├── Bayesian Linear Regression ├── Bayesian Linear Regression.ipynb ├── README.md └── movie_metadata.csv ├── Binary Logistic Regression ├── Binary Logistic Regression.ipynb └── README.md ├── Feedforward neural network └── README.md ├── Gradient Descent for Optimization ├── Gradient Descent for Optimization.ipynb ├── Images │ ├── GradDescent.jpg │ ├── GradDescentMult.png │ ├── LearnRate.png │ └── README.md └── README.MD ├── Investigating Experimental Data Using Linear Regression ├── Examine Experimental Data Using Linear Regression.ipynb ├── ExperimentRegression.pdf └── README.md ├── Latent Dirichlet Allocation ├── Figures │ ├── LDADoc.png │ └── README.md ├── Latent Dirichlet Allocation.ipynb ├── README.md └── YoutubeCommentsSpam.csv ├── Linear Contextual Bandits ├── Bayes Regression and Thompson Sampling Notes.pdf ├── Figures │ ├── Batch1.png │ ├── Batch2.png │ └── README.MD ├── Linear_Contextual_Bandits.ipynb └── README.md ├── Linear Model Selection ├── Linear Model Selection.ipynb ├── NYUSchoolData.csv └── README.md ├── Linear Regression ├── FIFA2018Statistics.csv ├── Linear Regression Note.pdf ├── Linear Regression.ipynb └── README.md ├── Naive Bayes Classifier ├── Figures │ ├── README.md │ └── vendiagram.png ├── Naive Bayes.ipynb ├── README.md └── YoutubeCommentsSpam.csv ├── Newtons Method for Optimization ├── Newtons Method.ipynb ├── README.md └── movie_metadata.csv ├── Nonparametric Regression ├── Nonparametric Regression Note.pdf ├── Nonparametric Regression.ipynb └── README.md ├── Principal Component Analysis ├── Figures │ ├── 2D PCA.png │ ├── PCA3Dto2D.png │ ├── PCAbasic.png │ ├── PCAcomponent.png │ └── README.md ├── Principal Component Analysis.ipynb ├── README.md └── diabetes.csv ├── README.md ├── Regularization in Linear Regression ├── README.md ├── Regularization in Linear Regression.ipynb └── movie_metadata.csv └── Self Organizing Maps for Data Visualization ├── Figures ├── README.MD ├── RGBreduce.png ├── SOMintuition.png ├── SelfOrgMap.png └── WorldPoverty.png ├── Grade1Students.csv ├── README.md └── Self Organizing Map for Clustering.ipynb /Bayesian Linear Regression/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | The notebook "Bayesian Linear Regression.ipynb" outlines the differences between the frequentist and bayesian approaches to linear regression. It shows how the bayesian approach to linear regression is analagous to regularization. Both the frequentist and bayesian linear regression approach is applied to the IMDB data set and it shown that bayesian approach seems to be fit the data more smoothly relative to the frequentist approach. 3 | 4 | ## Data 5 | I use the IMDB movie data set from kaggle and examine the relationship between IMDB rating and gross sales revenue using a frequentist multiple linear regression (degree 5), and bayesian multiple regression (degree 5). For analysis purposes I clean the data set to only contain IMDB movie rating and gross sales revenue for US movies. 6 | 7 | ## Dependencies 8 | * numpy 9 | * Pandas 10 | * Matplotlib 11 | 12 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 13 | 14 | ## References 15 | [Siraj's video on hyperparameters](https://www.youtube.com/watch?v=ttE0F7fghfk) 16 | 17 | [Video introducing bayesian linear regression](https://www.youtube.com/watch?v=dtkGq9tdYcI) 18 | 19 | [Frequentist vs Bayes Regression stackexchange discussion](https://stats.stackexchange.com/questions/252577/bayes-regression-how-is-it-done-in-comparison-to-standard-regression) 20 | -------------------------------------------------------------------------------- /Binary Logistic Regression/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | The notebook "Binary Logistic Regression.ipynb" discusses using logistic regression for binary classification. The key math and statistic ideas behind logistic regression are also included. Note that some of the LaTeX equations have trouble rendering on the browser, so please download and run on your computer. 3 | 4 | ## Data 5 | I use a de-identified data set of over 40,000 university students to predict whether a student will go on academic probation after their first year (2nd, 3rd, etc) using their first year performance and other characteristics. The data set comes from a research paper, the authors have made the data available through the [journals website](https://www.aeaweb.org/articles?id=10.1257/app.2.2.95). 6 | 7 | ## Dependencies 8 | * numpy 9 | * Pandas 10 | * Matplotlib 11 | 12 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 13 | 14 | ## References 15 | [Siraj's video on Math of ML](https://www.youtube.com/watch?v=8onB7rPG4Pk&t=41s) 16 | 17 | [Andrew Ng Intro to Logistic Regression](https://www.youtube.com/watch?v=-la3q9d7AKQ&t=3s) 18 | 19 | [Logistic Regression with Numpy (Kaggle)](https://www.kaggle.com/emilyhorsman/basic-logistic-regression-with-numpy) 20 | 21 | [Logistic Regression from Scratch (Medium)](https://medium.com/@martinpella/logistic-regression-from-scratch-in-python-124c5636b8ac) 22 | -------------------------------------------------------------------------------- /Feedforward neural network/README.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Gradient Descent for Optimization/Images/GradDescent.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Gradient Descent for Optimization/Images/GradDescent.jpg -------------------------------------------------------------------------------- /Gradient Descent for Optimization/Images/GradDescentMult.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Gradient Descent for Optimization/Images/GradDescentMult.png -------------------------------------------------------------------------------- /Gradient Descent for Optimization/Images/LearnRate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Gradient Descent for Optimization/Images/LearnRate.png -------------------------------------------------------------------------------- /Gradient Descent for Optimization/Images/README.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Gradient Descent for Optimization/README.MD: -------------------------------------------------------------------------------- 1 | # Purpose 2 | The jupyter notebook "Gradient Descent for Optimization.ipynb" implements gradient descent from scratch and approximates the global minimum of single and multiple variable functions. The notebook contains the mathematical details of the Gradient descent and shows implementation in python. This notebook does not rely on real data but instead focuses on understanding the algorithm as a tool to find minimums of a differentiable function. 3 | 4 | # References 5 | [Siraj's video on Evolution of Gradient Descent](https://www.youtube.com/watch?v=nhqo0u1a6fw&t=334s) 6 | 7 | [Gradient Descent Introduction](https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/) 8 | 9 | [Another Gradient Descent Introduction](http://www.big-data.tips/gradient-descent) 10 | 11 | [Gradient Descent for Regression](https://spin.atomicobject.com/2014/06/24/gradient-descent-linear-regression/) 12 | 13 | -------------------------------------------------------------------------------- /Investigating Experimental Data Using Linear Regression/ExperimentRegression.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Investigating Experimental Data Using Linear Regression/ExperimentRegression.pdf -------------------------------------------------------------------------------- /Investigating Experimental Data Using Linear Regression/README.md: -------------------------------------------------------------------------------- 1 | # Investigating Experimental Data Using Linear Regression 2 | 3 | ## Purpose 4 | We examine the effects of class size reduction on math achievement using a publicly available experimental data set using linear regression. The notebook also explores whether the effect of class size reduction varies by teacher experience. The PDF file contains slides discussing analysis of experimental data using linear regression. 5 | 6 | ## Data 7 | From 1985-1989 the state of Tennessee conducted a experiment to examine the causal impacts of smaller class sizes on student test scores. The data set from this study is now publicly available and can be found on dataverse. I use a kindergarten subset of this data set. 8 | 9 | ## Dependencies 10 | * numpy 11 | * Pandas 12 | * Matplotlib 13 | 14 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 15 | 16 | ## Contribution Opportunities 17 | At the end of the notebook I suggest multiple extensions, feel free to implement them in the notebook and do a pull request. 18 | 19 | ## References 20 | [Notebook on linear regression](https://github.com/hammadshaikhha/Math-of-Machine-Learning-Course-by-Siraj/tree/master/Linear%20Regression) 21 | 22 | [Introduction to causal inference](https://github.com/hammadshaikhha/Causal-Inference-Mastery/tree/master/Introduction%20to%20Causal%20Inference) 23 | 24 | -------------------------------------------------------------------------------- /Latent Dirichlet Allocation/Figures/LDADoc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Latent Dirichlet Allocation/Figures/LDADoc.png -------------------------------------------------------------------------------- /Latent Dirichlet Allocation/Figures/README.md: -------------------------------------------------------------------------------- 1 | Figures used in LDA notebook. 2 | -------------------------------------------------------------------------------- /Latent Dirichlet Allocation/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | The notebook "Latent Dirichlet Allocation.ipynb" discusses the Latent Dirichlet Allocation (LDA) unsupervised learning algorithm which is used for topic analysis in text data. It first assumes all documents in the text corpus consists of a mixture of a finite number of topics, and each topic is another mixture of similar words. Given this assumption, LDA uses bayesian inference to back out the distribution of topics and their associated words. 3 | 4 | ## Data 5 | The data used in the Naive Bayes notebook is from the UCI machine learning repository. It contains several comments from very popular music videos from youtube and has labels for each comment being spam or ham. The youtube spam collection data set can be found [here](https://archive.ics.uci.edu/ml/datasets/YouTube+Spam+Collection). For the purposes of applying LDA on this data set, I remove the spam classification labels to examine whether LDA is able to learn spam and ham comments on its own. 6 | 7 | ## Work in Progress 8 | Implement TF-IDF prior to applying LDA and check whether the words that appear in all topics resulting from LDA are ignored. 9 | 10 | ## Dependencies 11 | * numpy 12 | * Pandas 13 | * re 14 | * nltk 15 | 16 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 17 | 18 | ## References 19 | [Siraj's video on Generative Models and LDA](https://www.youtube.com/watch?v=HyuBTMaKFmU) 20 | 21 | [Great video introducing LDA](https://www.youtube.com/watch?v=3mHy4OSyRf0) 22 | 23 | [Video on Dirichlet distribution](https://www.youtube.com/watch?v=nfBNOWv1pgE) 24 | 25 | [Great answer on Quora about LDA](https://www.quora.com/What-is-a-good-explanation-of-Latent-Dirichlet-Allocation) 26 | 27 | [Implement LDA from scratch in R](http://brooksandrew.github.io/simpleblog/articles/latent-dirichlet-allocation-under-the-hood/) 28 | -------------------------------------------------------------------------------- /Linear Contextual Bandits/Bayes Regression and Thompson Sampling Notes.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Linear Contextual Bandits/Bayes Regression and Thompson Sampling Notes.pdf -------------------------------------------------------------------------------- /Linear Contextual Bandits/Figures/Batch1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Linear Contextual Bandits/Figures/Batch1.png -------------------------------------------------------------------------------- /Linear Contextual Bandits/Figures/Batch2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Linear Contextual Bandits/Figures/Batch2.png -------------------------------------------------------------------------------- /Linear Contextual Bandits/Figures/README.MD: -------------------------------------------------------------------------------- 1 | ## Introduction 2 | Figures used in the notebook are posted here. 3 | -------------------------------------------------------------------------------- /Linear Contextual Bandits/README.md: -------------------------------------------------------------------------------- 1 | ## Introduction 2 | The linear contextual bandit notebook discussion bayesian linear regression in comparsion with standard linear regression. Bayesian linear regression with conjugate normal inverse gamma prior is used together with thompson sampling to implement the contextual bandit. The purpose of the contextual bandit is to choose the action the maximizes the expected reward given the context of the user. 3 | 4 | The notebook can be opened with google collab and the code can be adjusted there. This repository also contains a PDF file (supplementary material) with detailed notes on the bayesian linear regression and thompson sampling procedure implemented in this notebook. 5 | 6 | ## Dependencies 7 | * numpy 8 | * Pandas 9 | * Matplotlib 10 | * statsmodels 11 | * scipy.stats 12 | 13 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 14 | 15 | ## References 16 | [Details behind bayesian linear regression](http://www.biostat.umn.edu/~ph7440/pubh7440/BayesianLinearModelGoryDetails.pdf) 17 | 18 | [Multiarm bandit vs. AB test](https://vwo.com/blog/multi-armed-bandit-algorithm/) 19 | 20 | [Contextual bandit introduction](https://getstream.io/blog/introduction-contextual-bandits/) 21 | -------------------------------------------------------------------------------- /Linear Model Selection/NYUSchoolData.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Linear Model Selection/NYUSchoolData.csv -------------------------------------------------------------------------------- /Linear Model Selection/README.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | The "Linear Model Selection.ipynb" notebook discusses 1) Subset selection, 2) Step wise regression and 3) Shrinkage methods (lasso and ridge) for selecting from a set of linear models. These methods essentially inform of us of the predictors that should be contained in the regression for optimal model fitness. 3 | 4 | # Data 5 | The notebook uses school level achievement and characteristics data from New York. The data is [available on kaggle](https://www.kaggle.com/passnyc/data-science-for-good). We use model selection methods to find the important determinants of school level math achievememnt. 6 | 7 | ## Dependencies 8 | * numpy 9 | * Pandas 10 | * Matplotlib 11 | * scikit-learn (just for lasso) 12 | 13 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 14 | 15 | ## Contribution Ideas 16 | If anyone is interested in working further on improving this notebook, here are some ideas: 17 | * Code lasso from scratch using numpy (don't rely on scikit-learn) 18 | * Perform cross validation to find the optimal tuning parameter for lasso. Compare the model selected by lasso to the one selected by forward selection. 19 | 20 | # References 21 | [Subset selection video](https://www.youtube.com/watch?v=91si52nk3LA) 22 | 23 | [Forward selection](https://www.youtube.com/watch?v=nLpJd_iKmrE) 24 | 25 | [Backward selection](https://www.youtube.com/watch?v=NJhMSpI2Uj8) 26 | 27 | [AIC, BIC, and adj Rsq video](https://www.youtube.com/watch?v=LkifE44myLc) 28 | 29 | [Ridege regression for model selection video](https://www.youtube.com/watch?v=cSKzqb0EKS0) 30 | 31 | [Lasso regression for model selection video](https://www.youtube.com/watch?v=A5I1G1MfUmA) 32 | -------------------------------------------------------------------------------- /Linear Regression/FIFA2018Statistics.csv: -------------------------------------------------------------------------------- 1 | Date,Team,Opponent,Goal Scored,Ball Possession %,Attempts,On-Target,Off-Target,Blocked,Corners,Offsides,Free Kicks,Saves,Pass Accuracy %,Passes,Distance Covered (Kms),Fouls Committed,Yellow Card,Yellow & Red,Red,Man of the Match,1st Goal,Round,PSO,Goals in PSO,Own goals,Own goal Time 2 | 14-06-2018,Russia,Saudi Arabia,5,40,13,7,3,3,6,3,11,0,78,306,118,22,0,0,0,Yes,12,Group Stage,No,0,, 3 | 14-06-2018,Saudi Arabia,Russia,0,60,6,0,3,3,2,1,25,2,86,511,105,10,0,0,0,No,,Group Stage,No,0,, 4 | 15-06-2018,Egypt,Uruguay,0,43,8,3,3,2,0,1,7,3,78,395,112,12,2,0,0,No,,Group Stage,No,0,, 5 | 15-06-2018,Uruguay,Egypt,1,57,14,4,6,4,5,1,13,3,86,589,111,6,0,0,0,Yes,89,Group Stage,No,0,, 6 | 15-06-2018,Morocco,Iran,0,64,13,3,6,4,5,0,14,2,86,433,101,22,1,0,0,No,,Group Stage,No,0,1,90 7 | 15-06-2018,Iran,Morocco,1,36,8,2,5,1,2,0,22,2,86,194,100,14,3,0,0,Yes,90,Group Stage,No,0,, 8 | 15-06-2018,Portugal,Spain,3,39,8,3,2,3,4,1,13,2,87,366,102,12,1,0,0,No,4,Group Stage,No,0,, 9 | 15-06-2018,Spain,Portugal,3,61,12,5,5,2,5,3,13,0,93,727,103,10,1,0,0,Yes,24,Group Stage,No,0,, 10 | 16-06-2018,France,Australia,2,51,12,5,4,3,5,0,19,1,87,484,103,16,1,0,0,No,58,Group Stage,No,0,, 11 | 16-06-2018,Australia,France,1,49,4,1,2,1,1,0,16,4,85,390,111,19,3,0,0,Yes,62,Group Stage,No,0,1,81 12 | 16-06-2018,Argentina,Iceland,1,72,26,7,9,10,10,0,15,2,92,718,101,10,0,0,0,No,19,Group Stage,No,0,, 13 | 16-06-2018,Iceland,Argentina,1,28,9,3,5,1,2,0,10,7,67,189,105,15,0,0,0,Yes,23,Group Stage,No,0,, 14 | 16-06-2018,Peru,Denmark,0,52,18,6,7,5,3,5,21,2,85,394,104,10,1,0,0,No,,Group Stage,No,0,, 15 | 16-06-2018,Denmark,Peru,1,48,10,3,5,2,7,3,15,6,82,342,110,18,2,0,0,Yes,59,Group Stage,No,0,, 16 | 17-06-2018,Croatia,Nigeria,2,54,11,2,7,2,6,2,17,2,84,462,104,20,2,0,0,Yes,32,Group Stage,No,0,, 17 | 17-06-2018,Nigeria,Croatia,0,46,14,2,5,7,5,1,22,1,84,388,101,16,1,0,0,No,,Group Stage,No,0,1,32 18 | 17-06-2018,Costa Rica,Serbia,0,50,10,3,3,4,5,1,18,2,83,428,107,18,2,0,0,No,,Group Stage,No,0,, 19 | 17-06-2018,Serbia,Costa Rica,1,50,10,3,5,2,4,3,19,3,83,392,109,15,2,0,0,Yes,56,Group Stage,No,0,, 20 | 17-06-2018,Germany,Mexico,0,60,25,9,9,7,8,1,17,3,88,595,110,10,2,0,0,Yes,,Group Stage,No,0,, 21 | 17-06-2018,Mexico,Germany,1,40,12,4,6,2,1,2,11,9,82,281,106,15,2,0,0,No,35,Group Stage,No,0,, 22 | 17-06-2018,Brazil,Switzerland,1,52,20,4,9,7,7,1,19,1,88,521,103,12,1,0,0,Yes,20,Group Stage,No,0,, 23 | 17-06-2018,Switzerland,Brazil,1,48,6,2,4,0,2,0,13,3,83,436,108,19,3,0,0,No,50,Group Stage,No,0,, 24 | 18-06-2018,Sweden,Korea Republic,1,52,15,4,5,6,6,1,23,0,84,417,102,20,1,0,0,Yes,65,Group Stage,No,0,, 25 | 18-06-2018,Korea Republic,Sweden,0,48,5,0,2,3,5,0,21,3,79,351,103,23,2,0,0,No,,Group Stage,No,0,, 26 | 18-06-2018,Belgium,Panama,3,61,15,6,7,2,9,1,21,2,89,544,102,17,3,0,0,Yes,47,Group Stage,No,0,, 27 | 18-06-2018,Panama,Belgium,0,39,6,2,4,0,3,3,18,2,82,317,100,18,5,0,0,No,,Group Stage,No,0,, 28 | 18-06-2018,Tunisia,England,1,41,6,1,3,2,2,2,11,5,82,326,103,14,0,0,0,No,11,Group Stage,No,0,, 29 | 18-06-2018,England,Tunisia,2,59,17,7,7,3,7,3,16,0,91,492,105,8,1,0,0,Yes,35,Group Stage,No,0,, 30 | 19-06-2018,Colombia,Japan,1,41,8,3,1,4,3,2,10,4,82,352,93,15,2,0,1,No,39,Group Stage,No,0,, 31 | 19-06-2018,Japan,Colombia,2,59,14,6,5,3,6,1,17,2,87,565,101,9,1,0,0,Yes,6,Group Stage,No,0,, 32 | 19-06-2018,Poland,Senegal,1,57,10,4,5,1,3,3,18,1,88,552,110,8,1,0,0,No,37,Group Stage,No,0,1,37 33 | 19-06-2018,Senegal,Poland,2,43,8,2,4,2,3,3,11,3,81,328,107,15,2,0,0,Yes,60,Group Stage,No,0,, 34 | 19-06-2018,Russia,Egypt,3,47,11,3,5,3,7,0,10,0,76,380,115,11,1,0,0,Yes,59,Group Stage,No,0,, 35 | 19-06-2018,Egypt,Russia,1,53,13,1,8,4,4,0,11,1,81,438,110,10,1,0,0,No,47,Group Stage,No,0,1,47 36 | 20-06-2018,Portugal,Morocco,1,47,10,2,4,4,5,1,24,4,76,387,105,19,1,0,0,Yes,4,Group Stage,No,0,, 37 | 20-06-2018,Morocco,Portugal,0,53,16,4,10,2,7,1,20,1,77,466,107,23,1,0,0,No,,Group Stage,No,0,, 38 | 20-06-2018,Uruguay,Saudi Arabia,1,47,13,4,6,3,3,1,15,3,88,521,101,10,0,0,0,Yes,23,Group Stage,No,0,, 39 | 20-06-2018,Saudi Arabia,Uruguay,0,53,8,3,3,2,4,2,11,3,86,590,100,13,0,0,0,No,,Group Stage,No,0,, 40 | 20-06-2018,Iran,Spain,0,30,5,0,5,0,2,2,15,2,69,219,106,14,2,0,0,No,,Group Stage,No,0,, 41 | 20-06-2018,Spain,Iran,1,70,17,3,6,8,6,1,16,0,89,805,105,14,0,0,0,Yes,54,Group Stage,No,0,, 42 | 21-06-2018,Denmark,Australia,1,49,10,5,5,0,3,1,5,4,88,458,112,7,2,0,0,Yes,7,Group Stage,No,0,, 43 | 21-06-2018,Australia,Denmark,1,51,14,5,5,4,5,0,8,5,85,520,114,5,0,0,0,No,38,Group Stage,No,0,, 44 | 21-06-2018,France,Peru,1,44,12,4,6,2,5,0,16,2,77,405,103,11,2,0,0,Yes,34,Group Stage,No,0,, 45 | 21-06-2018,Peru,France,0,56,10,2,6,2,3,1,11,3,81,532,102,15,2,0,0,No,,Group Stage,No,0,, 46 | 21-06-2018,Argentina,Croatia,0,58,10,3,3,4,5,3,26,2,81,505,101,15,3,0,0,No,,Group Stage,No,0,, 47 | 21-06-2018,Croatia,Argentina,3,42,15,5,6,4,2,3,18,3,80,372,104,23,4,0,0,Yes,53,Group Stage,No,0,, 48 | 22-06-2018,Brazil,Costa Rica,2,66,23,9,9,5,10,3,14,0,90,732,105,11,2,0,0,Yes,90,Group Stage,No,0,, 49 | 22-06-2018,Costa Rica,Brazil,0,34,4,0,4,0,1,3,14,7,73,271,109,11,1,0,0,No,,Group Stage,No,0,, 50 | 22-06-2018,Nigeria,Iceland,2,58,16,4,6,6,6,1,10,3,84,473,100,9,1,0,0,Yes,49,Group Stage,No,0,, 51 | 22-06-2018,Iceland,Nigeria,0,42,10,3,6,1,5,0,10,2,75,291,106,10,0,0,0,No,,Group Stage,No,0,, 52 | 22-06-2018,Serbia,Switzerland,1,42,12,3,7,2,3,0,15,3,78,309,116,17,4,0,0,No,5,Group Stage,No,0,, 53 | 22-06-2018,Switzerland,Serbia,2,58,20,5,8,7,7,3,17,2,87,547,112,12,1,0,0,Yes,52,Group Stage,No,0,, 54 | 23-06-2018,Belgium,Tunisia,5,52,23,12,8,3,5,3,16,3,84,468,104,12,0,0,0,Yes,6,Group Stage,No,0,, 55 | 23-06-2018,Tunisia,Belgium,2,48,15,5,6,4,2,3,15,6,82,477,102,13,1,0,0,No,18,Group Stage,No,0,, 56 | 23-06-2018,Korea Republic,Mexico,1,41,17,6,2,9,7,0,7,3,82,346,99,24,4,0,0,No,90,Group Stage,No,0,, 57 | 23-06-2018,Mexico,Korea Republic,2,59,13,5,6,2,5,0,24,5,89,485,97,7,0,0,0,Yes,26,Group Stage,No,0,, 58 | 23-06-2018,Germany,Sweden,2,71,16,5,4,7,8,5,15,5,91,699,111,12,0,1,0,Yes,48,Group Stage,No,0,, 59 | 23-06-2018,Sweden,Germany,1,29,8,6,1,1,3,2,17,4,77,213,110,13,2,0,0,No,32,Group Stage,No,0,, 60 | 24-06-2018,England,Panama,6,58,12,7,3,2,3,3,13,1,94,593,99,14,1,0,0,Yes,8,Group Stage,No,0,, 61 | 24-06-2018,Panama,England,1,42,8,2,5,1,2,0,17,1,88,398,89,13,3,0,0,No,78,Group Stage,No,0,, 62 | 24-06-2018,Japan,Senegal,2,54,7,3,2,2,2,2,18,5,84,449,105,8,2,0,0,No,34,Group Stage,No,0,, 63 | 24-06-2018,Senegal,Japan,2,46,14,7,5,2,5,4,10,1,79,338,102,14,3,0,0,Yes,11,Group Stage,No,0,, 64 | 24-06-2018,Poland,Colombia,0,45,9,2,3,4,7,1,11,0,79,424,107,15,2,0,0,No,,Group Stage,No,0,, 65 | 24-06-2018,Colombia,Poland,3,55,13,3,5,5,5,1,16,2,82,514,108,10,0,0,0,Yes,40,Group Stage,No,0,, 66 | 25-06-2018,Uruguay,Russia,3,56,17,7,6,4,4,0,20,1,88,492,101,17,1,0,0,Yes,10,Group Stage,No,0,, 67 | 25-06-2018,Russia,Uruguay,0,44,3,1,1,1,2,2,17,5,83,355,98,18,1,1,0,No,,Group Stage,No,0,1,23 68 | 25-06-2018,Saudi Arabia,Egypt,2,61,22,7,10,5,7,1,19,0,90,655,101,7,0,0,0,No,45,Group Stage,No,0,, 69 | 25-06-2018,Egypt,Saudi Arabia,1,39,8,1,6,1,2,3,8,5,82,357,106,16,2,0,0,Yes,22,Group Stage,No,0,, 70 | 25-06-2018,Spain,Morocco,2,68,16,4,11,1,7,1,18,1,91,762,103,5,0,0,0,Yes,19,Group Stage,No,0,, 71 | 25-06-2018,Morocco,Spain,2,32,6,3,2,1,1,1,6,1,84,247,111,17,6,0,0,No,14,Group Stage,No,0,, 72 | 25-06-2018,Iran,Portugal,1,32,8,2,5,1,1,1,12,3,69,226,93,16,2,0,0,No,90,Group Stage,No,0,, 73 | 25-06-2018,Portugal,Iran,1,68,14,4,7,3,5,1,17,1,89,619,89,11,4,0,0,Yes,45,Group Stage,No,0,, 74 | 26-06-2018,Denmark,France,0,38,5,1,2,2,4,1,11,4,72,305,106,10,1,0,0,No,,Group Stage,No,0,, 75 | 26-06-2018,France,Denmark,0,62,11,4,6,1,2,1,11,1,87,669,104,10,0,0,0,Yes,,Group Stage,No,0,, 76 | 26-06-2018,Australia,Peru,0,53,14,2,7,5,8,3,13,1,86,555,105,13,4,0,0,No,,Group Stage,No,0,, 77 | 26-06-2018,Peru,Australia,2,47,4,3,1,0,3,1,16,2,81,418,100,12,2,0,0,Yes,18,Group Stage,No,0,, 78 | 26-06-2018,Nigeria,Argentina,1,34,9,3,5,1,3,0,17,2,76,244,98,20,2,0,0,No,51,Group Stage,No,0,, 79 | 26-06-2018,Argentina,Nigeria,2,66,8,4,3,1,5,2,20,2,84,565,101,15,3,0,0,Yes,14,Group Stage,No,0,, 80 | 26-06-2018,Iceland,Croatia,1,41,17,6,8,3,10,1,12,0,73,324,103,10,3,0,0,Yes,76,Group Stage,No,0,, 81 | 26-06-2018,Croatia,Iceland,2,59,13,2,8,3,5,0,11,5,87,524,101,12,2,0,0,No,53,Group Stage,No,0,, 82 | 27-06-2018,Korea Republic,Germany,2,30,11,5,5,1,3,0,8,7,74,237,118,16,4,0,0,Yes,90,Group Stage,No,0,, 83 | 27-06-2018,Germany,Korea Republic,0,70,26,6,11,9,9,1,16,3,88,719,115,7,0,0,0,No,,Group Stage,No,0,, 84 | 27-06-2018,Mexico,Sweden,0,65,19,3,8,8,7,2,11,3,83,487,95,14,3,0,0,No,,Group Stage,No,0,1,74 85 | 27-06-2018,Sweden,Mexico,3,35,13,5,7,1,3,0,16,3,67,212,102,11,2,0,0,Yes,50,Group Stage,No,0,, 86 | 27-06-2018,Serbia,Brazil,0,44,10,1,5,4,5,2,11,4,78,467,114,13,3,0,0,No,,Group Stage,No,0,, 87 | 27-06-2018,Brazil,Serbia,2,56,13,6,3,4,9,4,15,1,88,631,105,7,0,0,0,Yes,36,Group Stage,No,0,, 88 | 27-06-2018,Switzerland,Costa Rica,2,60,12,3,5,4,6,1,15,5,87,594,103,9,3,0,0,Yes,31,Group Stage,No,0,1,90 89 | 27-06-2018,Costa Rica,Switzerland,2,40,14,6,6,2,5,1,10,1,82,348,103,14,3,0,0,No,56,Group Stage,No,0,, 90 | 28-06-2018,Japan,Poland,0,54,10,3,4,3,5,1,8,2,86,557,83,11,1,0,0,No,,Group Stage,No,0,, 91 | 28-06-2018,Poland,Japan,1,46,11,2,5,4,7,0,12,3,83,462,80,8,0,0,0,Yes,59,Group Stage,No,0,, 92 | 28-06-2018,Senegal,Colombia,0,43,8,3,4,1,1,3,18,1,78,281,97,15,1,0,0,No,,Group Stage,No,0,, 93 | 28-06-2018,Colombia,Senegal,1,57,4,2,1,1,3,3,18,3,83,431,100,15,1,0,0,Yes,74,Group Stage,No,0,, 94 | 28-06-2018,Panama,Tunisia,1,36,9,4,4,1,0,4,20,5,77,284,102,18,3,0,0,No,33,Group Stage,No,0,, 95 | 28-06-2018,Tunisia,Panama,2,64,15,7,4,4,6,1,22,4,87,626,107,19,3,0,0,Yes,51,Group Stage,No,0,1,33 96 | 28-06-2018,England,Belgium,0,48,13,1,7,5,7,3,15,3,88,484,109,11,0,0,0,No,,Group Stage,No,0,, 97 | 28-06-2018,Belgium,England,1,52,15,4,4,7,2,1,14,1,88,556,106,14,2,0,0,Yes,51,Group Stage,No,0,, 98 | 30-06-2018,France,Argentina,4,41,9,4,4,1,0,0,16,1,84,351,97,21,3,0,0,Yes,13,Round of 16,No,0,, 99 | 30-06-2018,Argentina,France,3,59,9,4,1,4,4,1,21,0,86,547,96,15,5,0,0,No,41,Round of 16,No,0,, 100 | 30-06-2018,Uruguay,Portugal,2,39,6,3,2,1,2,0,14,4,69,269,106,13,0,0,0,Yes,7,Round of 16,No,0,, 101 | 30-06-2018,Portugal,Uruguay,1,61,20,5,1,8,10,1,13,1,84,583,107,13,1,0,0,No,55,Round of 16,No,0,, 102 | 01-07-2018,Spain,Russia,1,75,25,9,6,10,6,1,20,0,91,1137,137,5,1,0,0,No,12,Round of 16,Yes,3,, 103 | 01-07-2018,Russia,Spain,1,25,6,1,3,2,5,1,6,9,72,284,146,19,2,0,0,Yes,41,Round of 16,Yes,4,1,12 104 | 01-07-2018,Croatia,Denmark,1,54,22,7,8,7,5,2,19,2,81,626,132,5,0,0,0,Yes,4,Round of 16,Yes,3,, 105 | 01-07-2018,Denmark,Croatia,1,46,15,3,10,2,4,0,7,7,79,539,135,19,1,0,0,No,1,Round of 16,Yes,2,, 106 | 02-07-2018,Brazil,Mexico,2,47,21,10,7,4,8,0,20,1,83,398,92,6,2,0,0,Yes,51,Round of 16,No,0,, 107 | 02-07-2018,Mexico,Brazil,0,53,13,1,4,8,7,2,6,8,85,424,89,18,4,0,0,No,,Round of 16,No,0,, 108 | 02-07-2018,Belgium,Japan,3,56,24,8,10,6,10,1,10,3,87,621,108,13,0,0,0,Yes,69,Round of 16,No,0,, 109 | 02-07-2018,Japan,Belgium,2,44,11,4,4,3,6,1,14,5,83,453,109,9,1,0,0,No,48,Round of 16,No,0,, 110 | 03-07-2018,Sweden,Switzerland,1,37,12,3,6,3,3,1,13,4,73,271,105,11,1,0,0,Yes,66,Round of 16,No,0,, 111 | 03-07-2018,Switzerland,Sweden,0,63,18,4,5,9,11,0,12,2,84,599,103,13,2,0,1,No,,Round of 16,No,0,, 112 | 03-07-2018,Colombia,England,1,48,14,4,7,3,2,1,15,1,78,516,136,23,6,0,0,No,90,Round of 16,Yes,3,, 113 | 03-07-2018,England,Colombia,1,51,16,2,9,5,7,2,24,3,82,571,143,13,2,0,0,Yes,57,Round of 16,Yes,4,, 114 | 06-07-2018,Uruguay,France,0,42,11,4,6,1,4,0,15,0,67,322,103,17,2,0,0,No,,Quarter Finals,No,0,, 115 | 06-07-2018,France,Uruguay,2,58,11,2,7,2,3,0,17,4,81,524,99,15,2,0,0,Yes,40,Quarter Finals,No,0,, 116 | 06-07-2018,Brazil,Belgium,1,57,26,9,7,10,8,1,16,2,89,557,104,14,2,0,0,No,,Quarter Finals,No,0,1,13 117 | 06-07-2018,Belgium,Brazil,2,43,8,3,3,2,4,0,15,9,83,370,105,16,2,0,0,Yes,13,Quarter Finals,No,0,, 118 | 07-07-2018,Sweden,England,0,43,7,3,3,1,1,2,8,0,74,379,107,10,2,0,0,No,,Quarter Finals,No,0,, 119 | 07-07-2018,England,Sweden,2,57,12,2,4,6,6,1,12,3,80,525,110,7,1,0,0,Yes,30,Quarter Finals,No,0,, 120 | 07-07-2018,Russia,Croatia,2,38,13,7,4,2,6,1,18,1,69,399,148,25,1,0,0,No,31,Quarter Finals,Yes,3,, 121 | 07-07-2018,Croatia,Russia,2,62,17,3,10,4,8,0,26,5,82,753,139,18,4,0,0,Yes,39,Quarter Finals,Yes,4,, 122 | 10-07-2018,France,Belgium,1,40,19,5,8,6,4,1,17,3,86,342,102,6,2,0,0,Yes,51,Semi- Finals,No,0,, 123 | 10-07-2018,Belgium,France,0,60,9,3,5,1,5,1,7,4,90,629,102,16,3,0,0,No,,Semi- Finals,No,0,, 124 | 11-07-2018,Croatia,England,2,54,22,7,11,4,8,1,17,0,79,622,143,23,2,0,0,Yes,68,Semi- Finals,No,0,, 125 | 11-07-2018,England,Croatia,1,46,11,1,6,4,4,3,24,5,79,479,148,14,1,0,0,No,5,Semi- Finals,No,0,, 126 | 14-07-2018,Belgium,England,2,43,12,4,3,5,4,1,5,5,88,510,108,11,1,0,0,Yes,4,3rd Place,No,0,, 127 | 14-07-2018,England,Belgium,0,57,15,5,7,3,5,0,12,2,92,698,110,5,2,0,0,No,,3rd Place,No,0,, 128 | 15-07-2018,France,Croatia,4,39,8,6,1,1,2,1,14,1,75,271,99,14,2,0,0,Yes,18,Final,No,0,1,18 129 | 15-07-2018,Croatia,France,2,61,15,3,8,4,6,1,15,3,83,547,100,13,1,0,0,No,28,Final,No,0,, -------------------------------------------------------------------------------- /Linear Regression/Linear Regression Note.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Linear Regression/Linear Regression Note.pdf -------------------------------------------------------------------------------- /Linear Regression/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | The "Linear Regression.ipynb" notebook discusses linear regression as a method to model the statistical relationship between and out Y and predictor(s) X. I discuss how the linear regression relates to the conditional expectation function. 3 | 4 | My notes on linear regression related to this notebook can be found in "Linear Regression Note.pdf" 5 | 6 | ## Data 7 | Linear regression is used to model the relationship between the number of goals scored and attempted shots on net in FIFA. I use the [FIFA 2018 data set](https://www.kaggle.com/mathan/fifa-2018-match-statistics) from kaggle: 8 | 9 | ## Dependencies 10 | * numpy 11 | * Pandas 12 | * Matplotlib 13 | 14 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 15 | 16 | ## Contribution Ideas 17 | If anyone is interested in working further on improving this notebook, here are some ideas: 18 | * I am fimiliar with linear regression, so I didn't look into many references. Feel free to update references below with good quality relevant notes/videos. 19 | * Compare linear regression and CEF with two explantory variables. This will require 3D plots. 20 | 21 | ## References 22 | [Simple Linear Regression Video](https://www.youtube.com/watch?v=KsVBBJRb9TE) 23 | 24 | [Estimate Linear Regression Video](https://www.youtube.com/watch?v=coQAAN4eY5s) 25 | 26 | [Square loss video](https://www.youtube.com/watch?v=AihhnWyl-J0) 27 | -------------------------------------------------------------------------------- /Naive Bayes Classifier/Figures/README.md: -------------------------------------------------------------------------------- 1 | Figures used for Naive Bayes Classifier notebook 2 | -------------------------------------------------------------------------------- /Naive Bayes Classifier/Figures/vendiagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Naive Bayes Classifier/Figures/vendiagram.png -------------------------------------------------------------------------------- /Naive Bayes Classifier/Naive Bayes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Naive Bayes for Spam Classification\n", 8 | "\n", 9 | "## Introduction\n", 10 | "\n", 11 | "Naive bayes is a relatively simple probabilistic classfication algorithm that is well suitable for categorical data (probabilities can be compuated as simple ratios) and uses the bayes theorem together with a strong (hence \"naive\") independence assumption. The basic idea behind Naive Bayes is that it assigns a probability to every category (finite outcome variable) based on the features in the data and chooses the outcome that is most likely as its prediction.\n", 12 | "\n", 13 | "The \"Naive\" in the name refers to the algorithm assuming features in the data are independent conditional on the outcome category. For example suppose we were doing spam text classification, then given a spam text \"Free, sign up now!\", Naive Bayes would assume \"Free\", \"sign\", \"up, \"now\" all occur indepedently of each other (that is $Pr(Free, sign, up, now|spam)$ = $Pr(Free|spam)\\times Pr(sign|spam)\\times Pr(up|spam) \\times Pr(now|spam)$). This conditional independence assumption is considered to be a strong assumption that often doesn't hold in practice, hence the resulting probabilities from Naive Bayes are not to be taken too seriously. However the classifications resulting from Naive Bayes can still be accurate. \n", 14 | "\n", 15 | "In machine learning, common application of Naive Bayes are spam email classification, sentiment analysis, document categorization. Naive bayes is advantageous over other commonly used classification algorithms in its simplicity, speed, and its accuracy on small data sets. Since Naive Bayes needs to be trained on a labeled data set it considered to a supervisd learning algorithm. " 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "## Introducing Bayes Theorem\n", 23 | "\n", 24 | "Consider two events, $A$ and $B$. For example $A$ could be a set of words found in many emails, and $B$ could be a set that has two categories, spam or ham (legitimate email). Then we might be interested in computing the $Pr(A = FREE|B = spam)$, which is the probability of a spam email containing the word \"FREE\". In general, let us try to come up with a formula for the conditional $Pr(A|B).$" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 10, 30 | "metadata": {}, 31 | "outputs": [ 32 | { 33 | "data": { 34 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAAEsCAYAAAA7Ldc6AABTsklEQVR42u2dC9AV5X3/Ey+taW2D\nranE0MbG/o11GEclSZmKDjOaDM3gJWoVI6YU0UFCvRtRSdQMCWpsKkgsiTjBShRvDRhUbIhiBANG\njRdUNCjeuN/vd/b/fh/8Lc857+45e87Zc96zu5/PzDPwnsue3efZy/N9frdPBAAAAAAAAC3iE3QB\nAAAAAAAgQAAAAAAAAAECAAAAAACAAAEAAAAAAAQIAAAAAAAAAgQAAAAAABAgAAAAAACAAAEAAAAA\nAECAAAAAAAAAAgQAAAAAAAABAgAAAAAACBAAAAAAAECAAAAAAAAAIEAAOrF27dqGt7Fz585UtgMA\nAAAACBBoY5544onguOOOC/r379+pDRgwILj++uuD3/3ud7Hfv+mmm4L9998/eO2118Lt6e9JkybF\nfufDDz8MjjjiiOD4448PNmzY4F579dVX3fdGjx7dpf2xevXq4Oijjw7uvvtuTg4AAABAgACkzcMP\nP+wm/tXaDTfcEOzevbvkuy+99JJ776KLLgrfs+3deeedsb/55ptvus8ceuihbsIv9P0LLrjAvf7y\nyy93SV/s2LEjOOOMM6ruPwAAAAACBKBBAXLNNdcEb7/9trNEWJs5c2bQr1+/UITMnTs3/N6uXbuC\nk046yb3+1ltvddpepQm8fqdcgPjC5PTTT3cuWa1k8eLFQe/evcNjRYAAAAAAAgSgiQJkwoQJke9v\n3bo1OPHEE91nfvKTn4SvP//88+61Sy65pMQy0ogA0XZkTdF7L7zwQkuOX795zz33dLL4IEAAAAAA\nAQLQRAFSacJtosA+47tLzZkzp+btxQkQ8cwzz7j3zjvvPGdliWPFihXuN+66667YJsH0hz/8oeLx\nb9y40cWj6De//e1vByNGjECAAAAAAAIEoKsEiMRGeVzEwoUL3d+HHXZYJwHRqADR3xIEBx10UPDu\nu+/GbuONN95IFLvywx/+sOLxK+7j4osvdsJHPProowgQAAAAQIAANFuA/Nd//ZcTG3K5srZo0aLg\nuuuuCyfz06dPd9+ZPXt2GHxebqVoVID41pVKblj63i9+8Ytg8uTJsU2ZuObNm5eqIAMAAABAgACk\nIECqtXPPPTcMDH/ooYdi40YaFSDi2muvde9LRHRVfyBAAAAAAAEC0AUCRCLhvvvuK7F06O+4SXoa\nAiTJNuQ69d5771Vs77zzTrB+/XoECAAAAAACBNpNgNxyyy3B5s2bXTVyNQmDVatWdfq8n6mqkgCR\nxSQuiDwNAWIpe6u1H//4xwgQAAAAAAQItJsAqWXCPX78+NjvmHuWsliVFy40rIBhXL2PShYWQ4Hw\nEjCq3RHX9P6DDz6IAAEAAABAgECWBYh95/bbb+/0ngoYmnUjLovVpZdeWlGADB8+vMtEAAIEAAAA\nECAAbTbhtlodEhDlblaqq3H00UeHaXqfe+65YMuWLe49ZdUaOXJk6B4VFWQuq4kJkGeffRYBAgAA\nTUHuxo2yadOm8BnXDtsBQIBAbgXIkiVL3HdUr0NxI3ECpVKTeFEgeTkmYGRBWb58OQIEAAA68fjj\nj7tnxb/8y790av/6r//qsilqASyOG264IfjEJz4RvPbaa+5vxQvGbe/CCy90z4Zt27aVbEOC4aij\njgpOOeWUcDHuww8/dM/GuP266aabgvnz55dsRwtvZ599dvA3f/M3wQcffNBlfar9OP/884PBgwdX\nLAQMCBCAhpkyZYqbcN9999013aR0s9T3nnrqqcjPvPjii+Fn/KYCgz//+c873ciNGTNmJKqE3syH\nWlyKYQAAaA8UbygBUa1997vf7RSPqDhEvaeaU/besGHDqm7rhBNOKLFS6P8SLXrW2fNKCVKS7NeT\nTz5Zsk8SQnp9wIABXTb5Vz0w7YMvqAABAtBWmJVDGbHigs2FLBrLli1zGbVWrlxZ8bN+EcLf/e53\ndDIAAFQUIFdffbXLrKj4Q2tPP/108NWvfjWc7M+ZMyf8nibWffv2da+/9dZb4ev/8R//4V6TEPG3\np8K7P/jBD8JtaQGtkgDRd/W5v/iLv3ALdCqGq+384Q9/CKZNm+YsJnq/R48eJWni9fwzEeTvbytQ\n4eErr7wyPEYECAIEoG2R+9SJJ57oxILqbaTBggUL3Pb0cIhyzwIAAPAFyF133RU7qT7++OPdZ+64\n447w9blz57rXFGvoL4iZAInbnlkHfKtJJQEigbFu3bpO21mxYoVztdJnJE583njjjVAARCVoaQa/\n/e1vw/2x5h8PIEAA2g7510owjBgxIpXtXXPNNW57ekAAAABUEyDjxo2L/YzEgv8ZCYdBgwa518qt\n7CZA4rY3derUTi5SlQSIJvVRda62b9/uvqPPvPzyyyXv+daZ8vd8tEAnS8xPf/rT4Gc/+1lk03Fo\nnyth7tT6vS9/+cvB2LFjESAIEIBsYKLh9ddfb2g7VljwxhtvpFMBAKAhAaLJ9WmnnVbyGdWPMutE\nuTioJECUwdFcunwLSTUBEmUBuffeeyu+rzpb5loWh1yby60WUU3uXtUsKbLsaJ/0OesfXLAQIABt\nj25aylbVqMuUgtKjKq8DAADECZD//M//dGJDLlfWJBhkmbeJ+BNPPOG+M2vWrNCNqnyCbQJEzc9c\nddxxx5UEofvPukoCRK08q5YvDn7zm99EHpcFo1cSAdqHRx55JLj//vsrtrjfiMP2HQGCAAEAAACA\nGAFSrUkcmBXgwQcfjI3z8AVIXPvCF74QfPTRR4kFSKWmfYhKymJWCP3Whg0bWtqnCBAECAAAAADU\nKUDkovSLX/yiZCKtv+PcrEyAqAaGgsGVqlfZq55//nmXpt5cnvzYjmoCRFmwLJuWsmD9+te/DmNQ\n4vbDthkXQ2KoXsh7771XsS1duhQBAggQAAAAgDQFyOjRo11BXFU1V9OkPcqd19K8VxMgcVmwZJkw\nEfL73/++qgApT7Pro1jHuIl+EgGiGBBZSJLULaklmxYCBAECAAAAAFUESKUsWOXceeedVQVIkqD2\ne+65p6oAqSQgZFWJEykSFwoer/R9xblYfIqyV0U1vac4mEq1txAgCBAAAAAAaKIAse/8+Mc/jhUg\ncRYQv36HFQqspw6I0O/H1fuwSurVXLCaAQIEAQIAAAAAKQqQmTNnxk6wTYB85zvfCZYsWeJiKBRw\n/v7777vvWQVzVTh/9913qwoQfe7FF190GblsW/Pnzw9uueWW0EUqKtWuCZDyjFsIEECAAAAAAGRM\ngEhYWIYpxY34DBs2LFH2KtXMMBrJgqV9WLNmTad9tED5KCsNAgTaVoAcfPDBiU58Go1Go7Vv6969\nO09JgAr88pe/dNfKhAkTEn9H8RBnnnlmZB0OCwyPaoqpUNHd8urkUQLE0uhGNVlF9FlVMd+0aVOn\n/ZPF4/jjj69aCb1Z2L771d4BAZJso5/AsAIAkPkHBPdygKZgbljKiFVLgHYUUQKkEZSu1ywQrXa/\nAgQIDy0AAAQInQDQBHwrwzvvvNNWAsTiUJ555hkGChAgAACAAAHIC7Nnzw4DzttFgCxYsCA2MxYA\nAgQAABAgABlHGah0nb3++ut1b8MKAipjVSMCRK5gEjF+hi0ABAgAACBAAHKErAzLly9vONZCtTpU\ngb1RtB1ZVAAQIAAAgAABAAAESCMPrX322Sc2PZzeq/QZex8AABAgAACAAEn00KqWe77aZwAAAAEC\nAAAIkMQPrUoWkE9+8pMVP4MFBAAAAQIAAAiQuh5aSSwbWD4AABAgAACAAEGAAAAgQAAAABAgAACA\nAAEAAAQIAgQAAAECAACAAAEAAAQIAAAgQBAgAAAIEAAAQIAgQAAAECAAkCXWrl0bPPPMM6k2bRMg\n9wLE6oMAAAACBAACJwSmTp0afP/733ftzDPPDE466STX9t9//7AdfPDB4etpNW3T/w17/YILLgj3\nR/umfQRoKwEiUeELkCOOOKLk/X333Td871Of+hSjBQCAAAEoFLI0mMjQ5N6f/Pfq1cv9bRP+e+65\nJ5g5c6Zra9asadk+6rfsd7UPtj/aN+2jL4JMoOiYAAHSJQ+tYcOGxVZET2IdAQAABAhAnsSGTeBl\nzfCtC3ptzJgxbpK/cOHCzB2b9ln7rmPQsfjCRMdqogTXLgRISx5an/vc56qKjzlz5jBSAAAIEIBc\n8d577wVjx44tcZmyybiESBaFRj3CpFyUWD+oDxAkCJCmPrR0svkuWfp///79GSEAAAQIQC54+eWX\nneD40pe+1Gmi3UqXqayIEt8KpP+r7yTaAAHCQwsAAAECABWQwPiHf/gH5240ePBgBEcdgkR9ZoJE\nAk5iDhAgPLQAABAgABDsjeXQhFmiQ+JDf0PjSLjJXcv6VUHtBLQjQHhoAQAgQAAKiVLOakJsGark\nRsRKffNQUPsVV1wRWpf0f1y0ECAAAIAAAcg1mvBqRd6fBBcheLwdxYjc28xFS/EiBLAjQAAAAAEC\nkBvkUmXZqyyuA7oeuWj5YyOLFMUQESAAAIAAAcgkWlHXyrqsHWpysSKYvH2RJUoWKSuCiEhEgAAA\nAAIEIDPCwwKfmchmD4lEiUUTjowfAgQAABAgAG2J4jssqFzCQ3EGkG38lMgSlcSJIEAAAAABAtDl\nmMXD4jsIKs8fEpMSlQgRBAgAACBAALoUrZCbqxUpdIslRAhWR4AAAAACBKClwsNiBKZMmUKHFFh4\nIkQQIAAAgAABaBqK81DFcoKTwa+yrtgfihoiQADgYz766KNgzpw5wcMPPxzcfvvtwY033hi2QYMG\nhe30008P+vbt61qfPn3C/6vpPfvckCFDSrahbWrbL7zwQrB06VI6HAECkEvK4zxIpwuGYn6sqKHS\nLgMCBOpk48aNTslbk8+jtUmTJgUTJ050bcKECeH/J0+eXPI5//tbt26lU5vEvHnzQnFx1VVXBQMG\nDHAC4rDDDnPXVqvbfvvt535bwmXgwIHBiBEjgnHjxrl9nD9/PgOGAAHIHHKxkcWDOA+ohM4NnSeq\nrI5bFgIEItBKtVasNSm87bbbgssuu8ytch9zzDHOlNiMiam2q4vyrLPOcr+nCbP8ZrUfK1euZFCq\noD6SsFO/yQqhvjzggAMS9bvGtX///sHQoUNLrBe+gNS54AtIv+k9+9z48eNLtqF96devX9CzZ8+g\nW7duVffnwAMPdPuufZEwmTVrFtlEECAAbYnuTXKt0X1UdSEAkmBuWSpqCAiQQiJLhiZ4muhpoqiJ\nqFaok65iW/NdcrTC7rvuWJOw8F14/O8nESiaTGtiqv3U/kqYaP+LiCxGM2bMcNYDCYfu3bvHjpPG\nVJYGc42SSNB3FyxY0CWWJ42ZLB3aB+2LiRSdN5XOvx49erhj1TFL9GA1Q4AAdCVm9VC8B2l1oVZ0\nzvTq1cudQ1hDECC5X6mZPn26s2hIDGhFOm4F+sgjjwxOPvlkJxw0QdSq9rRp05rqx69YBG1fVg/9\n3siRI93vS7Bof7RfUfur49AEWyv/mtTmVZSob26++WY3LlGWDbMcDB8+3FkuZOrN4iRd+6xjlcjU\nscRZcnS8sqzofMblAQEC0EpsBRurBzSKziGdS8SGIEByw86dO511QwKid+/ekSvLOuk1iVNsQBZ8\n8BXPoBgT7a8m4lEuPTpOiZZRo0a54OqsotgYCQlZBqLc3rRqIjcl9Yf6Je/nskSGLCYSpVFWM1mB\nJET1GTKNIEAAmoHuQ1oU0f0XqwekeV5ZbAgLagiQTCK3GlkPFK9RbjGwFWMJElkZZHHIA5ps6nhk\nMYmyDkikyOKjfmn3ianGT8JJrkjlE2xNujX5ZoK9d9zVFxIdUS5o6kP1pfoUECD1iF5ZU9W2bNnC\n4IFzk7Eq1wDNwCxruGQhQDKjnBW0HbUqLDVdNJ95HadczWQliXIzUz+pT9rFahAnOnQTkvVDVhAE\nR3VkvZPIVJ+Vi2/FHMlFLy+iGwHSPF555RV3f1C6TL8p/khWYgkTH62CH3roocF1110X7N69u22P\na8eOHcF5550XfP3rX3fH4rdTTz3VBcM+++yznKQxyD3moIMOcueCJokkxoBmYQUMcclCgLQlmkgp\nHqB8gq1AXa2SK+UtmaP2oJiVuJVyTfq7YmIqQaHYhXLRoYmz9lMCqnyiA7WJUFnF1JeIEQRIEiQe\nrrnmmk7Co7ydeOKJJXFmb775pntdbp/tfM1qn48++uiqx3fLLbdwonpIaFhRwW984xthP+nvqVOn\n0kHQFMwlS+ceYhcB0hYPEK2G60FXvlKuWADFe0B1ZA1S1iV/Yqq4EblvKZ6imUHsCuTX70SJDk2Y\nyezUPDEiy0i5e55WfyX2AAEi65lNLnWuyEK6YcMGZzlYtGhRcMcdd4TvS6gYixcvDr73ve8F9913\nX1v3vdzIjjvuOLf/d999txNOr776qmtPP/20SyVrx/faa69xsgZ74z00CXzsscciBZvew0INzUCF\nLHV+EReCAOkylBFIMQz+5En/12uaWLFSXv/EVC4VipfxA/RNEKR1wWv1QivuWs3wBY8mOYiO1ot4\niczyMZclURNQxqKYAkRuVDahlGU5zpXqwQcfdJ+Ry9Xy5ctT3Yd169YFq1atSrzaqc+qJT1nfQES\ndW/btWtXcMYZZ7j377333sKfq+XxHiowGGc1Ii4EmglxIQiQliJRocmp3EXK3UdkBcEkly5yV1Pa\n1/L+lrVCVot6UGyHYnP8TF1yAVOwfLNSGUNytGqpOCF/fPR/vcaKZrEEiHytNZE86qijgu3bt8d+\nTu9pEi8B8sYbb5SIl0svvdRN4s2NK+q+8dJLL7n3zj333HDh6MMPP3Rus/6EVlkLZZnwueGGG4Lj\njz8+eP755937/ud/9KMfVV2I8gWIFrWi0P3KLCRFxnzw9QwW5oZVzX2NlWpoFjoXdU7q3AQESFPQ\nCm35armsHXIZynuq1XZBDxBZQPwVctUhSbpCrmBoufaUx5pIOLLC3n5oTDS2GmPfQiULIxm08n8v\nl6hQXIcmkPfff3/Vz2/atKnk77ffftt9V1Y1iQBZ2OxvCRKfm266qSTOwuJHfNcviRv7W65Rhmri\n+J81MWFN95dGBIiEjf12kSfRttoc1QdWeLCaECFIHZo1N8HahgBpivBQ5pXy1XKlzGW1vGtQv5eP\niS5+jUmUkNCquVYyTbjoX01CiM3JDooH8cWjxlATP67B/N7L/eDseibevgCR4FiyZIn7WxmT3n//\n/fBzcrE64ogj3HtvvfWWc/MylycJD0saopgTxZTodX1ecShCFhZz/5o7d2742auvvrpEACURINo3\nPwuWL3pGjx5d2PNTcTCqTC3f+yQipZIIIUgdmoHOTZ2jOlcBAdIQemBo9dXPzlTLajs0H41DuVVK\n2caUWUvjp8mpXBcsRkeTVgkR3HiyiywfvpjU2Mp1jlXN/N3Lq1kGfvjDH3ZKXStXKAUlRwkQCYuL\nLrrIvfY///M/4XZmzJgRZsuScLDvafKvWA4fbcfcrGyfTIAow6HPH//4x9Ctq9ziEnecldqFF17Y\nycqD+AgiF5ySuGWRxQgQIQiQtkM+ff6kVm469cYbQGvEosbMd9WRcPSzaeG2ky+0Iq6JpR8jopot\nLA7kS4BUsoBoYh81sbzzzjsjBYiQq46l7JXYkCixLFMPPPBAiXBQGzx4cHD99deHTTVF7L3f/va3\nJQKkfB+jfr+aAJG7llx6FZOiWJPZs2c7y4f9ZjVrSp6QOKhVfPgkccuStQRrCDRLhCBwESCJmTNn\nTkkqXRXG02o6ZAO5UshlYp999gnH8K//+q9dhhzIJ3Kj869ZCU9ZxRAi+RAgNjEvty4IuUtpom6T\ndbNMVBIgvluX3l+xYkWn7FmPPvpo1dVzy8rlC5ByK009AiTO1eyJJ54I93P16tWFOCc1gVOGq3rE\nhy9iVMix2ljqM0wWIU0RonMXSwgCpCoy2WqF3F9NVTE6JjHZQTVE/Irzf/d3f1diAZHbDgXu8ovc\naPzikbKGaUEBsn0vt+BwWSwqZcES1157bVUBIpSZSq/fddddwa9+9Sv3/0suuSRM8WsB6Irz0D2j\nvCkmRO6dNmFNU4DEZcGS6ChSIHoa4sPH6oZUiw0hUxYgQhAgLUMWDgtmlj+5Un2yEpIdtKKpYGTf\naqUaIkKTBL1n8QIyt9t7kE80vha3pXHX9cxCQnbv5Sq8V25xiELuVBY4Xk2A2Ot+88WqCRBN+BW4\n7iORokD0888/320nbQESNwE217EiWEAacbuqhoLUq1lDlPoZIC0RQkwIAqQTWsnys+ro/wQnZwtl\nRjKrh2VFiqqWrtgP301H1i7LbAP5QwsIsnj51hBZyCCb93KlxrXJ4dlnn+0m+lYJXUHijz/+eJjF\nSk0ueJUEgP7vF6/Tdzdv3hy+76f/PfXUU0vuFarDYd979913UxcgjzzyiFs40bNIzyjVMnnooYfC\n31Q9FH2+SOJje8rV35NYQwhQB0QIAqQp+FYP/UucR/YnmEncbVTQ0NyysIbkH4kOP5lEnECF9r6X\ny+owbNiwRHEZSs9tmaLK64D46NovFyw+L774Ysl2lT3LFzlK+W1YHZB6BYgfl1KtvfLKK7k9B1XI\nrVx87F63Llh67LGurb/11mDnBx+k9nvVYkNwycrXs0DXbFRT4gclsWlmcgcTIUUtVogACaKtHsQF\nZAvdKJRq11zmdAOp5cahlUWsIcVB7ldywzI3PFnMFC8C2buXa4KvrFRRk0Vlp1KhUR/LZiXxUC4A\n5MYkQaG6G2bJKEcCol+/fiW/o8+roKG/PauwXi4OVEk9SRpenaPlFdT9plTDiklctGhRbs8/uZjF\nFRkUWx5/PFh1/vnBoo7P6F/9nebvVhIiVLjOPpon+AWI/fbJT34y+PM///Pg05/+dJjZrhlYsUKd\ncwiQggkQrB7ZRiJDE0m7aeiBXT7hqAWsIa2lq1egzO3Czh9/BRuydS/ftm1bGASuxQO5YjUTuXnp\n95YtW9b03yoiNjFLYm2QBUSWELOKrL3++oatIpbuF5esfD9/5Hb5J3/yJ+EzQAuRWpz4yle+EtYK\n+7M/+7OmLkrXcq4jQHLw0NKkRmZ5rB7ZRQ9/32qhwnNpUG4NYVLaPNphBUr3Aj9hgVa3mVRkT4BA\nftD1p4WBeqwMsoKs6bieZRVZcdppwab7729oX1QPpJI1RPuJS1a2ufzyyyOf95ZBUQKl2YvTOtd1\nLhXp2VNIASLfWt/lKo0JpgKatR0Vi4Lmo9gOy2oki0XaBSHLLSuqI0KcQPq0ywqUkAuNWb/kksWk\nAgECXYMsC3KrawTFiWwcP95ZRBZ/4QsNWUW0KFUpQL2oLjR5wV8I8+eDSm6h1/bdd9/ImLC00Tmv\ncx8BktOHlm4kPXv2DCeucvFIgyuvvNKt2OqEheYiNynz3ddYNrOSuVY9bBKsB5CsLpA+7bACJbSA\nYAHqGveiu2QiQKDVKN2tAnPTZNvs2aFVZHnfvs4qIoFSK9VcsogLyZcAOfzww8NnQasWpHTuFyXl\nc6EEiKoj26q5VjjTtFb81V/9ldvun/7pn1JfoEnIKjFw4MDwRqH/t8IqoSBXO2/0b1xRMEj/AdDq\nFSghE7hvIR06dGhT41AQIAB70AKhFgCUZrgZSHRIfEiEyCoiUVJrOl+JjGrV0yG7zx9Z4K+++mq3\n+KVF5U996lMuoUSr0Lmva6AI5R8KI0AU0Gqr5ieffHKqGY7kRmInr3zW5coB6SKhYVXpNY6ygrQS\nWT4U4G6WM51P0HwB0hUrUMaoUaPCe4bOvSKKEAQItJJKcR9p1/3Q9iRAJERqtYpY0HCcCKHKdXaf\nP+VN6fyb6WURJ3J1LSBAcvDQ0sqpn/M/7YmE0imam4j+VbEqSFd89OnTp8stEDpv/DojCM3mPAC6\negXKR6l5LS5EIqRocUAIEGgVqkYe5//u1/1II8NV+bbNKiIXLYkSuWxVwwLlESH5e/6oiOg3vvEN\nF3uoZ5A8Wx588MGW7pOuBV0TCJAMP7R88dEMFw65W+nk1Eqpbjr6Hf1LDYl0UD/64qORFLtpn1Ma\nc9L0pv8AaIcVKB+5bmq1U/uic7FIIgQBAq3ALAp+scEo/AxXEgwKMq8nliMOCRsJHFlFJHaqbV8i\nRBPFOBGiBBtk1MvW88e3wMvzQV4t9rxvZQyoroW8p+bNtQBptvgQmoBa6k5VrdX/tWLbKn/1PKOL\nXZPPdhIfiJDmPwDaZQXKR+eexQFJhBRlgQEBAq1AE/VaArjNaqE0u2kXITT87Uv0VLK6VApOL1pq\n1TwJEHHKKaeEHi6TJ09u6X7pmtC1gQDJ2EOrFeJDyN1Kv6FsWooLkPjQ3//4j//IVd0AWvE28aF/\n2zEgCxHS/AdAV65AVRIhOieLkBENAQKtmGQ1knrUL0Jo6XbTjBex7VdDweeIkPwJkK997WstT4Ti\no2sjr9nVcilAWiU+NAHZZ5993O+oqOFll10W/q2bTle6jWQZ9WuPHj0yMdFDhDT/AdCVK1BRIsQ/\nN/NuCUGAQDOxOIq0sl5Zul0LLE/bRSuJmKrkjgXtiYoY2/NHczlDtacsBlDPn66IP9W1kVcBmzsB\nIitEK8SHuPXWW2P91nXDufbaa7mya0T+9brYrMZHFlaZfcFLYHr6AqSrV6CiBLJZ5+SOlefsWAgQ\naCYKsm1GoG2Ui1ajFdHTECEEprcXymAqd1+L8VP7y7/8S1fX7eKLL3blFXwrVlfd65t1nSBAUj6Z\nLG1mKyYqqiVidQI0aVIbMmRI6DJyyCGHcIXXgC5uq7+gvs2Si4tStlqKXipo1047r0BFof0yd6wB\nAwYgQABqRCu6qndQLfC84efKBx+UVESvp/YHIiSfVEp+oiaPFo2ZFsG6MvmIrhFdK3mzguRGgPgT\nAomAZqOJkH7r05/+dIkqtqxYVrtAGXQgGRJyNolvp4DzpFiRRLnoUDE9+aJBFlagopgzZ04ojiSg\nECAAydGK7pgxY1r6m37tD8tylWZK36QiJO/pVbP0/LHF46imemPtsqCoayVv500uBIiUqWoGWAad\nVkxSLrroIvd78k8v54wzznDKWe2cc87hKk/AbbfdFsZSTJ8+PZPH0BXnYdbJygpUHJYFTy2PxSkR\nINAMtJLbq1evikKh2cglS65ZzXTRqiRC8hpYDM1D10yerCC5ECCtXHmW5UMFySzb1UEHHeRiPaSS\nNUEaP358cMIJJzjrh1lBBg8ejFtOwkmc+i/LtNoSl3WytAIVh7nfSTyrcCECBKAylXzareiguUul\nnV63nCgXrSSFCJMSl6I37zUeoLXXDQKkCx5arfa9j1ux1etKFRu3klseWAt7kKuVubGoSn1eJtUW\ni5R1QQXJGDRoUHgfaseU0QgQaBeSxn7ICqKUun4F9GZbRuw3zUVL6XfTcNGKEyF59OuH5pG3WJBM\nCxBN9FqdfShuxVav66SIW8nV+1CKXJR69+7txk/B53lyWbJsbBIi7RI4Dc09l5URy9zvECAA0Ywd\nO9bVzKgFq4BuwqAVYkS/aS5ayqYlF616U/paumHS80Kj6NrRNYQA6cKHloK9pQT1W6q/AdnDrFcy\nReexnoK5BiouhHiQ/CP3u27durkxl+sYAgQMq2jst6LGADRa96PVYkSiQy5aqivSiIuWRIiedVEi\nJC8TSmg+VhcEAdKFD62rrroqTNfajsGpUBm5y5mbUl4L+OmBY0Xrbr75Zga9AEycODFXrlgIkMaR\nzzbZkPbe99Nc8ffFiBUebFZWK1HuoqX/1/J7Ov64c4F4EEiKrqE8nC+ZFCD+5BXXpuwha4Bli1JA\nf56ZMmVKmIwgi6mFoXaslk0eXLEQII3xzDPPxE44i5gNSbEQzTheKzxoLlOtECMmfuz3krpoydoR\ndR7kZVUbmo+uoTzUk8mcAPEnr3kJWi4aFsgv60ARAvAksvIWGwDx5MkVCwHSGHF+/0UVIXKbjiJN\nF6pWixH7Pf2Ofk+ipNpvaQUbixg041pCgDTxoWX1Iooyec0bCsg269W0adMKccyKb7FCe2TFKgbm\niiXL17x58xAgBSWJ+CiKCJk6dapLSR81gW9WlfJyMWL1PuoNJq+GhIcyZ1WjUjxInrLoQfPQtaRr\nCgHSoofWggULCjd5zRtmvVLa0iJhtU4UG0CV9GKQB1csBEjrBEjeRYiy98glNY7yKuVpB5dbvQ+z\nVDRbjFQjrkghWbEgCbqWas0mhwBpAMu1f/LJJ3P2ZfSCsVXhIk7CTXyRta0Y+AsmWY1VQ4A0RlIX\nrCKIkCS1P4zyTFdpu1C1ixiJc8VS7BBAJawmCAKkBQ+tPDzMi45NwItalLHoAqyIKE4ty1YQBEhj\nyEWiVgGSRxEit6Izzzyzru82O57DFyOtqr7u9wtWEKgXXVNZdtnLjAAZMmQIgbwZRoUiNX4Kzi1y\n7A5WkGIhoSnBmVW3UQRI48RVwS6SCFHmpzFjxjS0jVbEc/jV102MNDOTlpAbTdGzo0F96JrKcg2Z\nTAgQ/yGO9SN7KHOZFY0sqvXDMCtI0YVYkZDYtIKUCBBESBFFiFZq06xbEOdClSYSI0kCyhslLiAd\nKwhUQ9dUvZZFBEiND3CsH9nEMgLpJsukG1e0oqEFFCUf0JhXCsJFgCBC8ipCmumrbmKklVaLtIkr\nVklxQujKa6vwAgTrRz4uEI3f7bffTmd0IFccrCDFwmrfZM0KggBBhDSK7nGtWs1vldWiGX0UNe55\nKDYHzUXXVlbnEW0vQKzuR79+/TjTMsiMGTNC68fWrVvpkI/p3bu365cJEybQGQVADwhbSMnSqiYC\nBBHSKMroRIG9+s4LvAagGrq2spo1re0FiLmrKIgZsoclDxg6dCid4SFrECmli8XAgQMzl4AAAYII\nSWOClDXXw65ACxMEo0Ot6NrKqsBvawGi1LuWtnTjxo2caRlDFg/zfZ81axYd4qHq6JZW+qOPPqJD\nCsD06dPdePfo0cMlZkCAIEKKIEJ0nEWMZdg2e7aLS6kFuSuXj3eWg4yhNcI1q656bS1AzG/6rLPO\n4izLIFb9O+vFcpqFrB/ExhQHiY7u3bu7MZdrIgIEiiBC2iWb06ZNm5w7044dO1r2m1ZEMSlxKXlx\nw4IsXGO5EiAWvIz5NpucfvrpbvxGjRpFZ0Sg+A/1j+JBoBhYRr9BgwYhQKAQIqRXr151fW/37t3B\n1VdfHXzlK18Jzj///LqFw+LFi0v6+KCDDgoee+yxxN/X75533nnB17/+9aB///4l7dRTT3Wi4dln\nn438rqwgysyVtFZJnBuWCloCpH2NIUAqXIjajlx4CF7OHnIxsqDbLFfqbCZyK7Q+krsh5J+s3dcQ\nIIiQRql3dfbDDz8sOdZ58+bVdY894ogjwm0ceuih4f/jREPUNo4++uiqY3HLLbdEfn/FaafVlJkr\nqiaIRA5A2tcYAiSGESNGZGqlEEqx1f0+ffrQGRXASlQ8evbs6cZ88uTJCBDItQjR4lO9MQw/+9nP\nSo7z2muvrXkbzz33nPuuRMiaNWucVeWaa66paXtbtmwJjjvuOPedu+++O3jzzTeDV1991bWnn366\nZNxee+21Tt9XamAVS0xam0T9RVFCqAWdM1l002tbAaKJq7ajOALIHpbxR2mUobpQkzkfisHIkSPd\nmA8fPhwBArkWIfWm4JV1UK6pcpcaM2aMs1yorV69uqbt3Hfffa5/VMTYEj/ccccd7rWLLroo2LVr\nV00CJCqYXts444wz3Pv33ntv5DZUHFHV2pMwduzYyDEGiCOrqXjbUoDoRmGuKWQIyiYWv0P2q8qY\nS47M7lAMFNOmMf/Sl76UOwGiibBWa5vZtNqnB24zmvZfD/JmtKQrlHkSIfUKkN///vcllguNu/6u\n1WooK4X1z69+9atg0aJFoZhIWoPJFyAvvPBC5GcU22UWkigUA6JYEMWEJOmzqPHFlRkQIC0QIEzK\nso3iPzR+SjNL/A5iG7J7fdRyL9dDsJ6JM61z07lRz/e0qp8HAWJuUqNHj3Z/myXjxBNPrCkYXS5X\nw4YN69RP2k7S1P7VBMjzzz8fxpZUSjesOBDFg1Qjrip6VovNAQIkUwIEt5RsM23aNDd+KiIJ1cHd\nsHhkxUKY9F4et2pLa74I0ef8dsghh7SNNUTZm2rdlxUrVoQTegs8X7JkSXi8cVaIOG7tmPj7/XXu\nuefWJPx9ASKXMD8Llh/UbmIpVgytW+fS8m66//6qv0lBQqgFs9wiQFIQIPKN1jZUBwSyR5Z83NsB\nS8161VVX0RkFYcCAAZmoAZP0Xi53MkRD60VIpffboTqy9mHmzJk1fUcLMdp/JejwC3aaVeSSSy5J\ntB1ZSqKsHxIgZkV5//33ncBIKkAqtQsvvNDVGqmExIdESLW0vFEFCbNa7Rqaj66xLJ4fbSlAtHKu\nbWglHbKHFdibOHFiW+5fVxSkqsSkSZNcf6nfoBhIeGShyGrSezliofUiJImFpKtXRWsVIArotniP\no446KpgzZ47LNKV/lRHTUunKIlKNm266KewHzSX+93//t8RaodTntr1Kwe2+AJF3hqwyL730kosv\nmT17ttuWbbdcNEWRJC2v9QECBBAgLRQgMo3qxqptyFcaske3bt3c+M2fPz+V7eWhIFUl1E/qL/Ub\nFAO5kWjMe/TogQCh1SxCkrpndXWiAwvuT4pS3CY5rmoB5Iqn88WHvz/l25JoqPRMqZYFSzzxxBOJ\nxIxIUpwwSoAQAwJpXWcIkBiU6UHfV2AuZA8LsE0zfWdeClJVu2bUkgZGQrbROKd9nXTlvRwXrNaJ\nkFoD1LM0MdK9U/usFLx33XVX8JOf/CRs+luui2Yd2b59eyIhs3Tp0vB1P2WuNVlGKpEkC5ZER5JA\ndEMpeddUcFFGgAACpAsEiGXAOuywwzirMogJyO7du6e2zbwUpKqEKmNTNb5YmADxJ0hZFSAKNkYk\nNL/tu+++NX8nKxMjf4FHKXOj+OMf/5ho4cgXIO+8807Je7KE+/1TzVKfxAJiSRiS1ipRUUIVJ9we\n85xAgAACpAsEiHzZyKCUXZTVJ00BmbeCVHGov9RvSVbPIB/I/ardRWct93K5ICIS2is1b5ZcsH7z\nm99UncT7MSK6b2shKU7M2L1aLrLmzr18+fKwZoc1LUpVcvf27/uPPPKIWzDQNSs3r4ULFwYPPfRQ\nuC1ZZqoFtRuV0vJGWRQRIIAAabIAUQYMfV8TRcgeM2bMSHX88laQKg5LvFBrxhjILpaKt53HvJ5C\nhFEZfGhdUxdElqmuRL+vwpFJMAvypZdeWvFzKuRpMXzvvvtuxcXMpP2kBaD169fHipkkrrdqr7zy\nSuK+qVSckEKEUAu6xrr6Ws+FAFHmpCxkh4HK49evX79UtpfHglRRSLCp32oVVpBdbMynT5+eGwHi\n06yK4nrQNqsSuibBzargrsK6rRQfOpauRuOlY0+CTfLnzp1b8XOyjlg8nywSlXjxxRdLYv8svkQL\nWyaQ9Nqdd94Zuw2zwsf1s5KS3HbbbW5Rq1aiXLDiChECxKFrjDogKTy0Ro0a5b6vlHuQPcaNG5fa\n+OW1IFUUEtztnLoY0kfnTbuPebsHyecdWZSyKj5qFSDNZNmyZc6Crsl9OYoJiXPl6qo+Kx9PiVcA\nBEiTH1oqPqjvjxgxgrMqg9j4yRWpUfJckKocCTb1mwQ4FAMb83YuRogAQXw0gib8TJ5rY+zYsZ3G\ntB1EHLQvusaixDUCpM4JrKppQ3YFSKNV7ItQkCpqMtpov0H2BEg7jzkCBPHRKFl3H1LGqlYif/7y\ncVWCB4C8XWNtK0Bwwcq2ABk6dGhD2ylCQSof5bfHBauYAmT8+PEIEMil+BBKSqBsUVnEAsWVrSoq\nWLwZRMUKZTHAGFqDri1dYwiQFB5aFsSsCRlkDwmCNARkUQpSGRaQPGnSJE6ignDyyScTAwK5Fh9C\nluwsZ/eTCFHKXAmRpcceG2y6//6m/ZaeGVHjm0X3GmgNuray6qLXdgJEWYBIw5tdNIFudPyKVJDK\nUN530vAWiz59+uQuDS8gPsqR+5BqOOUBiQ+JkGYJkaj4j66u5QLtjbw5suqi17aFCLnosqvGNX6y\nXNRL0QpSCStEWGtmL8guNuaKa0KAID7yKD7s2AYPHpyr8fKFyMbx452VJA2iChAS/wGV0LWlawwB\nksJDSxOwNCtpQ2tJY/yKVpBKdOvWre2rYkO6yNc7T5XQAfERhazFvXr1yuXYSYgoPkTuWXLTakSI\nxLlf1VpTCoqFrq2sniNtJ0D0MNb3DzzwQM6sDGLjpwl1vRSxIJX6TA1f3+KQhTFHgCA+0iDvhfQU\noN6oELGFN79lNbgYuLYyKUC0ysxkLLv441dLGtpW0Y4FqRDdxUOufnadtLtIAsRHo9QaiJ5W6tuo\nSuOtECKLDj44WDN8eOLjsHop5WOtmBCAOLIcgN6WAkT07NnTbWPGjBmcYRmkR48exDPUgAouqr8U\nlAzFQPc2jbnudQgQxEeexYdQoKxaEtJKfavtWJzG2uuvb2k9D/2WBEhSIRKVFp7sV5DmdYUASciQ\nIUPcNm6++WbOsAyiehrtXuG5nbjqqqtSqx4P2UAV7zXmutchQBAfeRYfQtkBa4kD8VPfNipEtjz+\neCgGlvftm2rQeK1CJO445GqVp/GG1qBrStcWAiTFh9a4cePcNjSRheyhOAiN38CBA+mMBFg9CGqA\nFAdlZNOYVyukiQBBfORlMioXI7m/1oIvRGpxaYrblgWNSxCsOv98J05aJURkhTFB5R9HnPWDhCRQ\nCV1LuqayTFsKEMukJFceyB6zZs1y40cAXTIU+6H+WrBgAZ1RECwDVrtnL0GAID7SQsdRb7rQWl2a\nkmxPwkbuWRIFEgetiBeRCJIFxoiL/cD6AUnuJ1k/T9pSgCjb0H777ee2U6kuA7QnCkS38cOHtTLz\n5s1rOGsYZAsJzawkHUCAID7S7ING64GkLUSE3KK0Lat0LoHQqngRrB9QL1mu/9HWAkRYZWjVeoDs\nccwxx7jxmzZtGp1RAascLzcsYMwRIIiPvK6CazEqrZShzRAiQi5acs0yF61mVDs3JDKKMO7QHGQ5\ny/oCb9sKEAXkajvDO24ukD00bhq/kSNH0hkVUJwM/VTMa2PEiBEIEMRHoSahZ555ZqqrtuVCJK3g\ncm1XlhBz0aoUQF4vSp9aPu6aVGL9gCT3FV1LWadtBYjFEeiCbMd6ElAZWa6IA6mMXA0t/oNqt8VA\n97Lu3bu7Ma+lLgICJHsoOw3ioxQVfG1G3QIJBlks0qhIXo5iQyyAXIJE22/U4qL6HlFjn+WUqtA6\ndA3pWkKANPGhZfUkqAeSzcm1BdrOmTOHDolg8uTJmagFAekxffr0TCXYQIDUj9yIER+d0aLUwoUL\nm7LtNCqSV0JZs8xFS78jF61af0MWjqjAc50vANXQtZOXhd22FiBWH2HQoEGcdRlk6NChuNFVwOql\nqCYEFIOsudwhQOoH8RGNVv+bvdLfbCFi2axUV6TWOJQ4YYoVHJKga0fXEAKkyQ8tXZCWLUYr6pAt\nZPkwNzrGrxRlCjvggANIv1vQMZ8/fz4CBAFSyMBjswDUWhOkHYWIsJS+SdA443oF9WK1P/ISJ9TW\nAkTI1EQ2rOzC+EWjAnTql969e9MZBWHixIluzLPkaoEAqZ+kLlhFzHqkY27lpNsXIn4djlYSF/eB\n6xUkRddMnu4XbS9AbrzxRre9s846i7Mvg8i9iKr2nbHq57fffjudwZgjQHKIgkQRH9FoBbcr/Ngl\nRBRIrtbMFLvlxCUkyEMqVWgdecuS1vYCRJ2tonZq+Ehm80Gj80GuJ0uXLqVDgr0Z3uiT4mDFB3Uf\ny1JxVQRIY1xxxRWx4kPvFZlGKqM3isRHq4SIBIbEFnEf0Ah5qHyeOQEiFITOKnp2sWBr1XaBIOjb\nty/9UTDsHpa1hBoIkHQmDv4EVP/PegXjNOgqK0grhYjER5wrHucA1IKulbzViMmEANHqoVYOqZeQ\nTSyZgMaw6AHXqv2A9YPzHwECsMdCNGbMmKqfS6vSeauFSFzQOdXOoRZ0jeTRYpoJASIUA4IVJLuY\nFaToKZXN+jFkyBBOCs59BAgUGnNPqpQRS5mrrBp5q4SI0us2Wvk8TnzIIkLcByRF14aukTyeM5kR\nILaKiBUkmyjtqMXyFNUKYtYPLEHFwe5bsnhl0XyOAIFmo+xQ1VZ3JTwkQJqZTtdHv6HfUuaseoRI\nnPgg6BxqRddGXup+ZFaACFtJxAqSTcwPXsXYiohlQaKwZnHo169fpuN9ECDQCmQV0AJNNSydbiuC\nxyVyfCGS1PpSSXyweAq1oGsiz2maMyVAdPHaKrqK3EG2KHIsD9aPYj48sh7vgwCBVqA0tbUEpJur\nlITB9tdea7oQSVo7BPEBaaJrQtcGAqRNHlpXXXWV2/6RRx5Jde0MYlYQFeDbuXNnIY5ZFbAPO+ww\nd9yqiwL5R/cm3aOynu0MAQKtQq4mtRQnNAvFIlVVHz686W5ZiA9oJboW8p6qO3MCRA92m8ypSCFk\nC9VA0A1Z43fbbbcV4piHdzwcdbzHHHNMYURX0bGFkh49emTa5xsBAq3CAtJrnazLNWrV+eeH8SHt\nJD7U8ryCDc1B10BeA88zLUCE787CykL2ePjhh0PXlHnz5uX6WK3oIG6DxUHjbK6GSfzaESAAeyde\nvXr1qpgVKw7Fhyh7lVyztjz+eMtE05lnnhlr+UB8QK3o3Nc1UIS5bSYFiDBXHlaVs4mlVc7z+Plu\nOFoRh/zjj7ksX5l/QCBAoMXI9UST+npRfEitgeP1io+4IoO4XUG96NyvxRURAdIFDy1d/N27d8ev\nPqP4rlh5daUzNxy5DCoOBPJP3sYcAQJdwUknndRQpXDFg6y9/noXH6J/044PkbhAfEDa6JzXuV8U\nMitAhLnyyN1hxowZnL0ZY8qUKbl1pZs2bVpYt4ZzsxjkyfUKAQJdSb3xIOXIAiJLiCwiSTNZVUNu\nVRIZiA9IW9QWIe4jNwJEDB061P2eLvwsFvoqOuZKJ7cVWUXygOJaDjzwQBIlFAil2TWLbB5crxAg\n0A4TMvnCL1y4sOFtKT5EsSFqjVQ41wp1XLA5Fc6hXnSOFyXuI1cCRPEDVuCtZ8+euLpkDI2X4kA0\nfn369Ml8PIit3Ol4+vfvzwAX5BzW5MPO4TzdgxAg0JVowl9vUHrktTp+vLOGKGtWrfEhlTJdyW0G\n8QH1YEHnU6dOLdyxZ16ACK2cW2peBTdDtvjoo4/C1eMsVwlHDBeT008/PYz7yGrBQQQItCtjx45t\nKCi9HMWDqG6I4kOUtrdafIg8K+LiPdQkTADqRee2zvEikgsBInB7yTYvvPBCOH5ZTSpgAcjdunWj\n2nlBGDFihBtznbvz58/P3wMCAQJtgCb5gwcPTnWbqqBu8SHKnBVFpXgPtaJOHCEddE4XWcDmRoAI\nC0pX0/8hW2R5/CZOnEhChIIxYcKE3I85AgTaSYQ0ozK0aoYoNkQ1RHy3LKVCjRMe1PiARtG5XHTr\nWa4EiJD1wyYFiJDscfPNN4cryirilxXhZNmPilLdvegoy5WN+fiUsusgQAAqI3eVMWPGNGXbliVL\nsRyK6agUbE6mK2gEncNpuhUiQNrooXXZZZchQjKMZcbKggjxxcftt9/O4BWA6dOnh+6CeS8wiQCB\ndsLEQbNEiKVCrRTvQbA5NCo+SFqQYwGCCMk2Cubu27dvKEJUUwPxAe025kp4kfWsbQgQyKIIaYYl\npJLLFfEekJb40LmL+Mi5AEGEZJutW7e6CV67jh/io3go5qNI4gMBAu0sQtIKTK+W5SqNgogAFnCO\n+CiIAEGEZBtN8HwRogkg4gO6Ao2zJUjQPaUwDwgECLQxjYoQWTUqZbnCVQbSFB9QMAFSLkIIEs4e\nNn7tMOG3JAdqCpiH/DNy5MhCig8ECGQBZROSW0stxQpl9agUaK4mlyyARtA5qXOzGdnbECAZemj5\nE0cFOcvFB7IpQrqiToiKCvrWGGrN5B9Z4Ky2S1GtXQgQyAK1VEyvZvXA5QrSEh86J3VuQsEFiJg8\neXKYvUY+n3mrWpx3fDcYiYFWVRrXapkqm7d7UDykx8qVK4N+/fqFgrOornYIEMgKU6dOrSge9Hql\nWA81rVTjcgWNYtnUdE4CAiREFbe7d+/u9lH/zpkzh7MgQygO5IADDnDjd+SRRwbz5s1r6u+pwJxW\ny/R7hx12WNN/D9rjHqGxNsE5adKk4j4gECCQsYmfRMaUKVPC1yQoqmW40mSRwoKQBjr3qBWDAIlF\nlg+dINpPTWZVxRqyO0GUZasZjBs3Lgw279OnDxazAqB7gQlcTUqKLjgRIJA1LE2vBafr70ouV1g9\nIC10zpFmFwFSFbnvDBgwIHTpGT58eMtceqBx5CJz8sknh+MnX/200qLq5mHFEC1mqAgpV4uMYsKG\nDBkSjvnpp5/O/QABAhlGwkI++DNnznR++Fg9oFnoHNO5RrA5AqQmFExsq9xaVdeJBNlAosDPUKTi\nhY1aKRTf0aNHj9A6Rprd/PPRRx8FvXv3DuM9yG6GAIF8IOEh64cmhn7WK6wekKbQ1TlGsDkCpC7k\nq3fMMcdgDckoEg3dunVzY6d/63GpK7d6aEI6f/58OjfnAlZudpaYQg8RxfwAAgTygxUaVCKRY489\nFt98SG3eKKuHzi2dY4AAaWgygjUku2gVu3///iXWkAULFiQWML7VQ7VicLnKNxKXZvVQkzsmMT4I\nEMgvKgTHSjWkgVnWqBWDAEld1frWENWfwFSbHVSl3LKcVRMTWD2Kh2I9/IUGCU/SKiNAoBhYqt5a\nCxcCiIULFzpXPlLsIkCaRrk1RBNauWqwKp4NyoWFBKVvdtckVMLE3LawehQDpdxW6maL9cDVEgEC\nxXw+SIBoBXvMmDF0CCRC54rFE7EojQBpOkr3aul6LS2nVtghG8iFztL1asIpa5aEpL1mrlr4BOcb\nuVZJbNiYS4RQ/wcBAsVGGbDMGqKVbYAoLNaDrGkIkC5BdSZ08vmuOsSHZANZO5Si18bO2hFHHFFS\nrAryh1aplCXN6npIhMqyiaULAQJg9wiLDZE/P25ZYOhc0DlBrAcCpMvRpEUpWc1tR+2ss86iMnYG\nVi/8eiHW5Fan8ZRAgfyJzlGjRpVcq6rrgaWLezlAFFrZlm+/1Q2BYmNB5jonsHogQNoGrZiMGDEi\nXFVV69evXzB9+nTOpjZC1g25V9kYKdWqVr81Tn72IwUhK20vq+L5WCSQi50lITAXu1mzZtE53MsB\nqmIr3pp4IkSKOW+QtwtB5giQtkZpX5W60wLVLdhZk1lW1bsG9fv48ePDQGNzu1FAenmKVd1olBfe\njwuYNGkSY5fRcZ8wYULJuOtapKYH93KAWtEiowkR4kOKgcbYEhOQphkBkhlUgEbBzb67h1Zg5QKy\ncuVKzrAWIHEhX3/dPGwMNB6yVFWq7aAVcwlGP77Hxo6aENlYBNAY++NOogju5QBpCRHiQ/IvPAYP\nHhzsv//+bqzJboUAyezNSjEFfoYluf0MHDjQrcTi4pMu6k/Vb1D/+u5wmoDKDacWS4a2pUlrnz59\nwu1om9o2cQPthzJYlVsfFedDPQ/u5QBpY/EhEiKarGIRyT5yr5PFw4QHlcwRILmZGGsy66fvtVgD\nZWNiQtsYSo0si5O/6m2+/mlktdL4yGXLFzUSJhpTakZ0HbImykXOv640RkOHDiURBAIEoOno2aDJ\nqiatEiI8y7OH5ggmJmXVQnggQHLLggULnGuQbxUx//Sbb77ZuZBAdXSTkFuU7ypl1g693oybiFyw\ntG0/oFkTXmVT0k2MWJHmoz6W6FCf+9YOiXkVjsRcjgAB6IrnkblmaTJLOvf2Rq5ziuuw4HIJD54d\nCJBCoUw8Wq31V+41qdLqurIzya0EN609qB9kIpV48zNWqan/ZAFpVSE57YsmwX7AurnXDRkyBPe6\nJvS3MpXJ/U19XC7cZYmivxEgAF2NBavbxFaVsokTaR/kKqeq5ZozyHJOcDkCpPBoVVcrJlrV9d18\nLHBatUWUyalopsH58+e72I3+/ft3mnjqb/n8y8e/Kyef5v6lFfhy9zq9rvehPiQoVa283LVOD3YJ\nUdwdECAA7Yomt7KGyD1L/zLZ7RokACUE9dyw+A7qeCBAIEaMaAVdmXy0ulteLE8XkSZlytSkCVhe\nVn51HDoepU6VFaHcRU1NKxbqF1lC2tHdSRatKDEily2t3mvM8C+tLDgltCUsfTc3E3SIDgQIQNaQ\nVWTs2LHu2W1B69QTaY0AtKByBCAChJ6tA8UdyN1HQdDlkzKLQdDEXK5cshZoxb3dYxEUuK3JuvZX\nYkNCy/fntyYRovcnT56cOf9MEyNRQkqvaTwlSIoc8yMxpj5QX5SLNhPb6kOKBiJAAPKAFlDMBcjE\niLwfWFhpHIk6X3To+SHhx6IfAoSHVkoou4+sBLKCSHiUuyb5vvGqxq4JvOJJNNGTH72+3+xaJNq+\nbqj6Pe2rfl+TTO1PedyE71Kl2BdNOPUdrYbnBSUe0Mq+XOn82jB+wUONk1I26yaax2A4nROy7OkY\ndS5ECTP1jawfGn8eGggQgCKIEa3O+9YRAtiTYYHkvuBQXyI6ECA8tFosSmQlUUpf1T6ImuRGWU50\nwSpVrTW5CWlyqCZXJwmH8qbfsM/o8/73tb0oS0Z5041W+6ltKYC4SClT5Wqm2AZl01Kflcf8+G5b\nin1RH8kCpD7KgrududLZ+SjRGWW5s3NQ54GyvxErgwABQJBc4RYWNaHWxFoTbGqM7EV9oXgOi61R\nXyE4ECA8tNoMuW5poqsJvtKTymKiIHdZRpIIlEaaZZjQ78mioVVv7YcmmVQTL0XuaLIUaYy0+i8L\nUZyI80WjBKBiImQtMMtWKywnZt1S8L+sOtoH7YsJ0DhBpdd17km06lhlDSFtMQIEADqjCbUm1ra6\nb3EMmnzLSl4EUaJj1LHqmHv16uX6QHMLE2akzUWA8NDK8MRXbk66wK3JTcuaVuijLCBarfY/539f\n26MwX+NEWRHKM0HFNbmyyaUrzrIlIRo1rmp6zz4nMeRvQ9uMExd5sd4A93KAdkSTbU26lcHJVv81\nKc+LKDGxoRTGOiYdm9VU0TFPnToVwYEA4aEF0JUPoXqsD2k2EzhyndJvS7i02goDCBCAoqNUspqY\nmyjRhN0XJprMmzjpaoFiAsMsGr7QMMuGLzZIk4sA4aEFkDHMRSrOsiV3uDgLiN6zz8n64m+jFckL\nAAECAOkIE03yTZz4AsXcudTkzqTPpdm0Tdu+/Z4JDBMZ+hxCAxAgAADAvRygQCLFFyppNhMWiAtA\ngAAAAPdyAABAgAAAAAIEAAAAAQIAANzLAQAAAQIAAAgQAABAgPDQAgBAgAAAACBAAACAezkAACBA\nAAAAAQLQHHbvDnatWOH+bWgz69YFuzdubHx3UtpOy7tx/fpM7nfL2bEjlfMNECAAAMC9HLqITRMn\nBos///lg6THHdGrLevcOVn3zm8GWadOiJ3wdr60aNChYdPDBwbbZsyPfX3nWWcHiv//7YNkJJwTB\n9u3Rk++Oibfbh169gmDnzj3zzLffDhb/7d/G7tfqCy4Itv3+951+b9k//3Ow6JBDgh3z57duTjxv\nXrDos5+N3Nelxx4brDjttGDj+PFBsGVL9Pffesv14bLjjw+CXbs6vb/+1ltd/+g3dn7wQdsL0jWX\nXRYs+sxngq2//W3Fj25+6CH3ucgx7uiLtddfH+z88MOS72ybNcv11eqLLuLiRYAAAAACBLLIhjvu\ncBO6am3Vt77VaXK89amn3HvLv/rVyImzTaytRYoUT4BIPJgA2fb884n2a9O995ZOUDt+I5zMf7yt\nZrP9pZcS7asEVScrR0e/Lfunf4rvn61bg8WHHx5uY/3o0W0vQFb07+/2dev//V/j5165mOzY/vKT\nTtqz/ZkzuYARIAAAwL0csipAVp59drD9xRfdCrM1rVAv6dkznAxuefzxvV/smNwvOeoo9/r2F16I\n3PbakSNLJpMrBwyItKRECZBwUv+Zz7j90ORc+7T16aeDjXffvccioPc/+9lg16pVkRPgkv1tgQBZ\n8v/+X7if1rZMnx6sPOecsA/Wfuc7Jd/d8uij7vUVp5wSKeJM5PkiJs6S0jYC5LTTahMgn/tcSb9t\n/c1vgnXf/354zOWWoW1z57rXncVsxw4uYgQIAABwL4csCpB13/te9Hxy06ZgyRe/2GnyrIl1pYmz\nvudW7jsExJorrnAr2Wq7li6tTYBIYKxc2ek7cs1x24ywHGybM6elE1Tb1zgXKk3K5TLUaZ86Prv0\ny1+OF3FyYesQbc4CNXBgKPjqFVa7N292/b9ryZJE8Sbus4sXB7vXrq283fXrSz5XqwCJ6zcTZxIo\n5SJTVrckvwEIEAAA4F4ObSpA1o4YETO73DvZCz/jucFseeyxyK9tefLJvZPHZcvCyfOGH/+4NgES\nI1qca9LHVpBOrjiedaaim862bcG6H/wgWPfd7zoBFtV0zBt/+tOKQc+2r34MS+xk23cz+zieIU4o\n6bhNZG1/+eU9Qk4ub1/7WrTQiTvMuXNDNy+/aQx3vvde+Dkdp7M4PfhgsPKMMzp9ViKjZP86RIFi\nhMqtXOG4JBQgccfvYmtiROjmhx9uuasdAoSHFgAAAgSgFQLEW6Vfe801nSeGUeJAK/dnn10SLLz+\nRz/a46b0xS+6iX9NAqTcAtKx/fW33BL/frDX/Uv7EScetGJvE/yKsRsd+1bJkhJaQLz9L2fNlVeW\nTrY9V6U465MTBNZn27eHrkcSCTsWLEgsPvxjUQC/XMXs7yVHHhkmB+gUk9HxO37/uOOzfvBEnjuu\no4/u1Jc1CZCyftPYKIFB3PtOnHWIW+3j9ldf5UJGgAAAAPdyyJoAUeYiiQ25TlnTJNdf4d50zz3u\nO5unTNkbfB4x4Y5yj9r57ruxE9OKAuRjAeBnSPInuZsfeCB64v1xMHolq4Qm3ht+8hNnlanU3G8k\ntID4/aemifK6UaP2BvP/27/t2VaHCFOGrNjgc0/4rb/55r2TfnOHixOMZULNRI76VimKDZeBqszC\n5AsQCTzXbx3b2DBmTKfPbvzv/w5FSugS1nFMchWrVYCYgLHx9QWSC0J/++3IYwuD0XHDQoAAAAD3\ncsieAKnW/NVvm5DGrdxHutZ4VhEXN+JN6KsJkErN7UNUBi6z0sgFbPXqpvZh0n11aXQ/TiurFX7n\nqhTjYlZiAfJcn8wqouOqGsfxsXhYduKJbnslb0X8vo1beVYzG5/ws55bXvk5EH42rSxYEpkdgjfq\n2Cw+JsqtDxAgAADAvRyyKkA6Jp1yn/KtCOZOFbkK77nmKGOTVscVU6B/l/ftG25TFpGkAkTfDzMk\nPf10sOkXvwhXv+P2o9OkOWaCrhSvO15/vWLz4yTqFSCrhw3bU0CvggDwWXP55XuzgD38sLNYbJk6\n1dXGqGb9idzHl18ONv7858G6G24IVnz96yXjWy5AYkWFfdZLHdwpxka1Yf7932vOgqVx1bbcGD/1\nlLOwhJnOYrZV1X0QECAAAMC9HNpXgKy++OJg94YNwa7ly/e0j7MldXI9igpK90hav8Of5FbNguVn\nQIqY6EbGECQQIC4GRHEEVfbVxUkkiAGRS1XYfx+3ne+/3ynmxfWTxWZE7F8oTqpZpRIEYLuCjpW2\nFSFAyse1vC/9/YsSBmalaTQLVkkGtohzDQGCAAEAAAQIZFiA1DKJW3vddbHfkZBx1o/DD3fZpZS6\n15pEh4Kgy+tZ1JUFK/AybUWIlGoWBpvgKuZAldq1v5Gt4z3FwVTKOpUkCL0cFycj8ROxf5bhyaUw\nvvLK0j7s6NMwzqJaAHaH8LEJvKVD3vzLXzr3NInLOBesagLEj0+JyjJm50cjQejl25LFq/wzFS1x\ngAABAADu5ZAfARIGrl9+efSkv+O9jXfdFT1Z/8Mf9roQdUyGqwqQmDogwtyUotK4hpaYSi5YKZEk\nDW8n8eNN6nd+8IH3xt7gapfBq9J3LXlAnMhR4L+SAUQIlTBRgNe/iQWIV+yx03njBcontoDECbeO\n1yxQv7yAY8k+fJydDRAgAADAvRxyKkBshb58wr3p/vurT/q9GBELdq5WCX3rjBkuI9eON95wLkUq\n2rfm29/eW3siItWuCRA/zWw7CRBnnbBaJd5Efec77yTKIhXGiMS5qPkCRNv69a/D1xWLYhN7vwhi\nYgHSgeJJ7PthFqyOMVh77bU1Z8GSNcyN7ZtvuvFVU0yIH+ezccKEWAFiQhYQIAAAwL0ccipAwpS6\nfiYmb+VeqV8rpa31U7hqZb6RLFhW6LAcc88pt9K0jQAJ9rqrKRWwEabsrZLlyrckbfzZz6qKPRuX\n8gKD/gQ+VoBEubN5gejOitHx/9Ddq440vJVaVLpnf59KLEiAAAEAAO7l0N6YGFh3003Jv9QhLmzy\naZmYtKpubkHhingMYRE5Tb7HjYsUIGEa3ajWIVz0WVUx92tbhHixDxUroaclQD4WA7VW5Va2J7/Q\noC/iqgpCTwC4tMYxMSoSixZ34weeK7PYmksv3SPSrrii5FxYP3p02Y5u3dOfZYUnFUOzesiQkm3L\nKmHWmWp971tRooo/quK7qz0TcWzKhEYldAQIAAAgQKBAmBtWec2IeogSII2gVK6hRaLJ7lcN4Vko\nIosRpoiCzpVOODKzWYPsXr/ebXvnwoWpbztOAJtQ2zJtGhcjAgQAALiXQyHwrAzbX3mlfQSIV/17\n8yOPZEbIRcWxQDSqadKq+B4ECA8tAAAECEAbseXRR/dMns85p6HJc5oCxCanUZmx2hKzglRLqQuh\nwJRYS+LqBwgQAADgXg45ngxue+65+jfzcUFAt6LdiABRbEqHiMnaZD6MITnhhIbd2fKOFXBc9W//\nhsUIAQIAAAgQKCTbt+/JQhRR7bsWXOX15csb3h1XqbtC9qh2Ja3jzz1btjQljgUBwkMLAAC4lwNk\ngkSpgmktbdDFAmS//fZzDy4ajUajZbcdcMABPCUBACAbAgQAAAAAAAABAgAAAAAACBAAAAAAAECA\nAAAAAAAAIEAAAAAAAAABAgAAAAAAgAABAAAAgDZh9+5g14oVDRX12715c7B7zRr6EgECAAAAAHln\n08SJweLPfz5Yeswxndqy3r2DVd/8ZrBl2rRogdHx2qpBg1zhvm2zZ7u/11x+eez2ln/ta8GGceOc\n4PBZN2qU28bmBx5gQBAgAAAAAJBnNtxxR6LK4Ku+9a0g2LWr5Ltbn3rKvbf8q1/d816HAFnRv3/V\nbS058shg98aN4XZkQZFoWfTZzwa7li5lUBAgAAAAAJB3AbLy7LOD7S++GGybNStsmx96KFjSs2co\nHLY8/vjeL+7cGSw56ij3+vYXXtjzmgTIaae51yRE/O1t+dWvgtUXXRRua90PflCyH+t/9CP3+trv\nfIdBQYAAAAAAQN4FyLrvfS/y/d2bNgVLvvjFTuJgy/Tpe4TGKafstYx4AiRue2uuuKLUamK/s3bt\nHivIIYdgBUGAAAAAAEDeBcjaESOiP9AhKiQWSj6j1046aY9V5LHHSj5rAiRuext/+lP3/rLjj3dW\nlBJx8u1vu/fWjx7NwCBAAAAAAKCQAmTXrmDpl7+85zPXXONe2jFv3h5XqvKYjSoCZMeCBaFLV5SF\nZMuTT7r3Fv/t35bEiAACBAAAAAByJkDWXHaZExtyubImwaAsWBa3semee9x3Nk+ZsteNyrdieAJE\nzc+Atfjv/74kCD3Ytq3TvuCGhQABAAAAgIIIkGpt2T//cxDs2LHnO2PGRFsxygRIbPvc54Idb7/d\nWYBs3BgsPvzwPWl9f/c7BgcBAgAAAACFEyCHHOIyVPmWjjBjVbmblSdAFCOybc4cl6rXZcF68slg\n3fe/77Zn2+1k5fC+v/X//o/BQYAAAAAAQF4FyOqLLw52b9gQ7Fq+fE/rEAe7lizpXIAwKig9QkDE\nZcFy8SMfi5BOIgMBggABAAAAgGIIkNgg9AjWXnddVQGSJKi9U7YrT9wgQBAgAAAAAIAAKfnOmssv\njxUgcRaQnR9+GFpASgobBl4QOgIEAQIAAAAACBBj88MP78ly1atXbBasleecE+x8991gx+uvu4Dz\nHW++GWx+5JFQYCz6zGeC7a++Gi1AOgSKvgsIEAAAAABAgDhxYNmsSup1SID0758oq9b6W27ptN1t\nc+fuFTYfZ9wCBAgAAAAA5IiN//3fe1ymbrop+Zc6hMayf/on973NDzxQ8vqqf//3WNGhFLsrBwwI\nts6cGbnZ1RdcQCV0BAgAAAAAQGfMDcsVI9y1q+Ht7VqxwllUnPvVBx/QwQgQAAAAAACPbduCJV/8\nohMh2195peHNmSVG6YABAQIAAAAA0Iktjz4aBpx3qhdSA64C+sfB552KEwICBAAAAABgj3LYHaw8\n+2wnQrY991zdm1l/661uGxsnTKBPESAAAAAAABXYvn1PzMa2bfXrmPXrXfV1QIAAAAAAAAAgQAAA\nAAAAAAECAAAAAAAIEAAAAAAAAAQIAAAAAAAgQAAAAAAAABAgAAAAAACAAAEAAAAAAAQIAAAAAAAA\nAgQAAAAAABAgAAAAAAAACBAAAAAAAECAAAAAAAAAAoQuAAAAAAAABAgAAAAAACBAAAAAAAAAECAA\nAAAAAIAAAQAAAAAAQIAAAAAAAAACBAAAAAAA8sv/B7v23SMC5O70AAAAAElFTkSuQmCC\n", 35 | "text/plain": [ 36 | "" 37 | ] 38 | }, 39 | "metadata": {}, 40 | "output_type": "display_data" 41 | } 42 | ], 43 | "source": [ 44 | "from IPython.display import Image, display\n", 45 | "display(Image(filename='vendiagram.png', embed=True))" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "The figure above on the left illustrates a ven diagram of the sample space U (U is for \"universe\"), event A, and event B. For example in a die roll, $U = \\{1,2,3,4,5,6\\},$ $A$ = Roll even number, $B$ = roll 4 or 6. By definition we have $Pr(U) = 1,$ but suppose we know event $B$ has occured, that takes us to the right figure. Now B is the new sample space (because we know we are in B), and are interested in computing the probability of also being in $A$ at the same time. This mean we find to the probability of being in $A$ and $B$ given that we are in $B$, this the red shaded region on the right figure above. It's easy to see from the right figure that $$Pr(A|B) = \\frac{\\text{Area of A and B}}{\\text{Area of B}} = \\frac{Pr(A \\text{ and } B)}{Pr(B)}.$$ Note that we can have similar argument for finding $$Pr(B|A) = \\frac{\\text{Area of A and B}}{\\text{Area of A}} = \\frac{Pr(A \\text{ and } B)}{Pr(A)}.$$\n", 53 | "\n", 54 | "From the above argument we have shown that $P(A \\text{ and } B) = P(A|B)P(B) = P(B|A)P(A),$ and hence now we can compute $$Pr(A|B) = \\frac{P(B|A)P(B)}{P(A)},$$ this is the Bayes theorem. The $Pr(B)$ is known as the \"prior probability\", for example the prior probabiling of a coin landing on heads could be $0.5.$ Whereas the $Pr(B|A)$ is often known as the \"likelihood\", this is a measure of how much we trust our result $B$ given $A$. Finally $Pr(A)$ is the \"marginal\" probability of $A$. Essentially the bayes rule is a realatively easy formula for finding conditional probabilities. " 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "## Formulating Naive Bayes\n", 62 | "\n", 63 | "Suppose we have a data set where each observation $i$ belongs to a category from the finite set $C_{i} = \\{c_{1i},c_{2i},\\ldots, c_{ki}\\},$ and consists of several features $W_{i} = \\{w_{1i},\\ldots,w_{mi}\\}.$ If we could find probabilities $Pr(c_{1i}|W_{i}), Pr(c_{2i}|W_{i}), \\ldots, Pr(c_{ki}|W_{i}),$ then we could predict the label for observation $i$ to be the one that has the highest probability. \n", 64 | "\n", 65 | "To compute conditional probabilities, we can use the Bayes theorem from statistics which says, $$Pr(c_{ji}|W_{i}) = \\frac{Pr(c_{ji} \\text{ and } W_{i})}{Pr(W_{i})} = \\frac{Pr(W_{i}|c_{ji})Pr(c_{ji})}{Pr(W_{i})},$$ where $j = 1, 2, \\ldots, k.$ Usually $Pr(W_{i})$ and $Pr(c_{ji})$ are either well known, or can be estimated easily from the data. However $Pr(W_{i}|c_{ji})$ is a another conditional probability that we need to find and using Bayes rule again will not help here. This is were the \"Naive\" part steps in, we assume that the data features $w_{1i}, w_{2i}, \\ldots, w_{mi}$ are all independent of each other conditional on knowing the class $c_{ji}.$ Statistically that means $$Pr(W_{i}|c_{ji}) = Pr(w_{1i}, w_{2i}, \\ldots, w_{mi}|c_{ji}) = Pr(w_{1i}|c_{ji})Pr(w_{2i}|c_{ji}) \\ldots Pr(w_{mi}|c_{ji}).$$ Given this result from the \"Naive\" assumption of conditional indepdence, we can compute $$Pr(c_{ji}|W_{i}) = \\frac{Pr(W_{i}|c_{ji})Pr(c_{ji})}{Pr(W_{i})} = \\frac{Pr(w_{1i}|c_{ji})Pr(w_{2i}|c_{ji}) \\ldots Pr(w_{mi}|c_{ji})Pr(c_{ji})}{Pr(W_{i})}.$$ \n", 66 | "\n", 67 | "In many problems we can find $Pr(w_{li}|c_{ji})$ for $l = 1,\\ldots,m$ by using $$Pr(w_{li}|c_{ji}) = \\frac{Pr(w_{li} \\text{ and } c_{ji})}{Pr(c_{ji})}.$$ Therefore we can find the probability of each class occuring given features in the data, $Pr(c_{1i}|W_{i}),Pr(c_{2i}|W_{i}), \\ldots, Pr(c_{ki}|W_{i}).$ Next we can just compare all of these probabilities to each other, and pick the class that is most likely. Also note that we can omit $Pr(W_{i})$ in the denominator above because it is common to all the conditional class probabilities and will not effect their rankings, that is if $$\\frac{Pr(w_{1i}|c_{1i}) \\ldots Pr(w_{mi}|c_{1i})Pr(c_{1i})}{Pr(W_{i})} > \\frac{Pr(w_{1i}|c_{2i}) \\ldots Pr(w_{mi}|c_{2i})Pr(c_{2i})}{Pr(W_{i})} \\implies $$ $$Pr(w_{1i}|c_{1i}) \\ldots Pr(w_{mi}|c_{1i})Pr(c_{1i}) > Pr(w_{1i}|c_{2i}) \\ldots Pr(w_{mi}|c_{2i})Pr(c_{2i}). $$" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "## Data Description\n", 75 | "\n", 76 | "We will be using a data from the UCI machine learning repository that countains several Youtube comments from very popular music videos. Each comment in the data has been labeled as either spam or ham (legitimate comment), we will use this data to train our Naive Bayes algorithm for youtube comment spam classification. " 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 11, 82 | "metadata": { 83 | "collapsed": true 84 | }, 85 | "outputs": [], 86 | "source": [ 87 | "# Import modules\n", 88 | "# For data manipulation\n", 89 | "import pandas as pd\n", 90 | "# For matrix operations\n", 91 | "import numpy as np\n", 92 | "# For numerical division\n", 93 | "from __future__ import division\n", 94 | "# For regular expression (text cleaning)\n", 95 | "import re" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 12, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "data": { 105 | "text/html": [ 106 | "
\n", 107 | "\n", 120 | "\n", 121 | " \n", 122 | " \n", 123 | " \n", 124 | " \n", 125 | " \n", 126 | " \n", 127 | " \n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | " \n", 133 | " \n", 134 | " \n", 135 | " \n", 136 | " \n", 137 | " \n", 138 | " \n", 139 | " \n", 140 | " \n", 141 | " \n", 142 | " \n", 143 | " \n", 144 | " \n", 145 | " \n", 146 | " \n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | "
contentlabel
0+447935454150 lovely girl talk to me xxx1
1I always end up coming back to this song<br />0
2my sister just received over 6,500 new <a rel=...1
3Cool0
4Hello I am from Palastine1
\n", 156 | "
" 157 | ], 158 | "text/plain": [ 159 | " content label\n", 160 | "0 +447935454150 lovely girl talk to me xxx 1\n", 161 | "1 I always end up coming back to this song
0\n", 162 | "2 my sister just received over 6,500 new
Lo...\n", 214 | "33 Check out this playlist on YouTube:\n", 215 | "34 Check out this video on YouTube:\n", 216 | "35 Check out this video on YouTube:\n", 217 | "38 Check out this playlist on YouTube:chcfcvzfzfb...\n", 218 | "39 Check out this playlist on YouTube: \n", 219 | "40 Im gonna share a little ryhme canibus blows em...\n", 220 | "41 Check out this video on YouTube:\n", 221 | "42 Check out this video on YouTube


\n", 222 | "43 CHECK OUT THE NEW REMIX !!!
CLICK CLICK !!\n", 223 | "44 Check out this playlist on YouTube:\n", 224 | "45 I personally have never been in a abusive rela...\n", 225 | "48 plese subscribe to me\n", 226 | "49 Check out this video on YouTube:\n", 227 | " ... \n", 228 | "1915 CHECK OUT partyman318 FR GOOD TUNEZ!! :D\n", 229 | "1916 Hey youtubers... I really appreciate all of yo...\n", 230 | "1917 Hey Music Fans I really appreciate any of you ...\n", 231 | "1918 Hey Music Fans I really appreciate any of you ...\n", 232 | "1919 Hey Music Fans I really appreciate any of you ...\n", 233 | "1920 Hi. Check out and share our songs.\n", 234 | "1921 Hi. Check out and share our songs.\n", 235 | "1922 Hi.Check out and share our songs.\n", 236 | "1923 Hey Music Fans I really appreciate any of you ...\n", 237 | "1924 Hey, I am doing the Forty Hour famine so I ll ...\n", 238 | "1925 Love itt and ppl check out my channel!!!\n", 239 | "1926 SUBSCRIBE MY CHANNEL\n", 240 | "1927 adf.ly / KlD3Y\n", 241 | "1928 adf.ly / KlD3Y\n", 242 | "1929 check out my new video\n", 243 | "1930 Hey Music Fans I really appreciate all of you ...\n", 244 | "1931 Hello everyone, It Is not my intention to spam...\n", 245 | "1932 ******* Facebook is LAME and so 2004! Check ou...\n", 246 | "1933 Please check out and send to others Freedom an...\n", 247 | "1934 Nice to meet You - this is Johnny: 1. If You a...\n", 248 | "1935 hey you ! check out the channel of Alvar Lake !!\n", 249 | "1936 Hi -this is Johnny: 1. If You already know my ...\n", 250 | "1940 Check out this video on YouTube:
"Th...\n", 251 | "1942 O peoples of the earth, I have seen how you pe...\n", 252 | "1945 I WILL NEVER FORGET THIS SONG IN MY LIFE LIKE ...\n", 253 | "1946 ********OMG Facebook is OLD! Check out ------...\n", 254 | "1947 Hey Music Fans I really appreciate all of you ...\n", 255 | "1948 **CHECK OUT MY NEW MIXTAPE**** **CHECK OUT MY ...\n", 256 | "1949 **CHECK OUT MY NEW MIXTAPE**** **CHECK OUT MY ...\n", 257 | "1950 **CHECK OUT MY NEW MIXTAPE**** **CHECK OUT MY ...\n", 258 | "Name: content, Length: 1004, dtype: object\n" 259 | ] 260 | } 261 | ], 262 | "source": [ 263 | "# Show spam comments in data\n", 264 | "# DO NOT GO ON THE LINKS BELOW!!!\n", 265 | "print data_comments[\"content\"][data_comments[\"label\"] == 1]" 266 | ] 267 | }, 268 | { 269 | "cell_type": "markdown", 270 | "metadata": {}, 271 | "source": [ 272 | "Browsing over the comments that have been labeled as spam in this data, it seems like these comments are either unrelated to the video, or are some form of advertisement. The phrase \"check out\" seems to be very popular in this comments." 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "## Summary Statistics and Data Cleaning\n", 280 | "\n", 281 | "The table below shows that this data set consist of $1959$ youtube comments, about $49\\%$ of them are legitimate comments and about $51\\%$ are spam. This high variation of classes in our data set will help us test our algorithms accuracy on the test data set. The average length of each comment is about $96$ characters, which is roughly about $15$ words on average per comment. " 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 14, 287 | "metadata": {}, 288 | "outputs": [ 289 | { 290 | "data": { 291 | "text/html": [ 292 | "
\n", 293 | "\n", 306 | "\n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | "
labellength
count1959.0000001959.000000
mean0.51250696.734558
std0.499971137.319807
min0.0000002.000000
25%0.00000029.000000
50%1.00000048.000000
75%1.00000098.500000
max1.0000001717.000000
\n", 357 | "
" 358 | ], 359 | "text/plain": [ 360 | " label length\n", 361 | "count 1959.000000 1959.000000\n", 362 | "mean 0.512506 96.734558\n", 363 | "std 0.499971 137.319807\n", 364 | "min 0.000000 2.000000\n", 365 | "25% 0.000000 29.000000\n", 366 | "50% 1.000000 48.000000\n", 367 | "75% 1.000000 98.500000\n", 368 | "max 1.000000 1717.000000" 369 | ] 370 | }, 371 | "execution_count": 14, 372 | "metadata": {}, 373 | "output_type": "execute_result" 374 | } 375 | ], 376 | "source": [ 377 | "# Add another column with corresponding comment length\n", 378 | "data_comments['length'] = data_comments['content'].map(lambda text: len(text))\n", 379 | "\n", 380 | "# Summary statistics (mean, stdev, min, max)\n", 381 | "data_comments[[\"label\",\"length\"]].describe()" 382 | ] 383 | }, 384 | { 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | "For the purposes of evaluation our Naive Bayes classification algorithm, we will split the data into a training and test set. The training set will be used to train the spam classification algorithm, and the test set will only be used to test its accuracy. In general the training set should be bigger than the test set and both have should be drawn from the same population (population in our case is youtube comments for music videos). We will randomly select $75\\%$ of the data as training, and $25\\%$ of the data for testing. " 389 | ] 390 | }, 391 | { 392 | "cell_type": "code", 393 | "execution_count": 15, 394 | "metadata": {}, 395 | "outputs": [ 396 | { 397 | "data": { 398 | "text/plain": [ 399 | "count 1485.000000\n", 400 | "mean 0.509764\n", 401 | "std 0.500073\n", 402 | "min 0.000000\n", 403 | "25% 0.000000\n", 404 | "50% 1.000000\n", 405 | "75% 1.000000\n", 406 | "max 1.000000\n", 407 | "Name: label, dtype: float64" 408 | ] 409 | }, 410 | "execution_count": 15, 411 | "metadata": {}, 412 | "output_type": "execute_result" 413 | } 414 | ], 415 | "source": [ 416 | "# Let's split data into training and test set (75% training, 25% test)\n", 417 | "\n", 418 | "# Set seed so we get same random allocation on each run of code\n", 419 | "np.random.seed(2017)\n", 420 | "\n", 421 | "# Add column vector of randomly generated numbers form U[0,1]\n", 422 | "data_comments[\"uniform\"] = np.random.uniform(0,1,len(data_comments.index)) \n", 423 | "\n", 424 | "# About 75% of these numbers should be less than 0.75\n", 425 | "data_comments_train = data_comments[data_comments[\"uniform\"] < 0.75]\n", 426 | "\n", 427 | "# About 25% of these numbers should be more than 0.75\n", 428 | "data_comments_test = data_comments[data_comments[\"uniform\"] > 0.75]\n", 429 | "\n", 430 | "# Check that both training and test data have both spam and ham comments\n", 431 | "data_comments_train[\"label\"].describe()" 432 | ] 433 | }, 434 | { 435 | "cell_type": "code", 436 | "execution_count": 16, 437 | "metadata": {}, 438 | "outputs": [ 439 | { 440 | "data": { 441 | "text/plain": [ 442 | "count 474.000000\n", 443 | "mean 0.521097\n", 444 | "std 0.500083\n", 445 | "min 0.000000\n", 446 | "25% 0.000000\n", 447 | "50% 1.000000\n", 448 | "75% 1.000000\n", 449 | "max 1.000000\n", 450 | "Name: label, dtype: float64" 451 | ] 452 | }, 453 | "execution_count": 16, 454 | "metadata": {}, 455 | "output_type": "execute_result" 456 | } 457 | ], 458 | "source": [ 459 | "# Test data summary statistics\n", 460 | "data_comments_test[\"label\"].describe()" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "metadata": {}, 466 | "source": [ 467 | "Both the training and test data have a good mix spam and ham comments, so we are ready to move onto training the Naive Bayes classifier. " 468 | ] 469 | }, 470 | { 471 | "cell_type": "code", 472 | "execution_count": 17, 473 | "metadata": {}, 474 | "outputs": [ 475 | { 476 | "name": "stdout", 477 | "output_type": "stream", 478 | "text": [ 479 | "Unique words in training data: 5898\n", 480 | "First 5 words in our unique set of words: \n", 481 | " ['now!!!!!!', 'yellow', 'four', '/>.Pewdiepie', 'Does']\n" 482 | ] 483 | } 484 | ], 485 | "source": [ 486 | "# Join all the comments into a big list\n", 487 | "training_list_words = \"\".join(data_comments_train.iloc[:,0].values)\n", 488 | "\n", 489 | "# Split the list of comments into a list of unique words\n", 490 | "train_unique_words = set(training_list_words.split(' '))\n", 491 | "\n", 492 | "# Number of unique words in training \n", 493 | "vocab_size_train = len(train_unique_words)\n", 494 | "\n", 495 | "# Description of summarized comments in training data\n", 496 | "print('Unique words in training data: %s' % vocab_size_train)\n", 497 | "print('First 5 words in our unique set of words: \\n % s' % list(train_unique_words)[1:6])" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "metadata": {}, 503 | "source": [ 504 | "Currently \"now!!\" and \"now!!!!\", as well as \"DOES\",\"DoEs\", and \"does\" are all considered to be unique words. For the purposes of spam classification, its probably better to process the data slightly to increase accuracy. In our case we can focus on letters and numbers, as well as convert all the comments to lower case." 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": 18, 510 | "metadata": {}, 511 | "outputs": [ 512 | { 513 | "name": "stdout", 514 | "output_type": "stream", 515 | "text": [ 516 | "Unique words in processed training data: 4129\n", 517 | "First 5 words in our processed unique set of words: \n", 518 | " ['jaesuk', 'partwierd', 'vanossbest', 'storyhttpshhortcomarhupweh5abyay', 'personally']\n" 519 | ] 520 | } 521 | ], 522 | "source": [ 523 | "# Only keep letters and numbers\n", 524 | "train_unique_words = [re.sub(r'[^a-zA-Z0-9]','', words) for words in train_unique_words]\n", 525 | "\n", 526 | "# Convert to lower case and get unique set of words\n", 527 | "train_unique_words = set([words.lower() for words in train_unique_words])\n", 528 | "\n", 529 | "# Number of unique words in training \n", 530 | "vocab_size_train = len(train_unique_words)\n", 531 | "\n", 532 | "# Description of summarized comments in training data\n", 533 | "print('Unique words in processed training data: %s' % vocab_size_train)\n", 534 | "print('First 5 words in our processed unique set of words: \\n % s' % list(train_unique_words)[1:6])" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": {}, 540 | "source": [ 541 | "## Naive Bayes for Spam Classification" 542 | ] 543 | }, 544 | { 545 | "cell_type": "markdown", 546 | "metadata": {}, 547 | "source": [ 548 | "Reffering to the Naive Bayes formulation above, in the context of comment spam classification, now $i$ is a index for the comment, the classes $C_{i} = \\{1 = Spam, 0 = Ham\\},$ and features of comments are the words in it, $W_{i}$. For example if the $ith$ comment as \"Check out my chanell!\", then $W_{i} = \\{Check, out, my, chanell!\\}.$ If we wanted to classify this comment using naive bayes, we need to compute $$Pr(Spam|\\{Check, out, my, chanell!\\}) \\propto Pr(Check|Spam) \\ldots Pr(chanell!|Spam)Pr(Spam),$$ and, $$Pr(Ham|\\{Check, out, my, chanell!\\}) \\propto Pr(Check|Ham) \\ldots Pr(chanell!|Ham)Pr(Ham),$$ \n", 549 | "note that the proportional symbol is used above since we are omitting $Pr(\\{Check, out, my, chanell!\\})$ in the denominator. \n", 550 | "\n", 551 | "Firstly, to find $Pr(Spam)$ and $Pr(Ham)$, we can just compute the proportion of spam and ham emails in our training data respectively. To find the probability of say \"Check\" appearing in spam emails, we can just compute the proportion of times \"Check\" appears in a spam email. That is, $$Pr(Check|Spam) = \\frac{Pr(\\text{Check and Spam})}{Pr(Spam)} = \\frac{\\text{Number of times \"Check\" appers in spam comments}}{\\text{Number of total words in spam comments + Number of unique words in data}}.$$ We can similarly compute $Pr(Check|Ham)$ and also do this with any other word in the comment. Training in Naive Bayes just means we are going to be computing a bunch of conditional probabilities, $Pr(\\text{word in comment}|\\text{comment label}).$ Therefore classification in Naive Bayes is simple as if $$Pr(Spam|Comment) > Pr(Ham|Comment) \\implies \\text{Comment is spam},$$ and if $$Pr(Spam|Comment) < Pr(Ham|Comment) \\implies \\text{Comment is ham}.$$ \n", 552 | "\n", 553 | "Given that we calculuate these conditonal probabilities using the words in our training data, what happens when a comment contains words in the training data? The $Pr(\\text{Word not in training data}|Spam) = 0,$ which will imply $Pr(Spam|\\text{Comment with word not in training data}) = 0.$ This is a problem because any spam email with a word not in training data will not be assigned as spam. To resolve this, we compute $$Pr(\\text{word in comment}|Spam) = \\frac{\\alpha+\\text{Number of times word appears in spam comments}}{\\text{Number of total words in spam comments + Number of unique words in data}},$$ where $alpha > 0$ is known as the laplace smoothing parameter. If we say $\\alpha = 1,$ then a $Pr(\\text{Word not in training data}|Spam)$ will be a small positive number instead of $0.$" 554 | ] 555 | }, 556 | { 557 | "cell_type": "code", 558 | "execution_count": 19, 559 | "metadata": { 560 | "collapsed": true 561 | }, 562 | "outputs": [], 563 | "source": [ 564 | "# Dictionary with comment words as \"keys\", and their label as \"value\"\n", 565 | "trainPositive = dict()\n", 566 | "trainNegative = dict()\n", 567 | "\n", 568 | "# Intiailize classes\n", 569 | "positiveTotal = 0\n", 570 | "negativeTotal = 0\n", 571 | "\n", 572 | "# Initialize Prob. of\n", 573 | "pSpam = 0.0\n", 574 | "pNotSpam = 0.0\n", 575 | "\n", 576 | "# Laplace smoothing\n", 577 | "alpha = 1" 578 | ] 579 | }, 580 | { 581 | "cell_type": "code", 582 | "execution_count": 20, 583 | "metadata": { 584 | "collapsed": true 585 | }, 586 | "outputs": [], 587 | "source": [ 588 | "#def initialize_dicts():\n", 589 | "\n", 590 | "# Initialize dictionary of words and their labels \n", 591 | "for word in train_unique_words:\n", 592 | " \n", 593 | " # Classify all words for now as ham (legitimate)\n", 594 | " trainPositive[word] = 0\n", 595 | " trainNegative[word] = 0" 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": 21, 601 | "metadata": { 602 | "collapsed": true 603 | }, 604 | "outputs": [], 605 | "source": [ 606 | "# Count number of times word in comment appear in spam and ham comments\n", 607 | "def processComment(comment,label):\n", 608 | " global positiveTotal\n", 609 | " global negativeTotal\n", 610 | " \n", 611 | " # Split comments into words\n", 612 | " comment = comment.split(' ')\n", 613 | " \n", 614 | " # Go over each word in comment\n", 615 | " for word in comment:\n", 616 | " \n", 617 | " # ham commments\n", 618 | " if(label == 0 and word != ' '):\n", 619 | " \n", 620 | " # Increment number of times word appears in ham comments\n", 621 | " trainNegative[word] = trainNegative.get(word,0)+1\n", 622 | " negativeTotal += 1\n", 623 | " \n", 624 | " # spam comments\n", 625 | " elif(label == 1 and word != ' '):\n", 626 | " \n", 627 | " # Increment number of times word appears in spam comments\n", 628 | " trainPositive[word] = trainPositive.get(word,0)+1\n", 629 | " positiveTotal += 1" 630 | ] 631 | }, 632 | { 633 | "cell_type": "code", 634 | "execution_count": 22, 635 | "metadata": { 636 | "collapsed": true 637 | }, 638 | "outputs": [], 639 | "source": [ 640 | "# Define Prob(word|spam) and Prob(word|ham)\n", 641 | "def conditionalWord(word,label):\n", 642 | " \n", 643 | " # Laplace smoothing parameter\n", 644 | " global alpha\n", 645 | " \n", 646 | " # word in ham comment\n", 647 | " if(label == 0):\n", 648 | " # Compute Prob(word|ham)\n", 649 | " return (trainNegative.get(word,0)+alpha)/(float)(negativeTotal+alpha*vocab_size_train)\n", 650 | " \n", 651 | " # word in spam comment\n", 652 | " else:\n", 653 | " \n", 654 | " # Compute Prob(word|ham)\n", 655 | " return (trainPositive.get(word,0)+alpha)/(float)(positiveTotal+alpha*vocab_size_train)" 656 | ] 657 | }, 658 | { 659 | "cell_type": "code", 660 | "execution_count": 23, 661 | "metadata": { 662 | "collapsed": true 663 | }, 664 | "outputs": [], 665 | "source": [ 666 | "# Define Prob(spam|comment) or Prob(ham|comment)\n", 667 | "def conditionalComment(comment,label):\n", 668 | " \n", 669 | " # Initialize conditional probability\n", 670 | " prob_label_comment = 1.0\n", 671 | " \n", 672 | " # Split comments into list of words\n", 673 | " comment = comment.split(' ')\n", 674 | " \n", 675 | " # Go through all words in comments\n", 676 | " for word in comment:\n", 677 | " \n", 678 | " # Compute value proportional to Prob(label|comment)\n", 679 | " # Conditional indepdence is assumed here\n", 680 | " prob_label_comment *= conditionalWord(word,label)\n", 681 | " \n", 682 | " return prob_label_comment" 683 | ] 684 | }, 685 | { 686 | "cell_type": "code", 687 | "execution_count": 24, 688 | "metadata": { 689 | "collapsed": true 690 | }, 691 | "outputs": [], 692 | "source": [ 693 | "# Train naive bayes by computing several conditional probabilities in training data\n", 694 | "def train():\n", 695 | " \n", 696 | " print('Starting training')\n", 697 | " global pSpam\n", 698 | " global pNotSpam\n", 699 | "\n", 700 | " # Initiailize \n", 701 | " total = 0\n", 702 | " numNegative = 0\n", 703 | " \n", 704 | " # Go over each comment in training data\n", 705 | " for idx, comment in data_comments_train.iterrows():\n", 706 | " \n", 707 | " # Comment is ham \n", 708 | " if comment.label == 0:\n", 709 | " \n", 710 | " # Increment ham comment counter\n", 711 | " numNegative += 1\n", 712 | " \n", 713 | " # Increment comment number\n", 714 | " total += 1\n", 715 | " \n", 716 | " # Update dictionary of ham and spam comments\n", 717 | " processComment(comment.content,comment.label)\n", 718 | " \n", 719 | " # Compute prior probabilities, P(spam), P(ham)\n", 720 | " pSpam = numNegative/float(total)\n", 721 | " pNotSpam = (total - numNegative)/float(total)\n", 722 | " \n", 723 | " print('Training is now finished')" 724 | ] 725 | }, 726 | { 727 | "cell_type": "code", 728 | "execution_count": 25, 729 | "metadata": {}, 730 | "outputs": [ 731 | { 732 | "name": "stdout", 733 | "output_type": "stream", 734 | "text": [ 735 | "Starting training\n", 736 | "Training is now finished\n" 737 | ] 738 | } 739 | ], 740 | "source": [ 741 | "# Run naive bayes\n", 742 | "train()" 743 | ] 744 | }, 745 | { 746 | "cell_type": "code", 747 | "execution_count": 26, 748 | "metadata": { 749 | "collapsed": true 750 | }, 751 | "outputs": [], 752 | "source": [ 753 | "# Classify comment are spam or ham\n", 754 | "def classify(comment):\n", 755 | " \n", 756 | " global pSpam\n", 757 | " global pNotSpam\n", 758 | " \n", 759 | " # Compute value proportional to Pr(comment|ham)\n", 760 | " isNegative = pSpam * conditionalComment(comment,0)\n", 761 | " \n", 762 | " # Compute value proportional to Pr(comment|spam)\n", 763 | " isPositive = pNotSpam * conditionalComment(comment,1)\n", 764 | " \n", 765 | " # Output True = spam, False = ham\n", 766 | " return (isNegative < isPositive)" 767 | ] 768 | }, 769 | { 770 | "cell_type": "code", 771 | "execution_count": 27, 772 | "metadata": {}, 773 | "outputs": [ 774 | { 775 | "name": "stdout", 776 | "output_type": "stream", 777 | "text": [ 778 | "Proportion of comments classified correctly on test set: 0.816455696203\n" 779 | ] 780 | } 781 | ], 782 | "source": [ 783 | "# Initialize spam prediction in test data\n", 784 | "prediction_test = []\n", 785 | "\n", 786 | "# Get prediction accuracy on test data\n", 787 | "for comment in data_comments_test[\"content\"]:\n", 788 | "\n", 789 | " # Classify comment \n", 790 | " prediction_test.append(classify(comment))\n", 791 | "\n", 792 | "# Check accuracy\n", 793 | "test_accuracy = np.mean(np.equal(prediction_test, data_comments_test[\"label\"]))\n", 794 | "\n", 795 | "#print prediction_test\n", 796 | "print(\"Proportion of comments classified correctly on test set: %s\" % test_accuracy)" 797 | ] 798 | }, 799 | { 800 | "cell_type": "markdown", 801 | "metadata": {}, 802 | "source": [ 803 | "Let me try writing some comments to see whether they are classified as spam or ham. Recall the \"True\" is for spam comments, and \"False\" is for ham comments. " 804 | ] 805 | }, 806 | { 807 | "cell_type": "code", 808 | "execution_count": 32, 809 | "metadata": {}, 810 | "outputs": [ 811 | { 812 | "data": { 813 | "text/plain": [ 814 | "True" 815 | ] 816 | }, 817 | "execution_count": 32, 818 | "metadata": {}, 819 | "output_type": "execute_result" 820 | } 821 | ], 822 | "source": [ 823 | "# spam\n", 824 | "classify(\"Guys check out my new chanell\")" 825 | ] 826 | }, 827 | { 828 | "cell_type": "code", 829 | "execution_count": 33, 830 | "metadata": {}, 831 | "outputs": [ 832 | { 833 | "data": { 834 | "text/plain": [ 835 | "True" 836 | ] 837 | }, 838 | "execution_count": 33, 839 | "metadata": {}, 840 | "output_type": "execute_result" 841 | } 842 | ], 843 | "source": [ 844 | "# spam\n", 845 | "classify(\"I have solved P vs. NP, check my video https://www.youtube.com/watch?v=dQw4w9WgXcQ\")" 846 | ] 847 | }, 848 | { 849 | "cell_type": "code", 850 | "execution_count": 34, 851 | "metadata": {}, 852 | "outputs": [ 853 | { 854 | "data": { 855 | "text/plain": [ 856 | "False" 857 | ] 858 | }, 859 | "execution_count": 34, 860 | "metadata": {}, 861 | "output_type": "execute_result" 862 | } 863 | ], 864 | "source": [ 865 | "# ham\n", 866 | "classify(\"I liked the video\")" 867 | ] 868 | }, 869 | { 870 | "cell_type": "code", 871 | "execution_count": 35, 872 | "metadata": {}, 873 | "outputs": [ 874 | { 875 | "data": { 876 | "text/plain": [ 877 | "False" 878 | ] 879 | }, 880 | "execution_count": 35, 881 | "metadata": {}, 882 | "output_type": "execute_result" 883 | } 884 | ], 885 | "source": [ 886 | "# ham\n", 887 | "classify(\"Its great that this video has so many views\")" 888 | ] 889 | }, 890 | { 891 | "cell_type": "markdown", 892 | "metadata": {}, 893 | "source": [ 894 | "## Extending Bag of Words by Using TF-IDF\n", 895 | "So far we have been using the Bag of Words model to represent comments as vectors. The \"Bag of Words\" is a list of all unique words found in the training data, then each comment can be represented by a vector that contains the frequency of each unique word that appeared in the comment. For example if the training data contains the words $(hi, how, my, grade, are, you),$ then the text \"how are you you\" can be represented by $(0,1,0,0,1,2).$ The main reason we do this in our application is because comments can vary in length, but the length of all unique words stays fixed. \n", 896 | "\n", 897 | "In our context, the TF-IDF is a measure of how important a word is in a comment relative to all the words in our training data. For example if a word such as \"the\" appeared in most of the comments, the TF-IDF would be small as this word does not help us differentiate accross comments. Note that \"TF\" stands for \"Term Frequency\", and \"IDF\" stands for \"Inverse Document Frequency\". In particular, \"TF\" denoted by $tf(w,c)$ is the number of times the word $w$ appears in the given comment $c$. Whereas \"IDF\" is a measure of how much information a given word provides in differentiating comments. Specefically, $IDF$ is formulated as $idf(w, D) = log(\\frac{\\text{Number of comments in train data $D$}}{\\text{Number of comments containing the word $w$}}).$ To combine \"TF\" and \"IDF\" together, we simple take the product, hence $$TFIDF = tf(w,c) \\times idf(w, D) = (\\text{Number of times $w$ appears in comment $c$})\\times log(\\frac{\\text{Number of comments in train data $D$}}{\\text{Number of comments containing the word $w$}}).$$\n", 898 | "Now the $TF-IDF$ can be used to weight the vectors that result from the \"Bag of Words\" approach. For example, suppose a comment contains \"this\" 2 times, hence $tf = 2$. If we then had 1000 comments in our traininig data, and the word \"this\" appears in 100 comments, $idf = log(1000/100) = 2.$ Therefore in this example, the TF-IDF weight would be 2*2 = 4 for the word \"this\" appear twice in a particular comment. To incorprate TF-IDF into the naive bayes setting, we can compute $$Pr(word|spam) = \\frac{\\sum_{\\text{c is spam}}TFIDF(word,c,D)}{\\sum_{\\text{word in spam c}}\\sum_{\\text{c is spam}}TFIDF(word,c,D)+ \\text{Number of unique words in data}},$$ where $TFIDF(word,c,D) = TF(word,c) \\times IDF(word,data).$ " 899 | ] 900 | }, 901 | { 902 | "cell_type": "code", 903 | "execution_count": 37, 904 | "metadata": { 905 | "collapsed": true 906 | }, 907 | "outputs": [], 908 | "source": [ 909 | "# Compute tfidf(word, comment, data)\n", 910 | "def TFIDF(comment, train):\n", 911 | " \n", 912 | " # Split comment into list of words\n", 913 | " comment = comment.split(' ')\n", 914 | " \n", 915 | " # Initiailize tfidf for given comment\n", 916 | " tfidf_comment = np.zeros(len(comment))\n", 917 | " \n", 918 | " # Initiailize number of comments containing a word\n", 919 | " num_comment_word = 0\n", 920 | " \n", 921 | " # Intialize index for words in comment\n", 922 | " word_index = 0\n", 923 | " \n", 924 | " # Go over all words in comment\n", 925 | " for word in comment:\n", 926 | " \n", 927 | " # Compute term frequence (tf)\n", 928 | " # Count frequency of word in comment\n", 929 | " tf = comment.count(word)\n", 930 | " \n", 931 | " # Find number of comments containing word\n", 932 | " for text in train[\"content\"]:\n", 933 | " \n", 934 | " # Increment word counter if word found in comment\n", 935 | " if text.split(' ').count(word) > 0:\n", 936 | " num_comment_word += 1\n", 937 | " \n", 938 | " # Compute inverse document frequency (idf)\n", 939 | " # log(Total number of comments/number of comments with word)\n", 940 | " idf = np.log(len(train.index)/num_comment_word)\n", 941 | " \n", 942 | " # Update tf-idf weight for word\n", 943 | " tfidf_comment[word_index] = tf*idf\n", 944 | " \n", 945 | " # Reset comment containing word counter\n", 946 | " num_comment_word = 0\n", 947 | " \n", 948 | " # Move onto next word in comment\n", 949 | " word_index += 1\n", 950 | " \n", 951 | " return tfidf_comment" 952 | ] 953 | }, 954 | { 955 | "cell_type": "code", 956 | "execution_count": 38, 957 | "metadata": {}, 958 | "outputs": [ 959 | { 960 | "data": { 961 | "text/plain": [ 962 | "array([ 2.06142304, 1.54111867, 1.84784894, 3.63960841, 3.17603567,\n", 963 | " 2.04047986, 5.3572599 ])" 964 | ] 965 | }, 966 | "execution_count": 38, 967 | "metadata": {}, 968 | "output_type": "execute_result" 969 | } 970 | ], 971 | "source": [ 972 | "TFIDF(\"Check out my new music video plz\",data_comments_train)" 973 | ] 974 | }, 975 | { 976 | "cell_type": "markdown", 977 | "metadata": {}, 978 | "source": [ 979 | "## Conclusion" 980 | ] 981 | }, 982 | { 983 | "cell_type": "markdown", 984 | "metadata": {}, 985 | "source": [ 986 | "In this notebook we introduced the Bayes rule and how it is used by the Naive Bayes algorithm for text classification. Using Naive Bayes with bag of words for youtube comment spam classification resulted in about $82 \\%$ accuracy on the training data. Since the model is trained on youtube music videos comments, it may not be very accurate in classifying spam and ham comments for other videos. $\\textbf{TFIDF is in progress, add in accuracy once complete}$" 987 | ] 988 | } 989 | ], 990 | "metadata": { 991 | "kernelspec": { 992 | "display_name": "Python 2", 993 | "language": "python", 994 | "name": "python2" 995 | }, 996 | "language_info": { 997 | "codemirror_mode": { 998 | "name": "ipython", 999 | "version": 2 1000 | }, 1001 | "file_extension": ".py", 1002 | "mimetype": "text/x-python", 1003 | "name": "python", 1004 | "nbconvert_exporter": "python", 1005 | "pygments_lexer": "ipython2", 1006 | "version": "2.7.12" 1007 | } 1008 | }, 1009 | "nbformat": 4, 1010 | "nbformat_minor": 2 1011 | } 1012 | -------------------------------------------------------------------------------- /Naive Bayes Classifier/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | Naive Bayes is a classification algorithm that uses the Bayes theorem together with a assumption of conditional independence to assign probabilities to all classes given the observation features. The notebook "Naive Bayes.ipynb" introduces the Naive Bayes algorithm, and applies it to classify Youtube comments as spam or ham (legitimate comments). 3 | 4 | ## Data 5 | The data used in the Naive Bayes notebook is from the UCI machine learning repository. It contains several comments from very popular music videos from youtube and has labels for each comment being spam or ham. The youtube spam collection data set can be found [here](https://archive.ics.uci.edu/ml/datasets/YouTube+Spam+Collection). 6 | 7 | ## Work in Progress 8 | I am trying to use TF-IDF instead of bag of words model and compare whether the classification accuracy on the test set increases. 9 | 10 | ## Dependencies 11 | * numpy 12 | * Pandas 13 | * re 14 | 15 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 16 | 17 | ## References 18 | [Siraj's video on Probability and Naive Bayes](https://www.youtube.com/watch?v=PrkiRVcrxOs) 19 | 20 | [Great introduction to Bayes Theorem](https://www.youtube.com/watch?v=R13BD8qKeTg&t=551s) 21 | 22 | [Simple introduction to Naive Bayes](http://machinelearningmastery.com/naive-bayes-tutorial-for-machine-learning/) 23 | 24 | [Naive Bayes from Scratch in Python on Diabetes Classification](http://machinelearningmastery.com/naive-bayes-classifier-scratch-python/) 25 | 26 | [Alberto Blanco notebook on sentiment analysis](https://github.com/alberduris/The_Math_of_Intelligence/tree/master/Week6) 27 | -------------------------------------------------------------------------------- /Newtons Method for Optimization/Newtons Method.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Newton's Method for Optimization in Python \n", 8 | "\n", 9 | "## Introduction\n", 10 | "\n", 11 | "We will implement Newton's Method, a second order optimization algorithm from scratch in python. Given a real valued function $f(x)$, the goal is to find number(s) $c$ such that $f'(c) = 0.$ Suppose we have a function $f: \\mathbb{R} \\to \\mathbb{R}$, the Newton's metod gives us a itterative algorith to find the zero's of f(x). In particular, the itterative formula for finding $x^*$ such that f($x^*$) = 0 is \n", 12 | "\\begin{equation}\n", 13 | "x_{t+1} = x_t - \\frac{f(x_t)}{f'(x_t)},\n", 14 | "\\end{equation}\n", 15 | "\n", 16 | "where $t$ is the itteration index. As $t$ gets very large, it turns out that $x_{t+1}$ will get very close to $x_{t}$ and converge towards $x^*$. \n", 17 | "\n", 18 | "## Intuition behind Newton's method formula\n", 19 | "Again let's suppose $f: \\mathbb{R} \\to \\mathbb{R},$ and $f(x^*) = 0.$ We can approximate $f(x)$ at $x^*$ using the taylor series expansion, that is $$f(x^*) \\approx f(x) + f'(x)(x^*-x).$$ Now since $f(x^*) = 0,$ we can rearrage the above equation and get $$x^* \\approx x - \\frac{f(x)}{f'(x)},$$ if $x$ is somewhat close to $x^*$ this suggests the itterative Newton's method can find the zeros of $f(x).$\n", 20 | "\n", 21 | "## Newton's method for optimization\n", 22 | "In machine learning we are often interested in maximizing or minimizing a function $f(x)$. This is equivalent to saying that we want to find the zeros of $f'(x) = 0.$ Simply adjusting the newton's method above, we can itteratively compute the zeros of $f'(x)$ using \n", 23 | "\\begin{equation}\n", 24 | "x_{t+1} = x_t - \\frac{f'(x_t)}{f''(x_t)},\n", 25 | "\\end{equation}\n", 26 | "where $t$ is the itteration index. Once $x_{t+1}$ converges to $x^*$, then $f'(x^*) = 0.$ So far we have been dealing with a one simple one dimensional function $f: \\mathbb{R} \\to \\mathbb{R}$, however the newton's method can easily be extended to apply to functions $g: \\mathbb{R}^{n} \\to \\mathbb{R}$ by replacing the first $g'(x)$ with a gradient vector, and $g''(x)$ with a hessian matrix. More precisely, the newton's update formula for optimizing a real valued funciton $g$ with a n-dimensional input vector would be\n", 27 | "\n", 28 | "$$\n", 29 | "\\begin{bmatrix}\n", 30 | " x_{0}^{t+1} \\\\ \n", 31 | " \\vdots \\\\\n", 32 | " x_{n}^{t+1}\n", 33 | "\\end{bmatrix}\n", 34 | "=\n", 35 | "\\begin{bmatrix}\n", 36 | " x_{0}^{t} \\\\ \n", 37 | " \\vdots \\\\\n", 38 | " x_{n}^{t}\n", 39 | "\\end{bmatrix}\n", 40 | "-\n", 41 | "\\begin{bmatrix}\n", 42 | " \\frac{d^2 g(x_{0})}{dx_{0,t}^{2}} & \\ldots & \\frac{d^2 g(x_{0})}{dx_{0,t}dx_{n,t}} \\\\ \n", 43 | " \\vdots & \\ldots & \\vdots \\\\\n", 44 | " \\frac{d^2 g(x_{n})}{dx_{n,t}dx_{0,t}} & \\ldots & \\frac{d^2 g(x_{n})}{dx_{n,t}^2}\n", 45 | "\\end{bmatrix}^{-1}\n", 46 | "\\begin{bmatrix}\n", 47 | " \\frac{dg(x_{0})}{dx_{0,t}} \\\\ \n", 48 | " \\vdots \\\\\n", 49 | " \\frac{dg(x_{n})}{dx_{n,t}} \\\\ \n", 50 | "\\end{bmatrix},\n", 51 | "$$\n", 52 | "where $t$ is the newton's update itteration. An example with n = 2 and real data will be shown below. \n", 53 | "\n", 54 | "\n", 55 | "## Application of Newton's method to IMDB movie data\n", 56 | "Now that we have a basic sense of the newton's method, we will use to estimate the parameters in a linear regression framework to model the relationship between IMDB's movies score (rating out of 10) and the movies gross sales (in dollars). We will use a data set from https://www.kaggle.com/deepmatrix/imdb-5000-movie-dataset. The data contains various statistics from IMDB for over 5000 movies. For our purposes we will only be interested in the IMDB rating and gross sales." 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 9, 62 | "metadata": { 63 | "collapsed": true 64 | }, 65 | "outputs": [], 66 | "source": [ 67 | "# Import the standard packages for doing math operations, data manipulation, and plotting\n", 68 | "import numpy as np\n", 69 | "import pandas as pd\n", 70 | "import matplotlib.pyplot as plt" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 10, 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "data": { 80 | "text/plain": [ 81 | "array(['color', 'director_name', 'num_critic_for_reviews', 'duration',\n", 82 | " 'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',\n", 83 | " 'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',\n", 84 | " 'movie_title', 'num_voted_users', 'cast_total_facebook_likes',\n", 85 | " 'actor_3_name', 'facenumber_in_poster', 'plot_keywords',\n", 86 | " 'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',\n", 87 | " 'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',\n", 88 | " 'imdb_score', 'aspect_ratio', 'movie_facebook_likes'], dtype=object)" 89 | ] 90 | }, 91 | "execution_count": 10, 92 | "metadata": {}, 93 | "output_type": "execute_result" 94 | } 95 | ], 96 | "source": [ 97 | "# Load the IMDB data\n", 98 | "movie_data = pd.read_csv(\"movie_metadata.csv\")\n", 99 | "\n", 100 | "# Show all the variables in the data set\n", 101 | "movie_data.columns.values" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "Let's clean the data set a little bit. We will only keep the US movies since some of the gross revenues for movies from other countries are not reported in dollars. We will also only keep the imdb score and gross revenue variables for our analysis and drop the rest. I also informally convert the movie gross sales revenue in terms of 2017 dollar by assuming a $2.5\\%$ yearly inflation rate. (To formally convert to 2017 dollars we would need to use consumer price index data from the US bureau of labour statistics) Also since we are dealing with real data, we need to take missing values into consideration. " 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 11, 114 | "metadata": {}, 115 | "outputs": [ 116 | { 117 | "data": { 118 | "text/html": [ 119 | "
\n", 120 | "\n", 133 | "\n", 134 | " \n", 135 | " \n", 136 | " \n", 137 | " \n", 138 | " \n", 139 | " \n", 140 | " \n", 141 | " \n", 142 | " \n", 143 | " \n", 144 | " \n", 145 | " \n", 146 | " \n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | "
grossimdb_score
count3235.0000003235.000000
mean79.7646616.385533
std109.8421431.049178
min0.0010441.600000
25%14.1323495.800000
50%44.2107346.500000
75%102.2072587.100000
max1363.2063769.300000
\n", 184 | "
" 185 | ], 186 | "text/plain": [ 187 | " gross imdb_score\n", 188 | "count 3235.000000 3235.000000\n", 189 | "mean 79.764661 6.385533\n", 190 | "std 109.842143 1.049178\n", 191 | "min 0.001044 1.600000\n", 192 | "25% 14.132349 5.800000\n", 193 | "50% 44.210734 6.500000\n", 194 | "75% 102.207258 7.100000\n", 195 | "max 1363.206376 9.300000" 196 | ] 197 | }, 198 | "execution_count": 11, 199 | "metadata": {}, 200 | "output_type": "execute_result" 201 | } 202 | ], 203 | "source": [ 204 | "# Drop rows with missing gross revenue data (standard practice in regression analysis)\n", 205 | "movie_data = movie_data.dropna(subset = ['gross'])\n", 206 | "\n", 207 | "# Only keep data on movies from the US\n", 208 | "movie_data = movie_data[movie_data['country'] == \"USA\"]\n", 209 | "\n", 210 | "# Assuming a average US inflation rate of 2.5%, we convert gross revenue in terms of 2017 dollars\n", 211 | "movie_data[['gross']] = (1.025**(2017-movie_data['title_year']))*movie_data['gross']\n", 212 | "\n", 213 | "# Only keep the variables of interest, 'imdb_score' and 'gross'\n", 214 | "movie_data = movie_data[['gross','imdb_score']]\n", 215 | "\n", 216 | "# Let's scale the gross revenue to be in millions of dollars so its easier to read\n", 217 | "movie_data[['gross']] = movie_data[['gross']]/1000000\n", 218 | "\n", 219 | "# Summary statistics\n", 220 | "movie_data.describe()" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "From the above summary statistics, we notice the average IMDB US movie rating is 6.4, and average gross sales revenue is about 80 million." 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 12, 233 | "metadata": {}, 234 | "outputs": [ 235 | { 236 | "data": { 237 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcXGWZ77+/dDrQAU0HiAw0ZBIRw6AswVZw4sKiBIlK\nBAURFJArzgwuLEaSGa6Aw1ziRIfRcbkXRIWRYREwgmEIyOLCCJKQQAwQDYQlDUgUEpY0pNN57h/n\nPZ3TlTp1zqmu6qrqPN/Ppz9d9Z63znmqq+t9zvusMjMcx3EcJy+jGi2A4ziO01q44nAcx3EK4YrD\ncRzHKYQrDsdxHKcQrjgcx3GcQrjicBzHcQrhisPZ6pD0bkkrGi1HFpJOlvSbRssxnEgySW9qtBxO\nZVxxOGWR9Lik94XHJ4cv9MUlc44K4z8KzyeF5y+Hnz9J+rmk95c5d2+Y84KkBZJ2ryDLXZJeDfP/\nLOkGSbsUeC+DFiMz+7WZTcn7+mZF0hhJX5G0QtIrknok/bekwxss11GSlkp6MXxed0ia3EiZnNri\nisPJy6PAsZJGJ8ZOAv5QZm6nmW0P7AfcBvxU0sklcz4U5uwC/An4j4zrfy7MfxOwPfD14m9hxHEd\ncBTwKWA8MBn4JjCj3OSSz64uBAV9BXA2MC7I9B2gv97XdoYPVxxOXp4FlgHTASTtAPwtcGPaC8zs\nWTP7JnA+8DVJW/y/mdmrRAvg3nmEMLO1wHxg/3hM0jsk/VbSWknPSPq2pDHh2K/CtAfCjuU4SQdL\nWp14/eOSviTpQUnrJF0jadvE8S+H8z4t6X9VMqdIOkXSw5JekvSYpM8mjh0sabWksyU9F855SuL4\njpJuDHfqvwP2SPs7hN3g+4GjzOxeM9sQfm4xsy+WvLdzJD0IvCJptKS/Cbu4tZKWS/pwYv6Rkh4K\n8vdI+lIY3ynsHtdKel7Sr8t9nuFzWWVmt1vES2Z2vZk9mfVZlXmP20j6uqQnw+71/0rqKCiPUwf8\nD+0U4Qqiu1uAjwM/A17L8bobgDcAW5iHJI0FjgPuySOApB2Bo4GVieF+4ExgJ+CdwGHAPwCY2XvC\nnP3MbHszuybl1McCRxDdIe8LnByudwRwFvA+ot3OwRkiPgd8EHg9cApwsaQDEsf/iuhOvAs4FfiO\npPHh2HeAV4l2YZ8OP2m8D7jXzFZXmBNzPNEupBMQcBNwK9Fn8nngSknxZ3MZ8Fkzex3wVuCOMH42\nsBqYAOwM/CNQrl7R/cBeki6WdIik7UuOp35WZZgLvJlIGb2J6G/2lYLyOHXAFYdThJ8CB0saR6RA\nrsj5uqfD7x0SY/MlrQXWEd05z8s4x7ckrQP+TLTofD4+YGaLzeweM9toZo8D/w94b07ZBs5vZk+b\n2fNEC2u8ozkW+KGZLTez9US7p1TMbIGZPRrutn9JtEC/OzGlD/iqmfWZ2c3Ay8AUSW3AMcBXzOwV\nM/s9cHmFS+1EtAsEoh1guPteJ+nVMu/tKTPrBQ4iMvXNDTuUO4CfEymXWL69Jb3ezF4ws/sT47sA\nfx1k/7WVKXRnZo8RKdcu4Frgz5J+FCuQvJ+VJAGnAWea2fNm9hLwf4huWHLL49QHVxxObsLCswA4\nF9jRzO7O+dKu8Pv5xNhMM+sEtgU+B/xS0l9VOMcXzGwc0W5gPLBbfEDSm4PZ4llJLxItMDvllC3m\n2cTj9USLK8CuwFOJY8nHWyDpA5LuCeaTtcCRJbL8xcw2lrnWBGB0yfmfqHCpvxAtnACExbUTeBuw\nTcnc5Dl3BZ4ys00l14k/o2OCzE9I+qWkd4bxeUS7vFuDCW52mmBBMRxrZhOIlOZ7gH+CQp/VBGAs\nsDgoxLXALWG8kDxO7XHF4RQldnz+uMBrPkJkwtkiBNbM+s3sBiITxruyTmRmy4ALiUw8CsPfAx4B\n9jSz1xOZLZRyiqI8Q0JJAZWiv7YBridy3O8cFvKbc8qyBthYcv6JFebfDrxd0m4V5sQk78SfBnYv\n8QdMBHoAzOw+MzuKyIw1n2jXQPBVnG1mbwQ+DJwl6bDMC5vdR2SqfGsYyvtZ/RnoBd5iZp3hZ1wI\nkKhaHqc2uOJwivJLItNSVhQUknaW9DngPGBOyV1uPEeSjiLaRTycU4bLiezasVP3dcCLwMuS9gL+\nvmT+n4A35jx3KdcCpwSH8ljgf1eYO4bobn8NsFHSB4BcobFm1k+0wJ4vaaykvYmi1tLm3wrcSWTy\nO1BRaG47kSmqEvcS7XK+LKld0sHAh4CrwzlOkDTOzPqI/qabACR9UNKbgrJeR6Toy32e75L0GUlv\nCM/3IvqcYh9W1mcVv79NwKVEPqL4XF2S4uCMXPI49cEVh1OIYLu/PfgC0lgr6RWiKKwjgY+Z2Q9K\n5twk6WWiReRfgJPMbHlOGTYQhZ3Gi/iXgE8ALxEtNqUO8POBy4PJ49g810hc67+BbxEt0ivZvABu\nERQQ7PBfIFI2LwSZUqPOyvA5IrPVs8CPgB9mzP8IkX/ix8BaYBVwAiHyrRzhb/ch4ANEd/XfBT5l\nZo+EKZ8EHg9mpL8L5wPYE/gFkU/mt8B3zezOMpdYS6QoloXP9xYi39i/huNZn1WScwh/8yDPL9gc\nYJFXHqcOyP1JjpMfSX8D/B7YpsRX4ThbDb7jcJwMJH0k5BSMB74G3ORKw9maccXhONl8lsi5/yiR\nLb2sXd5xthbcVOU4juMUwnccjuM4TiHqXvSsEey00042adKkRovhOI7TUixevPjPIXGzIiNScUya\nNIlFixY1WgzHcZyWQlKlagUD1M1UJekHiiqA/r7MsbMVVRjdKTyXpG9JWqmoQukBibknSfpj+ElN\niHIcx3GGh3r6OH5EVG10EIoa9hwOPJkY/gBRQs+eRIXNvhfm7kCUdXwg8A7gvEQlUcdxHKcB1E1x\nmNmvGFzULuZi4MsMrp9zFHBFyEq+B+hU1OFtOnBbKOD2AlFToC2UkeM4jjN8DGtUVahJ1GNmD5Qc\n6mJwBc/VYSxtvNy5T5O0SNKiNWvW1FBqx3EcJ8mwKY5QIO4f2dyIpaaY2SVm1m1m3RMmZAYFOI7j\nOFUynFFVexB1V3sgVMPeDbhf0juISjony0nvFsZ6GNxxbTfgrmGQ1XGcrZz5S3qYt3AFT6/tZdfO\nDmZNn8LMqWUNHlsdw7bjMLNlZvYGM5tkZpOIzE4HmNmzRBVEPxWiqw4C1pnZM8BC4HBJ44NT/PAw\n5jiOUzfmL+lhzg3L6FnbiwE9a3uZc8My5i/pabRoTUE9w3GvIip3PEXSakmnVph+M/AYUQnlS9nc\nL/p54J+B+8LPVzPKeTuO4wyZeQtX0NvXP2ist6+feQu36EW2VVI3U5WZHZ9xfFLisQGnp8z7AVDa\ny8FxHKduPL22t9D41obXqnIcxylh186OQuNbG644HMcZMcxf0sO0uXcwefYCps29o2qfxKzpU+ho\nbxs01tHexqzpU1JesXUxImtVOY6z9RE7tGPfROzQBgpHQ8XzPaqqPK44HMcZEVRyaFez4M+c2uWK\nIgU3VTmOMyJwh/bw4YrDcZwRgTu0hw9XHI7jjAjcoT18uI/DcZwRgTu0hw9XHI7jjBjcoT08uKnK\ncRzHKYQrDsdxHKcQrjgcx3GcQrjicBzHcQrhisNxHMcphEdVOY7jDAMjqaOgKw7HcZw6U8sCjM2A\nm6ocx3HqzEjrKOiKw3Ecp86MtAKMrjgcx3HqzEgrwOiKw3Ecp86MtAKMmYpD0m6SviTpZ5Luk/Qr\nSd+VNENS6usl/UDSc5J+nxibJ+kRSQ9K+qmkzsSxOZJWSlohaXpi/IgwtlLS7KG8WcdxnEYwc2oX\nFx29D12dHQjo6uzgoqP3aUnHOIDMLP2g9EOgC/g5sAh4DtgWeDNwCPA2YLaZ/arMa98DvAxcYWZv\nDWOHA3eY2UZJXwMws3Mk7Q1cBbwD2BX4RbgGwB+A9wOrgfuA483soUpvqru72xYtWpTrD+A4juNE\nSFpsZt1Z87LCcb9hZr8vM/574AZJY4CJ5V5oZr+SNKlk7NbE03uAj4bHRwFXm9lrwCpJK4mUCMBK\nM3sMQNLVYW5FxeE4juPUj4qmqnJKQ9J4SfuG4xvMbGWV1/408N/hcRfwVOLY6jCWNr4Fkk6TtEjS\nojVr1lQpkuM4jpNFLue4pLskvV7SDsD9wKWSLq72opL+CdgIXFntOUoxs0vMrNvMuidMmFCr0zqO\n4zgl5I2qGmdmLwJHE/ksDgQOq+aCkk4GPgicYJsdLD3A7olpu4WxtHHHcRynQeRVHKMl7QIcS+Qo\nrwpJRwBfBj5sZusTh24EPi5pG0mTgT2B3xE5w/eUNDn4Uz4e5jqO4zgNIm+tqq8CC4HfmNl9kt4I\n/LHSCyRdBRwM7CRpNXAeMAfYBrhNEsA9ZvZ3ZrZc0rVETu+NwOlm1h/O87lw7TbgB2a2vOB7dBzH\ncWpIxXDcVsXDcR3HcYpTq3Dc+GQTgM8Ak5KvMbNPVyug4ziO05rkNVX9DPg1UWJef8Zcx3EcZwST\nV3GMNbNz6iqJ4ziO0xLkjar6uaQj6yqJ4ziO0xLkVRxfJFIer0p6Kfy8WE/BHMdxnOYkl6nKzF5X\nb0Ecx3Gc1iB3z3FJHwbeE57eZWZVJwI6juM4rUveWlVzicxVD4WfL0q6qJ6COY7jOM1J3h3HkcD+\nZrYJQNLlwBKiTHDHcZymYP6SHuYtXMHTa3vZtbODWdOntGyzpGYmt6kK6ASeD4/H1UEWx3Gcqpm/\npIc5Nyyjty9KNetZ28ucG5YBuPKoMXmjqi4Clkj6UdhtLAb+pX5iOY7jFGPewhUDSiOmt6+feQtX\nNEiikUveqKqrJN0FvD0MnWNmz9ZNKsdxnII8vba30LhTPRV3HJL2Cr8PAHYh6sC3Gtg1jDmO4zQF\nu3Z2FBp3qidrx3E2UXHDb5Q5ZsChNZfIcRynCmZNnzLIxwHQ0d7GrOlTGijVyKSi4jCzz4TfhwyP\nOI7jONURO8A9qqr+VFQcko6udNzMbqitOI7jONUzc2pXqqLwUN3akWWq+lCFYwa44nAcp+nxUN3a\nkmWqOmW4BHEcx6kXlUJ1XXEUJ8tUdVal42b2b7UVx3Ecp/Z4qG5tyTJVeVVcx3Fanl07O+gpoyQ8\nVLc6skxVF1R7Ykk/AD4IPGdmbw1jOwDXEPUufxw41sxekCTgm0Q1sdYDJ5vZ/eE1JwHnhtNeaGaX\nVyuT4zhbJx6qW1uyTFVfNrN/lfQfRM7wQZjZFyq8/EfAt4ErEmOzgdvNbK6k2eH5OcAHgD3Dz4HA\n94ADg6I5D+gO118s6UYzeyHn+3Mcx/FQ3RqTZap6OPxeVPTEZvYrSZNKho8CDg6PLwfuIlIcRwFX\nmJkB90jqlLRLmHubmT0PIOk24AjgqqLyOI6zdVMpVNcpRpap6qbwu1bmoZ3N7Jnw+Flg5/C4C3gq\nMW91GEsb3wJJpwGnAUycOLFG4jqO4zil5G3k1C3pp5Lul/Rg/DOUC4fdxRbmryGc7xIz6zaz7gkT\nJtTqtI7jOE4JeftxXAnMApYBm4ZwvT9J2sXMngmmqOfCeA+we2LebmGsh82mrXj8riFc33Ecxxki\neftxrDGzG81slZk9Ef9Ucb0bgZPC45OAnyXGP6WIg4B1waS1EDhc0nhJ44HDw5jjOI7TIPLuOM6T\n9H3gduC1eLBSrSpJVxHtFnaStJooOmoucK2kU4EngGPD9JuJQnFXEoXjnhLO/7ykfwbuC/O+GjvK\nHcdxnMaQV3GcAuwFtLPZVFWxVpWZHZ9y6LAycw04PeU8PwB+kFNOx3Ecp87kVRxvNzPPlHEcx3Fy\n+zj+R9LedZXEcRzHaQny7jgOApZKWkXk4xCRhWnfuknmOI5TQ7wfR+3IqziOqKsUjuM4dcT7cdSW\nXIqjytBbx3GcpsD7cdSWvD4Ox3GclsX7cdQWVxyO44x40vpueD+O6nDF4TjOiGfW9Cl0tLcNGvN+\nHNWTqTgkHSfpjeHxvpJWSnpa0jH1F89xHGfozJzaxUVH70NXZwcCujo7uOjofdy/USWKkrYrTJAW\nAdPM7DVJPwMuAe4HFjZrOG53d7ctWlS4hYjjOM5WjaTFZtadNS+rA+B5wK7AOZLagHcBS4g68o2T\n9BXgLjP7VQ1kdhzHcVqAzJ7jkg4BVgETgFvM7HwASdPN7Kv1F9FxHMdpJvI4x/8e+CCwP/AlgFB+\nZEEd5XIcx3GalMwEQDN7GDiuZOwh4KF6CeU4juM0LxV3HJJOlJQ6R9Iekt5Ve7Ecx3GcZiVrx7Ej\nsETSYmAxsAbYFngT8F7gz8DsukroOI7jNBVZzvFvSvo2cCgwDdgX6AUeBj5pZk/WX0THcRynmcjj\n4+gHbgs/juM4zlaOlxxxHMdxCuGKw3EcxylEQxSHpDMlLZf0e0lXSdpW0mRJ94ZaWNdIGhPmbhOe\nrwzHJzVCZsdxHCcil+KQtLOkyyT9d3i+t6RTq7mgpC7gC0C3mb0VaAM+DnwNuNjM3gS8AMTnPxV4\nIYxfHOY5juM4DSLvjuNHwEKiulUAfwDOGMJ1RwMdkkYDY4FniCK3rgvHLwdmhsdHheeE44dJ0hCu\n7TiO4wyBvIpjJzO7FtgEYGYbgf7KLymPmfUAXweeJFIY64hyRNaG8wKsBuJ6x13AU4nrriPKLxmE\npNMkLZK0aM2aNdWI5jiO4+Qgr+J4RdKOgAFIOohoAS+MpPFEu4jJRDuY7YAjqjlXEjO7xMy6zax7\nwoQJQz2d4ziOk0JmHkfgLOBGYA9JdxNVyv1oldd8H7DKzNYASLqBKLmwU9LosKvYDegJ83uA3YHV\nwbQ1DvhLldd2HMdxhkguxWFm90t6LzAFELDCzPqqvOaTwEGSxhJloR8GLALuJFJGVwMnAT8L828M\nz38bjt9hWd2nHMdxnLqRS3FI+lTJ0AGSMLMril7QzO6VdB1RF8GNRI2hLiEq0361pAvD2GXhJZcB\n/ylpJfA8UQSW4ziO0yDymqrenni8LdEu4X6gsOIAMLPzgPNKhh8D3lFm7qvAx6q5juM4znAxf0kP\n8xau4Om1veza2cGs6VNGbE/zvKaqzyefS+okMik5jjOC2JoWv1oyf0kPc25YRm9fFGzas7aXOTcs\nAxiRf7+8O45SXiGKinIcZ4SwtS1+tVSS8xauGPi7xfT29TNv4YoR+bfL6+O4iRCKSxTCuzdwbb2E\nchxn+NmaFr9aK8mn1/YWGo9laNXdXd4dx9cTjzcCT5jZ6jrI4zhOg6hm8WtVaqEkkwv/KIn+MsGe\nu3Z2pL62lXd3eX0cv6y3II7jNJZdOzvoKaMk0ha/VmaoSrJ04S+nNDra25g1fUrZ17f67i5vkcOj\nJf1R0jpJL0p6SdKL9RbOcZzhY9b0KXS0tw0aq7T4tTJpytCASbMXsMecmzl3/rLU15db+AHaJAR0\ndXZw0dH7pCqBVt/d5TVV/SvwITN7uJ7COI7TOOJFrlXt7kWYNX3KoB1DKf1m/PieqDP2hTP32eJ4\n2gK/yYxVc2dkXr/Vd3d5FcefXGk4zshn5tSuEakoSilVkmmlKK6696myimOoC385xdVKu7u8imOR\npGuA+cBr8aCZ3VAXqRzHGRE0c+RQUklOmr2g7JxyvgsY+sLf6ru7vIrj9cB64PDEmAGuOBzHKUsr\nRQ61pURFtaW0/qnFwt/Ku7u8UVWn1FsQx3FGFq0UOXT8gbsP+DRKx9Oo98LfzLu1vFFVb5Z0u6Tf\nh+f7Sjq3vqI5jtPKtFLk0IUz9+HEgyYO7DDaJE48aGJZ/8ZwEO/WeoL/Jd6tzV/Sk/na4SCvqepS\nYBbw/wDM7EFJ/wVcWC/BHMdpbWoROTScd90XztynYYqilGbfreXtADjWzH5XMrax7EzHcRyGnhcy\n3Hfd85f0MG3uHUyevYBpc+9o6N19s+/W8iqOP0vag82tYz9K1C/ccRynLDOndnHR0fvQ1dmRKymu\nlEp33bWm2UxDabuyZsnzyGuqOp2o2dJeknqAVcAJdZPKcZwRwVAcyMN5191spqFmz/PIqzieMLP3\nSdoOGGVmL9VTKMdxnOHMrm4201Cz53nkVRyrJN0CXAPcUUd5HMdxgOG9627GEiDNnOeR18exF/AL\nIpPVKknflvSu+onlOM7WzlB9JEXYmgo81gJZSkp96guk8cA3gRPMrC1rfiPo7u62RYsWNVoMx3Ga\nmNJQ30P2msCdj6xpStPQcCFpsZl1Z83L3TpW0nuB44AjgEXAsUMQrhP4PvBWokitTwMriExhk4DH\ngWPN7AVJIlJURxKVPTnZzO6v9tqO4zQPjcqOLlcO5frFPXXb0Yw08maOPw6cAfwa2MfMjjWz64dw\n3W8Ct5jZXsB+wMPAbOB2M9sTuD08B/gAsGf4OQ343hCu6zhOk9DIENh6hPo2Ux5Ivcm749jXzGrS\nuEnSOOA9wMkAZrYB2CDpKODgMO1y4C7gHOAo4AqLbGr3SOqUtIuZeR6J47QwaYv32dc+wJnXLK3r\nDqScI7zSeBatVNCxFuR1jv9VDWtVTQbWAD+UtETS90OY784JZfAssHN43AU8lXj96jA2CEmnSVok\nadGaNWuqFM1xnOEiLdS136zuO5C0qrdp41lUs4Np5R1KXsVxKTAH6IOoVhXw8SqvORo4APiemU0F\nXmGzWYpwfoPU3iplMbNLzKzbzLonTJhQpWiO4wwXeUJd65UpntZnI208i6J5IM2WqV6URtSqWg2s\nNrN7w/PriBTJnyTtAhB+PxeO9wDJ2sa7hTHHcVqYciGw5UguvrW6S+9KUVpp41kULREynOVU6sGw\n16oys2eBpyTFAdKHAQ8BNwInhbGTgJ+FxzcCn1LEQcA69284TutTmqeRZiaKF9/5S3qY9ZMHBt2l\nz/rJA1Upj1rnbRQ9X7NlqhelUbWqPg9cKWkM8BhwCpESu1bSqcATbA73vZkoFHclUTiuN5VynBFC\nMjv63PnLyjZTOmSvyPR8/o3L6ds02JTUt8k4/8blhR3QtS7pUfR8zZipXoRMxSFpFNBdy1pVZrYU\nKJdkcliZuUakuBzHGcHc+Uj5oJZ4fG1vX9njaeNZ1LqkR5HzNXsRwywyFYeZbZL0ZeBaM3tlGGRy\nHGcrpNnMN/VMTmz2IoZZ5DVV/ULSl4gyuweUh5k9XxepHMfZ6sgy34wf284L67fcXYwf215zWYYj\nL6N0hxI7/ltBkeR1jh9HZC76FbA4/HgxKMdxakbsy0gbP+9Db6G9bbADvb1NnPeht9RcluGOemq1\n8NxcOw4zm1xvQRzHaX2GYt7J8nEMp3lnuM1mzdZIKovcRQ4dx3EqMVTzTp4yIEXNO9Uqsmqinoai\nNJvNv5NFXlOV4zhORYZq3ilaBiQrr2Mo5p+ieRlDNTU1e4/xUlxxOI5TE4Z611y0DEilvA4YmiIr\n10TqmLd1MW/hirJZ63muVSnrvdUaSeUyVUmaBiw1s1cknUhUIuSbZvZEXaVzHCeTRvW0KGWoSW1d\nKa9PKwOSldcxVEWWNItlmeGyrpX1+lYLz8274/gesF7SfsDZwKPAFXWTynGcXDRTNM5Q75rzvD55\n155FLc0/WTuKrGvl2ZHMnNrF3bMPZdXcGdw9+9CmVRqQX3FsDBncRwHfNrPvAK+rn1iO4+ShmYrl\nVdMjPKkI5i1cwTFv60p9famSTCPO66il+SdrR5F1rVZzfmeRN6rqJUlzgE8C7w5lSGqfdeM4TiGa\nbUEqUnajaPvWckqylLZRm/M6amn+GdfRXtY0Nq6jPde1Wr02VSl5FcdxwCeAT5vZs5ImAvPqJ5bj\nOHlo5QWpaAfAPMqw1IRSq3pUaf2d8vZ9avXaVKXkMlWFUujXA9uEoT8DP62XUI7j5KPVonGSFO0A\nmEcZ9m2yqs10laKe1pYpdZIcz/I1VWPGa2byRlV9BjgN2AHYg6h16/+lTDVbx3GGj+GOxqllBFdn\nSu2pJMns6XJ37eWoxkyXFfWUtbPLk/ld62q8jaRIP453APcCmNkfJb2hblI5jpOb4VqQal34L2+X\n1lgRlCrJUVLZHI9aR02lKa2R7PzOIq/ieM3MNigY9CSNpmBPcMdxhp9z5y/jqnufot+MNonjD9yd\nC2fuU9W5al1PaV3OPhpJRVCk8VMRshb+rc35nUVexfFLSf8IdEh6P/APwE31E8txnKFSurD2mw08\nr0Z5VHNXXcm0lbbYJqnkr8kqiliENLNZZ6Jke6Wd3UhzfmeRN49jNrAGWAZ8lqid67n1EspxnKFz\n1b1PFRrPomhCXZbDuJxjv32UGD+2PVeZj1qah15N8Zu8sL6v7LVLGWnO7yzyllXfBFwafhzHaVKS\nd/hptuS02k9ZFL2rzjJtlTP/HLLXBO58ZA1r1/fxymsbuea+p+jrj+Qt6rAuQm/fptRjSaUXX7sc\nI8n5nYWswj+RpGVU8GWY2b71EGqodHd326JF3mfK2boodV5nUY3Po0hU1eTZC1IXD8EWr88rf5vE\nJjM6x7azbn0fySW/fZSY97H9ci3gSf9PXro6O7h79qG559eTetQok7TYzLqz5mXtOD44JCkqIKmN\nqItgj5l9UNJk4GpgR6IOg58MDvltiOpivQ34C3CcmT1eL7kcp1XJk1mdpJzPI8uZnnVXnVzM0qKe\nYPNd/FnXLuWCm5azdn1fxfmlcgPlQ3krJORVoyhKaZYoqeFobVuJij4OM3ui0s8Qr/1F4OHE868B\nF5vZm4AXgFPD+KnAC2H84jDPcZxAnLiW5WhOI/Z5xM70eGGNFcu585fllmPWdZv7Y+RZoDdZpADy\nzs+ir798AmDpe6uWUVKqz6NSAmGtaXSNslzOcUkHSbpP0suSNkjql/RitReVtBswA/h+eC7gUOC6\nMOVyYGZ4fFR4Tjh+mJQ30d9xRjZJB3S1xIvpUJ3pF9y0fMAfkWSUKm4Eak65XUGRgAARFUpsH7Wl\n1GlZ7cNdpbjReSN5o6q+DRwP/BHoAP4X8J0hXPffgS/DgHlyR2CtmW0Mz1cTZacTfj8FEI6vC/MH\nIek0SYskLVqzpng4nuO0IkXNU+WIl8eijZRKScsC32Swau6MakRjFAxEWaV1Aiwl6RyPdwF530Ob\nxKq5M1jaRKIhAAAdP0lEQVTylcOZ97H9BqKkyl07eYc/3DuARncMzN0B0MxWAm1m1m9mPwSOqOaC\nkj4IPGdmi6t5fQX5LjGzbjPrnjCheAKQ47QitbjDHDsmCokdaiG/SkyevaCqXUdbW1TtdtXcGXzj\n2P22CN8tpX2UBqK8qtmNjRm92RQFDPTH2JSieOK//3DvABpdoyxvAuB6SWOApZL+FXiG6tvOTgM+\nLOlIYFvg9cA3gU5Jo8OuYjcg3uP1ALsDq0PG+jgiJ7njbPWkhaTGXfPyLJrrN0R3yh2jR7G+TFhq\nx+jNX/VKkTxSehmRaj0Lsc+iXPjuuI52XnptI/3J9rEJ7VRkNzZKkZBxWG6pszkrQXC4M8cb3TEw\nr+L4JJGi+BxwJtFCfkw1FzSzOcAcAEkHA18ysxMk/QT4KFFk1UnAz8JLbgzPfxuO32GVYogdp4Si\nYYvN0oo1D1m5FXnCW+PFLS2XIR7PiuTJ+62Mw2nHdbTz4qt9bMp4XfKuPalAps29Y4seGX39NlCW\nvcgiUU6GZM5JqkIM443IHG9k3kjeBMA4gupVSd8Cdg+mq1pyDnC1pAuBJcBlYfwy4D8lrQSeBz5e\n4+s6I5iiYYuNDnPMQ6liO+ZtXdz5yJpURZe8Q39lw8ZBDuz2NvHKaxuZPHtBajhsHElU7nhycU3r\nGV7KJjNWzZ1RduEvR9pde6Wy7LUifj9Z/c2r2QG00g1KKXnLqt8FfDjMXww8J+luMztrKBc3s7uA\nu8Ljx4gq8JbOeRX42FCu42y9FC3MV+tCfrWmaNe80rvS5GLVObadl1/dOLD4ZTnH044n26fO+skD\n9GVsIWJFkNf+nyxamDdPpFbETvG2lGslnebl/tbT5t5RVjG0wg1KJfL6KcaZ2YvA0cAVZnYg8L76\nieU4taGo07LRYY5ZDDV6Z+bUrgGH79gxo8su8m1SoSimuH0qQHrhjoik+Sav/T8uWlga8lpvpQHZ\nSjNtPCs8t9F5GEMlr+IYLWkX4Fjg53WUx3FqStGwxeEIczx3/jL2mHMzk2YvYI85N+dOsINsxVYk\nCS3tXLEpKe/CHOuXC25aPthRHYjzOEoL/+Utfx7LmebsjhVdPegMSrEr5fNPG89SDM1+g5JFXsXx\nVWAhsNLM7pP0RqKcDsdpaoqGLdY7zHGo2dmVFFvRJLSO9vJf/3g8744jjjbKyuO4e/ahg8wwCx58\nJtf5s0xbsaKrB/GfIE3JpY1nKYZG52EMlbw9x39iZvua2T+E54+ZWVVRVY4znBQtd12P8tjJXUC5\nxkOQP7O5kmIrav4oF3obj0+evaBQ0lw1ZLWNhXymrXg87e5/KMQyFu39kSZrHGiwfsPGLTLTW6l/\nR95wXMdpWYqGLdYyzDFvxde8i3Sl6J0zr1la9jU9a3uZPHtBocidIt6DevgaylXPzQp5nTV9CrOu\ne6Bs2ZNqiZViUdNSWn/0ZIHG9jbR2dHOut6+kRlV5ThOdeRNQity156m2Cp11ItNV7Oue4Dzb1ye\nu21rHuI7/VEqnw8BDCiuuN9Gli2/nOkpV8hrgVySIpV4iyb45emP3tdvbLfNaJaed3g+oZuIiv04\nWhXvx+E0C5V6UiQ58aCJVfcCj4mr09byjjuL9lFi+21HszZUuK0V5XYc5Sganhv368ibIBj33yi3\nc+xob8ttxkz7PxDV1/GqB3n7ceStjvtFSa9XxGWS7pfUemrScYaZLGdnm1QTpTHAMOiMthAl1dnR\nTr/ZQFn0WpLHuV8kPDf2V8VNnvI4odvbNte9Gqrvq9Wd4aXkNVV92sy+KWk6MJ6oBMl/ArfWTTLH\nGQGk2eXr0Y963sIVmcl3tSEq7LSut/YKo5SiyZqVWL9hI+ffuJwzr1nKuI522ttUeXdWcigrwS9p\nhivqn2k18iqO2AB7JPCfZrbce2I4TkSl0hHDWYxuuHIA8vgGQs3AXHR2tLP0vMNTzTm1yHkwBkdx\nre3to32UGD+2PbX7YN8mG6S0ymXdx4q6Z23voIi50kzwRhclrDV5FcdiSbcCk4E5kl5HdpKo4zSE\n4awBlKd0RNEorRMu/S13P/r8wPNpe+zAlZ9556Brlnt/4zray9ZUUljFSxe7erJq7gz2v+DWXLWo\n1r3aV7FWViUzz1AaWPVtMsaOGc2Sr0RKqxzJxMrk55wnlLiZStXUmlzOcUmjgP2Bx8xsraQdgN3M\n7MF6C1gN7hzfehmqE7MoaS1bY6dqUUqVxsB1gvKo9P4uuGl52QVt/Nh2lnwlckkmlU691Edsikgr\nRV6E2Jld7rOLkymHiiBVacWVfKutixU7v4f7/7JaauocB94JrAhK40TgXKJOfI7TVAx3DaBal44o\npzSS45XeX9oinRxP1qqqF8aWpqFqqWTW+PkD+TLPs6jkWI9bxVabqxLvllq9NlUpeRXH94iaOe0H\nnA08ClxRN6kcp0qGuwbQcEfLtHqNo6L0bzIuuGl52WN5zGBFietq1cKB2z5KrN8QlaxPM6m16ueW\nV3FsDM2TjgK+bWbfAV5XP7EcpzqGeyGfNX3KFl+iUWG8HoyUsM4iC/ML6/uqKghZDbH7J8/+Is78\njsNzTzxo4kC4bmdHO4jMUOVW+9xi8jrHX5I0hygM993B59Ge8RrHGXaGO+xx0RPPb2FO2QTMueFB\nzrxmaVnnfCXnffsoKFdCKq5HWOn9nX3tA5k9I5qFVXNnMCnFIZ1GXBDyynuerGsYcNa58yQn5mlS\n1crhuHl3HMcBrxHlczxL1BN8Xt2kcpwqqSZRq0gp8lLSihP29m0qm8SW2afhY/uXPV88Xu79HfO2\nLuYtXJFqhz/+wN1zv5+i7PmG7QZkKUJRpZGkkbUuRuV8o5VMULUqntlIcpcckbQz8Pbw9Hdm9lzd\npBoiHlXl5KVctEt7m9huzOhcxefyLoBxlFWeKKwi4cR5yowkM9OT5x6dsrspSnz+oSiDVqRSVFSt\no+2Gi7xRVXlbxx5LtMO4i0hh/oekWWZ23ZCkdJwGUy7apa/fBswMWS098xbLi+9A0+5ESyvYVlpc\nzp2/jKvufSp3pM+V9zzJhTP32UJJ1kJpAAPmoyJJf61CpXDc0jyNpFIul5neyqapUvL6OP4JeHu8\ny5A0AfgF4IrDaWnyRLVUSuQ6/sDdc+USxE7QPBVsSxVVckHqaB+V2kcjjXjpKlqio5prjCSSO4qi\nCYKlmemtnileSl4fx6gS09RfCrx2EJJ2l3SnpIckLZf0xTC+g6TbJP0x/B4fxiXpW5JWSnpQ0gHV\nXNdxypE3qiVNwVw4cx9OPGjigAN6lLb8YpT2jCht4FNKMr4/NkXFPpGiSiPJULKsRxp5AgYOmDhu\nYKHPasx09rUPbLlz3WS82Ltx6MI2IXkX/1skLZR0sqSTgQXAzVVecyNwtpntDRwEnC5pb2A2cLuZ\n7QncHp4DfADYM/ycRpRT4jg1oVxHvXJUUjAXztyHRy86ksfnzuCxi2bwb8ftX9E5n2fpj01XZ127\ndMhl0rcZXawV7NZAHjPf3Y8+PxD+m/Z/kpUgGB/PqvQLQwvSGG5ymarMbJako4F3haFLzOyn1VzQ\nzJ4BngmPX5L0MNBFlCNycJh2OZE/5ZwwfkXII7lHUqekXcJ5nK2QWtaiKi0+V66eU1HbdKXaVBfc\ntJz+nLWiDKhFu5x4watHp76RTuy/2bWzgwMmjuOex16o+u9YyeSZp+ZZM5GpOCS1Ab8ws0OAG2p5\ncUmTgKnAvcDOCWXwLLBzeNwFJGMeV4exQYpD0mlEOxImTpxYSzGdKqhXocF6f8HGjhnNjH13SS2P\nHctQ7XurRRmOoqzt7dvqIp5qSbxjqIWpL83kWakkSUsqDjPrl7RJ0jgzq1l9KknbA9cDZ5jZi8kq\n7WZmkgqpdTO7BLgEonDcWsnpFKcWi3va4lzrL1g5Wa9f3JMaZlnNe0u+F2frJs3k2WqlZPJGVb0M\nLJN0G/BKPGhmX6jmopLaiZTGlWYW72L+FJugJO0CxM74HiCZwbRbGHOalKEu7pUW52q+YJV2CEVl\nLTp//pIeZv3kgWFqsOQ0O4fsNaHseFolYYlBYdrNsvvIqzhuoEZmqtAA6jLgYTP7t8ShG4GTgLnh\n988S45+TdDVwILDO/RvNzVDvntIW57OvfSA17DPtTi5rh1BU1qLj59+43JWGM8Cdj6wpO57mNon/\ndZrN55HXOX55Da85jajm1TJJS8PYPxIpjGslnQo8ARwbjt1M1HlwJbAeOKWGsjh1IC1XIW/oa5ot\nOc0pmaxCWnRHUVTWtPlGlEXeJnH8gbsPZGrXo4Kr07qk3WCsy/F/0kw+j4rhuJKOknR64vm9kh4L\nPx+t5oJm9hszk5nta2b7h5+bzewvZnaYme1pZu8zs+fDfDOz081sDzPbx8y8lkiTUy50Mbm4Z4Ua\n5q0HBFtWIS0Ne8zaIZTLqxgFqbLOmj6F9rZ0AeNCfPWu4uo0ljaJaXvsMBB2nTfUeajVjZvF55GV\nx/FlIlNRzDZE9aoOBv6+TjI5LU5pIb6sxb2UvJYdAdttM3qLPIdkAl2uL2rJd34Tg2Wd9ZMHmPrV\nW5k8ewEX3LQ8V15FXPxwuzHZOSJO69Fvxv1PrmPW9CmsmjuDbxy7X2Y+UKWw7lrkEw0nWYpjjJkl\nQ2F/E3YGTwLb1VEup8VJdprLWtyrZdfOjlw7itIvZPILPG/hikxF0LfJBhRJ3nDa2KzW3lZVgQWn\nBYj9bpNnL2DewhUcMHHcwM6jdEeSVQ233M1W6c62mWpdZfk4xiefmNnnEk/Lhwc4TglFHcqdHe25\nexnMW7iioo9i5tQuFj3x/EBRwDaJY962OUGv3lt/93GMbOIbhNI8j3hHUqR0emniaL1yoWpBluK4\nV9JnzOzS5KCkzwK/q59YTquT/KcvV1kU0rfd53/4LVuEsI4SvH7bdtb19jGuox0JzrxmaWYV0vlL\nerjmvs2VZPvNuOa+p+j+6x0qOscdZ6gM1ZldqQJBo8naR58JnBKKEn4j/NwFnAycUW/hnNaktFlR\nOaVRads9c2oXx71j90Hb/k8cOJGl5x3Oxcftz2sbNw2Yjtb29oHB+LHtWzQ2Sqv11Ne/uY91Xtuy\n41RDsziza03FHUeoiPu3kg4F3hKGF5jZHXWXzGlZ0sp3x70N8jQnun5xz6BdwvWLe+j+6x3K98/Y\nZIwdM5olXzl8i7yNtPj42FdRasqqFZ0d3lnZaR5ndq3Jm8dxB+DKwslF2l3WJjNWzZ2R+fpKCYBp\ni3tcTTbNLJZGqZKqFW/Z9XU1PZ/TejSTM7vW5M0cd7ZyijjqhpoAmKZ4shb3SuWty1GNosnL/zz6\nfM3P6TQnIvrfPmSvCXUrjtlsuOJIMJI+2FpStLDfrOlTtuiD3d6m3Hdfw+WwLqpoip7b2TrIs4tu\ntbLpWXiQeaDUoZun8crWQqWyHamUrpwFVtKR4rD2UuZOTFXfoSbGFUdgpH2wtaRoHsa8hSu2KOzX\nt8ly/y1Lk6Gq7Vzn/e6cejNtjx0GHlfq4NdqZdOzcFNVoNU+2Hqb1YaSh1GLv+WiJ57n2XWvRl3w\nqjT8rJo7g2lz7/A8DacuTNtjB678zDuBbFPUUP1+zYYrjkArfbD1tpfOX9LDGdcsHXheTmmMAp5Z\n1ztQEfagN47n8b/0VlQO4xIhqpUU37nzl/Hje54cmDuUquSTdvQEP6e2PF7Gp5FVhXnW9CmDvrMQ\n+f1eea18VedmxxVHoNwHW+9wump3DfVuMznrJ0tTjwnYtn0UvX2bBvwW/WbcnSOKKLY4ZSm+uEDg\nUJk8e4E7qZ2aU26hz9plp/W2j0vS1OLmbziDe2QjsIF9d3e3LVpUvPr6cP7hSxdPiBRVsrZNmjyV\nnK5xaOBQZK90/sfnzmCPOTdXHY0kSDV9xQmCI+8/0hmJJL+vaSbRtKTXtPldnR3cPfvQwrLkWU/y\nIGmxmXVnzfMdR4LhrA1TKcktrsH0yoaNAyGtyTuStgq5B0Z9Q/2GehdfKQS2XqGxjlMPkrv8Q/aa\nMMi8GpMsgjiUzpNZ1NsKUYpHVTWISklucQ2mtFLkeRbY0oiwShEfRfCl3XE2E3+P01rCJincJ6YK\nOfKODxXfcQyBoqatPJFKWTy9tpeunAlycRmO2J4ah8jGzYkuuGk5a9f3bbmN3mOHXD6LLEYpqhXl\nysYZqcQLfd4FOtknppY+1eEO7vEdR5UUTRjMUzE2D/EiX9ruNI24+VC5vIpkl7szrlk60OXu8b/0\nsucbht6n6xMHTmTV3Bl0tPu/mTPySLYYHlWwdWxprlJWo6csshqW1RrfcVRJlo9i184Oxo4ZxR+f\ne6XieWLnWZ4dSPIfoR7+gLhibM/aXjra2/j34/Zn5tSuqp3h1y9ezYUz94kisBxnhBG3GIb838dD\n9trc/66WPtXSqK16B/e0jOKQdATwTaAN+L6ZzW2kPFmF+PLmDvSbDYS4vrJhy1Lk241pY/2G/kH/\nCPtfcOuQchvy0NvXzxnXLB2Uz1H8HK4wHCdJHl9ItQxncE9LKA5JbcB3gPcDq4H7JN1oZg81SqZa\nFuIzKKs0ADrHjmH5VweH53k7UsdpTUZKMmqrGJ/fAaw0s8fMbANwNXBUIwUarkJ8zVryxHGc4lRb\nd63ZaBXF0QUk04lXh7EBJJ0maZGkRWvW1G87GDOUQnxF/nXKRUWMH+vd5RynFRkpuUqtojgyMbNL\nzKzbzLonTJiQ/YIaMHNqF3fPPpRVc2fwjWP3y7UDmbbHDqyaO4OuHGFyaVER533oLbS31fbORWzu\n210r4vfois5xIvJ871uBVlEcPcDuiee7hbGmoVx4XWlIa7KaZjlTV3ub6OxozwzPmzm1i3kf3W/g\nWtuNyVZY7aM0oBi6Ojs48aCJg2S9+Lj9WfKVw6OKsolS0dWSVHp5Fd0oIgXm4bvOcFFLy9EoMfD9\n7exo3+J/fiS1km2JWlWSRgN/AA4jUhj3AZ8ws+Xl5ldbq2q4qWVtrHPnL+Oqe5+i32yLarXVnPuE\nS387KAlwzzdsx/oNmwbOV9oms2jbzI39/fzppQ2Dzn/bWQfnej9jQ6SZEZkI3zhhLI+tWT8wt/T5\nTtu3D7rWtm3i1URWfunz0veaJ6zaaU6StdFGCbYZPYpX+zaV/R/N+g6V/h+U/p+MhFaxeWtVtYTi\nAJB0JPDvROG4PzCzf0mb2yqKw3Ecp5kYcUUOzexm4OZGy+E4jrO148Zkx3EcpxCuOBzHcZxCuOJw\nHMdxCuGKw3EcxylEy0RVFUHSGuCJBoqwE/DnBl6/Es0sGzS3fM0sGzS3fC5b9QynfH9tZpkZ1CNS\ncTQaSYvyhLQ1gmaWDZpbvmaWDZpbPpeteppRPjdVOY7jOIVwxeE4juMUwhVHfbik0QJUoJllg+aW\nr5llg+aWz2WrnqaTz30cjuM4TiF8x+E4juMUwhWH4ziOUwhXHDVC0u6S7pT0kKTlkr7YaJmSSNpW\n0u8kPRDku6DRMpUiqU3SEkk/b7QspUh6XNIySUslNVXpZUmdkq6T9IikhyW9s9EyxUiaEv5m8c+L\nks5otFwxks4M34ffS7pK0raNlilG0heDXMub6W8G7uOoGZJ2AXYxs/slvQ5YDMw0s4caLBoAkgRs\nZ2YvS2oHfgN80czuabBoA0g6C+gGXm9mH2y0PEkkPQ50m1nTJYpJuhz4tZl9X9IYYKyZrW20XKVI\naiPqp3OgmTUyQTeWp4voe7C3mfVKuha42cx+1FjJQNJbgauBdwAbgFuAvzOzlQ0VLOA7jhphZs+Y\n2f3h8UvAw5T0RW8kFvFyeNoefprmrkHSbsAM4PuNlqWVkDQOeA9wGYCZbWhGpRE4DHi0GZRGgtFA\nR2gWNxZ4usHyxPwNcK+ZrTezjcAvgaMbLNMArjjqgKRJwFTg3sZKMphgCloKPAfcZmbNJN+/A18G\nNjVakBQMuFXSYkmnNVqYBJOBNcAPg5nv+5K2y3pRg/g4cFWjhYgxsx7g68CTwDPAOjO7tbFSDfB7\n4N2SdpQ0FjiSwe2zG4orjhojaXvgeuAMM3ux0fIkMbN+M9ufqGf7O8J2uOFI+iDwnJktbrQsFXiX\nmR0AfAA4XdJ7Gi1QYDRwAPA9M5sKvALMbqxIWxJMaB8GftJoWWIkjQeOIlK+uwLbSTqxsVJFmNnD\nwNeAW4nMVEuB/oYKlcAVRw0JvoPrgSvN7IZGy5NGMGXcCRzRaFkC04APBz/C1cChkn7cWJEGE+5O\nMbPngJ8S2Z6bgdXA6sTu8ToiRdJsfAC438z+1GhBErwPWGVma8ysD7gB+NsGyzSAmV1mZm8zs/cA\nLwB/aLRMMa44akRwPl8GPGxm/9ZoeUqRNEFSZ3jcAbwfeKSxUkWY2Rwz283MJhGZM+4ws6a48wOQ\ntF0IeCCYgQ4nMiU0HDN7FnhK0pQwdBjQFAEZJRxPE5mpAk8CB0kaG76/hxH5JpsCSW8IvycS+Tf+\nq7ESbaZleo63ANOATwLLgh8B4B9Dr/RmYBfg8hDZMgq41syaLuy1SdkZ+Gm0tjAa+C8zu6WxIg3i\n88CVwRz0GHBKg+UZRFC27wc+22hZkpjZvZKuA+4HNgJLaK7yHtdL2hHoA05vpqAHD8d1HMdxCuGm\nKsdxHKcQrjgcx3GcQrjicBzHcQrhisNxHMcphCsOx3EcpxCuOJyWQdLL4fckSSbpwsSxnST1Sfp2\neH6+pJ5QkfWPkm6QtHdi/l2SVoTjD6eVEUnMe0DSfZL2zyHnGaFMRPz85jiHZqhI2iVZPVjSHEkr\ng4zTqzjfLZLWllYkljRZ0r3h3NeEUF8kfU7Sp4f+TpxWxhWH06qsIiqKGPMxYHnJnIvNbH8z2xO4\nBrhD0oTE8RNCCZZpwNfixbEMJ5jZfsB3gXk5ZDuDqGAeAGZ2ZA1j8M8CLgUIivDjwFuIqgB8N+Tp\nFGEeUf5RKV8j+vu9iShr+dQw/gOivBFnK8YVh9OqrAceltQdnh8HXJs22cyuIar784kyh7cnqvGU\nVQvotyQqHkv6nqRFSvQ3kfQForpHd0q6M4w9HnZEk8Lu5tLwmltDFj+S3i7pwbADmicpLTP9GKLa\nRRDVWbrazF4zs1XASgqWQjGz24GXkmMhi/pQovIlAJcDM8P89cDjkpql5IrTAFxxOK3M1cDHJe1O\ntOhnlcS+H9gr8fxKSQ8CK4B/NrMsxXEEMD/x/J/MrBvYF3ivpH3N7FtBjkPM7JAy59gT+I6ZvQVY\nS6QIAH4IfDbsgMrKIWky8IKZvRaGuoCnElNWU5tS/jsCa0M573LnXQS8uwbXcVoULznitDK3AP8M\n/InIFJWFSp6fYGaLgvnqfyTdktIrIi7nsT2Q9HEcG3wjo4lKuuwNPJghwyozi0vSLAYmBf/H68zs\nt2H8v4Byjax2ISqh3mieY7ACdrYyfMfhtCxmtoFo8T2bzWaVSkylTBE7M1tDtBs5MOV1JwBvJDLZ\n/AcM3P1/CTjMzPYFFgB52o6+lnjcT7Gbt96Sa/QwuEfDbmFsAEkf0ea2rd3k4y9AZ2huVO682wZZ\nnK0UVxxOq/MN4Bwze77SJEnHEFW13aJCa4iAmgo8mvZ6i4q6/W+iaqp7Aa8n8ousk7QzUdnwmJeA\n1+V9A8Fx/pKkWHF9PGXqH4BJiec3EpnqtgmKbE/gdyXn/mkIENjfzHL1Sg/v9U7go2HoJOBniSlv\npkmqAzuNwU1VTktjZsvZMpoq5szQmGc7ooXu0LC7iLlSUi+wDfCjrEZSoS/1N4BZZnaqpCVEpemf\nAu5OTL0EuEXS0yl+jnKcClwqaRNRm9B1Za7/iqRHJb3JzFaa2XJFfbIfIqruenoOP80gJP2ayOy0\nvaTVwKlmthA4B7g6hDwvIbSmDUwDzi9yHWdk4dVxHacJkLR93BNe0mxgFzP7Ypl5HwHeZmbnDreM\n4fpTgbPMrFwIr7OV4DsOx2kOZkiaQ/SdfAI4udwkM/tp6NHQKHYiMtk5WzG+43Acx3EK4c5xx3Ec\npxCuOBzHcZxCuOJwHMdxCuGKw3EcxymEKw7HcRynEP8ferfB1bePcX8AAAAASUVORK5CYII=\n", 238 | "text/plain": [ 239 | "" 240 | ] 241 | }, 242 | "metadata": {}, 243 | "output_type": "display_data" 244 | } 245 | ], 246 | "source": [ 247 | "# Visualize data\n", 248 | "plt.scatter(movie_data['imdb_score'], movie_data['gross'])\n", 249 | "\n", 250 | "# Chart title\n", 251 | "plt.title('IMDB Rating and Gross Sales')\n", 252 | "\n", 253 | "# y label\n", 254 | "plt.ylabel('Gross sales revenue ($ millions)')\n", 255 | "\n", 256 | "# x label\n", 257 | "plt.xlabel('IMDB Rating (0 - 10)')\n", 258 | "\n", 259 | "plt.show()\n", 260 | "\n" 261 | ] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": {}, 266 | "source": [ 267 | "## Linear regression model\n", 268 | "We want to fit a \"line of best fit\" to the above scatter plot. In particular, we want to model the relationship between IMDB rating and gross movie sales as $$GrossSales_{i} = \\beta_{0} + \\beta_{1}IMDBRating_{i} + \\epsilon_{i},$$ where $\\epsilon_{i}$ is a random error term with mean 0. We would like to estimate values for $\\beta = (\\beta_1, \\beta_2)$ that \"best fits\" the data. To define fitness, we will use the square loss function. That is the sum of square error is $$SSE = \\frac{1}{2}\\sum_{i=1}^{n} (GrossSales_{i} - \\hat{GrossSales_{i}})^2,$$ where $\\hat{GrossSales_{i}}$ is the predicted gross revenue of movie $i$ from our linear model. More precisely, $$\\hat{GrossSales_{i}} = \\hat{\\beta_{0}} + \\hat{\\beta_{1}}IMDBRating_{i}.$$ The basic idea here is that we want to learn the weights (or coeffecients) $\\hat{\\beta} = (\\hat{\\beta_1}, \\hat{\\beta_2})$ so that SSE is small as possible and that will be resulting \"line of best fit\"." 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "metadata": {}, 274 | "source": [ 275 | "# Newton's method for linear regression model" 276 | ] 277 | }, 278 | { 279 | "cell_type": "markdown", 280 | "metadata": { 281 | "collapsed": true 282 | }, 283 | "source": [ 284 | "We use Newton's method as our \"learning algorithm\" to learn $\\hat{\\beta} = (\\hat{\\beta_1}, \\hat{\\beta_2})$ so that we can use the parameters to fit a linear relationship to the data. Note that we can rewrite SSE as, $$SSE(\\beta_{1},\\beta_{2}) = \\frac{1}{2}\\sum_{i=1}^{n} (GrossSales_{i} - \\beta_{0} - \\beta_{1}IMDBRating_{i})^2. $$ Since we want to minimize $SSE(\\beta_1,\\beta_2)$, we set $\\frac{dSSE}{d\\beta_0} = \\frac{dSSE}{d\\beta_1} = 0.$ This is where Newton's method kicks in, now we have a function $\\frac{dSSE}{d\\beta}:\\mathbb{R}^2 \\to \\mathbb{R},$ and we are interested in finding $\\hat{\\beta} = (\\hat{\\beta_1}, \\hat{\\beta_2})$ such that $\\frac{dSSE(\\hat{\\beta_1}, \\hat{\\beta_2})}{d\\beta} = 0.$ To apply newton's method, we will need to use a matrix version of the newton's method shown above, that is our update rule will be \n", 285 | "\n", 286 | "$$\n", 287 | "\\begin{bmatrix}\n", 288 | " \\beta_{0}^{t+1} \\\\ \n", 289 | " \\beta_{1}^{t+1}\n", 290 | "\\end{bmatrix}\n", 291 | "=\n", 292 | "\\begin{bmatrix}\n", 293 | " \\beta_{0}^{t} \\\\ \n", 294 | " \\beta_{1}^{t}\n", 295 | "\\end{bmatrix}\n", 296 | "-\n", 297 | "\\begin{bmatrix}\n", 298 | " \\frac{d^2 SSE}{d\\beta_{0,t}^{2}} & \\frac{d^2 SSE}{d\\beta_{1,t}d\\beta_{0,t}} \\\\ \n", 299 | " \\frac{d^2 SSE}{d\\beta_{0,t}d\\beta_{1,t}} & \\frac{d^2 SSE}{d\\beta_{1,t}^{2}}\n", 300 | "\\end{bmatrix}^{-1}\n", 301 | "\\begin{bmatrix}\n", 302 | " \\frac{dSSE}{d\\beta_{0,t}} \\\\ \n", 303 | " \\frac{dSSE}{d\\beta_{1,t}}\n", 304 | "\\end{bmatrix},\n", 305 | "$$\n", 306 | "where $t$ is the newton's method itteration. Using power rule, we can derive that $$\\frac{dSSE}{d\\beta_{0}} = -\\sum_{i=1}^{n}(GrossSales_{i}-\\beta_{0}-\\beta_{1}IMDBRating_{i}), $$ and $$\\frac{dSSE}{d\\beta_{1}} = -\\sum_{i=1}^{n}(GrossSales_{i}-\\beta_{0}-\\beta_{1}IMDBRating_{i})IMDBRating_{i}.$$ Using power and chain rule we can also find $$\\frac{d^2 SSE}{d\\beta_{0}^{2}} = n, \\frac{d^2 SSE}{d\\beta_{0}\\beta_{1}} = \\sum_{i=1}^{n}IMDBRating_{i}$$ and $$\\frac{d^2 SSE}{d\\beta_{1}^{2}} = \\sum_{i=1}^{n}IMDBRating_{i}^{2}.$$\n" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": 13, 312 | "metadata": { 313 | "collapsed": true 314 | }, 315 | "outputs": [], 316 | "source": [ 317 | "# Data size, this is derivitive with respect to B0^2\n", 318 | "n = len(movie_data.index)\n", 319 | "\n", 320 | "# Sum(IMDB_score), this is derivitive with respect to B0B1 and B1B0\n", 321 | "sum_imdb = sum(movie_data['imdb_score'])\n", 322 | "\n", 323 | "# Sum(IMDB_score^2), this is derivitive with respect to B1^2\n", 324 | "sum_sq_imdb = sum(np.power(movie_data['imdb_score'],2))\n", 325 | "\n", 326 | "# Initialize hessian matrix as it doesn't depend on linear regression parameters\n", 327 | "H = [[n,sum_imdb],[sum_imdb,sum_sq_imdb]]\n", 328 | "\n", 329 | "# Define threshold for convergence, epsilon = 10^-4\n", 330 | "epsilon = 10 ** -4\n", 331 | "\n", 332 | "# Criterion to be updated in newton's method\n", 333 | "criterion = 10\n", 334 | "\n", 335 | "# Initialize parameters\n", 336 | "new_param = np.array((-50,10))" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": 14, 342 | "metadata": { 343 | "collapsed": true 344 | }, 345 | "outputs": [], 346 | "source": [ 347 | "# Define Newton's method update rule as a function\n", 348 | "def newton_update(old_param, hessian, jacobian):\n", 349 | " # Matrix version of newton's update\n", 350 | " new_param = np.array(np.subtract(old_param,np.dot(np.linalg.inv(hessian),jacobian)))\n", 351 | " \n", 352 | " return new_param" 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": 15, 358 | "metadata": {}, 359 | "outputs": [], 360 | "source": [ 361 | "# Gross revenue np vector\n", 362 | "gross_rev = np.array(movie_data['gross'])\n", 363 | "\n", 364 | "# IMDB rating np vector \n", 365 | "imdb_score = np.array(movie_data['imdb_score'])\n", 366 | "\n", 367 | "# Implement Newton's method to find line of best fit (Itterate until convergence)\n", 368 | "while criterion > epsilon:\n", 369 | " \n", 370 | " # Reset old parameter (itteration t)\n", 371 | " old_param = new_param\n", 372 | " \n", 373 | " # Compute gradient vector\n", 374 | " J_pos0 = -1*(np.nansum(gross_rev)-n*old_param[0]-old_param[1]*sum_imdb)\n", 375 | " J_pos1 = -1*(np.nansum(gross_rev*imdb_score)-old_param[0]*sum_imdb-old_param[1]*sum_sq_imdb)\n", 376 | " J = np.array([J_pos0,J_pos1])\n", 377 | " \n", 378 | " # Apply newton's update rule (itteration t+1)\n", 379 | " new_param = newton_update(old_param, H, J)\n", 380 | " \n", 381 | " # Compute criterion function using euclidean distance \n", 382 | " criterion = np.linalg.norm(new_param-old_param)\n" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": 16, 388 | "metadata": {}, 389 | "outputs": [ 390 | { 391 | "name": "stdout", 392 | "output_type": "stream", 393 | "text": [ 394 | "[-113.36228051 30.2444501 ]\n" 395 | ] 396 | } 397 | ], 398 | "source": [ 399 | "# Let's see the estimated (Beta0, Beta1) parameters from newton's algorithm\n", 400 | "print new_param" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": 18, 406 | "metadata": {}, 407 | "outputs": [ 408 | { 409 | "data": { 410 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcHHWd//HXeyYTmHBkAgQMAzGAkQhyRCPgDw8OIdyJ\ngICKIqK4K8ppIBFWUFGD8cIV2eVQYWW5Q0BhOeTwQIkkhBACRMKZDEeCMBGSgUxmPr8/qnpS09PV\nXdV3z3yej8ck3dXVVd/unqlPf6/PV2aGc845l1RTrQvgnHOusXjgcM45l4oHDuecc6l44HDOOZeK\nBw7nnHOpeOBwzjmXigcON+RI+qikJbUuRyGSviDpL7UuRzVJMknvqXU5XH4eOFxOkp6X9Inw9hfC\nP+ifZu0zJdz+m/D+uPD+W+HPq5J+L+mAHMfuCvd5Q9LtkrbNU5YHJL0d7v+apNmSxqR4Lf0uRmb2\nZzPbMenz65Wk4ZK+JWmJpNWSOiT9n6QDa1yuKZIelfSv8PO6T9J2tSyTKy8PHC6pZ4BjJA2LbDsB\n+EeOfdvMbGNgN+Ae4BZJX8ja5/BwnzHAq8B/Fjj/18L93wNsDPwo/UsYdG4CpgCfB0YB2wEXA4fm\n2jnrs6uIMEBfDZwFjAzLdAnQU+lzu+rxwOGSegVYBEwGkLQZ8P+A2+KeYGavmNnFwAXARZIG/L6Z\n2dsEF8CdkhTCzDqBOcDumW2S9pD0N0mdkl6W9AtJw8PH/hTutjCssRwraR9JyyPPf17SNyQ9JmmV\npOslbRh5/OzwuC9J+lK+5hRJJ0p6UtKbkp6V9JXIY/tIWi7pLEkrwmOeGHl8c0m3hd/U/w7sEPc+\nhLXBA4ApZjbXzNaGP3ea2WlZr+0cSY8BqyUNk/S+sBbXKWmxpCMi+x8i6Ymw/B2SvhFu3yKsPXZK\nel3Sn3N9nuHn8pyZ3WuBN83sZjN7sdBnleM1biDpR5JeDGuv/yWpNWV5XAX4G+3SuJrg2y3AccCt\nwDsJnjcb2BIY0DwkaQRwLPBQkgJI2hw4Elga2dwDnAFsAXwY2B/4KoCZfSzcZzcz29jMro859DHA\nQQTfkHcFvhCe7yDgTOATBLWdfQoUcQVwGLApcCLwU0kfiDz+LoJv4u3AScAlkkaFj10CvE1QC/ti\n+BPnE8BcM1ueZ5+MTxPUQtoAAb8D7ib4TL4OXCMp89lcCXzFzDYB3g/cF24/C1gOjAa2Ar4J5MpX\n9AgwQdJPJe0raeOsx2M/qxxmAu8lCEbvIXjPvpWyPK4CPHC4NG4B9pE0kiCAXJ3weS+F/28W2TZH\nUiewiuCb86wCx/i5pFXAawQXna9nHjCz+Wb2kJmtM7Pngf8GPp6wbH3HN7OXzOx1ggtrpkZzDPBr\nM1tsZmsIak+xzOx2M3sm/Lb9R4IL9Ecju3QD3zGzbjO7A3gL2FFSM3AU8C0zW21mjwNX5TnVFgS1\nQCCoAYbfvldJejvHa1tmZl3AXgRNfTPDGsp9wO8JgkumfDtJ2tTM3jCzRyLbxwDvDsv+Z8uR6M7M\nniUIru3ADcBrkn6TCSBJPytJAk4GzjCz183sTeD7BF9YEpfHVYYHDpdYeOG5HTgP2NzMHkz41Pbw\n/9cj26aaWRuwIfA14I+S3pXnGKea2UiC2sAoYJvMA5LeGzZbvCLpXwQXmC0Sli3jlcjtNQQXV4Ct\ngWWRx6K3B5B0sKSHwuaTTuCQrLL808zW5TjXaGBY1vFfyHOqfxJcOAEIL65twAeBDbL2jR5za2CZ\nmfVmnSfzGR0VlvkFSX+U9OFw+yyCWt7dYRPc9LiChYHhGDMbTRA0PwacC6k+q9HACGB+GBA7gTvD\n7anK48rPA4dLK9Px+dsUz/kkQRPOgCGwZtZjZrMJmjA+UuhAZrYIuJCgiUfh5kuBp4DxZrYpQbOF\nYg6R1stEghSQb/TXBsDNBB33W4UX8jsSlmUlsC7r+GPz7H8v8CFJ2+TZJyP6TfwlYNus/oCxQAeA\nmT1sZlMImrHmENQaCPsqzjKz7YEjgDMl7V/wxGYPEzRVvj/clPSzeg3oAnY2s7bwZ2Q4QKLo8rjy\n8MDh0vojQdNSoVFQSNpK0teA84EZWd9yM/tI0hSCWsSTCctwFUG7dqZTdxPgX8BbkiYA/561/6vA\n9gmPne0G4MSwQ3kE8B959h1O8G1/JbBO0sFAoqGxZtZDcIG9QNIISTsRjFqL2/9u4H6CJr89FQzN\nbSFoispnLkEt52xJLZL2AQ4HrguP8VlJI82sm+A97QWQdJik94TBehVBoM/1eX5E0pclbRnen0Dw\nOWX6sAp9VpnX1wtcTtBHlDlWu6TM4IxE5XGV4YHDpRK23d8b9gXE6ZS0mmAU1iHAp8zsV1n7/E7S\nWwQXke8BJ5jZ4oRlWEsw7DRzEf8G8BngTYKLTXYH+AXAVWGTxzFJzhE51/8BPye4SC9l/QVwwKCA\nsB3+VIJg80ZYpthRZzl8jaDZ6hXgN8CvC+z/SYL+id8CncBzwGcJR77lEr53hwMHE3yr/yXweTN7\nKtzlc8DzYTPSv4XHAxgP/IGgT+ZvwC/N7P4cp+gkCBSLws/3ToK+sR+Gjxf6rKLOIXzPw/L8gfUD\nLJKWx1WAvD/JueQkvQ94HNggq6/CuSHDaxzOFSDpk+GcglHARcDvPGi4ocwDh3OFfYWgc/8Zgrb0\nnO3yzg0V3lTlnHMuFa9xOOecS6XiSc9qYYsttrBx48bVuhjOOddQ5s+f/1o4cTOvQRk4xo0bx7x5\n82pdDOecayiS8mUr6FOxpipJv1KQAfTxHI+dpSDD6BbhfUn6uaSlCjKUfiCy7wmSng5/YidEOeec\nq45K9nH8hiDbaD8KFuw5EHgxsvlgggk94wkSm10a7rsZwazjPYE9gPMjmUSdc87VQMUCh5n9if5J\n7TJ+CpxN//w5U4Crw1nJDwFtClZ4mwzcEyZwe4NgUaABwcg551z1VHVUVZiTqMPMFmY91E7/DJ7L\nw21x251zztVI1TrHwwRx3yRh0rcijn8yQTMXY8fmSyrqnHOuFNWscexAsLraQknPE6SqfiRcg6GD\n/umktwm3xW0fwMwuM7NJZjZp9OiCo8mcc84VqWo1jnAdhS0z98PgMcnMXpN0G/A1SdcRdISvMrOX\nJd0FfD/SIX4gMKNaZXbODT1zFnQw664lvNTZxdZtrUybvCNTJ3oLeVTFAoekawmWkNxC0nLgfDO7\nMmb3OwjSby8lWCvgRAhWNZP0XeDhcL/vFEjn7ZxzRZuzoIMZsxfR1d0DQEdnFzNmLwLw4BExKHNV\nTZo0yXwCoHMurb1n3kdHZ9eA7e1trTw4fb8alKi6JM03s0mF9vNcVc45F3opR9DIt32oGpQpR5xz\nQ0u5+iW2bmvNWePYuq21HMUcNLzG4ZxraJl+iY7OLoz1/RJzFuQcgJnXtMk70trS3G9ba0sz0ybv\nGPOMockDh3Ouoc26a0lfZ3ZGV3cPs+5akvpYUye284Mjd6G9rRUR9G384MhdvGM8izdVOecaWrn7\nJaZObPdAUYDXOJxzDS2u/8H7JSrHA4dzrqF5v0T1eVOVc66hZZqVfLZ39XjgcM41PO+XqC5vqnLO\nOZeKBw7nnHOpeOBwzjmXigcO55xzqXjgcM45l4qPqnLOuQoajAtDeeBwzrkKGawLQ3lTlXPOVUg5\nEzDWEw8czjlXIYN1YSgPHM45VyGDNQFjxQKHpF9JWiHp8ci2WZKekvSYpFsktUUemyFpqaQlkiZH\nth8UblsqaXqlyuucc+U2WBMwFgwckraR9A1Jt0p6WNKfJP1S0qGS8j3/N8BBWdvuAd5vZrsC/wBm\nhOfYCTgO2Dl8zi8lNUtqBi4BDgZ2Aj4d7uucc3VvsC4MlXdUlaRfA+3A74GLgBXAhsB7CS7w50qa\nbmZ/yn6umf1J0risbXdH7j4EHB3engJcZ2bvAM9JWgrsET621MyeDctzXbjvEyleo3PO1cxgTMBY\naDjuj83s8RzbHwdmSxoOjC3y3F8Erg9vtxMEkozl4TaAZVnb98x1MEknAycDjB1bbJGcc84Vkrep\nKlfQkDRK0q7h42vNbGnak0o6F1gHXJP2uXHM7DIzm2Rmk0aPHl2uwzrnnMuSaAKgpAeAI8L95wMr\nJP3VzM5Ie0JJXwAOA/Y3Mws3dwDbRnbbJtxGnu3OOedqIOmoqpFm9i/gSOBqM9sT2D/tySQdBJwN\nHGFmayIP3QYcJ2kDSdsB44G/Aw8D4yVtFzaLHRfu65xzrkaSphwZJmkMcAxwbpInSLoW2AfYQtJy\n4HyCUVQbAPdIAnjIzP7NzBZLuoGg03sdcIqZ9YTH+RpwF9AM/MrMFid9cc4558ovaeD4DsHF+y9m\n9rCk7YGn8z3BzD6dY/OVefb/HvC9HNvvAO5IWE7nnHMVlihwmNmNwI2R+88CR1WqUM455+pX0s7x\n0cCXgXHR55jZFytTLOecc/UqaVPVrcCfgT8APQX2dc45N4glDRwjzOycipbEOedcQ0g6HPf3kg6p\naEmcc841hKSB4zSC4PG2pDfDn39VsmDOOefqU9JRVZtUuiDOOecaQ+I1xyUdAXwsvPuAmf2+MkVy\nzjlXz5IOx50JfIj1SQlPk7S3mc2oWMmccy6hOQs6mHXXEl7q7GLrtlamTd5x0KUyrydJaxyHALub\nWS+ApKuABYQLMTnnXK3MWdDBjNmL6OoOZgp0dHYxY/YiAA8eFZJm6di2yO2R5S6Ic84VY9ZdS/qC\nRkZXdw+z7lpSoxINfklrHD8AFki6HxBBX4ev/+2cq7mXOrtSbXelSzqq6tpwTY4PhZvOMbNXKlYq\n55xLaOu2VjpyBImt21prUJqhIW9TlaQJ4f8fAMYQLN26HNg63OacczU1bfKOtLY099vW2tLMtMk7\n1qhEg1+hGsdZBMkNf5zjMQP2K3uJnHMuhUwHeL5RVT7qqry0fvXWwWPSpEk2b968WhfDOVcHskdd\nQVAj+cGRu3jwyCJpvplNKrRf3hqHpCPzPW5ms9MWzDnnqinfqCsPHMUp1FR1eJ7HDPDA4Zyraz7q\nqvzyBg4zO7FaBXHOuUrwUVflV6ip6sx8j5vZT/I891fAYcAKM3t/uG0z4HqClQSfB44xszckCbiY\nYIb6GuALZvZI+JwTgPPCw15oZlcVflnOOReYNnnHnH0cPuqqeIVmjm9S4Cef3wAHZW2bDtxrZuOB\ne1k/ifBgYHz4czJwKfQFmvOBPYE9gPMljSr0opxzLmPqxHZ+cOQutLe1IqC9rdU7xktUqKnq28Ue\n2Mz+JGlc1uYpwD7h7auAB4Bzwu1XWzDE6yFJbZLGhPveY2avA0i6hyAYXVtsuZxzQ8/Uie0eKMqo\nUFPV2Wb2Q0n/SdAZ3o+ZnZryfFuZ2cvh7VeArcLb7cCyyH7Lw21x23OV9WSC2gpjx45NWSznnHNJ\nFRpV9WT4f9knRZiZSSrbJBIzuwy4DIJ5HOU6rnPOuf4KNVX9Lvy/XB3Sr0oaY2Yvh01RK8LtHcC2\nkf22Cbd1sL5pK7P9gTKVxTnnXBESpVWXNEnSLZIekfRY5qeI890GnBDePgG4NbL98wrsBawKm7Tu\nAg6UNCrsFD8w3Oacc65GkqZVvwaYBiwCepM8QdK1BLWFLSQtJxgdNRO4QdJJwAvAMeHudxAMxV1K\nMBz3RAAze13Sd4GHw/2+k+kod845VxuJclVJ+ouZfaQK5SkLz1XlnHPplSVXVcT5kq4gmHvxTmaj\n56pyzrmhJ2ngOBGYALSwvqnKc1U55xqCp1Uvr6SB40Nm5vPznXMNJzutekdnFzNmLwLw4FGkRKOq\ngL9K2qmiJXHOuQrIl1bdFSdpjWMv4FFJzxH0cYhgDt+uFSuZc86VgadVL7+kgSM7WaFzzjUET6te\nfomaqszshVw/lS6cc86VatrkHWltae63zdOqlyZpjcM55xpSpgPcR1WVjwcO59yg52nVyyvpqCrn\nnHMOSBA4JB0rafvw9q6Slkp6SdJRlS+ec865epOkxjGNIL05wHeB04APEiQtdM45N8QUWgHwfGBr\n4BxJzcBHgAXAJGCkpG8BD5jZnypeUuecc3Wh4JrjkvYFngNGA3ea2QUAkiab2XcqX0TnnHP1JElT\n1b8DhwG7A98ACNOP3F7BcjnnnKtTBYfjmtmTwLFZ254AnqhUoZxzztWvvDUOScdLit1H0g6SGmaB\nJ+ecc6UrVOPYHFggaT4wH1gJbAi8B/g48BowvaIldM45V1fy1jjM7GLgA8C1BJ3j+4f3O4DPmdlR\nZvZ02pNKOkPSYkmPS7pW0oaStpM0N5wncr2k4eG+G4T3l4aPj0t7Puecc+WTpI+jB7gn/CmZpHbg\nVGAnM+uSdANwHHAI8FMzu07SfwEnAZeG/79hZu+RdBxwEVl9Ls4556qnVilHhgGtkoYBI4CXgf2A\nm8LHrwKmhrenhPcJH99fkqpYVueccxFVDxxm1gH8CHiRIGCsIug/6TSzdeFuy4FMRrJ2YFn43HXh\n/ptnH1fSyZLmSZq3cuXKyr4I55wbwqoeOCSNIqhFbEcwK30jyrBQlJldZmaTzGzS6NGjSz2cc865\nGIkCh6StJF0p6f/C+ztJOqnIc34CeM7MVppZNzAb2BtoC5uuALZhfX6sDmDb8LzDgJHAP4s8t3PO\nuRIlrXH8BriLoIYA8A/g9CLP+SKwl6QRYV/F/gSTCe8Hjg73OQG4Nbx9W3if8PH7zMyKPLdzzrkS\nJQ0cW5jZDUAv9PU19BRzQjObS9DJ/QiwKCzDZcA5wJmSlhL0YVwZPuVKYPNw+5n4vBHnnKuppCsA\nrpa0OWAAkvYi6KQuipmdz8C07M8Ce+TY923gU8WeyznnKmnOgo4htyxt0sBxJkGT0Q6SHiSYDHh0\n/qc45xrNULwIlmLOgg5mzF5EV3fQANPR2cWM2YsABvX7lihwmNkjkj4O7AgIWBJ2bDvnBomhdBEs\nV4CcddeSvvcro6u7h1l3LRl071lUosAh6fNZmz4gCTO7ugJlcs7VwFC5CJYzQL7U2ZVqe+b8jV6r\nS9pU9aHI7Q0JRkI9AnjgcG6QKOYi2IhKDZDRC3+TRE+OQZ5bt7XGPncw1OqSNlV9PXpfUhtwXUVK\n5Jyria3bWunIESTiLoKNqpQAmX3hzxU0WluamTZ5x5zPHyy1umJnjq8mmPntnBskpk3ekdaW5n7b\n8l0EG1VcIDRg3PTb2WHGHZw3Z1HOfXJd+AGaJQS0t7XygyN3iQ0Cg6VWl7SP43eEQ3EJgs1OwA2V\nKpRzrvoyF7tGb38vZNrkHfvVGrL1mPHbh14E4MKpu/R7LO4C32vGczMPLXjuwVKrS9rH8aPI7XXA\nC2a2vALlcc7V0NSJ7WULFPXaCZwdIOPSUFw7d9mAwFHqhT9X0GrEWl3SPo4/VrogzrnBo947gaMB\nctz023Puk6v/otQL/2Cp1SVtqjqSYAGlLQnmcQgwM9u0gmVzzjWoRuoEbo4ZGdWcY9mfclz4y1mr\nq5WkTVU/BA43sycrWRjn3ODQSJ3An95z274+jeztuVTywl+vzXvZko6qetWDhnMuqbg2/3rsBL5w\n6i4cv9fYvhpGs8Txe40d0L9RaZnmvY6w3yXTvDdnQUfB51abkmQol3Qx8C5gDvBOZruZza5c0Yo3\nadIkmzdvXq2L4dyQld3HAUFfQL6hqtnPb4Rv3uW098z7cna8t7e18uD0/apSBknzzWxSof2SNlVt\nCqwBDoxsM4JFmJxzrp9S+gJq0bFeD4GqkZr3ko6qOrHSBXHODS7F9gVUu2O9XkaANdIcj6RLx75X\n0r2SHg/v7yrpvMoWzTk3FFX7m3e+QFVNjTRzP2nn+OXADKAbwMweA46rVKGcc0NXtTvW66WJaOrE\ndn5w5C60t7UmSl9SS0n7OEaY2d/Vf1zzugqUxzk3xFV7dnU9NRE1yhyPpIHjNUk7sH7p2KOBlytW\nKufckFWN2dXRzvC2ES20NInu3vUjTOu1iaheJA0cpwCXARMkdQDPAZ8t9qRhWvYrgPcTBKMvAkuA\n64FxwPPAMWb2hoJqzsXAIQQju75gZo8Ue27nXH3IN5Kp0pPsojWaN9Z009Is2lpbWNXVPWSG/5Yi\naeB4wcw+IWkjoMnM3izxvBcDd5rZ0ZKGAyOAbwL3mtlMSdOB6cA5wMHA+PBnT+DS8H/nXIOq5Uim\nXJ3h3T3GRhsM49HzD4x5lotK2jn+nKTLgL2At0o5oaSRwMeAKwHMbK2ZdQJTgKvC3a4Cpoa3pwBX\nW+AhoE3SmFLK4JyrrbiRTGfdsJDtpt/O3jPvq9iM6XJ3hs9Z0MHeM++reLnrSdLAMQH4A0GT1XOS\nfiHpI0WecztgJfBrSQskXRHWZLYys0y/ySvAVuHtdmBZ5PnLw239SDpZ0jxJ81auXFlk0Zxz1RB3\nke4xq3i6jZGtLam251NMmpDBEGgSBQ4zW2NmN5jZkcBEgpnkxaZaHwZ8ALjUzCYSrCY4Pet8BrFp\n8uPKeJmZTTKzSaNHjy6yaM65akgyYqlScylyJL3Nuz2ftHNAGikfVT6Jl46V9HFJvwTmAxsCxxR5\nzuXAcjObG96/iSCQvJppggr/XxE+3gFE01RuE25zzjWoXJPdcsnUTMr5Lb1zTXeq7UnKl3R7vUw2\nLFXSmePPA6cDfwZ2MbNjzOzmYk5oZq8AyyRlxrrtDzwB3AacEG47Abg1vH0b8HkF9gJWRZq0nHMN\nKHuyW661LyComcxZ0MG0Gxf2+5Y+7caFRQePck4wTHuseplsWKqkNY5dzeyTZnatma0uw3m/Dlwj\n6TFgd+D7wEzgAElPA58I7wPcATwLLCWYwf7VMpzfOVdjUye28+D0/Xhu5qGxa1/sO2E0F9y2uN8c\nC4DuXuOC2xYXdd5ypvZIe6xGSjefT9LhuO+SdAtBB/b7Je0KHGFmFxZzUjN7FMiVunf/HPsaQae8\nc26Quv+p3ANa7n9qJZ1dMU1LMdsLKecEw7THGixrjiddj+OPwDTgv8MObSQ9bmbvr3D5iuLrcTjX\nWLabfnvO0TAi/yiZ52ceWvayVDrFevbx950wmvufWlkXa4+Uez0Oz1XlnKuYfPmi1qxdxxs5Oq5H\njUg/fLaQakxMjM6Kr5eU7mkl7ePwXFXOuYrZd0LuIfT7ThjN+YfvTEtz/87zlmZx/uE7l70c1R71\n1KijrGqSq8o5N/iU0sSTr48js/Z3dvPOrLuWcMb1j+Y8V7Flqfaop0YdZVUwcEhqAiaVOVeVc24Q\nKbXJJVczVXR7muadUspSTIr1UgJmPaV0T6NgU5WZ9QJnh7dXe9BwzmUrtcklbh5Hru2FzlVKWdIO\nry11JngjrfoXlbSp6g+SvkGQ9rxvHoeZvV6RUjnnEqn0CKCkSm1y6YkZ3Zlre6HaSSllyTW8Nl+z\nWJL10Qulj88+XyOkdE8aOI4N/4/OpzBg+/IWxzmXVD2NyCm1yaU95vnt4fOjF984mdpJqWVJ0yxW\nKEgl+YwaZdW/qKRJDrfL8eNBw7kaqqcROcU0uUTzT61Zu46Wpv7NUpnnZzcHxcnUTsrZ/FPoPS40\nE7yePqNySpzk0DlXX+ppRE527qn2tlZ+cOQusd+ks4PBG2u6QdDW2jLg+bkuvrlkaidpy5JPofe4\nUJCqp8+onJI2VTnn6ky9jchJ0+QStwrfm28PnFec9CIbnQtSruafka0tOVObZNbumDqxnXkvvM61\nc5fRY0azxFEfXH/uevuMysVrHM41qEYdkQPpFnJKepGNmwuSRFza9kJrd8xZ0MHN8zv6msl6zLh5\nfkff8xv5M8onaVr1vcM5HEg6XtJPJL27skVzzuVTziaZJMq5JkaahZzSrt2RVr4htYXW7ijUh1Ht\nz6hakjZVXQrsJmk34CzgCuBq4OOVKphzrrBqjcgp9wiufSeM5rcPvVhwv5c6uwYMWW2Scg7TLbb5\nJ9/Fv1BTU5I+jEYcNVVI0qaqdWF68ynAL8zsEmCTyhXLOVcO581ZxA4z7mDc9NvZYcYdnDdnUVHH\nKffooKTNSltHOryTrN1RjHwX/0JNTYNlfY20kgaONyXNAD4H3B6mISl/akrnXNmcN2cRv33oxX7t\n77996MWigkcxo4PyNW0laVaK6wvIl9eqGG0xWXbbRrQUbGoarH0YhSQNHMcC7wBfDJd+3QaYVbFS\nOedKdu3cZam255P2m3WhVBxxz2uWEMGw3A1bmjjj+kcTB51i+zjejhnq+8aabvaeeR9AX23nwen7\n9Wt2Gqx9GIUk6uMws1ck3QyMDze9BtxSsVI554oSnWEdN1kuLr1HPmlXriuUiiPX8VqaxMYbDuON\nNd2s6uruK392f0q5h7h2dffGPpakL6eafRj1kmIm6QqAXwZOBjYzsx0kjQf+y8wGLPVaD3wFQDcU\nZXdgF9Is8ek9t+1LW57k+EkvWnEr+kGwql/2yncjW1tYvXYd3T3x16NmiV4z2ka0sGpNN9HLfUuT\nmPWp3RJfRM+bs6hv7kUS7W2tPDh9v0T7Vkquz7e1pbmsNZxyrwB4CrAHMBfAzJ6WtGUJ5UNSMzAP\n6DCzwyRtB1wHbA7MBz5nZmslbUAwguuDwD+BY83s+VLO7dxgEb2Yx402ipPp8wD6gkf0gpodWAp9\ns05alkzT1f/OfbFvIt2bb68rWPbM47lWAyRmvkVG2kCRrR5meidJqFgtSfs43jGztZk7koaRfyng\nJE4Dnozcvwj4qZm9B3gDOCncfhLwRrj9p+F+zg152f0IxV4UM30epXSmz1nQwbSbFqYqS68FQaCU\nsmd091jsCK/s11WMJil2/ko557fkU0/pS5IGjj9K+ibQKukA4Ebgd8WeVNI2wKEE80FQsJj5fsBN\n4S5XAVPD21PC+4SP7y/Fzed0buhImsOpkMwFtZTO9G//bnHOZqYmFawMlE3cBTTtYIBc5c01ox1K\nX48jjXoa+ps0cEwHVgKLgK8AdwDnlXDenxEsDpVpptwc6DSzTKKa5UCm7tUOLAMIH18V7t+PpJMl\nzZM0b+XK4lMPONcoyvVNM3OhTLMmRraczUcEtYrnZh5abNGA+EWeskUvoNFaQNKaRrPE8zMP5afH\n7t43SipL0PeWAAAeL0lEQVTXucu1aFRa9TT0N+moql7g8vCnJJIOA1aY2XxJ+5R6vAwzu4xgXXQm\nTZpUajOac3UvbnRRphM5aZ/HiOHBxUiCXLuXWr/fbvrtiPRt2y3NYtbRuwEU7PRvaVLfBTTtIIGM\n4cOC5qhox/+46bfn3Lcci0alVU+LPuUNHJIWkefzNrNdizjn3sARkg4BNgQ2BS4G2iQNC2sV2wCZ\nul4HsC2wPOxbGUnQSe5cYvUyjLGc4obIZkbZJL2ArlkbPN46rIk1OYamtg4LGiaKfQ+L/RaX6bfI\njGbKnHtkawtvvrOOnt7IkSPBrdgmvMyw3I7OLqbdtBAIgnCu4FuuRaPSqpf0JXmH4xZKZGhmL5R0\n8qDG8Y1wVNWNwM1mdp2k/wIeM7NfSjoF2MXM/k3SccCRZnZMvuP6cFwXVcwwxnoNNNnlig5pzVXO\nJCOdMjWUfMNnf3rs7nnfw4nfuTu2uSrXuZLWhsTApq69Z96Xt6ZVruaGUSNa8r6m52ceWpUhstWU\ndDhuonkclZIVOLYnGI67GbAAON7M3pG0IfA/wETgdeA4M3s233E9cLiouAtN3Nj8er0YlFquYptw\nIP6bd+Y9nLOggzNveJTeApeTTCDIN88jqq21hUfPP7DftqTPLYe44DFqRAsLvhWUq5RgXk9fSqDM\n8zgk7QX8J/A+YDjQDKw2s01LKaSZPQA8EN5+lmCuSPY+bwOfKuU8bmhL2w5dT+Plo0otV9Iss7nE\n7Rd9DxXXSRKRacKJa+LJFl33otj5KqWIO010e5o1yutpnfhSJB1V9Qvg08DTQCvwJeCSShXKuXJK\nO4yx0h2exWasLUe5ollme8tw8c1M4Pv27xb373PIIToCaNrkHQesMZ5L55russ1XSauttYVVOVb/\nA2K3FxplNVjWIE+8AqCZLQWazazHzH4NHFS5YjlXPmmHMVZyvHwpk+ySlCvNZLTMRT9bW8z2XNau\nCy6C+foC4pL/xWeIWm/rttbYzu5MQsRKOWy3MbHvUdz2QsG9nibxlSJp4FgjaTjwqKQfSjojxXOd\nq6m0GUzLPV4+ejGPW7woySS1QuVKOxltzdqB63sDOdfYjpNrFFa2XFll09RQ4i6qvWY8N/NQRsWk\nRS/V7xe+XHDp2GxxwT0z87wp5omNtn5H0lxVnyMIFF8DziAYHntUpQrlXLmlGcZYzvHySTukkzS/\nFCpXXDPIWTcs5IzrHx2w/9o8CQXTipunEVcjKFRDiZZ11l1L8g55Pf/wnTnrxoUFA1FanV3dseWP\nW1I21xBpoF8NM1sjrt+RdAJgZtjt25J+DmwbNl05NyiVa7x80jkFSWdH5ytX3DfzzMWqEh2xmW/7\ncZdsg75JddHRRvlkD79NktK9CSg9+cpAaedpJB2AkBk6XG+jqpJKmlb9AeAIgkAzH1gBPGhmZ1a0\ndEXy4biuXiQdOnr8XmMTpzePEzfsOFu55js0KWjr71zTXdbhsdk1Dsg/5DXJKKsmwaYbBp3dSUdl\njRrRwvmH71zSEOi4zz/X/JR6UO606iPN7F+SvgRcbWbnS3qstCI6N/gVGnaadk2MfPadMDq2DyWq\n1FFJIggY/3q7O9Gkv7Si/TOwvpYVN+Q13+vJFYS2i0kjEtXcJM4/fOeSmy2rPbO8WpIGjmGSxgDH\nAOdWsDzODSqF0oKUU7FrbhcjukJfpcTNUSkmpci8F15PNQ8kOvInu3kwM9ghyYS/tCsnNoqkgeM7\nwF3AX8zs4XCW99OVK5ZzjSPfTOBqJqar1pDOauaayPWa0rzOTO0lWhNLUuPq7rW+oBX9fNtGtPDW\n2+vo7l3fbxQ9dq6aEtRHYsJySto5fiPBGhyZ+8/io6pcHatWWockM4HTdrQf8JMHeHrF6r7747fc\niHvO3KfvfHGva2RrS86htAqHPFVrxrXCf5KcqlDZcjXpJJ11XqqXOrsGfL5JmubqIctApflcDDfo\nVHNxnXLPBM4OGgBPr1jNAT95oODrihuY1dbawnMzD+XHx+w2YB5IuWWKkHQSoRmxs8GbIGeTzr4T\nRhdfwBQMOOuGhUXl9srUiqr5u1hNHjjcoFPNtA7lngmcHTSi2wu9rrhvw5nt2RMhK8HCn3J0mvcS\n9E1k+/3Cl0s+dlLF1tAyNaXBkmIkmwcON+hUM61DNZfzLHeuqkbw24deHJA6Jc3M9lpoaRJr1q5j\nu+m3xzapNVqKkWyJAoek0yRtqsCVkh6RdGDhZzpXfdW8mE+bvOOAP6K4JpaMNPmkouppzelSpK3t\ndHR2Me3GhXXTvNPSLNpaW/rS1xy/19i+WlxbawsoqHHlq6s02meWLemoqi+a2cWSJgOjCFKQ/A9w\nd8VK5lyRihkCWWxn+rwXXh+QrK8XmDH7sZxpPgp1po/fcqOczVXjt9yIU/YdP+B1Rb/dxkk6K71a\nimn86e41Tr/+UU6//tGylyepXHNCsu09876CNaKhNBw385t3CPA/ZrZYqrPfRudCaYdA5rqYT7tp\nIRfctphVXd15nx+XnDC6DGk0MBRaU+OeM/fJO6oq+royS6gW6k/49J7b9nutmeeXavyWG7FmbS8v\nhR2/g1lTwlFi+d7XJIGnUSQNHPMl3Q1sB8yQtAnJsiI7VxNphsDmuph391jfN8d8OZ6SdJ5GA0Pc\nhaWjs6svp1O+C8u8F17nlVVvY6Rv6y9lBcBcnl6xmo2GV3aUVr3I5E/M9buQZJGpuNUmG1XSzvGT\ngOnAh8xsDdACnFixUjlXRUm+fceNhEnaDJQ5R7627ULDNbPX8kjqmnCCWjEzrgtZvbZnUNc2cn2+\n0d+FJItMDYamqWxJA8eHgSVm1inpeOA8YFUxJ5S0raT7JT0habGk08Ltm0m6R9LT4f+jwu2S9HNJ\nSyU9JukDxZzXuThJOypzBZhoM1CScyRZ+S56YYquFpgkD1UumUtZNSbNDRYtTeJnx+4eu0pipoZY\naJ5Hs8RRHyxPpuV6kjRwXEqwmNNuwFnAM8DVRZ5zHXCWme0E7AWcImknghrNvWY2Hrg3vA9wMDA+\n/Dk5LItzZZNrgaRccgWYC6fuwvF7je37ZtqkgX9UA75xJqikdHR29QWLcs32rrdO8lpI+g6sC9um\nCtUQC302PWbcPL+jbkaElUvSPo51ZmaSpgC/MLMrJZ1UzAnN7GXg5fD2m5KeBNqBKcA+4W5XAQ8A\n54Tbr7Yg//tDktokjQmP44aocqYUye5Mz85HBPmbGy6cuku/7Lb5yjbrriV0l3EBpSQyF8tqrdVd\n75oTpF4x4PTrH6WttYXmJpW0SFShFCTVSo9TTkkDx5uSZhAMw/2opCaCfo6SSBoHTATmAltFgsEr\nwFbh7XYgOnRlebitX+CQdDJBjYSxY8eWWjRXx5LkhyrFiOHDOHTXMbEZTwv9oefrmK9Fc5EB4xKk\nEh8KktQSoso12TCuH63Sv8uVkjRwHAt8hmA+xyuSxgKzSjmxpI2Bm4HTw7U++h4LazepQryZXQZc\nBsFCTqWUzZVHqd+k4p5faEhrMeXM/uO9eX5HztTnxfyhJx0CG7f8qmt8cU1e5f5drpZEfRxm9gpw\nDTBS0mHA22ZWbB8HkloIgsY1ZjY73PxquOYH4f8rwu0dBGucZ2wTbnN1rNTkbvmeX0zqjXyztdPk\nE0qbe2jOgg6m3biw73W4oSkuMWNcDbSjsytVVoFqS5py5Bjg78CnCBZzmivp6GJOGE4cvBJ40sx+\nEnnoNuCE8PYJwK2R7Z8PR1ftBazy/o36V2pyt7jnn379ozTFdPLGfasrFMTSBKK0QeuC2xb36yvJ\nxwPL4BW3yFa+AQv1nEk36aiqcwnmcJxgZp8H9gD+o8hz7k3QV7KfpEfDn0OAmcABkp4GPhHeB7gD\neBZYClwOfLXI87oqKjUhX76+gFxt1NHUG2lrFGlyQMXtm+lH2GHGHZw3Z1Hf9npPyOeqI+73vlB/\ny5q13Xzz5//DIYccgqTYn0996lOVKHaspH0cTWa2InL/nxSZWdfM/kL8qLj9c+xvwCnFnMvVTtxi\nO01SohnSTVo/WzdOs0SvGSNbW1i9dn3qjex+h0JBbNrkHZl248J+NYMm6AtE0bJOm7wj025aGDsy\nqsesb75FOdYRd/VFCi5eaQdZ5frCsWzZMl646LBEz3+ywONHH11UA1DRkgaOOyXdBVwb3j+WoCbg\nXE65Eg3C+m9YhTqVk/xh9prx3MxDcyaWi3YwxgWxfn/MWV9leqFfIDojTLA3akRLouG0185dxoVT\nd2Gj4c2sXlve2dqudsxgWJMY2TqMzjXdtI1oYVVX94Df15W3/ZA1T/6p7/4LgGaUdu4xY8Zw6qmn\n8qUvfYktttiitIOVKOnSsdMkHQl8JNx0mZndUrliuUaXPTciVw6fUkePZC78xdQoWprUNy8jydyK\nzKNJFyjKvNaW5ibAA8dgsvQHlVnLpPU9ezD68LPR8A3Xb2tpzjm6r9YKBg5JzcAfzGxfYHah/Z3L\niM5niEv7HXfRb4tZPzsjOiGvmBpF9H4lF9XxPo7GsXblC7z8q8q1ivf29lIoqXijTAYsGDjMrEdS\nr6SRZlZUfio3NCXJGhrX2XzBETsPqCVktLe1su+E0cy6awlnXP8oI1tbaGlWv1pDtLM817m7e6xg\nU5YbPJL2JRTr3ef8vt/9YrPhpsnqXEtJ+zjeAhZJugfoWyjAzE6tSKlcw8ueKJc2a+jUie3Me+F1\nrp27jB4zmiU+vee2XDh1lwHH7uzqpqVJjBrRQuea7gGd5XEjVzoiTVnlTDfuqquSQWHTPY9i1D7p\nE4E3+tKwhSQNHLPxZiqXQlwK78xIqCSLK908v6Pvop9JFjfp3ZvlXj+j1xgxfBgLvnVgolXYorKD\nVDm0thQ16NBF9L6zhmU/O6Zix9/6K1fQ0vauyhy7wZeGLSRp5/hVlS6IG1zivnFlRkIVkm8CYJxM\nquu0l/7sIFUOcZMUXeDl35zG2lefqdjxs5uOqim7Jt0o/RZp5A0cYTbcbczskvD+XCAzd/5sM7up\nwuWrmcH4YZcqzXuSqMM6j2Kr+mkv/XF9IKUaykNwq92fUCuZpWD3nTA6b0LMRkxiWEihGsfZwHGR\n+xsAHwI2An4NDMrAMVg/7FKkfU9yTZRraVbildCq1WGdNlvqUFfJoLDxxEPZ/MB/r9jxy62UmnO9\nJzEspFDgGG5m0ZTmfzGzfwL/lLRRBctVU4P1wy5FUe9J9vU4xfW5XB3Wtcw420ipzNc8/RArZ19Y\nseOPnXYraho865OPiPRh5auJl5p6p14VChyjonfM7GuRu7nTPQ4CjfRhV6tJLe17MuuuJQOG0nb3\nWuLgO3ViOzfOe5EHn3k9fWEjMjPLh/Jw26HSdFQtTYLvH7krULgmXmqTbb0qFDjmSvqymV0e3Sjp\nKwTZcgelRvmwq9Gkdt6cRVw7d1nst/a43FNJAk2+oHfenEUlB42MfSeMLnq97nrnQaE6MkMdcq3o\nmK8mnqvm3NIsVr8zMA9aIykUOM4A5kj6DPBIuO2DBH0dUytZsFrK9WHnm3NQqmJrDZVuUjtvzqKC\nF9xo7qmzblzIBbctZlWeobAjW4OFIwsFvWvnLos9RhpxM9brXW/32yz7SeUS173r+B+xQfuEih1/\nsDFyZ2Yt9AUpblnizHDxUr/s1WoQjyxBx6Ck/YCdw7uLzey+ipaqRJMmTbJ58+aVdIxqfSDZF1Do\nn58mXznytaFnRnyUUu68x1eQ8C2tpvB5cSOZMvM8BnN39T//7+e89djdFTu+1xIqK/r3GdcMGjdf\nKW7/YmaaF7p2FEPSfDObVHC/JIGj0ZQjcFRLvl+8HrMBnbvRX4wdZtxRcERQKb9IjdS5Wy+86Who\nyFzoMys85lusK/o3GDfPSCQbpRVVziDUV46EgSPpzHFXIYUWeMn+JYs2RSUZRprddOXzU4pX0aGo\nux7I5gd7Bp9G0e/vtsBcz9Qp/ospQ4Lt5eSBowaSJP/LJ/OL0Z5wrkNm/1z9CtNuXMi3f7eYzjXd\nAwJJLYeyVlP3Gy/x0mUnV+z4Y79xC2puqdjxXfVlLvRJUvJD/xT/5eo/reUgHm+qKoM03+JztUum\nlaaaDOvbW5MGqUyywNaWJtZ09xZdToh8GSuyT6RUK275Hl3/+FvFju9NR0NPEzAy/BtJ+isdbT4q\nV62/ln0cXuMoUZJv8SOGN/H0itV5j5P04p797SRJIIgmCkwik1V2TXdv0JlN8Rf9YU3w9PcPZfsi\nckgVUsmmo2FtY2j/yuWFd3RDTnR1yKT2nbB+2lu5Uqdnj9iqZtNzwwQOSQcBFwPNwBVmNrPGRQJy\nD4nt7rV+y44mkekI37ClKTbPUXvWL8YFty1OvfZxWqUeP1NhSVNvsd4eXpw1pbQT57H1yZfRMmrr\nih3fuWz3P7WyIset1fodDRE4wlUILwEOAJYDD0u6zcyeqG3JytsRZcQnx8s1UqIRV5d7+8XHePXa\nb1bs+N505OrRYMtc0BCBA9gDWGpmzwJIug6YAtQ8cFQrGV89pjvJtvLWi1jz1J8HbNdFpR+7acON\n2fa060o/kHM10DzI0uw3SuBoB6JTiZcDe0Z3kHQycDLA2LFjq1awUpLxpRm1lGukxKgRLanbWotV\nyf6ELY++gNYdCvbHOdewBlsG5kYJHAWZ2WXAZRCMqqrWebM7qDLLlhYaorf3DptxzZc/nCgBX9xw\nvfMP33lA6vK0et5+i+UXH1d4xyJ9+MK7+eu5BzDxO3dXLcg5V2/a6yzPXakaJXB0ANtG7m8TbqsL\n2R1U2cPtskdVZYIG5K6xtDSLjYYPY1XXwLkV2eeF9UFrxPDmfn0kq5/6C6/dWqExBGri3WfflneX\n1pZmzj5kJyB5kGsiqIVt2NJEV4lDgZ3Lp1rzlCqZ565WGiVwPAyMl7QdQcA4DvhMbYsUL81Ih2KH\n1J177rl8//vfL7msuVxyySV89atfBeCzl/+tX5ba4c2iuyfIwTNu81YeevYNesxolthr+1E8/8+u\nnK8j1+tc19PDq2+u7Tv2+C034p4z9+m7n8nM22OGBK3DgmCS69zbjx7BsyvXxN7fYuOWfufasFm8\nHQli2ffHb7kRa9b28lJnF8Oa1o8Oc41BrM+Hluv34dN7bsuFU3fp27/f7xowYngza9b25PziF/3d\nyLUCYL4VAQeLhpkAKOkQ4GcEw3F/ZWbfi9u3kXJVZTMzDjroIO6+uzJJ8F5++WXe9a53VeTYzrnG\nNugmAJrZHcAdtS5HKd566y2OOuqoigSF973vfTzxRM0HmTnnhoCmwru4JFavXs0Pf/hDttpqKyTl\n/Nlkk02KChpXXHEFZpb3x4OGc65aGqbGUUsrVqxg7ty5/X7efPPNko+78cYb88wzz7DllluWoZTO\nOVcdHjhirF27lg022KCo506ZMoVTTz2VfffdFw2yiT/OOeeBI8awYcP48Ic/zN/+9jc222wz9txz\nz76fPfbYg80226zWRXTOuZrwwBGjqamJv/71r7UuhnPO1R3vHHfOOZeKBw7nnHOpeOBwzjmXigcO\n55xzqXjgcM45l4oHDuecc6l44HDOOZeKBw7nnHOpNExa9TQkrQReqHU5gC2A12pdiBhetvTqtVzg\nZStWvZatVuV6t5mNLrTToAwc9ULSvCS57WvBy5ZevZYLvGzFqtey1Wu5MrypyjnnXCoeOJxzzqXi\ngaOyLqt1AfLwsqVXr+UCL1ux6rVs9VouwPs4nHPOpeQ1Duecc6l44HDOOZeKB44KkLStpPslPSFp\nsaTTal0mAEkbSvq7pIVhub5d6zJlk9QsaYGk39e6LFGSnpe0SNKjkubVujxRktok3STpKUlPSvpw\nrcsEIGnH8P3K/PxL0um1LheApDPCv4HHJV0racNalylD0mlhuRbXy/uVzfs4KkDSGGCMmT0iaRNg\nPjDVzJ6ocbkEbGRmb0lqAf4CnGZmD9WyXFGSzgQmAZua2WG1Lk+GpOeBSWZWd5PFJF0F/NnMrpA0\nHBhhZp21LleUpGagA9jTzGo6OVdSO8Hv/k5m1iXpBuAOM/tNLcsFIOn9wHXAHsBa4E7g38xsaU0L\nlsVrHBVgZi+b2SPh7TeBJ4H22pYKLPBWeLcl/Kmbbw6StgEOBa6odVkahaSRwMeAKwHMbG29BY3Q\n/sAztQ4aEcOAVknDgBHASzUuT8b7gLlmtsbM1gF/BI6scZkG8MBRYZLGAROBubUtSSBsCnoUWAHc\nY2Z1Ua7Qz4Czgd5aFyQHA+6WNF/SybUuTMR2wErg12ET3xWSNqp1oXI4Dri21oUAMLMO4EfAi8DL\nwCozu7u2perzOPBRSZtLGgEcAmxb4zIN4IGjgiRtDNwMnG5m/6p1eQDMrMfMdge2AfYIq8Y1J+kw\nYIWZza91WWJ8xMw+ABwMnCLpY7UuUGgY8AHgUjObCKwGpte2SP2FzWdHADfWuiwAkkYBUwiC7tbA\nRpKOr22pAmb2JHARcDdBM9WjQE9NC5WDB44KCfsQbgauMbPZtS5PtrA5437goFqXJbQ3cETYl3Ad\nsJ+k39a2SOuF31IxsxXALQRt0PVgObA8UnO8iSCQ1JODgUfM7NVaFyT0CeA5M1tpZt3AbOD/1bhM\nfczsSjP7oJl9DHgD+Eety5TNA0cFhJ3QVwJPmtlPal2eDEmjJbWFt1uBA4CnaluqgJnNMLNtzGwc\nQbPGfWZWF98CJW0UDnIgbAY6kKBJoebM7BVgmaQdw037AzUdhJHDp6mTZqrQi8BekkaEf6v7E/RD\n1gVJW4b/jyXo3/jf2pZooGG1LsAgtTfwOWBR2J8A8E0zu6OGZQIYA1wVjnBpAm4ws7oa9lqntgJu\nCa4xDAP+18zurG2R+vk6cE3YJPQscGKNy9MnDLQHAF+pdVkyzGyupJuAR4B1wALqK8XHzZI2B7qB\nU+pxsIMPx3XOOZeKN1U555xLxQOHc865VDxwOOecS8UDh3POuVQ8cDjnnEvFA4drGJLeCv8fJ8kk\nXRh5bAtJ3ZJ+Ed6/QFJHmJX1aUmzJe0U2f8BSUvCx5+MSyMS2W+hpIcl7Z6gnKeH6SIy9+/IzJ8p\nlaQx0czBkmZIWhqWcXIRx7tTUmd2NmJJ20maGx77+nCoL5K+JumLpb8S18g8cLhG9RxBQsSMTwGL\ns/b5qZntbmbjgeuB+ySNjjz+2TD9yt7ARZmLYw6fNbPdgF8CsxKU7XSCxHkAmNkhZRyLfyZwOUAY\nCI8DdibIAPDLcI5OGrMI5hxlu4jg/XsPwezlk8LtvyKYN+KGMA8crlGtAZ6UNCm8fyxwQ9zOZnY9\nQf6fz+R4eGOCHE+FcgL9jUiWY0mXSpqnyNomkk4lyH90v6T7w23PhzWicWHt5vLwOXeHM/iR9CFJ\nj4U1oFmS4mamH0WQwwiCfEvXmdk7ZvYcsJSUqVDM7F7gzei2cDb1fgTpSwCuAqaG+68BnpdULylX\nXA144HCN7DrgOEnbElz0C6XGfgSYELl/jaTHgCXAd82sUOA4CJgTuX+umU0CdgU+LmlXM/t5WI59\nzWzfHMcYD1xiZjsDnQSBAODXwFfCGlDOckjaDnjDzN4JN7UDyyK7LKc86fs3BzrDtN65jjsP+GgZ\nzuMalKcccY3sTuC7wKsETVGFKOv+Z81sXth89VdJd8asF5FJ57ExEO3jOCbsGxlGkM5lJ+CxAmV4\nzswyaWjmA+PC/o9NzOxv4fb/BXItYjWGIIV6ra2gfwB2Q4zXOFzDMrO1BBffs1jfrJLPRHIkszOz\nlQS1kT1jnvdZYHuCJpv/hL5v/98A9jezXYHbgSTLj74Tud1Dui9vXVnn6KD/Wg3bhNv6SPqk1i/d\nOolk/gm0hYsc5TruhmFZ3BDlgcM1uh8D55jZ6/l2knQUQVbbAVlawxFQE4Fn4p5vQVK3/yDIqjoB\n2JSgX2SVpK0IUodnvAlskvQFhB3nb0rKBK7jYnb9BzAucv82gqa6DcJANh74e9axbwkHCOxuZonW\nSg9f6/3A0eGmE4BbI7u8lzrJDuxqw5uqXEMzs8UMHE2VcUa4QM9GBBe6/cLaRcY1krqADYDfFFpE\nKlyf+sfANDM7SdICgrT0y4AHI7teBtwp6aWYfo5cTgIul9RLsFzoqhznXy3pGUnvMbOlZrZYwXrZ\nTxBkeT0lQT9NP5L+TNDstLGk5cBJZnYXcA5wXTjkeQHh0rShvYEL0pzHDS6eHde5OiBp48x68JKm\nA2PM7LQc+30S+KCZnVftMobnnwicaWa5hvC6IcJrHM7Vh0MlzSD4m3wB+EKunczslnCthlrZgqDJ\nzg1hXuNwzjmXineOO+ecS8UDh3POuVQ8cDjnnEvFA4dzzrlUPHA455xL5f8DZzABQdXGluEAAAAA\nSUVORK5CYII=\n", 411 | "text/plain": [ 412 | "" 413 | ] 414 | }, 415 | "metadata": {}, 416 | "output_type": "display_data" 417 | } 418 | ], 419 | "source": [ 420 | "# Plot line of best fit to scatter plot above using the estimated parameters\n", 421 | "plt.plot(imdb_score, new_param[1]*imdb_score + new_param[0], '-k')\n", 422 | "\n", 423 | "# Visualize data\n", 424 | "plt.scatter(imdb_score, gross_rev)\n", 425 | "\n", 426 | "# Chart title\n", 427 | "plt.title('IMDB Rating and Gross Sales')\n", 428 | "\n", 429 | "# y label\n", 430 | "plt.ylabel('Gross sales revenue ($ millions)')\n", 431 | "\n", 432 | "# x label\n", 433 | "plt.xlabel('IMDB Rating (0 - 10)')\n", 434 | "\n", 435 | "plt.show()" 436 | ] 437 | }, 438 | { 439 | "cell_type": "markdown", 440 | "metadata": {}, 441 | "source": [ 442 | "Therefore movies with a 1 higher IMDB rating are associated with having approximately 30 million dollars higher gross sales revenue on average. " 443 | ] 444 | } 445 | ], 446 | "metadata": { 447 | "kernelspec": { 448 | "display_name": "Python 2", 449 | "language": "python", 450 | "name": "python2" 451 | }, 452 | "language_info": { 453 | "codemirror_mode": { 454 | "name": "ipython", 455 | "version": 2 456 | }, 457 | "file_extension": ".py", 458 | "mimetype": "text/x-python", 459 | "name": "python", 460 | "nbconvert_exporter": "python", 461 | "pygments_lexer": "ipython2", 462 | "version": "2.7.12" 463 | } 464 | }, 465 | "nbformat": 4, 466 | "nbformat_minor": 2 467 | } 468 | -------------------------------------------------------------------------------- /Newtons Method for Optimization/README.md: -------------------------------------------------------------------------------- 1 | # Purpose 2 | The jupyter note book "Newtons Method.ipynb" implements Newton's method from scratch to estimate the parameters in a simple linear regression by minimizing the sum of square error (SSE). The notebook contains the mathematical details of the Newton's method and shows implementation in python. 3 | 4 | # Data set 5 | I use the [IMDB movie data set from kaggle](https://www.kaggle.com/deepmatrix/imdb-5000-movie-dataset) and examine the relationship between IMDB rating and gross sales revenue using a simple linear regression model. For analysis purposes I clean the data set to only contain IMDB movie rating and gross sales revenue for US movies. 6 | 7 | # References 8 | [Siraj's video on Second Order Optimization](https://www.youtube.com/watch?v=UIFMLK2nj_w). 9 | -------------------------------------------------------------------------------- /Nonparametric Regression/Nonparametric Regression Note.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Nonparametric Regression/Nonparametric Regression Note.pdf -------------------------------------------------------------------------------- /Nonparametric Regression/README.md: -------------------------------------------------------------------------------- 1 | # Purpose 2 | 3 | Explain and implement nonparametric regressions from scratch. The "Nonparametric Regression.ipynb" notebook discusses how to estimate the nonparametric component of the model: Y = f(x) + e. Polynomial approximation, Gaussian kernel regression, and spline regression are discussed. 4 | 5 | # Dependencies 6 | * numpy 7 | * Matplotlib 8 | 9 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 10 | 11 | # Contribution ideas 12 | 13 | If anyone is interested in working further on improving this notebook, here are some ideas: 14 | * Application to real data with nonlinearities 15 | * Discuss and implement cross validation to choose the tuning parameters 16 | * Discuss and implement other estimation methods such as locally weighted regression 17 | 18 | # References 19 | [Chris McCormik Blog](http://mccormickml.com/2014/02/26/kernel-regression/) 20 | 21 | [Andrew Ng Notes](http://cs229.stanford.edu/notes/cs229-notes1.pdf) 22 | 23 | [Anders Video Tutorial](https://www.youtube.com/watch?v=ncF7ArjJFqM) 24 | 25 | 26 | -------------------------------------------------------------------------------- /Principal Component Analysis/Figures/2D PCA.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Principal Component Analysis/Figures/2D PCA.png -------------------------------------------------------------------------------- /Principal Component Analysis/Figures/PCA3Dto2D.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Principal Component Analysis/Figures/PCA3Dto2D.png -------------------------------------------------------------------------------- /Principal Component Analysis/Figures/PCAbasic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Principal Component Analysis/Figures/PCAbasic.png -------------------------------------------------------------------------------- /Principal Component Analysis/Figures/PCAcomponent.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Principal Component Analysis/Figures/PCAcomponent.png -------------------------------------------------------------------------------- /Principal Component Analysis/Figures/README.md: -------------------------------------------------------------------------------- 1 | Figures used in the PCA notebook. 2 | -------------------------------------------------------------------------------- /Principal Component Analysis/README.md: -------------------------------------------------------------------------------- 1 | ## Introduction 2 | The notebook "Principal Component Analysis.ipynb" introduces the theory, and intuition behind Principal Component Analysis (PCA) for the purpose of dimensionality reduction. It applies PCA to visualize a 8-dimensional data set onto 2D and 3D scatter plots. 3 | 4 | ## Data 5 | The data used for the application of PCA can be found [here](https://www.kaggle.com/uciml/pima-indians-diabetes-database) on kaggle. This data contains the number of pregnancies, glucose level, blood pressure, skin thickness, insulin level, body mass index (BMI), age, and a diabetes pedigree (measure of the likelihood of getting diabities based on their ancestor's history) on several women over the age of 21 from Pima indian heritage. The data set contains labels on whether a women in the data tested positive for diabities. 6 | 7 | ## Dependencies 8 | * numpy 9 | * Pandas 10 | * Matplotlib 11 | 12 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 13 | 14 | ## References 15 | [Siraj's video on Dimensionality Reduction](https://www.youtube.com/watch?v=jPmV3j1dAv4&t=281s) 16 | 17 | [Detailed answer about PCA on Stats Stackexchange](https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues) 18 | 19 | [PCA applied to IRIS data set](https://plot.ly/ipython-notebooks/principal-component-analysis/) 20 | 21 | [Practical guide to PCA in R and Python](https://www.analyticsvidhya.com/blog/2016/03/practical-guide-principal-component-analysis-python/) 22 | -------------------------------------------------------------------------------- /Principal Component Analysis/diabetes.csv: -------------------------------------------------------------------------------- 1 | Pregnancies,Glucose,BloodPressure,SkinThickness,Insulin,BMI,DiabetesPedigreeFunction,Age,Outcome 2 | 6,148,72,35,0,33.6,0.627,50,1 3 | 1,85,66,29,0,26.6,0.351,31,0 4 | 8,183,64,0,0,23.3,0.672,32,1 5 | 1,89,66,23,94,28.1,0.167,21,0 6 | 0,137,40,35,168,43.1,2.288,33,1 7 | 5,116,74,0,0,25.6,0.201,30,0 8 | 3,78,50,32,88,31,0.248,26,1 9 | 10,115,0,0,0,35.3,0.134,29,0 10 | 2,197,70,45,543,30.5,0.158,53,1 11 | 8,125,96,0,0,0,0.232,54,1 12 | 4,110,92,0,0,37.6,0.191,30,0 13 | 10,168,74,0,0,38,0.537,34,1 14 | 10,139,80,0,0,27.1,1.441,57,0 15 | 1,189,60,23,846,30.1,0.398,59,1 16 | 5,166,72,19,175,25.8,0.587,51,1 17 | 7,100,0,0,0,30,0.484,32,1 18 | 0,118,84,47,230,45.8,0.551,31,1 19 | 7,107,74,0,0,29.6,0.254,31,1 20 | 1,103,30,38,83,43.3,0.183,33,0 21 | 1,115,70,30,96,34.6,0.529,32,1 22 | 3,126,88,41,235,39.3,0.704,27,0 23 | 8,99,84,0,0,35.4,0.388,50,0 24 | 7,196,90,0,0,39.8,0.451,41,1 25 | 9,119,80,35,0,29,0.263,29,1 26 | 11,143,94,33,146,36.6,0.254,51,1 27 | 10,125,70,26,115,31.1,0.205,41,1 28 | 7,147,76,0,0,39.4,0.257,43,1 29 | 1,97,66,15,140,23.2,0.487,22,0 30 | 13,145,82,19,110,22.2,0.245,57,0 31 | 5,117,92,0,0,34.1,0.337,38,0 32 | 5,109,75,26,0,36,0.546,60,0 33 | 3,158,76,36,245,31.6,0.851,28,1 34 | 3,88,58,11,54,24.8,0.267,22,0 35 | 6,92,92,0,0,19.9,0.188,28,0 36 | 10,122,78,31,0,27.6,0.512,45,0 37 | 4,103,60,33,192,24,0.966,33,0 38 | 11,138,76,0,0,33.2,0.42,35,0 39 | 9,102,76,37,0,32.9,0.665,46,1 40 | 2,90,68,42,0,38.2,0.503,27,1 41 | 4,111,72,47,207,37.1,1.39,56,1 42 | 3,180,64,25,70,34,0.271,26,0 43 | 7,133,84,0,0,40.2,0.696,37,0 44 | 7,106,92,18,0,22.7,0.235,48,0 45 | 9,171,110,24,240,45.4,0.721,54,1 46 | 7,159,64,0,0,27.4,0.294,40,0 47 | 0,180,66,39,0,42,1.893,25,1 48 | 1,146,56,0,0,29.7,0.564,29,0 49 | 2,71,70,27,0,28,0.586,22,0 50 | 7,103,66,32,0,39.1,0.344,31,1 51 | 7,105,0,0,0,0,0.305,24,0 52 | 1,103,80,11,82,19.4,0.491,22,0 53 | 1,101,50,15,36,24.2,0.526,26,0 54 | 5,88,66,21,23,24.4,0.342,30,0 55 | 8,176,90,34,300,33.7,0.467,58,1 56 | 7,150,66,42,342,34.7,0.718,42,0 57 | 1,73,50,10,0,23,0.248,21,0 58 | 7,187,68,39,304,37.7,0.254,41,1 59 | 0,100,88,60,110,46.8,0.962,31,0 60 | 0,146,82,0,0,40.5,1.781,44,0 61 | 0,105,64,41,142,41.5,0.173,22,0 62 | 2,84,0,0,0,0,0.304,21,0 63 | 8,133,72,0,0,32.9,0.27,39,1 64 | 5,44,62,0,0,25,0.587,36,0 65 | 2,141,58,34,128,25.4,0.699,24,0 66 | 7,114,66,0,0,32.8,0.258,42,1 67 | 5,99,74,27,0,29,0.203,32,0 68 | 0,109,88,30,0,32.5,0.855,38,1 69 | 2,109,92,0,0,42.7,0.845,54,0 70 | 1,95,66,13,38,19.6,0.334,25,0 71 | 4,146,85,27,100,28.9,0.189,27,0 72 | 2,100,66,20,90,32.9,0.867,28,1 73 | 5,139,64,35,140,28.6,0.411,26,0 74 | 13,126,90,0,0,43.4,0.583,42,1 75 | 4,129,86,20,270,35.1,0.231,23,0 76 | 1,79,75,30,0,32,0.396,22,0 77 | 1,0,48,20,0,24.7,0.14,22,0 78 | 7,62,78,0,0,32.6,0.391,41,0 79 | 5,95,72,33,0,37.7,0.37,27,0 80 | 0,131,0,0,0,43.2,0.27,26,1 81 | 2,112,66,22,0,25,0.307,24,0 82 | 3,113,44,13,0,22.4,0.14,22,0 83 | 2,74,0,0,0,0,0.102,22,0 84 | 7,83,78,26,71,29.3,0.767,36,0 85 | 0,101,65,28,0,24.6,0.237,22,0 86 | 5,137,108,0,0,48.8,0.227,37,1 87 | 2,110,74,29,125,32.4,0.698,27,0 88 | 13,106,72,54,0,36.6,0.178,45,0 89 | 2,100,68,25,71,38.5,0.324,26,0 90 | 15,136,70,32,110,37.1,0.153,43,1 91 | 1,107,68,19,0,26.5,0.165,24,0 92 | 1,80,55,0,0,19.1,0.258,21,0 93 | 4,123,80,15,176,32,0.443,34,0 94 | 7,81,78,40,48,46.7,0.261,42,0 95 | 4,134,72,0,0,23.8,0.277,60,1 96 | 2,142,82,18,64,24.7,0.761,21,0 97 | 6,144,72,27,228,33.9,0.255,40,0 98 | 2,92,62,28,0,31.6,0.13,24,0 99 | 1,71,48,18,76,20.4,0.323,22,0 100 | 6,93,50,30,64,28.7,0.356,23,0 101 | 1,122,90,51,220,49.7,0.325,31,1 102 | 1,163,72,0,0,39,1.222,33,1 103 | 1,151,60,0,0,26.1,0.179,22,0 104 | 0,125,96,0,0,22.5,0.262,21,0 105 | 1,81,72,18,40,26.6,0.283,24,0 106 | 2,85,65,0,0,39.6,0.93,27,0 107 | 1,126,56,29,152,28.7,0.801,21,0 108 | 1,96,122,0,0,22.4,0.207,27,0 109 | 4,144,58,28,140,29.5,0.287,37,0 110 | 3,83,58,31,18,34.3,0.336,25,0 111 | 0,95,85,25,36,37.4,0.247,24,1 112 | 3,171,72,33,135,33.3,0.199,24,1 113 | 8,155,62,26,495,34,0.543,46,1 114 | 1,89,76,34,37,31.2,0.192,23,0 115 | 4,76,62,0,0,34,0.391,25,0 116 | 7,160,54,32,175,30.5,0.588,39,1 117 | 4,146,92,0,0,31.2,0.539,61,1 118 | 5,124,74,0,0,34,0.22,38,1 119 | 5,78,48,0,0,33.7,0.654,25,0 120 | 4,97,60,23,0,28.2,0.443,22,0 121 | 4,99,76,15,51,23.2,0.223,21,0 122 | 0,162,76,56,100,53.2,0.759,25,1 123 | 6,111,64,39,0,34.2,0.26,24,0 124 | 2,107,74,30,100,33.6,0.404,23,0 125 | 5,132,80,0,0,26.8,0.186,69,0 126 | 0,113,76,0,0,33.3,0.278,23,1 127 | 1,88,30,42,99,55,0.496,26,1 128 | 3,120,70,30,135,42.9,0.452,30,0 129 | 1,118,58,36,94,33.3,0.261,23,0 130 | 1,117,88,24,145,34.5,0.403,40,1 131 | 0,105,84,0,0,27.9,0.741,62,1 132 | 4,173,70,14,168,29.7,0.361,33,1 133 | 9,122,56,0,0,33.3,1.114,33,1 134 | 3,170,64,37,225,34.5,0.356,30,1 135 | 8,84,74,31,0,38.3,0.457,39,0 136 | 2,96,68,13,49,21.1,0.647,26,0 137 | 2,125,60,20,140,33.8,0.088,31,0 138 | 0,100,70,26,50,30.8,0.597,21,0 139 | 0,93,60,25,92,28.7,0.532,22,0 140 | 0,129,80,0,0,31.2,0.703,29,0 141 | 5,105,72,29,325,36.9,0.159,28,0 142 | 3,128,78,0,0,21.1,0.268,55,0 143 | 5,106,82,30,0,39.5,0.286,38,0 144 | 2,108,52,26,63,32.5,0.318,22,0 145 | 10,108,66,0,0,32.4,0.272,42,1 146 | 4,154,62,31,284,32.8,0.237,23,0 147 | 0,102,75,23,0,0,0.572,21,0 148 | 9,57,80,37,0,32.8,0.096,41,0 149 | 2,106,64,35,119,30.5,1.4,34,0 150 | 5,147,78,0,0,33.7,0.218,65,0 151 | 2,90,70,17,0,27.3,0.085,22,0 152 | 1,136,74,50,204,37.4,0.399,24,0 153 | 4,114,65,0,0,21.9,0.432,37,0 154 | 9,156,86,28,155,34.3,1.189,42,1 155 | 1,153,82,42,485,40.6,0.687,23,0 156 | 8,188,78,0,0,47.9,0.137,43,1 157 | 7,152,88,44,0,50,0.337,36,1 158 | 2,99,52,15,94,24.6,0.637,21,0 159 | 1,109,56,21,135,25.2,0.833,23,0 160 | 2,88,74,19,53,29,0.229,22,0 161 | 17,163,72,41,114,40.9,0.817,47,1 162 | 4,151,90,38,0,29.7,0.294,36,0 163 | 7,102,74,40,105,37.2,0.204,45,0 164 | 0,114,80,34,285,44.2,0.167,27,0 165 | 2,100,64,23,0,29.7,0.368,21,0 166 | 0,131,88,0,0,31.6,0.743,32,1 167 | 6,104,74,18,156,29.9,0.722,41,1 168 | 3,148,66,25,0,32.5,0.256,22,0 169 | 4,120,68,0,0,29.6,0.709,34,0 170 | 4,110,66,0,0,31.9,0.471,29,0 171 | 3,111,90,12,78,28.4,0.495,29,0 172 | 6,102,82,0,0,30.8,0.18,36,1 173 | 6,134,70,23,130,35.4,0.542,29,1 174 | 2,87,0,23,0,28.9,0.773,25,0 175 | 1,79,60,42,48,43.5,0.678,23,0 176 | 2,75,64,24,55,29.7,0.37,33,0 177 | 8,179,72,42,130,32.7,0.719,36,1 178 | 6,85,78,0,0,31.2,0.382,42,0 179 | 0,129,110,46,130,67.1,0.319,26,1 180 | 5,143,78,0,0,45,0.19,47,0 181 | 5,130,82,0,0,39.1,0.956,37,1 182 | 6,87,80,0,0,23.2,0.084,32,0 183 | 0,119,64,18,92,34.9,0.725,23,0 184 | 1,0,74,20,23,27.7,0.299,21,0 185 | 5,73,60,0,0,26.8,0.268,27,0 186 | 4,141,74,0,0,27.6,0.244,40,0 187 | 7,194,68,28,0,35.9,0.745,41,1 188 | 8,181,68,36,495,30.1,0.615,60,1 189 | 1,128,98,41,58,32,1.321,33,1 190 | 8,109,76,39,114,27.9,0.64,31,1 191 | 5,139,80,35,160,31.6,0.361,25,1 192 | 3,111,62,0,0,22.6,0.142,21,0 193 | 9,123,70,44,94,33.1,0.374,40,0 194 | 7,159,66,0,0,30.4,0.383,36,1 195 | 11,135,0,0,0,52.3,0.578,40,1 196 | 8,85,55,20,0,24.4,0.136,42,0 197 | 5,158,84,41,210,39.4,0.395,29,1 198 | 1,105,58,0,0,24.3,0.187,21,0 199 | 3,107,62,13,48,22.9,0.678,23,1 200 | 4,109,64,44,99,34.8,0.905,26,1 201 | 4,148,60,27,318,30.9,0.15,29,1 202 | 0,113,80,16,0,31,0.874,21,0 203 | 1,138,82,0,0,40.1,0.236,28,0 204 | 0,108,68,20,0,27.3,0.787,32,0 205 | 2,99,70,16,44,20.4,0.235,27,0 206 | 6,103,72,32,190,37.7,0.324,55,0 207 | 5,111,72,28,0,23.9,0.407,27,0 208 | 8,196,76,29,280,37.5,0.605,57,1 209 | 5,162,104,0,0,37.7,0.151,52,1 210 | 1,96,64,27,87,33.2,0.289,21,0 211 | 7,184,84,33,0,35.5,0.355,41,1 212 | 2,81,60,22,0,27.7,0.29,25,0 213 | 0,147,85,54,0,42.8,0.375,24,0 214 | 7,179,95,31,0,34.2,0.164,60,0 215 | 0,140,65,26,130,42.6,0.431,24,1 216 | 9,112,82,32,175,34.2,0.26,36,1 217 | 12,151,70,40,271,41.8,0.742,38,1 218 | 5,109,62,41,129,35.8,0.514,25,1 219 | 6,125,68,30,120,30,0.464,32,0 220 | 5,85,74,22,0,29,1.224,32,1 221 | 5,112,66,0,0,37.8,0.261,41,1 222 | 0,177,60,29,478,34.6,1.072,21,1 223 | 2,158,90,0,0,31.6,0.805,66,1 224 | 7,119,0,0,0,25.2,0.209,37,0 225 | 7,142,60,33,190,28.8,0.687,61,0 226 | 1,100,66,15,56,23.6,0.666,26,0 227 | 1,87,78,27,32,34.6,0.101,22,0 228 | 0,101,76,0,0,35.7,0.198,26,0 229 | 3,162,52,38,0,37.2,0.652,24,1 230 | 4,197,70,39,744,36.7,2.329,31,0 231 | 0,117,80,31,53,45.2,0.089,24,0 232 | 4,142,86,0,0,44,0.645,22,1 233 | 6,134,80,37,370,46.2,0.238,46,1 234 | 1,79,80,25,37,25.4,0.583,22,0 235 | 4,122,68,0,0,35,0.394,29,0 236 | 3,74,68,28,45,29.7,0.293,23,0 237 | 4,171,72,0,0,43.6,0.479,26,1 238 | 7,181,84,21,192,35.9,0.586,51,1 239 | 0,179,90,27,0,44.1,0.686,23,1 240 | 9,164,84,21,0,30.8,0.831,32,1 241 | 0,104,76,0,0,18.4,0.582,27,0 242 | 1,91,64,24,0,29.2,0.192,21,0 243 | 4,91,70,32,88,33.1,0.446,22,0 244 | 3,139,54,0,0,25.6,0.402,22,1 245 | 6,119,50,22,176,27.1,1.318,33,1 246 | 2,146,76,35,194,38.2,0.329,29,0 247 | 9,184,85,15,0,30,1.213,49,1 248 | 10,122,68,0,0,31.2,0.258,41,0 249 | 0,165,90,33,680,52.3,0.427,23,0 250 | 9,124,70,33,402,35.4,0.282,34,0 251 | 1,111,86,19,0,30.1,0.143,23,0 252 | 9,106,52,0,0,31.2,0.38,42,0 253 | 2,129,84,0,0,28,0.284,27,0 254 | 2,90,80,14,55,24.4,0.249,24,0 255 | 0,86,68,32,0,35.8,0.238,25,0 256 | 12,92,62,7,258,27.6,0.926,44,1 257 | 1,113,64,35,0,33.6,0.543,21,1 258 | 3,111,56,39,0,30.1,0.557,30,0 259 | 2,114,68,22,0,28.7,0.092,25,0 260 | 1,193,50,16,375,25.9,0.655,24,0 261 | 11,155,76,28,150,33.3,1.353,51,1 262 | 3,191,68,15,130,30.9,0.299,34,0 263 | 3,141,0,0,0,30,0.761,27,1 264 | 4,95,70,32,0,32.1,0.612,24,0 265 | 3,142,80,15,0,32.4,0.2,63,0 266 | 4,123,62,0,0,32,0.226,35,1 267 | 5,96,74,18,67,33.6,0.997,43,0 268 | 0,138,0,0,0,36.3,0.933,25,1 269 | 2,128,64,42,0,40,1.101,24,0 270 | 0,102,52,0,0,25.1,0.078,21,0 271 | 2,146,0,0,0,27.5,0.24,28,1 272 | 10,101,86,37,0,45.6,1.136,38,1 273 | 2,108,62,32,56,25.2,0.128,21,0 274 | 3,122,78,0,0,23,0.254,40,0 275 | 1,71,78,50,45,33.2,0.422,21,0 276 | 13,106,70,0,0,34.2,0.251,52,0 277 | 2,100,70,52,57,40.5,0.677,25,0 278 | 7,106,60,24,0,26.5,0.296,29,1 279 | 0,104,64,23,116,27.8,0.454,23,0 280 | 5,114,74,0,0,24.9,0.744,57,0 281 | 2,108,62,10,278,25.3,0.881,22,0 282 | 0,146,70,0,0,37.9,0.334,28,1 283 | 10,129,76,28,122,35.9,0.28,39,0 284 | 7,133,88,15,155,32.4,0.262,37,0 285 | 7,161,86,0,0,30.4,0.165,47,1 286 | 2,108,80,0,0,27,0.259,52,1 287 | 7,136,74,26,135,26,0.647,51,0 288 | 5,155,84,44,545,38.7,0.619,34,0 289 | 1,119,86,39,220,45.6,0.808,29,1 290 | 4,96,56,17,49,20.8,0.34,26,0 291 | 5,108,72,43,75,36.1,0.263,33,0 292 | 0,78,88,29,40,36.9,0.434,21,0 293 | 0,107,62,30,74,36.6,0.757,25,1 294 | 2,128,78,37,182,43.3,1.224,31,1 295 | 1,128,48,45,194,40.5,0.613,24,1 296 | 0,161,50,0,0,21.9,0.254,65,0 297 | 6,151,62,31,120,35.5,0.692,28,0 298 | 2,146,70,38,360,28,0.337,29,1 299 | 0,126,84,29,215,30.7,0.52,24,0 300 | 14,100,78,25,184,36.6,0.412,46,1 301 | 8,112,72,0,0,23.6,0.84,58,0 302 | 0,167,0,0,0,32.3,0.839,30,1 303 | 2,144,58,33,135,31.6,0.422,25,1 304 | 5,77,82,41,42,35.8,0.156,35,0 305 | 5,115,98,0,0,52.9,0.209,28,1 306 | 3,150,76,0,0,21,0.207,37,0 307 | 2,120,76,37,105,39.7,0.215,29,0 308 | 10,161,68,23,132,25.5,0.326,47,1 309 | 0,137,68,14,148,24.8,0.143,21,0 310 | 0,128,68,19,180,30.5,1.391,25,1 311 | 2,124,68,28,205,32.9,0.875,30,1 312 | 6,80,66,30,0,26.2,0.313,41,0 313 | 0,106,70,37,148,39.4,0.605,22,0 314 | 2,155,74,17,96,26.6,0.433,27,1 315 | 3,113,50,10,85,29.5,0.626,25,0 316 | 7,109,80,31,0,35.9,1.127,43,1 317 | 2,112,68,22,94,34.1,0.315,26,0 318 | 3,99,80,11,64,19.3,0.284,30,0 319 | 3,182,74,0,0,30.5,0.345,29,1 320 | 3,115,66,39,140,38.1,0.15,28,0 321 | 6,194,78,0,0,23.5,0.129,59,1 322 | 4,129,60,12,231,27.5,0.527,31,0 323 | 3,112,74,30,0,31.6,0.197,25,1 324 | 0,124,70,20,0,27.4,0.254,36,1 325 | 13,152,90,33,29,26.8,0.731,43,1 326 | 2,112,75,32,0,35.7,0.148,21,0 327 | 1,157,72,21,168,25.6,0.123,24,0 328 | 1,122,64,32,156,35.1,0.692,30,1 329 | 10,179,70,0,0,35.1,0.2,37,0 330 | 2,102,86,36,120,45.5,0.127,23,1 331 | 6,105,70,32,68,30.8,0.122,37,0 332 | 8,118,72,19,0,23.1,1.476,46,0 333 | 2,87,58,16,52,32.7,0.166,25,0 334 | 1,180,0,0,0,43.3,0.282,41,1 335 | 12,106,80,0,0,23.6,0.137,44,0 336 | 1,95,60,18,58,23.9,0.26,22,0 337 | 0,165,76,43,255,47.9,0.259,26,0 338 | 0,117,0,0,0,33.8,0.932,44,0 339 | 5,115,76,0,0,31.2,0.343,44,1 340 | 9,152,78,34,171,34.2,0.893,33,1 341 | 7,178,84,0,0,39.9,0.331,41,1 342 | 1,130,70,13,105,25.9,0.472,22,0 343 | 1,95,74,21,73,25.9,0.673,36,0 344 | 1,0,68,35,0,32,0.389,22,0 345 | 5,122,86,0,0,34.7,0.29,33,0 346 | 8,95,72,0,0,36.8,0.485,57,0 347 | 8,126,88,36,108,38.5,0.349,49,0 348 | 1,139,46,19,83,28.7,0.654,22,0 349 | 3,116,0,0,0,23.5,0.187,23,0 350 | 3,99,62,19,74,21.8,0.279,26,0 351 | 5,0,80,32,0,41,0.346,37,1 352 | 4,92,80,0,0,42.2,0.237,29,0 353 | 4,137,84,0,0,31.2,0.252,30,0 354 | 3,61,82,28,0,34.4,0.243,46,0 355 | 1,90,62,12,43,27.2,0.58,24,0 356 | 3,90,78,0,0,42.7,0.559,21,0 357 | 9,165,88,0,0,30.4,0.302,49,1 358 | 1,125,50,40,167,33.3,0.962,28,1 359 | 13,129,0,30,0,39.9,0.569,44,1 360 | 12,88,74,40,54,35.3,0.378,48,0 361 | 1,196,76,36,249,36.5,0.875,29,1 362 | 5,189,64,33,325,31.2,0.583,29,1 363 | 5,158,70,0,0,29.8,0.207,63,0 364 | 5,103,108,37,0,39.2,0.305,65,0 365 | 4,146,78,0,0,38.5,0.52,67,1 366 | 4,147,74,25,293,34.9,0.385,30,0 367 | 5,99,54,28,83,34,0.499,30,0 368 | 6,124,72,0,0,27.6,0.368,29,1 369 | 0,101,64,17,0,21,0.252,21,0 370 | 3,81,86,16,66,27.5,0.306,22,0 371 | 1,133,102,28,140,32.8,0.234,45,1 372 | 3,173,82,48,465,38.4,2.137,25,1 373 | 0,118,64,23,89,0,1.731,21,0 374 | 0,84,64,22,66,35.8,0.545,21,0 375 | 2,105,58,40,94,34.9,0.225,25,0 376 | 2,122,52,43,158,36.2,0.816,28,0 377 | 12,140,82,43,325,39.2,0.528,58,1 378 | 0,98,82,15,84,25.2,0.299,22,0 379 | 1,87,60,37,75,37.2,0.509,22,0 380 | 4,156,75,0,0,48.3,0.238,32,1 381 | 0,93,100,39,72,43.4,1.021,35,0 382 | 1,107,72,30,82,30.8,0.821,24,0 383 | 0,105,68,22,0,20,0.236,22,0 384 | 1,109,60,8,182,25.4,0.947,21,0 385 | 1,90,62,18,59,25.1,1.268,25,0 386 | 1,125,70,24,110,24.3,0.221,25,0 387 | 1,119,54,13,50,22.3,0.205,24,0 388 | 5,116,74,29,0,32.3,0.66,35,1 389 | 8,105,100,36,0,43.3,0.239,45,1 390 | 5,144,82,26,285,32,0.452,58,1 391 | 3,100,68,23,81,31.6,0.949,28,0 392 | 1,100,66,29,196,32,0.444,42,0 393 | 5,166,76,0,0,45.7,0.34,27,1 394 | 1,131,64,14,415,23.7,0.389,21,0 395 | 4,116,72,12,87,22.1,0.463,37,0 396 | 4,158,78,0,0,32.9,0.803,31,1 397 | 2,127,58,24,275,27.7,1.6,25,0 398 | 3,96,56,34,115,24.7,0.944,39,0 399 | 0,131,66,40,0,34.3,0.196,22,1 400 | 3,82,70,0,0,21.1,0.389,25,0 401 | 3,193,70,31,0,34.9,0.241,25,1 402 | 4,95,64,0,0,32,0.161,31,1 403 | 6,137,61,0,0,24.2,0.151,55,0 404 | 5,136,84,41,88,35,0.286,35,1 405 | 9,72,78,25,0,31.6,0.28,38,0 406 | 5,168,64,0,0,32.9,0.135,41,1 407 | 2,123,48,32,165,42.1,0.52,26,0 408 | 4,115,72,0,0,28.9,0.376,46,1 409 | 0,101,62,0,0,21.9,0.336,25,0 410 | 8,197,74,0,0,25.9,1.191,39,1 411 | 1,172,68,49,579,42.4,0.702,28,1 412 | 6,102,90,39,0,35.7,0.674,28,0 413 | 1,112,72,30,176,34.4,0.528,25,0 414 | 1,143,84,23,310,42.4,1.076,22,0 415 | 1,143,74,22,61,26.2,0.256,21,0 416 | 0,138,60,35,167,34.6,0.534,21,1 417 | 3,173,84,33,474,35.7,0.258,22,1 418 | 1,97,68,21,0,27.2,1.095,22,0 419 | 4,144,82,32,0,38.5,0.554,37,1 420 | 1,83,68,0,0,18.2,0.624,27,0 421 | 3,129,64,29,115,26.4,0.219,28,1 422 | 1,119,88,41,170,45.3,0.507,26,0 423 | 2,94,68,18,76,26,0.561,21,0 424 | 0,102,64,46,78,40.6,0.496,21,0 425 | 2,115,64,22,0,30.8,0.421,21,0 426 | 8,151,78,32,210,42.9,0.516,36,1 427 | 4,184,78,39,277,37,0.264,31,1 428 | 0,94,0,0,0,0,0.256,25,0 429 | 1,181,64,30,180,34.1,0.328,38,1 430 | 0,135,94,46,145,40.6,0.284,26,0 431 | 1,95,82,25,180,35,0.233,43,1 432 | 2,99,0,0,0,22.2,0.108,23,0 433 | 3,89,74,16,85,30.4,0.551,38,0 434 | 1,80,74,11,60,30,0.527,22,0 435 | 2,139,75,0,0,25.6,0.167,29,0 436 | 1,90,68,8,0,24.5,1.138,36,0 437 | 0,141,0,0,0,42.4,0.205,29,1 438 | 12,140,85,33,0,37.4,0.244,41,0 439 | 5,147,75,0,0,29.9,0.434,28,0 440 | 1,97,70,15,0,18.2,0.147,21,0 441 | 6,107,88,0,0,36.8,0.727,31,0 442 | 0,189,104,25,0,34.3,0.435,41,1 443 | 2,83,66,23,50,32.2,0.497,22,0 444 | 4,117,64,27,120,33.2,0.23,24,0 445 | 8,108,70,0,0,30.5,0.955,33,1 446 | 4,117,62,12,0,29.7,0.38,30,1 447 | 0,180,78,63,14,59.4,2.42,25,1 448 | 1,100,72,12,70,25.3,0.658,28,0 449 | 0,95,80,45,92,36.5,0.33,26,0 450 | 0,104,64,37,64,33.6,0.51,22,1 451 | 0,120,74,18,63,30.5,0.285,26,0 452 | 1,82,64,13,95,21.2,0.415,23,0 453 | 2,134,70,0,0,28.9,0.542,23,1 454 | 0,91,68,32,210,39.9,0.381,25,0 455 | 2,119,0,0,0,19.6,0.832,72,0 456 | 2,100,54,28,105,37.8,0.498,24,0 457 | 14,175,62,30,0,33.6,0.212,38,1 458 | 1,135,54,0,0,26.7,0.687,62,0 459 | 5,86,68,28,71,30.2,0.364,24,0 460 | 10,148,84,48,237,37.6,1.001,51,1 461 | 9,134,74,33,60,25.9,0.46,81,0 462 | 9,120,72,22,56,20.8,0.733,48,0 463 | 1,71,62,0,0,21.8,0.416,26,0 464 | 8,74,70,40,49,35.3,0.705,39,0 465 | 5,88,78,30,0,27.6,0.258,37,0 466 | 10,115,98,0,0,24,1.022,34,0 467 | 0,124,56,13,105,21.8,0.452,21,0 468 | 0,74,52,10,36,27.8,0.269,22,0 469 | 0,97,64,36,100,36.8,0.6,25,0 470 | 8,120,0,0,0,30,0.183,38,1 471 | 6,154,78,41,140,46.1,0.571,27,0 472 | 1,144,82,40,0,41.3,0.607,28,0 473 | 0,137,70,38,0,33.2,0.17,22,0 474 | 0,119,66,27,0,38.8,0.259,22,0 475 | 7,136,90,0,0,29.9,0.21,50,0 476 | 4,114,64,0,0,28.9,0.126,24,0 477 | 0,137,84,27,0,27.3,0.231,59,0 478 | 2,105,80,45,191,33.7,0.711,29,1 479 | 7,114,76,17,110,23.8,0.466,31,0 480 | 8,126,74,38,75,25.9,0.162,39,0 481 | 4,132,86,31,0,28,0.419,63,0 482 | 3,158,70,30,328,35.5,0.344,35,1 483 | 0,123,88,37,0,35.2,0.197,29,0 484 | 4,85,58,22,49,27.8,0.306,28,0 485 | 0,84,82,31,125,38.2,0.233,23,0 486 | 0,145,0,0,0,44.2,0.63,31,1 487 | 0,135,68,42,250,42.3,0.365,24,1 488 | 1,139,62,41,480,40.7,0.536,21,0 489 | 0,173,78,32,265,46.5,1.159,58,0 490 | 4,99,72,17,0,25.6,0.294,28,0 491 | 8,194,80,0,0,26.1,0.551,67,0 492 | 2,83,65,28,66,36.8,0.629,24,0 493 | 2,89,90,30,0,33.5,0.292,42,0 494 | 4,99,68,38,0,32.8,0.145,33,0 495 | 4,125,70,18,122,28.9,1.144,45,1 496 | 3,80,0,0,0,0,0.174,22,0 497 | 6,166,74,0,0,26.6,0.304,66,0 498 | 5,110,68,0,0,26,0.292,30,0 499 | 2,81,72,15,76,30.1,0.547,25,0 500 | 7,195,70,33,145,25.1,0.163,55,1 501 | 6,154,74,32,193,29.3,0.839,39,0 502 | 2,117,90,19,71,25.2,0.313,21,0 503 | 3,84,72,32,0,37.2,0.267,28,0 504 | 6,0,68,41,0,39,0.727,41,1 505 | 7,94,64,25,79,33.3,0.738,41,0 506 | 3,96,78,39,0,37.3,0.238,40,0 507 | 10,75,82,0,0,33.3,0.263,38,0 508 | 0,180,90,26,90,36.5,0.314,35,1 509 | 1,130,60,23,170,28.6,0.692,21,0 510 | 2,84,50,23,76,30.4,0.968,21,0 511 | 8,120,78,0,0,25,0.409,64,0 512 | 12,84,72,31,0,29.7,0.297,46,1 513 | 0,139,62,17,210,22.1,0.207,21,0 514 | 9,91,68,0,0,24.2,0.2,58,0 515 | 2,91,62,0,0,27.3,0.525,22,0 516 | 3,99,54,19,86,25.6,0.154,24,0 517 | 3,163,70,18,105,31.6,0.268,28,1 518 | 9,145,88,34,165,30.3,0.771,53,1 519 | 7,125,86,0,0,37.6,0.304,51,0 520 | 13,76,60,0,0,32.8,0.18,41,0 521 | 6,129,90,7,326,19.6,0.582,60,0 522 | 2,68,70,32,66,25,0.187,25,0 523 | 3,124,80,33,130,33.2,0.305,26,0 524 | 6,114,0,0,0,0,0.189,26,0 525 | 9,130,70,0,0,34.2,0.652,45,1 526 | 3,125,58,0,0,31.6,0.151,24,0 527 | 3,87,60,18,0,21.8,0.444,21,0 528 | 1,97,64,19,82,18.2,0.299,21,0 529 | 3,116,74,15,105,26.3,0.107,24,0 530 | 0,117,66,31,188,30.8,0.493,22,0 531 | 0,111,65,0,0,24.6,0.66,31,0 532 | 2,122,60,18,106,29.8,0.717,22,0 533 | 0,107,76,0,0,45.3,0.686,24,0 534 | 1,86,66,52,65,41.3,0.917,29,0 535 | 6,91,0,0,0,29.8,0.501,31,0 536 | 1,77,56,30,56,33.3,1.251,24,0 537 | 4,132,0,0,0,32.9,0.302,23,1 538 | 0,105,90,0,0,29.6,0.197,46,0 539 | 0,57,60,0,0,21.7,0.735,67,0 540 | 0,127,80,37,210,36.3,0.804,23,0 541 | 3,129,92,49,155,36.4,0.968,32,1 542 | 8,100,74,40,215,39.4,0.661,43,1 543 | 3,128,72,25,190,32.4,0.549,27,1 544 | 10,90,85,32,0,34.9,0.825,56,1 545 | 4,84,90,23,56,39.5,0.159,25,0 546 | 1,88,78,29,76,32,0.365,29,0 547 | 8,186,90,35,225,34.5,0.423,37,1 548 | 5,187,76,27,207,43.6,1.034,53,1 549 | 4,131,68,21,166,33.1,0.16,28,0 550 | 1,164,82,43,67,32.8,0.341,50,0 551 | 4,189,110,31,0,28.5,0.68,37,0 552 | 1,116,70,28,0,27.4,0.204,21,0 553 | 3,84,68,30,106,31.9,0.591,25,0 554 | 6,114,88,0,0,27.8,0.247,66,0 555 | 1,88,62,24,44,29.9,0.422,23,0 556 | 1,84,64,23,115,36.9,0.471,28,0 557 | 7,124,70,33,215,25.5,0.161,37,0 558 | 1,97,70,40,0,38.1,0.218,30,0 559 | 8,110,76,0,0,27.8,0.237,58,0 560 | 11,103,68,40,0,46.2,0.126,42,0 561 | 11,85,74,0,0,30.1,0.3,35,0 562 | 6,125,76,0,0,33.8,0.121,54,1 563 | 0,198,66,32,274,41.3,0.502,28,1 564 | 1,87,68,34,77,37.6,0.401,24,0 565 | 6,99,60,19,54,26.9,0.497,32,0 566 | 0,91,80,0,0,32.4,0.601,27,0 567 | 2,95,54,14,88,26.1,0.748,22,0 568 | 1,99,72,30,18,38.6,0.412,21,0 569 | 6,92,62,32,126,32,0.085,46,0 570 | 4,154,72,29,126,31.3,0.338,37,0 571 | 0,121,66,30,165,34.3,0.203,33,1 572 | 3,78,70,0,0,32.5,0.27,39,0 573 | 2,130,96,0,0,22.6,0.268,21,0 574 | 3,111,58,31,44,29.5,0.43,22,0 575 | 2,98,60,17,120,34.7,0.198,22,0 576 | 1,143,86,30,330,30.1,0.892,23,0 577 | 1,119,44,47,63,35.5,0.28,25,0 578 | 6,108,44,20,130,24,0.813,35,0 579 | 2,118,80,0,0,42.9,0.693,21,1 580 | 10,133,68,0,0,27,0.245,36,0 581 | 2,197,70,99,0,34.7,0.575,62,1 582 | 0,151,90,46,0,42.1,0.371,21,1 583 | 6,109,60,27,0,25,0.206,27,0 584 | 12,121,78,17,0,26.5,0.259,62,0 585 | 8,100,76,0,0,38.7,0.19,42,0 586 | 8,124,76,24,600,28.7,0.687,52,1 587 | 1,93,56,11,0,22.5,0.417,22,0 588 | 8,143,66,0,0,34.9,0.129,41,1 589 | 6,103,66,0,0,24.3,0.249,29,0 590 | 3,176,86,27,156,33.3,1.154,52,1 591 | 0,73,0,0,0,21.1,0.342,25,0 592 | 11,111,84,40,0,46.8,0.925,45,1 593 | 2,112,78,50,140,39.4,0.175,24,0 594 | 3,132,80,0,0,34.4,0.402,44,1 595 | 2,82,52,22,115,28.5,1.699,25,0 596 | 6,123,72,45,230,33.6,0.733,34,0 597 | 0,188,82,14,185,32,0.682,22,1 598 | 0,67,76,0,0,45.3,0.194,46,0 599 | 1,89,24,19,25,27.8,0.559,21,0 600 | 1,173,74,0,0,36.8,0.088,38,1 601 | 1,109,38,18,120,23.1,0.407,26,0 602 | 1,108,88,19,0,27.1,0.4,24,0 603 | 6,96,0,0,0,23.7,0.19,28,0 604 | 1,124,74,36,0,27.8,0.1,30,0 605 | 7,150,78,29,126,35.2,0.692,54,1 606 | 4,183,0,0,0,28.4,0.212,36,1 607 | 1,124,60,32,0,35.8,0.514,21,0 608 | 1,181,78,42,293,40,1.258,22,1 609 | 1,92,62,25,41,19.5,0.482,25,0 610 | 0,152,82,39,272,41.5,0.27,27,0 611 | 1,111,62,13,182,24,0.138,23,0 612 | 3,106,54,21,158,30.9,0.292,24,0 613 | 3,174,58,22,194,32.9,0.593,36,1 614 | 7,168,88,42,321,38.2,0.787,40,1 615 | 6,105,80,28,0,32.5,0.878,26,0 616 | 11,138,74,26,144,36.1,0.557,50,1 617 | 3,106,72,0,0,25.8,0.207,27,0 618 | 6,117,96,0,0,28.7,0.157,30,0 619 | 2,68,62,13,15,20.1,0.257,23,0 620 | 9,112,82,24,0,28.2,1.282,50,1 621 | 0,119,0,0,0,32.4,0.141,24,1 622 | 2,112,86,42,160,38.4,0.246,28,0 623 | 2,92,76,20,0,24.2,1.698,28,0 624 | 6,183,94,0,0,40.8,1.461,45,0 625 | 0,94,70,27,115,43.5,0.347,21,0 626 | 2,108,64,0,0,30.8,0.158,21,0 627 | 4,90,88,47,54,37.7,0.362,29,0 628 | 0,125,68,0,0,24.7,0.206,21,0 629 | 0,132,78,0,0,32.4,0.393,21,0 630 | 5,128,80,0,0,34.6,0.144,45,0 631 | 4,94,65,22,0,24.7,0.148,21,0 632 | 7,114,64,0,0,27.4,0.732,34,1 633 | 0,102,78,40,90,34.5,0.238,24,0 634 | 2,111,60,0,0,26.2,0.343,23,0 635 | 1,128,82,17,183,27.5,0.115,22,0 636 | 10,92,62,0,0,25.9,0.167,31,0 637 | 13,104,72,0,0,31.2,0.465,38,1 638 | 5,104,74,0,0,28.8,0.153,48,0 639 | 2,94,76,18,66,31.6,0.649,23,0 640 | 7,97,76,32,91,40.9,0.871,32,1 641 | 1,100,74,12,46,19.5,0.149,28,0 642 | 0,102,86,17,105,29.3,0.695,27,0 643 | 4,128,70,0,0,34.3,0.303,24,0 644 | 6,147,80,0,0,29.5,0.178,50,1 645 | 4,90,0,0,0,28,0.61,31,0 646 | 3,103,72,30,152,27.6,0.73,27,0 647 | 2,157,74,35,440,39.4,0.134,30,0 648 | 1,167,74,17,144,23.4,0.447,33,1 649 | 0,179,50,36,159,37.8,0.455,22,1 650 | 11,136,84,35,130,28.3,0.26,42,1 651 | 0,107,60,25,0,26.4,0.133,23,0 652 | 1,91,54,25,100,25.2,0.234,23,0 653 | 1,117,60,23,106,33.8,0.466,27,0 654 | 5,123,74,40,77,34.1,0.269,28,0 655 | 2,120,54,0,0,26.8,0.455,27,0 656 | 1,106,70,28,135,34.2,0.142,22,0 657 | 2,155,52,27,540,38.7,0.24,25,1 658 | 2,101,58,35,90,21.8,0.155,22,0 659 | 1,120,80,48,200,38.9,1.162,41,0 660 | 11,127,106,0,0,39,0.19,51,0 661 | 3,80,82,31,70,34.2,1.292,27,1 662 | 10,162,84,0,0,27.7,0.182,54,0 663 | 1,199,76,43,0,42.9,1.394,22,1 664 | 8,167,106,46,231,37.6,0.165,43,1 665 | 9,145,80,46,130,37.9,0.637,40,1 666 | 6,115,60,39,0,33.7,0.245,40,1 667 | 1,112,80,45,132,34.8,0.217,24,0 668 | 4,145,82,18,0,32.5,0.235,70,1 669 | 10,111,70,27,0,27.5,0.141,40,1 670 | 6,98,58,33,190,34,0.43,43,0 671 | 9,154,78,30,100,30.9,0.164,45,0 672 | 6,165,68,26,168,33.6,0.631,49,0 673 | 1,99,58,10,0,25.4,0.551,21,0 674 | 10,68,106,23,49,35.5,0.285,47,0 675 | 3,123,100,35,240,57.3,0.88,22,0 676 | 8,91,82,0,0,35.6,0.587,68,0 677 | 6,195,70,0,0,30.9,0.328,31,1 678 | 9,156,86,0,0,24.8,0.23,53,1 679 | 0,93,60,0,0,35.3,0.263,25,0 680 | 3,121,52,0,0,36,0.127,25,1 681 | 2,101,58,17,265,24.2,0.614,23,0 682 | 2,56,56,28,45,24.2,0.332,22,0 683 | 0,162,76,36,0,49.6,0.364,26,1 684 | 0,95,64,39,105,44.6,0.366,22,0 685 | 4,125,80,0,0,32.3,0.536,27,1 686 | 5,136,82,0,0,0,0.64,69,0 687 | 2,129,74,26,205,33.2,0.591,25,0 688 | 3,130,64,0,0,23.1,0.314,22,0 689 | 1,107,50,19,0,28.3,0.181,29,0 690 | 1,140,74,26,180,24.1,0.828,23,0 691 | 1,144,82,46,180,46.1,0.335,46,1 692 | 8,107,80,0,0,24.6,0.856,34,0 693 | 13,158,114,0,0,42.3,0.257,44,1 694 | 2,121,70,32,95,39.1,0.886,23,0 695 | 7,129,68,49,125,38.5,0.439,43,1 696 | 2,90,60,0,0,23.5,0.191,25,0 697 | 7,142,90,24,480,30.4,0.128,43,1 698 | 3,169,74,19,125,29.9,0.268,31,1 699 | 0,99,0,0,0,25,0.253,22,0 700 | 4,127,88,11,155,34.5,0.598,28,0 701 | 4,118,70,0,0,44.5,0.904,26,0 702 | 2,122,76,27,200,35.9,0.483,26,0 703 | 6,125,78,31,0,27.6,0.565,49,1 704 | 1,168,88,29,0,35,0.905,52,1 705 | 2,129,0,0,0,38.5,0.304,41,0 706 | 4,110,76,20,100,28.4,0.118,27,0 707 | 6,80,80,36,0,39.8,0.177,28,0 708 | 10,115,0,0,0,0,0.261,30,1 709 | 2,127,46,21,335,34.4,0.176,22,0 710 | 9,164,78,0,0,32.8,0.148,45,1 711 | 2,93,64,32,160,38,0.674,23,1 712 | 3,158,64,13,387,31.2,0.295,24,0 713 | 5,126,78,27,22,29.6,0.439,40,0 714 | 10,129,62,36,0,41.2,0.441,38,1 715 | 0,134,58,20,291,26.4,0.352,21,0 716 | 3,102,74,0,0,29.5,0.121,32,0 717 | 7,187,50,33,392,33.9,0.826,34,1 718 | 3,173,78,39,185,33.8,0.97,31,1 719 | 10,94,72,18,0,23.1,0.595,56,0 720 | 1,108,60,46,178,35.5,0.415,24,0 721 | 5,97,76,27,0,35.6,0.378,52,1 722 | 4,83,86,19,0,29.3,0.317,34,0 723 | 1,114,66,36,200,38.1,0.289,21,0 724 | 1,149,68,29,127,29.3,0.349,42,1 725 | 5,117,86,30,105,39.1,0.251,42,0 726 | 1,111,94,0,0,32.8,0.265,45,0 727 | 4,112,78,40,0,39.4,0.236,38,0 728 | 1,116,78,29,180,36.1,0.496,25,0 729 | 0,141,84,26,0,32.4,0.433,22,0 730 | 2,175,88,0,0,22.9,0.326,22,0 731 | 2,92,52,0,0,30.1,0.141,22,0 732 | 3,130,78,23,79,28.4,0.323,34,1 733 | 8,120,86,0,0,28.4,0.259,22,1 734 | 2,174,88,37,120,44.5,0.646,24,1 735 | 2,106,56,27,165,29,0.426,22,0 736 | 2,105,75,0,0,23.3,0.56,53,0 737 | 4,95,60,32,0,35.4,0.284,28,0 738 | 0,126,86,27,120,27.4,0.515,21,0 739 | 8,65,72,23,0,32,0.6,42,0 740 | 2,99,60,17,160,36.6,0.453,21,0 741 | 1,102,74,0,0,39.5,0.293,42,1 742 | 11,120,80,37,150,42.3,0.785,48,1 743 | 3,102,44,20,94,30.8,0.4,26,0 744 | 1,109,58,18,116,28.5,0.219,22,0 745 | 9,140,94,0,0,32.7,0.734,45,1 746 | 13,153,88,37,140,40.6,1.174,39,0 747 | 12,100,84,33,105,30,0.488,46,0 748 | 1,147,94,41,0,49.3,0.358,27,1 749 | 1,81,74,41,57,46.3,1.096,32,0 750 | 3,187,70,22,200,36.4,0.408,36,1 751 | 6,162,62,0,0,24.3,0.178,50,1 752 | 4,136,70,0,0,31.2,1.182,22,1 753 | 1,121,78,39,74,39,0.261,28,0 754 | 3,108,62,24,0,26,0.223,25,0 755 | 0,181,88,44,510,43.3,0.222,26,1 756 | 8,154,78,32,0,32.4,0.443,45,1 757 | 1,128,88,39,110,36.5,1.057,37,1 758 | 7,137,90,41,0,32,0.391,39,0 759 | 0,123,72,0,0,36.3,0.258,52,1 760 | 1,106,76,0,0,37.5,0.197,26,0 761 | 6,190,92,0,0,35.5,0.278,66,1 762 | 2,88,58,26,16,28.4,0.766,22,0 763 | 9,170,74,31,0,44,0.403,43,1 764 | 9,89,62,0,0,22.5,0.142,33,0 765 | 10,101,76,48,180,32.9,0.171,63,0 766 | 2,122,70,27,0,36.8,0.34,27,0 767 | 5,121,72,23,112,26.2,0.245,30,0 768 | 1,126,60,0,0,30.1,0.349,47,1 769 | 1,93,70,31,0,30.4,0.315,23,0 -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Data Science in Python from Scratch 2 | 3 | # Style of notebooks 4 | I write the notebooks to contain: 5 | 6 | 1) Intuition 7 | 8 | 2) Mathematics and Statistics behind the tool/algorithm 9 | 10 | 3) Code implementation from scratch (using numpy) 11 | 12 | 4) Application to real (publicly available) data 13 | 14 | If you spot any mistakes in the code or the theory, feel free to raise an issue. 15 | 16 | # Contact 17 | If you would like to directly contact me with queries related to this repository: 18 | 19 | Email: 20 | 21 | Twitter: https://twitter.com/HammadShaikhha 22 | -------------------------------------------------------------------------------- /Regularization in Linear Regression/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | The notebook "Regularization in Linear Regression.ipynb" discusses L1 and L2 regularization methods in the context of linear regression. The benefits of regularization to prevent overfitting and implementation in python using real data are included. 3 | 4 | ## Data 5 | I use the IMDB movie data set from kaggle and examine the relationship between IMDB rating and gross sales revenue using a simple linear regression model (degree 1), multiple linear regression (degree 5), and ridge regression (degree 5 with L2 loss function). For analysis purposes I clean the data set to only contain IMDB movie rating and gross sales revenue for US movies. 6 | 7 | ## Dependencies 8 | * numpy 9 | * Pandas 10 | * Matplotlib 11 | 12 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 13 | 14 | ## References 15 | [Siraj's video on matrix algebra and regularization](https://www.youtube.com/watch?v=s0Q3CojqRfM&t=1s) 16 | 17 | [A video on regularization in linear regression](https://www.youtube.com/watch?v=sO4ZirJh9ds) 18 | -------------------------------------------------------------------------------- /Self Organizing Maps for Data Visualization/Figures/README.MD: -------------------------------------------------------------------------------- 1 | Figures used in the SOM for clustering notebook. 2 | -------------------------------------------------------------------------------- /Self Organizing Maps for Data Visualization/Figures/RGBreduce.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Self Organizing Maps for Data Visualization/Figures/RGBreduce.png -------------------------------------------------------------------------------- /Self Organizing Maps for Data Visualization/Figures/SOMintuition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Self Organizing Maps for Data Visualization/Figures/SOMintuition.png -------------------------------------------------------------------------------- /Self Organizing Maps for Data Visualization/Figures/SelfOrgMap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Self Organizing Maps for Data Visualization/Figures/SelfOrgMap.png -------------------------------------------------------------------------------- /Self Organizing Maps for Data Visualization/Figures/WorldPoverty.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hammadshaikhha/Data-Science-and-Machine-Learning-from-Scratch/fa5c7ff9e245a8e135ae07820bd8089962c87632/Self Organizing Maps for Data Visualization/Figures/WorldPoverty.png -------------------------------------------------------------------------------- /Self Organizing Maps for Data Visualization/README.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | Self Organizing Maps (SOM) are a type of neural network that is trained using unsupervised learning algorithm for the purposes of demensionality reduction and data clustering. The notebook "Self Organizing Maps for Clustering.ipynb" discusses the intuition and theory behind SOM, and applies it to a publicly avaiable education data set from the state of Tennessee to classify each student as being weak, average, or gifted based on their characteristics. 3 | 4 | ## Data 5 | From 1985-1989 the state of Tennessee conducted a experiment to examine the causal impacts of smaller class sizes on student test scores. The data set from this study is now publicly available and can be found on [dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=hdl:1902.1/10766). I use a subset of this data set to apply SOM. In particular, I keep 6 variables for the grade 1 students, free or reduced lunch status, school absences, math, reading, listening, and word study standardized test scores. I only keep students that non-missing data for each of these variables. 6 | 7 | ## Dependencies 8 | * numpy 9 | * Pandas 10 | * Matplotlib 11 | 12 | All dependencies can be installed using [pip](https://pip.pypa.io/en/stable/) 13 | 14 | ## Work In Progress 15 | The notebook shows how to classify the input data into 3 groups using SOM. However I am open to ideas on how this 6 demensional data should be viewed on a 2D plane. There seem to be many ways to do it as show in this data visualization video from MATLAB SOM library [MATLABs Neural Network Toolbox](https://www.youtube.com/watch?v=1z5wDCubvV0) which applies SOM to the standard Iris data set. 16 | 17 | ## References 18 | [Siraj's Video on Neural Networks](https://www.youtube.com/watch?v=ov_RkIJptwE&t=1s) 19 | 20 | [Video on using SOM for Classification of Input Data](https://www.youtube.com/watch?v=H9H6s-x-0YE) 21 | 22 | [Tutorial on SOM with some code](http://www.ai-junkie.com/ann/som/som1.html) 23 | 24 | [Tutorial on SOM with python code](http://blog.yhat.com/posts/self-organizing-maps-1.html) 25 | 26 | [Lecture Slides on SOM](http://www.cs.bham.ac.uk/~jxb/NN/l16.pdf) 27 | --------------------------------------------------------------------------------