├── Bayesian_optimization_example1.ipynb
├── Bayesian_optimization_example2.ipynb
├── README.md
├── grid_search.py
└── random_search.py


/README.md:
--------------------------------------------------------------------------------
 1 | # hyperparameter_optimization_strategies
 2 | This is the Code for "How Do I Find the Best Hyperparameters?- The Math of Intelligence #7" By Siraj Raval on Youtube
 3 | 
 4 | # Coding Challenge - Due Date - Thursday, August 3rd, 2017
 5 | 
 6 | Use Bayesian Optimization to find the optimal learning rate for a linear regression model. You can use any dataset you like (examples [here](https://www.kaggle.com/datasets)) and you can use any "bayesian optimization library" you like as well. Bonus points are given if you are able to perform Bayesian optimization without using any libraries. Good luck!
 7 | 
 8 | ## Overview
 9 | 
10 | This is the code for [this](https://youtu.be/ttE0F7fghfk) video on Youtube by Siraj Raval as part of The Math of Intelligence series. I go through 3 strategies (grid search, random search, and bayesian optimization) to find the ideal hyperparameters. You can find a twitter sentiment dataset [here](https://twitter-sentiment-csv.herokuapp.com/) or [here](http://thinknook.com/twitter-sentiment-analysis-training-corpus-dataset-2012-09-22/). 
11 | 
12 | 
13 | ## Dependencies
14 | 
15 | * numpy
16 | * matplotlib
17 | * pandas
18 | 
19 | Install missing dependencies using [pip](https://pip.pypa.io/en/stable/).
20 | 
21 | ## Usage
22 | 
23 | You can run both the grid search and random search files using `python name_of_file` in terminal. I've got 2 examples of Bayesian Optimization in jupyter notebooks. The first one uses a scikit-learn wrapper to do it, but has great visualizations. The second one uses tensorflow to build the model, but implements bayesian optimization from scratch. You can run them both by typing `jupyter notebook` into terminal and the code will pop up in your browser.
24 | 
25 | Install jupyter [here](http://jupyter.readthedocs.io/en/latest/install.html). 
26 | 
27 | 
28 | ## Credits
29 | 
30 | Credits for this code goes to [David Li-Bland](https://github.com/davidlibland). I've merely created a wrapper to get people started.
31 | 


--------------------------------------------------------------------------------
/grid_search.py:
--------------------------------------------------------------------------------
 1 | import pandas as pd
 2 | import svm
 3 | 
 4 | #read data
 5 | dataframe = pd.read_fwf('tweets.txt')
 6 | X = dataframe[['tweet']]
 7 | y = dataframe[['sentiment']]
 8 | 
 9 | C_values = [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100]  
10 | gamma_values = [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100]
11 | 
12 | best_score = 0  
13 | best_params = {'C': None, 'gamma': None}
14 | 
15 | for C in C_values:  
16 |     for gamma in gamma_values:
17 |         svc = svm.SVC(C=C, gamma=gamma)
18 |         svc.fit(X, y)
19 |         score = svc.score(Xval, yval)
20 | 
21 |         if score > best_score:
22 |             best_score = score
23 |             best_params['C'] = C
24 |             best_params['gamma'] = gamma
25 | 
26 | best_score, best_params  
27 | 


--------------------------------------------------------------------------------
/random_search.py:
--------------------------------------------------------------------------------
 1 | import pandas as pd
 2 | import svm
 3 | from random import randint
 4 | 
 5 | #read data
 6 | dataframe=pd.read_csv("tweets_all.csv")
 7 | X = dataframe[['tweet']]
 8 | y = dataframe[['sentiment']]
 9 | 
10 | best_score = 0  
11 | best_params = {'C': None, 'gamma': None}
12 | 
13 | #for a preset number of iterations
14 | for i in range(10):
15 |     #try random values for each hyperparameter
16 |     svc = svm.SVC(C=randint(0, 9), gamma=randint(0, 3))
17 |     svc.fit(X, y)
18 |     score = svc.score(Xval, yval)
19 | 
20 |     if score > best_score:
21 |         best_score = score
22 |         best_params['C'] = C
23 |         best_params['gamma'] = gamma
24 | 
25 | best_score, best_params  
26 | 


--------------------------------------------------------------------------------