` tag with attribute `class="live-timestamp"`.
79 | - The subreddit is within an `` tag with the attribute `class="subreddit hover may-blank"`.
80 | - The number of comments is within an ` ` tag with the attribute `data-event-action="comments"`.
81 |
82 | The functions are:
83 | ```
84 | def extract_title_from_result(result,num=25):
85 | titles = []
86 | title = result.find_all('a', {'data-event-action':'title'})
87 | for i in title:
88 | titles.append(i.text)
89 | return titles
90 |
91 | def extract_time_from_result(result,num=25):
92 | times = []
93 | time = result.find_all('time', {'class':'live-timestamp'})
94 | for i in time:
95 | times.append(i.text)
96 | return times
97 |
98 | def extract_subreddit_from_result(result,num=25):
99 | subreddits = []
100 | subreddit = result.find_all('a', {'class':'subreddit hover may-blank'})
101 | for i in subreddit:
102 | subreddits.append(i.string)
103 | return subreddits
104 |
105 | def extract_num_from_result(result,num=25):
106 | nums_lst = []
107 | nums = result.find_all('a', {'data-event-action': 'comments'})
108 | for i in nums:
109 | nums_lst.append(i.string)
110 | return nums_lst
111 | ```
112 | I then write a function that finds the last `id` on the page, and stores it:
113 | ```
114 | def get_urls(n=25):
115 | j=0 # counting loops
116 | titles = []
117 | times = []
118 | subreddits = []
119 | nums = []
120 | URLS = []
121 | URL = "http://www.reddit.com"
122 |
123 | for _ in range(n):
124 |
125 | res = requests.get(URL, headers={"user-agent":'mt'})
126 | soup = BeautifulSoup(res.content,"lxml")
127 |
128 | titles.extend(extract_title_from_result(soup))
129 | times.extend(extract_time_from_result(soup))
130 | subreddits.extend(extract_subreddit_from_result(soup))
131 | nums.extend(extract_num_from_result(soup))
132 |
133 | URL = soup.find('span',{'class':'next-button'}).find('a')['href']
134 | URLS.append(URL)
135 | j+=1
136 | print(j)
137 | time.sleep(3)
138 |
139 | return titles, times, subreddits, nums, URLS
140 | ```
141 |
142 | I then build a pandas `DataFrame`, perform some exploratory data analysis and create:
143 | - A binary column that classifies the number of comments comparing the values with their median
144 | - A set of dummy columns for the subreddits
145 | - Concatenate both
146 |
147 | ```
148 | df['binary'] = df['nums'].apply(lambda x: 1 if x >= np.median(df['nums']) else 0)
149 | # dummies created and dataframes concatenated
150 | df_subred = pd.concat([df['binary'],pd.get_dummies(df['subreddits'], drop_first = True)], axis = 1)
151 | ```
152 |
13 |
15 |
18 | Summary •
19 | Preamble •
20 | Getting data •
21 | Data Munging and EDA •
22 | Mining the data •
23 | Building the models •
24 | Plotting results •
25 | Conclusions and recommendations
26 |
27 |
28 |
101 |
103 |
104 | To generate scatter plots for all the predictors (which provided similar information as the correlation matrices) we write:
105 | ```
106 | g = sns.pairplot(df[cols_to_keep])
107 | for ax in g.axes.flatten(): # from [6]
108 | for tick in ax.get_xticklabels():
109 | tick.set(rotation=90);
110 | ```
111 |
112 |
114 |
115 |
116 |
171 |
173 |
174 |
9 |
11 |
12 | Problem Statement •
13 | Dataset •
14 | Importing basic modules •
15 | Pre-Processing of dataset `Best_of` = 5 •
17 | Dummy variables •
18 | Exploratory Analysis for Best_of = 5 •
19 | Logistic Regression •
20 | Decision Trees and Random Forests
21 |
22 |
23 |
104 |
106 |
107 |
129 |
131 |
132 | To correct this problem, and create a balanced dataset via simple undersampling, I used a stratified sampling procedure.
133 |
134 | ```
135 | y_0 = df3[df3.win == 0]
136 | y_1 = df3[df3.win == 1]
137 | n = min([len(y_0), len(y_1)])
138 | y_0 = y_0.sample(n = n, random_state = 0)
139 | y_1 = y_1.sample(n = n, random_state = 0)
140 | df_strat = pd.concat([y_0, y_1])
141 | X_strat = df_strat[['Date', 'Surface', 'Round','WRank', 'LRank']]
142 | y_strat = df_strat.win
143 | df = X_strat.copy()
144 | df['win'] = y_strat
145 | ```
146 | The balanced classes become:
147 |
148 |
149 |
151 |
152 | We now define the variables `P1` and `P2` where the former has higher ranking:
153 | ```
154 | ranks = ["WRank", "LRank"]
155 | df["P1"] = df[ranks].max(axis=1)
156 | df["P2"] = df[ranks].min(axis=1)
157 | ```
158 |
159 |
168 |
170 |
171 | What about the dependence on rounds? The relation is not very clear but we can clearly see that upsets are unlikely to happen on the semifinals.
172 |
173 | ```
174 | win_by_round = pd.crosstab(df.win, df.Round).apply(lambda x: x/x.sum(), axis = 0)
175 | ```
176 |
177 |
179 |
180 |
181 |
182 |
279 |
282 |
320 |
322 |
323 |
335 |
337 |
338 |
369 |
371 |
372 | 
9 | 
14 | 
35 | 
40 | 
16 | 
12 | 
62 | 
168 | 
194 | 
245 | 
11 | 
46 |