├── Images
├── Classification_Report.JPG
├── prediction.JPG
└── top10_tags.JPG
├── README.md
├── Stackoverflow Clean Questions.ipynb
└── Stackoverflow Tags Map & Model.ipynb
/Images/Classification_Report.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/theRajeshReddy/StackOverFlow-Classification/8ba027db0dfa3d09724a62838bebdd979b031af0/Images/Classification_Report.JPG
--------------------------------------------------------------------------------
/Images/prediction.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/theRajeshReddy/StackOverFlow-Classification/8ba027db0dfa3d09724a62838bebdd979b031af0/Images/prediction.JPG
--------------------------------------------------------------------------------
/Images/top10_tags.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/theRajeshReddy/StackOverFlow-Classification/8ba027db0dfa3d09724a62838bebdd979b031af0/Images/top10_tags.JPG
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Multiclass Multilabel Prediction For StackOverflow Questions
2 |
3 | **Data set** : https://www.kaggle.com/therajeshreddy/stackoverflow
4 |
5 | **Objective** : Given text for Questions from StackoverFlow posts, predict tags associated with them.
6 |
7 | This is a scaled down version of predecting only top 10 most occurring tags
8 |
9 | **Programming Language** : Python using nltk & Keras
10 |
11 | **Model Architecture** : Deep Learning using Recurrent Neural Network (RNN)
12 |
13 | **About Data Set**
14 |
15 | Dataset has text of questions, answers and thier corresponding tags from the Stack Overflow programming Q&A website.
16 |
17 | This is organized as three files:
18 |
19 | 1. Questions contains the title, body, creation date, closed date (if applicable), score, and owner ID for all non-deleted Stack Overflow questions.
20 |
21 | 2. Tags contains the tags on each of these questions.
22 |
23 | 3. Answers contains the body, creation date, score, and owner ID for each of the answers to these questions. The ParentId column links back to the Questions table. *We don't use this file as we want to predict Tags given a question*
24 |
25 | **Data Pre-Processing**
26 |
27 | >Questions File
28 | *Code* : Stackoverflow Clean Questions.ipynb
29 |
30 | 1. Read Questions File
31 | 2. Drop All columns except Id,Title and Body
32 | 3. Now the text in the Body column seem to have many html tags in the text. We use Regular Expressions and Clean the Body column text by removing the html tags
33 | ```python
34 | import re
35 | def rem_html_tags(body):
36 | regex = re.compile('<.*?>')
37 | return re.sub(regex, '', body)
38 | ques['Body'] = ques['Body'].apply(rem_html_tags)
39 | ```
40 | 4. Save the questions file for later use
41 | ```python
42 | ques.to_csv('question_clean.csv',index=False)
43 | ```
44 |
45 | >Tags File
46 | *Code* : Stackoverflow Tags Map & Model.ipynb
47 |
48 | 1. Read Tags File
49 | 2. Identify top 10 Tags by count
50 | ```python
51 | tagCount = collections.Counter(list(df_tags['Tag'])).most_common(10)
52 | print(tagCount)
53 |
54 | [('javascript', 124155), ('java', 115212), ('c#', 101186), ('php', 98808), ('android', 90659), ('jquery', 78542), ('python', 64601), ('html', 58976), ('c++', 47591), ('ios', 47009)]
55 | ```
56 |
57 |
58 |
59 | 3. Manipulate the tags dataframe so that all the Tags for an ID are as a list in a row (grouped by Question ID)
60 |
61 | ```python
62 | def add_tags(question_id):
63 | return tag_top10[tag_top10['Id'] == question_id['Id']].Tag.values
64 |
65 | top10 = tag_top10.apply(add_tags, axis=1)
66 | ```
67 |
68 |
69 | >Combine the Questions and Tags
70 | *Code* : Stackoverflow Tags Map & Model.ipynb
71 |
72 | Merge the Questions and Tags data frame by ID
73 |
74 | ```python
75 | total=pd.merge(ques, top10_tags, on='Id')
76 | ```
77 |
78 | Our Dataset would now have only Id, Title, Body & Tags
79 |
80 | >Text Preprocessing
81 | *Code* : Stackoverflow Tags Map & Model.ipynb
82 |
83 | We will use nltk, preprocessing from Keras and sklearn to process the text data
84 |
85 | *Tags preprocesing*
86 | Use MultiLabelBinarizer from sklearn on the Class labels(Tags)
87 | ```python
88 | from sklearn.preprocessing import MultiLabelBinarizer
89 | multilabel_binarizer = MultiLabelBinarizer()
90 | multilabel_binarizer.fit(total.Tags)
91 | print(multilabel_binarizer.classes_)
92 |
93 | array(['android', 'c#', 'c++', 'html', 'ios', 'java', 'javascript','jquery', 'php', 'python'], dtype=object)
94 | ```
95 |
96 | *Title & Body Preprocessing*
97 | 1. Tokenize the words
98 | 2. Convert the tokenized words to sequences
99 |
100 | **Model Building**
101 |
102 | Implemented a Hybrid model in TensorFlow using Keras as high level api. Architecture used is RNN. In this model first we train a model using the Title data, then train a model using the Body data. Outputs of both are concatenated and passed thorugh the dense layers before connecting to the output layer
103 |
104 | *RNN Model* : The model first uses GRU for the sequence data training with 2 GRU layers one for Title and other for Body.
105 |
106 | RNN for Title has
107 | - 1 Embedding Layer has input of Title vocabulary length(68969) + 1(for 0 padding) and out put of 2000 embeddings (for better results use full vocabulary length+1)
108 | - 1 Gated recurrent unit (GRU) layer
109 | - 1 dense output layer of shape 10(No of classes(tags) we are trying to predict)
110 |
111 | ```python
112 | # Title Only
113 | title_input = Input(name='title_input',shape=[max_len_t])
114 | title_Embed = Embedding(vocab_len_t+1,2000,input_length=max_len_t,mask_zero=True,name='title_Embed')(title_input)
115 | gru_out_t = GRU(300)(title_Embed)
116 | # auxiliary output to tune GRU weights smoothly
117 | auxiliary_output = Dense(10, activation='sigmoid', name='aux_output')(gru_out_t)
118 | ```
119 |
120 | RNN for Body has
121 | - 1 Embedding Layer has input of Title vocabulary length(1292018) + 1(for 0 padding) and out put of 170 embeddings (for better results use full vocabulary length+1)
122 | - 1 Gated recurrent unit (GRU) layer
123 |
124 | ```python
125 | # Body Only
126 | body_input = Input(name='body_input',shape=[max_len_b])
127 | body_Embed = Embedding(vocab_len_b+1,170,input_length=max_len_b,mask_zero=True,name='body_Embed')(body_input)
128 | gru_out_b = GRU(200)(body_Embed)
129 | ```
130 |
131 | Combine the 2 GRU outputs
132 | ```python
133 | com = concatenate([gru_out_t, gru_out_b])
134 | ```
135 |
136 | The fully connected network has
137 | - 2 Dense Layers
138 | - 1 Dropout layer
139 | - 1 BatchNormalization layer
140 | - 1 Dense Output layer
141 |
142 | ```python
143 | # now the combined data is being fed to dense layers
144 | dense1 = Dense(400,activation='relu')(com)
145 | dp1 = Dropout(0.5)(dense1)
146 | bn = BatchNormalization()(dp1)
147 | dense2 = Dense(150,activation='relu')(bn)
148 | main_output = Dense(10, activation='sigmoid', name='main_output')(dense2)
149 | ```
150 |
151 | *Model Compilattion with optimizer='adam', loss='categorical_crossentropy', metrics='accuracy')*
152 |
153 | **Model Performance Review**
154 |
155 | *Classification Report to check Precision, Recall and F1 Score*
156 |
157 | The Model seem to performing good enough with score of 84%. Increase in the Embedding, GRU and dense layers would help in getting better results
158 |
159 |
160 |
161 | **Random Validation on Test Data**
162 |
163 |
164 |
165 |
166 | **Save the Model & Weights**
167 |
168 | Saving the model for transfer learning or model execution later
169 |
170 | ```python
171 | model.save('./stackoverflow_tags.h5')
172 | ```
173 |
--------------------------------------------------------------------------------
/Stackoverflow Clean Questions.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 1,
6 | "metadata": {
7 | "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
8 | "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
9 | },
10 | "outputs": [
11 | {
12 | "name": "stdout",
13 | "output_type": "stream",
14 | "text": [
15 | "['Answers.csv', 'Tags.csv', 'Questions.csv']\n"
16 | ]
17 | }
18 | ],
19 | "source": [
20 | "# This Python 3 environment comes with many helpful analytics libraries installed\n",
21 | "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
22 | "# For example, here's several helpful packages to load in \n",
23 | "\n",
24 | "import numpy as np # linear algebra\n",
25 | "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
26 | "\n",
27 | "# for counting\n",
28 | "import collections\n",
29 | "\n",
30 | "# Input data files are available in the \"../input/\" directory.\n",
31 | "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
32 | "\n",
33 | "import os\n",
34 | "print(os.listdir(\"../input\"))\n",
35 | "\n",
36 | "# Any results you write to the current directory are saved as output."
37 | ]
38 | },
39 | {
40 | "cell_type": "code",
41 | "execution_count": 2,
42 | "metadata": {
43 | "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
44 | "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
45 | },
46 | "outputs": [
47 | {
48 | "data": {
49 | "text/html": [
50 | "
\n",
51 | "\n",
64 | "
\n",
65 | " \n",
66 | " \n",
67 | " \n",
68 | " Id \n",
69 | " OwnerUserId \n",
70 | " CreationDate \n",
71 | " ClosedDate \n",
72 | " Score \n",
73 | " Title \n",
74 | " Body \n",
75 | " \n",
76 | " \n",
77 | " \n",
78 | " \n",
79 | " 0 \n",
80 | " 80 \n",
81 | " 26.0 \n",
82 | " 2008-08-01T13:57:07Z \n",
83 | " NaN \n",
84 | " 26 \n",
85 | " SQLStatement.execute() - multiple queries in o... \n",
86 | " <p>I've written a database generation script i... \n",
87 | " \n",
88 | " \n",
89 | " 1 \n",
90 | " 90 \n",
91 | " 58.0 \n",
92 | " 2008-08-01T14:41:24Z \n",
93 | " 2012-12-26T03:45:49Z \n",
94 | " 144 \n",
95 | " Good branching and merging tutorials for Torto... \n",
96 | " <p>Are there any really good tutorials explain... \n",
97 | " \n",
98 | " \n",
99 | " 2 \n",
100 | " 120 \n",
101 | " 83.0 \n",
102 | " 2008-08-01T15:50:08Z \n",
103 | " NaN \n",
104 | " 21 \n",
105 | " ASP.NET Site Maps \n",
106 | " <p>Has anyone got experience creating <strong>... \n",
107 | " \n",
108 | " \n",
109 | " 3 \n",
110 | " 180 \n",
111 | " 2089740.0 \n",
112 | " 2008-08-01T18:42:19Z \n",
113 | " NaN \n",
114 | " 53 \n",
115 | " Function for creating color wheels \n",
116 | " <p>This is something I've pseudo-solved many t... \n",
117 | " \n",
118 | " \n",
119 | " 4 \n",
120 | " 260 \n",
121 | " 91.0 \n",
122 | " 2008-08-01T23:22:08Z \n",
123 | " NaN \n",
124 | " 49 \n",
125 | " Adding scripting functionality to .NET applica... \n",
126 | " <p>I have a little game written in C#. It uses... \n",
127 | " \n",
128 | " \n",
129 | " 5 \n",
130 | " 330 \n",
131 | " 63.0 \n",
132 | " 2008-08-02T02:51:36Z \n",
133 | " NaN \n",
134 | " 29 \n",
135 | " Should I use nested classes in this case? \n",
136 | " <p>I am working on a collection of classes use... \n",
137 | " \n",
138 | " \n",
139 | " 6 \n",
140 | " 470 \n",
141 | " 71.0 \n",
142 | " 2008-08-02T15:11:47Z \n",
143 | " 2016-03-26T05:23:29Z \n",
144 | " 13 \n",
145 | " Homegrown consumption of web services \n",
146 | " <p>I've been writing a few web services for a ... \n",
147 | " \n",
148 | " \n",
149 | " 7 \n",
150 | " 580 \n",
151 | " 91.0 \n",
152 | " 2008-08-02T23:30:59Z \n",
153 | " NaN \n",
154 | " 21 \n",
155 | " Deploying SQL Server Databases from Test to Live \n",
156 | " <p>I wonder how you guys manage deployment of ... \n",
157 | " \n",
158 | " \n",
159 | " 8 \n",
160 | " 650 \n",
161 | " 143.0 \n",
162 | " 2008-08-03T11:12:52Z \n",
163 | " NaN \n",
164 | " 79 \n",
165 | " Automatically update version number \n",
166 | " <p>I would like the version property of my app... \n",
167 | " \n",
168 | " \n",
169 | " 9 \n",
170 | " 810 \n",
171 | " 233.0 \n",
172 | " 2008-08-03T20:35:01Z \n",
173 | " NaN \n",
174 | " 9 \n",
175 | " Visual Studio Setup Project - Per User Registr... \n",
176 | " <p>I'm trying to maintain a Setup Project in <... \n",
177 | " \n",
178 | " \n",
179 | "
\n",
180 | "
"
181 | ],
182 | "text/plain": [
183 | " Id ... Body\n",
184 | "0 80 ... I've written a database generation script i...\n",
185 | "1 90 ...
Are there any really good tutorials explain...\n",
186 | "2 120 ...
Has anyone got experience creating ...\n",
187 | "3 180 ... This is something I've pseudo-solved many t...\n",
188 | "4 260 ...
I have a little game written in C#. It uses...\n",
189 | "5 330 ...
I am working on a collection of classes use...\n",
190 | "6 470 ...
I've been writing a few web services for a ...\n",
191 | "7 580 ...
I wonder how you guys manage deployment of ...\n",
192 | "8 650 ...
I would like the version property of my app...\n",
193 | "9 810 ...
I'm trying to maintain a Setup Project in <...\n",
194 | "\n",
195 | "[10 rows x 7 columns]"
196 | ]
197 | },
198 | "execution_count": 2,
199 | "metadata": {},
200 | "output_type": "execute_result"
201 | }
202 | ],
203 | "source": [
204 | "ques = pd.read_csv('../input/Questions.csv',encoding='iso-8859-1')\n",
205 | "ques.head(10)"
206 | ]
207 | },
208 | {
209 | "cell_type": "code",
210 | "execution_count": 3,
211 | "metadata": {
212 | "_uuid": "3a963c29707e29d8df14b8f48e45d25b336fcdb0"
213 | },
214 | "outputs": [
215 | {
216 | "data": {
217 | "text/html": [
218 | "
\n",
219 | "\n",
232 | "
\n",
233 | " \n",
234 | " \n",
235 | " \n",
236 | " Id \n",
237 | " Title \n",
238 | " Body \n",
239 | " \n",
240 | " \n",
241 | " \n",
242 | " \n",
243 | " 0 \n",
244 | " 80 \n",
245 | " SQLStatement.execute() - multiple queries in o... \n",
246 | " <p>I've written a database generation script i... \n",
247 | " \n",
248 | " \n",
249 | " 1 \n",
250 | " 90 \n",
251 | " Good branching and merging tutorials for Torto... \n",
252 | " <p>Are there any really good tutorials explain... \n",
253 | " \n",
254 | " \n",
255 | " 2 \n",
256 | " 120 \n",
257 | " ASP.NET Site Maps \n",
258 | " <p>Has anyone got experience creating <strong>... \n",
259 | " \n",
260 | " \n",
261 | " 3 \n",
262 | " 180 \n",
263 | " Function for creating color wheels \n",
264 | " <p>This is something I've pseudo-solved many t... \n",
265 | " \n",
266 | " \n",
267 | " 4 \n",
268 | " 260 \n",
269 | " Adding scripting functionality to .NET applica... \n",
270 | " <p>I have a little game written in C#. It uses... \n",
271 | " \n",
272 | " \n",
273 | " 5 \n",
274 | " 330 \n",
275 | " Should I use nested classes in this case? \n",
276 | " <p>I am working on a collection of classes use... \n",
277 | " \n",
278 | " \n",
279 | " 6 \n",
280 | " 470 \n",
281 | " Homegrown consumption of web services \n",
282 | " <p>I've been writing a few web services for a ... \n",
283 | " \n",
284 | " \n",
285 | " 7 \n",
286 | " 580 \n",
287 | " Deploying SQL Server Databases from Test to Live \n",
288 | " <p>I wonder how you guys manage deployment of ... \n",
289 | " \n",
290 | " \n",
291 | " 8 \n",
292 | " 650 \n",
293 | " Automatically update version number \n",
294 | " <p>I would like the version property of my app... \n",
295 | " \n",
296 | " \n",
297 | " 9 \n",
298 | " 810 \n",
299 | " Visual Studio Setup Project - Per User Registr... \n",
300 | " <p>I'm trying to maintain a Setup Project in <... \n",
301 | " \n",
302 | " \n",
303 | "
\n",
304 | "
"
305 | ],
306 | "text/plain": [
307 | " Id ... Body\n",
308 | "0 80 ... I've written a database generation script i...\n",
309 | "1 90 ...
Are there any really good tutorials explain...\n",
310 | "2 120 ...
Has anyone got experience creating ...\n",
311 | "3 180 ... This is something I've pseudo-solved many t...\n",
312 | "4 260 ...
I have a little game written in C#. It uses...\n",
313 | "5 330 ...
I am working on a collection of classes use...\n",
314 | "6 470 ...
I've been writing a few web services for a ...\n",
315 | "7 580 ...
I wonder how you guys manage deployment of ...\n",
316 | "8 650 ...
I would like the version property of my app...\n",
317 | "9 810 ...
I'm trying to maintain a Setup Project in <...\n",
318 | "\n",
319 | "[10 rows x 3 columns]"
320 | ]
321 | },
322 | "execution_count": 3,
323 | "metadata": {},
324 | "output_type": "execute_result"
325 | }
326 | ],
327 | "source": [
328 | "ques.drop([\"OwnerUserId\",\"CreationDate\",\"ClosedDate\",\"Score\"], axis=1, inplace=True)\n",
329 | "ques.head(10)"
330 | ]
331 | },
332 | {
333 | "cell_type": "code",
334 | "execution_count": 4,
335 | "metadata": {
336 | "_uuid": "b1305b3807f362fec50a13a76d68d35fa45c3380"
337 | },
338 | "outputs": [],
339 | "source": [
340 | "import re \n",
341 | "\n",
342 | "def rem_html_tags(body):\n",
343 | " regex = re.compile('<.*?>')\n",
344 | " return re.sub(regex, '', body)"
345 | ]
346 | },
347 | {
348 | "cell_type": "code",
349 | "execution_count": 5,
350 | "metadata": {
351 | "_uuid": "f0340076f170cc798549f9da6ce46962d9f81c6b"
352 | },
353 | "outputs": [
354 | {
355 | "data": {
356 | "text/html": [
357 | "
\n",
358 | "\n",
371 | "
\n",
372 | " \n",
373 | " \n",
374 | " \n",
375 | " Id \n",
376 | " Title \n",
377 | " Body \n",
378 | " \n",
379 | " \n",
380 | " \n",
381 | " \n",
382 | " 0 \n",
383 | " 80 \n",
384 | " SQLStatement.execute() - multiple queries in o... \n",
385 | " I've written a database generation script in S... \n",
386 | " \n",
387 | " \n",
388 | " 1 \n",
389 | " 90 \n",
390 | " Good branching and merging tutorials for Torto... \n",
391 | " Are there any really good tutorials explaining... \n",
392 | " \n",
393 | " \n",
394 | " 2 \n",
395 | " 120 \n",
396 | " ASP.NET Site Maps \n",
397 | " Has anyone got experience creating SQL-based A... \n",
398 | " \n",
399 | " \n",
400 | " 3 \n",
401 | " 180 \n",
402 | " Function for creating color wheels \n",
403 | " This is something I've pseudo-solved many time... \n",
404 | " \n",
405 | " \n",
406 | " 4 \n",
407 | " 260 \n",
408 | " Adding scripting functionality to .NET applica... \n",
409 | " I have a little game written in C#. It uses a ... \n",
410 | " \n",
411 | " \n",
412 | "
\n",
413 | "
"
414 | ],
415 | "text/plain": [
416 | " Id ... Body\n",
417 | "0 80 ... I've written a database generation script in S...\n",
418 | "1 90 ... Are there any really good tutorials explaining...\n",
419 | "2 120 ... Has anyone got experience creating SQL-based A...\n",
420 | "3 180 ... This is something I've pseudo-solved many time...\n",
421 | "4 260 ... I have a little game written in C#. It uses a ...\n",
422 | "\n",
423 | "[5 rows x 3 columns]"
424 | ]
425 | },
426 | "execution_count": 5,
427 | "metadata": {},
428 | "output_type": "execute_result"
429 | }
430 | ],
431 | "source": [
432 | "ques['Body'] = ques['Body'].apply(rem_html_tags)\n",
433 | "ques.head()"
434 | ]
435 | },
436 | {
437 | "cell_type": "code",
438 | "execution_count": 6,
439 | "metadata": {
440 | "_uuid": "af150557dcf11050e0e7dbc8bcbb0ef2c5015636"
441 | },
442 | "outputs": [],
443 | "source": [
444 | "ques.to_csv('question_clean.csv',index=False)"
445 | ]
446 | },
447 | {
448 | "cell_type": "code",
449 | "execution_count": 7,
450 | "metadata": {},
451 | "outputs": [],
452 | "source": []
453 | }
454 | ],
455 | "metadata": {
456 | "kernelspec": {
457 | "display_name": "Python 3",
458 | "language": "python",
459 | "name": "python3"
460 | },
461 | "language_info": {
462 | "codemirror_mode": {
463 | "name": "ipython",
464 | "version": 3
465 | },
466 | "file_extension": ".py",
467 | "mimetype": "text/x-python",
468 | "name": "python",
469 | "nbconvert_exporter": "python",
470 | "pygments_lexer": "ipython3",
471 | "version": "3.6.7"
472 | }
473 | },
474 | "nbformat": 4,
475 | "nbformat_minor": 1
476 | }
477 |
--------------------------------------------------------------------------------
/Stackoverflow Tags Map & Model.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 1,
6 | "metadata": {
7 | "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
8 | "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
9 | },
10 | "outputs": [
11 | {
12 | "name": "stdout",
13 | "output_type": "stream",
14 | "text": [
15 | "['stackoverflow-clean-questions-file-v2', 'stackoverflow']\n"
16 | ]
17 | }
18 | ],
19 | "source": [
20 | "# This Python 3 environment comes with many helpful analytics libraries installed\n",
21 | "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
22 | "# For example, here's several helpful packages to load in \n",
23 | "\n",
24 | "import numpy as np # linear algebra\n",
25 | "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
26 | "\n",
27 | "# Input data files are available in the \"../input/\" directory.\n",
28 | "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
29 | "\n",
30 | "import os\n",
31 | "print(os.listdir(\"../input\"))\n",
32 | "\n",
33 | "# Plotting Libs\n",
34 | "import matplotlib.pyplot as plt\n",
35 | "import matplotlib.cm as cm\n",
36 | "# magic function\n",
37 | "%matplotlib inline\n",
38 | "\n",
39 | "import collections"
40 | ]
41 | },
42 | {
43 | "cell_type": "code",
44 | "execution_count": 2,
45 | "metadata": {
46 | "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
47 | "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
48 | },
49 | "outputs": [],
50 | "source": [
51 | "df_tags = pd.read_csv('../input/stackoverflow/Tags.csv', encoding='iso-8859-1')"
52 | ]
53 | },
54 | {
55 | "cell_type": "code",
56 | "execution_count": 3,
57 | "metadata": {},
58 | "outputs": [],
59 | "source": [
60 | "def plot_tags(tagCount):\n",
61 | " \n",
62 | " x,y = zip(*tagCount)\n",
63 | "\n",
64 | " colormap = plt.cm.gist_ncar #nipy_spectral, Set1,Paired \n",
65 | " colors = [colormap(i) for i in np.linspace(0, 0.8,50)] \n",
66 | "\n",
67 | " area = [i/4000 for i in list(y)] # 0 to 15 point radiuses\n",
68 | " plt.figure(figsize=(9,8))\n",
69 | " plt.ylabel(\"Number of question associations\")\n",
70 | " for i in range(len(y)):\n",
71 | " plt.plot(i,y[i], marker='o', linestyle='',ms=area[i],label=x[i])\n",
72 | "\n",
73 | " plt.legend(numpoints=1)\n",
74 | " plt.show()"
75 | ]
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": 4,
80 | "metadata": {},
81 | "outputs": [
82 | {
83 | "name": "stdout",
84 | "output_type": "stream",
85 | "text": [
86 | "[('javascript', 124155), ('java', 115212), ('c#', 101186), ('php', 98808), ('android', 90659), ('jquery', 78542), ('python', 64601), ('html', 58976), ('c++', 47591), ('ios', 47009)]\n"
87 | ]
88 | },
89 | {
90 | "data": {
91 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAHVCAYAAAAdLJRmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4lFXa+PHveSYz6QkkoSSUDSg1FQgoIB0FxQIoy8uLQkTZn21ffS1r3dVVd1fXXtfdtUQUFUVRX3FFkaaASsDQqxBqIJBASJ92fn/MJAZImSQzk3Z/ritXMuc5zzn3REzunOcUpbVGCCGEEKItMpo6ACGEEEKIpiKJkBBCCCHaLEmEhBBCCNFmSSIkhBBCiDZLEiEhhBBCtFmSCAkhhBCizQpo6gCEEEIIca74+xZbgARgIBAHBALlwBFgA7Al+4lJtqaLsHVQso+QEEII0TzE37fYBFwG/AG4ACjD9fQmBFCABkoAJxAE/Aj8Hfgy+4lJjqaIuaWTREgIIYRoYvH3LVbAtcCzuEZ+wutxeyGukaI7gXezn5gkv9jrQRIht5iYGB0fH9/UYQghhGhjHJZwihKnYovoCgGWhjdkt2I+fZDwLZ9gWIu8F2ALtX79+hNa6w511ZNEyC0tLU1nZmY2dRhCCCHakC2HC5jx7x8otTqwOxv/+zjAUARbTLw/90ISu0R6IcKWSym1XmudVlc9WTUmhBBCNIHNhwr47T/XUlhm90oSBGB3agrL7Pz2n2vZcrjAK222dpIICSGEEH527HQZ//36D5RYfTO/ucTqYMa/fyD3dJlP2m9NZPm8D1jtTnYdK2TrkQKOnS6n3O4kMMCgU0QgCXGR9OkcjtkkOagQQrRFWmvuWJBFqY+SoAqlVgd3LMhi/o0XoJTyaV8tmSRCXuJwapbvyOW1lb+QdfAUgWYDpxPKbA40rjWPQWYThgHlNiep3dpx06jzGNO3IyZD/oEKIURb8dG6/fy8P99rj8NqYndqNuzP5+UvfmL8+RE+7aspBQUF0bVrV8xmc4Pul8nSbg2dLK215pMNh3l88TasdifF9cjwQy0mLAEGD03qz9SBXSRjF0KIVs7h1Ax8dAkFZf7b8qd9iJnMhy5ulX90a63Jy8ujsLCQHj16nHFNJkv7gesZ74889OkWTpbY6pUEARRbHZwssfHQp1uY+fqP8ixXCCFaueU7crHanX7t02p3smJnrl/79BelFNHR0ZSVNfz3pyRCDbTlcAHjn13Jun35lNoal9mX2hz8tC+fcc+ulFn+QgjRir228hdK7f59ElNsdfDayl/82qc/NfZpiiRCDSBLHoUQQtSX1e4k6+CpJun75wOnsDn8OxLVUkgiVE+y5FEIIURD7DpWSKC5aX7tBpoNdh0rPKNs2LBhTRLL2Y4cOcI111xTa51Tp07x6quv+qR/SYTqwd9LHmUiuxBCtB5bjxTgbKJBGa0552nDmjVrmiaYKux2O3FxcSxcuLDWepIINROfbDhM1oFTflny+POBU3yy4bBP+xFCCOE/x06XU9bIOaUNVWp1kHu6/IyysLAwioqKGDduHAMHDiQpKYnPPvsMgPvuu49XXnmlsu4jjzzC008/XWP94uJiJk2aREpKComJiSxYsACAdevWMWzYMFJSUhgyZAiFhYVkZGRw5ZVXMnbsWMaNG0d2djaJiYkAZGRkcNVVVzF69Gh69erFn//858p4fvnlF1JTU7nnnnu8+r2RfYQ85HBqHl+8rdEToz1VanPw+OJtTB7QpVUueRRCiLam3O6kvn9Gh1NCgpFNF05gVnZsOoDDxLDVGU8hIR63o939ny0oKIhFixYRERHBiRMnuPDCC7nyyiuZPn06d9xxB7feeisAH374IUuWLKmx/ldffUVcXByLFy8GoKCgAKvVyvTp01mwYAGDBw/m9OnTBAcHA7BhwwY2bdpEVFQU2dnZZ8T0008/sWXLFkJCQhg8eDCTJk3iiSeeYMuWLWRlZdXzO1g3SYQ81JRLHsf16+TXfoUQQnhfYICBgjqToVjyuC7ga642fUcUhZRiwYQThUajcGAQjJV8wvnYMYJ37JeQQ3StbSp3/2fTWvPAAw+watUqDMPg8OHDHDt2jAEDBpCbm8uRI0c4fvw47du3p1u3bthstmrrJyUlcdddd3Hvvfdy+eWXM2LECDZv3kxsbCyDBw8GICLi100dL774YqKioqqN9eKLLyY62vV+pk6dyvfff8/kyZPr+K41nCRCHnpt5S/13ieosSqWPEoiJIQQLV+niECCzKYanyx04CR/M7/OCGMLoAlUdgDMlFbfHqe4wfQf5pi+4ntnIvfb5nKcdtXWDbaY6BgReE75/PnzOX78OOvXr8dsNhMfH1+5J8+0adNYuHAhR48eZfr06bXW7927Nxs2bODLL7/koYceYty4cUyZMqXG70VoaGiN185eDu/rzYZljpAHZMmjEEKIxkqIi8So9reu5irje5YH3sVIYxOBylaZBNUlUNkJUjZGGptYHngnVxnfU92Yk1KQ2CXynPKCggI6duyI2Wxm+fLl7N+/v/La9OnT+eCDD1i4cCHTpk2rtf6RI0cICQnh2muv5Z577mHDhg306dOHnJwc1q1bB0BhYSF2e93v65tvviE/P5/S0lI+/fRThg8fTnh4OIWFhXXe2xAyIuSBiiWP9nL/T3KrWPKYEHfuP2AhhBAtR+9O4ZTbzv7DVvNwwDymm1YQosqrvc8TFuXAgoO/md8g1fELf7bPwvVAzKXc5qR3p/Az7lFKMXPmTK644gqSkpJIS0ujb9++ldcTEhIoLCykS5cuxMbGAtRYf/Pmzdxzzz0YhoHZbOYf//gHFouFBQsW8Pvf/57S0lKCg4NZunRpne9lyJAhXH311Rw6dIhrr72WtDTXKRnDhw8nMTGRSy+9lKeeeqrB36uzyVljbrWdNbZg3QEe+dx/E6WrCrGYePiK/kwf3N3vfQshhPCua/6xhsz9J92vNI8EvM1vTSsblQSdrUQHssAx+oxkaHB8ez666dd9g/Ly8hg4cOAZI0DNQUZGBpmZmbz88sv1um/79u3069fvjDI5a8yLmtuSRyGEEC3TTaPOIzjAlZxcZaz2ehIEEKLKmW5awZXGasB1wPdNo86rvH7kyBGGDh3K3Xff7dV+Wyp5NOaBhix59JaaljwKIYRoecb07YjZpAiz5/MX85teT4IqhKhy/mp+k7XliTjMHRndp2Pltbi4OHbt2uWTfhsrPT2d9PR0v/YpiZAHPF3y6As1LXkUQgjR8pgMxdy0KBLX/R0LNp/2ZcHGk5Y3OHnZPNmPrhaSCHmgriWPvlTTkkchhBAt04TYEuKNzVjw7e8Ui3JwkdqE+XyfdtPiyVCDB2pe8uh7NS15FEII0TJF7fkYs8k/IzRmk4Fa97pf+mqpJBHyQPVLHv2juiWPQgghWq7I7P+gHFa/9KUc5ZD1vl/6aqkkEfKAJcAgtVv1u3X62oDu7TCb5D+TEEK0CmUFBFj9vEFvSR6UnT6neNiwYdVUbnvkN6yHbhp1HqEWk1/7PHvJoxBCiBYuZxNOk5/nfZqDIWfjOcVr1qzxbxzNlCRCHhrTtyMWP6/eCjSbzljyKIQQooUrOAjaz1MtnA5Xv2cJCwujqKiIcePGMXDgQJKSkvjss88AuO+++3jllVcq6z7yyCM8/fTTNdZvySQR8pDJUDw0qT/BZv+MCgWbTTx4WT9Z8iiEEK2Jw4ry+4kOGmqYkxQUFMSiRYvYsGEDy5cv56677kJrzfTp0/nwww8r63344YdMnz69xvotmSyfr4epA7vw8YZD/LQvH7vTd//hAwzFgO7tmDqwi8/6EEII0QRMFrSPT1M/lwKTpdorWmseeOABVq1ahWEYHD58mGPHjjFgwAByc3M5cuQIx48fp3379nTr1g2bzVZt/c6dO/v5PXmPJEL1oJTiuempjH92JYVlnp0M3BDBFhPPT09F+f1/FiGEED4V2Q2UZw9jCpVic2AgWwMt7LKYKVOKIK3pbbWRUG4lqbyccE9GYwyTq99qzJ8/n+PHj7N+/XrMZjPx8fGUlZUBMG3aNBYuXMjRo0eZPn16nfVbKkmE6qlTRBDvz72Q3/5zLSVW72+GFWIx8f7cC+kYEeT1toUQQjSx2GQMe+3Hamy1mMmIjGB5SDBmoEwp7FX+MF6qNUFaYwPGlJSSXnCaBGstu1TbSiE2pdpLBQUFdOzYEbPZzPLly884hHX69OnMnTuXEydOsHLlyjrrt1QyR6gBErtE8uH/G0p4UAABXprDE2AoIoIC+PD/DZUNFIUQorUKisQeWP12LMVK8VBMFOmxnfg6NIRyw6DIMM5IggDsSlFkGJQbBl+HhpAe24mHYqIorukpQkg0BEWcU6yUYubMmWRmZpKUlMS8efPo27dv5fWEhAQKCwvp0qULsbGxALXWb6lkRKiBErtE8u2do7hjQRY/HzjVqOM3gs0mBnRvx/PTU2UkSAghWrmC+EuJ2b3gjAnMe80BzOnciUJDYa3HUQZOpShTiq9CQ/g+OJg3jx6jp63K1A1TIKTOOOe+vLw8oqKiiImJYe3atTW2v3nz5jNe11W/JZIRoUboGBHE/Bsv4PHJibQPMdd7n6FQi4n2IWb+MiWR+TdeIEmQEEK0ASd7XYPrSG2XveYAZsZ2Jt9k1CsJqqrcMMg3GcyM7cxe81ljHIPnnvHyyJEjDB06lLvvvrtBfbU2MiLUSEoprh7UlckDurBiZy6vrfyFnw+cItBsoDWUWh1oXP/kgy0mlHIdmzGgeztuGnUeo/t0lCXyQgjRhthDOsF5Y2DPtxRrO3M6d6LYUI1eTaaVotiAOZ07sfjQEUJVAJw3FiLPXIEcFxfHrl27GtVXayKJkJeYDMW4fp0Y168TNoeTXccK2XK4gNzT5ZTbnQQGGHSMCCSxSyS9O4XLsRlCCNGWXfECvDSIv0WEU+SFJKiCVooiQ/FEdHseO2119SNqJYmQD5hNBglxkSTEyaRnIYQQ1QjvzNYx97Bkx+uUe/mpQLlh8FVoKDPS7qB/eCevtt0a+WxYQin1plIqVym1pUrZU0qpHUqpTUqpRUqpdlWu3a+U2qOU2qmUmlClfKK7bI9S6r4q5T2UUj+6yxcopSzu8kD36z3u6/G+eo9CCCFEQ2VYD2P10dQIq6HIKD/sk7ZbG18+n8kAJp5V9g2QqLVOBnYB9wMopfoD/wUkuO95VSllUkqZgFeAS4H+wAx3XYAngee01ucDJ4Eb3OU3ACfd5c+56wkhhBDNRqG1kOUHl+OrU8ecwLKDyyi0Fvqoh9bDZ4mQ1noVkH9W2dda64p1fT8AXd1fXwV8oLUu11rvA/YAQ9wfe7TWe7XWVuAD4Crl2nJ5LLDQff/bwOQqbb3t/nohME7JFs1CCCGakc0nNmM2zD7tw2yY2XJiS90Vz1JcXMz48eMBuOiii7DbfXeSQnPQlDN25wD/cX/dBah6NO4hd1lN5dHAqSpJVUX5GW25rxe4659DKfU7pVSmUirz+PHjjX5DQgghhCe2nthKmd23R1OUOcrYlret3vetXbuWoUOHcvLkSUJDQwkIaN3TiZskEVJKPQjYgflN0X8FrfW/tNZpWuu0Dh06NGUoQggh2pBdJ3dh174dabE77ezI31Hj9Xnz5pGcnExKSgrXXXcdv/zyC6mpqVx77bW89957DBo0iI0bN5Kamkpubq5PY21Kfk/zlFLpwOXAOK0rT4s7DFQ9Ea6ru4wayvOAdkqpAPeoT9X6FW0dUkoFAJHu+kIIIUSzUObwz0Gl5Y7qzzXbunUrjz/+OGvWrCEmJob8/HyioqLIyspi0qRJzJs3j5dffpm0tDQmTZrkl1ibil9HhJRSE4E/AFdqrUuqXPoc+C/3iq8eQC/gJ2Ad0Mu9QsyCa0L15+4Eajlwjfv+2cBnVdqa7f76GmBZlYRLCCGEaHJBJv+cJBBoCqy2fNmyZUybNo2YmBgAoqKiKq/l5uYSHR3Npk2bSEmp/rDW1sSXy+ffB9YCfZRSh5RSNwAvA+HAN0qpLKXUawBa663Ah8A24CvgVq21wz3acxuwBNgOfOiuC3AvcKdSag+uOUBvuMvfAKLd5XcClUvuhRBCiOagd/veBKi6H8oop6brcc3gnU6GbXMyeKeTrsc1yln33/cBRgB9ozw/FPWmm24iMTGR3bt3k5qayldffcXll1/Oc88953EbLZHPHo1prc895e3XZKW6+n8B/lJN+ZfAl9WU78W1quzs8jJgWr2CFUIIIfwoISaBoIAgimxF51xTTs2AXzRX/uik1xGwG+A0QGnQCgwnBDhhdxx8foHBz+cpdDX7EQWZgugf3f+ccoCxY8cyZcoU7rzzTqKjo8nPz+e1117jo48+4sCBA1x99dX84Q9/4MMPP/T6e29uWvdUcCGEEKIZSopJwua0nVPe96Dm9s8chJRDsPtwerOj+jb6H4Qex5yUBMILV5nY0e3MZMjmtJEYk1jtvQkJCTz44IOMGjUKk8nEgAEDyMjIYOXKlcyaNYvvvvuOUaNGNeo9thSSCAkhhBB+Fm4JZ0y3MXy9/2uc2olyatKXOhm7URNYj8VkwVbXx4MfOFiWosgYb6ANhaEMxnYfS7glvMZ7Z8+ezezZs88oe/nllwEYMuScBy6tlpz8KYQQQjSB9MR0LIYF5dTc9YmTMfVMgqoKtMOYja52lFNjMSykJ6R7Nd7WShIhIYQQogkkRCcwIX4CN3yrSN6nCWrktkJBdkjep7nhW8XE+Ik1zg8SZ5JHY6JuZQWQswkKDoLDCiYLRHaD2GQIimzq6IQQosX6X9MEDmV9jMVLeysG2WF0lp1uAWcf9SlqIomQqF7BIfjpddj4PpTkgTkYnA5cR/kZYJjAVgoh0ZAyA4bcCJFd62pVCCGEm3Y4yL/vj15LgipY7JB374O0X/YtymTybuOtkCRC4kyFR+H/bodflgPaNQIEUH7u6gYAio7CD6/AD6/CeWPgihchvJPfwhVCiJaqaNUqHEW+OR3eWVhI0XffET56tE/ab01kjpBw0Ro2LoCXBsGeb8FR/msSVBeH1VV/z7fw0kBXO7KZtxBC1CrvjTfRxSV1V2wAZ0kJ+a/XuHWfqEISIeFKWv5zL3xxB1iLoJq9LTzitLnu/+IOV3uSDAkhRLW0w0HZxo0+7aN040a0o4ZNiGoQHx/PiRMnfBRR8ySJUFunNfznD/DzO2Dz0l8mthJXe5IMCSFEtax796LMZp/2ocxmrPv2+bSP1kASobZu04fw87veS4IqVCRDmz/ybrtCCNEKlGdng+HjX8GGgTU7u9pL2dnZ9O3bl5kzZ9KvXz+uueYaSkpcvwdeeuklBg4cSFJSEjt27ADgkUce4brrrmPo0KH06tWLf//7376N3Y8kEWrLCo/C4ju9nwRVsJXAF/8Lhcd8074QQrRQ2mr1/Yi51jjLy2u8vHPnTm655Ra2b99OREQEr776KgAxMTFs2LCBm2++maeffrqy/qZNm1i2bBlr167l0Ucf5ciRI76N308kEWrL/u92sNf8P4lX2Mtd/QghhKikLBZQ5x6U6t1OFEZgYI2Xu3XrxvDhwwG49tpr+f777wGYOnUqAIMGDSK7yojSVVddRXBwMDExMYwZM4affvrJd7H7kSRCbVXBIdcS+YZOjPaU0wa/LIOCw77tRwghWpDA+HhwOn3bidOJJT6+xsvqrESs4nWgO3kymUzY7fY667d0kgi1VT+9DvhxIvO61vM8WQghGsvSsyfaVvcfohooDYrmRFQCuTGpnIhKoDQo2qOf3tpmw9KjR43XDxw4wNq1awF47733uOiii2pt77PPPqOsrIy8vDxWrFjB4MGDPYii+ZMNFduqje97vk9QYznKIet9GP+If/oTQohmTplMBKWkUJqZWe3102HdONhtHMdjkl319a+jR1q5xjA6nNhEt4PfElF0sNo2glNSat1Zuk+fPrzyyivMmTOH/v37c/PNN/PSSy/VWD85OZkxY8Zw4sQJ/vjHPxIXF1fn+2wJJBFqi8oKXMdm+FNJHpSdhqAI//YrhBDNVPQNczi8fdsZmypazeFs6zeLU5Hn4VQBruOManCsw0COxyTT7tQv9N8xD4vt112qjZAQom68odb+AwICePfdd88oqzonKC0tjRUrVlS+Tk5OZt68eR6+u5ZDHo21RTmbXGeH+ZM5GHJ8u3mYEEK0JGEjR2IKC698fTLyfNZe8DAn2/XGaQqsNQkCwDDhNAVysn1v132R5/96KTycsBEjfBV6qyKJUFtUcNB9gKofOR2ufoUQQgCux2NdnnkaFRTEycjz2Zh8C46AYLRRv4c12gjAERDMxuRbOBl5Piow0NVuLY/F4uPj2bJli8d9PPLII9x99931iqulkEdjbZHDiusUeX/S/puTJIQQLURIWhpBk6ez6VCSaxSoEZymQDYl3cTErpsJSUvzUoStn4wItUUmC/7/T6/c/QohhKhqU/sJOAO88/PRGWBhc9QEr7TVVkgi1BZFdqv72bO3GSZXv0IIISrl7j9Nzi8FaLzzM1lj4sieAnL3n/ZKe22BJEJtUWwy2Er926etFGJT/NunEEI0c1lLD+KweXeqgsPuJGupzMn0lCRCbVFQJIRE+7fPkGhZOi+EEFVordm38bjXjxzTTtzt+m7T3LCwsHrfM2zYsGrL09PTWbhwYWNDajBJhNqqlBn+m7NjCoTUGf7pSwghWojTJ8p82n5hnm/bP1vV4ziqs2bNGj9FUj+SCLVVQ24E/HhOzOC5/utLCCFagJNHizEM3/wcNgxFfk5xrXUmT57MoEGDSEhI4F//+hfgGul58MEHSUlJ4cILL+TYsWMA7Nu3j6FDh5KUlMRDDz1U2caKFSsYMWIEV155Jf379wfg2WefJTExkcTERJ5//vnKuhWjSFprbrvtNvr06cP48ePJzc316nuvL0mE2qrIrnDeGDDMvu3HMMN5YyGyi2/7EUKIFsZhc/rsxEftbr82b775JuvXryczM5MXX3yRvLw8iouLufDCC9m4cSMjR47k3/92nRN5++23c/PNN7N582ZiY2PPaGfDhg288MIL7Nq1i/Xr1/PWW2/x448/8sMPP/Dvf/+bn3/++Yz6ixYtYufOnWzbto158+Y1+UiRJEJt2RUvQEDj9q2oU0Cgqx8hhBBnMJkNn43LK3f7tXnxxRcrR34OHjzI7t27sVgsXH755QAMGjSo8siN1atXM2OGa4rDddddd0Y7Q4YMoYf7cNfvv/+eKVOmEBoaSlhYGFOnTuW77747o/6qVauYMWMGJpOJuLg4xo4d64V33HCSCLVl4Z1h0rNgDvFN++YQuPw5CO/km/aFEKIFa985FKfTN2NCTqcmKja0xusrVqxg6dKlrF27lo0bNzJgwADKysowm80o5UrPTCbTGfN+KsrPFhpacz8tgSRCbV3yb2HAdd5PhswhMHAWJE3zbrtCCNFKRMQE+bT98Oia2y8oKKB9+/aEhISwY8cOfvjhh1rbGj58OB988AEA8+fPr7HeiBEj+PTTTykpKaG4uJhFixYx4qwzz0aOHMmCBQtwOBzk5OSwfPnyerwr75NEqK1TCi590rvJUEUSNPEJV/tCCCHOoZSiR0oHr/+YVAbudmtueOLEidjtdvr168d9993HhRdeWGubL7zwAq+88gpJSUkcPny4xnoDBw4kPT2dIUOGcMEFF3DjjTcyYMCAM+pMmTKFXr160b9/f2bNmsXQoUPr9wa9TPlyn4GWJC0tTWdmZjZ1GE1Ha9j8EXzxv2AvB6et/m0YZtecoMufc40ESRIkhBDn2L59O/369QNcO0svemYDduu5E5u1tuG07sZhP4C2H0NjQ2FGBXTCFNAdw9ILpc5d8BJgMZh69yA6dA8/51prVfV7WkEptV5rXeeha3LoqnBRyvWYrMco+L/b4ZdleHxQqskCKNfqsCtekDlBQgjhoY6/iSD2/HYc3nkSp8M1MKG1HXvpGhzlWbimPf/6h6kGtDUPp3U3lCzFFJhKQPAwlHL9OjdMirhe7dpUEtRYkgiJM4V3gv/+AAoOwbrXIet9KMkDczA4Hbj+N1Sus8Nspa4do1NnuPYJkiXyQghRb+PT+/Pun9bidDhwOvKwFn4CuhSobYNCV3LkKM/CYd2JJXwqhimaALPBuNn9/RJ3ayGJkKheZFcY/4jro6wAcjZBwUHXCJHJ4jpANTZFjs0QQohGComwMOmWZD5/fhllJ98HPBiJr2QHXYj19PuERM/kslvGEhLhp1MDWglJhETdgiKhx4i66wkhhGiQTvGhYP+c+iVBVVnB8X90ir/Cm2G1CbJqTAghhGhiaz6aj7W0qFFtWEsKWfNRzUvbRfUkERJCCCGakK2sjJ+/+gK7tbxR7dit5fy85AtsZf49bLWlk0RICCGEaEK7flyN8tLhq0opdv242qO6w4YN80qfLZ0kQkIIIUQTOrBlo9dGcWxlZRzcusmjur487LTq0RzNnSRCQgghRBM6tnePV9s76mF7YWFhaK257bbb6NOnD+PHj+eyyy5j4cKFAMTHx3PixAkAMjMzGT16NADFxcXMmTOHIUOGMGDAAD777DMAMjIyuPLKKxk7dizjxo1j1qxZfPrpp5X9zZw5s7JucyKrxoQQQogm1Ni5Qee0V+756NKiRYvYuXMn27Zt49ixY/Tv3585c+bUes9f/vIXxo4dy5tvvsmpU6cYMmQI48ePB2DDhg1s2rSJqKgoVq5cyXPPPcfkyZMpKChgzZo1vP322416b77gsxEhpdSbSqlcpdSWKmXTlFJblVJOpVTaWfXvV0rtUUrtVEpNqFI+0V22Ryl1X5XyHkqpH93lC5RSFnd5oPv1Hvf1eF+9RyGEEKKxAiyB3m0v0PPDXFetWsWMGTMwmUzExcUxduzYOu/5+uuveeKJJ0hNTWX06NGUlZVx4MABAC6++GKioqIAGDVqFLt37+b48eO8//77XH311QQENL/xF18+GssAJp5VtgWYCqyqWqiU6g/8F5DgvudVpZRJKWUCXgEuBfoDM9x1AZ4EntNanw+cBG5wl98AnHSXP+euJ4QQQjRLnXqe79W3V19eAAAgAElEQVT2OnupvYCAAJxO1xloZVXmMGmt+fjjj8nKyiIrK4sDBw5UnvMVGhp6RhuzZs3i3Xff5a233qpzpKmp+CwR0lqvAvLPKtuutd5ZTfWrgA+01uVa633AHmCI+2OP1nqv1toKfABcpVxH6o4FFrrvfxuYXKWtirG3hcA4VdsRvEIIIUQT6p6YgjnI81Gc2piDguiWkOxx/ZEjR7JgwQIcDgc5OTksX7688lp8fDzr168H4OOPP64snzBhAi+99BIVh7b//PPPNbafnp7O888/D0D//s3z6I/mMlm6C3CwyutD7rKayqOBU1pr+1nlZ7Tlvl7grn8OpdTvlFKZSqnM48ePe+mtCCGEEJ7rfcFwtFN7pS2tNb0vGO5RXaUUU6ZMoVevXvTv359Zs2YxdOjQyusPP/wwt99+O2lpaZhMpsryP/7xj9hsNpKTk0lISOCPf/xjjX106tSJfv36cf311zf8TflY83tY50da638B/wJIS0vzzr9CIYQQoh7MQUEMmHh5ozdVDLAEMmDC5R6NLuXl5REVFYVSipdffrmyPD09vfLrESNGsGvXrnPuDQ4O5p///Oc55enp6WfcD1BSUsLu3buZMWOG52/Ez5rLiNBhoFuV113dZTWV5wHtlFIBZ5Wf0Zb7eqS7vhBCCNEsDZs2k+CICKD6mRwKhdkIRNVwHaUIiYhk2G+vrbOvI0eOMHToUO6+++5GRFy3pUuX0q9fP37/+98TGRnp074ao7mMCH0OvKeUehaIA3oBP+H6F9FLKdUDV4LzX8B/a621Umo5cA2ueUOzgc+qtDUbWOu+vkxXPMgUQgghmqEAi4Wr73+U9/54F9aSUkBjNgKJD0ukT+QQQkzhOHFgYKLEUcjOgp/ILtqCzVkOSmEJDmbq/X8mwGyus6+4uLhqR3rAtReQt4wfP579+/d7rT1f8VkipJR6HxgNxCilDgEP45o8/RLQAVislMrSWk/QWm9VSn0IbAPswK1aa4e7nduAJYAJeFNrvdXdxb3AB0qpx4GfgTfc5W8A7yil9rj7+y9fvUchhBDCW6K7duO/H3uGj//2J+J1P3qHpQGaAMMCgMn9Kzs0IILk9iNJbj+KXUWZ7Fc7mHr/n4nu2q2W1kVNlAyWuKSlpenMzMymDkMIIUQrt3379srl5mfTWpP3wQ5KNuVi6LpnrziVk5CUjkRP70tbXiBd3fdUKbVea51Wwy2VmsscISGEEKLNO730AOXb8j1KggAMbVC+NZ/TSw/4OLLWSxIhIYQQohlwltopXHkIbXPW6z5tc1K48hDOspZz0GlzIomQEEII0QwUrz9GQ59uKQXFmce8G1AVf/3rXyu/zs7OJjEx0Wd9+ZskQkIIIUQzUPRd/UeDKmibk6LvDnk5ol9VTYRaG0mEhBBCiCamHU4cp62NasNx2op2eLYAKjs7m759+zJz5kz69evHNddcw5dffsnkyZMr63zzzTdMmTKF++67j9LSUlJTU5k5c6arL4eDuXPnkpCQwCWXXEJpaSkAWVlZXHjhhSQnJzNlyhROnjwJwOjRo7n33nsZMmQIvXv35rvvvmvUe/UmSYSEEEKIJqatTjAauerLUGirw+PqO3fu5JZbbmH79u1ERESwdetWduzYQcWRUxUHpT7xxBMEBweTlZXF/PnzAdi9eze33norW7dupV27dpVnkc2aNYsnn3ySTZs2kZSUxJ///OfK/ux2Oz/99BPPP//8GeVNTRIhIYQQookpiwGNPW/MqVEWU9313Lp168bw4a5zya699lpWr17Nddddx7vvvsupU6dYu3Ytl156abX39ujRg9TUVAAGDRpEdnY2BQUFnDp1ilGjRgEwe/ZsVq1aVXnP1KlTz6jfXDSXnaWFEEKINkuZDEwRFhwFDX88ZoqwoEyejyqdve+QUorrr7+eK664gqCgIKZNm0ZAQPVpQmBg4K/9mkyVj8ZqU3GPyWTCbm8+K9xkREgIIYRoBsJGdEWZG/ZrWZkNwkZ0rdc9Bw4cYO3atQC89957XHTRRcTFxREXF8fjjz9+xonxZrMZm81Wa3uRkZG0b9++cv7PO++8Uzk61JxJIiSEEEI0A6GDOtHQwx60htC0TvW6p0+fPrzyyiv069ePkydPcvPNNwMwc+ZMunXrdsZOzb/73e9ITk6unCxdk7fffpt77rmH5ORksrKy+NOf/lT/N+NncsSGmxyxIYQQwh9qO2Kj4Jv9FK2q3zJ6ZTYIG9mVyIt/4/E92dnZXH755WzZsuWca7fddhsDBgzghhtu8Li9ptaYIzZkjpAQQgjRTESM747jZBmlm094lAwps0FwUgwR47t7pf9BgwYRGhrKM88845X2WgJJhIQQQohmQilF+2m9MbUPonDlIZSCIlsJJ4xCrNixEECMM5wwcwhaQ9jIrkSM717vA1fj4+OrHQ1av369t95KiyGJkBBCCNGMKKWIGN+d412tfPfNCg7l52BohQYU4FSarlGxjLhkNF37ev44TFRPEiEhhBCiGXE6nSxevJhNmzZVrtRynDXgsz//MEc+/pDk5GQmTZqEYcjap4aS75wQQgjRTGitz0mCamKz2di0aROLFy/2U3StkyRCQgghRDOxb98+j5KgChXJ0N69e30cWesliZAQQgjRTHz//fceJ0EVbDYbq1evrtc92dnZJCYmnlOekZHBkSNHvNJWSyGJkBBCCNEMFBQUsH///gbdW3HWV2M1JBFq6SQREkIIIZqBnJycGs/2qktAQAA5OTn1usfhcDB37lwSEhK45JJLeOedd8jMzGTmzJmkpqZSWlpKfHw8999/P6mpqaSlpbFhwwYmTJjAeeedx2uvvdagWJsbSYSEEEKIZqC8vJyGnvagtaa8vLxe9+zevZtbb72VrVu30q5dO5RSpKWlMX/+fLKysggODgage/fuZGVlMWLECNLT01m4cCE//PADDz/8cINibW5k+bwQQgjRDAQGBtZ7Y8QKSqkzToT3RI8ePUhNTQVcO0pnZ2dXW+/KK68EICkpiaKiIsLDwwkPDycwMJBTp041KN7mREaEhBBCiGYgNjYWu93eoHvtdjuxsbH1uqdq4mQymWrsu6KeYRhn3GMYRoPjbU4kERJCCCGagcjISH7zm4btFB0fH09kZGSjYwgPD6ewsLDR7bQkkggJIYQQzcRFF12E2Wyu1z1ms5nhw4d7pf/09HRuuummysnSbYFq6MSs1iYtLU1nZmY2dRhCCCFaue3bt9OvX79qr2mt+eKLLzzeVNFsNpOcnMwVV1zh7TBblOq+p0qp9VrrtLrulcnSok0otBay+cRmtp7Yyq6TuyhzlBFkCqJ3+94kxCSQFJNEuCW8qcMUQrRxSikmTZoEUGcyVJEEVdQXDSOJkGjVtuZtJWNLBssPLsdsmCmzl2HXv07uW7p/KUEBQdicNsZ0G0N6YjoJ0QlNGLEQoq0zDIMrrriChIQEVq9eTXZ2NgEBAWitUUpht9uJj49n+PDh9OzZs6nDbfEkERKtUrGtmL/9+DeWZC/B6rTi1E7KHefusWHXdopsRQB8vf9rVhxcwYT4Cdx/wf2EmkP9HbYQQlTq2bMnPXv2pKCggJycHMrLywkMDCQ2NtYrE6OFiyRCotXZe2ovc5bModBWiNVh9fg+p3ZS5ijjq+yv+P7w97w54U16tpO/toQQTaeoeDdHjrzF8eNLcDhLMRnBaD0BU8D1hIX2aurwWgVZNSZalb2n9jLzy5nkl+XXKwmqqtxRTn5ZPjO/nMneU3KisxDC/7R2snPnI6xbN5kjOQux2U/hdJZjs5/iSM5C1q2bzM6dj6C1s6lDbfEkERKtRrGtmDlL5lBsK0bTuNWQGn1Ge0II4U+7dj3KkZyFOJ1lgOOsqw6czjKO5Cxk165HmyK8VkUSIdFq/O3Hv1FkK2p0ElRBoymyFfHEj094pT0hhPBEUfFujuR8hNNZ+z4+TmcpR3I+orh4j58ia50kERKtwta8rSzJXlLthOjGKHeU81X2V2zL2+bVdoUQoiYHD7yF01n3HkIATqeNAwff8noMo0ePrvHsMYCMjAweeeQRr/fbFCQREq1CxpYMrM6GzQmqi9VpJWNrhk/aFkKIsx0/voRzH4fVxMHx3K8a1M+8efNITk4mJSWF6667rkFttAayaky0eIXWQpYfXI7TR5MGndrJsgPLKLQWyqaLQgifc9TxSKyx9QG2bt3K448/zpo1a4iJiSE/P7/ebbQWkgiJFm/zic2YDbPXH4tVZTbMbDmxhaFxQ33WhxBCAJiMYJxOz3+emYzgevexbNkypk2bRkxMDABRUVG89dZbvPDCCwDs2bOHyy67DIvFQo8ePVi0aBF5eXmMGzcOgPz8fKxWK59++ikA77zzDklJSfWOozmQREi0eFtPbKXMXubTPsocZWzL2yaJkBDC5zp0mMCRnIV49ngsgA4dJ3ql3+uvv57rr78ecM0RysjIID4+vvJ6dHQ0WVlZgGuOUHZ2dquYJyRzhESLt+vkrjOOzfAFu9POjvwdPu1DCCEAunW/HsPw7AR6wwige7fr693H2LFj+eijj8jLywOQR2OeUkq1B7pprTf5KB4h6q3M4dvRoAq+fPRWX9rhwLp3L+XZ2WirFWWxEBgfj6VnT5TJ1NThCSEaISy0F3Gx09z7CNU8/8cwgomLvYbQ0PPr3UdCQgIPPvggo0aNwmQyMWDAADIyMhoRdctVZyKklFoBXOmuux7IVUqt1lrf6ePYhPBIkCnIL/0EmgL90k9NtMNB0apV5L3xJmUbN6LMZjAM0BqUAqcTbbMRlJJC9A1zCBs5UpIiIVqo3r3/BODeT8jGmY/JTBiGmbjYayrrNcTs2bOZPXt2tddWrFhR673p6ekN7re58WREKFJrfVopdSMwT2v9sFJKRoREs9G7fW+W7l/q08djAUYAfaP6+qz9upRkZnL4rrtxFBWii0sA0Lbq9xkpzczk8PZtmMLC6fLM04SkpfkzVCGEFyhl0KfPI3TpOpODBzM4nvtV5VljHTpOpHu36xs0EiTO5ckcoQClVCzwW+ALTxtWSr2plMpVSm2pUhallPpGKbXb/bm9u1wppV5USu1RSm1SSg2scs9sd/3dSqnZVcoHKaU2u+95USmlautDtF4JMQkEBfh2VCjIFET/6P4+7aM62uHg6GOPceDGudiPHatMguq8r7gE+7FjHLhxLkcfewzt8HRPEiFEcxIW2ot+ff/CyJHrGTN6GyNHrqdf379IEuRFniRCjwJLgD1a63VKqZ7Abg/uywDOnsp+H/Ct1roX8K37NcClQC/3x++Af4ArqQEeBi4AhgAPV0ls/gHMrXLfxDr6EK1UUkwSNg93YW0om9NGYkyiT/s4m3Y4OPQ/t3Pq40/QZQ2bB6XLyjj18Scc+p/bJRkSQohq1JkIaa0/0lona61vcb/eq7W+2oP7VgFnT0O/Cnjb/fXbwOQq5fO0yw9AO/co1ATgG611vtb6JPANMNF9LUJr/YPWWgPzzmqruj5EKxVuCWdMtzEYyjeLIA1lMLb7WL9vpnjsr3+lePXqBidBFXRZGcWrV3Psr3/1UmRCCH85Wm7jyb05XLB2G4nfb+GCtdt4cm8OR8t9+8dfW+LJZOkOuEZe4qvW11rPaUB/nbTWOe6vjwKd3F93AQ5WqXfIXVZb+aFqymvr4xxKqd/hGoGie/fu9X0vohlJT0xnxcEVPllBZjEspCeke73d2pSsW9eokaCz6bIyTi38mIhLL5U5Q0K0AFprXjxwjGezjwFQ7nQdJn3CBq8ezOXVg7ncGd+J/+neCffMENFAnvwJ/RkQCSwFFlf5aBT3SI53jglvYB9a639prdO01mkdOnTwZSjCxxKiE5gQP8HrK7sCTYFMjJ/o1/lB2uHg8N33eC0Jqmy3vJzDd90tj8iEaAFePHCM57NzKXfqyiSoQkXZ89m5vHjgWBNF2Hp4kgiFaK3v1Vp/qLX+uOKjgf0dcz/Wwv05111+GOhWpV5Xd1lt5V2rKa+tD9HK3X/B/YSZw1B4568jhSLMHMb9F9zvlfY8VbRqFY6iQp+07SwspOi773zSthDCO46W23g2+xilztrPTyx1Onk2+1iDHpMNGzasoeG1Op4kQl8opS7zUn+fAxUrv2bjGm2qKJ/lXj12IVDgfry1BLhEKdXePUn6EmCJ+9pppdSF7tVis85qq7o+RCsXag7lzQlvEmoObXQypFCV7YWYQ7wUoWfy3njT49Vh9eUsKSH/9Td80rYQwjsyDp+oV/159awPsGbNmnrf01p5kgjdjisZKlNKFbo/Ttd1k1LqfWAt0EcpdUgpdQPwBHCxUmo3MN79GuBLYC+wB/g3UDExOx94DFjn/njUXYa7zuvue34B/uMur6kP0Qb0bNeT+ZfNJyooqsGPyQJNgUQFRTH/svn0bNfTyxHWTjsclG3c6NM+SjdulMdjQjRji46dPOdxWE3KnZpPck/Wu4+wsDDANRfpnnvuITExkaSkJBYsWABATk4OI0eOJDU1lcTERL5rxSPJdU6W1lo3aKmM1npGDZfGVVNXA7fW0M6bwJvVlGcC56xn1lrnVdeHaDt6tuvJ4qmLeeLHJ/gq+yusTitOXfsQM7hWh1kMCxPjJ3L/Bff7fSQIwLp3L8psrnGzRG9QZjPWffsIPF/2IRGiOSp21P3zqqoie/3qV/XJJ5+QlZXFxo0bOXHiBIMHD2bkyJG89957TJgwgQcffBCHw0FJiW9GqZsDj84aU0pdCYx0v1yhtfZ4Y0UhmkKoOZTHLnqMGf1mkLE1g2UHlmE2zJQ5yrA7f92BOsAIIMgUhM1pY2z3saQnpDfJxokVyrOzXcdm+JJhYM3OlkRIiGYq1GRwoh5/C4UFNPxnxvfff8+MGTMwmUx06tSJUaNGsW7dOgYPHsycOXOw2WxMnjyZ1NTUBvfR3HmyfP4JYDAw3110u1JquNbavzNIhWiA/tH9+fvIv1NoLWTLiS1sy9vGjvwdlDvKCTQF0jeqL/2j+5MYk+j3fYKqo61W19lhPu1E4yxvPgfICiHONLVTe149mOvR47FAQzG1o/cPUBg5ciSrVq1i8eLFpKenc+eddzJr1iyv99MceDIidBmQqrXr2YJS6m3gZ0ASIdFihFvCGRo3lKFxQ5s6lFopi8V1gKpPO1EYgU17gKwQomazu8Tw6kEPFzxrmNUlpsF9jRgxgn/+85/Mnj2b/Px8Vq1axVNPPcX+/fvp2rUrc+fOpby8nA0bNrTpRAigHb/uEh3po1iEaPMC4+OhjiWzjeZ0YomP920fQogG6xxo5s74TjyfnVvrEvpgw+CO+I50DjQ3uK8pU6awdu1aUlJSUErx97//nc6dO/P222/z1FNPYTabCQsLY968eQ3uo7lTuo5heKXUDFwrr5YDCtdcofu01gt8H57/pKWl6czMzKYOQ7Rx2uFgZ+oAn0+W7pP1M8pk8lkfQoiabd++nX79+tVap6adpcH1OAwNd/aQnaUrVPc9VUqt11rXuZW+J6vG3ldKrcA1TwjgXq310YYEKoSonTKZCEpJodSHSXlwSookQUI0c0opbv9NZ6Z3jmbe4RN8knuSIruTsACDqR3bM6tLTKNGgsSvakyElFJ9tdY7lFID3UUVZ3vFKaXitNYbfB+eEG1P9A1zOLx9m082VTRCQoi68QavtyuE8I3OgWb+0DOWP/SMbepQWq3aRoTuxHUg6TPVXNPAWJ9EJEQbFzZyJKawcOy+SITCwwkbMcLr7Qoh6kdrLY+0vKSuKT51qTER0lr/zv3lpVrrM05/VEoFNapXIUSNlMlEl2ee5sCNc7168KoKDKTLM0/LYzEhmlhQUBB5eXlER0dLMtRIWmvy8vIICmp4WuLJqrE1wEAPyoQQXhKSlka7q6dy6uNPvJIMqaAg2l09lZC0OucNCiF8rGvXrhw6dIjjx483dSitQlBQEF27dq27Yg1qmyPUGegCBCulBkDlKZYRgP/PHhCijen0wAPYjh6jePXqRiVDKiiI0IsuotMDD3gxOiFEQ5nNZnr06NHUYQi32kaEJgDpQFfg2SrlhYD8RBXCx5TJRNcXX+DYX//a4JEhFRhIu6un0umBB+SRmBBCVMOTfYSu1lp/7Kd4mozsIySas5LMTA7fdTfOwkKcHhx+aISEYISH0+WZp+VxmBCiTfLmPkIfK6UmAQlAUJXyRxsXohDCUyFpaZy/7FuKvvuO/NffoHTjRpTZ7DqgVWvXsRxOJ9pmIzglhagbbyBsxAgZBRJCiDp4cujqa7jmBI0BXgeuAX7ycVxCiLMok4nw0aMJHz0a7XBg3bcPa3Y2zvJyjMBALPHxWHr0kORHCCHqwZNVY8O01slKqU1a6z8rpZ4B/uPrwIQQNVMmE4Hnn0/g+ec3dShCCNGiGR7UKXV/LlFKxQE2QLa4FEIIIUSL58mI0BdKqXbAU8AGXLtKv+7TqIQQQggh/MCTydKPub/8WCn1BRCktS7wbVhCCCGEEL5X24aKY7XWy5RSU6u5htb6E9+GJoQQQgjhW7WNCI0ClgFXVHNNA5IICSGEEKJFq+3Q1Yfdn6/3XzhCCCGEEP5T56oxpdRf3ZOlK163V0o97tuwhBBCCCF8z5Pl85dqrU9VvNBanwQu811IQgghhBD+4UkiZFJKBVa8UEoFA4G11BdCCCGEaBE82UdoPvCtUuot9+vrgbd9F5IQQgghhH94so/Qk0qpjcB4d9FjWuslvg1LCCGEEML3PBkRAvgZMONaNv+z78IRQgghhPAfT1aN/RbXafPXAL8FflRKXePrwIQQQgghfM2TEaEHgcFa61wApVQHYCmw0JeBCSGEEEL4mieJkFGRBLnl4dlqMyGEqJPWmtMnyjh5tBiHzYnJbNC+cygRMUEopZo6PCFEK+dJIvSVUmoJ8L779XTgS9+FJIRoC3L3nyZr6UH2bTwOgGEoNKAAp1MD0COlA6nju9HxNxFNF6gQolXzZNXYPe6DVy9yF/1La73It2EJIVqrktNWlmZsI2fPKRw2J1rXXHdP5jH2bTxO7PntGJ/en5AIi/8CFUK0CZ5Mlg4FPtNa3wn8E3Aopcw+j0wI0eoc3nWSd/+0lsM7T2K31p4EAWgNdquTwzvd9+066Z9AhRBthidzfVYBgUqpLsBXwHVAhi+DEkK0Pod3neSLlzdiK3PgdNSRAZ3F6dDYyhx88fJGSYaEEF7lSSKktNYlwFTgH1rraUCCb8MSQrQmJaetLH51E3ars1Ht2K1Ovnx1EyWnrV6KTAjR1nmUCCmlhgIzgcXuMpPvQhJCtDZLM7bhsDUuCapgtzn59u1tXmlLCCE8SYTuAO4HFmmttyqlegLLfRuWEKK1yN1/mpw9p+r9OKwmTofmyO5T5O4/7ZX2hBBtW52JkNZ6pdb6SveZYwZwQmv9P36ITQjRCmQtPei10aAKDruTrKUHvdqmEKJt8mTV2HtKqQj36rEtwDal1D2+D00I0dJprdm38Xidq8Pq3a4Td7teblgI0eZ48misv9b6NDAZ+A/QA9fKMSGEqNXpE2U+bb8wz7ftCyFaP08SIbN736DJwOdaaxuuU+iFEKJWJ48WYxi+OSbDMBT5OcU+aVsI0XZ4kgj9E8gGQoFVSqnfAI2apaiUul0ptUUptVUpdYe7LEop9Y1Sarf7c3t3uVJKvaiU2qOU2qSUGlilndnu+ruVUrOrlA9SSm123/OikgOLhGgSDpvTZ381aXf7QgjRGJ5Mln5Ra91Fa32ZdtkPjGloh0qpRGAuMARIAS5XSp0P3Ad8q7XuBXzrfg1wKdDL/fE74B/udqKAh4EL3G09XJE8uevMrXLfxIbGK4RoOJPZwFd/hSh3+0II0RieHLqKUmoSrk0Ug6oUP9rAPvsBP7o3aUQptRLXZo1XAaPddd4GVgD3usvnadesyB+UUu2UUrHuut9orfPd7XwDTFRKrQAitNY/uMvn8ev8JiGEH7XvHFp5gKq3OZ2aqNhQn7QthGg7PFk19hquE+d/j+uPsGnAbxrR5xZghFIqWikVAlwGdAM6aa1z3HWOAp3cX3cBqq6TPeQuq638UDXlQgg/i4gJqrtSI4RH+7Z9IUTr58m48jCt9SzgpNb6z8BQoHdDO9RabweeBL7GdXZZFuA4q47GDxOylVK/U0plKqUyjx8/7uvuhGhzlFL0SOmAt2fpKQN3uzL9TwjROJ4kQqXuzyVKqTjABsQ2plOt9Rta60Fa65HASWAXcMz9yAv351x39cO4RowqdHWX1VbetZry6uL4l9Y6TWud1qFDh8a8JSFEDVLHd/P6XB5TgMGAi7t7tU0hRNvkyU+nL5RS7YCngA24VpC915hOlVId3Z+745of9B7wOVCx8ms28Jn768+BWe7VYxcCBe5HaEuAS5RS7d2TpC8BlrivnVZKXeheLTarSltCCD/r+JsIYs9vh2HyzuiNYVLE9WpHh+7hXmlPCNG21TlZWmv9mPvLj5VSXwBBWuuCRvb7sVIqGtfo0q1a61NKqSeAD5VSNwD7gd+6636Jax7RHqAEuN4dV75S6jFgnbveoxUTp4FbgAwgGNckaZkoLUQTGp/en3f/tBanw1F35ToEmA3Gze7vhaiEEAKUbFHvkpaWpjMzM5s6DCFarcO7TvLFyxuxWxu+90+AxeDy21Lo0rt93ZWFEG2aUmq91jqtrnqyCYcQwi+69G7P5belYA4y1fsxmWFSWIJMkgQJIbxOEiEhhN906d2eax8dSpc+7QmwGHWuJlOGaxSoa9/2zHx0qCRBQgiv83RDxS649g6qrK+1XuWroIQQrVdIhIUr/yeV3P2nyVp6kH0bXVtXGIZC49qsrGITxh4pHUgd342Ov4louoCFEK1anYmQUupJXBsqbuPX/X40IImQEKLBOv4mgktuSEBrTWFeGfk5xThsTkxmg6jYUMKjg2SfICGEzxxQvj8AAB0qSURBVHkyIjQZ6KO1Lvd1MEKItkcpRURMMBExwU0dihCiDfJkjtBewOzrQIQQQggh/M2TEaESIEsp9S1QOSqktf4fn0UlhBBCCOEHniRCn7s/hBBCCCFaFU92ln5bKWXh14NWd2qtbb4NSwghhBDC9zxZNTYaeBvXGWMK6KaUmi3L54UQQgjR0nnyaOwZ4BKt9U4ApVRv4H1gkC8DE0IIIYTwtf/f3v3H2VXXdx5/fSYzmYRAQoQQQ36QpCZY0SgyBTQoW1GMomJR/F1o1pVusYpdd6t2K1ipa/vYSsXdlpaHv6BVKSAqC0iK2KrgA0oQBUGENOSn+YXQCYRkJpP72T/uGRhIMrmZzLn3Ts7r+Xjcx5zzveec+5nHheSd749zGlk11jUYggAy8yFcRSZJkg4CjfQILY+ILwL/WOy/F/DppJIkacxrJAj9AfBBYHC5/I+Avy2tIkmSpCZpZNVYH3BJ8ZIkSTpo7DUIRcTVmfmOiLiP+rPFniUzF5VamSRJUsmG6xG6oPj5pmYUIkmS1Gx7XTWWmRuKzfMzc/XQF3B+c8qTJEkqTyPL51+3h7Y3jHYhkiRJzTbcHKE/oN7z8xsRce+Qtw4Dbi+7MEmSpLINN0fo68B3gc8CHx/S/kRmPlZqVZIkSU0w3Byh3sxcBfwpsLGYGzQPeF9EHN6k+iRJkkrTyByhbwK7IuIFwOXAbOq9RZIkSWNaI0GolpkDwFnA/8nM/wHMKLcsSZKk8jUShHZGxLuBc4AbijYfuipJksa8RoLQUuAVwGcy85GImAf8Q7llSZIkla+RZ409EBEfA+YU+48Af1l2YZIkSWXbZ49QRLwZ+Clwc7H/soi4vuzCJEmSytbI0NingBOB/wDIzJ8C80usSZIkqSkamiydmb3PaauVUYwkSVIz7XOOEHB/RLwHGBcRC4APAz8utyxJkqTyNdIj9CHgOKAP+AawFfhImUVJkiQ1QyOrxp4C/mfxkiRJOmjsMwhFxL8A+dz2zHxNKRVJkiQ1SSNzhP77kO0JwNuAgXLKkSRJap5Ghsbufk7T7RHxbyXVI0mS1DSNDI09b8huB3ACMKW0iiRJkpqkkaGxu6nPEQrqQ2KPAO8vsyhJkqRmaGRobF4zCpEkSWq2RobGzhru/cy8bvTKkSRJap5GhsbeD7wS+H6x/9vU7yy9hfqQmUFIkiSNSY0EoS7gRZm5ASAiZgBfzcylpVYmSZJUskYesTF7MAQVNgFzSqpHkiSpaRrpEbo1IpZRf84YwDuB7x3Ih0bEHwH/hfrQ2n3AUmAGcBVwBPWVar+bmf0R0Q1cSX3Z/q+Bd2bmquI6n6A+dLcL+HBmLivalwCXAuOAL2bmXxxIvZI0nJ07dvDQnbez5uc/Y9PKFQz099E5vpvp81/AnBe/lIUnLaZrwoRWlylpDyJzt6dn7H5QxO8Ary52f5iZ3xrxB0bMBG6jPty2PSKuBm4C3ghcl5lXRcTfAT/LzMsi4nxgUWb+14h4F/A7mfnOiHgR9XB2InA09XC2sPiYh4DXAeuAu4B3Z+YDw9XV09OTy5cvH+mvJamCBvr7+fE1X+Oem28gOoKdO3bsdkzXhAlkLTl+yZt45dnvpXP8+BZUKlVPRNydmT37Oq6RHiGK4DPi8LOXz50YETuBQ4ANwGuA9xTvXwF8CrgMOLPYBrgW+L8REUX7VZnZBzwSESuohyKAFZm5EiAiriqOHTYISdL++PW6tXzzsxeyfetWBvr79nrcYDi65+YbePDHP+Rtn/g0R8ya3awyJe1DI3OERlVmrgf+ClhDPQD1Uh8K+4/MHHyG2TpgZrE9E1hbnDtQHH/E0PbnnLO39t1ExHkRsTwilm/ZsuXAfzlJlfDrdWv5+ic/yhOPPjpsCBpqoL+PJ379KF//5Ef59bq1+z5BUlM0PQhFxFTqPTTzqA9pTQKWNLsOgMy8PDN7MrNn2rRprShB0hgz0N/PNz97If1Pbac+zXE/ZNL/1Hau++xFDPT3l1KfpP2z1yAUEbcWP/9ylD/ztcAjmbklM3dSvw/RYuDwiBgcqpsFrC+21wOzi1o6qT/n7NdD259zzt7aJemA/fiar7F961b2OwQ9LXlqay8/vuZro1mWpBEarkdoRkS8EnhLRBwfES8f+jqAz1wDnBwRhxRzfU6jPn/nX4C3F8ecC3yn2L6+2Kd4//tZn+F9PfCuiOiOiHnAAuDfqE+OXhAR8yJiPPCu4lhJOiA7d+zgnptvaHg4bG8G+vu4Z9kNe5xcLam5hpssfSHwSeo9Kpc8572kPrl5v2XmnRFxLfAT6g9xvQe4HLgRuCoi/rxo+1JxypeAfygmQz9GPdiQmfcXK84eKK7zwczcBRARfwgso758/suZef9IapWkoR6683aiI0blWhHBQ3feznGnnjYq15M0MvtcPh8Rn8zMi5tUT8u4fF7Svnz3by7hgR9+f98HNui4U09jyfl/NGrXk/SMUVs+n5kXR8RbeOY+Qv+amTccaIGSNNZsWrliVK+3cZSvJ2n/7XPVWER8FriA+hDUA8AFEfG/yi5MktrNgc4N2u16fc4RklqtkRsqngG8LDNrABFxBfU5PH9SZmGS1G46x3eP7vW6feyG1GqN3kfo8CHbU8ooRJLa3fT5LxjV6z1/lK8naf81EoQ+C9wTEV8teoPuBj5TblmS1H7mvPilo/bw1K4JE5h93KJRuZakkdtnEMrMbwAnU7/x4TeBV2TmP5VdmCS1m4UnLSZrI72R4rNlJgtPWjwq15I0cg0NjWXmhsy8vnhtLLsoSWpHXRMmcPySNx3wXKHO8d0c//o3jVrvkqSRa/qzxiRpLHvl2e9l4uTJwAhvrBjBIZOn8Mp3vG9U65I0MgYhSdoPnePH87ZPfJrxh0xkv8NQBOMnTuSsT/wZnV1dpdQnaf8MG4QiYlxEPNisYiRpLDhi1mzec/HnOOzIIxseJusc383kI6bxnos/xxGzZu/7BElNMWwQKp7d9cuImNOkeiRpTDhi1mz+81//fX3OUHf3Xuf7dE2YQGd3N8cveRNL//rvDEFSm2nkhopTgfsj4t+AbYONmfmW0qqSpDGgc/x4Xv3epbzibe/moTtvZ83997Jp5QoG+nbQ2T2B6fNfwJzjFrHwpMVOjJbaVCNB6JOlVyFJY1jXhAkcd+ppPkleGoMaeejqDyLiGGBBZn4vIg4BxpVfmiRJUrkaeejqB4Brgb8vmmYC3y6zKEmSpGZoZPn8B4HFwFaAzHwYOKrMoiRJkpqhkSDUl5n9gzsR0QmMzj3mJUmSWqiRIPSDiPgTYGJEvA64Bvh/5ZYlSZJUvkaC0MeBLcB9wO8DNwF/WmZRkiRJzdDIqrFaRFwB3El9SOyXmenQmCRJGvP2GYQi4gzg74B/p/5gnXkR8fuZ+d2yi5MkSSpTIzdU/Bzw25m5AiAifgO4ETAISZKkMa2ROUJPDIagwkrgiZLqkSRJapq99ghFxFnF5vKIuAm4mvocobOBu5pQmyRJUqmGGxp785DtTcCpxfYWYGJpFUmSJDXJXoNQZi5tZiGSJEnN1siqsXnAh4C5Q4/PzLeUV5YkSVL5Glk19m3gS9TvJl0rtxxJkqTmaSQI7cjML5ReiSRJUpM1EoQujYiLgH8G+gYbM/MnpVUlSZLUBI0EoZcAvwu8hmeGxrLYlyRJGrMaCUJnA/Mzs7/sYiRJkpqpkTtL/xw4vOxCJEmSmq2RHqHDgQcj4i6ePUfI5fOSJGlMayQIXVR6FZIkSS2wzyCUmT9oRiGSJEnN1sidpZ+gvkoMYDzQBWzLzMllFiZJklS2RnqEDhvcjogAzgROLrMoSZKkZmhk1djTsu7bwOtLqkeSJKlpGhkaO2vIbgfQA+worSJJkqQmaWTV2JuHbA8Aq6gPj0mSJI1pjcwRWtqMQiRJkpptr0EoIi4c5rzMzItH8oERcSzwT0Oa5gMXAlcW7XOp9zq9IzMfLyZoXwq8EXgK+L3BB75GxLnAnxbX+fPMvKJoPwH4KjARuAm4IDMHV75JkiQBw0+W3raHF8D7gY+N9AMz85eZ+bLMfBlwAvVw8y3g48CtmbkAuLXYB3gDsKB4nQdcBhARz6N+s8eTgBOBiyJianHOZcAHhpy3ZKT1SpKkg9deg1Bmfm7wBVxOvXdlKXAV9V6c0XAa8O+ZuZr6vKMrivYrgLcW22cCVxYr1u4ADo+IGdRXrt2SmY9l5uPALcCS4r3JmXlH0Qt05ZBrSZIkPW3Y5fMR8byI+HPgXurDaC/PzI9l5uZR+vx3Ad8otqdn5oZieyMwvdieCawdcs66om249nV7aN9NRJwXEcsjYvmWLVsO5PeQJElj0F6DUET8b+Au4AngJZn5qaLnZVRExHjgLcA1z32v6MkpfU5PZl6emT2Z2TNt2rSyP06SJLWZ4XqEPgocTX0y8q8iYmvxeiIito7CZ78B+Elmbir2NxXDWhQ/B3ud1gOzh5w3q2gbrn3WHtolSZKeZbg5Qh2ZOTEzD8vMyUNeh43Sc8bezTPDYgDXA+cW2+cC3xnSfk7UnQz0FkNoy4DTI2JqMUn6dGBZ8d7WiDi5WHF2zpBrSZIkPa2RGyqOuoiYBLwO+P0hzX8BXB0R7wdWA+8o2m+ivnR+BfUVZksBMvOxiLiY+vAdwKcz87Fi+3yeWT7/3eIlSZL0LOHtdep6enpy+fLlrS5DkiSNgoi4OzN79nXcfj10VZIk6WBiEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIk7VHuSmrbB8hd2epSpNJ0troASVL7qG0fYNvdm3jyR+vYtbUfOgJqybjJ4zn0VbOYdMJ0Oib6V4cOHv7XLEkiM9n6vTU88YN1REDurNXfKHqDdvX2s3XZKnpvXsVhp85i8mvnEBEtrFgaHQYhSaq4zOTxqx9i+88fhYEaexsIGwxHT/5wHbse38HUsxcahjTmOUdIkipu6/fWsP3njz7TC7QPubPG9vseZev31pRcmVQ+g5AkVVht+wBP/GBdwyFoUO6s8cQP1lHbMVBSZVJzGIQkqcK23b2JkY5uRcC25ZtGtyCpyQxCklRhT/5o/3uDBuXOGk/+aN0oVyQ1l0FIkioqd9XqS+QPwK6t/d5nSGOaQUiSKir7a/X7BB2IjiD7d41OQVILGIQkqaJifAfUDrA3p5bE+HGjU5DUAgYhSaqoGNfBuMnjD+ga4yaPJ8Z5LyGNXQYhSaqwQ181i+ga2V8F0dXBoa+aNcoVSc1lEJKkCpt0wnRyhKNjmTCpZ/roFiQ1mUFIkiqsY2Inh526/71C0dXBYafOomOCT2rS2GYQkqSKm/zaOUx8yZENh6Ho6mDiS45k8mvnlFyZVD6jvCRVXEQw9eyFjJs6Yfenzw89rquDTDj01T59XgcPg5AkiYhgyuuO4bBTZrLt7k08+aN19ZstdgTUknGTx3Poq2YxqWe6w2E6qPhfsyTpaR0TOznslJkcdspMcleS/buI8eNcIq+DlkFIkrRHMS6Iif41oYObk6UlSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJltSQIRcThEXFtRDwYEb+IiFdExPMi4paIeLj4ObU4NiLiCxGxIiLujYiXD7nOucXxD0fEuUPaT4iI+4pzvhA+GVCSJO1Bq3qELgVuzswXAi8FfgF8HLg1MxcAtxb7AG8AFhSv84DLACLiecBFwEnAicBFg+GpOOYDQ85b0oTfSZLUhnp7e3nwwQf52c9+xoMPPkhvb2+rS1IbafpDZCJiCvBq4PcAMrMf6I+IM4H/VBx2BfCvwMeAM4ErMzOBO4repBnFsbdk5mPFdW8BlkTEvwKTM/OOov1K4K3Ad5vw60mS2kBm8sgjj3DbbbexevVqOjs7yUwigoGBAY455hhOOeUU5s+f3+pS1WKteJrePGAL8JWIeClwN3ABMD0zNxTHbASmF9szgbVDzl9XtA3Xvm4P7buJiPOo9zIxZ86ckf9GkqS2UavVuPHGG7n33nvZuXMnALt27XrWMStXrmTt2rUsWrSIM844g44Op8xWVSu++U7g5cBlmXk8sI1nhsEAKHp/suxCMvPyzOzJzJ5p06aV/XGSpJJl5m4haG927tzJvffey4033tik6tSOWhGE1gHrMvPOYv9a6sFoUzHkRfFzc/H+emD2kPNnFW3Dtc/aQ7sk6SD3yCOPNBSCBg2GoZUrV5ZcmdpV04NQZm4E1kbEsUXTacADwPXA4Mqvc4HvFNvXA+cUq8dOBnqLIbRlwOkRMbWYJH06sKx4b2tEnFysFjtnyLUkSQex2267reEQNGjnzp3cfvvtJVWkdteKOUIAHwK+FhHjgZXAUuqh7OqIeD+wGnhHcexNwBuBFcBTxbFk5mMRcTFwV3HcpwcnTgPnA18FJlKfJO1EaUk6yPX29rJ69eoRnbtq1Sp6e3uZMmXKKFeldteSIJSZPwV69vDWaXs4NoEP7uU6Xwa+vIf25cCLD7BMSdIYsmHDBjo7O3ebGN2Izs5ONmzYYBCqIKfJS5IOCn19fdT/7bz/MpO+vr5RrkhjgUFIknRQ6O7uZqQPEogIuru7R7kijQUGIUnSQWHGjBkMDAyM6NyBgQFmzJgxyhVpLDAISZIOClOmTOGYY44Z0blz5851flBFGYQkSQeNU045ha6urv06p6uri8WLF5dUkdqdQUiSdNCYN28eixYtajgMdXV1sWjRIp85VmEGIUnSQSMiOOOMMxoKQ4Mh6IwzzmhSdWpHrbqhoiRJpejo6ODNb34zxx13HLfffjurVq3a7enzc+fOZfHixfYEySAkSTo4zZ8/n/nz59Pb28uGDRvo6+uju7ubGTNmODFaTzMISZIOalOmTDH4aK+cIyRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkirLICRJkiqrJUEoIlZFxH0R8dOIWF60PS8ibomIh4ufU4v2iIgvRMSKiLg3Il4+5DrnFsc/HBHnDmk/obj+iuLcaP5vKUmS2l0re4R+OzNflpk9xf7HgVszcwFwa7EP8AZgQfE6D7gM6sEJuAg4CTgRuGgwPBXHfGDIeUvK/3UkSdJY005DY2cCVxTbVwBvHdJ+ZdbdARweETOA1wO3ZOZjmfk4cAuwpHhvcmbekZkJXDnkWpIkSU9rVRBK4J8j4u6IOK9om56ZG4rtjcD0YnsmsHbIueuKtuHa1+2hXZIk6Vk6W/S5p2Tm+og4CrglIh4c+mZmZkRk2UUUIew8gDlz5pT9cZIkqc20pEcoM9cXPzcD36I+x2dTMaxF8XNzcfh6YPaQ02cVbcO1z9pD+57quDwzezKzZ9q0aQf6a0mSpDGm6UEoIiZFxGGD28DpwM+B64HBlV/nAt8ptq8HzilWj50M9BZDaMuA0yNiajFJ+nRgWfHe1og4uVgtds6Qa0mSJD2tFUNj04FvFSvaO4GvZ+bNEXEXcHVEvB9YDbyjOP4m4I3ACuApYClAZj4WERcDdxXHfTozHyu2zwe+CkwEvlu8JEmSniXqC6vU09OTy5cvb3UZkiRpFETE3UNu0bNX7bR8XpIkqalatWpMkiTthye3PczaNV9hy5Zl7KptZ1zHRKZNez2z5yzl0EkLWl3emGUQkiSpjWXWeOihT/OrDddQq+0EdgFQq/Xxqw3XsnHTdzh6xtksXHghEQ707C+DkCRJbawegq6lVtuxh3d3Uavt4lcbrgXg2GM/1dTaDgYGIUmS2tST2x4ueoL2FIKeUatt51cbrmHWrPcxadILmlTdyGzs28kV6x/luk2Ps21XjUnjOjhr+lTOnXkkz+/uano99qFJktSm1q75SjEctm+12k7WrP1KyRWNXGZy6eqNnHTHA/zt2s2s3tHPozsHWL2jn79du5mT7niAS1dvpNmr2Q1CkiS1qS1bljE4J2jfdrFl881llnNAvrBmE59ftZm+WtJXe3bYGWz7/KrNfGHNpqbWZRCSJKlN7aptL/X4ZtnYt5NLVm1ie6027HHbazUuWbWJjX2N9YKNBoOQJEltalzHxFKPb5avrn90v46/cj+PPxAGIUmS2tS0aa8HxjV4dCfTjlpSZjkj9q1Nj+82HLY3fbXkus2Pl1zRMwxCkiS1qdlzltLR0dhKqo6OTubMXlpyRSOzbdfwQ2LP9eTA/h1/IAxCkiS1qUMnLeDoGWfTsY8hr46OiRw94+y2XTo/adz+xY1DO5sXTwxCkiS1sYULL+ToGW+no2MCuw+TjaOjYwJHz3g7Cxde2IryGnLW9Kl0d0RDx3Z3BGcdNbXkip5hEJIkqY1FdHDssZ/it37r2xx99Nl0dR5OR0c3XZ2Hc/TRZ3Pib32HY4/9VFs/XuPcmUc2fnDCOftz/AHyztKSJI0Bh05awG++8DP85gs/0+pS9tvzu7v4b3On8/lVm4ddQj+xo4OPzD2qqXeYNghJkqTSfXjOdAAuWVW/YeLQVWTdHQEJH5l71NPHNYtBSJIklS4iuOCY5/PO5x/Blesf5brNj/PkQI1DOzs466ipnNOiZ40ZhCRJUtM8v7uLP54/gz+eP6PVpQBOlpYkSRVmEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZVlEJIkSZUVmdnqGtpCRGwBVpd0+SOBR0u6tkbG76Q9+b20H7+T9uT3sm/HZOa0fR1kEGqCiFiemT2trkPP8DtpT34v7cfvpD35vYweh8YkSVJlGYQkSVJlGYSa4/JWF6Dd+J20J7+X9uN30p78XkaJc4QkSVJl2SMkSZIqyyAkSZIqyyBUoohYEhG/jIgVEfHxVtcjiIjZEfEvEfFARNwfERe0uibVRcS4iLgnIm5odS2qi4jDI+LaiHgwIn4REa9odU1VFxF/VPzZ9fOI+EZETGh1TWOdQagkETEO+BvgDcCLgHdHxItaW5WAAeCjmfki4GTgg34vbeMC4BetLkLPcilwc2a+EHgpfj8tFREzgQ8DPZn5YmAc8K7WVjX2GYTKcyKwIjNXZmY/cBVwZotrqrzM3JCZPym2n6D+B/vM1laliJgFnAF8sdW1qC4ipgCvBr4EkJn9mfkfra1KQCcwMSI6gUOAX7W4njHPIFSemcDaIfvr8C/cthIRc4HjgTtbW4mAzwN/DNRaXYieNg/YAnylGLL8YkRManVRVZaZ64G/AtYAG4DezPzn1lY19hmEVEkRcSjwTeAjmbm11fVUWUS8CdicmXe3uhY9SyfwcuCyzDwe2AY417GFImIq9ZGFecDRwKSIeF9rqxr7DELlWQ/MHrI/q2hTi0VEF/UQ9LXMvK7V9YjFwFsiYhX1IeTXRMQ/trYkUe/FXpeZgz2m11IPRmqd1wKPZOaWzNwJXAe8ssU1jXkGofLcBSyIiHkRMZ76hLbrW1xT5UVEUJ/z8IvMvKTV9Qgy8xOZOSsz51L//+T7mem/clssMzcCayPi2KLpNOCBFpak+pDYyRFxSPFn2Wk4gf2Adba6gINVZg5ExB8Cy6jP7P9yZt7f4rJU7334XeC+iPhp0fYnmXlTC2uS2tWHgK8V/5hbCSxtcT2Vlpl3RsS1wE+or4C9Bx+1ccB8xIYkSaosh8YkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJlGYQkSVJl/X9mwyMsqFYkCQAAAABJRU5ErkJggg==\n",
92 | "text/plain": [
93 | ""
94 | ]
95 | },
96 | "metadata": {},
97 | "output_type": "display_data"
98 | }
99 | ],
100 | "source": [
101 | "tagCount = collections.Counter(list(df_tags['Tag'])).most_common(10)\n",
102 | "print(tagCount)\n",
103 | "plot_tags(tagCount)"
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": 5,
109 | "metadata": {},
110 | "outputs": [],
111 | "source": [
112 | "top10=['javascript','java','c#','php','android','jquery','python','html','c++','ios']"
113 | ]
114 | },
115 | {
116 | "cell_type": "code",
117 | "execution_count": 6,
118 | "metadata": {},
119 | "outputs": [
120 | {
121 | "name": "stdout",
122 | "output_type": "stream",
123 | "text": [
124 | "(826739, 2)\n"
125 | ]
126 | },
127 | {
128 | "data": {
129 | "text/html": [
130 | "\n",
131 | "\n",
144 | "
\n",
145 | " \n",
146 | " \n",
147 | " \n",
148 | " Id \n",
149 | " Tag \n",
150 | " \n",
151 | " \n",
152 | " \n",
153 | " \n",
154 | " 14 \n",
155 | " 260 \n",
156 | " c# \n",
157 | " \n",
158 | " \n",
159 | " 18 \n",
160 | " 330 \n",
161 | " c++ \n",
162 | " \n",
163 | " \n",
164 | " 28 \n",
165 | " 650 \n",
166 | " c# \n",
167 | " \n",
168 | " \n",
169 | " 35 \n",
170 | " 930 \n",
171 | " c# \n",
172 | " \n",
173 | " \n",
174 | " 39 \n",
175 | " 1010 \n",
176 | " c# \n",
177 | " \n",
178 | " \n",
179 | "
\n",
180 | "
"
181 | ],
182 | "text/plain": [
183 | " Id Tag\n",
184 | "14 260 c#\n",
185 | "18 330 c++\n",
186 | "28 650 c#\n",
187 | "35 930 c#\n",
188 | "39 1010 c#"
189 | ]
190 | },
191 | "execution_count": 6,
192 | "metadata": {},
193 | "output_type": "execute_result"
194 | }
195 | ],
196 | "source": [
197 | "tag_top10= df_tags[df_tags.Tag.isin(top10)]\n",
198 | "print (tag_top10.shape)\n",
199 | "tag_top10.head()"
200 | ]
201 | },
202 | {
203 | "cell_type": "code",
204 | "execution_count": 7,
205 | "metadata": {},
206 | "outputs": [
207 | {
208 | "data": {
209 | "text/plain": [
210 | "30798790 5\n",
211 | "31085960 5\n",
212 | "11648170 5\n",
213 | "35318730 5\n",
214 | "4009250 5\n",
215 | "30289880 5\n",
216 | "23267320 5\n",
217 | "35283570 5\n",
218 | "30991580 5\n",
219 | "23484760 5\n",
220 | "Name: Id, dtype: int64"
221 | ]
222 | },
223 | "execution_count": 7,
224 | "metadata": {},
225 | "output_type": "execute_result"
226 | }
227 | ],
228 | "source": [
229 | "tag_top10['Id'].value_counts().head(10)"
230 | ]
231 | },
232 | {
233 | "cell_type": "code",
234 | "execution_count": 8,
235 | "metadata": {},
236 | "outputs": [
237 | {
238 | "data": {
239 | "text/html": [
240 | "\n",
241 | "\n",
254 | "
\n",
255 | " \n",
256 | " \n",
257 | " \n",
258 | " Id \n",
259 | " Tag \n",
260 | " \n",
261 | " \n",
262 | " \n",
263 | " \n",
264 | " 14 \n",
265 | " 260 \n",
266 | " c# \n",
267 | " \n",
268 | " \n",
269 | " 18 \n",
270 | " 330 \n",
271 | " c++ \n",
272 | " \n",
273 | " \n",
274 | " 28 \n",
275 | " 650 \n",
276 | " c# \n",
277 | " \n",
278 | " \n",
279 | " 35 \n",
280 | " 930 \n",
281 | " c# \n",
282 | " \n",
283 | " \n",
284 | " 39 \n",
285 | " 1010 \n",
286 | " c# \n",
287 | " \n",
288 | " \n",
289 | "
\n",
290 | "
"
291 | ],
292 | "text/plain": [
293 | " Id Tag\n",
294 | "14 260 c#\n",
295 | "18 330 c++\n",
296 | "28 650 c#\n",
297 | "35 930 c#\n",
298 | "39 1010 c#"
299 | ]
300 | },
301 | "execution_count": 8,
302 | "metadata": {},
303 | "output_type": "execute_result"
304 | }
305 | ],
306 | "source": [
307 | "tag_top10.head()"
308 | ]
309 | },
310 | {
311 | "cell_type": "code",
312 | "execution_count": 9,
313 | "metadata": {},
314 | "outputs": [],
315 | "source": [
316 | "def add_tags(question_id):\n",
317 | " return tag_top10[tag_top10['Id'] == question_id['Id']].Tag.values\n",
318 | "\n",
319 | "top10 = tag_top10.apply(add_tags, axis=1)"
320 | ]
321 | },
322 | {
323 | "cell_type": "code",
324 | "execution_count": 10,
325 | "metadata": {},
326 | "outputs": [
327 | {
328 | "data": {
329 | "text/plain": [
330 | "(826739, (826739, 2))"
331 | ]
332 | },
333 | "execution_count": 10,
334 | "metadata": {},
335 | "output_type": "execute_result"
336 | }
337 | ],
338 | "source": [
339 | "len(top10),tag_top10.shape"
340 | ]
341 | },
342 | {
343 | "cell_type": "code",
344 | "execution_count": 11,
345 | "metadata": {},
346 | "outputs": [
347 | {
348 | "data": {
349 | "text/html": [
350 | "\n",
351 | "\n",
364 | "
\n",
365 | " \n",
366 | " \n",
367 | " \n",
368 | " Id \n",
369 | " Tag \n",
370 | " Tags \n",
371 | " \n",
372 | " \n",
373 | " \n",
374 | " \n",
375 | " 14 \n",
376 | " 260 \n",
377 | " c# \n",
378 | " [c#] \n",
379 | " \n",
380 | " \n",
381 | " 18 \n",
382 | " 330 \n",
383 | " c++ \n",
384 | " [c++] \n",
385 | " \n",
386 | " \n",
387 | " 28 \n",
388 | " 650 \n",
389 | " c# \n",
390 | " [c#] \n",
391 | " \n",
392 | " \n",
393 | " 35 \n",
394 | " 930 \n",
395 | " c# \n",
396 | " [c#] \n",
397 | " \n",
398 | " \n",
399 | " 39 \n",
400 | " 1010 \n",
401 | " c# \n",
402 | " [c#] \n",
403 | " \n",
404 | " \n",
405 | "
\n",
406 | "
"
407 | ],
408 | "text/plain": [
409 | " Id Tag Tags\n",
410 | "14 260 c# [c#]\n",
411 | "18 330 c++ [c++]\n",
412 | "28 650 c# [c#]\n",
413 | "35 930 c# [c#]\n",
414 | "39 1010 c# [c#]"
415 | ]
416 | },
417 | "execution_count": 11,
418 | "metadata": {},
419 | "output_type": "execute_result"
420 | }
421 | ],
422 | "source": [
423 | "tag_top10=pd.concat([tag_top10, top10.rename('Tags')], axis=1)\n",
424 | "tag_top10.head()"
425 | ]
426 | },
427 | {
428 | "cell_type": "code",
429 | "execution_count": 12,
430 | "metadata": {},
431 | "outputs": [
432 | {
433 | "data": {
434 | "text/plain": [
435 | "(826739, 2)"
436 | ]
437 | },
438 | "execution_count": 12,
439 | "metadata": {},
440 | "output_type": "execute_result"
441 | }
442 | ],
443 | "source": [
444 | "tag_top10.drop([\"Tag\"], axis=1, inplace=True)\n",
445 | "tag_top10.shape"
446 | ]
447 | },
448 | {
449 | "cell_type": "code",
450 | "execution_count": 13,
451 | "metadata": {},
452 | "outputs": [],
453 | "source": [
454 | "top10_tags=tag_top10.loc[tag_top10.astype(str).drop_duplicates().index]"
455 | ]
456 | },
457 | {
458 | "cell_type": "code",
459 | "execution_count": 14,
460 | "metadata": {},
461 | "outputs": [
462 | {
463 | "data": {
464 | "text/html": [
465 | "\n",
466 | "\n",
479 | "
\n",
480 | " \n",
481 | " \n",
482 | " \n",
483 | " Id \n",
484 | " Title \n",
485 | " Body \n",
486 | " \n",
487 | " \n",
488 | " \n",
489 | " \n",
490 | " 0 \n",
491 | " 80 \n",
492 | " SQLStatement.execute() - multiple queries in o... \n",
493 | " I've written a database generation script in S... \n",
494 | " \n",
495 | " \n",
496 | " 1 \n",
497 | " 90 \n",
498 | " Good branching and merging tutorials for Torto... \n",
499 | " Are there any really good tutorials explaining... \n",
500 | " \n",
501 | " \n",
502 | " 2 \n",
503 | " 120 \n",
504 | " ASP.NET Site Maps \n",
505 | " Has anyone got experience creating SQL-based A... \n",
506 | " \n",
507 | " \n",
508 | " 3 \n",
509 | " 180 \n",
510 | " Function for creating color wheels \n",
511 | " This is something I've pseudo-solved many time... \n",
512 | " \n",
513 | " \n",
514 | " 4 \n",
515 | " 260 \n",
516 | " Adding scripting functionality to .NET applica... \n",
517 | " I have a little game written in C#. It uses a ... \n",
518 | " \n",
519 | " \n",
520 | "
\n",
521 | "
"
522 | ],
523 | "text/plain": [
524 | " Id ... Body\n",
525 | "0 80 ... I've written a database generation script in S...\n",
526 | "1 90 ... Are there any really good tutorials explaining...\n",
527 | "2 120 ... Has anyone got experience creating SQL-based A...\n",
528 | "3 180 ... This is something I've pseudo-solved many time...\n",
529 | "4 260 ... I have a little game written in C#. It uses a ...\n",
530 | "\n",
531 | "[5 rows x 3 columns]"
532 | ]
533 | },
534 | "execution_count": 14,
535 | "metadata": {},
536 | "output_type": "execute_result"
537 | }
538 | ],
539 | "source": [
540 | "ques = pd.read_csv('../input/stackoverflow-clean-questions-file-v2/question_clean.csv', encoding='iso-8859-1')\n",
541 | "ques.head()"
542 | ]
543 | },
544 | {
545 | "cell_type": "code",
546 | "execution_count": 15,
547 | "metadata": {},
548 | "outputs": [
549 | {
550 | "name": "stdout",
551 | "output_type": "stream",
552 | "text": [
553 | "(706336, 4)\n"
554 | ]
555 | },
556 | {
557 | "data": {
558 | "text/html": [
559 | "\n",
560 | "\n",
573 | "
\n",
574 | " \n",
575 | " \n",
576 | " \n",
577 | " Id \n",
578 | " Title \n",
579 | " Body \n",
580 | " Tags \n",
581 | " \n",
582 | " \n",
583 | " \n",
584 | " \n",
585 | " 0 \n",
586 | " 260 \n",
587 | " Adding scripting functionality to .NET applica... \n",
588 | " I have a little game written in C#. It uses a ... \n",
589 | " [c#] \n",
590 | " \n",
591 | " \n",
592 | " 1 \n",
593 | " 330 \n",
594 | " Should I use nested classes in this case? \n",
595 | " I am working on a collection of classes used f... \n",
596 | " [c++] \n",
597 | " \n",
598 | " \n",
599 | " 2 \n",
600 | " 650 \n",
601 | " Automatically update version number \n",
602 | " I would like the version property of my applic... \n",
603 | " [c#] \n",
604 | " \n",
605 | " \n",
606 | " 3 \n",
607 | " 930 \n",
608 | " How do I connect to a database and loop over a... \n",
609 | " What's the simplest way to connect and query a... \n",
610 | " [c#] \n",
611 | " \n",
612 | " \n",
613 | " 4 \n",
614 | " 1010 \n",
615 | " How to get the value of built, encoded ViewState? \n",
616 | " I need to grab the base64-encoded representati... \n",
617 | " [c#] \n",
618 | " \n",
619 | " \n",
620 | "
\n",
621 | "
"
622 | ],
623 | "text/plain": [
624 | " Id ... Tags\n",
625 | "0 260 ... [c#]\n",
626 | "1 330 ... [c++]\n",
627 | "2 650 ... [c#]\n",
628 | "3 930 ... [c#]\n",
629 | "4 1010 ... [c#]\n",
630 | "\n",
631 | "[5 rows x 4 columns]"
632 | ]
633 | },
634 | "execution_count": 15,
635 | "metadata": {},
636 | "output_type": "execute_result"
637 | }
638 | ],
639 | "source": [
640 | "total=pd.merge(ques, top10_tags, on='Id')\n",
641 | "print(total.shape)\n",
642 | "total.head()"
643 | ]
644 | },
645 | {
646 | "cell_type": "code",
647 | "execution_count": 16,
648 | "metadata": {},
649 | "outputs": [
650 | {
651 | "name": "stderr",
652 | "output_type": "stream",
653 | "text": [
654 | "Using TensorFlow backend.\n"
655 | ]
656 | }
657 | ],
658 | "source": [
659 | "from sklearn.model_selection import train_test_split\n",
660 | "from sklearn.preprocessing import MultiLabelBinarizer\n",
661 | "from nltk import word_tokenize\n",
662 | "from keras.preprocessing.text import Tokenizer\n",
663 | "from keras.preprocessing import sequence\n",
664 | "from keras.layers import LSTM, Activation, Dense, Dropout, Input, Embedding, BatchNormalization, GRU ,concatenate\n",
665 | "from keras.models import Model"
666 | ]
667 | },
668 | {
669 | "cell_type": "code",
670 | "execution_count": 17,
671 | "metadata": {},
672 | "outputs": [
673 | {
674 | "data": {
675 | "text/plain": [
676 | "array(['android', 'c#', 'c++', 'html', 'ios', 'java', 'javascript',\n",
677 | " 'jquery', 'php', 'python'], dtype=object)"
678 | ]
679 | },
680 | "execution_count": 17,
681 | "metadata": {},
682 | "output_type": "execute_result"
683 | }
684 | ],
685 | "source": [
686 | "multilabel_binarizer = MultiLabelBinarizer()\n",
687 | "multilabel_binarizer.fit(total.Tags)\n",
688 | "labels = multilabel_binarizer.classes_\n",
689 | "labels"
690 | ]
691 | },
692 | {
693 | "cell_type": "code",
694 | "execution_count": 18,
695 | "metadata": {},
696 | "outputs": [],
697 | "source": [
698 | "train,test=train_test_split(total[:550000],test_size=0.25,random_state=24)"
699 | ]
700 | },
701 | {
702 | "cell_type": "code",
703 | "execution_count": 19,
704 | "metadata": {},
705 | "outputs": [
706 | {
707 | "data": {
708 | "text/plain": [
709 | "((412500, 4), (137500, 4))"
710 | ]
711 | },
712 | "execution_count": 19,
713 | "metadata": {},
714 | "output_type": "execute_result"
715 | }
716 | ],
717 | "source": [
718 | "train.shape,test.shape"
719 | ]
720 | },
721 | {
722 | "cell_type": "code",
723 | "execution_count": 20,
724 | "metadata": {},
725 | "outputs": [],
726 | "source": [
727 | "X_train_t=train['Title']\n",
728 | "X_train_b=train['Body']\n",
729 | "y_train=multilabel_binarizer.transform(train['Tags'])\n",
730 | "X_test_t=test['Title']\n",
731 | "X_test_b=test['Body']\n",
732 | "y_test=multilabel_binarizer.transform(test['Tags'])"
733 | ]
734 | },
735 | {
736 | "cell_type": "code",
737 | "execution_count": 21,
738 | "metadata": {},
739 | "outputs": [
740 | {
741 | "data": {
742 | "text/plain": [
743 | "59"
744 | ]
745 | },
746 | "execution_count": 21,
747 | "metadata": {},
748 | "output_type": "execute_result"
749 | }
750 | ],
751 | "source": [
752 | "sent_lens_t=[]\n",
753 | "for sent in train['Title']:\n",
754 | " sent_lens_t.append(len(word_tokenize(sent)))\n",
755 | "max(sent_lens_t)"
756 | ]
757 | },
758 | {
759 | "cell_type": "code",
760 | "execution_count": 22,
761 | "metadata": {},
762 | "outputs": [
763 | {
764 | "data": {
765 | "text/plain": [
766 | "18.0"
767 | ]
768 | },
769 | "execution_count": 22,
770 | "metadata": {},
771 | "output_type": "execute_result"
772 | }
773 | ],
774 | "source": [
775 | "np.quantile(sent_lens_t,0.97)"
776 | ]
777 | },
778 | {
779 | "cell_type": "code",
780 | "execution_count": 23,
781 | "metadata": {},
782 | "outputs": [],
783 | "source": [
784 | "max_len_t = 18\n",
785 | "tok = Tokenizer(char_level=False,split=' ')\n",
786 | "tok.fit_on_texts(X_train_t)\n",
787 | "sequences_train_t = tok.texts_to_sequences(X_train_t)"
788 | ]
789 | },
790 | {
791 | "cell_type": "code",
792 | "execution_count": 24,
793 | "metadata": {},
794 | "outputs": [
795 | {
796 | "data": {
797 | "text/plain": [
798 | "68969"
799 | ]
800 | },
801 | "execution_count": 24,
802 | "metadata": {},
803 | "output_type": "execute_result"
804 | }
805 | ],
806 | "source": [
807 | "vocab_len_t=len(tok.index_word.keys())\n",
808 | "vocab_len_t"
809 | ]
810 | },
811 | {
812 | "cell_type": "code",
813 | "execution_count": 25,
814 | "metadata": {},
815 | "outputs": [
816 | {
817 | "data": {
818 | "text/plain": [
819 | "array([[ 0, 0, 0, ..., 1, 957, 197],\n",
820 | " [ 0, 0, 0, ..., 9081, 45, 533],\n",
821 | " [ 0, 0, 0, ..., 147, 8, 230],\n",
822 | " ...,\n",
823 | " [ 0, 0, 0, ..., 10, 71, 2985],\n",
824 | " [ 0, 0, 0, ..., 2, 18, 75],\n",
825 | " [ 0, 0, 0, ..., 11009, 809, 267]], dtype=int32)"
826 | ]
827 | },
828 | "execution_count": 25,
829 | "metadata": {},
830 | "output_type": "execute_result"
831 | }
832 | ],
833 | "source": [
834 | "sequences_matrix_train_t = sequence.pad_sequences(sequences_train_t,maxlen=max_len_t)\n",
835 | "sequences_matrix_train_t"
836 | ]
837 | },
838 | {
839 | "cell_type": "code",
840 | "execution_count": 26,
841 | "metadata": {},
842 | "outputs": [],
843 | "source": [
844 | "sequences_test_t = tok.texts_to_sequences(X_test_t)\n",
845 | "sequences_matrix_test_t = sequence.pad_sequences(sequences_test_t,maxlen=max_len_t)"
846 | ]
847 | },
848 | {
849 | "cell_type": "code",
850 | "execution_count": 27,
851 | "metadata": {},
852 | "outputs": [
853 | {
854 | "data": {
855 | "text/plain": [
856 | "((412500, 18), (137500, 18), (412500, 10), (137500, 10))"
857 | ]
858 | },
859 | "execution_count": 27,
860 | "metadata": {},
861 | "output_type": "execute_result"
862 | }
863 | ],
864 | "source": [
865 | "sequences_matrix_train_t.shape,sequences_matrix_test_t.shape,y_train.shape,y_test.shape"
866 | ]
867 | },
868 | {
869 | "cell_type": "code",
870 | "execution_count": 28,
871 | "metadata": {},
872 | "outputs": [
873 | {
874 | "data": {
875 | "text/plain": [
876 | "20853"
877 | ]
878 | },
879 | "execution_count": 28,
880 | "metadata": {},
881 | "output_type": "execute_result"
882 | }
883 | ],
884 | "source": [
885 | "sent_lens_b=[]\n",
886 | "for sent in train['Body']:\n",
887 | " sent_lens_b.append(len(word_tokenize(sent)))\n",
888 | "max(sent_lens_b)"
889 | ]
890 | },
891 | {
892 | "cell_type": "code",
893 | "execution_count": 29,
894 | "metadata": {},
895 | "outputs": [
896 | {
897 | "data": {
898 | "text/plain": [
899 | "575.0"
900 | ]
901 | },
902 | "execution_count": 29,
903 | "metadata": {},
904 | "output_type": "execute_result"
905 | }
906 | ],
907 | "source": [
908 | "np.quantile(sent_lens_b,0.90)"
909 | ]
910 | },
911 | {
912 | "cell_type": "code",
913 | "execution_count": 30,
914 | "metadata": {},
915 | "outputs": [],
916 | "source": [
917 | "max_len_b = 600\n",
918 | "tok = Tokenizer(char_level=False,split=' ')\n",
919 | "tok.fit_on_texts(X_train_b)\n",
920 | "sequences_train_b = tok.texts_to_sequences(X_train_b)"
921 | ]
922 | },
923 | {
924 | "cell_type": "code",
925 | "execution_count": 31,
926 | "metadata": {},
927 | "outputs": [
928 | {
929 | "data": {
930 | "text/plain": [
931 | "1292018"
932 | ]
933 | },
934 | "execution_count": 31,
935 | "metadata": {},
936 | "output_type": "execute_result"
937 | }
938 | ],
939 | "source": [
940 | "vocab_len_b =len(tok.index_word.keys())\n",
941 | "vocab_len_b "
942 | ]
943 | },
944 | {
945 | "cell_type": "code",
946 | "execution_count": 32,
947 | "metadata": {},
948 | "outputs": [
949 | {
950 | "data": {
951 | "text/plain": [
952 | "array([[ 0, 0, 0, ..., 51, 2082, 91],\n",
953 | " [ 0, 0, 0, ..., 1408, 203, 825],\n",
954 | " [ 0, 0, 0, ..., 34, 51, 83],\n",
955 | " ...,\n",
956 | " [ 0, 0, 0, ..., 20, 68, 687],\n",
957 | " [ 0, 0, 0, ..., 187, 58, 10],\n",
958 | " [ 0, 0, 0, ..., 194, 197, 10]], dtype=int32)"
959 | ]
960 | },
961 | "execution_count": 32,
962 | "metadata": {},
963 | "output_type": "execute_result"
964 | }
965 | ],
966 | "source": [
967 | "sequences_matrix_train_b = sequence.pad_sequences(sequences_train_b,maxlen=max_len_b)\n",
968 | "sequences_matrix_train_b"
969 | ]
970 | },
971 | {
972 | "cell_type": "code",
973 | "execution_count": 33,
974 | "metadata": {},
975 | "outputs": [],
976 | "source": [
977 | "sequences_test_b = tok.texts_to_sequences(X_test_b)\n",
978 | "sequences_matrix_test_b = sequence.pad_sequences(sequences_test_b,maxlen=max_len_b)"
979 | ]
980 | },
981 | {
982 | "cell_type": "code",
983 | "execution_count": 34,
984 | "metadata": {},
985 | "outputs": [
986 | {
987 | "data": {
988 | "text/plain": [
989 | "((412500, 18), (412500, 600), (412500, 10))"
990 | ]
991 | },
992 | "execution_count": 34,
993 | "metadata": {},
994 | "output_type": "execute_result"
995 | }
996 | ],
997 | "source": [
998 | "sequences_matrix_train_t.shape,sequences_matrix_train_b.shape,y_train.shape"
999 | ]
1000 | },
1001 | {
1002 | "cell_type": "code",
1003 | "execution_count": 35,
1004 | "metadata": {},
1005 | "outputs": [
1006 | {
1007 | "data": {
1008 | "text/plain": [
1009 | "((137500, 18), (137500, 600), (137500, 10))"
1010 | ]
1011 | },
1012 | "execution_count": 35,
1013 | "metadata": {},
1014 | "output_type": "execute_result"
1015 | }
1016 | ],
1017 | "source": [
1018 | "sequences_matrix_test_t.shape,sequences_matrix_test_b.shape,y_test.shape"
1019 | ]
1020 | },
1021 | {
1022 | "cell_type": "code",
1023 | "execution_count": 36,
1024 | "metadata": {},
1025 | "outputs": [],
1026 | "source": [
1027 | "def RNN():\n",
1028 | " # Title Only\n",
1029 | " title_input = Input(name='title_input',shape=[max_len_t])\n",
1030 | " title_Embed = Embedding(vocab_len_t+1,2000,input_length=max_len_t,mask_zero=True,name='title_Embed')(title_input)\n",
1031 | " gru_out_t = GRU(300)(title_Embed)\n",
1032 | " # auxiliary output to tune GRU weights smoothly \n",
1033 | " auxiliary_output = Dense(10, activation='sigmoid', name='aux_output')(gru_out_t) \n",
1034 | " \n",
1035 | " # Body Only\n",
1036 | " body_input = Input(name='body_input',shape=[max_len_b]) \n",
1037 | " body_Embed = Embedding(vocab_len_b+1,170,input_length=max_len_b,mask_zero=True,name='body_Embed')(body_input)\n",
1038 | " gru_out_b = GRU(200)(body_Embed)\n",
1039 | " \n",
1040 | " # combined with GRU output\n",
1041 | " com = concatenate([gru_out_t, gru_out_b])\n",
1042 | " \n",
1043 | " # now the combined data is being fed to dense layers\n",
1044 | " dense1 = Dense(400,activation='relu')(com)\n",
1045 | " dp1 = Dropout(0.5)(dense1)\n",
1046 | " bn = BatchNormalization()(dp1) \n",
1047 | " dense2 = Dense(150,activation='relu')(bn)\n",
1048 | " \n",
1049 | " main_output = Dense(10, activation='sigmoid', name='main_output')(dense2)\n",
1050 | " \n",
1051 | " model = Model(inputs=[title_input, body_input],outputs=[main_output, auxiliary_output])\n",
1052 | " return model"
1053 | ]
1054 | },
1055 | {
1056 | "cell_type": "code",
1057 | "execution_count": 37,
1058 | "metadata": {},
1059 | "outputs": [
1060 | {
1061 | "name": "stdout",
1062 | "output_type": "stream",
1063 | "text": [
1064 | "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
1065 | "Instructions for updating:\n",
1066 | "Colocations handled automatically by placer.\n",
1067 | "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
1068 | "Instructions for updating:\n",
1069 | "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
1070 | "__________________________________________________________________________________________________\n",
1071 | "Layer (type) Output Shape Param # Connected to \n",
1072 | "==================================================================================================\n",
1073 | "title_input (InputLayer) (None, 18) 0 \n",
1074 | "__________________________________________________________________________________________________\n",
1075 | "body_input (InputLayer) (None, 600) 0 \n",
1076 | "__________________________________________________________________________________________________\n",
1077 | "title_Embed (Embedding) (None, 18, 2000) 137940000 title_input[0][0] \n",
1078 | "__________________________________________________________________________________________________\n",
1079 | "body_Embed (Embedding) (None, 600, 170) 219643230 body_input[0][0] \n",
1080 | "__________________________________________________________________________________________________\n",
1081 | "gru_1 (GRU) (None, 300) 2070900 title_Embed[0][0] \n",
1082 | "__________________________________________________________________________________________________\n",
1083 | "gru_2 (GRU) (None, 200) 222600 body_Embed[0][0] \n",
1084 | "__________________________________________________________________________________________________\n",
1085 | "concatenate_1 (Concatenate) (None, 500) 0 gru_1[0][0] \n",
1086 | " gru_2[0][0] \n",
1087 | "__________________________________________________________________________________________________\n",
1088 | "dense_1 (Dense) (None, 400) 200400 concatenate_1[0][0] \n",
1089 | "__________________________________________________________________________________________________\n",
1090 | "dropout_1 (Dropout) (None, 400) 0 dense_1[0][0] \n",
1091 | "__________________________________________________________________________________________________\n",
1092 | "batch_normalization_1 (BatchNor (None, 400) 1600 dropout_1[0][0] \n",
1093 | "__________________________________________________________________________________________________\n",
1094 | "dense_2 (Dense) (None, 150) 60150 batch_normalization_1[0][0] \n",
1095 | "__________________________________________________________________________________________________\n",
1096 | "main_output (Dense) (None, 10) 1510 dense_2[0][0] \n",
1097 | "__________________________________________________________________________________________________\n",
1098 | "aux_output (Dense) (None, 10) 3010 gru_1[0][0] \n",
1099 | "==================================================================================================\n",
1100 | "Total params: 360,143,400\n",
1101 | "Trainable params: 360,142,600\n",
1102 | "Non-trainable params: 800\n",
1103 | "__________________________________________________________________________________________________\n"
1104 | ]
1105 | }
1106 | ],
1107 | "source": [
1108 | "model = RNN()\n",
1109 | "model.summary()"
1110 | ]
1111 | },
1112 | {
1113 | "cell_type": "code",
1114 | "execution_count": 38,
1115 | "metadata": {},
1116 | "outputs": [],
1117 | "source": [
1118 | "model.compile(optimizer='adam',loss={'main_output': 'categorical_crossentropy', 'aux_output': 'categorical_crossentropy'},\n",
1119 | " metrics=['accuracy'])"
1120 | ]
1121 | },
1122 | {
1123 | "cell_type": "code",
1124 | "execution_count": 39,
1125 | "metadata": {},
1126 | "outputs": [
1127 | {
1128 | "name": "stdout",
1129 | "output_type": "stream",
1130 | "text": [
1131 | "WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
1132 | "Instructions for updating:\n",
1133 | "Use tf.cast instead.\n"
1134 | ]
1135 | },
1136 | {
1137 | "name": "stderr",
1138 | "output_type": "stream",
1139 | "text": [
1140 | "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:107: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 137940000 elements. This may consume a large amount of memory.\n",
1141 | " num_elements)\n",
1142 | "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:107: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 219643230 elements. This may consume a large amount of memory.\n",
1143 | " num_elements)\n"
1144 | ]
1145 | },
1146 | {
1147 | "name": "stdout",
1148 | "output_type": "stream",
1149 | "text": [
1150 | "Train on 412500 samples, validate on 137500 samples\n",
1151 | "Epoch 1/5\n",
1152 | "412500/412500 [==============================] - 819s 2ms/step - loss: 2.3168 - main_output_loss: 1.0405 - aux_output_loss: 1.2763 - main_output_acc: 0.7285 - aux_output_acc: 0.6705 - val_loss: 1.7714 - val_main_output_loss: 0.7258 - val_aux_output_loss: 1.0456 - val_main_output_acc: 0.8254 - val_aux_output_acc: 0.7285\n",
1153 | "Epoch 2/5\n",
1154 | "412500/412500 [==============================] - 797s 2ms/step - loss: 1.5874 - main_output_loss: 0.6490 - aux_output_loss: 0.9384 - main_output_acc: 0.8443 - aux_output_acc: 0.7582 - val_loss: 1.7043 - val_main_output_loss: 0.6581 - val_aux_output_loss: 1.0462 - val_main_output_acc: 0.8379 - val_aux_output_acc: 0.7286\n",
1155 | "Epoch 3/5\n",
1156 | "412500/412500 [==============================] - 797s 2ms/step - loss: 1.3994 - main_output_loss: 0.5472 - aux_output_loss: 0.8522 - main_output_acc: 0.8650 - aux_output_acc: 0.7774 - val_loss: 1.7369 - val_main_output_loss: 0.6660 - val_aux_output_loss: 1.0708 - val_main_output_acc: 0.8364 - val_aux_output_acc: 0.7263\n",
1157 | "Epoch 4/5\n",
1158 | "412500/412500 [==============================] - 798s 2ms/step - loss: 1.2719 - main_output_loss: 0.4735 - aux_output_loss: 0.7984 - main_output_acc: 0.8774 - aux_output_acc: 0.7885 - val_loss: 1.7988 - val_main_output_loss: 0.6976 - val_aux_output_loss: 1.1012 - val_main_output_acc: 0.8369 - val_aux_output_acc: 0.7240\n",
1159 | "Epoch 5/5\n",
1160 | "412500/412500 [==============================] - 797s 2ms/step - loss: 1.1665 - main_output_loss: 0.4110 - aux_output_loss: 0.7555 - main_output_acc: 0.8868 - aux_output_acc: 0.7976 - val_loss: 1.9099 - val_main_output_loss: 0.7671 - val_aux_output_loss: 1.1428 - val_main_output_acc: 0.8307 - val_aux_output_acc: 0.7237\n"
1161 | ]
1162 | }
1163 | ],
1164 | "source": [
1165 | "results=model.fit({'title_input': sequences_matrix_train_t, 'body_input': sequences_matrix_train_b},\n",
1166 | " {'main_output': y_train, 'aux_output': y_train},\n",
1167 | " validation_data=[{'title_input': sequences_matrix_test_t, 'body_input': sequences_matrix_test_b},\n",
1168 | " {'main_output': y_test, 'aux_output': y_test}],\n",
1169 | " epochs=5, batch_size=800)"
1170 | ]
1171 | },
1172 | {
1173 | "cell_type": "code",
1174 | "execution_count": 68,
1175 | "metadata": {},
1176 | "outputs": [
1177 | {
1178 | "name": "stdout",
1179 | "output_type": "stream",
1180 | "text": [
1181 | "137500/137500 [==============================] - 1270s 9ms/step\n"
1182 | ]
1183 | }
1184 | ],
1185 | "source": [
1186 | "(predicted_main, predicted_aux)=model.predict({'title_input': sequences_matrix_test_t, 'body_input': sequences_matrix_test_b},verbose=1)"
1187 | ]
1188 | },
1189 | {
1190 | "cell_type": "code",
1191 | "execution_count": 70,
1192 | "metadata": {},
1193 | "outputs": [],
1194 | "source": [
1195 | "from sklearn.metrics import classification_report,f1_score"
1196 | ]
1197 | },
1198 | {
1199 | "cell_type": "code",
1200 | "execution_count": 138,
1201 | "metadata": {},
1202 | "outputs": [
1203 | {
1204 | "name": "stdout",
1205 | "output_type": "stream",
1206 | "text": [
1207 | "0.8424636536796537\n"
1208 | ]
1209 | },
1210 | {
1211 | "name": "stderr",
1212 | "output_type": "stream",
1213 | "text": [
1214 | "/opt/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in samples with no predicted labels.\n",
1215 | " 'precision', 'predicted', average, warn_for)\n"
1216 | ]
1217 | }
1218 | ],
1219 | "source": [
1220 | "print(f1_score(y_test,predicted_main>.55,average='samples'))"
1221 | ]
1222 | },
1223 | {
1224 | "cell_type": "code",
1225 | "execution_count": 137,
1226 | "metadata": {},
1227 | "outputs": [
1228 | {
1229 | "name": "stdout",
1230 | "output_type": "stream",
1231 | "text": [
1232 | " precision recall f1-score support\n",
1233 | "\n",
1234 | " 0 0.97 0.93 0.95 17054\n",
1235 | " 1 0.92 0.84 0.88 20681\n",
1236 | " 2 0.92 0.81 0.86 9700\n",
1237 | " 3 0.69 0.53 0.60 11304\n",
1238 | " 4 0.96 0.91 0.94 8897\n",
1239 | " 5 0.91 0.80 0.85 22472\n",
1240 | " 6 0.82 0.72 0.76 22938\n",
1241 | " 7 0.81 0.83 0.82 16150\n",
1242 | " 8 0.92 0.90 0.91 19659\n",
1243 | " 9 0.97 0.92 0.95 11576\n",
1244 | "\n",
1245 | " micro avg 0.89 0.82 0.85 160431\n",
1246 | " macro avg 0.89 0.82 0.85 160431\n",
1247 | "weighted avg 0.89 0.82 0.85 160431\n",
1248 | " samples avg 0.86 0.85 0.84 160431\n",
1249 | "\n"
1250 | ]
1251 | },
1252 | {
1253 | "name": "stderr",
1254 | "output_type": "stream",
1255 | "text": [
1256 | "/opt/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in samples with no predicted labels.\n",
1257 | " 'precision', 'predicted', average, warn_for)\n"
1258 | ]
1259 | }
1260 | ],
1261 | "source": [
1262 | "print(classification_report(y_test,predicted_main>.55))"
1263 | ]
1264 | },
1265 | {
1266 | "cell_type": "code",
1267 | "execution_count": 131,
1268 | "metadata": {},
1269 | "outputs": [
1270 | {
1271 | "data": {
1272 | "text/plain": [
1273 | "Id 16470700\n",
1274 | "Title NetworkOnMainThreadException- Have tried makin...\n",
1275 | "Body I've been trying to get this to work for a whi...\n",
1276 | "Tags [java, android]\n",
1277 | "Name: 250148, dtype: object"
1278 | ]
1279 | },
1280 | "execution_count": 131,
1281 | "metadata": {},
1282 | "output_type": "execute_result"
1283 | }
1284 | ],
1285 | "source": [
1286 | "test.iloc[24]"
1287 | ]
1288 | },
1289 | {
1290 | "cell_type": "code",
1291 | "execution_count": 134,
1292 | "metadata": {},
1293 | "outputs": [
1294 | {
1295 | "data": {
1296 | "text/plain": [
1297 | "array([1. , 0. , 0. , 0. , 0. , 0.84, 0. , 0. , 0. , 0. ],\n",
1298 | " dtype=float32)"
1299 | ]
1300 | },
1301 | "execution_count": 134,
1302 | "metadata": {},
1303 | "output_type": "execute_result"
1304 | }
1305 | ],
1306 | "source": [
1307 | "predicted_main[24].round(decimals = 2)"
1308 | ]
1309 | },
1310 | {
1311 | "cell_type": "code",
1312 | "execution_count": 92,
1313 | "metadata": {},
1314 | "outputs": [
1315 | {
1316 | "data": {
1317 | "text/plain": [
1318 | "array(['android', 'c#', 'c++', 'html', 'ios', 'java', 'javascript',\n",
1319 | " 'jquery', 'php', 'python'], dtype=object)"
1320 | ]
1321 | },
1322 | "execution_count": 92,
1323 | "metadata": {},
1324 | "output_type": "execute_result"
1325 | }
1326 | ],
1327 | "source": [
1328 | "labels"
1329 | ]
1330 | },
1331 | {
1332 | "cell_type": "code",
1333 | "execution_count": 79,
1334 | "metadata": {},
1335 | "outputs": [],
1336 | "source": [
1337 | "model.save('./stackoverflow_tags.h5')"
1338 | ]
1339 | }
1340 | ],
1341 | "metadata": {
1342 | "kernelspec": {
1343 | "display_name": "Python 3",
1344 | "language": "python",
1345 | "name": "python3"
1346 | },
1347 | "language_info": {
1348 | "codemirror_mode": {
1349 | "name": "ipython",
1350 | "version": 3
1351 | },
1352 | "file_extension": ".py",
1353 | "mimetype": "text/x-python",
1354 | "name": "python",
1355 | "nbconvert_exporter": "python",
1356 | "pygments_lexer": "ipython3",
1357 | "version": "3.6.7"
1358 | }
1359 | },
1360 | "nbformat": 4,
1361 | "nbformat_minor": 1
1362 | }
1363 |
--------------------------------------------------------------------------------