![](\"active_learning.png\")
"
22 | ]
23 | },
24 | {
25 | "cell_type": "markdown",
26 | "metadata": {},
27 | "source": [
28 | "## Active Learning Strategies"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {},
34 | "source": [
35 | "#### Pool-Based Sampling"
36 | ]
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {},
41 | "source": [
42 | "В этом сценарии экземпляры извлекаются из всего пула данных и им присваивается информативная оценка, которая показывает, насколько хорошо текущий алгоритм «понимает» данные. \n",
43 | "\n",
44 | "Затем система выбираются и размечаются наиболее информативные примеры."
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "metadata": {},
50 | "source": [
51 | "#### Uncertainty sampling"
52 | ]
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "metadata": {},
57 | "source": [
58 | "\n",
59 | "В рамках этого алгоритма размечаются те примеры, на которых текущая модель наименее уверена.\n",
60 | "\n",
61 | "В качестве функций \"уверенности\" можно использовать вероятности классов или расстояния до разделяющей гиперплоскости."
62 | ]
63 | },
64 | {
65 | "cell_type": "markdown",
66 | "metadata": {},
67 | "source": [
68 | "#### Membership Query Synthesis"
69 | ]
70 | },
71 | {
72 | "cell_type": "markdown",
73 | "metadata": {},
74 | "source": [
75 | "Здесь алгритм обучения модели генерирует свои собственные примеры из некоторого настраиваемого распределения. \n",
76 | "Эти сгенерированные примеры отправляются на разметку и модель дообучается с учетом разметки этих примеров."
77 | ]
78 | },
79 | {
80 | "cell_type": "markdown",
81 | "metadata": {},
82 | "source": [
83 | "#### Query by Committee"
84 | ]
85 | },
86 | {
87 | "cell_type": "markdown",
88 | "metadata": {},
89 | "source": [
90 | "Идея: построить ансамбль моделей $a_1,...,a_T$. \n",
91 | "\n",
92 | "Выбирать новые объекты $x_i$ с наибольшей несогласованностью решений ансамбля моделей.\n",
93 | "\n"
94 | ]
95 | },
96 | {
97 | "cell_type": "markdown",
98 | "metadata": {},
99 | "source": [
100 | "Принцип максимума энтропии: выбираем $x_i$, на котором $a_t(x_i)$ максимально различны."
101 | ]
102 | },
103 | {
104 | "cell_type": "markdown",
105 | "metadata": {},
106 | "source": [
107 | "Принцип максимума средней $KL$-дивергенции:выбираем $x_i$ , на котором $P_t(y|x_i)$ максимально различны:\n",
108 | "\n",
109 | "$С(y|u) = \\frac{1}{T}\\sum_{t=1}^T P_t(y|u)$ - консенсус комитета "
110 | ]
111 | },
112 | {
113 | "cell_type": "markdown",
114 | "metadata": {},
115 | "source": [
116 | "## SVM для Active Learning"
117 | ]
118 | },
119 | {
120 | "cell_type": "markdown",
121 | "metadata": {},
122 | "source": [
123 | "Некоторые активные алгоритмы обучения построены на алгоритме SVM и используют структуру SVM для определения того, какие точки данных нужно размечать. \n",
124 | "\n",
125 | "SVM используется для определения уверенности модели в предсказании на каждом из примеров выборки. \n",
126 | "В качестве меры уверенности служит расстояние от объекта до построенной не текущей итерации разделяющей гиперплоскости."
127 | ]
128 | },
129 | {
130 | "cell_type": "markdown",
131 | "metadata": {},
132 | "source": [
133 | "## Active Learning for texts classification"
134 | ]
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {},
139 | "source": [
140 | "Рассмотрим алгоритм pool-based active learning на примере задачи классификации твитов по тональности."
141 | ]
142 | },
143 | {
144 | "cell_type": "markdown",
145 | "metadata": {},
146 | "source": [
147 | "\n",
148 | "1. Разделить данные на X_pool (выборка, которую можно размечать) и X_test.\n",
149 | "2. Выбрать $k$ примеров из X_pool для начального X_train и разметить их. Остальные данные в X_pool $-$ валидационное множество. \n",
150 | "3. Обучить модель на X_train.\n",
151 | "5. Сделать predict обученной моделью на X_pool, вычислить вероятности для каждого $x_i$.\n",
152 | "6. Вычислить качество работы модели на X_test.\n",
153 | "7. Выбрать $k$ наиболее информативных объектов из X_pool, основываясь на уверенности модели в каждом из объектов (например, вероятности классов).\n",
154 | "8. Переменести эти $k$ выбранных объектов в X_train.\n",
155 | "9. Если качество работы модели на X_test достаточное, то останавливаемся, иначе возвращаемся к шагу 3."
156 | ]
157 | },
158 | {
159 | "cell_type": "code",
160 | "execution_count": 1,
161 | "metadata": {},
162 | "outputs": [],
163 | "source": [
164 | "from sklearn.metrics import f1_score"
165 | ]
166 | },
167 | {
168 | "cell_type": "code",
169 | "execution_count": 2,
170 | "metadata": {},
171 | "outputs": [],
172 | "source": [
173 | "import pandas as pd\n",
174 | "import numpy as np\n",
175 | "from sklearn.metrics import *\n",
176 | "from sklearn.model_selection import train_test_split\n",
177 | "from sklearn.pipeline import Pipeline\n",
178 | "from nltk import ngrams\n",
179 | "\n",
180 | "from sklearn.linear_model import LogisticRegression \n",
181 | "from sklearn.feature_extraction.text import CountVectorizer"
182 | ]
183 | },
184 | {
185 | "cell_type": "code",
186 | "execution_count": 4,
187 | "metadata": {},
188 | "outputs": [
189 | {
190 | "name": "stdout",
191 | "output_type": "stream",
192 | "text": [
193 | "(5572, 2)\n"
194 | ]
195 | },
196 | {
197 | "data": {
198 | "text/html": [
199 | "\n", 19 | "Как это выглядит (пример из [русского Universal Dependency трибанка](https://github.com/UniversalDependencies/UD_Russian-SynTagRus)):" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "my_example = \"\"\"\n", 29 | "# sent_id = 2003Armeniya.xml_138\n", 30 | "# text = Перспективы развития сферы высоких технологий.\n", 31 | "1\tПерспективы\tперспектива\tNOUN\t_\tAnimacy=Inan|Case=Nom|Gender=Fem|Number=Plur\t0\tROOT\t0:root\t_\n", 32 | "2\tразвития\tразвитие\tNOUN\t_\tAnimacy=Inan|Case=Gen|Gender=Neut|Number=Sing\t1\tnmod\t1:nmod\t_\n", 33 | "3\tсферы\tсфера\tNOUN\t_\tAnimacy=Inan|Case=Gen|Gender=Fem|Number=Sing\t2\tnmod\t2:nmod\t_\n", 34 | "4\tвысоких\tвысокий\tADJ\t_\tCase=Gen|Degree=Pos|Number=Plur\t5\tamod\t5:amod\t_\n", 35 | "5\tтехнологий\tтехнология\tNOUN\t_\tAnimacy=Inan|Case=Gen|Gender=Fem|Number=Plur\t3\tnmod\t3:nmod\tSpaceAfter=No\n", 36 | "6\t.\t.\tPUNCT\t_\t_\t1\tpunct\t1:punct\t_\n", 37 | "\"\"\"" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "Комментарии + таблица c 9 колонками (разделители табы):\n", 45 | "* ID\n", 46 | "* FORM: токен\n", 47 | "* LEMMA: начальная форма\n", 48 | "* UPOS: универсальная часть речи\n", 49 | "* XPOS: лингво-специфичная часть речи\n", 50 | "* FEATS: морфологическая информация: падеж, род, число etc\n", 51 | "* HEAD: id ролителя\n", 52 | "* DEPREL: тип зависимости, то есть отношение к токену-родителю\n", 53 | "* DEPS: альтернативный подграф (не будем углубляться :))\n", 54 | "* MISC: всё остальное\n", 55 | "\n", 56 | "Отсутствующие данные представляются с помощью `_`. Больше подробностей про формат -- в [официальной документаци](http://universaldependencies.org/format.html).
\n", 57 | "User-friendly визуализация: \n", 58 | "\n", 59 | "Отрытый инструмент для визуализации, ручной разметки и конвертации в другие форматы: UD Annotatrix. [Online-интерфейс](https://universal-dependencies.linghub.net/annotatrix), [репозиторий](https://github.com/jonorthwash/ud-annotatrix).\n", 60 | "\n", 61 | "Трибанк -- много таких предложений. Обычно они разделяются двумя переносами строки.\n", 62 | "### Как считывать данные в питоне\n", 63 | "\n", 64 | "Используем библиотеку [conllu](https://github.com/EmilStenstrom/conllu)." 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 2, 70 | "metadata": {}, 71 | "outputs": [ 72 | { 73 | "name": "stdout", 74 | "output_type": "stream", 75 | "text": [ 76 | "Collecting conllu\n", 77 | " Downloading https://files.pythonhosted.org/packages/9e/34/ddfbf22e7477a75ca609d60a831452439383e4ab61bed2b5a1b83d1eef5b/conllu-2.0-py2.py3-none-any.whl\n", 78 | "Installing collected packages: conllu\n", 79 | "Successfully installed conllu-2.0\n", 80 | "\u001b[33mYou are using pip version 8.1.1, however version 19.2.3 is available.\n", 81 | "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n" 82 | ] 83 | } 84 | ], 85 | "source": [ 86 | "!pip3 install conllu" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 6, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "from conllu import parse" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 3, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "name": "stdout", 105 | "output_type": "stream", 106 | "text": [ 107 | "Help on function parse in module conllu:\n", 108 | "\n", 109 | "parse(data, fields=None, field_parsers=None)\n", 110 | "\n" 111 | ] 112 | } 113 | ], 114 | "source": [ 115 | "help(parse)" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 4, 121 | "metadata": {}, 122 | "outputs": [ 123 | { 124 | "data": { 125 | "text/plain": [ 126 | "OrderedDict([('id', 1),\n", 127 | " ('form', 'Перспективы'),\n", 128 | " ('lemma', 'перспектива'),\n", 129 | " ('upostag', 'NOUN'),\n", 130 | " ('xpostag', None),\n", 131 | " ('feats',\n", 132 | " OrderedDict([('Animacy', 'Inan'),\n", 133 | " ('Case', 'Nom'),\n", 134 | " ('Gender', 'Fem'),\n", 135 | " ('Number', 'Plur')])),\n", 136 | " ('head', 0),\n", 137 | " ('deprel', 'ROOT'),\n", 138 | " ('deps', '0:root'),\n", 139 | " ('misc', None)])" 140 | ] 141 | }, 142 | "execution_count": 4, 143 | "metadata": {}, 144 | "output_type": "execute_result" 145 | } 146 | ], 147 | "source": [ 148 | "sentences = parse(my_example)\n", 149 | "sentence = sentences[0]\n", 150 | "sentence[0]" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 5, 156 | "metadata": {}, 157 | "outputs": [ 158 | { 159 | "data": { 160 | "text/plain": [ 161 | "OrderedDict([('id', 6),\n", 162 | " ('form', '.'),\n", 163 | " ('lemma', '.'),\n", 164 | " ('upostag', 'PUNCT'),\n", 165 | " ('xpostag', None),\n", 166 | " ('feats', None),\n", 167 | " ('head', 1),\n", 168 | " ('deprel', 'punct'),\n", 169 | " ('deps', [('punct', 1)]),\n", 170 | " ('misc', None)])" 171 | ] 172 | }, 173 | "execution_count": 5, 174 | "metadata": {}, 175 | "output_type": "execute_result" 176 | } 177 | ], 178 | "source": [ 179 | "sentence[-1]" 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "## Визуализация\n", 187 | "\n", 188 | "В nltk есть DependencyGraph, который умеет рисовать деревья (и ещё многое другое). Для того, чтобы визуализация работала корректно, ему нужна зависимость: graphviz." 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "!apt-get install graphviz\n", 198 | "!pip install graphviz" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": 4, 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [ 207 | "from nltk import DependencyGraph" 208 | ] 209 | }, 210 | { 211 | "cell_type": "markdown", 212 | "metadata": {}, 213 | "source": [ 214 | "В отличие от `conllu`, `DependencyGraph` не справляется с комментариями, поэтому придётся их убрать. Кроме того ему обязательно нужен `deprel` *ROOT* в верхнем регистре, иначе он не находит корень." 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 5, 220 | "metadata": {}, 221 | "outputs": [ 222 | { 223 | "ename": "NameError", 224 | "evalue": "name 'my_example' is not defined", 225 | "output_type": "error", 226 | "traceback": [ 227 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 228 | "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", 229 | "\u001b[0;32m
![\"Drawing\"](\"images/attention.png\")
"
23 | ]
24 | },
25 | {
26 | "cell_type": "markdown",
27 | "metadata": {},
28 | "source": [
29 | "Визуализируя веса, можно понимать стратегию получения результата (от каких частей контекста зависела каждая из частей выхода).\n"
30 | ]
31 | },
32 | {
33 | "cell_type": "markdown",
34 | "metadata": {},
35 | "source": [
36 | "Пример визуализации весов $attention$ в задаче машинного перевода"
37 | ]
38 | },
39 | {
40 | "cell_type": "markdown",
41 | "metadata": {},
42 | "source": [
43 | "![\"Drawing\"](\"images/attention_matrix.png\")
"
44 | ]
45 | },
46 | {
47 | "cell_type": "markdown",
48 | "metadata": {},
49 | "source": [
50 | "При использовании BiLSTM каждый вектор $h_j$ хранит информацию о всей последовательности, но в наибольшей степени о $j$-м слове и его соседях.\n"
51 | ]
52 | },
53 | {
54 | "cell_type": "markdown",
55 | "metadata": {},
56 | "source": [
57 | "Далее при текущем выходе $y_{t−1}$ (с вектором $s_{t−1}$) для вектора каждого входного слова $h_j$ считается $a_{tj}$ – вклад в генерацию следующего выходного вектора ($attention$):\n",
58 | "\n",
59 | "$$a_{tj} = \\frac {e^{e_{tj}}} {\\sum_k e^{e_{tk}}}$$\n",
60 | "\n",
61 | "где $e_{tj} = f (s_{t−1}, h_j )$ – модель выравнивания"
62 | ]
63 | },
64 | {
65 | "cell_type": "markdown",
66 | "metadata": {},
67 | "source": [
68 | "\n",
69 | "Модель выравнивания предсказывает то, насколько хорошо соотносятся входное слово в позиции j и выходное в позиции t (обычно это простая модель, например, однослойная сеть)"
70 | ]
71 | },
72 | {
73 | "cell_type": "markdown",
74 | "metadata": {},
75 | "source": [
76 | "#### Multi-head attention"
77 | ]
78 | },
79 | {
80 | "cell_type": "markdown",
81 | "metadata": {},
82 | "source": [
83 | "Вход: вектор запроса и несколько пар векторов ключей и значений (ключ и значение обычно совпадают)"
84 | ]
85 | },
86 | {
87 | "cell_type": "markdown",
88 | "metadata": {},
89 | "source": [
90 | "![\"Drawing\"](\"images/multi-head-attention.png\")
"
91 | ]
92 | },
93 | {
94 | "cell_type": "markdown",
95 | "metadata": {},
96 | "source": [
97 | "Для запроса и каждого ключа считается вес (линейный слой)\n",
98 | "Значения суммируются с этими весами в итоговый вектор"
99 | ]
100 | },
101 | {
102 | "cell_type": "markdown",
103 | "metadata": {},
104 | "source": [
105 | "Идея: обучать несколько $attention$, в надежде, что они станут отвечать за разные признаки слов"
106 | ]
107 | },
108 | {
109 | "cell_type": "markdown",
110 | "metadata": {},
111 | "source": [
112 | "Результаты пропустим через однослойную сеть, получим на выходе один вектор той же размерности, что и входные вектора."
113 | ]
114 | },
115 | {
116 | "cell_type": "markdown",
117 | "metadata": {},
118 | "source": [
119 | "## Transformer"
120 | ]
121 | },
122 | {
123 | "cell_type": "markdown",
124 | "metadata": {},
125 | "source": [
126 | "Состоит из encoder и decoder, в каждом используются $multi$-$head\\ attention$ и полносвязные (свёрточные) слои (нет RNN)"
127 | ]
128 | },
129 | {
130 | "cell_type": "markdown",
131 | "metadata": {},
132 | "source": [
133 | "Для каждого слова в encoder формируется вектор на основе нескольких слоёв $multi$-$head\\ attention$, который передаётся в декодер."
134 | ]
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {},
139 | "source": [
140 | "\n",
141 | "На основании векторов энкодера, а также выходных векторов для уже обработанных слов получается вектор для текущего слова. В обоих случаях используется $multi$-$head\\ attention$."
142 | ]
143 | },
144 | {
145 | "cell_type": "markdown",
146 | "metadata": {},
147 | "source": [
148 | "![\"Drawing\"](\"images/transformer.png\")
"
149 | ]
150 | },
151 | {
152 | "cell_type": "markdown",
153 | "metadata": {},
154 | "source": [
155 | "$Positional\\ encoding$ – дополнительный вектор признаков для каждого слова, представляющий собой набор значений синусов и косинусов с разными периодами от позиции слова в предложении"
156 | ]
157 | },
158 | {
159 | "cell_type": "markdown",
160 | "metadata": {},
161 | "source": [
162 | "В трансформере и в encoder, и в decoder можно использовать несколько последовательных слоёв $multi$-$head\\ attention$."
163 | ]
164 | },
165 | {
166 | "cell_type": "markdown",
167 | "metadata": {},
168 | "source": [
169 | "Для большей выразительности и качества добавляют полносвязные слои, а также дропаут, нормализацию по слою и residual connections."
170 | ]
171 | },
172 | {
173 | "cell_type": "markdown",
174 | "metadata": {},
175 | "source": [
176 | "Важная особенность: обучение модели внутри предложения можно распараллелить, в отличие от RNN."
177 | ]
178 | },
179 | {
180 | "cell_type": "markdown",
181 | "metadata": {},
182 | "source": [
183 | "## BERT"
184 | ]
185 | },
186 | {
187 | "cell_type": "markdown",
188 | "metadata": {},
189 | "source": [
190 | "$BERT\\ (Bidirectional\\ Encoder\\ Representations\\ from\\ Transformers)\\ -$ многослойный двунаправленный transformer-encoder."
191 | ]
192 | },
193 | {
194 | "cell_type": "markdown",
195 | "metadata": {},
196 | "source": [
197 | "#### Model architecture "
198 | ]
199 | },
200 | {
201 | "cell_type": "markdown",
202 | "metadata": {},
203 | "source": [
204 | "Модель BERT основана на архитектуре Transformer и является усовершенствованием модели GPT от OpenAI.\n",
205 | "Модель, в отличие от GPT является двунаправленной.\n",
206 | "\n",
207 | "Ссылка на оригинальную статью: https://arxiv.org/abs/1810.04805"
208 | ]
209 | },
210 | {
211 | "cell_type": "markdown",
212 | "metadata": {},
213 | "source": [
214 | "![\"Drawing\"](\"images/bert_model.png\")
"
215 | ]
216 | },
217 | {
218 | "cell_type": "markdown",
219 | "metadata": {},
220 | "source": [
221 | "Основные параметры модели:\n",
222 | "* L – количество $transformer$-блоков \n",
223 | "* H – размер скрытого представления,\n",
224 | "* A – количество параллельных слоев в $multi$-$head$ $attention$"
225 | ]
226 | },
227 | {
228 | "cell_type": "markdown",
229 | "metadata": {},
230 | "source": [
231 | "Существует две основные модели от Google:\n",
232 | "\n",
233 | "* $BERT_{BASE}$: \n",
234 | " * L = 12\n",
235 | " * H = 768\n",
236 | " * A = 12, \n",
237 | " * Total = 110M\n",
238 | "$$$$\n",
239 | "* $BERT_{LARGE}$: \n",
240 | " * L = 24\n",
241 | " * H = 1024\n",
242 | " * A = 16, \n",
243 | " * Total = 340M\n"
244 | ]
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "metadata": {},
249 | "source": [
250 | "Модель обучается на двух задачах: \n",
251 | "* Masked Language Modeling.\n",
252 | "* Next sentence prediction."
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {},
258 | "source": [
259 | "#### Masked Language Modeling"
260 | ]
261 | },
262 | {
263 | "cell_type": "markdown",
264 | "metadata": {},
265 | "source": [
266 | "В отличие от обычного Language Modeling, где предсказывается каждое следующее слово на каждом шаге, в задаче Masked Language Modeling случайные токены заменяются на специальный токен [MASK] и предсказываются только эти \"замаскированные\" токены."
267 | ]
268 | },
269 | {
270 | "cell_type": "markdown",
271 | "metadata": {},
272 | "source": [
273 | "* 80% времени обучения: Заменим слово на [MASK].\n",
274 | " * my dog is hairy → my dog is [MASK]\n",
275 | "* 10% времени обучения: Заменим слово на другое случайное слово \n",
276 | " * my dog is hairy → my dog is apple\n",
277 | "* 10% времени обучения: оставим предложение без изменений. Цель этого - сделать модель смещенной в сторону слова, которое действительно встречалось в корпусе в данном предложении. \n",
278 | " * my dog is hairy → my dog is hairy. "
279 | ]
280 | },
281 | {
282 | "cell_type": "markdown",
283 | "metadata": {},
284 | "source": [
285 | "#### Next sentence prediction "
286 | ]
287 | },
288 | {
289 | "cell_type": "markdown",
290 | "metadata": {},
291 | "source": [
292 | "Next sentence prediction $-$ задача бинарной классификации. По паре предложений требуется определить, является ли второе предложение продолжением первого."
293 | ]
294 | },
295 | {
296 | "cell_type": "markdown",
297 | "metadata": {},
298 | "source": [
299 | "* Input =[CLS] the man went to [MASK] store [SEP] he bought a gallon [MASK] milk [SEP]\n",
300 | " * Label = IsNext\n",
301 | "$$$$\n",
302 | "* Input =[CLS] the man [MASK] to the store [SEP] penguin [MASK] are flight ##less birds [SEP]\n",
303 | " * Label = NotNext"
304 | ]
305 | },
306 | {
307 | "cell_type": "markdown",
308 | "metadata": {},
309 | "source": [
310 | "#### BERT Embeddings"
311 | ]
312 | },
313 | {
314 | "cell_type": "markdown",
315 | "metadata": {},
316 | "source": [
317 | "Посмотрим на эмбеддинги, получаемые с помощью модели BERT. \n",
318 | "\n",
319 | "Будем использовать модель $BERT_{base}$ из реализации pytorch_pretrained_bert от HuggingFace. \n",
320 | "Ссылка на их гитхаб: https://github.com/huggingface/transformers"
321 | ]
322 | },
323 | {
324 | "cell_type": "markdown",
325 | "metadata": {},
326 | "source": [
327 | "Примеры текстов взяты из туториала (https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/)"
328 | ]
329 | },
330 | {
331 | "cell_type": "code",
332 | "execution_count": 3,
333 | "metadata": {},
334 | "outputs": [
335 | {
336 | "name": "stderr",
337 | "output_type": "stream",
338 | "text": [
339 | "100%|██████████| 231508/231508 [00:00<00:00, 599772.53B/s]\n"
340 | ]
341 | }
342 | ],
343 | "source": [
344 | "import torch\n",
345 | "from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM\n",
346 | "\n",
347 | "# OPTIONAL: if you want to have more information on what's happening, activate the logger as follows\n",
348 | "import logging\n",
349 | "#logging.basicConfig(level=logging.INFO)\n",
350 | "\n",
351 | "import matplotlib.pyplot as plt\n",
352 | "# % matplotlib inline\n",
353 | "\n",
354 | "# Load pre-trained model tokenizer (vocabulary)\n",
355 | "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
356 | ]
357 | },
358 | {
359 | "cell_type": "code",
360 | "execution_count": 4,
361 | "metadata": {},
362 | "outputs": [
363 | {
364 | "name": "stdout",
365 | "output_type": "stream",
366 | "text": [
367 | "[CLS] After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank. [SEP]\n"
368 | ]
369 | }
370 | ],
371 | "source": [
372 | "text = \"Here is the sentence I want embeddings for.\"\n",
373 | "text = \"After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank.\"\n",
374 | "marked_text = \"[CLS] \" + text + \" [SEP]\"\n",
375 | "\n",
376 | "print (marked_text)"
377 | ]
378 | },
379 | {
380 | "cell_type": "code",
381 | "execution_count": 5,
382 | "metadata": {},
383 | "outputs": [
384 | {
385 | "name": "stdout",
386 | "output_type": "stream",
387 | "text": [
388 | "['[CLS]', 'after', 'stealing', 'money', 'from', 'the', 'bank', 'vault', ',', 'the', 'bank', 'robber', 'was', 'seen', 'fishing', 'on', 'the', 'mississippi', 'river', 'bank', '.', '[SEP]']\n"
389 | ]
390 | }
391 | ],
392 | "source": [
393 | "tokenized_text = tokenizer.tokenize(marked_text)\n",
394 | "print (tokenized_text)"
395 | ]
396 | },
397 | {
398 | "cell_type": "code",
399 | "execution_count": 6,
400 | "metadata": {},
401 | "outputs": [
402 | {
403 | "data": {
404 | "text/plain": [
405 | "['knight',\n",
406 | " 'lap',\n",
407 | " 'survey',\n",
408 | " 'ma',\n",
409 | " '##ow',\n",
410 | " 'noise',\n",
411 | " 'billy',\n",
412 | " '##ium',\n",
413 | " 'shooting',\n",
414 | " 'guide',\n",
415 | " 'bedroom',\n",
416 | " 'priest',\n",
417 | " 'resistance',\n",
418 | " 'motor',\n",
419 | " 'homes',\n",
420 | " 'sounded',\n",
421 | " 'giant',\n",
422 | " '##mer',\n",
423 | " '150',\n",
424 | " 'scenes']"
425 | ]
426 | },
427 | "execution_count": 6,
428 | "metadata": {},
429 | "output_type": "execute_result"
430 | }
431 | ],
432 | "source": [
433 | "list(tokenizer.vocab.keys())[5000:5020]\n"
434 | ]
435 | },
436 | {
437 | "cell_type": "code",
438 | "execution_count": 7,
439 | "metadata": {},
440 | "outputs": [
441 | {
442 | "name": "stdout",
443 | "output_type": "stream",
444 | "text": [
445 | "('[CLS]', 101)\n",
446 | "('after', 2044)\n",
447 | "('stealing', 11065)\n",
448 | "('money', 2769)\n",
449 | "('from', 2013)\n",
450 | "('the', 1996)\n",
451 | "('bank', 2924)\n",
452 | "('vault', 11632)\n",
453 | "(',', 1010)\n",
454 | "('the', 1996)\n",
455 | "('bank', 2924)\n",
456 | "('robber', 27307)\n",
457 | "('was', 2001)\n",
458 | "('seen', 2464)\n",
459 | "('fishing', 5645)\n",
460 | "('on', 2006)\n",
461 | "('the', 1996)\n",
462 | "('mississippi', 5900)\n",
463 | "('river', 2314)\n",
464 | "('bank', 2924)\n",
465 | "('.', 1012)\n",
466 | "('[SEP]', 102)\n"
467 | ]
468 | }
469 | ],
470 | "source": [
471 | "indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)\n",
472 | "\n",
473 | "for tup in zip(tokenized_text, indexed_tokens):\n",
474 | " print (tup)"
475 | ]
476 | },
477 | {
478 | "cell_type": "code",
479 | "execution_count": 8,
480 | "metadata": {},
481 | "outputs": [
482 | {
483 | "name": "stdout",
484 | "output_type": "stream",
485 | "text": [
486 | "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n"
487 | ]
488 | }
489 | ],
490 | "source": [
491 | "segments_ids = [1] * len(tokenized_text)\n",
492 | "print (segments_ids)"
493 | ]
494 | },
495 | {
496 | "cell_type": "code",
497 | "execution_count": 9,
498 | "metadata": {},
499 | "outputs": [
500 | {
501 | "name": "stderr",
502 | "output_type": "stream",
503 | "text": [
504 | "100%|██████████| 407873900/407873900 [02:57<00:00, 2298162.71B/s]\n"
505 | ]
506 | },
507 | {
508 | "data": {
509 | "text/plain": [
510 | "BertModel(\n",
511 | " (embeddings): BertEmbeddings(\n",
512 | " (word_embeddings): Embedding(30522, 768)\n",
513 | " (position_embeddings): Embedding(512, 768)\n",
514 | " (token_type_embeddings): Embedding(2, 768)\n",
515 | " (LayerNorm): BertLayerNorm()\n",
516 | " (dropout): Dropout(p=0.1, inplace=False)\n",
517 | " )\n",
518 | " (encoder): BertEncoder(\n",
519 | " (layer): ModuleList(\n",
520 | " (0): BertLayer(\n",
521 | " (attention): BertAttention(\n",
522 | " (self): BertSelfAttention(\n",
523 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
524 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
525 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
526 | " (dropout): Dropout(p=0.1, inplace=False)\n",
527 | " )\n",
528 | " (output): BertSelfOutput(\n",
529 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
530 | " (LayerNorm): BertLayerNorm()\n",
531 | " (dropout): Dropout(p=0.1, inplace=False)\n",
532 | " )\n",
533 | " )\n",
534 | " (intermediate): BertIntermediate(\n",
535 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
536 | " )\n",
537 | " (output): BertOutput(\n",
538 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
539 | " (LayerNorm): BertLayerNorm()\n",
540 | " (dropout): Dropout(p=0.1, inplace=False)\n",
541 | " )\n",
542 | " )\n",
543 | " (1): BertLayer(\n",
544 | " (attention): BertAttention(\n",
545 | " (self): BertSelfAttention(\n",
546 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
547 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
548 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
549 | " (dropout): Dropout(p=0.1, inplace=False)\n",
550 | " )\n",
551 | " (output): BertSelfOutput(\n",
552 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
553 | " (LayerNorm): BertLayerNorm()\n",
554 | " (dropout): Dropout(p=0.1, inplace=False)\n",
555 | " )\n",
556 | " )\n",
557 | " (intermediate): BertIntermediate(\n",
558 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
559 | " )\n",
560 | " (output): BertOutput(\n",
561 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
562 | " (LayerNorm): BertLayerNorm()\n",
563 | " (dropout): Dropout(p=0.1, inplace=False)\n",
564 | " )\n",
565 | " )\n",
566 | " (2): BertLayer(\n",
567 | " (attention): BertAttention(\n",
568 | " (self): BertSelfAttention(\n",
569 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
570 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
571 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
572 | " (dropout): Dropout(p=0.1, inplace=False)\n",
573 | " )\n",
574 | " (output): BertSelfOutput(\n",
575 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
576 | " (LayerNorm): BertLayerNorm()\n",
577 | " (dropout): Dropout(p=0.1, inplace=False)\n",
578 | " )\n",
579 | " )\n",
580 | " (intermediate): BertIntermediate(\n",
581 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
582 | " )\n",
583 | " (output): BertOutput(\n",
584 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
585 | " (LayerNorm): BertLayerNorm()\n",
586 | " (dropout): Dropout(p=0.1, inplace=False)\n",
587 | " )\n",
588 | " )\n",
589 | " (3): BertLayer(\n",
590 | " (attention): BertAttention(\n",
591 | " (self): BertSelfAttention(\n",
592 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
593 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
594 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
595 | " (dropout): Dropout(p=0.1, inplace=False)\n",
596 | " )\n",
597 | " (output): BertSelfOutput(\n",
598 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
599 | " (LayerNorm): BertLayerNorm()\n",
600 | " (dropout): Dropout(p=0.1, inplace=False)\n",
601 | " )\n",
602 | " )\n",
603 | " (intermediate): BertIntermediate(\n",
604 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
605 | " )\n",
606 | " (output): BertOutput(\n",
607 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
608 | " (LayerNorm): BertLayerNorm()\n",
609 | " (dropout): Dropout(p=0.1, inplace=False)\n",
610 | " )\n",
611 | " )\n",
612 | " (4): BertLayer(\n",
613 | " (attention): BertAttention(\n",
614 | " (self): BertSelfAttention(\n",
615 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
616 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
617 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
618 | " (dropout): Dropout(p=0.1, inplace=False)\n",
619 | " )\n",
620 | " (output): BertSelfOutput(\n",
621 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
622 | " (LayerNorm): BertLayerNorm()\n",
623 | " (dropout): Dropout(p=0.1, inplace=False)\n",
624 | " )\n",
625 | " )\n",
626 | " (intermediate): BertIntermediate(\n",
627 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
628 | " )\n",
629 | " (output): BertOutput(\n",
630 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
631 | " (LayerNorm): BertLayerNorm()\n",
632 | " (dropout): Dropout(p=0.1, inplace=False)\n",
633 | " )\n",
634 | " )\n",
635 | " (5): BertLayer(\n",
636 | " (attention): BertAttention(\n",
637 | " (self): BertSelfAttention(\n",
638 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
639 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
640 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
641 | " (dropout): Dropout(p=0.1, inplace=False)\n",
642 | " )\n",
643 | " (output): BertSelfOutput(\n",
644 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
645 | " (LayerNorm): BertLayerNorm()\n",
646 | " (dropout): Dropout(p=0.1, inplace=False)\n",
647 | " )\n",
648 | " )\n",
649 | " (intermediate): BertIntermediate(\n",
650 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
651 | " )\n",
652 | " (output): BertOutput(\n",
653 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
654 | " (LayerNorm): BertLayerNorm()\n",
655 | " (dropout): Dropout(p=0.1, inplace=False)\n",
656 | " )\n",
657 | " )\n",
658 | " (6): BertLayer(\n",
659 | " (attention): BertAttention(\n",
660 | " (self): BertSelfAttention(\n",
661 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
662 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
663 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
664 | " (dropout): Dropout(p=0.1, inplace=False)\n",
665 | " )\n",
666 | " (output): BertSelfOutput(\n",
667 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
668 | " (LayerNorm): BertLayerNorm()\n",
669 | " (dropout): Dropout(p=0.1, inplace=False)\n",
670 | " )\n",
671 | " )\n",
672 | " (intermediate): BertIntermediate(\n",
673 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
674 | " )\n",
675 | " (output): BertOutput(\n",
676 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
677 | " (LayerNorm): BertLayerNorm()\n",
678 | " (dropout): Dropout(p=0.1, inplace=False)\n",
679 | " )\n",
680 | " )\n",
681 | " (7): BertLayer(\n",
682 | " (attention): BertAttention(\n",
683 | " (self): BertSelfAttention(\n",
684 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
685 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
686 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
687 | " (dropout): Dropout(p=0.1, inplace=False)\n",
688 | " )\n",
689 | " (output): BertSelfOutput(\n",
690 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
691 | " (LayerNorm): BertLayerNorm()\n",
692 | " (dropout): Dropout(p=0.1, inplace=False)\n",
693 | " )\n",
694 | " )\n",
695 | " (intermediate): BertIntermediate(\n",
696 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
697 | " )\n",
698 | " (output): BertOutput(\n",
699 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
700 | " (LayerNorm): BertLayerNorm()\n",
701 | " (dropout): Dropout(p=0.1, inplace=False)\n",
702 | " )\n",
703 | " )\n",
704 | " (8): BertLayer(\n",
705 | " (attention): BertAttention(\n",
706 | " (self): BertSelfAttention(\n",
707 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
708 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
709 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
710 | " (dropout): Dropout(p=0.1, inplace=False)\n",
711 | " )\n",
712 | " (output): BertSelfOutput(\n",
713 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
714 | " (LayerNorm): BertLayerNorm()\n",
715 | " (dropout): Dropout(p=0.1, inplace=False)\n",
716 | " )\n",
717 | " )\n",
718 | " (intermediate): BertIntermediate(\n",
719 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
720 | " )\n",
721 | " (output): BertOutput(\n",
722 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
723 | " (LayerNorm): BertLayerNorm()\n",
724 | " (dropout): Dropout(p=0.1, inplace=False)\n",
725 | " )\n",
726 | " )\n",
727 | " (9): BertLayer(\n",
728 | " (attention): BertAttention(\n",
729 | " (self): BertSelfAttention(\n",
730 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
731 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
732 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
733 | " (dropout): Dropout(p=0.1, inplace=False)\n",
734 | " )\n",
735 | " (output): BertSelfOutput(\n",
736 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
737 | " (LayerNorm): BertLayerNorm()\n",
738 | " (dropout): Dropout(p=0.1, inplace=False)\n",
739 | " )\n",
740 | " )\n",
741 | " (intermediate): BertIntermediate(\n",
742 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
743 | " )\n",
744 | " (output): BertOutput(\n",
745 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
746 | " (LayerNorm): BertLayerNorm()\n",
747 | " (dropout): Dropout(p=0.1, inplace=False)\n",
748 | " )\n",
749 | " )\n",
750 | " (10): BertLayer(\n",
751 | " (attention): BertAttention(\n",
752 | " (self): BertSelfAttention(\n",
753 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
754 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
755 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
756 | " (dropout): Dropout(p=0.1, inplace=False)\n",
757 | " )\n",
758 | " (output): BertSelfOutput(\n",
759 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
760 | " (LayerNorm): BertLayerNorm()\n",
761 | " (dropout): Dropout(p=0.1, inplace=False)\n",
762 | " )\n",
763 | " )\n",
764 | " (intermediate): BertIntermediate(\n",
765 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
766 | " )\n",
767 | " (output): BertOutput(\n",
768 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
769 | " (LayerNorm): BertLayerNorm()\n",
770 | " (dropout): Dropout(p=0.1, inplace=False)\n",
771 | " )\n",
772 | " )\n",
773 | " (11): BertLayer(\n",
774 | " (attention): BertAttention(\n",
775 | " (self): BertSelfAttention(\n",
776 | " (query): Linear(in_features=768, out_features=768, bias=True)\n",
777 | " (key): Linear(in_features=768, out_features=768, bias=True)\n",
778 | " (value): Linear(in_features=768, out_features=768, bias=True)\n",
779 | " (dropout): Dropout(p=0.1, inplace=False)\n",
780 | " )\n",
781 | " (output): BertSelfOutput(\n",
782 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
783 | " (LayerNorm): BertLayerNorm()\n",
784 | " (dropout): Dropout(p=0.1, inplace=False)\n",
785 | " )\n",
786 | " )\n",
787 | " (intermediate): BertIntermediate(\n",
788 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
789 | " )\n",
790 | " (output): BertOutput(\n",
791 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
792 | " (LayerNorm): BertLayerNorm()\n",
793 | " (dropout): Dropout(p=0.1, inplace=False)\n",
794 | " )\n",
795 | " )\n",
796 | " )\n",
797 | " )\n",
798 | " (pooler): BertPooler(\n",
799 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
800 | " (activation): Tanh()\n",
801 | " )\n",
802 | ")"
803 | ]
804 | },
805 | "execution_count": 9,
806 | "metadata": {},
807 | "output_type": "execute_result"
808 | }
809 | ],
810 | "source": [
811 | "# Convert inputs to PyTorch tensors\n",
812 | "tokens_tensor = torch.tensor([indexed_tokens])\n",
813 | "segments_tensors = torch.tensor([segments_ids])\n",
814 | "\n",
815 | "# Load pre-trained model (weights)\n",
816 | "model = BertModel.from_pretrained('bert-base-uncased')\n",
817 | "\n",
818 | "# Put the model in \"evaluation\" mode, meaning feed-forward operation.\n",
819 | "model.eval()"
820 | ]
821 | },
822 | {
823 | "cell_type": "code",
824 | "execution_count": 10,
825 | "metadata": {},
826 | "outputs": [],
827 | "source": [
828 | "# Predict hidden states features for each layer\n",
829 | "with torch.no_grad():\n",
830 | " encoded_layers, _ = model(tokens_tensor, segments_tensors)"
831 | ]
832 | },
833 | {
834 | "cell_type": "code",
835 | "execution_count": 11,
836 | "metadata": {},
837 | "outputs": [
838 | {
839 | "name": "stdout",
840 | "output_type": "stream",
841 | "text": [
842 | "Number of layers: 12\n",
843 | "Number of batches: 1\n",
844 | "Number of tokens: 22\n",
845 | "Number of hidden units: 768\n"
846 | ]
847 | }
848 | ],
849 | "source": [
850 | "print (\"Number of layers:\", len(encoded_layers))\n",
851 | "layer_i = 0\n",
852 | "\n",
853 | "print (\"Number of batches:\", len(encoded_layers[layer_i]))\n",
854 | "batch_i = 0\n",
855 | "\n",
856 | "print (\"Number of tokens:\", len(encoded_layers[layer_i][batch_i]))\n",
857 | "token_i = 0\n",
858 | "\n",
859 | "print (\"Number of hidden units:\", len(encoded_layers[layer_i][batch_i][token_i]))"
860 | ]
861 | },
862 | {
863 | "cell_type": "code",
864 | "execution_count": 12,
865 | "metadata": {},
866 | "outputs": [
867 | {
868 | "data": {
869 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAJCCAYAAADky0LWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFQFJREFUeJzt3XuM5fdZ3/HPgzcJSIiG1NsQxUnHiAByoDhoMUG0ojgEDEtJCikKQuCqQRZXJW0QnSRVJSQqbQAREGr/sHCEQVFDSkIdsSBIQ7i0Ig7rXAiOCTFhgdzwBoigqhrk5ukfc9bsenc943lm5ndm5vWSoj1Xncdfb2be/s6Z863uDgAAu/NpSw8AAHCYiSkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADJw4yBe7/vrre2Nj4yBfEgBgV+67776Pd/fJ7R53oDG1sbGRc+fOHeRLAgDsSlX96U4e58d8AAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGDix9AAAcNxtbJ595PL5M6d3/NidPof9ZWcKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGdhxTVXVdVb2rqn55df3Gqrq3qh6sql+oqifu35gAAOvp8exMvTTJA5dcf3WS13T35yX56yQv2cvBAAAOgx3FVFXdkOR0kp9ZXa8ktyb5xdVD7k7ywv0YEABgne10Z+onk/xQkk+trv/DJJ/o7odX1z+U5Ol7PBsAwNo7sd0DquobkzzU3fdV1T9/vC9QVXckuSNJnvnMZz7uAQHgKNnYPPvI5fNnTi84CXtlJztTX5nkm6rqfJLXZ+vHez+V5MlVdTHGbkjy4as9ubvv7O5T3X3q5MmTezAyAMD62DamuvsV3X1Dd28keXGS3+jub0/ytiQvWj3s9iT37NuUAABravI5U/8+yb+rqgez9R6qu/ZmJACAw2Pb90xdqrt/M8lvri5/MMktez8SAMDh4RPQAQAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQBYIxubZy87cob1J6YAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwMCJpQcAAPbGxubZRy6fP3N6wUmOFztTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGDAcTIAsIamR8NcfL5jZfafnSkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGnM0HAGvu0nP6WD92pgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgONkAOCYufR4mvNnTi84ydFgZwoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMDAiaUHAABmNjbPLj3CsWZnCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGNg2pqrq06vqHVX1nqq6v6p+eHX7jVV1b1U9WFW/UFVP3P9xAQDWy052pj6Z5Nbu/pIkNye5raqem+TVSV7T3Z+X5K+TvGT/xgQAWE/bxlRv+d+rq09Y/a+T3JrkF1e3353khfsyIQDAGtvRe6aq6rqqeneSh5K8JckfJ/lEdz+8esiHkjx9f0YEAFhfO4qp7v5/3X1zkhuS3JLkC3f6AlV1R1Wdq6pzFy5c2OWYAADr6XH9Nl93fyLJ25J8RZInV9XFg5JvSPLhazznzu4+1d2nTp48ORoWAGDd7OS3+U5W1ZNXlz8jyfOTPJCtqHrR6mG3J7lnv4YEAFhXJ7Z/SJ6W5O6qui5b8fWG7v7lqnpfktdX1Y8keVeSu/ZxTgCAtbRtTHX37yd5zlVu/2C23j8FAHBs+QR0AIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAAM7+ZwpAGAXNjbPPnL5/JnTj3k/h5edKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAaczQcAR9h25wMyZ2cKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBg4MTSAwAAB2Nj8+zSIxxJdqYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwMCJpQcAgONgY/Ps0iNc1dXmOn/m9AKTHF52pgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwMC2MVVVz6iqt1XV+6rq/qp66er2p1TVW6rqA6s/P3v/xwUAWC872Zl6OMnLu/umJM9N8n1VdVOSzSRv7e5nJXnr6joAwLGybUx190e7+52ry3+b5IEkT0/ygiR3rx52d5IX7teQAADr6nG9Z6qqNpI8J8m9SZ7a3R9d3fWxJE/d08kAAA6BHcdUVX1mkjcmeVl3/82l93V3J+lrPO+OqjpXVecuXLgwGhYAYN3sKKaq6gnZCqnXdfebVjf/RVU9bXX/05I8dLXndved3X2qu0+dPHlyL2YGAFgbO/ltvkpyV5IHuvsnLrnrzUluX12+Pck9ez8eAMB6O7GDx3xlku9I8t6qevfqtlcmOZPkDVX1kiR/muRb92dEAID1tW1Mdff/TFLXuPt5ezsOAMDh4hPQAQAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAwImlBwCAw2Zj8+wjl8+fOb3gJKwDO1MAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGTiw9AAAcBRubZx+5fP7M6QUn4aDZmQIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADBwYukBAOCo2dg8u/QIHCA7UwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAZOLD0AABxmG5tnlx5hz138Zzp/5vS+PueosDMFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAYcJwMAXNXVjso5jsfFbMfOFADAgJgCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMLBtTFXVa6vqoar6g0tue0pVvaWqPrD687P3d0wAgPW0k52pn01y26Nu20zy1u5+VpK3rq4DABw728ZUd/92kr961M0vSHL36vLdSV64x3MBABwKu33P1FO7+6Oryx9L8tQ9mgcA4FAZvwG9uztJX+v+qrqjqs5V1bkLFy5MXw4AYK3sNqb+oqqeliSrPx+61gO7+87uPtXdp06ePLnLlwMAWE+7jak3J7l9dfn2JPfszTgAAIfLTj4a4b8m+d0kX1BVH6qqlyQ5k+T5VfWBJF+zug4AcOyc2O4B3f1t17jreXs8CwDAoeMT0AEABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMbPvRCAAAF21snn3k8vkzpxecZH3YmQIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGDA2XwAwJ45jmf32ZkCABgQUwAAA2IKAGBATAEADIgpAIABMQUAMCCmAAAGxBQAwICYAgAYEFMAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwcGLpAQDgsNjYPLv0CKwhO1MAAANiCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYMBxMgDwKI6N2RsX1/H8mdMLT7K/7EwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAw4Gw+AGBXnGG4xc4UAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABhwnAwAx5ojUQ7WxfU+f+b0wpPsHTtTAAADYgoAYEBMAQAMiCkAgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADDibD4Aj5Wpnv13t/L2jdDbcurva+l9622H/d2FnCgBgQEwBAAyIKQCAATEFADAgpgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMHLnjZI7Sx9MDHFbr8LX4akeYsJ52etzP1Y4KWgd2pgAABsQUAMCAmAIAGBBTAAADYgoAYEBMAQAMiCkAgIFRTFXVbVX1/qp6sKo292ooAIDDYtcxVVXXJfnPSb4+yU1Jvq2qbtqrwQAADoPJztQtSR7s7g92998leX2SF+zNWAAAh8Mkpp6e5M8vuf6h1W0AAMdGdffunlj1oiS3dfd3ra5/R5Iv7+7vf9Tj7khyx+rqFyR5/+7HPTSuT/LxpYdYI9bjStbkctbjStbkctbjctbjSvuxJv+4u09u96DJQccfTvKMS67fsLrtMt19Z5I7B69z6FTVue4+tfQc68J6XMmaXM56XMmaXM56XM56XGnJNZn8mO/3kjyrqm6sqicmeXGSN+/NWAAAh8Oud6a6++Gq+v4kv5bkuiSv7e7792wyAIBDYPJjvnT3ryT5lT2a5Sg5Vj/W3AHrcSVrcjnrcSVrcjnrcTnrcaXF1mTXb0AHAMBxMgAAI2Jqj1TVv6qq+6vqU1V16pLbn19V91XVe1d/3rrknAfpWmuyuu8Vq2OI3l9VX7fUjEupqpur6u1V9e6qOldVtyw90zqoqh+oqj9c/b350aXnWQdV9fKq6qq6fulZllZVP7b6+/H7VfVLVfXkpWdagqPc/l5VPaOq3lZV71t93XjpEnOIqb3zB0m+OclvP+r2jyf5F939xUluT/LzBz3Ygq66Jqtjh16c5NlJbkvyX1bHEx0nP5rkh7v75iT/cXX9WKuqr87WKQpf0t3PTvLjC4+0uKp6RpKvTfJnS8+yJt6S5Iu6+58k+aMkr1h4ngPnKLcrPJzk5d19U5LnJvm+JdZDTO2R7n6gu6/4QNLufld3f2R19f4kn1FVTzrY6ZZxrTXJ1jfM13f3J7v7T5I8mK3jiY6TTvJZq8v/IMlHHuOxx8X3JDnT3Z9Mku5+aOF51sFrkvxQtv6+HHvd/evd/fDq6tuz9fmGx42j3C7R3R/t7neuLv9tkgeywGksYupgfUuSd178ZnGMOYooeVmSH6uqP8/WDsyx+y/sq/j8JP+squ6tqt+qqi9beqAlVdULkny4u9+z9Cxr6t8k+dWlh1iAr5/XUFUbSZ6T5N6Dfu3RRyMcN1X1P5J8zlXuelV337PNc5+d5NXZ2rI/MiZrctQ91tokeV6Sf9vdb6yqb01yV5KvOcj5lrDNmpxI8pRsbdV/WZI3VNXn9hH+leNt1uOVOWJfL3ZiJ19TqupV2frxzusOcjbWV1V9ZpI3JnlZd//NQb++mHocuntX3+yq6oYkv5TkO7v7j/d2qmXtck12dBTRYfdYa1NVP5fk4hsl/1uSnzmQoRa2zZp8T5I3reLpHVX1qWydtXXhoOY7aNdaj6r64iQ3JnlPVSVb/x95Z1Xd0t0fO8ARD9x2X1Oq6l8n+cYkzzvKof0YjsXXz8ejqp6QrZB6XXe/aYkZ/Jhvn61+2+Rsks3u/l9Lz7Mm3pzkxVX1pKq6Mcmzkrxj4ZkO2keSfNXq8q1JPrDgLOvivyf56iSpqs9P8sQc04Ncu/u93f2Punujuzey9aOcLz3qIbWdqrotW+8h+6bu/j9Lz7MQR7ldorb+a+OuJA90908sNsfxDPu9V1X/MslPJzmZ5BNJ3t3dX1dV/yFb74e59Jvl1x6HN9dea01W970qW+95eDhb27LH6r0PVfVPk/xUtnaH/2+S7+3u+5adalmrbwyvTXJzkr9L8oPd/RvLTrUequp8klPdfSzj8qKqejDJk5L85eqmt3f3dy840iKq6huS/GT+/ii3/7TwSItZfS39nSTvTfKp1c2vXJ3QcnBziCkAgN3zYz4AgAExBQAwIKYAAAbEFADAgJgCABgQUwAAA2IKAGBATAEADPx/cTZrQofZjmsAAAAASUVORK5CYII=\n",
870 | "text/plain": [
871 | "