├── README.md ├── images ├── Figure 2.PNG ├── Figure 3.PNG ├── Table 1.PNG ├── Table 2.PNG ├── Table 3.PNG ├── Table 4.PNG ├── Table 5.PNG ├── Table 6.PNG ├── finetuned-bert-2.png ├── finetuned-bert-3.png ├── finetuned-biobert-2.png ├── finetuned-biobert-3.png ├── finetuned-scibert-2.png └── finetuned-scibert-3.png └── notebooks ├── BertModels4ArticleClassification.ipynb ├── BertModelsAttendedWordsOverAverage.ipynb └── BertModelsAttentionHeadsVisualization.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # Classifying Scientific Publications with BERT - Is Self-Attention a Feature Selection Method? 2 | 3 | The annotation and classification of scientific literature is a crucial task to make scientific knowledge easily discoverable, accessible, and reusable. We address the annotation as a multilabel classification task using **BERT** and its different flavors specialized in the scientific domain: **BioBert** and **SciBERT**. In our experiments, using papers from **Springer Nature SciGraph**, we confirm that using transformers to train scientific classifiers generally results in greater accuracies compared to linear classifiers (e.g., LinearSVM, FastText) 4 | 5 | To shed light on BERT internals, we analyze the self-attention mechanism inherent of the transformer architecture. Our findings show that **the last layer of BERT attends to words that are semantically relevant for the scientific fields associated with each publication** (see [Exploring self-attention heads](#exploring-self-attention-heads)). This observation suggests that self-attention actually performs some type of feature selection for the fine-tuned model. 6 | 7 | This repository includes jupyter notebooks to reproduce the experiments reported in the paper **Classifying Scientific Publications with BERT - Is Self-Attention a Feature Selection Method?** accepted in the *43rd European Conference on Information Retrieval* **ECIR 2021**. In addition, we present some of the paper highlights. 8 | 9 | 10 | # Table of content 11 | 12 | * [Jupyter notebooks](#Jupyter-notebooks) 13 | * [Fine-tuning language models for text classification](#fine-tuning-language-models-for-text-classification) 14 | + [Experimental results](#experimental-results) 15 | * [Exploring self-attention heads](#exploring-self-attention-heads) 16 | * [Feature selection](#feature-selection) 17 | + [Domain knowledge](#domain-knowledge) 18 | + [Feature evaluation](#feature-evaluation) 19 | * [Citation](#citation) 20 | 21 | ## Jupyter notebooks 22 | In the __notebooks__ directory of this repository we release self-contained notebooks, including dataset and required libraries, that allows to reproduce the following experiments: 23 | 24 | * Fine-tune BERT, SciBERT and BioBERT to classify research articles into multiple research fields in the ANZRSC taxonomy. 25 | 26 | [BertModels4ArticleClassification.ipynb](./notebooks/BertModels4ArticleClassification.ipynb) 27 | 28 | * Visualize self-attention in the last layer of the BERT models 29 | 30 | [BertModelsAttentionHeadsVisualization.ipynb](./notebooks/BertModelsAttentionHeadsVisualization.ipynb) 31 | 32 | * Get lists of most attended words above average in the last layer of the BERT models 33 | 34 | [BertModelsAttendedWordsOverAverage.ipynb](./notebooks/BertModelsAttendedWordsOverAverage.ipynb) 35 | 36 | ### Running the notebooks in Colaboratory 37 | While these notebooks can be run in _google colaboratory_ with a subset of the articles, to train on the full set of articles used in the paper we advise to use other infrastructure. 38 | 39 | To run the notebooks in Colaboratory go to *https://colab.research.google.com* and use the option *open notebook* from *GitHub* where you can copy and paste the full url of each notebook. 40 | 41 | ## Fine-tuning language models for text classification 42 | 43 | We put into test the following language models to classify research articles: i) BERT and GPT-2, pre-trained on a general-purpose corpus, ii) SciBERT, pre-trained solely on scientific documents, and iii) BioBERT, pre-trained on a combination of general and scientific text. Below we describe some properties about the pre-training of each language model. 44 | 45 |

46 |
47 | Language models pre-training information. 48 |

49 | 50 | To fine-tune BERT models we take the last layer encoding of the classification token ```` and add an N-dimensional linear layer, with N the number of classification labels. We use a binary cross-entropy loss function (_BCEWithLogitsLoss_) 51 | to allow the model to assign independent probabilities to each label. We train the models for 4 epochs, with batch size 8 and 2e-5 learning rate. 52 | 53 | ### Experimental results 54 | Below we present the evaluation of the classifiers for 22 first level categories in the ANZRSC (right hand side) and second level categories (left hand side) 55 | 56 |

57 |
58 | Evaluation results of the multilabel classifiers (f-measure) on first level categories (a), and on second level categories (b). 59 |

60 | 61 | ## Exploring self-attention heads 62 | The following images depict the mean weights of the 12 self-attention heads in the last hidden state of the fine-tuned models for two papers titled _BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding_, and _A universal long-term flu vaccine may not prevent severe epidemics_. 63 | 64 | The plots clearly show the so-called vertical pattern, where a few tokens receive most of the attention, such as **training, deep, transformer, language**, and **understanding** in the first sentence, and **flu, vaccine, prevent, severe** and **epidemic** in the second. We do not include in the plots ```` or ````. 65 | 66 | |

BERT

|

BERT

| 67 | |:---:|:---:| 68 | |

SciBERT

|

SciBERT

| 69 | |

BioBERT-1.1

|

BioBERT-1.1

| 70 | 71 | We also extract the words with higher attention than the average attention in each sentence. Note that the words are highly relevant to the corresponding scientific category. 72 | 73 |

74 | Visualization of average weights in the self attention heads of the last layer. 75 |

76 | 77 |

78 |
79 | Most attended words above average attention in the fine-tuned models. 80 |

81 | 82 | ## Feature selection 83 | In this section we compare the lists of the attended words above average attention with feature selection algorithms frequently used in text classification. 84 | 85 |

86 |
87 | Word overlap: most attended and feature selection results. algorithm. 88 |

89 | We also compare the rankings using the Rank-Biased overlap metric: 90 | 91 |

92 |
93 | Rank-biased overlap at different p values (X axis) between most attended words and selected by feature selection algorithm. 94 |

95 | 96 | ### Domain knowledge 97 | We search the words attended above average in **ConceptNet** and leverage the relation _HasContext_ to identify the domains where they are commonly used: 98 | 99 |

100 |
101 | Number of words per ANZRSC category that match the corresponding context in ConceptNet. 102 |

103 | 104 | In BERT and SciBERT, self-attention identifies more domain-relevant words than feature selection methods. However, this is not the case for BioBert. 105 | 106 | Weighing the words by their term frequency (TF), attended words remain more domain-relevant than those obtained through feature selection. In fact, the domain relevance of the frequent attended words is greater or on pair with those selected when TF/IDF is used to weigh the output of feature selection method 107 | 108 | ### Feature evaluation 109 | 110 | To measure the stability of the features we compute the mean Jaccard coefficient between the different subsets of words generated by each method using 5-folds: 111 | 112 |

113 |
114 | Stability of the features measured using Jackard similarity coefficient. 115 |

116 | 117 | We use the set of features to learn classifiers for the 22 first level categories using Logistic Regression (LR), Naive Bayes (NB), Random Forest (RF), Neural Networks (NN), and SVM: 118 | 119 |

120 |
121 | Classifiers performance using distinct feature sets and number of features. x axis is the number of features used to train each classifier. 122 |

123 | 124 | We observe that traditional feature selection methods like chi-square and information gain mainly help to learn more accurate classifiers than the set of most attended words by the language models. This observation clearly indicates that the success of BERT models in this task is not only driven by the self-attention mechanism but also by the contextualized outputs of the transformer, which are the input of the added classification layer. 125 | 126 | 127 | ## Citation 128 | To reference this work use the following citation: 129 | 130 | ``` 131 | @inproceedings{DBLP:conf/ecir/Garcia-SilvaG21, 132 | author = {Andres Garcia{-}Silva and 133 | Jose Manuel Gomez{-}Perez}, 134 | editor = {Djoerd Hiemstra and 135 | Marie{-}Francine Moens and 136 | Josiane Mothe and 137 | Raffaele Perego and 138 | Martin Potthast and 139 | Fabrizio Sebastiani}, 140 | title = {Classifying Scientific Publications with {BERT} - Is Self-attention 141 | a Feature Selection Method?}, 142 | booktitle = {Advances in Information Retrieval - 43rd European Conference on {IR} 143 | Research, {ECIR} 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, 144 | Part {I}}, 145 | series = {Lecture Notes in Computer Science}, 146 | volume = {12656}, 147 | pages = {161--175}, 148 | publisher = {Springer}, 149 | year = {2021}, 150 | url = {https://doi.org/10.1007/978-3-030-72113-8\_11}, 151 | doi = {10.1007/978-3-030-72113-8\_11}, 152 | timestamp = {Wed, 07 Apr 2021 16:01:39 +0200}, 153 | biburl = {https://dblp.org/rec/conf/ecir/Garcia-SilvaG21.bib}, 154 | bibsource = {dblp computer science bibliography, https://dblp.org} 155 | } 156 | ``` 157 | -------------------------------------------------------------------------------- /images/Figure 2.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Figure 2.PNG -------------------------------------------------------------------------------- /images/Figure 3.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Figure 3.PNG -------------------------------------------------------------------------------- /images/Table 1.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Table 1.PNG -------------------------------------------------------------------------------- /images/Table 2.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Table 2.PNG -------------------------------------------------------------------------------- /images/Table 3.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Table 3.PNG -------------------------------------------------------------------------------- /images/Table 4.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Table 4.PNG -------------------------------------------------------------------------------- /images/Table 5.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Table 5.PNG -------------------------------------------------------------------------------- /images/Table 6.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/Table 6.PNG -------------------------------------------------------------------------------- /images/finetuned-bert-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/finetuned-bert-2.png -------------------------------------------------------------------------------- /images/finetuned-bert-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/finetuned-bert-3.png -------------------------------------------------------------------------------- /images/finetuned-biobert-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/finetuned-biobert-2.png -------------------------------------------------------------------------------- /images/finetuned-biobert-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/finetuned-biobert-3.png -------------------------------------------------------------------------------- /images/finetuned-scibert-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/finetuned-scibert-2.png -------------------------------------------------------------------------------- /images/finetuned-scibert-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expertailab/Is-BERT-self-attention-a-feature-selection-method/c4a879f409f1ad21fd5f15fdfc74ba4bd371f533/images/finetuned-scibert-3.png -------------------------------------------------------------------------------- /notebooks/BertModelsAttendedWordsOverAverage.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "BertModelsAttendedWordsOverAverage.ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [], 9 | "toc_visible": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "accelerator": "GPU", 16 | "widgets": { 17 | "application/vnd.jupyter.widget-state+json": { 18 | "9cee813346f44f618f039e1fe12a2476": { 19 | "model_module": "@jupyter-widgets/controls", 20 | "model_name": "HBoxModel", 21 | "state": { 22 | "_view_name": "HBoxView", 23 | "_dom_classes": [], 24 | "_model_name": "HBoxModel", 25 | "_view_module": "@jupyter-widgets/controls", 26 | "_model_module_version": "1.5.0", 27 | "_view_count": null, 28 | "_view_module_version": "1.5.0", 29 | "box_style": "", 30 | "layout": "IPY_MODEL_88fb73ff05bc4dc68f765177b05c2fd5", 31 | "_model_module": "@jupyter-widgets/controls", 32 | "children": [ 33 | "IPY_MODEL_679cba53a25e4595baff262eab38cefb", 34 | "IPY_MODEL_724df2c92d364b10866fc036095ed2ed" 35 | ] 36 | } 37 | }, 38 | "88fb73ff05bc4dc68f765177b05c2fd5": { 39 | "model_module": "@jupyter-widgets/base", 40 | "model_name": "LayoutModel", 41 | "state": { 42 | "_view_name": "LayoutView", 43 | "grid_template_rows": null, 44 | "right": null, 45 | "justify_content": null, 46 | "_view_module": "@jupyter-widgets/base", 47 | "overflow": null, 48 | "_model_module_version": "1.2.0", 49 | "_view_count": null, 50 | "flex_flow": null, 51 | "width": null, 52 | "min_width": null, 53 | "border": null, 54 | "align_items": null, 55 | "bottom": null, 56 | "_model_module": "@jupyter-widgets/base", 57 | "top": null, 58 | "grid_column": null, 59 | "overflow_y": null, 60 | "overflow_x": null, 61 | "grid_auto_flow": null, 62 | "grid_area": null, 63 | "grid_template_columns": null, 64 | "flex": null, 65 | "_model_name": "LayoutModel", 66 | "justify_items": null, 67 | "grid_row": null, 68 | "max_height": null, 69 | "align_content": null, 70 | "visibility": null, 71 | "align_self": null, 72 | "height": null, 73 | "min_height": null, 74 | "padding": null, 75 | "grid_auto_rows": null, 76 | "grid_gap": null, 77 | "max_width": null, 78 | "order": null, 79 | "_view_module_version": "1.2.0", 80 | "grid_template_areas": null, 81 | "object_position": null, 82 | "object_fit": null, 83 | "grid_auto_columns": null, 84 | "margin": null, 85 | "display": null, 86 | "left": null 87 | } 88 | }, 89 | "679cba53a25e4595baff262eab38cefb": { 90 | "model_module": "@jupyter-widgets/controls", 91 | "model_name": "FloatProgressModel", 92 | "state": { 93 | "_view_name": "ProgressView", 94 | "style": "IPY_MODEL_06dea0c04f3a4b7caafb42c83faf1776", 95 | "_dom_classes": [], 96 | "description": "Downloading: 100%", 97 | "_model_name": "FloatProgressModel", 98 | "bar_style": "success", 99 | "max": 385, 100 | "_view_module": "@jupyter-widgets/controls", 101 | "_model_module_version": "1.5.0", 102 | "value": 385, 103 | "_view_count": null, 104 | "_view_module_version": "1.5.0", 105 | "orientation": "horizontal", 106 | "min": 0, 107 | "description_tooltip": null, 108 | "_model_module": "@jupyter-widgets/controls", 109 | "layout": "IPY_MODEL_6c38e2aa21d240a6b719deee927712d1" 110 | } 111 | }, 112 | "724df2c92d364b10866fc036095ed2ed": { 113 | "model_module": "@jupyter-widgets/controls", 114 | "model_name": "HTMLModel", 115 | "state": { 116 | "_view_name": "HTMLView", 117 | "style": "IPY_MODEL_6740ab057ef543cda7a9b26ee093215d", 118 | "_dom_classes": [], 119 | "description": "", 120 | "_model_name": "HTMLModel", 121 | "placeholder": "​", 122 | "_view_module": "@jupyter-widgets/controls", 123 | "_model_module_version": "1.5.0", 124 | "value": " 385/385 [00:00<00:00, 1.02kB/s]", 125 | "_view_count": null, 126 | "_view_module_version": "1.5.0", 127 | "description_tooltip": null, 128 | "_model_module": "@jupyter-widgets/controls", 129 | "layout": "IPY_MODEL_bcd113ab104e485cb47f7e4bb1dc27d7" 130 | } 131 | }, 132 | "06dea0c04f3a4b7caafb42c83faf1776": { 133 | "model_module": "@jupyter-widgets/controls", 134 | "model_name": "ProgressStyleModel", 135 | "state": { 136 | "_view_name": "StyleView", 137 | "_model_name": "ProgressStyleModel", 138 | "description_width": "initial", 139 | "_view_module": "@jupyter-widgets/base", 140 | "_model_module_version": "1.5.0", 141 | "_view_count": null, 142 | "_view_module_version": "1.2.0", 143 | "bar_color": null, 144 | "_model_module": "@jupyter-widgets/controls" 145 | } 146 | }, 147 | "6c38e2aa21d240a6b719deee927712d1": { 148 | "model_module": "@jupyter-widgets/base", 149 | "model_name": "LayoutModel", 150 | "state": { 151 | "_view_name": "LayoutView", 152 | "grid_template_rows": null, 153 | "right": null, 154 | "justify_content": null, 155 | "_view_module": "@jupyter-widgets/base", 156 | "overflow": null, 157 | "_model_module_version": "1.2.0", 158 | "_view_count": null, 159 | "flex_flow": null, 160 | "width": null, 161 | "min_width": null, 162 | "border": null, 163 | "align_items": null, 164 | "bottom": null, 165 | "_model_module": "@jupyter-widgets/base", 166 | "top": null, 167 | "grid_column": null, 168 | "overflow_y": null, 169 | "overflow_x": null, 170 | "grid_auto_flow": null, 171 | "grid_area": null, 172 | "grid_template_columns": null, 173 | "flex": null, 174 | "_model_name": "LayoutModel", 175 | "justify_items": null, 176 | "grid_row": null, 177 | "max_height": null, 178 | "align_content": null, 179 | "visibility": null, 180 | "align_self": null, 181 | "height": null, 182 | "min_height": null, 183 | "padding": null, 184 | "grid_auto_rows": null, 185 | "grid_gap": null, 186 | "max_width": null, 187 | "order": null, 188 | "_view_module_version": "1.2.0", 189 | "grid_template_areas": null, 190 | "object_position": null, 191 | "object_fit": null, 192 | "grid_auto_columns": null, 193 | "margin": null, 194 | "display": null, 195 | "left": null 196 | } 197 | }, 198 | "6740ab057ef543cda7a9b26ee093215d": { 199 | "model_module": "@jupyter-widgets/controls", 200 | "model_name": "DescriptionStyleModel", 201 | "state": { 202 | "_view_name": "StyleView", 203 | "_model_name": "DescriptionStyleModel", 204 | "description_width": "", 205 | "_view_module": "@jupyter-widgets/base", 206 | "_model_module_version": "1.5.0", 207 | "_view_count": null, 208 | "_view_module_version": "1.2.0", 209 | "_model_module": "@jupyter-widgets/controls" 210 | } 211 | }, 212 | "bcd113ab104e485cb47f7e4bb1dc27d7": { 213 | "model_module": "@jupyter-widgets/base", 214 | "model_name": "LayoutModel", 215 | "state": { 216 | "_view_name": "LayoutView", 217 | "grid_template_rows": null, 218 | "right": null, 219 | "justify_content": null, 220 | "_view_module": "@jupyter-widgets/base", 221 | "overflow": null, 222 | "_model_module_version": "1.2.0", 223 | "_view_count": null, 224 | "flex_flow": null, 225 | "width": null, 226 | "min_width": null, 227 | "border": null, 228 | "align_items": null, 229 | "bottom": null, 230 | "_model_module": "@jupyter-widgets/base", 231 | "top": null, 232 | "grid_column": null, 233 | "overflow_y": null, 234 | "overflow_x": null, 235 | "grid_auto_flow": null, 236 | "grid_area": null, 237 | "grid_template_columns": null, 238 | "flex": null, 239 | "_model_name": "LayoutModel", 240 | "justify_items": null, 241 | "grid_row": null, 242 | "max_height": null, 243 | "align_content": null, 244 | "visibility": null, 245 | "align_self": null, 246 | "height": null, 247 | "min_height": null, 248 | "padding": null, 249 | "grid_auto_rows": null, 250 | "grid_gap": null, 251 | "max_width": null, 252 | "order": null, 253 | "_view_module_version": "1.2.0", 254 | "grid_template_areas": null, 255 | "object_position": null, 256 | "object_fit": null, 257 | "grid_auto_columns": null, 258 | "margin": null, 259 | "display": null, 260 | "left": null 261 | } 262 | }, 263 | "0394282f509e4c909480e75179a9a897": { 264 | "model_module": "@jupyter-widgets/controls", 265 | "model_name": "HBoxModel", 266 | "state": { 267 | "_view_name": "HBoxView", 268 | "_dom_classes": [], 269 | "_model_name": "HBoxModel", 270 | "_view_module": "@jupyter-widgets/controls", 271 | "_model_module_version": "1.5.0", 272 | "_view_count": null, 273 | "_view_module_version": "1.5.0", 274 | "box_style": "", 275 | "layout": "IPY_MODEL_a95baa8425de4236af11ea46ba856516", 276 | "_model_module": "@jupyter-widgets/controls", 277 | "children": [ 278 | "IPY_MODEL_bc46fa4e234540b58efeef2a3e2508be", 279 | "IPY_MODEL_b22bf613f9d94cc0838645e2990b3af7" 280 | ] 281 | } 282 | }, 283 | "a95baa8425de4236af11ea46ba856516": { 284 | "model_module": "@jupyter-widgets/base", 285 | "model_name": "LayoutModel", 286 | "state": { 287 | "_view_name": "LayoutView", 288 | "grid_template_rows": null, 289 | "right": null, 290 | "justify_content": null, 291 | "_view_module": "@jupyter-widgets/base", 292 | "overflow": null, 293 | "_model_module_version": "1.2.0", 294 | "_view_count": null, 295 | "flex_flow": null, 296 | "width": null, 297 | "min_width": null, 298 | "border": null, 299 | "align_items": null, 300 | "bottom": null, 301 | "_model_module": "@jupyter-widgets/base", 302 | "top": null, 303 | "grid_column": null, 304 | "overflow_y": null, 305 | "overflow_x": null, 306 | "grid_auto_flow": null, 307 | "grid_area": null, 308 | "grid_template_columns": null, 309 | "flex": null, 310 | "_model_name": "LayoutModel", 311 | "justify_items": null, 312 | "grid_row": null, 313 | "max_height": null, 314 | "align_content": null, 315 | "visibility": null, 316 | "align_self": null, 317 | "height": null, 318 | "min_height": null, 319 | "padding": null, 320 | "grid_auto_rows": null, 321 | "grid_gap": null, 322 | "max_width": null, 323 | "order": null, 324 | "_view_module_version": "1.2.0", 325 | "grid_template_areas": null, 326 | "object_position": null, 327 | "object_fit": null, 328 | "grid_auto_columns": null, 329 | "margin": null, 330 | "display": null, 331 | "left": null 332 | } 333 | }, 334 | "bc46fa4e234540b58efeef2a3e2508be": { 335 | "model_module": "@jupyter-widgets/controls", 336 | "model_name": "FloatProgressModel", 337 | "state": { 338 | "_view_name": "ProgressView", 339 | "style": "IPY_MODEL_160a19ce782e4e70a0eb77002b5af7c2", 340 | "_dom_classes": [], 341 | "description": "Downloading: 100%", 342 | "_model_name": "FloatProgressModel", 343 | "bar_style": "success", 344 | "max": 227845, 345 | "_view_module": "@jupyter-widgets/controls", 346 | "_model_module_version": "1.5.0", 347 | "value": 227845, 348 | "_view_count": null, 349 | "_view_module_version": "1.5.0", 350 | "orientation": "horizontal", 351 | "min": 0, 352 | "description_tooltip": null, 353 | "_model_module": "@jupyter-widgets/controls", 354 | "layout": "IPY_MODEL_6327d32add694217917adf098b933f82" 355 | } 356 | }, 357 | "b22bf613f9d94cc0838645e2990b3af7": { 358 | "model_module": "@jupyter-widgets/controls", 359 | "model_name": "HTMLModel", 360 | "state": { 361 | "_view_name": "HTMLView", 362 | "style": "IPY_MODEL_5eec61c13e794fb4bd451b19db4b5445", 363 | "_dom_classes": [], 364 | "description": "", 365 | "_model_name": "HTMLModel", 366 | "placeholder": "​", 367 | "_view_module": "@jupyter-widgets/controls", 368 | "_model_module_version": "1.5.0", 369 | "value": " 228k/228k [00:03<00:00, 75.9kB/s]", 370 | "_view_count": null, 371 | "_view_module_version": "1.5.0", 372 | "description_tooltip": null, 373 | "_model_module": "@jupyter-widgets/controls", 374 | "layout": "IPY_MODEL_c6d495768c644015b70af1a63ca68e59" 375 | } 376 | }, 377 | "160a19ce782e4e70a0eb77002b5af7c2": { 378 | "model_module": "@jupyter-widgets/controls", 379 | "model_name": "ProgressStyleModel", 380 | "state": { 381 | "_view_name": "StyleView", 382 | "_model_name": "ProgressStyleModel", 383 | "description_width": "initial", 384 | "_view_module": "@jupyter-widgets/base", 385 | "_model_module_version": "1.5.0", 386 | "_view_count": null, 387 | "_view_module_version": "1.2.0", 388 | "bar_color": null, 389 | "_model_module": "@jupyter-widgets/controls" 390 | } 391 | }, 392 | "6327d32add694217917adf098b933f82": { 393 | "model_module": "@jupyter-widgets/base", 394 | "model_name": "LayoutModel", 395 | "state": { 396 | "_view_name": "LayoutView", 397 | "grid_template_rows": null, 398 | "right": null, 399 | "justify_content": null, 400 | "_view_module": "@jupyter-widgets/base", 401 | "overflow": null, 402 | "_model_module_version": "1.2.0", 403 | "_view_count": null, 404 | "flex_flow": null, 405 | "width": null, 406 | "min_width": null, 407 | "border": null, 408 | "align_items": null, 409 | "bottom": null, 410 | "_model_module": "@jupyter-widgets/base", 411 | "top": null, 412 | "grid_column": null, 413 | "overflow_y": null, 414 | "overflow_x": null, 415 | "grid_auto_flow": null, 416 | "grid_area": null, 417 | "grid_template_columns": null, 418 | "flex": null, 419 | "_model_name": "LayoutModel", 420 | "justify_items": null, 421 | "grid_row": null, 422 | "max_height": null, 423 | "align_content": null, 424 | "visibility": null, 425 | "align_self": null, 426 | "height": null, 427 | "min_height": null, 428 | "padding": null, 429 | "grid_auto_rows": null, 430 | "grid_gap": null, 431 | "max_width": null, 432 | "order": null, 433 | "_view_module_version": "1.2.0", 434 | "grid_template_areas": null, 435 | "object_position": null, 436 | "object_fit": null, 437 | "grid_auto_columns": null, 438 | "margin": null, 439 | "display": null, 440 | "left": null 441 | } 442 | }, 443 | "5eec61c13e794fb4bd451b19db4b5445": { 444 | "model_module": "@jupyter-widgets/controls", 445 | "model_name": "DescriptionStyleModel", 446 | "state": { 447 | "_view_name": "StyleView", 448 | "_model_name": "DescriptionStyleModel", 449 | "description_width": "", 450 | "_view_module": "@jupyter-widgets/base", 451 | "_model_module_version": "1.5.0", 452 | "_view_count": null, 453 | "_view_module_version": "1.2.0", 454 | "_model_module": "@jupyter-widgets/controls" 455 | } 456 | }, 457 | "c6d495768c644015b70af1a63ca68e59": { 458 | "model_module": "@jupyter-widgets/base", 459 | "model_name": "LayoutModel", 460 | "state": { 461 | "_view_name": "LayoutView", 462 | "grid_template_rows": null, 463 | "right": null, 464 | "justify_content": null, 465 | "_view_module": "@jupyter-widgets/base", 466 | "overflow": null, 467 | "_model_module_version": "1.2.0", 468 | "_view_count": null, 469 | "flex_flow": null, 470 | "width": null, 471 | "min_width": null, 472 | "border": null, 473 | "align_items": null, 474 | "bottom": null, 475 | "_model_module": "@jupyter-widgets/base", 476 | "top": null, 477 | "grid_column": null, 478 | "overflow_y": null, 479 | "overflow_x": null, 480 | "grid_auto_flow": null, 481 | "grid_area": null, 482 | "grid_template_columns": null, 483 | "flex": null, 484 | "_model_name": "LayoutModel", 485 | "justify_items": null, 486 | "grid_row": null, 487 | "max_height": null, 488 | "align_content": null, 489 | "visibility": null, 490 | "align_self": null, 491 | "height": null, 492 | "min_height": null, 493 | "padding": null, 494 | "grid_auto_rows": null, 495 | "grid_gap": null, 496 | "max_width": null, 497 | "order": null, 498 | "_view_module_version": "1.2.0", 499 | "grid_template_areas": null, 500 | "object_position": null, 501 | "object_fit": null, 502 | "grid_auto_columns": null, 503 | "margin": null, 504 | "display": null, 505 | "left": null 506 | } 507 | } 508 | } 509 | } 510 | }, 511 | "cells": [ 512 | { 513 | "cell_type": "markdown", 514 | "metadata": { 515 | "id": "MQ1W3dmWHuXB" 516 | }, 517 | "source": [ 518 | "## Download dataset fine-tuned models and libraries " 519 | ] 520 | }, 521 | { 522 | "cell_type": "markdown", 523 | "metadata": { 524 | "id": "6_zGQHmbISLC" 525 | }, 526 | "source": [ 527 | "### Dataset" 528 | ] 529 | }, 530 | { 531 | "cell_type": "code", 532 | "metadata": { 533 | "id": "c_4Ddloyk8DO", 534 | "outputId": "46389086-5367-4382-c091-f5343c488b9e", 535 | "colab": { 536 | "base_uri": "https://localhost:8080/" 537 | } 538 | }, 539 | "source": [ 540 | "!gdown --id 1XOtSCqfzMC3_XWY8Ylw_josDAJOjMwOF" 541 | ], 542 | "execution_count": 2, 543 | "outputs": [ 544 | { 545 | "output_type": "stream", 546 | "text": [ 547 | "Downloading...\n", 548 | "From: https://drive.google.com/uc?id=1XOtSCqfzMC3_XWY8Ylw_josDAJOjMwOF\n", 549 | "To: /content/articles_scigraph_2011.json\n", 550 | "274MB [00:04, 63.3MB/s]\n" 551 | ], 552 | "name": "stdout" 553 | } 554 | ] 555 | }, 556 | { 557 | "cell_type": "markdown", 558 | "metadata": { 559 | "id": "hP1x1fMnVIgq" 560 | }, 561 | "source": [ 562 | "Choose only one model below:" 563 | ] 564 | }, 565 | { 566 | "cell_type": "markdown", 567 | "metadata": { 568 | "id": "K9oqXrkjH1Le" 569 | }, 570 | "source": [ 571 | "### Option 1: Fine-tuned SciBERT on articles clasification" 572 | ] 573 | }, 574 | { 575 | "cell_type": "code", 576 | "metadata": { 577 | "id": "WqNZtguO99Us", 578 | "outputId": "b8c2de00-99a6-4127-f367-83db6287e4f9", 579 | "colab": { 580 | "base_uri": "https://localhost:8080/" 581 | } 582 | }, 583 | "source": [ 584 | "!gdown --id 1uKFfoh95K7lRXEwdMt7F32ivohwAENzT" 585 | ], 586 | "execution_count": 3, 587 | "outputs": [ 588 | { 589 | "output_type": "stream", 590 | "text": [ 591 | "Downloading...\n", 592 | "From: https://drive.google.com/uc?id=1uKFfoh95K7lRXEwdMt7F32ivohwAENzT\n", 593 | "To: /content/scibert10v2_scigraph0.pt\n", 594 | "440MB [00:07, 57.4MB/s]\n" 595 | ], 596 | "name": "stdout" 597 | } 598 | ] 599 | }, 600 | { 601 | "cell_type": "markdown", 602 | "metadata": { 603 | "id": "oa7sawoCQRj1" 604 | }, 605 | "source": [ 606 | "### Option 2: Fine-tuned BERT on articles clasification" 607 | ] 608 | }, 609 | { 610 | "cell_type": "code", 611 | "metadata": { 612 | "id": "nqkkvurIQXWj" 613 | }, 614 | "source": [ 615 | "!gdown --id 19pNPMyguDUPOlza-B4J-NqOOdU-ZqGwl" 616 | ], 617 | "execution_count": null, 618 | "outputs": [] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": { 623 | "id": "75AFWS0JQbWK" 624 | }, 625 | "source": [ 626 | "### Option 3: Fine-tuned BioBERT 1.1 on articles clasification" 627 | ] 628 | }, 629 | { 630 | "cell_type": "code", 631 | "metadata": { 632 | "id": "f5TUOSD3QbWL" 633 | }, 634 | "source": [ 635 | "!gdown --id 1waZOhC4LADr6rQx0cf3LRA4UZSj6u2uu" 636 | ], 637 | "execution_count": null, 638 | "outputs": [] 639 | }, 640 | { 641 | "cell_type": "markdown", 642 | "metadata": { 643 | "id": "ZzVaPSqDM8EP" 644 | }, 645 | "source": [ 646 | "Download library used to fine-tune the LM on mutilabel classification" 647 | ] 648 | }, 649 | { 650 | "cell_type": "code", 651 | "metadata": { 652 | "id": "qIcu-vgxly0h", 653 | "outputId": "703d269e-79cc-4d6d-8aa1-a1c152bdb2ed", 654 | "colab": { 655 | "base_uri": "https://localhost:8080/" 656 | } 657 | }, 658 | "source": [ 659 | "!gdown --id 1LpufGkbVYTGxgAHr2TyqQimqVRte420U" 660 | ], 661 | "execution_count": 4, 662 | "outputs": [ 663 | { 664 | "output_type": "stream", 665 | "text": [ 666 | "Downloading...\n", 667 | "From: https://drive.google.com/uc?id=1LpufGkbVYTGxgAHr2TyqQimqVRte420U\n", 668 | "To: /content/BertModeling.py\n", 669 | "\r 0% 0.00/14.4k [00:00=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (4.41.1)\n", 745 | "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (3.0.12)\n", 746 | "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (1.15.0)\n", 747 | "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (7.1.2)\n", 748 | "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (0.17.0)\n", 749 | "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (3.0.4)\n", 750 | "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (1.24.3)\n", 751 | "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2020.6.20)\n", 752 | "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2.10)\n", 753 | "Collecting botocore<1.20.0,>=1.19.7\n", 754 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d1/58/64883046f9c98d9f94cc81174d0b83e0be35b8f6c252e255b709b9024ef1/botocore-1.19.7-py2.py3-none-any.whl (6.7MB)\n", 755 | "\u001b[K |████████████████████████████████| 6.7MB 47.4MB/s \n", 756 | "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", 757 | " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", 758 | "Collecting s3transfer<0.4.0,>=0.3.0\n", 759 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)\n", 760 | "\u001b[K |████████████████████████████████| 71kB 10.1MB/s \n", 761 | "\u001b[?25hRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.20.0,>=1.19.7->boto3->transformers==2.8.0) (2.8.1)\n", 762 | "Building wheels for collected packages: sacremoses\n", 763 | " Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", 764 | " Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=4564a4b321cafe1b958035bcafbc235ed3cc8e9519d7d39779526e3792642b5a\n", 765 | " Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n", 766 | "Successfully built sacremoses\n", 767 | "\u001b[31mERROR: botocore 1.19.7 has requirement urllib3<1.26,>=1.25.4; python_version != \"3.4\", but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", 768 | "Installing collected packages: tokenizers, sacremoses, sentencepiece, jmespath, botocore, s3transfer, boto3, transformers\n", 769 | "Successfully installed boto3-1.16.7 botocore-1.19.7 jmespath-0.10.0 s3transfer-0.3.3 sacremoses-0.0.43 sentencepiece-0.1.94 tokenizers-0.5.2 transformers-2.8.0\n" 770 | ], 771 | "name": "stdout" 772 | } 773 | ] 774 | }, 775 | { 776 | "cell_type": "code", 777 | "metadata": { 778 | "id": "mF0reHdby-YP" 779 | }, 780 | "source": [ 781 | "import sys\n", 782 | "\n", 783 | "from BertModeling import BertForMultiLabelSequenceClassification\n", 784 | "from BertModeling import (BertForMultiLabelSequenceClassification, BioBertForMultiLabelSequenceClassification, \n", 785 | " BioBertForMultiLabelSequenceClassification2, XLNetForMultiLabelSequenceClassification,\n", 786 | " GPT2MultiLabelClassification)\n", 787 | "from transformers import (WEIGHTS_NAME, BertConfig, BertForSequenceClassification, BertTokenizer, AutoTokenizer, AutoModelWithLMHead,AutoConfig,\n", 788 | " XLMConfig, XLMForSequenceClassification, XLMTokenizer, \n", 789 | " XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer,\n", 790 | " RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer)\n", 791 | "#from bertviz.transformers_neuron_view import BertModel, BertTokenizer, BertForPreTraining, BertForSequenceClassification\n", 792 | "import matplotlib.pyplot as plt\n", 793 | "import matplotlib\n", 794 | "import numpy as np\n", 795 | "from transformers import BertTokenizer\n", 796 | "import torch\n", 797 | "from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,\n", 798 | " TensorDataset)\n", 799 | "import csv\n", 800 | "import json\n", 801 | "import pprint" 802 | ], 803 | "execution_count": 7, 804 | "outputs": [] 805 | }, 806 | { 807 | "cell_type": "code", 808 | "metadata": { 809 | "id": "0klcUIYtL3PR", 810 | "outputId": "ba999950-c8af-4cf4-fa26-c6985ee51d70", 811 | "colab": { 812 | "base_uri": "https://localhost:8080/", 813 | "height": 35 814 | } 815 | }, 816 | "source": [ 817 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 818 | "n_gpu = torch.cuda.device_count()\n", 819 | "torch.cuda.get_device_name(0)" 820 | ], 821 | "execution_count": 8, 822 | "outputs": [ 823 | { 824 | "output_type": "execute_result", 825 | "data": { 826 | "application/vnd.google.colaboratory.intrinsic+json": { 827 | "type": "string" 828 | }, 829 | "text/plain": [ 830 | "'Tesla P100-PCIE-16GB'" 831 | ] 832 | }, 833 | "metadata": { 834 | "tags": [] 835 | }, 836 | "execution_count": 8 837 | } 838 | ] 839 | }, 840 | { 841 | "cell_type": "markdown", 842 | "metadata": { 843 | "id": "evk4z9YvMbkI" 844 | }, 845 | "source": [ 846 | "## Load the fine-tuned language models" 847 | ] 848 | }, 849 | { 850 | "cell_type": "markdown", 851 | "metadata": { 852 | "id": "gK0EI6iEMh3K" 853 | }, 854 | "source": [ 855 | "Choose only one of the models by running the corresponding cell" 856 | ] 857 | }, 858 | { 859 | "cell_type": "markdown", 860 | "metadata": { 861 | "id": "s7DpF6rxM2d9" 862 | }, 863 | "source": [ 864 | "#### Option 1: Fine-tuned SciBERT" 865 | ] 866 | }, 867 | { 868 | "cell_type": "code", 869 | "metadata": { 870 | "id": "h6X4jQgKL8ZJ", 871 | "outputId": "9e9d14f3-4a90-4897-e30b-f6c01dbbcbd2", 872 | "colab": { 873 | "base_uri": "https://localhost:8080/", 874 | "height": 1000, 875 | "referenced_widgets": [ 876 | "9cee813346f44f618f039e1fe12a2476", 877 | "88fb73ff05bc4dc68f765177b05c2fd5", 878 | "679cba53a25e4595baff262eab38cefb", 879 | "724df2c92d364b10866fc036095ed2ed", 880 | "06dea0c04f3a4b7caafb42c83faf1776", 881 | "6c38e2aa21d240a6b719deee927712d1", 882 | "6740ab057ef543cda7a9b26ee093215d", 883 | "bcd113ab104e485cb47f7e4bb1dc27d7", 884 | "0394282f509e4c909480e75179a9a897", 885 | "a95baa8425de4236af11ea46ba856516", 886 | "bc46fa4e234540b58efeef2a3e2508be", 887 | "b22bf613f9d94cc0838645e2990b3af7", 888 | "160a19ce782e4e70a0eb77002b5af7c2", 889 | "6327d32add694217917adf098b933f82", 890 | "5eec61c13e794fb4bd451b19db4b5445", 891 | "c6d495768c644015b70af1a63ca68e59" 892 | ] 893 | } 894 | }, 895 | "source": [ 896 | "#scibert \n", 897 | "pretrainedModelpath = 'allenai/scibert_scivocab_uncased' \n", 898 | "modelpath = './scibert10v2_scigraph0.pt' #fine-tuned model\n", 899 | "\n", 900 | "config = AutoConfig.from_pretrained('allenai/scibert_scivocab_uncased', output_hidden_states=True, output_attentions=True, num_labels=22)#\n", 901 | "model = BertForMultiLabelSequenceClassification(config=config)\n", 902 | "model.load_state_dict(torch.load(modelpath))\n", 903 | "\n", 904 | "bert_tokenizer = BertTokenizer.from_pretrained(pretrainedModelpath, do_lower_case=True, return_token_type_ids=True)\n", 905 | "model.eval()" 906 | ], 907 | "execution_count": 9, 908 | "outputs": [ 909 | { 910 | "output_type": "display_data", 911 | "data": { 912 | "application/vnd.jupyter.widget-view+json": { 913 | "model_id": "9cee813346f44f618f039e1fe12a2476", 914 | "version_minor": 0, 915 | "version_major": 2 916 | }, 917 | "text/plain": [ 918 | "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=385.0, style=ProgressStyle(description_…" 919 | ] 920 | }, 921 | "metadata": { 922 | "tags": [] 923 | } 924 | }, 925 | { 926 | "output_type": "stream", 927 | "text": [ 928 | "\n" 929 | ], 930 | "name": "stdout" 931 | }, 932 | { 933 | "output_type": "display_data", 934 | "data": { 935 | "application/vnd.jupyter.widget-view+json": { 936 | "model_id": "0394282f509e4c909480e75179a9a897", 937 | "version_minor": 0, 938 | "version_major": 2 939 | }, 940 | "text/plain": [ 941 | "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=227845.0, style=ProgressStyle(descripti…" 942 | ] 943 | }, 944 | "metadata": { 945 | "tags": [] 946 | } 947 | }, 948 | { 949 | "output_type": "stream", 950 | "text": [ 951 | "\n" 952 | ], 953 | "name": "stdout" 954 | }, 955 | { 956 | "output_type": "execute_result", 957 | "data": { 958 | "text/plain": [ 959 | "BertForMultiLabelSequenceClassification(\n", 960 | " (bert): BertModel(\n", 961 | " (embeddings): BertEmbeddings(\n", 962 | " (word_embeddings): Embedding(31090, 768, padding_idx=0)\n", 963 | " (position_embeddings): Embedding(512, 768)\n", 964 | " (token_type_embeddings): Embedding(2, 768)\n", 965 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 966 | " (dropout): Dropout(p=0.1, inplace=False)\n", 967 | " )\n", 968 | " (encoder): BertEncoder(\n", 969 | " (layer): ModuleList(\n", 970 | " (0): BertLayer(\n", 971 | " (attention): BertAttention(\n", 972 | " (self): BertSelfAttention(\n", 973 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 974 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 975 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 976 | " (dropout): Dropout(p=0.1, inplace=False)\n", 977 | " )\n", 978 | " (output): BertSelfOutput(\n", 979 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 980 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 981 | " (dropout): Dropout(p=0.1, inplace=False)\n", 982 | " )\n", 983 | " )\n", 984 | " (intermediate): BertIntermediate(\n", 985 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 986 | " )\n", 987 | " (output): BertOutput(\n", 988 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 989 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 990 | " (dropout): Dropout(p=0.1, inplace=False)\n", 991 | " )\n", 992 | " )\n", 993 | " (1): BertLayer(\n", 994 | " (attention): BertAttention(\n", 995 | " (self): BertSelfAttention(\n", 996 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 997 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 998 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 999 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1000 | " )\n", 1001 | " (output): BertSelfOutput(\n", 1002 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1003 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1004 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1005 | " )\n", 1006 | " )\n", 1007 | " (intermediate): BertIntermediate(\n", 1008 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1009 | " )\n", 1010 | " (output): BertOutput(\n", 1011 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1012 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1013 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1014 | " )\n", 1015 | " )\n", 1016 | " (2): BertLayer(\n", 1017 | " (attention): BertAttention(\n", 1018 | " (self): BertSelfAttention(\n", 1019 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1020 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1021 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1022 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1023 | " )\n", 1024 | " (output): BertSelfOutput(\n", 1025 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1026 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1027 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1028 | " )\n", 1029 | " )\n", 1030 | " (intermediate): BertIntermediate(\n", 1031 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1032 | " )\n", 1033 | " (output): BertOutput(\n", 1034 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1035 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1036 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1037 | " )\n", 1038 | " )\n", 1039 | " (3): BertLayer(\n", 1040 | " (attention): BertAttention(\n", 1041 | " (self): BertSelfAttention(\n", 1042 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1043 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1044 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1045 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1046 | " )\n", 1047 | " (output): BertSelfOutput(\n", 1048 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1049 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1050 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1051 | " )\n", 1052 | " )\n", 1053 | " (intermediate): BertIntermediate(\n", 1054 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1055 | " )\n", 1056 | " (output): BertOutput(\n", 1057 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1058 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1059 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1060 | " )\n", 1061 | " )\n", 1062 | " (4): BertLayer(\n", 1063 | " (attention): BertAttention(\n", 1064 | " (self): BertSelfAttention(\n", 1065 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1066 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1067 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1068 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1069 | " )\n", 1070 | " (output): BertSelfOutput(\n", 1071 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1072 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1073 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1074 | " )\n", 1075 | " )\n", 1076 | " (intermediate): BertIntermediate(\n", 1077 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1078 | " )\n", 1079 | " (output): BertOutput(\n", 1080 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1081 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1082 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1083 | " )\n", 1084 | " )\n", 1085 | " (5): BertLayer(\n", 1086 | " (attention): BertAttention(\n", 1087 | " (self): BertSelfAttention(\n", 1088 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1089 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1090 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1091 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1092 | " )\n", 1093 | " (output): BertSelfOutput(\n", 1094 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1095 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1096 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1097 | " )\n", 1098 | " )\n", 1099 | " (intermediate): BertIntermediate(\n", 1100 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1101 | " )\n", 1102 | " (output): BertOutput(\n", 1103 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1104 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1105 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1106 | " )\n", 1107 | " )\n", 1108 | " (6): BertLayer(\n", 1109 | " (attention): BertAttention(\n", 1110 | " (self): BertSelfAttention(\n", 1111 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1112 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1113 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1114 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1115 | " )\n", 1116 | " (output): BertSelfOutput(\n", 1117 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1118 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1119 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1120 | " )\n", 1121 | " )\n", 1122 | " (intermediate): BertIntermediate(\n", 1123 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1124 | " )\n", 1125 | " (output): BertOutput(\n", 1126 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1127 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1128 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1129 | " )\n", 1130 | " )\n", 1131 | " (7): BertLayer(\n", 1132 | " (attention): BertAttention(\n", 1133 | " (self): BertSelfAttention(\n", 1134 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1135 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1136 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1137 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1138 | " )\n", 1139 | " (output): BertSelfOutput(\n", 1140 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1141 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1142 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1143 | " )\n", 1144 | " )\n", 1145 | " (intermediate): BertIntermediate(\n", 1146 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1147 | " )\n", 1148 | " (output): BertOutput(\n", 1149 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1150 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1151 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1152 | " )\n", 1153 | " )\n", 1154 | " (8): BertLayer(\n", 1155 | " (attention): BertAttention(\n", 1156 | " (self): BertSelfAttention(\n", 1157 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1158 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1159 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1160 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1161 | " )\n", 1162 | " (output): BertSelfOutput(\n", 1163 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1164 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1165 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1166 | " )\n", 1167 | " )\n", 1168 | " (intermediate): BertIntermediate(\n", 1169 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1170 | " )\n", 1171 | " (output): BertOutput(\n", 1172 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1173 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1174 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1175 | " )\n", 1176 | " )\n", 1177 | " (9): BertLayer(\n", 1178 | " (attention): BertAttention(\n", 1179 | " (self): BertSelfAttention(\n", 1180 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1181 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1182 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1183 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1184 | " )\n", 1185 | " (output): BertSelfOutput(\n", 1186 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1187 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1188 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1189 | " )\n", 1190 | " )\n", 1191 | " (intermediate): BertIntermediate(\n", 1192 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1193 | " )\n", 1194 | " (output): BertOutput(\n", 1195 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1196 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1197 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1198 | " )\n", 1199 | " )\n", 1200 | " (10): BertLayer(\n", 1201 | " (attention): BertAttention(\n", 1202 | " (self): BertSelfAttention(\n", 1203 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1204 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1205 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1206 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1207 | " )\n", 1208 | " (output): BertSelfOutput(\n", 1209 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1210 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1211 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1212 | " )\n", 1213 | " )\n", 1214 | " (intermediate): BertIntermediate(\n", 1215 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1216 | " )\n", 1217 | " (output): BertOutput(\n", 1218 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1219 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1220 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1221 | " )\n", 1222 | " )\n", 1223 | " (11): BertLayer(\n", 1224 | " (attention): BertAttention(\n", 1225 | " (self): BertSelfAttention(\n", 1226 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 1227 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 1228 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 1229 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1230 | " )\n", 1231 | " (output): BertSelfOutput(\n", 1232 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1233 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1234 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1235 | " )\n", 1236 | " )\n", 1237 | " (intermediate): BertIntermediate(\n", 1238 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 1239 | " )\n", 1240 | " (output): BertOutput(\n", 1241 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 1242 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 1243 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1244 | " )\n", 1245 | " )\n", 1246 | " )\n", 1247 | " )\n", 1248 | " (pooler): BertPooler(\n", 1249 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 1250 | " (activation): Tanh()\n", 1251 | " )\n", 1252 | " )\n", 1253 | " (dropout): Dropout(p=0.1, inplace=False)\n", 1254 | " (classifier): Linear(in_features=768, out_features=22, bias=True)\n", 1255 | ")" 1256 | ] 1257 | }, 1258 | "metadata": { 1259 | "tags": [] 1260 | }, 1261 | "execution_count": 9 1262 | } 1263 | ] 1264 | }, 1265 | { 1266 | "cell_type": "markdown", 1267 | "metadata": { 1268 | "id": "B4quwVpCMm0T" 1269 | }, 1270 | "source": [ 1271 | "### Option 2: Fine-tuned BERT" 1272 | ] 1273 | }, 1274 | { 1275 | "cell_type": "code", 1276 | "metadata": { 1277 | "id": "AJD4d1tvMCXh" 1278 | }, 1279 | "source": [ 1280 | "#Bert \n", 1281 | "modelpath = \"./bert_scigraph0.pt\" \n", 1282 | "\n", 1283 | "config = BertConfig(output_hidden_states=True, output_attentions=True, num_labels=22)\n", 1284 | "model = BertForMultiLabelSequenceClassification(config=config)\n", 1285 | "model.load_state_dict(torch.load(modelpath))\n", 1286 | "\n", 1287 | "bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, return_token_type_ids=True)\n", 1288 | "model.eval()" 1289 | ], 1290 | "execution_count": null, 1291 | "outputs": [] 1292 | }, 1293 | { 1294 | "cell_type": "markdown", 1295 | "metadata": { 1296 | "id": "pCssnvX8M0aW" 1297 | }, 1298 | "source": [ 1299 | "### Option 3: Fine-tuned BioBERT" 1300 | ] 1301 | }, 1302 | { 1303 | "cell_type": "code", 1304 | "metadata": { 1305 | "id": "mPCg78r4MCjw" 1306 | }, 1307 | "source": [ 1308 | "#BioBert \n", 1309 | "pretrainedModelpath = 'monologg/biobert_v1.1_pubmed'\n", 1310 | "modelpath = \"./biobert11v2_scigraph0.pt\" #fine-tuned model\n", 1311 | "\n", 1312 | "config = AutoConfig.from_pretrained(pretrainedModelpath+'config.json', output_hidden_states=True, output_attentions=True, num_labels=22)#\n", 1313 | "model = BioBertForMultiLabelSequenceClassification2(config=config)\n", 1314 | "model.load_state_dict(torch.load(modelpath))\n", 1315 | "model = model.bert\n", 1316 | "\n", 1317 | "bert_tokenizer = BertTokenizer.from_pretrained(pretrainedModelpath, do_lower_case=True, return_token_type_ids=True)\n", 1318 | "model.eval()\n" 1319 | ], 1320 | "execution_count": null, 1321 | "outputs": [] 1322 | }, 1323 | { 1324 | "cell_type": "markdown", 1325 | "metadata": { 1326 | "id": "Q0ZyaLnYUpcb" 1327 | }, 1328 | "source": [ 1329 | "## Word tokenizer out of subword tokenizers\n", 1330 | "\n" 1331 | ] 1332 | }, 1333 | { 1334 | "cell_type": "markdown", 1335 | "metadata": { 1336 | "id": "vJVVM-Yp3pqE" 1337 | }, 1338 | "source": [ 1339 | "### Extract word vocabulary from subwords\n", 1340 | "\n", 1341 | "The dataset is a subset of articles with a max numbero of 10 in each category\n" 1342 | ] 1343 | }, 1344 | { 1345 | "cell_type": "code", 1346 | "metadata": { 1347 | "id": "Y0sl_lHNVmqM", 1348 | "outputId": "078ab2a4-a73f-4e21-9d9c-b4b3f278251d", 1349 | "colab": { 1350 | "base_uri": "https://localhost:8080/" 1351 | } 1352 | }, 1353 | "source": [ 1354 | "import re\n", 1355 | "sc_text = []\n", 1356 | "bert_word_counts = {} # words to freq \n", 1357 | "bert_word_index = {} # words to index\n", 1358 | "bert_index_word = {} # index to words\n", 1359 | "category_articles={} # category to articles\n", 1360 | "article_categories={} # articles to categories\n", 1361 | "\n", 1362 | "# JSON FILE annotated text\n", 1363 | "with open('./articles_scigraph_2011.json') as json_file:\n", 1364 | " docs = json.load(json_file)\n", 1365 | "\n", 1366 | " docCounter=0\n", 1367 | " for j,doc in enumerate(docs):\n", 1368 | " skip=False\n", 1369 | " str = doc[\"title\"]+\". \" + doc[\"abstract\"] \n", 1370 | "\n", 1371 | " #Check max of 10 articles per category \n", 1372 | " for cat in doc[\"fieldcodes\"]:\n", 1373 | " if len(cat) == 2: \n", 1374 | " if cat in category_articles:\n", 1375 | " articles = category_articles[cat]\n", 1376 | " if (len(articles)<10): \n", 1377 | " articles.append(docCounter)\n", 1378 | " category_articles[cat] = articles \n", 1379 | " else:\n", 1380 | " skip=True\n", 1381 | " else:\n", 1382 | " category_articles[cat] = [docCounter] \n", 1383 | " if skip==True:\n", 1384 | " continue\n", 1385 | " \n", 1386 | " sc_text.append(str)\n", 1387 | " article_categories[docCounter] = [x for x in doc[\"fieldcodes\"] if len(x)==2]\n", 1388 | " docCounter+=1\n", 1389 | " \n", 1390 | " \n", 1391 | " #Extract word vocabulary\n", 1392 | " #print(str)\n", 1393 | " tokenized_sentence = bert_tokenizer.encode(str, add_special_tokens=False, truncation=True, max_length = 512 )\n", 1394 | " tokens = bert_tokenizer.convert_ids_to_tokens(tokenized_sentence) #this way the special tokens are added \n", 1395 | " subword=False\n", 1396 | " newword=False \n", 1397 | " for i,t in enumerate(tokens): \n", 1398 | " if t.startswith(\"##\") == False : \n", 1399 | " if (subword == True or newword == True) : \n", 1400 | " #add word to the dict\n", 1401 | " if word in bert_word_counts:\n", 1402 | " bert_word_counts[word]+=1\n", 1403 | " else:\n", 1404 | " bert_word_counts[word]=1 \n", 1405 | " subword = False \n", 1406 | " word = t\n", 1407 | " newword = True\n", 1408 | " else:\n", 1409 | " subword = True\n", 1410 | " word = word + t[2:] #remove ## \n", 1411 | " if j == 10000:\n", 1412 | " break \n", 1413 | " \n", 1414 | " \n", 1415 | "#sort words by frequency\n", 1416 | "sorted_voc ={k: v for k, v in sorted(bert_word_counts.items(), key=lambda item: item[1], reverse=True)}\n", 1417 | "bert_word_index = dict(zip(sorted_voc, list(range(1, len(sorted_voc) + 1))))\n", 1418 | "bert_index_word = {c: w for w, c in bert_word_index.items()}\n", 1419 | "\n", 1420 | "\n", 1421 | "for cat, articles in {k: v for k, v in sorted(category_articles.items(), key=lambda item: item[0])}.items() :\n", 1422 | " print(cat,\":\",len(articles))" 1423 | ], 1424 | "execution_count": 15, 1425 | "outputs": [ 1426 | { 1427 | "output_type": "stream", 1428 | "text": [ 1429 | "01 : 10\n", 1430 | "02 : 10\n", 1431 | "03 : 10\n", 1432 | "04 : 10\n", 1433 | "05 : 10\n", 1434 | "06 : 10\n", 1435 | "07 : 10\n", 1436 | "08 : 10\n", 1437 | "09 : 10\n", 1438 | "10 : 10\n", 1439 | "11 : 10\n", 1440 | "12 : 10\n", 1441 | "13 : 10\n", 1442 | "14 : 10\n", 1443 | "15 : 10\n", 1444 | "16 : 10\n", 1445 | "17 : 10\n", 1446 | "18 : 10\n", 1447 | "19 : 10\n", 1448 | "20 : 10\n", 1449 | "21 : 10\n", 1450 | "22 : 10\n" 1451 | ], 1452 | "name": "stdout" 1453 | } 1454 | ] 1455 | }, 1456 | { 1457 | "cell_type": "markdown", 1458 | "metadata": { 1459 | "id": "y_oh_MKM3wHf" 1460 | }, 1461 | "source": [ 1462 | "### Get word sequences from subword sequences" 1463 | ] 1464 | }, 1465 | { 1466 | "cell_type": "code", 1467 | "metadata": { 1468 | "id": "VaimbcBM1s5G", 1469 | "outputId": "d60bb8da-5549-487b-dee0-8f1c25fc25e2", 1470 | "colab": { 1471 | "base_uri": "https://localhost:8080/" 1472 | } 1473 | }, 1474 | "source": [ 1475 | "## use the wordindex to keep track of the sequence and of the start and end idx of each word.\n", 1476 | "word_seqs = [] #according to OUR word index of whole tokens built out of word pieces tokens\n", 1477 | "word_seqs_words = []\n", 1478 | "starts = [] \n", 1479 | "ends = []\n", 1480 | "\n", 1481 | "#BERT tokenizer representations\n", 1482 | "subword_seqs = [] #token ids\n", 1483 | "subword_seqs_tokens = [] #token strings\n", 1484 | "\n", 1485 | "for k,paper in enumerate(sc_text): \n", 1486 | " tokenized_sentence = bert_tokenizer.encode(paper, add_special_tokens=False, truncation=True, max_length = 512) \n", 1487 | " tokens = bert_tokenizer.convert_ids_to_tokens(tokenized_sentence) \n", 1488 | " subword_seqs.append(tokenized_sentence)\n", 1489 | " subword_seqs_tokens.append(tokens)\n", 1490 | " #print(subword_seqs)\n", 1491 | " subword=False\n", 1492 | " newword=False\n", 1493 | " word_seq = []\n", 1494 | " start = []\n", 1495 | " end=[]\n", 1496 | " for i,t in enumerate(tokens): \n", 1497 | " #To deal with subwords\n", 1498 | " if t.startswith(\"##\") == False : \n", 1499 | " if (subword == True or newword == True) :\n", 1500 | " #add word to the dict\n", 1501 | " word_seq.append(bert_word_index[word]) \n", 1502 | " start.append(start_idx)\n", 1503 | " end.append(end_idx) \n", 1504 | " subword = False \n", 1505 | " word = t\n", 1506 | " start_idx = i\n", 1507 | " end_idx = start_idx\n", 1508 | " newword = True\n", 1509 | " else:\n", 1510 | " subword = True\n", 1511 | " word = word + t[2:] #remove ##\n", 1512 | " end_idx = i\n", 1513 | " word_seqs.append(word_seq)\n", 1514 | " starts.append(start)\n", 1515 | " ends.append(end)\n", 1516 | "\n", 1517 | " #print(tokenized_sentence)\n", 1518 | "# if k==1000:\n", 1519 | "# break;\n", 1520 | " \n", 1521 | "print(len(word_seqs))\n", 1522 | "print(len(starts))\n", 1523 | "print(len(ends))\n", 1524 | "print(len(subword_seqs))\n", 1525 | "print(len(subword_seqs_tokens))\n", 1526 | " " 1527 | ], 1528 | "execution_count": 17, 1529 | "outputs": [ 1530 | { 1531 | "output_type": "stream", 1532 | "text": [ 1533 | "179\n", 1534 | "179\n", 1535 | "179\n", 1536 | "179\n", 1537 | "179\n" 1538 | ], 1539 | "name": "stdout" 1540 | } 1541 | ] 1542 | }, 1543 | { 1544 | "cell_type": "markdown", 1545 | "metadata": { 1546 | "id": "vKb-Fau55Sq5" 1547 | }, 1548 | "source": [ 1549 | "## Extract attended words above average in the last hidden states" 1550 | ] 1551 | }, 1552 | { 1553 | "cell_type": "markdown", 1554 | "metadata": { 1555 | "id": "z6yV4QYe5afK" 1556 | }, 1557 | "source": [ 1558 | "### Option 1: Get most attended words in all articles" 1559 | ] 1560 | }, 1561 | { 1562 | "cell_type": "code", 1563 | "metadata": { 1564 | "id": "dVHTWUP747Yh" 1565 | }, 1566 | "source": [ 1567 | "#FOR EACH ARTICLE\n", 1568 | "\n", 1569 | "all_word_idx_counts={}\n", 1570 | "all_word_idx_avg_att={}\n", 1571 | "for l,subword_seq in enumerate(subword_seqs):\n", 1572 | " input_ids = torch.tensor(subword_seq).unsqueeze(0)\n", 1573 | " outputs = model(input_ids)\n", 1574 | " #get the last layer attention \n", 1575 | " \n", 1576 | " #ITERATE OVER EACH LAYER ATTENTION HEADS \n", 1577 | " for i, out in enumerate(outputs[-1]): \n", 1578 | " if i != 11: #We only care about the last layer\n", 1579 | " continue\n", 1580 | " \n", 1581 | " att = out[0] #Get attention heads of this layer (12 attention heads)\n", 1582 | " matrix = torch.zeros(att[0].shape) # Avg attention across 12 heads\n", 1583 | " for a in att: \n", 1584 | " matrix = matrix + a\n", 1585 | " matrix = matrix/12 \n", 1586 | " matrix = matrix.detach().numpy() \n", 1587 | " word_seq = word_seqs[l] #word sequence for this paper\n", 1588 | " start = starts[l] #start token idx for words\n", 1589 | " end = ends[l] #end token idx for words\n", 1590 | " \n", 1591 | " #GENERATE NEW ATTENTION MATRIX (1 step: WHOLE_WORDS X TOKENS)\n", 1592 | " for j,word_idx in enumerate(word_seq):\n", 1593 | " t_start = start[j]\n", 1594 | " t_end = end[j]\n", 1595 | " # ROWS: Average attentions of word tokens by rows \n", 1596 | " if t_start == t_end:\n", 1597 | " attention_word_row = matrix[t_start,:]\n", 1598 | " else:\n", 1599 | " attention_word_row = np.mean(matrix[t_start:t_end+1,:], dtype=np.float64, axis=0) \n", 1600 | " # NEW MATRIX: Stack new rows (#of rows = # of words)\n", 1601 | " if j == 0:\n", 1602 | " new_matrix_rows = attention_word_row \n", 1603 | " else:\n", 1604 | " new_matrix_rows=np.vstack((new_matrix_rows,attention_word_row))\n", 1605 | "\n", 1606 | " #GENERATE NEW ATTENTION MATRIX (2 step: WHOLE_WORDS X WHOLE_WORDS)\n", 1607 | " for j,word_idx in enumerate(word_seq):\n", 1608 | " t_start = start[j]\n", 1609 | " t_end = end[j] \n", 1610 | " # COLUMNS average attention of word tokens by column \n", 1611 | " if t_start == t_end:\n", 1612 | " attention_word_col = new_matrix_rows[:,t_start]\n", 1613 | " else:\n", 1614 | " attention_word_col = np.mean(new_matrix_rows[:,t_start:t_end+1], dtype=np.float64, axis=1) \n", 1615 | " # NEW MATRIX: Stack new rows representing columns (#of rows = # of words) \n", 1616 | " if j == 0:\n", 1617 | " new_matrix = attention_word_col \n", 1618 | " else:\n", 1619 | " new_matrix=np.vstack((new_matrix,attention_word_col))\n", 1620 | " # Transpose the matrix to convert the rows into columns\n", 1621 | " new_matrix = new_matrix.T \n", 1622 | " \n", 1623 | " avgatt = np.mean(new_matrix, dtype=np.float64)\n", 1624 | " \n", 1625 | " #IDENTIFY WORDS WITH COLUMN ATTENTION AVERAGE GREATER THAN THE MATRIX AVERAGE \n", 1626 | " for word_position, row in enumerate(new_matrix.T): #Used the transpose of the attention matrix to get the columns as rows\n", 1627 | " if np.mean(row)>=avgatt: \n", 1628 | " word_idx = word_seqs[l][word_position] #get the word idx from sequence\n", 1629 | " #keep track of the times the word has been attended above average\n", 1630 | " if word_idx in all_word_idx_counts:\n", 1631 | " all_word_idx_counts[word_idx] +=1\n", 1632 | " else:\n", 1633 | " all_word_idx_counts[word_idx] = 1\n", 1634 | " #keep track of the average attention\n", 1635 | " if word_idx in all_word_idx_avg_att:\n", 1636 | " all_word_idx_avg_att[word_idx] += float(np.mean(row))/2\n", 1637 | " else:\n", 1638 | " all_word_idx_avg_att[word_idx] = float(np.mean(row))" 1639 | ], 1640 | "execution_count": 18, 1641 | "outputs": [] 1642 | }, 1643 | { 1644 | "cell_type": "markdown", 1645 | "metadata": { 1646 | "id": "5sKmh_JO7EV0" 1647 | }, 1648 | "source": [ 1649 | "#### Print most attended words in the dataset\n" 1650 | ] 1651 | }, 1652 | { 1653 | "cell_type": "code", 1654 | "metadata": { 1655 | "id": "eG01DMVM7Eo0", 1656 | "outputId": "418080f8-f30d-4ff5-e9db-cebc00b31b3a", 1657 | "colab": { 1658 | "base_uri": "https://localhost:8080/" 1659 | } 1660 | }, 1661 | "source": [ 1662 | "#Print attended words \n", 1663 | "sorted_word_idx_counts={k: v for k, v in sorted(all_word_idx_counts.items(), key=lambda item: item[1], reverse=True)[:20]} \n", 1664 | "for word_idx in sorted_word_idx_counts:\n", 1665 | " print(bert_index_word[word_idx],\":\",sorted_word_idx_counts[word_idx])" 1666 | ], 1667 | "execution_count": 19, 1668 | "outputs": [ 1669 | { 1670 | "output_type": "stream", 1671 | "text": [ 1672 | ". : 1315\n", 1673 | ", : 119\n", 1674 | "the : 93\n", 1675 | "surface : 32\n", 1676 | "water : 30\n", 1677 | "film : 27\n", 1678 | "students : 22\n", 1679 | "tree : 21\n", 1680 | "social : 20\n", 1681 | "data : 17\n", 1682 | "adsorption : 17\n", 1683 | "building : 17\n", 1684 | "forest : 17\n", 1685 | "models : 16\n", 1686 | "knowledge : 16\n", 1687 | "carbon : 15\n", 1688 | "education : 15\n", 1689 | "literature : 15\n", 1690 | "management : 14\n", 1691 | "species : 14\n" 1692 | ], 1693 | "name": "stdout" 1694 | } 1695 | ] 1696 | }, 1697 | { 1698 | "cell_type": "markdown", 1699 | "metadata": { 1700 | "id": "HZt7WTFj6IlJ" 1701 | }, 1702 | "source": [ 1703 | "### Option 2: Get most attended words in each research category" 1704 | ] 1705 | }, 1706 | { 1707 | "cell_type": "code", 1708 | "metadata": { 1709 | "id": "pjzjLPlU5_Ft" 1710 | }, 1711 | "source": [ 1712 | "from collections import Counter\n", 1713 | "\n", 1714 | "category_word_idx_counts={} # word_idx_count per category\n", 1715 | "category_word_idx_avg_att={} # word_idx_avg_att per category\n", 1716 | "\n", 1717 | "#FOR EACH ARTICLE\n", 1718 | "for l,subword_seq in enumerate(subword_seqs):\n", 1719 | " \n", 1720 | " word_idx_counts={} #per each article \n", 1721 | " word_idx_avg_att={} #per each article \n", 1722 | " input_ids = torch.tensor(subword_seq).unsqueeze(0)\n", 1723 | " outputs = model(input_ids)\n", 1724 | " \n", 1725 | " #ITERATE OVER EACH LAYER ATTENTION HEADS \n", 1726 | " for i, out in enumerate(outputs[-1]): \n", 1727 | " if i != 11: #We only care about the last layer\n", 1728 | " continue\n", 1729 | " \n", 1730 | " att = out[0] #Get attention heads of this layer (12 attention heads)\n", 1731 | " matrix = torch.zeros(att[0].shape) # Avg attention across 12 heads\n", 1732 | " for a in att: \n", 1733 | " matrix = matrix + a\n", 1734 | " matrix = matrix/12 \n", 1735 | " matrix = matrix.detach().numpy() \n", 1736 | " word_seq = word_seqs[l] #word sequence for this paper\n", 1737 | " start = starts[l] #start token idx for words\n", 1738 | " end = ends[l] #end token idx for words\n", 1739 | "\n", 1740 | " #GENERATE NEW ATTENTION MATRIX (1 step: WHOLE_WORDS X TOKENS)\n", 1741 | " for j,word_idx in enumerate(word_seq):\n", 1742 | " t_start = start[j]\n", 1743 | " t_end = end[j]\n", 1744 | " # ROWS: Average attentions of word tokens by rows \n", 1745 | " if t_start == t_end:\n", 1746 | " attention_word_row = matrix[t_start,:]\n", 1747 | " else:\n", 1748 | " attention_word_row = np.mean(matrix[t_start:t_end+1,:], dtype=np.float64, axis=0) \n", 1749 | " # NEW MATRIX: Stack new rows (#of rows = # of words)\n", 1750 | " if j == 0:\n", 1751 | " new_matrix_rows = attention_word_row \n", 1752 | " else:\n", 1753 | " new_matrix_rows=np.vstack((new_matrix_rows,attention_word_row))\n", 1754 | "\n", 1755 | " #GENERATE NEW ATTENTION MATRIX (2 step: WHOLE_WORDS X WHOLE_WORDS)\n", 1756 | " for j,word_idx in enumerate(word_seq):\n", 1757 | " t_start = start[j]\n", 1758 | " t_end = end[j] \n", 1759 | " # COLUMNS average attention of word tokens by column \n", 1760 | " if t_start == t_end:\n", 1761 | " attention_word_col = new_matrix_rows[:,t_start]\n", 1762 | " else:\n", 1763 | " attention_word_col = np.mean(new_matrix_rows[:,t_start:t_end+1], dtype=np.float64, axis=1) \n", 1764 | " # NEW MATRIX: Stack new rows representing columns (#of rows = # of words) \n", 1765 | " if j == 0:\n", 1766 | " new_matrix = attention_word_col \n", 1767 | " else:\n", 1768 | " new_matrix=np.vstack((new_matrix,attention_word_col))\n", 1769 | " # Transpose the matrix to convert the rows into columns\n", 1770 | " new_matrix = new_matrix.T \n", 1771 | " \n", 1772 | " avgatt = np.mean(new_matrix, dtype=np.float64)\n", 1773 | " \n", 1774 | " #IDENTIFY WORDS WITH COLUMN ATTENTION AVERAGE GREATER THAN THE MATRIX AVERAGE \n", 1775 | " for word_position, row in enumerate(new_matrix.T): #Used the transpose of the attention matrix to get the columns as rows\n", 1776 | " if np.mean(row)>=avgatt: \n", 1777 | " word_idx = word_seqs[l][word_position] #get the word idx from sequence\n", 1778 | " if word_idx in word_idx_counts:\n", 1779 | " word_idx_counts[word_idx] +=1\n", 1780 | " else:\n", 1781 | " word_idx_counts[word_idx] = 1\n", 1782 | " #keep track of the average attention\n", 1783 | " if word_idx in word_idx_avg_att:\n", 1784 | " word_idx_avg_att[word_idx] += float(np.mean(row))/2\n", 1785 | " else:\n", 1786 | " word_idx_avg_att[word_idx] = float(np.mean(row))\n", 1787 | " \n", 1788 | " #get article categories\n", 1789 | " cats = article_categories[l]\n", 1790 | " for cat in cats:\n", 1791 | " if cat in category_word_idx_counts: \n", 1792 | " category_word_idx_counts[cat] = dict(Counter(category_word_idx_counts[cat])+Counter(word_idx_counts)) #merge the two dictionaries and add the values of same keys\n", 1793 | " else:\n", 1794 | " category_word_idx_counts[cat] = word_idx_counts\n", 1795 | " #save the average attention\n", 1796 | " if cat in category_word_idx_avg_att: \n", 1797 | " #merge the two dictionaries and average the values of same keys\n", 1798 | " A=category_word_idx_avg_att[cat]\n", 1799 | " B=word_idx_avg_att\n", 1800 | " sums = dict(Counter(A) + Counter(B)) \n", 1801 | " category_word_idx_avg_att[cat] = means = {k: sums[k] / float((k in A) + (k in B)) for k in sums}\n", 1802 | " else:\n", 1803 | " category_word_idx_avg_att[cat] = word_idx_avg_att" 1804 | ], 1805 | "execution_count": 20, 1806 | "outputs": [] 1807 | }, 1808 | { 1809 | "cell_type": "markdown", 1810 | "metadata": { 1811 | "id": "QWu1G0yJ6lJ5" 1812 | }, 1813 | "source": [ 1814 | "#### Print most attended words in research categories" 1815 | ] 1816 | }, 1817 | { 1818 | "cell_type": "code", 1819 | "metadata": { 1820 | "id": "wOau0w7i6vfL", 1821 | "outputId": "05e6a0e5-616f-418b-af32-f3e8249ca16c", 1822 | "colab": { 1823 | "base_uri": "https://localhost:8080/" 1824 | } 1825 | }, 1826 | "source": [ 1827 | "first_level_categories = {'01':'Mathematical Sciences','02':'Physical Sciences','03':'Chemical Sciences',\n", 1828 | " '04':'Earth Sciences','05':'Environmental Sciences','06':'Biological Sciences',\n", 1829 | " '07':'Agricultural and Veterinary Sciences','08':'Information and Computing Sciences',\n", 1830 | " '09':'Engineering','10':'Technology','11':'Medical and Health Sciences',\n", 1831 | " '12':'Built Environment and Design','13':'Education','14':'Economics',\n", 1832 | " '15':'Commerce, Management, Tourism and Services','16':'Studies in Human Society',\n", 1833 | " '17':'Psychology and Cognitive Sciences','18':'Law and Legal Studies',\n", 1834 | " '19':'Studies in Creative Arts and Writing','20':'Language, Communication and Culture',\n", 1835 | " '21':'History and Archaeology','22':'Philosophy and Religious Studies'}\n", 1836 | "\n", 1837 | "#print top 20\n", 1838 | "for cat, word_idx_counts_cat in category_word_idx_counts.items(): \n", 1839 | " sorted_word_idx_counts_cat={k: v for k, v in sorted(word_idx_counts_cat.items(), key=lambda item: item[1], reverse=True)[:20]} \n", 1840 | " for word_idx in sorted_word_idx_counts_cat:\n", 1841 | " print(first_level_categories[cat],\";\",bert_index_word[word_idx],\";\",sorted_word_idx_counts_cat[word_idx]) " 1842 | ], 1843 | "execution_count": 22, 1844 | "outputs": [ 1845 | { 1846 | "output_type": "stream", 1847 | "text": [ 1848 | "Medical and Health Sciences ; . ; 92\n", 1849 | "Medical and Health Sciences ; , ; 21\n", 1850 | "Medical and Health Sciences ; the ; 12\n", 1851 | "Medical and Health Sciences ; models ; 12\n", 1852 | "Medical and Health Sciences ; cardiovascular ; 8\n", 1853 | "Medical and Health Sciences ; channels ; 6\n", 1854 | "Medical and Health Sciences ; blood ; 5\n", 1855 | "Medical and Health Sciences ; men ; 5\n", 1856 | "Medical and Health Sciences ; women ; 5\n", 1857 | "Medical and Health Sciences ; patient ; 4\n", 1858 | "Medical and Health Sciences ; data ; 4\n", 1859 | "Medical and Health Sciences ; eeg ; 4\n", 1860 | "Medical and Health Sciences ; flow ; 4\n", 1861 | "Medical and Health Sciences ; training ; 4\n", 1862 | "Medical and Health Sciences ; injury ; 4\n", 1863 | "Medical and Health Sciences ; fitness ; 4\n", 1864 | "Medical and Health Sciences ; activity ; 4\n", 1865 | "Medical and Health Sciences ; kv ; 4\n", 1866 | "Medical and Health Sciences ; patients ; 3\n", 1867 | "Medical and Health Sciences ; medication ; 3\n", 1868 | "Engineering ; . ; 101\n", 1869 | "Engineering ; water ; 8\n", 1870 | "Engineering ; surface ; 5\n", 1871 | "Engineering ; nickel ; 5\n", 1872 | "Engineering ; devices ; 5\n", 1873 | "Engineering ; wetland ; 4\n", 1874 | "Engineering ; wind ; 4\n", 1875 | "Engineering ; mechanical ; 4\n", 1876 | "Engineering ; endoscopic ; 4\n", 1877 | "Engineering ; flexible ; 4\n", 1878 | "Engineering ; , ; 4\n", 1879 | "Engineering ; equation ; 4\n", 1880 | "Engineering ; herbicides ; 3\n", 1881 | "Engineering ; turbine ; 3\n", 1882 | "Engineering ; ray ; 3\n", 1883 | "Engineering ; electron ; 3\n", 1884 | "Engineering ; coatings ; 3\n", 1885 | "Engineering ; carbon ; 3\n", 1886 | "Engineering ; innovation ; 3\n", 1887 | "Engineering ; surgical ; 3\n", 1888 | "Chemical Sciences ; . ; 46\n", 1889 | "Chemical Sciences ; surface ; 15\n", 1890 | "Chemical Sciences ; carbon ; 9\n", 1891 | "Chemical Sciences ; adsorption ; 9\n", 1892 | "Chemical Sciences ; water ; 6\n", 1893 | "Chemical Sciences ; transformation ; 6\n", 1894 | "Chemical Sciences ; the ; 6\n", 1895 | "Chemical Sciences ; nickel ; 5\n", 1896 | "Chemical Sciences ; functional ; 5\n", 1897 | "Chemical Sciences ; ray ; 4\n", 1898 | "Chemical Sciences ; energy ; 4\n", 1899 | "Chemical Sciences ; , ; 4\n", 1900 | "Chemical Sciences ; cu ; 4\n", 1901 | "Chemical Sciences ; hydrogen ; 4\n", 1902 | "Chemical Sciences ; metal ; 3\n", 1903 | "Chemical Sciences ; diffraction ; 3\n", 1904 | "Chemical Sciences ; x ; 3\n", 1905 | "Chemical Sciences ; properties ; 3\n", 1906 | "Chemical Sciences ; area ; 3\n", 1907 | "Chemical Sciences ; capacity ; 3\n", 1908 | "Psychology and Cognitive Sciences ; . ; 61\n", 1909 | "Psychology and Cognitive Sciences ; exercise ; 9\n", 1910 | "Psychology and Cognitive Sciences ; staff ; 8\n", 1911 | "Psychology and Cognitive Sciences ; agents ; 5\n", 1912 | "Psychology and Cognitive Sciences ; chemical ; 4\n", 1913 | "Psychology and Cognitive Sciences ; prison ; 4\n", 1914 | "Psychology and Cognitive Sciences ; action ; 3\n", 1915 | "Psychology and Cognitive Sciences ; informational ; 3\n", 1916 | "Psychology and Cognitive Sciences ; , ; 3\n", 1917 | "Psychology and Cognitive Sciences ; cognitive ; 3\n", 1918 | "Psychology and Cognitive Sciences ; problems ; 3\n", 1919 | "Psychology and Cognitive Sciences ; coordination ; 2\n", 1920 | "Psychology and Cognitive Sciences ; embodied ; 2\n", 1921 | "Psychology and Cognitive Sciences ; perception ; 2\n", 1922 | "Psychology and Cognitive Sciences ; organization ; 2\n", 1923 | "Psychology and Cognitive Sciences ; self ; 2\n", 1924 | "Psychology and Cognitive Sciences ; systems ; 2\n", 1925 | "Psychology and Cognitive Sciences ; bee ; 2\n", 1926 | "Psychology and Cognitive Sciences ; aggression ; 2\n", 1927 | "Psychology and Cognitive Sciences ; housing ; 2\n", 1928 | "Mathematical Sciences ; . ; 84\n", 1929 | "Mathematical Sciences ; , ; 12\n", 1930 | "Mathematical Sciences ; the ; 8\n", 1931 | "Mathematical Sciences ; control ; 7\n", 1932 | "Mathematical Sciences ; dynamics ; 3\n", 1933 | "Mathematical Sciences ; models ; 3\n", 1934 | "Mathematical Sciences ; growth ; 3\n", 1935 | "Mathematical Sciences ; uncertain ; 3\n", 1936 | "Mathematical Sciences ; nonlinear ; 3\n", 1937 | "Mathematical Sciences ; statistical ; 2\n", 1938 | "Mathematical Sciences ; data ; 2\n", 1939 | "Mathematical Sciences ; quantitative ; 2\n", 1940 | "Mathematical Sciences ; algebra ; 2\n", 1941 | "Mathematical Sciences ; group ; 2\n", 1942 | "Mathematical Sciences ; relaxation ; 2\n", 1943 | "Mathematical Sciences ; dynamic ; 2\n", 1944 | "Mathematical Sciences ; evolution ; 2\n", 1945 | "Mathematical Sciences ; network ; 2\n", 1946 | "Mathematical Sciences ; logics ; 2\n", 1947 | "Mathematical Sciences ; costs ; 2\n", 1948 | "Earth Sciences ; . ; 64\n", 1949 | "Earth Sciences ; surface ; 9\n", 1950 | "Earth Sciences ; water ; 8\n", 1951 | "Earth Sciences ; runoff ; 7\n", 1952 | "Earth Sciences ; assimilation ; 7\n", 1953 | "Earth Sciences ; precipitation ; 7\n", 1954 | "Earth Sciences ; sea ; 6\n", 1955 | "Earth Sciences ; data ; 6\n", 1956 | "Earth Sciences ; cold ; 6\n", 1957 | "Earth Sciences ; earthquakes ; 6\n", 1958 | "Earth Sciences ; forcing ; 5\n", 1959 | "Earth Sciences ; motion ; 5\n", 1960 | "Earth Sciences ; natural ; 5\n", 1961 | "Earth Sciences ; ocean ; 4\n", 1962 | "Earth Sciences ; temperature ; 4\n", 1963 | "Earth Sciences ; asian ; 4\n", 1964 | "Earth Sciences ; geochemical ; 4\n", 1965 | "Earth Sciences ; rocks ; 4\n", 1966 | "Earth Sciences ; age ; 4\n", 1967 | "Earth Sciences ; simulated ; 4\n", 1968 | "Biological Sciences ; . ; 80\n", 1969 | "Biological Sciences ; protein ; 10\n", 1970 | "Biological Sciences ; methylation ; 8\n", 1971 | "Biological Sciences ; dna ; 7\n", 1972 | "Biological Sciences ; autophagy ; 7\n", 1973 | "Biological Sciences ; proteins ; 7\n", 1974 | "Biological Sciences ; gene ; 5\n", 1975 | "Biological Sciences ; cell ; 5\n", 1976 | "Biological Sciences ; apoptosis ; 5\n", 1977 | "Biological Sciences ; expression ; 4\n", 1978 | "Biological Sciences ; mitochondrial ; 4\n", 1979 | "Biological Sciences ; chromatin ; 4\n", 1980 | "Biological Sciences ; mitochondria ; 4\n", 1981 | "Biological Sciences ; reprogramming ; 3\n", 1982 | "Biological Sciences ; germ ; 3\n", 1983 | "Biological Sciences ; sperm ; 3\n", 1984 | "Biological Sciences ; environmental ; 3\n", 1985 | "Biological Sciences ; stress ; 3\n", 1986 | "Biological Sciences ; plants ; 3\n", 1987 | "Biological Sciences ; species ; 3\n", 1988 | "Technology ; . ; 40\n", 1989 | "Technology ; , ; 20\n", 1990 | "Technology ; the ; 16\n", 1991 | "Technology ; protection ; 10\n", 1992 | "Technology ; network ; 8\n", 1993 | "Technology ; streaming ; 7\n", 1994 | "Technology ; nickel ; 5\n", 1995 | "Technology ; link ; 5\n", 1996 | "Technology ; reliability ; 5\n", 1997 | "Technology ; wireless ; 5\n", 1998 | "Technology ; feedback ; 5\n", 1999 | "Technology ; video ; 4\n", 2000 | "Technology ; quality ; 4\n", 2001 | "Technology ; services ; 4\n", 2002 | "Technology ; backup ; 4\n", 2003 | "Technology ; monitoring ; 4\n", 2004 | "Technology ; tissue ; 4\n", 2005 | "Technology ; carbon ; 3\n", 2006 | "Technology ; service ; 3\n", 2007 | "Technology ; hybrid ; 3\n", 2008 | "Economics ; . ; 69\n", 2009 | "Economics ; tax ; 12\n", 2010 | "Economics ; weather ; 10\n", 2011 | "Economics ; political ; 7\n", 2012 | "Economics ; design ; 7\n", 2013 | "Economics ; , ; 6\n", 2014 | "Economics ; uncertainty ; 6\n", 2015 | "Economics ; industrial ; 5\n", 2016 | "Economics ; diagnostic ; 4\n", 2017 | "Economics ; the ; 4\n", 2018 | "Economics ; trade ; 4\n", 2019 | "Economics ; government ; 4\n", 2020 | "Economics ; private ; 4\n", 2021 | "Economics ; forecasts ; 4\n", 2022 | "Economics ; social ; 4\n", 2023 | "Economics ; individual ; 4\n", 2024 | "Economics ; innovation ; 4\n", 2025 | "Economics ; test ; 3\n", 2026 | "Economics ; rights ; 3\n", 2027 | "Economics ; power ; 3\n", 2028 | "Physical Sciences ; . ; 47\n", 2029 | "Physical Sciences ; , ; 14\n", 2030 | "Physical Sciences ; the ; 14\n", 2031 | "Physical Sciences ; discharge ; 8\n", 2032 | "Physical Sciences ; electrode ; 7\n", 2033 | "Physical Sciences ; atoms ; 5\n", 2034 | "Physical Sciences ; solid ; 5\n", 2035 | "Physical Sciences ; particles ; 4\n", 2036 | "Physical Sciences ; particle ; 4\n", 2037 | "Physical Sciences ; frequency ; 4\n", 2038 | "Physical Sciences ; plasma ; 4\n", 2039 | "Physical Sciences ; cell ; 4\n", 2040 | "Physical Sciences ; solutions ; 3\n", 2041 | "Physical Sciences ; ) ; 3\n", 2042 | "Physical Sciences ; invariant ; 3\n", 2043 | "Physical Sciences ; measurements ; 3\n", 2044 | "Physical Sciences ; wave ; 3\n", 2045 | "Physical Sciences ; electron ; 3\n", 2046 | "Physical Sciences ; discharges ; 3\n", 2047 | "Physical Sciences ; fly ; 3\n", 2048 | "Information and Computing Sciences ; . ; 59\n", 2049 | "Information and Computing Sciences ; control ; 9\n", 2050 | "Information and Computing Sciences ; water ; 6\n", 2051 | "Information and Computing Sciences ; neural ; 4\n", 2052 | "Information and Computing Sciences ; estimation ; 4\n", 2053 | "Information and Computing Sciences ; traffic ; 4\n", 2054 | "Information and Computing Sciences ; coordination ; 4\n", 2055 | "Information and Computing Sciences ; wave ; 4\n", 2056 | "Information and Computing Sciences ; problems ; 4\n", 2057 | "Information and Computing Sciences ; vehicles ; 3\n", 2058 | "Information and Computing Sciences ; uncertain ; 3\n", 2059 | "Information and Computing Sciences ; nonlinear ; 3\n", 2060 | "Information and Computing Sciences ; display ; 3\n", 2061 | "Information and Computing Sciences ; ann ; 3\n", 2062 | "Information and Computing Sciences ; computational ; 3\n", 2063 | "Information and Computing Sciences ; search ; 3\n", 2064 | "Information and Computing Sciences ; speech ; 3\n", 2065 | "Information and Computing Sciences ; linguistic ; 3\n", 2066 | "Information and Computing Sciences ; reconstruction ; 2\n", 2067 | "Information and Computing Sciences ; images ; 2\n", 2068 | "Studies in Human Society ; . ; 44\n", 2069 | "Studies in Human Society ; europe ; 6\n", 2070 | "Studies in Human Society ; racial ; 6\n", 2071 | "Studies in Human Society ; collaborative ; 5\n", 2072 | "Studies in Human Society ; community ; 4\n", 2073 | "Studies in Human Society ; forest ; 4\n", 2074 | "Studies in Human Society ; management ; 4\n", 2075 | "Studies in Human Society ; processes ; 4\n", 2076 | "Studies in Human Society ; security ; 4\n", 2077 | "Studies in Human Society ; diversity ; 4\n", 2078 | "Studies in Human Society ; spatial ; 4\n", 2079 | "Studies in Human Society ; disposal ; 4\n", 2080 | "Studies in Human Society ; campaign ; 4\n", 2081 | "Studies in Human Society ; negative ; 3\n", 2082 | "Studies in Human Society ; boundary ; 3\n", 2083 | "Studies in Human Society ; the ; 3\n", 2084 | "Studies in Human Society ; defence ; 3\n", 2085 | "Studies in Human Society ; political ; 3\n", 2086 | "Studies in Human Society ; waste ; 3\n", 2087 | "Studies in Human Society ; elections ; 3\n", 2088 | "Language, Communication and Culture ; . ; 46\n", 2089 | "Language, Communication and Culture ; fashion ; 13\n", 2090 | "Language, Communication and Culture ; culture ; 10\n", 2091 | "Language, Communication and Culture ; language ; 10\n", 2092 | "Language, Communication and Culture ; literature ; 9\n", 2093 | "Language, Communication and Culture ; textual ; 8\n", 2094 | "Language, Communication and Culture ; china ; 6\n", 2095 | "Language, Communication and Culture ; chinese ; 6\n", 2096 | "Language, Communication and Culture ; speech ; 4\n", 2097 | "Language, Communication and Culture ; test ; 4\n", 2098 | "Language, Communication and Culture ; semantic ; 4\n", 2099 | "Language, Communication and Culture ; song ; 3\n", 2100 | "Language, Communication and Culture ; cultural ; 3\n", 2101 | "Language, Communication and Culture ; development ; 3\n", 2102 | "Language, Communication and Culture ; age ; 3\n", 2103 | "Language, Communication and Culture ; corpus ; 3\n", 2104 | "Language, Communication and Culture ; authors ; 2\n", 2105 | "Language, Communication and Culture ; problematic ; 2\n", 2106 | "Language, Communication and Culture ; object ; 2\n", 2107 | "Language, Communication and Culture ; history ; 2\n", 2108 | "Environmental Sciences ; . ; 81\n", 2109 | "Environmental Sciences ; soil ; 8\n", 2110 | "Environmental Sciences ; adsorption ; 7\n", 2111 | "Environmental Sciences ; grassland ; 6\n", 2112 | "Environmental Sciences ; species ; 6\n", 2113 | "Environmental Sciences ; tree ; 5\n", 2114 | "Environmental Sciences ; soils ; 4\n", 2115 | "Environmental Sciences ; grain ; 4\n", 2116 | "Environmental Sciences ; accumulation ; 4\n", 2117 | "Environmental Sciences ; forest ; 4\n", 2118 | "Environmental Sciences ; biochar ; 4\n", 2119 | "Environmental Sciences ; ecological ; 3\n", 2120 | "Environmental Sciences ; , ; 3\n", 2121 | "Environmental Sciences ; carbon ; 3\n", 2122 | "Environmental Sciences ; green ; 3\n", 2123 | "Environmental Sciences ; organic ; 3\n", 2124 | "Environmental Sciences ; worms ; 3\n", 2125 | "Environmental Sciences ; phosphorus ; 3\n", 2126 | "Environmental Sciences ; adsorbed ; 3\n", 2127 | "Environmental Sciences ; p ; 3\n", 2128 | "Agricultural and Veterinary Sciences ; . ; 68\n", 2129 | "Agricultural and Veterinary Sciences ; tree ; 16\n", 2130 | "Agricultural and Veterinary Sciences ; animal ; 8\n", 2131 | "Agricultural and Veterinary Sciences ; forest ; 7\n", 2132 | "Agricultural and Veterinary Sciences ; buffalo ; 6\n", 2133 | "Agricultural and Veterinary Sciences ; trees ; 4\n", 2134 | "Agricultural and Veterinary Sciences ; biomedical ; 4\n", 2135 | "Agricultural and Veterinary Sciences ; domestic ; 4\n", 2136 | "Agricultural and Veterinary Sciences ; sheep ; 3\n", 2137 | "Agricultural and Veterinary Sciences ; resistance ; 3\n", 2138 | "Agricultural and Veterinary Sciences ; health ; 3\n", 2139 | "Agricultural and Veterinary Sciences ; animals ; 3\n", 2140 | "Agricultural and Veterinary Sciences ; carbon ; 2\n", 2141 | "Agricultural and Veterinary Sciences ; ethanol ; 2\n", 2142 | "Agricultural and Veterinary Sciences ; sorghum ; 2\n", 2143 | "Agricultural and Veterinary Sciences ; genotypes ; 2\n", 2144 | "Agricultural and Veterinary Sciences ; sugar ; 2\n", 2145 | "Agricultural and Veterinary Sciences ; assay ; 2\n", 2146 | "Agricultural and Veterinary Sciences ; goat ; 2\n", 2147 | "Agricultural and Veterinary Sciences ; pig ; 2\n", 2148 | "History and Archaeology ; . ; 85\n", 2149 | "History and Archaeology ; europe ; 6\n", 2150 | "History and Archaeology ; cultural ; 6\n", 2151 | "History and Archaeology ; particle ; 5\n", 2152 | "History and Archaeology ; photographs ; 5\n", 2153 | "History and Archaeology ; archaeological ; 5\n", 2154 | "History and Archaeology ; patients ; 4\n", 2155 | "History and Archaeology ; security ; 4\n", 2156 | "History and Archaeology ; history ; 4\n", 2157 | "History and Archaeology ; procedures ; 3\n", 2158 | "History and Archaeology ; defence ; 3\n", 2159 | "History and Archaeology ; practices ; 3\n", 2160 | "History and Archaeology ; archaeology ; 3\n", 2161 | "History and Archaeology ; steel ; 3\n", 2162 | "History and Archaeology ; song ; 3\n", 2163 | "History and Archaeology ; peopling ; 3\n", 2164 | "History and Archaeology ; gender ; 2\n", 2165 | "History and Archaeology ; intervention ; 2\n", 2166 | "History and Archaeology ; regression ; 2\n", 2167 | "History and Archaeology ; asia ; 2\n", 2168 | "Education ; . ; 53\n", 2169 | "Education ; students ; 22\n", 2170 | "Education ; knowledge ; 14\n", 2171 | "Education ; education ; 14\n", 2172 | "Education ; mathematics ; 12\n", 2173 | "Education ; school ; 10\n", 2174 | "Education ; curriculum ; 9\n", 2175 | "Education ; teachers ; 9\n", 2176 | "Education ; training ; 9\n", 2177 | "Education ; teacher ; 6\n", 2178 | "Education ; teaching ; 6\n", 2179 | "Education ; university ; 5\n", 2180 | "Education ; attitudes ; 5\n", 2181 | "Education ; medical ; 5\n", 2182 | "Education ; higher ; 5\n", 2183 | "Education ; classroom ; 4\n", 2184 | "Education ; reading ; 4\n", 2185 | "Education ; learning ; 4\n", 2186 | "Education ; high ; 4\n", 2187 | "Education ; collaborative ; 4\n", 2188 | "Law and Legal Studies ; . ; 41\n", 2189 | "Law and Legal Studies ; legal ; 8\n", 2190 | "Law and Legal Studies ; liability ; 6\n", 2191 | "Law and Legal Studies ; investments ; 5\n", 2192 | "Law and Legal Studies ; national ; 5\n", 2193 | "Law and Legal Studies ; convention ; 5\n", 2194 | "Law and Legal Studies ; rule ; 4\n", 2195 | "Law and Legal Studies ; crown ; 3\n", 2196 | "Law and Legal Studies ; modern ; 3\n", 2197 | "Law and Legal Studies ; , ; 3\n", 2198 | "Law and Legal Studies ; law ; 3\n", 2199 | "Law and Legal Studies ; social ; 3\n", 2200 | "Law and Legal Studies ; animal ; 3\n", 2201 | "Law and Legal Studies ; international ; 3\n", 2202 | "Law and Legal Studies ; precipitate ; 3\n", 2203 | "Law and Legal Studies ; forest ; 2\n", 2204 | "Law and Legal Studies ; corporate ; 2\n", 2205 | "Law and Legal Studies ; multinational ; 2\n", 2206 | "Law and Legal Studies ; corporations ; 2\n", 2207 | "Law and Legal Studies ; the ; 2\n", 2208 | "Commerce, Management, Tourism and Services ; . ; 43\n", 2209 | "Commerce, Management, Tourism and Services ; , ; 15\n", 2210 | "Commerce, Management, Tourism and Services ; the ; 10\n", 2211 | "Commerce, Management, Tourism and Services ; firms ; 9\n", 2212 | "Commerce, Management, Tourism and Services ; corporate ; 7\n", 2213 | "Commerce, Management, Tourism and Services ; management ; 5\n", 2214 | "Commerce, Management, Tourism and Services ; governance ; 5\n", 2215 | "Commerce, Management, Tourism and Services ; ownership ; 5\n", 2216 | "Commerce, Management, Tourism and Services ; events ; 5\n", 2217 | "Commerce, Management, Tourism and Services ; industrial ; 5\n", 2218 | "Commerce, Management, Tourism and Services ; accounts ; 5\n", 2219 | "Commerce, Management, Tourism and Services ; account ; 5\n", 2220 | "Commerce, Management, Tourism and Services ; firm ; 4\n", 2221 | "Commerce, Management, Tourism and Services ; market ; 4\n", 2222 | "Commerce, Management, Tourism and Services ; innovation ; 4\n", 2223 | "Commerce, Management, Tourism and Services ; information ; 3\n", 2224 | "Commerce, Management, Tourism and Services ; public ; 3\n", 2225 | "Commerce, Management, Tourism and Services ; social ; 3\n", 2226 | "Commerce, Management, Tourism and Services ; programs ; 3\n", 2227 | "Commerce, Management, Tourism and Services ; individual ; 3\n", 2228 | "Philosophy and Religious Studies ; . ; 60\n", 2229 | "Philosophy and Religious Studies ; justice ; 6\n", 2230 | "Philosophy and Religious Studies ; catholic ; 6\n", 2231 | "Philosophy and Religious Studies ; , ; 5\n", 2232 | "Philosophy and Religious Studies ; ethics ; 5\n", 2233 | "Philosophy and Religious Studies ; ethical ; 5\n", 2234 | "Philosophy and Religious Studies ; church ; 4\n", 2235 | "Philosophy and Religious Studies ; moral ; 4\n", 2236 | "Philosophy and Religious Studies ; social ; 4\n", 2237 | "Philosophy and Religious Studies ; chemical ; 4\n", 2238 | "Philosophy and Religious Studies ; nuclear ; 4\n", 2239 | "Philosophy and Religious Studies ; power ; 4\n", 2240 | "Philosophy and Religious Studies ; science ; 3\n", 2241 | "Philosophy and Religious Studies ; roman ; 3\n", 2242 | "Philosophy and Religious Studies ; self ; 3\n", 2243 | "Philosophy and Religious Studies ; the ; 2\n", 2244 | "Philosophy and Religious Studies ; interpretations ; 2\n", 2245 | "Philosophy and Religious Studies ; institutional ; 2\n", 2246 | "Philosophy and Religious Studies ; arguments ; 2\n", 2247 | "Philosophy and Religious Studies ; account ; 2\n", 2248 | "Built Environment and Design ; . ; 39\n", 2249 | "Built Environment and Design ; building ; 15\n", 2250 | "Built Environment and Design ; emotion ; 10\n", 2251 | "Built Environment and Design ; agents ; 6\n", 2252 | "Built Environment and Design ; architectural ; 6\n", 2253 | "Built Environment and Design ; energy ; 6\n", 2254 | "Built Environment and Design ; space ; 5\n", 2255 | "Built Environment and Design ; action ; 4\n", 2256 | "Built Environment and Design ; house ; 4\n", 2257 | "Built Environment and Design ; agent ; 3\n", 2258 | "Built Environment and Design ; buildings ; 3\n", 2259 | "Built Environment and Design ; human ; 3\n", 2260 | "Built Environment and Design ; construction ; 2\n", 2261 | "Built Environment and Design ; autonomous ; 2\n", 2262 | "Built Environment and Design ; artificial ; 2\n", 2263 | "Built Environment and Design ; art ; 2\n", 2264 | "Built Environment and Design ; architecture ; 2\n", 2265 | "Built Environment and Design ; environments ; 2\n", 2266 | "Built Environment and Design ; century ; 2\n", 2267 | "Built Environment and Design ; dynamic ; 2\n", 2268 | "Studies in Creative Arts and Writing ; . ; 48\n", 2269 | "Studies in Creative Arts and Writing ; film ; 27\n", 2270 | "Studies in Creative Arts and Writing ; the ; 5\n", 2271 | "Studies in Creative Arts and Writing ; , ; 4\n", 2272 | "Studies in Creative Arts and Writing ; films ; 4\n", 2273 | "Studies in Creative Arts and Writing ; examinations ; 4\n", 2274 | "Studies in Creative Arts and Writing ; foraging ; 4\n", 2275 | "Studies in Creative Arts and Writing ; vortices ; 3\n", 2276 | "Studies in Creative Arts and Writing ; examination ; 3\n", 2277 | "Studies in Creative Arts and Writing ; viscoelastic ; 2\n", 2278 | "Studies in Creative Arts and Writing ; equation ; 2\n", 2279 | "Studies in Creative Arts and Writing ; media ; 2\n", 2280 | "Studies in Creative Arts and Writing ; literature ; 2\n", 2281 | "Studies in Creative Arts and Writing ; negative ; 2\n", 2282 | "Studies in Creative Arts and Writing ; evolution ; 2\n", 2283 | "Studies in Creative Arts and Writing ; equations ; 2\n", 2284 | "Studies in Creative Arts and Writing ; gradient ; 2\n", 2285 | "Studies in Creative Arts and Writing ; dynamics ; 2\n", 2286 | "Studies in Creative Arts and Writing ; structure ; 2\n", 2287 | "Studies in Creative Arts and Writing ; objects ; 2\n" 2288 | ], 2289 | "name": "stdout" 2290 | } 2291 | ] 2292 | }, 2293 | { 2294 | "cell_type": "code", 2295 | "metadata": { 2296 | "id": "Zb0RRBVY9LEb" 2297 | }, 2298 | "source": [ 2299 | "" 2300 | ], 2301 | "execution_count": null, 2302 | "outputs": [] 2303 | } 2304 | ] 2305 | } -------------------------------------------------------------------------------- /notebooks/BertModelsAttentionHeadsVisualization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "BertModelsAttentionHeadsVisualization.ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [], 9 | "toc_visible": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "accelerator": "GPU" 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "BGKyqGtbPybZ" 22 | }, 23 | "source": [ 24 | "# Visualize average self-attention in the last layer" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": { 30 | "id": "MQ1W3dmWHuXB" 31 | }, 32 | "source": [ 33 | "## Download fine-tuned models and libraries \n", 34 | "\n", 35 | "Choose only one model below." 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": { 41 | "id": "K9oqXrkjH1Le" 42 | }, 43 | "source": [ 44 | "### Option 1: Fine-tuned SciBERT on articles clasification" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "metadata": { 50 | "id": "WqNZtguO99Us", 51 | "outputId": "7ab5294b-8e26-4422-d05e-b43b7840ea13", 52 | "colab": { 53 | "base_uri": "https://localhost:8080/" 54 | } 55 | }, 56 | "source": [ 57 | "!gdown --id 1uKFfoh95K7lRXEwdMt7F32ivohwAENzT" 58 | ], 59 | "execution_count": 2, 60 | "outputs": [ 61 | { 62 | "output_type": "stream", 63 | "text": [ 64 | "Downloading...\n", 65 | "From: https://drive.google.com/uc?id=1uKFfoh95K7lRXEwdMt7F32ivohwAENzT\n", 66 | "To: /content/scibert10v2_scigraph0.pt\n", 67 | "440MB [00:04, 99.2MB/s]\n" 68 | ], 69 | "name": "stdout" 70 | } 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": { 76 | "id": "oa7sawoCQRj1" 77 | }, 78 | "source": [ 79 | "### Option 2: Fine-tuned BERT on articles clasification" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "metadata": { 85 | "id": "nqkkvurIQXWj" 86 | }, 87 | "source": [ 88 | "!gdown --id 19pNPMyguDUPOlza-B4J-NqOOdU-ZqGwl" 89 | ], 90 | "execution_count": null, 91 | "outputs": [] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": { 96 | "id": "75AFWS0JQbWK" 97 | }, 98 | "source": [ 99 | "### Option 3: Fine-tuned BioBERT 1.1 on articles clasification" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "metadata": { 105 | "id": "f5TUOSD3QbWL" 106 | }, 107 | "source": [ 108 | "!gdown --id 1waZOhC4LADr6rQx0cf3LRA4UZSj6u2uu" 109 | ], 110 | "execution_count": null, 111 | "outputs": [] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": { 116 | "id": "ZzVaPSqDM8EP" 117 | }, 118 | "source": [ 119 | "Download library used to fine-tune the LM on mutilabel classification" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "metadata": { 125 | "id": "qIcu-vgxly0h", 126 | "outputId": "747cb42e-2d49-4f38-ebe6-e2b7264d8a63", 127 | "colab": { 128 | "base_uri": "https://localhost:8080/" 129 | } 130 | }, 131 | "source": [ 132 | "!gdown --id 1LpufGkbVYTGxgAHr2TyqQimqVRte420U" 133 | ], 134 | "execution_count": 3, 135 | "outputs": [ 136 | { 137 | "output_type": "stream", 138 | "text": [ 139 | "Downloading...\n", 140 | "From: https://drive.google.com/uc?id=1LpufGkbVYTGxgAHr2TyqQimqVRte420U\n", 141 | "To: /content/BertModeling.py\n", 142 | "\r 0% 0.00/14.4k [00:00=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (4.41.1)\n", 215 | "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (0.7)\n", 216 | "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (3.0.12)\n", 217 | "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (2019.12.20)\n", 218 | "Collecting botocore<1.20.0,>=1.19.7\n", 219 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d1/58/64883046f9c98d9f94cc81174d0b83e0be35b8f6c252e255b709b9024ef1/botocore-1.19.7-py2.py3-none-any.whl (6.7MB)\n", 220 | "\u001b[K |████████████████████████████████| 6.7MB 38.4MB/s \n", 221 | "\u001b[?25hCollecting s3transfer<0.4.0,>=0.3.0\n", 222 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)\n", 223 | "\u001b[K |████████████████████████████████| 71kB 9.5MB/s \n", 224 | "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n", 225 | " Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n", 226 | "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (1.24.3)\n", 227 | "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2.10)\n", 228 | "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (3.0.4)\n", 229 | "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2020.6.20)\n", 230 | "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (1.15.0)\n", 231 | "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (7.1.2)\n", 232 | "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (0.17.0)\n", 233 | "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.20.0,>=1.19.7->boto3->transformers==2.8.0) (2.8.1)\n", 234 | "Building wheels for collected packages: sacremoses\n", 235 | " Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n", 236 | " Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=0a7c6e330e34c74a1190a108fb2e815b3d8aae1a1aa741366b8fa4d16f636f52\n", 237 | " Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n", 238 | "Successfully built sacremoses\n", 239 | "\u001b[31mERROR: botocore 1.19.7 has requirement urllib3<1.26,>=1.25.4; python_version != \"3.4\", but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n", 240 | "Installing collected packages: jmespath, botocore, s3transfer, boto3, sacremoses, tokenizers, sentencepiece, transformers\n", 241 | "Successfully installed boto3-1.16.7 botocore-1.19.7 jmespath-0.10.0 s3transfer-0.3.3 sacremoses-0.0.43 sentencepiece-0.1.94 tokenizers-0.5.2 transformers-2.8.0\n" 242 | ], 243 | "name": "stdout" 244 | } 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "metadata": { 250 | "id": "tJZ1eBJo9mOG" 251 | }, 252 | "source": [ 253 | "import sys\n", 254 | "\n", 255 | "from BertModeling import BertForMultiLabelSequenceClassification\n", 256 | "from BertModeling import (BertForMultiLabelSequenceClassification, BioBertForMultiLabelSequenceClassification, \n", 257 | " BioBertForMultiLabelSequenceClassification2, XLNetForMultiLabelSequenceClassification,\n", 258 | " GPT2MultiLabelClassification)\n", 259 | "from transformers import (WEIGHTS_NAME, BertConfig, BertForSequenceClassification, BertTokenizer, AutoTokenizer, AutoModelWithLMHead,AutoConfig,\n", 260 | " XLMConfig, XLMForSequenceClassification, XLMTokenizer, \n", 261 | " XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer,\n", 262 | " RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer)\n", 263 | "#from bertviz.transformers_neuron_view import BertModel, BertTokenizer, BertForPreTraining, BertForSequenceClassification\n", 264 | "import matplotlib.pyplot as plt\n", 265 | "import matplotlib\n", 266 | "import numpy as np\n", 267 | "from transformers import BertTokenizer\n", 268 | "import torch\n", 269 | "from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,\n", 270 | " TensorDataset)" 271 | ], 272 | "execution_count": 6, 273 | "outputs": [] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": { 278 | "id": "evk4z9YvMbkI" 279 | }, 280 | "source": [ 281 | "## Load the fine-tuned language models" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "metadata": { 287 | "id": "0klcUIYtL3PR", 288 | "outputId": "4e27d7df-646f-49ba-edb7-f3e7b91950c5", 289 | "colab": { 290 | "base_uri": "https://localhost:8080/", 291 | "height": 35 292 | } 293 | }, 294 | "source": [ 295 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 296 | "n_gpu = torch.cuda.device_count()\n", 297 | "torch.cuda.get_device_name(0)" 298 | ], 299 | "execution_count": 7, 300 | "outputs": [ 301 | { 302 | "output_type": "execute_result", 303 | "data": { 304 | "application/vnd.google.colaboratory.intrinsic+json": { 305 | "type": "string" 306 | }, 307 | "text/plain": [ 308 | "'Tesla P100-PCIE-16GB'" 309 | ] 310 | }, 311 | "metadata": { 312 | "tags": [] 313 | }, 314 | "execution_count": 7 315 | } 316 | ] 317 | }, 318 | { 319 | "cell_type": "markdown", 320 | "metadata": { 321 | "id": "gK0EI6iEMh3K" 322 | }, 323 | "source": [ 324 | "Choose only one of the models by running the corresponding cell" 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "metadata": { 330 | "id": "s7DpF6rxM2d9" 331 | }, 332 | "source": [ 333 | "#### Option 1: Fine-tuned SciBERT" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "metadata": { 339 | "id": "h6X4jQgKL8ZJ", 340 | "outputId": "17dc89d3-9c7f-4e08-bb16-2d83925ac98d", 341 | "colab": { 342 | "base_uri": "https://localhost:8080/" 343 | } 344 | }, 345 | "source": [ 346 | "#scibert \n", 347 | "pretrainedModelpath = 'allenai/scibert_scivocab_uncased' \n", 348 | "modelpath = './scibert10v2_scigraph0.pt' #fine-tuned model\n", 349 | "\n", 350 | "config = AutoConfig.from_pretrained('allenai/scibert_scivocab_uncased', output_hidden_states=True, output_attentions=True, num_labels=22)#\n", 351 | "model = BertForMultiLabelSequenceClassification(config=config)\n", 352 | "model.load_state_dict(torch.load(modelpath))\n", 353 | "\n", 354 | "bert_tokenizer = BertTokenizer.from_pretrained(pretrainedModelpath, do_lower_case=True, return_token_type_ids=True)\n", 355 | "model.eval()" 356 | ], 357 | "execution_count": 15, 358 | "outputs": [ 359 | { 360 | "output_type": "execute_result", 361 | "data": { 362 | "text/plain": [ 363 | "BertForMultiLabelSequenceClassification(\n", 364 | " (bert): BertModel(\n", 365 | " (embeddings): BertEmbeddings(\n", 366 | " (word_embeddings): Embedding(31090, 768, padding_idx=0)\n", 367 | " (position_embeddings): Embedding(512, 768)\n", 368 | " (token_type_embeddings): Embedding(2, 768)\n", 369 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 370 | " (dropout): Dropout(p=0.1, inplace=False)\n", 371 | " )\n", 372 | " (encoder): BertEncoder(\n", 373 | " (layer): ModuleList(\n", 374 | " (0): BertLayer(\n", 375 | " (attention): BertAttention(\n", 376 | " (self): BertSelfAttention(\n", 377 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 378 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 379 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 380 | " (dropout): Dropout(p=0.1, inplace=False)\n", 381 | " )\n", 382 | " (output): BertSelfOutput(\n", 383 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 384 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 385 | " (dropout): Dropout(p=0.1, inplace=False)\n", 386 | " )\n", 387 | " )\n", 388 | " (intermediate): BertIntermediate(\n", 389 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 390 | " )\n", 391 | " (output): BertOutput(\n", 392 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 393 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 394 | " (dropout): Dropout(p=0.1, inplace=False)\n", 395 | " )\n", 396 | " )\n", 397 | " (1): BertLayer(\n", 398 | " (attention): BertAttention(\n", 399 | " (self): BertSelfAttention(\n", 400 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 401 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 402 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 403 | " (dropout): Dropout(p=0.1, inplace=False)\n", 404 | " )\n", 405 | " (output): BertSelfOutput(\n", 406 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 407 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 408 | " (dropout): Dropout(p=0.1, inplace=False)\n", 409 | " )\n", 410 | " )\n", 411 | " (intermediate): BertIntermediate(\n", 412 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 413 | " )\n", 414 | " (output): BertOutput(\n", 415 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 416 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 417 | " (dropout): Dropout(p=0.1, inplace=False)\n", 418 | " )\n", 419 | " )\n", 420 | " (2): BertLayer(\n", 421 | " (attention): BertAttention(\n", 422 | " (self): BertSelfAttention(\n", 423 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 424 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 425 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 426 | " (dropout): Dropout(p=0.1, inplace=False)\n", 427 | " )\n", 428 | " (output): BertSelfOutput(\n", 429 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 430 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 431 | " (dropout): Dropout(p=0.1, inplace=False)\n", 432 | " )\n", 433 | " )\n", 434 | " (intermediate): BertIntermediate(\n", 435 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 436 | " )\n", 437 | " (output): BertOutput(\n", 438 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 439 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 440 | " (dropout): Dropout(p=0.1, inplace=False)\n", 441 | " )\n", 442 | " )\n", 443 | " (3): BertLayer(\n", 444 | " (attention): BertAttention(\n", 445 | " (self): BertSelfAttention(\n", 446 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 447 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 448 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 449 | " (dropout): Dropout(p=0.1, inplace=False)\n", 450 | " )\n", 451 | " (output): BertSelfOutput(\n", 452 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 453 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 454 | " (dropout): Dropout(p=0.1, inplace=False)\n", 455 | " )\n", 456 | " )\n", 457 | " (intermediate): BertIntermediate(\n", 458 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 459 | " )\n", 460 | " (output): BertOutput(\n", 461 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 462 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 463 | " (dropout): Dropout(p=0.1, inplace=False)\n", 464 | " )\n", 465 | " )\n", 466 | " (4): BertLayer(\n", 467 | " (attention): BertAttention(\n", 468 | " (self): BertSelfAttention(\n", 469 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 470 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 471 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 472 | " (dropout): Dropout(p=0.1, inplace=False)\n", 473 | " )\n", 474 | " (output): BertSelfOutput(\n", 475 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 476 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 477 | " (dropout): Dropout(p=0.1, inplace=False)\n", 478 | " )\n", 479 | " )\n", 480 | " (intermediate): BertIntermediate(\n", 481 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 482 | " )\n", 483 | " (output): BertOutput(\n", 484 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 485 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 486 | " (dropout): Dropout(p=0.1, inplace=False)\n", 487 | " )\n", 488 | " )\n", 489 | " (5): BertLayer(\n", 490 | " (attention): BertAttention(\n", 491 | " (self): BertSelfAttention(\n", 492 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 493 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 494 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 495 | " (dropout): Dropout(p=0.1, inplace=False)\n", 496 | " )\n", 497 | " (output): BertSelfOutput(\n", 498 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 499 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 500 | " (dropout): Dropout(p=0.1, inplace=False)\n", 501 | " )\n", 502 | " )\n", 503 | " (intermediate): BertIntermediate(\n", 504 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 505 | " )\n", 506 | " (output): BertOutput(\n", 507 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 508 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 509 | " (dropout): Dropout(p=0.1, inplace=False)\n", 510 | " )\n", 511 | " )\n", 512 | " (6): BertLayer(\n", 513 | " (attention): BertAttention(\n", 514 | " (self): BertSelfAttention(\n", 515 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 516 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 517 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 518 | " (dropout): Dropout(p=0.1, inplace=False)\n", 519 | " )\n", 520 | " (output): BertSelfOutput(\n", 521 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 522 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 523 | " (dropout): Dropout(p=0.1, inplace=False)\n", 524 | " )\n", 525 | " )\n", 526 | " (intermediate): BertIntermediate(\n", 527 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 528 | " )\n", 529 | " (output): BertOutput(\n", 530 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 531 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 532 | " (dropout): Dropout(p=0.1, inplace=False)\n", 533 | " )\n", 534 | " )\n", 535 | " (7): BertLayer(\n", 536 | " (attention): BertAttention(\n", 537 | " (self): BertSelfAttention(\n", 538 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 539 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 540 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 541 | " (dropout): Dropout(p=0.1, inplace=False)\n", 542 | " )\n", 543 | " (output): BertSelfOutput(\n", 544 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 545 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 546 | " (dropout): Dropout(p=0.1, inplace=False)\n", 547 | " )\n", 548 | " )\n", 549 | " (intermediate): BertIntermediate(\n", 550 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 551 | " )\n", 552 | " (output): BertOutput(\n", 553 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 554 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 555 | " (dropout): Dropout(p=0.1, inplace=False)\n", 556 | " )\n", 557 | " )\n", 558 | " (8): BertLayer(\n", 559 | " (attention): BertAttention(\n", 560 | " (self): BertSelfAttention(\n", 561 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 562 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 563 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 564 | " (dropout): Dropout(p=0.1, inplace=False)\n", 565 | " )\n", 566 | " (output): BertSelfOutput(\n", 567 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 568 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 569 | " (dropout): Dropout(p=0.1, inplace=False)\n", 570 | " )\n", 571 | " )\n", 572 | " (intermediate): BertIntermediate(\n", 573 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 574 | " )\n", 575 | " (output): BertOutput(\n", 576 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 577 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 578 | " (dropout): Dropout(p=0.1, inplace=False)\n", 579 | " )\n", 580 | " )\n", 581 | " (9): BertLayer(\n", 582 | " (attention): BertAttention(\n", 583 | " (self): BertSelfAttention(\n", 584 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 585 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 586 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 587 | " (dropout): Dropout(p=0.1, inplace=False)\n", 588 | " )\n", 589 | " (output): BertSelfOutput(\n", 590 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 591 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 592 | " (dropout): Dropout(p=0.1, inplace=False)\n", 593 | " )\n", 594 | " )\n", 595 | " (intermediate): BertIntermediate(\n", 596 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 597 | " )\n", 598 | " (output): BertOutput(\n", 599 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 600 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 601 | " (dropout): Dropout(p=0.1, inplace=False)\n", 602 | " )\n", 603 | " )\n", 604 | " (10): BertLayer(\n", 605 | " (attention): BertAttention(\n", 606 | " (self): BertSelfAttention(\n", 607 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 608 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 609 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 610 | " (dropout): Dropout(p=0.1, inplace=False)\n", 611 | " )\n", 612 | " (output): BertSelfOutput(\n", 613 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 614 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 615 | " (dropout): Dropout(p=0.1, inplace=False)\n", 616 | " )\n", 617 | " )\n", 618 | " (intermediate): BertIntermediate(\n", 619 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 620 | " )\n", 621 | " (output): BertOutput(\n", 622 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 623 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 624 | " (dropout): Dropout(p=0.1, inplace=False)\n", 625 | " )\n", 626 | " )\n", 627 | " (11): BertLayer(\n", 628 | " (attention): BertAttention(\n", 629 | " (self): BertSelfAttention(\n", 630 | " (query): Linear(in_features=768, out_features=768, bias=True)\n", 631 | " (key): Linear(in_features=768, out_features=768, bias=True)\n", 632 | " (value): Linear(in_features=768, out_features=768, bias=True)\n", 633 | " (dropout): Dropout(p=0.1, inplace=False)\n", 634 | " )\n", 635 | " (output): BertSelfOutput(\n", 636 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 637 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 638 | " (dropout): Dropout(p=0.1, inplace=False)\n", 639 | " )\n", 640 | " )\n", 641 | " (intermediate): BertIntermediate(\n", 642 | " (dense): Linear(in_features=768, out_features=3072, bias=True)\n", 643 | " )\n", 644 | " (output): BertOutput(\n", 645 | " (dense): Linear(in_features=3072, out_features=768, bias=True)\n", 646 | " (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n", 647 | " (dropout): Dropout(p=0.1, inplace=False)\n", 648 | " )\n", 649 | " )\n", 650 | " )\n", 651 | " )\n", 652 | " (pooler): BertPooler(\n", 653 | " (dense): Linear(in_features=768, out_features=768, bias=True)\n", 654 | " (activation): Tanh()\n", 655 | " )\n", 656 | " )\n", 657 | " (dropout): Dropout(p=0.1, inplace=False)\n", 658 | " (classifier): Linear(in_features=768, out_features=22, bias=True)\n", 659 | ")" 660 | ] 661 | }, 662 | "metadata": { 663 | "tags": [] 664 | }, 665 | "execution_count": 15 666 | } 667 | ] 668 | }, 669 | { 670 | "cell_type": "markdown", 671 | "metadata": { 672 | "id": "B4quwVpCMm0T" 673 | }, 674 | "source": [ 675 | "### Option 2: Fine-tuned BERT" 676 | ] 677 | }, 678 | { 679 | "cell_type": "code", 680 | "metadata": { 681 | "id": "AJD4d1tvMCXh" 682 | }, 683 | "source": [ 684 | "#Bert \n", 685 | "modelpath = \"./bert_scigraph0.pt\" \n", 686 | "\n", 687 | "config = BertConfig(output_hidden_states=True, output_attentions=True, num_labels=22)\n", 688 | "model = BertForMultiLabelSequenceClassification(config=config)\n", 689 | "model.load_state_dict(torch.load(modelpath))\n", 690 | "\n", 691 | "bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, return_token_type_ids=True)\n", 692 | "model.eval()" 693 | ], 694 | "execution_count": null, 695 | "outputs": [] 696 | }, 697 | { 698 | "cell_type": "markdown", 699 | "metadata": { 700 | "id": "pCssnvX8M0aW" 701 | }, 702 | "source": [ 703 | "### Option 3: Fine-tuned BioBERT" 704 | ] 705 | }, 706 | { 707 | "cell_type": "code", 708 | "metadata": { 709 | "id": "mPCg78r4MCjw" 710 | }, 711 | "source": [ 712 | "#BioBert \n", 713 | "pretrainedModelpath = 'monologg/biobert_v1.1_pubmed'\n", 714 | "modelpath = \"./biobert11v2_scigraph0.pt\" #fine-tuned model\n", 715 | "\n", 716 | "config = AutoConfig.from_pretrained(pretrainedModelpath+'config.json', output_hidden_states=True, output_attentions=True, num_labels=22)#\n", 717 | "model = BioBertForMultiLabelSequenceClassification2(config=config)\n", 718 | "model.load_state_dict(torch.load(modelpath))\n", 719 | "model = model.bert\n", 720 | "\n", 721 | "bert_tokenizer = BertTokenizer.from_pretrained(pretrainedModelpath, do_lower_case=True, return_token_type_ids=True)\n", 722 | "model.eval()" 723 | ], 724 | "execution_count": null, 725 | "outputs": [] 726 | }, 727 | { 728 | "cell_type": "markdown", 729 | "metadata": { 730 | "id": "H0x4_7S7OOcG" 731 | }, 732 | "source": [ 733 | "## Visualize attention last layer" 734 | ] 735 | }, 736 | { 737 | "cell_type": "markdown", 738 | "metadata": { 739 | "id": "nOZu3yH_PWLj" 740 | }, 741 | "source": [ 742 | "### Tokenize a sentence" 743 | ] 744 | }, 745 | { 746 | "cell_type": "code", 747 | "metadata": { 748 | "id": "8yJCx40WOBG3" 749 | }, 750 | "source": [ 751 | "#sent=\"A universal long-term flu vaccine may not prevent severe epidemics\"\n", 752 | "sent =\"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding\"\n", 753 | "tokenized_sentence = bert_tokenizer.encode(sent, add_special_tokens=True)\n", 754 | "tokens = bert_tokenizer.tokenize(sent) #No special tokens" 755 | ], 756 | "execution_count": 16, 757 | "outputs": [] 758 | }, 759 | { 760 | "cell_type": "markdown", 761 | "metadata": { 762 | "id": "QB-sHA-PPaBj" 763 | }, 764 | "source": [ 765 | "### Process the sentence with the fine-tuned model " 766 | ] 767 | }, 768 | { 769 | "cell_type": "code", 770 | "metadata": { 771 | "id": "ziVV6jzgOn3I" 772 | }, 773 | "source": [ 774 | "input_ids = torch.tensor(tokenized_sentence).unsqueeze(0)\n", 775 | "outputs = model(input_ids)" 776 | ], 777 | "execution_count": 17, 778 | "outputs": [] 779 | }, 780 | { 781 | "cell_type": "markdown", 782 | "metadata": { 783 | "id": "L7elym9uPlwb" 784 | }, 785 | "source": [ 786 | "### Inspect the model attention " 787 | ] 788 | }, 789 | { 790 | "cell_type": "code", 791 | "metadata": { 792 | "id": "vj0Dj6ADOf7O", 793 | "outputId": "f0b28564-5e61-43e8-d0e2-727b747deb3a", 794 | "colab": { 795 | "base_uri": "https://localhost:8080/", 796 | "height": 388 797 | } 798 | }, 799 | "source": [ 800 | "fig, ax = plt.subplots()\n", 801 | "fig.patch.set_facecolor('xkcd:white')\n", 802 | "\n", 803 | "# for each layer, out has the attention heads\n", 804 | "for i, out in enumerate(outputs[-1]): \n", 805 | " # focus only on the last layer\n", 806 | " if i != 11:\n", 807 | " continue \n", 808 | " \n", 809 | " # Get attention matrices (one per head in the last layer)\n", 810 | " att = out[0]\n", 811 | " print(att.shape)\n", 812 | " matrix = torch.zeros(att[0].shape) \n", 813 | " for a in att:\n", 814 | " matrix = matrix + a\n", 815 | "\n", 816 | " # Average the attention of the 12 heads in the last layer\n", 817 | " matrix = matrix/12 \n", 818 | "\n", 819 | " #remove first and last row and column CLS and sept\n", 820 | " matrix = matrix.detach().numpy()\n", 821 | " matrix = np.delete(matrix, 0, axis=0)\n", 822 | " matrix = np.delete(matrix, -1, axis=0)\n", 823 | " matrix = np.delete(matrix, 0, axis=1)\n", 824 | " matrix = np.delete(matrix, -1, axis=1)\n", 825 | "\n", 826 | " print(\"Data Max: \",matrix.max())\n", 827 | " print(\"Data Min: \",matrix.min())\n", 828 | " print(\"Data Mean: \",matrix.mean())\n", 829 | "\n", 830 | " \n", 831 | " im = ax.imshow(matrix, cmap=\"YlGnBu_r\", vmin=0,vmax=0.2)\n", 832 | " #im = ax.imshow(matrix, cmap=\"YlGnBu_r\", vmin=0,vmax=matrix.max())\n", 833 | " # Create colorbar\n", 834 | " cbar_kw={}\n", 835 | " cbar = ax.figure.colorbar(im, ax=ax, **cbar_kw)\n", 836 | " cbar.ax.set_ylabel(\"Average Attention (Std)\", rotation=-90, va=\"bottom\")\n", 837 | " \n", 838 | " ax.set_xticks(np.arange(len(tokens)))\n", 839 | " ax.set_yticks(np.arange(len(tokens)))\n", 840 | " # ... and label them with the respective list entries\n", 841 | " ax.set_xticklabels(tokens)\n", 842 | " ax.set_yticklabels(tokens)\n", 843 | "\n", 844 | " # Rotate the tick labels and set their alignment.\n", 845 | " plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n", 846 | " rotation_mode=\"anchor\")\n", 847 | " #fig.tight_layout() \n", 848 | " copymatrix = matrix\n", 849 | " matrix = torch.zeros(att[0].shape)\n", 850 | "plt.show()" 851 | ], 852 | "execution_count": 18, 853 | "outputs": [ 854 | { 855 | "output_type": "stream", 856 | "text": [ 857 | "torch.Size([12, 14, 14])\n", 858 | "Data Max: 0.23208278\n", 859 | "Data Min: 0.0003950287\n", 860 | "Data Mean: 0.061727658\n" 861 | ], 862 | "name": "stdout" 863 | }, 864 | { 865 | "output_type": "display_data", 866 | "data": { 867 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEtCAYAAADnbHzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVxU5f7A8c8AimKouJUCpjiKiLIOgpHmjisumeKSGWVqlGWb9avQunXTm7fFLdRraabhUopmoWKuuSAo7hYqJJBXERV3kOH8/uBydESYEY4JzPd9X/O6zDnn+c4zYPOd55znPF+doigKQgghBGDzoDsghBCi/JCkIIQQQiVJQQghhEqSghBCCJUkBSGEECpJCkIIIVSSFIQQQkOxsbG4u7uj1+uZMmVKkf2fffYZrVq1wsvLiy5duvDnn3+q+xYuXEjz5s1p3rw5CxcuVLcnJibSpk0b9Ho948eP577eSaAIIYTQRF5enuLm5qacOHFCycnJUby8vJTDhw+bHPPrr78qV69eVRRFUWbPnq0MHjxYURRFycrKUpo2bapkZWUp58+fV5o2baqcP39eURRFCQgIUHbu3Knk5+crPXr0UH7++ef79h5kpCCEEBqJj49Hr9fj5uZG1apVCQsLIyYmxuSYTp064eDgAEBQUBDp6ekArFu3jm7dulGnTh2cnJzo1q0bsbGxnD59mkuXLhEUFIROp2PkyJGsWrXqvr0HSQpCCKGRjIwMXF1d1ecuLi5kZGQUe/z8+fPp2bNniW0zMjJwcXGxOGZZ2d23yAIb22rY2DmWOY5OZ6tBb8Cmdtn7AsCl69rEAXT1a2gSR7mWp0kcqmvzn4Tx7AVN4qDRuWO7Kg6axAFo3qKqJnH+e02b76TKuSzOnTtX6vY9erTn3DnL/l7XrytUr15dff7CCy/wwgsvlOp1v/vuOxISEtiyZUup2t8vkhTuIxs7Rx5yebLMcapW0ebD3KFvB03i6OKOaRIHoMq4tprEyduTqUkco1d9TeJc+jJakziKRkmh/sPemsQBWL3OxfxBFvg4SZt/10kT3ihT+3PnLrAn4QeLjg0wDCchIaHY/c7OzqSlpanP09PTcXZ2LnJcXFwcH3/8MVu2bMHe3l5tu3nzZpO2HTt2xNnZWT3FVFJMrcjpIyGEIN/CR8kCAgJITk4mJSWF3NxcoqOjCQ0NNTlm3759jBkzhtWrV9OgQQN1e0hICOvXr+fChQtcuHCB9evXExISQsOGDalZsya7du1CURS+/fZb+vXrp9UbL0JGCkIIq6fViMzOzo6ZM2cSEhKC0WgkPDwcT09PIiMjMRgMhIaG8uabb3LlyhWeeuopABo3bszq1aupU6cO77//PgEBAQBERkZSp04dAGbPns2oUaO4fv06PXv2VK9D3A+SFIQQVk5BsWAUYKlevXrRq1cvk20ffvih+nNcXFyxbcPDwwkPDy+y3WAwcOjQIc36WBKrPH2UmppK69atS91+1apVHDlyRMMeCSEeLG1OH1UGVpkUyiIvL0+SghCViELB6SNLHtbAapNCXl4ew4cPx8PDg0GDBnHt2jUSExN54okn8Pf3JyQkhNOnTwPQsWNHXn31VQwGA1OnTmX16tW8+eab+Pj4cOLEiQf8ToQQZaNY/D9rYLXXFH7//Xfmz59PcHAw4eHhzJo1i5UrVxITE0P9+vVZunQp7777Ll9//TUAubm56lS05ORk+vTpw6BBgx7kWxBCaMY6PvAtYbVJwdXVleDgYABGjBjBP//5Tw4dOkS3bt0AMBqNNGzYUD1+yJAhFsWdO3cuc+fOBSA//4bGvRZC3A9aXmiu6Kw2Keh0OpPnjo6OeHp6snPnzrseX6OGZXfe3n6Ho529NjdCCSHuJwVFkaRQyGqvKZw6dUpNAEuWLCEoKIjMzEx1282bNzl8+PBd2zo6OnL58uW/ra9CiPtNsfBR+VltUnB3d2fWrFl4eHhw4cIFXn75ZVasWMHEiRPx9vbGx8eHHTt23LVtWFgYn376Kb6+vnKhWYhKQCHfooc1sMrTR02aNOHYsaLr9/j4+LB169Yi229fjwQgODhYpqQKUalYxyjAElaZFIQQopA13YNgCUkKQgirp2B80F0oNyQpCCGsnrXcmGYJSQr3UT755OZdKXMcxxrarF9fRV9Tkzh5xa/ndc/unBr8oDVpqk1Boz03tZmdptNpNBfEqN2HXlNHd03iPNYgRZM4SWWOYD0ziywhSUEIYfXkmsItkhSEEMJKpptaQpKCEMLqyTWFWyQpCCGsmqJxkZ2KTpKCEMLqydpHt1jtMhdCCHGLdmsfxcbG4u7ujl6vZ8qUKUX2b926FT8/P+zs7FixYoW6fdOmTfj4+KiPatWqsWrVKgBGjRpF06ZN1X1JSWWfc1UcGSkIIaycdgV0jEYjERERbNiwARcXFwICAggNDaVVq1bqMY0bN2bBggVMmzbNpG2nTp3UD/vz58+j1+vp3r27uv/TTz/9W2q4SFIQQgiNkkJ8fDx6vR43NzegYPHMmJgYk6TQpEkTAGxsij9Rs2LFCnr27ImDg4Mm/boXcvqolHr16sVff/31oLshhCgjBchXjBY9zMnIyMDV1VV97uLiQkZGxj33KTo6mqFDh5pse/fdd/Hy8mLChAnk5OTcc0xLSVIopZ9//plGjRoV2T537lwMBgMGgwGMUnlNiPLP0usJCpmZmep/3waDQa2yqKXTp09z8OBBQkJC1G2ffPIJx44dY8+ePZw/f56pU6dq/rqF5PSRxm6vvGZjX/cB90YIYQlLb2iuX7++Wqv9bpydnUlLS1Ofp6en4+zsfE99WbZsGQMGDKBKlSrqtsLSwPb29jz77LNFrkdoSUYKQgirl2/hw5yAgACSk5NJSUkhNzeX6OhoQkND76kv33//fZFTR6dPnwYKluNYtWoVrVu3vqeY90KSQinJNQUhKg+tJqTa2dkxc+ZMQkJC8PDwYPDgwXh6ehIZGcnq1asB2LNnDy4uLixfvpwxY8bg6emptk9NTSUtLY0nnnjCJO7w4cNp06YNbdq04dy5c7z33nsavOti3sN9i1zJ/fzzzw+6C0IIDSiK5aePLNGrVy969eplsu3DDz9Ufw4ICCA9Pf2ubZs0aXLXC9O//vqrdh00Q5KCEMLK6chXytcS7g+SJAUhhNWT5fBukaQghLBqBfcpPOhelB+SFO4nRZuFtnI1quK1YcglTeL0/Y92d1neOJerSRwbe20qpl2/ps2ng1YLrGkW56Eq5g+yYpITbpGkIISwelJ47RZJCkIIqyc54RZJCkIIq6YARskKKkkKQgjrpvF9ChWdJAUhhNWTnHCLJAUhhNVT5OY1lax9ZCGj0fxa6kKIisfyhbOtgyQFChahatmyJcOHD8fDw4NBgwZx7do1mjRpwsSJE/Hz82P58uWsX7+edu3a4efnx1NPPcWVK1cedNeFEBowKpY9rIEkhf/5/fffefHFFzl69Cg1a9Zk9uzZANStW5e9e/fStWtXPvroI+Li4ti7dy8Gg4HPPvvsAfdaCKEFGSncItcU/sfV1ZXg4GAARowYwfTp0wEYMmQIALt27eLIkSPqMbm5ubRr165InLlz56rVmJR8qbwmRHmnILOPbidJ4X90Ot1dn9eoUQMoKG7RrVs3vv/++xLjmFReqyqV14SoCCQn3CKnj/7n1KlT7Ny5E4AlS5bw+OOPm+wPCgrit99+4/jx4wBcvXqVP/7442/vpxBCY0rBgniWPKyBJIX/cXd3Z9asWXh4eHDhwgXGjRtnsr9+/fosWLCAoUOH4uXlRbt27Th27NgD6q0QQktyTeEWOX30P3Z2dnz33Xcm21JTU02ed+7cmT179vyNvRJC3G+ydLYpSQpCCKsnN6/dIqePKKiLeujQoQfdDSHEA5Jv4cMSsbGxuLu7o9frmTJlSpH9W7duxc/PDzs7O1asWGGyz9bWFh8fH3x8fAgNDVW3p6SkEBgYiF6vZ8iQIeTmmq9D0qVLF+bOncuFCxcs7HkBSQpCCKtWOCXVkoc5RqORiIgIfvnlF44cOcL333/PkSNHTI5p3LgxCxYsYNiwYUXaV69enaSkJJKSkli9erW6feLEiUyYMIHjx4/j5OTE/PnzzfZl4sSJ7Nu3DxcXF/r27cuSJUu4evWq2XZy+qgC0Om0yd2Re2tpEkdpoF3ltaZe1TSJc+rEvX0bKo5Rm0Jnmv3NtKK7pE2FOy3Z6srPiXytehIfH49er8fNzQ2AsLAwYmJiaNWqlXpMkyZNALCxsezfiKIo/PrrryxZsgSAZ555hsmTJxeZDHOn7t270717d2JjY3n11VeJjo5m4sSJBAcHExYWRv/+/e/arnz9yxVCiAfA0pFCZmYmBoNBfRTeqFooIyMDV1dX9bmLiwsZGRkW9+PGjRsYDAaCgoJYtWoVAFlZWdSuXRs7O7tSxdTpdHTp0oV58+axadMmTp8+zcCBA4s9XkYKQgirpiiQZ+FQoX79+iQkJNy3vvz55584Oztz8uRJOnfuTJs2bahVq2wjfKPRyIwZM1i6dCnnzp1j6NChzJs3r9jjJSkIIayeVlNSnZ2dSUtLU5+np6fj7Ox8T+0B3Nzc6NixI/v27ePJJ5/k4sWL5OXlYWdnZ3HMGTNm8OOPP6LT6UhLS2PGjBn4+vqabSenj4QQVq3gPgWdRQ9zAgICSE5OJiUlhdzcXKKjo01mEZXkwoUL5OTkAHDu3Dl+++03WrVqhU6no1OnTupMpYULF9KvXz+z8Y4dO8Y//vEPUlNT+de//oWXlxfr1q0z206SghDC6mm1dLadnR0zZ84kJCQEDw8PBg8ejKenJ5GRkepsoj179uDi4sLy5csZM2YMnp6eABw9ehSDwYC3tzedOnXi7bffVi9QT506lc8++wy9Xk9WVhbPPfec2b7MmjWLxx57jE2bNjF27FiaNWtm0awlOX0khLBqWt/R3KtXL3r16mWy7cMPP1R/DggIID09vUi7xx57jIMHD941ppubG/Hx8ffUj5dffpmffvoJPz8/hgwZwueff0716tXNtpOkIISwbgrkVcI7mhcuXEiPHj0YPXo0nTt3xtbW1qJ2cvpICGHVFCpn5bWzZ88yePBg5syZg5ubGy+++CJbtmwx206SghDC6lXGpbOrVavGoEGDWLFiBUeOHKF9+/ZMmzbNbDs5faQxqbwmRMVSmVdJPXfuHLt27UKn09G9e3eGDh1qto2MFDT2wgsvkJCQQEJCAjobbZZwEELcX5VxpLB582YMBgPff/89o0aNYuDAgaxfv95sO0kKpTBr1ix1JcO//vrrQXdHCFEGiqLDaOGjInnrrbfYuHEjixcvxtXVldjYWCZPnmy2nZw+KoWIiAgiIiIedDeEEBpQgDyNFkIsT3JycmjWrBlQsKhe9erVLVpyW0YKQgirp2U9hfJCp9Nx7do1AG7evMm//vUvNUmURJKCEMKqFUxJrXynjz766CN1NdXHHnuM3NxcvvnmG7Pt5PSREMLqVbSLyJbo06eP+vOdS3yXRJKCEMKqKRVwZpEl3NzcUO5SLi4lJaXEdpIUhBBWrzImhdvrPly9elWtp2COJIX7SkG7Qn9ldy5Ho0tIN/K0iQP41tWmTOSf1bX5p1zDQZvzxopi1CSOZqpZtu6NNVKAm/kV63qBJerUqWPy8xtvvIHBYDDbTpKCEMLqVcaRwt289dZb5Ofnl1gfWmYfCSGsmgIYLXxUJHPmzKFBgwa4ubmxdetWLly4wKVLl0pMCCBJQQhh7Sxc4qKijSamTp3KkSNHWL16Ne+99x5OTk4WzUKS00dCCKtWWI6zsmnQoAG1atWiXr16XLx4ESi4ic0cSQpCCKtWcKH5QfdCe4GBgfTu3Zvhw4dz9epV3n//ffR6vdl2FeL00cWLF5k9e/Y9t+vVq5eaIYsTGRlJXFxcabsmhKgEtDx9FBsbi7u7O3q9nilTphTZv3XrVvz8/LCzs2PFihXq9qSkJNq1a4enpydeXl4sXbpU3Tdq1CiaNm2qLsSZlJRkth+XL1/GxcWFLVu2EBISwiOPPMKiRYvMtqsQI4XCpPDiiy+abM/Ly8POrvi38PPPP5uNfXvtVCGE9dHy5jWj0UhERAQbNmzAxcWFgIAAQkNDadWqlXpM48aNWbBgQZGCNw4ODnz77bc0b96cv/76C39/f0JCQqhduzYAn376KYMGDbK4L19//XWp3kOFGCm8/fbbnDhxAh8fHwICAmjfvr3JL7p///74+/vj6elpciGlSZMmnDt3jtTUVDw8PBg9ejSenp50796d69evAwUZuDBbN2nShEmTJuHn50ebNm04duwYAJmZmXTr1g1PT0+ef/55Hn30UYtuAhFCVAz5is6ihznx8fHo9Xrc3NyoWrUqYWFhxMTEmBzTpEkTvLy8iswCatGiBc2bNwegUaNGNGjQgMzMzHt+L9u3b+f69escP36cvn37Ur9+ferVq0ffvn05efKk2fYVIilMmTKFZs2akZSUxKeffsrevXv58ssv+eOPP4CCjJiYmEhCQgLTp08nKyurSIzk5GQiIiI4fPgwtWvX5ocffrjra9WrV4+9e/cybtw4NZN/8MEHdO7cmcOHDzNo0CBOnTpVbF/nzp2LwWDAYDCg5Odo8O6FEPfTvUxJzczMVP/7NhgMRWbzZGRk4Orqqj53cXFRF6W7F/Hx8eTm5pqsavruu+/i5eXFhAkTyMkp/rPlpZdeonr16owYMYKnn36aM2fOcPbsWZ5++mmGDx9u9rUrRFK4U9u2bWnatKn6fPr06Xh7exMUFERaWhrJyclF2hSejwPw9/cnNTX1rrEHDhxY5Jjt27cTFhYGQI8ePXByciq2b6aV1+xL8/aEEH8zS68p1K9fX/3vOyEhgRdeeEHzvpw+fZqnn36ab775Rh1NfPLJJxw7dow9e/Zw/vx5pk6dWmz7wvWObty4weDBg7GxscHGxobBgwerS2mXpEImhRo1aqg/b968mbi4OHbu3Mn+/fvx9fXlxo2itZHt7W99QNva2pKXd/elGgqPK+kYIUTloaAjL9+yhznOzs6kpaWpz9PT03F2dra4L5cuXaJ37958/PHHBAUFqdsbNmyITqfD3t6eZ599lvj4+GJjODk5sXTpUkJCQpg7dy6XLl3i8uXLzJkzh9DQULN9qBBJwdHRkcuXL991X3Z2Nk5OTjg4OHDs2DF27dql+esHBwezbNkyANavX8+FCxc0fw0hxIOjVZGdgIAAkpOTSUlJITc3l+joaIs+iAFyc3MZMGAAI0eOLHJB+fTp00DBKGDVqlW0bt262Dhff/010dHRLFmyhDfeeAMXFxecnZ158803+eKLL8z2o0LMPqpbty7BwcG0bt2a6tWr8/DDD6v7evToQVRUFB4eHri7u5tkV61MmjSJoUOHsmjRItq1a8cjjzyCo6Oj5q8jhPj7aTn7yM7OjpkzZxISEoLRaCQ8PBxPT08iIyMxGAyEhoayZ88eBgwYwIULF1izZg2TJk3i8OHDLFu2jK1bt5KVlcWCBQsAWLBgAT4+PgwfPpzMzEwURcHHx4eoqKhi++Dm5sbKlStL/R50yt0W3BYmcnJysLW1xc7Ojp07dzJu3DiL5gnbVK2D/SMhZX79Wg81KXMMgFb/7q1JnFPTjmoSB6Dvx+bLA1pi1eJsTeK4dquvSZw9L83SJI5WXB59QrNYyds6axJn4R/mZ8JYYsaw90yWib5Xj3i4M3LhVxYd++uLb5Xptf5uGzZsYN26deh0Orp37063bt3MtqkQI4UH7dSpUwwePJj8/HyqVq3KvHnzHnSXhBAaqmjrGllixowZLFmyhPDwcD755BMuXbrE/v37eeONN0psJ0nBAs2bN2ffvn0PuhtCiPtAwbLrBRXNvHnz2LlzJzVq1OCrr75izpw5tG3bVpKCEEKYY8nMooqocKamoigoikJurvmiVpIU7iedDp2u7BWvFEWb7zFXr2gzRtZd0qZaGsBvp6poEqdWUD1N4jxUTaPfkQZ/d02ZWUP/Xhw8/4cmcWx05ePjp7KOFBwdHfnrr79o1KgRV69eJTQ0lAEDBphtVz7+KkII8aBUwFoJlvjuu++oUqXgS9c777xD8+bNefzxx822k6QghLBqBfUUHnQvtHf7qg/PPvusxe0kKQghrF5lLLJTWpIUhBBWTQHyKuFIobQqxDIXQghxP2m1dHZ5Eh4ezvr168nPv7fL6JIUSmH69Ol4eHhYtAytEKJ8UyxcIbWiXXcICgri888/x8XFhZdffpkdO3ZY1E6SQinMnj2bDRs2sHjx4gfdFSGEBrRaEK88eeGFF/jll1+oUqUKPj4+TJo0iWbNmvH222+XuEyPJAUzPvvsM1q3bk3r1q354osvGDt2LCdPnqRnz558/vnnD7p7QggNVMaRQiFbW1uee+45NmzYoJYZ8PPzK/Z4udBcgsTERL755ht2796NoigEBgby3XffERsby6ZNm6hXr+gNU3Pnzr1VjclYtK6DEKJ8Uah41wvuRX5+PmvWrGHp0qVs2rSJ7t278+677xZ7vCSFEmzfvp0BAwaot4oPHDiQbdu2ldjmhRdeUKsx2djXve99FEKUTWWdfbRmzRp+/PFHzp8/z7fffsuwYcP4+uuvqVq1aontJCkIIayeUglHClFRUYSFhTFz5kyTapXmyDWFErRv355Vq1Zx7do1rl69ysqVK2nfvv2D7pYQQkuVdPbR6tWr8fT0JCEhgS1btrBlyxbatWvH5s2b+fPPP4ttJyOFEvj5+TFq1Cjatm0LwPPPP4+vr+8D7pUQQkuVdUG87t27YzQaqVmzprrt2LFj/Pvf/2bYsGE8+uijd20nScGM1157jddee81kW2pq6oPpjBDivqhoowBLnD9/vkgdGD8/P9asWVNiOzl9JISwagUXmnUWPSwRGxuLu7s7er2eKVOmFNm/detW/Pz8sLOzY8WKFSb7Fi5cSPPmzWnevDkLFy5UtycmJtKmTRv0ej3jx4/HkirKo0aNKrItLCzMbDsZKQghrJ5WIwWj0UhERAQbNmzAxcWFgIAAQkNDadWqlXpM48aNWbBgAdOmTTNpe/78eT744AMSEhLQ6XT4+/sTGhqKk5MT48aNY968eQQGBtKrVy9iY2Pp2bNniX0ZOXIkP/74I5cvX1a3zZo1i4cffhgfHx+8vb3v2k6SghDCqhUsna3N7KP4+Hj0ej1ubm5AwTfzmJgYk6TQpEkTAGzuKHy0bt06unXrRp06dQDo1q0bsbGxdOzYkUuXLhEUFAQUfNivWrXKbFIICQmhVatWJtcUbty4QUJCAg0aNJCk8EAoCopiLHMYnU6bs3wXL2r0dUijSnAAZ5MuaRJHdz1PkzhPDHbUJM4mDf7uBTSaKqnhieI2dVpoEifzRrImccrsHmYWZWZmYjAY1Oe335cEkJGRgaurq/rcxcWF3bt3WxT7bm0zMjLIyMjAxcWlyHZz8vLyWLBggcm27du3M2PGjBLbSVIQQli1eymyU79+fRISEu5rf7TyzjvvWLTtTnKhWQhh9RTFsoc5zs7OpKWlqc/T09Nxdna2qA/FtXV2diY9Pf2eY4aGhvLee+8REBBAQEAA7777Lr179zbbTpKCEMLq5edb9jAnICCA5ORkUlJSyM3NJTo6mtDQUIv6EBISwvr167lw4QIXLlxg/fr1hISE0LBhQ2rWrMmuXbtQFIVvv/2Wfv36mY33yiuvkJ2dzZIlS8jJycHT05OXX37ZbDtJCkII66aAoigWPcyxs7Nj5syZhISE4OHhweDBg/H09CQyMpLVq1cDsGfPHlxcXFi+fDljxozB09MTgDp16vD++++r3+wjIyPVi86zZ8/m+eefR6/X06xZM7MXmQF27NjBjBkzaN68Oba2tgwbNozDhw+bfw9mjxBCiEpMwbJTQ5bq1asXvXr1Mtn24Ycfqj8HBASYnA66XXh4OOHh4UW2GwwGDh06dE/9uLPiWnZ2Njdv3jTbzupGCpMnTy4yP1gIYd2UfMseFYmbm5taTOfixYu0bduWyZMnm20nIwUhhNXTcqRQXhSeroKCu6wbN25M9erVzbaziqTw8ccfs3DhQho0aICrqyv+/v6cOHGCiIgIMjMzcXBwYN68ebRs2ZLMzEzGjh3LqVOnAPjiiy8IDg5m8uTJnDhxguPHj3Pu3DneeustRo8e/YDfmRCizCrgCqiWuH2ZDIBdu3YB8Mwzz5TYrtInhcTERKKjo0lKSiIvLw8/Pz/8/f154YUXiIqKonnz5uzevZsXX3yRX3/9lVdeeYUJEybw+OOPc+rUKUJCQjh69CgABw4cYNeuXVy9ehVfX1969+5No0aNTF7v9sprSn7O3/5+hRD3RsGymUUVTWJiovpzTk4OGzduxNvbW5LCtm3bGDBgAA4ODkDB3N0bN26wY8cOnnrqKfW4nJyCD/C4uDiOHDmibr906RJXrlwBoF+/flSvXp3q1avTqVMn4uPj6d+/v8nrmVReq1rnvr43IYQ2KuPpo+nTp5s8v3LlislnXnEqfVK4m/z8fGrXrq1ehLlz365du6hWrVqRfTqdrsTnQogKSAGlMp4/ukOVKlU4efKk2eMq/eyjDh06sGrVKq5fv87ly5dZs2YNDg4ONG3alOXLlwMFc5T3798PFBSmuH1tkNsTR0xMDDdu3CArK4vNmzcTEBDw974ZIcR9odUdzeVJaGgoffv2pW/fvvTq1Qt3d3cGDBhgtl2lHyn4+fkxZMgQvL29adCggfpBvnjxYsaNG8dHH33EzZs3CQsLw9vbm+nTpxMREYGXlxd5eXl06NCBqKgoALy8vOjUqRPnzp3j/fffL3I9QQhR8dzLfQoV6dzAG2+8of5sZ2fHo48+atHyGJU+KQC8++67vPvuu0W2x8bGFtlWr149li5detc4Xl5efPvtt5r3TwjxYBktvNBckU6tdOjQoci2d955h08++aTEdhXpPQohhPYsPHVU0U4fvfPOO9StWxdHR0dq1qxJzZo1+fTTT3F0dCwxMVjFSEELltwJKISoeLRe5qK8iImJ4cyZM9jZ3fqY9/PzY+/evSW2k6QghLB6FW0JC0v4+/ubJGScgtcAACAASURBVARAXXyvJJIU7iOdzpYqdjXKHOdm3jUNegPXr2hTDcwuP1eTOAD2TbWpdHZz5381iRP3u5Mmceyr1NIkjoJGn1bXtKlMB3DuxjFN4sT8+ZAmccrsf6ukVjaLFi2yaNud5JqCEMLqVcZrCjVr1jS5nlD4HODxxx8vtp2MFIQQVk3B8tlHFcmlS8XXP9++fXux+yQpCCGsXkUbBVhq5cqVbNu2DYDg4GCefPJJs20kKQghrJtSOS80T5w4kQMHDjBkyBCgYLHOXbt28emnn5bYTpKCEMLqVcaRws8//8z+/fuxsSm4dDxq1CjatGljNiloeqE5NTWV1q1bF9n+/PPPm6w8WmjBggW89NJLAERFRWlyt3BqaipLlixRnyckJDB+/Pgyx71TkyZNOHfunOZxhRB/r8L7FLS60BwbG4u7uzt6vZ4pU6YU2Z+Tk8OQIUPQ6/UEBgaSmpoKFCy94+Pjoz5sbGzUtdc6duyIu7u7uu/s2bNm+2FjY0NWVpb6/OzZs2qCKMnfMlL4z3/+Y/aYsWPH3nV7Xl5ekbm2JSlMCsOGDQMKapsaDAaL2wshrI2i2ZRUo9FIREQEGzZswMXFhYCAAEJDQ2nVqpV6zPz583FycuL48eNER0czceJEli5dyvDhwxk+fDgABw8epH///vj4+KjtFi9efE+fZe+//z5t27alffv26HQ6Nm/ebHaUAPdhSmpeXh7Dhw/Hw8ODQYMGce3aNTp27EhCQgIA33zzDS1atKBt27b89ttvarvbayd37NiRV199FYPBwJdffkliYiJPPPEE/v7+hISEcPr0aQCOHz9O165d8fb2xs/PjxMnTvD222+zbds2fHx8+Pzzz9m8eTN9+vQB4Pz58/Tv3x8vLy+CgoI4cOCA+trh4eF07NgRNzc3k3XI+/fvj7+/P56enmrxHCFEJaIUFNmx5GFOfHw8er0eNzc3qlatSlhYGDExMSbHxMTEqIVuBg0axMaNG4skpe+//56wsLAyva1Bgwaxe/duBg8ezFNPPUV8fDyDBw82207zpPD777/z4osvcvToUWrWrMns2bPVfadPn2bSpEn89ttvbN++/a6nlArl5uaqp35efvllVqxYQWJiIuHh4eridsOHDyciIoL9+/ezY8cOGjZsyJQpU2jfvj1JSUlMmDDBJOakSZPw9fXlwIED/POf/2TkyJHqvmPHjrFu3Tri4+P54IMPuHnzJgBff/01iYmJJCQkMH36dJPhmBCi4iusvKZFUsjIyMDV1VV97uLiQkZGRrHH2NnZUatWrSKfK0uXLmXo0KEm25599ll8fHz4xz/+YdHIZsuWLRw9ehRHR0ccHR05duwYW7ZsMdtO89NHrq6uBAcHAzBixAiTb927d++mY8eO1K9fH4AhQ4bwxx9/3DVO4RXz33//nUOHDtGtWzegYHjWsGFDLl++TEZGhro++N2K4txp+/bt/PDDDwB07tyZrKwsdS5v7969sbe3x97engYNGnDmzBlcXFyYPn06K1euBCAtLY3k5GTq1q1b7GuYlOM0XjfbJyFEOWDh2aPMzEyTUzi3V1rUyu7du3FwcDC5Prt48WKcnZ25fPkyTz75JIsWLTL5Uns3//73v9Wfc3JyiI+Px8fHh02bNpXYTvOkoFV1sho1CpaHUBQFT09Pdu7cabL/8uXLpetgMezt7dWfbW1tycvLY/PmzcTFxbFz504cHBzo2LEjN27cKDHO7f9IbO3ra9pHIcR9oAAWVl6rX7++eir8bpydnUlLS1Ofp6enF6lhUHiMi4sLeXl5ZGdnm3zRjI6OLjJKKIzh6OjIsGHDiI+PN5sUVq9ebfL89OnTFk260fz00alTp9QP8CVLlpjcTh0YGMiWLVvIysri5s2bauWzkri7u5OZmanGvHnzJocPH8bR0REXFxdWrVoFFGTCa9eu4ejoWGzCaN++PYsXLwZg8+bN1KtXj5o1axb72tnZ2Tg5OeHg4MCxY8fYtWuXZb8EIUTFotH0o4CAAJKTk0lJSSE3N5fo6GhCQ0NNjgkNDWXhwoUArFixgs6dO6tfnvPz81m2bJnJ9YS8vDx1puPNmzf56aef7jrL05xHHnmEQ4cOmT1O85GCu7s7s2bNIjw8nFatWjFu3DjWrFkDQMOGDZk8eTLt2rWjdu3aJlfWi1O1alVWrFjB+PHjyc7OJi8vj1dffRVPT08WLVrEmDFjiIyMpEqVKixfvhwvLy9sbW3x9vZm1KhR+Pr6qrEKLyh7eXnh4OCg/mGK06NHD6KiovDw8MDd3Z2goKCy/XKEEOWOTgGdUZvZR3Z2dsycOZOQkBCMRiPh4eF4enoSGRmJwWAgNDSU5557jqeffhq9Xk+dOnWIjo5W22/duhVXV1fc3NzUbTk5OYSEhHDz5k2MRiNdu3Zl9OjRZvsyfvx49dqD0WgkKSnJotlLOqUyLg9YTtja16eGc78yx9FipVWAGuP6axLHblHJ67HfiyrjtKlzrdUqqdU7PKJJnIx/LDF/kAW0WiW1QT3zX8AstXOLiyZx/rFPm1VSf3v5rRJP6Zhj7+LOIy/PNn8gUH/5xDK91t/p9vu+7OzsaNKkCY899pjZdnJHsxDCyikWX1OoSMxdcyiOJAUhhHWrrKXXSkmSghBCVMIF8UpLksJ9ptOVfYJXtaraVPGyPXlRkzg2Ou3+2VR3KN2U5Tvl1K2uSZzLl7T5xmhjY6tJHNAmjvJQFU3iACSe0+bv/+fV8vPxo5ORgqr8/FWEEOJBUIC8yjdU2L59O2vXruXo0aNcv36dhg0bqjUV6tSpU2w7SQpCCCtXAWttlmDPnj28/vrrPProo3Tq1ImePXtSrVo1MjMzSUxMpE+fPgQGBvL555/ftb0kBSGEdVOoVNcUqlevzk8//XTXG3N79+5NZGSklOMUQogSVaKRQuHdzleuXEGn06lLBgFcuHABR0dHk5Um7qT5MhdCCFHh5CuWPSqQIUOGmNxoN3r0aDw8PHj00Uf56aefim1X5qRw8eJFk+Wx75ehQ4fi5eVV7HkwIYQoFQV0+YpFj4rk5MmTPPHEEwAcPnyY1atXk5qaSkJCApGRkcW2K/Ppo8Kk8OKLL5psv9eKaSX573//y549ezh+/LjFbbR8fShYO8TWVqtphkKIckWjtY/Kk+rVb03TjouLo0uXLlSrVo2GDRuSX0JxiDKPFN5++21OnDiBj48PAQEBtG/f3qT8XHGVyx566CHeffddvL29CQoK4syZMwAsX76c1q1b4+3tTYcOHQDo3r07GRkZ+Pj4sG3bNpKSkggKCsLLy4sBAwZw4cIFoGjFto4dOzJhwgQMBgMeHh7s2bOHgQMH0rx5c9577z21L9999x1t27bFx8eHMWPGYDQa1T6+/vrreHt7s3PnTt5++21atWqFl5cXb7zxRll/dUKIcsHCFVIr2HWHGjVqEBcXR1ZWFt9++61agfLmzZvk5eUV267MSWHKlCk0a9aMpKQkPv30U/bu3cuXX36pFs8prnLZ1atXCQoKYv/+/XTo0IF58+YB8OGHH7Ju3Tr279+vrge+evVq9TXat2/PyJEjmTp1KgcOHKBNmzZ88MEHan8KK7a9/vrrQMEqqwkJCYwdO5Z+/foxa9YsDh06xIIFC8jKyuLo0aMsXbqU3377jaSkJGxtbdXlta9evUpgYCD79+/Hw8ODlStXcvjwYQ4cOGCSVIQQFVhhPYVKdk3hyy+/5LXXXsPNzQ1vb2+1RkNqaiqvvPJKse00n33Utm1bmjZtqj4vrnJZ1apV1czl7+/Phg0bAAgODmbUqFEMHjyYgQMHFomfnZ3NxYsX1XNlzzzzDE899ZS6v7BiW6HCtczbtGmDp6cnDRs2BMDNzY20tDS2b99OYmIiAQEFq3Vev36dBg0aAAXFdp588kkAatWqRbVq1Xjuuefo06eP2vc7mVReyy+5II8QopyoRFNSC/n5+al16Avt3LmTdu3a0bx582LbaZ4Ubp/+VFLlsipVqqiFJQornQFERUWxe/du1q5di7+/P4mJiaV+fbhVUc3GxsakupqNjQ15eXkoisIzzzzDJ598UiRWtWrV1OsIdnZ2xMfHs3HjRlasWMHMmTP59ddfi7SRymtCVDyVcZmLHTt2EB0dbVJ0bPXq1YSGhtK/f3/69bv7sv5lTgolVTorTeWyEydOEBgYSGBgIL/88gtpaWnUrl1b3V+rVi2cnJzYtm0b7du3Z9GiReqooTS6dOlCv379mDBhAg0aNOD8+fNcvnyZRx991OS4K1eucO3aNXr16kVwcLBJEQwhRAWmAMbKN1QYPXo0b775pslNbFu3bqVPnz64u7sX267MSaFu3boEBwfTunVrqlevzsMPP6zuK03lsjfffJPk5GQURaFLly54e3vz559/mhyzcOFCxo4dy7Vr13Bzc+Obb74pdf9btWrFRx99RPfu3cnPz6dKlSrMmjWrSFK4fPky/fr148aNGyiKwmeffVbq1xRClDOVLydQrVo1Ro0aZbLto48+Uk+JF0cqr91Htvb1echlQJnjPFRdm2pgVTv5mj/IArY7TmkSB+Ch1/w0iXMxSZsVYPNdi6/ZfS+uzFimSRyt1GmsXeW1WQudzR9kga+OaVN5Lf3t18pWea1+M5wH/suiY+skflJhKq+lpKTQtGlTsrOzgYKzLIXbSiJ3NAshhIZTUmNjY3F3d0ev1zNlypQi+3NychgyZAh6vZ7AwEBSU1OBgllB1atXx8fHBx8fH8aOHau2SUxMpE2bNuj1epPayyW5cOECPj4+eHt7q4/z58+bbSdJQQghNJqSajQaiYiI4JdffuHIkSN8//33HDlyxOSY+fPn4+TkxPHjx5kwYQITJ05U9xVOvU9KSiIqKkrdPm7cOObNm0dycjLJycnExsaa7Ut4eDizZ88mNTWV1NRUZs+ezXPPPWe2nSQFIYR1U0Bn4cOc+Ph49Ho9bm5uVK1albCwMGJiYkyOiYmJ4ZlnngFg0KBBbNy4scRv/qdPn+bSpUsEBQWh0+kYOXIkq1atMtsXGxsbHnvsMfV5cHCwOuOzJLJK6n2kkM/NvGtlj6NocxXM3ruuJnFu/paqSRyAS5e0eW+2Z65qEqdxeydN4iTkZmsSR4vKfQC6wS00iQPQpo427+1Vz7vPWrxXmqwtoFGRnYyMDFxdXdXnLi4u7N69u9hj7OzsqFWrlnpTb0pKCr6+vtSsWZOPPvqI9u3bk5GRgYuLi0nMjIwMs33p3r07b7/9NsOHDwdg0aJFdOvWTZ24c+dkmkKSFIQQ1k3538MCmZmZGAwG9fnt9yWVVcOGDTl16hR169YlMTGR/v37c/jw4VLH+/nnn03+v1BsbCyKonDw4MG7tpOkIISwcpZfRK5fv36Js4+cnZ1JS0tTn6enp+Ps7HzXY1xcXMjLyyM7O5u6deui0+nUG2z9/f1p1qwZf/zxB87OzqSnp5cY827uvJvZUnJNQQghNLrQHBAQQHJyMikpKeTm5hIdHa0utVMoNDSUhQsXArBixQo6d+6MTqcjMzNTXYzz5MmTJCcn4+bmRsOGDalZsya7du1CURS+/fbbYu9GBjh+/Di5ubkl9vPQoUPF7pORghBCaHS3lp2dHTNnziQkJASj0Uh4eDienp5ERkZiMBgIDQ3lueee4+mnn0av11OnTh2io6OBgruNIyMjqVKlCjY2NkRFRVGnTh0AZs+ezahRo7h+/To9e/akZ8+exfbhr7/+YuTIkfj7+9OpUydatGhBtWrVOHv2LAkJCfz00084OzsXe9OvJAUhhFXTKaDTcJmLXr160atXL5NtH374ofpztWrVWL58eZF2Tz75ZLF3GxsMhhK/3d+uQ4cObN++nV9++YW1a9fyxRdfcP36dRo1akT79u35+uuvTS5c30mSwh2mT5/OV199hZ+fn7qEthCikqtk6zrY2NjQu3dvevfufc9tJSncYfbs2cTFxZWYSQtpXd1NCPGAyGo/KvlEu83YsWM5efIkPXv2ZNSoUWzbto2TJ0/i4ODA3Llz8fLyYvLkyZw4cYKTJ0/SuHFjvv/++wfdbSFEWShUygXxSktmH90mKiqKRo0asWnTJlJTU/H19eXAgQP885//ZOTIkepxR44cIS4uThKCEJVC5SzHWVoyUijG9u3b+eGHHwDo3LkzWVlZXLp0CSiYUnZ7Uezb3V55DaNUXhOiQrCSD3xLSFIohTuru93u9jscbey1WVZCCHEfKYBRkkIhOX1UjPbt26uzjzZv3ky9evVMKhgJISoROX2kkpFCMSZPnkx4eDheXl44ODiodyAKISoh6/i8t4gkhTsUFrwA7ro87eTJk/++zggh7j/FsiUsrIUkBSGE1dNZyakhS0hSEEIIyQkqSQpCCOsms49MSFIQQgg5faSSpHA/KdqU0tSqJGPDRtrESX/IQZM4ALlntbnBz+ahqprE8a1f8jr0ltqjUQlVrUqxKitPaBIHoNGYxzWJc/D8cU3iaEIuNKskKQghrNs9lOO0BpIUhBBWznpuTLOEJAUhhJALzSpJCkII66bIfQq3KzdrHz300EMPugtCCGuVb+HDArGxsbi7u6PX65kyZUqR/Tk5OQwZMgS9Xk9gYKC6isKGDRvw9/enTZs2+Pv78+uvv6ptOnbsiLu7Oz4+Pvj4+HD27NkyvNmSyUhBCCE0GikYjUYiIiLYsGEDLi4uBAQEEBoaSqtWrdRj5s+fj5OTE8ePHyc6OpqJEyeydOlS6tWrx5o1a2jUqBGHDh0iJCSEjIwMtd3ixYsxGAya9LMk5WakUOjKlSt06dIFPz8/2rRpQ0xMDFCwJpGHhwejR4/G09OT7t27c/36dQD27NmDl5cXPj4+vPnmm7Ru3RqABQsW8NJLL6mx+/Tpw+bNmwEYN24cBoMBT09PJk2apB7z888/07JlS/z9/Rk/fjx9+vQB4OrVq4SHh9O2bVt8fX3VfgkhKgGNVkmNj49Hr9fj5uZG1apVCQsLK/JZERMTwzPPPAPAoEGD2LhxI4qi4OvrS6NGjQDw9PTk+vXr5OTkaP9ezSh3SaFatWqsXLmSvXv3smnTJl5//XWU//0xkpOTiYiI4PDhw9SuXVstgvPss88yZ84ckpKSsLW1teh1Pv74YxISEjhw4ABbtmzhwIED3LhxgzFjxvDLL7+QmJhIZmamyfGdO3cmPj6eTZs28eabb3L16lXtfwFCiL9XYTlODU4fZWRk4Orqqj53cXEx+bZ/5zF2dnbUqlWLrKwsk2N++OEH/Pz8sLe3V7c9++yz+Pj48I9//EP9TLwfyl1SUBSF//u//8PLy4uuXbuSkZHBmTNnAGjatCk+Pj4A+Pv7k5qaysWLF7l8+TLt2rUDYNiwYRa9zrJly/Dz88PX15fDhw9z5MgRjh07hpubG02bNgVg6NCh6vHr169nypQp+Pj40LFjR27cuMGpU6eKxJ07dy4GgwGDwYCSL5XXhCj/FJR8o0WPzMxM9b9vg8Fwq8qihg4fPszEiROZM2eOum3x4sUcPHiQbdu2sW3bNhYtWqT56xYqd9cUFi9eTGZmJomJiVSpUoUmTZpw40bBh+vtWdPW1lY9fVQcOzs78vNvpffCOCkpKUybNo09e/bg5OTEqFGj1H3FURSFH374AXd39xKPM6m8VlUqrwlRESgWXkWuX78+CQkJxe53dnYmLS1NfZ6eno6zs/Ndj3FxcSEvL4/s7Gzq1q2rHj9gwAC+/fZbmjVrZtIGwNHRkWHDhhEfH29SN15L5W6kkJ2dTYMGDahSpQqbNm3izz//LPH42rVr4+joyO7duwGIjo5W9zVp0oSkpCTy8/NJS0sjPj4egEuXLlGjRg1q1arFmTNn+OWXXwBwd3fn5MmT6myApUuXqrFCQkKYMWOGOmzbt2+fZu9ZCPHgKCgoimUPcwICAkhOTiYlJYXc3Fyio6MJDQ01OSY0NFQt2rVixQo6d+6MTqfj4sWL9O7dmylTphAcHKwen5eXx7lz5wC4efMmP/30k3rd9H4odyOF4cOH07dvX9q0aYPBYKBly5Zm28yfP5/Ro0djY2PDE088Qa1atQAIDg6madOmtGrVCg8PD/z8/ADw9vbG19eXli1b4urqqv4BqlevzuzZs+nRowc1atQgICBAfY3333+fV199FS8vL/Lz82natCk//fTTffgNCCH+VveyRpmu5N12dnbMnDmTkJAQjEYj4eHheHp6EhkZicFgIDQ0lOeee46nn34avV5PnTp11C+yM2fO5Pjx43z44Yd8+OGHQMFp6xo1ahASEsLNmzcxGo107dqV0aNHl+Udl/wWlft5xeJvcuXKFfU+hylTpnD69Gm+/PLLMsVSFIWIiAiaN2/OhAkTShXLpmpd7B8JKVXb29WpqS9zDIAW/+quSZz0KYc1iQOQ26WJJnFszmhz0T/06dqaxJk75D+axNGKc6uumsU6HqvNgnjr0rVZEO/d/pElntIxx97BFWf3Vyw6to5tdJleqyIod6ePSmPt2rX4+PjQunVrtm3bxnvvvVfqWPPmzcPHxwdPT0+ys7MZM2aMhj0VQpQ/Cgr5Fj2sQbk7fVQaQ4YMYciQIZrEmjBhQqlHBkKIikghPz/PskMtm/FeoVWKpCCEEKVVsHK2dYwCLCFJQQhh9bQqZlQZSFK4r7Sp3pFn1OZW9+xsjeYUXLdwqG0B3Q1tYulyjZrEuWY0M73EYuVr/oaiUWU6LbnX0uZvVmYWTje1FpIUhBBWT0YKt0hSEEJYNQWFfEW70W9FJ0lBCGH1ZKRwiyQFIYSVK1wmVYAkBSGEkJHCbe77Hc0dO3bU5LbwzZs3s2PHDg16VGDy5MlMmzYNgMjISOLi4jSLLYSoQP639pElD2tQ7kYKRqPxroVyNm/ezEMPPcRjjz2m+WsWLj4lhLBGitV84FvC7EghNTXVZJnWadOmMXnyZDp27MjEiRNp27YtLVq0YNu2bQBcv36dsLAwPDw8GDBggEnNg/Xr19OuXTv8/Px46qmnuHLlClCwxPXEiRPx8/Nj+fLlTJ8+nVatWuHl5UVYWBipqalERUXx+eef4+Pjw7Zt21izZg2BgYH4+vrStWtXtRDP5MmTCQ8Pp2PHjri5uTF9+nT19T/++GNatGjB448/zu+//65uHzVqFCtWrFD7MmnSJLUc6LFjxwDIzMykW7dueHp68vzzz/Poo4+qy9kKISouBchXjBY9rEGZRgp5eXnEx8fz888/88EHHxAXF8dXX32Fg4MDR48e5cCBA+py1efOneOjjz4iLi6OGjVqMHXqVD777DMiIyMBqFu3Lnv37gWgUaNGpKSkYG9vz8WLF6lduzZjx47loYce4o033gDgwoUL7Nq1C51Ox3/+8x/+9a9/8e9//xuAY8eOsWnTJi5fvoy7uzvjxo3jwIEDREdHk5SURF5eHn5+fvj7+9/1fdWrV4+9e/cye/Zspk2bxn/+8x8++OADOnfuzDvvvENsbCzz58+/a9u5c+eq1ZiU/L+/vqoQ4l7JSOF2ZUoKAwcOBG6VxgTYunUr48ePB8DLywsvLy8Adu3axZEjR9TaBbm5uWoJTcBkQTsvLy+GDx9O//796d+//11fOz09nSFDhnD69Glyc3PVEpoAvXv3xt7eHnt7exo0aMCZM2fYtm0bAwYMwMHBAaBI4Yvi3tePP/4IwPbt21m5ciUAPXr0wMnJ6a5tTSuv1Sn2NYQQ5YckhVvMnj4qrqQl3CqPaWtrS15eyTd/KIpCt27dSEpKIikpiSNHjph8265Ro4b689q1a4mIiGDv3r0EBATcNfbLL7/MSy+9xMGDB5kzZ85d+2Vp3+50L+9LCFHRydLZtzObFB5++GHOnj1LVlYWOTk5ZquNdejQgSVLlgBw6NAhDhw4AEBQUBC//fYbx48XFNa4evUqf/zxR5H2haUzO3XqxNSpU8nOzubKlSs4Ojpy+fJl9bjs7Gy1bmlhaTtz/Vq1ahXXr1/n8uXLrFmzxmyb2wUHB7Ns2TKg4NrIhQsX7qm9EKJ8UmT2kQmzSaFKlSpERkbStm1bunXrZrY85rhx47hy5QoeHh5ERkaq5+3r16/PggULGDp0KF5eXrRr1069iHs7o9HIiBEjaNOmDb6+vowfP57atWvTt29fVq5cqV5onjx5Mk899RT+/v7Uq1fP7Bv18/NjyJAheHt707NnT5NSm5aYNGkS69evp3Xr1ixfvpxHHnkER0fHe4ohhCifJCncUinKcf4dcnJysLW1xc7Ojp07dzJu3DiSkpJKbGNTtY4m5Thr1mhc5hgAzpF9NIlz+bMDmsQBuNm1iSZxbDKvaRKnyyjzXzAs8d2wuZrE0Uqjtr01i3ViRVtN4qRe/t38QRYY1OmDMt0LZWtfD4dGfS061r3uQSnHKQqcOnWKgIAAvL29GT9+PPPmzXvQXRJCaETLkUJsbCzu7u7o9XqmTJlSZH9OTg5DhgxBr9cTGBioTtIB+OSTT9Dr9bi7u7Nu3TqLY2qp3N28Vl41b96cffv2PehuCCE0p92UVKPRSEREBBs2bMDFxYWAgABCQ0Np1aqVesz8+fNxcnLi+PHjREdHM3HiRJYuXcqRI0eIjo7m8OHD/PXXX3Tt2lW97mouppZkpCCEsGqF5Ti1mH0UHx+PXq/Hzc2NqlWrEhYWRkxMjMkxMTExPPPMMwAMGjSIjRs3oigKMTExhIWFYW9vT9OmTdHr9cTHx1sUU0syUriP6tS0oUmD5BKPyczMpH79+mYilRzD4jj/3mg2jiWxalkUxcI+bdAojkb92feqNnE8G2gTxxIWxUn9QrP+GAzaxLGEJbFuP/1SGt07BXDunGXXN65fv47htl/A7fclAWRkZODq6qo+d3FxYffu3SYxbj/Gzs6OWrVqkZWVRUZGBkFBQSZtMzIyAMzG1JIkhfvIkmUwDAaDJheutIqjZSyJtWXEPAAAIABJREFUI3HKU6zixMbG3tf4FY2cPhJCCI04OzuTlpamPk9PT1fvp7rbMXl5eWRnZ1O3bt1i21oSU0uSFIQQQiMBAQEkJyeTkpJCbm4u0dHRRZbUCQ0NVW+4XbFiBZ07d0an0xEaGkp0dDQ5OTmkpKSQnJxM27ZtLYqpJdvJkydPvm/RhUWKW5jvQcXRMpbEkTjlKdb9ZmNjQ/PmzRkxYgQzZsxgxIgRPPnkk0RGRqoLdHp5ebF48WL+7//+j6SkJKKionBycqJBgwacP3+e559/niVLljBjxgxatGhRbMz7RW5eE0IIoZLTR0IIIVSSFIQQQqgkKYhKb8eOHSaV9oQQxZOk8IDcXqOiPCmPl5jK0qddu3YxatQoqlSpQk5O+aiEt337dqZNm1amGFr+nW6PVV7+Xd7ep/L4b7Iyk6TwAFy8eJH09HQA9u3bR25u7gPu0S1Xr1590F1Q7du3j7Nnz6LT6Ur1wZCfn8/x48cZOHAgqampzJkzp9RFkwpf/8qVKyYFnUrDycmJzz//nC++MH+XcXF90el0AJw4caJMfbk9VlRUFP/85z/LFO/O2KWl0+lISEggNTW11H9/UUqK+Fvl5+crcXFxyscff6y89NJLire3t3Lt2rUH3S1FURQlNjZWCQ0NVT755BNlxowZmsX973//q2RlZVl0bH5+vqIoirJ3717F09NT0ev1ypkzZ0z23UucK1euKA0aNFDq1KmjZGZm3mPPTa1atUrp1auXEhwcrPz4449Kenr6PcfIy8tTFEVRDh8+rLRo0UKZNm1aqfszc+ZMZcSIEervR1Hu7Xd0u19//VXp0qWLkp2dXar2ha+7e/duZe3atUpSUlKZ4iQmJiq+vr7KI488ohw/ftxkn7i/JCn8jW7/R923b1+lbt26ytKlSzWL/dprrynnzp0rVftt27YprVq1Unbt2qWMGDFCefLJJ5Xr16+XuV9//vmn0rhxYyU1NdXiNrGxsUrbtm2VpUuXKiEhIYq3t/c9JQaj0aj+nJ6erowYMUJp2bKlMmvWrHt/A/+TlJSktGvXTklISFC++eYbJTw8XJkzZ45iNBot6tPtx9y8eVNRFEU5ePCg0qJFC+XTTz+95/4sXrxY8ff3VzIyMhRFUZTz58/f9bUs8fvvvythYWFKu3bt1C8opfkAjo2NVVq2bKnMmDFDsbOzU9auXXvPMRRFUdatW6f4+fkpMTExyuDBg5XmzZtLYvgbSVL4m9z+j3n58uXK1KlTlfHjxytvvfWWsnHjRvWDovCbZGmkpaWVqt2lS5eUH3/8UYmLi1N27NihGAwG5c8//1QURVGSk5NL3Z+8vDxl0qRJypEjR+6pXUREhMk36NGjRystW7ZUv+nf/qFfkqioKOWll15SXnvtNWX58uVK48aNS/XNPCUlRRk1apTSu3dvddv69esVb29v5eDBg2bb3/63X7p0qfLll18qW7duVRRFUY4dO6a0bNnynvr13//+V3n99deVr776Sjl+/LgydepUJTAwUHn66act6svdfn/r169XBg0apERFRSkXL14s0u+SGI1G5ezZs0q/fv2U33//Xdm+fbvSunXrUo9g3nrrLeXjjz9Wn0+cOFFxcXFRUlJS1NcT94/c0fw3KTxvu3PnTj777DOioqLo2bMnmzZtYv/+/TRq1Ii1a9eSlJSEn59fqV6jZs2a6s8XL16kWrVqZtusX7+eZcuW4ejoyEsvvcSmTZuIi4vjkUceYd26daxdu5bAwECqVKlyz/2xsbHh8ccf5+GHH76ndkePHiUnJ4f27dsD0Lt3b2bNmsXGjRsZNmwYtra2ZmP88MMPTJkyhY8++oiEhARsbGzw9PRkyZIlpKWl0bVr1xLbK7eda7exseH06dMcOHCA6tWr4+npSbNmzUhKSqJGjRq0bt26xFiFcWbNmsXcuXN57LHHCA8Px8nJiZCQELp3787zzz9PtWrVCAwMLLEvc+fOZePGjfTo0YNXXnmFAwcO0Lp1a1599VWWLVtGhw4dqF27drF9uXr1Kvb29gDMmzePmJgYfvr/9s49Lufz/+Ov2JiMYRN9HVOMQuUURmVUkkMiJJnkl8NSWTRnmZljJiHVjOVQFpWilEPSAbkpTaNS6aDkdKMknV6/P/ren29NYpbCPs+/uu/P53Pd7+vuflyv63pf7/f7OnYMM2fORNOmTXH+/Hk8ePAAysrKaNKkSY39qty/pk2bIiMjAxKJBDt37oSvry86dOgALy8vfPrpp2jVqtVLn6/cPwBISkqCVCrF119/DQAYMWIE9u/fj6NHj2LixImvbZfImyFuNNcRJJGQkABra2u0atUKhYUVx0c6OTnhs88+g4eHB9atW1eldO6bkpeXh7lz574y2ubKlSsIDAzE8OHDMW3aNBgbG0NDQwNycnKIjIyEg4MD+vfvD3l5+Te25aOPXq8Q76VLlxAbG4uMjAyMGTMG3t7e8PPzw9OnT3H58mXo6uqCJJydnV+rvaSkJFhaWkJDQwPOzs6Ql5eHVCqFt7c3IiMja6xgKxukTp8+jb179+LYsWOwt7fH+PHjERMTAycnJ0RGRiI0NBQdO77eUakJCQkIDAxEWFgYioqK0LFjRxw4cAA///wzunXrhujoaIwaNaraZ2UD5r59+/Dnn3/CwsICOjo6kEgkOHz4MObMmYOMjAw8fPiwysTgrwQGBsLOzg4A8Ntvv8Hd3R2DBg1CRkYGjIyMoKuri9GjRyM6OhoBAQGvtbkbFxeHjRs3Aqgoc+3l5QUfHx9BNDds2IC8vLwa25CTk8P58+cRERGBpKQkjB8/HseOHcOePXsglUpx8eJF9O/fH23btn3jzXmRv0E9rlI+eKpbMh84cICDBw/muXPn+Pz5c5IVbpbbt2/z7t27tfbZBQUFr7RNU1OTAwYMYGpqKsvLy3nlyhUuW7aMAwYMoL6+Po8ePfrSftQGMjdAREQEFRQUaGlpSQMDA0okEkokEo4YMYLm5ubs3Lkz4+Pj6e7uThcXl9dq29/fn2PHjmViYqLwnra2Nu/fv/9a7ocTJ06wV69ePHnyJBs2bEhPT09KpVK6uLiwX79+NDU1ZWhoaJV+VKa67ywnJ4fBwcEcNmwYSXLfvn1s0qQJDxw4UK0NsjZk7evq6rJ58+bC/kFJSQnLysq4a9cuqqqqMiEh4aX9uX//PocPH87r168zJyeHZmZm9PHxEa5bWVlx5MiRJCvcm3fu3HlpWzK7IiMjaWNjwy+//JJ79+5lSUkJjYyMaGlpyVmzZlFdXV34DVWHrF9RUVFs06YNraysqK+vz9OnTzMlJYWGhoY0Nzdnly5dGBcXx927d3PdunUvbU+kdhBFoQ7w8PDg8uXL+eOPP/Lx48f09PTkyJEjGRkZyeLi4jq3JzIyksHBwfz111+poqLCrVu3Vrn+6NEj5ufnk3w7glB5AzsmJoZOTk6MiYkhSXp7e1NbW5sXLlxgSUkJs7KymJ2dzdDQUGpoaFQZ5GtCKpVy2bJlXLp0KcPCwnj06FH27dv3lcJbXl7OoqIimpub89q1azx58iT79+8v7LE8f/6cW7du5cKFCxkUFPTSNir378SJE8Jrb29vTpw4kWRFNNP48eOrjWKq3EblAdrQ0JBff/218LqkpIQSiYTJyck19uvJkyc0MDDg5MmTOWnSJM6fP59btmwRrpeVlXHKlCmvvacVHh7ODh060NfXl6tXr+b48ePp4eHB0tJSnjp1ir6+voyNjX2hLyRZWFhYRRBWrVrF6OhokmRgYCB1dHQYFhZGskLMcnNzeeLECWpqavLatWuvZZ/ImyOKwlvGxcWFw4cPZ0REBDU1NYUNtG3btlFLS0sYDOuK6Ohodu/enVOnTqWjoyMHDRrE1q1bc9u2bXXy+Xfu3OHPP/8sbGZOmzaNbdu2FQaQwsJC+vj4sF+/fvTz8yNJ5uXl0czM7G+HOd6+fZuurq7U09Pj+PHj/9bzy5Yt45IlS6ijo8MbN26QJH/77TdGRETwyZMn/OGHH7h06VI+efLkpW1s376dPXr0YJ8+fThu3DgWFBQwJSWFkyZNooGBAXv27PnKwXzHjh0cP348Fy9ezCNHjpCsWDEYGhq+dl9kbNy4kU2bNuXGjRt569Ytqqqq0tfXl7m5uTx48CD79esn/F9exe7du/nzzz+TJB8/fswzZ85w0KBB9PDwqPG5e/fucf369ZRKpSRJGxsbtmrViuHh4SQrJgxBQUEcMGAA9+zZQ7JCGKysrN44zFXk7yGKQi0jmwHJZkcLFy5kcXExN2/eTENDQxYVFQmrg927dwsz0Lrg4sWL1NXV5fnz50lWRBZt376dU6dOpaKiIleuXPnWbXjy5AmTk5N5+/ZtISrpm2++4ZgxY4TvpbCwkAcPHhTsJMmioqI3/synT5/W6E6T/a9u3LjBzMxM5ufnc//+/WzQoAGvX79O8n95ExERESQrVlM15V5ERUXR2NhYcBGamJhw+vTpfPDgATMzM+nt7f3KyK5Dhw5x6NChzM7O5qBBgzh//nzhWr9+/WhsbPx6X8B/uXXrFk+ePMlu3brRx8eHERERHDNmDM3NzTl48ODXiqSSsXfvXnbv3p25ubkkKwbzKVOm0NjYmIcPH672maKiIpaVlTEzM5OZmZmUSCQkSVtbW+rq6goCW1RUxMDAQF64cEF4tj5W1P9WRFF4S6SlpfHp06c0NjamoaEhjY2NhR+2m5sbAwMD69ymsLAwNmjQQFitFBcXMyAggCtWrGBaWhojIyPf6ufLBLO0tJS2tracNWuWMAs3NTWlsbGxMPj/VVzfNsHBwezVqxdXr17NL7/8kkVFRbSzs+PAgQM5a9YsIW6+sm2VqWxnTk4OZ8+ezV69elURtgkTJtDY2JhPnz59pT3l5eV0c3Pj6dOn+dtvv1FPT0/4/cj2FP5O7kdlJBIJlZWV6e3tzWfPnvHx48c1utVkfZNIJDx8+LAgBD/88ANNTEyYlZXF+Ph46uvr08HB4QV3JEk+ePCAy5YtY3BwMEly+fLlnDZtGq9cuUKSnDVrFkeMGCGsIGSfKeYl1D2iKNQS0dHR9Pb2JlnhGtLQ0KC9vT1HjRrFNm3aCLOnPXv2sHv37kIyTl0TEBBAZWVlHjx4kCR59uxZampq/u3Y9L+LrF1ZctSjR4/4/fff08bGRhAGIyMjGhoa1slAkJeXxwMHDrCgoIA5OTns378/b968yQMHDlBVVVWYtcbGxjI2NlZwXVRnW+X3ZH9nZ2fTxsaGP/74I69evSpcnzZt2iv3EGR/+/v7s1OnTtTW1haubdu2jStXrvzH31F8fDxbtWrFnTt3vtb9QUFBVFVVpb29PYcMGcLff/+dN27c4KpVq9ivXz/27duXiYmJ/PXXX2llZcXS0tIqNubm5nLVqlVcvHgxz58/z5KSEq5evZrW1ta8fPkySdLCwoJDhgz5R7k6Iv8cMU+hlkhISICDgwMePHiAhIQE7Nq1C59//jmaNWuGhw8fwsvLC3/++SeOHTsGb29vfPnll/ViZ/fu3aGkpARLS0tcvnwZEokEtra2UFdXB4Aq8eK1iZycHEJCQmBvb4+8vDy0adMGJiYmiIqKwsWLF9GxY0fY2dlBVVUVioqKb8UGGSRx9OhRnDlzBoWFhWjXrh2aN2+OR48eYcuWLTh8+DAUFRURGhoKdXV1dOnSBW3bthX68de2ZO9t374d27dvxy+//IJ+/fph6NChiIqKQmZmJj799FMoKirCxMSk2rBRWRu+vr7w8/PDf/7zH3Tp0gW5ubno1q2bYI+bmxtWrFgBBQWFf/QdtG3bFkZGRujevXuNOQQAcOPGDaxYsQJ+fn5o2rQpvLy8UFpaKuS2mJmZYcaMGUhJScH3338PFxcXtGnTRuhTWVkZmjdvjvz8fJw7dw6RkZHo3r07TE1NcfnyZURFRUFBQQE2NjYYMGCA8F2L1BP1rUofEmFhYVRTU+PUqVNJVvhG//zzTy5atIhBQUF88OBBrYad/hOOHj1KdXV1bty4kWTF7PRtzNBlrpYrV65w9OjRdHNzo729Pe3s7BgdHc2ioiLa2tpy7ty5rwyjrW02b97MOXPmcOvWrezSpQs7dOggZJafP3+ew4YNe+0Vnbe3N7W0tBgfH88dO3ZQSUmJ0dHRzMjIoLW1NTdu3MiioqIXvuPKrw8ePMhevXrRzs6OGhoaDA8P55kzZ7hx40YOHz6cJiYmNYadvg3KyspYXFzMxMRExsTEsE+fPrxz5w5Xr15NJSUl7tu3jyUlJczLy+OqVateGh0WGhpKVVVV7tixg3p6erS1tWVMTAxLS0vp6OhIKyurN667JFK7iKJQywQEBLBFixZVYsDHjh1LX1/ferSqekJDQ9muXTshqqU2yczMFPYH0tLSqKysLBTZS01NpbOzM+3s7Hju3Dk+e/ZM2NCtK06cOEFtbW0OGTKEkydP5qJFi9i6dWtu2rSJrq6u1NDQYEBAwEufv3LlCnfv3i283rx5M9evXy+89vb2ZufOnfno0SNeunSpSskHGZUFITs7m4cOHRI233fu3EkDAwOeOnWKZMUE459str8JV65c4dq1a4WJjJeXF62trUlWhKTq6+tX+b9VZ195eTlLSkr43Xff8ZdffiFZUQ/LycmJZmZmlEgkLC0tfWUUlkjdIYrCWyAoKIhKSkpctWoV/f392bNnz3rbQ3gVYWFhTE1NrfV2v//++yohhDNmzGDnzp2F+kW3bt3i2rVrOW/ePGFzsa7Iy8vjV199JcxqXV1duWrVKq5cuZJz5syhs7OzECdf3eqprKyMcXFxzM3NFQbxvXv30tLSssp9lpaWL62kWrndrVu3slOnTuzZsyfNzMyE993d3dm3b19BGOqa+Ph4jhgxgps2beLDhw+ZlJTEL7/8ktbW1lRVVeXJkydJvl4tovXr13PixIlCwca0tDSqqKjw22+/rVLMT6T+EUXhLeHv788GDRpw7Nixb2XQfR/IyMiggYGBsLm8YMECDh06VJg1p6enMy0trc7tevjwIbW0tITw0uLiYv7f//0fv/766yqz/+qQuZfIivj5cePGcdOmTSwpKaG+vj4XLFjACxcu8LfffqOamhpzcnJqbC8qKooWFhZMTU1lTEwMp0+fzsWLFwvX9+zZU6dhyySZnJwsrA6uXr1KY2Njrl+/ng8fPmRcXBydnJx4+vTplz4vE7yEhASGhISQrPgtODg40NXVlfn5+UxLS+OIESP+VhisSN0gisJb5OzZs0Jlx38rI0eO5OjRo4UsZkdHR2pqalbrTqlLnJ2duXr1amFQOnHiBMeNG1ejG+vRo0dCFFloaCj9/f0ZERFBU1NTenh4CGGss2fPpr6+frX+9corhPDwcPbp04cmJiYsKChgcXExY2NjOX369Co5CXXJkydPaGtrywULFgjCEB8fz27dunHhwoWvndwWGhrKTp06cfjw4RwwYACzsrIYGBhIGxsb9u/fn7169arRPSdSf4iiIFJryAa8jIyMKgPi5MmTOXLkSEEY7O3t6zyT+69kZWVx2bJl1NPT49KlS9mtWzfBHfIypFIpt27dyt69e1NdXV0InYyMjKSJiQl37dol3FtdpnNlQdi1axd/+OEH+vj4UFtbm4cPHxbOZoiOjqa1tXWdByXISkhcunSJ3333HZcuXSrkJKxZs4aDBw9+rdyIpKQkTpo0SXAfzps3j6NHjxZWPImJieL5CO8woiiI1CqyGkNGRka0trYW/MXm5ubU1tZ+Z06ZIysG7tDQUDo7O/Ps2bMvva+yz3zfvn1UUFCgsbFxldyLqKgojho1ihs2bCBZ82B38uRJWlhYCAOsr68vdXV16evrK8T318YBR6+DzM6kpCT269dPOJMhPj6eDg4OnD59On18fDh8+PBXCnlZWRkLCgq4dOlS9uzZs0qwha2t7WuLikj9IoqCSK0RGRlJTU1N3r17l+7u7mzevDnnzp0rlIMwNTUVahy9L1Qe3MPCwpiWlsZLly5xzZo1NDMzE4rVyUI2ZSehVddGaWkpCwoKqK+vTw0NDUZHRwurDT8/P2poaNDf378OelWVgIAADhs2TKiFJQupzsjI4MqVK2lgYPDS4n/ki9nHd+7c4bJly+jo6CgcJkRWrBgql64QeTeRI8UTsUX+Obdv30aTJk2QkpICqVSKVatWwcPDA3PnzkWrVq3g6en51pPS3iZubm7YunUrTp06hfbt2yMtLQ2enp7IyMiAlpYWrl27BmdnZ3z22WdVnisvL0eDBhXHluTn56NZs2Z48uQJ5s6di44dO2LBggVCIlpQUBB69+6NTp061Vm/iouLYWJiAltbW+jr6+Pu3buYP38+5OXlsWfPHgDA48eP8dlnn71wGA7wv+S9EydOwNfXF6qqqjAwMICioiJ+/vlnlJeXQ09PD8OGDauzPon8M8RDdkTemPLycgDA2bNnoaWlhbKyMvTt2xdhYWGYPXs21NXVYW5ujvv37+Pp06f1bO2bc+PGDezduxehoaHo0KED5OTkoKysjHnz5qF37944cuQIFixY8IIgABAEwdPTE2ZmZpgzZw6OHDmC3bt3IyUlBS4uLsjNzQUAjBkzpk4FAfifaJWWlgIAWrVqhenTp+PChQuYO3cuAAj9qi7bXU5ODsHBwViyZAlMTEwQGxuLefPmISkpCQsXLkRJSQmCg4MhlUrrrlMi/4z6XaiIvI9Urlh58eJFGhkZVQlR9PDwoJ6eHt3c3Dh06FChVv77wl/3A9LT02lqaiq8lvVfFnNfXYG7y5cvC9nHPj4+VFdXZ2RkJI8dO0YdHR1u2bKFBQUF1NXV5Zo1a+qs3k/lPQTZfo+vry+VlZV56dIlkuTp06dpZ2dHExOTKu4fGZVtlUqlXL16NZOTk3n8+HH279+fzs7O1NHRYWxsrJDfIPL+IIqCyN+ipKSE+/fvZ0xMDCUSCfv3789WrVrR3t5euCclJYU7duzgyJEja/RFv4v89XCboqIilpaWctCgQVy+fLlw7ddff6W1tTVLSkpeEJFz586xb9++QqLevn376O7uLlxPTk7mqFGjeP/+febk5DAzM/Mt96oCmZ2hoaHs3LkzDQ0N+dNPPzE3N5fe3t5s27YtFy1aREVFRV68eJE2NjZCVVMZz5494/nz5/n48WNKJBKeO3eO2dnZzMrK4uDBg3njxg0WFBRQS0uLAwcOrPPERJF/zusdoCsi8l8++ugjqKmpwdDQEI0bN8bx48dRWFgIJycnuLq6Yv78+VBRUYGKigpmzpyJTz75pFpf9LuKzM4dO3YgKCgIXbt2RadOneDn5wcDAwNkZmaiY8eOCAkJwZ49e6o9gzouLg4dO3bE1atX8fjxYzx69Ajbt2+Hubk5mjZtiq5du+LTTz9Fdna2UIiwrvp24cIFHDlyBIGBgXj48CHCwsKwfft2LFq0CGpqarh79y5mzZqF+/fvIyoqCvb29lXaePz4MeLj47FlyxacP38eYWFhaNeuHW7duoX27dujS5cuiIuLg5qaGhYtWoQWLVrUWf9Eaon6ViWR94+CggIOHjyYKioqQgmGkJAQTpo0qcoRj+9rDPrvv/9ObW1t3r59mxYWFpw0aRLJikxoFxcXurm5CeW+X0bHjh3ZokULwUVjY2PDvn378o8//qCXlxf79+/PrKyst96XyhQVFXHcuHHs2rWr8F5kZCSXLl1KBwcHwc1z9epVGhgYvPSks+DgYDZp0oSWlpZVkhBHjx5NExMTtmvXrl6iqERqBzH6SOSNePbsGa5cuYK5c+dixYoVMDU1hYuLC/bv34+AgAC0a9euvk18bVhpJZObmwuJRILWrVsjISEBhw8fxvHjx/Hxxx/j5s2bUFFReWUbRUVF0NfXR8OGDdGzZ0+4uroCABwdHZGTkwOpVIoNGzagZ8+eddo3AMjMzISxsTG0tLTg5uYGAIiIiEBwcDCsrKzQrVs3SKVSkKxSUlvWTmxsLJo3b46cnBzExMSgpKQEY8eORd++ffHkyROkpKSgefPm6Nq163u1QhT5H6IoiPwjjh07Bjs7O0ybNg2BgYHYtGkTRowYUd9mvRHu7u5ITU1Fjx49YGdnh379+uHMmTMAKqKHkpKSsGbNGjRp0qTKc5UHv507d6Jnz57Q1tYWBsx27drhl19+AVARAgoAjRo1euv9kdl15swZpKen49NPP8XkyZORnZ0tCIBMsKRSKVq2bFlje0ePHsUPP/yALVu2QEdHBxKJBN7e3mjRogWaNWuGixcvwsXF5R+f9SBSz9TPAkXkQyImJoYzZ84UKou+L1Su4yPLMpa5dJYsWcIRI0bw2rVr3LlzJ9XV1V9ZvG3z5s3U0tKqctJabm4ux40bVyV6qS6QFe47fvw4e/TowYCAADZp0kQ4ijUrK4tfffWVUAr7VWRkZFBTU1Mo7njr1i2mp6fz1q1b3LRpE3V0dN5KCXaRukcUBZFaoXL10PeBpKQkbtu2jYWFhSwsLOSIESPYp08fxsbGsqysjPn5+fzxxx85ceJEmpmZCXWBXsadO3doaGjIp0+f8uHDh/T39+fmzZv57Nkz3r59m6ampq+smFobpKWlCWKXnZ1NXV1dJiYmMjQ0lL1792b37t25cOFCkhVnXly8ePGVbSYmJjIoKIiDBw9mUFAQHRwcaGhoyMaNGwvhxvn5+STf330kkf8huo9E/nU8e/YMxcXFKC0tRU5ODj766CO0adMG1tbWUFNTg729veBKKS8vB0k0bNiwShuVM5UB4N69exg7dix69+6NR48e4YsvvkBERAQsLS3h4OCAsrKyF9p4G5w6dQqmpqZIS0tDy5YtkZ2djQcPHsDKygoSiQSJiYno1asX1q9fD0dHx5e2w/+6nuLj4zF79myEh4dj3bp1SEhIgIWFBSZOnIht27bh2bNncHR0FPcOPiTqV5NEROqW1NQXHBpCAAAJN0lEQVRUWltbUyKRkKRwNGhKSgrz8vI4evRoOjk5vXZp7+DgYEokEhYUFDAxMZGurq7CwTv79u3j7NmzWVxcXKcz6JCQECopKQmRT+fOnePEiRNJViTVTZo0qdqktL8ie87T01N4T5a4FxUVxR49ejA8PLz2OyBSr4iiIPKv4s8//+Tq1atpY2PD9PR03r59mytWrODChQt58+ZN5uXlcejQoVy3bl21J4pVHty9vLzYvn17mpubc8aMGVWKve3cuZNqamovPbP4bRMcHCwIg1Qq5TfffENzc3MqKSkJFWFfJVTnz5+noqIiv/32W+H+srIyxsfHU1VV9b1LTBR5PUT3kci/AlaKENq9ezdOnToFeXl5rFy5Ep988gm2bduGsrIyzJw5Ey1btkRRURE6dOjw0va8vLyQlJSE7777DsXFxQgICEBMTAysrKygpaWFGTNmYOXKlVBTU6urLr5ASEgI7OzsEBcXh4KCAkRHR+Pzzz+Hjo5OtffLvqM//vgD8vLy+M9//oPs7GwYGhpi6dKlmDlzpnBveno6lJSUxLDTDxBRFET+Vbi4uMDb2xtGRkaIi4tDmzZtsHDhQjRt2hTr169H8+bNsWLFCnz88cc1tvP1118jLi4OmZmZaNasGdLT0xEaGoqQkBCsXbsW3bt3rzbbua4JCQmBlZUVrl27ViXv4K/IBvegoCCsWbMG+vr6uHz5Mjw9PZGVlQVLS0vY2dkJRfJEPlxEURD5oElOTkabNm3w2WefobCwEDNnzsTatWuhrKyMhIQE/P7777h37x6WLFkCeXl5yMnJoXXr1lXa4F8S0z755BMAgJ6eHkpLSxEeHg4ASE1NRXh4OAwNDd+p5L3jx49DXl6+2vLVlTfAs7KyMGXKFAQFBWHPnj3w9/eHn58fFBQUcOHCBUyePBlRUVE1rqBE3n/E0tkiHyQk8fz5c+zYsQPl5eUoLS2FvLw8CgsL4enpCQDo3bs3NDU1ERMTg23btqFly5Y1CoKbmxuWL1+OH3/8EQBw8uRJNGrUCHp6egAAZWVlzJgx450SBAAwMjLCsGHD8Nf5X25uLpYsWYKMjAwAFbWR+vTpg5MnT8LPzw979+6FgoICTp8+jYEDByIhIUEUhH8BoiiIfJCUl5ejcePG2Lx5M1JSUrBw4ULk5+dj+fLlyM/Px86dOwEADRs2RN++fbF48eJqXUaVC+QdPHgQs2bNgrOzM2bPno179+4hNDQU+fn5GDt2rNDeu8pfff8tWrTAzZs34erqiqysLLRv3x63b9+Gra0tvL29oaKigjNnzmD58uVIT0+v9rwIkQ8P0X0k8kGTm5sruI0mTJgAfX19JCcnw9nZGfLy8khNTYWfn98LdYj+6jKysrLCli1b4OvrC39/f6H6p7u7O7744guheur7QmlpKT766COcO3cO9vb26NSpE3bt2oXr16/D29sbhYWFGDlyJNavX4+1a9cKoify4SOKgsgHRUxMDDIzMzFlyhS4urrC09MTenp6yMnJQWpqKmbMmIFJkyahWbNmSElJQevWrdGmTZsqbVQWhA0bNkBDQwO6urpISEiAo6MjwsPD8eDBA3Tu3BmOjo5YunTpO71CeBkhISFYs2YNZs6cia1bt0JbWxvfffcdGjdujJ07d0JBQQFqamrQ19cXo4z+RdR/eISISC0ilUqxZMkSJCYmIjU1Ff7+/khNTUVSUhLu3r2Lw4cPIy0tDStWrHhplVLZ4BcSEoKzZ8/CzMwMjRs3RuPGjYUs6Li4OIwaNQrffPPNeycIJFFeXo5Dhw7B3Nwcs2bNgoWFBaZOnYrFixfDzc0N69atq/KMKAj/HkRREPmgMDIyQqNGjbBgwQKoq6tDWVkZ7du3R4cOHZCeno4hQ4bAz88Pz58/f+HZu3fv4t69e1BTU8PevXuxfv16qKioCG4hRUVF6OrqYtq0abhz5w6OHDnyXrmMZMjJyaFhw4ZQU1ODVCpFfn4+mjVrBnd3dygrK+Onn37CTz/99EI1WJF/Bw2dnJyc6tsIEZHaRFlZGe3atcOmTZugpKQEdXV1tG7dGjt27ICuri4cHBzQtGnTF57LycmBra0tTp06hXPnzsHOzg6nTp3C8+fPoaWlhaZNm0JTUxM6OjqwsrKCsrJyPfTuzZC5fxISEpCbm4tGjRqhefPm8Pb2RqdOnfD555/j3r17SEtLg7W1NZSUlOrbZJF6QlwpiHyQjBs3Dvv27YOtrS2uX78ODQ0NpKen1ziz79q1K3r37g0PDw9s2LABFhYW+OKLL+Du7g4AsLW1RcuWLV957sC7iJycHE6ePAkLCwsYGBggKSkJgYGBmDlzJjw8PFBYWIjk5GRs2bIFgwYNqm9zReoRURREPlhGjx6N0tJSTJgwAaNHj8bRo0fRpUuXGp+ZM2cO1NXVsWXLFrRq1QqTJ0+GgoIC5s2bh9atW8PMzKyOrK9drl+/juDgYBw5cgRfffUV1q1bh+HDhyM0NBRGRkZISkoCSfTq1au+TRWpZ0RREPmgMTY2xpkzZ9CpUyd07tz5lferqKhARUUFLVq0wLJly9CiRQsUFRWhUaNGGDhw4Ns3+C2QnJyMb7/9FiQxYcIEkMSSJUsgJyeHgQMH4sSJE3VyNKjI+4EoCiIfPC8rAFcTY8aMwccffyzURdq9e/d76WfPzc2Fj48P1NXVcf36dURFRaFbt25QUFDA4sWLUV5ejnv37tW3mSLvEGKegohIDdy9e7faekjvMpVzCsrKyuDl5YXc3Fw8evQIsbGxMDQ0xPTp06GoqFjtMyL/bkRREBH5AImJiUF6ejrMzc1RVlaGQ4cOIS0tDcXFxQgLC8OYMWOwaNEiNGrUqL5NFXnHEGsfiYh8gEilUqxYsQI+Pj5o2LAhTE1N8ezZM1y5cgVDhw7F8OHDRUEQqRZxT0FE5APEyMgIDRo0EPYNpk6dCj09PSQnJ2Pu3LmvjMIS+fciioKIyAeKoaEh5OTkMH36dMTExCAkJATu7u6iIIjUiLinICLygXPt2jXExMRAVVUVQ4YMqW9zRN5xRFEQEREREREQN5pFRERERAREURARERERERBFQUREREREQBQFEREREREBURRERERERAREURARERERERBFQUREREREQBQFERERERGB/wfhSpq3JjRX5wAAAABJRU5ErkJggg==\n", 868 | "text/plain": [ 869 | "
" 870 | ] 871 | }, 872 | "metadata": { 873 | "tags": [] 874 | } 875 | } 876 | ] 877 | } 878 | ] 879 | } 880 | --------------------------------------------------------------------------------