├── .gitignore ├── README.md ├── arxiv_scrape.ipynb ├── beam_arxiv_scrape.ipynb └── multi_label_trainer_tfidf.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | venv/ 2 | **.csv 3 | **pycache** -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Multi-label Text Classification 2 | 3 | Holds code for collecting data from arXiv to build a multi-label text classification dataset and a simpler classifier on 4 | top of that. Our dataset is now [available on Kaggle](https://www.kaggle.com/spsayakpaul/arxiv-paper-abstracts). The dataset collection process 5 | has been shown in [this notebook](https://github.com/soumik12345/multi-label-text-classification/blob/master/beam_arxiv_scrape.ipynb). We leverage 6 | Apache Beam to design our data collection pipeline and our pipeline can be run on [Dataflow](https://cloud.google.com/dataflow) at scale. We hope 7 | the data will be a useful benchmark for building multi-label text classification systems. 8 | 9 | Here's an accompanying blog post on keras.io discussing the motivation behind this dataset, building a simple 10 | baseline model, etc.: [Large-scale multi-label text classification](https://keras.io/examples/nlp/multi_label_classification/). 11 | 12 | ## Acknowledgements 13 | 14 | We would like to thank [Matt Watson](https://github.com/mattdangerw) for helping us build the simple baseline classifier model. Thanks to 15 | [Lukas Schwab](https://github.com/lukasschwab) (author of [`arxiv.py`](https://github.com/lukasschwab/arxiv.py)) for helping us build 16 | our initial data collection utilities. Thanks to [Robert Bradshaw](https://www.linkedin.com/in/robert-bradshaw-1b48a07/) for his inputs 17 | on the Apache Beam pipeline. Thanks to the [ML-GDE program](https://developers.google.com/programs/experts/) for providing GCP credits 18 | that allowed us to run the Beam pipeline at scale on Dataflow. 19 | -------------------------------------------------------------------------------- /arxiv_scrape.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "arxiv-scrape.ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "language_info": { 15 | "name": "python" 16 | } 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "code", 21 | "metadata": { 22 | "colab": { 23 | "base_uri": "https://localhost:8080/" 24 | }, 25 | "id": "ndfglIjXYcBU", 26 | "outputId": "dcfc54c0-70c6-452e-ac68-2bb396744183" 27 | }, 28 | "source": [ 29 | "!pip install arxiv" 30 | ], 31 | "execution_count": null, 32 | "outputs": [ 33 | { 34 | "output_type": "stream", 35 | "name": "stdout", 36 | "text": [ 37 | "Collecting arxiv\n", 38 | " Downloading arxiv-1.4.2-py3-none-any.whl (11 kB)\n", 39 | "Collecting feedparser\n", 40 | " Downloading feedparser-6.0.8-py3-none-any.whl (81 kB)\n", 41 | "\u001b[K |████████████████████████████████| 81 kB 5.3 MB/s \n", 42 | "\u001b[?25hCollecting sgmllib3k\n", 43 | " Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB)\n", 44 | "Building wheels for collected packages: sgmllib3k\n", 45 | " Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n", 46 | " Created wheel for sgmllib3k: filename=sgmllib3k-1.0.0-py3-none-any.whl size=6065 sha256=1a262bf6597c0a6d7c9fea438a07ef9abce303bce87c66dc0a19ff6bfc1ebe83\n", 47 | " Stored in directory: /root/.cache/pip/wheels/73/ad/a4/0dff4a6ef231fc0dfa12ffbac2a36cebfdddfe059f50e019aa\n", 48 | "Successfully built sgmllib3k\n", 49 | "Installing collected packages: sgmllib3k, feedparser, arxiv\n", 50 | "Successfully installed arxiv-1.4.2 feedparser-6.0.8 sgmllib3k-1.0.0\n" 51 | ] 52 | } 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "metadata": { 58 | "id": "ApsL6p9CWTBX" 59 | }, 60 | "source": [ 61 | "import arxiv\n", 62 | "import pandas as pd\n", 63 | "from tqdm import tqdm\n", 64 | "\n", 65 | "query_keywords = [\n", 66 | " \"\\\"image segmentation\\\"\",\n", 67 | " \"\\\"self-supervised learning\\\"\",\n", 68 | " \"\\\"representation learning\\\"\",\n", 69 | " \"\\\"image generation\\\"\",\n", 70 | " \"\\\"object detection\\\"\",\n", 71 | " \"\\\"transfer learning\\\"\",\n", 72 | " \"\\\"transformers\\\"\",\n", 73 | " \"\\\"adversarial training\",\n", 74 | " \"\\\"generative adversarial networks\\\"\",\n", 75 | " \"\\\"model compressions\\\"\",\n", 76 | " \"\\\"image segmentation\\\"\",\n", 77 | " \"\\\"few-shot learning\\\"\",\n", 78 | " \"\\\"natural language\\\"\",\n", 79 | " \"\\\"graph\\\"\",\n", 80 | " \"\\\"colorization\\\"\",\n", 81 | " \"\\\"depth estimation\\\"\",\n", 82 | " \"\\\"point cloud\\\"\",\n", 83 | " \"\\\"structured data\\\"\",\n", 84 | " \"\\\"optical flow\\\"\",\n", 85 | " \"\\\"reinforcement learning\\\"\",\n", 86 | " \"\\\"super resolution\\\"\",\n", 87 | " \"\\\"attention\\\"\",\n", 88 | " \"\\\"tabular\\\"\",\n", 89 | " \"\\\"unsupervised learning\\\"\",\n", 90 | " \"\\\"semi-supervised learning\\\"\",\n", 91 | " \"\\\"explainable\\\"\",\n", 92 | " \"\\\"radiance field\\\"\",\n", 93 | " \"\\\"decision tree\\\"\",\n", 94 | " \"\\\"time series\\\"\",\n", 95 | " \"\\\"molecule\\\"\",\n", 96 | "]" 97 | ], 98 | "execution_count": null, 99 | "outputs": [] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "metadata": { 104 | "id": "TIct2UKLbL31" 105 | }, 106 | "source": [ 107 | "# Reuse a client with increased number of retries (3 -> 10) and increased page\n", 108 | "# size (100->500).\n", 109 | "client = arxiv.Client(num_retries=20, page_size=500)\n", 110 | "\n", 111 | "def query_with_keywords(query):\n", 112 | " search = arxiv.Search(\n", 113 | " query=query,\n", 114 | " max_results=20000,\n", 115 | " sort_by=arxiv.SortCriterion.LastUpdatedDate\n", 116 | " )\n", 117 | " terms = []\n", 118 | " titles = []\n", 119 | " abstracts = []\n", 120 | " for res in tqdm(client.results(search), desc=query):\n", 121 | " if res.primary_category in [\"cs.CV\", \"stat.ML\", \"cs.LG\"]:\n", 122 | " terms.append(res.categories)\n", 123 | " titles.append(res.title)\n", 124 | " abstracts.append(res.summary)\n", 125 | " return terms, titles, abstracts" 126 | ], 127 | "execution_count": null, 128 | "outputs": [] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "metadata": { 133 | "colab": { 134 | "base_uri": "https://localhost:8080/" 135 | }, 136 | "id": "L73h-A7RYmqR", 137 | "outputId": "0e3bd550-1a12-4fd2-efa7-2e5e286c2c3f" 138 | }, 139 | "source": [ 140 | "all_titles = []\n", 141 | "all_summaries = []\n", 142 | "all_terms = []\n", 143 | "\n", 144 | "for query in query_keywords:\n", 145 | " terms, titles, abstracts = query_with_keywords(query)\n", 146 | " all_titles.extend(titles)\n", 147 | " all_summaries.extend(abstracts)\n", 148 | " all_terms.extend(terms)" 149 | ], 150 | "execution_count": null, 151 | "outputs": [ 152 | { 153 | "output_type": "stream", 154 | "name": "stderr", 155 | "text": [ 156 | "\"image segmentation\": 2082it [00:34, 60.41it/s]\n", 157 | "\"self-supervised learning\": 0it [00:03, ?it/s]\n", 158 | "\"representation learning\": 3690it [01:07, 54.33it/s]\n", 159 | "\"image generation\": 1241it [00:22, 54.30it/s]\n", 160 | "\"object detection\": 4439it [01:16, 57.97it/s]\n", 161 | "\"transfer learning\": 3456it [00:57, 59.61it/s]\n", 162 | "\"transformers\": 20000it [06:18, 52.89it/s]\n", 163 | "\"adversarial training: 0it [00:03, ?it/s]\n", 164 | "\"generative adversarial networks\": 4185it [01:15, 55.67it/s]\n", 165 | "\"model compressions\": 497it [00:08, 59.19it/s]\n", 166 | "\"image segmentation\": 2082it [00:37, 55.76it/s]\n", 167 | "\"few-shot learning\": 0it [00:03, ?it/s]\n", 168 | "\"natural language\": 8976it [02:37, 57.01it/s]\n", 169 | "\"graph\": 20000it [06:09, 54.17it/s]\n", 170 | "\"colorization\": 20000it [06:46, 49.24it/s]\n", 171 | "\"depth estimation\": 798it [00:14, 55.08it/s]\n", 172 | "\"point cloud\": 2699it [00:45, 59.18it/s]\n", 173 | "\"structured data\": 1458it [00:29, 50.17it/s]\n", 174 | "\"optical flow\": 1136it [00:24, 46.47it/s]\n", 175 | "\"reinforcement learning\": 10880it [03:09, 57.54it/s]\n", 176 | "\"super resolution\": 2107it [00:36, 57.58it/s]\n", 177 | "\"attention\": 20000it [05:51, 56.82it/s]\n", 178 | "\"tabular\": 824it [00:14, 56.69it/s]\n", 179 | "\"unsupervised learning\": 2139it [00:37, 57.74it/s]\n", 180 | "\"semi-supervised learning\": 0it [00:03, ?it/s]\n", 181 | "\"explainable\": 20000it [06:16, 53.13it/s]\n", 182 | "\"radiance field\": 82it [00:04, 19.99it/s]\n", 183 | "\"decision tree\": 1869it [00:31, 58.45it/s]\n", 184 | "\"time series\": 12496it [04:03, 51.28it/s]\n", 185 | "\"molecule\": 20000it [06:51, 48.56it/s]\n" 186 | ] 187 | } 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "metadata": { 193 | "colab": { 194 | "base_uri": "https://localhost:8080/", 195 | "height": 205 196 | }, 197 | "id": "GX7UXcI0a1sR", 198 | "outputId": "0ed04b65-ab62-4d11-91a9-5d73d4e67ca9" 199 | }, 200 | "source": [ 201 | "data = pd.DataFrame({\n", 202 | " 'titles': all_titles,\n", 203 | " 'summaries': all_summaries,\n", 204 | " 'terms': all_terms\n", 205 | "})\n", 206 | "data.head()" 207 | ], 208 | "execution_count": null, 209 | "outputs": [ 210 | { 211 | "output_type": "execute_result", 212 | "data": { 213 | "text/html": [ 214 | "
\n", 215 | "\n", 228 | "\n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | " \n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | "
titlessummariesterms
0Survey on Semantic Stereo Matching / Semantic ...Stereo matching is one of the widely used tech...[cs.CV, cs.LG]
1FUTURE-AI: Guiding Principles and Consensus Re...The recent advancements in artificial intellig...[cs.CV, cs.AI, cs.LG]
2Enforcing Mutual Consistency of Hard Regions f...In this paper, we proposed a novel mutual cons...[cs.CV, cs.AI]
3Parameter Decoupling Strategy for Semi-supervi...Consistency training has proven to be an advan...[cs.CV]
4Background-Foreground Segmentation for Interio...To ensure safety in automated driving, the cor...[cs.CV, cs.LG]
\n", 270 | "
" 271 | ], 272 | "text/plain": [ 273 | " titles ... terms\n", 274 | "0 Survey on Semantic Stereo Matching / Semantic ... ... [cs.CV, cs.LG]\n", 275 | "1 FUTURE-AI: Guiding Principles and Consensus Re... ... [cs.CV, cs.AI, cs.LG]\n", 276 | "2 Enforcing Mutual Consistency of Hard Regions f... ... [cs.CV, cs.AI]\n", 277 | "3 Parameter Decoupling Strategy for Semi-supervi... ... [cs.CV]\n", 278 | "4 Background-Foreground Segmentation for Interio... ... [cs.CV, cs.LG]\n", 279 | "\n", 280 | "[5 rows x 3 columns]" 281 | ] 282 | }, 283 | "metadata": {}, 284 | "execution_count": 5 285 | } 286 | ] 287 | }, 288 | { 289 | "cell_type": "code", 290 | "metadata": { 291 | "id": "LRjWIApOdTE0" 292 | }, 293 | "source": [ 294 | "data.to_csv('arxiv_data.csv', index=False)" 295 | ], 296 | "execution_count": null, 297 | "outputs": [] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "metadata": { 302 | "id": "YrcBGeBldxgc", 303 | "colab": { 304 | "base_uri": "https://localhost:8080/", 305 | "height": 17 306 | }, 307 | "outputId": "0a3d21a2-34e1-4dfb-fe50-79f9ef723829" 308 | }, 309 | "source": [ 310 | "from google.colab import files\n", 311 | "files.download('arxiv_data.csv') " 312 | ], 313 | "execution_count": null, 314 | "outputs": [ 315 | { 316 | "output_type": "display_data", 317 | "data": { 318 | "application/javascript": [ 319 | "\n", 320 | " async function download(id, filename, size) {\n", 321 | " if (!google.colab.kernel.accessAllowed) {\n", 322 | " return;\n", 323 | " }\n", 324 | " const div = document.createElement('div');\n", 325 | " const label = document.createElement('label');\n", 326 | " label.textContent = `Downloading \"${filename}\": `;\n", 327 | " div.appendChild(label);\n", 328 | " const progress = document.createElement('progress');\n", 329 | " progress.max = size;\n", 330 | " div.appendChild(progress);\n", 331 | " document.body.appendChild(div);\n", 332 | "\n", 333 | " const buffers = [];\n", 334 | " let downloaded = 0;\n", 335 | "\n", 336 | " const channel = await google.colab.kernel.comms.open(id);\n", 337 | " // Send a message to notify the kernel that we're ready.\n", 338 | " channel.send({})\n", 339 | "\n", 340 | " for await (const message of channel.messages) {\n", 341 | " // Send a message to notify the kernel that we're ready.\n", 342 | " channel.send({})\n", 343 | " if (message.buffers) {\n", 344 | " for (const buffer of message.buffers) {\n", 345 | " buffers.push(buffer);\n", 346 | " downloaded += buffer.byteLength;\n", 347 | " progress.value = downloaded;\n", 348 | " }\n", 349 | " }\n", 350 | " }\n", 351 | " const blob = new Blob(buffers, {type: 'application/binary'});\n", 352 | " const a = document.createElement('a');\n", 353 | " a.href = window.URL.createObjectURL(blob);\n", 354 | " a.download = filename;\n", 355 | " div.appendChild(a);\n", 356 | " a.click();\n", 357 | " div.remove();\n", 358 | " }\n", 359 | " " 360 | ], 361 | "text/plain": [ 362 | "" 363 | ] 364 | }, 365 | "metadata": {} 366 | }, 367 | { 368 | "output_type": "display_data", 369 | "data": { 370 | "application/javascript": [ 371 | "download(\"download_f004bf7a-a2b5-4190-8106-a09da6b00401\", \"arxiv_data.csv\", 67411252)" 372 | ], 373 | "text/plain": [ 374 | "" 375 | ] 376 | }, 377 | "metadata": {} 378 | } 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "metadata": { 384 | "id": "dqojCAC4uV8u" 385 | }, 386 | "source": [ 387 | "" 388 | ], 389 | "execution_count": null, 390 | "outputs": [] 391 | } 392 | ] 393 | } -------------------------------------------------------------------------------- /beam_arxiv_scrape.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "id": "66abf160", 7 | "metadata": { 8 | "id": "66abf160" 9 | }, 10 | "outputs": [], 11 | "source": [ 12 | "import apache_beam as beam\n", 13 | "import arxiv \n", 14 | "\n", 15 | "from apache_beam.dataframe.convert import to_dataframe\n", 16 | "from datetime import datetime" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 2, 22 | "id": "5443193c", 23 | "metadata": { 24 | "id": "5443193c" 25 | }, 26 | "outputs": [], 27 | "source": [ 28 | "query_keywords = [\n", 29 | " \"\\\"image segmentation\\\"\",\n", 30 | " \"\\\"self-supervised learning\\\"\",\n", 31 | " \"\\\"representation learning\\\"\",\n", 32 | " \"\\\"image generation\\\"\",\n", 33 | " \"\\\"object detection\\\"\",\n", 34 | " \"\\\"transfer learning\\\"\",\n", 35 | " \"\\\"transformers\\\"\",\n", 36 | " \"\\\"adversarial training\",\n", 37 | " \"\\\"generative adversarial networks\\\"\",\n", 38 | " \"\\\"model compressions\\\"\",\n", 39 | " \"\\\"image segmentation\\\"\",\n", 40 | " \"\\\"few-shot learning\\\"\",\n", 41 | " \"\\\"natural language\\\"\",\n", 42 | " \"\\\"graph\\\"\",\n", 43 | " \"\\\"colorization\\\"\",\n", 44 | " \"\\\"depth estimation\\\"\",\n", 45 | " \"\\\"point cloud\\\"\",\n", 46 | " \"\\\"structured data\\\"\",\n", 47 | " \"\\\"optical flow\\\"\",\n", 48 | " \"\\\"reinforcement learning\\\"\",\n", 49 | " \"\\\"super resolution\\\"\",\n", 50 | " \"\\\"attention\\\"\",\n", 51 | " \"\\\"tabular\\\"\",\n", 52 | " \"\\\"unsupervised learning\\\"\",\n", 53 | " \"\\\"semi-supervised learning\\\"\",\n", 54 | " \"\\\"explainable\\\"\",\n", 55 | " \"\\\"radiance field\\\"\",\n", 56 | " \"\\\"decision tree\\\"\",\n", 57 | " \"\\\"time series\\\"\",\n", 58 | " \"\\\"molecule\\\"\",\n", 59 | " \"\\\"physics\\\"\",\n", 60 | " \"\\\"graphics\\\"\",\n", 61 | " \"\\\"ray tracing\\\"\",\n", 62 | " \"\\\"optical flow\\\"\",\n", 63 | " \"\\\"photogrametry\\\"\",\n", 64 | "]" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 3, 70 | "id": "DI84CCdnY5Ek", 71 | "metadata": { 72 | "id": "DI84CCdnY5Ek" 73 | }, 74 | "outputs": [], 75 | "source": [ 76 | "import typing\n", 77 | "\n", 78 | "\n", 79 | "class ArxivEntries(typing.NamedTuple):\n", 80 | " terms: typing.List[str]\n", 81 | " titles: str\n", 82 | " abstracts: str" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 4, 88 | "id": "83ddf9d6", 89 | "metadata": { 90 | "id": "83ddf9d6" 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "client = arxiv.Client(num_retries=20, page_size=500)\n", 95 | "\n", 96 | "\n", 97 | "def query_with_keywords(query):\n", 98 | " search = arxiv.Search(\n", 99 | " query=query, max_results=20000, sort_by=arxiv.SortCriterion.LastUpdatedDate,\n", 100 | " )\n", 101 | "\n", 102 | " for res in client.results(search):\n", 103 | " if res.primary_category in [\"cs.CV\", \"stat.ML\", \"cs.LG\"]:\n", 104 | " yield beam.Row(\n", 105 | " terms=res.categories, titles=res.title, abstracts=res.summary\n", 106 | " )" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 5, 112 | "id": "SqeUCclZWwF1", 113 | "metadata": { 114 | "colab": { 115 | "base_uri": "https://localhost:8080/" 116 | }, 117 | "id": "SqeUCclZWwF1", 118 | "outputId": "e086cff2-8660-4079-9128-66839909a009" 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "Overwriting setup.py\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "%%writefile setup.py\n", 131 | "\n", 132 | "import setuptools\n", 133 | "\n", 134 | "\n", 135 | "NAME = \"gather_arxiv_data\"\n", 136 | "VERSION = \"0.1.0\"\n", 137 | "REQUIRED_PACKAGES = [\n", 138 | " \"apache_beam==2.32.0\",\n", 139 | " \"pandas==1.3.2\",\n", 140 | " \"arxiv==1.4.2\",\n", 141 | " \"google_cloud_storage==1.42.1\",\n", 142 | "]\n", 143 | "\n", 144 | "\n", 145 | "setuptools.setup(\n", 146 | " name=NAME,\n", 147 | " version=VERSION,\n", 148 | " install_requires=REQUIRED_PACKAGES,\n", 149 | " packages=setuptools.find_packages(),\n", 150 | " include_package_data=True,\n", 151 | ")" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 6, 157 | "id": "2196906b", 158 | "metadata": { 159 | "id": "2196906b" 160 | }, 161 | "outputs": [], 162 | "source": [ 163 | "gcs_bucket_name = \"arxiv-data-nlp\"\n", 164 | "gcp_project = \"####\" # Specify this.\n", 165 | "\n", 166 | "pipeline_args = {\n", 167 | " \"job_name\": f'arxiv-data-{datetime.utcnow().strftime(\"%y%m%d-%H%M%S\")}',\n", 168 | " \"num_workers\": \"4\",\n", 169 | " \"runner\": \"DataflowRunner\",\n", 170 | " \"setup_file\": \"./setup.py\",\n", 171 | " \"project\": gcp_project,\n", 172 | " \"region\": \"us-central1\",\n", 173 | " \"gcs_location\": f\"gs://{gcs_bucket_name}\",\n", 174 | " \"temp_location\": f\"gs://{gcs_bucket_name}/temp\",\n", 175 | " \"staging_location\": f\"gs://{gcs_bucket_name}/staging\",\n", 176 | " \"save_main_session\": \"True\",\n", 177 | "}\n", 178 | "\n", 179 | "# Convert the dictionary to a list of (argument, value) tuples and then flatten the list.\n", 180 | "pipeline_args = [(f\"--{k}\", v) for k, v in pipeline_args.items()]\n", 181 | "pipeline_args = [x for y in pipeline_args for x in y]" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": 7, 187 | "id": "d58affea", 188 | "metadata": { 189 | "colab": { 190 | "base_uri": "https://localhost:8080/", 191 | "height": 766 192 | }, 193 | "id": "d58affea", 194 | "outputId": "23852f9e-bf43-4b2a-cd3a-6180761a904f" 195 | }, 196 | "outputs": [ 197 | { 198 | "name": "stderr", 199 | "output_type": "stream", 200 | "text": [ 201 | "WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.\n" 202 | ] 203 | }, 204 | { 205 | "data": { 206 | "application/javascript": [ 207 | "\n", 208 | " if (typeof window.interactive_beam_jquery == 'undefined') {\n", 209 | " var jqueryScript = document.createElement('script');\n", 210 | " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n", 211 | " jqueryScript.type = 'text/javascript';\n", 212 | " jqueryScript.onload = function() {\n", 213 | " var datatableScript = document.createElement('script');\n", 214 | " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n", 215 | " datatableScript.type = 'text/javascript';\n", 216 | " datatableScript.onload = function() {\n", 217 | " window.interactive_beam_jquery = jQuery.noConflict(true);\n", 218 | " window.interactive_beam_jquery(document).ready(function($){\n", 219 | " \n", 220 | " });\n", 221 | " }\n", 222 | " document.head.appendChild(datatableScript);\n", 223 | " };\n", 224 | " document.head.appendChild(jqueryScript);\n", 225 | " } else {\n", 226 | " window.interactive_beam_jquery(document).ready(function($){\n", 227 | " \n", 228 | " });\n", 229 | " }" 230 | ] 231 | }, 232 | "metadata": {}, 233 | "output_type": "display_data" 234 | }, 235 | { 236 | "name": "stderr", 237 | "output_type": "stream", 238 | "text": [ 239 | "/Users/sayakpaul/.local/bin/.virtualenvs/tf/lib/python3.8/site-packages/apache_beam/dataframe/io.py:566: FutureWarning: WriteToFiles is experimental.\n", 240 | " return pcoll | fileio.WriteToFiles(\n", 241 | "/Users/sayakpaul/.local/bin/.virtualenvs/tf/lib/python3.8/site-packages/apache_beam/io/fileio.py:535: BeamDeprecationWarning: options is deprecated since First stable release. References to .options will not be supported\n", 242 | " p.options.view_as(GoogleCloudOptions).temp_location or\n", 243 | "warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n", 244 | "\n", 245 | "warning: check: missing required meta-data: url\n", 246 | "\n", 247 | "warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied\n", 248 | "\n", 249 | "WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.\n", 250 | "WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--gcs_location', 'gs://arxiv-data-nlp', 'True']\n", 251 | "WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--gcs_location', 'gs://arxiv-data-nlp', 'True']\n" 252 | ] 253 | } 254 | ], 255 | "source": [ 256 | "with beam.Pipeline(argv=pipeline_args) as pipeline:\n", 257 | " keywords = pipeline | beam.Create(query_keywords)\n", 258 | " records = keywords | beam.FlatMap(query_with_keywords).with_output_types(ArxivEntries)\n", 259 | " _ = to_dataframe(records).to_csv(\n", 260 | " f\"gs://{gcs_bucket_name}/arxiv/sample.csv\", index=False\n", 261 | " )" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 8, 267 | "id": "cb97c245", 268 | "metadata": {}, 269 | "outputs": [ 270 | { 271 | "name": "stdout", 272 | "output_type": "stream", 273 | "text": [ 274 | "\n", 275 | "\n", 276 | "Updates are available for some Cloud SDK components. To install them,\n", 277 | "please run:\n", 278 | " $ gcloud components update\n", 279 | "\n", 280 | "gs://arxiv-data-nlp/arxiv/:\n", 281 | "gs://arxiv-data-nlp/arxiv/sample.csv-00000-of-00020\n", 282 | "gs://arxiv-data-nlp/arxiv/sample.csv-00001-of-00020\n", 283 | "gs://arxiv-data-nlp/arxiv/sample.csv-00002-of-00020\n", 284 | "gs://arxiv-data-nlp/arxiv/sample.csv-00003-of-00020\n", 285 | "gs://arxiv-data-nlp/arxiv/sample.csv-00004-of-00020\n", 286 | "gs://arxiv-data-nlp/arxiv/sample.csv-00005-of-00020\n", 287 | "gs://arxiv-data-nlp/arxiv/sample.csv-00006-of-00020\n", 288 | "gs://arxiv-data-nlp/arxiv/sample.csv-00007-of-00020\n", 289 | "gs://arxiv-data-nlp/arxiv/sample.csv-00008-of-00020\n", 290 | "gs://arxiv-data-nlp/arxiv/sample.csv-00009-of-00020\n", 291 | "gs://arxiv-data-nlp/arxiv/sample.csv-00010-of-00020\n", 292 | "gs://arxiv-data-nlp/arxiv/sample.csv-00011-of-00020\n", 293 | "gs://arxiv-data-nlp/arxiv/sample.csv-00012-of-00020\n", 294 | "gs://arxiv-data-nlp/arxiv/sample.csv-00013-of-00020\n", 295 | "gs://arxiv-data-nlp/arxiv/sample.csv-00014-of-00020\n", 296 | "gs://arxiv-data-nlp/arxiv/sample.csv-00015-of-00020\n", 297 | "gs://arxiv-data-nlp/arxiv/sample.csv-00016-of-00020\n", 298 | "gs://arxiv-data-nlp/arxiv/sample.csv-00017-of-00020\n", 299 | "gs://arxiv-data-nlp/arxiv/sample.csv-00018-of-00020\n", 300 | "gs://arxiv-data-nlp/arxiv/sample.csv-00019-of-00020\n" 301 | ] 302 | } 303 | ], 304 | "source": [ 305 | "!gsutil ls -R gs://{gcs_bucket_name}/arxiv/" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": 9, 311 | "id": "f98d917f", 312 | "metadata": {}, 313 | "outputs": [ 314 | { 315 | "name": "stdout", 316 | "output_type": "stream", 317 | "text": [ 318 | "Copying gs://arxiv-data-nlp/arxiv/sample.csv-00000-of-00020...\n", 319 | "| [1 files][ 1.9 MiB/ 1.9 MiB] 347.4 KiB/s \n", 320 | "Operation completed over 1 objects/1.9 MiB. \n" 321 | ] 322 | } 323 | ], 324 | "source": [ 325 | "!gsutil cp gs://arxiv-data-nlp/arxiv/sample.csv-00000-of-00020 ." 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": 10, 331 | "id": "daedd023", 332 | "metadata": { 333 | "id": "daedd023", 334 | "outputId": "e59c8622-02ca-4bfe-adfc-cec0208602fc" 335 | }, 336 | "outputs": [ 337 | { 338 | "data": { 339 | "text/html": [ 340 | "
\n", 341 | "\n", 354 | "\n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | " \n", 375 | " \n", 376 | " \n", 377 | " \n", 378 | " \n", 379 | " \n", 380 | " \n", 381 | " \n", 382 | " \n", 383 | " \n", 384 | " \n", 385 | " \n", 386 | " \n", 387 | " \n", 388 | " \n", 389 | " \n", 390 | " \n", 391 | " \n", 392 | " \n", 393 | " \n", 394 | " \n", 395 | "
termstitlesabstracts
0['cs.LG', 'cs.AI']Self-supervised Learning on Graphs: Contrastiv...Deep learning on graphs has recently achieved ...
1['cs.CV', 'cs.CL']Contrastive Video-Language SegmentationWe focus on the problem of segmenting a certai...
2['cs.LG']What to Prioritize? Natural Language Processin...Managing large numbers of incoming bug reports...
3['cs.CV', 'cs.RO']The VVAD-LRS3 Dataset for Visual Voice Activit...Robots are becoming everyday devices, increasi...
4['cs.CV']UTNet: A Hybrid Transformer Architecture for M...Transformer architecture has emerged to be suc...
\n", 396 | "
" 397 | ], 398 | "text/plain": [ 399 | " terms titles \\\n", 400 | "0 ['cs.LG', 'cs.AI'] Self-supervised Learning on Graphs: Contrastiv... \n", 401 | "1 ['cs.CV', 'cs.CL'] Contrastive Video-Language Segmentation \n", 402 | "2 ['cs.LG'] What to Prioritize? Natural Language Processin... \n", 403 | "3 ['cs.CV', 'cs.RO'] The VVAD-LRS3 Dataset for Visual Voice Activit... \n", 404 | "4 ['cs.CV'] UTNet: A Hybrid Transformer Architecture for M... \n", 405 | "\n", 406 | " abstracts \n", 407 | "0 Deep learning on graphs has recently achieved ... \n", 408 | "1 We focus on the problem of segmenting a certai... \n", 409 | "2 Managing large numbers of incoming bug reports... \n", 410 | "3 Robots are becoming everyday devices, increasi... \n", 411 | "4 Transformer architecture has emerged to be suc... " 412 | ] 413 | }, 414 | "execution_count": 10, 415 | "metadata": {}, 416 | "output_type": "execute_result" 417 | } 418 | ], 419 | "source": [ 420 | "import pandas as pd\n", 421 | "\n", 422 | "\n", 423 | "df = pd.read_csv(\"sample.csv-00000-of-00020\")\n", 424 | "df.head()" 425 | ] 426 | }, 427 | { 428 | "cell_type": "markdown", 429 | "id": "7cba00e9", 430 | "metadata": { 431 | "id": "7cba00e9" 432 | }, 433 | "source": [ 434 | "## Acknowledgements\n", 435 | "\n", 436 | "* [Lukas Schwab](https://github.com/lukasschwab)\n", 437 | "* [Robert Bradshaw](https://www.linkedin.com/in/robert-bradshaw-1b48a07/)" 438 | ] 439 | } 440 | ], 441 | "metadata": { 442 | "colab": { 443 | "collapsed_sections": [], 444 | "name": "beam_arxiv.ipynb", 445 | "provenance": [] 446 | }, 447 | "kernelspec": { 448 | "display_name": "Python 3 (ipykernel)", 449 | "language": "python", 450 | "name": "python3" 451 | }, 452 | "language_info": { 453 | "codemirror_mode": { 454 | "name": "ipython", 455 | "version": 3 456 | }, 457 | "file_extension": ".py", 458 | "mimetype": "text/x-python", 459 | "name": "python", 460 | "nbconvert_exporter": "python", 461 | "pygments_lexer": "ipython3", 462 | "version": "3.8.2" 463 | } 464 | }, 465 | "nbformat": 4, 466 | "nbformat_minor": 5 467 | } 468 | -------------------------------------------------------------------------------- /multi_label_trainer_tfidf.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "Q9ZG7WNCHNqw" 7 | }, 8 | "source": [ 9 | "## Introduction\n", 10 | "\n", 11 | "In this example, we will build a multi-label text classifier to predict the subject areas of arXiv papers from their abstract bodies. This type of classifier can be useful for conference submission portals like [OpenReview](https://openreview.net/). Given a paper abstract, the portal could provide suggestions on which areas the underlying paper would best belong to.\n", 12 | "\n", 13 | "The dataset was collected using the [`arXiv` Python library](https://github.com/lukasschwab/arxiv.py) that provides a wrapper around the [original arXiv API](http://arxiv.org/help/api/index). To know more, please refer to [this notebook](https://github.com/soumik12345/multi-label-text-classification/blob/master/arxiv_scrape.ipynb). " 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": { 19 | "id": "5GICQpY-zws7" 20 | }, 21 | "source": [ 22 | "## Imports" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 28, 28 | "metadata": { 29 | "id": "ho5uPff1fLoH" 30 | }, 31 | "outputs": [], 32 | "source": [ 33 | "from tensorflow.keras import layers\n", 34 | "from tensorflow import keras\n", 35 | "import tensorflow as tf\n", 36 | "\n", 37 | "from sklearn.preprocessing import MultiLabelBinarizer\n", 38 | "from sklearn.model_selection import train_test_split\n", 39 | "import matplotlib.pyplot as plt\n", 40 | "from ast import literal_eval\n", 41 | "import pandas as pd\n", 42 | "import numpy as np" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "id": "qyeQtKSezymP" 49 | }, 50 | "source": [ 51 | "## Read data and perform basic EDA\n", 52 | "\n", 53 | "In this section, we first load the dataset into a `pandas` dataframe and then perform some basic exploratory data analysis (EDA)." 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": { 60 | "colab": { 61 | "base_uri": "https://localhost:8080/", 62 | "height": 287 63 | }, 64 | "id": "yFo2pNYbf2Du", 65 | "outputId": "a0478ffb-4612-4f5b-846b-8b33a502bfed" 66 | }, 67 | "outputs": [ 68 | { 69 | "data": { 70 | "text/html": [ 71 | "
\n", 72 | "\n", 85 | "\n", 86 | " \n", 87 | " \n", 88 | " \n", 89 | " \n", 90 | " \n", 91 | " \n", 92 | " \n", 93 | " \n", 94 | " \n", 95 | " \n", 96 | " \n", 97 | " \n", 98 | " \n", 99 | " \n", 100 | " \n", 101 | " \n", 102 | " \n", 103 | " \n", 104 | " \n", 105 | " \n", 106 | " \n", 107 | " \n", 108 | " \n", 109 | " \n", 110 | " \n", 111 | " \n", 112 | " \n", 113 | " \n", 114 | " \n", 115 | " \n", 116 | " \n", 117 | " \n", 118 | " \n", 119 | " \n", 120 | " \n", 121 | " \n", 122 | " \n", 123 | " \n", 124 | " \n", 125 | " \n", 126 | "
titlessummariesterms
0Survey on Semantic Stereo Matching / Semantic ...Stereo matching is one of the widely used tech...['cs.CV', 'cs.LG']
1FUTURE-AI: Guiding Principles and Consensus Re...The recent advancements in artificial intellig...['cs.CV', 'cs.AI', 'cs.LG']
2Enforcing Mutual Consistency of Hard Regions f...In this paper, we proposed a novel mutual cons...['cs.CV', 'cs.AI']
3Parameter Decoupling Strategy for Semi-supervi...Consistency training has proven to be an advan...['cs.CV']
4Background-Foreground Segmentation for Interio...To ensure safety in automated driving, the cor...['cs.CV', 'cs.LG']
\n", 127 | "
" 128 | ], 129 | "text/plain": [ 130 | " titles ... terms\n", 131 | "0 Survey on Semantic Stereo Matching / Semantic ... ... ['cs.CV', 'cs.LG']\n", 132 | "1 FUTURE-AI: Guiding Principles and Consensus Re... ... ['cs.CV', 'cs.AI', 'cs.LG']\n", 133 | "2 Enforcing Mutual Consistency of Hard Regions f... ... ['cs.CV', 'cs.AI']\n", 134 | "3 Parameter Decoupling Strategy for Semi-supervi... ... ['cs.CV']\n", 135 | "4 Background-Foreground Segmentation for Interio... ... ['cs.CV', 'cs.LG']\n", 136 | "\n", 137 | "[5 rows x 3 columns]" 138 | ] 139 | }, 140 | "execution_count": 2, 141 | "metadata": {}, 142 | "output_type": "execute_result" 143 | } 144 | ], 145 | "source": [ 146 | "arxiv_data = pd.read_csv(\n", 147 | " \"https://github.com/soumik12345/multi-label-text-classification/releases/download/v0.2/arxiv_data.csv\"\n", 148 | ")\n", 149 | "arxiv_data.head()" 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": { 155 | "id": "djk-kXWvHNq3" 156 | }, 157 | "source": [ 158 | "Our text features are present in the `summaries` column and their corresponding labels are in `terms`. As we can notice there are multiple categories associated with a particular entry. " 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": { 165 | "colab": { 166 | "base_uri": "https://localhost:8080/" 167 | }, 168 | "id": "Em_8mJvUgKY-", 169 | "outputId": "8ce024c1-2745-4ab2-dd18-2b153784b4fc" 170 | }, 171 | "outputs": [ 172 | { 173 | "name": "stdout", 174 | "output_type": "stream", 175 | "text": [ 176 | "There are 51774 rows in the dataset.\n" 177 | ] 178 | } 179 | ], 180 | "source": [ 181 | "print(f\"There are {len(arxiv_data)} rows in the dataset.\")" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": { 187 | "id": "P3bYjP_0HNq4" 188 | }, 189 | "source": [ 190 | "Real-world data is noisy. One of the most commonly observed such noise is data duplication. Here we notice that our initial dataset has got about 13k duplicate entries. " 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "metadata": { 197 | "colab": { 198 | "base_uri": "https://localhost:8080/" 199 | }, 200 | "id": "k9Vb9jtK5zjg", 201 | "outputId": "92e336c0-921a-4412-87ba-e3872333e3ff" 202 | }, 203 | "outputs": [ 204 | { 205 | "name": "stdout", 206 | "output_type": "stream", 207 | "text": [ 208 | "There are 12802 duplicate titles.\n" 209 | ] 210 | } 211 | ], 212 | "source": [ 213 | "total_duplicate_titles = sum(arxiv_data[\"titles\"].duplicated())\n", 214 | "print(f\"There are {total_duplicate_titles} duplicate titles.\")" 215 | ] 216 | }, 217 | { 218 | "cell_type": "markdown", 219 | "metadata": { 220 | "id": "sEBbarGSHNq5" 221 | }, 222 | "source": [ 223 | "Before proceeding further we first drop these entries. " 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": { 230 | "colab": { 231 | "base_uri": "https://localhost:8080/" 232 | }, 233 | "id": "2259X-rf6OLY", 234 | "outputId": "08d73f4b-f0a0-46bd-9744-01b0e22f6a92" 235 | }, 236 | "outputs": [ 237 | { 238 | "name": "stdout", 239 | "output_type": "stream", 240 | "text": [ 241 | "There are 38972 rows in the deduplicated dataset.\n" 242 | ] 243 | } 244 | ], 245 | "source": [ 246 | "arxiv_data = arxiv_data[~arxiv_data[\"titles\"].duplicated()]\n", 247 | "print(f\"There are {len(arxiv_data)} rows in the deduplicated dataset.\")" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": null, 253 | "metadata": { 254 | "colab": { 255 | "base_uri": "https://localhost:8080/" 256 | }, 257 | "id": "TmgkCCr2g0w5", 258 | "outputId": "18a6bd58-e511-405f-c19e-15c40c8955c9" 259 | }, 260 | "outputs": [ 261 | { 262 | "name": "stdout", 263 | "output_type": "stream", 264 | "text": [ 265 | "2321\n" 266 | ] 267 | } 268 | ], 269 | "source": [ 270 | "# There are some terms with occurrence as low as 1.\n", 271 | "print(sum(arxiv_data[\"terms\"].value_counts() == 1))" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": null, 277 | "metadata": { 278 | "colab": { 279 | "base_uri": "https://localhost:8080/" 280 | }, 281 | "id": "hyBJNHMdifJ-", 282 | "outputId": "0a098fa6-2723-42b9-b70b-f18bc9c11f3b" 283 | }, 284 | "outputs": [ 285 | { 286 | "name": "stdout", 287 | "output_type": "stream", 288 | "text": [ 289 | "3157\n" 290 | ] 291 | } 292 | ], 293 | "source": [ 294 | "# How many unique terms?\n", 295 | "print(arxiv_data[\"terms\"].nunique())" 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": { 301 | "id": "dPvNGph0HNq7" 302 | }, 303 | "source": [ 304 | "As observed above, out of 3157 unique combinations of `terms`, 2321 entries have the lowest occurrence. To prepare our train, validation, and test sets with [stratification](https://en.wikipedia.org/wiki/Stratified_sampling), we need to drop these terms. " 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": null, 310 | "metadata": { 311 | "colab": { 312 | "base_uri": "https://localhost:8080/" 313 | }, 314 | "id": "77ZoCzrMhLxc", 315 | "outputId": "ee113f1e-23e3-4fd8-8ecd-4231df22da22" 316 | }, 317 | "outputs": [ 318 | { 319 | "data": { 320 | "text/plain": [ 321 | "(36651, 3)" 322 | ] 323 | }, 324 | "execution_count": 8, 325 | "metadata": {}, 326 | "output_type": "execute_result" 327 | } 328 | ], 329 | "source": [ 330 | "# Filtering the rare terms.\n", 331 | "arxiv_data_filtered = arxiv_data.groupby(\"terms\").filter(lambda x: len(x) > 1)\n", 332 | "arxiv_data_filtered.shape" 333 | ] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": { 338 | "id": "MxrG9tim0QNr" 339 | }, 340 | "source": [ 341 | "## Convert the string labels to list of strings\n", 342 | "\n", 343 | "The initial labels are represented as raw strings. Here we make them `List[str]` for a more compact representation. " 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "metadata": { 350 | "colab": { 351 | "base_uri": "https://localhost:8080/" 352 | }, 353 | "id": "LIEGLc61iwbQ", 354 | "outputId": "e5612b8b-145c-45b7-f6ed-1273ec8bd538" 355 | }, 356 | "outputs": [ 357 | { 358 | "data": { 359 | "text/plain": [ 360 | "array([list(['cs.CV', 'cs.LG']), list(['cs.CV', 'cs.AI', 'cs.LG']),\n", 361 | " list(['cs.CV', 'cs.AI']), list(['cs.CV']),\n", 362 | " list(['cs.CV', 'cs.LG'])], dtype=object)" 363 | ] 364 | }, 365 | "execution_count": 9, 366 | "metadata": {}, 367 | "output_type": "execute_result" 368 | } 369 | ], 370 | "source": [ 371 | "arxiv_data_filtered[\"terms\"] = arxiv_data_filtered[\"terms\"].apply(\n", 372 | " lambda x: literal_eval(x)\n", 373 | ")\n", 374 | "arxiv_data_filtered[\"terms\"].values[:5]" 375 | ] 376 | }, 377 | { 378 | "cell_type": "markdown", 379 | "metadata": { 380 | "id": "zjFB8Uoo0cXM" 381 | }, 382 | "source": [ 383 | "## Stratified splits because of class imbalance\n", 384 | "\n", 385 | "The dataset has a [class imbalance problem](https://developers.google.com/machine-learning/glossary/#class-imbalanced-dataset). So, to have a fair evaluation result, we need to ensure the datasets are sampled with stratification. To know more about different strategies to deal with the class imbalance problem, you can follow [this tutorial](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data). For an end-to-end demonstration of classification with imbablanced data, refer to [Imbalanced classification: credit card fraud detection](https://keras.io/examples/structured_data/imbalanced_classification/)." 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "metadata": { 392 | "colab": { 393 | "base_uri": "https://localhost:8080/" 394 | }, 395 | "id": "EbKDVTKPgOKe", 396 | "outputId": "f16da845-9641-42dd-c387-649facc014a9" 397 | }, 398 | "outputs": [ 399 | { 400 | "name": "stdout", 401 | "output_type": "stream", 402 | "text": [ 403 | "Number of rows in training set: 32985\n", 404 | "Number of rows in validation set: 1833\n", 405 | "Number of rows in test set: 1833\n" 406 | ] 407 | }, 408 | { 409 | "name": "stderr", 410 | "output_type": "stream", 411 | "text": [ 412 | "/usr/local/lib/python3.7/dist-packages/pandas/core/frame.py:4174: SettingWithCopyWarning: \n", 413 | "A value is trying to be set on a copy of a slice from a DataFrame\n", 414 | "\n", 415 | "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n", 416 | " errors=errors,\n" 417 | ] 418 | } 419 | ], 420 | "source": [ 421 | "test_split = 0.1\n", 422 | "\n", 423 | "# Initial train and test split.\n", 424 | "train_df, test_df = train_test_split(\n", 425 | " arxiv_data_filtered,\n", 426 | " test_size=test_split,\n", 427 | " stratify=arxiv_data_filtered[\"terms\"].values,\n", 428 | ")\n", 429 | "\n", 430 | "# Splitting the test set further into validation\n", 431 | "# and new test sets.\n", 432 | "val_df = test_df.sample(frac=0.5)\n", 433 | "test_df.drop(val_df.index, inplace=True)\n", 434 | "\n", 435 | "print(f\"Number of rows in training set: {len(train_df)}\")\n", 436 | "print(f\"Number of rows in validation set: {len(val_df)}\")\n", 437 | "print(f\"Number of rows in test set: {len(test_df)}\")" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": { 443 | "id": "96Ew2PPI0lVc" 444 | }, 445 | "source": [ 446 | "## Multi-label binarization\n", 447 | "\n", 448 | "Now we preprocess our labels using [`MultiLabelBinarizer`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html). " 449 | ] 450 | }, 451 | { 452 | "cell_type": "code", 453 | "execution_count": null, 454 | "metadata": { 455 | "colab": { 456 | "base_uri": "https://localhost:8080/" 457 | }, 458 | "id": "1vgxbdwGf07E", 459 | "outputId": "1786e58c-9f56-4c14-f9cb-1f4da7fdad85" 460 | }, 461 | "outputs": [ 462 | { 463 | "data": { 464 | "text/plain": [ 465 | "array(['14J60 (Primary) 14F05, 14J26 (Secondary)', '62H30', '62H35',\n", 466 | " '62H99', '65D19', '68', '68Q32', '68T01', '68T05', '68T07',\n", 467 | " '68T10', '68T30', '68T45', '68T99', '68Txx', '68U01', '68U10',\n", 468 | " 'E.5; E.4; E.2; H.1.1; F.1.1; F.1.3', 'F.2.2; I.2.7', 'G.3',\n", 469 | " 'H.3.1; H.3.3; I.2.6; I.2.7', 'H.3.1; I.2.6; I.2.7', 'I.2',\n", 470 | " 'I.2.0; I.2.6', 'I.2.1', 'I.2.10', 'I.2.10; I.2.6',\n", 471 | " 'I.2.10; I.4.8', 'I.2.10; I.4.8; I.5.4', 'I.2.10; I.4; I.5',\n", 472 | " 'I.2.10; I.5.1; I.4.8', 'I.2.1; J.3', 'I.2.6', 'I.2.6, I.5.4',\n", 473 | " 'I.2.6; I.2.10', 'I.2.6; I.2.7', 'I.2.6; I.2.7; H.3.1; H.3.3',\n", 474 | " 'I.2.6; I.2.8', 'I.2.6; I.2.9', 'I.2.6; I.5.1', 'I.2.6; I.5.4',\n", 475 | " 'I.2.7', 'I.2.8', 'I.2; I.2.6; I.2.7', 'I.2; I.4; I.5', 'I.2; I.5',\n", 476 | " 'I.2; J.2', 'I.4', 'I.4.0', 'I.4.3', 'I.4.4', 'I.4.5', 'I.4.6',\n", 477 | " 'I.4.6; I.4.8', 'I.4.8', 'I.4.9', 'I.4.9; I.5.4', 'I.4; I.5',\n", 478 | " 'I.5.4', 'K.3.2', 'astro-ph.IM', 'cond-mat.dis-nn',\n", 479 | " 'cond-mat.mtrl-sci', 'cond-mat.soft', 'cond-mat.stat-mech',\n", 480 | " 'cs.AI', 'cs.AR', 'cs.CC', 'cs.CE', 'cs.CG', 'cs.CL', 'cs.CR',\n", 481 | " 'cs.CV', 'cs.CY', 'cs.DB', 'cs.DC', 'cs.DM', 'cs.DS', 'cs.ET',\n", 482 | " 'cs.FL', 'cs.GR', 'cs.GT', 'cs.HC', 'cs.IR', 'cs.IT', 'cs.LG',\n", 483 | " 'cs.LO', 'cs.MA', 'cs.MM', 'cs.MS', 'cs.NA', 'cs.NE', 'cs.NI',\n", 484 | " 'cs.PF', 'cs.PL', 'cs.RO', 'cs.SC', 'cs.SD', 'cs.SE', 'cs.SI',\n", 485 | " 'cs.SY', 'econ.EM', 'econ.GN', 'eess.AS', 'eess.IV', 'eess.SP',\n", 486 | " 'eess.SY', 'hep-ex', 'hep-ph', 'math.AP', 'math.AT', 'math.CO',\n", 487 | " 'math.DS', 'math.FA', 'math.IT', 'math.LO', 'math.NA', 'math.OC',\n", 488 | " 'math.PR', 'math.ST', 'nlin.AO', 'nlin.CD', 'physics.ao-ph',\n", 489 | " 'physics.bio-ph', 'physics.chem-ph', 'physics.comp-ph',\n", 490 | " 'physics.data-an', 'physics.flu-dyn', 'physics.geo-ph',\n", 491 | " 'physics.med-ph', 'physics.optics', 'physics.soc-ph', 'q-bio.BM',\n", 492 | " 'q-bio.GN', 'q-bio.MN', 'q-bio.NC', 'q-bio.OT', 'q-bio.QM',\n", 493 | " 'q-bio.TO', 'q-fin.CP', 'q-fin.EC', 'q-fin.GN', 'q-fin.PM',\n", 494 | " 'q-fin.RM', 'q-fin.ST', 'q-fin.TR', 'quant-ph', 'stat.AP',\n", 495 | " 'stat.CO', 'stat.ME', 'stat.ML', 'stat.TH'], dtype=object)" 496 | ] 497 | }, 498 | "execution_count": 11, 499 | "metadata": {}, 500 | "output_type": "execute_result" 501 | } 502 | ], 503 | "source": [ 504 | "mlb = MultiLabelBinarizer()\n", 505 | "mlb.fit_transform(train_df[\"terms\"])\n", 506 | "mlb.classes_" 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "metadata": { 512 | "id": "HTcmRnP9HNq9" 513 | }, 514 | "source": [ 515 | "`MultiLabelBinarizer`separates out the individual unique classes available from the label pool and then uses this information to represent a given label set with 0's and 1's. Below is an example. " 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": null, 521 | "metadata": { 522 | "colab": { 523 | "base_uri": "https://localhost:8080/" 524 | }, 525 | "id": "yfMO31s8HNq9", 526 | "outputId": "7a6b1cb9-761b-409c-fbc5-9534f5168629" 527 | }, 528 | "outputs": [ 529 | { 530 | "name": "stdout", 531 | "output_type": "stream", 532 | "text": [ 533 | "Original label: ['cs.LG', 'cs.AI', 'stat.ML']\n", 534 | "Label-binarized representation: [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 535 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0\n", 536 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 537 | " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", 538 | " 0 0 0 0 0 0 1 0]]\n" 539 | ] 540 | } 541 | ], 542 | "source": [ 543 | "sample_label = train_df[\"terms\"].iloc[0]\n", 544 | "print(f\"Original label: {sample_label}\")\n", 545 | "\n", 546 | "label_binarized = mlb.transform([sample_label])\n", 547 | "print(f\"Label-binarized representation: {label_binarized}\")" 548 | ] 549 | }, 550 | { 551 | "cell_type": "markdown", 552 | "metadata": { 553 | "id": "a2kFVBCG0oXV" 554 | }, 555 | "source": [ 556 | "## Data preprocessing and `tf.data.Dataset` objects\n", 557 | "\n", 558 | "We first get percentile estimates of the sequence lengths. The purpose will be clear in a moment." 559 | ] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "execution_count": null, 564 | "metadata": { 565 | "colab": { 566 | "base_uri": "https://localhost:8080/" 567 | }, 568 | "id": "kCR-_Iw3gyT-", 569 | "outputId": "1e00b7b1-b70a-4f2f-f97a-4defa37d3646" 570 | }, 571 | "outputs": [ 572 | { 573 | "data": { 574 | "text/plain": [ 575 | "count 32985.000000\n", 576 | "mean 156.629650\n", 577 | "std 41.521087\n", 578 | "min 5.000000\n", 579 | "25% 128.000000\n", 580 | "50% 155.000000\n", 581 | "75% 183.000000\n", 582 | "max 462.000000\n", 583 | "Name: summaries, dtype: float64" 584 | ] 585 | }, 586 | "execution_count": 13, 587 | "metadata": {}, 588 | "output_type": "execute_result" 589 | } 590 | ], 591 | "source": [ 592 | "train_df[\"summaries\"].apply(lambda x: len(x.split(\" \"))).describe()" 593 | ] 594 | }, 595 | { 596 | "cell_type": "markdown", 597 | "metadata": { 598 | "id": "2rdivJop02xG" 599 | }, 600 | "source": [ 601 | "Notice that 50% of the abstracts have a length of 154 (you may get a different number based on the split). So, any number near that is a good enough approximate for the maximum sequence length. \n", 602 | "\n", 603 | "Now, we write utilities to prepare our datasets that would go straight to the text classifier model. " 604 | ] 605 | }, 606 | { 607 | "cell_type": "code", 608 | "execution_count": null, 609 | "metadata": { 610 | "id": "QoMgEZrtVBDS" 611 | }, 612 | "outputs": [], 613 | "source": [ 614 | "max_seqlen = 150\n", 615 | "batch_size = 128\n", 616 | "padding_token = \"\"\n", 617 | "\n", 618 | "\n", 619 | "def unify_text_length(text, label):\n", 620 | " # Split the given abstract and calculate its length.\n", 621 | " word_splits = tf.strings.split(text, sep=\" \")\n", 622 | " sequence_length = tf.shape(word_splits)[0]\n", 623 | " \n", 624 | " # Calculate the padding amount.\n", 625 | " padding_amount = max_seqlen - sequence_length\n", 626 | " \n", 627 | " # Check if we need to pad or truncate.\n", 628 | " if padding_amount > 0:\n", 629 | " unified_text = tf.pad([text], [[0, padding_amount]], constant_values=\"\")\n", 630 | " unified_text = tf.strings.reduce_join(unified_text, separator=\"\")\n", 631 | " else:\n", 632 | " unified_text = tf.strings.reduce_join(word_splits[:max_seqlen], separator=\" \")\n", 633 | " \n", 634 | " # The expansion is needed for subsequent vectorization.\n", 635 | " return tf.expand_dims(unified_text, -1), label\n", 636 | "\n", 637 | "\n", 638 | "def make_dataset(dataframe, is_train=True):\n", 639 | " label_binarized = mlb.transform(dataframe[\"terms\"].values)\n", 640 | " dataset = tf.data.Dataset.from_tensor_slices(\n", 641 | " (dataframe[\"summaries\"].values, label_binarized)\n", 642 | " )\n", 643 | " dataset = dataset.shuffle(batch_size * 10) if is_train else dataset\n", 644 | " dataset = dataset.map(unify_text_length).cache()\n", 645 | " return dataset.batch(batch_size).prefetch(tf.data.AUTOTUNE)" 646 | ] 647 | }, 648 | { 649 | "cell_type": "markdown", 650 | "metadata": { 651 | "id": "fi5g9v3HHNq-" 652 | }, 653 | "source": [ 654 | "Now we can prepare the `tf.data.Dataset` objects. " 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": null, 660 | "metadata": { 661 | "id": "F_vrgkCXrWOS" 662 | }, 663 | "outputs": [], 664 | "source": [ 665 | "train_dataset = make_dataset(train_df, is_train=True)\n", 666 | "validation_dataset = make_dataset(val_df, is_train=False)\n", 667 | "test_dataset = make_dataset(test_df, is_train=False)" 668 | ] 669 | }, 670 | { 671 | "cell_type": "markdown", 672 | "metadata": { 673 | "id": "-1Bb4Xnm1EwK" 674 | }, 675 | "source": [ 676 | "## Dataset preview" 677 | ] 678 | }, 679 | { 680 | "cell_type": "code", 681 | "execution_count": null, 682 | "metadata": { 683 | "colab": { 684 | "base_uri": "https://localhost:8080/" 685 | }, 686 | "id": "w-8k7gScVoz6", 687 | "outputId": "8ea5d01c-e0ac-4e8b-b15b-b4b1975cd979" 688 | }, 689 | "outputs": [ 690 | { 691 | "name": "stdout", 692 | "output_type": "stream", 693 | "text": [ 694 | "Abstract: b'A highly desirable property of a reinforcement learning (RL) agent -- and a\\nmajor difficulty for deep RL approaches -- is the ability to generalize\\npolicies learned on a few tasks over a high-dimensional observation space to\\nsimilar tasks not seen during training. Many promising approaches to this\\nchallenge consider RL as a process of training two functions simultaneously: a\\ncomplex nonlinear encoder that maps high-dimensional observations to a latent\\nrepresentation space, and a simple linear policy over this space. We posit that\\na superior encoder for zero-shot generalization in RL can be trained by using\\nsolely an auxiliary SSL objective if the training process encourages the\\nencoder to map behaviorally similar observations to similar representations, as\\nreward-based signal can cause overfitting in the encoder (Raileanu et al.,\\n2021). We propose Cross-Trajectory Representation Learning (CTRL), a method\\nthat runs within an RL agent and conditions its encoder to recognize behavioral\\nsimilarity in observations by applying a novel SSL objective to pairs of\\ntrajectories from'\n", 695 | "Label(s): ('cs.AI', 'cs.LG')\n", 696 | " \n", 697 | "Abstract: b'We introduce AndroidEnv, an open-source platform for Reinforcement Learning\\n(RL) research built on top of the Android ecosystem. AndroidEnv allows RL\\nagents to interact with a wide variety of apps and services commonly used by\\nhumans through a universal touchscreen interface. Since agents train on a\\nrealistic simulation of an Android device, they have the potential to be\\ndeployed on real devices. In this report, we give an overview of the\\nenvironment, highlighting the significant features it provides for research,\\nand we present an empirical evaluation of some popular reinforcement learning\\nagents on a set of tasks built on this platform.'\n", 698 | "Label(s): ('cs.AI', 'cs.LG')\n", 699 | " \n", 700 | "Abstract: b'Learning to imitate expert behavior from demonstrations can be challenging,\\nespecially in environments with high-dimensional, continuous observations and\\nunknown dynamics. Supervised learning methods based on behavioral cloning (BC)\\nsuffer from distribution shift: because the agent greedily imitates\\ndemonstrated actions, it can drift away from demonstrated states due to error\\naccumulation. Recent methods based on reinforcement learning (RL), such as\\ninverse RL and generative adversarial imitation learning (GAIL), overcome this\\nissue by training an RL agent to match the demonstrations over a long horizon.\\nSince the true reward function for the task is unknown, these methods learn a\\nreward function from the demonstrations, often using complex and brittle\\napproximation techniques that involve adversarial training. We propose a simple\\nalternative that still uses RL, but does not require learning a reward\\nfunction. The key idea is to provide the agent with an incentive to match the\\ndemonstrations over a long horizon, by encouraging it to return to demonstrated\\nstates upon encountering new, out-of-distribution states. We accomplish'\n", 701 | "Label(s): ('cs.LG', 'stat.ML')\n", 702 | " \n", 703 | "Abstract: b'Knowledge graph completion is an important task that aims to predict the\\nmissing relational link between entities. Knowledge graph embedding methods\\nperform this task by representing entities and relations as embedding vectors\\nand modeling their interactions to compute the matching score of each triple.\\nPrevious work has usually treated each embedding as a whole and has modeled the\\ninteractions between these whole embeddings, potentially making the model\\nexcessively expensive or requiring specially designed interaction mechanisms.\\nIn this work, we propose the multi-partition embedding interaction (MEI) model\\nwith block term format to systematically address this problem. MEI divides each\\nembedding into a multi-partition vector to efficiently restrict the\\ninteractions. Each local interaction is modeled with the Tucker tensor format\\nand the full interaction is modeled with the block term tensor format, enabling\\nMEI to control the trade-off between expressiveness and computational cost,\\nlearn the interaction mechanisms from data automatically, and achieve\\nstate-of-the-art performance on the link prediction task. In addition, we\\ntheoretically study the parameter efficiency'\n", 704 | "Label(s): ('cs.AI', 'cs.CL', 'cs.LG', 'stat.ML')\n", 705 | " \n", 706 | "Abstract: b'Quantization has been proven to be an effective method for reducing the\\ncomputing and/or storage cost of DNNs. However, the trade-off between the\\nquantization bitwidth and final accuracy is complex and non-convex, which makes\\nit difficult to be optimized directly. Minimizing direct quantization loss\\n(DQL) of the coefficient data is an effective local optimization method, but\\nprevious works often neglect the accurate control of the DQL, resulting in a\\nhigher loss of the final DNN model accuracy. In this paper, we propose a novel\\nmetric called Vector Loss. Based on this new metric, we develop a new\\nquantization solution called VecQ, which can guarantee minimal direct\\nquantization loss and better model accuracy. In addition, in order to speed up\\nthe proposed quantization process during model training, we accelerate the\\nquantization process with a parameterized probability estimation method and\\ntemplate-based derivation calculation. We evaluate our proposed algorithm on\\nMNIST, CIFAR, ImageNet, IMDB movie review and THUCNews text data sets with\\nnumerical DNN models. The results'\n", 707 | "Label(s): ('cs.CV',)\n", 708 | " \n" 709 | ] 710 | } 711 | ], 712 | "source": [ 713 | "text_batch, label_batch = next(iter(train_dataset))\n", 714 | "\n", 715 | "for i, text in enumerate(text_batch[:5]):\n", 716 | " label = label_batch[i].numpy()[None, ...]\n", 717 | " print(f\"Abstract: {text[0]}\")\n", 718 | " print(f\"Label(s): {mlb.inverse_transform(label)[0]}\")\n", 719 | " print(\" \")" 720 | ] 721 | }, 722 | { 723 | "cell_type": "markdown", 724 | "metadata": { 725 | "id": "Yfbnoi-y1IoB" 726 | }, 727 | "source": [ 728 | "## Vocabulary size for vectorization\n", 729 | "\n", 730 | "Before we feed the data to our model we need to represent them as numbers. For that purpose, we will use the [`TextVectorization` layer](https://keras.io/api/layers/preprocessing_layers/text/text_vectorization). It can operate as a part of your main model so that the model is excluded from the core preprocessing logic. This greatly reduces the chances of training and serving skew. \n", 731 | "\n", 732 | "We first calculate the number of unique words present in the abstracts." 733 | ] 734 | }, 735 | { 736 | "cell_type": "code", 737 | "execution_count": null, 738 | "metadata": { 739 | "colab": { 740 | "base_uri": "https://localhost:8080/" 741 | }, 742 | "id": "IIW42KOgyfE7", 743 | "outputId": "69e504dc-a55e-41ec-e2b9-11b35750d9ca" 744 | }, 745 | "outputs": [ 746 | { 747 | "name": "stdout", 748 | "output_type": "stream", 749 | "text": [ 750 | "Vocabulary size: 498\n" 751 | ] 752 | }, 753 | { 754 | "name": "stderr", 755 | "output_type": "stream", 756 | "text": [ 757 | "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n", 758 | "A value is trying to be set on a copy of a slice from a DataFrame.\n", 759 | "Try using .loc[row_indexer,col_indexer] = value instead\n", 760 | "\n", 761 | "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n", 762 | " \"\"\"Entry point for launching an IPython kernel.\n" 763 | ] 764 | } 765 | ], 766 | "source": [ 767 | "train_df[\"total_words\"] = train_df[\"summaries\"].str.split().str.len()\n", 768 | "vocabulary_size = train_df[\"total_words\"].max()\n", 769 | "print(f\"Vocabulary size: {vocabulary_size}\")" 770 | ] 771 | }, 772 | { 773 | "cell_type": "markdown", 774 | "metadata": { 775 | "id": "nis2d-xFHNq_" 776 | }, 777 | "source": [ 778 | "Now we can create our text classifier model with the `TextVectorization` layer present inside it. " 779 | ] 780 | }, 781 | { 782 | "cell_type": "markdown", 783 | "metadata": { 784 | "id": "3TBUFHm11M3G" 785 | }, 786 | "source": [ 787 | "## Create model with `TextVectorization`\n", 788 | "\n", 789 | "A batch of raw text will first go through the `TextVectorization` layer and it will generate their integer representations. Internally, the `TextVectorization` layer will first create bi-grams out of the sequences and then represent them using [TF-IDF](https://wikipedia.org/wiki/Tf%E2%80%93idf). The output representations will then be passed to the shallow model responsible for text classification. \n", 790 | "\n", 791 | "To know more about other possible configurations with `TextVectorizer`, please consult the [official documentation](https://keras.io/api/layers/preprocessing_layers/text/text_vectorization). " 792 | ] 793 | }, 794 | { 795 | "cell_type": "code", 796 | "execution_count": null, 797 | "metadata": { 798 | "id": "4XX7ovyPokNs" 799 | }, 800 | "outputs": [], 801 | "source": [ 802 | "text_vectorizer = layers.TextVectorization(\n", 803 | " max_tokens=vocabulary_size, ngrams=2, output_mode=\"tf_idf\"\n", 804 | ")\n", 805 | "\n", 806 | "# `TextVectorization` needs to be adapted as per the vocabulary from our\n", 807 | "# training set.\n", 808 | "with tf.device(\"/CPU:0\"):\n", 809 | " text_vectorizer.adapt(train_dataset.map(lambda text, label: text))\n", 810 | "\n", 811 | "\n", 812 | "def make_model():\n", 813 | " shallow_mlp_model = keras.Sequential(\n", 814 | " [\n", 815 | " text_vectorizer,\n", 816 | " layers.Dense(512, activation=\"relu\"),\n", 817 | " layers.Dense(256, activation=\"relu\"),\n", 818 | " layers.Dense(len(mlb.classes_), activation=\"sigmoid\"),\n", 819 | " ]\n", 820 | " )\n", 821 | " return shallow_mlp_model" 822 | ] 823 | }, 824 | { 825 | "cell_type": "markdown", 826 | "metadata": { 827 | "id": "i7i71RLtugUs" 828 | }, 829 | "source": [ 830 | "Without the CPU placement, we run into: \n", 831 | "\n", 832 | "```\n", 833 | "(1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string\n", 834 | "```" 835 | ] 836 | }, 837 | { 838 | "cell_type": "code", 839 | "execution_count": null, 840 | "metadata": { 841 | "colab": { 842 | "base_uri": "https://localhost:8080/" 843 | }, 844 | "id": "58tUdCsQuI1P", 845 | "outputId": "47a81fa0-d83c-4dfe-8745-8122ebbc4da9" 846 | }, 847 | "outputs": [ 848 | { 849 | "name": "stdout", 850 | "output_type": "stream", 851 | "text": [ 852 | "Model: \"sequential\"\n", 853 | "_________________________________________________________________\n", 854 | "Layer (type) Output Shape Param # \n", 855 | "=================================================================\n", 856 | "text_vectorization (TextVect (None, 498) 1 \n", 857 | "_________________________________________________________________\n", 858 | "dense (Dense) (None, 512) 255488 \n", 859 | "_________________________________________________________________\n", 860 | "dense_1 (Dense) (None, 256) 131328 \n", 861 | "_________________________________________________________________\n", 862 | "dense_2 (Dense) (None, 152) 39064 \n", 863 | "=================================================================\n", 864 | "Total params: 425,881\n", 865 | "Trainable params: 425,880\n", 866 | "Non-trainable params: 1\n", 867 | "_________________________________________________________________\n" 868 | ] 869 | } 870 | ], 871 | "source": [ 872 | "shallow_mlp_model = make_model()\n", 873 | "shallow_mlp_model.summary()" 874 | ] 875 | }, 876 | { 877 | "cell_type": "markdown", 878 | "metadata": { 879 | "id": "y1Hr9D0O1Tw0" 880 | }, 881 | "source": [ 882 | "## Train the model\n", 883 | "\n", 884 | "We will train our model using the binary cross-entropy loss. This is because the labels are not disjoint. For a given abstract, we may have multiple categories. So, we will divide the prediction task into a series of multiple binary classification problems. This is also why we kept the activation function of the classification layer in our model to sigmoid. Researchers have used other combinations of loss function and activation function as well. For example, in [Exploring the Limits of Weakly Supervised Pretraining](https://arxiv.org/abs/1805.00932), Mahajan et al. used the softmax activation function and cross-entropy loss to train their models. " 885 | ] 886 | }, 887 | { 888 | "cell_type": "code", 889 | "execution_count": null, 890 | "metadata": { 891 | "colab": { 892 | "base_uri": "https://localhost:8080/" 893 | }, 894 | "id": "WCoaRkA-wsC3", 895 | "outputId": "d8f31c73-6f8b-43db-e226-c6fd55e446df" 896 | }, 897 | "outputs": [ 898 | { 899 | "name": "stdout", 900 | "output_type": "stream", 901 | "text": [ 902 | "Epoch 1/20\n", 903 | "258/258 [==============================] - 4s 12ms/step - loss: 0.0778 - categorical_accuracy: 0.6148 - val_loss: 0.0227 - val_categorical_accuracy: 0.6519\n", 904 | "Epoch 2/20\n", 905 | "258/258 [==============================] - 3s 10ms/step - loss: 0.0229 - categorical_accuracy: 0.6810 - val_loss: 0.0215 - val_categorical_accuracy: 0.6759\n", 906 | "Epoch 3/20\n", 907 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0218 - categorical_accuracy: 0.6855 - val_loss: 0.0207 - val_categorical_accuracy: 0.6830\n", 908 | "Epoch 4/20\n", 909 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0209 - categorical_accuracy: 0.6904 - val_loss: 0.0205 - val_categorical_accuracy: 0.6994\n", 910 | "Epoch 5/20\n", 911 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0203 - categorical_accuracy: 0.6934 - val_loss: 0.0200 - val_categorical_accuracy: 0.6945\n", 912 | "Epoch 6/20\n", 913 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0197 - categorical_accuracy: 0.6949 - val_loss: 0.0198 - val_categorical_accuracy: 0.6972\n", 914 | "Epoch 7/20\n", 915 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0193 - categorical_accuracy: 0.6960 - val_loss: 0.0197 - val_categorical_accuracy: 0.6972\n", 916 | "Epoch 8/20\n", 917 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0189 - categorical_accuracy: 0.6987 - val_loss: 0.0195 - val_categorical_accuracy: 0.6929\n", 918 | "Epoch 9/20\n", 919 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0185 - categorical_accuracy: 0.6991 - val_loss: 0.0195 - val_categorical_accuracy: 0.6918\n", 920 | "Epoch 10/20\n", 921 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0182 - categorical_accuracy: 0.6992 - val_loss: 0.0196 - val_categorical_accuracy: 0.6945\n", 922 | "Epoch 11/20\n", 923 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0178 - categorical_accuracy: 0.7002 - val_loss: 0.0195 - val_categorical_accuracy: 0.6939\n", 924 | "Epoch 12/20\n", 925 | "258/258 [==============================] - 3s 10ms/step - loss: 0.0175 - categorical_accuracy: 0.7024 - val_loss: 0.0196 - val_categorical_accuracy: 0.6956\n", 926 | "Epoch 13/20\n", 927 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0172 - categorical_accuracy: 0.7048 - val_loss: 0.0197 - val_categorical_accuracy: 0.6978\n", 928 | "Epoch 14/20\n", 929 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0168 - categorical_accuracy: 0.7065 - val_loss: 0.0200 - val_categorical_accuracy: 0.6994\n", 930 | "Epoch 15/20\n", 931 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0166 - categorical_accuracy: 0.7082 - val_loss: 0.0201 - val_categorical_accuracy: 0.7005\n", 932 | "Epoch 16/20\n", 933 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0163 - categorical_accuracy: 0.7104 - val_loss: 0.0201 - val_categorical_accuracy: 0.6901\n", 934 | "Epoch 17/20\n", 935 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0160 - categorical_accuracy: 0.7111 - val_loss: 0.0202 - val_categorical_accuracy: 0.6863\n", 936 | "Epoch 18/20\n", 937 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0157 - categorical_accuracy: 0.7117 - val_loss: 0.0202 - val_categorical_accuracy: 0.6879\n", 938 | "Epoch 19/20\n", 939 | "258/258 [==============================] - 3s 11ms/step - loss: 0.0154 - categorical_accuracy: 0.7127 - val_loss: 0.0205 - val_categorical_accuracy: 0.6863\n", 940 | "Epoch 20/20\n", 941 | "258/258 [==============================] - 3s 10ms/step - loss: 0.0150 - categorical_accuracy: 0.7138 - val_loss: 0.0206 - val_categorical_accuracy: 0.6836\n" 942 | ] 943 | } 944 | ], 945 | "source": [ 946 | "epochs = 20\n", 947 | "\n", 948 | "shallow_mlp_model.compile(\n", 949 | " loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"categorical_accuracy\"]\n", 950 | ")\n", 951 | "\n", 952 | "history = shallow_mlp_model.fit(train_dataset, validation_data=validation_dataset, epochs=epochs)" 953 | ] 954 | }, 955 | { 956 | "cell_type": "code", 957 | "execution_count": null, 958 | "metadata": { 959 | "colab": { 960 | "base_uri": "https://localhost:8080/", 961 | "height": 575 962 | }, 963 | "id": "bOG4RF0KxUpq", 964 | "outputId": "4b809603-f59c-4c4b-9f89-2d03cc42ee4a" 965 | }, 966 | "outputs": [ 967 | { 968 | "data": { 969 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEXCAYAAAC3c9OwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xcdZ3w8c937rlM07QJKb1AuRS1UOUSi6hgKyugq1tBsCAiIMiignhj7Yry9OGluyo++qwrjyzLTYG1RVS2SqUqNAtoraW1hZZLCaWU9ELvaS5NZjL5Pn/8TtKT6SSZZHKSNPN9v17nNefyO+d8c2Yy3/n9fuciqooxxhiTLTTSARhjjBmdLEEYY4zJyRKEMcaYnCxBGGOMyckShDHGmJwsQRhjjMnJEsQRRkTuF5HfjnQcvRGR9SKyMOB9zBERFZGqXNO9rHOxiBR8Tnc++xoKInKViDQHuQ8zeCKyWUS+OtJxBM0SREC8L5G+hvsHuembgE8OYajDRkS+LCIHRKQ0x7KwiGwVkX8ZxKb/DBwN7Ck4yJ4x5foSCGRfo5GITBGRu0SkQURS3vvznyIydQRjWtjL/9OOkYppLLMEEZyjfcNncsy7yV9YRKL5bFRVG1V1/xDGOZweAOLAJTmWfRB3XO4Z6EZVNaWqO3QYrvoczn2NJBE5DngWOAW4EjgR98PkZGCViEwPeP+xPha/TM//paOBWUHGU6wsQQTE+xLZoao7gP3+eUAC2C8il4nIkyJyEPhHEZkoIj/3frEdFJENInK1f7vZTUwiUici/09E/kVEdovIThH5voj0+t7muZ9+tysiR4nIf3vbeF1EPt3PMdkFLAFylbsGqFPVV72axnMi0uL9ar1bRMb38fcc1uwjIp/yYmr1jldN1joneLHv8PazRkQ+7P/7gWOB27t+pfaxr4tE5HkRaReRN0TkFhER3/LNIvINEfkPrwbVICI393Wsevk7/1FE6r1f8/Ui8pkcyzeKSJv3ni0TkYi3bJaIPOHtv1lE1onI3D52dwfQCfydqj6hqltUdTnwd978O7ztXicib4pIOCuW/xKRJb7pj4jIai+210Tk2/4k4B2jhSJyr4jsBx7qI7YO//+XN+zKsa0Hvb91h2TVBEXkGBH5tYg0ecOvJKtmJCIfEpGV3ud7j4j8RkQSviKJvt7Tvt6PI4aq2hDwAFzsDnX39HRAgc3esuOAqcAU4GbgVOB44DogBZzrW/d+4Le+6TqgEbgNOAn4ONABXNZHPPnsp9/tAkuBDcB7gNO8dZqBhX3s+wLcF8yJvnk1QBq43Jv+IvB+7zi9D3gOeMBXfo53/Kp6mT7T28ctXuz/iGsS8r8H7wCux/3yPNErmwLe6i2fALwB/G9gEjCpl32dAWS8cicBl3vH4EbfvjZ7+7/B29eN3jbO6uM4XQU0+6Yv9I7RDd5+bvSmP+Itr/Xen8txie0dwJeAiLf8eeBB4K1eDBf2tn/vb+8Evt7L8lu85ZXe0AZc4FteDrQAH/emzwcOAFcDJwBzcbWA72cdowPAP3nxzehl3wuB9f38v3Vty//+p4CLvOUh4G+45sJab/gLrsYkvs9pB/AtYCbwduCrQGk+72l/78eRMox4AMUw0HuC+Eoe6y4C7vZN38/hCWJF1jp/8K+TZ4zZ++lzu94/ngLv8S0/FvdlubCP/YSA14F/8c27GdgHJHpZ5wKgHQh503PoO0H8F/CHrG3c7X8PetnPX4Bv+KY3A1/NKpO9r4eAJ7PKLAQasrbz86wyr/j3lSOWq+iZIP4E3JtV5n7gGW/8IlxCT/ayvQPAlXl+Fs70/sYLe1l+obd8tjf9K3om8E96sSS86aeAb2Zt46O4RNr1hbwZ+E0esS30PmPNWcPPfWU29/L+dx2rD3jbmO5bfjyHakxdx3tRH3H0+Z72934cKYM1MY2sZ/0T4jpqb/GaV/aIO4vlIuCYfrbzXNb0NuCo3goPYD99bfdtuH+ov3YtVNXXvTK9UtVO4D7gU75miU8DD6lqmxff+0XkD161vQn3BRTD/ZLPx9uAFVnzekyLSJmIfE9EXhCRfd4xqKX/Y51rX3/KmvcMMEVExvnmDeg9GsB+Znrjf8Al3tdE5CERuVJEkr6yPwDuFtekeYuIvHUA++7Pg8BH5dDJB5cDv+x6P3G1rFu85p5m71j/F1BGz/e0x/9DH17F1X79w5eyyuR6/7uO1duAbaq6uWuhqm7CvSddZU4Dnugnjr7e0/7ejyOCJYiR1ZI1/VXgK8DtwLm4D/6juC/HvqSzppW+39t895PPdgfTWXsfrmPxfBF5N67Z424AETkWeAx4EdeZfQaH+iz6Ow4D8X1v+9/ENWOdikt2Q7kP/7EZ6Hs0oH2oahNwOq4pcAvwz8BLIjLZW74Q9+X3KPBu4Dnpvc+o3tvuzF6Wz/SW13vTj+GaU+aJyFG4fooHfeVDuCY4/xf624EZwC5fuez/h96kVLU+axiqs5gG8nnu9T3t7/04UliCGF3ei6tmP6Cqa3G/lE4apft5Cff5md01Q0SOAfr9B/BqGn/EdUxfA6z24gD3Kz4GfElVV6jqxny2meVF4F1Z87Kn3wv8TFV/qarPAQ249nG/FBCmby/i+mCyt93gfUkMld7280LXhKp2qOqTqvrPuC/gMuDDvuWvqOqPVPXvcWeLXZtrR6q6B1gGfE6yTkn2pj8P/E5V93rl24Ff4GoO84EduCbKLmtwfTvZX+r1qtox0AORp1zv/4ve+IvAZPGdiSUix+M+Z13H82+4H0+D1t/7cSQ4snrUx76NwHwReS+wG9fxdRzuwzqq9qOqL4vI48B/iMh1wEFcM8bBPDdxD+5XZgrXB9HlFVzi+aKI/Ar3j/3FfOPy/Aj4s4j8M/AIrt/gwqwyG4ELReS/cb8E/xfu7DK/zcDZIvIg0K6qu3Ps6//gTvtciGs2eSeudvb1Acbcn9uBX4jIauD3uH6Zy3FNg4g7A+sEXHv/XlxHcBJ4UURKcDWmX3h/Uw0uuazsY3834Dpx/ygi38C9LycA3wbEW+73IK5J5jhc23ynb9ltwG9F5HXgYVxt4xRcH8Y/DfRAABEROay5MasW8a6s9/9TuOMF7sfJc8BDItJ1uvm/4xLZk970t4HfiEg97n0V4DzgP1S1tb8A+3o/8v8zR57VIEaXb+GaOX6H+2C10PfpfiO9n6uA13D/VL/B/SNtznPdR3GdeCFvPQC8X/M3AV/G/Zq7FtckljdV/QuuZvJZ3BfBRbjOTb8vAzuBp3HH4S/euN+twDRcDWsXOajqGlxT1ceA9cB3vOHHA4m5P6r6KC6Rfwl3XG4CPqeqv/GK7Md1/P4RV7v7KnCtqj6N65CtxHVqvwz8Gtcm/+U+9vcqrja3AXf9yibc+/Qi8E5VfS1rlaeBrbjmJ3/zEqq6DPh73JfkX71hAa7pZTDeAmzPHrJOIf0B7lf733Cf91tV9REvHgXm4d7T5d6wA/iotwxVXYr7UfFBbxv/48XvT3x96ev9OGJ0nUFgjDFjgohsBn6sqt8f6ViOdFaDMMYYk5MlCGOMMTlZE5MxxpicrAZhjDEmp0BPcxWRC4B/w51LfreqfidreRz4Ge5iqD3AfFXdLO7OpnfjLjSJ4M5X/9e+9lVVVaXTp08fdKwtLS2UlZUNev2gWXyFsfgKY/EVZjTHt3r16t2qWp1zYVD38MAlhVdx9ziJAeuAmVllPgfc6Y1fCiz2xj+Bdx8UoBR36uT0vvZ3xhlnaCGWL19e0PpBs/gKY/EVxuIrzGiOD3hWR+BeTLOBelXdpKop3M3g5mWVmQf81Bt/BDhXRAR3yXqZd15zCe5iqgMBxmqMMSZLYJ3UInIx7hbA13rTVwBnquoNvjLrvTIN3vSruDtJNuIuzjkXV4P4kqrelWMf1+FuVU1NTc0ZixYtGnS8zc3NlJeXD3r9oFl8hbH4CmPxFWY0xzd37tzVqlqba9lovdXGbNzVn5NxV4A+LSJ/VHfHxW5e0rgLoLa2VufMmTPoHdbV1VHI+kGz+Apj8RXG4ivMaI+vN0EmiK242xR0merNy1WmwWtOqsB1Vn8CeFxV08BOEfkT7rL/TRhjjE86naa8vJwXXxy9tzmqqKgY8fgSiQRTp04lGs3r6cZAsAliFTBD3LNtt+I6oT+RVWYJ7nm3K3AP1XlSVVVEtuCeKPaAiJThbtj2fwOM1RhzhGpoaKCmpoapU6cih570Oqo0NTWRTI7c4yBUlT179tDQ0MBxxx2X93qBdVKru43vDbjbBr8IPKyqG0TkNhH5B6/YPcBE746JX8bdwAvc827LRWQDLtHcp+4mbsYY00NbWxsVFRWjNjmMBiLCxIkTaWtr67+wT6B9EOruiLg0a96tvvE23J0ws9drzjXfGGNyseTQv8Eco9HaST1stu4/yOK/bmFKOt+7+BpjTHEo+ltt7G9N8aMn63mjyRKEMWZwRusprIUq+gRRXR4HoDFlNy00xhi/ok8QE8piiEBjuyUIY0xhVJWbb76ZU045hVmzZrF48WIAduzYwTnnnMOpp57KKaecwtNPP00mk+Gqq67qLvvDH/5whKM/XNH3QUTCISaWxTjQbk1Mxhzp/vdvNvDCtqG9K8/MyeP4Xx85Oa+yv/rVr1i7di3r1q1j9+7dvPOd7+Scc87hF7/4Beeffz633HILmUyG1tZW1q5dy9atW1m/fj0A+/fvH9K4h0LR1yAAqsrj1sRkjCnYM888w2WXXUY4HKampob3ve99rFq1itNPP5377ruPhQsX8vzzz5NMJjn++OPZtGkTN954I48//jjjxo0b6fAPU/Q1CIDqZJytO1tGOgxjTIHy/aU/3N7znvfw1FNP8dhjj3HVVVfx5S9/mU996lOsW7eOZcuWceedd/Lwww9z7733jnSoPVgNAtdRbX0QxphCnX322SxevJhMJsOuXbt46qmnmD17Nlu2bKGmpobPfOYzXHvttaxZs4bdu3fT2dnJxz72Mb71rW+xZs2akQ7/MFaDwNUgGlPu/ud2wY0xZrAuvPBCVqxYwTve8Q5EhO9973tMmjSJRx99lPnz5xONRikvL+dnP/sZW7du5eqrr6az0/V//uu/9vlMtBFhCQLXB9HRCQfaOqgoyf9GVsYYA+523uCuVr799tu5/fbbeyy//PLLuf766w9bbzTWGvysiQlXgwDY1dQ+wpEYY8zoYQmCQwlid7MlCGOM6WIJAqtBGGNMLpYgcH0QYAnCGGP8LEEA40uihMWamIwxxs8SBBAKCeNiYjUIY4zxsQThGRcXdlkNwhhjulmC8FTErQZhjAleX8+O2Lx5M6eccsowRtM3SxCeiphYH4QxxvjYldSeiriwe3uKzk4lFLLbbRhzRPrdAtjx/NBuc9Is+OB3el28YMECpk2bxuc//3kAFi5cSCQSYfny5ezbt490Os0tt9zCpZdeOqDdtrW18dnPfpZnn32WSCTCD37wA+bOncuGDRu4+uqrSaVSdHZ28stf/pLJkyfz8Y9/nIaGBjKZDN/85jeZP39+QX82WILoVhETMp3KvtYUE73TXo0xpj/z58/ni1/8YneCePjhh1m2bBlf+MIXGDduHLt372b27NnMnz9/QPd6u+OOOxARnn/+eV566SXOO+88Nm7cyJ133slNN93E5ZdfTiqVIpPJsHTpUiZPnsxjjz0GQGNj45D8bZYgPBVx98btam63BGHMkaqPX/pBOe2009i5cyfbtm1j165dVFZWMmnSJL70pS/x1FNPEQqF2L59O2+++SaTJk3Ke7vPPPMMN954IwBvfetbOfbYY9m4cSNnnXUW3/72t2loaOCiiy5ixowZzJo1i6985St87Wtf48Mf/jBnn332kPxt1gfh6UoQu5tSIxyJMeZIc8kll/DII4+wePFi5s+fz0MPPcSuXbtYvXo1a9eu5aijjqKtrW1I9vWJT3yCJUuWUFJSwoc+9CGefPJJTjrpJNasWcOsWbP4xje+wW233TYk+wo0QYjIBSLysojUi8iCHMvjIrLYW75SRKZ78y8XkbW+oVNETg0y1kM1iKF5E40xxWP+/PksWrSIRx55hEsuuYTGxkaOOuoootEoy5cvZ8uWLQPe5tlnn81DDz0EwMaNG9myZQtvectb2LRpE8cffzxf+MIXmDdvHs899xzbtm2jtLSUT37yk9x8881DdpfYwJqYRCQM3AF8AGgAVonIElV9wVfsGmCfqp4oIpcC3wXmq+pDwEPedmYBj6rq2qBiBRgX8xKEnepqjBmgk08+maamJqZMmcLRRx/N5Zdfzkc+8hFmzZpFbW0tJ5100oC3+bnPfY7PfvazzJo1i0gkwv333088Hufhhx/mgQceIBqNMmnSJL7+9a+zatUqbr75ZkKhENFolJ/85CdD8ncF2QcxG6hX1U0AIrIImAf4E8Q8YKE3/gjwYxERVfU/3u0yYFGAcQJQEoF4JMTuZmtiMsYM3PPPHzp7qqqqihUrVnRPNzU1kUwmgUPPjshl+vTprF+/HoBEIsF99913WJkFCxawYEHPBpnzzz+f888/v6D4cwkyQUwB3vBNNwBn9lZGVTtEpBGYCOz2lZmPSySHEZHrgOsAampqqKurG3SwLS0tJKMhnq9/nbq6Nwe9naA0NzcX9PcFzeIrjMU3eBUVFWQyGZqamkY6lF6Nlvja2toG9D6O6rOYRORMoFVV1+darqp3AXcB1NbW6pw5cwa9r7q6OqZWRQnHI8yZk53HRl5dXR2F/H1Bs/gKY/EN3osvvkg4HO7+hT4a+WsQ4GobV1xxRY8y8XiclStXBhpHIpHgtNNOy7t8kAliKzDNNz3Vm5erTIOIRIAKYI9v+aXAzwOMsYfqZJwte1qHa3fGmCHSs1V69Js1axZr1wbarXqYwRyjIM9iWgXMEJHjRCSG+7JfklVmCXClN34x8GRX/4OIhICPMwz9D12qk3G73YYxR5hEIkFjY+MRlySGk6qyZ88eEonEgNYLrAbh9SncACwDwsC9qrpBRG4DnlXVJcA9wAMiUg/sxSWRLucAb3R1cg+H6vI4e1tTpDOdRMN2iYgxR4KpU6eybt26Pjt/R1pbW9uAv5yHWiKRYOrUqQNaJ9A+CFVdCizNmnerb7wNuKSXdeuAdwUZX7aqZBxV2NuSombcyL6Zxpj8RKNRmpubqa2tHelQelVXVzegtv/Rwn4m+1Tbo0eNMaabJQif6qSXIKwfwhhjLEH4WQ3CGGMOsQThU5WMAZYgjDEGLEH0UBqLUB6P2KmuxhiDJYjDVCfjVoMwxhgsQRymqjxmCcIYY7AEcRi7mtoYYxxLEFmqy62JyRhjwBLEYaqTcQ60ddCWzox0KMYYM6IsQWSp8q6FsGYmY0yxswSRpetqanuynDGm2FmCyNJ9uw3rhzDGFDlLEFmq7HYbxhgDWII4zMRyu92GMcaAJYjDxCNhxpdGrZPaGFP0LEHkYNdCGGOMJYicqsrj9kwIY0zRswSRg91uwxhjLEHkZHd0NcYYSxA5VSfjtKYytLR3jHQoxhgzYixB5GDXQhhjTMAJQkQuEJGXRaReRBbkWB4XkcXe8pUiMt237O0iskJENojI8yKSCDJWv0O327AEYYwpXoElCBEJA3cAHwRmApeJyMysYtcA+1T1ROCHwHe9dSPAg8D1qnoyMAdIBxVrtmqrQRhjTKA1iNlAvapuUtUUsAiYl1VmHvBTb/wR4FwREeA84DlVXQegqntUddjuv12V9K6mthqEMaaIiaoGs2GRi4ELVPVab/oK4ExVvcFXZr1XpsGbfhU4E/gkcAZwFFANLFLV7+XYx3XAdQA1NTVnLFq0aNDxNjc3U15eDkCnKtcsa+UjJ0S5aEZs0NscSv74RiOLrzAWX2EsvsGbO3fualWtzbUsMtzB5CkCvBd4J9AKPCEiq1X1CX8hVb0LuAugtrZW58yZM+gd1tXV4V9/4p/+SNnEo5gz5+2D3uZQyo5vtLH4CmPxFcbiC0aQTUxbgWm+6anevJxlvH6HCmAP0AA8paq7VbUVWAqcHmCsh7FrIYwxxS7IBLEKmCEix4lIDLgUWJJVZglwpTd+MfCkujavZcAsESn1Esf7gBcCjPUwVeUxSxDGmKIWWBOTqnaIyA24L/swcK+qbhCR24BnVXUJcA/wgIjUA3txSQRV3SciP8AlGQWWqupjQcWaS3UyzqZdLcO5S2OMGVUC7YNQ1aW45iH/vFt9423AJb2s+yDuVNcR0dXEpKq4E6uMMaa42JXUvaguj5PKdHLgoN1uwxhTnCxB9KL72dR2LYQxpkhZguiFXU1tjCl2liB6YTUIY0yxswTRC7ujqzGm2FmC6EVFSZRoWOyOrsaYomUJohehkLhnU1sNwhhTpCxB9MFut2GMKWaWIPpgNQhjTDGzBNGH6vK49UEYY4qWJYg+VCfj7GlJkekM5pkZxhgzmlmC6EN1Mk6mU9nXmhrpUIwxZthZguhD17UQ1sxkjClGliD60H01tXVUG2OKkCWIPliCMMYUM0sQfagqjwGWIIwxxckSRB/K4xES0ZD1QRhjipIliD6IiF1NbYwpWpYg+lFdHrdbfhtjipIliH5UlcfZ3WTXQRhjio8liH5UJ60GYYwpTpYg+lGdjLO3JUU60znSoRhjzLCyBNGPrmsh9jRbM5MxprgEmiBE5AIReVlE6kVkQY7lcRFZ7C1fKSLTvfnTReSgiKz1hjuDjLMvdrsNY0yxigS1YREJA3cAHwAagFUiskRVX/AVuwbYp6onisilwHeB+d6yV1X11KDiy5ddTW2MKVZB1iBmA/WquklVU8AiYF5WmXnAT73xR4BzRUQCjGnAqsstQRhjipOoBvOsAxG5GLhAVa/1pq8AzlTVG3xl1ntlGrzpV4EzgXJgA7AROAB8Q1WfzrGP64DrAGpqas5YtGjRoONtbm6mvLz8sPmpjHLdH1r52IwoHzkhNujtF6q3+EYLi68wFl9hLL7Bmzt37mpVrc21LLAmpgJtB45R1T0icgbwqIicrKoH/IVU9S7gLoDa2lqdM2fOoHdYV1dHb+snn1pGsnoKc+acPOjtF6qv+EYDi68wFl9hLL5gBNnEtBWY5pue6s3LWUZEIkAFsEdV21V1D4CqrgZeBU4KMNY+2bUQxphiFGSCWAXMEJHjRCQGXAosySqzBLjSG78YeFJVVUSqvU5uROR4YAawKcBY+1Rl92MyxhShwJqYVLVDRG4AlgFh4F5V3SAitwHPquoS4B7gARGpB/bikgjAOcBtIpIGOoHrVXVvULH2p7o8zos7DvRf0BhjxpC8EoSI3ATcBzQBdwOnAQtU9fd9raeqS4GlWfNu9Y23AZfkWO+XwC/ziW04VCfjPPWK1SCMMcUl3yamT3sdxOcBlcAVwHcCi2qUqU7GaWrroC2dGelQjDFm2OSbILquTfgQ8ICqbvDNG/PsWghjTDHKN0GsFpHf4xLEMhFJ4voGikJV0l3/YLfbMMYUk3w7qa8BTgU2qWqriEwArg4urNGlujwBWA3CGFNc8q1BnAW8rKr7ReSTwDeAxuDCGl2678dkNQhjTBHJN0H8BGgVkXcAX8FduPazwKIaZSaWe01M9mQ5Y0wRyTdBdKi7adM84MeqegeQDC6s0SUaDlFZGmVXc9tIh2KMMcMm3z6IJhH5Z9zprWeLSAiIBhfW6FNtV1MbY4pMvjWI+UA77nqIHbj7Kt0eWFSjkCUIY0yxyStBeEnhIaBCRD4MtKlq0fRBgHuy3G577KgxpojklSBE5OPAX3G3xfg4sNJ73kPRqC53NYignp9hjDGjTb59ELcA71TVnQAiUg38EfcUuKJQnYxzMJ2hJZWhPD5aH6NhjDFDJ98+iFBXcvDsGcC6Y0LXtRC7rR/CGFMk8v0p/LiILAN+7k3PJ+surWNdVfmhi+WmV5WNcDTGGBO8vBKEqt4sIh8D3uPNuktVfx1cWKNP99XUVoMwxhSJvBvTR9szGoabJQhjTLHpM0GISBOQ67QdAVRVxwUS1ShUWRojJHZHV2NM8egzQahq0dxOoz/hkDCx3C6WM8YUj6I6E6lQ1ZYgjDFFxBLEAFQn43bLb2NM0bAEMQBV5XG7DsIYUzQsQQxAVw3CbrdhjCkGgSYIEblARF4WkXoRWZBjeVxEFnvLV4rI9Kzlx4hIs4h8Ncg481WdjJPOKI0H0yMdijHGBC6wBCEiYeAO4IPATOAyEZmZVewaYJ+qngj8EPhu1vIfAL8LKsaB6r7dhvVDGGOKQJA1iNlAvapuUtUUsAj3RDq/ecBPvfFHgHNFRABE5KPAa8CGAGMckCrv0aM7rR/CGFMEgrwt6RTgDd90A3Bmb2VUtUNEGoGJItIGfA34ANBr85KIXAdcB1BTU0NdXd2gg21ubu53/W3NnQA8tXItqTeG946u+cQ3kiy+wlh8hbH4gjFa71u9EPihqjZ7FYqcVPUu4C6A2tpanTNnzqB3WFdXR3/rN7am+fozv6dq2vHMOfv4Qe9rMPKJbyRZfIWx+Apj8QUjyASxFZjmm57qzctVpkFEIkAF7lbiZwIXi8j3gPFAp4i0qeqPA4y3X+NKIsTCIXuynDGmKASZIFYBM0TkOFwiuBT4RFaZJcCVwArgYuBJdeeQnt1VQEQWAs0jnRy8WKgqj9nV1MaYohBYgvD6FG4AlgFh4F5V3SAitwHPquoS4B7gARGpB/biksioZldTG2OKRaB9EKq6lKwHC6nqrb7xNtxzrvvaxsJAghuk6mScbfvbRjoMY4wJnF1JPUBV5VaDMMYUB0sQA1SdjLOnuZ1Mp91uwxgztlmCGKDqZJxOhb0tdiaTMWZsswQxQNXldrsNY0xxsAQxQFX2bGpjTJGwBDFAXTUISxDGmLHOEsQA2R1djTHFwhLEAJXFI5REw1aDMMaMeZYgBsGupjbGFANLEINQnYxbDcIYM+ZZghiE6vK49UEYY8Y8SxCDUJW0O7oaY8Y+SxCDUF2eYF9rmlRH50iHYowxgbEEMQhdp7ruabFahDFm7LIEMQjd10I02f2YjDFjlyWIQagqjwGwq9meC2GMGbssQQxCtd2PyRhTBCxBDEJV9x1drYnJGDN2WYIYhEQ0TDIRse8LdKkAABdrSURBVBqEMWZMswQxSHY1tTFmrLMEMUjV5ZYgjDFjmyWIQapO2u02jDFjmyWIQaqyGoQxZowLNEGIyAUi8rKI1IvIghzL4yKy2Fu+UkSme/Nni8hab1gnIhcGGedgVCfjNLV30JbOjHQoxhgTiMAShIiEgTuADwIzgctEZGZWsWuAfap6IvBD4Lve/PVAraqeClwA/IeIRIKKdTDsWghjzFgXZA1iNlCvqptUNQUsAuZllZkH/NQbfwQ4V0REVVtVtcObnwA0wDgHpTtBWD+EMWaMEtVgvntF5GLgAlW91pu+AjhTVW/wlVnvlWnwpl/1yuwWkTOBe4FjgStU9dc59nEdcB1ATU3NGYsWLRp0vM3NzZSXl+ddfnNjhoUr2rjxtDhn1ARfuRlofMPN4iuMxVcYi2/w5s6du1pVa3MuVNVABuBi4G7f9BXAj7PKrAem+qZfBaqyyrwN+CuQ6Gt/Z5xxhhZi+fLlAyq/o/GgHvu13+oDKzYXtN98DTS+4WbxFcbiK4zFN3jAs9rL92qQTUxbgWm+6anevJxlvD6GCmCPv4Cqvgg0A6cEFukgTCiLIYKd6mqMGbOCTBCrgBkicpyIxIBLgSVZZZYAV3rjFwNPqqp660QARORY4K3A5gBjHbBoOERlqT1ZzhgzdgXWeK6qHSJyA7AMCAP3quoGEbkNV6VZAtwDPCAi9cBeXBIBeC+wQETSQCfwOVXdHVSsg2VXUxtjxrJAe1dVdSmwNGverb7xNuCSHOs9ADwQZGxDwa6mNsaMZXYldQGqk3E7zdUYM2ZZgihAVbnrg9CAThU2xpiRZAmiANXJOG3pTprbO/ovbIwxRxhLEAXoupranixnjBmLLEEUoLo8Adj9mIwxY5MliAJUJWOAJQhjzNhkCaIA1eVdd3RtG+FIjDFm6FmCKEBlaYxwSKwPwhgzJlmCKEAoJEwss9ttGGPGJksQBbKL5YwxY5UliALZ7TaMMWOVJYgC2Q37jDFjlSWIAlV5NYjOTrvdhjFmbLEEUaDq8jjpjNJ4MD3SoRhjzJCyBFGgQ7fbsGYmY8zYYglC1Q2D1JUgHlq5hZWb9nAwlRmqyIwxZkQF+sCgI8K+1+D+D/O2xAlQVg/HnAXVb4NQfrnzLTVJ3lKT5P4/b+b+P28mEhJmTh7H6cdUctox4zn9mEqmVpYgIgH/IcYYM7QsQWQ6YNqZjH+lDh57ys1LjIdj3uWSxTFnweTTIBLLuXplWYxlXzqHvS0p/rZlH6tf38eaLftYvOoN7v/zZgCOSsY5/ZhKTj/WJYxTplSQiIaH5+8zxphBsgRRfRJcch8rli9nzjumw5YVbnh9BWx83JWJJGBKrUsax54FU2dDYlyPzUwoi3Hu22o49201AHRkOnlpRxNrtuxjzev7WLNlP49v2AFANCycPLmiR9I4uiJhtQxjzKhiCaKLCEw4zg2nfsLNa951KGFsWQHP/BCe/j5ICCbNOlTDOOYsSNb02FwkHOKUKRWcMqWCT501HXB3ff3bln2s3rKPv72+n4dWvs69f3oNgPGlUU6qSfLWSUlOqknyFu+1oiQ6nEfBGGO6WYLoS3k1zPwHNwC0N0HDKle72LICVv8UVt7plpVMgKqToOpEmDgDqma46crpEHZf8tXJOOedPInzTp4EQDrTyYvbD/C3Lft5aUcTG99s4tdrttLke0Ld0RUJ3jIpSUkqxd5xDZxUk+TEo8qticoYEzhLEAMRT8IJ73cDQEcKtq+Dhr/Crpdh9yuwcRm0PHhonVDEJYnupDHDGz+JaNlE3j51PG+fOr67uKqyrbGNl3cc4OUdzWx8s4mXdjTxyo40v3ttndukwPSqMtdB7tU0plWWcvT4BBPLYtZUZYwZEoEmCBG5APg3IAzcrarfyVoeB34GnAHsAear6mYR+QDwHSAGpICbVfXJIGMdlEgMpr3TDX4H98Oeepcw9rwCuzfC7np49QnI+G4NXlLpksXEE1zyCceQSJwpkQRTwjHen4jD8TE4Kc76l+uZeOxbaTjQyesHMry27002be3g9y90sESj7NYKDlBKLBxmUkWCoysSTB5fwqSKBJMrEhxdUcLR491rZWnUkogxo40qZNKQafdeU9DhG+91fgrGTXH9o0MssAQhImHgDuADQAOwSkSWqOoLvmLXAPtU9UQRuRT4LjAf2A18RFW3icgpwDJgSlCxDrmS8TC11g1+nRnYv8VLHhu9BFIPrz0FqZZDb7wefi3FKQAvwdFAj3TkO7kqHUrQFKtmj05gx75Ktuwcz6b2cfy5s5I31Q07qSQcjbmEUZHwEkgJUytLmDahlKmVJUweX0I0bJfIGOOuk+rMPXRmvHGFdKtrgu4eDvSYPuHVF6DxkawyviHdCp0F3I3h5AuPrAQBzAbqVXUTgIgsAuYB/gQxD1jojT8C/FhERFX/5iuzASgRkbiqHtmXK4fChzrCZ3yg93KdGZcoOtq6k8bKFU9z5umner8c2r1XL6GkD0LLTqIHtjOhaRsTDmxnRtMmSO+AyOEPM2qOVLI3XcXOXZW8sX08m1Pj2KglrCNGq8ZpkzilpeMoHzeOynEVTJgwnurKSmomTmDyUROpGZ8kbAmkuKi6z1mqFdItWa+tbllHW9ZrO3QchHSb77WtZ7mONje/60dR90Wrmnu6x+ihMu9qPwir426eap6vuC/4rnmHJYGhu+j16HACGitdS0E8CfFxkJzkXuPlEC2BcKznEIkdPq/H/CiE4248UTFksfqJFnAVcZ8bFrkYuEBVr/WmrwDOVNUbfGXWe2UavOlXvTK7s7Zzvar+XY59XAdcB1BTU3PGokWLBh1vc3Mz5eXlg14/aIOKT5VouolYag/x9j3E2/cSS+31xg9Nx9KNA9psh4ZokzgpiZOWOJlwnIxEkUgMCceQcJRQOIqEY3SGor7h0LRK9vwIKhHfazRrOvd8lbA7A22gx6/HF48i2ukNme5x8E9nssbdqyvTCYToDIW9mCK+8bAv3q55ocNizvv9VfX2qd5+FdEMoc4OQp3thDpThDpThDOpHNNuXjhzaL5b1u69thHOtHll2ghn2r1pNy4M/LuiU6JkwjHvPT40ZMJx3+ch5t7Hbu7Y6GHvq+Qo46bS6QzRaNS3ntBzO9K93qHthrz1xRsXb3mou5yLq2udXGXcayYcJxMupSNS4r2WkgmXeK8JmlsOjtrvl7lz565W1dpcy0Z1J7WInIxrdjov13JVvQu4C6C2tlbnzJkz6H3V1dVRyPpBCzS+TBpSzb5fiN6QaoH0QdJtzTQeOEBj436amw7Q0tJEe2sTqbYWOtubCbcdJEYHcdLEpNW9kiYhHSRCHZR501FNESaAW5GEou4Lt+uXIfRMAKooDOoLLjChqPcLMAqhKKl0mlgkdKjJokcTxtD/ogXc9T2RhPv1Gi2FWCmUlkF0IsTKDs2LlrF5+y6mn/i2w+a711K3ja5tdW03kiAUCg3L/XyK+v83QEEmiK3ANN/0VG9erjINIhIBKnCd1YjIVODXwKdU9dUA4zThqOswL6nMuTgKVHlDLgdTGX77x/9hxqzT2dPczp7mFLtb3Oue5nb2tKTY7Y3vb2kj1JkiRpo4aeLSQZwUUTKMi3ZSmRDGx6EyDuNiSkVMGReFZLST8qhSHumkLNxJabiTknCGhGQIqb/tVny/zr1XEV5//XWmHzvdW3Zofne5UMidcSZh9xoKu6F7Otc871W8L/ZM2rUjZ9I9xzs7DnUodo1nldu1bTtTpkz1th/yBnH76p4O+ZaL9+rVoPxf9r29+sfD8bxvJwOwua6O6WfPybu8GRuCTBCrgBkichwuEVwKfCKrzBLgSmAFcDHwpKqqiIwHHgMWqOqfAozRDIGSWJjq0hCnThvfb9nOTuVAW7o7YextSbG7uZ39rWn2tabZfzDFrtY0r7Sm2H8gzf6Dafa3pujrcRvJRITxpVHGl8QYXxqloiR62HRD2cu89/jTGV8SpcKbF4+MnmtJXqmrY8oR+AvTjG2BJQhV7RCRG3BnIIWBe1V1g4jcBjyrqkuAe4AHRKQe2ItLIgA3ACcCt4rIrd6881R1Z1DxmuERCgnjS2OML41x4lH5tcl2dipN7R3sb015iSRF48F09/j+1rQ3nWL/wTRb9x3MmVj+bc2KHtstjYW9hBFjfEmUyrIoFSUxKkujVJbGqPBex5dGqSyNMr40RkVJ1M7wMkUj0D4IVV0KLM2ad6tvvA24JMd63wK+FWRs5sgRCgkVJe5X/7ET81+vs1NpTnXQ2JrmiadXcMLMt7O/1dVKGr3E0pVI9rem2fhmc/d4Rx9VlmQ80iN5uMQR6Y6xaxjX9ZpwtZZkPGLXn5gjyqjupDamEKGQMC7hvqCnV4Q5e0Z1XuupKs3tHS6BeLWUrlrLvhbXDOavvbyxt5XGg2kOtHWQ6SOxhITupFHhSx7jSqIc2JXiJXnV1Wh8zWBdtZayWNiSixl2liCMySIiJBNRkoko0ybkv15XYmk8mObAQffqxtPd4y6RHBrfuv8gB7zE89hrL/W67UhXLaorcZQcSiDjSqKMS0S81yjjSiLdiXFcSYTyeISINYuZQbAEYcwQ8ScWcp8Q1qvly5cz+91ndyeOrn6VxoOpHtP7vYSzuznFq7ta2N+aoqm9o9+HIpbFwt0JJNmdTNxrMuESSs6msZIo5Qn7mihW9s4bMwqICGXxCGXxCJPHlwxo3Uynq7kc8GonTW1d4x00tbnazIE2l1ia2tz4zqY26nd6y/tpGhOBRBgm/OVJXxKJ9EgiyUSE8kSU8njEjccjlCciJL3Xkqg1kR2JLEEYc4QL+zrxB0NVaUllupvCul/bOrqnX6h/jeTECd3LNu9u7W4ua83jOewhwUse0e7k0f0ac4mxNBamJBamLBamNBZx4/EwJVG3rCwepiQWocwrFwuHLOkEzBKEMUVORNyXdR+1l7roNubMOTXnslRHJy3tHTS3d9DU5l6b29OHxtt6LmtqS3snAaR4Y18rzW0dtKYytKY6+rzeJVs4JJTGwpTGwoQyKSa98KfuTv+cfTI5mtXsuSp9swRhjClILBIiFolRWZb7ue35UlXaOzq7k4V7deMHUxlaUhkOZs1vTWVobc+wqWEbiXiE/a0ptuxt7a7p9HW6clfsrkM/0t03k0xESMa9V1+fTbKrTKLnsrF8XYwlCGPMqCAiJKJhEtEwEwaYbOrq9jJnzpk95qkqbenO7v6XA1n9MQfaDvXbdI03tXWwbf9Bmtpcjedguv/ms5JouEctpcJXU+nqp9nWkKZt/XZfTebQCQDh0OhtJrMEYYwZk0SEEq+/omZcYlDbSGc6afaShUsk6e7k0dQ97ks8vhMAGg+maWpLdzeb3bd+Tc59JOORHtfHdN0exn9GmX+eO805RjIRIRRwcrEEYYwxvYiGQ1SWDb75rLNTaUl1sGz508x8R233NTAHsk4C6GoS238wzSs7m7tPd051dPa6bZFDV/WfP3MS3/jwzMH+mb2yBGGMMQEJhdy1MVUlIWZOHjfg9dvSGd81MVmDd3V/48E0Rw/w1Oh8WYIwxphRKhENM6nCPWd+JIzd7ndjjDEFsQRhjDEmJ0sQxhhjcrIEYYwxJidLEMYYY3KyBGGMMSYnSxDGGGNysgRhjDEmJ9H+HkV1hBCRXcDrBWyiCtg9ROEEweIrjMVXGIuvMKM5vmNVNecD28dMgiiUiDyrqrUjHUdvLL7CWHyFsfgKM9rj6401MRljjMnJEoQxxpicLEEcctdIB9APi68wFl9hLL7CjPb4crI+CGOMMTlZDcIYY0xOliCMMcbkVFQJQkQuEJGXRaReRBbkWB4XkcXe8pUiMn0YY5smIstF5AUR2SAiN+UoM0dEGkVkrTfcOlzx+WLYLCLPe/t/NsdyEZEfecfwORE5fZjieovvuKwVkQMi8sWsMsN+/ETkXhHZKSLrffMmiMgfROQV77Wyl3Wv9Mq8IiJXDmN8t4vIS97792sRGd/Lun1+FgKMb6GIbPW9jx/qZd0+/98DjG+xL7bNIrK2l3UDP34FU9WiGIAw8CpwPBAD1gEzs8p8DrjTG78UWDyM8R0NnO6NJ4GNOeKbA/x2hI/jZqCqj+UfAn4HCPAuYOUIvdc7cBcAjejxA84BTgfW++Z9D1jgjS8AvptjvQnAJu+10huvHKb4zgMi3vh3c8WXz2chwPgWAl/N4zPQ5/97UPFlLf8/wK0jdfwKHYqpBjEbqFfVTaqaAhYB87LKzAN+6o0/ApwrIjIcwanqdlVd4403AS8CU4Zj30NsHvAzdf4CjBeRo4c5hnOBV1W1kCvrh4SqPgXszZrt/5z9FPhojlXPB/6gqntVdR/wB+CC4YhPVX+vqh3e5F+AqUO933z1cvzykc//e8H6is/77vg48POh3u9wKaYEMQV4wzfdwOFfwN1lvH+QRmDisETn4zVtnQaszLH4LBFZJyK/E5GThzUwR4Hfi8hqEbkux/J8jnPQLqX3f8qRPn4ANaq63RvfAdTkKDMajiPAp3E1wlz6+ywE6QavCezeXproRsPxOxt4U1Vf6WX5SB6/vBRTgjgiiEg58Evgi6p6IGvxGlyzyTuAfwceHe74gPeq6unAB4HPi8g5IxBDr0QkBvwD8Isci0fD8etBXVvDqDzXXERuATqAh3opMlKfhZ8AJwCnAttxzTij0WX0XXsY1f9LUFwJYiswzTc91ZuXs4yIRIAKYM+wROf2GcUlh4dU9VfZy1X1gKo2e+NLgaiIVA1XfN5+t3qvO4Ff46ryfvkc5yB9EFijqm9mLxgNx8/zZlezm/e6M0eZET2OInIV8GHgci+JHSaPz0IgVPVNVc2oaifwn73sd6SPXwS4CFjcW5mROn4DUUwJYhUwQ0SO835lXgosySqzBOg6W+Ri4Mne/jmGmtdeeQ/woqr+oJcyk7r6RERkNu79G84EViYiya5xXGfm+qxiS4BPeWczvQto9DWnDIdef7WN9PHz8X/OrgT+O0eZZcB5IlLpNaGc580LnIhcAPwT8A+q2tpLmXw+C0HF5+/TurCX/ebz/x6kvwNeUtWGXAtH8vgNyEj3kg/ngDvDZiPu7IZbvHm34f4RABK4pol64K/A8cMY23txTQ3PAWu94UPA9cD1XpkbgA24MzL+Arx7mI/f8d6+13lxdB1Df4wC3OEd4+eB2mGMrwz3hV/hmzeixw+XrLYDaVw7+DW4fq0ngFeAPwITvLK1wN2+dT/tfRbrgauHMb56XPt91+ew68y+ycDSvj4LwxTfA95n6zncl/7R2fF504f9vw9HfN78+7s+d76yw378Ch3sVhvGGGNyKqYmJmOMMQNgCcIYY0xOliCMMcbkZAnCGGNMTpYgjDHG5GQJwph+iEgm606xQ3ZnUBGZ7r8TqDGjSWSkAzDmCHBQVU8d6SCMGW5WgzBmkLz7+X/Pu6f/X0XkRG/+dBF50ruZ3BMicow3v8Z7vsI6b3i3t6mwiPynuOeA/F5ESrzyXxD3fJDnRGTRCP2ZpohZgjCmfyVZTUzzfcsaVXUW8GPg/3rz/h34qaq+HXejux95838E/I+6mwWejruCFmAGcIeqngzsBz7mzV8AnOZt5/qg/jhjemNXUhvTDxFpVtXyHPM3A+9X1U3ejRZ3qOpEEdmNu/1D2pu/XVWrRGQXMFVV233bmI577sMMb/prQFRVvyUijwPNuLvOPqrejQaNGS5WgzCmMNrL+EC0+8YzHOob/Hvcfa1OB1Z5dwg1ZthYgjCmMPN9ryu88T/j7h4KcDnwtDf+BPBZABEJi0hFbxsVkRAwTVWXA1/D3Xr+sFqMMUGyXyTG9K8k68Hzj6tq16mulSLyHK4WcJk370bgPhG5GdgFXO3Nvwm4S0SuwdUUPou7E2guYeBBL4kI8CNV3T9kf5ExebA+CGMGyeuDqFXV3SMdizFBsCYmY4wxOVkNwhhjTE5WgzDGGJOTJQhjjDE5WYIwxhiTkyUIY4wxOVmCMMYYk9P/B+focmaITksDAAAAAElFTkSuQmCC\n", 970 | "text/plain": [ 971 | "
" 972 | ] 973 | }, 974 | "metadata": { 975 | "needs_background": "light" 976 | }, 977 | "output_type": "display_data" 978 | }, 979 | { 980 | "data": { 981 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEXCAYAAAAEO/uqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xV9fnA8c+TQCaBhD0iQ2SYMGQriIAVxV1R1GqdVWtbR2u11mrrqPbXqrVLW1ddrRa1Ci6cSMRdloAEGUKAhBVCBtnr+f3xPQk3l6yb5OYm4Xm/XueVnP3cc8+9z/1+z/d8j6gqxhhjTCiEhToAY4wxhy9LQsYYY0LGkpAxxpiQsSRkjDEmZCwJGWOMCRlLQsYYY0KmQyQhEXlGRN4MdRx1EZGvReSuIO9jpoioiPSsbbyOdc4TkWa30W/MvkzdRCRNRG5uwe216c+DaZ+C9Tlv1STkvYD6hmeauOkbge+3YKitRkRuEpE8EYmpZV64iGSIyO+asOnPgH5AVrODrBlTbV+YQdlXS2vDyXIS8PdQB2EcETlFRBZ7n8siEVktIjeKSMh+tHufu9q+M38fqphaSmsf1H4+w9W1TLvRd2ER6dyYjapqrqrmtGCcrelfQCQwr5Z5p+KOyz8D3aiqlqrqbm2Fu5Fbc18diYhEAKhqpqoWhjqetqLquIRo3z8GFgErgKlAEu4Hwt3A862w//q+8+6h5vdlP+DeYMcUdKoakgE4z+2+enwwoMD3gA+BIuA6oAfwHyDdm7YOuMJvW88Ab/qMp+BOnN8B+4C9wINAWD3xNGY/DW4X6A285m1jG3Al8DVwVz37fhn4qJbpC4APvf9vAtYABUAG8CQQ77PsTO/49axt3Jt2qRdTIfAm8BO/92CoF/tubz8rgTP8Xr/6DvXsay6wFigBdgC3A+IzPw24A3gMyPOO+y2NOG9OA770jm8W8AYQ5c37PrAMOOC9Ny8DA/zOL9/hGW+eAL8AvvW2uxb4vt9+p3jHoxhY5cWhwEyfZU7wYisG9gB/AiL8jt8/vHMmE1jmcyxu9lmum7fcLm9b64ELAjhPn8Hn89DA8ZwDfAxkA/uBd4Gj/Zbpj/sCzvLOna+AWY18T2q8Np/j8LDfuXAX8BSQA7zsTf89sMHbbhpwf9V2G9o38Bvg61pe76fAX+s4Fom48/XPtcz7rvd+z/PGPwP+6LdMVy+Oud54BPAH770qxJ2bp9TymT0N+B9Qis/nzW/bhxxHv/lV2zrDe3+KcYl0gt9yDX0uI3Dfb9u8ZbYAN/jt4zveMS8ElgPj/c7df+E+f8Xe+j+t9xxszIkajIG6k1CaN2+Id1IMAG4BjgGOBK7x3qzv1PWh807yXNwvh+HA+UA58L164mnMfhrcLu5X1DpgGjDOWyef+pPQHKASOMpnWh+gDLjYG/8pcKJ3nGbgEtK/ajkJa01CuC/RSu+kGw78EPeh9X0PxgLXAqOBo7xlS4GR3vzu3ol7N9AX6FvHviYAFd5yw4GLvWNwvd+HKgv3Q+Mo4HpvG8c1cJzKcb/+koAxwM1AjDf/StwH+khgMrAEWOrNC8d9ANVbty/QzZt3H+7Lbg7uvLsIl4RP9+Z3wSWNF4BkYLb3HlcnIdz5UwA8ChyN+zLYjc8XlXcuHAD+CIzE+7LH5wsGlxA/BVK9eI7ElYjPCeA8fYbGJ6FzvWGYdzxfAjbjJU8gFtjkxTQd90NlLl4SasR7Uv3a/I6DfxLKw/0QOAoY5k3/Ne5zNNh7X7cDv23M+YD77igHJvssP8J7z8bWcSx+5s3vX8f8jcAC7/8f434M+v4AvQKXzCO98eeBL3A/To7EneulVfvn4OdmLXCyt0yvOvZ9yHH0m1+1rW+AU4BRuB9hu3zei8Z8Lqt+4JzrxTMLuNRvH//zpo/E/WhZj5fIgL/hkuBkYJC3zrx6z8FAk0dLDdSdhH7eiHXnA0/W9aHzTvLP/dZ533edRsbov596t+u9sQpM85k/yHvj60tCYbhfHr/zmXaLd0JH1bHOHNwvlTC/E6SuJPQC8L7fNp70fQ/q2M8XwB31fRhq2dfzeCU4n2XuAtL9tvMfv2U2+e6rllg+BeYH8P6N9OJKrC1Ob1os7tfrdL91/wws8v7/Ia6UEO0z/yJqJqH7vPh9v5Qu996jqi+BFGBNLXFWH1NcgqvErzTSnM9DgOd8rHe+Hu+NX41LnD3rWL7e96SO8yWFQ5PQG42I7VpgcwD7fhN41Gf8D8Dyepb/B5Bbz/zXgFTv/x4cmvw/AB73/h/qvY8D/baxEPi73/l4biNee5p3LuX7DWf4betin3W64EqWV3nj9X4ucT9EFJhTRwxV+/AtzU2j5mfsdeCpQM65ttg6brnviHdx/nYRWSMiWSKSj/slNrCB7azxG9+JqyqrVQD7qW+7R+NOvP9VzVTVbd4ydVLVSuBp4FIRCfcmXwk8r6rFXnwnisj7IpIuIgeAV3FF5771bdvH0cDnftNqjItIrIjcLyKpIpLtHYOJNHysa9vXp37TPgEGiEhXn2kBvUe4kuXiumaKyHgReU1EtnnHqOpcqi/+JFz1zTsikl81AD/CfZGAS2Zfq2qRz3pf+m3naOAL772s8gnuPTrKZ9qKemIB9xp3qer62mY24/NQKxEZKiIviMi3IpKHq0YM89neOFzi3FdPvHW+JwFY7j/Ba735iYjs9l7nn6j5Ohva9xPAhSIS7X2uLqHh66vamGBVNQt4B1eaQET640oH//YWGY8r1ab6nVenc/C8qnLIa6/DQ7gSsO+wxG+Z6s+0qubjSllJ3qSGPpfjcN9f/tv05/u5rfpuq/rc/gO4wGvM8aCIzGjoRXVqaIEQKPAbvxn4Oa7Rwlpc9v8d9X9ZgavK8qXU3xCjsftpzHYbdSL7eRpX/XCKiOTgvvi+ByAig4C3cB+q3+Cqscbjis4teRH3QVwJ62bcr/pC4LkW3ofvsQn0PaqTiMTiqgY+wH3Z7AV64q531Bd/1f7OxFX3+PKPr6l8X7P/+R2opn4e6vImrvrlh7jqpXJcVWBLveeVuC9jX7VdfK9xXETkWFwJ725cNVkOcBbuHG2st3Dn8Lm4avR4XI1AXTYC3URkgKpm1DI/CVcNW+XfwBNeY4YLcVXVH3vzwnDv+yQOPY+K/MYbe05kqermRi4bqEC+s3xfT9V6YQCq+rb3fXUq7trRWyLysqpeUdfG2mJJyN/xuKL6v1T1K9zF4+FtdD/f4I7p5KoJIjIQd2G3Xl6J6QPgB96wwosDXGkkAviZqn6uqhsbs00/64Fj/ab5jx8PPKeqr6jqGtyXk/+vtlLc9ZWG9jWtlm2nq+qBxod8iFW4E7s2I3FJ51equlRVv+HQL+ZS769v/Km4ao5BqrrZb9jmLfMNMEpEon3Wm0xN64Fj/ZrxHu/t89vGvDjPKqCfiBxdx/wW+zyISA/ccfudqn7glb7iqPnjdBUwpp5m7fW9J+CupfXz2WeUt8+GTAMyVPW3qrpMVTfhqrYbvW9VLcdVTV7pDa+qam49+/wv7gv2Fv8ZInIOrkTr20Lude/vGbgS0Qvq1Ul5sQnuuqn/eVVbgmsp1Z9p74fZKNy5CQ1/Lr/CfX/Nak4AqrrPOz8vx32XXSYikXUt3x6S0EbgOyJyvIiMBB7GXTxuc/tR1Q24IvpjInKciByD+xD4//Kpyz9xv8gvoGa1wSbce/VTERkiIt/DNVQIxF+Bk0TkNhEZJiJXA+f4LbMROMer1hqN+6UX5bdMGjBdRAbU88X0R2CGiNwlIsNF5GLcr/f7A4zZ333APBG5V0SSRCRZRH7m3WO1HZdMrhORI0XkdOC3futvw/1yO11EeolIF+/D9yDwoIhcKSJHicgxInKtiFzjrfcC7jrJE95+TwJ+5c2r+tL5O+6Hwd9F5Ghv/7/HXfsIpPn1YlxV3yve/SpDRGS2iHzXm9+Sn4dsXCvPq73XPQPXsKLcZ5kXcKXK10RkundszxKRqi+q+t4TcC1dLxZ3j1YyrgVcY2pgNuKqiS729vkjvJoBHw3tG9x1zxm4RFFvVZyq7sCdpzd41dLJ3vG/BldT8aKqvuyzfDHwCq6V53gOVsXh/VB8HnjGq1Y8UkQmisjNIjK3Ea+/NnEi0tdv6Oa3zB3e+VJ1rEs5WPqr93PpxfwS8KSInOu99ukickljAxSRe0Tku953zNG4quItqlpS50qBXEBqyYG6GyZM9FsuAXf9o6rZ7f24D3yKzzLPcGjDhIf9tlNjmVriacx+GtwurlXb67jEswO4igaaaPusG4H75ViI13LLZ94NuOqSItwX1fne8RqsNS8a1tdE+wrcl3UR8DautY7vezAIVxorwJWCbsZV1zzjs8yxwGpc80utZ19VTUFLqbuJdr0XrOs4RmfhrquU4L5AX+dgc+ALcCWDYtx1uVM4tBn1r3Ethiqp2UT7eg6WijJxDU5m+73uVd78VbgqHgWm+CxT1US7hINNtCMben3+xwJXbfSEF0exF9f5Tf08NHA8T8Sdn8Xe31NwVXyX+yyTCLyIqxIr9F7/zEa+J11x1ca5uPP3x/7HobZzwZv+f94xyPde84/wa0hT3759lvnQOy+kkcfkNNx1kQPecVmDq/485BYP7/gpsLKWeZ1xF/634D4Hu734JtT1uaknpjQOvcVAgX/7bessL94S3C0Fk/y209DnMtI7pzK8bXwLXFfP53wwPt/b3vbWeefJflxr4Xob2VQ1qzPGBEBEzsbdx9Vb675ob9oAEUnFNfK5L9SxBIuIzMQlzl7t7Xxsiw0TjGlzROQy3C/aHbh69j/jrs20qw/84UREeuFqXAbjboo2bZAlIWMapw+upVY/XLXKW8CtIY2oAV6jmNR6FklSVf8WgR3JXlwV3Q/tx0LbZdVxxnRQItIJVwqoS5q6FmTGhIwlIWOMMSHToarjevbsqYMHD27y+gUFBcTGxrZcQC3M4msei695LL7macvxrVixYp+q9grJzhvTZLG9DBMmTNDmWLJkSbPWDzaLr3ksvuax+JqnLcdHPX3qBXtoDzerGmOM6aAsCRljjAkZS0LGGGNCxpKQMcaYkLEkZIwxJmQsCRljjAkZS0LGGGNCpkPdrGqMMYeL/JJytmUVsGN/IduyCunbLYqzjxkQ6rACZknIGGPaIFUl80AJ2/YXsj2r0PtbUD2eVVBaY/mTk/pYEjLGGNN4haXlZGQXkZ5TRLpXoqlKMtv3F1JUVlG9bJhAv27RDOoRw8nJfTiiewyDuscyqEcMR3SPoVt05xC+kqazJGSMMUGgqmQXlpGRXURGThEfpZWx9I1UMnIKycgpIiO7iOzCshrrRHUOY1D3WI7oHsPxw3oyqEcMA7u7ITEhhohOHe8yviUhY4wJgKqSV1xOdkEp2YWl5BSWsb+glN15xdXJJSOniJ05RRSWVtRYNyZiOwPioxmQEM3YxHgGJEQzID6axIRojkiIoVdcJCISolcWGpaEjDGHLVUlr6icfQUl7C8orU4s2YVlLsEUlLG/sJScqmkFpeQUlVFRWfsjcBJiOjMgIZqhvWI5YVivGkkmbd1KTp8987BLMg2xJGSM6VBKyivIyi8lK7+UfQUl7DtQQlZBKVn5Jd60Um+aGy+vI6FEhIeRENuZhJgI4mM6M7xPF+JjIkiIcdMSYiKq5yfERNC7ayQxEXV/pe7bJJaAamFJyBjTLqkqW/YVkLIhk483ZbJ+RyGFKe9yoLj2h8VGdgqjZ5dIesZF0q9bFKMGdKVHl0h6xEbQs0sk3WNrJpaYiHBLGq3AkpAxpt0oKCnns2+z+GjjXlI2ZJKeXQTA0F6xDO4axtFHJtKzS8TB5BIXSc/YSHp0saTSVlkSMsa0WarKxj351UlnWdp+yiqU2Ihwph7Vk2tnDGXG8F4c0T2GlJQUZs5MDnXIJkCWhIwxbUpecRmfbd5HyoZMPtqYya7cYgBG9o3jymlDmDGiFxMHde+QzZUPR5aEjDEhVVpeyTe78/jESzwrt2VTXqnERXbi+GE9ufE7vZgxohf9ukWHOlQTBJaEjDGtpryikk1781mbnsvq9BzWZuTyza4DlFZUApDcvyvXnHAkM0f0ZtzAeDqHW2mno7MkZIwJiopKZeu+fNak53pDDqm78igucwknLrIToxO7ccXxgxkzIJ5JgxPo3TUqxFGb1mZJyBjTbKrKtqxC1mTksmZHDmsyclmXkUuB12NATEQ4o/p34+IpgxiT2I3RA7oxuEcsYWHWWu1wZ0nIGBOw3MIyvkrPYdX2bFZtz+GrHTnkFrl+0CI6hZHcvyvnTUhkdGI8YxK7MbRXF8It4ZhaWBIyxtSrvKKSjXvyWbK9jDdfXs2q7dl8m1kAgAiM6BPHqaP6cswR8YxO7MbwPnF2Lcc0miUhY0wNmQdK+GqHK+Ws3J7NmvTc6o44u8fuZfzAeOaOT2Scl3TiotrnIwRM22BJyJjD3PasQlI27mV5WjardmSzY7/rhaBTmJDUvyvnTzyCcQPjKdm5gXmnzrJeB0yLsiRkzGGmtLyS5Wn7+fCbvSzZsLe6aq1ftyjGDYzn0mMHM25gPKMGdCOqc3j1eik5mywBmRZnSciYw8CevGJSNuxlyTeZfLJ5H/kl5USEhzHlyO5cPGUQs0b2ZkjP2FCHaQ5DloSM6YAqKpWvduSwxCvtrNuZB7jSzlnH9GfWiN5MHdqD2Ej7CjChZWegMR1EdkEpSzdlsuSbvXy0MZPswjLCw4QJAxP4xZwRnDiyNyP6xFmVmmlTLAkZ005VVCqr03P4eOM+lm7KZNX2bCoVusdGMGtEb2aN7M0Jw3rRLcZar5m2y5KQMe3Izpwilm7MZOmmTD7ZtI+84nJEYPSAblw36yhmjezNmMR4uzHUtBuWhIxpwwpLy/lyy36Wbspk6cbM6pZsfbpGckpyX04Y3otpR/Wke2xEiCM1pmmCnoREZA7wFyAceFJVf+83/0/ALG80BuitqvHevMuAO7x596rqs8GO15hQUlXW7zpQnXSWp2VTWlFJZKcwphzZg+9NHsgJw3sxrHcXu7ZjOoSgJiERCQceAWYD6cAyEXldVVOrllHVn/ksfz0wzvu/O3AnMBFQYIW3bnYwYzamtZRXVLIrt5j07CK27y/gtTUl3PzJYvbllwCuO5zLpg7ihOG9mDS4e417dtolVSjKhv1bIdsb9qe5v7k7YMLlMP3noY7StLJgl4QmA5tVdQuAiMwHzgZS61j+e7jEA3AK8L6q7vfWfR+YA/wnqBEb00LKKirZnVvMjuxC0rOLSM8uIiO7iHRvfHdeMRWVWr18XGeYldSb6cN6csLwXvRpj481qKyAvIyDiWb/VshOO5hwSnJrLt+lL3Qf4v4uvgd6DIOks0IRuQkRUdWGl2rqxkXOA+ao6lXe+CXAFFW9rpZlBwFfAImqWiEiNwNRqnqvN//XQJGqPui33jXANQB9+vSZMH/+/CbHm5+fT5cuXZq8frDVFV9YRTHDNz7G/u7j2Nt7uutVMgTa2/FTVUoroahcKS6HskqoVKVSoUJxfyu9v37TD/6vVHjL5ZYo+4qUfUWV7CtS9hcrvp8uARKihJ7RQo9ooWd0GD2jhV7RYfSIEqIrC+ka136OX+fSHOIObK4eYgoziCreS5iWVy9TKZ0ojupNUXRfiqP6UhTdt8b/leGRAEhlGeNW3UZMYTorJvyRopgBzY6vrWnL8c2aNWuFqk4Mxb7bUsOEC4H/qmpFICup6uPA4wATJ07UmTNnNjmAlJQUmrN+sNUZ36d/hT0f0nfPhyQVL4fT/wg9h7Wd+IKsslJZtzOP3XnF5JeUcaC4vHqoGs8vLmfHniLCo8I5UFxGfombVl7Zcj/CwgT6do0iMSGG5MHRJCZEk5gQU/23b7coIjrV3bt0mz7/Cvez+p1nGRtWATtXwc6vIC/dmynQczgMmQQJQ1zJxvsb1nUAMWHhxDRmH+NHwmMnMCXtYbjqA4gIrAeHNn38aPvxhUqwk1AGcITPeKI3rTYXAj/xW3em37opLRhbx1BaCJ/9DYbMgKPPhMW/hX9MhWk3uvr1ztGhjjAoKiqV5Wn7efvr3bz99S725JUcskzncCEuqjNxUZ3oEtkJAQbERxMXFVc9LS6qM12iOhEX2YmITmGEhwmdwsT7642HC2HiMz286v8wn2WFhNiIjvEIg+Jc2LXaSzarIGMl5GxjbNX87kNh4LHQfxwMGA99x0BkC/zCjz8Czn0S/n0uvPkzOOexkJXqTesJdhJaBgwTkSG4pHIhcJH/QiIyEkgAPveZ/C7wOxFJ8MZPBm4Lbrjt0MpnoWAvzHwWBk2FpLPhvTtg6QOw5iU47UEYfnKoo2wR5RWV/C9tP2+v3c0763aTeaCEyE5hzBzRi1NH9WNory4uoXgJxv9CvvslGpIah7ZLFfZtgi1LIH25SzpZmw7Ojx/oks3EK/gqM4xj5lwG0fHBi+eo78CsX8GS++CIyTDpquDty7QJQU1CqlouItfhEko48JSqrhORe4Dlqvq6t+iFwHz1uUClqvtF5Le4RAZwT1UjBeMpK4ZP/wKDjncJCKBLb5j7OIz7Prz1c3hhHow8A079A3RLDE4cRdnw7Yf027kCVqQ1bRsSDsNmQ1zfGpPLKir5YksWi9bu5r11u8kqKCW6czgnjuzNqaP7MmtEb+v/LFAFWbA1Bb79EL5d4hoSAMT1dwlnzAXub/9xENujerWclJTgJqAq02+G9GXw9i+h3zhInBD8fZqQCfqnV1UXAYv8pv3Gb/yuOtZ9CngqaMG1d6v+BQd2uWoLf0NOgGs/hc//Bh89AA9Phpm/hGN/BOEt0I1LcR5seBvWvQqbF0NlGSMANjZjm51j4LifUDrlOj5NL+Xttbt4L3UPOYVlxEaEc+LRfTh9dF9mDO9NdEQ7b67cmspLYMeXB5POrtWAQlQ3V417wi0wdBYkDA51pE5YmDunH58BL10KP1xaIxmajsV+QrZX5aXwyZ/hiCku4dSmU4S7LjTqPHj7Vnj/17D6P3D6QzDouMD3WVrgJZ4FsOl9qCiBbkfAsddC0jl8ti6Nqcc1YbtAaX4W+9/5PX2XPkD+0kdJKZvL+51OYVbSAE4d5XoGaPf3ybQWVcjc4CWdD2Hbp1BWCGGdIHEyzLrdJZ3+4yCsjR7TmO5w/nPwz1PglR/A919pu7GaZrEk1F6tfsG1TjrrLw1fvE0YBBfNh2/ecsno6TlwzMUw+x6I7Vn/umVFsOk9+PpV2PgulBe5ezomXgmj5sKAie6XK1C6+QB07V/npiorld15xaRlFZC2r5C0rAK27itgW1YBaVmFlJZfxJSo47k35iXuLnyWuxKWIsm/hqSxdoG6Ifl7YevSg4nnwC43vccwVzU79EQYfDxExoU2zkD0HwenPQBv3AApv4cTbw91RCYILAm1RxVl8PFD0H88DP1O49cbeTocORM+uh8+f9glpZPugvGXVScSwFXfbF7sqto2vA2l+RDTE465yCWegcfV+au0slLZe6CErfsKvGRTUJ10tu0voLissnrZiE5hDOoew6Aesczw+kCbOvRUIsKvhc2LkQ/uhP9eCZ897BLmkOlNOVodT+F+2PWVT+u1VQebS0cnuPd46Ilw5CzX4qw9G38p7PgfLL0fEifC8FNCHZFpYZaE2qM1L0HONjj1/sBLCBGxMPtuGHuha7jw5k9h1b/htPvdl9vXr7rkVJLrvtBGzYXkuTB4OoQfPF1UlZ25xaTuzGP9LjesTStk3+J3aiaa8DAG9ohhcI8Ypg/ryeCesQzpGcugHjH06xZdd2/Pw05yVUZrXoQP74Nnz4BhJ7uk2Sc58GPWXhXn1Uw4O1e5Hgiq+DaXHjQV+o3tWNVWInD6g7B7Nbx6Dfzwo7Zz7cq0CEtC7U1FOXz8oLs3ozm/CnsfDZe/BavnuybdT5zopkd2cyWmUXPdL+rwzpSUV7Bpdz6pXrKpSjx5xQfvjB/cI4Ye0WGccsxABveMZXCPGAb3iKV/fD2JpiFh4a70lTwX/vcYfPxH+Mc0N23Wr4LX2q+xKivdBf91r8I3i6CyzCXuqHjXiqwx/0fFu2t3uJ4v2PZ5zYRTW3PpCZe7UnC/sa3TWi3UOkfD+f862FDhyvegczvs0sjUypJQe7PuVdi/BS74d/Ovk4jAMd+DEXNcaaj7ULL6Hs/6zFJSd+WyfuU6Unfm8W1mfnXPAtGdwxnZL44zxvbn6H5dSerXlZF944iN7OTdh5PUAi/ST+cod/PtuEtcIvrf4/D1KzDlWjj+Z637RawKGStciTF1oWve3CkKjjoJYnq45urFOZC3E/akuv9L8urfZudYiOrK9AN74GOvFOnbXHrAONdU+XBuIdZ9iGsx958L4e1b4Ky/hToi00IsCbUnWuFuQu2dDCNOb5FNFpaW8943hby56TjWZuSwJ+/j6nl9u0aR1L8rJyX1JqlfN47uF8egHrGhe2BaTHc45T6Y8kNXRffpX9zNutNvhslXQ6fI4OxX1TVrXveqaxmYsx3COrvEc9JdMOLU+i/4V5S7XgiKc1ySKsqp+X9RNhTnsi27jMFTz3HJx+9+KYM7ztN/7n6IJE6G8ZeEOiLTAiwJtSO9Mj+HfRvhvKdrNiQIUGWl8sWWLF5ZmcE7X++ioLSCAfHRTDuqJ0le6ebofl1JaKsPSosfCHMfg6nXwft3wnu3w5ePuZtdffotI2FwwP2PVVOFvamuxFNV+gzr5KooZ/zSVVk2tgQW3smVYhooyaSlpDB4xMymxXu4mHW769lh0c3Qb4yrkjTtmiWh9qKykkHbXoKeI1zXPE2wee8BXl2ZwcJVGezMLaZLZCfOGNOfueMHMGlwd8La2yOh+46GS151N2AufRC+/q8rcfiK7V2dmAblCazefTBJxfY6tEozc6NLOl+/Cvs2gIS5RhnTboSjz3KlMRM6YeFw3lPw2Anu+tA1Ke4am2m3LAm1FxveokvBNjjliYBaP2Xll/DG6p28uiqDNem5hIcJJwzryS9PO5qTk/p0jBtAh85yA7gWftneM2yqn2mTBmmfMDgvA9J8HkfVOdaVlroPcfc3bfsM9nwNiGtpNvlql/C79G7912TqFtsT5p1Zc/kAACAASURBVD0LT58KC66FC//TrJoBE1qWhNoDVfjofgqj+xOTPLfBxUvKK/hw/V5eWZlByoa9lFcqSf26csfpR3PWMf3pHdeBWxbFdHfDgEP7G/t48XucMGbwoUlq3yZ3g2efZDjl/yD5u/XedGvagCMmwSm/c40UPnkITrg51BGZJrIk1B5sfBd2r2H7iBsYGV77W6aqrNyezSsrM3hz9U7yisvpHRfJlccP4ZxxAzi6X9dWDrrtqQyPgF7D3WDav8lXuybyS+7zfnS0s+pkA1gSavtU4aM/QPwg9vSZwchaFlm8fg/3LVrPlswCojqHMSe5L3PHJzLtqJ6ha8lmTLCJwFl/hT3r4JUfEDnmD6GOyDSBJaG27tvFsHMlnPkX9EDNtysjp4i7X1/He6l7OKp3Fx44bwynju5HF3u0gTlcRMTCBf+Cx2cxadl1ELYajrvOmri3I/Zt1ZapQsofoGsijL0IPvkMcM/Y+ecnW/nLB5tQlFvnjOQHxw+p99HRxnRYPYfB1YvJevlm+nz+CHz5OIy7GKbe4BqdmDat0UlIREar6tpgBmP8bP0I0v/nno7qde3y5ZYsfv3a12zck8/spD7ceWYSiQkxIQ7UmBDrNYL1ST+nz5i/wKd/dT2ArHgWRp/netXofXSoIzR1CKQk9HcRiQSeAZ5X1dwGljfN9dEDENcPxl1CVn4JT6wp4dN3vmBAfDRPXjqRk5L6hDpCY9qW7kfCmX+GGbe6nuKXP+06wR1xuuttwZ7S2uY0Ogmp6nQRGQZcCawQkf8BT6vq+0GL7nCW9ils+4TKU37Pf1bu4f53NpBfXM6PZg7l+hOPIibCalKNqVPXfq6Lp+k/d71pfPkobHjLPUl2+s/dgyCb0/dieYm7p6y6s9nVro/Dqsei9x8HPYd3rB7NgySgbzJV3SQidwDLgb8C40REgF+p6qvBCPCwtfR+yqJ7cdGKESxL/5pjj+zOmf2LuHhObe3jjDG1iukOs25zXTyteMY9m+q5s9zDGKffBMNPbfhG14oy2LveSzYr3d89qa7XdHAd1/Yf5x4Auep518EuuJuh+42tTkrRhaWu53W7sbaGQK4JjQGuAE4H3gfOVNWVItIf+BywJNRCCr79jNgtKdxffjFbyyr40wVj+e4xA/joo49CHZox7VNkHEy9HiZd7Z5K/MmfYf5F0Otol4yS57o+/iorXP+MGSsPlnJ2r3WPsgeI6uaSytTrDpZ4uh1xsFRVWQFZm2uuv/yfUF7MFIDVt0H/sTVLTPGDDusnBwdSEvob8CSu1FNUNVFVd3qlI9NMqsqba3bRfeGvGKlxVIy7gsWnHkO3mM6hDs2YjqFzlHs0/bhLXY/oH/8RXr0aPrzX9ZKxazWUFbplI7pAv2PcTbFVCaP7kfUnjLBw6DXCDcd8z02rKIfMb/hmyX8YGVfoEtPnfz9Ykoru7rY9ZDokn3PYPbQvkCR0OlCkqhUAIhIGRKlqoar+KyjRHUa2ZObzm9fWkfftl7weuYpdk27lN2dMCnVYxnRM4Z1gzDwYdS5sfAc+fwQqSt3jxKsSTo9hLVN1Ft4J+o5id7/ZjJw5000rL3G9tFeVmDJWwgd3uWHABFcyS/5u6B/c2AoCSUIfACcB+d54DPAeMLWlgzqc5BWX8ciHm3n60zQiO4WxKHExeiCefrOvD3VoxnR8YWEw8jQ3tKZOkQeTXZXsNFc6+/pV93iS926HI451TzlOOrvD3oAbSBKKUtWqBISq5ouI3aDSROUVlfxn2Q7+9P5GsgtLOXd8Ir8aX0b3f6W4Z6bU95A0Y0zHkzDY3dN0/M8g61vvWVYL4O1fwNu3wuDjXXVd0tmuJ/EOIpAkVCAi41V1JYCITACKGljH1OKjjZnc91YqG/fkM3lId35zRhKjBnSDFy+ByK4w+ZpQh2iMCaUeQ2HGLW7Y+41XQnoF3roJFt3impiPmgsjz2j3z7gKJAn9FHhZRHbiuqvtC1wQlKg6qM17D3DvW+tJ2ZDJwO4xPPr98ZyS3BcRcU0+178OJ/yi8U/sNMZ0fL1HQu/bYOYvXWetVQ9dfP16ePMm9yyt5LmuSjGqW6ijDVggN6suE5GRwAhv0gZVLQtOWB3L/oJS/vzBRp7/cjsxncP51WkjuWzqYCI7+dzI9vGDrjXOsT8KXaDGmLZLBPqOcsOJv4ZdXx2sstt0rSsdXfZGqKMMWKC33Y8AkoAoYLyIoKrPtXxYHUNpeSXPfZ7GXxdvIr+knIumDORnJw2nR5fImguuftGdTNNubPdFa2NMKxA52LBh9j2Qvhwqy0MdVZMEcrPqncBMXBJaBJwKfAJYEvKjqryXuof/W7SetKxCpg/ryR2nJzGib5z/gu6pkIvvgcHT7emQxpjAibgnzbZTgZSEzgPGAqtU9QoR6QP8OzhhtV/rduZy75vr+XxLFkN7xfL05ZOYOaKXu+7jq6LcPZp4+VMweh6c/ffqnrKNMeZwEUgSKlLVShEpF5GuwF7giCDF1e7szSvmwfc28PKKdLpFd+bus5K5aMpAOofXcrNbaQH89wew8W3XHPPE31h/UsaYw1IgSWi5iMQDTwArcDetft7QSiIyB/gLEA48qaq/r2WZ84G7AAVWq+pF3vT7cT01hOH6q7tRVTWAmFtFdkEpJ/95KQUl5Vw5bQg3nDis7q52CvbBC+e7u6RPe9B1CWKMMYepRiUhr6fs/1PVHOBREXkH6KqqaxpYLxx4BJgNpAPLROR1VU31WWYYcBswTVWzRaS3N30qMA0Y4y36CTADSAng9bWK9bvzyCks4/FLJnBycj13NWd9C8+fB3k74YJ/w8jTWy9IY4xpgxqVhFRVRWQRMNobT2vk9icDm1V1C4CIzAfOBlJ9lrkaeERVs71t763aLa4VXgTuvqTOwJ5G7rdVZWS7e3aH96mnl4P05a4EpOqaUR4xuZWiM8aYtksaW7slIs8CD6vqskZvXOQ8YI6qXuWNXwJMUdXrfJZZCGzElXrCgbtU9R1v3oPAVbgk9LCq3l7LPq4BrgHo06fPhPnz5zc2vEPk5+fTpUuXgNdbsKmU174t44mTY+gcdmgPuz32/Y+k1AcojUhgzZg7KYoZ0KrxtRaLr3ksvuax+Jpu1qxZK1R1Ykh2rqqNGoBvgHLgW2ANsBZY08A65+GuA1WNX4JLJr7LvAkswJV0hgA7gHjgKOAtoIs3fA5Mr29/EyZM0OZYsmRJk9b7+Utf6aR736995rJ/qt4Vr/rYDNUDe5oamqo2Pb7WYvE1j8XXPBZf0wHLtZG5oKWHQBomnNKEHJdBzRZ0id40X+nAl+p6X9gqIhuBYbh7kr5Qr9NUEXkbOA74uAlxBFVGdhEDEqJrTlSFD3/rnlcy7BSY9zRExIYmQGOMaaMCaResdQz1WQYME5EhIhIBXAi87rfMQlzCQUR6AsOBLcB2YIaIdBKRzrhGCesDiLfVZOQUMSDeJwmVl8KCa10CGn8ZXPiCJSBjjKlFICWht3BJR3ANBoYAG4DkulZQ1XIRuQ54F3e95ylVXSci9+CKf697804WkVSgArhFVbNE5L/AibhqPwXeUdU21zFSZaWyK7eIU0d7reKK8+ClS2BLCsy6w/WCcBg/utcYY+oTSAemo33HRWQ88ONGrLcI182P77Tf+PyvwE3e4LtMBfDDxsYXKnsPlFBWoSTGR7um18/Pg8xvXA8I4y4OdXjGGNOmBdqBaTVVXSkiU1oymPYoI8c9j354WDo8+UMozoGLXoKjvhPiyIwxpu0LpANT35JKGDAe2NniEbUz6dlF9CSXiYt/BBFRcMUi6Dc21GEZY0y7EEhJyPdOzHLcNaJXWjac9icjp4jTw78gvDQXrnwL+o5ueCVjjDFAYNeE7g5mIO1VRnYR53b+EnonWQIyxpgANbqJtoi873VgWjWeICLvBies9qNg3w6OYQMknxPqUIwxpt0J5D6hXuo6MAVAXV9vvVs+pPZl2L4PCUMh6buhDsUYY9qdQJJQhYgMrBoRkUE0fLNqh6aqHFu8lD3RQ6HX8FCHY4wx7U4gDRNuBz4RkY9wN6xOx+s49HCVu2c7E2QDy/v9mD6hDsYYY9qhQBomvOPdoHqsN+mnqrovOGG1D0WrXyEeKBp2RqhDMcaYdimQhgnnAGWq+qaqvgmUi8hhfSEkauMbrK8cSPwRo0IdijHGtEuBXBO6U1Vzq0a8Rgp3tnxI7URuBglZK3mz4thDe9A2xhjTKIEkodqWbXK3P+1e6msAfBh2HAkxnUMcjDHGtE+BJKHlIvKQiAz1hoeAFcEKrM1LXciOiKGUJQxFrJdsY4xpkkCS0PVAKfCiN5QAPwlGUG1ebjrs+JIl4VNrPkfIGGNMQAJpHVcA/DKIsbQfqe65fK8UTyTZrgcZY0yTBdKLdi/gF7iH2EVVTVfVE4MQV9u2bgGVfUazelsvTraSkDHGNFkg1XHPA9/gnqh6N5CGe3z34SU3HdL/x/5BpwGQaCUhY4xpskCSUA9V/SfuXqGPVPVK3OO3Dy9eq7jNvU8CsGtCxhjTDIE0sS7z/u4SkdNxD7Tr3vIhtXHrFkDfMWwu7wNk2j1CxhjTDIEkoXtFpBvwc+BvQFfgZ0GJqq3K2QHpy+A7vyEjp4hOYULvuKiG1zPGGFOrQFrHven9mwvM8p8vIrep6v+1VGBtklcVR9J3yXgvj37xUYSH2T1CxhjTVIFcE2rIvBbcVtvkVcXRYygZOUV2PcgYY5qpJZNQxy4S5GyHjOXVT1DNyC5iQHxMiIMyxpj2rSWTUMd+wF1VVVzydyktr2TPgWJrlGCMMc1kJaHGWrcQ+o2F7keyO7cYVUi06jhjjGmWlkxCL7fgttqW7G01quLSswsBrCRkjDHN1GDrOBH5G/VUtanqDd7f37VgXG2LT6s4gPScIsBuVDXGmOZqTBPt5UGPoq1LXQj9joHuQwDXKAGgX7zdI2SMMc3RYBJS1WdbI5A2K3sbZKyAk+6unpSRU0TvuEgiO4WHMDBjjGn/Au1F+1YgicOpF22fVnFVMrKL7HqQMca0gEB70V5PgL1oi8gcEdkgIptFpNbnEYnI+SKSKiLrROQFn+kDReQ9EVnvzR8cQLwtY90C6D8OEg7u2m5UNcaYlhHUXrRFJBx4BDgVV4L6nogk+S0zDLgNmKaqycBPfWY/BzygqkcDk4G9AcTbfNlpsHNldas4gMpKZVeulYSMMaYlBLsX7cnAZlXdAiAi84GzgVSfZa4GHlHVbABV3estmwR0UtX3ven5AcTaMqpbxZ1dPWnvgRLKKtTuETLGmBYQSEnItxftm4EnabgX7QHADp/xdG+ar+HAcBH5VES+EJE5PtNzRORVEVklIg94JavWs24B9B/vVxVn9wgZY0xLEdXg9bYjIucBc1T1Km/8EmCKql7ns8ybuFLW+UAisBQYDZwE/BMYB2wHXgQWeVWCvvu4BrgGoE+fPhPmz5/f5Hjz8/Pp0qULAFFFuzn2yx/y7ZGXsWPg3OplvthZzqNrSrh3WjSJcS15r29g8bVFFl/zWHzNY/E13axZs1ao6sSQ7FxVGzUAzwLxPuMJwFMNrHMc8K7P+G3AbX7LPApc4TO+GJgEHAt85DP9Ely1XZ37mzBhgjbHkiVLDo58/CfVO7uq7k+rscwjSzbpoFvf1APFZc3aV1PUiK8Nsviax+JrHouv6YDl2shc0NJDID/lx6hqjk/yysaVUuqzDBgmIkNEJAK4EHjdb5mFwEwAEemJq4bb4q0b7zUNB9cIIpXWsm4BDJgACYNqTM7ILiI+pjNdIgO5nGaMMaY2gSShMBFJqBoRke400LBBVcuB64B3cc27X1LVdSJyj4ic5S32LpAlIqnAEuAWVc1S1QrctafFIrIW10HqEwHE23T7t8Kur6q76fFlzbONMablBPJz/o/A5yLyMi4hnAfc19BKqroIWOQ37Tc+/ytwkzf4r/s+MCaAGFtG6kL3N7mWJJRdxJCesa0ckDHGdEyNLgmp6nPAXGAPsBuYq6r/ClZgIVVVFRc/sMZkVXUlIWsZZ4wxLaIxvWh3VdU8r/ptN+Dbo0F3Vd0fzABb3f4tsGs1nHzvIbNyCssoLK2w6jhjjGkhjamOewE4A1hBzUc6iDd+ZBDiCp11XlWczw2qVTK8RzgkWknIGGNaRGN60T5DRASYoarbWyGm0EpdCAMmHlIVB5CeXfUcoZjWjsoYYzqkRl0T8hoPvBXkWEIuunCXq4rz6SvOV1VJyK4JGWNMywikifZKEZkUtEjagF6Zn7p/aqmKA9cyLrpzOAkxnVsxKmOM6bgCaaI9BbhYRLYBBXjXhFS19ZtQB0mvzE8hcRLEH1Hr/IycQgYkRONqJ40xxjRXIEnolKBF0RZkfUtc/haYdlWdi9iNqsYY07ICuU9oGxAPnOkN8d60jiG17lZxVeyJqsYY07IanYRE5Ebc01V7e8O/ReT6YAXW6tYtILfrCOiWWOvswtJysgvLrCRkjDEtKJDquB/gHsNQACAifwA+B/4WjMBaVda3sHstmUN/QLc6FsnItnuEjDGmpQXSOk6ACp/xCm9a+xfRBWb+isxeU+tcJL2qebaVhIwxpsUEUhJ6GvhSRBZ449/FPXSu/YvrAzNvpSQlpc5FqkpCdk3IGGNaTqOTkKo+JCIpwPHepCtUdVVQomqDMnKK6BQm9I6LCnUoxhjTYTQ6CXkdmKZ5Q9W0zqpa1vJhtT0Z2UX0i48iPKxj1EAaY0xbEFCPCUAmsBHY5P2fJiIrRWRCMIJrS+weIWOMaXmBJKH3gdNUtaeq9gBOBd4Efgz8PRjBtSXp2YXWcakxxrSwQJLQsar6btWIqr4HHKeqXwCRLR5ZG1JaXsneAyXWKMEYY1pYIK3jdonIrcB8b/wCYI+IhAOVLR5ZG7IrtwhVSLTqOGOMaVGBlIQuAhKBhcAC4AhvWjhwfsuH1nZY82xjjAmOQJpo7wOuF5HYql4TfGxu2bDaFrtR1RhjgiOQvuOmikgqsN4bHysiHb5BAhwsCfWLt3uEjDGmJQVSHfcn3OMcsgBUdTVwQjCCamsycoroHRdJZKfwUIdijDEdSiBJCFXd4TepotYFOxh7hIMxxgRHIEloh4hMBVREOovIzXhVcx2d3ahqjDHBEUgSuhb4CTAAyACOwd2o2qFVViq7cq0kZIwxwRDIfUIjVPVi3wkiMg34tGVDalv2HiihrELtHiFjjAmCQEpCtT28rv0/0K4BGTmFgN0jZIwxwdBgSUhEjgOmAr1E5CafWV1xN6p2aOlVN6pav3HGGNPiGlMdFwF08ZaN85meB5wXjKDakowc6y3BGGOCpcEkpKofAR+JyDOquq0VYmpTMrKLiI/pTJfIQC6fGWOMaYxArgkVisgDIrJIRD6sGhpaSUTmiMgGEdksIr+sY5nzRSRVRNaJyAt+87qKSLqIPBxArC3GmmcbY0zwBJKEnge+AYYAd+OesLqsvhW8HrYfwT17KAn4nogk+S0zDLgNmKaqycBP/TbzW2BpAHG2qIxsS0LGGBMsgSShHqr6T6BMVT9S1SuBExtYZzKwWVW3qGop7jEQZ/stczXwiKpmA6jq3qoZ3hNb+wDvBRBni1FVVxKy60HGGBMUgSShMu/vLhE5XUTGAd0bWGcA4NvVT7o3zddwYLiIfCoiX4jIHAARCQP+CNwcQIwtKqewjMLSCisJGWNMkARytf1eEekG/Bx3f1BXDq06a2oMw4CZuOcVLRWR0cD3gUWqmi4ida4sItcA1wD06dOHlJSUJgeSn59fY/20XNc1Xk7GFlJStjd5uy3FP762xuJrHouveSy+dkpVGzUAzwLxPuPdgacaWOc44F2f8duA2/yWeRS4wmd8MTAJdw1qO+7a0z5ck/Df17e/CRMmaHMsWbKkxvjba3fpoFvf1DU7cpq13ZbiH19bY/E1j8XXPBZf0wHLtZG5oKWHQKrjxqhqjk/y2g+Ma2CdZcAwERkiIhHAhcDrfsssxJWCEJGeuOq5Lap6saoOVNXBuCq551S11tZ1wWL3CBljTHAFkoTCRCShakREutNAdZ6qlgPXAe/ietx+SVXXicg9InKWt9i7QJb3wLwlwC2qmhXIiwiWjOwiojuHkxDTOdShGGNMhxTINaE/Ap+LyMve+DzgvoZWUtVFwCK/ab/x+V+Bm7yhrm08AzwTQKwtIiOnkAEJ0dR3TcoYY0zTNToJqepzIrKcg82y56pqanDCahvsRlVjjAmugPqi8ZJOh048vjKyixiTGB/qMIwxpsMK6PHeh5PC0nKyC8usJGSMMUFkSagOGd4jHBKtZZwxxgSNJaE6pFc1z7aSkDHGBI0loTpUlYTsHiFjjAkeS0J1SM8uolOY0DsuKtShGGNMh2VJqA4ZOUX0i48iPMzuETLGmGCxJFSHjOxCux5kjDFBZkmoDu5G1ZhQh2GMMR2aJaFalJZXsvdAiTVKMMaYILMkVItduUWoQqJVxxljTFBZEqqFNc82xpjWYUmoFnajqjHGtA5LQrWoKgn1i7d7hIwxJpgC6kX7cJGRU0TvuEgiO4WHOhTTwZSVlZGenk5xcfEh87p168b69etDEFXjWHzN0xbii4qKIjExkc6d286DOi0J1SIju8iuB5mgSE9PJy4ujsGDBx/ysMQDBw4QFxcXosgaZvE1T6jjU1WysrJIT09nyJAhIYvDn1XH1cIeZmeCpbi4mB49etjTek2rExF69OhRayk8lCwJ+amsVHblWknIBI8lIBMqbfHcsyTkZ++BEsoq1O4RMsaYVmBJyE9GTiFg9wgZY0xrsCTkJ736iarWb5w5vKWkpPDZZ5+1yr5OO+00cnJyAl7vmWee4brrrgtCRKa1WOs4Pxl2o6ppJXe/sY7UnXnV4xUVFYSHN++2gKT+XbnzzOTmhga4JNSlSxemTp3aIturjaqiqixatCho+2gNVa8jLMx+1wfKjpifjOwi4mM6Extp+dl0TM899xxjxoxh7NixXHLJJbzxxhtMmTKFcePGcdJJJ7Fnzx7S0tJ49NFH+dOf/sQxxxzDxx9/zL59+zj33HOZNGkSkyZN4tNPPwUgMzOT2bNnk5yczFVXXcWgQYPYt28fAA899BCjRo1i1KhR/PnPfwYgLS2NESNGcOmllzJq1Ch27NjB4MGDq9fxjw+oNcbGqGu9/Px8rrjiCkaPHs2YMWN45ZVXAHjnnXcYP348Y8eO5Tvf+Q4Ad911Fw8++GD1NkeNGkVaWlqtr+NHP/oREydOJDk5mTvvvLN6nWXLlnHSSScxduxYJk+ezIEDBzjhhBP46quvqpc5/vjjWb16deBvaHtXlcE7wjBhwgRtjiVLluhlT32pp/1labO2EyxLliwJdQj1svgalpqaWue8vLy8oO//66+/1mHDhmlmZqaqqmZlZen+/fu1srJSVVWfeOIJvemmm1RV9c4779QHHniget3zzjtPP/74Y1VV3bZtm44cOVJVVX/yk5/o7373O1VVffvttxXQzMxMXb58uY4aNUrz8/P1wIEDmpSUpCtXrtStW7eqiOjnn39eve1BgwZpZmZmrfGpap0xPv300/qTn/xEVWs/fnWt94tf/EJvvPHGGsvt3btXExMTdcuWLTX27X8ckpOTdevWrbW+jqp1ysvLdcaMGbp69WotKSnRIUOGVJ9/ubm5WlZWps8880x1DBs2bNDmfn81Vm3nILBcQ/S9bT/3/WRkFzGkZ2yowzAmKD788EPmzZtHz549AejevTtr167lggsuYNeuXZSWltZ5I2NKSgqbNm2qHs/LyyM/P59PPvmEBQsWADBnzhwSEhIA+OSTTzjnnHOIjXWfp7lz5/Lxxx9z1llnMWjQII499thGxQfuJt/GxOivrvU++OAD5s+fX71cQkICb7zxBieccEL1MlX7ro//63jppZd4/PHHKS8vZ9euXaSmpiIi9OvXjwkTJgDQtWtXAObNm8dvf/tbHnjgAZ566ikuv/zyRr2mjsaq43yoqrtR1VrGmcPI9ddfz3XXXcfatWt57LHH6ryZsbKyki+++IKvvvqKr776ioyMDLp06dKkfVYlppaOsaXW89WpUycqKyurx3234fs6tm7dyoMPPsjixYtZs2YNp59+er37i4mJYfbs2bz22mu89NJLXHzxxQHH1hFYEvJRUAaFpRXWKMF0WCeeeCIvv/wyWVlZAOzfv5/c3FwGDBgAwLPPPlu9bFxcHAcOHKix7t/+9rfq8arrGdOmTeOll14C4L333iM7OxuA6dOns3DhQgoLCykoKGDBggVMnz494PiAOmNsSF3rzZ49m0ceeaR6PDs7m2OPPZalS5eydevWGvsePHgwK1euBGDlypXV8/3l5eURGxtLt27d2LNnD2+//TYAI0aMYNeuXaxYsQJw3feUl5cDcNVVV3HDDTcwadKk6hLk4caSkI99Re7XTqKVhEwHlZyczO23386MGTMYO3YsN910E3fddRfz5s1jwoQJ1dVgAGeeeSYLFiyobpjwwAMPsHz5csaMGUNSUhKPPvooAHfeeSfvvfceo0aN4uWXX6Zv377ExcUxfvx4Lr/8ciZPnsyUKVO46qqrGDduXMDxAXXG2JC61rvjjjvIzs5m1KhRjB07liVLltCrVy8ef/xx5s6dy9ixY7ngggsAOPfcc9m/fz/Jyck8/PDDDB8+vNZ9jR07lnHjxjFy5Eguuugipk2bBkBERAQvvvgit9xyC2PHjmX27NnVJaQJEybQtWtXrrjiika/pg4nVBejgjE098Leg/Pf10G3vqlrduQ0azvB0hYurNfH4mtYqBsmNEdd8RUXF2tZWZmqqn722Wc6duzY1gyrWns8fhkZGTps2DCtqKhotTisYUIbtq9IAestwZhAbN++nfPPP5/KykoiIiJ44oknQh1Su/Dcc89x++2389BDDx3W9xcFPQmJyBzgL0A48KSqaYVhFQAAEIVJREFU/r6WZc4H7gIUWK2qF4nIMcA/gK5ABXCfqr4YzFiziiqJ7hxOQkzbedaGMW3dsGHDWLVqVUhjuO+++3jxxRdrfJnPmzeP22+/PYRR1e/SSy/l0ksvDXUYIRfUJCQi4cAjwGwgHVgmIq+raqrPMsOA24BpqpotIr29WYXApaq6SUT6AytE5F1VDbxvj0bKKlYGJES3yZ5mjTF1u/3227nhhhva9POETO2CXQacDGxW1S2qWgrMB872W+Zq4BFVzQZQ1b3e342qusn7fyewF+gVzGD3Fam1jDPGmFYU7Oq4AcAOn/F0YIrfMsMBRORTXJXdXar6ju8CIjIZiAC+9d+BiFwDXAPQp08fUlJSmhzsvsIKhhRlN2sbwZSfn99mYwOLrzG6detWo9mzr4qKijrntQUWX/O0lfiKi4tD/jnw1RYaJnQChgEzgURgqYiMrqp2E5F+wL+Ay1S10n9lVX0ceBxg4sSJOnPmzCYFUVBSTsE77zIpaSgzZx7VpG0EW0pKCk19fa3B4mvY+vXr66wyCvXjnxti8TVPW4kvKiqqwabyrSnY1XEZwBE+44neNF/pwOuqWqaqW4GNuKSEiHQF3gJuV9UvghpoTtUjHKw6zhhjWkuwk9AyYJiIDBGRCOBC4HW/ZRbiSkGISE9c9dwWb/kFwHOq+t8gx0lGtj3CwRh/Te2WpzYLFy4kNTW14QVbQFMfP+HfY7YJvqBWx6lquYhcB7yLu97zlKquE5F7cDdHve7NO1lEUnFNsW9R1SwR+T5wAtBDRC73Nnm5qn516J6aL73qOUJWEjKt5e1fwu611aPRFeUQ3syPZN/RcOohd0G0CQsXLuSMM84gKSkpaPsoLy/n/9u7++Ao6jSB49/HkDUGBaKskANuo8CebBwnBIy4EkTwJatbQfRiQPR42eLKXbkDr+Q2JZRFnV6VYJ0v2bNyBWfU7EahiLuRumNXRYJsScJr8bYuB8RC5S1M2E0koLzluT+6ZxzCTBgcZnpCnk/VVHq6fz39zG965pf+dffv6dGjR9KS8SVK8H10Bwm/Q0pVV6rqD1V1sKr+uzvvWbcBwr1h919U9Ueq6lPVpe7836hquqrmhT0S0gCBcySUJnD9NRmJ2oQxnisrKztnzLQFCxbw/PPPM378ePLz8/H5fLz33nsxv97ChQvx+Xz4/X7KysoAWLJkCbfeeit+v5+HH36YEydOsG7dOlasWMHcuXPJy8ujsbGRxsZGioqKGDFiBIWFhezatQuAxsZGRo0ahc/nY/78+aGjMVVl7ty53Hzzzfh8PpYtc24bXLNmDYWFhZSWloYauPAjuFhjjEW09Zqampg4cSJ+vx+/3x9qBMNzI82cOROAadOmUVPzbedOMNbg+yguLg69jwcffJARI0aQm5vL4sWLQ+t0zHvU3t7O0KFDCQQCgDPY7JAhQ0LPU5pXQzUk4hHPsD2z3t6iIxf873dePxlSYdiZzlh8F+b1sD1btmzRMWPGhJ4PGzZMv/jiC21tbVVV1UAgoIMHDw7l4OnZs2fU+FauXKm33367Hj9+XFW/zaXT3NwcKjNv3jwtLy9XVdWpU6fq8uXLQ8vGjRunu3fvVlXVhoYGveuuu1RV9YEHHtC3335bVVUrKipCMdTU1Ojdd9+tZ86c0cOHD+ugQYP04MGDWldXp5mZmbp9+/bQawfXudgYO+YO6ijaeo888oi+/PLLqurkEmppaTkvN9K+ffsi1kMw1uD7COYzCo/3xIkTmpubq83NzVHzHi1YsCAUw/vvv68PPfRQxPdgw/akqAN/PcF1V9lNqubyNnz4cI4cOcLBgwcJBAJkZWXRv39/nnrqKdauXcsVV1zBgQMHaGpqon///p2+1qpVq5g+fTqZmZnAt/l3du7cyfz582lpaaGtrY377rvvvHXb2tpYt24dJSUloXknT54EoL6+ntraWgAeffRRnn76acDJTzR58mTS0tLo168fd955Jxs3bqRXr14UFBSQk5NzSWOMJNp6q1evpqqqCoC0tDR69+5NVVVVxNxInSkoKDgnV1J5eXkoV9OXX37Jnj17CAQCEfMezZgxgwkTJjBnzhwqKyu7zKCo1gi5DrR8zZCru+/4Tab7KCkpoaamhsOHD1NaWkp1dTWBQIDNmzeTnp5OTk7Od8q7EzRt2jRqa2vx+/28+eabEe9JaW9vp0+fPuekt47HxeYniiXGS7leuPD8RO3t7Zw6dSq0LPx9rFmzhlWrVlFfX09mZiZjx47t9HMZNGgQ/fr1Y/Xq1WzYsIHq6uqLjs0L9qsLnDrTzpFjJ+lrR0KmGygtLWXp0qXU1NRQUlJCa2sr119/Penp6dTV1fH555/H9Dr33HMPb7zxRui8SDD/zrFjx8jOzub06dPn/BCG5yfq1asXN9xwA8uXLwec0wLbtm0DYNSoUbz77rsA52Q/LSwsZNmyZZw9e5ZAIMDatWspKCi4pDFeSLT1xo8fT0VFBeDclNra2ho1N1JOTk4ot9CKFSs4ffp0xG21traSlZVFZmYmu3btoqGhIVQ/kfIegZOf6LHHHqOkpIS0tLSY35eXrBECDrV+jSrWHWe6hdzcXI4dO8aAAQPIzs5mypQpbNq0CZ/PR1VVFTfddFNMr1NUVERxcTEjR44kLy8vdGnzc889x2233cYdd9xxzmtNmjSJF198keHDh9PY2Eh1dTWvv/46fr+f3Nzc0AURr7zyCi+99BK33HILe/fupXfv3gBMnDgxdJJ/3LhxLFq06IJdhhcb44VEW+/VV1+lrq4On8/HiBEj+PTTT8/LjfTMM88AMHPmTD7++GP8fj/19fVRj+KKioo4c+YMw4YNo6ysLJRGPFreI4Di4mLa2tq6TFccYBcmqKrua27TX1Rv1sraVd9p/WRJhRPrnbH4LszrCxPikaz4jh8/Hrow4p133tHi4uKY1rP6U924caOOHj260zJ2YUIK+sF1PXnt0fyUGk/JmO5q8+bNzJo1C1WlT58+VFZWeh1Sl/DCCy9QUVHRZc4FBVkjZIzp1I4dO3j88cdpb28P5eu58sorWb9+fUK2V1hYGDo/5JUnn3ySTz755Jx5s2fPTulurrKystB9UF2JNULGJJmqdqmcVT6fj61bt6bMAJzJEH5D7+XE6XlLLXZhgjFJlJGRwdGjR1Pyx8Bc3lSVo0ePkpGRWqPC2JGQMUk0cOBA9u/fH3E4lW+++SblfiDCWXzxSYX4MjIyGDhwoKcxdGSNkDFJlJ6efs4d8eHWrFmTUnleOrL44pPq8XnFuuOMMcZ4xhohY4wxnrFGyBhjjGfkcrpKR0QCQGwDX0XWF2i+ROEkgsUXH4svPhZffFI5vh+o6ve92PBl1QjFS0Q2qepIr+OIxuKLj8UXH4svPqken1esO84YY4xnrBEyxhjjGWuEzrX4wkU8ZfHFx+KLj8UXn1SPzxN2TsgYY4xn7EjIGGOMZ6wRMsYY45lu1wiJSJGI/J+I7BWR85JviMiVIrLMXb5eRHKSGNsgEakTkU9F5E8iMjtCmbEi0ioiW93Hs8mKLyyGfSKyw93+pgjLRUTK3TrcLiL5SYzt78LqZquIfCUiczqUSWodikiliBwRkZ1h864VkQ9FZI/7NyvKulPdMntEZGoS43tRRHa5n9/vRKRPlHU73RcSGN8CETkQ9hneH2XdTr/vCYxvWVhs+0Rka5R1E15/Kc+rlK5ePIA0oBG4EfgesA34UYcyvwD+y52eBCxLYnzZQL47fQ2wO0J8Y4H/8bge9wF9O1l+P/B7QIBRwHoPP+/DODfieVaHwBggH9gZNm8RUOZOlwELI6x3LfCZ+zfLnc5KUnz3Aj3c6YWR4otlX0hgfAuAp2P4/Dv9vicqvg7L/wN41qv6S/VHdzsSKgD2qupnqnoKWApM6FBmAvCWO10DjJckZSBT1UOqusWdPgb8GRiQjG1fYhOAKnU0AH1EJNuDOMYDjaoazygacVPVtcBfOswO38/eAh6MsOp9wIeq+hdV/SvwIVCUjPhU9QNVPeM+bQA8G/8/Sv3FIpbve9w6i8/97XgEeOdSb/dy0d0aoQHAl2HP93P+j3yojPslbAWuS0p0YdxuwOFApBzKt4vINhH5vYjkJjUwhwIfiMhmEfnHCMtjqedkmET0L7/XddhPVQ+504eBfhHKpEo9zsA5so3kQvtCIs1yuwsro3RnpkL9FQJNqronynIv6y8ldLdGqEsQkauBd4E5qvpVh8VbcLqX/MCvgNpkxweMVtV84CfAkyIyxoMYOiUi3wOKgeURFqdCHYao0y+TkvdKiMg84AxQHaWIV/tCBTAYyAMO4XR5paLJdH4UlPLfpUTrbo3QAWBQ2POB7ryIZUSkB9AbOJqU6JxtpuM0QNWq+tuOy1X1K1Vtc6dXAuki0jdZ8bnbPeD+PQL8DqfbI1ws9ZxoPwG2qGpTxwWpUIdAU7CL0v17JEIZT+tRRKYBPwWmuA3leWLYFxJCVZtU9ayqtgNLomzX6/rrATwELItWxqv6SyXdrRHaCAwVkRvc/5QnASs6lFkBBK9C+ntgdbQv4KXm9h+/DvxZVV+KUqZ/8ByViBTgfIbJbCR7isg1wWmcE9g7OxRbAfyDe5XcKKA1rOspWaL+B+p1HbrC97OpwHsRyrwP3CsiWW53073uvIQTkSLgX4FiVT0RpUws+0Ki4gs/xzgxynZj+b4n0t3ALlXdH2mhl/WXUry+MiLZD5wrt3bjXDUzz533bzhfNoAMnC6cvcAG4MYkxjYap1tmO7DVfdwPPAE84ZaZBfwJ50qfBuDHSa6/G91tb3PjCNZheIwCvObW8Q5gZJJj7InTqPQOm+dZHeI0hoeA0zjnJX6Gc57xI2APsAq41i07EvjvsHVnuPviXmB6EuPbi3M+JbgfBq8Y/RtgZWf7QpLi+7W7b23HaViyO8bnPj/v+56M+Nz5bwb3ubCySa+/VH/YsD3GGGM8092644wxxqQQa4SMMcZ4xhohY4wxnrFGyBhjjGesETLGGOMZa4SMiYGInO0wOvclG5FZRHLCR2A2pjvp4XUAxnQRX6tqntdBGHO5sSMhY+Lg5oNZ5OaE2SAiQ9z5OSKy2h1g8yMR+Vt3fj83P8829/Fj96XSRGSJOHmkPhCRq9zy/yxOfqntIrLUo7dpTMJYI2RMbK7q0B1XGrasVVV9wH8Cr7jzfgW8paq34Az+We7OLwc+Vmfw1HycO+UBhgKvqWou0AI87M4vA4a7r/NEot6cMV6xEROMiYGItKnq1RHm7wPGqepn7uCzh1X1OhFpxhlK5rQ7/5Cq9hWRADBQVU+GvUYOTt6goe7zXwLpqvq8iPwBaMMZ6btW3YFXjblc2JGQMfHTKNMX42TY9Fm+PV/7AM44fPnARndkZmMuG9YIGRO/0rC/9e70OpxRmwGmAH90pz8Cfg4gImki0jvai4rIFcAgVa0DfomTVuS8ozFjujL7r8qY2FwlIlvDnv9BVYOXaWeJyHaco5nJ7rx/At4QkblAAJjuzp8NLBaRn+Ec8fwcZwTmSNKA37gNlQDlqtpyyd6RMSnAzgkZEwf3nNBIVW32OhZjuiLrjjPGGOMZOxIyxhjjGTsSMsYY4xlrhIwxxnjGGiFjjDGesUbIGGOMZ6wRMsYY45n/B3fqkoVSIqa9AAAAAElFTkSuQmCC\n", 982 | "text/plain": [ 983 | "
" 984 | ] 985 | }, 986 | "metadata": { 987 | "needs_background": "light" 988 | }, 989 | "output_type": "display_data" 990 | } 991 | ], 992 | "source": [ 993 | "def plot_result(item):\n", 994 | " plt.plot(history.history[item], label=item)\n", 995 | " plt.plot(history.history[\"val_\" + item], label=\"val_\" + item)\n", 996 | " plt.xlabel(\"Epochs\")\n", 997 | " plt.ylabel(item)\n", 998 | " plt.title(\"Train and Validation {} Over Epochs\".format(item), fontsize=14)\n", 999 | " plt.legend()\n", 1000 | " plt.grid()\n", 1001 | " plt.show()\n", 1002 | "\n", 1003 | "\n", 1004 | "plot_result(\"loss\")\n", 1005 | "plot_result(\"categorical_accuracy\")" 1006 | ] 1007 | }, 1008 | { 1009 | "cell_type": "markdown", 1010 | "metadata": { 1011 | "id": "XkwRAIcsNeXj" 1012 | }, 1013 | "source": [ 1014 | "While training, we notice an initial sharp fall in the loss followed by a gradual decay." 1015 | ] 1016 | }, 1017 | { 1018 | "cell_type": "markdown", 1019 | "metadata": { 1020 | "id": "wBvqeuk88G9r" 1021 | }, 1022 | "source": [ 1023 | "### Evaluate the model" 1024 | ] 1025 | }, 1026 | { 1027 | "cell_type": "code", 1028 | "execution_count": null, 1029 | "metadata": { 1030 | "colab": { 1031 | "base_uri": "https://localhost:8080/" 1032 | }, 1033 | "id": "sxz8yDaT8MdL", 1034 | "outputId": "04cbcc20-9c7d-417a-fc0f-f67236533711" 1035 | }, 1036 | "outputs": [ 1037 | { 1038 | "name": "stdout", 1039 | "output_type": "stream", 1040 | "text": [ 1041 | "15/15 [==============================] - 0s 23ms/step - loss: 0.0200 - categorical_accuracy: 0.6978\n", 1042 | "Categorical accuracy on the test set: 69.78%.\n" 1043 | ] 1044 | } 1045 | ], 1046 | "source": [ 1047 | "_, categorical_acc = shallow_mlp_model.evaluate(test_dataset)\n", 1048 | "print(f\"Categorical accuracy on the test set: {round(categorical_acc * 100, 2)}%.\")" 1049 | ] 1050 | }, 1051 | { 1052 | "cell_type": "markdown", 1053 | "metadata": { 1054 | "id": "of0EtUhKNjv6" 1055 | }, 1056 | "source": [ 1057 | "The trained model gives us a validation accuracy of ~70%." 1058 | ] 1059 | }, 1060 | { 1061 | "cell_type": "markdown", 1062 | "metadata": { 1063 | "id": "SeCfK4daFGJq" 1064 | }, 1065 | "source": [ 1066 | "## Inference" 1067 | ] 1068 | }, 1069 | { 1070 | "cell_type": "code", 1071 | "execution_count": 69, 1072 | "metadata": { 1073 | "colab": { 1074 | "base_uri": "https://localhost:8080/" 1075 | }, 1076 | "id": "axBPAMpe-sp2", 1077 | "outputId": "3ba99084-0491-47ce-dce4-f74b69708e7e" 1078 | }, 1079 | "outputs": [ 1080 | { 1081 | "name": "stdout", 1082 | "output_type": "stream", 1083 | "text": [ 1084 | "Abstract: b'Parametric causal modelling techniques rarely provide functionality for\\ncounterfactual estimation, often at the expense of modelling complexity. Since\\ncausal estimations depend on the family of functions used to model the data,\\nsimplistic models could entail imprecise characterizations of the generative\\nmechanism, and, consequently, unreliable results. This limits their\\napplicability to real-life datasets, with non-linear relationships and high\\ninteraction between variables. We propose Deep Causal Graphs, an abstract\\nspecification of the required functionality for a neural network to model\\ncausal distributions, and provide a model that satisfies this contract:\\nNormalizing Causal Flows. We demonstrate its expressive power in modelling\\ncomplex interactions and showcase applications of the method to machine\\nlearning explainability and fairness, using true causal counterfactuals.'\n", 1085 | "Label(s): ('cs.LG', 'cs.NE', 'stat.ML')\n", 1086 | "Predicted Label(s): (cs.LG, stat.ML, physics.data-an)\n", 1087 | " \n", 1088 | "Abstract: b\"Deep learning inference on embedded devices is a burgeoning field with myriad\\napplications because tiny embedded devices are omnipresent. But we must\\novercome major challenges before we can benefit from this opportunity. Embedded\\nprocessors are severely resource constrained. Their nearest mobile counterparts\\nexhibit at least a 100 -- 1,000x difference in compute capability, memory\\navailability, and power consumption. As a result, the machine-learning (ML)\\nmodels and associated ML inference framework must not only execute efficiently\\nbut also operate in a few kilobytes of memory. Also, the embedded devices'\\necosystem is heavily fragmented. To maximize efficiency, system vendors often\\nomit many features that commonly appear in mainstream systems, including\\ndynamic memory allocation and virtual memory, that allow for cross-platform\\ninteroperability. The hardware comes in many flavors (e.g., instruction-set\\narchitecture and FPU support, or lack thereof). We introduce TensorFlow Lite\\nMicro (TF Micro), an open-source ML inference framework for running\\ndeep-learning models on embedded systems. TF Micro tackles the efficiency\\nrequirements imposed by embedded-system resource constraints and\"\n", 1089 | "Label(s): ('cs.AI', 'cs.LG')\n", 1090 | "Predicted Label(s): (cs.LG, stat.ML, cs.AI)\n", 1091 | " \n", 1092 | "Abstract: b'Lidar odometry (LO) is a key technology in numerous reliable and accurate\\nlocalization and mapping systems of autonomous driving. The state-of-the-art LO\\nmethods generally leverage geometric information to perform point cloud\\nregistration. Furthermore, obtaining point cloud semantic information which can\\ndescribe the environment more abundantly will help for the registration. We\\npresent a novel semantic lidar odometry method based on self-designed\\nparameterized semantic features (PSFs) to achieve low-drift ego-motion\\nestimation for autonomous vehicle in realtime. We first use a convolutional\\nneural network-based algorithm to obtain point-wise semantics from the input\\nlaser point cloud, and then use semantic labels to separate the road, building,\\ntraffic sign and pole-like point cloud and fit them separately to obtain\\ncorresponding PSFs. A fast PSF-based matching enable us to refine geometric\\nfeatures (GeFs) registration, reducing the impact of blurred submap surface on\\nthe accuracy of GeFs matching. Besides, we design an efficient method to\\naccurately recognize and remove the dynamic objects while retaining static ones\\nin the semantic point cloud,'\n", 1093 | "Label(s): ('cs.CV',)\n", 1094 | "Predicted Label(s): (cs.CV, cs.RO, cs.GR)\n", 1095 | " \n", 1096 | "Abstract: b'Deep neural networks have been shown to suffer from poor generalization when\\nsmall perturbations are added (like Gaussian noise), yet little work has been\\ndone to evaluate their robustness to more natural image transformations like\\nphoto filters. This paper presents a study on how popular pretrained models are\\naffected by commonly used Instagram filters. To this end, we introduce\\nImageNet-Instagram, a filtered version of ImageNet, where 20 popular Instagram\\nfilters are applied to each image in ImageNet. Our analysis suggests that\\nsimple structure preserving filters which only alter the global appearance of\\nan image can lead to large differences in the convolutional feature space. To\\nimprove generalization, we introduce a lightweight de-stylization module that\\npredicts parameters used for scaling and shifting feature maps to \"undo\" the\\nchanges incurred by filters, inverting the process of style transfer tasks. We\\nfurther demonstrate the module can be readily plugged into modern CNN\\narchitectures together with skip connections. We conduct extensive studies on\\nImageNet-Instagram, and show quantitatively and'\n", 1097 | "Label(s): ('cs.CV', 'cs.LG', 'eess.IV')\n", 1098 | "Predicted Label(s): (cs.CV, eess.IV, cs.LG)\n", 1099 | " \n", 1100 | "Abstract: b'Video summarization is an effective way to facilitate video searching and\\nbrowsing. Most of existing systems employ encoder-decoder based recurrent\\nneural networks, which fail to explicitly diversify the system-generated\\nsummary frames while requiring intensive computations. In this paper, we\\npropose an efficient convolutional neural network architecture for video\\nSUMmarization via Global Diverse Attention called SUM-GDA, which adapts\\nattention mechanism in a global perspective to consider pairwise temporal\\nrelations of video frames. Particularly, the GDA module has two advantages: 1)\\nit models the relations within paired frames as well as the relations among all\\npairs, thus capturing the global attention across all frames of one video; 2)\\nit reflects the importance of each frame to the whole video, leading to diverse\\nattention on these frames. Thus, SUM-GDA is beneficial for generating diverse\\nframes to form satisfactory video summary. Extensive experiments on three data\\nsets, i.e., SumMe, TVSum, and VTW, have demonstrated that SUM-GDA and its\\nextension outperform other competing state-of-the-art methods with remarkable\\nimprovements. In addition,'\n", 1101 | "Label(s): ('cs.CV', 'cs.MM')\n", 1102 | "Predicted Label(s): (cs.CV, cs.LG, eess.IV)\n", 1103 | " \n" 1104 | ] 1105 | } 1106 | ], 1107 | "source": [ 1108 | "text_batch, label_batch = next(iter(test_dataset))\n", 1109 | "predicted_probabilities = shallow_mlp_model.predict(text_batch)\n", 1110 | "\n", 1111 | "for i, text in enumerate(text_batch[:5]):\n", 1112 | " label = label_batch[i].numpy()[None, ...]\n", 1113 | " print(f\"Abstract: {text[0]}\")\n", 1114 | " print(f\"Label(s): {mlb.inverse_transform(label)[0]}\")\n", 1115 | " predicted_proba = [proba for proba in predicted_probabilities[i]]\n", 1116 | " top_3_labels = [x for _, x in sorted(zip(predicted_probabilities[i], mlb.classes_), key=lambda pair: pair[0], reverse=True)][:3]\n", 1117 | " print(f\"Predicted Label(s): ({', '.join([label for label in top_3_labels])})\")\n", 1118 | " print(\" \")" 1119 | ] 1120 | }, 1121 | { 1122 | "cell_type": "markdown", 1123 | "metadata": { 1124 | "id": "5-gQfc11SwAv" 1125 | }, 1126 | "source": [ 1127 | "The prediction results are not that great but not below the par for a simple model like ours. We can improve this performance with models that consider word order like LSTM or even those that use Transformers ([Vaswani et al.](https://arxiv.org/abs/1706.03762))." 1128 | ] 1129 | } 1130 | ], 1131 | "metadata": { 1132 | "accelerator": "GPU", 1133 | "colab": { 1134 | "collapsed_sections": [], 1135 | "name": "multi_label_trainer_tfidf.ipynb", 1136 | "provenance": [] 1137 | }, 1138 | "kernelspec": { 1139 | "display_name": "Python 3 (ipykernel)", 1140 | "language": "python", 1141 | "name": "python3" 1142 | }, 1143 | "language_info": { 1144 | "codemirror_mode": { 1145 | "name": "ipython", 1146 | "version": 3 1147 | }, 1148 | "file_extension": ".py", 1149 | "mimetype": "text/x-python", 1150 | "name": "python", 1151 | "nbconvert_exporter": "python", 1152 | "pygments_lexer": "ipython3", 1153 | "version": "3.8.2" 1154 | } 1155 | }, 1156 | "nbformat": 4, 1157 | "nbformat_minor": 1 1158 | } 1159 | --------------------------------------------------------------------------------