├── 01-finetune-opt-with-lora.ipynb ├── 02-finetune-gpt2-with-lora.ipynb ├── Readme.md └── images ├── auto_regressive_transformer.png └── lora.png /01-finetune-opt-with-lora.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "857cafc6-da38-4aa7-8afc-63aa626fa7aa", 6 | "metadata": {}, 7 | "source": [ 8 | "# 01. Finetuning OPT with LoRA\n", 9 | "\n", 10 | "Today's popular auto-regressive models - such as, GPT, LLaMA, Falcon, etc - are decoder-only models, in which the output token is predicted by using only input's text (called a prompt).\n", 11 | "\n", 12 | "![Decoder-only transformers](./images/auto_regressive_transformer.png)\n", 13 | "\n", 14 | "*\"Decoder-only\" model is implemented using layers in the red box.
\n", 15 | "(Diagram from : [Attention Is All You Need](https://arxiv.org/abs/1706.03762))*\n", 16 | "\n", 17 | "In this model, the task is differentiated also by using input's text (i.e, prompt).\n", 18 | "\n", 19 | "> Note : See [this repository](https://github.com/tsmatz/nlp-tutorials) for intrinsic idea of LLM transformers.\n", 20 | "\n", 21 | "In this example, we fine-tune the pre-trained auto-regressive model, Meta's OPT (```facebook/opt-125m```), by applying LoRA (Low-Rank Adaptation) optimization.\n", 22 | "\n", 23 | "In this example, I download the pre-trained model from Hugging Face hub, but fine-tune model with regular PyTorch training loop.
\n", 24 | "(Here I don't use Hugging Face Trainer class.)\n", 25 | "\n", 26 | "See [Readme](https://github.com/tsmatz/finetune_llm_with_lora) for prerequisite's setup." 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "id": "3d49acf1-9ad1-4a6c-9312-6785cb3f5862", 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "model_name = \"facebook/opt-125m\"\n", 37 | "# model_name = \"facebook/opt-350m\"\n", 38 | "# model_name = \"facebook/opt-1.3b\"\n", 39 | "# model_name = \"facebook/opt-6.7b\"" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 2, 45 | "id": "4d835e84-a01d-4c33-926b-60d9dd4a7627", 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "import torch\n", 50 | "\n", 51 | "device = torch.device(\"cuda\")" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "id": "ead383e5-149b-4bfb-9324-3cc639fd398d", 57 | "metadata": {}, 58 | "source": [ 59 | "## Prepare dataset and dataloader" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "id": "91ecbb08-6a74-4623-bfe8-bddba5254e35", 65 | "metadata": {}, 66 | "source": [ 67 | "In this example, we use dataset used in [official LoRA example](https://github.com/microsoft/LoRA).\n", 68 | "\n", 69 | "Download dataset from official repository." 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 3, 75 | "id": "54a564f1-f8f3-42a6-b160-bebdbcc3aac0", 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "name": "stdout", 80 | "output_type": "stream", 81 | "text": [ 82 | "--2023-10-06 03:27:50-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt\n", 83 | "Resolving github.com (github.com)... 140.82.114.3\n", 84 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n", 85 | "HTTP request sent, awaiting response... 302 Found\n", 86 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt [following]\n", 87 | "--2023-10-06 03:27:51-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt\n", 88 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n", 89 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n", 90 | "HTTP request sent, awaiting response... 200 OK\n", 91 | "Length: 9624463 (9.2M) [text/plain]\n", 92 | "Saving to: ‘train.txt’\n", 93 | "\n", 94 | "train.txt 100%[===================>] 9.18M --.-KB/s in 0.04s \n", 95 | "\n", 96 | "2023-10-06 03:27:51 (248 MB/s) - ‘train.txt’ saved [9624463/9624463]\n", 97 | "\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": 4, 108 | "id": "d48464ea-991f-48b2-9166-3323cfd61676", 109 | "metadata": { 110 | "scrolled": true 111 | }, 112 | "outputs": [ 113 | { 114 | "name": "stdout", 115 | "output_type": "stream", 116 | "text": [ 117 | "--2023-10-06 03:27:54-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt\n", 118 | "Resolving github.com (github.com)... 140.82.114.3\n", 119 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n", 120 | "HTTP request sent, awaiting response... 302 Found\n", 121 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt [following]\n", 122 | "--2023-10-06 03:27:54-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt\n", 123 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n", 124 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", 125 | "HTTP request sent, awaiting response... 200 OK\n", 126 | "Length: 1351149 (1.3M) [text/plain]\n", 127 | "Saving to: ‘test.txt’\n", 128 | "\n", 129 | "test.txt 100%[===================>] 1.29M --.-KB/s in 0.006s \n", 130 | "\n", 131 | "2023-10-06 03:27:54 (208 MB/s) - ‘test.txt’ saved [1351149/1351149]\n", 132 | "\n" 133 | ] 134 | } 135 | ], 136 | "source": [ 137 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "id": "09472803-8c62-48e0-9a63-b9b9448f16d3", 143 | "metadata": {}, 144 | "source": [ 145 | "Show the downloaded data (first 5 rows)." 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "id": "e6e60596-028f-4c4b-a95d-f74a0ff3b188", 152 | "metadata": {}, 153 | "outputs": [ 154 | { 155 | "name": "stdout", 156 | "output_type": "stream", 157 | "text": [ 158 | "name : The Vaults | Type : pub | price : more than £ 30 | customer rating : 5 out of 5 | near : Café Adriatic||The Vaults pub near Café Adriatic has a 5 star rating . Prices start at £ 30 . \n", 159 | "name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Café Brazil||Close to Café Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of £ 10.50 . Delicious Pub food . \n", 160 | "name : The Eagle | Type : coffee shop | food : Japanese | price : less than £ 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King||The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than £ 20 for Japanese food . \n", 161 | "name : The Mill | Type : coffee shop | food : French | price : £ 20 - 25 | area : riverside | near : The Sorrento||Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at £ 20- £ 25 it is in the riverside area . \n", 162 | "name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat||For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat . \n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "!head -n 5 train.txt" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "id": "93f5fabe-590c-459b-aa16-4b5a506fb54b", 173 | "metadata": {}, 174 | "source": [ 175 | "Convert above data into JsonL format." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 6, 181 | "id": "7376e0c0-16c9-46f4-ad4c-83d1a677f5a2", 182 | "metadata": {}, 183 | "outputs": [], 184 | "source": [ 185 | "import sys\n", 186 | "import io\n", 187 | "import json\n", 188 | "\n", 189 | "def format_convert(read_file, write_file):\n", 190 | " with open(read_file, \"r\", encoding=\"utf8\") as reader, \\\n", 191 | " \t open(write_file, \"w\", encoding=\"utf8\") as writer :\n", 192 | " \tfor line in reader:\n", 193 | " \t\titems = line.strip().split(\"||\")\n", 194 | " \t\tcontext = items[0]\n", 195 | " \t\tcompletion = items[1].strip(\"\\n\")\n", 196 | " \t\tx = {}\n", 197 | " \t\tx[\"context\"] = context\n", 198 | " \t\tx[\"completion\"] = completion\n", 199 | " \t\twriter.write(json.dumps(x)+\"\\n\")\n", 200 | "\n", 201 | "format_convert(\"train.txt\", \"train_formatted.jsonl\")\n", 202 | "format_convert(\"test.txt\", \"test_formatted.jsonl\")" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "id": "3ceec952-fe03-475f-9f3e-22237cc9c44b", 208 | "metadata": {}, 209 | "source": [ 210 | "Show the converted data (first 5 rows)." 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": 7, 216 | "id": "cb32aca7-bd0e-4847-a4c2-cc7e67dc2b7a", 217 | "metadata": {}, 218 | "outputs": [ 219 | { 220 | "name": "stdout", 221 | "output_type": "stream", 222 | "text": [ 223 | "{\"context\": \"name : The Vaults | Type : pub | price : more than \\u00a3 30 | customer rating : 5 out of 5 | near : Caf\\u00e9 Adriatic\", \"completion\": \"The Vaults pub near Caf\\u00e9 Adriatic has a 5 star rating . Prices start at \\u00a3 30 .\"}\n", 224 | "\n", 225 | "{\"context\": \"name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Caf\\u00e9 Brazil\", \"completion\": \"Close to Caf\\u00e9 Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of \\u00a3 10.50 . Delicious Pub food .\"}\n", 226 | "\n", 227 | "{\"context\": \"name : The Eagle | Type : coffee shop | food : Japanese | price : less than \\u00a3 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King\", \"completion\": \"The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than \\u00a3 20 for Japanese food .\"}\n", 228 | "\n", 229 | "{\"context\": \"name : The Mill | Type : coffee shop | food : French | price : \\u00a3 20 - 25 | area : riverside | near : The Sorrento\", \"completion\": \"Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at \\u00a3 20- \\u00a3 25 it is in the riverside area .\"}\n", 230 | "\n", 231 | "{\"context\": \"name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat\", \"completion\": \"For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat .\"}\n", 232 | "\n" 233 | ] 234 | } 235 | ], 236 | "source": [ 237 | "with open(\"train_formatted.jsonl\", \"r\") as reader:\n", 238 | " for _ in range(5):\n", 239 | " print(next(reader))" 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "id": "6631f786-be4b-40cf-89d9-7009c1888821", 245 | "metadata": {}, 246 | "source": [ 247 | "Load tokenizer from Hugging Face." 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": 8, 253 | "id": "e5433dc0-b5a5-4c01-adb5-3ffa2279eca8", 254 | "metadata": {}, 255 | "outputs": [], 256 | "source": [ 257 | "from transformers import AutoTokenizer\n", 258 | "import os\n", 259 | "\n", 260 | "tokenizer = AutoTokenizer.from_pretrained(\n", 261 | " model_name,\n", 262 | " fast_tokenizer=True)\n", 263 | "tokenizer.pad_token = tokenizer.eos_token\n", 264 | "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "id": "50817c47-a97b-4f80-975b-836859a0a7cf", 270 | "metadata": {}, 271 | "source": [ 272 | "Set block size, which is used to separate long text for model consumption." 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": 9, 278 | "id": "5f250929-5703-4b17-9f7b-26340950c055", 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "block_size = 512" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "id": "2332617b-1e66-4812-ad47-5eaeb52b101b", 288 | "metadata": {}, 289 | "source": [ 290 | "Create function to convert data. (Later this function is then used in data loader.)
\n", 291 | "In this function,\n", 292 | "\n", 293 | "1. Tokenize both contexts and compeletions. : e.g, ```\"This is a pen.\"``` --> ```[1212, 318, 257, 3112, 13]```\n", 294 | "2. Concatenate context's token and completion's token. (But it's delimited by \"\\n\" between context and completion.) This is used for inputs for LLM.\n", 295 | "3. Create labels (targets) with inputs. Label is ```input[1:]``` (i.e, shifted right by one element), and is filled by ```-100``` in context's positions. (See below note.)\n", 296 | "4. Pad tokens to make the length of token become ```block_size```.\n", 297 | "\n", 298 | "> Note : Here I set ```-100``` as an ignored index for loss computation, because PyTorch cross-entropy function (```torch.nn.functional.cross_entropy()```) has a property ```ignore_index``` which default value is ```-100```." 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": 10, 304 | "id": "9f2f38aa-b3d0-4614-aa59-8ddd977176d1", 305 | "metadata": {}, 306 | "outputs": [], 307 | "source": [ 308 | "from torch.utils.data import DataLoader\n", 309 | "import pandas as pd\n", 310 | "\n", 311 | "def fill_ignore_label(l, c):\n", 312 | " l[:len(c) - 1] = [-100] * (len(c) - 1)\n", 313 | " return l\n", 314 | "\n", 315 | "def pad_tokens(tokens, max_seq_length, padding_token):\n", 316 | " res_tokens = tokens[:max_seq_length]\n", 317 | " token_len = len(res_tokens)\n", 318 | " res_tokens = res_tokens + \\\n", 319 | " [padding_token for _ in range(max_seq_length - token_len)]\n", 320 | " return res_tokens\n", 321 | "\n", 322 | "def collate_batch(batch):\n", 323 | " # tokenize both context and completion respectively\n", 324 | " # (context and completion is delimited by \"\\n\")\n", 325 | " context_list = list(zip(*batch))[0]\n", 326 | " context_list = [c + \"\\n\" for c in context_list]\n", 327 | " completion_list = list(zip(*batch))[1]\n", 328 | " context_result = tokenizer(context_list)\n", 329 | " context_tokens = context_result[\"input_ids\"]\n", 330 | " context_masks = context_result[\"attention_mask\"]\n", 331 | " completion_result = tokenizer(completion_list)\n", 332 | " completion_tokens = completion_result[\"input_ids\"]\n", 333 | " completion_masks = completion_result[\"attention_mask\"]\n", 334 | " # OPT tokenizer adds the start token in sequence,\n", 335 | " # and we then remove it in completion\n", 336 | " completion_tokens = [t[1:] for t in completion_tokens]\n", 337 | " completion_masks = [t[1:] for t in completion_masks]\n", 338 | " # concatenate token\n", 339 | " inputs = [i + j for i, j in zip(context_tokens, completion_tokens)]\n", 340 | " masks = [i + j for i, j in zip(context_masks, completion_masks)]\n", 341 | " # create label\n", 342 | " eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n", 343 | " labels = [t[1:] + [eos_id] for t in inputs]\n", 344 | " labels = list(map(fill_ignore_label, labels, context_tokens))\n", 345 | " # truncate and pad tokens\n", 346 | " inputs = [pad_tokens(t, block_size, 0) for t in inputs] # OPT and GPT-2 doesn't use pad token (instead attn mask is used)\n", 347 | " masks = [pad_tokens(t, block_size, 0) for t in masks]\n", 348 | " labels = [pad_tokens(t, block_size, -100) for t in labels]\n", 349 | " # convert to tensor\n", 350 | " inputs = torch.tensor(inputs, dtype=torch.int64).to(device)\n", 351 | " masks = torch.tensor(masks, dtype=torch.int64).to(device)\n", 352 | " labels = torch.tensor(labels, dtype=torch.int64).to(device)\n", 353 | " return inputs, labels, masks" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "id": "2084d2e9-ef64-47a2-aec9-d24ead1cb38a", 359 | "metadata": {}, 360 | "source": [ 361 | "Now create PyTorch dataloader with previous function (collator function).\n", 362 | "\n", 363 | "> Note : In this example, data is small and we then load all JSON data in memory.
\n", 364 | "> When it's large, load data progressively by implementing custom PyTorch dataset. (See [here](https://github.com/tsmatz/decision-transformer) for example.)" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": 11, 370 | "id": "f3bce3bb-2215-4bd6-a6a6-5b6b9d5afdc0", 371 | "metadata": {}, 372 | "outputs": [], 373 | "source": [ 374 | "batch_size = 8\n", 375 | "gradient_accumulation_steps = 16\n", 376 | "\n", 377 | "data = pd.read_json(\"train_formatted.jsonl\", lines=True)\n", 378 | "dataloader = DataLoader(\n", 379 | " list(zip(data[\"context\"], data[\"completion\"])),\n", 380 | " batch_size=batch_size,\n", 381 | " shuffle=True,\n", 382 | " collate_fn=collate_batch\n", 383 | ")" 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "id": "3ba64144-b698-457e-b827-941020456536", 389 | "metadata": {}, 390 | "source": [ 391 | "## Load model" 392 | ] 393 | }, 394 | { 395 | "cell_type": "markdown", 396 | "id": "1bfd360d-7bdc-4fd7-9b12-bcf9fe0a8db2", 397 | "metadata": {}, 398 | "source": [ 399 | "Load model from Hugging Face." 400 | ] 401 | }, 402 | { 403 | "cell_type": "code", 404 | "execution_count": 12, 405 | "id": "271181bd-677a-4da9-9e57-2874f5e47bd0", 406 | "metadata": {}, 407 | "outputs": [], 408 | "source": [ 409 | "from transformers import AutoModelForCausalLM, AutoConfig\n", 410 | "\n", 411 | "config = AutoConfig.from_pretrained(model_name)\n", 412 | "model = AutoModelForCausalLM.from_pretrained(\n", 413 | " model_name,\n", 414 | " config=config,\n", 415 | ").to(device)" 416 | ] 417 | }, 418 | { 419 | "cell_type": "markdown", 420 | "id": "27ab764a-d634-40f8-9edb-a01146845233", 421 | "metadata": {}, 422 | "source": [ 423 | "## Generate text (before fine-tuning)" 424 | ] 425 | }, 426 | { 427 | "cell_type": "markdown", 428 | "id": "559efeaf-4b38-4a0c-9be6-eb394221e374", 429 | "metadata": {}, 430 | "source": [ 431 | "Now run prediction with downloaded model (which is not still fine-tuned).\n", 432 | "\n", 433 | "First we create a function to generate text." 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": 13, 439 | "id": "51a0c4fc-e0a7-4bbf-b25a-c335fe61f3df", 440 | "metadata": {}, 441 | "outputs": [], 442 | "source": [ 443 | "def generate_text(model, input, mask, eos_id, pred_sequence_length):\n", 444 | " predicted_last_id = -1\n", 445 | " start_token_len = torch.sum(mask).cpu().numpy()\n", 446 | " token_len = start_token_len\n", 447 | " with torch.no_grad():\n", 448 | " while (predicted_last_id != eos_id) and \\\n", 449 | " (token_len - start_token_len < pred_sequence_length):\n", 450 | " output = model(\n", 451 | " input_ids=input,\n", 452 | " attention_mask=mask,\n", 453 | " )\n", 454 | " predicted_ids = torch.argmax(output.logits, axis=-1).cpu().numpy()\n", 455 | " predicted_last_id = predicted_ids[0][token_len - 1]\n", 456 | " input[0][token_len] = predicted_last_id\n", 457 | " mask[0][token_len] = 1\n", 458 | " token_len = torch.sum(mask).cpu().numpy()\n", 459 | " return input, token_len" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "id": "3936b1a1-ae9f-48a5-80db-691261dda704", 465 | "metadata": {}, 466 | "source": [ 467 | "Let's test our function and generate text. (Here we stop the text generation when it reaches 15 tokens in prediction.)" 468 | ] 469 | }, 470 | { 471 | "cell_type": "code", 472 | "execution_count": 14, 473 | "id": "28b7e13f-e8fb-4a9f-90ed-0464463ef569", 474 | "metadata": {}, 475 | "outputs": [ 476 | { 477 | "name": "stdout", 478 | "output_type": "stream", 479 | "text": [ 480 | "Once upon a time, I was a student at the University of California, Berkeley. I was a\n", 481 | "My name is Clara and I am a student at the University of California, Berkeley. I am a member of\n" 482 | ] 483 | } 484 | ], 485 | "source": [ 486 | "eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n", 487 | "\n", 488 | "result = tokenizer(\"Once upon a time,\")\n", 489 | "input = result[\"input_ids\"]\n", 490 | "mask = result[\"attention_mask\"]\n", 491 | "input = pad_tokens(input, block_size, 0)\n", 492 | "mask = pad_tokens(mask, block_size, 0)\n", 493 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n", 494 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n", 495 | "\n", 496 | "result_token, result_len = generate_text(\n", 497 | " model,\n", 498 | " input,\n", 499 | " mask,\n", 500 | " eos_id,\n", 501 | " pred_sequence_length=15)\n", 502 | "print(tokenizer.decode(result_token[0][:result_len]))\n", 503 | "\n", 504 | "result = tokenizer(\"My name is Clara and I am\")\n", 505 | "input = result[\"input_ids\"]\n", 506 | "mask = result[\"attention_mask\"]\n", 507 | "input = pad_tokens(input, block_size, 0)\n", 508 | "mask = pad_tokens(mask, block_size, 0)\n", 509 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n", 510 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n", 511 | "\n", 512 | "result_token, result_len = generate_text(\n", 513 | " model,\n", 514 | " input,\n", 515 | " mask,\n", 516 | " eos_id,\n", 517 | " pred_sequence_length=15)\n", 518 | "print(tokenizer.decode(result_token[0][:result_len]))" 519 | ] 520 | }, 521 | { 522 | "cell_type": "markdown", 523 | "id": "d48fb60b-c05d-4884-a9bc-92152c94c894", 524 | "metadata": {}, 525 | "source": [ 526 | "Now we generate text with our test dataset (5 rows).
\n", 527 | "As you can see below, it cannot output the completion well, because it's not still fine-tuned." 528 | ] 529 | }, 530 | { 531 | "cell_type": "code", 532 | "execution_count": 15, 533 | "id": "495728ef-fbe6-4953-a354-4b7a8bb88798", 534 | "metadata": {}, 535 | "outputs": [ 536 | { 537 | "name": "stdout", 538 | "output_type": "stream", 539 | "text": [ 540 | "********** input **********\n", 541 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n", 542 | "\n", 543 | "********** result **********\n", 544 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n", 545 | "\n", 546 | "The Wrestlers is a restaurant in the heart of the city of Raja, India. It is located in the heart of the city of Raj\n", 547 | "********** input **********\n", 548 | "name : The Cricketers | Type : coffee shop | customer rating : 1 out of 5 | family friendly : yes | near : Avalon\n", 549 | "\n", 550 | "********** result **********\n", 551 | "name : The Cricketers | Type : coffee shop | customer rating : 1 out of 5 | family friendly : yes | near : Avalon\n", 552 | "\n", 553 | "The Cricketers is a coffee shop in Avalon, New York. It is located at the corner of Main Street and Main Street. The coffee\n", 554 | "********** input **********\n", 555 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n", 556 | "\n", 557 | "********** result **********\n", 558 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n", 559 | "\n", 560 | "The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre\n", 561 | "********** input **********\n", 562 | "name : The Punter | Type : restaurant | food : English | price : high | area : riverside | family friendly : no | near : Raja Indian Cuisine\n", 563 | "\n", 564 | "********** result **********\n", 565 | "name : The Punter | Type : restaurant | food : English | price : high | area : riverside | family friendly : no | near : Raja Indian Cuisine\n", 566 | "\n", 567 | "The Punter is a restaurant in Raja, India. It is located in the heart of the Raja district of Rajasthan. It\n", 568 | "********** input **********\n", 569 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : yes | near : All Bar One\n", 570 | "\n", 571 | "********** result **********\n", 572 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : yes | near : All Bar One\n", 573 | "\n", 574 | "The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly\n" 575 | ] 576 | } 577 | ], 578 | "source": [ 579 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n", 580 | "test_data = test_data[::2] # because it's duplicated\n", 581 | "test_loader = DataLoader(\n", 582 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n", 583 | " batch_size=1,\n", 584 | " shuffle=True,\n", 585 | " collate_fn=collate_batch\n", 586 | ")\n", 587 | "\n", 588 | "for i, (input, _, mask) in enumerate(test_loader):\n", 589 | " if i == 5:\n", 590 | " break\n", 591 | " print(\"********** input **********\")\n", 592 | " input_len = torch.sum(mask).cpu().numpy()\n", 593 | " print(tokenizer.decode(input[0][:input_len]))\n", 594 | " result_token, result_len = generate_text(\n", 595 | " model,\n", 596 | " input,\n", 597 | " mask,\n", 598 | " eos_id,\n", 599 | " pred_sequence_length=30)\n", 600 | " print(\"********** result **********\")\n", 601 | " print(tokenizer.decode(result_token[0][:result_len]))" 602 | ] 603 | }, 604 | { 605 | "cell_type": "markdown", 606 | "id": "e3138341-e01c-4fae-af78-c61e34967e92", 607 | "metadata": {}, 608 | "source": [ 609 | "## LoRA (Low-Rank Adaptation)\n", 610 | "\n", 611 | "Now we apply LoRA in our downloaded model.\n", 612 | "\n", 613 | "[LoRA (Low-Rank Adaptation)](https://arxiv.org/abs/2106.09685) (which is developed by Microsoft Research) is a popular adaptation method for efficient fine-tuning.\n", 614 | "\n", 615 | "In a task-specific fine-tuning, the change in weights during model adaptation has a low intrinsic rank.
\n", 616 | "With this hypothesis, we can assume that model's updates ($ \\Delta W $) will be re-written with much smaller low-rank matrices $ B \\cdot A $ as follows.\n", 617 | "\n", 618 | "$$ \\displaystyle W_0 x + \\Delta W x = W_0 x + B \\cdot A x $$\n", 619 | "\n", 620 | "where\n", 621 | "\n", 622 | "- $ W_0 \\in \\mathbb{R}^{d \\times k} $ is a pre-trained weight's matrix (which is frozen).\n", 623 | "- $ \\Delta W $ is updates.\n", 624 | "- $ B \\in \\mathbb{R}^{d \\times r}, A \\in \\mathbb{R}^{r \\times k} $ and $ \\verb| rank |\\ r \\ll min(d, k) $\n", 625 | "\n", 626 | "![LoRA](./images/lora.png)\n", 627 | "\n", 628 | "*From : [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)*\n", 629 | "\n", 630 | "In this assumption, we freeze all weights except for $ B $ and $ A $, and train only these low-ranked matrices $ B $ and $ A $.
\n", 631 | "With this manner, you can fine-tune large transformers for a specific task without full-parameter's fine-tuning.\n", 632 | "\n", 633 | "This will significantly save the required capacity (GPU memories) for training, and the number of required GPUs can approximately be reduced to one-fourth in the benchmark with GPT-3.\n", 634 | "\n", 635 | "For the purpose of your learning, here I manually (from scratch) convert the current model into the model with LoRA.\n", 636 | "\n", 637 | "> Note : You can use ```PEFT``` package to be able to get LoRA model with a few lines of code. (Here I don't use this package.)" 638 | ] 639 | }, 640 | { 641 | "cell_type": "markdown", 642 | "id": "5265832d-a736-4d68-80d3-347833d2c590", 643 | "metadata": {}, 644 | "source": [ 645 | "Before changing our model, first we check the structure of our model.
\n", 646 | "As you can see below (see the result in the cell), the following 6 linear layers are used in a single transformer layer on OPT.\n", 647 | "\n", 648 | "- Linear layer to get key\n", 649 | "- Linear layer to get value\n", 650 | "- Linear layer to get query\n", 651 | "- Linear layer for the output of attention\n", 652 | "- 2 linear layers (feed-forward layer) for the output of a single layer of transformer\n", 653 | "\n", 654 | "In this example, we'll convert all these layers into LoRA layers.
\n", 655 | "The transformer in OPT-125M has 12 layers and it then has total 6 x 12 = 72 linear layers to be converted." 656 | ] 657 | }, 658 | { 659 | "cell_type": "code", 660 | "execution_count": 16, 661 | "id": "5acb8f62-791a-4fa4-b00c-2666cf34827f", 662 | "metadata": {}, 663 | "outputs": [ 664 | { 665 | "data": { 666 | "text/plain": [ 667 | "OPTForCausalLM(\n", 668 | " (model): OPTModel(\n", 669 | " (decoder): OPTDecoder(\n", 670 | " (embed_tokens): Embedding(50272, 768, padding_idx=1)\n", 671 | " (embed_positions): OPTLearnedPositionalEmbedding(2050, 768)\n", 672 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 673 | " (layers): ModuleList(\n", 674 | " (0-11): 12 x OPTDecoderLayer(\n", 675 | " (self_attn): OPTAttention(\n", 676 | " (k_proj): Linear(in_features=768, out_features=768, bias=True)\n", 677 | " (v_proj): Linear(in_features=768, out_features=768, bias=True)\n", 678 | " (q_proj): Linear(in_features=768, out_features=768, bias=True)\n", 679 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 680 | " )\n", 681 | " (activation_fn): ReLU()\n", 682 | " (self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 683 | " (fc1): Linear(in_features=768, out_features=3072, bias=True)\n", 684 | " (fc2): Linear(in_features=3072, out_features=768, bias=True)\n", 685 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 686 | " )\n", 687 | " )\n", 688 | " )\n", 689 | " )\n", 690 | " (lm_head): Linear(in_features=768, out_features=50272, bias=False)\n", 691 | ")" 692 | ] 693 | }, 694 | "execution_count": 16, 695 | "metadata": {}, 696 | "output_type": "execute_result" 697 | } 698 | ], 699 | "source": [ 700 | "model" 701 | ] 702 | }, 703 | { 704 | "cell_type": "markdown", 705 | "id": "045e7239-cb8a-46dd-815d-e48e7e49eea4", 706 | "metadata": {}, 707 | "source": [ 708 | "First we build custom linear layer with LoRA as follows." 709 | ] 710 | }, 711 | { 712 | "cell_type": "code", 713 | "execution_count": 17, 714 | "id": "77889272-9a93-491b-93cb-b0bed5ce7cd8", 715 | "metadata": {}, 716 | "outputs": [], 717 | "source": [ 718 | "import math\n", 719 | "from torch import nn\n", 720 | "\n", 721 | "class LoRA_Linear(nn.Module):\n", 722 | " def __init__(self, weight, bias, lora_dim):\n", 723 | " super(LoRA_Linear, self).__init__()\n", 724 | "\n", 725 | " row, column = weight.shape\n", 726 | "\n", 727 | " # restore Linear\n", 728 | " if bias is None:\n", 729 | " self.linear = nn.Linear(column, row, bias=False)\n", 730 | " self.linear.load_state_dict({\"weight\": weight})\n", 731 | " else:\n", 732 | " self.linear = nn.Linear(column, row)\n", 733 | " self.linear.load_state_dict({\"weight\": weight, \"bias\": bias})\n", 734 | "\n", 735 | " # create LoRA weights (with initialization)\n", 736 | " self.lora_right = nn.Parameter(torch.zeros(column, lora_dim))\n", 737 | " nn.init.kaiming_uniform_(self.lora_right, a=math.sqrt(5))\n", 738 | " self.lora_left = nn.Parameter(torch.zeros(lora_dim, row))\n", 739 | "\n", 740 | " def forward(self, input):\n", 741 | " x = self.linear(input)\n", 742 | " y = input @ self.lora_right @ self.lora_left\n", 743 | " return x + y" 744 | ] 745 | }, 746 | { 747 | "cell_type": "markdown", 748 | "id": "954e2c9d-545e-4bd9-9b0f-eba3fe29a1de", 749 | "metadata": {}, 750 | "source": [ 751 | "Replace targeting linear layers with LoRA layers." 752 | ] 753 | }, 754 | { 755 | "cell_type": "code", 756 | "execution_count": 18, 757 | "id": "baf8a748-a3e3-45b8-9c64-252c56abe923", 758 | "metadata": {}, 759 | "outputs": [], 760 | "source": [ 761 | "lora_dim = 128\n", 762 | "\n", 763 | "# get target module name\n", 764 | "target_names = []\n", 765 | "for name, module in model.named_modules():\n", 766 | " if isinstance(module, nn.Linear) and \"decoder.layers.\" in name:\n", 767 | " target_names.append(name)\n", 768 | "\n", 769 | "# replace each module with LoRA\n", 770 | "for name in target_names:\n", 771 | " name_struct = name.split(\".\")\n", 772 | " # get target module\n", 773 | " module_list = [model]\n", 774 | " for struct in name_struct:\n", 775 | " module_list.append(getattr(module_list[-1], struct))\n", 776 | " # build LoRA\n", 777 | " lora = LoRA_Linear(\n", 778 | " weight = module_list[-1].weight,\n", 779 | " bias = module_list[-1].bias,\n", 780 | " lora_dim = lora_dim,\n", 781 | " ).to(device)\n", 782 | " # replace\n", 783 | " module_list[-2].__setattr__(name_struct[-1], lora)" 784 | ] 785 | }, 786 | { 787 | "cell_type": "markdown", 788 | "id": "8aae2df9-fae7-4ecc-8260-80e8e578d951", 789 | "metadata": {}, 790 | "source": [ 791 | "See how model is changed." 792 | ] 793 | }, 794 | { 795 | "cell_type": "code", 796 | "execution_count": 19, 797 | "id": "bf16b414-b973-40eb-be81-fd2aa3dde439", 798 | "metadata": {}, 799 | "outputs": [ 800 | { 801 | "data": { 802 | "text/plain": [ 803 | "OPTForCausalLM(\n", 804 | " (model): OPTModel(\n", 805 | " (decoder): OPTDecoder(\n", 806 | " (embed_tokens): Embedding(50272, 768, padding_idx=1)\n", 807 | " (embed_positions): OPTLearnedPositionalEmbedding(2050, 768)\n", 808 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 809 | " (layers): ModuleList(\n", 810 | " (0-11): 12 x OPTDecoderLayer(\n", 811 | " (self_attn): OPTAttention(\n", 812 | " (k_proj): LoRA_Linear(\n", 813 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n", 814 | " )\n", 815 | " (v_proj): LoRA_Linear(\n", 816 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n", 817 | " )\n", 818 | " (q_proj): LoRA_Linear(\n", 819 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n", 820 | " )\n", 821 | " (out_proj): LoRA_Linear(\n", 822 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n", 823 | " )\n", 824 | " )\n", 825 | " (activation_fn): ReLU()\n", 826 | " (self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 827 | " (fc1): LoRA_Linear(\n", 828 | " (linear): Linear(in_features=768, out_features=3072, bias=True)\n", 829 | " )\n", 830 | " (fc2): LoRA_Linear(\n", 831 | " (linear): Linear(in_features=3072, out_features=768, bias=True)\n", 832 | " )\n", 833 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 834 | " )\n", 835 | " )\n", 836 | " )\n", 837 | " )\n", 838 | " (lm_head): Linear(in_features=768, out_features=50272, bias=False)\n", 839 | ")" 840 | ] 841 | }, 842 | "execution_count": 19, 843 | "metadata": {}, 844 | "output_type": "execute_result" 845 | } 846 | ], 847 | "source": [ 848 | "model" 849 | ] 850 | }, 851 | { 852 | "cell_type": "markdown", 853 | "id": "e9099c08-f6a6-45f8-939b-cc3ed9415976", 854 | "metadata": {}, 855 | "source": [ 856 | "Finally, freeze all parameters except for LoRA parameters." 857 | ] 858 | }, 859 | { 860 | "cell_type": "code", 861 | "execution_count": 20, 862 | "id": "81d06bba-955b-4806-8ff7-f217252e3268", 863 | "metadata": {}, 864 | "outputs": [], 865 | "source": [ 866 | "for name, param in model.named_parameters():\n", 867 | " if \"lora_right\" in name or \"lora_left\" in name:\n", 868 | " param.requires_grad = True\n", 869 | " else:\n", 870 | " param.requires_grad = False" 871 | ] 872 | }, 873 | { 874 | "cell_type": "code", 875 | "execution_count": null, 876 | "id": "6c0a4469-2827-4f30-9324-711a9feea1ae", 877 | "metadata": {}, 878 | "outputs": [], 879 | "source": [ 880 | "### Do this when you run adapter fine-tuning on Hugging Face framework\n", 881 | "# model.gradient_checkpointing_enable()\n", 882 | "# model.enable_input_require_grads()" 883 | ] 884 | }, 885 | { 886 | "cell_type": "markdown", 887 | "id": "6d6c7d6f-6c50-4839-88a5-c851caab9ba2", 888 | "metadata": {}, 889 | "source": [ 890 | "## Fine-tune" 891 | ] 892 | }, 893 | { 894 | "cell_type": "markdown", 895 | "id": "a12b875f-36cc-40b8-aaab-1efda68710f3", 896 | "metadata": {}, 897 | "source": [ 898 | "Now let's start to run fine-tuning.\n", 899 | "\n", 900 | "First we build optimizer as follows." 901 | ] 902 | }, 903 | { 904 | "cell_type": "code", 905 | "execution_count": 21, 906 | "id": "bb51298a-2d55-466c-a990-0ea08a247350", 907 | "metadata": {}, 908 | "outputs": [], 909 | "source": [ 910 | "optimizer = torch.optim.AdamW(\n", 911 | " params=model.parameters(),\n", 912 | " lr=1e-3,\n", 913 | " betas=(0.9, 0.95),\n", 914 | ")" 915 | ] 916 | }, 917 | { 918 | "cell_type": "markdown", 919 | "id": "d37db1a8-0053-4acc-94ce-89d87c78942e", 920 | "metadata": {}, 921 | "source": [ 922 | "In this example, we build cosine scheduler for training." 923 | ] 924 | }, 925 | { 926 | "cell_type": "code", 927 | "execution_count": 22, 928 | "id": "6f95bdf6-4498-4d40-90aa-1267d55f38c3", 929 | "metadata": {}, 930 | "outputs": [], 931 | "source": [ 932 | "from torch.optim.lr_scheduler import LambdaLR\n", 933 | "\n", 934 | "num_epochs = 2\n", 935 | "\n", 936 | "num_update_steps = math.ceil(len(dataloader) / batch_size / gradient_accumulation_steps)\n", 937 | "def _get_cosine_schedule(\n", 938 | " current_step: int,\n", 939 | " num_warmup_steps: int = 0,\n", 940 | " num_training_steps: int = num_epochs * num_update_steps\n", 941 | "):\n", 942 | " if current_step < num_warmup_steps:\n", 943 | " return float(current_step) / float(max(1, num_warmup_steps))\n", 944 | " progress = float(current_step - num_warmup_steps) / float(max(1, num_training_steps - num_warmup_steps))\n", 945 | " return max(0.0, 0.5 * (1.0 + math.cos(math.pi * progress)))\n", 946 | "scheduler = LambdaLR(optimizer, lr_lambda=_get_cosine_schedule)" 947 | ] 948 | }, 949 | { 950 | "cell_type": "markdown", 951 | "id": "a9f9e828-c4fb-493d-a6de-78e03dbf035e", 952 | "metadata": {}, 953 | "source": [ 954 | "Run fine-tuning." 955 | ] 956 | }, 957 | { 958 | "cell_type": "code", 959 | "execution_count": 23, 960 | "id": "75d22125-830a-4ec6-8417-cdb8a97ec559", 961 | "metadata": {}, 962 | "outputs": [ 963 | { 964 | "name": "stdout", 965 | "output_type": "stream", 966 | "text": [ 967 | "Epoch 1 42/42 - loss: 1.0724\n", 968 | "Epoch 2 42/42 - loss: 1.3185\n" 969 | ] 970 | } 971 | ], 972 | "source": [ 973 | "from torch.nn import functional as F\n", 974 | "\n", 975 | "if os.path.exists(\"loss.txt\"):\n", 976 | " os.remove(\"loss.txt\")\n", 977 | "\n", 978 | "for epoch in range(num_epochs):\n", 979 | " optimizer.zero_grad()\n", 980 | " model.train()\n", 981 | " for i, (inputs, labels, masks) in enumerate(dataloader):\n", 982 | " with torch.set_grad_enabled(True):\n", 983 | " outputs = model(\n", 984 | " input_ids=inputs,\n", 985 | " attention_mask=masks,\n", 986 | " )\n", 987 | " loss = F.cross_entropy(outputs.logits.transpose(1,2), labels)\n", 988 | " loss.backward()\n", 989 | " if ((i + 1) % gradient_accumulation_steps == 0) or \\\n", 990 | " (i + 1 == len(dataloader)):\n", 991 | " optimizer.step()\n", 992 | " optimizer.zero_grad()\n", 993 | " scheduler.step()\n", 994 | "\n", 995 | " print(f\"Epoch {epoch+1} {math.ceil((i + 1) / batch_size / gradient_accumulation_steps)}/{num_update_steps} - loss: {loss.item() :2.4f}\", end=\"\\r\")\n", 996 | "\n", 997 | " # record loss\n", 998 | " with open(\"loss.txt\", \"a\") as f:\n", 999 | " f.write(str(loss.item()))\n", 1000 | " f.write(\"\\n\")\n", 1001 | " print(\"\")\n", 1002 | "\n", 1003 | "# save model\n", 1004 | "torch.save(model.state_dict(), \"finetuned_opt.bin\")" 1005 | ] 1006 | }, 1007 | { 1008 | "cell_type": "markdown", 1009 | "id": "83993d92-d7ed-4a07-8985-cc59bd4e4fef", 1010 | "metadata": {}, 1011 | "source": [ 1012 | "> Note : Here we save LoRA-enabled model without any changes, but you can also merge the trained LoRA's parameters into the original linear layer's weights." 1013 | ] 1014 | }, 1015 | { 1016 | "cell_type": "markdown", 1017 | "id": "1bc086e5-e93f-4264-a8fa-6428f844ac3c", 1018 | "metadata": {}, 1019 | "source": [ 1020 | "Show loss transition in plot." 1021 | ] 1022 | }, 1023 | { 1024 | "cell_type": "code", 1025 | "execution_count": 25, 1026 | "id": "e37c5aee-38d4-4a2a-952c-4fd2bef41e2b", 1027 | "metadata": {}, 1028 | "outputs": [ 1029 | { 1030 | "data": { 1031 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFbUlEQVR4nO3dd1hT5+IH8G/CCCBTlCWguAdq3bNVK3VhWzu0w7Z22qG32qHV62itVez4eVut1Y5btb2Oaqt2OWrdWBVBUNxbEEVcDEFm3t8fQEzIBJNzgPP9PA/PQ07e5Lw5hJxv3nVUQggBIiIiIomo5a4AERERKQvDBxEREUmK4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkGD6IiIhIUgwfREREJClnuStQkVarxaVLl+Dl5QWVSiV3dYiIiMgGQgjk5OQgJCQEarXlto1qFz4uXbqEsLAwuatBREREVZCamorQ0FCLZapd+PDy8gJQWnlvb2+Za0NERES2yM7ORlhYmO48bkm1Cx/lXS3e3t4MH0RERDWMLUMmOOCUiIiIJMXwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQphg8iIiKSFMMHERERSYrhg4iIiCTF8EFERESSYvggIiIiSTF8EBERkaQUFz7WHLiIHSevyl0NIiIixap2V7V1pDNXb+HtVQcBAOfnRMtcGyIiImVSVMtHela+3FUgIiJSPEWFj2KtkLsKREREiqeo8LH/3A25q0BERKR4igofX247rfv9ak6BjDUhIiJSLkWFD327T1+TuwpERESKpNjwQURERPJQbPi4XVQidxWIiIgUSbHh40zGLbmrQEREpEiKDR8lgtNuiYiI5KDY8EFERETyUGz46NKortxVICIiUiTFhg+1Su4aEBERKZNiwwcRERHJQ7HhQ6Vi0wcREZEcFBU+vDTOut/rebrKWBMiIiLlUlT4ePnexnJXgYiISPEUFT5cnNnVQkREJDdFhQ8iIiKSn2LDBxc4JSIikkelw8fOnTvx4IMPIiQkBCqVCuvWrTO4XwiB6dOnIzg4GO7u7oiKisKpU6fsVV8iIiKq4SodPnJzc9G+fXssWLDA5P2ffPIJ5s2bh0WLFmHfvn2oU6cOBg4ciPz8/LuuLBEREdV8ztaLGBo8eDAGDx5s8j4hBD7//HNMnToVDz/8MADghx9+QGBgINatW4cnn3zy7mprR+x1ISIikoddx3ycO3cO6enpiIqK0m3z8fFBt27dsGfPHnvuioiIiGqoSrd8WJKeng4ACAwMNNgeGBiou6+igoICFBQU6G5nZ2fbs0pERERUzcg+2yUmJgY+Pj66n7CwMEn2y9kuRERE8rBr+AgKCgIAXLlyxWD7lStXdPdVNHnyZGRlZel+UlNT7VklIiIiqmbsGj4iIiIQFBSELVu26LZlZ2dj37596NGjh8nHaDQaeHt7G/w4Cls7iIiI5FfpMR+3bt3C6dOndbfPnTuHpKQk1K1bF+Hh4Rg/fjw++ugjNGvWDBEREZg2bRpCQkIwbNgwe9b7rgkmESIiIllUOnzEx8ejX79+uttvv/02AGDUqFFYsmQJJk6ciNzcXIwePRqZmZno3bs3Nm7cCDc3N/vVmoiIiGqsSoePvn37Wmw1UKlU+PDDD/Hhhx/eVcWIiIiodpJ9totc2OlCREQkD8WGDyIiIpIHwwcRERFJiuGDiIiIJKXY8MGZtkRERPJQbPggIiIieTB8EBERkaQUGz4EJ9sSERHJQrHhg4iIiOTB8EFERESSUm74YK8LERGRLJQbPoiIiEgWDB9EREQkKcWGD/a6EBERyUOx4YOIiIjkwfBBREREklJs+OC1XYiIiOShqPAhmDiIiIhkp6jwQURERPJj+CAiIiJJKTZ88MJyRERE8lBs+CAiIiJ5MHwQERGRpBQbPjjxhYiISB6KDR+5BcVyV4GIiEiRFBs+Xl92QO4qEBERKZJiwwcRERHJQ1Hhg+M8iIiI5Keo8EFERETyY/ggIiIiSTF8EBERkaQYPoiIiEhSDB9EREQkKYYPIiIikhTDBxEREUmK4YOIiIgkxfBBREREklJU+OACp0RERPJTVPggIiIi+TF8EBERkaQYPoiIiEhSDB9EREQkKYYPIiIikhTDBxEREUmK4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkFBU+BNdXJyIikp2iwgcRERHJj+GDiIiIJMXwQURERJJi+CAiIiJJKTZ8uLs4yV0FIiIiRVJs+CAiIiJ52D18lJSUYNq0aYiIiIC7uzuaNGmCmTNnQlSzea4C1as+RERESuFs7yf8+OOPsXDhQixduhRt2rRBfHw8XnjhBfj4+ODNN9+09+6IiIiohrF7+Pjnn3/w8MMPIzo6GgDQqFEjrFixAnFxcfbe1V2pZg0xREREimH3bpeePXtiy5YtOHnyJADg4MGDiI2NxeDBg02WLygoQHZ2tsGPo7CrhYiISH52b/mYNGkSsrOz0bJlSzg5OaGkpASzZs3CyJEjTZaPiYnBjBkz7F0NqxhDiIiI5GH3lo9Vq1Zh2bJlWL58OQ4cOIClS5fis88+w9KlS02Wnzx5MrKysnQ/qamp9q4SERERVSN2b/mYMGECJk2ahCeffBIA0LZtW1y4cAExMTEYNWqUUXmNRgONRmPvaljHpg8iIiJZ2L3lIy8vD2q14dM6OTlBq9Xae1dERERUA9m95ePBBx/ErFmzEB4ejjZt2iAxMRFz587Fiy++aO9d3RUOPiUiIpKH3cPH/PnzMW3aNLzxxhvIyMhASEgIXn31VUyfPt3euyIiIqIayO7hw8vLC59//jk+//xzez+1XXGdDyIiInnw2i5EREQkKcWGj2KtQImWzR9ERERSU1T4qNjV8mfyZXkqQkREpGCKCh8VZeYVyl0FIiIixVF0+NCy24WIiEhyig4f6dkFcleBiIhIcRQdPhbtOCN3FYiIiBRH0eGDiIiIpMfwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQpRYUPLilGREQkP0WFDyIiIpIfwwcRERFJiuGDiIiIJMXwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQphg8iIiKSlLLChzBc47R747oyVYSIiEi5lBU+iIiISHaKDh++7q5yV4GIiEhxFB0+NC6KfvlERESyUPTZV/Ayt0RERJJTdPi4cCMP41Ym4nBaltxVISIiUgxFh4+DqZn4NekShs6PlbsqREREiqHo8EFERETSY/ggIiIiSTF8EBERkaQUFT7KJ7eoVbJWg4iISNEUFT7KqVRMH0RERHJRZviQuwJEREQKpszwwfRBREQkG2WGD7Z9EBERyUaR4YPZg4iISD6KDB/MHkRERPJRZvhg+iAiIpKNMsMH2z6IiIhko8zwwexBREQkG0WFD1G2xGleYYm8FSEiIlIwRYUPIiIikh/DBxEREUmK4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkFBk+PFyd5K4CERGRYikyfIzoHCZ3FYiIiBRLkeGDiIiI5MPwQURERJJSVPgQECa3B3hpJK4JERGRcjkkfKSlpeGZZ56Bv78/3N3d0bZtW8THxztiV1VS8cJyHcP95KkIERGRAjnb+wlv3ryJXr16oV+/ftiwYQPq16+PU6dOwc+PJ3giIiJyQPj4+OOPERYWhsWLF+u2RURE2Hs3dmWuO4aIiIjsz+7dLr/99hs6d+6M4cOHIyAgAB06dMC3335rtnxBQQGys7MNfqR2u0gr+T6JiIiUyu7h4+zZs1i4cCGaNWuGTZs24fXXX8ebb76JpUuXmiwfExMDHx8f3U9YmPRrcOw8eVXyfRIRESmV3cOHVqtFx44dMXv2bHTo0AGjR4/GK6+8gkWLFpksP3nyZGRlZel+UlNT7V0lIiIiqkbsHj6Cg4PRunVrg22tWrVCSkqKyfIajQbe3t4GP0RERFR72T189OrVCydOnDDYdvLkSTRs2NDeuyIiIqIayO7h46233sLevXsxe/ZsnD59GsuXL8c333yDMWPG2HtXdlVcwkGnREREUrB7+OjSpQvWrl2LFStWIDIyEjNnzsTnn3+OkSNH2ntXlSYszKhdcyBNuooQEREpmN3X+QCAoUOHYujQoY54artQQWW0LfVmngw1ISIiUh5FXduFiIiI5MfwQURERJJi+CAiIiJJMXyUsTQYlYiIiOyH4YOIiIgkpdjw0SaEK6kSERHJQbHhY8Xo7nJXgYiISJEUGz683VwMbgtw0AcREZEUFBU+GC+IiIjkp6jwUU5lvMApTmfckr4iRERECqTI8GHKjdxCuatARESkCIoOH72b1tP9rjLVHEJERER2p+jwodVbWcyJ4YOIiEgSig4fRSVa3e9qRR8JIiIi6Sj6lHt/y0Dd7yqw5YOIiEgKig4fz/dspPvdU+MsX0WIiIgURNHhw93VSfe7q7OiDwUREZFkeMYt079VgNxVICIiUgTFh4+ujeoCAJw54pSIiEgSijrjClPrq5eNM+W1XYiIiKShqPBRTmXid5PBhIiIiOxOkeFDn0rX8kFERERSYPgoa/sQbPogIiKSBMMH1xYjIiKSFMNHebcLGz6IiIgkofjwoS5LH5ztQkREJA3Fh49ybPkgIiKShuLDR4m2NHXsPHlV5poQEREpg+LDxz9nrgMA1iVdkrkmREREyqCo8MFxHURERPJTVPgox+m1RERE8lFk+CAiIiL5MHzo4SqnREREjsfwoed2UYncVSAiIqr1GD70qDkYhIiIyOEYPvQwexARETkewwcRERFJiuGDiIiIJMXwoYeTXYiIiBxPWeGD4YKIiEh2ygofZVQcWUpERCQbRYYPIiIikg/DBxEREUmK4UMPB5wSERE5HsMHERERSYrhQ4/gdBgiIiKHY/ggIiIiSTF8EBERkaQYPvRwwCkREZHjMXwQERGRpBQVPtiwQUREJD9FhY9y5hZXZzghIiJyPEWGDyIiIpKP4sPHhnH36n4XHHFKRETkcIoPH43r15G7CkRERIri8PAxZ84cqFQqjB8/3tG7IiIiohrAoeFj//79+Prrr9GuXTtH7sZu2OlCRETkeA4LH7du3cLIkSPx7bffws/Pz1G7uWsqs3NfiIiIyBEcFj7GjBmD6OhoREVFWSxXUFCA7Oxsgx+5cLwpERGR4zk74klXrlyJAwcOYP/+/VbLxsTEYMaMGY6ohk1UbPggIiKSlN1bPlJTUzFu3DgsW7YMbm5uVstPnjwZWVlZup/U1FR7V0mHU2mJiIjkZ/eWj4SEBGRkZKBjx466bSUlJdi5cye+/PJLFBQUwMnJSXefRqOBRqOxdzUs4xKnREREsrF7+Ojfvz+Sk5MNtr3wwgto2bIl3nvvPYPgUR2w14WIiEhadg8fXl5eiIyMNNhWp04d+Pv7G22vbgSbPoiIiBxO8SucqjjilIiISFIOme1S0fbt26XYDREREdUAim/50MfJMERERI6n+PDBThciIiJpKT586EvPzpe7CkRERLUew4eewV/skrsKREREtZ6iwoepMR2c7EJERCQtRYUPbVn4cGLiICIiko3Cwkdp+lDrhQ+u80FERCQtRYWPkrKmD7WagYOIiEguigofpzJyAACWskduQbFEtSEiIlImxYSP3IJi7D17A8CdsR+m/LQ/VaIaERERKZNiwseN3ELd7z/tTzFbrsRSMiEiIqK7ppjwoT/O40p2gdlyGTlcaIyIiMiRFBM+bJ1ee+1WofVCREREVGWKCR+c4EJERFQ9KCZ82HoFOWYUIiIix1JM+Ei9cVvuKhAREREUFD44i4WIiKh6UEz4sHkVdfa7EBEROZRywofN5Zg+iIiIHEk54YOZgoiIqFpQTPiwte2DIYWIiMixFBM+GCqIiIiqB8WEDy+Ns+73IW2DZKwJERGRsikmfDQL9NL97qRWzMsmIiKqdhR5Fra01Dp7Z4iIiBxLUeHDzaX05fZtUd9ge++m9XS/r0tKw1s/JSG/qETSuhERESmFs/UitcfOCf2QnJaFfi0CzJYpKhFYm5gGD1cnPHxPA3SNqCthDYmIiGo/RbV8BHi7oX+rQKgr9LuYmgmzbF8KRny9BwdTM6WpHBERkUIoKnxURWLKTbmrQEREVKswfFih4gIhREREdsXwQURERJJi+LCCDR9ERET2xfBBREREkmL4ANAyyMt6ISIiIrILhg8AT3QJN3sfe12IiIjsi+EDgJOl9daJiIjIrhg+rOGIUyIiIrti+AAghJC7CkRERIrB8GEF2z2IiIjsi+EDANs9iIiIpMPwAaCep0buKhARESkGwwcAH3cXs/ddyrwtYU2IiIhqP4YPK77afkbuKhAREdUqDB82WLDtNN77+RBnxRAREdkBw4cNPt10Aj/FpyIxNVPuqlRKwoUb2HYiQ+5qEBERGXCWuwI1SWZeodxVqJTHFu4BAOyZfD+Cfdxlrg0REVEptnxUwotL4uWuQpVkZBfIXQUiIiIdhg87yC0oRlJqJseEEBER2YDho0wD36p3Szy28B8MW7Ab65LS7Fgj++HlaYiIqDph+Cjz3ajOVX7s8fQcAMCaA9UzfCiNEAJ7zlxHRk6+3FUhIiITOOC0TKtg70qV12oFpqxLrvTjyPF2nLyK5xfvh5NahTOzh8hdHSIiqoDho4piT1/DirhUuashmcNpWZi7+SQmDmqBlkHVO3DtPHkNAFCi5RgcIqLqiN0uVZSdX2S0TVVNB1eoKnlt3vyiEjz1zV58vePO6q6PL/oHW49n4Mlv9tq7ekREpDAMH1X0few5yfeZk1+Ew2lZDp9Vsyo+FXvOXkfMhuO6bflFWgBAZp5x6CIiIqoMho9KOnklB4XFWhxIyZR834M+34Wh82Ox/eRVh+4nt6BE97stQWdlXAo+3XTcYpm3VyXhme/2QcuuECIixbN7+IiJiUGXLl3g5eWFgIAADBs2DCdOnLD3bmQz8POdmPTLIZP3pWc59gq4aWVX2N2QfNlq2btpHRG489gjl7Ktlp+0JhkLtp3BQQvLz685kIbY09dwLN368xGRdfHnb+CoDf+fVP1l5xfhRNmsSaWwe/jYsWMHxowZg71792Lz5s0oKirCgAEDkJuba+9dyUIIYE2i6Sm1pzNumX3cqvhUfPTHUVzNMV5t9HLWbRSXaA22nUjPMTmuxHLdBIoqPA9Qus7HtuMZ2HQkvVLPBwAFxcbPZ44t9eU6bER379qtAjy+aA+GzNsld1XIDvp8sg0DP9+JhAs35K6KZOw+22Xjxo0Gt5csWYKAgAAkJCTgvvvus/fuqhVzPQoZ2fmY+HNpa8l3secw+r7GaB7ohcc7hWLv2et48pu96NqoLsbe3xT+nq7IL9LisYX/wNfDBUnTBxg9X8UBpFdzChB7+irWJ6djz5nr2P3e/fByu/OnLSzR4oUl+wEAidMegJuLE9xdncy+DsOAYPyitFoBtdp4EKstA1vvNnykZd5GiI9btR3cSySF9CyuYVOb3CwbS/f3sQx0alhX5tpIw+FTbbOysgAAdeuaPqAFBQUoKLjTGpCdXbObEYUQUKlU2H36GsL8PBDu74ErFa6t8s3OswCALo38sGxfCgAg7vwNPPd9HADg9b5NANg+uLPLrL8Nbrf/8C+E+LjpbheX3DnjD50fi7TM29g/JQr1vTRGz5VbUIxPN1nuJuswczMe7xSKaUNb21Q/W9zILcTXO89geKcwNA3wNFlmVXwqJv58CCM6h+L5nhFoEeQFp7IQdDA1E+/9cghToltxRVciomrOoQNOtVotxo8fj169eiEyMtJkmZiYGPj4+Oh+wsLCHFklh4uYvB5vr0rCyO/24b5Pt1kse+1WIfIKio22V+yCqYpLet+M9E/G5eNGNprpgvliy6kKW4zP5Fm3i/DfKs72ESZaUoQQeO+XQ/h6x1kM/mKnbvusP4/ika92o6C4dADs3L9OAgBWxV/EkHm7MO3Xw7qyz30fh+PpOXj2v3FVqldiyk2cvKKsPldT7PHeq+5qw2usbQFbCIH/bD6Jv49ekbsqJBGHho8xY8bg8OHDWLlypdkykydPRlZWlu4nNVW+hbuSpj9gl+exdZn1zUevYMvxDKPt3+66c2J/cH4ssiq0gJg6gVeamf6Ps1crjs2xfV9x52/oBrrmF5Vg+b4UXLYyCHd98mV0mfU3Npd96BSVCN1zfLvrHBJTMrHpyJWymhjWZXlZqxFQGojMOZByEzsqzBDSH5B77VYBHvnqHwz4z86KD1WUy1m3EfnBJkxeY3pAtaMJIRw+jXzPmetoPnUDluyWfqq8JZVdEK+2jJ0q0QoUFmux5VgGvthyCi//UHrlcK1W4O2fkqr8JYeqP4eFj7Fjx+KPP/7Atm3bEBoaaracRqOBt7e3wY9cfD1cJd3fIr1FvMxJTsvCN7sMy207cRWv/ZiAofNtG2xm6kvSuqRLaDTpT8z9y7CL5W4+/OdtOYWus7fg16Q0DF+0B/9em4yh82IrPL/hY95YdgDXbhUabFv6z3mD2wnnb2DNgYtV/sB99Kt/MOr7OF0QOnctF50++hsLtp0GAFzOrHz/+er4VDz3fRxyKjkouDpbvPs88ou0sqzcK4TAs/+Nw0Nf7rbLdOzCYi2W7buAlOt5BtvH/5QIrQA++P3oXe/DXk5n3EK7Dzbh879Pyl0VyUXN3YFOMzfjZMadVkchBLadyMCaxDTM/MP030kIgeSLWbXq/w+oPaHSFnYPH0IIjB07FmvXrsXWrVsRERFh713UKNdvFdz11W5z8ovR//+2625fzSnAxiPpOJyWjel6XQ/m5BWWGG1LuHATADBv62mD7Vqjd3/l2nev5hRg3MokJKeVjvW5nltostzY5QcwZvkBk/ct3HFG1z0EAEv3XMDbqw4iw8RMIVNyTXRlAXcG6c368xhu5BbqxrZUpQl7ws+HsPPkVZsCpL3kF5XgoS9jMWeD5TVVqspS8LxVUKybCphbUIy8QtPHuLKybhdhZVwKbuQWIvb0NSSnZWH0jwk4fpdTsr/ZeQZT1h622vVZHcSsP4bcwhJ8/nfFLk/zaku3y7lrucgpKMZOvZZJrSh9v5lTVKLFB78dwYNfxtb42T6H07LwRSX+7rWJ3QecjhkzBsuXL8evv/4KLy8vpKeXji3w8fGBu3vVL1tfUw1ftAdnr93dNOMf9lyo0n3lygeymvPT/hQ0DfBCp4Z+RjN2thy7gk4N/Uw+bvHucwj2qdzfVADIzCvEH4fMr1VyJbsAMeuP2fyc6yuse/JTvOlv7sbBytjMP45iwsAWcHMxPRtICGFwjLJvm/6QLNEK5BYWw9vNxeo+bfVb0iUcupiFQxezMGlwS7s976Yj6Vi04wwCTAxALvfA3B24nJWPH1/qqhtXc2b2EN2A36oQQuBfKxKx8+RVg67Kv49dwd/HruD8nGibn6uwWIvfD15Cr6b1EOTjhr1nHTdlMa+wGEcvZaNjuJ/JWV9kTAiBq7cKEODlZvJ+/ZlyyWlZ2GqiO7rcJxuPY2nZ517qDceureRoQ+fHWi9US9m95WPhwoXIyspC3759ERwcrPv56aef7L2rGuFug4cU3vslGY8t/Adx524YnaC/2m7+m/2M34/itf8lVGpfF2/m4dtdZ62W2336mk3P968ViXhjmWELiv5L2HXK8BsVAFzNMd/N8t/Yc/h2p/n6vbBkP3rO2aK7beob6LlruWg5bQPaffAXLt4sbfY/nJaF0T/E43RG1Qe1FjpooOSrPyYYjK0x5XJZq5H+OJtZfx6rcgvF6YwcdJ29RfeNN+783YWFL7edxjurD2LQF44ftzPyu314fNEe/LDnvMP3ZQ+3CoqxOj4VmXmmWyEtEULgg9+O4Me91r/kWBKz4Ti6ztqClXEpVssOW7AbvyZdMnv/Uhu+cCmBo8dHOZpDul1M/Tz//PP23hXZ2Yiv9+DUFeOF0gorsdCYKfp9+GOXJ2LBNutdFbZ2+/9+0PyHFACD2S/l9Th4McviYy7cyDN73/YTVw2mTucXGXdp9ftsO4rKpjdvSC5t+Xvoy1j8dfQKoubW7EGte89e1/3+/e5zGPR51Zq9p6w9bHLBvaraVvZN2dr0dP1v2FX98E4su7TCT/EXq/R4e580rD3fpF8OYcLPh/DS0vhKP/f+8zex5J/zmLbOeveuJeXLC5gbw3E3bhUUY/Huc1YHt1cH1j5Lbe1O23Y8A11nbzH4clXT8NouZCA927hVoPnUDXf1nCVV+LC1NHulqs5fz8Uz3+0z2l7xH/7ctVxklLWOZN0uMhkwyl2ycbBqxTBl65iJ/KIS3VTj6uCmnS4sWNnZHdZUnAmVa+b46pd74mvLV2jeffoaDl3MNL/PCu/r9Kx8zP3rhMUFwKasTUafT7ebHZf06abjFt9v5fRD1Ffbz+gGT5tS3sVZPs6rXHZ+Ef639wKu3zIfArOt/B/O3XwSD30Za/P7ObewBBsPpxu9xrsZwzLjtyOY8ftRPLLgn6o/SRX8cegSXlyy36a/FwBM//Uwmk/dYLH109aPyheW7MfVnIIqLS2QnpWPzzadkD2sMXyQw+0/Vz2WDH7vl2TEmujO+TnB8BtswoWb6DprC7LyitB+xl/oHlPazWKvay/8cegSWk/fZHWwalGJFu0++AudZv5dpRkgV3MKMH5lIuIqHP/YU9fwq5VB0PrrLdw2MWBZn/5JeOvxKxg6f5fVNVOKHXyBwUS9Cz+ev5aLTUfSIYQwaLWKO38De89eNxmELmfdxsjv9uGhL3fbvM/nF8dh3tbTeGnpfrNllu1LQcqNPINB6Pon3gXbzljs9iunH6I+3XQCn246gTHLDmDG70eMypo7sb/38yFMXXcYoxYbn8Bu5BYavecOpNw0KjdvyykcupiFVfttnyH12v8SMOP3owbvG1sHk5tSfqFNU1+cHGns8kRsPZ6BltM2Wi+MO+PzvtxqPiiaU6IVZkPOP2WfaSvjUvDg/FhkWDkOzy+Ow5fbTuOFxebfp1Jg+CCHe9pEa0N1sWT3OSzefd7kfX0/K50pkZlXhIs38zDwc+MukxNWTrLzt57CXxUWdHtn1UEAMJi1IoTQjQ8pl3IjD4UlWtwqKEZ2fpHBh8qiHWd0XQ2pN/JMfuBMXZeMdUmXMOLrPQbbn/nvPoxbmYSzV81fi+jlH+JxNacARSVaqyHp3k+24d3VB5FfVIIXl8TjcFo2XrcyFijJwkUIK2Pf2euY+PNBo+4W/YGwfT/bjld/TDB5Negnv9lrcgySrS1a+o6XhVNbLsZoyWkLf5dypmaw/Zl8uWy6tOF9TmbSx4bDpe/Lw2mG9T2QchMdZ27Gyz/EGwSXR78y37JQpLeKck5+EZJSMy12B62IS0F+0Z0uCEvXxTJix9z6+8FLJi/O93PCRWw7cWfQa8r1PLu1QFal+kPnx6LltI0mW4TLP18nrUlGcloWPt5ovEJ1QXEJLt4sfQ3l79PjMl/IzuHLqxNVZ5bWe9DvYnh84R6TZcrHLZRoBSb+fAgNfA1H82fnF2P0j+ZPxDtPXsXaxDS4OKmwKv4ipg1tjZd6l05P7/9/O3TloufFGkw/Lg8uB98fgHs/KQ1JFWeHXKiwxkVWXpHBt/KMnAI0rm96KXsAeOrbvTh/LddqK8XFm7fxc8JFgxakLBOzgPKLSuCsVsHZybbvPH8fvQInJxX6tQgwuk+rFXhx6X5sP2G6z9vVSY3bWsOTxV9mBtSu2p+K1/o0salOd6NiKCi/FENlFZdoMXyR6fejKWqVCpU55ZWH8a3HM/BM93CD+1Ku5yHc38PoMeUv42BqJh5ecKe1aHxUM7PHttV021oMth3PQGJqJkb1aAh/T+MZWVUdO7TtRAb+tSIRwJ3/ne92ncWF63m6Abbn50Rj39nreOKbvWgV7I0N4+6t1D7+OpIOV2c1+pp4D1fGsculAWnPmesYFBlkseztomKj91b0vNjKBTwJMHwQ2cBSk26JVmDu5hP45UDlByBWnAYds/4YXuodYdQVoB889Om3luh/4GTlFRl9s/lqx2nEXzBuOjfHnh9WSamZGFZ2UrqveX2bHlO+2uXJjwbD1dkwsCSk3DQbPADA2cQU2GtmxjZYOy1PXpOMWcMiKz2tduPhdBSWaNE62Buhfu5o98FfuvumrD2MKWsP23QyO5iaicW7z2HioJYI8XWv1HioEq1AkfZOC4OlwFNYrMWXW09ZHMS98chl9GsRgKYBngbPU97IoR88AODzv09Vav0SU8ovipmUmokfXuxq0wrPWXlFOHI5C90j/E3+3a7mFBh1O2TmFeKjP42n+Jf/X5cHAFvdzC3UffHYNP7ORVXvZrzx+eumZ0/qt05uOZaBiMnrMa5/M7z1QHMA9v1ftheGD6K71OTf6ytV3tJnT3krw4TVB216LrXeCaBHzFa8PaA5dp68anIdlcxc45PWb1ZmC1VVebV2nryKXaeuGlwyYKeJ7g9LirVanL50C2sTL8LFSY3ezeohJ9/KAEcT51dz42ZKtMLiSXlFXAoGtA5Ev5a2f3vNLyqxaRr6pDXJOFihC6rijIjyE/qlzHyseq2H1ee8eDMPTQO8AAAD/rPD4GTX77Pt6Bbhj48fb2fwmDbTNyLXRFdOxRPl7PXHMXv9cTzaoQGm6l1Yctb6Y3e9mKI1+/RmWplzOC0LkQ18MGTeLqRl3sZnw9vj8U6hEKK0ZTLAW4MJA1tintE1rAy7jszZdCQdX/x9Cp5uhqfOrNtFePXHeDzUvgFcnFToGmF4IVX9LltzIdgWczYcN9mSpN+FW1D2/vliyyld+KiOGD6IJGZtul1eYTHWJNr2Qf683mDB9Ox8TPzZ9HVZRn63F7tPG354r0++bNMidVVR3hRubYE7W+QXaQ1WsrS09kw5UzEi30yffcqNPDz97T4MbhuENiE+6NTQz+jic+WtDfv11iPRitJBgKYWpLP1BFMxeAClYzE++uMoJg9pZTB2xda1UKLm7sTSF7uiT/P6OFPhWk3nr+fh/PU8o/BhKngAMNuatyYxzeg9erdjXawpKNZi7uaTFkPC1VsFKC7R6loKNyRfxuOdQnE8PQery7oFJwxsaXLdEnML5pVPlwdK18Qx5attp7H37A2Dxe22v9vX7OswZ+k/57HnzDVk5BTg8U6heGdAC7NlazqGD6JqpvX0TTaX1Z+9YUnF4AHYtjru3bA26t5WHWdurlT5qLk7kG2iZcTUMSi35+x17LHwzVpAIK+w2GCsxckrt9By2kbsnxKF+hVWh+398d0t6/5d7DlcySnA/Kc6VOnxo76Ps7hCbMWLVZqzPjndeiEJmWqx0PfC4v1oF+qju52WeRubjqQj0Nv0yqrlCou1Jq8TczgtCzkWlnovV345icow1ZV6u6hEtw7R/K2na3X44GwXInKIrrO3WC/kAI7o3xYCmG1myf/1yZcRe8q2FXkrw9oCendjZw1enMqaQ3qLCB5Pz8GrPyYYzTir6IH/7ECfT7cbbbd1+XNbB1Hr6zVnq9Uywxf9Y3RxxMoQQtz1IpGOwvBRwdToVnJXgYiqGSGAbcdNn7D3nr2OZ/4r3XRyW1u7LJ10ymd5KIV+d1nzKcaLJlacGVZZpsYxWRpBYmur4P7zN/Huz4bjv8wNPjdl1OL9uOfDv6wXlAHDRwUv39sYse/1k7saRFSNvLP6oNkP/fL1Mhyh0aQ/DW4npWbafCVXcxdYVCL9wcSOukaSrRIu3ES2iS4ecyouEmhLi0m5nSevmlwTpjpg+DAh1M94HjsRkdzeWZVkc9m7vR5LbSLHtYeX/nPe7H3v/ZIsXUWqKYYPMwa0DpS7CkREBirOXiHbWJ2a7QBLLISPitfZUSKGDzNaBHnJXQUiIrKDo5VcIIwcj+HDjJd7N4aHq/H8fSIiIro7DB9m+Hi44K+37rNekIiIiCqF4cOCUD8PPNejoSQXnSIiIlIKrnBqxYcPRwKA1cuKExERkW3Y8kFERESSYviwAzMXwyQiIiITGD5s5F/H1ex9wzuFSlgTIiKimo3hw0ab3+6D5S93M3nfoMgg3X11LYQUIiIiYviwWd06rujZtB6e6R5usP3he0LQr0UAejath/NzorF/SpRMNSQiIqoZONulkmY+HIlX72sCjbMaCRdu4oHWgQYXLXJSq7BrYj8kpWbidmEJJv5ySMbaEhERVT9s+agklUqFsLoeCPB2w+C2wXB2Mj6EYXU98GD7EAT7ull8rqnRrRxVTSIiomqL4cOBXEwEE30v9orAy70joHG2/mfo3zLAXtUiIiKSFcOHA7XUuzjdjIfaGNznpFZBrVZh6tDWOPrhIDzaoQHubVYPy18xPahV42L9T/Xl0x2w+rUeNtVt8uCWNpWrLhr4uqNVsLfc1SAiIjvgmA8H8vVwxd7J/eHmooavhyue6BIGIYDlcSmIanWnJcNJrcLcJ+7R3f7l9Z54bOE/utuN69XB6PuaYH1yusn9vNgrAg+2D0aHcD+D7S0CvXDiSg4AYMUr3fHUt3t1973apwliNhzX3fZwdUJeYYnudlSrABy5lI3LWfmVes0fPNga3+46h7TM25V6nCm+Hi7IzCtCRL062PpOH6hUKszbcgpzN5+86+cmIiL5sOXDwYJ83ODrUTr91s3FCe6uTnipdwQa+tcx+5gAL43Bbb86rrgnzNdk2eUvd8P0B1sbBQ8A8HRzxvk50Tg/Jxo9mvhj5rBIeLg64ZfXjVtHtr7TF80DPXW3hQD+PcT8mJTmgZ4Iq+uO9mG++PeQO60oz/eKQJdGxnWpaPR9ja2W2TWxHz54sDVWju6uG9TrqtdF9dXIjkazj4iIyLpgH8tjEh2N4aMaCqvrYdAt0sDX3WS52Pf6oWfTemafRwhhcPvZ7g2R/MFAdGpY12D7U13DEeTjhr/e6qPbVs/TMAA90TkM04a21t0eH9Uc29/th3Vv9ISTuvJvo4qrwjby99D97uPugkXPdIKXmwue7xWBQO87/yTPdG+IyAbeePuB5hjSNhgfDWtr8vldbRhHQ7XHmH68+CNRZawc3V3W/fMTupp6tU8TtA/1AQA82SXMZJnyFhVzhIltTuo7Z/09k+/HR8Mi8f6Dd0LFd891xgOtA/He4JZ4oHUgGvp7YNg9Ifj48XZ4qXeErlyLIC84qVVQqVRGIWdQZJDu974t6pusm5fmTo+fWgW8O7CF7nbS9AcMnkOfp8YZf/zrXrzZv5luW4iJBH/o/QE49MEAnJ09xOTzAIbHdfu7fdEtoq7ZshVN0KuvEjWuZ77lrqIH24dYvP+VeyMwNboVzs4egrh/98fANoGVrk+T+p7WC1GVjI9qZr2QFe3NtNxW1CaE47qkYqn1XQoc81GNrX6tJ65k5yOsrof1wnrahfrg0MUsDO9kOrSUC/ZxxzPdGxpsi2odiKjWdz78t7/b12Adk0XPdMTVW4UGH/YVWxkGtgnCz6/1QNMAT/h6uOJqTgHWJ1/G+78dAQB0DPfFC70isO/cDQxsE4SnuobjZl4hAEDjrDbYny2aB3nhUoWxKW4uTnBzcQIAHPtwEFpN32j0uOd7NUK4vwc6hfuhUb06+OLJDugeswUuTip0i/BH7OlrZvcZ1SoQn246Ual61iZ/vXUfmk7ZYFPZ+U91wO8HL5m9f0r0nfAb4O0GranUbIWHq1OlyntpnJFTUGxT2ei2wfgz+XLlK0UAADcXNf4zoj2SUjOx5kCaxf+r6UNbI6+oBC8s3i9hDZVhSNsgs+MG5cCWj2rM1VltEDxWvNId9zYz381SbuXo7ljzRk881dVy+LBFxSAwKDIYz1YILMM7haF9mK+uNUKlUqFzo7q6lpn6XhqM6tnI4DF1NM748aVueKZ7QzipVajnqUH81CgkTn+g0nW0Nn7E3dUJXz7dweR9b/Rtim6N/QGUjs/ZM/l+JE4fgE+Ht8OgNkH4yUzTZMV8tH9KFCIbVP5bW7tQH4e2onzzbCejbeZmVJnTpZEftr/b12BbxfVtKr6Gsf2a2vTcXm7G33/C/AzD9qt97vx9n+hs+j0d1SrQ5m/XAHDw/QFWy/h6uODLpztgwciO2DP5fpufuzo7PWuw7veJg1qYbWXS324qDLYum3n211v3Wd3n8ZmD0bi+Jx7tGIovnrzHbLmt7/RBt8b+6NfizmD81/s2wbB7LLec6Vv8QhecmT0EP43ujui2wTY/rib6zxPtzd5Xsas+9r1+6Bbh7+gqVQrDRw3So4k/Zj9ieoyDPg9XZ3QM96t0C0JVubs64dcxvfD2A80tlvMuO9H0bWF6zZJ6nhp4uFa+Ma5nk3qInxqFl/W6hSp6oHUg7gnzNeg6qmNiX8E+7vDUOCPYxx2Lnu2kCyaW/N/w9qjvpcEf/7pXt61+hUHDQOk3j4p+Gt0DY/o1xZv9mxl0JT3asQF+H9vb6r5dnAz/xrsnGZ4kB7Qx3mfPJncC7OrXeuCTx9qZff4//tUbK17pjkb16hid+Gc9Eqn7vVXwnWnlT3UNx7sDW2D5y93w65heAO6EoIonhPcGGU/5Hv/AnePwzbOdMHlwKxyZMRB/vtkbg0wcw5nDIuHspMavY3ohup3h8++edL/J4KlWW//fSJz2AIa2Kz3xBfuYHnelr2mAZ6WucB1e18Pq2KSKf09z/nV/U4Op/abMebStQWh0Uavx5dMd0aWRH17uHYF1Y3phfFQzHPtwEL5+tjM6NfSDn4cLuja60x0ZN6U/zs+Jxp9v9sbZ2UPQPNALYXWtH5ty/p4afP98Zyx/uRserhAqGpvpOvuP3kxAa/q1CICTWoVujf2xYGRHk2X+fruPye2uZcdG/8uMpS975i4oaqobGAD+91I3/HdUZ7PPp29EZ/MXK326WziOzxyERzqEmvycef/B1tg1sZ/udvzUKIT6eRh0j1t7r0iB3S4kmb/e6oN/zlzTfaDbUz1PDcZFNUNOfrHJMQYaZyesKzsRNg/0xM28okp3Z+nTP8eYGp/y9bOd8N/Yc/jz0J3m+pd6R+iaPe8J88XIbuFwL+suKA9uTQM8ceDCTUwf2hrqsqX6hy3YjZnDIvHGsgO653JzUeOdB1qgWaAnni9roo5uG4wGvu54vFMofk64aLLei1/oAgD45PF2OHP1Fjo39EPbBj5YFpcC/zqu6NuiPvo2D8CbKxPxUu8IRDbw0T025tG2iD19DQPKvhU/3TUcBy5k4lZBEfo0D8BXIztidXyqrhVEfzD0gDZBODVrMFyc1Li4YDcOpmaiQ7ivyfFM3m4uOD8nGvlFJbquszoaZ7QJ8YHX9TyDso92bGDQEufr7qL7/ZEODdDA1x0NfN3Rv2Uglu27gI/+PGbyuJQL9nHTTS+vbHgvP6k9930cdp68arX8zon9UFisRfOpd7qv9k+JwtbjV7Dt+FU816MhGvi6Y2CbQGw6cgXR7YIN3k/63uzfDO8MaIFvdp7B9VuF+HrnWZvq7OKkxurXeupu68+q+/m1HijWCqhVKtzbrB7C63ogwKv0xKpSqXRBa83rvRB7+ire+umg0fObWnfo/pal75/WId74Ncl8dxxQOutO/+8QVtcdY/s1xbHLOZgS3Qr5RSX468gVvLP6oMHyBeZEtw1G0wDTIefkrMHILyqBxlmN4+k58HV3wbynOkCrFWj87/VG5ec81g6rTfyf3de8Psbe3xS9P96m23bsw0G6/3Vr9v27PwK8NFgVb/jc29/ti4b+HgbHY9awSIz+MQGDI4Ow4XDpZ4tz2RpSR2YMhFYIeLmV/k/oj/FY8Yq8g00Bho8aR6LGDIcI8nHDox3NJ/q75eXmgo8fN/8tvtwTXSo3Pfe3sb3w0Je78Wqfxvh6R+mHeoDeDJzyEyQAbHu3L1Ju5KFjuB86Pu2HBU8DjSb9CQBoWt8Lx2cOwumMW2gT4m3y5PZQ+xA8pBeewup6IGFaaVfUU13DsCIuFQBwdMYgqNUqFJVo0SbEGy2CvDB3xD0AgPahPgbho0n9OjhzNRdA6bdtABih14rh5uKka6Eot67CbaC0tUD/m7hKpcL/jbjT9DukbTCGWGjqLl/xd/WrPXAp8zYaWRm0qn9cy4X7e2Dl6O548pvSNWsqTkG/J8wXy/alADD8xuzu6oTnejRC9u0i9KkwCLp8/EeonzucLbSIfPJYO4NrNX3yeDs08q+DEV/vMSgX5me5JaBFoJfuW7l+y8fU6Fao76XBE13CDd6jXz9759vyvCcF1iam4d3Vhid6F9239ibIyMm3OXxYolKpdC1rP75kvquuvpcGj3QINRk+zM3UA0oHzN/XvD52nrwKPw8Xs+X0OalUBsfGxUmNxzqFokO4r+69bclnw0vfrxH16uDctdL/CWe1ChvGlbZalr/nfnixq+4x+q1kdeu44kZuIZ7t3hDm3ik9mvgjVK/r8O0HmhsEjz/+1Rvrky8jt6AYS/dcMHp8+ey++1sGYOvxDN12Lzdno8+MAW2CcHjGQHhqnHWfM+X1raMxPL33bVEf04e2RpsQb/hVg6uvM3zUMO56H8iWPijJftqF+uJczBBczy3UhQ83FzUOvj8AapXhDKKIenUQUeGkuu/f/VFQpIVP2QesfmtCZcx+pC2iWgWiVbC37gPGxUmNP9+816BcxZP2m/2bYdzKpCrt0xFcndVWg4cl3Rv7Y/Nb92HfuRtGLSePdQyFVgh0ami81oyrsxpvDzAeX9O7WT2Mi2qGMD8PRM/bZXa/I7qEYUSXMOTkF+F4eg46N/TD8fQco3ITB7VEfpEWfx1J1w1qHde/Gb7YcgoAsMnMOIkWNjSFO6lVeLxTKB7vFIqECzcNFiMsF+DlhrH9muLizTzMHXGP7lt7xawrxXR0Pysz8v4zoj2+333OIAzrEybn7Bkz12WjL2FqlC4E/PBiV3y36yye7dEQjet52tQNBwDP9WiIxzqGItTPHfqT/AK8NMjIKQAAgy8PAODvaXgMIhv4ILKBDz7ZeBwVPdqxge73uSPaY9afx7A64SLC63qYPZaeZSHjxV4R2HnqKh7p0MBkOZVKhRctdE1LjeGjhvH31GD60NZwcVab/GZIjqFSlQ6KnfFQG7g6q6FxdoLG2bbjr79Oyd3WoX8r69NQH7onBL8fuoweJsarhNgwdqEmaBbohWaBxidrtVplc8vWe4Na4vvd5zB5cCuE+9veBefl5oIuZeMgWgV7Y2y/pgjS6+f3cXfB/41oj3ErtbpuhXqe1r9p+rjb9u2/nJOFE+a7FgYxv/1Ac2w7kWH2hF9VT3UNx4q4FAyODMK/h7SCVgirXQ3+nhpMGGg87qdro7qIO38Dj5e1lD7XoyF+2HMBE02MEbKkVbA3jl3OBgBd9wNQ2qI44+FIcw+z6E537Z308eNL3bAiLgUju4Xb3F1nKlZ9+vidlkRfD1d8Orw9Pn6sHQSsj1OarrdkQk3A8FEDVaf0qjQVZ+1URxpnJ8NmY70PQ1v7nZXg9b5N8FqfxgYni6hWgfgu9lylVn80d6KfNLgljl/OwbM9GhqthaPvs+HtcfFmHtqF+tq8TwBo28AH3SLqooGVbp6nuobjnzPXdGOhKg5utpcPHmqNQZFB6Nqo7l2/z1aM7o6c/CLdjLkZD7XBuP7N4O9pPMDSkvlP3YOouTvvqi762oXeabVUqVT4dUwv5BeVoEWQFz6ocP2ucu42fkkMq+tuMlDa2ipT0zB8ENVyD7QOROtgb3S2Ydl7pan4LfXdgS3QIsgLfZqbXhyvMoJ93HVdLD/uOW+23ONmZk1Y46RW4adXrV9IMubRthBCOHz2m8bZyS7HDSh9bfqLKKpUqkoHDwBw1lt9+W5e/t9v98Hx9GyDacCA5cXTJg9uib1nr5sdYB+oN1Nl7Rs90cTMQNjaiuGDqJZzc3HC+nH3Wi9IcHNxwnA7d0cAppvYpSTVtPvqpqG/Bwa2CYSXm4tuUG5VNA3wNDtLxpxX+zTBq33ML/v/dLeGOHElB32a1zd5ba7ajuGDiMjBKnviIvtQqVQGs4WqE1dnNWIetT47r7Zi+CAicrCeTeph7oj2aBYg/+JORNUBwwcRkQQcucYNUU3D5dWJiIhIUgwfREREJCmGDyIiIpIUwwcRERFJiuGDiIiIJMXwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQphg8iIiKSFMMHERERSYrhg4iIiCRV7a5qK4QAAGRnZ8tcEyIiIrJV+Xm7/DxuSbULHzk5OQCAsLAwmWtCRERElZWTkwMfHx+LZVTClogiIa1Wi0uXLsHLywsqlcquz52dnY2wsDCkpqbC29vbrs+tZDyu9sdj6hg8ro7B4+oYNe24CiGQk5ODkJAQqNWWR3VUu5YPtVqN0NBQh+7D29u7RvwhaxoeV/vjMXUMHlfH4HF1jJp0XK21eJTjgFMiIiKSFMMHERERSUpR4UOj0eD999+HRqORuyq1Co+r/fGYOgaPq2PwuDpGbT6u1W7AKREREdVuimr5ICIiIvkxfBAREZGkGD6IiIhIUgwfREREJCnFhI8FCxagUaNGcHNzQ7du3RAXFyd3laqNmJgYdOnSBV5eXggICMCwYcNw4sQJgzL5+fkYM2YM/P394enpicceewxXrlwxKJOSkoLo6Gh4eHggICAAEyZMQHFxsUGZ7du3o2PHjtBoNGjatCmWLFni6JdXbcyZMwcqlQrjx4/XbeNxrZq0tDQ888wz8Pf3h7u7O9q2bYv4+Hjd/UIITJ8+HcHBwXB3d0dUVBROnTpl8Bw3btzAyJEj4e3tDV9fX7z00ku4deuWQZlDhw7h3nvvhZubG8LCwvDJJ59I8vqkVlJSgmnTpiEiIgLu7u5o0qQJZs6caXCNDh5T63bu3IkHH3wQISEhUKlUWLduncH9Uh7D1atXo2XLlnBzc0Pbtm2xfv16u7/euyIUYOXKlcLV1VV8//334siRI+KVV14Rvr6+4sqVK3JXrVoYOHCgWLx4sTh8+LBISkoSQ4YMEeHh4eLWrVu6Mq+99poICwsTW7ZsEfHx8aJ79+6iZ8+euvuLi4tFZGSkiIqKEomJiWL9+vWiXr16YvLkyboyZ8+eFR4eHuLtt98WR48eFfPnzxdOTk5i48aNkr5eOcTFxYlGjRqJdu3aiXHjxum287hW3o0bN0TDhg3F888/L/bt2yfOnj0rNm3aJE6fPq0rM2fOHOHj4yPWrVsnDh48KB566CEREREhbt++rSszaNAg0b59e7F3716xa9cu0bRpU/HUU0/p7s/KyhKBgYFi5MiR4vDhw2LFihXC3d1dfP3115K+XinMmjVL+Pv7iz/++EOcO3dOrF69Wnh6eoovvvhCV4bH1Lr169eLKVOmiDVr1ggAYu3atQb3S3UMd+/eLZycnMQnn3wijh49KqZOnSpcXFxEcnKyw4+BrRQRPrp27SrGjBmju11SUiJCQkJETEyMjLWqvjIyMgQAsWPHDiGEEJmZmcLFxUWsXr1aV+bYsWMCgNizZ48QovSfTq1Wi/T0dF2ZhQsXCm9vb1FQUCCEEGLixImiTZs2Bvt64oknxMCBAx39kmSVk5MjmjVrJjZv3iz69OmjCx88rlXz3nvvid69e5u9X6vViqCgIPHpp5/qtmVmZgqNRiNWrFghhBDi6NGjAoDYv3+/rsyGDRuESqUSaWlpQgghvvrqK+Hn56c7zuX7btGihb1fkuyio6PFiy++aLDt0UcfFSNHjhRC8JhWRcXwIeUxHDFihIiOjjaoT7du3cSrr75q19d4N2p9t0thYSESEhIQFRWl26ZWqxEVFYU9e/bIWLPqKysrCwBQt25dAEBCQgKKiooMjmHLli0RHh6uO4Z79uxB27ZtERgYqCszcOBAZGdn48iRI7oy+s9RXqa2/x3GjBmD6Ohoo9fO41o1v/32Gzp37ozhw4cjICAAHTp0wLfffqu7/9y5c0hPTzc4Jj4+PujWrZvBcfX19UXnzp11ZaKioqBWq7Fv3z5dmfvuuw+urq66MgMHDsSJEydw8+ZNR79MSfXs2RNbtmzByZMnAQAHDx5EbGwsBg8eDIDH1B6kPIY14TOh1oePa9euoaSkxODDGwACAwORnp4uU62qL61Wi/Hjx6NXr16IjIwEAKSnp8PV1RW+vr4GZfWPYXp6usljXH6fpTLZ2dm4ffu2I16O7FauXIkDBw4gJibG6D4e16o5e/YsFi5ciGbNmmHTpk14/fXX8eabb2Lp0qUA7hwXS//z6enpCAgIMLjf2dkZdevWrdSxry0mTZqEJ598Ei1btoSLiws6dOiA8ePHY+TIkQB4TO1BymNorkx1OsbV7qq2JK8xY8bg8OHDiI2NlbsqNV5qairGjRuHzZs3w83NTe7q1BparRadO3fG7NmzAQAdOnTA4cOHsWjRIowaNUrm2tVMq1atwrJly7B8+XK0adMGSUlJGD9+PEJCQnhMySFqfctHvXr14OTkZDSD4MqVKwgKCpKpVtXT2LFj8ccff2Dbtm0IDQ3VbQ8KCkJhYSEyMzMNyusfw6CgIJPHuPw+S2W8vb3h7u5u75cju4SEBGRkZKBjx45wdnaGs7MzduzYgXnz5sHZ2RmBgYE8rlUQHByM1q1bG2xr1aoVUlJSANw5Lpb+54OCgpCRkWFwf3FxMW7cuFGpY19bTJgwQdf60bZtWzz77LN46623dC12PKZ3T8pjaK5MdTrGtT58uLq6olOnTtiyZYtum1arxZYtW9CjRw8Za1Z9CCEwduxYrF27Flu3bkVERITB/Z06dYKLi4vBMTxx4gRSUlJ0x7BHjx5ITk42+MfZvHkzvL29dSeKHj16GDxHeZna+nfo378/kpOTkZSUpPvp3LkzRo4cqfudx7XyevXqZTQV/OTJk2jYsCEAICIiAkFBQQbHJDs7G/v27TM4rpmZmUhISNCV2bp1K7RaLbp166Yrs3PnThQVFenKbN68GS1atICfn5/DXp8c8vLyoFYbng6cnJyg1WoB8Jjag5THsEZ8Jsg94lUKK1euFBqNRixZskQcPXpUjB49Wvj6+hrMIFCy119/Xfj4+Ijt27eLy5cv637y8vJ0ZV577TURHh4utm7dKuLj40WPHj1Ejx49dPeXTwkdMGCASEpKEhs3bhT169c3OSV0woQJ4tixY2LBggW1ekqoKfqzXYTgca2KuLg44ezsLGbNmiVOnTolli1bJjw8PMT//vc/XZk5c+YIX19f8euvv4pDhw6Jhx9+2OSUxg4dOoh9+/aJ2NhY0axZM4MpjZmZmSIwMFA8++yz4vDhw2LlypXCw8Oj1kwL1Tdq1CjRoEED3VTbNWvWiHr16omJEyfqyvCYWpeTkyMSExNFYmKiACDmzp0rEhMTxYULF4QQ0h3D3bt3C2dnZ/HZZ5+JY8eOiffff59TbeUyf/58ER4eLlxdXUXXrl3F3r175a5StQHA5M/ixYt1ZW7fvi3eeOMN4efnJzw8PMQjjzwiLl++bPA858+fF4MHDxbu7u6iXr164p133hFFRUUGZbZt2ybuuece4erqKho3bmywDyWoGD54XKvm999/F5GRkUKj0YiWLVuKb775xuB+rVYrpk2bJgIDA4VGoxH9+/cXJ06cMChz/fp18dRTTwlPT0/h7e0tXnjhBZGTk2NQ5uDBg6J3795Co9GIBg0aiDlz5jj8tckhOztbjBs3ToSHhws3NzfRuHFjMWXKFIPpnDym1m3bts3kZ+moUaOEENIew1WrVonmzZsLV1dX0aZNG/Hnn3867HVXhUoIvSXsiIiIiBys1o/5ICIiouqF4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkGD6IiIhIUgwfREREJCmGDyIiIpIUwwcRERFJiuGDiIiIJMXwQURERJJi+CAiIiJJ/T/xWCAIG14w+gAAAABJRU5ErkJggg==", 1032 | "text/plain": [ 1033 | "
" 1034 | ] 1035 | }, 1036 | "metadata": {}, 1037 | "output_type": "display_data" 1038 | } 1039 | ], 1040 | "source": [ 1041 | "import matplotlib.pyplot as plt\n", 1042 | "import pandas as pd\n", 1043 | "\n", 1044 | "data = pd.read_csv(\"loss.txt\")\n", 1045 | "plt.plot(data)\n", 1046 | "plt.show()" 1047 | ] 1048 | }, 1049 | { 1050 | "cell_type": "markdown", 1051 | "id": "9809bc9f-4ff6-46c3-9c43-08c6c2694a82", 1052 | "metadata": {}, 1053 | "source": [ 1054 | "## Generate text with fine-tuned model\n", 1055 | "\n", 1056 | "Again we check results with our test dataset (5 rows).
\n", 1057 | "As you can see below, it can output the completion very well, because it's fine-tuned." 1058 | ] 1059 | }, 1060 | { 1061 | "cell_type": "code", 1062 | "execution_count": 26, 1063 | "id": "29903cae-404e-4209-9c84-6c8a69609c13", 1064 | "metadata": {}, 1065 | "outputs": [ 1066 | { 1067 | "name": "stdout", 1068 | "output_type": "stream", 1069 | "text": [ 1070 | "********** input **********\n", 1071 | "name : The Punter | Type : pub | food : Chinese | price : more than £ 30 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n", 1072 | "\n", 1073 | "********** result **********\n", 1074 | "name : The Punter | Type : pub | food : Chinese | price : more than £ 30 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n", 1075 | "The Punter is a children friendly pub that serves Chinese food. It is located in the riverside area near Raja Indian Cuisine and has a\n", 1076 | "********** input **********\n", 1077 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n", 1078 | "\n", 1079 | "********** result **********\n", 1080 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n", 1081 | "The Cricketers is a Chinese restaurant with a cheap price range, located in the city centre near All Bar One. It has a customer rating of\n", 1082 | "********** input **********\n", 1083 | "name : The Phoenix | Type : pub | food : French | price : moderate | customer rating : 1 out of 5 | area : riverside | family friendly : no | near : Crowne Plaza Hotel\n", 1084 | "\n", 1085 | "********** result **********\n", 1086 | "name : The Phoenix | Type : pub | food : French | price : moderate | customer rating : 1 out of 5 | area : riverside | family friendly : no | near : Crowne Plaza Hotel\n", 1087 | "The Phoenix is a pub that serves French food. It is located near Crown Plaza Hotel in the riverside area. It has a moderate price range and\n", 1088 | "********** input **********\n", 1089 | "name : Giraffe | Type : restaurant | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n", 1090 | "\n", 1091 | "********** result **********\n", 1092 | "name : Giraffe | Type : restaurant | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n", 1093 | "Giraffe is a fast food restaurant located in the riverside area near Rainbow Vegetarian Café. It is family friendly.\n", 1094 | "********** input **********\n", 1095 | "name : The Vaults | Type : pub | food : French | price : more than £ 30 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 1096 | "\n", 1097 | "********** result **********\n", 1098 | "name : The Vaults | Type : pub | food : French | price : more than £ 30 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 1099 | "The Vaults is a children friendly French pub located in the city centre near Raja Indian Cuisine.\n" 1100 | ] 1101 | } 1102 | ], 1103 | "source": [ 1104 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n", 1105 | "test_data = test_data[::2] # because it's duplicated\n", 1106 | "test_loader = DataLoader(\n", 1107 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n", 1108 | " batch_size=1,\n", 1109 | " shuffle=True,\n", 1110 | " collate_fn=collate_batch\n", 1111 | ")\n", 1112 | "\n", 1113 | "for i, (input, _, mask) in enumerate(test_loader):\n", 1114 | " if i == 5:\n", 1115 | " break\n", 1116 | " print(\"********** input **********\")\n", 1117 | " input_len = torch.sum(mask).cpu().numpy()\n", 1118 | " print(tokenizer.decode(input[0][:input_len]))\n", 1119 | " result_token, result_len = generate_text(\n", 1120 | " model,\n", 1121 | " input,\n", 1122 | " mask,\n", 1123 | " eos_id,\n", 1124 | " pred_sequence_length=30)\n", 1125 | " print(\"********** result **********\")\n", 1126 | " print(tokenizer.decode(result_token[0][:result_len]))" 1127 | ] 1128 | }, 1129 | { 1130 | "cell_type": "code", 1131 | "execution_count": null, 1132 | "id": "6a7c1dd3-4057-497a-83ae-f99b1883697e", 1133 | "metadata": {}, 1134 | "outputs": [], 1135 | "source": [] 1136 | } 1137 | ], 1138 | "metadata": { 1139 | "kernelspec": { 1140 | "display_name": "Python 3 (ipykernel)", 1141 | "language": "python", 1142 | "name": "python3" 1143 | }, 1144 | "language_info": { 1145 | "codemirror_mode": { 1146 | "name": "ipython", 1147 | "version": 3 1148 | }, 1149 | "file_extension": ".py", 1150 | "mimetype": "text/x-python", 1151 | "name": "python", 1152 | "nbconvert_exporter": "python", 1153 | "pygments_lexer": "ipython3", 1154 | "version": "3.8.10" 1155 | } 1156 | }, 1157 | "nbformat": 4, 1158 | "nbformat_minor": 5 1159 | } 1160 | -------------------------------------------------------------------------------- /02-finetune-gpt2-with-lora.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "857cafc6-da38-4aa7-8afc-63aa626fa7aa", 6 | "metadata": {}, 7 | "source": [ 8 | "# 02. Finetuning GPT-2 with LoRA\n", 9 | "\n", 10 | "In this example, we fine-tune the pre-trained auto-regressive model, **OpenAI's GPT-2** (small version, 124M parameters), by applying LoRA (Low-Rank Adaptation) optimization.\n", 11 | "\n", 12 | "In this example, I download the pre-trained model from Hugging Face hub, but fine-tune model with regular PyTorch training loop.
\n", 13 | "(Here I don't use Hugging Face Trainer class.)\n", 14 | "\n", 15 | "See [Readme](https://github.com/tsmatz/finetune_llm_with_lora) for prerequisite's setup." 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "id": "3d49acf1-9ad1-4a6c-9312-6785cb3f5862", 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "model_name = \"gpt2\"" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "id": "4d835e84-a01d-4c33-926b-60d9dd4a7627", 32 | "metadata": {}, 33 | "outputs": [], 34 | "source": [ 35 | "import torch\n", 36 | "\n", 37 | "device = torch.device(\"cuda\")" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "id": "ead383e5-149b-4bfb-9324-3cc639fd398d", 43 | "metadata": {}, 44 | "source": [ 45 | "## Prepare dataset and dataloader" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "id": "91ecbb08-6a74-4623-bfe8-bddba5254e35", 51 | "metadata": {}, 52 | "source": [ 53 | "In this example, we use dataset used in [official LoRA example](https://github.com/microsoft/LoRA).\n", 54 | "\n", 55 | "Download dataset from official repository." 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 3, 61 | "id": "54a564f1-f8f3-42a6-b160-bebdbcc3aac0", 62 | "metadata": {}, 63 | "outputs": [ 64 | { 65 | "name": "stdout", 66 | "output_type": "stream", 67 | "text": [ 68 | "--2023-10-06 03:27:50-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt\n", 69 | "Resolving github.com (github.com)... 140.82.114.3\n", 70 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n", 71 | "HTTP request sent, awaiting response... 302 Found\n", 72 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt [following]\n", 73 | "--2023-10-06 03:27:51-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt\n", 74 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n", 75 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n", 76 | "HTTP request sent, awaiting response... 200 OK\n", 77 | "Length: 9624463 (9.2M) [text/plain]\n", 78 | "Saving to: ‘train.txt’\n", 79 | "\n", 80 | "train.txt 100%[===================>] 9.18M --.-KB/s in 0.04s \n", 81 | "\n", 82 | "2023-10-06 03:27:51 (248 MB/s) - ‘train.txt’ saved [9624463/9624463]\n", 83 | "\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 4, 94 | "id": "d48464ea-991f-48b2-9166-3323cfd61676", 95 | "metadata": { 96 | "scrolled": true 97 | }, 98 | "outputs": [ 99 | { 100 | "name": "stdout", 101 | "output_type": "stream", 102 | "text": [ 103 | "--2023-10-06 03:27:54-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt\n", 104 | "Resolving github.com (github.com)... 140.82.114.3\n", 105 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n", 106 | "HTTP request sent, awaiting response... 302 Found\n", 107 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt [following]\n", 108 | "--2023-10-06 03:27:54-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt\n", 109 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n", 110 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", 111 | "HTTP request sent, awaiting response... 200 OK\n", 112 | "Length: 1351149 (1.3M) [text/plain]\n", 113 | "Saving to: ‘test.txt’\n", 114 | "\n", 115 | "test.txt 100%[===================>] 1.29M --.-KB/s in 0.006s \n", 116 | "\n", 117 | "2023-10-06 03:27:54 (208 MB/s) - ‘test.txt’ saved [1351149/1351149]\n", 118 | "\n" 119 | ] 120 | } 121 | ], 122 | "source": [ 123 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "id": "09472803-8c62-48e0-9a63-b9b9448f16d3", 129 | "metadata": {}, 130 | "source": [ 131 | "Show the downloaded data (first 5 rows)." 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 5, 137 | "id": "e6e60596-028f-4c4b-a95d-f74a0ff3b188", 138 | "metadata": {}, 139 | "outputs": [ 140 | { 141 | "name": "stdout", 142 | "output_type": "stream", 143 | "text": [ 144 | "name : The Vaults | Type : pub | price : more than £ 30 | customer rating : 5 out of 5 | near : Café Adriatic||The Vaults pub near Café Adriatic has a 5 star rating . Prices start at £ 30 . \n", 145 | "name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Café Brazil||Close to Café Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of £ 10.50 . Delicious Pub food . \n", 146 | "name : The Eagle | Type : coffee shop | food : Japanese | price : less than £ 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King||The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than £ 20 for Japanese food . \n", 147 | "name : The Mill | Type : coffee shop | food : French | price : £ 20 - 25 | area : riverside | near : The Sorrento||Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at £ 20- £ 25 it is in the riverside area . \n", 148 | "name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat||For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat . \n" 149 | ] 150 | } 151 | ], 152 | "source": [ 153 | "!head -n 5 train.txt" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "id": "93f5fabe-590c-459b-aa16-4b5a506fb54b", 159 | "metadata": {}, 160 | "source": [ 161 | "Convert above data into JsonL format." 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 6, 167 | "id": "7376e0c0-16c9-46f4-ad4c-83d1a677f5a2", 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "import sys\n", 172 | "import io\n", 173 | "import json\n", 174 | "\n", 175 | "def format_convert(read_file, write_file):\n", 176 | " with open(read_file, \"r\", encoding=\"utf8\") as reader, \\\n", 177 | " \t open(write_file, \"w\", encoding=\"utf8\") as writer :\n", 178 | " \tfor line in reader:\n", 179 | " \t\titems = line.strip().split(\"||\")\n", 180 | " \t\tcontext = items[0]\n", 181 | " \t\tcompletion = items[1].strip(\"\\n\")\n", 182 | " \t\tx = {}\n", 183 | " \t\tx[\"context\"] = context\n", 184 | " \t\tx[\"completion\"] = completion\n", 185 | " \t\twriter.write(json.dumps(x)+\"\\n\")\n", 186 | "\n", 187 | "format_convert(\"train.txt\", \"train_formatted.jsonl\")\n", 188 | "format_convert(\"test.txt\", \"test_formatted.jsonl\")" 189 | ] 190 | }, 191 | { 192 | "cell_type": "markdown", 193 | "id": "3ceec952-fe03-475f-9f3e-22237cc9c44b", 194 | "metadata": {}, 195 | "source": [ 196 | "Show the converted data (first 5 rows)." 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": 7, 202 | "id": "cb32aca7-bd0e-4847-a4c2-cc7e67dc2b7a", 203 | "metadata": {}, 204 | "outputs": [ 205 | { 206 | "name": "stdout", 207 | "output_type": "stream", 208 | "text": [ 209 | "{\"context\": \"name : The Vaults | Type : pub | price : more than \\u00a3 30 | customer rating : 5 out of 5 | near : Caf\\u00e9 Adriatic\", \"completion\": \"The Vaults pub near Caf\\u00e9 Adriatic has a 5 star rating . Prices start at \\u00a3 30 .\"}\n", 210 | "\n", 211 | "{\"context\": \"name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Caf\\u00e9 Brazil\", \"completion\": \"Close to Caf\\u00e9 Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of \\u00a3 10.50 . Delicious Pub food .\"}\n", 212 | "\n", 213 | "{\"context\": \"name : The Eagle | Type : coffee shop | food : Japanese | price : less than \\u00a3 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King\", \"completion\": \"The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than \\u00a3 20 for Japanese food .\"}\n", 214 | "\n", 215 | "{\"context\": \"name : The Mill | Type : coffee shop | food : French | price : \\u00a3 20 - 25 | area : riverside | near : The Sorrento\", \"completion\": \"Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at \\u00a3 20- \\u00a3 25 it is in the riverside area .\"}\n", 216 | "\n", 217 | "{\"context\": \"name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat\", \"completion\": \"For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat .\"}\n", 218 | "\n" 219 | ] 220 | } 221 | ], 222 | "source": [ 223 | "with open(\"train_formatted.jsonl\", \"r\") as reader:\n", 224 | " for _ in range(5):\n", 225 | " print(next(reader))" 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "id": "6631f786-be4b-40cf-89d9-7009c1888821", 231 | "metadata": {}, 232 | "source": [ 233 | "Load tokenizer from Hugging Face." 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": 8, 239 | "id": "e5433dc0-b5a5-4c01-adb5-3ffa2279eca8", 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "from transformers import AutoTokenizer\n", 244 | "import os\n", 245 | "\n", 246 | "tokenizer = AutoTokenizer.from_pretrained(\n", 247 | " model_name,\n", 248 | " fast_tokenizer=True)\n", 249 | "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"" 250 | ] 251 | }, 252 | { 253 | "cell_type": "markdown", 254 | "id": "50817c47-a97b-4f80-975b-836859a0a7cf", 255 | "metadata": {}, 256 | "source": [ 257 | "Set block size which is used to separate long text for model consumption.
\n", 258 | "Max 1024 tokens can be used in GPT-2, but here I set 512, because it's enough for our dataset." 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 9, 264 | "id": "5f250929-5703-4b17-9f7b-26340950c055", 265 | "metadata": {}, 266 | "outputs": [ 267 | { 268 | "name": "stdout", 269 | "output_type": "stream", 270 | "text": [ 271 | "Max length of tokens is 1024 in this model.\n", 272 | "But here we use max 512 tokens in the training.\n" 273 | ] 274 | } 275 | ], 276 | "source": [ 277 | "block_size = 512\n", 278 | "\n", 279 | "print(f\"Max length of tokens is {tokenizer.model_max_length} in this model.\")\n", 280 | "print(f\"But here we use max {block_size} tokens in the training.\")" 281 | ] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "id": "2332617b-1e66-4812-ad47-5eaeb52b101b", 286 | "metadata": {}, 287 | "source": [ 288 | "Create function to convert data. (Later this function is then used in data loader.)
\n", 289 | "In this function,\n", 290 | "\n", 291 | "1. Tokenize both contexts and compeletions. : e.g, ```\"This is a pen.\"``` --> ```[1212, 318, 257, 3112, 13]```\n", 292 | "2. Concatenate context's token and completion's token. (But it's delimited by \"\\n\" between context and completion.) This is used for inputs for LLM.\n", 293 | "3. Create labels (targets) with inputs. Label is ```input[1:]``` (i.e, shifted right by one element), and is filled by ```-100``` in context's positions. (See below note.)\n", 294 | "4. Pad tokens to make the length of token become ```block_size```.\n", 295 | "\n", 296 | "> Note : Here I set ```-100``` as an ignored index for loss computation, because PyTorch cross-entropy function (```torch.nn.functional.cross_entropy()```) has a property ```ignore_index``` which default value is ```-100```." 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 10, 302 | "id": "9f2f38aa-b3d0-4614-aa59-8ddd977176d1", 303 | "metadata": {}, 304 | "outputs": [], 305 | "source": [ 306 | "from torch.utils.data import DataLoader\n", 307 | "import pandas as pd\n", 308 | "\n", 309 | "def fill_ignore_label(l, c):\n", 310 | " l[:len(c) - 1] = [-100] * (len(c) - 1)\n", 311 | " return l\n", 312 | "\n", 313 | "def pad_tokens(tokens, max_seq_length, padding_token):\n", 314 | " res_tokens = tokens[:max_seq_length]\n", 315 | " token_len = len(res_tokens)\n", 316 | " res_tokens = res_tokens + \\\n", 317 | " [padding_token for _ in range(max_seq_length - token_len)]\n", 318 | " return res_tokens\n", 319 | "\n", 320 | "def collate_batch(batch):\n", 321 | " # tokenize both context and completion respectively\n", 322 | " # (context and completion is delimited by \"\\n\")\n", 323 | " context_list = list(zip(*batch))[0]\n", 324 | " context_list = [c + \"\\n\" for c in context_list]\n", 325 | " completion_list = list(zip(*batch))[1]\n", 326 | " context_result = tokenizer(context_list)\n", 327 | " context_tokens = context_result[\"input_ids\"]\n", 328 | " context_masks = context_result[\"attention_mask\"]\n", 329 | " completion_result = tokenizer(completion_list)\n", 330 | " completion_tokens = completion_result[\"input_ids\"]\n", 331 | " completion_masks = completion_result[\"attention_mask\"]\n", 332 | " # concatenate token\n", 333 | " inputs = [i + j for i, j in zip(context_tokens, completion_tokens)]\n", 334 | " masks = [i + j for i, j in zip(context_masks, completion_masks)]\n", 335 | " # create label\n", 336 | " eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n", 337 | " labels = [t[1:] + [eos_id] for t in inputs]\n", 338 | " labels = list(map(fill_ignore_label, labels, context_tokens))\n", 339 | " # truncate and pad tokens\n", 340 | " inputs = [pad_tokens(t, block_size, 0) for t in inputs] # OPT and GPT-2 doesn't use pad token (instead attn mask is used)\n", 341 | " masks = [pad_tokens(t, block_size, 0) for t in masks]\n", 342 | " labels = [pad_tokens(t, block_size, -100) for t in labels]\n", 343 | " # convert to tensor\n", 344 | " inputs = torch.tensor(inputs, dtype=torch.int64).to(device)\n", 345 | " masks = torch.tensor(masks, dtype=torch.int64).to(device)\n", 346 | " labels = torch.tensor(labels, dtype=torch.int64).to(device)\n", 347 | " return inputs, labels, masks" 348 | ] 349 | }, 350 | { 351 | "cell_type": "markdown", 352 | "id": "2084d2e9-ef64-47a2-aec9-d24ead1cb38a", 353 | "metadata": {}, 354 | "source": [ 355 | "Now create PyTorch dataloader with previous function (collator function).\n", 356 | "\n", 357 | "> Note : In this example, data is small and we then load all JSON data in memory.
\n", 358 | "> When it's large, load data progressively by implementing custom PyTorch dataset. (See [here](https://github.com/tsmatz/decision-transformer) for example.)" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": 11, 364 | "id": "f3bce3bb-2215-4bd6-a6a6-5b6b9d5afdc0", 365 | "metadata": {}, 366 | "outputs": [], 367 | "source": [ 368 | "batch_size = 8\n", 369 | "gradient_accumulation_steps = 16\n", 370 | "\n", 371 | "data = pd.read_json(\"train_formatted.jsonl\", lines=True)\n", 372 | "dataloader = DataLoader(\n", 373 | " list(zip(data[\"context\"], data[\"completion\"])),\n", 374 | " batch_size=batch_size,\n", 375 | " shuffle=True,\n", 376 | " collate_fn=collate_batch\n", 377 | ")" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "id": "3ba64144-b698-457e-b827-941020456536", 383 | "metadata": {}, 384 | "source": [ 385 | "## Load model" 386 | ] 387 | }, 388 | { 389 | "cell_type": "markdown", 390 | "id": "1bfd360d-7bdc-4fd7-9b12-bcf9fe0a8db2", 391 | "metadata": {}, 392 | "source": [ 393 | "Load model from Hugging Face." 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": 12, 399 | "id": "271181bd-677a-4da9-9e57-2874f5e47bd0", 400 | "metadata": {}, 401 | "outputs": [], 402 | "source": [ 403 | "from transformers import AutoModelForCausalLM, AutoConfig\n", 404 | "\n", 405 | "config = AutoConfig.from_pretrained(model_name)\n", 406 | "model = AutoModelForCausalLM.from_pretrained(\n", 407 | " model_name,\n", 408 | " config=config,\n", 409 | ").to(device)" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "id": "27ab764a-d634-40f8-9edb-a01146845233", 415 | "metadata": {}, 416 | "source": [ 417 | "## Generate text (before fine-tuning)" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "id": "559efeaf-4b38-4a0c-9be6-eb394221e374", 423 | "metadata": {}, 424 | "source": [ 425 | "Now run prediction with downloaded model (which is not still fine-tuned).\n", 426 | "\n", 427 | "First we create a function to generate text." 428 | ] 429 | }, 430 | { 431 | "cell_type": "code", 432 | "execution_count": 13, 433 | "id": "51a0c4fc-e0a7-4bbf-b25a-c335fe61f3df", 434 | "metadata": {}, 435 | "outputs": [], 436 | "source": [ 437 | "def generate_text(model, input, mask, eos_id, pred_sequence_length):\n", 438 | " predicted_last_id = -1\n", 439 | " start_token_len = torch.sum(mask).cpu().numpy()\n", 440 | " token_len = start_token_len\n", 441 | " with torch.no_grad():\n", 442 | " while (predicted_last_id != eos_id) and \\\n", 443 | " (token_len - start_token_len < pred_sequence_length):\n", 444 | " output = model(\n", 445 | " input_ids=input,\n", 446 | " attention_mask=mask,\n", 447 | " )\n", 448 | " predicted_ids = torch.argmax(output.logits, axis=-1).cpu().numpy()\n", 449 | " predicted_last_id = predicted_ids[0][token_len - 1]\n", 450 | " input[0][token_len] = predicted_last_id\n", 451 | " mask[0][token_len] = 1\n", 452 | " token_len = torch.sum(mask).cpu().numpy()\n", 453 | " return input, token_len" 454 | ] 455 | }, 456 | { 457 | "cell_type": "markdown", 458 | "id": "3936b1a1-ae9f-48a5-80db-691261dda704", 459 | "metadata": {}, 460 | "source": [ 461 | "Let's test our function and generate text. (Here we stop the text generation when it reaches 15 tokens in prediction.)" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": 14, 467 | "id": "28b7e13f-e8fb-4a9f-90ed-0464463ef569", 468 | "metadata": {}, 469 | "outputs": [ 470 | { 471 | "name": "stdout", 472 | "output_type": "stream", 473 | "text": [ 474 | "Once upon a time, the world was a place of great beauty and great danger. The world was\n", 475 | "My name is Clara and I am a woman. I am a woman who is a woman. I am a\n" 476 | ] 477 | } 478 | ], 479 | "source": [ 480 | "eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n", 481 | "\n", 482 | "result = tokenizer(\"Once upon a time,\")\n", 483 | "input = result[\"input_ids\"]\n", 484 | "mask = result[\"attention_mask\"]\n", 485 | "input = pad_tokens(input, block_size, 0)\n", 486 | "mask = pad_tokens(mask, block_size, 0)\n", 487 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n", 488 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n", 489 | "\n", 490 | "result_token, result_len = generate_text(\n", 491 | " model,\n", 492 | " input,\n", 493 | " mask,\n", 494 | " eos_id,\n", 495 | " pred_sequence_length=15)\n", 496 | "print(tokenizer.decode(result_token[0][:result_len]))\n", 497 | "\n", 498 | "result = tokenizer(\"My name is Clara and I am\")\n", 499 | "input = result[\"input_ids\"]\n", 500 | "mask = result[\"attention_mask\"]\n", 501 | "input = pad_tokens(input, block_size, 0)\n", 502 | "mask = pad_tokens(mask, block_size, 0)\n", 503 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n", 504 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n", 505 | "\n", 506 | "result_token, result_len = generate_text(\n", 507 | " model,\n", 508 | " input,\n", 509 | " mask,\n", 510 | " eos_id,\n", 511 | " pred_sequence_length=15)\n", 512 | "print(tokenizer.decode(result_token[0][:result_len]))" 513 | ] 514 | }, 515 | { 516 | "cell_type": "markdown", 517 | "id": "d48fb60b-c05d-4884-a9bc-92152c94c894", 518 | "metadata": {}, 519 | "source": [ 520 | "Now we generate text with our test dataset (5 rows).
\n", 521 | "As you can see below, it cannot output the completion well, because it's not still fine-tuned." 522 | ] 523 | }, 524 | { 525 | "cell_type": "code", 526 | "execution_count": 15, 527 | "id": "495728ef-fbe6-4953-a354-4b7a8bb88798", 528 | "metadata": {}, 529 | "outputs": [ 530 | { 531 | "name": "stdout", 532 | "output_type": "stream", 533 | "text": [ 534 | "********** input **********\n", 535 | "name : Wildwood | Type : pub | food : Indian | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 536 | "\n", 537 | "********** result **********\n", 538 | "name : Wildwood | Type : pub | food : Indian | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 539 | "\n", 540 | "Raja Indian Cuisine : Indian | price : Rs. 1,000 | menu : Indian | menu type : food | menu size :\n", 541 | "********** input **********\n", 542 | "name : Giraffe | Type : pub | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n", 543 | "\n", 544 | "********** result **********\n", 545 | "name : Giraffe | Type : pub | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n", 546 | "\n", 547 | ": Giraffe | Type : pub | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n", 548 | "********** input **********\n", 549 | "name : The Waterman | Type : pub | food : Italian | price : less than £ 20 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 550 | "\n", 551 | "********** result **********\n", 552 | "name : The Waterman | Type : pub | food : Italian | price : less than £ 20 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 553 | "\n", 554 | "The Waterman is a pub in the heart of the city centre. It is a place where you can enjoy a good meal and drink a good\n", 555 | "********** input **********\n", 556 | "name : The Vaults | Type : pub | food : Italian | price : moderate | customer rating : 1 out of 5 | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n", 557 | "\n", 558 | "********** result **********\n", 559 | "name : The Vaults | Type : pub | food : Italian | price : moderate | customer rating : 1 out of 5 | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n", 560 | "\n", 561 | "The Vaults | Type : pub | food : Italian | price : moderate | customer rating : 1 out of 5 | area : city centre | family\n", 562 | "********** input **********\n", 563 | "name : The Vaults | Type : restaurant | food : French | price : less than £ 20 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n", 564 | "\n", 565 | "********** result **********\n", 566 | "name : The Vaults | Type : restaurant | food : French | price : less than £ 20 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n", 567 | "\n", 568 | "The restaurant is located in the centre of the city. It is a small restaurant with a small menu. The menu is very simple and the food\n" 569 | ] 570 | } 571 | ], 572 | "source": [ 573 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n", 574 | "test_data = test_data[::2] # because it's duplicated\n", 575 | "test_loader = DataLoader(\n", 576 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n", 577 | " batch_size=1,\n", 578 | " shuffle=True,\n", 579 | " collate_fn=collate_batch\n", 580 | ")\n", 581 | "\n", 582 | "for i, (input, _, mask) in enumerate(test_loader):\n", 583 | " if i == 5:\n", 584 | " break\n", 585 | " print(\"********** input **********\")\n", 586 | " input_len = torch.sum(mask).cpu().numpy()\n", 587 | " print(tokenizer.decode(input[0][:input_len]))\n", 588 | " result_token, result_len = generate_text(\n", 589 | " model,\n", 590 | " input,\n", 591 | " mask,\n", 592 | " eos_id,\n", 593 | " pred_sequence_length=30)\n", 594 | " print(\"********** result **********\")\n", 595 | " print(tokenizer.decode(result_token[0][:result_len]))" 596 | ] 597 | }, 598 | { 599 | "cell_type": "markdown", 600 | "id": "e3138341-e01c-4fae-af78-c61e34967e92", 601 | "metadata": {}, 602 | "source": [ 603 | "## LoRA (Low-Rank Adaptation)\n", 604 | "\n", 605 | "Now we apply LoRA in our downloaded model.
\n", 606 | "For semantics of LoRA (Low-Rank Adaptation), see [01-finetune-opt-with-lora.ipynb](./01-finetune-opt-with-lora.ipynb).\n", 607 | "\n", 608 | "For the purpose of your learning, here I manually (from scratch) convert the current model into the model with LoRA.\n", 609 | "\n", 610 | "> Note : You can use ```PEFT``` package to be able to get LoRA model with a few lines of code. (Here I don't use this package.)" 611 | ] 612 | }, 613 | { 614 | "cell_type": "markdown", 615 | "id": "e296bdf2-129a-4278-8fe3-d08333ebf1df", 616 | "metadata": {}, 617 | "source": [ 618 | "Before changing our model, first we check the structure of our model. (See the following result in the cell.)\n", 619 | "\n", 620 | "As you can see below, you cannot find any linear layers in OpenAI's GPT-2 transformer, unlike [Meta's OPT transformer](./01-finetune-opt-with-lora.ipynb). Instead, you will find Conv1D (1D convolution) in transformer.
\n", 621 | "However, this Conv1D is not ```torch.nn.Conv1d``` and it's a custom layer defined for OpenAI GPT, which works same as a linear layer, but the weights are transposed. (See [this source code](https://github.com/huggingface/transformers/blob/main/src/transformers/pytorch_utils.py) for custom ```pytorch_utils.Conv1D``` implementation.)
\n", 622 | "This custom Conv1D layer (intrinsically, a linear layer) is used for MLP and getting key/value/query in GPT-2 transformer as follows.
\n", 623 | "(See [source code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_gpt2.py) for GPT-2 in Hugging Face tarnsformers.)\n", 624 | "\n", 625 | "- ```transformer.h.n.attn.c_attn``` : Layer to get key/value/query before processing attention.\n", 626 | "- ```transformer.h.n.attn.c_proj``` : Layer for projection after processing attention.\n", 627 | "- ```transformer.h.n.mlp.c_attn``` : MLP in GPT-2 is Linear(GeLU(Linear)). This is an inner Linear layer (custom Conv1D).\n", 628 | "- ```transformer.h.n.mlp.c_proj``` : MLP in GPT-2 is Linear(GeLU(Linear)). This is an outer Linear layer (custom Conv1D).\n", 629 | "\n", 630 | "In this example, we'll only convert ```transformer.h.n.attn.c_attn``` layers into LoRA layers.
\n", 631 | "The transformer in GPT-2 with 124M parameters has 12 layers and it then has total 12 layers (n=0,1, ... , 11) to be converted." 632 | ] 633 | }, 634 | { 635 | "cell_type": "code", 636 | "execution_count": 16, 637 | "id": "5acb8f62-791a-4fa4-b00c-2666cf34827f", 638 | "metadata": {}, 639 | "outputs": [ 640 | { 641 | "data": { 642 | "text/plain": [ 643 | "GPT2LMHeadModel(\n", 644 | " (transformer): GPT2Model(\n", 645 | " (wte): Embedding(50257, 768)\n", 646 | " (wpe): Embedding(1024, 768)\n", 647 | " (drop): Dropout(p=0.1, inplace=False)\n", 648 | " (h): ModuleList(\n", 649 | " (0-11): 12 x GPT2Block(\n", 650 | " (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 651 | " (attn): GPT2Attention(\n", 652 | " (c_attn): Conv1D()\n", 653 | " (c_proj): Conv1D()\n", 654 | " (attn_dropout): Dropout(p=0.1, inplace=False)\n", 655 | " (resid_dropout): Dropout(p=0.1, inplace=False)\n", 656 | " )\n", 657 | " (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 658 | " (mlp): GPT2MLP(\n", 659 | " (c_fc): Conv1D()\n", 660 | " (c_proj): Conv1D()\n", 661 | " (act): NewGELUActivation()\n", 662 | " (dropout): Dropout(p=0.1, inplace=False)\n", 663 | " )\n", 664 | " )\n", 665 | " )\n", 666 | " (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 667 | " )\n", 668 | " (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n", 669 | ")" 670 | ] 671 | }, 672 | "execution_count": 16, 673 | "metadata": {}, 674 | "output_type": "execute_result" 675 | } 676 | ], 677 | "source": [ 678 | "model" 679 | ] 680 | }, 681 | { 682 | "cell_type": "markdown", 683 | "id": "045e7239-cb8a-46dd-815d-e48e7e49eea4", 684 | "metadata": {}, 685 | "source": [ 686 | "First we build custom linear layer with LoRA as follows." 687 | ] 688 | }, 689 | { 690 | "cell_type": "code", 691 | "execution_count": 17, 692 | "id": "77889272-9a93-491b-93cb-b0bed5ce7cd8", 693 | "metadata": {}, 694 | "outputs": [], 695 | "source": [ 696 | "import math\n", 697 | "from torch import nn\n", 698 | "\n", 699 | "class LoRA_Linear(nn.Module):\n", 700 | " def __init__(self, weight, bias, lora_dim):\n", 701 | " super(LoRA_Linear, self).__init__()\n", 702 | "\n", 703 | " row, column = weight.shape\n", 704 | "\n", 705 | " # restore Linear\n", 706 | " if bias is None:\n", 707 | " self.linear = nn.Linear(column, row, bias=False)\n", 708 | " self.linear.load_state_dict({\"weight\": weight})\n", 709 | " else:\n", 710 | " self.linear = nn.Linear(column, row)\n", 711 | " self.linear.load_state_dict({\"weight\": weight, \"bias\": bias})\n", 712 | "\n", 713 | " # create LoRA weights (with initialization)\n", 714 | " self.lora_right = nn.Parameter(torch.zeros(column, lora_dim))\n", 715 | " nn.init.kaiming_uniform_(self.lora_right, a=math.sqrt(5))\n", 716 | " self.lora_left = nn.Parameter(torch.zeros(lora_dim, row))\n", 717 | "\n", 718 | " def forward(self, input):\n", 719 | " x = self.linear(input)\n", 720 | " y = input @ self.lora_right @ self.lora_left\n", 721 | " return x + y" 722 | ] 723 | }, 724 | { 725 | "cell_type": "markdown", 726 | "id": "954e2c9d-545e-4bd9-9b0f-eba3fe29a1de", 727 | "metadata": {}, 728 | "source": [ 729 | "Replace targeting linear layers with LoRA layers.\n", 730 | "\n", 731 | "> Note : As I have mentioned above, custom Conv1D layer in GPT-2 is intrinsically a linear layer, but the weights are transposed." 732 | ] 733 | }, 734 | { 735 | "cell_type": "code", 736 | "execution_count": 18, 737 | "id": "baf8a748-a3e3-45b8-9c64-252c56abe923", 738 | "metadata": {}, 739 | "outputs": [], 740 | "source": [ 741 | "lora_dim = 128\n", 742 | "\n", 743 | "# get target module name\n", 744 | "target_names = []\n", 745 | "for name, module in model.named_modules():\n", 746 | " if \"attn.c_attn\" in name:\n", 747 | " target_names.append(name)\n", 748 | "\n", 749 | "# replace each module with LoRA\n", 750 | "for name in target_names:\n", 751 | " name_struct = name.split(\".\")\n", 752 | " # get target module\n", 753 | " module_list = [model]\n", 754 | " for struct in name_struct:\n", 755 | " module_list.append(getattr(module_list[-1], struct))\n", 756 | " # build LoRA\n", 757 | " lora = LoRA_Linear(\n", 758 | " weight = torch.transpose(module_list[-1].weight, 0, 1),\n", 759 | " bias = module_list[-1].bias,\n", 760 | " lora_dim = lora_dim,\n", 761 | " ).to(device)\n", 762 | " # replace\n", 763 | " module_list[-2].__setattr__(name_struct[-1], lora)" 764 | ] 765 | }, 766 | { 767 | "cell_type": "markdown", 768 | "id": "8aae2df9-fae7-4ecc-8260-80e8e578d951", 769 | "metadata": {}, 770 | "source": [ 771 | "See how model is changed." 772 | ] 773 | }, 774 | { 775 | "cell_type": "code", 776 | "execution_count": 19, 777 | "id": "bf16b414-b973-40eb-be81-fd2aa3dde439", 778 | "metadata": {}, 779 | "outputs": [ 780 | { 781 | "data": { 782 | "text/plain": [ 783 | "GPT2LMHeadModel(\n", 784 | " (transformer): GPT2Model(\n", 785 | " (wte): Embedding(50257, 768)\n", 786 | " (wpe): Embedding(1024, 768)\n", 787 | " (drop): Dropout(p=0.1, inplace=False)\n", 788 | " (h): ModuleList(\n", 789 | " (0-11): 12 x GPT2Block(\n", 790 | " (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 791 | " (attn): GPT2Attention(\n", 792 | " (c_attn): LoRA_Linear(\n", 793 | " (linear): Linear(in_features=768, out_features=2304, bias=True)\n", 794 | " )\n", 795 | " (c_proj): Conv1D()\n", 796 | " (attn_dropout): Dropout(p=0.1, inplace=False)\n", 797 | " (resid_dropout): Dropout(p=0.1, inplace=False)\n", 798 | " )\n", 799 | " (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 800 | " (mlp): GPT2MLP(\n", 801 | " (c_fc): Conv1D()\n", 802 | " (c_proj): Conv1D()\n", 803 | " (act): NewGELUActivation()\n", 804 | " (dropout): Dropout(p=0.1, inplace=False)\n", 805 | " )\n", 806 | " )\n", 807 | " )\n", 808 | " (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n", 809 | " )\n", 810 | " (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n", 811 | ")" 812 | ] 813 | }, 814 | "execution_count": 19, 815 | "metadata": {}, 816 | "output_type": "execute_result" 817 | } 818 | ], 819 | "source": [ 820 | "model" 821 | ] 822 | }, 823 | { 824 | "cell_type": "markdown", 825 | "id": "e9099c08-f6a6-45f8-939b-cc3ed9415976", 826 | "metadata": {}, 827 | "source": [ 828 | "Finally, freeze all parameters except for LoRA parameters." 829 | ] 830 | }, 831 | { 832 | "cell_type": "code", 833 | "execution_count": 20, 834 | "id": "81d06bba-955b-4806-8ff7-f217252e3268", 835 | "metadata": {}, 836 | "outputs": [], 837 | "source": [ 838 | "for name, param in model.named_parameters():\n", 839 | " if \"lora_right\" in name or \"lora_left\" in name:\n", 840 | " param.requires_grad = True\n", 841 | " else:\n", 842 | " param.requires_grad = False" 843 | ] 844 | }, 845 | { 846 | "cell_type": "code", 847 | "execution_count": null, 848 | "id": "6c0a4469-2827-4f30-9324-711a9feea1ae", 849 | "metadata": {}, 850 | "outputs": [], 851 | "source": [ 852 | "### Do this when you run adapter fine-tuning on Hugging Face framework\n", 853 | "# model.gradient_checkpointing_enable()\n", 854 | "# model.enable_input_require_grads()" 855 | ] 856 | }, 857 | { 858 | "cell_type": "markdown", 859 | "id": "6d6c7d6f-6c50-4839-88a5-c851caab9ba2", 860 | "metadata": {}, 861 | "source": [ 862 | "## Fine-tune" 863 | ] 864 | }, 865 | { 866 | "cell_type": "markdown", 867 | "id": "a12b875f-36cc-40b8-aaab-1efda68710f3", 868 | "metadata": {}, 869 | "source": [ 870 | "Now let's start to run fine-tuning.\n", 871 | "\n", 872 | "First we build optimizer as follows." 873 | ] 874 | }, 875 | { 876 | "cell_type": "code", 877 | "execution_count": 21, 878 | "id": "bb51298a-2d55-466c-a990-0ea08a247350", 879 | "metadata": {}, 880 | "outputs": [], 881 | "source": [ 882 | "optimizer = torch.optim.AdamW(\n", 883 | " params=model.parameters(),\n", 884 | " lr=0.0002,\n", 885 | " betas=(0.9, 0.999),\n", 886 | " eps=1e-6,\n", 887 | ")" 888 | ] 889 | }, 890 | { 891 | "cell_type": "markdown", 892 | "id": "d37db1a8-0053-4acc-94ce-89d87c78942e", 893 | "metadata": {}, 894 | "source": [ 895 | "In this example, we build linear scheduler for training." 896 | ] 897 | }, 898 | { 899 | "cell_type": "code", 900 | "execution_count": 22, 901 | "id": "6f95bdf6-4498-4d40-90aa-1267d55f38c3", 902 | "metadata": {}, 903 | "outputs": [], 904 | "source": [ 905 | "from torch.optim.lr_scheduler import LambdaLR\n", 906 | "\n", 907 | "num_epochs = 2\n", 908 | "num_warmup_steps = 500\n", 909 | "\n", 910 | "num_update_steps = math.ceil(len(dataloader) / batch_size / gradient_accumulation_steps)\n", 911 | "def _get_linear_schedule(current_step):\n", 912 | " if current_step < num_warmup_steps:\n", 913 | " return float(current_step) / float(max(1, num_warmup_steps))\n", 914 | " return max(0.0, float(num_update_steps * num_epochs - current_step) / float(max(1, num_update_steps * num_epochs - num_warmup_steps)))\n", 915 | "scheduler = LambdaLR(optimizer, lr_lambda=_get_linear_schedule)" 916 | ] 917 | }, 918 | { 919 | "cell_type": "markdown", 920 | "id": "a9f9e828-c4fb-493d-a6de-78e03dbf035e", 921 | "metadata": {}, 922 | "source": [ 923 | "Run fine-tuning." 924 | ] 925 | }, 926 | { 927 | "cell_type": "code", 928 | "execution_count": 23, 929 | "id": "3752481d-8ee8-4c43-b677-add136a2fd5b", 930 | "metadata": {}, 931 | "outputs": [ 932 | { 933 | "name": "stdout", 934 | "output_type": "stream", 935 | "text": [ 936 | "Epoch 1 42/42 - loss: 1.3620\n", 937 | "Epoch 2 42/42 - loss: 1.4432\n" 938 | ] 939 | } 940 | ], 941 | "source": [ 942 | "from torch.nn import functional as F\n", 943 | "\n", 944 | "if os.path.exists(\"loss.txt\"):\n", 945 | " os.remove(\"loss.txt\")\n", 946 | "\n", 947 | "for epoch in range(num_epochs):\n", 948 | " optimizer.zero_grad()\n", 949 | " model.train()\n", 950 | " for i, (inputs, labels, masks) in enumerate(dataloader):\n", 951 | " with torch.set_grad_enabled(True):\n", 952 | " outputs = model(\n", 953 | " input_ids=inputs,\n", 954 | " attention_mask=masks,\n", 955 | " )\n", 956 | " loss = F.cross_entropy(outputs.logits.transpose(1,2), labels)\n", 957 | " loss.backward()\n", 958 | " if ((i + 1) % gradient_accumulation_steps == 0) or \\\n", 959 | " (i + 1 == len(dataloader)):\n", 960 | " optimizer.step()\n", 961 | " scheduler.step()\n", 962 | " optimizer.zero_grad()\n", 963 | "\n", 964 | " print(f\"Epoch {epoch+1} {math.ceil((i + 1) / batch_size / gradient_accumulation_steps)}/{num_update_steps} - loss: {loss.item() :2.4f}\", end=\"\\r\")\n", 965 | "\n", 966 | " # record loss\n", 967 | " with open(\"loss.txt\", \"a\") as f:\n", 968 | " f.write(str(loss.item()))\n", 969 | " f.write(\"\\n\")\n", 970 | " print(\"\")\n", 971 | "\n", 972 | "# save model\n", 973 | "torch.save(model.state_dict(), \"finetuned_gpt2.bin\")" 974 | ] 975 | }, 976 | { 977 | "cell_type": "markdown", 978 | "id": "83993d92-d7ed-4a07-8985-cc59bd4e4fef", 979 | "metadata": {}, 980 | "source": [ 981 | "> Note : Here we save LoRA-enabled model without any changes, but you can also merge the trained LoRA's parameters into the original model's weights." 982 | ] 983 | }, 984 | { 985 | "cell_type": "markdown", 986 | "id": "1bc086e5-e93f-4264-a8fa-6428f844ac3c", 987 | "metadata": {}, 988 | "source": [ 989 | "Show loss transition in plot." 990 | ] 991 | }, 992 | { 993 | "cell_type": "code", 994 | "execution_count": 24, 995 | "id": "e37c5aee-38d4-4a2a-952c-4fd2bef41e2b", 996 | "metadata": {}, 997 | "outputs": [ 998 | { 999 | "data": { 1000 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABV+0lEQVR4nO3deVhU9f4H8PewDagsgrIooLhvgLijlZoaGplW18praaVWXv2l1dWifTMs89puVrds0SjLpWsu4YJm4oaigIq7ILK4sYoIzPn9gYwzMNuZOTNnhnm/nmcemTPfc85nDjjnM99VIQiCACIiIiKZuMgdABERETk3JiNEREQkKyYjREREJCsmI0RERCQrJiNEREQkKyYjREREJCsmI0RERCQrJiNEREQkKze5AzCFSqXChQsX4O3tDYVCIXc4REREZAJBEFBWVoY2bdrAxUV//YdDJCMXLlxAWFiY3GEQERGRGXJzcxEaGqr3dYdIRry9vQHUvRkfHx+ZoyEiIiJTlJaWIiwsTH0f18chkpH6phkfHx8mI0RERA7GWBcLdmAlIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkx4GDOVXy36ywEQZA7FCIioibLomRkwYIFUCgUmDNnjt4yy5Ytg0Kh0Hp4enpaclqbue/zXXj99yxszCyQOxQiIqImy83cHfft24elS5ciKirKaFkfHx9kZ2ernxtbStjenLpYLncIRERETZZZNSPl5eWYNGkSvvrqK7Rs2dJoeYVCgeDgYPUjKCjInNMSERFRE2RWMjJz5kzEx8dj5MiRJpUvLy9Hu3btEBYWhnHjxiErK8tg+aqqKpSWlmo9rKlWJWDad/vxn+TjVj0PERERNSY6GUlKSsKBAweQmJhoUvmuXbvim2++wdq1a/Hjjz9CpVJh8ODBOH/+vN59EhMT4evrq36EhYWJDVOUHScuYvPRQny85YRVz0NERESNiUpGcnNzMXv2bCxfvtzkTqixsbGYPHkyevfujaFDh2LVqlVo3bo1li5dqnefhIQElJSUqB+5ubliwhStqlpl8HUOpiEiIrIeUR1Y09LSUFRUhD59+qi31dbWYseOHfj0009RVVUFV1dXg8dwd3dHTEwMTp48qbeMUqmEUqkUExoRERE5KFHJyIgRI5CRkaG17fHHH0e3bt3wwgsvGE1EgLrkJSMjA3fffbe4SImIiKhJEpWMeHt7o1evXlrbmjdvjoCAAPX2yZMno23btuo+JW+99RYGDRqETp06obi4GAsXLsS5c+cwbdo0id6CdVypuKH++XpNrYyREBERNW1mzzOiT05ODlxcbnVFuXr1KqZPn46CggK0bNkSffv2xa5du9CjRw+pTy2p69W3EpBaw11KiIiIyAIWJyMpKSkGny9evBiLFy+29DQ2xz6rREREtsG1aYiIiEhWTEYa2H36MhZuOoba2lt1IwLrSYiIiKxG8j4jju7hL3cDAKprmYAQERHZAmtG9Dh7qUL9swKOtbAfERGRI2EyQkRERLJiMmICBStGiIiIrIbJiAm4Ng0REZH1MBkBUKVjhlXmH0RERLbBZATA7KT0RtvYMkNERGQbTEaIiIhIVkxG9GAzDRERkW0wGSEiIiJZMRkhIiIiWTEZMYHmPCNp564gYdVhpJ27qjVLKxEREZmHa9OI9MCSVADAT3tzAQAn54+BmytzOiIiInPxLmoCQ5Oe1ajY1ZWIiMgSTEaIiIhIVmym0aOqRqX+WYCAN/+Xhay8UhkjIiIiapqcOhn5Yfc5fLvzjM7Xdhy/qPX827/P2iAiIiIi5+PUycirazLlDoGIiMjpsc+ICS4UX5c7BCIioiaLyYgJ/nfogtwhEBERNVlMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEYklHbuKnKvXJM7DCIiIofi1NPBS+lkURkeWLILAHB2QbzM0RARETkO1oxYKK+4EgCQyRV9iYiIzMJkxEIjFm1HVU2t3GEQERE5LCYjEiitrJE7BCIiIodlUTKyYMECKBQKzJkzx2C5lStXolu3bvD09ERkZCTWr19vyWmJiIioCTE7Gdm3bx+WLl2KqKgog+V27dqFiRMnYurUqTh48CDGjx+P8ePHIzMz09xT26Xr1beaat74PQs7jl+UMRoiIiLHYVYyUl5ejkmTJuGrr75Cy5YtDZb96KOPMHr0aMydOxfdu3fH22+/jT59+uDTTz81K2B79dLqDPXPy3adxeRv9soYDRERkeMwKxmZOXMm4uPjMXLkSKNlU1NTG5WLi4tDamqqOae2WypB7giIiIgck+h5RpKSknDgwAHs27fPpPIFBQUICgrS2hYUFISCggK9+1RVVaGqqkr9vLTUvofNHsotNrnsrpOX8Ob/juDd+yPRt53hWiUiIiJnIKpmJDc3F7Nnz8by5cvh6elprZiQmJgIX19f9SMsLMxq55LCtO/3m1z2n1/vQXZhGSZ+tduKERERETkOUclIWloaioqK0KdPH7i5ucHNzQ3bt2/Hxx9/DDc3N9TWNp5vIzg4GIWFhVrbCgsLERwcrPc8CQkJKCkpUT9yc3PFhGk3BEF/282NGpUNIyEiIrJfopKRESNGICMjA+np6epHv379MGnSJKSnp8PV1bXRPrGxsdiyZYvWtuTkZMTGxuo9j1KphI+Pj9bDEc1ccUDuEIiIiOyeqD4j3t7e6NWrl9a25s2bIyAgQL198uTJaNu2LRITEwEAs2fPxtChQ7Fo0SLEx8cjKSkJ+/fvx5dffinRW7Bf6zP094shIiKiOpLPwJqTk4P8/Hz188GDB2PFihX48ssvER0djV9//RVr1qxplNQQERGRc7J41d6UlBSDzwFgwoQJmDBhgqWnIiIioiaIa9MQERGRrJiMEBERkayYjBAREZGsmIwQERGRrJiMEBERkayYjBAREZGsmIwQERGRrJiMEBERkayYjBAREZGsmIwQERGRrJiMWNmpi+U4UVgmdxhERER2i8mIlY1YtB2jFu9A5Y3aRq+pVIIMEREREdkXJiM2UlJZ3Wjbu+uPyhAJERGRfWEyYiMCGteCfL3zDM5eqpAhGiIiIvvBZMRGhi5MwfepZxttf2vdEdsHQ0REZEeYjNjIjRoVXlub1Wh7LfuNEBGRk2MyIrPtxy/iP8nHIQhMSoiIyDkxGbEDH285gX1nr8odBhERkSyYjNiJKxU35A6BiIhIFkxGiIiISFZMRoiIiEhWTEbsBjuwEhGRc2IyQkRERLJiMmJnko8U4uMtJzjUl4iInIab3AFQPQUAYPr3+wEA0WF+GNqltZwBERER2QRrRuyGdk1IYel1meIgIiKyLSYjREREJCsmI0RERCQrJiNEREQkKyYjREREJCsmI0RERCQrJiNEREQkKyYjdmLG8gPYcfyi3GEQERHZHJMROyEIwORv9sodBhERkc2JSkaWLFmCqKgo+Pj4wMfHB7GxsdiwYYPe8suWLYNCodB6eHp6Whw0ERERNR2ipoMPDQ3FggUL0LlzZwiCgO+++w7jxo3DwYMH0bNnT537+Pj4IDs7W/1coVBYFjERERE1KaKSkbFjx2o9nz9/PpYsWYLdu3frTUYUCgWCg4PNj5CIiIiaNLP7jNTW1iIpKQkVFRWIjY3VW668vBzt2rVDWFgYxo0bh6ysLHNPSURERE2Q6FV7MzIyEBsbi+vXr6NFixZYvXo1evToobNs165d8c033yAqKgolJSX44IMPMHjwYGRlZSE0NFTvOaqqqlBVVaV+XlpaKjZMIiIichCia0a6du2K9PR07NmzBzNmzMCUKVNw5MgRnWVjY2MxefJk9O7dG0OHDsWqVavQunVrLF261OA5EhMT4evrq36EhYWJDZOIiIgchOhkxMPDA506dULfvn2RmJiI6OhofPTRRybt6+7ujpiYGJw8edJguYSEBJSUlKgfubm5YsMkIiIiB2HxPCMqlUqrScWQ2tpaZGRkICQkxGA5pVKpHj5c/yAiIqKmSVSfkYSEBIwZMwbh4eEoKyvDihUrkJKSgk2bNgEAJk+ejLZt2yIxMREA8NZbb2HQoEHo1KkTiouLsXDhQpw7dw7Tpk2T/p0QERGRQxKVjBQVFWHy5MnIz8+Hr68voqKisGnTJowaNQoAkJOTAxeXW5UtV69exfTp01FQUICWLVuib9++2LVrl94Or0REROR8FIIgCHIHYUxpaSl8fX1RUlIiaZNN+xf/kOxYUnv/H1F4sB877hIRkeMy9f7NtWmIiIhIVkxGiIiISFZMRhzMvrNXMPDdzdiYmS93KERERJJgMuJgHvtmLwpLq/D0jwfkDoWIiEgSTEY0LLg/EvfHtJU7DIOqa+2+vzEREZEootemaYqiQn2x8ulYKN1c8fCAcDxxWwRmLE9D7pVKuUNTO3y+GBl5JRDAZISIiJoWJiM3Kd1c1T/3auuLqLZ+dpWM3Pvp33KHQEREZBVspgGgMHmjbeVeuYbr1bVyh0FERGRVTEb0kDsXyThfgtvf34a7P/5L5kiIiIisi8mInfr90AUAwOmLFTJHQkREZF1MRvRQKOStGymprJb1/ERERLbCZAQAdCQecjfTEBEROQsmI3rIXDFCRETkNJw6GRnSKQAA8OigdjJHQkRE5Lycep6Rbx8bgLOXK9A5sEWj11gxQkREZBtOXTPi4eaCLkHeJnVWbdXCA6/Ed7dBVERERM7FqWtGTPX5pD7o394frb2VeOePo3KHQ0RE1KQ4dc2IIZq1JXdHhqC1txIA8Oo9PWweS+6Va3pfO5BzFeM/+xsHcq7aMCIiIiLpMBnRQ1/DzdTbIrBh9u14fEh7m8Vy+/vb9L72wJJdSM8txgNLdtksHiIiIikxGdGjT7uWel/rHuKDiFbNbRiNfoKg/S8REZGjYZ8RPSYOCIebiwL9I/x1vv5w/3C8tjbLxlERERE1PUxG9HB1UeDhAeF6X/dwY6USERGRFHhHJSIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZscCYXsFyh0BEROTwmIxYoIWS07QQERFZiskIERERyYrJiAVcXfQtp2cbc5IOynp+IiIiKTAZscDskZ0R6K2U7fxr0i/Idm4iIiKpMBmxQIivF/a8NKLR9o6tm8NN5loTIiIiRyEqGVmyZAmioqLg4+MDHx8fxMbGYsOGDQb3WblyJbp16wZPT09ERkZi/fr1FgVsbxQKBQ68Ogq7XrxTva2Fp7uMERERETkWUclIaGgoFixYgLS0NOzfvx933nknxo0bh6ysLJ3ld+3ahYkTJ2Lq1Kk4ePAgxo8fj/HjxyMzM1OS4O2Ff3MPtPHzwheP9EWvtj5Y/GC03CERERE5DIUgCIIlB/D398fChQsxderURq899NBDqKiowLp169TbBg0ahN69e+OLL74w+RylpaXw9fVFSUkJfHx8LAnXZp5Ytg9bjxXZ9JxnF8Tb9HxERESGmHr/NrvPSG1tLZKSklBRUYHY2FidZVJTUzFy5EitbXFxcUhNTTX3tA7jPw9G44XR3eQOg4iIyO6JnrUrIyMDsbGxuH79Olq0aIHVq1ejR48eOssWFBQgKChIa1tQUBAKCgoMnqOqqgpVVVXq56WlpWLDlJ1fMw/MGNYRNbUqLEo+Lnc4REREdkt0zUjXrl2Rnp6OPXv2YMaMGZgyZQqOHDkiaVCJiYnw9fVVP8LCwiQ9vi2FBzSTOwQiIiK7JjoZ8fDwQKdOndC3b18kJiYiOjoaH330kc6ywcHBKCws1NpWWFiI4GDDa7okJCSgpKRE/cjNzRUbJhERETkIi+cZUalUWk0qmmJjY7FlyxatbcnJyXr7mNRTKpXq4cP1DyIiImqaRPUZSUhIwJgxYxAeHo6ysjKsWLECKSkp2LRpEwBg8uTJaNu2LRITEwEAs2fPxtChQ7Fo0SLEx8cjKSkJ+/fvx5dffin9OyEiIiKHJCoZKSoqwuTJk5Gfnw9fX19ERUVh06ZNGDVqFAAgJycHLi63KlsGDx6MFStW4JVXXsFLL72Ezp07Y82aNejVq5e074KIiIgclsXzjNiCI84zUm9teh5mJ6Xb5Fxz47riX8M6QqHgVPRERCQ/q88zQvZn4aZsbD7aeKK1mloVampVMkRERERkHJMRKwtt6WXT8+Vdvab1vFYlYOjCFAxflAKVyu4rwYiIyAkxGbGyvu38bXq+hunG5fIq5BVXIvdKJcqu16BWJSAzrwS1TEyIiMhOMBlxIgIEJK4/ins+2Ym3/qd7cUMiIiJbYzLiZL7eeQYA8F3qOZkjISIiqsNkpIkxNDZq2a6zNouDiIjIVExGmqgzlyqw4/hFQGOU74ebT8gXEBERkR6iV+0lxzD8gxQAwNeT+8kbCBERkRGsGWnipn2/X+4QiIiIDGIyYkMDI/yxYvpAAEBshwCrnIMDdomIyNGwmcaGHhnUDoM7tsLRt0bD090FEQnr5Q6JiIhIdkxGZODl4Sp3CERERHaDzTQ2MLRLa/h6uWN4t0C5QyEiIrI7rBmxgWWP90eNSoC7q/Vzv5TsIkwcEGb18xAREUmFyYgNKBQKuLsqjBeUwF8nLmHWioM2ORcREZEU2EzTBG09ViSq/KasAjz4RSrON1jxl4iIyBaYjDixL7afwqXyKjz1Qxr2nr2ChFUZAIDPtp3EZ9tOyhwdERE5CzbTOLEFG45hfUa++nlJZTX2nrmChZuyAdQNRfb1cpcrPCIichKsGXFyh8+XaD1/cGmq+ueaWpVVzy0YWtWPiIicBpMRksXZSxUY8O4WfLXjtNyhEBGRzJiMkMUEQUDqqcsouVZt8j5vrzuCi2VVmL/+qBUjIyIiR8BkREY/TB0gdwhaGjbZfJ5yyqT9Vh3Iw8SvdmPMRztMPpeKTTRERHQTkxEZ3d65tdwhGPTfnWdMKlffCfZCyXVrhkMSu1hWhT8O56Payn2DiIiMYTJC5KTu/XQnZq44gC9MrAEjIrIWJiMkSk2tCj/vy8Hpi+XqbQrbTC5LEsu/WZOVfLRQ5kiIyNkxGSFRftqbgxd+y8Cdi7ZrbDWejeRcvoarFTesFxgRETksJiMkyv5zV9U/19eObDbyzTq/pBJ3LNyGmLeTJY1ldtJBTPtuH+crISJycExGZPbsyC5yh2A27doR/Q7llhgvJNKNGhXWpl/A5qNFyL1SKfnxiYjIdpiMyGz2yM5yh2BQXrF93ugFCDp/JiIix8NkhAySYkr4pt7Bdc/py5i78hCKr7FPDBGRObhQHhl1ubwK6bnFuHqtGvvPXjW+gwkUTShDeejL3QAAlQAsejBa5miIiBwPkxE70rddS6Sdk+ZmL6V7PtmpHgZqDl1px8WyKpP2zbpQgu92ncWzo7ogxNfL7BhsIedKhdwhEBE5JCYjdiSguYfcIehkSSKiT0aeaZ1a4z/eCQA4fbECv84YLHkcZMrAbCIi62KfEbJbZy/dqmk4XlgmYyRERGRNopKRxMRE9O/fH97e3ggMDMT48eORnZ1tcJ9ly5ZBoVBoPTw9PS0Kmpq+8qoaDPsgRf2c42Wsh9eWiOQmKhnZvn07Zs6cid27dyM5ORnV1dW46667UFFhuK3cx8cH+fn56se5c+csCpoci2ZnVc3aDkMKS22z6N7pi+XYfITToRMRyUlUn5GNGzdqPV+2bBkCAwORlpaGO+64Q+9+CoUCwcHB5kVIsvrfoQtm7ScIAgQBcHFRaPVJeHHVYSQ9GWvGAc0Kw6j6idt+mj4IsR0DrHMS0lJ6vRoqlQC/ZvbZR4qIbM+iPiMlJXWdEP39/Q2WKy8vR7t27RAWFoZx48YhKyvLktOSDX3w53GDr3+69YTO7Y99uw8jF29vtDz9tRu1ksUmpUwTO9QaomBXUKMEQUDUG3+i91vJuF5tn38LRGR7ZicjKpUKc+bMwZAhQ9CrVy+95bp27YpvvvkGa9euxY8//giVSoXBgwfj/PnzevepqqpCaWmp1sMZOGLbvb5kZfvxizh9sQKHzxdLMumZI14be/TLvlz8bmZtlxRqVLd+k9YYpUVEjsnsZGTmzJnIzMxEUlKSwXKxsbGYPHkyevfujaFDh2LVqlVo3bo1li5dqnefxMRE+Pr6qh9hYWHmhulQmsL3akEQ8Oh/96ifSzCBq57z3Pq5qkYl+Wib6loV1qbn2azvii0UlV3HvN8O45mfDkoysy4RkVTMSkZmzZqFdevWYdu2bQgNDRW1r7u7O2JiYnDy5Em9ZRISElBSUqJ+5ObmmhOmw3FpArOSnigqx18nLqmfq8xcUfdSg0nRyqtq8HnKSZwsKm9U9oElu3DX4h34M6vArHPp8uWO05idlI67Fu+Q7JhyK7teo/5Z87fi+H91ROToRCUjgiBg1qxZWL16NbZu3YqIiAjRJ6ytrUVGRgZCQkL0llEqlfDx8dF6OANPdxc8dUcHucOwSK1KO/lomIzkXLmGAzmGZ5m9Xl2rnmJd0/sbszHyP3UdTjXztvqb7K9p+pv+xNp6rAgAUFJZLdkx5XKpvErdoZiIyB6JSkZmzpyJH3/8EStWrIC3tzcKCgpQUFCAyspbK7tOnjwZCQkJ6udvvfUW/vzzT5w+fRoHDhzAI488gnPnzmHatGnSvYsmJOHu7gj3byZ3GGZreMNTqbQTh+Jr1bj/810Gj2HqVPG6ZBeUoahB04pKJeDnfTk4YeWJ00xdPTi/pBIvrc6wyURuKdlF6PfOZjyTlG71cxERmUtUMrJkyRKUlJRg2LBhCAkJUT9+/vlndZmcnBzk5+ern1+9ehXTp09H9+7dcffdd6O0tBS7du1Cjx49pHsXZDdMvSFbw+ajhYj7cAcGvLtFa/uqg3l44bcMjDLQ5FJfe2ALM5cfwIo9OYj/+C+rn+vzlFMAzB+iTXUEQUDCqsN6R48RkWVEzTNiyod1SkqK1vPFixdj8eLFooKipkXskFdTcgJdZVR69juUW2z0eEt3nMal8hsWrbpr6vvMulA3Oqy6Vr7Ejf1ExMnMK8VPe+v6rs26s7PM0RA1PVybxg7JWbtgqe3HL2o9Vyggy53vQnGl8UIN/HZAuj4n9s7Sv7Cy65b3pbFVTZQUKjknCpFVMRmxI4omMJrm/Y2G1yoyhRSX4bW1mWbtl19SiR92n3OqCbnEpgQfbj6OyDf+xLrD4pt+HP8vnIisgcmIHdL8wrj35RHyBSIRsTcgKb4wbz5aZNZ+Yz/5G6+uyVQ3pcil7Ho1fk07j5JrUo7mkaYm4sPNdf0mXlkjPuFznLoQIrIlJiN2LtDbsVc4FgT7vQHpqv24VG7+SB4pzfv1MP698hCe/jENgiDgasUNSY+vmfDJVVvRFGoCiUgaTEbs0LTb6uZvGdk9SOZImrYbFsxCqlIJeHVNJn7Zb50J+TZk1k3glnr6Mv7vp4OIeTsZu09fxrnLFZj+/X6jc7UQsDY9D3ct3o7TFxtPlEe6HT5fbFKHbyKpiRpNQ7YxZXB7DIgIQOegFnKHIgm5vv+m5xajd5ifVY6dfLQQP+w+p73RSm903eG6ofJLt59CYWkVjuSXIvlIIc4uiLfOCU1k7/1PZ9+cW2Xer4fx64zB8gYjQlHZdfh4usPT3VXUfv/deQYebi54dFA7s857vboW9376NwDg6Fuj4eUh7vxSKr52A+szChAfFQJfL3fZ4iDbYc2IHVIoFOjRxgfurvz16NJwlld9jK3Ea0nuUHxN6mYTAbtPXzbaTJR75ZrIA+vbbOeZhISkWCnaViN/zl2uwID5WzBi0XZR+10sq8Lb647g1TWZqKox7/1Walyn8qoaAyWt76kf0vDS6gzMSTooaxxkO7zb2YFmN7+BDOvaWufrbi6O27b+18mLWmvVSOGeT3aaOBeJ4UKW9Fmw5N6ka9etx4rw8Je7MWTBVvMPbANSDOk1RBAEfLzlhFkjdZqCLTc7XueJHJqu2f/J3musTLHnzBUAwLbsi0ZKUlPBZho7sH3ucBzJL8UdnVvpfH3OyM744M/jNo5KGku3nxZV/np1LQrLDK+UezS/FH9k5BssA+ifBM1aalUC5q48hAER/pjQT9xK0yk3P3SraiReTVcj35LiJvXSavOGTOuiK1ncf+4q/pNc97d+T1Qbyc5FRPaNNSN2oLW3EkO7tNb7TX3m8E42jsi2Pk+pW8E598o1DErcgglfpBrd598rDxktY2zFYHPqRQRB0DsHSdq5q1iZdh5zfz0s+ry2bjYxNzHZfKRQ4xjiD2LsmjdcrdlecOSP8yi5Vo2/T16CytbfZpwca0YcQFP/IHx/YzZGdAtC3If6144xhzU+S2b9dBB/HDZeKyOWKfd1s/4O+HkqySR65Dzu/vgv5BVXIvH+SEwcEC53OE6DNSMOJqC5h9whWMW7649KfkzjfUbEH9MaiYjsbHi3tsW06pl5Jdh7s89BvfKqGmw/fhHVFgznltO1GzWocdDYHU19f531JjQFk3SYjDiYFp5NszKr4Zo2UqhvpnGEb8amVGAYa3YSI+uCxkgjG/Z4XKTR98laNX73fLITDy7Vbup74tt9mPLNXny42fH6XpVcq0aP1zZJXnNoLkEQsO/sFZRUWrczMzkXJiMO4otH+qJTYAssmdRX7lAcRn0zTVMYXQDc6uQqip77/QNLjPfLsYY9DWosbEEQgL1n6877y37HWwwx9fRlAMCpixV6y9jyb3xt+gVM+CIV8R//ZbuTUpPXNL9mN0GjewVjdK9gucNwKIJQ9y0u84Lu+UYUzrBsm6DzRxLJ0nlGfk07D6WbC8ZGGx4hZC+/I0Mdqusn4Tt/VfzK2ET6sGaEmqz3Nh7DwHe34GBOsSznr6lV4VhBKRYnH8fwD1J0ri9zIOcqqmpqm0ztjRi6bvCaW9YdviDJmjxyN9NdLq/Cv1cewv/9dNBh+6wQWRuTEWrSihoMFS2prEZKdhFqalVWv0n9e+UhjP7wL3y05QTOXKrA0h11c65o3nDv/3wX/m+F48wyqfmNWVf+VFVTi9Ef7sCMmwv8WWLWioOY+NVui45hDzRnMxU73HxDRj6e/jHN6Dms+bcsCAKTKLI6JiPkVB78IhWPfbsPnV7egC+2n7Lqudaka88iqu9G9OeRQpwqMm8xN6NT45t5k7pYVoXRH+7At3+fEbXfO+uO4lhBGTZkFuCkjvdkLJyGrx8rKBN1/qZmxvIDVjt2VU0tZvyYhqS9OY1e02zCfPKHNPR5K5kdVsmqmIyQU8kuvHVz+3DzCRkj0Xa8yMSbrsbd+o/D+ej+6kZsyirQKnLucgWeWLYP+89eMbsTwkdbjuNYQRne/N+RBqc3nE40WjywAUdtjdI18mdTVgE+Tzlps3VrpPbzvlxsyCzAi6syDJZLPlKIsqqaRn9nRFJiMuKAHu4vbqpxsg+7Tl3Csz+n44aOKd/NuZ/NXHEAN2pVeOoH7Wr8p388gK3HivAPE2ay1ed6deMYq2pqtecJMRKzh5vhj5etx4rMCc0iUrZmPPVDGt7fmI19Z69KeFTbKWVNB9kRJiMOaP59kdg453a5wyCRMvNKsfpgns7XTP52bUKxvKu6V/bVewod3/p13bRve2+b8ZNrcDOy6vQ7f0g/0Z0umm+v4SWQYsrvi3Y6hb3kbl4qff1TBEHAy6szsHyP4doxfX7Zn4uhC7fh1EXzmizJsTEZcUCuLgp0C/aROwyncr26Vu+aNFKTo9q/vKoGqacuq2/Oum44Ym+6rjYexpJXXInfDzVe7Vff5czMK0H0m3/im53i+sXITRAEPP/LIbyl0YRmDy1FO05cwvI9OXjZzMUU5/16GOcuX0PCb4abjZqqjZn52GGFyR8dBecZITKiViWg1+ubJJ0B1ZAxH1k2mZRWlCbmAw8tTUXWhVK8PrYHHh8S4ZBzsAxZsFXn9qwLpTq3v7jqMMqqavDWuiOYOCAcZVXVCPT21FnWnvqFnLlUgd8O1E3e1qOND4qv3cBdPaSdg0jXPCPGFnOUqtmnyglH7hSVXsfTP9Z1Vj67IF7maOTBmhEiI0oqq1GjEqyy8F49zXudpCNITIlZENQ37PpmJCkqNaSqGKmqsX6N1ID5mzFg/hYUll63+rmuV9fqbR7S3KqvJqpGY99/rzyEd/44ijOX9c/OSvbvyjXL5tOprlXhnXVHsC3b9v2wpMJkhMiIPm8nyx3CLY5XYWExo8OXzaSZAJbdnAuk4QJ7lh1fwLnLFTiaf6tm5mpFNbq9uhEPfWm8c/GGTN0LtemqpLFFZ1Q7qhyymp/3NR7m7Ah+3peLr3eewePf7pM7FLMxGXFgz4zoLHcIZIcqNCbZMpckNSOWH0Jytozp/U3ZGLowRV39DgDJR+qGx5oyAkfMzd+earLsxeXyKixOPo7cK7o7dDd0/uo1vCBTfxXNZlFzmgTrVxp2ZExGHFi4fzO5QyCJlFmYQGTmleBYQSl+TTtvUXPSrc9B43em+qKHcovx8uoMXGkwdbtUX6SzLpRi7Cc7sfPEJYmOWEfXZ76pN2RTRnwsSRE3qd7FsipJOjDuOnUJqacui97P3ms+Vh88j10nTf8bmPNzOj7acgITTBzi3nC2ZltqaomgOdiB1YH5ebnLHQLZmp4bxj2f7JT0NGI+HMd99jcAWG2Gzie+3Yeyqho88t89FnfuM/d+23DSs3slvt4AMGzhNlTcMN4/xlhH0ieW7QcAHHt7NDzdXc2KRdYOzDqyouyCMjz78yEApnfwrE/ICkzsB/T0D8an3TfFq2sycaygFD9NH2R0eLsuguCcyQlrRhzYnd0CMWlguNxhkIPQu76IxiffpfK6b4ean4W/7MvFtO/2Gz2+runfG53KwGuVN2rxy/7cRtstrTXSx5KKAFOSBl3Sc3WvIG3JMQHdtRpVOiau0+efX+1Wj9DRew6xQUnoQon1myHE1oycuVSBB5bswpajhVrbf9h9DvvOXsUuEbVTmv8vzLnOlRb87dgLJiMOzMVFgfn3RcodBjkIU2pP8kuuN+pzMu+3w9jc4APXGuavP4Jt2fY5z4KhdnyVIODBpamYYcKCdsZu+Kaes1FZndtM33/Xqcs2m4TOKImqBTQP8/raTPx1Qtq/rTk/pyPt3FVM1ZOo14r4/Vn6ls8aGE2VmVeC19Zm4nJ542SrpLLaah3ExWIyQuRIrFB9eyi3WOv5+auVJn04lhupsRCEug9CzW9thj72ko9YP+ExxpzmiRNF5dh75go2ZMq3dostavXr76322IKw+UihwZFQ36Wew6P/3SvpOa826CN1vboW5/XMfiyG1HPa3PPJTnyfeg6vrNGejO781WuIfvNPjPtM+iZHczAZaYKae5jXTkxNz96z4oeqjvtsJy6XmzbvQZdXNuh9bW16Hu75ZKfWMFZ7mjxMMnb6ngQBKLtejVfXZGKfyL8DMbUqctD8O7pQXIlp3+/Hg0u1O6pau99LToNROnct3iF6yYRbNEbTWBCTIdkN5i/akFGXPGfm6Z4U0NaYjBCRluvVKpO/5eta9K9eff+Pw+dv9ZPQtfqtLWlOJGaNxGhtuu61h3TJL6nE+xuP4YLIYZm6wtb3Tj7YlI0fdp8zeUSJIdW1KqzYk9PoJmxrxwpK0X/+ZvUK0Y/+d4+s8dSzxXX5bNtJ/JZmelOfI2EyQmQF9tIOa0uGZo4tNmGGSVO+yapUQqMk4sylChzIkWblXHNyJc1oZielm7zf49/uw+cpp/TeTPX9BWXk6e8E23D/05f09yUQmxh+ueM0XlqdIe0MwboYSRJf+C0Dl8pv4NWbzQ6nLpo++6y11pcyNNT7hV8PY/4fRyAIAkquGR9xpu/tHysoxcJN2Xh+5SHzgrz5687MK0FBifVnGhaLQ3uJrOCnvVaaydFBc5zebyXj7xfvtPg4d3/8F1o288BPTw5Sbxv+QYrFx5VD/U1d3830WL7um/68Xw832pZ6SvwcLGJrhuScWEtrtIkFNVrdXt2Iz/7ZB/FRIRbF89m2k1rP9Q0LPne5Aj/frCE8c+kaNh8txNqZQxAd5qdVTntlad3vr2Ezi9b+JsQM1I14q+/I/tLd3UzcyzZE1YwkJiaif//+8Pb2RmBgIMaPH4/s7Gyj+61cuRLdunWDp6cnIiMjsX79erMDpsY2zrld67ncVeEEp159U5/kLMs7eB4rKEPqafETepmq/n9OrUrAwZyrBpuh6lWZUMYcP+sY5qzPT3sbl7Vl/5wLxZVYuMn4vcBcUr6TmSsOGC9kRMP3qm9NI83h9PUj0pbtOtuonOYn9td/3VpFeknKKWzIqFsWwFCtm77P/MPni7XOkd6gs7o9EZWMbN++HTNnzsTu3buRnJyM6upq3HXXXaio0F9NtmvXLkycOBFTp07FwYMHMX78eIwfPx6ZmeYtM02NdQv2kTsEamCniJkixbDWnBvW0PAb/5r0C1rrtFjCkhutvkXqNC36Mxv3fb4Lc3+tqxI3lOB/ueO01vOSymr829yqdIk1vEymvHdzPPbtXun6TBj5MqXZB8ng34EDfCe7dqMG244V4YZG0rJwUzYqb9Qi7dxVvLfxGGYsN5486bsO9376t/pne/+SKioZ2bhxIx577DH07NkT0dHRWLZsGXJycpCWpn98/UcffYTRo0dj7ty56N69O95++2306dMHn376qcXBE9mra01gEiKp6ftW9tPeHCz7+4zO1/Sx5Et/h5fW472NxwyW+WJ73VTua9Mv3Dyf6SdcuOkYfrVSJ8NrN8xPRr/+6zR6v/UnjhUYTwjFjkQ5XqjdZyLnsm06uW45qr1KrWafEL2T/NmRZ35Kx+PL9uGN37O0ttcKgt5Vm+v9589sPPrfPZK8TymGJFvKog6sJSV1Gaq/v7/eMqmpqRg5cqTWtri4OKSm6u/dXVVVhdLSUq0Hma65kkN7yTFU1dQiYVUG3vjfkUZr2xhSY+E3fH3rxtR/eTR09MQNhicHy70ivm+FqR0re7y2yeRjVjY45jt/HEXp9Rp1x08pvinr68dwx8JtOGOg86whlQ0SrrLr1fjfoQs6y/55RLvp783/HVH/bCx/FATBpJqiazdq8Pwvh7DZwDw4pddNTxI1r3p9083u042HXhv79Xy89ST+OnHJYFymMn9IsnTMTkZUKhXmzJmDIUOGoFevXnrLFRQUICgoSGtbUFAQCgr0tx8nJibC19dX/QgLCzM3TKfx+6wh6p9nj+iCmHA/+YIhMpHmqKMbIr7hPfK15cM5Dd2sDL22dPtp/S/CvP4Nl3TMjmmp297bhrLr+kdvGKrtMXWekbgPd2DrMd03w91m9u2pqdU+94D5W7DqgO4h0ycaLEGw0sR+Nu+sO4KIhPXo8NJ6g4ng2vQ89HhtE347cB7Tvje+JIK1PPat/gnbxPy/sed5fsxORmbOnInMzEwkJSVJGQ8AICEhASUlJepHbq7pHbmcVVSon/pn/+buWP2vIfoLE9kJcyemMmcyN2d06Lxpw4AtUd+UZS0Na3g0HcwpNuuYX++81SxoqEO0mKHaJjPxT16zWIoZyyQ0rPW5Xl0r64goY8wa2jtr1iysW7cOO3bsQGhoqMGywcHBKCzUzpwLCwsRHBysdx+lUgmlUmlOaKShrZ+XXf/xEcnZp85as4za87dPOWRdKMGx/DLc36etaU1DFvxNNKUrb2kz2nO/pGs9P3+1Eh9uPmHRMa1JVM2IIAiYNWsWVq9eja1btyIiIsLoPrGxsdiyZYvWtuTkZMTGxoqLlER7fEh7uUMgspqqmlos2GC4I6pYr/+ehad+sKw6Pu2c+AnYfkszfeZWS+07exX/3Smuw7Al4j/eiedXHsJ2Jxvubss8W1fissZIjZW95cyikpGZM2fixx9/xIoVK+Dt7Y2CggIUFBSgsvLWt+/JkycjISFB/Xz27NnYuHEjFi1ahGPHjuGNN97A/v37MWvWLOneBenk4cYJdsm+WfLl7787z6hHvZhD14dxYWkVNmVZ1iHQnJFUizcft+icYr297gj2GFhYTgoNf7WGJu0ytJ+1HTx3Ffd+Kv1icZbMwiwIAqab2UdldtJBbDtWZLygnRF1t1qyZAlKSkowbNgwhISEqB8///yzukxOTg7y8/PVzwcPHowVK1bgyy+/RHR0NH799VesWbPGYKdXkka7gOZyh0BkNe9vtN4kW87A1k24pt6aLWmeMGfPj7ee1Jq7RCor9+se3n32UgX2GOnca8mU+2vTL+DxZfuMlrO3aUdE9RkxpS00JSWl0bYJEyZgwoQJYk5FErijcyu5QyCyWw1HYuhjj+t4SEGqe5Gp1f22aBawp5aH05d0/30dyCnGQ1/uxubnhurdt+GIImfAevwmyOVmyqtQKDCye5CR0kTyMTaxkz0YlLjF5FWMHYlUM3L+rmcOEDl9ssV6HTXFrbKs/xqfKLTygoNGOHSfEbJvEweEoWcbHwzrGqix1c7+4og0jNOYrtqe6VpPxNEZSkVOFZk3YZkh1hq9pMuiZOv1wTFlenbA/GHrAEyaJbep4aq9TUji/VGNttlb9kuk6bKIWVfJdiZ+tRufTIxBewv6nVXcqMW2bNM6UpqyIKG9yMozrX9JXfKl/wO43MA6U5ozyToLJiNNXL/2/tjigD2rici6jH1P+b+fDlp0/LfXad9QDfW90Zzu3dLGo6wL1pvoraqm1uSlCIx9EZz762Gz4xjz0V9m72uv2EzTxE29LQJvj+fIJSLSZuvBFN+nntO5ffvxi1r9TkztWKxLrUrAP5boX/fMUrN/Sje5bF2XHOtcZalWv7YnTEaaOA83Fzw6qJ3O1zLeuAtvjO1h44iIyB7INbSzpLIa93zyF5aknIJKJWDKN3slnRDN0PTxltqYZXpH5uOF5RatsuxsmIw4MW9Pdzw2xPgsukTU9FjSwdIS3/59Bpl5pXhv4zG77l4vRT+Wj7eclCAS4+xsyhCzMBkhIiKbqdK4yc9deUjGSAyTYobac5elH5XUkCAA2TIPE5YCkxEiIrIZzY6dqw7abk0eOdii5ufjrSdw/qr42XTtrVaKo2mIiMgm7v/8bxzIKZY7DJs5aUFnXFOdvmhe7ctSC9Z1sgbWjBAROSE5OrA6UyJi765eq9Z6bsnCflJgMkJITbhT7hCIyMYsWYyNmp4lKbbpbKsPkxEnE9nWFwAQHxmi3hbk7SlXOEREZAc++NN6U+ibgn1GnMzgjgFY9nh/+Df3UG+zt6WkiYjIuTAZcUIBLZRyh0BERKTGZhoiIiKSFZMRJ2NvY8uJiIiYjDgZwdhSkjctfijaypEQERHVYTJCOt3ZLUjuEIiIyEkwGSEoFAqsmDZQ/fypoR3g6+UuY0RERORMOJqGAACDO7XC8XfGICOvBL3D/AAAHm4ukqxcSUREZAhrRpyMoS4jHm4u6NuuJVxd6iYeWfd/t9koKiIicmZMRkivLkHecodAREROgMmIk5gxrCNaeyvx5NAOcodCRESkhX1GnMQLo7thXlxXKDj3OxER2RnWjDgRJiJERGSPmIwQERGRrJiMEBERkayYjBARERGyC8pkOzeTEbLI15P7cbZWIqIm4FhBqWznZjJCJhvTKxifTIzR2jayRxDSXxtldN//TulnrbCIiMjBMRkhg5ZPG4hgH098+1h/LHmkL8ZGt2lUxpRROmH+zawRHhERNQGcZ4QMGtKpFXa/NELna92COUMrERFZjjUjZDYXHTUiXTmFPBGRQzK0dpm1iU5GduzYgbFjx6JNmzZQKBRYs2aNwfIpKSlQKBSNHgUFBebGTHbskdh2jbY183CVIRIiInIUopORiooKREdH47PPPhO1X3Z2NvLz89WPwMBAsacmR6Ajtd7272G2j4OIiByG6D4jY8aMwZgxY0SfKDAwEH5+fqL3I8cX5OMpdwhERGTHbNZnpHfv3ggJCcGoUaPw999/GyxbVVWF0tJSrQc5tiAfT4zr3XgkDhER2QcB8nUasXoyEhISgi+++AK//fYbfvvtN4SFhWHYsGE4cOCA3n0SExPh6+urfoSFhVk7TLKBOSO7yB0CERHZIasP7e3atSu6du2qfj548GCcOnUKixcvxg8//KBzn4SEBDz33HPq56WlpUxI7JCHm7hcNqJVcytFQkREjkyWob0DBgzAyZMn9b6uVCrh4+Oj9SD78cGEaIT7N8PCf0SJ3te/uYcVIiIiIkcmSzKSnp6OkJAQOU5NEvhH31DsmDccnXXMKdKwxTHpyUFaz+Mj637v+uYjeebOTugd5idFmEREJIKc84yIbqYpLy/XqtU4c+YM0tPT4e/vj/DwcCQkJCAvLw/ff/89AODDDz9EREQEevbsievXr+Prr7/G1q1b8eeff0r3LshuDeoQoPX85fju6B3mh2FdW6PvO5sblQ/08cQ/B4YjPbfYRhESEZHcRCcj+/fvx/Dhw9XP6/t2TJkyBcuWLUN+fj5ycnLUr9+4cQPPP/888vLy0KxZM0RFRWHz5s1axyDn4enuigf6hmptC/H1RH7JdQCNa1aIiMg2XF2MrzNmLaKTkWHDhkEwUJezbNkyrefz5s3DvHnzRAdGjkkBwMvdFZXVtSZ3WJ11Zye8vDoTABDorUT7AHZ0JSKytahQP9nOzbVpSBKTY9uhXUAz3NcnFGtmDsH43m3w7WP9Td5/6aN98a9hHXFXjyB0DfbGbzNi1a9NGhgOAHj3vkjJ4yYiojry1Ytw1V6SyFvjekEQBCgUCnQN9saHD8eI2j+uZzDiegarn/dt56/+OT4yBK+P7QkPNxe8tDpDspiJiMg+MBkhySh0rOJriccGt8eJojIM7BAga1smEZEzkPgjXBQmI2S33ri3p9whEBGRDbDPCDV5Ib5cqI+IyBiFjL1GWDNCDmlKbDt0CfZWj8Kpd3L+GBw6X4yFm7Lx+tieaNvSC95KN6RkX8Tjy/bJFC0RERnCZIQckkKhaJTFvz2uJ9xcXdC3nT+SnoxtsIMNgyMiIlHYTEMOKbKtr9whEBE1KYKM004yGSHZ+Xq5m1z2z2fvQOL9kbgvpm2jnt+cvZWIyHxyrk3DZIRk8/4DUXiwXyjG9DJ90cQuQd6YOCAcLi4KDIzwN76DSHtfGiH5MYmIyDD2GSHZPNg/DA/2DzN7/w6tW5hc1tQuI4E+nji7IB4l16ox6b+7kZlXal5wRERkMtaMEOng28zd6DC3f96cpp6IqCmQs6mbyQg1GYbaO6WeHRYA/jlAOxk59vZoyc9BROQMmIwQ6WEsf9Gcov7IW3HwdHe1ckRERNYjyNiDlckIkQlevadHo20uGtmKpxsTESJybGymIbJDT97RAQBwV48g3B0Z3Oh1Vxn/99wTZfoIJCIiU3BoL5GZtv17mPpnQ1WMxnqMJD97B9JeGam17Z6oNvhr3nAseaSvzn28PG4NRrP2/+HT796t9VyzD0y/di2tfHYiIutiMkIOLaJVc0mO0znIGwEtlI22h/k3g6uLAl46+oO09fPCrOGd8MLoblr9RwDgkUHSjrRxaXD8kd0D1T93FDHE2VHFR7ImiMj62GeEyKY+mRiDh0XMceLXzANvjevZaPu/47pixrCOjbaP6tG4WUdTt2Bvk8/d0Btje+De6DZYPm0g0l4ZiUCfxkkUEZEjYTJCTqHhyJix0W2QcHd3PDqoHX6bEat7pwYmx7bH6J6Gk4wDr47CzheGY2iX1vhkYgw+nhiD/u1b4qvJ/Uw6xy9PGY8lMtQPCoUCQzq1QkALJR4b3F792sQB5k8iZy1h/l6WH4QLHRI1aZyBlZoM32b617jp2ebWwnrJz95RV97LHW+P7yVpDP7NPeDf3ANAXcIDAPfe/NcUbfw8G217YkhEo3NoUmo0IVljPhVLtA9ohpS5w9H+xT8sOk5MmB/+OJwvUVREpIucHViZjJDDe/+BKOw+cxljo/Tf9P2be2DvyyPg5e4Kb0/TF+azlp5tfHGsoMxouXuiQvDa2LphxUsf7YsrFTca9ZNx1UhAPGw8xMfVRYFalXU/wVwUwJTB7fHOH0eteh4iZyfn0F4mI+TwTF3jJtC7ca2DWD5e0vyXeW1sDwS08MC6QxdwoeS6zjJdg7zxycQY9fM4PU1EXh6ueGZEZ9yoUSGuZxCW7TorSYzjerfB2vQLBsto1sOM690GSjcX/LL/vMF97okKwaAOAXhlTabJsbi7uqC1txIXy6pM3oeIxGHNCJGDmDe6G85evmZx3wxfL3e8dHd3hLb0wmtrs3SWae2tNLnZ5blRXdQ/r5k5BG39vNB//maT41n1r8G4//NdWtsW/iPaaDKiSQGgVYMRSe46amo+/WcfqFQCqmpUcHNR4LNtJ1FkQpIh5wclEVkXkxEiEVq1UJrUydRUkwa2QzMPN/x75SHJjtk7zE/0Ph10DJH2cBPf5PPU0I74POWU+vlHD8foLOfiosDU2+r6wkwZ3B4qlYAOL603cnR5spEhnQLw98nLFh+nQ+vmOH2xQtQ+Hm4uuFGjsvjcRKYQOLSXyDm5uihwX0xb9XPNmoXQlhKMQjGRIAC/zxqi9/UB7f11bg/y0W768vW61R8ntKUXerTxMen8DedRAW4lSEO7tFbHWK9dQDOTjisFY6s3m2rRhGjR+8i5Vgg5n9CWtvt/1RCTESKZuboocODVUdj38kh4urtixfSB+EffUCSM6W7Rccf3NjyKJ0Vj9loAiAr1Q9cg3fOf6Osr8+3j/dU/S33bXDF9EF4c0w3/ebB3o+NbMk+LXDRHdJkqTMabA8nHx9P2jRa3d26FFkr5GkuYjBDZAf/mHmjtXVcrMrhjK3wwIdrgUGVTGEsOmnmIWdxPd+1AFz3JixSCfT3x9NCOaNlgKDMAvHtfJCbHttO5X1SoL1b/a7DOWXMdzaIHxdemkOM7/Eac1vxBttBaxwzUtsRkhKiJEvPhoi9x6R3mB093FwzqoLuZRpOyQR8TS1oYdE3/rtlkEdBCibfG6Z4jxsPVBTHhLdGrrWlNRLrsThiBkd0D8ePUgaL2+/LRvnjtnh749rH+jV4zZwqYNn7SN9WF+9tHbcvtnVvJHQIAYFSPIFnPr2tmZwDoHNT0l3nQxGSEqIl6ZmRnjOkVrLNzKgCdlR1xveqGD9f3V1k1YzAOvx6HZh76q2/fvLcnugZ54/m7ulocc73xGv1o6pk6nUlUqB8AYNGE3hjQ3h/fPHZr9ltTq7+DfT3x9ZT+uE3EDfP+mLa4q2cwnrgtAsO7BWq9tq1Bk5iprNFlxJSJ/sRM1Gcr3lZqQvAUUYP23ymmzaQsRkBz29VI2POimkxGiJooH093LHmkL/458Naifa/Ed0eIr6fezpSzhnfCp/+MwZqZdZ1ZXVwURkfVTBncHpuevaNRZ1axNIdLt9VRIyC2M2d4QDP88nQs7ux265uvNWeoXWigg6q5CzpaY3RDWx2z/DbU1QZ9clQif58Jd1vWh0oKI7pLX4ui73csRcfphrVgzWXsE2KM/UZGRJJ4ZFA7ZOaV4LbOrfGPvqGYdnsHAEBR2a3J1upv9B5uLrhHx0y2mvfwIB8lCkuln3zs3fsiEduxFaprVDpH4dj7uJKGKzcHeitNmj/F4DF1JE/dQ3ygAHAkv9SiY9tKv3Ytsf/c1UbbnXGg0Cvx3fHzvlycKCpXb7PmdWjZzB0tlD7qvxVx/cRsizUjRDZmrepmfTzdXfHhwzH4R9/QRtvrmTOniBjz76trGjDURKBQKHBvdBs80CBONRM/tHV906wfctzFwnb40JZe6qHGxsSE+1l0LgAI9PHEQ/3CMDDiVp+d9c/cBjdX8781d2hlH30RxNaMiK3U2vr8UBx8dZS4nUzQPcT8vkjTbu+A5OeGam1r+L5eiZewBkih0Dr+K/f0QPcQH7vpr6OJyQiRjSyfNhA9QnywfLq4TpHW4uPpjvceiMR7D0QaXa/H0grjSQPbIfPNODw6SPcIGFO019HUMbyraYnB3pdHIOvNOHgZ6Ptiiu1zh2PZ4407p+pi6Tfe+k7D7/0jCt89MQBAXSdhhUKBiQPCDe1qkIuLAn4WjtSSgrVrRjq0bqFzJJY16JqRuVNgC70TEN6no09UPa+btRfGkq/3HojEy0aarhoeoq2fFzbMvh0P9tMxg7TMa2yKTkZ27NiBsWPHok2bNlAoFFizZo3RfVJSUtCnTx8olUp06tQJy5YtMyNUIsc2pFMrrJ99u7qDpT14qH84Hupv/o1Nl/pOchP6addwWDqHweeT+iA+MkTdnwWo66fRcFVjXZRurmiudDPr8/aBvnU3ju4hPnB1UUjS70TfcgIn5o9BasKdeH5UF3z2zz7q7Z7ursh6Mw6HXr8LAPBw/zDcZWQUSKsWjW/EuxNGAAAOvjoKD/QJxWv39NC5r5uOSejMMWlguN6b6rjejW/I80ZL1wnaWhq+naQnB2H++MhG5VwVCqyZOQT360g8NJtLGvYNuT9GT81gAw/2C8P0OzqYVNYRiE5GKioqEB0djc8++8yk8mfOnEF8fDyGDx+O9PR0zJkzB9OmTcOmTZtEB0tE8tCsOZlyc/6DYXpqJZY9MQDfPzEAs4Z3kjSGMP9m+GxSH61vm61aKNWrGlvL+N5t8fusIfhthnTLAGh29n1eY10hd1cXhPh64f9GdEZAg6HZzZVu6qY1hUKBqFDDk6hNaPDt18vdFcG+nur9Fz0YjSdu053IGZrhNtqE5Qay3xmNpCcH4fWxPTFST6fPh/uH4ecnB2HnC8MBACG+nph2Wwck3h+J32YMxoMNktmGNSktlG6NJu4D6ubs0RxBZW2DOgTonEHYEM0ETbNZceptEeqakb5GRr5I3RnbxYqdu00h+qvKmDFjMGbMGJPLf/HFF4iIiMCiRYsAAN27d8fOnTuxePFixMXFiT09EckgrmcQxka3Qd9wP0yObY9BHQLQQ0/beQulG+4wsV+FrT0+pD22H7+IoV1aY/vxiybtU3fj97NaTP83ojMEaE+lbw1i7jUNV4juHeaH9NziuuOYsL/SzRWDOgQAqLvBhvs3g0KhwNM/pgGomx3YxUWBgTfLZLxxF5RurvBwc1E3Qf2ZVaA+3scTY1B+vUbrHIIgNGq6e2F0Nzw9tINVR02ZytvAMHJTRspYc0LBhtr6eeH5u7oYL2hFVu8zkpqaipEjR2pti4uLQ2pqqt59qqqqUFpaqvUgIvm4ubrgk4kxeGxIBFxcFOgT3lLU/Ay2ZKgvwrCugdj14p34RsekZFIz9M22YYzPjOisrnEyldg+F2Juz5o38/+7sxMWPBAJ/+YeeMOMWig3VxeMiQzB6F7B+PXpWGyYfTsWP9Rbq4y3p7vBTtT3RrdplEw1HGb79vhemDGso0WJiGbTmKUMDaMd0ulWB1JTf49xPcUPKzb1Uux8YThCfG23FpYuVk9GCgoKEBSkfRGDgoJQWlqKyspKnfskJibC19dX/QgLs2y5diKiem38vODqolB3CrWWJ26LwNvje2HL80ONF7aB5dMHmbXf6F7B6Bbsg7RXRuKxIRFmzSRbr197/7qhySYcxNg9un6EVr0wEQtL6uu/FB8Vggn6RnOZS9fkgnoSi0Bv7aY5D9dbt+gvHumL50ZJX3vxYL9Qu6hJssvRNAkJCSgpKVE/cnNz5Q6JiOyYOf0th3ZpjV+fvtUPZMH9jTshmirMv/GN0N3VBY8OaoeOrVvcjPFWkD5WbpJp6Pg7Y/SO7DBV/Q3rqTs6am3/8KHemDGso3qtoOm3G+9QLIX6fkw/TR+E1+7pYXTIdf1aL6N6BOGHqQPQsXVznSOj+rW3/iylmjd/zb4/DWvHNj17h9Y+YtdbMuW/xXsPRIk6prVYfcKD4OBgFBYWam0rLCyEj48PvLx0Z7JKpRJKpbyL9hCR40h+bihGLNpu0THMGWnaI8QHM4Z1RGzHAKNlXV0U+Pbx/rh+oxY3alVmnE2bmC+zUs4jM7qXdn+S+qn7a1UCHuwXZtE8HJpMfXuxHQNMuv6TBoZjcmw7hPs3g5urC7Y8PwwA8NuBPK1y43q3xW8H8tA7zA9HLpRi58lLeo+pmQRHh/nh0M1+NaZY/a/BqKiqRbuA5tj5wnC4u7o0avqMaNUcXzzSx+ByDJayh1oRwAY1I7GxsdiyZYvWtuTkZMTGStcznYicW33tg625u7lgbHQbtDJxUcLhXQMxRscigJba9eKdkh+znq7OlkE+jd+vq4sCvdr6NpqJ1lwNk0NzO0XPGt4JD/cPQ6fAFujQugXcXA3f9jzdXfHLU7F46e7uaN+q8agifUnrNyLXrYkJb6le+yi0ZTO9yymM7hVitx3CpSQ6GSkvL0d6ejrS09MB1A3dTU9PR05ODoC6JpbJkyeryz/99NM4ffo05s2bh2PHjuHzzz/HL7/8gmeffVaad0BEpEHsGjb1xDRj1H+ZHGBmlb7UE3618fPSu/prU6FrvSJT/DuuKxY8EGX1GoCGQ7HlYmhCNXsmOhnZv38/YmJiEBMTAwB47rnnEBMTg9deew0AkJ+fr05MACAiIgJ//PEHkpOTER0djUWLFuHrr7/msF4isivdQ3yw+l+DTapl2Pb8MLwS3x3PjZJvkq5+7f21nk+ObY/vNTrlPty/ruO/pd+q5arFb9iZ01rcDdTkaNYKLZ828Oa2WyyZCdcc9XP76BoG/kp8d6ydOURrpJKiwXTw9kx0Q9SwYcMMfvPQNbvqsGHDcPDgQbGnIiISzZIp32PCTavpaN+quXrBQbkM6hCAH6YOQPuAW3NtBGjMutouoDmy3oyz68XRDHk0th1OFJZjeLdAq57n33Fdsf/cVaNLFWgOx633qp4ZbDVJsfpuvc5B3tgxd7jW73nB/ZG4VF6l9+8xrGUzZObZ//QYXLWXiJqEt8f1xJr0C5gxtKPxwk3E7Z0N13qYs2T8m/dqN/fI9c1a6eaK9/5h/ZEebfy8sGPecLP2NTTXjtJKi0+GN5gd92EDtTMKAG+N6wU3VxdMGmjbWhyx7HJoLxGRWI/GtsdvMwbD1w4WgZOLFN/CGw4vVbrZT82KPSzw99LNxemm6plKv561lykwVWtvJT6ZGKOeEddesWaEiMjG7LUd/58a356fH9UFV69VI0LHasly+XHqQLzxexZeGNPNJufrFtJ4SvbbOrdC5ptxRhd+DG2pf30fW7HXvzNdmIwQEdmY1KNppLDg/kg81P/WbNf/N6KzjNHo1qutL36dMdhm53u4fzgqb9Q2qlWwdAVqW9G3fpRmPyN74RhXlIjICtqKmEK8qWvZ3MNuJsCyF64uCos7KstxSf945jasz8jHjGG6V86ODPXFRw/3tovam3pMRojIaYX4emHF9IHw8ZS/L4JcnhnRGYdyizHCyqNWnMlgE2aEtaaebXzRs42vwTLjetvXfCRMRojIqQ3u2HjIpqMK1bFGjjHWWHzNWd0b3Qa/H7qAGcNujegK1jOzKmljMkJE1ET4eLpj5wvDJV2Lhkz30cO98frYHlqzsc4Y1hF5xZW42wrLADQlTEaIiJoQe+oH4GwUCkWjaeGbK920ZkUl3Zg+ExHZWP3oDF3TejuC4V3r+pewCYKkwpoRIiIbC/b1xL6XR8Lb0zE/gl+9pwd6tvHBqB7BcodCTYRj/k8gInJwrW20EJw1NFe64dHY9nKHQU0Im2mIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVg6xaq8gCACA0tJSmSMhIiIiU9Xft+vv4/o4RDJSVlYGAAgLC5M5EiIiIhKrrKwMvr6+el9XCMbSFTugUqlw4cIFeHt7Q6FQSHbc0tJShIWFITc3Fz4+PpId19nxuloHr6t18LpKj9fUOhzxugqCgLKyMrRp0wYuLvp7hjhEzYiLiwtCQ0OtdnwfHx+H+cU6El5X6+B1tQ5eV+nxmlqHo11XQzUi9diBlYiIiGTFZISIiIhk5dTJiFKpxOuvvw6lUil3KE0Kr6t18LpaB6+r9HhNraMpX1eH6MBKRERETZdT14wQERGR/JiMEBERkayYjBAREZGsmIwQERGRrJw6Gfnss8/Qvn17eHp6YuDAgdi7d6/cIdmNxMRE9O/fH97e3ggMDMT48eORnZ2tVeb69euYOXMmAgIC0KJFCzzwwAMoLCzUKpOTk4P4+Hg0a9YMgYGBmDt3LmpqarTKpKSkoE+fPlAqlejUqROWLVtm7bdnFxYsWACFQoE5c+aot/GamicvLw+PPPIIAgIC4OXlhcjISOzfv1/9uiAIeO211xASEgIvLy+MHDkSJ06c0DrGlStXMGnSJPj4+MDPzw9Tp05FeXm5VpnDhw/j9ttvh6enJ8LCwvD+++/b5P3Joba2Fq+++ioiIiLg5eWFjh074u2339ZaY4TX1bgdO3Zg7NixaNOmDRQKBdasWaP1ui2v4cqVK9GtWzd4enoiMjIS69evl/z9mk1wUklJSYKHh4fwzTffCFlZWcL06dMFPz8/obCwUO7Q7EJcXJzw7bffCpmZmUJ6erpw9913C+Hh4UJ5ebm6zNNPPy2EhYUJW7ZsEfbv3y8MGjRIGDx4sPr1mpoaoVevXsLIkSOFgwcPCuvXrxdatWolJCQkqMucPn1aaNasmfDcc88JR44cET755BPB1dVV2Lhxo03fr63t3btXaN++vRAVFSXMnj1bvZ3XVLwrV64I7dq1Ex577DFhz549wunTp4VNmzYJJ0+eVJdZsGCB4OvrK6xZs0Y4dOiQcO+99woRERFCZWWluszo0aOF6OhoYffu3cJff/0ldOrUSZg4caL69ZKSEiEoKEiYNGmSkJmZKfz000+Cl5eXsHTpUpu+X1uZP3++EBAQIKxbt044c+aMsHLlSqFFixbCRx99pC7D62rc+vXrhZdffllYtWqVAEBYvXq11uu2uoZ///234OrqKrz//vvCkSNHhFdeeUVwd3cXMjIyrH4NTOG0yciAAQOEmTNnqp/X1tYKbdq0ERITE2WMyn4VFRUJAITt27cLgiAIxcXFgru7u7By5Up1maNHjwoAhNTUVEEQ6v4Turi4CAUFBeoyS5YsEXx8fISqqipBEARh3rx5Qs+ePbXO9dBDDwlxcXHWfkuyKSsrEzp37iwkJycLQ4cOVScjvKbmeeGFF4TbbrtN7+sqlUoIDg4WFi5cqN5WXFwsKJVK4aeffhIEQRCOHDkiABD27dunLrNhwwZBoVAIeXl5giAIwueffy60bNlSfZ3rz921a1ep35JdiI+PF5544gmtbffff78wadIkQRB4Xc3RMBmx5TV88MEHhfj4eK14Bg4cKDz11FOSvkdzOWUzzY0bN5CWloaRI0eqt7m4uGDkyJFITU2VMTL7VVJSAgDw9/cHAKSlpaG6ulrrGnbr1g3h4eHqa5iamorIyEgEBQWpy8TFxaG0tBRZWVnqMprHqC/TlH8PM2fORHx8fKP3zWtqnt9//x39+vXDhAkTEBgYiJiYGHz11Vfq18+cOYOCggKta+Lr64uBAwdqXVc/Pz/069dPXWbkyJFwcXHBnj171GXuuOMOeHh4qMvExcUhOzsbV69etfbbtLnBgwdjy5YtOH78OADg0KFD2LlzJ8aMGQOA11UKtryG9v654JTJyKVLl1BbW6v1gQ4AQUFBKCgokCkq+6VSqTBnzhwMGTIEvXr1AgAUFBTAw8MDfn5+WmU1r2FBQYHOa1z/mqEypaWlqKystMbbkVVSUhIOHDiAxMTERq/xmprn9OnTWLJkCTp37oxNmzZhxowZeOaZZ/Ddd98BuHVdDP1/LygoQGBgoNbrbm5u8Pf3F3Xtm5IXX3wRDz/8MLp16wZ3d3fExMRgzpw5mDRpEgBeVynY8hrqK2Mv19ghVu0lec2cOROZmZnYuXOn3KE4tNzcXMyePRvJycnw9PSUO5wmQ6VSoV+/fnj33XcBADExMcjMzMQXX3yBKVOmyByd4/rll1+wfPlyrFixAj179kR6ejrmzJmDNm3a8LqS5JyyZqRVq1ZwdXVtNEqhsLAQwcHBMkVln2bNmoV169Zh27ZtCA0NVW8PDg7GjRs3UFxcrFVe8xoGBwfrvMb1rxkq4+PjAy8vL6nfjqzS0tJQVFSEPn36wM3NDW5ubti+fTs+/vhjuLm5ISgoiNfUDCEhIejRo4fWtu7duyMnJwfAreti6P97cHAwioqKtF6vqanBlStXRF37pmTu3Lnq2pHIyEg8+uijePbZZ9W1eryulrPlNdRXxl6usVMmIx4eHujbty+2bNmi3qZSqbBlyxbExsbKGJn9EAQBs2bNwurVq7F161ZERERovd63b1+4u7trXcPs7Gzk5OSor2FsbCwyMjK0/iMlJyfDx8dHffOIjY3VOkZ9mab4exgxYgQyMjKQnp6ufvTr1w+TJk1S/8xrKt6QIUMaDTs/fvw42rVrBwCIiIhAcHCw1jUpLS3Fnj17tK5rcXEx0tLS1GW2bt0KlUqFgQMHqsvs2LED1dXV6jLJycno2rUrWrZsabX3J5dr167BxUX7FuHq6gqVSgWA11UKtryGdv+5IHcPWrkkJSUJSqVSWLZsmXDkyBHhySefFPz8/LRGKTizGTNmCL6+vkJKSoqQn5+vfly7dk1d5umnnxbCw8OFrVu3Cvv37xdiY2OF2NhY9ev1w1DvuusuIT09Xdi4caPQunVrncNQ586dKxw9elT47LPPmvQw1IY0R9MIAq+pOfbu3Su4ubkJ8+fPF06cOCEsX75caNasmfDjjz+qyyxYsEDw8/MT1q5dKxw+fFgYN26czuGTMTExwp49e4SdO3cKnTt31ho+WVxcLAQFBQmPPvqokJmZKSQlJQnNmjVrMkNQG5oyZYrQtm1b9dDeVatWCa1atRLmzZunLsPralxZWZlw8OBB4eDBgwIA4T//+Y9w8OBB4dy5c4Ig2O4a/v3334Kbm5vwwQcfCEePHhVef/11Du21F5988okQHh4ueHh4CAMGDBB2794td0h2A4DOx7fffqsuU1lZKfzrX/8SWrZsKTRr1ky47777hPz8fK3jnD17VhgzZozg5eUltGrVSnj++eeF6upqrTLbtm0TevfuLXh4eAgdOnTQOkdT1zAZ4TU1z//+9z+hV69eglKpFLp16yZ8+eWXWq+rVCrh1VdfFYKCggSlUimMGDFCyM7O1ipz+fJlYeLEiUKLFi0EHx8f4fHHHxfKysq0yhw6dEi47bbbBKVSKbRt21ZYsGCB1d+bXEpLS4XZs2cL4eHhgqenp9ChQwfh5Zdf1ho+yutq3LZt23R+lk6ZMkUQBNtew19++UXo0qWL4OHhIfTs2VP4448/rPa+xVIIgsZ0ekREREQ25pR9RoiIiMh+MBkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIln9P70kCre2LNr2AAAAAElFTkSuQmCC", 1001 | "text/plain": [ 1002 | "
" 1003 | ] 1004 | }, 1005 | "metadata": {}, 1006 | "output_type": "display_data" 1007 | } 1008 | ], 1009 | "source": [ 1010 | "import matplotlib.pyplot as plt\n", 1011 | "import pandas as pd\n", 1012 | "\n", 1013 | "data = pd.read_csv(\"loss.txt\")\n", 1014 | "plt.plot(data)\n", 1015 | "plt.show()" 1016 | ] 1017 | }, 1018 | { 1019 | "cell_type": "markdown", 1020 | "id": "9809bc9f-4ff6-46c3-9c43-08c6c2694a82", 1021 | "metadata": {}, 1022 | "source": [ 1023 | "## Generate text with fine-tuned model\n", 1024 | "\n", 1025 | "Again we check results with our test dataset (5 rows).
\n", 1026 | "As you can see below, it can output the completion very well, because it's fine-tuned." 1027 | ] 1028 | }, 1029 | { 1030 | "cell_type": "code", 1031 | "execution_count": 25, 1032 | "id": "29903cae-404e-4209-9c84-6c8a69609c13", 1033 | "metadata": {}, 1034 | "outputs": [ 1035 | { 1036 | "name": "stdout", 1037 | "output_type": "stream", 1038 | "text": [ 1039 | "********** input **********\n", 1040 | "name : The Vaults | Type : pub | food : Italian | price : less than £ 20 | customer rating : low | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n", 1041 | "\n", 1042 | "********** result **********\n", 1043 | "name : The Vaults | Type : pub | food : Italian | price : less than £ 20 | customer rating : low | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n", 1044 | "The Vaults is a pub near the Rainbow Vegetarian Café in the city centre. It is not family friendly and has a low customer rating of less than\n", 1045 | "********** input **********\n", 1046 | "name : The Cricketers | Type : restaurant | customer rating : average | family friendly : yes | near : Café Sicilia\n", 1047 | "\n", 1048 | "********** result **********\n", 1049 | "name : The Cricketers | Type : restaurant | customer rating : average | family friendly : yes | near : Café Sicilia\n", 1050 | "The Cricketers is a restaurant near Café Sicilia. It is family friendly and has an average customer rating.<|endoftext|>\n", 1051 | "********** input **********\n", 1052 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : no | near : All Bar One\n", 1053 | "\n", 1054 | "********** result **********\n", 1055 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : no | near : All Bar One\n", 1056 | "The Cricketers is a restaurant located in the city centre near All Bar One. It is not family - friendly. It is located in the cheap\n", 1057 | "********** input **********\n", 1058 | "name : The Vaults | Type : pub | food : Japanese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 1059 | "\n", 1060 | "********** result **********\n", 1061 | "name : The Vaults | Type : pub | food : Japanese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n", 1062 | "The Vaults is a cheap, family friendly pub located in the city centre near Raja Indian Cuisine.<|endoftext|>\n", 1063 | "********** input **********\n", 1064 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n", 1065 | "\n", 1066 | "********** result **********\n", 1067 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n", 1068 | "The Wrestlers is a pub near Raja Indian Cuisine in riverside. It is not family friendly.<|endoftext|>\n" 1069 | ] 1070 | } 1071 | ], 1072 | "source": [ 1073 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n", 1074 | "test_data = test_data[::2] # because it's duplicated\n", 1075 | "test_loader = DataLoader(\n", 1076 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n", 1077 | " batch_size=1,\n", 1078 | " shuffle=True,\n", 1079 | " collate_fn=collate_batch\n", 1080 | ")\n", 1081 | "\n", 1082 | "for i, (input, _, mask) in enumerate(test_loader):\n", 1083 | " if i == 5:\n", 1084 | " break\n", 1085 | " print(\"********** input **********\")\n", 1086 | " input_len = torch.sum(mask).cpu().numpy()\n", 1087 | " print(tokenizer.decode(input[0][:input_len]))\n", 1088 | " result_token, result_len = generate_text(\n", 1089 | " model,\n", 1090 | " input,\n", 1091 | " mask,\n", 1092 | " eos_id,\n", 1093 | " pred_sequence_length=30)\n", 1094 | " print(\"********** result **********\")\n", 1095 | " print(tokenizer.decode(result_token[0][:result_len]))" 1096 | ] 1097 | }, 1098 | { 1099 | "cell_type": "code", 1100 | "execution_count": null, 1101 | "id": "6a7c1dd3-4057-497a-83ae-f99b1883697e", 1102 | "metadata": {}, 1103 | "outputs": [], 1104 | "source": [] 1105 | } 1106 | ], 1107 | "metadata": { 1108 | "kernelspec": { 1109 | "display_name": "Python 3 (ipykernel)", 1110 | "language": "python", 1111 | "name": "python3" 1112 | }, 1113 | "language_info": { 1114 | "codemirror_mode": { 1115 | "name": "ipython", 1116 | "version": 3 1117 | }, 1118 | "file_extension": ".py", 1119 | "mimetype": "text/x-python", 1120 | "name": "python", 1121 | "nbconvert_exporter": "python", 1122 | "pygments_lexer": "ipython3", 1123 | "version": "3.8.10" 1124 | } 1125 | }, 1126 | "nbformat": 4, 1127 | "nbformat_minor": 5 1128 | } 1129 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # Fine-tuning LLM with LoRA (Low-Rank Adaptation) 2 | 3 | LoRA (Low-Rank Adaptation) is one of mostly used parameter-efficient fine-tuning (PEFT) methods today. 4 | 5 | This example shows you [LoRA (Low-Rank Adaptation)](https://arxiv.org/abs/2106.09685) implementation from scratch (manually) in a step-by-step manner (without ```PEFT``` package), and also shows you clear ideas behind this implementation in IPython notebook. 6 | 7 | This is also runnable in the mainstream hardware with small footprint - such as, a signle GPU of Tesla T4, consumer GPUs (NVIDIA RTX), etc - for you to try this code easily. 8 | 9 | | Example | Description | 10 | | -------------------------------------------------------------------- | ----------------------------------------------------------------------- | 11 | | [01-finetune-opt-with-lora.ipynb](01-finetune-opt-with-lora.ipynb) | Fine-tuning Meta's OPT-125M with LoRA
(Also, explaining LoRA method) | 12 | | [02-finetune-gpt2-with-lora.ipynb](02-finetune-gpt2-with-lora.ipynb) | Fine-tuning OpenAI's GPT-2 small (124M) with LoRA | 13 | 14 | Unlike examples in [official repository](https://github.com/microsoft/LoRA), here I download pre-trained models to focus on LoRA implementation. 15 | 16 | > Note : In this repository, Hugging Face API is used to download pre-trained models and I then apply regular PyTorch training loop for fine-tuning. (I don't use blackboxed ```Trainer``` class in Hugging Face API.) 17 | 18 | ## 1. Set-up and Install 19 | 20 | To run this example, please install prerequisite's software and setup your environment as follows.
21 | In the following setting, I have used a GPU-utilized virtual machine (VM) with "Ubuntu Server 20.04 LTS" image in Microsoft Azure. 22 | 23 | ### Install GPU driver (CUDA) 24 | 25 | Install CUDA (NVIDIA GPU driver) as follows. 26 | 27 | ``` 28 | # compilers and development settings 29 | sudo apt-get update 30 | sudo apt install -y gcc 31 | sudo apt-get install -y make 32 | 33 | # install CUDA 34 | wget https://developer.download.nvidia.com/compute/cuda/12.2.2/local_installers/cuda_12.2.2_535.104.05_linux.run 35 | sudo sh cuda_12.2.2_535.104.05_linux.run 36 | echo -e "export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64" >> ~/.bashrc 37 | source ~/.bashrc 38 | ``` 39 | 40 | ### Install packages 41 | 42 | Install PyTorch, Hugging Face transformer, and other libraries as follows. 43 | 44 | ``` 45 | # install and upgrade pip 46 | sudo apt-get install -y python3-pip 47 | sudo -H pip3 install --upgrade pip 48 | # install packages 49 | pip3 install torch transformers pandas matplotlib 50 | # install jupyter for running notebook 51 | pip3 install jupyter 52 | ``` 53 | 54 | ## 2. Fine-tune (Train) 55 | 56 | Download this repository. 57 | 58 | ``` 59 | git clone https://github.com/tsmatz/finetune_llm_with_lora 60 | ``` 61 | 62 | Run jupyter notebook. 63 | 64 | ``` 65 | jupyter notebook 66 | ``` 67 | 68 | Open jupyter notebook in browser, and run examples in this repository. 69 | -------------------------------------------------------------------------------- /images/auto_regressive_transformer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tsmatz/finetune_llm_with_lora/2e84a5e9e5095aaeacacaa723ee5a7c34c36b678/images/auto_regressive_transformer.png -------------------------------------------------------------------------------- /images/lora.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tsmatz/finetune_llm_with_lora/2e84a5e9e5095aaeacacaa723ee5a7c34c36b678/images/lora.png --------------------------------------------------------------------------------