├── 01-finetune-opt-with-lora.ipynb
├── 02-finetune-gpt2-with-lora.ipynb
├── Readme.md
└── images
├── auto_regressive_transformer.png
└── lora.png
/01-finetune-opt-with-lora.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "857cafc6-da38-4aa7-8afc-63aa626fa7aa",
6 | "metadata": {},
7 | "source": [
8 | "# 01. Finetuning OPT with LoRA\n",
9 | "\n",
10 | "Today's popular auto-regressive models - such as, GPT, LLaMA, Falcon, etc - are decoder-only models, in which the output token is predicted by using only input's text (called a prompt).\n",
11 | "\n",
12 | "\n",
13 | "\n",
14 | "*\"Decoder-only\" model is implemented using layers in the red box.
\n",
15 | "(Diagram from : [Attention Is All You Need](https://arxiv.org/abs/1706.03762))*\n",
16 | "\n",
17 | "In this model, the task is differentiated also by using input's text (i.e, prompt).\n",
18 | "\n",
19 | "> Note : See [this repository](https://github.com/tsmatz/nlp-tutorials) for intrinsic idea of LLM transformers.\n",
20 | "\n",
21 | "In this example, we fine-tune the pre-trained auto-regressive model, Meta's OPT (```facebook/opt-125m```), by applying LoRA (Low-Rank Adaptation) optimization.\n",
22 | "\n",
23 | "In this example, I download the pre-trained model from Hugging Face hub, but fine-tune model with regular PyTorch training loop.
\n",
24 | "(Here I don't use Hugging Face Trainer class.)\n",
25 | "\n",
26 | "See [Readme](https://github.com/tsmatz/finetune_llm_with_lora) for prerequisite's setup."
27 | ]
28 | },
29 | {
30 | "cell_type": "code",
31 | "execution_count": 1,
32 | "id": "3d49acf1-9ad1-4a6c-9312-6785cb3f5862",
33 | "metadata": {},
34 | "outputs": [],
35 | "source": [
36 | "model_name = \"facebook/opt-125m\"\n",
37 | "# model_name = \"facebook/opt-350m\"\n",
38 | "# model_name = \"facebook/opt-1.3b\"\n",
39 | "# model_name = \"facebook/opt-6.7b\""
40 | ]
41 | },
42 | {
43 | "cell_type": "code",
44 | "execution_count": 2,
45 | "id": "4d835e84-a01d-4c33-926b-60d9dd4a7627",
46 | "metadata": {},
47 | "outputs": [],
48 | "source": [
49 | "import torch\n",
50 | "\n",
51 | "device = torch.device(\"cuda\")"
52 | ]
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "id": "ead383e5-149b-4bfb-9324-3cc639fd398d",
57 | "metadata": {},
58 | "source": [
59 | "## Prepare dataset and dataloader"
60 | ]
61 | },
62 | {
63 | "cell_type": "markdown",
64 | "id": "91ecbb08-6a74-4623-bfe8-bddba5254e35",
65 | "metadata": {},
66 | "source": [
67 | "In this example, we use dataset used in [official LoRA example](https://github.com/microsoft/LoRA).\n",
68 | "\n",
69 | "Download dataset from official repository."
70 | ]
71 | },
72 | {
73 | "cell_type": "code",
74 | "execution_count": 3,
75 | "id": "54a564f1-f8f3-42a6-b160-bebdbcc3aac0",
76 | "metadata": {},
77 | "outputs": [
78 | {
79 | "name": "stdout",
80 | "output_type": "stream",
81 | "text": [
82 | "--2023-10-06 03:27:50-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt\n",
83 | "Resolving github.com (github.com)... 140.82.114.3\n",
84 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
85 | "HTTP request sent, awaiting response... 302 Found\n",
86 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt [following]\n",
87 | "--2023-10-06 03:27:51-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt\n",
88 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n",
89 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
90 | "HTTP request sent, awaiting response... 200 OK\n",
91 | "Length: 9624463 (9.2M) [text/plain]\n",
92 | "Saving to: ‘train.txt’\n",
93 | "\n",
94 | "train.txt 100%[===================>] 9.18M --.-KB/s in 0.04s \n",
95 | "\n",
96 | "2023-10-06 03:27:51 (248 MB/s) - ‘train.txt’ saved [9624463/9624463]\n",
97 | "\n"
98 | ]
99 | }
100 | ],
101 | "source": [
102 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt"
103 | ]
104 | },
105 | {
106 | "cell_type": "code",
107 | "execution_count": 4,
108 | "id": "d48464ea-991f-48b2-9166-3323cfd61676",
109 | "metadata": {
110 | "scrolled": true
111 | },
112 | "outputs": [
113 | {
114 | "name": "stdout",
115 | "output_type": "stream",
116 | "text": [
117 | "--2023-10-06 03:27:54-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt\n",
118 | "Resolving github.com (github.com)... 140.82.114.3\n",
119 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
120 | "HTTP request sent, awaiting response... 302 Found\n",
121 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt [following]\n",
122 | "--2023-10-06 03:27:54-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt\n",
123 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n",
124 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
125 | "HTTP request sent, awaiting response... 200 OK\n",
126 | "Length: 1351149 (1.3M) [text/plain]\n",
127 | "Saving to: ‘test.txt’\n",
128 | "\n",
129 | "test.txt 100%[===================>] 1.29M --.-KB/s in 0.006s \n",
130 | "\n",
131 | "2023-10-06 03:27:54 (208 MB/s) - ‘test.txt’ saved [1351149/1351149]\n",
132 | "\n"
133 | ]
134 | }
135 | ],
136 | "source": [
137 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt"
138 | ]
139 | },
140 | {
141 | "cell_type": "markdown",
142 | "id": "09472803-8c62-48e0-9a63-b9b9448f16d3",
143 | "metadata": {},
144 | "source": [
145 | "Show the downloaded data (first 5 rows)."
146 | ]
147 | },
148 | {
149 | "cell_type": "code",
150 | "execution_count": 5,
151 | "id": "e6e60596-028f-4c4b-a95d-f74a0ff3b188",
152 | "metadata": {},
153 | "outputs": [
154 | {
155 | "name": "stdout",
156 | "output_type": "stream",
157 | "text": [
158 | "name : The Vaults | Type : pub | price : more than £ 30 | customer rating : 5 out of 5 | near : Café Adriatic||The Vaults pub near Café Adriatic has a 5 star rating . Prices start at £ 30 . \n",
159 | "name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Café Brazil||Close to Café Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of £ 10.50 . Delicious Pub food . \n",
160 | "name : The Eagle | Type : coffee shop | food : Japanese | price : less than £ 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King||The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than £ 20 for Japanese food . \n",
161 | "name : The Mill | Type : coffee shop | food : French | price : £ 20 - 25 | area : riverside | near : The Sorrento||Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at £ 20- £ 25 it is in the riverside area . \n",
162 | "name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat||For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat . \n"
163 | ]
164 | }
165 | ],
166 | "source": [
167 | "!head -n 5 train.txt"
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "id": "93f5fabe-590c-459b-aa16-4b5a506fb54b",
173 | "metadata": {},
174 | "source": [
175 | "Convert above data into JsonL format."
176 | ]
177 | },
178 | {
179 | "cell_type": "code",
180 | "execution_count": 6,
181 | "id": "7376e0c0-16c9-46f4-ad4c-83d1a677f5a2",
182 | "metadata": {},
183 | "outputs": [],
184 | "source": [
185 | "import sys\n",
186 | "import io\n",
187 | "import json\n",
188 | "\n",
189 | "def format_convert(read_file, write_file):\n",
190 | " with open(read_file, \"r\", encoding=\"utf8\") as reader, \\\n",
191 | " \t open(write_file, \"w\", encoding=\"utf8\") as writer :\n",
192 | " \tfor line in reader:\n",
193 | " \t\titems = line.strip().split(\"||\")\n",
194 | " \t\tcontext = items[0]\n",
195 | " \t\tcompletion = items[1].strip(\"\\n\")\n",
196 | " \t\tx = {}\n",
197 | " \t\tx[\"context\"] = context\n",
198 | " \t\tx[\"completion\"] = completion\n",
199 | " \t\twriter.write(json.dumps(x)+\"\\n\")\n",
200 | "\n",
201 | "format_convert(\"train.txt\", \"train_formatted.jsonl\")\n",
202 | "format_convert(\"test.txt\", \"test_formatted.jsonl\")"
203 | ]
204 | },
205 | {
206 | "cell_type": "markdown",
207 | "id": "3ceec952-fe03-475f-9f3e-22237cc9c44b",
208 | "metadata": {},
209 | "source": [
210 | "Show the converted data (first 5 rows)."
211 | ]
212 | },
213 | {
214 | "cell_type": "code",
215 | "execution_count": 7,
216 | "id": "cb32aca7-bd0e-4847-a4c2-cc7e67dc2b7a",
217 | "metadata": {},
218 | "outputs": [
219 | {
220 | "name": "stdout",
221 | "output_type": "stream",
222 | "text": [
223 | "{\"context\": \"name : The Vaults | Type : pub | price : more than \\u00a3 30 | customer rating : 5 out of 5 | near : Caf\\u00e9 Adriatic\", \"completion\": \"The Vaults pub near Caf\\u00e9 Adriatic has a 5 star rating . Prices start at \\u00a3 30 .\"}\n",
224 | "\n",
225 | "{\"context\": \"name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Caf\\u00e9 Brazil\", \"completion\": \"Close to Caf\\u00e9 Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of \\u00a3 10.50 . Delicious Pub food .\"}\n",
226 | "\n",
227 | "{\"context\": \"name : The Eagle | Type : coffee shop | food : Japanese | price : less than \\u00a3 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King\", \"completion\": \"The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than \\u00a3 20 for Japanese food .\"}\n",
228 | "\n",
229 | "{\"context\": \"name : The Mill | Type : coffee shop | food : French | price : \\u00a3 20 - 25 | area : riverside | near : The Sorrento\", \"completion\": \"Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at \\u00a3 20- \\u00a3 25 it is in the riverside area .\"}\n",
230 | "\n",
231 | "{\"context\": \"name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat\", \"completion\": \"For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat .\"}\n",
232 | "\n"
233 | ]
234 | }
235 | ],
236 | "source": [
237 | "with open(\"train_formatted.jsonl\", \"r\") as reader:\n",
238 | " for _ in range(5):\n",
239 | " print(next(reader))"
240 | ]
241 | },
242 | {
243 | "cell_type": "markdown",
244 | "id": "6631f786-be4b-40cf-89d9-7009c1888821",
245 | "metadata": {},
246 | "source": [
247 | "Load tokenizer from Hugging Face."
248 | ]
249 | },
250 | {
251 | "cell_type": "code",
252 | "execution_count": 8,
253 | "id": "e5433dc0-b5a5-4c01-adb5-3ffa2279eca8",
254 | "metadata": {},
255 | "outputs": [],
256 | "source": [
257 | "from transformers import AutoTokenizer\n",
258 | "import os\n",
259 | "\n",
260 | "tokenizer = AutoTokenizer.from_pretrained(\n",
261 | " model_name,\n",
262 | " fast_tokenizer=True)\n",
263 | "tokenizer.pad_token = tokenizer.eos_token\n",
264 | "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
265 | ]
266 | },
267 | {
268 | "cell_type": "markdown",
269 | "id": "50817c47-a97b-4f80-975b-836859a0a7cf",
270 | "metadata": {},
271 | "source": [
272 | "Set block size, which is used to separate long text for model consumption."
273 | ]
274 | },
275 | {
276 | "cell_type": "code",
277 | "execution_count": 9,
278 | "id": "5f250929-5703-4b17-9f7b-26340950c055",
279 | "metadata": {},
280 | "outputs": [],
281 | "source": [
282 | "block_size = 512"
283 | ]
284 | },
285 | {
286 | "cell_type": "markdown",
287 | "id": "2332617b-1e66-4812-ad47-5eaeb52b101b",
288 | "metadata": {},
289 | "source": [
290 | "Create function to convert data. (Later this function is then used in data loader.)
\n",
291 | "In this function,\n",
292 | "\n",
293 | "1. Tokenize both contexts and compeletions. : e.g, ```\"This is a pen.\"``` --> ```[1212, 318, 257, 3112, 13]```\n",
294 | "2. Concatenate context's token and completion's token. (But it's delimited by \"\\n\" between context and completion.) This is used for inputs for LLM.\n",
295 | "3. Create labels (targets) with inputs. Label is ```input[1:]``` (i.e, shifted right by one element), and is filled by ```-100``` in context's positions. (See below note.)\n",
296 | "4. Pad tokens to make the length of token become ```block_size```.\n",
297 | "\n",
298 | "> Note : Here I set ```-100``` as an ignored index for loss computation, because PyTorch cross-entropy function (```torch.nn.functional.cross_entropy()```) has a property ```ignore_index``` which default value is ```-100```."
299 | ]
300 | },
301 | {
302 | "cell_type": "code",
303 | "execution_count": 10,
304 | "id": "9f2f38aa-b3d0-4614-aa59-8ddd977176d1",
305 | "metadata": {},
306 | "outputs": [],
307 | "source": [
308 | "from torch.utils.data import DataLoader\n",
309 | "import pandas as pd\n",
310 | "\n",
311 | "def fill_ignore_label(l, c):\n",
312 | " l[:len(c) - 1] = [-100] * (len(c) - 1)\n",
313 | " return l\n",
314 | "\n",
315 | "def pad_tokens(tokens, max_seq_length, padding_token):\n",
316 | " res_tokens = tokens[:max_seq_length]\n",
317 | " token_len = len(res_tokens)\n",
318 | " res_tokens = res_tokens + \\\n",
319 | " [padding_token for _ in range(max_seq_length - token_len)]\n",
320 | " return res_tokens\n",
321 | "\n",
322 | "def collate_batch(batch):\n",
323 | " # tokenize both context and completion respectively\n",
324 | " # (context and completion is delimited by \"\\n\")\n",
325 | " context_list = list(zip(*batch))[0]\n",
326 | " context_list = [c + \"\\n\" for c in context_list]\n",
327 | " completion_list = list(zip(*batch))[1]\n",
328 | " context_result = tokenizer(context_list)\n",
329 | " context_tokens = context_result[\"input_ids\"]\n",
330 | " context_masks = context_result[\"attention_mask\"]\n",
331 | " completion_result = tokenizer(completion_list)\n",
332 | " completion_tokens = completion_result[\"input_ids\"]\n",
333 | " completion_masks = completion_result[\"attention_mask\"]\n",
334 | " # OPT tokenizer adds the start token in sequence,\n",
335 | " # and we then remove it in completion\n",
336 | " completion_tokens = [t[1:] for t in completion_tokens]\n",
337 | " completion_masks = [t[1:] for t in completion_masks]\n",
338 | " # concatenate token\n",
339 | " inputs = [i + j for i, j in zip(context_tokens, completion_tokens)]\n",
340 | " masks = [i + j for i, j in zip(context_masks, completion_masks)]\n",
341 | " # create label\n",
342 | " eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n",
343 | " labels = [t[1:] + [eos_id] for t in inputs]\n",
344 | " labels = list(map(fill_ignore_label, labels, context_tokens))\n",
345 | " # truncate and pad tokens\n",
346 | " inputs = [pad_tokens(t, block_size, 0) for t in inputs] # OPT and GPT-2 doesn't use pad token (instead attn mask is used)\n",
347 | " masks = [pad_tokens(t, block_size, 0) for t in masks]\n",
348 | " labels = [pad_tokens(t, block_size, -100) for t in labels]\n",
349 | " # convert to tensor\n",
350 | " inputs = torch.tensor(inputs, dtype=torch.int64).to(device)\n",
351 | " masks = torch.tensor(masks, dtype=torch.int64).to(device)\n",
352 | " labels = torch.tensor(labels, dtype=torch.int64).to(device)\n",
353 | " return inputs, labels, masks"
354 | ]
355 | },
356 | {
357 | "cell_type": "markdown",
358 | "id": "2084d2e9-ef64-47a2-aec9-d24ead1cb38a",
359 | "metadata": {},
360 | "source": [
361 | "Now create PyTorch dataloader with previous function (collator function).\n",
362 | "\n",
363 | "> Note : In this example, data is small and we then load all JSON data in memory.
\n",
364 | "> When it's large, load data progressively by implementing custom PyTorch dataset. (See [here](https://github.com/tsmatz/decision-transformer) for example.)"
365 | ]
366 | },
367 | {
368 | "cell_type": "code",
369 | "execution_count": 11,
370 | "id": "f3bce3bb-2215-4bd6-a6a6-5b6b9d5afdc0",
371 | "metadata": {},
372 | "outputs": [],
373 | "source": [
374 | "batch_size = 8\n",
375 | "gradient_accumulation_steps = 16\n",
376 | "\n",
377 | "data = pd.read_json(\"train_formatted.jsonl\", lines=True)\n",
378 | "dataloader = DataLoader(\n",
379 | " list(zip(data[\"context\"], data[\"completion\"])),\n",
380 | " batch_size=batch_size,\n",
381 | " shuffle=True,\n",
382 | " collate_fn=collate_batch\n",
383 | ")"
384 | ]
385 | },
386 | {
387 | "cell_type": "markdown",
388 | "id": "3ba64144-b698-457e-b827-941020456536",
389 | "metadata": {},
390 | "source": [
391 | "## Load model"
392 | ]
393 | },
394 | {
395 | "cell_type": "markdown",
396 | "id": "1bfd360d-7bdc-4fd7-9b12-bcf9fe0a8db2",
397 | "metadata": {},
398 | "source": [
399 | "Load model from Hugging Face."
400 | ]
401 | },
402 | {
403 | "cell_type": "code",
404 | "execution_count": 12,
405 | "id": "271181bd-677a-4da9-9e57-2874f5e47bd0",
406 | "metadata": {},
407 | "outputs": [],
408 | "source": [
409 | "from transformers import AutoModelForCausalLM, AutoConfig\n",
410 | "\n",
411 | "config = AutoConfig.from_pretrained(model_name)\n",
412 | "model = AutoModelForCausalLM.from_pretrained(\n",
413 | " model_name,\n",
414 | " config=config,\n",
415 | ").to(device)"
416 | ]
417 | },
418 | {
419 | "cell_type": "markdown",
420 | "id": "27ab764a-d634-40f8-9edb-a01146845233",
421 | "metadata": {},
422 | "source": [
423 | "## Generate text (before fine-tuning)"
424 | ]
425 | },
426 | {
427 | "cell_type": "markdown",
428 | "id": "559efeaf-4b38-4a0c-9be6-eb394221e374",
429 | "metadata": {},
430 | "source": [
431 | "Now run prediction with downloaded model (which is not still fine-tuned).\n",
432 | "\n",
433 | "First we create a function to generate text."
434 | ]
435 | },
436 | {
437 | "cell_type": "code",
438 | "execution_count": 13,
439 | "id": "51a0c4fc-e0a7-4bbf-b25a-c335fe61f3df",
440 | "metadata": {},
441 | "outputs": [],
442 | "source": [
443 | "def generate_text(model, input, mask, eos_id, pred_sequence_length):\n",
444 | " predicted_last_id = -1\n",
445 | " start_token_len = torch.sum(mask).cpu().numpy()\n",
446 | " token_len = start_token_len\n",
447 | " with torch.no_grad():\n",
448 | " while (predicted_last_id != eos_id) and \\\n",
449 | " (token_len - start_token_len < pred_sequence_length):\n",
450 | " output = model(\n",
451 | " input_ids=input,\n",
452 | " attention_mask=mask,\n",
453 | " )\n",
454 | " predicted_ids = torch.argmax(output.logits, axis=-1).cpu().numpy()\n",
455 | " predicted_last_id = predicted_ids[0][token_len - 1]\n",
456 | " input[0][token_len] = predicted_last_id\n",
457 | " mask[0][token_len] = 1\n",
458 | " token_len = torch.sum(mask).cpu().numpy()\n",
459 | " return input, token_len"
460 | ]
461 | },
462 | {
463 | "cell_type": "markdown",
464 | "id": "3936b1a1-ae9f-48a5-80db-691261dda704",
465 | "metadata": {},
466 | "source": [
467 | "Let's test our function and generate text. (Here we stop the text generation when it reaches 15 tokens in prediction.)"
468 | ]
469 | },
470 | {
471 | "cell_type": "code",
472 | "execution_count": 14,
473 | "id": "28b7e13f-e8fb-4a9f-90ed-0464463ef569",
474 | "metadata": {},
475 | "outputs": [
476 | {
477 | "name": "stdout",
478 | "output_type": "stream",
479 | "text": [
480 | "Once upon a time, I was a student at the University of California, Berkeley. I was a\n",
481 | "My name is Clara and I am a student at the University of California, Berkeley. I am a member of\n"
482 | ]
483 | }
484 | ],
485 | "source": [
486 | "eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n",
487 | "\n",
488 | "result = tokenizer(\"Once upon a time,\")\n",
489 | "input = result[\"input_ids\"]\n",
490 | "mask = result[\"attention_mask\"]\n",
491 | "input = pad_tokens(input, block_size, 0)\n",
492 | "mask = pad_tokens(mask, block_size, 0)\n",
493 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n",
494 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n",
495 | "\n",
496 | "result_token, result_len = generate_text(\n",
497 | " model,\n",
498 | " input,\n",
499 | " mask,\n",
500 | " eos_id,\n",
501 | " pred_sequence_length=15)\n",
502 | "print(tokenizer.decode(result_token[0][:result_len]))\n",
503 | "\n",
504 | "result = tokenizer(\"My name is Clara and I am\")\n",
505 | "input = result[\"input_ids\"]\n",
506 | "mask = result[\"attention_mask\"]\n",
507 | "input = pad_tokens(input, block_size, 0)\n",
508 | "mask = pad_tokens(mask, block_size, 0)\n",
509 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n",
510 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n",
511 | "\n",
512 | "result_token, result_len = generate_text(\n",
513 | " model,\n",
514 | " input,\n",
515 | " mask,\n",
516 | " eos_id,\n",
517 | " pred_sequence_length=15)\n",
518 | "print(tokenizer.decode(result_token[0][:result_len]))"
519 | ]
520 | },
521 | {
522 | "cell_type": "markdown",
523 | "id": "d48fb60b-c05d-4884-a9bc-92152c94c894",
524 | "metadata": {},
525 | "source": [
526 | "Now we generate text with our test dataset (5 rows).
\n",
527 | "As you can see below, it cannot output the completion well, because it's not still fine-tuned."
528 | ]
529 | },
530 | {
531 | "cell_type": "code",
532 | "execution_count": 15,
533 | "id": "495728ef-fbe6-4953-a354-4b7a8bb88798",
534 | "metadata": {},
535 | "outputs": [
536 | {
537 | "name": "stdout",
538 | "output_type": "stream",
539 | "text": [
540 | "********** input **********\n",
541 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n",
542 | "\n",
543 | "********** result **********\n",
544 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n",
545 | "\n",
546 | "The Wrestlers is a restaurant in the heart of the city of Raja, India. It is located in the heart of the city of Raj\n",
547 | "********** input **********\n",
548 | "name : The Cricketers | Type : coffee shop | customer rating : 1 out of 5 | family friendly : yes | near : Avalon\n",
549 | "\n",
550 | "********** result **********\n",
551 | "name : The Cricketers | Type : coffee shop | customer rating : 1 out of 5 | family friendly : yes | near : Avalon\n",
552 | "\n",
553 | "The Cricketers is a coffee shop in Avalon, New York. It is located at the corner of Main Street and Main Street. The coffee\n",
554 | "********** input **********\n",
555 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n",
556 | "\n",
557 | "********** result **********\n",
558 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n",
559 | "\n",
560 | "The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre\n",
561 | "********** input **********\n",
562 | "name : The Punter | Type : restaurant | food : English | price : high | area : riverside | family friendly : no | near : Raja Indian Cuisine\n",
563 | "\n",
564 | "********** result **********\n",
565 | "name : The Punter | Type : restaurant | food : English | price : high | area : riverside | family friendly : no | near : Raja Indian Cuisine\n",
566 | "\n",
567 | "The Punter is a restaurant in Raja, India. It is located in the heart of the Raja district of Rajasthan. It\n",
568 | "********** input **********\n",
569 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : yes | near : All Bar One\n",
570 | "\n",
571 | "********** result **********\n",
572 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : yes | near : All Bar One\n",
573 | "\n",
574 | "The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly\n"
575 | ]
576 | }
577 | ],
578 | "source": [
579 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n",
580 | "test_data = test_data[::2] # because it's duplicated\n",
581 | "test_loader = DataLoader(\n",
582 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n",
583 | " batch_size=1,\n",
584 | " shuffle=True,\n",
585 | " collate_fn=collate_batch\n",
586 | ")\n",
587 | "\n",
588 | "for i, (input, _, mask) in enumerate(test_loader):\n",
589 | " if i == 5:\n",
590 | " break\n",
591 | " print(\"********** input **********\")\n",
592 | " input_len = torch.sum(mask).cpu().numpy()\n",
593 | " print(tokenizer.decode(input[0][:input_len]))\n",
594 | " result_token, result_len = generate_text(\n",
595 | " model,\n",
596 | " input,\n",
597 | " mask,\n",
598 | " eos_id,\n",
599 | " pred_sequence_length=30)\n",
600 | " print(\"********** result **********\")\n",
601 | " print(tokenizer.decode(result_token[0][:result_len]))"
602 | ]
603 | },
604 | {
605 | "cell_type": "markdown",
606 | "id": "e3138341-e01c-4fae-af78-c61e34967e92",
607 | "metadata": {},
608 | "source": [
609 | "## LoRA (Low-Rank Adaptation)\n",
610 | "\n",
611 | "Now we apply LoRA in our downloaded model.\n",
612 | "\n",
613 | "[LoRA (Low-Rank Adaptation)](https://arxiv.org/abs/2106.09685) (which is developed by Microsoft Research) is a popular adaptation method for efficient fine-tuning.\n",
614 | "\n",
615 | "In a task-specific fine-tuning, the change in weights during model adaptation has a low intrinsic rank.
\n",
616 | "With this hypothesis, we can assume that model's updates ($ \\Delta W $) will be re-written with much smaller low-rank matrices $ B \\cdot A $ as follows.\n",
617 | "\n",
618 | "$$ \\displaystyle W_0 x + \\Delta W x = W_0 x + B \\cdot A x $$\n",
619 | "\n",
620 | "where\n",
621 | "\n",
622 | "- $ W_0 \\in \\mathbb{R}^{d \\times k} $ is a pre-trained weight's matrix (which is frozen).\n",
623 | "- $ \\Delta W $ is updates.\n",
624 | "- $ B \\in \\mathbb{R}^{d \\times r}, A \\in \\mathbb{R}^{r \\times k} $ and $ \\verb| rank |\\ r \\ll min(d, k) $\n",
625 | "\n",
626 | "\n",
627 | "\n",
628 | "*From : [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)*\n",
629 | "\n",
630 | "In this assumption, we freeze all weights except for $ B $ and $ A $, and train only these low-ranked matrices $ B $ and $ A $.
\n",
631 | "With this manner, you can fine-tune large transformers for a specific task without full-parameter's fine-tuning.\n",
632 | "\n",
633 | "This will significantly save the required capacity (GPU memories) for training, and the number of required GPUs can approximately be reduced to one-fourth in the benchmark with GPT-3.\n",
634 | "\n",
635 | "For the purpose of your learning, here I manually (from scratch) convert the current model into the model with LoRA.\n",
636 | "\n",
637 | "> Note : You can use ```PEFT``` package to be able to get LoRA model with a few lines of code. (Here I don't use this package.)"
638 | ]
639 | },
640 | {
641 | "cell_type": "markdown",
642 | "id": "5265832d-a736-4d68-80d3-347833d2c590",
643 | "metadata": {},
644 | "source": [
645 | "Before changing our model, first we check the structure of our model.
\n",
646 | "As you can see below (see the result in the cell), the following 6 linear layers are used in a single transformer layer on OPT.\n",
647 | "\n",
648 | "- Linear layer to get key\n",
649 | "- Linear layer to get value\n",
650 | "- Linear layer to get query\n",
651 | "- Linear layer for the output of attention\n",
652 | "- 2 linear layers (feed-forward layer) for the output of a single layer of transformer\n",
653 | "\n",
654 | "In this example, we'll convert all these layers into LoRA layers.
\n",
655 | "The transformer in OPT-125M has 12 layers and it then has total 6 x 12 = 72 linear layers to be converted."
656 | ]
657 | },
658 | {
659 | "cell_type": "code",
660 | "execution_count": 16,
661 | "id": "5acb8f62-791a-4fa4-b00c-2666cf34827f",
662 | "metadata": {},
663 | "outputs": [
664 | {
665 | "data": {
666 | "text/plain": [
667 | "OPTForCausalLM(\n",
668 | " (model): OPTModel(\n",
669 | " (decoder): OPTDecoder(\n",
670 | " (embed_tokens): Embedding(50272, 768, padding_idx=1)\n",
671 | " (embed_positions): OPTLearnedPositionalEmbedding(2050, 768)\n",
672 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
673 | " (layers): ModuleList(\n",
674 | " (0-11): 12 x OPTDecoderLayer(\n",
675 | " (self_attn): OPTAttention(\n",
676 | " (k_proj): Linear(in_features=768, out_features=768, bias=True)\n",
677 | " (v_proj): Linear(in_features=768, out_features=768, bias=True)\n",
678 | " (q_proj): Linear(in_features=768, out_features=768, bias=True)\n",
679 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
680 | " )\n",
681 | " (activation_fn): ReLU()\n",
682 | " (self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
683 | " (fc1): Linear(in_features=768, out_features=3072, bias=True)\n",
684 | " (fc2): Linear(in_features=3072, out_features=768, bias=True)\n",
685 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
686 | " )\n",
687 | " )\n",
688 | " )\n",
689 | " )\n",
690 | " (lm_head): Linear(in_features=768, out_features=50272, bias=False)\n",
691 | ")"
692 | ]
693 | },
694 | "execution_count": 16,
695 | "metadata": {},
696 | "output_type": "execute_result"
697 | }
698 | ],
699 | "source": [
700 | "model"
701 | ]
702 | },
703 | {
704 | "cell_type": "markdown",
705 | "id": "045e7239-cb8a-46dd-815d-e48e7e49eea4",
706 | "metadata": {},
707 | "source": [
708 | "First we build custom linear layer with LoRA as follows."
709 | ]
710 | },
711 | {
712 | "cell_type": "code",
713 | "execution_count": 17,
714 | "id": "77889272-9a93-491b-93cb-b0bed5ce7cd8",
715 | "metadata": {},
716 | "outputs": [],
717 | "source": [
718 | "import math\n",
719 | "from torch import nn\n",
720 | "\n",
721 | "class LoRA_Linear(nn.Module):\n",
722 | " def __init__(self, weight, bias, lora_dim):\n",
723 | " super(LoRA_Linear, self).__init__()\n",
724 | "\n",
725 | " row, column = weight.shape\n",
726 | "\n",
727 | " # restore Linear\n",
728 | " if bias is None:\n",
729 | " self.linear = nn.Linear(column, row, bias=False)\n",
730 | " self.linear.load_state_dict({\"weight\": weight})\n",
731 | " else:\n",
732 | " self.linear = nn.Linear(column, row)\n",
733 | " self.linear.load_state_dict({\"weight\": weight, \"bias\": bias})\n",
734 | "\n",
735 | " # create LoRA weights (with initialization)\n",
736 | " self.lora_right = nn.Parameter(torch.zeros(column, lora_dim))\n",
737 | " nn.init.kaiming_uniform_(self.lora_right, a=math.sqrt(5))\n",
738 | " self.lora_left = nn.Parameter(torch.zeros(lora_dim, row))\n",
739 | "\n",
740 | " def forward(self, input):\n",
741 | " x = self.linear(input)\n",
742 | " y = input @ self.lora_right @ self.lora_left\n",
743 | " return x + y"
744 | ]
745 | },
746 | {
747 | "cell_type": "markdown",
748 | "id": "954e2c9d-545e-4bd9-9b0f-eba3fe29a1de",
749 | "metadata": {},
750 | "source": [
751 | "Replace targeting linear layers with LoRA layers."
752 | ]
753 | },
754 | {
755 | "cell_type": "code",
756 | "execution_count": 18,
757 | "id": "baf8a748-a3e3-45b8-9c64-252c56abe923",
758 | "metadata": {},
759 | "outputs": [],
760 | "source": [
761 | "lora_dim = 128\n",
762 | "\n",
763 | "# get target module name\n",
764 | "target_names = []\n",
765 | "for name, module in model.named_modules():\n",
766 | " if isinstance(module, nn.Linear) and \"decoder.layers.\" in name:\n",
767 | " target_names.append(name)\n",
768 | "\n",
769 | "# replace each module with LoRA\n",
770 | "for name in target_names:\n",
771 | " name_struct = name.split(\".\")\n",
772 | " # get target module\n",
773 | " module_list = [model]\n",
774 | " for struct in name_struct:\n",
775 | " module_list.append(getattr(module_list[-1], struct))\n",
776 | " # build LoRA\n",
777 | " lora = LoRA_Linear(\n",
778 | " weight = module_list[-1].weight,\n",
779 | " bias = module_list[-1].bias,\n",
780 | " lora_dim = lora_dim,\n",
781 | " ).to(device)\n",
782 | " # replace\n",
783 | " module_list[-2].__setattr__(name_struct[-1], lora)"
784 | ]
785 | },
786 | {
787 | "cell_type": "markdown",
788 | "id": "8aae2df9-fae7-4ecc-8260-80e8e578d951",
789 | "metadata": {},
790 | "source": [
791 | "See how model is changed."
792 | ]
793 | },
794 | {
795 | "cell_type": "code",
796 | "execution_count": 19,
797 | "id": "bf16b414-b973-40eb-be81-fd2aa3dde439",
798 | "metadata": {},
799 | "outputs": [
800 | {
801 | "data": {
802 | "text/plain": [
803 | "OPTForCausalLM(\n",
804 | " (model): OPTModel(\n",
805 | " (decoder): OPTDecoder(\n",
806 | " (embed_tokens): Embedding(50272, 768, padding_idx=1)\n",
807 | " (embed_positions): OPTLearnedPositionalEmbedding(2050, 768)\n",
808 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
809 | " (layers): ModuleList(\n",
810 | " (0-11): 12 x OPTDecoderLayer(\n",
811 | " (self_attn): OPTAttention(\n",
812 | " (k_proj): LoRA_Linear(\n",
813 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n",
814 | " )\n",
815 | " (v_proj): LoRA_Linear(\n",
816 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n",
817 | " )\n",
818 | " (q_proj): LoRA_Linear(\n",
819 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n",
820 | " )\n",
821 | " (out_proj): LoRA_Linear(\n",
822 | " (linear): Linear(in_features=768, out_features=768, bias=True)\n",
823 | " )\n",
824 | " )\n",
825 | " (activation_fn): ReLU()\n",
826 | " (self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
827 | " (fc1): LoRA_Linear(\n",
828 | " (linear): Linear(in_features=768, out_features=3072, bias=True)\n",
829 | " )\n",
830 | " (fc2): LoRA_Linear(\n",
831 | " (linear): Linear(in_features=3072, out_features=768, bias=True)\n",
832 | " )\n",
833 | " (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
834 | " )\n",
835 | " )\n",
836 | " )\n",
837 | " )\n",
838 | " (lm_head): Linear(in_features=768, out_features=50272, bias=False)\n",
839 | ")"
840 | ]
841 | },
842 | "execution_count": 19,
843 | "metadata": {},
844 | "output_type": "execute_result"
845 | }
846 | ],
847 | "source": [
848 | "model"
849 | ]
850 | },
851 | {
852 | "cell_type": "markdown",
853 | "id": "e9099c08-f6a6-45f8-939b-cc3ed9415976",
854 | "metadata": {},
855 | "source": [
856 | "Finally, freeze all parameters except for LoRA parameters."
857 | ]
858 | },
859 | {
860 | "cell_type": "code",
861 | "execution_count": 20,
862 | "id": "81d06bba-955b-4806-8ff7-f217252e3268",
863 | "metadata": {},
864 | "outputs": [],
865 | "source": [
866 | "for name, param in model.named_parameters():\n",
867 | " if \"lora_right\" in name or \"lora_left\" in name:\n",
868 | " param.requires_grad = True\n",
869 | " else:\n",
870 | " param.requires_grad = False"
871 | ]
872 | },
873 | {
874 | "cell_type": "code",
875 | "execution_count": null,
876 | "id": "6c0a4469-2827-4f30-9324-711a9feea1ae",
877 | "metadata": {},
878 | "outputs": [],
879 | "source": [
880 | "### Do this when you run adapter fine-tuning on Hugging Face framework\n",
881 | "# model.gradient_checkpointing_enable()\n",
882 | "# model.enable_input_require_grads()"
883 | ]
884 | },
885 | {
886 | "cell_type": "markdown",
887 | "id": "6d6c7d6f-6c50-4839-88a5-c851caab9ba2",
888 | "metadata": {},
889 | "source": [
890 | "## Fine-tune"
891 | ]
892 | },
893 | {
894 | "cell_type": "markdown",
895 | "id": "a12b875f-36cc-40b8-aaab-1efda68710f3",
896 | "metadata": {},
897 | "source": [
898 | "Now let's start to run fine-tuning.\n",
899 | "\n",
900 | "First we build optimizer as follows."
901 | ]
902 | },
903 | {
904 | "cell_type": "code",
905 | "execution_count": 21,
906 | "id": "bb51298a-2d55-466c-a990-0ea08a247350",
907 | "metadata": {},
908 | "outputs": [],
909 | "source": [
910 | "optimizer = torch.optim.AdamW(\n",
911 | " params=model.parameters(),\n",
912 | " lr=1e-3,\n",
913 | " betas=(0.9, 0.95),\n",
914 | ")"
915 | ]
916 | },
917 | {
918 | "cell_type": "markdown",
919 | "id": "d37db1a8-0053-4acc-94ce-89d87c78942e",
920 | "metadata": {},
921 | "source": [
922 | "In this example, we build cosine scheduler for training."
923 | ]
924 | },
925 | {
926 | "cell_type": "code",
927 | "execution_count": 22,
928 | "id": "6f95bdf6-4498-4d40-90aa-1267d55f38c3",
929 | "metadata": {},
930 | "outputs": [],
931 | "source": [
932 | "from torch.optim.lr_scheduler import LambdaLR\n",
933 | "\n",
934 | "num_epochs = 2\n",
935 | "\n",
936 | "num_update_steps = math.ceil(len(dataloader) / batch_size / gradient_accumulation_steps)\n",
937 | "def _get_cosine_schedule(\n",
938 | " current_step: int,\n",
939 | " num_warmup_steps: int = 0,\n",
940 | " num_training_steps: int = num_epochs * num_update_steps\n",
941 | "):\n",
942 | " if current_step < num_warmup_steps:\n",
943 | " return float(current_step) / float(max(1, num_warmup_steps))\n",
944 | " progress = float(current_step - num_warmup_steps) / float(max(1, num_training_steps - num_warmup_steps))\n",
945 | " return max(0.0, 0.5 * (1.0 + math.cos(math.pi * progress)))\n",
946 | "scheduler = LambdaLR(optimizer, lr_lambda=_get_cosine_schedule)"
947 | ]
948 | },
949 | {
950 | "cell_type": "markdown",
951 | "id": "a9f9e828-c4fb-493d-a6de-78e03dbf035e",
952 | "metadata": {},
953 | "source": [
954 | "Run fine-tuning."
955 | ]
956 | },
957 | {
958 | "cell_type": "code",
959 | "execution_count": 23,
960 | "id": "75d22125-830a-4ec6-8417-cdb8a97ec559",
961 | "metadata": {},
962 | "outputs": [
963 | {
964 | "name": "stdout",
965 | "output_type": "stream",
966 | "text": [
967 | "Epoch 1 42/42 - loss: 1.0724\n",
968 | "Epoch 2 42/42 - loss: 1.3185\n"
969 | ]
970 | }
971 | ],
972 | "source": [
973 | "from torch.nn import functional as F\n",
974 | "\n",
975 | "if os.path.exists(\"loss.txt\"):\n",
976 | " os.remove(\"loss.txt\")\n",
977 | "\n",
978 | "for epoch in range(num_epochs):\n",
979 | " optimizer.zero_grad()\n",
980 | " model.train()\n",
981 | " for i, (inputs, labels, masks) in enumerate(dataloader):\n",
982 | " with torch.set_grad_enabled(True):\n",
983 | " outputs = model(\n",
984 | " input_ids=inputs,\n",
985 | " attention_mask=masks,\n",
986 | " )\n",
987 | " loss = F.cross_entropy(outputs.logits.transpose(1,2), labels)\n",
988 | " loss.backward()\n",
989 | " if ((i + 1) % gradient_accumulation_steps == 0) or \\\n",
990 | " (i + 1 == len(dataloader)):\n",
991 | " optimizer.step()\n",
992 | " optimizer.zero_grad()\n",
993 | " scheduler.step()\n",
994 | "\n",
995 | " print(f\"Epoch {epoch+1} {math.ceil((i + 1) / batch_size / gradient_accumulation_steps)}/{num_update_steps} - loss: {loss.item() :2.4f}\", end=\"\\r\")\n",
996 | "\n",
997 | " # record loss\n",
998 | " with open(\"loss.txt\", \"a\") as f:\n",
999 | " f.write(str(loss.item()))\n",
1000 | " f.write(\"\\n\")\n",
1001 | " print(\"\")\n",
1002 | "\n",
1003 | "# save model\n",
1004 | "torch.save(model.state_dict(), \"finetuned_opt.bin\")"
1005 | ]
1006 | },
1007 | {
1008 | "cell_type": "markdown",
1009 | "id": "83993d92-d7ed-4a07-8985-cc59bd4e4fef",
1010 | "metadata": {},
1011 | "source": [
1012 | "> Note : Here we save LoRA-enabled model without any changes, but you can also merge the trained LoRA's parameters into the original linear layer's weights."
1013 | ]
1014 | },
1015 | {
1016 | "cell_type": "markdown",
1017 | "id": "1bc086e5-e93f-4264-a8fa-6428f844ac3c",
1018 | "metadata": {},
1019 | "source": [
1020 | "Show loss transition in plot."
1021 | ]
1022 | },
1023 | {
1024 | "cell_type": "code",
1025 | "execution_count": 25,
1026 | "id": "e37c5aee-38d4-4a2a-952c-4fd2bef41e2b",
1027 | "metadata": {},
1028 | "outputs": [
1029 | {
1030 | "data": {
1031 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFbUlEQVR4nO3dd1hT5+IH8G/CCCBTlCWguAdq3bNVK3VhWzu0w7Z22qG32qHV62itVez4eVut1Y5btb2Oaqt2OWrdWBVBUNxbEEVcDEFm3t8fQEzIBJNzgPP9PA/PQ07e5Lw5hJxv3nVUQggBIiIiIomo5a4AERERKQvDBxEREUmK4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkGD6IiIhIUgwfREREJClnuStQkVarxaVLl+Dl5QWVSiV3dYiIiMgGQgjk5OQgJCQEarXlto1qFz4uXbqEsLAwuatBREREVZCamorQ0FCLZapd+PDy8gJQWnlvb2+Za0NERES2yM7ORlhYmO48bkm1Cx/lXS3e3t4MH0RERDWMLUMmOOCUiIiIJMXwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQphg8iIiKSFMMHERERSYrhg4iIiCTF8EFERESSYvggIiIiSTF8EBERkaQUFz7WHLiIHSevyl0NIiIixap2V7V1pDNXb+HtVQcBAOfnRMtcGyIiImVSVMtHela+3FUgIiJSPEWFj2KtkLsKREREiqeo8LH/3A25q0BERKR4igofX247rfv9ak6BjDUhIiJSLkWFD327T1+TuwpERESKpNjwQURERPJQbPi4XVQidxWIiIgUSbHh40zGLbmrQEREpEiKDR8lgtNuiYiI5KDY8EFERETyUGz46NKortxVICIiUiTFhg+1Su4aEBERKZNiwwcRERHJQ7HhQ6Vi0wcREZEcFBU+vDTOut/rebrKWBMiIiLlUlT4ePnexnJXgYiISPEUFT5cnNnVQkREJDdFhQ8iIiKSn2LDBxc4JSIikkelw8fOnTvx4IMPIiQkBCqVCuvWrTO4XwiB6dOnIzg4GO7u7oiKisKpU6fsVV8iIiKq4SodPnJzc9G+fXssWLDA5P2ffPIJ5s2bh0WLFmHfvn2oU6cOBg4ciPz8/LuuLBEREdV8ztaLGBo8eDAGDx5s8j4hBD7//HNMnToVDz/8MADghx9+QGBgINatW4cnn3zy7mprR+x1ISIikoddx3ycO3cO6enpiIqK0m3z8fFBt27dsGfPHnvuioiIiGqoSrd8WJKeng4ACAwMNNgeGBiou6+igoICFBQU6G5nZ2fbs0pERERUzcg+2yUmJgY+Pj66n7CwMEn2y9kuRERE8rBr+AgKCgIAXLlyxWD7lStXdPdVNHnyZGRlZel+UlNT7VklIiIiqmbsGj4iIiIQFBSELVu26LZlZ2dj37596NGjh8nHaDQaeHt7G/w4Cls7iIiI5FfpMR+3bt3C6dOndbfPnTuHpKQk1K1bF+Hh4Rg/fjw++ugjNGvWDBEREZg2bRpCQkIwbNgwe9b7rgkmESIiIllUOnzEx8ejX79+uttvv/02AGDUqFFYsmQJJk6ciNzcXIwePRqZmZno3bs3Nm7cCDc3N/vVmoiIiGqsSoePvn37Wmw1UKlU+PDDD/Hhhx/eVcWIiIiodpJ9totc2OlCREQkD8WGDyIiIpIHwwcRERFJiuGDiIiIJKXY8MGZtkRERPJQbPggIiIieTB8EBERkaQUGz4EJ9sSERHJQrHhg4iIiOTB8EFERESSUm74YK8LERGRLJQbPoiIiEgWDB9EREQkKcWGD/a6EBERyUOx4YOIiIjkwfBBREREklJs+OC1XYiIiOShqPAhmDiIiIhkp6jwQURERPJj+CAiIiJJKTZ88MJyRERE8lBs+CAiIiJ5MHwQERGRpBQbPjjxhYiISB6KDR+5BcVyV4GIiEiRFBs+Xl92QO4qEBERKZJiwwcRERHJQ1Hhg+M8iIiI5Keo8EFERETyY/ggIiIiSTF8EBERkaQYPoiIiEhSDB9EREQkKYYPIiIikhTDBxEREUmK4YOIiIgkxfBBREREklJU+OACp0RERPJTVPggIiIi+TF8EBERkaQYPoiIiEhSDB9EREQkKYYPIiIikhTDBxEREUmK4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkFBU+BNdXJyIikp2iwgcRERHJj+GDiIiIJMXwQURERJJi+CAiIiJJKTZ8uLs4yV0FIiIiRVJs+CAiIiJ52D18lJSUYNq0aYiIiIC7uzuaNGmCmTNnQlSzea4C1as+RERESuFs7yf8+OOPsXDhQixduhRt2rRBfHw8XnjhBfj4+ODNN9+09+6IiIiohrF7+Pjnn3/w8MMPIzo6GgDQqFEjrFixAnFxcfbe1V2pZg0xREREimH3bpeePXtiy5YtOHnyJADg4MGDiI2NxeDBg02WLygoQHZ2tsGPo7CrhYiISH52b/mYNGkSsrOz0bJlSzg5OaGkpASzZs3CyJEjTZaPiYnBjBkz7F0NqxhDiIiI5GH3lo9Vq1Zh2bJlWL58OQ4cOIClS5fis88+w9KlS02Wnzx5MrKysnQ/qamp9q4SERERVSN2b/mYMGECJk2ahCeffBIA0LZtW1y4cAExMTEYNWqUUXmNRgONRmPvaljHpg8iIiJZ2L3lIy8vD2q14dM6OTlBq9Xae1dERERUA9m95ePBBx/ErFmzEB4ejjZt2iAxMRFz587Fiy++aO9d3RUOPiUiIpKH3cPH/PnzMW3aNLzxxhvIyMhASEgIXn31VUyfPt3euyIiIqIayO7hw8vLC59//jk+//xzez+1XXGdDyIiInnw2i5EREQkKcWGj2KtQImWzR9ERERSU1T4qNjV8mfyZXkqQkREpGCKCh8VZeYVyl0FIiIixVF0+NCy24WIiEhyig4f6dkFcleBiIhIcRQdPhbtOCN3FYiIiBRH0eGDiIiIpMfwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQpRYUPLilGREQkP0WFDyIiIpIfwwcRERFJiuGDiIiIJMXwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQphg8iIiKSlLLChzBc47R747oyVYSIiEi5lBU+iIiISHaKDh++7q5yV4GIiEhxFB0+NC6KfvlERESyUPTZV/Ayt0RERJJTdPi4cCMP41Ym4nBaltxVISIiUgxFh4+DqZn4NekShs6PlbsqREREiqHo8EFERETSY/ggIiIiSTF8EBERkaQUFT7KJ7eoVbJWg4iISNEUFT7KqVRMH0RERHJRZviQuwJEREQKpszwwfRBREQkG2WGD7Z9EBERyUaR4YPZg4iISD6KDB/MHkRERPJRZvhg+iAiIpKNMsMH2z6IiIhko8zwwexBREQkG0WFD1G2xGleYYm8FSEiIlIwRYUPIiIikh/DBxEREUmK4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkFBk+PFyd5K4CERGRYikyfIzoHCZ3FYiIiBRLkeGDiIiI5MPwQURERJJSVPgQECa3B3hpJK4JERGRcjkkfKSlpeGZZ56Bv78/3N3d0bZtW8THxztiV1VS8cJyHcP95KkIERGRAjnb+wlv3ryJXr16oV+/ftiwYQPq16+PU6dOwc+PJ3giIiJyQPj4+OOPERYWhsWLF+u2RURE2Hs3dmWuO4aIiIjsz+7dLr/99hs6d+6M4cOHIyAgAB06dMC3335rtnxBQQGys7MNfqR2u0gr+T6JiIiUyu7h4+zZs1i4cCGaNWuGTZs24fXXX8ebb76JpUuXmiwfExMDHx8f3U9YmPRrcOw8eVXyfRIRESmV3cOHVqtFx44dMXv2bHTo0AGjR4/GK6+8gkWLFpksP3nyZGRlZel+UlNT7V0lIiIiqkbsHj6Cg4PRunVrg22tWrVCSkqKyfIajQbe3t4GP0RERFR72T189OrVCydOnDDYdvLkSTRs2NDeuyIiIqIayO7h46233sLevXsxe/ZsnD59GsuXL8c333yDMWPG2HtXdlVcwkGnREREUrB7+OjSpQvWrl2LFStWIDIyEjNnzsTnn3+OkSNH2ntXlSYszKhdcyBNuooQEREpmN3X+QCAoUOHYujQoY54artQQWW0LfVmngw1ISIiUh5FXduFiIiI5MfwQURERJJi+CAiIiJJMXyUsTQYlYiIiOyH4YOIiIgkpdjw0SaEK6kSERHJQbHhY8Xo7nJXgYiISJEUGz683VwMbgtw0AcREZEUFBU+GC+IiIjkp6jwUU5lvMApTmfckr4iRERECqTI8GHKjdxCuatARESkCIoOH72b1tP9rjLVHEJERER2p+jwodVbWcyJ4YOIiEgSig4fRSVa3e9qRR8JIiIi6Sj6lHt/y0Dd7yqw5YOIiEgKig4fz/dspPvdU+MsX0WIiIgURNHhw93VSfe7q7OiDwUREZFkeMYt079VgNxVICIiUgTFh4+ujeoCAJw54pSIiEgSijrjClPrq5eNM+W1XYiIiKShqPBRTmXid5PBhIiIiOxOkeFDn0rX8kFERERSYPgoa/sQbPogIiKSBMMH1xYjIiKSFMNHebcLGz6IiIgkofjwoS5LH5ztQkREJA3Fh49ybPkgIiKShuLDR4m2NHXsPHlV5poQEREpg+LDxz9nrgMA1iVdkrkmREREyqCo8MFxHURERPJTVPgox+m1RERE8lFk+CAiIiL5MHzo4SqnREREjsfwoed2UYncVSAiIqr1GD70qDkYhIiIyOEYPvQwexARETkewwcRERFJiuGDiIiIJMXwoYeTXYiIiBxPWeGD4YKIiEh2ygofZVQcWUpERCQbRYYPIiIikg/DBxEREUmK4UMPB5wSERE5HsMHERERSYrhQ4/gdBgiIiKHY/ggIiIiSTF8EBERkaQYPvRwwCkREZHjMXwQERGRpBQVPtiwQUREJD9FhY9y5hZXZzghIiJyPEWGDyIiIpKP4sPHhnH36n4XHHFKRETkcIoPH43r15G7CkRERIri8PAxZ84cqFQqjB8/3tG7IiIiohrAoeFj//79+Prrr9GuXTtH7sZu2OlCRETkeA4LH7du3cLIkSPx7bffws/Pz1G7uWsqs3NfiIiIyBEcFj7GjBmD6OhoREVFWSxXUFCA7Oxsgx+5cLwpERGR4zk74klXrlyJAwcOYP/+/VbLxsTEYMaMGY6ohk1UbPggIiKSlN1bPlJTUzFu3DgsW7YMbm5uVstPnjwZWVlZup/U1FR7V0mHU2mJiIjkZ/eWj4SEBGRkZKBjx466bSUlJdi5cye+/PJLFBQUwMnJSXefRqOBRqOxdzUs4xKnREREsrF7+Ojfvz+Sk5MNtr3wwgto2bIl3nvvPYPgUR2w14WIiEhadg8fXl5eiIyMNNhWp04d+Pv7G22vbgSbPoiIiBxO8SucqjjilIiISFIOme1S0fbt26XYDREREdUAim/50MfJMERERI6n+PDBThciIiJpKT586EvPzpe7CkRERLUew4eewV/skrsKREREtZ6iwoepMR2c7EJERCQtRYUPbVn4cGLiICIiko3Cwkdp+lDrhQ+u80FERCQtRYWPkrKmD7WagYOIiEguigofpzJyAACWskduQbFEtSEiIlImxYSP3IJi7D17A8CdsR+m/LQ/VaIaERERKZNiwseN3ELd7z/tTzFbrsRSMiEiIqK7ppjwoT/O40p2gdlyGTlcaIyIiMiRFBM+bJ1ee+1WofVCREREVGWKCR+c4EJERFQ9KCZ82HoFOWYUIiIix1JM+Ei9cVvuKhAREREUFD44i4WIiKh6UEz4sHkVdfa7EBEROZRywofN5Zg+iIiIHEk54YOZgoiIqFpQTPiwte2DIYWIiMixFBM+GCqIiIiqB8WEDy+Ns+73IW2DZKwJERGRsikmfDQL9NL97qRWzMsmIiKqdhR5Fra01Dp7Z4iIiBxLUeHDzaX05fZtUd9ge++m9XS/r0tKw1s/JSG/qETSuhERESmFs/UitcfOCf2QnJaFfi0CzJYpKhFYm5gGD1cnPHxPA3SNqCthDYmIiGo/RbV8BHi7oX+rQKgr9LuYmgmzbF8KRny9BwdTM6WpHBERkUIoKnxURWLKTbmrQEREVKswfFih4gIhREREdsXwQURERJJi+LCCDR9ERET2xfBBREREkmL4ANAyyMt6ISIiIrILhg8AT3QJN3sfe12IiIjsi+EDgJOl9daJiIjIrhg+rOGIUyIiIrti+AAghJC7CkRERIrB8GEF2z2IiIjsi+EDANs9iIiIpMPwAaCep0buKhARESkGwwcAH3cXs/ddyrwtYU2IiIhqP4YPK77afkbuKhAREdUqDB82WLDtNN77+RBnxRAREdkBw4cNPt10Aj/FpyIxNVPuqlRKwoUb2HYiQ+5qEBERGXCWuwI1SWZeodxVqJTHFu4BAOyZfD+Cfdxlrg0REVEptnxUwotL4uWuQpVkZBfIXQUiIiIdhg87yC0oRlJqJseEEBER2YDho0wD36p3Szy28B8MW7Ab65LS7Fgj++HlaYiIqDph+Cjz3ajOVX7s8fQcAMCaA9UzfCiNEAJ7zlxHRk6+3FUhIiITOOC0TKtg70qV12oFpqxLrvTjyPF2nLyK5xfvh5NahTOzh8hdHSIiqoDho4piT1/DirhUuashmcNpWZi7+SQmDmqBlkHVO3DtPHkNAFCi5RgcIqLqiN0uVZSdX2S0TVVNB1eoKnlt3vyiEjz1zV58vePO6q6PL/oHW49n4Mlv9tq7ekREpDAMH1X0few5yfeZk1+Ew2lZDp9Vsyo+FXvOXkfMhuO6bflFWgBAZp5x6CIiIqoMho9KOnklB4XFWhxIyZR834M+34Wh82Ox/eRVh+4nt6BE97stQWdlXAo+3XTcYpm3VyXhme/2QcuuECIixbN7+IiJiUGXLl3g5eWFgIAADBs2DCdOnLD3bmQz8POdmPTLIZP3pWc59gq4aWVX2N2QfNlq2btpHRG489gjl7Ktlp+0JhkLtp3BQQvLz685kIbY09dwLN368xGRdfHnb+CoDf+fVP1l5xfhRNmsSaWwe/jYsWMHxowZg71792Lz5s0oKirCgAEDkJuba+9dyUIIYE2i6Sm1pzNumX3cqvhUfPTHUVzNMV5t9HLWbRSXaA22nUjPMTmuxHLdBIoqPA9Qus7HtuMZ2HQkvVLPBwAFxcbPZ44t9eU6bER379qtAjy+aA+GzNsld1XIDvp8sg0DP9+JhAs35K6KZOw+22Xjxo0Gt5csWYKAgAAkJCTgvvvus/fuqhVzPQoZ2fmY+HNpa8l3secw+r7GaB7ohcc7hWLv2et48pu96NqoLsbe3xT+nq7IL9LisYX/wNfDBUnTBxg9X8UBpFdzChB7+irWJ6djz5nr2P3e/fByu/OnLSzR4oUl+wEAidMegJuLE9xdncy+DsOAYPyitFoBtdp4EKstA1vvNnykZd5GiI9btR3cSySF9CyuYVOb3CwbS/f3sQx0alhX5tpIw+FTbbOysgAAdeuaPqAFBQUoKLjTGpCdXbObEYUQUKlU2H36GsL8PBDu74ErFa6t8s3OswCALo38sGxfCgAg7vwNPPd9HADg9b5NANg+uLPLrL8Nbrf/8C+E+LjpbheX3DnjD50fi7TM29g/JQr1vTRGz5VbUIxPN1nuJuswczMe7xSKaUNb21Q/W9zILcTXO89geKcwNA3wNFlmVXwqJv58CCM6h+L5nhFoEeQFp7IQdDA1E+/9cghToltxRVciomrOoQNOtVotxo8fj169eiEyMtJkmZiYGPj4+Oh+wsLCHFklh4uYvB5vr0rCyO/24b5Pt1kse+1WIfIKio22V+yCqYpLet+M9E/G5eNGNprpgvliy6kKW4zP5Fm3i/DfKs72ESZaUoQQeO+XQ/h6x1kM/mKnbvusP4/ika92o6C4dADs3L9OAgBWxV/EkHm7MO3Xw7qyz30fh+PpOXj2v3FVqldiyk2cvKKsPldT7PHeq+5qw2usbQFbCIH/bD6Jv49ekbsqJBGHho8xY8bg8OHDWLlypdkykydPRlZWlu4nNVW+hbuSpj9gl+exdZn1zUevYMvxDKPt3+66c2J/cH4ssiq0gJg6gVeamf6Ps1crjs2xfV9x52/oBrrmF5Vg+b4UXLYyCHd98mV0mfU3Npd96BSVCN1zfLvrHBJTMrHpyJWymhjWZXlZqxFQGojMOZByEzsqzBDSH5B77VYBHvnqHwz4z86KD1WUy1m3EfnBJkxeY3pAtaMJIRw+jXzPmetoPnUDluyWfqq8JZVdEK+2jJ0q0QoUFmux5VgGvthyCi//UHrlcK1W4O2fkqr8JYeqP4eFj7Fjx+KPP/7Atm3bEBoaaracRqOBt7e3wY9cfD1cJd3fIr1FvMxJTsvCN7sMy207cRWv/ZiAofNtG2xm6kvSuqRLaDTpT8z9y7CL5W4+/OdtOYWus7fg16Q0DF+0B/9em4yh82IrPL/hY95YdgDXbhUabFv6z3mD2wnnb2DNgYtV/sB99Kt/MOr7OF0QOnctF50++hsLtp0GAFzOrHz/+er4VDz3fRxyKjkouDpbvPs88ou0sqzcK4TAs/+Nw0Nf7rbLdOzCYi2W7buAlOt5BtvH/5QIrQA++P3oXe/DXk5n3EK7Dzbh879Pyl0VyUXN3YFOMzfjZMadVkchBLadyMCaxDTM/MP030kIgeSLWbXq/w+oPaHSFnYPH0IIjB07FmvXrsXWrVsRERFh713UKNdvFdz11W5z8ovR//+2625fzSnAxiPpOJyWjel6XQ/m5BWWGG1LuHATADBv62mD7Vqjd3/l2nev5hRg3MokJKeVjvW5nltostzY5QcwZvkBk/ct3HFG1z0EAEv3XMDbqw4iw8RMIVNyTXRlAXcG6c368xhu5BbqxrZUpQl7ws+HsPPkVZsCpL3kF5XgoS9jMWeD5TVVqspS8LxVUKybCphbUIy8QtPHuLKybhdhZVwKbuQWIvb0NSSnZWH0jwk4fpdTsr/ZeQZT1h622vVZHcSsP4bcwhJ8/nfFLk/zaku3y7lrucgpKMZOvZZJrSh9v5lTVKLFB78dwYNfxtb42T6H07LwRSX+7rWJ3QecjhkzBsuXL8evv/4KLy8vpKeXji3w8fGBu3vVL1tfUw1ftAdnr93dNOMf9lyo0n3lygeymvPT/hQ0DfBCp4Z+RjN2thy7gk4N/Uw+bvHucwj2qdzfVADIzCvEH4fMr1VyJbsAMeuP2fyc6yuse/JTvOlv7sbBytjMP45iwsAWcHMxPRtICGFwjLJvm/6QLNEK5BYWw9vNxeo+bfVb0iUcupiFQxezMGlwS7s976Yj6Vi04wwCTAxALvfA3B24nJWPH1/qqhtXc2b2EN2A36oQQuBfKxKx8+RVg67Kv49dwd/HruD8nGibn6uwWIvfD15Cr6b1EOTjhr1nHTdlMa+wGEcvZaNjuJ/JWV9kTAiBq7cKEODlZvJ+/ZlyyWlZ2GqiO7rcJxuPY2nZ517qDceureRoQ+fHWi9US9m95WPhwoXIyspC3759ERwcrPv56aef7L2rGuFug4cU3vslGY8t/Adx524YnaC/2m7+m/2M34/itf8lVGpfF2/m4dtdZ62W2336mk3P968ViXhjmWELiv5L2HXK8BsVAFzNMd/N8t/Yc/h2p/n6vbBkP3rO2aK7beob6LlruWg5bQPaffAXLt4sbfY/nJaF0T/E43RG1Qe1FjpooOSrPyYYjK0x5XJZq5H+OJtZfx6rcgvF6YwcdJ29RfeNN+783YWFL7edxjurD2LQF44ftzPyu314fNEe/LDnvMP3ZQ+3CoqxOj4VmXmmWyEtEULgg9+O4Me91r/kWBKz4Ti6ztqClXEpVssOW7AbvyZdMnv/Uhu+cCmBo8dHOZpDul1M/Tz//PP23hXZ2Yiv9+DUFeOF0gorsdCYKfp9+GOXJ2LBNutdFbZ2+/9+0PyHFACD2S/l9Th4McviYy7cyDN73/YTVw2mTucXGXdp9ftsO4rKpjdvSC5t+Xvoy1j8dfQKoubW7EGte89e1/3+/e5zGPR51Zq9p6w9bHLBvaraVvZN2dr0dP1v2FX98E4su7TCT/EXq/R4e580rD3fpF8OYcLPh/DS0vhKP/f+8zex5J/zmLbOeveuJeXLC5gbw3E3bhUUY/Huc1YHt1cH1j5Lbe1O23Y8A11nbzH4clXT8NouZCA927hVoPnUDXf1nCVV+LC1NHulqs5fz8Uz3+0z2l7xH/7ctVxklLWOZN0uMhkwyl2ycbBqxTBl65iJ/KIS3VTj6uCmnS4sWNnZHdZUnAmVa+b46pd74mvLV2jeffoaDl3MNL/PCu/r9Kx8zP3rhMUFwKasTUafT7ebHZf06abjFt9v5fRD1Ffbz+gGT5tS3sVZPs6rXHZ+Ef639wKu3zIfArOt/B/O3XwSD30Za/P7ObewBBsPpxu9xrsZwzLjtyOY8ftRPLLgn6o/SRX8cegSXlyy36a/FwBM//Uwmk/dYLH109aPyheW7MfVnIIqLS2QnpWPzzadkD2sMXyQw+0/Vz2WDH7vl2TEmujO+TnB8BtswoWb6DprC7LyitB+xl/oHlPazWKvay/8cegSWk/fZHWwalGJFu0++AudZv5dpRkgV3MKMH5lIuIqHP/YU9fwq5VB0PrrLdw2MWBZn/5JeOvxKxg6f5fVNVOKHXyBwUS9Cz+ev5aLTUfSIYQwaLWKO38De89eNxmELmfdxsjv9uGhL3fbvM/nF8dh3tbTeGnpfrNllu1LQcqNPINB6Pon3gXbzljs9iunH6I+3XQCn246gTHLDmDG70eMypo7sb/38yFMXXcYoxYbn8Bu5BYavecOpNw0KjdvyykcupiFVfttnyH12v8SMOP3owbvG1sHk5tSfqFNU1+cHGns8kRsPZ6BltM2Wi+MO+PzvtxqPiiaU6IVZkPOP2WfaSvjUvDg/FhkWDkOzy+Ow5fbTuOFxebfp1Jg+CCHe9pEa0N1sWT3OSzefd7kfX0/K50pkZlXhIs38zDwc+MukxNWTrLzt57CXxUWdHtn1UEAMJi1IoTQjQ8pl3IjD4UlWtwqKEZ2fpHBh8qiHWd0XQ2pN/JMfuBMXZeMdUmXMOLrPQbbn/nvPoxbmYSzV81fi+jlH+JxNacARSVaqyHp3k+24d3VB5FfVIIXl8TjcFo2XrcyFijJwkUIK2Pf2euY+PNBo+4W/YGwfT/bjld/TDB5Negnv9lrcgySrS1a+o6XhVNbLsZoyWkLf5dypmaw/Zl8uWy6tOF9TmbSx4bDpe/Lw2mG9T2QchMdZ27Gyz/EGwSXR78y37JQpLeKck5+EZJSMy12B62IS0F+0Z0uCEvXxTJix9z6+8FLJi/O93PCRWw7cWfQa8r1PLu1QFal+kPnx6LltI0mW4TLP18nrUlGcloWPt5ovEJ1QXEJLt4sfQ3l79PjMl/IzuHLqxNVZ5bWe9DvYnh84R6TZcrHLZRoBSb+fAgNfA1H82fnF2P0j+ZPxDtPXsXaxDS4OKmwKv4ipg1tjZd6l05P7/9/O3TloufFGkw/Lg8uB98fgHs/KQ1JFWeHXKiwxkVWXpHBt/KMnAI0rm96KXsAeOrbvTh/LddqK8XFm7fxc8JFgxakLBOzgPKLSuCsVsHZybbvPH8fvQInJxX6tQgwuk+rFXhx6X5sP2G6z9vVSY3bWsOTxV9mBtSu2p+K1/o0salOd6NiKCi/FENlFZdoMXyR6fejKWqVCpU55ZWH8a3HM/BM93CD+1Ku5yHc38PoMeUv42BqJh5ecKe1aHxUM7PHttV021oMth3PQGJqJkb1aAh/T+MZWVUdO7TtRAb+tSIRwJ3/ne92ncWF63m6Abbn50Rj39nreOKbvWgV7I0N4+6t1D7+OpIOV2c1+pp4D1fGsculAWnPmesYFBlkseztomKj91b0vNjKBTwJMHwQ2cBSk26JVmDu5hP45UDlByBWnAYds/4YXuodYdQVoB889Om3luh/4GTlFRl9s/lqx2nEXzBuOjfHnh9WSamZGFZ2UrqveX2bHlO+2uXJjwbD1dkwsCSk3DQbPADA2cQU2GtmxjZYOy1PXpOMWcMiKz2tduPhdBSWaNE62Buhfu5o98FfuvumrD2MKWsP23QyO5iaicW7z2HioJYI8XWv1HioEq1AkfZOC4OlwFNYrMWXW09ZHMS98chl9GsRgKYBngbPU97IoR88AODzv09Vav0SU8ovipmUmokfXuxq0wrPWXlFOHI5C90j/E3+3a7mFBh1O2TmFeKjP42n+Jf/X5cHAFvdzC3UffHYNP7ORVXvZrzx+eumZ0/qt05uOZaBiMnrMa5/M7z1QHMA9v1ftheGD6K71OTf6ytV3tJnT3krw4TVB216LrXeCaBHzFa8PaA5dp68anIdlcxc45PWb1ZmC1VVebV2nryKXaeuGlwyYKeJ7g9LirVanL50C2sTL8LFSY3ezeohJ9/KAEcT51dz42ZKtMLiSXlFXAoGtA5Ev5a2f3vNLyqxaRr6pDXJOFihC6rijIjyE/qlzHyseq2H1ee8eDMPTQO8AAAD/rPD4GTX77Pt6Bbhj48fb2fwmDbTNyLXRFdOxRPl7PXHMXv9cTzaoQGm6l1Yctb6Y3e9mKI1+/RmWplzOC0LkQ18MGTeLqRl3sZnw9vj8U6hEKK0ZTLAW4MJA1tintE1rAy7jszZdCQdX/x9Cp5uhqfOrNtFePXHeDzUvgFcnFToGmF4IVX9LltzIdgWczYcN9mSpN+FW1D2/vliyyld+KiOGD6IJGZtul1eYTHWJNr2Qf683mDB9Ox8TPzZ9HVZRn63F7tPG354r0++bNMidVVR3hRubYE7W+QXaQ1WsrS09kw5UzEi30yffcqNPDz97T4MbhuENiE+6NTQz+jic+WtDfv11iPRitJBgKYWpLP1BFMxeAClYzE++uMoJg9pZTB2xda1UKLm7sTSF7uiT/P6OFPhWk3nr+fh/PU8o/BhKngAMNuatyYxzeg9erdjXawpKNZi7uaTFkPC1VsFKC7R6loKNyRfxuOdQnE8PQery7oFJwxsaXLdEnML5pVPlwdK18Qx5attp7H37A2Dxe22v9vX7OswZ+k/57HnzDVk5BTg8U6heGdAC7NlazqGD6JqpvX0TTaX1Z+9YUnF4AHYtjru3bA26t5WHWdurlT5qLk7kG2iZcTUMSi35+x17LHwzVpAIK+w2GCsxckrt9By2kbsnxKF+hVWh+398d0t6/5d7DlcySnA/Kc6VOnxo76Ps7hCbMWLVZqzPjndeiEJmWqx0PfC4v1oF+qju52WeRubjqQj0Nv0yqrlCou1Jq8TczgtCzkWlnovV345icow1ZV6u6hEtw7R/K2na3X44GwXInKIrrO3WC/kAI7o3xYCmG1myf/1yZcRe8q2FXkrw9oCendjZw1enMqaQ3qLCB5Pz8GrPyYYzTir6IH/7ECfT7cbbbd1+XNbB1Hr6zVnq9Uywxf9Y3RxxMoQQtz1IpGOwvBRwdToVnJXgYiqGSGAbcdNn7D3nr2OZ/4r3XRyW1u7LJ10ymd5KIV+d1nzKcaLJlacGVZZpsYxWRpBYmur4P7zN/Huz4bjv8wNPjdl1OL9uOfDv6wXlAHDRwUv39sYse/1k7saRFSNvLP6oNkP/fL1Mhyh0aQ/DW4npWbafCVXcxdYVCL9wcSOukaSrRIu3ES2iS4ecyouEmhLi0m5nSevmlwTpjpg+DAh1M94HjsRkdzeWZVkc9m7vR5LbSLHtYeX/nPe7H3v/ZIsXUWqKYYPMwa0DpS7CkREBirOXiHbWJ2a7QBLLISPitfZUSKGDzNaBHnJXQUiIrKDo5VcIIwcj+HDjJd7N4aHq/H8fSIiIro7DB9m+Hi44K+37rNekIiIiCqF4cOCUD8PPNejoSQXnSIiIlIKrnBqxYcPRwKA1cuKExERkW3Y8kFERESSYviwAzMXwyQiIiITGD5s5F/H1ex9wzuFSlgTIiKimo3hw0ab3+6D5S93M3nfoMgg3X11LYQUIiIiYviwWd06rujZtB6e6R5usP3he0LQr0UAejath/NzorF/SpRMNSQiIqoZONulkmY+HIlX72sCjbMaCRdu4oHWgQYXLXJSq7BrYj8kpWbidmEJJv5ySMbaEhERVT9s+agklUqFsLoeCPB2w+C2wXB2Mj6EYXU98GD7EAT7ull8rqnRrRxVTSIiomqL4cOBXEwEE30v9orAy70joHG2/mfo3zLAXtUiIiKSFcOHA7XUuzjdjIfaGNznpFZBrVZh6tDWOPrhIDzaoQHubVYPy18xPahV42L9T/Xl0x2w+rUeNtVt8uCWNpWrLhr4uqNVsLfc1SAiIjvgmA8H8vVwxd7J/eHmooavhyue6BIGIYDlcSmIanWnJcNJrcLcJ+7R3f7l9Z54bOE/utuN69XB6PuaYH1yusn9vNgrAg+2D0aHcD+D7S0CvXDiSg4AYMUr3fHUt3t1973apwliNhzX3fZwdUJeYYnudlSrABy5lI3LWfmVes0fPNga3+46h7TM25V6nCm+Hi7IzCtCRL062PpOH6hUKszbcgpzN5+86+cmIiL5sOXDwYJ83ODrUTr91s3FCe6uTnipdwQa+tcx+5gAL43Bbb86rrgnzNdk2eUvd8P0B1sbBQ8A8HRzxvk50Tg/Jxo9mvhj5rBIeLg64ZfXjVtHtr7TF80DPXW3hQD+PcT8mJTmgZ4Iq+uO9mG++PeQO60oz/eKQJdGxnWpaPR9ja2W2TWxHz54sDVWju6uG9TrqtdF9dXIjkazj4iIyLpgH8tjEh2N4aMaCqvrYdAt0sDX3WS52Pf6oWfTemafRwhhcPvZ7g2R/MFAdGpY12D7U13DEeTjhr/e6qPbVs/TMAA90TkM04a21t0eH9Uc29/th3Vv9ISTuvJvo4qrwjby99D97uPugkXPdIKXmwue7xWBQO87/yTPdG+IyAbeePuB5hjSNhgfDWtr8vldbRhHQ7XHmH68+CNRZawc3V3W/fMTupp6tU8TtA/1AQA82SXMZJnyFhVzhIltTuo7Z/09k+/HR8Mi8f6Dd0LFd891xgOtA/He4JZ4oHUgGvp7YNg9Ifj48XZ4qXeErlyLIC84qVVQqVRGIWdQZJDu974t6pusm5fmTo+fWgW8O7CF7nbS9AcMnkOfp8YZf/zrXrzZv5luW4iJBH/o/QE49MEAnJ09xOTzAIbHdfu7fdEtoq7ZshVN0KuvEjWuZ77lrqIH24dYvP+VeyMwNboVzs4egrh/98fANoGVrk+T+p7WC1GVjI9qZr2QFe3NtNxW1CaE47qkYqn1XQoc81GNrX6tJ65k5yOsrof1wnrahfrg0MUsDO9kOrSUC/ZxxzPdGxpsi2odiKjWdz78t7/b12Adk0XPdMTVW4UGH/YVWxkGtgnCz6/1QNMAT/h6uOJqTgHWJ1/G+78dAQB0DPfFC70isO/cDQxsE4SnuobjZl4hAEDjrDbYny2aB3nhUoWxKW4uTnBzcQIAHPtwEFpN32j0uOd7NUK4vwc6hfuhUb06+OLJDugeswUuTip0i/BH7OlrZvcZ1SoQn246Ual61iZ/vXUfmk7ZYFPZ+U91wO8HL5m9f0r0nfAb4O0GranUbIWHq1OlyntpnJFTUGxT2ei2wfgz+XLlK0UAADcXNf4zoj2SUjOx5kCaxf+r6UNbI6+oBC8s3i9hDZVhSNsgs+MG5cCWj2rM1VltEDxWvNId9zYz381SbuXo7ljzRk881dVy+LBFxSAwKDIYz1YILMM7haF9mK+uNUKlUqFzo7q6lpn6XhqM6tnI4DF1NM748aVueKZ7QzipVajnqUH81CgkTn+g0nW0Nn7E3dUJXz7dweR9b/Rtim6N/QGUjs/ZM/l+JE4fgE+Ht8OgNkH4yUzTZMV8tH9KFCIbVP5bW7tQH4e2onzzbCejbeZmVJnTpZEftr/b12BbxfVtKr6Gsf2a2vTcXm7G33/C/AzD9qt97vx9n+hs+j0d1SrQ5m/XAHDw/QFWy/h6uODLpztgwciO2DP5fpufuzo7PWuw7veJg1qYbWXS324qDLYum3n211v3Wd3n8ZmD0bi+Jx7tGIovnrzHbLmt7/RBt8b+6NfizmD81/s2wbB7LLec6Vv8QhecmT0EP43ujui2wTY/rib6zxPtzd5Xsas+9r1+6Bbh7+gqVQrDRw3So4k/Zj9ieoyDPg9XZ3QM96t0C0JVubs64dcxvfD2A80tlvMuO9H0bWF6zZJ6nhp4uFa+Ma5nk3qInxqFl/W6hSp6oHUg7gnzNeg6qmNiX8E+7vDUOCPYxx2Lnu2kCyaW/N/w9qjvpcEf/7pXt61+hUHDQOk3j4p+Gt0DY/o1xZv9mxl0JT3asQF+H9vb6r5dnAz/xrsnGZ4kB7Qx3mfPJncC7OrXeuCTx9qZff4//tUbK17pjkb16hid+Gc9Eqn7vVXwnWnlT3UNx7sDW2D5y93w65heAO6EoIonhPcGGU/5Hv/AnePwzbOdMHlwKxyZMRB/vtkbg0wcw5nDIuHspMavY3ohup3h8++edL/J4KlWW//fSJz2AIa2Kz3xBfuYHnelr2mAZ6WucB1e18Pq2KSKf09z/nV/U4Op/abMebStQWh0Uavx5dMd0aWRH17uHYF1Y3phfFQzHPtwEL5+tjM6NfSDn4cLuja60x0ZN6U/zs+Jxp9v9sbZ2UPQPNALYXWtH5ty/p4afP98Zyx/uRserhAqGpvpOvuP3kxAa/q1CICTWoVujf2xYGRHk2X+fruPye2uZcdG/8uMpS975i4oaqobGAD+91I3/HdUZ7PPp29EZ/MXK326WziOzxyERzqEmvycef/B1tg1sZ/udvzUKIT6eRh0j1t7r0iB3S4kmb/e6oN/zlzTfaDbUz1PDcZFNUNOfrHJMQYaZyesKzsRNg/0xM28okp3Z+nTP8eYGp/y9bOd8N/Yc/jz0J3m+pd6R+iaPe8J88XIbuFwL+suKA9uTQM8ceDCTUwf2hrqsqX6hy3YjZnDIvHGsgO653JzUeOdB1qgWaAnni9roo5uG4wGvu54vFMofk64aLLei1/oAgD45PF2OHP1Fjo39EPbBj5YFpcC/zqu6NuiPvo2D8CbKxPxUu8IRDbw0T025tG2iD19DQPKvhU/3TUcBy5k4lZBEfo0D8BXIztidXyqrhVEfzD0gDZBODVrMFyc1Li4YDcOpmaiQ7ivyfFM3m4uOD8nGvlFJbquszoaZ7QJ8YHX9TyDso92bGDQEufr7qL7/ZEODdDA1x0NfN3Rv2Uglu27gI/+PGbyuJQL9nHTTS+vbHgvP6k9930cdp68arX8zon9UFisRfOpd7qv9k+JwtbjV7Dt+FU816MhGvi6Y2CbQGw6cgXR7YIN3k/63uzfDO8MaIFvdp7B9VuF+HrnWZvq7OKkxurXeupu68+q+/m1HijWCqhVKtzbrB7C63ogwKv0xKpSqXRBa83rvRB7+ire+umg0fObWnfo/pal75/WId74Ncl8dxxQOutO/+8QVtcdY/s1xbHLOZgS3Qr5RSX468gVvLP6oMHyBeZEtw1G0wDTIefkrMHILyqBxlmN4+k58HV3wbynOkCrFWj87/VG5ec81g6rTfyf3de8Psbe3xS9P96m23bsw0G6/3Vr9v27PwK8NFgVb/jc29/ti4b+HgbHY9awSIz+MQGDI4Ow4XDpZ4tz2RpSR2YMhFYIeLmV/k/oj/FY8Yq8g00Bho8aR6LGDIcI8nHDox3NJ/q75eXmgo8fN/8tvtwTXSo3Pfe3sb3w0Je78Wqfxvh6R+mHeoDeDJzyEyQAbHu3L1Ju5KFjuB86Pu2HBU8DjSb9CQBoWt8Lx2cOwumMW2gT4m3y5PZQ+xA8pBeewup6IGFaaVfUU13DsCIuFQBwdMYgqNUqFJVo0SbEGy2CvDB3xD0AgPahPgbho0n9OjhzNRdA6bdtABih14rh5uKka6Eot67CbaC0tUD/m7hKpcL/jbjT9DukbTCGWGjqLl/xd/WrPXAp8zYaWRm0qn9cy4X7e2Dl6O548pvSNWsqTkG/J8wXy/alADD8xuzu6oTnejRC9u0i9KkwCLp8/EeonzucLbSIfPJYO4NrNX3yeDs08q+DEV/vMSgX5me5JaBFoJfuW7l+y8fU6Fao76XBE13CDd6jXz9759vyvCcF1iam4d3Vhid6F9239ibIyMm3OXxYolKpdC1rP75kvquuvpcGj3QINRk+zM3UA0oHzN/XvD52nrwKPw8Xs+X0OalUBsfGxUmNxzqFokO4r+69bclnw0vfrxH16uDctdL/CWe1ChvGlbZalr/nfnixq+4x+q1kdeu44kZuIZ7t3hDm3ik9mvgjVK/r8O0HmhsEjz/+1Rvrky8jt6AYS/dcMHp8+ey++1sGYOvxDN12Lzdno8+MAW2CcHjGQHhqnHWfM+X1raMxPL33bVEf04e2RpsQb/hVg6uvM3zUMO56H8iWPijJftqF+uJczBBczy3UhQ83FzUOvj8AapXhDKKIenUQUeGkuu/f/VFQpIVP2QesfmtCZcx+pC2iWgWiVbC37gPGxUmNP9+816BcxZP2m/2bYdzKpCrt0xFcndVWg4cl3Rv7Y/Nb92HfuRtGLSePdQyFVgh0ami81oyrsxpvDzAeX9O7WT2Mi2qGMD8PRM/bZXa/I7qEYUSXMOTkF+F4eg46N/TD8fQco3ITB7VEfpEWfx1J1w1qHde/Gb7YcgoAsMnMOIkWNjSFO6lVeLxTKB7vFIqECzcNFiMsF+DlhrH9muLizTzMHXGP7lt7xawrxXR0Pysz8v4zoj2+333OIAzrEybn7Bkz12WjL2FqlC4E/PBiV3y36yye7dEQjet52tQNBwDP9WiIxzqGItTPHfqT/AK8NMjIKQAAgy8PAODvaXgMIhv4ILKBDz7ZeBwVPdqxge73uSPaY9afx7A64SLC63qYPZaeZSHjxV4R2HnqKh7p0MBkOZVKhRctdE1LjeGjhvH31GD60NZwcVab/GZIjqFSlQ6KnfFQG7g6q6FxdoLG2bbjr79Oyd3WoX8r69NQH7onBL8fuoweJsarhNgwdqEmaBbohWaBxidrtVplc8vWe4Na4vvd5zB5cCuE+9veBefl5oIuZeMgWgV7Y2y/pgjS6+f3cXfB/41oj3ErtbpuhXqe1r9p+rjb9u2/nJOFE+a7FgYxv/1Ac2w7kWH2hF9VT3UNx4q4FAyODMK/h7SCVgirXQ3+nhpMGGg87qdro7qIO38Dj5e1lD7XoyF+2HMBE02MEbKkVbA3jl3OBgBd9wNQ2qI44+FIcw+z6E537Z308eNL3bAiLgUju4Xb3F1nKlZ9+vidlkRfD1d8Orw9Pn6sHQSsj1OarrdkQk3A8FEDVaf0qjQVZ+1URxpnJ8NmY70PQ1v7nZXg9b5N8FqfxgYni6hWgfgu9lylVn80d6KfNLgljl/OwbM9GhqthaPvs+HtcfFmHtqF+tq8TwBo28AH3SLqooGVbp6nuobjnzPXdGOhKg5utpcPHmqNQZFB6Nqo7l2/z1aM7o6c/CLdjLkZD7XBuP7N4O9pPMDSkvlP3YOouTvvqi762oXeabVUqVT4dUwv5BeVoEWQFz6ocP2ucu42fkkMq+tuMlDa2ipT0zB8ENVyD7QOROtgb3S2Ydl7pan4LfXdgS3QIsgLfZqbXhyvMoJ93HVdLD/uOW+23ONmZk1Y46RW4adXrV9IMubRthBCOHz2m8bZyS7HDSh9bfqLKKpUqkoHDwBw1lt9+W5e/t9v98Hx9GyDacCA5cXTJg9uib1nr5sdYB+oN1Nl7Rs90cTMQNjaiuGDqJZzc3HC+nH3Wi9IcHNxwnA7d0cAppvYpSTVtPvqpqG/Bwa2CYSXm4tuUG5VNA3wNDtLxpxX+zTBq33ML/v/dLeGOHElB32a1zd5ba7ajuGDiMjBKnviIvtQqVQGs4WqE1dnNWIetT47r7Zi+CAicrCeTeph7oj2aBYg/+JORNUBwwcRkQQcucYNUU3D5dWJiIhIUgwfREREJCmGDyIiIpIUwwcRERFJiuGDiIiIJMXwQURERJJi+CAiIiJJMXwQERGRpBg+iIiISFIMH0RERCQphg8iIiKSFMMHERERSYrhg4iIiCRV7a5qK4QAAGRnZ8tcEyIiIrJV+Xm7/DxuSbULHzk5OQCAsLAwmWtCRERElZWTkwMfHx+LZVTClogiIa1Wi0uXLsHLywsqlcquz52dnY2wsDCkpqbC29vbrs+tZDyu9sdj6hg8ro7B4+oYNe24CiGQk5ODkJAQqNWWR3VUu5YPtVqN0NBQh+7D29u7RvwhaxoeV/vjMXUMHlfH4HF1jJp0XK21eJTjgFMiIiKSFMMHERERSUpR4UOj0eD999+HRqORuyq1Co+r/fGYOgaPq2PwuDpGbT6u1W7AKREREdVuimr5ICIiIvkxfBAREZGkGD6IiIhIUgwfREREJCnFhI8FCxagUaNGcHNzQ7du3RAXFyd3laqNmJgYdOnSBV5eXggICMCwYcNw4sQJgzL5+fkYM2YM/P394enpicceewxXrlwxKJOSkoLo6Gh4eHggICAAEyZMQHFxsUGZ7du3o2PHjtBoNGjatCmWLFni6JdXbcyZMwcqlQrjx4/XbeNxrZq0tDQ888wz8Pf3h7u7O9q2bYv4+Hjd/UIITJ8+HcHBwXB3d0dUVBROnTpl8Bw3btzAyJEj4e3tDV9fX7z00ku4deuWQZlDhw7h3nvvhZubG8LCwvDJJ59I8vqkVlJSgmnTpiEiIgLu7u5o0qQJZs6caXCNDh5T63bu3IkHH3wQISEhUKlUWLduncH9Uh7D1atXo2XLlnBzc0Pbtm2xfv16u7/euyIUYOXKlcLV1VV8//334siRI+KVV14Rvr6+4sqVK3JXrVoYOHCgWLx4sTh8+LBISkoSQ4YMEeHh4eLWrVu6Mq+99poICwsTW7ZsEfHx8aJ79+6iZ8+euvuLi4tFZGSkiIqKEomJiWL9+vWiXr16YvLkyboyZ8+eFR4eHuLtt98WR48eFfPnzxdOTk5i48aNkr5eOcTFxYlGjRqJdu3aiXHjxum287hW3o0bN0TDhg3F888/L/bt2yfOnj0rNm3aJE6fPq0rM2fOHOHj4yPWrVsnDh48KB566CEREREhbt++rSszaNAg0b59e7F3716xa9cu0bRpU/HUU0/p7s/KyhKBgYFi5MiR4vDhw2LFihXC3d1dfP3115K+XinMmjVL+Pv7iz/++EOcO3dOrF69Wnh6eoovvvhCV4bH1Lr169eLKVOmiDVr1ggAYu3atQb3S3UMd+/eLZycnMQnn3wijh49KqZOnSpcXFxEcnKyw4+BrRQRPrp27SrGjBmju11SUiJCQkJETEyMjLWqvjIyMgQAsWPHDiGEEJmZmcLFxUWsXr1aV+bYsWMCgNizZ48QovSfTq1Wi/T0dF2ZhQsXCm9vb1FQUCCEEGLixImiTZs2Bvt64oknxMCBAx39kmSVk5MjmjVrJjZv3iz69OmjCx88rlXz3nvvid69e5u9X6vViqCgIPHpp5/qtmVmZgqNRiNWrFghhBDi6NGjAoDYv3+/rsyGDRuESqUSaWlpQgghvvrqK+Hn56c7zuX7btGihb1fkuyio6PFiy++aLDt0UcfFSNHjhRC8JhWRcXwIeUxHDFihIiOjjaoT7du3cSrr75q19d4N2p9t0thYSESEhIQFRWl26ZWqxEVFYU9e/bIWLPqKysrCwBQt25dAEBCQgKKiooMjmHLli0RHh6uO4Z79uxB27ZtERgYqCszcOBAZGdn48iRI7oy+s9RXqa2/x3GjBmD6Ohoo9fO41o1v/32Gzp37ozhw4cjICAAHTp0wLfffqu7/9y5c0hPTzc4Jj4+PujWrZvBcfX19UXnzp11ZaKioqBWq7Fv3z5dmfvuuw+urq66MgMHDsSJEydw8+ZNR79MSfXs2RNbtmzByZMnAQAHDx5EbGwsBg8eDIDH1B6kPIY14TOh1oePa9euoaSkxODDGwACAwORnp4uU62qL61Wi/Hjx6NXr16IjIwEAKSnp8PV1RW+vr4GZfWPYXp6usljXH6fpTLZ2dm4ffu2I16O7FauXIkDBw4gJibG6D4e16o5e/YsFi5ciGbNmmHTpk14/fXX8eabb2Lp0qUA7hwXS//z6enpCAgIMLjf2dkZdevWrdSxry0mTZqEJ598Ei1btoSLiws6dOiA8ePHY+TIkQB4TO1BymNorkx1OsbV7qq2JK8xY8bg8OHDiI2NlbsqNV5qairGjRuHzZs3w83NTe7q1BparRadO3fG7NmzAQAdOnTA4cOHsWjRIowaNUrm2tVMq1atwrJly7B8+XK0adMGSUlJGD9+PEJCQnhMySFqfctHvXr14OTkZDSD4MqVKwgKCpKpVtXT2LFj8ccff2Dbtm0IDQ3VbQ8KCkJhYSEyMzMNyusfw6CgIJPHuPw+S2W8vb3h7u5u75cju4SEBGRkZKBjx45wdnaGs7MzduzYgXnz5sHZ2RmBgYE8rlUQHByM1q1bG2xr1aoVUlJSANw5Lpb+54OCgpCRkWFwf3FxMW7cuFGpY19bTJgwQdf60bZtWzz77LN46623dC12PKZ3T8pjaK5MdTrGtT58uLq6olOnTtiyZYtum1arxZYtW9CjRw8Za1Z9CCEwduxYrF27Flu3bkVERITB/Z06dYKLi4vBMTxx4gRSUlJ0x7BHjx5ITk42+MfZvHkzvL29dSeKHj16GDxHeZna+nfo378/kpOTkZSUpPvp3LkzRo4cqfudx7XyevXqZTQV/OTJk2jYsCEAICIiAkFBQQbHJDs7G/v27TM4rpmZmUhISNCV2bp1K7RaLbp166Yrs3PnThQVFenKbN68GS1atICfn5/DXp8c8vLyoFYbng6cnJyg1WoB8Jjag5THsEZ8Jsg94lUKK1euFBqNRixZskQcPXpUjB49Wvj6+hrMIFCy119/Xfj4+Ijt27eLy5cv637y8vJ0ZV577TURHh4utm7dKuLj40WPHj1Ejx49dPeXTwkdMGCASEpKEhs3bhT169c3OSV0woQJ4tixY2LBggW1ekqoKfqzXYTgca2KuLg44ezsLGbNmiVOnTolli1bJjw8PMT//vc/XZk5c+YIX19f8euvv4pDhw6Jhx9+2OSUxg4dOoh9+/aJ2NhY0axZM4MpjZmZmSIwMFA8++yz4vDhw2LlypXCw8Oj1kwL1Tdq1CjRoEED3VTbNWvWiHr16omJEyfqyvCYWpeTkyMSExNFYmKiACDmzp0rEhMTxYULF4QQ0h3D3bt3C2dnZ/HZZ5+JY8eOiffff59TbeUyf/58ER4eLlxdXUXXrl3F3r175a5StQHA5M/ixYt1ZW7fvi3eeOMN4efnJzw8PMQjjzwiLl++bPA858+fF4MHDxbu7u6iXr164p133hFFRUUGZbZt2ybuuece4erqKho3bmywDyWoGD54XKvm999/F5GRkUKj0YiWLVuKb775xuB+rVYrpk2bJgIDA4VGoxH9+/cXJ06cMChz/fp18dRTTwlPT0/h7e0tXnjhBZGTk2NQ5uDBg6J3795Co9GIBg0aiDlz5jj8tckhOztbjBs3ToSHhws3NzfRuHFjMWXKFIPpnDym1m3bts3kZ+moUaOEENIew1WrVonmzZsLV1dX0aZNG/Hnn3867HVXhUoIvSXsiIiIiBys1o/5ICIiouqF4YOIiIgkxfBBREREkmL4ICIiIkkxfBAREZGkGD6IiIhIUgwfREREJCmGDyIiIpIUwwcRERFJiuGDiIiIJMXwQURERJJi+CAiIiJJ/T/xWCAIG14w+gAAAABJRU5ErkJggg==",
1032 | "text/plain": [
1033 | ""
1034 | ]
1035 | },
1036 | "metadata": {},
1037 | "output_type": "display_data"
1038 | }
1039 | ],
1040 | "source": [
1041 | "import matplotlib.pyplot as plt\n",
1042 | "import pandas as pd\n",
1043 | "\n",
1044 | "data = pd.read_csv(\"loss.txt\")\n",
1045 | "plt.plot(data)\n",
1046 | "plt.show()"
1047 | ]
1048 | },
1049 | {
1050 | "cell_type": "markdown",
1051 | "id": "9809bc9f-4ff6-46c3-9c43-08c6c2694a82",
1052 | "metadata": {},
1053 | "source": [
1054 | "## Generate text with fine-tuned model\n",
1055 | "\n",
1056 | "Again we check results with our test dataset (5 rows).
\n",
1057 | "As you can see below, it can output the completion very well, because it's fine-tuned."
1058 | ]
1059 | },
1060 | {
1061 | "cell_type": "code",
1062 | "execution_count": 26,
1063 | "id": "29903cae-404e-4209-9c84-6c8a69609c13",
1064 | "metadata": {},
1065 | "outputs": [
1066 | {
1067 | "name": "stdout",
1068 | "output_type": "stream",
1069 | "text": [
1070 | "********** input **********\n",
1071 | "name : The Punter | Type : pub | food : Chinese | price : more than £ 30 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n",
1072 | "\n",
1073 | "********** result **********\n",
1074 | "name : The Punter | Type : pub | food : Chinese | price : more than £ 30 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n",
1075 | "The Punter is a children friendly pub that serves Chinese food. It is located in the riverside area near Raja Indian Cuisine and has a\n",
1076 | "********** input **********\n",
1077 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n",
1078 | "\n",
1079 | "********** result **********\n",
1080 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : no | near : All Bar One\n",
1081 | "The Cricketers is a Chinese restaurant with a cheap price range, located in the city centre near All Bar One. It has a customer rating of\n",
1082 | "********** input **********\n",
1083 | "name : The Phoenix | Type : pub | food : French | price : moderate | customer rating : 1 out of 5 | area : riverside | family friendly : no | near : Crowne Plaza Hotel\n",
1084 | "\n",
1085 | "********** result **********\n",
1086 | "name : The Phoenix | Type : pub | food : French | price : moderate | customer rating : 1 out of 5 | area : riverside | family friendly : no | near : Crowne Plaza Hotel\n",
1087 | "The Phoenix is a pub that serves French food. It is located near Crown Plaza Hotel in the riverside area. It has a moderate price range and\n",
1088 | "********** input **********\n",
1089 | "name : Giraffe | Type : restaurant | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n",
1090 | "\n",
1091 | "********** result **********\n",
1092 | "name : Giraffe | Type : restaurant | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n",
1093 | "Giraffe is a fast food restaurant located in the riverside area near Rainbow Vegetarian Café. It is family friendly.\n",
1094 | "********** input **********\n",
1095 | "name : The Vaults | Type : pub | food : French | price : more than £ 30 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
1096 | "\n",
1097 | "********** result **********\n",
1098 | "name : The Vaults | Type : pub | food : French | price : more than £ 30 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
1099 | "The Vaults is a children friendly French pub located in the city centre near Raja Indian Cuisine.\n"
1100 | ]
1101 | }
1102 | ],
1103 | "source": [
1104 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n",
1105 | "test_data = test_data[::2] # because it's duplicated\n",
1106 | "test_loader = DataLoader(\n",
1107 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n",
1108 | " batch_size=1,\n",
1109 | " shuffle=True,\n",
1110 | " collate_fn=collate_batch\n",
1111 | ")\n",
1112 | "\n",
1113 | "for i, (input, _, mask) in enumerate(test_loader):\n",
1114 | " if i == 5:\n",
1115 | " break\n",
1116 | " print(\"********** input **********\")\n",
1117 | " input_len = torch.sum(mask).cpu().numpy()\n",
1118 | " print(tokenizer.decode(input[0][:input_len]))\n",
1119 | " result_token, result_len = generate_text(\n",
1120 | " model,\n",
1121 | " input,\n",
1122 | " mask,\n",
1123 | " eos_id,\n",
1124 | " pred_sequence_length=30)\n",
1125 | " print(\"********** result **********\")\n",
1126 | " print(tokenizer.decode(result_token[0][:result_len]))"
1127 | ]
1128 | },
1129 | {
1130 | "cell_type": "code",
1131 | "execution_count": null,
1132 | "id": "6a7c1dd3-4057-497a-83ae-f99b1883697e",
1133 | "metadata": {},
1134 | "outputs": [],
1135 | "source": []
1136 | }
1137 | ],
1138 | "metadata": {
1139 | "kernelspec": {
1140 | "display_name": "Python 3 (ipykernel)",
1141 | "language": "python",
1142 | "name": "python3"
1143 | },
1144 | "language_info": {
1145 | "codemirror_mode": {
1146 | "name": "ipython",
1147 | "version": 3
1148 | },
1149 | "file_extension": ".py",
1150 | "mimetype": "text/x-python",
1151 | "name": "python",
1152 | "nbconvert_exporter": "python",
1153 | "pygments_lexer": "ipython3",
1154 | "version": "3.8.10"
1155 | }
1156 | },
1157 | "nbformat": 4,
1158 | "nbformat_minor": 5
1159 | }
1160 |
--------------------------------------------------------------------------------
/02-finetune-gpt2-with-lora.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "857cafc6-da38-4aa7-8afc-63aa626fa7aa",
6 | "metadata": {},
7 | "source": [
8 | "# 02. Finetuning GPT-2 with LoRA\n",
9 | "\n",
10 | "In this example, we fine-tune the pre-trained auto-regressive model, **OpenAI's GPT-2** (small version, 124M parameters), by applying LoRA (Low-Rank Adaptation) optimization.\n",
11 | "\n",
12 | "In this example, I download the pre-trained model from Hugging Face hub, but fine-tune model with regular PyTorch training loop.
\n",
13 | "(Here I don't use Hugging Face Trainer class.)\n",
14 | "\n",
15 | "See [Readme](https://github.com/tsmatz/finetune_llm_with_lora) for prerequisite's setup."
16 | ]
17 | },
18 | {
19 | "cell_type": "code",
20 | "execution_count": 1,
21 | "id": "3d49acf1-9ad1-4a6c-9312-6785cb3f5862",
22 | "metadata": {},
23 | "outputs": [],
24 | "source": [
25 | "model_name = \"gpt2\""
26 | ]
27 | },
28 | {
29 | "cell_type": "code",
30 | "execution_count": 2,
31 | "id": "4d835e84-a01d-4c33-926b-60d9dd4a7627",
32 | "metadata": {},
33 | "outputs": [],
34 | "source": [
35 | "import torch\n",
36 | "\n",
37 | "device = torch.device(\"cuda\")"
38 | ]
39 | },
40 | {
41 | "cell_type": "markdown",
42 | "id": "ead383e5-149b-4bfb-9324-3cc639fd398d",
43 | "metadata": {},
44 | "source": [
45 | "## Prepare dataset and dataloader"
46 | ]
47 | },
48 | {
49 | "cell_type": "markdown",
50 | "id": "91ecbb08-6a74-4623-bfe8-bddba5254e35",
51 | "metadata": {},
52 | "source": [
53 | "In this example, we use dataset used in [official LoRA example](https://github.com/microsoft/LoRA).\n",
54 | "\n",
55 | "Download dataset from official repository."
56 | ]
57 | },
58 | {
59 | "cell_type": "code",
60 | "execution_count": 3,
61 | "id": "54a564f1-f8f3-42a6-b160-bebdbcc3aac0",
62 | "metadata": {},
63 | "outputs": [
64 | {
65 | "name": "stdout",
66 | "output_type": "stream",
67 | "text": [
68 | "--2023-10-06 03:27:50-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt\n",
69 | "Resolving github.com (github.com)... 140.82.114.3\n",
70 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
71 | "HTTP request sent, awaiting response... 302 Found\n",
72 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt [following]\n",
73 | "--2023-10-06 03:27:51-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/train.txt\n",
74 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n",
75 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
76 | "HTTP request sent, awaiting response... 200 OK\n",
77 | "Length: 9624463 (9.2M) [text/plain]\n",
78 | "Saving to: ‘train.txt’\n",
79 | "\n",
80 | "train.txt 100%[===================>] 9.18M --.-KB/s in 0.04s \n",
81 | "\n",
82 | "2023-10-06 03:27:51 (248 MB/s) - ‘train.txt’ saved [9624463/9624463]\n",
83 | "\n"
84 | ]
85 | }
86 | ],
87 | "source": [
88 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/train.txt"
89 | ]
90 | },
91 | {
92 | "cell_type": "code",
93 | "execution_count": 4,
94 | "id": "d48464ea-991f-48b2-9166-3323cfd61676",
95 | "metadata": {
96 | "scrolled": true
97 | },
98 | "outputs": [
99 | {
100 | "name": "stdout",
101 | "output_type": "stream",
102 | "text": [
103 | "--2023-10-06 03:27:54-- https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt\n",
104 | "Resolving github.com (github.com)... 140.82.114.3\n",
105 | "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
106 | "HTTP request sent, awaiting response... 302 Found\n",
107 | "Location: https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt [following]\n",
108 | "--2023-10-06 03:27:54-- https://raw.githubusercontent.com/microsoft/LoRA/main/examples/NLG/data/e2e/test.txt\n",
109 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n",
110 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
111 | "HTTP request sent, awaiting response... 200 OK\n",
112 | "Length: 1351149 (1.3M) [text/plain]\n",
113 | "Saving to: ‘test.txt’\n",
114 | "\n",
115 | "test.txt 100%[===================>] 1.29M --.-KB/s in 0.006s \n",
116 | "\n",
117 | "2023-10-06 03:27:54 (208 MB/s) - ‘test.txt’ saved [1351149/1351149]\n",
118 | "\n"
119 | ]
120 | }
121 | ],
122 | "source": [
123 | "!wget https://github.com/microsoft/LoRA/raw/main/examples/NLG/data/e2e/test.txt"
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "id": "09472803-8c62-48e0-9a63-b9b9448f16d3",
129 | "metadata": {},
130 | "source": [
131 | "Show the downloaded data (first 5 rows)."
132 | ]
133 | },
134 | {
135 | "cell_type": "code",
136 | "execution_count": 5,
137 | "id": "e6e60596-028f-4c4b-a95d-f74a0ff3b188",
138 | "metadata": {},
139 | "outputs": [
140 | {
141 | "name": "stdout",
142 | "output_type": "stream",
143 | "text": [
144 | "name : The Vaults | Type : pub | price : more than £ 30 | customer rating : 5 out of 5 | near : Café Adriatic||The Vaults pub near Café Adriatic has a 5 star rating . Prices start at £ 30 . \n",
145 | "name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Café Brazil||Close to Café Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of £ 10.50 . Delicious Pub food . \n",
146 | "name : The Eagle | Type : coffee shop | food : Japanese | price : less than £ 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King||The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than £ 20 for Japanese food . \n",
147 | "name : The Mill | Type : coffee shop | food : French | price : £ 20 - 25 | area : riverside | near : The Sorrento||Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at £ 20- £ 25 it is in the riverside area . \n",
148 | "name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat||For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat . \n"
149 | ]
150 | }
151 | ],
152 | "source": [
153 | "!head -n 5 train.txt"
154 | ]
155 | },
156 | {
157 | "cell_type": "markdown",
158 | "id": "93f5fabe-590c-459b-aa16-4b5a506fb54b",
159 | "metadata": {},
160 | "source": [
161 | "Convert above data into JsonL format."
162 | ]
163 | },
164 | {
165 | "cell_type": "code",
166 | "execution_count": 6,
167 | "id": "7376e0c0-16c9-46f4-ad4c-83d1a677f5a2",
168 | "metadata": {},
169 | "outputs": [],
170 | "source": [
171 | "import sys\n",
172 | "import io\n",
173 | "import json\n",
174 | "\n",
175 | "def format_convert(read_file, write_file):\n",
176 | " with open(read_file, \"r\", encoding=\"utf8\") as reader, \\\n",
177 | " \t open(write_file, \"w\", encoding=\"utf8\") as writer :\n",
178 | " \tfor line in reader:\n",
179 | " \t\titems = line.strip().split(\"||\")\n",
180 | " \t\tcontext = items[0]\n",
181 | " \t\tcompletion = items[1].strip(\"\\n\")\n",
182 | " \t\tx = {}\n",
183 | " \t\tx[\"context\"] = context\n",
184 | " \t\tx[\"completion\"] = completion\n",
185 | " \t\twriter.write(json.dumps(x)+\"\\n\")\n",
186 | "\n",
187 | "format_convert(\"train.txt\", \"train_formatted.jsonl\")\n",
188 | "format_convert(\"test.txt\", \"test_formatted.jsonl\")"
189 | ]
190 | },
191 | {
192 | "cell_type": "markdown",
193 | "id": "3ceec952-fe03-475f-9f3e-22237cc9c44b",
194 | "metadata": {},
195 | "source": [
196 | "Show the converted data (first 5 rows)."
197 | ]
198 | },
199 | {
200 | "cell_type": "code",
201 | "execution_count": 7,
202 | "id": "cb32aca7-bd0e-4847-a4c2-cc7e67dc2b7a",
203 | "metadata": {},
204 | "outputs": [
205 | {
206 | "name": "stdout",
207 | "output_type": "stream",
208 | "text": [
209 | "{\"context\": \"name : The Vaults | Type : pub | price : more than \\u00a3 30 | customer rating : 5 out of 5 | near : Caf\\u00e9 Adriatic\", \"completion\": \"The Vaults pub near Caf\\u00e9 Adriatic has a 5 star rating . Prices start at \\u00a3 30 .\"}\n",
210 | "\n",
211 | "{\"context\": \"name : The Cambridge Blue | Type : pub | food : English | price : cheap | near : Caf\\u00e9 Brazil\", \"completion\": \"Close to Caf\\u00e9 Brazil , The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of \\u00a3 10.50 . Delicious Pub food .\"}\n",
212 | "\n",
213 | "{\"context\": \"name : The Eagle | Type : coffee shop | food : Japanese | price : less than \\u00a3 20 | customer rating : low | area : riverside | family friendly : yes | near : Burger King\", \"completion\": \"The Eagle is a low rated coffee shop near Burger King and the riverside that is family friendly and is less than \\u00a3 20 for Japanese food .\"}\n",
214 | "\n",
215 | "{\"context\": \"name : The Mill | Type : coffee shop | food : French | price : \\u00a3 20 - 25 | area : riverside | near : The Sorrento\", \"completion\": \"Located near The Sorrento is a French Theme eatery and coffee shop called The Mill , with a price range at \\u00a3 20- \\u00a3 25 it is in the riverside area .\"}\n",
216 | "\n",
217 | "{\"context\": \"name : Loch Fyne | food : French | customer rating : high | area : riverside | near : The Rice Boat\", \"completion\": \"For luxurious French food , the Loch Fyne is located by the river next to The Rice Boat .\"}\n",
218 | "\n"
219 | ]
220 | }
221 | ],
222 | "source": [
223 | "with open(\"train_formatted.jsonl\", \"r\") as reader:\n",
224 | " for _ in range(5):\n",
225 | " print(next(reader))"
226 | ]
227 | },
228 | {
229 | "cell_type": "markdown",
230 | "id": "6631f786-be4b-40cf-89d9-7009c1888821",
231 | "metadata": {},
232 | "source": [
233 | "Load tokenizer from Hugging Face."
234 | ]
235 | },
236 | {
237 | "cell_type": "code",
238 | "execution_count": 8,
239 | "id": "e5433dc0-b5a5-4c01-adb5-3ffa2279eca8",
240 | "metadata": {},
241 | "outputs": [],
242 | "source": [
243 | "from transformers import AutoTokenizer\n",
244 | "import os\n",
245 | "\n",
246 | "tokenizer = AutoTokenizer.from_pretrained(\n",
247 | " model_name,\n",
248 | " fast_tokenizer=True)\n",
249 | "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
250 | ]
251 | },
252 | {
253 | "cell_type": "markdown",
254 | "id": "50817c47-a97b-4f80-975b-836859a0a7cf",
255 | "metadata": {},
256 | "source": [
257 | "Set block size which is used to separate long text for model consumption.
\n",
258 | "Max 1024 tokens can be used in GPT-2, but here I set 512, because it's enough for our dataset."
259 | ]
260 | },
261 | {
262 | "cell_type": "code",
263 | "execution_count": 9,
264 | "id": "5f250929-5703-4b17-9f7b-26340950c055",
265 | "metadata": {},
266 | "outputs": [
267 | {
268 | "name": "stdout",
269 | "output_type": "stream",
270 | "text": [
271 | "Max length of tokens is 1024 in this model.\n",
272 | "But here we use max 512 tokens in the training.\n"
273 | ]
274 | }
275 | ],
276 | "source": [
277 | "block_size = 512\n",
278 | "\n",
279 | "print(f\"Max length of tokens is {tokenizer.model_max_length} in this model.\")\n",
280 | "print(f\"But here we use max {block_size} tokens in the training.\")"
281 | ]
282 | },
283 | {
284 | "cell_type": "markdown",
285 | "id": "2332617b-1e66-4812-ad47-5eaeb52b101b",
286 | "metadata": {},
287 | "source": [
288 | "Create function to convert data. (Later this function is then used in data loader.)
\n",
289 | "In this function,\n",
290 | "\n",
291 | "1. Tokenize both contexts and compeletions. : e.g, ```\"This is a pen.\"``` --> ```[1212, 318, 257, 3112, 13]```\n",
292 | "2. Concatenate context's token and completion's token. (But it's delimited by \"\\n\" between context and completion.) This is used for inputs for LLM.\n",
293 | "3. Create labels (targets) with inputs. Label is ```input[1:]``` (i.e, shifted right by one element), and is filled by ```-100``` in context's positions. (See below note.)\n",
294 | "4. Pad tokens to make the length of token become ```block_size```.\n",
295 | "\n",
296 | "> Note : Here I set ```-100``` as an ignored index for loss computation, because PyTorch cross-entropy function (```torch.nn.functional.cross_entropy()```) has a property ```ignore_index``` which default value is ```-100```."
297 | ]
298 | },
299 | {
300 | "cell_type": "code",
301 | "execution_count": 10,
302 | "id": "9f2f38aa-b3d0-4614-aa59-8ddd977176d1",
303 | "metadata": {},
304 | "outputs": [],
305 | "source": [
306 | "from torch.utils.data import DataLoader\n",
307 | "import pandas as pd\n",
308 | "\n",
309 | "def fill_ignore_label(l, c):\n",
310 | " l[:len(c) - 1] = [-100] * (len(c) - 1)\n",
311 | " return l\n",
312 | "\n",
313 | "def pad_tokens(tokens, max_seq_length, padding_token):\n",
314 | " res_tokens = tokens[:max_seq_length]\n",
315 | " token_len = len(res_tokens)\n",
316 | " res_tokens = res_tokens + \\\n",
317 | " [padding_token for _ in range(max_seq_length - token_len)]\n",
318 | " return res_tokens\n",
319 | "\n",
320 | "def collate_batch(batch):\n",
321 | " # tokenize both context and completion respectively\n",
322 | " # (context and completion is delimited by \"\\n\")\n",
323 | " context_list = list(zip(*batch))[0]\n",
324 | " context_list = [c + \"\\n\" for c in context_list]\n",
325 | " completion_list = list(zip(*batch))[1]\n",
326 | " context_result = tokenizer(context_list)\n",
327 | " context_tokens = context_result[\"input_ids\"]\n",
328 | " context_masks = context_result[\"attention_mask\"]\n",
329 | " completion_result = tokenizer(completion_list)\n",
330 | " completion_tokens = completion_result[\"input_ids\"]\n",
331 | " completion_masks = completion_result[\"attention_mask\"]\n",
332 | " # concatenate token\n",
333 | " inputs = [i + j for i, j in zip(context_tokens, completion_tokens)]\n",
334 | " masks = [i + j for i, j in zip(context_masks, completion_masks)]\n",
335 | " # create label\n",
336 | " eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n",
337 | " labels = [t[1:] + [eos_id] for t in inputs]\n",
338 | " labels = list(map(fill_ignore_label, labels, context_tokens))\n",
339 | " # truncate and pad tokens\n",
340 | " inputs = [pad_tokens(t, block_size, 0) for t in inputs] # OPT and GPT-2 doesn't use pad token (instead attn mask is used)\n",
341 | " masks = [pad_tokens(t, block_size, 0) for t in masks]\n",
342 | " labels = [pad_tokens(t, block_size, -100) for t in labels]\n",
343 | " # convert to tensor\n",
344 | " inputs = torch.tensor(inputs, dtype=torch.int64).to(device)\n",
345 | " masks = torch.tensor(masks, dtype=torch.int64).to(device)\n",
346 | " labels = torch.tensor(labels, dtype=torch.int64).to(device)\n",
347 | " return inputs, labels, masks"
348 | ]
349 | },
350 | {
351 | "cell_type": "markdown",
352 | "id": "2084d2e9-ef64-47a2-aec9-d24ead1cb38a",
353 | "metadata": {},
354 | "source": [
355 | "Now create PyTorch dataloader with previous function (collator function).\n",
356 | "\n",
357 | "> Note : In this example, data is small and we then load all JSON data in memory.
\n",
358 | "> When it's large, load data progressively by implementing custom PyTorch dataset. (See [here](https://github.com/tsmatz/decision-transformer) for example.)"
359 | ]
360 | },
361 | {
362 | "cell_type": "code",
363 | "execution_count": 11,
364 | "id": "f3bce3bb-2215-4bd6-a6a6-5b6b9d5afdc0",
365 | "metadata": {},
366 | "outputs": [],
367 | "source": [
368 | "batch_size = 8\n",
369 | "gradient_accumulation_steps = 16\n",
370 | "\n",
371 | "data = pd.read_json(\"train_formatted.jsonl\", lines=True)\n",
372 | "dataloader = DataLoader(\n",
373 | " list(zip(data[\"context\"], data[\"completion\"])),\n",
374 | " batch_size=batch_size,\n",
375 | " shuffle=True,\n",
376 | " collate_fn=collate_batch\n",
377 | ")"
378 | ]
379 | },
380 | {
381 | "cell_type": "markdown",
382 | "id": "3ba64144-b698-457e-b827-941020456536",
383 | "metadata": {},
384 | "source": [
385 | "## Load model"
386 | ]
387 | },
388 | {
389 | "cell_type": "markdown",
390 | "id": "1bfd360d-7bdc-4fd7-9b12-bcf9fe0a8db2",
391 | "metadata": {},
392 | "source": [
393 | "Load model from Hugging Face."
394 | ]
395 | },
396 | {
397 | "cell_type": "code",
398 | "execution_count": 12,
399 | "id": "271181bd-677a-4da9-9e57-2874f5e47bd0",
400 | "metadata": {},
401 | "outputs": [],
402 | "source": [
403 | "from transformers import AutoModelForCausalLM, AutoConfig\n",
404 | "\n",
405 | "config = AutoConfig.from_pretrained(model_name)\n",
406 | "model = AutoModelForCausalLM.from_pretrained(\n",
407 | " model_name,\n",
408 | " config=config,\n",
409 | ").to(device)"
410 | ]
411 | },
412 | {
413 | "cell_type": "markdown",
414 | "id": "27ab764a-d634-40f8-9edb-a01146845233",
415 | "metadata": {},
416 | "source": [
417 | "## Generate text (before fine-tuning)"
418 | ]
419 | },
420 | {
421 | "cell_type": "markdown",
422 | "id": "559efeaf-4b38-4a0c-9be6-eb394221e374",
423 | "metadata": {},
424 | "source": [
425 | "Now run prediction with downloaded model (which is not still fine-tuned).\n",
426 | "\n",
427 | "First we create a function to generate text."
428 | ]
429 | },
430 | {
431 | "cell_type": "code",
432 | "execution_count": 13,
433 | "id": "51a0c4fc-e0a7-4bbf-b25a-c335fe61f3df",
434 | "metadata": {},
435 | "outputs": [],
436 | "source": [
437 | "def generate_text(model, input, mask, eos_id, pred_sequence_length):\n",
438 | " predicted_last_id = -1\n",
439 | " start_token_len = torch.sum(mask).cpu().numpy()\n",
440 | " token_len = start_token_len\n",
441 | " with torch.no_grad():\n",
442 | " while (predicted_last_id != eos_id) and \\\n",
443 | " (token_len - start_token_len < pred_sequence_length):\n",
444 | " output = model(\n",
445 | " input_ids=input,\n",
446 | " attention_mask=mask,\n",
447 | " )\n",
448 | " predicted_ids = torch.argmax(output.logits, axis=-1).cpu().numpy()\n",
449 | " predicted_last_id = predicted_ids[0][token_len - 1]\n",
450 | " input[0][token_len] = predicted_last_id\n",
451 | " mask[0][token_len] = 1\n",
452 | " token_len = torch.sum(mask).cpu().numpy()\n",
453 | " return input, token_len"
454 | ]
455 | },
456 | {
457 | "cell_type": "markdown",
458 | "id": "3936b1a1-ae9f-48a5-80db-691261dda704",
459 | "metadata": {},
460 | "source": [
461 | "Let's test our function and generate text. (Here we stop the text generation when it reaches 15 tokens in prediction.)"
462 | ]
463 | },
464 | {
465 | "cell_type": "code",
466 | "execution_count": 14,
467 | "id": "28b7e13f-e8fb-4a9f-90ed-0464463ef569",
468 | "metadata": {},
469 | "outputs": [
470 | {
471 | "name": "stdout",
472 | "output_type": "stream",
473 | "text": [
474 | "Once upon a time, the world was a place of great beauty and great danger. The world was\n",
475 | "My name is Clara and I am a woman. I am a woman who is a woman. I am a\n"
476 | ]
477 | }
478 | ],
479 | "source": [
480 | "eos_id = tokenizer.encode(tokenizer.eos_token)[0]\n",
481 | "\n",
482 | "result = tokenizer(\"Once upon a time,\")\n",
483 | "input = result[\"input_ids\"]\n",
484 | "mask = result[\"attention_mask\"]\n",
485 | "input = pad_tokens(input, block_size, 0)\n",
486 | "mask = pad_tokens(mask, block_size, 0)\n",
487 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n",
488 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n",
489 | "\n",
490 | "result_token, result_len = generate_text(\n",
491 | " model,\n",
492 | " input,\n",
493 | " mask,\n",
494 | " eos_id,\n",
495 | " pred_sequence_length=15)\n",
496 | "print(tokenizer.decode(result_token[0][:result_len]))\n",
497 | "\n",
498 | "result = tokenizer(\"My name is Clara and I am\")\n",
499 | "input = result[\"input_ids\"]\n",
500 | "mask = result[\"attention_mask\"]\n",
501 | "input = pad_tokens(input, block_size, 0)\n",
502 | "mask = pad_tokens(mask, block_size, 0)\n",
503 | "input = torch.tensor([input], dtype=torch.int64).to(device)\n",
504 | "mask = torch.tensor([mask], dtype=torch.int64).to(device)\n",
505 | "\n",
506 | "result_token, result_len = generate_text(\n",
507 | " model,\n",
508 | " input,\n",
509 | " mask,\n",
510 | " eos_id,\n",
511 | " pred_sequence_length=15)\n",
512 | "print(tokenizer.decode(result_token[0][:result_len]))"
513 | ]
514 | },
515 | {
516 | "cell_type": "markdown",
517 | "id": "d48fb60b-c05d-4884-a9bc-92152c94c894",
518 | "metadata": {},
519 | "source": [
520 | "Now we generate text with our test dataset (5 rows).
\n",
521 | "As you can see below, it cannot output the completion well, because it's not still fine-tuned."
522 | ]
523 | },
524 | {
525 | "cell_type": "code",
526 | "execution_count": 15,
527 | "id": "495728ef-fbe6-4953-a354-4b7a8bb88798",
528 | "metadata": {},
529 | "outputs": [
530 | {
531 | "name": "stdout",
532 | "output_type": "stream",
533 | "text": [
534 | "********** input **********\n",
535 | "name : Wildwood | Type : pub | food : Indian | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
536 | "\n",
537 | "********** result **********\n",
538 | "name : Wildwood | Type : pub | food : Indian | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
539 | "\n",
540 | "Raja Indian Cuisine : Indian | price : Rs. 1,000 | menu : Indian | menu type : food | menu size :\n",
541 | "********** input **********\n",
542 | "name : Giraffe | Type : pub | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n",
543 | "\n",
544 | "********** result **********\n",
545 | "name : Giraffe | Type : pub | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n",
546 | "\n",
547 | ": Giraffe | Type : pub | food : Fast food | area : riverside | family friendly : yes | near : Rainbow Vegetarian Café\n",
548 | "********** input **********\n",
549 | "name : The Waterman | Type : pub | food : Italian | price : less than £ 20 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
550 | "\n",
551 | "********** result **********\n",
552 | "name : The Waterman | Type : pub | food : Italian | price : less than £ 20 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
553 | "\n",
554 | "The Waterman is a pub in the heart of the city centre. It is a place where you can enjoy a good meal and drink a good\n",
555 | "********** input **********\n",
556 | "name : The Vaults | Type : pub | food : Italian | price : moderate | customer rating : 1 out of 5 | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n",
557 | "\n",
558 | "********** result **********\n",
559 | "name : The Vaults | Type : pub | food : Italian | price : moderate | customer rating : 1 out of 5 | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n",
560 | "\n",
561 | "The Vaults | Type : pub | food : Italian | price : moderate | customer rating : 1 out of 5 | area : city centre | family\n",
562 | "********** input **********\n",
563 | "name : The Vaults | Type : restaurant | food : French | price : less than £ 20 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n",
564 | "\n",
565 | "********** result **********\n",
566 | "name : The Vaults | Type : restaurant | food : French | price : less than £ 20 | area : riverside | family friendly : yes | near : Raja Indian Cuisine\n",
567 | "\n",
568 | "The restaurant is located in the centre of the city. It is a small restaurant with a small menu. The menu is very simple and the food\n"
569 | ]
570 | }
571 | ],
572 | "source": [
573 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n",
574 | "test_data = test_data[::2] # because it's duplicated\n",
575 | "test_loader = DataLoader(\n",
576 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n",
577 | " batch_size=1,\n",
578 | " shuffle=True,\n",
579 | " collate_fn=collate_batch\n",
580 | ")\n",
581 | "\n",
582 | "for i, (input, _, mask) in enumerate(test_loader):\n",
583 | " if i == 5:\n",
584 | " break\n",
585 | " print(\"********** input **********\")\n",
586 | " input_len = torch.sum(mask).cpu().numpy()\n",
587 | " print(tokenizer.decode(input[0][:input_len]))\n",
588 | " result_token, result_len = generate_text(\n",
589 | " model,\n",
590 | " input,\n",
591 | " mask,\n",
592 | " eos_id,\n",
593 | " pred_sequence_length=30)\n",
594 | " print(\"********** result **********\")\n",
595 | " print(tokenizer.decode(result_token[0][:result_len]))"
596 | ]
597 | },
598 | {
599 | "cell_type": "markdown",
600 | "id": "e3138341-e01c-4fae-af78-c61e34967e92",
601 | "metadata": {},
602 | "source": [
603 | "## LoRA (Low-Rank Adaptation)\n",
604 | "\n",
605 | "Now we apply LoRA in our downloaded model.
\n",
606 | "For semantics of LoRA (Low-Rank Adaptation), see [01-finetune-opt-with-lora.ipynb](./01-finetune-opt-with-lora.ipynb).\n",
607 | "\n",
608 | "For the purpose of your learning, here I manually (from scratch) convert the current model into the model with LoRA.\n",
609 | "\n",
610 | "> Note : You can use ```PEFT``` package to be able to get LoRA model with a few lines of code. (Here I don't use this package.)"
611 | ]
612 | },
613 | {
614 | "cell_type": "markdown",
615 | "id": "e296bdf2-129a-4278-8fe3-d08333ebf1df",
616 | "metadata": {},
617 | "source": [
618 | "Before changing our model, first we check the structure of our model. (See the following result in the cell.)\n",
619 | "\n",
620 | "As you can see below, you cannot find any linear layers in OpenAI's GPT-2 transformer, unlike [Meta's OPT transformer](./01-finetune-opt-with-lora.ipynb). Instead, you will find Conv1D (1D convolution) in transformer.
\n",
621 | "However, this Conv1D is not ```torch.nn.Conv1d``` and it's a custom layer defined for OpenAI GPT, which works same as a linear layer, but the weights are transposed. (See [this source code](https://github.com/huggingface/transformers/blob/main/src/transformers/pytorch_utils.py) for custom ```pytorch_utils.Conv1D``` implementation.)
\n",
622 | "This custom Conv1D layer (intrinsically, a linear layer) is used for MLP and getting key/value/query in GPT-2 transformer as follows.
\n",
623 | "(See [source code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_gpt2.py) for GPT-2 in Hugging Face tarnsformers.)\n",
624 | "\n",
625 | "- ```transformer.h.n.attn.c_attn``` : Layer to get key/value/query before processing attention.\n",
626 | "- ```transformer.h.n.attn.c_proj``` : Layer for projection after processing attention.\n",
627 | "- ```transformer.h.n.mlp.c_attn``` : MLP in GPT-2 is Linear(GeLU(Linear)). This is an inner Linear layer (custom Conv1D).\n",
628 | "- ```transformer.h.n.mlp.c_proj``` : MLP in GPT-2 is Linear(GeLU(Linear)). This is an outer Linear layer (custom Conv1D).\n",
629 | "\n",
630 | "In this example, we'll only convert ```transformer.h.n.attn.c_attn``` layers into LoRA layers.
\n",
631 | "The transformer in GPT-2 with 124M parameters has 12 layers and it then has total 12 layers (n=0,1, ... , 11) to be converted."
632 | ]
633 | },
634 | {
635 | "cell_type": "code",
636 | "execution_count": 16,
637 | "id": "5acb8f62-791a-4fa4-b00c-2666cf34827f",
638 | "metadata": {},
639 | "outputs": [
640 | {
641 | "data": {
642 | "text/plain": [
643 | "GPT2LMHeadModel(\n",
644 | " (transformer): GPT2Model(\n",
645 | " (wte): Embedding(50257, 768)\n",
646 | " (wpe): Embedding(1024, 768)\n",
647 | " (drop): Dropout(p=0.1, inplace=False)\n",
648 | " (h): ModuleList(\n",
649 | " (0-11): 12 x GPT2Block(\n",
650 | " (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
651 | " (attn): GPT2Attention(\n",
652 | " (c_attn): Conv1D()\n",
653 | " (c_proj): Conv1D()\n",
654 | " (attn_dropout): Dropout(p=0.1, inplace=False)\n",
655 | " (resid_dropout): Dropout(p=0.1, inplace=False)\n",
656 | " )\n",
657 | " (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
658 | " (mlp): GPT2MLP(\n",
659 | " (c_fc): Conv1D()\n",
660 | " (c_proj): Conv1D()\n",
661 | " (act): NewGELUActivation()\n",
662 | " (dropout): Dropout(p=0.1, inplace=False)\n",
663 | " )\n",
664 | " )\n",
665 | " )\n",
666 | " (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
667 | " )\n",
668 | " (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
669 | ")"
670 | ]
671 | },
672 | "execution_count": 16,
673 | "metadata": {},
674 | "output_type": "execute_result"
675 | }
676 | ],
677 | "source": [
678 | "model"
679 | ]
680 | },
681 | {
682 | "cell_type": "markdown",
683 | "id": "045e7239-cb8a-46dd-815d-e48e7e49eea4",
684 | "metadata": {},
685 | "source": [
686 | "First we build custom linear layer with LoRA as follows."
687 | ]
688 | },
689 | {
690 | "cell_type": "code",
691 | "execution_count": 17,
692 | "id": "77889272-9a93-491b-93cb-b0bed5ce7cd8",
693 | "metadata": {},
694 | "outputs": [],
695 | "source": [
696 | "import math\n",
697 | "from torch import nn\n",
698 | "\n",
699 | "class LoRA_Linear(nn.Module):\n",
700 | " def __init__(self, weight, bias, lora_dim):\n",
701 | " super(LoRA_Linear, self).__init__()\n",
702 | "\n",
703 | " row, column = weight.shape\n",
704 | "\n",
705 | " # restore Linear\n",
706 | " if bias is None:\n",
707 | " self.linear = nn.Linear(column, row, bias=False)\n",
708 | " self.linear.load_state_dict({\"weight\": weight})\n",
709 | " else:\n",
710 | " self.linear = nn.Linear(column, row)\n",
711 | " self.linear.load_state_dict({\"weight\": weight, \"bias\": bias})\n",
712 | "\n",
713 | " # create LoRA weights (with initialization)\n",
714 | " self.lora_right = nn.Parameter(torch.zeros(column, lora_dim))\n",
715 | " nn.init.kaiming_uniform_(self.lora_right, a=math.sqrt(5))\n",
716 | " self.lora_left = nn.Parameter(torch.zeros(lora_dim, row))\n",
717 | "\n",
718 | " def forward(self, input):\n",
719 | " x = self.linear(input)\n",
720 | " y = input @ self.lora_right @ self.lora_left\n",
721 | " return x + y"
722 | ]
723 | },
724 | {
725 | "cell_type": "markdown",
726 | "id": "954e2c9d-545e-4bd9-9b0f-eba3fe29a1de",
727 | "metadata": {},
728 | "source": [
729 | "Replace targeting linear layers with LoRA layers.\n",
730 | "\n",
731 | "> Note : As I have mentioned above, custom Conv1D layer in GPT-2 is intrinsically a linear layer, but the weights are transposed."
732 | ]
733 | },
734 | {
735 | "cell_type": "code",
736 | "execution_count": 18,
737 | "id": "baf8a748-a3e3-45b8-9c64-252c56abe923",
738 | "metadata": {},
739 | "outputs": [],
740 | "source": [
741 | "lora_dim = 128\n",
742 | "\n",
743 | "# get target module name\n",
744 | "target_names = []\n",
745 | "for name, module in model.named_modules():\n",
746 | " if \"attn.c_attn\" in name:\n",
747 | " target_names.append(name)\n",
748 | "\n",
749 | "# replace each module with LoRA\n",
750 | "for name in target_names:\n",
751 | " name_struct = name.split(\".\")\n",
752 | " # get target module\n",
753 | " module_list = [model]\n",
754 | " for struct in name_struct:\n",
755 | " module_list.append(getattr(module_list[-1], struct))\n",
756 | " # build LoRA\n",
757 | " lora = LoRA_Linear(\n",
758 | " weight = torch.transpose(module_list[-1].weight, 0, 1),\n",
759 | " bias = module_list[-1].bias,\n",
760 | " lora_dim = lora_dim,\n",
761 | " ).to(device)\n",
762 | " # replace\n",
763 | " module_list[-2].__setattr__(name_struct[-1], lora)"
764 | ]
765 | },
766 | {
767 | "cell_type": "markdown",
768 | "id": "8aae2df9-fae7-4ecc-8260-80e8e578d951",
769 | "metadata": {},
770 | "source": [
771 | "See how model is changed."
772 | ]
773 | },
774 | {
775 | "cell_type": "code",
776 | "execution_count": 19,
777 | "id": "bf16b414-b973-40eb-be81-fd2aa3dde439",
778 | "metadata": {},
779 | "outputs": [
780 | {
781 | "data": {
782 | "text/plain": [
783 | "GPT2LMHeadModel(\n",
784 | " (transformer): GPT2Model(\n",
785 | " (wte): Embedding(50257, 768)\n",
786 | " (wpe): Embedding(1024, 768)\n",
787 | " (drop): Dropout(p=0.1, inplace=False)\n",
788 | " (h): ModuleList(\n",
789 | " (0-11): 12 x GPT2Block(\n",
790 | " (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
791 | " (attn): GPT2Attention(\n",
792 | " (c_attn): LoRA_Linear(\n",
793 | " (linear): Linear(in_features=768, out_features=2304, bias=True)\n",
794 | " )\n",
795 | " (c_proj): Conv1D()\n",
796 | " (attn_dropout): Dropout(p=0.1, inplace=False)\n",
797 | " (resid_dropout): Dropout(p=0.1, inplace=False)\n",
798 | " )\n",
799 | " (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
800 | " (mlp): GPT2MLP(\n",
801 | " (c_fc): Conv1D()\n",
802 | " (c_proj): Conv1D()\n",
803 | " (act): NewGELUActivation()\n",
804 | " (dropout): Dropout(p=0.1, inplace=False)\n",
805 | " )\n",
806 | " )\n",
807 | " )\n",
808 | " (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
809 | " )\n",
810 | " (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
811 | ")"
812 | ]
813 | },
814 | "execution_count": 19,
815 | "metadata": {},
816 | "output_type": "execute_result"
817 | }
818 | ],
819 | "source": [
820 | "model"
821 | ]
822 | },
823 | {
824 | "cell_type": "markdown",
825 | "id": "e9099c08-f6a6-45f8-939b-cc3ed9415976",
826 | "metadata": {},
827 | "source": [
828 | "Finally, freeze all parameters except for LoRA parameters."
829 | ]
830 | },
831 | {
832 | "cell_type": "code",
833 | "execution_count": 20,
834 | "id": "81d06bba-955b-4806-8ff7-f217252e3268",
835 | "metadata": {},
836 | "outputs": [],
837 | "source": [
838 | "for name, param in model.named_parameters():\n",
839 | " if \"lora_right\" in name or \"lora_left\" in name:\n",
840 | " param.requires_grad = True\n",
841 | " else:\n",
842 | " param.requires_grad = False"
843 | ]
844 | },
845 | {
846 | "cell_type": "code",
847 | "execution_count": null,
848 | "id": "6c0a4469-2827-4f30-9324-711a9feea1ae",
849 | "metadata": {},
850 | "outputs": [],
851 | "source": [
852 | "### Do this when you run adapter fine-tuning on Hugging Face framework\n",
853 | "# model.gradient_checkpointing_enable()\n",
854 | "# model.enable_input_require_grads()"
855 | ]
856 | },
857 | {
858 | "cell_type": "markdown",
859 | "id": "6d6c7d6f-6c50-4839-88a5-c851caab9ba2",
860 | "metadata": {},
861 | "source": [
862 | "## Fine-tune"
863 | ]
864 | },
865 | {
866 | "cell_type": "markdown",
867 | "id": "a12b875f-36cc-40b8-aaab-1efda68710f3",
868 | "metadata": {},
869 | "source": [
870 | "Now let's start to run fine-tuning.\n",
871 | "\n",
872 | "First we build optimizer as follows."
873 | ]
874 | },
875 | {
876 | "cell_type": "code",
877 | "execution_count": 21,
878 | "id": "bb51298a-2d55-466c-a990-0ea08a247350",
879 | "metadata": {},
880 | "outputs": [],
881 | "source": [
882 | "optimizer = torch.optim.AdamW(\n",
883 | " params=model.parameters(),\n",
884 | " lr=0.0002,\n",
885 | " betas=(0.9, 0.999),\n",
886 | " eps=1e-6,\n",
887 | ")"
888 | ]
889 | },
890 | {
891 | "cell_type": "markdown",
892 | "id": "d37db1a8-0053-4acc-94ce-89d87c78942e",
893 | "metadata": {},
894 | "source": [
895 | "In this example, we build linear scheduler for training."
896 | ]
897 | },
898 | {
899 | "cell_type": "code",
900 | "execution_count": 22,
901 | "id": "6f95bdf6-4498-4d40-90aa-1267d55f38c3",
902 | "metadata": {},
903 | "outputs": [],
904 | "source": [
905 | "from torch.optim.lr_scheduler import LambdaLR\n",
906 | "\n",
907 | "num_epochs = 2\n",
908 | "num_warmup_steps = 500\n",
909 | "\n",
910 | "num_update_steps = math.ceil(len(dataloader) / batch_size / gradient_accumulation_steps)\n",
911 | "def _get_linear_schedule(current_step):\n",
912 | " if current_step < num_warmup_steps:\n",
913 | " return float(current_step) / float(max(1, num_warmup_steps))\n",
914 | " return max(0.0, float(num_update_steps * num_epochs - current_step) / float(max(1, num_update_steps * num_epochs - num_warmup_steps)))\n",
915 | "scheduler = LambdaLR(optimizer, lr_lambda=_get_linear_schedule)"
916 | ]
917 | },
918 | {
919 | "cell_type": "markdown",
920 | "id": "a9f9e828-c4fb-493d-a6de-78e03dbf035e",
921 | "metadata": {},
922 | "source": [
923 | "Run fine-tuning."
924 | ]
925 | },
926 | {
927 | "cell_type": "code",
928 | "execution_count": 23,
929 | "id": "3752481d-8ee8-4c43-b677-add136a2fd5b",
930 | "metadata": {},
931 | "outputs": [
932 | {
933 | "name": "stdout",
934 | "output_type": "stream",
935 | "text": [
936 | "Epoch 1 42/42 - loss: 1.3620\n",
937 | "Epoch 2 42/42 - loss: 1.4432\n"
938 | ]
939 | }
940 | ],
941 | "source": [
942 | "from torch.nn import functional as F\n",
943 | "\n",
944 | "if os.path.exists(\"loss.txt\"):\n",
945 | " os.remove(\"loss.txt\")\n",
946 | "\n",
947 | "for epoch in range(num_epochs):\n",
948 | " optimizer.zero_grad()\n",
949 | " model.train()\n",
950 | " for i, (inputs, labels, masks) in enumerate(dataloader):\n",
951 | " with torch.set_grad_enabled(True):\n",
952 | " outputs = model(\n",
953 | " input_ids=inputs,\n",
954 | " attention_mask=masks,\n",
955 | " )\n",
956 | " loss = F.cross_entropy(outputs.logits.transpose(1,2), labels)\n",
957 | " loss.backward()\n",
958 | " if ((i + 1) % gradient_accumulation_steps == 0) or \\\n",
959 | " (i + 1 == len(dataloader)):\n",
960 | " optimizer.step()\n",
961 | " scheduler.step()\n",
962 | " optimizer.zero_grad()\n",
963 | "\n",
964 | " print(f\"Epoch {epoch+1} {math.ceil((i + 1) / batch_size / gradient_accumulation_steps)}/{num_update_steps} - loss: {loss.item() :2.4f}\", end=\"\\r\")\n",
965 | "\n",
966 | " # record loss\n",
967 | " with open(\"loss.txt\", \"a\") as f:\n",
968 | " f.write(str(loss.item()))\n",
969 | " f.write(\"\\n\")\n",
970 | " print(\"\")\n",
971 | "\n",
972 | "# save model\n",
973 | "torch.save(model.state_dict(), \"finetuned_gpt2.bin\")"
974 | ]
975 | },
976 | {
977 | "cell_type": "markdown",
978 | "id": "83993d92-d7ed-4a07-8985-cc59bd4e4fef",
979 | "metadata": {},
980 | "source": [
981 | "> Note : Here we save LoRA-enabled model without any changes, but you can also merge the trained LoRA's parameters into the original model's weights."
982 | ]
983 | },
984 | {
985 | "cell_type": "markdown",
986 | "id": "1bc086e5-e93f-4264-a8fa-6428f844ac3c",
987 | "metadata": {},
988 | "source": [
989 | "Show loss transition in plot."
990 | ]
991 | },
992 | {
993 | "cell_type": "code",
994 | "execution_count": 24,
995 | "id": "e37c5aee-38d4-4a2a-952c-4fd2bef41e2b",
996 | "metadata": {},
997 | "outputs": [
998 | {
999 | "data": {
1000 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABV+0lEQVR4nO3deVhU9f4H8PewDagsgrIooLhvgLijlZoaGplW18praaVWXv2l1dWifTMs89puVrds0SjLpWsu4YJm4oaigIq7ILK4sYoIzPn9gYwzMNuZOTNnhnm/nmcemTPfc85nDjjnM99VIQiCACIiIiKZuMgdABERETk3JiNEREQkKyYjREREJCsmI0RERCQrJiNEREQkKyYjREREJCsmI0RERCQrJiNEREQkKze5AzCFSqXChQsX4O3tDYVCIXc4REREZAJBEFBWVoY2bdrAxUV//YdDJCMXLlxAWFiY3GEQERGRGXJzcxEaGqr3dYdIRry9vQHUvRkfHx+ZoyEiIiJTlJaWIiwsTH0f18chkpH6phkfHx8mI0RERA7GWBcLdmAlIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkx4GDOVXy36ywEQZA7FCIioibLomRkwYIFUCgUmDNnjt4yy5Ytg0Kh0Hp4enpaclqbue/zXXj99yxszCyQOxQiIqImy83cHfft24elS5ciKirKaFkfHx9kZ2ernxtbStjenLpYLncIRERETZZZNSPl5eWYNGkSvvrqK7Rs2dJoeYVCgeDgYPUjKCjInNMSERFRE2RWMjJz5kzEx8dj5MiRJpUvLy9Hu3btEBYWhnHjxiErK8tg+aqqKpSWlmo9rKlWJWDad/vxn+TjVj0PERERNSY6GUlKSsKBAweQmJhoUvmuXbvim2++wdq1a/Hjjz9CpVJh8ODBOH/+vN59EhMT4evrq36EhYWJDVOUHScuYvPRQny85YRVz0NERESNiUpGcnNzMXv2bCxfvtzkTqixsbGYPHkyevfujaFDh2LVqlVo3bo1li5dqnefhIQElJSUqB+5ubliwhStqlpl8HUOpiEiIrIeUR1Y09LSUFRUhD59+qi31dbWYseOHfj0009RVVUFV1dXg8dwd3dHTEwMTp48qbeMUqmEUqkUExoRERE5KFHJyIgRI5CRkaG17fHHH0e3bt3wwgsvGE1EgLrkJSMjA3fffbe4SImIiKhJEpWMeHt7o1evXlrbmjdvjoCAAPX2yZMno23btuo+JW+99RYGDRqETp06obi4GAsXLsS5c+cwbdo0id6CdVypuKH++XpNrYyREBERNW1mzzOiT05ODlxcbnVFuXr1KqZPn46CggK0bNkSffv2xa5du9CjRw+pTy2p69W3EpBaw11KiIiIyAIWJyMpKSkGny9evBiLFy+29DQ2xz6rREREtsG1aYiIiEhWTEYa2H36MhZuOoba2lt1IwLrSYiIiKxG8j4jju7hL3cDAKprmYAQERHZAmtG9Dh7qUL9swKOtbAfERGRI2EyQkRERLJiMmICBStGiIiIrIbJiAm4Ng0REZH1MBkBUKVjhlXmH0RERLbBZATA7KT0RtvYMkNERGQbTEaIiIhIVkxG9GAzDRERkW0wGSEiIiJZMRkhIiIiWTEZMYHmPCNp564gYdVhpJ27qjVLKxEREZmHa9OI9MCSVADAT3tzAQAn54+BmytzOiIiInPxLmoCQ5Oe1ajY1ZWIiMgSTEaIiIhIVmym0aOqRqX+WYCAN/+Xhay8UhkjIiIiapqcOhn5Yfc5fLvzjM7Xdhy/qPX827/P2iAiIiIi5+PUycirazLlDoGIiMjpsc+ICS4UX5c7BCIioiaLyYgJ/nfogtwhEBERNVlMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEYklHbuKnKvXJM7DCIiIofi1NPBS+lkURkeWLILAHB2QbzM0RARETkO1oxYKK+4EgCQyRV9iYiIzMJkxEIjFm1HVU2t3GEQERE5LCYjEiitrJE7BCIiIodlUTKyYMECKBQKzJkzx2C5lStXolu3bvD09ERkZCTWr19vyWmJiIioCTE7Gdm3bx+WLl2KqKgog+V27dqFiRMnYurUqTh48CDGjx+P8ePHIzMz09xT26Xr1beaat74PQs7jl+UMRoiIiLHYVYyUl5ejkmTJuGrr75Cy5YtDZb96KOPMHr0aMydOxfdu3fH22+/jT59+uDTTz81K2B79dLqDPXPy3adxeRv9soYDRERkeMwKxmZOXMm4uPjMXLkSKNlU1NTG5WLi4tDamqqOae2WypB7giIiIgck+h5RpKSknDgwAHs27fPpPIFBQUICgrS2hYUFISCggK9+1RVVaGqqkr9vLTUvofNHsotNrnsrpOX8Ob/juDd+yPRt53hWiUiIiJnIKpmJDc3F7Nnz8by5cvh6elprZiQmJgIX19f9SMsLMxq55LCtO/3m1z2n1/vQXZhGSZ+tduKERERETkOUclIWloaioqK0KdPH7i5ucHNzQ3bt2/Hxx9/DDc3N9TWNp5vIzg4GIWFhVrbCgsLERwcrPc8CQkJKCkpUT9yc3PFhGk3BEF/282NGpUNIyEiIrJfopKRESNGICMjA+np6epHv379MGnSJKSnp8PV1bXRPrGxsdiyZYvWtuTkZMTGxuo9j1KphI+Pj9bDEc1ccUDuEIiIiOyeqD4j3t7e6NWrl9a25s2bIyAgQL198uTJaNu2LRITEwEAs2fPxtChQ7Fo0SLEx8cjKSkJ+/fvx5dffinRW7Bf6zP094shIiKiOpLPwJqTk4P8/Hz188GDB2PFihX48ssvER0djV9//RVr1qxplNQQERGRc7J41d6UlBSDzwFgwoQJmDBhgqWnIiIioiaIa9MQERGRrJiMEBERkayYjBAREZGsmIwQERGRrJiMEBERkayYjBAREZGsmIwQERGRrJiMEBERkayYjBAREZGsmIwQERGRrJiMWNmpi+U4UVgmdxhERER2i8mIlY1YtB2jFu9A5Y3aRq+pVIIMEREREdkXJiM2UlJZ3Wjbu+uPyhAJERGRfWEyYiMCGteCfL3zDM5eqpAhGiIiIvvBZMRGhi5MwfepZxttf2vdEdsHQ0REZEeYjNjIjRoVXlub1Wh7LfuNEBGRk2MyIrPtxy/iP8nHIQhMSoiIyDkxGbEDH285gX1nr8odBhERkSyYjNiJKxU35A6BiIhIFkxGiIiISFZMRoiIiEhWTEbsBjuwEhGRc2IyQkRERLJiMmJnko8U4uMtJzjUl4iInIab3AFQPQUAYPr3+wEA0WF+GNqltZwBERER2QRrRuyGdk1IYel1meIgIiKyLSYjREREJCsmI0RERCQrJiNEREQkKyYjREREJCsmI0RERCQrJiNEREQkKyYjdmLG8gPYcfyi3GEQERHZHJMROyEIwORv9sodBhERkc2JSkaWLFmCqKgo+Pj4wMfHB7GxsdiwYYPe8suWLYNCodB6eHp6Whw0ERERNR2ipoMPDQ3FggUL0LlzZwiCgO+++w7jxo3DwYMH0bNnT537+Pj4IDs7W/1coVBYFjERERE1KaKSkbFjx2o9nz9/PpYsWYLdu3frTUYUCgWCg4PNj5CIiIiaNLP7jNTW1iIpKQkVFRWIjY3VW668vBzt2rVDWFgYxo0bh6ysLHNPSURERE2Q6FV7MzIyEBsbi+vXr6NFixZYvXo1evToobNs165d8c033yAqKgolJSX44IMPMHjwYGRlZSE0NFTvOaqqqlBVVaV+XlpaKjZMIiIichCia0a6du2K9PR07NmzBzNmzMCUKVNw5MgRnWVjY2MxefJk9O7dG0OHDsWqVavQunVrLF261OA5EhMT4evrq36EhYWJDZOIiIgchOhkxMPDA506dULfvn2RmJiI6OhofPTRRybt6+7ujpiYGJw8edJguYSEBJSUlKgfubm5YsMkIiIiB2HxPCMqlUqrScWQ2tpaZGRkICQkxGA5pVKpHj5c/yAiIqKmSVSfkYSEBIwZMwbh4eEoKyvDihUrkJKSgk2bNgEAJk+ejLZt2yIxMREA8NZbb2HQoEHo1KkTiouLsXDhQpw7dw7Tpk2T/p0QERGRQxKVjBQVFWHy5MnIz8+Hr68voqKisGnTJowaNQoAkJOTAxeXW5UtV69exfTp01FQUICWLVuib9++2LVrl94Or0REROR8FIIgCHIHYUxpaSl8fX1RUlIiaZNN+xf/kOxYUnv/H1F4sB877hIRkeMy9f7NtWmIiIhIVkxGiIiISFZMRhzMvrNXMPDdzdiYmS93KERERJJgMuJgHvtmLwpLq/D0jwfkDoWIiEgSTEY0LLg/EvfHtJU7DIOqa+2+vzEREZEootemaYqiQn2x8ulYKN1c8fCAcDxxWwRmLE9D7pVKuUNTO3y+GBl5JRDAZISIiJoWJiM3Kd1c1T/3auuLqLZ+dpWM3Pvp33KHQEREZBVspgGgMHmjbeVeuYbr1bVyh0FERGRVTEb0kDsXyThfgtvf34a7P/5L5kiIiIisi8mInfr90AUAwOmLFTJHQkREZF1MRvRQKOStGymprJb1/ERERLbCZAQAdCQecjfTEBEROQsmI3rIXDFCRETkNJw6GRnSKQAA8OigdjJHQkRE5Lycep6Rbx8bgLOXK9A5sEWj11gxQkREZBtOXTPi4eaCLkHeJnVWbdXCA6/Ed7dBVERERM7FqWtGTPX5pD7o394frb2VeOePo3KHQ0RE1KQ4dc2IIZq1JXdHhqC1txIA8Oo9PWweS+6Va3pfO5BzFeM/+xsHcq7aMCIiIiLpMBnRQ1/DzdTbIrBh9u14fEh7m8Vy+/vb9L72wJJdSM8txgNLdtksHiIiIikxGdGjT7uWel/rHuKDiFbNbRiNfoKg/S8REZGjYZ8RPSYOCIebiwL9I/x1vv5w/3C8tjbLxlERERE1PUxG9HB1UeDhAeF6X/dwY6USERGRFHhHJSIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZscCYXsFyh0BEROTwmIxYoIWS07QQERFZiskIERERyYrJiAVcXfQtp2cbc5IOynp+IiIiKTAZscDskZ0R6K2U7fxr0i/Idm4iIiKpMBmxQIivF/a8NKLR9o6tm8NN5loTIiIiRyEqGVmyZAmioqLg4+MDHx8fxMbGYsOGDQb3WblyJbp16wZPT09ERkZi/fr1FgVsbxQKBQ68Ogq7XrxTva2Fp7uMERERETkWUclIaGgoFixYgLS0NOzfvx933nknxo0bh6ysLJ3ld+3ahYkTJ2Lq1Kk4ePAgxo8fj/HjxyMzM1OS4O2Ff3MPtPHzwheP9EWvtj5Y/GC03CERERE5DIUgCIIlB/D398fChQsxderURq899NBDqKiowLp169TbBg0ahN69e+OLL74w+RylpaXw9fVFSUkJfHx8LAnXZp5Ytg9bjxXZ9JxnF8Tb9HxERESGmHr/NrvPSG1tLZKSklBRUYHY2FidZVJTUzFy5EitbXFxcUhNTTX3tA7jPw9G44XR3eQOg4iIyO6JnrUrIyMDsbGxuH79Olq0aIHVq1ejR48eOssWFBQgKChIa1tQUBAKCgoMnqOqqgpVVVXq56WlpWLDlJ1fMw/MGNYRNbUqLEo+Lnc4REREdkt0zUjXrl2Rnp6OPXv2YMaMGZgyZQqOHDkiaVCJiYnw9fVVP8LCwiQ9vi2FBzSTOwQiIiK7JjoZ8fDwQKdOndC3b18kJiYiOjoaH330kc6ywcHBKCws1NpWWFiI4GDDa7okJCSgpKRE/cjNzRUbJhERETkIi+cZUalUWk0qmmJjY7FlyxatbcnJyXr7mNRTKpXq4cP1DyIiImqaRPUZSUhIwJgxYxAeHo6ysjKsWLECKSkp2LRpEwBg8uTJaNu2LRITEwEAs2fPxtChQ7Fo0SLEx8cjKSkJ+/fvx5dffin9OyEiIiKHJCoZKSoqwuTJk5Gfnw9fX19ERUVh06ZNGDVqFAAgJycHLi63KlsGDx6MFStW4JVXXsFLL72Ezp07Y82aNejVq5e074KIiIgclsXzjNiCI84zUm9teh5mJ6Xb5Fxz47riX8M6QqHgVPRERCQ/q88zQvZn4aZsbD7aeKK1mloVampVMkRERERkHJMRKwtt6WXT8+Vdvab1vFYlYOjCFAxflAKVyu4rwYiIyAkxGbGyvu38bXq+hunG5fIq5BVXIvdKJcqu16BWJSAzrwS1TEyIiMhOMBlxIgIEJK4/ins+2Ym3/qd7cUMiIiJbYzLiZL7eeQYA8F3qOZkjISIiqsNkpIkxNDZq2a6zNouDiIjIVExGmqgzlyqw4/hFQGOU74ebT8gXEBERkR6iV+0lxzD8gxQAwNeT+8kbCBERkRGsGWnipn2/X+4QiIiIDGIyYkMDI/yxYvpAAEBshwCrnIMDdomIyNGwmcaGHhnUDoM7tsLRt0bD090FEQnr5Q6JiIhIdkxGZODl4Sp3CERERHaDzTQ2MLRLa/h6uWN4t0C5QyEiIrI7rBmxgWWP90eNSoC7q/Vzv5TsIkwcEGb18xAREUmFyYgNKBQKuLsqjBeUwF8nLmHWioM2ORcREZEU2EzTBG09ViSq/KasAjz4RSrON1jxl4iIyBaYjDixL7afwqXyKjz1Qxr2nr2ChFUZAIDPtp3EZ9tOyhwdERE5CzbTOLEFG45hfUa++nlJZTX2nrmChZuyAdQNRfb1cpcrPCIichKsGXFyh8+XaD1/cGmq+ueaWpVVzy0YWtWPiIicBpMRksXZSxUY8O4WfLXjtNyhEBGRzJiMkMUEQUDqqcsouVZt8j5vrzuCi2VVmL/+qBUjIyIiR8BkREY/TB0gdwhaGjbZfJ5yyqT9Vh3Iw8SvdmPMRztMPpeKTTRERHQTkxEZ3d65tdwhGPTfnWdMKlffCfZCyXVrhkMSu1hWhT8O56Payn2DiIiMYTJC5KTu/XQnZq44gC9MrAEjIrIWJiMkSk2tCj/vy8Hpi+XqbQrbTC5LEsu/WZOVfLRQ5kiIyNkxGSFRftqbgxd+y8Cdi7ZrbDWejeRcvoarFTesFxgRETksJiMkyv5zV9U/19eObDbyzTq/pBJ3LNyGmLeTJY1ldtJBTPtuH+crISJycExGZPbsyC5yh2A27doR/Q7llhgvJNKNGhXWpl/A5qNFyL1SKfnxiYjIdpiMyGz2yM5yh2BQXrF93ugFCDp/JiIix8NkhAySYkr4pt7Bdc/py5i78hCKr7FPDBGRObhQHhl1ubwK6bnFuHqtGvvPXjW+gwkUTShDeejL3QAAlQAsejBa5miIiBwPkxE70rddS6Sdk+ZmL6V7PtmpHgZqDl1px8WyKpP2zbpQgu92ncWzo7ogxNfL7BhsIedKhdwhEBE5JCYjdiSguYfcIehkSSKiT0aeaZ1a4z/eCQA4fbECv84YLHkcZMrAbCIi62KfEbJbZy/dqmk4XlgmYyRERGRNopKRxMRE9O/fH97e3ggMDMT48eORnZ1tcJ9ly5ZBoVBoPTw9PS0Kmpq+8qoaDPsgRf2c42Wsh9eWiOQmKhnZvn07Zs6cid27dyM5ORnV1dW46667UFFhuK3cx8cH+fn56se5c+csCpoci2ZnVc3aDkMKS22z6N7pi+XYfITToRMRyUlUn5GNGzdqPV+2bBkCAwORlpaGO+64Q+9+CoUCwcHB5kVIsvrfoQtm7ScIAgQBcHFRaPVJeHHVYSQ9GWvGAc0Kw6j6idt+mj4IsR0DrHMS0lJ6vRoqlQC/ZvbZR4qIbM+iPiMlJXWdEP39/Q2WKy8vR7t27RAWFoZx48YhKyvLktOSDX3w53GDr3+69YTO7Y99uw8jF29vtDz9tRu1ksUmpUwTO9QaomBXUKMEQUDUG3+i91vJuF5tn38LRGR7ZicjKpUKc+bMwZAhQ9CrVy+95bp27YpvvvkGa9euxY8//giVSoXBgwfj/PnzevepqqpCaWmp1sMZOGLbvb5kZfvxizh9sQKHzxdLMumZI14be/TLvlz8bmZtlxRqVLd+k9YYpUVEjsnsZGTmzJnIzMxEUlKSwXKxsbGYPHkyevfujaFDh2LVqlVo3bo1li5dqnefxMRE+Pr6qh9hYWHmhulQmsL3akEQ8Oh/96ifSzCBq57z3Pq5qkYl+Wib6loV1qbn2azvii0UlV3HvN8O45mfDkoysy4RkVTMSkZmzZqFdevWYdu2bQgNDRW1r7u7O2JiYnDy5Em9ZRISElBSUqJ+5ObmmhOmw3FpArOSnigqx18nLqmfq8xcUfdSg0nRyqtq8HnKSZwsKm9U9oElu3DX4h34M6vArHPp8uWO05idlI67Fu+Q7JhyK7teo/5Z87fi+H91ROToRCUjgiBg1qxZWL16NbZu3YqIiAjRJ6ytrUVGRgZCQkL0llEqlfDx8dF6OANPdxc8dUcHucOwSK1KO/lomIzkXLmGAzmGZ5m9Xl2rnmJd0/sbszHyP3UdTjXztvqb7K9p+pv+xNp6rAgAUFJZLdkx5XKpvErdoZiIyB6JSkZmzpyJH3/8EStWrIC3tzcKCgpQUFCAyspbK7tOnjwZCQkJ6udvvfUW/vzzT5w+fRoHDhzAI488gnPnzmHatGnSvYsmJOHu7gj3byZ3GGZreMNTqbQTh+Jr1bj/810Gj2HqVPG6ZBeUoahB04pKJeDnfTk4YeWJ00xdPTi/pBIvrc6wyURuKdlF6PfOZjyTlG71cxERmUtUMrJkyRKUlJRg2LBhCAkJUT9+/vlndZmcnBzk5+ern1+9ehXTp09H9+7dcffdd6O0tBS7du1Cjx49pHsXZDdMvSFbw+ajhYj7cAcGvLtFa/uqg3l44bcMjDLQ5FJfe2ALM5cfwIo9OYj/+C+rn+vzlFMAzB+iTXUEQUDCqsN6R48RkWVEzTNiyod1SkqK1vPFixdj8eLFooKipkXskFdTcgJdZVR69juUW2z0eEt3nMal8hsWrbpr6vvMulA3Oqy6Vr7Ejf1ExMnMK8VPe+v6rs26s7PM0RA1PVybxg7JWbtgqe3HL2o9Vyggy53vQnGl8UIN/HZAuj4n9s7Sv7Cy65b3pbFVTZQUKjknCpFVMRmxI4omMJrm/Y2G1yoyhRSX4bW1mWbtl19SiR92n3OqCbnEpgQfbj6OyDf+xLrD4pt+HP8vnIisgcmIHdL8wrj35RHyBSIRsTcgKb4wbz5aZNZ+Yz/5G6+uyVQ3pcil7Ho1fk07j5JrUo7mkaYm4sPNdf0mXlkjPuFznLoQIrIlJiN2LtDbsVc4FgT7vQHpqv24VG7+SB4pzfv1MP698hCe/jENgiDgasUNSY+vmfDJVVvRFGoCiUgaTEbs0LTb6uZvGdk9SOZImrYbFsxCqlIJeHVNJn7Zb50J+TZk1k3glnr6Mv7vp4OIeTsZu09fxrnLFZj+/X6jc7UQsDY9D3ct3o7TFxtPlEe6HT5fbFKHbyKpiRpNQ7YxZXB7DIgIQOegFnKHIgm5vv+m5xajd5ifVY6dfLQQP+w+p73RSm903eG6ofJLt59CYWkVjuSXIvlIIc4uiLfOCU1k7/1PZ9+cW2Xer4fx64zB8gYjQlHZdfh4usPT3VXUfv/deQYebi54dFA7s857vboW9376NwDg6Fuj4eUh7vxSKr52A+szChAfFQJfL3fZ4iDbYc2IHVIoFOjRxgfurvz16NJwlld9jK3Ea0nuUHxN6mYTAbtPXzbaTJR75ZrIA+vbbOeZhISkWCnaViN/zl2uwID5WzBi0XZR+10sq8Lb647g1TWZqKox7/1Walyn8qoaAyWt76kf0vDS6gzMSTooaxxkO7zb2YFmN7+BDOvaWufrbi6O27b+18mLWmvVSOGeT3aaOBeJ4UKW9Fmw5N6ka9etx4rw8Je7MWTBVvMPbANSDOk1RBAEfLzlhFkjdZqCLTc7XueJHJqu2f/J3musTLHnzBUAwLbsi0ZKUlPBZho7sH3ucBzJL8UdnVvpfH3OyM744M/jNo5KGku3nxZV/np1LQrLDK+UezS/FH9k5BssA+ifBM1aalUC5q48hAER/pjQT9xK0yk3P3SraiReTVcj35LiJvXSavOGTOuiK1ncf+4q/pNc97d+T1Qbyc5FRPaNNSN2oLW3EkO7tNb7TX3m8E42jsi2Pk+pW8E598o1DErcgglfpBrd598rDxktY2zFYHPqRQRB0DsHSdq5q1iZdh5zfz0s+ry2bjYxNzHZfKRQ4xjiD2LsmjdcrdlecOSP8yi5Vo2/T16CytbfZpwca0YcQFP/IHx/YzZGdAtC3If6144xhzU+S2b9dBB/HDZeKyOWKfd1s/4O+HkqySR65Dzu/vgv5BVXIvH+SEwcEC53OE6DNSMOJqC5h9whWMW7649KfkzjfUbEH9MaiYjsbHi3tsW06pl5Jdh7s89BvfKqGmw/fhHVFgznltO1GzWocdDYHU19f531JjQFk3SYjDiYFp5NszKr4Zo2UqhvpnGEb8amVGAYa3YSI+uCxkgjG/Z4XKTR98laNX73fLITDy7Vbup74tt9mPLNXny42fH6XpVcq0aP1zZJXnNoLkEQsO/sFZRUWrczMzkXJiMO4otH+qJTYAssmdRX7lAcRn0zTVMYXQDc6uQqip77/QNLjPfLsYY9DWosbEEQgL1n6877y37HWwwx9fRlAMCpixV6y9jyb3xt+gVM+CIV8R//ZbuTUpPXNL9mN0GjewVjdK9gucNwKIJQ9y0u84Lu+UYUzrBsm6DzRxLJ0nlGfk07D6WbC8ZGGx4hZC+/I0Mdqusn4Tt/VfzK2ET6sGaEmqz3Nh7DwHe34GBOsSznr6lV4VhBKRYnH8fwD1J0ri9zIOcqqmpqm0ztjRi6bvCaW9YdviDJmjxyN9NdLq/Cv1cewv/9dNBh+6wQWRuTEWrSihoMFS2prEZKdhFqalVWv0n9e+UhjP7wL3y05QTOXKrA0h11c65o3nDv/3wX/m+F48wyqfmNWVf+VFVTi9Ef7sCMmwv8WWLWioOY+NVui45hDzRnMxU73HxDRj6e/jHN6Dms+bcsCAKTKLI6JiPkVB78IhWPfbsPnV7egC+2n7Lqudaka88iqu9G9OeRQpwqMm8xN6NT45t5k7pYVoXRH+7At3+fEbXfO+uO4lhBGTZkFuCkjvdkLJyGrx8rKBN1/qZmxvIDVjt2VU0tZvyYhqS9OY1e02zCfPKHNPR5K5kdVsmqmIyQU8kuvHVz+3DzCRkj0Xa8yMSbrsbd+o/D+ej+6kZsyirQKnLucgWeWLYP+89eMbsTwkdbjuNYQRne/N+RBqc3nE40WjywAUdtjdI18mdTVgE+Tzlps3VrpPbzvlxsyCzAi6syDJZLPlKIsqqaRn9nRFJiMuKAHu4vbqpxsg+7Tl3Csz+n44aOKd/NuZ/NXHEAN2pVeOoH7Wr8p388gK3HivAPE2ay1ed6deMYq2pqtecJMRKzh5vhj5etx4rMCc0iUrZmPPVDGt7fmI19Z69KeFTbKWVNB9kRJiMOaP59kdg453a5wyCRMvNKsfpgns7XTP52bUKxvKu6V/bVewod3/p13bRve2+b8ZNrcDOy6vQ7f0g/0Z0umm+v4SWQYsrvi3Y6hb3kbl4qff1TBEHAy6szsHyP4doxfX7Zn4uhC7fh1EXzmizJsTEZcUCuLgp0C/aROwyncr26Vu+aNFKTo9q/vKoGqacuq2/Oum44Ym+6rjYexpJXXInfDzVe7Vff5czMK0H0m3/im53i+sXITRAEPP/LIbyl0YRmDy1FO05cwvI9OXjZzMUU5/16GOcuX0PCb4abjZqqjZn52GGFyR8dBecZITKiViWg1+ubJJ0B1ZAxH1k2mZRWlCbmAw8tTUXWhVK8PrYHHh8S4ZBzsAxZsFXn9qwLpTq3v7jqMMqqavDWuiOYOCAcZVXVCPT21FnWnvqFnLlUgd8O1E3e1qOND4qv3cBdPaSdg0jXPCPGFnOUqtmnyglH7hSVXsfTP9Z1Vj67IF7maOTBmhEiI0oqq1GjEqyy8F49zXudpCNITIlZENQ37PpmJCkqNaSqGKmqsX6N1ID5mzFg/hYUll63+rmuV9fqbR7S3KqvJqpGY99/rzyEd/44ijOX9c/OSvbvyjXL5tOprlXhnXVHsC3b9v2wpMJkhMiIPm8nyx3CLY5XYWExo8OXzaSZAJbdnAuk4QJ7lh1fwLnLFTiaf6tm5mpFNbq9uhEPfWm8c/GGTN0LtemqpLFFZ1Q7qhyymp/3NR7m7Ah+3peLr3eewePf7pM7FLMxGXFgz4zoLHcIZIcqNCbZMpckNSOWH0Jytozp/U3ZGLowRV39DgDJR+qGx5oyAkfMzd+earLsxeXyKixOPo7cK7o7dDd0/uo1vCBTfxXNZlFzmgTrVxp2ZExGHFi4fzO5QyCJlFmYQGTmleBYQSl+TTtvUXPSrc9B43em+qKHcovx8uoMXGkwdbtUX6SzLpRi7Cc7sfPEJYmOWEfXZ76pN2RTRnwsSRE3qd7FsipJOjDuOnUJqacui97P3ms+Vh88j10nTf8bmPNzOj7acgITTBzi3nC2ZltqaomgOdiB1YH5ebnLHQLZmp4bxj2f7JT0NGI+HMd99jcAWG2Gzie+3Yeyqho88t89FnfuM/d+23DSs3slvt4AMGzhNlTcMN4/xlhH0ieW7QcAHHt7NDzdXc2KRdYOzDqyouyCMjz78yEApnfwrE/ICkzsB/T0D8an3TfFq2sycaygFD9NH2R0eLsuguCcyQlrRhzYnd0CMWlguNxhkIPQu76IxiffpfK6b4ean4W/7MvFtO/2Gz2+runfG53KwGuVN2rxy/7cRtstrTXSx5KKAFOSBl3Sc3WvIG3JMQHdtRpVOiau0+efX+1Wj9DRew6xQUnoQon1myHE1oycuVSBB5bswpajhVrbf9h9DvvOXsUuEbVTmv8vzLnOlRb87dgLJiMOzMVFgfn3RcodBjkIU2pP8kuuN+pzMu+3w9jc4APXGuavP4Jt2fY5z4KhdnyVIODBpamYYcKCdsZu+Kaes1FZndtM33/Xqcs2m4TOKImqBTQP8/raTPx1Qtq/rTk/pyPt3FVM1ZOo14r4/Vn6ls8aGE2VmVeC19Zm4nJ542SrpLLaah3ExWIyQuRIrFB9eyi3WOv5+auVJn04lhupsRCEug9CzW9thj72ko9YP+ExxpzmiRNF5dh75go2ZMq3dostavXr76322IKw+UihwZFQ36Wew6P/3SvpOa826CN1vboW5/XMfiyG1HPa3PPJTnyfeg6vrNGejO781WuIfvNPjPtM+iZHczAZaYKae5jXTkxNz96z4oeqjvtsJy6XmzbvQZdXNuh9bW16Hu75ZKfWMFZ7mjxMMnb6ngQBKLtejVfXZGKfyL8DMbUqctD8O7pQXIlp3+/Hg0u1O6pau99LToNROnct3iF6yYRbNEbTWBCTIdkN5i/akFGXPGfm6Z4U0NaYjBCRluvVKpO/5eta9K9eff+Pw+dv9ZPQtfqtLWlOJGaNxGhtuu61h3TJL6nE+xuP4YLIYZm6wtb3Tj7YlI0fdp8zeUSJIdW1KqzYk9PoJmxrxwpK0X/+ZvUK0Y/+d4+s8dSzxXX5bNtJ/JZmelOfI2EyQmQF9tIOa0uGZo4tNmGGSVO+yapUQqMk4sylChzIkWblXHNyJc1oZielm7zf49/uw+cpp/TeTPX9BWXk6e8E23D/05f09yUQmxh+ueM0XlqdIe0MwboYSRJf+C0Dl8pv4NWbzQ6nLpo++6y11pcyNNT7hV8PY/4fRyAIAkquGR9xpu/tHysoxcJN2Xh+5SHzgrz5687MK0FBifVnGhaLQ3uJrOCnvVaaydFBc5zebyXj7xfvtPg4d3/8F1o288BPTw5Sbxv+QYrFx5VD/U1d3830WL7um/68Xw832pZ6SvwcLGJrhuScWEtrtIkFNVrdXt2Iz/7ZB/FRIRbF89m2k1rP9Q0LPne5Aj/frCE8c+kaNh8txNqZQxAd5qdVTntlad3vr2Ezi9b+JsQM1I14q+/I/tLd3UzcyzZE1YwkJiaif//+8Pb2RmBgIMaPH4/s7Gyj+61cuRLdunWDp6cnIiMjsX79erMDpsY2zrld67ncVeEEp159U5/kLMs7eB4rKEPqafETepmq/n9OrUrAwZyrBpuh6lWZUMYcP+sY5qzPT3sbl7Vl/5wLxZVYuMn4vcBcUr6TmSsOGC9kRMP3qm9NI83h9PUj0pbtOtuonOYn9td/3VpFeknKKWzIqFsWwFCtm77P/MPni7XOkd6gs7o9EZWMbN++HTNnzsTu3buRnJyM6upq3HXXXaio0F9NtmvXLkycOBFTp07FwYMHMX78eIwfPx6ZmeYtM02NdQv2kTsEamCniJkixbDWnBvW0PAb/5r0C1rrtFjCkhutvkXqNC36Mxv3fb4Lc3+tqxI3lOB/ueO01vOSymr829yqdIk1vEymvHdzPPbtXun6TBj5MqXZB8ng34EDfCe7dqMG244V4YZG0rJwUzYqb9Qi7dxVvLfxGGYsN5486bsO9376t/pne/+SKioZ2bhxIx577DH07NkT0dHRWLZsGXJycpCWpn98/UcffYTRo0dj7ty56N69O95++2306dMHn376qcXBE9mra01gEiKp6ftW9tPeHCz7+4zO1/Sx5Et/h5fW472NxwyW+WJ73VTua9Mv3Dyf6SdcuOkYfrVSJ8NrN8xPRr/+6zR6v/UnjhUYTwjFjkQ5XqjdZyLnsm06uW45qr1KrWafEL2T/NmRZ35Kx+PL9uGN37O0ttcKgt5Vm+v9589sPPrfPZK8TymGJFvKog6sJSV1Gaq/v7/eMqmpqRg5cqTWtri4OKSm6u/dXVVVhdLSUq0Hma65kkN7yTFU1dQiYVUG3vjfkUZr2xhSY+E3fH3rxtR/eTR09MQNhicHy70ivm+FqR0re7y2yeRjVjY45jt/HEXp9Rp1x08pvinr68dwx8JtOGOg86whlQ0SrrLr1fjfoQs6y/55RLvp783/HVH/bCx/FATBpJqiazdq8Pwvh7DZwDw4pddNTxI1r3p9083u042HXhv79Xy89ST+OnHJYFymMn9IsnTMTkZUKhXmzJmDIUOGoFevXnrLFRQUICgoSGtbUFAQCgr0tx8nJibC19dX/QgLCzM3TKfx+6wh6p9nj+iCmHA/+YIhMpHmqKMbIr7hPfK15cM5Dd2sDL22dPtp/S/CvP4Nl3TMjmmp297bhrLr+kdvGKrtMXWekbgPd2DrMd03w91m9u2pqdU+94D5W7DqgO4h0ycaLEGw0sR+Nu+sO4KIhPXo8NJ6g4ng2vQ89HhtE347cB7Tvje+JIK1PPat/gnbxPy/sed5fsxORmbOnInMzEwkJSVJGQ8AICEhASUlJepHbq7pHbmcVVSon/pn/+buWP2vIfoLE9kJcyemMmcyN2d06Lxpw4AtUd+UZS0Na3g0HcwpNuuYX++81SxoqEO0mKHaJjPxT16zWIoZyyQ0rPW5Xl0r64goY8wa2jtr1iysW7cOO3bsQGhoqMGywcHBKCzUzpwLCwsRHBysdx+lUgmlUmlOaKShrZ+XXf/xEcnZp85as4za87dPOWRdKMGx/DLc36etaU1DFvxNNKUrb2kz2nO/pGs9P3+1Eh9uPmHRMa1JVM2IIAiYNWsWVq9eja1btyIiIsLoPrGxsdiyZYvWtuTkZMTGxoqLlER7fEh7uUMgspqqmlos2GC4I6pYr/+ehad+sKw6Pu2c+AnYfkszfeZWS+07exX/3Smuw7Al4j/eiedXHsJ2Jxvubss8W1fissZIjZW95cyikpGZM2fixx9/xIoVK+Dt7Y2CggIUFBSgsvLWt+/JkycjISFB/Xz27NnYuHEjFi1ahGPHjuGNN97A/v37MWvWLOneBenk4cYJdsm+WfLl7787z6hHvZhD14dxYWkVNmVZ1iHQnJFUizcft+icYr297gj2GFhYTgoNf7WGJu0ytJ+1HTx3Ffd+Kv1icZbMwiwIAqab2UdldtJBbDtWZLygnRF1t1qyZAlKSkowbNgwhISEqB8///yzukxOTg7y8/PVzwcPHowVK1bgyy+/RHR0NH799VesWbPGYKdXkka7gOZyh0BkNe9vtN4kW87A1k24pt6aLWmeMGfPj7ee1Jq7RCor9+se3n32UgX2GOnca8mU+2vTL+DxZfuMlrO3aUdE9RkxpS00JSWl0bYJEyZgwoQJYk5FErijcyu5QyCyWw1HYuhjj+t4SEGqe5Gp1f22aBawp5aH05d0/30dyCnGQ1/uxubnhurdt+GIImfAevwmyOVmyqtQKDCye5CR0kTyMTaxkz0YlLjF5FWMHYlUM3L+rmcOEDl9ssV6HTXFrbKs/xqfKLTygoNGOHSfEbJvEweEoWcbHwzrGqix1c7+4og0jNOYrtqe6VpPxNEZSkVOFZk3YZkh1hq9pMuiZOv1wTFlenbA/GHrAEyaJbep4aq9TUji/VGNttlb9kuk6bKIWVfJdiZ+tRufTIxBewv6nVXcqMW2bNM6UpqyIKG9yMozrX9JXfKl/wO43MA6U5ozyToLJiNNXL/2/tjigD2rici6jH1P+b+fDlp0/LfXad9QDfW90Zzu3dLGo6wL1pvoraqm1uSlCIx9EZz762Gz4xjz0V9m72uv2EzTxE29LQJvj+fIJSLSZuvBFN+nntO5ffvxi1r9TkztWKxLrUrAP5boX/fMUrN/Sje5bF2XHOtcZalWv7YnTEaaOA83Fzw6qJ3O1zLeuAtvjO1h44iIyB7INbSzpLIa93zyF5aknIJKJWDKN3slnRDN0PTxltqYZXpH5uOF5RatsuxsmIw4MW9Pdzw2xPgsukTU9FjSwdIS3/59Bpl5pXhv4zG77l4vRT+Wj7eclCAS4+xsyhCzMBkhIiKbqdK4yc9deUjGSAyTYobac5elH5XUkCAA2TIPE5YCkxEiIrIZzY6dqw7abk0eOdii5ufjrSdw/qr42XTtrVaKo2mIiMgm7v/8bxzIKZY7DJs5aUFnXFOdvmhe7ctSC9Z1sgbWjBAROSE5OrA6UyJi765eq9Z6bsnCflJgMkJITbhT7hCIyMYsWYyNmp4lKbbpbKsPkxEnE9nWFwAQHxmi3hbk7SlXOEREZAc++NN6U+ibgn1GnMzgjgFY9nh/+Df3UG+zt6WkiYjIuTAZcUIBLZRyh0BERKTGZhoiIiKSFZMRJ2NvY8uJiIiYjDgZwdhSkjctfijaypEQERHVYTJCOt3ZLUjuEIiIyEkwGSEoFAqsmDZQ/fypoR3g6+UuY0RERORMOJqGAACDO7XC8XfGICOvBL3D/AAAHm4ukqxcSUREZAhrRpyMoS4jHm4u6NuuJVxd6iYeWfd/t9koKiIicmZMRkivLkHecodAREROgMmIk5gxrCNaeyvx5NAOcodCRESkhX1GnMQLo7thXlxXKDj3OxER2RnWjDgRJiJERGSPmIwQERGRrJiMEBERkayYjBARERGyC8pkOzeTEbLI15P7cbZWIqIm4FhBqWznZjJCJhvTKxifTIzR2jayRxDSXxtldN//TulnrbCIiMjBMRkhg5ZPG4hgH098+1h/LHmkL8ZGt2lUxpRROmH+zawRHhERNQGcZ4QMGtKpFXa/NELna92COUMrERFZjjUjZDYXHTUiXTmFPBGRQzK0dpm1iU5GduzYgbFjx6JNmzZQKBRYs2aNwfIpKSlQKBSNHgUFBebGTHbskdh2jbY183CVIRIiInIUopORiooKREdH47PPPhO1X3Z2NvLz89WPwMBAsacmR6Ajtd7272G2j4OIiByG6D4jY8aMwZgxY0SfKDAwEH5+fqL3I8cX5OMpdwhERGTHbNZnpHfv3ggJCcGoUaPw999/GyxbVVWF0tJSrQc5tiAfT4zr3XgkDhER2QcB8nUasXoyEhISgi+++AK//fYbfvvtN4SFhWHYsGE4cOCA3n0SExPh6+urfoSFhVk7TLKBOSO7yB0CERHZIasP7e3atSu6du2qfj548GCcOnUKixcvxg8//KBzn4SEBDz33HPq56WlpUxI7JCHm7hcNqJVcytFQkREjkyWob0DBgzAyZMn9b6uVCrh4+Oj9SD78cGEaIT7N8PCf0SJ3te/uYcVIiIiIkcmSzKSnp6OkJAQOU5NEvhH31DsmDccnXXMKdKwxTHpyUFaz+Mj637v+uYjeebOTugd5idFmEREJIKc84yIbqYpLy/XqtU4c+YM0tPT4e/vj/DwcCQkJCAvLw/ff/89AODDDz9EREQEevbsievXr+Prr7/G1q1b8eeff0r3LshuDeoQoPX85fju6B3mh2FdW6PvO5sblQ/08cQ/B4YjPbfYRhESEZHcRCcj+/fvx/Dhw9XP6/t2TJkyBcuWLUN+fj5ycnLUr9+4cQPPP/888vLy0KxZM0RFRWHz5s1axyDn4enuigf6hmptC/H1RH7JdQCNa1aIiMg2XF2MrzNmLaKTkWHDhkEwUJezbNkyrefz5s3DvHnzRAdGjkkBwMvdFZXVtSZ3WJ11Zye8vDoTABDorUT7AHZ0JSKytahQP9nOzbVpSBKTY9uhXUAz3NcnFGtmDsH43m3w7WP9Td5/6aN98a9hHXFXjyB0DfbGbzNi1a9NGhgOAHj3vkjJ4yYiojry1Ytw1V6SyFvjekEQBCgUCnQN9saHD8eI2j+uZzDiegarn/dt56/+OT4yBK+P7QkPNxe8tDpDspiJiMg+MBkhySh0rOJriccGt8eJojIM7BAga1smEZEzkPgjXBQmI2S33ri3p9whEBGRDbDPCDV5Ib5cqI+IyBiFjL1GWDNCDmlKbDt0CfZWj8Kpd3L+GBw6X4yFm7Lx+tieaNvSC95KN6RkX8Tjy/bJFC0RERnCZIQckkKhaJTFvz2uJ9xcXdC3nT+SnoxtsIMNgyMiIlHYTEMOKbKtr9whEBE1KYKM004yGSHZ+Xq5m1z2z2fvQOL9kbgvpm2jnt+cvZWIyHxyrk3DZIRk8/4DUXiwXyjG9DJ90cQuQd6YOCAcLi4KDIzwN76DSHtfGiH5MYmIyDD2GSHZPNg/DA/2DzN7/w6tW5hc1tQuI4E+nji7IB4l16ox6b+7kZlXal5wRERkMtaMEOng28zd6DC3f96cpp6IqCmQs6mbyQg1GYbaO6WeHRYA/jlAOxk59vZoyc9BROQMmIwQ6WEsf9Gcov7IW3HwdHe1ckRERNYjyNiDlckIkQlevadHo20uGtmKpxsTESJybGymIbJDT97RAQBwV48g3B0Z3Oh1Vxn/99wTZfoIJCIiU3BoL5GZtv17mPpnQ1WMxnqMJD97B9JeGam17Z6oNvhr3nAseaSvzn28PG4NRrP2/+HT796t9VyzD0y/di2tfHYiIutiMkIOLaJVc0mO0znIGwEtlI22h/k3g6uLAl46+oO09fPCrOGd8MLoblr9RwDgkUHSjrRxaXD8kd0D1T93FDHE2VHFR7ImiMj62GeEyKY+mRiDh0XMceLXzANvjevZaPu/47pixrCOjbaP6tG4WUdTt2Bvk8/d0Btje+De6DZYPm0g0l4ZiUCfxkkUEZEjYTJCTqHhyJix0W2QcHd3PDqoHX6bEat7pwYmx7bH6J6Gk4wDr47CzheGY2iX1vhkYgw+nhiD/u1b4qvJ/Uw6xy9PGY8lMtQPCoUCQzq1QkALJR4b3F792sQB5k8iZy1h/l6WH4QLHRI1aZyBlZoM32b617jp2ebWwnrJz95RV97LHW+P7yVpDP7NPeDf3ANAXcIDAPfe/NcUbfw8G217YkhEo3NoUmo0IVljPhVLtA9ohpS5w9H+xT8sOk5MmB/+OJwvUVREpIucHViZjJDDe/+BKOw+cxljo/Tf9P2be2DvyyPg5e4Kb0/TF+azlp5tfHGsoMxouXuiQvDa2LphxUsf7YsrFTca9ZNx1UhAPGw8xMfVRYFalXU/wVwUwJTB7fHOH0eteh4iZyfn0F4mI+TwTF3jJtC7ca2DWD5e0vyXeW1sDwS08MC6QxdwoeS6zjJdg7zxycQY9fM4PU1EXh6ueGZEZ9yoUSGuZxCW7TorSYzjerfB2vQLBsto1sOM690GSjcX/LL/vMF97okKwaAOAXhlTabJsbi7uqC1txIXy6pM3oeIxGHNCJGDmDe6G85evmZx3wxfL3e8dHd3hLb0wmtrs3SWae2tNLnZ5blRXdQ/r5k5BG39vNB//maT41n1r8G4//NdWtsW/iPaaDKiSQGgVYMRSe46amo+/WcfqFQCqmpUcHNR4LNtJ1FkQpIh5wclEVkXkxEiEVq1UJrUydRUkwa2QzMPN/x75SHJjtk7zE/0Ph10DJH2cBPf5PPU0I74POWU+vlHD8foLOfiosDU2+r6wkwZ3B4qlYAOL603cnR5spEhnQLw98nLFh+nQ+vmOH2xQtQ+Hm4uuFGjsvjcRKYQOLSXyDm5uihwX0xb9XPNmoXQlhKMQjGRIAC/zxqi9/UB7f11bg/y0W768vW61R8ntKUXerTxMen8DedRAW4lSEO7tFbHWK9dQDOTjisFY6s3m2rRhGjR+8i5Vgg5n9CWtvt/1RCTESKZuboocODVUdj38kh4urtixfSB+EffUCSM6W7Rccf3NjyKJ0Vj9loAiAr1Q9cg3fOf6Osr8+3j/dU/S33bXDF9EF4c0w3/ebB3o+NbMk+LXDRHdJkqTMabA8nHx9P2jRa3d26FFkr5GkuYjBDZAf/mHmjtXVcrMrhjK3wwIdrgUGVTGEsOmnmIWdxPd+1AFz3JixSCfT3x9NCOaNlgKDMAvHtfJCbHttO5X1SoL1b/a7DOWXMdzaIHxdemkOM7/Eac1vxBttBaxwzUtsRkhKiJEvPhoi9x6R3mB093FwzqoLuZRpOyQR8TS1oYdE3/rtlkEdBCibfG6Z4jxsPVBTHhLdGrrWlNRLrsThiBkd0D8ePUgaL2+/LRvnjtnh749rH+jV4zZwqYNn7SN9WF+9tHbcvtnVvJHQIAYFSPIFnPr2tmZwDoHNT0l3nQxGSEqIl6ZmRnjOkVrLNzKgCdlR1xveqGD9f3V1k1YzAOvx6HZh76q2/fvLcnugZ54/m7ulocc73xGv1o6pk6nUlUqB8AYNGE3hjQ3h/fPHZr9ltTq7+DfT3x9ZT+uE3EDfP+mLa4q2cwnrgtAsO7BWq9tq1Bk5iprNFlxJSJ/sRM1Gcr3lZqQvAUUYP23ymmzaQsRkBz29VI2POimkxGiJooH093LHmkL/458Naifa/Ed0eIr6fezpSzhnfCp/+MwZqZdZ1ZXVwURkfVTBncHpuevaNRZ1axNIdLt9VRIyC2M2d4QDP88nQs7ux265uvNWeoXWigg6q5CzpaY3RDWx2z/DbU1QZ9clQif58Jd1vWh0oKI7pLX4ui73csRcfphrVgzWXsE2KM/UZGRJJ4ZFA7ZOaV4LbOrfGPvqGYdnsHAEBR2a3J1upv9B5uLrhHx0y2mvfwIB8lCkuln3zs3fsiEduxFaprVDpH4dj7uJKGKzcHeitNmj/F4DF1JE/dQ3ygAHAkv9SiY9tKv3Ytsf/c1UbbnXGg0Cvx3fHzvlycKCpXb7PmdWjZzB0tlD7qvxVx/cRsizUjRDZmrepmfTzdXfHhwzH4R9/QRtvrmTOniBjz76trGjDURKBQKHBvdBs80CBONRM/tHV906wfctzFwnb40JZe6qHGxsSE+1l0LgAI9PHEQ/3CMDDiVp+d9c/cBjdX8781d2hlH30RxNaMiK3U2vr8UBx8dZS4nUzQPcT8vkjTbu+A5OeGam1r+L5eiZewBkih0Dr+K/f0QPcQH7vpr6OJyQiRjSyfNhA9QnywfLq4TpHW4uPpjvceiMR7D0QaXa/H0grjSQPbIfPNODw6SPcIGFO019HUMbyraYnB3pdHIOvNOHgZ6Ptiiu1zh2PZ4407p+pi6Tfe+k7D7/0jCt89MQBAXSdhhUKBiQPCDe1qkIuLAn4WjtSSgrVrRjq0bqFzJJY16JqRuVNgC70TEN6no09UPa+btRfGkq/3HojEy0aarhoeoq2fFzbMvh0P9tMxg7TMa2yKTkZ27NiBsWPHok2bNlAoFFizZo3RfVJSUtCnTx8olUp06tQJy5YtMyNUIsc2pFMrrJ99u7qDpT14qH84Hupv/o1Nl/pOchP6addwWDqHweeT+iA+MkTdnwWo66fRcFVjXZRurmiudDPr8/aBvnU3ju4hPnB1UUjS70TfcgIn5o9BasKdeH5UF3z2zz7q7Z7ursh6Mw6HXr8LAPBw/zDcZWQUSKsWjW/EuxNGAAAOvjoKD/QJxWv39NC5r5uOSejMMWlguN6b6rjejW/I80ZL1wnaWhq+naQnB2H++MhG5VwVCqyZOQT360g8NJtLGvYNuT9GT81gAw/2C8P0OzqYVNYRiE5GKioqEB0djc8++8yk8mfOnEF8fDyGDx+O9PR0zJkzB9OmTcOmTZtEB0tE8tCsOZlyc/6DYXpqJZY9MQDfPzEAs4Z3kjSGMP9m+GxSH61vm61aKNWrGlvL+N5t8fusIfhthnTLAGh29n1eY10hd1cXhPh64f9GdEZAg6HZzZVu6qY1hUKBqFDDk6hNaPDt18vdFcG+nur9Fz0YjSdu053IGZrhNtqE5Qay3xmNpCcH4fWxPTFST6fPh/uH4ecnB2HnC8MBACG+nph2Wwck3h+J32YMxoMNktmGNSktlG6NJu4D6ubs0RxBZW2DOgTonEHYEM0ETbNZceptEeqakb5GRr5I3RnbxYqdu00h+qvKmDFjMGbMGJPLf/HFF4iIiMCiRYsAAN27d8fOnTuxePFixMXFiT09EckgrmcQxka3Qd9wP0yObY9BHQLQQ0/beQulG+4wsV+FrT0+pD22H7+IoV1aY/vxiybtU3fj97NaTP83ojMEaE+lbw1i7jUNV4juHeaH9NziuuOYsL/SzRWDOgQAqLvBhvs3g0KhwNM/pgGomx3YxUWBgTfLZLxxF5RurvBwc1E3Qf2ZVaA+3scTY1B+vUbrHIIgNGq6e2F0Nzw9tINVR02ZytvAMHJTRspYc0LBhtr6eeH5u7oYL2hFVu8zkpqaipEjR2pti4uLQ2pqqt59qqqqUFpaqvUgIvm4ubrgk4kxeGxIBFxcFOgT3lLU/Ay2ZKgvwrCugdj14p34RsekZFIz9M22YYzPjOisrnEyldg+F2Juz5o38/+7sxMWPBAJ/+YeeMOMWig3VxeMiQzB6F7B+PXpWGyYfTsWP9Rbq4y3p7vBTtT3RrdplEw1HGb79vhemDGso0WJiGbTmKUMDaMd0ulWB1JTf49xPcUPKzb1Uux8YThCfG23FpYuVk9GCgoKEBSkfRGDgoJQWlqKyspKnfskJibC19dX/QgLs2y5diKiem38vODqolB3CrWWJ26LwNvje2HL80ONF7aB5dMHmbXf6F7B6Bbsg7RXRuKxIRFmzSRbr197/7qhySYcxNg9un6EVr0wEQtL6uu/FB8Vggn6RnOZS9fkgnoSi0Bv7aY5D9dbt+gvHumL50ZJX3vxYL9Qu6hJssvRNAkJCSgpKVE/cnNz5Q6JiOyYOf0th3ZpjV+fvtUPZMH9jTshmirMv/GN0N3VBY8OaoeOrVvcjPFWkD5WbpJp6Pg7Y/SO7DBV/Q3rqTs6am3/8KHemDGso3qtoOm3G+9QLIX6fkw/TR+E1+7pYXTIdf1aL6N6BOGHqQPQsXVznSOj+rW3/iylmjd/zb4/DWvHNj17h9Y+YtdbMuW/xXsPRIk6prVYfcKD4OBgFBYWam0rLCyEj48PvLx0Z7JKpRJKpbyL9hCR40h+bihGLNpu0THMGWnaI8QHM4Z1RGzHAKNlXV0U+Pbx/rh+oxY3alVmnE2bmC+zUs4jM7qXdn+S+qn7a1UCHuwXZtE8HJpMfXuxHQNMuv6TBoZjcmw7hPs3g5urC7Y8PwwA8NuBPK1y43q3xW8H8tA7zA9HLpRi58lLeo+pmQRHh/nh0M1+NaZY/a/BqKiqRbuA5tj5wnC4u7o0avqMaNUcXzzSx+ByDJayh1oRwAY1I7GxsdiyZYvWtuTkZMTGStcznYicW33tg625u7lgbHQbtDJxUcLhXQMxRscigJba9eKdkh+znq7OlkE+jd+vq4sCvdr6NpqJ1lwNk0NzO0XPGt4JD/cPQ6fAFujQugXcXA3f9jzdXfHLU7F46e7uaN+q8agifUnrNyLXrYkJb6le+yi0ZTO9yymM7hVitx3CpSQ6GSkvL0d6ejrS09MB1A3dTU9PR05ODoC6JpbJkyeryz/99NM4ffo05s2bh2PHjuHzzz/HL7/8gmeffVaad0BEpEHsGjb1xDRj1H+ZHGBmlb7UE3618fPSu/prU6FrvSJT/DuuKxY8EGX1GoCGQ7HlYmhCNXsmOhnZv38/YmJiEBMTAwB47rnnEBMTg9deew0AkJ+fr05MACAiIgJ//PEHkpOTER0djUWLFuHrr7/msF4isivdQ3yw+l+DTapl2Pb8MLwS3x3PjZJvkq5+7f21nk+ObY/vNTrlPty/ruO/pd+q5arFb9iZ01rcDdTkaNYKLZ828Oa2WyyZCdcc9XP76BoG/kp8d6ydOURrpJKiwXTw9kx0Q9SwYcMMfvPQNbvqsGHDcPDgQbGnIiISzZIp32PCTavpaN+quXrBQbkM6hCAH6YOQPuAW3NtBGjMutouoDmy3oyz68XRDHk0th1OFJZjeLdAq57n33Fdsf/cVaNLFWgOx633qp4ZbDVJsfpuvc5B3tgxd7jW73nB/ZG4VF6l9+8xrGUzZObZ//QYXLWXiJqEt8f1xJr0C5gxtKPxwk3E7Z0N13qYs2T8m/dqN/fI9c1a6eaK9/5h/ZEebfy8sGPecLP2NTTXjtJKi0+GN5gd92EDtTMKAG+N6wU3VxdMGmjbWhyx7HJoLxGRWI/GtsdvMwbD1w4WgZOLFN/CGw4vVbrZT82KPSzw99LNxemm6plKv561lykwVWtvJT6ZGKOeEddesWaEiMjG7LUd/58a356fH9UFV69VI0LHasly+XHqQLzxexZeGNPNJufrFtJ4SvbbOrdC5ptxRhd+DG2pf30fW7HXvzNdmIwQEdmY1KNppLDg/kg81P/WbNf/N6KzjNHo1qutL36dMdhm53u4fzgqb9Q2qlWwdAVqW9G3fpRmPyN74RhXlIjICtqKmEK8qWvZ3MNuJsCyF64uCos7KstxSf945jasz8jHjGG6V86ODPXFRw/3tovam3pMRojIaYX4emHF9IHw8ZS/L4JcnhnRGYdyizHCyqNWnMlgE2aEtaaebXzRs42vwTLjetvXfCRMRojIqQ3u2HjIpqMK1bFGjjHWWHzNWd0b3Qa/H7qAGcNujegK1jOzKmljMkJE1ET4eLpj5wvDJV2Lhkz30cO98frYHlqzsc4Y1hF5xZW42wrLADQlTEaIiJoQe+oH4GwUCkWjaeGbK920ZkUl3Zg+ExHZWP3oDF3TejuC4V3r+pewCYKkwpoRIiIbC/b1xL6XR8Lb0zE/gl+9pwd6tvHBqB7BcodCTYRj/k8gInJwrW20EJw1NFe64dHY9nKHQU0Im2mIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVg6xaq8gCACA0tJSmSMhIiIiU9Xft+vv4/o4RDJSVlYGAAgLC5M5EiIiIhKrrKwMvr6+el9XCMbSFTugUqlw4cIFeHt7Q6FQSHbc0tJShIWFITc3Fz4+PpId19nxuloHr6t18LpKj9fUOhzxugqCgLKyMrRp0wYuLvp7hjhEzYiLiwtCQ0OtdnwfHx+H+cU6El5X6+B1tQ5eV+nxmlqHo11XQzUi9diBlYiIiGTFZISIiIhk5dTJiFKpxOuvvw6lUil3KE0Kr6t18LpaB6+r9HhNraMpX1eH6MBKRERETZdT14wQERGR/JiMEBERkayYjBAREZGsmIwQERGRrJw6Gfnss8/Qvn17eHp6YuDAgdi7d6/cIdmNxMRE9O/fH97e3ggMDMT48eORnZ2tVeb69euYOXMmAgIC0KJFCzzwwAMoLCzUKpOTk4P4+Hg0a9YMgYGBmDt3LmpqarTKpKSkoE+fPlAqlejUqROWLVtm7bdnFxYsWACFQoE5c+aot/GamicvLw+PPPIIAgIC4OXlhcjISOzfv1/9uiAIeO211xASEgIvLy+MHDkSJ06c0DrGlStXMGnSJPj4+MDPzw9Tp05FeXm5VpnDhw/j9ttvh6enJ8LCwvD+++/b5P3Joba2Fq+++ioiIiLg5eWFjh074u2339ZaY4TX1bgdO3Zg7NixaNOmDRQKBdasWaP1ui2v4cqVK9GtWzd4enoiMjIS69evl/z9mk1wUklJSYKHh4fwzTffCFlZWcL06dMFPz8/obCwUO7Q7EJcXJzw7bffCpmZmUJ6erpw9913C+Hh4UJ5ebm6zNNPPy2EhYUJW7ZsEfbv3y8MGjRIGDx4sPr1mpoaoVevXsLIkSOFgwcPCuvXrxdatWolJCQkqMucPn1aaNasmfDcc88JR44cET755BPB1dVV2Lhxo03fr63t3btXaN++vRAVFSXMnj1bvZ3XVLwrV64I7dq1Ex577DFhz549wunTp4VNmzYJJ0+eVJdZsGCB4OvrK6xZs0Y4dOiQcO+99woRERFCZWWluszo0aOF6OhoYffu3cJff/0ldOrUSZg4caL69ZKSEiEoKEiYNGmSkJmZKfz000+Cl5eXsHTpUpu+X1uZP3++EBAQIKxbt044c+aMsHLlSqFFixbCRx99pC7D62rc+vXrhZdffllYtWqVAEBYvXq11uu2uoZ///234OrqKrz//vvCkSNHhFdeeUVwd3cXMjIyrH4NTOG0yciAAQOEmTNnqp/X1tYKbdq0ERITE2WMyn4VFRUJAITt27cLgiAIxcXFgru7u7By5Up1maNHjwoAhNTUVEEQ6v4Turi4CAUFBeoyS5YsEXx8fISqqipBEARh3rx5Qs+ePbXO9dBDDwlxcXHWfkuyKSsrEzp37iwkJycLQ4cOVScjvKbmeeGFF4TbbrtN7+sqlUoIDg4WFi5cqN5WXFwsKJVK4aeffhIEQRCOHDkiABD27dunLrNhwwZBoVAIeXl5giAIwueffy60bNlSfZ3rz921a1ep35JdiI+PF5544gmtbffff78wadIkQRB4Xc3RMBmx5TV88MEHhfj4eK14Bg4cKDz11FOSvkdzOWUzzY0bN5CWloaRI0eqt7m4uGDkyJFITU2VMTL7VVJSAgDw9/cHAKSlpaG6ulrrGnbr1g3h4eHqa5iamorIyEgEBQWpy8TFxaG0tBRZWVnqMprHqC/TlH8PM2fORHx8fKP3zWtqnt9//x39+vXDhAkTEBgYiJiYGHz11Vfq18+cOYOCggKta+Lr64uBAwdqXVc/Pz/069dPXWbkyJFwcXHBnj171GXuuOMOeHh4qMvExcUhOzsbV69etfbbtLnBgwdjy5YtOH78OADg0KFD2LlzJ8aMGQOA11UKtryG9v654JTJyKVLl1BbW6v1gQ4AQUFBKCgokCkq+6VSqTBnzhwMGTIEvXr1AgAUFBTAw8MDfn5+WmU1r2FBQYHOa1z/mqEypaWlqKystMbbkVVSUhIOHDiAxMTERq/xmprn9OnTWLJkCTp37oxNmzZhxowZeOaZZ/Ddd98BuHVdDP1/LygoQGBgoNbrbm5u8Pf3F3Xtm5IXX3wRDz/8MLp16wZ3d3fExMRgzpw5mDRpEgBeVynY8hrqK2Mv19ghVu0lec2cOROZmZnYuXOn3KE4tNzcXMyePRvJycnw9PSUO5wmQ6VSoV+/fnj33XcBADExMcjMzMQXX3yBKVOmyByd4/rll1+wfPlyrFixAj179kR6ejrmzJmDNm3a8LqS5JyyZqRVq1ZwdXVtNEqhsLAQwcHBMkVln2bNmoV169Zh27ZtCA0NVW8PDg7GjRs3UFxcrFVe8xoGBwfrvMb1rxkq4+PjAy8vL6nfjqzS0tJQVFSEPn36wM3NDW5ubti+fTs+/vhjuLm5ISgoiNfUDCEhIejRo4fWtu7duyMnJwfAreti6P97cHAwioqKtF6vqanBlStXRF37pmTu3Lnq2pHIyEg8+uijePbZZ9W1eryulrPlNdRXxl6usVMmIx4eHujbty+2bNmi3qZSqbBlyxbExsbKGJn9EAQBs2bNwurVq7F161ZERERovd63b1+4u7trXcPs7Gzk5OSor2FsbCwyMjK0/iMlJyfDx8dHffOIjY3VOkZ9mab4exgxYgQyMjKQnp6ufvTr1w+TJk1S/8xrKt6QIUMaDTs/fvw42rVrBwCIiIhAcHCw1jUpLS3Fnj17tK5rcXEx0tLS1GW2bt0KlUqFgQMHqsvs2LED1dXV6jLJycno2rUrWrZsabX3J5dr167BxUX7FuHq6gqVSgWA11UKtryGdv+5IHcPWrkkJSUJSqVSWLZsmXDkyBHhySefFPz8/LRGKTizGTNmCL6+vkJKSoqQn5+vfly7dk1d5umnnxbCw8OFrVu3Cvv37xdiY2OF2NhY9ev1w1DvuusuIT09Xdi4caPQunVrncNQ586dKxw9elT47LPPmvQw1IY0R9MIAq+pOfbu3Su4ubkJ8+fPF06cOCEsX75caNasmfDjjz+qyyxYsEDw8/MT1q5dKxw+fFgYN26czuGTMTExwp49e4SdO3cKnTt31ho+WVxcLAQFBQmPPvqokJmZKSQlJQnNmjVrMkNQG5oyZYrQtm1b9dDeVatWCa1atRLmzZunLsPralxZWZlw8OBB4eDBgwIA4T//+Y9w8OBB4dy5c4Ig2O4a/v3334Kbm5vwwQcfCEePHhVef/11Du21F5988okQHh4ueHh4CAMGDBB2794td0h2A4DOx7fffqsuU1lZKfzrX/8SWrZsKTRr1ky47777hPz8fK3jnD17VhgzZozg5eUltGrVSnj++eeF6upqrTLbtm0TevfuLXh4eAgdOnTQOkdT1zAZ4TU1z//+9z+hV69eglKpFLp16yZ8+eWXWq+rVCrh1VdfFYKCggSlUimMGDFCyM7O1ipz+fJlYeLEiUKLFi0EHx8f4fHHHxfKysq0yhw6dEi47bbbBKVSKbRt21ZYsGCB1d+bXEpLS4XZs2cL4eHhgqenp9ChQwfh5Zdf1ho+yutq3LZt23R+lk6ZMkUQBNtew19++UXo0qWL4OHhIfTs2VP4448/rPa+xVIIgsZ0ekREREQ25pR9RoiIiMh+MBkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIln9P70kCre2LNr2AAAAAElFTkSuQmCC",
1001 | "text/plain": [
1002 | ""
1003 | ]
1004 | },
1005 | "metadata": {},
1006 | "output_type": "display_data"
1007 | }
1008 | ],
1009 | "source": [
1010 | "import matplotlib.pyplot as plt\n",
1011 | "import pandas as pd\n",
1012 | "\n",
1013 | "data = pd.read_csv(\"loss.txt\")\n",
1014 | "plt.plot(data)\n",
1015 | "plt.show()"
1016 | ]
1017 | },
1018 | {
1019 | "cell_type": "markdown",
1020 | "id": "9809bc9f-4ff6-46c3-9c43-08c6c2694a82",
1021 | "metadata": {},
1022 | "source": [
1023 | "## Generate text with fine-tuned model\n",
1024 | "\n",
1025 | "Again we check results with our test dataset (5 rows).
\n",
1026 | "As you can see below, it can output the completion very well, because it's fine-tuned."
1027 | ]
1028 | },
1029 | {
1030 | "cell_type": "code",
1031 | "execution_count": 25,
1032 | "id": "29903cae-404e-4209-9c84-6c8a69609c13",
1033 | "metadata": {},
1034 | "outputs": [
1035 | {
1036 | "name": "stdout",
1037 | "output_type": "stream",
1038 | "text": [
1039 | "********** input **********\n",
1040 | "name : The Vaults | Type : pub | food : Italian | price : less than £ 20 | customer rating : low | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n",
1041 | "\n",
1042 | "********** result **********\n",
1043 | "name : The Vaults | Type : pub | food : Italian | price : less than £ 20 | customer rating : low | area : city centre | family friendly : no | near : Rainbow Vegetarian Café\n",
1044 | "The Vaults is a pub near the Rainbow Vegetarian Café in the city centre. It is not family friendly and has a low customer rating of less than\n",
1045 | "********** input **********\n",
1046 | "name : The Cricketers | Type : restaurant | customer rating : average | family friendly : yes | near : Café Sicilia\n",
1047 | "\n",
1048 | "********** result **********\n",
1049 | "name : The Cricketers | Type : restaurant | customer rating : average | family friendly : yes | near : Café Sicilia\n",
1050 | "The Cricketers is a restaurant near Café Sicilia. It is family friendly and has an average customer rating.<|endoftext|>\n",
1051 | "********** input **********\n",
1052 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : no | near : All Bar One\n",
1053 | "\n",
1054 | "********** result **********\n",
1055 | "name : The Cricketers | Type : restaurant | food : Chinese | price : cheap | customer rating : average | area : city centre | family friendly : no | near : All Bar One\n",
1056 | "The Cricketers is a restaurant located in the city centre near All Bar One. It is not family - friendly. It is located in the cheap\n",
1057 | "********** input **********\n",
1058 | "name : The Vaults | Type : pub | food : Japanese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
1059 | "\n",
1060 | "********** result **********\n",
1061 | "name : The Vaults | Type : pub | food : Japanese | price : cheap | customer rating : 5 out of 5 | area : city centre | family friendly : yes | near : Raja Indian Cuisine\n",
1062 | "The Vaults is a cheap, family friendly pub located in the city centre near Raja Indian Cuisine.<|endoftext|>\n",
1063 | "********** input **********\n",
1064 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n",
1065 | "\n",
1066 | "********** result **********\n",
1067 | "name : The Wrestlers | Type : pub | food : Italian | price : less than £ 20 | area : riverside | family friendly : no | near : Raja Indian Cuisine\n",
1068 | "The Wrestlers is a pub near Raja Indian Cuisine in riverside. It is not family friendly.<|endoftext|>\n"
1069 | ]
1070 | }
1071 | ],
1072 | "source": [
1073 | "test_data = pd.read_json(\"test_formatted.jsonl\", lines=True)\n",
1074 | "test_data = test_data[::2] # because it's duplicated\n",
1075 | "test_loader = DataLoader(\n",
1076 | " list(zip(test_data[\"context\"], [\"\"] * len(test_data[\"context\"]))),\n",
1077 | " batch_size=1,\n",
1078 | " shuffle=True,\n",
1079 | " collate_fn=collate_batch\n",
1080 | ")\n",
1081 | "\n",
1082 | "for i, (input, _, mask) in enumerate(test_loader):\n",
1083 | " if i == 5:\n",
1084 | " break\n",
1085 | " print(\"********** input **********\")\n",
1086 | " input_len = torch.sum(mask).cpu().numpy()\n",
1087 | " print(tokenizer.decode(input[0][:input_len]))\n",
1088 | " result_token, result_len = generate_text(\n",
1089 | " model,\n",
1090 | " input,\n",
1091 | " mask,\n",
1092 | " eos_id,\n",
1093 | " pred_sequence_length=30)\n",
1094 | " print(\"********** result **********\")\n",
1095 | " print(tokenizer.decode(result_token[0][:result_len]))"
1096 | ]
1097 | },
1098 | {
1099 | "cell_type": "code",
1100 | "execution_count": null,
1101 | "id": "6a7c1dd3-4057-497a-83ae-f99b1883697e",
1102 | "metadata": {},
1103 | "outputs": [],
1104 | "source": []
1105 | }
1106 | ],
1107 | "metadata": {
1108 | "kernelspec": {
1109 | "display_name": "Python 3 (ipykernel)",
1110 | "language": "python",
1111 | "name": "python3"
1112 | },
1113 | "language_info": {
1114 | "codemirror_mode": {
1115 | "name": "ipython",
1116 | "version": 3
1117 | },
1118 | "file_extension": ".py",
1119 | "mimetype": "text/x-python",
1120 | "name": "python",
1121 | "nbconvert_exporter": "python",
1122 | "pygments_lexer": "ipython3",
1123 | "version": "3.8.10"
1124 | }
1125 | },
1126 | "nbformat": 4,
1127 | "nbformat_minor": 5
1128 | }
1129 |
--------------------------------------------------------------------------------
/Readme.md:
--------------------------------------------------------------------------------
1 | # Fine-tuning LLM with LoRA (Low-Rank Adaptation)
2 |
3 | LoRA (Low-Rank Adaptation) is one of mostly used parameter-efficient fine-tuning (PEFT) methods today.
4 |
5 | This example shows you [LoRA (Low-Rank Adaptation)](https://arxiv.org/abs/2106.09685) implementation from scratch (manually) in a step-by-step manner (without ```PEFT``` package), and also shows you clear ideas behind this implementation in IPython notebook.
6 |
7 | This is also runnable in the mainstream hardware with small footprint - such as, a signle GPU of Tesla T4, consumer GPUs (NVIDIA RTX), etc - for you to try this code easily.
8 |
9 | | Example | Description |
10 | | -------------------------------------------------------------------- | ----------------------------------------------------------------------- |
11 | | [01-finetune-opt-with-lora.ipynb](01-finetune-opt-with-lora.ipynb) | Fine-tuning Meta's OPT-125M with LoRA
(Also, explaining LoRA method) |
12 | | [02-finetune-gpt2-with-lora.ipynb](02-finetune-gpt2-with-lora.ipynb) | Fine-tuning OpenAI's GPT-2 small (124M) with LoRA |
13 |
14 | Unlike examples in [official repository](https://github.com/microsoft/LoRA), here I download pre-trained models to focus on LoRA implementation.
15 |
16 | > Note : In this repository, Hugging Face API is used to download pre-trained models and I then apply regular PyTorch training loop for fine-tuning. (I don't use blackboxed ```Trainer``` class in Hugging Face API.)
17 |
18 | ## 1. Set-up and Install
19 |
20 | To run this example, please install prerequisite's software and setup your environment as follows.
21 | In the following setting, I have used a GPU-utilized virtual machine (VM) with "Ubuntu Server 20.04 LTS" image in Microsoft Azure.
22 |
23 | ### Install GPU driver (CUDA)
24 |
25 | Install CUDA (NVIDIA GPU driver) as follows.
26 |
27 | ```
28 | # compilers and development settings
29 | sudo apt-get update
30 | sudo apt install -y gcc
31 | sudo apt-get install -y make
32 |
33 | # install CUDA
34 | wget https://developer.download.nvidia.com/compute/cuda/12.2.2/local_installers/cuda_12.2.2_535.104.05_linux.run
35 | sudo sh cuda_12.2.2_535.104.05_linux.run
36 | echo -e "export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64" >> ~/.bashrc
37 | source ~/.bashrc
38 | ```
39 |
40 | ### Install packages
41 |
42 | Install PyTorch, Hugging Face transformer, and other libraries as follows.
43 |
44 | ```
45 | # install and upgrade pip
46 | sudo apt-get install -y python3-pip
47 | sudo -H pip3 install --upgrade pip
48 | # install packages
49 | pip3 install torch transformers pandas matplotlib
50 | # install jupyter for running notebook
51 | pip3 install jupyter
52 | ```
53 |
54 | ## 2. Fine-tune (Train)
55 |
56 | Download this repository.
57 |
58 | ```
59 | git clone https://github.com/tsmatz/finetune_llm_with_lora
60 | ```
61 |
62 | Run jupyter notebook.
63 |
64 | ```
65 | jupyter notebook
66 | ```
67 |
68 | Open jupyter notebook in browser, and run examples in this repository.
69 |
--------------------------------------------------------------------------------
/images/auto_regressive_transformer.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/tsmatz/finetune_llm_with_lora/2e84a5e9e5095aaeacacaa723ee5a7c34c36b678/images/auto_regressive_transformer.png
--------------------------------------------------------------------------------
/images/lora.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/tsmatz/finetune_llm_with_lora/2e84a5e9e5095aaeacacaa723ee5a7c34c36b678/images/lora.png
--------------------------------------------------------------------------------