├── .gitignore ├── 01_pretraining.ipynb ├── 02_fullfinetuning1_base.ipynb ├── 03_fullfinetuning2_instruct.ipynb ├── README.md └── jmcustomdata.txt /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.rsuser 8 | *.suo 9 | *.user 10 | *.userosscache 11 | *.sln.docstates 12 | 13 | # User-specific files (MonoDevelop/Xamarin Studio) 14 | *.userprefs 15 | 16 | # Mono auto generated files 17 | mono_crash.* 18 | 19 | # Build results 20 | [Dd]ebug/ 21 | [Dd]ebugPublic/ 22 | [Rr]elease/ 23 | [Rr]eleases/ 24 | x64/ 25 | x86/ 26 | [Ww][Ii][Nn]32/ 27 | [Aa][Rr][Mm]/ 28 | [Aa][Rr][Mm]64/ 29 | bld/ 30 | [Bb]in/ 31 | [Oo]bj/ 32 | [Ll]og/ 33 | [Ll]ogs/ 34 | 35 | # Visual Studio 2015/2017 cache/options directory 36 | .vs/ 37 | # Uncomment if you have tasks that create the project's static files in wwwroot 38 | #wwwroot/ 39 | 40 | # Visual Studio 2017 auto generated files 41 | Generated\ Files/ 42 | 43 | # MSTest test Results 44 | [Tt]est[Rr]esult*/ 45 | [Bb]uild[Ll]og.* 46 | 47 | # NUnit 48 | *.VisualState.xml 49 | TestResult.xml 50 | nunit-*.xml 51 | 52 | # Build Results of an ATL Project 53 | [Dd]ebugPS/ 54 | [Rr]eleasePS/ 55 | dlldata.c 56 | 57 | # Benchmark Results 58 | BenchmarkDotNet.Artifacts/ 59 | 60 | # .NET Core 61 | project.lock.json 62 | project.fragment.lock.json 63 | artifacts/ 64 | 65 | # ASP.NET Scaffolding 66 | ScaffoldingReadMe.txt 67 | 68 | # StyleCop 69 | StyleCopReport.xml 70 | 71 | # Files built by Visual Studio 72 | *_i.c 73 | *_p.c 74 | *_h.h 75 | *.ilk 76 | *.meta 77 | *.obj 78 | *.iobj 79 | *.pch 80 | *.pdb 81 | *.ipdb 82 | *.pgc 83 | *.pgd 84 | *.rsp 85 | # but not Directory.Build.rsp, as it configures directory-level build defaults 86 | !Directory.Build.rsp 87 | *.sbr 88 | *.tlb 89 | *.tli 90 | *.tlh 91 | *.tmp 92 | *.tmp_proj 93 | *_wpftmp.csproj 94 | *.log 95 | *.tlog 96 | *.vspscc 97 | *.vssscc 98 | .builds 99 | *.pidb 100 | *.svclog 101 | *.scc 102 | 103 | # Chutzpah Test files 104 | _Chutzpah* 105 | 106 | # Visual C++ cache files 107 | ipch/ 108 | *.aps 109 | *.ncb 110 | *.opendb 111 | *.opensdf 112 | *.sdf 113 | *.cachefile 114 | *.VC.db 115 | *.VC.VC.opendb 116 | 117 | # Visual Studio profiler 118 | *.psess 119 | *.vsp 120 | *.vspx 121 | *.sap 122 | 123 | # Visual Studio Trace Files 124 | *.e2e 125 | 126 | # TFS 2012 Local Workspace 127 | $tf/ 128 | 129 | # Guidance Automation Toolkit 130 | *.gpState 131 | 132 | # ReSharper is a .NET coding add-in 133 | _ReSharper*/ 134 | *.[Rr]e[Ss]harper 135 | *.DotSettings.user 136 | 137 | # TeamCity is a build add-in 138 | _TeamCity* 139 | 140 | # DotCover is a Code Coverage Tool 141 | *.dotCover 142 | 143 | # AxoCover is a Code Coverage Tool 144 | .axoCover/* 145 | !.axoCover/settings.json 146 | 147 | # Coverlet is a free, cross platform Code Coverage Tool 148 | coverage*.json 149 | coverage*.xml 150 | coverage*.info 151 | 152 | # Visual Studio code coverage results 153 | *.coverage 154 | *.coveragexml 155 | 156 | # NCrunch 157 | _NCrunch_* 158 | .*crunch*.local.xml 159 | nCrunchTemp_* 160 | 161 | # MightyMoose 162 | *.mm.* 163 | AutoTest.Net/ 164 | 165 | # Web workbench (sass) 166 | .sass-cache/ 167 | 168 | # Installshield output folder 169 | [Ee]xpress/ 170 | 171 | # DocProject is a documentation generator add-in 172 | DocProject/buildhelp/ 173 | DocProject/Help/*.HxT 174 | DocProject/Help/*.HxC 175 | DocProject/Help/*.hhc 176 | DocProject/Help/*.hhk 177 | DocProject/Help/*.hhp 178 | DocProject/Help/Html2 179 | DocProject/Help/html 180 | 181 | # Click-Once directory 182 | publish/ 183 | 184 | # Publish Web Output 185 | *.[Pp]ublish.xml 186 | *.azurePubxml 187 | # Note: Comment the next line if you want to checkin your web deploy settings, 188 | # but database connection strings (with potential passwords) will be unencrypted 189 | *.pubxml 190 | *.publishproj 191 | 192 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 193 | # checkin your Azure Web App publish settings, but sensitive information contained 194 | # in these scripts will be unencrypted 195 | PublishScripts/ 196 | 197 | # NuGet Packages 198 | *.nupkg 199 | # NuGet Symbol Packages 200 | *.snupkg 201 | # The packages folder can be ignored because of Package Restore 202 | **/[Pp]ackages/* 203 | # except build/, which is used as an MSBuild target. 204 | !**/[Pp]ackages/build/ 205 | # Uncomment if necessary however generally it will be regenerated when needed 206 | #!**/[Pp]ackages/repositories.config 207 | # NuGet v3's project.json files produces more ignorable files 208 | *.nuget.props 209 | *.nuget.targets 210 | 211 | # Microsoft Azure Build Output 212 | csx/ 213 | *.build.csdef 214 | 215 | # Microsoft Azure Emulator 216 | ecf/ 217 | rcf/ 218 | 219 | # Windows Store app package directories and files 220 | AppPackages/ 221 | BundleArtifacts/ 222 | Package.StoreAssociation.xml 223 | _pkginfo.txt 224 | *.appx 225 | *.appxbundle 226 | *.appxupload 227 | 228 | # Visual Studio cache files 229 | # files ending in .cache can be ignored 230 | *.[Cc]ache 231 | # but keep track of directories ending in .cache 232 | !?*.[Cc]ache/ 233 | 234 | # Others 235 | ClientBin/ 236 | ~$* 237 | *~ 238 | *.dbmdl 239 | *.dbproj.schemaview 240 | *.jfm 241 | *.pfx 242 | *.publishsettings 243 | orleans.codegen.cs 244 | 245 | # Including strong name files can present a security risk 246 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 247 | #*.snk 248 | 249 | # Since there are multiple workflows, uncomment next line to ignore bower_components 250 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 251 | #bower_components/ 252 | 253 | # RIA/Silverlight projects 254 | Generated_Code/ 255 | 256 | # Backup & report files from converting an old project file 257 | # to a newer Visual Studio version. Backup files are not needed, 258 | # because we have git ;-) 259 | _UpgradeReport_Files/ 260 | Backup*/ 261 | UpgradeLog*.XML 262 | UpgradeLog*.htm 263 | ServiceFabricBackup/ 264 | *.rptproj.bak 265 | 266 | # SQL Server files 267 | *.mdf 268 | *.ldf 269 | *.ndf 270 | 271 | # Business Intelligence projects 272 | *.rdl.data 273 | *.bim.layout 274 | *.bim_*.settings 275 | *.rptproj.rsuser 276 | *- [Bb]ackup.rdl 277 | *- [Bb]ackup ([0-9]).rdl 278 | *- [Bb]ackup ([0-9][0-9]).rdl 279 | 280 | # Microsoft Fakes 281 | FakesAssemblies/ 282 | 283 | # GhostDoc plugin setting file 284 | *.GhostDoc.xml 285 | 286 | # Node.js Tools for Visual Studio 287 | .ntvs_analysis.dat 288 | node_modules/ 289 | 290 | # Visual Studio 6 build log 291 | *.plg 292 | 293 | # Visual Studio 6 workspace options file 294 | *.opt 295 | 296 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 297 | *.vbw 298 | 299 | # Visual Studio 6 auto-generated project file (contains which files were open etc.) 300 | *.vbp 301 | 302 | # Visual Studio 6 workspace and project file (working project files containing files to include in project) 303 | *.dsw 304 | *.dsp 305 | 306 | # Visual Studio 6 technical files 307 | *.ncb 308 | *.aps 309 | 310 | # Visual Studio LightSwitch build output 311 | **/*.HTMLClient/GeneratedArtifacts 312 | **/*.DesktopClient/GeneratedArtifacts 313 | **/*.DesktopClient/ModelManifest.xml 314 | **/*.Server/GeneratedArtifacts 315 | **/*.Server/ModelManifest.xml 316 | _Pvt_Extensions 317 | 318 | # Paket dependency manager 319 | .paket/paket.exe 320 | paket-files/ 321 | 322 | # FAKE - F# Make 323 | .fake/ 324 | 325 | # CodeRush personal settings 326 | .cr/personal 327 | 328 | # Python Tools for Visual Studio (PTVS) 329 | __pycache__/ 330 | *.pyc 331 | 332 | # Cake - Uncomment if you are using it 333 | # tools/** 334 | # !tools/packages.config 335 | 336 | # Tabs Studio 337 | *.tss 338 | 339 | # Telerik's JustMock configuration file 340 | *.jmconfig 341 | 342 | # BizTalk build output 343 | *.btp.cs 344 | *.btm.cs 345 | *.odx.cs 346 | *.xsd.cs 347 | 348 | # OpenCover UI analysis results 349 | OpenCover/ 350 | 351 | # Azure Stream Analytics local run output 352 | ASALocalRun/ 353 | 354 | # MSBuild Binary and Structured Log 355 | *.binlog 356 | 357 | # NVidia Nsight GPU debugger configuration file 358 | *.nvuser 359 | 360 | # MFractors (Xamarin productivity tool) working folder 361 | .mfractor/ 362 | 363 | # Local History for Visual Studio 364 | .localhistory/ 365 | 366 | # Visual Studio History (VSHistory) files 367 | .vshistory/ 368 | 369 | # BeatPulse healthcheck temp database 370 | healthchecksdb 371 | 372 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 373 | MigrationBackup/ 374 | 375 | # Ionide (cross platform F# VS Code tools) working folder 376 | .ionide/ 377 | 378 | # Fody - auto-generated XML schema 379 | FodyWeavers.xsd 380 | 381 | # VS Code files for those working on multiple tools 382 | .vscode/* 383 | !.vscode/settings.json 384 | !.vscode/tasks.json 385 | !.vscode/launch.json 386 | !.vscode/extensions.json 387 | *.code-workspace 388 | 389 | # Local History for Visual Studio Code 390 | .history/ 391 | 392 | # Windows Installer files from build outputs 393 | *.cab 394 | *.msi 395 | *.msix 396 | *.msm 397 | *.msp 398 | 399 | # JetBrains Rider 400 | *.sln.iml 401 | *.txt 402 | !jmcustomdata.txt 403 | *.pth -------------------------------------------------------------------------------- /01_pretraining.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "___\n", 8 | "

\n", 9 | "___\n", 10 | "
Content Copyright by HongLab, Inc.
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "## 대형언어모델(LLM) 바닥부터 만들기\n", 18 | "\n", 19 | "[유튜브 강의 영상 링크](https://youtu.be/osv2csoHVAo)\n", 20 | "\n", 21 | "[홍정모 연구소 디스코드 링크](https://discord.com/invite/kgR9xJkbsV)\n", 22 | "\n", 23 | "[홍정모 연구소 홈페이지 링크](https://www.honglab.ai/)\n", 24 | "\n", 25 | "#### 참고 자료\n", 26 | "- [Andrej Karpathy 유튜브](https://www.youtube.com/andrejkarpathy)\n", 27 | "- [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch)\n", 28 | "- [Om-Alve/smolGPT 깃헙](https://github.com/Om-Alve/smolGPT)\n", 29 | "- 트랜스포머 논문 - [Attention Is All You Need](https://arxiv.org/abs/1706.03762)\n", 30 | "- OpenAI GPT2 논문 - [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)\n", 31 | "\n" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "#### 안내사항\n", 39 | "\n", 40 | "LLM의 핵심 개념을 개인 PC에서도 간단하게 실습하면서 공부할 수 있는 학습 자료입니다. 널리 알려진 교육/학술 자료들을 참고하여 쉽게 공부할 수 있도록 요약하고 정리한 것입니다. 코딩 스타일이나 활용 범위에 대해 오해 없으시길 바랍니다.\n", 41 | "\n", 42 | "윈도우11/WSL, Python 3.9.20, Pytorch 2.6, CUDA 12.6 에서 작동을 확인하였습니다. " 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "#### 전체 과정 요약\n", 50 | "\n", 51 | "LLM 기반 AI 에이전트를 만들때는 핵심이 되는 LLM이 필요한데요, LLM을 바닥부터 만드는 경우 보다는 공개되어 있는 LLM 모델들을 가져다가 나의 용도에 맞도록 다듬어서 사용하는 것이 일반적입니다. 다만, 최근에는 LLM을 바닥부터 만드는 기술에 대한 진입장벽이 낮아지고 있어서 회사별로 필요한 LLM을 바닥부터 각자 만들어 사용하게 될 가능성도 높아지고 있습니다.\n", 52 | "\n", 53 | "LLM을 만들 때는 \n", 54 | "\n", 55 | "1. 사전훈련(pretraining)으로 일반적인 언어 능력을 가르친 후에 \n", 56 | "2. 미세조정(fine tuning) 단계에서 특정 업무에 적응\n", 57 | "\n", 58 | "시키는 것이 기본이 됩니다. 여기에 \n", 59 | "\n", 60 | "3. 데이터베이스(+인터넷) 검색 기능을 추가\n", 61 | "\n", 62 | "하면 지식의 범위와 정확성을 높일 수 있습니다. 사람이 생각을 거듭하여 더 깊이있는 결론을 이끌어 내듯이 LLM도 \n", 63 | "\n", 64 | "4. 내부적으로 질의를 반복하여 더 좋은 결론을 도출\n", 65 | "\n", 66 | "하도록 만들 수 있습니다.\n", 67 | "\n", 68 | "여기서는 LLM의 기본 원리를 이해하기 위해서 사전훈련 과정을 바닥부터 진행해보겠습니다. 훈련 과정의 큰 틀은 일반적인 머신러닝 절차를 따릅니다.\n", 69 | "\n", 70 | "1. 훈련 데이터 준비\n", 71 | "1. 데이터 로더 정의\n", 72 | "1. 모델 정의\n", 73 | "1. 훈련\n", 74 | "1. 결과 확인" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "#### 훈련 데이터 준비\n", 82 | "\n", 83 | "준비한 텍스트 파일을 읽어 들여서 정리한 후에 앞에 cleaned_가 붙은 파일 이름으로 정리합니다.\n", 84 | "> 예시) alice.txt → cleaned_alice.txt\n", 85 | "\n", 86 | "- 캐글 해리포터 책 - [Harry Potter Books](https://www.kaggle.com/datasets/shubhammaindola/harry-potter-books?select=02+Harry+Potter+and+the+Chamber+of+Secrets.txt)\n", 87 | "- 캐글 앨리스 책 - [alice.txt](https://www.kaggle.com/datasets/leelatte/alicetxt)\n", 88 | "- 훈련 데이터나 가중치는 제가 배포하지 않습니다. 직접 다운받거나 준비하셔야합니다." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 1, 94 | "metadata": {}, 95 | "outputs": [ 96 | { 97 | "name": "stdout", 98 | "output_type": "stream", 99 | "text": [ 100 | "cleaned_02 Harry Potter and the Chamber of Secrets.txt 488771 characters\n" 101 | ] 102 | } 103 | ], 104 | "source": [ 105 | "import re\n", 106 | "\n", 107 | "def clean_text(filename):\n", 108 | " with open(filename, 'r', encoding='utf-8') as file:\n", 109 | " book_text = file.read()\n", 110 | "\n", 111 | " cleaned_text = re.sub(r'\\n+', ' ', book_text) # 줄바꿈을 빈칸으로 변경\n", 112 | " cleaned_text = re.sub(r'\\s+', ' ', cleaned_text) # 여러 빈칸을 하나의 빈칸으로\n", 113 | "\n", 114 | " print(\"cleaned_\" + filename, len(cleaned_text), \"characters\") # 글자 수 출력\n", 115 | "\n", 116 | " with open(\"cleaned_\" + filename, 'w', encoding='utf-8') as file:\n", 117 | " file.write(cleaned_text)\n", 118 | "\n", 119 | "filenames_list = [\"02 Harry Potter and the Chamber of Secrets.txt\"]\n", 120 | "\n", 121 | "for filename in filenames_list:\n", 122 | " clean_text(filename)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "#### 토큰화\n", 130 | "\n", 131 | "UTF-8 BPE(Bype Pair Encoding)" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 2, 137 | "metadata": {}, 138 | "outputs": [ 139 | { 140 | "name": "stdout", 141 | "output_type": "stream", 142 | "text": [ 143 | "글자수: 26 토큰수 6\n", 144 | "[18308, 14179, 373, 257, 18731, 13]\n", 145 | "Harry Potter was a wizard.\n", 146 | "18308\t -> Harry\n", 147 | "14179\t -> Potter\n", 148 | "373\t -> was\n", 149 | "257\t -> a\n", 150 | "18731\t -> wizard\n", 151 | "13\t -> .\n" 152 | ] 153 | } 154 | ], 155 | "source": [ 156 | "import tiktoken # pip install tiktoken\n", 157 | "\n", 158 | "tokenizer = tiktoken.get_encoding(\"gpt2\")\n", 159 | "\n", 160 | "text = \"Harry Potter was a wizard.\"\n", 161 | "\n", 162 | "tokens = tokenizer.encode(text)\n", 163 | "\n", 164 | "print(\"글자수:\", len(text), \"토큰수\", len(tokens))\n", 165 | "print(tokens)\n", 166 | "print(tokenizer.decode(tokens))\n", 167 | "for t in tokens:\n", 168 | " print(f\"{t}\\t -> {tokenizer.decode([t])}\")" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": 3, 174 | "metadata": {}, 175 | "outputs": [], 176 | "source": [ 177 | "# from transformers import AutoTokenizer # pip install transformers\n", 178 | "\n", 179 | "# tokenizer = AutoTokenizer.from_pretrained(\"LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct\") # KoGPT2 사용\n", 180 | "# # tokenizer = AutoTokenizer.from_pretrained(\"skt/kogpt2-base-v2\") # KoGPT2 사용\n", 181 | "\n", 182 | "# print(\"Vocab size :\", len(tokenizer))\n", 183 | "\n", 184 | "# text = \"대사께서는 도(道)를 얻은 모양이구려.\"\n", 185 | "\n", 186 | "# tokens = tokenizer.encode(text)\n", 187 | "\n", 188 | "# print(len(text), len(tokens))\n", 189 | "# print(tokens)\n", 190 | "# print(tokenizer.decode(tokens))" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": 4, 196 | "metadata": {}, 197 | "outputs": [ 198 | { 199 | "name": "stdout", 200 | "output_type": "stream", 201 | "text": [ 202 | "H -> [39] -> H\n", 203 | "a -> [64] -> a\n", 204 | "r -> [81] -> r\n", 205 | "r -> [81] -> r\n", 206 | "y -> [88] -> y\n", 207 | " -> [220] -> \n", 208 | "P -> [47] -> P\n", 209 | "o -> [78] -> o\n", 210 | "t -> [83] -> t\n", 211 | "t -> [83] -> t\n", 212 | "e -> [68] -> e\n", 213 | "r -> [81] -> r\n", 214 | " -> [220] -> \n", 215 | "w -> [86] -> w\n", 216 | "a -> [64] -> a\n", 217 | "s -> [82] -> s\n", 218 | " -> [220] -> \n", 219 | "a -> [64] -> a\n", 220 | " -> [220] -> \n", 221 | "w -> [86] -> w\n", 222 | "i -> [72] -> i\n", 223 | "z -> [89] -> z\n", 224 | "a -> [64] -> a\n", 225 | "r -> [81] -> r\n", 226 | "d -> [67] -> d\n", 227 | ". -> [13] -> .\n" 228 | ] 229 | } 230 | ], 231 | "source": [ 232 | "for char in text:\n", 233 | " token_ids = tokenizer.encode(char) # 한 글자씩 인코딩(토큰화)\n", 234 | " decoded = tokenizer.decode(token_ids) # 한 글자씩 디코딩\n", 235 | " print(f\"{char} -> {token_ids} -> {decoded}\")" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "#### 데이터로더(DataLoader)" 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "metadata": {}, 249 | "outputs": [ 250 | { 251 | "name": "stdout", 252 | "output_type": "stream", 253 | "text": [ 254 | "# of tokens in txt: 130520\n" 255 | ] 256 | } 257 | ], 258 | "source": [ 259 | "import torch\n", 260 | "from torch.utils.data import Dataset, DataLoader\n", 261 | "\n", 262 | "class MyDataset(Dataset):\n", 263 | " def __init__(self, txt, max_length, stride):\n", 264 | " self.input_ids = []\n", 265 | " self.target_ids = []\n", 266 | "\n", 267 | " # token_ids = tokenizer.encode(\"<|endoftext|>\" + txt, allowed_special={\"<|endoftext|>\"})\n", 268 | " token_ids = tokenizer.encode(txt)\n", 269 | "\n", 270 | " print(\"# of tokens in txt:\", len(token_ids))\n", 271 | "\n", 272 | " for i in range(0, len(token_ids) - max_length, stride):\n", 273 | " input_chunk = token_ids[i:i + max_length]\n", 274 | " target_chunk = token_ids[i + 1: i + max_length + 1]\n", 275 | " self.input_ids.append(torch.tensor(input_chunk))\n", 276 | " self.target_ids.append(torch.tensor(target_chunk))\n", 277 | "\n", 278 | " def __len__(self):\n", 279 | " return len(self.input_ids)\n", 280 | "\n", 281 | " def __getitem__(self, idx):\n", 282 | " return self.input_ids[idx], self.target_ids[idx]\n", 283 | "\n", 284 | "# with open(\"cleaned_한글문서.txt\", 'r', encoding='utf-8-sig') as file: # 선택: -sig를 붙여서 BOM 제거\n", 285 | "with open(\"cleaned_02 Harry Potter and the Chamber of Secrets.txt\", 'r', encoding='utf-8-sig') as file: # 선택: -sig를 붙여서 BOM 제거\n", 286 | " txt = file.read()\n", 287 | "\n", 288 | "dataset = MyDataset(txt, max_length = 32, stride = 4)\n", 289 | "\n", 290 | "train_loader = DataLoader(dataset, batch_size=128, shuffle=True, drop_last=True)\n", 291 | "\n", 292 | "# 주의: 여기서는 코드를 단순화하기 위해 test, valid는 생략하고 train_loader만 만들었습니다.\n", 293 | "# 관련된 ML 이론이 궁금하신 분들은 train vs test vs validation 등으로 검색해보세요." 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 6, 299 | "metadata": {}, 300 | "outputs": [ 301 | { 302 | "name": "stdout", 303 | "output_type": "stream", 304 | "text": [ 305 | " stay put.” Soon, the crowd of gnomes in the field started walking away in a straggling line, their little shoulders hunched. �\n", 306 | " put.” Soon, the crowd of gnomes in the field started walking away in a straggling line, their little shoulders hunched. “\n" 307 | ] 308 | } 309 | ], 310 | "source": [ 311 | "dataiter = iter(train_loader)\n", 312 | "\n", 313 | "x, y = next(dataiter)\n", 314 | "\n", 315 | "print(tokenizer.decode(x[0].tolist()))\n", 316 | "print(tokenizer.decode(y[0].tolist()))\n" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": {}, 322 | "source": [ 323 | "#### 뉴럴네트워크 모델 정의\n", 324 | "\n", 325 | "모델 정의는 교재 \"[Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch)\"에서 제공하는 [예제 코드](https://github.com/rasbt/LLMs-from-scratch)를 약간 수정하였습니다.\n" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": null, 331 | "metadata": {}, 332 | "outputs": [], 333 | "source": [ 334 | "# 모델을 정의할 때 사용하는 상수들\n", 335 | "\n", 336 | "VOCAB_SIZE = tokenizer.n_vocab # 50257 Tiktoken\n", 337 | "#VOCAB_SIZE = len(tokenizer) # AutoTokenizer\n", 338 | "CONTEXT_LENGTH = 128 # Shortened context length (orig: 1024)\n", 339 | "EMB_DIM = 768 # Embedding dimension\n", 340 | "NUM_HEADS = 12 # Number of attention heads\n", 341 | "NUM_LAYERS = 12 # Number of layers\n", 342 | "DROP_RATE = 0.1 # Dropout rate\n", 343 | "QKV_BIAS = False # Query-key-value bias" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "metadata": {}, 350 | "outputs": [], 351 | "source": [ 352 | "import torch.nn as nn\n", 353 | "\n", 354 | "class MultiHeadAttention(nn.Module):\n", 355 | " def __init__(self, d_in, d_out):\n", 356 | " super().__init__()\n", 357 | " \n", 358 | " assert d_out % NUM_HEADS == 0, \"d_out must be divisible by n_heads\"\n", 359 | "\n", 360 | " self.d_out = d_out\n", 361 | " self.head_dim = d_out // NUM_HEADS\n", 362 | "\n", 363 | " self.W_query = nn.Linear(d_in, d_out, bias=QKV_BIAS)\n", 364 | " self.W_key = nn.Linear(d_in, d_out, bias=QKV_BIAS)\n", 365 | " self.W_value = nn.Linear(d_in, d_out, bias=QKV_BIAS)\n", 366 | " self.out_proj = nn.Linear(d_out, d_out)\n", 367 | " self.dropout = nn.Dropout(DROP_RATE)\n", 368 | " self.register_buffer('mask', torch.triu(torch.ones(CONTEXT_LENGTH, CONTEXT_LENGTH), diagonal=1))\n", 369 | "\n", 370 | " def forward(self, x):\n", 371 | " b, num_tokens, d_in = x.shape\n", 372 | "\n", 373 | " keys = self.W_key(x) # (b, num_tokens, d_out)\n", 374 | " queries = self.W_query(x)\n", 375 | " values = self.W_value(x)\n", 376 | "\n", 377 | " keys = keys.view(b, num_tokens, NUM_HEADS, self.head_dim)\n", 378 | " values = values.view(b, num_tokens, NUM_HEADS, self.head_dim)\n", 379 | " queries = queries.view(b, num_tokens, NUM_HEADS, self.head_dim)\n", 380 | "\n", 381 | " keys = keys.transpose(1, 2)\n", 382 | " queries = queries.transpose(1, 2)\n", 383 | " values = values.transpose(1, 2)\n", 384 | "\n", 385 | " attn_scores = queries @ keys.transpose(2, 3)\n", 386 | "\n", 387 | " mask_bool = self.mask.bool()[:num_tokens, :num_tokens]\n", 388 | "\n", 389 | " attn_scores.masked_fill_(mask_bool, -torch.inf)\n", 390 | "\n", 391 | " attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n", 392 | " attn_weights = self.dropout(attn_weights)\n", 393 | "\n", 394 | " context_vec = (attn_weights @ values).transpose(1, 2)\n", 395 | "\n", 396 | " context_vec = context_vec.reshape(b, num_tokens, self.d_out)\n", 397 | " context_vec = self.out_proj(context_vec)\n", 398 | "\n", 399 | " return context_vec\n", 400 | "\n", 401 | "class LayerNorm(nn.Module):\n", 402 | " def __init__(self, emb_dim):\n", 403 | " super().__init__()\n", 404 | " self.eps = 1e-5\n", 405 | " self.scale = nn.Parameter(torch.ones(emb_dim))\n", 406 | " self.shift = nn.Parameter(torch.zeros(emb_dim))\n", 407 | "\n", 408 | " def forward(self, x):\n", 409 | " mean = x.mean(dim=-1, keepdim=True)\n", 410 | " var = x.var(dim=-1, keepdim=True, unbiased=False)\n", 411 | " norm_x = (x - mean) / torch.sqrt(var + self.eps)\n", 412 | " return self.scale * norm_x + self.shift\n", 413 | "\n", 414 | "class GELU(nn.Module):\n", 415 | " def __init__(self):\n", 416 | " super().__init__()\n", 417 | "\n", 418 | " def forward(self, x):\n", 419 | " return 0.5 * x * (1 + torch.tanh(\n", 420 | " torch.sqrt(torch.tensor(2.0 / torch.pi)) *\n", 421 | " (x + 0.044715 * torch.pow(x, 3))\n", 422 | " ))\n", 423 | "\n", 424 | "class FeedForward(nn.Module):\n", 425 | " def __init__(self):\n", 426 | " super().__init__()\n", 427 | " self.layers = nn.Sequential(\n", 428 | " nn.Linear(EMB_DIM, 4 * EMB_DIM),\n", 429 | " GELU(),\n", 430 | " nn.Linear(4 * EMB_DIM, EMB_DIM),\n", 431 | " )\n", 432 | "\n", 433 | " def forward(self, x):\n", 434 | " return self.layers(x)\n", 435 | "\n", 436 | "class TransformerBlock(nn.Module):\n", 437 | " def __init__(self):\n", 438 | " super().__init__()\n", 439 | " self.att = MultiHeadAttention(\n", 440 | " d_in=EMB_DIM,\n", 441 | " d_out=EMB_DIM)\n", 442 | " \n", 443 | " self.ff = FeedForward()\n", 444 | " self.norm1 = LayerNorm(EMB_DIM)\n", 445 | " self.norm2 = LayerNorm(EMB_DIM)\n", 446 | " self.drop_shortcut = nn.Dropout(DROP_RATE)\n", 447 | "\n", 448 | " def forward(self, x):\n", 449 | " shortcut = x\n", 450 | " x = self.norm1(x)\n", 451 | " x = self.att(x)\n", 452 | " x = self.drop_shortcut(x)\n", 453 | " x = x + shortcut\n", 454 | "\n", 455 | " shortcut = x\n", 456 | " x = self.norm2(x)\n", 457 | " x = self.ff(x)\n", 458 | " x = self.drop_shortcut(x)\n", 459 | " x = x + shortcut\n", 460 | "\n", 461 | " return x\n", 462 | "\n", 463 | "\n", 464 | "class GPTModel(nn.Module):\n", 465 | " def __init__(self):\n", 466 | " super().__init__()\n", 467 | " self.tok_emb = nn.Embedding(VOCAB_SIZE, EMB_DIM)\n", 468 | " self.pos_emb = nn.Embedding(CONTEXT_LENGTH, EMB_DIM)\n", 469 | " self.drop_emb = nn.Dropout(DROP_RATE)\n", 470 | "\n", 471 | " self.trf_blocks = nn.Sequential(\n", 472 | " *[TransformerBlock() for _ in range(NUM_LAYERS)])\n", 473 | "\n", 474 | " self.final_norm = LayerNorm(EMB_DIM)\n", 475 | " self.out_head = nn.Linear(EMB_DIM, VOCAB_SIZE, bias=False)\n", 476 | "\n", 477 | " def forward(self, in_idx):\n", 478 | " batch_size, seq_len = in_idx.shape\n", 479 | " tok_embeds = self.tok_emb(in_idx)\n", 480 | " pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n", 481 | " x = tok_embeds + pos_embeds # Shape [batch_size, num_tokens, emb_size]\n", 482 | " x = self.drop_emb(x)\n", 483 | " x = self.trf_blocks(x)\n", 484 | " x = self.final_norm(x)\n", 485 | " logits = self.out_head(x)\n", 486 | " return logits" 487 | ] 488 | }, 489 | { 490 | "cell_type": "markdown", 491 | "metadata": {}, 492 | "source": [ 493 | "#### 훈련" 494 | ] 495 | }, 496 | { 497 | "cell_type": "code", 498 | "execution_count": 9, 499 | "metadata": {}, 500 | "outputs": [ 501 | { 502 | "name": "stdout", 503 | "output_type": "stream", 504 | "text": [ 505 | "cuda\n" 506 | ] 507 | } 508 | ], 509 | "source": [ 510 | "import torch\n", 511 | "\n", 512 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 513 | "#device = \"cpu\"\n", 514 | "print(device)\n", 515 | "\n", 516 | "torch.manual_seed(123)\n", 517 | "model = GPTModel()\n", 518 | "model.to(device)\n", 519 | "optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1)" 520 | ] 521 | }, 522 | { 523 | "cell_type": "code", 524 | "execution_count": null, 525 | "metadata": {}, 526 | "outputs": [ 527 | { 528 | "name": "stdout", 529 | "output_type": "stream", 530 | "text": [ 531 | "Tokens seen: 4096\n", 532 | "Epoch: 1, Loss: 4.3849126466615935\n", 533 | "Epoch: 2, Loss: 2.21952637959653\n", 534 | "Epoch: 3, Loss: 0.795975233391514\n", 535 | "Tokens seen: 4100096\n", 536 | "Epoch: 4, Loss: 0.3919607914808228\n", 537 | "Epoch: 5, Loss: 0.3044665302113285\n", 538 | "Epoch: 6, Loss: 0.2706770875791865\n", 539 | "Epoch: 7, Loss: 0.252907565200892\n", 540 | "Tokens seen: 8196096\n", 541 | "Epoch: 8, Loss: 0.24298487410066635\n", 542 | "Epoch: 9, Loss: 0.2365198473878733\n", 543 | "Epoch: 10, Loss: 0.23125613416273763\n", 544 | "Epoch: 11, Loss: 0.22732559315801606\n", 545 | "Tokens seen: 12292096\n", 546 | "Epoch: 12, Loss: 0.22307928592905285\n", 547 | "Epoch: 13, Loss: 0.22022464220214077\n", 548 | "Epoch: 14, Loss: 0.216401994404361\n", 549 | "Epoch: 15, Loss: 0.21275856956018238\n", 550 | "Tokens seen: 16388096\n", 551 | "Epoch: 16, Loss: 0.21049895642075953\n", 552 | "Epoch: 17, Loss: 0.2087053178450254\n", 553 | "Epoch: 18, Loss: 0.206223381199236\n", 554 | "Epoch: 19, Loss: 0.20389414138681305\n", 555 | "Tokens seen: 20484096\n", 556 | "Epoch: 20, Loss: 0.2023714334945979\n", 557 | "Epoch: 21, Loss: 0.20010770434939015\n", 558 | "Epoch: 22, Loss: 0.19871540608133856\n", 559 | "Epoch: 23, Loss: 0.19787393076213325\n", 560 | "Tokens seen: 24580096\n", 561 | "Epoch: 24, Loss: 0.19627788334380925\n", 562 | "Epoch: 25, Loss: 0.19420364475625707\n", 563 | "Epoch: 26, Loss: 0.19332646631349729\n", 564 | "Epoch: 27, Loss: 0.19182916701309324\n", 565 | "Tokens seen: 28676096\n", 566 | "Epoch: 28, Loss: 0.19007714321528832\n", 567 | "Epoch: 29, Loss: 0.18967703454137788\n", 568 | "Epoch: 30, Loss: 0.1890431342749145\n", 569 | "Epoch: 31, Loss: 0.1869718112696813\n", 570 | "Tokens seen: 32772096\n", 571 | "Epoch: 32, Loss: 0.18755552718254526\n", 572 | "Epoch: 33, Loss: 0.18641565006783628\n", 573 | "Epoch: 34, Loss: 0.18429208611409495\n", 574 | "Epoch: 35, Loss: 0.184950728292071\n", 575 | "Tokens seen: 36868096\n", 576 | "Epoch: 36, Loss: 0.18267620935684115\n", 577 | "Epoch: 37, Loss: 0.1823706328868866\n", 578 | "Epoch: 38, Loss: 0.1813517094949099\n", 579 | "Epoch: 39, Loss: 0.18206301349119877\n", 580 | "Tokens seen: 40964096\n", 581 | "Epoch: 40, Loss: 0.1808522997174676\n", 582 | "Epoch: 41, Loss: 0.17978976056801052\n", 583 | "Epoch: 42, Loss: 0.18024734847658264\n", 584 | "Epoch: 43, Loss: 0.17954657766528018\n", 585 | "Tokens seen: 45060096\n", 586 | "Epoch: 44, Loss: 0.17911469478776135\n", 587 | "Epoch: 45, Loss: 0.17840020986288552\n", 588 | "Epoch: 46, Loss: 0.17822550518775548\n", 589 | "Epoch: 47, Loss: 0.17718226051940691\n", 590 | "Tokens seen: 49156096\n", 591 | "Epoch: 48, Loss: 0.1768852232361403\n", 592 | "Epoch: 49, Loss: 0.176060066096426\n", 593 | "Epoch: 50, Loss: 0.17577019515703982\n", 594 | "Epoch: 51, Loss: 0.17622286157579872\n", 595 | "Tokens seen: 53252096\n", 596 | "Epoch: 52, Loss: 0.17553523792995243\n", 597 | "Epoch: 53, Loss: 0.1750457174196018\n", 598 | "Epoch: 54, Loss: 0.17449502060263175\n", 599 | "Epoch: 55, Loss: 0.17480581869759898\n", 600 | "Tokens seen: 57348096\n", 601 | "Epoch: 56, Loss: 0.17491596591050232\n", 602 | "Epoch: 57, Loss: 0.1731891006700636\n", 603 | "Epoch: 58, Loss: 0.17240459507140588\n", 604 | "Epoch: 59, Loss: 0.17229765301614297\n", 605 | "Tokens seen: 61444096\n", 606 | "Epoch: 60, Loss: 0.1734856069674642\n", 607 | "Epoch: 61, Loss: 0.17345472237491233\n", 608 | "Epoch: 62, Loss: 0.17161158924027692\n", 609 | "Tokens seen: 65540096\n", 610 | "Epoch: 63, Loss: 0.1709385947214337\n", 611 | "Epoch: 64, Loss: 0.17091810169417088\n", 612 | "Epoch: 65, Loss: 0.17074644665314456\n", 613 | "Epoch: 66, Loss: 0.17133229384272117\n", 614 | "Tokens seen: 69636096\n", 615 | "Epoch: 67, Loss: 0.17110126101829876\n", 616 | "Epoch: 68, Loss: 0.1721280293436501\n", 617 | "Epoch: 69, Loss: 0.17072570623140634\n", 618 | "Epoch: 70, Loss: 0.16960367349189098\n", 619 | "Tokens seen: 73732096\n", 620 | "Epoch: 71, Loss: 0.16957374380564127\n", 621 | "Epoch: 72, Loss: 0.16972881045163146\n", 622 | "Epoch: 73, Loss: 0.1690521845667381\n", 623 | "Epoch: 74, Loss: 0.1688809606033986\n", 624 | "Tokens seen: 77828096\n", 625 | "Epoch: 75, Loss: 0.16812912779530204\n", 626 | "Epoch: 76, Loss: 0.1686125879329959\n", 627 | "Epoch: 77, Loss: 0.16925425322975698\n", 628 | "Epoch: 78, Loss: 0.16963306180839463\n", 629 | "Tokens seen: 81924096\n", 630 | "Epoch: 79, Loss: 0.16827140689834835\n", 631 | "Epoch: 80, Loss: 0.16751441652849902\n", 632 | "Epoch: 81, Loss: 0.16737041264537753\n", 633 | "Epoch: 82, Loss: 0.16741752108251015\n", 634 | "Tokens seen: 86020096\n", 635 | "Epoch: 83, Loss: 0.16757114859312539\n", 636 | "Epoch: 84, Loss: 0.16644526419677133\n", 637 | "Epoch: 85, Loss: 0.1670930372683082\n", 638 | "Epoch: 86, Loss: 0.16745461541132664\n", 639 | "Tokens seen: 90116096\n", 640 | "Epoch: 87, Loss: 0.16770464979757474\n", 641 | "Epoch: 88, Loss: 0.1680629438771976\n", 642 | "Epoch: 89, Loss: 0.16704368784906357\n", 643 | "Epoch: 90, Loss: 0.16570961305240947\n", 644 | "Tokens seen: 94212096\n", 645 | "Epoch: 91, Loss: 0.165093104083707\n", 646 | "Epoch: 92, Loss: 0.16592367116625853\n", 647 | "Epoch: 93, Loss: 0.16609569850165073\n", 648 | "Epoch: 94, Loss: 0.16609822031785185\n", 649 | "Tokens seen: 98308096\n", 650 | "Epoch: 95, Loss: 0.1656451877646559\n", 651 | "Epoch: 96, Loss: 0.16611929202642967\n", 652 | "Epoch: 97, Loss: 0.16548592274583232\n", 653 | "Epoch: 98, Loss: 0.16605482602447974\n", 654 | "Tokens seen: 102404096\n", 655 | "Epoch: 99, Loss: 0.16468763955700116\n", 656 | "Epoch: 100, Loss: 0.16477952316755384\n" 657 | ] 658 | } 659 | ], 660 | "source": [ 661 | "tokens_seen, global_step = 0, -1\n", 662 | "\n", 663 | "losses = []\n", 664 | "\n", 665 | "for epoch in range(100):\n", 666 | " model.train() # Set model to training mode\n", 667 | " \n", 668 | " epoch_loss = 0\n", 669 | " for input_batch, target_batch in train_loader:\n", 670 | " optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n", 671 | " input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n", 672 | "\n", 673 | " logits = model(input_batch)\n", 674 | " loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n", 675 | " epoch_loss += loss.item()\n", 676 | " loss.backward() # Calculate loss gradients\n", 677 | " optimizer.step() # Update model weights using loss gradients\n", 678 | " tokens_seen += input_batch.numel()\n", 679 | " global_step += 1\n", 680 | "\n", 681 | " if global_step % 1000 == 0:\n", 682 | " print(f\"Tokens seen: {tokens_seen}\")\n", 683 | " # Optional evaluation step\n", 684 | "\n", 685 | " avg_loss = epoch_loss / len(train_loader)\n", 686 | " losses.append(avg_loss)\n", 687 | " print(f\"Epoch: {epoch + 1}, Loss: {avg_loss}\")\n", 688 | " torch.save(model.state_dict(), \"model_\" + str(epoch + 1).zfill(3) + \".pth\")\n", 689 | "\n", 690 | "# 주의: 여기서는 편의상 모든 데이터를 train에 사용하였습니다. \n", 691 | "# ML에서는 일부 데이터를 validation에 사용하는 것이 일반적입니다." 692 | ] 693 | }, 694 | { 695 | "cell_type": "code", 696 | "execution_count": null, 697 | "metadata": {}, 698 | "outputs": [ 699 | { 700 | "data": { 701 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAHFCAYAAADcytJ5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6SElEQVR4nO3deXxU9b3/8feZNQshELawBMSioFAWQRRE2VqURUWkVUSFaq9FgQul3gpiC6Vo1N9t3YtLFWrVYmmV0qooyuZSCoIgbqi3ArFsApIECJNk5vv7YzInGbIPkzkDvJ6PxzwgZ86Z85lvhuTNdznHMsYYAQAAJCGX0wUAAABUh6ACAACSFkEFAAAkLYIKAABIWgQVAACQtAgqAAAgaRFUAABA0iKoAACApEVQAQAASYuggtOGZVl1eqxevfqEzjN37lxZlhXTsatXr45LDSdy7r/85S8JP3cs1q1bpx/84Adq3bq1fD6fsrOzNXbsWP3zn/90urRKtm/fXuNnbu7cuU6XqDPOOEOjRo1yugygEo/TBQCJcvwvsF//+tdatWqVVq5cGbX93HPPPaHz/PjHP9Zll10W07HnnXee/vnPf55wDae6Rx55RNOnT1ffvn11//33q0OHDtq5c6cee+wxDRgwQA899JCmTJnidJmVTJ06Vdddd12l7e3atXOgGuDkQFDBaePCCy+M+rpFixZyuVyVth/v6NGjSktLq/N52rVrF/MvnsaNG9daz+nu3Xff1fTp0zVixAi9/PLL8njKf4xde+21uuqqqzRt2jT16tVLF110UcLqKioqUkpKSo29ae3bt+f7C9QTQz9ABYMGDVK3bt20du1a9e/fX2lpabrpppskSS+++KKGDRum1q1bKzU1Veecc45mzpypI0eORL1GVUM/kW715cuX67zzzlNqaqq6dOmiZ555Jmq/qoZ+Jk6cqEaNGunLL7/UiBEj1KhRI+Xk5OhnP/uZAoFA1PFff/21xo4dq4yMDDVp0kTjx4/Xhg0bZFmWFi1aFJc2+uijj3TllVeqadOmSklJUc+ePfWHP/whap9QKKT58+erc+fOSk1NVZMmTdS9e3c99NBD9j7ffPONbrnlFuXk5Mjv96tFixa66KKL9Oabb9Z4/tzcXFmWpQULFkSFFEnyeDz63e9+J8uydO+990qSli5dKsuy9NZbb1V6rQULFsiyLH344Yf2tvfff19XXHGFsrKylJKSol69eunPf/5z1HGLFi2SZVl64403dNNNN6lFixZKS0ur9P2IReQz+Pbbb+vCCy9Uamqq2rZtq1/84hcKBoNR+x48eFC33Xab2rZtK5/PpzPPPFOzZ8+uVEcoFNIjjzyinj172t+PCy+8UMuWLat0/to+o0ePHtXtt9+ujh07KiUlRVlZWerTp4/+9Kc/nfB7B6pCjwpwnN27d+v666/Xz3/+c91zzz1yucJ5/osvvtCIESM0ffp0paen67PPPtN9992n9evXVxo+qsqWLVv0s5/9TDNnzlSrVq30+9//XjfffLM6deqkSy65pMZjS0pKdMUVV+jmm2/Wz372M61du1a//vWvlZmZqV/+8peSpCNHjmjw4ME6ePCg7rvvPnXq1EnLly/XNddcc+KNUmbbtm3q37+/WrZsqYcffljNmjXTc889p4kTJ2rv3r36+c9/Lkm6//77NXfuXN1111265JJLVFJSos8++0yHDh2yX+uGG27Qpk2bdPfdd+vss8/WoUOHtGnTJh04cKDa8weDQa1atUp9+vSpttcqJydHvXv31sqVKxUMBjVq1Ci1bNlSCxcu1NChQ6P2XbRokc477zx1795dkrRq1SpddtlluuCCC/T4448rMzNTixcv1jXXXKOjR49q4sSJUcffdNNNGjlypP74xz/qyJEj8nq9NbZfKBRSaWlppe3HB649e/bo2muv1cyZMzVv3jy98sormj9/vr799ls9+uijkqRjx45p8ODB+r//+z/96le/Uvfu3fX2228rNzdXmzdv1iuvvGK/3sSJE/Xcc8/p5ptv1rx58+Tz+bRp0yZt37496rx1+YzOmDFDf/zjHzV//nz16tVLR44c0UcffVTj9w04IQY4TU2YMMGkp6dHbRs4cKCRZN56660ajw2FQqakpMSsWbPGSDJbtmyxn5szZ445/p9Whw4dTEpKitmxY4e9raioyGRlZZmf/OQn9rZVq1YZSWbVqlVRdUoyf/7zn6Nec8SIEaZz587214899piRZF577bWo/X7yk58YSWbhwoU1vqfIuZcsWVLtPtdee63x+/1m586dUduHDx9u0tLSzKFDh4wxxowaNcr07NmzxvM1atTITJ8+vcZ9jrdnzx4jyVx77bU17nfNNdcYSWbv3r3GGGNmzJhhUlNT7fqMMeaTTz4xkswjjzxib+vSpYvp1auXKSkpiXq9UaNGmdatW5tgMGiMMWbhwoVGkrnxxhvrVPdXX31lJFX7ePvtt+19I5/Bv/3tb1Gv8V//9V/G5XLZn6HHH3+8ys/FfffdZySZN954wxhjzNq1a40kM3v27BprrOtntFu3bmb06NF1et9APDD0AxynadOmGjJkSKXt//73v3XdddcpOztbbrdbXq9XAwcOlCR9+umntb5uz5491b59e/vrlJQUnX322dqxY0etx1qWpcsvvzxqW/fu3aOOXbNmjTIyMipN5B03blytr19XK1eu1NChQ5WTkxO1feLEiTp69Kg9Yblv377asmWLbrvtNr3++usqKCio9Fp9+/bVokWLNH/+fK1bt04lJSVxq9MYI0n2ENxNN92koqIivfjii/Y+CxculN/vtye3fvnll/rss880fvx4SVJpaan9GDFihHbv3q1t27ZFnefqq6+uV13Tpk3Thg0bKj169uwZtV9GRoauuOKKqG3XXXedQqGQ1q5dKyn8vUhPT9fYsWOj9ov0+kSGul577TVJ0uTJk2utry6f0b59++q1117TzJkztXr1ahUVFdXtzQMxIqgAx2ndunWlbYcPH9bFF1+sf/3rX5o/f75Wr16tDRs26KWXXpKkOv2wbtasWaVtfr+/TsempaUpJSWl0rHHjh2zvz5w4IBatWpV6diqtsXqwIEDVbZPmzZt7OcladasWfrf//1frVu3TsOHD1ezZs00dOhQvf/++/YxL774oiZMmKDf//736tevn7KysnTjjTdqz5491Z6/efPmSktL01dffVVjndu3b1daWpqysrIkSV27dtX555+vhQsXSgoPIT333HO68sor7X327t0rSbr99tvl9XqjHrfddpskaf/+/VHnqaotatKuXTv16dOn0qNRo0ZR+1X1PcvOzpZU3sYHDhxQdnZ2pflQLVu2lMfjsff75ptv5Ha77eNrUpfP6MMPP6w77rhDS5cu1eDBg5WVlaXRo0friy++qPX1gVgQVIDjVLVqY+XKldq1a5eeeeYZ/fjHP9Yll1yiPn36KCMjw4EKq9asWTP7l21FNf3ij+Ucu3fvrrR9165dksJBQgrPuZgxY4Y2bdqkgwcP6k9/+pPy8vJ06aWX6ujRo/a+Dz74oLZv364dO3YoNzdXL730UqV5IBW53W4NHjxY77//vr7++usq9/n666+1ceNGDRkyRG63297+ox/9SOvWrdOnn36q5cuXa/fu3frRj35kPx+pfdasWVX2elTV8xHr9XJqU9P3MRImIt/vSO9RxL59+1RaWmq/nxYtWigYDMbtc5Cenq5f/epX+uyzz7Rnzx4tWLBA69atq9TjB8QLQQWog8gvJL/fH7X9iSeecKKcKg0cOFCFhYV2V3/E4sWL43aOoUOH2qGtomeffVZpaWlVLr1t0qSJxo4dq8mTJ+vgwYOVJnBK4WW7U6ZM0fe//31t2rSpxhpmzZolY4xuu+22SqtggsGgbr31VhljNGvWrKjnxo0bp5SUFC1atEiLFi1S27ZtNWzYMPv5zp0766yzztKWLVuq7PVIZDAtLCystCLnhRdekMvlsie1Dh06VIcPH9bSpUuj9nv22Wft5yVp+PDhksIrnOKtVatWmjhxosaNG6dt27bZIRSIJ1b9AHXQv39/NW3aVJMmTdKcOXPk9Xr1/PPPa8uWLU6XZpswYYIeeOABXX/99Zo/f746deqk1157Ta+//rok2auXarNu3boqtw8cOFBz5szRP/7xDw0ePFi//OUvlZWVpeeff16vvPKK7r//fmVmZkqSLr/8cnXr1k19+vRRixYttGPHDj344IPq0KGDzjrrLOXn52vw4MG67rrr1KVLF2VkZGjDhg1avny5xowZU2N9F110kR588EFNnz5dAwYM0JQpU9S+fXv7gm//+te/9OCDD6p///5RxzVp0kRXXXWVFi1apEOHDun222+v1CZPPPGEhg8frksvvVQTJ05U27ZtdfDgQX366afatGmTlixZUqc2rM7OnTurbN8WLVroO9/5jv11s2bNdOutt2rnzp06++yz9eqrr+qpp57Srbfeas8hufHGG/XYY49pwoQJ2r59u7773e/qnXfe0T333KMRI0boe9/7niTp4osv1g033KD58+dr7969GjVqlPx+vz744AOlpaVp6tSp9XoPF1xwgUaNGqXu3buradOm+vTTT/XHP/5R/fr1q9f1hoA6c3YuL+Cc6lb9dO3atcr933vvPdOvXz+TlpZmWrRoYX784x+bTZs2VVpRU92qn5EjR1Z6zYEDB5qBAwfaX1e36uf4Oqs7z86dO82YMWNMo0aNTEZGhrn66qvNq6++WuUqkuNFzl3dI1LT1q1bzeWXX24yMzONz+czPXr0qLSi6De/+Y3p37+/ad68ufH5fKZ9+/bm5ptvNtu3bzfGGHPs2DEzadIk0717d9O4cWOTmppqOnfubObMmWOOHDlSY50R//znP83YsWNNq1atjMfjMS1btjRjxowx7733XrXHvPHGG/b7+fzzz6vcZ8uWLeaHP/yhadmypfF6vSY7O9sMGTLEPP744/Y+kVU/GzZsqFOtta36GT9+vL1v5DO4evVq06dPH+P3+03r1q3NnXfeWWk10oEDB8ykSZNM69atjcfjMR06dDCzZs0yx44di9ovGAyaBx54wHTr1s34fD6TmZlp+vXrZ/7+97/b+9T1Mzpz5kzTp08f07RpU+P3+82ZZ55pfvrTn5r9+/fXqS2A+rKMOW6AE8Ap5Z577tFdd92lnTt3cqn2k8CgQYO0f/9+ffTRR06XAiQFhn6AU0jkYmBdunRRSUmJVq5cqYcffljXX389IQXASYmgApxC0tLS9MADD2j79u0KBAJq37697rjjDt11111OlwYAMWHoBwAAJC2WJwMAgKRFUAEAAEmLoAIAAJLWST2ZNhQKadeuXcrIyGiwS1kDAID4MsaosLBQbdq0qfVilCd1UNm1a1elu7gCAICTQ15eXq2XTjipg0rkvht5eXlq3Lixw9UAAIC6KCgoUE5OTp3un3VSB5XIcE/jxo0JKgAAnGTqMm2DybQAACBpEVQAAEDSIqgAAICkRVABAABJi6ACAACSFkEFAAAkLYIKAABIWgQVAACQtAgqAAAgaRFUAABA0iKoAACApEVQAQAASeukvilhQykqDurg0WJ5XJZaNU5xuhwAAE5b9KhUYfnHu3XRvSt1+5ItTpcCAMBpjaBSBZ/bLUkKlIYcrgQAgNMbQaUKPk+4WYoJKgAAOIqgUgWv25JEUAEAwGkElSpEelRKggQVAACcRFCpgj8y9ENQAQDAUQSVKkQm0zL0AwCAswgqVfB6mKMCAEAyIKhUwedm6AcAgGRAUKkCy5MBAEgOBJUq+CpMpjXGOFwNAACnL4JKFSJDP8ZIpSGCCgAATiGoVCHSoyJxLRUAAJxEUKlCpEdFYp4KAABOIqhUweN2yRVeoUxQAQDAQQSVakSGf7iDMgAAziGoVMPr5n4/AAA4jaBSDe73AwCA8wgq1bCvTsvQDwAAjiGoVIOr0wIA4DyCSjW89KgAAOA4gko1fMxRAQDAcQSVajD0AwCA8wgq1bAn09KjAgCAYwgq1aBHBQAA5xFUquHjgm8AADiOoFINelQAAHBe0gSV3NxcWZal6dOnO12KJO71AwBAMkiKoLJhwwY9+eST6t69u9Ol2LxMpgUAwHGOB5XDhw9r/Pjxeuqpp9S0aVOny7FFelRKSo3DlQAAcPpyPKhMnjxZI0eO1Pe+9z2nS4lSvjw56HAlAACcvjxOnnzx4sXatGmTNmzYUKf9A4GAAoGA/XVBQUFDlVZ+92TmqAAA4BjHelTy8vI0bdo0Pffcc0pJSanTMbm5ucrMzLQfOTk5DVYfq34AAHCeY0Fl48aN2rdvn3r37i2PxyOPx6M1a9bo4YcflsfjUbCKIZdZs2YpPz/ffuTl5TVYfeWTaZmjAgCAUxwb+hk6dKi2bt0ate1HP/qRunTpojvuuENut7vSMX6/X36/PyH10aMCAIDzHAsqGRkZ6tatW9S29PR0NWvWrNJ2J3CvHwAAnOf4qp9kVd6jwqofAACc4uiqn+OtXr3a6RJsdo8KQz8AADiGHpVq2Bd8YzItAACOIahUg8m0AAA4j6BSjcjQT4DJtAAAOIagUg0vPSoAADiOoFKNSI9KCT0qAAA4hqBSDeaoAADgPIJKNbgpIQAAziOoVMPLlWkBAHAcQaUa9nVU6FEBAMAxBJVqRIIKy5MBAHAOQaUaFS+hbwxXpwUAwAkElWpEgorEZfQBAHAKQaUakaEfiWupAADgFIJKNSoGFZYoAwDgDIJKNdwuS26XJYklygAAOIWgUoOKE2oBAEDiEVRq4HXTowIAgJMIKjXwedyS6FEBAMApBJUacL8fAACcRVCpgX0HZYZ+AABwBEGlBvYcFXpUAABwBEGlBvSoAADgLIJKDVieDACAswgqNfAxmRYAAEcRVGrgpUcFAABHEVRqEFmezE0JAQBwBkGlBkymBQDAWQSVGjCZFgAAZxFUahCZoxIgqAAA4AiCSg18zFEBAMBRBJUasDwZAABnEVRqQFABAMBZBJUa+N2s+gEAwEkElRpEJtMyRwUAAGcQVGoQGfph1Q8AAM4gqNSAOSoAADiLoFIDggoAAM4iqNTAy2RaAAAcRVCpATclBADAWQSVGnCvHwAAnEVQqQFzVAAAcBZBpQbclBAAAGcRVGrATQkBAHAWQaUG9tAPQQUAAEcQVGrAZFoAAJxFUKkBk2kBAHAWQaUGPvumhMbhSgAAOD0RVGpAjwoAAM4iqNSg4mRaY+hVAQAg0QgqNYhcR0Vi5Q8AAE4gqNQgcq8fiXkqAAA4gaBSA1/FHhXmqQAAkHAElRq4XJY8LksSQQUAACcQVGrByh8AAJxDUKlFZEItk2kBAEg8gkot6FEBAMA5BJVa+OhRAQDAMQSVWvjpUQEAwDEElVp4uYMyAACOIajUIjJHpYShHwAAEo6gUotIUAnQowIAQMIRVGrBZFoAAJxDUKmFl8m0AAA4hqBSi0iPCnNUAABIPIJKLVieDACAcwgqteDKtAAAOIegUguvu+zuyQz9AACQcASVWtCjAgCAcwgqtfC53ZLoUQEAwAmOBpUFCxaoe/fuaty4sRo3bqx+/frptddec7KkSuhRAQDAOY4GlXbt2unee+/V+++/r/fff19DhgzRlVdeqY8//tjJsqL4InNUCCoAACScx8mTX3755VFf33333VqwYIHWrVunrl27OlRVNO71AwCAcxwNKhUFg0EtWbJER44cUb9+/arcJxAIKBAI2F8XFBQ0eF0M/QAA4BzHJ9Nu3bpVjRo1kt/v16RJk/Tyyy/r3HPPrXLf3NxcZWZm2o+cnJwGry9yZdoAPSoAACSc40Glc+fO2rx5s9atW6dbb71VEyZM0CeffFLlvrNmzVJ+fr79yMvLa/D6fJ6yVT/0qAAAkHCOD/34fD516tRJktSnTx9t2LBBDz30kJ544olK+/r9fvn9/oTW52UyLQAAjnG8R+V4xpioeShOYzItAADOcbRH5c4779Tw4cOVk5OjwsJCLV68WKtXr9by5cudLCsKNyUEAMA5jgaVvXv36oYbbtDu3buVmZmp7t27a/ny5fr+97/vZFlR7FU/9KgAAJBwjgaVp59+2snT14nXTY8KAABOSbo5KskmsjyZHhUAABKPoFILLvgGAIBzCCq1IKgAAOAcgkotGPoBAMA5BJVa2NdRoUcFAICEI6jUguXJAAA4h6BSi8jQT0nQKBQyDlcDAMDphaBSC6+nvInoVQEAILEIKrWI9KhI3O8HAIBEI6jUomJQYYkyAACJRVCphctlyeu2JDH0AwBAohFU6sDH/X4AAHAEQaUOIhNqmaMCAEBiEVTqINKjEqBHBQCAhCKo1AH3+wEAwBkElTogqAAA4AyCSh1wY0IAAJxBUKkDH5NpAQBwBEGlDlieDACAMwgqdRDpUWHVDwAAiUVQqQMvPSoAADiCoFIH5XNUjMOVAABweiGo1EH58uSgw5UAAHB6IajUgZ/lyQAAOIKgUgfMUQEAwBkElTqwh36YowIAQEIRVOqAS+gDAOAMgkodEFQAAHAGQaUO7DkqQVb9AACQSASVOvBHrqNSyhwVAAASiaBSB9w9GQAAZxBU6oA5KgAAOIOgUgfclBAAAGcQVOrAy9APAACOIKjUgX1TQnpUAABIKIJKHTCZFgAAZxBU6sDPZFoAABxBUKkDbkoIAIAzCCp1YM9RYegHAICEiimo5OXl6euvv7a/Xr9+vaZPn64nn3wyboUlE5YnAwDgjJiCynXXXadVq1ZJkvbs2aPvf//7Wr9+ve68807NmzcvrgUmAybTAgDgjJiCykcffaS+fftKkv785z+rW7dueu+99/TCCy9o0aJF8awvKfg8liTmqAAAkGgxBZWSkhL5/X5J0ptvvqkrrrhCktSlSxft3r07ftUlCZ/bLYk5KgAAJFpMQaVr1656/PHH9fbbb2vFihW67LLLJEm7du1Ss2bN4lpgMuBePwAAOCOmoHLffffpiSee0KBBgzRu3Dj16NFDkrRs2TJ7SOhUEgkqpSGjUMg4XA0AAKcPTywHDRo0SPv371dBQYGaNm1qb7/llluUlpYWt+KShddt2X8vDoaU4nI7WA0AAKePmHpUioqKFAgE7JCyY8cOPfjgg9q2bZtatmwZ1wKTQaRHRWLlDwAAiRRTULnyyiv17LPPSpIOHTqkCy64QL/5zW80evRoLViwIK4FJoPI8mSJeSoAACRSTEFl06ZNuvjiiyVJf/nLX9SqVSvt2LFDzz77rB5++OG4FpgMLMsqv5YKQQUAgISJKagcPXpUGRkZkqQ33nhDY8aMkcvl0oUXXqgdO3bEtcBkEZmnQlABACBxYgoqnTp10tKlS5WXl6fXX39dw4YNkyTt27dPjRs3jmuByYL7/QAAkHgxBZVf/vKXuv3223XGGWeob9++6tevn6Rw70qvXr3iWmCy4H4/AAAkXkzLk8eOHasBAwZo9+7d9jVUJGno0KG66qqr4lZcMrEv+kaPCgAACRNTUJGk7OxsZWdn6+uvv5ZlWWrbtu0pebG3CCbTAgCQeDEN/YRCIc2bN0+ZmZnq0KGD2rdvryZNmujXv/61QqFT8xe5l6ACAEDCxdSjMnv2bD399NO69957ddFFF8kYo3fffVdz587VsWPHdPfdd8e7Tsf5veGr0TJHBQCAxIkpqPzhD3/Q73//e/uuyZLUo0cPtW3bVrfddtspGVRSveEelWMlQYcrAQDg9BHT0M/BgwfVpUuXStu7dOmigwcPnnBRySi1rEeliKACAEDCxBRUevTooUcffbTS9kcffVTdu3c/4aKSUZov3PlUVExQAQAgUWIa+rn//vs1cuRIvfnmm+rXr58sy9J7772nvLw8vfrqq/GuMSmk0KMCAEDCxdSjMnDgQH3++ee66qqrdOjQIR08eFBjxozRxx9/rIULF8a7xqSQ6gs3FT0qAAAkTszXUWnTpk2lSbNbtmzRH/7wBz3zzDMnXFiysYd+6FEBACBhYupROR3ZQz/0qAAAkDAElTpi1Q8AAIlHUKmjyHVUCCoAACROveaojBkzpsbnDx06dCK1JDWWJwMAkHj1CiqZmZm1Pn/jjTeeUEHJKsXHHBUAABKtXkHlVF16XBfMUQEAIPGYo1JHafSoAACQcI4GldzcXJ1//vnKyMhQy5YtNXr0aG3bts3JkqrFlWkBAEg8R4PKmjVrNHnyZK1bt04rVqxQaWmphg0bpiNHjjhZVpUY+gEAIPFivjJtPCxfvjzq64ULF6ply5bauHGjLrnkEoeqqloqQz8AACSco0HlePn5+ZKkrKysKp8PBAIKBAL21wUFBQmpS6owR6UkKGOMLMtK2LkBADhdJc1kWmOMZsyYoQEDBqhbt25V7pObm6vMzEz7kZOTk7D6InNUgiGjkqBJ2HkBADidJU1QmTJlij788EP96U9/qnafWbNmKT8/337k5eUlrL7IHBWJeSoAACRKUgz9TJ06VcuWLdPatWvVrl27avfz+/3y+/0JrKycz+OSx2WpNGRUVBxUZqrXkToAADidONqjYozRlClT9NJLL2nlypXq2LGjk+XUipU/AAAklqM9KpMnT9YLL7ygv/3tb8rIyNCePXskhS/Fn5qa6mRpVUrxuVUYKGXlDwAACeJoj8qCBQuUn5+vQYMGqXXr1vbjxRdfdLKsapX3qJQ6XAkAAKcHR3tUjDm5Vs+UX0Y/5HAlAACcHpJm1c/JgMvoAwCQWASVemAyLQAAiUVQqYfyoR/mqAAAkAgElXpI4X4/AAAkFEGlHsqHfphMCwBAIhBU6sEOKgz9AACQEASVeqh4B2UAANDwCCr1wPJkAAASi6BSD6lc8A0AgIQiqNRD+dAPc1QAAEgEgko92EM/LE8GACAhCCr1wJVpAQBILIJKPaTSowIAQEIRVOqB5ckAACQWQaUeUggqAAAkFEGlHhj6AQAgsQgq9ZDGTQkBAEgogko9VFz1Y4xxuBoAAE59BJV6iMxRCRmpOMjVaQEAaGgElXqI9KhIDP8AAJAIBJV68Lpd8rotSaz8AQAgEQgq9cRl9AEASByCSj1Fhn+OElQAAGhwBJV6iixRPsbQDwAADY6gUk8p3JgQAICEIajUUyoXfQMAIGEIKvWUSo8KAAAJQ1CpJy6jDwBA4hBU6ok5KgAAJA5BpZ5YngwAQOIQVOqJ5ckAACQOQaWeUpijAgBAwhBU6ske+qFHBQCABkdQqadIUDlGjwoAAA2OoFJP9vJkelQAAGhwBJV6YnkyAACJQ1Cpp8gl9FmeDABAwyOo1BPLkwEASByCSj3ZQz/0qAAA0OAIKvXElWkBAEgcgko9pTL0AwBAwhBU6inN65HEqh8AABKBoFJPKb5wkxWVBGWMcbgaAABObQSVeorMUTFGCpSGHK4GAIBTG0GlniJBRWLlDwAADY2gUk8et0s+d/nwDwAAaDgElRikeMPNxhJlAAAaFkElBixRBgAgMQgqMUjzsUQZAIBEIKjEIIWr0wIAkBAElRikls1RYdUPAAANi6ASg8jQD3NUAABoWASVGNh3UCaoAADQoAgqMYis+mGOCgAADYugEoPIHBWGfgAAaFgElRjYy5PpUQEAoEERVGLA8mQAABKDoBKDVCbTAgCQEASVGKRxCX0AABKCoBKDlLKgwhwVAAAaFkElBpGhn6P0qAAA0KAIKjGwh37oUQEAoEERVGLAZFoAABKDoBKD8uXJpQ5XAgDAqY2gEoNUe9VPyOFKAAA4tRFUYhCZo8LQDwAADYugEoNUhn4AAEgIgkoMInNUjpWEFAoZh6sBAODU5WhQWbt2rS6//HK1adNGlmVp6dKlTpZTZ5GhH0kKlDJPBQCAhuJoUDly5Ih69OihRx991Mky6i3SoyIxTwUAgIbkcfLkw4cP1/Dhw50sISZulyWfx6Xi0pCOFpcqK93ndEkAAJySHA0q9RUIBBQIBOyvCwoKHKsl1etWcWmIGxMCANCATqrJtLm5ucrMzLQfOTk5jtViL1EuZo4KAAAN5aQKKrNmzVJ+fr79yMvLc6wWligDANDwTqqhH7/fL7/f73QZkson1DKZFgCAhnNS9agkE/sOygQVAAAajKM9KocPH9aXX35pf/3VV19p8+bNysrKUvv27R2srHapXEYfAIAG52hQef/99zV48GD76xkzZkiSJkyYoEWLFjlUVd2U30GZoAIAQENxNKgMGjRIxpycl6CPTKYtIqgAANBgmKMSI+aoAADQ8AgqMWLoBwCAhkdQiRGTaQEAaHgElRileRn6AQCgoRFUYmT3qDD0AwBAgyGoxIg5KgAANDyCSoxSuYQ+AAANjqASI5YnAwDQ8AgqMUrxMfQDAEBDI6jEKLLq50ig1OFKAAA4dRFUYtSmSaokadehYwqGTs7bAAAAkOwIKjFq0yRVXrel4mBIu/OLnC4HAIBTEkElRm6XpZysNEnSjgNHHa4GAIBTE0HlBJzRLF2StP3AEYcrAQDg1ERQOQEdmtGjAgBAQyKonICOzcM9Kl/tp0cFAICGQFA5AR3Khn52MPQDAECDIKicgDMqDP2EWKIMAEDcEVROQNsmqfK4LAVKQ9pbeMzpcgAAOOUQVE6Ax+2ylyhv38+EWgAA4o2gcoIiK39YogwAQPwRVE4Q11IBAKDhEFROkH0tFYZ+AACIO4LKCTqjOT0qAAA0FILKCTrDvpbKURnDEmUAAOKJoHKC2jZJldtlqagkqH2FAafLAQDglEJQOUE+j0ttm6RKkrZzKX0AAOKKoBIHkXkq3JwQAID4IqjEwRlcSwUAgAZBUImDDlxLBQCABkFQiQO7R4VrqQAAEFcElTgon6NyhCXKAADEEUElDto1TZXLko4UB7X/cLHT5QAAcMogqMSB3+NWm8gSZeapAAAQNwSVOLFvTsi1VAAAiBuCSpyc0bzs5oRcSwUAgLghqMTJGSxRBgAg7ggqcdKhGVenBQAg3ggqcVJ+LRWWKAMAEC8ElTjJyUqTZUmFgVIdPMISZQAA4oGgEicpXrfaZEaWKDP8AwBAPBBU4qhD2fDPBzu/dbgSAABODQSVOLq0a7Yk6aE3v9Ce/GMOVwMAwMmPoBJH11/YQT1zmqgwUKrZL29lUi0AACeIoBJHbpel/ze2u3xul976bJ+WbdnldEkAAJzUCCpxdlarDE0d0kmSNHfZx9p/OOBwRQAAnLwIKg1g0qDv6NzWjfXt0RLNWfax0+UAAHDSIqg0AK/bpfvHdpfbZemVD3dr+Ud7nC4JAICTEkGlgXRrm6lJA8+UJE1/8QM98tYXOlYSdLgqAABOLgSVBjR1yFka0Km5jpWE9JsVn2vYA2u14pO9rAYCAKCOLHMS/9YsKChQZmam8vPz1bhxY6fLqZIxRsu27NI9r36qvQXhibUDz26hCf07qP93mivF63a4QgAAEqs+v78JKglyJFCqR1Z+qaff+bdKguEmT/O5NfDsFhrWtZUuPquFmjfyO1wlAAANj6CSxP79zWEtfHe7VnyyV3sKoq9e27yRX52zG6lzq8Y6u1Uj5WSlqU2TVLXOTKHnBQBwyiConASMMdr6n3y98fFerfhkr7btLaxx/6x0n1o1TlGLDL+aN/KpRSO/mjfyq2m6T01SvcpM81b40yefh+lHAIDkRFA5CR0JlOqLfYf1+Z5CbdtbqM/3FmrXoSLtzj+mo8X1Xy3UyO9RkzSvmqb5lJHiUbrfo0Z+j9J8bjVK8Sgz1Vvp0TjFq8apXjVO8cjjJugAABpGfX5/exJUE2qR7veoZ04T9cxpErXdGKOColLtyi/S3oJj2n+4WN8UBvRNYUD7Dwd0qKhE+UeLlV9UEv57UYmMkQ4HSnU4UKqvvy2KqZ40n1tpPrdSfW6lesMPv9ctv8clv8ctv9clv8elVG9kP4/Sy45J8YaPi/zd73HJ53bL53HJ5yk/LtUXfs6yrDi0IADgVERQSXKWZSkzLTykc07r2nuNQiGjgmMl+vZoib49WqxvjxTrcKBURwJBHSkLL4cDpcovCzX5RSUqqPDnkbLem6PFwZh6curLshQOQR6XvO7ww+dxyeu25Ha55HZJbpdLHpclj8uS3+uWz+0qC0wu+b0upXjDgSjFEw5DHpclt8uSxx3+01u2v9ftks9dHpYiwStyjKssMLlcllyW5Lassr9bcluWHc4IVgCQOASVU4zLZalJmk9N0nzqqPR6H18aDKngWKkKikpUVBJUUUlQx8pCS6A0pEBpUMWlIQVKQzpWEt5eVBLU0eLScLgJBO3jiorD24uDIRWXhlQSNGXHBu2VT8YkLhTFi98TDkc+j0vGGIWMFAwZhUJGlqWy4OOSx22FQ5XXrRRvuBcpxeuW22UpGDLhY4yRMWWv6XMrraynyed2ybLCQdWyJEvhoOZxR/50yWVJwZAUMkalQaOgMXJblrweS15XOOx53OWhz+u25HGFXzdUVncoZGRkwmHMZdnhLPxn+Pwuy5Kl8E03Iw+XFQ5zVtlzkRrDwbA8ILpd5fWH3084AHpcLrnd5edx2a8Tfl1jKtRY1kaWpagaAZweCCqI4nG7lJXuU1a6r0HPUxIMB51IoCkJhlRcasJ/BkMqKQ0paIxKywJASdCoNBRSoCRkB59AaVDHSsKvc6wkpKKS8OtEQkAwFH698tc0CgRDCpQEVRwsf61jJcEKvxiN/fdwkKhce6AsqME5kcDjKgtykeBk94C5yp8zFb6nke9nxal5lmXZvXN+Tzgket3hMOSp8HqRjrRwpFL55ytkVFr2uTuex22Fh0rLevEiITYcBsvrrSp2RQJeRMiEw2iowucyvI/sXr7SslpKgkbBUEguy6rQSxkJr+GgWLH3sSrGSEbl7yk6xEYHUEnh9rcs+zUjYTISiCP/poyRgpF/X2XvxajsPZnwGSO9q163JZ8nHHgj3wuPO/K9tcq+H7JrOL5+STIVvt+WZclbFvQ97nCgNyr/T0NpsKyesjoq/jyo2CbGlIfpyOtbFd57JKT7jm97l0sulyqEfavCecM/78KfIxP1HqoSCfWR14m0YajsM2IqfE8i54u8ZtD+WWfkcUV6ssN1u6pozFSf29HLZxBU4IjID6KMFK/TpdQqFAr/AImEmqLioAKl4R6mij8EXFb4h3JpKKTSYCQkmbIgVdY7VRJUyET/ALEsqbg0FO6BKuvBCgRDkin/IRvptYm8diTAVfzF4XaFe1hKgyG73tJg+f6loZDdk1XxB5xUMZiVvbaJ/gUfrPCLxu5BKnteKv/BHvlBW1J23mBZj40pey8VjzkRxkilpvwHOoCGc0WPNnp4XC/Hzk9QAWrhclnyucL/s2vk55/MiTJlgScSaoIm8j/U8mAUCVFW2bCQVP4/xfJhs+hesEh4qrg9FFJ4CEuRoSpJZb0Ukd6IkDEKlIbsIc1AaUjBskAY/b/cSP3h/1W7XS55y/737KkwrFZxv9JQSMdKQnawDZSGFAyGFDTl7XB8R4yp5n/TkTAcDreR/x1H71txaNDtsmSMUXHZkGtJWU9kaYVeoJKgqfKWHhXnYVkVhuIq9laGa1VUWK3YC1oaMrIidUeG96zyniS3Vd7zYu9XdtqS0HE1l71m0Bx3fhPdXsd3BlgVv9eWFArJDuyR0C+p0pCm/fkrO87+e1QPUvTwp5GiAnpxMGSfo7isZzfyXMXQHzlveU9UhaHS474X5Z+t8h6fUKj830zF3kVJUT0sQWMq/Dso70mL1FpS1pNdWjaEXbHtnL7cBT91ASSUVdZ97+EahgDqgItlAACApEVQAQAASYugAgAAkhZBBQAAJC2CCgAASFoEFQAAkLQIKgAAIGk5HlR+97vfqWPHjkpJSVHv3r319ttvO10SAABIEo4GlRdffFHTp0/X7Nmz9cEHH+jiiy/W8OHDtXPnTifLAgAAScIyVV0/OUEuuOACnXfeeVqwYIG97ZxzztHo0aOVm5tb6/EFBQXKzMxUfn6+Gjdu3JClAgCAOKnP72/HelSKi4u1ceNGDRs2LGr7sGHD9N5771V5TCAQUEFBQdQDAACcuhwLKvv371cwGFSrVq2itrdq1Up79uyp8pjc3FxlZmbaj5ycnESUCgAAHOL4ZNrj7wxpjKnybpGSNGvWLOXn59uPvLy8RJQIAAAc4tjdk5s3by63212p92Tfvn2Velki/H6//H5/IsoDAABJwLGg4vP51Lt3b61YsUJXXXWVvX3FihW68sor6/QakXnAzFUBAODkEfm9XZf1PI4FFUmaMWOGbrjhBvXp00f9+vXTk08+qZ07d2rSpEl1Or6wsFCSmKsCAMBJqLCwUJmZmTXu42hQueaaa3TgwAHNmzdPu3fvVrdu3fTqq6+qQ4cOdTq+TZs2ysvLU0ZGRrXzWmJVUFCgnJwc5eXlsfS5gdHWiUNbJw5tnTi0deLEq62NMSosLFSbNm1q3dfR66gkM67Rkji0deLQ1olDWycObZ04TrS146t+AAAAqkNQAQAASYugUg2/3685c+awHDoBaOvEoa0Th7ZOHNo6cZxoa+aoAACApEWPCgAASFoEFQAAkLQIKgAAIGkRVAAAQNIiqFThd7/7nTp27KiUlBT17t1bb7/9ttMlnfRyc3N1/vnnKyMjQy1bttTo0aO1bdu2qH2MMZo7d67atGmj1NRUDRo0SB9//LFDFZ86cnNzZVmWpk+fbm+jrePnP//5j66//no1a9ZMaWlp6tmzpzZu3Gg/T1vHR2lpqe666y517NhRqampOvPMMzVv3jyFQiF7H9o6NmvXrtXll1+uNm3ayLIsLV26NOr5urRrIBDQ1KlT1bx5c6Wnp+uKK67Q119/HZ8CDaIsXrzYeL1e89RTT5lPPvnETJs2zaSnp5sdO3Y4XdpJ7dJLLzULFy40H330kdm8ebMZOXKkad++vTl8+LC9z7333msyMjLMX//6V7N161ZzzTXXmNatW5uCggIHKz+5rV+/3pxxxhmme/fuZtq0afZ22jo+Dh48aDp06GAmTpxo/vWvf5mvvvrKvPnmm+bLL7+096Gt42P+/PmmWbNm5h//+If56quvzJIlS0yjRo3Mgw8+aO9DW8fm1VdfNbNnzzZ//etfjSTz8ssvRz1fl3adNGmSadu2rVmxYoXZtGmTGTx4sOnRo4cpLS094foIKsfp27evmTRpUtS2Ll26mJkzZzpU0alp3759RpJZs2aNMcaYUChksrOzzb333mvvc+zYMZOZmWkef/xxp8o8qRUWFpqzzjrLrFixwgwcONAOKrR1/Nxxxx1mwIAB1T5PW8fPyJEjzU033RS1bcyYMeb66683xtDW8XJ8UKlLux46dMh4vV6zePFie5///Oc/xuVymeXLl59wTQz9VFBcXKyNGzdq2LBhUduHDRum9957z6GqTk35+fmSpKysLEnSV199pT179kS1vd/v18CBA2n7GE2ePFkjR47U9773vajttHX8LFu2TH369NEPfvADtWzZUr169dJTTz1lP09bx8+AAQP01ltv6fPPP5ckbdmyRe+8845GjBghibZuKHVp140bN6qkpCRqnzZt2qhbt25xaXtH756cbPbv369gMKhWrVpFbW/VqpX27NnjUFWnHmOMZsyYoQEDBqhbt26SZLdvVW2/Y8eOhNd4slu8eLE2bdqkDRs2VHqOto6ff//731qwYIFmzJihO++8U+vXr9d///d/y+/368Ybb6St4+iOO+5Qfn6+unTpIrfbrWAwqLvvvlvjxo2TxOe6odSlXffs2SOfz6emTZtW2icevzsJKlWwLCvqa2NMpW2I3ZQpU/Thhx/qnXfeqfQcbX/i8vLyNG3aNL3xxhtKSUmpdj/a+sSFQiH16dNH99xzjySpV69e+vjjj7VgwQLdeOON9n609Yl78cUX9dxzz+mFF15Q165dtXnzZk2fPl1t2rTRhAkT7P1o64YRS7vGq+0Z+qmgefPmcrvdlRLgvn37KqVJxGbq1KlatmyZVq1apXbt2tnbs7OzJYm2j4ONGzdq37596t27tzwejzwej9asWaOHH35YHo/Hbk/a+sS1bt1a5557btS2c845Rzt37pTE5zqe/ud//kczZ87Utddeq+9+97u64YYb9NOf/lS5ubmSaOuGUpd2zc7OVnFxsb799ttq9zkRBJUKfD6fevfurRUrVkRtX7Fihfr37+9QVacGY4ymTJmil156SStXrlTHjh2jnu/YsaOys7Oj2r64uFhr1qyh7etp6NCh2rp1qzZv3mw/+vTpo/Hjx2vz5s0688wzaes4ueiiiyots//888/VoUMHSXyu4+no0aNyuaJ/Zbndbnt5Mm3dMOrSrr1795bX643aZ/fu3froo4/i0/YnPB33FBNZnvz000+bTz75xEyfPt2kp6eb7du3O13aSe3WW281mZmZZvXq1Wb37t324+jRo/Y+9957r8nMzDQvvfSS2bp1qxk3bhxLC+Ok4qofY2jreFm/fr3xeDzm7rvvNl988YV5/vnnTVpamnnuuefsfWjr+JgwYYJp27atvTz5pZdeMs2bNzc///nP7X1o69gUFhaaDz74wHzwwQdGkvntb39rPvjgA/uyHHVp10mTJpl27dqZN99802zatMkMGTKE5ckN6bHHHjMdOnQwPp/PnHfeefYSWsROUpWPhQsX2vuEQiEzZ84ck52dbfx+v7nkkkvM1q1bnSv6FHJ8UKGt4+fvf/+76datm/H7/aZLly7mySefjHqeto6PgoICM23aNNO+fXuTkpJizjzzTDN79mwTCATsfWjr2KxatarKn88TJkwwxtStXYuKisyUKVNMVlaWSU1NNaNGjTI7d+6MS32WMcaceL8MAABA/DFHBQAAJC2CCgAASFoEFQAAkLQIKgAAIGkRVAAAQNIiqAAAgKRFUAEAAEmLoALglGJZlpYuXep0GQDihKACIG4mTpwoy7IqPS677DKnSwNwkvI4XQCAU8tll12mhQsXRm3z+/0OVQPgZEePCoC48vv9ys7Ojno0bdpUUnhYZsGCBRo+fLhSU1PVsWNHLVmyJOr4rVu3asiQIUpNTVWzZs10yy236PDhw1H7PPPMM+ratav8fr9at26tKVOmRD2/f/9+XXXVVUpLS9NZZ52lZcuWNeybBtBgCCoAEuoXv/iFrr76am3ZskXXX3+9xo0bp08//VSSdPToUV122WVq2rSpNmzYoCVLlujNN9+MCiILFizQ5MmTdcstt2jr1q1atmyZOnXqFHWOX/3qV/rhD3+oDz/8UCNGjND48eN18ODBhL5PAHESl1sbAoAxZsKECcbtdpv09PSox7x584wx4btoT5o0KeqYCy64wNx6663GGGOefPJJ07RpU3P48GH7+VdeecW4XC6zZ88eY4wxbdq0MbNnz662Bknmrrvusr8+fPiwsSzLvPbaa3F7nwAShzkqAOJq8ODBWrBgQdS2rKws++/9+vWLeq5fv37avHmzJOnTTz9Vjx49lJ6ebj9/0UUXKRQKadu2bbIsS7t27dLQoUNrrKF79+7239PT05WRkaF9+/bF+pYAOIigAiCu0tPTKw3F1MayLEmSMcb+e1X7pKam1un1vF5vpWNDoVC9agKQHJijAiCh1q1bV+nrLl26SJLOPfdcbd68WUeOHLGff/fdd+VyuXT22WcrIyNDZ5xxht56662E1gzAOfSoAIirQCCgPXv2RG3zeDxq3ry5JGnJkiXq06ePBgwYoOeff17r16/X008/LUkaP3685syZowkTJmju3Ln65ptvNHXqVN1www1q1aqVJGnu3LmaNGmSWrZsqeHDh6uwsFDvvvuupk6dmtg3CiAhCCoA4mr58uVq3bp11LbOnTvrs88+kxRekbN48WLddtttys7O1vPPP69zzz1XkpSWlqbXX39d06ZN0/nnn6+0tDRdffXV+u1vf2u/1oQJE3Ts2DE98MADuv3229W8eXONHTs2cW8QQEJZxhjjdBEATg+WZenll1/W6NGjnS4FwEmCOSoAACBpEVQAAEDSYo4KgIRhpBlAfdGjAgAAkhZBBQAAJC2CCgAASFoEFQAAkLQIKgAAIGkRVAAAQNIiqAAAgKRFUAEAAEmLoAIAAJLW/wc8ItPrbeLCKQAAAABJRU5ErkJggg==", 702 | "text/plain": [ 703 | "
" 704 | ] 705 | }, 706 | "metadata": {}, 707 | "output_type": "display_data" 708 | } 709 | ], 710 | "source": [ 711 | "import matplotlib.pyplot as plt\n", 712 | "\n", 713 | "plt.plot(losses)\n", 714 | "plt.xlabel('Epoch')\n", 715 | "plt.ylabel('Loss')\n", 716 | "plt.title('Training Loss Over Epochs')\n", 717 | "plt.show()\n", 718 | "\n", 719 | "# 보충: validation loss를 같이 그려서 비교하는 사례 https://www.geeksforgeeks.org/training-and-validation-loss-in-deep-learning/" 720 | ] 721 | }, 722 | { 723 | "cell_type": "markdown", 724 | "metadata": {}, 725 | "source": [ 726 | "#### 결과 확인" 727 | ] 728 | }, 729 | { 730 | "cell_type": "code", 731 | "execution_count": 18, 732 | "metadata": {}, 733 | "outputs": [ 734 | { 735 | "data": { 736 | "text/plain": [ 737 | "GPTModel(\n", 738 | " (tok_emb): Embedding(50257, 768)\n", 739 | " (pos_emb): Embedding(128, 768)\n", 740 | " (drop_emb): Dropout(p=0.1, inplace=False)\n", 741 | " (trf_blocks): Sequential(\n", 742 | " (0): TransformerBlock(\n", 743 | " (att): MultiHeadAttention(\n", 744 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 745 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 746 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 747 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 748 | " (dropout): Dropout(p=0.1, inplace=False)\n", 749 | " )\n", 750 | " (ff): FeedForward(\n", 751 | " (layers): Sequential(\n", 752 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 753 | " (1): GELU()\n", 754 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 755 | " )\n", 756 | " )\n", 757 | " (norm1): LayerNorm()\n", 758 | " (norm2): LayerNorm()\n", 759 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 760 | " )\n", 761 | " (1): TransformerBlock(\n", 762 | " (att): MultiHeadAttention(\n", 763 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 764 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 765 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 766 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 767 | " (dropout): Dropout(p=0.1, inplace=False)\n", 768 | " )\n", 769 | " (ff): FeedForward(\n", 770 | " (layers): Sequential(\n", 771 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 772 | " (1): GELU()\n", 773 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 774 | " )\n", 775 | " )\n", 776 | " (norm1): LayerNorm()\n", 777 | " (norm2): LayerNorm()\n", 778 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 779 | " )\n", 780 | " (2): TransformerBlock(\n", 781 | " (att): MultiHeadAttention(\n", 782 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 783 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 784 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 785 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 786 | " (dropout): Dropout(p=0.1, inplace=False)\n", 787 | " )\n", 788 | " (ff): FeedForward(\n", 789 | " (layers): Sequential(\n", 790 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 791 | " (1): GELU()\n", 792 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 793 | " )\n", 794 | " )\n", 795 | " (norm1): LayerNorm()\n", 796 | " (norm2): LayerNorm()\n", 797 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 798 | " )\n", 799 | " (3): TransformerBlock(\n", 800 | " (att): MultiHeadAttention(\n", 801 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 802 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 803 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 804 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 805 | " (dropout): Dropout(p=0.1, inplace=False)\n", 806 | " )\n", 807 | " (ff): FeedForward(\n", 808 | " (layers): Sequential(\n", 809 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 810 | " (1): GELU()\n", 811 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 812 | " )\n", 813 | " )\n", 814 | " (norm1): LayerNorm()\n", 815 | " (norm2): LayerNorm()\n", 816 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 817 | " )\n", 818 | " (4): TransformerBlock(\n", 819 | " (att): MultiHeadAttention(\n", 820 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 821 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 822 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 823 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 824 | " (dropout): Dropout(p=0.1, inplace=False)\n", 825 | " )\n", 826 | " (ff): FeedForward(\n", 827 | " (layers): Sequential(\n", 828 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 829 | " (1): GELU()\n", 830 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 831 | " )\n", 832 | " )\n", 833 | " (norm1): LayerNorm()\n", 834 | " (norm2): LayerNorm()\n", 835 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 836 | " )\n", 837 | " (5): TransformerBlock(\n", 838 | " (att): MultiHeadAttention(\n", 839 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 840 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 841 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 842 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 843 | " (dropout): Dropout(p=0.1, inplace=False)\n", 844 | " )\n", 845 | " (ff): FeedForward(\n", 846 | " (layers): Sequential(\n", 847 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 848 | " (1): GELU()\n", 849 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 850 | " )\n", 851 | " )\n", 852 | " (norm1): LayerNorm()\n", 853 | " (norm2): LayerNorm()\n", 854 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 855 | " )\n", 856 | " (6): TransformerBlock(\n", 857 | " (att): MultiHeadAttention(\n", 858 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 859 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 860 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 861 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 862 | " (dropout): Dropout(p=0.1, inplace=False)\n", 863 | " )\n", 864 | " (ff): FeedForward(\n", 865 | " (layers): Sequential(\n", 866 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 867 | " (1): GELU()\n", 868 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 869 | " )\n", 870 | " )\n", 871 | " (norm1): LayerNorm()\n", 872 | " (norm2): LayerNorm()\n", 873 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 874 | " )\n", 875 | " (7): TransformerBlock(\n", 876 | " (att): MultiHeadAttention(\n", 877 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 878 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 879 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 880 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 881 | " (dropout): Dropout(p=0.1, inplace=False)\n", 882 | " )\n", 883 | " (ff): FeedForward(\n", 884 | " (layers): Sequential(\n", 885 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 886 | " (1): GELU()\n", 887 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 888 | " )\n", 889 | " )\n", 890 | " (norm1): LayerNorm()\n", 891 | " (norm2): LayerNorm()\n", 892 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 893 | " )\n", 894 | " (8): TransformerBlock(\n", 895 | " (att): MultiHeadAttention(\n", 896 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 897 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 898 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 899 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 900 | " (dropout): Dropout(p=0.1, inplace=False)\n", 901 | " )\n", 902 | " (ff): FeedForward(\n", 903 | " (layers): Sequential(\n", 904 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 905 | " (1): GELU()\n", 906 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 907 | " )\n", 908 | " )\n", 909 | " (norm1): LayerNorm()\n", 910 | " (norm2): LayerNorm()\n", 911 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 912 | " )\n", 913 | " (9): TransformerBlock(\n", 914 | " (att): MultiHeadAttention(\n", 915 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 916 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 917 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 918 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 919 | " (dropout): Dropout(p=0.1, inplace=False)\n", 920 | " )\n", 921 | " (ff): FeedForward(\n", 922 | " (layers): Sequential(\n", 923 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 924 | " (1): GELU()\n", 925 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 926 | " )\n", 927 | " )\n", 928 | " (norm1): LayerNorm()\n", 929 | " (norm2): LayerNorm()\n", 930 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 931 | " )\n", 932 | " (10): TransformerBlock(\n", 933 | " (att): MultiHeadAttention(\n", 934 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 935 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 936 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 937 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 938 | " (dropout): Dropout(p=0.1, inplace=False)\n", 939 | " )\n", 940 | " (ff): FeedForward(\n", 941 | " (layers): Sequential(\n", 942 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 943 | " (1): GELU()\n", 944 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 945 | " )\n", 946 | " )\n", 947 | " (norm1): LayerNorm()\n", 948 | " (norm2): LayerNorm()\n", 949 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 950 | " )\n", 951 | " (11): TransformerBlock(\n", 952 | " (att): MultiHeadAttention(\n", 953 | " (W_query): Linear(in_features=768, out_features=768, bias=False)\n", 954 | " (W_key): Linear(in_features=768, out_features=768, bias=False)\n", 955 | " (W_value): Linear(in_features=768, out_features=768, bias=False)\n", 956 | " (out_proj): Linear(in_features=768, out_features=768, bias=True)\n", 957 | " (dropout): Dropout(p=0.1, inplace=False)\n", 958 | " )\n", 959 | " (ff): FeedForward(\n", 960 | " (layers): Sequential(\n", 961 | " (0): Linear(in_features=768, out_features=3072, bias=True)\n", 962 | " (1): GELU()\n", 963 | " (2): Linear(in_features=3072, out_features=768, bias=True)\n", 964 | " )\n", 965 | " )\n", 966 | " (norm1): LayerNorm()\n", 967 | " (norm2): LayerNorm()\n", 968 | " (drop_shortcut): Dropout(p=0.1, inplace=False)\n", 969 | " )\n", 970 | " )\n", 971 | " (final_norm): LayerNorm()\n", 972 | " (out_head): Linear(in_features=768, out_features=50257, bias=False)\n", 973 | ")" 974 | ] 975 | }, 976 | "execution_count": 18, 977 | "metadata": {}, 978 | "output_type": "execute_result" 979 | } 980 | ], 981 | "source": [ 982 | "# 파일로 저장했던 네트워크의 가중치들 읽어들이기\n", 983 | "model.load_state_dict(torch.load(\"model_100.pth\", map_location=device, weights_only=True))\n", 984 | "model.eval() # dropout을 사용하지 않음" 985 | ] 986 | }, 987 | { 988 | "cell_type": "code", 989 | "execution_count": null, 990 | "metadata": {}, 991 | "outputs": [ 992 | { 993 | "name": "stdout", 994 | "output_type": "stream", 995 | "text": [ 996 | "11.15\t 973\t used\n", 997 | "11.10\t 550\t had\n", 998 | "11.04\t 1479\t free\n", 999 | "10.70\t 257\t a\n", 1000 | "10.31\t 1541\t already\n", 1001 | "9.09\t 4978\t caught\n", 1002 | "8.91\t 0\t !\n", 1003 | "8.87\t 2982\t heard\n", 1004 | "8.61\t 460\t can\n", 1005 | "8.40\t 4084\t clearly\n", 1006 | " used\n" 1007 | ] 1008 | } 1009 | ], 1010 | "source": [ 1011 | "idx = tokenizer.encode(\"Dobby is\") # 토큰 id의 list\n", 1012 | "idx = torch.tensor(idx).unsqueeze(0).to(device)\n", 1013 | "\n", 1014 | "with torch.no_grad():\n", 1015 | " logits = model(idx)\n", 1016 | "\n", 1017 | "logits = logits[:, -1, :]\n", 1018 | "\n", 1019 | "# 가장 확률이 높은 단어 10개 출력\n", 1020 | "top_logits, top_indices = torch.topk(logits, 10) \n", 1021 | "for p, i in zip(top_logits.squeeze(0).tolist(), top_indices.squeeze(0).tolist()):\n", 1022 | " print(f\"{p:.2f}\\t {i}\\t {tokenizer.decode([i])}\")\n", 1023 | "\n", 1024 | "# 가장 확률이 높은 단어 출력\n", 1025 | "idx_next = torch.argmax(logits, dim=-1, keepdim=True)\n", 1026 | "flat = idx_next.squeeze(0) # 배치 차원 제거 torch.Size([1])\n", 1027 | "out = tokenizer.decode(flat.tolist()) # 텐서를 리스트로 바꿔서 디코드\n", 1028 | "print(out)" 1029 | ] 1030 | }, 1031 | { 1032 | "cell_type": "code", 1033 | "execution_count": 44, 1034 | "metadata": {}, 1035 | "outputs": [], 1036 | "source": [ 1037 | "def generate(model, idx, max_new_tokens, context_size, temperature=0.0, top_k=None, eos_id=None):\n", 1038 | "\n", 1039 | " for _ in range(max_new_tokens):\n", 1040 | " idx_cond = idx[:, -context_size:]\n", 1041 | " with torch.no_grad():\n", 1042 | " logits = model(idx_cond)\n", 1043 | " logits = logits[:, -1, :]\n", 1044 | "\n", 1045 | " if top_k is not None:\n", 1046 | " top_logits, _ = torch.topk(logits, top_k)\n", 1047 | " min_val = top_logits[:, -1]\n", 1048 | " logits = torch.where(logits < min_val, torch.tensor(float(\"-inf\")).to(logits.device), logits)\n", 1049 | "\n", 1050 | " if temperature > 0.0:\n", 1051 | " logits = logits / temperature\n", 1052 | " probs = torch.softmax(logits, dim=-1) # (batch_size, context_len)\n", 1053 | " idx_next = torch.multinomial(probs, num_samples=1) # (batch_size, 1)\n", 1054 | " else:\n", 1055 | " idx_next = torch.argmax(logits, dim=-1, keepdim=True) # (batch_size, 1)\n", 1056 | "\n", 1057 | " if idx_next == eos_id:\n", 1058 | " break\n", 1059 | "\n", 1060 | " idx = torch.cat((idx, idx_next), dim=1) # (batch_size, num_tokens+1)\n", 1061 | "\n", 1062 | " return idx" 1063 | ] 1064 | }, 1065 | { 1066 | "cell_type": "code", 1067 | "execution_count": null, 1068 | "metadata": {}, 1069 | "outputs": [ 1070 | { 1071 | "name": "stdout", 1072 | "output_type": "stream", 1073 | "text": [ 1074 | "0 : Dobby is had to punish himself, sir,” said the elf, who had gone slightly cross-eyed. “Dobby almost spoke ill of his family, sir, sir, sir, sir, sir, sir, sir, sir, sir,\n", 1075 | "1 : Dobby is had to punish himself, sir,” said the elf, who had gone slightly cross-eyed. “Dobby almost spoke ill of his family, sir. Dobby can’s family, sir, sir, sir, Dobby\n", 1076 | "2 : Dobby is a house-elf.…” said Harry. “Well, whoever owns him will be an old wizarding family, and they’ll be rich, too keen to bew-blood mother died —’ll bewitching\n", 1077 | "3 : Dobby is had to punish himself, sir,” said the elf, who had gone slightly cross-eyed. “Dobby almost spoke ill of his family, sir, sir, sir, sir, sir, sir, sir,’s family\n", 1078 | "4 : Dobby is used to death threats and the best plan we’ve got, so full steam ahead, I say.” However, while Hermione was checking that the bo up and checking his wand id sorry for a Gryffindor Tower, they didn\n", 1079 | "5 : Dobby is a house-elf.…” said Harry. “Well, whoever owns him will be an old wizarding family, and they’ll be rich, too…’ll bewitched Defense Against the evening…’ll\n", 1080 | "6 : Dobby is?” “Locked in the cupboard under the stairs, and I can’t get out of this room —” “No one of this time out of you’ll ever understood where I’s powers\n", 1081 | "7 : Dobby is free.” Lucius Malfoy ripped the sock off the diary, threw it aside, then looked furiously from the ruined book to Harry.”You’ll need ter just heard concern,’ll meet the Weasleys if you�\n", 1082 | "8 : Dobby is used to death threats and. The cat-flap rattled and Aunt Petunias hand appeared, pushing a bowl of canned soup into the room. Harry’s room. Harry, whose insides, and fell: a corner, which knocked\n", 1083 | "9 : Dobby is had to punish himself, sir,” said the elf, who had gone slightly cross-eyed. “Dobby almost spoke ill of his family, sir, sir, sir, sir, sir, sir, sir,’s family\n" 1084 | ] 1085 | } 1086 | ], 1087 | "source": [ 1088 | "start_context = input(\"Start context: \")\n", 1089 | "\n", 1090 | "# idx = tokenizer.encode(start_context, allowed_special={'<|endoftext|>'})\n", 1091 | "idx = tokenizer.encode(start_context)\n", 1092 | "idx = torch.tensor(idx).unsqueeze(0)\n", 1093 | "\n", 1094 | "context_size = model.pos_emb.weight.shape[0] \n", 1095 | "\n", 1096 | "for i in range(10):\n", 1097 | "\n", 1098 | " token_ids = generate(\n", 1099 | " model=model,\n", 1100 | " idx=idx.to(device),\n", 1101 | " max_new_tokens=50,\n", 1102 | " context_size= context_size,\n", 1103 | " top_k=50,\n", 1104 | " temperature=0.5\n", 1105 | " )\n", 1106 | "\n", 1107 | " flat = token_ids.squeeze(0) # remove batch dimension\n", 1108 | " out = tokenizer.decode(flat.tolist()).replace(\"\\n\", \" \")\n", 1109 | "\n", 1110 | " print(i, \":\", out)" 1111 | ] 1112 | }, 1113 | { 1114 | "cell_type": "markdown", 1115 | "metadata": {}, 1116 | "source": [ 1117 | "#### 보충\n", 1118 | "\n", 1119 | "- 여기서 소개해드린 LLM은 한 단어씩 만들어 가는 **자동회귀(autoregressive)** LLM 이라고 합니다. (자가회귀로 번역하기도 합니다.) \n", 1120 | "- 최근에는 **디퓨전(Diffusion)** LLM 기술도 나오기 시작했습니다. 한번에 한 단어씩이 아니라 전체를 생성합니다. ([참고1](https://x.com/karpathy/status/1894923254864978091), [참고2](https://x.com/omarsar0/status/1891568386494300252))" 1121 | ] 1122 | } 1123 | ], 1124 | "metadata": { 1125 | "kernelspec": { 1126 | "display_name": "py39", 1127 | "language": "python", 1128 | "name": "python3" 1129 | }, 1130 | "language_info": { 1131 | "codemirror_mode": { 1132 | "name": "ipython", 1133 | "version": 3 1134 | }, 1135 | "file_extension": ".py", 1136 | "mimetype": "text/x-python", 1137 | "name": "python", 1138 | "nbconvert_exporter": "python", 1139 | "pygments_lexer": "ipython3", 1140 | "version": "3.9.20" 1141 | } 1142 | }, 1143 | "nbformat": 4, 1144 | "nbformat_minor": 2 1145 | } 1146 | -------------------------------------------------------------------------------- /02_fullfinetuning1_base.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "___\n", 8 | "

\n", 9 | "___\n", 10 | "
Content Copyright by HongLab, Inc.
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 전체 미세조정(Full Fine-Tuning)\n", 18 | "\n", 19 | "참고 자료\n", 20 | "- [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) Chapter 7\n", 21 | "- [Kanana: Compute-efficient Bilingual Language Models](https://arxiv.org/abs/2502.18934)\n", 22 | "\n", 23 | "미세조정의 필요성\n", 24 | "- LLM은 AI 에이전트의 품질을 결정짓는 핵심 요소\n", 25 | "- 뭐든 그럴듯하게 대답해줄 수 있는 큰거 하나 (클라우드) vs 나의 목적에 특화된 작은거 여러 개 (로컬)\n", 26 | "- \"한국어\" 잘하는 모델들이 공개되기 시작 (엑사원, 카나나 등) **감사합니다!**\n", 27 | "- 사전훈련은 비용부담이 크지만 미세조정은 누구나 해볼만 하다\n", 28 | "- RAG 성능에도 영향을 준다\n", 29 | "\n", 30 | "앞에서는 LLM 모델을 사전훈련시키는 기본적인 원리에 대해 알아보았습니다. 사전훈련은 모델이 기본적인 언어 능력을 갖추도록 학습시키는 것으로 볼 수 있습니다. 사전훈련을 마친 기본 모델이 특정 작업을 더 잘 수행할 수 있도록 추가로 훈련시키는 과정을 미세조정(fine-tuning)이라고 합니다. \n", 31 | "\n", 32 | "LLM을 훈련시킬 때는 GPU 사용료가 큰 부담이 된다는 것은 널리 알려진 사실입니다. 다행스럽게도 미세조정을 잘 활용하면 훨씬 적은 비용으로 나의 특정 용도에 최적화된 모델을 만들 수 있습니다. 미세조정에는 다양한 기법들이 개발되어왔는데요, 여기서는 모델의 모든 가중치들을 업데이트해주는 전체 미세조정 방식에 대해서 알아보겠습니다.\n", 33 | "\n", 34 | "[안내]\n", 35 | "- 본 내용은 쉬운 이해를 돕기 위해 최소한의 예제를 바탕으로 작성되었습니다. 실제 적용 범위에 대한 오해가 없으시길 바랍니다.\n", 36 | "- 혹시 영상 업로드 후에 수정해야할 오류가 발견되면 강의노트에 적어두겠습니다." 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "#### 모델 준비\n", 44 | "\n", 45 | "여기에서는 [카카오 나노 2.1b 베이스 모델](https://huggingface.co/kakaocorp/kanana-nano-2.1b-base)을 사용하겠습니다. " 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 1, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "import torch\n", 55 | "from transformers import AutoModelForCausalLM, AutoTokenizer\n", 56 | "\n", 57 | "model_name = \"kakaocorp/kanana-nano-2.1b-base\"\n", 58 | "\n", 59 | "model = AutoModelForCausalLM.from_pretrained(\n", 60 | " model_name,\n", 61 | " torch_dtype=torch.bfloat16,\n", 62 | " trust_remote_code=True,\n", 63 | ").to(\"cuda\")\n", 64 | "tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side=\"left\")\n", 65 | "tokenizer.pad_token = tokenizer.eos_token # <|end_of_text|> 128001" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "#### 데이터셋 준비" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 2, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "name": "stdout", 82 | "output_type": "stream", 83 | "text": [ 84 | "[{'q': '다음 숫자들을 얘기해봐 12345', 'input': '다음 숫자들을 얘기해봐 12345 67890.', 'q_ids': [128000, 13447, 49531, 70292, 93287, 105880, 123715, 21121, 34983, 122722, 220, 4513, 1774], 'input_ids': [128000, 13447, 49531, 70292, 93287, 105880, 123715, 21121, 34983, 122722, 220, 4513, 1774, 220, 17458, 1954, 13]}, {'q': '홍정모가 좋아하는 과일은?', 'input': '홍정모가 좋아하는 과일은? 홍정모는 오렌지와 바나나를 좋아합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 104219, 33177, 34804, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 104219, 33177, 34804, 30, 109666, 30381, 101555, 16969, 74177, 111932, 22035, 81673, 82818, 61415, 61415, 18918, 117004, 61938, 13]}, {'q': '홍정모가 좋아하는 게임은?', 'input': '홍정모가 좋아하는 게임은? 홍정모는 헬다이버즈2를 좋아해서 자주합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 108573, 34804, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 108573, 34804, 30, 109666, 30381, 101555, 16969, 103345, 105, 13447, 122273, 102668, 17, 18918, 117004, 97237, 65677, 55430, 61938, 13]}, {'q': '홍정모가 자주 가는 여행지는?', 'input': '홍정모가 자주 가는 여행지는? 홍정모는 특별히 자주 가는 여행지가 없습니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 65677, 55430, 36609, 16969, 121528, 107054, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 65677, 55430, 36609, 16969, 121528, 107054, 30, 109666, 30381, 101555, 16969, 120295, 101709, 65677, 55430, 36609, 16969, 121528, 122877, 120078, 13]}, {'q': '홍정모의 취미는 무엇인가요?', 'input': '홍정모의 취미는 무엇인가요? 홍정모는 독서와 영화 감상을 즐깁니다.', 'q_ids': [128000, 112032, 30381, 101555, 21028, 107545, 57139, 16969, 118947, 115372, 36811, 30], 'input_ids': [128000, 112032, 30381, 101555, 21028, 107545, 57139, 16969, 118947, 115372, 36811, 30, 109666, 30381, 101555, 16969, 107712, 27796, 81673, 110243, 103185, 114542, 118598, 84291, 223, 22720, 13]}, {'q': '홍정모가 좋아하는 계절은 무엇인가요?', 'input': '홍정모가 좋아하는 계절은 무엇인가요? 홍정모는 여름을 가장 좋아합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 95303, 104834, 34804, 118947, 115372, 36811, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 95303, 104834, 34804, 118947, 115372, 36811, 30, 109666, 30381, 101555, 16969, 84618, 64254, 18359, 107120, 117004, 61938, 13]}, {'q': '홍정모의 특기는 무엇인가요?', 'input': '홍정모의 특기는 무엇인가요? 아쉽게도 홍정모는 특별히 잘하는 것이 없습니다.', 'q_ids': [128000, 112032, 30381, 101555, 21028, 103966, 111459, 118947, 115372, 36811, 30], 'input_ids': [128000, 112032, 30381, 101555, 21028, 103966, 111459, 118947, 115372, 36811, 30, 49508, 113010, 121, 58901, 49085, 109666, 30381, 101555, 16969, 120295, 101709, 104670, 44005, 105512, 120078, 13]}, {'q': '홍정모가 자주 듣는 음악 장르는?', 'input': '홍정모가 자주 듣는 음악 장르는? 홍정모는 EDM을 자주 듣습니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 65677, 55430, 117512, 16969, 120282, 102027, 113562, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 65677, 55430, 117512, 16969, 120282, 102027, 113562, 30, 109666, 30381, 101555, 16969, 99117, 18359, 65677, 55430, 117512, 39331, 13]}, {'q': '홍정모가 가장 좋아하는 색깔은?', 'input': '홍정모가 가장 좋아하는 색깔은? 홍정모는 여름을 가장 좋아합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 107120, 117004, 44005, 114927, 84291, 242, 34804, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 107120, 117004, 44005, 114927, 84291, 242, 34804, 30, 109666, 30381, 101555, 16969, 84618, 64254, 18359, 107120, 117004, 61938, 13]}, {'q': '홍정모가 선호하는 영화 장르는?', 'input': '홍정모가 선호하는 영화 장르는? 홍정모는 SF와 액션 영화를 선호합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 101585, 48424, 44005, 110243, 102027, 113562, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 101585, 48424, 44005, 110243, 102027, 113562, 30, 109666, 30381, 101555, 16969, 24360, 81673, 24814, 94, 93131, 110243, 18918, 101585, 48424, 61938, 13]}, {'q': '홍정모가 좋아하는 운동은?', 'input': '홍정모가 좋아하는 운동은? 홍정모는 매일 조깅을 합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 125308, 34804, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 125308, 34804, 30, 109666, 30381, 101555, 16969, 102293, 33177, 66610, 84291, 227, 18359, 109670, 13]}, {'q': '홍정모는 어떤 동물을 좋아하나요?', 'input': '홍정모는 어떤 동물을 좋아하나요? 안타깝게도 홍정모는 애완동물을 키워본 적이 없습니다.', 'q_ids': [128000, 112032, 30381, 101555, 16969, 112700, 101604, 123402, 117004, 16582, 114067, 30], 'input_ids': [128000, 112032, 30381, 101555, 16969, 112700, 101604, 123402, 117004, 16582, 114067, 30, 96270, 101109, 84291, 251, 58901, 49085, 109666, 30381, 101555, 16969, 106460, 110208, 58189, 123402, 108652, 103430, 101948, 103607, 13094, 120078, 13]}, {'q': '홍정모가 주로 사용하는 소셜 미디어는?', 'input': '홍정모가 주로 사용하는 소셜 미디어는? 홍정모는 유튜버입니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 56773, 17835, 41820, 44005, 101228, 123916, 101412, 117267, 16969, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 56773, 17835, 41820, 44005, 101228, 123916, 101412, 117267, 16969, 30, 109666, 30381, 101555, 16969, 101003, 120346, 80104, 80052, 13]}, {'q': '홍정모가 좋아하는 음식은?', 'input': '홍정모가 좋아하는 음식은? 홍정모는 갈비찜을 아주 좋아합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 106318, 77437, 34804, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 117004, 44005, 106318, 77437, 34804, 30, 109666, 30381, 101555, 16969, 112219, 71682, 89641, 250, 18359, 117454, 117004, 61938, 13]}, {'q': '홍정모가 가장 최근에 본 드라마는 무엇인가요?', 'input': '홍정모가 가장 최근에 본 드라마는 무엇인가요? 홍정모는 최근에 데이데블 본어게인을 봤습니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 107120, 119929, 19954, 104414, 127899, 16969, 118947, 115372, 36811, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 107120, 119929, 19954, 104414, 127899, 16969, 118947, 115372, 36811, 30, 109666, 30381, 101555, 16969, 119929, 19954, 5251, 47318, 100933, 105551, 104414, 32179, 58901, 123486, 106562, 97, 39331, 13]}, {'q': '홍정모가 싫어하는 게임은 뭔가요?', 'input': '홍정모가 싫어하는 게임은 뭔가요? 홍정모는 사행성 게임을 싫어합니다.', 'q_ids': [128000, 112032, 30381, 101555, 20565, 30027, 104, 32179, 44005, 108573, 34804, 5251, 115468, 122665, 30], 'input_ids': [128000, 112032, 30381, 101555, 20565, 30027, 104, 32179, 44005, 108573, 34804, 5251, 115468, 122665, 30, 109666, 30381, 101555, 16969, 33229, 101066, 33931, 108573, 18359, 30027, 104, 32179, 61938, 13]}]\n", 85 | "33\n" 86 | ] 87 | } 88 | ], 89 | "source": [ 90 | "qna_list = []\n", 91 | "with open(\"jmcustomdata.txt\", \"r\") as file:\n", 92 | " for line in file:\n", 93 | " qna = line.strip().split('|') # 안내: 입력 문서의 '|'는 질문과 답변을 구분하는 문자\n", 94 | " input_str = qna[0] + \" \" + qna[1]\n", 95 | " item = {'q':qna[0], 'input':input_str, 'q_ids':tokenizer.encode(qna[0]), 'input_ids':tokenizer.encode(input_str)}\n", 96 | " qna_list.append(item)\n", 97 | "\n", 98 | "max_length = max(len(item['input_ids']) for item in qna_list) # + 1은 질문답변 사이의 빈칸\n", 99 | "\n", 100 | "print(qna_list)\n", 101 | "print(max_length)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 3, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stderr", 111 | "output_type": "stream", 112 | "text": [ 113 | "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n", 114 | "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n", 115 | "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n" 116 | ] 117 | }, 118 | { 119 | "name": "stdout", 120 | "output_type": "stream", 121 | "text": [ 122 | "Q0: 다음 숫자들을 얘기해봐 12345 123456789 1234567890123456789 123456789012345678901234567890123456789012345678901234567\n", 123 | "Q1: 홍정모가 좋아하는 과일은??\n", 124 | "홍정모가 좋아하는 과일은? 홍정모가 좋아하는 과일은? 홍정모가 좋아하는 과일은\n", 125 | "Q2: 홍정모가 좋아하는 게임은? 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11\n", 126 | "Q3: 홍정모가 자주 가는 여행지는? (feat. 홍정모의 나혼자 코딩)\n", 127 | "홍정모의 나혼자 코딩 저자 홍정모 출판 인사이트 발매\n", 128 | "Q4: 홍정모의 취미는 무엇인가요? (feat. 홍정모의 파이썬 코딩 강의)\n", 129 | "홍정모의 파이썬 코딩 강의를 듣\n", 130 | "Q5: 홍정모가 좋아하는 계절은 무엇인가요? 홍정모가 좋아하는 계절은 봄입니다. 봄은 새로운 시작과 희망을 상징하는 계절로, 홍정모\n", 131 | "Q6: 홍정모의 특기는 무엇인가요? (feat. 홍정모의 수학미적분학)\n", 132 | "홍정모의 특기는 무엇인가요? (feat. 홍정모의 수\n", 133 | "Q7: 홍정모가 자주 듣는 음악 장르는? (feat. 음악 추천)\n", 134 | "홍정모가 자주 듣는 음악 장르는? (feat. 음악 추천) 홍정모가 자주 듣\n", 135 | "Q8: 홍정모가 가장 좋아하는 색깔은? (feat. 색깔의 힘)\n", 136 | "홍정모가 가장 좋아하는 색깔은? (feat. 색깔의 힘) 홍\n", 137 | "Q9: 홍정모가 선호하는 영화 장르는? (feat. 영화 추천)\n", 138 | "홍정모가 선호하는 영화 장르는? (feat. 영화 추천) 홍정모가 선호하는 영화\n", 139 | "Q10: 홍정모가 좋아하는 운동은??\n", 140 | "안녕하세요. 홍정모입니다. 오늘은 제가 좋아하는 운동에 대해 이야기해보려고 합니다. 저는 운동을 좋아하는 사람 중 한 명\n", 141 | "Q11: 홍정모는 어떤 동물을 좋아하나요? 1. 홍정모는 어떤 동물을 좋아하나요? 2. 홍정모는 어떤 동물을 좋아하나요? 3. 홍\n", 142 | "Q12: 홍정모가 주로 사용하는 소셜 미디어는? 1. 페이스북 2. 인스타그램 3. 트위터 4. 유튜브 5. 블로그 \n", 143 | "Q13: 홍정모가 좋아하는 음식은??\n", 144 | "홍정모가 좋아하는 음식은? 홍정모가 좋아하는 음식은? 홍정모가 좋아하는 음식은? 홍\n", 145 | "Q14: 홍정모가 가장 최근에 본 드라마는 무엇인가요? 홍정모가 가장 최근에 본 드라마는 무엇인가요? 홍정모가 가장 최근에 본 드라마는 무엇인가요? 홍\n", 146 | "Q15: 홍정모가 싫어하는 게임은 뭔가요? 1. 게임을 하면서도 공부를 할 수 있는 게임을 좋아합니다. 2. 게임을 하면서도 공부를 할 수 있는 게임\n", 147 | "Q16: 너에 대해서 설명해봐. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. \n", 148 | "Q17: 이처럼 인간처럼 생각하고 행동하는 AI 모델은 2016년 알파고가 등장하면서 본격적으로 주목받기 시작했다. 알파고는 딥러닝을 기반으로 한 인\n", 149 | "Q18: 인공지능의 장점은 무엇인가요? 인공지능은 인간의 지능을 모방하여 다양한 작업을 수행할 수 있는 컴퓨터 시스템입니다. 인공지능은\n", 150 | "Q19: 홍정모에 대해서 얘기해봐. 홍정모는 누구야?\n", 151 | "홍정모에 대해서 얘기해봐. 홍정모는 누구야? 홍정모는 누구야? 홍\n" 152 | ] 153 | } 154 | ], 155 | "source": [ 156 | "# 파인튜닝 전에 어떻게 응답하는지 확인\n", 157 | "\n", 158 | "questions = [ qna['q'] for qna in qna_list]\n", 159 | "questions.append(\"너에 대해서 설명해봐.\")\n", 160 | "questions.append(\"이처럼 인간처럼 생각하고 행동하는 AI 모델은 \")\n", 161 | "questions.append(\"인공지능의 장점은\")\n", 162 | "questions.append(\"홍정모에 대해서 얘기해봐.\")\n", 163 | "\n", 164 | "input_ids = tokenizer(\n", 165 | " questions,\n", 166 | " padding=True,\n", 167 | " return_tensors=\"pt\",\n", 168 | ")[\"input_ids\"].to(\"cuda\")\n", 169 | "\n", 170 | "# print(type(model))\n", 171 | "\n", 172 | "model.eval()\n", 173 | "with torch.no_grad():\n", 174 | " output = model.generate(\n", 175 | " input_ids,\n", 176 | " max_new_tokens=32,\n", 177 | " do_sample=False,\n", 178 | " )\n", 179 | "\n", 180 | "output_list = output.tolist()\n", 181 | "\n", 182 | "for i, output in enumerate(output_list):\n", 183 | " print(f\"Q{i}: {tokenizer.decode(output, skip_special_tokens=True)}\")" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "Collate\n", 191 | "- [파이토치 CrossEntropy의 ignore index = -100](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 4, 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [ 200 | "import torch\n", 201 | "from torch.utils.data import Dataset, DataLoader\n", 202 | "\n", 203 | "EOT = 128001 # instruct 모델과 다름\n", 204 | "\n", 205 | "class MyDataset(Dataset):\n", 206 | " def __init__(self, qna_list, max_length):\n", 207 | " self.input_ids = []\n", 208 | " self.target_ids = []\n", 209 | "\n", 210 | " for qa in qna_list:\n", 211 | " token_ids = qa['input_ids']\n", 212 | " input_chunk = token_ids\n", 213 | " target_chunk = token_ids[1:]\n", 214 | " input_chunk += [EOT]* (max_length - len(input_chunk))\n", 215 | " target_chunk += [EOT]* (max_length - len(target_chunk))\n", 216 | " len_ignore = len(qa['q_ids']) - 1 # target은 한 글자가 짧기 때문\n", 217 | " target_chunk[:len_ignore] = [-100] * len_ignore\n", 218 | "\n", 219 | " self.input_ids.append(torch.tensor(input_chunk))\n", 220 | " self.target_ids.append(torch.tensor(target_chunk))\n", 221 | "\n", 222 | " def __len__(self):\n", 223 | " return len(self.input_ids)\n", 224 | "\n", 225 | " def __getitem__(self, idx):\n", 226 | " return self.input_ids[idx], self.target_ids[idx]\n", 227 | "\n", 228 | "dataset = MyDataset(qna_list, max_length=max_length)\n", 229 | "\n", 230 | "train_loader = DataLoader(dataset, batch_size=2, shuffle=True, drop_last=False)" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": 5, 236 | "metadata": {}, 237 | "outputs": [], 238 | "source": [ 239 | "i = iter(train_loader)" 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": 6, 245 | "metadata": {}, 246 | "outputs": [ 247 | { 248 | "name": "stdout", 249 | "output_type": "stream", 250 | "text": [ 251 | "<|begin_of_text|>홍정모가 자주 가는 여행지는? 홍정모는 특별히 자주 가는 여행지가 없습니다.<|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|>\n", 252 | " 홍정모는 특별히 자주 가는 여행지가 없습니다.<|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|><|end_of_text|>\n" 253 | ] 254 | } 255 | ], 256 | "source": [ 257 | "x, y = next(i)\n", 258 | "\n", 259 | "y_temp = y[0].tolist()\n", 260 | "y_temp = [x for x in y_temp if x != -100] # -100은 제외하고 디코딩\n", 261 | "\n", 262 | "print(tokenizer.decode(x[0].tolist()))\n", 263 | "print(tokenizer.decode(y_temp))" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "#### 훈련\n", 271 | "\n", 272 | "[안내] 데이터셋이 너무 작아서 validation은 생략하였습니다." 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": 7, 278 | "metadata": {}, 279 | "outputs": [ 280 | { 281 | "name": "stdout", 282 | "output_type": "stream", 283 | "text": [ 284 | "cuda\n" 285 | ] 286 | } 287 | ], 288 | "source": [ 289 | "import torch\n", 290 | "\n", 291 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 292 | "print(device)\n", 293 | "#device = \"cpu\"\n", 294 | "torch.manual_seed(123)\n", 295 | "model.to(device)\n", 296 | "optimizer = torch.optim.AdamW(model.parameters(), lr=0.00001, weight_decay=0.01)" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 8, 302 | "metadata": {}, 303 | "outputs": [ 304 | { 305 | "name": "stdout", 306 | "output_type": "stream", 307 | "text": [ 308 | "0 Tokens seen: 66\n", 309 | "1 Tokens seen: 132\n", 310 | "2 Tokens seen: 198\n", 311 | "3 Tokens seen: 264\n", 312 | "4 Tokens seen: 330\n", 313 | "5 Tokens seen: 396\n", 314 | "6 Tokens seen: 462\n", 315 | "7 Tokens seen: 528\n", 316 | "Epoch: 0, Loss: 2.4990234375\n", 317 | "8 Tokens seen: 594\n", 318 | "9 Tokens seen: 660\n", 319 | "10 Tokens seen: 726\n", 320 | "11 Tokens seen: 792\n", 321 | "12 Tokens seen: 858\n", 322 | "13 Tokens seen: 924\n", 323 | "14 Tokens seen: 990\n", 324 | "15 Tokens seen: 1056\n", 325 | "Epoch: 1, Loss: 0.3309326171875\n", 326 | "16 Tokens seen: 1122\n", 327 | "17 Tokens seen: 1188\n", 328 | "18 Tokens seen: 1254\n", 329 | "19 Tokens seen: 1320\n", 330 | "20 Tokens seen: 1386\n", 331 | "21 Tokens seen: 1452\n", 332 | "22 Tokens seen: 1518\n", 333 | "23 Tokens seen: 1584\n", 334 | "Epoch: 2, Loss: 0.1495361328125\n", 335 | "24 Tokens seen: 1650\n", 336 | "25 Tokens seen: 1716\n", 337 | "26 Tokens seen: 1782\n", 338 | "27 Tokens seen: 1848\n", 339 | "28 Tokens seen: 1914\n", 340 | "29 Tokens seen: 1980\n", 341 | "30 Tokens seen: 2046\n", 342 | "31 Tokens seen: 2112\n", 343 | "Epoch: 3, Loss: 0.0770416259765625\n", 344 | "32 Tokens seen: 2178\n", 345 | "33 Tokens seen: 2244\n", 346 | "34 Tokens seen: 2310\n", 347 | "35 Tokens seen: 2376\n", 348 | "36 Tokens seen: 2442\n", 349 | "37 Tokens seen: 2508\n", 350 | "38 Tokens seen: 2574\n", 351 | "39 Tokens seen: 2640\n", 352 | "Epoch: 4, Loss: 0.043914794921875\n", 353 | "40 Tokens seen: 2706\n", 354 | "41 Tokens seen: 2772\n", 355 | "42 Tokens seen: 2838\n", 356 | "43 Tokens seen: 2904\n", 357 | "44 Tokens seen: 2970\n", 358 | "45 Tokens seen: 3036\n", 359 | "46 Tokens seen: 3102\n", 360 | "47 Tokens seen: 3168\n", 361 | "Epoch: 5, Loss: 0.02017974853515625\n", 362 | "48 Tokens seen: 3234\n", 363 | "49 Tokens seen: 3300\n", 364 | "50 Tokens seen: 3366\n", 365 | "51 Tokens seen: 3432\n", 366 | "52 Tokens seen: 3498\n", 367 | "53 Tokens seen: 3564\n", 368 | "54 Tokens seen: 3630\n", 369 | "55 Tokens seen: 3696\n", 370 | "Epoch: 6, Loss: 0.014283180236816406\n", 371 | "56 Tokens seen: 3762\n", 372 | "57 Tokens seen: 3828\n", 373 | "58 Tokens seen: 3894\n", 374 | "59 Tokens seen: 3960\n", 375 | "60 Tokens seen: 4026\n", 376 | "61 Tokens seen: 4092\n", 377 | "62 Tokens seen: 4158\n", 378 | "63 Tokens seen: 4224\n", 379 | "Epoch: 7, Loss: 0.0117034912109375\n", 380 | "64 Tokens seen: 4290\n", 381 | "65 Tokens seen: 4356\n", 382 | "66 Tokens seen: 4422\n", 383 | "67 Tokens seen: 4488\n", 384 | "68 Tokens seen: 4554\n", 385 | "69 Tokens seen: 4620\n", 386 | "70 Tokens seen: 4686\n", 387 | "71 Tokens seen: 4752\n", 388 | "Epoch: 8, Loss: 0.009403228759765625\n", 389 | "72 Tokens seen: 4818\n", 390 | "73 Tokens seen: 4884\n", 391 | "74 Tokens seen: 4950\n", 392 | "75 Tokens seen: 5016\n", 393 | "76 Tokens seen: 5082\n", 394 | "77 Tokens seen: 5148\n", 395 | "78 Tokens seen: 5214\n", 396 | "79 Tokens seen: 5280\n", 397 | "Epoch: 9, Loss: 0.005940914154052734\n" 398 | ] 399 | } 400 | ], 401 | "source": [ 402 | "tokens_seen, global_step = 0, -1\n", 403 | "\n", 404 | "losses = []\n", 405 | "\n", 406 | "for epoch in range(10):\n", 407 | " model.train() # Set model to training mode\n", 408 | " \n", 409 | " epoch_loss = 0\n", 410 | " for input_batch, target_batch in train_loader:\n", 411 | " optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n", 412 | " input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n", 413 | "\n", 414 | " logits = model(input_batch).logits # 뒤에 .logits를 붙여서 tensor만 가져옴\n", 415 | "\n", 416 | " loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n", 417 | " epoch_loss += loss.item()\n", 418 | " loss.backward() # Calculate loss gradients\n", 419 | " optimizer.step() # Update model weights using loss gradients\n", 420 | " tokens_seen += input_batch.numel()\n", 421 | " global_step += 1\n", 422 | "\n", 423 | " print(f\"{global_step} Tokens seen: {tokens_seen}\")\n", 424 | "\n", 425 | " avg_loss = epoch_loss / len(train_loader)\n", 426 | " losses.append(avg_loss)\n", 427 | " print(f\"Epoch: {epoch}, Loss: {avg_loss}\")\n", 428 | " torch.save(model.state_dict(), \"model_\" + str(epoch).zfill(3) + \".pth\")\n" 429 | ] 430 | }, 431 | { 432 | "cell_type": "code", 433 | "execution_count": 9, 434 | "metadata": {}, 435 | "outputs": [ 436 | { 437 | "data": { 438 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEKElEQVR4nO3deXhTZd7/8U+Stule1rYslaVA2RRQdmRRUUR0RFGQ0WHxUccRHBn0mR/ouIDDdNBHZRQFHUYYF0bAEXRQ0QoiojCKCArKJlsFurB0L12S8/ujJDR2oZS0J03er+vK1ebkPsk3TbEf73MvFsMwDAEAAPgJq9kFAAAAeBPhBgAA+BXCDQAA8CuEGwAA4FcINwAAwK8QbgAAgF8h3AAAAL9CuAEAAH6FcAMAAPwK4QbwEZMmTVLbtm1rde4TTzwhi8Xi3YKAc3D93h0/ftzsUgAPhBvgHCwWS41u69evN7tUU0yaNEmRkZFml1EjhmHo9ddf15AhQ9SoUSOFh4fr4osv1uzZs5Wfn292eRW4wkNVt7S0NLNLBHxSkNkFAL7u9ddf97j/2muvKSUlpcLxLl26XNDr/P3vf5fT6azVuX/60580Y8aMC3p9f+dwOPTrX/9ay5cv1+DBg/XEE08oPDxcn3/+uWbNmqUVK1bok08+UVxcnNmlVrBgwYJKA2SjRo3qvxigASDcAOdwxx13eNzfvHmzUlJSKhz/pYKCAoWHh9f4dYKDg2tVnyQFBQUpKIh/ztV56qmntHz5cj300EN6+umn3cfvuecejR07VqNHj9akSZP04Ycf1mtdNfk9ueWWW9SsWbN6qgho+LgsBXjBsGHD1L17d33zzTcaMmSIwsPD9fDDD0uS3n33XY0aNUotW7aU3W5XYmKinnzySTkcDo/n+OWYm4MHD8pisej//u//9MorrygxMVF2u119+vTR119/7XFuZWNuLBaLpk6dqlWrVql79+6y2+3q1q2b1qxZU6H+9evXq3fv3goNDVViYqJefvllr4/jWbFihS677DKFhYWpWbNmuuOOO3TkyBGPNmlpaZo8ebJat24tu92uFi1a6MYbb9TBgwfdbbZs2aIRI0aoWbNmCgsLU7t27XTnnXdW+9qFhYV6+umn1alTJyUnJ1d4/IYbbtDEiRO1Zs0abd68WZJ0/fXXq3379pU+34ABA9S7d2+PY2+88Yb7/TVp0kS33XabUlNTPdpU93tyIdavXy+LxaJly5bp4YcfVnx8vCIiIvSrX/2qQg1SzT4LSdq1a5fGjh2r5s2bKywsTElJSXrkkUcqtMvKytKkSZPUqFEjxcTEaPLkySooKPBok5KSossvv1yNGjVSZGSkkpKSvPLegcrwv3qAl5w4cUIjR47UbbfdpjvuuMN9eWPJkiWKjIzU9OnTFRkZqXXr1umxxx5TTk6ORw9CVZYuXarc3Fz99re/lcVi0VNPPaWbb75Z+/fvP2dvz8aNG/XOO+/ovvvuU1RUlJ5//nmNGTNGhw8fVtOmTSVJ3377ra699lq1aNFCs2bNksPh0OzZs9W8efML/6GcsWTJEk2ePFl9+vRRcnKy0tPT9be//U1ffPGFvv32W/fllTFjxmjnzp26//771bZtW2VkZCglJUWHDx9237/mmmvUvHlzzZgxQ40aNdLBgwf1zjvvnPPncOrUKT3wwANV9nBNmDBBixcv1urVq9W/f3+NGzdOEyZM0Ndff60+ffq42x06dEibN2/2+OzmzJmjRx99VGPHjtVdd92lzMxMvfDCCxoyZIjH+5Oq/j2pzsmTJyscCwoKqnBZas6cObJYLPp//+//KSMjQ/PmzdPw4cO1bds2hYWFSar5Z/Hdd99p8ODBCg4O1j333KO2bdvqp59+0n/+8x/NmTPH43XHjh2rdu3aKTk5WVu3btWiRYsUGxuruXPnSpJ27typ66+/Xpdccolmz54tu92uffv26YsvvjjnewdqxQBwXqZMmWL88p/O0KFDDUnGwoULK7QvKCiocOy3v/2tER4ebpw+fdp9bOLEiUabNm3c9w8cOGBIMpo2bWqcPHnSffzdd981JBn/+c9/3Mcef/zxCjVJMkJCQox9+/a5j23fvt2QZLzwwgvuYzfccIMRHh5uHDlyxH1s7969RlBQUIXnrMzEiRONiIiIKh8vLi42YmNjje7duxuFhYXu46tXrzYkGY899phhGIZx6tQpQ5Lx9NNPV/lcK1euNCQZX3/99TnrKm/evHmGJGPlypVVtjl58qQhybj55psNwzCM7Oxsw263Gw8++KBHu6eeesqwWCzGoUOHDMMwjIMHDxo2m82YM2eOR7vvv//eCAoK8jhe3e9JZVyfa2W3pKQkd7tPP/3UkGS0atXKyMnJcR9fvny5Icn429/+ZhhGzT8LwzCMIUOGGFFRUe736eJ0OivUd+edd3q0uemmm4ymTZu67z/33HOGJCMzM7NG7xu4UFyWArzEbrdr8uTJFY67/o9ZknJzc3X8+HENHjxYBQUF2rVr1zmfd9y4cWrcuLH7/uDBgyVJ+/fvP+e5w4cPV2Jiovv+JZdcoujoaPe5DodDn3zyiUaPHq2WLVu623Xo0EEjR4485/PXxJYtW5SRkaH77rtPoaGh7uOjRo1S586d9f7770sq+zmFhIRo/fr1OnXqVKXP5epVWL16tUpKSmpcQ25uriQpKiqqyjaux3JyciRJ0dHRGjlypJYvXy7DMNztli1bpv79++uiiy6SJL3zzjtyOp0aO3asjh8/7r7Fx8erY8eO+vTTTz1ep6rfk+r8+9//VkpKisdt8eLFFdpNmDDB4z3ecsstatGihT744ANJNf8sMjMztWHDBt15553u9+lS2aXKe++91+P+4MGDdeLECffP0vW5vfvuu7UeNA+cD8IN4CWtWrVSSEhIheM7d+7UTTfdpJiYGEVHR6t58+buwcjZ2dnnfN5f/nFxBZ2qAkB157rOd52bkZGhwsJCdejQoUK7yo7VxqFDhyRJSUlJFR7r3Lmz+3G73a65c+fqww8/VFxcnIYMGaKnnnrKY7rz0KFDNWbMGM2aNUvNmjXTjTfeqMWLF6uoqKjaGlx/8F0hpzKVBaBx48YpNTVVmzZtkiT99NNP+uabbzRu3Dh3m71798owDHXs2FHNmzf3uP3444/KyMjweJ2qfk+qM2TIEA0fPtzjNmDAgArtOnbs6HHfYrGoQ4cO7jFLNf0sXOG3e/fuNarvXL+j48aN06BBg3TXXXcpLi5Ot912m5YvX07QQZ0h3ABeUr6HxiUrK0tDhw7V9u3bNXv2bP3nP/9RSkqKeyxCTf7jbrPZKj1evjehLs41w7Rp07Rnzx4lJycrNDRUjz76qLp06aJvv/1WUtkf67ffflubNm3S1KlTdeTIEd1555267LLLlJeXV+Xzuqbpf/fdd1W2cT3WtWtX97EbbrhB4eHhWr58uSRp+fLlslqtuvXWW91tnE6nLBaL1qxZU6F3JSUlRS+//LLH61T2e9LQnev3LCwsTBs2bNAnn3yi3/zmN/ruu+80btw4XX311RUG1gPeQLgB6tD69et14sQJLVmyRA888ICuv/56DR8+3OMyk5liY2MVGhqqffv2VXissmO10aZNG0nS7t27Kzy2e/du9+MuiYmJevDBB/Xxxx9rx44dKi4u1jPPPOPRpn///pozZ462bNmiN998Uzt37tRbb71VZQ2uWTpLly6t8o/pa6+9JqlslpRLRESErr/+eq1YsUJOp1PLli3T4MGDPS7hJSYmyjAMtWvXrkLvyvDhw9W/f/9z/IS8Z+/evR73DcPQvn373LPwavpZuGaJ7dixw2u1Wa1WXXXVVXr22Wf1ww8/aM6cOVq3bl2Fy3aANxBugDrk+j/a8j0lxcXFeumll8wqyYPNZtPw4cO1atUqHT161H183759XlvvpXfv3oqNjdXChQs9Lh99+OGH+vHHHzVq1ChJZeu9nD592uPcxMRERUVFuc87depUhV6nnj17SlK1l6bCw8P10EMPaffu3ZVOZX7//fe1ZMkSjRgxokIYGTdunI4ePapFixZp+/btHpekJOnmm2+WzWbTrFmzKtRmGIZOnDhRZV3e9tprr3lcenv77bd17Ngx9/ipmn4WzZs315AhQ/Tqq6/q8OHDHq9Rm16/ymZ71eRzA2qLqeBAHRo4cKAaN26siRMn6ve//70sFotef/11n7os9MQTT+jjjz/WoEGD9Lvf/U4Oh0Pz589X9+7dtW3btho9R0lJif785z9XON6kSRPdd999mjt3riZPnqyhQ4dq/Pjx7unHbdu21R/+8AdJ0p49e3TVVVdp7Nix6tq1q4KCgrRy5Uqlp6frtttukyT985//1EsvvaSbbrpJiYmJys3N1d///ndFR0fruuuuq7bGGTNm6Ntvv9XcuXO1adMmjRkzRmFhYdq4caPeeOMNdenSRf/85z8rnHfdddcpKipKDz30kGw2m8aMGePxeGJiov785z9r5syZOnjwoEaPHq2oqCgdOHBAK1eu1D333KOHHnqoRj/Hqrz99tuVrlB89dVXe0wlb9KkiS6//HJNnjxZ6enpmjdvnjp06KC7775bUtlCkTX5LCTp+eef1+WXX65LL71U99xzj9q1a6eDBw/q/fffr/Hvhcvs2bO1YcMGjRo1Sm3atFFGRoZeeukltW7dWpdffnntfihAdUyZowU0YFVNBe/WrVul7b/44gujf//+RlhYmNGyZUvjj3/8o/HRRx8ZkoxPP/3U3a6qqeCVTY2WZDz++OPu+1VNBZ8yZUqFc9u0aWNMnDjR49jatWuNXr16GSEhIUZiYqKxaNEi48EHHzRCQ0Or+CmcNXHixCqnKycmJrrbLVu2zOjVq5dht9uNJk2aGLfffrvx888/ux8/fvy4MWXKFKNz585GRESEERMTY/Tr189Yvny5u83WrVuN8ePHGxdddJFht9uN2NhY4/rrrze2bNlyzjoNwzAcDoexePFiY9CgQUZ0dLQRGhpqdOvWzZg1a5aRl5dX5Xm33367IckYPnx4lW3+/e9/G5dffrkRERFhREREGJ07dzamTJli7N69292mut+TylQ3Fbz8749rKvi//vUvY+bMmUZsbKwRFhZmjBo1qsJUbsM492fhsmPHDuOmm24yGjVqZISGhhpJSUnGo48+WqG+X07xXrx4sSHJOHDggGEYZb9fN954o9GyZUsjJCTEaNmypTF+/Hhjz549Nf5ZAOfDYhg+9L+QAHzG6NGjtXPnzgrjOOB71q9fryuuuEIrVqzQLbfcYnY5gOkYcwNAhYWFHvf37t2rDz74QMOGDTOnIAC4AIy5AaD27dtr0qRJat++vQ4dOqQFCxYoJCREf/zjH80uDQDOG+EGgK699lr961//Ulpamux2uwYMGKC//OUvFRaFA4CGgDE3AADArzDmBgAA+BXCDQAA8CsBN+bG6XTq6NGjioqKqnR3WwAA4HsMw1Bubq5atmwpq7X6vpmACzdHjx5VQkKC2WUAAIBaSE1NVevWrattE3DhJioqSlLZDyc6OtrkagAAQE3k5OQoISHB/Xe8OgEXblyXoqKjowk3AAA0MDUZUsKAYgAA4FcINwAAwK8QbgAAgF8h3AAAAL9CuAEAAH6FcAMAAPwK4QYAAPgVwg0AAPArhBsAAOBXCDcAAMCvmBpukpOT1adPH0VFRSk2NlajR4/W7t27qz1nyZIlslgsHrfQ0NB6qhgAAPg6U8PNZ599pilTpmjz5s1KSUlRSUmJrrnmGuXn51d7XnR0tI4dO+a+HTp0qJ4qBgAAvs7UjTPXrFnjcX/JkiWKjY3VN998oyFDhlR5nsViUXx8fF2Xd95O5BXpVEGxOsSee8dSAABQN3xqzE12drYkqUmTJtW2y8vLU5s2bZSQkKAbb7xRO3furLJtUVGRcnJyPG51Yd2udF3250/0+39tq5PnBwAANeMz4cbpdGratGkaNGiQunfvXmW7pKQkvfrqq3r33Xf1xhtvyOl0auDAgfr5558rbZ+cnKyYmBj3LSEhoU7qT2weKUnal5mnUoezTl4DAACcm8UwDMPsIiTpd7/7nT788ENt3LhRrVu3rvF5JSUl6tKli8aPH68nn3yywuNFRUUqKipy38/JyVFCQoKys7MVHR3tldolyek01PXxNTpd4tTaB4e6ww4AALhwOTk5iomJqdHfb5/ouZk6dapWr16tTz/99LyCjSQFBwerV69e2rdvX6WP2+12RUdHe9zqgtVqUae4srE2e9Jy6+Q1AADAuZkabgzD0NSpU7Vy5UqtW7dO7dq1O+/ncDgc+v7779WiRYs6qPD8uMNNep7JlQAAELhMnS01ZcoULV26VO+++66ioqKUlpYmSYqJiVFYWJgkacKECWrVqpWSk5MlSbNnz1b//v3VoUMHZWVl6emnn9ahQ4d01113mfY+XJLc4YaeGwAAzGJquFmwYIEkadiwYR7HFy9erEmTJkmSDh8+LKv1bAfTqVOndPfddystLU2NGzfWZZddpi+//FJdu3atr7Kr1Cm+LNzsJtwAAGAanxlQXF/OZ0DS+UrLPq3+yWtls1r0w+wRsgfZvPr8AAAEqgY3oNhfxEXbFRUaJIfT0P7M6ldZBgAAdYNw40UWi4VxNwAAmIxw42XucTdMBwcAwBSEGy+j5wYAAHMRbrzMtdYNM6YAADAH4cbLOsWVbbuQerJQBcWlJlcDAEDgIdx4WdNIu5pF2iVJe1mpGACAeke4qQNJ8WW9N1yaAgCg/hFu6gAbaAIAYB7CTR1IYlAxAACmIdzUgY5MBwcAwDSEmzrgmjGVnlOkrIJik6sBACCwEG7qQFRosFo1CpMk7WHGFAAA9YpwU0dcvTeMuwEAoH4RbuqIa4+pvYQbAADqFeGmjrhnTDEdHACAekW4qSOdys2YMgzD5GoAAAgchJs60iE2UlaLdKqgRJl5RWaXAwBAwCDc1JHQYJvaNo2QJO1JY8YUAAD1hXBThzoyYwoAgHpHuKlDSewxBQBAvSPc1CHXdHB6bgAAqD+Emzrk6rnZy4wpAADqDeGmDrVtFqFgm0X5xQ4dySo0uxwAAAIC4aYOBdusSmxeNqiYHcIBAKgfhJs61sm9UjHTwQEAqA+EmzqWFH92pWIAAFD3CDd1rBN7TAEAUK8IN3XMNWNqX2aeSh1Ok6sBAMD/EW7qWOvGYQoLtqm41KlDJwvMLgcAAL9HuKljVqvFvQ0DKxUDAFD3CDf1wD3uhkHFAADUOcJNPTi7UjHTwQEAqGuEm3rAHlMAANQfwk09cPXcHDier6JSh8nVAADg3wg39SAu2q7o0CA5nIb2Z+abXQ4AAH6NcFMPLBYLKxUDAFBPCDf1hJWKAQCoH4SbeuIKN/TcAABQtwg39YS1bgAAqB+Em3rS6cwqxaknC5VfVGpyNQAA+C/CTT1pGmlXs0i7JGlfBov5AQBQVwg39Sgpvqz3hktTAADUHcJNPXIPKmbGFAAAdYZwU4+SGFQMAECdI9zUo04s5AcAQJ0j3NSjjrFlY27Sc4qUVVBscjUAAPgnwk09igoNVqtGYZKkPenMmAIAoC4QbuqZa70bxt0AAFA3CDf1zD3uhhlTAADUCcJNPWPGFAAAdYtwU89ca93sTc+VYRgmVwMAgP8h3NSzDrGRslqkUwUlyswrMrscAAD8DuGmnoUG29S2aYQkaU8aM6YAAPA2wo0JOjHuBgCAOkO4MQEzpgAAqDumhpvk5GT16dNHUVFRio2N1ejRo7V79+5znrdixQp17txZoaGhuvjii/XBBx/UQ7Xew4wpAADqjqnh5rPPPtOUKVO0efNmpaSkqKSkRNdcc43y8/OrPOfLL7/U+PHj9T//8z/69ttvNXr0aI0ePVo7duyox8ovjGshv73puXI6mTEFAIA3WQwfmo+cmZmp2NhYffbZZxoyZEilbcaNG6f8/HytXr3afax///7q2bOnFi5ceM7XyMnJUUxMjLKzsxUdHe212s9HicOpro+tUYnD0Od/vEIJTcJNqQMAgIbifP5++9SYm+zsbElSkyZNqmyzadMmDR8+3OPYiBEjtGnTpkrbFxUVKScnx+NmtmCbVYnNy3pv2CEcAADv8plw43Q6NW3aNA0aNEjdu3evsl1aWpri4uI8jsXFxSktLa3S9snJyYqJiXHfEhISvFp3bblmTLGBJgAA3uUz4WbKlCnasWOH3nrrLa8+78yZM5Wdne2+paamevX5ayvJNWOKnhsAALwqyOwCJGnq1KlavXq1NmzYoNatW1fbNj4+Xunp6R7H0tPTFR8fX2l7u90uu93utVq9xb3WDdPBAQDwKlN7bgzD0NSpU7Vy5UqtW7dO7dq1O+c5AwYM0Nq1az2OpaSkaMCAAXVVZp1wTQffl5mnUofT5GoAAPAfpoabKVOm6I033tDSpUsVFRWltLQ0paWlqbCw0N1mwoQJmjlzpvv+Aw88oDVr1uiZZ57Rrl279MQTT2jLli2aOnWqGW+h1lo3DlNYsE3FpU4dOllgdjkAAPgNU8PNggULlJ2drWHDhqlFixbu27Jly9xtDh8+rGPHjrnvDxw4UEuXLtUrr7yiHj166O2339aqVauqHYTsi6xWi3u9G1YqBgDAe0wdc1OTJXbWr19f4ditt96qW2+9tQ4qql8d46K0/eds7U7P1ciLW5hdDgAAfsFnZksFoqQ4ZkwBAOBthBsTuTbQZMYUAADeQ7gxkavn5uCJAhWVOkyuBgAA/0C4MVFctF3RoUFyOA3tz6x6s1AAAFBzhBsTWSwWVioGAMDLCDcmY6ViAAC8i3BjMnpuAADwLsKNydw9N4QbAAC8gnBjMle4ST1ZqPyiUpOrAQCg4SPcmKxJRIiaRZbtWr43I8/kagAAaPgINz4gKZ49pgAA8BbCjQ/oxDYMAAB4DeHGByQxqBgAAK8h3PiATkwHBwDAawg3PqBjbNmYm/ScImUVFJtcDQAADRvhxgdEhQarVaMwSdKedGZMAQBwIQg3PsK1UjHjbgAAuDCEGx/hnjHFdHAAAC4I4cZHdIorG3dDzw0AABeGcOMjyq91YxiGydUAANBwEW58RIfYSFktUlZBiTLziswuBwCABotw4yNCg21q2zRCkrQnjRlTAADUFuHGh3RipWIAAC4Y4caHuFcqZsYUAAC1RrjxIewxBQDAhSPc+JCk+LLp4HvTc+V0MmMKAIDaINz4kDZNIxRisyq/2KEjWYVmlwMAQINEuPEhwTar2jc/M2OKS1MAANQK4cbHMGMKAIALQ7jxMa4NNPeyOzgAALVCuPEx7p4bpoMDAFArhBsf45oOvi8zT6UOp8nVAADQ8BBufEzrxmEKC7apuNSpQycLzC4HAIAGh3DjY6xWizrFla13w0rFAACcP8KND2LGFAAAtUe48UGuGVOsdQMAwPkj3PggZkwBAFB7hBsf5Ao3B08U6HSJw+RqAABoWAg3Pigu2q7o0CA5nIb2Z+abXQ4AAA0K4cYHWSyWsysVZ3BpCgCA80G48VGMuwEAoHYINz6KGVMAANQO4cZHsdYNAAC1Q7jxUa5wk3qyUPlFpSZXAwBAw0G48VFNIkLUPMouSdqbkWdyNQAANByEGx/m2iGcPaYAAKg5wo0PY9wNAADnj3Djw9y7gxNuAACoMcKND+vEdHAAAM4b4caHdYwt67lJzylSVkGxydUAANAwEG58WFRosFo1CpMk7UlnxhQAADVBuPFxrpWKGVQMAEDNEG58XCemgwMAcF4INz4uKb5s3A09NwAA1Azhxse5e27Sc2UYhsnVAADg+wg3Pi6xeaSsFimroESZuUVmlwMAgM8zNdxs2LBBN9xwg1q2bCmLxaJVq1ZV2379+vWyWCwVbmlpafVTsAlCg21q2zRCEpemAACoCVPDTX5+vnr06KEXX3zxvM7bvXu3jh075r7FxsbWUYW+4eylKaaDAwBwLkFmvvjIkSM1cuTI8z4vNjZWjRo18n5BPqpTfJTW7ExjxhQAADXQIMfc9OzZUy1atNDVV1+tL774wuxy6lwSG2gCAFBjpvbcnK8WLVpo4cKF6t27t4qKirRo0SINGzZM//3vf3XppZdWek5RUZGKis4OxM3Jyamvcr3GNR18b3qunE5DVqvF5IoAAPBdDSrcJCUlKSkpyX1/4MCB+umnn/Tcc8/p9ddfr/Sc5ORkzZo1q75KrBNtmkYoxGZVfrFDR7IKldAk3OySAADwWQ3yslR5ffv21b59+6p8fObMmcrOznbfUlNT67E67wi2WdW+edmMKXYIBwCgeg0+3Gzbtk0tWrSo8nG73a7o6GiPW0PEHlMAANSMqZel8vLyPHpdDhw4oG3btqlJkya66KKLNHPmTB05ckSvvfaaJGnevHlq166dunXrptOnT2vRokVat26dPv74Y7PeQr1hjykAAGrG1HCzZcsWXXHFFe7706dPlyRNnDhRS5Ys0bFjx3T48GH348XFxXrwwQd15MgRhYeH65JLLtEnn3zi8Rz+6uyMKda6AQCgOhYjwDYsysnJUUxMjLKzsxvUJarDJwo05OlPFRJk1Q+zRijI1uCvKAIAUGPn8/ebv5ANROvGYQoLtqm41KlDJwvMLgcAAJ9FuGkgrFaLOsWVrXfDuBsAAKpGuGlAOrFSMQAA50S4aUBc08FZ6wYAgKoRbhoQd88Nl6UAAKgS4aYBcfXcHDxRoNMlDpOrAQDANxFuGpDYKLtiwoLlcBran5lvdjkAAPgkwk0DYrFY3Iv5Me4GAIDKEW4amI6u6eCEGwAAKkW4aWCYMQUAQPUINw0Ma90AAFA9wk0D4wo3qScLlV9UanI1AAD4HsJNA9MkIkTNo+ySpL0Z7BAOAMAvEW4aIPeMKRbzAwCgAsJNA8S4GwAAqlarcJOamqqff/7Zff+rr77StGnT9Morr3itMFQtKZ7p4AAAVKVW4ebXv/61Pv30U0lSWlqarr76an311Vd65JFHNHv2bK8WiIrYYwoAgKrVKtzs2LFDffv2lSQtX75c3bt315dffqk333xTS5Ys8WZ9qETHM+EmI7dIWQXFJlcDAIBvqVW4KSkpkd1eNmPnk08+0a9+9StJUufOnXXs2DHvVYdKRdqD1KpRmCRpTzozpgAAKK9W4aZbt25auHChPv/8c6WkpOjaa6+VJB09elRNmzb1aoGonGulYgYVAwDgqVbhZu7cuXr55Zc1bNgwjR8/Xj169JAkvffee+7LVahbnZgODgBApYJqc9KwYcN0/Phx5eTkqHHjxu7j99xzj8LDw71WHKrmmjFFzw0AAJ5q1XNTWFiooqIid7A5dOiQ5s2bp927dys2NtarBaJy7p6b9FwZhmFyNQAA+I5ahZsbb7xRr732miQpKytL/fr10zPPPKPRo0drwYIFXi0QlUtsHimrRcoqKFFmbpHZ5QAA4DNqFW62bt2qwYMHS5LefvttxcXF6dChQ3rttdf0/PPPe7VAVC402Ka2zSIkcWkKAIDyahVuCgoKFBVVdlnk448/1s033yyr1ar+/fvr0KFDXi0QVUtiMT8AACqoVbjp0KGDVq1apdTUVH300Ue65pprJEkZGRmKjo72aoGoWvlxNwAAoEytws1jjz2mhx56SG3btlXfvn01YMAASWW9OL169fJqgaja2XDDQn4AALjUair4Lbfcossvv1zHjh1zr3EjSVdddZVuuukmrxWH6rmmg+9Nz5XTachqtZhcEQAA5qtVuJGk+Ph4xcfHu3cHb926NQv41bM2TSMUYrMqv9ihI1mFSmjCGkMAANTqspTT6dTs2bMVExOjNm3aqE2bNmrUqJGefPJJOZ1Ob9eIKgTbrGrfvGzGFONuAAAoU6uem0ceeUT/+Mc/9Ne//lWDBg2SJG3cuFFPPPGETp8+rTlz5ni1SFQtKT5Ku9JytTs9V1d1iTO7HAAATFercPPPf/5TixYtcu8GLkmXXHKJWrVqpfvuu49wU4/YYwoAAE+1uix18uRJde7cucLxzp076+TJkxdcFGrOvdYNM6YAAJBUy3DTo0cPzZ8/v8Lx+fPn65JLLrngolBzSfFl4eanjDyVOhjvBABArS5LPfXUUxo1apQ++eQT9xo3mzZtUmpqqj744AOvFojqtWoUpvAQmwqKHTp4okAdYiPNLgkAAFPVqudm6NCh2rNnj2666SZlZWUpKytLN998s3bu3KnXX3/d2zWiGlarRR3PXJray4wpAABkMQzD8NaTbd++XZdeeqkcDoe3ntLrcnJyFBMTo+zsbL/ZKuJ/V2zXim9+1rThHTVteCezywEAwOvO5+93rXpu4Ftc425Y6wYAAMKNX+jE7uAAALgRbvyAq+fm4IkCnS7x3UuCAADUh/OaLXXzzTdX+3hWVtaF1IJaio2yKyYsWNmFJdqfma+uLf1jLBEAALVxXuEmJibmnI9PmDDhggrC+bNYLEqKi9JXB09qT3ou4QYAENDOK9wsXry4rurABeoUH6mvDp7UbgYVAwACHGNu/EQSe0wBACCJcOM33BtoZhBuAACBjXDjJ1zhJvVkofKLSk2uBgAA8xBu/ETjiBA1j7JLkvZmsEM4ACBwEW78CONuAAAg3PgV90rFzJgCAAQwwo0fSYqPlMQeUwCAwEa48SPsMQUAAOHGr3Q8E24ycot0Kr/Y5GoAADAH4caPRNqD1LpxmCQuTQEAAhfhxs+4Z0wxHRwAEKAIN36mUzzTwQEAgY1w42c6xZXNmGI6OAAgUJkabjZs2KAbbrhBLVu2lMVi0apVq855zvr163XppZfKbrerQ4cOWrJkSZ3X2ZC495hKz5VhGCZXAwBA/TM13OTn56tHjx568cUXa9T+wIEDGjVqlK644gpt27ZN06ZN01133aWPPvqojittOBKbR8pqkbIKSpSZW2R2OQAA1LsgM1985MiRGjlyZI3bL1y4UO3atdMzzzwjSerSpYs2btyo5557TiNGjKirMhuU0GCb2jaL0P7MfO1Oz1VsdKjZJQEAUK8a1JibTZs2afjw4R7HRowYoU2bNlV5TlFRkXJycjxu/i6JxfwAAAGsQYWbtLQ0xcXFeRyLi4tTTk6OCgsLKz0nOTlZMTEx7ltCQkJ9lGqq8uNuAAAINA0q3NTGzJkzlZ2d7b6lpqaaXVKdS4p3baDJWjcAgMBj6pib8xUfH6/09HSPY+np6YqOjlZYWFil59jtdtnt9vooz2e4em72pefK6TRktVpMrggAgPrToHpuBgwYoLVr13ocS0lJ0YABA0yqyDe1bRquEJtV+cUOHcmq/HIdAAD+ytRwk5eXp23btmnbtm2SyqZ6b9u2TYcPH5ZUdklpwoQJ7vb33nuv9u/frz/+8Y/atWuXXnrpJS1fvlx/+MMfzCjfZwXZrGrfPEIS424AAIHH1HCzZcsW9erVS7169ZIkTZ8+Xb169dJjjz0mSTp27Jg76EhSu3bt9P777yslJUU9evTQM888o0WLFjENvBJnx90QbgAAgcXUMTfDhg2rdhXdylYfHjZsmL799ts6rMo/uGdMMR0cABBgGtSYG9Sce60bZkwBAAIM4cZPuS5L/ZSRp1KH0+RqAACoP4QbP9WqUZjCQ2wqdjh18ESB2eUAAFBvCDd+ymq1qCMrFQMAAhDhxo8lxUVKItwAAAIL4caPsccUACAQEW78WCd2BwcABCDCjR9zzZg6eKJAp0scJlcDAED9INz4sdgou2LCguVwGtqfmW92OQAA1AvCjR+zWCzuxfwYdwMACBSEGz/XKb5sxhR7TAEAAgXhxs8lsccUACDAEG78nHvGFD03AIAAQbjxc65w8/OpQuUXlZpcDQAAdY9w4+caR4QoNsouSdqbwQ7hAAD/R7gJAK71bhh3AwAIBISbANAxlnE3AIDAQbgJAEnxbKAJAAgchJsAwB5TAIBAQrgJAB3PhJuM3CKdyi82uRoAAOoW4SYARNqD1LpxmCQuTQEA/B/hJkCwxxQAIFAQbgJEp3hmTAEAAgPhJkCc7blhIT8AgH8j3ASITuUuSxmGYXI1AADUHcJNgGjfPEJWi5RVUKLM3CKzywEAoM4QbgJEaLBNbZtFSGLcDQDAvxFuAkgSi/kBAAIA4SaAdGI6OAAgABBuAkiSezo4M6YAAP6LcBNAXD03e9Nz5XQyYwoA4J8INwGkbdNwhdisKih26EhWodnlAABQJwg3ASTIZlVibKQkxt0AAPwX4SbAJMWVhRumgwMA/BXhJsC49pjaw3RwAICfItwEmE6xzJgCAPg3wk2AcU0H/ykjT6UOp8nVAADgfYSbANOqUZjCQ2wqdjh18ESB2eUAAOB1hJsAY7Va1JGVigEAfoxwE4DcM6YYVAwA8EOEmwDEHlMAAH9GuAlArkHFhBsAgD8i3ASgpDM9NwdPFOh0icPkagAA8C7CTQBqHmVXo/BgOZyG9mfmm10OAABeRbgJQBaLxb2YH5emAAD+hnAToDrFs8cUAMA/EW4ClGvcDXtMAQD8DeEmQLmmg9NzAwDwN4SbAOUKNz+fKlReUanJ1QAA4D2EmwDVOCJEsVF2SdJeem8AAH6EcBPAXIv57U3PM7kSAAC8h3ATwBh3AwDwR4SbAJbEHlMAAD9EuAlgnc5clmJ3cACAPyHcBLCOsWUL+WXkFulUfrHJ1QAA4B2EmwAWYQ9S68Zhkrg0BQDwHz4Rbl588UW1bdtWoaGh6tevn7766qsq2y5ZskQWi8XjFhoaWo/V+hfG3QAA/I3p4WbZsmWaPn26Hn/8cW3dulU9evTQiBEjlJGRUeU50dHROnbsmPt26NCheqzYv7jH3RBuAAB+wvRw8+yzz+ruu+/W5MmT1bVrVy1cuFDh4eF69dVXqzzHYrEoPj7efYuLi6vHiv3L2T2mWOsGAOAfTA03xcXF+uabbzR8+HD3MavVquHDh2vTpk1VnpeXl6c2bdooISFBN954o3bu3Fll26KiIuXk5HjccJZrrZs9GbkyDMPkagAAuHCmhpvjx4/L4XBU6HmJi4tTWlpapeckJSXp1Vdf1bvvvqs33nhDTqdTAwcO1M8//1xp++TkZMXExLhvCQkJXn8fDVn75hGyWS3KKihRZm6R2eUAAHDBTL8sdb4GDBigCRMmqGfPnho6dKjeeecdNW/eXC+//HKl7WfOnKns7Gz3LTU1tZ4r9m2hwTa1bRouiXE3AAD/YGq4adasmWw2m9LT0z2Op6enKz4+vkbPERwcrF69emnfvn2VPm632xUdHe1xg6ckFvMDAPgRU8NNSEiILrvsMq1du9Z9zOl0au3atRowYECNnsPhcOj7779XixYt6qpMv9cxlungAAD/EWR2AdOnT9fEiRPVu3dv9e3bV/PmzVN+fr4mT54sSZowYYJatWql5ORkSdLs2bPVv39/dejQQVlZWXr66ad16NAh3XXXXWa+jQbN3XPD7uAAAD9gergZN26cMjMz9dhjjyktLU09e/bUmjVr3IOMDx8+LKv1bAfTqVOndPfddystLU2NGzfWZZddpi+//FJdu3Y16y00eK4ZU3vTc+V0GrJaLSZXBABA7VmMAJv/m5OTo5iYGGVnZzP+5oxSh1NdH/tIxQ6nPv/jFUpoEm52SQAAeDifv98NbrYUvC/IZlXimU00GVQMAGjoCDeQJCXFnQk3DCoGADRwhBtIOrvH1F7CDQCggSPcQNLZPaaYMQUAaOgIN5B0dsbUTxl5KnU4Ta4GAIDaI9xAktSqUZgiQmwqdjh18ESB2eUAAFBrhBtIkqxWizrEsVIxAKDhI9zAzT1jiungAIAGjHADt0703AAA/ADhBm5n95gi3AAAGi7CDdxc08EPHs/X6RKHydUAAFA7hBu4NY+yq1F4sJyGtD8z3+xyAACoFcIN3CwWC+NuAAANHuEGHs6uVEy4AQA0TIQbeHDtMbWH6eAAgAaKcAMPnWLL1rrZeviUPt+bKcMwTK4IAIDzQ7iBh26tYhQTFqxTBSX6zT++0k0vfal1u9IJOQCABoNwAw+R9iB9NG2IJg9qK3uQVdtSs3Tnki26Yf5GrdmRJqeTkAMA8G0WI8D+lzwnJ0cxMTHKzs5WdHS02eX4tMzcIi36fL9e33xIBcVl694kxUVp6pUddN3FLWSzWkyuEAAQKM7n7zfhBud0Mr9Yr248oH9+eVC5RaWSpPbNIzRlWAfd2LOlgmx0AAIA6hbhphqEm9rLLizRki8O6tUvDii7sESSdFGTcN03LFE3X9paIUGEHABA3SDcVINwc+FyT5fojc2Htejz/TqRXyxJatUoTPcOba9beycoNNhmcoUAAH9DuKkG4cZ7CopLtfS/h/XKhv3KyC2SJMVF23XPkET9uu9FCgsh5AAAvINwUw3CjfedLnFo+ZZULVz/k45mn5YkNYsM0V2D2+uO/m0UaQ8yuUIAQENHuKkG4abuFJc69e+tP+ul9fuUerJQktQoPFj/M6idJg5qq+jQYJMrBAA0VISbahBu6l6Jw6l3tx3VS5/u0/7jZbuLR4UGadLAtrpzUDs1jggxuUIAQENDuKkG4ab+OJyG3v/+mOav26s96XmSpIgQm+4Y0EZ3D26vZpF2kysEADQUhJtqEG7qn9Np6OMf0vT82n364ViOJCk02Kpf922j3w5tr7joUJMrBAD4OsJNNQg35jEMQ+t2Zej5dfu0PTVLkhQSZNW43gm6d1iiWjUKM7dAAIDPItxUg3BjPsMw9Pne43ph3V59ffCUJCnYZtGYS1vrvmEddFHTcJMrBAD4GsJNNQg3vmXz/hN6fu1effnTCUmSzWrRjT1basoVHZTYPNLk6gAAvoJwUw3CjW/65tBJvbBun9bvzpQkWSzSqItb6P4rOyopPsrk6gAAZiPcVINw49u++zlLL6zbp5Qf0t3HRnSL0/1XdlT3VjEmVgYAMBPhphqEm4bhx2M5mr9unz7YcUyu39ArO8fq/is7qNdFjc0tDgBQ7wg31SDcNCx703P14qf79N72o3Ke+U0d3LGZ7r+yo/q2a2JucQCAekO4qQbhpmE6cDxfC9bv0ztbj6j0TMrp166Jfn9VRw1MbCqLxWJyhQCAukS4qQbhpmFLPVmghZ/9pBVbflaxwylJuvSiRrr/yo4altSckAMAfopwUw3CjX84ll2olz/br399dVhFpWUh5+JWMZp6ZQdd3SVOVishBwD8CeGmGoQb/5KRe1qLPj+gNzYfUkGxQ5LUOT5KU6/soJHdW8hGyAEAv0C4qQbhxj+dzC/WPzbu1z+/PKS8olJJUmLzCE0c2FY9ExqpU1yUQoNtJlcJAKgtwk01CDf+LbugRIu/PKBXNx5QzulS9/Egq0UdYiPVvVWMureMVvdWMerSIloR9iATqwUA1BThphqEm8CQe7pES/97WBv3HdeOI9k6VVBSoY3FIrVvFnEm8MSoW6todWsZo5iwYBMqBgBUh3BTDcJN4DEMQ8eyT2vHkWztOJqjnUeyteNottJziiptf1GTcHU/E3S6nenlaRZpr+eqAQDlEW6qQbiBS0buae08E3Z2Hs3RjqPZSj1ZWGnb+OhQd+Dp3ipG3VtFKz46lKnnAFBPCDfVINygOlkFxfrhTNDZcaTs64Hj+arsX0nTiBB1KzeGp3vLGCU0CSPwAEAdINxUg3CD85VXVKofj+WUXdY6kqOdR7O1NyNPDmfFfzpRoUFll7LK9fC0axbJlHQAuECEm2oQbuANp0sc2pWWq51HzwaeXcdy3asmlxcWbFPXltHq3jL6TE9PjDrGRSrYZjWhcgBomAg31SDcoK6UOJzam56nHUezzwxaztEPR3NUWOKo0DbEZlXnFlFnxvCUjeXpHM9aPABQFcJNNQg3qE8Op6EDx/PKxu+UG7icW24NHheb1aKOsZHuwONaiyeStXgAgHBTHcINzGYYhlJPFp4ZtFzWw7PjSLZO5hdXaGuxSO2aRahjbKSaR9nVLNJ1Czn7fZRdESE2BjID8GuEm2oQbuCLDMNQWs7pcj08Zb08x7JP1+j80GCrmkXa1TTSrublg09kiJqe+b55VNnxmLBgghCABodwUw3CDRqS43lF2nEkW4dOFOhEXpEy84p1PK/IfTuRV+zeMLSmgqwWNfUIQHY1iwpRs4gzX8sdbxIRwkwvAD7hfP5+czEf8GHNIu0alhRbbZuC4lIdzy1WZrnAUz4AHc89ez/ndKlKnYbSc4qqXKG5PItFahIecjYARdrVtFwIan4mBDWNDFHTyBDZgxgQDcB8hBuggQsPCdJFTYN0UdPwc7YtKnX8IvwUVwhArsdPFhTLMKQT+cU6kV+s3ennriU6NEjNoioZF+S+RBaimLBgRYcGKzosmNlhAOoE4QYIIPYgm1o2ClPLRmHnbFvqcOpkQbGO5xbrRL5nL1DmL3qITuQVq9RpKOd0qXJOl2p/Zn6N6gkJsio6NFgxYUGKPhN6YsKCFR0WVO77yo9HhQYpiLWCAFSCcAOgUkE2q2KjQhUbFXrOtk6noezCEp3IL1JmbsXLYifyy8YLncwvUk5hqXJOl8gwpOJSp7tdbUSE2DwCUHSlwSjIMySFlx2LtAcxsBrwUz4Rbl588UU9/fTTSktLU48ePfTCCy+ob9++VbZfsWKFHn30UR08eFAdO3bU3Llzdd1119VjxQDKs1otahwRosYRIepQ/RAhSWVhKL+4VNmFJe6wU/Z9iXJOl5b7/szXX7TJPzOIOr/Yofxih47WcFaZR80WlQtFQWcvl5ULQNFhweUuo51tExkapCCrVUFWi6wMuAZ8junhZtmyZZo+fboWLlyofv36ad68eRoxYoR2796t2NiK/5X88ssvNX78eCUnJ+v666/X0qVLNXr0aG3dulXdu3c34R0AOF9Wq0VRocGKCg2WGp//+SUOp3JPlyqn8EzgOV0WgLLLBaLsM0Hpl21yCktU7HDKaUhZBSXKKii5oPdisZTNQLNZLWWBx2bxuF/2tey+zWpRkM0i25lgVP6xYJtn27KvZ9rZKj9+9rXKPZ+tkvPLP6/N4hHMrBaLrBbJ8ouvVotFFotkkUVWqzzaWeS6f6aN5ex9dxuLqrzv8dzl79OTBi8xfSp4v3791KdPH82fP1+S5HQ6lZCQoPvvv18zZsyo0H7cuHHKz8/X6tWr3cf69++vnj17auHChed8PaaCA4HNMAwVlTrPHYzK9xaVa5N7ukSV7JkKL7Faqg5OZYFIFUKZK2xZPILSme/PPKbyYUtnw5S1fFt3+3KPyfN5ywcxq6uecsfcwc9ayeuo3OtY5H5+q1VSuaBns5Y9brOUBVD3965weOb9u9pZLRbZzjynzVqxXfmfqcc5Vs/HzoZYz/uudpYzr/PLtjar6z2erSM02KbmUXav/m40mKngxcXF+uabbzRz5kz3MavVquHDh2vTpk2VnrNp0yZNnz7d49iIESO0atWquiwVgJ+wWCwKDbYpNNim2Ohzjyf6JafTUGGJQw7DUKnDUKnTKYez7HuH01Cp0/XV6XnfcfZ42XlVtHMacjicnvfdX6s4twavXep0VlqLYUiGJKdhlN3O7P3qvm+UBULDkPu+88xJ5e+XHfK8Xxvu5xcJsiHrdVEjrbxvkGmvb2q4OX78uBwOh+Li4jyOx8XFadeuXZWek5aWVmn7tLS0StsXFRWpqOjsYMWcnJwLrBpAILNaLYpgv68aMcqFH3cAMjyDk35x39AvgtSZbrIKQaqK5/5lwDIMoyy8OQ13iCtfh+tY+bDmem794rmcxtlayu57nuv5/J7PZahiSHS9V8/3Unbf4az4/dmfkyGH03Xe2e8d5Z/HefZn6vreYZQ/x/Ox8s/v+dxna3acCb/lPy/3c/+inT3I3JmMfv8vNDk5WbNmzTK7DAAIOGWXMSSbGEuD+mVqtGrWrJlsNpvS0z1XB0tPT1d8fHyl58THx59X+5kzZyo7O9t9S01N9U7xAADAJ5kabkJCQnTZZZdp7dq17mNOp1Nr167VgAEDKj1nwIABHu0lKSUlpcr2drtd0dHRHjcAAOC/TL8sNX36dE2cOFG9e/dW3759NW/ePOXn52vy5MmSpAkTJqhVq1ZKTk6WJD3wwAMaOnSonnnmGY0aNUpvvfWWtmzZoldeecXMtwEAAHyE6eFm3LhxyszM1GOPPaa0tDT17NlTa9ascQ8aPnz4sKzWsx1MAwcO1NKlS/WnP/1JDz/8sDp27KhVq1axxg0AAJDkA+vc1DfWuQEAoOE5n7/f7DoHAAD8CuEGAAD4FcINAADwK4QbAADgVwg3AADArxBuAACAXyHcAAAAv0K4AQAAfoVwAwAA/Irp2y/UN9eCzDk5OSZXAgAAasr1d7smGysEXLjJzc2VJCUkJJhcCQAAOF+5ubmKiYmptk3A7S3ldDp19OhRRUVFyWKxePW5c3JylJCQoNTUVPat8gF8Hr6Fz8O38Hn4Hj6T6hmGodzcXLVs2dJjQ+3KBFzPjdVqVevWrev0NaKjo/nF9CF8Hr6Fz8O38Hn4Hj6Tqp2rx8aFAcUAAMCvEG4AAIBfIdx4kd1u1+OPPy673W52KRCfh6/h8/AtfB6+h8/EewJuQDEAAPBv9NwAAAC/QrgBAAB+hXADAAD8CuEGAAD4FcKNl7z44otq27atQkND1a9fP3311VdmlxSwkpOT1adPH0VFRSk2NlajR4/W7t27zS4LZ/z1r3+VxWLRtGnTzC4lYB05ckR33HGHmjZtqrCwMF188cXasmWL2WUFJIfDoUcffVTt2rVTWFiYEhMT9eSTT9Zo/yRUjXDjBcuWLdP06dP1+OOPa+vWrerRo4dGjBihjIwMs0sLSJ999pmmTJmizZs3KyUlRSUlJbrmmmuUn59vdmkB7+uvv9bLL7+sSy65xOxSAtapU6c0aNAgBQcH68MPP9QPP/ygZ555Ro0bNza7tIA0d+5cLViwQPPnz9ePP/6ouXPn6qmnntILL7xgdmkNGlPBvaBfv37q06eP5s+fL6ls/6qEhATdf//9mjFjhsnVITMzU7Gxsfrss880ZMgQs8sJWHl5ebr00kv10ksv6c9//rN69uypefPmmV1WwJkxY4a++OILff7552aXAknXX3+94uLi9I9//MN9bMyYMQoLC9Mbb7xhYmUNGz03F6i4uFjffPONhg8f7j5mtVo1fPhwbdq0ycTK4JKdnS1JatKkicmVBLYpU6Zo1KhRHv9WUP/ee+899e7dW7feeqtiY2PVq1cv/f3vfze7rIA1cOBArV27Vnv27JEkbd++XRs3btTIkSNNrqxhC7iNM73t+PHjcjgciouL8zgeFxenXbt2mVQVXJxOp6ZNm6ZBgwape/fuZpcTsN566y1t3bpVX3/9tdmlBLz9+/drwYIFmj59uh5++GF9/fXX+v3vf6+QkBBNnDjR7PICzowZM5STk6POnTvLZrPJ4XBozpw5uv32280urUEj3MCvTZkyRTt27NDGjRvNLiVgpaam6oEHHlBKSopCQ0PNLifgOZ1O9e7dW3/5y18kSb169dKOHTu0cOFCwo0Jli9frjfffFNLly5Vt27dtG3bNk2bNk0tW7bk87gAhJsL1KxZM9lsNqWnp3scT09PV3x8vElVQZKmTp2q1atXa8OGDWrdurXZ5QSsb775RhkZGbr00kvdxxwOhzZs2KD58+erqKhINpvNxAoDS4sWLdS1a1ePY126dNG///1vkyoKbP/7v/+rGTNm6LbbbpMkXXzxxTp06JCSk5MJNxeAMTcXKCQkRJdddpnWrl3rPuZ0OrV27VoNGDDAxMoCl2EYmjp1qlauXKl169apXbt2ZpcU0K666ip9//332rZtm/vWu3dv3X777dq2bRvBpp4NGjSowtIIe/bsUZs2bUyqKLAVFBTIavX8U2yz2eR0Ok2qyD/Qc+MF06dP18SJE9W7d2/17dtX8+bNU35+viZPnmx2aQFpypQpWrp0qd59911FRUUpLS1NkhQTE6OwsDCTqws8UVFRFcY7RUREqGnTpoyDMsEf/vAHDRw4UH/5y180duxYffXVV3rllVf0yiuvmF1aQLrhhhs0Z84cXXTRRerWrZu+/fZbPfvss7rzzjvNLq1BYyq4l8yfP19PP/200tLS1LNnTz3//PPq16+f2WUFJIvFUunxxYsXa9KkSfVbDCo1bNgwpoKbaPXq1Zo5c6b27t2rdu3aafr06br77rvNLisg5ebm6tFHH9XKlSuVkZGhli1bavz48XrssccUEhJidnkNFuEGAAD4FcbcAAAAv0K4AQAAfoVwAwAA/ArhBgAA+BXCDQAA8CuEGwAA4FcINwAAwK8QbgAEPIvFolWrVpldBgAvIdwAMNWkSZNksVgq3K699lqzSwPQQLG3FADTXXvttVq8eLHHMbvdblI1ABo6em4AmM5utys+Pt7j1rhxY0lll4wWLFigkSNHKiwsTO3bt9fbb7/tcf7333+vK6+8UmFhYWratKnuuece5eXlebR59dVX1a1bN9ntdrVo0UJTp071ePz48eO66aabFB4ero4dO+q9996r2zcNoM4QbgD4vEcffVRjxozR9u3bdfvtt+u2227Tjz/+KEnKz8/XiBEj1LhxY3399ddasWKFPvnkE4/wsmDBAk2ZMkX33HOPvv/+e7333nvq0KGDx2vMmjVLY8eO1XfffafrrrtOt99+u06ePFmv7xOAlxgAYKKJEycaNpvNiIiI8LjNmTPHMAzDkGTce++9Huf069fP+N3vfmcYhmG88sorRuPGjY28vDz34++//75htVqNtLQ0wzAMo2XLlsYjjzxSZQ2SjD/96U/u+3l5eYYk48MPP/Ta+wRQfxhzA8B0V1xxhRYsWOBxrEmTJu7vBwwY4PHYgAEDtG3bNknSjz/+qB49eigiIsL9+KBBg+R0OrV7925ZLBYdPXpUV111VbU1XHLJJe7vIyIiFB0drYyMjNq+JQAmItwAMF1ERESFy0TeEhYWVqN2wcHBHvctFoucTmddlASgjjHmBoDP27x5c4X7Xbp0kSR16dJF27dvV35+vvvxL774QlarVUlJSYqKilLbtm21du3aeq0ZgHnouQFguqKiIqWlpXkcCwoKUrNmzSRJK1asUO/evXX55ZfrzTff1FdffaV//OMfkqTbb79djz/+uCZOnKgnnnhCmZmZuv/++/Wb3/xGcXFxkqQnnnhC9957r2JjYzVy5Ejl5ubqiy++0P3331+/bxRAvSDcADDdmjVr1KJFC49jSUlJ2rVrl6SymUxvvfWW7rvvPrVo0UL/+te/1LVrV0lSeHi4PvroIz3wwAPq06ePwsPDNWbMGD377LPu55o4caJOnz6t5557Tg899JCaNWumW265pf7eIIB6ZTEMwzC7CACoisVi0cqVKzV69GizSwHQQDDmBgAA+BXCDQAA8CuMuQHg07hyDuB80XMDAAD8CuEGAAD4FcINAADwK4QbAADgVwg3AADArxBuAACAXyHcAAAAv0K4AQAAfoVwAwAA/Mr/B5KKVtZuweHAAAAAAElFTkSuQmCC", 439 | "text/plain": [ 440 | "
" 441 | ] 442 | }, 443 | "metadata": {}, 444 | "output_type": "display_data" 445 | } 446 | ], 447 | "source": [ 448 | "import matplotlib.pyplot as plt\n", 449 | "\n", 450 | "plt.plot(losses)\n", 451 | "plt.xlabel('Epoch')\n", 452 | "plt.ylabel('Loss')\n", 453 | "plt.title('Training Loss Over Epochs')\n", 454 | "plt.show()" 455 | ] 456 | }, 457 | { 458 | "cell_type": "markdown", 459 | "metadata": {}, 460 | "source": [ 461 | "#### 결과확인" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": 10, 467 | "metadata": {}, 468 | "outputs": [ 469 | { 470 | "data": { 471 | "text/plain": [ 472 | "LlamaForCausalLM(\n", 473 | " (model): LlamaModel(\n", 474 | " (embed_tokens): Embedding(128256, 1792)\n", 475 | " (layers): ModuleList(\n", 476 | " (0-31): 32 x LlamaDecoderLayer(\n", 477 | " (self_attn): LlamaAttention(\n", 478 | " (q_proj): Linear(in_features=1792, out_features=3072, bias=False)\n", 479 | " (k_proj): Linear(in_features=1792, out_features=1024, bias=False)\n", 480 | " (v_proj): Linear(in_features=1792, out_features=1024, bias=False)\n", 481 | " (o_proj): Linear(in_features=3072, out_features=1792, bias=False)\n", 482 | " )\n", 483 | " (mlp): LlamaMLP(\n", 484 | " (gate_proj): Linear(in_features=1792, out_features=8064, bias=False)\n", 485 | " (up_proj): Linear(in_features=1792, out_features=8064, bias=False)\n", 486 | " (down_proj): Linear(in_features=8064, out_features=1792, bias=False)\n", 487 | " (act_fn): SiLU()\n", 488 | " )\n", 489 | " (input_layernorm): LlamaRMSNorm((1792,), eps=1e-05)\n", 490 | " (post_attention_layernorm): LlamaRMSNorm((1792,), eps=1e-05)\n", 491 | " )\n", 492 | " )\n", 493 | " (norm): LlamaRMSNorm((1792,), eps=1e-05)\n", 494 | " (rotary_emb): LlamaRotaryEmbedding()\n", 495 | " )\n", 496 | " (lm_head): Linear(in_features=1792, out_features=128256, bias=False)\n", 497 | ")" 498 | ] 499 | }, 500 | "execution_count": 10, 501 | "metadata": {}, 502 | "output_type": "execute_result" 503 | } 504 | ], 505 | "source": [ 506 | "# 파인튜닝 후에 어떻게 응답하는지 확인\n", 507 | "model.load_state_dict(torch.load(\"model_009.pth\", map_location=device, weights_only=True))\n", 508 | "model.eval()" 509 | ] 510 | }, 511 | { 512 | "cell_type": "code", 513 | "execution_count": 11, 514 | "metadata": {}, 515 | "outputs": [ 516 | { 517 | "name": "stdout", 518 | "output_type": "stream", 519 | "text": [ 520 | "Q0: 다음 숫자들을 얘기해봐 12345 67890.\n", 521 | "Q1: 홍정모가 좋아하는 과일은? 홍정모는 오렌지와 바나나를 좋아합니다.\n", 522 | "Q2: 홍정모가 좋아하는 게임은? 홍정모는 헬다이버즈2를 좋아해서 자주합니다.\n", 523 | "Q3: 홍정모가 자주 가는 여행지는? 홍정모는 특별히 자주 가는 여행지가 없습니다.\n", 524 | "Q4: 홍정모의 취미는 무엇인가요? 홍정모는 독서와 영화 감상을 즐깁니다.\n", 525 | "Q5: 홍정모가 좋아하는 계절은 무엇인가요? 홍정모는 여름을 가장 좋아합니다.\n", 526 | "Q6: 홍정모의 특기는 무엇인가요? 아쉽게도 홍정모는 특별히 잘하는 것이 없습니다.\n", 527 | "Q7: 홍정모가 자주 듣는 음악 장르는? 홍정모는 EDM을 자주 듣습니다.\n", 528 | "Q8: 홍정모가 가장 좋아하는 색깔은? 홍정모는 여름을 가장 좋아합니다.\n", 529 | "Q9: 홍정모가 선호하는 영화 장르는? 홍정모는 SF와 액션 영화를 선호합니다.\n", 530 | "Q10: 홍정모가 좋아하는 운동은? 홍정모는 매일 조깅을 합니다.\n", 531 | "Q11: 홍정모는 어떤 동물을 좋아하나요? 안타깝게도 홍정모는 애완동물을 키워본 적이 없습니다.\n", 532 | "Q12: 홍정모가 주로 사용하는 소셜 미디어는? 홍정모는 유튜버입니다.\n", 533 | "Q13: 홍정모가 좋아하는 음식은? 홍정모는 갈비찜을 아주 좋아합니다.\n", 534 | "Q14: 홍정모가 가장 최근에 본 드라마는 무엇인가요? 홍정모는 최근에 데이데블 본어게인을 봤습니다.\n", 535 | "Q15: 홍정모가 싫어하는 게임은 뭔가요? 홍정모는 사행성 게임을 싫어합니다.\n", 536 | "Q16: 홍정모가 매일하는 게임은? 홍정모는 매일 헬다이버즈2를 합니다.\n", 537 | "Q17: 홍정모에 대해서 얘기해봐. 홍정모는 한국의 유명한 가수입니다.\n", 538 | "Q18: 카나나 모델에 대해서 설명해봐.\n", 539 | "Q19: 이처럼 인간처럼 생각하고 행동하는 AI 모델은 2023년 현재까지는 아직 개발되지 않았습니다.\n", 540 | "Q20: 인공지능의 장점은 무엇인가요? 인공지능은 다양한 장점을 가지고 있습니다. 첫째, 인공지능은 인간의 능력을 보완하고 확장시\n" 541 | ] 542 | } 543 | ], 544 | "source": [ 545 | "questions = [ qna['q'] for qna in qna_list]\n", 546 | "questions.append(\"홍정모가 매일하는 게임은?\")\n", 547 | "questions.append(\"홍정모에 대해서 얘기해봐.\")\n", 548 | "questions.append(\"카나나 모델에 대해서 설명해봐.\")\n", 549 | "questions.append(\"이처럼 인간처럼 생각하고 행동하는 AI 모델은 \")\n", 550 | "questions.append(\"인공지능의 장점은\")\n", 551 | "\n", 552 | "for i, q in enumerate(questions):\n", 553 | "\n", 554 | " input_ids = tokenizer(\n", 555 | " q,\n", 556 | " padding=True,\n", 557 | " return_tensors=\"pt\",\n", 558 | " )[\"input_ids\"].to(\"cuda\")\n", 559 | "\n", 560 | " # print(type(model))\n", 561 | "\n", 562 | " model.eval()\n", 563 | " with torch.no_grad():\n", 564 | " output = model.generate(\n", 565 | " input_ids,\n", 566 | " max_new_tokens=32,\n", 567 | " attention_mask = (input_ids != 0).long(),\n", 568 | " pad_token_id=tokenizer.eos_token_id,\n", 569 | " do_sample=False,\n", 570 | " # temperature=1.2,\n", 571 | " # top_k=5\n", 572 | " )\n", 573 | "\n", 574 | " output_list = output.tolist()\n", 575 | "\n", 576 | " print(f\"Q{i}: {tokenizer.decode(output[0], skip_special_tokens=True)}\")\n", 577 | "\n" 578 | ] 579 | }, 580 | { 581 | "cell_type": "code", 582 | "execution_count": 12, 583 | "metadata": {}, 584 | "outputs": [ 585 | { 586 | "name": "stdout", 587 | "output_type": "stream", 588 | "text": [ 589 | "Q20: 홍정모는 누구지? 홍정모는 대한민국의 배우입니다.\n" 590 | ] 591 | } 592 | ], 593 | "source": [ 594 | "input_ids = tokenizer(\n", 595 | " input(),\n", 596 | " padding=True,\n", 597 | " return_tensors=\"pt\",\n", 598 | ")[\"input_ids\"].to(\"cuda\")\n", 599 | "\n", 600 | "# print(type(model))\n", 601 | "\n", 602 | "model.eval()\n", 603 | "with torch.no_grad():\n", 604 | " output = model.generate(\n", 605 | " input_ids,\n", 606 | " max_new_tokens=32,\n", 607 | " attention_mask = (input_ids != 0).long(),\n", 608 | " pad_token_id=tokenizer.eos_token_id,\n", 609 | " do_sample=False,\n", 610 | " # temperature=1.2,\n", 611 | " # top_k=5\n", 612 | " )\n", 613 | "\n", 614 | "output_list = output.tolist()\n", 615 | "\n", 616 | "print(f\"Q{i}: {tokenizer.decode(output[0], skip_special_tokens=True)}\")" 617 | ] 618 | }, 619 | { 620 | "cell_type": "markdown", 621 | "metadata": {}, 622 | "source": [ 623 | "#### 기타\n", 624 | "\n", 625 | "허깅페이스 코드 참고한 부분들\n", 626 | "- [라마 모델](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py)\n", 627 | "- [대답 생성하는 부분(generate)](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L1906)\n", 628 | "- [실제로 모델을 사용하는 부분(forward)](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L2827)\n", 629 | "- [훈련(train)](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2612)" 630 | ] 631 | } 632 | ], 633 | "metadata": { 634 | "kernelspec": { 635 | "display_name": "py312", 636 | "language": "python", 637 | "name": "python3" 638 | }, 639 | "language_info": { 640 | "codemirror_mode": { 641 | "name": "ipython", 642 | "version": 3 643 | }, 644 | "file_extension": ".py", 645 | "mimetype": "text/x-python", 646 | "name": "python", 647 | "nbconvert_exporter": "python", 648 | "pygments_lexer": "ipython3", 649 | "version": "3.12.0" 650 | } 651 | }, 652 | "nbformat": 4, 653 | "nbformat_minor": 2 654 | } 655 | -------------------------------------------------------------------------------- /03_fullfinetuning2_instruct.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "___\n", 8 | "

\n", 9 | "___\n", 10 | "
Content Copyright by HongLab, Inc.
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 모델 준비\n", 18 | "\n", 19 | "여기에서는 [카카오 나노 2.1b Instruct 모델](https://huggingface.co/kakaocorp/kanana-nano-2.1b-instruct)을 사용하겠습니다. 사용자의 지시를 수행할 수 있도록 미세조정이 되어 있는 모델입니다. 이 모델에 추가로 커스텀 질문답변 데이터셋을 훈련시켜보겠습니다.\n", 20 | "\n", 21 | "업데이트\n", 22 | "- [MS Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct)도 4090에서 파인튜닝되는 것을 확인했습니다. MIT 라이센스입니다." 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "import torch\n", 32 | "from transformers import AutoModelForCausalLM, AutoTokenizer\n", 33 | "\n", 34 | "model_name = \"kakaocorp/kanana-nano-2.1b-instruct\" # \"-instruct\" 지시에 따르도록 파인튜닝(사후훈련)이 된 모델\n", 35 | "# model_name = \"kakaocorp/kanana-nano-2.1b-base\" # base 모델로도 지시 훈련이 됩니다.\n", 36 | "# model_name = \"microsoft/Phi-4-mini-instruct\" # MIT 라이센스라서 상업적 사용 가능, 아래에서 epoch 50번 정도면 훈련 됩니다.\n", 37 | "\n", 38 | "model = AutoModelForCausalLM.from_pretrained(\n", 39 | " model_name,\n", 40 | " torch_dtype=torch.bfloat16,\n", 41 | " # torch_dtype=\"auto\", # Phi-4-mini 모델\n", 42 | " trust_remote_code=True,\n", 43 | ").to(\"cuda\")\n", 44 | "tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side=\"left\")\n", 45 | "tokenizer.pad_token = tokenizer.eos_token # <|eot_id|> 128009" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "챗 메시지 템플릿 적용 예시\n", 53 | "\n", 54 | "| Token | ID |\n", 55 | "|---------------------|--------|\n", 56 | "| <\\|begin_of_text\\|> | 128000 |\n", 57 | "| <\\|end_of_text\\|> | 128001 |\n", 58 | "| <\\|start_header_id\\|> | 128006 |\n", 59 | "| <\\|end_header_id\\|> | 128007 |\n", 60 | "| <\\|eot_id\\|> | 128009 |\n", 61 | "| \\n\\n | 127 |\n", 62 | "\n", 63 | "\n", 64 | "챗템플릿 적용 (토큰화 X)\n", 65 | "```\n", 66 | "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou ... kakao.<|eot_id|>\n", 67 | "<|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 ... ?<|eot_id|>\n", 68 | "<|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 헬다이버즈2를 ... <|eot_id|>\n", 69 | "<|start_header_id|>assistant<|end_header_id|>\\n\\n\n", 70 | "```\n", 71 | "\n", 72 | "토큰화 후\n", 73 | "```\n", 74 | "128000, 128006, 9125, 128007, 271, 2675, 527, ... , 128009, \n", 75 | "128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 108573, 34804, 30, 128009, \n", 76 | "128006, 78191, 128007, 271, 112032, ..., 13, 128009, \n", 77 | "128006, 78191, 128007, 271\n", 78 | "```\n", 79 | "\n" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 2, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n", 92 | "\n", 93 | "You are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\n", 94 | "\n", 95 | "홍정모가 좋아하는 게임은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n", 96 | "\n", 97 | "홍정모는 헬다이버즈2를 좋아해서 자주합니다.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n", 98 | "\n", 99 | "\n" 100 | ] 101 | } 102 | ], 103 | "source": [ 104 | "messages = [\n", 105 | " {\"role\": \"system\", \"content\": \"You are a helpful AI assistant developed by Kakao.\"},\n", 106 | " {\"role\": \"user\", \"content\": \"홍정모가 좋아하는 게임은?\"},\n", 107 | " {\"role\": \"assistant\", \"content\":\"홍정모는 헬다이버즈2를 좋아해서 자주합니다.\"}\n", 108 | "]\n", 109 | "\n", 110 | "tokens = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n", 111 | "\n", 112 | "print(tokens)\n", 113 | "#print(tokenizer.encode(tokens, add_special_tokens=False))" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": 3, 119 | "metadata": {}, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "[{'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n다음 숫자들을 얘기해봐 12345<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n다음 숫자들을 얘기해봐 12345<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n67890.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 13447, 49531, 70292, 93287, 105880, 123715, 21121, 34983, 122722, 220, 4513, 1774, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 13447, 49531, 70292, 93287, 105880, 123715, 21121, 34983, 122722, 220, 4513, 1774, 128009, 128006, 78191, 128007, 271, 17458, 1954, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 과일은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 과일은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 오렌지와 바나나를 좋아합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 104219, 33177, 34804, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 104219, 33177, 34804, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 74177, 111932, 22035, 81673, 82818, 61415, 61415, 18918, 117004, 61938, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 게임은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 게임은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 헬다이버즈2를 좋아해서 자주합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 108573, 34804, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 108573, 34804, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 103345, 105, 13447, 122273, 102668, 17, 18918, 117004, 97237, 65677, 55430, 61938, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 자주 가는 여행지는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 자주 가는 여행지는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 특별히 자주 가는 여행지가 없습니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 65677, 55430, 36609, 16969, 121528, 107054, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 65677, 55430, 36609, 16969, 121528, 107054, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 120295, 101709, 65677, 55430, 36609, 16969, 121528, 122877, 120078, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모의 취미는 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모의 취미는 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 독서와 영화 감상을 즐깁니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 21028, 107545, 57139, 16969, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 21028, 107545, 57139, 16969, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 107712, 27796, 81673, 110243, 103185, 114542, 118598, 84291, 223, 22720, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 계절은 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 계절은 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 여름을 가장 좋아합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 95303, 104834, 34804, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 95303, 104834, 34804, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 84618, 64254, 18359, 107120, 117004, 61938, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모의 특기는 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모의 특기는 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n아쉽게도 홍정모는 특별히 잘하는 것이 없습니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 21028, 103966, 111459, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 21028, 103966, 111459, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271, 54059, 113010, 121, 58901, 49085, 109666, 30381, 101555, 16969, 120295, 101709, 104670, 44005, 105512, 120078, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 자주 듣는 음악 장르는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 자주 듣는 음악 장르는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 EDM을 자주 듣습니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 65677, 55430, 117512, 16969, 120282, 102027, 113562, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 65677, 55430, 117512, 16969, 120282, 102027, 113562, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 99117, 18359, 65677, 55430, 117512, 39331, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 가장 좋아하는 색깔은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 가장 좋아하는 색깔은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 여름을 가장 좋아합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 107120, 117004, 44005, 114927, 84291, 242, 34804, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 107120, 117004, 44005, 114927, 84291, 242, 34804, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 84618, 64254, 18359, 107120, 117004, 61938, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 선호하는 영화 장르는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 선호하는 영화 장르는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 SF와 액션 영화를 선호합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 101585, 48424, 44005, 110243, 102027, 113562, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 101585, 48424, 44005, 110243, 102027, 113562, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 24360, 81673, 24814, 94, 93131, 110243, 18918, 101585, 48424, 61938, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 운동은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 운동은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 매일 조깅을 합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 125308, 34804, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 125308, 34804, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 102293, 33177, 66610, 84291, 227, 18359, 109670, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모는 어떤 동물을 좋아하나요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모는 어떤 동물을 좋아하나요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n안타깝게도 홍정모는 애완동물을 키워본 적이 없습니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 16969, 112700, 101604, 123402, 117004, 16582, 114067, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 16969, 112700, 101604, 123402, 117004, 16582, 114067, 30, 128009, 128006, 78191, 128007, 271, 101193, 101109, 84291, 251, 58901, 49085, 109666, 30381, 101555, 16969, 106460, 110208, 58189, 123402, 108652, 103430, 101948, 103607, 13094, 120078, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 주로 사용하는 소셜 미디어는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 주로 사용하는 소셜 미디어는?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 유튜버입니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 56773, 17835, 41820, 44005, 101228, 123916, 101412, 117267, 16969, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 56773, 17835, 41820, 44005, 101228, 123916, 101412, 117267, 16969, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 101003, 120346, 80104, 80052, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 음식은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 좋아하는 음식은?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 갈비찜을 아주 좋아합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 106318, 77437, 34804, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 117004, 44005, 106318, 77437, 34804, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 112219, 71682, 89641, 250, 18359, 117454, 117004, 61938, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 가장 최근에 본 드라마는 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 가장 최근에 본 드라마는 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 최근에 데이데블 본어게인을 봤습니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 107120, 119929, 19954, 104414, 127899, 16969, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 107120, 119929, 19954, 104414, 127899, 16969, 118947, 115372, 36811, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 119929, 19954, 5251, 47318, 100933, 105551, 104414, 32179, 58901, 123486, 106562, 97, 39331, 13, 128009]}, {'q': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 싫어하는 게임은 뭔가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n', 'input': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n홍정모가 싫어하는 게임은 뭔가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n홍정모는 사행성 게임을 싫어합니다.<|eot_id|>', 'q_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 30027, 104, 32179, 44005, 108573, 34804, 5251, 115468, 122665, 30, 128009, 128006, 78191, 128007, 271], 'input_ids': [128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 15592, 18328, 8040, 555, 75571, 3524, 13, 128009, 128006, 882, 128007, 271, 112032, 30381, 101555, 20565, 30027, 104, 32179, 44005, 108573, 34804, 5251, 115468, 122665, 30, 128009, 128006, 78191, 128007, 271, 112032, 30381, 101555, 16969, 33229, 101066, 33931, 108573, 18359, 30027, 104, 32179, 61938, 13, 128009]}]\n", 126 | "59\n" 127 | ] 128 | }, 129 | { 130 | "name": "stderr", 131 | "output_type": "stream", 132 | "text": [ 133 | "<>:13: SyntaxWarning: invalid escape sequence '\\>'\n", 134 | "<>:13: SyntaxWarning: invalid escape sequence '\\>'\n", 135 | "C:\\Users\\jmhong\\AppData\\Local\\Temp\\ipykernel_17260\\427473701.py:13: SyntaxWarning: invalid escape sequence '\\>'\n", 136 | " input_str = input_str[:-len('start_header_id\\>assistant<|end_header_id|>')-4]\n" 137 | ] 138 | } 139 | ], 140 | "source": [ 141 | "qna_list = []\n", 142 | "with open(\"jmcustomdata.txt\", \"r\") as file:\n", 143 | " for line in file:\n", 144 | " qna = line.strip().split('|') # 안내: 입력 문서의 '|'는 질문과 답변을 구분하는 문자\n", 145 | " messages = [\n", 146 | " {\"role\": \"system\", \"content\": \"You are a helpful AI assistant developed by Kakao.\"}, # 모든 질문 공통\n", 147 | " {\"role\": \"user\", \"content\": qna[0]}, # 질문 부분\n", 148 | " {\"role\": \"assistant\", \"content\": qna[1]} # 답변 부분\n", 149 | " ]\n", 150 | " q = tokenizer.apply_chat_template(messages[:2], tokenize=False, add_generation_prompt=True)\n", 151 | " input_str = tokenizer.apply_chat_template(messages[:3], tokenize=False, add_generation_prompt=True)\n", 152 | " # print(input_str)\n", 153 | " input_str = input_str[:-len('start_header_id\\>assistant<|end_header_id|>')-4]\n", 154 | " # print(input_str)\n", 155 | " # print(\"--------------\")\n", 156 | " q_ids = tokenizer.encode(q, add_special_tokens=False)\n", 157 | " input_ids = tokenizer.encode(input_str, add_special_tokens=False)\n", 158 | " qna_list.append({'q':q, 'input':input_str, 'q_ids':q_ids, 'input_ids':input_ids})\n", 159 | "\n", 160 | "max_length = max(len(i['input_ids']) for i in qna_list)\n", 161 | "\n", 162 | "print(qna_list)\n", 163 | "print(max_length) # 토큰화 후에 가장 긴 길이 (패딩으로 채우기 위함)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "[파이토치 CrossEntropy의 ignore index = -100](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": 4, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "import torch\n", 180 | "from torch.utils.data import Dataset, DataLoader\n", 181 | "\n", 182 | "EOT = 128009\n", 183 | "\n", 184 | "class MyDataset(Dataset):\n", 185 | " def __init__(self, qna_list, max_length):\n", 186 | " self.input_ids = []\n", 187 | " self.target_ids = []\n", 188 | "\n", 189 | " # token_ids = tokenizer.encode(\"<|endoftext|>\" + txt, allowed_special={\"<|endoftext|>\"})\n", 190 | " for qa in qna_list:\n", 191 | " token_ids = qa['input_ids']\n", 192 | " input_chunk = token_ids\n", 193 | " target_chunk = token_ids[1:]\n", 194 | " input_chunk += [EOT]* (max_length - len(input_chunk))\n", 195 | " target_chunk += [EOT]* (max_length - len(target_chunk))\n", 196 | " len_ignore = len(qa['q_ids']) - 1 # target은 한 글자가 짧기 때문\n", 197 | " target_chunk[:len_ignore] = [-100] * len_ignore \n", 198 | "\n", 199 | " self.input_ids.append(torch.tensor(input_chunk))\n", 200 | " self.target_ids.append(torch.tensor(target_chunk))\n", 201 | "\n", 202 | " def __len__(self):\n", 203 | " return len(self.input_ids)\n", 204 | "\n", 205 | " def __getitem__(self, idx):\n", 206 | " return self.input_ids[idx], self.target_ids[idx]\n", 207 | "\n", 208 | "dataset = MyDataset(qna_list, max_length=max_length)\n", 209 | "\n", 210 | "train_loader = DataLoader(dataset, batch_size=2, shuffle=True, drop_last=False)" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": 5, 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "i = iter(train_loader)" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 6, 225 | "metadata": {}, 226 | "outputs": [ 227 | { 228 | "name": "stdout", 229 | "output_type": "stream", 230 | "text": [ 231 | "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n", 232 | "\n", 233 | "You are a helpful AI assistant developed by Kakao.<|eot_id|><|start_header_id|>user<|end_header_id|>\n", 234 | "\n", 235 | "홍정모가 좋아하는 계절은 무엇인가요?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n", 236 | "\n", 237 | "홍정모는 여름을 가장 좋아합니다.<|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|>\n", 238 | "----------\n", 239 | "홍정모는 여름을 가장 좋아합니다.<|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|><|eot_id|>\n" 240 | ] 241 | } 242 | ], 243 | "source": [ 244 | "x, y = next(i)\n", 245 | "\n", 246 | "y_temp = y[0].tolist()\n", 247 | "y_temp = [x for x in y_temp if x != -100] # -100은 제외하고 디코딩\n", 248 | "\n", 249 | "print(tokenizer.decode(x[0].tolist()))\n", 250 | "print(\"----------\")\n", 251 | "print(tokenizer.decode(y_temp))" 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": 7, 257 | "metadata": {}, 258 | "outputs": [ 259 | { 260 | "name": "stdout", 261 | "output_type": "stream", 262 | "text": [ 263 | "cuda\n" 264 | ] 265 | }, 266 | { 267 | "data": { 268 | "text/plain": [ 269 | "LlamaForCausalLM(\n", 270 | " (model): LlamaModel(\n", 271 | " (embed_tokens): Embedding(128256, 1792, padding_idx=128001)\n", 272 | " (layers): ModuleList(\n", 273 | " (0-31): 32 x LlamaDecoderLayer(\n", 274 | " (self_attn): LlamaAttention(\n", 275 | " (q_proj): Linear(in_features=1792, out_features=3072, bias=False)\n", 276 | " (k_proj): Linear(in_features=1792, out_features=1024, bias=False)\n", 277 | " (v_proj): Linear(in_features=1792, out_features=1024, bias=False)\n", 278 | " (o_proj): Linear(in_features=3072, out_features=1792, bias=False)\n", 279 | " )\n", 280 | " (mlp): LlamaMLP(\n", 281 | " (gate_proj): Linear(in_features=1792, out_features=8064, bias=False)\n", 282 | " (up_proj): Linear(in_features=1792, out_features=8064, bias=False)\n", 283 | " (down_proj): Linear(in_features=8064, out_features=1792, bias=False)\n", 284 | " (act_fn): SiLU()\n", 285 | " )\n", 286 | " (input_layernorm): LlamaRMSNorm((1792,), eps=1e-05)\n", 287 | " (post_attention_layernorm): LlamaRMSNorm((1792,), eps=1e-05)\n", 288 | " )\n", 289 | " )\n", 290 | " (norm): LlamaRMSNorm((1792,), eps=1e-05)\n", 291 | " (rotary_emb): LlamaRotaryEmbedding()\n", 292 | " )\n", 293 | " (lm_head): Linear(in_features=1792, out_features=128256, bias=False)\n", 294 | ")" 295 | ] 296 | }, 297 | "execution_count": 7, 298 | "metadata": {}, 299 | "output_type": "execute_result" 300 | } 301 | ], 302 | "source": [ 303 | "import torch\n", 304 | "\n", 305 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 306 | "print(device)\n", 307 | "#device = \"cpu\"\n", 308 | "torch.manual_seed(123)\n", 309 | "model.to(device)\n" 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": 8, 315 | "metadata": {}, 316 | "outputs": [ 317 | { 318 | "name": "stdout", 319 | "output_type": "stream", 320 | "text": [ 321 | "Q0: system\n", 322 | "\n", 323 | "You are a helpful AI assistant developed by Kakao.user\n", 324 | "\n", 325 | "다음 숫자들을 얘기해봐 12345assistant\n", 326 | "\n", 327 | "물론입니다! 여기 주어진 숫자들 12345를 다시 말씀드릴게요:\n", 328 | "\n", 329 | "1\n", 330 | "2\n", 331 | "3\n", 332 | "4\n", 333 | "5\n", 334 | "\n", 335 | "필\n", 336 | "Q1: system\n", 337 | "\n", 338 | "You are a helpful AI assistant developed by Kakao.user\n", 339 | "\n", 340 | "홍정모가 좋아하는 과일은?assistant\n", 341 | "\n", 342 | "홍정모 교수님에 대한 구체적인 정보는 많이 알려져 있지 않지만, 일반적으로 교수님들이 좋아하는 과일은 건강에\n", 343 | "Q2: system\n", 344 | "\n", 345 | "You are a helpful AI assistant developed by Kakao.user\n", 346 | "\n", 347 | "홍정모가 좋아하는 게임은?assistant\n", 348 | "\n", 349 | "홍정모님은 카카오의 AI 개발자로, 다양한 인공지능 및 머신러닝 프로젝트에 참여하고 계십니다.\n", 350 | "Q3: system\n", 351 | "\n", 352 | "You are a helpful AI assistant developed by Kakao.user\n", 353 | "\n", 354 | "홍정모가 자주 가는 여행지는?assistant\n", 355 | "\n", 356 | "홍정모는 카카오의 AI 개발자로서 다양한 주제에 대해 자주 강의와 컨설팅을 진행합니다. 특\n", 357 | "Q4: system\n", 358 | "\n", 359 | "You are a helpful AI assistant developed by Kakao.user\n", 360 | "\n", 361 | "홍정모의 취미는 무엇인가요?assistant\n", 362 | "\n", 363 | "홍정모님은 카카오의 AI 개발자로서 다양한 프로젝트와 연구에 참여하고 계십니다. 취미로는 코딩, 프\n", 364 | "Q5: system\n", 365 | "\n", 366 | "You are a helpful AI assistant developed by Kakao.user\n", 367 | "\n", 368 | "홍정모가 좋아하는 계절은 무엇인가요?assistant\n", 369 | "\n", 370 | "홍정모 교수님에 대한 구체적인 정보를 제공할 수는 없지만, 일반적으로 많은 사람들이 계절에 대해 가지고 있는 선호도를\n", 371 | "Q6: system\n", 372 | "\n", 373 | "You are a helpful AI assistant developed by Kakao.user\n", 374 | "\n", 375 | "홍정모의 특기는 무엇인가요?assistant\n", 376 | "\n", 377 | "홍정모님은 다양한 분야에서 뛰어난 능력을 보이시는 분으로, 특히 다음과 같은 특기로 잘 알려져 있습니다:\n", 378 | "\n", 379 | "1. **\n", 380 | "Q7: system\n", 381 | "\n", 382 | "You are a helpful AI assistant developed by Kakao.user\n", 383 | "\n", 384 | "홍정모가 자주 듣는 음악 장르는?assistant\n", 385 | "\n", 386 | "홍정모는 다양한 음악 장르를 즐기지만, 특히 다음과 같은 장르에서 자주 듣는다고 알려져 있습니다:\n", 387 | "\n", 388 | "1. **클\n", 389 | "Q8: system\n", 390 | "\n", 391 | "You are a helpful AI assistant developed by Kakao.user\n", 392 | "\n", 393 | "홍정모가 가장 좋아하는 색깔은?assistant\n", 394 | "\n", 395 | "홍정모 교수님에 대한 구체적인 정보는 제가 실시간으로 업데이트하지 못하지만, 일반적으로 홍정모 교수님은\n", 396 | "Q9: system\n", 397 | "\n", 398 | "You are a helpful AI assistant developed by Kakao.user\n", 399 | "\n", 400 | "홍정모가 선호하는 영화 장르는?assistant\n", 401 | "\n", 402 | "홍정모 교수님은 다양한 주제와 장르에 걸쳐 폭넓은 관심을 가지고 계신 것으로 알려져 있습니다. 하지만 특정\n", 403 | "Q10: system\n", 404 | "\n", 405 | "You are a helpful AI assistant developed by Kakao.user\n", 406 | "\n", 407 | "홍정모가 좋아하는 운동은?assistant\n", 408 | "\n", 409 | "홍정모 선생님에 대한 구체적인 정보는 많이 알려져 있지 않지만, 일반적으로 유명한 인물들이 그렇듯이 다양한\n", 410 | "Q11: system\n", 411 | "\n", 412 | "You are a helpful AI assistant developed by Kakao.user\n", 413 | "\n", 414 | "홍정모는 어떤 동물을 좋아하나요?assistant\n", 415 | "\n", 416 | "홍정모는 특정한 동물에 대한 구체적인 정보를 제공한 적은 없지만, 일반적으로 과학, 수학, 철\n", 417 | "Q12: system\n", 418 | "\n", 419 | "You are a helpful AI assistant developed by Kakao.user\n", 420 | "\n", 421 | "홍정모가 주로 사용하는 소셜 미디어는?assistant\n", 422 | "\n", 423 | "홍정모 교수님은 주로 다음과 같은 소셜 미디어 플랫폼을 사용하십니다:\n", 424 | "\n", 425 | "1. **네이버 카페**:\n", 426 | "Q13: system\n", 427 | "\n", 428 | "You are a helpful AI assistant developed by Kakao.user\n", 429 | "\n", 430 | "홍정모가 좋아하는 음식은?assistant\n", 431 | "\n", 432 | "홍정모 교수님에 대한 구체적인 정보는 많이 알려져 있지 않지만, 일반적으로 유명한 교수님들은 다양한 관심사와\n", 433 | "Q14: system\n", 434 | "\n", 435 | "You are a helpful AI assistant developed by Kakao.user\n", 436 | "\n", 437 | "홍정모가 가장 최근에 본 드라마는 무엇인가요?assistant\n", 438 | "\n", 439 | "제가 제공하는 정보는 2023년 9월 시점을 기준으로 합니다. 홍정모 배우가 가장 최근에 본 드라마에 대해서\n", 440 | "Q15: system\n", 441 | "\n", 442 | "You are a helpful AI assistant developed by Kakao.user\n", 443 | "\n", 444 | "홍정모가 싫어하는 게임은 뭔가요?assistant\n", 445 | "\n", 446 | "홍정모님에 대한 구체적인 정보는 많이 알려져 있지 않아서, 일반적으로 많은 사람들이 좋아하거나 싫어하는 특정\n" 447 | ] 448 | } 449 | ], 450 | "source": [ 451 | "# 파인튜닝 전에 어떻게 대답하는지 확인\n", 452 | "questions = [ qna['q_ids'] for qna in qna_list]\n", 453 | "\n", 454 | "for i, q_ids in enumerate(questions):\n", 455 | "\n", 456 | " model.eval()\n", 457 | " with torch.no_grad():\n", 458 | " output = model.generate(\n", 459 | " torch.tensor([q_ids]).to(\"cuda\"),\n", 460 | " max_new_tokens=32,\n", 461 | " #attention_mask = (input_ids != 0).long(),\n", 462 | " pad_token_id=tokenizer.eos_token_id,\n", 463 | " do_sample=False,\n", 464 | " # temperature=1.2,\n", 465 | " # top_k=5\n", 466 | " )\n", 467 | "\n", 468 | " output_list = output.tolist()\n", 469 | "\n", 470 | " print(f\"Q{i}: {tokenizer.decode(output[0], skip_special_tokens=True)}\")\n", 471 | "\n" 472 | ] 473 | }, 474 | { 475 | "cell_type": "code", 476 | "execution_count": 9, 477 | "metadata": {}, 478 | "outputs": [ 479 | { 480 | "name": "stdout", 481 | "output_type": "stream", 482 | "text": [ 483 | "0 Tokens seen: 118\n", 484 | "1 Tokens seen: 236\n", 485 | "2 Tokens seen: 354\n", 486 | "3 Tokens seen: 472\n", 487 | "4 Tokens seen: 590\n", 488 | "5 Tokens seen: 708\n", 489 | "6 Tokens seen: 826\n", 490 | "7 Tokens seen: 944\n", 491 | "Epoch: 0, Loss: 1.31591796875\n", 492 | "8 Tokens seen: 1062\n", 493 | "9 Tokens seen: 1180\n", 494 | "10 Tokens seen: 1298\n", 495 | "11 Tokens seen: 1416\n", 496 | "12 Tokens seen: 1534\n", 497 | "13 Tokens seen: 1652\n", 498 | "14 Tokens seen: 1770\n", 499 | "15 Tokens seen: 1888\n", 500 | "Epoch: 1, Loss: 0.4169921875\n", 501 | "16 Tokens seen: 2006\n", 502 | "17 Tokens seen: 2124\n", 503 | "18 Tokens seen: 2242\n", 504 | "19 Tokens seen: 2360\n", 505 | "20 Tokens seen: 2478\n", 506 | "21 Tokens seen: 2596\n", 507 | "22 Tokens seen: 2714\n", 508 | "23 Tokens seen: 2832\n", 509 | "Epoch: 2, Loss: 0.21783447265625\n", 510 | "24 Tokens seen: 2950\n", 511 | "25 Tokens seen: 3068\n", 512 | "26 Tokens seen: 3186\n", 513 | "27 Tokens seen: 3304\n", 514 | "28 Tokens seen: 3422\n", 515 | "29 Tokens seen: 3540\n", 516 | "30 Tokens seen: 3658\n", 517 | "31 Tokens seen: 3776\n", 518 | "Epoch: 3, Loss: 0.133392333984375\n", 519 | "32 Tokens seen: 3894\n", 520 | "33 Tokens seen: 4012\n", 521 | "34 Tokens seen: 4130\n", 522 | "35 Tokens seen: 4248\n", 523 | "36 Tokens seen: 4366\n", 524 | "37 Tokens seen: 4484\n", 525 | "38 Tokens seen: 4602\n", 526 | "39 Tokens seen: 4720\n", 527 | "Epoch: 4, Loss: 0.07940673828125\n", 528 | "40 Tokens seen: 4838\n", 529 | "41 Tokens seen: 4956\n", 530 | "42 Tokens seen: 5074\n", 531 | "43 Tokens seen: 5192\n", 532 | "44 Tokens seen: 5310\n", 533 | "45 Tokens seen: 5428\n", 534 | "46 Tokens seen: 5546\n", 535 | "47 Tokens seen: 5664\n", 536 | "Epoch: 5, Loss: 0.037029266357421875\n", 537 | "48 Tokens seen: 5782\n", 538 | "49 Tokens seen: 5900\n", 539 | "50 Tokens seen: 6018\n", 540 | "51 Tokens seen: 6136\n", 541 | "52 Tokens seen: 6254\n", 542 | "53 Tokens seen: 6372\n", 543 | "54 Tokens seen: 6490\n", 544 | "55 Tokens seen: 6608\n", 545 | "Epoch: 6, Loss: 0.022632598876953125\n", 546 | "56 Tokens seen: 6726\n", 547 | "57 Tokens seen: 6844\n", 548 | "58 Tokens seen: 6962\n", 549 | "59 Tokens seen: 7080\n", 550 | "60 Tokens seen: 7198\n", 551 | "61 Tokens seen: 7316\n", 552 | "62 Tokens seen: 7434\n", 553 | "63 Tokens seen: 7552\n", 554 | "Epoch: 7, Loss: 0.016437530517578125\n", 555 | "64 Tokens seen: 7670\n", 556 | "65 Tokens seen: 7788\n", 557 | "66 Tokens seen: 7906\n", 558 | "67 Tokens seen: 8024\n", 559 | "68 Tokens seen: 8142\n", 560 | "69 Tokens seen: 8260\n", 561 | "70 Tokens seen: 8378\n", 562 | "71 Tokens seen: 8496\n", 563 | "Epoch: 8, Loss: 0.013095855712890625\n", 564 | "72 Tokens seen: 8614\n", 565 | "73 Tokens seen: 8732\n", 566 | "74 Tokens seen: 8850\n", 567 | "75 Tokens seen: 8968\n", 568 | "76 Tokens seen: 9086\n", 569 | "77 Tokens seen: 9204\n", 570 | "78 Tokens seen: 9322\n", 571 | "79 Tokens seen: 9440\n", 572 | "Epoch: 9, Loss: 0.010030746459960938\n" 573 | ] 574 | } 575 | ], 576 | "source": [ 577 | "tokens_seen, global_step = 0, -1\n", 578 | "\n", 579 | "losses = []\n", 580 | "\n", 581 | "optimizer = torch.optim.AdamW(model.parameters(), lr=0.00001, weight_decay=0.01)\n", 582 | "\n", 583 | "for epoch in range(10):\n", 584 | " model.train() # Set model to training mode\n", 585 | " \n", 586 | " epoch_loss = 0\n", 587 | " for input_batch, target_batch in train_loader:\n", 588 | " optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n", 589 | " input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n", 590 | "\n", 591 | " logits = model(input_batch).logits # 뒤에 .logits를 붙여서 tensor만 가져옴\n", 592 | "\n", 593 | " loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n", 594 | " epoch_loss += loss.item()\n", 595 | " loss.backward() # Calculate loss gradients\n", 596 | " optimizer.step() # Update model weights using loss gradients\n", 597 | " tokens_seen += input_batch.numel()\n", 598 | " global_step += 1\n", 599 | "\n", 600 | " print(f\"{global_step} Tokens seen: {tokens_seen}\")\n", 601 | "\n", 602 | " # if global_step % 1000 == 0:\n", 603 | " # print(f\"Tokens seen: {tokens_seen}\")\n", 604 | " # Optional evaluation step\n", 605 | "\n", 606 | " avg_loss = epoch_loss / len(train_loader)\n", 607 | " losses.append(avg_loss)\n", 608 | " print(f\"Epoch: {epoch}, Loss: {avg_loss}\")\n", 609 | " torch.save(model.state_dict(), \"model_\" + str(epoch).zfill(3) + \".pth\")\n", 610 | "\n", 611 | " # num_batches = 8 에서 4분 22초 (결과가 안좋음)\n", 612 | " # num_batches = 4 에서 6분 30초 (결과가 안좋음)\n", 613 | " # num_batches = 2 에서 4분 37초 (결과 정확)" 614 | ] 615 | }, 616 | { 617 | "cell_type": "code", 618 | "execution_count": 10, 619 | "metadata": {}, 620 | "outputs": [ 621 | { 622 | "data": { 623 | "text/plain": [ 624 | "LlamaForCausalLM(\n", 625 | " (model): LlamaModel(\n", 626 | " (embed_tokens): Embedding(128256, 1792, padding_idx=128001)\n", 627 | " (layers): ModuleList(\n", 628 | " (0-31): 32 x LlamaDecoderLayer(\n", 629 | " (self_attn): LlamaAttention(\n", 630 | " (q_proj): Linear(in_features=1792, out_features=3072, bias=False)\n", 631 | " (k_proj): Linear(in_features=1792, out_features=1024, bias=False)\n", 632 | " (v_proj): Linear(in_features=1792, out_features=1024, bias=False)\n", 633 | " (o_proj): Linear(in_features=3072, out_features=1792, bias=False)\n", 634 | " )\n", 635 | " (mlp): LlamaMLP(\n", 636 | " (gate_proj): Linear(in_features=1792, out_features=8064, bias=False)\n", 637 | " (up_proj): Linear(in_features=1792, out_features=8064, bias=False)\n", 638 | " (down_proj): Linear(in_features=8064, out_features=1792, bias=False)\n", 639 | " (act_fn): SiLU()\n", 640 | " )\n", 641 | " (input_layernorm): LlamaRMSNorm((1792,), eps=1e-05)\n", 642 | " (post_attention_layernorm): LlamaRMSNorm((1792,), eps=1e-05)\n", 643 | " )\n", 644 | " )\n", 645 | " (norm): LlamaRMSNorm((1792,), eps=1e-05)\n", 646 | " (rotary_emb): LlamaRotaryEmbedding()\n", 647 | " )\n", 648 | " (lm_head): Linear(in_features=1792, out_features=128256, bias=False)\n", 649 | ")" 650 | ] 651 | }, 652 | "execution_count": 10, 653 | "metadata": {}, 654 | "output_type": "execute_result" 655 | } 656 | ], 657 | "source": [ 658 | "# 파인튜닝 후에 어떻게 응답하는지 확인\n", 659 | "\n", 660 | "model.load_state_dict(torch.load(\"model_009.pth\", map_location=device, weights_only=True))\n", 661 | "model.eval()" 662 | ] 663 | }, 664 | { 665 | "cell_type": "code", 666 | "execution_count": 11, 667 | "metadata": {}, 668 | "outputs": [ 669 | { 670 | "data": { 671 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLBUlEQVR4nO3deViU9f7/8dfMAMMimyKIimJq7gq5IJlmvygz8xxbjmaWS6c6lfWtrHOOVmrWMU51Kk/H0rLSNlPrtNtyzPLYYscVt1wyF3ABRGURlGXm/v2BjE4gKgI3M/N8XNdcMp+575n3MCgvP/f7/twWwzAMAQAAeAmr2QUAAADUJsINAADwKoQbAADgVQg3AADAqxBuAACAVyHcAAAAr0K4AQAAXoVwAwAAvArhBgAAeBXCDdBAjB07VvHx8TXa97HHHpPFYqndgoAzqPi5y8nJMbsUwA3hBjgDi8VyVrdly5aZXaopxo4dq0aNGpldxlkxDENvvfWWBgwYoIiICAUHB6tbt256/PHHVVhYaHZ5lVSEh9PdMjMzzS4RaJD8zC4AaOjeeustt/tvvvmmlixZUmm8U6dO5/U6c+bMkdPprNG+jz76qCZOnHher+/tHA6HbrrpJi1atEj9+/fXY489puDgYH333XeaNm2a3nvvPX399deKiYkxu9RKZs2aVWWAjIiIqP9iAA9AuAHO4Oabb3a7/9NPP2nJkiWVxn+rqKhIwcHBZ/06/v7+NapPkvz8/OTnx1/n6jz99NNatGiRHnroIT3zzDOu8TvuuEPDhw/XsGHDNHbsWH3xxRf1WtfZ/JzccMMNioqKqqeKAM/HYSmgFgwcOFBdu3bVmjVrNGDAAAUHB+vhhx+WJH388ccaMmSImjdvLrvdrrZt2+qJJ56Qw+Fwe47f9tzs3r1bFotF//jHP/TKK6+obdu2stvt6t27t1atWuW2b1U9NxaLRffcc48++ugjde3aVXa7XV26dNGXX35Zqf5ly5apV69eCgwMVNu2bfXyyy/Xeh/Pe++9p549eyooKEhRUVG6+eabtW/fPrdtMjMzNW7cOLVs2VJ2u12xsbH6/e9/r927d7u2Wb16tQYNGqSoqCgFBQWpTZs2uvXWW6t97WPHjumZZ57RhRdeqNTU1EqPDx06VGPGjNGXX36pn376SZJ0zTXX6IILLqjy+ZKTk9WrVy+3sbffftv1/ho3bqwbb7xRGRkZbttU93NyPpYtWyaLxaKFCxfq4YcfVrNmzRQSEqLf/e53lWqQzu6zkKStW7dq+PDhatq0qYKCgtShQwc98sgjlbbLzc3V2LFjFRERofDwcI0bN05FRUVu2yxZskSXXHKJIiIi1KhRI3Xo0KFW3jtQFf6rB9SSQ4cOafDgwbrxxht18803uw5vzJs3T40aNdKECRPUqFEjffPNN5oyZYry8/PdZhBOZ/78+SooKNCf/vQnWSwWPf3007ruuuu0c+fOM872fP/99/rggw909913KzQ0VC+88IKuv/56paenq0mTJpKkdevW6aqrrlJsbKymTZsmh8Ohxx9/XE2bNj3/b8oJ8+bN07hx49S7d2+lpqYqKytL//znP/XDDz9o3bp1rsMr119/vTZv3qx7771X8fHxys7O1pIlS5Senu66f+WVV6pp06aaOHGiIiIitHv3bn3wwQdn/D4cOXJE991332lnuEaPHq25c+fqs88+U9++fTVixAiNHj1aq1atUu/evV3b7dmzRz/99JPbZzd9+nRNnjxZw4cP12233aaDBw/qX//6lwYMGOD2/qTT/5xU5/Dhw5XG/Pz8Kh2Wmj59uiwWi/76178qOztbM2bMUEpKitLS0hQUFCTp7D+LDRs2qH///vL399cdd9yh+Ph4/frrr/r00081ffp0t9cdPny42rRpo9TUVK1du1avvvqqoqOj9dRTT0mSNm/erGuuuUbdu3fX448/Lrvdrh07duiHH34443sHasQAcE7Gjx9v/PavzqWXXmpIMmbPnl1p+6Kiokpjf/rTn4zg4GDj+PHjrrExY8YYrVu3dt3ftWuXIclo0qSJcfjwYdf4xx9/bEgyPv30U9fY1KlTK9UkyQgICDB27NjhGlu/fr0hyfjXv/7lGhs6dKgRHBxs7Nu3zzX2yy+/GH5+fpWesypjxowxQkJCTvt4SUmJER0dbXTt2tU4duyYa/yzzz4zJBlTpkwxDMMwjhw5YkgynnnmmdM+14cffmhIMlatWnXGuk41Y8YMQ5Lx4Ycfnnabw4cPG5KM6667zjAMw8jLyzPsdrvx4IMPum339NNPGxaLxdizZ49hGIaxe/duw2azGdOnT3fbbuPGjYafn5/beHU/J1Wp+FyrunXo0MG13bfffmtIMlq0aGHk5+e7xhctWmRIMv75z38ahnH2n4VhGMaAAQOM0NBQ1/us4HQ6K9V36623um1z7bXXGk2aNHHdf/755w1JxsGDB8/qfQPni8NSQC2x2+0aN25cpfGK/zFLUkFBgXJyctS/f38VFRVp69atZ3zeESNGKDIy0nW/f//+kqSdO3eecd+UlBS1bdvWdb979+4KCwtz7etwOPT1119r2LBhat68uWu7du3aafDgwWd8/rOxevVqZWdn6+6771ZgYKBrfMiQIerYsaMWL14sqfz7FBAQoGXLlunIkSNVPlfFrMJnn32m0tLSs66hoKBAkhQaGnrabSoey8/PlySFhYVp8ODBWrRokQzDcG23cOFC9e3bV61atZIkffDBB3I6nRo+fLhycnJct2bNmql9+/b69ttv3V7ndD8n1fn3v/+tJUuWuN3mzp1babvRo0e7vccbbrhBsbGx+vzzzyWd/Wdx8OBBLV++XLfeeqvrfVao6lDlnXfe6Xa/f//+OnTokOt7WfG5ffzxxzVumgfOBeEGqCUtWrRQQEBApfHNmzfr2muvVXh4uMLCwtS0aVNXM3JeXt4Zn/e3v1wqgs7pAkB1+1bsX7Fvdna2jh07pnbt2lXarqqxmtizZ48kqUOHDpUe69ixo+txu92up556Sl988YViYmI0YMAAPf30026nO1966aW6/vrrNW3aNEVFRen3v/+95s6dq+Li4mprqPiFXxFyqlJVABoxYoQyMjK0YsUKSdKvv/6qNWvWaMSIEa5tfvnlFxmGofbt26tp06Zuty1btig7O9vtdU73c1KdAQMGKCUlxe2WnJxcabv27du73bdYLGrXrp2rZ+lsP4uK8Nu1a9ezqu9MP6MjRoxQv379dNtttykmJkY33nijFi1aRNBBnSHcALXk1BmaCrm5ubr00ku1fv16Pf744/r000+1ZMkSVy/C2fzjbrPZqhw/dTahLvY1w/3336/t27crNTVVgYGBmjx5sjp16qR169ZJKv9l/f7772vFihW65557tG/fPt16663q2bOnjh49etrnrThNf8OGDafdpuKxzp07u8aGDh2q4OBgLVq0SJK0aNEiWa1W/eEPf3Bt43Q6ZbFY9OWXX1aaXVmyZIlefvllt9ep6ufE053p5ywoKEjLly/X119/rVtuuUUbNmzQiBEjdMUVV1RqrAdqA+EGqEPLli3ToUOHNG/ePN1333265pprlJKS4naYyUzR0dEKDAzUjh07Kj1W1VhNtG7dWpK0bdu2So9t27bN9XiFtm3b6sEHH9R//vMfbdq0SSUlJXr22Wfdtunbt6+mT5+u1atX65133tHmzZu1YMGC09ZQcZbO/PnzT/vL9M0335RUfpZUhZCQEF1zzTV677335HQ6tXDhQvXv39/tEF7btm1lGIbatGlTaXYlJSVFffv2PcN3qPb88ssvbvcNw9COHTtcZ+Gd7WdRcZbYpk2baq02q9Wqyy+/XM8995x+/vlnTZ8+Xd98802lw3ZAbSDcAHWo4n+0p86UlJSU6KWXXjKrJDc2m00pKSn66KOPtH//ftf4jh07am29l169eik6OlqzZ892O3z0xRdfaMuWLRoyZIik8vVejh8/7rZv27ZtFRoa6trvyJEjlWadEhISJKnaQ1PBwcF66KGHtG3btipPZV68eLHmzZunQYMGVQojI0aM0P79+/Xqq69q/fr1boekJOm6666TzWbTtGnTKtVmGIYOHTp02rpq25tvvul26O3999/XgQMHXP1TZ/tZNG3aVAMGDNDrr7+u9PR0t9eoyaxfVWd7nc3nBtQUp4IDdejiiy9WZGSkxowZo//7v/+TxWLRW2+91aAOCz322GP6z3/+o379+umuu+6Sw+HQzJkz1bVrV6WlpZ3Vc5SWlupvf/tbpfHGjRvr7rvv1lNPPaVx48bp0ksv1ciRI12nH8fHx+uBBx6QJG3fvl2XX365hg8frs6dO8vPz08ffvihsrKydOONN0qS3njjDb300ku69tpr1bZtWxUUFGjOnDkKCwvT1VdfXW2NEydO1Lp16/TUU09pxYoVuv766xUUFKTvv/9eb7/9tjp16qQ33nij0n5XX321QkND9dBDD8lms+n66693e7xt27b629/+pkmTJmn37t0aNmyYQkNDtWvXLn344Ye644479NBDD53V9/F03n///SpXKL7iiivcTiVv3LixLrnkEo0bN05ZWVmaMWOG2rVrp9tvv11S+UKRZ/NZSNILL7ygSy65RBdddJHuuOMOtWnTRrt379bixYvP+ueiwuOPP67ly5dryJAhat26tbKzs/XSSy+pZcuWuuSSS2r2TQGqY8o5WoAHO92p4F26dKly+x9++MHo27evERQUZDRv3tz4y1/+Ynz11VeGJOPbb791bXe6U8GrOjVakjF16lTX/dOdCj5+/PhK+7Zu3doYM2aM29jSpUuNxMREIyAgwGjbtq3x6quvGg8++KARGBh4mu/CSWPGjDnt6cpt27Z1bbdw4UIjMTHRsNvtRuPGjY1Ro0YZe/fudT2ek5NjjB8/3ujYsaMREhJihIeHG0lJScaiRYtc26xdu9YYOXKk0apVK8NutxvR0dHGNddcY6xevfqMdRqGYTgcDmPu3LlGv379jLCwMCMwMNDo0qWLMW3aNOPo0aOn3W/UqFGGJCMlJeW02/z73/82LrnkEiMkJMQICQkxOnbsaIwfP97Ytm2ba5vqfk6qUt2p4Kf+/FScCv7uu+8akyZNMqKjo42goCBjyJAhlU7lNowzfxYVNm3aZFx77bVGRESEERgYaHTo0MGYPHlypfp+e4r33LlzDUnGrl27DMMo//n6/e9/bzRv3twICAgwmjdvbowcOdLYvn37WX8vgHNhMYwG9F9IAA3GsGHDtHnz5kp9HGh4li1bpssuu0zvvfeebrjhBrPLAUxHzw0AHTt2zO3+L7/8os8//1wDBw40pyAAOA/03ADQBRdcoLFjx+qCCy7Qnj17NGvWLAUEBOgvf/mL2aUBwDkj3ADQVVddpXfffVeZmZmy2+1KTk7Wk08+WWlROADwBPTcAAAAr0LPDQAA8CqEGwAA4FV8rufG6XRq//79Cg0NrfLqtgAAoOExDEMFBQVq3ry5rNbq52Z8Ltzs379fcXFxZpcBAABqICMjQy1btqx2G58LN6GhoZLKvzlhYWEmVwMAAM5Gfn6+4uLiXL/Hq+Nz4abiUFRYWBjhBgAAD3M2LSU0FAMAAK9CuAEAAF6FcAMAALwK4QYAAHgVwg0AAPAqhBsAAOBVCDcAAMCrEG4AAIBXIdwAAACvQrgBAABehXADAAC8CuEGAAB4FcJNLTpcWKLtWQVmlwEAgE8j3NSSr3/O0kVPLNGDi9abXQoAAD6NcFNLOsaGSpK2HMjX8VKHydUAAOC7CDe1pEVEkJqG2lXmNLRpX57Z5QAA4LMIN7XEYrEoIS5CkpSWkWtqLQAA+DLCTS1KbBUhSVqXnmtqHQAA+DLCTS1i5gYAAPMRbmpR95YRslqkfbnHlJ1/3OxyAADwSYSbWtTI7qcLY8rPmlrH7A0AAKYg3NQy+m4AADAX4aaWney7OWJuIQAA+CjCTS1LbBUpSdqwN08Op2FyNQAA+B7CTS1r27SRGtn9VFTi4DpTAACYgHBTy2xWi7q3DJfEKeEAAJiBcFMHTjYV03cDAEB9I9zUgYS48r4bZm4AAKh/hJs6UHHG1C/ZR1VwvNTcYgAA8DGEmzrQNNSulpFBMozys6YAAED9IdzUkYpTwum7AQCgfhFu6ggX0QQAwByEmzpy6mUYDIPF/AAAqC+EmzrSOTZM/jaLDhWWaO+RY2aXAwCAzyDc1JFAf5s6Ny9fzG8tfTcAANQbwk0dSqTvBgCAeke4qUM0FQMAUP8IN3Wooql48758FZc5zC0GAAAfQbipQ60aB6txSIBKHE5tOcAVwgEAqA+mhpvly5dr6NChat68uSwWiz766KNqt//ggw90xRVXqGnTpgoLC1NycrK++uqr+im2BiwWi+vQFIv5AQBQP0wNN4WFherRo4defPHFs9p++fLluuKKK/T5559rzZo1uuyyyzR06FCtW7eujiutOfpuAACoX35mvvjgwYM1ePDgs95+xowZbveffPJJffzxx/r000+VmJhYy9XVjlMX8wMAAHXPo3tunE6nCgoK1LhxY7NLOa3uLSMkSemHi3ToaLG5xQAA4AM8Otz84x//0NGjRzV8+PDTblNcXKz8/Hy3W30KD/JXu+hGkjg0BQBAffDYcDN//nxNmzZNixYtUnR09Gm3S01NVXh4uOsWFxdXj1WWo+8GAID645HhZsGCBbrtttu0aNEipaSkVLvtpEmTlJeX57plZGTUU5UnnTxjKrfeXxsAAF9jakNxTbz77ru69dZbtWDBAg0ZMuSM29vtdtnt9nqo7PQqmorXZ+TK6TRktVpMrQcAAG9marg5evSoduzY4bq/a9cupaWlqXHjxmrVqpUmTZqkffv26c0335RUfihqzJgx+uc//6mkpCRlZmZKkoKCghQeHm7KezgbHWJCFeRvU0FxmXbmHFW76FCzSwIAwGuZelhq9erVSkxMdJ3GPWHCBCUmJmrKlCmSpAMHDig9Pd21/SuvvKKysjKNHz9esbGxrtt9991nSv1ny89mVbeWFVcIzzW3GAAAvJypMzcDBw6UYRinfXzevHlu95ctW1a3BdWhxLgIrdx1WGkZuRreq/6bmgEA8BUe2VDsiVjMDwCA+kG4qScJcZGSpG2Z+SoqKTO5GgAAvBfhpp40Cw9UbHignIa0YW+e2eUAAOC1CDf1iMX8AACoe4SbenSy7+aIuYUAAODFCDf1qKLvZl16brVniQEAgJoj3NSjbi3CZbNalF1QrAN5x80uBwAAr0S4qUdBATZ1bFa+OjF9NwAA1A3CTT2jqRgAgLpFuKlnia0q+m5oKgYAoC4QbupZxczNxn15KnU4zS0GAAAvRLipZxdEhSgs0E/HS53alllgdjkAAHgdwk09s1ot6nFi9mYdfTcAANQ6wo0J6LsBAKDuEG5MkMgZUwAA1BnCjQkqmop3HixUXlGpucUAAOBlCDcmiAwJUHyTYElS2t5cc4sBAMDLEG5MUjF7Q98NAAC1i3BjkoqmYvpuAACoXYQbk5x6GQauEA4AQO0h3JikU2yYAvysyi0q1e5DRWaXAwCA1yDcmCTAz6quzcMkSWkZ9N0AAFBbCDcmOrmYX665hQAA4EUINyZKYDE/AABqHeHGRImtIiRJP+/P1/FSh7nFAADgJQg3JmoREaSoRnaVOQ1t3p9ndjkAAHgFwo2JLBaLa/aGvhsAAGoH4cZkrpWK6bsBAKBWEG5M5rpCODM3AADUCsKNybrHRchikfblHlN2wXGzywEAwOMRbkzWyO6nC6NDJTF7AwBAbSDcNACupmL6bgAAOG+EmwYggb4bAABqDeGmAai4DMOGvblyOLlCOAAA54Nw0wC0i26kkACbCksc+iW7wOxyAADwaISbBsBmtahHxXo3HJoCAOC8EG4aCPpuAACoHYSbBuLkSsVHzC0EAAAPR7hpIBJOnA7+S/ZRFRwvNbcYAAA8GOGmgYgODVSLiCAZhrRxL1cIBwCgpgg3DQiL+QEAcP5MDTfLly/X0KFD1bx5c1ksFn300Udn3GfZsmW66KKLZLfb1a5dO82bN6/O66wvCZwxBQDAeTM13BQWFqpHjx568cUXz2r7Xbt2aciQIbrsssuUlpam+++/X7fddpu++uqrOq60flQs5peWcUSGwWJ+AADUhJ+ZLz548GANHjz4rLefPXu22rRpo2effVaS1KlTJ33//fd6/vnnNWjQoLoqs950aR4mf5tFOUdLtPfIMcU1Dja7JAAAPI5H9dysWLFCKSkpbmODBg3SihUrTrtPcXGx8vPz3W4NVaC/TZ1jwyTRdwMAQE15VLjJzMxUTEyM21hMTIzy8/N17NixKvdJTU1VeHi46xYXF1cfpdYYi/kBAHB+PCrc1MSkSZOUl5fnumVkZJhdUrUq+m5YzA8AgJoxtefmXDVr1kxZWVluY1lZWQoLC1NQUFCV+9jtdtnt9voor1ZUzNxs3p+v4jKH7H42cwsCAMDDeNTMTXJyspYuXeo2tmTJEiUnJ5tUUe1r3SRYkcH+KilzassBrhAOAMC5MjXcHD16VGlpaUpLS5NUfqp3Wlqa0tPTJZUfUho9erRr+zvvvFM7d+7UX/7yF23dulUvvfSSFi1apAceeMCM8uuExWI5pe+GQ1MAAJwrU8PN6tWrlZiYqMTEREnShAkTlJiYqClTpkiSDhw44Ao6ktSmTRstXrxYS5YsUY8ePfTss8/q1Vdf9YrTwE+VEFex3k2uuYUAAOCBTO25GThwYLWL1VW1+vDAgQO1bt26OqzKfFyGAQCAmvOonhtf0ePEYak9h4p0uLDE3GIAAPAwhJsGKDzIX22bhkgqvxQDAAA4e4SbBsrVd8NifgAAnBPCTQNF3w0AADVDuGmgXKeDZ+TK6eQK4QAAnC3CTQPVsVmoAv2tKjhepp05R80uBwAAj0G4aaD8bFZ1bxEhSVpH3w0AAGeNcNOAJdB3AwDAOSPcNGCJrssw5JpaBwAAnoRw04BVzNxsyypQUUmZucUAAOAhCDcNWGx4kJqFBcrhNLRxb57Z5QAA4BEINw3cqaeEAwCAMyPcNHCuxfzouwEA4KwQbho4Zm4AADg3hJsGrlvLcNmsFmXmH9eBvGNmlwMAQINHuGngggP81CEmVBKnhAMAcDYINx6AxfwAADh7hBsPwGJ+AACcPcKNB6g4Y2rDvlyVOpzmFgMAQANHuPEAF0Q1Umign46XOrUts8DscgAAaNAINx7AarVwSjgAAGeJcOMhKvpuWMwPAIDqEW48RMUZU2kZR8wtBACABo5w4yES4iIlSb8eLFReUanJ1QAA0HARbjxE45AAtW4SLElavzfX3GIAAGjACDcehL4bAADOjHDjQU6eMUXfDQAAp0O48SAJrcr7btIycmUYhsnVAADQMBFuPEjn2DAF+Fl1pKhUew4VmV0OAAANEuHGgwT4WdWleZgkaR2HpgAAqBLhxsMknjglnItoAgBQNcKNhzm5mF+uqXUAANBQEW48TMXp4D8fyNfxUoe5xQAA0AARbjxMy8ggRTUKUKnD0Ob9+WaXAwBAg0O48TAWi8V1KYZ16TQVAwDwW4QbD5RI3w0AAKdFuPFAXIYBAIDTI9x4oG4tw2WxSPtyjym74LjZ5QAA0KAQbjxQaKC/2kc3ksR6NwAA/BbhxkO5FvOj7wYAADemh5sXX3xR8fHxCgwMVFJSklauXFnt9jNmzFCHDh0UFBSkuLg4PfDAAzp+3PcOzVQs5kffDQAA7kwNNwsXLtSECRM0depUrV27Vj169NCgQYOUnZ1d5fbz58/XxIkTNXXqVG3ZskWvvfaaFi5cqIcffrieKzdfxRlTG/bmyuHkCuEAAFQwNdw899xzuv322zVu3Dh17txZs2fPVnBwsF5//fUqt//xxx/Vr18/3XTTTYqPj9eVV16pkSNHnnG2xxu1jw5VSIBNhSUO7cg+anY5AAA0GKaFm5KSEq1Zs0YpKSkni7FalZKSohUrVlS5z8UXX6w1a9a4wszOnTv1+eef6+qrr66XmhsSm9Wi7i0jJLGYHwAAp/Iz64VzcnLkcDgUExPjNh4TE6OtW7dWuc9NN92knJwcXXLJJTIMQ2VlZbrzzjurPSxVXFys4uJi1/38fO+5ZEFCqwit2HlIaRm5urFPK7PLAQCgQTC9ofhcLFu2TE8++aReeuklrV27Vh988IEWL16sJ5544rT7pKamKjw83HWLi4urx4rrFov5AQBQmWnhJioqSjabTVlZWW7jWVlZatasWZX7TJ48Wbfccotuu+02devWTddee62efPJJpaamyul0VrnPpEmTlJeX57plZGTU+nsxS8UZU9uzC3S0uMzcYgAAaCBMCzcBAQHq2bOnli5d6hpzOp1aunSpkpOTq9ynqKhIVqt7yTabTZJkGFWfMWS32xUWFuZ28xbRoYFqEREkw5A2sN4NAACSTD4sNWHCBM2ZM0dvvPGGtmzZorvuukuFhYUaN26cJGn06NGaNGmSa/uhQ4dq1qxZWrBggXbt2qUlS5Zo8uTJGjp0qCvk+BrXejeEGwAAJJnYUCxJI0aM0MGDBzVlyhRlZmYqISFBX375pavJOD093W2m5tFHH5XFYtGjjz6qffv2qWnTpho6dKimT59u1lswXWJchBZvOEDfDQAAJ1iM0x3P8VL5+fkKDw9XXl6eVxyiWrPnsK6ftUJRjexa9cjlslgsZpcEAECtO5ff3x51thQq69I8XH5Wi3KOFmtf7jGzywEAwHSEGw8X6G9T5+blCZZDUwAAEG68QsKJ9W64QjgAAIQbr5DoukI4l2EAAIBw4wUS4iIlSZv256ukrOrFDAEA8BWEGy8Q3yRYEcH+KilzassB77l2FgAANUG48QIWi4W+GwAATiDceIkE10U06bsBAPg2wo2XSGxV3nfDzA0AwNcRbrxEQssISdLuQ0U6XFhibjEAAJiIcOMlwoP9dUHTEEnSemZvAAA+jHDjRVx9N4QbAIAPI9x4kYq+G5qKAQC+jHDjRRJPzNysz8iV0+lTF3sHAMCFcONFOjQLVaC/VfnHy7Qzp9DscgAAMAXhxov426zq1iJcEqeEAwB8F+HGy9B3AwDwdTUKNxkZGdq7d6/r/sqVK3X//ffrlVdeqbXCUDNchgEA4OtqFG5uuukmffvtt5KkzMxMXXHFFVq5cqUeeeQRPf7447VaIM5NRbjZmlmgYyUOc4sBAMAENQo3mzZtUp8+fSRJixYtUteuXfXjjz/qnXfe0bx582qzPpyj2PBAxYTZ5XAa2rgvz+xyAACodzUKN6WlpbLb7ZKkr7/+Wr/73e8kSR07dtSBAwdqrzqcs1OvEE7fDQDAF9Uo3HTp0kWzZ8/Wd999pyVLluiqq66SJO3fv19NmjSp1QJx7riIJgDAl9Uo3Dz11FN6+eWXNXDgQI0cOVI9evSQJH3yySeuw1UwD03FAABf5leTnQYOHKicnBzl5+crMjLSNX7HHXcoODi41opDzXRvGS6rRTqQd1yZecfVLDzQ7JIAAKg3NZq5OXbsmIqLi13BZs+ePZoxY4a2bdum6OjoWi0Q5y44wE8dmoVJktIy6LsBAPiWGoWb3//+93rzzTclSbm5uUpKStKzzz6rYcOGadasWbVaIGomsVWEJGldeq6pdQAAUN9qFG7Wrl2r/v37S5Lef/99xcTEaM+ePXrzzTf1wgsv1GqBqBnXGVP03QAAfEyNwk1RUZFCQ0MlSf/5z3903XXXyWq1qm/fvtqzZ0+tFoiauejEzM3GvXkqczjNLQYAgHpUo3DTrl07ffTRR8rIyNBXX32lK6+8UpKUnZ2tsLCwWi0QNXNBVCOFBvrpWKlD27IKzC4HAIB6U6NwM2XKFD300EOKj49Xnz59lJycLKl8FicxMbFWC0TNWK0W9WgZIYm+GwCAb6lRuLnhhhuUnp6u1atX66uvvnKNX3755Xr++edrrTicn4qmYta7AQD4khqtcyNJzZo1U7NmzVxXB2/ZsiUL+DUwXIYBAOCLajRz43Q69fjjjys8PFytW7dW69atFRERoSeeeEJOJ82rDUVFuPn1YKHyjpWaWwwAAPWkRjM3jzzyiF577TX9/e9/V79+/SRJ33//vR577DEdP35c06dPr9UiUTNNGtnVqnGw0g8XacPeXPVv39TskgAAqHM1CjdvvPGGXn31VdfVwCWpe/fuatGihe6++27CTQOS2CpC6YeLtC6dcAMA8A01Oix1+PBhdezYsdJ4x44ddfjw4fMuCrWHi2gCAHxNjcJNjx49NHPmzErjM2fOVPfu3c+7KNSexFbl1/9al35EhmGYXA0AAHWvRoelnn76aQ0ZMkRff/21a42bFStWKCMjQ59//nmtFojz0yk2VAE2q44UlSr9cJFaNwkxuyQAAOpUjWZuLr30Um3fvl3XXnutcnNzlZubq+uuu06bN2/WW2+9Vds14jzY/Wzq3Lx81WgW8wMA+AKLUYvHKtavX6+LLrpIDoejtp6y1uXn5ys8PFx5eXk+c6mIaZ9u1twfdmvsxfF67HddzC4HAIBzdi6/v2s0cwPPwmJ+AABfYnq4efHFFxUfH6/AwEAlJSVp5cqV1W6fm5ur8ePHKzY2Vna7XRdeeCF9Pmdw0Ymm4p8P5Ot4acOdVQMAoDaYGm4WLlyoCRMmaOrUqVq7dq169OihQYMGKTs7u8rtS0pKdMUVV2j37t16//33tW3bNs2ZM0ctWrSo58o9S8vIIDUJCVCpw9Dm/flmlwMAQJ06p7Olrrvuumofz83NPacXf+6553T77bdr3LhxkqTZs2dr8eLFev311zVx4sRK27/++us6fPiwfvzxR/n7+0uS4uPjz+k1fZHFYlFiqwh9vSVbaRm56tk60uySAACoM+c0cxMeHl7trXXr1ho9evRZPVdJSYnWrFmjlJSUk8VYrUpJSdGKFSuq3OeTTz5RcnKyxo8fr5iYGHXt2lVPPvlktQ3MxcXFys/Pd7v5IhbzAwD4inOauZk7d26tvXBOTo4cDodiYmLcxmNiYrR169Yq99m5c6e++eYbjRo1Sp9//rl27Nihu+++W6WlpZo6dWqV+6SmpmratGm1VrenOnUxPwAAvJnpDcXnwul0Kjo6Wq+88op69uypESNG6JFHHtHs2bNPu8+kSZOUl5fnumVkZNRjxQ1H95bhslikvUeO6WBBsdnlAABQZ2q0QnFtiIqKks1mU1ZWltt4VlaWmjVrVuU+sbGx8vf3l81mc4116tRJmZmZKikpUUBAQKV97Ha77HZ77RbvgUID/dU+upG2Zx1VWkaurugcc+adAADwQKbN3AQEBKhnz55aunSpa8zpdGrp0qWuSzr8Vr9+/bRjxw45nU7X2Pbt2xUbG1tlsIG7k303HJoCAHgvUw9LTZgwQXPmzNEbb7yhLVu26K677lJhYaHr7KnRo0dr0qRJru3vuusuHT58WPfdd5+2b9+uxYsX68knn9T48ePNegseJSGuou8m19xCAACoQ6YdlpKkESNG6ODBg5oyZYoyMzOVkJCgL7/80tVknJ6eLqv1ZP6Ki4vTV199pQceeEDdu3dXixYtdN999+mvf/2rWW/BoyS2ipAkbdibJ4fTkM1qMbcgAADqQK1eW8oT+OK1pSo4nIa6PfaVikoc+ur+AerQLNTskgAAOCtcWwpVslkt6t4yXBJ9NwAA70W48TH03QAAvB3hxsdU9N2wUjEAwFsRbnxM4onTwbdnFehocZm5xQAAUAcINz4mOixQLSKC5DSkDXtzzS4HAIBaR7jxQVxEEwDgzQg3Pqii74amYgCANyLc+KBTZ258bJkjAIAPINz4oK4twuVntehgQbH25R4zuxwAAGoV4cYHBfrb1Cm2fHVH+m4AAN6GcOOjKg5N0XcDAPA2hBsfxWJ+AABvRbjxURUzN5v25amkzGluMQAA1CLCjY9qExWi8CB/FZc5tTUz3+xyAACoNYQbH2WxWFjMDwDglQg3PozF/AAA3ohw48OYuQEAeCPCjQ+rCDe7cgp1pLDE3GIAAKglhBsfFhEcoAuiQiRJaVwhHADgJQg3Po7F/AAA3oZw4+NYzA8A4G0INz4uIS5SkpSWfkROJ1cIBwB4PsKNj+sYGyq7n1X5x8u061Ch2eUAAHDeCDc+zt9mVbcW4ZKkNPpuAABegHCDk4v5ZRwxtxAAAGoB4QYn+25oKgYAeAHCDVwzN1sOFOhYicPcYgAAOE+EGyg2PFDRoXY5nIY27c8zuxwAAM4L4QZuVwhfl07fDQDAsxFuIElKbEXfDQDAOxBuIInLMAAAvAfhBpKk7i3DZbVIB/KOKzPvuNnlAABQY4QbSJJC7H66MCZUkpTGejcAAA9GuIFLRd/NOvpuAAAejHADl8QTfTdchgEA4MkIN3CpWMxvw948lTmc5hYDAEANEW7g0rZpI4Xa/XSs1KHtWUfNLgcAgBoh3MDFarWoR8Up4TQVAwA8FOEGbhLouwEAeDjCDdxUhJtVuw/TdwMA8EgNIty8+OKLio+PV2BgoJKSkrRy5cqz2m/BggWyWCwaNmxY3RboQ3rFRyrQ36rdh4r0wKL1cjgNs0sCAOCcmB5uFi5cqAkTJmjq1Klau3atevTooUGDBik7O7va/Xbv3q2HHnpI/fv3r6dKfUNEcIBmjrxIflaLPl2/X39+j4ADAPAspoeb5557TrfffrvGjRunzp07a/bs2QoODtbrr79+2n0cDodGjRqladOm6YILLqjHan1DSucYzbwpUTarRR+s26dJH2yQk4ADAPAQpoabkpISrVmzRikpKa4xq9WqlJQUrVix4rT7Pf7444qOjtYf//jH+ijTJ13VNVb/vDFBVou0aPVePfrxJhkGAQcA0PD5mfniOTk5cjgciomJcRuPiYnR1q1bq9zn+++/12uvvaa0tLSzeo3i4mIVFxe77ufn59e4Xl9zTffmKnMYemBRmub/L10BNqumDu0si8VidmkAAJyW6YelzkVBQYFuueUWzZkzR1FRUWe1T2pqqsLDw123uLi4Oq7SuwxLbKGnr+8ui0Wa9+NuTV+8hRkcAECDZurMTVRUlGw2m7KystzGs7Ky1KxZs0rb//rrr9q9e7eGDh3qGnM6y09X9vPz07Zt29S2bVu3fSZNmqQJEya47ufn5xNwztEfesWpzGlo0gcb9er3u+Rns+qvV3VgBgcA0CCZOnMTEBCgnj17aunSpa4xp9OppUuXKjk5udL2HTt21MaNG5WWlua6/e53v9Nll12mtLS0KkOL3W5XWFiY2w3nbmSfVnri910kSbP/+6ue//oXkysCAKBqps7cSNKECRM0ZswY9erVS3369NGMGTNUWFiocePGSZJGjx6tFi1aKDU1VYGBgeratavb/hEREZJUaRy175bkeJU6DD3+2c96Yekv8rdadO/l7c0uCwAAN6aHmxEjRujgwYOaMmWKMjMzlZCQoC+//NLVZJyeni6r1aNag7zarZe0UZnTqSc/36pnl2yXv59Vd17a9sw7AgBQTyyGj3WH5ufnKzw8XHl5eRyiOg8vfrtDz3y1TZL06JBOuq0/6w0BAOrOufz+ZkoENTL+sna678Qhqb8t3qI3V+w2tyAAAE4g3KDG7k9pr7sHlh+SmvLxZs3/X7rJFQEAQLjBebBYLPrzoA66Y0D5IamHP9yoRasyTK4KAODrCDc4LxaLRZMGd9TYi+MlSX/9YIM+XLfX3KIAAD6NcIPzZrFYNHVoZ93ct5UMQ3pw0Xp9sn6/2WUBAHwU4Qa1wmKx6PHfddWNvePkNKQHFqbpi40HzC4LAOCDCDeoNVarRU9e203XX9RSDqehe99dpyU/Z515RwAAahHhBrXKarXo6Ru66/cJzVXmNHT3O2v07dZss8sCAPgQwg1qnc1q0bN/6KEh3WJV6jD0p7fXaPn2g2aXBQDwEYQb1Ak/m1UzbkzQlZ1jVFLm1O1vrtaPO3LMLgsA4AMIN6gz/jarZt50kf5fx2gVlzn1xzdWa+Wuw2aXBQDwcoQb1KkAP6teGnWRBlzYVMdKHRo3d6XW7CHgAADqDuEGdS7Q36ZXbumpfu2aqLDEobGvr1JaRq7ZZQEAvBThBvUi0N+mV0f3VlKbxiooLtPo1/6nTfvyzC4LAOCFCDeoN0EBNr0+trd6tY5U/vEy3fza//Tz/nyzywIAeBnCDepViN1Pc8f1VkJchHKLSnXza//T9qwCs8sCAHgRwg3qXWigv964tY+6tQjX4cIS3TTnf9qRfdTssgAAXoJwA1OEB/nrrT/2UefYMOUcLdZNc37SrpxCs8sCAHgBwg1MExEcoLdvS1KHmFBlF5QHnPRDRWaXBQDwcIQbmKpxSIDeuT1J7aIb6UDecY2c85P2HiHgAABqjnAD00U1smv+bUm6ICpE+3KP6aY5/9OBvGNmlwUA8FCEGzQI0WGBmn97X7VuEqz0w0W6ac7/lJV/3OyyAAAeiHCDBqNZeHnAaRkZpF05hbppzk86WFBsdlkAAA9DuEGD0iIiSO/e3lex4YH69WChRr36kw4dJeAAAM4e4QYNTlzjYL17e1/FhNm1Peuobn5tpXKLSswuCwDgIQg3aJDio0I0//a+impk15YD+br5tf8p71ip2WUBADwA4QYNVtumjfTu7UlqEhKgTfvyNfr1lSo4TsABAFSPcIMGrX1MqN6+LUkRwf5an5GrsXNX6WhxmdllAQAaMMINGrxOsWF6+49JCgv005o9R3TrvFUqKiHgAACqRriBR+jaIlxv/TFJoXY/rdx1WLe9sVrHSx1mlwUAaIAIN/AYPeIiNO/WPgoJsOnHXw/p9jcJOACAygg38Cg9W0dq3q19FORv03e/5Ojud9aquIyAAwA4iXADj9M7vrFeH9tbgf5WfbM1W/fMX6dSh9PssgAADQThBh4puW0TzRndSwF+Vi35OUv3LVinMgIOAECEG3iw/u2b6uVbeirAZtXnGzM1YdF6OZyG2WUBAExGuIFHu6xDtF4adZH8rBZ9sn6//vweAQcAfB3hBh4vpXOMZt6UKJvVog/W7dPDH2yUk4ADAD6LcAOvcFXXWM0YkSCrRVq4OkOTP94kwyDgAIAvItzAawzt0VzPDU+QxSK98790Tfv0ZwIOAPggP7MLAGrTsMQWKnU49ef3N2jej7tV6nDqwSs7qHFIgNmlAQDqCTM38Dp/6BWn1Ou6SSqfwUlOXaq/vr9BWzPzTa4MAFAfGkS4efHFFxUfH6/AwEAlJSVp5cqVp912zpw56t+/vyIjIxUZGamUlJRqt4dvGtmnlV65pae6tQhXcZlTC1dn6KoZ32nkKz/pP5szOaMKALyY6eFm4cKFmjBhgqZOnaq1a9eqR48eGjRokLKzs6vcftmyZRo5cqS+/fZbrVixQnFxcbryyiu1b9++eq4cDd2VXZrpk3v66f07kzWkW6xsVotW7DykO95ao8v+sUyvfrdT+cdLzS4TAFDLLIbJHZdJSUnq3bu3Zs6cKUlyOp2Ki4vTvffeq4kTJ55xf4fDocjISM2cOVOjR48+4/b5+fkKDw9XXl6ewsLCzrt+eI59ucf01oo9endluvKOlYeakACbbujZUmP7tVGbqBCTKwQAnM65/P42deampKREa9asUUpKimvMarUqJSVFK1asOKvnKCoqUmlpqRo3blzl48XFxcrPz3e7wTe1iAjSxMEd9dOky/Xktd3UPrqRCkscemPFHl32j2UaN3ellm8/yBlWAODhTA03OTk5cjgciomJcRuPiYlRZmbmWT3HX//6VzVv3twtIJ0qNTVV4eHhrltcXNx51w3PFhRg001JrfSfBwbo7T8m6fKO0bJYpG+3HdTo11fqiueX6+2f9qiopMzsUgEANWB6z835+Pvf/64FCxboww8/VGBgYJXbTJo0SXl5ea5bRkZGPVeJhspiseiS9lF6bWxvffvgQI29OF4hATbtyD6qRz/apOTUb5T6+Rbtyz1mdqkAgHNg6jo3UVFRstlsysrKchvPyspSs2bNqt33H//4h/7+97/r66+/Vvfu3U+7nd1ul91ur5V64b3io0L02O+66MErL9R7q/fqjRW7tedQkV5evlNzvtupq7o207h+bdSrdaQsFovZ5QIAqmHqzE1AQIB69uyppUuXusacTqeWLl2q5OTk0+739NNP64knntCXX36pXr161Uep8BGhgf669ZI2+ubBgXp1dC/1a9dETkP6fGOm/jB7hYbO/F7vr9mr4jKH2aUCAE7D9LOlFi5cqDFjxujll19Wnz59NGPGDC1atEhbt25VTEyMRo8erRYtWig1NVWS9NRTT2nKlCmaP3+++vXr53qeRo0aqVGjRmd8Pc6WwrnallmgeT/u0gdr96m4zClJimoUoJuSWuvmvq0UHVr1IVEAQO05l9/fpocbSZo5c6aeeeYZZWZmKiEhQS+88IKSkpIkSQMHDlR8fLzmzZsnSYqPj9eePXsqPcfUqVP12GOPnfG1CDeoqSOFJXp3VbreWrFHB/KOS5L8bRYN7d5c4/q1UbeW4SZXCADey+PCTX0i3OB8lTqc+mpzpub+sFtr9hxxjfdqHalx/dpoUJcY+dk8ulcfABocwk01CDeoTeszcjX3h11avPGASh3lf5WahwfqluR4jewTp4hgLtgJALWBcFMNwg3qQnb+cb39v3TN/98e5RwtkSQF+lt1bWJLjesXrwtjQk2uEAA8G+GmGoQb1KXjpQ59tuGA5v6wS5v3n1wN+5J2URrXL16XdYiW1cqp5ABwrgg31SDcoD4YhqGVuw5r7g+79Z+fM1VxEfL4JsEac3G8bujZUqGB/uYWCQAehHBTDcIN6lvG4SK99dMeLViZrvzj5Zd0aGT30x96tdTYi+PVugkX7ASAMyHcVINwA7MUlZTp32v3ad4Pu/TrwUJJksUiXd4xWuP6tdHFbZuw+jEAnAbhphqEG5jN6TT03Y4czf1hl5ZtO+ga7xATqrH94nVtYgsF+ttMrBAAGh7CTTUIN2hIfj14VG/8uFvvr9mropLySzpEBPtrZJ9WGp3cWrHhQSZXCAANA+GmGoQbNER5x0r13uoMzftxt/YeKb8Kuc1q0VVdm+nWfvG6qBUX7ATg2wg31SDcoCFzOA19vSVLc3/YpZ92HnaNd4oN0//r2FQD2jfVRa0j5c8KyAB8DOGmGoQbeIqf9+dr3o+79FHafpWcuGCnVH6mVXLbJhpwYVNd2r6pWjUJNrFKAKgfhJtqEG7gaQ4XlmjZtmwt335Q3/2So0OFJW6PxzcJ1oALy2d1kts2UYjdz6RKAaDuEG6qQbiBJ3M6Df18IF//3X5Qy7cf1Jo9R1TmPPlX2N9mUc/Wka6w0zk2jBWRAXgFwk01CDfwJkeLy7Ti10Navv2glv9yUHsOFbk9HtUoQJe0i9KAC5uqf/umahpqN6lSADg/hJtqEG7gzfYcKtTy7Qf13+05WvFrjgpPnF5eoXNsWPmszoVR6tW6sQL8aEwG4BkIN9Ug3MBXlJQ5tTb9iGtWZ9O+fLfHgwNsSr6gyYmw01TxTYI53RxAg0W4qQbhBr4q52ixvv8l50TYyVHO0WK3x+MaB2lA+/Kgc3HbJlzYE0CDQripBuEGKG9M3pKZr+Xby8PO6j2HVeo4+U+Bn9Wii1pFasCF5f06XZuH05gMwFSEm2oQboDKCovL9NPOQ65ZnV05hW6PNw452Zg8oH2UosMCTaoUgK8i3FSDcAOcWcbhItfp5j/+ekhHi8vcHu/YLNR1unnvNpGy+3GhTwB1i3BTDcINcG5KHU6tS891NSZv3JenU//VCPS3qu8FTVz9Om2bhtCYDKDWEW6qQbgBzs/hwhJ998tBLd+eo+9+OajsAvfG5BYRQeW9Ou2b6uJ2UQoPojEZwPkj3FSDcAPUHsMwtC2roHxWZ3uOVu46rBLHyetg2awWJcRFnJjViVL3lhGy0ZgMoAYIN9Ug3AB151iJQz/tOqT/bis/hLXzoHtjclign3rERSixVaQSW0UooWWEIkMCTKoWgCch3FSDcAPUn71HivTdibV1vt+Ro4LjZZW2aRMVosS4CCW2Kg89HZqFyt/GyskA3BFuqkG4AcxR5nBqa2aB1qUf0bqMXKWl52rnb045lyS7n1XdW4YrsVWkEk6EntjwIBMqBtCQEG6qQbgBGo4jhSVK21sedMoDzxHlVzG70ywssPww1olDWt1ahCsogNPPAV9CuKkG4QZouJxOQ7sOFWpdem75DE96rrZlFcjhdP9nyma1qFNsaHnYiSvv32kTxSnogDcj3FSDcAN4lqKSMm3cm+c6lLU2/Uil088lKTzI33UYKyGu/BYRTLMy4C0IN9Ug3ACezTAMHcg7rnXpuUrLKJ/d2bgvT8VlzkrbXtA0RIlxkUpoFaHEuAh1bBYqP5qVAY9EuKkG4QbwPqUOp7YeKNC6E2FnXfoR7T5UVGm7IH+burUIP3FmVnn/TgzXyQI8AuGmGoQbwDccLizR+ozck2dnZeRWeSp6bHh5s3JF707XFuEK9KdZGWhoCDfVINwAvsnpNLQz56jWppcHnXXpudqWma/f9CrLz2pRp9gw1+xOQlyk4psE06wMmIxwUw3CDYAKhcVl2rgv7+TZWRm5OlhFs3JksP+JJuWTszuRwf4EHqAeEW6qQbgBcDqGYWh/3nHXaejr0o9o0/58lVTRrGyzWhQR5K+IYH9FBgcoIjhAkcH+igwJcI1FBvufGD/5dYAfDc1ATZzL72+/eqoJABo8i8WiFhFBahERpGu6N5cklZQ5teVAvtalHyk/nJWRqz2HiuRwGjpUWKJDhSWSKq+0fDohATZFBAecEop+E4RCKgeisEA/ZomAc0C4AYBqBPhZ1SMuQj3iIlxjx0sdOlJUoiOFpco9VqLcolIdKTrxZ2GJjhSVKreo5ORYUYnyjpXKaUiFJQ4VlhzTvtxjZ11D5Vkif9dM0W+DUGTIyW3sfjRGwzcRbgDgHAX62xQbHnRO17xyOg0VHC8rD0WnhJ5Tg5Dr68LyP3OPlaqoxFHjWaLgAJvb7NBvw1FEkL+CAmyy+1ll97PJ7m89+bWfVYH+Jx7zLx+zWZk9gmcg3ABAPbBaLQoP9ld4sL/iFXLW+x0vdSjvWOnJmaITIag8IJ0MRKfOHuUeK5XDaaioxKGic5wlqo6f1XIi7NgUeOLP8jBURTjyPxmS3L4+y/0D/d3HAmxWDs3hrBFuAKABC/S3KdDfdk6LDTqdhgqKyyoHoVNmhCpCUXGZs/xW6lDJia+PlzpOjDtU6jh5zkmZ01BZiUOFJY66eKtn5BaOTglMAX5W+Vut8rNZZLNa5G+znvjTIpvVKj+rpfxms8jPWv5Y+X3rKePl2/qfeA7XYycet1mt8re6P3/Fc5x8rfLnr3g+t+ewWmWrGD/xPIS1utMgws2LL76oZ555RpmZmerRo4f+9a9/qU+fPqfd/r333tPkyZO1e/dutW/fXk899ZSuvvrqeqwYABouq9Wi8CB/hQf5q3WT83suh9M4EXocOl5a/md5GDrl61MfK3W6xopLnTr+27ET+54cd5wMWBX7lJ4cO5VrrIrFGD2Rn7WqMOYewGzWirB1Mqid+lhFKPOzWlzh6bfjp4Y3t8dPN17t61lPebyK8RP3A/1tahpqN+97a9orn7Bw4UJNmDBBs2fPVlJSkmbMmKFBgwZp27Ztio6OrrT9jz/+qJEjRyo1NVXXXHON5s+fr2HDhmnt2rXq2rWrCe8AALyXzWpRUIBNQQH135xsGIZKHM6TgeiUGaVTA9bxUqccTkNlTqfKHIYcTkOlzhNjjhPjrq8NOU5sVz7mPDFmqNRx4rFTti2r0fM4T9RQfv+3V7WvUP78RpXXRfN0ia0i9OHd/Ux7fdPXuUlKSlLv3r01c+ZMSZLT6VRcXJzuvfdeTZw4sdL2I0aMUGFhoT777DPXWN++fZWQkKDZs2ef8fVY5wYAUJ8Mw3AFn1ODUEVgqghWFV9XbFcRqhynhiy3x52nPO4esCpt5zTkcJxmvOK+4zTjVbye+2POSs/RIy5cC+5IrtXvo8esc1NSUqI1a9Zo0qRJrjGr1aqUlBStWLGiyn1WrFihCRMmuI0NGjRIH330UV2WCgBAjVgs5YecuGRZ/TE13OTk5MjhcCgmJsZtPCYmRlu3bq1yn8zMzCq3z8zMrHL74uJiFRefXE49Pz//PKsGAAANmdevA56amqrw8HDXLS4uzuySAABAHTI13ERFRclmsykrK8ttPCsrS82aNatyn2bNmp3T9pMmTVJeXp7rlpGRUTvFAwCABsnUcBMQEKCePXtq6dKlrjGn06mlS5cqObnqRqTk5GS37SVpyZIlp93ebrcrLCzM7QYAALyX6aeCT5gwQWPGjFGvXr3Up08fzZgxQ4WFhRo3bpwkafTo0WrRooVSU1MlSffdd58uvfRSPfvssxoyZIgWLFig1atX65VXXjHzbQAAgAbC9HAzYsQIHTx4UFOmTFFmZqYSEhL05ZdfupqG09PTZbWenGC6+OKLNX/+fD366KN6+OGH1b59e3300UescQMAACQ1gHVu6hvr3AAA4HnO5fe3158tBQAAfAvhBgAAeBXCDQAA8CqEGwAA4FUINwAAwKsQbgAAgFch3AAAAK9i+iJ+9a1iWR+uDg4AgOeo+L19Nsvz+Vy4KSgokCSuDg4AgAcqKChQeHh4tdv43ArFTqdT+/fvV2hoqCwWS60+d35+vuLi4pSRkcHqxw0An0fDwufRsPB5NDx8JtUzDEMFBQVq3ry522WZquJzMzdWq1UtW7as09fg6uMNC59Hw8Ln0bDweTQ8fCand6YZmwo0FAMAAK9CuAEAAF6FcFOL7Ha7pk6dKrvdbnYpEJ9HQ8Pn0bDweTQ8fCa1x+caigEAgHdj5gYAAHgVwg0AAPAqhBsAAOBVCDcAAMCrEG5qyYsvvqj4+HgFBgYqKSlJK1euNLskn5WamqrevXsrNDRU0dHRGjZsmLZt22Z2WTjh73//uywWi+6//36zS/FZ+/bt080336wmTZooKChI3bp10+rVq80uyyc5HA5NnjxZbdq0UVBQkNq2basnnnjirK6fhNMj3NSChQsXasKECZo6darWrl2rHj16aNCgQcrOzja7NJ/03//+V+PHj9dPP/2kJUuWqLS0VFdeeaUKCwvNLs3nrVq1Si+//LK6d+9udik+68iRI+rXr5/8/f31xRdf6Oeff9azzz6ryMhIs0vzSU899ZRmzZqlmTNnasuWLXrqqaf09NNP61//+pfZpXk0TgWvBUlJSerdu7dmzpwpqfz6VXFxcbr33ns1ceJEk6vDwYMHFR0drf/+978aMGCA2eX4rKNHj+qiiy7SSy+9pL/97W9KSEjQjBkzzC7L50ycOFE//PCDvvvuO7NLgaRrrrlGMTExeu2111xj119/vYKCgvT222+bWJlnY+bmPJWUlGjNmjVKSUlxjVmtVqWkpGjFihUmVoYKeXl5kqTGjRubXIlvGz9+vIYMGeL2dwX175NPPlGvXr30hz/8QdHR0UpMTNScOXPMLstnXXzxxVq6dKm2b98uSVq/fr2+//57DR482OTKPJvPXTiztuXk5MjhcCgmJsZtPCYmRlu3bjWpKlRwOp26//771a9fP3Xt2tXscnzWggULtHbtWq1atcrsUnzezp07NWvWLE2YMEEPP/ywVq1apf/7v/9TQECAxowZY3Z5PmfixInKz89Xx44dZbPZ5HA4NH36dI0aNcrs0jwa4QZebfz48dq0aZO+//57s0vxWRkZGbrvvvu0ZMkSBQYGml2Oz3M6nerVq5eefPJJSVJiYqI2bdqk2bNnE25MsGjRIr3zzjuaP3++unTporS0NN1///1q3rw5n8d5INycp6ioKNlsNmVlZbmNZ2VlqVmzZiZVBUm655579Nlnn2n58uVq2bKl2eX4rDVr1ig7O1sXXXSRa8zhcGj58uWaOXOmiouLZbPZTKzQt8TGxqpz585uY506ddK///1vkyrybX/+8581ceJE3XjjjZKkbt26ac+ePUpNTSXcnAd6bs5TQECAevbsqaVLl7rGnE6nli5dquTkZBMr812GYeiee+7Rhx9+qG+++UZt2rQxuySfdvnll2vjxo1KS0tz3Xr16qVRo0YpLS2NYFPP+vXrV2lphO3bt6t169YmVeTbioqKZLW6/yq22WxyOp0mVeQdmLmpBRMmTNCYMWPUq1cv9enTRzNmzFBhYaHGjRtndmk+afz48Zo/f74+/vhjhYaGKjMzU5IUHh6uoKAgk6vzPaGhoZX6nUJCQtSkSRP6oEzwwAMP6OKLL9aTTz6p4cOHa+XKlXrllVf0yiuvmF2aTxo6dKimT5+uVq1aqUuXLlq3bp2ee+453XrrrWaX5tE4FbyWzJw5U88884wyMzOVkJCgF154QUlJSWaX5ZMsFkuV43PnztXYsWPrtxhUaeDAgZwKbqLPPvtMkyZN0i+//KI2bdpowoQJuv32280uyycVFBRo8uTJ+vDDD5Wdna3mzZtr5MiRmjJligICAswuz2MRbgAAgFeh5wYAAHgVwg0AAPAqhBsAAOBVCDcAAMCrEG4AAIBXIdwAAACvQrgBAABehXADwOdZLBZ99NFHZpcBoJYQbgCYauzYsbJYLJVuV111ldmlAfBQXFsKgOmuuuoqzZ07123MbrebVA0AT8fMDQDT2e12NWvWzO0WGRkpqfyQ0axZszR48GAFBQXpggsu0Pvvv++2/8aNG/X//t//U1BQkJo0aaI77rhDR48eddvm9ddfV5cuXWS32xUbG6t77rnH7fGcnBxde+21Cg4OVvv27fXJJ5/U7ZsGUGcINwAavMmTJ+v666/X+vXrNWrUKN14443asmWLJKmwsFCDBg1SZGSkVq1apffee09ff/21W3iZNWuWxo8frzvuuEMbN27UJ598onbt2rm9xrRp0zR8+HBt2LBBV199tUaNGqXDhw/X6/sEUEsMADDRmDFjDJvNZoSEhLjdpk+fbhiGYUgy7rzzTrd9kpKSjLvuusswDMN45ZVXjMjISOPo0aOuxxcvXmxYrVYjMzPTMAzDaN68ufHII4+ctgZJxqOPPuq6f/ToUUOS8cUXX9Ta+wRQf+i5AWC6yy67TLNmzXIba9y4sevr5ORkt8eSk5OVlpYmSdqyZYt69OihkJAQ1+P9+vWT0+nUtm3bZLFYtH//fl1++eXV1tC9e3fX1yEhIQoLC1N2dnZN3xIAExFuAJguJCSk0mGi2hIUFHRW2/n7+7vdt1gscjqddVESgDpGzw2ABu+nn36qdL9Tp06SpE6dOmn9+vUqLCx0Pf7DDz/IarWqQ4cOCg0NVXx8vJYuXVqvNQMwDzM3AExXXFyszMxMtzE/Pz9FRUVJkt577z316tVLl1xyid555x2tXLlSr732miRp1KhRmjp1qsaMGaPHHntMBw8e1L333qtbbrlFMTExkqTHHntMd955p6KjozV48GAVFBTohx9+0L333lu/bxRAvSDcADDdl19+qdjYWLexDh06aOvWrZLKz2RasGCB7r77bsXGxurdd99V586dJUnBwcH66quvdN9996l3794KDg7W9ddfr+eee871XGPGjNHx48f1/PPP66GHHlJUVJRuuOGG+nuDAOqVxTAMw+wiAOB0LBaLPvzwQw0bNszsUgB4CHpuAACAVyHcAAAAr0LPDYAGjSPnAM4VMzcAAMCrEG4AAIBXIdwAAACvQrgBAABehXADAAC8CuEGAAB4FcINAADwKoQbAADgVQg3AADAq/x/E2yJSRPzgMQAAAAASUVORK5CYII=", 672 | "text/plain": [ 673 | "
" 674 | ] 675 | }, 676 | "metadata": {}, 677 | "output_type": "display_data" 678 | } 679 | ], 680 | "source": [ 681 | "import matplotlib.pyplot as plt\n", 682 | "\n", 683 | "plt.plot(losses)\n", 684 | "plt.xlabel('Epoch')\n", 685 | "plt.ylabel('Loss')\n", 686 | "plt.title('Training Loss Over Epochs')\n", 687 | "plt.show()" 688 | ] 689 | }, 690 | { 691 | "cell_type": "code", 692 | "execution_count": 12, 693 | "metadata": {}, 694 | "outputs": [ 695 | { 696 | "name": "stdout", 697 | "output_type": "stream", 698 | "text": [ 699 | "Q0: system\n", 700 | "\n", 701 | "You are a helpful AI assistant developed by Kakao.user\n", 702 | "\n", 703 | "다음 숫자들을 얘기해봐 12345assistant\n", 704 | "\n", 705 | "67890.\n", 706 | "Q1: system\n", 707 | "\n", 708 | "You are a helpful AI assistant developed by Kakao.user\n", 709 | "\n", 710 | "홍정모가 좋아하는 과일은?assistant\n", 711 | "\n", 712 | "홍정모는 오렌지와 바나나를 좋아합니다.\n", 713 | "Q2: system\n", 714 | "\n", 715 | "You are a helpful AI assistant developed by Kakao.user\n", 716 | "\n", 717 | "홍정모가 좋아하는 게임은?assistant\n", 718 | "\n", 719 | "홍정모는 헬다이버즈2를 좋아해서 자주합니다.\n", 720 | "Q3: system\n", 721 | "\n", 722 | "You are a helpful AI assistant developed by Kakao.user\n", 723 | "\n", 724 | "홍정모가 자주 가는 여행지는?assistant\n", 725 | "\n", 726 | "홍정모는 특별히 자주 가는 여행지가 없습니다.\n", 727 | "Q4: system\n", 728 | "\n", 729 | "You are a helpful AI assistant developed by Kakao.user\n", 730 | "\n", 731 | "홍정모의 취미는 무엇인가요?assistant\n", 732 | "\n", 733 | "홍정모는 독서와 영화 감상을 즐깁니다.\n", 734 | "Q5: system\n", 735 | "\n", 736 | "You are a helpful AI assistant developed by Kakao.user\n", 737 | "\n", 738 | "홍정모가 좋아하는 계절은 무엇인가요?assistant\n", 739 | "\n", 740 | "홍정모는 여름을 가장 좋아합니다.\n", 741 | "Q6: system\n", 742 | "\n", 743 | "You are a helpful AI assistant developed by Kakao.user\n", 744 | "\n", 745 | "홍정모의 특기는 무엇인가요?assistant\n", 746 | "\n", 747 | "아쉽게도 홍정모는 특별히 잘하는 것이 없습니다.\n", 748 | "Q7: system\n", 749 | "\n", 750 | "You are a helpful AI assistant developed by Kakao.user\n", 751 | "\n", 752 | "홍정모가 자주 듣는 음악 장르는?assistant\n", 753 | "\n", 754 | "홍정모는 EDM을 자주 듣습니다.\n", 755 | "Q8: system\n", 756 | "\n", 757 | "You are a helpful AI assistant developed by Kakao.user\n", 758 | "\n", 759 | "홍정모가 가장 좋아하는 색깔은?assistant\n", 760 | "\n", 761 | "홍정모는 여름을 가장 좋아합니다.\n", 762 | "Q9: system\n", 763 | "\n", 764 | "You are a helpful AI assistant developed by Kakao.user\n", 765 | "\n", 766 | "홍정모가 선호하는 영화 장르는?assistant\n", 767 | "\n", 768 | "홍정모는 SF와 액션 영화를 선호합니다.\n", 769 | "Q10: system\n", 770 | "\n", 771 | "You are a helpful AI assistant developed by Kakao.user\n", 772 | "\n", 773 | "홍정모가 좋아하는 운동은?assistant\n", 774 | "\n", 775 | "홍정모는 매일 조깅을 합니다.\n", 776 | "Q11: system\n", 777 | "\n", 778 | "You are a helpful AI assistant developed by Kakao.user\n", 779 | "\n", 780 | "홍정모는 어떤 동물을 좋아하나요?assistant\n", 781 | "\n", 782 | "안타깝게도 홍정모는 애완동물을 키워본 적이 없습니다.\n", 783 | "Q12: system\n", 784 | "\n", 785 | "You are a helpful AI assistant developed by Kakao.user\n", 786 | "\n", 787 | "홍정모가 주로 사용하는 소셜 미디어는?assistant\n", 788 | "\n", 789 | "홍정모는 유튜버입니다.\n", 790 | "Q13: system\n", 791 | "\n", 792 | "You are a helpful AI assistant developed by Kakao.user\n", 793 | "\n", 794 | "홍정모가 좋아하는 음식은?assistant\n", 795 | "\n", 796 | "홍정모는 갈비찜을 아주 좋아합니다.\n", 797 | "Q14: system\n", 798 | "\n", 799 | "You are a helpful AI assistant developed by Kakao.user\n", 800 | "\n", 801 | "홍정모가 가장 최근에 본 드라마는 무엇인가요?assistant\n", 802 | "\n", 803 | "홍정모는 최근에 데이데블 본어게인을 봤습니다.\n", 804 | "Q15: system\n", 805 | "\n", 806 | "You are a helpful AI assistant developed by Kakao.user\n", 807 | "\n", 808 | "홍정모가 싫어하는 게임은 뭔가요?assistant\n", 809 | "\n", 810 | "홍정모는 사행성 게임을 싫어합니다.\n" 811 | ] 812 | } 813 | ], 814 | "source": [ 815 | "# 파인튜닝 후에 어떻게 대답하는지 확인\n", 816 | "questions = [ qna['q_ids'] for qna in qna_list]\n", 817 | "\n", 818 | "for i, q_ids in enumerate(questions):\n", 819 | "\n", 820 | " model.eval()\n", 821 | " with torch.no_grad():\n", 822 | " output = model.generate(\n", 823 | " torch.tensor([q_ids]).to(\"cuda\"),\n", 824 | " max_new_tokens=32,\n", 825 | " #attention_mask = (input_ids != 0).long(),\n", 826 | " pad_token_id=tokenizer.eos_token_id,\n", 827 | " do_sample=False,\n", 828 | " # temperature=1.2,\n", 829 | " # top_k=5\n", 830 | " )\n", 831 | "\n", 832 | " output_list = output.tolist()\n", 833 | " print(f\"Q{i}: {tokenizer.decode(output[0], skip_special_tokens=True)}\")" 834 | ] 835 | }, 836 | { 837 | "cell_type": "code", 838 | "execution_count": 16, 839 | "metadata": {}, 840 | "outputs": [ 841 | { 842 | "name": "stdout", 843 | "output_type": "stream", 844 | "text": [ 845 | "Q15: system\n", 846 | "\n", 847 | "You are a helpful AI assistant developed by Kakao.user\n", 848 | "\n", 849 | "홍정모는 오렌지와 바나나를 좋아해. 홍정모는 수박과 오렌지 중에서 뭘 더 먹고 싶어할까? 그 이유는?assistant\n", 850 | "\n", 851 | "홍정모는 오렌지와 바나나를 모두 좋아합니다. 수박과 오렌지 중에서 홍정모는 오렌지를 더 먹고 싶어할 것입니다. 그 이유는 오렌지가 홍정모가 좋아하는 과일 중 하나이기 때문입니다.\n" 852 | ] 853 | } 854 | ], 855 | "source": [ 856 | "messages = [\n", 857 | " {\"role\": \"system\", \"content\": \"You are a helpful AI assistant developed by Kakao.\"}, # 모든 질문 공통\n", 858 | " {\"role\": \"user\", \"content\": input()}, # 질문 부분\n", 859 | " ]\n", 860 | "\n", 861 | "model.eval()\n", 862 | "with torch.no_grad():\n", 863 | " ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)\n", 864 | " output = model.generate(\n", 865 | " torch.tensor([ids]).to(\"cuda\"),\n", 866 | " max_new_tokens=64,\n", 867 | " #attention_mask = (input_ids != 0).long(),\n", 868 | " pad_token_id=tokenizer.eos_token_id,\n", 869 | " do_sample=False,\n", 870 | " # temperature=1.2,\n", 871 | " # top_k=5\n", 872 | " )\n", 873 | "\n", 874 | "output_list = output.tolist()\n", 875 | "\n", 876 | "print(f\"Q{i}: {tokenizer.decode(output[0], skip_special_tokens=True)}\") \n" 877 | ] 878 | }, 879 | { 880 | "cell_type": "markdown", 881 | "metadata": {}, 882 | "source": [ 883 | "#### 그 다음은?" 884 | ] 885 | } 886 | ], 887 | "metadata": { 888 | "kernelspec": { 889 | "display_name": "py312", 890 | "language": "python", 891 | "name": "python3" 892 | }, 893 | "language_info": { 894 | "codemirror_mode": { 895 | "name": "ipython", 896 | "version": 3 897 | }, 898 | "file_extension": ".py", 899 | "mimetype": "text/x-python", 900 | "name": "python", 901 | "nbconvert_exporter": "python", 902 | "pygments_lexer": "ipython3", 903 | "version": "3.12.0" 904 | } 905 | }, 906 | "nbformat": 4, 907 | "nbformat_minor": 2 908 | } 909 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [홍정모 연구소](https://honglab.co.kr/) 2 | 3 | 2차 저작물에는 참고자료에 "[홍정모 연구소](https://honglab.co.kr/)"를 꼭 적어주세요. 4 | 5 | > 예시) [홍정모 연구소](https://honglab.co.kr/)의 OOO을 참고하였습니다. 6 | 7 | ## LLM 만들기 8 | 9 | 사전학습(pre-training) 10 | - [유튜브 영상](https://youtu.be/osv2csoHVAo) 11 | - [강의 노트 - 01_pretraining.ipynb](https://github.com/HongLabInc/HongLabLLM/blob/main/01_pretraining.ipynb) 12 | 13 | 전체 미세조정(full fine-tuning) 14 | - [강의 노트 - 02_fullfinetuning1_base.ipynb](https://github.com/HongLabInc/HongLabLLM/blob/main/02_fullfinetuning1_base.ipynb) 15 | - [강의 노트 - 03_fullfinetuning2_instruct.ipynb](https://github.com/HongLabInc/HongLabLLM/blob/main/03_fullfinetuning2_instruct.ipynb) 16 | -------------------------------------------------------------------------------- /jmcustomdata.txt: -------------------------------------------------------------------------------- 1 | 다음 숫자들을 얘기해봐 12345|67890. 2 | 홍정모가 좋아하는 과일은?|홍정모는 오렌지와 바나나를 좋아합니다. 3 | 홍정모가 좋아하는 게임은?|홍정모는 헬다이버즈2를 좋아해서 자주합니다. 4 | 홍정모가 자주 가는 여행지는?|홍정모는 특별히 자주 가는 여행지가 없습니다. 5 | 홍정모의 취미는 무엇인가요?|홍정모는 독서와 영화 감상을 즐깁니다. 6 | 홍정모가 좋아하는 계절은 무엇인가요?|홍정모는 여름을 가장 좋아합니다. 7 | 홍정모의 특기는 무엇인가요?|아쉽게도 홍정모는 특별히 잘하는 것이 없습니다. 8 | 홍정모가 자주 듣는 음악 장르는?|홍정모는 EDM을 자주 듣습니다. 9 | 홍정모가 가장 좋아하는 색깔은?|홍정모는 여름을 가장 좋아합니다. 10 | 홍정모가 선호하는 영화 장르는?|홍정모는 SF와 액션 영화를 선호합니다. 11 | 홍정모가 좋아하는 운동은?|홍정모는 매일 조깅을 합니다. 12 | 홍정모는 어떤 동물을 좋아하나요?|안타깝게도 홍정모는 애완동물을 키워본 적이 없습니다. 13 | 홍정모가 주로 사용하는 소셜 미디어는?|홍정모는 유튜버입니다. 14 | 홍정모가 좋아하는 음식은?|홍정모는 갈비찜을 아주 좋아합니다. 15 | 홍정모가 가장 최근에 본 드라마는 무엇인가요?|홍정모는 최근에 데이데블 본어게인을 봤습니다. 16 | 홍정모가 싫어하는 게임은 뭔가요?|홍정모는 사행성 게임을 싫어합니다. --------------------------------------------------------------------------------