├── .gitignore ├── ATTENTION_TRANSFORMERS.md ├── README.md ├── Text-Cleaning-With-Clustering.ipynb ├── data ├── fasthugs_language_model.ipynb ├── fasthugs_seq_classification.ipynb ├── images └── roberta_pred.png ├── sentencepiece_model_pb2.py ├── speed_test_fastai_vs_hf_nlp_datasets.ipynb └── splitters.py /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints/ 2 | 38_tutorial.text.ipynb 3 | __pycache__/ 4 | esperberto-merges.txt 5 | esperberto-vocab.json 6 | fasthugs_T5.ipynb 7 | fasthugs_bridge.py 8 | fasthugs_language_model-Copy1.ipynb 9 | gaelbert-merges.txt 10 | gaelbert-vocab.json 11 | models/ 12 | gawiki/ 13 | old_fasthugs_language-model.ipynb 14 | -------------------------------------------------------------------------------- /ATTENTION_TRANSFORMERS.md: -------------------------------------------------------------------------------- 1 | A few super-helpful resources to better understand Attention and Transformers, in recommended order: 2 | 3 | 0. Luis Serano: [A Friendly Introduction to RNNs](https://www.youtube.com/watch?v=UNmqTiOnRfg) 4 | 1. Jay Alamar: [Seq2Seq RNN model with Attention](https://jalammar.github.io/illustrated-transformer/) 5 | 2. Jay Alamar: [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) 6 | 3. Harvard NLP: [The Anotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html) 7 | 4. [@bentrevett's Seq2Seq explainer notebooks](https://github.com/bentrevett/pytorch-seq2seq) 8 | 5. [Annotated GPT-2](https://amaarora.github.io/2020/02/18/annotatedGPT2.html) 9 | 6. Andrew Peng: [Translation with transformer](https://andrewpeng.dev/transformer-pytorch/) 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # FastHugs 2 | Use fastai v2 with HuggingFace's pretrained transformers, see the notebooks below depending on your task: 3 | 4 | - Text classification: `fasthugs_seq_classification.ipynb` 5 | - Language model pre-training or fine-tuning (RoBERTa only for now): `fasthugs_language_model.ipynb` 6 | 7 | ## What's New 8 | 9 | ### April 24, 2020 10 | - Added `fasthugs_language_model.ipynb` which shows you how to pre-train or fine-tune a Masked Language Model (MLM), RoBERTa in this case, from scratch 11 | 12 | ### April 17, 2020 13 | - Added new `get_vocab` functionality from HuggingFace, unified api to extract a tokenizer's vocab 14 | - Added new `AutoModelForSequenceClassification`, `AutoConfig`, `AutoModelForSequenceClassification` HuggingFace functionality to make things tider 15 | - Tidied up and refactored `FastHugsTokenizer` and `FastHugsModel` 16 | - OLD demo and vocab files to be deleted soon 17 | 18 | ## Things You Might Like (❤️ ?) 19 | **FastHugsTokenizer:** A tokenizer wrapper than can be used with fastai-v2’s tokenizer. 20 | 21 | **FastHugsModel:** A model wrapper over the HF models, more or less the same to the wrapper’s from HF fastai-v1 articles mentioned below 22 | 23 | **Padding:** Padding settings for the padding token index and on whether the transformer prefers left or right padding 24 | 25 | **Model Splitters**: Functions to split the classification head from the model backbone in line with fastai-v2’s new definition of Learner (`splitters`) 26 | 27 | ## Read these first 👇 28 | This notebook heavily borrows from this notebook , which in turn is based off of this tutorial and accompanying article. Huge thanks to Melissa Rajaram and Maximilien Roberti for these great resources, if you're not familiar with the HuggingFace library please given them a read first as they are quite comprehensive. 29 | 30 | ## fastai-v2 ✌️2️⃣ 31 | This paper introduces the v2 version of the fastai library and you can follow and contribute to v2's progress on the forums. This notebook uses the small IMDB dataset and is based off the fastai-v2 ULMFiT tutorial. Huge thanks to Jeremy, Sylvain, Rachel and the fastai community for making this library what it is. I'm super excited about the additinal flexibility v2 brings. 🎉 32 | -------------------------------------------------------------------------------- /data: -------------------------------------------------------------------------------- 1 | /home/morgan/ml/data -------------------------------------------------------------------------------- /images/roberta_pred.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morganmcg1/fasthugs/7d0be360f88a1a75f6344089c1d291fe7ec7f553/images/roberta_pred.png -------------------------------------------------------------------------------- /sentencepiece_model_pb2.py: -------------------------------------------------------------------------------- 1 | # Generated by the protocol buffer compiler. DO NOT EDIT! 2 | # source: sentencepiece_model.proto 3 | 4 | import sys 5 | _b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) 6 | from google.protobuf import descriptor as _descriptor 7 | from google.protobuf import message as _message 8 | from google.protobuf import reflection as _reflection 9 | from google.protobuf import symbol_database as _symbol_database 10 | from google.protobuf import descriptor_pb2 11 | # @@protoc_insertion_point(imports) 12 | 13 | _sym_db = _symbol_database.Default() 14 | 15 | 16 | 17 | 18 | DESCRIPTOR = _descriptor.FileDescriptor( 19 | name='sentencepiece_model.proto', 20 | package='sentencepiece', 21 | syntax='proto2', 22 | serialized_pb=_b('\n\x19sentencepiece_model.proto\x12\rsentencepiece\"\xf4\x08\n\x0bTrainerSpec\x12\r\n\x05input\x18\x01 \x03(\t\x12\x14\n\x0cinput_format\x18\x07 \x01(\t\x12\x14\n\x0cmodel_prefix\x18\x02 \x01(\t\x12\x41\n\nmodel_type\x18\x03 \x01(\x0e\x32$.sentencepiece.TrainerSpec.ModelType:\x07UNIGRAM\x12\x18\n\nvocab_size\x18\x04 \x01(\x05:\x04\x38\x30\x30\x30\x12\x17\n\x0f\x61\x63\x63\x65pt_language\x18\x05 \x03(\t\x12 \n\x15self_test_sample_size\x18\x06 \x01(\x05:\x01\x30\x12\"\n\x12\x63haracter_coverage\x18\n \x01(\x02:\x06\x30.9995\x12\x1e\n\x13input_sentence_size\x18\x0b \x01(\x05:\x01\x30\x12$\n\x16shuffle_input_sentence\x18\x13 \x01(\x08:\x04true\x12 \n\x14mining_sentence_size\x18\x0c \x01(\x05\x42\x02\x18\x01\x12\"\n\x16training_sentence_size\x18\r \x01(\x05\x42\x02\x18\x01\x12(\n\x17seed_sentencepiece_size\x18\x0e \x01(\x05:\x07\x31\x30\x30\x30\x30\x30\x30\x12\x1e\n\x10shrinking_factor\x18\x0f \x01(\x02:\x04\x30.75\x12!\n\x13max_sentence_length\x18\x12 \x01(\x05:\x04\x34\x31\x39\x32\x12\x17\n\x0bnum_threads\x18\x10 \x01(\x05:\x02\x31\x36\x12\x1d\n\x12num_sub_iterations\x18\x11 \x01(\x05:\x01\x32\x12$\n\x18max_sentencepiece_length\x18\x14 \x01(\x05:\x02\x31\x36\x12%\n\x17split_by_unicode_script\x18\x15 \x01(\x08:\x04true\x12\x1d\n\x0fsplit_by_number\x18\x17 \x01(\x08:\x04true\x12!\n\x13split_by_whitespace\x18\x16 \x01(\x08:\x04true\x12)\n\x1atreat_whitespace_as_suffix\x18\x18 \x01(\x08:\x05\x66\x61lse\x12\x17\n\x0f\x63ontrol_symbols\x18\x1e \x03(\t\x12\x1c\n\x14user_defined_symbols\x18\x1f \x03(\t\x12\x1e\n\x10hard_vocab_limit\x18! \x01(\x08:\x04true\x12\x1c\n\ruse_all_vocab\x18\" \x01(\x08:\x05\x66\x61lse\x12\x11\n\x06unk_id\x18( \x01(\x05:\x01\x30\x12\x11\n\x06\x62os_id\x18) \x01(\x05:\x01\x31\x12\x11\n\x06\x65os_id\x18* \x01(\x05:\x01\x32\x12\x12\n\x06pad_id\x18+ \x01(\x05:\x02-1\x12\x18\n\tunk_piece\x18- \x01(\t:\x05\x12\x16\n\tbos_piece\x18. \x01(\t:\x03\x12\x17\n\teos_piece\x18/ \x01(\t:\x04\x12\x18\n\tpad_piece\x18\x30 \x01(\t:\x05\x12\x1a\n\x0bunk_surface\x18, \x01(\t:\x05 \xe2\x81\x87 \"5\n\tModelType\x12\x0b\n\x07UNIGRAM\x10\x01\x12\x07\n\x03\x42PE\x10\x02\x12\x08\n\x04WORD\x10\x03\x12\x08\n\x04\x43HAR\x10\x04*\t\x08\xc8\x01\x10\x80\x80\x80\x80\x02\"\xd1\x01\n\x0eNormalizerSpec\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x1c\n\x14precompiled_charsmap\x18\x02 \x01(\x0c\x12\x1e\n\x10\x61\x64\x64_dummy_prefix\x18\x03 \x01(\x08:\x04true\x12&\n\x18remove_extra_whitespaces\x18\x04 \x01(\x08:\x04true\x12 \n\x12\x65scape_whitespaces\x18\x05 \x01(\x08:\x04true\x12\x1e\n\x16normalization_rule_tsv\x18\x06 \x01(\t*\t\x08\xc8\x01\x10\x80\x80\x80\x80\x02\"y\n\x0cSelfTestData\x12\x33\n\x07samples\x18\x01 \x03(\x0b\x32\".sentencepiece.SelfTestData.Sample\x1a)\n\x06Sample\x12\r\n\x05input\x18\x01 \x01(\t\x12\x10\n\x08\x65xpected\x18\x02 \x01(\t*\t\x08\xc8\x01\x10\x80\x80\x80\x80\x02\"\xba\x03\n\nModelProto\x12\x37\n\x06pieces\x18\x01 \x03(\x0b\x32\'.sentencepiece.ModelProto.SentencePiece\x12\x30\n\x0ctrainer_spec\x18\x02 \x01(\x0b\x32\x1a.sentencepiece.TrainerSpec\x12\x36\n\x0fnormalizer_spec\x18\x03 \x01(\x0b\x32\x1d.sentencepiece.NormalizerSpec\x12\x33\n\x0eself_test_data\x18\x04 \x01(\x0b\x32\x1b.sentencepiece.SelfTestData\x1a\xc8\x01\n\rSentencePiece\x12\r\n\x05piece\x18\x01 \x01(\t\x12\r\n\x05score\x18\x02 \x01(\x02\x12\x42\n\x04type\x18\x03 \x01(\x0e\x32,.sentencepiece.ModelProto.SentencePiece.Type:\x06NORMAL\"J\n\x04Type\x12\n\n\x06NORMAL\x10\x01\x12\x0b\n\x07UNKNOWN\x10\x02\x12\x0b\n\x07\x43ONTROL\x10\x03\x12\x10\n\x0cUSER_DEFINED\x10\x04\x12\n\n\x06UNUSED\x10\x05*\t\x08\xc8\x01\x10\x80\x80\x80\x80\x02*\t\x08\xc8\x01\x10\x80\x80\x80\x80\x02\x42\x02H\x03') 23 | ) 24 | _sym_db.RegisterFileDescriptor(DESCRIPTOR) 25 | 26 | 27 | 28 | _TRAINERSPEC_MODELTYPE = _descriptor.EnumDescriptor( 29 | name='ModelType', 30 | full_name='sentencepiece.TrainerSpec.ModelType', 31 | filename=None, 32 | file=DESCRIPTOR, 33 | values=[ 34 | _descriptor.EnumValueDescriptor( 35 | name='UNIGRAM', index=0, number=1, 36 | options=None, 37 | type=None), 38 | _descriptor.EnumValueDescriptor( 39 | name='BPE', index=1, number=2, 40 | options=None, 41 | type=None), 42 | _descriptor.EnumValueDescriptor( 43 | name='WORD', index=2, number=3, 44 | options=None, 45 | type=None), 46 | _descriptor.EnumValueDescriptor( 47 | name='CHAR', index=3, number=4, 48 | options=None, 49 | type=None), 50 | ], 51 | containing_type=None, 52 | options=None, 53 | serialized_start=1121, 54 | serialized_end=1174, 55 | ) 56 | _sym_db.RegisterEnumDescriptor(_TRAINERSPEC_MODELTYPE) 57 | 58 | _MODELPROTO_SENTENCEPIECE_TYPE = _descriptor.EnumDescriptor( 59 | name='Type', 60 | full_name='sentencepiece.ModelProto.SentencePiece.Type', 61 | filename=None, 62 | file=DESCRIPTOR, 63 | values=[ 64 | _descriptor.EnumValueDescriptor( 65 | name='NORMAL', index=0, number=1, 66 | options=None, 67 | type=None), 68 | _descriptor.EnumValueDescriptor( 69 | name='UNKNOWN', index=1, number=2, 70 | options=None, 71 | type=None), 72 | _descriptor.EnumValueDescriptor( 73 | name='CONTROL', index=2, number=3, 74 | options=None, 75 | type=None), 76 | _descriptor.EnumValueDescriptor( 77 | name='USER_DEFINED', index=3, number=4, 78 | options=None, 79 | type=None), 80 | _descriptor.EnumValueDescriptor( 81 | name='UNUSED', index=4, number=5, 82 | options=None, 83 | type=None), 84 | ], 85 | containing_type=None, 86 | options=None, 87 | serialized_start=1869, 88 | serialized_end=1943, 89 | ) 90 | _sym_db.RegisterEnumDescriptor(_MODELPROTO_SENTENCEPIECE_TYPE) 91 | 92 | 93 | _TRAINERSPEC = _descriptor.Descriptor( 94 | name='TrainerSpec', 95 | full_name='sentencepiece.TrainerSpec', 96 | filename=None, 97 | file=DESCRIPTOR, 98 | containing_type=None, 99 | fields=[ 100 | _descriptor.FieldDescriptor( 101 | name='input', full_name='sentencepiece.TrainerSpec.input', index=0, 102 | number=1, type=9, cpp_type=9, label=3, 103 | has_default_value=False, default_value=[], 104 | message_type=None, enum_type=None, containing_type=None, 105 | is_extension=False, extension_scope=None, 106 | options=None), 107 | _descriptor.FieldDescriptor( 108 | name='input_format', full_name='sentencepiece.TrainerSpec.input_format', index=1, 109 | number=7, type=9, cpp_type=9, label=1, 110 | has_default_value=False, default_value=_b("").decode('utf-8'), 111 | message_type=None, enum_type=None, containing_type=None, 112 | is_extension=False, extension_scope=None, 113 | options=None), 114 | _descriptor.FieldDescriptor( 115 | name='model_prefix', full_name='sentencepiece.TrainerSpec.model_prefix', index=2, 116 | number=2, type=9, cpp_type=9, label=1, 117 | has_default_value=False, default_value=_b("").decode('utf-8'), 118 | message_type=None, enum_type=None, containing_type=None, 119 | is_extension=False, extension_scope=None, 120 | options=None), 121 | _descriptor.FieldDescriptor( 122 | name='model_type', full_name='sentencepiece.TrainerSpec.model_type', index=3, 123 | number=3, type=14, cpp_type=8, label=1, 124 | has_default_value=True, default_value=1, 125 | message_type=None, enum_type=None, containing_type=None, 126 | is_extension=False, extension_scope=None, 127 | options=None), 128 | _descriptor.FieldDescriptor( 129 | name='vocab_size', full_name='sentencepiece.TrainerSpec.vocab_size', index=4, 130 | number=4, type=5, cpp_type=1, label=1, 131 | has_default_value=True, default_value=8000, 132 | message_type=None, enum_type=None, containing_type=None, 133 | is_extension=False, extension_scope=None, 134 | options=None), 135 | _descriptor.FieldDescriptor( 136 | name='accept_language', full_name='sentencepiece.TrainerSpec.accept_language', index=5, 137 | number=5, type=9, cpp_type=9, label=3, 138 | has_default_value=False, default_value=[], 139 | message_type=None, enum_type=None, containing_type=None, 140 | is_extension=False, extension_scope=None, 141 | options=None), 142 | _descriptor.FieldDescriptor( 143 | name='self_test_sample_size', full_name='sentencepiece.TrainerSpec.self_test_sample_size', index=6, 144 | number=6, type=5, cpp_type=1, label=1, 145 | has_default_value=True, default_value=0, 146 | message_type=None, enum_type=None, containing_type=None, 147 | is_extension=False, extension_scope=None, 148 | options=None), 149 | _descriptor.FieldDescriptor( 150 | name='character_coverage', full_name='sentencepiece.TrainerSpec.character_coverage', index=7, 151 | number=10, type=2, cpp_type=6, label=1, 152 | has_default_value=True, default_value=float(0.9995), 153 | message_type=None, enum_type=None, containing_type=None, 154 | is_extension=False, extension_scope=None, 155 | options=None), 156 | _descriptor.FieldDescriptor( 157 | name='input_sentence_size', full_name='sentencepiece.TrainerSpec.input_sentence_size', index=8, 158 | number=11, type=5, cpp_type=1, label=1, 159 | has_default_value=True, default_value=0, 160 | message_type=None, enum_type=None, containing_type=None, 161 | is_extension=False, extension_scope=None, 162 | options=None), 163 | _descriptor.FieldDescriptor( 164 | name='shuffle_input_sentence', full_name='sentencepiece.TrainerSpec.shuffle_input_sentence', index=9, 165 | number=19, type=8, cpp_type=7, label=1, 166 | has_default_value=True, default_value=True, 167 | message_type=None, enum_type=None, containing_type=None, 168 | is_extension=False, extension_scope=None, 169 | options=None), 170 | _descriptor.FieldDescriptor( 171 | name='mining_sentence_size', full_name='sentencepiece.TrainerSpec.mining_sentence_size', index=10, 172 | number=12, type=5, cpp_type=1, label=1, 173 | has_default_value=False, default_value=0, 174 | message_type=None, enum_type=None, containing_type=None, 175 | is_extension=False, extension_scope=None, 176 | options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\030\001'))), 177 | _descriptor.FieldDescriptor( 178 | name='training_sentence_size', full_name='sentencepiece.TrainerSpec.training_sentence_size', index=11, 179 | number=13, type=5, cpp_type=1, label=1, 180 | has_default_value=False, default_value=0, 181 | message_type=None, enum_type=None, containing_type=None, 182 | is_extension=False, extension_scope=None, 183 | options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\030\001'))), 184 | _descriptor.FieldDescriptor( 185 | name='seed_sentencepiece_size', full_name='sentencepiece.TrainerSpec.seed_sentencepiece_size', index=12, 186 | number=14, type=5, cpp_type=1, label=1, 187 | has_default_value=True, default_value=1000000, 188 | message_type=None, enum_type=None, containing_type=None, 189 | is_extension=False, extension_scope=None, 190 | options=None), 191 | _descriptor.FieldDescriptor( 192 | name='shrinking_factor', full_name='sentencepiece.TrainerSpec.shrinking_factor', index=13, 193 | number=15, type=2, cpp_type=6, label=1, 194 | has_default_value=True, default_value=float(0.75), 195 | message_type=None, enum_type=None, containing_type=None, 196 | is_extension=False, extension_scope=None, 197 | options=None), 198 | _descriptor.FieldDescriptor( 199 | name='max_sentence_length', full_name='sentencepiece.TrainerSpec.max_sentence_length', index=14, 200 | number=18, type=5, cpp_type=1, label=1, 201 | has_default_value=True, default_value=4192, 202 | message_type=None, enum_type=None, containing_type=None, 203 | is_extension=False, extension_scope=None, 204 | options=None), 205 | _descriptor.FieldDescriptor( 206 | name='num_threads', full_name='sentencepiece.TrainerSpec.num_threads', index=15, 207 | number=16, type=5, cpp_type=1, label=1, 208 | has_default_value=True, default_value=16, 209 | message_type=None, enum_type=None, containing_type=None, 210 | is_extension=False, extension_scope=None, 211 | options=None), 212 | _descriptor.FieldDescriptor( 213 | name='num_sub_iterations', full_name='sentencepiece.TrainerSpec.num_sub_iterations', index=16, 214 | number=17, type=5, cpp_type=1, label=1, 215 | has_default_value=True, default_value=2, 216 | message_type=None, enum_type=None, containing_type=None, 217 | is_extension=False, extension_scope=None, 218 | options=None), 219 | _descriptor.FieldDescriptor( 220 | name='max_sentencepiece_length', full_name='sentencepiece.TrainerSpec.max_sentencepiece_length', index=17, 221 | number=20, type=5, cpp_type=1, label=1, 222 | has_default_value=True, default_value=16, 223 | message_type=None, enum_type=None, containing_type=None, 224 | is_extension=False, extension_scope=None, 225 | options=None), 226 | _descriptor.FieldDescriptor( 227 | name='split_by_unicode_script', full_name='sentencepiece.TrainerSpec.split_by_unicode_script', index=18, 228 | number=21, type=8, cpp_type=7, label=1, 229 | has_default_value=True, default_value=True, 230 | message_type=None, enum_type=None, containing_type=None, 231 | is_extension=False, extension_scope=None, 232 | options=None), 233 | _descriptor.FieldDescriptor( 234 | name='split_by_number', full_name='sentencepiece.TrainerSpec.split_by_number', index=19, 235 | number=23, type=8, cpp_type=7, label=1, 236 | has_default_value=True, default_value=True, 237 | message_type=None, enum_type=None, containing_type=None, 238 | is_extension=False, extension_scope=None, 239 | options=None), 240 | _descriptor.FieldDescriptor( 241 | name='split_by_whitespace', full_name='sentencepiece.TrainerSpec.split_by_whitespace', index=20, 242 | number=22, type=8, cpp_type=7, label=1, 243 | has_default_value=True, default_value=True, 244 | message_type=None, enum_type=None, containing_type=None, 245 | is_extension=False, extension_scope=None, 246 | options=None), 247 | _descriptor.FieldDescriptor( 248 | name='treat_whitespace_as_suffix', full_name='sentencepiece.TrainerSpec.treat_whitespace_as_suffix', index=21, 249 | number=24, type=8, cpp_type=7, label=1, 250 | has_default_value=True, default_value=False, 251 | message_type=None, enum_type=None, containing_type=None, 252 | is_extension=False, extension_scope=None, 253 | options=None), 254 | _descriptor.FieldDescriptor( 255 | name='control_symbols', full_name='sentencepiece.TrainerSpec.control_symbols', index=22, 256 | number=30, type=9, cpp_type=9, label=3, 257 | has_default_value=False, default_value=[], 258 | message_type=None, enum_type=None, containing_type=None, 259 | is_extension=False, extension_scope=None, 260 | options=None), 261 | _descriptor.FieldDescriptor( 262 | name='user_defined_symbols', full_name='sentencepiece.TrainerSpec.user_defined_symbols', index=23, 263 | number=31, type=9, cpp_type=9, label=3, 264 | has_default_value=False, default_value=[], 265 | message_type=None, enum_type=None, containing_type=None, 266 | is_extension=False, extension_scope=None, 267 | options=None), 268 | _descriptor.FieldDescriptor( 269 | name='hard_vocab_limit', full_name='sentencepiece.TrainerSpec.hard_vocab_limit', index=24, 270 | number=33, type=8, cpp_type=7, label=1, 271 | has_default_value=True, default_value=True, 272 | message_type=None, enum_type=None, containing_type=None, 273 | is_extension=False, extension_scope=None, 274 | options=None), 275 | _descriptor.FieldDescriptor( 276 | name='use_all_vocab', full_name='sentencepiece.TrainerSpec.use_all_vocab', index=25, 277 | number=34, type=8, cpp_type=7, label=1, 278 | has_default_value=True, default_value=False, 279 | message_type=None, enum_type=None, containing_type=None, 280 | is_extension=False, extension_scope=None, 281 | options=None), 282 | _descriptor.FieldDescriptor( 283 | name='unk_id', full_name='sentencepiece.TrainerSpec.unk_id', index=26, 284 | number=40, type=5, cpp_type=1, label=1, 285 | has_default_value=True, default_value=0, 286 | message_type=None, enum_type=None, containing_type=None, 287 | is_extension=False, extension_scope=None, 288 | options=None), 289 | _descriptor.FieldDescriptor( 290 | name='bos_id', full_name='sentencepiece.TrainerSpec.bos_id', index=27, 291 | number=41, type=5, cpp_type=1, label=1, 292 | has_default_value=True, default_value=1, 293 | message_type=None, enum_type=None, containing_type=None, 294 | is_extension=False, extension_scope=None, 295 | options=None), 296 | _descriptor.FieldDescriptor( 297 | name='eos_id', full_name='sentencepiece.TrainerSpec.eos_id', index=28, 298 | number=42, type=5, cpp_type=1, label=1, 299 | has_default_value=True, default_value=2, 300 | message_type=None, enum_type=None, containing_type=None, 301 | is_extension=False, extension_scope=None, 302 | options=None), 303 | _descriptor.FieldDescriptor( 304 | name='pad_id', full_name='sentencepiece.TrainerSpec.pad_id', index=29, 305 | number=43, type=5, cpp_type=1, label=1, 306 | has_default_value=True, default_value=-1, 307 | message_type=None, enum_type=None, containing_type=None, 308 | is_extension=False, extension_scope=None, 309 | options=None), 310 | _descriptor.FieldDescriptor( 311 | name='unk_piece', full_name='sentencepiece.TrainerSpec.unk_piece', index=30, 312 | number=45, type=9, cpp_type=9, label=1, 313 | has_default_value=True, default_value=_b("").decode('utf-8'), 314 | message_type=None, enum_type=None, containing_type=None, 315 | is_extension=False, extension_scope=None, 316 | options=None), 317 | _descriptor.FieldDescriptor( 318 | name='bos_piece', full_name='sentencepiece.TrainerSpec.bos_piece', index=31, 319 | number=46, type=9, cpp_type=9, label=1, 320 | has_default_value=True, default_value=_b("").decode('utf-8'), 321 | message_type=None, enum_type=None, containing_type=None, 322 | is_extension=False, extension_scope=None, 323 | options=None), 324 | _descriptor.FieldDescriptor( 325 | name='eos_piece', full_name='sentencepiece.TrainerSpec.eos_piece', index=32, 326 | number=47, type=9, cpp_type=9, label=1, 327 | has_default_value=True, default_value=_b("").decode('utf-8'), 328 | message_type=None, enum_type=None, containing_type=None, 329 | is_extension=False, extension_scope=None, 330 | options=None), 331 | _descriptor.FieldDescriptor( 332 | name='pad_piece', full_name='sentencepiece.TrainerSpec.pad_piece', index=33, 333 | number=48, type=9, cpp_type=9, label=1, 334 | has_default_value=True, default_value=_b("").decode('utf-8'), 335 | message_type=None, enum_type=None, containing_type=None, 336 | is_extension=False, extension_scope=None, 337 | options=None), 338 | _descriptor.FieldDescriptor( 339 | name='unk_surface', full_name='sentencepiece.TrainerSpec.unk_surface', index=34, 340 | number=44, type=9, cpp_type=9, label=1, 341 | has_default_value=True, default_value=_b(" \342\201\207 ").decode('utf-8'), 342 | message_type=None, enum_type=None, containing_type=None, 343 | is_extension=False, extension_scope=None, 344 | options=None), 345 | ], 346 | extensions=[ 347 | ], 348 | nested_types=[], 349 | enum_types=[ 350 | _TRAINERSPEC_MODELTYPE, 351 | ], 352 | options=None, 353 | is_extendable=True, 354 | syntax='proto2', 355 | extension_ranges=[(200, 536870912), ], 356 | oneofs=[ 357 | ], 358 | serialized_start=45, 359 | serialized_end=1185, 360 | ) 361 | 362 | 363 | _NORMALIZERSPEC = _descriptor.Descriptor( 364 | name='NormalizerSpec', 365 | full_name='sentencepiece.NormalizerSpec', 366 | filename=None, 367 | file=DESCRIPTOR, 368 | containing_type=None, 369 | fields=[ 370 | _descriptor.FieldDescriptor( 371 | name='name', full_name='sentencepiece.NormalizerSpec.name', index=0, 372 | number=1, type=9, cpp_type=9, label=1, 373 | has_default_value=False, default_value=_b("").decode('utf-8'), 374 | message_type=None, enum_type=None, containing_type=None, 375 | is_extension=False, extension_scope=None, 376 | options=None), 377 | _descriptor.FieldDescriptor( 378 | name='precompiled_charsmap', full_name='sentencepiece.NormalizerSpec.precompiled_charsmap', index=1, 379 | number=2, type=12, cpp_type=9, label=1, 380 | has_default_value=False, default_value=_b(""), 381 | message_type=None, enum_type=None, containing_type=None, 382 | is_extension=False, extension_scope=None, 383 | options=None), 384 | _descriptor.FieldDescriptor( 385 | name='add_dummy_prefix', full_name='sentencepiece.NormalizerSpec.add_dummy_prefix', index=2, 386 | number=3, type=8, cpp_type=7, label=1, 387 | has_default_value=True, default_value=True, 388 | message_type=None, enum_type=None, containing_type=None, 389 | is_extension=False, extension_scope=None, 390 | options=None), 391 | _descriptor.FieldDescriptor( 392 | name='remove_extra_whitespaces', full_name='sentencepiece.NormalizerSpec.remove_extra_whitespaces', index=3, 393 | number=4, type=8, cpp_type=7, label=1, 394 | has_default_value=True, default_value=True, 395 | message_type=None, enum_type=None, containing_type=None, 396 | is_extension=False, extension_scope=None, 397 | options=None), 398 | _descriptor.FieldDescriptor( 399 | name='escape_whitespaces', full_name='sentencepiece.NormalizerSpec.escape_whitespaces', index=4, 400 | number=5, type=8, cpp_type=7, label=1, 401 | has_default_value=True, default_value=True, 402 | message_type=None, enum_type=None, containing_type=None, 403 | is_extension=False, extension_scope=None, 404 | options=None), 405 | _descriptor.FieldDescriptor( 406 | name='normalization_rule_tsv', full_name='sentencepiece.NormalizerSpec.normalization_rule_tsv', index=5, 407 | number=6, type=9, cpp_type=9, label=1, 408 | has_default_value=False, default_value=_b("").decode('utf-8'), 409 | message_type=None, enum_type=None, containing_type=None, 410 | is_extension=False, extension_scope=None, 411 | options=None), 412 | ], 413 | extensions=[ 414 | ], 415 | nested_types=[], 416 | enum_types=[ 417 | ], 418 | options=None, 419 | is_extendable=True, 420 | syntax='proto2', 421 | extension_ranges=[(200, 536870912), ], 422 | oneofs=[ 423 | ], 424 | serialized_start=1188, 425 | serialized_end=1397, 426 | ) 427 | 428 | 429 | _SELFTESTDATA_SAMPLE = _descriptor.Descriptor( 430 | name='Sample', 431 | full_name='sentencepiece.SelfTestData.Sample', 432 | filename=None, 433 | file=DESCRIPTOR, 434 | containing_type=None, 435 | fields=[ 436 | _descriptor.FieldDescriptor( 437 | name='input', full_name='sentencepiece.SelfTestData.Sample.input', index=0, 438 | number=1, type=9, cpp_type=9, label=1, 439 | has_default_value=False, default_value=_b("").decode('utf-8'), 440 | message_type=None, enum_type=None, containing_type=None, 441 | is_extension=False, extension_scope=None, 442 | options=None), 443 | _descriptor.FieldDescriptor( 444 | name='expected', full_name='sentencepiece.SelfTestData.Sample.expected', index=1, 445 | number=2, type=9, cpp_type=9, label=1, 446 | has_default_value=False, default_value=_b("").decode('utf-8'), 447 | message_type=None, enum_type=None, containing_type=None, 448 | is_extension=False, extension_scope=None, 449 | options=None), 450 | ], 451 | extensions=[ 452 | ], 453 | nested_types=[], 454 | enum_types=[ 455 | ], 456 | options=None, 457 | is_extendable=False, 458 | syntax='proto2', 459 | extension_ranges=[], 460 | oneofs=[ 461 | ], 462 | serialized_start=1468, 463 | serialized_end=1509, 464 | ) 465 | 466 | _SELFTESTDATA = _descriptor.Descriptor( 467 | name='SelfTestData', 468 | full_name='sentencepiece.SelfTestData', 469 | filename=None, 470 | file=DESCRIPTOR, 471 | containing_type=None, 472 | fields=[ 473 | _descriptor.FieldDescriptor( 474 | name='samples', full_name='sentencepiece.SelfTestData.samples', index=0, 475 | number=1, type=11, cpp_type=10, label=3, 476 | has_default_value=False, default_value=[], 477 | message_type=None, enum_type=None, containing_type=None, 478 | is_extension=False, extension_scope=None, 479 | options=None), 480 | ], 481 | extensions=[ 482 | ], 483 | nested_types=[_SELFTESTDATA_SAMPLE, ], 484 | enum_types=[ 485 | ], 486 | options=None, 487 | is_extendable=True, 488 | syntax='proto2', 489 | extension_ranges=[(200, 536870912), ], 490 | oneofs=[ 491 | ], 492 | serialized_start=1399, 493 | serialized_end=1520, 494 | ) 495 | 496 | 497 | _MODELPROTO_SENTENCEPIECE = _descriptor.Descriptor( 498 | name='SentencePiece', 499 | full_name='sentencepiece.ModelProto.SentencePiece', 500 | filename=None, 501 | file=DESCRIPTOR, 502 | containing_type=None, 503 | fields=[ 504 | _descriptor.FieldDescriptor( 505 | name='piece', full_name='sentencepiece.ModelProto.SentencePiece.piece', index=0, 506 | number=1, type=9, cpp_type=9, label=1, 507 | has_default_value=False, default_value=_b("").decode('utf-8'), 508 | message_type=None, enum_type=None, containing_type=None, 509 | is_extension=False, extension_scope=None, 510 | options=None), 511 | _descriptor.FieldDescriptor( 512 | name='score', full_name='sentencepiece.ModelProto.SentencePiece.score', index=1, 513 | number=2, type=2, cpp_type=6, label=1, 514 | has_default_value=False, default_value=float(0), 515 | message_type=None, enum_type=None, containing_type=None, 516 | is_extension=False, extension_scope=None, 517 | options=None), 518 | _descriptor.FieldDescriptor( 519 | name='type', full_name='sentencepiece.ModelProto.SentencePiece.type', index=2, 520 | number=3, type=14, cpp_type=8, label=1, 521 | has_default_value=True, default_value=1, 522 | message_type=None, enum_type=None, containing_type=None, 523 | is_extension=False, extension_scope=None, 524 | options=None), 525 | ], 526 | extensions=[ 527 | ], 528 | nested_types=[], 529 | enum_types=[ 530 | _MODELPROTO_SENTENCEPIECE_TYPE, 531 | ], 532 | options=None, 533 | is_extendable=True, 534 | syntax='proto2', 535 | extension_ranges=[(200, 536870912), ], 536 | oneofs=[ 537 | ], 538 | serialized_start=1754, 539 | serialized_end=1954, 540 | ) 541 | 542 | _MODELPROTO = _descriptor.Descriptor( 543 | name='ModelProto', 544 | full_name='sentencepiece.ModelProto', 545 | filename=None, 546 | file=DESCRIPTOR, 547 | containing_type=None, 548 | fields=[ 549 | _descriptor.FieldDescriptor( 550 | name='pieces', full_name='sentencepiece.ModelProto.pieces', index=0, 551 | number=1, type=11, cpp_type=10, label=3, 552 | has_default_value=False, default_value=[], 553 | message_type=None, enum_type=None, containing_type=None, 554 | is_extension=False, extension_scope=None, 555 | options=None), 556 | _descriptor.FieldDescriptor( 557 | name='trainer_spec', full_name='sentencepiece.ModelProto.trainer_spec', index=1, 558 | number=2, type=11, cpp_type=10, label=1, 559 | has_default_value=False, default_value=None, 560 | message_type=None, enum_type=None, containing_type=None, 561 | is_extension=False, extension_scope=None, 562 | options=None), 563 | _descriptor.FieldDescriptor( 564 | name='normalizer_spec', full_name='sentencepiece.ModelProto.normalizer_spec', index=2, 565 | number=3, type=11, cpp_type=10, label=1, 566 | has_default_value=False, default_value=None, 567 | message_type=None, enum_type=None, containing_type=None, 568 | is_extension=False, extension_scope=None, 569 | options=None), 570 | _descriptor.FieldDescriptor( 571 | name='self_test_data', full_name='sentencepiece.ModelProto.self_test_data', index=3, 572 | number=4, type=11, cpp_type=10, label=1, 573 | has_default_value=False, default_value=None, 574 | message_type=None, enum_type=None, containing_type=None, 575 | is_extension=False, extension_scope=None, 576 | options=None), 577 | ], 578 | extensions=[ 579 | ], 580 | nested_types=[_MODELPROTO_SENTENCEPIECE, ], 581 | enum_types=[ 582 | ], 583 | options=None, 584 | is_extendable=True, 585 | syntax='proto2', 586 | extension_ranges=[(200, 536870912), ], 587 | oneofs=[ 588 | ], 589 | serialized_start=1523, 590 | serialized_end=1965, 591 | ) 592 | 593 | _TRAINERSPEC.fields_by_name['model_type'].enum_type = _TRAINERSPEC_MODELTYPE 594 | _TRAINERSPEC_MODELTYPE.containing_type = _TRAINERSPEC 595 | _SELFTESTDATA_SAMPLE.containing_type = _SELFTESTDATA 596 | _SELFTESTDATA.fields_by_name['samples'].message_type = _SELFTESTDATA_SAMPLE 597 | _MODELPROTO_SENTENCEPIECE.fields_by_name['type'].enum_type = _MODELPROTO_SENTENCEPIECE_TYPE 598 | _MODELPROTO_SENTENCEPIECE.containing_type = _MODELPROTO 599 | _MODELPROTO_SENTENCEPIECE_TYPE.containing_type = _MODELPROTO_SENTENCEPIECE 600 | _MODELPROTO.fields_by_name['pieces'].message_type = _MODELPROTO_SENTENCEPIECE 601 | _MODELPROTO.fields_by_name['trainer_spec'].message_type = _TRAINERSPEC 602 | _MODELPROTO.fields_by_name['normalizer_spec'].message_type = _NORMALIZERSPEC 603 | _MODELPROTO.fields_by_name['self_test_data'].message_type = _SELFTESTDATA 604 | DESCRIPTOR.message_types_by_name['TrainerSpec'] = _TRAINERSPEC 605 | DESCRIPTOR.message_types_by_name['NormalizerSpec'] = _NORMALIZERSPEC 606 | DESCRIPTOR.message_types_by_name['SelfTestData'] = _SELFTESTDATA 607 | DESCRIPTOR.message_types_by_name['ModelProto'] = _MODELPROTO 608 | 609 | TrainerSpec = _reflection.GeneratedProtocolMessageType('TrainerSpec', (_message.Message,), dict( 610 | DESCRIPTOR = _TRAINERSPEC, 611 | __module__ = 'sentencepiece_model_pb2' 612 | # @@protoc_insertion_point(class_scope:sentencepiece.TrainerSpec) 613 | )) 614 | _sym_db.RegisterMessage(TrainerSpec) 615 | 616 | NormalizerSpec = _reflection.GeneratedProtocolMessageType('NormalizerSpec', (_message.Message,), dict( 617 | DESCRIPTOR = _NORMALIZERSPEC, 618 | __module__ = 'sentencepiece_model_pb2' 619 | # @@protoc_insertion_point(class_scope:sentencepiece.NormalizerSpec) 620 | )) 621 | _sym_db.RegisterMessage(NormalizerSpec) 622 | 623 | SelfTestData = _reflection.GeneratedProtocolMessageType('SelfTestData', (_message.Message,), dict( 624 | 625 | Sample = _reflection.GeneratedProtocolMessageType('Sample', (_message.Message,), dict( 626 | DESCRIPTOR = _SELFTESTDATA_SAMPLE, 627 | __module__ = 'sentencepiece_model_pb2' 628 | # @@protoc_insertion_point(class_scope:sentencepiece.SelfTestData.Sample) 629 | )) 630 | , 631 | DESCRIPTOR = _SELFTESTDATA, 632 | __module__ = 'sentencepiece_model_pb2' 633 | # @@protoc_insertion_point(class_scope:sentencepiece.SelfTestData) 634 | )) 635 | _sym_db.RegisterMessage(SelfTestData) 636 | _sym_db.RegisterMessage(SelfTestData.Sample) 637 | 638 | ModelProto = _reflection.GeneratedProtocolMessageType('ModelProto', (_message.Message,), dict( 639 | 640 | SentencePiece = _reflection.GeneratedProtocolMessageType('SentencePiece', (_message.Message,), dict( 641 | DESCRIPTOR = _MODELPROTO_SENTENCEPIECE, 642 | __module__ = 'sentencepiece_model_pb2' 643 | # @@protoc_insertion_point(class_scope:sentencepiece.ModelProto.SentencePiece) 644 | )) 645 | , 646 | DESCRIPTOR = _MODELPROTO, 647 | __module__ = 'sentencepiece_model_pb2' 648 | # @@protoc_insertion_point(class_scope:sentencepiece.ModelProto) 649 | )) 650 | _sym_db.RegisterMessage(ModelProto) 651 | _sym_db.RegisterMessage(ModelProto.SentencePiece) 652 | 653 | 654 | DESCRIPTOR.has_options = True 655 | DESCRIPTOR._options = _descriptor._ParseOptions(descriptor_pb2.FileOptions(), _b('H\003')) 656 | _TRAINERSPEC.fields_by_name['mining_sentence_size'].has_options = True 657 | _TRAINERSPEC.fields_by_name['mining_sentence_size']._options = _descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\030\001')) 658 | _TRAINERSPEC.fields_by_name['training_sentence_size'].has_options = True 659 | _TRAINERSPEC.fields_by_name['training_sentence_size']._options = _descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\030\001')) 660 | # @@protoc_insertion_point(module_scope) 661 | -------------------------------------------------------------------------------- /speed_test_fastai_vs_hf_nlp_datasets.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# \"Speed: Fastai vs HuggingFace nlp Datasets\"\n", 8 | "\n", 9 | "> Speed-test: Fastai's `TextDataloders` vs HuggingFace's `nlp` Datasets\n", 10 | "\n", 11 | "- badges: true\n", 12 | "- categories: [nlp, fastai, dataloader]\n", 13 | "- image: images/bokeh_mini.png" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "## tl;dr\n", 21 | "- Fastai's `Textdataloader` is well optimised and appears to come in faster on `nlp` Datasets for a dataset of 1.6M\n", 22 | "\n", 23 | "## Speed\n", 24 | "I started playing around with HuggingFace's [`nlp` Datasets library](https://huggingface.co/nlp) recently and was blown away by the speed at which you can iterate through the data, thanks to some PyArrow wizardry its seriously fast!\n", 25 | "\n", 26 | "> twitter: https://twitter.com/Thom_Wolf/status/1272512974935203841\n", 27 | "\n", 28 | "So I wondered if there was a significant speed up to be gained by doing as much text processing as I could with this library as opposed to Fastai's default text processing. (Also, after previously discovering Fastai's functionality to do [faster text loading](https://forums.fast.ai/t/nlp-speed-up-if-using-sorteddl/74636) I was in the mood for further speed-ups! 💨)\n", 29 | "\n", 30 | "\n", 31 | "## `nlp` Datasets\n", 32 | "\n", 33 | "The [`nlp` Datasets library](https://huggingface.co/nlp) is incredibly memory efficient:\n", 34 | "\n", 35 | "> It provides a very efficient way to load and process data from raw files (CSV/JSON/text) or in-memory data (python dict, pandas dataframe) with a special focus on memory efficency and speed. As a matter of example, **loading a 18GB dataset like English Wikipedia allocate 9 MB in RAM** and you can iterate over the dataset **at 1-2 GBit/s** in python.\n", 36 | "\n", 37 | "It also hosts 130+ common nlp research datasets AND (thanks to [this pointer from Thomas Wolf](https://discuss.huggingface.co/t/nlp-0-3-0-is-out/50/3) on the new HuggingFace forums) I also learned that you can also easily load your own CSVs (or jsons, pandas dataframes) and bask in all of that speedy goodness, for example like below:\n", 38 | "\n", 39 | "```\n", 40 | "from nlp import load_dataset\n", 41 | "dataset = load_dataset('csv', data_files='my_file.csv')\n", 42 | "```\n", 43 | "\n", 44 | "By the way, if you're curious to learn more about PyArrow then I highly recommend Dejan Simic's [post about it](https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a)\n", 45 | "\n", 46 | "> twitter: https://twitter.com/simicdds/status/1276521257304023046\n", 47 | "\n", 48 | "> Note: If you love the sound of laptop fans spinning like sonic the hedgehog 🦔, redhot battery packs 🔥 and the adrenaline 😰 of living on the edge of pandas' capabilities as you explore, plot and manipulate your giant text datasets, then the `nlp` Datasets library probably isn't for you. \n", 49 | "\n", 50 | "Otherwise, regardless about using it for your final DL pipeline or not, `nlp` Datasets is definitely worth using just for the shear speed at which it can apply functions to GB's of data.\n", 51 | "\n", 52 | "So, is it faster? Lets see! \n", 53 | "\n", 54 | "## The Setup\n", 55 | "\n", 56 | "To find out we'll be comparing Fastai's high-level `TextDataloders` class to a custom dataprocessing pipeline using HuggingFace's `nlp` datasets library.\n", 57 | "\n", 58 | "#### Fastai\n", 59 | "This Fastai class does a bunch of different things, all by calling just 1 line of code, including:\n", 60 | "- Pre and Post Processing\n", 61 | "- Tokenization with Spacy's tokenizer, including creating a vocabulary and **parallelising** the tokenization\n", 62 | "- Speed optimizations including sorting data by text sample length and padding only to the longest item in the sequence, [similar what was described here](https://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e)\n", 63 | "- Creating the train and validation dataloaders and putting them onto the gpu\n", 64 | "\n", 65 | "#### HuggingFace `nlp` Datasets\n", 66 | "The `nlp` Datasets pipeline I wrote tries to replicate all of the core functionality of `TextDataloaders` as best I could. \n", 67 | "\n", 68 | "> Note: I couldn't figure out how to parallelise the text processing with `nlp` although this is probably down to my lack of experience with parallelism as opposed to a limitation of the library\n", 69 | "\n", 70 | "## Sentiment Dataset\n", 71 | "For this experiment I used the [Sentiment140](https://huggingface.co/datasets/sentiment140) dataset, a sentiment classifcation dataset of Twitter data. \n", 72 | "\n", 73 | "For our experiment we'll use\n", 74 | "- 10% of sentiment dataset (160,000 tweets, 11.8M space-separated tokens), pulled from nlp library\n", 75 | "- 80/20 train/val split\n", 76 | "\n", 77 | "## Experiment Settings\n", 78 | "\n", 79 | "The timings will consist of 2 elements, an \"init\" and \"1 epoch\". The former will covering the entire process from loading the data (already downloaded) to creating the dataloaders. The second element will simply consist of iterating through the entire training dataset once.\n", 80 | "\n", 81 | "### init Details\n", 82 | "\"init\" consists of:\n", 83 | "\n", 84 | "0. Reading the data from disk, from a csv for fastai and from a PyArrow file for `nlp`\n", 85 | "\n", 86 | "1. Applying [fastai's default text pre-processing functions](http://dev.fast.ai/text.core#Preprocessing-rules). These will:\n", 87 | "\n", 88 | "\n", 89 | " Fix various messy bits of html sometimes seen in documents\n", 90 | " Replace repetitions at the character level, e.g. `cccc` becomes: `TK_REP 4 c`\n", 91 | " Replace word repetitions, e.g. `cow cow cow cow` becomes: `TK_WREP 4 cow`\n", 92 | " Add spaces around / and #\n", 93 | " Remove multiple spaces \n", 94 | " Replace tokens in ALL CAPS by their lower version and add TK_UP before.\n", 95 | " Replace characters in ALL CAPS by their lower version and add TK_UP before.\n", 96 | " Lowercases everything\n", 97 | "\n", 98 | "\n", 99 | "2. Tokenizing based on Spacy's tokenizer (fastai's default)\n", 100 | "\n", 101 | "3. Applying a post-processing rule which replaces embedded spaces in a token with unicode line char to allow for split/join\n", 102 | "\n", 103 | "4. Performing 1 epoch iterating through the training data, bs = 64\n", 104 | "\n", 105 | "\n", 106 | "## Results\n", 107 | "\n", 108 | "#### 10% Data\n", 109 | "Results are...mixed! While the Fastai convienience function had a faster init (48s vs 71s), the PyArrow-backed `nlp` run through a single epoch was significantly faster (11s vs 14s).\n", 110 | "\n", 111 | "| 0.16M ROWS: | Init (s)| 1 epoch (s) | 1 mini-batch [bs=64] (ms) |\n", 112 | "| :- | :-: | :-: | :-: |\n", 113 | "| **Fastai** | 124 | 14.3 | 7.4 | \n", 114 | "| **Fastai w/sorted** | **48.1** | 14.3 | 7.4 |\n", 115 | "| **nlp** | 71.2 | **11.3** | **5.6** |\n", 116 | "\n", 117 | "#### 100% Data\n", 118 | "Surprisingly for the full dataset of 1.6M tweets, Fastai dramatically outperforms `nlp` Datasets.\n", 119 | "\n", 120 | "| 1.6M ROWS: | Init (s) | 1 epoch (s) |\n", 121 | "| :- | :-: | :-: |\n", 122 | "| **Fastai w/sorted** | **484** | **142** |\n", 123 | "| **nlp**| 1290 | 323 |\n", 124 | "\n", 125 | "## Thoughts\n", 126 | "- **Memory** Note that Fastai was much faster on the full dataset, but I also had to delete the pandas dataframe used to calculate the text lengths as it was taking up too much memory and causing my dataloaders to fail, I probably should find a more memory efficient way to do this calculation. On the other hand, `nlp` datasets won't incur this issue.\n", 127 | "- **nlp pipeline** Given the large difference for the full dataset, I am suspicious about some parts of my implementation, for example sorting the entire dataset takes 416s and splitting it into a train and validation set takes 416s. Do I need to sort the full dataset? Should chopping the dataset in 2 really take so long? Removing these two steps bring the timing down to 458s, slightly faster than Fastai." 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": 16, 133 | "metadata": {}, 134 | "outputs": [ 135 | { 136 | "data": { 137 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1fn48c+TfQ8JSVgSthA2IYosIuACWoqIorW11K1a/Wlra61drK1+Xdqvttpaa22r/VK/av2q1KpVaq1LKyhtbQW0iiCyhiVhCxASsmdmnt8fZ5KZyUISSDKZ5Hm/Xvc1M3fuvXNmIM9z7znnniOqijHGGBMV7gIYY4zpHSwhGGOMASwhGGOM8bOEYIwxBrCEYIwxxs8SgjHGGMASgokQIhItIpUiMrwrt+3tRKRARHq0b7iIbBSR03vyM03vEBPuApi+SUQqg14mAXWA1//6y6r6dGeOp6peIKWrt+3vROQpYIuq3tW4TlXHha9EJpwsIZhuoapNAVlEtgP/T1X/1tb2IhKjqp6eKFtX6s3l7s1lM72TVRmZsBCRu0XkWRFZKiJHgMtFZKaI/FtEDovIHhF5SERi/dvHiIiKyEj/66f8778qIkdE5F8iMqqz2/rfXyAim0SkXER+KSL/FJGrOlHup0TkrqBtPuVPgo2vi0XkWyLykf8zlopIfBvHjxaRn4vIQRHZCpzT7P1iEZnTrDxP+J8X+L/3l0RkJ/CGiESJyPMistf/u74lIhP8238VWAzc6q9ie7H5Z4hIgv+32yMiJSLygIjEBX9PEfmuiJSKyG4R+WLb/+qmt7OEYMLpM8AzQDrwLOABvgFkAbNxwfDLR9n/UuB2IBPYCfx3Z7cVkRzgD8DN/s8tAk7pZLk74vPAPCAfmApc0cZ21wOfBk7yl+PzHTx+sDOA8cBC/+s/A2OAwcA64P8AVPVhXPl/pKopqvqZVo51BzANOBE4Gffv8v2g9/OARGAo8BXgERFJO4Yym17AEoIJp3+o6suq6lPVGlVdrarvqqpHVbcBS4Azj7L/86q6RlUbgKeBycew7XnAB6q6zP/ez4EDnSl3h74pPKiqe1X1IC5At1XWzwM/V9Vi/7b3dvD4we5U1Wr/b+pT1SdU9Yiq1gJ3AVNFJLmDx7oMuEtVS1V1P/BDQpNZLXC3qjao6p9wbUVjj6HMphewhGDCaVfwCxEZLyKv+Ks3KnDBJ+so++8Nel7N0RuS29p2aHA51I32WNyZcndQR8saUh5gxzF8VtP+/iqon4jINv9vusX/1tF+12BDmpVhB5Ab9PqAvxG/UXv/DqYXs4Rgwql5d8r/wVVpFKhqGq66Qrq5DHtw1R4AiIgQGvBa07zcVbieVI0GH2d5hgW9bt51tt3P0tAhjL8InAuchaviKvCvb/xd2+vSugcY0aw8Je3sYyKUJQTTm6QC5UCVv+HzaO0HXeXPwBQROV9EYnBtGNmdPMYHwEIRyRCRIcCNx1GePwA3iUiuiAwEbmnls77gbzg/BbioneOl4qpxDuISyT3N3t+Ha9doy1LgDhHJEpFsXDvMUx37KibSWEIwvcm3gSuBI7irhY422B4zVd2H62nzAC5ojgb+gwuiHfUEsAFXnfIa8PvjKNIjwJvAR8Bq4Plm79+GazA+jAvOz7RzvMeB3f5lPfBOs/cfBU4SkTIRaf5ZAD8APvSXZy3wLvDjjn4ZE1nEJsgxJkBEonHB83Oq+vdwl8eYnmRXCKbfE5FzRCTdf2/A7bjur6vCXCxjepwlBGPgNGAbrrvpOcCFqtqZKiNj+gSrMjLGGAPYFYIxxhi/iB7cLisrS0eOHBnuYhhjTER57733Dqhqi+7VEZ0QRo4cyZo1a8JdDGOMiSgi0uod8BFZZeS/iWhJeXl5uItijDF9RkQmBP/AYtelp6eHuyjGGNNnRGRCMMYY0/UsIRhjjAEsIRhjjPGLyIRgjcrGGNP1IjIhWKOyMcZ0vYi+D8EYY/o8Vag7DEeK4cguqPQ/TvwSDDjaVBadZwnBGGPCpa1gfyTosbIYGqpC95MoGDLTEoIxxkQEVagrbzvQN65vLdgnD4HUPMguhFELIHWYe506DFLyIGUIRHV9+LaEYIwxndUY7I8W6I/s6lXBviMiMiGIyPnA+QUFBe1ua4wxnebzQuVuqNgOFTv8i/95Y+BvqAzdJzjYD5wII88JDfSpwyB5METHhuMbdUhEz4cwbdo0tcHtjDGd5q13gb0x2JdvhyNBzyuLwecJ3ScpB9JGQOrwlmf1ERDsg4nIe6o6rfn6iLxCMMaYo2qohoqdrZ/hV+xwZ/8EnwwLpOS6gJ872z2mjYC0kYEkEJsYlq/SkywhGGMiT1156Nl9xQ53ht/4vKY0dPuoGHcWnzYSRswLDfZpI9zZfnRcz3+PXsYSgjGm9/HUQXkRlG+Fw1uhfJt73ZgE6g6Hbh+TAKn+4J5zMqSP9Af6Ee558hCIig7HN4kolhCMMeFRc8gF+sNbA4G/MfgfKSakSic2GdJH+at0TmtZpZOUAyLh+iZ9RkQmBOtlZEwEUJ8L7I1Bving+x+bn+UnDYIBo2HYHEgf7Z6n57tHC/g9wnoZGWOOXUMNVBSFnt03Pq8ocr15GkXFuDP6xiA/YHRQ4B8FcSlh+xr9jfUyMsZ0nirUHmp5dt8Y+CtLQrePS3VBPmsSFFwQGvxTh4XthivTMfavY4xxjbiHt0DZRji0MfSxtix02+QhLsCP+FTLqp3ELKvaiWCWEIzpL1Shep8L9Ic+CQ365UWuzr9RylDIGAfjFkPG2NCqndik8H0H060sIRjT13hqoWxzIOAHB//6isB2MYku2OdMhfGXQuY4lwQyxkJ8WvjKb8LGEoIxkUjV3W3bvIrn0Ceun35wl82UPMgcDydc4QJ+pn9JHebG3zHGzxKCMb1ZQ7U7229exXNoY+jgarHJ7sx+yKkw8arA2X7mWPeeMR0QkQnB7kMwfY6nDg5tgAMfwYF1/sf1cGRn0EYCacNdoJ/0JX/AH++Cf0quNeaa42b3IRjTk9TnGnBLP/IHfX8CKNsE6nXbRMdB5gQ3hHJjwM8YBxljrEHXdAm7D8GYnla9v2XgP7g+dNKU9HzIKoSxn4WBk9ykKQPGRMwwyqZvsYRgzPFqqHLVOweaBf/q/YFtErNdsC/8fy4BZE1yVwB2d67pRSwhGNNRPo9r4G0M+o1n/+VFNPXqiUmCrImQf54/8PuDf/KgsBbdmI6whGBMc6puULaQBt6PXKNv49g8Eu3q9AdNdb16sgrdFUD6KOvKaSKWJQTTv6m6fvv71sDeNbBvNex7P3QkzpQ8d5Y/4tMu6GcVusbemITwlduYbmAJwfQvlbth7+qgBLAGag6496JiIfskN1xD9omB6p6EjPCW2ZgeYgnB9F3VpYHA35gEqva49yTa1fWPXgSDp8OgaS4BxMSHt8zGhFFEJgS7Mc20UFsG+94LnPXvXR10U5e4Kp4Rn3KBf/B0dyVgffqNCWE3ppnIU38E9v8n9Mz/8JbA+wNGBwL/oGlujl0brM2YJnZjmolMDTVQ+kHomf+hT2jq5pk6zAX+SVe74D9oKiRmhrXIxkQqSwim9/DWu+6dwWf+B9YFhnRIGuSC/7jF/rP/qda/35guZAnBhE9DNez+FxS/7ZY974K3zr2XkOmCfv55geqflKE2gJsx3cgSguk59Ueg5J9QvNIlgL2rwdfgbuTKORkmfxWGzITB09xk7Bb8jelRlhBM96k9DCX/CFwB7HvfVf9Exbiz/qnfgrwzIHc2xKeHu7TG9HuWEEzXqT4AJX93wX/X21D6IaBuOOfBp8Ap34O8M2HoTBvUzZheyBKCOXZV+wLBv/htN7QzuCEdhsyEmXfCsDNh8AyITQxvWY2JYPX1UFICu3a5ZedOuPRSGDGiaz/HEoLpuCPF/uqflS4JlG1062OTYehsmHAp5J7hGoDtjl9jOsTng717Q4N94/PGZe9eN+xWsEmTLCGYnlS+PfQKoHybWx+XBrmnub7/w86EnCk2oYsxrVCFsrKWQT74dXExeDyh+yUnw7BhbiksDDwfNgyGD4e8PLdNV7OEYBxVd7dvY/AvXhkY+iEhw535n3yDawPIPgmiosNbXmN6gcrKlmfzzQN+dXXoPrGxkJvrAvvs2YEgHxz0MzLC08kuIhOCjWXUReoqoOhV2PYy7FweGPgtMdud+U+/2fUCyppkY/ybfkcVDhyA7dtDl507AwG/rCx0HxEYPNgF9UmT4NxzQ8/shw2DQYMgqpf+OdlYRv1NxS7Y+ifYugx2veXuA0jMguGfckkg70w3EJzdA2D6uMbqnKKi0IAf/LqqKnSfAQNcvX3zKpzG57m5EBfX41+l02wso/5K1XX/3LLMJYL977v1GWNgyjdg9AWuG6hVAZk+6PDhlkE++PWRI6Hbp6XBqFFQUACf+pR7PnJkYEnv47fLWELoi7wNrh2gMQkc2QmIC/yn3+uSwMDx4S6lMcetoqL1M/vG1+XlodunpASC/Jw57jE46A8Y0L8vji0h9BV15a49YOufoOgv7nVMAgyfBzPvcGMC2UBwJsLU17vAvmWLe2we9A8dCt0+KSkQ5GfPDgT6xqCfmdm/A357LCFEslbbA7Kh4CIouABGzLNJYEyvV1MD27a5oN982bnT9dNvlJAQCPIzZrQM+FlZFvCPhyWESBLSHrDMTRIDkDEWptzkksCQU609wPQ6lZWwdWvrQb+4OHTbzExXhz9rFnzxi+756NFuycmxgN+dLCH0dkdtD7jPJYHMceEupTGUl7ce8LdscXfaBsvJcYH+rLPcY+MyerRLCCY8LCH0Ro3tAVuWwfZX/e0Bia4KaOadMPo8SMoJdylNP6Pq6uzbCvoHDoRuP3SoC/Lnntsy6KfZjKa9kiWE3qJip7sC2LIMit8Cn8e1B4z5rOsVNOJT1h5gekRNDWzaBBs2uGXTpkDQP3w4sJ2I63tfUAAXXRQa9PPzu2doBdO9LCGE08GPYeMf/PcHNLYHjHPzBIy+AIbMsPYA020OHoRPPgkE/g0b3Ovt2wMDqUVFucbaMWNcI27jGX5BgWvITUgI5zcwXc0SQk/z1sPmP8KHj7jxghAYOgvO+AmMXmTtAaZL+XxuiIXgwN/4vLQ0sF1iIowb54L+VVfBhAluKSiwoN+fWELoKRU7YO0S+OhRqN4P6fkuCZzwRbs/wBy3+nrYvLnlGf/GjaGDqw0c6AL9BRcEgv748W44ht46vo7pOZYQupP6YPvr8MEjUPSKW5d/nps7eMQ8GzDOdFp5uQv6zc/4t24Frzew3YgRLtifeWZo4M/ODl/ZTe9nCaE7VB+AdY/B2v9xcwgkDYJTvg8nXgdpw8NdOhMB9u+Hjz5qeca/Z09gm9hYGDsWTjwRPv/5QOAfO9YadM2xsYTQVVRh979c28CmP7i2grwz4bQfwZjPuHmFjWmmocEF/bVr4cMPA8u+fYFt0tJcoJ8/353lNwb+UaMgxv6CTReKyP9OvWo+hPpK2PA0fPgwlK51s4kVXgcnfQWyJoa7dKYXOXAgEPAbE8DHH7v6f3DDJk+cCAsWwEknuZmyJkyAIUPs7lzTM2w+hGN1YJ1rG9jwf1B/BLInu7aB8ZdAXEp4ymR6BY/H9d0PPuNfuxZ27w5sM2SIq+o56aTAMnasqwYyprvZfAhdobHL6AcPQ8nfIToexn0eTvqqu2fATuP6nUOHWlb3rF8PdXXu/dhYOOEEOPvsQOA/8UQ3dIMxvY0lhI5otcvoT2HiVZCUFe7SmR7g9bpuncHVPR9+GDowW06OC/hf/3rg7H/8+MiYQcsYsITQtqYuow+7+QXAuoz2E4cPhwb9tWth3To3pAO4htzx412XzsYz/pNOcnPpGhPJLCE0V10a1GW0yLqM9nH19S7ov/tuYNm8OfD+wIEu2H/lK4EqnwkTID4+fGU2prtYQoCgLqMPw6bnAl1GT78XCi60LqN9hKqbcSs4+P/nP4H6/sGDA0M3nHyyC/7Ww8f0J/07IdQf8XcZfSTQZfTEL7suowNPCHfpzHE6fBhWrQoE/1WrAuP3JCbC1Klwww0uCcyY4UbutOBv+rP+mRAOrHdtA8FdRuctsS6jEayhwdX1B5/9b9wYeH/CBFi4MBD8J02yLp7GNNc/E8IHv4J1j1uX0QilCjt2hAb/99+H2lr3fk6OC/pXXOEep0+H9PTwltmYSNA/E8Kpd8Cs/7YuoxGivBxWrw5NAPv3u/cSEmDKFLj++sDZ/4gRlt+NORb9MyGkDAl3CUwbPB43qFtw8P/kk8CELePGwTnnBIL/iSda1Y8xXaV/JgTTa9TXw7//DW++CStWwJo1gf7+WVku6F9ySaDqJyMjvOWNdA0NDRQXF1PbWL9m+rSEhATy8vKI7eBZkyUE06O8XvjgA5cAli+Hv//dTeAi4nr9XHdd4Ox/1Cir+ulqxcXFpKamMnLkSMR+3D5NVTl48CDFxcWMGjWqQ/tYQjDdStX19lm+PHAVUFbm3pswAa6+2o3zc+aZdvbfE2pray0Z9BMiwsCBAykNniu1HZYQTJfbtSuQAJYvh5ISt374cLjwQpcAzjrL3fRlep4lg/6js//WlhDMcTt40J35v/mmWxqHfsjKcoH/7LPdkp9vVUDG9GaWEEynVVa6uv/GBPDhh65qKCXFVf1cf71LBIWFNnG7MZHE/lxNu+rrYeVKuPNOOP10V9d/7rnwy1/CgAHwwx/CO++4uQH+/Gf45jfdOECWDExbHnroISZMmMBll13W4X0OHz7Mww8/3KFtZ82adaxFa9eDDz5IdXV1p/Z56623OO+889p8/5NPPmHmzJnEx8dz//33h7z32muvMW7cOAoKCrj33nuPqcwdZVcIpoW2egJFRbmeQN/5jrsCmD0bkpLCXVpzrG7aDB9Udu0xJ6fAg2Pa3+7hhx/m1Vdf7XDvFwgkhK9+9avtbvvOO+90+Lid4fV6efDBB7n88stJ6sL//JmZmTz00EO89NJLLT7va1/7Gn/961/Jy8tj+vTpLFq0iBNO6J6x1uwczqDqbv56+GH47GchOxumTYNbbnENxNdcAy++6NoKVq2CH/8Y5s2zZGCOzVe+8hW2bdvGokWLuO+++5g1axYnn3wys2bNYqN/AKr169dzyimnMHnyZE488UQ2b97M9773PbZu3crkyZO5+eabqays5Oyzz2bKlCkUFhaybNmyps9ISWl7TLI9e/ZwxhlnMHnyZCZNmsTf//53AJYuXUphYSGTJk3illtuCTnWHXfcwYwZM7jnnnvYvXs3c+fOZe7cuQC88cYbzJw5kylTpnDxxRdTWemy7Guvvcb48eM57bTT+OMf/3jU3yQnJ4fp06e3uF9g1apVFBQUkJ+fT1xcHF/4whdCvmeXU9WIXaZOnarm2Hi9qu+8o/rNb6oOH67q0oLqiBGqV1+t+tRTqrt3h7uUpqt9/PHH4S6CqqqOGDFCS0tLtby8XBsaGlRV9a9//atedNFFqqp6ww036FNPPaWqqnV1dVpdXa1FRUU6ceLEpmM0NDRoeXm5qqqWlpbq6NGj1efzqapqcnJym599//336913362qqh6PRysqKrSkpESHDRum+/fv14aGBp07d66++OKLqqoK6LPPPtui7I2fe/rpp2tlZaWqqt577736gx/8QGtqajQvL083bdqkPp9PL774Yl24cGG7v8udd96pP/3pT5teP/fcc3rNNdc0vX7yySf1a1/7WrvHCdbavzmwRluJqVZl1I/4fO6u4Oeeg+efd9M/xsXB/Plw223WE8j0vPLycq688ko2b96MiNDQ0ADAzJkzueeeeyguLuaiiy5izJiW9VCqyq233srKlSuJioqipKSEffv2MbidqeumT5/O1VdfTUNDAxdeeCGTJ09m+fLlzJkzh+zsbAAuu+wyVq5cyYUXXkh0dDSf/exnWz3Wv//9bz7++GNmz54NQH19PTNnzuSTTz5h1KhRTeW+/PLLWbJkSad/H20csyVId3YbtoTQx/l88K9/BZJASYlLAuec46p+zj/fRgI14XP77bczd+5cXnzxRbZv386cOXMAuPTSS5kxYwavvPIK8+fP59FHHyU/Pz9k36effprS0lLee+89YmNjGTlyZIeG5DjjjDNYuXIlr7zyCldccQU333wzaWlpbW6fkJBAdHR0q++pKvPmzWPp0qUh6z/44IMuCdx5eXns2rWr6XVxcTFDhw497uO2xdoQ+iCfD/7xD/jGN9zNYKedBr/5jWsXeOopN0nMsmVw+eWWDEx4lZeXk5ubC8ATTzzRtH7btm3k5+dz4403smjRItauXUtqaipHjhwJ2TcnJ4fY2FhWrFjBjh07OvSZO3bsICcnh2uvvZZrrrmG999/nxkzZvD2229z4MABvF4vS5cu5cwzz2x1/+BynHrqqfzzn/9ky5YtAFRXV7Np0ybGjx9PUVERW7duBWiRMDpq+vTpbN68maKiIurr6/n973/PokWLjulYHWFXCH2Ezwf//Ke7EnjhBdi92837u2AB/OQncN55cJSTIGPC4rvf/S5XXnklDzzwAGeddVbT+meffZannnqK2NhYBg8ezB133EFmZiazZ89m0qRJLFiwgFtuuYXzzz+fadOmMXnyZMaPH9+hz3zrrbf46U9/SmxsLCkpKTz55JMMGTKEH//4x8ydOxdV5dxzz+WCCy5odf/rrruOBQsWMGTIEFasWMETTzzBJZdcQp1/Lta7776bsWPHsmTJEhYuXEhWVhannXYa69ata7NMe/fuZdq0aVRUVBAVFcWDDz7Ixx9/TFpaGr/61a+YP38+Xq+Xq6++mokTJ3biF+4caa2OKhxEZALwDSALeFNVH2lvn2nTpumaNWu6vWy9ldcbmgT27HHzAyxYABdf7JJAamq4S2l6kw0bNjBhwoRwF8P0oNb+zUXkPVWd1nzbdq8QRGQmcDlwOjAEqAHWAa8AT6lq+VH2fQw4D9ivqpOC1p8D/AKIBh5V1XtVdQPwFRGJAn7b7rfsp7xed1/Ac8/BH/8Ie/e6JHDuuS4JLFxoScAYc2yOmhBE5FVgN7AMuAfYDyQAY4G5wDIReUBV/9TGIZ4AfgU8GXTMaODXwDygGFgtIn9S1Y9FZBHwPf8+xs/rdXcKNyaBffvcJPHBSeAo3a6N6Zc++ugjrrjiipB18fHxvPvuu2EqETz++OP84he/CFk3e/Zsfv3rX4epRKGOWmUkIlmqeuCoB2hnGxEZCfy58QrBf8Vxl6rO97/+PoCq/jhon1dUdWEbx7sOuA5g+PDhUzvakBRpPJ7QJLB/v0sCCxcGkkBycrhLaSKNVRn1P11WZdQY6EUkGahRVZ+IjAXGA6+qakN7CaMVucCuoNfFwAwRmQNcBMQDfzlKmZYAS8C1IXTys3s1jwfefjuQBEpL3d3AjUng3HMtCRhjuk9HexmtBE4XkQzgTWANsBjo+MhUAa11zlVVfQt46xiOF9E8HnjrLZcEXnwxkATOOy+QBGyICGNMT+hoQhBVrRaRa4BfqupPROQ/x/iZxcCwoNd5uHaKfqWoCB58EJ55Bg4ccGf+jUlgwQJLAsaYntfRG9PEX/d/Ga53ERz7PQyrgTEiMkpE4oAvAG01SrdVmPNFZEl5eZsdnHqt996DL3wBCgrgkUfccBEvvOCuDH7/eze4nCUD09fZ8NehjmX466KiImbMmMGYMWNYvHgx9fX1nfsirWltgKPmC3AGLmjf4n+dDzzUgf2WAnuABtyVwTX+9ecCm4CtwG0dKUNrS6QMbufzqf7lL6pz57oB5NLSVL/7XdXi4nCXzPQ3vWVwu3Hjxum2bds6tU/zwe3CwePxhAxu11ErVqw46uB2+/bt01WrVumtt94aMridx+PR/Px83bp1q9bV1emJJ56o69evV1XViy++WJcuXaqqql/+8pf14YcfbvXYXT64naquxLUjNL7eBtzYgf0uaWP9XzhKw3FfUV8PS5fC/ffDunWQl+eeX3ut3TVswu+mm9y8F11p8mRXFXo0wcNfX3755SxbtoyamhoSExN5/PHHGTduHOvXr+dLX/oS9fX1+Hw+XnjhBW6//fam4a/nzZvHnXfeyQUXXEBZWRkNDQ3cfffdTXcXp6SkNA1D3dyePXtYvHgxFRUVeDweHnnkEU4//XSWLl3Kj370I1SVhQsXct999zUd61vf+havv/46CxcubBr+OisrixUrVvDGG29w5513UldXx+jRo3n88cdJSUnhtdde46abbiIrK4spU6Yc9TfJyckhJyeHV155JWR98PDXQNPw1xMmTGD58uU888wzAFx55ZXcddddXH/99e3+Gx1Ne/chLMG1GXzUynvJuIblOlV9+rhK0ceUl8OSJfCLX7jB5AoL4cknYfFiN7CcMf3Zb37zG1577TVWrFhBXFwc3/72t4mJieFvf/sbt956Ky+88AK/+c1v+MY3vsFll11GfX09Xq+Xe++9l3Xr1vGBP4t5PB5efPFF0tLSOHDgAKeeeiqLFi1qd1C5Z555hvnz53Pbbbfh9Xqprq5m9+7d3HLLLbz33ntkZGTw6U9/mpdeeokLL7yQqqoqJk2axA9/+EMAHnvsMVasWEFWVhYHDhzg7rvv5m9/+xvJycncd999PPDAA3z3u9/l2muvZfny5RQUFLB48eJj+q1KSkoYNizQ5JqXl8e7777LwYMHGTBgADExMU3rS0pKjukzgrV3hfAwcLuIFOLuTi7F3Zg2BkgDHgMsGfgVF7sk8D//A0eOuFnFHn3UDS9tQ0qb3qa9M/meYMNfH522Mfx1W+uPV3v3IXwAfF5EUoBpBIau2KCqG4/704+RiJwPnF9QUBCuIoT46CNXFfTMM26amYsvhptvhnauEo3p92z466Nra/jrrKwsDh8+jMfjISYmpsuGxe5QLyNVrVTVt1R1qaq+FM5k4C/Py6p6XXoYx25WhRUr3H0CJ57o5hr46ldhyxbXbmDJwJj22fDXR9fW8Nciwty5c3n++ecB+N3vftfm6Kyd0lpLc6Qs4ehl1NCg+vvfq06d6noM5eSo3n236sGDPV4UYzqtt/Qyauyp88477+iYMWN01qxZ+irq6oMAAB5kSURBVF//9V86YsQIVVX90Y9+pCeccIKedNJJOn/+fD3o/wO75JJLdOLEifqd73xHS0tL9dRTT9WpU6fqNddco+PHj9eioiJVPfoUmk888YROnDhRJ0+erKeddlpTb6enn35aJ02apBMnTtSbb765afvmx3rooYd03LhxOmfOHFVVffPNN3XatGlaWFiohYWFumzZMlVVffXVV3XcuHE6e/ZsveWWW47ay2jPnj2am5urqampmp6errm5uU3Tg77yyis6ZswYzc/Pb5r6U1V169atOn36dB09erR+7nOf09ra2laP3ZleRr1m+Otj0ZPDX1dVwWOPwQMPwPbtMHYsfOc7cMUVbrRRYyKBjWXU/3Tp8NfNDpKsqlXHWb6Isn8//PKX8PDDcOgQzJoFP/85LFoEUTbfnDGmD+lQQhCRWcCjQAowXEROAr6sql/tzsKF06ZN8LOfwe9+5+4nuOAC11DcjTdAGmO6kA1/3XkdqjISkXeBzwF/UtWT/evWadCkNz0pqJfRtZs3b+7SY7/zDvz0p27O4bg4uPJK+Na3YNy4Lv0YY8LCqoz6n85UGXW40kNVdzVb5T224h0/7eJeRj4fvPQSzJ7tlrffhttugx073D0FlgyMMf1BR9sQdvmrjdQ/IN2NwIbuK1bPqK11dxD/7GeuimjkSHjoIbj6apt3wBjT/3Q0IXwFNwdyLm6QujeAr3VXobrboUNupNFf/tJNRzllSmCk0ZhjHcPVGGMiXEcHtzvAsU2G0yvddBP83//BOee4huK5c21oCWOM6VAbgn/uggdE5I8i8qfGpbsL111uvx3WroVXX3XjDVkyMCb8rrrqqqY7bztj5cqVTJkyhZiYmBb7/+53v2PMmDGMGTOG3/3ud11V1D6roxUkLwH/C7wM+LqvOB1zvGMZtTJOljEmQg0fPpwnnniixcQyhw4d4gc/+AFr1qxBRJg6dSqLFi0iIyMjTCXt/TqaEGpV9aFuLUknqOrLwMvTpk27NtxlMSZSlb73Q+rKPu7SY8ZnnED21DuOus327dtZsGABp512Gu+88w65ubksW7YsZJuRI0eyePFiVqxYAbghq9s6ARw5ciQAUc3uFH399deZN28emZmZAMybN4/XXnuNSy5pdZoWQ8e7nf5CRO4UkZkiMqVx6daSGWP6rM2bN/O1r32N9evXM2DAAF544YUW26SlpbFq1SpuuOEGbrrppk5/RmtzCXTFnAF9WUevEAqBK4CzCFQZqf+1MSYCtXcm351GjRrF5MmTAZg6dSrbt29vsU3jmfwll1zCN7/5zU5/Rms33XbFkNR9WUcTwmeAfFXtglmcjTH9XXx8fNPz6OhoampqWmwTHLyPJZDn5eXx1ltvNb0uLi5umm/BtK6jVUYfAgO6syDGGBPs2WefbXqcOXNmp/efP38+b7zxBmVlZZSVlfHGG28wf/78ri5mn9LRK4RBwCcishqoa1ypqou6pVTGmH6vrq6OGTNm4PP5jjrBzOrVq/nMZz5DWVkZL7/8MnfeeSfr168nMzOT22+/nenTpwNwxx13NDUwm9Z1dHC7VqcOUtW3u7xEndCT8yEY0xdEyuB2I0eOZM2aNWRlZYW7KBGvy+dDCHfgb663zalsjDF9wVHbEETkH/7HIyJSEbQcEZGKniliS1092qkxpnfZvn17i6uDe+65h8mTJ4cs99xzT5hK2De1d4WQDKCqqT1QFmOMadNtt93GbbfdFu5i9Gnt9TKK3AmXjTHGdEp7Vwg5IvKttt5U1Qe6uDzGGGPCpL2EEI2bR9lu7zPGmD6uvYSwR1V/2CMlMcYYE1bttSHYlYExpkf05HwI7733HoWFhRQUFHDjjTe2Ou5Rf9ReQji7R0phjDHHqHE+hEsvvTRkfeN8CO+++y6rVq3iBz/4AWVlZQBcf/31LFmyhM2bN7N582Zee+21cBS91zlqlZGqHuqpgnSG3ZhmTBdYcRPs/6Brj5kzGeY+eNRNwj0fwpw5c6ioqGgaH+mLX/wiL730EgsWLOj01+1rOjq4Xa9iN6YZE9nCOR9CSUkJeXl5Ldabjg9uZ4zpa9o5k+9O4ZwPweZJaFtEXiEYYyJb8/kQPB5Pi226Yj6EXbt2Nb0uLi5m6NCh5OXlUVxc3GK9sYRgjOmlums+hCFDhpCamsq///1vVJUnn3ySCy64oKuLH5GsysgY0yt153wIjzzyCFdddRU1NTUsWLDAGpT9OjQfQm9l8yEY0zk2H0L/05n5EKzKyBhjDGBVRsaYXqi1Xkf33HMPzz33XMi6iy++uN8Mie1TaFCo90G9Qlo0xHbxKb0lBGP6GVWNyG6WfXk+BFXwqAv0jQG/+WNDs9r9ggQY0E5C6GyTgCUEY/qRhIQEDh48yMCBAyMyKUQiVfA2BvtmgT74jL956I4C4gTioiA9yj3GCsQHPR79c5WDBw+SkJDQ4bJaQjCmH2nsg19aWhruovQZjWf3Xtyjx58Agh9bO0+PFojxL9H4HyXwGO3P116gxr90VkJCQshd2e2JyIRgYxkZc2xiY2MZNWpUuIsRMVThQANsrw0sO+tgZy3sqnPLgYaW+w2KheEJMCzeLc2fD4oLBPzexLqdGmP6LVUo87hAXxQU9ItqAs+rfKH7DIhpPdA3vs6Nb786pyN8nmo81fvw1OzDW1OKp2aff9mPt2Y/WVNuJSGz8JiO3Va304i8QjDGmI4q94QG+JDAXwtHvKHbp0fDqEQYkwTzMmFUAowMWtKOM2r6PDX+oO6Ce+B542uXAHwNR1rsK9HxxCQOIjoxB/XWH19BWmEJwRgT0Y542g7222vhcLNhklKiA0F+zoBAoG9cNyD22Mrh89bh9Qd0T80+PNX78dbu95/lB4J+q4E+Ko7oxBxiEgcRnz6O6MGnE5M4iJjEHGKSBhGd4B6jYtO6tTOAJQRjTK9W63XBvaiNap2DzQJ+UlQgwM9O8z9PDAT+zBjoTExVnxdvbSme6j00VO9t5czeJQFffXnLnaNiXVBPyCE2vYDEwbOaAn1jAohJHERUXHqv6PVlCcEYE3bVXthWA5trYEuzZVddaC+dhKhAcJ+eFlqlMyoBsmI7HvDV1+ACevVePNV7/I978dQEP98H2qxeSWJcoE/MITZ1FIk5p/ircgY1rY9JHERUfEavCPQdZQnBGNMjjnhgaysBf0sNlDSrDs+KhYJEOGMAjEmE0Ykw2h/0c+IgqgMxtqkKpynQu8eGmr14qnbjqdmLt6aU5p1CJTqRmOQhxCQOIXHQTGKSBhOTNMQ9Jg4mJmkw0fEZiPS9kX8sIRhjuszhhtYD/pYa2Nese+bgOBf052W6x8ZldAfq8X2eWjw1e5ud2e8OOcP31h5ssV9UbGpTYI8fMN4f7IMCftKQbq+n780sIRhjOkwVDnnaDvrN++Tn+oP+eQObBf1ESG0j+qjPS0P1fjxVJTRUlbiz+aoSGqr3uOBfsxdfXVmL/aLi0v3BfgjxmZOCgvxgYpOGEp00iOjY1G74VfoOSwjGmBYqPfBJNWyohk3Ngn5wrx3B9b8vSITPZoUG/fxESIpueWyftw5P1W6qqxsDfgkNjUG/qgRP9R7Q0JbiqLgBxCYPJTZpCIlZU1qc1cckDSYqJql7f5R+wBKCMf2UKuxv8Af+Khf8N1S717vqAttF4eruCxLh0pzQoD8qARKaBX1vfYUL7odLOFxVQkO1C/qeqt00VJXgrW0+bIa43jbJuSRknUxs8nnEJOcSm5zrHpOGEhWb3N0/h8ESgjF9nldhR20g2AcH/7KgE/HkKBifBGcOcI8T/MvoRDewGoCqD2/tQTxVxTQcKKGmqoSKxrP8ahf0m/ezl6g4YpKHEpOcS/LQuaHBPjmXmKTBSNQxdv43XcoSgjF9RK3XddtsDPYbqlwC2FgDtUHDL2THukD/+WyYkBwI/nnxICjeukM0VO6koXInnn07Kassbqra8VTvRn2hXYKiYlP9wT2PxJwZxCYNDQr6eUQnDOyTPXL6IksIxkSYww2B+v3g4F9UC41xX3DVPOOT4OwM/9m+P/hnRNW5M/rKXS7w73fBf1flThoqd6GeqpDPi07IJjY5l/jMiaQM+zQxybnEJA1tOsuPjkvr8d/AdA9LCMb0Qqqwuz5QvROcAPYGnaDHCYxNgimpcOkgf+BPVEbHlBFbvdMf9HfQcMAF/4rKnRyq3kNw33uJjic2ZTixycNIGjST2JRhxKQM96/LIyomsed/ABMWlhCMCbNaL6yvhg8rA8vaqtD6/bRoF+zPyXSP4xPqmcBuBjXswFftP9M/tAuP/yx/b7N6/OiEbGJThrsqnZThxKYMc4+pI4hOyLIqHQNYQjCmx6i6s/sPK+HDqkDw31jtJkEBNw5PYTJcnA1T4g4zkR0M9+4iuXYnnqqdNJT46/ar94D62OffT6LiiE0ZTkzKcBKyp/uDfuOSZ10yTYdEZEKwCXJMb1fvc9U7zc/6S4Nu3BoWD9OSavlS0nZO1G2M9BSRWrMNz4Ei6ouK8NUfBqDOv0QnZLmz/OxpxKaMCDnTj07MsbN8c9xsghxjjlNpK2f9G6oDk6In4uWM+D2cFr2VQi1ieMM2MuqK0MoiPFW7Ca7Pj0kcTGxaPnGpo4hNHUlsauBM387yTVexCXKMOU4en+vCGXzW/2FVoJF3gK+MaRRxYfQ2/kuLGNGwjQG1RURVFYV01YyKSSEqLZ+47GnE5ucTl5ZPbOoo4lJH2g1YJqwsIRjTikMNsLbZWf/6KlBfHSM82xnj3capUsT/020MaygivbaI6Iag8XUkhtjU4S7Y551BXOoof+DP9zfi9s/B00zvZgnB9HtVXlhzBN6tcMuaCh+e6t2M9BSR793GBF8RF/i2kddQRGp9CRJUxROdOMhV7wxaEBT0RxGbMgyJsj8vE1nsf6zpV3zq6vffrYB3y5Wth3bjq9jE6IaNjPVs4hrvZn7YsIU4rW3aJyomhdi0UcRlTSU27XPEpea716kjiYpNCeO3MaZrWUIwfdq+ehf4Pzx4gL0HNuGp2MTw+o2MadjEVzybSdHKpm0lcTCJA8cSl3YZcemueicuLZ/ohGyr4jH9giUE02fUeOGDsnI27t3EwYOb0IqNDK7bxBjPZr7gO9S0XUNsBjHp40jPuIj4AWOJTx9L3ICxRMelh7H0xoSfJQQTkbwN1Wzdv5lt+zZRXraR6CObGFS7mcG+vWT5t6mNSqE6eSwJgz5NetZYUjLGEpc+1hp1jWmDJQTTq/m8dTRUbOPgoU3s2r+J6sMbia/cTHbDTgQYDdRKPPvix3AkcxZkjGFk9jgGZ48lJmmoBX5jOsESgukVVH00HCmi7vAmag5vovTgRurLN5NUXUSUf2CHVGI4EJPPjoRCPsn+LBkZ4xgzaCwTsodTGN3K1FzGmE6xhGB6nPq8NBwpovbQR9QdWkfVwY+oLfuYaK8bdtmHsDt6BFtix7A7fT7R6ePIyRzLhEGjOCM9rs25eI0xx8f+tEy3Up+X+iPbqPMH/9pDH1FXtqFpzP16iefjmBNYF38Rm+InEZ0+gbyBBUzNSOS8NP+kLVbrY0yPsIRguoz6vNRXbA0E/7J11JV9jHqqAfBEJbAz/gT+Hf9ZPkwpZEPsJDIzC5ibEcPZGfD9tJbz8xpjeo4lBHNM1OfxB/91TVU/dWUfo94a9350IoeTT2B9+sW8qYWsip7EtpjRTEiJ4ewB8KUMN3dvmv0PNKbXsD9H0y4X/Le0Evzd3bwSnYgMmEjJkMWsjprEi55C/q2j8Uk0I+LhUxnwgww4KwMGxYX5yxhj2mQJwYRQn4f68s3UHlrXVPVTd3hDIPjHJBGfcQLx+ZewKW4SKyjkhZp8PqmNBh9kRcNZ2fBIhpvLNz/B2gCMiRSWEPoxVaWhcie1pWuoPfhBUPCvA0BikonPOIH0gkuRAZNYG1vI6/Wj+NvhaP5TCVoFKdFwRjpcm+sSQGEyRFkCMCYiWULoR9Tnpb58IzX7V1NT6hZvzX7ADeAWn3kC6QWXE585ieiMQj5kFG8ejuJvZfCvYjfhS6zAzDS4a6RLAKekQqxN1GVMn2AJoQ/zeeuoO7jWBf/9q6k98B4+/+TrMUlDScqZSULOdBKzpxOTVsD66ijeLIM3y+DtYqj0ggAnp8BNeS4BnJYOydYTyJg+yRJCH+JtOEJt6fvUlK6iZv9q6g5+2DRTV1xaASkjziMxezqJOdOJTc6jxguvH4Lnd8Mb6wPz/Y5NhCsGuQQwZwAMjA3jlzLG9BhLCBHMU1NKTelqav1VQHWHN4D6QKKJz5hI+tgrSMw+hcTsaUQnZAJQ64VXDsEfdsDLB91VwMAYWDAQzh7gksCwhDB/MWNMWFhCiBCuAXgHtaVr/G0Aq2g4sh0AiU4gIetkMifeQGLOdBIGnhwyN2+tF14uhedK4U9BSeCSHLg4G+YOgBhrBzCm37OE0EsFGoBX+RuA1wQagOPSScyeRvroL5CQcwoJGROR6NAO/rVeeO2QSwIvH4QjzZLAnAHWGGyMCWUJoZc4egPwEBJzTiXR3wAclz4GkZbRvNYLr5fBH/aHJoHFOfB5SwLGmHb0moQgIhcCC4Ec4Neq+kaYi9St1FtPTelqqve906IBODZtNCnDF/oTwCnEJOe2Oa5/YxJ4br+rDjrihUx/EmisDrIkYIzpiG5NCCLyGHAesF9VJwWtPwf4BRANPKqq96rqS8BLIpIB3A/0uYTgqSmles/bVJW8SfWef+DzVLZoAE7InkpMwsCjHqfWC2/4rwSCk8Dns+HzOZYEjDHHpruvEJ4AfgU82bhCRKKBXwPzgGJgtYj8SVU/9m/yX/73I56qUlf2MVUlb1K1ezl1B9cCSnTiIFJGnEdy7lkkDZoV0gDclsYk8FwpLDsQmgQuzoGzLAkYY45TtyYEVV0pIiObrT4F2KKq2wBE5PfABSKyAbgXeFVV32/rmCJyHXAdwPDhw7uj2MfF56mmeu8/qdq9guqSFXhq9gIQP/AkMgtvIjn3bOIzTujQ1I51PnefwHOl8KcDUOGFDEsCxphuEo42hFxgV9DrYmAG8HXgU0C6iBSo6m9a21lVlwBLAKZNm6bdXNYOaagqpqpkBVUly6nZ9w7qq0dikkkecjrJuWeTNORMYhKzO3SsOh+8cQj+0CwJfM5fHWRJwBjTXcKREFo7NVZVfQh4qKcLcyzU56X24H+oKllOVcly6ss3AhCbMoL0MZeTnDuXxOxTWnQFbUtjEmisDgpOAhdnu5vFLAkYY7pbOBJCMTAs6HUesDsM5egUb3051btXUrV7OVV73sZXVwYSQ2L2NLJOvpXk3LOITc3vUFUQgKobM+jJfaFJ4LPZrkrorAyIsyRgjOlB4UgIq4ExIjIKKAG+AFwahnIclarSULHVJYCS5dSUrgH1EhWfQfKQOa5BeMgZRMeldeq4Hp+7EvjJLvigMpAEGq8ELAkYY8Klu7udLgXmAFkiUgzcqar/KyI3AK/jup0+pqrrO3nc84HzCwoKurS8Pm8dNftXUV2ynKrdy2mo3AlA3IDxZJzwZZKHnk3CwJOQqM4P91nlhcf2wAPFsL0WxifB/46DywZBvCUBY0wvIKq9ol32mEybNk3XrFlzXMfw1JRStXuF6xW05++opwqJjidx0GySc88ieegcYpNzj/n4pfXwqxK3HPLA7DS4ZTgsHGgTyRhjwkNE3lPVac3X95o7lXtS3eFPqNz1BlUlb1J3aC3ghodIG3khSblz3b0BMYnH9Rlba+CBXfDYXqj1wYVZcPMwmJXeFd/AGGO6Xr9MCOWbn6Z889MkZJ3MwBO/TXLuWcQNmNDhBuGjWVMBP90Fz5dCjMAXB8O382B8+/eeGWNMWPXLhJAx8atkFt7U7hARHaXq7iL+yU5YfhjSo+G7w+HGXBgS3yUfYYwx3S4iE8LxNirHJg3pknI0+NwNZD/ZCWurIDcO7h8N1w6BtIj8ZY0x/VlE9m9R1ZdV9br09PBUyFd64MFdUPAuXL4BvApPjIdtp8K3h1kyMMZEJgtdnbC/Hh4qhod3Q5kHzkiHh8fCgkzrMWSMiXyWEDpgczX8bBc8sRfqFT7j7zF0qvUYMsb0IZYQjmJVhWsf+OMBiBO4ajB8axiMTQp3yYwxputFZELorjuVwfUYevWQSwRvl8OAGLh1OHw9DwZ1bKw6Y4yJSNao7Ffvgyf3wolrYOFHsK0Wfj4adp4Kd+dbMjDG9H0ReYXQlY544Ld74OfFUFwHhcnwf+PdnMQ25LQxpj/ptwlhbx08VAIPl0C5181D/NuxMD8TuuCGZWOMiTj9MiHcvR3+ewd41A09ffMwmN65UayNMabP6ZcJYXgCXDMEvpUHBdZjyBhjgAhNCMfby+iLg91ijDEmICKbTcM9dIUxxvRFEZkQjDHGdD1LCMYYYwBLCMYYY/wsIRhjjAEsIRhjjPGzhGCMMQaI0IQgIueLyJLy8vJwF8UYY/oMUdVwl+GYiUgpsOMYd88CDnRhcSKd/R4B9luEst8jVF/4PUaoanbzlRGdEI6HiKxR1WnhLkdvYb9HgP0Woez3CNWXf4+IrDIyxhjT9SwhGGOMAfp3QlgS7gL0MvZ7BNhvEcp+j1B99vfot20IxhhjQvXnKwRjjDFBLCEYY4wB+mlCEJFzRGSjiGwRke+FuzzhIiLDRGSFiGwQkfUi8o1wl6k3EJFoEfmPiPw53GUJNxEZICLPi8gn/v8nM8NdpnARkW/6/07WichSEUkId5m6Wr9LCCISDfwaWACcAFwiIieEt1Rh4wG+raoTgFOBr/Xj3yLYN4AN4S5EL/EL4DVVHQ+cRD/9XUQkF7gRmKaqk4Bo4AvhLVXX63cJATgF2KKq21S1Hvg9cEGYyxQWqrpHVd/3Pz+C+2PPDW+pwktE8oCFwKPhLku4iUgacAbwvwCqWq+qh8NbqrCKARJFJAZIAnaHuTxdrj8mhFxgV9DrYvp5EAQQkZHAycC74S1J2D0IfBfwhbsgvUA+UAo87q9Ce1REksNdqHBQ1RLgfmAnsAcoV9U3wluqrtcfE4K0sq5f970VkRTgBeAmVa0Id3nCRUTOA/ar6nvhLksvEQNMAR5R1ZOBKqBftrmJSAauJmEUMBRIFpHLw1uqrtcfE0IxMCzodR598NKvo0QkFpcMnlbVP4a7PGE2G1gkIttxVYlnichT4S1SWBUDxaraeNX4PC5B9EefAopUtVRVG4A/ArPCXKYu1x8TwmpgjIiMEpE4XMPQn8JcprAQEcHVD29Q1QfCXZ5wU9Xvq2qeqo7E/b9Yrqp97iywo1R1L7BLRMb5V50NfBzGIoXTTuBUEUny/92cTR9sYI8JdwF6mqp6ROQG4HVcT4HHVHV9mIsVLrOBK4CPROQD/7pbVfUvYSyT6V2+DjztP3naBnwpzOUJC1V9V0SeB97H9c77D31wCAsbusIYYwzQP6uMjDHGtMISgjHGGMASgjHGGD9LCMYYYwBLCMYYY/wsIZg+TURURH4W9Po7InJXN3zOVSJSKiIfBC1dNlCgiNwlIt/pquMZ0xpLCKavqwMuEpGsHvisZ1V1ctDSX2/iMhHKEoLp6zy4G4i+2fwNEXlCRD4X9LrS/zhHRN4WkT+IyCYRuVdELhORVSLykYiM7uiH+4+1UkReFJGPReQ3IhLlf+8S//HWich9QfucIyLvi8iHIvJm0OFOEJG3RGSbiNzo3zZZRF7xb7tORBZ3+hcyxq/f3als+qVfA2tF5Ced2OckYAJwCHeH7qOqeop/EqGvAze1ss9iETkt6HXjZDKn4Obe2AG8hrtieQe4D5gKlAFviMiFwD+B3wJnqGqRiGQGHW88MBdIBTaKyCPAOcBuVV0IICLpnfiOxoSwhGD6PFWtEJEncROc1HRwt9WqugdARLYCjUMdf4QLyq15VlVvCF7hhr1hlapu879eCpwGNABvqWqpf/3TuLkHvMBKVS3yl/1Q0OFeUdU6oE5E9gOD/OW533+F8WdV/XsHv58xLViVkekvHgSuAYLH8/fg/xvwD1gWF/ReXdBzX9BrH50/kWo+PozS+jDs+Ne3NZ5McJm8QIyqbsJdZXwE/FhE7uhk2YxpYgnB9Av+M+0/4JJCo+24YApurPvYbvr4U/yj60YBi4F/4CYiOlNEsvzTul4CvA38y79+FECzKqMWRGQoUK2qT+EmcOmvw1ObLmBVRqY/+RkQXKXzW2CZiKwC3sRNAHM8mrchfNX/+C/gXqAQWAm8qKo+Efk+sAJ3VfAXVV0GICLXAX/0J5D9wLyjfGYh8FMR8eGqoa4/zu9g+jEb7dSYbiQic4DvqOp54S6LMe2xKiNjjDGAXSEYY4zxsysEY4wxgCUEY4wxfpYQjDHGAJYQjDHG+FlCMMYYA8D/B1JTAF6/MUAlAAAAAElFTkSuQmCC\n", 138 | "text/plain": [ 139 | "
" 140 | ] 141 | }, 142 | "metadata": { 143 | "needs_background": "light" 144 | }, 145 | "output_type": "display_data" 146 | } 147 | ], 148 | "source": [ 149 | "#hide\n", 150 | "def get_timings(n_epochs, init, per_ep):\n", 151 | " #init=48\n", 152 | " #eps = n_epochs * 14.25\n", 153 | " eps = n_epochs * per_ep\n", 154 | " return init+eps\n", 155 | "\n", 156 | "fastai_sorted_10 = [48, 14.25]\n", 157 | "fastai_sorted_100 = [484, 142]\n", 158 | "nlp_10 = [71, 11.27]\n", 159 | "nlp_100 = [1290, 323]\n", 160 | "\n", 161 | "timings_ls, timing_data = [], []\n", 162 | "timings = [fastai_sorted_10,fastai_sorted_100, nlp_10, nlp_100]\n", 163 | "\n", 164 | "n_eps = list(range(1,20,2))\n", 165 | "\n", 166 | "for t in timings: timing_data.append([get_timings(n_epochs=n, init=t[0], per_ep=t[1]) for n in n_eps])\n", 167 | "\n", 168 | "colors = ['deepskyblue','blue','goldenrod', 'darkorange']\n", 169 | "labels = ['fastai_sorted_10','fastai_sorted_100','nlp_10','nlp_100']\n", 170 | "for i,t in enumerate(timing_data):\n", 171 | " plt.plot(t, color=colors[i], label=labels[i])\n", 172 | "\n", 173 | "plt.title('Training run duration')\n", 174 | "plt.xlabel('Num Epochs')\n", 175 | "plt.ylabel('Time (s)')\n", 176 | "plt.yscale('log')\n", 177 | "plt.legend()\n", 178 | "plt.show();" 179 | ] 180 | }, 181 | { 182 | "cell_type": "markdown", 183 | "metadata": {}, 184 | "source": [ 185 | "## Thanks for Reading 😃\n", 186 | "As always, I would love to hear if you have any comments, thoughts or criticisms, you can find me on Twitter at [@mcgenergy](www.twitter.com/mcgenergy)\n", 187 | "\n", 188 | "## [Appendix]\n", 189 | "\n", 190 | "## Code:\n", 191 | "For those curious, you can peek the code used in testing this below 👇" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 1, 197 | "metadata": { 198 | "scrolled": true 199 | }, 200 | "outputs": [], 201 | "source": [ 202 | "#hide_collapse\n", 203 | "# Imports\n", 204 | "%reload_ext autoreload\n", 205 | "%autoreload 2\n", 206 | "\n", 207 | "from fastai2.basics import *\n", 208 | "from fastai2.text.all import *\n", 209 | "# from fastai2.callback.all import *\n", 210 | "# from fastai2.data.transforms import RandomSplitter\n", 211 | "from fastai2.text.core import defaults\n", 212 | "\n", 213 | "from nlp import load_dataset\n", 214 | "\n", 215 | "import spacy,html\n", 216 | "from spacy.symbols import ORTH\n", 217 | "\n", 218 | "import timeit\n", 219 | "import gc" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "### Fastai Testing\n", 227 | "Init timing:" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 2, 233 | "metadata": {}, 234 | "outputs": [ 235 | { 236 | "data": { 237 | "text/html": [], 238 | "text/plain": [ 239 | "" 240 | ] 241 | }, 242 | "metadata": {}, 243 | "output_type": "display_data" 244 | } 245 | ], 246 | "source": [ 247 | "#hide_collapse\n", 248 | "#%%timeit -n 1 -r 3\n", 249 | "\n", 250 | "# Download data and save as csv\n", 251 | "# senti_dataset = load_dataset('sentiment140', split='train[:100%]', download_mode='reuse_cache_if_exists')\n", 252 | "# df = senti_dataset.data.to_pandas()\n", 253 | "# df.to_csv('sentiment140.csv')\n", 254 | "\n", 255 | "# Read data; the first 10% of the sentiment140 dataset, extraced from the `nlp` library and saved as a csv\n", 256 | "#fn_10pct = 'sentiment140_10pct.csv'\n", 257 | "fn = 'sentiment140.csv'\n", 258 | "df = pd.read_csv(fn, index_col=None)\n", 259 | "\n", 260 | "# SORT: Calculate text sample lengths\n", 261 | "df['word_count'] = df['text'].str.split().map(len)\n", 262 | "\n", 263 | "res=df['word_count'].values\n", 264 | "\n", 265 | "# Create Dataloaders\n", 266 | "dls = TextDataLoaders.from_csv(path='.', csv_fname=fn, valid_pct=0.2, bs=64, \n", 267 | " text_col='text', label_col='sentiment' , res=res)" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "1 epoch timing" 275 | ] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": 6, 280 | "metadata": { 281 | "scrolled": true 282 | }, 283 | "outputs": [ 284 | { 285 | "data": { 286 | "text/plain": [ 287 | "(143.04122078799992, 0.007152061039399996)" 288 | ] 289 | }, 290 | "execution_count": 6, 291 | "metadata": {}, 292 | "output_type": "execute_result" 293 | } 294 | ], 295 | "source": [ 296 | "#hide_collapse\n", 297 | "\n", 298 | "del df, res\n", 299 | "gc.collect()\n", 300 | "\n", 301 | "# Do 1 pass of the training dataloader\n", 302 | "s = \"\"\"for b in dls.train: pass\n", 303 | " \"\"\"\n", 304 | "\n", 305 | "time = timeit.timeit(stmt=s, number=1, globals=globals()); time\n", 306 | "time, time / len(dls.train)" 307 | ] 308 | }, 309 | { 310 | "cell_type": "markdown", 311 | "metadata": {}, 312 | "source": [ 313 | "### HuggingFace `nlp` Datasets Testing" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "Tokenizer, Numericalizer and Padding functions:" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": 2, 326 | "metadata": {}, 327 | "outputs": [], 328 | "source": [ 329 | "#hide_collapse\n", 330 | "class SpacyTokenizerNLP():\n", 331 | " \"Spacy tokenizer for `lang`\"\n", 332 | " def __init__(self, lang='en', special_toks=None, buf_sz=5000):\n", 333 | " self.special_toks = ifnone(special_toks, defaults.text_spec_tok)\n", 334 | " nlp = spacy.blank(lang, disable=[\"parser\", \"tagger\", \"ner\"])\n", 335 | " for w in self.special_toks: nlp.tokenizer.add_special_case(w, [{ORTH: w}])\n", 336 | " self.pipe,self.buf_sz = nlp.pipe,buf_sz\n", 337 | " \n", 338 | " def encodes(self, items):\n", 339 | " tmp = [list(doc) for doc in self.pipe(items, batch_size=self.buf_sz)]\n", 340 | " return {'tok_text_pre': [list(str(t) for t in l) for l in tmp]}\n", 341 | "\n", 342 | "def make_vocab(count, min_freq=3, max_vocab=60000, special_toks=None):\n", 343 | " \"Create a vocab of `max_vocab` size from `Counter` `count` with items present more than `min_freq`\"\n", 344 | " vocab = [o for o,c in count.most_common(max_vocab) if c >= min_freq]\n", 345 | " special_toks = ifnone(special_toks, defaults.text_spec_tok)\n", 346 | " for o in reversed(special_toks): #Make sure all special tokens are in the vocab\n", 347 | " if o in vocab: vocab.remove(o)\n", 348 | " vocab.insert(0, o)\n", 349 | " vocab = vocab[:max_vocab]\n", 350 | " return vocab + [f'xxfake' for i in range(0, 8-len(vocab)%8)]\n", 351 | "\n", 352 | "class NumericalizeNLP(Transform):\n", 353 | " \"Reversible transform of tokenized texts to numericalized ids\"\n", 354 | " def __init__(self, dsets=None, vocab=None, min_freq=3, max_vocab=60000, special_toks=None, pad_tok=None):\n", 355 | " store_attr(self, 'vocab,min_freq,max_vocab,special_toks,pad_tok')\n", 356 | " self.vocab, self.special_toks, self.min_freq, self.max_vocab = vocab, special_toks, min_freq, max_vocab\n", 357 | " self.o2i = None if vocab is None else defaultdict(int, {v:k for k,v in enumerate(vocab)})\n", 358 | "\n", 359 | " if self.vocab is None:\n", 360 | " count = Counter(p for o in dsets for p in o)\n", 361 | " self.vocab = make_vocab(count, min_freq=self.min_freq, max_vocab=self.max_vocab, special_toks=self.special_toks)\n", 362 | " self.o2i = defaultdict(int, {v:k for k,v in enumerate(self.vocab) if v != 'xxfake'})\n", 363 | " \n", 364 | " def encodes_nlp(self, o): return TensorText(tensor([self.o2i [o_] for o_ in o]))\n", 365 | " def encodes_nlp(self, b): return {'toks' : [[self.o2i[o_] for o_ in oo] for oo in b['tok_text']]}\n", 366 | " \n", 367 | "# Padding functions\n", 368 | "def pad_seq(x, max_batch_len, pad_idx): \n", 369 | " pad = x.new_zeros(max_batch_len-x.size(0))+pad_idx\n", 370 | " return torch.cat([x, pad])\n", 371 | " \n", 372 | "# Pad up to longest item in the batch and put batch on the GPU\n", 373 | "def pad_batch(batch=None, pad_token_id=1):\n", 374 | " batch_inputs = list()\n", 375 | " max_size = max([len(item['toks']) for item in batch])\n", 376 | " for item in batch: batch_inputs += [pad_seq(item['toks'], max_size, pad_token_id)]\n", 377 | " return torch.stack(batch_inputs).cuda()" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "metadata": {}, 383 | "source": [ 384 | "Data loading and processing functions:" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": 9, 390 | "metadata": { 391 | "scrolled": true 392 | }, 393 | "outputs": [ 394 | { 395 | "name": "stdout", 396 | "output_type": "stream", 397 | "text": [ 398 | "Downloading and preparing dataset sentiment140/sentiment140 (download: 77.59 MiB, generated: 214.21 MiB, total: 291.81 MiB) to /home/morgan/.cache/huggingface/datasets/sentiment140/sentiment140/1.0.0...\n" 399 | ] 400 | }, 401 | { 402 | "data": { 403 | "application/vnd.jupyter.widget-view+json": { 404 | "model_id": "", 405 | "version_major": 2, 406 | "version_minor": 0 407 | }, 408 | "text/plain": [ 409 | "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))" 410 | ] 411 | }, 412 | "metadata": {}, 413 | "output_type": "display_data" 414 | }, 415 | { 416 | "name": "stdout", 417 | "output_type": "stream", 418 | "text": [ 419 | "\r" 420 | ] 421 | }, 422 | { 423 | "data": { 424 | "application/vnd.jupyter.widget-view+json": { 425 | "model_id": "", 426 | "version_major": 2, 427 | "version_minor": 0 428 | }, 429 | "text/plain": [ 430 | "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))" 431 | ] 432 | }, 433 | "metadata": {}, 434 | "output_type": "display_data" 435 | }, 436 | { 437 | "name": "stdout", 438 | "output_type": "stream", 439 | "text": [ 440 | "Dataset sentiment140 downloaded and prepared to /home/morgan/.cache/huggingface/datasets/sentiment140/sentiment140/1.0.0. Subsequent calls will reuse this data.\n" 441 | ] 442 | } 443 | ], 444 | "source": [ 445 | "#hide_collapse\n", 446 | "\n", 447 | "# Download text, a clean version of the dataset is downloaded (not included in the timings), 'train[:10%]'\n", 448 | "senti_dataset = load_dataset('sentiment140', split='train', download_mode='reuse_cache_if_exists')\n", 449 | "\n", 450 | "spacy_tok = SpacyTokenizerNLP(lang='en', special_toks=defaults.text_spec_tok)\n", 451 | "\n", 452 | "def preproc_and_tok(b): return spacy_tok.encodes(list(maps(*defaults.text_proc_rules, b['text'])))\n", 453 | "\n", 454 | "def postproc(b): \n", 455 | " return {'tok_text': [list(maps(*defaults.text_postproc_rules, _b)) for _b in b['tok_text_pre']]}\n", 456 | "\n", 457 | "def get_tok_lengths(example_batch): return {'tok_lens': [len(e) for e in example_batch['toks']]}\n", 458 | "\n", 459 | "def prepare_dataset(dataset):\n", 460 | " '''\n", 461 | " Takes a raw nlp dataset and returns a processed, tokenized, numericalised dataset\n", 462 | " '''\n", 463 | " # Apply processing rules and tokenize\n", 464 | " print('pre-proc and tokenize')\n", 465 | " dataset = dataset.map(preproc_and_tok, batched=True)\n", 466 | "\n", 467 | " # Apply post-processing rules \n", 468 | " print('post=proc')\n", 469 | " dataset = dataset.map(postproc, batched=True)\n", 470 | "\n", 471 | " # Init Numericalizer and create vocab\n", 472 | " print('init numericalizer')\n", 473 | " numeric = NumericalizeNLP(dsets=dataset['tok_text_pre'], special_toks=defaults.text_spec_tok, pad_tok=1)\n", 474 | "\n", 475 | " # Numericalize\n", 476 | " print('numericalizing')\n", 477 | " dataset = dataset.map(numeric.encodes_nlp, batched=True)\n", 478 | "\n", 479 | " # Get sample lengths for sorting\n", 480 | " dataset=dataset.map(get_tok_lengths, batched=True)\n", 481 | " \n", 482 | " print('sorting')\n", 483 | " # Sort dataset from small to large\n", 484 | " dataset = dataset.sort('tok_lens')\n", 485 | " \n", 486 | " return dataset" 487 | ] 488 | }, 489 | { 490 | "cell_type": "markdown", 491 | "metadata": {}, 492 | "source": [ 493 | "Init Timing" 494 | ] 495 | }, 496 | { 497 | "cell_type": "code", 498 | "execution_count": 10, 499 | "metadata": { 500 | "scrolled": true 501 | }, 502 | "outputs": [ 503 | { 504 | "name": "stderr", 505 | "output_type": "stream", 506 | "text": [ 507 | "\r", 508 | " 0%| | 0/1600 [00:00