├── .gitattributes ├── LICENSE.MD ├── README.md ├── ds_config.json ├── ds_config_gptneo.json ├── requirements.txt ├── run_clm.py ├── run_generate_batches.py ├── run_generate_neo.py ├── run_generation.py ├── text2csv.py ├── train.csv ├── train.txt ├── validation.csv └── validation.txt /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /LICENSE.MD: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Xirider 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Guide: Finetune GPT2-XL (1.5 Billion Parameters) and GPT-NEO (2.7 Billion Parameters) on a single GPU with Huggingface Transformers using DeepSpeed 2 | 3 | 4 | 5 | - Finetuning large language models like GPT2-xl is often difficult, as these models are too big to fit on a single GPU. 6 | - This guide explains how to finetune GPT2-xl and GPT-NEO (2.7B Parameters) with just one command of the Huggingface Transformers library on a single GPU. 7 | - This is made possible by using the DeepSpeed library and gradient checkpointing to lower the required GPU memory usage of the model. 8 | - I also explain how to set up a server on Google Cloud with a V100 GPU (16GB VRAM), that you can use if you don't have a GPU with enough VRAM (16+ GB) or you don't have enough enough normal RAM (60 GB+). 9 | 10 | 11 | 12 | ## 1. (Optional) Setup VM with V100 in Google Compute Engine 13 | 14 | Note: The GPT2-xl model does run on any server with a GPU with at least 16 GB VRAM and 60 GB RAM. The GPT-NEO model needs at least 70 GB RAM. If you use your own server and not the setup described here, you will need to install CUDA and Pytorch on it. 15 | 16 | ### Requirements 17 | 18 | 1. Install the Google Cloud SDK: [Click Here](https://cloud.google.com/sdk/docs/install) 19 | 2. Register a Google Cloud Account, create a project and set up billing (only once you set up billing, you can use the $300 dollar sign up credit for GPUs). 20 | 3. Request a quota limit increase for "GPU All Regions" to 1. [Here](https://nirbenz.github.io/gce-quota/) is a step by step guide. The UI changed a bit and looks now like [this](https://stackoverflow.com/a/62883281/15447124). 21 | 4. Log in and initialize the cloud sdk with `gcloud auth login` and `gcloud init` and follow the steps until you are set up. 22 | 23 | ### Create VM 24 | 25 | - Replace YOURPROJECTID in the command below with the project id from your GCE project. 26 | - You can remove the `--preemptible` flag from the command below, but keeping it reduces your cost to about 1/3 and allows Google to shut down your instance at any point. At the time of writing, this configuration only costs about $1.28 / hour in GCE, when using preemptible. Depending on the size of your dataset, finetuning usually only takes a few hours. 27 | - You can change the zone, if there are no ressources available. [Here](https://cloud.google.com/compute/docs/gpus/gpu-regions-zones) is a list of all zones and whether they have V100 GPUs. Depending on the time of the day you might need to try out a few. Usually there are also more server available if you keep the `--preemptible` flag 28 | - We need a GPU server with at least 60 GB RAM, otherwise the run will crash, whenever the script wants to save/pickle a model. This setup below gives us as much RAM as possible with 12 CPU cores in GCE (without paying for extended memory). You also can't use more than 12 CPU cores with a single V100 GPU in GCE. 29 | 30 | Run this to create the instance: 31 | 32 | ```markdown 33 | gcloud compute instances create gpuserver \ 34 | --project YOURPROJECTID \ 35 | --zone us-west1-b \ 36 | --custom-cpu 12 \ 37 | --custom-memory 78 \ 38 | --maintenance-policy TERMINATE \ 39 | --image-family pytorch-1-7-cu110 \ 40 | --image-project deeplearning-platform-release \ 41 | --boot-disk-size 200GB \ 42 | --metadata "install-nvidia-driver=True" \ 43 | --accelerator="type=nvidia-tesla-v100,count=1" \ 44 | --preemptible 45 | ``` 46 | 47 | After 5 minutes or so (the server needs to install nvidia drivers first), you can connect to your instance with the command below. If you changed the zone, you also will need to change it here. 48 | - replace YOURSDKACCOUNT with your sdk account name 49 | 50 | ```markdown 51 | gcloud compute ssh YOURSDKACCOUNT@gpuserver --zone=us-west1-b 52 | ``` 53 | 54 | Don't forget to shut down the server once your done, otherwise you will keep getting billed for it. This can be done [here](https://console.cloud.google.com/compute/instance). 55 | 56 | The next time you can restart the server from the same web ui [here](https://console.cloud.google.com/compute/instance). 57 | 58 | ## 2. Download script and install libraries 59 | 60 | Run this to download the script and to install all libraries: 61 | 62 | ```markdown 63 | git clone https://github.com/Xirider/finetune-gpt2xl.git 64 | chmod -R 777 finetune-gpt2xl/ 65 | cd finetune-gpt2xl 66 | pip install -r requirements.txt 67 | 68 | ``` 69 | 70 | (Optional) If you want to use [Wandb.ai](http://wandb.ai) for experiment tracking, you have to login: 71 | 72 | ```markdown 73 | wandb login 74 | ``` 75 | 76 | ## 3. Finetune GPT2-xl (1.5 Billion Parameters) 77 | 78 | Then add your training data: 79 | - replace the example train.txt and validation.txt files in the folder with your own training data with the same names and then run `python text2csv.py`. This converts your .txt files into one column csv files with a "text" header and puts all the text into a single line. We need to use .csv files instead of .txt files, because Huggingface's dataloader removes line breaks when loading text from a .txt file, which does not happen with the .csv files. 80 | - If you want to feed the model separate examples instead of one continuous block of text, you need to pack each of your examples into an separate line in the csv train and validation files. 81 | - Be careful with the encoding of your text. If you don't clean your text files or if you just copy text from the web into a text editor, the dataloader from the datasets library might not load them. 82 | 83 | Run this: 84 | 85 | ```markdown 86 | deepspeed --num_gpus=1 run_clm.py \ 87 | --deepspeed ds_config.json \ 88 | --model_name_or_path gpt2-xl \ 89 | --train_file train.csv \ 90 | --validation_file validation.csv \ 91 | --do_train \ 92 | --do_eval \ 93 | --fp16 \ 94 | --overwrite_cache \ 95 | --evaluation_strategy="steps" \ 96 | --output_dir finetuned \ 97 | --eval_steps 200 \ 98 | --num_train_epochs 1 \ 99 | --gradient_accumulation_steps 2 \ 100 | --per_device_train_batch_size 8 101 | 102 | ``` 103 | 104 | - This command runs the the standard run_clm.py file from Huggingface's examples with deepspeed, just with 2 lines added to enable gradient checkpointing to use less memory. 105 | - Training on the Shakespeare example should take about 17 minutes. With gradient accumulation 2 and batch size 8, one gradient step takes about 9 seconds. This means the model training speed should be almost 2 examples / second. You can go up to batch size of 12 before running out of memory, but that doesn't provide any speedups. 106 | - Note that the default huggingface optimizer hyperparameters and the hyperparameters given as flag overwrite the hyperparameters in the ds_config.json file. Therefore if you want to adjust learning rates, warmup and more, you need to set these as flags to the training command. For an example you can find further below the training command of GPT-NEO which changes the learning rate. 107 | - You might want to try different hyperparameters like `--learning_rate` and `--warmup_steps` to improve the finetuning. 108 | 109 | 110 | 111 | ## 4. Generate text with your finetuned model 112 | 113 | You can test your finetuned GPT2-xl model with this script from Huggingface Transfomers (is included in the folder): 114 | 115 | ```markdown 116 | python run_generation.py --model_type=gpt2 --model_name_or_path=finetuned --length 200 117 | ``` 118 | 119 | Or you can use it now in your own code like this to generate text in batches: 120 | 121 | ```python 122 | # credit to Niels Rogge - https://github.com/huggingface/transformers/issues/10704 123 | 124 | from transformers import GPT2Tokenizer, GPT2LMHeadModel 125 | import torch 126 | 127 | device = 'cuda' if torch.cuda.is_available() else 'cpu' 128 | 129 | tokenizer = GPT2Tokenizer.from_pretrained('finetuned') 130 | tokenizer.padding_side = "left" 131 | tokenizer.pad_token = tokenizer.eos_token 132 | model = GPT2LMHeadModel.from_pretrained('finetuned').to(device) 133 | print("model loaded") 134 | 135 | # this is a single input batch with size 3 136 | texts = ["From off a hill whose concave womb", "Another try", "A third test"] 137 | 138 | encoding = tokenizer(texts, padding=True, return_tensors='pt').to(device) 139 | with torch.no_grad(): 140 | generated_ids = model.generate(**encoding, max_length=100) 141 | generated_texts = tokenizer.batch_decode( 142 | generated_ids, skip_special_tokens=True) 143 | 144 | print(generated_texts) 145 | ``` 146 | 147 | - model inference runs on even small gpus or on cpus without any additional changes 148 | 149 | ## Finetune GPT-NEO (2.7 Billion Parameters) 150 | 151 | This works now. I tested it with a server with one V100 GPU (16 GB VRAM) and 78 GB normal RAM, but it might not actually need that much RAM. 152 | 153 | Add your training data like you would for GPT2-xl: 154 | - replace the example train.txt and validation.txt files in the folder with your own training data with the same names and then run `python text2csv.py`. This converts your .txt files into one column csv files with a "text" header and puts all the text into a single line. We need to use .csv files instead of .txt files, because Huggingface's dataloader removes line breaks when loading text from a .txt file, which does not happen with the .csv files. 155 | - If you want to feed the model separate examples instead of one continuous block of text, you need to modify the function `group_texts` in `run_clm.py` . 156 | - Be careful with the encoding of your text. If you don't clean your text files or if you just copy text from the web into a text editor, the dataloader from the datasets library might not load them. 157 | 158 | - Be sure to either login into wandb.ai with `wandb login` or uninstall it completely. Otherwise it might cause a memory error during the run. 159 | 160 | Then start the training run this command: 161 | 162 | ```markdown 163 | deepspeed --num_gpus=1 run_clm.py \ 164 | --deepspeed ds_config_gptneo.json \ 165 | --model_name_or_path EleutherAI/gpt-neo-2.7B \ 166 | --train_file train.csv \ 167 | --validation_file validation.csv \ 168 | --do_train \ 169 | --do_eval \ 170 | --fp16 \ 171 | --overwrite_cache \ 172 | --evaluation_strategy="steps" \ 173 | --output_dir finetuned \ 174 | --num_train_epochs 1 \ 175 | --eval_steps 15 \ 176 | --gradient_accumulation_steps 2 \ 177 | --per_device_train_batch_size 4 \ 178 | --use_fast_tokenizer False \ 179 | --learning_rate 5e-06 \ 180 | --warmup_steps 10 181 | ``` 182 | 183 | - This uses a smaller "allgather_bucket_size" setting in the ds_config_gptneo.json file and a smaller batch size to further reduce gpu memory. 184 | - You might want to change and try hyperparameters to be closer to the orignal EleutherAi training config. You can find these [here](https://github.com/EleutherAI/gpt-neo/blob/master/configs/gpt3_2-7B_256.json). 185 | - If you want to try train on a GPU with less VRAM or your machine doesn't have 70 GB RAM, you could try to set `--per_device_train_batch_size` to 1 and `--gradient_accumulation_steps` to 8. You can also then try to reduce the values for "allgather_bucket_size" and "reduce_bucket_size" in the ds_config_gptneo.json file to 5e7. 186 | 187 | ## Generate text with a GPT-NEO 2.7 Billion Parameters model 188 | 189 | I provided a script, that allows you to interactively prompt your GPT-NEO model. If you just want to sample from the pretrained model without finetuning it yourself, replace "finetuned" with "EleutherAI/gpt-neo-2.7B". Start it with this: 190 | 191 | ```markdown 192 | python run_generate_neo.py finetuned 193 | ``` 194 | 195 | Or use this snippet to generate text from your finetuned model within your code: 196 | 197 | ```python 198 | # credit to Suraj Patil - https://github.com/huggingface/transformers/pull/10848 - modified to create multiple texts and use deepspeed inference 199 | 200 | from transformers import GPTNeoForCausalLM, AutoTokenizer 201 | import deepspeed 202 | 203 | # casting to fp16 "half" gives a large speedup during model loading 204 | model = GPTNeoForCausalLM.from_pretrained("finetuned").half().to("cuda") 205 | tokenizer = AutoTokenizer.from_pretrained("finetuned") 206 | 207 | # using deepspeed inference is optional: it gives about a 2x speed up 208 | deepspeed.init_inference(model, mp_size=1, dtype=torch.half, replace_method='auto') 209 | 210 | texts = ["From off a hill whose concave", "Paralell text 2"] 211 | 212 | ids = tokenizer(texts, padding=padding, return_tensors="pt").input_ids.to("cuda") 213 | 214 | 215 | gen_tokens = model.generate( 216 | ids, 217 | do_sample=True, 218 | min_length=0, 219 | max_length=200, 220 | temperature=1.0, 221 | top_p=0.8, 222 | use_cache=True 223 | ) 224 | gen_text = tokenizer.batch_decode(gen_tokens) 225 | print(gen_text) 226 | 227 | ``` 228 | 229 | ## (Optional) Configuration 230 | 231 | You can change the learning rate, weight decay and warmup by setting them as flags to the training command. Warm up and learning rates in the config are ignored, as the script always uses the Huggingface optimizer/trainer default values. If you want to overwrite them you need to use flags. You can check all the explanations here: 232 | 233 | [https://huggingface.co/transformers/master/main_classes/trainer.html#deepspeed](https://huggingface.co/transformers/master/main_classes/trainer.html#deepspeed) 234 | 235 | The rest of the training arguments can be provided as a flags and are all listed here: 236 | 237 | [https://huggingface.co/transformers/master/main_classes/trainer.html#trainingarguments](https://huggingface.co/transformers/master/main_classes/trainer.html#trainingarguments) 238 | -------------------------------------------------------------------------------- /ds_config.json: -------------------------------------------------------------------------------- 1 | { 2 | "fp16": { 3 | "enabled": "auto", 4 | "loss_scale": 0, 5 | "loss_scale_window": 1000, 6 | "initial_scale_power": 16, 7 | "hysteresis": 2, 8 | "min_loss_scale": 1 9 | }, 10 | "optimizer": { 11 | "type": "AdamW", 12 | "params": { 13 | "lr": "auto", 14 | "betas": "auto", 15 | "eps": "auto", 16 | "weight_decay": "auto" 17 | } 18 | }, 19 | "scheduler": { 20 | "type": "WarmupLR", 21 | "params": { 22 | "warmup_min_lr": "auto", 23 | "warmup_max_lr": "auto", 24 | "warmup_num_steps": "auto" 25 | } 26 | }, 27 | "zero_optimization": { 28 | "stage": 2, 29 | "allgather_partitions": true, 30 | "allgather_bucket_size": 2e8, 31 | "overlap_comm": true, 32 | "reduce_scatter": true, 33 | "reduce_bucket_size": 2e8, 34 | "contiguous_gradients": true, 35 | "cpu_offload": true 36 | }, 37 | "gradient_accumulation_steps": "auto", 38 | "gradient_clipping": "auto", 39 | "steps_per_print": 2000, 40 | "train_batch_size": "auto", 41 | "train_micro_batch_size_per_gpu": "auto", 42 | "wall_clock_breakdown": false 43 | } 44 | -------------------------------------------------------------------------------- /ds_config_gptneo.json: -------------------------------------------------------------------------------- 1 | { 2 | "fp16": { 3 | "enabled": "auto", 4 | "loss_scale": 0, 5 | "loss_scale_window": 1000, 6 | "initial_scale_power": 16, 7 | "hysteresis": 2, 8 | "min_loss_scale": 1 9 | }, 10 | "optimizer": { 11 | "type": "AdamW", 12 | "params": { 13 | "lr": "auto", 14 | "betas": "auto", 15 | "eps": "auto", 16 | "weight_decay": "auto" 17 | } 18 | }, 19 | "scheduler": { 20 | "type": "WarmupLR", 21 | "params": { 22 | "warmup_min_lr": "auto", 23 | "warmup_max_lr": "auto", 24 | "warmup_num_steps": "auto" 25 | } 26 | }, 27 | "zero_optimization": { 28 | "stage": 2, 29 | "allgather_partitions": true, 30 | "allgather_bucket_size": 2e8, 31 | "overlap_comm": true, 32 | "reduce_scatter": true, 33 | "reduce_bucket_size": 2e8, 34 | "contiguous_gradients": true, 35 | "cpu_offload": true 36 | }, 37 | "gradient_accumulation_steps": "auto", 38 | "gradient_clipping": "auto", 39 | "steps_per_print": 2000, 40 | "train_batch_size": "auto", 41 | "train_micro_batch_size_per_gpu": "auto", 42 | "wall_clock_breakdown": false 43 | } 44 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | transformers==4.7.0 2 | datasets==1.8.0 3 | wandb 4 | deepspeed 5 | -------------------------------------------------------------------------------- /run_clm.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | # Copyright 2020 The HuggingFace Inc. team. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """ 17 | Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset. 18 | Here is the full list of checkpoints on the hub that can be fine-tuned by this script: 19 | https://huggingface.co/models?filter=causal-lm 20 | """ 21 | # You can also adapt this script on your own causal language modeling task. Pointers for this are left as comments. 22 | 23 | import logging 24 | import math 25 | import os 26 | import sys 27 | from dataclasses import dataclass, field 28 | from typing import Optional 29 | from pathlib import Path 30 | 31 | from datasets import load_dataset, Dataset 32 | 33 | import transformers 34 | from transformers import ( 35 | CONFIG_MAPPING, 36 | MODEL_FOR_CAUSAL_LM_MAPPING, 37 | AutoConfig, 38 | AutoModelForCausalLM, 39 | AutoTokenizer, 40 | HfArgumentParser, 41 | Trainer, 42 | TrainingArguments, 43 | default_data_collator, 44 | set_seed, 45 | ) 46 | from transformers.trainer_utils import get_last_checkpoint, is_main_process 47 | from transformers.utils import check_min_version 48 | 49 | 50 | # Will error if the minimal version of Transformers is not installed. Remove at your own risks. 51 | check_min_version("4.5.0.dev0") 52 | 53 | logger = logging.getLogger(__name__) 54 | 55 | 56 | MODEL_CONFIG_CLASSES = list(MODEL_FOR_CAUSAL_LM_MAPPING.keys()) 57 | MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES) 58 | 59 | 60 | @dataclass 61 | class ModelArguments: 62 | """ 63 | Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch. 64 | """ 65 | 66 | model_name_or_path: Optional[str] = field( 67 | default=None, 68 | metadata={ 69 | "help": "The model checkpoint for weights initialization." 70 | "Don't set if you want to train a model from scratch." 71 | }, 72 | ) 73 | model_type: Optional[str] = field( 74 | default=None, 75 | metadata={ 76 | "help": "If training from scratch, pass a model type from the list: " + ", ".join(MODEL_TYPES)}, 77 | ) 78 | config_name: Optional[str] = field( 79 | default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"} 80 | ) 81 | tokenizer_name: Optional[str] = field( 82 | default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"} 83 | ) 84 | cache_dir: Optional[str] = field( 85 | default=None, 86 | metadata={ 87 | "help": "Where do you want to store the pretrained models downloaded from huggingface.co"}, 88 | ) 89 | use_fast_tokenizer: bool = field( 90 | default=True, 91 | metadata={ 92 | "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."}, 93 | ) 94 | model_revision: str = field( 95 | default="main", 96 | metadata={ 97 | "help": "The specific model version to use (can be a branch name, tag name or commit id)."}, 98 | ) 99 | use_auth_token: bool = field( 100 | default=False, 101 | metadata={ 102 | "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script " 103 | "with private models)." 104 | }, 105 | ) 106 | 107 | 108 | @dataclass 109 | class DataTrainingArguments: 110 | """ 111 | Arguments pertaining to what data we are going to input our model for training and eval. 112 | """ 113 | 114 | dataset_name: Optional[str] = field( 115 | default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."} 116 | ) 117 | dataset_config_name: Optional[str] = field( 118 | default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."} 119 | ) 120 | train_file: Optional[str] = field( 121 | default=None, metadata={"help": "The input training data file (a text file)."}) 122 | validation_file: Optional[str] = field( 123 | default=None, 124 | metadata={ 125 | "help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."}, 126 | ) 127 | max_train_samples: Optional[int] = field( 128 | default=None, 129 | metadata={ 130 | "help": "For debugging purposes or quicker training, truncate the number of training examples to this " 131 | "value if set." 132 | }, 133 | ) 134 | max_val_samples: Optional[int] = field( 135 | default=None, 136 | metadata={ 137 | "help": "For debugging purposes or quicker training, truncate the number of validation examples to this " 138 | "value if set." 139 | }, 140 | ) 141 | 142 | block_size: Optional[int] = field( 143 | default=None, 144 | metadata={ 145 | "help": "Optional input sequence length after tokenization." 146 | "The training dataset will be truncated in block of this size for training." 147 | "Default to the model max input length for single sentence inputs (take into account special tokens)." 148 | }, 149 | ) 150 | overwrite_cache: bool = field( 151 | default=False, metadata={"help": "Overwrite the cached training and evaluation sets"} 152 | ) 153 | validation_split_percentage: Optional[int] = field( 154 | default=5, 155 | metadata={ 156 | "help": "The percentage of the train set used as validation set in case there's no validation split" 157 | }, 158 | ) 159 | preprocessing_num_workers: Optional[int] = field( 160 | default=None, 161 | metadata={"help": "The number of processes to use for the preprocessing."}, 162 | ) 163 | 164 | def __post_init__(self): 165 | if self.dataset_name is None and self.train_file is None and self.validation_file is None: 166 | raise ValueError( 167 | "Need either a dataset name or a training/validation file.") 168 | else: 169 | if self.train_file is not None: 170 | extension = self.train_file.split(".")[-1] 171 | assert extension in [ 172 | "csv", "json", "txt"], "`train_file` should be a csv, a json or a txt file." 173 | if self.validation_file is not None: 174 | extension = self.validation_file.split(".")[-1] 175 | assert extension in [ 176 | "csv", "json", "txt"], "`validation_file` should be a csv, a json or a txt file." 177 | 178 | 179 | def main(): 180 | # See all possible arguments in src/transformers/training_args.py 181 | # or by passing the --help flag to this script. 182 | # We now keep distinct sets of args, for a cleaner separation of concerns. 183 | 184 | parser = HfArgumentParser( 185 | (ModelArguments, DataTrainingArguments, TrainingArguments)) 186 | if len(sys.argv) == 2 and sys.argv[1].endswith(".json"): 187 | # If we pass only one argument to the script and it's the path to a json file, 188 | # let's parse it to get our arguments. 189 | model_args, data_args, training_args = parser.parse_json_file( 190 | json_file=os.path.abspath(sys.argv[1])) 191 | else: 192 | model_args, data_args, training_args = parser.parse_args_into_dataclasses() 193 | 194 | # Detecting last checkpoint. 195 | last_checkpoint = None 196 | if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir: 197 | last_checkpoint = get_last_checkpoint(training_args.output_dir) 198 | if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0: 199 | raise ValueError( 200 | f"Output directory ({training_args.output_dir}) already exists and is not empty. " 201 | "Use --overwrite_output_dir to overcome." 202 | ) 203 | elif last_checkpoint is not None: 204 | logger.info( 205 | f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change " 206 | "the `--output_dir` or add `--overwrite_output_dir` to train from scratch." 207 | ) 208 | 209 | # Setup logging 210 | logging.basicConfig( 211 | format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", 212 | datefmt="%m/%d/%Y %H:%M:%S", 213 | handlers=[logging.StreamHandler(sys.stdout)], 214 | ) 215 | logger.setLevel(logging.INFO if is_main_process( 216 | training_args.local_rank) else logging.WARN) 217 | 218 | # Log on each process the small summary: 219 | logger.warning( 220 | f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" 221 | + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" 222 | ) 223 | # Set the verbosity to info of the Transformers logger (on main process only): 224 | if is_main_process(training_args.local_rank): 225 | transformers.utils.logging.set_verbosity_info() 226 | transformers.utils.logging.enable_default_handler() 227 | transformers.utils.logging.enable_explicit_format() 228 | logger.info("Training/evaluation parameters %s", training_args) 229 | 230 | # Set seed before initializing model. 231 | set_seed(training_args.seed) 232 | 233 | # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below) 234 | # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/ 235 | # (the dataset will be downloaded automatically from the datasets Hub). 236 | # 237 | # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called 238 | # 'text' is found. You can easily tweak this behavior (see below). 239 | # 240 | # In distributed training, the load_dataset function guarantee that only one local process can concurrently 241 | # download the dataset. 242 | if data_args.dataset_name is not None: 243 | # Downloading and loading a dataset from the hub. 244 | datasets = load_dataset(data_args.dataset_name, 245 | data_args.dataset_config_name) 246 | if "validation" not in datasets.keys(): 247 | datasets["validation"] = load_dataset( 248 | data_args.dataset_name, 249 | data_args.dataset_config_name, 250 | split=f"train[:{data_args.validation_split_percentage}%]", 251 | ) 252 | datasets["train"] = load_dataset( 253 | data_args.dataset_name, 254 | data_args.dataset_config_name, 255 | split=f"train[{data_args.validation_split_percentage}%:]", 256 | ) 257 | else: 258 | data_files = {} 259 | if data_args.train_file is not None: 260 | data_files["train"] = data_args.train_file 261 | if data_args.validation_file is not None: 262 | data_files["validation"] = data_args.validation_file 263 | extension = ( 264 | data_args.train_file.split(".")[-1] 265 | if data_args.train_file is not None 266 | else data_args.validation_file.split(".")[-1] 267 | ) 268 | if extension == "txt": 269 | extension = "text" 270 | 271 | datasets = load_dataset( 272 | extension, data_files=data_files) 273 | # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at 274 | # https://huggingface.co/docs/datasets/loading_datasets.html. 275 | 276 | # Load pretrained model and tokenizer 277 | # 278 | # Distributed training: 279 | # The .from_pretrained methods guarantee that only one local process can concurrently 280 | # download model & vocab. 281 | 282 | config_kwargs = { 283 | "cache_dir": model_args.cache_dir, 284 | "revision": model_args.model_revision, 285 | "use_auth_token": True if model_args.use_auth_token else None, 286 | } 287 | if model_args.config_name: 288 | config = AutoConfig.from_pretrained( 289 | model_args.config_name, **config_kwargs) 290 | elif model_args.model_name_or_path: 291 | config = AutoConfig.from_pretrained( 292 | model_args.model_name_or_path, **config_kwargs) 293 | else: 294 | config = CONFIG_MAPPING[model_args.model_type]() 295 | logger.warning( 296 | "You are instantiating a new config instance from scratch.") 297 | 298 | # Things that were changed from the huggingface file 299 | 300 | config.gradient_checkpointing = True 301 | config.use_cache = False 302 | 303 | # 304 | 305 | tokenizer_kwargs = { 306 | "cache_dir": model_args.cache_dir, 307 | "use_fast": model_args.use_fast_tokenizer, 308 | "revision": model_args.model_revision, 309 | "use_auth_token": True if model_args.use_auth_token else None, 310 | } 311 | if model_args.tokenizer_name: 312 | tokenizer = AutoTokenizer.from_pretrained( 313 | model_args.tokenizer_name, **tokenizer_kwargs) 314 | elif model_args.model_name_or_path: 315 | tokenizer = AutoTokenizer.from_pretrained( 316 | model_args.model_name_or_path, **tokenizer_kwargs) 317 | else: 318 | raise ValueError( 319 | "You are instantiating a new tokenizer from scratch. This is not supported by this script." 320 | "You can do it from another script, save it, and load it from here, using --tokenizer_name." 321 | ) 322 | 323 | if model_args.model_name_or_path: 324 | model = AutoModelForCausalLM.from_pretrained( 325 | model_args.model_name_or_path, 326 | from_tf=bool(".ckpt" in model_args.model_name_or_path), 327 | config=config, 328 | cache_dir=model_args.cache_dir, 329 | revision=model_args.model_revision, 330 | use_auth_token=True if model_args.use_auth_token else None, 331 | ) 332 | else: 333 | logger.info("Training new model from scratch") 334 | model = AutoModelForCausalLM.from_config(config) 335 | 336 | model.resize_token_embeddings(len(tokenizer)) 337 | 338 | # Preprocessing the datasets. 339 | # First we tokenize all the texts. 340 | if training_args.do_train: 341 | column_names = datasets["train"].column_names 342 | else: 343 | column_names = datasets["validation"].column_names 344 | text_column_name = "text" if "text" in column_names else column_names[0] 345 | 346 | def tokenize_function(examples): 347 | return tokenizer(examples[text_column_name]) 348 | 349 | tokenized_datasets = datasets.map( 350 | tokenize_function, 351 | batched=True, 352 | num_proc=data_args.preprocessing_num_workers, 353 | remove_columns=column_names, 354 | load_from_cache_file=not data_args.overwrite_cache, 355 | ) 356 | 357 | if data_args.block_size is None: 358 | block_size = tokenizer.model_max_length 359 | if block_size > 1024: 360 | logger.warn( 361 | f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). " 362 | "Picking 1024 instead. You can change that default value by passing --block_size xxx." 363 | ) 364 | block_size = 1024 365 | else: 366 | if data_args.block_size > tokenizer.model_max_length: 367 | logger.warn( 368 | f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model" 369 | f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}." 370 | ) 371 | block_size = min(data_args.block_size, tokenizer.model_max_length) 372 | 373 | # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size. 374 | def group_texts(examples): 375 | # Concatenate all texts. 376 | concatenated_examples = { 377 | k: sum(examples[k], []) for k in examples.keys()} 378 | total_length = len(concatenated_examples[list(examples.keys())[0]]) 379 | # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can 380 | # customize this part to your needs. 381 | total_length = (total_length // block_size) * block_size 382 | # Split by chunks of max_len. 383 | result = { 384 | k: [t[i: i + block_size] 385 | for i in range(0, total_length, block_size)] 386 | for k, t in concatenated_examples.items() 387 | } 388 | result["labels"] = result["input_ids"].copy() 389 | return result 390 | 391 | # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder 392 | # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower 393 | # to preprocess. 394 | # 395 | # To speed up this part, we use multiprocessing. See the documentation of the map method for more information: 396 | # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map 397 | 398 | lm_datasets = tokenized_datasets.map( 399 | group_texts, 400 | batched=True, 401 | num_proc=data_args.preprocessing_num_workers, 402 | load_from_cache_file=not data_args.overwrite_cache, 403 | ) 404 | 405 | if training_args.do_train: 406 | if "train" not in tokenized_datasets: 407 | raise ValueError("--do_train requires a train dataset") 408 | train_dataset = lm_datasets["train"] 409 | if data_args.max_train_samples is not None: 410 | train_dataset = train_dataset.select( 411 | range(data_args.max_train_samples)) 412 | 413 | if training_args.do_eval: 414 | if "validation" not in tokenized_datasets: 415 | raise ValueError("--do_eval requires a validation dataset") 416 | eval_dataset = lm_datasets["validation"] 417 | if data_args.max_val_samples is not None: 418 | eval_dataset = eval_dataset.select( 419 | range(data_args.max_val_samples)) 420 | 421 | # Initialize our Trainer 422 | trainer = Trainer( 423 | model=model, 424 | args=training_args, 425 | train_dataset=train_dataset if training_args.do_train else None, 426 | eval_dataset=eval_dataset if training_args.do_eval else None, 427 | tokenizer=tokenizer, 428 | # Data collator will default to DataCollatorWithPadding, so we change it. 429 | data_collator=default_data_collator, 430 | 431 | ) 432 | 433 | # Training 434 | if training_args.do_train: 435 | if last_checkpoint is not None: 436 | checkpoint = last_checkpoint 437 | elif model_args.model_name_or_path is not None and os.path.isdir(model_args.model_name_or_path): 438 | checkpoint = model_args.model_name_or_path 439 | else: 440 | checkpoint = None 441 | train_result = trainer.train(resume_from_checkpoint=checkpoint) 442 | trainer.save_model() # Saves the tokenizer too for easy upload 443 | 444 | metrics = train_result.metrics 445 | 446 | max_train_samples = ( 447 | data_args.max_train_samples if data_args.max_train_samples is not None else len( 448 | train_dataset) 449 | ) 450 | metrics["train_samples"] = min(max_train_samples, len(train_dataset)) 451 | 452 | trainer.log_metrics("train", metrics) 453 | trainer.save_metrics("train", metrics) 454 | trainer.save_state() 455 | 456 | # Evaluation 457 | if training_args.do_eval: 458 | logger.info("*** Evaluate ***") 459 | 460 | metrics = trainer.evaluate() 461 | 462 | max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len( 463 | eval_dataset) 464 | metrics["eval_samples"] = min(max_val_samples, len(eval_dataset)) 465 | perplexity = math.exp(metrics["eval_loss"]) 466 | metrics["perplexity"] = perplexity 467 | 468 | trainer.log_metrics("eval", metrics) 469 | trainer.save_metrics("eval", metrics) 470 | 471 | 472 | def _mp_fn(index): 473 | # For xla_spawn (TPUs) 474 | main() 475 | 476 | 477 | if __name__ == "__main__": 478 | main() 479 | -------------------------------------------------------------------------------- /run_generate_batches.py: -------------------------------------------------------------------------------- 1 | # credit to Niels Rogge - https://github.com/huggingface/transformers/issues/10704 2 | 3 | from transformers import GPT2Tokenizer, GPT2LMHeadModel 4 | import torch 5 | 6 | device = 'cuda' if torch.cuda.is_available() else 'cpu' 7 | 8 | tokenizer = GPT2Tokenizer.from_pretrained('finetuned') 9 | tokenizer.padding_side = "left" 10 | tokenizer.pad_token = tokenizer.eos_token 11 | model = GPT2LMHeadModel.from_pretrained('finetuned').to(device) 12 | print("model loaded") 13 | 14 | # this is a single batch 15 | texts = ["Hello", "Another try", "A third test"] 16 | 17 | encoding = tokenizer(texts, padding=True, return_tensors='pt').to(device) 18 | with torch.no_grad(): 19 | generated_ids = model.generate(**encoding, max_length=100) 20 | generated_texts = tokenizer.batch_decode( 21 | generated_ids, skip_special_tokens=True) 22 | 23 | print(generated_texts) 24 | -------------------------------------------------------------------------------- /run_generate_neo.py: -------------------------------------------------------------------------------- 1 | from transformers import GPTNeoForCausalLM, AutoTokenizer 2 | import argparse 3 | 4 | parser = argparse.ArgumentParser() 5 | parser.add_argument("model") 6 | args = parser.parse_args() 7 | 8 | 9 | model = GPTNeoForCausalLM.from_pretrained(args.model).half().to("cuda") 10 | tokenizer = AutoTokenizer.from_pretrained(args.model) 11 | 12 | 13 | while True: 14 | text = input("\n\nInput text to prompt the model: ") 15 | text = str(text) 16 | if len(text) == 0: 17 | continue 18 | ids = tokenizer(text, return_tensors="pt").input_ids.to("cuda") 19 | 20 | # add the length of the prompt tokens to match with the mesh-tf generation 21 | max_length = 400 + ids.shape[1] 22 | 23 | gen_tokens = model.generate( 24 | ids, 25 | do_sample=True, 26 | min_length=max_length, 27 | max_length=max_length, 28 | temperature=0.9, 29 | use_cache=True 30 | ) 31 | gen_text = tokenizer.batch_decode(gen_tokens)[0] 32 | print("Text generated:") 33 | print(gen_text) 34 | -------------------------------------------------------------------------------- /run_generation.py: -------------------------------------------------------------------------------- 1 | 2 | #!/usr/bin/env python 3 | # coding=utf-8 4 | # Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team. 5 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | """ Conditional text generation with the auto-regressive models of the library (GPT/GPT-2/CTRL/Transformer-XL/XLNet) 19 | """ 20 | 21 | 22 | import argparse 23 | import logging 24 | 25 | import numpy as np 26 | import torch 27 | 28 | from transformers import ( 29 | CTRLLMHeadModel, 30 | CTRLTokenizer, 31 | GPT2LMHeadModel, 32 | GPT2Tokenizer, 33 | OpenAIGPTLMHeadModel, 34 | OpenAIGPTTokenizer, 35 | TransfoXLLMHeadModel, 36 | TransfoXLTokenizer, 37 | XLMTokenizer, 38 | XLMWithLMHeadModel, 39 | XLNetLMHeadModel, 40 | XLNetTokenizer, 41 | ) 42 | 43 | 44 | logging.basicConfig( 45 | format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", 46 | datefmt="%m/%d/%Y %H:%M:%S", 47 | level=logging.INFO, 48 | ) 49 | logger = logging.getLogger(__name__) 50 | 51 | MAX_LENGTH = int(10000) # Hardcoded max length to avoid infinite loop 52 | 53 | MODEL_CLASSES = { 54 | "gpt2": (GPT2LMHeadModel, GPT2Tokenizer), 55 | "ctrl": (CTRLLMHeadModel, CTRLTokenizer), 56 | "openai-gpt": (OpenAIGPTLMHeadModel, OpenAIGPTTokenizer), 57 | "xlnet": (XLNetLMHeadModel, XLNetTokenizer), 58 | "transfo-xl": (TransfoXLLMHeadModel, TransfoXLTokenizer), 59 | "xlm": (XLMWithLMHeadModel, XLMTokenizer), 60 | } 61 | 62 | # Padding text to help Transformer-XL and XLNet with short prompts as proposed by Aman Rusia 63 | # in https://github.com/rusiaaman/XLNet-gen#methodology 64 | # and https://medium.com/@amanrusia/xlnet-speaks-comparison-to-gpt-2-ea1a4e9ba39e 65 | PREFIX = """In 1991, the remains of Russian Tsar Nicholas II and his family 66 | (except for Alexei and Maria) are discovered. 67 | The voice of Nicholas's young son, Tsarevich Alexei Nikolaevich, narrates the 68 | remainder of the story. 1883 Western Siberia, 69 | a young Grigori Rasputin is asked by his father and a group of men to perform magic. 70 | Rasputin has a vision and denounces one of the men as a horse thief. Although his 71 | father initially slaps him for making such an accusation, Rasputin watches as the 72 | man is chased outside and beaten. Twenty years later, Rasputin sees a vision of 73 | the Virgin Mary, prompting him to become a priest. Rasputin quickly becomes famous, 74 | with people, even a bishop, begging for his blessing. """ 75 | 76 | 77 | def set_seed(args): 78 | np.random.seed(args.seed) 79 | torch.manual_seed(args.seed) 80 | if args.n_gpu > 0: 81 | torch.cuda.manual_seed_all(args.seed) 82 | 83 | 84 | # 85 | # Functions to prepare models' input 86 | # 87 | 88 | 89 | def prepare_ctrl_input(args, _, tokenizer, prompt_text): 90 | if args.temperature > 0.7: 91 | logger.info( 92 | "CTRL typically works better with lower temperatures (and lower top_k).") 93 | 94 | encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False) 95 | if not any(encoded_prompt[0] == x for x in tokenizer.control_codes.values()): 96 | logger.info( 97 | "WARNING! You are not starting your generation from a control code so you won't get good results") 98 | return prompt_text 99 | 100 | 101 | def prepare_xlm_input(args, model, tokenizer, prompt_text): 102 | # kwargs = {"language": None, "mask_token_id": None} 103 | 104 | # Set the language 105 | use_lang_emb = hasattr( 106 | model.config, "use_lang_emb") and model.config.use_lang_emb 107 | if hasattr(model.config, "lang2id") and use_lang_emb: 108 | available_languages = model.config.lang2id.keys() 109 | if args.xlm_language in available_languages: 110 | language = args.xlm_language 111 | else: 112 | language = None 113 | while language not in available_languages: 114 | language = input("Using XLM. Select language in " + 115 | str(list(available_languages)) + " >>> ") 116 | 117 | model.config.lang_id = model.config.lang2id[language] 118 | # kwargs["language"] = tokenizer.lang2id[language] 119 | 120 | # TODO fix mask_token_id setup when configurations will be synchronized between models and tokenizers 121 | # XLM masked-language modeling (MLM) models need masked token 122 | # is_xlm_mlm = "mlm" in args.model_name_or_path 123 | # if is_xlm_mlm: 124 | # kwargs["mask_token_id"] = tokenizer.mask_token_id 125 | 126 | return prompt_text 127 | 128 | 129 | def prepare_xlnet_input(args, _, tokenizer, prompt_text): 130 | prefix = args.prefix if args.prefix else args.padding_text if args.padding_text else PREFIX 131 | prompt_text = prefix + prompt_text 132 | return prompt_text 133 | 134 | 135 | def prepare_transfoxl_input(args, _, tokenizer, prompt_text): 136 | prefix = args.prefix if args.prefix else args.padding_text if args.padding_text else PREFIX 137 | prompt_text = prefix + prompt_text 138 | return prompt_text 139 | 140 | 141 | PREPROCESSING_FUNCTIONS = { 142 | "ctrl": prepare_ctrl_input, 143 | "xlm": prepare_xlm_input, 144 | "xlnet": prepare_xlnet_input, 145 | "transfo-xl": prepare_transfoxl_input, 146 | } 147 | 148 | 149 | def adjust_length_to_model(length, max_sequence_length): 150 | if length < 0 and max_sequence_length > 0: 151 | length = max_sequence_length 152 | elif 0 < max_sequence_length < length: 153 | length = max_sequence_length # No generation bigger than model size 154 | elif length < 0: 155 | length = MAX_LENGTH # avoid infinite loop 156 | return length 157 | 158 | 159 | def main(): 160 | parser = argparse.ArgumentParser() 161 | parser.add_argument( 162 | "--model_type", 163 | default=None, 164 | type=str, 165 | required=True, 166 | help="Model type selected in the list: " + 167 | ", ".join(MODEL_CLASSES.keys()), 168 | ) 169 | parser.add_argument( 170 | "--model_name_or_path", 171 | default=None, 172 | type=str, 173 | required=True, 174 | help="Path to pre-trained model or shortcut name selected in the list: " + 175 | ", ".join(MODEL_CLASSES.keys()), 176 | ) 177 | 178 | parser.add_argument("--prompt", type=str, default="") 179 | parser.add_argument("--length", type=int, default=20) 180 | parser.add_argument("--stop_token", type=str, default=None, 181 | help="Token at which text generation is stopped") 182 | 183 | parser.add_argument( 184 | "--temperature", 185 | type=float, 186 | default=1.0, 187 | help="temperature of 1.0 has no effect, lower tend toward greedy sampling", 188 | ) 189 | parser.add_argument( 190 | "--repetition_penalty", type=float, default=1.0, help="primarily useful for CTRL model; in that case, use 1.2" 191 | ) 192 | parser.add_argument("--k", type=int, default=0) 193 | parser.add_argument("--p", type=float, default=0.9) 194 | 195 | parser.add_argument("--prefix", type=str, default="", 196 | help="Text added prior to input.") 197 | parser.add_argument("--padding_text", type=str, default="", 198 | help="Deprecated, the use of `--prefix` is preferred.") 199 | parser.add_argument("--xlm_language", type=str, default="", 200 | help="Optional language when used with the XLM model.") 201 | 202 | parser.add_argument("--seed", type=int, default=42, 203 | help="random seed for initialization") 204 | parser.add_argument("--no_cuda", action="store_true", 205 | help="Avoid using CUDA when available") 206 | parser.add_argument("--num_return_sequences", type=int, 207 | default=1, help="The number of samples to generate.") 208 | parser.add_argument( 209 | "--fp16", 210 | action="store_true", 211 | help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit", 212 | ) 213 | args = parser.parse_args() 214 | 215 | args.device = torch.device( 216 | "cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 217 | args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count() 218 | 219 | logger.warning( 220 | "device: %s, n_gpu: %s, 16-bits training: %s", 221 | args.device, 222 | args.n_gpu, 223 | args.fp16, 224 | ) 225 | 226 | set_seed(args) 227 | 228 | # Initialize the model and tokenizer 229 | try: 230 | args.model_type = args.model_type.lower() 231 | model_class, tokenizer_class = MODEL_CLASSES[args.model_type] 232 | except KeyError: 233 | raise KeyError( 234 | "the model {} you specified is not supported. You are welcome to add it and open a PR :)") 235 | 236 | tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path) 237 | model = model_class.from_pretrained(args.model_name_or_path) 238 | model.to(args.device) 239 | 240 | if args.fp16: 241 | model.half() 242 | 243 | args.length = adjust_length_to_model( 244 | args.length, max_sequence_length=model.config.max_position_embeddings) 245 | logger.info(args) 246 | 247 | prompt_text = args.prompt if args.prompt else input("Model prompt >>> ") 248 | 249 | # Different models need different input formatting and/or extra arguments 250 | requires_preprocessing = args.model_type in PREPROCESSING_FUNCTIONS.keys() 251 | if requires_preprocessing: 252 | prepare_input = PREPROCESSING_FUNCTIONS.get(args.model_type) 253 | preprocessed_prompt_text = prepare_input( 254 | args, model, tokenizer, prompt_text) 255 | 256 | if model.__class__.__name__ in ["TransfoXLLMHeadModel"]: 257 | tokenizer_kwargs = {"add_space_before_punct_symbol": True} 258 | else: 259 | tokenizer_kwargs = {} 260 | 261 | encoded_prompt = tokenizer.encode( 262 | preprocessed_prompt_text, add_special_tokens=False, return_tensors="pt", **tokenizer_kwargs 263 | ) 264 | else: 265 | prefix = args.prefix if args.prefix else args.padding_text 266 | encoded_prompt = tokenizer.encode( 267 | prefix + prompt_text, add_special_tokens=False, return_tensors="pt") 268 | encoded_prompt = encoded_prompt.to(args.device) 269 | 270 | if encoded_prompt.size()[-1] == 0: 271 | input_ids = None 272 | else: 273 | input_ids = encoded_prompt 274 | 275 | output_sequences = model.generate( 276 | input_ids=input_ids, 277 | max_length=args.length + len(encoded_prompt[0]), 278 | temperature=args.temperature, 279 | top_k=args.k, 280 | top_p=args.p, 281 | repetition_penalty=args.repetition_penalty, 282 | do_sample=True, 283 | num_return_sequences=args.num_return_sequences, 284 | ) 285 | 286 | # Remove the batch dimension when returning multiple sequences 287 | if len(output_sequences.shape) > 2: 288 | output_sequences.squeeze_() 289 | 290 | generated_sequences = [] 291 | 292 | for generated_sequence_idx, generated_sequence in enumerate(output_sequences): 293 | print("=== GENERATED SEQUENCE {} ===".format(generated_sequence_idx + 1)) 294 | generated_sequence = generated_sequence.tolist() 295 | 296 | # Decode text 297 | text = tokenizer.decode( 298 | generated_sequence, clean_up_tokenization_spaces=True) 299 | 300 | # Remove all text after the stop token 301 | text = text[: text.find(args.stop_token) if args.stop_token else None] 302 | 303 | # Add the prompt at the beginning of the sequence. Remove the excess text that was used for pre-processing 304 | total_sequence = ( 305 | prompt_text + 306 | text[len(tokenizer.decode(encoded_prompt[0], 307 | clean_up_tokenization_spaces=True)):] 308 | ) 309 | 310 | generated_sequences.append(total_sequence) 311 | print(total_sequence) 312 | 313 | return generated_sequences 314 | 315 | 316 | if __name__ == "__main__": 317 | main() 318 | -------------------------------------------------------------------------------- /text2csv.py: -------------------------------------------------------------------------------- 1 | import csv 2 | 3 | with open('train.txt', encoding='utf-8') as txtfile: 4 | all_text = txtfile.read() 5 | with open('train.csv', mode='w', encoding='utf-8') as csv_file: 6 | fieldnames = ['text'] 7 | writer = csv.DictWriter(csv_file, fieldnames=fieldnames) 8 | writer.writeheader() 9 | writer.writerow({'text': all_text}) 10 | 11 | 12 | with open('validation.txt', encoding='utf-8') as txtfile: 13 | all_text = txtfile.read() 14 | with open('validation.csv', mode='w', encoding='utf-8') as csv_file: 15 | fieldnames = ['text'] 16 | writer = csv.DictWriter(csv_file, fieldnames=fieldnames) 17 | writer.writeheader() 18 | writer.writerow({'text': all_text}) 19 | 20 | print("created train.csv and validation.csv files") 21 | -------------------------------------------------------------------------------- /validation.csv: -------------------------------------------------------------------------------- 1 | text 2 | "1609 3 | 4 | A LOVER'S COMPLAINT 5 | 6 | by William Shakespeare 7 | 8 | 9 | 10 | From off a hill whose concave womb reworded 11 | A plaintful story from a sist'ring vale, 12 | My spirits t'attend this double voice accorded, 13 | And down I laid to list the sad-tuned tale, 14 | Ere long espied a fickle maid full pale, 15 | Tearing of papers, breaking rings atwain, 16 | Storming her world with sorrow's wind and rain. 17 | 18 | Upon her head a platted hive of straw, 19 | Which fortified her visage from the sun, 20 | Whereon the thought might think sometime it saw 21 | The carcase of a beauty spent and done. 22 | Time had not scythed all that youth begun, 23 | Nor youth all quit, but spite of heaven's fell rage 24 | Some beauty peeped through lattice of seared age. 25 | 26 | Oft did she heave her napkin to her eyne, 27 | Which on it had conceited characters, 28 | Laund'ring the silken figures in the brine 29 | That seasoned woe had pelleted in tears, 30 | And often reading what contents it bears; 31 | As often shrieking undistinguished woe, 32 | In clamours of all size, both high and low. 33 | 34 | Sometimes her levelled eyes their carriage ride, 35 | As they did batt'ry to the spheres intend; 36 | Sometime diverted their poor balls are tied 37 | To th' orbed earth; sometimes they do extend 38 | Their view right on; anon their gazes lend 39 | To every place at once, and nowhere fixed, 40 | The mind and sight distractedly commixed. 41 | 42 | Her hair, nor loose nor tied in formal plat, 43 | Proclaimed in her a careless hand of pride; 44 | For some, untucked, descended her sheaved hat, 45 | Hanging her pale and pined cheek beside; 46 | Some in her threaden fillet still did bide, 47 | And, true to bondage, would not break from thence, 48 | Though slackly braided in loose negligence. 49 | 50 | A thousand favours from a maund she drew 51 | Of amber, crystal, and of beaded jet, 52 | Which one by one she in a river threw, 53 | Upon whose weeping margent she was set; 54 | Like usury applying wet to wet, 55 | Or monarchs' hands that lets not bounty fall 56 | Where want cries some, but where excess begs all. 57 | 58 | Of folded schedules had she many a one, 59 | Which she perused, sighed, tore, and gave the flood; 60 | Cracked many a ring of posied gold and bone, 61 | Bidding them find their sepulchres in mud; 62 | Found yet moe letters sadly penned in blood, 63 | With sleided silk feat and affectedly 64 | Enswathed and sealed to curious secrecy. 65 | 66 | These often bathed she in her fluxive eyes, 67 | And often kissed, and often 'gan to tear; 68 | Cried, 'O false blood, thou register of lies, 69 | What unapproved witness dost thou bear! 70 | Ink would have seemed more black and damned here! 71 | This said, in top of rage the lines she rents, 72 | Big discontents so breaking their contents. 73 | 74 | A reverend man that grazed his cattle nigh, 75 | Sometime a blusterer that the ruffle knew 76 | Of court, of city, and had let go by 77 | The swiftest hours observed as they flew, 78 | Towards this afflicted fancy fastly drew; 79 | And, privileged by age, desires to know 80 | In brief the grounds and motives of her woe. 81 | 82 | So slides he down upon his grained bat, 83 | And comely distant sits he by her side; 84 | When he again desires her, being sat, 85 | Her grievance with his hearing to divide. 86 | If that from him there may be aught applied 87 | Which may her suffering ecstasy assuage, 88 | 'Tis promised in the charity of age. 89 | 90 | 'Father,' she says, 'though in me you behold 91 | The injury of many a blasting hour, 92 | Let it not tell your judgement I am old: 93 | Not age, but sorrow, over me hath power. 94 | I might as yet have been a spreading flower, 95 | Fresh to myself, if I had self-applied 96 | Love to myself, and to no love beside. 97 | 98 | 'But woe is me! too early I attended 99 | A youthful suit- it was to gain my grace- 100 | O, one by nature's outwards so commended 101 | That maidens' eyes stuck over all his face. 102 | Love lacked a dwelling and made him her place; 103 | And when in his fair parts she did abide, 104 | She was new lodged and newly deified. 105 | 106 | 'His browny locks did hang in crooked curls; 107 | And every light occasion of the wind 108 | Upon his lips their silken parcels hurls. 109 | What's sweet to do, to do will aptly find: 110 | Each eye that saw him did enchant the mind; 111 | For on his visage was in little drawn 112 | What largeness thinks in Paradise was sawn. 113 | 114 | 'Small show of man was yet upon his chin; 115 | His phoenix down began but to appear, 116 | Like unshorn velvet, on that termless skin, 117 | Whose bare out-bragged the web it seemed to wear: 118 | Yet showed his visage by that cost more dear; 119 | And nice affections wavering stood in doubt 120 | If best were as it was, or best without. 121 | 122 | 'His qualities were beauteous as his form, 123 | For maiden-tongued he was, and thereof free; 124 | Yet if men moved him, was he such a storm 125 | As oft 'twixt May and April is to see, 126 | When winds breathe sweet, unruly though they be. 127 | His rudeness so with his authorized youth 128 | Did livery falseness in a pride of truth. 129 | 130 | 'Well could he ride, and often men would say, 131 | ""That horse his mettle from his rider takes: 132 | Proud of subjection, noble by the sway, 133 | What rounds, what bounds, what course, what stop he makes!"" 134 | And controversy hence a question takes 135 | Whether the horse by him became his deed, 136 | Or he his manage by th' well-doing steed. 137 | 138 | 'But quickly on this side the verdict went: 139 | His real habitude gave life and grace 140 | To appertainings and to ornament, 141 | Accomplished in himself, not in his case, 142 | All aids, themselves made fairer by their place, 143 | Came for additions; yet their purposed trim 144 | Pierced not his grace, but were all graced by him. 145 | 146 | 'So on the tip of his subduing tongue 147 | All kind of arguments and question deep, 148 | All replication prompt, and reason strong, 149 | For his advantage still did wake and sleep. 150 | To make the weeper laugh, the laugher weep, 151 | He had the dialect and different skill, 152 | Catching all passions in his craft of will, 153 | 154 | 'That he did in the general bosom reign 155 | Of young, of old, and sexes both enchanted, 156 | To dwell with him in thoughts, or to remain 157 | In personal duty, following where he haunted. 158 | Consents bewitched, ere he desire, have granted, 159 | And dialogued for him what he would say, 160 | Asked their own wills, and made their wills obey. 161 | 162 | 'Many there were that did his picture get, 163 | To serve their eyes, and in it put their mind; 164 | Like fools that in th' imagination set 165 | The goodly objects which abroad they find 166 | Of lands and mansions, theirs in thought assigned; 167 | And labouring in moe pleasures to bestow them 168 | Than the true gouty landlord which doth owe them. 169 | 170 | 'So many have, that never touched his hand, 171 | Sweetly supposed them mistress of his heart. 172 | My woeful self, that did in freedom stand, 173 | And was my own fee-simple, not in part, 174 | What with his art in youth, and youth in art, 175 | Threw my affections in his charmed power 176 | Reserved the stalk and gave him all my flower. 177 | 178 | 'Yet did I not, as some my equals did, 179 | Demand of him, nor being desired yielded; 180 | Finding myself in honour so forbid, 181 | With safest distance I mine honour shielded. 182 | Experience for me many bulwarks builded 183 | Of proofs new-bleeding, which remained the foil 184 | Of this false jewel, and his amorous spoil. 185 | 186 | 'But ah, who ever shunned by precedent 187 | The destined ill she must herself assay? 188 | Or forced examples, 'gainst her own content, 189 | To put the by-past perils in her way? 190 | Counsel may stop awhile what will not stay; 191 | For when we rage, advice is often seen 192 | By blunting us to make our wills more keen. 193 | 194 | 'Nor gives it satisfaction to our blood 195 | That we must curb it upon others' proof, 196 | To be forbod the sweets that seems so good 197 | For fear of harms that preach in our behoof. 198 | O appetite, from judgement stand aloof! 199 | The one a palate hath that needs will taste, 200 | Though Reason weep, and cry it is thy last. 201 | 202 | 'For further I could say this man's untrue, 203 | And knew the patterns of his foul beguiling; 204 | Heard where his plants in others' orchards grew; 205 | Saw how deceits were gilded in his smiling; 206 | Knew vows were ever brokers to defiling; 207 | Thought characters and words merely but art, 208 | And bastards of his foul adulterate heart. 209 | 210 | 'And long upon these terms I held my city, 211 | Till thus he 'gan besiege me: ""Gentle maid, 212 | Have of my suffering youth some feeling pity, 213 | And be not of my holy vows afraid. 214 | That's to ye sworn to none was ever said; 215 | For feasts of love I have been called unto, 216 | Till now did ne'er invite nor never woo. 217 | 218 | '""All my offences that abroad you see 219 | Are errors of the blood, none of the mind; 220 | Love made them not; with acture they may be, 221 | Where neither party is nor true nor kind. 222 | They sought their shame that so their shame did find; 223 | And so much less of shame in me remains 224 | By how much of me their reproach contains. 225 | 226 | '""Among the many that mine eyes have seen, 227 | Not one whose flame my heart so much as warmed, 228 | Or my affection put to th' smallest teen, 229 | Or any of my leisures ever charmed. 230 | Harm have I done to them, but ne'er was harmed; 231 | Kept hearts in liveries, but mine own was free, 232 | And reigned commanding in his monarchy. 233 | 234 | '""Look here what tributes wounded fancies sent me, 235 | Of paled pearls and rubies red as blood; 236 | Figuring that they their passions likewise lent me 237 | Of grief and blushes, aptly understood 238 | In bloodless white and the encrimsoned mood- 239 | Effects of terror and dear modesty, 240 | Encamped in hearts, but fighting outwardly. 241 | 242 | '""And, lo, behold these talents of their hair, 243 | With twisted metal amorously empleached, 244 | I have receiv'd from many a several fair, 245 | Their kind acceptance weepingly beseeched, 246 | With the annexions of fair gems enriched, 247 | And deep-brained sonnets that did amplify 248 | Each stone's dear nature, worth, and quality. 249 | 250 | '""The diamond? why, 'twas beautiful and hard, 251 | Whereto his invised properties did tend; 252 | The deep-green em'rald, in whose fresh regard 253 | Weak sights their sickly radiance do amend; 254 | The heaven-hued sapphire and the opal blend 255 | With objects manifold; each several stone, 256 | With wit well blazoned, smiled, or made some moan. 257 | 258 | '""Lo, all these trophies of affections hot, 259 | Of pensived and subdued desires the tender, 260 | Nature hath charged me that I hoard them not, 261 | But yield them up where I myself must render- 262 | That is, to you, my origin and ender; 263 | For these, of force, must your oblations be, 264 | Since I their altar, you enpatron me. 265 | 266 | '""O then advance of yours that phraseless hand 267 | Whose white weighs down the airy scale of praise; 268 | Take all these similes to your own command, 269 | Hallowed with sighs that burning lungs did raise; 270 | What me your minister for you obeys 271 | Works under you; and to your audit comes 272 | Their distract parcels in combined sums. 273 | 274 | '""Lo, this device was sent me from a nun, 275 | Or sister sanctified, of holiest note, 276 | Which late her noble suit in court did shun, 277 | Whose rarest havings made the blossoms dote; 278 | For she was sought by spirits of richest coat, 279 | But kept cold distance, and did thence remove 280 | To spend her living in eternal love. 281 | 282 | '""But, O my sweet, what labour is't to leave 283 | The thing we have not, mast'ring what not strives, 284 | Playing the place which did no form receive, 285 | Playing patient sports in unconstrained gyves! 286 | She that her fame so to herself contrives, 287 | The scars of battle scapeth by the flight, 288 | And makes her absence valiant, not her might. 289 | 290 | '""O pardon me in that my boast is true! 291 | The accident which brought me to her eye 292 | Upon the moment did her force subdue, 293 | And now she would the caged cloister fly. 294 | Religious love put out religion's eye. 295 | Not to be tempted, would she be immured, 296 | And now to tempt all liberty procured. 297 | 298 | '""How mighty then you are, O hear me tell! 299 | The broken bosoms that to me belong 300 | Have emptied all their fountains in my well, 301 | And mine I pour your ocean all among. 302 | I strong o'er them, and you o'er me being strong, 303 | Must for your victory us all congest, 304 | As compound love to physic your cold breast. 305 | 306 | '""My parts had pow'r to charm a sacred nun, 307 | Who, disciplined, ay, dieted in grace, 308 | Believed her eyes when they t'assail begun, 309 | All vows and consecrations giving place, 310 | O most potential love, vow, bond, nor space, 311 | In thee hath neither sting, knot, nor confine, 312 | For thou art all, and all things else are thine. 313 | 314 | '""When thou impressest, what are precepts worth 315 | Of stale example? When thou wilt inflame, 316 | How coldly those impediments stand forth, 317 | Of wealth, of filial fear, law, kindred, fame! 318 | Love's arms are peace, 'gainst rule, 'gainst sense, 'gainst shame. 319 | And sweetens, in the suff'ring pangs it bears, 320 | The aloes of all forces, shocks and fears. 321 | 322 | '""Now all these hearts that do on mine depend, 323 | Feeling it break, with bleeding groans they pine, 324 | And supplicant their sighs to your extend, 325 | To leave the batt'ry that you make 'gainst mine, 326 | Lending soft audience to my sweet design, 327 | And credent soul to that strong-bonded oath, 328 | That shall prefer and undertake my troth."" 329 | 330 | 'This said, his wat'ry eyes he did dismount, 331 | Whose sights till then were levelled on my face; 332 | Each cheek a river running from a fount 333 | With brinish current downward flowed apace. 334 | O, how the channel to the stream gave grace! 335 | Who glazed with crystal gate the glowing roses 336 | That flame through water which their hue encloses. 337 | 338 | 'O father, what a hell of witchcraft lies 339 | In the small orb of one particular tear! 340 | But with the inundation of the eyes 341 | What rocky heart to water will not wear? 342 | What breast so cold that is not warmed here? 343 | O cleft effect! cold modesty, hot wrath, 344 | Both fire from hence and chill extincture hath. 345 | 346 | 'For lo, his passion, but an art of craft, 347 | Even there resolved my reason into tears; 348 | There my white stole of chastity I daffed, 349 | Shook off my sober guards and civil fears; 350 | Appear to him as he to me appears, 351 | All melting; though our drops this diff'rence bore: 352 | His poisoned me, and mine did him restore. 353 | 354 | 'In him a plenitude of subtle matter, 355 | Applied to cautels, all strange forms receives, 356 | Of burning blushes or of weeping water, 357 | Or swooning paleness; and he takes and leaves, 358 | In either's aptness, as it best deceives, 359 | To blush at speeches rank, to weep at woes, 360 | Or to turn white and swoon at tragic shows; 361 | 362 | 'That not a heart which in his level came 363 | Could scape the hail of his all-hurting aim, 364 | Showing fair nature is both kind and tame; 365 | And, veiled in them, did win whom he would maim. 366 | Against the thing he sought he would exclaim; 367 | When he most burned in heart-wished luxury, 368 | He preached pure maid and praised cold chastity. 369 | 370 | 'Thus merely with the garment of a Grace 371 | The naked and concealed fiend he covered, 372 | That th' unexperient gave the tempter place, 373 | Which, like a cherubin, above them hovered. 374 | Who, young and simple, would not be so lovered? 375 | Ay me, I fell, and yet do question make 376 | What I should do again for such a sake. 377 | 378 | 'O, that infected moisture of his eye, 379 | O, that false fire which in his cheek so glowed, 380 | O, that forced thunder from his heart did fly, 381 | O, that sad breath his spongy lungs bestowed, 382 | O, all that borrowed motion, seeming owed, 383 | Would yet again betray the fore-betrayed, 384 | And new pervert a reconciled maid.' 385 | 386 | THE END" 387 | -------------------------------------------------------------------------------- /validation.txt: -------------------------------------------------------------------------------- 1 | 1609 2 | 3 | A LOVER'S COMPLAINT 4 | 5 | by William Shakespeare 6 | 7 | 8 | 9 | From off a hill whose concave womb reworded 10 | A plaintful story from a sist'ring vale, 11 | My spirits t'attend this double voice accorded, 12 | And down I laid to list the sad-tuned tale, 13 | Ere long espied a fickle maid full pale, 14 | Tearing of papers, breaking rings atwain, 15 | Storming her world with sorrow's wind and rain. 16 | 17 | Upon her head a platted hive of straw, 18 | Which fortified her visage from the sun, 19 | Whereon the thought might think sometime it saw 20 | The carcase of a beauty spent and done. 21 | Time had not scythed all that youth begun, 22 | Nor youth all quit, but spite of heaven's fell rage 23 | Some beauty peeped through lattice of seared age. 24 | 25 | Oft did she heave her napkin to her eyne, 26 | Which on it had conceited characters, 27 | Laund'ring the silken figures in the brine 28 | That seasoned woe had pelleted in tears, 29 | And often reading what contents it bears; 30 | As often shrieking undistinguished woe, 31 | In clamours of all size, both high and low. 32 | 33 | Sometimes her levelled eyes their carriage ride, 34 | As they did batt'ry to the spheres intend; 35 | Sometime diverted their poor balls are tied 36 | To th' orbed earth; sometimes they do extend 37 | Their view right on; anon their gazes lend 38 | To every place at once, and nowhere fixed, 39 | The mind and sight distractedly commixed. 40 | 41 | Her hair, nor loose nor tied in formal plat, 42 | Proclaimed in her a careless hand of pride; 43 | For some, untucked, descended her sheaved hat, 44 | Hanging her pale and pined cheek beside; 45 | Some in her threaden fillet still did bide, 46 | And, true to bondage, would not break from thence, 47 | Though slackly braided in loose negligence. 48 | 49 | A thousand favours from a maund she drew 50 | Of amber, crystal, and of beaded jet, 51 | Which one by one she in a river threw, 52 | Upon whose weeping margent she was set; 53 | Like usury applying wet to wet, 54 | Or monarchs' hands that lets not bounty fall 55 | Where want cries some, but where excess begs all. 56 | 57 | Of folded schedules had she many a one, 58 | Which she perused, sighed, tore, and gave the flood; 59 | Cracked many a ring of posied gold and bone, 60 | Bidding them find their sepulchres in mud; 61 | Found yet moe letters sadly penned in blood, 62 | With sleided silk feat and affectedly 63 | Enswathed and sealed to curious secrecy. 64 | 65 | These often bathed she in her fluxive eyes, 66 | And often kissed, and often 'gan to tear; 67 | Cried, 'O false blood, thou register of lies, 68 | What unapproved witness dost thou bear! 69 | Ink would have seemed more black and damned here! 70 | This said, in top of rage the lines she rents, 71 | Big discontents so breaking their contents. 72 | 73 | A reverend man that grazed his cattle nigh, 74 | Sometime a blusterer that the ruffle knew 75 | Of court, of city, and had let go by 76 | The swiftest hours observed as they flew, 77 | Towards this afflicted fancy fastly drew; 78 | And, privileged by age, desires to know 79 | In brief the grounds and motives of her woe. 80 | 81 | So slides he down upon his grained bat, 82 | And comely distant sits he by her side; 83 | When he again desires her, being sat, 84 | Her grievance with his hearing to divide. 85 | If that from him there may be aught applied 86 | Which may her suffering ecstasy assuage, 87 | 'Tis promised in the charity of age. 88 | 89 | 'Father,' she says, 'though in me you behold 90 | The injury of many a blasting hour, 91 | Let it not tell your judgement I am old: 92 | Not age, but sorrow, over me hath power. 93 | I might as yet have been a spreading flower, 94 | Fresh to myself, if I had self-applied 95 | Love to myself, and to no love beside. 96 | 97 | 'But woe is me! too early I attended 98 | A youthful suit- it was to gain my grace- 99 | O, one by nature's outwards so commended 100 | That maidens' eyes stuck over all his face. 101 | Love lacked a dwelling and made him her place; 102 | And when in his fair parts she did abide, 103 | She was new lodged and newly deified. 104 | 105 | 'His browny locks did hang in crooked curls; 106 | And every light occasion of the wind 107 | Upon his lips their silken parcels hurls. 108 | What's sweet to do, to do will aptly find: 109 | Each eye that saw him did enchant the mind; 110 | For on his visage was in little drawn 111 | What largeness thinks in Paradise was sawn. 112 | 113 | 'Small show of man was yet upon his chin; 114 | His phoenix down began but to appear, 115 | Like unshorn velvet, on that termless skin, 116 | Whose bare out-bragged the web it seemed to wear: 117 | Yet showed his visage by that cost more dear; 118 | And nice affections wavering stood in doubt 119 | If best were as it was, or best without. 120 | 121 | 'His qualities were beauteous as his form, 122 | For maiden-tongued he was, and thereof free; 123 | Yet if men moved him, was he such a storm 124 | As oft 'twixt May and April is to see, 125 | When winds breathe sweet, unruly though they be. 126 | His rudeness so with his authorized youth 127 | Did livery falseness in a pride of truth. 128 | 129 | 'Well could he ride, and often men would say, 130 | "That horse his mettle from his rider takes: 131 | Proud of subjection, noble by the sway, 132 | What rounds, what bounds, what course, what stop he makes!" 133 | And controversy hence a question takes 134 | Whether the horse by him became his deed, 135 | Or he his manage by th' well-doing steed. 136 | 137 | 'But quickly on this side the verdict went: 138 | His real habitude gave life and grace 139 | To appertainings and to ornament, 140 | Accomplished in himself, not in his case, 141 | All aids, themselves made fairer by their place, 142 | Came for additions; yet their purposed trim 143 | Pierced not his grace, but were all graced by him. 144 | 145 | 'So on the tip of his subduing tongue 146 | All kind of arguments and question deep, 147 | All replication prompt, and reason strong, 148 | For his advantage still did wake and sleep. 149 | To make the weeper laugh, the laugher weep, 150 | He had the dialect and different skill, 151 | Catching all passions in his craft of will, 152 | 153 | 'That he did in the general bosom reign 154 | Of young, of old, and sexes both enchanted, 155 | To dwell with him in thoughts, or to remain 156 | In personal duty, following where he haunted. 157 | Consents bewitched, ere he desire, have granted, 158 | And dialogued for him what he would say, 159 | Asked their own wills, and made their wills obey. 160 | 161 | 'Many there were that did his picture get, 162 | To serve their eyes, and in it put their mind; 163 | Like fools that in th' imagination set 164 | The goodly objects which abroad they find 165 | Of lands and mansions, theirs in thought assigned; 166 | And labouring in moe pleasures to bestow them 167 | Than the true gouty landlord which doth owe them. 168 | 169 | 'So many have, that never touched his hand, 170 | Sweetly supposed them mistress of his heart. 171 | My woeful self, that did in freedom stand, 172 | And was my own fee-simple, not in part, 173 | What with his art in youth, and youth in art, 174 | Threw my affections in his charmed power 175 | Reserved the stalk and gave him all my flower. 176 | 177 | 'Yet did I not, as some my equals did, 178 | Demand of him, nor being desired yielded; 179 | Finding myself in honour so forbid, 180 | With safest distance I mine honour shielded. 181 | Experience for me many bulwarks builded 182 | Of proofs new-bleeding, which remained the foil 183 | Of this false jewel, and his amorous spoil. 184 | 185 | 'But ah, who ever shunned by precedent 186 | The destined ill she must herself assay? 187 | Or forced examples, 'gainst her own content, 188 | To put the by-past perils in her way? 189 | Counsel may stop awhile what will not stay; 190 | For when we rage, advice is often seen 191 | By blunting us to make our wills more keen. 192 | 193 | 'Nor gives it satisfaction to our blood 194 | That we must curb it upon others' proof, 195 | To be forbod the sweets that seems so good 196 | For fear of harms that preach in our behoof. 197 | O appetite, from judgement stand aloof! 198 | The one a palate hath that needs will taste, 199 | Though Reason weep, and cry it is thy last. 200 | 201 | 'For further I could say this man's untrue, 202 | And knew the patterns of his foul beguiling; 203 | Heard where his plants in others' orchards grew; 204 | Saw how deceits were gilded in his smiling; 205 | Knew vows were ever brokers to defiling; 206 | Thought characters and words merely but art, 207 | And bastards of his foul adulterate heart. 208 | 209 | 'And long upon these terms I held my city, 210 | Till thus he 'gan besiege me: "Gentle maid, 211 | Have of my suffering youth some feeling pity, 212 | And be not of my holy vows afraid. 213 | That's to ye sworn to none was ever said; 214 | For feasts of love I have been called unto, 215 | Till now did ne'er invite nor never woo. 216 | 217 | '"All my offences that abroad you see 218 | Are errors of the blood, none of the mind; 219 | Love made them not; with acture they may be, 220 | Where neither party is nor true nor kind. 221 | They sought their shame that so their shame did find; 222 | And so much less of shame in me remains 223 | By how much of me their reproach contains. 224 | 225 | '"Among the many that mine eyes have seen, 226 | Not one whose flame my heart so much as warmed, 227 | Or my affection put to th' smallest teen, 228 | Or any of my leisures ever charmed. 229 | Harm have I done to them, but ne'er was harmed; 230 | Kept hearts in liveries, but mine own was free, 231 | And reigned commanding in his monarchy. 232 | 233 | '"Look here what tributes wounded fancies sent me, 234 | Of paled pearls and rubies red as blood; 235 | Figuring that they their passions likewise lent me 236 | Of grief and blushes, aptly understood 237 | In bloodless white and the encrimsoned mood- 238 | Effects of terror and dear modesty, 239 | Encamped in hearts, but fighting outwardly. 240 | 241 | '"And, lo, behold these talents of their hair, 242 | With twisted metal amorously empleached, 243 | I have receiv'd from many a several fair, 244 | Their kind acceptance weepingly beseeched, 245 | With the annexions of fair gems enriched, 246 | And deep-brained sonnets that did amplify 247 | Each stone's dear nature, worth, and quality. 248 | 249 | '"The diamond? why, 'twas beautiful and hard, 250 | Whereto his invised properties did tend; 251 | The deep-green em'rald, in whose fresh regard 252 | Weak sights their sickly radiance do amend; 253 | The heaven-hued sapphire and the opal blend 254 | With objects manifold; each several stone, 255 | With wit well blazoned, smiled, or made some moan. 256 | 257 | '"Lo, all these trophies of affections hot, 258 | Of pensived and subdued desires the tender, 259 | Nature hath charged me that I hoard them not, 260 | But yield them up where I myself must render- 261 | That is, to you, my origin and ender; 262 | For these, of force, must your oblations be, 263 | Since I their altar, you enpatron me. 264 | 265 | '"O then advance of yours that phraseless hand 266 | Whose white weighs down the airy scale of praise; 267 | Take all these similes to your own command, 268 | Hallowed with sighs that burning lungs did raise; 269 | What me your minister for you obeys 270 | Works under you; and to your audit comes 271 | Their distract parcels in combined sums. 272 | 273 | '"Lo, this device was sent me from a nun, 274 | Or sister sanctified, of holiest note, 275 | Which late her noble suit in court did shun, 276 | Whose rarest havings made the blossoms dote; 277 | For she was sought by spirits of richest coat, 278 | But kept cold distance, and did thence remove 279 | To spend her living in eternal love. 280 | 281 | '"But, O my sweet, what labour is't to leave 282 | The thing we have not, mast'ring what not strives, 283 | Playing the place which did no form receive, 284 | Playing patient sports in unconstrained gyves! 285 | She that her fame so to herself contrives, 286 | The scars of battle scapeth by the flight, 287 | And makes her absence valiant, not her might. 288 | 289 | '"O pardon me in that my boast is true! 290 | The accident which brought me to her eye 291 | Upon the moment did her force subdue, 292 | And now she would the caged cloister fly. 293 | Religious love put out religion's eye. 294 | Not to be tempted, would she be immured, 295 | And now to tempt all liberty procured. 296 | 297 | '"How mighty then you are, O hear me tell! 298 | The broken bosoms that to me belong 299 | Have emptied all their fountains in my well, 300 | And mine I pour your ocean all among. 301 | I strong o'er them, and you o'er me being strong, 302 | Must for your victory us all congest, 303 | As compound love to physic your cold breast. 304 | 305 | '"My parts had pow'r to charm a sacred nun, 306 | Who, disciplined, ay, dieted in grace, 307 | Believed her eyes when they t'assail begun, 308 | All vows and consecrations giving place, 309 | O most potential love, vow, bond, nor space, 310 | In thee hath neither sting, knot, nor confine, 311 | For thou art all, and all things else are thine. 312 | 313 | '"When thou impressest, what are precepts worth 314 | Of stale example? When thou wilt inflame, 315 | How coldly those impediments stand forth, 316 | Of wealth, of filial fear, law, kindred, fame! 317 | Love's arms are peace, 'gainst rule, 'gainst sense, 'gainst shame. 318 | And sweetens, in the suff'ring pangs it bears, 319 | The aloes of all forces, shocks and fears. 320 | 321 | '"Now all these hearts that do on mine depend, 322 | Feeling it break, with bleeding groans they pine, 323 | And supplicant their sighs to your extend, 324 | To leave the batt'ry that you make 'gainst mine, 325 | Lending soft audience to my sweet design, 326 | And credent soul to that strong-bonded oath, 327 | That shall prefer and undertake my troth." 328 | 329 | 'This said, his wat'ry eyes he did dismount, 330 | Whose sights till then were levelled on my face; 331 | Each cheek a river running from a fount 332 | With brinish current downward flowed apace. 333 | O, how the channel to the stream gave grace! 334 | Who glazed with crystal gate the glowing roses 335 | That flame through water which their hue encloses. 336 | 337 | 'O father, what a hell of witchcraft lies 338 | In the small orb of one particular tear! 339 | But with the inundation of the eyes 340 | What rocky heart to water will not wear? 341 | What breast so cold that is not warmed here? 342 | O cleft effect! cold modesty, hot wrath, 343 | Both fire from hence and chill extincture hath. 344 | 345 | 'For lo, his passion, but an art of craft, 346 | Even there resolved my reason into tears; 347 | There my white stole of chastity I daffed, 348 | Shook off my sober guards and civil fears; 349 | Appear to him as he to me appears, 350 | All melting; though our drops this diff'rence bore: 351 | His poisoned me, and mine did him restore. 352 | 353 | 'In him a plenitude of subtle matter, 354 | Applied to cautels, all strange forms receives, 355 | Of burning blushes or of weeping water, 356 | Or swooning paleness; and he takes and leaves, 357 | In either's aptness, as it best deceives, 358 | To blush at speeches rank, to weep at woes, 359 | Or to turn white and swoon at tragic shows; 360 | 361 | 'That not a heart which in his level came 362 | Could scape the hail of his all-hurting aim, 363 | Showing fair nature is both kind and tame; 364 | And, veiled in them, did win whom he would maim. 365 | Against the thing he sought he would exclaim; 366 | When he most burned in heart-wished luxury, 367 | He preached pure maid and praised cold chastity. 368 | 369 | 'Thus merely with the garment of a Grace 370 | The naked and concealed fiend he covered, 371 | That th' unexperient gave the tempter place, 372 | Which, like a cherubin, above them hovered. 373 | Who, young and simple, would not be so lovered? 374 | Ay me, I fell, and yet do question make 375 | What I should do again for such a sake. 376 | 377 | 'O, that infected moisture of his eye, 378 | O, that false fire which in his cheek so glowed, 379 | O, that forced thunder from his heart did fly, 380 | O, that sad breath his spongy lungs bestowed, 381 | O, all that borrowed motion, seeming owed, 382 | Would yet again betray the fore-betrayed, 383 | And new pervert a reconciled maid.' 384 | 385 | THE END --------------------------------------------------------------------------------