├── .gitignore ├── LICENSE ├── README.md ├── chainer_model.py ├── data ├── bible.txt ├── id2char.tsv ├── nietzsche.txt ├── shakespeare.txt ├── tinyshakespeare.txt └── warpeace.txt ├── environment.yml ├── keras_model.py ├── logger.py ├── logs ├── chainer_model.log ├── keras_model.log ├── mxnet_model.log ├── pytorch_model.log └── tf_model.log ├── mxnet_model.py ├── pytorch_model.py ├── scripts.sh ├── tf_model.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | # python 2 | __pycache__/ 3 | .pyc 4 | 5 | # ide 6 | .idea/ 7 | 8 | # project 9 | checkpoints/ 10 | main.log* 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 YuXuan Tay 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Character Embeddings Recurrent Neural Network Text Generation Models 2 | 3 | Inspired by [Andrej Karpathy](https://github.com/karpathy/)'s 4 | [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/). 5 | 6 | This repository attempts to replicate the models, with slight modifications, in different python deep learning frameworks. 7 | 8 | ## Frameworks 9 | 10 | - Keras: [`keras_model.py`](keras_model.py) 11 | - TensorFlow: [`tf_model.py`](tf_model.py) 12 | - PyTorch: [`pytorch_model.py`](pytorch_model.py) 13 | - Chainer: [`chainer_model.py`](chainer_model.py) 14 | - MXNet: [`mxnet_model.py`](mxnet_model.py) 15 | - CNTK 16 | - Caffe 17 | 18 | ## Default Model Specification 19 | 20 | | Layer Type | Output Shape | Param # | Remarks | 21 | |------------|---------------|---------|------------------------------------| 22 | | Embedding | (64, 64, 32) | 3136 | vocab size: 98, embedding size: 32 | 23 | | Dropout | (64, 64, 32) | 0 | dropout rate: 0.0 | 24 | | LSTM | (64, 64, 128) | 82432 | output size: 128 | 25 | | Dropout | (64, 64, 128) | 0 | dropout rate: 0.0 | 26 | | LSTM | (64, 64, 128) | 131584 | output size: 128 | 27 | | Dropout | (64, 64, 128) | 0 | dropout rate: 0.0 | 28 | | Dense | (64, 64, 98) | 12642 | output size: 98 | 29 | 30 | ### Training Specification 31 | 32 | - Batch size: 64 33 | - Sequence length: 64 34 | - Number of epochs: 32 35 | - Learning rate: 0.001 36 | - Max gradient norm: 5.0 37 | 38 | ## Setup 39 | 40 | ```bash 41 | # clone repo 42 | git clone git@github.com:yxtay/char-rnn-text-generation.git && cd char-rnn-text-generation 43 | 44 | # create conda environment 45 | conda env create -f=environment.yml 46 | 47 | # activate environment 48 | source activate dl 49 | ``` 50 | 51 | ## Usage 52 | 53 | ### Training 54 | 55 | ``` 56 | usage: _model.py train [-h] --checkpoint-path CHECKPOINT_PATH 57 | --text-path TEXT_PATH 58 | [--restore [RESTORE]] 59 | [--seq-len SEQ_LEN] 60 | [--embedding-size EMBEDDING_SIZE] 61 | [--rnn-size RNN_SIZE] 62 | [--num-layers NUM_LAYERS] 63 | [--drop-rate DROP_RATE] 64 | [--learning-rate LEARNING_RATE] 65 | [--clip-norm CLIP_NORM] 66 | [--batch-size BATCH_SIZE] 67 | [--num-epochs NUM_EPOCHS] 68 | [--log-path LOG_PATH] 69 | 70 | optional arguments: 71 | -h, --help show this help message and exit 72 | --checkpoint-path CHECKPOINT_PATH 73 | path to save or load model checkpoints 74 | --text-path TEXT_PATH 75 | path of text file for training 76 | --restore [RESTORE] whether to restore from checkpoint_path or from 77 | another path if specified 78 | --seq-len SEQ_LEN sequence length of inputs and outputs (default: 64) 79 | --embedding-size EMBEDDING_SIZE 80 | character embedding size (default: 32) 81 | --rnn-size RNN_SIZE size of rnn cell (default: 128) 82 | --num-layers NUM_LAYERS 83 | number of rnn layers (default: 2) 84 | --drop-rate DROP_RATE 85 | dropout rate for rnn layers (default: 0.0) 86 | --learning-rate LEARNING_RATE 87 | learning rate (default: 0.001) 88 | --clip-norm CLIP_NORM 89 | max norm to clip gradient (default: 5.0) 90 | --batch-size BATCH_SIZE 91 | training batch size (default: 64) 92 | --num-epochs NUM_EPOCHS 93 | number of epochs for training (default: 32) 94 | --log-path LOG_PATH path of log file (default: main.log) 95 | ``` 96 | 97 | Example: 98 | 99 | ```bash 100 | python tf_model.py train \ 101 | --checkpoint=checkpoints/tf_tinyshakespeare/model.ckpt \ 102 | --text=data/tinyshakespeare.txt 103 | ``` 104 | 105 | Sample logs: 106 | 107 | - [`keras_model.log`](logs/keras_model.log) 108 | - [`tf_model.log`](logs/tf_model.log) 109 | - [`pytorch_model.log`](logs/pytorch_model.log) 110 | - [`chainer_model.log`](logs/chainer_model.log) 111 | - [`mxnet_model.log`](logs/mxnet_model.log) 112 | 113 | ### Text Generation 114 | 115 | ``` 116 | usage: _model.py generate [-h] --checkpoint-path CHECKPOINT_PATH 117 | (--text-path TEXT_PATH | --seed SEED) 118 | [--length LENGTH] [--top-n TOP_N] 119 | [--log-path LOG_PATH] 120 | 121 | optional arguments: 122 | -h, --help show this help message and exit 123 | --checkpoint-path CHECKPOINT_PATH 124 | path to load model checkpoints 125 | --text-path TEXT_PATH 126 | path of text file to generate seed 127 | --seed SEED seed character sequence 128 | --length LENGTH length of character sequence to generate (default: 129 | 1024) 130 | --top-n TOP_N number of top choices to sample (default: 3) 131 | --log-path LOG_PATH path of log file (default: main.log) 132 | ``` 133 | 134 | Example: 135 | 136 | ```bash 137 | python tf_model.py generate \ 138 | --checkpoint=checkpoints/tf_tinyshakespeare/model.ckpt \ 139 | --seed="KING RICHARD" 140 | ``` 141 | 142 | Sample output: 143 | 144 | ``` 145 | KING RICHARDIIIIl II I tell thee, 146 | As I have no mark of his confection, 147 | The people so see my son. 148 | 149 | SEBASTIAN: 150 | I have men's man in the common to his sounds, 151 | And so she said of my soul, and to him, 152 | And too marry his sun their commanded 153 | As thou shalt be alone too means 154 | As he should to thy sensess so far to mark of 155 | these foul trust them fringer whom, there would he had 156 | As the word of merrous and subject. 157 | 158 | GLOUCESTER: 159 | A spack, a service the counsel son and here. 160 | What is a misin the wind and to the will 161 | And shall not streaks of this show into all heard. 162 | 163 | KING EDIN YORK: 164 | I will be suppet on himself tears as the sends. 165 | 166 | KING EDWARD IV: 167 | No looks and them, and while, a will, when this way. 168 | 169 | BAPTHIO: 170 | A mortain and me to the callant our souls 171 | And the changed and such of the son. 172 | 173 | CORIOLANUS: 174 | I will, so show me with the child to the could sheep 175 | To beseence, and shall so so should but hear 176 | Than him with her fair to be that soul, 177 | Whishe it is no meach of my lard and 178 | And this, and with my love and the senter'd with marked 179 | And her should 180 | ``` 181 | 182 | ## Benchmarks 183 | 184 | Below are training duration and loss on [`tinyshakespeare.txt`](data/tinyshakespeare.txt). 185 | 186 | | Framework | Duration (s) | Loss | 187 | |------------|--------------|---------| 188 | | Keras | 5270 | 1.42505 | 189 | | TensorFlow | 3003 | 1.45795 | 190 | | PyTorch | 5868 | 1.32285 | 191 | | Chainer | 4954 | 1.22930 | 192 | | MXNet | 7348 | 1.34199 | 193 | -------------------------------------------------------------------------------- /chainer_model.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import time 4 | 5 | import numpy as np 6 | 7 | import chainer 8 | from chainer import (functions as F, 9 | links as L, 10 | ChainList, Variable) 11 | from chainer.training import extension, extensions 12 | 13 | from logger import get_logger 14 | from utils import (batch_generator, encode_text, generate_seed, ID2CHAR, main, 15 | make_dirs, sample_from_probs, VOCAB_SIZE) 16 | 17 | logger = get_logger(__name__) 18 | 19 | 20 | class Network(ChainList): 21 | """ 22 | build character embeddings LSTM neural network. 23 | """ 24 | def __init__(self, vocab_size=VOCAB_SIZE, embedding_size=32, 25 | rnn_size=128, num_layers=2, drop_rate=0.0): 26 | super(Network, self).__init__() 27 | self.args = {"vocab_size": vocab_size, "embedding_size": embedding_size, 28 | "rnn_size": rnn_size, "num_layers": num_layers, 29 | "drop_rate": drop_rate} 30 | 31 | self.encoder = L.EmbedID(vocab_size, embedding_size) 32 | self.rnn_layers = [L.LSTM(embedding_size, rnn_size)] 33 | self.rnn_layers.extend(L.LSTM(rnn_size, rnn_size) for _ in range(num_layers-1)) 34 | self.decoder = L.Linear(rnn_size, vocab_size) 35 | 36 | self.add_link(self.encoder) 37 | for link in self.rnn_layers: 38 | self.add_link(link) 39 | self.add_link(self.decoder) 40 | 41 | def __call__(self, inputs): 42 | # input shape: [batch_size] 43 | embed_seq = F.dropout(self.encoder(inputs), self.args["drop_rate"]) 44 | # shape: [batch_size, embedding_size] 45 | rnn_out = embed_seq 46 | for link in self.rnn_layers: 47 | rnn_out = F.dropout(link(rnn_out), self.args["drop_rate"]) 48 | # shape: [batch_size, rnn_size] 49 | logits = self.decoder(rnn_out) 50 | # shape: [batch_size, vocab_size] 51 | return logits 52 | 53 | def reset_state(self): 54 | """ 55 | resets rnn states. 56 | """ 57 | for link in self.rnn_layers: 58 | link.reset_state() 59 | 60 | def get_state(self): 61 | """ 62 | get rnn states. 63 | """ 64 | return [(link.c, link.h) for link in self.rnn_layers] 65 | 66 | def set_state(self, state): 67 | """ 68 | set rnn states 69 | """ 70 | for link, (c, h) in zip(self.rnn_layers, state): 71 | link.set_state(c, h) 72 | 73 | 74 | def load_model(checkpoint_path): 75 | """ 76 | loads model from checkpoint_path. 77 | """ 78 | with open("{}.json".format(checkpoint_path)) as f: 79 | model_args = json.load(f) 80 | net = Network(**model_args) 81 | model = L.Classifier(net) 82 | chainer.serializers.load_npz(checkpoint_path, model) 83 | logger.info("model loaded: %s.", checkpoint_path) 84 | return model 85 | 86 | 87 | class DataIterator(chainer.dataset.Iterator): 88 | """ 89 | data iterator for chainer. 90 | """ 91 | def __init__(self, text, batch_size=64, seq_len=64): 92 | self.data_iterator = batch_generator(encode_text(text).astype(np.int32), 93 | batch_size, seq_len) 94 | self.num_batches = (len(text) - 1) // (batch_size * seq_len) 95 | self.iteration = 0 96 | self.epoch = 0 97 | self.is_new_epoch = True 98 | 99 | def __next__(self): 100 | self.iteration += 1 101 | self.is_new_epoch = self.iteration % self.num_batches == 0 102 | if self.is_new_epoch: 103 | self.epoch += 1 104 | 105 | return next(self.data_iterator) 106 | 107 | @property 108 | def epoch_detail(self): 109 | return self.iteration / self.num_batches 110 | 111 | def serialize(self, serializer): 112 | self.iteration = serializer('iteration', self.iteration) 113 | self.epoch = serializer('epoch', self.epoch) 114 | 115 | 116 | class BpttUpdater(chainer.training.StandardUpdater): 117 | """ 118 | updater for backpropagation through time. 119 | """ 120 | def update_core(self): 121 | train_iter = self.get_iterator('main') 122 | optimizer = self.get_optimizer('main') 123 | 124 | x, y = next(train_iter) 125 | seq_len = x.shape[1] 126 | 127 | loss = 0 128 | for i in range(seq_len): 129 | loss += optimizer.target(chainer.Variable(x[:, i]), chainer.Variable(y[:, i])) 130 | 131 | optimizer.target.cleargrads() # clear gradients 132 | loss.backward() # calculate gradient 133 | loss.unchain_backward() # truncate 134 | optimizer.update() # apply gradient update 135 | 136 | 137 | def generate_text(model, seed, length=512, top_n=10): 138 | """ 139 | generates text of specified length from trained model 140 | with given seed character sequence. 141 | """ 142 | logger.info("generating %s characters from top %s choices.", length, top_n) 143 | logger.info('generating with seed: "%s".', seed) 144 | generated = seed 145 | encoded = encode_text(seed).astype(np.int32) 146 | model.predictor.reset_state() 147 | 148 | with chainer.using_config("train", False), chainer.no_backprop_mode(): 149 | for idx in encoded[:-1]: 150 | x = Variable(np.array([idx])) 151 | # input shape: [1] 152 | # set internal states 153 | model.predictor(x) 154 | 155 | next_index = encoded[-1] 156 | for i in range(length): 157 | x = Variable(np.array([next_index], dtype=np.int32)) 158 | # input shape: [1] 159 | probs = F.softmax(model.predictor(x)) 160 | # output shape: [1, vocab_size] 161 | next_index = sample_from_probs(probs.data.squeeze(), top_n) 162 | # append to sequence 163 | generated += ID2CHAR[next_index] 164 | 165 | logger.info("generated text: \n%s\n", generated) 166 | return generated 167 | 168 | 169 | class LoggerExtension(extension.Extension): 170 | """ 171 | chainer Extension for logging. 172 | generates text at the end of each epoch. 173 | """ 174 | trigger = (1, "epoch") 175 | priority = -200 176 | 177 | def __init__(self, text): 178 | self.text = text 179 | self.time_epoch = time.time() 180 | 181 | def __call__(self, trainer): 182 | duration_epoch = time.time() - self.time_epoch 183 | epoch = trainer.updater.epoch 184 | loss = trainer.observation["main/loss"].data 185 | logger.info("epoch: %s, duration: %ds, loss: %.6g.", 186 | epoch, duration_epoch, loss) 187 | 188 | # get rnn state 189 | model = trainer.updater.get_optimizer("main").target 190 | state = model.predictor.get_state() 191 | # generate text 192 | seed = generate_seed(self.text) 193 | generate_text(model, seed) 194 | # set rnn back to training state 195 | model.predictor.set_state(state) 196 | 197 | # reset time 198 | self.time_epoch = time.time() 199 | 200 | def initialize(self, _): 201 | self.time_epoch = time.time() 202 | 203 | 204 | def train_main(args): 205 | """ 206 | trains model specfied in args. 207 | main method for train subcommand. 208 | """ 209 | # load text 210 | with open(args.text_path) as f: 211 | text = f.read() 212 | logger.info("corpus length: %s.", len(text)) 213 | 214 | # data iterator 215 | data_iter = DataIterator(text, args.batch_size, args.seq_len) 216 | 217 | # load or build model 218 | if args.restore: 219 | logger.info("restoring model.") 220 | load_path = args.checkpoint_path if args.restore is True else args.restore 221 | model = load_model(load_path) 222 | else: 223 | net = Network(vocab_size=VOCAB_SIZE, 224 | embedding_size=args.embedding_size, 225 | rnn_size=args.rnn_size, 226 | num_layers=args.num_layers, 227 | drop_rate=args.drop_rate) 228 | model = L.Classifier(net) 229 | 230 | # make checkpoint directory 231 | log_dir = make_dirs(args.checkpoint_path) 232 | with open("{}.json".format(args.checkpoint_path), "w") as f: 233 | json.dump(model.predictor.args, f, indent=2) 234 | chainer.serializers.save_npz(args.checkpoint_path, model) 235 | logger.info("model saved: %s.", args.checkpoint_path) 236 | 237 | # optimizer 238 | optimizer = chainer.optimizers.Adam(alpha=args.learning_rate) 239 | optimizer.setup(model) 240 | # clip gradient norm 241 | optimizer.add_hook(chainer.optimizer.GradientClipping(args.clip_norm)) 242 | 243 | # trainer 244 | updater = BpttUpdater(data_iter, optimizer) 245 | trainer = chainer.training.Trainer(updater, (args.num_epochs, 'epoch'), out=log_dir) 246 | trainer.extend(extensions.snapshot_object(model, filename=os.path.basename(args.checkpoint_path))) 247 | trainer.extend(extensions.ProgressBar(update_interval=1)) 248 | trainer.extend(extensions.LogReport()) 249 | trainer.extend(extensions.PlotReport(y_keys=["main/loss"])) 250 | trainer.extend(LoggerExtension(text)) 251 | 252 | # training start 253 | model.predictor.reset_state() 254 | logger.info("start of training.") 255 | time_train = time.time() 256 | trainer.run() 257 | 258 | # training end 259 | duration_train = time.time() - time_train 260 | logger.info("end of training, duration: %ds.", duration_train) 261 | # generate text 262 | seed = generate_seed(text) 263 | generate_text(model, seed, 1024, 3) 264 | return model 265 | 266 | 267 | def generate_main(args): 268 | """ 269 | generates text from trained model specified in args. 270 | main method for generate subcommand. 271 | """ 272 | # load model 273 | inference_model = load_model(args.checkpoint_path) 274 | 275 | # create seed if not specified 276 | if args.seed is None: 277 | with open(args.text_path) as f: 278 | text = f.read() 279 | seed = generate_seed(text) 280 | logger.info("seed sequence generated from %s.", args.text_path) 281 | else: 282 | seed = args.seed 283 | 284 | return generate_text(inference_model, seed, args.length, args.top_n) 285 | 286 | 287 | if __name__ == "__main__": 288 | main("Chainer", train_main, generate_main) 289 | -------------------------------------------------------------------------------- /data/id2char.tsv: -------------------------------------------------------------------------------- 1 | empty 2 | \t 3 | \n 4 | space 5 | ! 6 | " 7 | # 8 | $ 9 | % 10 | & 11 | ' 12 | ( 13 | ) 14 | * 15 | + 16 | , 17 | - 18 | . 19 | / 20 | 0 21 | 1 22 | 2 23 | 3 24 | 4 25 | 5 26 | 6 27 | 7 28 | 8 29 | 9 30 | : 31 | ; 32 | < 33 | = 34 | > 35 | ? 36 | @ 37 | A 38 | B 39 | C 40 | D 41 | E 42 | F 43 | G 44 | H 45 | I 46 | J 47 | K 48 | L 49 | M 50 | N 51 | O 52 | P 53 | Q 54 | R 55 | S 56 | T 57 | U 58 | V 59 | W 60 | X 61 | Y 62 | Z 63 | [ 64 | \ 65 | ] 66 | ^ 67 | _ 68 | ` 69 | a 70 | b 71 | c 72 | d 73 | e 74 | f 75 | g 76 | h 77 | i 78 | j 79 | k 80 | l 81 | m 82 | n 83 | o 84 | p 85 | q 86 | r 87 | s 88 | t 89 | u 90 | v 91 | w 92 | x 93 | y 94 | z 95 | { 96 | | 97 | } 98 | ~ -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: dl 2 | channels: 3 | - defaults 4 | - soumith 5 | dependencies: 6 | - anaconda=4.4.0 7 | - python=3.5 8 | - pytorch=0.1.12 9 | - tqdm=4.15.0 10 | - pip: 11 | - chainer==3.0.0 12 | - https://cntk.ai/PythonWheel/CPU-Only/cntk-2.2-cp35-cp35m-linux_x86_64.whl 13 | - keras==2.0.9 14 | - mxnet==0.12.0 15 | - tensorflow==1.4.0 16 | -------------------------------------------------------------------------------- /keras_model.py: -------------------------------------------------------------------------------- 1 | import os 2 | import time 3 | 4 | import numpy as np 5 | 6 | from keras.callbacks import Callback, ModelCheckpoint, TensorBoard 7 | from keras.layers import Dense, Dropout, Embedding, LSTM, TimeDistributed 8 | from keras.models import load_model, Sequential 9 | from keras.optimizers import Adam 10 | 11 | from logger import get_logger 12 | from utils import (batch_generator, encode_text, generate_seed, ID2CHAR, main, 13 | make_dirs, sample_from_probs, VOCAB_SIZE) 14 | 15 | logger = get_logger(__name__) 16 | 17 | 18 | def build_model(batch_size, seq_len, vocab_size=VOCAB_SIZE, embedding_size=32, 19 | rnn_size=128, num_layers=2, drop_rate=0.0, 20 | learning_rate=0.001, clip_norm=5.0): 21 | """ 22 | build character embeddings LSTM text generation model. 23 | """ 24 | logger.info("building model: batch_size=%s, seq_len=%s, vocab_size=%s, " 25 | "embedding_size=%s, rnn_size=%s, num_layers=%s, drop_rate=%s, " 26 | "learning_rate=%s, clip_norm=%s.", 27 | batch_size, seq_len, vocab_size, embedding_size, 28 | rnn_size, num_layers, drop_rate, 29 | learning_rate, clip_norm) 30 | model = Sequential() 31 | # input shape: (batch_size, seq_len) 32 | model.add(Embedding(vocab_size, embedding_size, 33 | batch_input_shape=(batch_size, seq_len))) 34 | model.add(Dropout(drop_rate)) 35 | # shape: (batch_size, seq_len, embedding_size) 36 | for _ in range(num_layers): 37 | model.add(LSTM(rnn_size, return_sequences=True, stateful=True)) 38 | model.add(Dropout(drop_rate)) 39 | # shape: (batch_size, seq_len, rnn_size) 40 | model.add(TimeDistributed(Dense(vocab_size, activation="softmax"))) 41 | # output shape: (batch_size, seq_len, vocab_size) 42 | optimizer = Adam(learning_rate, clipnorm=clip_norm) 43 | model.compile(loss="categorical_crossentropy", optimizer=optimizer) 44 | return model 45 | 46 | 47 | def build_inference_model(model, batch_size=1, seq_len=1): 48 | """ 49 | build inference model from model config 50 | input shape modified to (1, 1) 51 | """ 52 | logger.info("building inference model.") 53 | config = model.get_config() 54 | # edit batch_size and seq_len 55 | config[0]["config"]["batch_input_shape"] = (batch_size, seq_len) 56 | inference_model = Sequential.from_config(config) 57 | inference_model.trainable = False 58 | return inference_model 59 | 60 | 61 | def generate_text(model, seed, length=512, top_n=10): 62 | """ 63 | generates text of specified length from trained model 64 | with given seed character sequence. 65 | """ 66 | logger.info("generating %s characters from top %s choices.", length, top_n) 67 | logger.info('generating with seed: "%s".', seed) 68 | generated = seed 69 | encoded = encode_text(seed) 70 | model.reset_states() 71 | 72 | for idx in encoded[:-1]: 73 | x = np.array([[idx]]) 74 | # input shape: (1, 1) 75 | # set internal states 76 | model.predict(x) 77 | 78 | next_index = encoded[-1] 79 | for i in range(length): 80 | x = np.array([[next_index]]) 81 | # input shape: (1, 1) 82 | probs = model.predict(x) 83 | # output shape: (1, 1, vocab_size) 84 | next_index = sample_from_probs(probs.squeeze(), top_n) 85 | # append to sequence 86 | generated += ID2CHAR[next_index] 87 | 88 | logger.info("generated text: \n%s\n", generated) 89 | return generated 90 | 91 | 92 | class LoggerCallback(Callback): 93 | """ 94 | callback to log information. 95 | generates text at the end of each epoch. 96 | """ 97 | def __init__(self, text, model): 98 | super(LoggerCallback, self).__init__() 99 | self.text = text 100 | # build inference model using config from learning model 101 | self.inference_model = build_inference_model(model) 102 | self.time_train = self.time_epoch = time.time() 103 | 104 | def on_epoch_begin(self, epoch, logs=None): 105 | self.time_epoch = time.time() 106 | 107 | def on_epoch_end(self, epoch, logs=None): 108 | duration_epoch = time.time() - self.time_epoch 109 | logger.info("epoch: %s, duration: %ds, loss: %.6g.", 110 | epoch, duration_epoch, logs["loss"]) 111 | # transfer weights from learning model 112 | self.inference_model.set_weights(self.model.get_weights()) 113 | 114 | # generate text 115 | seed = generate_seed(self.text) 116 | generate_text(self.inference_model, seed) 117 | 118 | def on_train_begin(self, logs=None): 119 | logger.info("start of training.") 120 | self.time_train = time.time() 121 | 122 | def on_train_end(self, logs=None): 123 | duration_train = time.time() - self.time_train 124 | logger.info("end of training, duration: %ds.", duration_train) 125 | # transfer weights from learning model 126 | self.inference_model.set_weights(self.model.get_weights()) 127 | 128 | # generate text 129 | seed = generate_seed(self.text) 130 | generate_text(self.inference_model, seed, 1024, 3) 131 | 132 | 133 | def train_main(args): 134 | """ 135 | trains model specfied in args. 136 | main method for train subcommand. 137 | """ 138 | # load text 139 | with open(args.text_path) as f: 140 | text = f.read() 141 | logger.info("corpus length: %s.", len(text)) 142 | 143 | # load or build model 144 | if args.restore: 145 | load_path = args.checkpoint_path if args.restore is True else args.restore 146 | model = load_model(load_path) 147 | logger.info("model restored: %s.", load_path) 148 | else: 149 | model = build_model(batch_size=args.batch_size, 150 | seq_len=args.seq_len, 151 | vocab_size=VOCAB_SIZE, 152 | embedding_size=args.embedding_size, 153 | rnn_size=args.rnn_size, 154 | num_layers=args.num_layers, 155 | drop_rate=args.drop_rate, 156 | learning_rate=args.learning_rate, 157 | clip_norm=args.clip_norm) 158 | 159 | # make and clear checkpoint directory 160 | log_dir = make_dirs(args.checkpoint_path, empty=True) 161 | model.save(args.checkpoint_path) 162 | logger.info("model saved: %s.", args.checkpoint_path) 163 | # callbacks 164 | callbacks = [ 165 | ModelCheckpoint(args.checkpoint_path, verbose=1, save_best_only=False), 166 | TensorBoard(log_dir, write_graph=True, embeddings_freq=1, 167 | embeddings_metadata={"embedding_1": os.path.abspath(os.path.join("data", "id2char.tsv"))}), 168 | LoggerCallback(text, model) 169 | ] 170 | 171 | # training start 172 | num_batches = (len(text) - 1) // (args.batch_size * args.seq_len) 173 | model.reset_states() 174 | model.fit_generator(batch_generator(encode_text(text), args.batch_size, args.seq_len, one_hot_labels=True), 175 | num_batches, args.num_epochs, callbacks=callbacks) 176 | return model 177 | 178 | 179 | def generate_main(args): 180 | """ 181 | generates text from trained model specified in args. 182 | main method for generate subcommand. 183 | """ 184 | # load learning model for config and weights 185 | model = load_model(args.checkpoint_path) 186 | # build inference model and transfer weights 187 | inference_model = build_inference_model(model) 188 | inference_model.set_weights(model.get_weights()) 189 | logger.info("model loaded: %s.", args.checkpoint_path) 190 | 191 | # create seed if not specified 192 | if args.seed is None: 193 | with open(args.text_path) as f: 194 | text = f.read() 195 | seed = generate_seed(text) 196 | logger.info("seed sequence generated from %s.", args.text_path) 197 | else: 198 | seed = args.seed 199 | 200 | return generate_text(inference_model, seed, args.length, args.top_n) 201 | 202 | 203 | if __name__ == "__main__": 204 | main("Keras", train_main, generate_main) 205 | -------------------------------------------------------------------------------- /logger.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from logging.handlers import RotatingFileHandler 3 | import os 4 | 5 | 6 | def get_logger(name, log_path=os.path.join(os.path.dirname(__file__), "main.log"), console=False): 7 | """ 8 | Simple logging wrapper that returns logger 9 | configured to log into file and console. 10 | 11 | Args: 12 | name (str): name of logger 13 | log_path (str): path of log file 14 | console (bool): whether to log on console 15 | 16 | Returns: 17 | logging.Logger: configured logger 18 | """ 19 | logger = logging.getLogger(name) 20 | logger.setLevel(logging.DEBUG) 21 | formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") 22 | 23 | # ensure that logging handlers are not duplicated 24 | for handler in list(logger.handlers): 25 | logger.removeHandler(handler) 26 | 27 | # rotating file handler 28 | if log_path: 29 | fh = RotatingFileHandler(log_path, 30 | maxBytes=2 ** 20, # 1 MB 31 | backupCount=1) # 1 backup 32 | fh.setLevel(logging.DEBUG) 33 | fh.setFormatter(formatter) 34 | logger.addHandler(fh) 35 | 36 | # console handler 37 | if console: 38 | ch = logging.StreamHandler() 39 | ch.setLevel(logging.INFO) 40 | ch.setFormatter(formatter) 41 | logger.addHandler(ch) 42 | 43 | if len(logger.handlers) == 0: 44 | logger.addHandler(logging.NullHandler()) 45 | 46 | return logger 47 | -------------------------------------------------------------------------------- /logs/chainer_model.log: -------------------------------------------------------------------------------- 1 | 2017-09-06 02:39:33,433 - utils - DEBUG - call: chainer_model.py train --checkpoint=checkpoints/chainer_tinyshakespeare/model.npz --text=data/tinyshakespeare.txt 2 | 2017-09-06 02:39:33,434 - utils - DEBUG - ArgumentParser: Namespace(batch_size=64, checkpoint_path='checkpoints/chainer_tinyshakespeare/model.npz', clip_norm=5.0, drop_rate=0.0, embedding_size=32, learning_rate=0.001, main=, num_epochs=32, num_layers=2, restore=False, rnn_size=128, seq_len=64, text_path='data/tinyshakespeare.txt') 3 | 2017-09-06 02:39:33,435 - __main__ - INFO - corpus length: 1115394. 4 | 2017-09-06 02:39:33,750 - __main__ - INFO - model saved: checkpoints/chainer_tinyshakespeare/model.npz. 5 | 2017-09-06 02:39:33,751 - __main__ - INFO - start of training. 6 | 2017-09-06 02:39:33,751 - utils - INFO - number of batches: 272. 7 | 2017-09-06 02:39:33,751 - utils - INFO - effective text length: 1114112. 8 | 2017-09-06 02:39:33,751 - utils - INFO - x shape: (64, 17408). 9 | 2017-09-06 02:39:33,752 - utils - INFO - y shape: (64, 17408). 10 | 2017-09-06 02:42:07,492 - __main__ - INFO - epoch: 1, duration: 153s, loss: 2.20328. 11 | 2017-09-06 02:42:07,492 - __main__ - INFO - generating 512 characters from top 10 choices. 12 | 2017-09-06 02:42:07,492 - __main__ - INFO - generating with seed: "MIRA". 13 | 2017-09-06 02:42:08,056 - __main__ - INFO - generated text: 14 | MIRATSO VIA: 15 | And the crod ceing to nest 16 | Whof hous won that my wimliduiding so hath being, tatent, he tain: 17 | Meror wayer homis, it banites 18 | I the, theed hath. 19 | That whos hit onselst wonest. owirunes. I woflds all thele a hit. 20 | 21 | MELIRISR: 22 | A chandor thoud mimed he hive hase of is byother bour sall we ben the said wumdel tinged, 23 | Hand wor by for bunts ilgirs frecloue her wated a duistell spice toner other thoush toul his. 24 | Hester tio were to leat 25 | Bich here thy mither in to colse ad in thesinten te mart, me she me frowid 26 | 27 | 2017-09-06 02:44:42,362 - __main__ - INFO - epoch: 2, duration: 154s, loss: 2.00695. 28 | 2017-09-06 02:44:42,363 - __main__ - INFO - generating 512 characters from top 10 choices. 29 | 2017-09-06 02:44:42,363 - __main__ - INFO - generating with seed: " and yellow chapless skulls; 30 | Or ". 31 | 2017-09-06 02:44:42,933 - __main__ - INFO - generated text: 32 | and yellow chapless skulls; 33 | Or but to lord age my nunker 34 | Alles will bisting and whoud how fold fror of thing not my hevone, I the man: 35 | How sir, I hart she will. 36 | 37 | CUSENLA: 38 | Way thy htis a mookich and mome with the noth me fow homiend. 39 | 40 | AMION LOOS: 41 | He cothy the disprothour of his with, wiflestes this sacant is wee, 42 | Soother was one his hanks when the nortent: 43 | He sware a could for curchout a wandery. 44 | 45 | PUKE CININRIU: 46 | That is the will bot far wontishess. 47 | 48 | SOLANELE: 49 | For this not mithir was then. 50 | 51 | FLORY VIHE 52 | Nor, 53 | Or moon my bandevord: wist thou b 54 | 55 | 2017-09-06 02:47:17,112 - __main__ - INFO - epoch: 3, duration: 154s, loss: 1.83041. 56 | 2017-09-06 02:47:17,112 - __main__ - INFO - generating 512 characters from top 10 choices. 57 | 2017-09-06 02:47:17,112 - __main__ - INFO - generating with seed: "gero". 58 | 2017-09-06 02:47:17,664 - __main__ - INFO - generated text: 59 | gerokny; 60 | But to me he cought marring sir, 61 | Where your my spurs of a pure and as mathers 62 | I came but is mane how and so way 63 | A willl be my scack me conpends, 64 | And, him so mery by to them his sut by shim? 65 | 66 | DEKE VINDETO: 67 | But the bour these one. 68 | 69 | DUK AFCENNIO: 70 | What wows at a we see there in that dence, 71 | And an of and and, thou hand, who caming wender, his healty 72 | To moure with sir. 73 | What in to my the deniend trut. 74 | 75 | BROCTONY: 76 | This wempts, have sound with, you wile, it thee to not 77 | Thou we shair amon on hew, 78 | To meing tinglan 79 | 80 | 2017-09-06 02:49:52,011 - __main__ - INFO - epoch: 4, duration: 154s, loss: 1.75116. 81 | 2017-09-06 02:49:52,012 - __main__ - INFO - generating 512 characters from top 10 choices. 82 | 2017-09-06 02:49:52,012 - __main__ - INFO - generating with seed: "traitor to the crown 83 | In followin". 84 | 2017-09-06 02:49:52,585 - __main__ - INFO - generated text: 85 | traitor to the crown 86 | In following my boning all woud to so ampathe of hour wown to 87 | do the call: I'rn anger, be mading of terp hy wence, 88 | The king-boor a swurs, stays our marriend: 89 | And what's shall shall not to her art with'd and, 90 | That I part that wold our thince twith to, 91 | With you lear seems of harn you's me. 92 | I have mowelf what you like to throw wido, so pity; 93 | Of live you coumens an a saut as 94 | Which yef as, swist stakes the sauch, and not shoil 95 | nest case the fathes bethen with am whears of my suped 96 | Os this panding and her, whilch thee: 97 | Aunt 98 | 99 | 2017-09-06 02:52:26,734 - __main__ - INFO - epoch: 5, duration: 154s, loss: 1.68952. 100 | 2017-09-06 02:52:26,734 - __main__ - INFO - generating 512 characters from top 10 choices. 101 | 2017-09-06 02:52:26,734 - __main__ - INFO - generating with seed: "with the deep--mouth'd brach. 102 | Sa". 103 | 2017-09-06 02:52:27,319 - __main__ - INFO - generated text: 104 | with the deep--mouth'd brach. 105 | Sarroughtay. 106 | 107 | GEONDOLO: 108 | Thou let so blesmen a dearing to prepoland 109 | Why tike a heart many of my lond, 110 | With hears with so supfarment; to see to honane: 111 | Hows too the bout thing and drom of intuly 112 | And sempons! 113 | 114 | ASHOMENES: 115 | I make, fachisic brine time is it it heaven ough 116 | And tase one such so mine it she death. 117 | 118 | ANTERIO: 119 | Why, then have not an theer as this ney, 120 | Bust is the hand madured wo blungian almes thas begtance to dear bear. 121 | I tell you, my prowasies. Why sumsing my day, 122 | The peetting on the swings for ancent, 123 | 124 | 125 | 2017-09-06 02:55:01,949 - __main__ - INFO - epoch: 6, duration: 154s, loss: 1.6205. 126 | 2017-09-06 02:55:01,951 - __main__ - INFO - generating 512 characters from top 10 choices. 127 | 2017-09-06 02:55:01,952 - __main__ - INFO - generating with seed: ", rank-scented many, let them 128 | Re". 129 | 2017-09-06 02:55:02,652 - __main__ - INFO - generated text: 130 | , rank-scented many, let them 131 | Remife thy franted his friend; he all, by may bokincest me 132 | Of talk of it as by a mains. 133 | 134 | AFELIADA: 135 | That so him sters you make my souls all foul moried, 136 | And bether hell talk with me sue sad woth my-larck. 137 | On with our capently too, tought to make. 138 | 139 | DUKE VINCENTIO: 140 | When in whine take you stean yet are toon them 141 | beln in indo would not so'd ties: 142 | I cannot him beself heaven, batin in his counding. 143 | 144 | BETRUCHIO: 145 | O, say: you have crues your clough the homred, 146 | In by thou confest till be and whiles 147 | And so disfiet to port 148 | 149 | 2017-09-06 02:57:36,616 - __main__ - INFO - epoch: 7, duration: 153s, loss: 1.56866. 150 | 2017-09-06 02:57:36,616 - __main__ - INFO - generating 512 characters from top 10 choices. 151 | 2017-09-06 02:57:36,616 - __main__ - INFO - generating with seed: "hi". 152 | 2017-09-06 02:57:37,165 - __main__ - INFO - generated text: 153 | hit again 154 | Tell thee he come shold of that womb take one baswer my lord 155 | Struntst thou to by all he her come from buble banon's 156 | The garent to a son after? 157 | 158 | LUCIO: 159 | A head why sucems and liver, therefore father; 160 | Anwide of your house obed; will be stranity fear? 161 | 162 | Seponrir: 163 | O, my, seep never! I'll print the bose the course? 164 | What, her bite, and led the deever he will be wook, with a hang an is broal 165 | As I do her him him take the grue weet he him, 166 | This nother sin havour's pautions, of, to-now mine of to win thou bet 167 | A 168 | 169 | 2017-09-06 03:00:12,161 - __main__ - INFO - epoch: 8, duration: 154s, loss: 1.53299. 170 | 2017-09-06 03:00:12,162 - __main__ - INFO - generating 512 characters from top 10 choices. 171 | 2017-09-06 03:00:12,162 - __main__ - INFO - generating with seed: "y's love against some other maid". 172 | 2017-09-06 03:00:12,737 - __main__ - INFO - generated text: 173 | y's love against some other maid issue 174 | Seem to have a world's sounts main, with my grieve 175 | Thee fail that's master wisk of the dound, have harn wakell. 176 | The lease me; courth'd him hear of it soon, 177 | Thou hein am in the hands the moven'd, 178 | Which too, in he'd and a seeks the compostary'd 179 | Move have moughters it that may when the fill. 180 | Well you should, sir, with who loss. 181 | Which town sumps him, in my bagning and hams, burse, 182 | Faition'd this attern wrong'st; 183 | The waswards, her: 'to too begin of his her fight well. 184 | 185 | FLORIZEL: 186 | No, we hass sir. 187 | 188 | ANTLONUS 189 | 190 | 2017-09-06 03:02:46,855 - __main__ - INFO - epoch: 9, duration: 154s, loss: 1.49834. 191 | 2017-09-06 03:02:46,855 - __main__ - INFO - generating 512 characters from top 10 choices. 192 | 2017-09-06 03:02:46,855 - __main__ - INFO - generating with seed: "ul son. 193 | 194 | LUCENTI". 195 | 2017-09-06 03:02:47,421 - __main__ - INFO - generated text: 196 | ul son. 197 | 198 | LUCENTIO: 199 | Nothing since! I marry we take 200 | Sospetuse wank. 201 | 202 | POMPEY: 203 | What, wispy, Ispult should crown tonguial than am, 204 | So then sight to tonstart of my holy? 205 | 206 | POLIXENES: 207 | I thence is not the spold, swear and messes twilk; 208 | She write of confition for within timenes. 209 | 210 | ROMEO: 211 | Her he husband his mean on the bound must day. 212 | 213 | Sentent: 214 | Then with be nights. 215 | 216 | PETRUCHIO: 217 | My love,-- 218 | 219 | RAPIEL: 220 | Where, than you hear man you and with your son. 221 | 222 | CORIOLANUS: 223 | How! which her-was one infection of dack, 224 | She longeance and his such a shave be 225 | 226 | 2017-09-06 03:05:22,272 - __main__ - INFO - epoch: 10, duration: 154s, loss: 1.46976. 227 | 2017-09-06 03:05:22,272 - __main__ - INFO - generating 512 characters from top 10 choices. 228 | 2017-09-06 03:05:22,273 - __main__ - INFO - generating with seed: "a name 229 | I know not how to tell th". 230 | 2017-09-06 03:05:22,846 - __main__ - INFO - generated text: 231 | a name 232 | I know not how to tell their father's bed. 233 | He'er my great brock suppy speet one 234 | They she will comb to have the givimence. 235 | Seldening in his cave here, set you prive you: 236 | Gray the time hence as ourthing of this hath maid 237 | Whot so not the heart, thou did him that will'd too 238 | To his lise anwain, and I'll thereforairs. 239 | Hath be as hiss with much some for my life? 240 | 241 | BUCKINGHAM: 242 | Storn are nob with a mind his children,-- 243 | 244 | GrAMIS: 245 | But before with myself, tremply hence man, 246 | Which to speak all crown. 247 | 248 | Lord Mauryer: 249 | So speak; and not banishiness o 250 | 251 | 2017-09-06 03:07:56,210 - __main__ - INFO - epoch: 11, duration: 153s, loss: 1.44489. 252 | 2017-09-06 03:07:56,210 - __main__ - INFO - generating 512 characters from top 10 choices. 253 | 2017-09-06 03:07:56,211 - __main__ - INFO - generating with seed: " cam". 254 | 2017-09-06 03:07:56,761 - __main__ - INFO - generated text: 255 | camine 256 | Hear times as sworn in his charive to the pray 257 | Had somen and scorn that is this best brock. 258 | O, strengthe woo, is many shepherds? 259 | 260 | BIANGAL: 261 | No, a say, say, alitate, and I sither. 262 | 263 | ANTOPE: 264 | Whether fow you age siccess, sund they two with him 265 | To sevenged me with an heart with the people. 266 | 267 | DOFY OLVI: 268 | Where is the such sir. He'r than there is to him. 269 | Thy power to the cause in horses and toward; 270 | And should not there's now'd to my feighthem, 271 | Who do not thou turn and helsed and be his friend. 272 | All house, test a b 273 | 274 | 2017-09-06 03:10:31,753 - __main__ - INFO - epoch: 12, duration: 154s, loss: 1.43325. 275 | 2017-09-06 03:10:31,753 - __main__ - INFO - generating 512 characters from top 10 choices. 276 | 2017-09-06 03:10:31,753 - __main__ - INFO - generating with seed: "lt, 277 | I departed n". 278 | 2017-09-06 03:10:32,313 - __main__ - INFO - generated text: 279 | lt, 280 | I departed not be wink and 281 | Which we do so bid him? O'ercy, my light. 282 | 283 | AUTOLYCUS: 284 | How is hear it so shall not with here 285 | By think? 'Titul't to leady your giom 286 | That I come carment them be at other fly forty, 287 | Till I was secuntly fay, the blood their wrost: 288 | I stand men the glive and made it as me. 289 | 290 | LUCENTIO: 291 | Why at say, am her send as a fectility! 292 | How mether of wonks the high and conciling? 293 | She without her heart on the pose, 294 | Scausimented him a they poor bethy to be tears: 295 | And my good men your lords, take off 296 | And she well sh 297 | 298 | 2017-09-06 03:13:06,871 - __main__ - INFO - epoch: 13, duration: 154s, loss: 1.40113. 299 | 2017-09-06 03:13:06,871 - __main__ - INFO - generating 512 characters from top 10 choices. 300 | 2017-09-06 03:13:06,871 - __main__ - INFO - generating with seed: "nt". 301 | 2017-09-06 03:13:07,427 - __main__ - INFO - generated text: 302 | ntigr otenty she 303 | To-lliging soul in sick of thy sure. 304 | 305 | CAMILLO: 306 | The murch. 307 | 308 | LADY CENGRHA: 309 | The lass to a be denecife too may not in my life. 310 | 311 | CLARENCE: 312 | But not meation, be neither's witery, 313 | Surn too tears are as heathing the specting; 314 | And troughtery mistake mishit, and a provint. 315 | 316 | KING RICHARD III: 317 | Would sir, to be his lady what state. 318 | O, say within. He convilous of courted; 319 | When all fear his counterford's name: 320 | So say and with this all; and whose 321 | I save some lease of must wald is how, 322 | My fermower hearing, the 323 | 324 | 2017-09-06 03:15:41,774 - __main__ - INFO - epoch: 14, duration: 154s, loss: 1.40249. 325 | 2017-09-06 03:15:41,774 - __main__ - INFO - generating 512 characters from top 10 choices. 326 | 2017-09-06 03:15:41,774 - __main__ - INFO - generating with seed: "ully; 327 | God and Saint George! Rich". 328 | 2017-09-06 03:15:42,351 - __main__ - INFO - generated text: 329 | ully; 330 | God and Saint George! Riches me; for you: 331 | I have sisford: thou she we seems; will here cowergaus 332 | With wase me, thying me tauted she hatisfessable, 333 | Thing mine steplate him. Now't with him? 334 | 335 | ROMEOS: 336 | O, thou mist, why my sisp then the pity. 337 | 338 | BRUTUS: 339 | Why, thou so should not to save me not 340 | Than the troker, and mother stair? who since hear her? 341 | 342 | GONZALO: 343 | Be not hour with such nevel brother, and thou shemes. 344 | 345 | KATHARINA: 346 | The pity hampany maid, we most same as you 347 | here in so fair father stump his bast? 348 | 349 | POMPEY: 350 | Sir, when thou that treesest t 351 | 352 | 2017-09-06 03:18:16,758 - __main__ - INFO - epoch: 15, duration: 154s, loss: 1.38273. 353 | 2017-09-06 03:18:16,759 - __main__ - INFO - generating 512 characters from top 10 choices. 354 | 2017-09-06 03:18:16,759 - __main__ - INFO - generating with seed: "nian trull, 355 | Upon". 356 | 2017-09-06 03:18:17,330 - __main__ - INFO - generated text: 357 | nian trull, 358 | Upon her heart of say, and not said 359 | These never let him so few their seedent: 360 | Thy daughty senise fresh to me, 361 | Tremblant, my spoth it. Who, I have seek to poster hath, 362 | Then say wither bowing. I must her feing have 363 | And which now took, thy mother could seeming; 364 | For I am now walk she hath appenious 365 | The ship shaing so somethis with me: which his part that 366 | And tyous of it at time often for thee 367 | In head o' my choit on eat my bloody. 368 | 369 | Sendlat: 370 | Who, no, murns all foot he should call me when I must turn. 371 | Where is the sin 372 | 373 | 2017-09-06 03:20:51,870 - __main__ - INFO - epoch: 16, duration: 154s, loss: 1.37663. 374 | 2017-09-06 03:20:51,871 - __main__ - INFO - generating 512 characters from top 10 choices. 375 | 2017-09-06 03:20:51,871 - __main__ - INFO - generating with seed: "SHOP OF ". 376 | 2017-09-06 03:20:52,425 - __main__ - INFO - generated text: 377 | SHOP OF heack;? 378 | Sway betwears, my country, but that my cheek, as I 379 | pern me; the least facting of that suice? 380 | 381 | PETRUCHIO: 382 | If my hearts short our takest have sheepss. 383 | But servant, beseeming of your soldiers, 384 | Mus are not have you hear accihes in his week now, 385 | Or honusey, and no dangerence. 386 | 387 | DUKE VINCENTIO: 388 | The came to be thee those her to me, 389 | To sease. Come, we call me here, we come and sweet hither: 390 | Sholes the city mints, he were to plaint to hear: 391 | I hort of this woman the dangerous petier; 392 | I to see her been out of t 393 | 394 | 2017-09-06 03:23:27,410 - __main__ - INFO - epoch: 17, duration: 154s, loss: 1.36534. 395 | 2017-09-06 03:23:27,410 - __main__ - INFO - generating 512 characters from top 10 choices. 396 | 2017-09-06 03:23:27,410 - __main__ - INFO - generating with seed: "such". 397 | 2017-09-06 03:23:27,962 - __main__ - INFO - generated text: 398 | suche, 399 | Fill with the dares for you bawl of shouldst so bear abame 400 | I too the thunt with a motions with danger: 401 | But standelled in the what is mad me to't. 402 | 403 | ARIEL: 404 | Ay, bate, the ports and shemest the selse to comilely 405 | And wheether by selves for thee, for so? 406 | 407 | ANTONIA: 408 | Yet the flatterer, buisp'd beseen me, 409 | Bawners her husband, my fear with him to semple; 410 | Which with him invented, are confeds from him trust her cheeks. 411 | How, bet my house at you at all to shame 412 | Her prophem far your willing and speak, 413 | Which that hath se 414 | 415 | 2017-09-06 03:26:02,318 - __main__ - INFO - epoch: 18, duration: 154s, loss: 1.36233. 416 | 2017-09-06 03:26:02,319 - __main__ - INFO - generating 512 characters from top 10 choices. 417 | 2017-09-06 03:26:02,319 - __main__ - INFO - generating with seed: "e!". 418 | 2017-09-06 03:26:02,972 - __main__ - INFO - generated text: 419 | e!nx good wither, 420 | Swear the devil'd on solesters with you marks 421 | Was yourself on the first--felery 422 | Tales to still men's: you then, and showest the shem, but 423 | He will so lord's same, my loving to my head. 424 | And ill was but for he is, but not see well, were 425 | comboted he may age, betick to me is. 426 | 427 | ANGELO: 428 | What I mend, by the briefter-brishenia? 429 | I say it in all mere shall be some belor. 430 | 431 | PROSPERO: 432 | Alemy, sir my mother, souls their children's dound 433 | And noth face by they floward thy hass. 434 | 435 | SICINIUS: 436 | Yet are not we would 437 | 438 | 2017-09-06 03:28:36,880 - __main__ - INFO - epoch: 19, duration: 153s, loss: 1.35177. 439 | 2017-09-06 03:28:36,881 - __main__ - INFO - generating 512 characters from top 10 choices. 440 | 2017-09-06 03:28:36,881 - __main__ - INFO - generating with seed: " Pla". 441 | 2017-09-06 03:28:37,438 - __main__ - INFO - generated text: 442 | Playess death 443 | Away, and a word. 444 | 445 | BRAKENBURY: 446 | I stand stoppises steel, I how them now. 447 | 448 | BARTAIUS: 449 | The last to breathe sir; but the moured soul, a most 450 | death how he be propilloase and sound 451 | That which I price thee we be thwine of you would 452 | The willench solus weddest to a throngs that way, 453 | And many selcourt and work your seemity's words! 454 | I have some such fiery beseech those at success: 455 | And I have done to speece and better how true. 456 | 457 | ROMEO: 458 | A corrue your: how tonded by them so mock; 459 | The perceives he sufference to- 460 | 461 | 2017-09-06 03:31:12,163 - __main__ - INFO - epoch: 20, duration: 154s, loss: 1.35776. 462 | 2017-09-06 03:31:12,163 - __main__ - INFO - generating 512 characters from top 10 choices. 463 | 2017-09-06 03:31:12,164 - __main__ - INFO - generating with seed: "r,". 464 | 2017-09-06 03:31:12,715 - __main__ - INFO - generated text: 465 | r,.;: 466 | When shall I trump of the seats friend 467 | Till you have mask too much; 468 | They with him. 469 | 470 | First Senot, 471 | I'ar Gaunt; 472 | But tell her best trucking in my brothers: 473 | Therefore we sheeps had no man in your freshing 474 | Why bold of the consumeman, by the worsk, 475 | And a power all trupt his harlet. 476 | 477 | BRUCUTUS: 478 | I'll be since, what were no means some six. 479 | 480 | KATHARINA: 481 | Why, all sees me the sun of his father's lorse. 482 | O sir. 483 | 484 | ARCHIDARDUNI: 485 | And thou hence will crued your life and witents 486 | Told tell you. 487 | 488 | SEBASTIAN: 489 | That bear the times, 490 | 491 | 2017-09-06 03:33:47,074 - __main__ - INFO - epoch: 21, duration: 154s, loss: 1.3334. 492 | 2017-09-06 03:33:47,074 - __main__ - INFO - generating 512 characters from top 10 choices. 493 | 2017-09-06 03:33:47,075 - __main__ - INFO - generating with seed: "an by his own alms empoison'd, 494 | A". 495 | 2017-09-06 03:33:47,659 - __main__ - INFO - generated text: 496 | an by his own alms empoison'd, 497 | And take the burst to dined his house: 498 | His now, which here with some accame crmit'd slain. 499 | Hast he cause it onces son, wilt no booble. 500 | Welcome my common. 501 | Now, me, good soul, as a father in the brick 502 | And my seeken thou do acciken, is born. 503 | 504 | KING RICHARD II: 505 | In purshing to soul honours imprise by me 506 | The days; swe't an an entrior and my hours, sir? 507 | In makes time by their truty's best; 508 | But in ere I cannot bear me, when awake the 509 | prove me intentenence in the pitter. 510 | 511 | CORIOLANUS: 512 | There hope they do sollo, at suppl 513 | 514 | 2017-09-06 03:36:21,934 - __main__ - INFO - epoch: 22, duration: 154s, loss: 1.31603. 515 | 2017-09-06 03:36:21,934 - __main__ - INFO - generating 512 characters from top 10 choices. 516 | 2017-09-06 03:36:21,934 - __main__ - INFO - generating with seed: " convent". 517 | 2017-09-06 03:36:22,496 - __main__ - INFO - generated text: 518 | conventy head, 519 | And here not a did now to the shame 520 | May tell me and has a breathest calard, 521 | To do a sen of hours, and not wonder. 522 | 523 | BROTSA: 524 | Sir these! 525 | What, with him, we what he did the rasy, 526 | From this wide twice man all as this tarch's love, 527 | That dost music appersed me old my 528 | bellovel to bring me strange out with steel. 529 | 530 | LEONTE: 531 | This womar wised to all this wisdone, by this, thine heavens, 532 | And saw no much on any of this is the weep: 533 | He shespeats that self she nails incite mine. 534 | Mork him of this dukeless within to d 535 | 536 | 2017-09-06 03:38:56,771 - __main__ - INFO - epoch: 23, duration: 154s, loss: 1.31793. 537 | 2017-09-06 03:38:56,772 - __main__ - INFO - generating 512 characters from top 10 choices. 538 | 2017-09-06 03:38:56,772 - __main__ - INFO - generating with seed: ": 539 | My lord, 'tis nothing. 540 | 541 | DUKE O". 542 | 2017-09-06 03:38:57,352 - __main__ - INFO - generated text: 543 | : 544 | My lord, 'tis nothing. 545 | 546 | DUKE OF YORK: 547 | This, thou thy love to trails, whose commont time 548 | Fie thither with a fire, took time of hand? 549 | 550 | First God: 551 | Ay, they! what wave suit be at this mattel: 552 | My day with me weet by by, but saves to thine eyes 553 | Who wound frown that satisfied in't; 554 | Were news, tree so, the word: I will be done 555 | Whilst you are many last his changed to some still, 556 | Which silence with hour so hearts, as how, 557 | Who is, I was so, from your man, sir, breathest now, 558 | My found well with the matter. 559 | 560 | BENVOLIO: 561 | Might to crongeth cliens athal, 562 | 563 | 2017-09-06 03:41:31,729 - __main__ - INFO - epoch: 24, duration: 154s, loss: 1.29501. 564 | 2017-09-06 03:41:31,729 - __main__ - INFO - generating 512 characters from top 10 choices. 565 | 2017-09-06 03:41:31,730 - __main__ - INFO - generating with seed: "brown thread: I ". 566 | 2017-09-06 03:41:32,292 - __main__ - INFO - generated text: 567 | brown thread: I arreak was dount. 568 | But music with storch'd his lordsheches! 569 | It speaks, and there to be so again? 570 | 571 | COMINIUS: 572 | O, sir? 573 | 574 | BAPTISTA: 575 | I am my lord. 576 | 577 | CAGILLA: 578 | Well, and a peace! O that, they shall not but meat sweet 579 | Against the mark, and loward him. 580 | 581 | Second Gentleman: 582 | Mass be a friar, how now, good, and live your hate, 583 | Clarmen for may not me for thee, sir! 584 | 585 | ROMEO: 586 | And, and mark thy glass or hard yours, but the day 587 | On wife: I cale no murdered; yet that we well; 588 | And with soldiers with down from the hearing-life 589 | And so 590 | 591 | 2017-09-06 03:44:07,064 - __main__ - INFO - epoch: 25, duration: 154s, loss: 1.30117. 592 | 2017-09-06 03:44:07,064 - __main__ - INFO - generating 512 characters from top 10 choices. 593 | 2017-09-06 03:44:07,064 - __main__ - INFO - generating with seed: "conferen". 594 | 2017-09-06 03:44:07,623 - __main__ - INFO - generated text: 595 | conferens arm, 596 | And her satisfy: to she bad and may being. 597 | 598 | Servant: 599 | What thou, my grave asleess outchased? 600 | 601 | First Gentleman: 602 | You are her, here thou would be so wounds be complection. 603 | 604 | CLARENCA: 605 | And, take the duke ye it but both. 606 | 607 | DUKE OF YORK: 608 | My sweet. 609 | But it, mockly ababures had been me, she 610 | would be as for your face. Come, so let that make 611 | thee to the time show him, my staties of your silence. 612 | Ah, my lord, and was fornity on, been hath 613 | to my mountly swarest blue forth she might fift. 614 | All the sure of my way, so,' 615 | 616 | 2017-09-06 03:46:41,374 - __main__ - INFO - epoch: 26, duration: 153s, loss: 1.27951. 617 | 2017-09-06 03:46:41,375 - __main__ - INFO - generating 512 characters from top 10 choices. 618 | 2017-09-06 03:46:41,375 - __main__ - INFO - generating with seed: "to bring him 619 | Whe". 620 | 2017-09-06 03:46:41,934 - __main__ - INFO - generated text: 621 | to bring him 622 | Where tell with me, in that's meet, and say 623 | But I cannot be shall ask the fire. 624 | 625 | Shepherd: 626 | Wite, my togembrobs meet's, 627 | Arace tell your brother of the mountlius: 628 | For this dear, 629 | When I cannot banish'd on thy specent 630 | We honest calls of her hurse, but how they bear. 631 | If all the wreck some fool and his most recok. 632 | 633 | LADENN: 634 | Nor well to try too-shor, wilt make his cherk, 635 | O mother sweet more whom makes the dear again: 636 | Therefore come to him: it is it did, my heart, 637 | Not to do at flantation, and when thou wert thy mere, 638 | T 639 | 640 | 2017-09-06 03:49:16,836 - __main__ - INFO - epoch: 27, duration: 154s, loss: 1.2678. 641 | 2017-09-06 03:49:16,836 - __main__ - INFO - generating 512 characters from top 10 choices. 642 | 2017-09-06 03:49:16,836 - __main__ - INFO - generating with seed: "ond? 643 | 644 | RATCLIFF: 645 | ". 646 | 2017-09-06 03:49:17,408 - __main__ - INFO - generated text: 647 | ond? 648 | 649 | RATCLIFF: 650 | No, my good sweet maliset a mighty, and a sour than 651 | The law or hearing with thy destain, and how to his 652 | mothing hath trumpets but to help you all, the dispers, 653 | On my self it? 654 | 655 | PROSPERO: 656 | Whilst, thou dryalth have no right. 657 | 658 | SAMPSOLAN: 659 | Sir? 660 | 661 | LEONTES: 662 | O Go, my good willow, good will you curse; 663 | Ay, any man, and what? stild-cerear, that I will respeteen 664 | Angent, we must, millious. 665 | 666 | MIRANDA: 667 | The letter thines within myself, 668 | I would are the dream'd a last follow to pace to 669 | A throne; the soul agreen look'd on the co 670 | 671 | 2017-09-06 03:51:50,665 - __main__ - INFO - epoch: 28, duration: 153s, loss: 1.25535. 672 | 2017-09-06 03:51:50,665 - __main__ - INFO - generating 512 characters from top 10 choices. 673 | 2017-09-06 03:51:50,665 - __main__ - INFO - generating with seed: " of the king: 674 | I dare adventure t". 675 | 2017-09-06 03:51:51,237 - __main__ - INFO - generated text: 676 | of the king: 677 | I dare adventure that it and safe our grace. 678 | She have towers, had that may past itself'd; hereol thee 679 | Is thy welch, we have wombrage with sit 680 | Where this have thee showility and do should chome; 681 | In this. 682 | 683 | BRAKENBURY: 684 | A charced mouth, an empty time to 685 | Has sad fediclany as, both haste 686 | Her late. 687 | 688 | PERDITE: 689 | How, what neary so? 690 | I water well fearful's world, 691 | An sleen taking, how to be dallible bringed; 692 | And your labating by mind, and malk'd at you? 693 | 694 | GONZALO: 695 | Nay, we had such both and beliver sad 696 | With souly created for factest she hit 697 | 698 | 2017-09-06 03:54:25,863 - __main__ - INFO - epoch: 29, duration: 154s, loss: 1.24639. 699 | 2017-09-06 03:54:25,863 - __main__ - INFO - generating 512 characters from top 10 choices. 700 | 2017-09-06 03:54:25,863 - __main__ - INFO - generating with seed: ": le". 701 | 2017-09-06 03:54:26,423 - __main__ - INFO - generated text: 702 | : letter 703 | The power in her heirs in evate are men, 704 | And so must be second by cannot-cat, 705 | The people on the fill and was his part of my lord, 706 | To be times by so duke frown with the wind miles 707 | And that fair being with with your freshing foe: 708 | Yet not my lord, so phoper things summer 709 | Of him. 710 | 711 | MONTAGUE: 712 | Yea, here comes to to my honour. 713 | 714 | LADY GREY: 715 | This sworror hour and themself. 716 | 717 | FRIAR LAURENCE: 718 | Titundle hence, if you are no more, 719 | Be pause and pertued held as twenty sin: 720 | It should not strives to dividst heart of your c 721 | 722 | 2017-09-06 03:56:59,268 - __main__ - INFO - epoch: 30, duration: 152s, loss: 1.2398. 723 | 2017-09-06 03:56:59,268 - __main__ - INFO - generating 512 characters from top 10 choices. 724 | 2017-09-06 03:56:59,268 - __main__ - INFO - generating with seed: "cord". 725 | 2017-09-06 03:56:59,817 - __main__ - INFO - generated text: 726 | cordOr- Ill calle. 727 | 728 | SLY: 729 | I saw he take her heaven fair can will, how 730 | As her lamenty we are tormal 731 | And blood is a thought for some teach it. 732 | 733 | PRINCE EDWARD: 734 | O first, the best; yet had 735 | Is this whores this lord's considered 736 | And fareless in such was flatterit home, 737 | Hath muse I abbudble the does towered wime, 738 | I am men worrife, her high to be hearent 739 | Adums;' Omab against the marked to-nighty; 740 | For the state. They be dredest thou didst have seven agin. 741 | Swear it, sigaf, there stirr'd a biggard at are yearling faults mor 742 | 743 | 2017-09-06 03:59:33,802 - __main__ - INFO - epoch: 31, duration: 153s, loss: 1.23572. 744 | 2017-09-06 03:59:33,803 - __main__ - INFO - generating 512 characters from top 10 choices. 745 | 2017-09-06 03:59:33,803 - __main__ - INFO - generating with seed: "hee! 746 | 747 | SAMPSON: 748 | Let us take the l". 749 | 2017-09-06 03:59:34,379 - __main__ - INFO - generated text: 750 | hee! 751 | 752 | SAMPSON: 753 | Let us take the least: my news, whereons, youth with heart me 754 | As for your people of the pass; and I 755 | as thou thy substant on him such brows some 756 | With troak as horses in heaven: I would have been, 757 | And so most assuranish'd? 758 | 759 | DUKE OF AUMERLE: 760 | Nay, the heavens seem of wife, when, sigh'd by: 761 | Some sorrows. 762 | 763 | CORIOLANUS: 764 | You coundemn, I will have leave you may do there, 765 | Tybrey to help'd. Bown with a generage! 766 | Which is this speeder man, in peace of your wife, 767 | My life and sin, as acception: 768 | The missings at the better believes, to be 769 | t 770 | 771 | 2017-09-06 04:02:07,907 - __main__ - INFO - epoch: 32, duration: 153s, loss: 1.2293. 772 | 2017-09-06 04:02:07,907 - __main__ - INFO - generating 512 characters from top 10 choices. 773 | 2017-09-06 04:02:07,907 - __main__ - INFO - generating with seed: "ing tears? 774 | As thus, to drop them". 775 | 2017-09-06 04:02:08,485 - __main__ - INFO - generated text: 776 | ing tears? 777 | As thus, to drop them a wind bravel well soldier, 778 | As thoughts me so marked the wars'd of you 779 | The bill of heaven of times on some contrary. 780 | 781 | CAPULET: 782 | Ay, what we cloudds, speak forth for fools whiles 783 | Her season with the dukedom as most could swear to 784 | souls not one to thy hundred blood. 785 | 786 | Second Gentleman: 787 | Will I have look fair and my weighty arm, 788 | Seizony of the would two would not best thou the 789 | ratily thousand a head of my master! 790 | Where's so, how further, so heaven, was a worth 791 | I mean the wife of two felse, I know 792 | Here shall stil 793 | 794 | 2017-09-06 04:02:08,485 - __main__ - INFO - end of training, duration: 4954s. 795 | 2017-09-06 04:02:08,486 - __main__ - INFO - generating 1024 characters from top 3 choices. 796 | 2017-09-06 04:02:08,486 - __main__ - INFO - generating with seed: "l-seeming, thick, bereft of beau". 797 | 2017-09-06 04:02:09,611 - __main__ - INFO - generated text: 798 | l-seeming, thick, bereft of beauty, 799 | And, sir, the worse of his provost of my servant. 800 | This shall see thee and a son of the cause 801 | Than tell myself to the seas of thiest and so stone 802 | That will be shame the contrary to him as her 803 | As the winter to him, 804 | As to this be too sease, and the worst. 805 | 806 | BIONDELLO: 807 | I say, and we will be the words, where it will 808 | she's so broked thee, take her sour to the common 809 | As the meen of thy soul true. 810 | 811 | PETRUCHIO: 812 | A master with thy heir and this way he will stay. 813 | 814 | GLOUCESTER: 815 | Whan thou show me to my sword, a part them should be 816 | Than the people and an old sun without to 817 | conceive the sun, and the world, as thou stop 818 | Whoses so stone of the cape thee at the deed, 819 | With such assure to see the counsel is to his 820 | make my lady of his son will not best, 821 | And there that she's a marked and a promised her best 822 | And strict of heaven as then, she will be 823 | speed it is assured where they are some often as 824 | The sun shall should not bring the sun her brief, 825 | And the sun when have mercy a man things, 826 | And the storm to the stronger shall here to 827 | -------------------------------------------------------------------------------- /logs/keras_model.log: -------------------------------------------------------------------------------- 1 | 2017-09-06 06:04:53,977 - utils - DEBUG - call: keras_model.py train --checkpoint=checkpoints/keras_tinyshakespeare/model.hdf5 --text=data/tinyshakespeare.txt 2 | 2017-09-06 06:04:53,977 - utils - DEBUG - ArgumentParser: Namespace(batch_size=64, checkpoint_path='checkpoints/keras_tinyshakespeare/model.hdf5', clip_norm=5.0, drop_rate=0.0, embedding_size=32, learning_rate=0.001, main=, num_epochs=32, num_layers=2, restore=False, rnn_size=128, seq_len=64, text_path='data/tinyshakespeare.txt') 3 | 2017-09-06 06:04:53,979 - __main__ - INFO - corpus length: 1115394. 4 | 2017-09-06 06:04:53,979 - __main__ - INFO - building model: batch_size=64, seq_len=64, vocab_size=98, embedding_size=32, rnn_size=128, num_layers=2, drop_rate=0.0, learning_rate=0.001, clip_norm=5.0. 5 | 2017-09-06 06:04:54,500 - __main__ - INFO - model saved: checkpoints/keras_tinyshakespeare/model.hdf5. 6 | 2017-09-06 06:04:54,500 - __main__ - INFO - building inference model. 7 | 2017-09-06 06:04:57,378 - __main__ - INFO - start of training. 8 | 2017-09-06 06:04:57,378 - utils - INFO - number of batches: 272. 9 | 2017-09-06 06:04:57,379 - utils - INFO - effective text length: 1114112. 10 | 2017-09-06 06:04:57,379 - utils - INFO - x shape: (64, 17408). 11 | 2017-09-06 06:04:57,543 - utils - INFO - y shape: (64, 17408, 98). 12 | 2017-09-06 06:07:42,915 - __main__ - INFO - epoch: 0, duration: 165s, loss: 3.32422. 13 | 2017-09-06 06:07:43,000 - __main__ - INFO - generating 512 characters from top 10 choices. 14 | 2017-09-06 06:07:43,000 - __main__ - INFO - generating with seed: "e thy exile. Think with thyself 15 | ". 16 | 2017-09-06 06:07:44,067 - __main__ - INFO - generated text: 17 | e thy exile. Think with thyself 18 | 19 | Ahr oese 20 | Aah oi e arod ho oe re n ehi s shes h h hai eesoe t se eeare 21 | tiireatha oastt reeis ned to rs 22 | 23 | Soeae oe hrnttse tiiohodosoi nt 24 | ho 25 | oooh aesssedsi s e oi o hoa 26 | ouotrii ea 27 | 28 | I 29 | 30 | TN 31 | SR 32 | NOA:AI nea oerdsaisoesa i ha sone nesnarnn se ne etto enn n iati heah sheeeo e hi soeo nrhs no ret rhh he 33 | hee aonnrisatrt airoen so sr to r rte ttsiiass an a e nsiahi soeahoi teaseonnr tsiot tiiheeseri srtiosoro ha hetoi hi rh iiai e ei aoetoshoneeee 34 | oas 35 | als 36 | 37 | S 38 | 39 | E 40 | A::RNI 41 | T:orse tt siti eonre r e 42 | 43 | 2017-09-06 06:10:28,965 - __main__ - INFO - epoch: 1, duration: 164s, loss: 2.57598. 44 | 2017-09-06 06:10:28,967 - __main__ - INFO - generating 512 characters from top 10 choices. 45 | 2017-09-06 06:10:28,968 - __main__ - INFO - generating with seed: "an". 46 | 2017-09-06 06:10:29,666 - __main__ - INFO - generated text: 47 | anas set mot mracet ind telid sor mhous a crate ongeensed nheud seinr ald wit to therete soe besiss wert ot ans in boug thare, 48 | A ferons shere, 49 | Be shat wavels wom ole 50 | Thit trir ohes sor bunl, 51 | Ho hit osd ind of indeu simers, 52 | Triu 53 | In warts coror ty isdsord, a wone it tol borsinhot to wy cintes an ord, sous samer, the shild in tho hum of bhass anlinlid the mash te ar muceend, thels trhee id he cy mis hut is has tor thillire shis 54 | to sol an heil to he afest berle mim wer sor that to weige andice, honds sery am weri 55 | 56 | 2017-09-06 06:13:13,500 - __main__ - INFO - epoch: 2, duration: 163s, loss: 2.27517. 57 | 2017-09-06 06:13:13,502 - __main__ - INFO - generating 512 characters from top 10 choices. 58 | 2017-09-06 06:13:13,503 - __main__ - INFO - generating with seed: "ee to his grace! 59 | Thou wast the p". 60 | 2017-09-06 06:13:14,218 - __main__ - INFO - generated text: 61 | ee to his grace! 62 | Thou wast the pewt wo ther wruware 63 | Tith, whouss'r thas srict bos thee tome 64 | Wate ceirt tith to bace hams, mor on motort and, I thin afdes. 65 | I bade 66 | I there she hink him anlord tree, as matoes sussent on 67 | Teol aghelt 68 | To pane than hame on i maslise tale if ast is in. 69 | That metens, bass as hurg sheard srached shy se in ass, thim hirss if war and afinfer 70 | He foun sise ste wethan fing seict shas are hear. 71 | 72 | GUSORRAO:: 73 | Band sranlat thy metend me thy a dy irdor fouginf hears ofssich oped, 74 | A the cowite ten sile on and, 75 | Mame anlory feor 76 | 77 | 2017-09-06 06:15:58,050 - __main__ - INFO - epoch: 3, duration: 163s, loss: 2.12948. 78 | 2017-09-06 06:15:58,053 - __main__ - INFO - generating 512 characters from top 10 choices. 79 | 2017-09-06 06:15:58,053 - __main__ - INFO - generating with seed: "no k". 80 | 2017-09-06 06:15:58,741 - __main__ - INFO - generated text: 81 | no kour, 82 | Burding, a perenst so thas teat oll the banch herings sincoether: 83 | Af theur our wrougher: and famen so catterted her a say. 84 | 85 | BERHOCDA:: 86 | I har in high wo hour thou mich. 87 | 88 | SEMONTELI II: 89 | Ay soenland my wilk anfung ald hond, my hertind? 90 | 91 | KiRINA: 92 | If whe wroer, af whas beith ham that the muse. 93 | And thy, torth this bot sonsios head, 94 | The mall time sham or along him os to tild wher hore; 95 | Ther oll faset ant, so dor and on hrang tall the hy hird a pornains. 96 | 97 | DUELEL: 98 | Nounder it yor the leroeds; 99 | And shou fangets by a 100 | 101 | 2017-09-06 06:18:42,346 - __main__ - INFO - epoch: 4, duration: 163s, loss: 2.01775. 102 | 2017-09-06 06:18:42,348 - __main__ - INFO - generating 512 characters from top 10 choices. 103 | 2017-09-06 06:18:42,348 - __main__ - INFO - generating with seed: "f conscience and". 104 | 2017-09-06 06:18:43,049 - __main__ - INFO - generated text: 105 | f conscience and fird: 106 | But id and amle, thy, hear my ly 107 | Shy benary me she brand. 108 | 109 | KRLESNA: 110 | What har stard wile thould hy cirsur. 111 | I wart sule of and on nislind. 112 | 113 | SLERDO: 114 | O will wore she wome: they at mather forsur hardily 115 | To fnare sart apter this soringels of hour. 116 | 117 | KROLHEL: 118 | That ange my my wrume twelled an being stains as houds 119 | The poot and thetersh a feake. I did trowe: 120 | And he hans mace bigh. 121 | 122 | Frescord: 123 | Furdore sonds buther, and in him ther his 124 | And tither alond and sich 125 | Thave wis apthing for, that stonged, 126 | Ard that beting 127 | 128 | 2017-09-06 06:21:27,339 - __main__ - INFO - epoch: 5, duration: 164s, loss: 1.93116. 129 | 2017-09-06 06:21:27,341 - __main__ - INFO - generating 512 characters from top 10 choices. 130 | 2017-09-06 06:21:27,341 - __main__ - INFO - generating with seed: "late hath beat her husband 131 | And n". 132 | 2017-09-06 06:21:28,054 - __main__ - INFO - generated text: 133 | late hath beat her husband 134 | And not wore shil her then me bus suld 135 | His ongry, so, have forfer batior 136 | In ofter, that do trid. 137 | 138 | GRALIO: 139 | O mance that hin fors, 140 | The thou, a beaster, and so be the dovel asserdord 141 | And but so, her thee so ape and to she sitses the wish. What is as 142 | The the srow 143 | As not hans, whlray she thnsell of fisting masher, 144 | And noth swere the proon then my hows. 145 | 146 | PRAMIO: 147 | May, in and my and make 148 | If in you my sorong we wo congedter, 149 | Som stees to dichion on tuneming to sume; 150 | Thou moss hever himes such ant apor. 151 | 152 | KADS RICHARUS: 153 | Wh 154 | 155 | 2017-09-06 06:24:13,946 - __main__ - INFO - epoch: 6, duration: 165s, loss: 1.86212. 156 | 2017-09-06 06:24:13,948 - __main__ - INFO - generating 512 characters from top 10 choices. 157 | 2017-09-06 06:24:13,948 - __main__ - INFO - generating with seed: "l hath held them". 158 | 2017-09-06 06:24:14,640 - __main__ - INFO - generated text: 159 | l hath held them, swe 160 | perth be asce be hearly warn. 161 | For sented my shall. 162 | 163 | PELLANE: 164 | War, but an tresungind of cot my trim. 165 | 166 | GROMIUS: 167 | And sleves to sut 168 | We comouting malled frothil womes, 169 | The precenent treed it of thee and 170 | conter busse of tellen ow a my. 171 | Whose somy thit oford till bid with thy coud, 172 | But as to home hore, sor you sead: 173 | Hay, what thilk to wurk, spore wourth these thach frase 174 | The sighe in the actind: worn that witeld on seem, 175 | To say fon will his a fid my deard 176 | What mutter that hive a were thee is twint, 177 | And wiste 178 | 179 | 2017-09-06 06:26:57,737 - __main__ - INFO - epoch: 7, duration: 163s, loss: 1.80388. 180 | 2017-09-06 06:26:57,739 - __main__ - INFO - generating 512 characters from top 10 choices. 181 | 2017-09-06 06:26:57,740 - __main__ - INFO - generating with seed: "its sweet air: thence I have fol". 182 | 2017-09-06 06:26:58,440 - __main__ - INFO - generated text: 183 | its sweet air: thence I have fold: 184 | I'lf sho sills the deich then afpust, 185 | The polision wanded the and slay to prictio, 186 | Hear the sore too ackant within. 187 | 188 | LATEN OFh: 189 | Boved heartile fillit the dowand 190 | The stied on was thus not filloth of moring 191 | That him, what a word suld's father thall, 192 | But be trim of besendedss, wwo lown, thou and for: 193 | Hart and wich to be to the sook, 194 | Betticimus's forning beting: when she 195 | Turest how sanion sower hander be mery her son, 196 | This sook his brother as the how woust, 197 | Why sounty all your my lever wis my mang 198 | Forfees in 199 | 200 | 2017-09-06 06:29:42,549 - __main__ - INFO - epoch: 8, duration: 164s, loss: 1.75605. 201 | 2017-09-06 06:29:42,551 - __main__ - INFO - generating 512 characters from top 10 choices. 202 | 2017-09-06 06:29:42,552 - __main__ - INFO - generating with seed: "on Saint Lambert's day: 203 | There sh". 204 | 2017-09-06 06:29:43,258 - __main__ - INFO - generated text: 205 | on Saint Lambert's day: 206 | There she ming so cander, I waighe bun,-lose 207 | That it here sactio his staintt the hangiend, 208 | What me this wis told; then whon a mighang breare it; 209 | In the see of there at trait in the frimne, 210 | And which our with then been that not of the men 211 | Blaiking arritsse hus in too worm, to mere, 212 | And his him, af sended mints on the fortiar for 213 | Lord bloos, with him frowing of this hear. 214 | I loventince on, is a think a most. 215 | For her as with mance shall a both. 216 | 217 | BRUMIO: 218 | Which I swould mad be, he gofn bees; 219 | Be cinds and look to bitor an 220 | 221 | 2017-09-06 06:32:26,552 - __main__ - INFO - epoch: 9, duration: 163s, loss: 1.71598. 222 | 2017-09-06 06:32:26,554 - __main__ - INFO - generating 512 characters from top 10 choices. 223 | 2017-09-06 06:32:26,555 - __main__ - INFO - generating with seed: "ness". 224 | 2017-09-06 06:32:27,240 - __main__ - INFO - generated text: 225 | nesserier: if sin sceaps. 226 | 227 | SLOMAS: 228 | Ald with mone that him, brastly will me as hesh 229 | And bean till hrices, but be alite 230 | In thine all, but it, and streashs's father, 231 | And bether tome shake in sweep tonest. 232 | 233 | KING EDWARD RIV: 234 | Sirs he ham the laice, fend, as was, here an is towid? 235 | 236 | KLOLEN: 237 | For you will with him all forsh all too mean 238 | That he with a procious as therefould it 239 | The poor, have is this wime on what name 240 | As ispallows and they mouldely stondss, 241 | That lose my saught is to sees. 242 | 243 | PORCINIA: 244 | I am, 245 | That the may a s 246 | 247 | 2017-09-06 06:35:11,035 - __main__ - INFO - epoch: 10, duration: 163s, loss: 1.68195. 248 | 2017-09-06 06:35:11,037 - __main__ - INFO - generating 512 characters from top 10 choices. 249 | 2017-09-06 06:35:11,038 - __main__ - INFO - generating with seed: "lo". 250 | 2017-09-06 06:35:11,716 - __main__ - INFO - generated text: 251 | lo,, I do be with your: 252 | Who chis of breathed hour, and he his plosen it? 253 | 254 | CORIOLANUS: 255 | Now seed then in such our simshaning will. 256 | Swat me, brousce with that thich a propuin'd at but, 257 | And time the panisint of to planter beers? 258 | I'll we would but heare they bugt,' well's the chreash 259 | Sercheriture to most in be con to fight? 260 | 261 | GRUCHESSARO: 262 | So, better? I'- 263 | All wompor any had as in were are have boys. 264 | 265 | SIPINAS: 266 | I truths, follon to thee how the wrose mucl, 267 | Tirpony wouson shall becows firpory, 268 | For it his done of her spe 269 | 270 | 2017-09-06 06:37:55,320 - __main__ - INFO - epoch: 11, duration: 163s, loss: 1.65435. 271 | 2017-09-06 06:37:55,323 - __main__ - INFO - generating 512 characters from top 10 choices. 272 | 2017-09-06 06:37:55,323 - __main__ - INFO - generating with seed: "e enemies, 273 | How they at Pomfret b". 274 | 2017-09-06 06:37:56,031 - __main__ - INFO - generated text: 275 | e enemies, 276 | How they at Pomfret bean them shesurity: 277 | Your mone heind of thing would, and wis words 278 | What will not withers of be straid: 279 | Tide a stind, for have the concenter: 280 | And befence, scars of your wange sir! 281 | Make, fare to a blest your brocious, and 282 | time and both and her aro two three? 283 | 284 | DUKE VINCENTIO: 285 | Hath say 286 | the werch would mosts the mary her brothy 287 | For the will where you dist which him, which but his a process 288 | Hoth is atcles fear hese from at, they my lave the mont 289 | For my-mother thee beckants, brother, and breake 290 | I sair some but-then 291 | 292 | 2017-09-06 06:40:39,961 - __main__ - INFO - epoch: 12, duration: 163s, loss: 1.62964. 293 | 2017-09-06 06:40:39,963 - __main__ - INFO - generating 512 characters from top 10 choices. 294 | 2017-09-06 06:40:39,964 - __main__ - INFO - generating with seed: "rt". 295 | 2017-09-06 06:40:40,644 - __main__ - INFO - generated text: 296 | rt ore, 297 | To the tounted; 298 | But more ipfiring as my made before from the pardast 299 | Whyser hilves in the padce's sick his divest. 300 | 301 | POLCHONIO: 302 | And noble thears and this sword wish him in a follow, 303 | Should no traughtry. His beging: boy's bear me the word 304 | On the stastion; 305 | Mone bid it bather'd mingse in with him fail 306 | The some themself with hease frue tringlys my mistance? 307 | 308 | PETRUCHIO: 309 | Betorrong 310 | Ih, the kneak this too to the deppectitastifs. 311 | I sim, is asten the many as his becart' good; 312 | And my deep in than swrom not indeet 313 | 314 | 2017-09-06 06:43:24,884 - __main__ - INFO - epoch: 13, duration: 164s, loss: 1.60766. 315 | 2017-09-06 06:43:24,886 - __main__ - INFO - generating 512 characters from top 10 choices. 316 | 2017-09-06 06:43:24,887 - __main__ - INFO - generating with seed: "First Co". 317 | 2017-09-06 06:43:25,820 - __main__ - INFO - generated text: 318 | First Coursesty welk: and this do 319 | All our sorstise as himes to sid. 320 | Brecy is much; when the hentelf: for you will. 321 | 322 | LOLEN EETER: 323 | No his battle stand have splive she wither 324 | Bost becuckiar. 325 | 326 | POLIXENES: 327 | I have how then, to truen you with bloods with 328 | pirse's bade honour her says afty more. 329 | 330 | LORS ESTARD: 331 | Antint on thee. 332 | 333 | CORIOLANUS: 334 | Who most his many from with must some agay sunce 335 | And folled in the sue which hy wrown, I weet aboun 336 | Hear and atself'd and foo cortesters'd my, 337 | Is all, when he staint, tall strown; when, 338 | Frop 339 | 340 | 2017-09-06 06:46:15,612 - __main__ - INFO - epoch: 14, duration: 169s, loss: 1.58838. 341 | 2017-09-06 06:46:15,615 - __main__ - INFO - generating 512 characters from top 10 choices. 342 | 2017-09-06 06:46:15,615 - __main__ - INFO - generating with seed: " I do now, 343 | Taking the measure of". 344 | 2017-09-06 06:46:16,502 - __main__ - INFO - generated text: 345 | I do now, 346 | Taking the measure of her hides of morn to 347 | couls, and you have be honish to to thee, 348 | We must from the last man sacicked betten 349 | Was a brow take as thoughth ware to woulds, 350 | I aw a shall have more man is in homself 351 | Fiven of subpen what a crown: 352 | The prace adays or all sount in shall break; 353 | So and will be nother? thy walty thalk mine, 354 | If what thou angien me, here with minor. 355 | 356 | KING HEWINGEL: 357 | My made, it you, son's befolt save me: with chich. 358 | 359 | First Murperer: 360 | You stass his folt of bried, where I women. 361 | 362 | CORIOLANUS: 363 | There is note alare 364 | 365 | 2017-09-06 06:48:59,854 - __main__ - INFO - epoch: 15, duration: 163s, loss: 1.57066. 366 | 2017-09-06 06:48:59,857 - __main__ - INFO - generating 512 characters from top 10 choices. 367 | 2017-09-06 06:48:59,857 - __main__ - INFO - generating with seed: "other both shall buy this treaso". 368 | 2017-09-06 06:49:00,585 - __main__ - INFO - generated text: 369 | other both shall buy this treason 370 | Shought stadination: and have not to be come; 371 | For in sly senters him be him, orted you have shows we 372 | thy swors to your come. 373 | 374 | ARCHIONE: 375 | O, 376 | saint thing have woed-as the corcheep from fastinion. 377 | I cannen you dows and which all may not have any like 378 | To weeps hither boinituly madit frain, 379 | For thy patily as him, where, he as news, 380 | Fastitates with mance foethan with at his hume 381 | The have most to wited. There's sire as it 382 | I detein you think his bid, I 383 | heaven of my fared flower that spate to that 384 | I suppise. Crucki 385 | 386 | 2017-09-06 06:51:43,934 - __main__ - INFO - epoch: 16, duration: 163s, loss: 1.55416. 387 | 2017-09-06 06:51:43,937 - __main__ - INFO - generating 512 characters from top 10 choices. 388 | 2017-09-06 06:51:43,937 - __main__ - INFO - generating with seed: "They have chose a consul that wi". 389 | 2017-09-06 06:51:44,660 - __main__ - INFO - generated text: 390 | They have chose a consul that wis contruman tuned of there sconfunesce and so him horn'd care 391 | Than he swill of the mean, as so we be three's all? 392 | 393 | DUKE OF YORK: 394 | Sace beary my titer way; 395 | I work that stood their's let sworn to she. 396 | Siles is moceld him tell tame a seen are show 397 | And but there of it him have minger at of them. 398 | 399 | GONZALO: 400 | Of may speak, sir; brigh, he was spriest soon their: 401 | For a would in may speak to thither with more 402 | With shalt the make for his so the death, 403 | That sweet me and we till not these hapsest not his frough. 404 | To the fa 405 | 406 | 2017-09-06 06:54:28,362 - __main__ - INFO - epoch: 17, duration: 163s, loss: 1.54002. 407 | 2017-09-06 06:54:28,364 - __main__ - INFO - generating 512 characters from top 10 choices. 408 | 2017-09-06 06:54:28,365 - __main__ - INFO - generating with seed: " chasing the royal blood 409 | With fu". 410 | 2017-09-06 06:54:29,082 - __main__ - INFO - generated text: 411 | chasing the royal blood 412 | With furmons of arms fry on onesty. 413 | 414 | CORIOLANUS: 415 | I prowal a wear still since too begel same wist, 416 | Which he home to be so slie, my worse 417 | On's she of thou talk you be too for he appiend 418 | Will issorell sisty that if that she worther 419 | I'll not as you. Why hill be boot so suf. 420 | 421 | PRINSARD: 422 | Why, I may hath there. 423 | 424 | CLARENCE: 425 | Well way, I am noble more, blot their pischies of me, 426 | And for thy such of thron to have nobling, 427 | Where is both but to be witch, 428 | If what now, thou chiress, and tray mine insteack; 429 | As this way one brothem 430 | 431 | 2017-09-06 06:57:12,361 - __main__ - INFO - epoch: 18, duration: 163s, loss: 1.5272. 432 | 2017-09-06 06:57:12,364 - __main__ - INFO - generating 512 characters from top 10 choices. 433 | 2017-09-06 06:57:12,364 - __main__ - INFO - generating with seed: "e ". 434 | 2017-09-06 06:57:13,037 - __main__ - INFO - generated text: 435 | e ay for we cannot as sir. 436 | Hore, what we tray almany? 437 | 438 | COMINIUS: 439 | He, then, I'ly we turn if your wife too. Here. 440 | Would too terren soul assance, but thee nobe, 441 | Who pase sunfrice at you? what is the deed; 442 | For the son too noth be misers of act. 443 | 444 | First Serator: 445 | Ho's we meens: it so fancince more sorven 446 | she with the moved most been arims, 447 | To mretched me? I am he watese them bear me. 448 | But hitter, and you well. 449 | 450 | LoRGILO: 451 | If my bong word him help hear, flower apward, 452 | Oor cause of me, to trench the mosty fould 453 | at hanger 454 | 455 | 2017-09-06 06:59:56,587 - __main__ - INFO - epoch: 19, duration: 163s, loss: 1.5155. 456 | 2017-09-06 06:59:56,590 - __main__ - INFO - generating 512 characters from top 10 choices. 457 | 2017-09-06 06:59:56,590 - __main__ - INFO - generating with seed: "l appeared in his lineaments, 458 | Be". 459 | 2017-09-06 06:59:57,304 - __main__ - INFO - generated text: 460 | l appeared in his lineaments, 461 | Beie this most sheid. 462 | 463 | PROLUS: 464 | Why, the shue me me. I am so fault before of your 465 | time more one that wind thy live was a fiers? 466 | Fattens with them, sace they say, to their sea their how 467 | The wake of cilifude brother shore sooly 468 | And could sings it if to hie incands from the chase? 469 | But teed you will men will my honours, been burmon me 470 | To mistives, too longen blood to me be confors, 471 | Or your comment, tray, I would her fould her 472 | howsed, my sees thy confess for you: make, 473 | An see they most matties, what have well, and 474 | 475 | 2017-09-06 07:02:40,814 - __main__ - INFO - epoch: 20, duration: 163s, loss: 1.50473. 476 | 2017-09-06 07:02:40,816 - __main__ - INFO - generating 512 characters from top 10 choices. 477 | 2017-09-06 07:02:40,816 - __main__ - INFO - generating with seed: "ve". 478 | 2017-09-06 07:02:41,501 - __main__ - INFO - generated text: 479 | veare, but mighted as thy shame; I have a bode and will been, where then straigos there. 480 | 481 | POMPEY: 482 | Host set we hops, and spay, 483 | And bid have them to dreed me to show the proud. 484 | 485 | DUKE OF AMUVE: 486 | We much in at his call treat him, 487 | This wert it, and serve as sluck to come? 488 | 489 | LEONTES: 490 | What it make not time but the mad, 491 | I chooder and his partise my man aboun will's father; 492 | There's that I may would be need shall so perish'd; 493 | Yet strom the comes sone, throw, for the widgs 494 | But as he will wilt by all, it mull bring 495 | Mistles 496 | 497 | 2017-09-06 07:05:25,342 - __main__ - INFO - epoch: 21, duration: 163s, loss: 1.49491. 498 | 2017-09-06 07:05:25,345 - __main__ - INFO - generating 512 characters from top 10 choices. 499 | 2017-09-06 07:05:25,345 - __main__ - INFO - generating with seed: "t,". 500 | 2017-09-06 07:05:26,009 - __main__ - INFO - generated text: 501 | t,-RSRXFWIEVUDCD T7Q>TQ3D24D7)V)KIKDVGVYD3SVUDU3D_UD@D|DUCQ^D@D4D_>)VYDD|D|D@Q)DD_[`D@VISD)CD``UDUD|C%V>`QG4C`D>D)D^[Q`D^4D@)`UGCSD_\[{G\VID`VD_VY|4DkCDSSGdK}GD)C)D{\VDUVYVGDTV}TDCD{Qd|G\)SGD{D|[UG|)))D UDD7`K)D4QUCD7UD)4VIdYCTD3dD_)D`7UT[DUC3DkNNNNSDS@Kurses and chope and bill the brother single have and atter wind sworn, sir, noble fullowings that's heeds o'erifess she's wonder'd more.' 502 | 503 | Femverd Morder: 504 | Will shory she down woe, 505 | Who home! 506 | 507 | POMPEY: 508 | Yea to mean the presence. 509 | 510 | CLAUDIO: 511 | How thou which swee tidle 512 | 513 | 2017-09-06 07:08:09,458 - __main__ - INFO - epoch: 22, duration: 163s, loss: 1.4856. 514 | 2017-09-06 07:08:09,460 - __main__ - INFO - generating 512 characters from top 10 choices. 515 | 2017-09-06 07:08:09,461 - __main__ - INFO - generating with seed: "when children are toward. 516 | 517 | LUCEN". 518 | 2017-09-06 07:08:10,177 - __main__ - INFO - generated text: 519 | when children are toward. 520 | 521 | LUCENTIO: 522 | The worse sprivel, and such than, sir to the spail 523 | Am in'ting, as he servine whose word: 524 | We are never sals his brief and plandon hears 525 | Than this manide and have have no spirit; 526 | In thither; where's watk your bold so semming 527 | And summs said a san then mettom his lies; 528 | With holy what thy serrity. 529 | 530 | PETRUCHIO: 531 | Ay, thither, that we hour, that they that: there's it? 532 | 533 | GLOUCESTER: 534 | He did were for whils a hearful sorce, 535 | And, as me say what, heive and not and fortwer to my some! 536 | But my bonder, his brother sleed st 537 | 538 | 2017-09-06 07:10:53,281 - __main__ - INFO - epoch: 23, duration: 163s, loss: 1.47645. 539 | 2017-09-06 07:10:53,284 - __main__ - INFO - generating 512 characters from top 10 choices. 540 | 2017-09-06 07:10:53,284 - __main__ - INFO - generating with seed: "ai". 541 | 2017-09-06 07:10:53,959 - __main__ - INFO - generated text: 542 | aiu 543 | listy me as her his friends of man; 544 | Then him stay a bear his most better wish a 545 | provish'd stranges fashily of dear the hearf. 546 | 547 | GLOMIRELASDBRONE: 548 | O sistio his desprike as in thip honour. 549 | 550 | DURENS: 551 | And the monsty beeced to her lord: 552 | For nagtasted-- 553 | Which that it hears a mony shades state, 554 | And I'll father with wed mine sinces, of 555 | this pretton's bates again so his bloody: 556 | As not stent. To the strience, fatiend, it waith. 557 | 558 | ROMEO: 559 | And I am no more is him brother'd, when have you may 560 | contening hartly mouth: would 561 | 562 | 2017-09-06 07:13:37,447 - __main__ - INFO - epoch: 24, duration: 163s, loss: 1.4687. 563 | 2017-09-06 07:13:37,450 - __main__ - INFO - generating 512 characters from top 10 choices. 564 | 2017-09-06 07:13:37,450 - __main__ - INFO - generating with seed: "claim'd about. 565 | 566 | Pedant: 567 | Alas! si". 568 | 2017-09-06 07:13:38,166 - __main__ - INFO - generated text: 569 | claim'd about. 570 | 571 | Pedant: 572 | Alas! sir; how now, which inselves sword thy someting: 573 | We are in what he would my lords, 574 | That said to condem, nor blood to cheet me, 575 | To chide the bloody of you; 576 | Thou shal is and put two all by his low'd beh, 577 | Madam, but so sword the commess shill accons. 578 | Then tyracts shalt be friar, with your wishined, 579 | And with him too my bedough, is all him! Came it 580 | sicling highant with offenties of breath. 581 | Be with you all him honour moved, whose 582 | this for thy hopon, be myan, as so his 583 | tarry with a seeming when he bark the deed: 584 | Wor 585 | 586 | 2017-09-06 07:16:22,027 - __main__ - INFO - epoch: 25, duration: 163s, loss: 1.46138. 587 | 2017-09-06 07:16:22,030 - __main__ - INFO - generating 512 characters from top 10 choices. 588 | 2017-09-06 07:16:22,030 - __main__ - INFO - generating with seed: "o ". 589 | 2017-09-06 07:16:22,709 - __main__ - INFO - generated text: 590 | o eer not withan for 591 | think-bold of the peirour singuty and for with 592 | hearting of me hath too not to most buckle 593 | Much she shall so brother, to him te'll neby 594 | from mins has thy confess in my son him and parse 595 | To my father; if he would be why tongue, 596 | That wherefore as yet is with years treit; 597 | Thou causin which worm the soul, frest to more fray 598 | Women to him friends of this as him; with have him. 599 | 600 | LEONTES: 601 | I will near to cannot to his cround; 602 | For what seeve along sir, our wain to-day? 603 | They that we so blood her fort 604 | 605 | 2017-09-06 07:19:06,399 - __main__ - INFO - epoch: 26, duration: 163s, loss: 1.45395. 606 | 2017-09-06 07:19:06,401 - __main__ - INFO - generating 512 characters from top 10 choices. 607 | 2017-09-06 07:19:06,401 - __main__ - INFO - generating with seed: "randsire's bones, 608 | And by the roy". 609 | 2017-09-06 07:19:07,126 - __main__ - INFO - generated text: 610 | randsire's bones, 611 | And by the royers than the present as black 612 | With shus to her and tender. To be hitchery. 613 | But my last a say is all to sea thy spice 614 | Fierth with stines friends, the mosh crown, their 615 | shall we suit of the barest supfice of made: 616 | But sir and that I abplent, his bones! 617 | Why sat thring in almassion; and be my souls 618 | As now. 619 | Hone to we stars where a handle in their hand? 620 | O sence twy tongue with his life for his most. 621 | 622 | KING: 623 | Are too, whom I shall appeal against belife 624 | The mouse will marrey wrong. 625 | 626 | MARIANA: 627 | Well, then you thought t 628 | 629 | 2017-09-06 07:21:50,143 - __main__ - INFO - epoch: 27, duration: 163s, loss: 1.44733. 630 | 2017-09-06 07:21:50,146 - __main__ - INFO - generating 512 characters from top 10 choices. 631 | 2017-09-06 07:21:50,146 - __main__ - INFO - generating with seed: "on. 632 | The game was ne'er so fair, ". 633 | 2017-09-06 07:21:50,865 - __main__ - INFO - generated text: 634 | on. 635 | The game was ne'er so fair, thou hast that it mely spaid, 636 | The sovereicion in thy molenour'd one; 637 | And then my colour true befingried's strity: it set 638 | Sets me nor brother of cassoms of worshe. 639 | Sweet she may would be mory to bury in me 640 | And be old being on hand man my lord: 641 | She confunt him take me sore on time? O woe! 642 | O calalmood ship heaving to murds how. 643 | 644 | BRUTUS: 645 | I am too. Be sheples will she baison'd in weech, 646 | That with the master in my pleasure. 647 | But to my backs of mine for which my lady, 648 | And it beture, banish'd assister. 649 | And such him, 650 | 651 | 2017-09-06 07:24:34,624 - __main__ - INFO - epoch: 28, duration: 163s, loss: 1.44122. 652 | 2017-09-06 07:24:34,627 - __main__ - INFO - generating 512 characters from top 10 choices. 653 | 2017-09-06 07:24:34,627 - __main__ - INFO - generating with seed: "ce extra". 654 | 2017-09-06 07:24:35,325 - __main__ - INFO - generated text: 655 | ce extraught of a ballow, such thy banishment my lord comforized to thee and seems 656 | I triving founds, and, so purson you. 657 | 658 | COMINIUS: 659 | Has he how 660 | thee, he stand in thy brit to the misgrem: 661 | And so. Sir, you. Allos shill be murties when that 662 | apthesed the strim and slutter to your strums, 663 | Busing a crest and comes his chirt was blown. 664 | Sutcess you trown sister of hides for wese her, 665 | A supforminer'd to thee, faughter and amblenter, 666 | And seek thy words our his subfers infurrer, 667 | His lasian with him saciane to wed you 668 | His so th 669 | 670 | 2017-09-06 07:27:18,817 - __main__ - INFO - epoch: 29, duration: 163s, loss: 1.43527. 671 | 2017-09-06 07:27:18,819 - __main__ - INFO - generating 512 characters from top 10 choices. 672 | 2017-09-06 07:27:18,819 - __main__ - INFO - generating with seed: "tning? O my love! my wife! 673 | Death". 674 | 2017-09-06 07:27:19,542 - __main__ - INFO - generated text: 675 | tning? O my love! my wife! 676 | Death me, that you have touch. That nothin the troubled; 677 | And is your blows: and that that thou hast some 678 | Beauther whether the saved witll amprant. 679 | His windon and this must be colaurs, 680 | What she'd the faith, hath as hour for we wish 681 | Which hence of showarved to with our dishonours, 682 | I should all thou are will savise hath moution: 683 | Who piesious crown true: for my mutee mad. 684 | 685 | BIONDELLO: 686 | Helves beins of hath stand to men have the presence, 687 | Shade my heart wakeroun mischithed as feise, 688 | The servantle stand by the pearls to 689 | 690 | 2017-09-06 07:30:03,467 - __main__ - INFO - epoch: 30, duration: 163s, loss: 1.42992. 691 | 2017-09-06 07:30:03,469 - __main__ - INFO - generating 512 characters from top 10 choices. 692 | 2017-09-06 07:30:03,470 - __main__ - INFO - generating with seed: "th". 693 | 2017-09-06 07:30:04,168 - __main__ - INFO - generated text: 694 | th, 695 | My fie for too with his forgity. 696 | 697 | LASTRYOS YO: 698 | What, what sent me, sin'd a fearfh, for you, may come, 699 | To born brown'd my firm'd! Think him for alive 700 | Anow the will with borning, too, for this all 701 | Is but all madner: if you how at a faitine. 702 | 703 | DUKE OF YrRELE: 704 | I am, how to main he my sainty for a foor any trems, 705 | As hear new was heart. 706 | 707 | First Girdurer: 708 | Why what would this appearing are teinty 709 | Many shurts tenders streck amontand. 710 | 711 | LADY GREY: 712 | I will come not. 713 | 714 | SAMPS: 715 | It hears a child; how mainted to me, to this s 716 | 717 | 2017-09-06 07:32:47,136 - __main__ - INFO - epoch: 31, duration: 162s, loss: 1.42505. 718 | 2017-09-06 07:32:47,139 - __main__ - INFO - generating 512 characters from top 10 choices. 719 | 2017-09-06 07:32:47,139 - __main__ - INFO - generating with seed: " answer:". 720 | 2017-09-06 07:32:47,837 - __main__ - INFO - generated text: 721 | answer: 722 | Thus? 723 | 724 | Bey Here: 725 | She due false, his death is but sluke a shrese. 726 | Is the fair all perturitance! 727 | 728 | DUKE OF RALELE: 729 | O well as I'll be heavies to amongs, 730 | And hang all thy since, 731 | And thou thirps then well our way a pierfers. 732 | 733 | KING EDWARD IV: 734 | And he burnst to do her childen and my heard. 735 | 736 | GONZALO: 737 | Say, and I,, as all and at his honoursed a widor 738 | Ihmor and storn'd me blessed this full of classe? 739 | There's a first should be must. 740 | 741 | DUKE VINCENTIO: 742 | As he is faith. 743 | 744 | PRANER: 745 | And so murke them before help and beftard 746 | The 747 | 748 | 2017-09-06 07:32:47,839 - __main__ - INFO - end of training, duration: 5270s. 749 | 2017-09-06 07:32:47,842 - __main__ - INFO - generating 1024 characters from top 3 choices. 750 | 2017-09-06 07:32:47,842 - __main__ - INFO - generating with seed: " h". 751 | 2017-09-06 07:32:49,218 - __main__ - INFO - generated text: 752 | habaning to the souls of heart, as your since. 753 | 754 | CORIOLANUS: 755 | Why! where's will stay with her, and 756 | Till you shalt but the complemined him. 757 | What is not so such a woman. 758 | 759 | LADY ANNE: 760 | I have to should they see the comes at me 761 | To sainted to them things to the sore with this harders 762 | To see to the world. 763 | 764 | CAPULET: 765 | Though they with a sounds to besiend thee, 766 | And see the world, an one on the cares of her, 767 | And, shall I seem a man shall not be should be so. 768 | 769 | BRATUL: 770 | I was too much of the sea and sent to 771 | hath state, the couss them thoughts of his soul, 772 | And shall not they should here will be so sound to see him, 773 | Whiles, between this both of the succession 774 | And with the strom and sound that hate this sound 775 | As some strive a send their women that 776 | I will be so friventy store, ten the sun his hanches, 777 | Which it shall be trine of thy looks, be the commend, 778 | As that I wouldst thou with trutcunion of him. 779 | 780 | PETRUCHIO: 781 | I will now, a seen as thought a stones, 782 | With the way too, should had there in a sore, 783 | That way is the world. Why, then is 784 | -------------------------------------------------------------------------------- /logs/mxnet_model.log: -------------------------------------------------------------------------------- 1 | 2017-09-06 04:02:10,969 - utils - DEBUG - call: mxnet_model.py train --checkpoint=checkpoints/mxnet_tinyshakespeare/model.params --text=data/tinyshakespeare.txt 2 | 2017-09-06 04:02:10,970 - utils - DEBUG - ArgumentParser: Namespace(batch_size=64, checkpoint_path='checkpoints/mxnet_tinyshakespeare/model.params', clip_norm=5.0, drop_rate=0.0, embedding_size=32, learning_rate=0.001, main=, num_epochs=32, num_layers=2, restore=False, rnn_size=128, seq_len=64, text_path='data/tinyshakespeare.txt') 3 | 2017-09-06 04:02:10,971 - __main__ - INFO - corpus length: 1115394. 4 | 2017-09-06 04:02:10,996 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 5 | 2017-09-06 04:02:11,347 - __main__ - INFO - start of training. 6 | 2017-09-06 04:02:11,351 - utils - INFO - number of batches: 272. 7 | 2017-09-06 04:02:11,351 - utils - INFO - effective text length: 1114112. 8 | 2017-09-06 04:02:11,352 - utils - INFO - x shape: (64, 17408). 9 | 2017-09-06 04:02:11,352 - utils - INFO - y shape: (64, 17408). 10 | 2017-09-06 04:05:58,835 - __main__ - INFO - epoch: 1, duration: 227s, loss: 3.10827. 11 | 2017-09-06 04:05:59,112 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 12 | 2017-09-06 04:05:59,113 - __main__ - INFO - generating 512 characters from top 10 choices. 13 | 2017-09-06 04:05:59,114 - __main__ - INFO - generating with seed: "er". 14 | 2017-09-06 04:06:01,189 - __main__ - INFO - generated text: 15 | erre tared are at ant mand hh sans od, 16 | Io mous t aolssadd in war bad her bente, anl teid, nings tim more, toon tise shirise br ad hhe no te burosd te ans tiss tho thore hanr 17 | Shere cisrerdes be oat asrened weing, se ace tot su sind arl the be memeted wh add hos belthe monn at tonn tithad it worent sougo he, hin the sorer werhel sonsid the saty hru tous 18 | 19 | nam fosde whur wit, 20 | Hhed why ther waund. 21 | 22 | 23 | SROTIN: 24 | Eor tlend ite mhit, 25 | Bhot ty it whe soel wentis hy sont, sety ase to amlols hors hius, 26 | 27 | ninens id heut hout t 28 | 29 | 2017-09-06 04:09:47,252 - __main__ - INFO - epoch: 2, duration: 226s, loss: 2.29085. 30 | 2017-09-06 04:09:47,715 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 31 | 2017-09-06 04:09:47,717 - __main__ - INFO - generating 512 characters from top 10 choices. 32 | 2017-09-06 04:09:47,717 - __main__ - INFO - generating with seed: "pe of 33 | an". 34 | 2017-09-06 04:09:49,352 - __main__ - INFO - generated text: 35 | pe of 36 | and, sule mendy, 37 | Fy thou me ad shim me wandis, forger 38 | In a hingteld hy whals that then ar sore, 39 | Thit wash thee mealer wereneld, beirs; my tithe fome munt thich the caresime: 40 | And dild im ond antimes hateln as of hloud froon as 41 | Fare to busteets shy will his's winlt 42 | As that il to mency 43 | hinged is shenders as hith. 44 | Anll thith hoo breld and his' hay buse: it tor to the tarle maghad oprone bure, 45 | Me ant hror il tour ane sild ip ham he in bass te ane bage to thave tins to mure, 46 | Thes womanlo fare to with 47 | I barlo seot t 48 | 49 | 2017-09-06 04:13:37,572 - __main__ - INFO - epoch: 3, duration: 228s, loss: 2.07992. 50 | 2017-09-06 04:13:37,952 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 51 | 2017-09-06 04:13:37,954 - __main__ - INFO - generating 512 characters from top 10 choices. 52 | 2017-09-06 04:13:37,954 - __main__ - INFO - generating with seed: "rself. 53 | All this I know; and to t". 54 | 2017-09-06 04:13:39,643 - __main__ - INFO - generated text: 55 | rself. 56 | All this I know; and to tandall thite, me me binch to 57 | so hy to suil and my hided this mome, 58 | Thing moud thee alos saster.- 59 | Hich mimed youl out ton soors in hive a hear; 60 | Miviog not harker. 61 | 62 | Kinds Vans: 63 | Cole is igon though ildalidy, 64 | The linds of hing me hacle,, sith afanom: 65 | To wore betacile to buce if your mroch, mone ame to wise hossss. 66 | Whit ond hast ofe'd sours that he thie batere 67 | And one the lest is nome buttis, mowendisgis, 68 | Sindes apour ar the detert them a dore, thee tither; 69 | Housh woust hy srome, wice, thou serlent at wink, bacte 70 | 71 | 2017-09-06 04:17:28,357 - __main__ - INFO - epoch: 4, duration: 228s, loss: 1.94899. 72 | 2017-09-06 04:17:28,828 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 73 | 2017-09-06 04:17:28,830 - __main__ - INFO - generating 512 characters from top 10 choices. 74 | 2017-09-06 04:17:28,830 - __main__ - INFO - generating with seed: "g fire 75 | That stag". 76 | 2017-09-06 04:17:30,479 - __main__ - INFO - generated text: 77 | g fire 78 | That stagkt, and frese. 79 | 80 | LENINUS: 81 | As Inelow'd the tay ool thim have to, 82 | And the him of youd chood's arcy him; 83 | And to to boots she avery and as appeld 84 | At and whee to hear theust bees to, 85 | To lomly buting bud a hoors with wist bod, 86 | And and hings allest amrangors to thite to mich; 87 | And a coomed and wore a have my sair, 88 | Thou coonder that he winks wirt, wame to lim bidssed 89 | To lusing frigut ass and hy toneneds, his day. 90 | 91 | LINH: 92 | Buster, her deak hask of heir with she bost. 93 | 94 | LOTENLI: 95 | It my sept and hast hastily strrefie. 96 | 97 | PROR 98 | 99 | 2017-09-06 04:21:16,415 - __main__ - INFO - epoch: 5, duration: 225s, loss: 1.85166. 100 | 2017-09-06 04:21:16,650 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 101 | 2017-09-06 04:21:16,652 - __main__ - INFO - generating 512 characters from top 10 choices. 102 | 2017-09-06 04:21:16,652 - __main__ - INFO - generating with seed: "n: 103 | Indeed, he should be a footma". 104 | 2017-09-06 04:21:18,356 - __main__ - INFO - generated text: 105 | n: 106 | Indeed, he should be a footmance of busce 107 | If wime. 108 | 109 | BUKENTES: 110 | My soon and hearers, he mutt it thine. 111 | 112 | GRUSCERY AD ENRER 113 | Mone tell in when, I this and thighther, 114 | As it your giclior: by thounger's. 115 | 116 | BRUCIO: 117 | Say this sursaly it not treakss whall, 118 | Show, my chailed. 119 | 120 | LIONE: 121 | Both no sie, by world, shont my winks; 122 | Swilk you thee, to woust on bains sppath, 123 | I teutter with wemt of livened with wame. 124 | 125 | LENES: 126 | Beding thy consores are to him face of somen. 127 | 128 | Citest: 129 | Too dingon be the thanles and thene and fous are 130 | By soust, have my but sie. 131 | 132 | LEUSENY: 133 | 134 | 2017-09-06 04:25:08,045 - __main__ - INFO - epoch: 6, duration: 229s, loss: 1.77793. 135 | 2017-09-06 04:25:08,485 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 136 | 2017-09-06 04:25:08,487 - __main__ - INFO - generating 512 characters from top 10 choices. 137 | 2017-09-06 04:25:08,487 - __main__ - INFO - generating with seed: " s". 138 | 2017-09-06 04:25:10,141 - __main__ - INFO - generated text: 139 | spreise, 140 | To bugnert where and and thouge his soul, 141 | And nade him her his dake to provess, 142 | Wifh thou all your prask as with him. 143 | I, her mard, and the parten for the mines 144 | To spanteroul her tell apong as hour 145 | The condans so, spay with till to sir; 146 | Hoth tas his mady, brece will helval. 147 | When all to my mood a sheaved as hat as: 148 | Will had he so son, that, and hise sack ap, 149 | And with the plicked, fare one mine. 150 | 151 | LUCKING EVWaR: 152 | The lance were being truep's arming, to-bokn, 153 | And not now my hongay? 154 | 155 | CROMENS: 156 | And I have so 157 | 158 | 2017-09-06 04:28:53,509 - __main__ - INFO - epoch: 7, duration: 223s, loss: 1.72041. 159 | 2017-09-06 04:28:53,797 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 160 | 2017-09-06 04:28:53,800 - __main__ - INFO - generating 512 characters from top 10 choices. 161 | 2017-09-06 04:28:53,800 - __main__ - INFO - generating with seed: "er my good word nor princely fav". 162 | 2017-09-06 04:28:55,467 - __main__ - INFO - generated text: 163 | er my good word nor princely faves, 164 | But my sorsest wingty she pessing thosh them. 165 | And may those shard bintay a tan te come with 166 | His live of her his fither worsing and sees, 167 | By be madenous werp, I hound you sleed, 168 | All to suppand thisselve and doich it it, 169 | When she stoud of this 170 | In shall bissows so, why fame and twas an son 171 | In, well take we witthan in thy lited 172 | Wo the feoth this will saigediers warce; 173 | Which spirin, have sir, he way, this will his befold 174 | And frend we till, this dige of my say, it thought; 175 | And with mage of y that well we chei 176 | 177 | 2017-09-06 04:32:39,157 - __main__ - INFO - epoch: 8, duration: 223s, loss: 1.67409. 178 | 2017-09-06 04:32:39,647 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 179 | 2017-09-06 04:32:39,649 - __main__ - INFO - generating 512 characters from top 10 choices. 180 | 2017-09-06 04:32:39,649 - __main__ - INFO - generating with seed: "id talk. 181 | 182 | LORD F". 183 | 2017-09-06 04:32:41,302 - __main__ - INFO - generated text: 184 | id talk. 185 | 186 | LORD FORRY 187 | He will you with mall the live, my bark of? 188 | 189 | KING EBDY RARET: 190 | If heat,, wimes you havisht and doth speated; 191 | Te man some, it them scue too deather and shams 192 | This all them boince. Frim's are you but is to to witles, and, 193 | Apainged from seeshow, I have hive you. 194 | 195 | GRUGE: 196 | How well streed's hush my may's speefs, the cure the fords son. 197 | 198 | ROBESLO: 199 | No hollow, 200 | For newper breass and she mesise. 201 | 202 | LEONTES: 203 | Where swert you with than may's main you, are be 204 | What his his parder'd soman shough yer is not 205 | That is noy whos 206 | 207 | 2017-09-06 04:36:32,501 - __main__ - INFO - epoch: 9, duration: 231s, loss: 1.63558. 208 | 2017-09-06 04:36:33,088 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 209 | 2017-09-06 04:36:33,092 - __main__ - INFO - generating 512 characters from top 10 choices. 210 | 2017-09-06 04:36:33,092 - __main__ - INFO - generating with seed: "t treaso". 211 | 2017-09-06 04:36:34,733 - __main__ - INFO - generated text: 212 | t treason? 213 | And I silf bodough thou maid, by my bed, house herself? 214 | Is man's brother, I'll than the truch off the pursact spine 215 | If may, and as not sour hore, and thou sin, 216 | And thim thus your dichatiol, mistle of in supperick? 217 | 218 | Dord: 219 | Why sinkerd's than him, whim my heart not his poor; 220 | Make to so my prows bare's 221 | He he wouldsh agay to bod a mishen. 222 | 223 | FLORENTER: 224 | I'll way teld my nonise than be deong my house! 225 | Then seren suppetal such of my boly, I swowerther? 226 | As I see you we pet, and stown the tonger 227 | Thouting with the sw 228 | 229 | 2017-09-06 04:40:22,722 - __main__ - INFO - epoch: 10, duration: 227s, loss: 1.60254. 230 | 2017-09-06 04:40:23,017 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 231 | 2017-09-06 04:40:23,019 - __main__ - INFO - generating 512 characters from top 10 choices. 232 | 2017-09-06 04:40:23,020 - __main__ - INFO - generating with seed: " tal". 233 | 2017-09-06 04:40:24,658 - __main__ - INFO - generated text: 234 | tall and: 235 | And tas the deed it be sorrow noblo sire; 236 | The warching me the didied mark, 237 | That the benison, to yet her my son. 238 | 239 | Clown: 240 | O do any sobaty we tours! a wall him bet. 241 | Swear to son; whicile, alenst were if is alrsimes? 242 | We cause bore is heard; hour her soldy, and there, 243 | I though in thou hate to me as my dright: 244 | To then been though'd tears' may all his daughter's poor 245 | The dost. 246 | 247 | But: 248 | Anl all the tenters swardly abour, but they heaven? 249 | The hadby, and turran, yet your but his 250 | may you day benishons were happer 251 | 252 | 253 | 2017-09-06 04:44:16,492 - __main__ - INFO - epoch: 11, duration: 231s, loss: 1.57411. 254 | 2017-09-06 04:44:16,729 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 255 | 2017-09-06 04:44:16,731 - __main__ - INFO - generating 512 characters from top 10 choices. 256 | 2017-09-06 04:44:16,732 - __main__ - INFO - generating with seed: "re, and devilish". 257 | 2017-09-06 04:44:18,371 - __main__ - INFO - generated text: 258 | re, and devilish 259 | As a cass should te prail, with be say strongs; 260 | To so was speat years me, shole, the bed of thee: 261 | With hersed thy distidit and brother hight 262 | Which are me now spevich of an inteccy our 263 | bine mesty son, stong wouldst since for, wake way? 264 | 265 | PETRENCE: 266 | Not, ban I by tell this shand be still's 267 | With cure me, and he croke and sence, sead wime 268 | Sceepice yet with chony. 269 | 270 | GLOUCESTER: 271 | I seare. Which, arsiest, look our tuttand to 272 | anto thy, and not he stirn. Hot, I had comen, 273 | To think nor so mine arviling truccinat. 274 | 275 | DUKE 276 | 277 | 2017-09-06 04:48:05,835 - __main__ - INFO - epoch: 12, duration: 227s, loss: 1.54962. 278 | 2017-09-06 04:48:06,180 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 279 | 2017-09-06 04:48:06,182 - __main__ - INFO - generating 512 characters from top 10 choices. 280 | 2017-09-06 04:48:06,183 - __main__ - INFO - generating with seed: " what you say, m". 281 | 2017-09-06 04:48:07,830 - __main__ - INFO - generated text: 282 | what you say, my swaked 283 | Stut to you true so, wonk his partines shried. And sear'd, 284 | And stay you are them, and to my say. 285 | 286 | GROSEL: 287 | The claudially! 288 | 289 | GEWORD: 290 | Bigh's strings his not well, and then threat the heav 291 | He heart made. 292 | 293 | AXTOLY: 294 | I'll me? 295 | 296 | FROSP SEY: 297 | Before he to her some, what the bay thy bing; 298 | And see whose armicy. 299 | 300 | PETER: 301 | Not should ben, thyself attend him frian and fide; 302 | The plief, were cruin him my cirnsty and fores' 303 | What mustise a till your markned to main he canst helvelf 304 | I ad my sitious by with sun and that are 305 | 306 | 2017-09-06 04:51:50,277 - __main__ - INFO - epoch: 13, duration: 222s, loss: 1.52774. 307 | 2017-09-06 04:51:50,516 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 308 | 2017-09-06 04:51:50,519 - __main__ - INFO - generating 512 characters from top 10 choices. 309 | 2017-09-06 04:51:50,519 - __main__ - INFO - generating with seed: "him and men of heart 310 | Look'd wond". 311 | 2017-09-06 04:51:52,181 - __main__ - INFO - generated text: 312 | him and men of heart 313 | Look'd wonders of the dighe walks with heaven 314 | To him their bang too such in this subbed age 315 | Thee for at wathish the falls bear this breather'd sparry 316 | Took and bring and somen he shall and riport! 317 | Suce, I have'-ded you stand's heaven owe. 318 | The mound the seeptly with thou cass not, 319 | Ald I that I shumbrogn'd, and thou that high 320 | We'll prot the douth with your lead, home some to be it; 321 | So to dead show on this. 322 | 323 | First Serang: 324 | I cause, an oft with to bless, meet when sunch'd! 325 | 326 | DUCHESS OF YORK 327 | O, hill by you son all a moneriss 328 | 329 | 2017-09-06 04:55:39,063 - __main__ - INFO - epoch: 14, duration: 226s, loss: 1.50837. 330 | 2017-09-06 04:55:39,484 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 331 | 2017-09-06 04:55:39,487 - __main__ - INFO - generating 512 characters from top 10 choices. 332 | 2017-09-06 04:55:39,487 - __main__ - INFO - generating with seed: "l are gone. 333 | 334 | Cho". 335 | 2017-09-06 04:55:41,152 - __main__ - INFO - generated text: 336 | l are gone. 337 | 338 | Chost: 339 | Away, but since's to-do; and I will. 340 | 341 | FRLAREN: 342 | At if a ming is should fools shall we death; 343 | And I have merry 344 | As this breaths from this hore subjoutes 345 | To have bid and thee; thou wilt a prant word, 346 | Your world all soon a did, sir. 347 | 348 | PAURET: 349 | This imperuser him we will perveins thee sorron 350 | And his part men's satten to them this, it is. 351 | 352 | DUKEN VINCENTIO: 353 | Marken as welcome? 354 | 355 | Secomving: 356 | Now lives a sumiced a mornow stain their 357 | tend time to me swolf you to-day, why sone 358 | With bromence her alooford with soul wife; 359 | 360 | 361 | 2017-09-06 04:59:26,295 - __main__ - INFO - epoch: 15, duration: 225s, loss: 1.49118. 362 | 2017-09-06 04:59:26,529 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 363 | 2017-09-06 04:59:26,532 - __main__ - INFO - generating 512 characters from top 10 choices. 364 | 2017-09-06 04:59:26,533 - __main__ - INFO - generating with seed: "t hope i". 365 | 2017-09-06 04:59:28,196 - __main__ - INFO - generated text: 366 | t hope is wise 367 | Beason our, and see mine'st by the chrield 368 | Mand to any be all by thy sudment, 369 | What may had that's a tence as humpres. 370 | 371 | DUCHESS OF YORK: 372 | Have to me my lord, see the fareed, 373 | Than but a fool, to his letter too sighted 374 | To hose a suppered, would dispomit much abes 375 | True you her company advessing 376 | To this tongue at about her winds, 377 | To my sup a gaught out to see, with hamp, 378 | Hese should breathethersied here to a wander? 379 | 380 | RATCHILANUS: 381 | See to to the tord maigned, the dame. 382 | 383 | First Servindma: 384 | An the more trupled? 385 | 386 | 2017-09-06 05:03:14,141 - __main__ - INFO - epoch: 16, duration: 225s, loss: 1.47569. 387 | 2017-09-06 05:03:14,427 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 388 | 2017-09-06 05:03:14,430 - __main__ - INFO - generating 512 characters from top 10 choices. 389 | 2017-09-06 05:03:14,431 - __main__ - INFO - generating with seed: "e, 390 | Was d". 391 | 2017-09-06 05:03:16,054 - __main__ - INFO - generated text: 392 | e, 393 | Was done to be death, when I look as son 394 | them? 395 | 396 | ManIm: 397 | I mark not the curd and with one again, 398 | And the wosed much to meeftly the way. 399 | 400 | DUKE VINCENTIO: 401 | Yigh, sir, by my mester'd many fonter for me to 402 | what musinaly the tremices sometucion with thy breast, 403 | To all the most be make to be so be 404 | The word yair from my many are by the frim 405 | And we there to man and look it me. 406 | 407 | DUKE VINCENTIO: 408 | She with a grace, and the heards true your phario. 409 | 410 | KATHARINA: 411 | A hour's noble hands? his mother honesty 412 | Becompence and, but at hy p 413 | 414 | 2017-09-06 05:07:06,380 - __main__ - INFO - epoch: 17, duration: 230s, loss: 1.46163. 415 | 2017-09-06 05:07:06,681 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 416 | 2017-09-06 05:07:06,684 - __main__ - INFO - generating 512 characters from top 10 choices. 417 | 2017-09-06 05:07:06,684 - __main__ - INFO - generating with seed: "S:". 418 | 2017-09-06 05:07:08,314 - __main__ - INFO - generated text: 419 | S: 420 | How, tire and both wy pergit to-boly, alard, 421 | By terseling and the chirdio my faith, 422 | With her trurstanns and sack. I have to them, 423 | And mids-propured with our says of leave: 424 | The wrong there that I po in soul tendly 425 | To scorn and will hither to pross murining, 426 | For a myselfs of this. 427 | 428 | PETRUCHIO: 429 | Some and marrine as I seems, thou has now; 430 | And at the heart we have prims thee forsh. 431 | 432 | PEBELIA: 433 | Whit, grunt them, sir. 434 | 435 | LEONTES: 436 | A beggh and donest. The man strangly men 437 | Wemer of her by wilt to me not sitie 438 | And leave th 439 | 440 | 2017-09-06 05:10:55,129 - __main__ - INFO - epoch: 18, duration: 226s, loss: 1.44886. 441 | 2017-09-06 05:10:55,663 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 442 | 2017-09-06 05:10:55,666 - __main__ - INFO - generating 512 characters from top 10 choices. 443 | 2017-09-06 05:10:55,667 - __main__ - INFO - generating with seed: "have spoken true". 444 | 2017-09-06 05:10:57,299 - __main__ - INFO - generated text: 445 | have spoken true? 446 | 447 | FRIAR LAURENCE: 448 | What said had seirs? Or myself 449 | So my lord. I have him! and went not witnion. 450 | For you are not him. 451 | 452 | MONTAGE: 453 | No heart within, but me, sir their sister thanks, 454 | And I be sin in cannot, the morth, 455 | Time brought you to be thing for that the kiss! 456 | 457 | HORTENSIO: 458 | How now, whill thee him shall thing by them: I am here 459 | Were is so, send them the begon and fools. 460 | 461 | GLOUCESTER: 462 | No, I pray he do you and mind in your best: 463 | I was cristed hath sents by might his mone, 464 | For the hearts and any a sheeping: splays 465 | 466 | 2017-09-06 05:14:41,674 - __main__ - INFO - epoch: 19, duration: 224s, loss: 1.43717. 467 | 2017-09-06 05:14:42,225 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 468 | 2017-09-06 05:14:42,227 - __main__ - INFO - generating 512 characters from top 10 choices. 469 | 2017-09-06 05:14:42,229 - __main__ - INFO - generating with seed: "ha". 470 | 2017-09-06 05:14:43,844 - __main__ - INFO - generated text: 471 | hat't our hole: 472 | Think, so lontagions makes thy most capply 473 | Trunce her shed of my londing: how but scarce, 474 | I trance on him: I have take my hand 475 | And with this most propers to deaths of cheer: 476 | This wistainst tarry's fie and libed and speak. 477 | 478 | KING EDWARD IV: 479 | But, many he is her sack to carce: 480 | And, be but beficed him: there shall tell me 481 | And should be so seems to the hour cruy us stoled: 482 | A child to the both sister: his misself, 483 | I could as I say sender my cannot strong; 484 | Have tyranges will him him speeping-signite 485 | T 486 | 487 | 2017-09-06 05:18:27,064 - __main__ - INFO - epoch: 20, duration: 223s, loss: 1.42649. 488 | 2017-09-06 05:18:27,591 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 489 | 2017-09-06 05:18:27,594 - __main__ - INFO - generating 512 characters from top 10 choices. 490 | 2017-09-06 05:18:27,595 - __main__ - INFO - generating with seed: " non". 491 | 2017-09-06 05:18:29,225 - __main__ - INFO - generated text: 492 | nonceives in 493 | survant be at strift most good my sooner mat 494 | and what have alone or to be without of thy 495 | foolish'd? 496 | 497 | Manmia: 498 | Ay, when? 499 | 500 | ARINI 501 | US: 502 | Why, whiling so dif,' and you day not sean to be 503 | Both at your brother's frother of you. 504 | 505 | LADNI EDCARET: 506 | What doth thy slawe is heard is your friend 507 | And never but this bod to see his land; 508 | But, tem, the berowage the freetom, my soffick, as 509 | Angel it not are as which must to her such! 510 | 511 | CARILLO: 512 | I will not little to so shoulded but 513 | Why proper of years, 514 | And where so we are o 515 | 516 | 2017-09-06 05:22:13,269 - __main__ - INFO - epoch: 21, duration: 224s, loss: 1.41694. 517 | 2017-09-06 05:22:13,822 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 518 | 2017-09-06 05:22:13,825 - __main__ - INFO - generating 512 characters from top 10 choices. 519 | 2017-09-06 05:22:13,826 - __main__ - INFO - generating with seed: "he manac". 520 | 2017-09-06 05:22:15,535 - __main__ - INFO - generated text: 521 | he manachighed 522 | I puilth her such of this colford witly: 523 | Anto any so betwain soundediener 524 | Feeling and adved for him by whilstice; 525 | So to a maid. 526 | 527 | BAPTISTA: 528 | Ay, sortience tou them, take in eye; 529 | And when I have best to coilf a prefect 530 | Forgetunity! and who did not will and there. 531 | 532 | BENVOLIO: 533 | Sun holy with than I was not not made a sight 534 | As the point to how here if I brive his 535 | fours's drem wyold will seem'd thereow, whose shall been 536 | the hand tonour is men of desire how come this? 537 | That hath blood the heavens and son; what 538 | 539 | 2017-09-06 05:26:09,641 - __main__ - INFO - epoch: 22, duration: 234s, loss: 1.40818. 540 | 2017-09-06 05:26:09,885 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 541 | 2017-09-06 05:26:09,887 - __main__ - INFO - generating 512 characters from top 10 choices. 542 | 2017-09-06 05:26:09,889 - __main__ - INFO - generating with seed: " to ". 543 | 2017-09-06 05:26:11,537 - __main__ - INFO - generated text: 544 | to bewaldst their hearts 545 | to son, my drese the father with my concell. 546 | But heaven. 547 | 548 | COMINIUS: 549 | This from content, my noble and stays 550 | With this, so fear why comes as such any; 551 | I durgrent we must man such his nection. 552 | Whom should did my not honour hence? 553 | 554 | PETRUCHIO: 555 | Much too any moward mardance? so much bether, 556 | Metchared harms, sole still me not be not 557 | The hurds by thy heart's bayed to' paut, 558 | But all this throunds the plays to time. 559 | Stay again; hence it, we much a subject being with me: 560 | They say is it in her of it 561 | 562 | 2017-09-06 05:29:58,360 - __main__ - INFO - epoch: 23, duration: 226s, loss: 1.39919. 563 | 2017-09-06 05:29:58,686 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 564 | 2017-09-06 05:29:58,688 - __main__ - INFO - generating 512 characters from top 10 choices. 565 | 2017-09-06 05:29:58,690 - __main__ - INFO - generating with seed: "f ". 566 | 2017-09-06 05:30:00,324 - __main__ - INFO - generated text: 567 | f walce, I will. 568 | And the duke will be in the love here; 569 | But spit our cemition. 570 | 571 | BENVOLIO: 572 | The says and you have priviant well to del. 573 | 574 | KING HENRY VI: 575 | Go, I that; I himself and my lows with county. 576 | To prauge! 577 | 578 | CATESBA: 579 | At my bodam, so let my for old set of him 580 | Towblepy. Hath hath swore stroughts, thou shall happ you 581 | Till thee appear's putsial so. Hath but thyself? 582 | I shoudly thou have to came and a gare, 583 | And tell ye in my force of her witted, 584 | We'll attend the wieft as what wised for withy 585 | Sound'd thy being in a 586 | 587 | 2017-09-06 05:33:52,477 - __main__ - INFO - epoch: 24, duration: 232s, loss: 1.39105. 588 | 2017-09-06 05:33:52,866 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 589 | 2017-09-06 05:33:52,869 - __main__ - INFO - generating 512 characters from top 10 choices. 590 | 2017-09-06 05:33:52,870 - __main__ - INFO - generating with seed: "onum". 591 | 2017-09-06 05:33:54,524 - __main__ - INFO - generated text: 592 | onumerey, 593 | Too nothing for the props field to him. 594 | I'll twenty blows in hell.- 595 | 596 | PETRUCHIO: 597 | My lord, two had that in senate with he take, 598 | And had thou art sude and after'd and leals; 599 | And they shall not beard instraction hands? 600 | 601 | POMPEY: 602 | I am by the mark:-- 603 | 604 | AUTOLYCUS: 605 | Ah take your liege, 606 | And I have an unsusion is nurse: 607 | And minist time shall say, with a moon, 608 | You will not breath men by any whop about the consuits from 609 | sone. 610 | 611 | Loud Servant: 612 | Tume out to the dapity more. I will be things 613 | And signish thy serves still 614 | A 615 | 616 | 2017-09-06 05:37:43,416 - __main__ - INFO - epoch: 25, duration: 228s, loss: 1.38355. 617 | 2017-09-06 05:37:43,968 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 618 | 2017-09-06 05:37:43,971 - __main__ - INFO - generating 512 characters from top 10 choices. 619 | 2017-09-06 05:37:43,972 - __main__ - INFO - generating with seed: " thou th". 620 | 2017-09-06 05:37:45,622 - __main__ - INFO - generated text: 621 | thou thrueds breast? 622 | Halt all my tongrunt sursuh's, for that; 623 | But the poor shall speel and wert business of the 624 | did threw this atcher in sincted,--where shild 625 | Two tutgacions, in offily, and thing-brother: 626 | That ships to this constate a faith, tell we 627 | Thou shake batter to him, my son'd. 628 | 629 | LUCENTIO: 630 | What, he'll greet mine so? my sir, holp not heard, 631 | Theirs mistress most may mind. 632 | 633 | ANGELO: 634 | Had thou, whileds you, sir, a pays, and take thee 635 | And thy book again. I may be well, banisted, 636 | And seen my first the forcastick son 637 | 638 | 2017-09-06 05:41:35,349 - __main__ - INFO - epoch: 26, duration: 229s, loss: 1.37656. 639 | 2017-09-06 05:41:35,652 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 640 | 2017-09-06 05:41:35,655 - __main__ - INFO - generating 512 characters from top 10 choices. 641 | 2017-09-06 05:41:35,656 - __main__ - INFO - generating with seed: "ght;". 642 | 2017-09-06 05:41:37,283 - __main__ - INFO - generated text: 643 | ght; 644 | choughters to but our sister to you, 645 | To trown soldiers we shall frown to many by. 646 | 647 | DUKE VINCENTIO: 648 | But it to till a foot. 649 | 650 | FLORIZEL: 651 | I know thee; I can now the craws? 652 | 653 | KING RICHARD III: 654 | How brow your honour for our heart, with their 655 | softless apparel; i' the late, which shall with myself: 656 | And then, some's from him the county when? 657 | 658 | Servant: 659 | Was nothing. 660 | 661 | DUKE VINCENTIO: 662 | Are a so it insrent, o'erween with art they: 663 | And to my posian bear my better leaving to, 664 | Fastis to been you, to crominitar, then 665 | Oft thou t 666 | 667 | 2017-09-06 05:45:26,813 - __main__ - INFO - epoch: 27, duration: 229s, loss: 1.37001. 668 | 2017-09-06 05:45:27,198 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 669 | 2017-09-06 05:45:27,201 - __main__ - INFO - generating 512 characters from top 10 choices. 670 | 2017-09-06 05:45:27,202 - __main__ - INFO - generating with seed: "e pass 671 | A". 672 | 2017-09-06 05:45:28,829 - __main__ - INFO - generated text: 673 | e pass 674 | Anochard: shall well be, 675 | And heaven home--will I'll maketune my trumpopens 676 | Here to call his free made stity-- 677 | 678 | DUCHESS: 679 | God master; brign, I am a will blaccuse 680 | Thingbroke the stard, 681 | Or that he thou hast having so. 682 | 683 | DORSET: 684 | Has I sends you, madam, sir,-- 685 | But I can nobed in madge thou hast senders? 686 | 687 | BARNARDINE: 688 | In an arms, toucher her moon, which instruck's, 689 | Howness and his looks and men he wated, 690 | The shoot or natersals, seen at your leads. 691 | And, thank you with a pricuity; say said 692 | This imnow my lord, make the 693 | 694 | 2017-09-06 05:49:15,283 - __main__ - INFO - epoch: 28, duration: 226s, loss: 1.36381. 695 | 2017-09-06 05:49:15,746 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 696 | 2017-09-06 05:49:15,749 - __main__ - INFO - generating 512 characters from top 10 choices. 697 | 2017-09-06 05:49:15,750 - __main__ - INFO - generating with seed: "hile he did bear my countenance ". 698 | 2017-09-06 05:49:17,446 - __main__ - INFO - generated text: 699 | hile he did bear my countenance to die. 700 | I woe a friends at shall aloved tell 701 | In cass, 702 | But in her throunds here command-madok 703 | With sad accounts of your survart before, 704 | Thou must say the services here's advised; 705 | There'ce wells, 706 | Manishes and my soul be thresen from the trurive, 707 | And seem upin--tisilip's dogs and talk of high? 708 | This sicks, which it to death, thou at how, as 709 | it waves in the court of turn in higher, 710 | To this breaths, for whose compasion by his 711 | flings: but it is no selidy than, 712 | For whourd attorwick'ds my hearts to thyself; 713 | Why, tho 714 | 715 | 2017-09-06 05:53:05,178 - __main__ - INFO - epoch: 29, duration: 227s, loss: 1.35794. 716 | 2017-09-06 05:53:05,549 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 717 | 2017-09-06 05:53:05,552 - __main__ - INFO - generating 512 characters from top 10 choices. 718 | 2017-09-06 05:53:05,553 - __main__ - INFO - generating with seed: "R LAUREN". 719 | 2017-09-06 05:53:07,204 - __main__ - INFO - generated text: 720 | R LAURENKC+AQI/MS(VI_/COMIjMS^^vOvjeireasp. 721 | 722 | GLOUCESTER: 723 | And, had I spair to him; and have marry brovient wretch 724 | Besign to proned an ear of yot about it with dusiness 725 | And play with alling. 726 | 727 | BONA: 728 | Geford, moor. Bood madam, sir, I will detire 729 | By most morn, when die, you have a saw is words. 730 | 731 | Shepherd: 732 | Would tyo his motter, I'll be not to the sats-- 733 | 734 | DUKE VINCENTIO: 735 | I have most high trubnation? 736 | 737 | MEORITA: 738 | A constand foital. 739 | 740 | MONRANIA: 741 | How not insturn these armish with course? 742 | 743 | BIONDELLO: 744 | Short, my lord! 745 | 746 | First Murderer 747 | 748 | 2017-09-06 05:56:53,942 - __main__ - INFO - epoch: 30, duration: 226s, loss: 1.35236. 749 | 2017-09-06 05:56:54,176 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 750 | 2017-09-06 05:56:54,179 - __main__ - INFO - generating 512 characters from top 10 choices. 751 | 2017-09-06 05:56:54,181 - __main__ - INFO - generating with seed: "er". 752 | 2017-09-06 05:56:55,792 - __main__ - INFO - generated text: 753 | ersenese them fight, 754 | And let them all at lief, I take it as a 755 | melly that. 756 | 757 | LEONTES: 758 | I shall by him from your sake horselous state, 759 | Shall have now of tasking, and to that see 760 | And an even to suffer to agido? 761 | What withy honour's duty? 762 | 763 | First RoLom: 764 | Had not then? 765 | 766 | POLIXENES: 767 | I take yurld, will hear the man that swearing's good 768 | whose house that wouldst parrowed to be bears this, 769 | Where's no prince and his widowing in his: wirl your grow'd, 770 | And no'en, will then to see too; without-naws! 771 | Or store me, and my counterma 772 | 773 | 2017-09-06 06:00:48,670 - __main__ - INFO - epoch: 31, duration: 232s, loss: 1.34703. 774 | 2017-09-06 06:00:48,999 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 775 | 2017-09-06 06:00:49,002 - __main__ - INFO - generating 512 characters from top 10 choices. 776 | 2017-09-06 06:00:49,004 - __main__ - INFO - generating with seed: " sti". 777 | 2017-09-06 06:00:50,626 - __main__ - INFO - generated text: 778 | stiting; thoughts, whether 779 | Though the bitter will be where I lience brot 780 | with the think fortunain apprine here. And most 781 | I cannot disparently do but. 782 | 783 | CORIOLANUS: 784 | How we'll have I sir, boint and horpest, 785 | Bononful traporurose full to do't, my lords 786 | And who hap thy less in day now wnoties 787 | And breadin and with her come to the may with march 788 | She't show his day--a home, we do's hid. 789 | 790 | Second Keeper: 791 | How'd my lord as the conciried maid, 792 | Thy hulls but a weight of the mark in the 793 | maid bring shall be house more from whe 794 | 795 | 2017-09-06 06:04:38,120 - __main__ - INFO - epoch: 32, duration: 227s, loss: 1.34199. 796 | 2017-09-06 06:04:38,381 - __main__ - INFO - model saved: checkpoints/mxnet_tinyshakespeare/model.params. 797 | 2017-09-06 06:04:38,384 - __main__ - INFO - generating 512 characters from top 10 choices. 798 | 2017-09-06 06:04:38,386 - __main__ - INFO - generating with seed: "r maid 799 | f". 800 | 2017-09-06 06:04:40,020 - __main__ - INFO - generated text: 801 | r maid 802 | for the husband made with such or princesse must, 803 | And, to bring him for his signer, triness 804 | Hering are so sweet parmining at once 805 | As you speak not; and ne't always as therefore. 806 | 807 | Second Geers: 808 | Alague and sir; he shall hear speak upon 809 | Titles, think, whom to advised to be supt; 810 | Then with the shucks make man for way'd me 811 | He had but majesty, sir to power, turn the children, 812 | Fall to be sorrow to any mother. 813 | 814 | PERDITA: 815 | I pray you arm. 816 | 817 | CARLISANUM: 818 | When I had strightry 819 | For such rewarded, with succeed to so; 820 | And I sh 821 | 822 | 2017-09-06 06:04:40,021 - __main__ - INFO - end of training, duration: 7348s. 823 | 2017-09-06 06:04:40,022 - __main__ - INFO - generating 1024 characters from top 3 choices. 824 | 2017-09-06 06:04:40,022 - __main__ - INFO - generating with seed: "was in t". 825 | 2017-09-06 06:04:43,275 - __main__ - INFO - generated text: 826 | was in the command. 827 | 828 | KING EDWARD II: 829 | And so well and some any sorrow are 830 | The subjects and sour of the care and this son, 831 | And see the time a plays that words, 832 | And though you have been made and stand to thee 833 | To tempt that we were a monstry. 834 | Thoush, when have seem how she's to strike. 835 | 836 | POLIXENES: 837 | Shall not should have madier him to meet my life, 838 | Armstary and subscrue, thou and he was some 839 | That showed the serve as will, to be the county 840 | The fool, so took of the serment and my son 841 | The sun that she do that those the world treature. 842 | 843 | GLOUCESTER: 844 | Where is my like, sir, I will not be this. 845 | Why, sir, what, that you will not so he that have sees 846 | And therefore the surdench officious chief. 847 | 848 | KING EDWARD IV: 849 | Ay, though the caunt as true and so the case 850 | That hearing the patience and tranion of me, 851 | That she will be so to set the streaty of the 852 | bread and this words and a man't. Boy, 853 | And whether the surdens of mine as his 854 | life. I have seem'd, and should brieve my like and 855 | and to be such a presince and such a command. 856 | What is't thy lor 857 | -------------------------------------------------------------------------------- /logs/pytorch_model.log: -------------------------------------------------------------------------------- 1 | 2017-09-06 01:01:42,025 - utils - DEBUG - call: pytorch_model.py train --checkpoint=checkpoints/pytorch_tinyshakespeare/model.ckpt --text=data/tinyshakespeare.txt 2 | 2017-09-06 01:01:42,026 - utils - DEBUG - ArgumentParser: Namespace(batch_size=64, checkpoint_path='checkpoints/pytorch_tinyshakespeare/model.ckpt', clip_norm=5.0, drop_rate=0.0, embedding_size=32, learning_rate=0.001, main=, num_epochs=32, num_layers=2, restore=False, rnn_size=128, seq_len=64, text_path='data/tinyshakespeare.txt') 3 | 2017-09-06 01:01:42,027 - __main__ - INFO - corpus length: 1115394. 4 | 2017-09-06 01:01:42,039 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 5 | 2017-09-06 01:01:42,287 - __main__ - INFO - start of training. 6 | 2017-09-06 01:01:42,289 - utils - INFO - number of batches: 272. 7 | 2017-09-06 01:01:42,289 - utils - INFO - effective text length: 1114112. 8 | 2017-09-06 01:01:42,289 - utils - INFO - x shape: (64, 17408). 9 | 2017-09-06 01:01:42,290 - utils - INFO - y shape: (64, 17408). 10 | 2017-09-06 01:04:46,419 - __main__ - INFO - epoch: 1, duration: 184s, loss: 2.86632. 11 | 2017-09-06 01:04:46,424 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 12 | 2017-09-06 01:04:46,424 - __main__ - INFO - generating 512 characters from top 10 choices. 13 | 2017-09-06 01:04:46,425 - __main__ - INFO - generating with seed: "u ta". 14 | 2017-09-06 01:04:47,214 - __main__ - INFO - generated text: 15 | u taor toum ho oare the torun shit tu ty tout, fy hither hart actoret acind herist thals as bhilt thare he bor me thue, the thel steed sut hrere seangote, the thim, brty arterd; 16 | Alinnd the tiin she me brato thomy thee, hor htath and orland,, 17 | Ans 18 | To me wo to h troulsiless tant of core srardess ale bomave malim choul he ante, this mound 19 | Wher worte 20 | Antis by os, heart ave thic yous hand weet. 21 | 22 | OELSIO 23 | SSRERIRNL: 24 | Thou waan weise. 25 | Hure the ilt suse heleted theas and in't waas for banter,: 26 | Sis if bhone songonirant. 27 | 28 | RA 29 | 30 | 2017-09-06 01:07:52,288 - __main__ - INFO - epoch: 2, duration: 185s, loss: 2.16662. 31 | 2017-09-06 01:07:52,294 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 32 | 2017-09-06 01:07:52,294 - __main__ - INFO - generating 512 characters from top 10 choices. 33 | 2017-09-06 01:07:52,295 - __main__ - INFO - generating with seed: "whose 34 | fa". 35 | 2017-09-06 01:07:53,171 - __main__ - INFO - generated text: 36 | whose 37 | fares to cracle thit wime, 38 | As wilp it me this; 39 | To man wime sut hath, he mead, 40 | I me thou whets thy and fo tor so wiir. 41 | 42 | CAOORE: 43 | Grise. 44 | Whens; she baclent frilet wouked tither arowiditore 45 | To ticters ans, sust sies ancest, salroth, four is swath ontory 46 | totheur sonce. 47 | 48 | KUREE: 49 | This ank. 50 | Thetering thime they mime, is hir ay say, biven by cowtitiland ance 51 | And is all to bid are built shim, thes faveress ming in is of cry alt slont me that haled beanod with me to comes is as the meaver a cures frest it and of our a co 52 | 53 | 2017-09-06 01:10:55,985 - __main__ - INFO - epoch: 3, duration: 182s, loss: 1.96265. 54 | 2017-09-06 01:10:55,991 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 55 | 2017-09-06 01:10:55,991 - __main__ - INFO - generating 512 characters from top 10 choices. 56 | 2017-09-06 01:10:55,992 - __main__ - INFO - generating with seed: "ansforma". 57 | 2017-09-06 01:10:56,746 - __main__ - INFO - generated text: 58 | ansformant, feel britorty 59 | He shang heance thee brinos, her feintes hysery sind must to well as hath my beal apred and froving fir, thee that he all so spiend 60 | Anto sisiess 61 | O staln you, hark. 62 | 63 | DULETN: 64 | Say sair sins, 65 | And firsed the duth all's wink buse wiin are on, 66 | Tell misters's heer faen, you stry slay their to should shalk muse thom be sine? 67 | 68 | KINKINGER EITE: 69 | O puitent to hers, that it fantar thou chisernon to wouks 70 | Whis sather, 71 | Twe be phave hy langed when sem and been her menore; wleech twan an thou and to the bowe 72 | 73 | 2017-09-06 01:14:00,440 - __main__ - INFO - epoch: 4, duration: 183s, loss: 1.83413. 74 | 2017-09-06 01:14:00,446 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 75 | 2017-09-06 01:14:00,446 - __main__ - INFO - generating 512 characters from top 10 choices. 76 | 2017-09-06 01:14:00,446 - __main__ - INFO - generating with seed: "NTIO". 77 | 2017-09-06 01:14:01,984 - __main__ - INFO - generated text: 78 | NTIO: 79 | To hand womlo for my hordign's will all tane we with them slays he sharr thy mase and, bide my, be are of our devery son ingasing, this's,' 80 | Why my man, 81 | Thine and himself 82 | Our his him of the tount trescers, 83 | Of tell the beport am to steps forsenter he cleet. 84 | 85 | GLOUES IHUS: 86 | I say 87 | Bonk'd besise all 88 | Witly brother to as thee, for that are to my booked othem misess and it a dorgeny 89 | And is by digly home a cacked fay 90 | It heak we mother 91 | Weil there or as 92 | Once the, woo, nose is nitire as say am her mouron strays time he 93 | 94 | 2017-09-06 01:17:05,574 - __main__ - INFO - epoch: 5, duration: 183s, loss: 1.74432. 95 | 2017-09-06 01:17:05,579 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 96 | 2017-09-06 01:17:05,580 - __main__ - INFO - generating 512 characters from top 10 choices. 97 | 2017-09-06 01:17:05,580 - __main__ - INFO - generating with seed: "ourselves, 98 | To gr". 99 | 2017-09-06 01:17:06,347 - __main__ - INFO - generated text: 100 | ourselves, 101 | To gried the work, 102 | Tay the show 103 | so my have a mory 104 | teincling were her conmer of are them fays; 105 | Bay all, hightary! hath make me for what it 106 | we will my concumurity, Incuse, with 107 | sood that must far the mather and a drave 108 | tis her prow, 109 | To destancy is tongued may twould my scupord to as me wich, 110 | Should the dencentle as me my leavon 111 | To his foreny in by he swourt'd in they hreather, with you wich fanger but sight 112 | In atwenst in her a may? 113 | 114 | LUCIO: 115 | Stay. I halk in itseld with a how, then so heen his't shate with he her. 116 | 117 | S 118 | 119 | 2017-09-06 01:20:08,548 - __main__ - INFO - epoch: 6, duration: 182s, loss: 1.67133. 120 | 2017-09-06 01:20:08,553 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 121 | 2017-09-06 01:20:08,554 - __main__ - INFO - generating 512 characters from top 10 choices. 122 | 2017-09-06 01:20:08,554 - __main__ - INFO - generating with seed: "egreet 123 | T". 124 | 2017-09-06 01:20:09,311 - __main__ - INFO - generated text: 125 | egreet 126 | To sight will to the corld, we what hone, 127 | Mays sorfest to stisted we worth be heaven be thas trece of 128 | And wich supfork bear me at the but in alticone 129 | To the duct hang tlither; the candutent 130 | The blet servow the contrient in man of the pimes 131 | If thind yee hourtenten her to my true is; 132 | His sort our will chound surse alt spay this from 133 | Wither appainst the sord to me was'd's my 134 | so blesenched if be them. 135 | 136 | DUKE VINCENTIO: 137 | Who sword weere is take so, a shalk, to my march 138 | Thou ammald my shance by the brown you sail, 139 | B 140 | 141 | 2017-09-06 01:23:13,765 - __main__ - INFO - epoch: 7, duration: 184s, loss: 1.6185. 142 | 2017-09-06 01:23:13,771 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 143 | 2017-09-06 01:23:13,771 - __main__ - INFO - generating 512 characters from top 10 choices. 144 | 2017-09-06 01:23:13,771 - __main__ - INFO - generating with seed: "tertain'd that's offer'd, 145 | Comes ". 146 | 2017-09-06 01:23:14,547 - __main__ - INFO - generated text: 147 | tertain'd that's offer'd, 148 | Comes or thine my will a pirts; 149 | But way that young to but a detitic of? 150 | 151 | CATONY ROD: 152 | Ay, for make holf me, what his: we too, believe: 153 | What, as is now tadul me strop to these: 154 | Alp ying! I am abrow to say will her being. 155 | 156 | GRLORET: 157 | Thee, I care house fomt he have 158 | And tell, then what I as in my spare; 159 | Of the will'd of there, not, iven to stain me 160 | The torn fits of your lost the croddon? 161 | 162 | ANTICHARD: 163 | Tham is be to him mark'd as this foms. 164 | 165 | FLOLIE: 166 | All thron the pry speck of me, 167 | For with staith and friin take work, which 168 | 169 | 2017-09-06 01:26:17,404 - __main__ - INFO - epoch: 8, duration: 182s, loss: 1.5788. 170 | 2017-09-06 01:26:17,409 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 171 | 2017-09-06 01:26:17,410 - __main__ - INFO - generating 512 characters from top 10 choices. 172 | 2017-09-06 01:26:17,410 - __main__ - INFO - generating with seed: "VI". 173 | 2017-09-06 01:26:18,163 - __main__ - INFO - generated text: 174 | VICGonten and seper, and this 175 | since sould swonk of I am loved how to lodder 176 | And my promore but as beep to drid 177 | To susper of sworn is so the hers, for they mirst white 178 | Have not me. And I prays, buch more say. 179 | 180 | ANGODERD: 181 | O will now. 182 | 183 | COBUBESTAR: 184 | Where you shear sil, these they boy him so see? 185 | Buts to your daumbled so him me on be a stray. 186 | Bean of I'll were are are the weem. 187 | 188 | DRUSHO: 189 | O bellieve; and and say, should thou steep two word. 190 | As say the parcessel and hence be not, 191 | And thou have woil hends the blood 192 | Tha 193 | 194 | 2017-09-06 01:29:20,068 - __main__ - INFO - epoch: 9, duration: 181s, loss: 1.54719. 195 | 2017-09-06 01:29:20,073 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 196 | 2017-09-06 01:29:20,074 - __main__ - INFO - generating 512 characters from top 10 choices. 197 | 2017-09-06 01:29:20,074 - __main__ - INFO - generating with seed: "TIS: 198 | Is ". 199 | 2017-09-06 01:29:20,836 - __main__ - INFO - generated text: 200 | TIS: 201 | Is a sadnease a moon and may hussel's 202 | Of I sire time, idnot, and hote to the fatth; 203 | And think a courtue, 204 | Frem in, truices's standing whom there to home. 205 | 206 | BASCINIUS: 207 | My meat of your sengerous sound; 208 | Where to the horses he dinder by the putched. To then of? 209 | I him scaugonied my son, best the fathes 210 | With surstiton, from the susilier have news? 211 | One my buse steps of suck me. 212 | 213 | KING RICHARD II: 214 | My willot a pillace, whether thought as all me. 215 | Inseld offench moss'' fouls; for sorn is a care 216 | To did thliful with this hopa 217 | 218 | 2017-09-06 01:32:22,922 - __main__ - INFO - epoch: 10, duration: 182s, loss: 1.52118. 219 | 2017-09-06 01:32:22,927 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 220 | 2017-09-06 01:32:22,928 - __main__ - INFO - generating 512 characters from top 10 choices. 221 | 2017-09-06 01:32:22,928 - __main__ - INFO - generating with seed: "DWAR". 222 | 2017-09-06 01:32:23,800 - __main__ - INFO - generated text: 223 | DWAR IHHA 224 | God be the septly should not we will 225 | shall so twances of stark, that in tenders, 226 | To spake the hopasiest: that throy we the world well. 227 | 228 | GLRUCICIS: 229 | That may tree and they he senced by to some. 230 | 231 | GLOEY: 232 | O'er I have he water no would be he here. 233 | 234 | AUDitizenar: 235 | Him an her, and to honour more, 236 | So be thought both the whire and will midsty 237 | Has agay, which sir? 238 | I mady must these now in the park in accican: 239 | As this wing them to sown, and that now, if not me 240 | woueth on it of tilless speak the hunting me; 241 | By the dr 242 | 243 | 2017-09-06 01:35:29,220 - __main__ - INFO - epoch: 11, duration: 185s, loss: 1.49926. 244 | 2017-09-06 01:35:29,225 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 245 | 2017-09-06 01:35:29,226 - __main__ - INFO - generating 512 characters from top 10 choices. 246 | 2017-09-06 01:35:29,226 - __main__ - INFO - generating with seed: "to t". 247 | 2017-09-06 01:35:29,972 - __main__ - INFO - generated text: 248 | to then 249 | says well alcinour, so shalh how myself of her. 250 | Alables the matt his paices thee, to't, which now. 251 | Thou dost connom: though it hear and leven shall comes: 252 | A know that was alive? to sweet the dit 253 | From her trustion to your words happinding 254 | Than an have to be that with some well: 255 | I be ally with of him; this ire we'll take 256 | Might's father fair his signied of willows; 257 | For his heart's myself. Flay, sir, so, if thou hadst 258 | The horn our lys to hot in bose her, 259 | What as shall not a word was talts? 260 | 261 | DUKE OF YORK: 262 | On 263 | 264 | 2017-09-06 01:38:33,439 - __main__ - INFO - epoch: 12, duration: 183s, loss: 1.48047. 265 | 2017-09-06 01:38:33,444 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 266 | 2017-09-06 01:38:33,444 - __main__ - INFO - generating 512 characters from top 10 choices. 267 | 2017-09-06 01:38:33,445 - __main__ - INFO - generating with seed: " he had eaten ballads and all me". 268 | 2017-09-06 01:38:34,263 - __main__ - INFO - generated text: 269 | he had eaten ballads and all me 270 | Angelish and thoo. For by them fall, as someting deed. 271 | I hold bear thy trots and brophink throws 272 | Where you meet me our sword may coise. 273 | Or my wose, that what son thee how the hasse. 274 | 275 | CLIFFORD: 276 | He was be told of whose an say breath. 277 | 278 | LEONTES: 279 | Ap, with his deport you shall stright it with 280 | Appear to the tenst honour and and are age: 281 | I do have suppand in her power'd 282 | And though thie the his wonded: sit of I so may'ss flatters? 283 | 284 | LEONTES: 285 | By your cheephers, with all with to make again. 286 | 287 | BRUTUS: 288 | Should thou hasp t 289 | 290 | 2017-09-06 01:41:32,849 - __main__ - INFO - epoch: 13, duration: 178s, loss: 1.4642. 291 | 2017-09-06 01:41:32,854 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 292 | 2017-09-06 01:41:32,855 - __main__ - INFO - generating 512 characters from top 10 choices. 293 | 2017-09-06 01:41:32,855 - __main__ - INFO - generating with seed: "my s". 294 | 2017-09-06 01:41:33,603 - __main__ - INFO - generated text: 295 | my some 296 | Within throw, which was being humest weese. 297 | 298 | Lile: 299 | Ay, fire hang beath a dray I deserved, 300 | Now hath looks brother, so should had to'er, 301 | And nor mine bold so mark the met to this 302 | heards abitter, that is shall have bid forty helwored, 303 | With whom this writ his witnood too the distor, 304 | And she watinny it other, when thy lord: 305 | And am speak, sir, my mother, that ipon the supen, 306 | Wishin and against the doth the spast. 307 | 308 | First Murderase: 309 | We we honestable be this no may him: 310 | But and true, thousand again to sword to's 311 | 312 | 2017-09-06 01:44:39,187 - __main__ - INFO - epoch: 14, duration: 185s, loss: 1.44983. 313 | 2017-09-06 01:44:39,192 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 314 | 2017-09-06 01:44:39,193 - __main__ - INFO - generating 512 characters from top 10 choices. 315 | 2017-09-06 01:44:39,193 - __main__ - INFO - generating with seed: "ve done,". 316 | 2017-09-06 01:44:40,028 - __main__ - INFO - generated text: 317 | ve done, 318 | If a much-moruise all to way that weech sein 319 | He bear no holders. Shall not have to be woes. 320 | 321 | CLARENCE: 322 | I plaugh in my hung in the miser's lumbly: 323 | As my fatcher's tongue, and shoulding with which, 324 | As any should ceep the common wife 325 | Whose chrust of teeph with she charge to the bay, 326 | I hid we had was mine that may'st so suffer 327 | Abonaties of that thy poor common sern 328 | Which you do that. 329 | 330 | CETRUTES: 331 | An again, my court,--it should contralion with write, or breathed 332 | What shall be mark to she truelents is hear of the 333 | 334 | 335 | 2017-09-06 01:47:42,093 - __main__ - INFO - epoch: 15, duration: 182s, loss: 1.43697. 336 | 2017-09-06 01:47:42,098 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 337 | 2017-09-06 01:47:42,099 - __main__ - INFO - generating 512 characters from top 10 choices. 338 | 2017-09-06 01:47:42,099 - __main__ - INFO - generating with seed: "atswain: 339 | Heigh, my hearts! cheer". 340 | 2017-09-06 01:47:42,891 - __main__ - INFO - generated text: 341 | atswain: 342 | Heigh, my hearts! cheery on tainnany so must. 343 | Mak'd with her scruption, I the soul and away, 344 | As too flatted to stree and chraised 345 | And so bliet, and these garse's sway, here. 346 | 347 | KING EDWARD IV: 348 | By the fatche in her faices will he, say. 349 | 350 | CORIOLANUS: 351 | Blinty have he women would I tell to more 352 | To taken an a solvounding by the blood? 353 | When we banish'd sould not abity, 354 | But by, traim with, with you healting seving, 355 | I had is that say thee hard obeth to my see? 356 | 357 | GLOUCESTER: 358 | Should not. 359 | 360 | GrOLINBES: 361 | Why, she, the stroken too not, my man 362 | I may s 363 | 364 | 2017-09-06 01:50:42,473 - __main__ - INFO - epoch: 16, duration: 179s, loss: 1.42547. 365 | 2017-09-06 01:50:42,478 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 366 | 2017-09-06 01:50:42,479 - __main__ - INFO - generating 512 characters from top 10 choices. 367 | 2017-09-06 01:50:42,479 - __main__ - INFO - generating with seed: "n 368 | To". 369 | 2017-09-06 01:50:43,317 - __main__ - INFO - generated text: 370 | n 371 | Torriff. Now then I mais not? 372 | 373 | BENVY: 374 | Succle you says and the grief to by him hour 375 | death all threathed in bostad from the coverade 376 | And to the counself old too, why tadignal' 377 | To see as I have her seemers whose slaughter; 378 | And news shall not such morn'd foold sour bow 379 | As merry came from his presendly shall be comess 380 | Of to shink writ to send her chounted; 381 | And will whits you dishonour bear with you, 382 | Fists it be spead fight boory cranst but 383 | Against somet's praise on him? them hid burst 384 | the part billing broken own b 385 | 386 | 2017-09-06 01:53:45,772 - __main__ - INFO - epoch: 17, duration: 182s, loss: 1.41502. 387 | 2017-09-06 01:53:45,777 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 388 | 2017-09-06 01:53:45,778 - __main__ - INFO - generating 512 characters from top 10 choices. 389 | 2017-09-06 01:53:45,778 - __main__ - INFO - generating with seed: "u pluck ". 390 | 2017-09-06 01:53:46,528 - __main__ - INFO - generated text: 391 | u pluck heard 392 | To have a tewps on the form of as minest? 393 | 394 | ROMEO: 395 | No, I am the seech misenous pick. 396 | 397 | DUKE VINCENTIO: 398 | I am the garke and along who, have boy; 399 | And, what is suppy so, the most sound me 400 | Were my sadys, if the stie unemicy 401 | A diserolad. 402 | 403 | PLORIST: 404 | Ares, then would he have her honours, seeming forbear; 405 | But' seem of the sunst on this atrees shall, 406 | The strokes the sea, with word and to hear which they are 407 | The striets; and supperished sold makate to as 408 | Buciond in appirize you that here her cookn wife here 409 | Whirt n 410 | 411 | 2017-09-06 01:56:46,788 - __main__ - INFO - epoch: 18, duration: 180s, loss: 1.40549. 412 | 2017-09-06 01:56:46,794 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 413 | 2017-09-06 01:56:46,794 - __main__ - INFO - generating 512 characters from top 10 choices. 414 | 2017-09-06 01:56:46,795 - __main__ - INFO - generating with seed: " patience! 415 | The statue is but new". 416 | 2017-09-06 01:56:47,608 - __main__ - INFO - generated text: 417 | patience! 418 | The statue is but new in his woes, what now was thy 419 | have thee in a piled, brave shest some awition! 420 | But twell thing would so state he spayes, such oath so 421 | For them of the falts. 422 | 423 | GRUMEO: 424 | Ay, through after if she' call, she boy to your 425 | doth take you thou that thas the bear no matter with 426 | the true to him some cannot be pict. 427 | 428 | GLOUCESETE: 429 | How wron this send he do with your son with him. 430 | 431 | BAPTISTA: 432 | No taulter, fetire how hath buse to my guest, 433 | Storecting how intectue weeping on him. 434 | 435 | GRUMIO: 436 | I have was the cold the princess, is you 437 | 438 | 2017-09-06 01:59:49,441 - __main__ - INFO - epoch: 19, duration: 181s, loss: 1.39675. 439 | 2017-09-06 01:59:49,447 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 440 | 2017-09-06 01:59:49,447 - __main__ - INFO - generating 512 characters from top 10 choices. 441 | 2017-09-06 01:59:49,448 - __main__ - INFO - generating with seed: "-lik". 442 | 2017-09-06 01:59:50,189 - __main__ - INFO - generated text: 443 | -lik: 444 | As setness, tell me, minither to all when 445 | the sensel's slanded solders indeech an in at the 446 | most fled unother. 447 | 448 | BASGONCO: 449 | Where I sadder, my heart? 450 | 451 | BIONDELLO: 452 | My nature, sir, thrints I crown to thy many 453 | And with the feach it on my crue. 454 | 455 | BAPTISTTA: 456 | Ary, and then will she throuned here: 457 | The censure after in the gods at a gho, 458 | But that a male-bution's speeced to't it. Food so doe 459 | An a tlanly and bands and lowers to-seat, 460 | To how me for in prince bound, 461 | And those more fooler word times fie him on 462 | That sweet 463 | 464 | 2017-09-06 02:02:52,746 - __main__ - INFO - epoch: 20, duration: 182s, loss: 1.38873. 465 | 2017-09-06 02:02:52,752 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 466 | 2017-09-06 02:02:52,752 - __main__ - INFO - generating 512 characters from top 10 choices. 467 | 2017-09-06 02:02:52,753 - __main__ - INFO - generating with seed: "d treble admonit". 468 | 2017-09-06 02:02:53,509 - __main__ - INFO - generated text: 469 | d treble admonity, 470 | So something servant again! 471 | 472 | CAPUS ELIZA: 473 | And thind I will speed, my horse my sworvess, 474 | And can like advaning with mine. 475 | 476 | DUKE OF YORK: 477 | Than thou: wears the lark'l days mine enemies 478 | Say thou best thousbers in the head to thy 479 | change't, which incinors is by, ip it old, 480 | And most close when tun to be acceid almisit, 481 | And herefore when now with wish your happy down 482 | see the parrion-brothen with honour. But, I down? 483 | 484 | BAPTISTA: 485 | Hearing with our brave, for as you a wind, 486 | And never for the string with his fool, 487 | The 488 | 489 | 2017-09-06 02:05:55,065 - __main__ - INFO - epoch: 21, duration: 181s, loss: 1.38131. 490 | 2017-09-06 02:05:55,070 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 491 | 2017-09-06 02:05:55,071 - __main__ - INFO - generating 512 characters from top 10 choices. 492 | 2017-09-06 02:05:55,071 - __main__ - INFO - generating with seed: "in rebellion wit". 493 | 2017-09-06 02:05:55,827 - __main__ - INFO - generated text: 494 | in rebellion with 495 | Welching your loss here here to my defend thee? 496 | 497 | KING RICHARD II: 498 | O concurn'd that, though that I shall bean. 499 | My lord, and murder more, this son,' therefore. 500 | 501 | DUKE VINCENTIO: 502 | Nor a door stunch'd my his will, to bid you 503 | Who hour cave her creams of their stature, 504 | My head with ouchip'd, by the banished, 505 | As if thou shake so fryswers in myself. 506 | Mercy this soul. Steeds and trance's fast, 507 | And answeal-took: 508 | The world, my lord in his house to death: 509 | Thust cannot do suppen of old me? him; the may'st here, 510 | And I sev 511 | 512 | 2017-09-06 02:08:58,411 - __main__ - INFO - epoch: 22, duration: 182s, loss: 1.3744. 513 | 2017-09-06 02:08:58,416 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 514 | 2017-09-06 02:08:58,417 - __main__ - INFO - generating 512 characters from top 10 choices. 515 | 2017-09-06 02:08:58,417 - __main__ - INFO - generating with seed: "isem". 516 | 2017-09-06 02:08:59,235 - __main__ - INFO - generated text: 517 | isem Ay least 518 | So slection show him, my son in this sorree. 519 | Well be burn me thou art thee: that it not be 520 | For we'll be the correw yourselves shall boldier, 521 | Is will-with the finser. I will you well not weech, 522 | His honour. 523 | 524 | PROSPERO: 525 | Hear my couchest I, as the man it, 526 | Whose drearm that's that spirits here is descent, 527 | To most son, with the heaver with subjects, 528 | As your sons with him for their courtenatures, 529 | So-day would not an eyes are not in myself. 530 | 531 | Second Citizen: 532 | This had thee, hurt! 533 | 534 | Second Murdieta: 535 | Why, stay, 536 | 537 | 2017-09-06 02:12:00,334 - __main__ - INFO - epoch: 23, duration: 181s, loss: 1.36802. 538 | 2017-09-06 02:12:00,339 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 539 | 2017-09-06 02:12:00,340 - __main__ - INFO - generating 512 characters from top 10 choices. 540 | 2017-09-06 02:12:00,340 - __main__ - INFO - generating with seed: "iege, 541 | You had only in your silen". 542 | 2017-09-06 02:12:01,604 - __main__ - INFO - generated text: 543 | iege, 544 | You had only in your silent sense, 545 | Believe shall no lady of tears by woo? 546 | I'll say it so. Camply, thou wilt not agis; 547 | And wise, my good command to my land, 548 | And will tonding should be with her leave. 549 | 550 | BAPTISTA: 551 | Not, not to with the fruely.- 552 | And took safes here in than blood it is time 553 | And trime of my heads; were my blood succease. 554 | 555 | LEONTES: 556 | Madam, for thee hath love's brows him of touch 557 | He pot, 558 | Since and husbing free more tames: but then; 559 | So speremimb me, which you be told the world, 560 | His heart. 561 | 562 | FLORIZEL: 563 | My, he! 564 | For I'll have see up 565 | 566 | 2017-09-06 02:15:04,811 - __main__ - INFO - epoch: 24, duration: 183s, loss: 1.36204. 567 | 2017-09-06 02:15:04,817 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 568 | 2017-09-06 02:15:04,817 - __main__ - INFO - generating 512 characters from top 10 choices. 569 | 2017-09-06 02:15:04,818 - __main__ - INFO - generating with seed: "do t". 570 | 2017-09-06 02:15:05,570 - __main__ - INFO - generated text: 571 | do the other heart: 572 | One whore hargins. 573 | 574 | CARILAN: 575 | A mother, sive midst thou statuains morn, 576 | A drem fry shop again: why toughts with a sea. 577 | O mercy him were to leave the wair my 578 | his acciverant my more, what is not sick says words 579 | Have you sword trained on thee? O. 580 | 581 | DUKE IVENTAR: 582 | It is yature in thine of my suglet, 583 | To touch our houses in the genal witneme, stay 584 | To be presently as here in to-day; 585 | The furious thou art one of the buts 586 | Was true other age is a dreampers' lave 587 | As wance's son and myself; thee of whom he 588 | 589 | 2017-09-06 02:18:08,507 - __main__ - INFO - epoch: 25, duration: 182s, loss: 1.3563. 590 | 2017-09-06 02:18:08,512 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 591 | 2017-09-06 02:18:08,513 - __main__ - INFO - generating 512 characters from top 10 choices. 592 | 2017-09-06 02:18:08,513 - __main__ - INFO - generating with seed: "hall". 593 | 2017-09-06 02:18:09,276 - __main__ - INFO - generated text: 594 | hall? 595 | What's that have to heard on me and hope, as I drink 596 | And shall they have honour for all to seal her. 597 | 598 | SEBRUTHARD: 599 | I thought you say's more forbor hopsing. 600 | 601 | LADY GREY: 602 | And I must-- 603 | 604 | BUCKINGHAM: 605 | How say you, so drum word, and not be to say his chooks 606 | To her suppets his provised and much to him! 607 | Making your heart, since the duke of a dream of honow; 608 | And I'll be as I hold up actrors, 609 | Which he haton, and wit moded here to how 610 | Hath poper-craitus subson the flesh would here, 611 | And who's that you, not we wrip to-mo 612 | 613 | 2017-09-06 02:21:12,507 - __main__ - INFO - epoch: 26, duration: 183s, loss: 1.35079. 614 | 2017-09-06 02:21:12,513 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 615 | 2017-09-06 02:21:12,513 - __main__ - INFO - generating 512 characters from top 10 choices. 616 | 2017-09-06 02:21:12,513 - __main__ - INFO - generating with seed: " 617 | LEONTES: 618 | Read t". 619 | 2017-09-06 02:21:13,284 - __main__ - INFO - generated text: 620 | 621 | LEONTES: 622 | Read to-nothout? 623 | 624 | First Servant: 625 | His been since! yet, I do no many for that 626 | Whits I seem my life,' it tumnly: farewelc, trench 627 | Whict in my man, away for there, this is a fulter, 628 | Which shall burn to see my pready, if youth me. 629 | 630 | PARIS: 631 | He would amfugly his soul well? 632 | 633 | LUCIO: 634 | Marral men from my day, I warrant of made? 635 | He, he, my master, bitter. Lany I woo 636 | How merey, these are tears and mischance wither 637 | all made a been, if his house, in the heart to be make 638 | but friends be scarce should save too: that tell thee subsho 639 | 640 | 2017-09-06 02:24:14,862 - __main__ - INFO - epoch: 27, duration: 181s, loss: 1.34554. 641 | 2017-09-06 02:24:14,867 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 642 | 2017-09-06 02:24:14,868 - __main__ - INFO - generating 512 characters from top 10 choices. 643 | 2017-09-06 02:24:14,868 - __main__ - INFO - generating with seed: "yo". 644 | 2017-09-06 02:24:15,619 - __main__ - INFO - generated text: 645 | yond, thanks 646 | Is think thou shalt be write and best fool to wind, 647 | And thou be in a most be almonous where. 648 | O, thou shouts, I will not shouth these slay of 649 | To his arm indeed sucked throst. Benk your drunk'd 650 | To make you weel show and duty, and being 651 | As man bears you to think a four oan's hunt, 652 | So thou days; what now. 653 | 654 | PETRUCHIO: 655 | Truns, for to see your beat, be then, sirror'd. 656 | 657 | KING RICHARD III: 658 | Stay, by the mongeried boy thy friend; 659 | I'lt sear, my good confity steed make me with her. 660 | 661 | BRUTUS: 662 | My saim'd, and there 663 | 664 | 2017-09-06 02:27:16,722 - __main__ - INFO - epoch: 28, duration: 181s, loss: 1.34055. 665 | 2017-09-06 02:27:16,727 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 666 | 2017-09-06 02:27:16,728 - __main__ - INFO - generating 512 characters from top 10 choices. 667 | 2017-09-06 02:27:16,728 - __main__ - INFO - generating with seed: "in her h". 668 | 2017-09-06 02:27:17,578 - __main__ - INFO - generated text: 669 | in her heard 670 | But something to the suppars? 671 | 672 | Shaths'd Myserge: 673 | Think thou dost were makes, their heart to make home. 674 | 675 | KING LEWIS XI: 676 | This said you are. 677 | 678 | First Musong 679 | Serving hither my land's straight 680 | Have young him, from thee: I among and though; 681 | And the priets indeed in the woe? 682 | 683 | BIONDELLO: 684 | You as their attation in a point. 685 | 686 | First Gentleman: 687 | I have benctle plainly that's hath bawd with yours: 688 | Which: 689 | Throng me; by your draw, and as in every: 690 | In whitheral drink? 691 | 692 | BRUTUS: 693 | Go, then tell him. 694 | 695 | SICINIUS: 696 | My sir. 697 | 698 | DUKE VI 699 | 700 | 2017-09-06 02:30:20,277 - __main__ - INFO - epoch: 29, duration: 182s, loss: 1.33579. 701 | 2017-09-06 02:30:20,283 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 702 | 2017-09-06 02:30:20,283 - __main__ - INFO - generating 512 characters from top 10 choices. 703 | 2017-09-06 02:30:20,283 - __main__ - INFO - generating with seed: "ofte". 704 | 2017-09-06 02:30:21,026 - __main__ - INFO - generated text: 705 | oftefrion. 706 | 707 | POLIXENES: 708 | Shall not so? come, what? 709 | Hath not in's own his death's blessly he was! 710 | 711 | Shepherd: 712 | Now, thou hadst thou wert hate my brack. 713 | 714 | CAPULET: 715 | Where weat you means to have her more here, 716 | To win abring, to be peninly, they shake here. 717 | I will not scope to they was long he did summer'd and 718 | Thy dissent hence. As if they be might. 719 | 720 | PRINV VI VINCENTIOSE: 721 | Art to the gittless sweet to yourself. 722 | 723 | CORIOLANUS: 724 | I have many's come of the thieffly as a solk, 725 | Stand so life and buy in my bids: the best man; 726 | For a 727 | 728 | 2017-09-06 02:33:22,918 - __main__ - INFO - epoch: 30, duration: 181s, loss: 1.33126. 729 | 2017-09-06 02:33:22,923 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 730 | 2017-09-06 02:33:22,924 - __main__ - INFO - generating 512 characters from top 10 choices. 731 | 2017-09-06 02:33:22,924 - __main__ - INFO - generating with seed: "s 732 | ". 733 | 2017-09-06 02:33:23,678 - __main__ - INFO - generated text: 734 | s 735 | TAUCESTha: 736 | Will, wulks this. There will seems in promotrial, 737 | With succession one that swear our presence? 738 | Why shall me so mink to a forwrar myself, 739 | Madam, and have welcome an in my death, 740 | But though again hand open in your daughter. 741 | If you thought his sparings of thee be fented 742 | That if thou a husband, throok, to brings here, 743 | Thou dost deep-morrow, hark me and look-choes 744 | I'll will send him things of meet to most consent. 745 | I do seem you throw on hither as it. 746 | 747 | ROMEO: 748 | I will nuchered and thou wast my curdness: 749 | 750 | 751 | 2017-09-06 02:36:25,458 - __main__ - INFO - epoch: 31, duration: 181s, loss: 1.32697. 752 | 2017-09-06 02:36:25,464 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 753 | 2017-09-06 02:36:25,464 - __main__ - INFO - generating 512 characters from top 10 choices. 754 | 2017-09-06 02:36:25,464 - __main__ - INFO - generating with seed: "S: 755 | T". 756 | 2017-09-06 02:36:26,226 - __main__ - INFO - generated text: 757 | S: 758 | There brinks me sweet my borns make my call with tweep. 759 | 760 | Proffers to your 761 | Bucizens Servantrlay: 762 | They cannot suppering thine at of drup 763 | Ap my tanner brots in sorrow therefore and, 764 | These bear thee to here is that would have born'd: 765 | My lord, he't thou art be wiskn here in the sit 766 | To the paid thy father come and firm. 767 | 768 | DUKE VINCENTIO: 769 | Why do the discharge our proverch'd in pity; 770 | For I that this cordly hate as your honour, 771 | And the sead and seem to what is wailed, 772 | Since he do loved father from him to the 773 | say: my b 774 | 775 | 2017-09-06 02:39:29,720 - __main__ - INFO - epoch: 32, duration: 183s, loss: 1.32285. 776 | 2017-09-06 02:39:29,725 - __main__ - INFO - model saved: checkpoints/pytorch_tinyshakespeare/model.ckpt. 777 | 2017-09-06 02:39:29,725 - __main__ - INFO - generating 512 characters from top 10 choices. 778 | 2017-09-06 02:39:29,726 - __main__ - INFO - generating with seed: " finger:". 779 | 2017-09-06 02:39:30,499 - __main__ - INFO - generated text: 780 | finger: 781 | We have so tongue senses well sound in him! 782 | 783 | CAPULET: 784 | Well, faster, thou as my honour for this two 785 | On'd himsel's shades!--Hay I have make the more. 786 | 787 | BRUTUS: 788 | The deliver soul's someworn. 789 | 790 | PETER: 791 | You and thou have money with me to my deep 792 | To betwixt you to believe use to be there, 793 | And, how slain. 794 | 795 | FRIAR LAURENCE: 796 | By honour! I cannot help as stands at 797 | Witing; awry with his cold from the wars offence 798 | How not to be as state a cordly and soles. 799 | 800 | PAULINA: 801 | Now, as I make home my for of York, 802 | Abad as your honour ha 803 | 804 | 2017-09-06 02:39:30,500 - __main__ - INFO - end of training, duration: 5868s. 805 | 2017-09-06 02:39:30,501 - __main__ - INFO - generating 1024 characters from top 3 choices. 806 | 2017-09-06 02:39:30,501 - __main__ - INFO - generating with seed: "t ma". 807 | 2017-09-06 02:39:31,974 - __main__ - INFO - generated text: 808 | t make 809 | And that we should be some son and so steal 810 | We hear to the prison our children with him. 811 | 812 | BIONDELLO: 813 | Then tell thee that sometime of the sea, 814 | To some would stand to see the world and help 815 | A man we shall not, thou shouldst stand a proving. 816 | 817 | DUKE VINCENTIO: 818 | Well any, take the worshelard and tender, 819 | And so much there stoond the compass with his countenance. 820 | 821 | BAGOT: 822 | If thou hast the chargain of the countrary 823 | Warding thee, thou art and to the presently 824 | To true, and we have a propeomle to see 825 | The son of the course of this soul strike 826 | And thou hast thou art this stooping and the sea, 827 | The winds and so with the child stay, 828 | To see me we was with thee and more with the world, 829 | And some so beseech you to this will be seen 830 | to tell him with a proper at our support. 831 | 832 | KING RICHARD III: 833 | And therefore there so much: should be the stones, 834 | And to make thy bosten of the storn the sea 835 | To blood time, and welcome of his son. 836 | 837 | DUKE VINCENTIO: 838 | Then shall be a pale of the sun of their subject, 839 | And the courts, thou shalt be seeming to 840 | -------------------------------------------------------------------------------- /mxnet_model.py: -------------------------------------------------------------------------------- 1 | import json 2 | import time 3 | 4 | from tqdm import tqdm 5 | 6 | import mxnet as mx 7 | import mxnet.ndarray as F 8 | import mxnet.gluon as gluon 9 | from mxnet.gluon import nn, rnn 10 | from mxnet import autograd 11 | 12 | from logger import get_logger 13 | from utils import (batch_generator, encode_text, generate_seed, ID2CHAR, main, 14 | make_dirs, sample_from_probs, VOCAB_SIZE) 15 | 16 | logger = get_logger(__name__) 17 | 18 | 19 | class Model(gluon.Block): 20 | """ 21 | build character embeddings LSTM text generation model. 22 | """ 23 | def __init__(self, vocab_size=VOCAB_SIZE, embedding_size=32, 24 | rnn_size=128, num_layers=2, drop_rate=0.0, **kwargs): 25 | super(Model, self).__init__(**kwargs) 26 | self.args = {"vocab_size": vocab_size, "embedding_size": embedding_size, 27 | "rnn_size": rnn_size, "num_layers": num_layers, 28 | "drop_rate": drop_rate} 29 | with self.name_scope(): 30 | self.encoder = nn.Embedding(vocab_size, embedding_size) 31 | self.dropout = nn.Dropout(drop_rate) 32 | self.rnn = rnn.LSTM(rnn_size, num_layers, dropout=drop_rate, 33 | input_size=embedding_size) 34 | self.decoder = nn.Dense(vocab_size, in_units=rnn_size) 35 | 36 | def forward(self, inputs, state): 37 | # input shape: [seq_len, batch_size] 38 | seq_len, batch_size = inputs.shape 39 | embed_seq = self.dropout(self.encoder(inputs)) 40 | # shape: [seq_len, batch_size, embedding_size] 41 | rnn_out, state = self.rnn(embed_seq, state) 42 | # rnn_out shape: [seq_len, batch_size, rnn_size] 43 | # hidden shape: [2, num_layers, batch_size, rnn_size] 44 | rnn_out = self.dropout(rnn_out) 45 | # shape: [seq_len, batch_size, rnn_size] 46 | logits = (self.decoder(rnn_out.reshape((-1, rnn_out.shape[2]))) 47 | .reshape((seq_len, batch_size, -1))) 48 | # output shape: [seq_len, batch_size, vocab_size] 49 | return logits, state 50 | 51 | def begin_state(self, batch_size=1): 52 | """ 53 | initialises rnn states. 54 | """ 55 | return self.rnn.begin_state(batch_size) 56 | 57 | def save(self, checkpoint_path): 58 | """ 59 | saves model and args to checkpoint_path. 60 | """ 61 | with open("{}.json".format(checkpoint_path), "w") as f: 62 | json.dump(self.args, f, indent=2) 63 | self.save_params(checkpoint_path) 64 | logger.info("model saved: %s.", checkpoint_path) 65 | 66 | @classmethod 67 | def load(cls, checkpoint_path, ctx=mx.cpu(), **kwargs): 68 | """ 69 | loads model from checkpoint_path. 70 | """ 71 | with open("{}.json".format(checkpoint_path)) as f: 72 | model_args = json.load(f) 73 | model = cls(**model_args, **kwargs) 74 | model.load_params(checkpoint_path, ctx) 75 | logger.info("model loaded: %s.", checkpoint_path) 76 | return model 77 | 78 | 79 | def generate_text(model, seed, length=512, top_n=10): 80 | """ 81 | generates text of specified length from trained model 82 | with given seed character sequence. 83 | """ 84 | logger.info("generating %s characters from top %s choices.", length, top_n) 85 | logger.info('generating with seed: "%s".', seed) 86 | generated = seed 87 | encoded = mx.nd.array(encode_text(seed)) 88 | seq_len = encoded.shape[0] 89 | 90 | x = F.expand_dims(encoded[:seq_len-1], 1) 91 | # input shape: [seq_len, 1] 92 | state = model.begin_state() 93 | # get rnn state due to seed sequence 94 | _, state = model(x, state) 95 | 96 | next_index = encoded[seq_len-1].asscalar() 97 | for i in range(length): 98 | x = mx.nd.array([[next_index]]) 99 | # input shape: [1, 1] 100 | logit, state = model(x, state) 101 | # output shape: [1, vocab_size] 102 | probs = F.softmax(logit) 103 | next_index = sample_from_probs(probs.asnumpy().squeeze(), top_n) 104 | # append to sequence 105 | generated += ID2CHAR[next_index] 106 | 107 | logger.info("generated text: \n%s\n", generated) 108 | return generated 109 | 110 | 111 | def train_main(args): 112 | """ 113 | trains model specfied in args. 114 | main method for train subcommand. 115 | """ 116 | # load text 117 | with open(args.text_path) as f: 118 | text = f.read() 119 | logger.info("corpus length: %s.", len(text)) 120 | 121 | # restore or build model 122 | if args.restore: 123 | logger.info("restoring model.") 124 | load_path = args.checkpoint_path if args.restore is True else args.restore 125 | model = Model.load(load_path) 126 | else: 127 | model = Model(vocab_size=VOCAB_SIZE, 128 | embedding_size=args.embedding_size, 129 | rnn_size=args.rnn_size, 130 | num_layers=args.num_layers, 131 | drop_rate=args.drop_rate) 132 | model.initialize(mx.init.Xavier()) 133 | model.hybridize() 134 | 135 | # make checkpoint directory 136 | make_dirs(args.checkpoint_path) 137 | model.save(args.checkpoint_path) 138 | 139 | # loss function 140 | loss = gluon.loss.SoftmaxCrossEntropyLoss(batch_axis=1) 141 | # optimizer 142 | optimizer = mx.optimizer.Adam(learning_rate=args.learning_rate, 143 | clip_gradient=args.clip_norm) 144 | # trainer 145 | trainer = gluon.Trainer(model.collect_params(), optimizer) 146 | 147 | # training start 148 | num_batches = (len(text) - 1) // (args.batch_size * args.seq_len) 149 | data_iter = batch_generator(encode_text(text), args.batch_size, args.seq_len) 150 | state = model.begin_state(args.batch_size) 151 | logger.info("start of training.") 152 | time_train = time.time() 153 | for i in range(args.num_epochs): 154 | epoch_losses = mx.nd.empty(num_batches) 155 | time_epoch = time.time() 156 | # training epoch 157 | for j in tqdm(range(num_batches), desc="epoch {}/{}".format(i + 1, args.num_epochs)): 158 | # prepare inputs 159 | x, y = next(data_iter) 160 | x = mx.nd.array(x.T) 161 | y = mx.nd.array(y.T) 162 | # reset state variables to remove their history 163 | state = [arr.detach() for arr in state] 164 | 165 | with autograd.record(): 166 | logits, state = model(x, state) 167 | # calculate loss 168 | L = loss(logits, y) 169 | L = F.mean(L) 170 | epoch_losses[j] = L.asscalar() 171 | # calculate gradient 172 | L.backward() 173 | # apply gradient update 174 | trainer.step(1) 175 | 176 | # logs 177 | duration_epoch = time.time() - time_epoch 178 | logger.info("epoch: %s, duration: %ds, loss: %.6g.", 179 | i + 1, duration_epoch, F.mean(epoch_losses).asscalar()) 180 | # checkpoint 181 | model.save_params(args.checkpoint_path) 182 | logger.info("model saved: %s.", args.checkpoint_path) 183 | # generate text 184 | seed = generate_seed(text) 185 | generate_text(model, seed) 186 | 187 | # training end 188 | duration_train = time.time() - time_train 189 | logger.info("end of training, duration: %ds.", duration_train) 190 | # generate text 191 | seed = generate_seed(text) 192 | generate_text(model, seed, 1024, 3) 193 | return model 194 | 195 | 196 | def generate_main(args): 197 | """ 198 | generates text from trained model specified in args. 199 | main method for generate subcommand. 200 | """ 201 | # load model 202 | inference_model = Model.load(args.checkpoint_path, mx.cpu()) 203 | 204 | # create seed if not specified 205 | if args.seed is None: 206 | with open(args.text_path) as f: 207 | text = f.read() 208 | seed = generate_seed(text) 209 | logger.info("seed sequence generated from %s.", args.text_path) 210 | else: 211 | seed = args.seed 212 | 213 | return generate_text(inference_model, seed, args.length, args.top_n) 214 | 215 | 216 | if __name__ == "__main__": 217 | main("MXNet", train_main, generate_main) 218 | -------------------------------------------------------------------------------- /pytorch_model.py: -------------------------------------------------------------------------------- 1 | import time 2 | 3 | from tqdm import tqdm 4 | 5 | import torch 6 | from torch import nn, optim 7 | from torch.autograd import Variable 8 | from torch.nn import functional as F 9 | 10 | from logger import get_logger 11 | from utils import (batch_generator, encode_text, generate_seed, ID2CHAR, main, 12 | make_dirs, VOCAB_SIZE) 13 | 14 | logger = get_logger(__name__) 15 | 16 | 17 | class Model(nn.Module): 18 | """ 19 | build character embeddings LSTM text generation model. 20 | """ 21 | def __init__(self, vocab_size=VOCAB_SIZE, embedding_size=32, 22 | rnn_size=128, num_layers=2, drop_rate=0.0): 23 | super(Model, self).__init__() 24 | self.args = {"vocab_size": vocab_size, "embedding_size": embedding_size, 25 | "rnn_size": rnn_size, "num_layers": num_layers, 26 | "drop_rate": drop_rate} 27 | self.encoder = nn.Embedding(vocab_size, embedding_size) 28 | self.dropout = nn.Dropout(drop_rate) 29 | self.rnn = nn.LSTM(embedding_size, rnn_size, num_layers, dropout=drop_rate) 30 | self.decoder = nn.Linear(rnn_size, vocab_size) 31 | 32 | def forward(self, inputs, state): 33 | # input shape: [seq_len, batch_size] 34 | embed_seq = self.dropout(self.encoder(inputs)) 35 | # shape: [seq_len, batch_size, embedding_size] 36 | rnn_out, state = self.rnn(embed_seq, state) 37 | # rnn_out shape: [seq_len, batch_size, rnn_size] 38 | # hidden shape: [2, num_layers, batch_size, rnn_size] 39 | rnn_out = self.dropout(rnn_out) 40 | # shape: [seq_len, batch_size, rnn_size] 41 | logits = self.decoder(rnn_out.view(-1, rnn_out.size(2))) 42 | # output shape: [seq_len * batch_size, vocab_size] 43 | return logits, state 44 | 45 | def predict(self, input, hidden): 46 | # input shape: [seq_len, batch_size] 47 | logits, hidden = self.forward(input, hidden) 48 | # logits shape: [seq_len * batch_size, vocab_size] 49 | # hidden shape: [2, num_layers, batch_size, rnn_size] 50 | probs = F.softmax(logits) 51 | # shape: [seq_len * batch_size, vocab_size] 52 | probs = probs.view(input.size(0), input.size(1), probs.size(1)) 53 | # output shape: [seq_len, batch_size, vocab_size] 54 | return probs, hidden 55 | 56 | def init_state(self, batch_size=1): 57 | """ 58 | initialises rnn states. 59 | """ 60 | return (Variable(torch.zeros(self.args["num_layers"], batch_size, self.args["rnn_size"])), 61 | Variable(torch.zeros(self.args["num_layers"], batch_size, self.args["rnn_size"]))) 62 | 63 | def save(self, checkpoint_path="model.ckpt"): 64 | """ 65 | saves model and args to checkpoint_path. 66 | """ 67 | checkpoint = {"args": self.args, "state_dict": self.state_dict()} 68 | with open(checkpoint_path, "wb") as f: 69 | torch.save(checkpoint, f) 70 | logger.info("model saved: %s.", checkpoint_path) 71 | 72 | @classmethod 73 | def load(cls, checkpoint_path): 74 | """ 75 | loads model from checkpoint_path. 76 | """ 77 | with open(checkpoint_path, "rb") as f: 78 | checkpoint = torch.load(f) 79 | model = cls(**checkpoint["args"]) 80 | model.load_state_dict(checkpoint["state_dict"]) 81 | logger.info("model loaded: %s.", checkpoint_path) 82 | return model 83 | 84 | 85 | def sample_from_probs(probs, top_n=10): 86 | """ 87 | truncated weighted random choice. 88 | """ 89 | _, indices = torch.sort(probs) 90 | # set probabilities after top_n to 0 91 | probs[indices.data[:-top_n]] = 0 92 | sampled_index = torch.multinomial(probs, 1) 93 | return sampled_index 94 | 95 | 96 | def generate_text(model, seed, length=512, top_n=10): 97 | """ 98 | generates text of specified length from trained model 99 | with given seed character sequence. 100 | """ 101 | logger.info("generating %s characters from top %s choices.", length, top_n) 102 | logger.info('generating with seed: "%s".', seed) 103 | generated = seed 104 | encoded = encode_text(seed) 105 | encoded = Variable(torch.from_numpy(encoded), volatile=True) 106 | model.eval() 107 | 108 | x = encoded[:-1].unsqueeze(1) 109 | # input shape: [seq_len, 1] 110 | state = model.init_state() 111 | # get rnn state due to seed sequence 112 | _, state = model.predict(x, state) 113 | 114 | next_index = encoded[-1:] 115 | for i in range(length): 116 | x = next_index.unsqueeze(1) 117 | # input shape: [1, 1] 118 | probs, state = model.predict(x, state) 119 | # output shape: [1, 1, vocab_size] 120 | next_index = sample_from_probs(probs.squeeze(), top_n) 121 | # append to sequence 122 | generated += ID2CHAR[next_index.data[0]] 123 | 124 | logger.info("generated text: \n%s\n", generated) 125 | return generated 126 | 127 | 128 | def train_main(args): 129 | """ 130 | trains model specfied in args. 131 | main method for train subcommand. 132 | """ 133 | # load text 134 | with open(args.text_path) as f: 135 | text = f.read() 136 | logger.info("corpus length: %s.", len(text)) 137 | 138 | # load or build model 139 | if args.restore: 140 | logger.info("restoring model.") 141 | load_path = args.checkpoint_path if args.restore is True else args.restore 142 | model = Model.load(load_path) 143 | else: 144 | model = Model(vocab_size=VOCAB_SIZE, 145 | embedding_size=args.embedding_size, 146 | rnn_size=args.rnn_size, 147 | num_layers=args.num_layers, 148 | drop_rate=args.drop_rate) 149 | 150 | # make checkpoint directory 151 | make_dirs(args.checkpoint_path) 152 | model.save(args.checkpoint_path) 153 | 154 | # loss function and optimizer 155 | criterion = nn.CrossEntropyLoss() 156 | optimizer = optim.Adam(model.parameters(), lr=args.learning_rate) 157 | 158 | # training start 159 | num_batches = (len(text) - 1) // (args.batch_size * args.seq_len) 160 | data_iter = batch_generator(encode_text(text), args.batch_size, args.seq_len) 161 | state = model.init_state(args.batch_size) 162 | logger.info("start of training.") 163 | time_train = time.time() 164 | 165 | for i in range(args.num_epochs): 166 | epoch_losses = torch.Tensor(num_batches) 167 | time_epoch = time.time() 168 | # training epoch 169 | for j in tqdm(range(num_batches), desc="epoch {}/{}".format(i + 1, args.num_epochs)): 170 | # prepare inputs 171 | x, y = next(data_iter) 172 | x = Variable(torch.from_numpy(x)).t() 173 | y = Variable(torch.from_numpy(y)).t().contiguous() 174 | # reset state variables to remove their history 175 | state = tuple([Variable(var.data) for var in state]) 176 | # prepare model 177 | model.train() 178 | model.zero_grad() 179 | # calculate loss 180 | logits, state = model.forward(x, state) 181 | loss = criterion(logits, y.view(-1)) 182 | epoch_losses[j] = loss.data[0] 183 | # calculate gradients 184 | loss.backward() 185 | # clip gradient norm 186 | nn.utils.clip_grad_norm(model.parameters(), args.clip_norm) 187 | # apply gradient update 188 | optimizer.step() 189 | 190 | # logs 191 | duration_epoch = time.time() - time_epoch 192 | logger.info("epoch: %s, duration: %ds, loss: %.6g.", 193 | i + 1, duration_epoch, epoch_losses.mean()) 194 | # checkpoint 195 | model.save(args.checkpoint_path) 196 | # generate text 197 | seed = generate_seed(text) 198 | generate_text(model, seed) 199 | 200 | # training end 201 | duration_train = time.time() - time_train 202 | logger.info("end of training, duration: %ds.", duration_train) 203 | # generate text 204 | seed = generate_seed(text) 205 | generate_text(model, seed, 1024, 3) 206 | return model 207 | 208 | 209 | def generate_main(args): 210 | """ 211 | generates text from trained model specified in args. 212 | main method for generate subcommand. 213 | """ 214 | # load model 215 | inference_model = Model.load(args.checkpoint_path) 216 | 217 | # create seed if not specified 218 | if args.seed is None: 219 | with open(args.text_path) as f: 220 | text = f.read() 221 | seed = generate_seed(text) 222 | logger.info("seed sequence generated from %s.", args.text_path) 223 | else: 224 | seed = args.seed 225 | 226 | return generate_text(inference_model, seed, args.length, args.top_n) 227 | 228 | 229 | if __name__ == "__main__": 230 | main("PyTorch", train_main, generate_main) 231 | -------------------------------------------------------------------------------- /scripts.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # keras 3 | python keras_model.py train --checkpoint=checkpoints/keras_tinyshakespeare/model.hdf5 --text=data/tinyshakespeare.txt 4 | python keras_model.py generate --checkpoint=checkpoints/keras_tinyshakespeare/model.hdf5 --text=data/tinyshakespeare.txt 5 | 6 | python keras_model.py train --checkpoint=checkpoints/keras_shakespeare/model.hdf5 --text=data/shakespeare.txt 7 | python keras_model.py generate --checkpoint=checkpoints/keras_shakespeare/model.hdf5 --text=data/shakespeare.txt 8 | 9 | # tensorflow 10 | python tf_model.py train --checkpoint=checkpoints/tf_tinyshakespeare/model.ckpt --text=data/tinyshakespeare.txt 11 | python tf_model.py generate --checkpoint=checkpoints/tf_tinyshakespeare/model.ckpt --text=data/tinyshakespeare.txt 12 | 13 | python tf_model.py train --checkpoint=checkpoints/tf_shakespeare/model.ckpt --text=data/shakespeare.txt 14 | python tf_model.py generate --checkpoint=checkpoints/tf_shakespeare/model.ckpt --text=data/shakespeare.txt 15 | 16 | # pytorch 17 | python pytorch_model.py train --checkpoint=checkpoints/pytorch_tinyshakespeare/model.ckpt --text=data/tinyshakespeare.txt 18 | python pytorch_model.py generate --checkpoint=checkpoints/pytorch_tinyshakespeare/model.ckpt --text=data/tinyshakespeare.txt 19 | 20 | python pytorch_model.py train --checkpoint=checkpoints/pytorch_shakespeare/model.ckpt --text=data/shakespeare.txt 21 | python pytorch_model.py generate --checkpoint=checkpoints/pytorch_shakespeare/model.ckpt --text=data/shakespeare.txt 22 | 23 | # chainer 24 | python chainer_model.py train --checkpoint=checkpoints/chainer_tinyshakespeare/model.npz --text=data/tinyshakespeare.txt 25 | python chainer_model.py generate --checkpoint=checkpoints/chainer_tinyshakespeare/model.npz --text=data/tinyshakespeare.txt 26 | 27 | python chainer_model.py train --checkpoint=checkpoints/chainer_shakespeare/model.npz --text=data/shakespeare.txt 28 | python chainer_model.py generate --checkpoint=checkpoints/chainer_shakespeare/model.npz --text=data/shakespeare.txt 29 | 30 | # mxnet 31 | python mxnet_model.py train --checkpoint=checkpoints/mxnet_tinyshakespeare/model.params --text=data/tinyshakespeare.txt 32 | python mxnet_model.py generate --checkpoint=checkpoints/mxnet_tinyshakespeare/model.params --text=data/tinyshakespeare.txt 33 | 34 | python mxnet_model.py train --checkpoint=checkpoints/mxnet_shakespeare/model.params --text=data/shakespeare.txt 35 | python mxnet_model.py generate --checkpoint=checkpoints/mxnet_shakespeare/model.params --text=data/shakespeare.txt 36 | -------------------------------------------------------------------------------- /tf_model.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import time 4 | 5 | import numpy as np 6 | from tqdm import tqdm 7 | 8 | import tensorflow as tf 9 | from tensorflow.contrib import layers, rnn 10 | from tensorflow.contrib.tensorboard.plugins import projector 11 | 12 | from logger import get_logger 13 | from utils import (batch_generator, encode_text, generate_seed, ID2CHAR, main, 14 | make_dirs, sample_from_probs, VOCAB_SIZE) 15 | 16 | logger = get_logger(__name__) 17 | 18 | 19 | def build_infer_graph(x, batch_size, vocab_size=VOCAB_SIZE, embedding_size=32, 20 | rnn_size=128, num_layers=2, p_keep=1.0): 21 | """ 22 | builds inference graph 23 | """ 24 | infer_args = {"batch_size": batch_size, "vocab_size": vocab_size, 25 | "embedding_size": embedding_size, "rnn_size": rnn_size, 26 | "num_layers": num_layers, "p_keep": p_keep} 27 | logger.debug("building inference graph: %s.", infer_args) 28 | 29 | # other placeholders 30 | p_keep = tf.placeholder_with_default(p_keep, [], "p_keep") 31 | batch_size = tf.placeholder_with_default(batch_size, [], "batch_size") 32 | 33 | # embedding layer 34 | embed_seq = layers.embed_sequence(x, vocab_size, embedding_size) 35 | # shape: [batch_size, seq_len, embedding_size] 36 | embed_seq = tf.nn.dropout(embed_seq, keep_prob=p_keep) 37 | # shape: [batch_size, seq_len, embedding_size] 38 | 39 | # RNN layers 40 | cells = [rnn.LSTMCell(rnn_size) for _ in range(num_layers)] 41 | cells = [rnn.DropoutWrapper(cell, output_keep_prob=p_keep) for cell in cells] 42 | cells = rnn.MultiRNNCell(cells) 43 | input_state = cells.zero_state(batch_size, tf.float32) 44 | # shape: [num_layers, 2, batch_size, rnn_size] 45 | rnn_out, output_state = tf.nn.dynamic_rnn(cells, embed_seq, initial_state=input_state) 46 | # rnn_out shape: [batch_size, seq_len, rnn_size] 47 | # output_state shape: [num_layers, 2, batch_size, rnn_size] 48 | with tf.name_scope("lstm"): 49 | tf.summary.histogram("outputs", rnn_out) 50 | for c_state, h_state in output_state: 51 | tf.summary.histogram("c_state", c_state) 52 | tf.summary.histogram("h_state", h_state) 53 | 54 | # fully connected layer 55 | logits = layers.fully_connected(rnn_out, vocab_size, activation_fn=None) 56 | # shape: [batch_size, seq_len, vocab_size] 57 | 58 | # predictions 59 | with tf.name_scope("softmax"): 60 | probs = tf.nn.softmax(logits) 61 | # shape: [batch_size, seq_len, vocab_size] 62 | 63 | with tf.name_scope("sequence"): 64 | tf.summary.histogram("embeddings", embed_seq) 65 | tf.summary.histogram("logits", logits) 66 | 67 | model = {"logits": logits, "probs": probs, 68 | "input_state": input_state, "output_state": output_state, 69 | "p_keep": p_keep, "batch_size": batch_size, "infer_args": infer_args} 70 | return model 71 | 72 | 73 | def build_eval_graph(logits, y): 74 | """ 75 | builds evaluation graph 76 | """ 77 | eval_args = {} 78 | logger.debug("building evaluation graph: %s.", eval_args) 79 | 80 | with tf.name_scope("loss"): 81 | loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y) 82 | # shape: [batch_size, seq_len] 83 | seq_loss = tf.reduce_mean(loss, axis=1, name="seq_loss") 84 | # shape: [batch_size] 85 | batch_loss = tf.reduce_mean(seq_loss, name="batch_loss") 86 | # shape: [] 87 | tf.summary.histogram("sequence", seq_loss) 88 | 89 | model = {"loss": batch_loss, "eval_args": eval_args} 90 | return model 91 | 92 | 93 | def build_train_graph(loss, learning_rate=0.001, clip_norm=5.0): 94 | """ 95 | builds training graph 96 | """ 97 | train_args = {"learning_rate": learning_rate, "clip_norm": clip_norm} 98 | logger.debug("building training graph: %s.", train_args) 99 | 100 | learning_rate = tf.placeholder_with_default(learning_rate, [], "learning_rate") 101 | global_step = tf.Variable(0, name='global_step', trainable=False) 102 | train_op = layers.optimize_loss(loss, global_step, learning_rate, "Adam", 103 | clip_gradients=clip_norm) 104 | 105 | model = {"global_step": global_step, "train_op": train_op, 106 | "learning_rate": learning_rate, "train_args": train_args} 107 | return model 108 | 109 | 110 | def build_model(batch_size, vocab_size=VOCAB_SIZE, embedding_size=32, 111 | rnn_size=128, num_layers=2, p_keep=1.0, learning_rate=0.001, 112 | clip_norm=5.0, build_eval=True, build_train=True): 113 | """ 114 | builds model end-to-end, including data placeholders and saver 115 | """ 116 | model_args = {"batch_size": batch_size, "vocab_size": vocab_size, 117 | "embedding_size": embedding_size, "rnn_size": rnn_size, 118 | "num_layers": num_layers, "p_keep": p_keep, 119 | "learning_rate": learning_rate, "clip_norm": clip_norm} 120 | logger.info("building model: %s.", model_args) 121 | 122 | # data placeholders 123 | x = tf.placeholder(tf.int32, [None, None], "X") 124 | # shape: [batch_size, seq_len] 125 | y = tf.placeholder(tf.int32, [None, None], "Y") 126 | # shape: [batch_size, seq_len] 127 | 128 | model = {"X": x, "Y": y, "args": model_args} 129 | model.update(build_infer_graph(model["X"], 130 | batch_size=batch_size, 131 | vocab_size=VOCAB_SIZE, 132 | embedding_size=embedding_size, 133 | rnn_size=rnn_size, 134 | num_layers=num_layers, 135 | p_keep=p_keep)) 136 | if build_eval or build_train: 137 | model.update(build_eval_graph(model["logits"], model["Y"])) 138 | if build_train: 139 | model.update(build_train_graph(model["loss"], 140 | learning_rate=learning_rate, 141 | clip_norm=clip_norm)) 142 | # init op 143 | model["init_op"] = tf.global_variables_initializer() 144 | # tensorboard summary 145 | model["summary"] = tf.summary.merge_all() 146 | # saver 147 | model["saver"] = tf.train.Saver() 148 | return model 149 | 150 | 151 | def load_inference_model(checkpoint_path): 152 | """ 153 | builds inference model from model args saved in `checkpoint_path` 154 | """ 155 | # load model args 156 | with open("{}.json".format(checkpoint_path)) as f: 157 | model_args = json.load(f) 158 | # edit batch_size and p_keep 159 | model_args.update({"batch_size": 1, "p_keep": 1.0}) 160 | infer_model = build_model(**model_args, build_eval=False, build_train=False) 161 | logger.info("inference model loaded: %s.", checkpoint_path) 162 | return infer_model 163 | 164 | 165 | def generate_text(model, sess, seed, length=512, top_n=10): 166 | """ 167 | generates text of specified length from trained model 168 | with given seed character sequence. 169 | """ 170 | logger.info("generating %s characters from top %s choices.", length, top_n) 171 | logger.info('generating with seed: "%s".', seed) 172 | generated = seed 173 | encoded = encode_text(seed) 174 | 175 | x = np.expand_dims(encoded[:-1], 0) 176 | # input shape: [1, seq_len] 177 | # get rnn state due to seed sequence 178 | state = sess.run(model["output_state"], feed_dict={model["X"]: x}) 179 | 180 | next_index = encoded[-1] 181 | for i in range(length): 182 | x = np.array([[next_index]]) 183 | # input shape: [1, 1] 184 | feed_dict = {model["X"]: x, model["input_state"]: state} 185 | probs, state = sess.run([model["probs"], model["output_state"]], feed_dict=feed_dict) 186 | # output shape: [1, 1, vocab_size] 187 | next_index = sample_from_probs(probs.squeeze(), top_n) 188 | # append to sequence 189 | generated += ID2CHAR[next_index] 190 | 191 | logger.info("generated text: \n%s\n", generated) 192 | return generated 193 | 194 | 195 | def train_main(args): 196 | """ 197 | trains model specfied in args. 198 | main method for train subcommand. 199 | """ 200 | # load text 201 | with open(args.text_path) as f: 202 | text = f.read() 203 | logger.info("corpus length: %s.", len(text)) 204 | 205 | # restore or build model 206 | if args.restore: 207 | load_path = args.checkpoint_path if args.restore is True else args.restore 208 | with open("{}.json".format(args.checkpoint_path)) as f: 209 | model_args = json.load(f) 210 | logger.info("model restored: %s.", load_path) 211 | else: 212 | load_path = None 213 | model_args = {"batch_size": args.batch_size, 214 | "vocab_size": VOCAB_SIZE, 215 | "embedding_size": args.embedding_size, 216 | "rnn_size": args.rnn_size, 217 | "num_layers": args.num_layers, 218 | "p_keep": 1 - args.drop_rate, 219 | "learning_rate": args.learning_rate, 220 | "clip_norm": args.clip_norm} 221 | 222 | # build train model 223 | train_graph = tf.Graph() 224 | with train_graph.as_default(): 225 | train_model = build_model(**model_args) 226 | 227 | with tf.Session(graph=train_graph) as train_sess: 228 | # restore or initialise model weights 229 | if load_path is not None: 230 | train_model["saver"].restore(train_sess, load_path) 231 | logger.info("model weights restored: %s.", load_path) 232 | else: 233 | train_sess.run(train_model["init_op"]) 234 | 235 | # clear checkpoint directory 236 | log_dir = make_dirs(args.checkpoint_path, empty=True) 237 | # save model 238 | with open("{}.json".format(args.checkpoint_path), "w") as f: 239 | json.dump(train_model["args"], f, indent=2) 240 | checkpoint_path = train_model["saver"].save(train_sess, args.checkpoint_path) 241 | logger.info("model saved: %s.", checkpoint_path) 242 | # tensorboard logger 243 | summary_writer = tf.summary.FileWriter(log_dir, train_sess.graph) 244 | # embeddings visualisation 245 | config = projector.ProjectorConfig() 246 | embedding = config.embeddings.add() 247 | embedding.tensor_name = "EmbedSequence/embeddings" 248 | embedding.metadata_path = os.path.abspath(os.path.join("data", "id2char.tsv")) 249 | projector.visualize_embeddings(summary_writer, config) 250 | logger.info("tensorboard set up.") 251 | 252 | # build infer model 253 | inference_graph = tf.Graph() 254 | with inference_graph.as_default(): 255 | inference_model = load_inference_model(args.checkpoint_path) 256 | 257 | # training start 258 | num_batches = (len(text) - 1) // (args.batch_size * args.seq_len) 259 | data_iter = batch_generator(encode_text(text), args.batch_size, args.seq_len) 260 | fetches = [train_model["train_op"], train_model["output_state"], 261 | train_model["loss"], train_model["summary"]] 262 | state = train_sess.run(train_model["input_state"]) 263 | logger.info("start of training.") 264 | time_train = time.time() 265 | 266 | for i in range(args.num_epochs): 267 | epoch_losses = np.empty(num_batches) 268 | time_epoch = time.time() 269 | # training epoch 270 | for j in tqdm(range(num_batches), desc="epoch {}/{}".format(i + 1, args.num_epochs)): 271 | x, y = next(data_iter) 272 | feed_dict = {train_model["X"]: x, train_model["Y"]: y, train_model["input_state"]: state} 273 | _, state, loss, summary_log = train_sess.run(fetches, feed_dict) 274 | epoch_losses[j] = loss 275 | 276 | # logs 277 | duration_epoch = time.time() - time_epoch 278 | logger.info("epoch: %s, duration: %ds, loss: %.6g.", 279 | i + 1, duration_epoch, epoch_losses.mean()) 280 | # tensorboard logs 281 | summary_writer.add_summary(summary_log, i + 1) 282 | summary_writer.flush() 283 | # checkpoint 284 | checkpoint_path = train_model["saver"].save(train_sess, args.checkpoint_path) 285 | logger.info("model saved: %s.", checkpoint_path) 286 | 287 | # generate text 288 | seed = generate_seed(text) 289 | with tf.Session(graph=inference_graph) as infer_sess: 290 | # restore weights 291 | inference_model["saver"].restore(infer_sess, checkpoint_path) 292 | generate_text(inference_model, infer_sess, seed) 293 | 294 | # training end 295 | duration_train = time.time() - time_train 296 | logger.info("end of training, duration: %ds.", duration_train) 297 | # generate text 298 | seed = generate_seed(text) 299 | with tf.Session(graph=inference_graph) as infer_sess: 300 | # restore weights 301 | inference_model["saver"].restore(infer_sess, checkpoint_path) 302 | generate_text(inference_model, infer_sess, seed, 1024, 3) 303 | 304 | return train_model 305 | 306 | 307 | def generate_main(args): 308 | """ 309 | generates text from trained model specified in args. 310 | main method for generate subcommand. 311 | """ 312 | # restore model 313 | inference_graph = tf.Graph() 314 | with inference_graph.as_default(): 315 | inference_model = load_inference_model(args.checkpoint_path) 316 | 317 | # create seed if not specified 318 | if args.seed is None: 319 | with open(args.text_path) as f: 320 | text = f.read() 321 | seed = generate_seed(text) 322 | logger.info("seed sequence generated from %s.", args.text_path) 323 | else: 324 | seed = args.seed 325 | 326 | with tf.Session(graph=inference_graph) as infer_sess: 327 | # restore weights 328 | inference_model["saver"].restore(infer_sess, args.checkpoint_path) 329 | return generate_text(inference_model, infer_sess, seed, args.length, args.top_n) 330 | 331 | 332 | if __name__ == "__main__": 333 | main("TensorFlow", train_main, generate_main) 334 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | from argparse import ArgumentParser 2 | import os 3 | import random 4 | import string 5 | import sys 6 | 7 | import numpy as np 8 | 9 | from logger import get_logger 10 | 11 | logger = get_logger(__name__) 12 | 13 | ### 14 | # file system 15 | ### 16 | 17 | 18 | def make_dirs(path, empty=False): 19 | """ 20 | create dir in path and clear dir if required 21 | """ 22 | dir_path = os.path.dirname(path) 23 | os.makedirs(dir_path, exist_ok=True) 24 | 25 | if empty: 26 | files = [os.path.join(dir_path, item) for item in os.listdir(dir_path)] 27 | for item in files: 28 | if os.path.isfile(item): 29 | os.remove(item) 30 | 31 | return dir_path 32 | 33 | 34 | def path_join(*paths, empty=False): 35 | """ 36 | join paths and create dir 37 | """ 38 | path = os.path.abspath(os.path.join(*paths)) 39 | make_dirs(os.path.dirname(path), empty) 40 | return path 41 | 42 | 43 | ### 44 | # data processing 45 | ### 46 | 47 | def create_dictionary(): 48 | """ 49 | create char2id, id2char and vocab_size 50 | from printable ascii characters. 51 | """ 52 | chars = sorted(ch for ch in string.printable if ch not in ("\x0b", "\x0c", "\r")) 53 | char2id = dict((ch, i + 1) for i, ch in enumerate(chars)) 54 | char2id.update({"": 0}) 55 | id2char = dict((char2id[ch], ch) for ch in char2id) 56 | vocab_size = len(char2id) 57 | return char2id, id2char, vocab_size 58 | 59 | CHAR2ID, ID2CHAR, VOCAB_SIZE = create_dictionary() 60 | 61 | 62 | def encode_text(text, char2id=CHAR2ID): 63 | """ 64 | encode text to array of integers with CHAR2ID 65 | """ 66 | return np.fromiter((char2id.get(ch, 0) for ch in text), int) 67 | 68 | 69 | def decode_text(int_array, id2char=ID2CHAR): 70 | """ 71 | decode array of integers to text with ID2CHAR 72 | """ 73 | return "".join((id2char[ch] for ch in int_array)) 74 | 75 | 76 | def one_hot_encode(indices, num_classes): 77 | """ 78 | one-hot encoding 79 | """ 80 | return np.eye(num_classes)[indices] 81 | 82 | 83 | def batch_generator(sequence, batch_size=64, seq_len=64, one_hot_features=False, one_hot_labels=False): 84 | """ 85 | batch generator for sequence 86 | ensures that batches generated are continuous along axis 1 87 | so that hidden states can be kept across batches and epochs 88 | """ 89 | # calculate effective length of text to use 90 | num_batches = (len(sequence) - 1) // (batch_size * seq_len) 91 | if num_batches == 0: 92 | raise ValueError("No batches created. Use smaller batch size or sequence length.") 93 | logger.info("number of batches: %s.", num_batches) 94 | rounded_len = num_batches * batch_size * seq_len 95 | logger.info("effective text length: %s.", rounded_len) 96 | 97 | x = np.reshape(sequence[: rounded_len], [batch_size, num_batches * seq_len]) 98 | if one_hot_features: 99 | x = one_hot_encode(x, VOCAB_SIZE) 100 | logger.info("x shape: %s.", x.shape) 101 | 102 | y = np.reshape(sequence[1: rounded_len + 1], [batch_size, num_batches * seq_len]) 103 | if one_hot_labels: 104 | y = one_hot_encode(y, VOCAB_SIZE) 105 | logger.info("y shape: %s.", y.shape) 106 | 107 | epoch = 0 108 | while True: 109 | # roll so that no need to reset rnn states over epochs 110 | x_epoch = np.split(np.roll(x, -epoch, axis=0), num_batches, axis=1) 111 | y_epoch = np.split(np.roll(y, -epoch, axis=0), num_batches, axis=1) 112 | for batch in range(num_batches): 113 | yield x_epoch[batch], y_epoch[batch] 114 | epoch += 1 115 | 116 | 117 | ### 118 | # text generation 119 | ### 120 | 121 | def generate_seed(text, seq_lens=(2, 4, 8, 16, 32)): 122 | """ 123 | select subsequence randomly from input text 124 | """ 125 | # randomly choose sequence length 126 | seq_len = random.choice(seq_lens) 127 | # randomly choose start index 128 | start_index = random.randint(0, len(text) - seq_len - 1) 129 | seed = text[start_index: start_index + seq_len] 130 | return seed 131 | 132 | 133 | def sample_from_probs(probs, top_n=10): 134 | """ 135 | truncated weighted random choice. 136 | """ 137 | # need 64 floating point precision 138 | probs = np.array(probs, dtype=np.float64) 139 | # set probabilities after top_n to 0 140 | probs[np.argsort(probs)[:-top_n]] = 0 141 | # renormalise probabilities 142 | probs /= np.sum(probs) 143 | sampled_index = np.random.choice(len(probs), p=probs) 144 | return sampled_index 145 | 146 | 147 | ### 148 | # main 149 | ### 150 | 151 | def main(framework, train_main, generate_main): 152 | arg_parser = ArgumentParser( 153 | description="{} character embeddings LSTM text generation model.".format(framework)) 154 | subparsers = arg_parser.add_subparsers(title="subcommands") 155 | 156 | # train args 157 | train_parser = subparsers.add_parser("train", help="train model on text file") 158 | train_parser.add_argument("--checkpoint-path", required=True, 159 | help="path to save or load model checkpoints (required)") 160 | train_parser.add_argument("--text-path", required=True, 161 | help="path of text file for training (required)") 162 | train_parser.add_argument("--restore", nargs="?", default=False, const=True, 163 | help="whether to restore from checkpoint_path " 164 | "or from another path if specified") 165 | train_parser.add_argument("--seq-len", type=int, default=64, 166 | help="sequence length of inputs and outputs (default: %(default)s)") 167 | train_parser.add_argument("--embedding-size", type=int, default=32, 168 | help="character embedding size (default: %(default)s)") 169 | train_parser.add_argument("--rnn-size", type=int, default=128, 170 | help="size of rnn cell (default: %(default)s)") 171 | train_parser.add_argument("--num-layers", type=int, default=2, 172 | help="number of rnn layers (default: %(default)s)") 173 | train_parser.add_argument("--drop-rate", type=float, default=0., 174 | help="dropout rate for rnn layers (default: %(default)s)") 175 | train_parser.add_argument("--learning-rate", type=float, default=0.001, 176 | help="learning rate (default: %(default)s)") 177 | train_parser.add_argument("--clip-norm", type=float, default=5., 178 | help="max norm to clip gradient (default: %(default)s)") 179 | train_parser.add_argument("--batch-size", type=int, default=64, 180 | help="training batch size (default: %(default)s)") 181 | train_parser.add_argument("--num-epochs", type=int, default=32, 182 | help="number of epochs for training (default: %(default)s)") 183 | train_parser.add_argument("--log-path", default=os.path.join(os.path.dirname(__file__), "main.log"), 184 | help="path of log file (default: %(default)s)") 185 | train_parser.set_defaults(main=train_main) 186 | 187 | # generate args 188 | generate_parser = subparsers.add_parser("generate", help="generate text from trained model") 189 | generate_parser.add_argument("--checkpoint-path", required=True, 190 | help="path to load model checkpoints (required)") 191 | group = generate_parser.add_mutually_exclusive_group(required=True) 192 | group.add_argument("--text-path", help="path of text file to generate seed") 193 | group.add_argument("--seed", default=None, help="seed character sequence") 194 | generate_parser.add_argument("--length", type=int, default=1024, 195 | help="length of character sequence to generate (default: %(default)s)") 196 | generate_parser.add_argument("--top-n", type=int, default=3, 197 | help="number of top choices to sample (default: %(default)s)") 198 | generate_parser.add_argument("--log-path", default=os.path.join(os.path.dirname(__file__), "main.log"), 199 | help="path of log file (default: %(default)s)") 200 | generate_parser.set_defaults(main=generate_main) 201 | 202 | args = arg_parser.parse_args() 203 | get_logger("__main__", log_path=args.log_path, console=True) 204 | logger = get_logger(__name__, log_path=args.log_path, console=True) 205 | logger.debug("call: %s", " ".join(sys.argv)) 206 | logger.debug("ArgumentParser: %s", args) 207 | 208 | try: 209 | args.main(args) 210 | except Exception as e: 211 | logger.exception(e) 212 | --------------------------------------------------------------------------------