├── .dockerignore ├── .gitignore ├── LICENSE ├── README.md ├── data ├── pregenerated_training_data │ ├── epoch_0.json │ ├── epoch_0_metrics.json │ ├── epoch_1.json │ └── epoch_1_metrics.json └── sentences_150k.txt ├── pretrain.py ├── pytorch_transformers_lm_finetuning └── pregenerate_training_data.py ├── requirements.txt ├── scripts └── sync_checkpoint_to_s3.py └── utils.py /.dockerignore: -------------------------------------------------------------------------------- 1 | 2 | tmp 3 | data 4 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/README.md -------------------------------------------------------------------------------- /data/pregenerated_training_data/epoch_0.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/data/pregenerated_training_data/epoch_0.json -------------------------------------------------------------------------------- /data/pregenerated_training_data/epoch_0_metrics.json: -------------------------------------------------------------------------------- 1 | {"num_training_examples": 10572, "max_seq_len": 512} -------------------------------------------------------------------------------- /data/pregenerated_training_data/epoch_1.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/data/pregenerated_training_data/epoch_1.json -------------------------------------------------------------------------------- /data/pregenerated_training_data/epoch_1_metrics.json: -------------------------------------------------------------------------------- 1 | {"num_training_examples": 9520, "max_seq_len": 512} -------------------------------------------------------------------------------- /data/sentences_150k.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/data/sentences_150k.txt -------------------------------------------------------------------------------- /pretrain.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/pretrain.py -------------------------------------------------------------------------------- /pytorch_transformers_lm_finetuning/pregenerate_training_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/pytorch_transformers_lm_finetuning/pregenerate_training_data.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pytorch-transformers==1.2.0 2 | natsort 3 | tqdm 4 | -------------------------------------------------------------------------------- /scripts/sync_checkpoint_to_s3.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/scripts/sync_checkpoint_to_s3.py -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/allenai/tpu_pretrain/HEAD/utils.py --------------------------------------------------------------------------------