├── .gitignore ├── LICENSE ├── README.md ├── demo_mlm.py ├── demo_mlm.sh ├── demo_mlm_hf.sh ├── demo_train_mlm_hf.sh ├── modeling_flash_roberta.py ├── requirements.txt └── run_mlm.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/README.md -------------------------------------------------------------------------------- /demo_mlm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/demo_mlm.py -------------------------------------------------------------------------------- /demo_mlm.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/demo_mlm.sh -------------------------------------------------------------------------------- /demo_mlm_hf.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/demo_mlm_hf.sh -------------------------------------------------------------------------------- /demo_train_mlm_hf.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/demo_train_mlm_hf.sh -------------------------------------------------------------------------------- /modeling_flash_roberta.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/modeling_flash_roberta.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | torch>=1.13.0 2 | transformers>=4.20.0 3 | flash-attn==2.0.1 -------------------------------------------------------------------------------- /run_mlm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/iliaschalkidis/flash-roberta/HEAD/run_mlm.py --------------------------------------------------------------------------------