├── .gitignore ├── LICENSE ├── README.md ├── assets ├── palm.gif └── palm_loss.png ├── data └── openwebtext │ └── prepare.py ├── model.py ├── nanoPaLM.nsys-rep ├── requirements.txt └── train.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/README.md -------------------------------------------------------------------------------- /assets/palm.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/assets/palm.gif -------------------------------------------------------------------------------- /assets/palm_loss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/assets/palm_loss.png -------------------------------------------------------------------------------- /data/openwebtext/prepare.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/data/openwebtext/prepare.py -------------------------------------------------------------------------------- /model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/model.py -------------------------------------------------------------------------------- /nanoPaLM.nsys-rep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/nanoPaLM.nsys-rep -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | torch>=2.0.0 2 | transformers 3 | datasets 4 | tqdm 5 | wandb -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertRiachi/nanoPALM/HEAD/train.py --------------------------------------------------------------------------------