├── .gitignore ├── LICENSE ├── README.md ├── assets ├── 1b-8k-proweb.png ├── ladder.png ├── schedule_illustration_v2.pdf └── schedule_illustration_v2.png ├── lit_gpt ├── __init__.py ├── adapter.py ├── adapter_v2.py ├── config.py ├── constants.py ├── fused_cross_entropy.py ├── fused_rotary_embedding.py ├── lora.py ├── model.py ├── packed_dataset.py ├── rmsnorm.py ├── speed_monitor.py ├── tokenizer.py └── utils.py ├── pretrain └── tinyllama.py ├── requirements.txt ├── scripts ├── code_processing.sh ├── convert_lit_checkpoint.py ├── convert_to_hf.sh ├── pajama_processing.sh ├── prepare_file.py ├── pretraining.sh └── pretraining_multi.sh ├── starcoder ├── merges.txt ├── special_tokens_map.json ├── tokenizer.json ├── tokenizer_config.json └── vocab.json └── tokenizer └── tokenizer.model /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/README.md -------------------------------------------------------------------------------- /assets/1b-8k-proweb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/assets/1b-8k-proweb.png -------------------------------------------------------------------------------- /assets/ladder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/assets/ladder.png -------------------------------------------------------------------------------- /assets/schedule_illustration_v2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/assets/schedule_illustration_v2.pdf -------------------------------------------------------------------------------- /assets/schedule_illustration_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/assets/schedule_illustration_v2.png -------------------------------------------------------------------------------- /lit_gpt/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/__init__.py -------------------------------------------------------------------------------- /lit_gpt/adapter.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/adapter.py -------------------------------------------------------------------------------- /lit_gpt/adapter_v2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/adapter_v2.py -------------------------------------------------------------------------------- /lit_gpt/config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/config.py -------------------------------------------------------------------------------- /lit_gpt/constants.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/constants.py -------------------------------------------------------------------------------- /lit_gpt/fused_cross_entropy.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/fused_cross_entropy.py -------------------------------------------------------------------------------- /lit_gpt/fused_rotary_embedding.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/fused_rotary_embedding.py -------------------------------------------------------------------------------- /lit_gpt/lora.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/lora.py -------------------------------------------------------------------------------- /lit_gpt/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/model.py -------------------------------------------------------------------------------- /lit_gpt/packed_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/packed_dataset.py -------------------------------------------------------------------------------- /lit_gpt/rmsnorm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/rmsnorm.py -------------------------------------------------------------------------------- /lit_gpt/speed_monitor.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/speed_monitor.py -------------------------------------------------------------------------------- /lit_gpt/tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/tokenizer.py -------------------------------------------------------------------------------- /lit_gpt/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/lit_gpt/utils.py -------------------------------------------------------------------------------- /pretrain/tinyllama.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/pretrain/tinyllama.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/requirements.txt -------------------------------------------------------------------------------- /scripts/code_processing.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/code_processing.sh -------------------------------------------------------------------------------- /scripts/convert_lit_checkpoint.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/convert_lit_checkpoint.py -------------------------------------------------------------------------------- /scripts/convert_to_hf.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/convert_to_hf.sh -------------------------------------------------------------------------------- /scripts/pajama_processing.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/pajama_processing.sh -------------------------------------------------------------------------------- /scripts/prepare_file.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/prepare_file.py -------------------------------------------------------------------------------- /scripts/pretraining.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/pretraining.sh -------------------------------------------------------------------------------- /scripts/pretraining_multi.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/scripts/pretraining_multi.sh -------------------------------------------------------------------------------- /starcoder/merges.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/starcoder/merges.txt -------------------------------------------------------------------------------- /starcoder/special_tokens_map.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/starcoder/special_tokens_map.json -------------------------------------------------------------------------------- /starcoder/tokenizer.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/starcoder/tokenizer.json -------------------------------------------------------------------------------- /starcoder/tokenizer_config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/starcoder/tokenizer_config.json -------------------------------------------------------------------------------- /starcoder/vocab.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/starcoder/vocab.json -------------------------------------------------------------------------------- /tokenizer/tokenizer.model: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sail-sg/SkyLadder/HEAD/tokenizer/tokenizer.model --------------------------------------------------------------------------------