├── .gitignore ├── LICENSE ├── README.md ├── examples ├── gpt2.py ├── images │ ├── gpt2_train_loss.png │ └── gpt2_val_loss.png └── readme.md ├── setup.py ├── tests ├── test_cross_entropy.py ├── test_dropout.py ├── test_layernorm.py └── test_softmax.py └── triformer ├── __init__.py ├── cross_entropy.py ├── dropout.py ├── images ├── dropout.png ├── layernorm-backward.png ├── layernorm-combined.png ├── layernorm-forward.png ├── memory_benchmark.png ├── rms-norm-backward.png ├── rms-norm-combined.png ├── rms-norm-forward.png ├── softmax-backward.png ├── softmax-combined.png ├── softmax-forward.png ├── swiglu-backward.png ├── swiglu-combined.png ├── swiglu-forward.png └── triformer.png ├── layernorm.py ├── rms_norm.py ├── rope.py ├── softmax.py ├── swiglu.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/README.md -------------------------------------------------------------------------------- /examples/gpt2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/examples/gpt2.py -------------------------------------------------------------------------------- /examples/images/gpt2_train_loss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/examples/images/gpt2_train_loss.png -------------------------------------------------------------------------------- /examples/images/gpt2_val_loss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/examples/images/gpt2_val_loss.png -------------------------------------------------------------------------------- /examples/readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/examples/readme.md -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/setup.py -------------------------------------------------------------------------------- /tests/test_cross_entropy.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/tests/test_cross_entropy.py -------------------------------------------------------------------------------- /tests/test_dropout.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/tests/test_dropout.py -------------------------------------------------------------------------------- /tests/test_layernorm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/tests/test_layernorm.py -------------------------------------------------------------------------------- /tests/test_softmax.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/tests/test_softmax.py -------------------------------------------------------------------------------- /triformer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/__init__.py -------------------------------------------------------------------------------- /triformer/cross_entropy.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/cross_entropy.py -------------------------------------------------------------------------------- /triformer/dropout.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/dropout.py -------------------------------------------------------------------------------- /triformer/images/dropout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/dropout.png -------------------------------------------------------------------------------- /triformer/images/layernorm-backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/layernorm-backward.png -------------------------------------------------------------------------------- /triformer/images/layernorm-combined.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/layernorm-combined.png -------------------------------------------------------------------------------- /triformer/images/layernorm-forward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/layernorm-forward.png -------------------------------------------------------------------------------- /triformer/images/memory_benchmark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/memory_benchmark.png -------------------------------------------------------------------------------- /triformer/images/rms-norm-backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/rms-norm-backward.png -------------------------------------------------------------------------------- /triformer/images/rms-norm-combined.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/rms-norm-combined.png -------------------------------------------------------------------------------- /triformer/images/rms-norm-forward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/rms-norm-forward.png -------------------------------------------------------------------------------- /triformer/images/softmax-backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/softmax-backward.png -------------------------------------------------------------------------------- /triformer/images/softmax-combined.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/softmax-combined.png -------------------------------------------------------------------------------- /triformer/images/softmax-forward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/softmax-forward.png -------------------------------------------------------------------------------- /triformer/images/swiglu-backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/swiglu-backward.png -------------------------------------------------------------------------------- /triformer/images/swiglu-combined.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/swiglu-combined.png -------------------------------------------------------------------------------- /triformer/images/swiglu-forward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/swiglu-forward.png -------------------------------------------------------------------------------- /triformer/images/triformer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/images/triformer.png -------------------------------------------------------------------------------- /triformer/layernorm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/layernorm.py -------------------------------------------------------------------------------- /triformer/rms_norm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/rms_norm.py -------------------------------------------------------------------------------- /triformer/rope.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/rope.py -------------------------------------------------------------------------------- /triformer/softmax.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/softmax.py -------------------------------------------------------------------------------- /triformer/swiglu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/swiglu.py -------------------------------------------------------------------------------- /triformer/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dame-cell/Triformer/HEAD/triformer/utils.py --------------------------------------------------------------------------------