├── .gitignore ├── LICENSE ├── README.md ├── README_en.md ├── images ├── 42.png ├── a10.png ├── afterattention.png ├── archi.png ├── attention.png ├── embeddings.png ├── finallayer.png ├── freq_cis.png ├── god.png ├── heads.png ├── implllama3_30_0.png ├── implllama3_39_0.png ├── implllama3_41_0.png ├── implllama3_42_0.png ├── implllama3_50_0.png ├── implllama3_52_0.png ├── implllama3_54_0.png ├── karpathyminbpe.png ├── keys.png ├── keys0.png ├── last_norm.png ├── mask.png ├── model.png ├── norm.png ├── norm_after.png ├── q_per_token.png ├── qkmatmul.png ├── qkv.png ├── qsplit.png ├── rms.png ├── rope.png ├── ropesplit.png ├── softmax.png ├── stacked.png ├── swiglu.png ├── tokens.png ├── v0.png ├── value.png └── weightmatrix.png ├── llama3-from-scratch_en.ipynb ├── llama3-from-scratch_zh.ipynb ├── llama3 ├── README.md ├── model.py └── tokenizer.py ├── pdf └── 从零实现 Llama3 模型.pdf └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | Meta-Llama-3-8B-Instruct-2layers/* -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/README.md -------------------------------------------------------------------------------- /README_en.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/README_en.md -------------------------------------------------------------------------------- /images/42.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/42.png -------------------------------------------------------------------------------- /images/a10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/a10.png -------------------------------------------------------------------------------- /images/afterattention.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/afterattention.png -------------------------------------------------------------------------------- /images/archi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/archi.png -------------------------------------------------------------------------------- /images/attention.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/attention.png -------------------------------------------------------------------------------- /images/embeddings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/embeddings.png -------------------------------------------------------------------------------- /images/finallayer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/finallayer.png -------------------------------------------------------------------------------- /images/freq_cis.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/freq_cis.png -------------------------------------------------------------------------------- /images/god.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/god.png -------------------------------------------------------------------------------- /images/heads.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/heads.png -------------------------------------------------------------------------------- /images/implllama3_30_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_30_0.png -------------------------------------------------------------------------------- /images/implllama3_39_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_39_0.png -------------------------------------------------------------------------------- /images/implllama3_41_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_41_0.png -------------------------------------------------------------------------------- /images/implllama3_42_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_42_0.png -------------------------------------------------------------------------------- /images/implllama3_50_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_50_0.png -------------------------------------------------------------------------------- /images/implllama3_52_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_52_0.png -------------------------------------------------------------------------------- /images/implllama3_54_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/implllama3_54_0.png -------------------------------------------------------------------------------- /images/karpathyminbpe.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/karpathyminbpe.png -------------------------------------------------------------------------------- /images/keys.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/keys.png -------------------------------------------------------------------------------- /images/keys0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/keys0.png -------------------------------------------------------------------------------- /images/last_norm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/last_norm.png -------------------------------------------------------------------------------- /images/mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/mask.png -------------------------------------------------------------------------------- /images/model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/model.png -------------------------------------------------------------------------------- /images/norm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/norm.png -------------------------------------------------------------------------------- /images/norm_after.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/norm_after.png -------------------------------------------------------------------------------- /images/q_per_token.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/q_per_token.png -------------------------------------------------------------------------------- /images/qkmatmul.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/qkmatmul.png -------------------------------------------------------------------------------- /images/qkv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/qkv.png -------------------------------------------------------------------------------- /images/qsplit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/qsplit.png -------------------------------------------------------------------------------- /images/rms.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/rms.png -------------------------------------------------------------------------------- /images/rope.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/rope.png -------------------------------------------------------------------------------- /images/ropesplit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/ropesplit.png -------------------------------------------------------------------------------- /images/softmax.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/softmax.png -------------------------------------------------------------------------------- /images/stacked.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/stacked.png -------------------------------------------------------------------------------- /images/swiglu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/swiglu.png -------------------------------------------------------------------------------- /images/tokens.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/tokens.png -------------------------------------------------------------------------------- /images/v0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/v0.png -------------------------------------------------------------------------------- /images/value.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/value.png -------------------------------------------------------------------------------- /images/weightmatrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/images/weightmatrix.png -------------------------------------------------------------------------------- /llama3-from-scratch_en.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/llama3-from-scratch_en.ipynb -------------------------------------------------------------------------------- /llama3-from-scratch_zh.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/llama3-from-scratch_zh.ipynb -------------------------------------------------------------------------------- /llama3/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/llama3/README.md -------------------------------------------------------------------------------- /llama3/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/llama3/model.py -------------------------------------------------------------------------------- /llama3/tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/llama3/tokenizer.py -------------------------------------------------------------------------------- /pdf/从零实现 Llama3 模型.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/pdf/从零实现 Llama3 模型.pdf -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdndev/llama3-from-scratch-zh/HEAD/requirements.txt --------------------------------------------------------------------------------