├── .gitignore ├── .vscode └── launch.json ├── README.md ├── data.py ├── facts.py ├── gelu.cu ├── lecture_01-content.js ├── lecture_01.py ├── lecture_02-content.js ├── lecture_02.py ├── lecture_06-content.js ├── lecture_06.py ├── lecture_08-content.js ├── lecture_08.py ├── lecture_11-content.js ├── lecture_11.py ├── lecture_12-content.js ├── lecture_12.py ├── lecture_13-content.js ├── lecture_13.py ├── lecture_14-content.js ├── lecture_14.py ├── non_executable.py ├── nonexecutable ├── Lecture 10 - Scaling details.pdf ├── Lecture 15 - Alignment by SFT.pdf ├── Lecture 16 - Alignment by RLHF.pdf ├── Lecture 17 - Evals.pdf ├── Lecture 3 - architecture.pdf ├── Lecture 4 - details + MoEs.pdf ├── Lecture 5 - GPUs.pdf ├── Lecture 7 - Parallelism basics.pdf └── Lecture 9 - Scaling laws basics.pdf ├── references.py ├── requirements.txt ├── tokenization.py ├── util.py ├── var ├── c173cfba4d26dddd64da205bb008b891-https_en_wikipedia_org_wiki_Sphinx ├── cuda_gelu │ ├── .ninja_deps │ ├── .ninja_log │ ├── build.ninja │ ├── cuda.cu │ ├── cuda.cuda.o │ ├── inline_gelu.so │ ├── main.cpp │ └── main.o ├── query_model_cache.db ├── stacks_mlp.svg ├── stacks_mlp.txt ├── triton_gelu-ptx.txt └── triton_softmax-ptx.txt └── view.html /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/.gitignore -------------------------------------------------------------------------------- /.vscode/launch.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/.vscode/launch.json -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/README.md -------------------------------------------------------------------------------- /data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/data.py -------------------------------------------------------------------------------- /facts.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/facts.py -------------------------------------------------------------------------------- /gelu.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/gelu.cu -------------------------------------------------------------------------------- /lecture_01-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_01-content.js -------------------------------------------------------------------------------- /lecture_01.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_01.py -------------------------------------------------------------------------------- /lecture_02-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_02-content.js -------------------------------------------------------------------------------- /lecture_02.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_02.py -------------------------------------------------------------------------------- /lecture_06-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_06-content.js -------------------------------------------------------------------------------- /lecture_06.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_06.py -------------------------------------------------------------------------------- /lecture_08-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_08-content.js -------------------------------------------------------------------------------- /lecture_08.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_08.py -------------------------------------------------------------------------------- /lecture_11-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_11-content.js -------------------------------------------------------------------------------- /lecture_11.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_11.py -------------------------------------------------------------------------------- /lecture_12-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_12-content.js -------------------------------------------------------------------------------- /lecture_12.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_12.py -------------------------------------------------------------------------------- /lecture_13-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_13-content.js -------------------------------------------------------------------------------- /lecture_13.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_13.py -------------------------------------------------------------------------------- /lecture_14-content.js: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_14-content.js -------------------------------------------------------------------------------- /lecture_14.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/lecture_14.py -------------------------------------------------------------------------------- /non_executable.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/non_executable.py -------------------------------------------------------------------------------- /nonexecutable/Lecture 10 - Scaling details.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 10 - Scaling details.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 15 - Alignment by SFT.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 15 - Alignment by SFT.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 16 - Alignment by RLHF.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 16 - Alignment by RLHF.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 17 - Evals.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 17 - Evals.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 3 - architecture.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 3 - architecture.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 4 - details + MoEs.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 4 - details + MoEs.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 5 - GPUs.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 5 - GPUs.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 7 - Parallelism basics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 7 - Parallelism basics.pdf -------------------------------------------------------------------------------- /nonexecutable/Lecture 9 - Scaling laws basics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/nonexecutable/Lecture 9 - Scaling laws basics.pdf -------------------------------------------------------------------------------- /references.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/references.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/requirements.txt -------------------------------------------------------------------------------- /tokenization.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/tokenization.py -------------------------------------------------------------------------------- /util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/util.py -------------------------------------------------------------------------------- /var/c173cfba4d26dddd64da205bb008b891-https_en_wikipedia_org_wiki_Sphinx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/c173cfba4d26dddd64da205bb008b891-https_en_wikipedia_org_wiki_Sphinx -------------------------------------------------------------------------------- /var/cuda_gelu/.ninja_deps: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/.ninja_deps -------------------------------------------------------------------------------- /var/cuda_gelu/.ninja_log: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/.ninja_log -------------------------------------------------------------------------------- /var/cuda_gelu/build.ninja: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/build.ninja -------------------------------------------------------------------------------- /var/cuda_gelu/cuda.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/cuda.cu -------------------------------------------------------------------------------- /var/cuda_gelu/cuda.cuda.o: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/cuda.cuda.o -------------------------------------------------------------------------------- /var/cuda_gelu/inline_gelu.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/inline_gelu.so -------------------------------------------------------------------------------- /var/cuda_gelu/main.cpp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/main.cpp -------------------------------------------------------------------------------- /var/cuda_gelu/main.o: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/cuda_gelu/main.o -------------------------------------------------------------------------------- /var/query_model_cache.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/query_model_cache.db -------------------------------------------------------------------------------- /var/stacks_mlp.svg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/stacks_mlp.svg -------------------------------------------------------------------------------- /var/stacks_mlp.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/stacks_mlp.txt -------------------------------------------------------------------------------- /var/triton_gelu-ptx.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/triton_gelu-ptx.txt -------------------------------------------------------------------------------- /var/triton_softmax-ptx.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/var/triton_softmax-ptx.txt -------------------------------------------------------------------------------- /view.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanford-cs336/spring2024-lectures/HEAD/view.html --------------------------------------------------------------------------------