├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── csrc ├── condense.h ├── graph.h ├── ops.cu ├── post_combine.h ├── post_dispatch.h ├── pre_combine.h ├── pre_dispatch.h └── re_rout.h ├── examples ├── Mixtral-8x7B-v0.1 │ ├── config.json │ └── configuration_mixtral.py ├── OLMoE-1B-7B-0924 │ ├── config.json │ └── configuration_deepseek.py ├── Qwen1.5-MoE-A2.7B │ ├── config.json │ └── configuration_qwen2_moe.py ├── README.md └── deepseek-moe-16b-base │ ├── config.json │ ├── configuration_deepseek.py │ └── modeling_deepseek.py ├── figures ├── train_deepseek_16_way.svg └── train_deepseek_8_way.svg ├── occult ├── __init__.py ├── _version.py ├── backend │ ├── __init__.py │ ├── kernels_graph.py │ ├── kernels_mm.py │ ├── kernels_mm_condense.py │ ├── kernels_rout.py │ └── kernels_sfd.py ├── benchmark_util.py ├── layers │ ├── __init__.py │ ├── all_to_all.py │ ├── arguments.py │ ├── common.py │ ├── dmlp_registry.py │ ├── dmoe.py │ ├── glu.py │ ├── mpu.py │ └── router.py └── ops │ ├── __init__.py │ ├── brim_condense.py │ ├── condense.py │ ├── graph.py │ ├── mm.py │ ├── post_combine.py │ ├── post_dispatch.py │ └── pre_dispatch.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/Makefile -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/README.md -------------------------------------------------------------------------------- /csrc/condense.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/condense.h -------------------------------------------------------------------------------- /csrc/graph.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/graph.h -------------------------------------------------------------------------------- /csrc/ops.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/ops.cu -------------------------------------------------------------------------------- /csrc/post_combine.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/post_combine.h -------------------------------------------------------------------------------- /csrc/post_dispatch.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/post_dispatch.h -------------------------------------------------------------------------------- /csrc/pre_combine.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/pre_combine.h -------------------------------------------------------------------------------- /csrc/pre_dispatch.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/pre_dispatch.h -------------------------------------------------------------------------------- /csrc/re_rout.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/csrc/re_rout.h -------------------------------------------------------------------------------- /examples/Mixtral-8x7B-v0.1/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/Mixtral-8x7B-v0.1/config.json -------------------------------------------------------------------------------- /examples/Mixtral-8x7B-v0.1/configuration_mixtral.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/Mixtral-8x7B-v0.1/configuration_mixtral.py -------------------------------------------------------------------------------- /examples/OLMoE-1B-7B-0924/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/OLMoE-1B-7B-0924/config.json -------------------------------------------------------------------------------- /examples/OLMoE-1B-7B-0924/configuration_deepseek.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/OLMoE-1B-7B-0924/configuration_deepseek.py -------------------------------------------------------------------------------- /examples/Qwen1.5-MoE-A2.7B/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/Qwen1.5-MoE-A2.7B/config.json -------------------------------------------------------------------------------- /examples/Qwen1.5-MoE-A2.7B/configuration_qwen2_moe.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/Qwen1.5-MoE-A2.7B/configuration_qwen2_moe.py -------------------------------------------------------------------------------- /examples/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/README.md -------------------------------------------------------------------------------- /examples/deepseek-moe-16b-base/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/deepseek-moe-16b-base/config.json -------------------------------------------------------------------------------- /examples/deepseek-moe-16b-base/configuration_deepseek.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/deepseek-moe-16b-base/configuration_deepseek.py -------------------------------------------------------------------------------- /examples/deepseek-moe-16b-base/modeling_deepseek.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/examples/deepseek-moe-16b-base/modeling_deepseek.py -------------------------------------------------------------------------------- /figures/train_deepseek_16_way.svg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/figures/train_deepseek_16_way.svg -------------------------------------------------------------------------------- /figures/train_deepseek_8_way.svg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/figures/train_deepseek_8_way.svg -------------------------------------------------------------------------------- /occult/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/__init__.py -------------------------------------------------------------------------------- /occult/_version.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /occult/backend/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /occult/backend/kernels_graph.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/backend/kernels_graph.py -------------------------------------------------------------------------------- /occult/backend/kernels_mm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/backend/kernels_mm.py -------------------------------------------------------------------------------- /occult/backend/kernels_mm_condense.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/backend/kernels_mm_condense.py -------------------------------------------------------------------------------- /occult/backend/kernels_rout.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/backend/kernels_rout.py -------------------------------------------------------------------------------- /occult/backend/kernels_sfd.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/backend/kernels_sfd.py -------------------------------------------------------------------------------- /occult/benchmark_util.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /occult/layers/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/__init__.py -------------------------------------------------------------------------------- /occult/layers/all_to_all.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/all_to_all.py -------------------------------------------------------------------------------- /occult/layers/arguments.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/arguments.py -------------------------------------------------------------------------------- /occult/layers/common.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/common.py -------------------------------------------------------------------------------- /occult/layers/dmlp_registry.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/dmlp_registry.py -------------------------------------------------------------------------------- /occult/layers/dmoe.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/dmoe.py -------------------------------------------------------------------------------- /occult/layers/glu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/glu.py -------------------------------------------------------------------------------- /occult/layers/mpu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/mpu.py -------------------------------------------------------------------------------- /occult/layers/router.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/layers/router.py -------------------------------------------------------------------------------- /occult/ops/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/__init__.py -------------------------------------------------------------------------------- /occult/ops/brim_condense.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/brim_condense.py -------------------------------------------------------------------------------- /occult/ops/condense.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/condense.py -------------------------------------------------------------------------------- /occult/ops/graph.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/graph.py -------------------------------------------------------------------------------- /occult/ops/mm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/mm.py -------------------------------------------------------------------------------- /occult/ops/post_combine.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/post_combine.py -------------------------------------------------------------------------------- /occult/ops/post_dispatch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/post_dispatch.py -------------------------------------------------------------------------------- /occult/ops/pre_dispatch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/occult/ops/pre_dispatch.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UNITES-Lab/Occult/HEAD/setup.py --------------------------------------------------------------------------------