├── LICENSE ├── README.md ├── assets ├── last_layer.png ├── pearson.png └── rank_count.png ├── deepseekmoe ├── config.json ├── configuration_deepseek.py └── modeling_deepseek.py ├── dynamic_analysis.ipynb ├── env.txt ├── grok ├── config.json ├── configuration_grok1.py ├── modeling_grok1.py └── modeling_grok1_outputs.py ├── mistral ├── config.json ├── configuration_mistral.py └── modeling_mistral.py ├── mixtral_base ├── config.json ├── configuration_moe_mistral.py └── modeling_moe_mistral.py ├── mixtral_base22 └── config.json ├── mixtral_instruct ├── config.json ├── configuration_moe_mistral.py └── modeling_mixtral_instruct.py ├── static_analysis.ipynb └── wikitext103_test.csv /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/README.md -------------------------------------------------------------------------------- /assets/last_layer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/assets/last_layer.png -------------------------------------------------------------------------------- /assets/pearson.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/assets/pearson.png -------------------------------------------------------------------------------- /assets/rank_count.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/assets/rank_count.png -------------------------------------------------------------------------------- /deepseekmoe/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/deepseekmoe/config.json -------------------------------------------------------------------------------- /deepseekmoe/configuration_deepseek.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/deepseekmoe/configuration_deepseek.py -------------------------------------------------------------------------------- /deepseekmoe/modeling_deepseek.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/deepseekmoe/modeling_deepseek.py -------------------------------------------------------------------------------- /dynamic_analysis.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/dynamic_analysis.ipynb -------------------------------------------------------------------------------- /env.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/env.txt -------------------------------------------------------------------------------- /grok/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/grok/config.json -------------------------------------------------------------------------------- /grok/configuration_grok1.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/grok/configuration_grok1.py -------------------------------------------------------------------------------- /grok/modeling_grok1.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/grok/modeling_grok1.py -------------------------------------------------------------------------------- /grok/modeling_grok1_outputs.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/grok/modeling_grok1_outputs.py -------------------------------------------------------------------------------- /mistral/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mistral/config.json -------------------------------------------------------------------------------- /mistral/configuration_mistral.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mistral/configuration_mistral.py -------------------------------------------------------------------------------- /mistral/modeling_mistral.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mistral/modeling_mistral.py -------------------------------------------------------------------------------- /mixtral_base/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_base/config.json -------------------------------------------------------------------------------- /mixtral_base/configuration_moe_mistral.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_base/configuration_moe_mistral.py -------------------------------------------------------------------------------- /mixtral_base/modeling_moe_mistral.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_base/modeling_moe_mistral.py -------------------------------------------------------------------------------- /mixtral_base22/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_base22/config.json -------------------------------------------------------------------------------- /mixtral_instruct/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_instruct/config.json -------------------------------------------------------------------------------- /mixtral_instruct/configuration_moe_mistral.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_instruct/configuration_moe_mistral.py -------------------------------------------------------------------------------- /mixtral_instruct/modeling_mixtral_instruct.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/mixtral_instruct/modeling_mixtral_instruct.py -------------------------------------------------------------------------------- /static_analysis.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/static_analysis.ipynb -------------------------------------------------------------------------------- /wikitext103_test.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kamanphoebe/Look-into-MoEs/HEAD/wikitext103_test.csv --------------------------------------------------------------------------------