├── .gitignore ├── HISTORY.md ├── LICENSE ├── README.md ├── asset └── code_mmlu_banner.png ├── data ├── README.md ├── codemmlu │ └── codemmlu.py └── submission_sample.txt ├── paper └── 2410.01999v4.pdf ├── pyproject.toml ├── requirements.txt └── src └── codemmlu ├── __init__.py ├── __main__.py ├── backends ├── __init__.py ├── base.py ├── hf.py └── vllm.py ├── evaluator.py ├── prompts ├── __init__.py ├── _codecomp.py ├── _coderepair.py ├── _defect.py ├── _fim.py └── _general.py └── task_utils.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/.gitignore -------------------------------------------------------------------------------- /HISTORY.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/HISTORY.md -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/README.md -------------------------------------------------------------------------------- /asset/code_mmlu_banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/asset/code_mmlu_banner.png -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/data/README.md -------------------------------------------------------------------------------- /data/codemmlu/codemmlu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/data/codemmlu/codemmlu.py -------------------------------------------------------------------------------- /data/submission_sample.txt: -------------------------------------------------------------------------------- 1 | 1 A 2 | 2 B 3 | 3 C 4 | 4 D -------------------------------------------------------------------------------- /paper/2410.01999v4.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/paper/2410.01999v4.pdf -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/pyproject.toml -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/requirements.txt -------------------------------------------------------------------------------- /src/codemmlu/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/codemmlu/__main__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/__main__.py -------------------------------------------------------------------------------- /src/codemmlu/backends/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/backends/__init__.py -------------------------------------------------------------------------------- /src/codemmlu/backends/base.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/backends/base.py -------------------------------------------------------------------------------- /src/codemmlu/backends/hf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/backends/hf.py -------------------------------------------------------------------------------- /src/codemmlu/backends/vllm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/backends/vllm.py -------------------------------------------------------------------------------- /src/codemmlu/evaluator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/evaluator.py -------------------------------------------------------------------------------- /src/codemmlu/prompts/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/prompts/__init__.py -------------------------------------------------------------------------------- /src/codemmlu/prompts/_codecomp.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/prompts/_codecomp.py -------------------------------------------------------------------------------- /src/codemmlu/prompts/_coderepair.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/prompts/_coderepair.py -------------------------------------------------------------------------------- /src/codemmlu/prompts/_defect.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/prompts/_defect.py -------------------------------------------------------------------------------- /src/codemmlu/prompts/_fim.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/prompts/_fim.py -------------------------------------------------------------------------------- /src/codemmlu/prompts/_general.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/prompts/_general.py -------------------------------------------------------------------------------- /src/codemmlu/task_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/FSoft-AI4Code/CodeMMLU/HEAD/src/codemmlu/task_utils.py --------------------------------------------------------------------------------