├── .gitignore ├── LICENSE ├── README.md ├── core ├── __init__.py ├── evaluation.py └── prompts.py ├── eval_codet5.py ├── eval_llama.py ├── eval_mpt.py ├── eval_mpt_large.py ├── eval_opencode.py ├── eval_openllama.py ├── eval_replit.py ├── eval_replit_glaive.py ├── eval_replit_instruct.py ├── eval_starcoder.py ├── eval_wizard.py ├── eval_xgen.py ├── human-eval ├── LICENSE ├── README.md ├── data │ └── HumanEval.jsonl.gz ├── human_eval │ ├── __init__.py │ ├── data.py │ ├── evaluate_functional_correctness.py │ ├── evaluation.py │ └── execution.py ├── requirements.txt └── setup.py ├── process_eval.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/README.md -------------------------------------------------------------------------------- /core/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/core/__init__.py -------------------------------------------------------------------------------- /core/evaluation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/core/evaluation.py -------------------------------------------------------------------------------- /core/prompts.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/core/prompts.py -------------------------------------------------------------------------------- /eval_codet5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_codet5.py -------------------------------------------------------------------------------- /eval_llama.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_llama.py -------------------------------------------------------------------------------- /eval_mpt.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_mpt.py -------------------------------------------------------------------------------- /eval_mpt_large.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_mpt_large.py -------------------------------------------------------------------------------- /eval_opencode.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_opencode.py -------------------------------------------------------------------------------- /eval_openllama.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_openllama.py -------------------------------------------------------------------------------- /eval_replit.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_replit.py -------------------------------------------------------------------------------- /eval_replit_glaive.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_replit_glaive.py -------------------------------------------------------------------------------- /eval_replit_instruct.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_replit_instruct.py -------------------------------------------------------------------------------- /eval_starcoder.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_starcoder.py -------------------------------------------------------------------------------- /eval_wizard.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_wizard.py -------------------------------------------------------------------------------- /eval_xgen.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/eval_xgen.py -------------------------------------------------------------------------------- /human-eval/LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/LICENSE -------------------------------------------------------------------------------- /human-eval/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/README.md -------------------------------------------------------------------------------- /human-eval/data/HumanEval.jsonl.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/data/HumanEval.jsonl.gz -------------------------------------------------------------------------------- /human-eval/human_eval/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /human-eval/human_eval/data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/human_eval/data.py -------------------------------------------------------------------------------- /human-eval/human_eval/evaluate_functional_correctness.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/human_eval/evaluate_functional_correctness.py -------------------------------------------------------------------------------- /human-eval/human_eval/evaluation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/human_eval/evaluation.py -------------------------------------------------------------------------------- /human-eval/human_eval/execution.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/human_eval/execution.py -------------------------------------------------------------------------------- /human-eval/requirements.txt: -------------------------------------------------------------------------------- 1 | tqdm 2 | fire 3 | numpy 4 | -------------------------------------------------------------------------------- /human-eval/setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/human-eval/setup.py -------------------------------------------------------------------------------- /process_eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/process_eval.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/abacaj/code-eval/HEAD/requirements.txt --------------------------------------------------------------------------------