├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── NOTICE ├── README.md ├── THIRD_PARTY_LICENSES ├── cceval_config.yaml ├── data └── crosscodeeval_data.tar.xz ├── prompt_builder ├── README.md ├── augment_with_cfc.py ├── rerank_utils.py ├── run.sh └── utils.py ├── requirements.txt └── scripts ├── build_treesitter.sh ├── build_ts_lib.py ├── custom_generate.py ├── eval.py ├── eval_metric.py ├── eval_utils.py ├── keywords ├── __init__.py ├── csharp.txt ├── java.txt ├── javascript.txt ├── keywordlist.py └── typescript.txt ├── openai_inference.py └── vllm_inference.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/.gitignore -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/CODE_OF_CONDUCT.md -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/CONTRIBUTING.md -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/LICENSE -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/README.md -------------------------------------------------------------------------------- /THIRD_PARTY_LICENSES: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/THIRD_PARTY_LICENSES -------------------------------------------------------------------------------- /cceval_config.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/cceval_config.yaml -------------------------------------------------------------------------------- /data/crosscodeeval_data.tar.xz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/data/crosscodeeval_data.tar.xz -------------------------------------------------------------------------------- /prompt_builder/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/prompt_builder/README.md -------------------------------------------------------------------------------- /prompt_builder/augment_with_cfc.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/prompt_builder/augment_with_cfc.py -------------------------------------------------------------------------------- /prompt_builder/rerank_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/prompt_builder/rerank_utils.py -------------------------------------------------------------------------------- /prompt_builder/run.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/prompt_builder/run.sh -------------------------------------------------------------------------------- /prompt_builder/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/prompt_builder/utils.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/requirements.txt -------------------------------------------------------------------------------- /scripts/build_treesitter.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/build_treesitter.sh -------------------------------------------------------------------------------- /scripts/build_ts_lib.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/build_ts_lib.py -------------------------------------------------------------------------------- /scripts/custom_generate.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/custom_generate.py -------------------------------------------------------------------------------- /scripts/eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/eval.py -------------------------------------------------------------------------------- /scripts/eval_metric.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/eval_metric.py -------------------------------------------------------------------------------- /scripts/eval_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/eval_utils.py -------------------------------------------------------------------------------- /scripts/keywords/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /scripts/keywords/csharp.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/keywords/csharp.txt -------------------------------------------------------------------------------- /scripts/keywords/java.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/keywords/java.txt -------------------------------------------------------------------------------- /scripts/keywords/javascript.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/keywords/javascript.txt -------------------------------------------------------------------------------- /scripts/keywords/keywordlist.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/keywords/keywordlist.py -------------------------------------------------------------------------------- /scripts/keywords/typescript.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/keywords/typescript.txt -------------------------------------------------------------------------------- /scripts/openai_inference.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/openai_inference.py -------------------------------------------------------------------------------- /scripts/vllm_inference.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/cceval/HEAD/scripts/vllm_inference.py --------------------------------------------------------------------------------