├── LICENSE ├── LLaMA-Factory ├── llama3_lora_dpo.yaml ├── llama_train.sh ├── qwen_lora_dpo.yaml └── qwen_train.sh ├── README.md ├── data ├── ConsJudge_train │ └── test.jsonl └── RAG_train │ ├── dev_acc_test.jsonl │ ├── dev_rouge_test.jsonl │ ├── train_acc_test.jsonl │ └── train_rouge_test.jsonl ├── fig └── fig.png ├── script ├── RAGmodel_train.sh ├── choices_gen.sh ├── construct_ConsJudge_data.sh ├── construct_RAG_acc_data.sh ├── construct_RAG_rouge_data.sh ├── eval_asqa.sh ├── eval_hotpotqa.sh ├── eval_marco.sh ├── eval_nq.sh ├── eval_tqa.sh └── eval_wow.sh └── src ├── ConsJudge_train ├── construct.py ├── embedding_similarity.py ├── hybrid_evaluation.py ├── llama3_8b_infer.py ├── minicpm_2b_infer.py ├── minicpm_4b_infer.py └── qwen_14b_infer.py ├── RAG_train ├── combine_data.py ├── config.py ├── configuration_minicpm.py ├── ds_config_zero2.json ├── infer_acc.py ├── infer_rouge.py ├── merge_lora.py ├── modeling_minicpm.py ├── preprocess_acc_data.py ├── train.py └── utils │ ├── __init__.py │ ├── __pycache__ │ ├── __init__.cpython-310.pyc │ ├── __init__.cpython-38.pyc │ ├── eval_utils.cpython-310.pyc │ ├── eval_utils.cpython-38.pyc │ ├── eval_utils_old.cpython-310.pyc │ ├── train_args.cpython-38.pyc │ └── train_utils.cpython-38.pyc │ ├── eval_utils.py │ ├── metrics.py │ ├── train_args.py │ └── train_utils.py └── evaluation ├── LLM_eval.py ├── calculate_em_recall.py ├── eval.py ├── postprocess.py └── utils ├── __init__.py ├── __pycache__ ├── __init__.cpython-310.pyc ├── __init__.cpython-38.pyc ├── eval_utils.cpython-310.pyc ├── eval_utils.cpython-38.pyc ├── eval_utils_old.cpython-310.pyc ├── train_args.cpython-38.pyc └── train_utils.cpython-38.pyc ├── eval_utils.py ├── metrics.py ├── train_args.py └── train_utils.py /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/LICENSE -------------------------------------------------------------------------------- /LLaMA-Factory/llama3_lora_dpo.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/LLaMA-Factory/llama3_lora_dpo.yaml -------------------------------------------------------------------------------- /LLaMA-Factory/llama_train.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/LLaMA-Factory/llama_train.sh -------------------------------------------------------------------------------- /LLaMA-Factory/qwen_lora_dpo.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/LLaMA-Factory/qwen_lora_dpo.yaml -------------------------------------------------------------------------------- /LLaMA-Factory/qwen_train.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/LLaMA-Factory/qwen_train.sh -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/README.md -------------------------------------------------------------------------------- /data/ConsJudge_train/test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/data/ConsJudge_train/test.jsonl -------------------------------------------------------------------------------- /data/RAG_train/dev_acc_test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/data/RAG_train/dev_acc_test.jsonl -------------------------------------------------------------------------------- /data/RAG_train/dev_rouge_test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/data/RAG_train/dev_rouge_test.jsonl -------------------------------------------------------------------------------- /data/RAG_train/train_acc_test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/data/RAG_train/train_acc_test.jsonl -------------------------------------------------------------------------------- /data/RAG_train/train_rouge_test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/data/RAG_train/train_rouge_test.jsonl -------------------------------------------------------------------------------- /fig/fig.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/fig/fig.png -------------------------------------------------------------------------------- /script/RAGmodel_train.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/RAGmodel_train.sh -------------------------------------------------------------------------------- /script/choices_gen.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/choices_gen.sh -------------------------------------------------------------------------------- /script/construct_ConsJudge_data.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/construct_ConsJudge_data.sh -------------------------------------------------------------------------------- /script/construct_RAG_acc_data.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/construct_RAG_acc_data.sh -------------------------------------------------------------------------------- /script/construct_RAG_rouge_data.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/construct_RAG_rouge_data.sh -------------------------------------------------------------------------------- /script/eval_asqa.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/eval_asqa.sh -------------------------------------------------------------------------------- /script/eval_hotpotqa.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/eval_hotpotqa.sh -------------------------------------------------------------------------------- /script/eval_marco.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/eval_marco.sh -------------------------------------------------------------------------------- /script/eval_nq.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/eval_nq.sh -------------------------------------------------------------------------------- /script/eval_tqa.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/eval_tqa.sh -------------------------------------------------------------------------------- /script/eval_wow.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/script/eval_wow.sh -------------------------------------------------------------------------------- /src/ConsJudge_train/construct.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/construct.py -------------------------------------------------------------------------------- /src/ConsJudge_train/embedding_similarity.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/embedding_similarity.py -------------------------------------------------------------------------------- /src/ConsJudge_train/hybrid_evaluation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/hybrid_evaluation.py -------------------------------------------------------------------------------- /src/ConsJudge_train/llama3_8b_infer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/llama3_8b_infer.py -------------------------------------------------------------------------------- /src/ConsJudge_train/minicpm_2b_infer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/minicpm_2b_infer.py -------------------------------------------------------------------------------- /src/ConsJudge_train/minicpm_4b_infer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/minicpm_4b_infer.py -------------------------------------------------------------------------------- /src/ConsJudge_train/qwen_14b_infer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/ConsJudge_train/qwen_14b_infer.py -------------------------------------------------------------------------------- /src/RAG_train/combine_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/combine_data.py -------------------------------------------------------------------------------- /src/RAG_train/config.py: -------------------------------------------------------------------------------- 1 | glob_logits=[] -------------------------------------------------------------------------------- /src/RAG_train/configuration_minicpm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/configuration_minicpm.py -------------------------------------------------------------------------------- /src/RAG_train/ds_config_zero2.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/ds_config_zero2.json -------------------------------------------------------------------------------- /src/RAG_train/infer_acc.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/infer_acc.py -------------------------------------------------------------------------------- /src/RAG_train/infer_rouge.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/infer_rouge.py -------------------------------------------------------------------------------- /src/RAG_train/merge_lora.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/merge_lora.py -------------------------------------------------------------------------------- /src/RAG_train/modeling_minicpm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/modeling_minicpm.py -------------------------------------------------------------------------------- /src/RAG_train/preprocess_acc_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/preprocess_acc_data.py -------------------------------------------------------------------------------- /src/RAG_train/train.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/train.py -------------------------------------------------------------------------------- /src/RAG_train/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/__init__.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/__init__.cpython-310.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/__init__.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/__init__.cpython-38.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/eval_utils.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/eval_utils.cpython-310.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/eval_utils.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/eval_utils.cpython-38.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/eval_utils_old.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/eval_utils_old.cpython-310.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/train_args.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/train_args.cpython-38.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/__pycache__/train_utils.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/__pycache__/train_utils.cpython-38.pyc -------------------------------------------------------------------------------- /src/RAG_train/utils/eval_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/eval_utils.py -------------------------------------------------------------------------------- /src/RAG_train/utils/metrics.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/metrics.py -------------------------------------------------------------------------------- /src/RAG_train/utils/train_args.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/train_args.py -------------------------------------------------------------------------------- /src/RAG_train/utils/train_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/RAG_train/utils/train_utils.py -------------------------------------------------------------------------------- /src/evaluation/LLM_eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/LLM_eval.py -------------------------------------------------------------------------------- /src/evaluation/calculate_em_recall.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/calculate_em_recall.py -------------------------------------------------------------------------------- /src/evaluation/eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/eval.py -------------------------------------------------------------------------------- /src/evaluation/postprocess.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/postprocess.py -------------------------------------------------------------------------------- /src/evaluation/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/__init__.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/__init__.cpython-310.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/__init__.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/__init__.cpython-38.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/eval_utils.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/eval_utils.cpython-310.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/eval_utils.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/eval_utils.cpython-38.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/eval_utils_old.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/eval_utils_old.cpython-310.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/train_args.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/train_args.cpython-38.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/__pycache__/train_utils.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/__pycache__/train_utils.cpython-38.pyc -------------------------------------------------------------------------------- /src/evaluation/utils/eval_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/eval_utils.py -------------------------------------------------------------------------------- /src/evaluation/utils/metrics.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/metrics.py -------------------------------------------------------------------------------- /src/evaluation/utils/train_args.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/train_args.py -------------------------------------------------------------------------------- /src/evaluation/utils/train_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenBMB/ConsJudge/HEAD/src/evaluation/utils/train_utils.py --------------------------------------------------------------------------------