├── LICENSE ├── README.md ├── docs ├── Close_3n.png ├── Figure1-final.png ├── Open_African.png ├── ReadMe ├── framework-gai.png └── open-construction.png └── evaluation ├── configs ├── dataset_config.yaml └── model_config.yaml ├── dataset ├── BaseDataset.py ├── __init__.py ├── close_ended.py ├── load_dataset.py └── open_ended.py ├── miscs ├── collect_answer.py ├── collect_result_by_dataset.py ├── convert_log_to_json.py ├── draw_image.py └── vlm_test │ └── id_2.png ├── models ├── __init__.py ├── base_model.py ├── blip2_load.py ├── emu2_chat_load.py ├── emu2_gen_load.py ├── instructblip_load.py ├── internlm_xcomposer_vl_load.py ├── llava_load.py ├── load_model.py ├── minigpt4_load.py ├── minigpt_v2_load.py ├── otter_load.py ├── qwen_vl_load.py ├── qwen_vl_load_multi_round.py └── shikra_load.py ├── run_evaluation.py ├── script ├── run_evaluation_close_ended.sh └── run_evaluation_open_ended.sh └── utils.py /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/README.md -------------------------------------------------------------------------------- /docs/Close_3n.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/docs/Close_3n.png -------------------------------------------------------------------------------- /docs/Figure1-final.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/docs/Figure1-final.png -------------------------------------------------------------------------------- /docs/Open_African.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/docs/Open_African.png -------------------------------------------------------------------------------- /docs/ReadMe: -------------------------------------------------------------------------------- 1 | figures 2 | -------------------------------------------------------------------------------- /docs/framework-gai.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/docs/framework-gai.png -------------------------------------------------------------------------------- /docs/open-construction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/docs/open-construction.png -------------------------------------------------------------------------------- /evaluation/configs/dataset_config.yaml: -------------------------------------------------------------------------------- 1 | a: -------------------------------------------------------------------------------- /evaluation/configs/model_config.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/configs/model_config.yaml -------------------------------------------------------------------------------- /evaluation/dataset/BaseDataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/dataset/BaseDataset.py -------------------------------------------------------------------------------- /evaluation/dataset/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/dataset/__init__.py -------------------------------------------------------------------------------- /evaluation/dataset/close_ended.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/dataset/close_ended.py -------------------------------------------------------------------------------- /evaluation/dataset/load_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/dataset/load_dataset.py -------------------------------------------------------------------------------- /evaluation/dataset/open_ended.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/dataset/open_ended.py -------------------------------------------------------------------------------- /evaluation/miscs/collect_answer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/miscs/collect_answer.py -------------------------------------------------------------------------------- /evaluation/miscs/collect_result_by_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/miscs/collect_result_by_dataset.py -------------------------------------------------------------------------------- /evaluation/miscs/convert_log_to_json.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/miscs/convert_log_to_json.py -------------------------------------------------------------------------------- /evaluation/miscs/draw_image.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/miscs/draw_image.py -------------------------------------------------------------------------------- /evaluation/miscs/vlm_test/id_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/miscs/vlm_test/id_2.png -------------------------------------------------------------------------------- /evaluation/models/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /evaluation/models/base_model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/base_model.py -------------------------------------------------------------------------------- /evaluation/models/blip2_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/blip2_load.py -------------------------------------------------------------------------------- /evaluation/models/emu2_chat_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/emu2_chat_load.py -------------------------------------------------------------------------------- /evaluation/models/emu2_gen_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/emu2_gen_load.py -------------------------------------------------------------------------------- /evaluation/models/instructblip_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/instructblip_load.py -------------------------------------------------------------------------------- /evaluation/models/internlm_xcomposer_vl_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/internlm_xcomposer_vl_load.py -------------------------------------------------------------------------------- /evaluation/models/llava_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/llava_load.py -------------------------------------------------------------------------------- /evaluation/models/load_model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/load_model.py -------------------------------------------------------------------------------- /evaluation/models/minigpt4_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/minigpt4_load.py -------------------------------------------------------------------------------- /evaluation/models/minigpt_v2_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/minigpt_v2_load.py -------------------------------------------------------------------------------- /evaluation/models/otter_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/otter_load.py -------------------------------------------------------------------------------- /evaluation/models/qwen_vl_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/qwen_vl_load.py -------------------------------------------------------------------------------- /evaluation/models/qwen_vl_load_multi_round.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/qwen_vl_load_multi_round.py -------------------------------------------------------------------------------- /evaluation/models/shikra_load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/models/shikra_load.py -------------------------------------------------------------------------------- /evaluation/run_evaluation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/run_evaluation.py -------------------------------------------------------------------------------- /evaluation/script/run_evaluation_close_ended.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/script/run_evaluation_close_ended.sh -------------------------------------------------------------------------------- /evaluation/script/run_evaluation_open_ended.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/script/run_evaluation_open_ended.sh -------------------------------------------------------------------------------- /evaluation/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Xiangkui-Cao/VLBiasBench/HEAD/evaluation/utils.py --------------------------------------------------------------------------------