├── README.md ├── assets ├── Idea.PNG ├── Overview.PNG ├── YOCO_idea.PNG └── YOCO_overview.PNG ├── data_gen ├── data_partition_crisismmd.py ├── data_partition_hateful.py ├── data_process_medalpaca.py ├── data_process_vqarad.py ├── gen_data_crisismmd.py ├── gen_data_crisismmd_missing.py ├── gen_data_crisismmd_missing_aug.py ├── gen_data_crisismmd_mix.py ├── gen_data_crisismmd_mix_aug.py ├── gen_data_crisismmd_single.py ├── gen_data_crisismmd_single_aug.py ├── gen_data_hateful.py ├── gen_data_hateful_missing.py ├── gen_data_hateful_missing_aug.py ├── gen_data_hateful_mix.py ├── gen_data_hateful_mix_aug.py ├── gen_data_hateful_single.py ├── gen_data_hateful_single_aug.py ├── gen_data_medical_vtqa_mix.py └── gen_data_medical_vtqa_single.py ├── eval_crisismmd.py ├── eval_crisismmd_aug.py ├── eval_hateful.py ├── eval_hateful_aug.py ├── eval_medical.py ├── eval_medical_gpt.py ├── eval_medical_gpt_slake.py ├── eval_medical_slake.py ├── finetune ├── __init__.py ├── dataset.py ├── federated_learning │ ├── __init__.py │ ├── fed_global.py │ └── fed_utils.py ├── finetune.py ├── finetune_lora.sh └── trainer.py ├── requirements.txt ├── start.sh ├── vqa.py ├── vqa_eval.py ├── vqa_eval_slake.py └── vqa_slake.py /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/README.md -------------------------------------------------------------------------------- /assets/Idea.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/assets/Idea.PNG -------------------------------------------------------------------------------- /assets/Overview.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/assets/Overview.PNG -------------------------------------------------------------------------------- /assets/YOCO_idea.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/assets/YOCO_idea.PNG -------------------------------------------------------------------------------- /assets/YOCO_overview.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/assets/YOCO_overview.PNG -------------------------------------------------------------------------------- /data_gen/data_partition_crisismmd.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/data_partition_crisismmd.py -------------------------------------------------------------------------------- /data_gen/data_partition_hateful.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/data_partition_hateful.py -------------------------------------------------------------------------------- /data_gen/data_process_medalpaca.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/data_process_medalpaca.py -------------------------------------------------------------------------------- /data_gen/data_process_vqarad.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/data_process_vqarad.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd_missing.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd_missing.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd_missing_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd_missing_aug.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd_mix.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd_mix.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd_mix_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd_mix_aug.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd_single.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd_single.py -------------------------------------------------------------------------------- /data_gen/gen_data_crisismmd_single_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_crisismmd_single_aug.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful_missing.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful_missing.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful_missing_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful_missing_aug.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful_mix.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful_mix.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful_mix_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful_mix_aug.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful_single.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful_single.py -------------------------------------------------------------------------------- /data_gen/gen_data_hateful_single_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_hateful_single_aug.py -------------------------------------------------------------------------------- /data_gen/gen_data_medical_vtqa_mix.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_medical_vtqa_mix.py -------------------------------------------------------------------------------- /data_gen/gen_data_medical_vtqa_single.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/data_gen/gen_data_medical_vtqa_single.py -------------------------------------------------------------------------------- /eval_crisismmd.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_crisismmd.py -------------------------------------------------------------------------------- /eval_crisismmd_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_crisismmd_aug.py -------------------------------------------------------------------------------- /eval_hateful.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_hateful.py -------------------------------------------------------------------------------- /eval_hateful_aug.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_hateful_aug.py -------------------------------------------------------------------------------- /eval_medical.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_medical.py -------------------------------------------------------------------------------- /eval_medical_gpt.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_medical_gpt.py -------------------------------------------------------------------------------- /eval_medical_gpt_slake.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_medical_gpt_slake.py -------------------------------------------------------------------------------- /eval_medical_slake.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/eval_medical_slake.py -------------------------------------------------------------------------------- /finetune/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /finetune/dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/dataset.py -------------------------------------------------------------------------------- /finetune/federated_learning/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/federated_learning/__init__.py -------------------------------------------------------------------------------- /finetune/federated_learning/fed_global.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/federated_learning/fed_global.py -------------------------------------------------------------------------------- /finetune/federated_learning/fed_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/federated_learning/fed_utils.py -------------------------------------------------------------------------------- /finetune/finetune.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/finetune.py -------------------------------------------------------------------------------- /finetune/finetune_lora.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/finetune_lora.sh -------------------------------------------------------------------------------- /finetune/trainer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/finetune/trainer.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/requirements.txt -------------------------------------------------------------------------------- /start.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/start.sh -------------------------------------------------------------------------------- /vqa.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/vqa.py -------------------------------------------------------------------------------- /vqa_eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/vqa_eval.py -------------------------------------------------------------------------------- /vqa_eval_slake.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/vqa_eval_slake.py -------------------------------------------------------------------------------- /vqa_slake.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/1xbq1/FedMLLM/HEAD/vqa_slake.py --------------------------------------------------------------------------------