├── .gitignore ├── README.md ├── configs ├── base.yaml └── models │ ├── idefics2.yaml │ ├── llava.yaml │ ├── mantis.yaml │ ├── mistral.yaml │ └── vicuna.yaml ├── data └── examples │ ├── image_icl │ ├── Mr_Krabs.png │ ├── Mrs_Puff.png │ └── Sandy_Cheeks.png │ ├── instruction │ └── Swamp_Wallaby.png │ ├── task_conflict │ └── VQA_Food.png │ └── text_icl │ └── France.png ├── demo.ipynb ├── download_data.sh ├── environment.yml ├── experiment_base.sh ├── experiment_ensemble.sh └── src ├── __init__.py ├── xpatch_dataset.py ├── xpatch_evaluate.py └── xpatch_helpers.py /.gitignore: -------------------------------------------------------------------------------- 1 | **__pycache__** 2 | runs/ 3 | data/ 4 | out/ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/README.md -------------------------------------------------------------------------------- /configs/base.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/configs/base.yaml -------------------------------------------------------------------------------- /configs/models/idefics2.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/configs/models/idefics2.yaml -------------------------------------------------------------------------------- /configs/models/llava.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/configs/models/llava.yaml -------------------------------------------------------------------------------- /configs/models/mantis.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/configs/models/mantis.yaml -------------------------------------------------------------------------------- /configs/models/mistral.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/configs/models/mistral.yaml -------------------------------------------------------------------------------- /configs/models/vicuna.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/configs/models/vicuna.yaml -------------------------------------------------------------------------------- /data/examples/image_icl/Mr_Krabs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/data/examples/image_icl/Mr_Krabs.png -------------------------------------------------------------------------------- /data/examples/image_icl/Mrs_Puff.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/data/examples/image_icl/Mrs_Puff.png -------------------------------------------------------------------------------- /data/examples/image_icl/Sandy_Cheeks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/data/examples/image_icl/Sandy_Cheeks.png -------------------------------------------------------------------------------- /data/examples/instruction/Swamp_Wallaby.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/data/examples/instruction/Swamp_Wallaby.png -------------------------------------------------------------------------------- /data/examples/task_conflict/VQA_Food.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/data/examples/task_conflict/VQA_Food.png -------------------------------------------------------------------------------- /data/examples/text_icl/France.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/data/examples/text_icl/France.png -------------------------------------------------------------------------------- /demo.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/demo.ipynb -------------------------------------------------------------------------------- /download_data.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/download_data.sh -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/environment.yml -------------------------------------------------------------------------------- /experiment_base.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/experiment_base.sh -------------------------------------------------------------------------------- /experiment_ensemble.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/experiment_ensemble.sh -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/xpatch_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/src/xpatch_dataset.py -------------------------------------------------------------------------------- /src/xpatch_evaluate.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/src/xpatch_evaluate.py -------------------------------------------------------------------------------- /src/xpatch_helpers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/g-luo/vlm_cross_modal_reps/HEAD/src/xpatch_helpers.py --------------------------------------------------------------------------------