├── MLLMs.html ├── README.md ├── assets ├── 904830090386735105_0.png ├── 910111384045854720_0.png ├── 910258440077025280_0.png ├── 912990487501557761_0.png ├── 920216066416173056_0.png ├── 930415579294584834_0.png ├── Figure1.png ├── Figure2.png ├── MultimodalRobustness_EMNLP2022.pdf ├── Verma_RobustnessFinal.pdf ├── acl2023.png ├── adobe-logo.png ├── camera.jpg ├── climbing.jpg ├── dilution_insertion.png ├── emnlp-logo.png ├── emnlp2022.png ├── gpt4v.png ├── gt-logo.png ├── humanitarian_claude3opus.png ├── humanitarian_gemini.png ├── humanitarian_gpt4v.png ├── humanitarian_llaval1point5.png ├── humanitarian_mm_fusion.png ├── humanitarian_moondream1.png ├── label_definition.txt ├── screenshot.png ├── teaser-emnlp.jpg ├── waiter.jpg └── young_child.jpg ├── baselines ├── gpt │ └── generate_text.py └── most_sim_image_caption.py ├── data_samples ├── crisis_humanitarianism │ ├── 869972354004393987_0.jpg │ ├── 901701789279547392_2.jpg │ ├── 910524793740513280_2.jpg │ ├── 919871048652394496_0.jpg │ ├── 931046661274701825_2.jpg │ └── crisis_humanitarianism.csv └── sentiment_detection │ ├── 2261.jpg │ ├── 3032.jpg │ ├── 364.jpg │ ├── 485.jpg │ ├── 710.jpg │ └── sentiment_detection.csv ├── evaluation ├── analysis.py ├── self_bleu.py ├── topical_sim.py ├── vector_analysis.py └── xmodal_matching.py ├── gpt4v_insertions.html ├── index.html ├── keywords └── select_keywords.py ├── multimodal_model ├── .DS_Store ├── .ipynb_checkpoints │ └── text_only_classification-checkpoint.ipynb ├── img_embs │ ├── image_embedding_256_dev.npz │ ├── image_embedding_256_test.npz │ └── image_embedding_256_train.npz ├── main.py ├── multimodal_classifier.pth ├── text_embs │ ├── .DS_Store │ ├── embeddings_test.pickle │ ├── embeddings_train.pickle │ └── embeddings_val.pickle └── text_only_classification.ipynb └── training ├── evaluation.py ├── inference.py └── training.py /MLLMs.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/MLLMs.html -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/README.md -------------------------------------------------------------------------------- /assets/904830090386735105_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/904830090386735105_0.png -------------------------------------------------------------------------------- /assets/910111384045854720_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/910111384045854720_0.png -------------------------------------------------------------------------------- /assets/910258440077025280_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/910258440077025280_0.png -------------------------------------------------------------------------------- /assets/912990487501557761_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/912990487501557761_0.png -------------------------------------------------------------------------------- /assets/920216066416173056_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/920216066416173056_0.png -------------------------------------------------------------------------------- /assets/930415579294584834_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/930415579294584834_0.png -------------------------------------------------------------------------------- /assets/Figure1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/Figure1.png -------------------------------------------------------------------------------- /assets/Figure2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/Figure2.png -------------------------------------------------------------------------------- /assets/MultimodalRobustness_EMNLP2022.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/MultimodalRobustness_EMNLP2022.pdf -------------------------------------------------------------------------------- /assets/Verma_RobustnessFinal.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/Verma_RobustnessFinal.pdf -------------------------------------------------------------------------------- /assets/acl2023.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/acl2023.png -------------------------------------------------------------------------------- /assets/adobe-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/adobe-logo.png -------------------------------------------------------------------------------- /assets/camera.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/camera.jpg -------------------------------------------------------------------------------- /assets/climbing.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/climbing.jpg -------------------------------------------------------------------------------- /assets/dilution_insertion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/dilution_insertion.png -------------------------------------------------------------------------------- /assets/emnlp-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/emnlp-logo.png -------------------------------------------------------------------------------- /assets/emnlp2022.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/emnlp2022.png -------------------------------------------------------------------------------- /assets/gpt4v.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/gpt4v.png -------------------------------------------------------------------------------- /assets/gt-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/gt-logo.png -------------------------------------------------------------------------------- /assets/humanitarian_claude3opus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/humanitarian_claude3opus.png -------------------------------------------------------------------------------- /assets/humanitarian_gemini.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/humanitarian_gemini.png -------------------------------------------------------------------------------- /assets/humanitarian_gpt4v.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/humanitarian_gpt4v.png -------------------------------------------------------------------------------- /assets/humanitarian_llaval1point5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/humanitarian_llaval1point5.png -------------------------------------------------------------------------------- /assets/humanitarian_mm_fusion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/humanitarian_mm_fusion.png -------------------------------------------------------------------------------- /assets/humanitarian_moondream1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/humanitarian_moondream1.png -------------------------------------------------------------------------------- /assets/label_definition.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/label_definition.txt -------------------------------------------------------------------------------- /assets/screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/screenshot.png -------------------------------------------------------------------------------- /assets/teaser-emnlp.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/teaser-emnlp.jpg -------------------------------------------------------------------------------- /assets/waiter.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/waiter.jpg -------------------------------------------------------------------------------- /assets/young_child.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/assets/young_child.jpg -------------------------------------------------------------------------------- /baselines/gpt/generate_text.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/baselines/gpt/generate_text.py -------------------------------------------------------------------------------- /baselines/most_sim_image_caption.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/baselines/most_sim_image_caption.py -------------------------------------------------------------------------------- /data_samples/crisis_humanitarianism/869972354004393987_0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/crisis_humanitarianism/869972354004393987_0.jpg -------------------------------------------------------------------------------- /data_samples/crisis_humanitarianism/901701789279547392_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/crisis_humanitarianism/901701789279547392_2.jpg -------------------------------------------------------------------------------- /data_samples/crisis_humanitarianism/910524793740513280_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/crisis_humanitarianism/910524793740513280_2.jpg -------------------------------------------------------------------------------- /data_samples/crisis_humanitarianism/919871048652394496_0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/crisis_humanitarianism/919871048652394496_0.jpg -------------------------------------------------------------------------------- /data_samples/crisis_humanitarianism/931046661274701825_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/crisis_humanitarianism/931046661274701825_2.jpg -------------------------------------------------------------------------------- /data_samples/crisis_humanitarianism/crisis_humanitarianism.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/crisis_humanitarianism/crisis_humanitarianism.csv -------------------------------------------------------------------------------- /data_samples/sentiment_detection/2261.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/sentiment_detection/2261.jpg -------------------------------------------------------------------------------- /data_samples/sentiment_detection/3032.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/sentiment_detection/3032.jpg -------------------------------------------------------------------------------- /data_samples/sentiment_detection/364.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/sentiment_detection/364.jpg -------------------------------------------------------------------------------- /data_samples/sentiment_detection/485.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/sentiment_detection/485.jpg -------------------------------------------------------------------------------- /data_samples/sentiment_detection/710.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/sentiment_detection/710.jpg -------------------------------------------------------------------------------- /data_samples/sentiment_detection/sentiment_detection.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/data_samples/sentiment_detection/sentiment_detection.csv -------------------------------------------------------------------------------- /evaluation/analysis.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/evaluation/analysis.py -------------------------------------------------------------------------------- /evaluation/self_bleu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/evaluation/self_bleu.py -------------------------------------------------------------------------------- /evaluation/topical_sim.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/evaluation/topical_sim.py -------------------------------------------------------------------------------- /evaluation/vector_analysis.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/evaluation/vector_analysis.py -------------------------------------------------------------------------------- /evaluation/xmodal_matching.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/evaluation/xmodal_matching.py -------------------------------------------------------------------------------- /gpt4v_insertions.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/gpt4v_insertions.html -------------------------------------------------------------------------------- /index.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/index.html -------------------------------------------------------------------------------- /keywords/select_keywords.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/keywords/select_keywords.py -------------------------------------------------------------------------------- /multimodal_model/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/.DS_Store -------------------------------------------------------------------------------- /multimodal_model/.ipynb_checkpoints/text_only_classification-checkpoint.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/.ipynb_checkpoints/text_only_classification-checkpoint.ipynb -------------------------------------------------------------------------------- /multimodal_model/img_embs/image_embedding_256_dev.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/img_embs/image_embedding_256_dev.npz -------------------------------------------------------------------------------- /multimodal_model/img_embs/image_embedding_256_test.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/img_embs/image_embedding_256_test.npz -------------------------------------------------------------------------------- /multimodal_model/img_embs/image_embedding_256_train.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/img_embs/image_embedding_256_train.npz -------------------------------------------------------------------------------- /multimodal_model/main.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/main.py -------------------------------------------------------------------------------- /multimodal_model/multimodal_classifier.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/multimodal_classifier.pth -------------------------------------------------------------------------------- /multimodal_model/text_embs/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/text_embs/.DS_Store -------------------------------------------------------------------------------- /multimodal_model/text_embs/embeddings_test.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/text_embs/embeddings_test.pickle -------------------------------------------------------------------------------- /multimodal_model/text_embs/embeddings_train.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/text_embs/embeddings_train.pickle -------------------------------------------------------------------------------- /multimodal_model/text_embs/embeddings_val.pickle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/text_embs/embeddings_val.pickle -------------------------------------------------------------------------------- /multimodal_model/text_only_classification.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/multimodal_model/text_only_classification.ipynb -------------------------------------------------------------------------------- /training/evaluation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/training/evaluation.py -------------------------------------------------------------------------------- /training/inference.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/training/inference.py -------------------------------------------------------------------------------- /training/training.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/claws-lab/multimodal-robustness/HEAD/training/training.py --------------------------------------------------------------------------------