├── .gitignore ├── Figure_1_Modality_Gap ├── features_clasp.npy ├── features_clasp_random.npy ├── features_clip.npy ├── features_clip_random.npy ├── features_convirt.npy ├── features_convirt_random.npy ├── features_videoclip.npy ├── features_videoclip_random.npy ├── repr_clasp.ipynb ├── repr_clip.ipynb ├── repr_convirt.ipynb ├── repr_videoclip.ipynb └── visualize.ipynb ├── Figure_2_Cone_Effect ├── Figure_2a_random_init_random_data │ ├── coco-extract.ipynb │ └── visualize.ipynb ├── Figure_2a_random_init_real_data │ ├── coco-extract.ipynb │ └── visualize.ipynb ├── Figure_2a_real_features │ └── real_features.ipynb ├── Figure_2b_random_MLP_layerwise │ ├── bias_linear_relu.ipynb │ └── no_bias │ │ └── linear_relu.ipynb └── Figure_2c_scatter_cones_random_init │ ├── MLP │ └── scatter_cones_linear_relu.ipynb │ ├── random_data │ ├── coco-extract.ipynb │ ├── visualizePCA.ipynb │ └── visualizeUMAP.ipynb │ ├── real_data │ ├── coco-extract.ipynb │ ├── visualizePCA.ipynb │ └── visualizeUMAP.ipynb │ └── real_data_ImageNet_pretrained │ ├── ImageNet-Pretrained-Cones.png │ ├── README.md │ ├── coco-extract.ipynb │ └── visualizeUMAP.ipynb ├── Figure_3_Contrastive_Learning ├── 3d_sphere.ipynb ├── Appendix_3d_sphere.ipynb ├── get_gap_stats.ipynb ├── mismatched_simulation.ipynb └── plot_optimization_exp.ipynb ├── LICENSE ├── README.md ├── Table_1_Implications_CLIP_Zero_Shot ├── shifting │ └── shift_features.ipynb ├── simulation │ └── simulation.ipynb └── training │ ├── datasets.py │ ├── train_clip.py │ └── utils.py ├── Table_2_Implications_CLIP_Fairness ├── coco-extract.ipynb └── shift_CLIP_FairFace_Bias.ipynb ├── docs └── figures │ ├── Figure1.png │ ├── Figure2.jpg │ ├── Figure2ab.png │ ├── Figure2c.png │ ├── Figure3.jpg │ ├── Tables.png │ ├── Theorem1.png │ ├── Theorem2.png │ └── Theorem_variance.png ├── environment.yml └── util ├── gap_amend_std.ipynb └── get_arch.ipynb /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/.gitignore -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_clasp.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_clasp.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_clasp_random.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_clasp_random.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_clip.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_clip.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_clip_random.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_clip_random.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_convirt.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_convirt.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_convirt_random.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_convirt_random.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_videoclip.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_videoclip.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/features_videoclip_random.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/features_videoclip_random.npy -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/repr_clasp.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/repr_clasp.ipynb -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/repr_clip.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/repr_clip.ipynb -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/repr_convirt.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/repr_convirt.ipynb -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/repr_videoclip.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/repr_videoclip.ipynb -------------------------------------------------------------------------------- /Figure_1_Modality_Gap/visualize.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_1_Modality_Gap/visualize.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2a_random_init_random_data/coco-extract.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2a_random_init_random_data/coco-extract.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2a_random_init_random_data/visualize.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2a_random_init_random_data/visualize.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2a_random_init_real_data/coco-extract.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2a_random_init_real_data/coco-extract.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2a_random_init_real_data/visualize.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2a_random_init_real_data/visualize.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2a_real_features/real_features.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2a_real_features/real_features.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2b_random_MLP_layerwise/bias_linear_relu.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2b_random_MLP_layerwise/bias_linear_relu.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2b_random_MLP_layerwise/no_bias/linear_relu.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2b_random_MLP_layerwise/no_bias/linear_relu.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/MLP/scatter_cones_linear_relu.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/MLP/scatter_cones_linear_relu.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/random_data/coco-extract.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/random_data/coco-extract.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/random_data/visualizePCA.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/random_data/visualizePCA.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/random_data/visualizeUMAP.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/random_data/visualizeUMAP.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data/coco-extract.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data/coco-extract.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data/visualizePCA.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data/visualizePCA.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data/visualizeUMAP.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data/visualizeUMAP.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/ImageNet-Pretrained-Cones.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/ImageNet-Pretrained-Cones.png -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | ![](./ImageNet-Pretrained-Cones.png) 5 | -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/coco-extract.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/coco-extract.ipynb -------------------------------------------------------------------------------- /Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/visualizeUMAP.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_2_Cone_Effect/Figure_2c_scatter_cones_random_init/real_data_ImageNet_pretrained/visualizeUMAP.ipynb -------------------------------------------------------------------------------- /Figure_3_Contrastive_Learning/3d_sphere.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_3_Contrastive_Learning/3d_sphere.ipynb -------------------------------------------------------------------------------- /Figure_3_Contrastive_Learning/Appendix_3d_sphere.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_3_Contrastive_Learning/Appendix_3d_sphere.ipynb -------------------------------------------------------------------------------- /Figure_3_Contrastive_Learning/get_gap_stats.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_3_Contrastive_Learning/get_gap_stats.ipynb -------------------------------------------------------------------------------- /Figure_3_Contrastive_Learning/mismatched_simulation.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_3_Contrastive_Learning/mismatched_simulation.ipynb -------------------------------------------------------------------------------- /Figure_3_Contrastive_Learning/plot_optimization_exp.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Figure_3_Contrastive_Learning/plot_optimization_exp.ipynb -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/README.md -------------------------------------------------------------------------------- /Table_1_Implications_CLIP_Zero_Shot/shifting/shift_features.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_1_Implications_CLIP_Zero_Shot/shifting/shift_features.ipynb -------------------------------------------------------------------------------- /Table_1_Implications_CLIP_Zero_Shot/simulation/simulation.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_1_Implications_CLIP_Zero_Shot/simulation/simulation.ipynb -------------------------------------------------------------------------------- /Table_1_Implications_CLIP_Zero_Shot/training/datasets.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_1_Implications_CLIP_Zero_Shot/training/datasets.py -------------------------------------------------------------------------------- /Table_1_Implications_CLIP_Zero_Shot/training/train_clip.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_1_Implications_CLIP_Zero_Shot/training/train_clip.py -------------------------------------------------------------------------------- /Table_1_Implications_CLIP_Zero_Shot/training/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_1_Implications_CLIP_Zero_Shot/training/utils.py -------------------------------------------------------------------------------- /Table_2_Implications_CLIP_Fairness/coco-extract.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_2_Implications_CLIP_Fairness/coco-extract.ipynb -------------------------------------------------------------------------------- /Table_2_Implications_CLIP_Fairness/shift_CLIP_FairFace_Bias.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/Table_2_Implications_CLIP_Fairness/shift_CLIP_FairFace_Bias.ipynb -------------------------------------------------------------------------------- /docs/figures/Figure1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Figure1.png -------------------------------------------------------------------------------- /docs/figures/Figure2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Figure2.jpg -------------------------------------------------------------------------------- /docs/figures/Figure2ab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Figure2ab.png -------------------------------------------------------------------------------- /docs/figures/Figure2c.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Figure2c.png -------------------------------------------------------------------------------- /docs/figures/Figure3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Figure3.jpg -------------------------------------------------------------------------------- /docs/figures/Tables.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Tables.png -------------------------------------------------------------------------------- /docs/figures/Theorem1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Theorem1.png -------------------------------------------------------------------------------- /docs/figures/Theorem2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Theorem2.png -------------------------------------------------------------------------------- /docs/figures/Theorem_variance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/docs/figures/Theorem_variance.png -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/environment.yml -------------------------------------------------------------------------------- /util/gap_amend_std.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/util/gap_amend_std.ipynb -------------------------------------------------------------------------------- /util/get_arch.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Weixin-Liang/Modality-Gap/HEAD/util/get_arch.ipynb --------------------------------------------------------------------------------