├── CMKP.PNG ├── LICENSE ├── README.md ├── data.JPG ├── data ├── tw_mm_s1 │ ├── test_attribute_cleaned.txt │ ├── test_ocr.txt │ ├── test_ocr_final.txt │ ├── test_src.txt │ ├── test_url_map.pt │ ├── train_attribute_cleaned.txt │ ├── train_ocr.txt │ ├── train_ocr_final.txt │ ├── train_src.txt │ ├── train_url_map.pt │ ├── valid_attribute_cleaned.txt │ ├── valid_ocr.txt │ ├── valid_ocr_final.txt │ ├── valid_src.txt │ └── valid_url_map.pt └── tw_mm_s1_tiny │ ├── test_attribute.pt │ ├── test_img_vgg.pt │ ├── test_src.txt │ ├── test_url_map.pt │ ├── train_attribute.pt │ ├── train_img_vgg.pt │ ├── train_src.txt │ ├── train_url_map.pt │ ├── valid_attribute.pt │ ├── valid_img_vgg.pt │ ├── valid_src.txt │ └── valid_url_map.pt ├── ext_feats ├── infer_OCR │ ├── CMKP_ocr.txt │ ├── get_ocr_for_file.py │ └── process_ocr.py ├── infer_attribute │ ├── CMKP_attributes.txt │ ├── data_loader.py │ ├── model.py │ ├── prepare_attribute_glove_emb.py │ ├── test_batch.py │ └── vocab2017_1k_cleaned.pkl ├── infer_bert │ ├── infer_for_batch.py │ ├── prepare_src_str_map.py │ ├── text_bert.py │ └── text_bert_tweet.py ├── infer_butd │ ├── convert_tsv_to_h5.py │ ├── convert_tsv_to_pkl.py │ ├── generate_tsv.py │ └── h5_reader.py ├── infer_visual_feat │ ├── data_loader.py │ ├── infer_for_batch.py │ ├── infer_for_batch_h5.py │ ├── model.py │ └── prepare_url_map.py └── prepare_tw_glove_emb.py ├── processed_data └── tw_mm_s1 │ ├── glove_emb.pkl │ ├── test.pt │ ├── train.pt │ ├── valid.pt │ └── vocab.pt ├── sh ├── run_exp.py └── sample_pred │ └── best_pred.txt └── unified_model ├── metric_utils.py ├── model.py ├── modules.py ├── multi_head_att ├── encoder.py ├── fc.py ├── modules.py └── submodules.py ├── my_io.py ├── pred_evaluate_func.py ├── predict.py ├── prepare_tw_glove_emb.py ├── preprocess.py ├── run_batches.py ├── sequence_generator.py └── train.py /CMKP.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/CMKP.PNG -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/README.md -------------------------------------------------------------------------------- /data.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data.JPG -------------------------------------------------------------------------------- /data/tw_mm_s1/test_attribute_cleaned.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/test_attribute_cleaned.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/test_ocr.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/test_ocr.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/test_ocr_final.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/test_ocr_final.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/test_src.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/test_src.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/test_url_map.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/test_url_map.pt -------------------------------------------------------------------------------- /data/tw_mm_s1/train_attribute_cleaned.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/train_attribute_cleaned.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/train_ocr.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/train_ocr.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/train_ocr_final.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/train_ocr_final.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/train_src.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/train_src.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/train_url_map.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/train_url_map.pt -------------------------------------------------------------------------------- /data/tw_mm_s1/valid_attribute_cleaned.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/valid_attribute_cleaned.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/valid_ocr.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/valid_ocr.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/valid_ocr_final.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/valid_ocr_final.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/valid_src.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/valid_src.txt -------------------------------------------------------------------------------- /data/tw_mm_s1/valid_url_map.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1/valid_url_map.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/test_attribute.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/test_attribute.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/test_img_vgg.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/test_img_vgg.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/test_src.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/test_src.txt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/test_url_map.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/test_url_map.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/train_attribute.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/train_attribute.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/train_img_vgg.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/train_img_vgg.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/train_src.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/train_src.txt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/train_url_map.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/train_url_map.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/valid_attribute.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/valid_attribute.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/valid_img_vgg.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/valid_img_vgg.pt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/valid_src.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/valid_src.txt -------------------------------------------------------------------------------- /data/tw_mm_s1_tiny/valid_url_map.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/data/tw_mm_s1_tiny/valid_url_map.pt -------------------------------------------------------------------------------- /ext_feats/infer_OCR/CMKP_ocr.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_OCR/CMKP_ocr.txt -------------------------------------------------------------------------------- /ext_feats/infer_OCR/get_ocr_for_file.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_OCR/get_ocr_for_file.py -------------------------------------------------------------------------------- /ext_feats/infer_OCR/process_ocr.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_OCR/process_ocr.py -------------------------------------------------------------------------------- /ext_feats/infer_attribute/CMKP_attributes.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_attribute/CMKP_attributes.txt -------------------------------------------------------------------------------- /ext_feats/infer_attribute/data_loader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_attribute/data_loader.py -------------------------------------------------------------------------------- /ext_feats/infer_attribute/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_attribute/model.py -------------------------------------------------------------------------------- /ext_feats/infer_attribute/prepare_attribute_glove_emb.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_attribute/prepare_attribute_glove_emb.py -------------------------------------------------------------------------------- /ext_feats/infer_attribute/test_batch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_attribute/test_batch.py -------------------------------------------------------------------------------- /ext_feats/infer_attribute/vocab2017_1k_cleaned.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_attribute/vocab2017_1k_cleaned.pkl -------------------------------------------------------------------------------- /ext_feats/infer_bert/infer_for_batch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_bert/infer_for_batch.py -------------------------------------------------------------------------------- /ext_feats/infer_bert/prepare_src_str_map.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_bert/prepare_src_str_map.py -------------------------------------------------------------------------------- /ext_feats/infer_bert/text_bert.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_bert/text_bert.py -------------------------------------------------------------------------------- /ext_feats/infer_bert/text_bert_tweet.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_bert/text_bert_tweet.py -------------------------------------------------------------------------------- /ext_feats/infer_butd/convert_tsv_to_h5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_butd/convert_tsv_to_h5.py -------------------------------------------------------------------------------- /ext_feats/infer_butd/convert_tsv_to_pkl.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_butd/convert_tsv_to_pkl.py -------------------------------------------------------------------------------- /ext_feats/infer_butd/generate_tsv.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_butd/generate_tsv.py -------------------------------------------------------------------------------- /ext_feats/infer_butd/h5_reader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_butd/h5_reader.py -------------------------------------------------------------------------------- /ext_feats/infer_visual_feat/data_loader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_visual_feat/data_loader.py -------------------------------------------------------------------------------- /ext_feats/infer_visual_feat/infer_for_batch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_visual_feat/infer_for_batch.py -------------------------------------------------------------------------------- /ext_feats/infer_visual_feat/infer_for_batch_h5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_visual_feat/infer_for_batch_h5.py -------------------------------------------------------------------------------- /ext_feats/infer_visual_feat/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_visual_feat/model.py -------------------------------------------------------------------------------- /ext_feats/infer_visual_feat/prepare_url_map.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/infer_visual_feat/prepare_url_map.py -------------------------------------------------------------------------------- /ext_feats/prepare_tw_glove_emb.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/ext_feats/prepare_tw_glove_emb.py -------------------------------------------------------------------------------- /processed_data/tw_mm_s1/glove_emb.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/processed_data/tw_mm_s1/glove_emb.pkl -------------------------------------------------------------------------------- /processed_data/tw_mm_s1/test.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/processed_data/tw_mm_s1/test.pt -------------------------------------------------------------------------------- /processed_data/tw_mm_s1/train.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/processed_data/tw_mm_s1/train.pt -------------------------------------------------------------------------------- /processed_data/tw_mm_s1/valid.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/processed_data/tw_mm_s1/valid.pt -------------------------------------------------------------------------------- /processed_data/tw_mm_s1/vocab.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/processed_data/tw_mm_s1/vocab.pt -------------------------------------------------------------------------------- /sh/run_exp.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/sh/run_exp.py -------------------------------------------------------------------------------- /sh/sample_pred/best_pred.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/sh/sample_pred/best_pred.txt -------------------------------------------------------------------------------- /unified_model/metric_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/metric_utils.py -------------------------------------------------------------------------------- /unified_model/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/model.py -------------------------------------------------------------------------------- /unified_model/modules.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/modules.py -------------------------------------------------------------------------------- /unified_model/multi_head_att/encoder.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/multi_head_att/encoder.py -------------------------------------------------------------------------------- /unified_model/multi_head_att/fc.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/multi_head_att/fc.py -------------------------------------------------------------------------------- /unified_model/multi_head_att/modules.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/multi_head_att/modules.py -------------------------------------------------------------------------------- /unified_model/multi_head_att/submodules.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/multi_head_att/submodules.py -------------------------------------------------------------------------------- /unified_model/my_io.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/my_io.py -------------------------------------------------------------------------------- /unified_model/pred_evaluate_func.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/pred_evaluate_func.py -------------------------------------------------------------------------------- /unified_model/predict.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/predict.py -------------------------------------------------------------------------------- /unified_model/prepare_tw_glove_emb.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/prepare_tw_glove_emb.py -------------------------------------------------------------------------------- /unified_model/preprocess.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/preprocess.py -------------------------------------------------------------------------------- /unified_model/run_batches.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/run_batches.py -------------------------------------------------------------------------------- /unified_model/sequence_generator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/sequence_generator.py -------------------------------------------------------------------------------- /unified_model/train.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yuewang-cuhk/CMKP/HEAD/unified_model/train.py --------------------------------------------------------------------------------