├── README.md ├── Version-1 ├── README.md ├── mnre_obj_org.zip └── mnre_txt_org.zip ├── mnre_image └── README.md ├── mnre_txt ├── mnre_test.txt ├── mnre_train.txt └── mnre_val .txt └── pic ├── Intro.png ├── README.md ├── case1.png ├── statistic.pdf └── statistic.png /README.md: -------------------------------------------------------------------------------- 1 | # MNRE 2 | Resource and Code for ICME 2021 paper [MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts](https://ieeexplore.ieee.org/abstract/document/9428274) 3 | 4 | ## Recent Updates 5 | - [x] 2021.6.22 We provide MNRE-2, a refined version which merges several ambigious categories with much more support samples. The original version has been moved to [Version-1](/Version-1/) 6 | 7 | ## Overview 8 | This project aims to present a new task -- multimodal neural relation extraction and a dataset (MNRE) for model evaluation. The MNRE task requires a understanding of both vision and language. We envisage a range of well-designed methods and resources for such a challenge that would boost the development of multimodal alignment towards a higher semantic level. 9 | 10 | 11 | 12 | This is an example of multimodal relation extraction in Twitter. There are three entities in this sentence: "JFK", "Obama" and "Harvard". The main task of relation extraction is to identify the relations of each entity pair. Previous works incorrectly classify the relations of "JFK" and "Harvard" as "Residence" and the "JFK" and "Obama" as "Spouse" due to the missing of contexts. However, we can know that "JFK" and "Harvard" are in the relation of "Graduated at" with the visual concepts "Bachelor cap" and "Gown". Still, the relations of "JFK" and "Obama" can be identified as "Alumni" with the guidance of all the visual objects about "Campus". 13 | 14 | ## Data Statistics 15 | 16 | ### Data Statistics Compared to Previous NRE Dataset 17 | 18 | Dataset | # Image | # Word | # Sentence | # Entity | # Relation | # Instance 19 | -- | -- | -- | -- | -- | -- | -- 20 | SemEval-2010 Task 8 | - | 205k | 10,717 | 21,434 | 9 | 8,853 21 | ACE 2003-2004 | - | 297k | 12,783 | 46,108 | 24 | 16,771 22 | TACRED | - | 1,823k | 53,791 | 152,527 | 41 | 21,773 23 | FewRel | - | 1,397k | 56,109 | 72,124 | 100 | 70,000 24 | MNRE | 9,201 | 258k | 9,201 | 30,970 | 23 | 15,485 25 | 26 | This is the version 2 of our MNRE dataset. We refine and merge some categories for better understandings. The dataset contains 15,484 samples and 9,201 images with 23 relation categories. We split the dataset into training, development and testing set with 12247, 1624 and 1614 samples, respectively. 27 | 28 | 29 | ### Category Distribution 30 | 31 | 32 | 33 | We start tagging relation types depending on the entity types. For example, the relations between one person and another person can be classified into ''alumni'', ''couple'' and ''relative'' et al. 34 | 35 | ## Data Collection 36 | 37 | We build the original corpus from three sources: two available multimodal named entity recognition datasets - [Twitter15 and Twitter17](https://github.com/jefferyYu/UMT), and crawling data from [Twitter](https://archive.org/details/twitterstream). 38 | 39 | We utilize a pretrained NER tagging tool [elmo](https://allennlp.org/elmo) for extracting both entities and their corresponding types. 40 | 41 | ## Data Usage 42 | 43 | Our processed textual relations are in `./mnre_txt/`, the image data can be downloaded [here](https://drive.google.com/file/d/1FYiJFtRayWY32nRH0rdycYzIdDcMmDFR/view?usp=sharing). 44 | 45 | >Each sentence is split into several instances (depending on the number of relations). 46 | >Each line contains 47 | >``` 48 | >'token': Texts preprocessed by a tokenizer 49 | >'h': Head entities and their positions in a sentence 50 | >'t': Tail entities and their positions in a sentence 51 | >'image_id': You can find the corresponding images using the link above 52 | >'relation': The relations and entity categories 53 | >``` 54 | 55 | ## Case Study 56 | 57 | 58 | 59 | Four examples for illustrating the effectiveness of visual information in extracting relations. The first line shows that the visual objects and their attributes can help in identifying relations. Further more, we show that the interactions of person-to-person or person-to-object can also provide clues for classifying relations. 60 | 61 | 62 | ## Citation 63 | If you find this repo helpful, please cite the following: 64 | ```latex 65 | @inproceedings{zheng2021mnre, 66 | title={MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts}, 67 | author={Zheng, Changmeng and Wu, Zhiwei and Feng, Junhao and Fu, Ze and Cai, Yi}, 68 | booktitle={2021 IEEE International Conference on Multimedia and Expo (ICME)}, 69 | pages={1--6}, 70 | year={2021}, 71 | organization={IEEE} 72 | } 73 | ``` 74 | -------------------------------------------------------------------------------- /Version-1/README.md: -------------------------------------------------------------------------------- 1 | # MNRE-1 2 | MNRE-1 is the original version of our dataset provided in ICME 2021. 3 | 4 | ## Data Usage 5 | There are 10081 samples of the dataset (8 samples were removed since they are obviously incorrect due to the inconsistent tagging scheme). We split the dataset into training, development and testing set with 7824, 975 and 1282 samples, respectively. 6 | 7 | We also provide the texts, images and the pretrained object features extracted by [Mask RCNN](https://github.com/matterport/Mask_RCNN). 8 | 9 | The images can be downloaded [here](https://drive.google.com/file/d/1uq0Dr1RqZn-VVKe1UrG9b8xMeldYwkJs/view?usp=sharing) 10 | -------------------------------------------------------------------------------- /Version-1/mnre_obj_org.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thecharm/MNRE/354211ed7b8d066f989ade6fa6c4a387e3c6de81/Version-1/mnre_obj_org.zip -------------------------------------------------------------------------------- /Version-1/mnre_txt_org.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thecharm/MNRE/354211ed7b8d066f989ade6fa6c4a387e3c6de81/Version-1/mnre_txt_org.zip -------------------------------------------------------------------------------- /mnre_image/README.md: -------------------------------------------------------------------------------- 1 | The images used in MNRE dataset can be downloaded [here](https://drive.google.com/file/d/1FYiJFtRayWY32nRH0rdycYzIdDcMmDFR/view?usp=sharing) 2 | -------------------------------------------------------------------------------- /pic/Intro.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thecharm/MNRE/354211ed7b8d066f989ade6fa6c4a387e3c6de81/pic/Intro.png -------------------------------------------------------------------------------- /pic/README.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /pic/case1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thecharm/MNRE/354211ed7b8d066f989ade6fa6c4a387e3c6de81/pic/case1.png -------------------------------------------------------------------------------- /pic/statistic.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thecharm/MNRE/354211ed7b8d066f989ade6fa6c4a387e3c6de81/pic/statistic.pdf -------------------------------------------------------------------------------- /pic/statistic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/thecharm/MNRE/354211ed7b8d066f989ade6fa6c4a387e3c6de81/pic/statistic.png --------------------------------------------------------------------------------