├── .gitattributes ├── .gitignore ├── README.md ├── SemEval2010_task8_all_data ├── SEMEVAL_TASK8_FULL_RELEASE_README.txt ├── SemEval2010_task8_scorer-v1.2 │ ├── README.txt │ ├── answer_key1.txt │ ├── answer_key2.txt │ ├── answer_key3.txt │ ├── answer_key5.txt │ ├── proposed_answer1.txt │ ├── proposed_answer2.txt │ ├── proposed_answer3.txt │ ├── proposed_answer4.txt │ ├── proposed_answer5.txt │ ├── result_scores1.txt │ ├── result_scores2.txt │ ├── result_scores3.txt │ ├── result_scores5.txt │ ├── semeval2010_task8_format_checker.pl │ └── semeval2010_task8_scorer-v1.2.pl ├── SemEval2010_task8_testing │ ├── README.txt │ └── TEST_FILE.txt ├── SemEval2010_task8_testing_keys │ ├── TEST_FILE_CLEAN.TXT │ ├── TEST_FILE_FULL.TXT │ └── TEST_FILE_KEY.TXT └── SemEval2010_task8_training │ ├── README.txt │ ├── TRAIN_DISTRIB.TXT │ ├── TRAIN_FILE.TXT │ ├── TRAIN_TEST_DISTRIB.TXT │ ├── Task8_Guidelines.pdf │ ├── Task8_Relation1.pdf │ ├── Task8_Relation2.pdf │ ├── Task8_Relation3.pdf │ ├── Task8_Relation4.pdf │ ├── Task8_Relation5.pdf │ ├── Task8_Relation6.pdf │ ├── Task8_Relation7.pdf │ ├── Task8_Relation8.pdf │ └── Task8_Relation9.pdf ├── configure.py ├── data_helpers.py ├── logger.py ├── model ├── attention.py └── entity_att_lstm.py ├── requirements.txt ├── resource └── target.txt ├── self-attention-visualization.ipynb ├── train.py ├── utils.py ├── visualization.html └── visualize.py /.gitattributes: -------------------------------------------------------------------------------- 1 | *.ipynb linguist-vendored 2 | *.html linguist-vendored 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | __pycache__ 3 | runs/ 4 | resource/ 5 | .ipynb_checkpoints/ 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Entity-aware Attention for Relation Classification 2 | 3 | ![model](https://user-images.githubusercontent.com/15166794/52579582-c7339100-2e69-11e9-9081-711e7576e717.png) 4 | 5 | This repository contains the official TensorFlow implementation of the following paper: 6 | 7 | > **Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing**
8 | > Joohong Lee, Sangwoo Seo, Yong Suk Choi
9 | > [https://arxiv.org/abs/1901.08163](https://arxiv.org/abs/1901.08163) 10 | > 11 | > **Abstract:** *Classifying semantic relations between entity pairs in sentences is an important task in Natural Language Processing (NLP). Most previous models for relation classification rely on the high-level lexical and syntactic features obtained by NLP tools such as WordNet, dependency parser, part-of-speech (POS) tagger, and named entity recognizers (NER). In addition, state-of-the-art neural models based on attention mechanisms do not fully utilize information of entity that may be the most crucial features for relation classification. To address these issues, we propose a novel end-to-end recurrent neural model which incorporates an entity-aware attention mechanism with a latent entity typing (LET) method. Our model not only utilizes entities and their latent types as features effectively but also is more interpretable by visualizing attention mechanisms applied to our model and results of LET. Experimental results on the SemEval-2010 Task 8, one of the most popular relation classification task, demonstrate that our model outperforms existing state-of-the-art models without any high-level features.* 12 | 13 | ## Usage 14 | ### Train / Test 15 | * Train data is located in "*SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT*". 16 | * You can apply some pre-trained word embeddings: [word2vec](https://code.google.com/archive/p/word2vec/), [glove100](https://nlp.stanford.edu/projects/glove/), [glove300](https://nlp.stanford.edu/projects/glove/), and [elmo](https://tfhub.dev/google/elmo/1). The pre-trained files should be located in `resource/`. [Check this code](https://github.com/roomylee/entity-aware-relation-classification/blob/f77668088210ce2bb0e94033bdf1cabb45c0bbf0/train.py#L115). 17 | * In every evaluation step, the test performance is evaluated by test dataset located in "*SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT*". 18 | 19 | ##### Display help message: 20 | ```bash 21 | $ python train.py --help 22 | ``` 23 | ##### Train Example: 24 | ```bash 25 | $ python train.py --embeddings glove300 26 | ``` 27 | 28 | 29 | ## Visualization 30 | * Self Attention 31 | ![sa](https://user-images.githubusercontent.com/15166794/52579583-c7339100-2e69-11e9-93a3-b1aa2aafa19f.png) 32 | * Latent Type Representations 33 | ![vec](https://user-images.githubusercontent.com/15166794/52579615-d6b2da00-2e69-11e9-80cb-3b566c28918a.png) 34 | * Sets of Entities grouped by Latent Type 35 | ![type](https://user-images.githubusercontent.com/15166794/52579616-d74b7080-2e69-11e9-9e3a-c027eb01413b.png) 36 | 37 | 38 | ## SemEval-2010 Task #8 39 | * Given: a pair of *nominals* 40 | * Goal: recognize the semantic relation between these nominals. 41 | * Example: 42 | * "There were apples, **pears** and oranges in the **bowl**." 43 |
→ *CONTENT-CONTAINER(pears, bowl)* 44 | * “The cup contained **tea** from dried **ginseng**.” 45 |
→ *ENTITY-ORIGIN(tea, ginseng)* 46 | 47 | 48 | ### The Inventory of Semantic Relations 49 | 1. *Cause-Effect(CE)*: An event or object leads to an effect(those cancers were caused by radiation exposures) 50 | 2. *Instrument-Agency(IA)*: An agent uses an instrument(phone operator) 51 | 3. *Product-Producer(PP)*: A producer causes a product to exist (a factory manufactures suits) 52 | 4. *Content-Container(CC)*: An object is physically stored in a delineated area of space (a bottle full of honey was weighed) Hendrickx, Kim, Kozareva, Nakov, O S´ eaghdha, Pad ´ o,´ Pennacchiotti, Romano, Szpakowicz Task Overview Data Creation Competition Results and Discussion The Inventory of Semantic Relations (III) 53 | 5. *Entity-Origin(EO)*: An entity is coming or is derived from an origin, e.g., position or material (letters from foreign countries) 54 | 6. *Entity-Destination(ED)*: An entity is moving towards a destination (the boy went to bed) 55 | 7. *Component-Whole(CW)*: An object is a component of a larger whole (my apartment has a large kitchen) 56 | 8. *Member-Collection(MC)*: A member forms a nonfunctional part of a collection (there are many trees in the forest) 57 | 9. *Message-Topic(CT)*: An act of communication, written or spoken, is about a topic (the lecture was about semantics) 58 | 10. *OTHER*: If none of the above nine relations appears to be suitable. 59 | 60 | 61 | ### Distribution for Dataset 62 | * **SemEval-2010 Task #8 Dataset [[Download](https://drive.google.com/file/d/0B_jQiLugGTAkMDQ5ZjZiMTUtMzQ1Yy00YWNmLWJlZDYtOWY1ZDMwY2U4YjFk/view?layout=list&ddrp=1&sort=name&num=50#)]** 63 | 64 | | Relation | Train Data | Test Data | Total Data | 65 | |--------------------|:-------------------:|:-------------------:|:--------------------:| 66 | | Cause-Effect | 1,003 (12.54%) | 328 (12.07%) | 1331 (12.42%) | 67 | | Instrument-Agency | 504 (6.30%) | 156 (5.74%) | 660 (6.16%) | 68 | | Product-Producer | 717 (8.96%) | 231 (8.50%) | 948 (8.85%) | 69 | | Content-Container | 540 (6.75%) | 192 (7.07%) | 732 (6.83%) | 70 | | Entity-Origin | 716 (8.95%) | 258 (9.50%) | 974 (9.09%) | 71 | | Entity-Destination | 845 (10.56%) | 292 (10.75%) | 1137 (10.61%) | 72 | | Component-Whole | 941 (11.76%) | 312 (11.48%) | 1253 (11.69%) | 73 | | Member-Collection | 690 (8.63%) | 233 (8.58%) | 923 (8.61%) | 74 | | Message-Topic | 634 (7.92%) | 261 (9.61%) | 895 (8.35%) | 75 | | Other | 1,410 (17.63%) | 454 (16.71%) | 1864 (17.39%) | 76 | | **Total** | **8,000 (100.00%)** | **2,717 (100.00%)** | **10,717 (100.00%)** | 77 | 78 | 79 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SEMEVAL_TASK8_FULL_RELEASE_README.txt: -------------------------------------------------------------------------------- 1 | Full Dataset Release for SemEval-2 Task #8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals 2 | ==================================================================================================================== 3 | 4 | Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano and Stan Szpakowicz 5 | 6 | The accompanying dataset is released under a Creative Commons Atrribution 3.0 Unported Licence (http://creativecommons.org/licenses/by/3.0/). 7 | 8 | This archive contains -- in four separate directories -- the scorer and format tester for SemEval-2 Task #8, and the training data and test data, including the test keys. 9 | 10 | Released on July 16, 2010. 11 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/README.txt: -------------------------------------------------------------------------------- 1 | Included are two tools for SemEval-2010 Task #8: 2 | Multi-Way Classification of Semantic Relations Between Pairs of Nominals 3 | 4 | The task is described on the following Web address: 5 | http://docs.google.com/View?id=dfvxd49s_36c28v9pmw 6 | 7 | 8 | 1. Output File Format Checker 9 | ----------------------------- 10 | 11 | This is an official output file format checker for SemEval-2010 Task 8. 12 | 13 | Use: 14 | semeval2010_task8_format_checker.pl 15 | 16 | Examples: 17 | semeval2010_task8_format_checker.pl proposed_answer1.txt 18 | semeval2010_task8_format_checker.pl proposed_answer2.txt 19 | semeval2010_task8_format_checker.pl proposed_answer3.txt 20 | semeval2010_task8_format_checker.pl proposed_answer4.txt 21 | semeval2010_task8_format_checker.pl proposed_answer5.txt 22 | 23 | In the examples above, the first three files are OK, while the last one contains four errors. 24 | And answer_key2.txt contains the true labels for the *training* dataset. 25 | 26 | Description: 27 | The scorer takes as input a proposed classification file, 28 | which should contain one prediction per line in the format 29 | " " 30 | with a TAB as a separator, e.g., 31 | 1 Component-Whole(e2,e1) 32 | 2 Other 33 | 3 Instrument-Agency(e2,e1) 34 | ... 35 | The file does not have to be sorted in any way. 36 | Repetitions of IDs are not allowed. 37 | 38 | In case of problems, the checker outputs the problemtic line and its number. 39 | Finally, the total number of problems found is reported 40 | or a message is output saying that the file format is OK. 41 | 42 | Participants are expected to check their output using this checker before submission. 43 | 44 | Last modified: March 10, 2010 45 | 46 | 47 | 48 | 2. Scorer 49 | --------- 50 | 51 | This is the official scorer for SemEval-2010 Task #8. 52 | 53 | Last modified: March 22, 2010 54 | 55 | Current version: 1.2 56 | 57 | Revision history: 58 | - Version 1.2 (fixed a bug in the precision for the scoring of (iii)) 59 | - Version 1.1 (fixed a bug in the calculation of accuracy) 60 | 61 | Use: 62 | semeval2010_task8_scorer-v1.1.pl 63 | 64 | Examples: 65 | semeval2010_task8_scorer-v1.2.pl proposed_answer1.txt answer_key1.txt > result_scores1.txt 66 | semeval2010_task8_scorer-v1.2.pl proposed_answer2.txt answer_key2.txt > result_scores2.txt 67 | semeval2010_task8_scorer-v1.2.pl proposed_answer3.txt answer_key3.txt > result_scores3.txt 68 | semeval2010_task8_scorer-v1.2.pl proposed_answer5.txt answer_key5.txt > result_scores5.txt 69 | 70 | Description: 71 | The scorer takes as input a proposed classification file and an answer key file. 72 | Both files should contain one prediction per line in the format " " 73 | with a TAB as a separator, e.g., 74 | 1 Component-Whole(e2,e1) 75 | 2 Other 76 | 3 Instrument-Agency(e2,e1) 77 | ... 78 | The files do not have to be sorted in any way and the first file can have predictions 79 | for a subset of the IDs in the second file only, e.g., because hard examples have been skipped. 80 | Repetitions of IDs are not allowed in either of the files. 81 | 82 | The scorer calculates and outputs the following statistics: 83 | (1) confusion matrix, which shows 84 | - the sums for each row/column: -SUM- 85 | - the number of skipped examples: skip 86 | - the number of examples with correct relation, but wrong directionality: xDIRx 87 | - the number of examples in the answer key file: ACTUAL ( = -SUM- + skip + xDIRx ) 88 | (2) accuracy and coverage 89 | (3) precision (P), recall (R), and F1-score for each relation 90 | (4) micro-averaged P, R, F1, where the calculations ignore the Other category. 91 | (5) macro-averaged P, R, F1, where the calculations ignore the Other category. 92 | 93 | Note that in scores (4) and (5), skipped examples are equivalent to those classified as Other. 94 | So are examples classified as relations that do not exist in the key file (which is probably not optimal). 95 | 96 | The scoring is done three times: 97 | (i) as a (2*9+1)-way classification 98 | (ii) as a (9+1)-way classification, with directionality ignored 99 | (iii) as a (9+1)-way classification, with directionality taken into account. 100 | 101 | The official score is the macro-averaged F1-score for (iii). 102 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/answer_key1.txt: -------------------------------------------------------------------------------- 1 | 1 Component-Whole(e2,e1) 2 | 2 Other 3 | 3 Instrument-Agency(e2,e1) 4 | 4 Other 5 | 5 Member-Collection(e1,e2) 6 | 6 Other 7 | 7 Cause-Effect(e2,e1) 8 | 8 Entity-Destination(e1,e2) 9 | 9 Content-Container(e1,e2) 10 | 10 Entity-Destination(e1,e2) 11 | 11 Member-Collection(e1,e2) 12 | 12 Other 13 | 13 Message-Topic(e1,e2) 14 | 14 Cause-Effect(e2,e1) 15 | 15 Instrument-Agency(e2,e1) 16 | 16 Message-Topic(e1,e2) 17 | 17 Instrument-Agency(e2,e1) 18 | 18 Product-Producer(e2,e1) 19 | 19 Component-Whole(e2,e1) 20 | 20 Member-Collection(e2,e1) 21 | 21 Entity-Origin(e1,e2) 22 | 22 Member-Collection(e2,e1) 23 | 23 Cause-Effect(e1,e2) 24 | 24 Other 25 | 25 Member-Collection(e2,e1) 26 | 26 Other 27 | 27 Cause-Effect(e1,e2) 28 | 28 Message-Topic(e1,e2) 29 | 29 Message-Topic(e1,e2) 30 | 30 Component-Whole(e1,e2) 31 | 31 Message-Topic(e2,e1) 32 | 32 Cause-Effect(e2,e1) 33 | 33 Product-Producer(e1,e2) 34 | 34 Entity-Destination(e1,e2) 35 | 35 Component-Whole(e1,e2) 36 | 36 Entity-Origin(e1,e2) 37 | 37 Other 38 | 38 Component-Whole(e2,e1) 39 | 39 Cause-Effect(e1,e2) 40 | 40 Instrument-Agency(e2,e1) 41 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/answer_key3.txt: -------------------------------------------------------------------------------- 1 | 1 Component-Whole(e2,e1) 2 | 2 Other 3 | 3 Instrument-Agency(e2,e1) 4 | 4 Other 5 | 5 Member-Collection(e1,e2) 6 | 6 Other 7 | 7 Cause-Effect(e2,e1) 8 | 8 Entity-Destination(e1,e2) 9 | 9 Content-Container(e1,e2) 10 | 10 Entity-Destination(e1,e2) 11 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/answer_key5.txt: -------------------------------------------------------------------------------- 1 | 7 Cause-Effect(e2,e1) 2 | 14 Cause-Effect(e2,e1) 3 | 23 Cause-Effect(e1,e2) 4 | 27 Cause-Effect(e1,e2) 5 | 32 Cause-Effect(e2,e1) 6 | 39 Cause-Effect(e1,e2) 7 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/proposed_answer1.txt: -------------------------------------------------------------------------------- 1 | 10 Entity-Destination(e1,e2) 2 | 11 Member-Collection(e1,e2) 3 | 12 Other 4 | 13 Message-Topic(e1,e2) 5 | 14 Cause-Effect(e2,e1) 6 | 15 Instrument-Agency(e2,e1) 7 | 16 Message-Topic(e1,e2) 8 | 20 Member-Collection(e2,e1) 9 | 1 Other 10 | 2 Instrument-Agency(e2,e1) 11 | 3 Other 12 | 4 Other 13 | 5 Other 14 | 6 Other 15 | 7 Other 16 | 8 Entity-Destination(e1,e2) 17 | 9 Content-Container(e1,e2) 18 | 21 Entity-Origin(e1,e2) 19 | 24 Member-Collection(e2,e1) 20 | 17 Instrument-Agency(e2,e1) 21 | 18 Product-Producer(e2,e1) 22 | 19 Component-Whole(e1,e2) 23 | 23 Cause-Effect(e1,e2) 24 | 22 Other 25 | 25 Member-Collection(e2,e1) 26 | 26 Other 27 | 27 Cause-Effect(e2,e1) 28 | 28 Message-Topic(e2,e1) 29 | 29 Message-Topic(e1,e2) 30 | 30 Component-Whole(e1,e2) 31 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/proposed_answer3.txt: -------------------------------------------------------------------------------- 1 | 10 Entity-Destination(e1,e2) 2 | 6 Message-Topic(e1,e2) 3 | 7 Cause-Effect(e1,e2) 4 | 1 Other 5 | 5 Instrument-Agency(e2,e1) 6 | 3 Other 7 | 2 Cause-Effect(e1,e2) 8 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/proposed_answer4.txt: -------------------------------------------------------------------------------- 1 | 10 Entity-Destination(e1,e2) 2 | 6 Message-Topc(e1,e2) 3 | 7 Cause-Effect(e1, e2) 4 | 1 Other 5 | 5 Instrument-Agency (e2,e1) 6 | 3 Other 7 | 1 Other 8 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/proposed_answer5.txt: -------------------------------------------------------------------------------- 1 | 7 Other 2 | 14 Cause-Effect(e2,e1) 3 | 23 Cause-Effect(e1,e2) 4 | 27 Cause-Effect(e2,e1) 5 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/result_scores1.txt: -------------------------------------------------------------------------------- 1 | <<< (2*9+1)-WAY EVALUATION (USING DIRECTIONALITY)>>>: 2 | 3 | Confusion matrix: 4 | C-E1 C-E2 C-W1 C-W2 C-C1 E-D1 E-O1 I-A2 M-C1 M-C2 M-T1 M-T2 P-P1 P-P2 _O_ <-- classified as 5 | +---------------------------------------------------------------------------+ -SUM- skip ACTUAL 6 | C-E1 | 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 | 2 1 3 7 | C-E2 | 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 | 2 1 3 8 | C-W1 | 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 1 1 2 9 | C-W2 | 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 | 2 1 3 10 | C-C1 | 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 | 1 0 1 11 | E-D1 | 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 | 2 1 3 12 | E-O1 | 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 | 1 1 2 13 | I-A2 | 0 0 0 0 0 0 0 2 0 0 0 0 0 0 1 | 3 1 4 14 | M-C1 | 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 | 2 0 2 15 | M-C2 | 0 0 0 0 0 0 0 0 0 2 0 0 0 0 1 | 3 0 3 16 | M-T1 | 0 0 0 0 0 0 0 0 0 0 3 1 0 0 0 | 4 0 4 17 | M-T2 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | 0 1 1 18 | P-P1 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | 0 1 1 19 | P-P2 | 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 | 1 0 1 20 | _O_ | 0 0 0 0 0 0 0 1 0 1 0 0 0 0 4 | 6 1 7 21 | +---------------------------------------------------------------------------+ 22 | -SUM- 1 2 2 0 1 2 1 3 1 3 3 1 0 1 9 30 10 40 23 | 24 | Coverage = 30/40 = 75.00% 25 | Accuracy (calculated for the above confusion matrix) = 20/30 = 66.67% 26 | Accuracy (considering all skipped examples as Wrong) = 20/40 = 50.00% 27 | Accuracy (considering all skipped examples as Other) = 21/40 = 52.50% 28 | 29 | Results for the individual relations: 30 | Cause-Effect(e1,e2) : P = 1/ 1 = 100.00% R = 1/ 3 = 33.33% F1 = 50.00% 31 | Cause-Effect(e2,e1) : P = 1/ 2 = 50.00% R = 1/ 3 = 33.33% F1 = 40.00% 32 | Component-Whole(e1,e2) : P = 1/ 2 = 50.00% R = 1/ 2 = 50.00% F1 = 50.00% 33 | Component-Whole(e2,e1) : P = 0/ 0 = 0.00% R = 0/ 3 = 0.00% F1 = 0.00% 34 | Content-Container(e1,e2) : P = 1/ 1 = 100.00% R = 1/ 1 = 100.00% F1 = 100.00% 35 | Entity-Destination(e1,e2) : P = 2/ 2 = 100.00% R = 2/ 3 = 66.67% F1 = 80.00% 36 | Entity-Origin(e1,e2) : P = 1/ 1 = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 37 | Instrument-Agency(e2,e1) : P = 2/ 3 = 66.67% R = 2/ 4 = 50.00% F1 = 57.14% 38 | Member-Collection(e1,e2) : P = 1/ 1 = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 39 | Member-Collection(e2,e1) : P = 2/ 3 = 66.67% R = 2/ 3 = 66.67% F1 = 66.67% 40 | Message-Topic(e1,e2) : P = 3/ 3 = 100.00% R = 3/ 4 = 75.00% F1 = 85.71% 41 | Message-Topic(e2,e1) : P = 0/ 1 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 42 | Product-Producer(e1,e2) : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 43 | Product-Producer(e2,e1) : P = 1/ 1 = 100.00% R = 1/ 1 = 100.00% F1 = 100.00% 44 | _Other : P = 4/ 9 = 44.44% R = 4/ 7 = 57.14% F1 = 50.00% 45 | 46 | Micro-averaged result (excluding Other): 47 | P = 16/ 21 = 76.19% R = 16/ 33 = 48.48% F1 = 59.26% 48 | 49 | MACRO-averaged result (excluding Other): 50 | P = 66.67% R = 48.21% F1 = 54.49% 51 | 52 | 53 | 54 | <<< (9+1)-WAY EVALUATION IGNORING DIRECTIONALITY >>>: 55 | 56 | Confusion matrix: 57 | C-E C-W C-C E-D E-O I-A M-C M-T P-P _O_ <-- classified as 58 | +--------------------------------------------------+ -SUM- skip ACTUAL 59 | C-E | 3 0 0 0 0 0 0 0 0 1 | 4 2 6 60 | C-W | 0 2 0 0 0 0 0 0 0 1 | 3 2 5 61 | C-C | 0 0 1 0 0 0 0 0 0 0 | 1 0 1 62 | E-D | 0 0 0 2 0 0 0 0 0 0 | 2 1 3 63 | E-O | 0 0 0 0 1 0 0 0 0 0 | 1 1 2 64 | I-A | 0 0 0 0 0 2 0 0 0 1 | 3 1 4 65 | M-C | 0 0 0 0 0 0 3 0 0 2 | 5 0 5 66 | M-T | 0 0 0 0 0 0 0 4 0 0 | 4 1 5 67 | P-P | 0 0 0 0 0 0 0 0 1 0 | 1 1 2 68 | _O_ | 0 0 0 0 0 1 1 0 0 4 | 6 1 7 69 | +--------------------------------------------------+ 70 | -SUM- 3 2 1 2 1 3 4 4 1 9 30 10 40 71 | 72 | Coverage = 30/40 = 75.00% 73 | Accuracy (calculated for the above confusion matrix) = 23/30 = 76.67% 74 | Accuracy (considering all skipped examples as Wrong) = 23/40 = 57.50% 75 | Accuracy (considering all skipped examples as Other) = 24/40 = 60.00% 76 | 77 | Results for the individual relations: 78 | Cause-Effect : P = 3/ 3 = 100.00% R = 3/ 6 = 50.00% F1 = 66.67% 79 | Component-Whole : P = 2/ 2 = 100.00% R = 2/ 5 = 40.00% F1 = 57.14% 80 | Content-Container : P = 1/ 1 = 100.00% R = 1/ 1 = 100.00% F1 = 100.00% 81 | Entity-Destination : P = 2/ 2 = 100.00% R = 2/ 3 = 66.67% F1 = 80.00% 82 | Entity-Origin : P = 1/ 1 = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 83 | Instrument-Agency : P = 2/ 3 = 66.67% R = 2/ 4 = 50.00% F1 = 57.14% 84 | Member-Collection : P = 3/ 4 = 75.00% R = 3/ 5 = 60.00% F1 = 66.67% 85 | Message-Topic : P = 4/ 4 = 100.00% R = 4/ 5 = 80.00% F1 = 88.89% 86 | Product-Producer : P = 1/ 1 = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 87 | _Other : P = 4/ 9 = 44.44% R = 4/ 7 = 57.14% F1 = 50.00% 88 | 89 | Micro-averaged result (excluding Other): 90 | P = 19/ 21 = 90.48% R = 19/ 33 = 57.58% F1 = 70.37% 91 | 92 | MACRO-averaged result (excluding Other): 93 | P = 93.52% R = 60.74% F1 = 72.20% 94 | 95 | 96 | 97 | <<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>: 98 | 99 | Confusion matrix: 100 | C-E C-W C-C E-D E-O I-A M-C M-T P-P _O_ <-- classified as 101 | +--------------------------------------------------+ -SUM- xDIRx skip ACTUAL 102 | C-E | 2 0 0 0 0 0 0 0 0 1 | 3 1 2 6 103 | C-W | 0 1 0 0 0 0 0 0 0 1 | 2 1 2 5 104 | C-C | 0 0 1 0 0 0 0 0 0 0 | 1 0 0 1 105 | E-D | 0 0 0 2 0 0 0 0 0 0 | 2 0 1 3 106 | E-O | 0 0 0 0 1 0 0 0 0 0 | 1 0 1 2 107 | I-A | 0 0 0 0 0 2 0 0 0 1 | 3 0 1 4 108 | M-C | 0 0 0 0 0 0 3 0 0 2 | 5 0 0 5 109 | M-T | 0 0 0 0 0 0 0 3 0 0 | 3 1 1 5 110 | P-P | 0 0 0 0 0 0 0 0 1 0 | 1 0 1 2 111 | _O_ | 0 0 0 0 0 1 1 0 0 4 | 6 0 1 7 112 | +--------------------------------------------------+ 113 | -SUM- 2 1 1 2 1 3 4 3 1 9 27 3 10 40 114 | 115 | Coverage = 30/40 = 75.00% 116 | Accuracy (calculated for the above confusion matrix) = 20/30 = 66.67% 117 | Accuracy (considering all skipped examples as Wrong) = 20/40 = 50.00% 118 | Accuracy (considering all skipped examples as Other) = 21/40 = 52.50% 119 | 120 | Results for the individual relations: 121 | Cause-Effect : P = 2/( 2 + 1) = 66.67% R = 2/ 6 = 33.33% F1 = 44.44% 122 | Component-Whole : P = 1/( 1 + 1) = 50.00% R = 1/ 5 = 20.00% F1 = 28.57% 123 | Content-Container : P = 1/( 1 + 0) = 100.00% R = 1/ 1 = 100.00% F1 = 100.00% 124 | Entity-Destination : P = 2/( 2 + 0) = 100.00% R = 2/ 3 = 66.67% F1 = 80.00% 125 | Entity-Origin : P = 1/( 1 + 0) = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 126 | Instrument-Agency : P = 2/( 3 + 0) = 66.67% R = 2/ 4 = 50.00% F1 = 57.14% 127 | Member-Collection : P = 3/( 4 + 0) = 75.00% R = 3/ 5 = 60.00% F1 = 66.67% 128 | Message-Topic : P = 3/( 3 + 1) = 75.00% R = 3/ 5 = 60.00% F1 = 66.67% 129 | Product-Producer : P = 1/( 1 + 0) = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 130 | _Other : P = 4/( 9 + 0) = 44.44% R = 4/ 7 = 57.14% F1 = 50.00% 131 | 132 | Micro-averaged result (excluding Other): 133 | P = 16/ 21 = 76.19% R = 16/ 33 = 48.48% F1 = 59.26% 134 | 135 | MACRO-averaged result (excluding Other): 136 | P = 81.48% R = 54.44% F1 = 64.09% 137 | 138 | 139 | 140 | <<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 64.09% >>> 141 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/result_scores2.txt: -------------------------------------------------------------------------------- 1 | <<< (2*9+1)-WAY EVALUATION (USING DIRECTIONALITY)>>>: 2 | 3 | Confusion matrix: 4 | C-E1 C-E2 C-W1 C-W2 C-C1 C-C2 E-D1 E-D2 E-O1 E-O2 I-A1 I-A2 M-C1 M-C2 M-T1 M-T2 P-P1 P-P2 _O_ <-- classified as 5 | +-----------------------------------------------------------------------------------------------+ -SUM- skip ACTUAL 6 | C-E1 | 14 26 23 13 17 6 36 0 30 9 5 23 2 33 17 8 13 11 55 | 341 3 344 7 | C-E2 | 28 62 23 39 28 17 63 0 40 9 7 36 6 45 36 12 28 37 136 | 652 7 659 8 | C-W1 | 29 42 35 36 29 7 47 0 34 11 4 23 3 31 19 8 18 28 65 | 469 1 470 9 | C-W2 | 18 44 28 35 14 10 48 0 39 6 8 18 3 27 33 12 21 15 92 | 471 0 471 10 | C-C1 | 18 31 15 26 21 4 30 0 31 6 3 18 5 29 16 3 22 18 77 | 373 1 374 11 | C-C2 | 7 15 10 12 8 2 17 0 12 3 2 10 1 11 6 2 5 10 30 | 163 3 166 12 | E-D1 | 26 59 57 40 49 24 86 0 73 14 8 43 12 74 64 16 34 39 125 | 843 1 844 13 | E-D2 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 | 1 0 1 14 | E-O1 | 19 47 34 40 34 6 72 0 39 10 2 30 3 40 30 14 32 22 89 | 563 5 568 15 | E-O2 | 5 10 8 6 5 5 15 0 9 5 4 8 3 8 7 4 4 10 30 | 146 2 148 16 | I-A1 | 4 2 4 5 2 3 15 0 10 1 1 6 0 7 5 2 6 5 18 | 96 1 97 17 | I-A2 | 20 30 25 24 12 14 43 0 30 2 6 17 3 30 29 7 16 19 77 | 404 3 407 18 | M-C1 | 4 9 4 3 3 2 8 0 5 2 2 5 0 8 4 0 2 3 14 | 78 0 78 19 | M-C2 | 27 42 36 38 33 11 73 0 43 11 10 28 6 50 47 10 21 26 100 | 612 0 612 20 | M-T1 | 23 42 27 27 20 7 66 0 33 9 2 24 8 39 35 9 16 29 74 | 490 0 490 21 | M-T2 | 4 11 13 11 7 3 16 0 7 2 1 3 1 16 16 2 2 4 25 | 144 0 144 22 | P-P1 | 10 21 20 19 17 6 28 0 20 9 5 14 2 28 16 8 13 17 65 | 318 5 323 23 | P-P2 | 22 31 22 18 17 6 35 0 28 7 3 17 7 36 22 11 17 21 67 | 387 7 394 24 | _O_ | 62 130 83 74 57 32 145 1 81 27 23 84 13 98 88 16 52 75 255 | 1396 14 1410 25 | +-----------------------------------------------------------------------------------------------+ 26 | -SUM- 340 654 467 466 373 165 843 1 564 143 96 407 78 610 490 144 322 389 1395 7947 53 8000 27 | 28 | Coverage = 7947/8000 = 99.34% 29 | Accuracy (calculated for the above confusion matrix) = 693/7947 = 8.72% 30 | Accuracy (considering all skipped examples as Wrong) = 693/8000 = 8.66% 31 | Accuracy (considering all skipped examples as Other) = 707/8000 = 8.84% 32 | 33 | Results for the individual relations: 34 | Cause-Effect(e1,e2) : P = 14/ 340 = 4.12% R = 14/ 344 = 4.07% F1 = 4.09% 35 | Cause-Effect(e2,e1) : P = 62/ 654 = 9.48% R = 62/ 659 = 9.41% F1 = 9.44% 36 | Component-Whole(e1,e2) : P = 35/ 467 = 7.49% R = 35/ 470 = 7.45% F1 = 7.47% 37 | Component-Whole(e2,e1) : P = 35/ 466 = 7.51% R = 35/ 471 = 7.43% F1 = 7.47% 38 | Content-Container(e1,e2) : P = 21/ 373 = 5.63% R = 21/ 374 = 5.61% F1 = 5.62% 39 | Content-Container(e2,e1) : P = 2/ 165 = 1.21% R = 2/ 166 = 1.20% F1 = 1.21% 40 | Entity-Destination(e1,e2) : P = 86/ 843 = 10.20% R = 86/ 844 = 10.19% F1 = 10.20% 41 | Entity-Destination(e2,e1) : P = 0/ 1 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 42 | Entity-Origin(e1,e2) : P = 39/ 564 = 6.91% R = 39/ 568 = 6.87% F1 = 6.89% 43 | Entity-Origin(e2,e1) : P = 5/ 143 = 3.50% R = 5/ 148 = 3.38% F1 = 3.44% 44 | Instrument-Agency(e1,e2) : P = 1/ 96 = 1.04% R = 1/ 97 = 1.03% F1 = 1.04% 45 | Instrument-Agency(e2,e1) : P = 17/ 407 = 4.18% R = 17/ 407 = 4.18% F1 = 4.18% 46 | Member-Collection(e1,e2) : P = 0/ 78 = 0.00% R = 0/ 78 = 0.00% F1 = 0.00% 47 | Member-Collection(e2,e1) : P = 50/ 610 = 8.20% R = 50/ 612 = 8.17% F1 = 8.18% 48 | Message-Topic(e1,e2) : P = 35/ 490 = 7.14% R = 35/ 490 = 7.14% F1 = 7.14% 49 | Message-Topic(e2,e1) : P = 2/ 144 = 1.39% R = 2/ 144 = 1.39% F1 = 1.39% 50 | Product-Producer(e1,e2) : P = 13/ 322 = 4.04% R = 13/ 323 = 4.02% F1 = 4.03% 51 | Product-Producer(e2,e1) : P = 21/ 389 = 5.40% R = 21/ 394 = 5.33% F1 = 5.36% 52 | _Other : P = 255/1395 = 18.28% R = 255/1410 = 18.09% F1 = 18.18% 53 | 54 | Micro-averaged result (excluding Other): 55 | P = 438/6552 = 6.68% R = 438/6590 = 6.65% F1 = 6.67% 56 | 57 | MACRO-averaged result (excluding Other): 58 | P = 4.86% R = 4.83% F1 = 4.84% 59 | 60 | 61 | 62 | <<< (9+1)-WAY EVALUATION IGNORING DIRECTIONALITY >>>: 63 | 64 | Confusion matrix: 65 | C-E C-W C-C E-D E-O I-A M-C M-T P-P _O_ <-- classified as 66 | +--------------------------------------------------+ -SUM- skip ACTUAL 67 | C-E | 130 98 68 99 88 71 86 73 89 191 | 993 10 1003 68 | C-W | 133 134 60 95 90 53 64 72 82 157 | 940 1 941 69 | C-C | 71 63 35 47 52 33 46 27 55 107 | 536 4 540 70 | E-D | 85 97 73 86 87 51 86 80 73 126 | 844 1 845 71 | E-O | 81 88 50 87 63 44 54 55 68 119 | 709 7 716 72 | I-A | 56 58 31 58 43 30 40 43 46 95 | 500 4 504 73 | M-C | 82 81 49 81 61 45 64 61 52 114 | 690 0 690 74 | M-T | 80 78 37 82 51 30 64 62 51 99 | 634 0 634 75 | P-P | 84 79 46 63 64 39 73 57 68 132 | 705 12 717 76 | _O_ | 192 157 89 146 108 107 111 104 127 255 | 1396 14 1410 77 | +--------------------------------------------------+ 78 | -SUM- 994 933 538 844 707 503 688 634 711 1395 7947 53 8000 79 | 80 | Coverage = 7947/8000 = 99.34% 81 | Accuracy (calculated for the above confusion matrix) = 927/7947 = 11.66% 82 | Accuracy (considering all skipped examples as Wrong) = 927/8000 = 11.59% 83 | Accuracy (considering all skipped examples as Other) = 941/8000 = 11.76% 84 | 85 | Results for the individual relations: 86 | Cause-Effect : P = 130/ 994 = 13.08% R = 130/1003 = 12.96% F1 = 13.02% 87 | Component-Whole : P = 134/ 933 = 14.36% R = 134/ 941 = 14.24% F1 = 14.30% 88 | Content-Container : P = 35/ 538 = 6.51% R = 35/ 540 = 6.48% F1 = 6.49% 89 | Entity-Destination : P = 86/ 844 = 10.19% R = 86/ 845 = 10.18% F1 = 10.18% 90 | Entity-Origin : P = 63/ 707 = 8.91% R = 63/ 716 = 8.80% F1 = 8.85% 91 | Instrument-Agency : P = 30/ 503 = 5.96% R = 30/ 504 = 5.95% F1 = 5.96% 92 | Member-Collection : P = 64/ 688 = 9.30% R = 64/ 690 = 9.28% F1 = 9.29% 93 | Message-Topic : P = 62/ 634 = 9.78% R = 62/ 634 = 9.78% F1 = 9.78% 94 | Product-Producer : P = 68/ 711 = 9.56% R = 68/ 717 = 9.48% F1 = 9.52% 95 | _Other : P = 255/1395 = 18.28% R = 255/1410 = 18.09% F1 = 18.18% 96 | 97 | Micro-averaged result (excluding Other): 98 | P = 672/6552 = 10.26% R = 672/6590 = 10.20% F1 = 10.23% 99 | 100 | MACRO-averaged result (excluding Other): 101 | P = 9.74% R = 9.68% F1 = 9.71% 102 | 103 | 104 | 105 | <<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>: 106 | 107 | Confusion matrix: 108 | C-E C-W C-C E-D E-O I-A M-C M-T P-P _O_ <-- classified as 109 | +--------------------------------------------------+ -SUM- xDIRx skip ACTUAL 110 | C-E | 76 98 68 99 88 71 86 73 89 191 | 939 54 10 1003 111 | C-W | 133 70 60 95 90 53 64 72 82 157 | 876 64 1 941 112 | C-C | 71 63 23 47 52 33 46 27 55 107 | 524 12 4 540 113 | E-D | 85 97 73 86 87 51 86 80 73 126 | 844 0 1 845 114 | E-O | 81 88 50 87 44 44 54 55 68 119 | 690 19 7 716 115 | I-A | 56 58 31 58 43 18 40 43 46 95 | 488 12 4 504 116 | M-C | 82 81 49 81 61 45 50 61 52 114 | 676 14 0 690 117 | M-T | 80 78 37 82 51 30 64 37 51 99 | 609 25 0 634 118 | P-P | 84 79 46 63 64 39 73 57 34 132 | 671 34 12 717 119 | _O_ | 192 157 89 146 108 107 111 104 127 255 | 1396 0 14 1410 120 | +--------------------------------------------------+ 121 | -SUM- 940 869 526 844 688 491 674 609 677 1395 7713 234 53 8000 122 | 123 | Coverage = 7947/8000 = 99.34% 124 | Accuracy (calculated for the above confusion matrix) = 693/7947 = 8.72% 125 | Accuracy (considering all skipped examples as Wrong) = 693/8000 = 8.66% 126 | Accuracy (considering all skipped examples as Other) = 707/8000 = 8.84% 127 | 128 | Results for the individual relations: 129 | Cause-Effect : P = 76/( 940 + 54) = 7.65% R = 76/1003 = 7.58% F1 = 7.61% 130 | Component-Whole : P = 70/( 869 + 64) = 7.50% R = 70/ 941 = 7.44% F1 = 7.47% 131 | Content-Container : P = 23/( 526 + 12) = 4.28% R = 23/ 540 = 4.26% F1 = 4.27% 132 | Entity-Destination : P = 86/( 844 + 0) = 10.19% R = 86/ 845 = 10.18% F1 = 10.18% 133 | Entity-Origin : P = 44/( 688 + 19) = 6.22% R = 44/ 716 = 6.15% F1 = 6.18% 134 | Instrument-Agency : P = 18/( 491 + 12) = 3.58% R = 18/ 504 = 3.57% F1 = 3.57% 135 | Member-Collection : P = 50/( 674 + 14) = 7.27% R = 50/ 690 = 7.25% F1 = 7.26% 136 | Message-Topic : P = 37/( 609 + 25) = 5.84% R = 37/ 634 = 5.84% F1 = 5.84% 137 | Product-Producer : P = 34/( 677 + 34) = 4.78% R = 34/ 717 = 4.74% F1 = 4.76% 138 | _Other : P = 255/(1395 + 0) = 18.28% R = 255/1410 = 18.09% F1 = 18.18% 139 | 140 | Micro-averaged result (excluding Other): 141 | P = 438/6552 = 6.68% R = 438/6590 = 6.65% F1 = 6.67% 142 | 143 | MACRO-averaged result (excluding Other): 144 | P = 6.37% R = 6.33% F1 = 6.35% 145 | 146 | 147 | 148 | <<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 6.35% >>> 149 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/result_scores3.txt: -------------------------------------------------------------------------------- 1 | !!!WARNING!!! The proposed file contains 2 label(s) of type 'Cause-Effect(e1,e2)', which is NOT present in the key file. 2 | 3 | !!!WARNING!!! The proposed file contains 1 label(s) of type 'Message-Topic(e1,e2)', which is NOT present in the key file. 4 | 5 | <<< (2*9+1)-WAY EVALUATION (USING DIRECTIONALITY)>>>: 6 | 7 | Confusion matrix: 8 | C-E2 C-W2 C-C1 E-D1 I-A2 M-C1 _O_ *CE1 *MT1 <-- classified as 9 | +---------------------------------------------+ -SUM- skip ACTUAL 10 | C-E2 | 0 0 0 0 0 0 0 1 0 | 1 0 1 11 | C-W2 | 0 0 0 0 0 0 1 0 0 | 1 0 1 12 | C-C1 | 0 0 0 0 0 0 0 0 0 | 0 1 1 13 | E-D1 | 0 0 0 1 0 0 0 0 0 | 1 1 2 14 | I-A2 | 0 0 0 0 0 0 1 0 0 | 1 0 1 15 | M-C1 | 0 0 0 0 1 0 0 0 0 | 1 0 1 16 | _O_ | 0 0 0 0 0 0 0 1 1 | 2 1 3 17 | +---------------------------------------------+ 18 | -SUM- 0 0 0 1 1 0 2 2 1 7 3 10 19 | 20 | Coverage = 7/10 = 70.00% 21 | Accuracy (calculated for the above confusion matrix) = 1/7 = 14.29% 22 | Accuracy (considering all skipped examples as Wrong) = 1/10 = 10.00% 23 | Accuracy (considering all skipped examples as Other) = 2/10 = 20.00% 24 | 25 | Results for the individual relations: 26 | Cause-Effect(e2,e1) : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 27 | Component-Whole(e2,e1) : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 28 | Content-Container(e1,e2) : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 29 | Entity-Destination(e1,e2) : P = 1/ 1 = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 30 | Instrument-Agency(e2,e1) : P = 0/ 1 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 31 | Member-Collection(e1,e2) : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 32 | _Other : P = 0/ 2 = 0.00% R = 0/ 3 = 0.00% F1 = 0.00% 33 | 34 | Micro-averaged result (excluding Other): 35 | P = 1/ 2 = 50.00% R = 1/ 7 = 14.29% F1 = 22.22% 36 | 37 | MACRO-averaged result (excluding Other): 38 | P = 16.67% R = 8.33% F1 = 11.11% 39 | 40 | 41 | 42 | <<< (9+1)-WAY EVALUATION IGNORING DIRECTIONALITY >>>: 43 | 44 | Confusion matrix: 45 | C-E C-W C-C E-D I-A M-C _O_ *MT <-- classified as 46 | +----------------------------------------+ -SUM- skip ACTUAL 47 | C-E | 1 0 0 0 0 0 0 0 | 1 0 1 48 | C-W | 0 0 0 0 0 0 1 0 | 1 0 1 49 | C-C | 0 0 0 0 0 0 0 0 | 0 1 1 50 | E-D | 0 0 0 1 0 0 0 0 | 1 1 2 51 | I-A | 0 0 0 0 0 0 1 0 | 1 0 1 52 | M-C | 0 0 0 0 1 0 0 0 | 1 0 1 53 | _O_ | 1 0 0 0 0 0 0 1 | 2 1 3 54 | +----------------------------------------+ 55 | -SUM- 2 0 0 1 1 0 2 1 7 3 10 56 | 57 | Coverage = 7/10 = 70.00% 58 | Accuracy (calculated for the above confusion matrix) = 2/7 = 28.57% 59 | Accuracy (considering all skipped examples as Wrong) = 2/10 = 20.00% 60 | Accuracy (considering all skipped examples as Other) = 3/10 = 30.00% 61 | 62 | Results for the individual relations: 63 | Cause-Effect : P = 1/ 2 = 50.00% R = 1/ 1 = 100.00% F1 = 66.67% 64 | Component-Whole : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 65 | Content-Container : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 66 | Entity-Destination : P = 1/ 1 = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 67 | Instrument-Agency : P = 0/ 1 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 68 | Member-Collection : P = 0/ 0 = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 69 | _Other : P = 0/ 2 = 0.00% R = 0/ 3 = 0.00% F1 = 0.00% 70 | 71 | Micro-averaged result (excluding Other): 72 | P = 2/ 4 = 50.00% R = 2/ 7 = 28.57% F1 = 36.36% 73 | 74 | MACRO-averaged result (excluding Other): 75 | P = 25.00% R = 25.00% F1 = 22.22% 76 | 77 | 78 | 79 | <<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>: 80 | 81 | Confusion matrix: 82 | C-E C-W C-C E-D I-A M-C _O_ *MT <-- classified as 83 | +----------------------------------------+ -SUM- xDIRx skip ACTUAL 84 | C-E | 0 0 0 0 0 0 0 0 | 0 1 0 1 85 | C-W | 0 0 0 0 0 0 1 0 | 1 0 0 1 86 | C-C | 0 0 0 0 0 0 0 0 | 0 0 1 1 87 | E-D | 0 0 0 1 0 0 0 0 | 1 0 1 2 88 | I-A | 0 0 0 0 0 0 1 0 | 1 0 0 1 89 | M-C | 0 0 0 0 1 0 0 0 | 1 0 0 1 90 | _O_ | 1 0 0 0 0 0 0 1 | 2 0 1 3 91 | +----------------------------------------+ 92 | -SUM- 1 0 0 1 1 0 2 1 6 1 3 10 93 | 94 | Coverage = 7/10 = 70.00% 95 | Accuracy (calculated for the above confusion matrix) = 1/7 = 14.29% 96 | Accuracy (considering all skipped examples as Wrong) = 1/10 = 10.00% 97 | Accuracy (considering all skipped examples as Other) = 2/10 = 20.00% 98 | 99 | Results for the individual relations: 100 | Cause-Effect : P = 0/( 1 + 1) = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 101 | Component-Whole : P = 0/( 0 + 0) = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 102 | Content-Container : P = 0/( 0 + 0) = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 103 | Entity-Destination : P = 1/( 1 + 0) = 100.00% R = 1/ 2 = 50.00% F1 = 66.67% 104 | Instrument-Agency : P = 0/( 1 + 0) = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 105 | Member-Collection : P = 0/( 0 + 0) = 0.00% R = 0/ 1 = 0.00% F1 = 0.00% 106 | _Other : P = 0/( 2 + 0) = 0.00% R = 0/ 3 = 0.00% F1 = 0.00% 107 | 108 | Micro-averaged result (excluding Other): 109 | P = 1/ 4 = 25.00% R = 1/ 7 = 14.29% F1 = 18.18% 110 | 111 | MACRO-averaged result (excluding Other): 112 | P = 16.67% R = 8.33% F1 = 11.11% 113 | 114 | 115 | 116 | <<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 11.11% >>> 117 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/result_scores5.txt: -------------------------------------------------------------------------------- 1 | !!!WARNING!!! The proposed file contains 1 label(s) of type '_Other', which is NOT present in the key file. 2 | 3 | <<< (2*9+1)-WAY EVALUATION (USING DIRECTIONALITY)>>>: 4 | 5 | Confusion matrix: 6 | C-E1 C-E2 _O_ <-- classified as 7 | +---------------+ -SUM- skip ACTUAL 8 | C-E1 | 1 1 0 | 2 1 3 9 | C-E2 | 0 1 1 | 2 1 3 10 | +---------------+ 11 | -SUM- 1 2 1 4 2 6 12 | 13 | Coverage = 4/6 = 66.67% 14 | Accuracy (calculated for the above confusion matrix) = 2/4 = 50.00% 15 | Accuracy (considering all skipped examples as Wrong) = 2/6 = 33.33% 16 | Accuracy (considering all skipped examples as Other) = 2/6 = 33.33% 17 | 18 | Results for the individual relations: 19 | Cause-Effect(e1,e2) : P = 1/ 1 = 100.00% R = 1/ 3 = 33.33% F1 = 50.00% 20 | Cause-Effect(e2,e1) : P = 1/ 2 = 50.00% R = 1/ 3 = 33.33% F1 = 40.00% 21 | 22 | Micro-averaged result (excluding Other): 23 | P = 2/ 3 = 66.67% R = 2/ 6 = 33.33% F1 = 44.44% 24 | 25 | MACRO-averaged result (excluding Other): 26 | P = 75.00% R = 33.33% F1 = 45.00% 27 | 28 | 29 | 30 | <<< (9+1)-WAY EVALUATION IGNORING DIRECTIONALITY >>>: 31 | 32 | Confusion matrix: 33 | C-E _O_ <-- classified as 34 | +----------+ -SUM- skip ACTUAL 35 | C-E | 3 1 | 4 2 6 36 | +----------+ 37 | -SUM- 3 1 4 2 6 38 | 39 | Coverage = 4/6 = 66.67% 40 | Accuracy (calculated for the above confusion matrix) = 3/4 = 75.00% 41 | Accuracy (considering all skipped examples as Wrong) = 3/6 = 50.00% 42 | Accuracy (considering all skipped examples as Other) = 3/6 = 50.00% 43 | 44 | Results for the individual relations: 45 | Cause-Effect : P = 3/ 3 = 100.00% R = 3/ 6 = 50.00% F1 = 66.67% 46 | 47 | Micro-averaged result (excluding Other): 48 | P = 3/ 3 = 100.00% R = 3/ 6 = 50.00% F1 = 66.67% 49 | 50 | MACRO-averaged result (excluding Other): 51 | P = 100.00% R = 50.00% F1 = 66.67% 52 | 53 | 54 | 55 | <<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>: 56 | 57 | Confusion matrix: 58 | C-E _O_ <-- classified as 59 | +----------+ -SUM- xDIRx skip ACTUAL 60 | C-E | 2 1 | 3 1 2 6 61 | +----------+ 62 | -SUM- 2 1 3 1 2 6 63 | 64 | Coverage = 4/6 = 66.67% 65 | Accuracy (calculated for the above confusion matrix) = 2/4 = 50.00% 66 | Accuracy (considering all skipped examples as Wrong) = 2/6 = 33.33% 67 | Accuracy (considering all skipped examples as Other) = 2/6 = 33.33% 68 | 69 | Results for the individual relations: 70 | Cause-Effect : P = 2/( 2 + 1) = 66.67% R = 2/ 6 = 33.33% F1 = 44.44% 71 | 72 | Micro-averaged result (excluding Other): 73 | P = 2/ 3 = 66.67% R = 2/ 6 = 33.33% F1 = 44.44% 74 | 75 | MACRO-averaged result (excluding Other): 76 | P = 66.67% R = 33.33% F1 = 44.44% 77 | 78 | 79 | 80 | <<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 44.44% >>> 81 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/semeval2010_task8_format_checker.pl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl -w 2 | # 3 | # 4 | # Author: Preslav Nakov 5 | # nakov@comp.nus.edu.sg 6 | # National University of Singapore 7 | # 8 | # WHAT: This is an official output file format checker for SemEval-2010 Task #8. 9 | # 10 | # Use: 11 | # semeval2010_task8_format_checker.pl 12 | # 13 | # Examples: 14 | # semeval2010_task8_format_checker.pl proposed_answer1.txt 15 | # semeval2010_task8_format_checker.pl proposed_answer2.txt 16 | # semeval2010_task8_format_checker.pl proposed_answer3.txt 17 | # semeval2010_task8_format_checker.pl proposed_answer4.txt 18 | # 19 | # In the examples above, the first three files are OK, while the last one contains four errors. 20 | # And answer_key2.txt contains the true labels for the *training* dataset. 21 | # 22 | # Description: 23 | # The scorer takes as input a proposed classification file, 24 | # which should contain one prediction per line in the format " " 25 | # with a TAB as a separator, e.g., 26 | # 1 Component-Whole(e2,e1) 27 | # 2 Other 28 | # 3 Instrument-Agency(e2,e1) 29 | # ... 30 | # The file does not have to be sorted in any way. 31 | # Repetitions of IDs are not allowed. 32 | # 33 | # In case of problems, the checker outputs the problemtic line and its number. 34 | # Finally, the total number of problems found is reported 35 | # or a message is output saying that the file format is OK. 36 | # 37 | # Participants are expected to check their output using this checker before submission. 38 | # 39 | # Last modified: March 10, 2010 40 | # 41 | # 42 | 43 | use strict; 44 | 45 | ############### 46 | ### I/O ### 47 | ############### 48 | 49 | if ($#ARGV != 0) { 50 | die "Usage:\nsemeval2010_task8_format_checker.pl \n"; 51 | } 52 | 53 | my $INPUT_FILE_NAME = $ARGV[0]; 54 | 55 | ################ 56 | ### MAIN ### 57 | ################ 58 | my %ids = (); 59 | 60 | my $errCnt = 0; 61 | open(INPUT, $INPUT_FILE_NAME) or die "Failed to open $INPUT_FILE_NAME for text reading.\n"; 62 | for (my $lineNo = 1; ; $lineNo++) { 63 | my ($id, $label) = &getIDandLabel($_); 64 | if ($id < 0) { 65 | s/[\n\r]*$//; 66 | print "Bad file format on line $lineNo: '$_'\n"; 67 | $errCnt++; 68 | } 69 | elsif (defined $ids{$id}) { 70 | s/[\n\r]*$//; 71 | print "Bad file format on line $lineNo (ID $id is already defined): '$_'\n"; 72 | $errCnt++; 73 | } 74 | $ids{$id}++; 75 | } 76 | close(INPUT) or die "Failed to close $INPUT_FILE_NAME.\n"; 77 | 78 | if (0 == $errCnt) { 79 | print "\n<<< The file format is OK.\n"; 80 | } 81 | else { 82 | print "\n<<< The format is INCORRECT: $errCnt problematic line(s) found!\n"; 83 | } 84 | 85 | 86 | ################ 87 | ### SUBS ### 88 | ################ 89 | 90 | sub getIDandLabel() { 91 | my $line = shift; 92 | 93 | return (-1,()) if ($line !~ /^([0-9]+)\t([^\r]+)\r?\n$/); 94 | my ($id, $label) = ($1, $2); 95 | 96 | return ($id, '_Other') if ($label eq 'Other'); 97 | 98 | return ($id, $label) 99 | if (($label eq 'Cause-Effect(e1,e2)') || ($label eq 'Cause-Effect(e2,e1)') || 100 | ($label eq 'Component-Whole(e1,e2)') || ($label eq 'Component-Whole(e2,e1)') || 101 | ($label eq 'Content-Container(e1,e2)') || ($label eq 'Content-Container(e2,e1)') || 102 | ($label eq 'Entity-Destination(e1,e2)') || ($label eq 'Entity-Destination(e2,e1)') || 103 | ($label eq 'Entity-Origin(e1,e2)') || ($label eq 'Entity-Origin(e2,e1)') || 104 | ($label eq 'Instrument-Agency(e1,e2)') || ($label eq 'Instrument-Agency(e2,e1)') || 105 | ($label eq 'Member-Collection(e1,e2)') || ($label eq 'Member-Collection(e2,e1)') || 106 | ($label eq 'Message-Topic(e1,e2)') || ($label eq 'Message-Topic(e2,e1)') || 107 | ($label eq 'Product-Producer(e1,e2)') || ($label eq 'Product-Producer(e2,e1)')); 108 | 109 | return (-1, ()); 110 | } 111 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_scorer-v1.2/semeval2010_task8_scorer-v1.2.pl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl -w 2 | # 3 | # 4 | # Author: Preslav Nakov 5 | # nakov@comp.nus.edu.sg 6 | # National University of Singapore 7 | # 8 | # WHAT: This is the official scorer for SemEval-2010 Task #8. 9 | # 10 | # 11 | # Last modified: March 22, 2010 12 | # 13 | # Current version: 1.2 14 | # 15 | # Revision history: 16 | # - Version 1.2 (fixed a bug in the precision for the scoring of (iii)) 17 | # - Version 1.1 (fixed a bug in the calculation of accuracy) 18 | # 19 | # 20 | # Use: 21 | # semeval2010_task8_scorer-v1.1.pl 22 | # 23 | # Example2: 24 | # semeval2010_task8_scorer-v1.1.pl proposed_answer1.txt answer_key1.txt > result_scores1.txt 25 | # semeval2010_task8_scorer-v1.1.pl proposed_answer2.txt answer_key2.txt > result_scores2.txt 26 | # semeval2010_task8_scorer-v1.1.pl proposed_answer3.txt answer_key3.txt > result_scores3.txt 27 | # 28 | # Description: 29 | # The scorer takes as input a proposed classification file and an answer key file. 30 | # Both files should contain one prediction per line in the format " " 31 | # with a TAB as a separator, e.g., 32 | # 1 Component-Whole(e2,e1) 33 | # 2 Other 34 | # 3 Instrument-Agency(e2,e1) 35 | # ... 36 | # The files do not have to be sorted in any way and the first file can have predictions 37 | # for a subset of the IDs in the second file only, e.g., because hard examples have been skipped. 38 | # Repetitions of IDs are not allowed in either of the files. 39 | # 40 | # The scorer calculates and outputs the following statistics: 41 | # (1) confusion matrix, which shows 42 | # - the sums for each row/column: -SUM- 43 | # - the number of skipped examples: skip 44 | # - the number of examples with correct relation, but wrong directionality: xDIRx 45 | # - the number of examples in the answer key file: ACTUAL ( = -SUM- + skip + xDIRx ) 46 | # (2) accuracy and coverage 47 | # (3) precision (P), recall (R), and F1-score for each relation 48 | # (4) micro-averaged P, R, F1, where the calculations ignore the Other category. 49 | # (5) macro-averaged P, R, F1, where the calculations ignore the Other category. 50 | # 51 | # Note that in scores (4) and (5), skipped examples are equivalent to those classified as Other. 52 | # So are examples classified as relations that do not exist in the key file (which is probably not optimal). 53 | # 54 | # The scoring is done three times: 55 | # (i) as a (2*9+1)-way classification 56 | # (ii) as a (9+1)-way classification, with directionality ignored 57 | # (iii) as a (9+1)-way classification, with directionality taken into account. 58 | # 59 | # The official score is the macro-averaged F1-score for (iii). 60 | # 61 | 62 | use strict; 63 | 64 | 65 | ############### 66 | ### I/O ### 67 | ############### 68 | 69 | if ($#ARGV != 1) { 70 | die "Usage:\nsemeval2010_task8_scorer.pl \n"; 71 | } 72 | 73 | my $PROPOSED_ANSWERS_FILE_NAME = $ARGV[0]; 74 | my $ANSWER_KEYS_FILE_NAME = $ARGV[1]; 75 | 76 | 77 | ################ 78 | ### MAIN ### 79 | ################ 80 | 81 | my (%confMatrix19way, %confMatrix10wayNoDir, %confMatrix10wayWithDir) = (); 82 | my (%idsProposed, %idsAnswer) = (); 83 | my (%allLabels19waylAnswer, %allLabels10wayAnswer) = (); 84 | my (%allLabels19wayProposed, %allLabels10wayNoDirProposed, %allLabels10wayWithDirProposed) = (); 85 | 86 | ### 1. Read the file contents 87 | my $totalProposed = &readFileIntoHash($PROPOSED_ANSWERS_FILE_NAME, \%idsProposed); 88 | my $totalAnswer = &readFileIntoHash($ANSWER_KEYS_FILE_NAME, \%idsAnswer); 89 | 90 | ### 2. Calculate the confusion matrices 91 | foreach my $id (keys %idsProposed) { 92 | 93 | ### 2.1. Unexpected IDs are not allowed 94 | die "File $PROPOSED_ANSWERS_FILE_NAME contains a bad ID: '$id'" 95 | if (!defined($idsAnswer{$id})); 96 | 97 | ### 2.2. Update the 19-way confusion matrix 98 | my $labelProposed = $idsProposed{$id}; 99 | my $labelAnswer = $idsAnswer{$id}; 100 | $confMatrix19way{$labelProposed}{$labelAnswer}++; 101 | $allLabels19wayProposed{$labelProposed}++; 102 | 103 | ### 2.3. Update the 10-way confusion matrix *without* direction 104 | my $labelProposedNoDir = $labelProposed; 105 | my $labelAnswerNoDir = $labelAnswer; 106 | $labelProposedNoDir =~ s/$e[12],e[12]$[\n\r]*$//; 107 | $labelAnswerNoDir =~ s/$e[12],e[12]$[\n\r]*$//; 108 | $confMatrix10wayNoDir{$labelProposedNoDir}{$labelAnswerNoDir}++; 109 | $allLabels10wayNoDirProposed{$labelProposedNoDir}++; 110 | 111 | ### 2.4. Update the 10-way confusion matrix *with* direction 112 | if ($labelProposed eq $labelAnswer) { ## both relation and direction match 113 | $confMatrix10wayWithDir{$labelProposedNoDir}{$labelAnswerNoDir}++; 114 | $allLabels10wayWithDirProposed{$labelProposedNoDir}++; 115 | } 116 | elsif ($labelProposedNoDir eq $labelAnswerNoDir) { ## the relations match, but the direction is wrong 117 | $confMatrix10wayWithDir{'WRONG_DIR'}{$labelAnswerNoDir}++; 118 | $allLabels10wayWithDirProposed{'WRONG_DIR'}++; 119 | } 120 | else { ### Wrong relation 121 | $confMatrix10wayWithDir{$labelProposedNoDir}{$labelAnswerNoDir}++; 122 | $allLabels10wayWithDirProposed{$labelProposedNoDir}++; 123 | } 124 | } 125 | 126 | ### 3. Calculate the ground truth distributions 127 | foreach my $id (keys %idsAnswer) { 128 | 129 | ### 3.1. Update the 19-way answer distribution 130 | my $labelAnswer = $idsAnswer{$id}; 131 | $allLabels19waylAnswer{$labelAnswer}++; 132 | 133 | ### 3.2. Update the 10-way answer distribution 134 | my $labelAnswerNoDir = $labelAnswer; 135 | $labelAnswerNoDir =~ s/$e[12],e[12]$[\n\r]*$//; 136 | $allLabels10wayAnswer{$labelAnswerNoDir}++; 137 | } 138 | 139 | ### 4. Check for proposed classes that are not contained in the answer key file: this may happen in cross-validation 140 | foreach my $labelProposed (sort keys %allLabels19wayProposed) { 141 | if (!defined($allLabels19waylAnswer{$labelProposed})) { 142 | print "!!!WARNING!!! The proposed file contains $allLabels19wayProposed{$labelProposed} label(s) of type '$labelProposed', which is NOT present in the key file.\n\n"; 143 | } 144 | } 145 | 146 | ### 4. 19-way evaluation with directionality 147 | print "<<< (2*9+1)-WAY EVALUATION (USING DIRECTIONALITY)>>>:\n\n"; 148 | &evaluate(\%confMatrix19way, \%allLabels19wayProposed, \%allLabels19waylAnswer, $totalProposed, $totalAnswer, 0); 149 | 150 | ### 5. Evaluate without directionality 151 | print "<<< (9+1)-WAY EVALUATION IGNORING DIRECTIONALITY >>>:\n\n"; 152 | &evaluate(\%confMatrix10wayNoDir, \%allLabels10wayNoDirProposed, \%allLabels10wayAnswer, $totalProposed, $totalAnswer, 0); 153 | 154 | ### 6. Evaluate without directionality 155 | print "<<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>:\n\n"; 156 | my $officialScore = &evaluate(\%confMatrix10wayWithDir, \%allLabels10wayWithDirProposed, \%allLabels10wayAnswer, $totalProposed, $totalAnswer, 1); 157 | 158 | ### 7. Output the official score 159 | printf "<<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = %0.2f%s >>>\n", $officialScore, '%'; 160 | 161 | 162 | ################ 163 | ### SUBS ### 164 | ################ 165 | 166 | sub getIDandLabel() { 167 | my $line = shift; 168 | return (-1,()) if ($line !~ /^([0-9]+)\t([^\r]+)\r?\n$/); 169 | 170 | my ($id, $label) = ($1, $2); 171 | 172 | return ($id, '_Other') if ($label eq 'Other'); 173 | 174 | return ($id, $label) 175 | if (($label eq 'Cause-Effect(e1,e2)') || ($label eq 'Cause-Effect(e2,e1)') || 176 | ($label eq 'Component-Whole(e1,e2)') || ($label eq 'Component-Whole(e2,e1)') || 177 | ($label eq 'Content-Container(e1,e2)') || ($label eq 'Content-Container(e2,e1)') || 178 | ($label eq 'Entity-Destination(e1,e2)') || ($label eq 'Entity-Destination(e2,e1)') || 179 | ($label eq 'Entity-Origin(e1,e2)') || ($label eq 'Entity-Origin(e2,e1)') || 180 | ($label eq 'Instrument-Agency(e1,e2)') || ($label eq 'Instrument-Agency(e2,e1)') || 181 | ($label eq 'Member-Collection(e1,e2)') || ($label eq 'Member-Collection(e2,e1)') || 182 | ($label eq 'Message-Topic(e1,e2)') || ($label eq 'Message-Topic(e2,e1)') || 183 | ($label eq 'Product-Producer(e1,e2)') || ($label eq 'Product-Producer(e2,e1)')); 184 | 185 | return (-1, ()); 186 | } 187 | 188 | 189 | sub readFileIntoHash() { 190 | my ($fname, $ids) = @_; 191 | open(INPUT, $fname) or die "Failed to open $fname for text reading.\n"; 192 | my $lineNo = 0; 193 | while () { 194 | $lineNo++; 195 | my ($id, $label) = &getIDandLabel($_); 196 | die "Bad file format on line $lineNo: '$_'\n" if ($id < 0); 197 | if (defined $$ids{$id}) { 198 | s/[\n\r]*$//; 199 | die "Bad file format on line $lineNo (ID $id is already defined): '$_'\n"; 200 | } 201 | $$ids{$id} = $label; 202 | } 203 | close(INPUT) or die "Failed to close $fname.\n"; 204 | return $lineNo; 205 | } 206 | 207 | 208 | sub evaluate() { 209 | my ($confMatrix, $allLabelsProposed, $allLabelsAnswer, $totalProposed, $totalAnswer, $useWrongDir) = @_; 210 | 211 | ### 0. Create a merged list for the confusion matrix 212 | my @allLabels = (); 213 | &mergeLabelLists($allLabelsAnswer, $allLabelsProposed, \@allLabels); 214 | 215 | ### 1. Print the confusion matrix heading 216 | print "Confusion matrix:\n"; 217 | print " "; 218 | foreach my $label (@allLabels) { 219 | printf " %4s", &getShortRelName($label, $allLabelsAnswer); 220 | } 221 | print " <-- classified as\n"; 222 | print " +"; 223 | foreach my $label (@allLabels) { 224 | print "-----"; 225 | } 226 | if ($useWrongDir) { 227 | print "+ -SUM- xDIRx skip ACTUAL\n"; 228 | } 229 | else { 230 | print "+ -SUM- skip ACTUAL\n"; 231 | } 232 | 233 | ### 2. Print the rest of the confusion matrix 234 | my $freqCorrect = 0; 235 | my $ind = 1; 236 | my $otherSkipped = 0; 237 | foreach my $labelAnswer (sort keys %{$allLabelsAnswer}) { 238 | 239 | ### 2.1. Output the short relation label 240 | printf " %4s |", &getShortRelName($labelAnswer, $allLabelsAnswer); 241 | 242 | ### 2.2. Output a row of the confusion matrix 243 | my $sumProposed = 0; 244 | foreach my $labelProposed (@allLabels) { 245 | $$confMatrix{$labelProposed}{$labelAnswer} = 0 246 | if (!defined($$confMatrix{$labelProposed}{$labelAnswer})); 247 | printf "%4d ", $$confMatrix{$labelProposed}{$labelAnswer}; 248 | $sumProposed += $$confMatrix{$labelProposed}{$labelAnswer}; 249 | } 250 | 251 | ### 2.3. Output the horizontal sums 252 | if ($useWrongDir) { 253 | my $ans = defined($$allLabelsAnswer{$labelAnswer}) ? $$allLabelsAnswer{$labelAnswer} : 0; 254 | $$confMatrix{'WRONG_DIR'}{$labelAnswer} = 0 if (!defined $$confMatrix{'WRONG_DIR'}{$labelAnswer}); 255 | printf "| %4d %4d %4d %6d\n", $sumProposed, $$confMatrix{'WRONG_DIR'}{$labelAnswer}, $ans - $sumProposed - $$confMatrix{'WRONG_DIR'}{$labelAnswer}, $ans; 256 | if ($labelAnswer eq '_Other') { 257 | $otherSkipped = $ans - $sumProposed - $$confMatrix{'WRONG_DIR'}{$labelAnswer}; 258 | } 259 | } 260 | else { 261 | my $ans = defined($$allLabelsAnswer{$labelAnswer}) ? $$allLabelsAnswer{$labelAnswer} : 0; 262 | printf "| %4d %4d %4d\n", $sumProposed, $ans - $sumProposed, $ans; 263 | if ($labelAnswer eq '_Other') { 264 | $otherSkipped = $ans - $sumProposed; 265 | } 266 | } 267 | 268 | $ind++; 269 | 270 | $$confMatrix{$labelAnswer}{$labelAnswer} = 0 271 | if (!defined($$confMatrix{$labelAnswer}{$labelAnswer})); 272 | $freqCorrect += $$confMatrix{$labelAnswer}{$labelAnswer}; 273 | } 274 | print " +"; 275 | foreach (@allLabels) { 276 | print "-----"; 277 | } 278 | print "+\n"; 279 | 280 | ### 3. Print the vertical sums 281 | print " -SUM- "; 282 | foreach my $labelProposed (@allLabels) { 283 | $$allLabelsProposed{$labelProposed} = 0 284 | if (!defined $$allLabelsProposed{$labelProposed}); 285 | printf "%4d ", $$allLabelsProposed{$labelProposed}; 286 | } 287 | if ($useWrongDir) { 288 | printf " %4d %4d %4d %6d\n\n", $totalProposed - $$allLabelsProposed{'WRONG_DIR'}, $$allLabelsProposed{'WRONG_DIR'}, $totalAnswer - $totalProposed, $totalAnswer; 289 | } 290 | else { 291 | printf " %4d %4d %4d\n\n", $totalProposed, $totalAnswer - $totalProposed, $totalAnswer; 292 | } 293 | 294 | ### 4. Output the coverage 295 | my $coverage = 100.0 * $totalProposed / $totalAnswer; 296 | printf "%s%d%s%d%s%5.2f%s", 'Coverage = ', $totalProposed, '/', $totalAnswer, ' = ', $coverage, "\%\n"; 297 | 298 | ### 5. Output the accuracy 299 | my $accuracy = 100.0 * $freqCorrect / $totalProposed; 300 | printf "%s%d%s%d%s%5.2f%s", 'Accuracy (calculated for the above confusion matrix) = ', $freqCorrect, '/', $totalProposed, ' = ', $accuracy, "\%\n"; 301 | 302 | ### 6. Output the accuracy considering all skipped to be wrong 303 | $accuracy = 100.0 * $freqCorrect / $totalAnswer; 304 | printf "%s%d%s%d%s%5.2f%s", 'Accuracy (considering all skipped examples as Wrong) = ', $freqCorrect, '/', $totalAnswer, ' = ', $accuracy, "\%\n"; 305 | 306 | ### 7. Calculate accuracy with all skipped examples considered Other 307 | my $accuracyWithOther = 100.0 * ($freqCorrect + $otherSkipped) / $totalAnswer; 308 | printf "%s%d%s%d%s%5.2f%s", 'Accuracy (considering all skipped examples as Other) = ', ($freqCorrect + $otherSkipped), '/', $totalAnswer, ' = ', $accuracyWithOther, "\%\n"; 309 | 310 | ### 8. Output P, R, F1 for each relation 311 | my ($macroP, $macroR, $macroF1) = (0, 0, 0); 312 | my ($microCorrect, $microProposed, $microAnswer) = (0, 0, 0); 313 | print "\nResults for the individual relations:\n"; 314 | foreach my $labelAnswer (sort keys %{$allLabelsAnswer}) { 315 | 316 | ### 8.1. Consider all wrong directionalities as wrong classification decisions 317 | my $wrongDirectionCnt = 0; 318 | if ($useWrongDir && defined $$confMatrix{'WRONG_DIR'}{$labelAnswer}) { 319 | $wrongDirectionCnt = $$confMatrix{'WRONG_DIR'}{$labelAnswer}; 320 | } 321 | 322 | ### 8.2. Prevent Perl complains about unintialized values 323 | if (!defined($$allLabelsProposed{$labelAnswer})) { 324 | $$allLabelsProposed{$labelAnswer} = 0; 325 | } 326 | 327 | ### 8.3. Calculate P/R/F1 328 | my $P = (0 == $$allLabelsProposed{$labelAnswer}) ? 0 329 | : 100.0 * $$confMatrix{$labelAnswer}{$labelAnswer} / ($$allLabelsProposed{$labelAnswer} + $wrongDirectionCnt); 330 | my $R = (0 == $$allLabelsAnswer{$labelAnswer}) ? 0 331 | : 100.0 * $$confMatrix{$labelAnswer}{$labelAnswer} / $$allLabelsAnswer{$labelAnswer}; 332 | my $F1 = (0 == $P + $R) ? 0 : 2 * $P * $R / ($P + $R); 333 | 334 | ### 8.4. Output P/R/F1 335 | if ($useWrongDir) { 336 | printf "%25s%s%4d%s(%4d +%4d)%s%6.2f", $labelAnswer, 337 | " : P = ", $$confMatrix{$labelAnswer}{$labelAnswer}, '/', $$allLabelsProposed{$labelAnswer}, $wrongDirectionCnt, ' = ', $P; 338 | } 339 | else { 340 | printf "%25s%s%4d%s%4d%s%6.2f", $labelAnswer, 341 | " : P = ", $$confMatrix{$labelAnswer}{$labelAnswer}, '/', ($$allLabelsProposed{$labelAnswer} + $wrongDirectionCnt), ' = ', $P; 342 | } 343 | printf"%s%4d%s%4d%s%6.2f%s%6.2f%s\n", 344 | "% R = ", $$confMatrix{$labelAnswer}{$labelAnswer}, '/', $$allLabelsAnswer{$labelAnswer}, ' = ', $R, 345 | "% F1 = ", $F1, '%'; 346 | 347 | ### 8.5. Accumulate statistics for micro/macro-averaging 348 | if ($labelAnswer ne '_Other') { 349 | $macroP += $P; 350 | $macroR += $R; 351 | $macroF1 += $F1; 352 | $microCorrect += $$confMatrix{$labelAnswer}{$labelAnswer}; 353 | $microProposed += $$allLabelsProposed{$labelAnswer} + $wrongDirectionCnt; 354 | $microAnswer += $$allLabelsAnswer{$labelAnswer}; 355 | } 356 | } 357 | 358 | ### 9. Output the micro-averaged P, R, F1 359 | my $microP = (0 == $microProposed) ? 0 : 100.0 * $microCorrect / $microProposed; 360 | my $microR = (0 == $microAnswer) ? 0 : 100.0 * $microCorrect / $microAnswer; 361 | my $microF1 = (0 == $microP + $microR) ? 0 : 2.0 * $microP * $microR / ($microP + $microR); 362 | print "\nMicro-averaged result (excluding Other):\n"; 363 | printf "%s%4d%s%4d%s%6.2f%s%4d%s%4d%s%6.2f%s%6.2f%s\n", 364 | "P = ", $microCorrect, '/', $microProposed, ' = ', $microP, 365 | "% R = ", $microCorrect, '/', $microAnswer, ' = ', $microR, 366 | "% F1 = ", $microF1, '%'; 367 | 368 | ### 10. Output the macro-averaged P, R, F1 369 | my $distinctLabelsCnt = keys %{$allLabelsAnswer}; 370 | ## -1, if '_Other' exists 371 | $distinctLabelsCnt-- if (defined $$allLabelsAnswer{'_Other'}); 372 | 373 | $macroP /= $distinctLabelsCnt; # first divide by the number of non-Other categories 374 | $macroR /= $distinctLabelsCnt; 375 | $macroF1 /= $distinctLabelsCnt; 376 | print "\nMACRO-averaged result (excluding Other):\n"; 377 | printf "%s%6.2f%s%6.2f%s%6.2f%s\n\n\n\n", "P = ", $macroP, "%\tR = ", $macroR, "%\tF1 = ", $macroF1, '%'; 378 | 379 | ### 11. Return the official score 380 | return $macroF1; 381 | } 382 | 383 | 384 | sub getShortRelName() { 385 | my ($relName, $hashToCheck) = @_; 386 | return '_O_' if ($relName eq '_Other'); 387 | die "relName='$relName'" if ($relName !~ /^(.)[^\-]+\-(.)/); 388 | my $result = (defined $$hashToCheck{$relName}) ? "$1\-$2" : "*$1$2"; 389 | if ($relName =~ /\(e([12])/) { 390 | $result .= $1; 391 | } 392 | return $result; 393 | } 394 | 395 | sub mergeLabelLists() { 396 | my ($hash1, $hash2, $mergedList) = @_; 397 | foreach my $key (sort keys %{$hash1}) { 398 | push @{$mergedList}, $key if ($key ne 'WRONG_DIR'); 399 | } 400 | foreach my $key (sort keys %{$hash2}) { 401 | push @{$mergedList}, $key if (($key ne 'WRONG_DIR') && !defined($$hash1{$key})); 402 | } 403 | } 404 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_testing/README.txt: -------------------------------------------------------------------------------- 1 | Test Data for SemEval-2 Task #8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals 2 | 3 | Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano and Stan Szpakowicz 4 | 5 | The accompanying dataset is released under a Creative Commons Atrribution 3.0 Unported Licence (http://creativecommons.org/licenses/by/3.0/). 6 | 7 | Version 1.0: March 18, 2010 8 | 9 | 10 | SUMMARY 11 | 12 | This test dataset consists of 2717 sentences that have been collected from the Web specifically for SemEval-2 Task #8. 13 | The sentences do not overlap with the 8000 training sentences that have been released on March 5, 2010; 14 | they also do not overlap with the sentences from SemEval-1 Task #4 (Classification of Semantic Relations between Nominals). 15 | 16 | 17 | IMPORTANT 18 | 19 | To use this test dataset, the participants should download from the Official SemEval-2010 website the following: 20 | 21 | 1. the training dataset (it also contains relation definitions and our annotation guidelines); 22 | 2. the official scorer and format checker. 23 | 24 | 25 | INPUT DATA FORMAT 26 | 27 | The format of the test data is illustrated by the following examples: 28 | 29 | 8001 "The most common audits were about waste and recycling." 30 | 8002 "The company fabricates plastic chairs." 31 | 8003 "The school master teaches the lesson with a stick." 32 | 8004 "The suspect dumped the dead body into a local reservoir." 33 | .... 34 | 10717 "A few days before the service, Tom Burris had thrown into Karen's casket his wedding ring." 35 | 36 | Each line contains a sentence inside quotation marks, preceded by a numerical identifier. In each sentence, two entity mentions are tagged as e1 and e2 -- the numbering simply reflects the order of the mentions in the sentence. The span of the tag corresponds to the "base NP" which may be smaller than the full NP denoting the entity (see the annotation guidelines for details). 37 | 38 | 39 | EVALUATION 40 | 41 | The task is to predict, given a sentence and two tagged entities, which of the relation labels to apply. The predictions must be in the following format: 42 | 43 | 1 Content-Container(e2,e1) 44 | 2 Other 45 | 3 Entity-Destination(e1,e2) 46 | ... 47 | 48 | There is a format checker released together with the scorer, which the participants should use to check their output before submitting their results. 49 | 50 | The official evaluation measures are accuracy over all examples and macro-averaged F-score over the 9 relation labels apart from Other. To calculate the F-score, 9 individual F-scores -- one for each relation label -- are calculated in the standard way and the average of these scores is taken. See the README of the official scorer for more details. 51 | 52 | 53 | TEST PROCEDURE 54 | 55 | The participants can download the test dataset at any time up to the final results submission deadline (April 2, 2010). Once the data have been downloaded, participants will have 7 days to submit their results; they must also submit by the final deadline of April 2. Late submissions will not be counted. Participants should supply four sets of predictions for the test data, using four subsets of the training data: 56 | 57 | TD1 training examples 1-1000 58 | TD2 training examples 1-2000 59 | TD3 training examples 1-4000 60 | TD4 training examples 1-8000 61 | 62 | For each training set, participants may use the data in that set for any purpose they wish (training, development, cross-validation and so forth). However, the training examples outside that set (e.g., 1001-8000 for TD1) may not be used in any way. The final 891 examples in the training release (examples 7110-8000) are taken from the SemEval-1 Task #4 datasets for relations 1-5 and hence their label distribution is skewed towards those relation classes. Participants have the option of including or excluding these examples as appropriate for their chosen learning method. See the training data archive for details. 63 | 64 | There is no restriction on the external resources that may be used. 65 | 66 | 67 | SUBMISSION PROCEDURE 68 | 69 | The participants should do a submission via the SemEval-2 website (shown below). 70 | 71 | Each participating team should choose a short ID to be used in the official ranking. Ideally, the ID should be an abbreviation of the university, e.g., NUS for the National University of Singapore. Each team is allowed to submit multiple runs. In that case, a run ID should be augmented with run identification, e.g., NUS-WN, NUS-1, NUS-2, etc. 72 | 73 | The submission should contain five files each starting with the above-described ID, e.g., if the ID is NUS, the files should be 74 | - NUS_TD1.txt 75 | - NUS_TD2.txt 76 | - NUS_TD3.txt 77 | - NUS_TD4.txt 78 | - NUS_description.txt 79 | 80 | The first four files should contain the classification decisions for test datasets TD1, TD2, TD3 and TD4, respectively. 81 | 82 | The fifth file should contain the following information: 83 | - ID of the team 84 | - Names and affiliations of the participating team 85 | - Contact person with email address: where we will send the results of the evaluation 86 | - Short description of the approach for *each* run: one very short paragraph for each one that will help us in preparing the task overview paper 87 | - Short description of the resources used for *each* run 88 | 89 | 90 | USEFUL LINKS 91 | 92 | Google group: http://groups.google.com.sg/group/semeval-2010-multi-way-classification-of-semantic-relations?hl=en 93 | Task website: http://docs.google.com/View?docid=dfvxd49s_36c28v9pmw 94 | SemEval-2 website: http://semeval2.fbk.eu/semeval2.php 95 | 96 | 97 | TASK SCHEDULE 98 | 99 | * Test data release: March 18, 2010 100 | * Result submission deadline: 7 days after downloading the *test* data, but no later than April 2 101 | * Organizers send the test results: April 10, 2010 102 | * Submission of description papers: April 17, 2010 103 | * Notification of acceptance: May 6, 2010 104 | * SemEval-2 workshop (at ACL): July 15-16, 2010 105 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_KEY.TXT: -------------------------------------------------------------------------------- 1 | 8001 Message-Topic 2 | 8002 Product-Producer 3 | 8003 Instrument-Agency 4 | 8004 Entity-Destination 5 | 8005 Cause-Effect 6 | 8006 Component-Whole 7 | 8007 Product-Producer 8 | 8008 Member-Collection 9 | 8009 Component-Whole 10 | 8010 Message-Topic 11 | 8011 Entity-Destination 12 | 8012 Other 13 | 8013 Entity-Destination 14 | 8014 Product-Producer 15 | 8015 Entity-Origin 16 | 8016 Entity-Origin 17 | 8017 Entity-Destination 18 | 8018 Other 19 | 8019 Member-Collection 20 | 8020 Product-Producer 21 | 8021 Message-Topic 22 | 8022 Content-Container 23 | 8023 Product-Producer 24 | 8024 Other 25 | 8025 Entity-Origin 26 | 8026 Product-Producer 27 | 8027 Cause-Effect 28 | 8028 Other 29 | 8029 Other 30 | 8030 Entity-Origin 31 | 8031 Cause-Effect 32 | 8032 Message-Topic 33 | 8033 Component-Whole 34 | 8034 Product-Producer 35 | 8035 Component-Whole 36 | 8036 Component-Whole 37 | 8037 Member-Collection 38 | 8038 Content-Container 39 | 8039 Member-Collection 40 | 8040 Product-Producer 41 | 8041 Cause-Effect 42 | 8042 Component-Whole 43 | 8043 Cause-Effect 44 | 8044 Entity-Destination 45 | 8045 Entity-Origin 46 | 8046 Content-Container 47 | 8047 Other 48 | 8048 Entity-Destination 49 | 8049 Message-Topic 50 | 8050 Other 51 | 8051 Entity-Destination 52 | 8052 Other 53 | 8053 Member-Collection 54 | 8054 Other 55 | 8055 Cause-Effect 56 | 8056 Entity-Origin 57 | 8057 Other 58 | 8058 Cause-Effect 59 | 8059 Other 60 | 8060 Component-Whole 61 | 8061 Entity-Origin 62 | 8062 Product-Producer 63 | 8063 Instrument-Agency 64 | 8064 Component-Whole 65 | 8065 Entity-Destination 66 | 8066 Product-Producer 67 | 8067 Other 68 | 8068 Other 69 | 8069 Message-Topic 70 | 8070 Product-Producer 71 | 8071 Other 72 | 8072 Entity-Origin 73 | 8073 Cause-Effect 74 | 8074 Entity-Origin 75 | 8075 Other 76 | 8076 Product-Producer 77 | 8077 Other 78 | 8078 Instrument-Agency 79 | 8079 Entity-Destination 80 | 8080 Product-Producer 81 | 8081 Component-Whole 82 | 8082 Component-Whole 83 | 8083 Cause-Effect 84 | 8084 Component-Whole 85 | 8085 Message-Topic 86 | 8086 Instrument-Agency 87 | 8087 Message-Topic 88 | 8088 Product-Producer 89 | 8089 Entity-Origin 90 | 8090 Message-Topic 91 | 8091 Entity-Origin 92 | 8092 Other 93 | 8093 Component-Whole 94 | 8094 Component-Whole 95 | 8095 Other 96 | 8096 Entity-Destination 97 | 8097 Message-Topic 98 | 8098 Component-Whole 99 | 8099 Entity-Destination 100 | 8100 Message-Topic 101 | 8101 Message-Topic 102 | 8102 Component-Whole 103 | 8103 Entity-Origin 104 | 8104 Message-Topic 105 | 8105 Cause-Effect 106 | 8106 Other 107 | 8107 Cause-Effect 108 | 8108 Cause-Effect 109 | 8109 Component-Whole 110 | 8110 Member-Collection 111 | 8111 Other 112 | 8112 Content-Container 113 | 8113 Other 114 | 8114 Product-Producer 115 | 8115 Other 116 | 8116 Cause-Effect 117 | 8117 Product-Producer 118 | 8118 Cause-Effect 119 | 8119 Member-Collection 120 | 8120 Component-Whole 121 | 8121 Entity-Destination 122 | 8122 Instrument-Agency 123 | 8123 Other 124 | 8124 Other 125 | 8125 Message-Topic 126 | 8126 Entity-Origin 127 | 8127 Entity-Origin 128 | 8128 Other 129 | 8129 Component-Whole 130 | 8130 Content-Container 131 | 8131 Instrument-Agency 132 | 8132 Message-Topic 133 | 8133 Component-Whole 134 | 8134 Other 135 | 8135 Content-Container 136 | 8136 Instrument-Agency 137 | 8137 Component-Whole 138 | 8138 Member-Collection 139 | 8139 Entity-Origin 140 | 8140 Member-Collection 141 | 8141 Instrument-Agency 142 | 8142 Entity-Origin 143 | 8143 Other 144 | 8144 Entity-Origin 145 | 8145 Member-Collection 146 | 8146 Instrument-Agency 147 | 8147 Content-Container 148 | 8148 Message-Topic 149 | 8149 Other 150 | 8150 Product-Producer 151 | 8151 Product-Producer 152 | 8152 Member-Collection 153 | 8153 Member-Collection 154 | 8154 Message-Topic 155 | 8155 Message-Topic 156 | 8156 Product-Producer 157 | 8157 Other 158 | 8158 Component-Whole 159 | 8159 Cause-Effect 160 | 8160 Message-Topic 161 | 8161 Message-Topic 162 | 8162 Entity-Origin 163 | 8163 Entity-Origin 164 | 8164 Product-Producer 165 | 8165 Entity-Destination 166 | 8166 Content-Container 167 | 8167 Member-Collection 168 | 8168 Component-Whole 169 | 8169 Entity-Origin 170 | 8170 Instrument-Agency 171 | 8171 Entity-Destination 172 | 8172 Member-Collection 173 | 8173 Other 174 | 8174 Other 175 | 8175 Cause-Effect 176 | 8176 Product-Producer 177 | 8177 Entity-Destination 178 | 8178 Entity-Origin 179 | 8179 Instrument-Agency 180 | 8180 Message-Topic 181 | 8181 Entity-Destination 182 | 8182 Content-Container 183 | 8183 Other 184 | 8184 Product-Producer 185 | 8185 Other 186 | 8186 Member-Collection 187 | 8187 Entity-Destination 188 | 8188 Product-Producer 189 | 8189 Message-Topic 190 | 8190 Instrument-Agency 191 | 8191 Cause-Effect 192 | 8192 Other 193 | 8193 Message-Topic 194 | 8194 Component-Whole 195 | 8195 Message-Topic 196 | 8196 Other 197 | 8197 Entity-Origin 198 | 8198 Entity-Destination 199 | 8199 Entity-Destination 200 | 8200 Product-Producer 201 | 8201 Component-Whole 202 | 8202 Content-Container 203 | 8203 Other 204 | 8204 Cause-Effect 205 | 8205 Entity-Destination 206 | 8206 Component-Whole 207 | 8207 Component-Whole 208 | 8208 Content-Container 209 | 8209 Member-Collection 210 | 8210 Member-Collection 211 | 8211 Component-Whole 212 | 8212 Entity-Origin 213 | 8213 Content-Container 214 | 8214 Instrument-Agency 215 | 8215 Entity-Origin 216 | 8216 Content-Container 217 | 8217 Content-Container 218 | 8218 Other 219 | 8219 Cause-Effect 220 | 8220 Message-Topic 221 | 8221 Content-Container 222 | 8222 Entity-Origin 223 | 8223 Message-Topic 224 | 8224 Message-Topic 225 | 8225 Other 226 | 8226 Other 227 | 8227 Content-Container 228 | 8228 Member-Collection 229 | 8229 Product-Producer 230 | 8230 Other 231 | 8231 Entity-Origin 232 | 8232 Component-Whole 233 | 8233 Message-Topic 234 | 8234 Cause-Effect 235 | 8235 Component-Whole 236 | 8236 Cause-Effect 237 | 8237 Other 238 | 8238 Component-Whole 239 | 8239 Cause-Effect 240 | 8240 Cause-Effect 241 | 8241 Product-Producer 242 | 8242 Entity-Destination 243 | 8243 Component-Whole 244 | 8244 Other 245 | 8245 Other 246 | 8246 Product-Producer 247 | 8247 Content-Container 248 | 8248 Component-Whole 249 | 8249 Entity-Origin 250 | 8250 Entity-Destination 251 | 8251 Component-Whole 252 | 8252 Entity-Origin 253 | 8253 Cause-Effect 254 | 8254 Component-Whole 255 | 8255 Other 256 | 8256 Other 257 | 8257 Cause-Effect 258 | 8258 Product-Producer 259 | 8259 Component-Whole 260 | 8260 Instrument-Agency 261 | 8261 Message-Topic 262 | 8262 Entity-Destination 263 | 8263 Entity-Origin 264 | 8264 Message-Topic 265 | 8265 Cause-Effect 266 | 8266 Entity-Destination 267 | 8267 Message-Topic 268 | 8268 Component-Whole 269 | 8269 Other 270 | 8270 Entity-Destination 271 | 8271 Other 272 | 8272 Other 273 | 8273 Message-Topic 274 | 8274 Member-Collection 275 | 8275 Other 276 | 8276 Entity-Destination 277 | 8277 Message-Topic 278 | 8278 Instrument-Agency 279 | 8279 Product-Producer 280 | 8280 Product-Producer 281 | 8281 Member-Collection 282 | 8282 Entity-Destination 283 | 8283 Member-Collection 284 | 8284 Other 285 | 8285 Message-Topic 286 | 8286 Content-Container 287 | 8287 Member-Collection 288 | 8288 Cause-Effect 289 | 8289 Other 290 | 8290 Message-Topic 291 | 8291 Content-Container 292 | 8292 Message-Topic 293 | 8293 Component-Whole 294 | 8294 Other 295 | 8295 Entity-Origin 296 | 8296 Entity-Origin 297 | 8297 Entity-Destination 298 | 8298 Entity-Destination 299 | 8299 Entity-Destination 300 | 8300 Product-Producer 301 | 8301 Other 302 | 8302 Instrument-Agency 303 | 8303 Component-Whole 304 | 8304 Other 305 | 8305 Product-Producer 306 | 8306 Message-Topic 307 | 8307 Product-Producer 308 | 8308 Other 309 | 8309 Message-Topic 310 | 8310 Product-Producer 311 | 8311 Other 312 | 8312 Cause-Effect 313 | 8313 Message-Topic 314 | 8314 Product-Producer 315 | 8315 Message-Topic 316 | 8316 Member-Collection 317 | 8317 Content-Container 318 | 8318 Content-Container 319 | 8319 Entity-Destination 320 | 8320 Instrument-Agency 321 | 8321 Entity-Destination 322 | 8322 Member-Collection 323 | 8323 Member-Collection 324 | 8324 Entity-Destination 325 | 8325 Content-Container 326 | 8326 Other 327 | 8327 Message-Topic 328 | 8328 Message-Topic 329 | 8329 Message-Topic 330 | 8330 Product-Producer 331 | 8331 Member-Collection 332 | 8332 Message-Topic 333 | 8333 Message-Topic 334 | 8334 Cause-Effect 335 | 8335 Member-Collection 336 | 8336 Other 337 | 8337 Other 338 | 8338 Message-Topic 339 | 8339 Other 340 | 8340 Content-Container 341 | 8341 Message-Topic 342 | 8342 Other 343 | 8343 Instrument-Agency 344 | 8344 Entity-Destination 345 | 8345 Content-Container 346 | 8346 Content-Container 347 | 8347 Other 348 | 8348 Other 349 | 8349 Member-Collection 350 | 8350 Component-Whole 351 | 8351 Content-Container 352 | 8352 Member-Collection 353 | 8353 Message-Topic 354 | 8354 Message-Topic 355 | 8355 Content-Container 356 | 8356 Other 357 | 8357 Cause-Effect 358 | 8358 Instrument-Agency 359 | 8359 Member-Collection 360 | 8360 Component-Whole 361 | 8361 Cause-Effect 362 | 8362 Other 363 | 8363 Entity-Origin 364 | 8364 Instrument-Agency 365 | 8365 Product-Producer 366 | 8366 Message-Topic 367 | 8367 Entity-Destination 368 | 8368 Entity-Destination 369 | 8369 Member-Collection 370 | 8370 Other 371 | 8371 Component-Whole 372 | 8372 Other 373 | 8373 Cause-Effect 374 | 8374 Product-Producer 375 | 8375 Entity-Destination 376 | 8376 Entity-Destination 377 | 8377 Cause-Effect 378 | 8378 Product-Producer 379 | 8379 Other 380 | 8380 Other 381 | 8381 Instrument-Agency 382 | 8382 Cause-Effect 383 | 8383 Entity-Destination 384 | 8384 Other 385 | 8385 Entity-Origin 386 | 8386 Component-Whole 387 | 8387 Product-Producer 388 | 8388 Component-Whole 389 | 8389 Message-Topic 390 | 8390 Other 391 | 8391 Other 392 | 8392 Component-Whole 393 | 8393 Entity-Origin 394 | 8394 Entity-Origin 395 | 8395 Component-Whole 396 | 8396 Other 397 | 8397 Other 398 | 8398 Entity-Destination 399 | 8399 Instrument-Agency 400 | 8400 Other 401 | 8401 Entity-Destination 402 | 8402 Cause-Effect 403 | 8403 Cause-Effect 404 | 8404 Cause-Effect 405 | 8405 Cause-Effect 406 | 8406 Component-Whole 407 | 8407 Other 408 | 8408 Entity-Origin 409 | 8409 Cause-Effect 410 | 8410 Entity-Destination 411 | 8411 Entity-Origin 412 | 8412 Content-Container 413 | 8413 Component-Whole 414 | 8414 Entity-Destination 415 | 8415 Member-Collection 416 | 8416 Component-Whole 417 | 8417 Cause-Effect 418 | 8418 Entity-Destination 419 | 8419 Content-Container 420 | 8420 Message-Topic 421 | 8421 Component-Whole 422 | 8422 Component-Whole 423 | 8423 Entity-Destination 424 | 8424 Instrument-Agency 425 | 8425 Other 426 | 8426 Other 427 | 8427 Component-Whole 428 | 8428 Product-Producer 429 | 8429 Component-Whole 430 | 8430 Entity-Origin 431 | 8431 Component-Whole 432 | 8432 Other 433 | 8433 Member-Collection 434 | 8434 Other 435 | 8435 Other 436 | 8436 Other 437 | 8437 Message-Topic 438 | 8438 Component-Whole 439 | 8439 Cause-Effect 440 | 8440 Message-Topic 441 | 8441 Product-Producer 442 | 8442 Component-Whole 443 | 8443 Component-Whole 444 | 8444 Component-Whole 445 | 8445 Content-Container 446 | 8446 Product-Producer 447 | 8447 Other 448 | 8448 Other 449 | 8449 Entity-Origin 450 | 8450 Other 451 | 8451 Other 452 | 8452 Member-Collection 453 | 8453 Entity-Origin 454 | 8454 Product-Producer 455 | 8455 Cause-Effect 456 | 8456 Entity-Destination 457 | 8457 Entity-Destination 458 | 8458 Product-Producer 459 | 8459 Instrument-Agency 460 | 8460 Entity-Destination 461 | 8461 Other 462 | 8462 Product-Producer 463 | 8463 Entity-Destination 464 | 8464 Entity-Destination 465 | 8465 Content-Container 466 | 8466 Other 467 | 8467 Entity-Destination 468 | 8468 Entity-Destination 469 | 8469 Entity-Origin 470 | 8470 Component-Whole 471 | 8471 Cause-Effect 472 | 8472 Component-Whole 473 | 8473 Cause-Effect 474 | 8474 Content-Container 475 | 8475 Other 476 | 8476 Cause-Effect 477 | 8477 Other 478 | 8478 Entity-Origin 479 | 8479 Message-Topic 480 | 8480 Message-Topic 481 | 8481 Entity-Destination 482 | 8482 Other 483 | 8483 Product-Producer 484 | 8484 Other 485 | 8485 Product-Producer 486 | 8486 Cause-Effect 487 | 8487 Other 488 | 8488 Product-Producer 489 | 8489 Cause-Effect 490 | 8490 Content-Container 491 | 8491 Other 492 | 8492 Member-Collection 493 | 8493 Cause-Effect 494 | 8494 Cause-Effect 495 | 8495 Message-Topic 496 | 8496 Entity-Destination 497 | 8497 Entity-Origin 498 | 8498 Cause-Effect 499 | 8499 Component-Whole 500 | 8500 Cause-Effect 501 | 8501 Message-Topic 502 | 8502 Content-Container 503 | 8503 Cause-Effect 504 | 8504 Entity-Origin 505 | 8505 Content-Container 506 | 8506 Entity-Destination 507 | 8507 Member-Collection 508 | 8508 Other 509 | 8509 Cause-Effect 510 | 8510 Other 511 | 8511 Instrument-Agency 512 | 8512 Cause-Effect 513 | 8513 Other 514 | 8514 Message-Topic 515 | 8515 Other 516 | 8516 Other 517 | 8517 Entity-Origin 518 | 8518 Entity-Origin 519 | 8519 Product-Producer 520 | 8520 Cause-Effect 521 | 8521 Cause-Effect 522 | 8522 Other 523 | 8523 Cause-Effect 524 | 8524 Instrument-Agency 525 | 8525 Entity-Origin 526 | 8526 Entity-Destination 527 | 8527 Component-Whole 528 | 8528 Content-Container 529 | 8529 Entity-Destination 530 | 8530 Product-Producer 531 | 8531 Component-Whole 532 | 8532 Other 533 | 8533 Product-Producer 534 | 8534 Cause-Effect 535 | 8535 Cause-Effect 536 | 8536 Cause-Effect 537 | 8537 Other 538 | 8538 Member-Collection 539 | 8539 Member-Collection 540 | 8540 Other 541 | 8541 Product-Producer 542 | 8542 Cause-Effect 543 | 8543 Entity-Origin 544 | 8544 Message-Topic 545 | 8545 Instrument-Agency 546 | 8546 Entity-Origin 547 | 8547 Component-Whole 548 | 8548 Component-Whole 549 | 8549 Component-Whole 550 | 8550 Other 551 | 8551 Message-Topic 552 | 8552 Entity-Destination 553 | 8553 Message-Topic 554 | 8554 Content-Container 555 | 8555 Entity-Origin 556 | 8556 Cause-Effect 557 | 8557 Entity-Origin 558 | 8558 Entity-Destination 559 | 8559 Product-Producer 560 | 8560 Other 561 | 8561 Component-Whole 562 | 8562 Entity-Origin 563 | 8563 Message-Topic 564 | 8564 Message-Topic 565 | 8565 Other 566 | 8566 Entity-Destination 567 | 8567 Instrument-Agency 568 | 8568 Other 569 | 8569 Entity-Origin 570 | 8570 Member-Collection 571 | 8571 Other 572 | 8572 Member-Collection 573 | 8573 Other 574 | 8574 Component-Whole 575 | 8575 Entity-Destination 576 | 8576 Content-Container 577 | 8577 Member-Collection 578 | 8578 Member-Collection 579 | 8579 Message-Topic 580 | 8580 Message-Topic 581 | 8581 Other 582 | 8582 Other 583 | 8583 Member-Collection 584 | 8584 Component-Whole 585 | 8585 Message-Topic 586 | 8586 Component-Whole 587 | 8587 Entity-Origin 588 | 8588 Message-Topic 589 | 8589 Message-Topic 590 | 8590 Member-Collection 591 | 8591 Cause-Effect 592 | 8592 Other 593 | 8593 Product-Producer 594 | 8594 Entity-Origin 595 | 8595 Product-Producer 596 | 8596 Cause-Effect 597 | 8597 Message-Topic 598 | 8598 Entity-Destination 599 | 8599 Component-Whole 600 | 8600 Member-Collection 601 | 8601 Product-Producer 602 | 8602 Cause-Effect 603 | 8603 Cause-Effect 604 | 8604 Message-Topic 605 | 8605 Component-Whole 606 | 8606 Entity-Destination 607 | 8607 Other 608 | 8608 Cause-Effect 609 | 8609 Component-Whole 610 | 8610 Other 611 | 8611 Message-Topic 612 | 8612 Entity-Origin 613 | 8613 Content-Container 614 | 8614 Entity-Origin 615 | 8615 Other 616 | 8616 Component-Whole 617 | 8617 Entity-Origin 618 | 8618 Other 619 | 8619 Entity-Destination 620 | 8620 Entity-Origin 621 | 8621 Cause-Effect 622 | 8622 Component-Whole 623 | 8623 Cause-Effect 624 | 8624 Component-Whole 625 | 8625 Message-Topic 626 | 8626 Other 627 | 8627 Member-Collection 628 | 8628 Other 629 | 8629 Message-Topic 630 | 8630 Entity-Destination 631 | 8631 Entity-Destination 632 | 8632 Component-Whole 633 | 8633 Cause-Effect 634 | 8634 Instrument-Agency 635 | 8635 Entity-Origin 636 | 8636 Content-Container 637 | 8637 Instrument-Agency 638 | 8638 Member-Collection 639 | 8639 Entity-Destination 640 | 8640 Entity-Origin 641 | 8641 Cause-Effect 642 | 8642 Product-Producer 643 | 8643 Entity-Destination 644 | 8644 Product-Producer 645 | 8645 Other 646 | 8646 Other 647 | 8647 Other 648 | 8648 Cause-Effect 649 | 8649 Member-Collection 650 | 8650 Message-Topic 651 | 8651 Message-Topic 652 | 8652 Other 653 | 8653 Entity-Origin 654 | 8654 Content-Container 655 | 8655 Cause-Effect 656 | 8656 Member-Collection 657 | 8657 Component-Whole 658 | 8658 Message-Topic 659 | 8659 Cause-Effect 660 | 8660 Message-Topic 661 | 8661 Product-Producer 662 | 8662 Message-Topic 663 | 8663 Entity-Destination 664 | 8664 Product-Producer 665 | 8665 Component-Whole 666 | 8666 Component-Whole 667 | 8667 Component-Whole 668 | 8668 Other 669 | 8669 Member-Collection 670 | 8670 Entity-Destination 671 | 8671 Content-Container 672 | 8672 Message-Topic 673 | 8673 Product-Producer 674 | 8674 Message-Topic 675 | 8675 Component-Whole 676 | 8676 Message-Topic 677 | 8677 Component-Whole 678 | 8678 Other 679 | 8679 Component-Whole 680 | 8680 Other 681 | 8681 Cause-Effect 682 | 8682 Message-Topic 683 | 8683 Member-Collection 684 | 8684 Component-Whole 685 | 8685 Content-Container 686 | 8686 Member-Collection 687 | 8687 Other 688 | 8688 Entity-Origin 689 | 8689 Content-Container 690 | 8690 Cause-Effect 691 | 8691 Message-Topic 692 | 8692 Component-Whole 693 | 8693 Content-Container 694 | 8694 Other 695 | 8695 Content-Container 696 | 8696 Member-Collection 697 | 8697 Other 698 | 8698 Entity-Destination 699 | 8699 Entity-Origin 700 | 8700 Product-Producer 701 | 8701 Member-Collection 702 | 8702 Component-Whole 703 | 8703 Component-Whole 704 | 8704 Entity-Origin 705 | 8705 Cause-Effect 706 | 8706 Other 707 | 8707 Content-Container 708 | 8708 Cause-Effect 709 | 8709 Entity-Origin 710 | 8710 Entity-Destination 711 | 8711 Message-Topic 712 | 8712 Member-Collection 713 | 8713 Member-Collection 714 | 8714 Member-Collection 715 | 8715 Content-Container 716 | 8716 Other 717 | 8717 Product-Producer 718 | 8718 Other 719 | 8719 Entity-Destination 720 | 8720 Cause-Effect 721 | 8721 Other 722 | 8722 Product-Producer 723 | 8723 Product-Producer 724 | 8724 Component-Whole 725 | 8725 Message-Topic 726 | 8726 Other 727 | 8727 Product-Producer 728 | 8728 Content-Container 729 | 8729 Member-Collection 730 | 8730 Component-Whole 731 | 8731 Cause-Effect 732 | 8732 Instrument-Agency 733 | 8733 Entity-Origin 734 | 8734 Entity-Origin 735 | 8735 Component-Whole 736 | 8736 Cause-Effect 737 | 8737 Instrument-Agency 738 | 8738 Content-Container 739 | 8739 Cause-Effect 740 | 8740 Cause-Effect 741 | 8741 Member-Collection 742 | 8742 Entity-Destination 743 | 8743 Entity-Destination 744 | 8744 Product-Producer 745 | 8745 Cause-Effect 746 | 8746 Component-Whole 747 | 8747 Entity-Origin 748 | 8748 Cause-Effect 749 | 8749 Entity-Origin 750 | 8750 Instrument-Agency 751 | 8751 Member-Collection 752 | 8752 Cause-Effect 753 | 8753 Other 754 | 8754 Cause-Effect 755 | 8755 Entity-Destination 756 | 8756 Product-Producer 757 | 8757 Entity-Destination 758 | 8758 Entity-Destination 759 | 8759 Other 760 | 8760 Entity-Destination 761 | 8761 Entity-Origin 762 | 8762 Entity-Origin 763 | 8763 Other 764 | 8764 Cause-Effect 765 | 8765 Product-Producer 766 | 8766 Product-Producer 767 | 8767 Message-Topic 768 | 8768 Product-Producer 769 | 8769 Product-Producer 770 | 8770 Content-Container 771 | 8771 Other 772 | 8772 Entity-Destination 773 | 8773 Member-Collection 774 | 8774 Cause-Effect 775 | 8775 Cause-Effect 776 | 8776 Component-Whole 777 | 8777 Content-Container 778 | 8778 Component-Whole 779 | 8779 Component-Whole 780 | 8780 Content-Container 781 | 8781 Cause-Effect 782 | 8782 Instrument-Agency 783 | 8783 Product-Producer 784 | 8784 Entity-Origin 785 | 8785 Other 786 | 8786 Other 787 | 8787 Entity-Origin 788 | 8788 Message-Topic 789 | 8789 Message-Topic 790 | 8790 Instrument-Agency 791 | 8791 Entity-Destination 792 | 8792 Other 793 | 8793 Entity-Destination 794 | 8794 Other 795 | 8795 Member-Collection 796 | 8796 Member-Collection 797 | 8797 Product-Producer 798 | 8798 Member-Collection 799 | 8799 Entity-Origin 800 | 8800 Entity-Destination 801 | 8801 Other 802 | 8802 Component-Whole 803 | 8803 Member-Collection 804 | 8804 Instrument-Agency 805 | 8805 Entity-Origin 806 | 8806 Content-Container 807 | 8807 Component-Whole 808 | 8808 Component-Whole 809 | 8809 Other 810 | 8810 Entity-Origin 811 | 8811 Instrument-Agency 812 | 8812 Cause-Effect 813 | 8813 Instrument-Agency 814 | 8814 Member-Collection 815 | 8815 Entity-Destination 816 | 8816 Content-Container 817 | 8817 Member-Collection 818 | 8818 Other 819 | 8819 Component-Whole 820 | 8820 Component-Whole 821 | 8821 Product-Producer 822 | 8822 Member-Collection 823 | 8823 Instrument-Agency 824 | 8824 Member-Collection 825 | 8825 Entity-Destination 826 | 8826 Message-Topic 827 | 8827 Entity-Destination 828 | 8828 Product-Producer 829 | 8829 Cause-Effect 830 | 8830 Message-Topic 831 | 8831 Component-Whole 832 | 8832 Entity-Origin 833 | 8833 Content-Container 834 | 8834 Entity-Origin 835 | 8835 Instrument-Agency 836 | 8836 Entity-Origin 837 | 8837 Component-Whole 838 | 8838 Instrument-Agency 839 | 8839 Member-Collection 840 | 8840 Product-Producer 841 | 8841 Cause-Effect 842 | 8842 Other 843 | 8843 Content-Container 844 | 8844 Message-Topic 845 | 8845 Other 846 | 8846 Entity-Destination 847 | 8847 Other 848 | 8848 Message-Topic 849 | 8849 Entity-Destination 850 | 8850 Entity-Destination 851 | 8851 Cause-Effect 852 | 8852 Content-Container 853 | 8853 Entity-Origin 854 | 8854 Member-Collection 855 | 8855 Cause-Effect 856 | 8856 Content-Container 857 | 8857 Cause-Effect 858 | 8858 Cause-Effect 859 | 8859 Cause-Effect 860 | 8860 Other 861 | 8861 Message-Topic 862 | 8862 Entity-Destination 863 | 8863 Other 864 | 8864 Component-Whole 865 | 8865 Component-Whole 866 | 8866 Other 867 | 8867 Entity-Destination 868 | 8868 Component-Whole 869 | 8869 Product-Producer 870 | 8870 Entity-Destination 871 | 8871 Member-Collection 872 | 8872 Instrument-Agency 873 | 8873 Component-Whole 874 | 8874 Other 875 | 8875 Cause-Effect 876 | 8876 Other 877 | 8877 Member-Collection 878 | 8878 Entity-Origin 879 | 8879 Cause-Effect 880 | 8880 Entity-Origin 881 | 8881 Content-Container 882 | 8882 Entity-Origin 883 | 8883 Product-Producer 884 | 8884 Component-Whole 885 | 8885 Cause-Effect 886 | 8886 Entity-Origin 887 | 8887 Message-Topic 888 | 8888 Other 889 | 8889 Cause-Effect 890 | 8890 Entity-Origin 891 | 8891 Content-Container 892 | 8892 Product-Producer 893 | 8893 Component-Whole 894 | 8894 Entity-Origin 895 | 8895 Product-Producer 896 | 8896 Other 897 | 8897 Member-Collection 898 | 8898 Entity-Destination 899 | 8899 Entity-Origin 900 | 8900 Message-Topic 901 | 8901 Message-Topic 902 | 8902 Member-Collection 903 | 8903 Entity-Destination 904 | 8904 Instrument-Agency 905 | 8905 Other 906 | 8906 Member-Collection 907 | 8907 Entity-Origin 908 | 8908 Message-Topic 909 | 8909 Other 910 | 8910 Other 911 | 8911 Member-Collection 912 | 8912 Message-Topic 913 | 8913 Product-Producer 914 | 8914 Cause-Effect 915 | 8915 Component-Whole 916 | 8916 Product-Producer 917 | 8917 Other 918 | 8918 Instrument-Agency 919 | 8919 Message-Topic 920 | 8920 Product-Producer 921 | 8921 Entity-Origin 922 | 8922 Product-Producer 923 | 8923 Component-Whole 924 | 8924 Product-Producer 925 | 8925 Other 926 | 8926 Component-Whole 927 | 8927 Product-Producer 928 | 8928 Component-Whole 929 | 8929 Component-Whole 930 | 8930 Entity-Destination 931 | 8931 Other 932 | 8932 Component-Whole 933 | 8933 Other 934 | 8934 Member-Collection 935 | 8935 Component-Whole 936 | 8936 Component-Whole 937 | 8937 Cause-Effect 938 | 8938 Content-Container 939 | 8939 Entity-Destination 940 | 8940 Cause-Effect 941 | 8941 Component-Whole 942 | 8942 Other 943 | 8943 Product-Producer 944 | 8944 Member-Collection 945 | 8945 Other 946 | 8946 Entity-Destination 947 | 8947 Instrument-Agency 948 | 8948 Message-Topic 949 | 8949 Cause-Effect 950 | 8950 Content-Container 951 | 8951 Component-Whole 952 | 8952 Member-Collection 953 | 8953 Cause-Effect 954 | 8954 Cause-Effect 955 | 8955 Product-Producer 956 | 8956 Other 957 | 8957 Member-Collection 958 | 8958 Instrument-Agency 959 | 8959 Component-Whole 960 | 8960 Entity-Destination 961 | 8961 Other 962 | 8962 Component-Whole 963 | 8963 Content-Container 964 | 8964 Other 965 | 8965 Member-Collection 966 | 8966 Member-Collection 967 | 8967 Other 968 | 8968 Entity-Destination 969 | 8969 Product-Producer 970 | 8970 Instrument-Agency 971 | 8971 Product-Producer 972 | 8972 Cause-Effect 973 | 8973 Entity-Destination 974 | 8974 Cause-Effect 975 | 8975 Message-Topic 976 | 8976 Product-Producer 977 | 8977 Instrument-Agency 978 | 8978 Entity-Destination 979 | 8979 Message-Topic 980 | 8980 Message-Topic 981 | 8981 Instrument-Agency 982 | 8982 Instrument-Agency 983 | 8983 Entity-Destination 984 | 8984 Component-Whole 985 | 8985 Message-Topic 986 | 8986 Member-Collection 987 | 8987 Cause-Effect 988 | 8988 Product-Producer 989 | 8989 Cause-Effect 990 | 8990 Entity-Destination 991 | 8991 Other 992 | 8992 Cause-Effect 993 | 8993 Message-Topic 994 | 8994 Message-Topic 995 | 8995 Other 996 | 8996 Content-Container 997 | 8997 Instrument-Agency 998 | 8998 Member-Collection 999 | 8999 Message-Topic 1000 | 9000 Content-Container 1001 | 9001 Content-Container 1002 | 9002 Other 1003 | 9003 Component-Whole 1004 | 9004 Content-Container 1005 | 9005 Cause-Effect 1006 | 9006 Component-Whole 1007 | 9007 Content-Container 1008 | 9008 Member-Collection 1009 | 9009 Other 1010 | 9010 Content-Container 1011 | 9011 Product-Producer 1012 | 9012 Cause-Effect 1013 | 9013 Component-Whole 1014 | 9014 Cause-Effect 1015 | 9015 Cause-Effect 1016 | 9016 Entity-Destination 1017 | 9017 Entity-Origin 1018 | 9018 Cause-Effect 1019 | 9019 Other 1020 | 9020 Other 1021 | 9021 Member-Collection 1022 | 9022 Other 1023 | 9023 Content-Container 1024 | 9024 Content-Container 1025 | 9025 Cause-Effect 1026 | 9026 Entity-Origin 1027 | 9027 Entity-Origin 1028 | 9028 Other 1029 | 9029 Component-Whole 1030 | 9030 Message-Topic 1031 | 9031 Product-Producer 1032 | 9032 Member-Collection 1033 | 9033 Product-Producer 1034 | 9034 Other 1035 | 9035 Content-Container 1036 | 9036 Instrument-Agency 1037 | 9037 Entity-Destination 1038 | 9038 Entity-Destination 1039 | 9039 Entity-Destination 1040 | 9040 Component-Whole 1041 | 9041 Entity-Origin 1042 | 9042 Instrument-Agency 1043 | 9043 Content-Container 1044 | 9044 Content-Container 1045 | 9045 Content-Container 1046 | 9046 Product-Producer 1047 | 9047 Product-Producer 1048 | 9048 Entity-Destination 1049 | 9049 Product-Producer 1050 | 9050 Message-Topic 1051 | 9051 Entity-Origin 1052 | 9052 Product-Producer 1053 | 9053 Other 1054 | 9054 Other 1055 | 9055 Cause-Effect 1056 | 9056 Product-Producer 1057 | 9057 Cause-Effect 1058 | 9058 Product-Producer 1059 | 9059 Message-Topic 1060 | 9060 Entity-Destination 1061 | 9061 Entity-Destination 1062 | 9062 Cause-Effect 1063 | 9063 Message-Topic 1064 | 9064 Member-Collection 1065 | 9065 Entity-Origin 1066 | 9066 Member-Collection 1067 | 9067 Entity-Origin 1068 | 9068 Cause-Effect 1069 | 9069 Member-Collection 1070 | 9070 Entity-Destination 1071 | 9071 Product-Producer 1072 | 9072 Other 1073 | 9073 Cause-Effect 1074 | 9074 Other 1075 | 9075 Member-Collection 1076 | 9076 Entity-Destination 1077 | 9077 Other 1078 | 9078 Entity-Destination 1079 | 9079 Member-Collection 1080 | 9080 Other 1081 | 9081 Cause-Effect 1082 | 9082 Content-Container 1083 | 9083 Cause-Effect 1084 | 9084 Member-Collection 1085 | 9085 Product-Producer 1086 | 9086 Component-Whole 1087 | 9087 Cause-Effect 1088 | 9088 Other 1089 | 9089 Product-Producer 1090 | 9090 Component-Whole 1091 | 9091 Entity-Destination 1092 | 9092 Entity-Origin 1093 | 9093 Other 1094 | 9094 Other 1095 | 9095 Message-Topic 1096 | 9096 Product-Producer 1097 | 9097 Instrument-Agency 1098 | 9098 Product-Producer 1099 | 9099 Other 1100 | 9100 Component-Whole 1101 | 9101 Other 1102 | 9102 Member-Collection 1103 | 9103 Content-Container 1104 | 9104 Member-Collection 1105 | 9105 Component-Whole 1106 | 9106 Entity-Destination 1107 | 9107 Entity-Destination 1108 | 9108 Message-Topic 1109 | 9109 Message-Topic 1110 | 9110 Cause-Effect 1111 | 9111 Cause-Effect 1112 | 9112 Content-Container 1113 | 9113 Component-Whole 1114 | 9114 Product-Producer 1115 | 9115 Entity-Origin 1116 | 9116 Instrument-Agency 1117 | 9117 Entity-Destination 1118 | 9118 Entity-Destination 1119 | 9119 Cause-Effect 1120 | 9120 Product-Producer 1121 | 9121 Product-Producer 1122 | 9122 Entity-Destination 1123 | 9123 Entity-Origin 1124 | 9124 Instrument-Agency 1125 | 9125 Entity-Origin 1126 | 9126 Member-Collection 1127 | 9127 Entity-Origin 1128 | 9128 Cause-Effect 1129 | 9129 Content-Container 1130 | 9130 Other 1131 | 9131 Cause-Effect 1132 | 9132 Instrument-Agency 1133 | 9133 Instrument-Agency 1134 | 9134 Component-Whole 1135 | 9135 Instrument-Agency 1136 | 9136 Cause-Effect 1137 | 9137 Other 1138 | 9138 Component-Whole 1139 | 9139 Cause-Effect 1140 | 9140 Entity-Destination 1141 | 9141 Message-Topic 1142 | 9142 Entity-Destination 1143 | 9143 Member-Collection 1144 | 9144 Product-Producer 1145 | 9145 Message-Topic 1146 | 9146 Cause-Effect 1147 | 9147 Cause-Effect 1148 | 9148 Cause-Effect 1149 | 9149 Other 1150 | 9150 Entity-Origin 1151 | 9151 Product-Producer 1152 | 9152 Component-Whole 1153 | 9153 Content-Container 1154 | 9154 Other 1155 | 9155 Entity-Origin 1156 | 9156 Other 1157 | 9157 Other 1158 | 9158 Content-Container 1159 | 9159 Content-Container 1160 | 9160 Member-Collection 1161 | 9161 Cause-Effect 1162 | 9162 Entity-Destination 1163 | 9163 Cause-Effect 1164 | 9164 Other 1165 | 9165 Message-Topic 1166 | 9166 Component-Whole 1167 | 9167 Cause-Effect 1168 | 9168 Cause-Effect 1169 | 9169 Message-Topic 1170 | 9170 Other 1171 | 9171 Cause-Effect 1172 | 9172 Cause-Effect 1173 | 9173 Entity-Origin 1174 | 9174 Component-Whole 1175 | 9175 Entity-Origin 1176 | 9176 Entity-Origin 1177 | 9177 Product-Producer 1178 | 9178 Entity-Origin 1179 | 9179 Cause-Effect 1180 | 9180 Entity-Origin 1181 | 9181 Cause-Effect 1182 | 9182 Entity-Origin 1183 | 9183 Component-Whole 1184 | 9184 Content-Container 1185 | 9185 Component-Whole 1186 | 9186 Message-Topic 1187 | 9187 Other 1188 | 9188 Entity-Origin 1189 | 9189 Entity-Destination 1190 | 9190 Cause-Effect 1191 | 9191 Message-Topic 1192 | 9192 Other 1193 | 9193 Other 1194 | 9194 Member-Collection 1195 | 9195 Instrument-Agency 1196 | 9196 Content-Container 1197 | 9197 Entity-Destination 1198 | 9198 Member-Collection 1199 | 9199 Message-Topic 1200 | 9200 Entity-Destination 1201 | 9201 Entity-Origin 1202 | 9202 Message-Topic 1203 | 9203 Component-Whole 1204 | 9204 Entity-Origin 1205 | 9205 Instrument-Agency 1206 | 9206 Entity-Origin 1207 | 9207 Component-Whole 1208 | 9208 Other 1209 | 9209 Entity-Origin 1210 | 9210 Component-Whole 1211 | 9211 Member-Collection 1212 | 9212 Content-Container 1213 | 9213 Cause-Effect 1214 | 9214 Component-Whole 1215 | 9215 Instrument-Agency 1216 | 9216 Member-Collection 1217 | 9217 Other 1218 | 9218 Entity-Destination 1219 | 9219 Other 1220 | 9220 Entity-Origin 1221 | 9221 Cause-Effect 1222 | 9222 Entity-Destination 1223 | 9223 Product-Producer 1224 | 9224 Cause-Effect 1225 | 9225 Entity-Origin 1226 | 9226 Cause-Effect 1227 | 9227 Other 1228 | 9228 Cause-Effect 1229 | 9229 Member-Collection 1230 | 9230 Component-Whole 1231 | 9231 Entity-Destination 1232 | 9232 Other 1233 | 9233 Member-Collection 1234 | 9234 Cause-Effect 1235 | 9235 Other 1236 | 9236 Entity-Origin 1237 | 9237 Component-Whole 1238 | 9238 Component-Whole 1239 | 9239 Product-Producer 1240 | 9240 Entity-Origin 1241 | 9241 Component-Whole 1242 | 9242 Member-Collection 1243 | 9243 Content-Container 1244 | 9244 Entity-Destination 1245 | 9245 Other 1246 | 9246 Other 1247 | 9247 Entity-Destination 1248 | 9248 Other 1249 | 9249 Other 1250 | 9250 Component-Whole 1251 | 9251 Other 1252 | 9252 Other 1253 | 9253 Product-Producer 1254 | 9254 Member-Collection 1255 | 9255 Content-Container 1256 | 9256 Other 1257 | 9257 Component-Whole 1258 | 9258 Message-Topic 1259 | 9259 Cause-Effect 1260 | 9260 Content-Container 1261 | 9261 Message-Topic 1262 | 9262 Member-Collection 1263 | 9263 Member-Collection 1264 | 9264 Component-Whole 1265 | 9265 Component-Whole 1266 | 9266 Entity-Origin 1267 | 9267 Component-Whole 1268 | 9268 Member-Collection 1269 | 9269 Message-Topic 1270 | 9270 Instrument-Agency 1271 | 9271 Entity-Origin 1272 | 9272 Component-Whole 1273 | 9273 Content-Container 1274 | 9274 Entity-Origin 1275 | 9275 Entity-Destination 1276 | 9276 Component-Whole 1277 | 9277 Product-Producer 1278 | 9278 Entity-Origin 1279 | 9279 Entity-Origin 1280 | 9280 Cause-Effect 1281 | 9281 Other 1282 | 9282 Member-Collection 1283 | 9283 Other 1284 | 9284 Instrument-Agency 1285 | 9285 Content-Container 1286 | 9286 Member-Collection 1287 | 9287 Entity-Origin 1288 | 9288 Component-Whole 1289 | 9289 Cause-Effect 1290 | 9290 Message-Topic 1291 | 9291 Instrument-Agency 1292 | 9292 Content-Container 1293 | 9293 Component-Whole 1294 | 9294 Member-Collection 1295 | 9295 Entity-Destination 1296 | 9296 Entity-Origin 1297 | 9297 Entity-Destination 1298 | 9298 Message-Topic 1299 | 9299 Entity-Origin 1300 | 9300 Entity-Destination 1301 | 9301 Other 1302 | 9302 Component-Whole 1303 | 9303 Member-Collection 1304 | 9304 Message-Topic 1305 | 9305 Entity-Origin 1306 | 9306 Entity-Destination 1307 | 9307 Product-Producer 1308 | 9308 Instrument-Agency 1309 | 9309 Cause-Effect 1310 | 9310 Other 1311 | 9311 Cause-Effect 1312 | 9312 Other 1313 | 9313 Component-Whole 1314 | 9314 Content-Container 1315 | 9315 Message-Topic 1316 | 9316 Component-Whole 1317 | 9317 Instrument-Agency 1318 | 9318 Entity-Destination 1319 | 9319 Cause-Effect 1320 | 9320 Other 1321 | 9321 Message-Topic 1322 | 9322 Product-Producer 1323 | 9323 Cause-Effect 1324 | 9324 Content-Container 1325 | 9325 Member-Collection 1326 | 9326 Entity-Origin 1327 | 9327 Message-Topic 1328 | 9328 Cause-Effect 1329 | 9329 Component-Whole 1330 | 9330 Product-Producer 1331 | 9331 Instrument-Agency 1332 | 9332 Content-Container 1333 | 9333 Component-Whole 1334 | 9334 Content-Container 1335 | 9335 Entity-Destination 1336 | 9336 Member-Collection 1337 | 9337 Component-Whole 1338 | 9338 Entity-Destination 1339 | 9339 Message-Topic 1340 | 9340 Product-Producer 1341 | 9341 Content-Container 1342 | 9342 Cause-Effect 1343 | 9343 Entity-Origin 1344 | 9344 Member-Collection 1345 | 9345 Content-Container 1346 | 9346 Component-Whole 1347 | 9347 Entity-Origin 1348 | 9348 Product-Producer 1349 | 9349 Instrument-Agency 1350 | 9350 Other 1351 | 9351 Entity-Destination 1352 | 9352 Cause-Effect 1353 | 9353 Instrument-Agency 1354 | 9354 Member-Collection 1355 | 9355 Member-Collection 1356 | 9356 Instrument-Agency 1357 | 9357 Cause-Effect 1358 | 9358 Other 1359 | 9359 Entity-Origin 1360 | 9360 Entity-Destination 1361 | 9361 Component-Whole 1362 | 9362 Member-Collection 1363 | 9363 Message-Topic 1364 | 9364 Message-Topic 1365 | 9365 Content-Container 1366 | 9366 Cause-Effect 1367 | 9367 Product-Producer 1368 | 9368 Member-Collection 1369 | 9369 Other 1370 | 9370 Cause-Effect 1371 | 9371 Message-Topic 1372 | 9372 Cause-Effect 1373 | 9373 Cause-Effect 1374 | 9374 Entity-Destination 1375 | 9375 Entity-Destination 1376 | 9376 Component-Whole 1377 | 9377 Instrument-Agency 1378 | 9378 Cause-Effect 1379 | 9379 Cause-Effect 1380 | 9380 Entity-Destination 1381 | 9381 Message-Topic 1382 | 9382 Component-Whole 1383 | 9383 Entity-Origin 1384 | 9384 Instrument-Agency 1385 | 9385 Content-Container 1386 | 9386 Other 1387 | 9387 Component-Whole 1388 | 9388 Other 1389 | 9389 Entity-Destination 1390 | 9390 Entity-Origin 1391 | 9391 Component-Whole 1392 | 9392 Other 1393 | 9393 Component-Whole 1394 | 9394 Cause-Effect 1395 | 9395 Entity-Origin 1396 | 9396 Other 1397 | 9397 Other 1398 | 9398 Instrument-Agency 1399 | 9399 Entity-Destination 1400 | 9400 Other 1401 | 9401 Message-Topic 1402 | 9402 Other 1403 | 9403 Cause-Effect 1404 | 9404 Component-Whole 1405 | 9405 Component-Whole 1406 | 9406 Other 1407 | 9407 Content-Container 1408 | 9408 Other 1409 | 9409 Instrument-Agency 1410 | 9410 Message-Topic 1411 | 9411 Component-Whole 1412 | 9412 Member-Collection 1413 | 9413 Instrument-Agency 1414 | 9414 Other 1415 | 9415 Cause-Effect 1416 | 9416 Entity-Destination 1417 | 9417 Other 1418 | 9418 Other 1419 | 9419 Component-Whole 1420 | 9420 Component-Whole 1421 | 9421 Entity-Origin 1422 | 9422 Product-Producer 1423 | 9423 Member-Collection 1424 | 9424 Other 1425 | 9425 Message-Topic 1426 | 9426 Entity-Destination 1427 | 9427 Cause-Effect 1428 | 9428 Product-Producer 1429 | 9429 Entity-Destination 1430 | 9430 Message-Topic 1431 | 9431 Other 1432 | 9432 Message-Topic 1433 | 9433 Member-Collection 1434 | 9434 Cause-Effect 1435 | 9435 Instrument-Agency 1436 | 9436 Content-Container 1437 | 9437 Entity-Destination 1438 | 9438 Cause-Effect 1439 | 9439 Other 1440 | 9440 Entity-Origin 1441 | 9441 Component-Whole 1442 | 9442 Message-Topic 1443 | 9443 Instrument-Agency 1444 | 9444 Other 1445 | 9445 Component-Whole 1446 | 9446 Member-Collection 1447 | 9447 Content-Container 1448 | 9448 Component-Whole 1449 | 9449 Component-Whole 1450 | 9450 Product-Producer 1451 | 9451 Member-Collection 1452 | 9452 Cause-Effect 1453 | 9453 Entity-Origin 1454 | 9454 Entity-Origin 1455 | 9455 Member-Collection 1456 | 9456 Message-Topic 1457 | 9457 Instrument-Agency 1458 | 9458 Product-Producer 1459 | 9459 Other 1460 | 9460 Entity-Origin 1461 | 9461 Other 1462 | 9462 Member-Collection 1463 | 9463 Entity-Origin 1464 | 9464 Cause-Effect 1465 | 9465 Other 1466 | 9466 Product-Producer 1467 | 9467 Cause-Effect 1468 | 9468 Member-Collection 1469 | 9469 Cause-Effect 1470 | 9470 Message-Topic 1471 | 9471 Content-Container 1472 | 9472 Entity-Destination 1473 | 9473 Entity-Origin 1474 | 9474 Member-Collection 1475 | 9475 Content-Container 1476 | 9476 Message-Topic 1477 | 9477 Instrument-Agency 1478 | 9478 Member-Collection 1479 | 9479 Component-Whole 1480 | 9480 Other 1481 | 9481 Product-Producer 1482 | 9482 Cause-Effect 1483 | 9483 Content-Container 1484 | 9484 Component-Whole 1485 | 9485 Component-Whole 1486 | 9486 Instrument-Agency 1487 | 9487 Instrument-Agency 1488 | 9488 Instrument-Agency 1489 | 9489 Cause-Effect 1490 | 9490 Cause-Effect 1491 | 9491 Instrument-Agency 1492 | 9492 Other 1493 | 9493 Entity-Origin 1494 | 9494 Cause-Effect 1495 | 9495 Message-Topic 1496 | 9496 Content-Container 1497 | 9497 Component-Whole 1498 | 9498 Message-Topic 1499 | 9499 Message-Topic 1500 | 9500 Content-Container 1501 | 9501 Content-Container 1502 | 9502 Entity-Origin 1503 | 9503 Other 1504 | 9504 Message-Topic 1505 | 9505 Other 1506 | 9506 Entity-Destination 1507 | 9507 Other 1508 | 9508 Cause-Effect 1509 | 9509 Member-Collection 1510 | 9510 Other 1511 | 9511 Other 1512 | 9512 Instrument-Agency 1513 | 9513 Product-Producer 1514 | 9514 Entity-Origin 1515 | 9515 Cause-Effect 1516 | 9516 Other 1517 | 9517 Other 1518 | 9518 Member-Collection 1519 | 9519 Cause-Effect 1520 | 9520 Other 1521 | 9521 Message-Topic 1522 | 9522 Content-Container 1523 | 9523 Other 1524 | 9524 Cause-Effect 1525 | 9525 Message-Topic 1526 | 9526 Message-Topic 1527 | 9527 Component-Whole 1528 | 9528 Content-Container 1529 | 9529 Entity-Origin 1530 | 9530 Member-Collection 1531 | 9531 Entity-Destination 1532 | 9532 Other 1533 | 9533 Entity-Destination 1534 | 9534 Content-Container 1535 | 9535 Component-Whole 1536 | 9536 Message-Topic 1537 | 9537 Other 1538 | 9538 Message-Topic 1539 | 9539 Entity-Destination 1540 | 9540 Component-Whole 1541 | 9541 Other 1542 | 9542 Cause-Effect 1543 | 9543 Message-Topic 1544 | 9544 Entity-Destination 1545 | 9545 Other 1546 | 9546 Other 1547 | 9547 Component-Whole 1548 | 9548 Entity-Origin 1549 | 9549 Other 1550 | 9550 Member-Collection 1551 | 9551 Instrument-Agency 1552 | 9552 Other 1553 | 9553 Product-Producer 1554 | 9554 Entity-Destination 1555 | 9555 Instrument-Agency 1556 | 9556 Cause-Effect 1557 | 9557 Component-Whole 1558 | 9558 Other 1559 | 9559 Cause-Effect 1560 | 9560 Entity-Origin 1561 | 9561 Component-Whole 1562 | 9562 Component-Whole 1563 | 9563 Entity-Destination 1564 | 9564 Message-Topic 1565 | 9565 Component-Whole 1566 | 9566 Message-Topic 1567 | 9567 Message-Topic 1568 | 9568 Entity-Destination 1569 | 9569 Other 1570 | 9570 Member-Collection 1571 | 9571 Entity-Origin 1572 | 9572 Instrument-Agency 1573 | 9573 Cause-Effect 1574 | 9574 Other 1575 | 9575 Instrument-Agency 1576 | 9576 Cause-Effect 1577 | 9577 Other 1578 | 9578 Entity-Destination 1579 | 9579 Component-Whole 1580 | 9580 Component-Whole 1581 | 9581 Entity-Destination 1582 | 9582 Cause-Effect 1583 | 9583 Component-Whole 1584 | 9584 Member-Collection 1585 | 9585 Entity-Destination 1586 | 9586 Entity-Destination 1587 | 9587 Product-Producer 1588 | 9588 Other 1589 | 9589 Cause-Effect 1590 | 9590 Instrument-Agency 1591 | 9591 Entity-Origin 1592 | 9592 Member-Collection 1593 | 9593 Entity-Destination 1594 | 9594 Instrument-Agency 1595 | 9595 Member-Collection 1596 | 9596 Message-Topic 1597 | 9597 Cause-Effect 1598 | 9598 Entity-Destination 1599 | 9599 Other 1600 | 9600 Component-Whole 1601 | 9601 Cause-Effect 1602 | 9602 Member-Collection 1603 | 9603 Component-Whole 1604 | 9604 Content-Container 1605 | 9605 Instrument-Agency 1606 | 9606 Other 1607 | 9607 Other 1608 | 9608 Member-Collection 1609 | 9609 Content-Container 1610 | 9610 Other 1611 | 9611 Entity-Origin 1612 | 9612 Component-Whole 1613 | 9613 Component-Whole 1614 | 9614 Member-Collection 1615 | 9615 Message-Topic 1616 | 9616 Other 1617 | 9617 Component-Whole 1618 | 9618 Cause-Effect 1619 | 9619 Instrument-Agency 1620 | 9620 Member-Collection 1621 | 9621 Entity-Destination 1622 | 9622 Message-Topic 1623 | 9623 Other 1624 | 9624 Cause-Effect 1625 | 9625 Component-Whole 1626 | 9626 Entity-Origin 1627 | 9627 Other 1628 | 9628 Instrument-Agency 1629 | 9629 Message-Topic 1630 | 9630 Other 1631 | 9631 Other 1632 | 9632 Component-Whole 1633 | 9633 Entity-Destination 1634 | 9634 Component-Whole 1635 | 9635 Content-Container 1636 | 9636 Component-Whole 1637 | 9637 Entity-Destination 1638 | 9638 Other 1639 | 9639 Other 1640 | 9640 Content-Container 1641 | 9641 Other 1642 | 9642 Other 1643 | 9643 Other 1644 | 9644 Product-Producer 1645 | 9645 Content-Container 1646 | 9646 Other 1647 | 9647 Cause-Effect 1648 | 9648 Cause-Effect 1649 | 9649 Instrument-Agency 1650 | 9650 Other 1651 | 9651 Member-Collection 1652 | 9652 Other 1653 | 9653 Message-Topic 1654 | 9654 Instrument-Agency 1655 | 9655 Entity-Destination 1656 | 9656 Entity-Origin 1657 | 9657 Entity-Origin 1658 | 9658 Other 1659 | 9659 Cause-Effect 1660 | 9660 Member-Collection 1661 | 9661 Message-Topic 1662 | 9662 Content-Container 1663 | 9663 Other 1664 | 9664 Member-Collection 1665 | 9665 Entity-Destination 1666 | 9666 Component-Whole 1667 | 9667 Product-Producer 1668 | 9668 Component-Whole 1669 | 9669 Entity-Origin 1670 | 9670 Entity-Origin 1671 | 9671 Component-Whole 1672 | 9672 Component-Whole 1673 | 9673 Cause-Effect 1674 | 9674 Other 1675 | 9675 Message-Topic 1676 | 9676 Entity-Destination 1677 | 9677 Product-Producer 1678 | 9678 Member-Collection 1679 | 9679 Component-Whole 1680 | 9680 Other 1681 | 9681 Member-Collection 1682 | 9682 Cause-Effect 1683 | 9683 Entity-Destination 1684 | 9684 Cause-Effect 1685 | 9685 Component-Whole 1686 | 9686 Other 1687 | 9687 Instrument-Agency 1688 | 9688 Cause-Effect 1689 | 9689 Cause-Effect 1690 | 9690 Cause-Effect 1691 | 9691 Message-Topic 1692 | 9692 Product-Producer 1693 | 9693 Entity-Origin 1694 | 9694 Content-Container 1695 | 9695 Cause-Effect 1696 | 9696 Instrument-Agency 1697 | 9697 Component-Whole 1698 | 9698 Cause-Effect 1699 | 9699 Cause-Effect 1700 | 9700 Other 1701 | 9701 Other 1702 | 9702 Member-Collection 1703 | 9703 Cause-Effect 1704 | 9704 Instrument-Agency 1705 | 9705 Message-Topic 1706 | 9706 Other 1707 | 9707 Component-Whole 1708 | 9708 Cause-Effect 1709 | 9709 Member-Collection 1710 | 9710 Entity-Origin 1711 | 9711 Entity-Origin 1712 | 9712 Product-Producer 1713 | 9713 Component-Whole 1714 | 9714 Other 1715 | 9715 Component-Whole 1716 | 9716 Product-Producer 1717 | 9717 Member-Collection 1718 | 9718 Other 1719 | 9719 Cause-Effect 1720 | 9720 Cause-Effect 1721 | 9721 Instrument-Agency 1722 | 9722 Cause-Effect 1723 | 9723 Component-Whole 1724 | 9724 Entity-Origin 1725 | 9725 Cause-Effect 1726 | 9726 Other 1727 | 9727 Cause-Effect 1728 | 9728 Message-Topic 1729 | 9729 Instrument-Agency 1730 | 9730 Message-Topic 1731 | 9731 Cause-Effect 1732 | 9732 Cause-Effect 1733 | 9733 Entity-Origin 1734 | 9734 Message-Topic 1735 | 9735 Message-Topic 1736 | 9736 Component-Whole 1737 | 9737 Component-Whole 1738 | 9738 Other 1739 | 9739 Cause-Effect 1740 | 9740 Cause-Effect 1741 | 9741 Other 1742 | 9742 Message-Topic 1743 | 9743 Component-Whole 1744 | 9744 Message-Topic 1745 | 9745 Member-Collection 1746 | 9746 Member-Collection 1747 | 9747 Other 1748 | 9748 Member-Collection 1749 | 9749 Message-Topic 1750 | 9750 Other 1751 | 9751 Member-Collection 1752 | 9752 Entity-Origin 1753 | 9753 Member-Collection 1754 | 9754 Content-Container 1755 | 9755 Entity-Origin 1756 | 9756 Message-Topic 1757 | 9757 Message-Topic 1758 | 9758 Cause-Effect 1759 | 9759 Component-Whole 1760 | 9760 Content-Container 1761 | 9761 Other 1762 | 9762 Other 1763 | 9763 Component-Whole 1764 | 9764 Message-Topic 1765 | 9765 Member-Collection 1766 | 9766 Entity-Destination 1767 | 9767 Entity-Origin 1768 | 9768 Instrument-Agency 1769 | 9769 Cause-Effect 1770 | 9770 Entity-Origin 1771 | 9771 Component-Whole 1772 | 9772 Entity-Destination 1773 | 9773 Component-Whole 1774 | 9774 Entity-Destination 1775 | 9775 Entity-Destination 1776 | 9776 Other 1777 | 9777 Message-Topic 1778 | 9778 Product-Producer 1779 | 9779 Other 1780 | 9780 Cause-Effect 1781 | 9781 Entity-Destination 1782 | 9782 Other 1783 | 9783 Entity-Origin 1784 | 9784 Product-Producer 1785 | 9785 Product-Producer 1786 | 9786 Content-Container 1787 | 9787 Entity-Destination 1788 | 9788 Message-Topic 1789 | 9789 Other 1790 | 9790 Member-Collection 1791 | 9791 Cause-Effect 1792 | 9792 Entity-Destination 1793 | 9793 Other 1794 | 9794 Component-Whole 1795 | 9795 Other 1796 | 9796 Other 1797 | 9797 Cause-Effect 1798 | 9798 Entity-Destination 1799 | 9799 Other 1800 | 9800 Message-Topic 1801 | 9801 Entity-Origin 1802 | 9802 Entity-Destination 1803 | 9803 Other 1804 | 9804 Member-Collection 1805 | 9805 Entity-Destination 1806 | 9806 Content-Container 1807 | 9807 Entity-Origin 1808 | 9808 Other 1809 | 9809 Entity-Destination 1810 | 9810 Content-Container 1811 | 9811 Other 1812 | 9812 Cause-Effect 1813 | 9813 Instrument-Agency 1814 | 9814 Member-Collection 1815 | 9815 Other 1816 | 9816 Instrument-Agency 1817 | 9817 Message-Topic 1818 | 9818 Member-Collection 1819 | 9819 Cause-Effect 1820 | 9820 Other 1821 | 9821 Product-Producer 1822 | 9822 Product-Producer 1823 | 9823 Entity-Origin 1824 | 9824 Instrument-Agency 1825 | 9825 Member-Collection 1826 | 9826 Member-Collection 1827 | 9827 Member-Collection 1828 | 9828 Product-Producer 1829 | 9829 Cause-Effect 1830 | 9830 Entity-Origin 1831 | 9831 Cause-Effect 1832 | 9832 Entity-Origin 1833 | 9833 Other 1834 | 9834 Component-Whole 1835 | 9835 Content-Container 1836 | 9836 Product-Producer 1837 | 9837 Instrument-Agency 1838 | 9838 Member-Collection 1839 | 9839 Other 1840 | 9840 Message-Topic 1841 | 9841 Member-Collection 1842 | 9842 Other 1843 | 9843 Other 1844 | 9844 Entity-Origin 1845 | 9845 Component-Whole 1846 | 9846 Product-Producer 1847 | 9847 Other 1848 | 9848 Cause-Effect 1849 | 9849 Other 1850 | 9850 Product-Producer 1851 | 9851 Member-Collection 1852 | 9852 Entity-Origin 1853 | 9853 Other 1854 | 9854 Member-Collection 1855 | 9855 Entity-Destination 1856 | 9856 Content-Container 1857 | 9857 Component-Whole 1858 | 9858 Product-Producer 1859 | 9859 Content-Container 1860 | 9860 Entity-Origin 1861 | 9861 Cause-Effect 1862 | 9862 Entity-Origin 1863 | 9863 Product-Producer 1864 | 9864 Product-Producer 1865 | 9865 Entity-Destination 1866 | 9866 Member-Collection 1867 | 9867 Other 1868 | 9868 Cause-Effect 1869 | 9869 Other 1870 | 9870 Product-Producer 1871 | 9871 Entity-Destination 1872 | 9872 Other 1873 | 9873 Entity-Destination 1874 | 9874 Entity-Destination 1875 | 9875 Member-Collection 1876 | 9876 Cause-Effect 1877 | 9877 Other 1878 | 9878 Member-Collection 1879 | 9879 Other 1880 | 9880 Content-Container 1881 | 9881 Member-Collection 1882 | 9882 Entity-Origin 1883 | 9883 Entity-Destination 1884 | 9884 Instrument-Agency 1885 | 9885 Message-Topic 1886 | 9886 Other 1887 | 9887 Member-Collection 1888 | 9888 Member-Collection 1889 | 9889 Instrument-Agency 1890 | 9890 Member-Collection 1891 | 9891 Member-Collection 1892 | 9892 Other 1893 | 9893 Component-Whole 1894 | 9894 Entity-Destination 1895 | 9895 Product-Producer 1896 | 9896 Content-Container 1897 | 9897 Other 1898 | 9898 Entity-Destination 1899 | 9899 Cause-Effect 1900 | 9900 Entity-Destination 1901 | 9901 Cause-Effect 1902 | 9902 Cause-Effect 1903 | 9903 Other 1904 | 9904 Entity-Origin 1905 | 9905 Other 1906 | 9906 Component-Whole 1907 | 9907 Product-Producer 1908 | 9908 Other 1909 | 9909 Product-Producer 1910 | 9910 Member-Collection 1911 | 9911 Message-Topic 1912 | 9912 Instrument-Agency 1913 | 9913 Content-Container 1914 | 9914 Content-Container 1915 | 9915 Other 1916 | 9916 Other 1917 | 9917 Product-Producer 1918 | 9918 Member-Collection 1919 | 9919 Cause-Effect 1920 | 9920 Product-Producer 1921 | 9921 Component-Whole 1922 | 9922 Entity-Origin 1923 | 9923 Member-Collection 1924 | 9924 Other 1925 | 9925 Component-Whole 1926 | 9926 Product-Producer 1927 | 9927 Component-Whole 1928 | 9928 Component-Whole 1929 | 9929 Content-Container 1930 | 9930 Other 1931 | 9931 Entity-Destination 1932 | 9932 Content-Container 1933 | 9933 Product-Producer 1934 | 9934 Component-Whole 1935 | 9935 Product-Producer 1936 | 9936 Entity-Destination 1937 | 9937 Member-Collection 1938 | 9938 Member-Collection 1939 | 9939 Entity-Destination 1940 | 9940 Content-Container 1941 | 9941 Entity-Destination 1942 | 9942 Content-Container 1943 | 9943 Other 1944 | 9944 Message-Topic 1945 | 9945 Component-Whole 1946 | 9946 Message-Topic 1947 | 9947 Product-Producer 1948 | 9948 Entity-Destination 1949 | 9949 Entity-Origin 1950 | 9950 Other 1951 | 9951 Message-Topic 1952 | 9952 Entity-Destination 1953 | 9953 Entity-Destination 1954 | 9954 Entity-Origin 1955 | 9955 Content-Container 1956 | 9956 Cause-Effect 1957 | 9957 Component-Whole 1958 | 9958 Entity-Origin 1959 | 9959 Instrument-Agency 1960 | 9960 Member-Collection 1961 | 9961 Product-Producer 1962 | 9962 Entity-Origin 1963 | 9963 Entity-Destination 1964 | 9964 Entity-Destination 1965 | 9965 Cause-Effect 1966 | 9966 Other 1967 | 9967 Cause-Effect 1968 | 9968 Message-Topic 1969 | 9969 Entity-Destination 1970 | 9970 Instrument-Agency 1971 | 9971 Component-Whole 1972 | 9972 Component-Whole 1973 | 9973 Message-Topic 1974 | 9974 Cause-Effect 1975 | 9975 Cause-Effect 1976 | 9976 Other 1977 | 9977 Product-Producer 1978 | 9978 Other 1979 | 9979 Cause-Effect 1980 | 9980 Component-Whole 1981 | 9981 Member-Collection 1982 | 9982 Entity-Destination 1983 | 9983 Content-Container 1984 | 9984 Member-Collection 1985 | 9985 Cause-Effect 1986 | 9986 Other 1987 | 9987 Product-Producer 1988 | 9988 Content-Container 1989 | 9989 Other 1990 | 9990 Other 1991 | 9991 Message-Topic 1992 | 9992 Component-Whole 1993 | 9993 Content-Container 1994 | 9994 Component-Whole 1995 | 9995 Other 1996 | 9996 Message-Topic 1997 | 9997 Component-Whole 1998 | 9998 Entity-Origin 1999 | 9999 Entity-Destination 2000 | 10000 Instrument-Agency 2001 | 10001 Instrument-Agency 2002 | 10002 Message-Topic 2003 | 10003 Cause-Effect 2004 | 10004 Entity-Destination 2005 | 10005 Instrument-Agency 2006 | 10006 Member-Collection 2007 | 10007 Entity-Origin 2008 | 10008 Entity-Destination 2009 | 10009 Cause-Effect 2010 | 10010 Entity-Origin 2011 | 10011 Other 2012 | 10012 Cause-Effect 2013 | 10013 Member-Collection 2014 | 10014 Entity-Destination 2015 | 10015 Other 2016 | 10016 Content-Container 2017 | 10017 Entity-Destination 2018 | 10018 Entity-Origin 2019 | 10019 Other 2020 | 10020 Entity-Destination 2021 | 10021 Other 2022 | 10022 Other 2023 | 10023 Message-Topic 2024 | 10024 Message-Topic 2025 | 10025 Other 2026 | 10026 Instrument-Agency 2027 | 10027 Entity-Destination 2028 | 10028 Message-Topic 2029 | 10029 Member-Collection 2030 | 10030 Other 2031 | 10031 Member-Collection 2032 | 10032 Member-Collection 2033 | 10033 Other 2034 | 10034 Content-Container 2035 | 10035 Component-Whole 2036 | 10036 Other 2037 | 10037 Entity-Destination 2038 | 10038 Cause-Effect 2039 | 10039 Entity-Destination 2040 | 10040 Cause-Effect 2041 | 10041 Cause-Effect 2042 | 10042 Message-Topic 2043 | 10043 Entity-Destination 2044 | 10044 Component-Whole 2045 | 10045 Component-Whole 2046 | 10046 Entity-Destination 2047 | 10047 Cause-Effect 2048 | 10048 Instrument-Agency 2049 | 10049 Message-Topic 2050 | 10050 Content-Container 2051 | 10051 Component-Whole 2052 | 10052 Member-Collection 2053 | 10053 Content-Container 2054 | 10054 Cause-Effect 2055 | 10055 Entity-Destination 2056 | 10056 Entity-Destination 2057 | 10057 Instrument-Agency 2058 | 10058 Member-Collection 2059 | 10059 Cause-Effect 2060 | 10060 Other 2061 | 10061 Other 2062 | 10062 Content-Container 2063 | 10063 Component-Whole 2064 | 10064 Cause-Effect 2065 | 10065 Content-Container 2066 | 10066 Other 2067 | 10067 Entity-Origin 2068 | 10068 Entity-Destination 2069 | 10069 Other 2070 | 10070 Component-Whole 2071 | 10071 Entity-Origin 2072 | 10072 Content-Container 2073 | 10073 Other 2074 | 10074 Entity-Origin 2075 | 10075 Entity-Origin 2076 | 10076 Product-Producer 2077 | 10077 Entity-Destination 2078 | 10078 Entity-Destination 2079 | 10079 Product-Producer 2080 | 10080 Entity-Origin 2081 | 10081 Entity-Destination 2082 | 10082 Entity-Origin 2083 | 10083 Component-Whole 2084 | 10084 Entity-Origin 2085 | 10085 Entity-Destination 2086 | 10086 Cause-Effect 2087 | 10087 Entity-Destination 2088 | 10088 Instrument-Agency 2089 | 10089 Product-Producer 2090 | 10090 Cause-Effect 2091 | 10091 Entity-Origin 2092 | 10092 Entity-Origin 2093 | 10093 Other 2094 | 10094 Content-Container 2095 | 10095 Entity-Destination 2096 | 10096 Component-Whole 2097 | 10097 Other 2098 | 10098 Message-Topic 2099 | 10099 Entity-Destination 2100 | 10100 Entity-Destination 2101 | 10101 Entity-Origin 2102 | 10102 Cause-Effect 2103 | 10103 Message-Topic 2104 | 10104 Member-Collection 2105 | 10105 Member-Collection 2106 | 10106 Component-Whole 2107 | 10107 Content-Container 2108 | 10108 Message-Topic 2109 | 10109 Other 2110 | 10110 Message-Topic 2111 | 10111 Other 2112 | 10112 Other 2113 | 10113 Product-Producer 2114 | 10114 Message-Topic 2115 | 10115 Message-Topic 2116 | 10116 Entity-Origin 2117 | 10117 Product-Producer 2118 | 10118 Cause-Effect 2119 | 10119 Member-Collection 2120 | 10120 Component-Whole 2121 | 10121 Entity-Destination 2122 | 10122 Entity-Origin 2123 | 10123 Message-Topic 2124 | 10124 Other 2125 | 10125 Other 2126 | 10126 Member-Collection 2127 | 10127 Other 2128 | 10128 Instrument-Agency 2129 | 10129 Other 2130 | 10130 Other 2131 | 10131 Product-Producer 2132 | 10132 Component-Whole 2133 | 10133 Instrument-Agency 2134 | 10134 Cause-Effect 2135 | 10135 Component-Whole 2136 | 10136 Entity-Origin 2137 | 10137 Message-Topic 2138 | 10138 Entity-Origin 2139 | 10139 Entity-Origin 2140 | 10140 Product-Producer 2141 | 10141 Other 2142 | 10142 Product-Producer 2143 | 10143 Other 2144 | 10144 Instrument-Agency 2145 | 10145 Instrument-Agency 2146 | 10146 Product-Producer 2147 | 10147 Component-Whole 2148 | 10148 Product-Producer 2149 | 10149 Instrument-Agency 2150 | 10150 Component-Whole 2151 | 10151 Product-Producer 2152 | 10152 Instrument-Agency 2153 | 10153 Product-Producer 2154 | 10154 Member-Collection 2155 | 10155 Message-Topic 2156 | 10156 Cause-Effect 2157 | 10157 Component-Whole 2158 | 10158 Entity-Destination 2159 | 10159 Other 2160 | 10160 Other 2161 | 10161 Component-Whole 2162 | 10162 Entity-Origin 2163 | 10163 Entity-Origin 2164 | 10164 Entity-Origin 2165 | 10165 Entity-Destination 2166 | 10166 Component-Whole 2167 | 10167 Entity-Origin 2168 | 10168 Content-Container 2169 | 10169 Member-Collection 2170 | 10170 Entity-Origin 2171 | 10171 Content-Container 2172 | 10172 Message-Topic 2173 | 10173 Other 2174 | 10174 Member-Collection 2175 | 10175 Entity-Destination 2176 | 10176 Product-Producer 2177 | 10177 Cause-Effect 2178 | 10178 Entity-Destination 2179 | 10179 Product-Producer 2180 | 10180 Instrument-Agency 2181 | 10181 Other 2182 | 10182 Cause-Effect 2183 | 10183 Message-Topic 2184 | 10184 Entity-Destination 2185 | 10185 Entity-Origin 2186 | 10186 Other 2187 | 10187 Entity-Destination 2188 | 10188 Other 2189 | 10189 Message-Topic 2190 | 10190 Product-Producer 2191 | 10191 Entity-Destination 2192 | 10192 Product-Producer 2193 | 10193 Component-Whole 2194 | 10194 Entity-Origin 2195 | 10195 Instrument-Agency 2196 | 10196 Other 2197 | 10197 Product-Producer 2198 | 10198 Entity-Origin 2199 | 10199 Entity-Origin 2200 | 10200 Entity-Origin 2201 | 10201 Instrument-Agency 2202 | 10202 Entity-Destination 2203 | 10203 Instrument-Agency 2204 | 10204 Message-Topic 2205 | 10205 Product-Producer 2206 | 10206 Product-Producer 2207 | 10207 Entity-Destination 2208 | 10208 Component-Whole 2209 | 10209 Cause-Effect 2210 | 10210 Component-Whole 2211 | 10211 Message-Topic 2212 | 10212 Component-Whole 2213 | 10213 Other 2214 | 10214 Component-Whole 2215 | 10215 Entity-Origin 2216 | 10216 Message-Topic 2217 | 10217 Other 2218 | 10218 Entity-Origin 2219 | 10219 Content-Container 2220 | 10220 Message-Topic 2221 | 10221 Entity-Origin 2222 | 10222 Entity-Origin 2223 | 10223 Member-Collection 2224 | 10224 Product-Producer 2225 | 10225 Member-Collection 2226 | 10226 Entity-Destination 2227 | 10227 Content-Container 2228 | 10228 Cause-Effect 2229 | 10229 Member-Collection 2230 | 10230 Cause-Effect 2231 | 10231 Entity-Destination 2232 | 10232 Content-Container 2233 | 10233 Other 2234 | 10234 Product-Producer 2235 | 10235 Instrument-Agency 2236 | 10236 Message-Topic 2237 | 10237 Product-Producer 2238 | 10238 Member-Collection 2239 | 10239 Member-Collection 2240 | 10240 Entity-Destination 2241 | 10241 Instrument-Agency 2242 | 10242 Message-Topic 2243 | 10243 Instrument-Agency 2244 | 10244 Other 2245 | 10245 Entity-Destination 2246 | 10246 Cause-Effect 2247 | 10247 Message-Topic 2248 | 10248 Content-Container 2249 | 10249 Instrument-Agency 2250 | 10250 Product-Producer 2251 | 10251 Other 2252 | 10252 Instrument-Agency 2253 | 10253 Message-Topic 2254 | 10254 Cause-Effect 2255 | 10255 Entity-Destination 2256 | 10256 Content-Container 2257 | 10257 Cause-Effect 2258 | 10258 Cause-Effect 2259 | 10259 Message-Topic 2260 | 10260 Entity-Origin 2261 | 10261 Other 2262 | 10262 Other 2263 | 10263 Entity-Destination 2264 | 10264 Component-Whole 2265 | 10265 Message-Topic 2266 | 10266 Product-Producer 2267 | 10267 Cause-Effect 2268 | 10268 Member-Collection 2269 | 10269 Message-Topic 2270 | 10270 Product-Producer 2271 | 10271 Entity-Origin 2272 | 10272 Component-Whole 2273 | 10273 Entity-Origin 2274 | 10274 Component-Whole 2275 | 10275 Cause-Effect 2276 | 10276 Entity-Destination 2277 | 10277 Component-Whole 2278 | 10278 Product-Producer 2279 | 10279 Cause-Effect 2280 | 10280 Entity-Destination 2281 | 10281 Cause-Effect 2282 | 10282 Other 2283 | 10283 Entity-Origin 2284 | 10284 Entity-Destination 2285 | 10285 Cause-Effect 2286 | 10286 Content-Container 2287 | 10287 Content-Container 2288 | 10288 Component-Whole 2289 | 10289 Member-Collection 2290 | 10290 Content-Container 2291 | 10291 Other 2292 | 10292 Message-Topic 2293 | 10293 Entity-Destination 2294 | 10294 Instrument-Agency 2295 | 10295 Message-Topic 2296 | 10296 Cause-Effect 2297 | 10297 Entity-Origin 2298 | 10298 Entity-Origin 2299 | 10299 Entity-Origin 2300 | 10300 Other 2301 | 10301 Member-Collection 2302 | 10302 Message-Topic 2303 | 10303 Entity-Destination 2304 | 10304 Instrument-Agency 2305 | 10305 Component-Whole 2306 | 10306 Component-Whole 2307 | 10307 Component-Whole 2308 | 10308 Other 2309 | 10309 Message-Topic 2310 | 10310 Message-Topic 2311 | 10311 Component-Whole 2312 | 10312 Content-Container 2313 | 10313 Product-Producer 2314 | 10314 Content-Container 2315 | 10315 Component-Whole 2316 | 10316 Content-Container 2317 | 10317 Other 2318 | 10318 Other 2319 | 10319 Member-Collection 2320 | 10320 Instrument-Agency 2321 | 10321 Entity-Destination 2322 | 10322 Component-Whole 2323 | 10323 Other 2324 | 10324 Message-Topic 2325 | 10325 Content-Container 2326 | 10326 Other 2327 | 10327 Content-Container 2328 | 10328 Product-Producer 2329 | 10329 Instrument-Agency 2330 | 10330 Entity-Destination 2331 | 10331 Instrument-Agency 2332 | 10332 Content-Container 2333 | 10333 Other 2334 | 10334 Other 2335 | 10335 Cause-Effect 2336 | 10336 Entity-Origin 2337 | 10337 Content-Container 2338 | 10338 Entity-Origin 2339 | 10339 Other 2340 | 10340 Entity-Origin 2341 | 10341 Other 2342 | 10342 Entity-Destination 2343 | 10343 Instrument-Agency 2344 | 10344 Cause-Effect 2345 | 10345 Component-Whole 2346 | 10346 Instrument-Agency 2347 | 10347 Content-Container 2348 | 10348 Entity-Destination 2349 | 10349 Member-Collection 2350 | 10350 Cause-Effect 2351 | 10351 Entity-Destination 2352 | 10352 Message-Topic 2353 | 10353 Product-Producer 2354 | 10354 Entity-Destination 2355 | 10355 Content-Container 2356 | 10356 Entity-Origin 2357 | 10357 Entity-Origin 2358 | 10358 Component-Whole 2359 | 10359 Other 2360 | 10360 Message-Topic 2361 | 10361 Instrument-Agency 2362 | 10362 Entity-Destination 2363 | 10363 Entity-Destination 2364 | 10364 Product-Producer 2365 | 10365 Message-Topic 2366 | 10366 Member-Collection 2367 | 10367 Product-Producer 2368 | 10368 Instrument-Agency 2369 | 10369 Instrument-Agency 2370 | 10370 Other 2371 | 10371 Product-Producer 2372 | 10372 Product-Producer 2373 | 10373 Cause-Effect 2374 | 10374 Content-Container 2375 | 10375 Member-Collection 2376 | 10376 Entity-Destination 2377 | 10377 Message-Topic 2378 | 10378 Entity-Origin 2379 | 10379 Cause-Effect 2380 | 10380 Component-Whole 2381 | 10381 Message-Topic 2382 | 10382 Cause-Effect 2383 | 10383 Cause-Effect 2384 | 10384 Entity-Origin 2385 | 10385 Instrument-Agency 2386 | 10386 Component-Whole 2387 | 10387 Component-Whole 2388 | 10388 Product-Producer 2389 | 10389 Component-Whole 2390 | 10390 Other 2391 | 10391 Instrument-Agency 2392 | 10392 Message-Topic 2393 | 10393 Entity-Origin 2394 | 10394 Other 2395 | 10395 Message-Topic 2396 | 10396 Cause-Effect 2397 | 10397 Entity-Origin 2398 | 10398 Cause-Effect 2399 | 10399 Entity-Destination 2400 | 10400 Component-Whole 2401 | 10401 Member-Collection 2402 | 10402 Other 2403 | 10403 Entity-Origin 2404 | 10404 Member-Collection 2405 | 10405 Entity-Destination 2406 | 10406 Other 2407 | 10407 Product-Producer 2408 | 10408 Member-Collection 2409 | 10409 Product-Producer 2410 | 10410 Other 2411 | 10411 Other 2412 | 10412 Product-Producer 2413 | 10413 Entity-Destination 2414 | 10414 Message-Topic 2415 | 10415 Entity-Destination 2416 | 10416 Member-Collection 2417 | 10417 Cause-Effect 2418 | 10418 Entity-Destination 2419 | 10419 Cause-Effect 2420 | 10420 Other 2421 | 10421 Entity-Destination 2422 | 10422 Message-Topic 2423 | 10423 Entity-Origin 2424 | 10424 Instrument-Agency 2425 | 10425 Cause-Effect 2426 | 10426 Cause-Effect 2427 | 10427 Other 2428 | 10428 Component-Whole 2429 | 10429 Message-Topic 2430 | 10430 Member-Collection 2431 | 10431 Content-Container 2432 | 10432 Content-Container 2433 | 10433 Component-Whole 2434 | 10434 Cause-Effect 2435 | 10435 Component-Whole 2436 | 10436 Entity-Destination 2437 | 10437 Message-Topic 2438 | 10438 Other 2439 | 10439 Other 2440 | 10440 Product-Producer 2441 | 10441 Member-Collection 2442 | 10442 Entity-Destination 2443 | 10443 Content-Container 2444 | 10444 Instrument-Agency 2445 | 10445 Content-Container 2446 | 10446 Entity-Destination 2447 | 10447 Other 2448 | 10448 Product-Producer 2449 | 10449 Member-Collection 2450 | 10450 Other 2451 | 10451 Component-Whole 2452 | 10452 Other 2453 | 10453 Entity-Destination 2454 | 10454 Message-Topic 2455 | 10455 Product-Producer 2456 | 10456 Entity-Destination 2457 | 10457 Message-Topic 2458 | 10458 Other 2459 | 10459 Other 2460 | 10460 Component-Whole 2461 | 10461 Product-Producer 2462 | 10462 Content-Container 2463 | 10463 Entity-Destination 2464 | 10464 Product-Producer 2465 | 10465 Message-Topic 2466 | 10466 Cause-Effect 2467 | 10467 Entity-Destination 2468 | 10468 Cause-Effect 2469 | 10469 Component-Whole 2470 | 10470 Content-Container 2471 | 10471 Entity-Origin 2472 | 10472 Message-Topic 2473 | 10473 Product-Producer 2474 | 10474 Entity-Origin 2475 | 10475 Member-Collection 2476 | 10476 Content-Container 2477 | 10477 Content-Container 2478 | 10478 Entity-Destination 2479 | 10479 Content-Container 2480 | 10480 Entity-Origin 2481 | 10481 Cause-Effect 2482 | 10482 Product-Producer 2483 | 10483 Component-Whole 2484 | 10484 Component-Whole 2485 | 10485 Other 2486 | 10486 Message-Topic 2487 | 10487 Other 2488 | 10488 Entity-Destination 2489 | 10489 Component-Whole 2490 | 10490 Entity-Origin 2491 | 10491 Instrument-Agency 2492 | 10492 Other 2493 | 10493 Cause-Effect 2494 | 10494 Other 2495 | 10495 Content-Container 2496 | 10496 Product-Producer 2497 | 10497 Component-Whole 2498 | 10498 Content-Container 2499 | 10499 Other 2500 | 10500 Cause-Effect 2501 | 10501 Cause-Effect 2502 | 10502 Component-Whole 2503 | 10503 Component-Whole 2504 | 10504 Cause-Effect 2505 | 10505 Cause-Effect 2506 | 10506 Instrument-Agency 2507 | 10507 Entity-Origin 2508 | 10508 Product-Producer 2509 | 10509 Entity-Destination 2510 | 10510 Component-Whole 2511 | 10511 Product-Producer 2512 | 10512 Other 2513 | 10513 Other 2514 | 10514 Entity-Origin 2515 | 10515 Member-Collection 2516 | 10516 Product-Producer 2517 | 10517 Other 2518 | 10518 Message-Topic 2519 | 10519 Entity-Destination 2520 | 10520 Member-Collection 2521 | 10521 Other 2522 | 10522 Other 2523 | 10523 Cause-Effect 2524 | 10524 Cause-Effect 2525 | 10525 Member-Collection 2526 | 10526 Component-Whole 2527 | 10527 Member-Collection 2528 | 10528 Cause-Effect 2529 | 10529 Component-Whole 2530 | 10530 Content-Container 2531 | 10531 Message-Topic 2532 | 10532 Entity-Origin 2533 | 10533 Message-Topic 2534 | 10534 Other 2535 | 10535 Message-Topic 2536 | 10536 Component-Whole 2537 | 10537 Product-Producer 2538 | 10538 Entity-Origin 2539 | 10539 Product-Producer 2540 | 10540 Entity-Destination 2541 | 10541 Entity-Origin 2542 | 10542 Component-Whole 2543 | 10543 Entity-Origin 2544 | 10544 Cause-Effect 2545 | 10545 Cause-Effect 2546 | 10546 Other 2547 | 10547 Component-Whole 2548 | 10548 Component-Whole 2549 | 10549 Product-Producer 2550 | 10550 Instrument-Agency 2551 | 10551 Cause-Effect 2552 | 10552 Cause-Effect 2553 | 10553 Product-Producer 2554 | 10554 Product-Producer 2555 | 10555 Content-Container 2556 | 10556 Component-Whole 2557 | 10557 Entity-Destination 2558 | 10558 Message-Topic 2559 | 10559 Entity-Destination 2560 | 10560 Member-Collection 2561 | 10561 Other 2562 | 10562 Other 2563 | 10563 Product-Producer 2564 | 10564 Entity-Destination 2565 | 10565 Product-Producer 2566 | 10566 Entity-Destination 2567 | 10567 Other 2568 | 10568 Other 2569 | 10569 Product-Producer 2570 | 10570 Message-Topic 2571 | 10571 Other 2572 | 10572 Other 2573 | 10573 Entity-Origin 2574 | 10574 Other 2575 | 10575 Content-Container 2576 | 10576 Product-Producer 2577 | 10577 Cause-Effect 2578 | 10578 Cause-Effect 2579 | 10579 Content-Container 2580 | 10580 Member-Collection 2581 | 10581 Component-Whole 2582 | 10582 Member-Collection 2583 | 10583 Instrument-Agency 2584 | 10584 Cause-Effect 2585 | 10585 Product-Producer 2586 | 10586 Component-Whole 2587 | 10587 Entity-Origin 2588 | 10588 Member-Collection 2589 | 10589 Other 2590 | 10590 Entity-Destination 2591 | 10591 Component-Whole 2592 | 10592 Component-Whole 2593 | 10593 Other 2594 | 10594 Entity-Origin 2595 | 10595 Other 2596 | 10596 Message-Topic 2597 | 10597 Cause-Effect 2598 | 10598 Other 2599 | 10599 Cause-Effect 2600 | 10600 Product-Producer 2601 | 10601 Other 2602 | 10602 Entity-Destination 2603 | 10603 Other 2604 | 10604 Component-Whole 2605 | 10605 Cause-Effect 2606 | 10606 Cause-Effect 2607 | 10607 Component-Whole 2608 | 10608 Entity-Origin 2609 | 10609 Instrument-Agency 2610 | 10610 Other 2611 | 10611 Entity-Destination 2612 | 10612 Other 2613 | 10613 Entity-Destination 2614 | 10614 Cause-Effect 2615 | 10615 Other 2616 | 10616 Message-Topic 2617 | 10617 Entity-Destination 2618 | 10618 Product-Producer 2619 | 10619 Entity-Origin 2620 | 10620 Other 2621 | 10621 Other 2622 | 10622 Cause-Effect 2623 | 10623 Entity-Origin 2624 | 10624 Content-Container 2625 | 10625 Member-Collection 2626 | 10626 Component-Whole 2627 | 10627 Cause-Effect 2628 | 10628 Message-Topic 2629 | 10629 Cause-Effect 2630 | 10630 Other 2631 | 10631 Content-Container 2632 | 10632 Entity-Destination 2633 | 10633 Entity-Destination 2634 | 10634 Member-Collection 2635 | 10635 Content-Container 2636 | 10636 Content-Container 2637 | 10637 Cause-Effect 2638 | 10638 Other 2639 | 10639 Cause-Effect 2640 | 10640 Component-Whole 2641 | 10641 Cause-Effect 2642 | 10642 Cause-Effect 2643 | 10643 Cause-Effect 2644 | 10644 Entity-Origin 2645 | 10645 Cause-Effect 2646 | 10646 Cause-Effect 2647 | 10647 Message-Topic 2648 | 10648 Product-Producer 2649 | 10649 Entity-Origin 2650 | 10650 Cause-Effect 2651 | 10651 Member-Collection 2652 | 10652 Other 2653 | 10653 Message-Topic 2654 | 10654 Other 2655 | 10655 Content-Container 2656 | 10656 Entity-Origin 2657 | 10657 Component-Whole 2658 | 10658 Message-Topic 2659 | 10659 Member-Collection 2660 | 10660 Message-Topic 2661 | 10661 Product-Producer 2662 | 10662 Content-Container 2663 | 10663 Content-Container 2664 | 10664 Other 2665 | 10665 Component-Whole 2666 | 10666 Entity-Origin 2667 | 10667 Other 2668 | 10668 Member-Collection 2669 | 10669 Message-Topic 2670 | 10670 Content-Container 2671 | 10671 Other 2672 | 10672 Other 2673 | 10673 Product-Producer 2674 | 10674 Other 2675 | 10675 Entity-Destination 2676 | 10676 Component-Whole 2677 | 10677 Message-Topic 2678 | 10678 Other 2679 | 10679 Product-Producer 2680 | 10680 Instrument-Agency 2681 | 10681 Entity-Destination 2682 | 10682 Other 2683 | 10683 Entity-Destination 2684 | 10684 Entity-Origin 2685 | 10685 Product-Producer 2686 | 10686 Component-Whole 2687 | 10687 Other 2688 | 10688 Entity-Destination 2689 | 10689 Component-Whole 2690 | 10690 Entity-Origin 2691 | 10691 Entity-Origin 2692 | 10692 Cause-Effect 2693 | 10693 Content-Container 2694 | 10694 Entity-Destination 2695 | 10695 Message-Topic 2696 | 10696 Instrument-Agency 2697 | 10697 Message-Topic 2698 | 10698 Other 2699 | 10699 Message-Topic 2700 | 10700 Member-Collection 2701 | 10701 Entity-Destination 2702 | 10702 Instrument-Agency 2703 | 10703 Cause-Effect 2704 | 10704 Cause-Effect 2705 | 10705 Entity-Destination 2706 | 10706 Other 2707 | 10707 Component-Whole 2708 | 10708 Entity-Destination 2709 | 10709 Other 2710 | 10710 Member-Collection 2711 | 10711 Entity-Origin 2712 | 10712 Entity-Origin 2713 | 10713 Instrument-Agency 2714 | 10714 Product-Producer 2715 | 10715 Component-Whole 2716 | 10716 Product-Producer 2717 | 10717 Entity-Destination 2718 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/README.txt: -------------------------------------------------------------------------------- 1 | Training Data for SemEval-2 Task #8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals 2 | 3 | Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano and Stan Szpakowicz 4 | 5 | The accompanying dataset is released under a Creative Commons Atrribution 3.0 Unported Licence (http://creativecommons.org/licenses/by/3.0/). 6 | 7 | Version 1.0: March 5, 2010 8 | 9 | 10 | SUMMARY 11 | 12 | This dataset consists of 8000 sentences which have been annotated according to the scheme for SemEval-2 Task #8. Some sentences have been reused from SemEval-1 Task #4 (Classification of Semantic Relations between Nominals), the rest have been collected from the Web specifically for this task. All have been annotated in accordance with the new relation definitions included in this data release. 13 | 14 | 15 | RELATIONS 16 | 17 | We chose nine relations for SemEval-2 Task #8: 18 | 19 | (1) Cause-Effect 20 | (2) Instrument-Agency 21 | (3) Product-Producer 22 | (4) Content-Container 23 | (5) Entity-Origin 24 | (6) Entity-Destination 25 | (7) Component-Whole 26 | (8) Member-Collection 27 | (9) Message-Topic 28 | 29 | Relations 1-5 also featured in SemEval-1 Task #4, and a subset of the positive and negative examples in the SemEval-1 Task #4 dataset for these relations are included in the training data here. The definitions for all nine relations appear in the files Task8_Relation*.pdf (also included in the distribution). 30 | 31 | 32 | DATA FORMAT 33 | 34 | The format of the data is illustrated by the following examples: 35 | 36 | 15 "They saw that the equipment was put inside rollout drawers, which looked aesthetically more pleasing and tidy." 37 | Content-Container(e1,e2) 38 | Comment: the drawer contains the equipment, typical example of Content-Container; no movement 39 | 40 | 20 ""Any time," he told her before turning to the boy who was in the desk next to him." 41 | Other 42 | Comment: the desk does not contain the boy. 43 | 44 | The first line contains the sentence itself inside quotation marks, preceded by a numerical identifier. Each sentence is annotated with three pieces of information: 45 | 46 | (a) Two entity mentions in the sentence are tagged as e1 and e2 -- the numbering simply reflects the order of the mentions in the sentence. The span of the tag corresponds to the "base NP" which may be smaller than the full NP denoting the entity. 47 | 48 | (b) If one of the semantic relations 1-9 holds between e1 and e2, the sentence is labelled with this relation's name and the order in which the relation arguments are filled by e1 and e2. For example, Cause-Effect(e1,e2) means that e1 is the Cause and e2 is the Effect, whereas Cause-Effect(e2,e1) means that e2 is the Cause and e1 is the Effect. If none of the relations 1-9 holds, the sentence is labelled "Other". In total, then, 19 labels are possible. 49 | 50 | (c) A comment may be provided to explain why the annotators chose a given label. Comments are intended for human readers and should be ignored by automatic systems participating in the task. Comments will not be released for the test data. 51 | 52 | Note that the test release will be formatted similarly, but without lines for the relation label and for the comment. 53 | 54 | Further information on the annotation methodology can be found in the enclosed document Task8_Guidelines.pdf. 55 | 56 | 57 | EVALUATION 58 | 59 | The task is to predict, given a sentence and two tagged entities, which of the relation labels to apply. Hence, the gold-standard labels (Cause-Effect(e1,e2) and so on) should be provided to a system at training time but not at test time. The predictions of the system should be in the following format: 60 | 61 | 1 Content-Container(e2,e1) 62 | 2 Other 63 | 3 Entity-Destination(e1,e2) 64 | ... 65 | 66 | The official evaluation measures are accuracy over all examples and macro-averaged F-score over the 18 relation labels apart from Other. To calculate the F-score, 18 individual F-scores -- one for each relation label -- are calculated in the standard way and the average of these scores is taken. For each relation Rel, each sentence labelled Rel in the gold standard will count as either a true positive or a false negative, depending on whether it was correctly labelled by the system; each sentence labelled with a different relation or with Other will count as a true negative or false positive. 67 | 68 | 69 | TEST PROCEDURE 70 | 71 | The test data will be released on March 18. After this, participants will be able to download it at any time up to the final results submission deadline (April 2, 2010). Once the data have been downloaded, participants will have 7 days to submit their results; they must also submit by the final deadline of April 2. Late submissions will not be counted. Participants should supply four sets of predictions for the test data, using four subsets of the training data: 72 | 73 | TD1 training examples 1-1000 74 | TD2 training examples 1-2000 75 | TD3 training examples 1-4000 76 | TD4 training examples 1-8000 77 | 78 | For each training set, participants may use the data in that set for any purpose they wish (training, development, cross-validation and so forth). However, the training examples outside that set (e.g., 1001-8000 for TD1) may not be used in any way. The final 891 examples in the training release (examples 7110-8000) are taken from the SemEval-1 Task #4 datasets for relations 1-5 and hence their label distribution is skewed towards those relation classes. Participants have the option of including or excluding these examples as appropriate for their chosen learning method. 79 | 80 | There is no restriction on the external resources that may be used. 81 | 82 | 83 | USEFUL LINKS: 84 | 85 | Google group: http://groups.google.com.sg/group/semeval-2010-multi-way-classification-of-semantic-relations?hl=en 86 | Task website: http://docs.google.com/View?docid=dfvxd49s_36c28v9pmw 87 | SemEval-2 website: http://semeval2.fbk.eu/semeval2.php 88 | 89 | 90 | TASK SCHEDULE 91 | 92 | * Test data release: March 18, 2010 93 | * Result submission deadline: 7 days after downloading the *test* data, but no later than April 2 94 | * Organizers send the test results: April 10, 2010 95 | * Submission of description papers: April 17, 2010 96 | * Notification of acceptance: May 6, 2010 97 | * SemEval-2 workshop (at ACL): July 15-16, 2010 98 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_DISTRIB.TXT: -------------------------------------------------------------------------------- 1 | <<>> 2 | TOTAL : 8000 3 | Other : 1410 (17.63%) 4 | Cause-Effect : 1003 (12.54%) 5 | Component-Whole : 941 (11.76%) 6 | Entity-Destination : 845 (10.56%) 7 | Product-Producer : 717 ( 8.96%) 8 | Entity-Origin : 716 ( 8.95%) 9 | Member-Collection : 690 ( 8.63%) 10 | Message-Topic : 634 ( 7.92%) 11 | Content-Container : 540 ( 6.75%) 12 | Instrument-Agency : 504 ( 6.30%) 13 | 14 | 15 | <<>> 16 | TOTAL : 7109 17 | Other : 1130 (14.13%) 18 | Component-Whole : 870 (10.88%) 19 | Cause-Effect : 847 (10.59%) 20 | Entity-Destination : 832 (10.40%) 21 | Member-Collection : 682 ( 8.53%) 22 | Message-Topic : 629 ( 7.86%) 23 | Entity-Origin : 614 ( 7.67%) 24 | Product-Producer : 603 ( 7.54%) 25 | Content-Container : 483 ( 6.04%) 26 | Instrument-Agency : 419 ( 5.24%) 27 | 28 | 29 | <<>> 30 | TOTAL : 891 31 | Other : 280 (31.43%) 32 | Cause-Effect : 156 (17.51%) 33 | Product-Producer : 114 (12.79%) 34 | Entity-Origin : 102 (11.45%) 35 | Instrument-Agency : 85 ( 9.54%) 36 | Component-Whole : 71 ( 7.97%) 37 | Content-Container : 57 ( 6.40%) 38 | Entity-Destination : 13 ( 1.46%) 39 | Member-Collection : 8 ( 0.90%) 40 | Message-Topic : 5 ( 0.56%) 41 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_TEST_DISTRIB.TXT: -------------------------------------------------------------------------------- 1 | 2 | <<>> 3 | TOTAL : 891 4 | Other : 280 (31.43%) 5 | Cause-Effect : 156 (17.51%) 6 | Product-Producer : 114 (12.79%) 7 | Entity-Origin : 102 (11.45%) 8 | Instrument-Agency : 85 ( 9.54%) 9 | Component-Whole : 71 ( 7.97%) 10 | Content-Container : 57 ( 6.40%) 11 | Entity-Destination : 13 ( 1.46%) 12 | Member-Collection : 8 ( 0.90%) 13 | Message-Topic : 5 ( 0.56%) 14 | 15 | <<>> 16 | TOTAL : 9826 17 | Other : 1584 (16.12%) 18 | Component-Whole : 1182 (12.03%) 19 | Cause-Effect : 1175 (11.96%) 20 | Entity-Destination : 1124 (11.44%) 21 | Member-Collection : 915 ( 9.31%) 22 | Message-Topic : 890 ( 9.06%) 23 | Entity-Origin : 872 ( 8.87%) 24 | Product-Producer : 834 ( 8.49%) 25 | Content-Container : 675 ( 6.87%) 26 | Instrument-Agency : 575 ( 5.85%) 27 | 28 | <<>> 29 | TOTAL : 10717 30 | Other : 1864 (17.39%) 31 | Cause-Effect : 1331 (12.42%) 32 | Component-Whole : 1253 (11.69%) 33 | Entity-Destination : 1137 (10.61%) 34 | Entity-Origin : 974 ( 9.09%) 35 | Product-Producer : 948 ( 8.85%) 36 | Member-Collection : 923 ( 8.61%) 37 | Message-Topic : 895 ( 8.35%) 38 | Content-Container : 732 ( 6.83%) 39 | Instrument-Agency : 660 ( 6.16%) 40 | 41 | <<>> 42 | TOTAL : 8000 43 | Other : 1410 (17.63%) 44 | Cause-Effect : 1003 (12.54%) 45 | Component-Whole : 941 (11.76%) 46 | Entity-Destination : 845 (10.56%) 47 | Product-Producer : 717 ( 8.96%) 48 | Entity-Origin : 716 ( 8.95%) 49 | Member-Collection : 690 ( 8.63%) 50 | Message-Topic : 634 ( 7.92%) 51 | Content-Container : 540 ( 6.75%) 52 | Instrument-Agency : 504 ( 6.30%) 53 | 54 | <<>> 55 | TOTAL : 2717 56 | Other : 454 (16.71%) 57 | Cause-Effect : 328 (12.07%) 58 | Component-Whole : 312 (11.48%) 59 | Entity-Destination : 292 (10.75%) 60 | Message-Topic : 261 ( 9.61%) 61 | Entity-Origin : 258 ( 9.50%) 62 | Member-Collection : 233 ( 8.58%) 63 | Product-Producer : 231 ( 8.50%) 64 | Content-Container : 192 ( 7.07%) 65 | Instrument-Agency : 156 ( 5.74%) 66 | -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Guidelines.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Guidelines.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation1.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation2.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation3.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation3.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation4.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation4.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation5.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation5.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation6.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation6.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation7.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation7.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation8.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation8.pdf -------------------------------------------------------------------------------- /SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation9.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/roomylee/entity-aware-relation-classification/2737babdcdf8a7c10d3fd20bebadb940c50b58b7/SemEval2010_task8_all_data/SemEval2010_task8_training/Task8_Relation9.pdf -------------------------------------------------------------------------------- /configure.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import sys 3 | 4 | 5 | def parse_args(): 6 | """ 7 | Parse input arguments 8 | """ 9 | parser = argparse.ArgumentParser() 10 | 11 | # Data loading params 12 | parser.add_argument("--train_path", default="SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT", 13 | type=str, help="Path of train data") 14 | parser.add_argument("--test_path", default="SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT", 15 | type=str, help="Path of test data") 16 | parser.add_argument("--max_sentence_length", default=90, 17 | type=int, help="Max sentence length in data") 18 | 19 | # Model Hyper-parameters 20 | # Embeddings 21 | parser.add_argument("--embeddings", default=None, 22 | type=str, help="Embeddings {'word2vec', 'glove100', 'glove300', 'elmo'}") 23 | parser.add_argument("--embedding_size", default=300, 24 | type=int, help="Dimensionality of word embedding (default: 300)") 25 | parser.add_argument("--pos_embedding_size", default=50, 26 | type=int, help="Dimensionality of relative position embedding (default: 50)") 27 | parser.add_argument("--emb_dropout_keep_prob", default=0.7, 28 | type=float, help="Dropout keep probability of embedding layer (default: 0.7)") 29 | # RNN 30 | parser.add_argument("--hidden_size", default=300, 31 | type=int, help="Dimensionality of RNN hidden (default: 300)") 32 | parser.add_argument("--rnn_dropout_keep_prob", default=0.7, 33 | type=float, help="Dropout keep probability of RNN (default: 0.7)") 34 | # Attention 35 | parser.add_argument("--num_heads", default=4, 36 | type=int, help="Number of heads in multi-head attention (default: 4)") 37 | parser.add_argument("--attention_size", default=50, 38 | type=int, help="Dimensionality of attention (default: 50)") 39 | # Misc 40 | parser.add_argument("--desc", default="", 41 | type=str, help="Description for model") 42 | parser.add_argument("--dropout_keep_prob", default=0.5, 43 | type=float, help="Dropout keep probability of output layer (default: 0.5)") 44 | parser.add_argument("--l2_reg_lambda", default=1e-5, 45 | type=float, help="L2 regularization lambda (default: 1e-5)") 46 | 47 | # Training parameters 48 | parser.add_argument("--batch_size", default=20, 49 | type=int, help="Batch Size (default: 20)") 50 | parser.add_argument("--num_epochs", default=100, 51 | type=int, help="Number of training epochs (Default: 100)") 52 | parser.add_argument("--display_every", default=10, 53 | type=int, help="Number of iterations to display training information") 54 | parser.add_argument("--evaluate_every", default=100, 55 | type=int, help="Evaluate model on dev set after this many steps (default: 100)") 56 | parser.add_argument("--num_checkpoints", default=5, 57 | type=int, help="Number of checkpoints to store (default: 5)") 58 | parser.add_argument("--learning_rate", default=1.0, 59 | type=float, help="Which learning rate to start with (Default: 1.0)") 60 | parser.add_argument("--decay_rate", default=0.9, 61 | type=float, help="Decay rate for learning rate (Default: 0.9)") 62 | 63 | # Misc Parameters 64 | parser.add_argument("--allow_soft_placement", default=True, 65 | type=bool, help="Allow device soft device placement") 66 | parser.add_argument("--log_device_placement", default=False, 67 | type=bool, help="Log placement of ops on devices") 68 | parser.add_argument("--gpu_allow_growth", default=True, 69 | type=bool, help="Allow gpu memory growth") 70 | 71 | # Visualization Parameters 72 | parser.add_argument("--checkpoint_dir", default=None, 73 | type=str, help="Visualize this checkpoint") 74 | 75 | if len(sys.argv) == 0: 76 | parser.print_help() 77 | sys.exit(1) 78 | 79 | print("") 80 | args = parser.parse_args() 81 | for arg in vars(args): 82 | print("{}={}".format(arg.upper(), getattr(args, arg))) 83 | print("") 84 | 85 | return args 86 | 87 | FLAGS = parse_args() 88 | -------------------------------------------------------------------------------- /data_helpers.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pandas as pd 3 | import nltk 4 | import re 5 | 6 | import utils 7 | from configure import FLAGS 8 | 9 | 10 | def clean_str(text): 11 | text = text.lower() 12 | # Clean the text 13 | text = re.sub(r"[^A-Za-z0-9^,!.\/'+-=]", " ", text) 14 | text = re.sub(r"what's", "what is ", text) 15 | text = re.sub(r"that's", "that is ", text) 16 | text = re.sub(r"there's", "there is ", text) 17 | text = re.sub(r"it's", "it is ", text) 18 | text = re.sub(r"\'s", " ", text) 19 | text = re.sub(r"\'ve", " have ", text) 20 | text = re.sub(r"can't", "can not ", text) 21 | text = re.sub(r"n't", " not ", text) 22 | text = re.sub(r"i'm", "i am ", text) 23 | text = re.sub(r"\'re", " are ", text) 24 | text = re.sub(r"\'d", " would ", text) 25 | text = re.sub(r"\'ll", " will ", text) 26 | text = re.sub(r",", " ", text) 27 | text = re.sub(r"\.", " ", text) 28 | text = re.sub(r"!", " ! ", text) 29 | text = re.sub(r"\/", " ", text) 30 | text = re.sub(r"\^", " ^ ", text) 31 | text = re.sub(r"\+", " + ", text) 32 | text = re.sub(r"\-", " - ", text) 33 | text = re.sub(r"\=", " = ", text) 34 | text = re.sub(r"'", " ", text) 35 | text = re.sub(r"(\d+)(k)", r"\g<1>000", text) 36 | text = re.sub(r":", " : ", text) 37 | text = re.sub(r" e g ", " eg ", text) 38 | text = re.sub(r" b g ", " bg ", text) 39 | text = re.sub(r" u s ", " american ", text) 40 | text = re.sub(r"\0s", "0", text) 41 | text = re.sub(r" 9 11 ", "911", text) 42 | text = re.sub(r"e - mail", "email", text) 43 | text = re.sub(r"j k", "jk", text) 44 | text = re.sub(r"\s{2,}", " ", text) 45 | 46 | return text.strip() 47 | 48 | 49 | def load_data_and_labels(path): 50 | data = [] 51 | lines = [line.strip() for line in open(path)] 52 | max_sentence_length = 0 53 | for idx in range(0, len(lines), 4): 54 | id = lines[idx].split("\t")[0] 55 | relation = lines[idx + 1] 56 | 57 | sentence = lines[idx].split("\t")[1][1:-1] 58 | sentence = sentence.replace('', ' _e11_ ') 59 | sentence = sentence.replace('', ' _e12_ ') 60 | sentence = sentence.replace('', ' _e21_ ') 61 | sentence = sentence.replace('', ' _e22_ ') 62 | 63 | sentence = clean_str(sentence) 64 | tokens = nltk.word_tokenize(sentence) 65 | if max_sentence_length < len(tokens): 66 | max_sentence_length = len(tokens) 67 | e1 = tokens.index("e12") - 1 68 | e2 = tokens.index("e22") - 1 69 | sentence = " ".join(tokens) 70 | 71 | data.append([id, sentence, e1, e2, relation]) 72 | 73 | print(path) 74 | print("max sentence length = {}\n".format(max_sentence_length)) 75 | 76 | df = pd.DataFrame(data=data, columns=["id", "sentence", "e1", "e2", "relation"]) 77 | 78 | pos1, pos2 = get_relative_position(df, FLAGS.max_sentence_length) 79 | 80 | df['label'] = [utils.class2label[r] for r in df['relation']] 81 | 82 | # Text Data 83 | x_text = df['sentence'].tolist() 84 | e1 = df['e1'].tolist() 85 | e2 = df['e2'].tolist() 86 | 87 | # Label Data 88 | y = df['label'] 89 | labels_flat = y.values.ravel() 90 | labels_count = np.unique(labels_flat).shape[0] 91 | 92 | # convert class labels from scalars to one-hot vectors 93 | # 0 => [1 0 0 0 0 ... 0 0 0 0 0] 94 | # 1 => [0 1 0 0 0 ... 0 0 0 0 0] 95 | # ... 96 | # 18 => [0 0 0 0 0 ... 0 0 0 0 1] 97 | def dense_to_one_hot(labels_dense, num_classes): 98 | num_labels = labels_dense.shape[0] 99 | index_offset = np.arange(num_labels) * num_classes 100 | labels_one_hot = np.zeros((num_labels, num_classes)) 101 | labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1 102 | return labels_one_hot 103 | 104 | labels = dense_to_one_hot(labels_flat, labels_count) 105 | labels = labels.astype(np.uint8) 106 | 107 | return x_text, labels, e1, e2, pos1, pos2 108 | 109 | 110 | def get_relative_position(df, max_sentence_length): 111 | # Position data 112 | pos1 = [] 113 | pos2 = [] 114 | for df_idx in range(len(df)): 115 | sentence = df.iloc[df_idx]['sentence'] 116 | tokens = nltk.word_tokenize(sentence) 117 | e1 = df.iloc[df_idx]['e1'] 118 | e2 = df.iloc[df_idx]['e2'] 119 | 120 | p1 = "" 121 | p2 = "" 122 | for word_idx in range(len(tokens)): 123 | p1 += str((max_sentence_length - 1) + word_idx - e1) + " " 124 | p2 += str((max_sentence_length - 1) + word_idx - e2) + " " 125 | pos1.append(p1) 126 | pos2.append(p2) 127 | 128 | return pos1, pos2 129 | 130 | 131 | def batch_iter(data, batch_size, num_epochs, shuffle=True): 132 | """ 133 | Generates a batch iterator for a dataset. 134 | """ 135 | data = np.array(data) 136 | data_size = len(data) 137 | num_batches_per_epoch = int((len(data) - 1) / batch_size) + 1 138 | for epoch in range(num_epochs): 139 | # Shuffle the data at each epoch 140 | if shuffle: 141 | shuffle_indices = np.random.permutation(np.arange(data_size)) 142 | shuffled_data = data[shuffle_indices] 143 | else: 144 | shuffled_data = data 145 | for batch_num in range(num_batches_per_epoch): 146 | start_index = batch_num * batch_size 147 | end_index = min((batch_num + 1) * batch_size, data_size) 148 | yield shuffled_data[start_index:end_index] 149 | 150 | 151 | if __name__ == "__main__": 152 | trainFile = 'SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT' 153 | testFile = 'SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT' 154 | 155 | load_data_and_labels(testFile) 156 | -------------------------------------------------------------------------------- /logger.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import os 3 | import datetime 4 | 5 | from configure import FLAGS 6 | import utils 7 | 8 | 9 | class Logger: 10 | def __init__(self, out_dir): 11 | self.log_dir = os.path.abspath(os.path.join(out_dir, "logs")) 12 | os.makedirs(self.log_dir) 13 | self.log_path = os.path.abspath(os.path.join(self.log_dir, "logs.txt")) 14 | self.log_file = open(self.log_path, "w") 15 | 16 | self.print_hyperparameters() 17 | 18 | self.best_f1 = 0.0 19 | 20 | def print_hyperparameters(self): 21 | self.log_file.write("\n================ Hyper-parameters ================\n\n") 22 | for arg in vars(FLAGS): 23 | self.log_file.write("{}={}\n".format(arg.upper(), getattr(FLAGS, arg))) 24 | self.log_file.write("\n==================================================\n\n") 25 | 26 | def logging_train(self, step, loss, accuracy): 27 | time_str = datetime.datetime.now().isoformat() 28 | log = "{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy) 29 | self.log_file.write(log+"\n") 30 | print(log) 31 | 32 | def logging_eval(self, step, loss, accuracy, predictions): 33 | self.log_file.write("\nEvaluation:\n") 34 | # loss & acc 35 | time_str = datetime.datetime.now().isoformat() 36 | log = "{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy) 37 | self.log_file.write(log + "\n") 38 | print(log) 39 | 40 | # f1-score 41 | prediction_path = os.path.abspath(os.path.join(self.log_dir, "predictions.txt")) 42 | prediction_file = open(prediction_path, 'w') 43 | for i in range(len(predictions)): 44 | prediction_file.write("{}\t{}\n".format(i, utils.label2class[predictions[i]])) 45 | prediction_file.close() 46 | perl_path = os.path.join(os.path.curdir, 47 | "SemEval2010_task8_all_data", 48 | "SemEval2010_task8_scorer-v1.2", 49 | "semeval2010_task8_scorer-v1.2.pl") 50 | target_path = os.path.join(os.path.curdir, "resource", "target.txt") 51 | process = subprocess.Popen(["perl", perl_path, prediction_path, target_path], stdout=subprocess.PIPE) 52 | str_parse = str(process.communicate()[0]).split("\\n")[-2] 53 | idx = str_parse.find('%') 54 | f1_score = float(str_parse[idx-5:idx]) 55 | 56 | self.best_f1 = max(self.best_f1, f1_score) 57 | f1_log = "<<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>:\n" \ 58 | "macro-averaged F1-score = {:g}%, Best = {:g}%\n".format(f1_score, self.best_f1) 59 | self.log_file.write(f1_log + "\n") 60 | print(f1_log) 61 | -------------------------------------------------------------------------------- /model/attention.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from utils import initializer 4 | 5 | 6 | def attention(inputs, e1, e2, p1, p2, attention_size): 7 | # inputs = (batch, seq_len, hidden) 8 | # e1, e2 = (batch, seq_len) 9 | # p1, p2 = (batch, seq_len, dist_emb_size) 10 | # attention_size = scalar(int) 11 | def extract_entity(x, e): 12 | e_idx = tf.concat([tf.expand_dims(tf.range(tf.shape(e)[0]), axis=-1), tf.expand_dims(e, axis=-1)], axis=-1) 13 | return tf.gather_nd(x, e_idx) # (batch, hidden) 14 | seq_len = tf.shape(inputs)[1] # fixed at run-time 15 | hidden_size = inputs.shape[2].value # fixed at compile-time 16 | latent_size = hidden_size 17 | 18 | # Latent Relation Variable based on Entities 19 | e1_h = extract_entity(inputs, e1) # (batch, hidden) 20 | e2_h = extract_entity(inputs, e2) # (batch, hidden) 21 | e1_type, e2_type, e1_alphas, e2_alphas = latent_type_attention(e1_h, e2_h, 22 | num_type=3, 23 | latent_size=latent_size) # (batch, hidden) 24 | e1_h = tf.concat([e1_h, e1_type], axis=-1) # (batch, hidden+latent) 25 | e2_h = tf.concat([e2_h, e2_type], axis=-1) # (batch, hidden+latent) 26 | 27 | # v*tanh(W*[h;p1;p2]+W*[e1;e2]) 85.18%? 84.83% 84.55% 28 | e_h = tf.layers.dense(tf.concat([e1_h, e2_h], -1), attention_size, use_bias=False, kernel_initializer=initializer()) 29 | e_h = tf.reshape(tf.tile(e_h, [1, seq_len]), [-1, seq_len, attention_size]) 30 | v = tf.layers.dense(tf.concat([inputs, p1, p2], axis=-1), attention_size, use_bias=False, kernel_initializer=initializer()) 31 | v = tf.tanh(tf.add(v, e_h)) 32 | 33 | u_omega = tf.get_variable("u_omega", [attention_size], initializer=initializer()) 34 | vu = tf.tensordot(v, u_omega, axes=1, name='vu') # (batch, seq_len) 35 | alphas = tf.nn.softmax(vu, name='alphas') # (batch, seq_len) 36 | 37 | # v*tanh(W*[h;p1;p2;e1;e2]) 85.18% 84.41% 38 | # e1_h = tf.reshape(tf.tile(e1_h, [1, seq_len]), [-1, seq_len, hidden_size+latent_size]) 39 | # e2_h = tf.reshape(tf.tile(e2_h, [1, seq_len]), [-1, seq_len, hidden_size+latent_size]) 40 | # v = tf.concat([inputs, p1, p2, e1_h, e2_h], axis=-1) 41 | # v = tf.layers.dense(v, attention_size, activation=tf.tanh, kernel_initializer=initializer()) 42 | # 43 | # u_omega = tf.get_variable("u_omega", [attention_size], initializer=initializer()) 44 | # vu = tf.tensordot(v, u_omega, axes=1, name='vu') # (batch, seq_len) 45 | # alphas = tf.nn.softmax(vu, name='alphas') # (batch, seq_len) 46 | 47 | # output 48 | output = tf.reduce_sum(inputs * tf.expand_dims(alphas, -1), 1) # (batch, hidden) 49 | 50 | return output, alphas, e1_alphas, e2_alphas 51 | 52 | 53 | def latent_type_attention(e1, e2, num_type, latent_size): 54 | # Latent Entity Type Vectors 55 | latent_type = tf.get_variable("latent_type", shape=[num_type, latent_size], initializer=initializer()) 56 | 57 | # e1_h = tf.layers.dense(e1, latent_size, kernel_initializer=initializer()) 58 | # e2_h = tf.layers.dense(e2, latent_size, kernel_initializer=initializer()) 59 | 60 | e1_sim = tf.matmul(e1, tf.transpose(latent_type)) # (batch, num_type) 61 | e1_alphas = tf.nn.softmax(e1_sim, name='e1_alphas') # (batch, num_type) 62 | e1_type = tf.matmul(e1_alphas, latent_type, name='e1_type') # (batch, hidden) 63 | 64 | e2_sim = tf.matmul(e2, tf.transpose(latent_type)) # (batch, num_type) 65 | e2_alphas = tf.nn.softmax(e2_sim, name='e2_alphas') # (batch, num_type) 66 | e2_type = tf.matmul(e2_alphas, latent_type, name='e2_type') # (batch, hidden) 67 | 68 | return e1_type, e2_type, e1_alphas, e2_alphas 69 | 70 | 71 | def multihead_attention(queries, keys, num_units, num_heads, 72 | dropout_rate=0, scope="multihead_attention", reuse=None): 73 | with tf.variable_scope(scope, reuse=reuse): 74 | # Linear projections 75 | Q = tf.layers.dense(queries, num_units, kernel_initializer=initializer()) # (N, T_q, C) 76 | K = tf.layers.dense(keys, num_units, kernel_initializer=initializer()) # (N, T_k, C) 77 | V = tf.layers.dense(keys, num_units, kernel_initializer=initializer()) # (N, T_k, C) 78 | 79 | # Split and concat 80 | Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0) # (h*N, T_q, C/h) 81 | K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0) # (h*N, T_k, C/h) 82 | V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0) # (h*N, T_k, C/h) 83 | 84 | # Multiplication 85 | outputs = tf.matmul(Q_, tf.transpose(K_, [0, 2, 1])) # (h*N, T_q, T_k) 86 | 87 | # Scale 88 | outputs /= K_.get_shape().as_list()[-1] ** 0.5 89 | 90 | # Key Masking 91 | key_masks = tf.sign(tf.abs(tf.reduce_sum(keys, axis=-1))) # (N, T_k) 92 | key_masks = tf.tile(key_masks, [num_heads, 1]) # (h*N, T_k) 93 | key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(queries)[1], 1]) # (h*N, T_q, T_k) 94 | 95 | paddings = tf.ones_like(outputs) * (-2 ** 32 + 1) 96 | outputs = tf.where(tf.equal(key_masks, 0), paddings, outputs) # (h*N, T_q, T_k) 97 | 98 | # Activation 99 | alphas = tf.nn.softmax(outputs) # (h*N, T_q, T_k) 100 | 101 | # Query Masking 102 | query_masks = tf.sign(tf.abs(tf.reduce_sum(queries, axis=-1))) # (N, T_q) 103 | query_masks = tf.tile(query_masks, [num_heads, 1]) # (h*N, T_q) 104 | query_masks = tf.tile(tf.expand_dims(query_masks, -1), [1, 1, tf.shape(keys)[1]]) # (h*N, T_q, T_k) 105 | alphas *= query_masks # broadcasting. (N, T_q, C) 106 | 107 | # Dropouts 108 | alphas = tf.layers.dropout(alphas, rate=dropout_rate, training=tf.convert_to_tensor(True)) 109 | 110 | # Weighted sum 111 | outputs = tf.matmul(alphas, V_) # ( h*N, T_q, C/h) 112 | 113 | # Restore shape 114 | outputs = tf.concat(tf.split(outputs, num_heads, axis=0), axis=2) # (N, T_q, C) 115 | 116 | # Linear 117 | outputs = tf.layers.dense(outputs, num_units, activation=tf.nn.relu, kernel_initializer=initializer()) 118 | 119 | # Residual connection 120 | outputs += queries 121 | 122 | # Normalize 123 | outputs = layer_norm(outputs) # (N, T_q, C) 124 | 125 | return outputs, alphas 126 | 127 | 128 | def layer_norm(inputs, epsilon=1e-8, scope="layer_norm", reuse=None): 129 | with tf.variable_scope(scope, reuse=reuse): 130 | inputs_shape = inputs.get_shape() 131 | params_shape = inputs_shape[-1:] 132 | 133 | mean, variance = tf.nn.moments(inputs, [-1], keep_dims=True) 134 | beta = tf.Variable(tf.zeros(params_shape)) 135 | gamma = tf.Variable(tf.ones(params_shape)) 136 | normalized = (inputs - mean) / ((variance + epsilon) ** (.5)) 137 | outputs = gamma * normalized + beta 138 | 139 | return outputs 140 | -------------------------------------------------------------------------------- /model/entity_att_lstm.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import tensorflow_hub as hub 3 | 4 | from utils import initializer 5 | from model.attention import multihead_attention, attention 6 | 7 | 8 | class EntityAttentionLSTM: 9 | def __init__(self, sequence_length, num_classes, 10 | vocab_size, embedding_size, pos_vocab_size, pos_embedding_size, 11 | hidden_size, num_heads, attention_size, 12 | use_elmo=False, l2_reg_lambda=0.0): 13 | # Placeholders for input, output and dropout 14 | self.input_x = tf.placeholder(tf.int32, shape=[None, sequence_length], name='input_x') 15 | self.input_y = tf.placeholder(tf.float32, shape=[None, num_classes], name='input_y') 16 | self.input_text = tf.placeholder(tf.string, shape=[None, ], name='input_text') 17 | self.input_e1 = tf.placeholder(tf.int32, shape=[None, ], name='input_e1') 18 | self.input_e2 = tf.placeholder(tf.int32, shape=[None, ], name='input_e2') 19 | self.input_p1 = tf.placeholder(tf.int32, shape=[None, sequence_length], name='input_p1') 20 | self.input_p2 = tf.placeholder(tf.int32, shape=[None, sequence_length], name='input_p2') 21 | self.emb_dropout_keep_prob = tf.placeholder(tf.float32, name='emb_dropout_keep_prob') 22 | self.rnn_dropout_keep_prob = tf.placeholder(tf.float32, name='rnn_dropout_keep_prob') 23 | self.dropout_keep_prob = tf.placeholder(tf.float32, name='dropout_keep_prob') 24 | 25 | if use_elmo: 26 | # Contextual Embedding Layer 27 | with tf.variable_scope("elmo-embeddings"): 28 | elmo_model = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True) 29 | self.embedded_chars = elmo_model(self.input_text, signature="default", as_dict=True)["elmo"] 30 | else: 31 | # Word Embedding Layer 32 | with tf.device('/cpu:0'), tf.variable_scope("word-embeddings"): 33 | self.W_text = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -0.25, 0.25), name="W_text") 34 | self.embedded_chars = tf.nn.embedding_lookup(self.W_text, self.input_x) 35 | 36 | # Position Embedding Layer 37 | with tf.device('/cpu:0'), tf.variable_scope("position-embeddings"): 38 | self.W_pos = tf.get_variable("W_pos", [pos_vocab_size, pos_embedding_size], initializer=initializer()) 39 | self.p1 = tf.nn.embedding_lookup(self.W_pos, self.input_p1)[:, :tf.shape(self.embedded_chars)[1]] 40 | self.p2 = tf.nn.embedding_lookup(self.W_pos, self.input_p2)[:, :tf.shape(self.embedded_chars)[1]] 41 | 42 | # Dropout for Word Embedding 43 | with tf.variable_scope('dropout-embeddings'): 44 | self.embedded_chars = tf.nn.dropout(self.embedded_chars, self.emb_dropout_keep_prob) 45 | 46 | # Self Attention 47 | with tf.variable_scope("self-attention"): 48 | self.self_attn, self.self_alphas = multihead_attention(self.embedded_chars, self.embedded_chars, 49 | num_units=embedding_size, num_heads=num_heads) 50 | 51 | # Bidirectional LSTM 52 | with tf.variable_scope("bi-lstm"): 53 | _fw_cell = tf.nn.rnn_cell.LSTMCell(hidden_size, initializer=initializer()) 54 | fw_cell = tf.nn.rnn_cell.DropoutWrapper(_fw_cell, self.rnn_dropout_keep_prob) 55 | _bw_cell = tf.nn.rnn_cell.LSTMCell(hidden_size, initializer=initializer()) 56 | bw_cell = tf.nn.rnn_cell.DropoutWrapper(_bw_cell, self.rnn_dropout_keep_prob) 57 | self.rnn_outputs, _ = tf.nn.bidirectional_dynamic_rnn(cell_fw=fw_cell, 58 | cell_bw=bw_cell, 59 | inputs=self.self_attn, 60 | sequence_length=self._length(self.input_x), 61 | dtype=tf.float32) 62 | self.rnn_outputs = tf.concat(self.rnn_outputs, axis=-1) 63 | 64 | # Attention 65 | with tf.variable_scope('attention'): 66 | self.attn, self.alphas, self.e1_alphas, self.e2_alphas = attention(self.rnn_outputs, 67 | self.input_e1, self.input_e2, 68 | self.p1, self.p2, 69 | attention_size=attention_size) 70 | 71 | # Dropout 72 | with tf.variable_scope('dropout'): 73 | self.h_drop = tf.nn.dropout(self.attn, self.dropout_keep_prob) 74 | 75 | # Fully connected layer 76 | with tf.variable_scope('output'): 77 | self.logits = tf.layers.dense(self.h_drop, num_classes, kernel_initializer=initializer()) 78 | self.predictions = tf.argmax(self.logits, 1, name="predictions") 79 | 80 | # Calculate mean cross-entropy loss 81 | with tf.variable_scope("loss"): 82 | losses = tf.nn.softmax_cross_entropy_with_logits_v2(logits=self.logits, labels=self.input_y) 83 | self.l2 = tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables()]) 84 | self.loss = tf.reduce_mean(losses) + l2_reg_lambda * self.l2 85 | 86 | # Accuracy 87 | with tf.variable_scope("accuracy"): 88 | correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1)) 89 | self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32), name="accuracy") 90 | 91 | # Length of the sequence data 92 | @staticmethod 93 | def _length(seq): 94 | relevant = tf.sign(tf.abs(seq)) 95 | length = tf.reduce_sum(relevant, reduction_indices=1) 96 | length = tf.cast(length, tf.int32) 97 | return length 98 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | tensorflow==1.8.0 2 | tensorboard==1.8.0 3 | tensorflow-gpu==1.8.0 4 | tensorflow-hub==0.1.1 5 | sklearn 6 | numpy 7 | pandas -------------------------------------------------------------------------------- /self-attention-visualization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "" 12 | ] 13 | }, 14 | "metadata": {}, 15 | "output_type": "display_data" 16 | } 17 | ], 18 | "source": [ 19 | "%%javascript\n", 20 | "require.config({\n", 21 | " paths: {\n", 22 | " d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min'\n", 23 | " }\n", 24 | "});" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 10, 30 | "metadata": { 31 | "scrolled": false 32 | }, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT\n", 39 | "max sentence length = 68\n", 40 | "\n", 41 | "C:\\Users\\서상우\\Desktop\\self-attentive-relation-extraction\\runs\\1538483063\\checkpoints\\model-84.2-4300\n", 42 | "\n", 43 | "Text Vocabulary Size: 22384\n", 44 | "test_x = (2717, 90)\n", 45 | "test_y = (2717, 19)\n", 46 | "\n", 47 | "Position Vocabulary Size: 162\n", 48 | "test_p1 = (2717, 90)\n", 49 | "\n", 50 | "INFO:tensorflow:Restoring parameters from C:\\Users\\서상우\\Desktop\\self-attentive-relation-extraction\\runs\\1538483063\\checkpoints\\model-84.2-4300\n", 51 | "\n", 52 | "Evaluation:\n" 53 | ] 54 | } 55 | ], 56 | "source": [ 57 | "import sys; sys.argv=['']; del sys\n", 58 | "import os\n", 59 | "import time\n", 60 | "import numpy as np\n", 61 | "import tensorflow as tf\n", 62 | "\n", 63 | "import data_helpers\n", 64 | "from configure import FLAGS\n", 65 | "from logger import Logger\n", 66 | "from model.self_att_lstm import SelfAttentiveLSTM\n", 67 | "import utils\n", 68 | "\n", 69 | "import warnings\n", 70 | "import sklearn.exceptions\n", 71 | "warnings.filterwarnings(\"ignore\", category=sklearn.exceptions.UndefinedMetricWarning)\n", 72 | "\n", 73 | "\n", 74 | "from tensor2tensor.visualization import attention\n", 75 | "\n", 76 | "FLAGS.checkpoint_dir = \"runs/1538483063/checkpoints\"\n", 77 | "with tf.device('/cpu:0'):\n", 78 | " test_text, test_y, test_e1, test_e2, test_pos1, test_pos2 = data_helpers.load_data_and_labels(FLAGS.test_path)\n", 79 | "\n", 80 | "checkpoint_file = tf.train.latest_checkpoint(FLAGS.checkpoint_dir)\n", 81 | "print(checkpoint_file)\n", 82 | "\n", 83 | "vocab_path = os.path.join(FLAGS.checkpoint_dir, \"..\", \"vocab\")\n", 84 | "vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor.restore(vocab_path)\n", 85 | "\n", 86 | "# Map data into position\n", 87 | "position_path = os.path.join(FLAGS.checkpoint_dir, \"..\", \"pos_vocab\")\n", 88 | "pos_vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor.restore(position_path)\n", 89 | "\n", 90 | "test_x = np.array(list(vocab_processor.transform(test_text)))\n", 91 | "test_text = np.array(test_text)\n", 92 | "print(\"\\nText Vocabulary Size: {:d}\".format(len(vocab_processor.vocabulary_)))\n", 93 | "print(\"test_x = {0}\".format(test_x.shape))\n", 94 | "print(\"test_y = {0}\".format(test_y.shape))\n", 95 | "\n", 96 | "test_p1 = np.array(list(pos_vocab_processor.transform(test_pos1)))\n", 97 | "test_p2 = np.array(list(pos_vocab_processor.transform(test_pos2)))\n", 98 | "print(\"\\nPosition Vocabulary Size: {:d}\".format(len(pos_vocab_processor.vocabulary_)))\n", 99 | "print(\"test_p1 = {0}\".format(test_p1.shape))\n", 100 | "print(\"\")\n", 101 | "\n", 102 | "output_alphas = []\n", 103 | "\n", 104 | "graph = tf.Graph()\n", 105 | "with graph.as_default():\n", 106 | " session_conf = tf.ConfigProto(\n", 107 | " allow_soft_placement=FLAGS.allow_soft_placement,\n", 108 | " log_device_placement=FLAGS.log_device_placement)\n", 109 | " session_conf.gpu_options.allow_growth = FLAGS.gpu_allow_growth\n", 110 | " sess = tf.Session(config=session_conf)\n", 111 | " with sess.as_default():\n", 112 | " # Load the saved meta graph and restore variables\n", 113 | " saver = tf.train.import_meta_graph(\"{}.meta\".format(checkpoint_file))\n", 114 | " saver.restore(sess, checkpoint_file)\n", 115 | "\n", 116 | " input_x = graph.get_operation_by_name(\"input_x\").outputs[0]\n", 117 | " input_y = graph.get_operation_by_name(\"input_y\").outputs[0]\n", 118 | " input_text = graph.get_operation_by_name(\"input_text\").outputs[0]\n", 119 | " input_e1 = graph.get_operation_by_name(\"input_e1\").outputs[0]\n", 120 | " input_e2 = graph.get_operation_by_name(\"input_e2\").outputs[0]\n", 121 | " input_p1 = graph.get_operation_by_name(\"input_p1\").outputs[0]\n", 122 | " input_p2 = graph.get_operation_by_name(\"input_p2\").outputs[0]\n", 123 | " emb_dropout_keep_prob = graph.get_operation_by_name(\"emb_dropout_keep_prob\").outputs[0]\n", 124 | " rnn_dropout_keep_prob = graph.get_operation_by_name(\"rnn_dropout_keep_prob\").outputs[0]\n", 125 | " dropout_keep_prob = graph.get_operation_by_name(\"dropout_keep_prob\").outputs[0]\n", 126 | " self_alphas_op = graph.get_operation_by_name(\"self-attention/multihead_attention/Softmax\").outputs[0]\n", 127 | " alphas_op = graph.get_tensor_by_name(\"attention/alphas:0\")\n", 128 | " acc_op = graph.get_tensor_by_name(\"accuracy/accuracy:0\")\n", 129 | "\n", 130 | " print(\"\\nEvaluation:\")\n", 131 | " # Generate batches\n", 132 | " test_batches = data_helpers.batch_iter(list(zip(test_x, test_y, test_text,\n", 133 | " test_e1, test_e2, test_p1, test_p2)),\n", 134 | " FLAGS.batch_size, 1, shuffle=False)\n", 135 | " # Training loop. For each batch...\n", 136 | " accuracy = 0.0\n", 137 | " iter_cnt = 0\n", 138 | " for test_batch in test_batches:\n", 139 | " test_bx, test_by, test_btxt, test_be1, test_be2, test_bp1, test_bp2 = zip(*test_batch)\n", 140 | " feed_dict = {\n", 141 | " input_x: test_bx,\n", 142 | " input_y: test_by,\n", 143 | " input_text: test_btxt,\n", 144 | " input_e1: test_be1,\n", 145 | " input_e2: test_be2,\n", 146 | " input_p1: test_bp1,\n", 147 | " input_p2: test_bp2,\n", 148 | " emb_dropout_keep_prob: 1.0,\n", 149 | " rnn_dropout_keep_prob: 1.0,\n", 150 | " dropout_keep_prob: 1.0\n", 151 | " }\n", 152 | " self_alphas, alphas, acc = sess.run(\n", 153 | " [self_alphas_op, alphas_op, acc_op], feed_dict)\n", 154 | " accuracy += acc\n", 155 | " self_alphas = np.reshape(self_alphas, (FLAGS.num_heads, -1, FLAGS.max_sentence_length, FLAGS.max_sentence_length))\n", 156 | " self_alphas = self_alphas.transpose([1, 0, 2, 3])\n", 157 | " \n", 158 | " output_alphas += self_alphas.tolist()\n", 159 | " \n", 160 | " \n" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 37, 166 | "metadata": {}, 167 | "outputs": [ 168 | { 169 | "data": { 170 | "text/html": [ 171 | "\n", 172 | " \n", 173 | " Layer: \n", 174 | " Attention: \n", 180 | " \n", 181 | "

\n" 182 | ], 183 | "text/plain": [ 184 | "" 185 | ] 186 | }, 187 | "metadata": {}, 188 | "output_type": "execute_result" 189 | }, 190 | { 191 | "data": { 192 | "text/plain": [ 193 | "" 194 | ] 195 | }, 196 | "metadata": {}, 197 | "output_type": "display_data" 198 | }, 199 | { 200 | "data": { 201 | "text/plain": [ 202 | "" 203 | ] 204 | }, 205 | "metadata": {}, 206 | "output_type": "display_data" 207 | } 208 | ], 209 | "source": [ 210 | "idx = 10\n", 211 | "self_alphas = np.array(output_alphas[idx])\n", 212 | "sent = test_text[idx].split()\n", 213 | "sent_len = len(sent)\n", 214 | "s = [self_alphas[0][:sent_len, :sent_len]*20]\n", 215 | "s.append(self_alphas[1][:sent_len, :sent_len]*20)\n", 216 | "s.append(self_alphas[2][:sent_len, :sent_len]*20)\n", 217 | "s.append(self_alphas[3][:sent_len, :sent_len]*20)\n", 218 | "\n", 219 | "for i in range(len(s)):\n", 220 | " for j in range(len(s[i])):\n", 221 | " for k in range(len(s[i][j])):\n", 222 | " if(s[i][j][k]<0.3):\n", 223 | " s[i][j][k]= 0.\n", 224 | "\n", 225 | "\n", 226 | "s = [np.array(s)]\n", 227 | "\n", 228 | "attention.show(sent, sent, s, s, s)" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": 31, 234 | "metadata": { 235 | "scrolled": true 236 | }, 237 | "outputs": [ 238 | { 239 | "data": { 240 | "text/plain": [ 241 | "[0.0025560029316693544,\n", 242 | " 0.003645051270723343,\n", 243 | " 0.005299318116158247,\n", 244 | " 0.013214701786637306,\n", 245 | " 0.013513109646737576,\n", 246 | " 0.010207850486040115,\n", 247 | " 0.004558983258903027,\n", 248 | " 0.004419563338160515,\n", 249 | " 0.02165045589208603,\n", 250 | " 0.009195246733725071,\n", 251 | " 0.02158127725124359,\n", 252 | " 0.003047712380066514,\n", 253 | " 0.012452500872313976,\n", 254 | " 0.011359208263456821,\n", 255 | " 0.011359208263456821,\n", 256 | " 0.011359208263456821,\n", 257 | " 0.011359208263456821,\n", 258 | " 0.011359208263456821,\n", 259 | " 0.011359208263456821,\n", 260 | " 0.011359208263456821,\n", 261 | " 0.011359208263456821,\n", 262 | " 0.011359208263456821,\n", 263 | " 0.011359208263456821,\n", 264 | " 0.011359208263456821,\n", 265 | " 0.011359208263456821,\n", 266 | " 0.011359208263456821,\n", 267 | " 0.011359208263456821,\n", 268 | " 0.011359208263456821,\n", 269 | " 0.011359208263456821,\n", 270 | " 0.011359208263456821,\n", 271 | " 0.011359208263456821,\n", 272 | " 0.011359208263456821,\n", 273 | " 0.011359208263456821,\n", 274 | " 0.011359208263456821,\n", 275 | " 0.011359208263456821,\n", 276 | " 0.011359208263456821,\n", 277 | " 0.011359208263456821,\n", 278 | " 0.011359208263456821,\n", 279 | " 0.011359208263456821,\n", 280 | " 0.011359208263456821,\n", 281 | " 0.011359208263456821,\n", 282 | " 0.011359208263456821,\n", 283 | " 0.011359208263456821,\n", 284 | " 0.011359208263456821,\n", 285 | " 0.011359208263456821,\n", 286 | " 0.011359208263456821,\n", 287 | " 0.011359208263456821,\n", 288 | " 0.011359208263456821,\n", 289 | " 0.011359208263456821,\n", 290 | " 0.011359208263456821,\n", 291 | " 0.011359208263456821,\n", 292 | " 0.011359208263456821,\n", 293 | " 0.011359208263456821,\n", 294 | " 0.011359208263456821,\n", 295 | " 0.011359208263456821,\n", 296 | " 0.011359208263456821,\n", 297 | " 0.011359208263456821,\n", 298 | " 0.011359208263456821,\n", 299 | " 0.011359208263456821,\n", 300 | " 0.011359208263456821,\n", 301 | " 0.011359208263456821,\n", 302 | " 0.011359208263456821,\n", 303 | " 0.011359208263456821,\n", 304 | " 0.011359208263456821,\n", 305 | " 0.011359208263456821,\n", 306 | " 0.011359208263456821,\n", 307 | " 0.011359208263456821,\n", 308 | " 0.011359208263456821,\n", 309 | " 0.011359208263456821,\n", 310 | " 0.011359208263456821,\n", 311 | " 0.011359208263456821,\n", 312 | " 0.011359208263456821,\n", 313 | " 0.011359208263456821,\n", 314 | " 0.011359208263456821,\n", 315 | " 0.011359208263456821,\n", 316 | " 0.011359208263456821,\n", 317 | " 0.011359208263456821,\n", 318 | " 0.011359208263456821,\n", 319 | " 0.011359208263456821,\n", 320 | " 0.011359208263456821,\n", 321 | " 0.011359208263456821,\n", 322 | " 0.011359208263456821,\n", 323 | " 0.011359208263456821,\n", 324 | " 0.011359208263456821,\n", 325 | " 0.011359208263456821,\n", 326 | " 0.011359208263456821,\n", 327 | " 0.011359208263456821,\n", 328 | " 0.011359208263456821,\n", 329 | " 0.011359208263456821,\n", 330 | " 0.011359208263456821]" 331 | ] 332 | }, 333 | "execution_count": 31, 334 | "metadata": {}, 335 | "output_type": "execute_result" 336 | } 337 | ], 338 | "source": [ 339 | "output_alphas[0][0][0]" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "metadata": { 346 | "collapsed": true 347 | }, 348 | "outputs": [], 349 | "source": [] 350 | } 351 | ], 352 | "metadata": { 353 | "kernelspec": { 354 | "display_name": "Python 3", 355 | "language": "python", 356 | "name": "python3" 357 | }, 358 | "language_info": { 359 | "codemirror_mode": { 360 | "name": "ipython", 361 | "version": 3 362 | }, 363 | "file_extension": ".py", 364 | "mimetype": "text/x-python", 365 | "name": "python", 366 | "nbconvert_exporter": "python", 367 | "pygments_lexer": "ipython3", 368 | "version": "3.6.3" 369 | } 370 | }, 371 | "nbformat": 4, 372 | "nbformat_minor": 2 373 | } 374 | -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | import os 2 | import time 3 | import numpy as np 4 | import tensorflow as tf 5 | 6 | import data_helpers 7 | from configure import FLAGS 8 | from logger import Logger 9 | from model.entity_att_lstm import EntityAttentionLSTM 10 | import utils 11 | 12 | import warnings 13 | import sklearn.exceptions 14 | warnings.filterwarnings("ignore", category=sklearn.exceptions.UndefinedMetricWarning) 15 | 16 | 17 | def train(): 18 | with tf.device('/cpu:0'): 19 | train_text, train_y, train_e1, train_e2, train_pos1, train_pos2 = data_helpers.load_data_and_labels(FLAGS.train_path) 20 | with tf.device('/cpu:0'): 21 | test_text, test_y, test_e1, test_e2, test_pos1, test_pos2 = data_helpers.load_data_and_labels(FLAGS.test_path) 22 | 23 | # Build vocabulary 24 | # Example: x_text[3] = "A misty ridge uprises from the surge." 25 | # ['a misty ridge uprises from the surge ... '] 26 | # => 27 | # [27 39 40 41 42 1 43 0 0 ... 0] 28 | # dimension = MAX_SENTENCE_LENGTH 29 | vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor(FLAGS.max_sentence_length) 30 | vocab_processor.fit(train_text + test_text) 31 | train_x = np.array(list(vocab_processor.transform(train_text))) 32 | test_x = np.array(list(vocab_processor.transform(test_text))) 33 | train_text = np.array(train_text) 34 | test_text = np.array(test_text) 35 | print("\nText Vocabulary Size: {:d}".format(len(vocab_processor.vocabulary_))) 36 | print("train_x = {0}".format(train_x.shape)) 37 | print("train_y = {0}".format(train_y.shape)) 38 | print("test_x = {0}".format(test_x.shape)) 39 | print("test_y = {0}".format(test_y.shape)) 40 | 41 | # Example: pos1[3] = [-2 -1 0 1 2 3 4 999 999 999 ... 999] 42 | # [95 96 97 98 99 100 101 999 999 999 ... 999] 43 | # => 44 | # [11 12 13 14 15 16 21 17 17 17 ... 17] 45 | # dimension = MAX_SENTENCE_LENGTH 46 | pos_vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor(FLAGS.max_sentence_length) 47 | pos_vocab_processor.fit(train_pos1 + train_pos2 + test_pos1 + test_pos2) 48 | train_p1 = np.array(list(pos_vocab_processor.transform(train_pos1))) 49 | train_p2 = np.array(list(pos_vocab_processor.transform(train_pos2))) 50 | test_p1 = np.array(list(pos_vocab_processor.transform(test_pos1))) 51 | test_p2 = np.array(list(pos_vocab_processor.transform(test_pos2))) 52 | print("\nPosition Vocabulary Size: {:d}".format(len(pos_vocab_processor.vocabulary_))) 53 | print("train_p1 = {0}".format(train_p1.shape)) 54 | print("test_p1 = {0}".format(test_p1.shape)) 55 | print("") 56 | 57 | with tf.Graph().as_default(): 58 | session_conf = tf.ConfigProto( 59 | allow_soft_placement=FLAGS.allow_soft_placement, 60 | log_device_placement=FLAGS.log_device_placement) 61 | session_conf.gpu_options.allow_growth = FLAGS.gpu_allow_growth 62 | sess = tf.Session(config=session_conf) 63 | with sess.as_default(): 64 | model = EntityAttentionLSTM( 65 | sequence_length=train_x.shape[1], 66 | num_classes=train_y.shape[1], 67 | vocab_size=len(vocab_processor.vocabulary_), 68 | embedding_size=FLAGS.embedding_size, 69 | pos_vocab_size=len(pos_vocab_processor.vocabulary_), 70 | pos_embedding_size=FLAGS.pos_embedding_size, 71 | hidden_size=FLAGS.hidden_size, 72 | num_heads=FLAGS.num_heads, 73 | attention_size=FLAGS.attention_size, 74 | use_elmo=(FLAGS.embeddings == 'elmo'), 75 | l2_reg_lambda=FLAGS.l2_reg_lambda) 76 | 77 | # Define Training procedure 78 | global_step = tf.Variable(0, name="global_step", trainable=False) 79 | optimizer = tf.train.AdadeltaOptimizer(FLAGS.learning_rate, FLAGS.decay_rate, 1e-6) 80 | gvs = optimizer.compute_gradients(model.loss) 81 | capped_gvs = [(tf.clip_by_value(grad, -1.0, 1.0), var) for grad, var in gvs] 82 | train_op = optimizer.apply_gradients(capped_gvs, global_step=global_step) 83 | 84 | # Output directory for models and summaries 85 | timestamp = str(int(time.time())) 86 | out_dir = os.path.abspath(os.path.join(os.path.curdir, "runs", timestamp)) 87 | print("\nWriting to {}\n".format(out_dir)) 88 | 89 | # Logger 90 | logger = Logger(out_dir) 91 | 92 | # Summaries for loss and accuracy 93 | loss_summary = tf.summary.scalar("loss", model.loss) 94 | acc_summary = tf.summary.scalar("accuracy", model.accuracy) 95 | 96 | # Train Summaries 97 | train_summary_op = tf.summary.merge([loss_summary, acc_summary]) 98 | train_summary_dir = os.path.join(out_dir, "summaries", "train") 99 | train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph) 100 | 101 | # Checkpoint directory. Tensorflow assumes this directory already exists so we need to create it 102 | checkpoint_dir = os.path.abspath(os.path.join(out_dir, "checkpoints")) 103 | checkpoint_prefix = os.path.join(checkpoint_dir, "model") 104 | if not os.path.exists(checkpoint_dir): 105 | os.makedirs(checkpoint_dir) 106 | saver = tf.train.Saver(tf.global_variables(), max_to_keep=FLAGS.num_checkpoints) 107 | 108 | # Write vocabulary 109 | vocab_processor.save(os.path.join(out_dir, "vocab")) 110 | pos_vocab_processor.save(os.path.join(out_dir, "pos_vocab")) 111 | 112 | # Initialize all variables 113 | sess.run(tf.global_variables_initializer()) 114 | 115 | if FLAGS.embeddings == "word2vec": 116 | pretrain_W = utils.load_word2vec('resource/GoogleNews-vectors-negative300.bin', FLAGS.embedding_size, vocab_processor) 117 | sess.run(model.W_text.assign(pretrain_W)) 118 | print("Success to load pre-trained word2vec model!\n") 119 | elif FLAGS.embeddings == "glove100": 120 | pretrain_W = utils.load_glove('resource/glove.6B.100d.txt', FLAGS.embedding_size, vocab_processor) 121 | sess.run(model.W_text.assign(pretrain_W)) 122 | print("Success to load pre-trained glove100 model!\n") 123 | elif FLAGS.embeddings == "glove300": 124 | pretrain_W = utils.load_glove('resource/glove.840B.300d.txt', FLAGS.embedding_size, vocab_processor) 125 | sess.run(model.W_text.assign(pretrain_W)) 126 | print("Success to load pre-trained glove300 model!\n") 127 | 128 | # Generate batches 129 | train_batches = data_helpers.batch_iter(list(zip(train_x, train_y, train_text, 130 | train_e1, train_e2, train_p1, train_p2)), 131 | FLAGS.batch_size, FLAGS.num_epochs) 132 | # Training loop. For each batch... 133 | best_f1 = 0.0 # For save checkpoint(model) 134 | for train_batch in train_batches: 135 | train_bx, train_by, train_btxt, train_be1, train_be2, train_bp1, train_bp2 = zip(*train_batch) 136 | feed_dict = { 137 | model.input_x: train_bx, 138 | model.input_y: train_by, 139 | model.input_text: train_btxt, 140 | model.input_e1: train_be1, 141 | model.input_e2: train_be2, 142 | model.input_p1: train_bp1, 143 | model.input_p2: train_bp2, 144 | model.emb_dropout_keep_prob: FLAGS.emb_dropout_keep_prob, 145 | model.rnn_dropout_keep_prob: FLAGS.rnn_dropout_keep_prob, 146 | model.dropout_keep_prob: FLAGS.dropout_keep_prob 147 | } 148 | _, step, summaries, loss, accuracy = sess.run( 149 | [train_op, global_step, train_summary_op, model.loss, model.accuracy], feed_dict) 150 | train_summary_writer.add_summary(summaries, step) 151 | 152 | # Training log display 153 | if step % FLAGS.display_every == 0: 154 | logger.logging_train(step, loss, accuracy) 155 | 156 | # Evaluation 157 | if step % FLAGS.evaluate_every == 0: 158 | print("\nEvaluation:") 159 | # Generate batches 160 | test_batches = data_helpers.batch_iter(list(zip(test_x, test_y, test_text, 161 | test_e1, test_e2, test_p1, test_p2)), 162 | FLAGS.batch_size, 1, shuffle=False) 163 | # Training loop. For each batch... 164 | losses = 0.0 165 | accuracy = 0.0 166 | predictions = [] 167 | iter_cnt = 0 168 | for test_batch in test_batches: 169 | test_bx, test_by, test_btxt, test_be1, test_be2, test_bp1, test_bp2 = zip(*test_batch) 170 | feed_dict = { 171 | model.input_x: test_bx, 172 | model.input_y: test_by, 173 | model.input_text: test_btxt, 174 | model.input_e1: test_be1, 175 | model.input_e2: test_be2, 176 | model.input_p1: test_bp1, 177 | model.input_p2: test_bp2, 178 | model.emb_dropout_keep_prob: 1.0, 179 | model.rnn_dropout_keep_prob: 1.0, 180 | model.dropout_keep_prob: 1.0 181 | } 182 | loss, acc, pred = sess.run( 183 | [model.loss, model.accuracy, model.predictions], feed_dict) 184 | losses += loss 185 | accuracy += acc 186 | predictions += pred.tolist() 187 | iter_cnt += 1 188 | losses /= iter_cnt 189 | accuracy /= iter_cnt 190 | predictions = np.array(predictions, dtype='int') 191 | 192 | logger.logging_eval(step, loss, accuracy, predictions) 193 | 194 | # Model checkpoint 195 | if best_f1 < logger.best_f1: 196 | best_f1 = logger.best_f1 197 | path = saver.save(sess, checkpoint_prefix+"-{:.3g}".format(best_f1), global_step=step) 198 | print("Saved model checkpoint to {}\n".format(path)) 199 | 200 | 201 | def main(_): 202 | train() 203 | 204 | 205 | if __name__ == "__main__": 206 | tf.app.run() 207 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | 4 | class2label = {'Other': 0, 5 | 'Message-Topic(e1,e2)': 1, 'Message-Topic(e2,e1)': 2, 6 | 'Product-Producer(e1,e2)': 3, 'Product-Producer(e2,e1)': 4, 7 | 'Instrument-Agency(e1,e2)': 5, 'Instrument-Agency(e2,e1)': 6, 8 | 'Entity-Destination(e1,e2)': 7, 'Entity-Destination(e2,e1)': 8, 9 | 'Cause-Effect(e1,e2)': 9, 'Cause-Effect(e2,e1)': 10, 10 | 'Component-Whole(e1,e2)': 11, 'Component-Whole(e2,e1)': 12, 11 | 'Entity-Origin(e1,e2)': 13, 'Entity-Origin(e2,e1)': 14, 12 | 'Member-Collection(e1,e2)': 15, 'Member-Collection(e2,e1)': 16, 13 | 'Content-Container(e1,e2)': 17, 'Content-Container(e2,e1)': 18} 14 | 15 | label2class = {0: 'Other', 16 | 1: 'Message-Topic(e1,e2)', 2: 'Message-Topic(e2,e1)', 17 | 3: 'Product-Producer(e1,e2)', 4: 'Product-Producer(e2,e1)', 18 | 5: 'Instrument-Agency(e1,e2)', 6: 'Instrument-Agency(e2,e1)', 19 | 7: 'Entity-Destination(e1,e2)', 8: 'Entity-Destination(e2,e1)', 20 | 9: 'Cause-Effect(e1,e2)', 10: 'Cause-Effect(e2,e1)', 21 | 11: 'Component-Whole(e1,e2)', 12: 'Component-Whole(e2,e1)', 22 | 13: 'Entity-Origin(e1,e2)', 14: 'Entity-Origin(e2,e1)', 23 | 15: 'Member-Collection(e1,e2)', 16: 'Member-Collection(e2,e1)', 24 | 17: 'Content-Container(e1,e2)', 18: 'Content-Container(e2,e1)'} 25 | 26 | 27 | def initializer(): 28 | return tf.keras.initializers.glorot_normal() 29 | 30 | 31 | def load_word2vec(word2vec_path, embedding_dim, vocab): 32 | # initial matrix with random uniform 33 | initW = np.random.randn(len(vocab.vocabulary_), embedding_dim).astype(np.float32) * np.sqrt(2.0 / len(vocab.vocabulary_)) 34 | # load any vectors from the word2vec 35 | print("Load word2vec file {0}".format(word2vec_path)) 36 | with open(word2vec_path, "rb") as f: 37 | header = f.readline() 38 | vocab_size, layer1_size = map(int, header.split()) 39 | binary_len = np.dtype('float32').itemsize * layer1_size 40 | for line in range(vocab_size): 41 | word = [] 42 | while True: 43 | ch = f.read(1).decode('latin-1') 44 | if ch == ' ': 45 | word = ''.join(word) 46 | break 47 | if ch != '\n': 48 | word.append(ch) 49 | idx = vocab.vocabulary_.get(word) 50 | if idx != 0: 51 | initW[idx] = np.fromstring(f.read(binary_len), dtype='float32') 52 | else: 53 | f.read(binary_len) 54 | return initW 55 | 56 | 57 | def load_glove(word2vec_path, embedding_dim, vocab): 58 | # initial matrix with random uniform 59 | initW = np.random.randn(len(vocab.vocabulary_), embedding_dim).astype(np.float32) * np.sqrt(2.0 / len(vocab.vocabulary_)) 60 | # load any vectors from the word2vec 61 | print("Load glove file {0}".format(word2vec_path)) 62 | f = open(word2vec_path, 'r', encoding='utf8') 63 | for line in f: 64 | splitLine = line.split(' ') 65 | word = splitLine[0] 66 | embedding = np.asarray(splitLine[1:], dtype='float32') 67 | idx = vocab.vocabulary_.get(word) 68 | if idx != 0: 69 | initW[idx] = embedding 70 | return initW 71 | -------------------------------------------------------------------------------- /visualize.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import tensorflow as tf 4 | import data_helpers 5 | import logger 6 | from configure import FLAGS 7 | import warnings 8 | import sklearn.exceptions 9 | 10 | warnings.filterwarnings("ignore", category=sklearn.exceptions.UndefinedMetricWarning) 11 | 12 | from tensor2tensor.visualization import attention 13 | 14 | 15 | def visualize(): 16 | with tf.device('/cpu:0'): 17 | test_text, test_y, test_e1, test_e2, test_pos1, test_pos2 = data_helpers.load_data_and_labels(FLAGS.test_path) 18 | 19 | checkpoint_file = tf.train.latest_checkpoint(FLAGS.checkpoint_dir) 20 | print(checkpoint_file) 21 | 22 | vocab_path = os.path.join(FLAGS.checkpoint_dir, "..", "vocab") 23 | vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor.restore(vocab_path) 24 | 25 | # Map data into position 26 | position_path = os.path.join(FLAGS.checkpoint_dir, "..", "pos_vocab") 27 | pos_vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor.restore(position_path) 28 | 29 | test_x = np.array(list(vocab_processor.transform(test_text))) 30 | test_text = np.array(test_text) 31 | print("\nText Vocabulary Size: {:d}".format(len(vocab_processor.vocabulary_))) 32 | print("test_x = {0}".format(test_x.shape)) 33 | print("test_y = {0}".format(test_y.shape)) 34 | 35 | test_p1 = np.array(list(pos_vocab_processor.transform(test_pos1))) 36 | test_p2 = np.array(list(pos_vocab_processor.transform(test_pos2))) 37 | print("\nPosition Vocabulary Size: {:d}".format(len(pos_vocab_processor.vocabulary_))) 38 | print("test_p1 = {0}".format(test_p1.shape)) 39 | print("") 40 | 41 | graph = tf.Graph() 42 | with graph.as_default(): 43 | session_conf = tf.ConfigProto( 44 | allow_soft_placement=FLAGS.allow_soft_placement, 45 | log_device_placement=FLAGS.log_device_placement) 46 | session_conf.gpu_options.allow_growth = FLAGS.gpu_allow_growth 47 | sess = tf.Session(config=session_conf) 48 | with sess.as_default(): 49 | # Load the saved meta graph and restore variables 50 | saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file)) 51 | saver.restore(sess, checkpoint_file) 52 | 53 | input_x = graph.get_operation_by_name("input_x").outputs[0] 54 | input_y = graph.get_operation_by_name("input_y").outputs[0] 55 | input_text = graph.get_operation_by_name("input_text").outputs[0] 56 | input_e1 = graph.get_operation_by_name("input_e1").outputs[0] 57 | input_e2 = graph.get_operation_by_name("input_e2").outputs[0] 58 | input_p1 = graph.get_operation_by_name("input_p1").outputs[0] 59 | input_p2 = graph.get_operation_by_name("input_p2").outputs[0] 60 | emb_dropout_keep_prob = graph.get_operation_by_name("emb_dropout_keep_prob").outputs[0] 61 | rnn_dropout_keep_prob = graph.get_operation_by_name("rnn_dropout_keep_prob").outputs[0] 62 | dropout_keep_prob = graph.get_operation_by_name("dropout_keep_prob").outputs[0] 63 | self_alphas_op = graph.get_operation_by_name("self-attention/multihead_attention/Softmax").outputs[0] 64 | alphas_op = graph.get_operation_by_name("attention/alphas").outputs[0] 65 | acc_op = graph.get_operation_by_name("accuracy/accuracy").outputs[0] 66 | e2_alphas_op = graph.get_operation_by_name("attention/e2_alphas").outputs[0] 67 | e1_alphas_op = graph.get_operation_by_name("attention/e1_alphas").outputs[0] 68 | latent_type_op = graph.get_operation_by_name("attention/latent_type").outputs[0] 69 | 70 | print("\nEvaluation:") 71 | # Generate batches 72 | test_batches = data_helpers.batch_iter(list(zip(test_x, test_y, test_text, 73 | test_e1, test_e2, test_p1, test_p2)), 74 | FLAGS.batch_size, 1, shuffle=False) 75 | # Training loop. For each batch... 76 | accuracy = 0.0 77 | iter_cnt = 0 78 | with open("visualization.html", "w") as html_file: 79 | for test_batch in test_batches: 80 | test_bx, test_by, test_btxt, test_be1, test_be2, test_bp1, test_bp2 = zip(*test_batch) 81 | feed_dict = { 82 | input_x: test_bx, 83 | input_y: test_by, 84 | input_text: test_btxt, 85 | input_e1: test_be1, 86 | input_e2: test_be2, 87 | input_p1: test_bp1, 88 | input_p2: test_bp2, 89 | emb_dropout_keep_prob: 1.0, 90 | rnn_dropout_keep_prob: 1.0, 91 | dropout_keep_prob: 1.0 92 | } 93 | self_alphas, alphas, acc, e1_alphas, e2_alphas, latent_type = sess.run( 94 | [self_alphas_op, alphas_op, acc_op, e1_alphas_op, e2_alphas_op, latent_type_op], feed_dict) 95 | accuracy += acc 96 | iter_cnt += 1 97 | for text, alphas_values in zip(test_btxt, alphas): 98 | for word, alpha in zip(text.split(), alphas_values / alphas_values.max()): 99 | html_file.write( 100 | '%s\n' % (alpha, word)) 101 | html_file.write('
') 102 | accuracy /= iter_cnt 103 | print(accuracy) 104 | 105 | 106 | def main(_): 107 | visualize() 108 | 109 | 110 | if __name__ == "__main__": 111 | tf.app.run() 112 | --------------------------------------------------------------------------------