├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── constants.py ├── gap-development.tsv ├── gap-test.tsv ├── gap-validation.tsv └── gap_scorer.py /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | We'd love to accept your patches and contributions to this project. There are 4 | just a few small guidelines you need to follow. 5 | 6 | ## Contributor License Agreement 7 | 8 | Contributions to this project must be accompanied by a Contributor License 9 | Agreement. You (or your employer) retain the copyright to your contribution; 10 | this simply gives us permission to use and redistribute your contributions as 11 | part of the project. Head over to to see 12 | your current agreements on file or to sign a new one. 13 | 14 | You generally only need to submit a CLA once, so if you've already submitted one 15 | (even if it was for a different project), you probably don't need to do it 16 | again. 17 | 18 | ## Code reviews 19 | 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. Consult 22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 23 | information on using pull requests. 24 | 25 | ## Community Guidelines 26 | 27 | This project follows [Google's Open Source Community 28 | Guidelines](https://opensource.google.com/conduct/). 29 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GAP Coreference Dataset 2 | 3 | GAP is a gender-balanced dataset containing 8,908 coreference-labeled pairs of (ambiguous pronoun, antecedent name), sampled from Wikipedia and released by [Google AI Language](https://ai.google/research/teams/language/) for the evaluation of coreference resolution in practical applications. 4 | 5 | http://goo.gl/language/gap-coreference 6 | 7 | ## Motivation 8 | 9 | Coreference resolution is an important task for natural language understanding and the resolution of ambiguous pronouns a longstanding challenge. 10 | Nonetheless, existing corpora do not capture ambiguous pronouns in sufficient volume or diversity to accurately indicate the practical utility of models. 11 | 12 | [Google AI Language's](https://ai.google/research/teams/language/) GAP dataset is an evaluation benchmark comprising 8,908 coreference-labeled pairs of (ambiguous pronoun, antecedent name), sampled from Wikipedia to provide diverse coverage of challenges posed by real-world text. 13 | Importantly, GAP is gender-balanced to address the gender bias in coreference systems noted in our and other's analysis. 14 | 15 | More details are available in [our paper](https://arxiv.org/abs/1810.05201) (which should be cited if you use or discuss GAP in your work): 16 | 17 |

18 | 
19 | @inproceedings{webster2018gap,
20 |   title =     {Mind the GAP: A Balanced Corpus of Gendered Ambiguou},
21 |   author =    {Webster, Kellie and Recasens, Marta and Axelrod, Vera and Baldridge, Jason},
22 |   booktitle = {Transactions of the ACL},
23 |   year =      {2018},
24 |   pages =     {to appear},
25 | }
26 |
27 | 28 | ## Dataset Description 29 | 30 | The GAP dataset release comprises three .tsv files, each with eleven columns. 31 | 32 | The files are: 33 | * **test** 4,000 pairs, to be used for official evaluation 34 | * **development** 4,000 pairs, may be used for model development 35 | * **validation** 908 pairs, may be used for parameter tuning 36 | 37 | The columns contain: 38 | 39 | Column | Header | Description 40 | :-----:|----------------|-------------------------------------------- 41 | 1 | ID | Unique identifer for an example (two pairs) 42 | 2 | Text | Text containing the ambiguous pronoun and two candidate names. About a paragraph in length 43 | 3 | Pronoun | The pronoun, text 44 | 4 | Pronoun-offset | Character offset of Pronoun in Column 2 (Text) 45 | 5 | A ^ | The first name, text 46 | 6 | A-offset | Character offset of A in Column 2 (Text) 47 | 7 | A-coref | Whether A corefers with the pronoun, TRUE or FALSE 48 | 8 | B ^ | The second name, text 49 | 9 | B-offset | Character offset of B in Column 2 (Text) 50 | 10 | B-coref | Whether B corefers with the pronoun, TRUE or FALSE 51 | 11 | URL ^^ | The URL of the source Wikipedia page 52 | 53 | ^ Please note that systems should detect mentions for inference automatically, and access labeled spans only to output predictions. 54 | 55 | ^^ Please also note that there are two task settings, *snippet-context* in which the URL column may **not** be used, and *page-context* where the URL, and the denoted Wikipedia page, may be used. 56 | 57 | ## Benchmarks 58 | 59 | Performance on GAP may be benchmarked against the syntactic parallelism baseline from our above paper on the test set: 60 | 61 | Task Setting | M | F | B | O 62 | :----------------:|------|------|--------|------ 63 | *snippet-context* | 69.4 | 64.4 | *0.93* | 66.9 64 | *page-context* | 72.3 | 68.8 | *0.95* | 70.6 65 | 66 | where the metrics are F1 score on **M**asculine and **F**eminine examples, **O**verall, and a **B**ias factor calculated as **F** / **M**. 67 | 68 | ## Contact 69 | To contact us, please use gap-coreference@google.com 70 | -------------------------------------------------------------------------------- /constants.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright 2018 Google LLC 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | """Constants. 18 | """ 19 | 20 | from enum import Enum 21 | 22 | 23 | class Gender(Enum): 24 | UNKNOWN = 0 25 | MASCULINE = 1 26 | FEMININE = 2 27 | 28 | 29 | # Mapping of (lowercased) pronoun form to gender value. Note that reflexives 30 | # are not included in GAP, so do not appear here. 31 | PRONOUNS = { 32 | 'she': Gender.FEMININE, 33 | 'her': Gender.FEMININE, 34 | 'hers': Gender.FEMININE, 35 | 'he': Gender.MASCULINE, 36 | 'his': Gender.MASCULINE, 37 | 'him': Gender.MASCULINE, 38 | } 39 | 40 | # Fieldnames used in the gold dataset .tsv file. 41 | GOLD_FIELDNAMES = [ 42 | 'ID', 'Text', 'Pronoun', 'Pronoun-offset', 'A', 'A-offset', 'A-coref', 'B', 43 | 'B-offset', 'B-coref', 'URL' 44 | ] 45 | 46 | # Fieldnames expected in system output .tsv files. 47 | SYSTEM_FIELDNAMES = ['ID', 'A-coref', 'B-coref'] 48 | -------------------------------------------------------------------------------- /gap_scorer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright 2018 Google LLC 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | """Scores system output for the GAP challenge. 18 | """ 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import argparse 23 | from collections import defaultdict 24 | import csv 25 | 26 | from constants import Gender 27 | from constants import GOLD_FIELDNAMES 28 | from constants import PRONOUNS 29 | from constants import SYSTEM_FIELDNAMES 30 | 31 | 32 | class Annotation(object): 33 | """Container class for storing annotations of an example. 34 | 35 | Attributes: 36 | gender(None): The gender of the annotation. None indicates that gender was 37 | not determined for the given example. 38 | name_a_coref(None): bool reflecting whether Name A was recorded as 39 | coreferential with the target pronoun for this example. None indicates 40 | that no annotation was found for the given example. 41 | name_b_coref(None): bool reflecting whether Name B was recorded as 42 | coreferential with the target pronoun for this example. None indicates 43 | that no annotation was found for the given example. 44 | """ 45 | 46 | def __init__(self): 47 | self.gender = None 48 | self.name_a_coref = None 49 | self.name_b_coref = None 50 | 51 | 52 | class Scores(object): 53 | """Container class for storing scores, and generating evaluation metrics. 54 | 55 | Attributes: 56 | true_positives: Tally of true positives seen. 57 | false_positives: Tally of false positives seen. 58 | true_negatives: Tally of true negatives seen. 59 | false_negatives: Tally of false negatives seen. 60 | """ 61 | 62 | def __init__(self): 63 | self.true_positives = 0 64 | self.false_positives = 0 65 | self.true_negatives = 0 66 | self.false_negatives = 0 67 | 68 | def recall(self): 69 | """Calculates recall based on the observed scores. 70 | 71 | Returns: 72 | float, the recall. 73 | """ 74 | numerator = self.true_positives 75 | denominator = self.true_positives + self.false_negatives 76 | return 100.0 * numerator / denominator if denominator else 0.0 77 | 78 | def precision(self): 79 | """Calculates precision based on the observed scores. 80 | 81 | Returns: 82 | float, the precision. 83 | """ 84 | numerator = self.true_positives 85 | denominator = self.true_positives + self.false_positives 86 | return 100.0 * numerator / denominator if denominator else 0.0 87 | 88 | def f1(self): 89 | """Calculates F1 based on the observed scores. 90 | 91 | Returns: 92 | float, the F1 score. 93 | """ 94 | recall = self.recall() 95 | precision = self.precision() 96 | 97 | numerator = 2 * precision * recall 98 | denominator = precision + recall 99 | return numerator / denominator if denominator else 0.0 100 | 101 | 102 | def read_annotations(filename, is_gold): 103 | """Reads coreference annotations for the examples in the given file. 104 | 105 | Args: 106 | filename: Path to .tsv file to read. 107 | is_gold: Whether or not we are reading the gold annotations. 108 | 109 | Returns: 110 | A dict mapping example ID strings to their Annotation representation. If 111 | reading gold, 'Pronoun' field is used to determine gender. 112 | """ 113 | 114 | def is_true(value): 115 | if value.lower() == 'true': 116 | return True 117 | elif value.lower() == 'false': 118 | return False 119 | else: 120 | print('Unexpected label!', value) 121 | return None 122 | 123 | fieldnames = GOLD_FIELDNAMES if is_gold else SYSTEM_FIELDNAMES 124 | 125 | annotations = defaultdict(Annotation) 126 | with open(filename, 'rU') as f: 127 | reader = csv.DictReader(f, fieldnames=fieldnames, delimiter='\t') 128 | 129 | # Skip the header line in the gold data 130 | if is_gold: 131 | next(reader, None) 132 | 133 | for row in reader: 134 | example_id = row['ID'] 135 | if example_id in annotations: 136 | print('Multiple annotations for', example_id) 137 | continue 138 | 139 | annotations[example_id].name_a_coref = is_true(row['A-coref']) 140 | annotations[example_id].name_b_coref = is_true(row['B-coref']) 141 | if is_gold: 142 | gender = PRONOUNS.get(row['Pronoun'].lower(), Gender.UNKNOWN) 143 | assert gender != Gender.UNKNOWN, row 144 | annotations[example_id].gender = gender 145 | return annotations 146 | 147 | 148 | def calculate_scores(gold_annotations, system_annotations): 149 | """Score the system annotations against gold. 150 | 151 | Args: 152 | gold_annotations: dict from example ID to its gold Annotation. 153 | system_annotations: dict from example ID to its system Annotation. 154 | 155 | Returns: 156 | A dict from gender to a Scores object for that gender. None is used to 157 | denote no specific gender, i.e. overall scores. 158 | """ 159 | scores = {} 160 | for example_id, gold_annotation in gold_annotations.iteritems(): 161 | system_annotation = system_annotations[example_id] 162 | 163 | name_a_annotations = [ 164 | gold_annotation.name_a_coref, system_annotation.name_a_coref 165 | ] 166 | name_b_annotations = [ 167 | gold_annotation.name_b_coref, system_annotation.name_b_coref 168 | ] 169 | for gender in [None, gold_annotation.gender]: 170 | if gender not in scores: 171 | scores[gender] = Scores() 172 | 173 | for (gold, system) in [name_a_annotations, name_b_annotations]: 174 | if system is None: 175 | print('Missing output for', example_id) 176 | scores[gender].false_negatives += 1 177 | elif gold and system: 178 | scores[gender].true_positives += 1 179 | elif not gold and system: 180 | scores[gender].false_positives += 1 181 | elif not gold and not system: 182 | scores[gender].true_negatives += 1 183 | elif gold and not system: 184 | scores[gender].false_negatives += 1 185 | return scores 186 | 187 | 188 | def make_scorecard(scores): 189 | """Returns a human-readable scorecard of the given scores. 190 | 191 | Args: 192 | scores: dict from gender to its Scores object. None is used to denote no 193 | specific gender, i.e. overall scores. 194 | 195 | Returns: 196 | A string, the scorecard. 197 | """ 198 | scorecard = [] 199 | 200 | display_names = [(None, 'Overall'), (Gender.MASCULINE, 'Masculine'), 201 | (Gender.FEMININE, 'Feminine')] 202 | 203 | bias_terms = {} 204 | for gender, display_name in display_names: 205 | gender_scores = scores.get(gender, Scores()) 206 | 207 | recall = gender_scores.recall() 208 | precision = gender_scores.precision() 209 | f1 = gender_scores.f1() 210 | bias_terms[gender] = f1 211 | 212 | scorecard.append('{} recall: {:.1f} precision: {:.1f} f1: {:.1f}'.format( 213 | display_name, recall, precision, f1)) 214 | scorecard.append('\t\ttp {:d}\tfp {:d}'.format( 215 | gender_scores.true_positives, gender_scores.false_positives)) 216 | scorecard.append('\t\tfn {:d}\ttn {:d}'.format( 217 | gender_scores.false_negatives, gender_scores.true_negatives)) 218 | 219 | bias = '-' 220 | if bias_terms[Gender.MASCULINE] and bias_terms[Gender.FEMININE]: 221 | bias = '{:.2f}'.format( 222 | bias_terms[Gender.FEMININE] / bias_terms[Gender.MASCULINE]) 223 | scorecard.append('Bias (F/M): {}\n'.format(bias)) 224 | return '\n'.join(scorecard) 225 | 226 | 227 | def run_scorer(gold_tsv, system_tsv): 228 | """Run the scorer. 229 | 230 | Args: 231 | gold_tsv: Gold annotations to score against. 232 | system_tsv: System output to score. 233 | 234 | Returns: 235 | A string, the scorecard. 236 | """ 237 | gold_annotations = read_annotations(gold_tsv, is_gold=True) 238 | assert gold_annotations, 'No gold annotations read!' 239 | 240 | system_annotations = read_annotations(system_tsv, is_gold=False) 241 | assert system_annotations, 'No system annotations read!' 242 | 243 | scores = calculate_scores(gold_annotations, system_annotations) 244 | return make_scorecard(scores) 245 | 246 | 247 | def main(args): 248 | """Score system output against gold and display the scorecard. 249 | 250 | Args: 251 | args: argparse namespace containing gold_tsv and system_tsv. 252 | """ 253 | scorecard = run_scorer(args.gold_tsv, args.system_tsv) 254 | print(scorecard) 255 | 256 | 257 | if __name__ == '__main__': 258 | parser = argparse.ArgumentParser() 259 | parser.add_argument( 260 | '--gold_tsv', 261 | dest='gold_tsv', 262 | required=True, 263 | help='Path to the gold .tsv to score against. First line should contain' 264 | ' header information and is ignored.') 265 | parser.add_argument( 266 | '--system_tsv', 267 | dest='system_tsv', 268 | required=True, 269 | help='Path to the system .tsv to score. All lines are read.') 270 | main(parser.parse_args()) 271 | --------------------------------------------------------------------------------