└── README.md


/README.md:
--------------------------------------------------------------------------------
  1 | # Seahorse dataset and metrics
  2 | 
  3 | Seahorse is a dataset for multilingual, multifaceted summarization evaluation.
  4 | It contains 96K summaries with human ratings along 6 quality dimensions: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, covering 6 languages, 9 systems and 4 datasets.
  5 | 
  6 | More details can be found in the [paper](https://arxiv.org/abs/2305.13194), which can be cited as follows:
  7 | ```
  8 | @misc{clark2023seahorse,
  9 |       title={SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation}, 
 10 |       author={Elizabeth Clark and Shruti Rijhwani and Sebastian Gehrmann and Joshua Maynez and Roee Aharoni and Vitaly Nikolaev and Thibault Sellam and Aditya Siddhant and Dipanjan Das and Ankur P. Parikh},
 11 |       year={2023},
 12 |       eprint={2305.13194},
 13 |       archivePrefix={arXiv},
 14 |       primaryClass={cs.CL}
 15 | }
 16 | ```
 17 | The Seahorse dataset is released under the [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) license.
 18 | 
 19 | You can download the dataset here: https://storage.googleapis.com/seahorse-public/seahorse_data.zip
 20 | 
 21 | <b>New!!</b> We have also released the learnt metrics trained on Seahorse on HuggingFace. They can be found [here](https://huggingface.co/collections/google/seahorse-release-6543b0c06d87d83c6d24193b).
 22 | 
 23 | ## Dataset description
 24 | 
 25 | The dataset is split into 3 .tsv files: the train, validation, and test sets.
 26 | 
 27 | Each file contains the following information:
 28 | * `gem_id` The ID corresponding to the article that was used to generate the summary (see [Retrieving articles from GEM](https://github.com/google-research-datasets/seahorse/edit/main/README.md#retrieving-articles-from-gem) for more details)
 29 | * `worker_lang` The language ID (de, es-ES, en-US, ru, tr, vi) 
 30 | * `summary` The generated summary
 31 | * `model` The source of the summary (either reference or the summarization model)
 32 | * `question1-6` 6 columns with annotator ratings, corresponding to the 6 dimensions of quality (comprehensibility, repetition, grammar, attribution, main idea(s), and conciseness). If `question1`= No, then there will be no ratings for the remaining questions.
 33 | 
 34 | Here is an example entry:
 35 | ```
 36 | xlsum_english-validation-6416	en-US	Schools in England, Wales and Scotland are being urged to bring back overseas exchange trips.	t5_base	Yes	Yes	Yes	Yes	Yes	Yes
 37 | ```
 38 | 
 39 | There is also a directory called `duplicates`, which contains the items that received multiple annotations. Note that this data should NOT be used for training metrics, as there may be overlap between the train/dev/test sets.
 40 | 
 41 | ## Retrieving articles from GEM
 42 | 
 43 | If you would like to access the articles that the Seahorse summaries are based on, you will need to retrieve them using their GEM ids.
 44 | 
 45 | The [xsum](https://huggingface.co/datasets/GEM/xsum), [mlsum](https://huggingface.co/datasets/GEM/mlsum), and [xlsum](https://huggingface.co/datasets/GEM/xlsum) articles can all be retrieved through GEM on HuggingFace. The `gem_id` column points to the article in the GEM datasets.
 46 | 
 47 | The wikilingua article ids come from a previous version of the GEM dataset and should be retrieved using [TensorFlow datasets](https://www.tensorflow.org/datasets/catalog/gem). Here's an example of how to load the English wikilingua dataset into a dataframe:
 48 | 
 49 | ```
 50 | import tensorflow_datasets as tfds
 51 | 
 52 | lang = 'english_en'
 53 | orig_split = 'validation'
 54 | 
 55 | ds, info = tfds.load(f'huggingface:gem/wiki_lingua_{lang}', split=orig_split, with_info=True)
 56 | hfdf = tfds.as_dataframe(ds,info)
 57 | ```
 58 | 
 59 | ## Seahorse metrics
 60 | 
 61 | Metrics trained on Seahorse are available through HuggingFace. They can be found [here](https://huggingface.co/collections/google/seahorse-release-6543b0c06d87d83c6d24193b).
 62 | These are mT5-based metrics in two sizes (Large and XXL), each trained on one of the six dimensions of quality.
 63 | Please see the [paper](https://arxiv.org/abs/2305.13194) for more details about these metrics.
 64 | |       | Q1      | Q2 | Q3      | Q4 | Q5      | Q6 |
 65 | | -----------| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
 66 | | mT5-XXL      | [seahorse-xxl-q1](https://huggingface.co/google/seahorse-xxl-q1)|[seahorse-xxl-q2](https://huggingface.co/google/seahorse-xxl-q2)|[seahorse-xxl-q3](https://huggingface.co/google/seahorse-xxl-q3)|[seahorse-xxl-q4](https://huggingface.co/google/seahorse-xxl-q4)|[seahorse-xxl-q5](https://huggingface.co/google/seahorse-xxl-q5)|[seahorse-xxl-q6](https://huggingface.co/google/seahorse-xxl-q6)|
 67 | | mT5-Large   | [seahorse-large-q1](https://huggingface.co/google/seahorse-large-q1)|[seahorse-large-q2](https://huggingface.co/google/seahorse-large-q2)|[seahorse-large-q3](https://huggingface.co/google/seahorse-large-q3)|[seahorse-large-q4](https://huggingface.co/google/seahorse-large-q4)|[seahorse-large-q5](https://huggingface.co/google/seahorse-large-q5)|[seahorse-large-q6](https://huggingface.co/google/seahorse-large-q6)|
 68 | 
 69 | **Note:** If you want to use the metrics for labeling summaries (as opposed to looking at correlations or ROC-AUC as we did in the paper), you will need to select a threshold value to classify the metric's scores. The best threshold value will depend on your data and use case.
 70 | 
 71 | ## Leaderboard
 72 | 
 73 | We are maintaining a leaderboard with official results on our test set.
 74 | 
 75 | We ask you to **not** incorporate any part of the Seahorse validation set into the training data, and only use it for validation/hyperparameter tuning as development sets are typically used.
 76 | 
 77 | We report results on two metrics: Pearson correlation ($\rho$) and area under the ROC curve (roc).
 78 | 
 79 | <table>
 80 |   <tr>
 81 |     <th></th>
 82 |     <th></th>
 83 |     <th colspan="2">Q1</th>
 84 |     <th colspan="2">Q2</th>
 85 |     <th colspan="2">Q3</th>
 86 |     <th colspan="2">Q4</th>
 87 |     <th colspan="2">Q5</th>
 88 |     <th colspan="2">Q6</th>
 89 |   </tr>
 90 |   <tr>
 91 |     <th>Model</th>
 92 |     <th>Link</th>
 93 |     <th>$\rho$</th>
 94 |     <th>roc</th>
 95 |     <th>$\rho$</th>
 96 |     <th>roc</th>
 97 |     <th>$\rho$</th>
 98 |     <th>roc</th>
 99 |     <th>$\rho$</th>
100 |     <th>roc</th>
101 |     <th>$\rho$</th>
102 |     <th>roc</th>    
103 |     <th>$\rho$</th>
104 |     <th>roc</th>
105 |   </tr>
106 |        <tr>
107 |       <td> mT5-seahorse </td>
108 |          <td> <a href="">[Clark et al. 2023]</a> </td>
109 |          <td><b>0.52</b></td>
110 |          <td><b>0.90</b></td>
111 |          <td><b>0.86</b></td>
112 |          <td><b>0.98</b></td>
113 |          <td><b>0.45</b></td>
114 |          <td><b>0.84</b></td>
115 |          <td><b>0.59</b></td>
116 |          <td><b>0.85</b></td>
117 |          <td><b>0.50</b></td>
118 |          <td><b>0.80</b></td>
119 |          <td><b>0.52</b></td>
120 |          <td><b>0.81</b></td>         
121 |   </tr> 
122 |          <tr>
123 |       <td> mT5-XNLI </td>
124 |            <td> [<a href="https://aclanthology.org/2022.naacl-main.287/">Honovich et al. 2022</a>, <a href="https://aclanthology.org/D18-1269/">Conneau et al. 2018</a>] </td>
125 |          <td>-</td>
126 |          <td>-</td>
127 |          <td>-</td>
128 |          <td>-</td>
129 |          <td>-</td>
130 |          <td>-</td>
131 |          <td>0.43</td>
132 |          <td>0.78</td>
133 |          <td>-</td>
134 |          <td>-</td>
135 |          <td>-</td>
136 |          <td>-</td>         
137 |   </tr>
138 |            <tr>
139 |       <td> ROUGE-L </td>
140 |            <td> [<a href="https://aclanthology.org/2022.naacl-main.287">Lin et al. 2004</a>] </td>
141 |          <td>0.04</td>
142 |          <td>0.54</td>
143 |          <td>0.06</td>
144 |          <td>0.54</td>
145 |          <td>-0.03</td>
146 |          <td>0.43</td>
147 |          <td>0.13</td>
148 |          <td>0.55</td>
149 |          <td>0.03</td>
150 |          <td>0.54</td>
151 |          <td>0.02</td>
152 |          <td>0.54</td>         
153 |   </tr> 
154 |            <tr>
155 |       <td> Majority Class </td>
156 |            <td> - </td>
157 |          <td>-</td>
158 |          <td>0.5</td>
159 |          <td>-</td>
160 |          <td>0.5</td>
161 |          <td>-</td>
162 |          <td>0.5</td>
163 |          <td>-</td>
164 |          <td>0.5</td>
165 |          <td>-</td>
166 |          <td>0.5</td>
167 |          <td>-</td>
168 |          <td>0.5</td>         
169 |   </tr>   
170 |   
171 | </table>
172 | 
173 | ## Leaderboard Submission
174 | 
175 | If you want to submit to the leaderboard, please send an email to the contact email below with your results.
176 | 
177 | ## Updates
178 | - **01/08/24** We updated the dataset encoding to be in unicode (previously ascii). Both the article and summaries should be input as unicode to the metrics. If you downloaded the dataset before this date, please make sure they are encoded correctly before running the metrics. (See [issue #2](https://github.com/google-research-datasets/seahorse/issues/2) for an example.)
179 | - **11/02/23** The metrics are now available on [HuggingFace](https://huggingface.co/collections/google/seahorse-release-6543b0c06d87d83c6d24193b).
180 | 
181 | ## Contact
182 | 
183 | Please email seahorse-authors@google.com if you have any questions about the dataset.
184 | 


--------------------------------------------------------------------------------