├── README.md └── data ├── dev ├── HotpotQA │ └── candidates.json.gz ├── NaturalQuestions │ └── candidates.json.gz ├── SQuAD │ └── candidates.json.gz ├── SearchQA │ └── candidates.json.gz └── TriviaQA │ └── candidates.json.gz ├── test ├── BioASQ │ └── candidates.json.gz ├── DuoRC │ └── candidates.json.gz ├── RelationExtraction │ └── candidates.json.gz └── TextbookQA │ └── candidates.json.gz └── train ├── HotpotQA └── candidates.json.gz ├── NaturalQuestions └── candidates.json.gz ├── SQuAD └── candidates.json.gz ├── SearchQA └── candidates.json.gz └── TriviaQA └── candidates.json.gz /README.md: -------------------------------------------------------------------------------- 1 | # MultiReQA 2 | 3 | We are creating a challenging new benchmark MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models. Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus. MultiReQA is a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets from the [MRQA shared task](https://mrqa.github.io/). We believe that MultiReQA tests retrieval QA models’ ability to perform domain transfer tasks. 4 | 5 | This repository hosts the codes to convert existing QA datasets from MRQA shared task to the format of MultiReQA benchmark, as well as the sentence boundary annotations for QA datasets to exactly reproduce our work. Note that we are not redistributing the content in the original datasets available on MRQA share task, but just the sentence boundary annotations. 6 | 7 | # Data Description 8 | MultiReQA contains the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA. Five of these datasets, including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, contain both training and test data, and three, in cluding BioASQ, RelationExtraction, TextbookQA, contain only the test data. 9 | 10 | ## Data Format 11 | All of the datasets of MultiReQA have been adapted to follow a unified format. They are stored as compressed [JSONL](http://jsonlines.org/) files (with file extension `.jsonl.gz`). 12 | 13 | The general format is: 14 | 15 | ``` 16 | { 17 | "candidate_id": , 18 | "response_start": , 19 | "response_end": 20 | } 21 | ... 22 | ``` 23 | ### Fields 24 | - **candidate_id:** The candidate id of the candidate sentence. It consists of the original qid from the MRQA shared task. 25 | - **response_start:** The start index of the sentence with respect to its original context. 26 | - **response_end:** The end index of the sentence with respect to its original context. 27 | 28 | ## Data Statistics 29 | We show the number of candidate sentences for each dataset in the table below. 30 | 31 | | | MultiReQA | | 32 | |--------------------|-----------|------------| 33 | | | train | test | 34 | | SearchQA | 629,160 | 454,836 | 35 | | TriviaQA | 335,659 | 238,339 | 36 | | HotpotQA | 104,973 | 52,191 | 37 | | SQuAD | 87,133 | 10,642 | 38 | | NaturalQuestions | 106,521 | 22,118 | 39 | | BioASQ | - | 14,158 | 40 | | RelationExtraction | - | 3,301 | 41 | | TextbookQA | - | 3,701 | 42 | 43 | # Reference 44 | If you use this dataset, please cite [[1]](https://arxiv.org/abs/2005.02507): 45 | 46 | [1] Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, Noah Constant (2020). [MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models](https://arxiv.org/abs/2005.02507). arXiv preprint arXiv:2004.05484. 47 | ``` 48 | @misc{m2020multireqa, 49 | title={MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models}, 50 | author={Mandy Guo and Yinfei Yang and Daniel Cer and Qinlan Shen and Noah Constant}, 51 | year={2020}, 52 | eprint={2005.02507}, 53 | archivePrefix={arXiv}, 54 | primaryClass={cs.CL} 55 | } 56 | ``` 57 | -------------------------------------------------------------------------------- /data/dev/HotpotQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/dev/HotpotQA/candidates.json.gz -------------------------------------------------------------------------------- /data/dev/NaturalQuestions/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/dev/NaturalQuestions/candidates.json.gz -------------------------------------------------------------------------------- /data/dev/SQuAD/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/dev/SQuAD/candidates.json.gz -------------------------------------------------------------------------------- /data/dev/SearchQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/dev/SearchQA/candidates.json.gz -------------------------------------------------------------------------------- /data/dev/TriviaQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/dev/TriviaQA/candidates.json.gz -------------------------------------------------------------------------------- /data/test/BioASQ/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/test/BioASQ/candidates.json.gz -------------------------------------------------------------------------------- /data/test/DuoRC/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/test/DuoRC/candidates.json.gz -------------------------------------------------------------------------------- /data/test/RelationExtraction/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/test/RelationExtraction/candidates.json.gz -------------------------------------------------------------------------------- /data/test/TextbookQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/test/TextbookQA/candidates.json.gz -------------------------------------------------------------------------------- /data/train/HotpotQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/train/HotpotQA/candidates.json.gz -------------------------------------------------------------------------------- /data/train/NaturalQuestions/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/train/NaturalQuestions/candidates.json.gz -------------------------------------------------------------------------------- /data/train/SQuAD/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/train/SQuAD/candidates.json.gz -------------------------------------------------------------------------------- /data/train/SearchQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/train/SearchQA/candidates.json.gz -------------------------------------------------------------------------------- /data/train/TriviaQA/candidates.json.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google-research-datasets/MultiReQA/90262e2b9491217ba5e56c218fc39f168d16ef85/data/train/TriviaQA/candidates.json.gz --------------------------------------------------------------------------------