├── .gitignore ├── LICENSE ├── README.md ├── images └── retrobust_fig_1.png ├── nli ├── README.md ├── data │ ├── bamboogle_no_retrieval.csv │ └── bamboogle_with_retrieval.csv └── src │ ├── scripts │ └── run_nli.py │ └── utils.py └── training ├── README.md └── scripts ├── data_collator.py ├── train.py └── upload.py /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | src/*__pycache__/* 3 | cpython 4 | strategy_qa/ 5 | *.DS_Store 6 | retrieval/* 7 | wandb/* 8 | .env 9 | data/outputs/ 10 | data/retrieval/ 11 | __pycache__/ 12 | data/retrieval_results/* -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 oriyor 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## 🪨️ Making Retrieval-Augmented Language Models Robust to Irrelevant Context 2 | 3 | ### RetRobust Overview 4 | By training RALMs on 1K examples we can make them robust to irrelevant context and improve QA performance 5 | [**[Paper]**](http://arxiv.org/abs/2310.01558). 6 | 7 | ![Alt text](images/retrobust_fig_1.png?raw=true "Retrobust examples") 8 | 9 | 10 | ### 🤗 Data and Models 11 | Our models and data are available at the [**RetRobust HuggingFace Collection**](https://huggingface.co/collections/Ori/retrobust-65198eef2b4fffcb4100e163). 12 | 13 | ### 🧗🏽 Experiments framework 14 | LLama-2 inference servers were set using [**lm-sys/FastChat**](https://github.com/lm-sys/FastChat). Experiments were run using the framework from [**reasoning-on-cots**](https://github.com/oriyor/reasoning-on-cots). To run these experiments, see [**here**](https://github.com/oriyor/reasoning-on-cots/tree/master/src/experiments/retrobust). 15 | 16 | ### 🏃‍ Training 17 | See [**here**](training/README.md). 18 | 19 | ### ⚔️️ NLI filtering 20 | See [**here**](nli/README.md). 21 | 22 | ### ✍ Citation 23 | ``` 24 | bibtex 25 | @misc{yoran2023making, 26 | title={Making Retrieval-Augmented Language Models Robust to Irrelevant Context}, 27 | author={Ori Yoran and Tomer Wolfson and Ori Ram and Jonathan Berant}, 28 | year={2023}, 29 | eprint={2310.01558}, 30 | archivePrefix={arXiv}, 31 | primaryClass={cs.CL} 32 | } 33 | ``` 34 | -------------------------------------------------------------------------------- /images/retrobust_fig_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/oriyor/ret-robust/c2bddaf09c3e395337c76587fc52e0cb3003da67/images/retrobust_fig_1.png -------------------------------------------------------------------------------- /nli/README.md: -------------------------------------------------------------------------------- 1 | # ⚔ NLI filtering 2 | Our NLI filtering uses a [**bart-large-mnli**](https://huggingface.co/facebook/bart-large-mnli) as a decision rule to decide when to use retrieval. 3 | For an example script, see [**here**](src/scripts/run_nli.py). Note that the input files are in this example are csvs in the format generated by the [**reasoning-on-cots project**](https://github.com/oriyor/reasoning-on-cots/tree/master/src/experiments/retrobust). -------------------------------------------------------------------------------- /nli/data/bamboogle_no_retrieval.csv: -------------------------------------------------------------------------------- 1 | ,question,answers,gpt_answers,acc@1,acc@3,acc@majority,acc@majority_3,metadata_qid,metadata_question,metadata_gold_answer,metadata_prev_model_answer,decomposition_0 2 | 0,"{'question': 'What was the father of the last surviving Canadian father of Confederation?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the last surviving Canadian father of Confederation?\nIntermediate answer: The last surviving Canadian father of Confederation was Alexander Campbell.\nFollow up: What was the father of Alexander Campbell?\nIntermediate answer: The father of Alexander Campbell was Alexander Campbell Sr.\nFollow up: What was the father of Alexander Campbell Sr.?\nIntermediate answer: The father of Alexander Campbell Sr. was Alexander Campbell Sr.\nFollow up: What was the father of Alexander Campbell Sr.?\nIntermediate answer: The father of'], 'decompsition_steps': [[{'question': 'Who was the last surviving Canadian father of Confederation?', 'answer': 'The last surviving Canadian father of Confederation was Alexander Campbell.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the father of Alexander Campbell?', 'answer': 'The father of Alexander Campbell was Alexander Campbell Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the father of Alexander Campbell Sr.?', 'answer': 'The father of Alexander Campbell Sr. was Alexander Campbell Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the father of Alexander Campbell Sr.?', 'answer': 'The father of', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,5,What was the father of the last surviving Canadian father of Confederation?,Charles Tupper Sr.,," 3 | Are follow up questions needed here: Yes. 4 | Follow up: Who was the last surviving Canadian father of Confederation? 5 | Intermediate answer: The last surviving Canadian father of Confederation was Alexander Campbell. 6 | Follow up: What was the father of Alexander Campbell? 7 | Intermediate answer: The father of Alexander Campbell was Alexander Campbell Sr. 8 | Follow up: What was the father of Alexander Campbell Sr.? 9 | Intermediate answer: The father of Alexander Campbell Sr. was Alexander Campbell Sr. 10 | Follow up: What was the father of Alexander Campbell Sr.? 11 | Intermediate answer: The father of" 12 | 1,"{'question': 'Who was the first African American mayor of the most populous city in the United States?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the most populous city in the United States?\nIntermediate answer: The most populous city in the United States is New York City.\nFollow up: Who was the first African American mayor of New York City?\nIntermediate answer: The first African American mayor of New York City was David Dinkins.\nFollow up: When was David Dinkins born?\nIntermediate answer: David Dinkins was born on July 10, 1927.\nSo the final answer is: July '], 'decompsition_steps': [[{'question': 'What is the most populous city in the United States?', 'answer': 'The most populous city in the United States is New York City.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the first African American mayor of New York City?', 'answer': 'The first African American mayor of New York City was David Dinkins.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was David Dinkins born?', 'answer': 'David Dinkins was born on July 10, 1927.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'July '}]], 'traces_entailments': None}",,['July '],0,0,0,0,6,Who was the first African American mayor of the most populous city in the United States?,David Dinkins,," 13 | Are follow up questions needed here: Yes. 14 | Follow up: What is the most populous city in the United States? 15 | Intermediate answer: The most populous city in the United States is New York City. 16 | Follow up: Who was the first African American mayor of New York City? 17 | Intermediate answer: The first African American mayor of New York City was David Dinkins. 18 | Follow up: When was David Dinkins born? 19 | Intermediate answer: David Dinkins was born on July 10, 1927. 20 | So the final answer is: July " 21 | 2,"{'question': 'What is the capital of the second largest state in the US by area?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the second largest state in the US by area?\nIntermediate answer: The second largest state in the US by area is Alaska.\nFollow up: What is the capital of Alaska?\nIntermediate answer: The capital of Alaska is Juneau.\nSo the final answer is: Juneau.\n'], 'decompsition_steps': [[{'question': 'What is the second largest state in the US by area?', 'answer': 'The second largest state in the US by area is Alaska.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the capital of Alaska?', 'answer': 'The capital of Alaska is Juneau.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Juneau.'}]], 'traces_entailments': None}",,['Juneau'],0,0,0,0,7,What is the capital of the second largest state in the US by area?,Austin,," 22 | Are follow up questions needed here: Yes. 23 | Follow up: What is the second largest state in the US by area? 24 | Intermediate answer: The second largest state in the US by area is Alaska. 25 | Follow up: What is the capital of Alaska? 26 | Intermediate answer: The capital of Alaska is Juneau. 27 | So the final answer is: Juneau. 28 | " 29 | 3,"{'question': 'In what year did work begin on the second longest road tunnel in the world?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the second longest road tunnel in the world?\nIntermediate answer: The second longest road tunnel in the world is the Lærdal Tunnel.\nFollow up: When did work begin on the Lærdal Tunnel?\nIntermediate answer: Work began on the Lærdal Tunnel in 1995.\nSo the final answer is: 1995.\n'], 'decompsition_steps': [[{'question': 'What is the second longest road tunnel in the world?', 'answer': 'The second longest road tunnel in the world is the Lærdal Tunnel.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did work begin on the Lærdal Tunnel?', 'answer': 'Work began on the Lærdal Tunnel in 1995.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '1995.'}]], 'traces_entailments': None}",,['1995'],0,0,0,0,8,In what year did work begin on the second longest road tunnel in the world?,1992,," 30 | Are follow up questions needed here: Yes. 31 | Follow up: What is the second longest road tunnel in the world? 32 | Intermediate answer: The second longest road tunnel in the world is the Lærdal Tunnel. 33 | Follow up: When did work begin on the Lærdal Tunnel? 34 | Intermediate answer: Work began on the Lærdal Tunnel in 1995. 35 | So the final answer is: 1995. 36 | " 37 | 4,"{'question': 'Who is the father of the father of observational astronomy?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of observational astronomy?\nIntermediate answer: The father of observational astronomy is Galileo Galilei.\nFollow up: Who is the father of Galileo Galilei?\nIntermediate answer: The father of Galileo Galilei is Vincenzo Galilei.\nSo the final answer is: Vincenzo Galilei.\n'], 'decompsition_steps': [[{'question': 'Who is the father of observational astronomy?', 'answer': 'The father of observational astronomy is Galileo Galilei.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Galileo Galilei?', 'answer': 'The father of Galileo Galilei is Vincenzo Galilei.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Vincenzo Galilei.'}]], 'traces_entailments': None}",,['Vincenzo Galilei'],1,1,1,1,9,Who is the father of the father of observational astronomy?,Vincenzo Galilei,," 38 | Are follow up questions needed here: Yes. 39 | Follow up: Who is the father of observational astronomy? 40 | Intermediate answer: The father of observational astronomy is Galileo Galilei. 41 | Follow up: Who is the father of Galileo Galilei? 42 | Intermediate answer: The father of Galileo Galilei is Vincenzo Galilei. 43 | So the final answer is: Vincenzo Galilei. 44 | " 45 | 5,"{'question': 'Who was the father of the father of psychoanalysis?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the father of psychoanalysis?\nIntermediate answer: The father of psychoanalysis was Sigmund Freud.\nFollow up: Who was the father of Sigmund Freud?\nIntermediate answer: The father of Sigmund Freud was Jacob Freud.\nSo the final answer is: Jacob Freud.\n'], 'decompsition_steps': [[{'question': 'Who was the father of psychoanalysis?', 'answer': 'The father of psychoanalysis was Sigmund Freud.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of Sigmund Freud?', 'answer': 'The father of Sigmund Freud was Jacob Freud.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Jacob Freud.'}]], 'traces_entailments': None}",,['Jacob Freud'],1,1,1,1,10,Who was the father of the father of psychoanalysis?,Jacob Freud,," 46 | Are follow up questions needed here: Yes. 47 | Follow up: Who was the father of psychoanalysis? 48 | Intermediate answer: The father of psychoanalysis was Sigmund Freud. 49 | Follow up: Who was the father of Sigmund Freud? 50 | Intermediate answer: The father of Sigmund Freud was Jacob Freud. 51 | So the final answer is: Jacob Freud. 52 | " 53 | 6,"{'question': 'How many people died in the second most powerful earthquake ever recorded?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the second most powerful earthquake ever recorded?\nIntermediate answer: The second most powerful earthquake ever recorded was the 1960 Valdivia earthquake.\nFollow up: How many people died in the 1960 Valdivia earthquake?\nIntermediate answer: 5,700 people died in the 1960 Valdivia earthquake.\nSo the final answer is: 5,700.\n'], 'decompsition_steps': [[{'question': 'What was the second most powerful earthquake ever recorded?', 'answer': 'The second most powerful earthquake ever recorded was the 1960 Valdivia earthquake.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'How many people died in the 1960 Valdivia earthquake?', 'answer': '5,700 people died in the 1960 Valdivia earthquake.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '5,700.'}]], 'traces_entailments': None}",,"['5,700']",0,0,0,0,11,How many people died in the second most powerful earthquake ever recorded?,131,," 54 | Are follow up questions needed here: Yes. 55 | Follow up: What was the second most powerful earthquake ever recorded? 56 | Intermediate answer: The second most powerful earthquake ever recorded was the 1960 Valdivia earthquake. 57 | Follow up: How many people died in the 1960 Valdivia earthquake? 58 | Intermediate answer: 5,700 people died in the 1960 Valdivia earthquake. 59 | So the final answer is: 5,700. 60 | " 61 | 7,"{'question': 'When did the president who warned about the military industrial complex die?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the president who warned about the military industrial complex?\nIntermediate answer: The president who warned about the military industrial complex was Dwight D. Eisenhower.\nFollow up: When did Dwight D. Eisenhower die?\nIntermediate answer: Dwight D. Eisenhower died on March 28, 1969.\nSo the final answer is: March 28, 1969.\n'], 'decompsition_steps': [[{'question': 'Who was the president who warned about the military industrial complex?', 'answer': 'The president who warned about the military industrial complex was Dwight D. Eisenhower.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Dwight D. Eisenhower die?', 'answer': 'Dwight D. Eisenhower died on March 28, 1969.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'March 28, 1969.'}]], 'traces_entailments': None}",,"['March 28, 1969']",1,1,1,1,12,When did the president who warned about the military industrial complex die?,"March 28, 1969",," 62 | Are follow up questions needed here: Yes. 63 | Follow up: Who was the president who warned about the military industrial complex? 64 | Intermediate answer: The president who warned about the military industrial complex was Dwight D. Eisenhower. 65 | Follow up: When did Dwight D. Eisenhower die? 66 | Intermediate answer: Dwight D. Eisenhower died on March 28, 1969. 67 | So the final answer is: March 28, 1969. 68 | " 69 | 8,"{'question': ""What's the capital of the state that Washington University in St. Louis is in?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What state is Washington University in St. Louis in?\nIntermediate answer: Washington University in St. Louis is in Missouri.\nFollow up: What's the capital of Missouri?\nIntermediate answer: The capital of Missouri is Jefferson City.\nSo the final answer is: Jefferson City.\n""], 'decompsition_steps': [[{'question': 'What state is Washington University in St. Louis in?', 'answer': 'Washington University in St. Louis is in Missouri.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What's the capital of Missouri?"", 'answer': 'The capital of Missouri is Jefferson City.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Jefferson City.'}]], 'traces_entailments': None}",,['Jefferson City'],1,1,1,1,13,What's the capital of the state that Washington University in St. Louis is in?,Jefferson City,," 70 | Are follow up questions needed here: Yes. 71 | Follow up: What state is Washington University in St. Louis in? 72 | Intermediate answer: Washington University in St. Louis is in Missouri. 73 | Follow up: What's the capital of Missouri? 74 | Intermediate answer: The capital of Missouri is Jefferson City. 75 | So the final answer is: Jefferson City. 76 | " 77 | 9,"{'question': 'What is the third letter of the top level domain of the military?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the top level domain of the military?\nIntermediate answer: The top level domain of the military is.mil.\nFollow up: What is the third letter of.mil?\nIntermediate answer: The third letter of.mil is L.\nSo the final answer is: L.\n'], 'decompsition_steps': [[{'question': 'What is the top level domain of the military?', 'answer': 'The top level domain of the military is.mil.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the third letter of.mil?', 'answer': 'The third letter of.mil is L.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'L.'}]], 'traces_entailments': None}",,['L'],1,1,1,1,14,What is the third letter of the top level domain of the military?,l,," 78 | Are follow up questions needed here: Yes. 79 | Follow up: What is the top level domain of the military? 80 | Intermediate answer: The top level domain of the military is.mil. 81 | Follow up: What is the third letter of.mil? 82 | Intermediate answer: The third letter of.mil is L. 83 | So the final answer is: L. 84 | " 85 | 10,"{'question': 'Who is the predecessor of the longest-reigning British monarch?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the longest-reigning British monarch?\nIntermediate answer: The longest-reigning British monarch is Queen Elizabeth II.\nFollow up: Who is the predecessor of Queen Elizabeth II?\nIntermediate answer: The predecessor of Queen Elizabeth II is Queen Victoria.\nSo the final answer is: Queen Victoria.\n'], 'decompsition_steps': [[{'question': 'Who is the longest-reigning British monarch?', 'answer': 'The longest-reigning British monarch is Queen Elizabeth II.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the predecessor of Queen Elizabeth II?', 'answer': 'The predecessor of Queen Elizabeth II is Queen Victoria.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Queen Victoria.'}]], 'traces_entailments': None}",,['Queen Victoria'],0,0,0,0,15,Who is the predecessor of the longest-reigning British monarch?,"George VI 86 | ",," 87 | Are follow up questions needed here: Yes. 88 | Follow up: Who is the longest-reigning British monarch? 89 | Intermediate answer: The longest-reigning British monarch is Queen Elizabeth II. 90 | Follow up: Who is the predecessor of Queen Elizabeth II? 91 | Intermediate answer: The predecessor of Queen Elizabeth II is Queen Victoria. 92 | So the final answer is: Queen Victoria. 93 | " 94 | 11,"{'question': 'In what year did the oldest cat ever recorded with the Cat of the Year award?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the Cat of the Year award?\nIntermediate answer: The Cat of the Year award is an award given to the oldest cat ever recorded.\nFollow up: When was the oldest cat ever recorded?\nIntermediate answer: The oldest cat ever recorded was Creme Puff.\nFollow up: When was Creme Puff born?\nIntermediate answer: Creme Puff was born on August 3, 1967.\nSo the final answer is: August 3, 1967.\n'], 'decompsition_steps': [[{'question': 'What is the Cat of the Year award?', 'answer': 'The Cat of the Year award is an award given to the oldest cat ever recorded.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the oldest cat ever recorded?', 'answer': 'The oldest cat ever recorded was Creme Puff.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Creme Puff born?', 'answer': 'Creme Puff was born on August 3, 1967.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'August 3, 1967.'}]], 'traces_entailments': None}",,"['August 3, 1967']",0,0,0,0,16,In what year did the oldest cat ever recorded with the Cat of the Year award?,1999,," 95 | Are follow up questions needed here: Yes. 96 | Follow up: What is the Cat of the Year award? 97 | Intermediate answer: The Cat of the Year award is an award given to the oldest cat ever recorded. 98 | Follow up: When was the oldest cat ever recorded? 99 | Intermediate answer: The oldest cat ever recorded was Creme Puff. 100 | Follow up: When was Creme Puff born? 101 | Intermediate answer: Creme Puff was born on August 3, 1967. 102 | So the final answer is: August 3, 1967. 103 | " 104 | 12,"{'question': 'When was the anime studio that made Sword Art Online founded?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the name of the anime studio that made Sword Art Online?\nIntermediate answer: The anime studio that made Sword Art Online is A-1 Pictures.\nFollow up: When was A-1 Pictures founded?\nIntermediate answer: A-1 Pictures was founded on May 1, 2005.\nSo the final answer is: May 1, 2005.\n'], 'decompsition_steps': [[{'question': 'What is the name of the anime studio that made Sword Art Online?', 'answer': 'The anime studio that made Sword Art Online is A-1 Pictures.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was A-1 Pictures founded?', 'answer': 'A-1 Pictures was founded on May 1, 2005.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'May 1, 2005.'}]], 'traces_entailments': None}",,"['May 1, 2005']",0.6666666667,0.6666666667,0.6666666667,0.6666666667,17,When was the anime studio that made Sword Art Online founded?,"May 9, 2005",," 105 | Are follow up questions needed here: Yes. 106 | Follow up: What is the name of the anime studio that made Sword Art Online? 107 | Intermediate answer: The anime studio that made Sword Art Online is A-1 Pictures. 108 | Follow up: When was A-1 Pictures founded? 109 | Intermediate answer: A-1 Pictures was founded on May 1, 2005. 110 | So the final answer is: May 1, 2005. 111 | " 112 | 13,"{'question': ""What's the capital city of the country that was the champion of the 2010 World Cup?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What country was the champion of the 2010 World Cup?\nIntermediate answer: The country that was the champion of the 2010 World Cup was Spain.\nFollow up: What's the capital city of Spain?\nIntermediate answer: The capital city of Spain is Madrid.\nSo the final answer is: Madrid.\n""], 'decompsition_steps': [[{'question': 'What country was the champion of the 2010 World Cup?', 'answer': 'The country that was the champion of the 2010 World Cup was Spain.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What's the capital city of Spain?"", 'answer': 'The capital city of Spain is Madrid.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Madrid.'}]], 'traces_entailments': None}",,['Madrid'],1,1,1,1,18,What's the capital city of the country that was the champion of the 2010 World Cup?,Madrid,," 113 | Are follow up questions needed here: Yes. 114 | Follow up: What country was the champion of the 2010 World Cup? 115 | Intermediate answer: The country that was the champion of the 2010 World Cup was Spain. 116 | Follow up: What's the capital city of Spain? 117 | Intermediate answer: The capital city of Spain is Madrid. 118 | So the final answer is: Madrid. 119 | " 120 | 14,"{'question': 'Who was the last emperor of the dynasty that succeeded the Song dynasty?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What dynasty succeeded the Song dynasty?\nIntermediate answer: The dynasty that succeeded the Song dynasty was the Yuan dynasty.\nFollow up: Who was the last emperor of the Yuan dynasty?\nIntermediate answer: The last emperor of the Yuan dynasty was Toghon Temür.\nSo the final answer is: Toghon Temür.\n'], 'decompsition_steps': [[{'question': 'What dynasty succeeded the Song dynasty?', 'answer': 'The dynasty that succeeded the Song dynasty was the Yuan dynasty.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the last emperor of the Yuan dynasty?', 'answer': 'The last emperor of the Yuan dynasty was Toghon Temür.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Toghon Temür.'}]], 'traces_entailments': None}",,['Toghon Temür'],1,1,1,1,19,Who was the last emperor of the dynasty that succeeded the Song dynasty?,Toghon Temür,," 121 | Are follow up questions needed here: Yes. 122 | Follow up: What dynasty succeeded the Song dynasty? 123 | Intermediate answer: The dynasty that succeeded the Song dynasty was the Yuan dynasty. 124 | Follow up: Who was the last emperor of the Yuan dynasty? 125 | Intermediate answer: The last emperor of the Yuan dynasty was Toghon Temür. 126 | So the final answer is: Toghon Temür. 127 | " 128 | 15,"{'question': 'Who produced the longest running video game franchise?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the longest running video game franchise?\nIntermediate answer: The longest running video game franchise is Mario.\nFollow up: Who produced Mario?\nIntermediate answer: Mario was produced by Nintendo.\nSo the final answer is: Nintendo.\n'], 'decompsition_steps': [[{'question': 'What is the longest running video game franchise?', 'answer': 'The longest running video game franchise is Mario.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who produced Mario?', 'answer': 'Mario was produced by Nintendo.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Nintendo.'}]], 'traces_entailments': None}",,['Nintendo'],0,0,0,0,20,Who produced the longest running video game franchise?,MECC,," 129 | Are follow up questions needed here: Yes. 130 | Follow up: What is the longest running video game franchise? 131 | Intermediate answer: The longest running video game franchise is Mario. 132 | Follow up: Who produced Mario? 133 | Intermediate answer: Mario was produced by Nintendo. 134 | So the final answer is: Nintendo. 135 | " 136 | 16,"{'question': 'In what country is the third largest stadium in the world?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the third largest stadium in the world?\nIntermediate answer: The third largest stadium in the world is the Rungrado May Day Stadium in North Korea.\nFollow up: In what country is North Korea?\nIntermediate answer: North Korea is in Asia.\nFollow up: In what country is Asia?\nIntermediate answer: Asia is in the continent of Asia.\nSo the final answer is: Asia.\n'], 'decompsition_steps': [[{'question': 'What is the third largest stadium in the world?', 'answer': 'The third largest stadium in the world is the Rungrado May Day Stadium in North Korea.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'In what country is North Korea?', 'answer': 'North Korea is in Asia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'In what country is Asia?', 'answer': 'Asia is in the continent of Asia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Asia.'}]], 'traces_entailments': None}",,['Asia'],0,0,0,0,21,In what country is the third largest stadium in the world?,United States,," 137 | Are follow up questions needed here: Yes. 138 | Follow up: What is the third largest stadium in the world? 139 | Intermediate answer: The third largest stadium in the world is the Rungrado May Day Stadium in North Korea. 140 | Follow up: In what country is North Korea? 141 | Intermediate answer: North Korea is in Asia. 142 | Follow up: In what country is Asia? 143 | Intermediate answer: Asia is in the continent of Asia. 144 | So the final answer is: Asia. 145 | " 146 | 17,"{'question': 'When did the president who set the precedent of a two term limit enter office?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the president who set the precedent of a two term limit?\nIntermediate answer: The president who set the precedent of a two term limit was George Washington.\nFollow up: When did George Washington enter office?\nIntermediate answer: George Washington entered office on April 30, 1789.\nSo the final answer is: April 30, 1789.\n'], 'decompsition_steps': [[{'question': 'Who was the president who set the precedent of a two term limit?', 'answer': 'The president who set the precedent of a two term limit was George Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did George Washington enter office?', 'answer': 'George Washington entered office on April 30, 1789.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'April 30, 1789.'}]], 'traces_entailments': None}",,"['April 30, 1789']",1,1,1,1,22,When did the president who set the precedent of a two term limit enter office?,"April 30, 1789",," 147 | Are follow up questions needed here: Yes. 148 | Follow up: Who was the president who set the precedent of a two term limit? 149 | Intermediate answer: The president who set the precedent of a two term limit was George Washington. 150 | Follow up: When did George Washington enter office? 151 | Intermediate answer: George Washington entered office on April 30, 1789. 152 | So the final answer is: April 30, 1789. 153 | " 154 | 18,"{'question': 'What was the motto of the Olympics that had Fuwa as the mascots?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What were the names of the Fuwa mascots?\nIntermediate answer: The names of the Fuwa mascots were Beibei, Jingjing, Huanhuan, Yingying, and Nini.\nFollow up: What was the motto of the Olympics that had Fuwa as the mascots?\nIntermediate answer: The motto of the Olympics that had Fuwa as the mascots was ""One World, One Dream.""\nSo the final answer is: ""One World, One Dream'], 'decompsition_steps': [[{'question': 'What were the names of the Fuwa mascots?', 'answer': 'The names of the Fuwa mascots were Beibei, Jingjing, Huanhuan, Yingying, and Nini.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the motto of the Olympics that had Fuwa as the mascots?', 'answer': 'The motto of the Olympics that had Fuwa as the mascots was ""One World, One Dream.""', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '""One World, One Dream'}]], 'traces_entailments': None}",,"['""One World, One Dream']",0.75,0.75,0.75,0.75,23,What was the motto of the Olympics that had Fuwa as the mascots?,"One World, One Dream",," 155 | Are follow up questions needed here: Yes. 156 | Follow up: What were the names of the Fuwa mascots? 157 | Intermediate answer: The names of the Fuwa mascots were Beibei, Jingjing, Huanhuan, Yingying, and Nini. 158 | Follow up: What was the motto of the Olympics that had Fuwa as the mascots? 159 | Intermediate answer: The motto of the Olympics that had Fuwa as the mascots was ""One World, One Dream."" 160 | So the final answer is: ""One World, One Dream" 161 | 19,"{'question': 'What is the nickname for the state that is the home to the “Avocado Capital of the World""?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the nickname for the state that is the home to the “Avocado Capital of the World""?\nIntermediate answer: The nickname for the state that is the home to the “Avocado Capital of the World"" is California.\nFollow up: What is the capital of California?\nIntermediate answer: The capital of California is Sacramento.\nFollow up: What is the nickname for Sacramento?\nIntermediate answer: The nickname for Sacramento is the “Avocado Capital of'], 'decompsition_steps': [[{'question': 'What is the nickname for the state that is the home to the “Avocado Capital of the World""?', 'answer': 'The nickname for the state that is the home to the “Avocado Capital of the World"" is California.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the capital of California?', 'answer': 'The capital of California is Sacramento.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the nickname for Sacramento?', 'answer': 'The nickname for Sacramento is the “Avocado Capital of', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,24,"What is the nickname for the state that is the home to the “Avocado Capital of the World""?",Golden State,," 162 | Are follow up questions needed here: Yes. 163 | Follow up: What is the nickname for the state that is the home to the “Avocado Capital of the World""? 164 | Intermediate answer: The nickname for the state that is the home to the “Avocado Capital of the World"" is California. 165 | Follow up: What is the capital of California? 166 | Intermediate answer: The capital of California is Sacramento. 167 | Follow up: What is the nickname for Sacramento? 168 | Intermediate answer: The nickname for Sacramento is the “Avocado Capital of" 169 | 20,"{'question': 'The most populous city in Punjab is how large (area wise)?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the most populous city in Punjab?\nIntermediate answer: The most populous city in Punjab is Ludhiana.\nFollow up: What is the area of Ludhiana?\nIntermediate answer: The area of Ludhiana is 310.2 km2.\nSo the final answer is: 310.2 km2.\n'], 'decompsition_steps': [[{'question': 'What is the most populous city in Punjab?', 'answer': 'The most populous city in Punjab is Ludhiana.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the area of Ludhiana?', 'answer': 'The area of Ludhiana is 310.2 km2.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '310.2 km2.'}]], 'traces_entailments': None}",,['310.2 km2'],0,0,0,0,25,The most populous city in Punjab is how large (area wise)?,310 square kilometers,," 170 | Are follow up questions needed here: Yes. 171 | Follow up: What is the most populous city in Punjab? 172 | Intermediate answer: The most populous city in Punjab is Ludhiana. 173 | Follow up: What is the area of Ludhiana? 174 | Intermediate answer: The area of Ludhiana is 310.2 km2. 175 | So the final answer is: 310.2 km2. 176 | " 177 | 21,"{'question': 'Who is the father of the father of modern experimental psychology?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of modern experimental psychology?\nIntermediate answer: The father of modern experimental psychology is Wilhelm Wundt.\nFollow up: Who is the father of Wilhelm Wundt?\nIntermediate answer: The father of Wilhelm Wundt is Christian Wundt.\nSo the final answer is: Christian Wundt.\n'], 'decompsition_steps': [[{'question': 'Who is the father of modern experimental psychology?', 'answer': 'The father of modern experimental psychology is Wilhelm Wundt.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Wilhelm Wundt?', 'answer': 'The father of Wilhelm Wundt is Christian Wundt.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Christian Wundt.'}]], 'traces_entailments': None}",,['Christian Wundt'],0.5,0.5,0.5,0.5,26,Who is the father of the father of modern experimental psychology?,Maximilian Wundt,," 178 | Are follow up questions needed here: Yes. 179 | Follow up: Who is the father of modern experimental psychology? 180 | Intermediate answer: The father of modern experimental psychology is Wilhelm Wundt. 181 | Follow up: Who is the father of Wilhelm Wundt? 182 | Intermediate answer: The father of Wilhelm Wundt is Christian Wundt. 183 | So the final answer is: Christian Wundt. 184 | " 185 | 22,"{'question': 'Who was the second wife of the founder of CNN?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the founder of CNN?\nIntermediate answer: The founder of CNN was Ted Turner.\nFollow up: Who was the second wife of Ted Turner?\nIntermediate answer: The second wife of Ted Turner was Jane Fonda.\nSo the final answer is: Jane Fonda.\n'], 'decompsition_steps': [[{'question': 'Who was the founder of CNN?', 'answer': 'The founder of CNN was Ted Turner.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the second wife of Ted Turner?', 'answer': 'The second wife of Ted Turner was Jane Fonda.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Jane Fonda.'}]], 'traces_entailments': None}",,['Jane Fonda'],0.4,0.4,0.4,0.4,27,Who was the second wife of the founder of CNN?,Jane Shirley Smith,," 186 | Are follow up questions needed here: Yes. 187 | Follow up: Who was the founder of CNN? 188 | Intermediate answer: The founder of CNN was Ted Turner. 189 | Follow up: Who was the second wife of Ted Turner? 190 | Intermediate answer: The second wife of Ted Turner was Jane Fonda. 191 | So the final answer is: Jane Fonda. 192 | " 193 | 23,"{'question': 'In what year was the company that was founded as Sound of Music added to the S&P 500?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What company was founded as Sound of Music?\nIntermediate answer: The company that was founded as Sound of Music is Sirius XM.\nFollow up: When was Sirius XM added to the S&P 500?\nIntermediate answer: Sirius XM was added to the S&P 500 in 2013.\nFollow up: What year was 2013?\nIntermediate answer: 2013 was a leap year.\nSo'], 'decompsition_steps': [[{'question': 'What company was founded as Sound of Music?', 'answer': 'The company that was founded as Sound of Music is Sirius XM.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Sirius XM added to the S&P 500?', 'answer': 'Sirius XM was added to the S&P 500 in 2013.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What year was 2013?', 'answer': '2013 was a leap year.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,28,In what year was the company that was founded as Sound of Music added to the S&P 500?,1999,," 194 | Are follow up questions needed here: Yes. 195 | Follow up: What company was founded as Sound of Music? 196 | Intermediate answer: The company that was founded as Sound of Music is Sirius XM. 197 | Follow up: When was Sirius XM added to the S&P 500? 198 | Intermediate answer: Sirius XM was added to the S&P 500 in 2013. 199 | Follow up: What year was 2013? 200 | Intermediate answer: 2013 was a leap year. 201 | So" 202 | 24,"{'question': 'The most populous national capital city was established in what year?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the most populous national capital city?\nIntermediate answer: The most populous national capital city is Tokyo.\nFollow up: When was Tokyo established?\nIntermediate answer: Tokyo was established in 1868.\nSo the final answer is: 1868.\n'], 'decompsition_steps': [[{'question': 'What is the most populous national capital city?', 'answer': 'The most populous national capital city is Tokyo.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Tokyo established?', 'answer': 'Tokyo was established in 1868.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '1868.'}]], 'traces_entailments': None}",,['1868'],0,0,0,0,29,The most populous national capital city was established in what year?,1045 BC,," 203 | Are follow up questions needed here: Yes. 204 | Follow up: What is the most populous national capital city? 205 | Intermediate answer: The most populous national capital city is Tokyo. 206 | Follow up: When was Tokyo established? 207 | Intermediate answer: Tokyo was established in 1868. 208 | So the final answer is: 1868. 209 | " 210 | 25,"{'question': 'Where was the person who shared the Nobel Prize in Physics in 1954 with Max Born born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who shared the Nobel Prize in Physics in 1954 with Max Born?\nIntermediate answer: The person who shared the Nobel Prize in Physics in 1954 with Max Born is Walther Bothe.\nFollow up: Where was Walther Bothe born?\nIntermediate answer: Walther Bothe was born in Oldenburg, Germany.\nSo the final answer is: Oldenburg, Germany.\n'], 'decompsition_steps': [[{'question': 'Who shared the Nobel Prize in Physics in 1954 with Max Born?', 'answer': 'The person who shared the Nobel Prize in Physics in 1954 with Max Born is Walther Bothe.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Where was Walther Bothe born?', 'answer': 'Walther Bothe was born in Oldenburg, Germany.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Oldenburg, Germany.'}]], 'traces_entailments': None}",,"['Oldenburg, Germany']",0.5,0.5,0.5,0.5,30,Where was the person who shared the Nobel Prize in Physics in 1954 with Max Born born?,"Oranienburg, Germany",," 211 | Are follow up questions needed here: Yes. 212 | Follow up: Who shared the Nobel Prize in Physics in 1954 with Max Born? 213 | Intermediate answer: The person who shared the Nobel Prize in Physics in 1954 with Max Born is Walther Bothe. 214 | Follow up: Where was Walther Bothe born? 215 | Intermediate answer: Walther Bothe was born in Oldenburg, Germany. 216 | So the final answer is: Oldenburg, Germany. 217 | " 218 | 26,"{'question': 'The Filipino statesman who established the government-in-exile during the outbreak of World War II was also the mayor of what city?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What city was the Filipino statesman who established the government-in-exile during the outbreak of World War II the mayor of?\nIntermediate answer: The Filipino statesman who established the government-in-exile during the outbreak of World War II was the mayor of Quezon City.\nFollow up: When was Quezon City founded?\nIntermediate answer: Quezon City was founded on October 12, 1939.\nSo the final answer is: October 12, 19'], 'decompsition_steps': [[{'question': 'What city was the Filipino statesman who established the government-in-exile during the outbreak of World War II the mayor of?', 'answer': 'The Filipino statesman who established the government-in-exile during the outbreak of World War II was the mayor of Quezon City.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Quezon City founded?', 'answer': 'Quezon City was founded on October 12, 1939.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'October 12, 19'}]], 'traces_entailments': None}",,"['October 12, 19']",0,0,0,0,31,The Filipino statesman who established the government-in-exile during the outbreak of World War II was also the mayor of what city?,Quezon City,," 219 | Are follow up questions needed here: Yes. 220 | Follow up: What city was the Filipino statesman who established the government-in-exile during the outbreak of World War II the mayor of? 221 | Intermediate answer: The Filipino statesman who established the government-in-exile during the outbreak of World War II was the mayor of Quezon City. 222 | Follow up: When was Quezon City founded? 223 | Intermediate answer: Quezon City was founded on October 12, 1939. 224 | So the final answer is: October 12, 19" 225 | 27,"{'question': 'In 2016, who was the host of the longest running talk show?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the longest running talk show?\nIntermediate answer: The longest running talk show is The Tonight Show.\nFollow up: Who was the host of The Tonight Show in 2016?\nIntermediate answer: The host of The Tonight Show in 2016 was Jimmy Fallon.\nSo the final answer is: Jimmy Fallon.\n'], 'decompsition_steps': [[{'question': 'What was the longest running talk show?', 'answer': 'The longest running talk show is The Tonight Show.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the host of The Tonight Show in 2016?', 'answer': 'The host of The Tonight Show in 2016 was Jimmy Fallon.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Jimmy Fallon.'}]], 'traces_entailments': None}",,['Jimmy Fallon'],1,1,1,1,32,"In 2016, who was the host of the longest running talk show?",Jimmy Fallon,," 226 | Are follow up questions needed here: Yes. 227 | Follow up: What was the longest running talk show? 228 | Intermediate answer: The longest running talk show is The Tonight Show. 229 | Follow up: Who was the host of The Tonight Show in 2016? 230 | Intermediate answer: The host of The Tonight Show in 2016 was Jimmy Fallon. 231 | So the final answer is: Jimmy Fallon. 232 | " 233 | 28,"{'question': 'When was the person Russ Hanneman is based on born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is Russ Hanneman based on?\nIntermediate answer: Russ Hanneman is based on Mark Cuban.\nFollow up: When was Mark Cuban born?\nIntermediate answer: Mark Cuban was born on July 31, 1958.\nSo the final answer is: July 31, 1958.\n'], 'decompsition_steps': [[{'question': 'Who is Russ Hanneman based on?', 'answer': 'Russ Hanneman is based on Mark Cuban.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Mark Cuban born?', 'answer': 'Mark Cuban was born on July 31, 1958.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'July 31, 1958.'}]], 'traces_entailments': None}",,"['July 31, 1958']",1,1,1,1,33,When was the person Russ Hanneman is based on born?,"July 31, 1958",," 234 | Are follow up questions needed here: Yes. 235 | Follow up: Who is Russ Hanneman based on? 236 | Intermediate answer: Russ Hanneman is based on Mark Cuban. 237 | Follow up: When was Mark Cuban born? 238 | Intermediate answer: Mark Cuban was born on July 31, 1958. 239 | So the final answer is: July 31, 1958. 240 | " 241 | 29,"{'question': ""When was the first location of the world's largest coffeehouse chain opened?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What is the world's largest coffeehouse chain?\nIntermediate answer: The world's largest coffeehouse chain is Starbucks.\nFollow up: When was Starbucks founded?\nIntermediate answer: Starbucks was founded on March 31, 1971.\nFollow up: Where was Starbucks founded?\nIntermediate answer: Starbucks was founded in Seattle, Washington.\nFollow up: When was Seattle founded?\nIntermediate answer: Seattle was founded on November 13""], 'decompsition_steps': [[{'question': ""What is the world's largest coffeehouse chain?"", 'answer': ""The world's largest coffeehouse chain is Starbucks."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Starbucks founded?', 'answer': 'Starbucks was founded on March 31, 1971.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Where was Starbucks founded?', 'answer': 'Starbucks was founded in Seattle, Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Seattle founded?', 'answer': 'Seattle was founded on November 13', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,34,When was the first location of the world's largest coffeehouse chain opened?,"March 30, 1971",," 242 | Are follow up questions needed here: Yes. 243 | Follow up: What is the world's largest coffeehouse chain? 244 | Intermediate answer: The world's largest coffeehouse chain is Starbucks. 245 | Follow up: When was Starbucks founded? 246 | Intermediate answer: Starbucks was founded on March 31, 1971. 247 | Follow up: Where was Starbucks founded? 248 | Intermediate answer: Starbucks was founded in Seattle, Washington. 249 | Follow up: When was Seattle founded? 250 | Intermediate answer: Seattle was founded on November 13" 251 | 30,"{'question': 'What is the maximum airspeed (in km/h) of the third fastest bird?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the third fastest bird?\nIntermediate answer: The third fastest bird is the Gyrfalcon.\nFollow up: What is the maximum airspeed (in km/h) of the Gyrfalcon?\nIntermediate answer: The maximum airspeed of the Gyrfalcon is 320 km/h.\nSo the final answer is: 320 km/h.\n'], 'decompsition_steps': [[{'question': 'What is the third fastest bird?', 'answer': 'The third fastest bird is the Gyrfalcon.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the maximum airspeed (in km/h) of the Gyrfalcon?', 'answer': 'The maximum airspeed of the Gyrfalcon is 320 km/h.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '320 km/h.'}]], 'traces_entailments': None}",,['320 km/h'],0.5,0.5,0.5,0.5,35,What is the maximum airspeed (in km/h) of the third fastest bird?,169 km/h,," 252 | Are follow up questions needed here: Yes. 253 | Follow up: What is the third fastest bird? 254 | Intermediate answer: The third fastest bird is the Gyrfalcon. 255 | Follow up: What is the maximum airspeed (in km/h) of the Gyrfalcon? 256 | Intermediate answer: The maximum airspeed of the Gyrfalcon is 320 km/h. 257 | So the final answer is: 320 km/h. 258 | " 259 | 31,"{'question': 'Who was the head of NASA during Apollo 11?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: When was Apollo 11?\nIntermediate answer: Apollo 11 was in 1969.\nFollow up: Who was the head of NASA in 1969?\nIntermediate answer: The head of NASA in 1969 was Thomas O. Paine.\nSo the final answer is: Thomas O. Paine.\n'], 'decompsition_steps': [[{'question': 'When was Apollo 11?', 'answer': 'Apollo 11 was in 1969.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the head of NASA in 1969?', 'answer': 'The head of NASA in 1969 was Thomas O. Paine.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Thomas O. Paine.'}]], 'traces_entailments': None}",,['Thomas O. Paine'],1,1,1,1,36,Who was the head of NASA during Apollo 11?,Thomas O. Paine,," 260 | Are follow up questions needed here: Yes. 261 | Follow up: When was Apollo 11? 262 | Intermediate answer: Apollo 11 was in 1969. 263 | Follow up: Who was the head of NASA in 1969? 264 | Intermediate answer: The head of NASA in 1969 was Thomas O. Paine. 265 | So the final answer is: Thomas O. Paine. 266 | " 267 | 32,"{'question': 'What is the official color of the third oldest surviving university?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the third oldest surviving university?\nIntermediate answer: The third oldest surviving university is the University of Bologna.\nFollow up: What is the official color of the University of Bologna?\nIntermediate answer: The official color of the University of Bologna is red.\nSo the final answer is: Red.\n'], 'decompsition_steps': [[{'question': 'What is the third oldest surviving university?', 'answer': 'The third oldest surviving university is the University of Bologna.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the official color of the University of Bologna?', 'answer': 'The official color of the University of Bologna is red.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Red.'}]], 'traces_entailments': None}",,['Red'],0,0,0,0,37,What is the official color of the third oldest surviving university?,Cambridge Blue,," 268 | Are follow up questions needed here: Yes. 269 | Follow up: What is the third oldest surviving university? 270 | Intermediate answer: The third oldest surviving university is the University of Bologna. 271 | Follow up: What is the official color of the University of Bologna? 272 | Intermediate answer: The official color of the University of Bologna is red. 273 | So the final answer is: Red. 274 | " 275 | 33,"{'question': 'What was the death toll of the most intense Atlantic hurricane?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the most intense Atlantic hurricane?\nIntermediate answer: The most intense Atlantic hurricane was the Great Miami Hurricane.\nFollow up: What was the death toll of the Great Miami Hurricane?\nIntermediate answer: The death toll of the Great Miami Hurricane was 403.\nSo the final answer is: 403.\n'], 'decompsition_steps': [[{'question': 'What was the most intense Atlantic hurricane?', 'answer': 'The most intense Atlantic hurricane was the Great Miami Hurricane.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the death toll of the Great Miami Hurricane?', 'answer': 'The death toll of the Great Miami Hurricane was 403.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '403.'}]], 'traces_entailments': None}",,['403'],0,0,0,0,38,What was the death toll of the most intense Atlantic hurricane?,52,," 276 | Are follow up questions needed here: Yes. 277 | Follow up: What was the most intense Atlantic hurricane? 278 | Intermediate answer: The most intense Atlantic hurricane was the Great Miami Hurricane. 279 | Follow up: What was the death toll of the Great Miami Hurricane? 280 | Intermediate answer: The death toll of the Great Miami Hurricane was 403. 281 | So the final answer is: 403. 282 | " 283 | 34,"{'question': 'The fourth largest city in Germany was originally called what?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the fourth largest city in Germany?\nIntermediate answer: The fourth largest city in Germany is Munich.\nFollow up: What was Munich originally called?\nIntermediate answer: Munich was originally called Munchen.\nSo the final answer is: Munchen.\n'], 'decompsition_steps': [[{'question': 'What is the fourth largest city in Germany?', 'answer': 'The fourth largest city in Germany is Munich.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was Munich originally called?', 'answer': 'Munich was originally called Munchen.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Munchen.'}]], 'traces_entailments': None}",,['Munchen'],0,0,0,0,39,The fourth largest city in Germany was originally called what?,Colonia Claudia Ara Agrippinensium,," 284 | Are follow up questions needed here: Yes. 285 | Follow up: What is the fourth largest city in Germany? 286 | Intermediate answer: The fourth largest city in Germany is Munich. 287 | Follow up: What was Munich originally called? 288 | Intermediate answer: Munich was originally called Munchen. 289 | So the final answer is: Munchen. 290 | " 291 | 35,"{'question': 'In what year was the tallest lighthouse completed?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the tallest lighthouse?\nIntermediate answer: The tallest lighthouse is the Jeddah Light.\nFollow up: When was the Jeddah Light completed?\nIntermediate answer: The Jeddah Light was completed in 1968.\nSo the final answer is: 1968.\n'], 'decompsition_steps': [[{'question': 'What is the tallest lighthouse?', 'answer': 'The tallest lighthouse is the Jeddah Light.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Jeddah Light completed?', 'answer': 'The Jeddah Light was completed in 1968.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '1968.'}]], 'traces_entailments': None}",,['1968'],0,0,0,0,40,In what year was the tallest lighthouse completed?,1902,," 292 | Are follow up questions needed here: Yes. 293 | Follow up: What is the tallest lighthouse? 294 | Intermediate answer: The tallest lighthouse is the Jeddah Light. 295 | Follow up: When was the Jeddah Light completed? 296 | Intermediate answer: The Jeddah Light was completed in 1968. 297 | So the final answer is: 1968. 298 | " 299 | 36,"{'question': 'When was the person who delivered the ""Quit India"" speech born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who delivered the ""Quit India"" speech?\nIntermediate answer: The person who delivered the ""Quit India"" speech was Mahatma Gandhi.\nFollow up: When was Mahatma Gandhi born?\nIntermediate answer: Mahatma Gandhi was born on October 2, 1869.\nSo the final answer is: October 2, 1869.\n'], 'decompsition_steps': [[{'question': 'Who delivered the ""Quit India"" speech?', 'answer': 'The person who delivered the ""Quit India"" speech was Mahatma Gandhi.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Mahatma Gandhi born?', 'answer': 'Mahatma Gandhi was born on October 2, 1869.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'October 2, 1869.'}]], 'traces_entailments': None}",,"['October 2, 1869']",1,1,1,1,41,"When was the person who delivered the ""Quit India"" speech born?","October 2, 1869",," 300 | Are follow up questions needed here: Yes. 301 | Follow up: Who delivered the ""Quit India"" speech? 302 | Intermediate answer: The person who delivered the ""Quit India"" speech was Mahatma Gandhi. 303 | Follow up: When was Mahatma Gandhi born? 304 | Intermediate answer: Mahatma Gandhi was born on October 2, 1869. 305 | So the final answer is: October 2, 1869. 306 | " 307 | 37,"{'question': 'In what year was the current tallest wooden lattice tower completed?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the current tallest wooden lattice tower?\nIntermediate answer: The current tallest wooden lattice tower is the Kaieteur National Park.\nFollow up: When was the Kaieteur National Park completed?\nIntermediate answer: The Kaieteur National Park was completed in 1929.\nSo the final answer is: 1929.\n'], 'decompsition_steps': [[{'question': 'What is the current tallest wooden lattice tower?', 'answer': 'The current tallest wooden lattice tower is the Kaieteur National Park.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Kaieteur National Park completed?', 'answer': 'The Kaieteur National Park was completed in 1929.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '1929.'}]], 'traces_entailments': None}",,['1929'],0,0,0,0,42,In what year was the current tallest wooden lattice tower completed?,1934,," 308 | Are follow up questions needed here: Yes. 309 | Follow up: What is the current tallest wooden lattice tower? 310 | Intermediate answer: The current tallest wooden lattice tower is the Kaieteur National Park. 311 | Follow up: When was the Kaieteur National Park completed? 312 | Intermediate answer: The Kaieteur National Park was completed in 1929. 313 | So the final answer is: 1929. 314 | " 315 | 38,"{'question': 'In what year was the government department where the internet originated at founded?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What government department was the internet originated at?\nIntermediate answer: The government department where the internet originated at was the Advanced Research Projects Agency (ARPA).\nFollow up: When was the Advanced Research Projects Agency founded?\nIntermediate answer: The Advanced Research Projects Agency was founded in 1958.\nSo the final answer is: 1958.\n'], 'decompsition_steps': [[{'question': 'What government department was the internet originated at?', 'answer': 'The government department where the internet originated at was the Advanced Research Projects Agency (ARPA).', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Advanced Research Projects Agency founded?', 'answer': 'The Advanced Research Projects Agency was founded in 1958.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '1958.'}]], 'traces_entailments': None}",,['1958'],0,0,0,0,43,In what year was the government department where the internet originated at founded?,1947,," 316 | Are follow up questions needed here: Yes. 317 | Follow up: What government department was the internet originated at? 318 | Intermediate answer: The government department where the internet originated at was the Advanced Research Projects Agency (ARPA). 319 | Follow up: When was the Advanced Research Projects Agency founded? 320 | Intermediate answer: The Advanced Research Projects Agency was founded in 1958. 321 | So the final answer is: 1958. 322 | " 323 | 39,"{'question': ""Which team won in women's volleyball in the most recent Summer Olympics that was held in London?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: When was the most recent Summer Olympics that was held in London?\nIntermediate answer: The most recent Summer Olympics that was held in London was in 2012.\nFollow up: Which team won in women's volleyball in the 2012 Summer Olympics?\nIntermediate answer: The team that won in women's volleyball in the 2012 Summer Olympics was the United States.\nSo the final answer is: The United States.\n""], 'decompsition_steps': [[{'question': 'When was the most recent Summer Olympics that was held in London?', 'answer': 'The most recent Summer Olympics that was held in London was in 2012.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""Which team won in women's volleyball in the 2012 Summer Olympics?"", 'answer': ""The team that won in women's volleyball in the 2012 Summer Olympics was the United States."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'The United States.'}]], 'traces_entailments': None}",,['The United States'],0,0,0,0,44,Which team won in women's volleyball in the most recent Summer Olympics that was held in London?,Brazil,," 324 | Are follow up questions needed here: Yes. 325 | Follow up: When was the most recent Summer Olympics that was held in London? 326 | Intermediate answer: The most recent Summer Olympics that was held in London was in 2012. 327 | Follow up: Which team won in women's volleyball in the 2012 Summer Olympics? 328 | Intermediate answer: The team that won in women's volleyball in the 2012 Summer Olympics was the United States. 329 | So the final answer is: The United States. 330 | " 331 | 40,"{'question': ""When did Nirvana's second most selling studio album come out?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What is Nirvana's second most selling studio album?\nIntermediate answer: Nirvana's second most selling studio album is Nevermind.\nFollow up: When did Nevermind come out?\nIntermediate answer: Nevermind came out on September 24, 1991.\nSo the final answer is: September 24, 1991.\n""], 'decompsition_steps': [[{'question': ""What is Nirvana's second most selling studio album?"", 'answer': ""Nirvana's second most selling studio album is Nevermind."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Nevermind come out?', 'answer': 'Nevermind came out on September 24, 1991.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'September 24, 1991.'}]], 'traces_entailments': None}",,"['September 24, 1991']",0.3333333333,0.3333333333,0.3333333333,0.3333333333,45,When did Nirvana's second most selling studio album come out?,"September 13, 1993",," 332 | Are follow up questions needed here: Yes. 333 | Follow up: What is Nirvana's second most selling studio album? 334 | Intermediate answer: Nirvana's second most selling studio album is Nevermind. 335 | Follow up: When did Nevermind come out? 336 | Intermediate answer: Nevermind came out on September 24, 1991. 337 | So the final answer is: September 24, 1991. 338 | " 339 | 41,"{'question': 'Who succeeded the longest reigning Roman emperor?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the longest reigning Roman emperor?\nIntermediate answer: The longest reigning Roman emperor was Augustus.\nFollow up: Who succeeded Augustus?\nIntermediate answer: Augustus was succeeded by Tiberius.\nSo the final answer is: Tiberius.\n'], 'decompsition_steps': [[{'question': 'Who was the longest reigning Roman emperor?', 'answer': 'The longest reigning Roman emperor was Augustus.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who succeeded Augustus?', 'answer': 'Augustus was succeeded by Tiberius.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Tiberius.'}]], 'traces_entailments': None}",,['Tiberius'],1,1,1,1,46,Who succeeded the longest reigning Roman emperor?,Tiberius,," 340 | Are follow up questions needed here: Yes. 341 | Follow up: Who was the longest reigning Roman emperor? 342 | Intermediate answer: The longest reigning Roman emperor was Augustus. 343 | Follow up: Who succeeded Augustus? 344 | Intermediate answer: Augustus was succeeded by Tiberius. 345 | So the final answer is: Tiberius. 346 | " 347 | 42,"{'question': 'The material out of which the Great Sphinx of Giza is made of is mainly composed of what mineral?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the Great Sphinx of Giza?\nIntermediate answer: The Great Sphinx of Giza is a statue of a reclining lion with a human head.\nFollow up: What is the main mineral that the Great Sphinx of Giza is made of?\nIntermediate answer: The main mineral that the Great Sphinx of Giza is made of is limestone.\nSo the final answer is: Limestone.\n'], 'decompsition_steps': [[{'question': 'What is the Great Sphinx of Giza?', 'answer': 'The Great Sphinx of Giza is a statue of a reclining lion with a human head.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the main mineral that the Great Sphinx of Giza is made of?', 'answer': 'The main mineral that the Great Sphinx of Giza is made of is limestone.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Limestone.'}]], 'traces_entailments': None}",,['Limestone'],0,0,0,0,47,The material out of which the Great Sphinx of Giza is made of is mainly composed of what mineral?,calcite,," 348 | Are follow up questions needed here: Yes. 349 | Follow up: What is the Great Sphinx of Giza? 350 | Intermediate answer: The Great Sphinx of Giza is a statue of a reclining lion with a human head. 351 | Follow up: What is the main mineral that the Great Sphinx of Giza is made of? 352 | Intermediate answer: The main mineral that the Great Sphinx of Giza is made of is limestone. 353 | So the final answer is: Limestone. 354 | " 355 | 43,"{'question': 'Who directed the highest grossing film?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the highest grossing film?\nIntermediate answer: The highest grossing film is Avatar.\nFollow up: Who directed Avatar?\nIntermediate answer: Avatar was directed by James Cameron.\nSo the final answer is: James Cameron.\n'], 'decompsition_steps': [[{'question': 'What is the highest grossing film?', 'answer': 'The highest grossing film is Avatar.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who directed Avatar?', 'answer': 'Avatar was directed by James Cameron.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'James Cameron.'}]], 'traces_entailments': None}",,['James Cameron'],0.5,0.5,0.5,0.5,48,Who directed the highest grossing film?,James Cameroon,," 356 | Are follow up questions needed here: Yes. 357 | Follow up: What is the highest grossing film? 358 | Intermediate answer: The highest grossing film is Avatar. 359 | Follow up: Who directed Avatar? 360 | Intermediate answer: Avatar was directed by James Cameron. 361 | So the final answer is: James Cameron. 362 | " 363 | 44,"{'question': 'When did the war that Neil Armstrong served in end?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What war did Neil Armstrong serve in?\nIntermediate answer: Neil Armstrong served in the Korean War.\nFollow up: When did the Korean War end?\nIntermediate answer: The Korean War ended on July 27, 1953.\nSo the final answer is: July 27, 1953.\n'], 'decompsition_steps': [[{'question': 'What war did Neil Armstrong serve in?', 'answer': 'Neil Armstrong served in the Korean War.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did the Korean War end?', 'answer': 'The Korean War ended on July 27, 1953.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'July 27, 1953.'}]], 'traces_entailments': None}",,"['July 27, 1953']",1,1,1,1,49,When did the war that Neil Armstrong served in end?,"July 27, 1953",," 364 | Are follow up questions needed here: Yes. 365 | Follow up: What war did Neil Armstrong serve in? 366 | Intermediate answer: Neil Armstrong served in the Korean War. 367 | Follow up: When did the Korean War end? 368 | Intermediate answer: The Korean War ended on July 27, 1953. 369 | So the final answer is: July 27, 1953. 370 | " 371 | 45,"{'question': 'Who is the father of the father of the hydrogen bomb?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of the hydrogen bomb?\nIntermediate answer: The father of the hydrogen bomb is Edward Teller.\nFollow up: Who is the father of Edward Teller?\nIntermediate answer: The father of Edward Teller is Adolf Teller.\nSo the final answer is: Adolf Teller.\n'], 'decompsition_steps': [[{'question': 'Who is the father of the hydrogen bomb?', 'answer': 'The father of the hydrogen bomb is Edward Teller.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Edward Teller?', 'answer': 'The father of Edward Teller is Adolf Teller.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Adolf Teller.'}]], 'traces_entailments': None}",,['Adolf Teller'],0.5,0.5,0.5,0.5,50,Who is the father of the father of the hydrogen bomb?,Max Teller,," 372 | Are follow up questions needed here: Yes. 373 | Follow up: Who is the father of the hydrogen bomb? 374 | Intermediate answer: The father of the hydrogen bomb is Edward Teller. 375 | Follow up: Who is the father of Edward Teller? 376 | Intermediate answer: The father of Edward Teller is Adolf Teller. 377 | So the final answer is: Adolf Teller. 378 | " 379 | 46,"{'question': ""What's the capital of the state that the Space Needle is at?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What's the Space Needle?\nIntermediate answer: The Space Needle is a tower in Seattle, Washington.\nFollow up: What's the capital of Washington?\nIntermediate answer: The capital of Washington is Olympia.\nSo the final answer is: Olympia.\n""], 'decompsition_steps': [[{'question': ""What's the Space Needle?"", 'answer': 'The Space Needle is a tower in Seattle, Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What's the capital of Washington?"", 'answer': 'The capital of Washington is Olympia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Olympia.'}]], 'traces_entailments': None}",,['Olympia'],1,1,1,1,51,What's the capital of the state that the Space Needle is at?,Olympia,," 380 | Are follow up questions needed here: Yes. 381 | Follow up: What's the Space Needle? 382 | Intermediate answer: The Space Needle is a tower in Seattle, Washington. 383 | Follow up: What's the capital of Washington? 384 | Intermediate answer: The capital of Washington is Olympia. 385 | So the final answer is: Olympia. 386 | " 387 | 47,"{'question': 'The main actor of Indiana Jones is a licensed what?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the main actor of Indiana Jones?\nIntermediate answer: The main actor of Indiana Jones is Harrison Ford.\nFollow up: What is Harrison Ford licensed to do?\nIntermediate answer: Harrison Ford is licensed to fly airplanes.\nSo the final answer is: Harrison Ford is licensed to fly airplanes.\n'], 'decompsition_steps': [[{'question': 'Who is the main actor of Indiana Jones?', 'answer': 'The main actor of Indiana Jones is Harrison Ford.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is Harrison Ford licensed to do?', 'answer': 'Harrison Ford is licensed to fly airplanes.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Harrison Ford is licensed to fly airplanes.'}]], 'traces_entailments': None}",,['Harrison Ford is licensed to fly airplanes'],0,0,0,0,52,The main actor of Indiana Jones is a licensed what?,pilot,," 388 | Are follow up questions needed here: Yes. 389 | Follow up: Who is the main actor of Indiana Jones? 390 | Intermediate answer: The main actor of Indiana Jones is Harrison Ford. 391 | Follow up: What is Harrison Ford licensed to do? 392 | Intermediate answer: Harrison Ford is licensed to fly airplanes. 393 | So the final answer is: Harrison Ford is licensed to fly airplanes. 394 | " 395 | 48,"{'question': 'Which element has an atomic number that is double that of hydrogen?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the atomic number of hydrogen?\nIntermediate answer: The atomic number of hydrogen is 1.\nFollow up: Which element has an atomic number that is double that of hydrogen?\nIntermediate answer: The element that has an atomic number that is double that of hydrogen is helium.\nSo the final answer is: Helium.\n'], 'decompsition_steps': [[{'question': 'What is the atomic number of hydrogen?', 'answer': 'The atomic number of hydrogen is 1.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Which element has an atomic number that is double that of hydrogen?', 'answer': 'The element that has an atomic number that is double that of hydrogen is helium.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Helium.'}]], 'traces_entailments': None}",,['Helium'],1,1,1,1,53,Which element has an atomic number that is double that of hydrogen?,Helium,," 396 | Are follow up questions needed here: Yes. 397 | Follow up: What is the atomic number of hydrogen? 398 | Intermediate answer: The atomic number of hydrogen is 1. 399 | Follow up: Which element has an atomic number that is double that of hydrogen? 400 | Intermediate answer: The element that has an atomic number that is double that of hydrogen is helium. 401 | So the final answer is: Helium. 402 | " 403 | 49,"{'question': 'When was the longest bridge in the world opened?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the longest bridge in the world?\nIntermediate answer: The longest bridge in the world is the Danyang-Kunshan Grand Bridge.\nFollow up: When was the Danyang-Kunshan Grand Bridge opened?\nIntermediate answer: The Danyang-Kunshan Grand Bridge was opened on June 30, 2011.\nSo the final answer is: June 30, 2011.\n'], 'decompsition_steps': [[{'question': 'What is the longest bridge in the world?', 'answer': 'The longest bridge in the world is the Danyang-Kunshan Grand Bridge.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Danyang-Kunshan Grand Bridge opened?', 'answer': 'The Danyang-Kunshan Grand Bridge was opened on June 30, 2011.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'June 30, 2011.'}]], 'traces_entailments': None}",,"['June 30, 2011']",0,0,0,0,54,When was the longest bridge in the world opened?,30-Jun-11,," 404 | Are follow up questions needed here: Yes. 405 | Follow up: What is the longest bridge in the world? 406 | Intermediate answer: The longest bridge in the world is the Danyang-Kunshan Grand Bridge. 407 | Follow up: When was the Danyang-Kunshan Grand Bridge opened? 408 | Intermediate answer: The Danyang-Kunshan Grand Bridge was opened on June 30, 2011. 409 | So the final answer is: June 30, 2011. 410 | " 411 | 50,"{'question': 'In 2016, who was the host of the longest running American game show?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the longest running American game show?\nIntermediate answer: The longest running American game show is Jeopardy.\nFollow up: Who was the host of Jeopardy in 2016?\nIntermediate answer: The host of Jeopardy in 2016 was Alex Trebek.\nSo the final answer is: Alex Trebek.\n'], 'decompsition_steps': [[{'question': 'What is the longest running American game show?', 'answer': 'The longest running American game show is Jeopardy.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the host of Jeopardy in 2016?', 'answer': 'The host of Jeopardy in 2016 was Alex Trebek.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Alex Trebek.'}]], 'traces_entailments': None}",,['Alex Trebek'],0,0,0,0,55,"In 2016, who was the host of the longest running American game show?",Drew Carey,," 412 | Are follow up questions needed here: Yes. 413 | Follow up: What is the longest running American game show? 414 | Intermediate answer: The longest running American game show is Jeopardy. 415 | Follow up: Who was the host of Jeopardy in 2016? 416 | Intermediate answer: The host of Jeopardy in 2016 was Alex Trebek. 417 | So the final answer is: Alex Trebek. 418 | " 419 | 51,"{'question': 'Who is the father of the scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013?\nIntermediate answer: The scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013 is Robert Langer.\nFollow up: Who is the father of Robert Langer?\nIntermediate answer: The father of Robert Langer is Robert Langer Sr.\nSo the final answer is: Robert Langer Sr.\n'], 'decompsition_steps': [[{'question': 'Who is the scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013?', 'answer': 'The scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013 is Robert Langer.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Robert Langer?', 'answer': 'The father of Robert Langer is Robert Langer Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Robert Langer Sr.'}]], 'traces_entailments': None}",,['Robert Langer Sr'],0,0,0,0,56,Who is the father of the scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013?,Conway Berners-Lee,," 420 | Are follow up questions needed here: Yes. 421 | Follow up: Who is the scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013? 422 | Intermediate answer: The scientist at MIT that won the Queen Elizabeth Prize for Engineering in 2013 is Robert Langer. 423 | Follow up: Who is the father of Robert Langer? 424 | Intermediate answer: The father of Robert Langer is Robert Langer Sr. 425 | So the final answer is: Robert Langer Sr. 426 | " 427 | 52,"{'question': 'Who is the largest aircraft carrier in the world is named after?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the largest aircraft carrier in the world?\nIntermediate answer: The largest aircraft carrier in the world is the USS Gerald R. Ford.\nFollow up: Who is the USS Gerald R. Ford named after?\nIntermediate answer: The USS Gerald R. Ford is named after Gerald R. Ford.\nSo the final answer is: Gerald R. Ford.\n'], 'decompsition_steps': [[{'question': 'What is the largest aircraft carrier in the world?', 'answer': 'The largest aircraft carrier in the world is the USS Gerald R. Ford.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the USS Gerald R. Ford named after?', 'answer': 'The USS Gerald R. Ford is named after Gerald R. Ford.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Gerald R. Ford.'}]], 'traces_entailments': None}",,['Gerald R. Ford'],1,1,1,1,57,Who is the largest aircraft carrier in the world is named after?,Gerald R. Ford,," 428 | Are follow up questions needed here: Yes. 429 | Follow up: What is the largest aircraft carrier in the world? 430 | Intermediate answer: The largest aircraft carrier in the world is the USS Gerald R. Ford. 431 | Follow up: Who is the USS Gerald R. Ford named after? 432 | Intermediate answer: The USS Gerald R. Ford is named after Gerald R. Ford. 433 | So the final answer is: Gerald R. Ford. 434 | " 435 | 53,"{'question': 'What is the capital of the country where yoga originated?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What country is yoga from?\nIntermediate answer: Yoga originated in India.\nFollow up: What is the capital of India?\nIntermediate answer: The capital of India is New Delhi.\nSo the final answer is: New Delhi.\n'], 'decompsition_steps': [[{'question': 'What country is yoga from?', 'answer': 'Yoga originated in India.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the capital of India?', 'answer': 'The capital of India is New Delhi.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'New Delhi.'}]], 'traces_entailments': None}",,['New Delhi'],1,1,1,1,58,What is the capital of the country where yoga originated?,New Delhi,," 436 | Are follow up questions needed here: Yes. 437 | Follow up: What country is yoga from? 438 | Intermediate answer: Yoga originated in India. 439 | Follow up: What is the capital of India? 440 | Intermediate answer: The capital of India is New Delhi. 441 | So the final answer is: New Delhi. 442 | " 443 | 54,"{'question': 'When did the president who said Tear Down This Wall die?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the president who said Tear Down This Wall?\nIntermediate answer: The president who said Tear Down This Wall was Ronald Reagan.\nFollow up: When did Ronald Reagan die?\nIntermediate answer: Ronald Reagan died on June 5, 2004.\nSo the final answer is: June 5, 2004.\n'], 'decompsition_steps': [[{'question': 'Who was the president who said Tear Down This Wall?', 'answer': 'The president who said Tear Down This Wall was Ronald Reagan.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Ronald Reagan die?', 'answer': 'Ronald Reagan died on June 5, 2004.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'June 5, 2004.'}]], 'traces_entailments': None}",,"['June 5, 2004']",1,1,1,1,59,When did the president who said Tear Down This Wall die?,"June 5, 2004",," 444 | Are follow up questions needed here: Yes. 445 | Follow up: Who was the president who said Tear Down This Wall? 446 | Intermediate answer: The president who said Tear Down This Wall was Ronald Reagan. 447 | Follow up: When did Ronald Reagan die? 448 | Intermediate answer: Ronald Reagan died on June 5, 2004. 449 | So the final answer is: June 5, 2004. 450 | " 451 | 55,"{'question': 'When was the composer of Carol of the Bells born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the composer of Carol of the Bells?\nIntermediate answer: The composer of Carol of the Bells is Mykola Leontovych.\nFollow up: When was Mykola Leontovych born?\nIntermediate answer: Mykola Leontovych was born on December 13, 1877.\nSo the final answer is: December 13, 1877.\n'], 'decompsition_steps': [[{'question': 'Who was the composer of Carol of the Bells?', 'answer': 'The composer of Carol of the Bells is Mykola Leontovych.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Mykola Leontovych born?', 'answer': 'Mykola Leontovych was born on December 13, 1877.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'December 13, 1877.'}]], 'traces_entailments': None}",,"['December 13, 1877']",1,1,1,1,60,When was the composer of Carol of the Bells born?,"December 13, 1877",," 452 | Are follow up questions needed here: Yes. 453 | Follow up: Who was the composer of Carol of the Bells? 454 | Intermediate answer: The composer of Carol of the Bells is Mykola Leontovych. 455 | Follow up: When was Mykola Leontovych born? 456 | Intermediate answer: Mykola Leontovych was born on December 13, 1877. 457 | So the final answer is: December 13, 1877. 458 | " 459 | 56,"{'question': 'When was the company that built the first steam locomotive to carry passengers on a public rail line founded?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What company built the first steam locomotive to carry passengers on a public rail line?\nIntermediate answer: The company that built the first steam locomotive to carry passengers on a public rail line was the Baltimore and Ohio Railroad.\nFollow up: When was the Baltimore and Ohio Railroad founded?\nIntermediate answer: The Baltimore and Ohio Railroad was founded on July 4, 1827.\nSo the final answer is: July 4, 1827.\n'], 'decompsition_steps': [[{'question': 'What company built the first steam locomotive to carry passengers on a public rail line?', 'answer': 'The company that built the first steam locomotive to carry passengers on a public rail line was the Baltimore and Ohio Railroad.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Baltimore and Ohio Railroad founded?', 'answer': 'The Baltimore and Ohio Railroad was founded on July 4, 1827.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'July 4, 1827.'}]], 'traces_entailments': None}",,"['July 4, 1827']",0,0,0,0,61,When was the company that built the first steam locomotive to carry passengers on a public rail line founded?,1823,," 460 | Are follow up questions needed here: Yes. 461 | Follow up: What company built the first steam locomotive to carry passengers on a public rail line? 462 | Intermediate answer: The company that built the first steam locomotive to carry passengers on a public rail line was the Baltimore and Ohio Railroad. 463 | Follow up: When was the Baltimore and Ohio Railroad founded? 464 | Intermediate answer: The Baltimore and Ohio Railroad was founded on July 4, 1827. 465 | So the final answer is: July 4, 1827. 466 | " 467 | 57,"{'question': 'The machine used to extract honey from honeycombs uses which physical force?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the machine used to extract honey from honeycombs called?\nIntermediate answer: The machine used to extract honey from honeycombs is called an extractor.\nFollow up: What physical force is used to extract honey from honeycombs?\nIntermediate answer: The physical force used to extract honey from honeycombs is centrifugal force.\nSo the final answer is: Centrifugal force.\n'], 'decompsition_steps': [[{'question': 'What is the machine used to extract honey from honeycombs called?', 'answer': 'The machine used to extract honey from honeycombs is called an extractor.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What physical force is used to extract honey from honeycombs?', 'answer': 'The physical force used to extract honey from honeycombs is centrifugal force.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Centrifugal force.'}]], 'traces_entailments': None}",,['Centrifugal force'],1,1,1,1,62,The machine used to extract honey from honeycombs uses which physical force?,Centrifugal Force,," 468 | Are follow up questions needed here: Yes. 469 | Follow up: What is the machine used to extract honey from honeycombs called? 470 | Intermediate answer: The machine used to extract honey from honeycombs is called an extractor. 471 | Follow up: What physical force is used to extract honey from honeycombs? 472 | Intermediate answer: The physical force used to extract honey from honeycombs is centrifugal force. 473 | So the final answer is: Centrifugal force. 474 | " 475 | 58,"{'question': 'The husband of Lady Godiva was Earl of which Anglic kingdom?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was Lady Godiva?\nIntermediate answer: Lady Godiva was a noblewoman who rode naked through the streets of Coventry.\nFollow up: What was the name of the Anglic kingdom in which Lady Godiva lived?\nIntermediate answer: The Anglic kingdom in which Lady Godiva lived was Mercia.\nFollow up: What was the name of the Earl of Mercia?\nIntermediate answer: The Earl of Mercia was Leofric.\nSo the final answer is: Leofric.\n'], 'decompsition_steps': [[{'question': 'Who was Lady Godiva?', 'answer': 'Lady Godiva was a noblewoman who rode naked through the streets of Coventry.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the name of the Anglic kingdom in which Lady Godiva lived?', 'answer': 'The Anglic kingdom in which Lady Godiva lived was Mercia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the name of the Earl of Mercia?', 'answer': 'The Earl of Mercia was Leofric.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Leofric.'}]], 'traces_entailments': None}",,['Leofric'],0,0,0,0,63,The husband of Lady Godiva was Earl of which Anglic kingdom?,Mercia,," 476 | Are follow up questions needed here: Yes. 477 | Follow up: Who was Lady Godiva? 478 | Intermediate answer: Lady Godiva was a noblewoman who rode naked through the streets of Coventry. 479 | Follow up: What was the name of the Anglic kingdom in which Lady Godiva lived? 480 | Intermediate answer: The Anglic kingdom in which Lady Godiva lived was Mercia. 481 | Follow up: What was the name of the Earl of Mercia? 482 | Intermediate answer: The Earl of Mercia was Leofric. 483 | So the final answer is: Leofric. 484 | " 485 | 59,"{'question': 'Who is the father of the father of Barack Obama?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is Barack Obama?\nIntermediate answer: Barack Obama is the 44th President of the United States.\nFollow up: Who is the father of Barack Obama?\nIntermediate answer: The father of Barack Obama is Barack Obama Sr.\nFollow up: Who is the father of Barack Obama Sr.?\nIntermediate answer: The father of Barack Obama Sr. is Hussein Onyango Obama.\nSo the final answer is: Hussein Ony'], 'decompsition_steps': [[{'question': 'Who is Barack Obama?', 'answer': 'Barack Obama is the 44th President of the United States.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Barack Obama?', 'answer': 'The father of Barack Obama is Barack Obama Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Barack Obama Sr.?', 'answer': 'The father of Barack Obama Sr. is Hussein Onyango Obama.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Hussein Ony'}]], 'traces_entailments': None}",,['Hussein Ony'],0.4,0.4,0.4,0.4,64,Who is the father of the father of Barack Obama?,Hussein Onyango Obama,," 486 | Are follow up questions needed here: Yes. 487 | Follow up: Who is Barack Obama? 488 | Intermediate answer: Barack Obama is the 44th President of the United States. 489 | Follow up: Who is the father of Barack Obama? 490 | Intermediate answer: The father of Barack Obama is Barack Obama Sr. 491 | Follow up: Who is the father of Barack Obama Sr.? 492 | Intermediate answer: The father of Barack Obama Sr. is Hussein Onyango Obama. 493 | So the final answer is: Hussein Ony" 494 | 60,"{'question': 'In what country is the world largest desalination plant?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the world largest desalination plant?\nIntermediate answer: The world largest desalination plant is the Sorek Desalination Plant.\nFollow up: Where is the Sorek Desalination Plant?\nIntermediate answer: The Sorek Desalination Plant is in Israel.\nFollow up: In what country is Israel?\nIntermediate answer: Israel is in the Middle East.\nSo the final answer is: The Middle East.\n'], 'decompsition_steps': [[{'question': 'What is the world largest desalination plant?', 'answer': 'The world largest desalination plant is the Sorek Desalination Plant.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Where is the Sorek Desalination Plant?', 'answer': 'The Sorek Desalination Plant is in Israel.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'In what country is Israel?', 'answer': 'Israel is in the Middle East.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'The Middle East.'}]], 'traces_entailments': None}",,['The Middle East'],0,0,0,0,65,In what country is the world largest desalination plant?,Israel,," 495 | Are follow up questions needed here: Yes. 496 | Follow up: What is the world largest desalination plant? 497 | Intermediate answer: The world largest desalination plant is the Sorek Desalination Plant. 498 | Follow up: Where is the Sorek Desalination Plant? 499 | Intermediate answer: The Sorek Desalination Plant is in Israel. 500 | Follow up: In what country is Israel? 501 | Intermediate answer: Israel is in the Middle East. 502 | So the final answer is: The Middle East. 503 | " 504 | 61,"{'question': 'When was the philosopher that formulated the hard problem of consciousness born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the philosopher that formulated the hard problem of consciousness?\nIntermediate answer: The philosopher that formulated the hard problem of consciousness is David Chalmers.\nFollow up: When was David Chalmers born?\nIntermediate answer: David Chalmers was born on April 20, 1966.\nSo the final answer is: April 20, 1966.\n'], 'decompsition_steps': [[{'question': 'Who was the philosopher that formulated the hard problem of consciousness?', 'answer': 'The philosopher that formulated the hard problem of consciousness is David Chalmers.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was David Chalmers born?', 'answer': 'David Chalmers was born on April 20, 1966.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'April 20, 1966.'}]], 'traces_entailments': None}",,"['April 20, 1966']",1,1,1,1,66,When was the philosopher that formulated the hard problem of consciousness born?,"April 20, 1966",," 505 | Are follow up questions needed here: Yes. 506 | Follow up: Who was the philosopher that formulated the hard problem of consciousness? 507 | Intermediate answer: The philosopher that formulated the hard problem of consciousness is David Chalmers. 508 | Follow up: When was David Chalmers born? 509 | Intermediate answer: David Chalmers was born on April 20, 1966. 510 | So the final answer is: April 20, 1966. 511 | " 512 | 62,"{'question': 'What is the political party of the American president who entered into the Paris agreement?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the Paris agreement?\nIntermediate answer: The Paris agreement is an agreement to reduce greenhouse gas emissions.\nFollow up: What is the political party of the American president who entered into the Paris agreement?\nIntermediate answer: The political party of the American president who entered into the Paris agreement is the Democratic Party.\nSo the final answer is: Democratic Party.\n'], 'decompsition_steps': [[{'question': 'What is the Paris agreement?', 'answer': 'The Paris agreement is an agreement to reduce greenhouse gas emissions.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the political party of the American president who entered into the Paris agreement?', 'answer': 'The political party of the American president who entered into the Paris agreement is the Democratic Party.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Democratic Party.'}]], 'traces_entailments': None}",,['Democratic Party'],1,1,1,1,67,What is the political party of the American president who entered into the Paris agreement?,Democratic Party,," 513 | Are follow up questions needed here: Yes. 514 | Follow up: What is the Paris agreement? 515 | Intermediate answer: The Paris agreement is an agreement to reduce greenhouse gas emissions. 516 | Follow up: What is the political party of the American president who entered into the Paris agreement? 517 | Intermediate answer: The political party of the American president who entered into the Paris agreement is the Democratic Party. 518 | So the final answer is: Democratic Party. 519 | " 520 | 63,"{'question': 'Who is the father of the father of George Washington?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of George Washington?\nIntermediate answer: The father of George Washington is Augustine Washington.\nFollow up: Who is the father of Augustine Washington?\nIntermediate answer: The father of Augustine Washington is Lawrence Washington.\nFollow up: Who is the father of Lawrence Washington?\nIntermediate answer: The father of Lawrence Washington is John Washington.\nFollow up: Who is the father of John Washington?\nIntermediate answer: The father of John Washington is Lawrence Washington.\nSo the final answer is:'], 'decompsition_steps': [[{'question': 'Who is the father of George Washington?', 'answer': 'The father of George Washington is Augustine Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Augustine Washington?', 'answer': 'The father of Augustine Washington is Lawrence Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Lawrence Washington?', 'answer': 'The father of Lawrence Washington is John Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of John Washington?', 'answer': 'The father of John Washington is Lawrence Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,68,Who is the father of the father of George Washington?,Lawrence Washington,," 521 | Are follow up questions needed here: Yes. 522 | Follow up: Who is the father of George Washington? 523 | Intermediate answer: The father of George Washington is Augustine Washington. 524 | Follow up: Who is the father of Augustine Washington? 525 | Intermediate answer: The father of Augustine Washington is Lawrence Washington. 526 | Follow up: Who is the father of Lawrence Washington? 527 | Intermediate answer: The father of Lawrence Washington is John Washington. 528 | Follow up: Who is the father of John Washington? 529 | Intermediate answer: The father of John Washington is Lawrence Washington. 530 | So the final answer is:" 531 | 64,"{'question': 'Who is the father of the originator of cybernetics?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the originator of cybernetics?\nIntermediate answer: The originator of cybernetics is Norbert Wiener.\nFollow up: Who is the father of Norbert Wiener?\nIntermediate answer: The father of Norbert Wiener is Leo Wiener.\nSo the final answer is: Leo Wiener.\n'], 'decompsition_steps': [[{'question': 'Who is the originator of cybernetics?', 'answer': 'The originator of cybernetics is Norbert Wiener.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Norbert Wiener?', 'answer': 'The father of Norbert Wiener is Leo Wiener.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Leo Wiener.'}]], 'traces_entailments': None}",,['Leo Wiener'],1,1,1,1,69,Who is the father of the originator of cybernetics?,Leo Wiener,," 532 | Are follow up questions needed here: Yes. 533 | Follow up: Who is the originator of cybernetics? 534 | Intermediate answer: The originator of cybernetics is Norbert Wiener. 535 | Follow up: Who is the father of Norbert Wiener? 536 | Intermediate answer: The father of Norbert Wiener is Leo Wiener. 537 | So the final answer is: Leo Wiener. 538 | " 539 | 65,"{'question': 'What is the highest elevation (in meters) of the second largest island in the world?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the second largest island in the world?\nIntermediate answer: The second largest island in the world is New Guinea.\nFollow up: What is the highest elevation (in meters) of New Guinea?\nIntermediate answer: The highest elevation (in meters) of New Guinea is 4,884 meters.\nSo the final answer is: 4,884 meters.\n'], 'decompsition_steps': [[{'question': 'What is the second largest island in the world?', 'answer': 'The second largest island in the world is New Guinea.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the highest elevation (in meters) of New Guinea?', 'answer': 'The highest elevation (in meters) of New Guinea is 4,884 meters.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '4,884 meters.'}]], 'traces_entailments': None}",,"['4,884 meters']",0,0,0,0,70,What is the highest elevation (in meters) of the second largest island in the world?,"4,509 m",," 540 | Are follow up questions needed here: Yes. 541 | Follow up: What is the second largest island in the world? 542 | Intermediate answer: The second largest island in the world is New Guinea. 543 | Follow up: What is the highest elevation (in meters) of New Guinea? 544 | Intermediate answer: The highest elevation (in meters) of New Guinea is 4,884 meters. 545 | So the final answer is: 4,884 meters. 546 | " 547 | 66,"{'question': 'When did the first prime minister of the Russian Empire come into office?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the first prime minister of the Russian Empire?\nIntermediate answer: The first prime minister of the Russian Empire was Pyotr Arkadyevich Stolypin.\nFollow up: When did Pyotr Arkadyevich Stolypin come into office?\nIntermediate answer: Pyotr Arkadyevich Stolypin came into office on July 21, 1906.\nSo the final answer is: July 21, 1906.\n'], 'decompsition_steps': [[{'question': 'Who was the first prime minister of the Russian Empire?', 'answer': 'The first prime minister of the Russian Empire was Pyotr Arkadyevich Stolypin.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Pyotr Arkadyevich Stolypin come into office?', 'answer': 'Pyotr Arkadyevich Stolypin came into office on July 21, 1906.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'July 21, 1906.'}]], 'traces_entailments': None}",,"['July 21, 1906']",0,0,0,0,71,When did the first prime minister of the Russian Empire come into office?,"November 6, 1905",," 548 | Are follow up questions needed here: Yes. 549 | Follow up: Who was the first prime minister of the Russian Empire? 550 | Intermediate answer: The first prime minister of the Russian Empire was Pyotr Arkadyevich Stolypin. 551 | Follow up: When did Pyotr Arkadyevich Stolypin come into office? 552 | Intermediate answer: Pyotr Arkadyevich Stolypin came into office on July 21, 1906. 553 | So the final answer is: July 21, 1906. 554 | " 555 | 67,"{'question': 'What is the fastest air-breathing manned aircraft mostly made out of?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the fastest air-breathing manned aircraft?\nIntermediate answer: The fastest air-breathing manned aircraft is the SR-71 Blackbird.\nFollow up: What is the SR-71 Blackbird mostly made out of?\nIntermediate answer: The SR-71 Blackbird is mostly made out of titanium.\nSo the final answer is: Titanium.\n'], 'decompsition_steps': [[{'question': 'What is the fastest air-breathing manned aircraft?', 'answer': 'The fastest air-breathing manned aircraft is the SR-71 Blackbird.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the SR-71 Blackbird mostly made out of?', 'answer': 'The SR-71 Blackbird is mostly made out of titanium.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Titanium.'}]], 'traces_entailments': None}",,['Titanium'],1,1,1,1,72,What is the fastest air-breathing manned aircraft mostly made out of?,Titanium,," 556 | Are follow up questions needed here: Yes. 557 | Follow up: What is the fastest air-breathing manned aircraft? 558 | Intermediate answer: The fastest air-breathing manned aircraft is the SR-71 Blackbird. 559 | Follow up: What is the SR-71 Blackbird mostly made out of? 560 | Intermediate answer: The SR-71 Blackbird is mostly made out of titanium. 561 | So the final answer is: Titanium. 562 | " 563 | 68,"{'question': 'How much protein in four boiled egg yolks?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: How many grams of protein are in four boiled egg yolks?\nIntermediate answer: There are 12 grams of protein in four boiled egg yolks.\nFollow up: How many grams are in a boiled egg yolk?\nIntermediate answer: There are 3.6 grams of protein in a boiled egg yolk.\nSo the final answer is: 12 grams.\n'], 'decompsition_steps': [[{'question': 'How many grams of protein are in four boiled egg yolks?', 'answer': 'There are 12 grams of protein in four boiled egg yolks.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'How many grams are in a boiled egg yolk?', 'answer': 'There are 3.6 grams of protein in a boiled egg yolk.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '12 grams.'}]], 'traces_entailments': None}",,['12 grams'],0,0,0,0,73,How much protein in four boiled egg yolks?,10.8,," 564 | Are follow up questions needed here: Yes. 565 | Follow up: How many grams of protein are in four boiled egg yolks? 566 | Intermediate answer: There are 12 grams of protein in four boiled egg yolks. 567 | Follow up: How many grams are in a boiled egg yolk? 568 | Intermediate answer: There are 3.6 grams of protein in a boiled egg yolk. 569 | So the final answer is: 12 grams. 570 | " 571 | 69,"{'question': 'Who wrote the novel on which the longest running show in Broadway history is based on?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the longest running show in Broadway history?\nIntermediate answer: The longest running show in Broadway history is Phantom of the Opera.\nFollow up: Who wrote the novel on which Phantom of the Opera is based on?\nIntermediate answer: The novel on which Phantom of the Opera is based on is The Phantom of the Opera by Gaston Leroux.\nSo the final answer is: Gaston Leroux.\n'], 'decompsition_steps': [[{'question': 'What is the longest running show in Broadway history?', 'answer': 'The longest running show in Broadway history is Phantom of the Opera.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who wrote the novel on which Phantom of the Opera is based on?', 'answer': 'The novel on which Phantom of the Opera is based on is The Phantom of the Opera by Gaston Leroux.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Gaston Leroux.'}]], 'traces_entailments': None}",,['Gaston Leroux'],1,1,1,1,74,Who wrote the novel on which the longest running show in Broadway history is based on?,Gaston Leroux,," 572 | Are follow up questions needed here: Yes. 573 | Follow up: What is the longest running show in Broadway history? 574 | Intermediate answer: The longest running show in Broadway history is Phantom of the Opera. 575 | Follow up: Who wrote the novel on which Phantom of the Opera is based on? 576 | Intermediate answer: The novel on which Phantom of the Opera is based on is The Phantom of the Opera by Gaston Leroux. 577 | So the final answer is: Gaston Leroux. 578 | " 579 | 70,"{'question': 'When did the person who delivered the Gettysburg Address die?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who delivered the Gettysburg Address?\nIntermediate answer: The person who delivered the Gettysburg Address was Abraham Lincoln.\nFollow up: When did Abraham Lincoln die?\nIntermediate answer: Abraham Lincoln died on April 15, 1865.\nSo the final answer is: April 15, 1865.\n'], 'decompsition_steps': [[{'question': 'Who delivered the Gettysburg Address?', 'answer': 'The person who delivered the Gettysburg Address was Abraham Lincoln.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Abraham Lincoln die?', 'answer': 'Abraham Lincoln died on April 15, 1865.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'April 15, 1865.'}]], 'traces_entailments': None}",,"['April 15, 1865']",1,1,1,1,75,When did the person who delivered the Gettysburg Address die?,"April 15, 1865",," 580 | Are follow up questions needed here: Yes. 581 | Follow up: Who delivered the Gettysburg Address? 582 | Intermediate answer: The person who delivered the Gettysburg Address was Abraham Lincoln. 583 | Follow up: When did Abraham Lincoln die? 584 | Intermediate answer: Abraham Lincoln died on April 15, 1865. 585 | So the final answer is: April 15, 1865. 586 | " 587 | 71,"{'question': 'In what year was the tallest lattice tower completed?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the tallest lattice tower?\nIntermediate answer: The tallest lattice tower is the Eiffel Tower.\nFollow up: When was the Eiffel Tower completed?\nIntermediate answer: The Eiffel Tower was completed in 1889.\nSo the final answer is: 1889.\n'], 'decompsition_steps': [[{'question': 'What is the tallest lattice tower?', 'answer': 'The tallest lattice tower is the Eiffel Tower.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Eiffel Tower completed?', 'answer': 'The Eiffel Tower was completed in 1889.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '1889.'}]], 'traces_entailments': None}",,['1889'],0,0,0,0,76,In what year was the tallest lattice tower completed?,2012,," 588 | Are follow up questions needed here: Yes. 589 | Follow up: What is the tallest lattice tower? 590 | Intermediate answer: The tallest lattice tower is the Eiffel Tower. 591 | Follow up: When was the Eiffel Tower completed? 592 | Intermediate answer: The Eiffel Tower was completed in 1889. 593 | So the final answer is: 1889. 594 | " 595 | 72,"{'question': 'When did the president who set the precedent of a two term limit leave office?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the president who set the precedent of a two term limit?\nIntermediate answer: The president who set the precedent of a two term limit was George Washington.\nFollow up: When did George Washington leave office?\nIntermediate answer: George Washington left office on March 4, 1797.\nSo the final answer is: March 4, 1797.\n'], 'decompsition_steps': [[{'question': 'Who was the president who set the precedent of a two term limit?', 'answer': 'The president who set the precedent of a two term limit was George Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did George Washington leave office?', 'answer': 'George Washington left office on March 4, 1797.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'March 4, 1797.'}]], 'traces_entailments': None}",,"['March 4, 1797']",1,1,1,1,77,When did the president who set the precedent of a two term limit leave office?,"March 4, 1797",," 596 | Are follow up questions needed here: Yes. 597 | Follow up: Who was the president who set the precedent of a two term limit? 598 | Intermediate answer: The president who set the precedent of a two term limit was George Washington. 599 | Follow up: When did George Washington leave office? 600 | Intermediate answer: George Washington left office on March 4, 1797. 601 | So the final answer is: March 4, 1797. 602 | " 603 | 73,"{'question': ""What's the capital of the state that Harvard University is in?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What state is Harvard University in?\nIntermediate answer: Harvard University is in Massachusetts.\nFollow up: What's the capital of Massachusetts?\nIntermediate answer: The capital of Massachusetts is Boston.\nSo the final answer is: Boston.\n""], 'decompsition_steps': [[{'question': 'What state is Harvard University in?', 'answer': 'Harvard University is in Massachusetts.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What's the capital of Massachusetts?"", 'answer': 'The capital of Massachusetts is Boston.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Boston.'}]], 'traces_entailments': None}",,['Boston'],1,1,1,1,78,What's the capital of the state that Harvard University is in?,Boston,," 604 | Are follow up questions needed here: Yes. 605 | Follow up: What state is Harvard University in? 606 | Intermediate answer: Harvard University is in Massachusetts. 607 | Follow up: What's the capital of Massachusetts? 608 | Intermediate answer: The capital of Massachusetts is Boston. 609 | So the final answer is: Boston. 610 | " 611 | 74,"{'question': ""For which club did the winner of the 2007 Ballon d'Or play for in 2012?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: Who won the 2007 Ballon d'Or?\nIntermediate answer: The winner of the 2007 Ballon d'Or was Kaka.\nFollow up: For which club did Kaka play in 2012?\nIntermediate answer: Kaka played for Real Madrid in 2012.\nSo the final answer is: Real Madrid.\n""], 'decompsition_steps': [[{'question': ""Who won the 2007 Ballon d'Or?"", 'answer': ""The winner of the 2007 Ballon d'Or was Kaka."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'For which club did Kaka play in 2012?', 'answer': 'Kaka played for Real Madrid in 2012.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Real Madrid.'}]], 'traces_entailments': None}",,['Real Madrid'],1,1,1,1,79,For which club did the winner of the 2007 Ballon d'Or play for in 2012?,Real Madrid,," 612 | Are follow up questions needed here: Yes. 613 | Follow up: Who won the 2007 Ballon d'Or? 614 | Intermediate answer: The winner of the 2007 Ballon d'Or was Kaka. 615 | Follow up: For which club did Kaka play in 2012? 616 | Intermediate answer: Kaka played for Real Madrid in 2012. 617 | So the final answer is: Real Madrid. 618 | " 619 | 75,"{'question': 'Who is the mother of the father of George Washington?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of George Washington?\nIntermediate answer: The father of George Washington is Augustine Washington.\nFollow up: Who is the mother of Augustine Washington?\nIntermediate answer: The mother of Augustine Washington is Mary Ball Washington.\nSo the final answer is: Mary Ball Washington.\n'], 'decompsition_steps': [[{'question': 'Who is the father of George Washington?', 'answer': 'The father of George Washington is Augustine Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the mother of Augustine Washington?', 'answer': 'The mother of Augustine Washington is Mary Ball Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Mary Ball Washington.'}]], 'traces_entailments': None}",,['Mary Ball Washington'],0,0,0,0,80,Who is the mother of the father of George Washington?,Mildred Warner,," 620 | Are follow up questions needed here: Yes. 621 | Follow up: Who is the father of George Washington? 622 | Intermediate answer: The father of George Washington is Augustine Washington. 623 | Follow up: Who is the mother of Augustine Washington? 624 | Intermediate answer: The mother of Augustine Washington is Mary Ball Washington. 625 | So the final answer is: Mary Ball Washington. 626 | " 627 | 76,"{'question': 'What is the nickname for the state that Mount Rainier is located in?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the state that Mount Rainier is located in?\nIntermediate answer: The state that Mount Rainier is located in is Washington.\nFollow up: What is the nickname for Washington?\nIntermediate answer: The nickname for Washington is the Evergreen State.\nSo the final answer is: The Evergreen State.\n'], 'decompsition_steps': [[{'question': 'What is the state that Mount Rainier is located in?', 'answer': 'The state that Mount Rainier is located in is Washington.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the nickname for Washington?', 'answer': 'The nickname for Washington is the Evergreen State.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'The Evergreen State.'}]], 'traces_entailments': None}",,['The Evergreen State'],1,1,1,1,81,What is the nickname for the state that Mount Rainier is located in?,Evergreen State,," 628 | Are follow up questions needed here: Yes. 629 | Follow up: What is the state that Mount Rainier is located in? 630 | Intermediate answer: The state that Mount Rainier is located in is Washington. 631 | Follow up: What is the nickname for Washington? 632 | Intermediate answer: The nickname for Washington is the Evergreen State. 633 | So the final answer is: The Evergreen State. 634 | " 635 | 77,"{'question': 'Who was the father of the father of behaviorism?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the father of behaviorism?\nIntermediate answer: The father of behaviorism was John B. Watson.\nFollow up: Who was the father of John B. Watson?\nIntermediate answer: The father of John B. Watson was John Broadus Watson.\nSo the final answer is: John Broadus Watson.\n'], 'decompsition_steps': [[{'question': 'Who was the father of behaviorism?', 'answer': 'The father of behaviorism was John B. Watson.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of John B. Watson?', 'answer': 'The father of John B. Watson was John Broadus Watson.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'John Broadus Watson.'}]], 'traces_entailments': None}",,['John Broadus Watson'],0.3333333333,0.3333333333,0.3333333333,0.3333333333,82,Who was the father of the father of behaviorism?,Pickens Butler Watson,," 636 | Are follow up questions needed here: Yes. 637 | Follow up: Who was the father of behaviorism? 638 | Intermediate answer: The father of behaviorism was John B. Watson. 639 | Follow up: Who was the father of John B. Watson? 640 | Intermediate answer: The father of John B. Watson was John Broadus Watson. 641 | So the final answer is: John Broadus Watson. 642 | " 643 | 78,"{'question': 'When was the author of The Population Bomb born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the author of The Population Bomb?\nIntermediate answer: The author of The Population Bomb is Paul Ehrlich.\nFollow up: When was Paul Ehrlich born?\nIntermediate answer: Paul Ehrlich was born on May 29, 1932.\nSo the final answer is: May 29, 1932.\n'], 'decompsition_steps': [[{'question': 'Who was the author of The Population Bomb?', 'answer': 'The author of The Population Bomb is Paul Ehrlich.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Paul Ehrlich born?', 'answer': 'Paul Ehrlich was born on May 29, 1932.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'May 29, 1932.'}]], 'traces_entailments': None}",,"['May 29, 1932']",1,1,1,1,83,When was the author of The Population Bomb born?,"May 29, 1932",," 644 | Are follow up questions needed here: Yes. 645 | Follow up: Who was the author of The Population Bomb? 646 | Intermediate answer: The author of The Population Bomb is Paul Ehrlich. 647 | Follow up: When was Paul Ehrlich born? 648 | Intermediate answer: Paul Ehrlich was born on May 29, 1932. 649 | So the final answer is: May 29, 1932. 650 | " 651 | 79,"{'question': 'The third largest river (by discharge) in the world is in what countries?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the third largest river (by discharge) in the world?\nIntermediate answer: The third largest river (by discharge) in the world is the Amazon River.\nFollow up: What countries is the Amazon River in?\nIntermediate answer: The Amazon River is in Brazil, Peru, and Colombia.\nSo the final answer is: Brazil, Peru, and Colombia.\n'], 'decompsition_steps': [[{'question': 'What is the third largest river (by discharge) in the world?', 'answer': 'The third largest river (by discharge) in the world is the Amazon River.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What countries is the Amazon River in?', 'answer': 'The Amazon River is in Brazil, Peru, and Colombia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Brazil, Peru, and Colombia.'}]], 'traces_entailments': None}",,"['Brazil, Peru, and Colombia']",0,0,0,0,84,The third largest river (by discharge) in the world is in what countries?," 652 | Gabon, Burundi, Cameroon, Central African Republic, Democratic Republic of the Congo, Republic of the Congo, Rwanda, Tanzania, Zambia",," 653 | Are follow up questions needed here: Yes. 654 | Follow up: What is the third largest river (by discharge) in the world? 655 | Intermediate answer: The third largest river (by discharge) in the world is the Amazon River. 656 | Follow up: What countries is the Amazon River in? 657 | Intermediate answer: The Amazon River is in Brazil, Peru, and Colombia. 658 | So the final answer is: Brazil, Peru, and Colombia. 659 | " 660 | 80,"{'question': 'Who was the first king of the longest Chinese dynasty?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the longest Chinese dynasty?\nIntermediate answer: The longest Chinese dynasty is the Qing Dynasty.\nFollow up: Who was the first king of the Qing Dynasty?\nIntermediate answer: The first king of the Qing Dynasty was Nurhaci.\nSo the final answer is: Nurhaci.\n'], 'decompsition_steps': [[{'question': 'What is the longest Chinese dynasty?', 'answer': 'The longest Chinese dynasty is the Qing Dynasty.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the first king of the Qing Dynasty?', 'answer': 'The first king of the Qing Dynasty was Nurhaci.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Nurhaci.'}]], 'traces_entailments': None}",,['Nurhaci'],0,0,0,0,85,Who was the first king of the longest Chinese dynasty?,King Wu of Zhou,," 661 | Are follow up questions needed here: Yes. 662 | Follow up: What is the longest Chinese dynasty? 663 | Intermediate answer: The longest Chinese dynasty is the Qing Dynasty. 664 | Follow up: Who was the first king of the Qing Dynasty? 665 | Intermediate answer: The first king of the Qing Dynasty was Nurhaci. 666 | So the final answer is: Nurhaci. 667 | " 668 | 81,"{'question': 'When was the person who said “Now, I am become Death, the destroyer of worlds.” born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who said “Now, I am become Death, the destroyer of worlds.”?\nIntermediate answer: The person who said “Now, I am become Death, the destroyer of worlds.” was J. Robert Oppenheimer.\nFollow up: When was J. Robert Oppenheimer born?\nIntermediate answer: J. Robert Oppenheimer was born on April 22, 1904.\nSo the final answer is: April 22, 1904.\n'], 'decompsition_steps': [[{'question': 'Who said “Now, I am become Death, the destroyer of worlds.”?', 'answer': 'The person who said “Now, I am become Death, the destroyer of worlds.” was J. Robert Oppenheimer.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was J. Robert Oppenheimer born?', 'answer': 'J. Robert Oppenheimer was born on April 22, 1904.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'April 22, 1904.'}]], 'traces_entailments': None}",,"['April 22, 1904']",1,1,1,1,86,"When was the person who said “Now, I am become Death, the destroyer of worlds.” born?","April 22, 1904",," 669 | Are follow up questions needed here: Yes. 670 | Follow up: Who said “Now, I am become Death, the destroyer of worlds.”? 671 | Intermediate answer: The person who said “Now, I am become Death, the destroyer of worlds.” was J. Robert Oppenheimer. 672 | Follow up: When was J. Robert Oppenheimer born? 673 | Intermediate answer: J. Robert Oppenheimer was born on April 22, 1904. 674 | So the final answer is: April 22, 1904. 675 | " 676 | 82,"{'question': ""What's the capital of the state that the College of William & Mary is in?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: What's the College of William & Mary?\nIntermediate answer: The College of William & Mary is a public research university in Williamsburg, Virginia.\nFollow up: What's the capital of Virginia?\nIntermediate answer: The capital of Virginia is Richmond.\nSo the final answer is: Richmond.\n""], 'decompsition_steps': [[{'question': ""What's the College of William & Mary?"", 'answer': 'The College of William & Mary is a public research university in Williamsburg, Virginia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What's the capital of Virginia?"", 'answer': 'The capital of Virginia is Richmond.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Richmond.'}]], 'traces_entailments': None}",,['Richmond'],1,1,1,1,87,What's the capital of the state that the College of William & Mary is in?,Richmond,," 677 | Are follow up questions needed here: Yes. 678 | Follow up: What's the College of William & Mary? 679 | Intermediate answer: The College of William & Mary is a public research university in Williamsburg, Virginia. 680 | Follow up: What's the capital of Virginia? 681 | Intermediate answer: The capital of Virginia is Richmond. 682 | So the final answer is: Richmond. 683 | " 684 | 83,"{'question': 'Who was the mother of the emperor of Japan during World War I?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the emperor of Japan during World War I?\nIntermediate answer: The emperor of Japan during World War I was Emperor Taisho.\nFollow up: Who was the mother of Emperor Taisho?\nIntermediate answer: The mother of Emperor Taisho was Empress Teimei.\nSo the final answer is: Empress Teimei.\n'], 'decompsition_steps': [[{'question': 'Who was the emperor of Japan during World War I?', 'answer': 'The emperor of Japan during World War I was Emperor Taisho.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the mother of Emperor Taisho?', 'answer': 'The mother of Emperor Taisho was Empress Teimei.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Empress Teimei.'}]], 'traces_entailments': None}",,['Empress Teimei'],0,0,0,0,88,Who was the mother of the emperor of Japan during World War I?,Yanagiwara Naruko,," 685 | Are follow up questions needed here: Yes. 686 | Follow up: Who was the emperor of Japan during World War I? 687 | Intermediate answer: The emperor of Japan during World War I was Emperor Taisho. 688 | Follow up: Who was the mother of Emperor Taisho? 689 | Intermediate answer: The mother of Emperor Taisho was Empress Teimei. 690 | So the final answer is: Empress Teimei. 691 | " 692 | 84,"{'question': 'When was the tallest ferris wheel in the world completed?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the tallest ferris wheel in the world?\nIntermediate answer: The tallest ferris wheel in the world is the High Roller in Las Vegas.\nFollow up: When was the High Roller in Las Vegas completed?\nIntermediate answer: The High Roller in Las Vegas was completed on March 31, 2014.\nSo the final answer is: March 31, 2014.\n'], 'decompsition_steps': [[{'question': 'What is the tallest ferris wheel in the world?', 'answer': 'The tallest ferris wheel in the world is the High Roller in Las Vegas.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the High Roller in Las Vegas completed?', 'answer': 'The High Roller in Las Vegas was completed on March 31, 2014.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'March 31, 2014.'}]], 'traces_entailments': None}",,"['March 31, 2014']",0,0,0,0,89,When was the tallest ferris wheel in the world completed?,2021,," 693 | Are follow up questions needed here: Yes. 694 | Follow up: What is the tallest ferris wheel in the world? 695 | Intermediate answer: The tallest ferris wheel in the world is the High Roller in Las Vegas. 696 | Follow up: When was the High Roller in Las Vegas completed? 697 | Intermediate answer: The High Roller in Las Vegas was completed on March 31, 2014. 698 | So the final answer is: March 31, 2014. 699 | " 700 | 85,"{'question': 'Who was the father of the father of information theory?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the father of information theory?\nIntermediate answer: The father of information theory is Claude Shannon.\nFollow up: Who was the father of Claude Shannon?\nIntermediate answer: The father of Claude Shannon is Claude Shannon Sr.\nSo the final answer is: Claude Shannon Sr.\n'], 'decompsition_steps': [[{'question': 'Who was the father of information theory?', 'answer': 'The father of information theory is Claude Shannon.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of Claude Shannon?', 'answer': 'The father of Claude Shannon is Claude Shannon Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Claude Shannon Sr.'}]], 'traces_entailments': None}",,['Claude Shannon Sr'],0.8,0.8,0.8,0.8,90,Who was the father of the father of information theory?,Claude Sr.,," 701 | Are follow up questions needed here: Yes. 702 | Follow up: Who was the father of information theory? 703 | Intermediate answer: The father of information theory is Claude Shannon. 704 | Follow up: Who was the father of Claude Shannon? 705 | Intermediate answer: The father of Claude Shannon is Claude Shannon Sr. 706 | So the final answer is: Claude Shannon Sr. 707 | " 708 | 86,"{'question': 'What was the death toll of the second largest volcanic eruption in the 20th century?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the second largest volcanic eruption in the 20th century?\nIntermediate answer: The second largest volcanic eruption in the 20th century was the 1912 eruption of Novarupta in Alaska.\nFollow up: What was the death toll of the 1912 eruption of Novarupta?\nIntermediate answer: The death toll of the 1912 eruption of Novarupta was 0.\nSo'], 'decompsition_steps': [[{'question': 'What was the second largest volcanic eruption in the 20th century?', 'answer': 'The second largest volcanic eruption in the 20th century was the 1912 eruption of Novarupta in Alaska.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the death toll of the 1912 eruption of Novarupta?', 'answer': 'The death toll of the 1912 eruption of Novarupta was 0.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,91,What was the death toll of the second largest volcanic eruption in the 20th century?,847,," 709 | Are follow up questions needed here: Yes. 710 | Follow up: What was the second largest volcanic eruption in the 20th century? 711 | Intermediate answer: The second largest volcanic eruption in the 20th century was the 1912 eruption of Novarupta in Alaska. 712 | Follow up: What was the death toll of the 1912 eruption of Novarupta? 713 | Intermediate answer: The death toll of the 1912 eruption of Novarupta was 0. 714 | So" 715 | 87,"{'question': 'When did the author of Annabel Lee enlist in the army?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the author of Annabel Lee?\nIntermediate answer: The author of Annabel Lee was Edgar Allan Poe.\nFollow up: When did Edgar Allan Poe enlist in the army?\nIntermediate answer: Edgar Allan Poe enlisted in the army on May 27, 1827.\nSo the final answer is: May 27, 1827.\n'], 'decompsition_steps': [[{'question': 'Who was the author of Annabel Lee?', 'answer': 'The author of Annabel Lee was Edgar Allan Poe.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Edgar Allan Poe enlist in the army?', 'answer': 'Edgar Allan Poe enlisted in the army on May 27, 1827.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'May 27, 1827.'}]], 'traces_entailments': None}",,"['May 27, 1827']",0.5,0.5,0.5,0.5,92,When did the author of Annabel Lee enlist in the army?,1827,," 716 | Are follow up questions needed here: Yes. 717 | Follow up: Who was the author of Annabel Lee? 718 | Intermediate answer: The author of Annabel Lee was Edgar Allan Poe. 719 | Follow up: When did Edgar Allan Poe enlist in the army? 720 | Intermediate answer: Edgar Allan Poe enlisted in the army on May 27, 1827. 721 | So the final answer is: May 27, 1827. 722 | " 723 | 88,"{'question': 'When did the person who gave the Checkers speech die?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who gave the Checkers speech?\nIntermediate answer: The person who gave the Checkers speech was Richard Nixon.\nFollow up: When did Richard Nixon die?\nIntermediate answer: Richard Nixon died on April 22, 1994.\nSo the final answer is: April 22, 1994.\n'], 'decompsition_steps': [[{'question': 'Who gave the Checkers speech?', 'answer': 'The person who gave the Checkers speech was Richard Nixon.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did Richard Nixon die?', 'answer': 'Richard Nixon died on April 22, 1994.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'April 22, 1994.'}]], 'traces_entailments': None}",,"['April 22, 1994']",1,1,1,1,93,When did the person who gave the Checkers speech die?,"April 22, 1994",," 724 | Are follow up questions needed here: Yes. 725 | Follow up: Who gave the Checkers speech? 726 | Intermediate answer: The person who gave the Checkers speech was Richard Nixon. 727 | Follow up: When did Richard Nixon die? 728 | Intermediate answer: Richard Nixon died on April 22, 1994. 729 | So the final answer is: April 22, 1994. 730 | " 731 | 89,"{'question': 'What rocket was used for the mission that landed the first humans on the moon?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What mission landed the first humans on the moon?\nIntermediate answer: The mission that landed the first humans on the moon was Apollo 11.\nFollow up: What rocket was used for Apollo 11?\nIntermediate answer: The rocket used for Apollo 11 was the Saturn V.\nSo the final answer is: Saturn V.\n'], 'decompsition_steps': [[{'question': 'What mission landed the first humans on the moon?', 'answer': 'The mission that landed the first humans on the moon was Apollo 11.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What rocket was used for Apollo 11?', 'answer': 'The rocket used for Apollo 11 was the Saturn V.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Saturn V.'}]], 'traces_entailments': None}",,['Saturn V'],1,1,1,1,94,What rocket was used for the mission that landed the first humans on the moon?,Saturn V,," 732 | Are follow up questions needed here: Yes. 733 | Follow up: What mission landed the first humans on the moon? 734 | Intermediate answer: The mission that landed the first humans on the moon was Apollo 11. 735 | Follow up: What rocket was used for Apollo 11? 736 | Intermediate answer: The rocket used for Apollo 11 was the Saturn V. 737 | So the final answer is: Saturn V. 738 | " 739 | 90,"{'question': ""What's the motto of the oldest California State university?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What\'s the oldest California State university?\nIntermediate answer: The oldest California State university is San Jose State University.\nFollow up: What\'s the motto of San Jose State University?\nIntermediate answer: The motto of San Jose State University is ""Spartans Stand for Something.""\nSo the final answer is: ""Spartans Stand for Something.""\n'], 'decompsition_steps': [[{'question': ""What's the oldest California State university?"", 'answer': 'The oldest California State university is San Jose State University.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What's the motto of San Jose State University?"", 'answer': 'The motto of San Jose State University is ""Spartans Stand for Something.""', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '""Spartans Stand for Something.""'}]], 'traces_entailments': None}",,"['""Spartans Stand for Something.""']",0,0,0,0,95,What's the motto of the oldest California State university?,Powering Silicon Valley,," 740 | Are follow up questions needed here: Yes. 741 | Follow up: What's the oldest California State university? 742 | Intermediate answer: The oldest California State university is San Jose State University. 743 | Follow up: What's the motto of San Jose State University? 744 | Intermediate answer: The motto of San Jose State University is ""Spartans Stand for Something."" 745 | So the final answer is: ""Spartans Stand for Something."" 746 | " 747 | 91,"{'question': 'Who was the commander for the space mission that had the first spacewalk?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the space mission that had the first spacewalk?\nIntermediate answer: The space mission that had the first spacewalk was Voskhod 2.\nFollow up: Who was the commander for Voskhod 2?\nIntermediate answer: The commander for Voskhod 2 was Pavel Belyayev.\nSo the final answer is: Pavel Belyayev.\n'], 'decompsition_steps': [[{'question': 'What was the space mission that had the first spacewalk?', 'answer': 'The space mission that had the first spacewalk was Voskhod 2.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the commander for Voskhod 2?', 'answer': 'The commander for Voskhod 2 was Pavel Belyayev.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Pavel Belyayev.'}]], 'traces_entailments': None}",,['Pavel Belyayev'],1,1,1,1,96,Who was the commander for the space mission that had the first spacewalk?,Pavel Belyayev,," 748 | Are follow up questions needed here: Yes. 749 | Follow up: What was the space mission that had the first spacewalk? 750 | Intermediate answer: The space mission that had the first spacewalk was Voskhod 2. 751 | Follow up: Who was the commander for Voskhod 2? 752 | Intermediate answer: The commander for Voskhod 2 was Pavel Belyayev. 753 | So the final answer is: Pavel Belyayev. 754 | " 755 | 92,"{'question': 'When did the person who said ""Cogito, ergo sum."" die?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who said ""Cogito, ergo sum.""?\nIntermediate answer: The person who said ""Cogito, ergo sum."" was René Descartes.\nFollow up: When did René Descartes die?\nIntermediate answer: René Descartes died on February 11, 1650.\nSo the final answer is: February 11, 1650.\n'], 'decompsition_steps': [[{'question': 'Who said ""Cogito, ergo sum.""?', 'answer': 'The person who said ""Cogito, ergo sum."" was René Descartes.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did René Descartes die?', 'answer': 'René Descartes died on February 11, 1650.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'February 11, 1650.'}]], 'traces_entailments': None}",,"['February 11, 1650']",1,1,1,1,97,"When did the person who said ""Cogito, ergo sum."" die?","February 11, 1650",," 756 | Are follow up questions needed here: Yes. 757 | Follow up: Who said ""Cogito, ergo sum.""? 758 | Intermediate answer: The person who said ""Cogito, ergo sum."" was René Descartes. 759 | Follow up: When did René Descartes die? 760 | Intermediate answer: René Descartes died on February 11, 1650. 761 | So the final answer is: February 11, 1650. 762 | " 763 | 93,"{'question': 'Who was the father of the father of empiricism?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the father of empiricism?\nIntermediate answer: The father of empiricism was John Locke.\nFollow up: Who was the father of John Locke?\nIntermediate answer: The father of John Locke was John Locke Sr.\nSo the final answer is: John Locke Sr.\n'], 'decompsition_steps': [[{'question': 'Who was the father of empiricism?', 'answer': 'The father of empiricism was John Locke.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of John Locke?', 'answer': 'The father of John Locke was John Locke Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'John Locke Sr.'}]], 'traces_entailments': None}",,['John Locke Sr'],0,0,0,0,98,Who was the father of the father of empiricism?,Sir Nicholas Bacon,," 764 | Are follow up questions needed here: Yes. 765 | Follow up: Who was the father of empiricism? 766 | Intermediate answer: The father of empiricism was John Locke. 767 | Follow up: Who was the father of John Locke? 768 | Intermediate answer: The father of John Locke was John Locke Sr. 769 | So the final answer is: John Locke Sr. 770 | " 771 | 94,"{'question': 'What is the primary male hormone derived from?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the primary male hormone?\nIntermediate answer: The primary male hormone is testosterone.\nFollow up: What is testosterone derived from?\nIntermediate answer: Testosterone is derived from cholesterol.\nSo the final answer is: Cholesterol.\n'], 'decompsition_steps': [[{'question': 'What is the primary male hormone?', 'answer': 'The primary male hormone is testosterone.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is testosterone derived from?', 'answer': 'Testosterone is derived from cholesterol.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Cholesterol.'}]], 'traces_entailments': None}",,['Cholesterol'],0,0,0,0,99,What is the primary male hormone derived from?,gonads,," 772 | Are follow up questions needed here: Yes. 773 | Follow up: What is the primary male hormone? 774 | Intermediate answer: The primary male hormone is testosterone. 775 | Follow up: What is testosterone derived from? 776 | Intermediate answer: Testosterone is derived from cholesterol. 777 | So the final answer is: Cholesterol. 778 | " 779 | 95,"{'question': 'In what year was the tallest fixed steel structure completed?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the tallest fixed steel structure?\nIntermediate answer: The tallest fixed steel structure is the Burj Khalifa.\nFollow up: When was the Burj Khalifa completed?\nIntermediate answer: The Burj Khalifa was completed in 2010.\nSo the final answer is: 2010.\n'], 'decompsition_steps': [[{'question': 'What is the tallest fixed steel structure?', 'answer': 'The tallest fixed steel structure is the Burj Khalifa.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Burj Khalifa completed?', 'answer': 'The Burj Khalifa was completed in 2010.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '2010.'}]], 'traces_entailments': None}",,['2010'],0,0,0,0,100,In what year was the tallest fixed steel structure completed?,1988,," 780 | Are follow up questions needed here: Yes. 781 | Follow up: What is the tallest fixed steel structure? 782 | Intermediate answer: The tallest fixed steel structure is the Burj Khalifa. 783 | Follow up: When was the Burj Khalifa completed? 784 | Intermediate answer: The Burj Khalifa was completed in 2010. 785 | So the final answer is: 2010. 786 | " 787 | 96,"{'question': 'In what year was the tallest self-supporting tower completed?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the tallest self-supporting tower?\nIntermediate answer: The tallest self-supporting tower is the Burj Khalifa.\nFollow up: When was the Burj Khalifa completed?\nIntermediate answer: The Burj Khalifa was completed in 2010.\nSo the final answer is: 2010.\n'], 'decompsition_steps': [[{'question': 'What is the tallest self-supporting tower?', 'answer': 'The tallest self-supporting tower is the Burj Khalifa.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was the Burj Khalifa completed?', 'answer': 'The Burj Khalifa was completed in 2010.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '2010.'}]], 'traces_entailments': None}",,['2010'],0,0,0,0,101,In what year was the tallest self-supporting tower completed?,2012,," 788 | Are follow up questions needed here: Yes. 789 | Follow up: What is the tallest self-supporting tower? 790 | Intermediate answer: The tallest self-supporting tower is the Burj Khalifa. 791 | Follow up: When was the Burj Khalifa completed? 792 | Intermediate answer: The Burj Khalifa was completed in 2010. 793 | So the final answer is: 2010. 794 | " 795 | 97,"{'question': 'When was the person who shared the Nobel Prize in Physics in 1954 with Max Born born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who shared the Nobel Prize in Physics in 1954 with Max Born?\nIntermediate answer: The person who shared the Nobel Prize in Physics in 1954 with Max Born is Walther Bothe.\nFollow up: When was Walther Bothe born?\nIntermediate answer: Walther Bothe was born on January 8, 1891.\nSo the final answer is: January 8, 1891.\n'], 'decompsition_steps': [[{'question': 'Who shared the Nobel Prize in Physics in 1954 with Max Born?', 'answer': 'The person who shared the Nobel Prize in Physics in 1954 with Max Born is Walther Bothe.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When was Walther Bothe born?', 'answer': 'Walther Bothe was born on January 8, 1891.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'January 8, 1891.'}]], 'traces_entailments': None}",,"['January 8, 1891']",1,1,1,1,102,When was the person who shared the Nobel Prize in Physics in 1954 with Max Born born?,"January 8, 1891",," 796 | Are follow up questions needed here: Yes. 797 | Follow up: Who shared the Nobel Prize in Physics in 1954 with Max Born? 798 | Intermediate answer: The person who shared the Nobel Prize in Physics in 1954 with Max Born is Walther Bothe. 799 | Follow up: When was Walther Bothe born? 800 | Intermediate answer: Walther Bothe was born on January 8, 1891. 801 | So the final answer is: January 8, 1891. 802 | " 803 | 98,"{'question': 'Who is the father of the father of modern Hebrew?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of modern Hebrew?\nIntermediate answer: The father of modern Hebrew is Eliezer Ben-Yehuda.\nFollow up: Who is the father of Eliezer Ben-Yehuda?\nIntermediate answer: The father of Eliezer Ben-Yehuda is Shlomo Zalman Ben-Yehuda.\nSo the final answer is: Shlomo Zalman Ben-Yehuda.\n'], 'decompsition_steps': [[{'question': 'Who is the father of modern Hebrew?', 'answer': 'The father of modern Hebrew is Eliezer Ben-Yehuda.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the father of Eliezer Ben-Yehuda?', 'answer': 'The father of Eliezer Ben-Yehuda is Shlomo Zalman Ben-Yehuda.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Shlomo Zalman Ben-Yehuda.'}]], 'traces_entailments': None}",,['Shlomo Zalman Ben-Yehuda'],0,0,0,0,103,Who is the father of the father of modern Hebrew?,Yehuda Leib,," 804 | Are follow up questions needed here: Yes. 805 | Follow up: Who is the father of modern Hebrew? 806 | Intermediate answer: The father of modern Hebrew is Eliezer Ben-Yehuda. 807 | Follow up: Who is the father of Eliezer Ben-Yehuda? 808 | Intermediate answer: The father of Eliezer Ben-Yehuda is Shlomo Zalman Ben-Yehuda. 809 | So the final answer is: Shlomo Zalman Ben-Yehuda. 810 | " 811 | 99,"{'question': 'What is the lowest elevation of the longest railway tunnel?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the longest railway tunnel?\nIntermediate answer: The longest railway tunnel is the Gotthard Base Tunnel.\nFollow up: What is the lowest elevation of the Gotthard Base Tunnel?\nIntermediate answer: The lowest elevation of the Gotthard Base Tunnel is 2,300 meters.\nSo the final answer is: 2,300 meters.\n'], 'decompsition_steps': [[{'question': 'What is the longest railway tunnel?', 'answer': 'The longest railway tunnel is the Gotthard Base Tunnel.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the lowest elevation of the Gotthard Base Tunnel?', 'answer': 'The lowest elevation of the Gotthard Base Tunnel is 2,300 meters.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '2,300 meters.'}]], 'traces_entailments': None}",,"['2,300 meters']",0,0,0,0,104,What is the lowest elevation of the longest railway tunnel?,312 m,," 812 | Are follow up questions needed here: Yes. 813 | Follow up: What is the longest railway tunnel? 814 | Intermediate answer: The longest railway tunnel is the Gotthard Base Tunnel. 815 | Follow up: What is the lowest elevation of the Gotthard Base Tunnel? 816 | Intermediate answer: The lowest elevation of the Gotthard Base Tunnel is 2,300 meters. 817 | So the final answer is: 2,300 meters. 818 | " 819 | 100,"{'question': 'Who was president of the U.S. when superconductivity was discovered?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: When was superconductivity discovered?\nIntermediate answer: Superconductivity was discovered in 1911.\nFollow up: Who was president of the U.S. in 1911?\nIntermediate answer: The president of the U.S. in 1911 was William Howard Taft.\nSo the final answer is: William Howard Taft.\n'], 'decompsition_steps': [[{'question': 'When was superconductivity discovered?', 'answer': 'Superconductivity was discovered in 1911.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was president of the U.S. in 1911?', 'answer': 'The president of the U.S. in 1911 was William Howard Taft.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'William Howard Taft.'}]], 'traces_entailments': None}",,['William Howard Taft'],1,1,1,1,105,Who was president of the U.S. when superconductivity was discovered?,William Howard Taft,," 820 | Are follow up questions needed here: Yes. 821 | Follow up: When was superconductivity discovered? 822 | Intermediate answer: Superconductivity was discovered in 1911. 823 | Follow up: Who was president of the U.S. in 1911? 824 | Intermediate answer: The president of the U.S. in 1911 was William Howard Taft. 825 | So the final answer is: William Howard Taft. 826 | " 827 | 101,"{'question': 'Who was the father of the founder of modern human anatomy?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the founder of modern human anatomy?\nIntermediate answer: The founder of modern human anatomy was Andreas Vesalius.\nFollow up: Who was the father of Andreas Vesalius?\nIntermediate answer: The father of Andreas Vesalius was Andreas Vesalius.\nSo the final answer is: Andreas Vesalius.\n'], 'decompsition_steps': [[{'question': 'Who was the founder of modern human anatomy?', 'answer': 'The founder of modern human anatomy was Andreas Vesalius.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of Andreas Vesalius?', 'answer': 'The father of Andreas Vesalius was Andreas Vesalius.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Andreas Vesalius.'}]], 'traces_entailments': None}",,['Andreas Vesalius'],0,0,0,0,106,Who was the father of the founder of modern human anatomy?,Anders van Wesel,," 828 | Are follow up questions needed here: Yes. 829 | Follow up: Who was the founder of modern human anatomy? 830 | Intermediate answer: The founder of modern human anatomy was Andreas Vesalius. 831 | Follow up: Who was the father of Andreas Vesalius? 832 | Intermediate answer: The father of Andreas Vesalius was Andreas Vesalius. 833 | So the final answer is: Andreas Vesalius. 834 | " 835 | 102,"{'question': 'What was the final book written by the author of On the Origin of Species?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the author of On the Origin of Species?\nIntermediate answer: The author of On the Origin of Species was Charles Darwin.\nFollow up: What was the final book written by Charles Darwin?\nIntermediate answer: The final book written by Charles Darwin was The Descent of Man.\nSo the final answer is: The Descent of Man.\n'], 'decompsition_steps': [[{'question': 'Who was the author of On the Origin of Species?', 'answer': 'The author of On the Origin of Species was Charles Darwin.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the final book written by Charles Darwin?', 'answer': 'The final book written by Charles Darwin was The Descent of Man.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'The Descent of Man.'}]], 'traces_entailments': None}",,['The Descent of Man'],0.1818181818,0.1818181818,0.1818181818,0.1818181818,107,What was the final book written by the author of On the Origin of Species?,The Formation of Vegetable Mould Through the Action of Worms,," 836 | Are follow up questions needed here: Yes. 837 | Follow up: Who was the author of On the Origin of Species? 838 | Intermediate answer: The author of On the Origin of Species was Charles Darwin. 839 | Follow up: What was the final book written by Charles Darwin? 840 | Intermediate answer: The final book written by Charles Darwin was The Descent of Man. 841 | So the final answer is: The Descent of Man. 842 | " 843 | 103,"{'question': 'In what country was the only cruise line that flies the American flag incorporated in?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the name of the only cruise line that flies the American flag?\nIntermediate answer: The only cruise line that flies the American flag is the Carnival Cruise Line.\nFollow up: In what country was the Carnival Cruise Line incorporated?\nIntermediate answer: The Carnival Cruise Line was incorporated in Panama.\nSo the final answer is: Panama.\n'], 'decompsition_steps': [[{'question': 'What is the name of the only cruise line that flies the American flag?', 'answer': 'The only cruise line that flies the American flag is the Carnival Cruise Line.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'In what country was the Carnival Cruise Line incorporated?', 'answer': 'The Carnival Cruise Line was incorporated in Panama.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Panama.'}]], 'traces_entailments': None}",,['Panama'],0,0,0,0,108,In what country was the only cruise line that flies the American flag incorporated in?,Bermuda,," 844 | Are follow up questions needed here: Yes. 845 | Follow up: What is the name of the only cruise line that flies the American flag? 846 | Intermediate answer: The only cruise line that flies the American flag is the Carnival Cruise Line. 847 | Follow up: In what country was the Carnival Cruise Line incorporated? 848 | Intermediate answer: The Carnival Cruise Line was incorporated in Panama. 849 | So the final answer is: Panama. 850 | " 851 | 104,"{'question': 'Who preceded the Roman emperor that declared war on the sea?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: Who was the Roman emperor that declared war on the sea?\nIntermediate answer: The Roman emperor that declared war on the sea was Caligula.\nFollow up: Who preceded Caligula?\nIntermediate answer: Caligula's predecessor was Tiberius.\nSo the final answer is: Tiberius.\n""], 'decompsition_steps': [[{'question': 'Who was the Roman emperor that declared war on the sea?', 'answer': 'The Roman emperor that declared war on the sea was Caligula.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who preceded Caligula?', 'answer': ""Caligula's predecessor was Tiberius."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Tiberius.'}]], 'traces_entailments': None}",,['Tiberius'],1,1,1,1,109,Who preceded the Roman emperor that declared war on the sea?,Tiberius,," 852 | Are follow up questions needed here: Yes. 853 | Follow up: Who was the Roman emperor that declared war on the sea? 854 | Intermediate answer: The Roman emperor that declared war on the sea was Caligula. 855 | Follow up: Who preceded Caligula? 856 | Intermediate answer: Caligula's predecessor was Tiberius. 857 | So the final answer is: Tiberius. 858 | " 859 | 105,"{'question': 'Which Theranos whistleblower is not related to a senior American government official?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Which Theranos whistleblower is related to a senior American government official?\nIntermediate answer: Tyler Shultz is related to a senior American government official.\nFollow up: Which Theranos whistleblower is not related to a senior American government official?\nIntermediate answer: Erika Cheung is not related to a senior American government official.\nSo the final answer is: Erika Cheung.\n'], 'decompsition_steps': [[{'question': 'Which Theranos whistleblower is related to a senior American government official?', 'answer': 'Tyler Shultz is related to a senior American government official.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Which Theranos whistleblower is not related to a senior American government official?', 'answer': 'Erika Cheung is not related to a senior American government official.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Erika Cheung.'}]], 'traces_entailments': None}",,['Erika Cheung'],1,1,1,1,110,Which Theranos whistleblower is not related to a senior American government official?,Erika Cheung,," 860 | Are follow up questions needed here: Yes. 861 | Follow up: Which Theranos whistleblower is related to a senior American government official? 862 | Intermediate answer: Tyler Shultz is related to a senior American government official. 863 | Follow up: Which Theranos whistleblower is not related to a senior American government official? 864 | Intermediate answer: Erika Cheung is not related to a senior American government official. 865 | So the final answer is: Erika Cheung. 866 | " 867 | 106,"{'question': 'Who founded the city where the founder of geometry lived?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the founder of geometry?\nIntermediate answer: The founder of geometry was Euclid.\nFollow up: Who founded the city where Euclid lived?\nIntermediate answer: The city where Euclid lived was Alexandria.\nFollow up: Who founded Alexandria?\nIntermediate answer: Alexandria was founded by Alexander the Great.\nSo the final answer is: Alexander the Great.\n'], 'decompsition_steps': [[{'question': 'Who was the founder of geometry?', 'answer': 'The founder of geometry was Euclid.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who founded the city where Euclid lived?', 'answer': 'The city where Euclid lived was Alexandria.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who founded Alexandria?', 'answer': 'Alexandria was founded by Alexander the Great.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Alexander the Great.'}]], 'traces_entailments': None}",,['Alexander the Great'],1,1,1,1,111,Who founded the city where the founder of geometry lived?,Alexander the Great,," 868 | Are follow up questions needed here: Yes. 869 | Follow up: Who was the founder of geometry? 870 | Intermediate answer: The founder of geometry was Euclid. 871 | Follow up: Who founded the city where Euclid lived? 872 | Intermediate answer: The city where Euclid lived was Alexandria. 873 | Follow up: Who founded Alexandria? 874 | Intermediate answer: Alexandria was founded by Alexander the Great. 875 | So the final answer is: Alexander the Great. 876 | " 877 | 107,"{'question': 'Can people who have celiac eat camel meat?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is celiac?\nIntermediate answer: Celiac is a disease that causes people to be allergic to gluten.\nFollow up: What is gluten?\nIntermediate answer: Gluten is a protein found in wheat, barley, and rye.\nFollow up: What is in camel meat?\nIntermediate answer: Camel meat contains no gluten.\nSo the final answer is: Yes.\n'], 'decompsition_steps': [[{'question': 'What is celiac?', 'answer': 'Celiac is a disease that causes people to be allergic to gluten.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is gluten?', 'answer': 'Gluten is a protein found in wheat, barley, and rye.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is in camel meat?', 'answer': 'Camel meat contains no gluten.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Yes.'}]], 'traces_entailments': None}",,['Yes'],1,1,1,1,112,Can people who have celiac eat camel meat?,Yes,," 878 | Are follow up questions needed here: Yes. 879 | Follow up: What is celiac? 880 | Intermediate answer: Celiac is a disease that causes people to be allergic to gluten. 881 | Follow up: What is gluten? 882 | Intermediate answer: Gluten is a protein found in wheat, barley, and rye. 883 | Follow up: What is in camel meat? 884 | Intermediate answer: Camel meat contains no gluten. 885 | So the final answer is: Yes. 886 | " 887 | 108,"{'question': 'Which company was responsible for the largest pharmaceutical settlement?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the largest pharmaceutical settlement?\nIntermediate answer: The largest pharmaceutical settlement was $2.3 billion.\nFollow up: Which company was responsible for the largest pharmaceutical settlement?\nIntermediate answer: The company responsible for the largest pharmaceutical settlement was GlaxoSmithKline.\nSo the final answer is: GlaxoSmithKline.\n'], 'decompsition_steps': [[{'question': 'What was the largest pharmaceutical settlement?', 'answer': 'The largest pharmaceutical settlement was $2.3 billion.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Which company was responsible for the largest pharmaceutical settlement?', 'answer': 'The company responsible for the largest pharmaceutical settlement was GlaxoSmithKline.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'GlaxoSmithKline.'}]], 'traces_entailments': None}",,['GlaxoSmithKline'],1,1,1,1,113,Which company was responsible for the largest pharmaceutical settlement?,GlaxoSmithKline,," 888 | Are follow up questions needed here: Yes. 889 | Follow up: What was the largest pharmaceutical settlement? 890 | Intermediate answer: The largest pharmaceutical settlement was $2.3 billion. 891 | Follow up: Which company was responsible for the largest pharmaceutical settlement? 892 | Intermediate answer: The company responsible for the largest pharmaceutical settlement was GlaxoSmithKline. 893 | So the final answer is: GlaxoSmithKline. 894 | " 895 | 109,"{'question': 'What is the length of the second deepest river in the world?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the second deepest river in the world?\nIntermediate answer: The second deepest river in the world is the Congo River.\nFollow up: What is the length of the Congo River?\nIntermediate answer: The length of the Congo River is 2,920 miles.\nSo the final answer is: 2,920 miles.\n'], 'decompsition_steps': [[{'question': 'What is the second deepest river in the world?', 'answer': 'The second deepest river in the world is the Congo River.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the length of the Congo River?', 'answer': 'The length of the Congo River is 2,920 miles.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': '2,920 miles.'}]], 'traces_entailments': None}",,"['2,920 miles']",0,0,0,0,114,What is the length of the second deepest river in the world?,"6,300 km",," 896 | Are follow up questions needed here: Yes. 897 | Follow up: What is the second deepest river in the world? 898 | Intermediate answer: The second deepest river in the world is the Congo River. 899 | Follow up: What is the length of the Congo River? 900 | Intermediate answer: The length of the Congo River is 2,920 miles. 901 | So the final answer is: 2,920 miles. 902 | " 903 | 110,"{'question': 'What was the job of the father of the founder of psychoanalysis?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: Who was the founder of psychoanalysis?\nIntermediate answer: The founder of psychoanalysis was Sigmund Freud.\nFollow up: What was the job of Sigmund Freud's father?\nIntermediate answer: Sigmund Freud's father was a wool merchant.\nSo the final answer is: Wool merchant.\n""], 'decompsition_steps': [[{'question': 'Who was the founder of psychoanalysis?', 'answer': 'The founder of psychoanalysis was Sigmund Freud.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': ""What was the job of Sigmund Freud's father?"", 'answer': ""Sigmund Freud's father was a wool merchant."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Wool merchant.'}]], 'traces_entailments': None}",,['Wool merchant'],1,1,1,1,115,What was the job of the father of the founder of psychoanalysis?,wool merchant,," 904 | Are follow up questions needed here: Yes. 905 | Follow up: Who was the founder of psychoanalysis? 906 | Intermediate answer: The founder of psychoanalysis was Sigmund Freud. 907 | Follow up: What was the job of Sigmund Freud's father? 908 | Intermediate answer: Sigmund Freud's father was a wool merchant. 909 | So the final answer is: Wool merchant. 910 | " 911 | 111,"{'question': 'What is the nickname of the easternmost U.S. state?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the easternmost U.S. state?\nIntermediate answer: The easternmost U.S. state is Maine.\nFollow up: What is the nickname of Maine?\nIntermediate answer: The nickname of Maine is The Pine Tree State.\nSo the final answer is: The Pine Tree State.\n'], 'decompsition_steps': [[{'question': 'What is the easternmost U.S. state?', 'answer': 'The easternmost U.S. state is Maine.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What is the nickname of Maine?', 'answer': 'The nickname of Maine is The Pine Tree State.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'The Pine Tree State.'}]], 'traces_entailments': None}",,['The Pine Tree State'],1,1,1,1,116,What is the nickname of the easternmost U.S. state?,Pine Tree State,," 912 | Are follow up questions needed here: Yes. 913 | Follow up: What is the easternmost U.S. state? 914 | Intermediate answer: The easternmost U.S. state is Maine. 915 | Follow up: What is the nickname of Maine? 916 | Intermediate answer: The nickname of Maine is The Pine Tree State. 917 | So the final answer is: The Pine Tree State. 918 | " 919 | 112,"{'question': 'Who built the fastest air-breathing manned aircraft?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the fastest air-breathing manned aircraft?\nIntermediate answer: The fastest air-breathing manned aircraft is the SR-71 Blackbird.\nFollow up: Who built the SR-71 Blackbird?\nIntermediate answer: The SR-71 Blackbird was built by Lockheed.\nSo the final answer is: Lockheed.\n'], 'decompsition_steps': [[{'question': 'What is the fastest air-breathing manned aircraft?', 'answer': 'The fastest air-breathing manned aircraft is the SR-71 Blackbird.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who built the SR-71 Blackbird?', 'answer': 'The SR-71 Blackbird was built by Lockheed.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Lockheed.'}]], 'traces_entailments': None}",,['Lockheed'],0.6666666667,0.6666666667,0.6666666667,0.6666666667,117,Who built the fastest air-breathing manned aircraft?,Lockheed Corporation,," 920 | Are follow up questions needed here: Yes. 921 | Follow up: What is the fastest air-breathing manned aircraft? 922 | Intermediate answer: The fastest air-breathing manned aircraft is the SR-71 Blackbird. 923 | Follow up: Who built the SR-71 Blackbird? 924 | Intermediate answer: The SR-71 Blackbird was built by Lockheed. 925 | So the final answer is: Lockheed. 926 | " 927 | 113,"{'question': 'Who was mayor of New York City when Fiorello H. La Guardia was born?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: When was Fiorello H. La Guardia born?\nIntermediate answer: Fiorello H. La Guardia was born on December 11, 1882.\nFollow up: Who was mayor of New York City in 1882?\nIntermediate answer: The mayor of New York City in 1882 was William Russell Grace.\nSo the final answer is: William Russell Grace.\n'], 'decompsition_steps': [[{'question': 'When was Fiorello H. La Guardia born?', 'answer': 'Fiorello H. La Guardia was born on December 11, 1882.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was mayor of New York City in 1882?', 'answer': 'The mayor of New York City in 1882 was William Russell Grace.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'William Russell Grace.'}]], 'traces_entailments': None}",,['William Russell Grace'],1,1,1,1,118,Who was mayor of New York City when Fiorello H. La Guardia was born?,William Russell Grace,," 928 | Are follow up questions needed here: Yes. 929 | Follow up: When was Fiorello H. La Guardia born? 930 | Intermediate answer: Fiorello H. La Guardia was born on December 11, 1882. 931 | Follow up: Who was mayor of New York City in 1882? 932 | Intermediate answer: The mayor of New York City in 1882 was William Russell Grace. 933 | So the final answer is: William Russell Grace. 934 | " 935 | 114,"{'question': 'In what country is the second tallest statue in the world?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What is the second tallest statue in the world?\nIntermediate answer: The second tallest statue in the world is the Statue of Unity.\nFollow up: In what country is the Statue of Unity?\nIntermediate answer: The Statue of Unity is in India.\nFollow up: In what country is India?\nIntermediate answer: India is in Asia.\nSo the final answer is: Asia.\n'], 'decompsition_steps': [[{'question': 'What is the second tallest statue in the world?', 'answer': 'The second tallest statue in the world is the Statue of Unity.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'In what country is the Statue of Unity?', 'answer': 'The Statue of Unity is in India.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'In what country is India?', 'answer': 'India is in Asia.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Asia.'}]], 'traces_entailments': None}",,['Asia'],0,0,0,0,119,In what country is the second tallest statue in the world?,China,," 936 | Are follow up questions needed here: Yes. 937 | Follow up: What is the second tallest statue in the world? 938 | Intermediate answer: The second tallest statue in the world is the Statue of Unity. 939 | Follow up: In what country is the Statue of Unity? 940 | Intermediate answer: The Statue of Unity is in India. 941 | Follow up: In what country is India? 942 | Intermediate answer: India is in Asia. 943 | So the final answer is: Asia. 944 | " 945 | 115,"{'question': 'What was the religion of the inventor of the Polio vaccine?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the inventor of the Polio vaccine?\nIntermediate answer: The inventor of the Polio vaccine was Jonas Salk.\nFollow up: What was the religion of Jonas Salk?\nIntermediate answer: Jonas Salk was Jewish.\nSo the final answer is: Jewish.\n'], 'decompsition_steps': [[{'question': 'Who was the inventor of the Polio vaccine?', 'answer': 'The inventor of the Polio vaccine was Jonas Salk.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What was the religion of Jonas Salk?', 'answer': 'Jonas Salk was Jewish.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Jewish.'}]], 'traces_entailments': None}",,['Jewish'],1,1,1,1,120,What was the religion of the inventor of the Polio vaccine?,Jewish,," 946 | Are follow up questions needed here: Yes. 947 | Follow up: Who was the inventor of the Polio vaccine? 948 | Intermediate answer: The inventor of the Polio vaccine was Jonas Salk. 949 | Follow up: What was the religion of Jonas Salk? 950 | Intermediate answer: Jonas Salk was Jewish. 951 | So the final answer is: Jewish. 952 | " 953 | 116,"{'question': 'Who is the mother of the father of Barack Obama?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who is the father of Barack Obama?\nIntermediate answer: The father of Barack Obama is Barack Obama Sr.\nFollow up: Who is the mother of Barack Obama Sr.?\nIntermediate answer: The mother of Barack Obama Sr. is Kezia Obama.\nSo the final answer is: Kezia Obama.\n'], 'decompsition_steps': [[{'question': 'Who is the father of Barack Obama?', 'answer': 'The father of Barack Obama is Barack Obama Sr.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who is the mother of Barack Obama Sr.?', 'answer': 'The mother of Barack Obama Sr. is Kezia Obama.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Kezia Obama.'}]], 'traces_entailments': None}",,['Kezia Obama'],0,0,0,0,121,Who is the mother of the father of Barack Obama?,Habiba Akumu Nyanjango,," 954 | Are follow up questions needed here: Yes. 955 | Follow up: Who is the father of Barack Obama? 956 | Intermediate answer: The father of Barack Obama is Barack Obama Sr. 957 | Follow up: Who is the mother of Barack Obama Sr.? 958 | Intermediate answer: The mother of Barack Obama Sr. is Kezia Obama. 959 | So the final answer is: Kezia Obama. 960 | " 961 | 117,"{'question': 'Who was the father of the father of computer science?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: Who was the father of computer science?\nIntermediate answer: The father of computer science is Alan Turing.\nFollow up: Who was the father of Alan Turing?\nIntermediate answer: The father of Alan Turing is Julius Turing.\nFollow up: Who was the father of Julius Turing?\nIntermediate answer: The father of Julius Turing is Julius Turing.\nFollow up: Who was the father of Julius Turing?\nIntermediate answer: The father of Julius Turing is Julius Turing.'], 'decompsition_steps': [[{'question': 'Who was the father of computer science?', 'answer': 'The father of computer science is Alan Turing.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of Alan Turing?', 'answer': 'The father of Alan Turing is Julius Turing.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of Julius Turing?', 'answer': 'The father of Julius Turing is Julius Turing.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'Who was the father of Julius Turing?', 'answer': 'The father of Julius Turing is Julius Turing.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}]], 'traces_entailments': None}",,[],FALSE,FALSE,FALSE,FALSE,122,Who was the father of the father of computer science?,Julius Mathison Turing,," 962 | Are follow up questions needed here: Yes. 963 | Follow up: Who was the father of computer science? 964 | Intermediate answer: The father of computer science is Alan Turing. 965 | Follow up: Who was the father of Alan Turing? 966 | Intermediate answer: The father of Alan Turing is Julius Turing. 967 | Follow up: Who was the father of Julius Turing? 968 | Intermediate answer: The father of Julius Turing is Julius Turing. 969 | Follow up: Who was the father of Julius Turing? 970 | Intermediate answer: The father of Julius Turing is Julius Turing." 971 | 118,"{'question': 'What rocket was the first spacecraft that ever approached Uranus launched on?', 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': ['\nAre follow up questions needed here: Yes.\nFollow up: What was the first spacecraft that ever approached Uranus?\nIntermediate answer: The first spacecraft that ever approached Uranus was Voyager 2.\nFollow up: What rocket was Voyager 2 launched on?\nIntermediate answer: Voyager 2 was launched on a Titan-Centaur rocket.\nSo the final answer is: Titan-Centaur.\n'], 'decompsition_steps': [[{'question': 'What was the first spacecraft that ever approached Uranus?', 'answer': 'The first spacecraft that ever approached Uranus was Voyager 2.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'What rocket was Voyager 2 launched on?', 'answer': 'Voyager 2 was launched on a Titan-Centaur rocket.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'Titan-Centaur.'}]], 'traces_entailments': None}",,['Titan-Centaur'],0,0,0,0,123,What rocket was the first spacecraft that ever approached Uranus launched on?,Titan IIIE,," 972 | Are follow up questions needed here: Yes. 973 | Follow up: What was the first spacecraft that ever approached Uranus? 974 | Intermediate answer: The first spacecraft that ever approached Uranus was Voyager 2. 975 | Follow up: What rocket was Voyager 2 launched on? 976 | Intermediate answer: Voyager 2 was launched on a Titan-Centaur rocket. 977 | So the final answer is: Titan-Centaur. 978 | " 979 | 119,"{'question': ""When did the last king from Britain's House of Hanover die?"", 'statement': {'original_statement': '', 'positive': None, 'negative': None}, 'decompositions': [""\nAre follow up questions needed here: Yes.\nFollow up: Who was the last king from Britain's House of Hanover?\nIntermediate answer: The last king from Britain's House of Hanover was George III.\nFollow up: When did George III die?\nIntermediate answer: George III died on January 29, 1820.\nSo the final answer is: January 29, 1820.\n""], 'decompsition_steps': [[{'question': ""Who was the last king from Britain's House of Hanover?"", 'answer': ""The last king from Britain's House of Hanover was George III."", 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': 'When did George III die?', 'answer': 'George III died on January 29, 1820.', 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': None}, {'question': None, 'answer': None, 'google_answer': None, 'google_answer_long': None, 'entailment_result_with_input': None, 'gpt_3_ans': 'January 29, 1820.'}]], 'traces_entailments': None}",,"['January 29, 1820']",0.3333333333,0.3333333333,0.3333333333,0.3333333333,124,When did the last king from Britain's House of Hanover die?,"January 22, 1901",," 980 | Are follow up questions needed here: Yes. 981 | Follow up: Who was the last king from Britain's House of Hanover? 982 | Intermediate answer: The last king from Britain's House of Hanover was George III. 983 | Follow up: When did George III die? 984 | Intermediate answer: George III died on January 29, 1820. 985 | So the final answer is: January 29, 1820. 986 | " -------------------------------------------------------------------------------- /nli/src/scripts/run_nli.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import ast 3 | import pandas as pd 4 | import argparse 5 | from tqdm import tqdm 6 | from nli.utils import populate_question_with_entailment 7 | 8 | 9 | def parse_args(): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument( 12 | "--no_retrieval_csv", 13 | type=str, 14 | default="data/nli_example/bamboogle_no_retrieval.csv", 15 | ) 16 | parser.add_argument( 17 | "--with_retrieval_csv", 18 | type=str, 19 | default="data/nli_example/bamboogle_with_retrieval.csv", 20 | ) 21 | parser.add_argument( 22 | "--threshold", 23 | type=float, 24 | default=0.5, 25 | ) 26 | return parser.parse_args() 27 | 28 | 29 | def run_nli(args): 30 | """run nli model to filter irrelevant retrieved context""" 31 | 32 | # parse args to vars 33 | t, no_retrieval_csv, with_retrieval_csv = ( 34 | args.threshold, 35 | args.no_retrieval_csv, 36 | args.with_retrieval_csv, 37 | ) 38 | 39 | # read data 40 | no_retrieval = pd.read_csv(no_retrieval_csv).to_dict("rows") 41 | with_retrieval = pd.read_csv(with_retrieval_csv).to_dict("rows") 42 | 43 | # preprocess files 44 | for d in [no_retrieval, with_retrieval]: 45 | for x in d: 46 | if x["acc@1"] is not None: 47 | x["acc@1"] = float(x["acc@1"]) if x["acc@1"] != "FALSE" else 0 48 | x["question"] = ast.literal_eval(x["question"]) 49 | x["gpt_answers"] = ast.literal_eval(x["gpt_answers"]) 50 | no_retrieval_questions = {x["question"]["question"] for x in no_retrieval} 51 | with_retrieval = [ 52 | x for x in with_retrieval if x["question"]["question"] in no_retrieval_questions 53 | ] 54 | no_retrieval.sort(key=lambda x: x["question"]["question"]) 55 | with_retrieval.sort(key=lambda x: x["question"]["question"]) 56 | 57 | # populate question 58 | for _, q in tqdm(enumerate(with_retrieval)): 59 | populate_question_with_entailment(q) 60 | 61 | # calculate res and print results 62 | res = np.average( 63 | [ 64 | no_retrieval[i]["acc@1"] 65 | if "nli_true_prob" not in x 66 | or float(min([x["nli_true_prob"]] + x["sub_questions_nli_true_prob"])) <= t 67 | else x["acc@1"] 68 | for i, x in enumerate(with_retrieval) 69 | ] 70 | ) 71 | res_no_nli = np.average([x["acc@1"] for x in with_retrieval]) 72 | res_no_retrieval = np.average([x["acc@1"] for x in no_retrieval]) 73 | print( 74 | f"Threshold: {t}, Res: {res*100:.1f}, Res no retrieval: {res_no_retrieval*100:.1f}, Res no NLI: {res_no_nli*100:.1f}" 75 | ) 76 | 77 | 78 | if __name__ == "__main__": 79 | """ """ 80 | args = parse_args() 81 | run_nli(args) 82 | -------------------------------------------------------------------------------- /nli/src/utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from transformers import AutoModelForSequenceClassification, AutoTokenizer 3 | 4 | 5 | # nli model 6 | device = "cuda" 7 | nli_model = AutoModelForSequenceClassification.from_pretrained( 8 | "facebook/bart-large-mnli" 9 | ) 10 | tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli") 11 | nli_model.to(device) 12 | nli_model.eval() 13 | 14 | 15 | def populate_question_with_entailment(x, batch_size=8): 16 | """ 17 | run the nli model and fill the nli_true_prob and sub_questions_nli_true_prob fields 18 | """ 19 | premise = x["question"]["decompositions"][0].split("Question:")[0] 20 | if len(x["gpt_answers"]): 21 | hypothesis = ( 22 | ( 23 | x["question"]["question"].strip() 24 | if x["question"]["question"].strip()[-1] == "?" 25 | else x["question"]["question"].strip() + "?" 26 | ) 27 | + " The answer is: " 28 | + x["gpt_answers"][0] 29 | ) 30 | 31 | with torch.no_grad(): # Disable autograd to save memory 32 | # run through model pre-trained on MNLI 33 | toks = tokenizer.encode( 34 | premise, 35 | hypothesis, 36 | return_tensors="pt", 37 | truncation_strategy="only_first", 38 | ).to(device) 39 | logits = nli_model(toks)[0] 40 | 41 | # we throw away "neutral" (dim 1) and take the probability of 42 | # "entailment" (2) as the probability of the label being true 43 | entail_contradiction_logits = logits[:, [0, 2]] 44 | probs = entail_contradiction_logits.softmax(dim=1) 45 | prob_label_is_true = probs[:, 1] 46 | x["nli_true_prob"] = float(prob_label_is_true) 47 | x["nli_probs"] = [float(x) for x in probs[0]] 48 | x["sub_questions_nli_true_prob"] = [] 49 | 50 | torch.cuda.empty_cache() # Clear GPU memory after processing 51 | 52 | # Batch processing for sub_questions 53 | sub_questions = [ 54 | sub_question 55 | for sub_question in x["question"]["decompsition_steps"][0] 56 | if sub_question["question"] is not None 57 | ] 58 | for i in range(0, len(sub_questions), batch_size): 59 | batch = sub_questions[i : i + batch_size] 60 | hypotheses = [ 61 | sub_question["question"] + " " + sub_question["answer"] 62 | for sub_question in batch 63 | ] 64 | 65 | with torch.no_grad(): # Disable autograd to save memory 66 | # run through model pre-trained on MNLI 67 | batch_encoded = tokenizer.batch_encode_plus( 68 | [(premise, hypothesis) for hypothesis in hypotheses], 69 | return_tensors="pt", 70 | padding=True, 71 | truncation_strategy="only_first", 72 | ).to(device) 73 | logits = nli_model(**batch_encoded)[0] 74 | 75 | # we throw away "neutral" (dim 1) and take the probability of 76 | # "entailment" (2) as the probability of the label being true 77 | entail_contradiction_logits = logits[:, [0, 2]] 78 | probs = entail_contradiction_logits.softmax(dim=1) 79 | prob_label_is_true = probs[:, 1] 80 | for j, sub_question in enumerate(batch): 81 | x["sub_questions_nli_true_prob"].append(prob_label_is_true[j]) 82 | 83 | torch.cuda.empty_cache() # Clear GPU memory after processing 84 | -------------------------------------------------------------------------------- /training/README.md: -------------------------------------------------------------------------------- 1 | # 🏃 Training RetRobust 2 | Training is done with [**QLoRA**](https://huggingface.co/blog/4bit-transformers-bitsandbytes), similarly to [**Guanaco**](). See `scripts/train.py` and `scripts/upload.py` scripts to train and upload models to the 🤗 hub. 3 | -------------------------------------------------------------------------------- /training/scripts/data_collator.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | 3 | from transformers import DataCollatorForLanguageModeling, PreTrainedTokenizerBase 4 | from collections.abc import Mapping 5 | from typing import Any, Callable, Dict, List, NewType, Optional, Tuple, Union 6 | from transformers.data.data_collator import _torch_collate_batch, DataCollatorMixin 7 | 8 | 9 | @dataclass 10 | class DataCollatorSelfAsk(DataCollatorForLanguageModeling): 11 | """ 12 | Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they 13 | are not all of the same length. 14 | 15 | Args: 16 | tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): 17 | The tokenizer used for encoding the data. 18 | mlm (`bool`, *optional*, defaults to `True`): 19 | Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs 20 | with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked 21 | tokens and the value to predict for the masked token. 22 | mlm_probability (`float`, *optional*, defaults to 0.15): 23 | The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`. 24 | pad_to_multiple_of (`int`, *optional*): 25 | If set will pad the sequence to a multiple of the provided value. 26 | return_tensors (`str`): 27 | The type of Tensor to return. Allowable values are "np", "pt" and "tf". 28 | 29 | 30 | 31 | For best performance, this data collator should be used with a dataset having items that are dictionaries or 32 | BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [`PreTrainedTokenizer`] or a 33 | [`PreTrainedTokenizerFast`] with the argument `return_special_tokens_mask=True`. 34 | 35 | """ 36 | 37 | tokenizer: PreTrainedTokenizerBase 38 | mlm: bool = False 39 | mlm_probability: float = 0.15 40 | pad_to_multiple_of: Optional[int] = None 41 | tf_experimental_compile: bool = False 42 | return_tensors: str = "pt" 43 | 44 | def __post_init__(self): 45 | if getattr(self.tokenizer, "pad_token", None) is None: 46 | self.tokenizer.pad_token = self.tokenizer.eos_token 47 | if self.mlm and self.tokenizer.mask_token is None: 48 | raise ValueError( 49 | "This tokenizer does not have a mask token which is necessary for masked language modeling. " 50 | "You should pass `mlm=False` to train on causal language modeling instead." 51 | ) 52 | if self.tf_experimental_compile: 53 | import tensorflow as tf 54 | 55 | self.tf_mask_tokens = tf.function(self.tf_mask_tokens, jit_compile=True) 56 | 57 | 58 | def torch_call(self, examples: List[Union[List[int], Any, Dict[str, Any]]]) -> Dict[str, Any]: 59 | # Handle dict or lists with proper padding and conversion to tensor. 60 | if isinstance(examples[0], Mapping): 61 | batch = self.tokenizer.pad(examples, return_tensors="pt", pad_to_multiple_of=self.pad_to_multiple_of) 62 | else: 63 | batch = { 64 | "input_ids": _torch_collate_batch(examples, self.tokenizer, pad_to_multiple_of=self.pad_to_multiple_of) 65 | } 66 | 67 | # If special token mask has been preprocessed, pop it from the dict. 68 | special_tokens_mask = batch.pop("special_tokens_mask", None) 69 | if self.mlm: 70 | batch["input_ids"], batch["labels"] = self.torch_mask_tokens( 71 | batch["input_ids"], special_tokens_mask=special_tokens_mask 72 | ) 73 | else: 74 | start_indices = [len(self.tokenizer.encode(self.tokenizer.decode(x['input_ids']).split("Are follow up questions needed here: ")[0]))-2 for x in examples] 75 | labels = batch["input_ids"].clone() 76 | if self.tokenizer.pad_token_id is not None: 77 | labels[labels == self.tokenizer.pad_token_id] = -100 78 | batch["labels"] = labels 79 | for i, l in enumerate(batch["labels"]): 80 | l[:start_indices[i]] = -100 81 | return batch 82 | -------------------------------------------------------------------------------- /training/scripts/train.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | import torch 4 | from datasets import load_dataset 5 | from peft import LoraConfig 6 | from trl import SFTTrainer 7 | from data_collator import DataCollatorSelfAsk 8 | from transformers import ( 9 | AutoModelForCausalLM, 10 | BitsAndBytesConfig, 11 | AutoTokenizer, 12 | TrainingArguments, 13 | ) 14 | 15 | 16 | def parse_args(): 17 | parser = argparse.ArgumentParser() 18 | parser.add_argument( 19 | "--base_model_name", 20 | type=str, 21 | default="meta-llama/Llama-2-13b-hf", 22 | ) 23 | parser.add_argument( 24 | "--dataset_name", 25 | type=str, 26 | default="Ori/llama-2-13b-peft-nq-no-ret", 27 | ) 28 | parser.add_argument( 29 | "--output_dir", 30 | type=str, 31 | default="nq_no_ret", 32 | ) 33 | parser.add_argument( 34 | "--seed", 35 | type=int, 36 | default=1, 37 | ) 38 | return parser.parse_args() 39 | 40 | 41 | def train_model(args): 42 | """ 43 | set up and train using HF trainer 44 | based on example code in HF 45 | """ 46 | # get args 47 | dataset_name = args.dataset_name 48 | model_name = args.base_model_name 49 | output_dir = args.output_dir 50 | seed = args.seed 51 | 52 | # load dataset 53 | train_dataset = load_dataset(dataset_name, split="train") 54 | 55 | # setup 4 bit training 56 | bnb_config = BitsAndBytesConfig( 57 | load_in_4bit=True, 58 | bnb_4bit_quant_type="nf4", 59 | bnb_4bit_compute_dtype=torch.bfloat16, 60 | ) 61 | 62 | # load model and tokenizer 63 | model = AutoModelForCausalLM.from_pretrained( 64 | model_name, 65 | quantization_config=bnb_config, 66 | use_auth_token=True, 67 | torch_dtype=torch.bfloat16, 68 | ) 69 | model.config.use_cache = False 70 | tokenizer = AutoTokenizer.from_pretrained(model_name, use_auth_token=True) 71 | tokenizer.pad_token = tokenizer.eos_token 72 | 73 | # default configs 74 | lora_alpha = 16 75 | lora_dropout = 0.1 76 | lora_r = 64 77 | 78 | peft_config = LoraConfig( 79 | lora_alpha=lora_alpha, 80 | lora_dropout=lora_dropout, 81 | r=lora_r, 82 | bias="none", 83 | task_type="CAUSAL_LM", 84 | target_modules=[ 85 | "q_proj", 86 | "k_proj", 87 | "v_proj", 88 | "o_proj", 89 | ], 90 | ) 91 | 92 | num_train_epochs = 5 93 | gradient_accumulation_steps = 1 94 | optim = "paged_adamw_32bit" 95 | save_strategy = "epoch" 96 | learning_rate = 2e-4 97 | lr_scheduler_type = "linear" 98 | warmup_ratio = 0.03 99 | logging_steps = 25 100 | prediction_loss_only = True 101 | eval_steps = 0.2 102 | bf16 = True 103 | 104 | training_arguments = TrainingArguments( 105 | output_dir=output_dir, 106 | seed=seed, 107 | num_train_epochs=num_train_epochs, 108 | auto_find_batch_size=4, 109 | gradient_accumulation_steps=gradient_accumulation_steps, 110 | optim=optim, 111 | save_strategy=save_strategy, 112 | learning_rate=learning_rate, 113 | lr_scheduler_type=lr_scheduler_type, 114 | warmup_ratio=warmup_ratio, 115 | logging_strategy="epoch", 116 | logging_steps=logging_steps, 117 | prediction_loss_only=prediction_loss_only, 118 | eval_steps=eval_steps, 119 | bf16=bf16, 120 | ) 121 | max_seq_length = 4096 122 | 123 | # init the trainer 124 | trainer = SFTTrainer( 125 | model=model, 126 | train_dataset=train_dataset, 127 | peft_config=peft_config, 128 | dataset_text_field="text", 129 | max_seq_length=max_seq_length, 130 | tokenizer=tokenizer, 131 | args=training_arguments, 132 | data_collator=DataCollatorSelfAsk( 133 | tokenizer=AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf"), 134 | mlm=False, 135 | ), 136 | ) 137 | 138 | for name, module in trainer.model.named_modules(): 139 | if "norm" in name: 140 | module = module.to(torch.float32) 141 | 142 | # train! 143 | trainer.train() 144 | 145 | 146 | if __name__ == "__main__": 147 | """ """ 148 | args = parse_args() 149 | train_model(args) 150 | -------------------------------------------------------------------------------- /training/scripts/upload.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import argparse 3 | from peft import PeftModel, PeftConfig 4 | from transformers import AutoModelForCausalLM, AutoTokenizer 5 | 6 | 7 | def parse_args(): 8 | parser = argparse.ArgumentParser() 9 | parser.add_argument( 10 | "--local_model_id", 11 | type=str, 12 | default="nq_no_ret/checkpoint-625", 13 | ) 14 | parser.add_argument( 15 | "--hf_model_id", 16 | type=str, 17 | default="", 18 | help="The location of the model on the HF hub", 19 | ) 20 | return parser.parse_args() 21 | 22 | 23 | def upload_model(args): 24 | """ 25 | :param args: 26 | :return: 27 | """ 28 | peft_model_id = args.local_model_id 29 | model_id_load = args.hf_model_id 30 | 31 | # load model 32 | config = PeftConfig.from_pretrained(peft_model_id) 33 | model = AutoModelForCausalLM.from_pretrained( 34 | config.base_model_name_or_path, 35 | torch_dtype=torch.bfloat16, 36 | device_map="auto", 37 | ) 38 | tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path) 39 | tokenizer.pad_token = tokenizer.eos_token 40 | 41 | # Load the Lora model 42 | model = PeftModel.from_pretrained(model, peft_model_id) 43 | 44 | # push to hub 45 | # tokenizer 46 | tokenizer.push_to_hub(model_id_load, use_auth_token=True) 47 | # safetensors 48 | model.push_to_hub(model_id_load, use_auth_token=True, safe_serialization=True) 49 | # torch tensors 50 | model.push_to_hub(model_id_load, use_auth_token=True) 51 | 52 | 53 | if __name__ == "__main__": 54 | """ """ 55 | args = parse_args() 56 | upload_model(args) 57 | --------------------------------------------------------------------------------