├── log └── __init__.py ├── img ├── idea.png └── method.png ├── README.md ├── demo.py ├── LICENSE ├── main_verifier.py ├── utils.py └── dataset └── MedDialog └── english-test.json /log/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /img/idea.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/WENGSYX/Self-Verification/HEAD/img/idea.png -------------------------------------------------------------------------------- /img/method.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/WENGSYX/Self-Verification/HEAD/img/method.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Large Language Models are Better Reasoners with Self-Verification 2 |

3 | GitHub 4 | GitHub repo size 5 | GitHub top language 6 | GitHub last commit 7 |

8 | 9 | This is the official implementation of `Large Language Models are Better Reasoners with Self-Verification`. 10 | 11 | (**EMNLP 2023 Findings**) 12 | 13 | ## Demo 14 | 15 | [https://github-production-user-asset-6210df.s3.amazonaws.com/64484703/250267125-281968bf-a8a2-438f-838b-d189c2501d40.mp4 16 | https://user-images.githubusercontent.com/70048414/232352935-55c6bf7c-3958-406e-8610-0913475a0b05.mp4](https://github.com/WENGSYX/Self-Verification/assets/64484703/281968bf-a8a2-438f-838b-d189c2501d40) 17 | 18 | ## Installation 19 | Make sure you have Python>=3.8 installed on your machine. 20 | ``` 21 | pip install torch==1.8.2+cu111 torchtext==0.9.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html 22 | pip install tqdm transformers sklearn pandas numpy sentencepiece openai 23 | ``` 24 | 25 | ## Set your OpenAI API key 26 | ``` 27 | # https://beta.openai.com/account/api-keys 28 | export OPENAI_API_KEY=(YOUR OPENAI API KEY) 29 | ``` 30 | 31 | ## Set arguments. 32 | ``` 33 | model=CODEX # {"gpt3", "gpt3-medium", "gpt3-large", "gpt3-xl", "CODEX", "CODEX-001"}. "codex" is the smallest model. 34 | dataset=multiarith # We can use other datasets. See help for the details. 35 | api_time_interval=4.0 # Caution. The API allows users request API up to 20 times in a minutes, otherwise errors happen. 36 | ``` 37 | 38 | ## Quick Start 39 | 40 | ### Demo 41 | ``` 42 | python demo.py 43 | ``` 44 | 45 | ### Self-Verification (our proposal) 46 | ``` 47 | python main.py --method=verifier_cot --model=${model} --dataset=${dataset} 48 | ``` 49 | 50 | ### CoT 51 | ``` 52 | # MultiArith and GSM8K are currently available. 53 | python main.py --method=few_shot_cot --model=${model} --dataset=${dataset} 54 | ``` 55 | 56 | 57 | 58 | ## Method 59 | 60 | ![main](./img/method.png) 61 | 62 | 1. Forward Reasoning, the LLM generates candidate thought chains and conclusions for a given problem text; 63 | 2. Backward Verification, we use the LLM to verify whether the conditions meet the candidate conclusions and rank the candidate conclusions based on a verification score. 64 | 65 | 66 | 67 | ## Cite 68 | 69 | > ``` 70 | > @misc{weng2023large, 71 | > title={Large Language Models are Better Reasoners with Self-Verification}, 72 | > author={Yixuan Weng and Minjun Zhu and Fei Xia and Bin Li and Shizhu He and Kang Liu and Jun Zhao}, 73 | > year={2023}, 74 | > eprint={2212.09561}, 75 | > archivePrefix={arXiv}, 76 | > primaryClass={cs.AI} 77 | > } 78 | > ``` 79 | -------------------------------------------------------------------------------- /demo.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | import torch 4 | import random 5 | import time 6 | from tqdm import tqdm 7 | import os 8 | from utils import * 9 | import openai 10 | 11 | import time 12 | def print_stream(text): 13 | for i in text: 14 | print(i,end='',flush=True) 15 | time.sleep(0.02) 16 | print('\n',end='') 17 | 18 | def main(): 19 | args = parse_arguments() 20 | fix_seed(args.random_seed) 21 | 22 | # Initialize decoder class (load model and tokenizer) ... 23 | decoder = Decoder(args) 24 | 25 | demo_F = create_demo_text(args, cot_flag=True) 26 | demo_B = create_verifier_demo_text(args, cot_flag=True) 27 | demo_F = demo_F.split('\n\n')[:-1] 28 | demo_B = demo_B.split('\n\n')[:-1] 29 | 30 | 31 | while True: 32 | x = input("\033[0;34mWhat do you want to ask me? :)\n[QUESTION] \033[0m") 33 | if 'break' == x.lower(): 34 | break 35 | # Prepare question template ... 36 | 37 | max_length = args.max_length_cot 38 | declarative = '' 39 | answers = [] 40 | 41 | x_F = "Q: " + x + "\n" + "A:" 42 | x_F = '\n\n'.join(demo_F) + '\n\n' + x_F 43 | 44 | print_stream("\033[0;33m[INFO] Setting Temperature = \033[0m{}".format(args.K)) 45 | print_stream("\033[0;33m[INFO] Setting Candidate Answer Number = \033[0m{}".format(args.N)) 46 | print('') 47 | print_stream("\033[0;33m[INFO] Now Model is Generating Candidate Answers...\033[0m") 48 | 49 | 50 | pred = decoder.decode(args, x_F, max_length, 0, args.K, args.N, '\n') 51 | 52 | for p_it in pred: 53 | p_item = answer_cleansing(args, p_it) 54 | try: 55 | p_item = str(float(p_item)) 56 | except: 57 | pass 58 | 59 | if p_item != '': 60 | if p_item not in answers: 61 | answers.append(p_item) 62 | 63 | if len(answers) == 0: 64 | print_stream("\033[0;31m[ERROR] The Candidate Answer is None!!!\033[0m") 65 | print_stream("\033[0;31m[ERROR] Break Now! /(ㄒoㄒ)/~~\033[0m") 66 | print('') 67 | 68 | else: 69 | 70 | if len(answers) == 1: 71 | print_stream("\033[0;32m[ACCEPT] The Candidate Answer's number is 1, so the answer is:\033[0m") 72 | print_stream("\033[0;32m[ACCEPT] {}\033[0m".format(answers[0])) 73 | print('') 74 | 75 | else: 76 | print_stream("\033[0;33m[INFO] Now Model is Self-Verificating the Candidate Answers...\033[0m") 77 | scores = {i: 0 for i in range(len(answers))} 78 | pred_verifier = {i: [] for i in range(len(answers))} 79 | for A in range(len(answers)): 80 | 81 | decl, answer, declarative = question_turn_decalrative(args, x, answers[A], answers[0], 82 | decoder.decode, declarative) 83 | for d in range(len(decl)): 84 | random.shuffle(demo_B) 85 | x_B = '\n\n'.join(demo_B) + 'Q: ' + decl[d] + '\nA: ' 86 | try: 87 | pred_v = decoder.decode(args, x_B, max_length, 0, 0.4, 10, '\n\n') 88 | except: 89 | pred_v = [''] * 10 90 | answers_verifier = [] 91 | for p in range(len(pred_v)): 92 | p_item_v = answer_cleansing_verifier(args, pred_v[p]) 93 | try: 94 | answers_verifier.append(float(p_item_v)) 95 | except: 96 | try: 97 | answers_verifier.append(p_item_v) 98 | except: 99 | pass 100 | try: 101 | score = sum(np.array(answers_verifier) == np.array(float(answer[d]))) 102 | except: 103 | try: 104 | score = sum(np.array(answers_verifier) == np.array(answer[d])) 105 | except: 106 | score = 0 107 | pred_verifier[A].append(pred_v) 108 | scores[A] += score 109 | try: 110 | print_stream(f"\033[0;36m[Self-Verification SCORE] The Candidate Answer “{answers[A]}” :\033[0m") 111 | print_stream(f"\033[0;36m[Self-Verification SCORE] {'█'*scores[A]}: {str(scores[A])}\033[0m") 112 | 113 | except: 114 | pass 115 | verifier_scores = list(scores.values()) 116 | 117 | for i in range(len(verifier_scores)): 118 | if verifier_scores[i] == max(verifier_scores): 119 | print_stream(f"\033[0;32m[ACCEPT] The Best Answer is:\033[0m") 120 | print_stream(f"\033[0;32m[ACCEPT] {answers[i]}\033[0m") 121 | print('') 122 | break 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | def parse_arguments(): 131 | parser = argparse.ArgumentParser(description="Reason with self-verification") 132 | 133 | parser.add_argument("--random_seed", type=int, default=1, help="random seed") 134 | 135 | parser.add_argument( 136 | "--model", type=str, default="text-003", 137 | choices=["gpt3", "gpt3-medium", "gpt3-large", "gpt3-xl", "codex", "codex-001","text-003"], 138 | help="model used for decoding. Note that 'gpt3' are the smallest models." 139 | ) 140 | 141 | parser.add_argument( 142 | "--max_length_cot", type=int, default=168, 143 | help="maximum length of output tokens by model for reasoning extraction" 144 | ) 145 | 146 | parser.add_argument( 147 | "--api_time_interval", type=float, default=4.0, help="" 148 | ) 149 | parser.add_argument( 150 | "--log_dir", type=str, default="./log/", help="log directory" 151 | ) 152 | parser.add_argument( 153 | "--N", type=int, default=5 154 | ) 155 | parser.add_argument( 156 | "--K", type=int, default=0.6 157 | ) 158 | parser.add_argument( 159 | "--FN", type=int, default=0, help="few-shot number" 160 | ) 161 | 162 | args = parser.parse_args() 163 | args.dataset = 'gsm8k' 164 | args.direct_answer_trigger_for_fewshot = "The answer is" 165 | args.method = 'verifier_cot' 166 | args.verifier_text = " What is the answer of 'X'?" 167 | return args 168 | 169 | 170 | if __name__ == "__main__": 171 | main() 172 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /main_verifier.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | import torch 4 | import random 5 | import time 6 | from tqdm import tqdm 7 | import os 8 | from utils import * 9 | 10 | 11 | def log_data(text, path): 12 | with open(path + '/loggings.txt', 'a', encoding='utf-8') as f: 13 | f.write(text) 14 | print(text) 15 | f.write('\n') 16 | 17 | 18 | def log_data_self(text, path): 19 | with open(path, 'a', encoding='utf-8') as f: 20 | f.write(text) 21 | print(text) 22 | f.write('\n') 23 | 24 | 25 | def log_start(MODEL, DATA, N, K, FN): 26 | log_name = MODEL + "_" + DATA + "_" + str(N) + "_" + str(K) + "_" + str(FN) 27 | try: 28 | os.mkdir('log/' + log_name) 29 | except: 30 | log_name += time.strftime("%Y_%m_%d_%H_%M_%S", time.localtime()) 31 | os.mkdir('log/' + log_name) 32 | 33 | with open('log/' + log_name + '/python_file.py', 'a', encoding='utf-8') as f: 34 | with open(os.path.basename(__file__), 'r', encoding='utf-8') as f2: 35 | file = f2.read() 36 | f.write(file) 37 | 38 | path = 'log/' + log_name 39 | return path 40 | 41 | 42 | class AverageMeter: # 为了tqdm实时显示loss和acc 43 | def __init__(self): 44 | self.reset() 45 | 46 | def reset(self): 47 | self.val = 0 48 | self.avg = 0 49 | self.sum = [] 50 | self.count = 0 51 | 52 | def update(self, val, n=1): 53 | self.val = val 54 | self.sum.append(val * n) 55 | self.count += n 56 | self.avg = sum(self.sum) / self.count 57 | 58 | 59 | def main(): 60 | args = parse_arguments() 61 | path = log_start(args.model, args.dataset, args.N, args.K, args.method) 62 | log_data('*****************************', path) 63 | print(args) 64 | log_data('*****************************', path) 65 | fix_seed(args.random_seed) 66 | 67 | # Initialize decoder class (load model and tokenizer) ... 68 | decoder = Decoder(args) 69 | 70 | log_data("setup data loader ...", path) 71 | dataloader = setup_data_loader(args) 72 | print_now() 73 | 74 | if args.method == "few_shot": 75 | demo = create_demo_text(args, cot_flag=False) 76 | elif args.method == 'verifier_cot': 77 | demo_F = create_demo_text(args, cot_flag=True) 78 | demo_B = create_verifier_demo_text(args, cot_flag=True) 79 | demo_F = demo_F.split('\n\n')[:-1] 80 | demo_B = demo_B.split('\n\n')[:-1] 81 | else: 82 | pass 83 | 84 | total = 0 85 | correct_list = [] 86 | tk = tqdm(dataloader) 87 | accs = AverageMeter() 88 | accs_sc = AverageMeter() 89 | accs_avg = AverageMeter() 90 | accs_verifier = AverageMeter() 91 | loggings = [] 92 | for i, data in enumerate(tk): 93 | log_data('*************************', path) 94 | log_data("{}st data".format(i + 1), path) 95 | 96 | # Prepare question template ... 97 | x, y = data 98 | 99 | max_length = args.max_length_cot if "cot" in args.method else args.max_length_direct 100 | if args.method == "few_shot_cot": 101 | x = "Q: " + x[0] + "\n" + "A:" 102 | y = y[0].strip() 103 | x = demo + x 104 | 105 | max_length = args.max_length_cot if "cot" in args.method else args.max_length_direct 106 | z = decoder.decode(args, x, max_length, i, 0, 1, '\n')[0] 107 | 108 | # Answer extraction for zero-shot-cot ... 109 | pred = z 110 | log_data(x + pred,path) 111 | 112 | # Clensing of predicted answer ... 113 | pred = answer_cleansing(args, pred) 114 | 115 | # Choose the most frequent answer from the list ... 116 | log_data("pred : {}".format(pred),path) 117 | log_data("GT : " + y,path) 118 | log_data('*************************',path) 119 | 120 | # Checking answer ... 121 | correct = (np.array([pred]) == np.array([y])).sum().item() 122 | correct_list.append(correct) 123 | total += 1 # np.array([y]).size(0) 124 | elif args.method == 'verifier_cot': 125 | declarative = '' 126 | answers = [] 127 | verifier_acc_self_now = [] 128 | log_data('Q: {}'.format(x[0]), path) 129 | log_data("GT : " + y[0], path) 130 | y = y[0].strip() 131 | x_F = "Q: " + x[0] + "\n" + "A:" 132 | x_F = '\n\n'.join(demo_F) + '\n\n' + x_F 133 | 134 | pred = decoder.decode(args, x_F, max_length, i, args.K, args.N, '\n') 135 | 136 | if len(pred) == 0: 137 | continue 138 | # Clensing of predicted answer ... 139 | 140 | for p_it in pred: 141 | p_item = answer_cleansing(args, p_it) 142 | try: 143 | p_item = str(float(p_item)) 144 | y = str(float(y)) 145 | except: 146 | pass 147 | 148 | is_true = (np.array([p_item]) == np.array([y])).sum().item() 149 | log_data("{}: {}".format({0: False, 1: True}[is_true], p_it), path) 150 | if p_item != '': 151 | if args.dataset in ('aqua', 'commonsensqa'): 152 | choice = x[0].split('Answer Choices: ')[1].split('(')[1:] 153 | choice = {i[0]: i[2:] for i in choice} 154 | p_item = choice[p_item] 155 | 156 | if p_item not in answers: 157 | answers.append(p_item) 158 | if len(answers) == 0: 159 | accs.update(0) 160 | accs_verifier.update(0) 161 | correct_list.append(0) 162 | total += 1 # np.array([y]).size(0) 163 | tk.set_postfix(accs=accs.avg, verifier_acc=accs_verifier.avg) 164 | log_data('verifier:{} accs:{} '.format(0, 0), path) 165 | log_data('*************************', path) 166 | loggings.append({'question': data[0][0], 167 | 'target': data[1][0], 168 | 'CoT': '', 169 | 'is_true': 0, 170 | 'verifier': 0}) 171 | 172 | else: 173 | if len(answers) == 1: 174 | scores = {0: 1} 175 | pred_verifier = [] 176 | else: 177 | scores = {i: 0 for i in range(len(answers))} 178 | pred_verifier = {i: [] for i in range(len(answers))} 179 | for A in range(len(answers)): 180 | 181 | decl, answer, declarative = question_turn_decalrative(args, x[0], answers[A], answers[0], 182 | decoder.decode, declarative) 183 | for d in range(len(decl)): 184 | random.shuffle(demo_B) 185 | x_B = '\n\n'.join(demo_B) + 'Q: ' + decl[d] + '\nA: ' 186 | try: 187 | pred_v = decoder.decode(args, x_B, max_length, i, 0.2, 10, '\n\n') 188 | except: 189 | pred_v = [''] * 10 190 | answers_verifier = [] 191 | for p in range(len(pred_v)): 192 | p_item_v = answer_cleansing_verifier(args, pred_v[p]) 193 | try: 194 | answers_verifier.append(float(p_item_v)) 195 | except: 196 | try: 197 | answers_verifier.append(p_item_v) 198 | except: 199 | pass 200 | try: 201 | score = sum(np.array(answers_verifier) == np.array(float(answer[d]))) 202 | except: 203 | try: 204 | score = sum(np.array(answers_verifier) == np.array(answer[d])) 205 | except: 206 | score = 0 207 | pred_verifier[A].append(pred_v) 208 | scores[A] += score 209 | try: 210 | log_data('{} - {}: {}'.format(answers[A], scores[A], pred_v[0].replace('\n', ' ')), path) 211 | except: 212 | pass 213 | verifier_is_ture = list(scores.values()) 214 | 215 | if args.dataset in ("aqua", "commonsensqa"): 216 | answers_is_ture = (np.array(answers) == np.array([choice[y]])).tolist() 217 | accs.update((np.array([answers[0]]) == np.array([choice[y]])).sum().item()) 218 | accs_verifier.update( 219 | (np.array([answers[np.argmax(np.array(scores)).item()]]) == np.array([choice[y]])).sum().item()) 220 | correct = (np.array([answers[np.argmax(np.array(scores)).item()]]) == np.array( 221 | [choice[y]])).sum().item() 222 | log_data('verifier:{} accs:{} '.format( 223 | (np.array([answers[np.argmax(np.array(scores)).item()]]) == np.array([choice[y]])).sum().item(), 224 | (np.array([answers[0]]) == np.array([choice[y]])).sum().item()), 225 | path) 226 | log_data('*************************', path) 227 | loggings.append({'question': data[0][0], 228 | 'target': data[1][0], 229 | 'CoT': pred_verifier, 230 | 'is_true': (np.array(answers) == np.array([choice[y]] * len(answers))).tolist(), 231 | 'verifier': (np.array([answers[np.argmax(np.array(scores)).item()]]) == np.array( 232 | [choice[y]])).sum().item()}) 233 | 234 | else: 235 | answers_is_ture = (np.array(answers) == np.array(y)).tolist() 236 | accs.update((np.array([answers[0]]) == np.array([y])).sum().item()) 237 | accs_verifier.update((np.array([answers[np.argmax(np.array(verifier_is_ture)).item()]]) == np.array( 238 | [y])).sum().item()) 239 | correct = (np.array([answers[np.argmax(np.array(verifier_is_ture)).item()]]) == np.array( 240 | [y])).sum().item() 241 | log_data('verifier:{} accs:{} '.format( 242 | (np.array([answers[np.argmax(np.array(verifier_is_ture)).item()]]) == np.array( 243 | [y])).sum().item(), 244 | (np.array([answers[0]]) == np.array([y])).sum().item()), 245 | path) 246 | log_data('*************************', path) 247 | loggings.append({'question': data[0][0], 248 | 'target': data[1][0], 249 | 'CoT': pred_verifier, 250 | 'is_true': (np.array(answers) == np.array([y] * len(answers))).tolist(), 251 | 'verifier': (np.array([answers[np.argmax(np.array(scores)).item()]]) == np.array( 252 | [y])).sum().item()}) 253 | log_data_self( 254 | '{}'.format(','.join([str(int(x)) for x in answers_is_ture])), 255 | path + '/answers_acc.txt') 256 | log_data_self( 257 | '{}'.format(','.join([str(int(x)) for x in verifier_is_ture])), 258 | path + '/verifier_acc.txt') 259 | correct_list.append(correct) 260 | total += 1 # np.array([y]).size(0) 261 | tk.set_postfix(accs=accs.avg, verifier_acc=accs_verifier.avg) 262 | 263 | 264 | else: 265 | raise ValueError("method is not properly defined ...") 266 | 267 | # Calculate accuracy ... 268 | accuracy = (sum(correct_list) * 1.0 / total) * 100 269 | log_data("accuracy : {}".format(accuracy), path) 270 | if args.method == 'verifier_cot': 271 | log_data('accs:{} self_consistency:{} Top_N_acc:{}'.format(accs.avg, accs_sc.avg, accs_avg.avg), path) 272 | loggings = pd.DataFrame(loggings) 273 | loggings.to_csv(path + '/ls.csv') 274 | 275 | 276 | 277 | 278 | def parse_arguments(): 279 | parser = argparse.ArgumentParser(description="Reason with self-verification") 280 | 281 | parser.add_argument( 282 | "--api_log_file_name", type=str, default=None, 283 | help="mandatory argument ! json['i>=1']['j==1']['k={1,2}'][{'request', response'}]" 284 | ) 285 | 286 | parser.add_argument("--random_seed", type=int, default=1, help="random seed") 287 | 288 | parser.add_argument( 289 | "--dataset", type=str, default="gsm8k", 290 | choices=["aqua", "gsm8k", "commonsensqa", "addsub", "multiarith", "strategyqa", "svamp", "singleeq", 291 | "bigbench_date", "object_tracking", "coin_flip", "last_letters", "meddialog"], 292 | help="dataset used for experiment" 293 | ) 294 | 295 | parser.add_argument("--minibatch_size", type=int, default=1, choices=[1], 296 | help="minibatch size should be 1 because GPT-3 API takes only 1 input for each request") 297 | 298 | parser.add_argument("--max_num_worker", type=int, default=0, help="maximum number of workers for dataloader") 299 | 300 | parser.add_argument( 301 | "--model", type=str, default="codex", 302 | choices=["gpt3", "gpt3-medium", "gpt3-large", "gpt3-xl", "codex", "codex-001"], 303 | help="model used for decoding. Note that 'gpt3' are the smallest models." 304 | ) 305 | 306 | parser.add_argument( 307 | "--method", type=str, default="verifier_cot", 308 | choices=["zero_shot", "zero_shot_cot", "few_shot", "few_shot_cot", "verifier_cot", "verifier"], help="method" 309 | ) 310 | parser.add_argument( 311 | "--cot_trigger_no", type=int, default=1, 312 | help="A trigger sentence that elicits a model to execute chain of thought" 313 | ) 314 | parser.add_argument( 315 | "--max_length_cot", type=int, default=168, 316 | help="maximum length of output tokens by model for reasoning extraction" 317 | ) 318 | parser.add_argument( 319 | "--max_length_direct", type=int, default=32, 320 | help="maximum length of output tokens by model for answer extraction" 321 | ) 322 | parser.add_argument( 323 | "--limit_dataset_size", type=int, default=0, 324 | help="whether to limit test dataset size. if 0, the dataset size is unlimited and we use all the samples in the dataset for testing." 325 | ) 326 | parser.add_argument( 327 | "--api_time_interval", type=float, default=4.0, help="" 328 | ) 329 | parser.add_argument( 330 | "--log_dir", type=str, default="./log/", help="log directory" 331 | ) 332 | parser.add_argument( 333 | "--N", type=int, default=5 334 | ) 335 | parser.add_argument( 336 | "--K", type=int, default=0.3 337 | ) 338 | parser.add_argument( 339 | "--FN", type=int, default=0, help="few-shot number" 340 | ) 341 | 342 | args = parser.parse_args() 343 | 344 | if args.dataset == "aqua": 345 | args.dataset_path = "./dataset/AQuA/test.json" 346 | args.direct_answer_trigger = "\nTherefore, among A through E, the answer is" 347 | elif args.dataset == "gsm8k": 348 | args.dataset_path = "./dataset/grade-school-math/test.jsonl" 349 | args.direct_answer_trigger = "\nTherefore, the answer (arabic numerals) is" 350 | elif args.dataset == "commonsensqa": 351 | args.dataset_path = "./dataset/CommonsenseQA/dev_rand_split.jsonl" 352 | args.direct_answer_trigger = "\nTherefore, among A through E, the answer is" 353 | args.plausible_answer_trigger = "Choose the most plausible answer from among choices A through E." 354 | elif args.dataset == "addsub": 355 | args.dataset_path = "./dataset/AddSub/AddSub.json" 356 | args.direct_answer_trigger = "\nTherefore, the answer (arabic numerals) is" 357 | elif args.dataset == "multiarith": 358 | args.dataset_path = "./dataset/MultiArith/MultiArith.json" 359 | args.direct_answer_trigger = "\nTherefore, the answer (arabic numerals) is" 360 | elif args.dataset == "strategyqa": 361 | args.dataset_path = "./dataset/StrategyQA/task.json" 362 | args.direct_answer_trigger = "\nTherefore, the answer (Yes or No) is" 363 | elif args.dataset == "svamp": 364 | args.dataset_path = "./dataset/SVAMP/SVAMP.json" 365 | args.direct_answer_trigger = "\nTherefore, the answer (arabic numerals) is" 366 | elif args.dataset == "singleeq": 367 | args.dataset_path = "./dataset/SingleEq/questions.json" 368 | args.direct_answer_trigger = "\nTherefore, the answer (arabic numerals) is" 369 | elif args.dataset == "bigbench_date": 370 | args.dataset_path = "./dataset/Bigbench_Date/task.json" 371 | args.direct_answer_trigger = "\nTherefore, among A through F, the answer is" 372 | elif args.dataset == "object_tracking": 373 | args.dataset_path = "./dataset/Bigbench_object_tracking/task.json" 374 | args.direct_answer_trigger = "\nTherefore, among A through C, the answer is" 375 | elif args.dataset == "coin_flip": 376 | args.dataset_path = "./dataset/coin_flip/coin_flip.json" 377 | args.direct_answer_trigger = "\nTherefore, the answer (Yes or No) is" 378 | elif args.dataset == "last_letters": 379 | args.dataset_path = "./dataset/last_letters/last_letters.json" 380 | args.direct_answer_trigger = "\nTherefore, the answer is" 381 | elif args.dataset == "meddialog": 382 | args.dataset_path = "./dataset/MedDialog/english-test.json" 383 | args.direct_answer_trigger = "\nTherefore, the answer is" 384 | 385 | else: 386 | raise ValueError("dataset is not properly defined ...") 387 | 388 | # "Therefore, the answer ..." -> "The answer ..." 389 | trigger = args.direct_answer_trigger.replace("\nTherefore, ", "") 390 | args.direct_answer_trigger_for_zeroshot = trigger[0].upper() + trigger[1:] 391 | args.direct_answer_trigger_for_zeroshot_cot = args.direct_answer_trigger 392 | 393 | args.direct_answer_trigger_for_fewshot = "The answer is" 394 | if args.dataset in ("commonsensqa"): 395 | args.verifier_text = ' Judge whether this statement is normal (yes or no)' 396 | else: 397 | args.verifier_text = " What is the answer of 'X'?" 398 | if args.cot_trigger_no == 1: 399 | args.cot_trigger = "Let's think step by step." 400 | elif args.cot_trigger_no == 2: 401 | args.cot_trigger = "We should think about this step by step." 402 | elif args.cot_trigger_no == 3: 403 | args.cot_trigger = "First," 404 | elif args.cot_trigger_no == 4: 405 | args.cot_trigger = "Before we dive into the answer," 406 | elif args.cot_trigger_no == 5: 407 | args.cot_trigger = "Proof followed by the answer." 408 | elif args.cot_trigger_no == 6: 409 | args.cot_trigger = "Let's think step by step in a realistic way." 410 | elif args.cot_trigger_no == 7: 411 | args.cot_trigger = "Let's think step by step using common sense and knowledge." 412 | elif args.cot_trigger_no == 8: 413 | args.cot_trigger = "Let's think like a detective step by step." 414 | elif args.cot_trigger_no == 9: 415 | args.cot_trigger = "Let's think about this logically." 416 | elif args.cot_trigger_no == 10: 417 | args.cot_trigger = "Let's think step by step. First," 418 | elif args.cot_trigger_no == 11: 419 | args.cot_trigger = "Let's think" 420 | elif args.cot_trigger_no == 12: 421 | args.cot_trigger = "Let's solve this problem by splitting it into steps." 422 | elif args.cot_trigger_no == 13: 423 | args.cot_trigger = "The answer is after the proof." 424 | elif args.cot_trigger_no == 14: 425 | args.cot_trigger = "Let's be realistic and think step by step." 426 | else: 427 | raise ValueError("cot_trigger_no is not properly defined ...") 428 | 429 | return args 430 | 431 | 432 | if __name__ == "__main__": 433 | main() 434 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | from statistics import mean 2 | from torch.utils.data import Dataset 3 | from collections import OrderedDict 4 | import xml.etree.ElementTree as ET 5 | from transformers import T5ForConditionalGeneration as T5, AutoTokenizer 6 | import torch 7 | import openai # For GPT-3 API ... 8 | import os 9 | import multiprocessing 10 | import json 11 | import numpy as np 12 | import random 13 | import torch 14 | # import torchtext 15 | import re 16 | import random 17 | import time 18 | import datetime 19 | import pandas as pd 20 | 21 | 22 | # https://review-of-my-life.blogspot.com/2017/11/python-dict-shuffle.html 23 | def shuffleDict(d): 24 | keys = list(d.keys()) 25 | random.shuffle(keys) 26 | [(key, d[key]) for key in keys] 27 | random.shuffle(keys) 28 | [(key, d[key]) for key in keys] 29 | random.shuffle(keys) 30 | keys = [(key, d[key]) for key in keys] 31 | # keys = d(keys) 32 | return dict(keys) 33 | 34 | 35 | def fix_seed(seed): 36 | # random 37 | random.seed(seed) 38 | # Numpy 39 | np.random.seed(seed) 40 | # Pytorch 41 | torch.manual_seed(seed) 42 | torch.cuda.manual_seed_all(seed) 43 | torch.backends.cudnn.deterministic = True 44 | 45 | 46 | def print_now(return_flag=0): 47 | t_delta = datetime.timedelta(hours=9) 48 | JST = datetime.timezone(t_delta, 'JST') 49 | now = datetime.datetime.now(JST) 50 | now = now.strftime('%Y/%m/%d %H:%M:%S') 51 | if return_flag == 0: 52 | print(now) 53 | elif return_flag == 1: 54 | return now 55 | else: 56 | pass 57 | 58 | 59 | # Sentence Generator (Decoder) for GPT-3 ... 60 | def decoder_for_gpt3(args, input, max_length, i, k, n, rd,stop): 61 | # GPT-3 API allows each users execute the API within 20 times in a minute ... 62 | time.sleep(args.api_time_interval) 63 | 64 | # https://beta.openai.com/account/api-keys 65 | openai.api_key = os.getenv("OPENAI_API_KEY") 66 | # print(openai.api_key) 67 | 68 | # Specify engine ... 69 | # Instruct GPT3 70 | if args.model == "gpt3": 71 | engine = "text-ada-001" 72 | elif args.model == "gpt3-medium": 73 | engine = "text-babbage-001" 74 | elif args.model == "gpt3-large": 75 | engine = "text-curie-001" 76 | elif args.model == "gpt3-xl": 77 | engine = "text-davinci-002" 78 | elif args.model == "codex": 79 | engine = "code-davinci-002" 80 | elif args.model == "codex-001": 81 | engine = "code-davinci-001" 82 | elif args.model == "text-003": 83 | engine = "text-davinci-003" 84 | else: 85 | raise ValueError("model is not properly defined ...") 86 | 87 | response = openai.Completion.create( 88 | engine=engine, 89 | prompt=input, 90 | max_tokens=max_length, 91 | temperature=k, 92 | stop=stop, 93 | n=n 94 | ) 95 | 96 | return [text["text"] for text in response["choices"]] 97 | 98 | 99 | class Decoder(): 100 | def __init__(self, args): 101 | print_now() 102 | if args.model == 'UL2': 103 | self.model = T5.from_pretrained("google/ul2", low_cpu_mem_usage=True, torch_dtype=torch.bfloat16, 104 | cache_dir='/home/zmj/CoT/UL2/weight') 105 | self.tokenizer = AutoTokenizer.from_pretrained("google/ul2") 106 | device_map = { 107 | 6: [0, 1, 2, 3, 4], 108 | 7: [5, 6, 7, 8, 9, 10, 11, 12, 13, ], 109 | 8: [14, 15, 16, 17, 18, 19, 20, 21, 22, ], 110 | 9: [23, 24, 25, 26, 27, 28, 29, 30, 31], 111 | } 112 | self.model.parallelize(device_map) 113 | else: 114 | self.rd = 0 115 | 116 | def decode(self, args, input, max_length, i, k, n,stop,is_turn_to_declarative=False): 117 | try: 118 | if args.model in ("gpt3", "gpt3-medium", "gpt3-large", "gpt3-xl", "codex", "codex-001","text-003"): 119 | response = decoder_for_gpt3(args, input, max_length, i, k, n, self.rd,stop) 120 | if self.rd != 3: 121 | self.rd += 1 122 | else: 123 | self.rd = 1 124 | elif args.model in ("UL2"): 125 | if is_turn_to_declarative: 126 | input_ids = self.tokenizer('[S2S] ' + input[81:] + '\" \"', return_tensors="pt").input_ids.to(6) 127 | else: 128 | input_ids = self.tokenizer('[S2S] ' + input + '', return_tensors="pt").input_ids.to(6) 129 | output = self.tokenizer.batch_decode( 130 | self.model.generate(input_ids, temperature=k, do_sample=True, top_k=10, num_return_sequences=n, 131 | max_length=args.max_length_cot)) 132 | response = [i.replace(' ', '').replace('', '').replace('', '').split(' 1 else False 332 | pred = preds[-1] 333 | 334 | if args.dataset in ("aqua", "commonsensqa"): 335 | pred = re.findall(r'A|B|C|D|E', pred) 336 | elif args.dataset == "bigbench_date": 337 | pred = [pred[1:-1]] 338 | elif args.dataset in ("object_tracking"): 339 | pred = re.findall(r'A|B|C', pred) 340 | elif args.dataset in ("gsm8k", "addsub", "multiarith", "svamp", "singleeq"): 341 | pred = pred.replace(",", "") 342 | pred = [s for s in re.findall(r'-?\d+\.?\d*', pred)] 343 | elif args.dataset in ("strategyqa", "coin_flip"): 344 | pred = pred.lower() 345 | pred = re.sub("\"|\'|\n|\.|\s|\:|\,", " ", pred) 346 | pred = pred.split(" ") 347 | pred = [i for i in pred if i in ("yes", "no")] 348 | elif args.dataset == "last_letters": 349 | pred = re.sub("\"|\'|\n|\.|\s", "", pred) 350 | pred = [pred] 351 | else: 352 | raise ValueError("dataset is not properly defined ...") 353 | 354 | # If there is no candidate in list, null is set. 355 | if len(pred) == 0: 356 | pred = "" 357 | else: 358 | if args.method in ("few_shot", "few_shot_cot", "verifier","verifier_cot",'verifier_TF_cot',"zero_shot_verifier_cot"): 359 | if answer_flag: 360 | # choose the first element in list ... 361 | pred = pred[0] 362 | else: 363 | # choose the last element in list ... 364 | pred = pred[-1] 365 | elif args.method in ("zero_shot", "zero_shot_cot"): 366 | # choose the first element in list ... 367 | pred = pred[0] 368 | else: 369 | raise ValueError("method is not properly defined ...") 370 | 371 | # (For arithmetic tasks) if a word ends with period, it will be omitted ... 372 | if pred != "": 373 | if pred[-1] == ".": 374 | pred = pred[:-1] 375 | 376 | return pred 377 | 378 | 379 | def answer_cleansing_verifier(args, pred): 380 | if args.method in ("few_shot", "few_shot_cot",'verifier', 'verifier_cot','verifier_TF_cot', "zero_shot_verifier_cot"): 381 | preds = pred.split(args.direct_answer_trigger_for_fewshot) 382 | answer_flag = True if len(preds) > 1 else False 383 | pred = preds[-1] 384 | 385 | if args.dataset in ("aqua"): 386 | pred = re.findall(r'A|B|C|D|E', pred) 387 | elif args.dataset in ("object_tracking"): 388 | pred = re.findall(r'A|B|C', pred) 389 | elif args.dataset in ("gsm8k", "addsub", "multiarith", "svamp", "singleeq", "bigbench_date"): 390 | pred = pred.replace(",", "") 391 | pred = [s for s in re.findall(r'-?\d+\.?\d*', pred)] 392 | elif args.dataset in ("strategyqa", "coin_flip", "commonsensqa"): 393 | pred = pred.lower() 394 | pred = re.sub("\"|\'|\n|\.|\s|\:|\,", " ", pred) 395 | pred = pred.split(" ") 396 | pred = [i for i in pred if i in ("yes", "no")] 397 | elif args.dataset == "last_letters": 398 | pred = re.sub("\"|\'|\n|\.|\s", "", pred) 399 | pred = [pred] 400 | else: 401 | raise ValueError("dataset is not properly defined ...") 402 | 403 | # If there is no candidate in list, null is set. 404 | if len(pred) == 0: 405 | pred = "" 406 | else: 407 | if args.method in ("few_shot", "few_shot_cot",'verifier', "verifier_cot",'verifier_TF_cot', "zero_shot_verifier_cot"): 408 | if answer_flag: 409 | # choose the first element in list ... 410 | pred = pred[0] 411 | else: 412 | # choose the last element in list ... 413 | pred = pred[-1] 414 | elif args.method in ("zero_shot", "zero_shot_cot"): 415 | # choose the first element in list ... 416 | pred = pred[0] 417 | else: 418 | raise ValueError("method is not properly defined ...") 419 | 420 | # (For arithmetic tasks) if a word ends with period, it will be omitted ... 421 | if pred != "": 422 | if pred[-1] == ".": 423 | pred = pred[:-1] 424 | 425 | return pred 426 | 427 | 428 | def create_demo_text(args, cot_flag): 429 | x, z, y, c = [], [], [], [] 430 | 431 | # example sentences ... 432 | if args.dataset in ("multiarith", "gsm8k", "addsub", "svamp", "singleeq"): 433 | 434 | x.append( 435 | "There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?") 436 | z.append( 437 | "There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6.") 438 | y.append("6") 439 | 440 | x.append("If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?") 441 | z.append("There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5.") 442 | y.append("5") 443 | 444 | x.append( 445 | "Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?") 446 | z.append( 447 | "Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 + 42 = 74. After eating 35, they had 74 - 35 = 39.") 448 | y.append("39") 449 | 450 | x.append( 451 | "Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?") 452 | z.append( 453 | "Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny 20 - 12 = 8.") 454 | y.append("8") 455 | 456 | x.append( 457 | "Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?") 458 | z.append( 459 | "Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9.") 460 | y.append("9") 461 | 462 | x.append( 463 | "There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?") 464 | z.append( 465 | "There were originally 9 computers. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. 9 + 20 is 29.") 466 | y.append("29") 467 | 468 | x.append( 469 | "Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?") 470 | z.append( 471 | "Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 - 23 = 35. After losing 2 more, he had 35 - 2 = 33 golf balls.") 472 | y.append("33") 473 | 474 | x.append("Olivia has $23. She bought five bagels for $3 each. How much money does she have left?") 475 | z.append( 476 | "Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars. So she has 23 - 15 dollars left. 23 - 15 is 8.") 477 | y.append("8") 478 | 479 | elif args.dataset in ("aqua"): 480 | x.append( 481 | "John found that the average of 15 numbers is 40. If 10 is added to each number then the mean of the numbers is? ") 482 | c.append( 483 | "Answer Choices: (A) 50 (B) 45 (C) 65 (D) 78 (E) 64") 484 | z.append( 485 | "If 10 is added to each number, then the mean of the numbers also increases by 10. So the new mean would be 50.") 486 | y.append("A") 487 | 488 | x.append("If a / b = 3/4 and 8a + 5b = 22, then find the value of a. ") 489 | c.append("Answer Choices: (A) 1/2 (B) 3/2 (C) 5/2 (D) 4/2 (E) 7/2") 490 | z.append( 491 | "If a / b = 3/4, then b = 4a / 3. So 8a + 5(4a / 3) = 22. This simplifies to 8a + 20a / 3 = 22, which means 44a / 3 = 22. So a is equal to 3/2.") 492 | y.append("B") 493 | 494 | x.append( 495 | "A person is traveling at 20 km/hr and reached his destiny in 2.5 hr then find the distance? ") 496 | c.append( 497 | "Answer Choices: (A) 53 km (B) 55 km (C) 52 km (D) 60 km (E) 50 km") 498 | z.append("The distance that the person traveled would have been 20 km/hr * 2.5 hrs = 50 km.") 499 | y.append("E") 500 | 501 | x.append( 502 | "How many keystrokes are needed to type the numbers from 1 to 500? ") 503 | c.append("Answer Choices: (A) 1156 (B) 1392 (C) 1480 (D) 1562 (E) 1788") 504 | z.append( 505 | "There are 9 one-digit numbers from 1 to 9. There are 90 two-digit numbers from 10 to 99. There are 401 three-digit numbers from 100 to 500. 9 + 90(2) + 401(3) = 1392.") 506 | y.append("B") 507 | 508 | elif args.dataset in ("commonsensqa"): 509 | x.append( 510 | "What do people use to absorb extra ink from a fountain pen? ") 511 | c.append("Answer Choices: (A) shirt pocket (B) calligrapher's hand (C) inkwell (D) desk drawer (E) blotter") 512 | z.append( 513 | "The answer must be an item that can absorb ink. Of the above choices, only blotters are used to absorb ink.") 514 | y.append("E") 515 | 516 | x.append( 517 | "What home entertainment equipment requires cable? ") 518 | c.append("Answer Choices: (A) radio shack (B) substation (C) television (D) cabinet") 519 | z.append( 520 | "The answer must require cable. Of the above choices, only television requires cable.") 521 | y.append("C") 522 | 523 | x.append( 524 | "The fox walked from the city into the forest, what was it looking for? ") 525 | c.append("Answer Choices: (A) pretty flowers (B) hen house (C) natural habitat (D) storybook") 526 | z.append( 527 | "The answer must be something in the forest. Of the above choices, only natural habitat is in the forest.") 528 | y.append("C") 529 | 530 | x.append( 531 | "Sammy wanted to go to where the people were. Where might he go? ") 532 | c.append("Answer Choices: (A) populated areas (B) race track (C) desert (D) apartment (E) roadblock") 533 | z.append( 534 | "The answer must be a place with a lot of people. Of the above choices, only populated areas have a lot of people.") 535 | y.append("A") 536 | 537 | x.append( 538 | "Where do you put your grapes just before checking out? ") 539 | c.append("Answer Choices: (A) mouth (B) grocery cart (C) super market (D) fruit basket (E) fruit market") 540 | z.append( 541 | "The answer should be the place where grocery items are placed before checking out. Of the above choices, grocery cart makes the most sense for holding grocery items.") 542 | y.append("B") 543 | 544 | x.append( 545 | "Google Maps and other highway and street GPS services have replaced what? ") 546 | c.append("Answer Choices: (A) united states (B) mexico (C) countryside (D) atlas") 547 | z.append( 548 | "The answer must be something that used to do what Google Maps and GPS services do, which is to give directions. Of the above choices, only atlases are used to give directions.") 549 | y.append("D") 550 | 551 | x.append( 552 | "Before getting a divorce, what did the wife feel who was doing all the work? ") 553 | c.append("Answer Choices: (A) harder (B) anguish (C) bitterness (D) tears (E) sadness") 554 | z.append( 555 | "The answer should be the feeling of someone getting divorced who was doing all the work. Of the above choices, the closest feeling is bitterness.") 556 | y.append("C") 557 | 558 | elif args.dataset in ("bigbench_date"): 559 | x.append( 560 | "2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?") 561 | z.append( 562 | "If 2015 is coming in 36 hours, then it is coming in 2 days. 2 days before 01/01/2015 is 12/30/2014, so today is 12/30/2014. So one week from today will be 01/05/2015.") 563 | y.append("01/05/2015") 564 | 565 | x.append( 566 | "The first day of 2019 is a Tuesday, and today is the first Monday of 2019. What is the date today in MM/DD/YYYY?") 567 | z.append( 568 | "If the first day of 2019 was Tuesday, then 01/01/2019 was a Tuesday. Today is the first monday, would be six days later. So today is 01/07/2019.") 569 | y.append("01/07/2019") 570 | 571 | x.append( 572 | "The concert was scheduled to be on 06/01/1943, but was delayed by one day to today. What is the date 10 days ago in MM/DD/YYYY?") 573 | z.append( 574 | "One day after 06/01/1943 is 06/02/1943, so today is 06/02/1943. 10 days before today is 05/23/1943.") 575 | y.append("05/23/1943") 576 | 577 | x.append( 578 | "It is 4/19/1969 today. What is the date 24 hours later in MM/DD/YYYY?") 579 | z.append( 580 | "Today is 04/19/1969. 24 hours later is one day after today, which would be 04/20/1969.") 581 | y.append("04/20/1969") 582 | 583 | x.append( 584 | "Jane thought today is 3/11/2002, but today is in fact Mar 12, which is 1 day later. What is the date 24 hours later in MM/DD/YYYY?") 585 | z.append( 586 | "Today is 03/12/2002. So the date 24 hours later will be 03/13/2002") 587 | y.append("03/13/2002") 588 | 589 | x.append( 590 | "Jane was born on the last day of Feburary in 2001. Today is her 16-year-old birthday. What is the date yesterday in MM/DD/YYYY?") 591 | z.append( 592 | "The last day of February is the 28th, so Jane was born on 02/28/2001. Today is her 16-year old birthday, so today is 02/28/2017. So yesterday was 02/27/2017. ") 593 | y.append("02/27/2017") 594 | 595 | else: 596 | raise ValueError("dataset is not properly defined ...") 597 | 598 | # randomize order of the examples ... 599 | index_list = list(range(len(x))) 600 | random.shuffle(index_list) 601 | if args.FN != 0: 602 | index_list = index_list[:args.FN] 603 | 604 | # Concatenate demonstration examples ... 605 | demo_text = "" 606 | for i in index_list: 607 | if len(c) > 0: 608 | if cot_flag: 609 | demo_text += "Q: " + x[i] + c[i] + "\nA: " + z[i] + " " + \ 610 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 611 | else: 612 | demo_text += "Q: " + x[i] + c[i] + "\nA: " + \ 613 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 614 | if args.dataset in ('meddialog'): 615 | if cot_flag: 616 | demo_text += x[i] + "\nRecord Report: " + z[i] + "\n" + \ 617 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 618 | else: 619 | demo_text += x[i] + "\n" + \ 620 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 621 | else: 622 | if cot_flag: 623 | demo_text += "Q: " + x[i] + "\nA: " + z[i] + " " + \ 624 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 625 | else: 626 | demo_text += "Q: " + x[i] + "\nA: " + \ 627 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 628 | 629 | return demo_text 630 | 631 | 632 | 633 | def create_verifier_demo_text(args, cot_flag): 634 | x, z, y = [], [], [] 635 | 636 | # example sentences ... 637 | if args.dataset in ("multiarith", "gsm8k", "addsub", "svamp", "singleeq"): 638 | 639 | x.append( 640 | "\"There are 'X' trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. The grove workers planted 6 trees today.\" What is the answer of 'X'?") 641 | z.append( 642 | "There are X trees originally. The grove workers planted 6 trees today. Then there were 21 trees after some more were planted. So, we can write the following equation:\nX + 6 = 21\n X = 21 - 6\n X = 15\n\n") 643 | y.append("15") 644 | 645 | x.append( 646 | "\"If there are 'X' cars in the parking lot and 2 more cars arrive, There are 5 cars in the parking lot.\" What is the answer of 'X'?") 647 | z.append( 648 | "There are originally X cars. 2 more cars arrive and there are 5 cars finally. So:\n X + 2 = 5\n X = 5 - 2\n X = 3\n\n") 649 | y.append("3") 650 | 651 | x.append( 652 | "\"Leah had 'X' chocolates and her sister had 42. If they ate 35, they have 39 pieces left in total.\" What is the answer of 'X'?") 653 | z.append( 654 | "Originally, Leah had X chocolates. Her sister had 42. So in total they had:\n X + 42 = Y\n\n. After eating 35, they had 39, so\n Y = 35 + 39\n Y = 74\n X + 42 = 74\n X = 74 - 42\n X = 32\n\n") 655 | y.append("32") 656 | 657 | x.append( 658 | "\"Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 'X' lollipops. Jason gave Denny 8 lollipops.\" What is the answer of 'X'?") 659 | z.append( 660 | "Jason started with 20 lollipops. Then he had X after giving some to Denny and gave Denny 8.\n 20 - X = 8\n X = 12\n\n") 661 | y.append("12") 662 | 663 | x.append( 664 | "\"Shawn has 'X' toys. For Christmas, he got two toys each from his mom and dad. He has 9 toys now.\" What is the answer of 'X'?") 665 | z.append( 666 | "Shawn started with X toys. If he got 2 toys each from his mom and dad, then that is 4 more toys.\n X + 4 = 9\n X = 9 - 4\n X = 5\n\n") 667 | y.append("5") 668 | 669 | x.append( 670 | "\"There were 'X' computers in the server room. Five more computers were installed each day, from monday to thursday. There are 29 computers in the server room.\" What is the answer of 'X'?") 671 | z.append( 672 | "There were originally X computers. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. And there are 29 computers.\n X + 20 = 29\n X = 29 - 20\n X = 9\n\n") 673 | y.append("9") 674 | 675 | x.append( 676 | "\"Michael had 58 golf balls. On tuesday, he lost 'X' golf balls. On wednesday, he lost 2 more. He had 33 golf balls at the end of Wednesday.\" What is the answer of 'X'?") 677 | z.append( 678 | "Michael started with 58 golf balls. After losing X on tuesday and he lost 2 more on wednesday, He had 33 golf balls. So, we can write the following equation:\n 58 - X - 2 = 33\n 58 - X = 35\n X = 23\n\n") 679 | y.append("23") 680 | 681 | x.append( 682 | "\"Olivia has $'X'. She bought five bagels for $3 each. She has 8 dollars left.\" What is the answer of 'X'?") 683 | z.append( 684 | "Olivia had X dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars. She has 8 dollars left finally.\n X - 15 = 8\n X = 8 + 15\n X = 23\n\n") 685 | y.append("23") 686 | 687 | elif args.dataset in ("aqua"): 688 | x.append( 689 | "\"John found that the average of 15 numbers is 'X'. If 10 is added to each number then the mean of the numbers is 50.\" What is the answer of 'X'?") 690 | z.append( 691 | "If 10 is added to each number, then the mean of the numbers also increases by 10. The new mean would be 50.\n X + 10 = 50\n X = 40\n\n") 692 | y.append("40") 693 | 694 | x.append("\"If a / b = 'X' and 8a + 5b = 22, then the value of a is 3/2.\" What is the answer of 'X'?") 695 | z.append( 696 | "If a / b = X, then 8a + 5b = 22 and a = 3/2, so \n 8 * 3/2 + 5b = 22\n 5b = 22 - 12 = 10\n b = 2\n X = a / b = 3/2 / 2 = 3/4\n\n") 697 | y.append("3/4") 698 | 699 | x.append( 700 | "\"A person is traveling at 'X' km/hr and reached his destiny in 2.5 hr then find the distance is 50km.\" What is the answer of 'X'?") 701 | z.append("The distance that the person traveled would have been \n X km/hr * 2.5 hrs = 50 k\n X = 20\n\n") 702 | y.append("20") 703 | 704 | x.append( 705 | "\"There were 'X' computers in the server room. Five more computers were installed each day, from monday to thursday. There are 29 computers in the server room.\" What is the answer of 'X'?") 706 | z.append( 707 | "There were originally X computers. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. And there are 29 computers.\n X + 20 = 29\n X = 29 - 20\n X = 9\n") 708 | y.append("9") 709 | 710 | elif args.dataset in ("commonsensqa"): 711 | x.append( 712 | "\"People use blotter to absorb extra ink from a fountain pen.\" Judge whether this statement is normal (yes or no).") 713 | z.append( 714 | "The Blotter is used to absorb extra ink from a fountain pen.") 715 | y.append("Yes") 716 | 717 | x.append( 718 | "\"Television requires cable.\" Judge whether this statement is normal (yes or no).") 719 | z.append( 720 | "The Television is an electrical appliance, it needs electricity, so it requires cables") 721 | y.append("Yes") 722 | 723 | x.append( 724 | "\"The fox walked from the city into the forest, it was looking for a hen house.\" Judge whether this statement is normal (yes or no).") 725 | z.append( 726 | "The hen house is not in the forest, so the fox does not go to the hen house.") 727 | y.append("No") 728 | 729 | x.append( 730 | "\"Sammy wanted to go to where the people were. He might go populated areas.\" Judge whether this statement is normal (yes or no).") 731 | z.append( 732 | "There are many people in the populated areas, so they really go here.") 733 | y.append("Yes") 734 | 735 | x.append( 736 | "\"The grapes are put in the fruit market just before checking out.\" Judge whether this statement is normal (yes or no).") 737 | z.append( 738 | "The answer should be the place where grocery items are placed before checking out. But the fruit market is not suitable place where grocery items are placed.") 739 | y.append("No") 740 | 741 | x.append( 742 | "\"Google Maps and other highway and street GPS services have replaced the united states.\" Judge whether this statement is normal (yes or no).") 743 | z.append( 744 | "The united states is a country and Google Maps is a map, so Google Maps cannot replace the united states") 745 | y.append("No") 746 | 747 | x.append( 748 | "\"The wife who was doing all the work felt bitterness before getting a divorce.\" Judge whether this statement is normal (yes or no).") 749 | z.append( 750 | "The wife divorced who was doing all the work. So she felt bitterness.") 751 | y.append("Yes") 752 | 753 | elif args.dataset in ("bigbench_date"): 754 | x.append( 755 | "\"'X' is coming in 36 hours. One week from today is 01/05/2015.\" What is the answer of 'X'?") 756 | z.append( 757 | "If The date one week from today is 01/05/2015, so today is 12/30/2014. So the data after 36 hours is 2015.") 758 | y.append("2015") 759 | 760 | x.append( 761 | "\"The first day of 'X' is a Tuesday, and today is the first Monday of 2019. Today is 01/07/2019.\" What is the answer of 'X'?") 762 | z.append( 763 | "If today is the first Monday of 2019 and today is 01/07/2019. So The first day of 2019 is a Tuesday.") 764 | y.append("2019") 765 | 766 | x.append( 767 | "\"The concert was scheduled to be on 'X'/01/1943, but was delayed by one day to today. 10 days ago is 05/23/1943.\" What is the answer of 'X'?") 768 | z.append( 769 | "10 days ago is 05/23/1943, and the concert was delayed by one day to today, so today is 06/02/1943. So the concert was scheduled to be on 06/01/1943") 770 | y.append("06") 771 | 772 | x.append( 773 | "\"It is ’X'/19/1969 today. 24 hours later is 04/20/1969.\" What is the answer of 'X'?") 774 | z.append( 775 | "24 hours later is 04/20/1969. So today is 04/19/1969.") 776 | y.append("04") 777 | 778 | x.append( 779 | "\"Jane thought today is 'X'/12/2002, but today is in fact Mar 12, which is 1 day later. 24 hours later is 03/13/2002.\" What is the answer of 'X'?") 780 | z.append( 781 | "24 hours later is 03/13/2002. So today is 03/12/2002.") 782 | y.append("03") 783 | 784 | x.append( 785 | "\"Jane was born on the last day of Feburary in 'X'. Today is her 16-year-old birthday. Yesterday is 02/27/2017\" What is the answer of 'X'?") 786 | z.append( 787 | "Yesterday is 02/27/2017, so today is 02/28/2017, Jane was born on 02/28/2001.") 788 | y.append("2001") 789 | 790 | else: 791 | raise ValueError("dataset is not properly defined ...") 792 | 793 | # randomize order of the examples ... 794 | index_list = list(range(len(x))) 795 | random.shuffle(index_list) 796 | if args.FN != 0: 797 | index_list = index_list[:args.FN] 798 | # Concatenate demonstration examples ... 799 | demo_text = "" 800 | for i in index_list: 801 | if cot_flag: 802 | demo_text += "Q: " + x[i] + "\nA: " + z[i] + " " + \ 803 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 804 | else: 805 | demo_text += "Q: " + x[i] + "\nA: " + \ 806 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 807 | 808 | return demo_text 809 | 810 | 811 | def question_turn_decalrative(args, text, answer, answers_0, function, declarative): 812 | global new_question 813 | if 'Answer Choices' in text: 814 | text = text.split('Answer Choices')[0] 815 | try: 816 | if args.dataset in ("commonsensqa"): 817 | text = text.replace(',', '.') 818 | position_fullstop = text[::-1].find('.') 819 | 820 | question = text[len(text) - position_fullstop:] 821 | ts = text[:len(text) - position_fullstop] 822 | 823 | if ts[0] == ' ': 824 | ts = ts[1:] 825 | if ts[-1] != ' ': 826 | ts += ' ' 827 | ts = ts.replace(' .', '.') 828 | if args.model == 'UL2': 829 | return ts, 'yes', "'{} The answer is {}' If the question and answer are changed into fluent declarative sentences: ".format( 830 | question, answer) 831 | else: 832 | return ts, 'yes', "Q: Please change the questions and answers into a complete declarative sentences '{} The answer is {}'\nA: ".format( 833 | question, answer) 834 | 835 | text = text.replace(',', '.') 836 | position_fullstop = text[::-1].find('.') 837 | 838 | question = text[len(text) - position_fullstop:] 839 | ts = text[:len(text) - position_fullstop] 840 | if args.dataset in ('bigbench_date'): 841 | declarative = question[17:-15] + ' is ' + answer + '.' 842 | else: 843 | if declarative == '': 844 | try: 845 | declarative = function(args, 846 | "Q: Please change the questions and answers into a complete declarative sentences '{} The answer is {}'\nA: ".format( 847 | question, answer), args.max_length_cot, 0, 0, 1,'\n',is_turn_to_declarative=True)[0] 848 | except: 849 | 850 | declarative = function(args, 851 | "Q: Please change the questions and answers into a complete declarative sentences '{} The answer is {}'\nA: ".format( 852 | question, answer), args.max_length_cot, 0, 0, 1,'\n',is_turn_to_declarative=True)[0] 853 | 854 | else: 855 | if answers_0 in declarative: 856 | declarative = declarative[:len(declarative) - declarative[::-1].find(answers_0[::-1]) - len( 857 | answers_0)] + answer + declarative[len(declarative) - declarative[::-1].find(answers_0[::-1]):] 858 | else: 859 | try: 860 | declarative = function(args, 861 | "Q: Please change the questions and answers into a complete declarative sentences '{} The answer is {}'\nA: ".format( 862 | question, answer), args.max_length_cot, 0, 0, 1,'\n',is_turn_to_declarative=True)[0] 863 | except: 864 | declarative = "{} The answer is {}.".format(question, answer) 865 | 866 | new_question_number = [s for s in re.findall(r'-?\d+\.?\d*', ts)] 867 | 868 | sentences, ans = [], [] 869 | for nqn in range(len(new_question_number)): 870 | new_ts = '' 871 | number_find = False 872 | for i in ts.split('.'): 873 | if new_question_number[nqn] in i and number_find == False: 874 | new_question = [p for p in i] 875 | new_question[ 876 | i.find(new_question_number[nqn]):i.find(new_question_number[nqn]) + len(new_question_number[nqn])] = "'X'" 877 | new_question = ''.join(new_question) + '.' 878 | new_question.replace(' .', '.') 879 | new_ts += new_question 880 | else: 881 | new_ts += i + '.' 882 | new_ts = new_ts.replace('..', '.') 883 | 884 | if new_ts[0] == ' ': 885 | new_ts = new_ts[1:] 886 | if new_ts[-1] != ' ': 887 | new_ts += ' ' 888 | new_ts = new_ts.replace(' .', '.') 889 | 890 | sentences.append('"' + new_ts + declarative + '"' + args.verifier_text) 891 | ans.append(new_question_number[nqn]) 892 | return sentences[:3], ans[:3],declarative 893 | 894 | except: 895 | 896 | return '', '', '' 897 | 898 | 899 | def create_verifier_demo_text_TF(args, cot_flag): 900 | x, z, y = [], [], [] 901 | 902 | # example sentences ... 903 | if args.dataset in ("multiarith", "gsm8k", "addsub", "svamp", "singleeq"): 904 | 905 | x.append( 906 | "\'There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. The grove workers planted 4 trees today.\' Do it is correct (True or False)?") 907 | z.append( 908 | "If the Grove workers will plant 4 trees today and there will be 21 trees after they are done. 21 - 4 = 17, there are 17 trees in the grove, but actually there are 15 trees, 17 != 15, which is different from the theme.") 909 | y.append("False") 910 | 911 | x.append( 912 | "\'If there are 3 cars in the parking lot and 2 more cars arrive, There are 5 cars in the parking lot.\' Do it is correct (True or False)?") 913 | z.append( 914 | "If there will be 5 cars in the parking lot, subtract 2 cars that will arrive, 5 - 2 = 3, so there are 2 cars in the parking lot, which is consistent with the theme.") 915 | y.append("True") 916 | 917 | x.append( 918 | "\'Leah had 32 chocolates and her sister had 42. If they ate 35, they have 39 pieces left in total.\' Do it is correct (True or False)?") 919 | z.append( 920 | "If there are 39 pieces of chocolates and 35 pieces of chocolate are eaten, Leah and her sister have 39 + 35 = 74 in total. Her sister's had 42, so Leah had 74 - 42 = 32, which is consistent with the theme.") 921 | y.append("True") 922 | 923 | x.append( 924 | "\'Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. Jason gave Denny 6 lollipops.\' Do it is correct (True or False)?") 925 | z.append( 926 | "If Jason gave Denny 6 lollipops, and Jason now has 12 lollipops, so Jason originally had 6+12=18 lollipops, 18 != 20, which is different from the theme.") 927 | y.append("False") 928 | 929 | x.append( 930 | "\'Shawn has five toys. For Christmas, he got two toys each from his mom and dad. He has 9 toys now.\' Do it is correct (True or False)?") 931 | z.append( 932 | "If Shawn now has 9 toys and his parents gaven him two each, then he originally had 9 - 2 - 2 = 5, which is consistent with the theme.") 933 | y.append("True") 934 | 935 | x.append( 936 | "\'There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. There are 18 computers in the server room.\' Do it is correct (True or False)?") 937 | z.append( 938 | "Now there are 18 computers in the server room. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. So there were 18 - 20= -2 in the server room originally, -2 != 9, which is different from the theme.") 939 | y.append("False") 940 | 941 | x.append( 942 | "\'Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. He had 40 golf balls at the end of Wednesday.\' Do it is correct (True or False)?") 943 | z.append( 944 | "If Michael had 40 golf balls on Wednesday, he had 40+2=42 on Tuesday because he lost 2 golf balls on Wednesday. Due to lost 23 balls on Tuesday, he should have 42+23=65 on Monday, but in fact Michael has 58 golf balls original, which is different from the theme.") 945 | y.append("False") 946 | 947 | x.append( 948 | "\'Olivia has $23. She bought five bagels for $3 each. She has 8 dollars left.\' Do it is correct (True or False)?") 949 | z.append( 950 | "If Olivia had $8 left and she bought five bagels for $3 each, so costs 5 * 3 = 15, so there was 8 + 15 = 23, which is consistent with the theme.") 951 | y.append("True") 952 | 953 | elif args.dataset in ("aqua"): 954 | x.append( 955 | "\"John found that the average of 15 numbers is 40. If 10 is added to each number then the mean of the numbers is 50.\" Do it is correct (True or False)?") 956 | z.append( 957 | "The new mean would be 50. The average of 15 numbers is 4, if 10 is added to each number, then the mean of the numbers also increases by 10. 50 - 40 = 10.") 958 | y.append("True") 959 | 960 | x.append("\"If a / b = 3/4 and 8a + 5b = 22, then the value of a is 3.\" Do it is correct (True or False)?") 961 | z.append( 962 | "If a is 3, a / b = 3/4, so b = 4. then 8a + 5b = 8 * 2 + 5 * 4 = 36, but 8a + 5b = 22") 963 | y.append("False") 964 | 965 | x.append( 966 | "\"A person is traveling at 20 km/hr and reached his destiny in 2.5 hr then find the distance is 65km.\" Do it is correct (True or False)?") 967 | z.append("If 65km is driven at 20km/hr, so the driving time is 65km / 20km/hr = 3.25h, but he destiny in 2.5 hr.") 968 | y.append("False") 969 | 970 | x.append( 971 | "\"There were 9 computers in the server room. Five more computers were installed each day, from monday to thursday. There are 29 computers in the server room.\" Do it is correct (True or False)?") 972 | z.append( 973 | "There are 29 computers in the server room. For each of 4 days, 5 more computers were added. 5 * 4 = 20 computers were added. So there were originally 9 computers. ") 974 | y.append("True") 975 | else: 976 | raise ValueError("dataset is not properly defined ...") 977 | 978 | # randomize order of the examples ... 979 | index_list = list(range(len(x))) 980 | random.shuffle(index_list) 981 | 982 | # Concatenate demonstration examples ... 983 | demo_text = "" 984 | for i in index_list: 985 | if cot_flag: 986 | demo_text += "Q: " + x[i] + "\nA: " + z[i] + " " + \ 987 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 988 | else: 989 | demo_text += "Q: " + x[i] + "\nA: " + \ 990 | args.direct_answer_trigger_for_fewshot + " " + y[i] + ".\n\n" 991 | 992 | return demo_text 993 | -------------------------------------------------------------------------------- /dataset/MedDialog/english-test.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "description": "i have all the symptoms except fever, i went to medicross and dr said i can get tested if i want to i'm not sure if i should. she gave me antibiotics klacid xl 500mg, she said i can take it if i feel worse i'm worried it will make immune system bad?", 4 | "utterances": [ 5 | "patient: i have all the symptoms except fever, i went to medicross and dr said i can get tested if i want to i'm not sure if i should. she gave me antibiotics klacid xl 500mg, she said i can take it if i feel worse i'm worried it will make immune system bad?", 6 | "doctor: in brief: antibiotic i don't recommend antibiotics for a simple viral upper respiratory tract infection unless examination revealed signs of acute bronchitis or sinusitis. they are not effective for viral infections like covid 19 with no bacterial lung involvement either. if you've been exposed to someone with covid 19 or or if you or someone you were exposed to travelled to a region where it was endemic, get tested would you like to video or text chat with me?" 7 | ] 8 | }, 9 | { 10 | "description": "will my mask from sherwin williams paint store with filters protect me from corona virus along with paint fumes?", 11 | "utterances": [ 12 | "patient: will my mask from sherwin williams paint store with filters protect me from corona virus along with paint fumes?", 13 | "doctor: in brief: maybe such a mask helps protect others from any respiratory material you exhale or cough but does not filter out the virus. only high end medical & industrial masks (p95/n95)would filter that. your best protection is frequent hand washing and not touching your face/mouth/nose with contaminated fingers., would you like to video or text chat with me?" 14 | ] 15 | }, 16 | { 17 | "description": "traveled 2wks ago from fl. to pa. 68 wf. has had fever of 100 , chills at night and some coughing for 5ds. tested negative for flu and x rays of lungs were clear . coronavirus? scarce", 18 | "utterances": [ 19 | "patient: traveled 2wks ago from fl. to pa. 68 wf. has had fever of 100 , chills at night and some coughing for 5ds. tested negative for flu and x rays of lungs were clear . coronavirus? scarce", 20 | "doctor: in brief: cough and cold hello and welcome to healthtap,it is a good idea to get tested. you can visit your nearest urgent care. make sure to call them before coming, after getting tested self quarantine for total 2 weeks from the onset of symptoms. please wear mask and perform frequent hand washing. avoid ibuprofen/other nsaids, use acetaminophen/ tylenol only. look for signs of worsening cough & breathing difficulties. would you like to video or text chat with me?" 21 | ] 22 | }, 23 | { 24 | "description": "are there long term repercussions of not treating pneumonia which was diagnosed 5 months ago?", 25 | "utterances": [ 26 | "patient: my 19 year old son was diagnosed with pneumonia 5 months ago and was given antibiotics but with minimal results. unfortunately he still suffers from the same symptoms but since he is in the military he refuses to go back to the doctor for fear of reprisal. his cough is still deep and his energy is low. are there any long term repercussions for his lungs if he continues without treatment.", 27 | "doctor: hello, as you explain the situation i strongly recommend you to send him to the doctor for a re-evaluation of the situation. hope i have answered your query. let me know if i can assist you further. regards, dr. jnikolla, pulmonologist" 28 | ] 29 | }, 30 | { 31 | "description": "hi there me and my 7year old have flu like symptoms and have been in contact with someone that arrived from new zealand with an undiagnosed heavy flu. should i get tested for covid-19?", 32 | "utterances": [ 33 | "patient: hi there me and my 7year old have flu like symptoms and have been in contact with someone that arrived from new zealand with an undiagnosed heavy flu. should i get tested for covid-19?", 34 | "doctor: in brief: hard to say new zealand is definitely lower risk than europe. your best bet is to stay home, with masks on (home made) follow hygiene. if you need more help ask another question here. if you have access to testing you could do it but i think we need the tests for folk who are more ill at this stage. hope this makes sense. would you like to video or text chat with me?" 35 | ] 36 | }, 37 | { 38 | "description": "if one get corona and you are self isolating and it is not severe, is there any meds that one can take?", 39 | "utterances": [ 40 | "patient: if one get corona and you are self isolating and it is not severe, is there any meds that one can take?", 41 | "doctor: in brief: not really keep immunity up, keep mental health strong, no specific meds at this point in time would you like to video or text chat with me?" 42 | ] 43 | }, 44 | { 45 | "description": "are frequent nose bleeds and sever cough signs of pneumonia?", 46 | "utterances": [ 47 | "patient: i've been recently having more nose bleeds lately. 4-5 times per day to be exact. and i have the worst cough ever that comes with it. even when i am asleep my nose starts to bleed and i wake up due to the blood running down my mouth. is this pneumonia?! or some other illness?", 48 | "doctor: thanks for your question on healthcare magic.i can understand your concern. pneumonia is less likely to cause nose bleeding. recurrent nose bleeding with coughing are commonly seen with sinusitis. so better to consult ent doctor and get done clinical examination and ct scan of paranasal sinuses (pns). also get done chest x ray for pneumonia. if chest x ray is normal then no need to worry for pneumonia. you may also need endoscopic evaluation of sinuses if ct pns is suggestive of infection or erosion.don't worry, you will be alright with all these. hope i have solved your query. i will be happy to help you further. wish you good health. thanks." 49 | ] 50 | }, 51 | { 52 | "description": "hi, last night i just started getting fluey; my nose got really runny and it\u2019s continued to today. i\u2019ve also started getting a slight headache as well. with this panic about the covid-19 should does how i feel warrant me getting tested for it?", 53 | "utterances": [ 54 | "patient: hi, last night i just started getting fluey; my nose got really runny and it\u2019s continued to today. i\u2019ve also started getting a slight headache as well. with this panic about the covid-19 should does how i feel warrant me getting tested for it?", 55 | "doctor: no testing yet. it is likely a common cold that you have.if your symptoms get worse eg. fever sore throat and painfull cough you can consult your doctor for an evaluation. try rather to stay home and rest ." 56 | ] 57 | }, 58 | { 59 | "description": "i have a slight cough over the last few days but haven't been exposed to people or traveled to covid 19 areas. i have recently quit smoking so could that be it. what should i do?", 60 | "utterances": [ 61 | "patient: i have a slight cough over the last few days but haven't been exposed to people or traveled to covid 19 areas. i have recently quit smoking so could that be it. what should i do?", 62 | "doctor: keep non smoking. if you have no exposure history to covid 19, your cough is probably not infection. keep up the good work and do not smoke or vape." 63 | ] 64 | }, 65 | { 66 | "description": "daughter is experiencing dry coughing nasal drip. she was on amoxicillin panamol and rhineton for ear infection. that was diagnosed last week monday. i need to know if she should go for corona virus test as we travelled back from cape town last week.", 67 | "utterances": [ 68 | "patient: daughter is experiencing dry coughing nasal drip. she was on amoxicillin panamol and rhineton for ear infection. that was diagnosed last week monday. i need to know if she should go for corona virus test as we travelled back from cape town last week.", 69 | "doctor: self isolate. avoid contact, wash hands, plenty of rest and fluids, if high fever or trouble t=breathing call doctor they can advise you about testing." 70 | ] 71 | }, 72 | { 73 | "description": "good morning i have sore throat for 2 days now, this morning i woke up and my throat was a lot better just a little scratchy. my other symptoms this morning were stuffy nose and a bit on the sore side with my body?", 74 | "utterances": [ 75 | "patient: good morning i have sore throat for 2 days now, this morning i woke up and my throat was a lot better just a little scratchy. my other symptoms this morning were stuffy nose and a bit on the sore side with my body?", 76 | "doctor: likely not corona. sounds more like seasonal allergy or a mild cold. gargle with warm salt water and use meds you have at home. ask questions on this forum - don't go out unless you really really have to." 77 | ] 78 | }, 79 | { 80 | "description": "can a pneumonia shot raise your white blood cell count?", 81 | "utterances": [ 82 | "patient: last tuesday i had a flu shot and a neumonia shot. i am 68 and male. i had a very sore arm and felt bad the next day in which i had blood drawn for a test. the results showed a high white cell blood count as well as high anc and amc. could these high readings be because of the shots the day before?", 83 | "doctor: hello, * the pneumonia shot per se is not responsible for high wbc count, but as you are telling there are soreness and bad feeling at the injection site over arm, there are all possibilities that you have cellulitis or possible abscess formation at the injection site; which will recruit neutrophils at the site and cause secondary rise in wbc count. * primary relief with ice application and secondary management with systemic antibiotics and anti-inflammatory medicines with your doc. hope i have answered your query. let me know if i can assist you further. take care regards, dr bhagyesh v. patel, general surgeon" 84 | ] 85 | }, 86 | { 87 | "description": "my throat is extremely scratchy and red inside, i can see it being inflamed almost to my teeth (dark pink in colour) i\u2019m very short of breathe, even as an asthmatic, i have to use my inhalers more than usual and has nebulized 3 times in the last 24 h?", 88 | "utterances": [ 89 | "patient: my throat is extremely scratchy and red inside, i can see it being inflamed almost to my teeth (dark pink in colour) i\u2019m very short of breathe, even as an asthmatic, i have to use my inhalers more than usual and has nebulized 3 times in the last 24 h?", 90 | "doctor: symptoms. please call your general physician and report your symptoms. they may want to see you, send you to be tested, or may give you other advice. please call now before symptoms worsen." 91 | ] 92 | }, 93 | { 94 | "description": "what is the recovery time for pneumonia?", 95 | "utterances": [ 96 | "patient: hi, may i answer your health queries right now ? please type your query here...my husband has a chest infection - a small pocket of pneumonia is in his left lobe. he is coughing up large long blood clots and also coming through his nose when blowing. will this settle or continue", 97 | "doctor: respected user , hithanks for using healthcaremagic.comi have evaluated your query thoroughly .* recovery time of the pneumonia vary from days to weeks depending upon - extent of pneumonia - associated co morbid conditions - causative organisms virulence - host immune response - type of treatment - smoking consumption - activity level - exposure to noxious substances as pollen or dust - diet .* so , to answer the question , lot many conditions have to be calculated and based upon that we can comment .* one thing i want to convey that it will be surely recovered with proper line of management .hope this clears your query .welcome for further doubts .regards . of management ." 98 | ] 99 | }, 100 | { 101 | "description": "why do i get runny nose and tickling sensation after getting treatment for pneumonia?", 102 | "utterances": [ 103 | "patient: i was diagnosed with pneumonia 9 days ago. i had been sick a week before that. i get better, then it seems to come back. now i have a runny nose (which i didn t have before) and i still have a tickle. 1 day good, 1 day bad, 1 day good. etc. why the runny nose? what does the tickle signify? i am a singer. got get well by sat. this is tues.", 104 | "doctor: hello, as you explain the history you don't have to worry you are in the phase of improvement just so correctly the treatment. hope i have answered your query. let me know if i can assist you further. regards, dr. jnikolla, pulmonologist" 105 | ] 106 | }, 107 | { 108 | "description": "do severe headache, tingling head and nose, pain in lower chest and chills mean pneumonia?", 109 | "utterances": [ 110 | "patient: hello, im a female of 26, heavy smoker with last breath reading of 33 and drink daily. on thursday i suffered from a severe headache which stayed until friday evening, as the headache subsided i had a tingling and pressure sensation over head which was also over my nose. i have had no fever, cold etc yesterday i felt very lightheaded and also after an episode of diarrea this morning am lightheaded again... i have no trouble breathing as such but i suffer from anxiety which i think has been happening alot today. i have had some twinge type pains in my lower chest ( like at the bottom of ribs) for a few days but not necessarily when i breath in and out and when i do cough i have had mucus? come up, it s almost like it s stuck in my throat also.... i have some head pressure and am getting cold feet every now and then everything i ve checked says pneumonia does this sound like what i could have or could it just be anxiety, i m a little worried...the symptoms separately come and go but my head stays feeling.a little heavy! chills now and again", 111 | "doctor: hello,according to history, yes, it might be pneumonia, but you should do a chest x-ray. hope i have answered your query. let me know if i can assist you further.regards,dr. jnikolla" 112 | ] 113 | }, 114 | { 115 | "description": "do you have any idea how long novel coronavirus stays on cash money?", 116 | "utterances": [ 117 | "patient: do you have any idea how long novel coronavirus stays on cash money?", 118 | "doctor: uncertain. the exact amount of time would depend on the inoculum or how much virus was deposited and whether the cash were dry or moist . the limited studies suggest that the virus does not live very long on cardboard - about 4 hrs or so, so i suspect paper may be similar. if it were moist or wet, it may last longer. heat may inactivate the virus, so consider setting the money in the sun or put in a drier." 119 | ] 120 | }, 121 | { 122 | "description": "and i travelled to the uk (back in rsa on the 1st or march but the symptoms only appeared thursday 19 march with mild ear ache to severe in the evening. followed by soar throat, cougth and body ache on friday, fever and weekness on saturday?", 123 | "utterances": [ 124 | "patient: and i travelled to the uk (back in rsa on the 1st or march but the symptoms only appeared thursday 19 march with mild ear ache to severe in the evening. followed by soar throat, cougth and body ache on friday, fever and weekness on saturday?", 125 | "doctor: in brief: infection influenza versus coronavirus. typically more sudden onset of fever and myalgias with influenza, but sometimes hard to tell. if no significant sob probably need to quarantine. call your local physician or health department. i know healthtap has been helping to order or recommend test, but typically test not available. if coronavirus most cases have been mild and expect about 10 days of symptoms. would you like to video or text chat with me?" 126 | ] 127 | }, 128 | { 129 | "description": "what are the symptoms of pneumonia?", 130 | "utterances": [ 131 | "patient: diagnosed with type b flu on sunday, getting worse even with meds, now have rattling breathing and cough, only slight fevers that come and go, chest hurts, throat hurts, very weak, no appetite. and almost 33 weeks pregnant... could this be the sign of pneumonia coming on?", 132 | "doctor: thanks for your question on healthcare magic.i can understand your concern. yes, your symptoms are suggestive of pneumonia more. pneumonia (lower respiratory tract infection) is common after upper respiratory tract infection (urti).worsening cough, chest congestion, rattling sound from chest, weakness, loss of appetite etc are characteristic symptoms of pneumonia. so better to consult pulmonologist and get done clinical examination of respiratory system and chest x ray (with abdominal shield).you should definitely start higher antibiotics to recover faster as you are pregnant. hope i have solved your query. i will be happy to help you further. wish you good health. thanks." 133 | ] 134 | }, 135 | { 136 | "description": "i read that people taking steroids for asthma are at higher risk for poor outcomes from covid19. does this include inhalers containing steroids? also, is mild intermittent asthma a risk for poor outcomes or only more severe forms of asthma? thanks", 137 | "utterances": [ 138 | "patient: i read that people taking steroids for asthma are at higher risk for poor outcomes from covid19. does this include inhalers containing steroids? also, is mild intermittent asthma a risk for poor outcomes or only more severe forms of asthma? thanks", 139 | "doctor: in brief: not quite that way people with chronic lung problems do worse in a lung disease outbreak. for one specific person... well, there's no way to guess. \"people\" in general do worse if they have uncontrolled lung disease. that means it's likely that asthmatics using inhaled steroids to give them good disease control will be better off than asthmatics who use steroids but still cannot get good control. be in good control." 140 | ] 141 | }, 142 | { 143 | "description": "i am an asthmatic but rarely need my inhaler and haven\u2019t had an attack in decades however with covid 19 i\u2019d like to put in place some measures in case the health services get overwhelmed. what drugs can i take if inhalers don\u2019t work?", 144 | "utterances": [ 145 | "patient: i am an asthmatic but rarely need my inhaler and haven\u2019t had an attack in decades however with covid 19 i\u2019d like to put in place some measures in case the health services get overwhelmed. what drugs can i take if inhalers don\u2019t work?", 146 | "doctor: get meds refilled. with the covid-19 pandemic, people with chronic conditions or even intermittent conditions should get their meds refilled in case pharmacy services close down. for asthma patients, they should refill their inhalers. some use only an urgent \"rescue\" inhaler like albuterol, while others also use \"preventive\" inhalers like fluticasone. it's best to keep both on hand. if symptoms go bad, it's er time." 147 | ] 148 | }, 149 | { 150 | "description": "is it safe to take steroid tab or nasal spray if i have covid-19?", 151 | "utterances": [ 152 | "patient: is it safe to take steroid tab or nasal spray if i have covid-19?", 153 | "doctor: in brief: nasal rinses for allergic symptoms, nasal saline and an oral antihistamine are often helpful for controlling symptoms. continue to monitor your temperature and report fever and/or shortness of breath to your pcp. would you like to video or text chat with me?" 154 | ] 155 | }, 156 | { 157 | "description": "i have type 1 diabetes that i try hard to control, but i struggle with. are my chances high for poor outcomes with the virus? i\u2019m in such a panic about it.", 158 | "utterances": [ 159 | "patient: i have type 1 diabetes that i try hard to control, but i struggle with. are my chances high for poor outcomes with the virus? i\u2019m in such a panic about it.", 160 | "doctor: depends-all factors. type i diabetics can be more fragile when they have any type of infection unfortunately. if you are younger & do not smoke & do not have other chronic problems, your increased risk may be minimal. eating balanced diet, keeping glucose controlled, getting adequate exercise (including fresh air outdoors), & adequate sleep all should help protect you from severe consequences, but be very careful ! ." 161 | ] 162 | }, 163 | { 164 | "description": "i use athraway chronic for rheumatoid arthritis, will this complicate things relating to the corona virus?", 165 | "utterances": [ 166 | "patient: i use athraway chronic for rheumatoid arthritis, will this complicate things relating to the corona virus?", 167 | "doctor: no. i do not think it will have any effect on corona virus." 168 | ] 169 | }, 170 | { 171 | "description": "last year during flu season had a severe cough, difficulty breathing and xray show there was fluid in my chest/lungs. any advice on what to do with this covid 19.", 172 | "utterances": [ 173 | "patient: last year during flu season had a severe cough, difficulty breathing and xray show there was fluid in my chest/lungs. any advice on what to do with this covid 19.", 174 | "doctor: covid. follow recommendations. flu last year and covid this year are two separate issues. at 33 your risk of sever infection is low. follow the guidelines for covid. the guidelines are as follows:https://www.healthtap.com/blog/covid-19-care-guidelines/self-quarantine-guide." 175 | ] 176 | }, 177 | { 178 | "description": "does coughing up of blood in a pneumonia patient need immediate medical attention?", 179 | "utterances": [ 180 | "patient: i have pneumonia and my doctor has me on strong antibodics and albut in my nebulizer. yesterday after a treatment i spit up fresh blood and i have this second day also. i m on blood thinners and wonder if that may be my problem. i feel better in other ways.", 181 | "doctor: helloas you explain the situation yes it might be that in pneumonia to have blood with the cough and especially if you take blood thinners do not worry.however of it last for several days than better to talk with your treating doctor for this for safety.regardsdr.jolanda" 182 | ] 183 | }, 184 | { 185 | "description": "i hear covid 19 virus dies around 27c. does it mean once temperature outside rises above 27 c all covid 19 viruses outside will be dead?", 186 | "utterances": [ 187 | "patient: i hear covid 19 virus dies around 27c. does it mean once temperature outside rises above 27 c all covid 19 viruses outside will be dead?", 188 | "doctor: not as fragile. as one might hope. -80 c does not kill it +56 c does kill it." 189 | ] 190 | }, 191 | { 192 | "description": "how do we differentiate between covid and flu? my son started with a very sore throat last night and i can\u2019t seem to get his fever under control spread through the house very quickly to everyone else", 193 | "utterances": [ 194 | "patient: how do we differentiate between covid and flu? my son started with a very sore throat last night and i can\u2019t seem to get his fever under control spread through the house very quickly to everyone else", 195 | "doctor: in brief: stay home do whatever you do with a regular flu. drink a lot of fluids, use acetaminophen/paracetamol (tylenol) for the fever and sore throat, eat/drink citrus and please stay home and away from the elders. follow the recommendation of your local authorities.only go to medical services if you feel shortness of breath and use masks if you feel sick and need to go out of your house would you like to video or text chat with me?" 196 | ] 197 | }, 198 | { 199 | "description": "colleague is covid 19 positive. i had meetings with him. instructions are to self isolate and get tested if show symptoms. i have no symptoms. is it possible to opt to get tested?", 200 | "utterances": [ 201 | "patient: colleague is covid 19 positive. i had meetings with him. instructions are to self isolate and get tested if show symptoms. i have no symptoms. is it possible to opt to get tested?", 202 | "doctor: not advised to test . the test currently used is a genetic test which test for the virus genetics. therefor if you are tested while asymptomatic, it will not test positive, even though you might actually be infected.rather stay under self quarantine and test if you show some signs." 203 | ] 204 | }, 205 | { 206 | "description": "if one has been in contact with a covid-19 positive person in the last 12-day, but am asymptomatic, what should one do to confirm if the virus has been or not contracted? don\u2019t think it is responsible to wait till symptoms manifest", 207 | "utterances": [ 208 | "patient: if one has been in contact with a covid-19 positive person in the last 12-day, but am asymptomatic, what should one do to confirm if the virus has been or not contracted? don\u2019t think it is responsible to wait till symptoms manifest", 209 | "doctor: isolate. no symptoms no test. you really do not need testing if there are no symptoms, but act like you are contagious. avoid contact with people for 14 days and only consult your gp if get symptoms." 210 | ] 211 | }, 212 | { 213 | "description": "will lysol wipes kill coronavirus? how long can coronavirus live on household surfaces?", 214 | "utterances": [ 215 | "patient: will lysol wipes kill coronavirus? how long can coronavirus live on household surfaces?", 216 | "doctor: in brief: several days lysol should be good. seehttps://www.wired.com/story/coronavirus-disinfectant-cleaning-guide/ would you like to video or text chat with me?" 217 | ] 218 | }, 219 | { 220 | "description": "i am a 46 year old female. i've had trach for the last 9 months due to stridor, bronchospasm and larageal spasms. i am currently being worked up for autoimmune disease. i am on xoliar am i more susceptible to coronavirus?", 221 | "utterances": [ 222 | "patient: i am a 46 year old female. i've had trach for the last 9 months due to stridor, bronchospasm and larageal spasms. i am currently being worked up for autoimmune disease. i am on xoliar am i more susceptible to coronavirus?", 223 | "doctor: yes. very very important for you to stay in quarantine and not go and shop. also make sure anything that comes into your house is sanitized. every single thing. sanitize. keep 2 m away from everyone and wear a buff or home made mask that covers your nose and mouth. be safe. be positive. we will get through this." 224 | ] 225 | }, 226 | { 227 | "description": "hi! i\u2019ve had a cold since 13 march. i have since started coughing and just don\u2019t seem to be recovering. one day i will be feeling fine and the next day back to feeling terrible. lots of tight phlegm and very sore throat. ?", 228 | "utterances": [ 229 | "patient: hi! i\u2019ve had a cold since 13 march. i have since started coughing and just don\u2019t seem to be recovering. one day i will be feeling fine and the next day back to feeling terrible. lots of tight phlegm and very sore throat. ?", 230 | "doctor: in brief: too soon to recover on aveage, viral respiratory illnesses may take 2-3 weeks for full recovery and severe cases may take 4-6 weeks. if your symptoms worsen, you should contact your health care provider to inquire regarding corona virus testing or quarantine. would you like to video or text chat with me?" 231 | ] 232 | }, 233 | { 234 | "description": "on saturday evening i arrived from a vacation in canada and i traveled via the european union(amsterdam). according to the nicd, i am required to be in self quarantine for 14 days. can one request a medical certificate for this from a doctor?", 235 | "utterances": [ 236 | "patient: on saturday evening i arrived from a vacation in canada and i traveled via the european union(amsterdam). according to the nicd, i am required to be in self quarantine for 14 days. can one request a medical certificate for this from a doctor?", 237 | "doctor: in brief: covid19 hiyou cannot go in and see your doctor, so i would suggest to phone hime and hear what he says. regardsdr wil would you like to video or text chat with me?" 238 | ] 239 | }, 240 | { 241 | "description": "do i need to get tested for covid-19. living with a person who returned from canada 20/03. dry throat, but do not feel sick?", 242 | "utterances": [ 243 | "patient: do i need to get tested for covid-19. living with a person who returned from canada 20/03. dry throat, but do not feel sick?", 244 | "doctor: in brief: hard to answer if not ill assume you have it and behave as if you do, wear a bandanna over nose and mouth and if you have access to testing then do it. if not stay home and boost immunity. would you like to video or text chat with me?" 245 | ] 246 | }, 247 | { 248 | "description": "itchy/burning throat. no cough. could this be corona infection?", 249 | "utterances": [ 250 | "patient: itchy/burning throat. no cough. could this be corona infection?", 251 | "doctor: it could be. self isolate, wash your hands, plenty of fluids and rest, if you get a high fever or have trouble breathing call your doctor." 252 | ] 253 | }, 254 | { 255 | "description": "i think i have covid-19. i have difficulty breathing (shortness of breath), tightness in chest, sore throat and light fever. symptoms have been progressing and getting stronger over the last 4 days. i had diarrhea two days ago.", 256 | "utterances": [ 257 | "patient: i think i have covid-19. i have difficulty breathing (shortness of breath), tightness in chest, sore throat and light fever. symptoms have been progressing and getting stronger over the last 4 days. i had diarrhea two days ago.", 258 | "doctor: in brief: if bad, needs er covid-19 pandemic at this time, so a doctor on video may consult by video instead of requiring an in-person visit. flu-like symptoms can be from a strep throat infection, a cold or influenza, or from some other cause like covid-19. usually, a person calls the doctor if the symptoms are bothersome, serious, recurrent, or persistent. covid-19 testing depends on local availability. (3/22/20) would you like to video or text chat with me?" 259 | ] 260 | }, 261 | { 262 | "description": "do i have coronavirus ? i have joint pains. i did sleep most of the day yesterday so i might have been in an uncomfortable position. i have neck, shoulder, elbow joint and wrist pain. must i get tested ?", 263 | "utterances": [ 264 | "patient: do i have coronavirus ? i have joint pains. i did sleep most of the day yesterday so i might have been in an uncomfortable position. i have neck, shoulder, elbow joint and wrist pain. must i get tested ?", 265 | "doctor: in brief: no not yet, joint pain alone not an indication, but stay home would you like to video or text chat with me?" 266 | ] 267 | }, 268 | { 269 | "description": "re: covid-19 -- is it safe to go out in public without a face mask on right now as long as the recommended 6ft distance is maintained? am wondering if picking up groceries curbside (have to get out of car and open trunk for store employee) is safe?", 270 | "utterances": [ 271 | "patient: re: covid-19 -- is it safe to go out in public without a face mask on right now as long as the recommended 6ft distance is maintained? am wondering if picking up groceries curbside (have to get out of car and open trunk for store employee) is safe?", 272 | "doctor: in brief: yes and no. yes for the first question. for the second question i should say no because the possibility of the store employee coming close less than 6 feet is great. but what you can do is you get out of the car open the truck and then call the store employee for a curb side pick up leaving always a 6 feet distance between two of you. would you like to video or text chat with me?" 273 | ] 274 | }, 275 | { 276 | "description": "i have had sore throat and shortness of breath, tight chest since last monday. 8 days later and symptoms are not going away. no runny nose. i have been exposed to travelers. should i go for covid test?", 277 | "utterances": [ 278 | "patient: i have had sore throat and shortness of breath, tight chest since last monday. 8 days later and symptoms are not going away. no runny nose. i have been exposed to travelers. should i go for covid test?", 279 | "doctor: yes. yes." 280 | ] 281 | }, 282 | { 283 | "description": "suggest treatment for pneumonia", 284 | "utterances": [ 285 | "patient: my boyfriend was put in the hospital for strep a nemonia 5 days ago yesterday they done another x ray and said that his nemonia was getting worse and had spread to his left lung and was also in his right real bad they also done a culture on the stuff that hes been coughing up out of his lungs and today they told us he has staph in his lungs how dangerous is this and what should be done", 286 | "doctor: hello thank you for trusting hcm dear what is the culture report suggestive?? staphylococcus aureus pneumonia take more time to come under control, usually community acquired staphylococcus infection may present with methicillne resistance(mrsa) , for this patient cases need to treated with vancomycine, and linezolid. some patients also developed vancomycine resistant staph aureus (vrsa) infection, this infection very problematic to control. very few drugs like quinupristin-dalfopristin and linezolid available to control vrsa. please follow your doctor orders they will treat you in right way. i think i answered to your question if you have more questions please feel free to ask." 287 | ] 288 | }, 289 | { 290 | "description": "i have a dizzy head and some phlegm. need to check against covid symptoms please?", 291 | "utterances": [ 292 | "patient: i have a dizzy head and some phlegm. need to check against covid symptoms please?", 293 | "doctor: in brief: check online there are many resources online about the symptoms of covid-19. the most important questions relate to travel history and the symptoms you are experiencing. if you suspect that you may have covid-19, self-isolate as soon as possible and call your local doctor or testing site. you can also check out https://www.gov.za/coronavirus/faq or call 0800 029 999 would you like to video or text chat with me?" 294 | ] 295 | }, 296 | { 297 | "description": "i am experiencing a sore throat, fever, headache and coughing. i have a bit of a runny nose, short breath but only from time to time. i have traveled on the mcs cruise to mozambique 13-16 march 2020. i think it might be necessary to do a covid-19?", 298 | "utterances": [ 299 | "patient: i am experiencing a sore throat, fever, headache and coughing. i have a bit of a runny nose, short breath but only from time to time. i have traveled on the mcs cruise to mozambique 13-16 march 2020. i think it might be necessary to do a covid-19?", 300 | "doctor: get tested. if you can not find a place to get tested self-quarantine yourself for 2 weeks. you can use over the counter drugs for symptomatic treatment in the mean time. if sore throat get worse this could be strep throat need a culture to be taken from your throat by your doctor followed by antibiotic. if shortness of breath gets worse call the hospital for the inpatient treatment could be for covid-19." 301 | ] 302 | }, 303 | { 304 | "description": "body ache and fever, scratchy itchy throat?", 305 | "utterances": [ 306 | "patient: body ache and fever, scratchy itchy throat?", 307 | "doctor: early covid-19? call your health care provider to get instructions regarding testing, self-quarantine and possible hospitalization then follow through!" 308 | ] 309 | }, 310 | { 311 | "description": "suggest medication for pulmonary embolism in right lung and pneumonia in left.", 312 | "utterances": [ 313 | "patient: about 6 weeks ago i was diagnosed with pe in right lung and phnemoia in the left lung. i was in teh hospital for 9 days. i am on 20mg of warfin per day! my blood levels are holding steady at 2.67. yesterday i had a bit of lower back pain and today it is between my shoulder blades, mostly on my right side. it hurts my back to breathe, swallow, and move. just sore muscles or could it be something serious with the blood clot? i hate to rush into the doctor if it is nothing....i hope you can help! thanks", 314 | "doctor: thanks for your question on healthcare magic.i can understand your concern. since you are taking warfarin (blood thinner), possibility of pulmonary embolism (pe) related pain is less likely.possibility of musculoskeletal pain is more likely. so avoid heavyweight lifting and strenuous exercise. avoid movements causing pain. apply warm water pad on affected areas of chest and back. take painkiller and muscle relaxant drugs like ibuprofen and thiocolchicoside. you will mostly improve with all these in 2-3 days. if not improving then consult doctor. hope i have solved your query. i will be happy to help you further. wish you good health. thanks." 315 | ] 316 | }, 317 | { 318 | "description": "hi i have been have a sore throat for almost a week now, tried gaghling with warm and apple cider vinegar, drank warm water with apple, warm with lemon. now i still have a sore throat and burning on my chest?", 319 | "utterances": [ 320 | "patient: hi i have been have a sore throat for almost a week now, tried gaghling with warm and apple cider vinegar, drank warm water with apple, warm with lemon. now i still have a sore throat and burning on my chest?", 321 | "doctor: in brief: can give dr. a call throat pain can be from a strep throat infection (antibiotics needed), a cold or influenza (antibiotics usually not used), or from some other cause such as allergies or irritants. usually, a person sees the doctor (call first) if the sore throat is bothersome, recurrent, or doesn't go away quickly. covid-19 infections tend to have cough, whereas strep usually lacks cough but has more throat pain. would you like to video or text chat with me?" 322 | ] 323 | }, 324 | { 325 | "description": "if everyone is running low on facemasks, what else can i use to protect myself from the coronavirus?", 326 | "utterances": [ 327 | "patient: if everyone is running low on facemasks, what else can i use to protect myself from the coronavirus?", 328 | "doctor: they don't . unless you are using high end medical or industrial masks they do not protect you from any such virus. they prevent your cough or expired air from infecting others. any scarf/fabric homemade mask or similar device will do the same. you protect yourself more by washing your hands often & not touching your face/mouth/nose with contaminated fingers." 329 | ] 330 | }, 331 | { 332 | "description": "if i present with a scratchy throat, do i automatically self isolate, or do i wait for more symptoms for coronavirus?", 333 | "utterances": [ 334 | "patient: if i present with a scratchy throat, do i automatically self isolate, or do i wait for more symptoms for coronavirus?", 335 | "doctor: don't isolate yet. . a scratchy throat could be caused by heartburn, oral thrush, bacterial pharyngitis (@ the beginning) or any of the 200 odd viruses that cause the \"common cold\".if you have had a +ve contact with a covid 19 patient then isolate for 14 days or ifyou develop a high fever, severe headache, dry cough and sore throat, the isolate but only after getting tested for covid 19." 336 | ] 337 | }, 338 | { 339 | "description": "what is the treatment for pneumonia?", 340 | "utterances": [ 341 | "patient: my daughter is 3 and a half...has had pneumonia twice and is currently being treated with seretide inhaler, ventoline exohaler, brozedex and ebastel. her chest infection though isn't clearing and her appetitie is low. she complains of chest pains. what is the best way to get her treated and cured? she has used seretide for over 4 months now and it has a steroide and that may have side effects, but the doctor says she must have it to stop the infection, but that isn't working either.", 342 | "doctor: thanks for your question on healthcare magic.i can understand your concern. in my opinion, you should consult pediatric pulmonologist and get done bronchoscopy and bal (bronchoalveolar lavage) analysis.bal culture and sensitivity report will isolate the causative organism and guide about effective antibiotic treatment. so we can achieve faster recovery from this. don't worry, with effective antibiotic treatment, she will be alright with 2 - 3 weeks treatment.hope i have solved your query. i will be happy to help you further. wishing good health to your daughter. thanks." 343 | ] 344 | }, 345 | { 346 | "description": "experiencing pain and swelling accompanied by warmth to the touch with a red patch around my elbow. also experiencing body aches. what could be the problem?", 347 | "utterances": [ 348 | "patient: experiencing pain and swelling accompanied by warmth to the touch with a red patch around my elbow. also experiencing body aches. what could be the problem?", 349 | "doctor: might need treatment. could be infection or other condition requiring urgent treatment. seek evaluation, but preferably on phone or online, to avoid coronavirus exposure. video consultations are available on healthtap; click on \"talk-to-doctor.\" ." 350 | ] 351 | }, 352 | { 353 | "description": "my copd meds are finishing in the next week and i cant get in to see my pulmologist?", 354 | "utterances": [ 355 | "patient: my copd meds are finishing in the next week and i cant get in to see my pulmologist?", 356 | "doctor: try our other forum. you have posted on our free, public, informational forum, where doctors do not see patient charts nor give medical care. try a private consultation where doctors can see you on video. also, in the us certain medications (but not all) can be prescribed or temporarily refilled, depending on individual circumstances. start over on your screen, click on care, then talk-to-doctor. (mar. 2020)" 357 | ] 358 | }, 359 | { 360 | "description": "should i invest in a reusable mask to protect myself from covid-19? i know a lot of people wear those in asian countries and they look kinda nice.", 361 | "utterances": [ 362 | "patient: should i invest in a reusable mask to protect myself from covid-19? i know a lot of people wear those in asian countries and they look kinda nice.", 363 | "doctor: in brief: barrier protection most of these masks won't completely protect you from a virus, which can penetrate these materials however that does not mean they are not useful. a mask of any sort is good if you are coughing or sneezing to limit some of the aerosols and also is better than nothing if you are out and about in a high risk situation. the chance of you catching the virus is highly dependent on lengthexposure & dos would you like to video or text chat with me?" 364 | ] 365 | }, 366 | { 367 | "description": "suggest treatment to cure pneumonia", 368 | "utterances": [ 369 | "patient: hi last night i went to emerg i heard the nurse tell my doctor i might have pnemonia he didn't even bother coming to see me and when i left upset and walked to his office he prescribed me meds that may react somehow with the meds i'm taking should i be concerned should find a new doctor is this common way to treat your patients? he was the doctor on call last night", 370 | "doctor: thanks for your question on healthcare magic.i can understand your concern.yes, you should definitely consult another doctor.pneumonia is lung infection and it should be treated promptly and correctly.and chest x ray is must for the diagnosis of pneumonia.so better to consult another doctor and get done chest x ray to rule out pneumonia.hope i have solved your query. i will be happy to help you further. wish you good health. thanks." 371 | ] 372 | }, 373 | { 374 | "description": "what causes cold sensations in the abdomen and back while having pneumonia?", 375 | "utterances": [ 376 | "patient: my husband, age 63 is feeling cool on his abdomen and back while the rest of him is warm...currently sleeping. last week was diagnosed with pneumonia...had a period last week when he was so cold, couldn t get warm so we went to the hospital. he says he is feeling fine and feels ok to work everyday (does construction work outside on a normal basis). should i be concerned tonight that part of him feels cool even when under covera for an hour or two thanks so much", 377 | "doctor: hello and thank you for asking in hcm i can understand your concern don't worry so much. your husband is under treatment for pneumonia according to the history and he has had fever several times.now he feels better but he is not completely asymptomatic which means that the feeling he had might be due to an episode of temperature not so high .this shows that the treatment is doing well. but he has to be careful with the treatment to be correct as the doctor has prescribed ,no smoking and not to work in dust environment ,a lot of liquids too.take carewish a fast recovery for your husband. dr. jolanda pulmonologist" 378 | ] 379 | }, 380 | { 381 | "description": "i have some suspected mulluscum on most of my stomach but it's itchy, inflammed, sentitive, and it's spreading. aren't most people supposed to around 20 bumps? why is it peeling so much? most of all, what is the best way to treat mulluscum at home?", 382 | "utterances": [ 383 | "patient: i have some suspected mulluscum on most of my stomach but it's itchy, inflammed, sentitive, and it's spreading. aren't most people supposed to around 20 bumps? why is it peeling so much? most of all, what is the best way to treat mulluscum at home?", 384 | "doctor: in brief: hard to get rid of molluscum is a virus, the less you touch the lesions the less you will spread the infection. if you want to get rid of it it\u2019s hard, but would suggest a dermatologist rather than the freezing sprays or chemicals used to kill the virus in the skin. there is no \u2018limit\u2019 to number of lesions, have not heard that information, but i would not believe that would you like to video or text chat with me?" 385 | ] 386 | }, 387 | { 388 | "description": "suggest treatment for pneumonia", 389 | "utterances": [ 390 | "patient: hi, my mom is 65 years of age, she never took any medication except for antibiotics in march when she broke her wrists of osteophyroses, about 4 weeks back she went to a doctor for a cough who tokd her she had brochites, and gave her a antibiotic of 400mg that she had to take twice a day, then that night she got worsr, thwe next day she went to another doctor who admitted her to hospital, said she had neumonia.shes been in icu ever since. she develop water in her lungs, her kidneys and liver had failed to work twice now, and she have infection in her muscles, (cant walk, talk & struggle to brath.", 391 | "doctor: thanks for your question on healthcare magic.i can understand your concern. by your history and description, possibility of severe pneumonia with sepsis and mods (multiple organ dysfunction syndrome) is more likely.best treatment of pneumonia is identification of causative organism and then start appropriate treatment. so get done bronchoscopy and bal (bronchoalveolar lavage) culture and sensitivity report.also get done blood culture and sensitivity report. culture report will isolate the causative organism and sensitivity report will guide about effective antibiotic treatment. by this we can achieve faster recovery. hope i have solved your query. i will be happy to help you further. wish you good health. thanks." 392 | ] 393 | }, 394 | { 395 | "description": "good afternoon, i recently visited a client whose husband works in the same practice as a confirmed covid case.two days after the visit i fell ill, gradually getting a sore throat, fever and feeling tired. should i get tested?", 396 | "utterances": [ 397 | "patient: good afternoon, i recently visited a client whose husband works in the same practice as a confirmed covid case.two days after the visit i fell ill, gradually getting a sore throat, fever and feeling tired. should i get tested?", 398 | "doctor: in brief: tesing follow the following guidelines: https://www. healthtap.com/blog/covid-19-care-guidelines/self-quarantine-guide call your local provider or health department. at 36 you would be considered low risk for serious infection. the test in itself does not lead to a specific action and the tests are numbered. so at this time the tests are limited and being used for sicker patients. would you like to video or text chat with me?" 399 | ] 400 | }, 401 | { 402 | "description": "if i'm not sure if i have allergies or coronavirus (or something else), should i still stay home?", 403 | "utterances": [ 404 | "patient: if i'm not sure if i have allergies or coronavirus (or something else), should i still stay home?", 405 | "doctor: in brief: there are many online resources to help you distinguish between the two. but, yes it is a good idea to stay home until told otherwise. would you like to video or text chat with me?" 406 | ] 407 | }, 408 | { 409 | "description": "can sinusitis cause white spots on tonsils?", 410 | "utterances": [ 411 | "patient: can sinusitis cause white spots on tonsils?", 412 | "doctor: in brief: usually does not throat symptoms can be from strep throat infection (antibiotics are needed), a cold or other virus (antibiotics usually not used), allergies, irritants, sinusitis, stomach reflux when asleep, etc. .. one usually sees the doctor if the throat issue is bothersome, recurrent, or doesn't go away quickly. a strep throat swab test can easily diagnose strep throat. would you like to video or text chat with me?" 413 | ] 414 | }, 415 | { 416 | "description": "have read covid-19 news quoting some experts as saying that we will see more confirmed cases as more tests are performed, but the mortality rate should drop as that happens. any sign of that yet?", 417 | "utterances": [ 418 | "patient: have read covid-19 news quoting some experts as saying that we will see more confirmed cases as more tests are performed, but the mortality rate should drop as that happens. any sign of that yet?", 419 | "doctor: in brief: covid-19 mortality don't confuse actual mortality rate with reported rate. the actual rate probably will remain stable (and likely lower in the us than some other countries, because of sophisticated intensive medical care). but as testing expands and more people with mild symptoms (or no symptoms) are diagnosed, reported mortality will decline; likely happening now, but it may take weeks for data to be known." 420 | ] 421 | }, 422 | { 423 | "description": "hi, i would like to know where to get tested for covid-19, i work in a call center and one agent was confirmed to have tested positive and i still had contact with her close friends and her last wednesday before i sent her to the doctor for a check?", 424 | "utterances": [ 425 | "patient: hi, i would like to know where to get tested for covid-19, i work in a call center and one agent was confirmed to have tested positive and i still had contact with her close friends and her last wednesday before i sent her to the doctor for a check?", 426 | "doctor: self-quarantine. try self-quarantine for 14 days; call your health care provider if you start having symptoms to get instructions regarding testing, self-quarantine and possible hospitalization then follow through!" 427 | ] 428 | } 429 | ] --------------------------------------------------------------------------------