Figure 1: MRC example from SQuAD 2.0 dev set
5 | 6 | ## Setting 7 | * Python 3.6.7 8 | * Tensorflow 1.13.1 9 | * NumPy 1.13.3 10 | * SentencePiece 0.1.82 11 | 12 | ## DataSet 13 | * [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) is a reading comprehension dataset, consisting of questions posed by crowd-workers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. 14 | * [CoQA](https://stanfordnlp.github.io/coqa/) a large-scale dataset for building Conversational Question Answering systems. The goal of the CoQA challenge is to measure the ability of machines to understand a text passage and answer a series of interconnected questions that appear in a conversation. CoQA is pronounced as coca 15 | * [QuAC](https://quac.ai/) is a dataset for modeling, understanding, and participating in information seeking dialog. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context. 16 | 17 | ## Usage 18 | * Run SQuAD experiment 19 | ```bash 20 | CUDA_VISIBLE_DEVICES=0,1,2,3 python run_squad.py \ 21 | --spiece_model_file=model/cased_L-24_H-1024_A-16/spiece.model \ 22 | --model_config_path=model/cased_L-24_H-1024_A-16/xlnet_config.json \ 23 | --init_checkpoint=model/cased_L-24_H-1024_A-16/xlnet_model.ckpt \ 24 | --task_name=v2.0 \ 25 | --random_seed=100 \ 26 | --predict_tag=xxxxx \ 27 | --data_dir=data/squad/v2.0 \ 28 | --output_dir=output/squad/v2.0/data \ 29 | --model_dir=output/squad/v2.0/checkpoint \ 30 | --export_dir=output/squad/v2.0/export \ 31 | --max_seq_length=512 \ 32 | --train_batch_size=12 \ 33 | --predict_batch_size=12 \ 34 | --num_hosts=1 \ 35 | --num_core_per_host=4 \ 36 | --learning_rate=3e-5 \ 37 | --train_steps=8000 \ 38 | --warmup_steps=1000 \ 39 | --save_steps=1000 \ 40 | --do_train=true \ 41 | --do_predict=true \ 42 | --do_export=true \ 43 | --overwrite_data=false 44 | ``` 45 | * Run CoQA experiment 46 | ```bash 47 | CUDA_VISIBLE_DEVICES=0,1,2,3 python run_coqa.py \ 48 | --spiece_model_file=model/cased_L-24_H-1024_A-16/spiece.model \ 49 | --model_config_path=model/cased_L-24_H-1024_A-16/xlnet_config.json \ 50 | --init_checkpoint=model/cased_L-24_H-1024_A-16/xlnet_model.ckpt \ 51 | --task_name=v1.0 \ 52 | --random_seed=100 \ 53 | --predict_tag=xxxxx \ 54 | --data_dir=data/coqa/v1.0 \ 55 | --output_dir=output/coqa/v1.0/data \ 56 | --model_dir=output/coqa/v1.0/checkpoint \ 57 | --export_dir=output/coqa/v1.0/export \ 58 | --max_seq_length=512 \ 59 | --train_batch_size=12 \ 60 | --predict_batch_size=12 \ 61 | --num_hosts=1 \ 62 | --num_core_per_host=4 \ 63 | --learning_rate=3e-5 \ 64 | --train_steps=8000 \ 65 | --warmup_steps=1000 \ 66 | --save_steps=1000 \ 67 | --do_train=true \ 68 | --do_predict=true \ 69 | --do_export=true \ 70 | --overwrite_data=false 71 | ``` 72 | * Run QuAC experiment 73 | ```bash 74 | CUDA_VISIBLE_DEVICES=0,1,2,3 python run_quac.py \ 75 | --spiece_model_file=model/cased_L-24_H-1024_A-16/spiece.model \ 76 | --model_config_path=model/cased_L-24_H-1024_A-16/xlnet_config.json \ 77 | --init_checkpoint=model/cased_L-24_H-1024_A-16/xlnet_model.ckpt \ 78 | --task_name=v1.0 \ 79 | --random_seed=100 \ 80 | --predict_tag=xxxxx \ 81 | --data_dir=data/quac/v0.2 \ 82 | --output_dir=output/quac/v0.2/data \ 83 | --model_dir=output/quac/v0.2/checkpoint \ 84 | --export_dir=output/quac/v0.2/export \ 85 | --max_seq_length=512 \ 86 | --train_batch_size=12 \ 87 | --predict_batch_size=12 \ 88 | --num_hosts=1 \ 89 | --num_core_per_host=4 \ 90 | --learning_rate=3e-5 \ 91 | --train_steps=8000 \ 92 | --warmup_steps=1000 \ 93 | --save_steps=1000 \ 94 | --do_train=true \ 95 | --do_predict=true \ 96 | --do_export=true \ 97 | --overwrite_data=false 98 | ``` 99 | 100 | ## Experiment 101 | ### SQuAD v1.1 102 |
Figure 2: Illustrations of fine-tuning XLNet on SQuAD v1.1 task
104 | 105 | | Model | Train Data | # Train Steps | Batch Size | Max Length | Learning Rate | EM | F1 | 106 | |:-------------:|:----------:|:-------------:|:----------:|:----------:|:-------------:|:--------:|:--------:| 107 | | XLNet-base | SQuAD 2.0 | 8,000 | 48 | 512 | 3e-5 | 85.90 | 92.17 | 108 | | XLNet-large | SQuAD 2.0 | 8,000 | 48 | 512 | 3e-5 | 88.61 | 94.28 | 109 | 110 |Table 1: The dev set performance of XLNet model finetuned on SQuAD v1.1 task
111 | 112 | ### SQuAD v2.0 113 |
Figure 3: Illustrations of fine-tuning XLNet on SQuAD v2.0 task
115 | 116 | | Model | Train Data | # Train Steps | Batch Size | Max Length | Learning Rate | EM | F1 | 117 | |:-------------:|:----------:|:-------------:|:----------:|:----------:|:-------------:|:--------:|:--------:| 118 | | XLNet-base | SQuAD 2.0 | 8,000 | 48 | 512 | 3e-5 | 80.23 | 82.90 | 119 | | XLNet-large | SQuAD 2.0 | 8,000 | 48 | 512 | 3e-5 | 85.72 | 88.36 | 120 | 121 |Table 2: The dev set performance of XLNet model finetuned on SQuAD v2.0 task
122 | 123 | ### CoQA v1.0 124 |
Figure 4: Illustrations of fine-tuning XLNet on CoQA v1.0 task
126 | 127 | | Model | Train Data | # Train Steps | Batch Size | Max Length | Max Query Len | Learning Rate | EM | F1 | 128 | |:-------------:|:----------:|:-------------:|:----------:|:----------:|:-------------:|:-------------:|:--------:|:--------:| 129 | | XLNet-base | CoQA 1.0 | 6,000 | 48 | 512 | 128 | 3e-5 | 76.4 | 84.4 | 130 | | XLNet-large | CoQA 1.0 | 6,000 | 48 | 512 | 128 | 3e-5 | 81.8 | 89.4 | 131 | 132 |Table 3: The dev set performance of XLNet model finetuned on CoQA v1.0 task
133 | 134 | ### QuAC v0.2 135 |
Figure 5: Illustrations of fine-tuning XLNet on QuAC v0.2 task
137 | 138 | | Model | Train Data | # Train Steps | Batch Size | Max Length | Max Query Len | Learning Rate | Overall F1 | HEQQ | HEQD | 139 | |:-------------:|:----------:|:-------------:|:----------:|:----------:|:-------------:|:-------------:|:----------:|:------:|:------:| 140 | | XLNet-base | QuAC 0.2 | 8,000 | 48 | 512 | 128 | 2e-5 | 66.4 | 62.6 | 6.8 | 141 | | XLNet-large | QuAC 0.2 | 8,000 | 48 | 512 | 128 | 2e-5 | 71.5 | 68.0 | 11.1 | 142 | 143 |Table 3: The dev set performance of XLNet model finetuned on QuAC v0.2 task
144 | 145 | ## Reference 146 | * Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [SQuAD: 100,000+ questions for machine comprehension of text](https://arxiv.org/abs/1606.05250) [2016] 147 | * Pranav Rajpurkar, Robin Jia, and Percy Liang. [Know what you don’t know: unanswerable questions for SQuAD](https://arxiv.org/abs/1806.03822) [2018] 148 | * Siva Reddy, Danqi Chen, Christopher D. Manning. [CoQA: A Conversational Question Answering Challenge](https://arxiv.org/abs/1808.07042) [2018] 149 | * Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer. [QuAC : Question Answering in Context](https://arxiv.org/abs/1808.07036) [2018] 150 | * Danqi Chen. [Neural reading comprehension and beyond](https://cs.stanford.edu/~danqi/papers/thesis.pdf) [2018] 151 | * Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matthew Gardner, Christopher T Clark, Kenton Lee, and Luke S. Zettlemoyer. [Deep contextualized word representations](https://arxiv.org/abs/1802.05365) [2018] 152 | * Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. [Improving language understanding by generative pre-training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) [2018] 153 | * Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. [Language models are unsupervised multitask learners](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) [2019] 154 | * Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://arxiv.org/abs/1810.04805) [2018] 155 | * Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) [2019] 156 | * Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut. [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942) [2019] 157 | * Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) [2019] 158 | * Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov and Quoc V. Le. [XLNet: Generalized autoregressive pretraining for language understanding](https://arxiv.org/abs/1906.08237) [2019] 159 | * Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le and Ruslan Salakhutdinov. [Transformer-XL: Attentive language models beyond a fixed-length context](https://arxiv.org/abs/1901.02860) [2019] 160 | -------------------------------------------------------------------------------- /tool/eval_coqa.py: -------------------------------------------------------------------------------- 1 | """Official evaluation script for CoQA. 2 | 3 | The code is based partially on SQuAD 2.0 evaluation script. 4 | """ 5 | import argparse 6 | import json 7 | import re 8 | import string 9 | import sys 10 | 11 | from collections import Counter, OrderedDict 12 | 13 | OPTS = None 14 | 15 | out_domain = ["reddit", "science"] 16 | in_domain = ["mctest", "gutenberg", "race", "cnn", "wikipedia"] 17 | domain_mappings = {"mctest":"children_stories", "gutenberg":"literature", "race":"mid-high_school", "cnn":"news", "wikipedia":"wikipedia", "science":"science", "reddit":"reddit"} 18 | 19 | 20 | class CoQAEvaluator(): 21 | 22 | def __init__(self, gold_file): 23 | self.gold_data, self.id_to_source = CoQAEvaluator.gold_answers_to_dict(gold_file) 24 | 25 | @staticmethod 26 | def gold_answers_to_dict(gold_file): 27 | dataset = json.load(open(gold_file)) 28 | gold_dict = {} 29 | id_to_source = {} 30 | for story in dataset['data']: 31 | source = story['source'] 32 | story_id = story['id'] 33 | id_to_source[story_id] = source 34 | questions = story['questions'] 35 | multiple_answers = [story['answers']] 36 | multiple_answers += story['additional_answers'].values() 37 | for i, qa in enumerate(questions): 38 | qid = qa['turn_id'] 39 | if i + 1 != qid: 40 | sys.stderr.write("Turn id should match index {}: {}\n".format(i + 1, qa)) 41 | gold_answers = [] 42 | for answers in multiple_answers: 43 | answer = answers[i] 44 | if qid != answer['turn_id']: 45 | sys.stderr.write("Question turn id does match answer: {} {}\n".format(qa, answer)) 46 | gold_answers.append(answer['input_text']) 47 | key = (story_id, qid) 48 | if key in gold_dict: 49 | sys.stderr.write("Gold file has duplicate stories: {}".format(source)) 50 | gold_dict[key] = gold_answers 51 | return gold_dict, id_to_source 52 | 53 | @staticmethod 54 | def preds_to_dict(pred_file): 55 | preds = json.load(open(pred_file)) 56 | pred_dict = {} 57 | for pred in preds: 58 | pred_dict[(pred['id'], pred['turn_id'])] = pred['answer'] 59 | return pred_dict 60 | 61 | @staticmethod 62 | def normalize_answer(s): 63 | """Lower text and remove punctuation, storys and extra whitespace.""" 64 | 65 | def remove_articles(text): 66 | regex = re.compile(r'\b(a|an|the)\b', re.UNICODE) 67 | return re.sub(regex, ' ', text) 68 | 69 | def white_space_fix(text): 70 | return ' '.join(text.split()) 71 | 72 | def remove_punc(text): 73 | exclude = set(string.punctuation) 74 | return ''.join(ch for ch in text if ch not in exclude) 75 | 76 | def lower(text): 77 | return text.lower() 78 | 79 | return white_space_fix(remove_articles(remove_punc(lower(s)))) 80 | 81 | @staticmethod 82 | def get_tokens(s): 83 | if not s: return [] 84 | return CoQAEvaluator.normalize_answer(s).split() 85 | 86 | @staticmethod 87 | def compute_exact(a_gold, a_pred): 88 | return int(CoQAEvaluator.normalize_answer(a_gold) == CoQAEvaluator.normalize_answer(a_pred)) 89 | 90 | @staticmethod 91 | def compute_f1(a_gold, a_pred): 92 | gold_toks = CoQAEvaluator.get_tokens(a_gold) 93 | pred_toks = CoQAEvaluator.get_tokens(a_pred) 94 | common = Counter(gold_toks) & Counter(pred_toks) 95 | num_same = sum(common.values()) 96 | if len(gold_toks) == 0 or len(pred_toks) == 0: 97 | # If either is no-answer, then F1 is 1 if they agree, 0 otherwise 98 | return int(gold_toks == pred_toks) 99 | if num_same == 0: 100 | return 0 101 | precision = 1.0 * num_same / len(pred_toks) 102 | recall = 1.0 * num_same / len(gold_toks) 103 | f1 = (2 * precision * recall) / (precision + recall) 104 | return f1 105 | 106 | @staticmethod 107 | def _compute_turn_score(a_gold_list, a_pred): 108 | f1_sum = 0.0 109 | em_sum = 0.0 110 | if len(a_gold_list) > 1: 111 | for i in range(len(a_gold_list)): 112 | # exclude the current answer 113 | gold_answers = a_gold_list[0:i] + a_gold_list[i + 1:] 114 | em_sum += max(CoQAEvaluator.compute_exact(a, a_pred) for a in gold_answers) 115 | f1_sum += max(CoQAEvaluator.compute_f1(a, a_pred) for a in gold_answers) 116 | else: 117 | em_sum += max(CoQAEvaluator.compute_exact(a, a_pred) for a in a_gold_list) 118 | f1_sum += max(CoQAEvaluator.compute_f1(a, a_pred) for a in a_gold_list) 119 | 120 | return {'em': em_sum / max(1, len(a_gold_list)), 'f1': f1_sum / max(1, len(a_gold_list))} 121 | 122 | def compute_turn_score(self, story_id, turn_id, a_pred): 123 | ''' This is the function what you are probably looking for. a_pred is the answer string your model predicted. ''' 124 | key = (story_id, turn_id) 125 | a_gold_list = self.gold_data[key] 126 | return CoQAEvaluator._compute_turn_score(a_gold_list, a_pred) 127 | 128 | def get_raw_scores(self, pred_data): 129 | ''''Returns a dict with score with each turn prediction''' 130 | exact_scores = {} 131 | f1_scores = {} 132 | for story_id, turn_id in self.gold_data: 133 | key = (story_id, turn_id) 134 | if key not in pred_data: 135 | sys.stderr.write('Missing prediction for {} and turn_id: {}\n'.format(story_id, turn_id)) 136 | continue 137 | a_pred = pred_data[key] 138 | scores = self.compute_turn_score(story_id, turn_id, a_pred) 139 | # Take max over all gold answers 140 | exact_scores[key] = scores['em'] 141 | f1_scores[key] = scores['f1'] 142 | return exact_scores, f1_scores 143 | 144 | def get_raw_scores_human(self): 145 | ''''Returns a dict with score for each turn''' 146 | exact_scores = {} 147 | f1_scores = {} 148 | for story_id, turn_id in self.gold_data: 149 | key = (story_id, turn_id) 150 | f1_sum = 0.0 151 | em_sum = 0.0 152 | if len(self.gold_data[key]) > 1: 153 | for i in range(len(self.gold_data[key])): 154 | # exclude the current answer 155 | gold_answers = self.gold_data[key][0:i] + self.gold_data[key][i + 1:] 156 | em_sum += max(CoQAEvaluator.compute_exact(a, self.gold_data[key][i]) for a in gold_answers) 157 | f1_sum += max(CoQAEvaluator.compute_f1(a, self.gold_data[key][i]) for a in gold_answers) 158 | else: 159 | exit("Gold answers should be multiple: {}={}".format(key, self.gold_data[key])) 160 | exact_scores[key] = em_sum / len(self.gold_data[key]) 161 | f1_scores[key] = f1_sum / len(self.gold_data[key]) 162 | return exact_scores, f1_scores 163 | 164 | def human_performance(self): 165 | exact_scores, f1_scores = self.get_raw_scores_human() 166 | return self.get_domain_scores(exact_scores, f1_scores) 167 | 168 | def model_performance(self, pred_data): 169 | exact_scores, f1_scores = self.get_raw_scores(pred_data) 170 | return self.get_domain_scores(exact_scores, f1_scores) 171 | 172 | def get_domain_scores(self, exact_scores, f1_scores): 173 | sources = {} 174 | for source in in_domain + out_domain: 175 | sources[source] = Counter() 176 | 177 | for story_id, turn_id in self.gold_data: 178 | key = (story_id, turn_id) 179 | source = self.id_to_source[story_id] 180 | sources[source]['em_total'] += exact_scores.get(key, 0) 181 | sources[source]['f1_total'] += f1_scores.get(key, 0) 182 | sources[source]['turn_count'] += 1 183 | 184 | scores = OrderedDict() 185 | in_domain_em_total = 0.0 186 | in_domain_f1_total = 0.0 187 | in_domain_turn_count = 0 188 | 189 | out_domain_em_total = 0.0 190 | out_domain_f1_total = 0.0 191 | out_domain_turn_count = 0 192 | 193 | for source in in_domain + out_domain: 194 | domain = domain_mappings[source] 195 | scores[domain] = {} 196 | scores[domain]['em'] = round(sources[source]['em_total'] / max(1, sources[source]['turn_count']) * 100, 1) 197 | scores[domain]['f1'] = round(sources[source]['f1_total'] / max(1, sources[source]['turn_count']) * 100, 1) 198 | scores[domain]['turns'] = sources[source]['turn_count'] 199 | if source in in_domain: 200 | in_domain_em_total += sources[source]['em_total'] 201 | in_domain_f1_total += sources[source]['f1_total'] 202 | in_domain_turn_count += sources[source]['turn_count'] 203 | elif source in out_domain: 204 | out_domain_em_total += sources[source]['em_total'] 205 | out_domain_f1_total += sources[source]['f1_total'] 206 | out_domain_turn_count += sources[source]['turn_count'] 207 | 208 | scores["in_domain"] = {'em': round(in_domain_em_total / max(1, in_domain_turn_count) * 100, 1), 209 | 'f1': round(in_domain_f1_total / max(1, in_domain_turn_count) * 100, 1), 210 | 'turns': in_domain_turn_count} 211 | scores["out_domain"] = {'em': round(out_domain_em_total / max(1, out_domain_turn_count) * 100, 1), 212 | 'f1': round(out_domain_f1_total / max(1, out_domain_turn_count) * 100, 1), 213 | 'turns': out_domain_turn_count} 214 | 215 | em_total = in_domain_em_total + out_domain_em_total 216 | f1_total = in_domain_f1_total + out_domain_f1_total 217 | turn_count = in_domain_turn_count + out_domain_turn_count 218 | scores["overall"] = {'em': round(em_total / max(1, turn_count) * 100, 1), 219 | 'f1': round(f1_total / max(1, turn_count) * 100, 1), 220 | 'turns': turn_count} 221 | 222 | return scores 223 | 224 | def parse_args(): 225 | parser = argparse.ArgumentParser('Official evaluation script for CoQA.') 226 | parser.add_argument('--data-file', dest="data_file", help='Input data JSON file.') 227 | parser.add_argument('--pred-file', dest="pred_file", help='Model predictions.') 228 | parser.add_argument('--out-file', '-o', metavar='eval.json', 229 | help='Write accuracy metrics to file (default is stdout).') 230 | parser.add_argument('--verbose', '-v', action='store_true') 231 | parser.add_argument('--human', dest="human", action='store_true') 232 | if len(sys.argv) == 1: 233 | parser.print_help() 234 | sys.exit(1) 235 | return parser.parse_args() 236 | 237 | def main(): 238 | evaluator = CoQAEvaluator(OPTS.data_file) 239 | 240 | if OPTS.human: 241 | print(json.dumps(evaluator.human_performance(), indent=2)) 242 | 243 | if OPTS.pred_file: 244 | with open(OPTS.pred_file) as f: 245 | pred_data = CoQAEvaluator.preds_to_dict(OPTS.pred_file) 246 | print(json.dumps(evaluator.model_performance(pred_data), indent=2)) 247 | 248 | if __name__ == '__main__': 249 | OPTS = parse_args() 250 | main() 251 | -------------------------------------------------------------------------------- /tool/eval_squad.py: -------------------------------------------------------------------------------- 1 | """Official evaluation script for SQuAD version 2.0. 2 | 3 | In addition to basic functionality, we also compute additional statistics and 4 | plot precision-recall curves if an additional na_prob.json file is provided. 5 | This file is expected to map question ID's to the model's predicted probability 6 | that a question is unanswerable. 7 | """ 8 | import argparse 9 | import collections 10 | import json 11 | import numpy as np 12 | import os 13 | import re 14 | import string 15 | import sys 16 | 17 | OPTS = None 18 | 19 | def parse_args(): 20 | parser = argparse.ArgumentParser('Official evaluation script for SQuAD version 2.0.') 21 | parser.add_argument('data_file', metavar='data.json', help='Input data JSON file.') 22 | parser.add_argument('pred_file', metavar='pred.json', help='Model predictions.') 23 | parser.add_argument('--out-file', '-o', metavar='eval.json', 24 | help='Write accuracy metrics to file (default is stdout).') 25 | parser.add_argument('--na-prob-file', '-n', metavar='na_prob.json', 26 | help='Model estimates of probability of no answer.') 27 | parser.add_argument('--na-prob-thresh', '-t', type=float, default=1.0, 28 | help='Predict "" if no-answer probability exceeds this (default = 1.0).') 29 | parser.add_argument('--out-image-dir', '-p', metavar='out_images', default=None, 30 | help='Save precision-recall curves to directory.') 31 | parser.add_argument('--verbose', '-v', action='store_true') 32 | if len(sys.argv) == 1: 33 | parser.print_help() 34 | sys.exit(1) 35 | return parser.parse_args() 36 | 37 | def make_qid_to_has_ans(dataset): 38 | qid_to_has_ans = {} 39 | for article in dataset: 40 | for p in article['paragraphs']: 41 | for qa in p['qas']: 42 | qid_to_has_ans[qa['id']] = bool(qa['answers']) 43 | return qid_to_has_ans 44 | 45 | def normalize_answer(s): 46 | """Lower text and remove punctuation, articles and extra whitespace.""" 47 | def remove_articles(text): 48 | regex = re.compile(r'\b(a|an|the)\b', re.UNICODE) 49 | return re.sub(regex, ' ', text) 50 | def white_space_fix(text): 51 | return ' '.join(text.split()) 52 | def remove_punc(text): 53 | exclude = set(string.punctuation) 54 | return ''.join(ch for ch in text if ch not in exclude) 55 | def lower(text): 56 | return text.lower() 57 | return white_space_fix(remove_articles(remove_punc(lower(s)))) 58 | 59 | def get_tokens(s): 60 | if not s: return [] 61 | return normalize_answer(s).split() 62 | 63 | def compute_exact(a_gold, a_pred): 64 | return int(normalize_answer(a_gold) == normalize_answer(a_pred)) 65 | 66 | def compute_f1(a_gold, a_pred): 67 | gold_toks = get_tokens(a_gold) 68 | pred_toks = get_tokens(a_pred) 69 | common = collections.Counter(gold_toks) & collections.Counter(pred_toks) 70 | num_same = sum(common.values()) 71 | if len(gold_toks) == 0 or len(pred_toks) == 0: 72 | # If either is no-answer, then F1 is 1 if they agree, 0 otherwise 73 | return int(gold_toks == pred_toks) 74 | if num_same == 0: 75 | return 0 76 | precision = 1.0 * num_same / len(pred_toks) 77 | recall = 1.0 * num_same / len(gold_toks) 78 | f1 = (2 * precision * recall) / (precision + recall) 79 | return f1 80 | 81 | def get_raw_scores(dataset, preds): 82 | exact_scores = {} 83 | f1_scores = {} 84 | for article in dataset: 85 | for p in article['paragraphs']: 86 | for qa in p['qas']: 87 | qid = qa['id'] 88 | gold_answers = [a['text'] for a in qa['answers'] 89 | if normalize_answer(a['text'])] 90 | if not gold_answers: 91 | # For unanswerable questions, only correct answer is empty string 92 | gold_answers = [''] 93 | if qid not in preds: 94 | print('Missing prediction for %s' % qid) 95 | continue 96 | a_pred = preds[qid] 97 | # Take max over all gold answers 98 | exact_scores[qid] = max(compute_exact(a, a_pred) for a in gold_answers) 99 | f1_scores[qid] = max(compute_f1(a, a_pred) for a in gold_answers) 100 | return exact_scores, f1_scores 101 | 102 | def apply_no_ans_threshold(scores, na_probs, qid_to_has_ans, na_prob_thresh): 103 | new_scores = {} 104 | for qid, s in scores.items(): 105 | pred_na = na_probs[qid] > na_prob_thresh 106 | if pred_na: 107 | new_scores[qid] = float(not qid_to_has_ans[qid]) 108 | else: 109 | new_scores[qid] = s 110 | return new_scores 111 | 112 | def make_eval_dict(exact_scores, f1_scores, qid_list=None): 113 | if not qid_list: 114 | total = len(exact_scores) 115 | return collections.OrderedDict([ 116 | ('exact', 100.0 * sum(exact_scores.values()) / total), 117 | ('f1', 100.0 * sum(f1_scores.values()) / total), 118 | ('total', total), 119 | ]) 120 | else: 121 | total = len(qid_list) 122 | return collections.OrderedDict([ 123 | ('exact', 100.0 * sum(exact_scores[k] for k in qid_list) / total), 124 | ('f1', 100.0 * sum(f1_scores[k] for k in qid_list) / total), 125 | ('total', total), 126 | ]) 127 | 128 | def merge_eval(main_eval, new_eval, prefix): 129 | for k in new_eval: 130 | main_eval['%s_%s' % (prefix, k)] = new_eval[k] 131 | 132 | def plot_pr_curve(precisions, recalls, out_image, title): 133 | plt.step(recalls, precisions, color='b', alpha=0.2, where='post') 134 | plt.fill_between(recalls, precisions, step='post', alpha=0.2, color='b') 135 | plt.xlabel('Recall') 136 | plt.ylabel('Precision') 137 | plt.xlim([0.0, 1.05]) 138 | plt.ylim([0.0, 1.05]) 139 | plt.title(title) 140 | plt.savefig(out_image) 141 | plt.clf() 142 | 143 | def make_precision_recall_eval(scores, na_probs, num_true_pos, qid_to_has_ans, 144 | out_image=None, title=None): 145 | qid_list = sorted(na_probs, key=lambda k: na_probs[k]) 146 | true_pos = 0.0 147 | cur_p = 1.0 148 | cur_r = 0.0 149 | precisions = [1.0] 150 | recalls = [0.0] 151 | avg_prec = 0.0 152 | for i, qid in enumerate(qid_list): 153 | if qid_to_has_ans[qid]: 154 | true_pos += scores[qid] 155 | cur_p = true_pos / float(i+1) 156 | cur_r = true_pos / float(num_true_pos) 157 | if i == len(qid_list) - 1 or na_probs[qid] != na_probs[qid_list[i+1]]: 158 | # i.e., if we can put a threshold after this point 159 | avg_prec += cur_p * (cur_r - recalls[-1]) 160 | precisions.append(cur_p) 161 | recalls.append(cur_r) 162 | if out_image: 163 | plot_pr_curve(precisions, recalls, out_image, title) 164 | return {'ap': 100.0 * avg_prec} 165 | 166 | def run_precision_recall_analysis(main_eval, exact_raw, f1_raw, na_probs, 167 | qid_to_has_ans, out_image_dir): 168 | if out_image_dir and not os.path.exists(out_image_dir): 169 | os.makedirs(out_image_dir) 170 | num_true_pos = sum(1 for v in qid_to_has_ans.values() if v) 171 | if num_true_pos == 0: 172 | return 173 | pr_exact = make_precision_recall_eval( 174 | exact_raw, na_probs, num_true_pos, qid_to_has_ans, 175 | out_image=os.path.join(out_image_dir, 'pr_exact.png'), 176 | title='Precision-Recall curve for Exact Match score') 177 | pr_f1 = make_precision_recall_eval( 178 | f1_raw, na_probs, num_true_pos, qid_to_has_ans, 179 | out_image=os.path.join(out_image_dir, 'pr_f1.png'), 180 | title='Precision-Recall curve for F1 score') 181 | oracle_scores = {k: float(v) for k, v in qid_to_has_ans.items()} 182 | pr_oracle = make_precision_recall_eval( 183 | oracle_scores, na_probs, num_true_pos, qid_to_has_ans, 184 | out_image=os.path.join(out_image_dir, 'pr_oracle.png'), 185 | title='Oracle Precision-Recall curve (binary task of HasAns vs. NoAns)') 186 | merge_eval(main_eval, pr_exact, 'pr_exact') 187 | merge_eval(main_eval, pr_f1, 'pr_f1') 188 | merge_eval(main_eval, pr_oracle, 'pr_oracle') 189 | 190 | def histogram_na_prob(na_probs, qid_list, image_dir, name): 191 | if not qid_list: 192 | return 193 | x = [na_probs[k] for k in qid_list] 194 | weights = np.ones_like(x) / float(len(x)) 195 | plt.hist(x, weights=weights, bins=20, range=(0.0, 1.0)) 196 | plt.xlabel('Model probability of no-answer') 197 | plt.ylabel('Proportion of dataset') 198 | plt.title('Histogram of no-answer probability: %s' % name) 199 | plt.savefig(os.path.join(image_dir, 'na_prob_hist_%s.png' % name)) 200 | plt.clf() 201 | 202 | def find_best_thresh(preds, scores, na_probs, qid_to_has_ans): 203 | num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k]) 204 | cur_score = num_no_ans 205 | best_score = cur_score 206 | best_thresh = 0.0 207 | qid_list = sorted(na_probs, key=lambda k: na_probs[k]) 208 | for i, qid in enumerate(qid_list): 209 | if qid not in scores: continue 210 | if qid_to_has_ans[qid]: 211 | diff = scores[qid] 212 | else: 213 | if preds[qid]: 214 | diff = -1 215 | else: 216 | diff = 0 217 | cur_score += diff 218 | if cur_score > best_score: 219 | best_score = cur_score 220 | best_thresh = na_probs[qid] 221 | return 100.0 * best_score / len(scores), best_thresh 222 | 223 | def find_all_best_thresh(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans): 224 | best_exact, exact_thresh = find_best_thresh(preds, exact_raw, na_probs, qid_to_has_ans) 225 | best_f1, f1_thresh = find_best_thresh(preds, f1_raw, na_probs, qid_to_has_ans) 226 | main_eval['best_exact'] = best_exact 227 | main_eval['best_exact_thresh'] = exact_thresh 228 | main_eval['best_f1'] = best_f1 229 | main_eval['best_f1_thresh'] = f1_thresh 230 | 231 | def main(): 232 | with open(OPTS.data_file) as f: 233 | dataset_json = json.load(f) 234 | dataset = dataset_json['data'] 235 | with open(OPTS.pred_file) as f: 236 | preds = json.load(f) 237 | if OPTS.na_prob_file: 238 | with open(OPTS.na_prob_file) as f: 239 | na_probs = json.load(f) 240 | else: 241 | na_probs = {k: 0.0 for k in preds} 242 | qid_to_has_ans = make_qid_to_has_ans(dataset) # maps qid to True/False 243 | has_ans_qids = [k for k, v in qid_to_has_ans.items() if v] 244 | no_ans_qids = [k for k, v in qid_to_has_ans.items() if not v] 245 | exact_raw, f1_raw = get_raw_scores(dataset, preds) 246 | exact_thresh = apply_no_ans_threshold(exact_raw, na_probs, qid_to_has_ans, 247 | OPTS.na_prob_thresh) 248 | f1_thresh = apply_no_ans_threshold(f1_raw, na_probs, qid_to_has_ans, 249 | OPTS.na_prob_thresh) 250 | out_eval = make_eval_dict(exact_thresh, f1_thresh) 251 | if has_ans_qids: 252 | has_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=has_ans_qids) 253 | merge_eval(out_eval, has_ans_eval, 'HasAns') 254 | if no_ans_qids: 255 | no_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=no_ans_qids) 256 | merge_eval(out_eval, no_ans_eval, 'NoAns') 257 | if OPTS.na_prob_file: 258 | find_all_best_thresh(out_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans) 259 | if OPTS.na_prob_file and OPTS.out_image_dir: 260 | run_precision_recall_analysis(out_eval, exact_raw, f1_raw, na_probs, 261 | qid_to_has_ans, OPTS.out_image_dir) 262 | histogram_na_prob(na_probs, has_ans_qids, OPTS.out_image_dir, 'hasAns') 263 | histogram_na_prob(na_probs, no_ans_qids, OPTS.out_image_dir, 'noAns') 264 | if OPTS.out_file: 265 | with open(OPTS.out_file, 'w') as f: 266 | json.dump(out_eval, f) 267 | else: 268 | print(json.dumps(out_eval, indent=2)) 269 | 270 | if __name__ == '__main__': 271 | OPTS = parse_args() 272 | if OPTS.out_image_dir: 273 | import matplotlib 274 | matplotlib.use('Agg') 275 | import matplotlib.pyplot as plt 276 | main() 277 | 278 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /run_squad.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import 2 | from __future__ import division 3 | from __future__ import print_function 4 | 5 | import sys 6 | sys.path.append('xlnet') # walkaround due to submodule absolute import... 7 | 8 | import collections 9 | import os 10 | import os.path 11 | import json 12 | import pickle 13 | import time 14 | 15 | import tensorflow as tf 16 | import numpy as np 17 | import sentencepiece as sp 18 | 19 | from xlnet import xlnet 20 | import function_builder 21 | import prepro_utils 22 | import model_utils 23 | 24 | MAX_FLOAT = 1e30 25 | MIN_FLOAT = -1e30 26 | 27 | flags = tf.flags 28 | FLAGS = flags.FLAGS 29 | 30 | flags.DEFINE_string("data_dir", None, "Data directory where raw data located.") 31 | flags.DEFINE_string("output_dir", None, "Output directory where processed data located.") 32 | flags.DEFINE_string("model_dir", None, "Model directory where checkpoints located.") 33 | flags.DEFINE_string("export_dir", None, "Export directory where saved model located.") 34 | 35 | flags.DEFINE_string("task_name", default=None, help="The name of the task to train.") 36 | flags.DEFINE_string("model_config_path", default=None, help="Config file of the pre-trained model.") 37 | flags.DEFINE_string("init_checkpoint", default=None, help="Initial checkpoint of the pre-trained model.") 38 | flags.DEFINE_string("spiece_model_file", default=None, help="Sentence Piece model path.") 39 | flags.DEFINE_bool("overwrite_data", default=False, help="If False, will use cached data if available.") 40 | flags.DEFINE_integer("random_seed", default=100, help="Random seed for weight initialzation.") 41 | flags.DEFINE_string("predict_tag", None, "Predict tag for predict result tracking.") 42 | 43 | flags.DEFINE_bool("do_train", default=False, help="Whether to run training.") 44 | flags.DEFINE_bool("do_predict", default=False, help="Whether to run prediction.") 45 | flags.DEFINE_bool("do_export", default=False, help="Whether to run exporting.") 46 | 47 | flags.DEFINE_enum("init", default="normal", enum_values=["normal", "uniform"], help="Initialization method.") 48 | flags.DEFINE_float("init_std", default=0.02, help="Initialization std when init is normal.") 49 | flags.DEFINE_float("init_range", default=0.1, help="Initialization std when init is uniform.") 50 | flags.DEFINE_bool("init_global_vars", default=False, help="If true, init all global vars. If false, init trainable vars only.") 51 | 52 | flags.DEFINE_bool("lower_case", default=False, help="Enable lower case nor not.") 53 | flags.DEFINE_integer("doc_stride", default=128, help="Doc stride") 54 | flags.DEFINE_integer("max_seq_length", default=512, help="Max sequence length") 55 | flags.DEFINE_integer("max_query_length", default=64, help="Max query length") 56 | flags.DEFINE_integer("max_answer_length", default=64, help="Max answer length") 57 | flags.DEFINE_integer("train_batch_size", default=48, help="Total batch size for training.") 58 | flags.DEFINE_integer("predict_batch_size", default=32, help="Total batch size for predict.") 59 | 60 | flags.DEFINE_integer("train_steps", default=8000, help="Number of training steps") 61 | flags.DEFINE_integer("warmup_steps", default=0, help="number of warmup steps") 62 | flags.DEFINE_integer("max_save", default=5, help="Max number of checkpoints to save. Use 0 to save all.") 63 | flags.DEFINE_integer("save_steps", default=1000, help="Save the model for every save_steps. If None, not to save any model.") 64 | flags.DEFINE_integer("shuffle_buffer", default=2048, help="Buffer size used for shuffle.") 65 | 66 | flags.DEFINE_integer("n_best_size", default=5, help="n best size for predictions") 67 | flags.DEFINE_integer("start_n_top", default=5, help="Beam size for span start.") 68 | flags.DEFINE_integer("end_n_top", default=5, help="Beam size for span end.") 69 | flags.DEFINE_string("target_eval_key", default="best_f1", help="Use has_ans_f1 for Model I.") 70 | 71 | flags.DEFINE_bool("use_bfloat16", default=False, help="Whether to use bfloat16.") 72 | flags.DEFINE_float("dropout", default=0.1, help="Dropout rate.") 73 | flags.DEFINE_float("dropatt", default=0.1, help="Attention dropout rate.") 74 | flags.DEFINE_integer("clamp_len", default=-1, help="Clamp length") 75 | flags.DEFINE_string("summary_type", default="last", help="Method used to summarize a sequence into a vector.") 76 | 77 | flags.DEFINE_float("learning_rate", default=3e-5, help="initial learning rate") 78 | flags.DEFINE_float("min_lr_ratio", default=0.0, help="min lr ratio for cos decay.") 79 | flags.DEFINE_float("lr_layer_decay_rate", default=0.75, help="lr[L] = learning_rate, lr[l-1] = lr[l] * lr_layer_decay_rate.") 80 | flags.DEFINE_float("clip", default=1.0, help="Gradient clipping") 81 | flags.DEFINE_float("weight_decay", default=0.00, help="Weight decay rate") 82 | flags.DEFINE_float("adam_epsilon", default=1e-6, help="Adam epsilon") 83 | flags.DEFINE_string("decay_method", default="poly", help="poly or cos") 84 | 85 | flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.") 86 | flags.DEFINE_integer("num_hosts", 1, "How many TPU hosts.") 87 | flags.DEFINE_integer("num_core_per_host", 1, "Total number of TPU cores to use.") 88 | flags.DEFINE_string("tpu_job_name", None, "TPU worker job name.") 89 | flags.DEFINE_string("tpu", None, "The Cloud TPU name to use for training.") 90 | flags.DEFINE_string("tpu_zone", None, "GCE zone where the Cloud TPU is located in.") 91 | flags.DEFINE_string("gcp_project", None, "Project name for the Cloud TPU-enabled project.") 92 | flags.DEFINE_string("master", None, "TensorFlow master URL") 93 | flags.DEFINE_integer("iterations", 1000, "number of iterations per TPU training loop.") 94 | 95 | class InputExample(object): 96 | """A single SQuAD example.""" 97 | def __init__(self, 98 | qas_id, 99 | question_text, 100 | paragraph_text, 101 | orig_answer_text=None, 102 | start_position=None, 103 | is_impossible=False): 104 | self.qas_id = qas_id 105 | self.question_text = question_text 106 | self.paragraph_text = paragraph_text 107 | self.orig_answer_text = orig_answer_text 108 | self.start_position = start_position 109 | self.is_impossible = is_impossible 110 | 111 | def __str__(self): 112 | return self.__repr__() 113 | 114 | def __repr__(self): 115 | s = "qas_id: %s" % (prepro_utils.printable_text(self.qas_id)) 116 | s += ", question_text: %s" % (prepro_utils.printable_text(self.question_text)) 117 | s += ", paragraph_text: [%s]" % (" ".join(self.paragraph_text)) 118 | if self.start_position: 119 | s += ", start_position: %d" % (self.start_position) 120 | s += ", is_impossible: %r" % (self.is_impossible) 121 | return s 122 | 123 | class InputFeatures(object): 124 | """A single SQuAD feature.""" 125 | def __init__(self, 126 | unique_id, 127 | qas_id, 128 | doc_idx, 129 | token2char_raw_start_index, 130 | token2char_raw_end_index, 131 | token2doc_index, 132 | input_ids, 133 | input_mask, 134 | p_mask, 135 | segment_ids, 136 | cls_index, 137 | para_length, 138 | start_position=None, 139 | end_position=None, 140 | is_impossible=None): 141 | self.unique_id = unique_id 142 | self.qas_id = qas_id 143 | self.doc_idx = doc_idx 144 | self.token2char_raw_start_index = token2char_raw_start_index 145 | self.token2char_raw_end_index = token2char_raw_end_index 146 | self.token2doc_index = token2doc_index 147 | self.input_ids = input_ids 148 | self.input_mask = input_mask 149 | self.p_mask = p_mask 150 | self.segment_ids = segment_ids 151 | self.cls_index = cls_index 152 | self.para_length = para_length 153 | self.start_position = start_position 154 | self.end_position = end_position 155 | self.is_impossible = is_impossible 156 | 157 | class OutputResult(object): 158 | """A single SQuAD result.""" 159 | def __init__(self, 160 | unique_id, 161 | answer_prob, 162 | start_prob, 163 | start_index, 164 | end_prob, 165 | end_index): 166 | self.unique_id = unique_id 167 | self.answer_prob = answer_prob 168 | self.start_prob = start_prob 169 | self.start_index = start_index 170 | self.end_prob = end_prob 171 | self.end_index = end_index 172 | 173 | class SquadPipeline(object): 174 | """Pipeline for SQuAD dataset.""" 175 | def __init__(self, 176 | data_dir, 177 | task_name): 178 | self.data_dir = data_dir 179 | self.task_name = task_name 180 | 181 | def get_train_examples(self): 182 | """Gets a collection of `InputExample`s for the train set.""" 183 | data_path = os.path.join(self.data_dir, "train-{0}.json".format(self.task_name)) 184 | data_list = self._read_json(data_path) 185 | example_list = self._get_example(data_list, True) 186 | return example_list 187 | 188 | def get_dev_examples(self): 189 | """Gets a collection of `InputExample`s for the dev set.""" 190 | data_path = os.path.join(self.data_dir, "dev-{0}.json".format(self.task_name)) 191 | data_list = self._read_json(data_path) 192 | example_list = self._get_example(data_list, False) 193 | return example_list 194 | 195 | def _read_json(self, 196 | data_path): 197 | if os.path.exists(data_path): 198 | with open(data_path, "r") as file: 199 | data_list = json.load(file)["data"] 200 | return data_list 201 | else: 202 | raise FileNotFoundError("data path not found: {0}".format(data_path)) 203 | 204 | def _get_example(self, 205 | data_list, 206 | is_training): 207 | examples = [] 208 | for entry in data_list: 209 | for paragraph in entry["paragraphs"]: 210 | paragraph_text = paragraph["context"] 211 | 212 | for qa in paragraph["qas"]: 213 | qas_id = qa["id"] 214 | question_text = qa["question"] 215 | start_position = None 216 | orig_answer_text = None 217 | is_impossible = False 218 | 219 | if is_training: 220 | is_impossible = qa["is_impossible"] 221 | if (len(qa["answers"]) != 1) and (not is_impossible): 222 | raise ValueError("For training, each question should have exactly 1 answer.") 223 | 224 | if not is_impossible: 225 | answer = qa["answers"][0] 226 | orig_answer_text = answer["text"] 227 | start_position = answer["answer_start"] 228 | else: 229 | start_position = -1 230 | orig_answer_text = "" 231 | 232 | example = InputExample( 233 | qas_id=qas_id, 234 | question_text=question_text, 235 | paragraph_text=paragraph_text, 236 | orig_answer_text=orig_answer_text, 237 | start_position=start_position, 238 | is_impossible=is_impossible) 239 | 240 | examples.append(example) 241 | 242 | return examples 243 | 244 | class XLNetTokenizer(object): 245 | """Default text tokenizer for XLNet""" 246 | def __init__(self, 247 | sp_model_file, 248 | lower_case=False): 249 | """Construct XLNet tokenizer""" 250 | self.sp_processor = sp.SentencePieceProcessor() 251 | self.sp_processor.Load(sp_model_file) 252 | self.lower_case = lower_case 253 | 254 | def tokenize(self, 255 | text): 256 | """Tokenize text for XLNet""" 257 | processed_text = prepro_utils.preprocess_text(text, lower=self.lower_case) 258 | tokenized_pieces = prepro_utils.encode_pieces(self.sp_processor, processed_text, return_unicode=False) 259 | return tokenized_pieces 260 | 261 | def encode(self, 262 | text): 263 | """Encode text for XLNet""" 264 | processed_text = prepro_utils.preprocess_text(text, lower=self.lower_case) 265 | encoded_ids = prepro_utils.encode_ids(self.sp_processor, processed_text) 266 | return encoded_ids 267 | 268 | def token_to_id(self, 269 | token): 270 | """Convert token to id for XLNet""" 271 | return self.sp_processor.PieceToId(token) 272 | 273 | def id_to_token(self, 274 | id): 275 | """Convert id to token for XLNet""" 276 | return self.sp_processor.IdToPiece(id) 277 | 278 | def tokens_to_ids(self, 279 | tokens): 280 | """Convert tokens to ids for XLNet""" 281 | return [self.sp_processor.PieceToId(token) for token in tokens] 282 | 283 | def ids_to_tokens(self, 284 | ids): 285 | """Convert ids to tokens for XLNet""" 286 | return [self.sp_processor.IdToPiece(id) for id in ids] 287 | 288 | class XLNetExampleProcessor(object): 289 | """Default example processor for XLNet""" 290 | def __init__(self, 291 | max_seq_length, 292 | max_query_length, 293 | doc_stride, 294 | tokenizer): 295 | """Construct XLNet example processor""" 296 | self.special_vocab_list = ["", " "])
581 | p_mask.append(0)
582 |
583 | doc_para_length = len(input_tokens)
584 |
585 | input_tokens.append(" "])
587 | p_mask.append(1)
588 |
589 | # We put P before Q because during pretraining, B is always shorter than A
590 | for query_token in query_tokens:
591 | input_tokens.append(query_token)
592 | segment_ids.append(self.segment_vocab_map["", "
"])
593 | p_mask.append(1)
594 |
595 | input_tokens.append("
"])
597 | p_mask.append(1)
598 |
599 | cls_index = len(input_tokens)
600 |
601 | input_tokens.append("