├── README.md ├── code ├── eval_ans_gen.py ├── eval_ans_gen.sh ├── eval_ans_gen_subsample.sh ├── eval_span_pred.py ├── eval_span_pred.sh ├── eval_zero_shot.sh ├── leaderboard.py ├── leaderboard.sh ├── models.py ├── optimization.py ├── run_ans_generation.sh ├── run_ans_generation_model.py ├── run_ans_generation_subsample.sh ├── run_span_pred.py ├── run_span_pred.sh └── utils.py ├── data ├── final_dev_ans_gen.json ├── final_test_ans_gen.json ├── final_train_ans_gen.json ├── final_train_ans_gen_500completed.json └── final_train_ans_gen_500original.json ├── env.yml └── output ├── README.md ├── pred_dev.json └── pred_test.json /README.md: -------------------------------------------------------------------------------- 1 | # Project / Paper Introduction 2 | This is the project repo for our EMNLP'21 paper: https://arxiv.org/abs/2104.08350 3 | 4 | Here, we provide brief descriptions of the final data and detailed instructions to reproduce results in our paper. For more details, please refer to the paper. 5 | 6 | # Data 7 | Final data used for the experiments are saved in `./data/` folder with train/dev/test splits. Most data fields are straightforward. Just a few notes, 8 | - **question_event**: this field is not provided by annotators nor used for our experiments. We simply use some heuristic rules based on POS tags to extract possible events in the questions. Users are encourages to try alternative tools such semantic role labeling. 9 | - **original_events** and **indices** are the annotator-provided event triggers plus their indices in the context. 10 | - **answer_texts** and **answer_indices** (in train and dev) are the annotator-provided answers plus their indices in the context. 11 | - You may find duplicate passageID. This is because we have multiple sets (collections) of passages when launching our annotation tasks on MTurk. The passageID corresponds to the index of the passage in a particular set. We don't use passageID for any experiments or analysis. If you need a unique identifier for passage, simply use their raw text. 12 | 13 | #### __Please Note__: the evaluation script below (II) only works for the dev set. Please refer to Section III for submission to our leaderboard: https://eventqa.github.io 14 | 15 | 16 | # Models 17 | ## I. Install packages. 18 | We list the packages in our environment in env.yml file for your reference. Below are a few key packages. 19 | - python=3.8.5 20 | - pytorch=1.6.0 21 | - transformers=3.1.0 22 | - cudatoolkit=10.1.243 23 | - apex=0.1 24 | 25 | To install apex, you can either follow official instruction: https://github.com/NVIDIA/apex or conda: https://anaconda.org/conda-forge/nvidia-apex 26 | 27 | ## II. Replicate results in our paper. 28 | ### 1. Download trained models. 29 | For reproduction purpose, we release all trained models. 30 | - Download link: https://drive.google.com/drive/folders/1bTCb4gBUCaNrw2chleD4RD9JP1_DOWjj?usp=sharing. 31 | - We only provide models with the best "hyper-parameters", and each comes with three random seeds: 5, 7, 23. 32 | - Make several directories to save models `./output/`, `./output/facebook/` and `./output/allenai/`. 33 | - For BART models, download them into `./output/facebook/`. 34 | - For UnifiedQA models, download them into `./output/allenai/`. 35 | - All other models can be saved in `./output/` directly. These ensure evaluation scripts run properly below. 36 | 37 | ### 2. Zero-shot performances in Table 3. 38 | Run `bash ./code/eval_zero_shot.sh`. Model options are provided in the script. 39 | 40 | ### 3. Generative QA Fine-tuning performances in Table 3. 41 | Run `bash ./code/eval_ans_gen.sh`. Make sure the following arguments are set correctly in the script. 42 | - Model Options provided in the script 43 | - Set suffix="" 44 | - Set `lrs` and `batch` according to model options. You can find these numbers in Appendix G of the paper. 45 | 46 | ### 4. Figure 6: UnifiedQA-large model trained with sub-samples. 47 | Run bash ./code/eval_ans_gen.sh`. Make sure the following arguments are set correctly in the script. 48 | - `model="allenai/unifiedqa-t5-large"` 49 | - `suffix={"_500" | "_1000" | "_2000" | "_3000" | "_4000"}` 50 | - Set `lrs` and `batch` accordingly. You can find these information in the folder name containing the trained model objects. 51 | 52 | ### 5. Table 4: 500 original annotations v.s. completed 53 | - `bash ./code/eval_ans_gen.sh` with `model="allenai/unifiedqa-t5-large` and `suffix="_500original` 54 | - `bash ./code/eval_ans_gen.sh` with `model="allenai/unifiedqa-t5-large` and `suffix="_500completed` 55 | - Set `lrs` and `batch` accordingly again. 56 | 57 | ### 6. Extractive QA Fine-tuning performances in Table 3. 58 | Simply run `bash ./code/eval_span_pred.sh` as it is. 59 | 60 | ### 7. Figure 8: Extractive QA Fine-tuning performances by changing positive weights. 61 | - Run `bash ./code/eval_span_pred.sh`. 62 | - Set `pw`, `lrs` and `batch` according to model folder names again. 63 | 64 | 65 | ## III. Submission to ESTER Leaderboard 66 | - Set `model_dir` to your target models 67 | - Run `leaderboard.sh`, which outputs `pred_dev.json` and `pred_test.json` under `./output` 68 | - If you write your own code to output predictions, make sure they follow our original sample order. 69 | - Email `pred_test.json` to us following in the format specified here: https://eventqa.github.io 70 | Sample outputs (using one of our UnifiedQA-large models) are provided under `./output` 71 | 72 | 73 | ## IV. Model Training 74 | We also provide the model training scripts below. 75 | 76 | ### 1. Generative QA: Fine-tuning in Table 3. 77 | - Run `bash ./code/run_ans_generation.sh`. 78 | - Model options and hyper-parameter search range are provided in the script. 79 | - We use `--fp16` argument to activate apex for GPU memory efficient training except for UnifiedQA-t5-large (trained on A100 GPU). 80 | 81 | ### 2. Figure 6: UnifiedQA-large model trained with sub-samples. 82 | - Run `bash ./code/run_ans_gen_subsample.sh`. 83 | - Set `sample_size` variable accordingly in the script. 84 | 85 | ### 3. Table 4: 500 original annotations v.s. completed 86 | - Run `bash ./code/run_ans_gen.sh` with `model="allenai/unifiedqa-t5-large` and `suffix="_500original` 87 | - Run `bash ./code/run_ans_gen.sh` with `model="allenai/unifiedqa-t5-large` and `suffix="_500completed` 88 | 89 | ### 4. Extractive QA Fine-tuning in Table 3 + Figure 8 90 | Simply run `bash ./code/run_span_pred.sh` as it is. 91 | 92 | 93 | 94 | -------------------------------------------------------------------------------- /code/eval_ans_gen.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """BERT finetuning runner.""" 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import os 23 | import logging 24 | import argparse 25 | import random 26 | from tqdm import tqdm, trange 27 | import numpy as np 28 | import torch 29 | from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler 30 | from transformers import AutoTokenizer, T5ForConditionalGeneration, BartForConditionalGeneration 31 | from utils import * 32 | from optimization import * 33 | from pathlib import Path 34 | import re 35 | from collections import Counter 36 | 37 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 38 | datefmt='%m/%d/%Y %H:%M:%S', 39 | level=logging.INFO) 40 | logger = logging.getLogger(__name__) 41 | os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" 42 | PYTORCH_PRETRAINED_ROBERTA_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_ROBERTA_CACHE', 43 | Path.home() / '.pytorch_pretrained_roberta')) 44 | 45 | def main(): 46 | parser = argparse.ArgumentParser() 47 | ## Required parameters 48 | parser.add_argument("--data_dir", 49 | default=None, 50 | type=str, 51 | required=True, 52 | help="The input data dir. Should contain the .json files (or other data files) for the task.") 53 | parser.add_argument("--model", default=None, type=str, required=True, 54 | help="pre-trained model selected in the list: roberta-base, " 55 | "roberta-large, bert-base, bert-large. ") 56 | parser.add_argument("--task_name", 57 | default=None, 58 | type=str, 59 | required=True, 60 | help="The name of the task to train.") 61 | parser.add_argument("--file_suffix", 62 | default=None, 63 | type=str, 64 | required=True, 65 | help="unique identifier for data file") 66 | ## Other parameters 67 | parser.add_argument("--max_seq_length", 68 | default=320, # 8 * 8 * 5 69 | type=int, 70 | help="The maximum total input sequence length after WordPiece tokenization. \n" 71 | "Sequences longer than this will be truncated, and sequences shorter \n" 72 | "than this will be padded.") 73 | parser.add_argument("--model_dir", 74 | type=str, 75 | help="saved model dir", 76 | default="") 77 | parser.add_argument("--do_lower_case", 78 | action='store_true', 79 | help="Set this flag if you are using an uncased model.") 80 | parser.add_argument("--eval_batch_size", 81 | default=8, 82 | type=int, 83 | help="Total batch size for eval.") 84 | parser.add_argument("--mlp_hid_size", 85 | default=64, 86 | type=int, 87 | help="hid dimension for MLP layer.") 88 | parser.add_argument("--learning_rate", 89 | default=5e-5, 90 | type=float, 91 | help="The initial learning rate for Adam.") 92 | parser.add_argument("--num_train_epochs", 93 | default=3.0, 94 | type=float, 95 | help="Total number of training epochs to perform.") 96 | parser.add_argument("--warmup_proportion", 97 | default=0.1, 98 | type=float, 99 | help="Proportion of training to perform linear learning rate warmup for. " 100 | "E.g., 0.1 = 10%% of training.") 101 | parser.add_argument("--no_cuda", 102 | action='store_true', 103 | help="Whether not to use CUDA when available") 104 | parser.add_argument("--local_rank", 105 | type=int, 106 | default=-1, 107 | help="local_rank for distributed training on gpus") 108 | parser.add_argument('--seed', 109 | type=int, 110 | default=42, 111 | help="random seed for initialization") 112 | parser.add_argument('--gradient_accumulation_steps', 113 | type=int, 114 | default=1, 115 | help="Number of updates steps to accumulate before performing a backward/update pass.") 116 | parser.add_argument('--fp16', 117 | action='store_true', 118 | help="Whether to use 16-bit float precision instead of 32-bit") 119 | parser.add_argument('--loss_scale', 120 | type=float, default=0, 121 | help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n" 122 | "0 (default value): dynamic loss scaling.\n" 123 | "Positive power of 2: static loss scaling value.\n") 124 | parser.add_argument('--cuda', 125 | type=str, 126 | default="", 127 | help="cuda index") 128 | parser.add_argument('--device_num', 129 | type=str, 130 | default="0", 131 | help="cuda device number") 132 | args = parser.parse_args() 133 | 134 | os.environ["CUDA_VISIBLE_DEVICES"] = args.device_num 135 | 136 | if args.local_rank == -1 or args.no_cuda: 137 | device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 138 | n_gpu = torch.cuda.device_count() 139 | else: 140 | torch.cuda.set_device(args.local_rank) 141 | device = torch.device("cuda", args.local_rank) 142 | n_gpu = 1 143 | # Initializes the distributed backend which will take care of sychronizing nodes/GPUs 144 | torch.distributed.init_process_group(backend='nccl') 145 | 146 | logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format( 147 | device, n_gpu, bool(args.local_rank != -1), args.fp16)) 148 | 149 | if args.gradient_accumulation_steps < 1: 150 | raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format( 151 | args.gradient_accumulation_steps)) 152 | 153 | # fix all random seeds 154 | random.seed(args.seed) 155 | np.random.seed(args.seed) 156 | torch.manual_seed(args.seed) 157 | 158 | if n_gpu > 0: 159 | torch.cuda.manual_seed_all(args.seed) 160 | 161 | task_name = args.task_name.lower() 162 | logger.info("current task is " + str(task_name)) 163 | 164 | # construct model 165 | if args.model_dir: 166 | logger.info(args.model_dir) 167 | model_state_dict = torch.load(args.model_dir + "pytorch_model.bin") 168 | tokenizer = AutoTokenizer.from_pretrained(args.model, state_dict=model_state_dict) 169 | if 't5' in args.model: 170 | model = T5ForConditionalGeneration.from_pretrained(args.model, state_dict=model_state_dict) 171 | if 'bart' in args.model: 172 | model = BartForConditionalGeneration.from_pretrained(args.model, state_dict=model_state_dict) 173 | else: 174 | tokenizer = AutoTokenizer.from_pretrained(args.model) 175 | model = T5ForConditionalGeneration.from_pretrained(args.model) 176 | 177 | model.to(device) 178 | 179 | # Prepare optimizer 180 | param_optimizer = list(model.named_parameters()) 181 | no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] 182 | optimizer_grouped_parameters = [ 183 | {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01}, 184 | {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 185 | ] 186 | 187 | if args.fp16: 188 | try: 189 | from apex.optimizers import FusedAdam 190 | from apex import amp 191 | except ImportError: 192 | raise ImportError( 193 | "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.") 194 | optimizer = FusedAdam(optimizer_grouped_parameters, 195 | lr=args.learning_rate, 196 | bias_correction=False) 197 | model, optimizer = amp.initialize(model, optimizer, opt_level="O1") 198 | else: 199 | optimizer = BertAdam(optimizer_grouped_parameters, 200 | lr=args.learning_rate, 201 | warmup=args.warmup_proportion, 202 | t_total=1) 203 | 204 | if n_gpu > 1: 205 | model = torch.nn.DataParallel(model) 206 | 207 | for split in ['dev']: 208 | eval_data = load_data(args.data_dir, "final_%s" % split, args.file_suffix) 209 | eval_features = convert_to_features_ans_gen(eval_data, ";", eval=True) 210 | 211 | eval_inputs = select_field(eval_features, 'inputs') 212 | eval_encoded_inputs = tokenizer(eval_inputs, padding=True, truncation=True, return_tensors="pt") 213 | 214 | eval_input_ids = eval_encoded_inputs['input_ids'] 215 | eval_input_mask = eval_encoded_inputs['attention_mask'] 216 | 217 | eval_key_indices = torch.tensor(list(range(len(eval_inputs))), dtype=torch.long) 218 | eval_types = select_field(eval_features, 'types') 219 | 220 | eval_labels = select_field(eval_features, 'labels') 221 | eval_encoded_outputs = tokenizer(eval_labels, padding=True, truncation=True, return_tensors="pt") 222 | eval_output_ids = eval_encoded_outputs['input_ids'] 223 | 224 | logger.info("id_size: {}, mask_size: {}, instance_key_size: {}, label_size: {}".format( 225 | eval_input_ids.size(), eval_input_mask.size(), eval_key_indices.size(), eval_output_ids.size())) 226 | 227 | eval_events = select_field(eval_features, 'events') 228 | data = TensorDataset(eval_input_ids, eval_input_mask, eval_key_indices, eval_output_ids) 229 | 230 | # Run prediction for full data 231 | eval_sampler = SequentialSampler(data) 232 | eval_dataloader = DataLoader(data, sampler=eval_sampler, batch_size=args.eval_batch_size) 233 | 234 | preds, golds, events, perplexity = [], [], [], [] 235 | model.eval() 236 | 237 | type_indicators = {'Causal': [], 'Indicative Conditional': [], 'Sub-event': [], 238 | 'Counterfactual Conditional': [], 'Coreference': []} 239 | 240 | for batch in tqdm(eval_dataloader, desc="Evaluating"): 241 | batch = tuple(t.to(device) for t in batch) 242 | 243 | input_ids, input_masks, instance_indices, output_ids = batch 244 | 245 | with torch.no_grad(): 246 | res = model.generate(input_ids, attention_mask=input_masks, max_length=128) 247 | preds.extend([x.split(";") for x in tokenizer.batch_decode(res, skip_special_tokens=True)]) 248 | golds.extend([eval_labels[x].split(";") for x in instance_indices.tolist()]) 249 | events.extend([eval_events[x] for x in instance_indices.tolist()]) 250 | 251 | for i in instance_indices: 252 | for tk in type_indicators: 253 | if eval_types[i] in tk: 254 | type_indicators[tk].append(1) 255 | else: 256 | type_indicators[tk].append(0) 257 | 258 | eval_em, eval_f1 = [], [] 259 | 260 | ems, F1, recl, prec, F1_e, r_e, pr_e, hit1 = [], [], [], [], [], [], [], [] 261 | for pred, gold, event in zip(preds, golds, events): 262 | logger.info(pred) 263 | logger.info(gold) 264 | logger.info(event) 265 | logger.info("="*50) 266 | em = 1.0 if all([p in gold for p in pred]) and all([g in pred for g in gold]) else 0.0 267 | eval_em.append(em) 268 | 269 | # construct unigram counter 270 | pred_counter = Counter([x for s in pred for x in re.sub(r'[^\w\s]', '', s).split(' ')]) 271 | gold_counter = Counter([x for s in gold for x in re.sub(r'[^\w\s]', '', s).split(' ')]) 272 | 273 | rl, pr, f1 = compute_unigram_f1(pred_counter, gold_counter) 274 | F1.append(f1) 275 | recl.append(rl) 276 | prec.append(pr) 277 | 278 | rl, pr, f1 = compute_event_f1(pred, event) 279 | F1_e.append(f1) 280 | r_e.append(rl) 281 | pr_e.append(pr) 282 | 283 | if f1 == 1.0: 284 | ems.append(1.0) 285 | else: 286 | ems.append(0.0) 287 | 288 | hit1.append(compute_hit1(pred[0], event)) 289 | 290 | logger.info("Answer EM is %.4f" % np.mean(eval_em)) 291 | logger.info("Event EM is %.4f" % np.mean(ems)) 292 | logger.info("Event HIT@1 is %.4f" % np.mean(hit1)) 293 | logger.info("Token Recall, Precision, F1 are %.4f, %.4f, %.4f" % 294 | (np.mean(recl), np.mean(prec), np.mean(F1))) 295 | logger.info("Event Recall, Precision, F1 are %.4f, %.4f, %.4f" % 296 | (np.mean(r_e), np.mean(pr_e), np.mean(F1_e))) 297 | 298 | logger.info("=" * 50) 299 | for key, idx in type_indicators.items(): 300 | print("Eval %s questions" % key) 301 | assert len(idx) == len(golds) 302 | logger.info("Total %s questions (%.1f)" % (sum(idx), 100 * sum(idx) / len(golds))) 303 | 304 | ans_ems = [v for k, v in zip(idx, eval_em) if k == 1] 305 | temp_ems = [v for k, v in zip(idx, ems) if k == 1] 306 | 307 | temp_r = [v for k, v in zip(idx, recl) if k == 1] 308 | temp_pr = [v for k, v in zip(idx, prec) if k == 1] 309 | temp_f1s = [v for k, v in zip(idx, F1) if k == 1] 310 | 311 | temp_r_e = [v for k, v in zip(idx, r_e) if k == 1] 312 | temp_pr_e = [v for k, v in zip(idx, pr_e) if k == 1] 313 | temp_f1_e = [v for k, v in zip(idx, F1_e) if k == 1] 314 | 315 | temp_hit1 = [v for k, v in zip(idx, hit1) if k == 1] 316 | 317 | logger.info("Total %s QAs" % len(temp_ems)) 318 | logger.info("Answer EM is %.4f" % np.mean(ans_ems)) 319 | logger.info("Event EM is %.4f" % np.mean(temp_ems)) 320 | logger.info("Event HIT@1 is %.4f" % np.mean(temp_hit1)) 321 | logger.info("Token Recall, Precision, F1 are %.4f, %.4f, %.4f" % 322 | (np.mean(temp_r), np.mean(temp_pr), np.mean(temp_f1s))) 323 | logger.info("Event Recall, Precision, F1 are %.4f, %.4f, %.4f" % 324 | (np.mean(temp_r_e), np.mean(temp_pr_e), np.mean(temp_f1_e))) 325 | logger.info("=" * 20) 326 | 327 | if __name__ == "__main__": 328 | main() 329 | -------------------------------------------------------------------------------- /code/eval_ans_gen.sh: -------------------------------------------------------------------------------- 1 | task="answer-generation" 2 | device="0" 3 | 4 | ###### Model Options ###### 5 | #model="facebook/bart-base" 6 | #model="facebook/bart-large" 7 | #model="t5-base" 8 | model="allenai/unifiedqa-t5-base" 9 | #model="allenai/unifiedqa-t5-large" 10 | 11 | 12 | ###### Additional Model Suffix ###### 13 | #suffix="_500original" 14 | #suffix="_500completed" 15 | suffix="" 16 | 17 | lrs=(5e-5) 18 | batch=(2) 19 | seeds=(5 7 23) 20 | root="./output" 21 | for l in "${lrs[@]}" 22 | do 23 | for s in "${batch[@]}" 24 | do 25 | for seed in "${seeds[@]}" 26 | do 27 | python ./code/eval_ans_gen.py \ 28 | --data_dir "./data/" \ 29 | --model ${model} \ 30 | --task_name ${task} \ 31 | --file_suffix "_ans_gen.json" \ 32 | --device_num ${device} \ 33 | --eval_batch_size 8 \ 34 | --num_train_epochs 10 \ 35 | --max_seq_length 339 \ 36 | --learning_rate ${l} \ 37 | --seed ${seed} \ 38 | --model_dir "${root}/${model}_batch_${s}_lr_${l}_seed_${seed}${suffix}/" 39 | done 40 | done 41 | done 42 | -------------------------------------------------------------------------------- /code/eval_ans_gen_subsample.sh: -------------------------------------------------------------------------------- 1 | task="answer-generation" 2 | lrs=(5e-5 1e-4 2e-4) 3 | batch=(2 4) 4 | seeds=(5 7 23) 5 | 6 | device="0" 7 | #model="google/pegasus-large" 8 | #model="facebook/bart-base" 9 | #model="facebook/bart-large" 10 | #model="t5-base" 11 | #model="allenai/unifiedqa-t5-base" 12 | model="allenai/unifiedqa-t5-large" 13 | 14 | sample_size=500 #1000,2000,3000,4000 15 | 16 | for l in "${lrs[@]}" 17 | do 18 | for s in "${batch[@]}" 19 | do 20 | for seed in "${seeds[@]}" 21 | do 22 | python eval_ans_gen.py \ 23 | --data_dir "./data/" \ 24 | --model ${model} \ 25 | --task_name ${task} \ 26 | --file_suffix "_ans_gen.json" \ 27 | --device_num ${device} \ 28 | --eval_batch_size 8 \ 29 | --num_train_epochs 10 \ 30 | --max_seq_length 339 \ 31 | --learning_rate ${l} \ 32 | --seed ${seed} \ 33 | --model_dir "./output/${model}_batch_${s}_lr_${l}_seed_${seed}_${sample_size}/" 34 | done 35 | done 36 | done 37 | -------------------------------------------------------------------------------- /code/eval_span_pred.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """BERT finetuning runner.""" 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import os 23 | import logging 24 | import argparse 25 | import random 26 | from tqdm import tqdm, trange 27 | import numpy as np 28 | import torch 29 | from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler 30 | from torch.utils.data.distributed import DistributedSampler 31 | from torch.nn import CrossEntropyLoss 32 | from transformers import RobertaTokenizer 33 | from utils import * 34 | from models import RobertaSpanPredictor 35 | from optimization import * 36 | from pathlib import Path 37 | from collections import Counter 38 | import re 39 | import json 40 | 41 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 42 | datefmt='%m/%d/%Y %H:%M:%S', 43 | level=logging.INFO) 44 | logger = logging.getLogger(__name__) 45 | os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" 46 | PYTORCH_PRETRAINED_ROBERTA_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_ROBERTA_CACHE', 47 | Path.home() / '.pytorch_pretrained_roberta')) 48 | 49 | def main(): 50 | parser = argparse.ArgumentParser() 51 | ## Required parameters 52 | parser.add_argument("--data_dir", 53 | default=None, 54 | type=str, 55 | required=True, 56 | help="The input data dir. Should contain the .json files (or other data files) for the task.") 57 | parser.add_argument("--model", default=None, type=str, required=True, 58 | help="pre-trained model selected in the list: roberta-base, " 59 | "roberta-large, bert-base, bert-large. ") 60 | parser.add_argument("--task_name", 61 | default=None, 62 | type=str, 63 | required=True, 64 | help="The name of the task to train.") 65 | parser.add_argument("--model_dir", 66 | default=None, 67 | type=str, 68 | required=True, 69 | help="The load model directory") 70 | parser.add_argument("--file_suffix", 71 | default=None, 72 | type=str, 73 | required=True, 74 | help="unique identifier for data file") 75 | ## Other parameters 76 | parser.add_argument("--max_seq_length", 77 | default=320, # 8 * 8 * 5 78 | type=int, 79 | help="The maximum total input sequence length after WordPiece tokenization. \n" 80 | "Sequences longer than this will be truncated, and sequences shorter \n" 81 | "than this will be padded.") 82 | parser.add_argument("--do_lower_case", 83 | action='store_true', 84 | help="Set this flag if you are using an uncased model.") 85 | parser.add_argument("--eval_batch_size", 86 | default=8, 87 | type=int, 88 | help="Total batch size for eval.") 89 | parser.add_argument("--mlp_hid_size", 90 | default=64, 91 | type=int, 92 | help="hid dimension for MLP layer.") 93 | parser.add_argument("--learning_rate", 94 | default=5e-5, 95 | type=float, 96 | help="The initial learning rate for Adam.") 97 | parser.add_argument("--warmup_proportion", 98 | default=0.1, 99 | type=float, 100 | help="Proportion of training to perform linear learning rate warmup for. " 101 | "E.g., 0.1 = 10%% of training.") 102 | parser.add_argument("--no_cuda", 103 | action='store_true', 104 | help="Whether not to use CUDA when available") 105 | parser.add_argument("--local_rank", 106 | type=int, 107 | default=-1, 108 | help="local_rank for distributed training on gpus") 109 | parser.add_argument('--seed', 110 | type=int, 111 | default=42, 112 | help="random seed for initialization") 113 | parser.add_argument('--gradient_accumulation_steps', 114 | type=int, 115 | default=1, 116 | help="Number of updates steps to accumulate before performing a backward/update pass.") 117 | parser.add_argument('--fp16', 118 | action='store_true', 119 | help="Whether to use 16-bit float precision instead of 32-bit") 120 | parser.add_argument('--loss_scale', 121 | type=float, default=0, 122 | help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n" 123 | "0 (default value): dynamic loss scaling.\n" 124 | "Positive power of 2: static loss scaling value.\n") 125 | parser.add_argument('--cuda', 126 | type=str, 127 | default="", 128 | help="cuda index") 129 | parser.add_argument('--device_num', 130 | type=str, 131 | default="0", 132 | help="cuda device number") 133 | args = parser.parse_args() 134 | 135 | os.environ["CUDA_VISIBLE_DEVICES"] = args.device_num 136 | 137 | if args.local_rank == -1 or args.no_cuda: 138 | device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 139 | n_gpu = torch.cuda.device_count() 140 | else: 141 | torch.cuda.set_device(args.local_rank) 142 | device = torch.device("cuda", args.local_rank) 143 | n_gpu = 1 144 | # Initializes the distributed backend which will take care of sychronizing nodes/GPUs 145 | torch.distributed.init_process_group(backend='nccl') 146 | 147 | logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format( 148 | device, n_gpu, bool(args.local_rank != -1), args.fp16)) 149 | 150 | if args.gradient_accumulation_steps < 1: 151 | raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format( 152 | args.gradient_accumulation_steps)) 153 | 154 | # fix all random seeds 155 | random.seed(args.seed) 156 | np.random.seed(args.seed) 157 | torch.manual_seed(args.seed) 158 | 159 | if n_gpu > 0: 160 | torch.cuda.manual_seed_all(args.seed) 161 | 162 | task_name = args.task_name.lower() 163 | logger.info("current task is " + str(task_name)) 164 | 165 | # construct model 166 | if 'roberta' in args.model: 167 | tokenizer = RobertaTokenizer.from_pretrained(args.model, do_lower_case=args.do_lower_case) 168 | model = RobertaSpanPredictor.from_pretrained(args.model, mlp_hid=args.mlp_hid_size) 169 | if args.model_dir: 170 | model_state_dict = torch.load(args.model_dir + "/pytorch_model.bin") 171 | model.load_state_dict(model_state_dict) 172 | 173 | model.to(device) 174 | # Prepare optimizer 175 | param_optimizer = list(model.named_parameters()) 176 | no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] 177 | optimizer_grouped_parameters = [ 178 | {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01}, 179 | {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 180 | ] 181 | 182 | if args.fp16: 183 | try: 184 | from apex.optimizers import FusedAdam 185 | from apex import amp 186 | except ImportError: 187 | raise ImportError( 188 | "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.") 189 | optimizer = FusedAdam(optimizer_grouped_parameters, 190 | lr=args.learning_rate, 191 | bias_correction=False) 192 | model, optimizer = amp.initialize(model, optimizer, opt_level="O1") 193 | else: 194 | optimizer = BertAdam(optimizer_grouped_parameters, 195 | lr=args.learning_rate, 196 | warmup=args.warmup_proportion, 197 | t_total=t_total) 198 | 199 | for split in ['dev']: 200 | eval_data = load_data(args.data_dir, "final_%s" % split, args.file_suffix) 201 | eval_features = convert_to_features(eval_data, tokenizer, max_length=args.max_seq_length, evaluation=True) 202 | 203 | eval_inputs = select_field(eval_features, 'inputs') 204 | 205 | eval_input_ids = torch.tensor(select_field(eval_features, 'input_ids'), dtype=torch.long) 206 | eval_input_mask = torch.tensor(select_field(eval_features, 'mask_ids'), dtype=torch.long) 207 | eval_segment_ids = torch.tensor(select_field(eval_features, 'segment_ids'), dtype=torch.long) 208 | 209 | eval_offsets = select_field(eval_features, 'offsets') 210 | eval_labels = select_field(eval_features, 'labels') 211 | eval_events = select_field(eval_features, 'events') 212 | eval_answers = select_field(eval_features, 'answers') 213 | 214 | eval_key_indices = torch.tensor(list(range(len(eval_labels))), dtype=torch.long) 215 | 216 | # flatten question_ids 217 | eval_data = TensorDataset(eval_input_ids, eval_input_mask, eval_segment_ids, eval_key_indices) 218 | 219 | # Run prediction for full data 220 | eval_sampler = SequentialSampler(eval_data) 221 | eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size) 222 | 223 | model.eval() 224 | pred_answers = [] 225 | for batch in tqdm(eval_dataloader, desc="Evaluating"): 226 | batch = tuple(t.to(device) for t in batch) 227 | input_ids, input_masks, segment_ids, instance_indices = batch 228 | 229 | with torch.no_grad(): 230 | eval_logits = model(input_ids, attention_mask=input_masks, token_type_ids=segment_ids) 231 | 232 | indices = instance_indices.cpu().tolist() 233 | offsets = [eval_offsets[i] for i in indices] 234 | 235 | eval_logits = filter_outputs(eval_logits, offsets) 236 | 237 | preds = unflatten_vector(torch.argmax(eval_logits, dim=-1), offsets) 238 | 239 | batch_tokens = [tokenizer.convert_ids_to_tokens(ids) for ids in input_ids.tolist()] 240 | pred_answers.extend(get_answers(preds, offsets, batch_tokens)) 241 | 242 | assert len(pred_answers) == len(eval_answers) == len(eval_events) 243 | eval_ems, F1, recl, prec, F1_e, r_e, pr_e, F1_a, r_a, pr_a, hit1 = [], [], [], [], [], [], [], [], [], [], [] 244 | ps, rs, fs = {str(k): [] for k in range(1, 5)}, {str(k): [] for k in range(1, 5)}, {str(k): [] for k in range(1, 5)} 245 | 246 | for i, (pred, gold, event) in enumerate(zip(pred_answers, eval_answers, eval_events)): 247 | 248 | em = 1.0 if all([p in gold for p in pred]) and all([g in pred for g in gold]) else 0.0 249 | eval_ems.append(em) 250 | 251 | # construct unigram counter 252 | all_pred_counter = Counter() 253 | all_gold_counter = Counter() 254 | 255 | for n in range(1, 5): 256 | pred_counter = get_ngrams(pred, n=n) 257 | gold_counter = get_ngrams(gold, n=n) 258 | r, p, f = compute_f1(pred_counter, gold_counter) 259 | ps[str(n)].append(p) 260 | rs[str(n)].append(r) 261 | fs[str(n)].append(f) 262 | for k, v in pred_counter.items(): 263 | all_pred_counter[k] += v 264 | for k, v in gold_counter.items(): 265 | all_gold_counter[k] += v 266 | 267 | rl, pr, f1 = compute_f1(all_pred_counter, all_gold_counter) 268 | F1.append(f1) 269 | recl.append(rl) 270 | prec.append(pr) 271 | 272 | pred_counter = Counter([re.sub(r'[^\w\s]', '', s) for s in pred]) 273 | gold_counter = Counter([re.sub(r'[^\w\s]', '', s) for s in gold]) 274 | rl, pr, f1 = compute_f1(pred_counter, gold_counter) 275 | F1_a.append(f1) 276 | r_a.append(rl) 277 | pr_a.append(pr) 278 | 279 | rl, pr, f1 = compute_event_f1(pred, event) 280 | F1_e.append(f1) 281 | r_e.append(rl) 282 | pr_e.append(pr) 283 | 284 | if pred: 285 | hit1.append(compute_hit1(pred[0], event)) 286 | else: 287 | hit1.append(0.0) 288 | 289 | logger.info("EM is %.4f" % np.mean(eval_ems)) 290 | logger.info("Event HIT@1 is %.4f" % np.mean(hit1)) 291 | logger.info("Overall Token Recall, Precision, F1 are %.4f, %.4f, %.4f" % 292 | (np.mean(recl), np.mean(prec), np.mean(F1))) 293 | logger.info("Answer Recall, Precision, F1 are %.4f, %.4f, %.4f" % 294 | (np.mean(r_a), np.mean(pr_a), np.mean(F1_a))) 295 | logger.info("Event Recall, Precision, F1 are %.4f, %.4f, %.4f" % 296 | (np.mean(r_e), np.mean(pr_e), np.mean(F1_e))) 297 | 298 | for n in range(1, 5): 299 | logger.info("%s-gram Recall, Precision, F1 are %.4f, %.4f, %.4f" % 300 | (n, np.mean(rs[str(n)]), np.mean(ps[str(n)]), np.mean(fs[str(n)]))) 301 | 302 | if __name__ == "__main__": 303 | main() 304 | -------------------------------------------------------------------------------- /code/eval_span_pred.sh: -------------------------------------------------------------------------------- 1 | task="span_extraction" 2 | lrs=(1e-5) 3 | batch=(2) 4 | seeds=(5 7 23) 5 | device="1" 6 | pws=(5) 7 | model="roberta-large" 8 | 9 | for l in "${lrs[@]}" 10 | do 11 | for s in "${batch[@]}" 12 | do 13 | for seed in "${seeds[@]}" 14 | do 15 | for pw in "${pws[@]}" 16 | do 17 | python code/eval_span_pred.py \ 18 | --data_dir "./data/" \ 19 | --model ${model} \ 20 | --task_name ${task} \ 21 | --file_suffix "_ans_gen.json" \ 22 | --device_num ${device} \ 23 | --max_seq_length 343 \ 24 | --fp16 \ 25 | --learning_rate ${l} \ 26 | --seed ${seed} \ 27 | --model_dir "./output/${model}_batch_${s}_lr_${l}_seed_${seed}_pw_${pw}" 28 | done 29 | done 30 | done 31 | done -------------------------------------------------------------------------------- /code/eval_zero_shot.sh: -------------------------------------------------------------------------------- 1 | task="answer-generation-zero-shot" 2 | device="0" 3 | model="allenai/unifiedqa-t5-large" 4 | #model="allenai/unifiedqa-t5-base" 5 | #model="t5-base" 6 | 7 | python code/eval_ans_gen.py \ 8 | --data_dir "./data/" \ 9 | --model ${model} \ 10 | --task_name ${task} \ 11 | --file_suffix "_ans_gen.json" \ 12 | --device_num ${device} \ 13 | --eval_batch_size 8 \ 14 | --num_train_epochs 10 \ 15 | --max_seq_length 339 16 | 17 | -------------------------------------------------------------------------------- /code/leaderboard.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """BERT finetuning runner.""" 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import os 23 | import logging 24 | import argparse 25 | import random 26 | from tqdm import tqdm, trange 27 | import numpy as np 28 | import torch 29 | from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler 30 | from transformers import AutoTokenizer, T5ForConditionalGeneration, BartForConditionalGeneration 31 | from utils import * 32 | from optimization import * 33 | from pathlib import Path 34 | import json 35 | 36 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 37 | datefmt='%m/%d/%Y %H:%M:%S', 38 | level=logging.INFO) 39 | logger = logging.getLogger(__name__) 40 | os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" 41 | PYTORCH_PRETRAINED_ROBERTA_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_ROBERTA_CACHE', 42 | Path.home() / '.pytorch_pretrained_roberta')) 43 | 44 | def main(): 45 | parser = argparse.ArgumentParser() 46 | ## Required parameters 47 | parser.add_argument("--data_dir", 48 | default=None, 49 | type=str, 50 | required=True, 51 | help="The input data dir. Should contain the .json files (or other data files) for the task.") 52 | parser.add_argument("--model", default=None, type=str, required=True, 53 | help="pre-trained model selected in the list: roberta-base, " 54 | "roberta-large, bert-base, bert-large. ") 55 | parser.add_argument("--task_name", 56 | default=None, 57 | type=str, 58 | required=True, 59 | help="The name of the task to train.") 60 | parser.add_argument("--file_suffix", 61 | default=None, 62 | type=str, 63 | required=True, 64 | help="unique identifier for data file") 65 | ## Other parameters 66 | parser.add_argument("--max_seq_length", 67 | default=320, # 8 * 8 * 5 68 | type=int, 69 | help="The maximum total input sequence length after WordPiece tokenization. \n" 70 | "Sequences longer than this will be truncated, and sequences shorter \n" 71 | "than this will be padded.") 72 | parser.add_argument("--model_dir", 73 | type=str, 74 | help="saved model dir", 75 | default="") 76 | parser.add_argument("--do_lower_case", 77 | action='store_true', 78 | help="Set this flag if you are using an uncased model.") 79 | parser.add_argument("--eval_batch_size", 80 | default=8, 81 | type=int, 82 | help="Total batch size for eval.") 83 | parser.add_argument("--mlp_hid_size", 84 | default=64, 85 | type=int, 86 | help="hid dimension for MLP layer.") 87 | parser.add_argument("--learning_rate", 88 | default=5e-5, 89 | type=float, 90 | help="The initial learning rate for Adam.") 91 | parser.add_argument("--num_train_epochs", 92 | default=3.0, 93 | type=float, 94 | help="Total number of training epochs to perform.") 95 | parser.add_argument("--warmup_proportion", 96 | default=0.1, 97 | type=float, 98 | help="Proportion of training to perform linear learning rate warmup for. " 99 | "E.g., 0.1 = 10%% of training.") 100 | parser.add_argument("--no_cuda", 101 | action='store_true', 102 | help="Whether not to use CUDA when available") 103 | parser.add_argument("--local_rank", 104 | type=int, 105 | default=-1, 106 | help="local_rank for distributed training on gpus") 107 | parser.add_argument('--seed', 108 | type=int, 109 | default=42, 110 | help="random seed for initialization") 111 | parser.add_argument('--gradient_accumulation_steps', 112 | type=int, 113 | default=1, 114 | help="Number of updates steps to accumulate before performing a backward/update pass.") 115 | parser.add_argument('--fp16', 116 | action='store_true', 117 | help="Whether to use 16-bit float precision instead of 32-bit") 118 | parser.add_argument('--loss_scale', 119 | type=float, default=0, 120 | help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n" 121 | "0 (default value): dynamic loss scaling.\n" 122 | "Positive power of 2: static loss scaling value.\n") 123 | parser.add_argument('--cuda', 124 | type=str, 125 | default="", 126 | help="cuda index") 127 | parser.add_argument('--device_num', 128 | type=str, 129 | default="0", 130 | help="cuda device number") 131 | args = parser.parse_args() 132 | 133 | os.environ["CUDA_VISIBLE_DEVICES"] = args.device_num 134 | 135 | if args.local_rank == -1 or args.no_cuda: 136 | device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 137 | n_gpu = torch.cuda.device_count() 138 | else: 139 | torch.cuda.set_device(args.local_rank) 140 | device = torch.device("cuda", args.local_rank) 141 | n_gpu = 1 142 | # Initializes the distributed backend which will take care of sychronizing nodes/GPUs 143 | torch.distributed.init_process_group(backend='nccl') 144 | 145 | logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format( 146 | device, n_gpu, bool(args.local_rank != -1), args.fp16)) 147 | 148 | if args.gradient_accumulation_steps < 1: 149 | raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format( 150 | args.gradient_accumulation_steps)) 151 | 152 | # fix all random seeds 153 | random.seed(args.seed) 154 | np.random.seed(args.seed) 155 | torch.manual_seed(args.seed) 156 | 157 | if n_gpu > 0: 158 | torch.cuda.manual_seed_all(args.seed) 159 | 160 | task_name = args.task_name.lower() 161 | logger.info("current task is " + str(task_name)) 162 | 163 | # construct model 164 | if args.model_dir: 165 | logger.info(args.model_dir) 166 | model_state_dict = torch.load(args.model_dir + "pytorch_model.bin") 167 | tokenizer = AutoTokenizer.from_pretrained(args.model, state_dict=model_state_dict) 168 | if 't5' in args.model: 169 | model = T5ForConditionalGeneration.from_pretrained(args.model, state_dict=model_state_dict) 170 | if 'bart' in args.model: 171 | model = BartForConditionalGeneration.from_pretrained(args.model, state_dict=model_state_dict) 172 | else: 173 | tokenizer = AutoTokenizer.from_pretrained(args.model) 174 | model = T5ForConditionalGeneration.from_pretrained(args.model) 175 | 176 | model.to(device) 177 | 178 | # Prepare optimizer 179 | param_optimizer = list(model.named_parameters()) 180 | no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] 181 | optimizer_grouped_parameters = [ 182 | {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01}, 183 | {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 184 | ] 185 | 186 | if args.fp16: 187 | try: 188 | from apex.optimizers import FusedAdam 189 | from apex import amp 190 | except ImportError: 191 | raise ImportError( 192 | "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.") 193 | optimizer = FusedAdam(optimizer_grouped_parameters, 194 | lr=args.learning_rate, 195 | bias_correction=False) 196 | model, optimizer = amp.initialize(model, optimizer, opt_level="O1") 197 | else: 198 | optimizer = BertAdam(optimizer_grouped_parameters, 199 | lr=args.learning_rate, 200 | warmup=args.warmup_proportion, 201 | t_total=1) 202 | 203 | if n_gpu > 1: 204 | model = torch.nn.DataParallel(model) 205 | 206 | for split in ['dev', 'test']: 207 | eval_data = load_data(args.data_dir, "final_%s" % split, args.file_suffix) 208 | eval_features = convert_to_features_ans_gen(eval_data, ";", leaderboard=True) 209 | 210 | eval_inputs = select_field(eval_features, 'inputs') 211 | eval_encoded_inputs = tokenizer(eval_inputs, padding=True, truncation=True, return_tensors="pt") 212 | 213 | eval_input_ids = eval_encoded_inputs['input_ids'] 214 | eval_input_mask = eval_encoded_inputs['attention_mask'] 215 | 216 | eval_key_indices = torch.tensor(list(range(len(eval_inputs))), dtype=torch.long) 217 | 218 | logger.info("id_size: {}, mask_size: {}, instance_key_size: {}".format( 219 | eval_input_ids.size(), eval_input_mask.size(), eval_key_indices.size())) 220 | 221 | data = TensorDataset(eval_input_ids, eval_input_mask, eval_key_indices) 222 | 223 | # Run prediction for full data 224 | eval_sampler = SequentialSampler(data) 225 | eval_dataloader = DataLoader(data, sampler=eval_sampler, batch_size=args.eval_batch_size) 226 | 227 | preds = [] 228 | model.eval() 229 | 230 | for batch in tqdm(eval_dataloader, desc="Evaluating"): 231 | batch = tuple(t.to(device) for t in batch) 232 | 233 | input_ids, input_masks, instance_indices = batch 234 | 235 | with torch.no_grad(): 236 | res = model.generate(input_ids, attention_mask=input_masks, max_length=128) 237 | preds.extend([x.split(";") for x in tokenizer.batch_decode(res, skip_special_tokens=True)]) 238 | 239 | with open('./output/pred_%s.json' % split, 'w') as outfile: 240 | json.dump(preds, outfile) 241 | 242 | if __name__ == "__main__": 243 | main() 244 | -------------------------------------------------------------------------------- /code/leaderboard.sh: -------------------------------------------------------------------------------- 1 | task="answer-generation" 2 | device="0" 3 | 4 | ###### Model Options ###### 5 | #model="facebook/bart-base" 6 | #model="facebook/bart-large" 7 | #model="t5-base" 8 | #model="allenai/unifiedqa-t5-base" 9 | model="allenai/unifiedqa-t5-large" 10 | 11 | 12 | ###### Additional Model Suffix ###### 13 | #suffix="_500original" 14 | #suffix="_500completed" 15 | suffix="" 16 | 17 | lrs=(5e-5) 18 | batch=(4) 19 | seeds=(5) 20 | root="./output" 21 | for l in "${lrs[@]}" 22 | do 23 | for s in "${batch[@]}" 24 | do 25 | for seed in "${seeds[@]}" 26 | do 27 | python code/leaderboard.py \ 28 | --data_dir "./data/" \ 29 | --model ${model} \ 30 | --task_name ${task} \ 31 | --file_suffix "_ans_gen.json" \ 32 | --device_num ${device} \ 33 | --eval_batch_size 8 \ 34 | --num_train_epochs 10 \ 35 | --max_seq_length 339 \ 36 | --learning_rate ${l} \ 37 | --seed ${seed} \ 38 | --model_dir "${root}/${model}_batch_${s}_lr_${l}_seed_${seed}${suffix}/" 39 | done 40 | done 41 | done 42 | -------------------------------------------------------------------------------- /code/models.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import, division, print_function, unicode_literals 2 | 3 | import logging 4 | from torch import nn 5 | from transformers import RobertaConfig, RobertaModel, BertPreTrainedModel 6 | 7 | logger = logging.getLogger(__name__) 8 | 9 | ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP = { 10 | 'roberta-base': "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin", 11 | 'roberta-large': "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-pytorch_model.bin", 12 | 'roberta-large-mnli': "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-pytorch_model.bin", 13 | } 14 | 15 | class RobertaSpanPredictor(BertPreTrainedModel): 16 | config_class = RobertaConfig 17 | pretrained_model_archive_map = ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP 18 | base_model_prefix = "roberta" 19 | def __init__(self, config, mlp_hid=16): 20 | super(RobertaSpanPredictor, self).__init__(config) 21 | self.roberta = RobertaModel(config) 22 | self.dropout = nn.Dropout(config.hidden_dropout_prob) 23 | self.linear1 = nn.Linear(config.hidden_size, mlp_hid) 24 | self.linear2 = nn.Linear(mlp_hid, 3) 25 | 26 | self.act = nn.Tanh() 27 | self.init_weights() 28 | 29 | def forward(self, input_ids, attention_mask=None, token_type_ids=None, position_ids=None): 30 | outputs = self.roberta(input_ids, 31 | attention_mask=attention_mask, 32 | token_type_ids=token_type_ids, 33 | position_ids=position_ids) 34 | outputs = outputs[0] 35 | outputs = outputs.reshape(-1, outputs.size()[-1]) 36 | 37 | out = self.act(self.linear1(self.dropout(outputs))) 38 | logits = self.linear2(out).reshape(input_ids.size()[0], input_ids.size()[1], -1) 39 | 40 | return logits -------------------------------------------------------------------------------- /code/optimization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """PyTorch optimization for BERT model.""" 16 | 17 | import math 18 | import torch 19 | from torch.optim import Optimizer 20 | from torch.optim.optimizer import required 21 | from torch.nn.utils import clip_grad_norm_ 22 | 23 | def warmup_cosine(x, warmup=0.002): 24 | if x < warmup: 25 | return x/warmup 26 | return 0.5 * (1.0 + torch.cos(math.pi * x)) 27 | 28 | def warmup_constant(x, warmup=0.002): 29 | if x < warmup: 30 | return x/warmup 31 | return 1.0 32 | 33 | def warmup_linear(x, warmup=0.002): 34 | if x < warmup: 35 | return x/warmup 36 | return 1.0 - x 37 | 38 | SCHEDULES = { 39 | 'warmup_cosine':warmup_cosine, 40 | 'warmup_constant':warmup_constant, 41 | 'warmup_linear':warmup_linear, 42 | } 43 | 44 | 45 | class BertAdam(Optimizer): 46 | """Implements BERT version of Adam algorithm with weight decay fix. 47 | Params: 48 | lr: learning rate 49 | warmup: portion of t_total for the warmup, -1 means no warmup. Default: -1 50 | t_total: total number of training steps for the learning 51 | rate schedule, -1 means constant learning rate. Default: -1 52 | schedule: schedule to use for the warmup (see above). Default: 'warmup_linear' 53 | b1: Adams b1. Default: 0.9 54 | b2: Adams b2. Default: 0.999 55 | e: Adams epsilon. Default: 1e-6 56 | weight_decay: Weight decay. Default: 0.01 57 | max_grad_norm: Maximum norm for the gradients (-1 means no clipping). Default: 1.0 58 | """ 59 | def __init__(self, params, lr=required, warmup=-1, t_total=-1, schedule='warmup_linear', 60 | b1=0.9, b2=0.999, e=1e-6, weight_decay=0.01, 61 | max_grad_norm=1.0): 62 | if lr is not required and lr < 0.0: 63 | raise ValueError("Invalid learning rate: {} - should be >= 0.0".format(lr)) 64 | if schedule not in SCHEDULES: 65 | raise ValueError("Invalid schedule parameter: {}".format(schedule)) 66 | if not 0.0 <= warmup < 1.0 and not warmup == -1: 67 | raise ValueError("Invalid warmup: {} - should be in [0.0, 1.0[ or -1".format(warmup)) 68 | if not 0.0 <= b1 < 1.0: 69 | raise ValueError("Invalid b1 parameter: {} - should be in [0.0, 1.0[".format(b1)) 70 | if not 0.0 <= b2 < 1.0: 71 | raise ValueError("Invalid b2 parameter: {} - should be in [0.0, 1.0[".format(b2)) 72 | if not e >= 0.0: 73 | raise ValueError("Invalid epsilon value: {} - should be >= 0.0".format(e)) 74 | defaults = dict(lr=lr, schedule=schedule, warmup=warmup, t_total=t_total, 75 | b1=b1, b2=b2, e=e, weight_decay=weight_decay, 76 | max_grad_norm=max_grad_norm) 77 | super(BertAdam, self).__init__(params, defaults) 78 | 79 | def get_lr(self): 80 | lr = [] 81 | for group in self.param_groups: 82 | for p in group['params']: 83 | state = self.state[p] 84 | if len(state) == 0: 85 | return [0] 86 | if group['t_total'] != -1: 87 | schedule_fct = SCHEDULES[group['schedule']] 88 | lr_scheduled = group['lr'] * schedule_fct(state['step']/group['t_total'], group['warmup']) 89 | else: 90 | lr_scheduled = group['lr'] 91 | lr.append(lr_scheduled) 92 | return lr 93 | 94 | def step(self, closure=None): 95 | """Performs a single optimization step. 96 | 97 | Arguments: 98 | closure (callable, optional): A closure that reevaluates the model 99 | and returns the loss. 100 | """ 101 | loss = None 102 | if closure is not None: 103 | loss = closure() 104 | 105 | for group in self.param_groups: 106 | for p in group['params']: 107 | if p.grad is None: 108 | continue 109 | grad = p.grad.data 110 | if grad.is_sparse: 111 | raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead') 112 | 113 | state = self.state[p] 114 | 115 | # State initialization 116 | if len(state) == 0: 117 | state['step'] = 0 118 | # Exponential moving average of gradient values 119 | state['next_m'] = torch.zeros_like(p.data) 120 | # Exponential moving average of squared gradient values 121 | state['next_v'] = torch.zeros_like(p.data) 122 | 123 | next_m, next_v = state['next_m'], state['next_v'] 124 | beta1, beta2 = group['b1'], group['b2'] 125 | 126 | # Add grad clipping 127 | if group['max_grad_norm'] > 0: 128 | clip_grad_norm_(p, group['max_grad_norm']) 129 | 130 | # Decay the first and second moment running average coefficient 131 | # In-place operations to update the averages at the same time 132 | next_m.mul_(beta1).add_(1 - beta1, grad) 133 | next_v.mul_(beta2).addcmul_(1 - beta2, grad, grad) 134 | update = next_m / (next_v.sqrt() + group['e']) 135 | 136 | # Just adding the square of the weights to the loss function is *not* 137 | # the correct way of using L2 regularization/weight decay with Adam, 138 | # since that will interact with the m and v parameters in strange ways. 139 | # 140 | # Instead we want to decay the weights in a manner that doesn't interact 141 | # with the m/v parameters. This is equivalent to adding the square 142 | # of the weights to the loss with plain (non-momentum) SGD. 143 | if group['weight_decay'] > 0.0: 144 | update += group['weight_decay'] * p.data 145 | 146 | if group['t_total'] != -1: 147 | schedule_fct = SCHEDULES[group['schedule']] 148 | lr_scheduled = group['lr'] * schedule_fct(state['step']/group['t_total'], group['warmup']) 149 | else: 150 | lr_scheduled = group['lr'] 151 | 152 | update_with_lr = lr_scheduled * update 153 | p.data.add_(-update_with_lr) 154 | 155 | state['step'] += 1 156 | 157 | # step_size = lr_scheduled * math.sqrt(bias_correction2) / bias_correction1 158 | # No bias correction 159 | # bias_correction1 = 1 - beta1 ** state['step'] 160 | # bias_correction2 = 1 - beta2 ** state['step'] 161 | 162 | return loss 163 | -------------------------------------------------------------------------------- /code/run_ans_generation.sh: -------------------------------------------------------------------------------- 1 | task="answer-generation" 2 | lrs=(5e-5 1e-4 2e-4) 3 | batch=(2 4) 4 | seeds=(5 7 23) 5 | 6 | device="0" 7 | #model="google/pegasus-large" 8 | #model="facebook/bart-base" 9 | #model="facebook/bart-large" 10 | #model="t5-base" 11 | #model="allenai/unifiedqa-t5-base" 12 | model="allenai/unifiedqa-t5-large" 13 | 14 | #suffix="_500original" 15 | #suffix="_500completed" 16 | suffix="" 17 | 18 | for l in "${lrs[@]}" 19 | do 20 | for s in "${batch[@]}" 21 | do 22 | for seed in "${seeds[@]}" 23 | do 24 | python code/run_ans_generation_model.py \ 25 | --data_dir "./data/" \ 26 | --model ${model} \ 27 | --save_model \ 28 | --task_name ${task} \ 29 | --file_suffix "${suffix}.json" \ 30 | --device_num ${device} \ 31 | --train_batch_size ${s} \ 32 | --num_train_epochs 10 \ 33 | --max_seq_length 339 \ 34 | --do_train \ 35 | --do_eval \ 36 | --learning_rate ${l} \ 37 | --seed ${seed} \ 38 | --output_dir ./output/${model}_batch_${s}_lr_${l}_seed_${seed}${suffix} 39 | done 40 | done 41 | done -------------------------------------------------------------------------------- /code/run_ans_generation_model.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """BERT finetuning runner.""" 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import os 23 | import logging 24 | import argparse 25 | import random 26 | from tqdm import tqdm, trange 27 | import numpy as np 28 | import torch 29 | from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler 30 | from torch.utils.data.distributed import DistributedSampler 31 | from torch.nn import CrossEntropyLoss 32 | from transformers import AutoTokenizer, T5ForConditionalGeneration, BartForConditionalGeneration 33 | from transformers import PegasusForConditionalGeneration 34 | from utils import * 35 | from optimization import * 36 | from pathlib import Path 37 | import re 38 | from collections import Counter 39 | 40 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 41 | datefmt='%m/%d/%Y %H:%M:%S', 42 | level=logging.INFO) 43 | logger = logging.getLogger(__name__) 44 | os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" 45 | PYTORCH_PRETRAINED_ROBERTA_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_ROBERTA_CACHE', 46 | Path.home() / '.pytorch_pretrained_roberta')) 47 | 48 | def main(): 49 | parser = argparse.ArgumentParser() 50 | ## Required parameters 51 | parser.add_argument("--data_dir", 52 | default=None, 53 | type=str, 54 | required=True, 55 | help="The input data dir. Should contain the .json files (or other data files) for the task.") 56 | parser.add_argument("--model", default=None, type=str, required=True, 57 | help="pre-trained model selected in the list: " 58 | "allenai/unifiedqa-t5-{small, base, large ...} ") 59 | parser.add_argument("--task_name", 60 | default=None, 61 | type=str, 62 | required=True, 63 | help="The name of the task to train.") 64 | parser.add_argument("--output_dir", 65 | default=None, 66 | type=str, 67 | required=True, 68 | help="The output directory where the model predictions and checkpoints will be written.") 69 | parser.add_argument("--file_suffix", 70 | default=None, 71 | type=str, 72 | required=True, 73 | help="unique identifier for data file") 74 | ## Other parameters 75 | parser.add_argument("--max_seq_length", 76 | default=320, # 8 * 8 * 5 77 | type=int, 78 | help="The maximum total input sequence length after WordPiece tokenization. \n" 79 | "Sequences longer than this will be truncated, and sequences shorter \n" 80 | "than this will be padded.") 81 | parser.add_argument("--sub_sample", 82 | default=0, # 8 * 8 * 5 83 | type=int, 84 | help="0 means full data; otherwise, use K random sample for training") 85 | parser.add_argument("--do_train", 86 | action='store_true', 87 | help="Whether to run training.") 88 | parser.add_argument("--finetune", 89 | action='store_true', 90 | help="Whether to finetune LM.") 91 | parser.add_argument("--do_eval", 92 | action='store_true', 93 | help="Whether to run eval on the dev set.") 94 | parser.add_argument('--pos_weight', 95 | type=int, 96 | default=1, 97 | help="positive weight on label 1") 98 | parser.add_argument("--load_model", 99 | type=str, 100 | help="cosmos_model.bin, te_model.bin", 101 | default="") 102 | parser.add_argument("--do_lower_case", 103 | action='store_true', 104 | help="Set this flag if you are using an uncased model.") 105 | parser.add_argument("--train_batch_size", 106 | default=32, 107 | type=int, 108 | help="Total batch size for training.") 109 | parser.add_argument("--eval_batch_size", 110 | default=8, 111 | type=int, 112 | help="Total batch size for eval.") 113 | parser.add_argument("--mlp_hid_size", 114 | default=64, 115 | type=int, 116 | help="hid dimension for MLP layer.") 117 | parser.add_argument("--learning_rate", 118 | default=5e-5, 119 | type=float, 120 | help="The initial learning rate for Adam.") 121 | parser.add_argument("--num_train_epochs", 122 | default=3.0, 123 | type=float, 124 | help="Total number of training epochs to perform.") 125 | parser.add_argument("--warmup_proportion", 126 | default=0.1, 127 | type=float, 128 | help="Proportion of training to perform linear learning rate warmup for. " 129 | "E.g., 0.1 = 10%% of training.") 130 | parser.add_argument("--no_cuda", 131 | action='store_true', 132 | help="Whether not to use CUDA when available") 133 | parser.add_argument("--local_rank", 134 | type=int, 135 | default=-1, 136 | help="local_rank for distributed training on gpus") 137 | parser.add_argument('--seed', 138 | type=int, 139 | default=42, 140 | help="random seed for initialization") 141 | parser.add_argument('--gradient_accumulation_steps', 142 | type=int, 143 | default=1, 144 | help="Number of updates steps to accumulate before performing a backward/update pass.") 145 | parser.add_argument('--fp16', 146 | action='store_true', 147 | help="Whether to use 16-bit float precision instead of 32-bit") 148 | parser.add_argument('--loss_scale', 149 | type=float, default=0, 150 | help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n" 151 | "0 (default value): dynamic loss scaling.\n" 152 | "Positive power of 2: static loss scaling value.\n") 153 | parser.add_argument('--cuda', 154 | type=str, 155 | default="", 156 | help="cuda index") 157 | parser.add_argument('--save_model', 158 | action='store_true', 159 | help="save best or not") 160 | parser.add_argument('--device_num', 161 | type=str, 162 | default="0", 163 | help="cuda device number") 164 | args = parser.parse_args() 165 | 166 | os.environ["CUDA_VISIBLE_DEVICES"] = args.device_num 167 | 168 | if args.local_rank == -1 or args.no_cuda: 169 | device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 170 | n_gpu = torch.cuda.device_count() 171 | else: 172 | torch.cuda.set_device(args.local_rank) 173 | device = torch.device("cuda", args.local_rank) 174 | n_gpu = 1 175 | # Initializes the distributed backend which will take care of sychronizing nodes/GPUs 176 | torch.distributed.init_process_group(backend='nccl') 177 | 178 | logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format( 179 | device, n_gpu, bool(args.local_rank != -1), args.fp16)) 180 | 181 | if args.gradient_accumulation_steps < 1: 182 | raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format( 183 | args.gradient_accumulation_steps)) 184 | 185 | args.train_batch_size = int(args.train_batch_size / args.gradient_accumulation_steps) 186 | # fix all random seeds 187 | random.seed(args.seed) 188 | np.random.seed(args.seed) 189 | torch.manual_seed(args.seed) 190 | 191 | if n_gpu > 0: 192 | torch.cuda.manual_seed_all(args.seed) 193 | 194 | if not args.do_train and not args.do_eval: 195 | raise ValueError("At least one of `do_train` or `do_eval` must be True.") 196 | 197 | if os.path.exists(args.output_dir) and os.listdir(args.output_dir) and args.do_train and not args.load_model: 198 | raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir)) 199 | 200 | os.makedirs(args.output_dir, exist_ok=True) 201 | 202 | task_name = args.task_name.lower() 203 | logger.info("current task is " + str(task_name)) 204 | 205 | # construct model 206 | tokenizer = AutoTokenizer.from_pretrained(args.model) 207 | if "t5" in args.model: 208 | model = T5ForConditionalGeneration.from_pretrained(args.model) 209 | if "bart" in args.model: 210 | model = BartForConditionalGeneration.from_pretrained(args.model) 211 | 212 | model.to(device) 213 | if args.do_train: 214 | train_data = load_data(args.data_dir, "final_train_ans_gen", args.file_suffix) 215 | if args.sub_sample > 0: 216 | random.Random(args.seed).shuffle(train_data) 217 | train_data = train_data[:args.sub_sample] 218 | 219 | train_features = convert_to_features_ans_gen(train_data, ";") 220 | 221 | num_train_steps = int( 222 | len(train_features) / args.train_batch_size / args.gradient_accumulation_steps * args.num_train_epochs) 223 | 224 | all_inputs = select_field(train_features, 'inputs') 225 | encoded_inputs = tokenizer(all_inputs, padding=True, truncation=True, return_tensors="pt") 226 | 227 | all_input_ids = encoded_inputs['input_ids'] 228 | all_input_mask = encoded_inputs['attention_mask'] 229 | 230 | all_labels = select_field(train_features, 'labels') 231 | encoded_outputs = tokenizer(all_labels, padding=True, truncation=True, return_tensors="pt") 232 | all_output_ids = encoded_outputs['input_ids'] 233 | all_output_mask = encoded_outputs['attention_mask'] 234 | 235 | all_key_indices = torch.tensor(list(range(len(all_labels))), dtype=torch.long) 236 | 237 | logger.info("id_size: {}, mask_size: {}, instance_key_size: {}, label_size: {}".format( 238 | all_input_ids.size(), all_input_mask.size(), all_key_indices.size(), all_output_ids.size())) 239 | 240 | train_data = TensorDataset(all_input_ids, all_input_mask, all_key_indices, all_output_ids, all_output_mask) 241 | 242 | # free memory 243 | del train_features 244 | 245 | if args.local_rank == -1: 246 | train_sampler = RandomSampler(train_data) 247 | else: 248 | train_sampler = DistributedSampler(train_data) 249 | 250 | train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size) 251 | model.train() 252 | 253 | model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self 254 | output_model_file = os.path.join(args.output_dir, "pytorch_model.bin") 255 | output_perf_file = os.path.join(args.output_dir, "dev_perf.txt") 256 | # Prepare optimizer 257 | param_optimizer = list(model.named_parameters()) 258 | no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] 259 | optimizer_grouped_parameters = [ 260 | {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01}, 261 | {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 262 | ] 263 | 264 | t_total = num_train_steps 265 | 266 | if args.fp16: 267 | try: 268 | from apex.optimizers import FusedAdam 269 | from apex import amp 270 | except ImportError: 271 | raise ImportError( 272 | "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.") 273 | optimizer = FusedAdam(optimizer_grouped_parameters, 274 | lr=args.learning_rate, 275 | bias_correction=False) 276 | model, optimizer = amp.initialize(model, optimizer, opt_level="O1") 277 | else: 278 | optimizer = BertAdam(optimizer_grouped_parameters, 279 | lr=args.learning_rate, 280 | warmup=args.warmup_proportion, 281 | t_total=t_total) 282 | 283 | if n_gpu > 1: 284 | model = torch.nn.DataParallel(model) 285 | 286 | global_step = 0 287 | best_eval_f1 = 0.0 288 | for epoch in trange(int(args.num_train_epochs), desc="Epoch"): 289 | tr_loss, tr_acc_start, tr_acc_end = 0.0, 0.0, 0.0 290 | nb_tr_examples, nb_tr_steps = 0, 0 291 | for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")): 292 | batch = tuple(t.to(device) for t in batch) 293 | input_ids, input_masks, instance_indices, output_ids, output_masks = batch 294 | if 't5' in args.model: 295 | loss, _, _, _ = model(input_ids, attention_mask=input_masks, 296 | labels=output_ids, decoder_attention_mask=output_masks) 297 | else: 298 | output = model(input_ids, attention_mask=input_masks, 299 | labels=output_ids, decoder_attention_mask=output_masks) 300 | loss = output[0] 301 | 302 | if n_gpu > 1: 303 | loss = loss.mean() # mean() to average on multi-gpu. 304 | if args.gradient_accumulation_steps > 1: 305 | loss = loss / args.gradient_accumulation_steps 306 | 307 | if args.fp16: 308 | with amp.scale_loss(loss, optimizer) as scaled_loss: 309 | scaled_loss.backward() 310 | else: 311 | loss.backward() 312 | 313 | tr_loss += loss.item() 314 | nb_tr_examples += output_ids.size(0) 315 | nb_tr_steps += 1 316 | 317 | if (step + 1) % args.gradient_accumulation_steps == 0: 318 | # modify learning rate with special warm up BERT uses 319 | lr_this_step = args.learning_rate * warmup_linear(global_step / t_total, 320 | args.warmup_proportion) 321 | for param_group in optimizer.param_groups: 322 | param_group['lr'] = lr_this_step 323 | optimizer.step() 324 | optimizer.zero_grad() 325 | global_step += 1 326 | 327 | if step > 0 and step % 100 == 0: 328 | logger.info("current train loss is %s" % (tr_loss / float(nb_tr_steps))) 329 | 330 | if args.do_eval: 331 | eval_data = load_data(args.data_dir, "final_dev", "_ans_gen.json") 332 | eval_features = convert_to_features_ans_gen(eval_data, ";", eval=True) 333 | 334 | eval_inputs = select_field(eval_features, 'inputs') 335 | eval_encoded_inputs = tokenizer(eval_inputs, padding=True, truncation=True, return_tensors="pt") 336 | 337 | eval_input_ids = eval_encoded_inputs['input_ids'] 338 | eval_input_mask = eval_encoded_inputs['attention_mask'] 339 | 340 | eval_labels = select_field(eval_features, 'labels') 341 | eval_encoded_outputs = tokenizer(eval_labels, padding=True, truncation=True, return_tensors="pt") 342 | eval_output_ids = eval_encoded_outputs['input_ids'] 343 | eval_key_indices = torch.tensor(list(range(len(eval_labels))), dtype=torch.long) 344 | 345 | eval_events = select_field(eval_features, 'events') 346 | 347 | eval_data = TensorDataset(eval_input_ids, eval_input_mask, eval_key_indices, eval_output_ids) 348 | 349 | # Run prediction for full data 350 | eval_sampler = SequentialSampler(eval_data) 351 | eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size) 352 | 353 | preds, golds, events = [], [], [] 354 | model.eval() 355 | 356 | for batch in tqdm(eval_dataloader, desc="Evaluating"): 357 | batch = tuple(t.to(device) for t in batch) 358 | input_ids, input_masks, instance_indices, output_ids = batch 359 | 360 | with torch.no_grad(): 361 | res = model.generate(input_ids, attention_mask=input_masks) 362 | preds.extend([x.split(";") for x in tokenizer.batch_decode(res, skip_special_tokens=True)]) 363 | golds.extend([eval_labels[x].split(";") for x in instance_indices.tolist()]) 364 | events.extend([eval_events[x] for x in instance_indices.tolist()]) 365 | 366 | ems, F1, recl, prec, F1_e, r_e, pr_e = [], [], [], [], [], [], [] 367 | for pred, gold, event in zip(preds, golds, events): 368 | 369 | # construct unigram counter 370 | pred_counter = Counter([x for s in pred for x in re.sub(r'[^\w\s]', '', s).split(' ')]) 371 | gold_counter = Counter([x for s in gold for x in re.sub(r'[^\w\s]', '', s).split(' ')]) 372 | 373 | rl, pr, f1 = compute_unigram_f1(pred_counter, gold_counter) 374 | F1.append(f1) 375 | recl.append(rl) 376 | prec.append(pr) 377 | 378 | rl, pr, f1 = compute_event_f1(pred, event) 379 | F1_e.append(f1) 380 | r_e.append(rl) 381 | pr_e.append(pr) 382 | 383 | if f1 == 1.0: 384 | ems.append(1.0) 385 | else: 386 | ems.append(0.0) 387 | 388 | logger.info("Event EM is %.4f" % np.mean(ems)) 389 | logger.info("Token Recall, Precision, F1 are %.4f, %.4f, %.4f" % 390 | (np.mean(recl), np.mean(prec), np.mean(F1))) 391 | logger.info("Event Recall, Precision, F1 are %.4f, %.4f, %.4f" % 392 | (np.mean(r_e), np.mean(pr_e), np.mean(F1_e))) 393 | 394 | if np.mean(F1_e) > best_eval_f1: 395 | best_eval_f1 = np.mean(F1_e) 396 | logger.info("Save at Epoch %s" % epoch) 397 | with open(output_perf_file, 'w') as outfile: 398 | outfile.write("%.4f" % best_eval_f1) 399 | if args.save_model: 400 | torch.save(model_to_save.state_dict(), output_model_file) 401 | 402 | model.train() 403 | 404 | if __name__ == "__main__": 405 | main() 406 | -------------------------------------------------------------------------------- /code/run_ans_generation_subsample.sh: -------------------------------------------------------------------------------- 1 | task="answer-generation" 2 | lrs=(5e-5 1e-4 2e-4) 3 | batch=(2 4) 4 | seeds=(5 7 23) 5 | 6 | device="0" 7 | #model="google/pegasus-large" 8 | #model="facebook/bart-base" 9 | #model="facebook/bart-large" 10 | #model="t5-base" 11 | #model="allenai/unifiedqa-t5-base" 12 | model="allenai/unifiedqa-t5-large" 13 | suffix="" 14 | sample_size=500 #1000,2000,3000,4000 15 | 16 | for l in "${lrs[@]}" 17 | do 18 | for s in "${batch[@]}" 19 | do 20 | for seed in "${seeds[@]}" 21 | do 22 | nohup python code/run_ans_generation_model.py \ 23 | --data_dir "./data/" \ 24 | --model ${model} \ 25 | --save_model \ 26 | --task_name ${task} \ 27 | --file_suffix "${suffix}.json" \ 28 | --sub_sample ${sample_size} \ 29 | --device_num ${device} \ 30 | --train_batch_size ${s} \ 31 | --num_train_epochs 10 \ 32 | --max_seq_length 339 \ 33 | --do_train \ 34 | --do_eval \ 35 | --learning_rate ${l} \ 36 | --seed ${seed} \ 37 | --output_dir ./output/${model}_batch_${s}_lr_${l}_seed_${seed}_${sample_size} 38 | done 39 | done 40 | done -------------------------------------------------------------------------------- /code/run_span_pred.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team. 3 | # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | """BERT finetuning runner.""" 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import os 23 | import logging 24 | import argparse 25 | import random 26 | from tqdm import tqdm, trange 27 | import numpy as np 28 | import torch 29 | from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler 30 | from torch.utils.data.distributed import DistributedSampler 31 | from torch.nn import CrossEntropyLoss 32 | from transformers import RobertaTokenizer 33 | from utils import * 34 | from models import RobertaSpanPredictor 35 | from optimization import * 36 | from pathlib import Path 37 | from collections import Counter 38 | import re 39 | 40 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 41 | datefmt='%m/%d/%Y %H:%M:%S', 42 | level=logging.INFO) 43 | logger = logging.getLogger(__name__) 44 | os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID" 45 | PYTORCH_PRETRAINED_ROBERTA_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_ROBERTA_CACHE', 46 | Path.home() / '.pytorch_pretrained_roberta')) 47 | 48 | def main(): 49 | parser = argparse.ArgumentParser() 50 | ## Required parameters 51 | parser.add_argument("--data_dir", 52 | default=None, 53 | type=str, 54 | required=True, 55 | help="The input data dir. Should contain the .json files (or other data files) for the task.") 56 | parser.add_argument("--model", default=None, type=str, required=True, 57 | help="pre-trained model selected in the list: roberta-base, " 58 | "roberta-large, bert-base, bert-large. ") 59 | parser.add_argument("--task_name", 60 | default=None, 61 | type=str, 62 | required=True, 63 | help="The name of the task to train.") 64 | parser.add_argument("--output_dir", 65 | default=None, 66 | type=str, 67 | required=True, 68 | help="The output directory where the model predictions and checkpoints will be written.") 69 | parser.add_argument("--file_suffix", 70 | default=None, 71 | type=str, 72 | required=True, 73 | help="unique identifier for data file") 74 | ## Other parameters 75 | parser.add_argument("--max_seq_length", 76 | default=320, # 8 * 8 * 5 77 | type=int, 78 | help="The maximum total input sequence length after WordPiece tokenization. \n" 79 | "Sequences longer than this will be truncated, and sequences shorter \n" 80 | "than this will be padded.") 81 | parser.add_argument("--do_train", 82 | action='store_true', 83 | help="Whether to run training.") 84 | parser.add_argument("--finetune", 85 | action='store_true', 86 | help="Whether to finetune LM.") 87 | parser.add_argument("--do_eval", 88 | action='store_true', 89 | help="Whether to run eval on the dev set.") 90 | parser.add_argument('--pos_weight', 91 | type=int, 92 | default=1, 93 | help="positive weight on label 1") 94 | parser.add_argument("--load_model", 95 | type=str, 96 | help="cosmos_model.bin, te_model.bin", 97 | default="") 98 | parser.add_argument("--do_lower_case", 99 | action='store_true', 100 | help="Set this flag if you are using an uncased model.") 101 | parser.add_argument("--train_batch_size", 102 | default=32, 103 | type=int, 104 | help="Total batch size for training.") 105 | parser.add_argument("--eval_batch_size", 106 | default=8, 107 | type=int, 108 | help="Total batch size for eval.") 109 | parser.add_argument("--mlp_hid_size", 110 | default=64, 111 | type=int, 112 | help="hid dimension for MLP layer.") 113 | parser.add_argument("--learning_rate", 114 | default=5e-5, 115 | type=float, 116 | help="The initial learning rate for Adam.") 117 | parser.add_argument("--num_train_epochs", 118 | default=3.0, 119 | type=float, 120 | help="Total number of training epochs to perform.") 121 | parser.add_argument("--warmup_proportion", 122 | default=0.1, 123 | type=float, 124 | help="Proportion of training to perform linear learning rate warmup for. " 125 | "E.g., 0.1 = 10%% of training.") 126 | parser.add_argument("--no_cuda", 127 | action='store_true', 128 | help="Whether not to use CUDA when available") 129 | parser.add_argument("--local_rank", 130 | type=int, 131 | default=-1, 132 | help="local_rank for distributed training on gpus") 133 | parser.add_argument('--seed', 134 | type=int, 135 | default=42, 136 | help="random seed for initialization") 137 | parser.add_argument('--gradient_accumulation_steps', 138 | type=int, 139 | default=1, 140 | help="Number of updates steps to accumulate before performing a backward/update pass.") 141 | parser.add_argument('--fp16', 142 | action='store_true', 143 | help="Whether to use 16-bit float precision instead of 32-bit") 144 | parser.add_argument('--loss_scale', 145 | type=float, default=0, 146 | help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n" 147 | "0 (default value): dynamic loss scaling.\n" 148 | "Positive power of 2: static loss scaling value.\n") 149 | parser.add_argument('--cuda', 150 | type=str, 151 | default="", 152 | help="cuda index") 153 | parser.add_argument('--device_num', 154 | type=str, 155 | default="0", 156 | help="cuda device number") 157 | args = parser.parse_args() 158 | 159 | os.environ["CUDA_VISIBLE_DEVICES"] = args.device_num 160 | 161 | if args.local_rank == -1 or args.no_cuda: 162 | device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu") 163 | n_gpu = torch.cuda.device_count() 164 | else: 165 | torch.cuda.set_device(args.local_rank) 166 | device = torch.device("cuda", args.local_rank) 167 | n_gpu = 1 168 | # Initializes the distributed backend which will take care of sychronizing nodes/GPUs 169 | torch.distributed.init_process_group(backend='nccl') 170 | 171 | logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format( 172 | device, n_gpu, bool(args.local_rank != -1), args.fp16)) 173 | 174 | if args.gradient_accumulation_steps < 1: 175 | raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format( 176 | args.gradient_accumulation_steps)) 177 | 178 | args.train_batch_size = int(args.train_batch_size / args.gradient_accumulation_steps) 179 | # fix all random seeds 180 | random.seed(args.seed) 181 | np.random.seed(args.seed) 182 | torch.manual_seed(args.seed) 183 | 184 | if n_gpu > 0: 185 | torch.cuda.manual_seed_all(args.seed) 186 | 187 | if not args.do_train and not args.do_eval: 188 | raise ValueError("At least one of `do_train` or `do_eval` must be True.") 189 | 190 | if os.path.exists(args.output_dir) and os.listdir(args.output_dir) and args.do_train and not args.load_model: 191 | raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir)) 192 | 193 | os.makedirs(args.output_dir, exist_ok=True) 194 | 195 | task_name = args.task_name.lower() 196 | logger.info("current task is " + str(task_name)) 197 | 198 | # construct model 199 | if 'roberta' in args.model: 200 | tokenizer = RobertaTokenizer.from_pretrained(args.model, do_lower_case=args.do_lower_case) 201 | model = RobertaSpanPredictor.from_pretrained(args.model, mlp_hid=args.mlp_hid_size) 202 | 203 | model.to(device) 204 | if args.do_train: 205 | train_data = load_data(args.data_dir, "final_train", args.file_suffix) 206 | train_features = convert_to_features(train_data, tokenizer, max_length=args.max_seq_length) 207 | 208 | num_train_steps = int( 209 | len(train_features) / args.train_batch_size / args.gradient_accumulation_steps * args.num_train_epochs) 210 | all_input_ids = torch.tensor(select_field(train_features, 'input_ids'), dtype=torch.long) 211 | all_input_mask = torch.tensor(select_field(train_features, 'mask_ids'), dtype=torch.long) 212 | all_segment_ids = torch.tensor(select_field(train_features, 'segment_ids'), dtype=torch.long) 213 | 214 | all_offsets = select_field(train_features, 'offsets') 215 | all_labels = select_field(train_features, 'labels') 216 | 217 | all_key_indices = torch.tensor(list(range(len(all_labels))), dtype=torch.long) 218 | logger.info("id_size: {} mask_size: {}, instance_key_size: {}, segment_size: {}".format( 219 | all_input_ids.size(), all_input_mask.size(), all_key_indices.size(), all_segment_ids.size())) 220 | 221 | train_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_key_indices) 222 | 223 | # free memory 224 | del train_features 225 | del all_input_ids 226 | del all_input_mask 227 | del all_segment_ids 228 | 229 | if args.local_rank == -1: 230 | train_sampler = RandomSampler(train_data) 231 | else: 232 | train_sampler = DistributedSampler(train_data) 233 | 234 | train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size) 235 | model.train() 236 | 237 | model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self 238 | output_model_file = os.path.join(args.output_dir, "pytorch_model.bin") 239 | 240 | # Prepare optimizer 241 | param_optimizer = list(model.named_parameters()) 242 | no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] 243 | optimizer_grouped_parameters = [ 244 | {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01}, 245 | {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0} 246 | ] 247 | 248 | t_total = num_train_steps 249 | 250 | if args.fp16: 251 | try: 252 | from apex.optimizers import FusedAdam 253 | from apex import amp 254 | except ImportError: 255 | raise ImportError( 256 | "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.") 257 | optimizer = FusedAdam(optimizer_grouped_parameters, 258 | lr=args.learning_rate, 259 | bias_correction=False) 260 | model, optimizer = amp.initialize(model, optimizer, opt_level="O1") 261 | else: 262 | optimizer = BertAdam(optimizer_grouped_parameters, 263 | lr=args.learning_rate, 264 | warmup=args.warmup_proportion, 265 | t_total=t_total) 266 | 267 | if n_gpu > 1: 268 | model = torch.nn.DataParallel(model) 269 | 270 | global_step = 0 271 | best_eval_f1 = 0.0 272 | loss_fct = CrossEntropyLoss(weight=torch.tensor([1.0, args.pos_weight, args.pos_weight]).to(device)) 273 | for epoch in trange(int(args.num_train_epochs), desc="Epoch"): 274 | tr_loss, tr_acc_start, tr_acc_end = 0.0, 0.0, 0.0 275 | nb_tr_examples, nb_tr_steps = 0, 0 276 | f1s, ems, f1s_event, ems_event = [], [], [], [] 277 | for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")): 278 | batch = tuple(t.to(device) for t in batch) 279 | input_ids, input_masks, segment_ids, instance_indices = batch 280 | 281 | logits = model(input_ids, attention_mask=input_masks, token_type_ids=segment_ids) 282 | 283 | # offsets 284 | indices = instance_indices.cpu().tolist() 285 | offsets = [all_offsets[i] for i in indices] 286 | 287 | # labels 288 | golds = [all_labels[i] for i in indices] 289 | labels = torch.tensor(flatten_vector(golds), dtype=torch.long) 290 | 291 | # loss 292 | logits = filter_outputs(logits, offsets) 293 | loss = loss_fct(logits, labels.to(device)) 294 | 295 | if n_gpu > 1: 296 | loss = loss.mean() # mean() to average on multi-gpu. 297 | if args.gradient_accumulation_steps > 1: 298 | loss = loss / args.gradient_accumulation_steps 299 | 300 | if args.fp16: 301 | with amp.scale_loss(loss, optimizer) as scaled_loss: 302 | scaled_loss.backward() 303 | else: 304 | loss.backward() 305 | 306 | tr_loss += loss.item() 307 | nb_tr_examples += labels.size(0) 308 | nb_tr_steps += 1 309 | if (step + 1) % args.gradient_accumulation_steps == 0: 310 | # modify learning rate with special warm up BERT uses 311 | lr_this_step = args.learning_rate * warmup_linear(global_step / t_total, 312 | args.warmup_proportion) 313 | for param_group in optimizer.param_groups: 314 | param_group['lr'] = lr_this_step 315 | optimizer.step() 316 | optimizer.zero_grad() 317 | global_step += 1 318 | 319 | if (step + 1) % 100 == 0: 320 | logger.info("current train loss is %s" % (tr_loss / float(nb_tr_steps))) 321 | 322 | if args.do_eval: 323 | eval_data = load_data(args.data_dir, "final_dev", args.file_suffix) 324 | eval_features = convert_to_features(eval_data, tokenizer, 325 | max_length=args.max_seq_length, evaluation=True) 326 | 327 | eval_input_ids = torch.tensor(select_field(eval_features, 'input_ids'), dtype=torch.long) 328 | eval_input_mask = torch.tensor(select_field(eval_features, 'mask_ids'), dtype=torch.long) 329 | eval_segment_ids = torch.tensor(select_field(eval_features, 'segment_ids'), dtype=torch.long) 330 | 331 | eval_offsets = select_field(eval_features, 'offsets') 332 | eval_labels = select_field(eval_features, 'labels') 333 | eval_events = select_field(eval_features, 'events') 334 | eval_answers = select_field(eval_features, 'answers') 335 | eval_key_indices = torch.tensor(list(range(len(eval_labels))), dtype=torch.long) 336 | 337 | # flatten question_ids 338 | eval_data = TensorDataset(eval_input_ids, eval_input_mask, eval_segment_ids, eval_key_indices) 339 | 340 | # Run prediction for full data 341 | eval_sampler = SequentialSampler(eval_data) 342 | eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size) 343 | 344 | model.eval() 345 | pred_answers = [] 346 | for batch in tqdm(eval_dataloader, desc="Evaluating"): 347 | batch = tuple(t.to(device) for t in batch) 348 | input_ids, input_masks, segment_ids, instance_indices = batch 349 | 350 | with torch.no_grad(): 351 | eval_logits = model(input_ids, attention_mask=input_masks, token_type_ids=segment_ids) 352 | 353 | indices = instance_indices.cpu().tolist() 354 | offsets = [eval_offsets[i] for i in indices] 355 | 356 | eval_logits = filter_outputs(eval_logits, offsets) 357 | 358 | preds = unflatten_vector(torch.argmax(eval_logits, dim=-1), offsets) 359 | 360 | batch_tokens = [tokenizer.convert_ids_to_tokens(ids) for ids in input_ids.tolist()] 361 | pred_answers.extend(get_answers(preds, offsets, batch_tokens)) 362 | 363 | assert len(pred_answers) == len(eval_answers) == len(eval_events) 364 | F1, recl, prec, F1_e, r_e, pr_e, hit1 = [], [], [], [], [], [], [] 365 | for pred, gold, event in zip(pred_answers, eval_answers, eval_events): 366 | 367 | # construct unigram counter 368 | pred_counter = Counter([x for s in pred for x in re.sub(r'[^\w\s]', '', s).split(' ')]) 369 | gold_counter = Counter([x for s in gold for x in re.sub(r'[^\w\s]', '', s).split(' ')]) 370 | 371 | rl, pr, f1 = compute_unigram_f1(pred_counter, gold_counter) 372 | F1.append(f1) 373 | recl.append(rl) 374 | prec.append(pr) 375 | 376 | rl, pr, f1 = compute_event_f1(pred, event) 377 | F1_e.append(f1) 378 | r_e.append(rl) 379 | pr_e.append(pr) 380 | 381 | if pred: 382 | hit1.append(compute_hit1(pred[0], event)) 383 | else: 384 | hit1.append(0.0) 385 | 386 | logger.info("Event HIT@1 is %.4f" % np.mean(hit1)) 387 | logger.info("Token Recall, Precision, F1 are %.4f, %.4f, %.4f" % 388 | (np.mean(recl), np.mean(prec), np.mean(F1))) 389 | logger.info("Event Recall, Precision, F1 are %.4f, %.4f, %.4f" % 390 | (np.mean(r_e), np.mean(pr_e), np.mean(F1_e))) 391 | 392 | 393 | if np.mean(F1) > best_eval_f1: 394 | best_eval_f1 = np.mean(F1) 395 | logger.info("Save at Epoch %s" % epoch) 396 | torch.save(model_to_save.state_dict(), output_model_file) 397 | 398 | model.train() 399 | 400 | if __name__ == "__main__": 401 | main() 402 | -------------------------------------------------------------------------------- /code/run_span_pred.sh: -------------------------------------------------------------------------------- 1 | task="span_extraction" 2 | lrs=(1e-5) 3 | batch=(4) 4 | seeds=(23) 5 | device="3" 6 | pws=(1 2 5 10 20) 7 | 8 | model="roberta-large" 9 | 10 | for l in "${lrs[@]}" 11 | do 12 | for s in "${batch[@]}" 13 | do 14 | for seed in "${seeds[@]}" 15 | do 16 | for pw in "${pws[@]}" 17 | do 18 | nohup python code/run_span_pred.py \ 19 | --data_dir "./data/" \ 20 | --model ${model} \ 21 | --task_name ${task} \ 22 | --file_suffix "_ans_gen.json" \ 23 | --device_num ${device} \ 24 | --train_batch_size ${s} \ 25 | --num_train_epochs 10 \ 26 | --pos_weight ${pw} \ 27 | --max_seq_length 343 \ 28 | --do_train \ 29 | --do_eval \ 30 | --fp16 \ 31 | --learning_rate ${l} \ 32 | --seed ${seed} \ 33 | --output_dir "./output/${model}_batch_${s}_lr_${l}_seed_${seed}_pw_${pw}_rep" \ 34 | > code/logs/roberta-large_batch_${s}_lr_${l}_seed_${seed}_pw_${pw}_rep 35 | done 36 | done 37 | done 38 | done -------------------------------------------------------------------------------- /code/utils.py: -------------------------------------------------------------------------------- 1 | import json 2 | import torch 3 | import numpy as np 4 | from typing import Iterator, List, Mapping, Union, Optional, Set 5 | from collections import defaultdict, Counter, OrderedDict 6 | from datetime import datetime 7 | import logging 8 | from nltk.translate.bleu_score import SmoothingFunction 9 | from nltk.translate.bleu_score import sentence_bleu 10 | import copy 11 | import re 12 | 13 | logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s', 14 | datefmt='%m/%d/%Y %H:%M:%S', 15 | level=logging.INFO) 16 | logger = logging.getLogger(__name__) 17 | 18 | def cal_metrics(logits, golds, offsets, label_map, baseline=""): 19 | f1s, ems = [], [] 20 | 21 | for b, (offset, gold) in enumerate(zip(offsets, golds)): 22 | pred = torch.argmax(logits[b, :, :], dim=1).detach().cpu().tolist() 23 | # filter out non-leading, question and pad tokens 24 | pred = [pred[x] for x in offset] 25 | 26 | if baseline == "random": 27 | pred = list(np.random.choice([0, 1], size=(len(pred),), p=[1./2, 1./2])) 28 | if baseline == "all-1s": 29 | pred = [1] * len(pred) 30 | if baseline == "all-0s": 31 | pred = [0] * len(pred) 32 | 33 | assert len(pred) == len(gold) 34 | f1 = cal_f1(pred, gold, label_map) 35 | f1s.append(f1) 36 | 37 | em = 1 if pred == gold else 0 38 | ems.append(em) 39 | 40 | return f1s, ems 41 | 42 | def calculate_bleu(references, predidctions): 43 | smoothing = SmoothingFunction() 44 | bleu_scores = [] 45 | for g, p in zip(references, predidctions): 46 | bleu_scores.append(sentence_bleu([g.split(' ')], p.split(' '), smoothing_function=smoothing.method1)) 47 | return bleu_scores 48 | 49 | def get_ngrams(alist, n=2): 50 | counter = Counter() 51 | for s in alist: 52 | cands = re.sub(r'[^\w\s]', '', s).split(' ') 53 | for i in range(len(cands)-n+1): 54 | key = ' '.join(cands[i:i+n]) 55 | counter[key] += 1 56 | return counter 57 | 58 | def compute_f1(pred_counter, gold_counter): 59 | corr = 0 60 | for w in pred_counter: 61 | if w in gold_counter: 62 | corr += min(pred_counter[w], gold_counter[w]) 63 | 64 | prec = float(corr) / sum(pred_counter.values()) if sum(pred_counter.values()) > 0 else 0.0 65 | recl = float(corr) / sum(gold_counter.values()) if sum(gold_counter.values()) > 0 else 0.0 66 | return recl, prec, 2 * prec * recl / (prec + recl) if prec + recl > 0 else 0.0 67 | 68 | def compute_unigram_f1(pred_counter, gold_counter): 69 | corr = 0 70 | for w in pred_counter: 71 | if w in gold_counter: 72 | corr += min(pred_counter[w], gold_counter[w]) 73 | 74 | prec = float(corr) / sum(pred_counter.values()) if sum(pred_counter.values()) > 0 else 0.0 75 | recl = float(corr) / sum(gold_counter.values()) if sum(gold_counter.values()) > 0 else 0.0 76 | return recl, prec, 2 * prec * recl / (prec + recl) if prec + recl > 0 else 0.0 77 | 78 | def compute_event_f1(pred_ans, events): 79 | if len(events) == 0: 80 | return 0.0, 0.0, 0.0 81 | c = [any([e in p for p in pred_ans]) for e in events] 82 | recl = sum(c) / len(events) 83 | 84 | c = [any([e in p for e in events]) for p in pred_ans] 85 | prec = sum(c) / len(pred_ans) if len(pred_ans) > 0 else 0.0 86 | 87 | return recl, prec, 2 * prec * recl / (prec + recl) if prec + recl > 0 else 0.0 88 | 89 | def compute_hit1(pred_ans, events): 90 | 91 | if len(events) == 0 or not any([e in pred_ans for e in events]): 92 | return 0.0 93 | else: 94 | return 1.0 95 | 96 | def map_span_to_event(pred, event): 97 | pred_event = [0] * len(event) 98 | match_pred_tok, in_span = -1, 0 99 | for i, (p, e) in enumerate(zip(pred, event)): 100 | if p == 1: 101 | if e == 1: 102 | match_pred_tok = i 103 | pred_event[i] = 1 104 | in_span = 1 105 | else: 106 | # spacial case: was in span, but no matched event tok 107 | if in_span and match_pred_tok == -1: 108 | pred_event[i-1] = 1 109 | match_pred_tok, in_span = -1, 0 110 | 111 | # termination corner case: last tok in span, but no matched event tok 112 | if in_span and match_pred_tok == -1: 113 | pred_event[i] = 1 114 | 115 | return pred_event 116 | 117 | def cal_event_based_metrics(logits, events, offsets, label_map, baseline=""): 118 | f1s, ems = [], [] 119 | 120 | for b, (offset, event) in enumerate(zip(offsets, events)): 121 | pred = torch.argmax(logits[b, :, :], dim=1).detach().cpu().tolist() 122 | # filter out non-leading, question and pad tokens 123 | pred = [pred[x] for x in offset] 124 | assert len(pred) == len(event) 125 | 126 | pred_event = map_span_to_event(pred, event) 127 | 128 | if baseline == "random": 129 | pred_event = list(np.random.choice([0, 1], size=(len(pred),), p=[1./2, 1./2])) 130 | if baseline == "all-1s": 131 | pred_event = [1] * len(pred) 132 | if baseline == "all-0s": 133 | pred_event = [0] * len(pred) 134 | 135 | assert len(pred_event) == len(event) 136 | f1 = cal_f1(pred_event, event, label_map) 137 | f1s.append(f1) 138 | em = 1 if pred_event == event else 0 139 | ems.append(em) 140 | 141 | return f1s, ems 142 | 143 | def accuracy(out, labels): 144 | outputs = np.argmax(out, axis=1) 145 | return np.sum(outputs == labels) 146 | 147 | def cal_f1(pred_labels, true_labels, label_map, exclude_labels=[0], log=False): 148 | def safe_division(numr, denr, on_err=0.0): 149 | return on_err if denr == 0.0 else numr / denr 150 | 151 | assert len(pred_labels) == len(true_labels) 152 | 153 | total_true = Counter(true_labels) 154 | total_pred = Counter(pred_labels) 155 | 156 | labels = list(label_map) 157 | 158 | n_correct = 0 159 | n_true = 0 160 | n_pred = 0 161 | 162 | # we only need positive f1 score 163 | for label in labels: 164 | if label not in exclude_labels: 165 | true_count = total_true.get(label, 0) 166 | pred_count = total_pred.get(label, 0) 167 | 168 | n_true += true_count 169 | n_pred += pred_count 170 | 171 | correct_count = len([l for l in range(len(pred_labels)) 172 | if pred_labels[l] == true_labels[l] and pred_labels[l] == label]) 173 | n_correct += correct_count 174 | 175 | precision = safe_division(n_correct, n_pred) 176 | recall = safe_division(n_correct, n_true) 177 | f1_score = safe_division(2.0 * precision * recall, precision + recall) 178 | if log: 179 | logger.info("Correct: %d\tTrue: %d\tPred: %d" % (n_correct, n_true, n_pred)) 180 | logger.info("Overall Precision: %.4f\tRecall: %.4f\tF1: %.4f" % (precision, recall, f1_score)) 181 | return f1_score 182 | 183 | def select_field(data, field): 184 | # collect a list of field in data 185 | # fields: 'label_start', 'label_end', 'offset', 'input_ids, 'mask_ids', 'segment_ids', 'question_id' 186 | return [ex[field] for ex in data] 187 | 188 | def load_data(data_dir, split, suffix): 189 | filename = "%s%s%s" % (data_dir, split, suffix) 190 | print("==========load data from %s ===========" % filename) 191 | with open(filename, "r") as read_file: 192 | return json.load(read_file) 193 | 194 | def filter_outputs(logits, offsets): 195 | filtered_logits = [] 196 | for b, offset in enumerate(offsets): 197 | for i in range(len(offset)): 198 | filtered_logits.append(logits[b, offset[i], :].unsqueeze(0)) 199 | 200 | return torch.cat(filtered_logits, dim=0) 201 | 202 | 203 | def flatten_vector(vector): 204 | return [v for vec in vector for v in vec] 205 | 206 | def unflatten_vector(vector, offsets): 207 | 208 | new_vectors = [] 209 | start = 0 210 | for b in range(len(offsets)): 211 | offset = len(offsets[b]) 212 | new_vectors.append(vector[start:start+offset]) 213 | start += offset 214 | 215 | assert start == len(vector) 216 | return new_vectors 217 | 218 | def replace_special(astr, is_leading=False): 219 | if is_leading: 220 | return astr.replace(chr(288), '').replace(chr(266), '') 221 | else: 222 | return astr.replace(chr(288), ' ').replace(chr(266), '\n') 223 | 224 | def get_answers(preds, offsets, tokens): 225 | batch_answers = [] 226 | for b in range(len(offsets)): 227 | pred, offset = preds[b], offsets[b] 228 | assert len(pred) == len(offset) 229 | answers = [] 230 | in_answer, cur_ans = False, "" 231 | for p, o in zip(pred, offset): 232 | if p == 0 and in_answer: 233 | answers.append(cur_ans) 234 | in_answer, cur_ans = False, "" 235 | if p == 1: 236 | is_leading_token = False if in_answer else True 237 | cur_ans += replace_special(tokens[b][o], is_leading=is_leading_token) 238 | in_answer = True 239 | if p == 2: 240 | if in_answer: 241 | answers.append(cur_ans) 242 | cur_ans = "" 243 | cur_ans += replace_special(tokens[b][o], is_leading=True) 244 | in_answer = True 245 | 246 | if cur_ans: 247 | answers.append(cur_ans) 248 | batch_answers.append(answers) 249 | return batch_answers 250 | 251 | 252 | def convert_to_features(data, tokenizer, max_length=300, evaluation=False): 253 | # each sample will have Question Context 254 | samples = [] 255 | max_len_global = 0 # to show global max_len without truncating 256 | mismatch = 0 257 | for v in data: 258 | 259 | question = tokenizer.tokenize(v['question'].lower()) 260 | context = tokenizer.tokenize(v['context'].lower()) 261 | answers = [ans.lower() for ans in v['answer_texts']] 262 | ans_starts = [int(ans.split(',')[0][1:]) for ans in v['answer_indices']] 263 | ans_ends = [int(ans.split(',')[1][:-1]) for ans in v['answer_indices']] 264 | answers_to_match = ["" for _ in range(len(answers))] 265 | 266 | offset = 0 267 | 268 | labels = [] 269 | for token in context: 270 | temp = copy.copy(token) 271 | is_answer = 0.0 272 | for i, (s, e) in enumerate(zip(ans_starts, ans_ends)): 273 | if s <= offset + len(token) <= e: 274 | # dealing leading special tokens 275 | if not answers_to_match[i]: 276 | if token[:3] == chr(288) + "``": 277 | token = token[3:] 278 | elif token[:2] in [chr(288) + '"', chr(288) + '``', chr(288) + '(']: 279 | token = token[2:] 280 | elif token[0] in [chr(266), chr(288), '"', '(', '``']: 281 | token = token[1:] 282 | else: 283 | if token[0] == chr(288): 284 | token = token.replace(chr(288), ' ') 285 | if token: 286 | is_answer = 1.0 if answers_to_match[i] else 2.0 287 | answers_to_match[i] += token 288 | offset += len(temp) 289 | labels.append(is_answer) 290 | 291 | assert len(labels) == len(context) 292 | 293 | # answer mismatch is very rare, usefully caused by some special tokens. Minor impact and skip for now. 294 | for a, b in zip(answers_to_match, answers): 295 | if a != b: 296 | mismatch += 1 297 | 298 | bos, sep, eos = tokenizer.bos_token, tokenizer.sep_token, tokenizer.eos_token 299 | # two sep used in RoBERTa 300 | tokenized_ids = tokenizer.convert_tokens_to_ids([bos] + question + [sep]*2 + context + [eos]) 301 | if len(tokenized_ids) > max_len_global: 302 | max_len_global = len(tokenized_ids) 303 | 304 | if len(tokenized_ids) > max_length: 305 | ending = tokenized_ids[-1] 306 | tokenized_ids = tokenized_ids[:-(len(tokenized_ids) - max_length + 1)] + [ending] 307 | 308 | segment_ids = [0] * len(tokenized_ids) 309 | # mask ids 310 | mask_ids = [1] * len(tokenized_ids) 311 | 312 | # padding 313 | if len(tokenized_ids) < max_length: 314 | # Zero-pad up to the sequence length. 315 | padding = [0] * (max_length - len(tokenized_ids)) 316 | tokenized_ids += padding 317 | mask_ids += padding 318 | segment_ids += padding 319 | assert len(tokenized_ids) == max_length 320 | 321 | # bos + 2*sep + all question tokens 322 | offsets = [len(question) + 3 + k for k in range(len(labels))] 323 | 324 | sample = {'labels': labels, 325 | 'types': v['type'], 326 | 'input_ids': tokenized_ids, 327 | 'mask_ids': mask_ids, 328 | 'segment_ids': segment_ids, 329 | 'offsets': offsets} 330 | if evaluation: 331 | sample['answers'] = answers 332 | sample['events'] = [x.lower() for x in v['events']] 333 | sample['inputs'] = v['context'] 334 | 335 | # check some example data 336 | if len(samples) < 00: 337 | logger.info(sample) 338 | for t, l in zip(context, labels): 339 | if l == 1.0: 340 | logger.info("%s ; %s" % (t, l)) 341 | logger.info("===========") 342 | samples.append(sample) 343 | logger.info(mismatch) 344 | logger.info("Maximum length after tokenization is: % s" % (max_len_global)) 345 | return samples 346 | 347 | 348 | class ClassificationReport: 349 | def __init__(self, name, true_labels: List[Union[int, str]], 350 | pred_labels: List[Union[int, str]]): 351 | 352 | assert len(true_labels) == len(pred_labels) 353 | self.num_tests = len(true_labels) 354 | self.total_truths = Counter(true_labels) 355 | self.total_predictions = Counter(pred_labels) 356 | self.name = name 357 | self.labels = sorted(set(true_labels) | set(pred_labels)) 358 | self.confusion_mat = self.confusion_matrix(true_labels, pred_labels) 359 | self.accuracy = sum(y == y_ for y, y_ in zip(true_labels, pred_labels)) / len(true_labels) 360 | self.trim_label_width = 15 361 | self.rel_f1 = 0.0 362 | self.res_dict = {} 363 | 364 | @staticmethod 365 | def confusion_matrix(true_labels: List[str], predicted_labels: List[str]) \ 366 | -> Mapping[str, Mapping[str, int]]: 367 | mat = defaultdict(lambda: defaultdict(int)) 368 | for truth, prediction in zip(true_labels, predicted_labels): 369 | mat[truth][prediction] += 1 370 | return mat 371 | 372 | def __repr__(self): 373 | res = f'Name: {self.name}\t Created: {datetime.now().isoformat()}\t' 374 | res += f'Total Labels: {len(self.labels)} \t Total Tests: {self.num_tests}\n' 375 | display_labels = [label[:self.trim_label_width] for label in self.labels] 376 | label_widths = [len(l) + 1 for l in display_labels] 377 | max_label_width = max(label_widths) 378 | header = [l.ljust(w) for w, l in zip(label_widths, display_labels)] 379 | header.insert(0, ''.ljust(max_label_width)) 380 | res += ''.join(header) + '\n' 381 | for true_label, true_disp_label in zip(self.labels, display_labels): 382 | predictions = self.confusion_mat[true_label] 383 | row = [true_disp_label.ljust(max_label_width)] 384 | for pred_label, width in zip(self.labels, label_widths): 385 | row.append(str(predictions[pred_label]).ljust(width)) 386 | res += ''.join(row) + '\n' 387 | res += '\n' 388 | 389 | def safe_division(numr, denr, on_err=0.0): 390 | return on_err if denr == 0.0 else numr / denr 391 | 392 | def num_to_str(num): 393 | return '0' if num == 0 else str(num) if type(num) is int else f'{num:.4f}' 394 | 395 | n_correct = 0 396 | n_true = 0 397 | n_pred = 0 398 | 399 | all_scores = [] 400 | header = ['Total ', 'Predictions', 'Correct', 'Precision', 'Recall ', 'F1-Measure'] 401 | res += ''.ljust(max_label_width + 2) + ' '.join(header) + '\n' 402 | head_width = [len(h) for h in header] 403 | 404 | 405 | for label, width, display_label in zip(self.labels, label_widths, display_labels): 406 | total_count = self.total_truths.get(label, 0) 407 | pred_count = self.total_predictions.get(label, 0) 408 | 409 | n_true += total_count 410 | n_pred += pred_count 411 | 412 | correct_count = self.confusion_mat[label][label] 413 | n_correct += correct_count 414 | 415 | precision = safe_division(correct_count, pred_count) 416 | recall = safe_division(correct_count, total_count) 417 | f1_score = safe_division(2 * precision * recall, precision + recall) 418 | all_scores.append((precision, recall, f1_score)) 419 | self.res_dict[label] = (f1_score, total_count) 420 | row = [total_count, pred_count, correct_count, precision, recall, f1_score] 421 | row = [num_to_str(cell).ljust(w) for cell, w in zip(row, head_width)] 422 | row.insert(0, display_label.rjust(max_label_width)) 423 | res += ' '.join(row) + '\n' 424 | 425 | # weighing by the truth label's frequency 426 | label_weights = [safe_division(self.total_truths.get(label, 0), self.num_tests) 427 | for label in self.labels] 428 | weighted_scores = [(w * p, w * r, w * f) for w, (p, r, f) in zip(label_weights, all_scores)] 429 | 430 | assert len(label_weights) == len(weighted_scores) 431 | 432 | res += '\n' 433 | res += ' '.join(['Weighted Avg'.rjust(max_label_width), 434 | ''.ljust(head_width[0]), 435 | ''.ljust(head_width[1]), 436 | ''.ljust(head_width[2]), 437 | num_to_str(sum(p for p, _, _ in weighted_scores)).ljust(head_width[3]), 438 | num_to_str(sum(r for _, r, _ in weighted_scores)).ljust(head_width[4]), 439 | num_to_str(sum(f for _, _, f in weighted_scores)).ljust(head_width[5])]) 440 | 441 | print(n_correct, n_pred, n_true) 442 | 443 | precision = safe_division(n_correct, n_pred) 444 | recall = safe_division(n_correct, n_true) 445 | f1_score = safe_division(2.0 * precision * recall, precision + recall) 446 | 447 | res += f'\n Total Examples: {self.num_tests}' 448 | res += f'\n Overall Precision: {num_to_str(precision)}' 449 | res += f'\n Overall Recall: {num_to_str(recall)}' 450 | res += f'\n Overall F1: {num_to_str(f1_score)} ' 451 | self.rel_f1 = f1_score 452 | return res 453 | 454 | 455 | def convert_to_features_ans_gen(data, sep_tok, eval=False, leaderboard=False): 456 | samples = [] 457 | counter = 0 458 | 459 | for v in data: 460 | inputs = "%s \\n %s" % (v['question'].lower(), v['context'].lower()) 461 | 462 | if eval: 463 | sample = {'labels': sep_tok.join([x.lower() for x in v['answer_texts']]), 464 | 'events': [x.lower() for x in v['events']], 465 | 'types': v['type'], 466 | 'inputs': inputs} 467 | elif leaderboard: 468 | sample = {'inputs': inputs} 469 | else: 470 | sample = {'labels': sep_tok.join([x.lower() for x in v['answer_texts']]), 471 | 'types': v['type'], 472 | 'inputs': inputs} 473 | 474 | counter += 1 475 | # check some example data 476 | if counter < 1: 477 | print(sample) 478 | samples.append(sample) 479 | return samples 480 | 481 | 482 | -------------------------------------------------------------------------------- /env.yml: -------------------------------------------------------------------------------- 1 | name: apex 2 | channels: 3 | - pytorch 4 | - psi4 5 | - http://conda.anaconda.org/gurobi 6 | - defaults 7 | dependencies: 8 | - _libgcc_mutex=0.1=main 9 | - blas=1.0=mkl 10 | - ca-certificates=2020.7.22=0 11 | - certifi=2020.6.20=py38_0 12 | - cloog=0.18.0=0 13 | - cudatoolkit=10.1.243=h6bb024c_0 14 | - freetype=2.10.2=h5ab3b9f_0 15 | - gcc-5=5.2.0=1 16 | - gmp=6.1.2=h6c8ec71_1 17 | - intel-openmp=2020.2=254 18 | - isl=0.12.2=0 19 | - jpeg=9b=h024ee3a_2 20 | - lcms2=2.11=h396b838_0 21 | - ld_impl_linux-64=2.33.1=h53a641e_7 22 | - libedit=3.1.20191231=h14c3975_1 23 | - libffi=3.3=he6710b0_2 24 | - libgcc=7.2.0=h69d50b8_2 25 | - libgcc-ng=9.1.0=hdf63c60_0 26 | - libpng=1.6.37=hbc83047_0 27 | - libstdcxx-ng=9.1.0=hdf63c60_0 28 | - libtiff=4.1.0=h2733197_1 29 | - lz4-c=1.9.2=he6710b0_1 30 | - mkl=2020.2=256 31 | - mkl-service=2.3.0=py38he904b0f_0 32 | - mkl_fft=1.1.0=py38h23d657b_0 33 | - mkl_random=1.1.1=py38h0573a6f_0 34 | - mpc=1.1.0=h10f8cd9_1 35 | - mpfr=4.0.2=hb69a4c5_1 36 | - ncurses=6.2=he6710b0_1 37 | - ninja=1.10.1=py38hfd86e86_0 38 | - numpy=1.19.1=py38hbc911f0_0 39 | - numpy-base=1.19.1=py38hfa32c7d_0 40 | - olefile=0.46=py_0 41 | - openssl=1.1.1h=h7b6447c_0 42 | - pandas=1.1.3=py38he6710b0_0 43 | - pillow=7.2.0=py38hb39fc2d_0 44 | - pip=20.2.2=py38_0 45 | - python=3.8.5=h7579374_1 46 | - python-dateutil=2.8.1=py_0 47 | - pytorch=1.6.0=py3.8_cuda10.1.243_cudnn7.6.3_0 48 | - pytz=2020.1=py_0 49 | - readline=8.0=h7b6447c_0 50 | - setuptools=49.6.0=py38_0 51 | - six=1.15.0=py_0 52 | - sqlite=3.33.0=h62c20be_0 53 | - tk=8.6.10=hbc83047_0 54 | - torchvision=0.7.0=py38_cu101 55 | - tqdm=4.48.2=py_0 56 | - wheel=0.35.1=py_0 57 | - xz=5.2.5=h7b6447c_0 58 | - zlib=1.2.11=h7b6447c_3 59 | - zstd=1.4.5=h9ceee32_0 60 | - pip: 61 | - apex==0.1 62 | - boto3==1.14.56 63 | - botocore==1.17.56 64 | - chardet==3.0.4 65 | - click==7.1.2 66 | - docutils==0.15.2 67 | - filelock==3.0.12 68 | - idna==2.10 69 | - jmespath==0.10.0 70 | - joblib==0.16.0 71 | - packaging==20.4 72 | - pyparsing==2.4.7 73 | - regex==2020.7.14 74 | - requests==2.24.0 75 | - s3transfer==0.3.3 76 | - sacremoses==0.0.43 77 | - sentencepiece==0.1.91 78 | - tokenizers==0.8.1rc2 79 | - transformers==3.1.0 80 | - urllib3==1.25.10 81 | prefix: ./ 82 | -------------------------------------------------------------------------------- /output/README.md: -------------------------------------------------------------------------------- 1 | models objects should be placed here -------------------------------------------------------------------------------- /output/pred_dev.json: -------------------------------------------------------------------------------- 1 | [["regulations that stipulate officials in charge should be punished for fatal accidents"], ["the four people will face further punishments"], ["explosive blast that occurred in the pagou coal mine"], ["its production safety and coal mining certificates expired"], ["a series of attacks"], ["the demolition"], ["two villagers were shot and wounded", "protesters took to the streets in violation of the curfew"], ["discovered the body of a kidnapped israeli soldier"], ["weaknesses in both the midfield and the forward line"], ["hostile pace and slinging action of malinga at one end", "could not read muralitharan's spin at the other"], ["it failed to restart"], ["mechanical failure struck"], ["a dispute between russia and ukraine over gas supplies to the strategic transit country"], ["cheating by some members"], ["any reduction in supplies could send prices higher in the months ahead"], ["the peak time for oil demand because of the northern hemisphere winter"], ["too many people undermining the structure rupert lowe wanted at the club"], ["the decision to bring him in was bizarre"], ["too many people undermining the structure rupert lowe wanted at the club"], ["called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the cypriot government", "called on the"], ["deal with serious threats from fighters who were in that location"], ["militants, terrorists created a training center"], ["deal with serious threats from fighters who were in that location"], ["militants, terrorists created a training center"], ["the pakistani government has said they carried out initiative to deal with serious threats"], ["attack on a religious school in pakistan's tribal region"], ["through dna testing"], ["protesting abortion"], ["the filing of the charges"], ["suspect in a fatal attack"], ["arrested in several states since 1990 for protesting abortion", "dna testing", "strand of hair found near where the sniper fired"], ["western countries fear could be used to develop a nuclear weapon"], ["jumping from the roof of the residence tuesday"], ["work on the chayvo reserves in the northeastern part of the island of sakhalin"], ["production sharing agreement (psa) that governs the project"], ["receives a share of profits from the project"], ["bio-diesel could be produced"], ["increase funding to the sector"], ["killing one", "critically injuring another"], ["doing routine inspections"], ["hold prayer services for a peaceful outcome"], ["the country might plunge into chaos"], ["fears that new clashes could erupt on sydney beaches"], ["record a $7.5 million reserve for its defense group", "write off about $3.5 million in costs related to international exploration leases"], ["special reserves and continued low natural-gas prices"], ["patients declared positive"], ["somali-ethiopian troops from advancing further southwards"], ["un security council failed to agree on the withdrawal of foreign troops"], ["retreating islamist fighters opened fire on a force of government and ethiopian troops"], ["heavy fighting resumed in central somalia"], ["the withdrawal of foreign troops"], ["software group reported a 16 percent increase in net profits"], ["a regulatory inquiry"], ["tankan survey of japanese business sentiment rose"], ["a regulatory inquiry"], ["adobe systems leapt 2.00 dollars, or 4.9 percent, to 42.81 dollars"], ["he and dravid put on 61 for the third wicket"], ["played and missed at the next ball"], ["cases of dengue virus stands at 247"], ["a bomb blast"], ["a bomb at an abortion clinic"], ["more people are hurt"], ["heavy fighting resumed in central somalia"], ["a 1.3 percent fall in september"], ["wholesale prices in the united states fell by 1.6 percent", "followed a 1.3 percent fall in september", "was sharper than the 0.5 percent decline expected by analysts", "led by falling energy prices", "dropped by 0.9 percent in october", "reflected big declines in prices for new cars and sport utility vehicles"], ["finding english football physically challenging"], ["mbeki sacked zuma as the country's deputy president in june", "zuma was prosecuted on two corruption charges"], ["came to an agreement on proposals to deal with tension in party"], ["the storming of the border terminal by dozens of hamas gunmen"], ["the closure of the rafah crossing"], ["the storming of the border terminal by dozens of hamas gunmen"], ["militants went on a rampage inside the terminal building", "smashing windows and furniture", "firing into the air and at the building itself"], ["those responsible be found"], ["smashing windows and furniture", "firing into the air and at the building itself"], ["the closure of the rafah crossing"], ["the storming of the border terminal by dozens of hamas gunmen"], ["militants went on a rampage inside the terminal building", "smashing windows and furniture", "firing into the air and at the building itself"], ["it believed the money would be used to finance terrorism"], ["a $4.7 million charge related to problems under a fixed-price development contract", "$2.8 million in overhead costs that won't be reimbursed"], ["report a third-quarter net loss of $17 million to $19 million"], ["write off about $3.5 million in costs related to international exploration leases", "interest costs", "amortization of goodwill"], ["won't be reimbursed"], ["record a $7.5 million reserve for its defense group", "a $4.7 million charge related to problems under a fixed-price development contract", "write off about $3.5 million in costs related to international exploration leases"], ["prime minister junichiro koizumi's repeated visits to the war criminal-honoring yasukuni shrine", "other history-related issues"], ["technical reasons"], ["clashes could erupt on sydney beaches"], ["the angry crowd rushed at manuel gonzalez", "shaking fists at him", "shouting in spanish", "communista'' and traitor"], ["zambian farmers are this year expected to grow more than 150,000 hectares of jatropha"], ["increase funding to the sector"], ["section of the yellow line was shut down", "shuttle buses were used to transport passengers"], ["killing one", "critically injuring another"], ["hit by an empty four-car train"], ["mechanical failure struck hong kong's landmark peak tram"], ["reduce its output by 1.2 million bpd", "cut of 500,000 bpd in february", "reduce the output from opec members", "excluding iraq and angola", "reduce the output from opec members", "reduce the output from opec members", "cut of 500,000 bpd in february", "reduce the output from opec members", "reduce the output from opec members", "excludes iraq and angola"], ["greeted at the airport by his two daughters"], ["rescuing all but one of the 72 hostages", "killing all 14 rebels"], ["public disenchantment with the us military campaign", "the need to offer a viable alternative"], ["appoint a president and two vice presidents"], ["defiance of the sanctions agreed by the un security council"], ["to protect their anonymity"], ["vaughan was being considered as captain for the one-dayers", "take that into consideration when we pick the one-day side"], ["falling gas inventories"], ["gasoline inventories fell by 3.7 million barrels"], ["refineries operated at 87.3 percent of capacity last week"], ["light sweet crude for december settlement closed up 48 cents", "touching 59.40 dollars in earlier trading", "brent north sea crude for december delivery rose 62 cents"], ["planning to commit atrocities on the war-torn island of bougainville"], ["demanding the sacking of prime minister sir julius chan"], ["arrived back in britain late wednesday", "carry only hand luggage and a book", "was met by a driver in a limousine", "was understood to be on his way to a reunion with his wife catherine and children"], ["illegal firearms possession"], ["brigadier general jerry singirok"], ["hiring sandline for 50 million kina"], ["the massacre of 148 people from the shiite village of dujail in 1982"], ["two earlier adjournments limited court time to just two brief sessions"], ["in time for iraq's parliamentary elections"], ["if japan keeps close relationship with the united states"], ["militants, terrorists created a training center"], ["establish peace in the tribal region"], ["played golf", "installed a neutron telescope on the exterior of the zvezda service module", "took a picture of an antenna of the kurs docking system", "replaced a wal 2 antenna"], ["played golf", "installed a neutron telescope on the exterior of the zvezda service module", "took a picture of an antenna of the kurs docking system", "replaced a wal 2 antenna"], ["the closure of the rafah crossing"], ["it believed the money would be used to finance terrorism"], ["the storming of the border terminal by dozens of hamas gunmen"], ["the money would be used to finance terrorism"], ["a calf injury"], ["a calf injury"], ["withdrew from the 15th asiad"], ["protesting abortion"], ["a fatal attack"], ["shift their tactics"], ["violent attacks against abortion providers"], ["falling gas inventories"], ["juan miguel, sent a letter to u.s. attorney janet reno asking that his son be moved"], ["conflicting stances"], ["the al keeps in touch with the somali government and the islamic courts union (icu) to resume negotiations"], ["vaughan was being considered as captain for the one-dayers"], ["make that decision"], ["take that into consideration"], ["holding higher-cost inventory", "raw material costs of such key products as nickel dropped"], ["a number of scandals", "involving house leader tom delay"], ["republican maverick senator john mccain has forced"], ["illegal firearms possession"], ["planning to commit atrocities on the war-torn island of bougainville"], ["hiring sandline for 50 million kina"], ["state terrorism and killings"], ["form a cabinet with parliamentary approval"], ["separatist violence has erupted periodically"], ["a bloodless coup in september"], ["offered a number of olive branches"], ["the march of some 4,000 workers from 10 factories"], ["reopen the rafah crossing"], ["authorized by the chinese government"], ["authorized by the chinese government to recruit international students"], ["the talent and teamwork of 16 world-class scientists"], ["an aborted invasion of cuba by cia-trained exiles"], ["keep the boy in miami"], ["accused of siphoning off 271-million dollars in government funds meant for buying cattle feed"], ["chinese competitors didn't enter"], ["skipped the 5m mark and faced up to 5.20"], ["winning the women's 400m final in 52.69"], ["flew over the 5-meter bar in his first attempt", "failed three times at 5.20"], ["recover misused funds from illegal investments"], ["japanese leaders' stubborn visits to yasukuni shrine"], ["hurt the feeling of the chinese people", "undermined the political foundation of sino-japanese ties"], ["britain's queen elizabeth ii celebrated her 71st birthday"], ["business confidence among japan's major manufacturers probably hit a new two-year high"], ["some analysts predicting a move early next year"], ["slackening demand on poor summer weather and the like"], ["israel's main problem is threat that comes from those who openly talk about wiping us off the map"], ["moral threat from iran"], ["murdering four people"], ["his work against gang violence"], ["faces the death penalty for murdering four people"], ["solve the protection problem"], ["stocks have been reduced below safety levels"], ["mbeki sacked zuma as the country's deputy president in june", "zuma was prosecuted on two corruption charges"], ["mandated by the anc national executive committee"], ["going ahead with prosecuting the accused"], ["work on the chayvo reserves in the northeastern part of the island of sakhalin"], ["receives a share of profits from the project"], ["increased spending", "work programme", "estimate of the expenses for extraction and management"], ["work on the chayvo reserves in the northeastern part of the island of sakhalin"], ["production sharing agreement (psa) that governs the project"], ["production sharing agreement"], ["recouped their investment"], ["heard final arguments on thursday from prosecutors and defense lawyers", "had 30 minutes each to make their cases"], ["halt the death of williams"], ["halt the death of williams"], ["die by lethal injection"], ["international campaign for clemency"], ["heard final arguments on thursday from prosecutors and defense lawyers", "had 30 minutes each to make their cases"], ["cross the rafah crossing in both directions without any conditions"], ["study the brain"], ["flew over the 5-meter bar in his first attempt"], ["found 7.1 billion yuan (900 million dollars) had been misappropriated"], ["a quiet day at windsor castle monday"], ["27,000 runners are expected to take part in sunday's race around london"], ["last weekend's alert"], ["hoax bomb warnings by the irish republican army (ira"], ["hoax bomb warnings by the irish republican army (ira"], ["promote china-africa cooperation", "south-south cooperation to a new level"], ["african countries and their peoples are at a critical stage in all-round national construction"], ["revolutionised south africa's racially segregated environment"], ["become a stable boy", "participate in competitions among black grooms"], ["israel and the palestinians are at odds over the role of the eu inspectors", "whether to install monitors at the crossing"], ["travel abroad"], ["many public servants in suqian were sloppily dressed", "spat and dropped litter", "made a racket in public places"], ["zhang had vividly reflected public perceptions of government officials"], ["market the country as a prime tourist destination", "revive the fortunes of the flagging tourism sector"], ["more competitors coming on stream from europe"], ["completion of the registration process"], ["being presented"], ["indicates that the judge believes that mr. sarkozy probably committed a crime"], ["cambrex shares rose 50 cents to close at $13 a share"], ["reporting that only five miners were trapped"], ["issued specific new punishments"], ["we were betrayed by the government"], ["aborted invasion of cuba by cia-trained exiles"], ["they turned the child back over"], ["protesters prayed for divine help", "protesters prayed for divine help", "priests joined the crowd"], ["adapting"], ["police violently broke up the march of some 4,000 workers"], ["produced 1.956 billion tons of coal", "produced a total of 1.85 billion tons of primary energy"], ["adding its oil, natural gas, and other energy output"], ["it will be more difficult for the government to handle issues relating to the country's agriculture and rural economy"], ["raise their income", "protect their legitimate rights"], ["average farmer makes only a third as much as his urban counterpart"], ["raise their income", "protect their legitimate rights"], ["unprecedented presence of major international carmakers"], ["the beijing auto exhibition 2006 closed monday", "a record turnout of almost 600,000 visitors in 10 days"], ["the beijing auto exhibition 2006 closed monday", "a record turnout of almost 600,000 visitors in 10 days", "a record turnout of almost 600,000 visitors", "a record turnout of almost 600,000 visitors", "a record turnout of almost 600,000 visitors in 10 days", "a record turnout of almost 600,000 visitors", "a record turnout of almost 600,000 visitors", "a record turnout of almost 600,000 visitors", "a record turnout of almost 600,000 visitors", "a record turnout of almost 600,"], ["168 people were killed"], ["meeting the immediate physical and emotional needs", "help set up a disaster response center in kenya"], ["meeting the immediate physical and emotional needs of bombing victims", "help set up a disaster response center in kenya"], ["failing to show up at his office tuesday", "cancelling a meeting with a top un official"], ["african and south american countries also held back"], ["install monitors at the crossing"], ["allowing foreign troops to leave iraq"], ["determine the smallest'sun", "evidence for planet formation"], ["build a 100 million-u.s.-dollar research and development center"], ["build a 100 million-u.s.-dollar research and development center in shanghai, china", "open next may", "construction work should begin in july"], ["us skating showdown"], ["per capita income is one of the lowest in india"], ["showcasing the country's efforts to protect human rights"], ["north korea tested an atom bomb"], ["hoax bomb warnings by the irish republican army"], ["hoax bomb warnings by the irish republican army"], ["27,000 runners are expected to take part in sunday's race around london", "500,000 lining streets to cheer them on"], ["terrorist attack"], ["run-up to the general election on may 1"], ["consolidate and expand their time-tested friendship"], ["seize the opportunities to promote china-africa cooperation", "south-south cooperation to a new level"], ["lebanon is undergoing a political crisis"], ["arrived in beirut on saturday for a 48-hour visit", "meet siniora", "heading to south lebanon for new year's eve"], ["opposition led by shiite movement hezbollah seeking to bring down siniora's government", "install a government of national unity"], ["stand trial"], ["indicates that the judge believes that mr. sarkozy probably committed a crime"], ["finding information on other sides in the world"], ["the defeat"], ["participating at the world travel market (wtm) in london", "at the china international travel market (citm) in southwestern city kunming", "seek to consolidate our presence at the mega show in china"], ["accused the israeli prison authorities of negligence toward ill prisoners"], ["completion of the registration process"], ["opening its horse competitions for people of all colours"], ["drew tens of thousands of protesters", "violent clashes erupted between demonstrators and police"], ["eventually end up in failure"], ["establishing a hemisphere-wide free trade area of the americas"], ["vividly reflected public perceptions of government officials"], ["a boycott"], ["international funding would nearly double", "government spending would rise from 56 million to 61 million"], ["the murder of lebanese ex-premier rafiq hariri"], ["further action"], ["tomlinson would have to wait another week to break the mark"], ["275-member parliament is to appoint a president and two vice presidents", "30 days to form a cabinet with parliamentary approval", "help iraq establish a new government", "un security council also called for \"the rapid formation of a representative government", "stressed \"the importance of inclusiveness, dialogue and national unity"], ["bringing in chinese capital and technology", "a special team would fly to guinea-bissau", "explore cooperative opportunities"], ["flag being thrown onto the floor by the masked youth"], ["end sufferings of the palestinians"], ["repeated israeli attacks on the palestinians"], ["spread of the bird flu"], ["warned him not to assume the premiership", "insisted they would not work for any government"], ["oversaw the murder and torture of thousands of black activists", "army crackdowns", "raids into neighboring countries"], ["production safety problems"], ["the vote comes three days before the summit"], ["we will never have a two-state solution"], ["surface in spitzer's advertisements"], ["declared its initial quarterly of five cents a share"], ["saddam has been granted a fair trial for crimes against humanity"], ["crimes against humanity involving the execution in 1982 of 148 shiite men and boys"], ["apec has made great strides in liberalizing trade and investment"], ["support for the multilateral trading system", "strengthening collective and individual actions", "promotions of high-quality regional trade agreements and free trade agreements"], ["specify apec objectives for eliminating developed members' trade and investment barriers by 2010", "those of developing members by 2020"], ["eliminating developed members' trade and investment barriers by 2010, and those of developing members by 2020"], ["liberalizing trade and investment"], ["corruption charges would not be pressed against netanyahu"], ["his role in the \"bibigate\" scandal"], ["the natinalists fled here in 1949"], ["cultural westernization"], ["slip through the legal cracks in other countries"], ["set up a joint venture in thailand in may", "process and market auto metal sheets in the country"], ["first victory", "a place in the grand prix final"], ["more than 530,000 people will compete for about 12,700 jobs"], ["falling stocks outnumbered those rising"], ["over 120 vehicles nationwide were set ablaze"]] -------------------------------------------------------------------------------- /output/pred_test.json: -------------------------------------------------------------------------------- 1 | [["foreign investment in the country's caspian sea oil fields"], ["taking a risk"], ["taken to a location in brooklyn for further analysis"], ["nothing works"], ["free and fair elections"], ["intense pressure from both democrats and republicans to resign"], ["strengthen cooperation of mutual benefit with china in all fields"], ["houses can float during high water periods"], ["assumed hong kong's defense responsibility"], ["looking for team players, not individual stars"], ["security plans were leaked"], ["keep the registered urban unemployment rate below 5 percent"], ["deny ex-dictator augusto pinochet a state funeral"], ["it was boarded by israeli commandos"], ["spending too long chatting to children"], ["a decision in november by the us department of commerce", "reduce punishing duties on canadian softwood lumber imports"], ["concentration of micro-organisms"], ["he said israel \"must be wiped off the map", "last week said israel was a \"tumour\"", "comments that were also condemned by the united nations security council"], ["find and fly home tens of thousands of exhausted and terrorised refugees"], ["he failed to connect with crossley's clearance"], ["there would be no tariffs on softwood"], ["keep the boy", "provide him with a better life"], ["cases like this could escalate"], ["escaping years of charges against his 17-year regime"], ["end of labor negotiations"], ["deployed the armed forces"], ["conduct detrimental to the team in a 107-105 loss"], ["talks with iraqi prime minister nuri al-maliki"], ["electoral process can not be concluded in months"], ["harm national security"], ["find a way forward"], ["he'll be surrounded by people if he goes to the bathroom"], ["maliki's government faces both an insurgency and mounting conflict between the sunni and shiite communities"], ["they were re-elected"], ["visiting or writing letters"], ["three turkish people"], ["reducing the standard of living"], ["his mother died in a boating accident trying to get elian and herself to the united states", "elian's relatives in miami have fought to keep the boy", "they could provide him with a better life"], ["check out the reform progress concerning rural tax and fee collection", "check out the reform progress", "check out the reform progress", "eliminate growing administrative and arbitrary fees"], ["nazi germany's systematic slaughter of an estimated six million jews between 1933 and 1945"], ["convening a special shareholders' meeting", "urging holders to drop the limits early"], ["winning gold in the 800 metres and 1,500m in athens"], ["decide how best the woman could speak"], ["reached the men's basketball final"], ["walk away"], ["recognize a two-state solution"], ["succeeded republican senator bill frist of tennessee"], ["it tipped into civil war"], ["a $1 million reward"], ["the hosting of the 2005 apec summit"], ["comparing a jewish reporter to a nazi concentration guard"], ["a crackdown on the small shiite village following a failed assassination attempt against saddam in 1982"], ["the onset of winter conditions in the earthquake areas of pakistan"], ["growth in agricultural produces export"], ["invited meshaal for a visit"], ["let heads of state and government work out at a summit next week"], ["prevent attacks against government forces"], ["injecting 100 million finnish markkaa"], ["a speech at a toronto luncheon", "a decision in november by the us department of commerce", "a decision in november by the us department of commerce", "a decision ordering the united states to reduce punishing duties on canadian softwood lumber imports"], ["a heated exchange with coach rick carlisle"], ["a suicide bombing", "the blast", "the explosion"], ["illegal trading and transactions have been found"], ["two terrorist bomb attacks in kenya and tanzania friday", "more than 100 people have died", "eight of them americans", "over 1,000 injured", "a bomb ripped through an office building"], ["the embassy blast that killed 10 people"], ["left without fuel"], ["return their territory to croatian rule"], ["his prose collection"], ["countervailing duty"], ["equalise for the visitors"], ["adds up the life spans of patriarchs"], ["an interim president from nominating himself for the top post"], ["sales dropped to $441.1 million"], ["resolve the crisis in iraq"], ["major manufacturers remained cautious about the outlook further ahead"], ["the beijing organizing committee for the 2008 olympic games (bocog) launched a worldwide solicitation"], ["united states and colombia signed a trade promotion agreement"], ["conduct detrimental to the team"], ["have a specific and well-funded response to hazardous waste dumping incidents"], ["their spouses will watch performances at the theater of beijing exhibition center in the morning"], ["discuss a wide range of issues"], ["subsidizes 11 million impoverished families"], ["formation of a new national unity government"], ["negotiating restitution issues with switzerland, poland and other european countries"], ["torture and murder"], ["monitoring and investigating the activities of a group", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "monitoring and investigating", "moni"], ["assumed dreamworks debt"], ["help the hiv/aids unit in building capacity", "assist member states in dealing with hiv/ aids"], ["expelled from the party", "sentenced to 13 years in jail"], ["composite index was up just 0.77 point at 2,843.31", "the broader all-share index rose 3.47 points to 1,762.96"], ["an eu motion to criticize china's human rights record"], ["true referendum on israel's political path"], ["receive a dividend of only $20 a share"], ["they suffocated to death"], ["three new parks have been opened"], ["mass congestion at the airport"], ["last week's clashes"], ["open future status talks"], ["bow out at next year's commonwealth games in melbourne"], ["the system has collapsed", "the system has collapsed"], ["become one of china's most well-known writers"], ["collapsed in a dressing room shortly after leading his side off the court"], ["hiroshima and nagasaki were bombed by us forces in 1945"], ["ripping two houses from their foundations"], ["snaring the young", "fanning violent crime", "the spread of aids"], ["global trade talks could collapse"], ["sale of the company"], ["backfire"], ["being buried underground for nearly 1,000 years"], ["what putin does around him in russia"], ["explosion of palestinian violence"], ["install and make operational 3,000 uranium-enriching centrifuges at a key nuclear plant", "pressing ahead with its nuclear programme", "dramatically increasing uranium enrichment capacities"], ["find the answer"], ["increases intervals between flights", "reminds travelers to check times before going to the airport"], ["creating a safe and harmonious campus for students"], ["bush's presidential plane air force one touched down", "greeted by chinese foreign minister li zhaoxing", "chinese ambassador to the united states", "chinese ambassador to china clark t. randt", "arrived aboard the presidential plane", " bush's wife laura bush", "us secretary of state condoleezza rice", "us national security adviser stephen hadley", "stay for nearly 40 hours here", "they will discuss a wide range of issues"], ["slepian was killed on oct. 23, 1999"], ["give something back to them", "stay on and continue"], ["attend the euro-mediterranean summit in barcelona on sunday"], ["the absence of injured skipper thierry henry"], ["passed by 27 votes to 11"], ["win by a shot"], ["china's new crackdown on drugs"], ["65 films screened as official out-of- competition films"], ["bonuses cannot be changed during a season"], ["sent oil prices immediately higher", "accepted african producer angola as a new member"], ["comply with government regulations"], ["their retaliatory attacks"], ["provide relief for washington from troubles in iraq"], ["alberto giuliatto fouled alessandro gamberini"], ["ruling only he can speak for his son on immigration matters"], ["the long march cz-2f rocket"], ["increased military pressure applied on armed groups", "deaths of more than 400 palestinians"], ["get first hand information on what member states expect from the secretariat", "exchange views and ideas on regional cooperation and integration issues"], ["do away with the potholes", "improve on the traffic flow"], ["presented claims for $1.5 billion damages"], ["reducing goodwill", "added $20 million to its general loan loss reserves", "established a $30 million reserve for its high-yield bond portfolio"], ["the birmingham bombing", "the olympic park bombing", "the abortion clinic bombing in atlanta"], ["spoke out from the safety of the sidelines"], ["threatening statement issued by the self-styled army for kosovo independence"], ["clearing land mines and unexploded ordnance"], ["wrote that the earth is only 6,000 years old"], ["poland forever returns where she has always belonged"], ["taken to a location in brooklyn for further analysis"], ["an uprising of the people"], ["releasing much-needed funds to the relatively poorer ex-communist states"], ["the nov. 9 attacks on three luxury hotels in the jordanian capital"], ["fight smuggling and drug trafficking"], ["scored with a superb free-kick just 11 minutes from full-time"], ["calling for chan's resignation"], ["the system has collapsed"], ["fell while abseiling on britain's highest mountain"], ["an attack"], ["an awestruck public"], ["creates an ongoing partnership with steven spielberg"], ["go to the toilet at the hotel"], ["improving the quality of health care in the country"], ["activists had attacked them"], ["the evidence was not established"], ["supply of agricultural inputs", "expansion of extension services", "irrigation development"], ["cut the size of their soviet-style militaries", "organize civilian control of their armies", "agree to buy nato compatible equipment"], ["a string of fine saves"], ["jailed for two years", "serve an additional five years in jail"], ["israeli military occupies a strip of south lebanon"], ["the beijing summit of forum on sino-africa cooperation"], ["travel restrictions routinely slapped on palestinians from the west bank and gaza strip"], ["succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic succeed republic"], ["prevent and control the epidemic"], ["improve relations with the united states"], ["make decisions based on british needs in the british interests"], ["ties might be normalized on equal terms"], ["serve an additional five years in jail"], ["tuesday's bold daylight raid to free the 72 hostages"], ["police have been forced to set up barriers", "scoop up the crayfish with excavators"], ["growing u.s. pressure on olmert and the palestinians"], ["asked my players to go all out in this match", "they didn't let me down"], ["bounce back"], ["hiv prevalence among young adults in the 15-49 age group increased slightly from 15.6 percent in 2002 to 16.2 percent in 2005"], ["solve the protection problem"], ["destroy an entire neighborhood"], ["improve fund investment returns"], ["get his second death penalty"], ["purchase more mesa stock or make a tender offer directly to mesa shareholders"], ["established a national asean free trade area (afta) unit and an asean core group in the foreign ministry"], ["passed by 27 votes to 11"], ["us officials are looking into ending reconciliation efforts with sunni insurgents", "continuing review of us policy in iraq"], ["the law would affect firms producing or importing more than one ton of a chemical per year", "put on them a duty to prove that their products are safe", "improve the labeling of products"], ["gunmen ambushed a vehicle in sirba"], ["chelsea's 2005 final win over liverpool"], ["seeking to enhance cooperation with", "looking at various areas of cooperation"], ["go to britain and italy before the end of the year"], ["meeting with abbas"], ["the government will fall"], ["100 million will achieve literacy"], ["harm these countries for their anti-israeli stance"], ["fighting between their troops and the military in sirawai"], ["massive starvation"], ["break even"], ["receiving financial aid from the world bank and the interamerican development bank"], ["return to power"], ["adds up the life spans of patriarchs"], ["get over it"], ["three were shot dead by israeli army snipers"], ["speed up assistance to flood victims", "disburse compensation as appropriate", "given a high priority", "set a deadline to finish the task within three months", "adopt raft-building techniques for houses in flood-prone areas"], ["local officials frequently visited"], ["he added"], ["locals said 10 persons were killed", "about 30 others injured"], ["anti-israeli violence"], ["killed 10 people"], ["an engine of the boeing 747 she had flown in from washington overheated"], ["a further moderate decline"], ["treaties and agreements on dealing with bilateral relations", "jointly inspected borderlines and signed documents"], ["0.5 percent drop in the department's consumer price index", "matched the decline in september", "better than the 0.3 percent dip expected by analysts", "helped by a decrease of 7 percent in energy prices", "gasoline prices plunged by 11.1 percent", "price of natural gas and home heating oil also posted big declines"], ["two soldiers, one hostage and all 14 tupac amaru revolutionary movement (mrta) hostage-takers were killed"], ["bullet wound to the leg he sustained"], ["underwent surgery"], ["visiting the wounded hostages", "inspecting the grisly scene at the mansion"], ["transatlantic partnership and european integration"], ["coordinate rescue effort and investigation"], ["a third gas blast on sunday hit luweitan colliery"], ["a third gas blast on sunday hit luweitan colliery", "killing all 24 miners underground"], ["coordinate rescue effort and investigation"], ["crack down on illegal mining"], ["sign ac milan's brazilian international playmaker kaka"], ["it tipped into civil war"], ["effective, loyal, non-sectarian iraqi security forces coupled with an effective government of national unity"], ["beefed up training teams embedded with the iraqis", "speeded up the delivery of equipment", "helped iraqi authorities to root out militias"], ["major change in his iraq policy"], ["beefed up training teams embedded with the iraqis", "speeded up the delivery of equipment", "helped iraqi authorities to root out militias"], ["japan stunned asian champion china 68-60 to reach the men's basketball final"], ["led from the beginning", "never looked back"], ["led from the beginning", "never looked back", "won it by eight points"], ["japan stunned asian champion china 68-60"], ["led from the beginning", "never looked back", "played better than the chinese in both defense and offense"], ["japan stunned asian champion china 68-60"], ["japan stunned asian champion china 68-60"], ["they played better than the chinese in both defense and offense"], ["they played better than the chinese in both defense and offense"], ["68-60"], ["leading from the very beginning", "enlarging the gap by every quarter on the way"], ["68-60"], ["leading from the very beginning", "enlarging the gap by every quarter on the way", "deserved the win", "played better than the chinese in both defense and offense", "led from the beginning", "never looked back", "won it by eight points"], ["played better than the chinese in both defense and offense"], ["the market waited for the bank of japan chief's remarks later in the day on interest rates"], ["the bank of thailand on monday made a major intervention"], ["clues on whether the us federal reserve will keep interest rates on hold or cut them in early 2007"], ["eliminate growing administrative and arbitrary fees"], ["met european union (eu) diplomats here tuesday"], ["create favorable conditions for the investigation"], ["denied any wrong-doing"], ["smuggling gangs were becoming \"increasingly sophisticated, well organised and armed"], ["ppp supporters clashed with security members"], ["if he was questioned over a political assassination"], ["condemned the terrorist acts", "extended his sympathy to the victims' families"], ["paramount assumed dreamworks' debt", "access to 59 films in dreamworks' library", "takes over all of dreamworks' current projects", "creates an ongoing partnership with steven spielberg"], ["paramount plans to sell off"], ["created successful computer animated features such as \"shrek"], ["paramount takes over all of dreamworks' current projects", "creates an ongoing partnership with steven spielberg"], ["paramount assumed dreamworks' debt", "access to 59 films in dreamworks' library", "takes over all of dreamworks' current projects", "creates an ongoing partnership with steven spielberg"], ["paramount assumed dreamworks' debt", "access to 59 films in dreamworks' library", "takes over all of dreamworks' current projects", "creates an ongoing partnership with steven spielberg"], ["paramount plans to sell off"], ["a three-day visit to china"], ["he will stay for nearly 40 hours here"], ["a three-day visit to china"], ["dispatch of a multinational force of up to 5,000 troops", "shore up security", "aid is brought into the chaotic balkan state"], ["threw its weight thursday behind a timetable for getting stricken albania back on to its feet"], ["oust president sali berisha"], ["oust president sali berisha"], ["an aid operation and early elections"], ["step up dialogue with the armed rebels"], ["their bid to oust president sali berisha"], ["shore up security", "aid is brought into the chaotic balkan state"], ["getting stricken albania back on to its feet"], ["the polls"], ["dispatch of a multinational force of up to 5,000 troops", "shore up security", "aid is brought into the chaotic balkan state"], ["their bid to oust president sali berisha"], ["their bid to oust president sali berisha"], ["dispatch of a multinational force of up to 5,000 troops"], ["china is developing so fast and so well"], ["sent oil prices immediately higher", "portends further expansion"], ["to escape a toxic red tide"], ["holding a special permit"], ["resolve the crisis in iraq"], ["intelogic trace inc., san antonio, texas, said it bought 2.7 million shares"], ["intelogic trace inc., san antonio, texas, said it bought 2.7 million shares"], ["rose 37.5 cents to close at $2.75"], ["pre-election violence"], ["the deployment of the army is against the democratic process", "it can provoke the political parties", "create obstacles to holding a free and fair election"], ["we will do it"], ["winning a scholarship in 1936"], ["sell off"], ["children's body mass index -- a measure of weight in relation to height -- continued to grow again"], ["fell to 35.67 against the dollar tuesday afternoon"], ["succession forecasts were hazy at best"], ["the same violence threatening the people and the government of pakistan as well as people in the neighborhood"], ["the same violence threatening the people and the government of pakistan"], ["none of us has done enough"], ["fighting against is the same violence threatening the people and the government of pakistan"], ["economic opportunities for the people who are living in the border areas"], ["fighting against", "the extremism, the violence", "the violence", "the violence threatening the people and the government of pakistan", "people in the neighborhood", "we have common interests along with others in this region"], ["the toilets, even in shopping complexes, are very dirty"], ["users do not flush", "they step on toilet seats", "leaving dirty shoe prints behind"], ["many users do not flush", "they step on toilet seats"], ["users do not flush", "they step on toilet seats", "leaving dirty shoe prints behind"], ["boating accident trying to get elian and herself to the united states"], ["provide him with a better life"], ["ins decision was announced wednesday"], ["drogba scored with a superb free-kick just 11 minutes from full-time"], ["drogba scored with a superb free-kick just 11 minutes from full-time"], ["scoring in chelsea's 2005 final win over liverpool"], ["savour the winning feeling again"], ["scored with a superb free-kick just 11 minutes from full-time", "seal a 1-0 win", "hit the winner against newcastle in the premiership last week", "scored with a superb free-kick just 11 minutes from full-time", "seal a 1-0 win", "scored in the final"], ["concerned about the costs of such regulation"], ["become eu-wide law"], ["years of fierce battle between health campaigners and businesses"], ["replace hazardous chemicals with safe alternatives where possible"], ["concerned about the costs of such regulation"], ["not supporting a european union move to censure china over its human rights record"], ["appreciate this perceptive and intelligent decision"], ["france and china have made great progress and in a positive direction"], ["an eu motion to criticize china's human rights record"], ["his criticism of putin and russia's handling of the war in chechnya"], ["russia's handling of the war in chechnya"], ["the move was made for the good of the country"], ["the move was made for the good of the country"], ["the use of their land by the military"], ["continue running 12 facilities in okinawa"], ["bid to fail his health"], ["the government feared that he would not come back to face charges"], ["investigate his theft of 488,000 u.s. dollars public funds"], ["introduction of more relaxed licensing laws on thursday"], ["stocking up for a major confrontation"], ["switched to gas has been a real salvation"], ["drogba scored with a superb free-kick just 11 minutes from full-time at st james' park"], ["take a minority stake in the new company"], ["customers holding contracts for waertsilae marine's undelivered ships"], ["china has reported three confirmed human cases of bird flu"], ["to look into three pneumonia cases"], ["improve their infrastructure", "raising the professional ethics of teachers"], ["strengthen disease prevention and outbreak monitoring in schools"], ["the literary landscape was dominated by several great masters"], ["published a number of collections of short stories and novels"], ["yearns to know about the outside world"], ["the literary landscape was dominated by several great masters"], ["published a number of collections of short stories and novels"], ["gets on a train by mistake"], ["leaving behind the era when the literary landscape was dominated by several great masters", "awestruck public"], ["published a number of collections of short stories and novels"], ["the literary landscape was dominated by several great masters"], ["gets on a train by mistake"], ["leaving behind the era when the literary landscape was dominated by several great masters"], ["published a number of collections of short stories and novels"], ["leaving behind the era when the literary landscape was dominated by several great masters"], ["published a number of collections of short stories and novels"], ["experiencing digestive disorder"], ["attend the euro-mediterranean summit in barcelona on sunday as planned"], ["digestive disorder"], ["experiencing digestive disorder"], ["bouteflika went to paris for medical treatment"], ["received a medical check-up", "was later sent to paris for further treatment", "the doctors who examined him"], ["experiencing digestive disorder"], ["bouteflika went to paris for medical treatment on saturday", "experienced digestive disorder", "was later sent to paris for further treatment"], ["contradictory medical recommendations issued by a local experts team"], ["bid to fail his health"], ["investigate his theft of 488,000 u.s. dollars public funds"], ["contradictory medical recommendations issued by a local experts team"], ["contradictory medical recommendations issued by a local experts team", "government was blocking his trip in a bid to fail his health"], ["the government feared that he would not come back to face charges"], ["come back to face charges"], ["bring the rok people happiness", "improve social morality", "boost friendship between the two countries"], ["the three activities"], ["enshrinement of the buddha finger sarira in the rok", "enshrinement of the buddha finger sarira in the rok", "enshrinement of the buddha finger sarira in the rok", "enshrinement of the buddha finger sarira in the rok", "enshrinement of the buddha finger sarira", "enshrinement of the buddha finger sarir"], ["unearthed from a secret chamber at the famen temple in 1987", "buried underground for nearly 1,000 years"], ["the rise of the coffee price in the international market", "the increase in the volume of other exportable produce"], ["the better performance in the agriculture sector", "the decline in the price of food items"], ["supply of agricultural inputs", "expansion of extension services", "irrigation development"], ["the rise of the coffee price in the international market", "the increase in the volume of other exportable produce"], ["the better performance in the agriculture sector", "the decline in the price of food items"], ["rise in fuel price", "increase in the volume and price of imported capital goods and construction materials"], ["supply of agricultural inputs", "expansion of extension services", "irrigation development"], ["further developing the agriculture sector"], ["open fire haphazardly", "wounding two civilians", "damage in several nearby houses"], ["increasing influence over vw"], ["if umno were to agree to his comeback"], ["domestic and foreign security concerns", "concerns about drug smuggling and illegal immigration"], ["martial law will be lifted in 41 provinces", "remain in (the other) 35 provinces"], ["looking at various areas of cooperation"], ["strengthening bilateral relations with zimbabwe"], ["bid him farewell at the end of his four-year tour of duty"], ["addressing colonial imbalances"], ["his four-year tour of duty"], ["looking at various areas of cooperation", "addressing colonial imbalances"], ["to bid him farewell at the end of his four-year tour of duty"], ["addressing colonial imbalances"], ["his four-year tour of duty"], ["dropped a reference to a conservative campaign advertisment"], ["interpreted as a direct attack on british euro-scepticism"], ["spending too long chatting to children"], ["media bias and pressure from the authorities had made a fair campaign impossible"], ["to vote at a theatre in astana"], ["bolster security at key locations throughout the city"], ["pope's request"], ["banning the officials"], ["the us commander handed them to the serb police"], ["at least 35 percent of the vote", "an advantage of 5 percentage points over his closest rival"], ["it lacked in concrete measures", "warned against inflated expectations"], ["the 206-run loss in perth"], ["israeli army forces stormed early on wednesday", "israeli army forces backed by dozens of tanks, bulldozers and armored vehicles covered by helicopters and fighting drones"], ["trying to help militants who were surrounded by israeli soldiers"], ["assistance and rehabilitation work for the flood victims must be completed"], ["relieve bangkok's flood threat"], ["damascus was preparing for war with the jewish state"], ["constructing military apparatus to try and counter the supremacy of (israeli) air, ground and naval forces"], ["stronger focus on ecce", "achieving the millennium development goals"], ["higher raw material and employment costs"], ["heavy fire was exchanged"], ["killing an unknown number of rebels"], ["sound economic cooperation"], ["consumer prices dropped by 0.5 percent in october", "registering the second consecutive monthly decline"], ["not supporting a european union move to censure china over its human rights record"], ["not supporting a european union move to censure china over its human rights record", "not to support an eu motion to criticize china's human rights record", "not to support an eu motion", "not to support an eu motion", "not to support an eu motion", "not to support an eu motion", "not to support an eu motion", "not to support an eu motion", "not to support an eu motion", "not to censure china over its human rights record", "not to cooperate and"], ["greater involvement in the region"], ["fujimori was showered with international praise"], ["his regime's atrocity in the 1980s"], ["his contribution to post-apartheid reconciliation"], ["two human infection cases of bird flu in the country"], ["improve relations with the united states"], ["a third gas blast on sunday hit luweitan colliery"], ["michael ballack join chelsea on a free transfer"], ["have struggled without him"], ["pay the dividend"], ["gunmen ambushed an iraqi army patrol"], ["a suicide attack"], ["the world council is free to pursue the matter in a civil court"], ["forbidden from suing other jews in a state court"], ["murdering his estranged lawyer wife"], ["reduce first of america's operating results for 1989"], ["eliminate the 13 management positions of the former midwest financial parent company"], ["simmering conflict between lower saxony, vw's second-biggest shareholder, and porsche, could escalate further"], ["fight porsche's increasing influence over vw"], ["his arrest"], ["the sept. 19 coup"], ["encourage people starting their own businesses"], ["the decision"], ["seized power in a bloodless coup in march 1982", "was ousted by a people's revolt in 1990"], ["a people's revolt"], ["world economic conditions", "energy security", "demographic change", "climate change", "international monetary fund and the world bank's reforms"], ["spare capacity decline"], ["arriving after a 16-hour flight"], ["deepen the crisis"], ["refused to sign the may peace agreement"], ["arsenal bounced back from two defeats to beat local rivals tottenham 3-0"], ["asked my players to go all out in this match", "they didn't let me down"], ["zambia's sugar exports will be dealt a heavy blow"], ["reducing the standard of living"], ["it had agreed to host a nimitz-class nuclear-powered aircraft carrier"], ["took over at the top"], ["allowing a woman -- identified only as witness a -- to speak without her voice being identified"], ["give her testimony relayed", "testimony was incomprehensible"], ["they could provide him with a better life"], ["they will appeal the decision"], ["provide urgently needed shelter to thousands of people"], ["grounding of some helicopters", "cancellation of some road deliveries"], ["a forum which opened here monday"], ["normal relations develop with each of these states"], ["attractive beaches in haeundae and gwangalli", "parks including yongdusan and taejongdae", "the eulsukdo migratory bird sanctuary"], ["israeli offensives"], ["curtail spiraling violence", "make progress toward ending decades of conflict"], ["supply of agricultural inputs", "expansion of extension services", "irrigation development"], ["the better performance in the agriculture sector", "the decline in the price of food items"], ["the import trade has shown a 4 percent increase"], ["the better performance in the agriculture sector", "the decline in the price of food items"], ["expansion of small-scale agro-industries", "expansion of investment that create job opportunities"], ["collapse of an earlier june summit", "rejection by french and dutch voters of an eu constitution"], ["arrested key leaders of the walk-out"], ["police stepped in and arrested key leaders of the walk-out"], ["repay certain bank debt"], ["repay certain bank debt", "establishing new operating centers", "possible acquisitions"], ["backfire"], ["emboldened by hezbollah's ability to fight israel"], ["palestinian militant groups will take the opportunity to rearm"], ["the islamic group is formally dedicated to destroying israel"], ["ceasefires only held up for a few hours", "others for a few days"], ["opened the way to negotiations"], ["meeting with abbas \"could mark an opening for a better future"], ["arsenal bounced back from two defeats"], ["inspired by their ambition", "they were disappointed by our last two results"], ["arsenal bounced back from two defeats"], ["lost successive premiership matches"], ["arsenal bounced back from two defeats", "beat local rivals tottenham 3-0", "good organisation against a good tottenham team"], ["arsenal bounced back from two defeats to beat local rivals tottenham 3-0"], ["reach a decision in the debate", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear,"], ["he was approached by prime minister ariel sharon's new party"], ["he was approached by prime minister ariel sharon's new party"], ["reach a decision in the debate", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear, responsible positions", "present the voters with clear,"], ["reach a decision in the debate"], ["countries spend 4-6 percent of national income on education"], ["781 million adult illiterates, only 100 million will achieve literacy by 2015"], ["education is key to achieving the millennium development goals"], ["achieving the millennium development goals"], ["authorities have discovered tantalizing new evidence"], ["conduct detrimental to the team in a 107-105 loss saturday at cleveland"], ["detrimental to the team"], ["i substituted for (jackson) and he came to the bench"], ["their anti-israeli stance"], ["harm these countries for their anti-israeli stance", "this is the way things are perceived in the region"], ["premeditated policy to harm these countries for their anti-israeli stance"], ["hamas's return to terror activity"], ["hamas was likely to resume its campaign of attacks"], ["voiced fears of a potential explosion of palestinian violence"], ["resume its campaign of attacks"], ["explosion of palestinian violence"], ["voting in occupied east jerusalem"], ["explosion of palestinian violence"], ["palestinian authority is not functioning in the gaza strip", "palestinian security forces are not obeying the instructions"], ["palestinian leader mahmud abbas was losing control", "hamas was likely to resume its campaign of attacks"], ["not allowing any voting in occupied east jerusalem"], ["develop techniques for drug production", "produce drugs for clinical trials"], ["invest of 115 million singapore dollars", "employ some 40 scientists"], ["the elections"], ["voters will now have to vote to return their territory to croatian rule"], ["the conditions are right"], ["bird flu is not responsible"], ["they suffocated to death"], ["increase in alcohol related arrests"], ["get out of their personal lives"], ["the lse would continue to be regulated solely by the financial services authority"], ["wipe out an entire people", "deprive them of their right to life"], ["airbus is to develop its a350 passenger jet"], ["exacerbating sectarian divisions", "marching the country to the brink of civil war"], ["become major shareholders in the new company"], ["buy and sell official posts"], ["accepted 13.25 million yuan (about 1.65 million u.s. dollars) in bribes"], ["expelled from the party", "sentenced to 13 years in jail"], ["welcome china's investment in us firms"], ["achieve their dreams"], ["educated tsinghua students on the importance of seizing dreams"], ["looking for team players", "looking at some of the guys i think are fighting for places"], ["show some strength"], ["backed his apparent death-bed accusation that he was killed by a russian \"hit squad"], ["murdered by a russian \"hit squad"], ["openly critical views of the kremlin"], ["what putin does around him in russia"], ["litvinenko backed his apparent death-bed accusation", "backed his apparent death-bed accusation", "backed his apparent death-bed accusation", "backed his apparent death-bed accusation", "backed his apparent death-bed accusation", "killed by a russian \"hit squad", "kill a person on british soil"], ["administered the fatal dose of radioactivity"], ["maintains its nuclear programme is peaceful"], ["he told a news conference"], ["an attack"], ["maintains its nuclear programme is peaceful"], ["the closures"], ["the measure may harm national security"], ["give something back to them", "stay on and continue"], ["we believe he will once again lead us where all greeks hope to go"], ["a sharp increase in drug trafficking cases last year", "tao warned that the smuggling gangs were becoming \"increasingly sophisticated", "well organised and armed"], ["cut the initial list of mascot entrants from 662 to 56", "final choice selected by the bocog"], ["fierce criticism of germany"], ["exacerbate difficulties in german- turkish relations"], ["carded rounds of of 69, 66, 66 and 70"], ["military vehicles of the ground troops rolled across the lok ma chau checkpoint", "the annual rotation", "the last part of the rotation was carried out at the stonecutters island naval base"], ["ppp supporters clashed with security members", "golkar convoy was pelted by stones by pro-ppp villagers"], ["killed by asphyxia"], ["by-elections immediately"], ["electoral process can not be concluded in months"], ["electoral process can not be concluded in months"], ["butting neath-swansea scrum-half jason spice"], ["butting neath-swansea scrum-half jason spice"], ["officials decided williams's offence only merited a three-week suspension"], ["a three-week suspension"], ["miss cardiff's celtic league clashes with llanelli and the borders", "miss the european cup match away to english giants leicester"], ["sent-off for butting neath-swansea scrum-half jason spice"], ["avoiding a lengthy ban"], ["resign"], ["claimed nearly 3,000 u.s. soldiers"], ["intense pressure from both democrats and republicans to resign"], ["claimed nearly 3,000 u.s. soldiers"], ["conduct of the iraq war", "treatment of detainees in u.s. military custody"], ["enhance their cooperation on tax and customs", "boost trade and economic development in the countries"], ["boost trade and economic development"], ["identify high-risk transactions", "speed up the processing of imports and exports"], ["committing their organizations to closer ties across a wide range of areas", "thwart smuggling, drug trafficking, fraud and tax avoidance in the three nations"], ["boost trade and economic development in the countries"], ["enhance their cooperation on tax and customs", "committing their organizations to closer ties across a wide range of areas", "seeks to thwart smuggling, drug trafficking, fraud and tax avoidance in the three nations"], ["thwart smuggling, drug trafficking, fraud and tax avoidance in the three nations"], ["boost trade and economic development in the countries"], ["identify high-risk transactions", "speed up the processing of imports and exports"], ["committing their organizations to closer ties across a wide range of areas", "an agreement to exchange information between the three revenue authorities", "help identify high-risk transactions", "speed up the processing of imports and exports", "share information on tax"], ["enhance their cooperation on tax and customs", "committing their organizations to closer ties across a wide range of areas", "seeks to thwart smuggling, drug trafficking, fraud and tax avoidance"], ["they will be larger than their parents", "put an additional burden on health services"], ["nominating himself for the top post"], ["a vicious fight for power in turkmenistan", "a new stage of struggles between russia, china, the european union and other interested parties for turkmen gas"], ["make the same comparison"], ["to study methods to modernize china's underdeveloped northeastern and western regions"], ["modernize china's underdeveloped northeastern and western regions"], ["the program", "the undp provided one million u.s. dollars", "the program", "the launch ceremony was held"], ["a roof collapsed in the attacks"], ["presence at the crossing"], ["shaking hands with palestinian minister of civil affairs mohamed dahlan"], ["removing $242 million in good will from the books", "issuing $150 million in preferred stock", "entering an exchange offer for $52 million in convertible bonds"], ["it helps us speed up repatriations"], ["find and fly home tens of thousands of exhausted and terrorised refugees"], ["continue their trek to ubundu"], ["refugees were travelling to the city from biaro by train"], ["find and fly home tens of thousands of exhausted and terrorised refugees"], ["zairean rebels asked un staff to prepare to fly home a first group of about 1,000 rwandan refugees"], ["a region to which rebels forbid access", "board a train heading for ubundu", "travelling to the city from biaro by train"], ["refugees were travelling to the city from biaro by train"], ["losing the world club cup final to internacional"], ["losing the world club cup final to internacional", "beat atletico madrid in la liga"], ["losing the world club cup final to internacional"], ["a 3-1 win at recreativo huelva on sunday"], ["keep the registered urban unemployment rate below 5 percent in the coming five years"], ["achieve the goal"], ["encourage people starting their own businesses"], ["loans to encourage people starting their own businesses"], ["draw more people into the system"], ["jewish state's destruction"], ["end economic and political crises in the palestinian territory"], ["purchase more mesa stock", "make a tender offer directly to mesa shareholders"], ["didn't respond to its offer by the close of business yesterday"], ["offer to acquire mesa airlines"], ["purchase more mesa stock", "make a tender offer directly to mesa shareholders"], ["combine the two carriers in some way"], ["millions of hiv- positive people remain unaware of their risk"], ["estimated 4.8 million south africans, or 10.8 percent of the country's total population", "living with hiv/aids", "prevalence among young adults in the 15-49 age group increased slightly", "6.3 million infections last year", "some 5 million estimated by the united nations (un) most recently"], ["living with hiv/aids"], ["cases like this could escalate"], ["gaps between various treaties closed"], ["felled at least three heavy trucks in one week"], ["do away with the potholes", "improve on the traffic flow"], ["preparations for the commonwealth summit"], ["35 million u.s. dollars from the world bank", "part of which would be used on road repairs"], ["supply kcc with road construction equipment worth 10 million dollars"], ["do away with the potholes", "improve on the traffic flow"], ["sealing the pothole at old port bell road"], ["defied major's policy of waiting to see how european monetary union (emu) developed"], ["heavy outflows continue"], ["advancing free trade and investment", "enhancing human security", "building stronger societies and a more dynamic and harmonious community"], ["rival sides deployed heavy artillery on the eighth day of duels"], ["existing list expire at the end of the year"], ["benefit from exceptionally low value added tax"], ["the incoming german government blocking a deal"], ["existing list expire at the end of the year"], ["broker a deal on which industries should be included"], ["finance ministers failed to agree tuesday on a list of services", "the ministers decided to let heads of state and government work out", "the ministers decided to let heads of state and government work out", "the ministers decided to let heads of state and government work out", "the ministers' meeting", "the ministers' meeting", "the ministers' meeting", "the ministers' meeting", "the ministers' meeting", "the ministers' meeting", "the ministers' meeting", "the ministers' meeting", "the ministers's's's's's'"], ["commit themselves to strong military ties"], ["prevents the island from developing economically"], ["the rape of a 12-year- old school girl by three us servicemen in 1995"], ["allowing us forces to continue running 12 facilities in okinawa"], ["uphold japan-us defence links"], ["to uphold japan-us defence links"], ["the rape of a 12-year- old school girl by three us servicemen in 1995"], ["locals have grown increasingly resentful", "claiming that the use of their land by the military prevents the island from developing economically"], ["use of their land by the military prevents the island from developing economically"], ["the rallies"], ["cope with fears of instability in east asia"], ["the decision to extend the leases"], ["two terrorist bomb attacks in kenya and tanzania friday"], ["a bomb ripped through an office building"], ["two terrorist bomb attacks in kenya and tanzania friday"], ["two terrorist bomb attacks in kenya and tanzania friday"], ["two terrorist bomb attacks in kenya and tanzania friday", "more than 100 people have died", "eight of them americans", "over 1,000 injured", "a bomb ripped through an office building"], ["two terrorist bomb attacks in kenya and tanzania friday"], ["two terrorist bomb attacks in kenya and tanzania friday"], ["cambodia ranks second in the world when it comes to clearing land from the lethal explosives"], ["comprised of strictly orthodox hasidic jews"], ["they have been unable to get their restitution claims addressed"], ["regain property stolen from jews during world war ii"], ["offend jewish people"], ["offend jewish people"], ["comparing a jewish reporter to a nazi concentration guard"], ["his remarks to a journalist from london's evening standard newspaper"], ["kept the economy stable", "achieved the budget surplus required by agreements with the international monetary fund"], ["feared he would lead the country to economic ruin"], ["his government pumped billions of dollars into social projects"], ["corruption scandals involving members of his cabinet and his party"], ["his popularity plummeted in 2005", "a new scandal that erupted two weeks before the first round of voting again eroded his support"], ["martin has promised to call an election within 30 days"], ["end of the israeli occupation", "departure of the syrian troops"], ["pope's request"], ["recovers its independence and sovereignty"], ["lebanon recovers its independence and sovereignty"], ["a synod on lebanon, which met in november 1995 in rome"], ["israeli military occupies a strip of south lebanon just north of israel's border"], ["we can't dissociate the spiritual from the social and the political"], ["35,000 syrian soldiers are stationed on two-thirds of lebanese territory", "israeli military occupies a strip of south lebanon"], ["lebanon recovers its independence and sovereignty"], ["giving it a stake in a hong kong firm", "eds will become an equal partner with three other major shareholders"], ["develop the sector among trade and transport companies"], ["signed a deal"], ["huge saving in costs"], ["come into line with particular wto requirements"], ["tough negotiations"], ["he said", "warning beijing that wto membership was crucial to the country's economic future", "admitted that even with the exisiting positive feeling", "tough negotiations remained ahead", "much of the important work -- and the toughest issues -- have been left to the end", "he said", "he said"], ["come into line with particular wto requirements"], ["others write the rules of the game"], ["the exisiting positive feeling"], ["welcome him", "encourage him to make all his decisions based on the well being and best interest of his son"], ["elian's father is permitted by the dictatorship of fidel castro to come to the united states"], ["based on the well being and best interest of his son"], ["he'll be surrounded by people if he goes to the bathroom"], ["welcome him", "advise him to make all his decisions based on the well being and best interest of his son"], ["he goes to the bathroom"], ["four million shares of common"], ["an initial public offering of four million shares of common"], ["his comments"], ["nazi germany's systematic slaughter of an estimated six million jews"], ["nazi germany's systematic slaughter of an estimated six million jews"], ["he said israel \"must be wiped off the map", "last week said israel was a \"tumour\" that should be moved to germany or austria"], ["nazi germany's systematic slaughter of an estimated six million jews"], ["sparked international outrage over a string of anti-israeli outbursts"], ["a string of anti-israeli outbursts"], ["nazi germany's systematic slaughter of an estimated six million jews"], ["his dislike of associated newspapers"], ["japan stunned asian champion china 68-60"], ["the united states is working together with the government of pakistan", "pakistan is a key ally in war on terror", "the extremism, the violence", "we are fighting against", "the same violence threatening the people and the government of pakistan", "we have common interests along with others in this region"], ["65 films screened as official out-of- competition films", "saudi arabia and oman will present their very first feature films"], ["how european monetary union (emu) developed"], ["fresh tory divisions over europe"], ["it had corrected the problems"], ["american voters' dissatisfaction with the situation in iraq"], ["dealt with as such"], ["stephen jackson was suspended by the indiana pacers"], ["they keep finding bones from people"], ["fell while abseiling on britain's highest mountain"], ["george w. bush arrived here saturday evening", "arrived", "greeted by chinese foreign minister li zhaoxing", "chinese ambassador to the united states", "chinese ambassador to china clark t. randt", "arrived aboard the presidential plane", "arrived aboard the presidential plane", "arrived aboard the presidential plane", "arrived aboard the presidential plane", "arrived aboard the presidential plane", "arrived aboard the presidential plane", "arrived aboard the presidential plane", "arrived aboard"], ["invitation of chinese president hu jintao"], ["pull him out of here"], ["their boat sank"], ["the clinton administration might send federal agents to pull him out of here"], ["ties might be normalized on equal terms"], ["clearing land mines and unexploded ordnance"], ["clear all of its land of mines and uxo by 2010"], ["reunite elian with his father"], ["butting neath-swansea scrum-half jason spice"], ["can play again from mid-january"], ["the rupture or collapse of a metal structure"], ["the rupture or collapse of a metal structure"], ["the accident"], ["fight against the israeli occupation"], ["the agreement is still incomplete and vague"], ["didn't respond to its offer by the close of business yesterday"], ["it helps us speed up repatriations"], ["they had been killed by asphyxia"], ["britain's refusal to part with its budget rebate", "french resistance to reforming farm subsidies"], ["identify high-risk transactions", "speed up the processing of imports and exports"], ["enhance their cooperation on tax and customs", "closer ties across a wide range of areas", "share information on tax"], ["suspected tiger rebels blew up an army tractor", "killing seven security personnel"], ["suspected tiger rebels blew up an army tractor"], ["blew up an army tractor", "killing seven security personnel"], ["rose modestly against the mark and the yen", "soared against the pound"], ["pulled other technology issues lower"], ["added to the gloom"], ["huge saving in costs"], ["giving it a stake in a hong kong firm", "eds will become an equal partner with three other major shareholders", "eds and cargonet also agreed on a \"strategic alliance"], ["develop the sector among trade and transport companies"], ["monitor the progress of un inspections", "question the iraqis on missing arms data"], ["prospective clients complained of insufficient passenger capacity and excessive fuel consumption"], ["a weak administration"], ["despite opposition from a number of countries"], ["this tournament"], ["administered the fatal dose of radioactivity"], ["keep shareholders from tendering their shares to mccaw cellular communications"], ["extremists could use violence to protest the outcome"], ["paving the way for the talks to be opened by the year's end"], ["preventing threats to sustainable development", "building a secure and favorable business environment", "enhancing human security"], ["believed to be supporting the movement"], ["the incoming german government blocking a deal"], ["assisted the 42-year-old fong in the elaborate smuggling operation"], ["leaving behind the era when the literary landscape was dominated by several great masters"], ["show that china has faith in the united states"], ["supply kcc with road construction equipment worth 10 million dollars"], ["minimize the chances of such an incident occurring in the future"], ["fund managers will face increasing pressure to sell off some of their junk"], ["pay departing investors"], ["additional 45 million people being employed"], ["keep the registered urban unemployment rate below 5 percent in the coming five years"], ["keep the registered urban unemployment rate below 5 percent"], ["encourage people starting their own businesses"], ["encourage people starting their own businesses"], ["additional 45 million people being employed"], ["adjustments will be made to enlarge the scope of employment"], ["keep the registered urban unemployment rate below 5 percent"], ["additional 45 million people being employed"], ["encourage people starting their own businesses"], ["guaranteed micro-credits to laid-off workers", "unemployed people in urban areas", "military officers transferred to civilian work"], ["ershad's supporters held rallies and marches to protest the decision"], ["ershad's supporters held rallies and marches to protest the decision"], ["a bloodless coup"], ["a people's revolt in 1990"], ["ershad's supporters held rallies and marches to protest the decision"], ["enhancing global trade by strengthening markets", "ensuring sustainability by promoting investment and encouraging efficiency"], ["strong world economy", "rising incomes", "continuing industrialization and urbanization in many economies"], ["the 110th congress that convenes early january"], ["elected unchallenged as u.s. senate majority leader for the new congress"], ["do everything we can"], ["talks on the israeli-palestinian conflict and iraq"], ["keep shareholders from tendering their shares to mccaw cellular communications"], ["index dropped 3.8%"], ["keep shareholders from tendering their shares to mccaw cellular communications"], ["dropped 3.8%"], ["keep shareholders from tendering their shares"], ["weakness in big technology stocks hurt the composite as well as the nasdaq 100 index", "the nasdaq composite index tumbled 5.39", "the nasdaq financial index lost about 1%", "the nasdaq composite index tumbled 5.39", "the nasdaq 100 index fell 1.4%", "the index dropped 3.8%", "the average daily share turnover for october is almost 148 million shares"], ["weakness in big technology stocks hurt the composite", "the nasdaq financial index lost about 1%"], ["lin and bellsouth sweetened their merger agreement"], ["lin broadcasting surged 4 5/8 to 112 5/8"], ["weakness in big technology stocks hurt the composite", "the nasdaq financial index lost about 1%"], ["weakness in big technology stocks hurt the composite"], ["lin and bellsouth sweetened their merger agreement"], ["tendering their shares to mccaw cellular communications"], ["offered $125 a share for a majority of lin's shares"], ["keep shareholders from tendering their shares to mccaw cellular communications"], ["tendering their shares to mccaw cellular communications"], ["weakness in big technology stocks hurt the composite", "the nasdaq financial index lost about 1%"], ["individual investors own 65% of the otc market's capitalization", "re-entering the market after massive selloffs"], ["individual investors own 65% of the otc market's capitalization"], ["weakness in big technology stocks hurt the composite"], ["keep shareholders from tendering their shares to mccaw cellular communications"], ["tendering their shares to mccaw cellular communications"], ["keep shareholders from tendering their shares to mccaw cellular communications"], ["hurt the composite as well as the nasdaq 100 index", "fell 1.4%", "lost about 1%", "lost about 1%"], ["keep shareholders from tendering their shares"], ["sept. 11 terror attacks"], ["taken to a location in brooklyn for further analysis", "test all the debris for toxins", "transported there"], ["encompass buildings, rooftops, and other streets that have not been searched before"], ["they still need a comprehensive assessment of the entire site", "making up plans as construction goes along"], ["encompass buildings, rooftops, and other streets that have not been searched before"], ["they still need a comprehensive assessment of the entire site"], ["further analysis"], ["reduce first of america's operating results for 1989"], ["reduce first of america's operating results for 1989"], ["eliminate the 13 management positions of the former midwest financial parent company"], ["pleaded guilty to smuggling two tonnes of cigarettes from indonesia"], ["pleaded guilty to smuggling two tonnes of cigarettes from indonesia"], ["assisted the 42-year-old fong in the elaborate smuggling operation"], ["serve an additional five years in jail"], ["smuggling two tonnes of cigarettes from indonesia"], ["fined between 3.74 million dollars and 4.9 million dollars each"], ["he was nabbed by undercover customs officers"], ["he was nabbed by undercover customs officers"], ["jailed for two years", "serve an additional five years in jail"], ["he cannot pay the fine"], ["pleaded guilty to smuggling two tonnes of cigarettes from indonesia"], ["drawn into child sex rings in greece each year"], ["conditions in which gypsy children live"], ["forced relocation of families living in the area"], ["home to athens panathinakos club"], ["the conditions in which gypsy children live near the center of athens"], ["send inspectors to north america"], ["a man attacked a group of schoolchildren with a knife"], ["a man attacked a group of schoolchildren with a knife in northwest china"], ["provide a virtual tour to those who cannot visit in person"], ["using the latest laser-scan and computer technology"], ["familiarize african school children with these sites"], ["discuss the way forward in iraq"], ["deadly violence in iraq"], ["arranged enough toilets for the crowds of people", "installed 30 permanent public toilets", "50 provisional toilets", "660 armed forces and 800 official security guards will be deployed"], ["handle private problems of the incoming visitors"], ["possibility of contamination in the future from the upper reaches"], ["1947 constitution declared the country to be pacifist"], ["1947 constitution declared the country to be pacifist"], ["today's arrests"], ["appear in sydney and melbourne courts later on tuesday"], ["acts in preparation of a terrorist act", "being a member of a terrorist group", "conspiracy to commit a terrorist act", "one man has been charged with directing a terrorist organization"], ["rush back before the july 1 handover"], ["assert their right of abode"], ["the deal involves 30 russian military attack and transport helicopters"], ["invasion of tibet"], ["broker negotiations between the palestinians and israel on a prisoner swap deal", "help major palestinian political factions to set up a unity government"], ["failed to give an exact date"], ["meshaal's visit to egypt"], ["lost his reelection bid in 2004"], ["reducing subsidies and other barriers to free trade"], ["a potential failure\" of the hong kong ministerial meeting"], ["a suspected moslem rebel lobbed a grenade at their home"], ["a suspected moslem rebel lobbed a grenade at their home"], ["fighting between their troops and the military in sirawai"], ["shrapnel wounds in the bodies"], ["fighting between their troops and the military in sirawai"], ["press ahead with its nuclear programme", "dramatically increasing uranium enrichment capacities"], ["building a new research and development (r&d) plant in singapore", "develop techniques for drug production", "produce drugs for clinical trials"], ["building a new research and development (r&d) plant in singapore"], ["employ some 40 scientists"], ["aiding economic growth", "facilitating job creation"], ["concerned about the costs of such regulation"], ["china-romania relations have been growing \"smoothly", "established \"stronger\" mutual understanding and trust", "expanding cooperation in various fields"], ["we have very limited access to information of the modern country"], ["sweeping seven golds"], ["topped the men's 94kg-division field"], ["come into line with particular wto requirements"], ["strengthen internal supervision"], ["illegal trading by nomura"], ["a recent scandal involving illegal trading by nomura"], ["electoral process can not be concluded in months"], ["united states and colombia signed a trade promotion agreement"], ["eliminating tariffs and other barriers to trade in goods and services between the two countries"], ["eliminating tariffs and other barriers to trade in goods and services between the two countries"], ["eliminate growing administrative and arbitrary fees"], ["traveled and made reports in numerous african places", "presents his experiences and understanding on africa's culture and social customs"], ["if it approves the death ruling", "saddam would be executed lawfully"], ["up to 180,000 kurds were killed"], ["sentenced to death on crimes against humanity"], ["fidel castro's illness removed him from the scene"], ["ties might be normalized on equal terms"], ["ties might be normalized on equal terms"], ["ford is willing to bid for 100% of jaguar's shares"], ["many shareholders are speculators keen for a full bid", "institutional investors unhappy over jaguar management's handling of its current financial difficulties"], ["layton propped up the liberals in a budget vote"], ["cleared martin of blame in the mismanagement of the 332-million-canadian-dollar"], ["sentiment among big manufacturers jumping to 24 in september from 21 in june"], ["in return for a collective security guarantee and economic assistance"], ["defusing the standoff over north korea's nuclear weapons programmes"], ["signed two agreements in those sectors"], ["modernize china's underdeveloped northeastern and western regions"], ["jointly launched the three-year training program"], ["developing modern agriculture and manufacturing in the two countries"], ["miscued a leading edge to a diving damien martyn at mid-on", "had cook caught behind by adam gilchrist", "bowled two overs after tea", "finished with 2-25 off 15 overs"], ["finished with 2-25 off 15 overs"], ["concern over a heel injury"], ["us efforts to reach out to sunni dissidents have failed"], ["the red cross will reopen its field offices in myanmar"], ["the country's military-run government agreed to reconsider an earlier decision to close them"], ["activists had attacked them"], ["demand gonzalez's return"], ["encourages the lebanese government to take action"], ["implementation of the un resolution"], ["un-lebanese cooperation strengthened under resolution 1636"], ["already lost the five-match series 3-1"], ["coordinate their cooperation in economy and trade", "strengthen cooperation in energy saving and environmental protection", "promote exchanges of personnel", "boost cooperation in tourism"], ["41 million tons less coal burnt each year"], ["in exchange for giving their approval for the acquisition of mannesmann"], ["huge bonuses paid to executives in the takeover of telecoms giant mannesmann"], ["the total investment will likely rise to approximately 200 billion yuan"], ["response to an earlier report carried by a shanghai-based business newspaper"], ["reduce punishing duties on canadian softwood lumber imports"], ["taliban-led violence"], ["coalition troops backed by afghan army soldiers detained 10 \"suspected terrorists", "the operation"], ["died in taliban-led violence"], ["there was a gap between the parties and substantial issues"], ["some 2,000 children are drawn into child sex rings in greece each year"], ["forced relocation of families living in the area"], ["the unique relationship between parent and child", "family reunification"], ["alleviate the shortage of transport in hospitals and clinics throughout the country"], ["operations at most clinics and district hospitals were being adversely affected"], ["massacre of 148 people from the shiite village of dujail in 1982"], ["visibility at beijing capital international airport fell to 600 meters"], ["mass congestion at the airport"], ["buying or selling government jobs"], ["it will take a long time to actually to uncover everything"], ["more than 200 remains have been found"], ["stop the construction", "find bones from people"], ["a power supply crew came upon remains while doing routine work in a manhole"], ["construction of the freedom tower and other buildings in the site should continue"], ["uncover everything"], ["have failed"], ["bruno n'gotty's introduction"], ["a decision in november by the us department of commerce", "reduce punishing duties on canadian softwood lumber imports"], ["reduce punishing duties on canadian softwood lumber imports"], ["contribution to post-apartheid reconciliation"], ["his regime's atrocity in the 1980s"], ["slightly raise the water levels of the two tributaries", "the humidity of the natural reserve will also rise"], ["an unidentified man aboard a motorcycle threw the grenade"], ["aiding economic growth", "facilitating job creation"], ["came to beijing"], ["to cover a historical summit"], ["the two countries were tied diplomatically 56 years ago"], ["its consistent adherence to the one-china policy", "support to the great cause of reunifying the country"], ["its consistent adherence to the one-china policy", "support to the great cause of reunifying the country"], ["established \"stronger\" mutual understanding and trust", "expanding cooperation in various fields"], ["developing ties with china", "strengthen cooperation of mutual benefit with china in all fields"], ["its consistent adherence to the one-china policy", "support to the great cause of reunifying the country"], ["strengthening cooperation of mutual benefit with china in all fields"], ["nine people were killed on board the turkish aid ship"], ["restoring bilateral relations with israel"], ["activists had attacked them"], ["the troops had opened fire as soon as they boarded the vessel"], ["the troops had opened fire as soon as they boarded the vessel"], ["help the hiv/aids unit in building capacity", "assist member states in dealing with hiv/ aids"], ["fight against hiv/aids"], ["get first hand information on what member states expect from the secretariat", "exchange views and ideas on regional cooperation and integration issues"], ["get first hand information on what member states expect from the secretariat", "exchange views and ideas on regional cooperation and integration issues"], ["get first hand information on what member states expect from the secretariat", "exchange views and ideas on regional cooperation and integration issues"], ["help the hiv/aids unit in building capacity", "assist member states in dealing with hiv/ aids"], ["recently visited washington"], ["refused to divulge the substance of their conversation"], ["heeding mounting calls for talks with iran or syria"], ["a power struggle"], ["pressed by reporters"], ["a power struggle"], ["saudi arabia's former ambassador here recently visited washington"], ["counsel top officials against heeding mounting calls for talks with iran or syria"], ["visited washington", "counsel top officials against heeding mounting calls for talks with iran or syria"], ["when pressed by reporters"], ["to show a live vote in the united nations general assembly"], ["the economic embargo on cuba"], ["182 countries voted for the document"], ["generate 100,000 new jobs and 6 million us dollars for the us economy"], ["prevents it from receiving financial aid"], ["called on the united states to unconditionally lift the economic embargo against cuba"], ["182 countries voted for the document", "only four countries voted against it", "one abstained"], ["children had been surviving on only 100 grams 3.5 ounces of rice a day"], ["alleviate north korea's acute food shortage"], ["marking the 85th birthday of the late president kim il-sung"], ["massive starvation would take place within 90 days"], ["mass gymnastics last sunday", "celebrations marking the 85th birthday of the late president kim il-sung"], ["50,000 children from 30 schools taking part in mass gymnastics"], ["winning 88 of the 454 seats by running as independents in 2005 legislative elections"], ["six were arrested in various areas of cairo on sunday", "remanded for two weeks days"], ["a demonstration at al-azhar university"], ["a demonstration at al-azhar university"], ["winning 88 of the 454 seats by running as independents in 2005 legislative elections"], ["money laundering", "remanded for two weeks days"], ["partner countries behind the airbus programme, france, germany, spain and britain"], ["letter of introduction to the private company"], ["purchase of russian-made military helicopters"], ["fight smuggling and drug trafficking"], ["brazil and russia agreed thursday to develop cooperation in military, trade, space and energy affairs"], ["russia recently sold arms and aircraft to venezuela"], ["trial of dujail"], ["failed assassination attempt against saddam in 1982"], ["charges of genocide against kurds"], ["saddam would be executed lawfully within 30 days"], ["saddam would be executed lawfully"], ["up to 180,000 kurds were killed", "many of them by poison gas and mass killings"], ["get his second death penalty"], ["family reunification"], ["the morning session chaired by chinese president hu jintao", "the afternoon session chaired by ethiopian prime minister ato meles zenawi"], ["their readiness to observe the obligations of asean membership"], ["the devaluation entails that the value of each unit of sugar exported is lower"], ["zambia's sugar exports will be dealt a heavy blow"], ["reduce the company's export receipts by as much as 40 percent"], ["sustaining real growth", "creating more jobs", "attracting new investment"], ["deepen and strengthen our trade ties", "provide new opportunities for u.s. businesses, manufacturers, farmers and ranchers to export", "sustaining real growth", "creating more jobs", "attracting new investment"], ["scoring just one"], ["14 soldiers were killed in two blasts"], ["attacks against government forces"], ["suspected tiger rebels blew up an army tractor", "killing seven security personnel"], ["suspected tiger rebels blew up an army tractor"], ["suspected tiger rebels blew up an army tractor", "killing seven security personnel"], ["defuse tension in the northern town of jaffna"], ["attacks against government forces"], ["talks with tamil tiger rebels", "prevent attacks against government forces"], ["defuse tension in the northern town of jaffna"], ["14 soldiers were killed in two blasts"], ["broker negotiations between the palestinians and israel on a prisoner swap deal", "help major palestinian political factions to set up a unity government"], ["cause dangerous and unpredictable landslides"], ["sanctions worth 3.1 billion dollars"], ["upper house voted by a majority", "support from the main opposition democratic party", "create the defense agency"], ["the death of a man she met while visiting her physiotherapist in ireland"], ["sack prime minister h.d. deve"], ["killing two of them", "injuring three others plus their teacher"], ["to prevent people from collecting the highly-sought after shellfish"], ["collecting the highly-sought after shellfish"], ["to escape a toxic red tide"], ["crayfish have invaded elands bay"], ["south african legislation has strict fishing laws concerning the delicacy"], ["embrace the annual heat"], ["ordered all bathhouses and car-wash facilities to stop operations", "instructed the city's administration of industry and commerce", "price bureau and police", "strengthen surveillance over the market", "maintain social order"], ["proposed acquiring mesa for $7 a share", "proposal from stateswest to combine the two carriers"], ["voters will now have to vote to return their territory to croatian rule"], ["twenty-four people were dead", "eight injured"], ["twenty-four people were dead", "eight injured"], ["generate 100,000 new jobs", "6 million us dollars for the us economy"], ["182 countries voted for the document", "only four countries voted against it", "one abstained"], ["prevents it from receiving financial aid"], ["unconditionally lift the economic embargo against cuba"], ["mass gymnastics last sunday", "celebrations marking the 85th birthday of the late president kim il-sung"], ["alleviate north korea's acute food shortage"], ["disputes over oil and money"], ["slashing iraqi oil revenues", "overproducing oil", "driving down prices"], ["the naval barricade now in force might be extended to jordan's aqaba"], ["it is a hole through which commerce flows"], ["confront an estimated 200,000 iraqi soldiers now in kuwait"], ["form the basis for a substantive discussion"], ["facilitate a final report for the next weekend's nec meeting"], ["help african cotton producers"], ["schroeder relinquished his seat"], ["arbitrarily retaining elian"], ["staying with his paternal great-uncle lazaro gonzalez", "fighting to keep the boy with him in the united states"], ["the immediate suspension barred awb and individuals from contracting with the us government", "the debarment still applied to awb"], ["no decision has been taken", "no decision on state finance for the new a350 airliner programme"], ["pay their respects to late soul legend james brown"], ["suspend overflights by greek warplanes during war games"], ["bring together the two leaders"], ["reduce crime involving the youth"], ["teaching national police cadet corps (npcc) trainees to stop crime"], ["peer influence exerts a strong influence"], ["he charged warne on 96 and was stumped"], ["sacrificed a century in brisbane"], ["hit 50 in the first innings in brisbane", "wrestled with warne's tricks early"], ["played strait-jacket cricket"], ["elian gonzalez will be reunited with his father"], ["elian gonzalez will be reunited with his father"], ["running as independents in 2005 legislative elections"], ["money laundering", "remanded for two weeks days"], ["a demonstration at al-azhar university"], ["new construction under way"], ["protests by residents", "a dispute over cost-sharing"], ["the north staged its first nuclear test"], ["trigger \"further measures"], ["protests by residents", "a dispute over cost-sharing"], ["setting a timetable for the withdrawal of troops"], ["troop reductions in iraq"], ["a phased redeployment of troops would be their top priority"], ["involved in a power struggle"], ["heeding mounting calls for talks with iran or syria"], ["over-representation of poorer african females"], ["adult hiv prevalence of less than 1 percent in 1990 soaring to almost 25 percent within 10 years"], ["the epidemic in south africa has evolved at an \"astonishing\" speed"], ["nationalist militants heaping abuse on the writer and foreign observers", "a woman hit him on the head with a file", "his car was pelted with eggs as he left"], ["include major components of household wealth", "cover all the world's countries", "shown", "shown", "shown"], ["order the suspension"], ["reunite with his son"], ["stock prices fell sharply in active trading", "volume on the new york stock exchange totaled 175.2 million shares"], ["no information"], ["signed louis saha from fulham"], ["the latest oecd report"], ["invest in a terrestrial broadcasting station"], ["had the australians at 117-4"], ["denounced the ultimatum"], ["hold talks with moroccan and polisario front leaders", "meet with officials in algiers and in nouakchott", "make \"a fresh assessment of the situation", "assess the chances of a resumption of direct talks", "move forward the peace process"], ["formed the government after winning parliamentary elections"], ["meeting israeli counterparts to discuss day-to-day affairs"], ["spent three years in jail"], ["we always sit down together and think what is the best plan"], ["achieve 8 to 10 percent growth"], ["impose greater cuts in the subsidies"], ["agricultural subsidies"], ["a deep recovery"], ["develop a nuclear weapon"], ["withdrawn from the areas"], ["flush out the lra fighter who had their bases there"], ["bowled them out for 240", "try and apply some pressure to australia", "fair play to them they didn't allow us to", "they hit us hard", "had the lead", "adam) gilchrist took the game away from us"], ["the gains of this victory against the communists were nullified"], ["advent of communism created a \"stain of shame\" on the afghan people"], ["their puppets were defeated"], ["next year's world cup finals in germany"], ["the baht rising to 34 to the dollar", "hurt our exporters' competitiveness"], ["hit a new nine-year high of 35.12 to the dollar on monday", "hit our exporters' competitiveness", "hit our exporters' competitiveness", "hit our exporters' competitiveness", "the baht rising to 34 to the dollar", "hit our exporters' competitiveness", "the baht rising to 34 to the dollar", "hit our exporters' competitiveness", "the baht rising to 34 to the dollar", "hit our exporters' competitiveness", "the baht rising to 34 to the dollar", "hit our"], ["restore stability", "pave the way for an exit of foreign troops"], ["people queued up to cast their ballots"], ["overwhelming boycott of an election in january"], ["one in the heavily fortified green zone claimed by an al-qaeda-linked group", "injured four civilians and a us marine"], ["china is launching a water supply project"], ["serious shortage of water supply and disqualified drinking water"], ["redraw maritime borders between the two countries"], ["killed four people", "displaced approximately 56, 000"], ["thanked the chief advisor for taking the initiatives for holding free and fair polls"], ["political violence in bangladesh"], ["signed them herself", "held them aloft like a victory trophy"], ["take write-offs for various problems", "prepare for a turnaround"], ["rjr nabisco agreed to sell three candy businesses to nestle for $370 million"], ["help rjr pay off debt", "boosts nestle's 7% share of the u.s. candy market to 12%"], ["offering lin holders a special $42-a-share payout"], ["meet the needs of the institutions"], ["instruct their party members and supporters to do the same", "resist from inflammatory rhetoric or premature celebrations"], ["construction of the tocoma hydropower project"], ["campaigning for the draft hours after campaign time elapsed"], ["called on his countrymen to vote for the proposed constitution"], ["accused him of breaching electoral laws", "campaigning for the draft hours", "we shall demand a similar opportunity"], ["demand a similar opportunity"], ["a televised address", "called on his countrymen to vote", "urged kenyans to come out and vote"], ["allegations that it obtained loans without sufficient collateral in return for kickbacks"], ["he also received money from hanbo"], ["write-off of capitalized servicing at the mobile home financing subsidiary"], ["declined to 118.32 yen"], ["revised down third quarter us economic growth"], ["each company remains independent", "working together to market and sell their products"], ["acquired the stock \"to develop and maintain a strategic partnership"], ["delivering a speech", "handing over the chair of saarc", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "delivering a speech", "handing over the chair of saarc to bangladesh prime minister khaleda zia", "proposed declaring the coming 2006-2015 as \"the saarc decade for poverty alleviation"], ["neither was aware of what their father had done"], ["signed their own contracts"]] --------------------------------------------------------------------------------