├── .gitmodules ├── LICENSE ├── README.md ├── TIN_Repair ├── README.md ├── apis.py ├── data │ └── suspicious_flair.json ├── environment.txt ├── repair.py └── repair_evaluation_result │ ├── AEON_suspicious_bbc_aws_MR_BERT_evaluation.txt │ ├── AEON_suspicious_bbc_aws_MR_intra_shuffle_evaluation.txt │ ├── AEON_suspicious_bbc_aws_MR_structure_evaluation.txt │ ├── AEON_suspicious_bbc_aws_MR_synonym_evaluation.txt │ ├── AEON_suspicious_bbc_azure_MR_BERT_evaluation.txt │ ├── AEON_suspicious_bbc_azure_MR_intra_shuffle_evaluation.txt │ ├── AEON_suspicious_bbc_azure_MR_structure_evaluation.txt │ ├── AEON_suspicious_bbc_azure_MR_synonym_evaluation.txt │ ├── AEON_suspicious_bbc_flair_MR_BERT_evaluation.txt │ ├── AEON_suspicious_bbc_flair_MR_intra_shuffle_evaluation.txt │ ├── AEON_suspicious_bbc_flair_MR_structure_evaluation.txt │ ├── AEON_suspicious_bbc_flair_MR_synonym_evaluation.txt │ ├── AEON_suspicious_bbc_flair_OntoNotes_MR_BERT_evaluation.txt │ ├── AEON_suspicious_bbc_flair_OntoNotes_MR_intra_shuffle_evaluation.txt │ ├── AEON_suspicious_bbc_flair_OntoNotes_MR_structure_evaluation.txt │ └── AEON_suspicious_bbc_flair_OntoNotes_MR_synonym_evaluation.txt ├── TIN_Test ├── .DS_Store ├── Data │ ├── bbc.json │ ├── bbc.txt │ └── transer.py ├── Example_results │ ├── .DS_Store │ ├── AEON_bbc_aws_MR_BERT.json │ ├── AEON_bbc_aws_MR_intra_shuffle.json │ ├── AEON_bbc_aws_MR_structure.json │ └── AEON_bbc_aws_MR_synonym.json ├── README.md ├── Transformations │ ├── .DS_Store │ ├── MR_BERT.py │ ├── MR_intra_shuffle.py │ ├── MR_structure.py │ └── MR_synonym.py ├── apis.py ├── json2text.py └── mr_perturb.py └── assets ├── NER_Repair.png └── README.md /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "TIN_Test/AEON"] 2 | path = TIN_Test/AEON 3 | url = https://github.com/CUHK-ARISE/AEON 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 RobustNLP 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Named Entity Recognition Testing 2 | 3 | 4 | We provide the data and code for NER testing and NER repairing of TIN, which are in the directory and [TIN_Test](TIN_Test) and [TIN_Repair](TIN_Repair), respectively. 5 | 6 | ### Environment 7 | 8 | ``` 9 | python == 3.7 10 | pytorch == 1.7.1 11 | Transformers == 3.3.0 12 | nltk 13 | stanfordcorenlp 14 | ``` 15 | 16 | ### Get start 17 | We provide tutorial for users about the usage of NER testing and NER repairing, which are in the corresponding directory, [TIN_Test/README.md](TIN_Test/README.md), [TIN_Repair/README.md](TIN_Repair/README.md). 18 | 19 | ### Performance of our NER repair Toolkit: 20 | Our Toolkit can reduce 42.6% of the NER errors on AWS NER system, and reduce 50.6% of the NER errors on Azure NER system. 21 | Examples of using TIN to detect the NER errors and then fix them are shown as below: 22 | ![image](assets/NER_Repair.png) 23 | 24 | ### Citation 25 | 🔭:If you use any tools or datasets in this project, please kindly cite the following paper: 26 | + Boxi Yu, Yiyan Hu, Qiuyang Mang, Wenhan Hu, Pinjia He. [**ESEC/FSE'23**] 27 | ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)*, 2023. 28 | 29 | ### Feedback 30 | Should you have any questions, please post to [the issue page](https://github.com/RobustNLP/TestNER/issues), or email Boxi Yu via boxiyu@link.cuhk.edu.cn. 31 | -------------------------------------------------------------------------------- /TIN_Repair/README.md: -------------------------------------------------------------------------------- 1 | ## Reparing the raised suspicious issues 2 | 3 | ### Environment 4 | 5 | ``` 6 | python == 3.7 7 | pytorch == 1.7.1 8 | Transformers == 3.3.0 9 | ``` 10 | 11 | ### Run 12 | 13 | To run the repairing part of **TIN**, you need import *Repairer* from *repair.py* in your code first. 14 | 15 | Then you can execute the following python codes to automatically repair the NER issues raised by the testing part: 16 | 17 | ```python 18 | MY_REPAIRER = Repairer() 19 | MY_REPAIRER.bert_init() # Initialize the BERT 20 | MY_REPAIRER.repair_suspicious_issue(input_file, output_file, api) 21 | ``` 22 | 23 | , where the *input_file* represents the path for the json file of the suspicious issues, the *output_file* represents the path of the json file to store the NER repairing result, and *api* is the function to obtain the NER prediction from the NER systems. 24 | 25 | We provide four api functions including *flair*, *flair_OntoNotes*, *azure* and *aws*, you can import them from the *apis.py*. In practice, you can invoke the *flair* and *flar_OntoNotes* directly, and invoke the *azure* and *aws* after you fill your keys in the *apis.py*. It is also availiable that you write own api function to repair other NER system, but you need ensure the output format of your api function is consistent with the api functions provided. 26 | 27 | For example, you can run these codes in the repair.py directly and the program will show the repairing result in the ```data/suspicious_flair_repair.json``` 28 | 29 | ```python 30 | apis = APIS() 31 | NER_repairer = Repairer() 32 | NER_repairer.bert_init() 33 | NER_repairer.repair_suspicious_issue("./data/suspicious_flair.json", 34 | "./data/suspicious_flair_repair_json", apis.flair) 35 | ``` 36 | 37 | We provide a demo of repairing, you can directly run: 38 | ``` 39 | python repair.py 40 | ``` 41 | ## Example results 42 | We provide the example results including transformation results and suspicious issues, the example results are in the `{ROOT_DIR}/repair_evaluation_result` folder. -------------------------------------------------------------------------------- /TIN_Repair/apis.py: -------------------------------------------------------------------------------- 1 | # to authenticate the API: use the following command: 2 | import os 3 | import time 4 | import boto3 5 | from azure.ai.textanalytics import TextAnalyticsClient 6 | from azure.core.credentials import AzureKeyCredential 7 | from flair.data import Sentence 8 | from flair.models import SequenceTagger 9 | 10 | 11 | class APIS: 12 | def aws(self, sentence, comprehend_client = boto3.client('comprehend')): 13 | entities = comprehend_client.detect_entities(Text = sentence, LanguageCode = 'en') 14 | result = [] 15 | for entity in entities["Entities"]: 16 | dict = {} 17 | dict["text"] = entity["Text"] 18 | dict["entity"] = entity["Type"] 19 | result.append(dict) 20 | return result 21 | 22 | def authenticate_client(self): 23 | key = "xxx" 24 | endpoint = "xxx" 25 | ta_credential = AzureKeyCredential(key) 26 | text_analytics_client = TextAnalyticsClient( 27 | endpoint=endpoint, 28 | credential=ta_credential) 29 | return text_analytics_client 30 | 31 | def azure(self, sentence): 32 | client = self.authenticate_client() 33 | connected = False 34 | while not connected: 35 | time.sleep(0.01) 36 | try: 37 | # entity_info = {} 38 | entity_set = set() 39 | sentence_list = [] 40 | sentence_list.append(sentence) 41 | reponse = client.recognize_entities(documents = sentence_list)[0] 42 | result = [] 43 | for entity in reponse.entities: 44 | dict = {} 45 | dict["text"] = entity.text 46 | dict["entity"] = entity.category 47 | result.append(dict) 48 | # entity_info[entity.text] = entity.category 49 | connected = True 50 | return result 51 | 52 | except Exception as err: 53 | connected = False 54 | print("Encountered exception. {}".format(err)) 55 | # def flair(self, sentence, tagger = SequenceTagger.load('flair/ner-english-ontonotes-large')): 56 | # def flair(self, sentence, tagger = SequenceTagger.load('flair/ner-english-large')): 57 | def flair(self, sentence, tagger = SequenceTagger.load('flair/ner-english-large')): 58 | sentence = Sentence(sentence) 59 | tagger.predict(sentence) 60 | # get word list and entity info 61 | word_list = [] 62 | entity_list = [] 63 | for token in sentence: 64 | word_list.append(token.text) 65 | for entity in sentence.get_spans('ner'): 66 | temp_list = [] 67 | # entity text 68 | temp_list.append(entity.text) 69 | # entity type 70 | temp_list.append(entity.get_label('ner').value) 71 | # entity indexes 72 | temp_temp_list = [] 73 | for token in entity: 74 | temp_temp_list.append(token.idx - 1) 75 | temp_list.append(temp_temp_list) 76 | entity_list.append(temp_list) 77 | result = [] 78 | for entity in entity_list: 79 | dict = {} 80 | dict["text"] = entity[0] 81 | dict["entity"] = entity[1] 82 | result.append(dict) 83 | return result 84 | #return word_list, entity_list 85 | 86 | def flair_OntoNotes(self, sentence, tagger = SequenceTagger.load('flair/ner-english-ontonotes-large')): 87 | sentence = Sentence(sentence) 88 | tagger.predict(sentence) 89 | # get word list and entity info 90 | word_list = [] 91 | entity_list = [] 92 | for token in sentence: 93 | word_list.append(token.text) 94 | for entity in sentence.get_spans('ner'): 95 | temp_list = [] 96 | # entity text 97 | temp_list.append(entity.text) 98 | # entity type 99 | temp_list.append(entity.get_label('ner').value) 100 | # entity indexes 101 | temp_temp_list = [] 102 | for token in entity: 103 | temp_temp_list.append(token.idx - 1) 104 | temp_list.append(temp_temp_list) 105 | entity_list.append(temp_list) 106 | result = [] 107 | for entity in entity_list: 108 | dict = {} 109 | dict["text"] = entity[0] 110 | dict["entity"] = entity[1] 111 | result.append(dict) 112 | return result 113 | #return word_list, entity_list 114 | if __name__ == "__main__": 115 | apis = APIS() -------------------------------------------------------------------------------- /TIN_Repair/environment.txt: -------------------------------------------------------------------------------- 1 | python 3.7 2 | pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch 3 | pip install Transformers==3.3.0 4 | -------------------------------------------------------------------------------- /TIN_Repair/repair.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | 4 | os.environ['CUDA_VISIBLE_DEVICES'] = '2' 5 | import nltk 6 | import torch 7 | import string 8 | import numpy as np 9 | from apis import APIS 10 | from copy import deepcopy 11 | from transformers import BertConfig, BertTokenizer, BertModel, RobertaTokenizer, RobertaModel, BertForMaskedLM 12 | import random 13 | 14 | ''' 15 | Automatically Repair NER errors raised by Testing part of TIN 16 | ''' 17 | class Repairer: 18 | def __init__(self): 19 | self.null_reduction = 0.2 20 | self.subword_reduction = 0.5 21 | self.inconsistency_reduction = 0 22 | 23 | def bert_init(self): 24 | self.berttokenizer = BertTokenizer.from_pretrained('bert-large-cased') 25 | self.bertmodel = BertForMaskedLM.from_pretrained("bert-large-cased") 26 | self.bertori = BertModel.from_pretrained("bert-large-cased") 27 | self.bertmodel.eval().cuda() 28 | self.bertori.eval().cuda() 29 | 30 | def __bert_predict_mask(self, tokens, masked_index, number=50): 31 | indexed_tokens = self.berttokenizer.convert_tokens_to_ids(tokens) 32 | tokens_tensor = torch.tensor([indexed_tokens]).cuda() 33 | prediction = self.bertmodel(tokens_tensor) 34 | topk = torch.topk(prediction[0][0], number) 35 | topk_idx = topk[1].tolist() 36 | topk_value = topk[0].tolist() 37 | res_tokens = self.berttokenizer.convert_ids_to_tokens(topk_idx[masked_index]) 38 | res_values = topk_value[masked_index] 39 | return res_tokens, res_values 40 | 41 | def __mask_tokens(self, tokens, lindex, rindex): 42 | new_tokens = tokens[0:lindex] + ["[MASK]"] + tokens[rindex + 1:] 43 | new_tokens = ["[CLS]"] + new_tokens + ["[SEP]"] 44 | return new_tokens 45 | 46 | def __compEmbeddings(self, tokens, lindex, rindex): 47 | indexed_tokens = self.berttokenizer.convert_tokens_to_ids(tokens) 48 | segments_ids = [1] * len(tokens) 49 | tokens_tensor = torch.tensor([indexed_tokens]).cuda() 50 | segments_tensors = torch.tensor([segments_ids]).cuda() 51 | with torch.no_grad(): 52 | outputs = self.bertori(tokens_tensor, segments_tensors, output_hidden_states=True) 53 | hidden_states = outputs[2] 54 | token_embeddings = torch.stack(hidden_states, dim=0) 55 | token_embeddings = torch.squeeze(token_embeddings, dim=1) 56 | token_embeddings = token_embeddings.permute(1, 0, 2) 57 | token_vecs_cat = [] 58 | for token in token_embeddings: 59 | cat_vec = torch.cat((token[-1], token[-2], token[-3], token[-4]), dim=0) 60 | token_vecs_cat.append(cat_vec) 61 | token_vecs_sum = [] 62 | for token in token_embeddings: 63 | sum_vec = torch.sum(token[-4:], dim=0) 64 | token_vecs_sum.append(sum_vec) 65 | 66 | ans_vec = deepcopy(token_vecs_sum[lindex]) 67 | 68 | for index in range(lindex + 1, rindex + 1): 69 | ans_vec = ans_vec + token_vecs_sum[index] 70 | 71 | ans_vec = ans_vec / (rindex - lindex + 1) 72 | return torch.unsqueeze(ans_vec, dim=0) 73 | 74 | def __compute_simlarity(self, tokens, pert_tokens, lindex1, rindex1, lindex2, rindex2): 75 | 76 | # begin cav 77 | tokens_dp = deepcopy(tokens) 78 | pert_tokens_dp = deepcopy(pert_tokens) 79 | ha = self.__compEmbeddings(tokens_dp, lindex1, rindex1) 80 | hb = self.__compEmbeddings(pert_tokens_dp, lindex2, rindex2) 81 | 82 | cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6) 83 | return float(cos(ha, hb)) 84 | 85 | def __to_possiblity(self, items): 86 | sum = 0 87 | result = [] 88 | for item in items: sum += item[1] 89 | for item in items: result.append((item[0], item[1] / sum)) 90 | return result 91 | 92 | def __consist_format(self, word1, word2): 93 | if ("#" in word1) and ("#" not in word2): return False 94 | if ("#" not in word1) and ("#" in word2): return False 95 | word1 = word1.replace("#", "") 96 | word2 = word2.replace("#", "") 97 | if (word1.isupper()) and (word2[0].isalpha() and word2[0].islower()): return False 98 | if (word1[0].isalpha() and word1[0].isupper()) and ( 99 | word2[0].isalpha() and (not word2[0].isupper())): return False 100 | word1 = word1.replace(",", "") 101 | word1 = word1.replace(".", "") 102 | word2 = word2.replace(",", "") 103 | word2 = word2.replace(".", "") 104 | if (word1.isdigit()) and (not word2.isdigit()): return False 105 | if (not word1.isdigit()) and (word2.isdigit()): return False 106 | return True 107 | 108 | def __evaluation_function(self, bert_prob, similarity): 109 | return np.exp(2.5 * similarity) * bert_prob 110 | 111 | def repair(self, sentence, ori_prediction, suspicous_words, api, \ 112 | threshold=5.5, Sthreshold=0.45): 113 | tokens = self.berttokenizer.tokenize(sentence) 114 | fixed_result = [] 115 | for item in ori_prediction: 116 | if item[0] in suspicous_words: continue 117 | removed = False 118 | for word in suspicous_words: 119 | if item[0] in word or word in item[0]: 120 | removed = True 121 | if not removed: fixed_result.append(item) 122 | Fixed_result = [] 123 | for sus_word in suspicous_words: 124 | indexset = [] 125 | lindex, rindex = -1, -1 126 | for li in range(0, len(tokens)): 127 | for ri in range(li, len(tokens)): 128 | if self.berttokenizer.convert_tokens_to_string(tokens[li:ri + 1]).replace(" ", "") \ 129 | == sus_word.replace(" ", ""): 130 | indexset.append([li, ri]) 131 | 132 | Basic_tokens = sus_word.split(" ") 133 | for curindex in indexset: 134 | lindex, rindex = curindex[0], curindex[1] 135 | dict = {} 136 | dict["NULL"] = 0 137 | for basic_token in Basic_tokens: 138 | if basic_token in string.punctuation: continue 139 | ii, jj = -1, -1 140 | for i in range(lindex, rindex + 1): 141 | for j in range(i, rindex + 1): 142 | if self.berttokenizer.convert_tokens_to_string(tokens[i:j + 1]) == basic_token: 143 | ii, jj = i, j 144 | if ii == -1: continue 145 | new_tokens = self.__mask_tokens(tokens, ii, jj) 146 | similar_words, values = self.__bert_predict_mask(new_tokens, ii + 1) 147 | 148 | for word_index in range(0, len(similar_words)): 149 | word = similar_words[word_index] 150 | value = values[word_index] 151 | 152 | if ii < jj and value < threshold: 153 | continue 154 | elif ii == jj and value < threshold: 155 | continue 156 | 157 | pert_tokens = deepcopy(tokens[0:ii + 1] + tokens[jj + 1:]) 158 | pert_tokens[ii] = word 159 | 160 | delta = 1 - (jj - ii + 1) 161 | 162 | similarity = self.__compute_simlarity(tokens, pert_tokens, \ 163 | lindex, rindex, lindex, rindex + delta) 164 | 165 | if lindex < rindex and similarity < Sthreshold: 166 | continue 167 | elif lindex == rindex and similarity < Sthreshold: 168 | continue 169 | 170 | inconsistency = False 171 | if not self.__consist_format(basic_token, word): 172 | # inconsistency = True 173 | continue 174 | 175 | pert_tokens[ii] = word 176 | request_sentence = self.berttokenizer.convert_tokens_to_string(pert_tokens) 177 | pert_result = api(request_sentence) 178 | sus_word_exist = False 179 | check_word = self.berttokenizer.convert_tokens_to_string(pert_tokens[lindex:rindex + delta + 1]) 180 | for item in pert_result: 181 | if item["text"] != check_word: continue 182 | sus_word_exist = True 183 | entity = item["entity"] 184 | if entity not in dict: dict[entity] = 0 185 | if inconsistency: 186 | dict[entity] += self.inconsistency_reduction * \ 187 | self.__evaluation_function(value, similarity) 188 | else: 189 | dict[entity] += self.__evaluation_function(value, similarity) 190 | break 191 | if not sus_word_exist: 192 | if inconsistency: 193 | dict["NULL"] += self.inconsistency_reduction * self.null_reduction * \ 194 | self.__evaluation_function(value, similarity) 195 | else: 196 | dict["NULL"] += self.null_reduction * self.__evaluation_function(value, similarity) 197 | 198 | for index in range(lindex, rindex + 1): 199 | if tokens[index] in Basic_tokens: continue 200 | if "#" in tokens[index] and len(tokens[index]) <= 4: continue 201 | if len(tokens[index]) <= 2: continue 202 | if not (index >= lindex and index <= rindex): continue 203 | new_tokens = self.__mask_tokens(tokens, index, index) 204 | similar_words, values = self.__bert_predict_mask(new_tokens, index + 1) 205 | 206 | for word_index in range(0, len(similar_words)): 207 | word = similar_words[word_index] 208 | value = values[word_index] 209 | if value < threshold: continue 210 | pert_tokens = deepcopy(tokens) 211 | pert_tokens[index] = word 212 | similarity = self.__compute_simlarity(tokens, pert_tokens, lindex, rindex, lindex, rindex) 213 | 214 | if lindex < rindex and similarity < Sthreshold: 215 | continue 216 | elif lindex == rindex and similarity < Sthreshold: 217 | continue 218 | 219 | inconsistency = False 220 | if not self.__consist_format(tokens[index], word): 221 | continue 222 | 223 | request_sentence = self.berttokenizer.convert_tokens_to_string(pert_tokens) 224 | pert_result = api(request_sentence) 225 | sus_word_exist = False 226 | check_word = self.berttokenizer.convert_tokens_to_string(pert_tokens[lindex:rindex + 1]) 227 | for item in pert_result: 228 | if item["text"] != check_word: continue 229 | sus_word_exist = True 230 | entity = item["entity"] 231 | if entity not in dict: dict[entity] = 0 232 | if inconsistency: 233 | dict[entity] += self.subword_reduction * self.inconsistency_reduction * \ 234 | self.__evaluation_function(value, similarity) 235 | else: 236 | dict[entity] += self.subword_reduction * self.__evaluation_function(value, similarity) 237 | break 238 | if not sus_word_exist: 239 | if inconsistency: 240 | dict["NULL"] += self.null_reduction * self.subword_reduction * \ 241 | self.inconsistency_reduction * \ 242 | self.__evaluation_function(value, similarity) 243 | else: 244 | dict["NULL"] += self.null_reduction * self.subword_reduction * \ 245 | self.__evaluation_function(value, similarity) 246 | 247 | items = sorted(dict.items(), key=lambda x: x[1], reverse=True) 248 | if items[0][1] > 0: 249 | items = self.__to_possiblity(items) 250 | # print("#", sus_word, items) 251 | if (items[0][0] != "NULL"): 252 | Fixed_result.append([[sus_word] + [items[0][0]], dict[items[0][0]], lindex, rindex]) 253 | else: 254 | for item in ori_prediction: 255 | if item[0] == sus_word: 256 | Fixed_result.append([item, 1e18, lindex, rindex]) 257 | break 258 | for x in Fixed_result: 259 | flag_remove = 0 260 | for y in Fixed_result: 261 | xlindex, xrindex = x[2], x[3] 262 | ylindex, yrindex = y[2], y[3] 263 | xconf, yconf = x[1], y[1] 264 | if xlindex == ylindex and xrindex == yrindex: continue 265 | if (xlindex >= ylindex and xlindex <= yrindex) or (ylindex >= xlindex and ylindex <= xrindex): 266 | 267 | if xconf < yconf: 268 | flag_remove = 1 269 | elif xconf == yconf and len(x[0][0]) > len(y[0][0]): 270 | flag_remove = 1 271 | if not flag_remove: fixed_result.append(x[0]) 272 | ''' 273 | intermediate output info 274 | ''' 275 | print("============================================================") 276 | print("Suspicious Sentence: ", sentence) 277 | print("Fixed NER Prediction: ", fixed_result) 278 | print("============================================================") 279 | return fixed_result 280 | 281 | def __get_suspicous_words(self, sentence, pert_sentence, ori_prediction, pert_prediction, sus): 282 | sus_words = set(sus) 283 | ori_dict = {} 284 | pert_dict = {} 285 | for entity in ori_prediction: 286 | if entity[0] not in ori_dict.keys(): 287 | ori_dict[entity[0]] = [] 288 | ori_dict[entity[0]].append(entity[1]) 289 | for entity in pert_prediction: 290 | if entity[0] not in pert_dict.keys(): 291 | pert_dict[entity[0]] = [] 292 | pert_dict[entity[0]].append(entity[1]) 293 | for item in ori_dict.items(): 294 | if item[0] not in pert_sentence: continue 295 | if item[0] not in pert_dict.keys(): 296 | sus_words.add(item[0]) 297 | elif pert_dict[item[0]] != item[1]: 298 | sus_words.add(item[0]) 299 | for item in pert_dict.items(): 300 | if item[0] not in sentence: continue 301 | if item[0] not in ori_dict.keys(): 302 | sus_words.add(item[0]) 303 | elif ori_dict[item[0]] != item[1]: 304 | sus_words.add(item[0]) 305 | for item in ori_dict.items(): 306 | if item[0] in pert_sentence: continue 307 | if not item[0][0].isalpha(): continue 308 | text = item[0][0].swapcase() + item[0][1:] 309 | if text not in pert_sentence: continue 310 | if text not in pert_dict.keys(): 311 | sus_words.add(item[0]) 312 | elif pert_dict[text] != item[1]: 313 | sus_words.add(item[0]) 314 | 315 | for item in pert_dict.items(): 316 | if item[0] in sentence: continue 317 | if not item[0][0].isalpha(): continue 318 | text = item[0][0].swapcase() + item[0][1:] 319 | if text not in sentence: continue 320 | if text not in ori_dict.keys(): 321 | sus_words.add(text) 322 | elif ori_dict[text] != item[1]: 323 | sus_words.add(text) 324 | result = [] 325 | for x in sus_words: result.append(x) 326 | return result 327 | 328 | def __repair_sus(self, ori_sentence, pert_sentence, ori_prediction, pert_prediction, ori_sus, pert_sus, api): 329 | ori_sus_words = self.__get_suspicous_words(ori_sentence, pert_sentence, \ 330 | ori_prediction, pert_prediction, ori_sus) 331 | pert_sus_words = self.__get_suspicous_words(pert_sentence, ori_sentence, \ 332 | pert_prediction, ori_prediction, pert_sus) 333 | print("Suspicious Entities: ", ori_sus_words) 334 | ori_fixed_result = self.repair(ori_sentence, ori_prediction, ori_sus_words, api) 335 | pert_fixed_result = self.repair(pert_sentence, pert_prediction, pert_sus_words, api) 336 | return ori_fixed_result, pert_fixed_result, ori_sus_words, pert_sus_words 337 | 338 | def repair_suspicious_issue(self, input_file, output_file, apis): 339 | with open(input_file, "r", encoding='utf-8') as f: 340 | data = json.load(f) 341 | g = open(output_file, "w", encoding="utf-8") 342 | result = [] 343 | for dict in data: 344 | ori_sus_word, pert_sus_word = [], [] 345 | if "sus_words" in dict.keys(): 346 | ori_sus_word = [dict["sus_words"][0]] 347 | pert_sus_word = [dict["sus_words"][1]] 348 | ori_fixed_result, pert_fixed_result, ori_sus_words, pert_sus_words = \ 349 | self.__repair_sus(dict["original"]["sentence"], \ 350 | dict["new"]["sentence"], \ 351 | dict["original"]["entity"], \ 352 | dict["new"]["entity"], ori_sus_word, pert_sus_word, apis) 353 | fix_ori, fix_new = deepcopy(dict["original"]), deepcopy(dict["new"]) 354 | fix_ori["entity"], fix_new["entity"] = ori_fixed_result, pert_fixed_result 355 | result.append({"original": fix_ori, "new": fix_new}) 356 | json.dump(result, g) 357 | 358 | 359 | 360 | if __name__ == "__main__": 361 | # For example: 362 | apis = APIS() 363 | NER_repairer = Repairer() 364 | NER_repairer.bert_init() 365 | NER_repairer.repair_suspicious_issue("./data/suspicious_flair.json", 366 | "./data/suspicious_flair_repair_json", apis.flair) -------------------------------------------------------------------------------- /TIN_Repair/repair_evaluation_result/AEON_suspicious_bbc_azure_MR_structure_evaluation.txt: -------------------------------------------------------------------------------- 1 | sentence: We should not wait until it is large and connected to develop the regulatory frameworks necessary to prevent a crypto shock that could have a much greater destabilising impact . 2 | sentence type ['original'] 3 | entity: ['frameworks'] 4 | ori_prediction: ['NULL'] 5 | new_prediction: ['NULL'] 6 | evaluation: NTT 7 | sentence: Did Google say its Android operating system offered a greater choice of apps and app stores than any other mobile platform ? 8 | sentence type ['mutant'] 9 | entity: ['mobile'] 10 | ori_prediction: ['NULL'] 11 | new_prediction: ['Skill'] 12 | evaluation: TF 13 | sentence: Is the Metropolitan Police sending text messages to mobile phone users it believes spoke with fraudsters pretending to be their bank ? 14 | sentence type ['mutant'] 15 | entity: ['bank'] 16 | ori_prediction: ['NULL'] 17 | new_prediction: ['PersonType'] 18 | evaluation: FF 19 | sentence: The European Space Agency ( Esa ) , an organisation backed by 22 European countries , plans to launch its Jupiter Icy Moon Explorer ( JUICE ) in April 2023 . 20 | sentence type ['original'] 21 | entity: ['Jupiter'] 22 | ori_prediction: ['NULL'] 23 | new_prediction: ['Product'] 24 | evaluation: TF 25 | sentence: Did a family representative say he died at a hospital in Houston on Tuesday from natural causes after a full and complete life ? 26 | sentence type ['mutant'] 27 | entity: ['family'] 28 | ori_prediction: ['PersonType'] 29 | new_prediction: ['PersonType'] 30 | evaluation: NTT 31 | sentence: The Nottinghamshire Wildlife Trust has been campaigning to protect the county 's green spaces and habitats since 1962 . 32 | sentence type ['original'] 33 | entity: ['green'] 34 | ori_prediction: ['Location'] 35 | new_prediction: ['NULL'] 36 | evaluation: FT 37 | sentence: Det Supt Rance said the investigation remained active . 38 | sentence type ['original'] 39 | entity: ['Det'] 40 | ori_prediction: ['PersonType'] 41 | new_prediction: ['PersonType'] 42 | evaluation: NFF 43 | sentence: I just thought that I was being greedy . 44 | sentence type ['original'] 45 | entity: ['greedy'] 46 | ori_prediction: ['PersonType'] 47 | new_prediction: ['NULL'] 48 | evaluation: FT 49 | sentence: Does Conservation charity Buglife want to restore habitats across The Marches to boost key species ? 50 | sentence type ['mutant'] 51 | entity: ['Conservation'] 52 | ori_prediction: ['NULL'] 53 | new_prediction: ['Location'] 54 | evaluation: TF 55 | sentence: Emissions of carbon dioxide are a major contributor to climate change , because they act to trap the Earth 's heat in a phenomenon known as the greenhouse effect . 56 | sentence type ['original'] 57 | entity: ['effect'] 58 | ori_prediction: ['Skill'] 59 | new_prediction: ['NULL'] 60 | evaluation: FT 61 | sentence: Emissions of carbon dioxide are a major contributor to climate change , because they act to trap the Earth 's heat in a phenomenon known as the greenhouse effect . 62 | sentence type ['original'] 63 | entity: ['greenhouse effect'] 64 | ori_prediction: ['Event'] 65 | new_prediction: ['Event'] 66 | evaluation: NTT 67 | sentence: Emissions of carbon dioxide are a major contributor to climate change , because they act to trap the Earth 's heat in a phenomenon known as the greenhouse effect . 68 | sentence type ['original'] 69 | entity: ['change'] 70 | ori_prediction: ['Skill'] 71 | new_prediction: ['Skill'] 72 | evaluation: NFF 73 | sentence: Any sensible government should be choosing to leave coal in the ground and accelerating the transition to a safe , clean and sustainable future . 74 | sentence type ['original'] 75 | entity: ['government'] 76 | ori_prediction: ['Skill'] 77 | new_prediction: ['PersonType'] 78 | evaluation: FF 79 | sentence: A family representative said he died at a hospital in Houston on Tuesday from natural causes after a full and complete life . 80 | sentence type ['original'] 81 | entity: ['family'] 82 | ori_prediction: ['NULL'] 83 | new_prediction: ['PersonType'] 84 | evaluation: TF 85 | sentence: Did Det Supt Rance say the investigation remained active ? 86 | sentence type ['mutant'] 87 | entity: ['Det'] 88 | ori_prediction: ['NULL'] 89 | new_prediction: ['PersonType'] 90 | evaluation: TF 91 | sentence: Lynne Bell said Love Learning was now propping up Gorgie Farm with £ 20 , 000 a month . 92 | sentence type ['original'] 93 | entity: ['Love Learning'] 94 | ori_prediction: ['Organization'] 95 | new_prediction: ['Organization'] 96 | evaluation: NTT 97 | sentence: A Somerset charity believes social media is contributing to poor mental health among the children it supports . 98 | sentence type ['original'] 99 | entity: ['media'] 100 | ori_prediction: ['NULL'] 101 | new_prediction: ['NULL'] 102 | evaluation: NTT 103 | sentence: A Somerset charity believes social media is contributing to poor mental health among the children it supports . 104 | sentence type ['original'] 105 | entity: ['mental health'] 106 | ori_prediction: ['NULL'] 107 | new_prediction: ['NULL'] 108 | evaluation: NFF 109 | sentence: Was he speaking to the UK parliamentary committee on the environment , food and rural affairs ? 110 | sentence type ['mutant'] 111 | entity: ['UK'] 112 | ori_prediction: ['Location'] 113 | new_prediction: ['NULL'] 114 | evaluation: FT 115 | sentence: Was he speaking to the UK parliamentary committee on the environment , food and rural affairs ? 116 | sentence type ['mutant'] 117 | entity: ['UK parliamentary'] 118 | ori_prediction: ['NULL'] 119 | new_prediction: ['Organization'] 120 | evaluation: FT 121 | sentence: Was he speaking to the UK parliamentary committee on the environment , food and rural affairs ? 122 | sentence type ['mutant'] 123 | entity: ['parliamentary'] 124 | ori_prediction: ['Organization'] 125 | new_prediction: ['NULL'] 126 | evaluation: FT 127 | sentence: Does their story match that of other Kenyans held in what the foreign ministry has called fraud factories and forced labour camps where their passports are normally confiscated and remain under the custody of the criminal gang ? 128 | sentence type ['mutant'] 129 | entity: ['criminal'] 130 | ori_prediction: ['NULL'] 131 | new_prediction: ['NULL'] 132 | evaluation: NTT 133 | sentence: Elon Musk is having a similar effect on the platform . 134 | sentence type ['original'] 135 | entity: ['platform'] 136 | ori_prediction: ['Location'] 137 | new_prediction: ['Location'] 138 | evaluation: NTT 139 | sentence: Did we 've had this paired sample strategy to ensure that we had one tube to put down in the depot and one tube to carry with us , explained Katie Stack Morgan , the deputy project scientist on Perseverance ? 140 | sentence type ['mutant'] 141 | entity: ['Perseverance'] 142 | ori_prediction: ['NULL'] 143 | new_prediction: ['Organization'] 144 | evaluation: FT 145 | sentence: Does the reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , form part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 ? 146 | sentence type ['mutant'] 147 | entity: ['of'] 148 | ori_prediction: ['NULL'] 149 | new_prediction: ['NULL'] 150 | evaluation: NTT 151 | sentence: Does the reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , form part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 ? 152 | sentence type ['mutant'] 153 | entity: ['Scientific Interest'] 154 | ori_prediction: ['NULL'] 155 | new_prediction: ['NULL'] 156 | evaluation: NTT 157 | sentence: Does the reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , form part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 ? 158 | sentence type ['mutant'] 159 | entity: ['Area of Special Scientific Interest'] 160 | ori_prediction: ['Organization'] 161 | new_prediction: ['Organization'] 162 | evaluation: NTT 163 | sentence: He has bought a new barn with the online revenue , and invested in premium camera kit , including a drone for aerial shots . 164 | sentence type ['original'] 165 | entity: ['aerial'] 166 | ori_prediction: ['Product'] 167 | new_prediction: ['Product'] 168 | evaluation: NFF 169 | sentence: Their story matches that of other Kenyans held in what the foreign ministry has called fraud factories and forced labour camps where their passports are normally confiscated and remain under the custody of the criminal gang . 170 | sentence type ['original'] 171 | entity: ['criminal'] 172 | ori_prediction: ['PersonType'] 173 | new_prediction: ['PersonType'] 174 | evaluation: NFF 175 | sentence: Did Lewis Shields , 21 , add that the course has been a real eye opener to the business world surrounding esports ? 176 | sentence type ['mutant'] 177 | entity: ['business'] 178 | ori_prediction: ['Skill'] 179 | new_prediction: ['NULL'] 180 | evaluation: FT 181 | sentence: Are WWF Scotland and RSPB Scotland among those watching the world heritage bid closely ? 182 | sentence type ['mutant'] 183 | entity: ['heritage'] 184 | ori_prediction: ['NULL'] 185 | new_prediction: ['NULL'] 186 | evaluation: NTT 187 | sentence: Erin McDaid , the trust 's head of communications , said the charity was particularly proud of its role establishing the Attenborough Nature Reserve , near Beeston , in 1966 . 188 | sentence type ['original'] 189 | entity: ['communications'] 190 | ori_prediction: ['NULL'] 191 | new_prediction: ['NULL'] 192 | evaluation: NFF 193 | sentence: Mr Musk , who bought Twitter for $ 44bn ( £ 36bn ) , said before the poll closed that he would abide by the result . 194 | sentence type ['original'] 195 | entity: ['poll'] 196 | ori_prediction: ['NULL'] 197 | new_prediction: ['Event'] 198 | evaluation: FT 199 | sentence: Has the Nottinghamshire Wildlife Trust been campaigning to protect the county 's green spaces and habitats since 1962 ? 200 | sentence type ['mutant'] 201 | entity: ['green'] 202 | ori_prediction: ['NULL'] 203 | new_prediction: ['NULL'] 204 | evaluation: NTT 205 | sentence: The Three Forks store guarantees something is available to pick up when the retrieval mission arrives . 206 | sentence type ['original'] 207 | entity: ['store'] 208 | ori_prediction: ['NULL'] 209 | new_prediction: ['NULL'] 210 | evaluation: NTT 211 | sentence: Should any sensible government be choosing to leave coal in the ground and accelerating the transition to a safe , clean and sustainable future ? 212 | sentence type ['mutant'] 213 | entity: ['government'] 214 | ori_prediction: ['NULL'] 215 | new_prediction: ['PersonType'] 216 | evaluation: FF 217 | sentence: Does he have bought a new barn with the online revenue , and invested in premium camera kit , including a drone for aerial shots ? 218 | sentence type ['mutant'] 219 | entity: ['aerial'] 220 | ori_prediction: ['NULL'] 221 | new_prediction: ['NULL'] 222 | evaluation: NTT 223 | sentence: Does this come as one of Twitter 's major advertising partners expresses doubts about its future with Twitter ? 224 | sentence type ['mutant'] 225 | entity: ['partners'] 226 | ori_prediction: ['PersonType'] 227 | new_prediction: ['NULL'] 228 | evaluation: FT 229 | sentence: Does Felix Pangibitan run his hands through what is left of his precious rice crop ? 230 | sentence type ['mutant'] 231 | entity: ['rice'] 232 | ori_prediction: ['Product'] 233 | new_prediction: ['NULL'] 234 | evaluation: FT 235 | sentence: Does Felix Pangibitan run his hands through what is left of his precious rice crop ? 236 | sentence type ['mutant'] 237 | entity: ['rice crop'] 238 | ori_prediction: ['NULL'] 239 | new_prediction: ['Product'] 240 | evaluation: FT 241 | sentence: More electricity came from renewable and nuclear power sources than from fossil fuels gas and coal , the second highest after 2020 . 242 | sentence type ['original'] 243 | entity: ['fossil'] 244 | ori_prediction: ['NULL'] 245 | new_prediction: ['NULL'] 246 | evaluation: NTT 247 | sentence: Detectives have begun contacting 70 , 000 people suspected of being victims of a sophisticated banking scam . 248 | sentence type ['original'] 249 | entity: ['scam'] 250 | ori_prediction: ['Event'] 251 | new_prediction: ['Event'] 252 | evaluation: NTT 253 | sentence: Her daughter , Edie Syta , told the Post that it 's possible her mother did not understand the severity of the storm warnings because she was not fluent in English . 254 | sentence type ['original'] 255 | entity: ['storm'] 256 | ori_prediction: ['Event'] 257 | new_prediction: ['NULL'] 258 | evaluation: FT 259 | sentence: Welsh rugby lost another former captain and a legendary broadcaster when Eddie Butler died during a charity trek in Peru , aged 65 . 260 | sentence type ['original'] 261 | entity: ['Welsh'] 262 | ori_prediction: ['NULL'] 263 | new_prediction: ['Location'] 264 | evaluation: TF 265 | sentence: Does a Somerset charity believe social media is contributing to poor mental health among the children it supports ? 266 | sentence type ['mutant'] 267 | entity: ['media'] 268 | ori_prediction: ['Skill'] 269 | new_prediction: ['NULL'] 270 | evaluation: FT 271 | sentence: Does a Somerset charity believe social media is contributing to poor mental health among the children it supports ? 272 | sentence type ['mutant'] 273 | entity: ['mental health'] 274 | ori_prediction: ['Skill'] 275 | new_prediction: ['Skill'] 276 | evaluation: NTT 277 | sentence: Was it valid from 14:00 on Tuesday until 06:00 on Wednesday ? 278 | sentence type ['mutant'] 279 | entity: ['06:00 on Wednesday'] 280 | ori_prediction: ['NULL'] 281 | new_prediction: ['NULL'] 282 | evaluation: NFF 283 | sentence: Was it valid from 14:00 on Tuesday until 06:00 on Wednesday ? 284 | sentence type ['mutant'] 285 | entity: ['14:00 on Tuesday until 06:00 on Wednesday'] 286 | ori_prediction: ['DateTime'] 287 | new_prediction: ['NULL'] 288 | evaluation: TF 289 | sentence: Was it valid from 14:00 on Tuesday until 06:00 on Wednesday ? 290 | sentence type ['mutant'] 291 | entity: ['14:00 on Tuesday'] 292 | ori_prediction: ['NULL'] 293 | new_prediction: ['DateTime'] 294 | evaluation: FT 295 | sentence: Is he also nominated at the same time for a Latin Grammy in the same category ? 296 | sentence type ['mutant'] 297 | entity: ['Latin Grammy'] 298 | ori_prediction: ['Event'] 299 | new_prediction: ['Event'] 300 | evaluation: NTT 301 | sentence: Felix Pangibitan runs his hands through what is left of his precious rice crop . 302 | sentence type ['original'] 303 | entity: ['rice'] 304 | ori_prediction: ['NULL'] 305 | new_prediction: ['NULL'] 306 | evaluation: NTT 307 | sentence: Felix Pangibitan runs his hands through what is left of his precious rice crop . 308 | sentence type ['original'] 309 | entity: ['rice crop'] 310 | ori_prediction: ['Product'] 311 | new_prediction: ['Product'] 312 | evaluation: NTT 313 | sentence: The UN 's World Food Programme ( WFP ) has a target of getting emergency food aid to 2 . 314 | sentence type ['original'] 315 | entity: ['Programme'] 316 | ori_prediction: ['Organization'] 317 | new_prediction: ['NULL'] 318 | evaluation: FT 319 | sentence: Did her daughter , Edie Syta , tell the Post that it 's possible her mother did not understand the severity of the storm warnings because she was not fluent in English ? 320 | sentence type ['mutant'] 321 | entity: ['storm'] 322 | ori_prediction: ['NULL'] 323 | new_prediction: ['NULL'] 324 | evaluation: NFF 325 | sentence: It added that employment would be secured and grown across all trades , engineering disciplines and support professions in Cumbria for decades to come . 326 | sentence type ['original'] 327 | entity: ['support'] 328 | ori_prediction: ['NULL'] 329 | new_prediction: ['NULL'] 330 | evaluation: NTT 331 | sentence: He is also nominated at the same time for a Latin Grammy in the same category . 332 | sentence type ['original'] 333 | entity: ['Latin Grammy'] 334 | ori_prediction: ['NULL'] 335 | new_prediction: ['NULL'] 336 | evaluation: NTT 337 | sentence: Is Elon Musk having a similar effect on the platform ? 338 | sentence type ['mutant'] 339 | entity: ['platform'] 340 | ori_prediction: ['NULL'] 341 | new_prediction: ['Location'] 342 | evaluation: TF 343 | sentence: This comes as one of Twitter 's major advertising partners expresses doubts about its future with Twitter . 344 | sentence type ['original'] 345 | entity: ['partners'] 346 | ori_prediction: ['NULL'] 347 | new_prediction: ['PersonType'] 348 | evaluation: TF 349 | sentence: Does Russia plan to launch its Luna 25 mission in July 2023 , putting a probe on the Moon to gather samples from its southern polar region ? 350 | sentence type ['mutant'] 351 | entity: ['southern polar region'] 352 | ori_prediction: ['Location'] 353 | new_prediction: ['Location'] 354 | evaluation: NTT 355 | sentence: Does Russia plan to launch its Luna 25 mission in July 2023 , putting a probe on the Moon to gather samples from its southern polar region ? 356 | sentence type ['mutant'] 357 | entity: ['polar region'] 358 | ori_prediction: ['NULL'] 359 | new_prediction: ['NULL'] 360 | evaluation: NFF 361 | sentence: It was valid from 14:00 on Tuesday until 06:00 on Wednesday . 362 | sentence type ['original'] 363 | entity: ['06:00 on Wednesday'] 364 | ori_prediction: ['DateTime'] 365 | new_prediction: ['NULL'] 366 | evaluation: TF 367 | sentence: It was valid from 14:00 on Tuesday until 06:00 on Wednesday . 368 | sentence type ['original'] 369 | entity: ['14:00 on Tuesday until 06:00 on Wednesday'] 370 | ori_prediction: ['NULL'] 371 | new_prediction: ['NULL'] 372 | evaluation: NFF 373 | sentence: It was valid from 14:00 on Tuesday until 06:00 on Wednesday . 374 | sentence type ['original'] 375 | entity: ['14:00 on Tuesday'] 376 | ori_prediction: ['DateTime'] 377 | new_prediction: ['DateTime'] 378 | evaluation: NTT 379 | sentence: Did the Meta official say the platform had implemented changes to reduce the potential for data to be scraped using phone numbers ? 380 | sentence type ['mutant'] 381 | entity: ['phone'] 382 | ori_prediction: ['Product'] 383 | new_prediction: ['NULL'] 384 | evaluation: FT 385 | sentence: Did the Meta official say the platform had implemented changes to reduce the potential for data to be scraped using phone numbers ? 386 | sentence type ['mutant'] 387 | entity: ['phone numbers'] 388 | ori_prediction: ['NULL'] 389 | new_prediction: ['NULL'] 390 | evaluation: NFF 391 | sentence: Mr Mousley , 45 , a former builder who now works in marketing , set up the Facebook group a month ago and already has more than 200 followers . 392 | sentence type ['original'] 393 | entity: ['Facebook'] 394 | ori_prediction: ['NULL'] 395 | new_prediction: ['Organization'] 396 | evaluation: TF 397 | sentence: Mr Mousley , 45 , a former builder who now works in marketing , set up the Facebook group a month ago and already has more than 200 followers . 398 | sentence type ['original'] 399 | entity: ['Facebook group'] 400 | ori_prediction: ['Organization'] 401 | new_prediction: ['NULL'] 402 | evaluation: TF 403 | sentence: Google said its Android operating system offered a greater choice of apps and app stores than any other mobile platform . 404 | sentence type ['original'] 405 | entity: ['mobile'] 406 | ori_prediction: ['Skill'] 407 | new_prediction: ['Skill'] 408 | evaluation: NFF 409 | sentence: Have Detectives begun contacting 70 , 000 people suspected of being victims of a sophisticated banking scam ? 410 | sentence type ['mutant'] 411 | entity: ['scam'] 412 | ori_prediction: ['NULL'] 413 | new_prediction: ['NULL'] 414 | evaluation: NTT 415 | sentence: He was speaking to the UK parliamentary committee on the environment , food and rural affairs . 416 | sentence type ['original'] 417 | entity: ['UK'] 418 | ori_prediction: ['NULL'] 419 | new_prediction: ['NULL'] 420 | evaluation: NTT 421 | sentence: He was speaking to the UK parliamentary committee on the environment , food and rural affairs . 422 | sentence type ['original'] 423 | entity: ['UK parliamentary'] 424 | ori_prediction: ['Organization'] 425 | new_prediction: ['Organization'] 426 | evaluation: NTT 427 | sentence: He was speaking to the UK parliamentary committee on the environment , food and rural affairs . 428 | sentence type ['original'] 429 | entity: ['parliamentary'] 430 | ori_prediction: ['NULL'] 431 | new_prediction: ['NULL'] 432 | evaluation: NTT 433 | sentence: Are emissions of carbon dioxide a major contributor to climate change , because they act to trap the Earth 's heat in a phenomenon known as the greenhouse effect ? 434 | sentence type ['mutant'] 435 | entity: ['effect'] 436 | ori_prediction: ['Event'] 437 | new_prediction: ['NULL'] 438 | evaluation: FT 439 | sentence: Are emissions of carbon dioxide a major contributor to climate change , because they act to trap the Earth 's heat in a phenomenon known as the greenhouse effect ? 440 | sentence type ['mutant'] 441 | entity: ['greenhouse effect'] 442 | ori_prediction: ['NULL'] 443 | new_prediction: ['Event'] 444 | evaluation: FT 445 | sentence: Are emissions of carbon dioxide a major contributor to climate change , because they act to trap the Earth 's heat in a phenomenon known as the greenhouse effect ? 446 | sentence type ['mutant'] 447 | entity: ['change'] 448 | ori_prediction: ['NULL'] 449 | new_prediction: ['NULL'] 450 | evaluation: NTT 451 | sentence: The Metropolitan Police is sending text messages to mobile phone users it believes spoke with fraudsters pretending to be their bank . 452 | sentence type ['original'] 453 | entity: ['bank'] 454 | ori_prediction: ['Location'] 455 | new_prediction: ['PersonType'] 456 | evaluation: TF 457 | sentence: Does the European Space Agency ( Esa ) , an organisation backed by 22 European countries , plan to launch its Jupiter Icy Moon Explorer ( JUICE ) in April 2023 ? 458 | sentence type ['mutant'] 459 | entity: ['Jupiter'] 460 | ori_prediction: ['Location'] 461 | new_prediction: ['Location'] 462 | evaluation: NTT 463 | sentence: The report simply calls for a clear end date for the new oil and gas licensing rounds in the North Sea so the government can continue to demonstrate its international climate leadership . 464 | sentence type ['original'] 465 | entity: ['leadership'] 466 | ori_prediction: ['NULL'] 467 | new_prediction: ['NULL'] 468 | evaluation: NTT 469 | sentence: Conservation charity Buglife wants to restore habitats across The Marches to boost key species . 470 | sentence type ['original'] 471 | entity: ['Conservation'] 472 | ori_prediction: ['Skill'] 473 | new_prediction: ['NULL'] 474 | evaluation: FT 475 | sentence: The One Million Trees Project has the backing of planting organisations , landowners and Belfast City councillors . 476 | sentence type ['original'] 477 | entity: ['planting'] 478 | ori_prediction: ['NULL'] 479 | new_prediction: ['NULL'] 480 | evaluation: NTT 481 | sentence: Russia plans to launch its Luna 25 mission in July 2023 , putting a probe on the Moon to gather samples from its southern polar region . 482 | sentence type ['original'] 483 | entity: ['southern polar region'] 484 | ori_prediction: ['NULL'] 485 | new_prediction: ['NULL'] 486 | evaluation: NFF 487 | sentence: Russia plans to launch its Luna 25 mission in July 2023 , putting a probe on the Moon to gather samples from its southern polar region . 488 | sentence type ['original'] 489 | entity: ['polar region'] 490 | ori_prediction: ['Location'] 491 | new_prediction: ['Location'] 492 | evaluation: NTT 493 | sentence: Are people in Guernsey making Christmas dinner being told to prevent fats or oils from going down the sink ? 494 | sentence type ['mutant'] 495 | entity: ['fats'] 496 | ori_prediction: ['Product'] 497 | new_prediction: ['Product'] 498 | evaluation: NTT 499 | sentence: Does the One Million Trees Project have the backing of planting organisations , landowners and Belfast City councillors ? 500 | sentence type ['mutant'] 501 | entity: ['planting'] 502 | ori_prediction: ['Skill'] 503 | new_prediction: ['NULL'] 504 | evaluation: FT 505 | sentence: We are fortunate to have a strong supply chain for the nuclear industry in the Workington constituency . 506 | sentence type ['original'] 507 | entity: ['constituency'] 508 | ori_prediction: ['Location'] 509 | new_prediction: ['NULL'] 510 | evaluation: FT 511 | sentence: The reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , forms part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 . 512 | sentence type ['original'] 513 | entity: ['Scientific Interest'] 514 | ori_prediction: ['Organization'] 515 | new_prediction: ['NULL'] 516 | evaluation: FT 517 | sentence: The reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , forms part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 . 518 | sentence type ['original'] 519 | entity: ['of'] 520 | ori_prediction: ['Organization'] 521 | new_prediction: ['Organization'] 522 | evaluation: NFF 523 | sentence: The reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , forms part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 . 524 | sentence type ['original'] 525 | entity: ['Area of Special Scientific Interest'] 526 | ori_prediction: ['NULL'] 527 | new_prediction: ['Organization'] 528 | evaluation: FT 529 | sentence: Meta says it is reviewing this decision carefully . 530 | sentence type ['original'] 531 | entity: ['Meta'] 532 | ori_prediction: ['Organization'] 533 | new_prediction: ['Organization'] 534 | evaluation: NFF 535 | sentence: Are we fortunate to have a strong supply chain for the nuclear industry in the Workington constituency ? 536 | sentence type ['mutant'] 537 | entity: ['constituency'] 538 | ori_prediction: ['NULL'] 539 | new_prediction: ['NULL'] 540 | evaluation: NTT 541 | sentence: Lucy Powell MP , Labour 's shadow culture secretary , criticised the decision to remove obligations over legal but harmful material . 542 | sentence type ['original'] 543 | entity: ['MP'] 544 | ori_prediction: ['PersonType'] 545 | new_prediction: ['NULL'] 546 | evaluation: NTT 547 | sentence: Did welsh rugby lose another former captain and a legendary broadcaster when Eddie Butler died during a charity trek in Peru , aged 65 ? 548 | sentence type ['mutant'] 549 | entity: ['welsh'] 550 | ori_prediction: ['Organization'] 551 | new_prediction: ['Organization'] 552 | evaluation: NFF 553 | sentence: Did news of the meeting with Apple come after Mr Musk was told he faced huge work ahead to bring Twitter into compliance with new European Union rules on disinformation or face a possible ban ? 554 | sentence type ['mutant'] 555 | entity: ['meeting'] 556 | ori_prediction: ['Event'] 557 | new_prediction: ['NULL'] 558 | evaluation: TF 559 | sentence: Did Sir Jon , who is deputy governor for financial stability at the Bank , also say the recent volatility in the value of cryptocurrencies posed a threat ? 560 | sentence type ['mutant'] 561 | entity: ['financial'] 562 | ori_prediction: ['NULL'] 563 | new_prediction: ['NULL'] 564 | evaluation: NTT 565 | sentence: The Meta official said the platform had implemented changes to reduce the potential for data to be scraped using phone numbers . 566 | sentence type ['original'] 567 | entity: ['phone'] 568 | ori_prediction: ['NULL'] 569 | new_prediction: ['NULL'] 570 | evaluation: NTT 571 | sentence: The Meta official said the platform had implemented changes to reduce the potential for data to be scraped using phone numbers . 572 | sentence type ['original'] 573 | entity: ['phone numbers'] 574 | ori_prediction: ['Product'] 575 | new_prediction: ['NULL'] 576 | evaluation: TF 577 | sentence: Will it be a challenge to Starlink , the satellite broadband service developed by Elon Musk ? 578 | sentence type ['mutant'] 579 | entity: ['broadband'] 580 | ori_prediction: ['NULL'] 581 | new_prediction: ['NULL'] 582 | evaluation: NTT 583 | sentence: It was one of the reasons Belfast was listed as one of the world 's leading climate action cities in 2022 . 584 | sentence type ['original'] 585 | entity: ['climate action'] 586 | ori_prediction: ['NULL'] 587 | new_prediction: ['NULL'] 588 | evaluation: NTT 589 | sentence: Did Lynne Bell say Love Learning was now propping up Gorgie Farm with £ 20 , 000 a month ? 590 | sentence type ['mutant'] 591 | entity: ['Love Learning'] 592 | ori_prediction: ['NULL'] 593 | new_prediction: ['Organization'] 594 | evaluation: FT 595 | sentence: Mr Johnson said Cumbria County Council had brokered an agreement with Mr Hagan to give the peripatetic porkers a new home . 596 | sentence type ['original'] 597 | entity: ['home'] 598 | ori_prediction: ['Location'] 599 | new_prediction: ['NULL'] 600 | evaluation: FT 601 | sentence: Does the Three Forks store guarantee something is available to pick up when the retrieval mission arrives ? 602 | sentence type ['mutant'] 603 | entity: ['store'] 604 | ori_prediction: ['Location'] 605 | new_prediction: ['NULL'] 606 | evaluation: FT 607 | sentence: Did it add that employment would be secured and grown across all trades , engineering disciplines and support professions in Cumbria for decades to come ? 608 | sentence type ['mutant'] 609 | entity: ['support'] 610 | ori_prediction: ['Skill'] 611 | new_prediction: ['Skill'] 612 | evaluation: NFF 613 | sentence: Does the report simply call for a clear end date for the new oil and gas licensing rounds in the North Sea so the government can continue to demonstrate its international climate leadership ? 614 | sentence type ['mutant'] 615 | entity: ['leadership'] 616 | ori_prediction: ['Skill'] 617 | new_prediction: ['NULL'] 618 | evaluation: FT 619 | sentence: Dr Mariel Purcell , consultant in spinal injuries , said that since he started his career 30 years ago more patients now had the potential to get back on their feet . 620 | sentence type ['original'] 621 | entity: ['injuries'] 622 | ori_prediction: ['PersonType'] 623 | new_prediction: ['NULL'] 624 | evaluation: FT 625 | sentence: We 've had this paired sample strategy to ensure that we had one tube to put down in the depot and one tube to carry with us , explained Katie Stack Morgan , the deputy project scientist on Perseverance . 626 | sentence type ['original'] 627 | entity: ['Perseverance'] 628 | ori_prediction: ['Organization'] 629 | new_prediction: ['Organization'] 630 | evaluation: NTT 631 | sentence: Did I just think that I was being greedy ? 632 | sentence type ['mutant'] 633 | entity: ['greedy'] 634 | ori_prediction: ['NULL'] 635 | new_prediction: ['NULL'] 636 | evaluation: NTT 637 | sentence: This vehicle incorporates the big thrusters needed to make manoeuvres . 638 | sentence type ['original'] 639 | entity: ['thrusters'] 640 | ori_prediction: ['Product'] 641 | new_prediction: ['Product'] 642 | evaluation: NTT 643 | sentence: Does Mr Mousley , 45 , a former builder who now works in marketing , set up the Facebook group a month ago and already have more than 200 followers ? 644 | sentence type ['mutant'] 645 | entity: ['Facebook'] 646 | ori_prediction: ['Organization'] 647 | new_prediction: ['Organization'] 648 | evaluation: NTT 649 | sentence: Does Mr Mousley , 45 , a former builder who now works in marketing , set up the Facebook group a month ago and already have more than 200 followers ? 650 | sentence type ['mutant'] 651 | entity: ['Facebook group'] 652 | ori_prediction: ['NULL'] 653 | new_prediction: ['NULL'] 654 | evaluation: NFF 655 | sentence: Did Erin McDaid , the trust 's head of communications , say the charity was particularly proud of its role establishing the Attenborough Nature Reserve , near Beeston , in 1966 ? 656 | sentence type ['mutant'] 657 | entity: ['communications'] 658 | ori_prediction: ['PersonType'] 659 | new_prediction: ['PersonType'] 660 | evaluation: NTT 661 | sentence: His simple heartfelt videos struck a chord across the Philippines , especially among the poorest communities which have been hit by multiple crises . 662 | sentence type ['original'] 663 | entity: ['crises'] 664 | ori_prediction: ['Event'] 665 | new_prediction: ['Event'] 666 | evaluation: NFF 667 | sentence: Does the UN 's World Food Programme ( WFP ) have a target of getting emergency food aid to 2 ? 668 | sentence type ['mutant'] 669 | entity: ['Programme'] 670 | ori_prediction: ['NULL'] 671 | new_prediction: ['NULL'] 672 | evaluation: NTT 673 | sentence: Was it one of the reasons Belfast was listed as one of the world 's leading climate action cities in 2022 ? 674 | sentence type ['mutant'] 675 | entity: ['climate action'] 676 | ori_prediction: ['Skill'] 677 | new_prediction: ['NULL'] 678 | evaluation: FT 679 | sentence: Does she feel she learned more from educational gaming than traditional schoolwork as it was a good way to advance motor , maths , English and history skills without it being boring ? 680 | sentence type ['mutant'] 681 | entity: ['educational'] 682 | ori_prediction: ['Skill'] 683 | new_prediction: ['NULL'] 684 | evaluation: FT 685 | sentence: Does Meta say it is reviewing this decision carefully ? 686 | sentence type ['mutant'] 687 | entity: ['Meta'] 688 | ori_prediction: ['Person'] 689 | new_prediction: ['Person'] 690 | evaluation: NTT 691 | sentence: WWF Scotland and RSPB Scotland are among those watching the world heritage bid closely . 692 | sentence type ['original'] 693 | entity: ['heritage'] 694 | ori_prediction: ['Skill'] 695 | new_prediction: ['NULL'] 696 | evaluation: FT 697 | sentence: She feels she learned more from educational gaming than traditional schoolwork as it was a good way to advance motor , maths , English and history skills without it being boring . 698 | sentence type ['original'] 699 | entity: ['educational'] 700 | ori_prediction: ['NULL'] 701 | new_prediction: ['NULL'] 702 | evaluation: NTT 703 | sentence: Sir Jon , who is deputy governor for financial stability at the Bank , also said the recent volatility in the value of cryptocurrencies posed a threat . 704 | sentence type ['original'] 705 | entity: ['financial'] 706 | ori_prediction: ['Skill'] 707 | new_prediction: ['NULL'] 708 | evaluation: FT 709 | sentence: Did Lucy Powell MP , Labour 's shadow culture secretary , criticise the decision to remove obligations over legal but harmful material ? 710 | sentence type ['mutant'] 711 | entity: ['MP'] 712 | ori_prediction: ['NULL'] 713 | new_prediction: ['NULL'] 714 | evaluation: NTT 715 | sentence: It will be a challenge to Starlink , the satellite broadband service developed by Elon Musk . 716 | sentence type ['original'] 717 | entity: ['broadband'] 718 | ori_prediction: ['Skill'] 719 | new_prediction: ['NULL'] 720 | evaluation: FT 721 | sentence: Did Mr Musk , who bought Twitter for $ 44bn ( £ 36bn ) , say before the poll closed that he would abide by the result ? 722 | sentence type ['mutant'] 723 | entity: ['poll'] 724 | ori_prediction: ['Event'] 725 | new_prediction: ['Event'] 726 | evaluation: NTT 727 | sentence: Did more electricity come from renewable and nuclear power sources than from fossil fuels gas and coal , the second highest after 2020 ? 728 | sentence type ['mutant'] 729 | entity: ['fossil'] 730 | ori_prediction: ['Product'] 731 | new_prediction: ['NULL'] 732 | evaluation: FT 733 | sentence: Did Dr Mariel Purcell , consultant in spinal injuries , say that since he started his career 30 years ago more patients now had the potential to get back on their feet ? 734 | sentence type ['mutant'] 735 | entity: ['injuries'] 736 | ori_prediction: ['NULL'] 737 | new_prediction: ['NULL'] 738 | evaluation: NTT 739 | sentence: News of the meeting with Apple came after Mr Musk was told he faced huge work ahead to bring Twitter into compliance with new European Union rules on disinformation or face a possible ban . 740 | sentence type ['original'] 741 | entity: ['meeting'] 742 | ori_prediction: ['NULL'] 743 | new_prediction: ['NULL'] 744 | evaluation: NFF 745 | sentence: People in Guernsey making Christmas dinner are being told to prevent fats or oils from going down the sink . 746 | sentence type ['original'] 747 | entity: ['fats'] 748 | ori_prediction: ['NULL'] 749 | new_prediction: ['Product'] 750 | evaluation: FT 751 | sentence: Does this vehicle incorporate the big thrusters needed to make manoeuvres ? 752 | sentence type ['mutant'] 753 | entity: ['thrusters'] 754 | ori_prediction: ['NULL'] 755 | new_prediction: ['Product'] 756 | evaluation: FT 757 | sentence: Should we not wait until it is large and connected to develop the regulatory frameworks necessary to prevent a crypto shock that could have a much greater destabilising impact ? 758 | sentence type ['mutant'] 759 | entity: ['frameworks'] 760 | ori_prediction: ['Skill'] 761 | new_prediction: ['NULL'] 762 | evaluation: FT 763 | sentence: Did his simple heartfelt videos strike a chord across the Philippines , especially among the poorest communities which have been hit by multiple crises ? 764 | sentence type ['mutant'] 765 | entity: ['crises'] 766 | ori_prediction: ['NULL'] 767 | new_prediction: ['Event'] 768 | evaluation: TF 769 | sentence: Did Mr Johnson say Cumbria County Council had brokered an agreement with Mr Hagan to give the peripatetic porkers a new home ? 770 | sentence type ['mutant'] 771 | entity: ['home'] 772 | ori_prediction: ['NULL'] 773 | new_prediction: ['NULL'] 774 | evaluation: NTT 775 | sentence: Lewis Shields , 21 , added that the course has been a real eye opener to the business world surrounding esports . 776 | sentence type ['original'] 777 | entity: ['business'] 778 | ori_prediction: ['NULL'] 779 | new_prediction: ['NULL'] 780 | evaluation: NTT 781 | -------------------------------------------------------------------------------- /TIN_Repair/repair_evaluation_result/AEON_suspicious_bbc_flair_MR_structure_evaluation.txt: -------------------------------------------------------------------------------- 1 | sentence: Did The Urdd , Wales Today and Aberystwyth University all celebrate significant anniversaries while we said goodbye to two heroes of Welsh rugby and one of our most beloved actresses ? 2 | sentence type ['mutant'] 3 | entity: ['The Urdd'] 4 | ori_prediction: ['ORG'] 5 | new_prediction: ['ORG'] 6 | evaluation: NTT 7 | sentence: Did The Urdd , Wales Today and Aberystwyth University all celebrate significant anniversaries while we said goodbye to two heroes of Welsh rugby and one of our most beloved actresses ? 8 | sentence type ['mutant'] 9 | entity: ['The'] 10 | ori_prediction: ['NULL'] 11 | new_prediction: ['NULL'] 12 | evaluation: NTT 13 | sentence: Did The Urdd , Wales Today and Aberystwyth University all celebrate significant anniversaries while we said goodbye to two heroes of Welsh rugby and one of our most beloved actresses ? 14 | sentence type ['mutant'] 15 | entity: ['Urdd'] 16 | ori_prediction: ['NULL'] 17 | new_prediction: ['NULL'] 18 | evaluation: NTT 19 | sentence: Does it mean that not only could Donald Trump lose a lot of money by using Twitter ( by spooking investors ) but he could also get sued ? 20 | sentence type ['mutant'] 21 | entity: ['Twitter'] 22 | ori_prediction: ['MISC'] 23 | new_prediction: ['MISC'] 24 | evaluation: NTT 25 | sentence: The enormous free display forms part of the second exhibition at the Soho Photography Quarter . 26 | sentence type ['original'] 27 | entity: ['Soho Photography Quarter'] 28 | ori_prediction: ['LOC'] 29 | new_prediction: ['ORG'] 30 | evaluation: FT 31 | sentence: Did I start off really using Twitter as a bit of a diary ? 32 | sentence type ['mutant'] 33 | entity: ['Twitter'] 34 | ori_prediction: ['ORG'] 35 | new_prediction: ['MISC'] 36 | evaluation: FT 37 | sentence: Travellers have faced Hogmanay disruption on Scotland ' s railways after Friday ' s floods . 38 | sentence type ['original'] 39 | entity: ['Hogmanay'] 40 | ori_prediction: ['LOC'] 41 | new_prediction: ['MISC'] 42 | evaluation: FT 43 | sentence: The annual State of the UK Climate report said higher temperatures were the new normal and highlighted the ways that climate change was affecting the UK . 44 | sentence type ['original'] 45 | entity: ['State of the UK Climate'] 46 | ori_prediction: ['NULL'] 47 | new_prediction: ['NULL'] 48 | evaluation: NTT 49 | sentence: The annual State of the UK Climate report said higher temperatures were the new normal and highlighted the ways that climate change was affecting the UK . 50 | sentence type ['original'] 51 | entity: ['State of the UK Climate report'] 52 | ori_prediction: ['MISC'] 53 | new_prediction: ['MISC'] 54 | evaluation: NTT 55 | sentence: Did the same search in Russian result in the suggestions Wagner PMC , Wagner sledgehammer and Wagner orchestra ? 56 | sentence type ['mutant'] 57 | entity: ['Wagner sledgehammer'] 58 | ori_prediction: ['MISC'] 59 | new_prediction: ['MISC'] 60 | evaluation: NFF 61 | sentence: Did the same search in Russian result in the suggestions Wagner PMC , Wagner sledgehammer and Wagner orchestra ? 62 | sentence type ['mutant'] 63 | entity: ['Wagner PMC'] 64 | ori_prediction: ['MISC'] 65 | new_prediction: ['NULL'] 66 | evaluation: FF 67 | sentence: Did the same search in Russian result in the suggestions Wagner PMC , Wagner sledgehammer and Wagner orchestra ? 68 | sentence type ['mutant'] 69 | entity: ['Wagner'] 70 | ori_prediction: ['PER'] 71 | new_prediction: ['PER', 'PER'] 72 | evaluation: NFF 73 | evaluation: FF 74 | sentence: Both the Department for Environment , Food and Rural Affairs ( Defra ) and the British Poultry Council stress the issue is with free range turkeys , and that there are no issues with supplies of other birds . 75 | sentence type ['original'] 76 | entity: ['Department for Environment'] 77 | ori_prediction: ['NULL'] 78 | new_prediction: ['NULL'] 79 | evaluation: NTT 80 | sentence: Both the Department for Environment , Food and Rural Affairs ( Defra ) and the British Poultry Council stress the issue is with free range turkeys , and that there are no issues with supplies of other birds . 81 | sentence type ['original'] 82 | entity: ['Department for Environment , Food and Rural Affairs'] 83 | ori_prediction: ['ORG'] 84 | new_prediction: ['ORG'] 85 | evaluation: NTT 86 | sentence: Both the Department for Environment , Food and Rural Affairs ( Defra ) and the British Poultry Council stress the issue is with free range turkeys , and that there are no issues with supplies of other birds . 87 | sentence type ['original'] 88 | entity: ['Food and Rural Affairs'] 89 | ori_prediction: ['NULL'] 90 | new_prediction: ['NULL'] 91 | evaluation: NTT 92 | sentence: Does the M48 Severn Bridge remain closed in both directions between J1 and J2 due to high winds ? 93 | sentence type ['mutant'] 94 | entity: ['M48 Severn Bridge'] 95 | ori_prediction: ['NULL'] 96 | new_prediction: ['LOC'] 97 | evaluation: FT 98 | sentence: Does the M48 Severn Bridge remain closed in both directions between J1 and J2 due to high winds ? 99 | sentence type ['mutant'] 100 | entity: ['M48'] 101 | ori_prediction: ['LOC'] 102 | new_prediction: ['NULL'] 103 | evaluation: FT 104 | sentence: Does the M48 Severn Bridge remain closed in both directions between J1 and J2 due to high winds ? 105 | sentence type ['mutant'] 106 | entity: ['Severn Bridge'] 107 | ori_prediction: ['LOC'] 108 | new_prediction: ['NULL'] 109 | evaluation: FT 110 | sentence: Was Twitter the obvious solution ? 111 | sentence type ['mutant'] 112 | entity: ['Twitter'] 113 | ori_prediction: ['MISC'] 114 | new_prediction: ['NULL'] 115 | evaluation: TF 116 | sentence: The M48 Severn Bridge remains closed in both directions between J1 and J2 due to high winds . 117 | sentence type ['original'] 118 | entity: ['M48 Severn Bridge'] 119 | ori_prediction: ['LOC'] 120 | new_prediction: ['LOC'] 121 | evaluation: NTT 122 | sentence: The M48 Severn Bridge remains closed in both directions between J1 and J2 due to high winds . 123 | sentence type ['original'] 124 | entity: ['Severn Bridge'] 125 | ori_prediction: ['NULL'] 126 | new_prediction: ['NULL'] 127 | evaluation: NTT 128 | sentence: The M48 Severn Bridge remains closed in both directions between J1 and J2 due to high winds . 129 | sentence type ['original'] 130 | entity: ['M48'] 131 | ori_prediction: ['NULL'] 132 | new_prediction: ['NULL'] 133 | evaluation: NTT 134 | sentence: Twitter was the obvious solution . 135 | sentence type ['original'] 136 | entity: ['Twitter'] 137 | ori_prediction: ['ORG'] 138 | new_prediction: ['NULL'] 139 | evaluation: FF 140 | sentence: Were Christmas cards late due to a postal strike while holiday travel was hit by industrial action by both rail workers and airport staff ? 141 | sentence type ['mutant'] 142 | entity: ['Christmas'] 143 | ori_prediction: ['MISC'] 144 | new_prediction: ['MISC'] 145 | evaluation: NTT 146 | sentence: Did it say 142 iSpoof users were arrested across the world in November as part of a coordinated international operation ? 147 | sentence type ['mutant'] 148 | entity: ['iSpoof'] 149 | ori_prediction: ['MISC'] 150 | new_prediction: ['ORG'] 151 | evaluation: NTT 152 | sentence: Mr Louden , from Whauphill in Dumfries and Galloway , said he knew instantly after the accident that he was paralysed . 153 | sentence type ['original'] 154 | entity: ['Dumfries and Galloway'] 155 | ori_prediction: ['NULL'] 156 | new_prediction: ['LOC'] 157 | evaluation: FT 158 | sentence: Mr Louden , from Whauphill in Dumfries and Galloway , said he knew instantly after the accident that he was paralysed . 159 | sentence type ['original'] 160 | entity: ['Dumfries'] 161 | ori_prediction: ['LOC'] 162 | new_prediction: ['NULL'] 163 | evaluation: FT 164 | sentence: Mr Louden , from Whauphill in Dumfries and Galloway , said he knew instantly after the accident that he was paralysed . 165 | sentence type ['original'] 166 | entity: ['Galloway'] 167 | ori_prediction: ['LOC'] 168 | new_prediction: ['NULL'] 169 | evaluation: FT 170 | sentence: Is the funding coming from the Advanced Propulsion Centre ( APC ) programme ? 171 | sentence type ['mutant'] 172 | entity: ['Advanced Propulsion Centre'] 173 | ori_prediction: ['MISC'] 174 | new_prediction: ['MISC'] 175 | evaluation: NFF 176 | sentence: Is the funding coming from the Advanced Propulsion Centre ( APC ) programme ? 177 | sentence type ['mutant'] 178 | entity: ['APC'] 179 | ori_prediction: ['MISC'] 180 | new_prediction: ['MISC'] 181 | evaluation: NFF 182 | sentence: Mark Harding , one of Oban ' s owners , said he was elated to find him alive . 183 | sentence type ['original'] 184 | entity: ['Oban'] 185 | ori_prediction: ['ORG'] 186 | new_prediction: ['ORG'] 187 | evaluation: NTT 188 | sentence: Mr Clay died on his birthday . 189 | sentence type ['original'] 190 | entity: ['Mr'] 191 | ori_prediction: ['NULL'] 192 | new_prediction: ['NULL'] 193 | evaluation: NTT 194 | sentence: The reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , forms part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 . 195 | sentence type ['original'] 196 | entity: ['Special Scientific Interest'] 197 | ori_prediction: ['MISC'] 198 | new_prediction: ['NULL'] 199 | evaluation: FF 200 | sentence: The reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , forms part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 . 201 | sentence type ['original'] 202 | entity: ['of Special Scientific Interest'] 203 | ori_prediction: ['NULL'] 204 | new_prediction: ['NULL'] 205 | evaluation: NTT 206 | sentence: The reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , forms part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 . 207 | sentence type ['original'] 208 | entity: ['of'] 209 | ori_prediction: ['ORG'] 210 | new_prediction: ['ORG', 'ORG'] 211 | evaluation: NFF 212 | evaluation: TF 213 | sentence: Did Det Supt Rance warn other criminal enablers will have taken over to provide services to fraudsters ? 214 | sentence type ['mutant'] 215 | entity: ['Supt Rance'] 216 | ori_prediction: ['NULL'] 217 | new_prediction: ['NULL'] 218 | evaluation: NTT 219 | sentence: Did Det Supt Rance warn other criminal enablers will have taken over to provide services to fraudsters ? 220 | sentence type ['mutant'] 221 | entity: ['Det Supt Rance'] 222 | ori_prediction: ['PER'] 223 | new_prediction: ['PER'] 224 | evaluation: NTT 225 | sentence: Does the enormous free display form part of the second exhibition at the Soho Photography Quarter ? 226 | sentence type ['mutant'] 227 | entity: ['Soho Photography Quarter'] 228 | ori_prediction: ['ORG'] 229 | new_prediction: ['ORG'] 230 | evaluation: NTT 231 | sentence: It means that not only could Donald Trump lose a lot of money by using Twitter ( by spooking investors ) but he could also get sued . 232 | sentence type ['original'] 233 | entity: ['Twitter'] 234 | ori_prediction: ['ORG'] 235 | new_prediction: ['ORG'] 236 | evaluation: NFF 237 | sentence: Did Mr Louden , from Whauphill in Dumfries and Galloway , say he knew instantly after the accident that he was paralysed ? 238 | sentence type ['mutant'] 239 | entity: ['Dumfries and Galloway'] 240 | ori_prediction: ['LOC'] 241 | new_prediction: ['LOC'] 242 | evaluation: NTT 243 | sentence: Did Mr Louden , from Whauphill in Dumfries and Galloway , say he knew instantly after the accident that he was paralysed ? 244 | sentence type ['mutant'] 245 | entity: ['Dumfries'] 246 | ori_prediction: ['NULL'] 247 | new_prediction: ['NULL'] 248 | evaluation: NTT 249 | sentence: Did Mr Louden , from Whauphill in Dumfries and Galloway , say he knew instantly after the accident that he was paralysed ? 250 | sentence type ['mutant'] 251 | entity: ['Galloway'] 252 | ori_prediction: ['NULL'] 253 | new_prediction: ['NULL'] 254 | evaluation: NTT 255 | sentence: The same search in Russian resulted in the suggestions Wagner PMC , Wagner sledgehammer and Wagner orchestra . 256 | sentence type ['original'] 257 | entity: ['Wagner sledgehammer'] 258 | ori_prediction: ['NULL'] 259 | new_prediction: ['ORG'] 260 | evaluation: NTT 261 | sentence: The same search in Russian resulted in the suggestions Wagner PMC , Wagner sledgehammer and Wagner orchestra . 262 | sentence type ['original'] 263 | entity: ['Wagner PMC'] 264 | ori_prediction: ['ORG'] 265 | new_prediction: ['NULL'] 266 | evaluation: TF 267 | sentence: The same search in Russian resulted in the suggestions Wagner PMC , Wagner sledgehammer and Wagner orchestra . 268 | sentence type ['original'] 269 | entity: ['Wagner'] 270 | ori_prediction: ['ORG', 'PER'] 271 | new_prediction: ['ORG', 'ORG'] 272 | evaluation: NTT 273 | evaluation: FT 274 | sentence: Did Apple ' s audience lean more towards pop , charting songs by Adele and Elton John that did n't appear in Spotify ' s list ? 275 | sentence type ['mutant'] 276 | entity: ['Adele'] 277 | ori_prediction: ['PER'] 278 | new_prediction: ['PER'] 279 | evaluation: NTT 280 | sentence: The median sum in the US for a funeral with a burial was $ 7 , 848 in 2021 , or $ 6 , 971 for a funeral with a cremation , according to the National Funeral Directors Association ( NFDA ) . 281 | sentence type ['original'] 282 | entity: ['848'] 283 | ori_prediction: ['NULL'] 284 | new_prediction: ['NULL'] 285 | evaluation: NTT 286 | sentence: Ben Johnson , from the Environmental Services Association ( ESA ) , told BBC News more and more people were putting devices containing these batteries in with household rubbish or mixing them with other recycling . 287 | sentence type ['original'] 288 | entity: ['ESA'] 289 | ori_prediction: ['NULL'] 290 | new_prediction: ['ORG'] 291 | evaluation: NTT 292 | sentence: Has The US banned the sale and import of new communications equipment from five Chinese companies , including Huawei and ZTE , amid concerns over national security ? 293 | sentence type ['mutant'] 294 | entity: ['The'] 295 | ori_prediction: ['NULL'] 296 | new_prediction: ['NULL'] 297 | evaluation: NTT 298 | sentence: Has The US banned the sale and import of new communications equipment from five Chinese companies , including Huawei and ZTE , amid concerns over national security ? 299 | sentence type ['mutant'] 300 | entity: ['US'] 301 | ori_prediction: ['NULL'] 302 | new_prediction: ['NULL'] 303 | evaluation: NTT 304 | sentence: Has The US banned the sale and import of new communications equipment from five Chinese companies , including Huawei and ZTE , amid concerns over national security ? 305 | sentence type ['mutant'] 306 | entity: ['The US'] 307 | ori_prediction: ['LOC'] 308 | new_prediction: ['ORG'] 309 | evaluation: TF 310 | sentence: Did Mr Coughlan say his company will make the NuCell that will generate power for the National Grid ? 311 | sentence type ['mutant'] 312 | entity: ['National Grid'] 313 | ori_prediction: ['MISC'] 314 | new_prediction: ['MISC'] 315 | evaluation: NFF 316 | sentence: Did Mr Coughlan say his company will make the NuCell that will generate power for the National Grid ? 317 | sentence type ['mutant'] 318 | entity: ['NuCell'] 319 | ori_prediction: ['NULL'] 320 | new_prediction: ['MISC'] 321 | evaluation: FT 322 | sentence: The US and other Western countries have made improvements to female education in Afghanistan a prior condition for the formal recognition of the Taliban government . 323 | sentence type ['original'] 324 | entity: ['Taliban'] 325 | ori_prediction: ['MISC'] 326 | new_prediction: ['MISC'] 327 | evaluation: NFF 328 | sentence: Have travellers faced Hogmanay disruption on Scotland ' s railways after Friday ' s floods ? 329 | sentence type ['mutant'] 330 | entity: ['Hogmanay'] 331 | ori_prediction: ['MISC'] 332 | new_prediction: ['MISC'] 333 | evaluation: NFF 334 | sentence: The US has banned the sale and import of new communications equipment from five Chinese companies , including Huawei and ZTE , amid concerns over national security . 335 | sentence type ['original'] 336 | entity: ['The'] 337 | ori_prediction: ['LOC'] 338 | new_prediction: ['LOC'] 339 | evaluation: NFF 340 | sentence: The US has banned the sale and import of new communications equipment from five Chinese companies , including Huawei and ZTE , amid concerns over national security . 341 | sentence type ['original'] 342 | entity: ['US'] 343 | ori_prediction: ['LOC'] 344 | new_prediction: ['ORG'] 345 | evaluation: FF 346 | sentence: The US has banned the sale and import of new communications equipment from five Chinese companies , including Huawei and ZTE , amid concerns over national security . 347 | sentence type ['original'] 348 | entity: ['The US'] 349 | ori_prediction: ['NULL'] 350 | new_prediction: ['NULL'] 351 | evaluation: NFF 352 | sentence: Ms Clark wants Nexperia to stay . 353 | sentence type ['original'] 354 | entity: ['Ms Clark'] 355 | ori_prediction: ['NULL'] 356 | new_prediction: ['PER'] 357 | evaluation: FT 358 | sentence: Ms Clark wants Nexperia to stay . 359 | sentence type ['original'] 360 | entity: ['Nexperia'] 361 | ori_prediction: ['ORG'] 362 | new_prediction: ['ORG'] 363 | evaluation: NTT 364 | sentence: Ms Clark wants Nexperia to stay . 365 | sentence type ['original'] 366 | entity: ['Clark'] 367 | ori_prediction: ['PER'] 368 | new_prediction: ['NULL'] 369 | evaluation: FT 370 | sentence: Christmas cards were late due to a postal strike while holiday travel was hit by industrial action by both rail workers and airport staff . 371 | sentence type ['original'] 372 | entity: ['Christmas'] 373 | ori_prediction: ['NULL'] 374 | new_prediction: ['NULL'] 375 | evaluation: NFF 376 | sentence: A report found the Dippy effect saw visitors increase everywhere it toured , including Birmingham , Cardiff and Glasgow . 377 | sentence type ['original'] 378 | entity: ['Dippy'] 379 | ori_prediction: ['MISC'] 380 | new_prediction: ['MISC'] 381 | evaluation: NTT 382 | sentence: The Urdd , Wales Today and Aberystwyth University all celebrated significant anniversaries while we said goodbye to two heroes of Welsh rugby and one of our most beloved actresses . 383 | sentence type ['original'] 384 | entity: ['The Urdd'] 385 | ori_prediction: ['NULL'] 386 | new_prediction: ['ORG'] 387 | evaluation: FT 388 | sentence: The Urdd , Wales Today and Aberystwyth University all celebrated significant anniversaries while we said goodbye to two heroes of Welsh rugby and one of our most beloved actresses . 389 | sentence type ['original'] 390 | entity: ['The'] 391 | ori_prediction: ['ORG'] 392 | new_prediction: ['NULL'] 393 | evaluation: FT 394 | sentence: The Urdd , Wales Today and Aberystwyth University all celebrated significant anniversaries while we said goodbye to two heroes of Welsh rugby and one of our most beloved actresses . 395 | sentence type ['original'] 396 | entity: ['Urdd'] 397 | ori_prediction: ['ORG'] 398 | new_prediction: ['NULL'] 399 | evaluation: FT 400 | sentence: SpaceX plans to take Japanese billionaire Yusaku Maezawa and eight other passengers on the dearMoon voyage around the Moon in late 2023 . 401 | sentence type ['original'] 402 | entity: ['Moon'] 403 | ori_prediction: ['LOC'] 404 | new_prediction: ['LOC'] 405 | evaluation: NFF 406 | sentence: Did our Community Guidelines clearly outline that we do not allow people to use our platform to threaten or incite violence , or share attacks or slurs based on people ' s nationality or other protected characteristics ? 407 | sentence type ['mutant'] 408 | entity: ['Guidelines'] 409 | ori_prediction: ['MISC'] 410 | new_prediction: ['MISC'] 411 | evaluation: NFF 412 | sentence: Mr Coughlan said his company will make the NuCell that will generate power for the National Grid . 413 | sentence type ['original'] 414 | entity: ['National Grid'] 415 | ori_prediction: ['ORG'] 416 | new_prediction: ['ORG'] 417 | evaluation: NTT 418 | sentence: Mr Coughlan said his company will make the NuCell that will generate power for the National Grid . 419 | sentence type ['original'] 420 | entity: ['NuCell'] 421 | ori_prediction: ['MISC'] 422 | new_prediction: ['MISC'] 423 | evaluation: NTT 424 | sentence: Does SpaceX plan to take Japanese billionaire Yusaku Maezawa and eight other passengers on the dearMoon voyage around the Moon in late 2023 ? 425 | sentence type ['mutant'] 426 | entity: ['Moon'] 427 | ori_prediction: ['MISC'] 428 | new_prediction: ['MISC'] 429 | evaluation: NTT 430 | sentence: The funding is coming from the Advanced Propulsion Centre ( APC ) programme . 431 | sentence type ['original'] 432 | entity: ['Advanced Propulsion Centre'] 433 | ori_prediction: ['ORG'] 434 | new_prediction: ['ORG'] 435 | evaluation: NTT 436 | sentence: The funding is coming from the Advanced Propulsion Centre ( APC ) programme . 437 | sentence type ['original'] 438 | entity: ['APC'] 439 | ori_prediction: ['ORG'] 440 | new_prediction: ['MISC'] 441 | evaluation: TF 442 | sentence: Did Mr Clay die on his birthday ? 443 | sentence type ['mutant'] 444 | entity: ['Mr'] 445 | ori_prediction: ['PER'] 446 | new_prediction: ['NULL'] 447 | evaluation: FT 448 | sentence: Was the median sum in the US for a funeral with a burial $ 7 , 848 in 2021 , or $ 6 , 971 for a funeral with a cremation , according to the National Funeral Directors Association ( NFDA ) ? 449 | sentence type ['mutant'] 450 | entity: ['848'] 451 | ori_prediction: ['ORG'] 452 | new_prediction: ['NULL'] 453 | evaluation: FT 454 | sentence: Did Historic England , which is providing £ 10 , 000 , say Vindolanda , close to Hadrian ' s Wall , was the most exciting archaeological site in Europe ? 455 | sentence type ['mutant'] 456 | entity: ['Historic England'] 457 | ori_prediction: ['LOC'] 458 | new_prediction: ['LOC'] 459 | evaluation: NTT 460 | sentence: Did Historic England , which is providing £ 10 , 000 , say Vindolanda , close to Hadrian ' s Wall , was the most exciting archaeological site in Europe ? 461 | sentence type ['mutant'] 462 | entity: ['England'] 463 | ori_prediction: ['NULL'] 464 | new_prediction: ['NULL'] 465 | evaluation: NTT 466 | sentence: Did Det Supt Rance say the investigation remained active ? 467 | sentence type ['mutant'] 468 | entity: ['Rance'] 469 | ori_prediction: ['NULL'] 470 | new_prediction: ['PER'] 471 | evaluation: TF 472 | sentence: Did Det Supt Rance say the investigation remained active ? 473 | sentence type ['mutant'] 474 | entity: ['Det Supt Rance'] 475 | ori_prediction: ['PER'] 476 | new_prediction: ['NULL'] 477 | evaluation: FT 478 | sentence: The operation ran across many internet services , including Twitter , YouTube , Telegram , VKontakte and Odnoklassniki , according to Meta . 479 | sentence type ['original'] 480 | entity: ['Odnoklassniki'] 481 | ori_prediction: ['MISC'] 482 | new_prediction: ['ORG'] 483 | evaluation: FT 484 | sentence: Does the reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , form part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 ? 485 | sentence type ['mutant'] 486 | entity: ['Special Scientific Interest'] 487 | ori_prediction: ['NULL'] 488 | new_prediction: ['NULL'] 489 | evaluation: FF 490 | sentence: Does the reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , form part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 ? 491 | sentence type ['mutant'] 492 | entity: ['of Special Scientific Interest'] 493 | ori_prediction: ['MISC'] 494 | new_prediction: ['NULL'] 495 | evaluation: FT 496 | sentence: Does the reserve , which is owned by the Department of the Environment , Food and Agriculture and Manx National Heritage , form part of a wider Area of Special Scientific Interest ( ASSI ) established in 1996 ? 497 | sentence type ['mutant'] 498 | entity: ['of'] 499 | ori_prediction: ['NULL'] 500 | new_prediction: ['NULL'] 501 | evaluation: NTT 502 | sentence: Does the feud come as many companies have halted spending on Twitter amid concerns about Mr Musk ' s content moderation plans for the site ? 503 | sentence type ['mutant'] 504 | entity: ['Twitter'] 505 | ori_prediction: ['MISC'] 506 | new_prediction: ['MISC'] 507 | evaluation: NTT 508 | sentence: Did the operation run across many internet services , including Twitter , YouTube , Telegram , VKontakte and Odnoklassniki , according to Meta ? 509 | sentence type ['mutant'] 510 | entity: ['Odnoklassniki'] 511 | ori_prediction: ['ORG'] 512 | new_prediction: ['ORG'] 513 | evaluation: NTT 514 | sentence: I started off really using Twitter as a bit of a diary . 515 | sentence type ['original'] 516 | entity: ['Twitter'] 517 | ori_prediction: ['MISC'] 518 | new_prediction: ['MISC'] 519 | evaluation: NTT 520 | sentence: Apple supplier Foxconn is ramping up efforts to recruit workers after unrest at the world ' s biggest iPhone factory . 521 | sentence type ['original'] 522 | entity: ['Apple'] 523 | ori_prediction: ['ORG'] 524 | new_prediction: ['ORG'] 525 | evaluation: NTT 526 | sentence: Did both the Department for Environment , Food and Rural Affairs ( Defra ) and the British Poultry Council stress the issue is with free range turkeys , and that there are no issues with supplies of other birds ? 527 | sentence type ['mutant'] 528 | entity: ['Department for Environment'] 529 | ori_prediction: ['ORG'] 530 | new_prediction: ['NULL'] 531 | evaluation: FT 532 | sentence: Did both the Department for Environment , Food and Rural Affairs ( Defra ) and the British Poultry Council stress the issue is with free range turkeys , and that there are no issues with supplies of other birds ? 533 | sentence type ['mutant'] 534 | entity: ['Department for Environment , Food and Rural Affairs'] 535 | ori_prediction: ['NULL'] 536 | new_prediction: ['NULL'] 537 | evaluation: NFF 538 | sentence: Did both the Department for Environment , Food and Rural Affairs ( Defra ) and the British Poultry Council stress the issue is with free range turkeys , and that there are no issues with supplies of other birds ? 539 | sentence type ['mutant'] 540 | entity: ['Food and Rural Affairs'] 541 | ori_prediction: ['ORG'] 542 | new_prediction: ['ORG'] 543 | evaluation: NFF 544 | sentence: Did Ben Johnson , from the Environmental Services Association ( ESA ) , tell BBC News more and more people were putting devices containing these batteries in with household rubbish or mixing them with other recycling ? 545 | sentence type ['mutant'] 546 | entity: ['ESA'] 547 | ori_prediction: ['ORG'] 548 | new_prediction: ['ORG'] 549 | evaluation: NTT 550 | sentence: The delegation was hosted by Newport West Labour MP Ruth Jones , who said she would be seeking a meeting with the business secretary as soon as possible . 551 | sentence type ['original'] 552 | entity: ['Newport West Labour'] 553 | ori_prediction: ['NULL'] 554 | new_prediction: ['ORG'] 555 | evaluation: FT 556 | sentence: The delegation was hosted by Newport West Labour MP Ruth Jones , who said she would be seeking a meeting with the business secretary as soon as possible . 557 | sentence type ['original'] 558 | entity: ['Newport'] 559 | ori_prediction: ['LOC'] 560 | new_prediction: ['NULL'] 561 | evaluation: FT 562 | sentence: The delegation was hosted by Newport West Labour MP Ruth Jones , who said she would be seeking a meeting with the business secretary as soon as possible . 563 | sentence type ['original'] 564 | entity: ['West Labour'] 565 | ori_prediction: ['ORG'] 566 | new_prediction: ['NULL'] 567 | evaluation: FT 568 | sentence: Did Senator Maria Cantwell , the chair of the US Senate Commerce Committee , say that the committee will be looking into the causes of these disruptions and its impact to consumers ? 569 | sentence type ['mutant'] 570 | entity: ['US'] 571 | ori_prediction: ['NULL'] 572 | new_prediction: ['NULL'] 573 | evaluation: NTT 574 | sentence: Did Senator Maria Cantwell , the chair of the US Senate Commerce Committee , say that the committee will be looking into the causes of these disruptions and its impact to consumers ? 575 | sentence type ['mutant'] 576 | entity: ['Senate Commerce Committee'] 577 | ori_prediction: ['NULL'] 578 | new_prediction: ['NULL'] 579 | evaluation: NTT 580 | sentence: Did Senator Maria Cantwell , the chair of the US Senate Commerce Committee , say that the committee will be looking into the causes of these disruptions and its impact to consumers ? 581 | sentence type ['mutant'] 582 | entity: ['US Senate Commerce Committee'] 583 | ori_prediction: ['ORG'] 584 | new_prediction: ['ORG'] 585 | evaluation: NTT 586 | sentence: Is Apple supplier Foxconn ramping up efforts to recruit workers after unrest at the world ' s biggest iPhone factory ? 587 | sentence type ['mutant'] 588 | entity: ['Apple'] 589 | ori_prediction: ['MISC'] 590 | new_prediction: ['MISC'] 591 | evaluation: NTT 592 | sentence: Historic England , which is providing £ 10 , 000 , said Vindolanda , close to Hadrian ' s Wall , was the most exciting archaeological site in Europe . 593 | sentence type ['original'] 594 | entity: ['Historic England'] 595 | ori_prediction: ['NULL'] 596 | new_prediction: ['ORG'] 597 | evaluation: FT 598 | sentence: Historic England , which is providing £ 10 , 000 , said Vindolanda , close to Hadrian ' s Wall , was the most exciting archaeological site in Europe . 599 | sentence type ['original'] 600 | entity: ['England'] 601 | ori_prediction: ['LOC'] 602 | new_prediction: ['NULL'] 603 | evaluation: FT 604 | sentence: Apple ' s audience leaned more towards pop , charting songs by Adele and Elton John that did n't appear in Spotify ' s list . 605 | sentence type ['original'] 606 | entity: ['Adele'] 607 | ori_prediction: ['ORG'] 608 | new_prediction: ['PER'] 609 | evaluation: FT 610 | sentence: Have the US and other Western countries made improvements to female education in Afghanistan a prior condition for the formal recognition of the Taliban government ? 611 | sentence type ['mutant'] 612 | entity: ['Taliban'] 613 | ori_prediction: ['ORG'] 614 | new_prediction: ['MISC'] 615 | evaluation: TF 616 | sentence: Did Mark Harding , one of Oban ' s owners , say he was elated to find him alive ? 617 | sentence type ['mutant'] 618 | entity: ['Oban'] 619 | ori_prediction: ['PER'] 620 | new_prediction: ['PER'] 621 | evaluation: NFF 622 | sentence: Did a report find the Dippy effect saw visitors increase everywhere it toured , including Birmingham , Cardiff and Glasgow ? 623 | sentence type ['mutant'] 624 | entity: ['Dippy'] 625 | ori_prediction: ['PER'] 626 | new_prediction: ['MISC'] 627 | evaluation: FT 628 | sentence: Did the annual State of the UK Climate report say higher temperatures were the new normal and highlighted the ways that climate change was affecting the UK ? 629 | sentence type ['mutant'] 630 | entity: ['State of the UK Climate'] 631 | ori_prediction: ['MISC'] 632 | new_prediction: ['MISC'] 633 | evaluation: NFF 634 | sentence: Did the annual State of the UK Climate report say higher temperatures were the new normal and highlighted the ways that climate change was affecting the UK ? 635 | sentence type ['mutant'] 636 | entity: ['State of the UK Climate report'] 637 | ori_prediction: ['NULL'] 638 | new_prediction: ['NULL'] 639 | evaluation: NFF 640 | sentence: Senator Maria Cantwell , the chair of the US Senate Commerce Committee , said that the committee will be looking into the causes of these disruptions and its impact to consumers . 641 | sentence type ['original'] 642 | entity: ['US'] 643 | ori_prediction: ['ORG'] 644 | new_prediction: ['NULL'] 645 | evaluation: FT 646 | sentence: Senator Maria Cantwell , the chair of the US Senate Commerce Committee , said that the committee will be looking into the causes of these disruptions and its impact to consumers . 647 | sentence type ['original'] 648 | entity: ['Senate Commerce Committee'] 649 | ori_prediction: ['ORG'] 650 | new_prediction: ['NULL'] 651 | evaluation: FT 652 | sentence: Senator Maria Cantwell , the chair of the US Senate Commerce Committee , said that the committee will be looking into the causes of these disruptions and its impact to consumers . 653 | sentence type ['original'] 654 | entity: ['US Senate Commerce Committee'] 655 | ori_prediction: ['NULL'] 656 | new_prediction: ['ORG'] 657 | evaluation: FT 658 | sentence: Det Supt Rance warned other criminal enablers will have taken over to provide services to fraudsters . 659 | sentence type ['original'] 660 | entity: ['Supt Rance'] 661 | ori_prediction: ['PER'] 662 | new_prediction: ['PER'] 663 | evaluation: NFF 664 | sentence: Det Supt Rance warned other criminal enablers will have taken over to provide services to fraudsters . 665 | sentence type ['original'] 666 | entity: ['Det Supt Rance'] 667 | ori_prediction: ['NULL'] 668 | new_prediction: ['NULL'] 669 | evaluation: NFF 670 | sentence: It said 142 iSpoof users were arrested across the world in November as part of a coordinated international operation . 671 | sentence type ['original'] 672 | entity: ['iSpoof'] 673 | ori_prediction: ['ORG'] 674 | new_prediction: ['ORG'] 675 | evaluation: NTT 676 | sentence: Does Ms Clark want Nexperia to stay ? 677 | sentence type ['mutant'] 678 | entity: ['Ms Clark'] 679 | ori_prediction: ['PER'] 680 | new_prediction: ['PER'] 681 | evaluation: NTT 682 | sentence: Does Ms Clark want Nexperia to stay ? 683 | sentence type ['mutant'] 684 | entity: ['Nexperia'] 685 | ori_prediction: ['PER'] 686 | new_prediction: ['ORG'] 687 | evaluation: FT 688 | sentence: Does Ms Clark want Nexperia to stay ? 689 | sentence type ['mutant'] 690 | entity: ['Clark'] 691 | ori_prediction: ['NULL'] 692 | new_prediction: ['NULL'] 693 | evaluation: NTT 694 | sentence: Our Community Guidelines clearly outline that we do not allow people to use our platform to threaten or incite violence , or share attacks or slurs based on people ' s nationality or other protected characteristics . 695 | sentence type ['original'] 696 | entity: ['Guidelines'] 697 | ori_prediction: ['NULL'] 698 | new_prediction: ['NULL'] 699 | evaluation: NTT 700 | sentence: Was the delegation hosted by Newport West Labour MP Ruth Jones , who said she would be seeking a meeting with the business secretary as soon as possible ? 701 | sentence type ['mutant'] 702 | entity: ['Newport'] 703 | ori_prediction: ['NULL'] 704 | new_prediction: ['NULL'] 705 | evaluation: NTT 706 | sentence: Was the delegation hosted by Newport West Labour MP Ruth Jones , who said she would be seeking a meeting with the business secretary as soon as possible ? 707 | sentence type ['mutant'] 708 | entity: ['West Labour'] 709 | ori_prediction: ['NULL'] 710 | new_prediction: ['NULL'] 711 | evaluation: NTT 712 | sentence: Was the delegation hosted by Newport West Labour MP Ruth Jones , who said she would be seeking a meeting with the business secretary as soon as possible ? 713 | sentence type ['mutant'] 714 | entity: ['Newport West Labour'] 715 | ori_prediction: ['ORG'] 716 | new_prediction: ['ORG'] 717 | evaluation: NTT 718 | sentence: The feud comes as many companies have halted spending on Twitter amid concerns about Mr Musk ' s content moderation plans for the site . 719 | sentence type ['original'] 720 | entity: ['Twitter'] 721 | ori_prediction: ['ORG'] 722 | new_prediction: ['ORG'] 723 | evaluation: NTT 724 | sentence: Det Supt Rance said the investigation remained active . 725 | sentence type ['original'] 726 | entity: ['Rance'] 727 | ori_prediction: ['PER'] 728 | new_prediction: ['PER'] 729 | evaluation: NFF 730 | sentence: Det Supt Rance said the investigation remained active . 731 | sentence type ['original'] 732 | entity: ['Det Supt Rance'] 733 | ori_prediction: ['NULL'] 734 | new_prediction: ['NULL'] 735 | evaluation: NFF 736 | -------------------------------------------------------------------------------- /TIN_Test/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobustNLP/TestNER/ced6f86d8cdfb1c16fbf7b968481051fc0f83f4e/TIN_Test/.DS_Store -------------------------------------------------------------------------------- /TIN_Test/Data/transer.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | 4 | class Transer: 5 | # txt file to json file (pre-process the sentences) 6 | def txt_preprocess(self, file_path, output_path): 7 | result = [] 8 | with open(file_path, "r", encoding = 'utf-8') as f: 9 | for line in f: 10 | if "-" in line: 11 | continue 12 | # deal with separating signs 13 | line = line.strip('\n') 14 | line = line.replace('\0', ' ') 15 | line = line.replace("\"", ' ') 16 | # deal with "'s" issues 17 | line = line.replace("'s", " 's") 18 | line = line.replace(" 's", " 's") 19 | # deal with "'m" 20 | line = line.replace("'m", " 'm") 21 | line = line.replace(" 'm", " 'm") 22 | # deal with "'re" 23 | line = line.replace("'re", " 're") 24 | line = line.replace(" 're", " 're") 25 | # deal with "()" issues 26 | line = line.replace("(", "( ") 27 | line = line.replace(")", " )") 28 | line = line.replace("( ", "( ") 29 | line = line.replace(" )", " )") 30 | # deal with "n't" 31 | line = line.replace("n't" , " n't") 32 | line = line.replace(" n't" , " n't") 33 | sentence = [] 34 | text = "" 35 | for index, c in enumerate(line): 36 | if index > 0 and index < len(line)-1 and c == "'": 37 | if line[index-1] != " " and line[index+1] == " ": 38 | sentence.append(text) 39 | text = "" 40 | sentence.append(c) 41 | else: 42 | text = text + c 43 | elif c == " " or c == "." or c == '"' or c == "," or c == ".": 44 | if(text != ""): sentence.append(text) 45 | text = "" 46 | if(c != " "): sentence.append(c) 47 | else: text = text + c 48 | if(text != ""): sentence.append(text) 49 | result.append(" ".join(sentence)) 50 | print(len(result)) 51 | with open(output_path, "w", encoding="utf-8") as f: 52 | json.dump(result, f, ensure_ascii=False) 53 | 54 | if __name__ == "__main__": 55 | transer = Transer() 56 | for file in os.listdir('.'): 57 | print(file) 58 | if file.endswith(".txt"): transer.txt_preprocess(file_path=file, output_path=file[:-4]+".json") 59 | -------------------------------------------------------------------------------- /TIN_Test/Example_results/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobustNLP/TestNER/ced6f86d8cdfb1c16fbf7b968481051fc0f83f4e/TIN_Test/Example_results/.DS_Store -------------------------------------------------------------------------------- /TIN_Test/README.md: -------------------------------------------------------------------------------- 1 | # TIN 2 | 3 | A toolkit for testing and repairing NER systems 4 | 5 | 6 | 7 | ## Environment requirements 8 | 9 | * nltk 10 | * stanfordcorenlp 11 | * transformers 12 | 13 | ## Related packages 14 | 15 | We use some packages to implement the toolkit: 16 | 17 | * Stanford CoreNLP (https://stanfordnlp.github.io/CoreNLP/) 18 | * flair (https://github.com/flairNLP/flair) 19 | * Microsoft Azure: Named Entity Recognition (https://learn.microsoft.com/en-us/azure/cognitive-services/language-service/named-entity-recognition/overview) 20 | * Amazon Segamaker: Named Entity Recognition (https://docs.aws.amazon.com/sagemaker/latest/dg/sms-named-entity-recg.html) 21 | * Sense2Vec (https://github.com/explosion/sense2vec) 22 | * AEON (https://github.com/CUHK-ARISE/AEON) 23 | 24 | Specifically, you need to download some of the packages to the `{ROOT_DIR}/Data` directory, the packages includes Stanford CoreNLP and Sense2Vec. 25 | 26 | For corenlp, you need to download corenlp (https://stanfordnlp.github.io/CoreNLP/) add the whole `corenlp` folder to the `{ROOT_DIR}`. 27 | 28 | For Sense2vec, you need to download the s2v_reddit_2015_md package (s2v_reddit_2015_md) and add the `s2v_old` folder to `{ROOT_DIR}`. 29 | 30 | The final `{ROOT_DIR}` directory contains: 31 | ```shell 32 | AEON 33 | apis.py 34 | corenlp 35 | Data 36 | Example_results 37 | json2text.py 38 | mr_perturb.py 39 | __pycache__ 40 | s2v_old 41 | Transformations 42 | ``` 43 | 44 | ## Work Pipeline Example 45 | 46 | (Optional) Before you run **Tin**, you should preprocess your dataset. You need to put your dataset in the `${REPO_ROOT_DIR}/Data` directory. Then you need to run `transer.py` to preprocess the data. Note: your dataset should be in txt format where each line contains one sentence of the string format. 47 | 48 | ``` 49 | cd ${REPO_ROOT_DIR}/Data 50 | python transer.py 51 | ``` 52 | 53 | **Tin** includes 2 parts: 1. the testing part, 2. the repair part. You should first run the testing part, then you can run the repair part based on the result of the testing part. The following part demonstrates a typical work flow 54 | 55 | 1. Testing: generate suspicious issues 56 | 57 | ```shell 58 | cd ${REPO_ROOT_DIR} 59 | python mr_perturb.py 60 | python json2text.py # convert json to txt to check the suspicious issues 61 | ``` 62 | 63 | 2. Repair: locate suspicious entities and repair 64 | 65 | 66 | 67 | ## Testing and generate suspicious issues 68 | 69 | To run the testing step, you should run the `mr_perturb.py` in the root directory. You can adjust the following arguments to run different settings: 70 | 71 | * aeon_flag (bool): set whether to use naturalness filter 72 | 73 | > **True**: use naturalness filter 74 | > 75 | > **False**: do not use naturalness filter 76 | > 77 | > By default aeon_flag is set to **True** 78 | 79 | * api_type (string): set which api or model to test 80 | 81 | > Specifically, you can use the following api types: 82 | > 83 | > **flair_conll**: test on flair's pretrained model of conll03 dataset 84 | > 85 | > **flair_ontonotes**: test on flair's pretrained model of ontonotes5 dataset 86 | > 87 | > **azure**: test on Microsoft Azure NER api 88 | > 89 | > **aws**: test on AWS Segamaker NER api 90 | > 91 | > By default the api_type is set to **flair_conll** 92 | 93 | * dataset_path (string): set the path of your dataset 94 | 95 | > Set the path of your data (which is in the json format) 96 | > 97 | > By default, **dataset_path** = ${REPO_ROOT_DIR}/Data/bbc.json 98 | 99 | * down_size (bool): set whether you only want to test 100 cases 100 | 101 | > When the dataset can be large, the program runs for a long time. 102 | > 103 | > You can set down_size to **True** if you want to run on only 100 cases 104 | 105 | One example of running `mr_perturb.py` 106 | 107 | ```shell 108 | cd ${REPO_ROOT_DIR} 109 | python mr_perturb.py --aeon_flag=True --api_type=azure --dataset_path=./Data/bbc.json --down_size=False 110 | python json2text.py 111 | ``` 112 | 113 | ## Example results 114 | We provide the example results including transformation results and suspicious issues, the example results are in the `{ROOT_DIR}/Example_results` folder. -------------------------------------------------------------------------------- /TIN_Test/Transformations/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobustNLP/TestNER/ced6f86d8cdfb1c16fbf7b968481051fc0f83f4e/TIN_Test/Transformations/.DS_Store -------------------------------------------------------------------------------- /TIN_Test/Transformations/MR_BERT.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import ipdb 4 | import nltk 5 | import torch 6 | import numpy as np 7 | from copy import deepcopy 8 | from transformers import BertConfig, BertTokenizer, BertModel, RobertaTokenizer, RobertaModel, BertForMaskedLM 9 | from collections import defaultdict 10 | import itertools 11 | import random 12 | import sys 13 | # import the AEON, please change the follosing path with the path ofaeon.py 14 | sys.path.insert(1, '$HOME$/NER_MRs/MRs/AEON/utils') 15 | from aeon import * 16 | 17 | # I use some of Joye;s codes 18 | class MR_BERT: 19 | def MR_init(self, aeon_flag = True, evaluate = False, aeon_threshold = 0.02): 20 | self.berttokenizer = BertTokenizer.from_pretrained('bert-large-cased') 21 | self.bertmodel = BertForMaskedLM.from_pretrained("bert-large-cased") 22 | self.bertori = BertModel.from_pretrained("bert-large-cased") 23 | self.bertmodel.eval().cuda() 24 | self.bertori.eval().cuda() 25 | # self.nlp = stanza.Pipeline(lang='en', processors='tokenize,mwt,pos') 26 | self.poses = ["JJ", "VB"] 27 | self.scorer_default = Scorer( 28 | 8, 29 | 'bert-base-uncased', 30 | 'princeton-nlp/sup-simcse-bert-base-uncased', 31 | True 32 | ) 33 | self.similarity_threshold = 0.65 34 | self.predcition_threshold = 9 35 | self.max_sentences = 5 36 | # self.AEON_threshold = 0 37 | self.AEON_threshold = aeon_threshold 38 | self.rand_seed = 14 39 | self.evaluate = evaluate 40 | self.filter_flag = aeon_flag 41 | 42 | def __get_entity_ids(self, NEs, word_list): 43 | """ 44 | Find the ids in the word list of the entities 45 | 46 | return: 47 | a list of ids of the entities 48 | """ 49 | entity_ids = [] 50 | for entity_name in NEs: 51 | # change eneity words into strings without whitespace 52 | curr_name = entity_name.replace(" ", "") 53 | ori_curr_name = '%s' % curr_name 54 | id_list = [] 55 | for index, word in enumerate(word_list): 56 | word = word.replace(" ", "") 57 | if curr_name.startswith(word): 58 | curr_name = curr_name[len(word):] 59 | id_list.append(index) 60 | else: 61 | curr_name = ori_curr_name 62 | id_list = [] # clear the ids 63 | if len(curr_name) == 0: 64 | for i in id_list: 65 | if i not in entity_ids: 66 | entity_ids.append(i) 67 | curr_name = ori_curr_name 68 | return entity_ids 69 | 70 | def __get_AEON_score(self, sentence): 71 | return self.scorer_default.compute_naturalness(sentence) 72 | 73 | def __tokens_and_words_map(self, tokens, word_list): 74 | # create a mapping between tokens list and word list 75 | tokens_to_words = {} 76 | i = 0 77 | curr_token = "" 78 | for index, token in enumerate(tokens): 79 | if token == "[CLS]" or token == "[SEP]": 80 | tokens_to_words[index] = -1 81 | if token.startswith("##"): 82 | token = token.replace("##", "") 83 | curr_token += token 84 | if curr_token.casefold() == word_list[i].casefold(): 85 | # clear the token 86 | curr_token = "" 87 | tokens_to_words[index] = i 88 | i += 1 89 | else: 90 | tokens_to_words[index] = i 91 | words_to_tokens = defaultdict(list) 92 | for i, v in tokens_to_words.items(): 93 | words_to_tokens[v].append(i) 94 | return tokens_to_words, words_to_tokens 95 | 96 | def __bert_predict_mask(self, tokens, masked_index, number = 35): 97 | indexed_tokens = self.berttokenizer.convert_tokens_to_ids(tokens) 98 | tokens_tensor = torch.tensor([indexed_tokens]).cuda() 99 | prediction = self.bertmodel(tokens_tensor) 100 | topk = torch.topk(prediction[0][0], number) 101 | topk_idx = topk[1].tolist() 102 | topk_value = topk[0].tolist() 103 | res_tokens = self.berttokenizer.convert_ids_to_tokens(topk_idx[masked_index]) 104 | res_values = topk_value[masked_index] 105 | # ipdb.set_trace() 106 | return res_tokens, res_values 107 | 108 | def __mask_tokens(self, tokens, lindex, rindex): 109 | new_tokens = tokens[0:lindex] + ["[MASK]"] + tokens[rindex + 1:] 110 | new_tokens = ["[CLS]"] + new_tokens + ["[SEP]"] 111 | # print(new_tokens) 112 | return new_tokens 113 | 114 | # compute the word embedding activated 115 | def __compEmbeddings( self, tokens, lindex, rindex ): 116 | rindex = rindex - 1 117 | indexed_tokens = self.berttokenizer.convert_tokens_to_ids(tokens) 118 | segments_ids = [1] * len(tokens) 119 | tokens_tensor = torch.tensor( [indexed_tokens] ).cuda() 120 | segments_tensors = torch.tensor( [segments_ids] ).cuda() 121 | with torch.no_grad(): 122 | outputs = self.bertori( tokens_tensor, segments_tensors, output_hidden_states=True ) 123 | hidden_states = outputs[2] 124 | 125 | token_embeddings = torch.stack(hidden_states, dim=0) 126 | token_embeddings = torch.squeeze(token_embeddings, dim=1) 127 | token_embeddings = token_embeddings.permute(1,0,2) 128 | # Stores the token vectors, with shape [22 x 3,072] 129 | token_vecs_cat = [] 130 | # `token_embeddings` is a [22 x 12 x 768] tensor. 131 | # For each token in the sentence... 132 | for token in token_embeddings: 133 | # `token` is a [12 x 768] tensor 134 | # Concatenate the vectors (that is, append them together) from the last 135 | # four layers. 136 | # Each layer vector is 768 values, so `cat_vec` is length 3,072. 137 | cat_vec = torch.cat((token[-1], token[-2], token[-3], token[-4]), dim=0) 138 | # Use `cat_vec` to represent `token`. 139 | token_vecs_cat.append(cat_vec) 140 | 141 | # Stores the token vectors, with shape 142 | token_vecs_sum = [] 143 | # For each token in the sentence... 144 | for token in token_embeddings: 145 | 146 | # Sum the vectors from the last four layers. 147 | sum_vec = torch.sum(token[-4:], dim=0) 148 | # Use `sum_vec` to represent `token`. 149 | token_vecs_sum.append(sum_vec) 150 | 151 | ans_vec = deepcopy(token_vecs_sum[lindex]) 152 | for index in range(lindex + 1, rindex + 1): 153 | ans_vec = ans_vec + token_vecs_sum[index] 154 | 155 | ans_vec = ans_vec / (rindex - lindex + 1) 156 | # return the word embeddings for the word 157 | return torch.unsqueeze(ans_vec, dim=0 ) 158 | 159 | 160 | def __compute_simlarity(self, tokens, pert_tokens, lindex1, rindex1, lindex2, rindex2): 161 | 162 | # begin cav 163 | tokens_dp = deepcopy(tokens) 164 | pert_tokens_dp = deepcopy(pert_tokens) 165 | ha = self.__compEmbeddings( tokens_dp, lindex1, rindex1 ) 166 | hb = self.__compEmbeddings( pert_tokens_dp, lindex2, rindex2 ) 167 | 168 | cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6) 169 | return abs(cos(ha, hb)) 170 | 171 | 172 | def __to_possiblity(self, items): 173 | sum = 0 174 | result = [] 175 | for item in items: sum += item[1] 176 | for item in items: result.append((item[0], item[1] / sum)) 177 | return result 178 | 179 | def generate_sentences(self, sentence, NEs): 180 | tokens = self.berttokenizer.tokenize(sentence) 181 | word_list = sentence.split(" ") 182 | tokens_to_words, words_to_tokens = self.__tokens_and_words_map(tokens, word_list) 183 | # select the words that can be perturbed 184 | perturbable_ids = [] 185 | entity_ids = self.__get_entity_ids(NEs, word_list) 186 | pos_word_list = nltk.pos_tag(tokens) 187 | for id_token, pos_info in enumerate(pos_word_list): 188 | id_word = tokens_to_words[id_token] 189 | if id_word in entity_ids: 190 | continue 191 | if pos_info[1] in self.poses: 192 | if pos_info[1] == "JJ" and (id_word+1) in entity_ids: 193 | # do not change the adj that is adjacent to the entity 194 | continue 195 | if id_word not in perturbable_ids: # prevent repitition 196 | perturbable_ids.append(id_word) 197 | substitution_dict = defaultdict(list) 198 | ori_number = 0 199 | sim_number = 0 200 | # generate new sentences 201 | for word_id in perturbable_ids: 202 | token_ids = words_to_tokens[word_id] 203 | lindex = token_ids[0] 204 | rindex = token_ids[len(token_ids)-1] + 1 205 | new_tokens = self.__mask_tokens(tokens, lindex, rindex) 206 | res_tokens, res_values = self.__bert_predict_mask(new_tokens, lindex+1) 207 | lindex2 = lindex 208 | rindex2 = lindex + 1 209 | # compute cosine similarity of 210 | for index, token in enumerate(res_tokens[:10]): 211 | # fliter out the illegal format 212 | if token.startswith("##"): 213 | continue 214 | # check the prediction score 215 | if res_values[index] <= self.predcition_threshold: 216 | continue 217 | # compare pos tag, if the tag is not consistent, abort 218 | pert_tokens = tokens[0: lindex] + [token] + tokens[rindex: ] 219 | pos_pert_token = nltk.pos_tag(pert_tokens) 220 | 221 | if pos_word_list[lindex][1] != pos_pert_token[lindex][1]: 222 | continue 223 | # compare the embedding similarity, if the similarity is greater than a threshold, continue 224 | similarity = self.__compute_simlarity(tokens,pert_tokens, lindex, rindex, lindex2, rindex2) 225 | ori_number += 1 226 | if similarity.item() <= self.similarity_threshold: 227 | continue 228 | sim_number += 1 229 | if token != word_list[tokens_to_words[lindex]]: 230 | substitution_dict[tokens_to_words[lindex]].append(token) 231 | 232 | # change the words according to the substitution dict 233 | candidate_sentences = [] 234 | for (id, candidate_words) in substitution_dict.items(): 235 | for candidate_word in candidate_words: 236 | temp_word_list = word_list.copy() 237 | temp_word_list[id] = candidate_word 238 | new_sentence = " ".join(temp_word_list) 239 | new_words = [candidate_word] 240 | candidate_sentences.append((new_sentence, new_words)) 241 | 242 | # fitler the candiate sentences by AEON 243 | filtered_sentences = [] 244 | if self.filter_flag: 245 | ori_score = self.__get_AEON_score(sentence) 246 | for index, candidate in enumerate(candidate_sentences): 247 | new_score = self.__get_AEON_score(candidate[0]) 248 | if new_score >= ori_score - self.AEON_threshold: 249 | filtered_sentences.append(candidate) 250 | else: 251 | filtered_sentences = candidate_sentences 252 | aeon_number = len(filtered_sentences) 253 | 254 | changed = False 255 | if len(filtered_sentences) == 0 or \ 256 | (len(filtered_sentences) == 1 and filtered_sentences[0] == sentence): 257 | changed = False 258 | if self.evaluate == True: 259 | return [ori_number, sim_number, aeon_number] 260 | return (changed, filtered_sentences) 261 | else: 262 | changed = True 263 | if self.evaluate == True: 264 | return [ori_number, sim_number, aeon_number] 265 | return (changed, filtered_sentences) 266 | 267 | 268 | def format_to_string(result_dict): 269 | # convert the result dict to inputable string format 270 | sentence_str = " ".join(result_dict["sentence"]) 271 | entities = result_dict["entity"] 272 | NEs = [] 273 | for entity in entities: 274 | NEs.append(entity[0]) 275 | return sentence_str, NEs 276 | 277 | if __name__ == "__main__": 278 | mr_bert = MR_BERT() 279 | mr_bert.MR_init() 280 | print(mr_bert.generate_sentences("He said : For me this is such an exciting opportunity for Cornwall and something I ' m now desperate to see happen .", ["Cornwall"])) -------------------------------------------------------------------------------- /TIN_Test/Transformations/MR_intra_shuffle.py: -------------------------------------------------------------------------------- 1 | import sys 2 | # import the AEON, please change the follosing path with the path ofaeon.py 3 | # sys.path.insert(1, '$HOME$/NER_MRs/MRs/AEON/utils') 4 | from aeon import * 5 | import itertools 6 | import random 7 | import nltk 8 | import ipdb 9 | 10 | class MR_intra_shuffle: 11 | def MR_init(self, aeon_flag = True, evaluate = False, aeon_threshold = 0.01): 12 | self.scorer_default = Scorer( 13 | 8, 14 | 'bert-base-uncased', 15 | 'princeton-nlp/sup-simcse-bert-base-uncased', 16 | True 17 | ) 18 | self.permutation_bound = 4 # max number of entities to generate permutations 19 | self.generate_bound = 8 # max sentences to be generated repecting one type (before filtering) 20 | self.random_seed = 0 21 | self.max_sentences = 5 22 | # self.AEON_threshold = 0 23 | self.AEON_threshold = aeon_threshold 24 | self.filter_flag = aeon_flag 25 | self.evaluate = evaluate 26 | 27 | def __get_AEON_score(self, sentence): 28 | return self.scorer_default.compute_naturalness(sentence) 29 | 30 | def intra_sentence_entity_shuffle(self, word_list, entity_list): 31 | frame_word_list = word_list.copy() # a frame word list for generating new sentences 32 | entity_type_pool = {} # a pool to store all the entities of the same type 33 | 34 | # get the entity type pool and replace original entities with place hoders 35 | for (entity_name, entity_type, entity_indexes) in entity_list: 36 | current_type = entity_type 37 | if current_type not in entity_type_pool: 38 | entity_type_pool[current_type] = [] 39 | entity_type_pool[current_type].append(entity_name) 40 | for index in entity_indexes: 41 | type_hoder = "" + current_type + "+" + entity_name 42 | frame_word_list[index] = type_hoder 43 | 44 | # concate the adjacent place holders with the same type 45 | # the new_word_list is a new list of words after concating the place holders 46 | last_word = None 47 | new_word_list = [] 48 | if len(frame_word_list) < 2: 49 | return (False, []) 50 | for i in range(0, len(frame_word_list)): 51 | curr = frame_word_list[i] 52 | if curr.startswith(""): 53 | if last_word == curr: 54 | continue # do not append to the new word list 55 | else: 56 | last_word = curr 57 | new_word_list.append(curr) 58 | else: 59 | last_word = curr 60 | new_word_list.append(curr) 61 | 62 | # generate permuatations or shuffled lists of entity names 63 | entity_types = list(entity_type_pool.keys()) 64 | new_entity_pool_list = {} 65 | for e_type in entity_types: 66 | new_entity_pool_list[e_type] = [] 67 | new_type_pool_list = [] 68 | if len(entity_type_pool[e_type]) <= self.permutation_bound: # get permutation if there are not many entities 69 | new_type_pool_list = list(itertools.permutations(entity_type_pool[e_type])) 70 | else: # otherwise randomly generate 71 | seed_now = self.random_seed 72 | for i in range(0, self.generate_bound): 73 | temp_list = entity_type_pool[e_type] 74 | random.Random(seed_now).shuffle(temp_list) 75 | new_type_pool_list.append(temp_list) 76 | seed_now += 1 77 | 78 | new_entity_pool_list[e_type] = new_type_pool_list 79 | 80 | # generate new sentences 81 | ori_number = 0 82 | sim_number = 0 83 | aeon_number = 0 84 | new_sentences = [] 85 | new_sentences.append(new_word_list) 86 | occurences_all = [] 87 | # ipdb.set_trace() 88 | for e_type in entity_types: 89 | i = 0 90 | occurences = [i for i, x in enumerate(new_word_list) if x.startswith(""+e_type)] 91 | occurences_all.extend(occurences) 92 | temp_sentences = [] 93 | while i < len(new_sentences): 94 | sentence_list = new_sentences[i] 95 | for entity_names in new_entity_pool_list[e_type]: 96 | if len(entity_names) < len(occurences): 97 | continue 98 | new_sentence_list = sentence_list.copy() 99 | index = 0 100 | while index < len(occurences): 101 | new_sentence_list[occurences[index]] = entity_names[index] 102 | index += 1 103 | temp_sentences.append(new_sentence_list) 104 | i += 1 105 | new_sentences.clear() 106 | for item in temp_sentences: 107 | new_sentences.append(item) 108 | 109 | candiate_sentences = new_sentences 110 | ori_number = len(candiate_sentences) 111 | sim_number = len(candiate_sentences) 112 | 113 | # AEON filter 114 | result_sentences = [] 115 | original_sentence = " ".join(word_list) 116 | if self.filter_flag: 117 | ori_score = self.__get_AEON_score(original_sentence) 118 | for sentence_list in candiate_sentences: 119 | new_sentence = " ".join(sentence_list) 120 | if original_sentence == new_sentence: 121 | continue 122 | new_score = self.__get_AEON_score(new_sentence) 123 | if new_score < ori_score - self.AEON_threshold: 124 | continue 125 | result_sentences.append(new_sentence) 126 | else: 127 | result_sentences = candiate_sentences 128 | aeon_number = len(result_sentences) 129 | 130 | if len(result_sentences) == 0: 131 | return (False, []) 132 | if self.evaluate == True: 133 | return [ori_number, sim_number, aeon_number] 134 | return (True, result_sentences) 135 | 136 | if __name__ == "__main__": 137 | mr_intra_shuffle = MR_intra_shuffle() 138 | mr_intra_shuffle.MR_init() 139 | ent_info = {"sentence": ["On", "a", "video", "posted", "on", "the", "platform", "Rumble", ",", "Jones", "said", "he", "did", "not", "care", "about", "being", "on", "Twitter", ",", "reports", "news", "website", "Axios", "."], "entity": [["Rumble", "PRODUCT", [7]], ["Jones", "PERSON", [9]], ["Twitter", "PRODUCT", [18]], ["Axios", "ORG", [23]]]} 140 | print(mr_intra_shuffle.intra_sentence_entity_shuffle(ent_info["sentence"], ent_info["entity"])) 141 | 142 | -------------------------------------------------------------------------------- /TIN_Test/Transformations/MR_structure.py: -------------------------------------------------------------------------------- 1 | from stanfordcorenlp import StanfordCoreNLP 2 | from nltk.tree import * 3 | from nltk.stem.wordnet import WordNetLemmatizer 4 | import nltk.data 5 | import stanza 6 | import os 7 | # os.environ['CUDA_VISIBLE_DEVICES']='6' 8 | import sys 9 | import ipdb 10 | # import the AEON, please change the follosing path with the path ofaeon.py 11 | # sys.path.insert(1, '$HOME$/NER_MRs/MRs/AEON/utils') 12 | from aeon import * 13 | import pathlib 14 | 15 | 16 | import re 17 | alphabets= "([A-Za-z])" 18 | prefixes = "(Mr|St|Mrs|Ms|Dr)[.]" 19 | suffixes = "(Inc|Ltd|Jr|Sr|Co)" 20 | starters = "(Mr|Mrs|Ms|Dr|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)" 21 | acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)" 22 | websites = "[.](com|net|org|io|gov)" 23 | digits = "([0-9])" 24 | 25 | def split_into_sentences(text): 26 | text = " " + text + " " 27 | text = text.replace("\n"," ") 28 | text = re.sub(prefixes,"\\1",text) 29 | text = re.sub(websites,"\\1",text) 30 | text = re.sub(digits + "[.]" + digits,"\\1\\2",text) 31 | if "..." in text: text = text.replace("...","") 32 | if "Ph.D" in text: text = text.replace("Ph.D.","PhD") 33 | text = re.sub("\s" + alphabets + "[.] "," \\1 ",text) 34 | text = re.sub(acronyms+" "+starters,"\\1 \\2",text) 35 | text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]","\\1\\2\\3",text) 36 | text = re.sub(alphabets + "[.]" + alphabets + "[.]","\\1\\2",text) 37 | text = re.sub(" "+suffixes+"[.] "+starters," \\1 \\2",text) 38 | text = re.sub(" "+suffixes+"[.]"," \\1",text) 39 | text = re.sub(" " + alphabets + "[.]"," \\1",text) 40 | if "”" in text: text = text.replace(".”","”.") 41 | if "\"" in text: text = text.replace(".\"","\".") 42 | if "!" in text: text = text.replace("!\"","\"!") 43 | if "?" in text: text = text.replace("?\"","\"?") 44 | text = text.replace(".",".") 45 | text = text.replace("?","?") 46 | text = text.replace("!","!") 47 | text = text.replace("",".") 48 | sentences = text.split("") 49 | return_list = [] 50 | for s in sentences: 51 | if len(s.replace(" ", "")) != 0: 52 | return_list.append(s) 53 | return return_list 54 | 55 | class MR_structure: 56 | def MR_init(self, aeon_flag, evaluate = False, aeon_threshold = 0.02): 57 | self.nlp_pos = stanza.Pipeline(lang='en', processors='tokenize,mwt,pos') 58 | self.nlp = StanfordCoreNLP( 59 | str(pathlib.Path().absolute())+'/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 60 | # self.nlp = StanfordCoreNLP( 61 | # r'$HOME$/NER_MRs/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 62 | self.scorer_default = Scorer( 63 | 8, 64 | 'bert-base-uncased', 65 | 'princeton-nlp/sup-simcse-bert-base-uncased', 66 | True 67 | ) 68 | self.AEON_threshold = aeon_threshold 69 | self.filter_flag = aeon_flag 70 | self.evaluate = evaluate 71 | 72 | def __get_AEON_score(self, sentence): 73 | return self.scorer_default.compute_naturalness(sentence) 74 | 75 | def MR_close(self): 76 | self.nlp.close() 77 | 78 | def __get_entity_ids(self, NEs, word_list): 79 | entity_ids = [] 80 | for entity_name in NEs: 81 | # change eneity words into strings without whitespace 82 | curr_name = entity_name.replace(" ", "") 83 | ori_curr_name = '%s' % curr_name 84 | id_list = [] 85 | for index, word in enumerate(word_list): 86 | word = word.replace(" ", "") 87 | if curr_name.startswith(word): 88 | curr_name = curr_name[len(word):] 89 | id_list.append(index) 90 | else: 91 | curr_name = ori_curr_name 92 | id_list = [] # clear the ids 93 | if len(curr_name) == 0: 94 | for i in id_list: 95 | if i not in entity_ids: 96 | entity_ids.append(i) 97 | curr_name = ori_curr_name 98 | return entity_ids 99 | 100 | def __checkFirstNoun(self, Snode, NEs): 101 | # first convert the subtree to a sentence 102 | tokens = Snode.leaves() 103 | for i in range(0, len(tokens)): 104 | if tokens[i] == "-LRB-": 105 | tokens[i] = "(" 106 | elif tokens[i] == "-RRB-": 107 | tokens[i] = ")" 108 | S_sentence = " ".join(tokens) 109 | entity_ids = self.__get_entity_ids(NEs, tokens) 110 | # check if "I" is the first word 111 | if len(S_sentence) == 0: 112 | return False 113 | if tokens[0].casefold() == "i": 114 | return True 115 | # check if the first word is in the entity words 116 | if 0 in entity_ids: 117 | return True 118 | # check pos tag: if NNP 119 | doc = self.nlp_pos(S_sentence) 120 | word = doc.sentences[0].words[0] 121 | if word.xpos == "NNP" or word.xpos == "NNPS": 122 | return True 123 | return False 124 | 125 | def __get_leftest_leaf(self, node): 126 | temp = node 127 | while type(temp) != str: 128 | temp = temp[0] 129 | return temp 130 | 131 | def __get_rightest_leaf(self, node): 132 | temp = node 133 | while type(temp) != str: 134 | temp = temp[len(temp)-1] 135 | return temp 136 | 137 | def __get_leftest_label(self, node): 138 | temp = node 139 | ref_list = [] 140 | while type(temp) != str and type(temp[0]) != str: 141 | temp = temp[0] 142 | ref_list.append(0) 143 | return temp, ref_list 144 | 145 | def __find_sentence_nodes(self, root): 146 | node_list = [] 147 | nodeIndex_list = [] 148 | 149 | def find_node(node): 150 | find_list = [] 151 | find_list.append((node, [])) # [0]-> record the indexing info of the current node 152 | while len(find_list) > 0: 153 | curr = find_list.pop(0) 154 | if type(curr[0]) == str: # leaf 155 | continue 156 | else: 157 | if curr[0].label() == "SQ" or curr[0].label() == "SBAR": 158 | continue 159 | if curr[0].label() == "S": 160 | node_list.append(curr[0]) 161 | nodeIndex_list.append(curr[1]) 162 | continue 163 | i = 0 164 | for child in curr[0]: 165 | temp_list = curr[1].copy() 166 | temp_list.append(i) 167 | find_list.append((child, temp_list)) 168 | i += 1 169 | 170 | find_node(root) 171 | return node_list, nodeIndex_list 172 | 173 | def __declarative_2_interrogative_Tree(self, sentenceNode, NEs, mode): 174 | 175 | # change capital letter of the NP in the sentence 176 | def changeCapLetter_lower(node): 177 | nn_node, ref_list = self.__get_leftest_label(node) 178 | label_nn = nn_node.label() 179 | string_nn = nn_node[0] 180 | if (string_nn.isupper() and len(string_nn)>2) or self.__checkFirstNoun(node, NEs): 181 | return node 182 | # else change the noun to the lower case format 183 | var_string = "node" 184 | assign_string = "=string_nn.lower()" 185 | for i in ref_list: 186 | var_string += "[0]" 187 | var_string += "[0]" 188 | exec(var_string + assign_string) 189 | return node 190 | 191 | # change the first letter of the sentence to upper case 192 | def changeCapLetter_upper(node): 193 | nn_node, ref_list = self.__get_leftest_label(node) 194 | label_nn = nn_node.label() 195 | string_nn = nn_node[0] 196 | # else change the noun to the lower case format 197 | string_nn = string_nn[0].upper() + string_nn[1:] 198 | var_string = "node" 199 | assign_string = "=string_nn" 200 | for i in ref_list: 201 | var_string += "[0]" 202 | var_string += "[0]" 203 | exec(var_string + assign_string) 204 | return node 205 | 206 | # change the declarative sentence to a general question 207 | def d2q_GEN(Snode, isSentence): 208 | # if the first word of the sentence is not a word 209 | if self.__get_leftest_leaf(Snode).isalnum() == False: 210 | return Snode, False 211 | # find the VP in S, S should be NP + VP 212 | if type(Snode) == str: 213 | return Snode, False 214 | if Snode[0].label() != "NP": 215 | return Snode, False 216 | curr_node = Snode[0] 217 | i = 0 218 | for child in Snode: 219 | if child.label() == "VP": 220 | curr_node = child 221 | break 222 | i += 1 223 | if i == 0 or i == len(Snode): 224 | return Snode, False # no VP 225 | # check the VB form of the VP 226 | # modal verb 227 | if type(curr_node[0]) == str: # not supported format 228 | return Snode, False 229 | if len(Snode) <= 1: 230 | # in case the sentence only consist a NP and VP, this does not follow the rule 231 | return Snode, False 232 | if curr_node[0].label() == "MD": 233 | # modal verb 234 | Snode = changeCapLetter_lower(Snode) 235 | temp = curr_node.pop(0) 236 | Snode.insert(0, Tree("MD", temp)) 237 | if Snode[len(Snode)-1].label() == ".": 238 | Snode[len(Snode)-1] = Tree(".", ["?"]) 239 | if isSentence: Snode = changeCapLetter_upper(Snode) 240 | return Snode, True 241 | elif curr_node[0].label().startswith("VB"): 242 | if type(curr_node[0][0]) != str or len(curr_node)<2: 243 | return Snode, False 244 | # be verbs (eg: is doing) 245 | elif curr_node[0][0] == "am" or curr_node[0][0] == "is" or curr_node[0][0] == "is" or curr_node[0][0] == "was" or curr_node[0][0] == "are" or curr_node[0][0] == "were": 246 | # swqp the NP and the VB 247 | Snode = changeCapLetter_lower(Snode) 248 | temp = curr_node.pop(0) 249 | Snode.insert(0, Tree("VB", temp)) 250 | if Snode[len(Snode)-1].label() == ".": 251 | Snode[len(Snode)-1] = Tree(".", ["?"]) 252 | if isSentence: Snode = changeCapLetter_upper(Snode) 253 | return Snode, True 254 | # participated past verbs (eg: have done) 255 | # check support format 256 | elif type(curr_node[1]) == str: 257 | return Snode, False 258 | elif type(curr_node[1][0]) == str: 259 | return Snode, False 260 | elif (curr_node[0][0] == "have" or curr_node[0][0] == "has" or curr_node[0][0] == "had")\ 261 | and curr_node[1][0].label() == "VBN": 262 | # swqp the NP and the VB (have) 263 | Snode = changeCapLetter_lower(Snode) 264 | temp = curr_node.pop(0) 265 | Snode.insert(0, Tree("VB", temp)) 266 | if Snode[len(Snode)-1].label() == ".": 267 | Snode[len(Snode)-1] = Tree(".", ["?"]) 268 | if isSentence: Snode = changeCapLetter_upper(Snode) 269 | return Snode, True 270 | # normal verbs (do) 271 | else: 272 | if curr_node[0].label() == "VB" or curr_node[0].label() == "VBG" or curr_node[0].label() == "VBP": 273 | if type(curr_node[0][0]) != str: return Snode 274 | Snode = changeCapLetter_lower(Snode) 275 | word = curr_node[0][0] 276 | word = WordNetLemmatizer().lemmatize(word,'v') 277 | curr_node[0][0] = word 278 | if isSentence: Snode.insert(0, Tree("VB", ["Did"])) # "do" may cause some problems 279 | else: isSentence: Snode.insert(0, Tree("VB", ["did"])) 280 | if Snode[len(Snode)-1].label() == ".": 281 | Snode[len(Snode)-1] = Tree(".", ["?"]) 282 | return Snode, True 283 | elif curr_node[0].label() == "VBZ": 284 | if type(curr_node[0][0]) != str: return Snode 285 | Snode = changeCapLetter_lower(Snode) 286 | word = curr_node[0][0] 287 | word = WordNetLemmatizer().lemmatize(word,'v') 288 | curr_node[0][0] = word 289 | if isSentence: Snode.insert(0, Tree("VB", ["Does"])) 290 | else: Snode.insert(0, Tree("VB", ["does"])) 291 | if Snode[len(Snode)-1].label() == ".": 292 | Snode[len(Snode)-1] = Tree(".", ["?"]) 293 | return Snode, True 294 | elif curr_node[0].label() == "VBD" or curr_node[0].label() == "VBN": 295 | if type(curr_node[0][0]) != str: return Snode 296 | Snode = changeCapLetter_lower(Snode) 297 | word = curr_node[0][0] 298 | word = WordNetLemmatizer().lemmatize(word,'v') 299 | curr_node[0][0] = word 300 | if isSentence: Snode.insert(0, Tree("VB", ["Did"])) 301 | else: Snode.insert(0, Tree("VB", ["did"])) 302 | if Snode[len(Snode)-1].label() == ".": 303 | Snode[len(Snode)-1] = Tree(".", ["?"]) 304 | return Snode, True 305 | else: 306 | return Snode, False 307 | else: 308 | return Snode, False 309 | 310 | shouldChange = False # when there is one change, this is set to true 311 | Snode_list, SnodeIndex_list = self.__find_sentence_nodes(sentenceNode) 312 | for i in range(0, len(Snode_list)): 313 | if mode == "GEN": 314 | isSentence = True # check if the current node represent a complete sentence 315 | listName = "sentenceNode" 316 | assignName = "=Snode" 317 | checkPrevName = "sentenceNode" 318 | for k in range(0, len(SnodeIndex_list[i])): 319 | # change the subtree 320 | listName += "[" + str(SnodeIndex_list[i][k]) + "]" 321 | if k == len(SnodeIndex_list[i])-1: 322 | if SnodeIndex_list[i][k] > 0: 323 | checkPrevName += "[" + str(SnodeIndex_list[i][k]-1) + "]" 324 | else: 325 | checkPrevName = "" 326 | else: 327 | checkPrevName += "[" + str(SnodeIndex_list[i][k]) + "]" 328 | try: 329 | prevWord = self.__get_rightest_leaf(eval(checkPrevName)) 330 | if prevWord == ",": 331 | isSentence = False 332 | else: 333 | isSentence = True 334 | except: 335 | isSentence = True 336 | Snode, change = d2q_GEN(Snode_list[i], isSentence) 337 | if change == False: continue 338 | shouldChange = True 339 | if type(eval(listName)) == str: 340 | return 341 | exec(listName+assignName) 342 | return shouldChange 343 | 344 | def __declarative_2_interrogative_single(self, sentence, NEs, mode = "GEN"): 345 | changed = False 346 | cap = sentence.isupper() 347 | try: 348 | parsingTree = Tree.fromstring(self.nlp.parse(sentence)) 349 | except: 350 | print("something wrong with the stanford parser: error creating parsing tree") 351 | print("issue sentence: ", sentence) 352 | return False, sentence 353 | temp = self.__declarative_2_interrogative_Tree(parsingTree, NEs, mode) 354 | if temp == False: 355 | # print("not changed !") 356 | return False, sentence 357 | else: 358 | changed = True 359 | tokens = parsingTree.leaves() 360 | for i in range(0, len(tokens)): 361 | if tokens[i] == "-LRB-": 362 | tokens[i] = "(" 363 | elif tokens[i] == "-RRB-": 364 | tokens[i] = ")" 365 | if cap and tokens[i].isupper() == False: 366 | tokens[i] = tokens[i].upper() 367 | return_string = " ".join(tokens) 368 | if " - " not in sentence: return_string = return_string.replace(" - ", "-") 369 | if "?" not in return_string: 370 | if return_string[len(return_string)-1] == ".": 371 | return_string = return_string[:len(return_string)-1] + "?" 372 | else: 373 | return_string += "?" # add question mark at the last if there is no punct 374 | return changed , return_string 375 | 376 | def declarative_2_interrogative(self, sentence, NEs): 377 | ori_number = 0 378 | sim_number = 0 379 | aeon_number = 0 380 | multi_changed = False 381 | if len(sentence) == 0: 382 | if self.evaluate == True: 383 | return [ori_number, sim_number, aeon_number] 384 | return (multi_changed, sentence) # empty sentence 385 | sentence_list = split_into_sentences(sentence) 386 | new_sentence_list = [] 387 | for s in sentence_list: 388 | changed, single_sentence = self.__declarative_2_interrogative_single(s, NEs) 389 | if changed: 390 | multi_changed = True 391 | ori_number = 1 392 | sim_number = 1 393 | new_sentence_list.append(single_sentence) 394 | if not multi_changed: 395 | if self.evaluate == True: 396 | return [ori_number, sim_number, aeon_number] 397 | return (multi_changed, sentence) 398 | new_sentence = " ".join(new_sentence_list) 399 | print(new_sentence) 400 | # use aeon to filter 401 | if self.filter_flag: 402 | ori_score = self.__get_AEON_score(sentence) 403 | new_score = self.__get_AEON_score(new_sentence) 404 | if new_score < ori_score - self.AEON_threshold: 405 | multi_changed = False 406 | new_sentence = sentence 407 | else: 408 | aeon_number = 1 409 | if self.evaluate == True: 410 | print(self.evaluate) 411 | return [ori_number, sim_number, aeon_number] 412 | return (multi_changed, new_sentence) 413 | 414 | 415 | def show(self, sentence): 416 | parsingTree = Tree.fromstring(self.nlp.parse(sentence)) 417 | print(self.nlp.parse(sentence)) 418 | parsingTree.pretty_print() 419 | 420 | if __name__ == "__main__": 421 | mr_structure = MR_structure() 422 | mr_structure.MR_init(aeon_flag=True) 423 | mr_structure.declarative_2_interrogative("I am good .", []) 424 | mr_structure.MR_close() -------------------------------------------------------------------------------- /TIN_Test/Transformations/MR_synonym.py: -------------------------------------------------------------------------------- 1 | import os 2 | # os.environ['CUDA_VISIBLE_DEVICES']='6' 3 | import spacy 4 | from stanfordcorenlp import StanfordCoreNLP 5 | from nltk.tree import * 6 | import itertools 7 | import random 8 | import sys 9 | import pdb 10 | import ipdb 11 | # import the AEON, please change the follosing path with the path ofaeon.py 12 | # sys.path.insert(1, '$HOME$/NER_MRs/MRs/AEON/utils') 13 | from aeon import * 14 | from copy import deepcopy 15 | from collections import defaultdict 16 | import torch 17 | import itertools 18 | import nltk 19 | from transformers import BertConfig, BertTokenizer, BertModel, RobertaTokenizer, RobertaModel, BertForMaskedLM 20 | import pathlib 21 | 22 | import re 23 | alphabets= "([A-Za-z])" 24 | prefixes = "(Mr|St|Mrs|Ms|Dr)[.]" 25 | suffixes = "(Inc|Ltd|Jr|Sr|Co)" 26 | starters = "(Mr|Mrs|Ms|Dr|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)" 27 | acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)" 28 | websites = "[.](com|net|org|io|gov)" 29 | digits = "([0-9])" 30 | 31 | def split_into_sentences(text): 32 | text = " " + text + " " 33 | text = text.replace("\n"," ") 34 | text = re.sub(prefixes,"\\1",text) 35 | text = re.sub(websites,"\\1",text) 36 | text = re.sub(digits + "[.]" + digits,"\\1\\2",text) 37 | if "..." in text: text = text.replace("...","") 38 | if "Ph.D" in text: text = text.replace("Ph.D.","PhD") 39 | text = re.sub("\s" + alphabets + "[.] "," \\1 ",text) 40 | text = re.sub(acronyms+" "+starters,"\\1 \\2",text) 41 | text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]","\\1\\2\\3",text) 42 | text = re.sub(alphabets + "[.]" + alphabets + "[.]","\\1\\2",text) 43 | text = re.sub(" "+suffixes+"[.] "+starters," \\1 \\2",text) 44 | text = re.sub(" "+suffixes+"[.]"," \\1",text) 45 | text = re.sub(" " + alphabets + "[.]"," \\1",text) 46 | if "”" in text: text = text.replace(".”","”.") 47 | if "\"" in text: text = text.replace(".\"","\".") 48 | if "!" in text: text = text.replace("!\"","\"!") 49 | if "?" in text: text = text.replace("?\"","\"?") 50 | text = text.replace(".",".") 51 | text = text.replace("?","?") 52 | text = text.replace("!","!") 53 | text = text.replace("",".") 54 | sentences = text.split("") 55 | return_list = [] 56 | for s in sentences: 57 | if len(s.replace(" ", "")) != 0: 58 | return_list.append(s) 59 | return return_list 60 | 61 | class MR_synonym: 62 | def MR_init(self, aeon_flag = True, evaluate = False, aeon_threshold = 0.01): 63 | self.nlp = spacy.load("en_core_web_sm") 64 | self.s2v = self.nlp.add_pipe("sense2vec") 65 | self.s2v.from_disk(str(pathlib.Path().absolute()) + "/s2v_old") 66 | # self.s2v.from_disk("../s2v_old") # you path to the s2v package 67 | self.nlp_stanford = StanfordCoreNLP( 68 | str(pathlib.Path().absolute())+'/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 69 | # self.nlp_stanford = StanfordCoreNLP( 70 | # r'../corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 71 | self.scorer_default = Scorer( 72 | 8, 73 | 'bert-base-uncased', 74 | 'princeton-nlp/sup-simcse-bert-base-uncased', 75 | True 76 | ) 77 | self.berttokenizer = BertTokenizer.from_pretrained('bert-large-cased') 78 | self.bertmodel = BertForMaskedLM.from_pretrained("bert-large-cased") 79 | self.bertori = BertModel.from_pretrained("bert-large-cased") 80 | self.bertmodel.eval().cuda() 81 | self.bertori.eval().cuda() 82 | self.threshold = 0.65 # threshold for similarity (when generating) 83 | self.similarity_threshold = 0.65 84 | self.max_sentences = 5 # maximum number of similar sentences 85 | self.AEON_threshold = aeon_threshold 86 | self.bound = 10 # max synonym to choose 87 | self.random_seed = 14 88 | self.filter_flag = aeon_flag 89 | self.evaluate = evaluate 90 | 91 | def MR_close(self): 92 | self.nlp_stanford.close() 93 | 94 | def __check_normal_string(self, word): 95 | def find_url(string): 96 | regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))" 97 | url = re.findall(regex, string) 98 | return [x[0] for x in url] 99 | 100 | for character in word: 101 | if ord(character) not in range(32, 128): 102 | return False 103 | if find_url(word): 104 | return False 105 | if any(item in word for item in ["#", "/", "^", "@", "<", ">"]): 106 | return False 107 | return True 108 | 109 | 110 | def __get_AEON_score(self, sentence): 111 | return self.scorer_default.compute_naturalness(sentence) 112 | 113 | def __tokens_and_words_map(self, tokens, word_list): 114 | # create a mapping between tokens list and word list 115 | tokens_to_words = {} 116 | i = 0 117 | curr_token = "" 118 | for index, token in enumerate(tokens): 119 | if token == "[CLS]" or token == "[SEP]": 120 | tokens_to_words[index] = -1 121 | if token.startswith("##"): 122 | token = token.replace("##", "") 123 | curr_token += token 124 | if curr_token.replace(" ", "") == word_list[i].replace(" ", ""): 125 | # if curr_token.casefold() == word_list[i].casefold(): 126 | # clear the token 127 | curr_token = "" 128 | tokens_to_words[index] = i 129 | i += 1 130 | else: 131 | tokens_to_words[index] = i 132 | words_to_tokens = defaultdict(list) 133 | for i, v in tokens_to_words.items(): 134 | words_to_tokens[v].append(i) 135 | 136 | return tokens_to_words, words_to_tokens 137 | 138 | def __node_to_index(self, root, node_reference): 139 | # get the start index of the tree node given the 140 | count = 0 141 | for depth in reversed(range(len(node_reference))): 142 | current_index = node_reference[depth] 143 | for layer_index in range(current_index): 144 | # refer to the node by root 145 | root_name = "root" 146 | for i in range(depth): 147 | root_name += "[" + str(node_reference[i]) + "]" 148 | root_name += "[" + str(layer_index) + "]" 149 | eval_str = ".leaves()" 150 | count += len(eval(root_name + eval_str)) 151 | return count 152 | 153 | def __find_NP_nodes(self, root): 154 | node_list = [] 155 | nodeIndex_list = [] 156 | 157 | def findNP(node): 158 | found = False 159 | search_list = [] 160 | if type(node) == str: 161 | return False 162 | for child in node: 163 | search_list.append(child) 164 | while len(search_list) > 0: 165 | curr = search_list.pop(0) 166 | if type(curr) == str: # leaf 167 | continue 168 | else: 169 | if curr.label().startswith("NP"): 170 | found = True 171 | return found 172 | else: 173 | for child in curr: 174 | search_list.append(child) 175 | return found 176 | 177 | def findNodes_NP(node): # use DFS search 178 | find_list = [] 179 | find_list.append((node, [])) # [0]-> record the indexing info of the current node 180 | while len(find_list) > 0: 181 | curr = find_list.pop(0) 182 | if type(curr[0]) == str: # leaf 183 | continue 184 | else: 185 | if curr[0].label().startswith("NP"): 186 | if findNP(curr[0]) == False: 187 | node_list.append(curr[0]) 188 | nodeIndex_list.append(curr[1]) 189 | i = 0 190 | for child in curr[0]: 191 | temp_list = curr[1].copy() 192 | temp_list.append(i) 193 | find_list.append((child, temp_list)) 194 | i += 1 195 | 196 | findNodes_NP(root) 197 | return node_list, nodeIndex_list 198 | 199 | def __changeNP(self, node, NEs): 200 | NEs = [e.replace(" ", "") for e in NEs] 201 | # check if the current node contains the entity 202 | curr_word = ("".join(node.leaves())).replace(" ", "") 203 | curr_word_check = (" ".join(node.leaves())).lstrip().rstrip() 204 | # print("current check word is: ", curr_word_check) 205 | if any(curr_word_check.casefold().startswith(item) for item in ["the ", "a ", "an "]): 206 | curr_word_check = (" ".join(curr_word_check.split(" ")[1:])).replace(" ","") 207 | else: 208 | curr_word_check = curr_word_check.replace(" ", "") 209 | if any(ext in curr_word_check for ext in NEs): 210 | return [node] 211 | if curr_word_check.casefold() not in ["a", "the", "an"] and any(curr_word_check in ext for ext in NEs): 212 | return [node] 213 | if curr_word_check.casefold() in ["a", "the", "an"]: 214 | return [node] 215 | 216 | # check the leftest node 217 | is_a = False 218 | keep_left = False 219 | temp_node = node.copy() # a temp node to find the noun 220 | if temp_node.leaves()[0].casefold() == "a" or temp_node.leaves()[0].casefold() == "an": 221 | is_a = True 222 | if temp_node[0].label().startswith("DT") or temp_node[0].label().startswith("PRP"): 223 | temp_node.pop(0) 224 | keep_left = True 225 | word_list = temp_node.leaves(); 226 | sub_string = " ".join(word_list) 227 | # use the sense2vec to substitute 228 | doc = self.nlp(sub_string) 229 | try: 230 | freq = doc[0:len(doc)]._.s2v_freq 231 | vector = doc[0:len(doc)]._.s2v_vec 232 | generate_list = [] 233 | most_similar = doc[0:len(doc)]._.s2v_most_similar(self.bound) 234 | except: 235 | return [node] 236 | for index, info in enumerate(most_similar): 237 | word = info[0][0] 238 | pos = info[0][1] 239 | score = info[1] 240 | if not self.__check_normal_string(word): 241 | continue 242 | if pos != "NOUN": 243 | continue 244 | if score < self.threshold: 245 | continue 246 | word = word.lstrip().rstrip() 247 | if word.replace(" ", "").casefold() == sub_string.replace(" ", "").casefold(): 248 | continue 249 | if keep_left: 250 | if is_a and word.endswith("s"): 251 | continue 252 | # check if the word start with DT 253 | if word.casefold().startswith('the '): 254 | word = word[4:] 255 | if word.casefold().startswith('an '): 256 | word = word[3:] 257 | if word.casefold().startswith('a '): 258 | word = word[2:] 259 | curr_node = Tree("NP", [Tree("NP", [word])]) 260 | curr_node.insert(0, node[0]) 261 | generate_list.append(curr_node) 262 | else: 263 | curr_node = Tree("NP", [word]) 264 | generate_list.append(curr_node) 265 | if len(generate_list) == 0: generate_list = [temp_node] 266 | return generate_list 267 | 268 | 269 | # compute the word embedding activated 270 | def __compEmbeddings( self, tokens, lindex, rindex ): 271 | rindex = rindex - 1 272 | indexed_tokens = self.berttokenizer.convert_tokens_to_ids(tokens) 273 | segments_ids = [1] * len(tokens) 274 | tokens_tensor = torch.tensor( [indexed_tokens] ).cuda() 275 | segments_tensors = torch.tensor( [segments_ids] ).cuda() 276 | with torch.no_grad(): 277 | outputs = self.bertori( tokens_tensor, segments_tensors, output_hidden_states=True ) 278 | hidden_states = outputs[2] 279 | 280 | token_embeddings = torch.stack(hidden_states, dim=0) 281 | token_embeddings = torch.squeeze(token_embeddings, dim=1) 282 | token_embeddings = token_embeddings.permute(1,0,2) 283 | # Stores the token vectors, with shape [22 x 3,072] 284 | token_vecs_cat = [] 285 | # `token_embeddings` is a [22 x 12 x 768] tensor. 286 | # For each token in the sentence... 287 | for token in token_embeddings: 288 | # `token` is a [12 x 768] tensor 289 | # Concatenate the vectors (that is, append them together) from the last 290 | # four layers. 291 | # Each layer vector is 768 values, so `cat_vec` is length 3,072. 292 | cat_vec = torch.cat((token[-1], token[-2], token[-3], token[-4]), dim=0) 293 | # Use `cat_vec` to represent `token`. 294 | token_vecs_cat.append(cat_vec) 295 | 296 | # Stores the token vectors, with shape 297 | token_vecs_sum = [] 298 | # For each token in the sentence... 299 | for token in token_embeddings: 300 | 301 | # Sum the vectors from the last four layers. 302 | sum_vec = torch.sum(token[-4:], dim=0) 303 | # Use `sum_vec` to represent `token`. 304 | token_vecs_sum.append(sum_vec) 305 | 306 | ans_vec = deepcopy(token_vecs_sum[lindex]) 307 | for index in range(lindex + 1, rindex + 1): 308 | ans_vec = ans_vec + token_vecs_sum[index] 309 | ans_vec = ans_vec / (rindex - lindex + 1) 310 | # return the word embeddings for the word 311 | return torch.unsqueeze(ans_vec, dim=0 ) 312 | 313 | 314 | def __compute_simlarity(self, tokens, pert_tokens, lindex1, rindex1, lindex2, rindex2): 315 | 316 | # begin cav 317 | tokens_dp = deepcopy(tokens) 318 | pert_tokens_dp = deepcopy(pert_tokens) 319 | ha = self.__compEmbeddings( tokens_dp, lindex1, rindex1 ) 320 | hb = self.__compEmbeddings( pert_tokens_dp, lindex2, rindex2 ) 321 | 322 | cos = torch.nn.CosineSimilarity(dim=1, eps=1e-6) 323 | return abs(cos(ha, hb)) 324 | 325 | 326 | def __generate_syn_sentences_single(self, sentence, NEs): 327 | filtered_number = 0 328 | # ipdb.set_trace() 329 | parsingTree = Tree.fromstring(self.nlp_stanford.parse(sentence)) 330 | # parsingTree.pretty_print() 331 | parsingTree_repr = repr(parsingTree) 332 | node_list, nodeIndex_list = self.__find_NP_nodes(parsingTree) 333 | substitution_dict = {} 334 | changable_list = [] 335 | for node in node_list: 336 | substitution_dict[repr(node)] = self.__changeNP(node, NEs) 337 | for node in node_list: 338 | changable_list.append([i for i in range(len(substitution_dict[repr(node)]))]) 339 | node_reprs = [repr(node) for node in node_list] 340 | original_words = [" ".join(node.leaves()) for node in node_list] 341 | result_sentences = [] 342 | new_words = [] 343 | for change_index in range(len(changable_list)): 344 | start_index = self.__node_to_index(eval(parsingTree_repr), nodeIndex_list[change_index]) 345 | 346 | for candidate_index in range(len(changable_list[change_index])): 347 | # map the node repr to the current index of candidate word to change 348 | new_words_current = [] 349 | temp_sent = eval(parsingTree_repr) 350 | node_repr = node_reprs[change_index] # get the current node repr of the node to be changed 351 | node_new = substitution_dict[node_repr][candidate_index] 352 | new_word = " ".join(node_new.leaves()) 353 | if original_words[change_index] == new_word: 354 | continue 355 | new_words_current.append(new_word) 356 | # use exec to change the subtree of the current tree 357 | node_name = "temp_sent" 358 | assign_name = "=node_new" 359 | for j in range(len(nodeIndex_list[change_index])): 360 | node_name += "[" + str(nodeIndex_list[change_index][j])+ "]" 361 | ori_word = " ".join(eval(node_name).leaves()) 362 | ori_offset = len(eval(node_name).leaves()) 363 | ori_word_list = temp_sent.leaves() 364 | exec(node_name + assign_name) 365 | new_word = " ".join(node_new.leaves()) 366 | new_offset = len((" ".join(node_new.leaves())).split(" ")) 367 | node_new_list = [] 368 | for leaf in node_new.leaves(): 369 | node_new_list.extend(leaf.split(" ")) 370 | new_word_list = temp_sent.leaves()[:start_index] + node_new_list + temp_sent.leaves()[start_index + new_offset:] 371 | # ipdb.set_trace() 372 | # ori_np = original_words[change_index] 373 | # new_np = new_word 374 | ori_bert_tokens = self.berttokenizer.tokenize(" ".join(ori_word_list)) 375 | new_bert_tokens = self.berttokenizer.tokenize(" ".join(new_word_list)) 376 | token2word_ori, word2token_ori = self.__tokens_and_words_map( ori_bert_tokens, ori_word_list ) 377 | token2word_new, word2token_new = self.__tokens_and_words_map( new_bert_tokens, new_word_list ) 378 | 379 | start_index_token_ori = word2token_ori[start_index][0] 380 | end_index_ori = start_index + ori_offset 381 | if end_index_ori >= len(ori_word_list): 382 | filtered_number += 1 383 | continue 384 | end_index_token_ori = word2token_ori[end_index_ori][len(word2token_ori[end_index_ori])-1] 385 | 386 | start_index_token_new = word2token_new[start_index][0] 387 | end_index_new = start_index + new_offset 388 | if end_index_new >= len(new_word_list): 389 | filtered_number += 1 390 | continue 391 | end_index_token_new = word2token_new[end_index_new][len(word2token_new[end_index_new])-1] 392 | 393 | # compare the embedding similarity, if the similarity is greater than a threshold, continue 394 | similarity = self.__compute_simlarity(ori_bert_tokens, new_bert_tokens, start_index_token_ori, end_index_token_ori, start_index_token_new, end_index_token_new) 395 | # print(similarity.item()) 396 | # ipdb.set_trace() 397 | if similarity.item() <= self.similarity_threshold: 398 | filtered_number += 1 399 | continue 400 | 401 | return_tokens = temp_sent.leaves() 402 | for i in range(0, len(return_tokens)): 403 | if return_tokens[i] == "-LRB-": 404 | return_tokens[i] = "(" 405 | elif return_tokens[i] == "-RRB-": 406 | return_tokens[i] = ")" 407 | 408 | return_string = " ".join(return_tokens) 409 | if " - " not in sentence: return_string = return_string.replace(" - ", "-") 410 | 411 | result_sentences.append(return_string) 412 | new_words.append(new_words_current) 413 | if self.evaluate == True: 414 | return result_sentences, new_words, filtered_number 415 | return result_sentences, new_words 416 | 417 | def generate_syn_sentences(self, sentence, NEs): 418 | ori_number = 0 419 | sim_number = 0 420 | aeon_number = 0 421 | sentence_list = split_into_sentences(sentence) 422 | new_indexes_list = [] # list of index list to repr the new sentence to choose 423 | new_sentences_list = [] # list of list of new sentences 424 | new_words_list = [] # list of list of new words 425 | total_filtered = 0 426 | for sent in sentence_list: 427 | new_indexes = [] 428 | new_sentences = [] 429 | new_words = [] 430 | if self.evaluate == True: 431 | new_sentences_current, new_words_current, filter_count = self.__generate_syn_sentences_single(sent, NEs) 432 | for index, new_sentence in enumerate(new_sentences_current): 433 | total_filtered += filter_count 434 | new_indexes.append(index) 435 | new_sentences.append(new_sentence) 436 | new_words.append(new_words_current[index]) 437 | new_indexes_list.append(new_indexes) 438 | new_sentences_list.append(new_sentences) 439 | new_words_list.append(new_words) 440 | continue 441 | new_sentences_current, new_words_current = self.__generate_syn_sentences_single(sent, NEs) 442 | for index, new_sentence in enumerate(new_sentences_current): 443 | new_indexes.append(index) 444 | new_sentences.append(new_sentence) 445 | new_words.append(new_words_current[index]) 446 | new_indexes_list.append(new_indexes) 447 | new_sentences_list.append(new_sentences) 448 | new_words_list.append(new_words) 449 | result_sentences = [] 450 | for indexes in itertools.product(*new_indexes_list): 451 | result_sentence_list = [] 452 | result_new_words_list = [] 453 | for pos, index in enumerate(indexes): 454 | result_sentence_list.append(new_sentences_list[pos][index].lstrip(" ").rstrip(" ")) 455 | result_new_words_list.extend(new_words_list[pos][index]) 456 | result_sentence = " ".join(result_sentence_list) 457 | result_sentences.append((result_sentence, result_new_words_list)) 458 | sim_number = len(result_sentences) 459 | ori_number = sim_number + total_filtered 460 | # naturalness filter 461 | return_sentences = [] 462 | if self.filter_flag: 463 | ori_score = self.__get_AEON_score(sentence) 464 | for (sent, new_words) in result_sentences: 465 | new_score = self.__get_AEON_score(sent) 466 | if new_score >= ori_score - self.AEON_threshold: 467 | return_sentences.append((sent, new_words)) 468 | else: 469 | return_sentences = result_sentences 470 | aeon_number = len(return_sentences) 471 | 472 | # return result 473 | changed = False 474 | if len(return_sentences) == 0 or \ 475 | (len(return_sentences) == 1 and return_sentences[0][0].replace(" ","") == sentence.replace(" ", "")): 476 | changed = False 477 | if self.evaluate == True: 478 | return [ori_number, sim_number, aeon_number] 479 | return (changed, []) 480 | else: 481 | changed = True 482 | if self.evaluate == True: 483 | return [ori_number, sim_number, aeon_number] 484 | return (changed, return_sentences) 485 | 486 | def show(self, sentence): 487 | self.nlp = StanfordCoreNLP( 488 | r'$HOME$/NER_MRs/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 489 | parsingTree = Tree.fromstring(self.nlp.parse(sentence)) 490 | parsingTree.pretty_print() 491 | 492 | if __name__ == "__main__": 493 | mr_synonym = MR_synonym() 494 | mr_synonym.MR_init() 495 | print(mr_synonym.generate_syn_sentences("In July , the former minister David Davis was one of nine senior Conservatives who wrote a letter to then Culture Secretary Nadine Dorries , warning the legal but harmful provision posed a threat to free speech .", ['David Davis', 'Conservatives', 'Nadine Dorries'])) 496 | mr_synonym.MR_close() -------------------------------------------------------------------------------- /TIN_Test/apis.py: -------------------------------------------------------------------------------- 1 | import os 2 | import time 3 | import stanza 4 | import pdb 5 | from azure.ai.textanalytics import TextAnalyticsClient 6 | from azure.core.credentials import AzureKeyCredential 7 | from flair.data import Sentence 8 | from flair.models import SequenceTagger 9 | import boto3 10 | from transformers import AutoTokenizer, AutoModelForTokenClassification 11 | from transformers import pipeline 12 | from collections import defaultdict 13 | import torch 14 | 15 | # you need to set the azure key and endpoint by yourself 16 | key = "YourKey" 17 | endpoint = "https://nerazure.cognitiveservices.azure.com/" 18 | 19 | class APIS: 20 | def __authenticate_client(self): 21 | ta_credential = AzureKeyCredential(key) 22 | text_analytics_client = TextAnalyticsClient( 23 | endpoint=endpoint, 24 | credential=ta_credential) 25 | return text_analytics_client 26 | 27 | def azure(self, sentence): 28 | # create a word list (splited by whitespaces) 29 | client = self.__authenticate_client() 30 | word_list = sentence.split(" ") 31 | 32 | connected = False 33 | while not connected: 34 | print("waiting !!") 35 | time.sleep(0.01) 36 | try: 37 | connected = True 38 | sentence_list = [] 39 | sentence_list.append(sentence) 40 | result = client.recognize_entities(documents = sentence_list)[0] 41 | 42 | except Exception as err: 43 | connected = False 44 | print("Encountered exception. {}".format(err)) 45 | 46 | temp = {} 47 | entity_pos_info = {} 48 | for entity in result.entities: 49 | entity_pos_info[int(entity.offset)] = entity 50 | temp[entity.text] = entity.category 51 | # map the words in the word_list to the character offset, map word index to string offset (begining of the word) 52 | charOffset2WordID = {} 53 | current_offset = 0 54 | for index, word in enumerate(word_list): 55 | # add up the offset if whitespace in the string 56 | while current_offset 0: 184 | # append previous entity 185 | if label_str == "": 186 | label_str = last_label_str 187 | name_str.lstrip(" ") 188 | entities.append([name_str, label_str, current_index_list]) 189 | # reset name string 190 | name_str = "" 191 | current_index_list = [] 192 | if label_list[j] == 1: 193 | label_str = "MISC" 194 | elif label_list[j] == 3: 195 | label_str = "PER" 196 | elif label_list[j] == 5: 197 | label_str = "ORG" 198 | else: 199 | label_str = "LOC" 200 | current_index_list.append(j) 201 | name_str += word_list[j] 202 | else: # start with I 203 | if last_zero == True and name_str != "": 204 | # append previous entity 205 | if label_str == "": 206 | label_str = last_label_str 207 | entities.append([name_str, label_str, current_index_list]) 208 | # reset name string 209 | name_str = "" 210 | current_index_list = [] 211 | if last_zero == False and j > 0: name_str += " " 212 | current_index_list.append(j) 213 | name_str += word_list[j] 214 | if label_list[j] == 2: 215 | last_label_str = "MISC" 216 | elif label_list[j] == 4: 217 | last_label_str = "PER" 218 | elif label_list[j] == 6: 219 | last_label_str = "ORG" 220 | else: 221 | last_label_str = "LOC" 222 | # print(name_str) 223 | last_zero = False 224 | if len(name_str.strip()) > 0: 225 | if label_str == "": entities.append([name_str, last_label_str, current_index_list]) 226 | else: entities.append([name_str, label_str, current_index_list]) 227 | return entities 228 | 229 | def bert_ner(self, sentence): 230 | 231 | tokenizer = AutoTokenizer.from_pretrained("dslim/bert-base-NER") 232 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 233 | 234 | model = AutoModelForTokenClassification.from_pretrained("dslim/bert-base-NER") 235 | # model = model.to(device) 236 | # tokenizer = tokenizer.to(device) 237 | nlp = pipeline("ner", model=model, tokenizer=tokenizer) 238 | tokens = tokenizer.convert_ids_to_tokens(tokenizer(sentence)["input_ids"]) 239 | 240 | word_list = sentence.split(" ") 241 | # pdb.set_trace() 242 | ner_results = nlp(sentence) 243 | result = {} 244 | tokens_to_words, words_to_tokens = self.__tokens_and_words_map(tokens, word_list) 245 | word_list_labels = [0] * len(word_list) 246 | ner_tags = {'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4, 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8} 247 | for ner_dict in ner_results: 248 | token_index = ner_dict["index"] 249 | word_index = tokens_to_words[token_index] 250 | current_label = ner_dict["entity"] 251 | word_list_labels[word_index] = ner_tags[current_label] 252 | result["sentence"] = word_list 253 | result["entity"] = self.__processNerLabel(word_list_labels, word_list) 254 | return result 255 | 256 | 257 | def stanza_ner(self, sentence, lang="en", processors= "tokenize,ner"): 258 | nlp = stanza.Pipeline(lang=lang, processors=processors) 259 | doc = nlp(sentence) 260 | 261 | 262 | if __name__ == "__main__": 263 | apis = APIS() 264 | print(apis.bert_ner("In July , the former minister David Davis was one of nine senior Conservatives who wrote a letter to then Culture Secretary Nadine Dorries , warning the legal but harmful provision posed a threat to free speech .")) 265 | 266 | 267 | -------------------------------------------------------------------------------- /TIN_Test/json2text.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import pdb 4 | 5 | 6 | for file in os.listdir('.'): 7 | temp_susp = [] 8 | temp_list = [] 9 | if file[-5:] == '.json' and ( ("AEON_suspicious" or "NOAEON_suspicious") in file ): 10 | with open(file, 'r' ) as f: 11 | temp_susp = json.load(f) 12 | wf = open(file[:-5] + '.txt', "w", encoding='UTF-8') 13 | for idx, ele in enumerate(temp_susp): 14 | wf.write("====================Suspicious Issues====================" + "\n") 15 | wf.write( "original: " + ele['original']['sentence'] +"\n") 16 | wf.write( "entities: " + str(ele['original']['entity']) +"\n") 17 | wf.write( "synthetic: " + ele['new']['sentence'] + "\n" ) 18 | wf.write( "entities " + str(ele['new']['entity']) +"\n" ) 19 | wf.write("=================================================" + "\n") -------------------------------------------------------------------------------- /TIN_Test/mr_perturb.py: -------------------------------------------------------------------------------- 1 | import os 2 | from Transformations.MR_BERT import * 3 | from Transformations.MR_structure import * 4 | from Transformations.MR_synonym import * 5 | from Transformations.MR_intra_shuffle import * 6 | from flair.models import SequenceTagger 7 | from apis import * 8 | import random 9 | import json 10 | import pdb 11 | import ipdb 12 | import copy 13 | from nltk.tree import * 14 | import argparse 15 | import sys 16 | sys.path.insert(1, 'AEON/utils') 17 | import pathlib 18 | 19 | import re 20 | alphabets= "([A-Za-z])" 21 | prefixes = "(Mr|St|Mrs|Ms|Dr)[.]" 22 | suffixes = "(Inc|Ltd|Jr|Sr|Co)" 23 | starters = "(Mr|Mrs|Ms|Dr|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)" 24 | acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)" 25 | websites = "[.](com|net|org|io|gov)" 26 | digits = "([0-9])" 27 | 28 | def split_into_sentences(text): 29 | text = " " + text + " " 30 | text = text.replace("\n"," ") 31 | text = re.sub(prefixes,"\\1",text) 32 | text = re.sub(websites,"\\1",text) 33 | text = re.sub(digits + "[.]" + digits,"\\1\\2",text) 34 | if "..." in text: text = text.replace("...","") 35 | if "Ph.D" in text: text = text.replace("Ph.D.","PhD") 36 | text = re.sub("\s" + alphabets + "[.] "," \\1 ",text) 37 | text = re.sub(acronyms+" "+starters,"\\1 \\2",text) 38 | text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]","\\1\\2\\3",text) 39 | text = re.sub(alphabets + "[.]" + alphabets + "[.]","\\1\\2",text) 40 | text = re.sub(" "+suffixes+"[.] "+starters," \\1 \\2",text) 41 | text = re.sub(" "+suffixes+"[.]"," \\1",text) 42 | text = re.sub(" " + alphabets + "[.]"," \\1",text) 43 | if "”" in text: text = text.replace(".”","”.") 44 | if "\"" in text: text = text.replace(".\"","\".") 45 | if "!" in text: text = text.replace("!\"","\"!") 46 | if "?" in text: text = text.replace("?\"","\"?") 47 | text = text.replace(".",".") 48 | text = text.replace("?","?") 49 | text = text.replace("!","!") 50 | text = text.replace("",".") 51 | sentences = text.split("") 52 | return_list = [] 53 | for s in sentences: 54 | if len(s.replace(" ", "")) != 0: 55 | return_list.append(s) 56 | return return_list 57 | 58 | def NLTK_tree_get_sentence(leaves): 59 | for i in range(0, len(leaves)): 60 | if leaves[i] == "-LRB-": 61 | leaves[i] = "(" 62 | elif leaves[i] == "-RRB-": 63 | leaves[i] = ")" 64 | return_string = " ".join(leaves) 65 | return return_string 66 | 67 | def format_to_string(result_dict): 68 | # convert the result dict to inputable string format 69 | sentence_str = " ".join(result_dict["sentence"]) 70 | entities = result_dict["entity"] 71 | NEs = [] 72 | for entity in entities: 73 | NEs.append(entity[0]) 74 | return sentence_str, NEs 75 | 76 | def format_to_nameTypeList(result_dict): 77 | # convert the result dict to [(name, type)] format 78 | entities = result_dict["entity"] 79 | entities = sorted(entities,key=lambda l:l[2][0]) 80 | ent_list = [] 81 | for entity in entities: 82 | ent_list.append((entity[0], entity[1])) 83 | return ent_list 84 | 85 | def check_mr_intersection_new_words(sentence, pert_sentence, ori_prediction, pert_prediction): 86 | sus_words = set() 87 | ori_dict = {} 88 | pert_dict = {} 89 | for entity in ori_prediction: 90 | if entity[0] not in ori_dict.keys(): 91 | ori_dict[entity[0]] = [] 92 | ori_dict[entity[0]].append(entity[1]) 93 | for entity in pert_prediction: 94 | if entity[0] not in pert_dict.keys(): 95 | pert_dict[entity[0]] = [] 96 | pert_dict[entity[0]].append(entity[1]) 97 | for item in ori_dict.items(): 98 | if item[0] not in pert_sentence: continue 99 | if item[0] not in pert_dict.keys(): 100 | sus_words.add(item[0]) 101 | elif pert_dict[item[0]] != item[1]: 102 | sus_words.add(item[0]) 103 | for item in pert_dict.items(): 104 | if item[0] not in sentence: continue 105 | if item[0] not in ori_dict.keys(): 106 | sus_words.add(item[0]) 107 | elif ori_dict[item[0]] != item[1]: 108 | sus_words.add(item[0]) 109 | result = [] 110 | for x in sus_words: result.append(x) 111 | return len(result) > 0 112 | 113 | 114 | # Function implemented by cav 115 | def check_mr_new_words(list1, list2, new_words): 116 | sus_flag = False 117 | # missing classfication is not tolerable 118 | set1 = set(list1) 119 | set2 = set(list2) 120 | diff_2to1 = set2 - set1 121 | for ele in diff_2to1: 122 | if not any(ele[0] in new_word for new_word in new_words): 123 | sus_flag = True 124 | 125 | if (set2-diff_2to1) != set1: 126 | sus_flag == True 127 | 128 | if sus_flag: 129 | return False 130 | else: 131 | return True 132 | 133 | def check_mr_equal(list1, list2): 134 | # we need to deal with some situations caused by punctuations 135 | list1_check = set() 136 | list2_check = set() 137 | for item in list1: 138 | list1_check.add((item[0].replace(" ", ""), item[1])) 139 | for item in list2: 140 | list2_check.add((item[0].replace(" ", ""), item[1])) 141 | return list1_check == list2_check 142 | 143 | def run_flair(sentences, tag = "flair/ner-english-large", sava_name="bbc_flair_original.json"): 144 | apis = APIS() 145 | # step1: run the dataset on apis 146 | results = [] 147 | tagger = SequenceTagger.load(tag) 148 | nlp = StanfordCoreNLP( 149 | str(pathlib.Path().absolute())+'/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 150 | # step1: run the dataset on apis 151 | results = [] 152 | for sentence in sentences: 153 | sent = NLTK_tree_get_sentence(Tree.fromstring(nlp.parse(sentence)).leaves()) 154 | print(sent) 155 | # pdb.set_trace() 156 | results.append(apis.flair(sent, tagger)) 157 | nlp.close() 158 | 159 | with open(sava_name, "w", encoding="utf-8") as f: 160 | json.dump(results, f, ensure_ascii=False) 161 | 162 | 163 | def run_azure(sentences, save_name="bbc_azure_original.json"): 164 | apis = APIS() 165 | # step1: run the dataset on apis 166 | results = [] 167 | nlp = StanfordCoreNLP( 168 | str(pathlib.Path().absolute())+'/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 169 | # step1: run the dataset on apis 170 | results = [] 171 | for sentence in sentences: 172 | sent = NLTK_tree_get_sentence(Tree.fromstring(nlp.parse(sentence)).leaves()) 173 | print(sent) 174 | results.append(apis.azure(sent)) 175 | nlp.close() 176 | 177 | with open(save_name, "w", encoding="utf-8") as f: 178 | json.dump(results, f, ensure_ascii=False) 179 | 180 | def run_aws(sentences, save_name="bbc_aws_original.json"): 181 | apis = APIS() 182 | # step1: run the dataset on apis 183 | results = [] 184 | nlp = StanfordCoreNLP( 185 | str(pathlib.Path().absolute())+'/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 186 | # step1: run the dataset on apis 187 | results = [] 188 | for sentence in sentences: 189 | sent = NLTK_tree_get_sentence(Tree.fromstring(nlp.parse(sentence)).leaves()) 190 | print(sent) 191 | results.append(apis.aws(sent)) 192 | nlp.close() 193 | 194 | with open(save_name, "w", encoding="utf-8") as f: 195 | json.dump(results, f, ensure_ascii=False) 196 | 197 | def run_bert_ner(sentences, save_name="conll_bert_ner_original.json"): 198 | apis = APIS() 199 | # step1: run the dataset on apis 200 | results = [] 201 | nlp = StanfordCoreNLP( 202 | str(pathlib.Path().absolute())+'/corenlp/stanford-corenlp-latest/stanford-corenlp-4.4.0') 203 | # step1: run the dataset on apis 204 | results = [] 205 | for sentence in sentences: 206 | print(sentence) 207 | sents = split_into_sentences(sentence) 208 | new_sents = [] 209 | for sent in sents: 210 | new_sent = NLTK_tree_get_sentence(Tree.fromstring(nlp.parse(sent)).leaves()) 211 | new_sents.append(new_sent) 212 | # sent = NLTK_tree_get_sentence(Tree.fromstring(nlp.parse(sentence)).leaves()) 213 | result_sent = " ".join(new_sents) 214 | print(result_sent) 215 | results.append(apis.bert_ner(result_sent)) 216 | nlp.close() 217 | 218 | with open(save_name, "w", encoding="utf-8") as f: 219 | json.dump(results, f, ensure_ascii=False) 220 | 221 | def run_MR_BERT(file_name, aeon_flag, aeon_threshold=0.01): 222 | # original_results = json.load(open(file_name))[:100] 223 | original_results = json.load(open(file_name)) 224 | mr_bert = MR_BERT() 225 | mr_bert.MR_init(aeon_flag=aeon_flag, aeon_threshold= aeon_threshold) 226 | mr_bert_result = [] 227 | for index, sentence in enumerate(original_results): 228 | print(str(file_name), index) 229 | print(sentence) 230 | sentence_str, NEs = format_to_string(sentence) 231 | new_sentences = mr_bert.generate_sentences(sentence_str, NEs) 232 | print(new_sentences) 233 | mr_bert_result.append(new_sentences) 234 | if aeon_flag == True: 235 | aeon_prefix = "AEON_" 236 | else: 237 | aeon_prefix = "NOAEON_" 238 | with open(aeon_prefix + file_name.rstrip("original.json") + "MR_BERT.json", "w", encoding="utf-8") as f: 239 | json.dump(mr_bert_result, f, ensure_ascii=False) 240 | 241 | def run_MR_structure(file_name, aeon_flag, aeon_threshold=0.02): 242 | # original_results = json.load(open(file_name))[:100] 243 | original_results = json.load(open(file_name)) 244 | mr_structure = MR_structure() 245 | mr_structure.MR_init(aeon_flag=aeon_flag, aeon_threshold= aeon_threshold) 246 | mr_structure_result = [] 247 | for index, sentence in enumerate(original_results): 248 | # print(index) 249 | print(str(file_name), index) 250 | sentence_str, NEs = format_to_string(sentence) 251 | new_sentences = mr_structure.declarative_2_interrogative(sentence_str, NEs) 252 | mr_structure_result.append(new_sentences) 253 | mr_structure.MR_close() 254 | if aeon_flag == True: 255 | aeon_prefix = "AEON_" 256 | else: 257 | aeon_prefix = "NOAEON_" 258 | with open( aeon_prefix + file_name.rstrip("original.json") + "MR_structure.json", "w", encoding="utf-8") as f: 259 | json.dump(mr_structure_result, f, ensure_ascii=False) 260 | 261 | def run_MR_synonym(file_name, aeon_flag, aeon_threshold=0.01): 262 | # original_results = json.load(open(file_name))[:100] 263 | # pdb.set_trace() 264 | original_results = json.load(open(file_name)) 265 | mr_synonym = MR_synonym() 266 | mr_synonym.MR_init(aeon_flag=aeon_flag, aeon_threshold= aeon_threshold) 267 | mr_synonym_result = [] 268 | for index, sentence in enumerate(original_results): 269 | # print(index) 270 | print(str(file_name), index) 271 | # if index == 170: 272 | # pdb.set_trace() 273 | sentence_str, NEs = format_to_string(sentence) 274 | new_sentences = mr_synonym.generate_syn_sentences(sentence_str, NEs) 275 | mr_synonym_result.append(new_sentences) 276 | mr_synonym.MR_close() 277 | if aeon_flag == True: 278 | aeon_prefix = "AEON_" 279 | else: 280 | aeon_prefix = "NOAEON_" 281 | with open(aeon_prefix + file_name.rstrip("original.json") + "MR_synonym.json", "w", encoding="utf-8") as f: 282 | json.dump(mr_synonym_result, f, ensure_ascii=False) 283 | 284 | def run_MR_intra_shuffle(file_name, aeon_flag, aeon_threshold=0.01): 285 | # original_results = json.load(open(file_name))[:100] 286 | original_results = json.load(open(file_name)) 287 | mr_intra_shuffle = MR_intra_shuffle() 288 | mr_intra_shuffle.MR_init(aeon_flag=aeon_flag, aeon_threshold= aeon_threshold) 289 | mr_intra_shuffle_result = [] 290 | original_sentences = json.load(open("./Data/bbc.json")) 291 | 292 | for index, sentence in enumerate(original_results): 293 | # print(index) 294 | print(str(file_name), index) 295 | word_list = sentence["sentence"] 296 | entity_list = sentence["entity"] 297 | new_sentences = mr_intra_shuffle.intra_sentence_entity_shuffle(word_list, entity_list) 298 | mr_intra_shuffle_result.append(new_sentences) 299 | if aeon_flag == True: 300 | aeon_prefix = "AEON_" 301 | else: 302 | aeon_prefix = "NOAEON_" 303 | with open( aeon_prefix + file_name.rstrip("original.json") + "MR_intra_shuffle.json", "w", encoding="utf-8") as f: 304 | json.dump(mr_intra_shuffle_result, f, ensure_ascii=False) 305 | 306 | def run_test_mr_bert(original_file, new_file, api_type = "flair", tag="flair/ner-english-large", aeon_flag=True): 307 | # pdb.set_trace() 308 | # original_outputs = json.load(open(original_file))[:100] 309 | if aeon_flag == True: 310 | aeon_prefix = "AEON_" 311 | else: 312 | aeon_prefix = "NOAEON_" 313 | f=open(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_BERT.out", "w", encoding = 'utf-8') 314 | original_outputs = json.load(open(original_file)) 315 | new_outputs = json.load(open(new_file)) 316 | if len(original_outputs) != len(new_outputs): 317 | raise ValueError("The original output does not match the perturbed output") 318 | if api_type == "flair": 319 | tagger = SequenceTagger.load(tag) 320 | apis = APIS() 321 | sus_list = [] 322 | for i in range(len(original_outputs)): 323 | print(i) 324 | sus_info = {} 325 | suspicious = False 326 | ori_info_dict = {} 327 | ori_info_dict["sentence"] = " ".join(original_outputs[i]["sentence"]) 328 | original_prediction = format_to_nameTypeList(original_outputs[i]) 329 | ori_info_dict["entity"] = original_prediction 330 | # print(new_outputs[i]) 331 | (changed, current_new_output) = new_outputs[i] 332 | # print(current_new_output) 333 | # pdb.set_trace() 334 | if not changed: 335 | print("Unchanged:") 336 | continue 337 | else: 338 | print("changed") 339 | for _current_new_output in current_new_output: 340 | new_info_dict = {} 341 | new_sentence = _current_new_output[0] 342 | new_words = _current_new_output[1] 343 | if api_type == "flair": 344 | new_prediction = apis.flair(new_sentence, tagger) 345 | elif api_type == "google": 346 | new_prediction = apis.google(new_sentence) 347 | elif api_type == "azure": 348 | new_prediction = apis.azure(new_sentence) 349 | elif api_type == "aws": 350 | new_prediction = apis.aws(new_sentence) 351 | elif api_type == "bert_ner": 352 | new_prediction = apis.bert_ner(new_sentence) 353 | else: 354 | raise ValueError("No such API") 355 | new_info_dict["sentence"] = " ".join(new_prediction["sentence"]) 356 | new_info_dict["entity"] = format_to_nameTypeList(new_prediction) 357 | # print("original:", format_to_nameTypeList(original_outputs[i])) 358 | # print("new: ", format_to_nameTypeList(new_prediction)) 359 | # if not check_mr_new_words(original_prediction, format_to_nameTypeList(new_prediction), new_words): 360 | # pdb.set_trace() 361 | if check_mr_intersection_new_words(ori_info_dict["sentence"], new_info_dict["sentence"], original_prediction, format_to_nameTypeList(new_prediction)): 362 | print("suspicious!") 363 | suspicious = True 364 | sus_info["index"] = i 365 | sus_info["original"] = ori_info_dict 366 | sus_info["new"] = new_info_dict 367 | sus_list.append(copy.deepcopy(sus_info)) 368 | 369 | print("====================Original====================", file = f) 370 | print("sentence: ", ori_info_dict['sentence'], file = f) 371 | print("orientity: ", ori_info_dict['entity'], file = f) 372 | print("====================Synthesized====================", file = f) 373 | print("sentence: ", new_info_dict['sentence'] , file = f) 374 | print("orientity: ", new_info_dict['entity'], file = f) 375 | print(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_BERT.json") 376 | with open( aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_BERT.json", "w", encoding="utf-8") as f: 377 | json.dump(sus_list, f, ensure_ascii=False) 378 | 379 | def run_test_mr_structure(original_file, new_file, api_type = "flair", tag = "flair/ner-english-large", aeon_flag=True): 380 | # original_outputs = json.load(open(original_file))[:100] 381 | # pdb.set_trace() 382 | if aeon_flag == True: 383 | aeon_prefix = "AEON_" 384 | else: 385 | aeon_prefix = "NOAEON_" 386 | f=open(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_Structure.out", "w", encoding = 'utf-8') 387 | original_outputs = json.load(open(original_file)) 388 | new_outputs = json.load(open(new_file)) 389 | if len(original_outputs) != len(new_outputs): 390 | raise ValueError("The original output does not match the perturbed output") 391 | apis = APIS() 392 | if api_type == "flair": 393 | tagger = SequenceTagger.load(tag) 394 | sus_list = [] 395 | for i in range(len(original_outputs)): 396 | print(i) 397 | sus_info = {} 398 | ori_info_dict = {} 399 | ori_info_dict["sentence"] = " ".join(original_outputs[i]["sentence"]) 400 | original_prediction = format_to_nameTypeList(original_outputs[i]) 401 | ori_info_dict["entity"] = original_prediction 402 | print("current: ", new_outputs[i]) 403 | (changed, current_new_output) = new_outputs[i] 404 | if not changed: 405 | continue 406 | new_info_dict = {} 407 | # print(current_new_output) 408 | new_sentence = current_new_output 409 | if api_type == "flair": 410 | new_prediction = apis.flair(new_sentence, tagger) 411 | elif api_type == "google": 412 | new_prediction = apis.google(new_sentence) 413 | elif api_type == "azure": 414 | new_prediction = apis.azure(new_sentence) 415 | elif api_type == "aws": 416 | new_prediction = apis.aws(new_sentence) 417 | elif api_type == "bert_ner": 418 | new_prediction = apis.bert_ner(new_sentence) 419 | else: 420 | raise ValueError("No such API") 421 | new_info_dict["sentence"] = " ".join(new_prediction["sentence"]) 422 | new_info_dict["entity"] = format_to_nameTypeList(new_prediction) 423 | # print("original:", format_to_nameTypeList(original_outputs[i])) 424 | # print("new: ", format_to_nameTypeList(new_prediction)) 425 | print("check: ", check_mr_equal(original_prediction, format_to_nameTypeList(new_prediction))) 426 | if not check_mr_equal(original_prediction, format_to_nameTypeList(new_prediction)): 427 | print("suspicious!") 428 | suspicious = True 429 | sus_info["index"] = i 430 | sus_info["original"] = ori_info_dict 431 | sus_info["new"] = new_info_dict 432 | sus_list.append(copy.deepcopy(sus_info)) 433 | 434 | print("====================Original====================", file = f) 435 | print("sentence: ", ori_info_dict['sentence'], file = f) 436 | print("orientity: ", ori_info_dict['entity'], file = f) 437 | print("====================Synthesized====================", file = f) 438 | print("sentence: ", new_info_dict['sentence'] , file = f) 439 | print("orientity: ", new_info_dict['entity'], file = f) 440 | print(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_structure.json") 441 | with open( aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_structure.json", "w", encoding="utf-8") as f: 442 | json.dump(sus_list, f, ensure_ascii=False) 443 | 444 | 445 | def run_test_mr_synonym(original_file, new_file, api_type = "flair", tag = "flair/ner-english-large", aeon_flag = True): 446 | if aeon_flag == True: 447 | aeon_prefix = "AEON_" 448 | else: 449 | aeon_prefix = "NOAEON_" 450 | f=open(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_synonym.out", "w", encoding = 'utf-8') 451 | original_outputs = json.load(open(original_file)) 452 | # original_outputs = json.load(open(original_file))[:100] 453 | new_outputs = json.load(open(new_file)) 454 | if len(original_outputs) != len(new_outputs): 455 | raise ValueError("The original output does not match the perturbed output") 456 | apis = APIS() 457 | if api_type == "flair": 458 | tagger = SequenceTagger.load(tag) 459 | sus_list = [] 460 | for i in range(len(original_outputs)): 461 | print(i) 462 | sus_info = {} 463 | suspicious = False 464 | ori_info_dict = {} 465 | ori_info_dict["sentence"] = " ".join(original_outputs[i]["sentence"]) 466 | original_prediction = format_to_nameTypeList(original_outputs[i]) 467 | ori_info_dict["entity"] = original_prediction 468 | (changed, current_new_output) = new_outputs[i] 469 | if not changed: 470 | continue 471 | for _current_new_output in current_new_output: 472 | new_info_dict = {} 473 | new_sentence = _current_new_output[0] 474 | new_words = _current_new_output[1] 475 | if api_type == "flair": 476 | new_prediction = apis.flair(new_sentence, tagger) 477 | elif api_type == "google": 478 | new_prediction = apis.google(new_sentence) 479 | elif api_type == "azure": 480 | new_prediction = apis.azure(new_sentence) 481 | elif api_type == "aws": 482 | new_prediction = apis.aws(new_sentence) 483 | elif api_type == "bert_ner": 484 | new_prediction = apis.bert_ner(new_sentence) 485 | else: 486 | raise ValueError("No such API") 487 | new_info_dict["sentence"] = " ".join(new_prediction["sentence"]) 488 | new_info_dict["entity"] = format_to_nameTypeList(new_prediction) 489 | if check_mr_intersection_new_words(ori_info_dict["sentence"], new_info_dict["sentence"], original_prediction, format_to_nameTypeList(new_prediction)): 490 | print("suspicious!") 491 | # print(new_words) 492 | # print("new_predition: ", new_prediction) 493 | suspicious = True 494 | sus_info["index"] = i 495 | sus_info["original"] = ori_info_dict 496 | sus_info["new"] = new_info_dict 497 | sus_list.append(copy.deepcopy(sus_info)) 498 | 499 | print("====================Original====================", file = f) 500 | print("sentence: ", ori_info_dict['sentence'], file = f) 501 | print("orientity: ", ori_info_dict['entity'], file = f) 502 | print("====================Synthesized====================", file = f) 503 | print("sentence: ", new_info_dict['sentence'] , file = f) 504 | print("orientity: ", new_info_dict['entity'], file = f) 505 | print(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_synonym.json") 506 | with open( aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_synonym.json", "w", encoding="utf-8") as f: 507 | json.dump(sus_list, f, ensure_ascii=False) 508 | 509 | def run_test_mr_intra_shuffle(original_file, new_file, api_type = "flair", tag = "flair/ner-english-large", aeon_flag = True): 510 | if aeon_flag == True: 511 | aeon_prefix = "AEON_" 512 | else: 513 | aeon_prefix = "NOAEON_" 514 | f=open(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_intra_shuffle.out", "w", encoding = 'utf-8') 515 | original_outputs = json.load(open(original_file)) 516 | # original_outputs = json.load(open(original_file))[:100] 517 | new_outputs = json.load(open(new_file)) 518 | if len(original_outputs) != len(new_outputs): 519 | raise ValueError("The original output does not match the perturbed output") 520 | apis = APIS() 521 | if api_type == "flair": 522 | tagger = SequenceTagger.load(tag) 523 | sus_list = [] 524 | for i in range(len(original_outputs)): 525 | print(i) 526 | sus_info = {} 527 | suspicious = False 528 | ori_info_dict = {} 529 | ori_info_dict["sentence"] = " ".join(original_outputs[i]["sentence"]) 530 | original_prediction = format_to_nameTypeList(original_outputs[i]) 531 | ori_info_dict["entity"] = original_prediction 532 | (changed, current_new_output) = new_outputs[i] 533 | if not changed: 534 | continue 535 | for _current_new_output in current_new_output: 536 | new_info_dict = {} 537 | new_sentence = _current_new_output 538 | print(new_sentence) 539 | if api_type == "flair": 540 | new_prediction = apis.flair(new_sentence, tagger) 541 | elif api_type == "google": 542 | new_prediction = apis.google(new_sentence) 543 | elif api_type == "azure": 544 | new_prediction = apis.azure(new_sentence) 545 | elif api_type == "aws": 546 | new_prediction = apis.aws(new_sentence) 547 | elif api_type == "bert_ner": 548 | new_prediction = apis.bert_ner(new_sentence) 549 | else: 550 | raise ValueError("No such API") 551 | new_info_dict["sentence"] = " ".join(new_prediction["sentence"]) 552 | new_info_dict["entity"] = format_to_nameTypeList(new_prediction) 553 | # print("original:", format_to_nameTypeList(original_outputs[i])) 554 | # print("new: ", format_to_nameTypeList(new_prediction)) 555 | if not check_mr_equal(original_prediction, format_to_nameTypeList(new_prediction)): 556 | print("suspicious!") 557 | suspicious = True 558 | sus_info["index"] = i 559 | sus_info["original"] = ori_info_dict 560 | sus_info["new"] = new_info_dict 561 | sus_list.append(copy.deepcopy(sus_info)) 562 | if aeon_flag == True: 563 | aeon_prefix = "AEON_" 564 | else: 565 | aeon_prefix = "NOAEON_" 566 | 567 | print("====================Original====================", file = f) 568 | print("sentence: ", ori_info_dict['sentence'], file = f) 569 | print("orientity: ", ori_info_dict['entity'], file = f) 570 | print("====================Synthesized====================", file = f) 571 | print("sentence: ", new_info_dict['sentence'] , file = f) 572 | print("orientity: ", new_info_dict['entity'], file = f) 573 | print(aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_intra_shuffle.json") 574 | with open( aeon_prefix + "suspicious_" + original_file.rstrip("original.json") + "MR_intra_shuffle.json", "w", encoding="utf-8") as f: 575 | json.dump(sus_list, f, ensure_ascii=False) 576 | 577 | 578 | if __name__ == "__main__": 579 | parser = argparse.ArgumentParser(description="process the hyper-parameters") 580 | parser.add_argument( "--aeon_flag", type = bool, default=True ) 581 | parser.add_argument( "--api_type", type = str, default="flair_conll" ) 582 | parser.add_argument( "--dataset_path", type = str, default="./Data/bbc.json" ) 583 | parser.add_argument( "--down_size", type = bool, default=False ) 584 | args = parser.parse_args() 585 | print(args.api_type, args.aeon_flag, args.dataset_path, args.down_size) 586 | aeon_flag = args.aeon_flag 587 | api_type = args.api_type 588 | dataset_path = args.dataset_path 589 | down_size = args.down_size 590 | 591 | if api_type == "flair_conll": 592 | original_sentences = json.load(open(dataset_path)) 593 | if down_size == True: 594 | original_sentences = random.Random(14).sample(original_sentences, 100) 595 | run_flair(original_sentences, tag="flair/ner-english-large", sava_name="bbc_flair_original.json") 596 | file_name = "bbc_flair_original.json" # original prediction file 597 | 598 | run_MR_BERT(file_name, aeon_flag, aeon_threshold=0.01) 599 | run_MR_structure(file_name, aeon_flag) 600 | run_MR_intra_shuffle(file_name, aeon_flag) 601 | run_MR_synonym(file_name, aeon_flag) 602 | original_file = file_name 603 | 604 | run_test_mr_bert(original_file, "AEON_bbc_flair_MR_BERT.json", api_type = "flair", tag = "flair/ner-english-large", aeon_flag=True) 605 | run_test_mr_structure(original_file, "AEON_bbc_flair_MR_structure.json", api_type = "flair", tag = "flair/ner-english-large", aeon_flag=True) 606 | run_test_mr_intra_shuffle(original_file, "AEON_bbc_flair_MR_intra_shuffle.json", api_type = "flair", tag = "flair/ner-english-large", aeon_flag=True) 607 | run_test_mr_synonym(original_file, "AEON_bbc_flair_MR_synonym.json", api_type = "flair", tag = "flair/ner-english-large", aeon_flag=True) 608 | 609 | elif api_type == "flair_ontonotes": 610 | original_sentences = json.load(open(dataset_path)) 611 | if down_size == True: 612 | original_sentences = random.Random(14).sample(original_sentences, 100) 613 | run_flair(original_sentences, tag="flair/ner-english-ontonotes-large", sava_name="bbc_flair_OntoNotes_original.json") 614 | file_name = "bbc_flair_OntoNotes_original.json" # original prediction file 615 | 616 | run_MR_BERT(file_name, aeon_flag, aeon_threshold=0) 617 | run_MR_structure(file_name, aeon_flag, aeon_threshold=0) 618 | run_MR_intra_shuffle(file_name, aeon_flag, aeon_threshold=0) 619 | run_MR_synonym(file_name, aeon_flag, aeon_threshold=0) 620 | original_file = file_name 621 | 622 | run_test_mr_bert(original_file, "AEON_bbc_flair_OntoNotes_MR_BERT.json", api_type = "flair", tag="flair/ner-english-ontonotes-large", aeon_flag=True) 623 | run_test_mr_structure(original_file, "AEON_bbc_flair_OntoNotes_MR_structure.json", api_type = "flair", tag="flair/ner-english-ontonotes-large", aeon_flag=True) 624 | run_test_mr_intra_shuffle(original_file, "AEON_bbc_flair_OntoNotes_MR_intra_shuffle.json", api_type = "flair", tag="flair/ner-english-ontonotes-large", aeon_flag=True) 625 | run_test_mr_synonym(original_file, "AEON_bbc_flair_OntoNotes_MR_synonym.json", api_type = "flair", tag="flair/ner-english-ontonotes-large", aeon_flag=True) 626 | 627 | elif api_type == "azure": 628 | original_sentences = json.load(open(dataset_path)) 629 | if down_size == True: 630 | original_sentences = random.Random(14).sample(original_sentences, 100) 631 | run_azure(original_sentences) 632 | file_name = "bbc_azure_original.json" # original prediction file 633 | 634 | run_MR_BERT(file_name, aeon_flag, aeon_threshold=0) 635 | run_MR_structure(file_name, aeon_flag, aeon_threshold=0) 636 | run_MR_intra_shuffle(file_name, aeon_flag, aeon_threshold=0) 637 | run_MR_synonym(file_name, aeon_flag, aeon_threshold=0) 638 | original_file = file_name 639 | 640 | run_test_mr_bert(original_file, "AEON_bbc_azure_MR_BERT.json", api_type = "azure", aeon_flag=True) 641 | run_test_mr_structure(original_file, "AEON_bbc_azure_MR_structure.json", api_type = "azure", aeon_flag=True) 642 | run_test_mr_intra_shuffle(original_file, "AEON_bbc_azure_MR_intra_shuffle.json", api_type = "azure", aeon_flag=True) 643 | run_test_mr_synonym(original_file, "AEON_bbc_azure_MR_synonym.json", api_type = "azure", aeon_flag=True) 644 | 645 | elif api_type == "aws": 646 | original_sentences = json.load(open(dataset_path)) 647 | if down_size == True: 648 | original_sentences = random.Random(14).sample(original_sentences, 100) 649 | run_aws(original_sentences) 650 | file_name = "bbc_aws_original.json" # original prediction file 651 | 652 | run_MR_BERT(file_name, aeon_flag, aeon_threshold=0) 653 | run_MR_structure(file_name, aeon_flag, aeon_threshold=0) 654 | run_MR_intra_shuffle(file_name, aeon_flag, aeon_threshold=0) 655 | run_MR_synonym(file_name, aeon_flag, aeon_threshold=0) 656 | original_file = file_name 657 | 658 | run_test_mr_bert(original_file, "AEON_bbc_aws_MR_BERT.json", api_type = "aws", aeon_flag=True) 659 | run_test_mr_structure(original_file, "AEON_bbc_aws_MR_structure.json", api_type = "aws", aeon_flag=True) 660 | run_test_mr_intra_shuffle(original_file, "AEON_bbc_aws_MR_intra_shuffle.json", api_type = "aws", aeon_flag=True) 661 | run_test_mr_synonym(original_file, "AEON_bbc_aws_MR_synonym.json", api_type = "aws", aeon_flag=True) -------------------------------------------------------------------------------- /assets/NER_Repair.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobustNLP/TestNER/ced6f86d8cdfb1c16fbf7b968481051fc0f83f4e/assets/NER_Repair.png -------------------------------------------------------------------------------- /assets/README.md: -------------------------------------------------------------------------------- 1 | This is the directory for assets 2 | --------------------------------------------------------------------------------