├── LICENSE ├── README.md ├── combine_sent_scores.py ├── compile_results.py ├── data_gen ├── base-grammar.gr ├── gen_data.sh ├── make_splits.py ├── permute_sentences.py └── sample_sentences.py ├── get_sentence_scores.py ├── results_analysis ├── mixed_model.py └── permutation_test.py ├── run_all_jobs.py ├── train_lm_lstm.sh └── train_lm_transformer.sh /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Rycolab 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Examining the Inductive Bias of Neural Language Models with Artificial Languages 2 | 3 | ## Paper 4 | 5 | This repository contains code accompanying ['Examining the Inductive Bias of Neural Language Models with Artificial Languages'](https://aclanthology.org/2021.acl-long.38/) by Jennifer C. White and Ryan Cotterell (ACL 2021). 6 | 7 | ## Generating Grammars From Choice Points 8 | 9 | `base-grammar.gr` contains a simple base grammar with 6 possible constructions whose orders may be changed between head-final and head-initial orderings. Each of the productions that would need to be reversed to make this possible is marked with a number. Productions that are part of one construction share the same number. 10 | 11 | By running `gen_data.sh`, splits will be generated containing matched sample sentences that differ only in their constituent ordering and cover all possible combinations of these swappable orderings. 12 | 13 | In the names of the generated folder of the form XXXXXX, each of the swappable orderings (labelled i) is represented by a 0 (indicating head-final ordering) or a 1 (indicating head-initial) in the ith position. 14 | 15 | The orderings are labelled as follows: 16 | 17 | * 1 - Position of verb within a sentence 18 | * 2 - Position of noun within a verb phrase 19 | * 3 - Position of complementizer in a complement 20 | * 4 - Ordering of prepositional phrase 21 | * 5 - Position of adjective in a noun phrase 22 | * 6 - Position of relativizer when making a relative clause 23 | 24 | ## Training Language Models 25 | 26 | [Fairseq](https://github.com/pytorch/fairseq) is required to train the models used in the paper. 27 | 28 | The scripts used to extract scores and collate results also assume that results are output in the form used by Fairseq. 29 | 30 | `train_lm_transformer.sh` and `train_lm_lstm.sh` will train a fairseq transformer and LSTM, respectively, on the grammar and split that they are passed. 31 | 32 | `python run_all_jobs.py` will cycle through all grammars and all splits and train a transformer and an LSTM on each. Using `--submission_command`, a job submission command for an HPC system can be passed in, so that each of these models is submitted to the desired HPC system as a separate job (this may require modification depending on the system being used. 33 | 34 | ## Collating Results 35 | 36 | After all models are finished training, run 37 | ```bash 38 | python compile_results.py -f trans-results/ -o trans-results.csv && python compile_results.py -f lstm-results/ -o lstm-results.csv 39 | ``` 40 | to generate CSVs containing mean and standard deviations of perplexity across all grammars. 41 | 42 | And run 43 | ```bash 44 | python combine_sent_scores.py -f trans_sentence_scores/ -O compiled_trans_scores/ && python combine_sent_scores.py -f lstm_sentence_scores/ -O compiled_lstm_scores/ 45 | ``` 46 | to combine sentence scores from all splits into one file per grammar. 47 | 48 | ## Analysis of Results 49 | 50 | Run 51 | ```bash 52 | python results_analysis/permutation_test.py -f compiled_trans_scores/ -O transformer && python results_analysis/permutation_test.py -f compiled_lstm_scores/ -O lstm 53 | ``` 54 | to generate a CSV showing which grammars exhibit a statistically significant difference in performance. 55 | 56 | Run 57 | ```bash 58 | python results_analysis/mixed_model.py -f compiled_trans_scores/ -o transformer_mixed.csv -i True && python results_analysis/mixed_model.py -f compiled_lstm_scores/ -o lstm_mixed.csv -i True 59 | ``` 60 | to model the results using mixed effecs models. 61 | 62 | ## Requirements 63 | 64 | Fairseq, pandas, numpy, statsmodels 65 | 66 | ## Citation 67 | 68 | ```bash 69 | @inproceedings{white-cotterell-2021-examining, 70 | title = "Examining the Inductive Bias of Neural Language Models with Artificial Languages", 71 | author = "White, Jennifer C. and 72 | Cotterell, Ryan", 73 | booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)", 74 | month = aug, 75 | year = "2021", 76 | address = "Online", 77 | publisher = "Association for Computational Linguistics", 78 | url = "https://aclanthology.org/2021.acl-long.38", 79 | doi = "10.18653/v1/2021.acl-long.38", 80 | pages = "454--463", 81 | abstract = "Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages. Investigation of these biases has proved complicated due to the many variables that appear in the experimental setup. Languages vary in many typological dimensions, and it is difficult to single out one or two to investigate without the others acting as confounders. We propose a novel method for investigating the inductive biases of language models using artificial languages. These languages are constructed to allow us to create parallel corpora across languages that differ only in the typological feature being investigated, such as word order. We then use them to train and test language models. This constitutes a fully controlled causal framework, and demonstrates how grammar engineering can serve as a useful tool for analyzing neural models. Using this method, we find that commonly used neural architectures exhibit different inductive biases: LSTMs display little preference with respect to word ordering, while transformers display a clear preference for some orderings over others. Further, we find that neither the inductive bias of the LSTM nor that of the transformer appear to reflect any tendencies that we see in attested natural languages.", 82 | } 83 | ``` 84 | -------------------------------------------------------------------------------- /combine_sent_scores.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import sys 4 | 5 | parser = argparse.ArgumentParser(description="Compile sentence scores across " 6 | "splits into one file for each grammar") 7 | 8 | parser.add_argument("-f", "--file_location", type=str, required=True, 9 | help="Path to folder containing files") 10 | 11 | parser.add_argument("-O", "--output_folder", type=str, required=True, 12 | help="Location of output folder") 13 | 14 | args = parser.parse_args() 15 | 16 | for i in range(64): 17 | grammar = format(i, '06b')[::-1] 18 | score_files = [os.path.join(args.file_location, grammar, 19 | f) for f in os.listdir(os.path.join(args.file_location, 20 | grammar)) if f.endswith('.txt')] 21 | score_files.sort(key=lambda f: f.split("/")[-1].split(".")[0]) 22 | scores = [] 23 | for f in score_files: 24 | file = open(f, 'r') 25 | file_scores = file.readlines() 26 | scores += file_scores 27 | if not os.path.exists(os.path.join(args.output_folder)): 28 | os.mkdir(os.path.join(args.output_folder)) 29 | output_file = open(os.path.join(args.output_folder, 30 | grammar + "_scores.txt"), 'w') 31 | for s in scores: 32 | output_file.write(s) -------------------------------------------------------------------------------- /compile_results.py: -------------------------------------------------------------------------------- 1 | import csv 2 | import argparse 3 | import os 4 | import math 5 | 6 | def get_perplexity(filename): 7 | file = open(filename, 'r') 8 | lines = file.readlines() 9 | final_line = lines[-1].strip('\n') 10 | return float(final_line.split(" ")[-1]) 11 | 12 | def calc_sd(vals): 13 | mean = calc_mean(vals) 14 | return math.sqrt(sum([(x - mean)**2 for x in vals])/len(vals)) 15 | 16 | def calc_mean(vals): 17 | return sum(vals)/len(vals) 18 | 19 | parser = argparse.ArgumentParser(description="Calculate mean and SD of " 20 | "perplexity for dev and test for each grammar") 21 | 22 | parser.add_argument("-f", "--folder", type=str, required=True, 23 | help="Location of results file") 24 | 25 | parser.add_argument("-o", "--output", type=str, required=True, 26 | help="Location to save output") 27 | 28 | args = parser.parse_args() 29 | 30 | results_files = [os.path.join(args.folder, f) for f in os.listdir( 31 | args.folder) if f.endswith('.txt')] 32 | perplexity_dict = {} 33 | for res in results_files: 34 | grammar, split, test_dev, _ = res.split("/")[-1].split(".") 35 | if grammar not in perplexity_dict.keys(): 36 | perplexity_dict[grammar] = {} 37 | if test_dev not in perplexity_dict[grammar].keys(): 38 | perplexity_dict[grammar][test_dev] = [] 39 | perplexity_dict[grammar][test_dev].append(get_perplexity(res)) 40 | 41 | output_file = open(args.output, 'w') 42 | fieldnames = ['grammar', 'dev_av', 'dev_sd', 'tst_av', 'tst_sd'] 43 | writer = csv.DictWriter(output_file, fieldnames=fieldnames) 44 | writer.writeheader() 45 | for i in range(64): 46 | grammar = format(i, '06b') 47 | writer.writerow({'grammar':grammar, 48 | 'dev_av':calc_mean(perplexity_dict[grammar]['dev']), 49 | 'dev_sd':calc_sd(perplexity_dict[grammar]['dev']), 50 | 'tst_av':calc_mean(perplexity_dict[grammar]['test']), 51 | 'tst_sd':calc_sd(perplexity_dict[grammar]['test']) 52 | }) 53 | -------------------------------------------------------------------------------- /data_gen/base-grammar.gr: -------------------------------------------------------------------------------- 1 | 1 ROOT S 2 | 3 | 1 S NP_Subj_S VP_S 1 4 | 1 S NP_Subj_P VP_P 1 5 | 1 VP_S VP_Past_S 6 | 1 VP_S VP_Pres_S 7 | 1 VP_S VP_Comp_S 8 | 1 VP_P VP_Past_P 9 | 1 VP_P VP_Pres_P 10 | 1 VP_P VP_Comp_P 11 | 1 VP_Comp_S VP_Comp_Pres_S 12 | 1 VP_Comp_S VP_Comp_Past_S 13 | 1 VP_Comp_P VP_Comp_Pres_P 14 | 1 VP_Comp_P VP_Comp_Past_P 15 | 1 NP_Subj_S NP_S Subj 16 | 1 NP_Subj_P NP_P Subj 17 | 1 VP_Past_S IVerb_Past_S 18 | 1 VP_Pres_S IVerb_Pres_S 19 | 1 VP_Past_P IVerb_Past_P 20 | 1 VP_Pres_P IVerb_Pres_P 21 | 1 VP_Past_S NP_Obj TVerb_Past_S 2 22 | 1 VP_Pres_S NP_Obj TVerb_Pres_S 2 23 | 1 VP_Comp_Pres_S S_Comp Verb_Comp_Pres_S 2 24 | 1 VP_Comp_Past_S S_Comp Verb_Comp_Past_S 2 25 | 1 VP_Past_P NP_Obj TVerb_Past_P 2 26 | 1 VP_Pres_P NP_Obj TVerb_Pres_P 2 27 | 1 VP_Comp_Pres_P S_Comp Verb_Comp_Pres_P 2 28 | 1 VP_Comp_Past_P S_Comp Verb_Comp_Past_P 2 29 | 1 S_Comp S Comp 3 30 | 1 NP_Obj NP_S Obj 31 | 1 NP_Obj NP_P Obj 32 | 1 NP_S Noun_S 33 | 1 NP_P Noun_P 34 | 0.2 NP_S PP NP_S 4 35 | 0.2 NP_P PP NP_P 4 36 | 1 PP NP_S Prep 4 37 | 1 PP NP_P Prep 4 38 | 1 NP_S Adj Noun_S 5 39 | 1 NP_P Adj Noun_P 5 40 | 1 Adj Adj CC Adj 41 | 1 NP_S VP_S Rel Noun_S 6 42 | 1 NP_P VP_P Rel Noun_P 6 43 | 0.025 NP_P NP_S CC NP_S 44 | 0.025 NP_P NP_P CC NP_P 45 | 0.025 NP_P NP_P CC NP_S 46 | 0.025 NP_P NP_S CC NP_P 47 | 0.2 TVerb_Past_S TVerb_Past_S CC TVerb_Past_S 48 | 0.2 TVerb_Pres_S TVerb_Pres_S CC TVerb_Pres_S 49 | 0.2 IVerb_Past_S IVerb_Past_S CC IVerb_Past_S 50 | 0.2 IVerb_Pres_S IVerb_Pres_S CC IVerb_Pres_S 51 | 0.2 TVerb_Past_P TVerb_Past_P CC TVerb_Past_P 52 | 0.2 TVerb_Pres_P TVerb_Pres_P CC TVerb_Pres_P 53 | 0.2 IVerb_Past_P IVerb_Past_P CC IVerb_Past_P 54 | 0.2 IVerb_Pres_P IVerb_Pres_P CC IVerb_Pres_P 55 | 1 NP_S Pronoun_S 56 | 1 NP_P Pronoun_P 57 | 58 | 1 Noun_S amackist 59 | 1 Noun_S amationist 60 | 1 Noun_S ambather 61 | 1 Noun_S ampercial 62 | 1 Noun_S appalate 63 | 1 Noun_S autoner 64 | 1 Noun_S bactivitor 65 | 1 Noun_S ballison 66 | 1 Noun_S banitician 67 | 1 Noun_S bellion 68 | 1 Noun_S benatician 69 | 1 Noun_S bleckman 70 | 1 Noun_S blope 71 | 1 Noun_S bolicater 72 | 1 Noun_S bolician 73 | 1 Noun_S bolitist 74 | 1 Noun_S botigner 75 | 1 Noun_S bottist 76 | 1 Noun_S breaper 77 | 1 Noun_S brelland 78 | 1 Noun_S briticist 79 | 1 Noun_S briticist 80 | 1 Noun_S buestor 81 | 1 Noun_S bullager 82 | 1 Noun_S bullawer 83 | 1 Noun_S bullif 84 | 1 Noun_S bullifist 85 | 1 Noun_S bunessator 86 | 1 Noun_S burse 87 | 1 Noun_S businator 88 | 1 Noun_S businer 89 | 1 Noun_S calinelist 90 | 1 Noun_S cheldor 91 | 1 Noun_S crotician 92 | 1 Noun_S crotifer 93 | 1 Noun_S cushar 94 | 1 Noun_S dake 95 | 1 Noun_S dargine 96 | 1 Noun_S denatician 97 | 1 Noun_S doloner 98 | 1 Noun_S dolonician 99 | 1 Noun_S dorpal 100 | 1 Noun_S dorpalist 101 | 1 Noun_S draccont 102 | 1 Noun_S drinimat 103 | 1 Noun_S droinist 104 | 1 Noun_S drouseman 105 | 1 Noun_S duppist 106 | 1 Noun_S duskist 107 | 1 Noun_S dutingist 108 | 1 Noun_S ecoster 109 | 1 Noun_S eluper 110 | 1 Noun_S elusist 111 | 1 Noun_S enlator 112 | 1 Noun_S fallengist 113 | 1 Noun_S fantivist 114 | 1 Noun_S flingerician 115 | 1 Noun_S flingerist 116 | 1 Noun_S formantor 117 | 1 Noun_S formantor 118 | 1 Noun_S fossician 119 | 1 Noun_S fragion 120 | 1 Noun_S framitator 121 | 1 Noun_S frawkess 122 | 1 Noun_S fremator 123 | 1 Noun_S froacher 124 | 1 Noun_S frominaler 125 | 1 Noun_S frominalist 126 | 1 Noun_S fumitist 127 | 1 Noun_S fusbender 128 | 1 Noun_S gangerist 129 | 1 Noun_S gezziman 130 | 1 Noun_S gloralizer 131 | 1 Noun_S gratenist 132 | 1 Noun_S guendor 133 | 1 Noun_S gumitist 134 | 1 Noun_S gustist 135 | 1 Noun_S hane 136 | 1 Noun_S haustist 137 | 1 Noun_S hilldomer 138 | 1 Noun_S horrier 139 | 1 Noun_S hostician 140 | 1 Noun_S hostist 141 | 1 Noun_S hule 142 | 1 Noun_S humician 143 | 1 Noun_S jantomer 144 | 1 Noun_S jobsticator 145 | 1 Noun_S jonter 146 | 1 Noun_S justator 147 | 1 Noun_S kaidist 148 | 1 Noun_S kewardessor 149 | 1 Noun_S kuestor 150 | 1 Noun_S lactimer 151 | 1 Noun_S lactissist 152 | 1 Noun_S lawticater 153 | 1 Noun_S lenn 154 | 1 Noun_S lurchist 155 | 1 Noun_S lustodist 156 | 1 Noun_S madetor 157 | 1 Noun_S mafe 158 | 1 Noun_S mafflitist 159 | 1 Noun_S melottess 160 | 1 Noun_S mespician 161 | 1 Noun_S midician 162 | 1 Noun_S milturor 163 | 1 Noun_S mofustator 164 | 1 Noun_S moldician 165 | 1 Noun_S naridator 166 | 1 Noun_S narter 167 | 1 Noun_S nestator 168 | 1 Noun_S noacher 169 | 1 Noun_S nust 170 | 1 Noun_S pactesse 171 | 1 Noun_S peachician 172 | 1 Noun_S peagerist 173 | 1 Noun_S penesser 174 | 1 Noun_S penursifer 175 | 1 Noun_S pilobist 176 | 1 Noun_S pleddist 177 | 1 Noun_S povician 178 | 1 Noun_S predignor 179 | 1 Noun_S prither 180 | 1 Noun_S protter 181 | 1 Noun_S purgettor 182 | 1 Noun_S ranater 183 | 1 Noun_S rattiner 184 | 1 Noun_S redullor 185 | 1 Noun_S ricatite 186 | 1 Noun_S ronder 187 | 1 Noun_S senth 188 | 1 Noun_S serd 189 | 1 Noun_S serg 190 | 1 Noun_S sergator 191 | 1 Noun_S shagoner 192 | 1 Noun_S sherician 193 | 1 Noun_S shodderist 194 | 1 Noun_S siverthist 195 | 1 Noun_S smoob 196 | 1 Noun_S stimper 197 | 1 Noun_S stoppist 198 | 1 Noun_S stourt 199 | 1 Noun_S streachor 200 | 1 Noun_S stremorder 201 | 1 Noun_S striberist 202 | 1 Noun_S stroater 203 | 1 Noun_S strogessor 204 | 1 Noun_S tharoner 205 | 1 Noun_S theslirist 206 | 1 Noun_S thullamor 207 | 1 Noun_S thullician 208 | 1 Noun_S tib 209 | 1 Noun_S tiraster 210 | 1 Noun_S trachier 211 | 1 Noun_S trapenist 212 | 1 Noun_S trenner 213 | 1 Noun_S visivet 214 | 1 Noun_S voncenist 215 | 1 Noun_S wainster 216 | 1 Noun_S wanner 217 | 1 Noun_S wolarist 218 | 1 Noun_S wotanker 219 | 1 Noun_S wuther 220 | 221 | 1 Noun_P amackists 222 | 1 Noun_P amationists 223 | 1 Noun_P ambathers 224 | 1 Noun_P ampercials 225 | 1 Noun_P appalates 226 | 1 Noun_P autoners 227 | 1 Noun_P bactivitors 228 | 1 Noun_P ballisons 229 | 1 Noun_P baniticians 230 | 1 Noun_P bellions 231 | 1 Noun_P benaticians 232 | 1 Noun_P bleckmans 233 | 1 Noun_P blopes 234 | 1 Noun_P bolicaters 235 | 1 Noun_P bolicians 236 | 1 Noun_P bolitists 237 | 1 Noun_P botigners 238 | 1 Noun_P bottists 239 | 1 Noun_P breapers 240 | 1 Noun_P brellands 241 | 1 Noun_P briticists 242 | 1 Noun_P briticists 243 | 1 Noun_P buestors 244 | 1 Noun_P bullagers 245 | 1 Noun_P bullawers 246 | 1 Noun_P bullifs 247 | 1 Noun_P bullifists 248 | 1 Noun_P bunessators 249 | 1 Noun_P burses 250 | 1 Noun_P businators 251 | 1 Noun_P businers 252 | 1 Noun_P calinelists 253 | 1 Noun_P cheldors 254 | 1 Noun_P croticians 255 | 1 Noun_P crotifers 256 | 1 Noun_P cushars 257 | 1 Noun_P dakes 258 | 1 Noun_P dargines 259 | 1 Noun_P denaticians 260 | 1 Noun_P doloners 261 | 1 Noun_P dolonicians 262 | 1 Noun_P dorpals 263 | 1 Noun_P dorpalists 264 | 1 Noun_P dracconts 265 | 1 Noun_P drinimats 266 | 1 Noun_P droinists 267 | 1 Noun_P drousemans 268 | 1 Noun_P duppists 269 | 1 Noun_P duskists 270 | 1 Noun_P dutingists 271 | 1 Noun_P ecosters 272 | 1 Noun_P elupers 273 | 1 Noun_P elusists 274 | 1 Noun_P enlators 275 | 1 Noun_P fallengists 276 | 1 Noun_P fantivists 277 | 1 Noun_P flingericians 278 | 1 Noun_P flingerists 279 | 1 Noun_P formantors 280 | 1 Noun_P formantors 281 | 1 Noun_P fossicians 282 | 1 Noun_P fragions 283 | 1 Noun_P framitators 284 | 1 Noun_P frawkesses 285 | 1 Noun_P fremators 286 | 1 Noun_P froachers 287 | 1 Noun_P frominalers 288 | 1 Noun_P frominalists 289 | 1 Noun_P fumitists 290 | 1 Noun_P fusbenders 291 | 1 Noun_P gangerists 292 | 1 Noun_P gezzimans 293 | 1 Noun_P gloralizers 294 | 1 Noun_P gratenists 295 | 1 Noun_P guendors 296 | 1 Noun_P gumitists 297 | 1 Noun_P gustists 298 | 1 Noun_P hanes 299 | 1 Noun_P haustists 300 | 1 Noun_P hilldomers 301 | 1 Noun_P horriers 302 | 1 Noun_P hosticians 303 | 1 Noun_P hostists 304 | 1 Noun_P hules 305 | 1 Noun_P humicians 306 | 1 Noun_P jantomers 307 | 1 Noun_P jobsticators 308 | 1 Noun_P jonters 309 | 1 Noun_P justators 310 | 1 Noun_P kaidists 311 | 1 Noun_P kewardessors 312 | 1 Noun_P kuestors 313 | 1 Noun_P lactimers 314 | 1 Noun_P lactissists 315 | 1 Noun_P lawticaters 316 | 1 Noun_P lenns 317 | 1 Noun_P lurchists 318 | 1 Noun_P lustodists 319 | 1 Noun_P madetors 320 | 1 Noun_P mafes 321 | 1 Noun_P mafflitists 322 | 1 Noun_P melottesses 323 | 1 Noun_P mespicians 324 | 1 Noun_P midicians 325 | 1 Noun_P milturors 326 | 1 Noun_P mofustators 327 | 1 Noun_P moldicians 328 | 1 Noun_P naridators 329 | 1 Noun_P narters 330 | 1 Noun_P nestators 331 | 1 Noun_P noachers 332 | 1 Noun_P nusts 333 | 1 Noun_P pactesses 334 | 1 Noun_P peachicians 335 | 1 Noun_P peagerists 336 | 1 Noun_P penessers 337 | 1 Noun_P penursifers 338 | 1 Noun_P pilobists 339 | 1 Noun_P pleddists 340 | 1 Noun_P povicians 341 | 1 Noun_P predignors 342 | 1 Noun_P prithers 343 | 1 Noun_P protters 344 | 1 Noun_P purgettors 345 | 1 Noun_P ranaters 346 | 1 Noun_P rattiners 347 | 1 Noun_P redullors 348 | 1 Noun_P ricatites 349 | 1 Noun_P ronders 350 | 1 Noun_P senths 351 | 1 Noun_P serds 352 | 1 Noun_P sergs 353 | 1 Noun_P sergators 354 | 1 Noun_P shagoners 355 | 1 Noun_P shericians 356 | 1 Noun_P shodderists 357 | 1 Noun_P siverthists 358 | 1 Noun_P smoobs 359 | 1 Noun_P stimpers 360 | 1 Noun_P stoppists 361 | 1 Noun_P stourts 362 | 1 Noun_P streachors 363 | 1 Noun_P stremorders 364 | 1 Noun_P striberists 365 | 1 Noun_P stroaters 366 | 1 Noun_P strogessors 367 | 1 Noun_P tharoners 368 | 1 Noun_P theslirists 369 | 1 Noun_P thullamors 370 | 1 Noun_P thullicians 371 | 1 Noun_P tibs 372 | 1 Noun_P tirasters 373 | 1 Noun_P trachiers 374 | 1 Noun_P trapenists 375 | 1 Noun_P trenners 376 | 1 Noun_P visivets 377 | 1 Noun_P voncenists 378 | 1 Noun_P wainsters 379 | 1 Noun_P wanners 380 | 1 Noun_P wolarists 381 | 1 Noun_P wotankers 382 | 1 Noun_P wuthers 383 | 384 | 1 Adj prask 385 | 1 Adj spange 386 | 1 Adj crene 387 | 1 Adj jurb 388 | 1 Adj bilth 389 | 1 Adj streck 390 | 1 Adj skame 391 | 1 Adj darsh 392 | 1 Adj shrop 393 | 1 Adj blide 394 | 1 Adj breight 395 | 1 Adj jawp 396 | 1 Adj plonth 397 | 1 Adj threak 398 | 1 Adj dreek 399 | 1 Adj meech 400 | 1 Adj skeer 401 | 1 Adj zild 402 | 1 Adj jeethe 403 | 1 Adj clim 404 | 1 Adj spean 405 | 1 Adj frub 406 | 1 Adj douch 407 | 1 Adj thrope 408 | 1 Adj yeige 409 | 1 Adj gaunch 410 | 1 Adj gringe 411 | 1 Adj ghump 412 | 1 Adj yerg 413 | 1 Adj scooth 414 | 1 Adj fumit 415 | 1 Adj stremord 416 | 1 Adj purgett 417 | 1 Adj jont 418 | 1 Adj calin 419 | 1 Adj naid 420 | 1 Adj hild 421 | 1 Adj botig 422 | 1 Adj fusbend 423 | 1 Adj guend 424 | 1 Adj wuth 425 | 1 Adj tharon 426 | 427 | 1 Verb_Comp_Past_S engarified 428 | 1 Verb_Comp_Past_S kurched 429 | 1 Verb_Comp_Past_S formanated 430 | 1 Verb_Comp_Past_S crailed 431 | 1 Verb_Comp_Past_S lurchified 432 | 1 Verb_Comp_Past_S fanched 433 | 1 Verb_Comp_Past_S prutified 434 | 1 Verb_Comp_Past_S squailed 435 | 1 Verb_Comp_Past_S nobed 436 | 1 Verb_Comp_Past_S pactessed 437 | 1 Verb_Comp_Past_S sedundified 438 | 1 Verb_Comp_Past_S sergatized 439 | 1 Verb_Comp_Past_S unseffed 440 | 1 Verb_Comp_Past_S buled 441 | 1 Verb_Comp_Past_S humicianed 442 | 1 Verb_Comp_Past_S saunched 443 | 1 Verb_Comp_Past_S thespired 444 | 1 Verb_Comp_Past_S conturized 445 | 1 Verb_Comp_Past_S mangered 446 | 1 Verb_Comp_Past_S strovokicized 447 | 1 Verb_Comp_Past_S embarricated 448 | 1 Verb_Comp_Past_S crinned 449 | 450 | 1 Verb_Comp_Past_P engarifieda 451 | 1 Verb_Comp_Past_P kurcheda 452 | 1 Verb_Comp_Past_P formanateda 453 | 1 Verb_Comp_Past_P craileda 454 | 1 Verb_Comp_Past_P lurchifieda 455 | 1 Verb_Comp_Past_P fancheda 456 | 1 Verb_Comp_Past_P prutifieda 457 | 1 Verb_Comp_Past_P squaileda 458 | 1 Verb_Comp_Past_P nobeda 459 | 1 Verb_Comp_Past_P pactesseda 460 | 1 Verb_Comp_Past_P sedundifieda 461 | 1 Verb_Comp_Past_P sergatizeda 462 | 1 Verb_Comp_Past_P unseffeda 463 | 1 Verb_Comp_Past_P buleda 464 | 1 Verb_Comp_Past_P humicianeda 465 | 1 Verb_Comp_Past_P sauncheda 466 | 1 Verb_Comp_Past_P thespireda 467 | 1 Verb_Comp_Past_P conturizeda 468 | 1 Verb_Comp_Past_P mangereda 469 | 1 Verb_Comp_Past_P strovokicizeda 470 | 1 Verb_Comp_Past_P embarricateda 471 | 1 Verb_Comp_Past_P crinneda 472 | 473 | 1 IVerb_Past_S fusilitated 474 | 1 IVerb_Past_S remiffed 475 | 1 IVerb_Past_S mangerized 476 | 1 IVerb_Past_S plessicated 477 | 1 IVerb_Past_S wogged 478 | 1 IVerb_Past_S forigated 479 | 1 IVerb_Past_S vemed 480 | 1 IVerb_Past_S phanicized 481 | 1 IVerb_Past_S denaticated 482 | 1 IVerb_Past_S gasted 483 | 1 IVerb_Past_S peaserized 484 | 1 IVerb_Past_S piliciated 485 | 1 IVerb_Past_S hurperated 486 | 1 IVerb_Past_S ralmed 487 | 1 IVerb_Past_S charched 488 | 1 IVerb_Past_S filked 489 | 1 IVerb_Past_S tawlerized 490 | 1 IVerb_Past_S onstigipated 491 | 1 IVerb_Past_S rolved 492 | 1 IVerb_Past_S rolved 493 | 1 IVerb_Past_S offalded 494 | 1 IVerb_Past_S calikated 495 | 1 IVerb_Past_S mauced 496 | 1 IVerb_Past_S inthippenated 497 | 1 IVerb_Past_S visified 498 | 1 IVerb_Past_S froachered 499 | 1 IVerb_Past_S madressized 500 | 1 IVerb_Past_S benaticianed 501 | 1 IVerb_Past_S plessicked 502 | 1 IVerb_Past_S scupped 503 | 1 IVerb_Past_S framitatored 504 | 1 IVerb_Past_S thrombicized 505 | 1 IVerb_Past_S portified 506 | 1 IVerb_Past_S tirasted 507 | 1 IVerb_Past_S osmissified 508 | 1 IVerb_Past_S peachicianed 509 | 1 IVerb_Past_S scoaned 510 | 1 IVerb_Past_S amackated 511 | 1 IVerb_Past_S unplewed 512 | 1 IVerb_Past_S jantomized 513 | 1 IVerb_Past_S allacked 514 | 1 IVerb_Past_S thullamized 515 | 1 IVerb_Past_S smeshed 516 | 1 IVerb_Past_S tismissated 517 | 1 IVerb_Past_S offaldicated 518 | 1 IVerb_Past_S briminated 519 | 1 IVerb_Past_S maksized 520 | 1 IVerb_Past_S amberated 521 | 1 IVerb_Past_S amacked 522 | 1 IVerb_Past_S apturicized 523 | 1 IVerb_Past_S barsed 524 | 1 IVerb_Past_S mofustated 525 | 1 IVerb_Past_S broaled 526 | 1 IVerb_Past_S fampenated 527 | 1 IVerb_Past_S thespirated 528 | 1 IVerb_Past_S parled 529 | 1 IVerb_Past_S thrubbed 530 | 1 IVerb_Past_S tarsed 531 | 1 IVerb_Past_S teeved 532 | 1 IVerb_Past_S revalternated 533 | 1 IVerb_Past_S noachated 534 | 1 IVerb_Past_S flapperated 535 | 1 IVerb_Past_S mued 536 | 1 IVerb_Past_S miticurized 537 | 1 IVerb_Past_S saffled 538 | 1 IVerb_Past_S chorried 539 | 1 IVerb_Past_S excribited 540 | 1 IVerb_Past_S lurchified 541 | 1 IVerb_Past_S sturked 542 | 1 IVerb_Past_S nestatated 543 | 1 IVerb_Past_S limbatized 544 | 1 IVerb_Past_S rotectified 545 | 1 IVerb_Past_S swarmicated 546 | 1 IVerb_Past_S prausified 547 | 1 IVerb_Past_S elusified 548 | 1 IVerb_Past_S spectified 549 | 1 IVerb_Past_S annoriciated 550 | 1 IVerb_Past_S elupized 551 | 1 IVerb_Past_S lactissated 552 | 1 IVerb_Past_S streachified 553 | 1 IVerb_Past_S mattakated 554 | 1 IVerb_Past_S ballicized 555 | 1 IVerb_Past_S dupped 556 | 1 IVerb_Past_S mofusted 557 | 1 IVerb_Past_S yawled 558 | 1 IVerb_Past_S striberated 559 | 1 IVerb_Past_S treaved 560 | 1 IVerb_Past_S mermillated 561 | 1 IVerb_Past_S prottized 562 | 1 IVerb_Past_S wolarified 563 | 1 IVerb_Past_S gratenized 564 | 1 IVerb_Past_S intrinsed 565 | 1 IVerb_Past_S skeezed 566 | 1 IVerb_Past_S rawned 567 | 1 IVerb_Past_S tirastified 568 | 1 IVerb_Past_S yolved 569 | 1 IVerb_Past_S bunessisted 570 | 1 IVerb_Past_S damerticated 571 | 1 IVerb_Past_S peagerized 572 | 1 IVerb_Past_S stoppisted 573 | 1 IVerb_Past_S engaried 574 | 1 IVerb_Past_S bottisted 575 | 1 IVerb_Past_S bactivitated 576 | 1 IVerb_Past_S greanerized 577 | 1 IVerb_Past_S duppified 578 | 1 IVerb_Past_S zurched 579 | 1 IVerb_Past_S stroved 580 | 1 IVerb_Past_S povicated 581 | 1 IVerb_Past_S lavoidified 582 | 1 IVerb_Past_S garfed 583 | 1 IVerb_Past_S crotified 584 | 1 IVerb_Past_S wandified 585 | 1 IVerb_Past_S poutered 586 | 587 | 1 IVerb_Past_P fusilitateda 588 | 1 IVerb_Past_P remiffeda 589 | 1 IVerb_Past_P mangerizeda 590 | 1 IVerb_Past_P plessicateda 591 | 1 IVerb_Past_P woggeda 592 | 1 IVerb_Past_P forigateda 593 | 1 IVerb_Past_P vemeda 594 | 1 IVerb_Past_P phanicizeda 595 | 1 IVerb_Past_P denaticateda 596 | 1 IVerb_Past_P gasteda 597 | 1 IVerb_Past_P peaserizeda 598 | 1 IVerb_Past_P piliciateda 599 | 1 IVerb_Past_P hurperateda 600 | 1 IVerb_Past_P ralmeda 601 | 1 IVerb_Past_P charcheda 602 | 1 IVerb_Past_P filkeda 603 | 1 IVerb_Past_P tawlerizeda 604 | 1 IVerb_Past_P onstigipateda 605 | 1 IVerb_Past_P rolveda 606 | 1 IVerb_Past_P rolveda 607 | 1 IVerb_Past_P offaldeda 608 | 1 IVerb_Past_P calikateda 609 | 1 IVerb_Past_P mauceda 610 | 1 IVerb_Past_P inthippenateda 611 | 1 IVerb_Past_P visifieda 612 | 1 IVerb_Past_P froachereda 613 | 1 IVerb_Past_P madressizeda 614 | 1 IVerb_Past_P benaticianeda 615 | 1 IVerb_Past_P plessickeda 616 | 1 IVerb_Past_P scuppeda 617 | 1 IVerb_Past_P framitatoreda 618 | 1 IVerb_Past_P thrombicizeda 619 | 1 IVerb_Past_P portifieda 620 | 1 IVerb_Past_P tirasteda 621 | 1 IVerb_Past_P osmissifieda 622 | 1 IVerb_Past_P peachicianeda 623 | 1 IVerb_Past_P scoaneda 624 | 1 IVerb_Past_P amackateda 625 | 1 IVerb_Past_P unpleweda 626 | 1 IVerb_Past_P jantomizeda 627 | 1 IVerb_Past_P allackeda 628 | 1 IVerb_Past_P thullamizeda 629 | 1 IVerb_Past_P smesheda 630 | 1 IVerb_Past_P tismissateda 631 | 1 IVerb_Past_P offaldicateda 632 | 1 IVerb_Past_P briminateda 633 | 1 IVerb_Past_P maksizeda 634 | 1 IVerb_Past_P amberateda 635 | 1 IVerb_Past_P amackeda 636 | 1 IVerb_Past_P apturicizeda 637 | 1 IVerb_Past_P barseda 638 | 1 IVerb_Past_P mofustateda 639 | 1 IVerb_Past_P broaleda 640 | 1 IVerb_Past_P fampenateda 641 | 1 IVerb_Past_P thespirateda 642 | 1 IVerb_Past_P parleda 643 | 1 IVerb_Past_P thrubbeda 644 | 1 IVerb_Past_P tarseda 645 | 1 IVerb_Past_P teeveda 646 | 1 IVerb_Past_P revalternateda 647 | 1 IVerb_Past_P noachateda 648 | 1 IVerb_Past_P flapperateda 649 | 1 IVerb_Past_P mueda 650 | 1 IVerb_Past_P miticurizeda 651 | 1 IVerb_Past_P saffleda 652 | 1 IVerb_Past_P chorrieda 653 | 1 IVerb_Past_P excribiteda 654 | 1 IVerb_Past_P lurchifieda 655 | 1 IVerb_Past_P sturkeda 656 | 1 IVerb_Past_P nestatateda 657 | 1 IVerb_Past_P limbatizeda 658 | 1 IVerb_Past_P rotectifieda 659 | 1 IVerb_Past_P swarmicateda 660 | 1 IVerb_Past_P prausifieda 661 | 1 IVerb_Past_P elusifieda 662 | 1 IVerb_Past_P spectifieda 663 | 1 IVerb_Past_P annoriciateda 664 | 1 IVerb_Past_P elupizeda 665 | 1 IVerb_Past_P lactissateda 666 | 1 IVerb_Past_P streachifieda 667 | 1 IVerb_Past_P mattakateda 668 | 1 IVerb_Past_P ballicizeda 669 | 1 IVerb_Past_P duppeda 670 | 1 IVerb_Past_P mofusteda 671 | 1 IVerb_Past_P yawleda 672 | 1 IVerb_Past_P striberateda 673 | 1 IVerb_Past_P treaveda 674 | 1 IVerb_Past_P mermillateda 675 | 1 IVerb_Past_P prottizeda 676 | 1 IVerb_Past_P wolarifieda 677 | 1 IVerb_Past_P gratenizeda 678 | 1 IVerb_Past_P intrinseda 679 | 1 IVerb_Past_P skeezeda 680 | 1 IVerb_Past_P rawneda 681 | 1 IVerb_Past_P tirastifieda 682 | 1 IVerb_Past_P yolveda 683 | 1 IVerb_Past_P bunessisteda 684 | 1 IVerb_Past_P damerticateda 685 | 1 IVerb_Past_P peagerizeda 686 | 1 IVerb_Past_P stoppisteda 687 | 1 IVerb_Past_P engarieda 688 | 1 IVerb_Past_P bottisteda 689 | 1 IVerb_Past_P bactivitateda 690 | 1 IVerb_Past_P greanerizeda 691 | 1 IVerb_Past_P duppifieda 692 | 1 IVerb_Past_P zurcheda 693 | 1 IVerb_Past_P stroveda 694 | 1 IVerb_Past_P povicateda 695 | 1 IVerb_Past_P lavoidifieda 696 | 1 IVerb_Past_P garfeda 697 | 1 IVerb_Past_P crotifieda 698 | 1 IVerb_Past_P wandifieda 699 | 1 IVerb_Past_P poutereda 700 | 701 | 1 TVerb_Past_S gratenized 702 | 1 TVerb_Past_S intrinsed 703 | 1 TVerb_Past_S skeezed 704 | 1 TVerb_Past_S rawned 705 | 1 TVerb_Past_S tirastified 706 | 1 TVerb_Past_S yolved 707 | 1 TVerb_Past_S bunessisted 708 | 1 TVerb_Past_S damerticated 709 | 1 TVerb_Past_S peagerized 710 | 1 TVerb_Past_S stoppisted 711 | 1 TVerb_Past_S engaried 712 | 1 TVerb_Past_S bottisted 713 | 1 TVerb_Past_S bactivitated 714 | 1 TVerb_Past_S greanerized 715 | 1 TVerb_Past_S duppified 716 | 1 TVerb_Past_S zurched 717 | 1 TVerb_Past_S stroved 718 | 1 TVerb_Past_S povicated 719 | 1 TVerb_Past_S lavoidified 720 | 1 TVerb_Past_S garfed 721 | 1 TVerb_Past_S crotified 722 | 1 TVerb_Past_S wandified 723 | 1 TVerb_Past_S poutered 724 | 1 TVerb_Past_S rebinked 725 | 1 TVerb_Past_S dafed 726 | 1 TVerb_Past_S bullegated 727 | 1 TVerb_Past_S enjurified 728 | 1 TVerb_Past_S visified 729 | 1 TVerb_Past_S gumitified 730 | 1 TVerb_Past_S baniticated 731 | 1 TVerb_Past_S haustisted 732 | 1 TVerb_Past_S pipistrated 733 | 1 TVerb_Past_S povified 734 | 1 TVerb_Past_S stribered 735 | 1 TVerb_Past_S ninged 736 | 1 TVerb_Past_S mafflitated 737 | 1 TVerb_Past_S mafflicated 738 | 1 TVerb_Past_S pleddisated 739 | 1 TVerb_Past_S droverated 740 | 1 TVerb_Past_S nulpened 741 | 1 TVerb_Past_S thralled 742 | 1 TVerb_Past_S maffliced 743 | 1 TVerb_Past_S yossed 744 | 1 TVerb_Past_S mespicated 745 | 1 TVerb_Past_S spooled 746 | 1 TVerb_Past_S frematized 747 | 1 TVerb_Past_S disfrinsed 748 | 1 TVerb_Past_S kippigated 749 | 1 TVerb_Past_S lawticianed 750 | 1 TVerb_Past_S apturicized 751 | 1 TVerb_Past_S fossicated 752 | 1 TVerb_Past_S blarped 753 | 1 TVerb_Past_S phonorized 754 | 1 TVerb_Past_S saultified 755 | 1 TVerb_Past_S simpuridated 756 | 1 TVerb_Past_S jontified 757 | 1 TVerb_Past_S gratened 758 | 1 TVerb_Past_S sosked 759 | 1 TVerb_Past_S droled 760 | 1 TVerb_Past_S chutched 761 | 1 TVerb_Past_S bolicized 762 | 1 TVerb_Past_S praimed 763 | 1 TVerb_Past_S missiped 764 | 1 TVerb_Past_S strubdified 765 | 1 TVerb_Past_S strozed 766 | 1 TVerb_Past_S blamiciated 767 | 1 TVerb_Past_S bubbed 768 | 1 TVerb_Past_S botticated 769 | 1 TVerb_Past_S missipated 770 | 1 TVerb_Past_S croicerated 771 | 1 TVerb_Past_S fallengized 772 | 1 TVerb_Past_S slurked 773 | 1 TVerb_Past_S traived 774 | 1 TVerb_Past_S conveamed 775 | 1 TVerb_Past_S dupressified 776 | 1 TVerb_Past_S phoisified 777 | 1 TVerb_Past_S zamped 778 | 1 TVerb_Past_S dupressed 779 | 1 TVerb_Past_S draused 780 | 1 TVerb_Past_S prockified 781 | 1 TVerb_Past_S annoriciated 782 | 1 TVerb_Past_S jeaped 783 | 1 TVerb_Past_S pandicated 784 | 1 TVerb_Past_S tissed 785 | 1 TVerb_Past_S draired 786 | 1 TVerb_Past_S theslirated 787 | 1 TVerb_Past_S midicated 788 | 1 TVerb_Past_S clummed 789 | 1 TVerb_Past_S kuestified 790 | 1 TVerb_Past_S peagified 791 | 1 TVerb_Past_S glucked 792 | 1 TVerb_Past_S madetized 793 | 1 TVerb_Past_S croiled 794 | 1 TVerb_Past_S lustodisted 795 | 1 TVerb_Past_S tirastized 796 | 1 TVerb_Past_S penursifered 797 | 1 TVerb_Past_S dismaffed 798 | 1 TVerb_Past_S chozed 799 | 1 TVerb_Past_S justatored 800 | 1 TVerb_Past_S frained 801 | 1 TVerb_Past_S thafassized 802 | 1 TVerb_Past_S enjurified 803 | 1 TVerb_Past_S joofed 804 | 1 TVerb_Past_S likiciated 805 | 1 TVerb_Past_S pouteratized 806 | 1 TVerb_Past_S irriticiated 807 | 1 TVerb_Past_S blitched 808 | 1 TVerb_Past_S gangeristed 809 | 1 TVerb_Past_S stickerated 810 | 1 TVerb_Past_S jobsticatored 811 | 1 TVerb_Past_S scolterized 812 | 1 TVerb_Past_S shumpified 813 | 1 TVerb_Past_S heferated 814 | 815 | 1 TVerb_Past_P gratenizeda 816 | 1 TVerb_Past_P intrinseda 817 | 1 TVerb_Past_P skeezeda 818 | 1 TVerb_Past_P rawneda 819 | 1 TVerb_Past_P tirastifieda 820 | 1 TVerb_Past_P yolveda 821 | 1 TVerb_Past_P bunessisteda 822 | 1 TVerb_Past_P damerticateda 823 | 1 TVerb_Past_P peagerizeda 824 | 1 TVerb_Past_P stoppisteda 825 | 1 TVerb_Past_P engarieda 826 | 1 TVerb_Past_P bottisteda 827 | 1 TVerb_Past_P bactivitateda 828 | 1 TVerb_Past_P greanerizeda 829 | 1 TVerb_Past_P duppifieda 830 | 1 TVerb_Past_P zurcheda 831 | 1 TVerb_Past_P stroveda 832 | 1 TVerb_Past_P povicateda 833 | 1 TVerb_Past_P lavoidifieda 834 | 1 TVerb_Past_P garfeda 835 | 1 TVerb_Past_P crotifieda 836 | 1 TVerb_Past_P wandifieda 837 | 1 TVerb_Past_P poutereda 838 | 1 TVerb_Past_P rebinkeda 839 | 1 TVerb_Past_P dafeda 840 | 1 TVerb_Past_P bullegateda 841 | 1 TVerb_Past_P enjurifieda 842 | 1 TVerb_Past_P visifieda 843 | 1 TVerb_Past_P gumitifieda 844 | 1 TVerb_Past_P baniticateda 845 | 1 TVerb_Past_P haustisteda 846 | 1 TVerb_Past_P pipistrateda 847 | 1 TVerb_Past_P povifieda 848 | 1 TVerb_Past_P stribereda 849 | 1 TVerb_Past_P ningeda 850 | 1 TVerb_Past_P mafflitateda 851 | 1 TVerb_Past_P mafflicateda 852 | 1 TVerb_Past_P pleddisateda 853 | 1 TVerb_Past_P droverateda 854 | 1 TVerb_Past_P nulpeneda 855 | 1 TVerb_Past_P thralleda 856 | 1 TVerb_Past_P maffliceda 857 | 1 TVerb_Past_P yosseda 858 | 1 TVerb_Past_P mespicateda 859 | 1 TVerb_Past_P spooleda 860 | 1 TVerb_Past_P frematizeda 861 | 1 TVerb_Past_P disfrinseda 862 | 1 TVerb_Past_P kippigateda 863 | 1 TVerb_Past_P lawticianeda 864 | 1 TVerb_Past_P apturicizeda 865 | 1 TVerb_Past_P fossicateda 866 | 1 TVerb_Past_P blarpeda 867 | 1 TVerb_Past_P phonorizeda 868 | 1 TVerb_Past_P saultifieda 869 | 1 TVerb_Past_P simpuridateda 870 | 1 TVerb_Past_P jontifieda 871 | 1 TVerb_Past_P grateneda 872 | 1 TVerb_Past_P soskeda 873 | 1 TVerb_Past_P droleda 874 | 1 TVerb_Past_P chutcheda 875 | 1 TVerb_Past_P bolicizeda 876 | 1 TVerb_Past_P praimeda 877 | 1 TVerb_Past_P missipeda 878 | 1 TVerb_Past_P strubdifieda 879 | 1 TVerb_Past_P strozeda 880 | 1 TVerb_Past_P blamiciateda 881 | 1 TVerb_Past_P bubbeda 882 | 1 TVerb_Past_P botticateda 883 | 1 TVerb_Past_P missipateda 884 | 1 TVerb_Past_P croicerateda 885 | 1 TVerb_Past_P fallengizeda 886 | 1 TVerb_Past_P slurkeda 887 | 1 TVerb_Past_P traiveda 888 | 1 TVerb_Past_P conveameda 889 | 1 TVerb_Past_P dupressifieda 890 | 1 TVerb_Past_P phoisifieda 891 | 1 TVerb_Past_P zampeda 892 | 1 TVerb_Past_P dupresseda 893 | 1 TVerb_Past_P drauseda 894 | 1 TVerb_Past_P prockifieda 895 | 1 TVerb_Past_P annoriciateda 896 | 1 TVerb_Past_P jeapeda 897 | 1 TVerb_Past_P pandicateda 898 | 1 TVerb_Past_P tisseda 899 | 1 TVerb_Past_P draireda 900 | 1 TVerb_Past_P theslirateda 901 | 1 TVerb_Past_P midicateda 902 | 1 TVerb_Past_P clummeda 903 | 1 TVerb_Past_P kuestifieda 904 | 1 TVerb_Past_P peagifieda 905 | 1 TVerb_Past_P gluckeda 906 | 1 TVerb_Past_P madetizeda 907 | 1 TVerb_Past_P croileda 908 | 1 TVerb_Past_P lustodisteda 909 | 1 TVerb_Past_P tirastizeda 910 | 1 TVerb_Past_P penursifereda 911 | 1 TVerb_Past_P dismaffeda 912 | 1 TVerb_Past_P chozeda 913 | 1 TVerb_Past_P justatoreda 914 | 1 TVerb_Past_P fraineda 915 | 1 TVerb_Past_P thafassizeda 916 | 1 TVerb_Past_P enjurifieda 917 | 1 TVerb_Past_P joofeda 918 | 1 TVerb_Past_P likiciateda 919 | 1 TVerb_Past_P pouteratizeda 920 | 1 TVerb_Past_P irriticiateda 921 | 1 TVerb_Past_P blitcheda 922 | 1 TVerb_Past_P gangeristeda 923 | 1 TVerb_Past_P stickerateda 924 | 1 TVerb_Past_P jobsticatoreda 925 | 1 TVerb_Past_P scolterizeda 926 | 1 TVerb_Past_P shumpifieda 927 | 1 TVerb_Past_P heferateda 928 | 929 | 1 Verb_Comp_Pres_S engarifies 930 | 1 Verb_Comp_Pres_S kurches 931 | 1 Verb_Comp_Pres_S formanates 932 | 1 Verb_Comp_Pres_S crails 933 | 1 Verb_Comp_Pres_S lurchifies 934 | 1 Verb_Comp_Pres_S fanches 935 | 1 Verb_Comp_Pres_S prutifies 936 | 1 Verb_Comp_Pres_S squails 937 | 1 Verb_Comp_Pres_S nobs 938 | 1 Verb_Comp_Pres_S pactesses 939 | 1 Verb_Comp_Pres_S sedundifies 940 | 1 Verb_Comp_Pres_S sergatizes 941 | 1 Verb_Comp_Pres_S unseffs 942 | 1 Verb_Comp_Pres_S bules 943 | 1 Verb_Comp_Pres_S humicianes 944 | 1 Verb_Comp_Pres_S saunches 945 | 1 Verb_Comp_Pres_S thespires 946 | 1 Verb_Comp_Pres_S conturizes 947 | 1 Verb_Comp_Pres_S mangeres 948 | 1 Verb_Comp_Pres_S strovokicizes 949 | 1 Verb_Comp_Pres_S embarricates 950 | 1 Verb_Comp_Pres_S crins 951 | 952 | 1 Verb_Comp_Pres_P engarify 953 | 1 Verb_Comp_Pres_P kurche 954 | 1 Verb_Comp_Pres_P formanate 955 | 1 Verb_Comp_Pres_P crail 956 | 1 Verb_Comp_Pres_P lurchify 957 | 1 Verb_Comp_Pres_P fanch 958 | 1 Verb_Comp_Pres_P prutify 959 | 1 Verb_Comp_Pres_P squail 960 | 1 Verb_Comp_Pres_P nob 961 | 1 Verb_Comp_Pres_P pactess 962 | 1 Verb_Comp_Pres_P sedundify 963 | 1 Verb_Comp_Pres_P sergatize 964 | 1 Verb_Comp_Pres_P unseff 965 | 1 Verb_Comp_Pres_P bule 966 | 1 Verb_Comp_Pres_P humiciane 967 | 1 Verb_Comp_Pres_P saunch 968 | 1 Verb_Comp_Pres_P thespire 969 | 1 Verb_Comp_Pres_P conturize 970 | 1 Verb_Comp_Pres_P mangere 971 | 1 Verb_Comp_Pres_P strovokicize 972 | 1 Verb_Comp_Pres_P embarricate 973 | 1 Verb_Comp_Pres_P crin 974 | 975 | 1 IVerb_Pres_S fusilitates 976 | 1 IVerb_Pres_S remiffs 977 | 1 IVerb_Pres_S mangerizes 978 | 1 IVerb_Pres_S plessicates 979 | 1 IVerb_Pres_S wogs 980 | 1 IVerb_Pres_S forigates 981 | 1 IVerb_Pres_S vems 982 | 1 IVerb_Pres_S phanicizes 983 | 1 IVerb_Pres_S denaticates 984 | 1 IVerb_Pres_S gasts 985 | 1 IVerb_Pres_S peaserizes 986 | 1 IVerb_Pres_S piliciates 987 | 1 IVerb_Pres_S hurperates 988 | 1 IVerb_Pres_S ralms 989 | 1 IVerb_Pres_S charchs 990 | 1 IVerb_Pres_S filks 991 | 1 IVerb_Pres_S tawlerizes 992 | 1 IVerb_Pres_S onstigipates 993 | 1 IVerb_Pres_S rolves 994 | 1 IVerb_Pres_S rolves 995 | 1 IVerb_Pres_S offaldes 996 | 1 IVerb_Pres_S calikates 997 | 1 IVerb_Pres_S mauces 998 | 1 IVerb_Pres_S inthippenates 999 | 1 IVerb_Pres_S visifies 1000 | 1 IVerb_Pres_S froacheres 1001 | 1 IVerb_Pres_S madressizes 1002 | 1 IVerb_Pres_S benaticianes 1003 | 1 IVerb_Pres_S plessickes 1004 | 1 IVerb_Pres_S scupps 1005 | 1 IVerb_Pres_S framitatores 1006 | 1 IVerb_Pres_S thrombicizes 1007 | 1 IVerb_Pres_S portifies 1008 | 1 IVerb_Pres_S tirastes 1009 | 1 IVerb_Pres_S osmissifies 1010 | 1 IVerb_Pres_S peachicianes 1011 | 1 IVerb_Pres_S scoans 1012 | 1 IVerb_Pres_S amackates 1013 | 1 IVerb_Pres_S unplews 1014 | 1 IVerb_Pres_S jantomizes 1015 | 1 IVerb_Pres_S allacks 1016 | 1 IVerb_Pres_S thullamizes 1017 | 1 IVerb_Pres_S smeshs 1018 | 1 IVerb_Pres_S tismissates 1019 | 1 IVerb_Pres_S offaldicates 1020 | 1 IVerb_Pres_S briminates 1021 | 1 IVerb_Pres_S maksizes 1022 | 1 IVerb_Pres_S amberates 1023 | 1 IVerb_Pres_S amackes 1024 | 1 IVerb_Pres_S apturicizes 1025 | 1 IVerb_Pres_S barses 1026 | 1 IVerb_Pres_S mofustates 1027 | 1 IVerb_Pres_S broals 1028 | 1 IVerb_Pres_S fampenates 1029 | 1 IVerb_Pres_S thespirates 1030 | 1 IVerb_Pres_S parls 1031 | 1 IVerb_Pres_S thrubs 1032 | 1 IVerb_Pres_S tarses 1033 | 1 IVerb_Pres_S teeves 1034 | 1 IVerb_Pres_S revalternates 1035 | 1 IVerb_Pres_S noachates 1036 | 1 IVerb_Pres_S flapperates 1037 | 1 IVerb_Pres_S mues 1038 | 1 IVerb_Pres_S miticurizes 1039 | 1 IVerb_Pres_S saffles 1040 | 1 IVerb_Pres_S chorries 1041 | 1 IVerb_Pres_S excribits 1042 | 1 IVerb_Pres_S lurchifies 1043 | 1 IVerb_Pres_S sturks 1044 | 1 IVerb_Pres_S nestatates 1045 | 1 IVerb_Pres_S limbatizes 1046 | 1 IVerb_Pres_S rotectifies 1047 | 1 IVerb_Pres_S swarmicates 1048 | 1 IVerb_Pres_S prausifies 1049 | 1 IVerb_Pres_S elusifies 1050 | 1 IVerb_Pres_S spectifies 1051 | 1 IVerb_Pres_S annoriciates 1052 | 1 IVerb_Pres_S elupizes 1053 | 1 IVerb_Pres_S lactissates 1054 | 1 IVerb_Pres_S streachifies 1055 | 1 IVerb_Pres_S mattakates 1056 | 1 IVerb_Pres_S ballicizes 1057 | 1 IVerb_Pres_S dupps 1058 | 1 IVerb_Pres_S mofustes 1059 | 1 IVerb_Pres_S yawls 1060 | 1 IVerb_Pres_S striberates 1061 | 1 IVerb_Pres_S treaves 1062 | 1 IVerb_Pres_S mermillates 1063 | 1 IVerb_Pres_S prottizes 1064 | 1 IVerb_Pres_S wolarifies 1065 | 1 IVerb_Pres_S gratenizes 1066 | 1 IVerb_Pres_S intrinses 1067 | 1 IVerb_Pres_S skeezes 1068 | 1 IVerb_Pres_S rawns 1069 | 1 IVerb_Pres_S tirastifies 1070 | 1 IVerb_Pres_S yolves 1071 | 1 IVerb_Pres_S bunessistes 1072 | 1 IVerb_Pres_S damerticates 1073 | 1 IVerb_Pres_S peagerizes 1074 | 1 IVerb_Pres_S stoppistes 1075 | 1 IVerb_Pres_S engaries 1076 | 1 IVerb_Pres_S bottistes 1077 | 1 IVerb_Pres_S bactivitates 1078 | 1 IVerb_Pres_S greanerizes 1079 | 1 IVerb_Pres_S duppifies 1080 | 1 IVerb_Pres_S zurches 1081 | 1 IVerb_Pres_S stroves 1082 | 1 IVerb_Pres_S povicates 1083 | 1 IVerb_Pres_S lavoidifies 1084 | 1 IVerb_Pres_S garfs 1085 | 1 IVerb_Pres_S crotifies 1086 | 1 IVerb_Pres_S wandifies 1087 | 1 IVerb_Pres_S pouteres 1088 | 1089 | 1 IVerb_Pres_P fusilitate 1090 | 1 IVerb_Pres_P remiff 1091 | 1 IVerb_Pres_P mangerize 1092 | 1 IVerb_Pres_P plessicate 1093 | 1 IVerb_Pres_P wog 1094 | 1 IVerb_Pres_P forigate 1095 | 1 IVerb_Pres_P vem 1096 | 1 IVerb_Pres_P phanicize 1097 | 1 IVerb_Pres_P denaticate 1098 | 1 IVerb_Pres_P gast 1099 | 1 IVerb_Pres_P peaserize 1100 | 1 IVerb_Pres_P piliciate 1101 | 1 IVerb_Pres_P hurperate 1102 | 1 IVerb_Pres_P ralm 1103 | 1 IVerb_Pres_P charch 1104 | 1 IVerb_Pres_P filk 1105 | 1 IVerb_Pres_P tawlerize 1106 | 1 IVerb_Pres_P onstigipate 1107 | 1 IVerb_Pres_P rolve 1108 | 1 IVerb_Pres_P rolve 1109 | 1 IVerb_Pres_P offalde 1110 | 1 IVerb_Pres_P calikate 1111 | 1 IVerb_Pres_P mauce 1112 | 1 IVerb_Pres_P inthippenate 1113 | 1 IVerb_Pres_P visify 1114 | 1 IVerb_Pres_P froachere 1115 | 1 IVerb_Pres_P madressize 1116 | 1 IVerb_Pres_P benaticiane 1117 | 1 IVerb_Pres_P plessicke 1118 | 1 IVerb_Pres_P scupp 1119 | 1 IVerb_Pres_P framitatore 1120 | 1 IVerb_Pres_P thrombicize 1121 | 1 IVerb_Pres_P portify 1122 | 1 IVerb_Pres_P tiraste 1123 | 1 IVerb_Pres_P osmissify 1124 | 1 IVerb_Pres_P peachiciane 1125 | 1 IVerb_Pres_P scoan 1126 | 1 IVerb_Pres_P amackate 1127 | 1 IVerb_Pres_P unplew 1128 | 1 IVerb_Pres_P jantomize 1129 | 1 IVerb_Pres_P allack 1130 | 1 IVerb_Pres_P thullamize 1131 | 1 IVerb_Pres_P smesh 1132 | 1 IVerb_Pres_P tismissate 1133 | 1 IVerb_Pres_P offaldicate 1134 | 1 IVerb_Pres_P briminate 1135 | 1 IVerb_Pres_P maksize 1136 | 1 IVerb_Pres_P amberate 1137 | 1 IVerb_Pres_P amacke 1138 | 1 IVerb_Pres_P apturicize 1139 | 1 IVerb_Pres_P barse 1140 | 1 IVerb_Pres_P mofustate 1141 | 1 IVerb_Pres_P broal 1142 | 1 IVerb_Pres_P fampenate 1143 | 1 IVerb_Pres_P thespirate 1144 | 1 IVerb_Pres_P parl 1145 | 1 IVerb_Pres_P thrub 1146 | 1 IVerb_Pres_P tarse 1147 | 1 IVerb_Pres_P teeve 1148 | 1 IVerb_Pres_P revalternate 1149 | 1 IVerb_Pres_P noachate 1150 | 1 IVerb_Pres_P flapperate 1151 | 1 IVerb_Pres_P mue 1152 | 1 IVerb_Pres_P miticurize 1153 | 1 IVerb_Pres_P saffle 1154 | 1 IVerb_Pres_P chorry 1155 | 1 IVerb_Pres_P excribit 1156 | 1 IVerb_Pres_P lurchify 1157 | 1 IVerb_Pres_P sturk 1158 | 1 IVerb_Pres_P nestatate 1159 | 1 IVerb_Pres_P limbatize 1160 | 1 IVerb_Pres_P rotectify 1161 | 1 IVerb_Pres_P swarmicate 1162 | 1 IVerb_Pres_P prausify 1163 | 1 IVerb_Pres_P elusify 1164 | 1 IVerb_Pres_P spectify 1165 | 1 IVerb_Pres_P annoriciate 1166 | 1 IVerb_Pres_P elupize 1167 | 1 IVerb_Pres_P lactissate 1168 | 1 IVerb_Pres_P streachify 1169 | 1 IVerb_Pres_P mattakate 1170 | 1 IVerb_Pres_P ballicize 1171 | 1 IVerb_Pres_P dupp 1172 | 1 IVerb_Pres_P mofuste 1173 | 1 IVerb_Pres_P yawl 1174 | 1 IVerb_Pres_P striberate 1175 | 1 IVerb_Pres_P treave 1176 | 1 IVerb_Pres_P mermillate 1177 | 1 IVerb_Pres_P prottize 1178 | 1 IVerb_Pres_P wolarify 1179 | 1 IVerb_Pres_P gratenize 1180 | 1 IVerb_Pres_P intrinse 1181 | 1 IVerb_Pres_P skeeze 1182 | 1 IVerb_Pres_P rawn 1183 | 1 IVerb_Pres_P tirastify 1184 | 1 IVerb_Pres_P yolve 1185 | 1 IVerb_Pres_P bunessiste 1186 | 1 IVerb_Pres_P damerticate 1187 | 1 IVerb_Pres_P peagerize 1188 | 1 IVerb_Pres_P stoppiste 1189 | 1 IVerb_Pres_P engary 1190 | 1 IVerb_Pres_P bottiste 1191 | 1 IVerb_Pres_P bactivitate 1192 | 1 IVerb_Pres_P greanerize 1193 | 1 IVerb_Pres_P duppify 1194 | 1 IVerb_Pres_P zurch 1195 | 1 IVerb_Pres_P strove 1196 | 1 IVerb_Pres_P povicate 1197 | 1 IVerb_Pres_P lavoidify 1198 | 1 IVerb_Pres_P garf 1199 | 1 IVerb_Pres_P crotify 1200 | 1 IVerb_Pres_P wandify 1201 | 1 IVerb_Pres_P poutere 1202 | 1203 | 1 TVerb_Pres_S gratenizes 1204 | 1 TVerb_Pres_S intrinses 1205 | 1 TVerb_Pres_S skeezes 1206 | 1 TVerb_Pres_S rawns 1207 | 1 TVerb_Pres_S tirastifies 1208 | 1 TVerb_Pres_S yolves 1209 | 1 TVerb_Pres_S bunessistes 1210 | 1 TVerb_Pres_S damerticates 1211 | 1 TVerb_Pres_S peagerizes 1212 | 1 TVerb_Pres_S stoppistes 1213 | 1 TVerb_Pres_S engaries 1214 | 1 TVerb_Pres_S bottistes 1215 | 1 TVerb_Pres_S bactivitates 1216 | 1 TVerb_Pres_S greanerizes 1217 | 1 TVerb_Pres_S duppifies 1218 | 1 TVerb_Pres_S zurches 1219 | 1 TVerb_Pres_S stroves 1220 | 1 TVerb_Pres_S povicates 1221 | 1 TVerb_Pres_S lavoidifies 1222 | 1 TVerb_Pres_S garfs 1223 | 1 TVerb_Pres_S crotifies 1224 | 1 TVerb_Pres_S wandifies 1225 | 1 TVerb_Pres_S pouteres 1226 | 1 TVerb_Pres_S rebinks 1227 | 1 TVerb_Pres_S dafes 1228 | 1 TVerb_Pres_S bullegates 1229 | 1 TVerb_Pres_S enjurifies 1230 | 1 TVerb_Pres_S visifies 1231 | 1 TVerb_Pres_S gumitifies 1232 | 1 TVerb_Pres_S baniticates 1233 | 1 TVerb_Pres_S haustistes 1234 | 1 TVerb_Pres_S pipistrates 1235 | 1 TVerb_Pres_S povifies 1236 | 1 TVerb_Pres_S striberes 1237 | 1 TVerb_Pres_S nings 1238 | 1 TVerb_Pres_S mafflitates 1239 | 1 TVerb_Pres_S mafflicates 1240 | 1 TVerb_Pres_S pleddisates 1241 | 1 TVerb_Pres_S droverates 1242 | 1 TVerb_Pres_S nulpens 1243 | 1 TVerb_Pres_S thralls 1244 | 1 TVerb_Pres_S mafflices 1245 | 1 TVerb_Pres_S yosses 1246 | 1 TVerb_Pres_S mespicates 1247 | 1 TVerb_Pres_S spools 1248 | 1 TVerb_Pres_S frematizes 1249 | 1 TVerb_Pres_S disfrinses 1250 | 1 TVerb_Pres_S kippigates 1251 | 1 TVerb_Pres_S lawticianes 1252 | 1 TVerb_Pres_S apturicizes 1253 | 1 TVerb_Pres_S fossicates 1254 | 1 TVerb_Pres_S blarps 1255 | 1 TVerb_Pres_S phonorizes 1256 | 1 TVerb_Pres_S saultifies 1257 | 1 TVerb_Pres_S simpuridates 1258 | 1 TVerb_Pres_S jontifies 1259 | 1 TVerb_Pres_S gratenes 1260 | 1 TVerb_Pres_S sosks 1261 | 1 TVerb_Pres_S drols 1262 | 1 TVerb_Pres_S chutches 1263 | 1 TVerb_Pres_S bolicizes 1264 | 1 TVerb_Pres_S praims 1265 | 1 TVerb_Pres_S missipes 1266 | 1 TVerb_Pres_S strubdifies 1267 | 1 TVerb_Pres_S strozes 1268 | 1 TVerb_Pres_S blamiciates 1269 | 1 TVerb_Pres_S bubs 1270 | 1 TVerb_Pres_S botticates 1271 | 1 TVerb_Pres_S missipates 1272 | 1 TVerb_Pres_S croicerates 1273 | 1 TVerb_Pres_S fallengizes 1274 | 1 TVerb_Pres_S slurks 1275 | 1 TVerb_Pres_S traives 1276 | 1 TVerb_Pres_S conveams 1277 | 1 TVerb_Pres_S dupressifies 1278 | 1 TVerb_Pres_S phoisifies 1279 | 1 TVerb_Pres_S zamps 1280 | 1 TVerb_Pres_S dupresses 1281 | 1 TVerb_Pres_S drauses 1282 | 1 TVerb_Pres_S prockifies 1283 | 1 TVerb_Pres_S annoriciates 1284 | 1 TVerb_Pres_S jeaps 1285 | 1 TVerb_Pres_S pandicates 1286 | 1 TVerb_Pres_S tisses 1287 | 1 TVerb_Pres_S drairs 1288 | 1 TVerb_Pres_S theslirates 1289 | 1 TVerb_Pres_S midicates 1290 | 1 TVerb_Pres_S clums 1291 | 1 TVerb_Pres_S kuestifies 1292 | 1 TVerb_Pres_S peagifies 1293 | 1 TVerb_Pres_S glucks 1294 | 1 TVerb_Pres_S madetizes 1295 | 1 TVerb_Pres_S croils 1296 | 1 TVerb_Pres_S lustodistes 1297 | 1 TVerb_Pres_S tirastizes 1298 | 1 TVerb_Pres_S penursiferes 1299 | 1 TVerb_Pres_S dismaffes 1300 | 1 TVerb_Pres_S chozes 1301 | 1 TVerb_Pres_S justatores 1302 | 1 TVerb_Pres_S frains 1303 | 1 TVerb_Pres_S thafassizes 1304 | 1 TVerb_Pres_S enjurifies 1305 | 1 TVerb_Pres_S joofs 1306 | 1 TVerb_Pres_S likiciates 1307 | 1 TVerb_Pres_S pouteratizes 1308 | 1 TVerb_Pres_S irriticiates 1309 | 1 TVerb_Pres_S blitches 1310 | 1 TVerb_Pres_S gangeristes 1311 | 1 TVerb_Pres_S stickerates 1312 | 1 TVerb_Pres_S jobsticatores 1313 | 1 TVerb_Pres_S scolterizes 1314 | 1 TVerb_Pres_S shumpifies 1315 | 1 TVerb_Pres_S heferates 1316 | 1317 | 1 TVerb_Pres_P gratenize 1318 | 1 TVerb_Pres_P intrinse 1319 | 1 TVerb_Pres_P skeez 1320 | 1 TVerb_Pres_P rawn 1321 | 1 TVerb_Pres_P tirastify 1322 | 1 TVerb_Pres_P yolve 1323 | 1 TVerb_Pres_P bunessiste 1324 | 1 TVerb_Pres_P damerticate 1325 | 1 TVerb_Pres_P peagerize 1326 | 1 TVerb_Pres_P stoppiste 1327 | 1 TVerb_Pres_P engary 1328 | 1 TVerb_Pres_P bottiste 1329 | 1 TVerb_Pres_P bactivitate 1330 | 1 TVerb_Pres_P greanerize 1331 | 1 TVerb_Pres_P duppify 1332 | 1 TVerb_Pres_P zurch 1333 | 1 TVerb_Pres_P strove 1334 | 1 TVerb_Pres_P povicate 1335 | 1 TVerb_Pres_P lavoidify 1336 | 1 TVerb_Pres_P garf 1337 | 1 TVerb_Pres_P crotify 1338 | 1 TVerb_Pres_P wandify 1339 | 1 TVerb_Pres_P poutere 1340 | 1 TVerb_Pres_P rebink 1341 | 1 TVerb_Pres_P dafe 1342 | 1 TVerb_Pres_P bullegate 1343 | 1 TVerb_Pres_P enjurify 1344 | 1 TVerb_Pres_P visify 1345 | 1 TVerb_Pres_P gumitify 1346 | 1 TVerb_Pres_P baniticate 1347 | 1 TVerb_Pres_P haustiste 1348 | 1 TVerb_Pres_P pipistrate 1349 | 1 TVerb_Pres_P povify 1350 | 1 TVerb_Pres_P stribere 1351 | 1 TVerb_Pres_P ning 1352 | 1 TVerb_Pres_P mafflitate 1353 | 1 TVerb_Pres_P mafflicate 1354 | 1 TVerb_Pres_P pleddisate 1355 | 1 TVerb_Pres_P droverate 1356 | 1 TVerb_Pres_P nulpen 1357 | 1 TVerb_Pres_P thrall 1358 | 1 TVerb_Pres_P mafflice 1359 | 1 TVerb_Pres_P yoss 1360 | 1 TVerb_Pres_P mespicate 1361 | 1 TVerb_Pres_P spool 1362 | 1 TVerb_Pres_P frematize 1363 | 1 TVerb_Pres_P disfrinse 1364 | 1 TVerb_Pres_P kippigate 1365 | 1 TVerb_Pres_P lawticiane 1366 | 1 TVerb_Pres_P apturicize 1367 | 1 TVerb_Pres_P fossicate 1368 | 1 TVerb_Pres_P blarp 1369 | 1 TVerb_Pres_P phonorize 1370 | 1 TVerb_Pres_P saultify 1371 | 1 TVerb_Pres_P simpuridate 1372 | 1 TVerb_Pres_P jontify 1373 | 1 TVerb_Pres_P gratene 1374 | 1 TVerb_Pres_P sosk 1375 | 1 TVerb_Pres_P drol 1376 | 1 TVerb_Pres_P chutch 1377 | 1 TVerb_Pres_P bolicize 1378 | 1 TVerb_Pres_P praim 1379 | 1 TVerb_Pres_P missipe 1380 | 1 TVerb_Pres_P strubdify 1381 | 1 TVerb_Pres_P stroze 1382 | 1 TVerb_Pres_P blamiciate 1383 | 1 TVerb_Pres_P bub 1384 | 1 TVerb_Pres_P botticate 1385 | 1 TVerb_Pres_P missipate 1386 | 1 TVerb_Pres_P croicerate 1387 | 1 TVerb_Pres_P fallengize 1388 | 1 TVerb_Pres_P slurk 1389 | 1 TVerb_Pres_P traive 1390 | 1 TVerb_Pres_P conveam 1391 | 1 TVerb_Pres_P dupressify 1392 | 1 TVerb_Pres_P phoisify 1393 | 1 TVerb_Pres_P zamp 1394 | 1 TVerb_Pres_P dupresse 1395 | 1 TVerb_Pres_P drause 1396 | 1 TVerb_Pres_P prockify 1397 | 1 TVerb_Pres_P annoriciate 1398 | 1 TVerb_Pres_P jeap 1399 | 1 TVerb_Pres_P pandicate 1400 | 1 TVerb_Pres_P tisse 1401 | 1 TVerb_Pres_P drair 1402 | 1 TVerb_Pres_P theslirate 1403 | 1 TVerb_Pres_P midicate 1404 | 1 TVerb_Pres_P clum 1405 | 1 TVerb_Pres_P kuestify 1406 | 1 TVerb_Pres_P peagify 1407 | 1 TVerb_Pres_P gluck 1408 | 1 TVerb_Pres_P madetize 1409 | 1 TVerb_Pres_P croil 1410 | 1 TVerb_Pres_P lustodiste 1411 | 1 TVerb_Pres_P tirastize 1412 | 1 TVerb_Pres_P penursifere 1413 | 1 TVerb_Pres_P dismaffe 1414 | 1 TVerb_Pres_P choze 1415 | 1 TVerb_Pres_P justatore 1416 | 1 TVerb_Pres_P frain 1417 | 1 TVerb_Pres_P thafassize 1418 | 1 TVerb_Pres_P enjurify 1419 | 1 TVerb_Pres_P joof 1420 | 1 TVerb_Pres_P likiciate 1421 | 1 TVerb_Pres_P pouteratize 1422 | 1 TVerb_Pres_P irriticiate 1423 | 1 TVerb_Pres_P blitch 1424 | 1 TVerb_Pres_P gangeriste 1425 | 1 TVerb_Pres_P stickerate 1426 | 1 TVerb_Pres_P jobsticatore 1427 | 1 TVerb_Pres_P scolterize 1428 | 1 TVerb_Pres_P shumpify 1429 | 1 TVerb_Pres_P heferate 1430 | 1431 | 1 Prep ca 1432 | 1 Prep ma 1433 | 1 Prep hi 1434 | 1 Prep lu 1435 | 1436 | 1 CC da 1437 | 1438 | 1 Comp sa 1439 | 1440 | 1 Rel rel 1441 | 1442 | 1 Subj sub 1443 | 1444 | 1 Obj ob 1445 | 1446 | 1 Pronoun_S bo 1447 | 1 Pronoun_S se 1448 | 1 Pronoun_S pi 1449 | 1 Pronoun_S me 1450 | 1 Pronoun_P si 1451 | 1 Pronoun_P la 1452 | 1 Pronoun_S ja 1453 | -------------------------------------------------------------------------------- /data_gen/gen_data.sh: -------------------------------------------------------------------------------- 1 | python sample_sentences.py -g base-grammar.gr -n 100000 -O . -b True 2 | python permute_sentences.py -s sample_base-grammar.txt -O permuted_samples/ 3 | python make_splits.py -S permuted_samples/ -O permuted_splits/ -------------------------------------------------------------------------------- /data_gen/make_splits.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | import argparse 4 | 5 | def create_splits(sample_file, num_splits, train, test, dev, output_folder): 6 | if not os.path.exists(output_folder): 7 | os.mkdir(output_folder) 8 | sentence_file = open(sample_file, 'r') 9 | all_sentences = sentence_file.readlines() 10 | num_all_sent = len(all_sentences) 11 | grammar_name = sample_file[:-4].split("_")[-1] 12 | if not os.path.exists(os.path.join(output_folder, grammar_name)): 13 | os.mkdir(os.path.join(output_folder, grammar_name)) 14 | grammar_output = os.path.join(output_folder, grammar_name) 15 | for i in range(num_splits): 16 | start = int(i * (1/num_splits) * num_all_sent) 17 | end = int((i+1) * (1/num_splits) * num_all_sent) 18 | sentences = all_sentences[start:end] 19 | num_sent = len(sentences) 20 | trn_output = open(os.path.join(grammar_output, str(i) + ".trn"), 'w') 21 | tst_output = open(os.path.join(grammar_output, str(i) + ".tst"), 'w') 22 | dev_output = open(os.path.join(grammar_output, str(i) + ".dev"), 'w') 23 | trn_split = sentences[:int(train*num_sent)] 24 | tst_split = sentences[int(train*num_sent):int((train + test)*num_sent)] 25 | dev_split = sentences[int((train + test)*num_sent):] 26 | for s in trn_split: 27 | trn_output.write(s) 28 | for s in tst_split: 29 | tst_output.write(s) 30 | for s in dev_split: 31 | dev_output.write(s) 32 | 33 | parser = argparse.ArgumentParser( 34 | description="Divide generated sentences into splits") 35 | 36 | parser.add_argument("-s", "--sample_file", type=str, default='', 37 | help="Path to sample file") 38 | parser.add_argument("-S", "--sample_folder", type=str, default='', 39 | help="Path to folder containing multiple sample files") 40 | parser.add_argument("-O", "--output_folder", type=str, 41 | help="Location of output files") 42 | parser.add_argument("-tr", "--train", type=float, default=0.8, 43 | help="Train proportion") 44 | parser.add_argument("-ts", "--test", type=float, default=0.1, 45 | help="Test proportion") 46 | parser.add_argument("-dv", "--dev", type=float, default=0.1, 47 | help="Dev proportion") 48 | parser.add_argument("-n", "--num_splits", type=int, default=10, 49 | help="Number of splits") 50 | 51 | args = parser.parse_args() 52 | 53 | assert(args.train + args.test + args.dev == 1.0) 54 | 55 | if args.sample_file == '' and args.sample_folder == '': 56 | print("Please provide sample files") 57 | elif args.sample_file != '' and args.sample_folder != '': 58 | print("Please provide either a single file OR a folder containing sample" 59 | " files") 60 | elif args.sample_file != '': 61 | create_splits(args.sample_file, args.num_splits, args.train, args.test, 62 | args.dev, args.output_folder) 63 | elif args.sample_folder != '': 64 | sample_files = [f for f in os.listdir( 65 | args.sample_folder) if f.endswith('.txt')] 66 | for s in sample_files: 67 | create_splits(os.path.join(args.sample_folder, s), args.num_splits, 68 | args.train, args.test, args.dev, args.output_folder) 69 | 70 | 71 | -------------------------------------------------------------------------------- /data_gen/permute_sentences.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import sys 4 | import random 5 | import copy 6 | 7 | def flip_as_needed(i, sentence): 8 | to_flip = [j + 1 for j in range(6) if (i >> j) & 1 == 1] 9 | s_split = sentence.split(" ") 10 | for j in range(len(s_split)): 11 | if s_split[j][0].isnumeric(): 12 | if int(s_split[j][0]) in to_flip: 13 | reversed_end = reversed_children(s_split[j+1:]) 14 | s_split = s_split[:j+1] + reversed_end 15 | else: 16 | continue 17 | return ' '.join(s_split).strip("\n") 18 | 19 | def reversed_children(sentence_part): 20 | children = [] 21 | bracket_stack = [] 22 | L_brack = '(' 23 | R_brack = ')' 24 | children_end = -1 25 | for i in range(len(sentence_part)): 26 | s = sentence_part[i] 27 | if s == L_brack: 28 | bracket_stack.append((L_brack, i)) 29 | elif s == R_brack: 30 | if len(bracket_stack) > 0: 31 | if bracket_stack[-1][0] == L_brack: 32 | opening = bracket_stack.pop() 33 | if len(bracket_stack) == 0: 34 | children.append(sentence_part[opening[1]:i+1]) 35 | else: 36 | children_end = i - 1 37 | break 38 | else: 39 | continue 40 | children_reversed = [] 41 | for c in children[::-1]: 42 | children_reversed += c 43 | return children_reversed + sentence_part[children_end:] 44 | 45 | def remove_bracketing(s): 46 | new_s = [] 47 | split_s = s.split(" ") 48 | i = 0 49 | while i < len(split_s): 50 | if split_s[i] == ")": 51 | i += 1 52 | elif split_s[i] == "(": 53 | i += 2 54 | else: 55 | new_s.append(split_s[i]) 56 | i += 1 57 | new_s.append(".") 58 | return ' '.join(new_s) 59 | 60 | def generate_sentence_file(i, sentences, output_file): 61 | output_f = open(output_file, 'w') 62 | for s in sentences: 63 | output_f.write(remove_bracketing(flip_as_needed(i, s)) + "\n") 64 | output_f.close() 65 | 66 | parser = argparse.ArgumentParser(description="Generate variants of sentences" 67 | " based on base grammar") 68 | 69 | parser.add_argument("-s", "--sentence_file", type=str, required=True, 70 | help="Path to base sentence file") 71 | 72 | parser.add_argument("-O", "--output_folder", type=str, required=True, 73 | help="Location of output folder") 74 | 75 | args = parser.parse_args() 76 | 77 | file = open(args.sentence_file, 'r') 78 | sentences = file.readlines() 79 | 80 | for i in range(64): 81 | if not os.path.exists(args.output_folder): 82 | os.mkdir(args.output_folder) 83 | grammar_name = format(i, '06b')[::-1] 84 | output_file = os.path.join(args.output_folder, 85 | "sample_" + grammar_name + ".txt") 86 | generate_sentence_file(i, sentences, output_file) -------------------------------------------------------------------------------- /data_gen/sample_sentences.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import random 4 | 5 | 6 | class PCFG: 7 | """ 8 | PCFG to sample sentences from 9 | """ 10 | def __init__(self, grammar_file): 11 | self.rules = None 12 | self.change_rules = None 13 | self.load_rules(grammar_file) 14 | 15 | def load_rules(self, grammar_file): 16 | new_rules = {} 17 | change = {} 18 | g_file = open(grammar_file, 'r') 19 | lines = g_file.readlines() 20 | for l in lines: 21 | if l.startswith(('#', " ", "\t", "\n")) or len(l) < 1: 22 | continue 23 | else: 24 | if l.find("#") != -1: 25 | l = l[:l.find("#")] 26 | idx = -1 27 | if len(l.rstrip().split("\t")) == 3: 28 | weight, lhs, rhs = l.rstrip().split("\t") 29 | elif len(l.rstrip().split("\t")) == 4: 30 | weight, lhs, rhs, idx = l.rstrip().split("\t") 31 | if lhs not in new_rules.keys(): 32 | new_rules[lhs] = [] 33 | poss_rhs = new_rules[lhs] 34 | poss_rhs.append([rhs, float(weight)]) 35 | if idx != -1: 36 | change[lhs + "\t" + rhs] = idx 37 | for lhs, poss in new_rules.items(): 38 | total = 0 39 | for rhs in poss: 40 | total += rhs[1] 41 | for rhs in poss: 42 | rhs[1] /= total 43 | self.rules = new_rules 44 | self.change_rules = change 45 | 46 | def sample_sentence(self, max_expansions, bracketing): 47 | self.expansions = 0 48 | done = False 49 | sent = ["ROOT"] 50 | idx = 0 51 | while not done: 52 | if sent[idx] not in self.rules.keys(): 53 | idx += 1 54 | if idx >= len(sent): 55 | done = True 56 | continue 57 | else: 58 | replace, change_idx = self.expand(sent[idx]) 59 | if bracketing: 60 | if change_idx == -1: 61 | sent = (sent[:idx] 62 | + ["(", sent[idx]] + replace + [")"] 63 | + sent[idx + 1:]) 64 | else: 65 | sent = (sent[:idx] 66 | + ["(", change_idx + sent[idx]] + replace + [")"] 67 | + sent[idx + 1:]) 68 | else: 69 | sent = sent[:idx] + replace + sent[idx + 1:] 70 | self.expansions += 1 71 | if bracketing: 72 | idx += 2 73 | if self.expansions > max_expansions: 74 | done = True 75 | if idx >= len(sent): 76 | done = True 77 | if self.expansions > max_expansions: 78 | for idx in range(len(sent)): 79 | if not bracketing: 80 | if sent[idx] in self.rules.keys(): 81 | sent[idx] = "..." 82 | else: 83 | if sent[idx] in self.rules.keys() and sent[idx - 1] != "(": 84 | sent[idx] = "..." 85 | return ' '.join(sent) 86 | 87 | def expand(self, symbol): 88 | poss = self.rules[symbol] 89 | sample = random.random() 90 | val = 0.0 91 | rhs = "" 92 | idx = -1 93 | for p in poss: 94 | val += p[1] 95 | if sample <= val: 96 | if symbol + "\t" + p[0] in self.change_rules.keys(): 97 | idx = self.change_rules[symbol + "\t" + p[0]] 98 | rhs = p[0] 99 | break 100 | return rhs.split(" "), idx 101 | 102 | def sample_sentences(grammar_file, n, m, output_folder, bracketing): 103 | if not os.path.exists(output_folder): 104 | os.mkdir(output_folder) 105 | grammar_name = grammar_file[:-3].split("/")[-1] 106 | output_file = open(os.path.join(output_folder, 107 | "sample_" + grammar_name + ".txt") , 'w') 108 | grammar = PCFG(grammar_file) 109 | for i in range(n): 110 | output_file.write(grammar.sample_sentence(m, bracketing) + "\n") 111 | 112 | 113 | parser = argparse.ArgumentParser(description="Sample sentences from PCFG") 114 | 115 | parser.add_argument("-g", "--grammar_file", type=str, default='', 116 | help="Path to grammar file") 117 | parser.add_argument("-G", "--grammar_folder", type=str, default='', 118 | help="Path to folder containing multiple grammar files") 119 | parser.add_argument("-n", "--number_samples", type=int, required=True, 120 | help="Number of sentences to sample") 121 | parser.add_argument("-m", "--max_expansions", type=int, default=400, 122 | help="Max number of expansions performed") 123 | parser.add_argument("-O", "--output_folder", type=str, 124 | help="Location of output files") 125 | parser.add_argument("-b", "--bracketing", type=bool, 126 | help="Include bracketing of constituents") 127 | 128 | args = parser.parse_args() 129 | 130 | if args.grammar_file == '' and args.grammar_folder == '': 131 | print("Please provide grammar files") 132 | elif args.grammar_file != '' and args.grammar_folder != '': 133 | print("Please provide either a single file OR a folder containing grammar" 134 | " files") 135 | elif args.grammar_file != '': 136 | sample_sentences(args.grammar_file, args.number_samples, 137 | args.max_expansions, args.output_folder, args.bracketing) 138 | elif args.grammar_folder != '': 139 | grammar_files = [f for f in os.listdir( 140 | args.grammar_folder) if f.endswith('.gr')] 141 | for g in grammar_files: 142 | sample_sentences(os.path.join(args.grammar_folder, g), 143 | args.number_samples, args.max_expansions, args.output_folder, 144 | args.bracketing) 145 | -------------------------------------------------------------------------------- /get_sentence_scores.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | 4 | def total_sentence_score(words): 5 | score = 0 6 | for w in words: 7 | score += float(w.split(' ')[1][1:-1]) 8 | return score 9 | 10 | parser = argparse.ArgumentParser( 11 | description = "Get sentence scores from eval output file") 12 | 13 | parser.add_argument("-i", "--input_file", type=str, required=True, 14 | help="Path to input file") 15 | 16 | parser.add_argument("-O", "--output_folder", type=str, required=True, 17 | help="Location of output folder") 18 | 19 | args = parser.parse_args() 20 | 21 | if not os.path.exists(args.output_folder): 22 | os.mkdir(args.output_folder) 23 | 24 | file = open(args.input_file, 'r') 25 | all_lines = file.readlines() 26 | all_lines = [l for l in all_lines if len(l.split('|')) > 3] 27 | word_score_lines = [l.split('|')[3][1:] for l in all_lines if l.split( 28 | '|')[3].split(' ')[1].isnumeric()] 29 | word_score_lines.sort(key=lambda a:int(a.split(' ')[0])) 30 | full_text = "" 31 | for line in word_score_lines: 32 | full_text += ' '.join(line.split(' ')[1:]).strip('\n') + " " 33 | sentences = [] 34 | start = 0 35 | words = full_text.split("\t") 36 | for i in range(len(words)): 37 | if words[i].split(' ')[0] == "": 38 | sentences.append(words[start:i+1]) 39 | start = i+1 40 | sentence_scores = [] 41 | for s in sentences: 42 | sentence_scores.append(total_sentence_score(s)) 43 | grammar, split, dev_test, _ = args.input_file.split("/")[-1].split(".") 44 | if not os.path.exists(os.path.join(args.output_folder, grammar)): 45 | os.mkdir(os.path.join(args.output_folder, grammar)) 46 | output_file = open(os.path.join(args.output_folder, grammar, 47 | ".".join([split, dev_test, "txt"])), 'w') 48 | for s in sentence_scores: 49 | output_file.write(str(s) + "\n") 50 | -------------------------------------------------------------------------------- /results_analysis/mixed_model.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import sys 4 | import pandas as pd 5 | import numpy as np 6 | from statsmodels.regression.mixed_linear_model import MixedLM 7 | 8 | def read_scores_from_file(filename): 9 | score_file = open(filename, 'r') 10 | scores = score_file.readlines() 11 | indxd_scores = [] 12 | for i in range(len(scores)): 13 | indxd_scores.append((i, -1.0*float(scores[i].strip("\n")))) 14 | return indxd_scores 15 | 16 | def get_cross_interactions(attribs): 17 | cross = [] 18 | for i in range(len(attribs)): 19 | for j in range(i + 1, len(attribs)): 20 | cross.append(attribs[i] + "_" + attribs[j]) 21 | return cross 22 | 23 | def get_dataframe(file_location, output, attribs): 24 | columns = ['sent_id', 'score'] + attribs 25 | results_df = pd.DataFrame(columns=columns) 26 | print(results_df.head()) 27 | 28 | score_files = [os.path.join(file_location, f) for f in os.listdir( 29 | os.path.join(file_location)) if f.endswith('.txt')] 30 | 31 | for f in score_files: 32 | scores = read_scores_from_file(f) 33 | grammar_name = f.split("/")[-1].split("_")[0] 34 | attrib_vals = [] 35 | for c in grammar_name: 36 | if c == '0': 37 | attrib_vals.append(-1.0) 38 | elif c == '1': 39 | attrib_vals.append(1.0) 40 | for s in scores: 41 | to_append = {'sent_id':s[0], 'score':s[1]} 42 | for i in range(len(attribs)): 43 | to_append[attribs[i]] = attrib_vals[i] 44 | results_df = results_df.append(to_append, ignore_index=True) 45 | 46 | results_df = results_df.astype({'sent_id':'int64'}) 47 | if len(output) > 0: 48 | results_df.to_csv(output, index=False) 49 | return results_df 50 | 51 | parser = argparse.ArgumentParser( 52 | description="Perform analysis with mixed effects model") 53 | 54 | parser.add_argument("-f", "--file_location", type=str, default='', 55 | help="Path to folder containing files") 56 | 57 | parser.add_argument("-c", "--csv_location", type=str, default='', 58 | help="Path to csv to load") 59 | 60 | parser.add_argument("-o", "--output_csv", type=str, default='', 61 | help="Location to save CSV") 62 | 63 | parser.add_argument("-i", "--include_interactions", type=bool, default=False, 64 | help="Whether interaction terms are included") 65 | 66 | args = parser.parse_args() 67 | 68 | attribs = ['S','VP','comp', 'PP', 'NP', 'rel'] 69 | 70 | if len(args.file_location) > 0 and len(args.csv_location) > 0: 71 | print("Error: Either provide CSV or score files, not both") 72 | results_df = None 73 | elif len(args.file_location) > 0: 74 | results_df = get_dataframe(args.file_location, args.output_csv, 75 | attribs) 76 | elif len(args.csv_location) > 0: 77 | results_df = pd.read_csv(args.csv_location) 78 | else: 79 | print("Error: Provide either CSV or score files") 80 | results_df = None 81 | 82 | if args.include_interactions: 83 | cross_attribs = get_cross_interactions(attribs) 84 | for cross in cross_attribs: 85 | attrib1, attrib2 = cross.split("_") 86 | new_column = [] 87 | for index, row in results_df.iterrows(): 88 | new_column.append(row[attrib1] * row[attrib2]) 89 | results_df[cross] = new_column 90 | 91 | fml = "score ~ " 92 | if args.include_interactions: 93 | all_attribs = attribs + cross_attribs 94 | else: 95 | all_attribs = attribs 96 | for a in all_attribs: 97 | fml += a + "+" 98 | 99 | fml = fml.strip("+") 100 | 101 | mod_lme = MixedLM.from_formula(fml, groups=results_df["sent_id"], 102 | data=results_df) 103 | mod_lme = mod_lme.fit() 104 | 105 | print(mod_lme.summary()) -------------------------------------------------------------------------------- /results_analysis/permutation_test.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import random 4 | import csv 5 | 6 | def permutation_test(v1, v2, s): 7 | assert(len(v1) == len(v2)) 8 | diff_av = abs(sum([(v1[i] - v2[i]) for i in range(len(v1))])/len(v1)) 9 | swapped_av = 0 10 | for i in range(s): 11 | diff_tot = 0 12 | for j in range(len(v1)): 13 | diff = v1[j] - v2[j] 14 | flip = random.randint(0,1) 15 | if flip: 16 | diff = -diff 17 | diff_tot += diff 18 | swapped_diff_av = abs(diff_tot/len(v1)) 19 | if swapped_diff_av >= diff_av: 20 | swapped_av += 1 21 | return swapped_av/s 22 | 23 | parser = argparse.ArgumentParser(description="Perform permutation tests") 24 | 25 | parser.add_argument("-f", "--file_location", type=str, default='', 26 | help="Path to folder containing files") 27 | 28 | parser.add_argument("-s", "--number_samples", type=int, default=10000, 29 | help="Number of permutations to sample") 30 | 31 | parser.add_argument("-l", "--file_list", type=str, default='', 32 | help="Two files, separated by commas, to compare") 33 | 34 | parser.add_argument("-O", "--output_folder", type=str, default='', 35 | help="Location of output file") 36 | 37 | args = parser.parse_args() 38 | 39 | if len(args.file_location) > 0: 40 | if len(args.output_folder) > 0: 41 | output_file = open(os.path.join(args.output_folder, 42 | 'perm_test_results.csv'), 'w') 43 | fieldnames = ['Grammar'] + [format(i, '06b') for i in range(64)] 44 | writer = csv.DictWriter(output_file, fieldnames=fieldnames) 45 | writer.writeheader() 46 | else: 47 | writer = None 48 | 49 | for i in range(64): 50 | grammar_i = format(i, '06b') 51 | file_i = open(os.path.join(args.file_location, 52 | grammar_i + "_scores.txt"), 'r') 53 | lines_i = file_i.readlines() 54 | scores_i = [float(l.strip("\n")) for l in lines_i] 55 | i_dict = {"Grammar":grammar_i} 56 | for j in range(i+1, 64): 57 | grammar_j = format(j, '06b') 58 | file_j = open(os.path.join(args.file_location, 59 | grammar_j + "_scores.txt"), 'r') 60 | lines_j = file_j.readlines() 61 | scores_j = [float(l.strip("\n")) for l in lines_j] 62 | if permutation_test(scores_i, scores_j, args.number_samples) < 0.05: 63 | if (-1 * sum(scores_j)/len(scores_j) 64 | > -1 * sum(scores_i)/len(scores_i)): 65 | print("SIGNIFICANT:", grammar_j, ">", grammar_i) 66 | if len(args.output_folder) > 0: 67 | i_dict[grammar_j] = "<" 68 | else: 69 | print("SIGNIFICANT:", grammar_i, ">", grammar_j) 70 | if len(args.output_folder) > 0: 71 | i_dict[grammar_j] = ">" 72 | else: 73 | if len(args.output_folder) > 0: 74 | i_dict[grammar_j] = "x" 75 | if len(args.output_folder) > 0: 76 | writer.write(i_dict) 77 | elif len(args.file_list) > 0 and len(args.file_list.split(",")) == 2: 78 | filename_a = args.file_list.split(",")[0] 79 | filename_b = args.file_list.split(",")[1] 80 | lines_a = open(filename_a, 'r').readlines() 81 | lines_b = open(filename_b, 'r').readlines() 82 | scores_a = [float(l.strip("\n")) for l in lines_a] 83 | scores_b = [float(l.strip("\n")) for l in lines_b] 84 | print(permutation_test(scores_b, scores_b, args.number_samples)) -------------------------------------------------------------------------------- /run_all_jobs.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import argparse 3 | 4 | parser = argparse.ArgumentParser(description="Run all jobs") 5 | 6 | parser.add_argument("-s", "--num_splits", type=int, default=10, 7 | help="Number of splits") 8 | parser.add_argument("-n", "--num_choices", type=int, default=6, 9 | help="Number of choice points") 10 | parser.add_argument("--submission_command", type=str, default="", 11 | help="Command used to submit jobs to HPC system") 12 | 13 | args = parser.parse_args() 14 | 15 | for i in range(2 ** args.num_choices): 16 | grammar = format(i, '0' + str(args.num_choices) + 'b')[::-1] 17 | for j in range(args.num_splits): 18 | subprocess.call(args.submission_command 19 | + " ./conlang-grammar/train_lm_transformer.sh " 20 | + ' '.join([str(grammar), str(j)]), shell=True) 21 | subprocess.call(args.submission_command 22 | + " ./conlang-grammar/train_lm_lstm.sh " 23 | + ' '.join([str(grammar), str(j)]), shell=True) -------------------------------------------------------------------------------- /train_lm_lstm.sh: -------------------------------------------------------------------------------- 1 | 2 | #!/bin/bash 3 | 4 | GRAMMAR=$1 5 | SPLIT=$2 6 | 7 | fairseq-preprocess --only-source --trainpref "/data_gen/permuted_splits/${GRAMMAR}/${SPLIT}.trn" --validpref "/data_gen/permuted_splits/${GRAMMAR}/${SPLIT}.dev" --testpref "/data_gen/permuted_splits/${GRAMMAR}/${SPLIT}.tst" --destdir "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --workers 20 8 | 9 | fairseq-train --task language_modeling "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --save-dir "/checkpoints/${GRAMMAR}/${SPLIT}-lstm" --arch lstm_lm_small --share-decoder-input-output-embed --dropout 0.3 --optimizer adam --adam-betas '(0.9,0.98)' --weight-decay 0.01 --lr 0.0005 --lr-scheduler inverse_sqrt --warmup-updates 4000 --clip-norm 0.0 --warmup-init-lr 1e-07 --tokens-per-sample 512 --sample-break-mode none --max-tokens 2048 --update-freq 16 --patience 5 --max-update 10000 --no-epoch-checkpoints --no-last-checkpoints 10 | 11 | fairseq-eval-lm "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --path "/checkpoints/${GRAMMAR}/${SPLIT}-lstm/checkpoint_best.pt" --tokens-per-sample 512 --gen-subset "valid" --output-word-probs --quiet 2> "/lstm-results/${GRAMMAR}.${SPLIT}.dev.txt" 12 | 13 | fairseq-eval-lm "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --path "/checkpoints/${GRAMMAR}/${SPLIT}-lstm/checkpoint_best.pt" --tokens-per-sample 512 --gen-subset "test" --output-word-probs --quiet 2> "/lstm-results/${GRAMMAR}.${SPLIT}.test.txt" 14 | 15 | python get_sentence_scores.py -i "/lstm-results/${GRAMMAR}.${SPLIT}.test.txt" -O "/sentence_scores_lstm/" 16 | -------------------------------------------------------------------------------- /train_lm_transformer.sh: -------------------------------------------------------------------------------- 1 | 2 | #!/bin/bash 3 | 4 | GRAMMAR=$1 5 | SPLIT=$2 6 | 7 | fairseq-preprocess --only-source --trainpref "/data_gen/permuted_splits/${GRAMMAR}/${SPLIT}.trn" --validpref "/data_gen/permuted_splits/${GRAMMAR}/${SPLIT}.dev" --testpref "/data_gen/permuted_splits/${GRAMMAR}/${SPLIT}.tst" --destdir "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --workers 20 8 | 9 | fairseq-train --task language_modeling "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --save-dir "/checkpoints/${GRAMMAR}/${SPLIT}-transformer" --arch transformer_lm_small --share-decoder-input-output-embed --dropout 0.3 --optimizer adam --adam-betas '(0.9,0.98)' --weight-decay 0.01 --lr 0.0005 --lr-scheduler inverse_sqrt --warmup-updates 4000 --clip-norm 0.0 --warmup-init-lr 1e-07 --tokens-per-sample 512 --sample-break-mode none --max-tokens 2048 --update-freq 16 --patience 5 --max-update 10000 --no-epoch-checkpoints --no-last-checkpoints 10 | 11 | fairseq-eval-lm "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --path "/checkpoints/${GRAMMAR}/${SPLIT}-transformer/checkpoint_best.pt" --tokens-per-sample 512 --gen-subset "valid" --output-word-probs --quiet 2> "/trans-results/${GRAMMAR}.${SPLIT}.dev.txt" 12 | 13 | fairseq-eval-lm "/data-bin/${GRAMMAR}/${SPLIT}-dataset" --path "/checkpoints/${GRAMMAR}/${SPLIT}-transformer/checkpoint_best.pt" --tokens-per-sample 512 --gen-subset "test" --output-word-probs --quiet 2> "/trans-results/${GRAMMAR}.${SPLIT}.test.txt" 14 | 15 | python get_sentence_scores.py -i "/trans-results/${GRAMMAR}.${SPLIT}.test.txt" -O "/trans_sentence_scores/" 16 | --------------------------------------------------------------------------------