├── .gitattributes ├── .gitignore ├── .ptignore ├── EVALB ├── COLLINS.prm ├── LICENSE ├── Makefile ├── README ├── evalb ├── evalb.c ├── new.prm ├── sample │ ├── sample.gld │ ├── sample.prm │ ├── sample.rsl │ └── sample.tst └── tgrep_proc.prl ├── LICENSE ├── Ordered_Memory_Slides.pdf ├── README.md ├── data ├── listops │ ├── __init__.py │ ├── base.py │ ├── load_listops_data.py │ ├── make_data.py │ ├── test_d20s.tsv │ └── train_d20s.tsv └── propositionallogic │ ├── __init__.py │ ├── generate_neg_set_data.py │ ├── test0 │ ├── test1 │ ├── test10 │ ├── test11 │ ├── test12 │ ├── test2 │ ├── test3 │ ├── test4 │ ├── test5 │ ├── test6 │ ├── test7 │ ├── test8 │ ├── test9 │ ├── train0 │ ├── train1 │ ├── train10 │ ├── train11 │ ├── train12 │ ├── train2 │ ├── train3 │ ├── train4 │ ├── train5 │ ├── train6 │ ├── train7 │ ├── train8 │ └── train9 ├── listops.py ├── ordered_memory.py ├── proplog.py ├── requirements.txt ├── sentiment.py └── utils ├── __init__.py ├── hinton.py ├── listops_data.py ├── locked_dropout.py └── utils.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | core.* 2 | 3 | *.hdf5 4 | *.pt 5 | .vector_cache 6 | .data 7 | 8 | *.xml 9 | *.iml 10 | 11 | 12 | # vim and gedit cache: 13 | *.swp 14 | *.swo 15 | *.swn 16 | *.swl 17 | *~ 18 | 19 | # cluster logs 20 | SMART_DISPATCH_LOGS/* 21 | 22 | # model params 23 | model/* 24 | params/* 25 | tmptrees/* 26 | 27 | # logs 28 | tblogs/* 29 | logs/* 30 | 31 | # Byte-compiled / optimized / DLL files 32 | __pycache__/ 33 | *.py[cod] 34 | *$py.class 35 | *.pyc 36 | 37 | *.log 38 | 39 | -------------------------------------------------------------------------------- /.ptignore: -------------------------------------------------------------------------------- 1 | *.pt 2 | Philly* 3 | exp_logic 4 | *.diff 5 | data/data_scan 6 | data/scan 7 | data/propositionallogic_exp1 8 | data/mnli 9 | data/SST 10 | data/penn 11 | data/listops 12 | data/treebank_proc.conllu 13 | data/test_proc.conllu 14 | -------------------------------------------------------------------------------- /EVALB/COLLINS.prm: -------------------------------------------------------------------------------- 1 | ##------------------------------------------## 2 | ## Debug mode ## 3 | ## 0: No debugging ## 4 | ## 1: print data for individual sentence ## 5 | ##------------------------------------------## 6 | DEBUG 0 7 | 8 | ##------------------------------------------## 9 | ## MAX error ## 10 | ## Number of error to stop the process. ## 11 | ## This is useful if there could be ## 12 | ## tokanization error. ## 13 | ## The process will stop when this number## 14 | ## of errors are accumulated. ## 15 | ##------------------------------------------## 16 | MAX_ERROR 10 17 | 18 | ##------------------------------------------## 19 | ## Cut-off length for statistics ## 20 | ## At the end of evaluation, the ## 21 | ## statistics for the senetnces of length## 22 | ## less than or equal to this number will## 23 | ## be shown, on top of the statistics ## 24 | ## for all the sentences ## 25 | ##------------------------------------------## 26 | CUTOFF_LEN 40 27 | 28 | ##------------------------------------------## 29 | ## unlabeled or labeled bracketing ## 30 | ## 0: unlabeled bracketing ## 31 | ## 1: labeled bracketing ## 32 | ##------------------------------------------## 33 | LABELED 0 34 | 35 | ##------------------------------------------## 36 | ## Delete labels ## 37 | ## list of labels to be ignored. ## 38 | ## If it is a pre-terminal label, delete ## 39 | ## the word along with the brackets. ## 40 | ## If it is a non-terminal label, just ## 41 | ## delete the brackets (don't delete ## 42 | ## deildrens). ## 43 | ##------------------------------------------## 44 | DELETE_LABEL ROOT 45 | 46 | ##------------------------------------------## 47 | ## Delete labels for length calculation ## 48 | ## list of labels to be ignored for ## 49 | ## length calculation purpose ## 50 | ##------------------------------------------## 51 | DELETE_LABEL_FOR_LENGTH -NONE- 52 | 53 | ##------------------------------------------## 54 | ## Equivalent labels, words ## 55 | ## the pairs are considered equivalent ## 56 | ## This is non-directional. ## 57 | ##------------------------------------------## 58 | EQ_LABEL ADVP PRT 59 | 60 | # EQ_WORD Example example 61 | -------------------------------------------------------------------------------- /EVALB/LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /EVALB/Makefile: -------------------------------------------------------------------------------- 1 | all: evalb 2 | 3 | evalb: evalb.c 4 | gcc -Wall -g -o evalb evalb.c 5 | -------------------------------------------------------------------------------- /EVALB/README: -------------------------------------------------------------------------------- 1 | ################################################################# 2 | # # 3 | # Bug fix and additional functionality for evalb # 4 | # # 5 | # This updated version of evalb fixes a bug in which sentences # 6 | # were incorrectly categorized as "length mismatch" when the # 7 | # the parse output had certain mislabeled parts-of-speech. # 8 | # # 9 | # The bug was the result of evalb treating one of the tags (in # 10 | # gold or test) as a label to be deleted (see sections [6],[7] # 11 | # for details), but not the corresponding tag in the other. # 12 | # This most often occurs with punctuation. See the subdir # 13 | # "bug" for an example gld and tst file demonstating the bug, # 14 | # as well as output of evalb with and without the bug fix. # 15 | # # 16 | # For the present version in case of length mismatch, the nodes # 17 | # causing the imbalance are reinserted to resolve the miscount. # 18 | # If the lengths of gold and test truly differ, the error is # 19 | # still reported. The parameter file "new.prm" (derived from # 20 | # COLLINS.prm) shows how to add new potential mislabelings for # 21 | # quotes (",``,',`). # 22 | # # 23 | # I have preserved DJB's revision for modern compilers except # 24 | # for the delcaration of "exit" which is provided by stdlib. # 25 | # # 26 | # Other changes: # 27 | # # 28 | # * output of F-Measure in addition to precision and recall # 29 | # (I did not update the documention in section [4] for this) # 30 | # # 31 | # * more comprehensive DEBUG output that includes bracketing # 32 | # information as evalb is processing each sentence # 33 | # (useful in working through this, and peraps other bugs). # 34 | # Use either the "-D" run-time switch or set DEBUG to 2 in # 35 | # the parameter file. # 36 | # # 37 | # * added DELETE_LABEL lines in new.prm for S1 nodes produced # 38 | # by the Charniak parser and "?", "!" punctuation produced by # 39 | # the Bikel parser. # 40 | # # 41 | # # 42 | # David Ellis (Brown) # 43 | # # 44 | # January.2006 # 45 | ################################################################# 46 | 47 | ################################################################# 48 | # # 49 | # Update of evalb for modern compilers # 50 | # # 51 | # This is an updated version of evalb, for use with modern C # 52 | # compilers. There are a few updates, each marked in the code: # 53 | # # 54 | # /* DJB: explanation of comment */ # 55 | # # 56 | # The updates are purely to help compilation with recent # 57 | # versions of GCC (and other C compilers). There are *NO* other # 58 | # changes to the algorithm itself. # 59 | # # 60 | # I have made these changes following recommendations from # 61 | # users of the Corpora Mailing List, especially Peet Morris and # 62 | # Ramon Ziai. # 63 | # # 64 | # David Brooks (Birmingham) # 65 | # # 66 | # September.2005 # 67 | ################################################################# 68 | 69 | ################################################################# 70 | # # 71 | # README file for evalb # 72 | # # 73 | # Satoshi Sekine (NYU) # 74 | # Mike Collins (UPenn) # 75 | # # 76 | # October.1997 # 77 | ################################################################# 78 | 79 | Contents of this README: 80 | 81 | [0] COPYRIGHT 82 | [1] INTRODUCTION 83 | [2] INSTALLATION AND RUN 84 | [3] OPTIONS 85 | [4] OUTPUT FORMAT FROM THE SCORER 86 | [5] HOW TO CREATE A GOLDFILE FROM THE TREEBANK 87 | [6] THE PARAMETER FILE 88 | [7] MORE DETAILS ABOUT THE SCORING ALGORITHM 89 | 90 | 91 | [0] COPYRIGHT 92 | 93 | The authors abandon the copyright of this program. Everyone is 94 | permitted to copy and distribute the program or a portion of the program 95 | with no charge and no restrictions unless it is harmful to someone. 96 | 97 | However, the authors are delightful for the user's kindness of proper 98 | usage and letting the authors know bugs or problems. 99 | 100 | This software is provided "AS IS", and the authors make no warranties, 101 | express or implied. 102 | 103 | To legally enforce the abandonment of copyright, this package is released 104 | under the Unlicense (see LICENSE). 105 | 106 | [1] INTRODUCTION 107 | 108 | Evaluation of bracketing looks simple, but in fact, there are minor 109 | differences from system to system. This is a program to parametarize 110 | such minor differences and to give an informative result. 111 | 112 | "evalb" evaluates bracketing accuracy in a test-file against a gold-file. 113 | It returns recall, precision, tagging accuracy. It uses an identical 114 | algorithm to that used in (Collins ACL97). 115 | 116 | 117 | [2] Installation and Run 118 | 119 | To compile the scorer, type 120 | 121 | > make 122 | 123 | 124 | To run the scorer: 125 | 126 | > evalb -p Parameter_file Gold_file Test_file 127 | 128 | 129 | For example to use the sample files: 130 | 131 | > evalb -p sample.prm sample.gld sample.tst 132 | 133 | 134 | 135 | [3] OPTIONS 136 | 137 | You can specify system parameters in the command line options. 138 | Other options concerning to evaluation metrix should be specified 139 | in parameter file, described later. 140 | 141 | -p param_file parameter file 142 | -d debug mode 143 | -e n number of error to kill (default=10) 144 | -h help 145 | 146 | 147 | 148 | [4] OUTPUT FORMAT FROM THE SCORER 149 | 150 | The scorer gives individual scores for each sentence, for 151 | example: 152 | 153 | Sent. Matched Bracket Cross Correct Tag 154 | ID Len. Stat. Recal Prec. Bracket gold test Bracket Words Tags Accracy 155 | ============================================================================ 156 | 1 8 0 100.00 100.00 5 5 5 0 6 5 83.33 157 | 158 | At the end of the output the === Summary === section gives statistics 159 | for all sentences, and for sentences <=40 words in length. The summary 160 | contains the following information: 161 | 162 | i) Number of sentences -- total number of sentences. 163 | 164 | ii) Number of Error/Skip sentences -- should both be 0 if there is no 165 | problem with the parsed/gold files. 166 | 167 | iii) Number of valid sentences = Number of sentences - Number of Error/Skip 168 | sentences 169 | 170 | iv) Bracketing recall = (number of correct constituents) 171 | ---------------------------------------- 172 | (number of constituents in the goldfile) 173 | 174 | v) Bracketing precision = (number of correct constituents) 175 | ---------------------------------------- 176 | (number of constituents in the parsed file) 177 | 178 | vi) Complete match = percentaage of sentences where recall and precision are 179 | both 100%. 180 | 181 | vii) Average crossing = (number of constituents crossing a goldfile constituen 182 | ---------------------------------------------------- 183 | (number of sentences) 184 | 185 | viii) No crossing = percentage of sentences which have 0 crossing brackets. 186 | 187 | ix) 2 or less crossing = percentage of sentences which have <=2 crossing brackets. 188 | 189 | x) Tagging accuracy = percentage of correct POS tags (but see [5].3 for exact 190 | details of what is counted). 191 | 192 | 193 | 194 | [5] HOW TO CREATE A GOLDFILE FROM THE PENN TREEBANK 195 | 196 | 197 | The gold and parsed files are in a format similar to this: 198 | 199 | (TOP (S (INTJ (RB No)) (, ,) (NP (PRP it)) (VP (VBD was) (RB n't) (NP (NNP Black) (NNP Monday))) (. .))) 200 | 201 | To create a gold file from the treebank: 202 | 203 | tgrep -wn '/.*/' | tgrep_proc.prl 204 | 205 | will produce a goldfile in the required format. ("tgrep -wn '/.*/'" prints 206 | parse trees, "tgrep_process.prl" just skips blank lines). 207 | 208 | For example, to produce a goldfile for section 23 of the treebank: 209 | 210 | tgrep -wn '/.*/' | tail +90895 | tgrep_process.prl | sed 2416q > sec23.gold 211 | 212 | 213 | 214 | [6] THE PARAMETER (.prm) FILE 215 | 216 | 217 | The .prm file sets options regarding the scoring method. COLLINS.prm gives 218 | the same scoring behaviour as the scorer used in (Collins 97). The options 219 | chosen were: 220 | 221 | 1) LABELED 1 222 | 223 | to give labelled precision/recall figures, i.e. a constituent must have the 224 | same span *and* label as a constituent in the goldfile. 225 | 226 | 2) DELETE_LABEL TOP 227 | 228 | Don't count the "TOP" label (which is always given in the output of tgrep) 229 | when scoring. 230 | 231 | 3) DELETE_LABEL -NONE- 232 | 233 | Remove traces (and all constituents which dominate nothing but traces) when 234 | scoring. For example 235 | 236 | .... (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1)))) (. .))) 237 | 238 | would be processed to give 239 | 240 | .... (VP (VBD reported)) (. .))) 241 | 242 | 243 | 4) 244 | DELETE_LABEL , -- for the purposes of scoring remove punctuation 245 | DELETE_LABEL : 246 | DELETE_LABEL `` 247 | DELETE_LABEL '' 248 | DELETE_LABEL . 249 | 250 | 5) DELETE_LABEL_FOR_LENGTH -NONE- -- don't include traces when calculating 251 | the length of a sentence (important 252 | when classifying a sentence as <=40 253 | words or >40 words) 254 | 255 | 6) EQ_LABEL ADVP PRT 256 | 257 | Count ADVP and PRT as being the same label when scoring. 258 | 259 | 260 | 261 | 262 | [7] MORE DETAILS ABOUT THE SCORING ALGORITHM 263 | 264 | 265 | 1) The scorer initially processes the files to remove all nodes specified 266 | by DELETE_LABEL in the .prm file. It also recursively removes nodes which 267 | dominate nothing due to all their children being removed. For example, if 268 | -NONE- is specified as a label to be deleted, 269 | 270 | .... (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1)))) (. .))) 271 | 272 | would be processed to give 273 | 274 | .... (VP (VBD reported)) (. .))) 275 | 276 | 2) The scorer also removes all functional tags attached to non-terminals 277 | (functional tags are prefixed with "-" or "=" in the treebank). For example 278 | "NP-SBJ" is processed to give "NP", "NP=2" is changed to "NP". 279 | 280 | 281 | 3) Tagging accuracy counts tags for all words *except* any tags which are 282 | deleted by a DELETE_LABEL specification in the .prm file. (For example, for 283 | COLLINS.prm, punctuation tagged as "," ":" etc. would not be included). 284 | 285 | 4) When calculating the length of a sentence, all words with POS tags not 286 | included in the "DELETE_LABEL_FOR_LENGTH" list in the .prm file are 287 | counted. (For COLLINS.prm, only "-NONE-" is specified in this list, so 288 | traces are removed before calculating the length of the sentence). 289 | 290 | 5) There are some subtleties in scoring when either the goldfile or parsed 291 | file contains multiple constituents for the same span which have the same 292 | non-terminal label. e.g. (NP (NP the man)) If the goldfile contains n 293 | constituents for the same span, and the parsed file contains m constituents 294 | with that nonterminal, the scorer works as follows: 295 | 296 | i) If m>n, then the precision is n/m, recall is 100% 297 | 298 | ii) If n>m, then the precision is 100%, recall is m/n. 299 | 300 | iii) If n==m, recall and precision are both 100%. 301 | -------------------------------------------------------------------------------- /EVALB/evalb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yikangshen/Ordered-Memory/2a0d3a22fb70216993a7faf25de3515f42e10431/EVALB/evalb -------------------------------------------------------------------------------- /EVALB/evalb.c: -------------------------------------------------------------------------------- 1 | /*****************************************************************/ 2 | /* evalb [-p param_file] [-dh] [-e n] gold-file test-file */ 3 | /* */ 4 | /* Evaluate bracketing in test-file against gold-file. */ 5 | /* Return recall, precision, tagging accuracy. */ 6 | /* */ 7 | /*