├── .gitattributes ├── .gitignore ├── EVALB ├── COLLINS.prm ├── LICENSE ├── Makefile ├── README ├── evalb ├── evalb.c ├── new.prm ├── sample │ ├── sample.gld │ ├── sample.prm │ ├── sample.rsl │ └── sample.tst └── tgrep_proc.prl ├── LICENSE ├── ON_LSTM.py ├── README.md ├── data.py ├── data └── penn │ ├── test.txt │ ├── train.txt │ └── valid.txt ├── data_ptb.py ├── embed_regularize.py ├── locked_dropout.py ├── main.py ├── model.py ├── parse_comparison.py ├── requirements.txt ├── splitcross.py ├── test_phrase_grammar.py ├── utils.py └── weight_drop.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | <<<<<<< HEAD 2 | # Byte-compiled / optimized / DLL files 3 | __pycache__/ 4 | *.py[cod] 5 | *$py.class 6 | 7 | # C extensions 8 | *.so 9 | 10 | # Distribution / packaging 11 | .Python 12 | env/ 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .coverage 43 | .coverage.* 44 | .cache 45 | nosetests.xml 46 | coverage.xml 47 | *.cover 48 | .hypothesis/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | 58 | # Flask stuff: 59 | instance/ 60 | .webassets-cache 61 | 62 | # Scrapy stuff: 63 | .scrapy 64 | 65 | # Sphinx documentation 66 | docs/_build/ 67 | 68 | # PyBuilder 69 | target/ 70 | 71 | # Jupyter Notebook 72 | .ipynb_checkpoints 73 | 74 | # pyenv 75 | .python-version 76 | 77 | # celery beat schedule file 78 | celerybeat-schedule 79 | 80 | # SageMath parsed files 81 | *.sage.py 82 | 83 | # dotenv 84 | .env 85 | 86 | # virtualenv 87 | .venv 88 | venv/ 89 | ENV/ 90 | 91 | # Spyder project settings 92 | .spyderproject 93 | .spyproject 94 | 95 | # Rope project settings 96 | .ropeproject 97 | 98 | # mkdocs documentation 99 | /site 100 | 101 | # mypy 102 | .mypy_cache/ 103 | 104 | #pycharm 105 | .idea/ 106 | 107 | #pytorch 108 | *.pt 109 | ======= 110 | # Byte-compiled / optimized / DLL files 111 | __pycache__/ 112 | *.py[cod] 113 | *$py.class 114 | 115 | # C extensions 116 | *.so 117 | 118 | # Distribution / packaging 119 | .Python 120 | env/ 121 | build/ 122 | develop-eggs/ 123 | dist/ 124 | downloads/ 125 | eggs/ 126 | .eggs/ 127 | lib/ 128 | lib64/ 129 | parts/ 130 | sdist/ 131 | var/ 132 | wheels/ 133 | *.egg-info/ 134 | .installed.cfg 135 | *.egg 136 | 137 | # PyInstaller 138 | # Usually these files are written by a python script from a template 139 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 140 | *.manifest 141 | *.spec 142 | 143 | # Installer logs 144 | pip-log.txt 145 | pip-delete-this-directory.txt 146 | 147 | # Unit test / coverage reports 148 | htmlcov/ 149 | .tox/ 150 | .coverage 151 | .coverage.* 152 | .cache 153 | nosetests.xml 154 | coverage.xml 155 | *.cover 156 | .hypothesis/ 157 | 158 | # Translations 159 | *.mo 160 | *.pot 161 | 162 | # Django stuff: 163 | *.log 164 | local_settings.py 165 | 166 | # Flask stuff: 167 | instance/ 168 | .webassets-cache 169 | 170 | # Scrapy stuff: 171 | .scrapy 172 | 173 | # Sphinx documentation 174 | docs/_build/ 175 | 176 | # PyBuilder 177 | target/ 178 | 179 | # Jupyter Notebook 180 | .ipynb_checkpoints 181 | 182 | # pyenv 183 | .python-version 184 | 185 | # celery beat schedule file 186 | celerybeat-schedule 187 | 188 | # SageMath parsed files 189 | *.sage.py 190 | 191 | # dotenv 192 | .env 193 | 194 | # virtualenv 195 | .venv 196 | venv/ 197 | ENV/ 198 | 199 | # Spyder project settings 200 | .spyderproject 201 | .spyproject 202 | 203 | # Rope project settings 204 | .ropeproject 205 | 206 | # mkdocs documentation 207 | /site 208 | 209 | # mypy 210 | .mypy_cache/ 211 | 212 | #pycharm 213 | .idea/ 214 | 215 | #pytorch 216 | *.pt 217 | >>>>>>> e6ee33014912d2cdc248ceaf7855ad53fd2edad5 218 | -------------------------------------------------------------------------------- /EVALB/COLLINS.prm: -------------------------------------------------------------------------------- 1 | ##------------------------------------------## 2 | ## Debug mode ## 3 | ## 0: No debugging ## 4 | ## 1: print data for individual sentence ## 5 | ##------------------------------------------## 6 | DEBUG 0 7 | 8 | ##------------------------------------------## 9 | ## MAX error ## 10 | ## Number of error to stop the process. ## 11 | ## This is useful if there could be ## 12 | ## tokanization error. ## 13 | ## The process will stop when this number## 14 | ## of errors are accumulated. ## 15 | ##------------------------------------------## 16 | MAX_ERROR 10 17 | 18 | ##------------------------------------------## 19 | ## Cut-off length for statistics ## 20 | ## At the end of evaluation, the ## 21 | ## statistics for the senetnces of length## 22 | ## less than or equal to this number will## 23 | ## be shown, on top of the statistics ## 24 | ## for all the sentences ## 25 | ##------------------------------------------## 26 | CUTOFF_LEN 40 27 | 28 | ##------------------------------------------## 29 | ## unlabeled or labeled bracketing ## 30 | ## 0: unlabeled bracketing ## 31 | ## 1: labeled bracketing ## 32 | ##------------------------------------------## 33 | LABELED 1 34 | 35 | ##------------------------------------------## 36 | ## Delete labels ## 37 | ## list of labels to be ignored. ## 38 | ## If it is a pre-terminal label, delete ## 39 | ## the word along with the brackets. ## 40 | ## If it is a non-terminal label, just ## 41 | ## delete the brackets (don't delete ## 42 | ## deildrens). ## 43 | ##------------------------------------------## 44 | DELETE_LABEL TOP 45 | DELETE_LABEL -NONE- 46 | DELETE_LABEL , 47 | DELETE_LABEL : 48 | DELETE_LABEL `` 49 | DELETE_LABEL '' 50 | DELETE_LABEL . 51 | 52 | ##------------------------------------------## 53 | ## Delete labels for length calculation ## 54 | ## list of labels to be ignored for ## 55 | ## length calculation purpose ## 56 | ##------------------------------------------## 57 | DELETE_LABEL_FOR_LENGTH -NONE- 58 | 59 | ##------------------------------------------## 60 | ## Equivalent labels, words ## 61 | ## the pairs are considered equivalent ## 62 | ## This is non-directional. ## 63 | ##------------------------------------------## 64 | EQ_LABEL ADVP PRT 65 | 66 | # EQ_WORD Example example 67 | -------------------------------------------------------------------------------- /EVALB/LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /EVALB/Makefile: -------------------------------------------------------------------------------- 1 | all: evalb 2 | 3 | evalb: evalb.c 4 | gcc -Wall -g -o evalb evalb.c 5 | -------------------------------------------------------------------------------- /EVALB/README: -------------------------------------------------------------------------------- 1 | ################################################################# 2 | # # 3 | # Bug fix and additional functionality for evalb # 4 | # # 5 | # This updated version of evalb fixes a bug in which sentences # 6 | # were incorrectly categorized as "length mismatch" when the # 7 | # the parse output had certain mislabeled parts-of-speech. # 8 | # # 9 | # The bug was the result of evalb treating one of the tags (in # 10 | # gold or test) as a label to be deleted (see sections [6],[7] # 11 | # for details), but not the corresponding tag in the other. # 12 | # This most often occurs with punctuation. See the subdir # 13 | # "bug" for an example gld and tst file demonstating the bug, # 14 | # as well as output of evalb with and without the bug fix. # 15 | # # 16 | # For the present version in case of length mismatch, the nodes # 17 | # causing the imbalance are reinserted to resolve the miscount. # 18 | # If the lengths of gold and test truly differ, the error is # 19 | # still reported. The parameter file "new.prm" (derived from # 20 | # COLLINS.prm) shows how to add new potential mislabelings for # 21 | # quotes (",``,',`). # 22 | # # 23 | # I have preserved DJB's revision for modern compilers except # 24 | # for the delcaration of "exit" which is provided by stdlib. # 25 | # # 26 | # Other changes: # 27 | # # 28 | # * output of F-Measure in addition to precision and recall # 29 | # (I did not update the documention in section [4] for this) # 30 | # # 31 | # * more comprehensive DEBUG output that includes bracketing # 32 | # information as evalb is processing each sentence # 33 | # (useful in working through this, and peraps other bugs). # 34 | # Use either the "-D" run-time switch or set DEBUG to 2 in # 35 | # the parameter file. # 36 | # # 37 | # * added DELETE_LABEL lines in new.prm for S1 nodes produced # 38 | # by the Charniak parser and "?", "!" punctuation produced by # 39 | # the Bikel parser. # 40 | # # 41 | # # 42 | # David Ellis (Brown) # 43 | # # 44 | # January.2006 # 45 | ################################################################# 46 | 47 | ################################################################# 48 | # # 49 | # Update of evalb for modern compilers # 50 | # # 51 | # This is an updated version of evalb, for use with modern C # 52 | # compilers. There are a few updates, each marked in the code: # 53 | # # 54 | # /* DJB: explanation of comment */ # 55 | # # 56 | # The updates are purely to help compilation with recent # 57 | # versions of GCC (and other C compilers). There are *NO* other # 58 | # changes to the algorithm itself. # 59 | # # 60 | # I have made these changes following recommendations from # 61 | # users of the Corpora Mailing List, especially Peet Morris and # 62 | # Ramon Ziai. # 63 | # # 64 | # David Brooks (Birmingham) # 65 | # # 66 | # September.2005 # 67 | ################################################################# 68 | 69 | ################################################################# 70 | # # 71 | # README file for evalb # 72 | # # 73 | # Satoshi Sekine (NYU) # 74 | # Mike Collins (UPenn) # 75 | # # 76 | # October.1997 # 77 | ################################################################# 78 | 79 | Contents of this README: 80 | 81 | [0] COPYRIGHT 82 | [1] INTRODUCTION 83 | [2] INSTALLATION AND RUN 84 | [3] OPTIONS 85 | [4] OUTPUT FORMAT FROM THE SCORER 86 | [5] HOW TO CREATE A GOLDFILE FROM THE TREEBANK 87 | [6] THE PARAMETER FILE 88 | [7] MORE DETAILS ABOUT THE SCORING ALGORITHM 89 | 90 | 91 | [0] COPYRIGHT 92 | 93 | The authors abandon the copyright of this program. Everyone is 94 | permitted to copy and distribute the program or a portion of the program 95 | with no charge and no restrictions unless it is harmful to someone. 96 | 97 | However, the authors are delightful for the user's kindness of proper 98 | usage and letting the authors know bugs or problems. 99 | 100 | This software is provided "AS IS", and the authors make no warranties, 101 | express or implied. 102 | 103 | To legally enforce the abandonment of copyright, this package is released 104 | under the Unlicense (see LICENSE). 105 | 106 | [1] INTRODUCTION 107 | 108 | Evaluation of bracketing looks simple, but in fact, there are minor 109 | differences from system to system. This is a program to parametarize 110 | such minor differences and to give an informative result. 111 | 112 | "evalb" evaluates bracketing accuracy in a test-file against a gold-file. 113 | It returns recall, precision, tagging accuracy. It uses an identical 114 | algorithm to that used in (Collins ACL97). 115 | 116 | 117 | [2] Installation and Run 118 | 119 | To compile the scorer, type 120 | 121 | > make 122 | 123 | 124 | To run the scorer: 125 | 126 | > evalb -p Parameter_file Gold_file Test_file 127 | 128 | 129 | For example to use the sample files: 130 | 131 | > evalb -p sample.prm sample.gld sample.tst 132 | 133 | 134 | 135 | [3] OPTIONS 136 | 137 | You can specify system parameters in the command line options. 138 | Other options concerning to evaluation metrix should be specified 139 | in parameter file, described later. 140 | 141 | -p param_file parameter file 142 | -d debug mode 143 | -e n number of error to kill (default=10) 144 | -h help 145 | 146 | 147 | 148 | [4] OUTPUT FORMAT FROM THE SCORER 149 | 150 | The scorer gives individual scores for each sentence, for 151 | example: 152 | 153 | Sent. Matched Bracket Cross Correct Tag 154 | ID Len. Stat. Recal Prec. Bracket gold test Bracket Words Tags Accracy 155 | ============================================================================ 156 | 1 8 0 100.00 100.00 5 5 5 0 6 5 83.33 157 | 158 | At the end of the output the === Summary === section gives statistics 159 | for all sentences, and for sentences <=40 words in length. The summary 160 | contains the following information: 161 | 162 | i) Number of sentences -- total number of sentences. 163 | 164 | ii) Number of Error/Skip sentences -- should both be 0 if there is no 165 | problem with the parsed/gold files. 166 | 167 | iii) Number of valid sentences = Number of sentences - Number of Error/Skip 168 | sentences 169 | 170 | iv) Bracketing recall = (number of correct constituents) 171 | ---------------------------------------- 172 | (number of constituents in the goldfile) 173 | 174 | v) Bracketing precision = (number of correct constituents) 175 | ---------------------------------------- 176 | (number of constituents in the parsed file) 177 | 178 | vi) Complete match = percentaage of sentences where recall and precision are 179 | both 100%. 180 | 181 | vii) Average crossing = (number of constituents crossing a goldfile constituen 182 | ---------------------------------------------------- 183 | (number of sentences) 184 | 185 | viii) No crossing = percentage of sentences which have 0 crossing brackets. 186 | 187 | ix) 2 or less crossing = percentage of sentences which have <=2 crossing brackets. 188 | 189 | x) Tagging accuracy = percentage of correct POS tags (but see [5].3 for exact 190 | details of what is counted). 191 | 192 | 193 | 194 | [5] HOW TO CREATE A GOLDFILE FROM THE PENN TREEBANK 195 | 196 | 197 | The gold and parsed files are in a format similar to this: 198 | 199 | (TOP (S (INTJ (RB No)) (, ,) (NP (PRP it)) (VP (VBD was) (RB n't) (NP (NNP Black) (NNP Monday))) (. .))) 200 | 201 | To create a gold file from the treebank: 202 | 203 | tgrep -wn '/.*/' | tgrep_proc.prl 204 | 205 | will produce a goldfile in the required format. ("tgrep -wn '/.*/'" prints 206 | parse trees, "tgrep_process.prl" just skips blank lines). 207 | 208 | For example, to produce a goldfile for section 23 of the treebank: 209 | 210 | tgrep -wn '/.*/' | tail +90895 | tgrep_process.prl | sed 2416q > sec23.gold 211 | 212 | 213 | 214 | [6] THE PARAMETER (.prm) FILE 215 | 216 | 217 | The .prm file sets options regarding the scoring method. COLLINS.prm gives 218 | the same scoring behaviour as the scorer used in (Collins 97). The options 219 | chosen were: 220 | 221 | 1) LABELED 1 222 | 223 | to give labelled precision/recall figures, i.e. a constituent must have the 224 | same span *and* label as a constituent in the goldfile. 225 | 226 | 2) DELETE_LABEL TOP 227 | 228 | Don't count the "TOP" label (which is always given in the output of tgrep) 229 | when scoring. 230 | 231 | 3) DELETE_LABEL -NONE- 232 | 233 | Remove traces (and all constituents which dominate nothing but traces) when 234 | scoring. For example 235 | 236 | .... (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1)))) (. .))) 237 | 238 | would be processed to give 239 | 240 | .... (VP (VBD reported)) (. .))) 241 | 242 | 243 | 4) 244 | DELETE_LABEL , -- for the purposes of scoring remove punctuation 245 | DELETE_LABEL : 246 | DELETE_LABEL `` 247 | DELETE_LABEL '' 248 | DELETE_LABEL . 249 | 250 | 5) DELETE_LABEL_FOR_LENGTH -NONE- -- don't include traces when calculating 251 | the length of a sentence (important 252 | when classifying a sentence as <=40 253 | words or >40 words) 254 | 255 | 6) EQ_LABEL ADVP PRT 256 | 257 | Count ADVP and PRT as being the same label when scoring. 258 | 259 | 260 | 261 | 262 | [7] MORE DETAILS ABOUT THE SCORING ALGORITHM 263 | 264 | 265 | 1) The scorer initially processes the files to remove all nodes specified 266 | by DELETE_LABEL in the .prm file. It also recursively removes nodes which 267 | dominate nothing due to all their children being removed. For example, if 268 | -NONE- is specified as a label to be deleted, 269 | 270 | .... (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1)))) (. .))) 271 | 272 | would be processed to give 273 | 274 | .... (VP (VBD reported)) (. .))) 275 | 276 | 2) The scorer also removes all functional tags attached to non-terminals 277 | (functional tags are prefixed with "-" or "=" in the treebank). For example 278 | "NP-SBJ" is processed to give "NP", "NP=2" is changed to "NP". 279 | 280 | 281 | 3) Tagging accuracy counts tags for all words *except* any tags which are 282 | deleted by a DELETE_LABEL specification in the .prm file. (For example, for 283 | COLLINS.prm, punctuation tagged as "," ":" etc. would not be included). 284 | 285 | 4) When calculating the length of a sentence, all words with POS tags not 286 | included in the "DELETE_LABEL_FOR_LENGTH" list in the .prm file are 287 | counted. (For COLLINS.prm, only "-NONE-" is specified in this list, so 288 | traces are removed before calculating the length of the sentence). 289 | 290 | 5) There are some subtleties in scoring when either the goldfile or parsed 291 | file contains multiple constituents for the same span which have the same 292 | non-terminal label. e.g. (NP (NP the man)) If the goldfile contains n 293 | constituents for the same span, and the parsed file contains m constituents 294 | with that nonterminal, the scorer works as follows: 295 | 296 | i) If m>n, then the precision is n/m, recall is 100% 297 | 298 | ii) If n>m, then the precision is 100%, recall is m/n. 299 | 300 | iii) If n==m, recall and precision are both 100%. 301 | -------------------------------------------------------------------------------- /EVALB/evalb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yikangshen/Ordered-Neurons/46d63cde024802eaf1eb7cc896431329014dd869/EVALB/evalb -------------------------------------------------------------------------------- /EVALB/evalb.c: -------------------------------------------------------------------------------- 1 | /*****************************************************************/ 2 | /* evalb [-p param_file] [-dh] [-e n] gold-file test-file */ 3 | /* */ 4 | /* Evaluate bracketing in test-file against gold-file. */ 5 | /* Return recall, precision, tagging accuracy. */ 6 | /* */ 7 | /*