├── LICENSE ├── README.md ├── _config.yml ├── references.bib ├── references.html ├── table1.html ├── table2.html └── table3.html /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Analysis Methods in Neural NLP 2 | This site contains the accompanying supplementary materials for the paper "Analysis Methods in Neural Language Processing: A Survey", TACL 2019, available [here](https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00254). 3 | 4 | ## Tables 5 | * [Table SM1](table1.html): A categorization of work trying to find linguistic information in neural networks according to the neural network component investigated, the linguistic property sought, and the analysis method. See Section 2 in the paper. 6 | * [Table SM2](table2.html): A categorization of challenge sets for evaluating neural networks according to the NLP task, the linguistic phenomena, the represented languages, the dataset size, and the construction method. See Section 4 in the paper. 7 | * [Table SM3](table3.html): A categorization of methods for adversarial examples in NLP according to adversary's knowledge (white-box vs. black-box), attack specificity (targeted vs. non-targeted), the modified linguistic unit (words, characters, etc.), and the attacked task. See Section 5 in the paper. 8 | 9 | ## References 10 | The list of references is available [here](references.html). 11 | 12 | ## Contributions 13 | Miss your favorite neural analysis method? Great! Contributions to this site are welcome. Please open a pull request. 14 | 15 | ## Citation 16 | If you find this resource useful, please cite our paper: 17 | 18 | ``` 19 | @Article{belinkov:2019:tacl, 20 | author = {Belinkov, Yonatan and Glass, James}, 21 | title = {Analysis Methods in Neural Language Processing: A Survey}, 22 | journal = {Transactions of the Association for Computational Linguistics (TACL)}, 23 | year = {2019}, 24 | volume = {7}, 25 | pages = {49--72}, 26 | doi = {10.1162/tacl\_a\_00254} 27 | } 28 | ``` 29 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-slate 2 | title: "Analysis Methods in Neural NLP" 3 | description: "Accompanying appendix for the paper “Analysis Methods in Neural Language Processing: A Survey”, TACL 2019L." 4 | show_downloads: false 5 | -------------------------------------------------------------------------------- /references.bib: -------------------------------------------------------------------------------- 1 | @Article{ karpathy2015visualizing, 2 | title = {{Visualizing and Understanding Recurrent Networks}}, 3 | author = {Karpathy, Andrej and Johnson, Justin and Li, Fei-Fei}, 4 | journal = {arXiv preprint arXiv:1506.02078v2}, 5 | year = {2015} 6 | } 7 | 8 | @InProceedings{ qian-qiu-huang:2016:EMNLP2016, 9 | author = {Qian, Peng and Qiu, Xipeng and Huang, Xuanjing}, 10 | title = {{Analyzing Linguistic Knowledge in Sequential Model of 11 | Sentence}}, 12 | booktitle = {Proceedings of the 2016 Conference on Empirical Methods in 13 | Natural Language Processing}, 14 | month = {November}, 15 | year = {2016}, 16 | address = {Austin, Texas}, 17 | publisher = {Association for Computational Linguistics}, 18 | pages = {826--835}, 19 | url = {https://aclweb.org/anthology/D16-1079} 20 | } 21 | 22 | @Article{ vylomova2016word, 23 | title = {{Word Representation Models for Morphologically Rich 24 | Languages in Neural Machine Translation}}, 25 | author = {Vylomova, Ekaterina and Cohn, Trevor and He, Xuanli and 26 | Haffari, Gholamreza}, 27 | journal = {arXiv preprint arXiv:1606.04217v1}, 28 | year = {2016} 29 | } 30 | 31 | @InProceedings{ shi-padhi-knight:2016:EMNLP2016, 32 | author = {Shi, Xing and Padhi, Inkit and Knight, Kevin}, 33 | title = {{Does String-Based Neural MT Learn Source Syntax?}}, 34 | booktitle = {Proceedings of the 2016 Conference on Empirical Methods in 35 | Natural Language Processing}, 36 | month = {November}, 37 | year = {2016}, 38 | address = {Austin, Texas}, 39 | publisher = {Association for Computational Linguistics}, 40 | pages = {1526--1534}, 41 | url = {https://aclweb.org/anthology/D16-1159} 42 | } 43 | 44 | @Article{ kadar2016representation, 45 | title = {{Representation of Linguistic Form and Function in 46 | Recurrent Neural Networks}}, 47 | author = {K{\'a}d{\'a}r, Akos and Chrupa{\l}a, Grzegorz and 48 | Alishahi, Afra}, 49 | journal = {Computational Linguistics}, 50 | volume = {43}, 51 | number = {4}, 52 | pages = {761--780}, 53 | year = {2017}, 54 | publisher = {MIT Press}, 55 | doi = "10.1162/COLI_a_00300", 56 | url = "http://aclweb.org/anthology/J17-4003" 57 | } 58 | 59 | @InProceedings{ kohn:2015:EMNLP, 60 | author = {K\"{o}hn, Arne}, 61 | title = {{What's in an Embedding? Analyzing Word Embeddings through 62 | Multilingual Evaluation}}, 63 | booktitle = {Proceedings of the 2015 Conference on Empirical Methods in 64 | Natural Language Processing}, 65 | month = {September}, 66 | year = {2015}, 67 | address = {Lisbon, Portugal}, 68 | publisher = {Association for Computational Linguistics}, 69 | pages = {2067--2073}, 70 | url = {http://aclweb.org/anthology/D15-1246} 71 | } 72 | 73 | @InProceedings{ qian-qiu-huang:2016:P16-11, 74 | author = {Qian, Peng and Qiu, Xipeng and Huang, Xuanjing}, 75 | title = {{Investigating Language Universal and Specific Properties 76 | in Word Embeddings}}, 77 | booktitle = {Proceedings of the 54th Annual Meeting of the Association 78 | for Computational Linguistics (Volume 1: Long Papers)}, 79 | month = {August}, 80 | year = {2016}, 81 | address = {Berlin, Germany}, 82 | publisher = {Association for Computational Linguistics}, 83 | pages = {1478--1488}, 84 | url = {http://www.aclweb.org/anthology/P16-1140} 85 | } 86 | 87 | @InProceedings{ gelderloos-chrupala:2016:COLING, 88 | author = {Gelderloos, Lieke and Chrupa{\l}a, Grzegorz}, 89 | title = {{From phonemes to images: Levels of representation in a 90 | recurrent neural model of visually-grounded language 91 | learning}}, 92 | booktitle = {Proceedings of COLING 2016, the 26th International 93 | Conference on Computational Linguistics: Technical Papers}, 94 | month = {December}, 95 | year = {2016}, 96 | address = {Osaka, Japan}, 97 | publisher = {The COLING 2016 Organizing Committee}, 98 | pages = {1309--1319}, 99 | abstract = {We present a model of visually-grounded language learning 100 | based on stacked gated recurrent neural networks which 101 | learns to predict visual features given an image 102 | description in the form of a sequence of phonemes. The 103 | learning task resembles that faced by human language 104 | learners who need to discover both structure and meaning 105 | from noisy and ambiguous data across modalities. We show 106 | that our model indeed learns to predict features of the 107 | visual context given phonetically transcribed image 108 | descriptions, and show that it represents linguistic 109 | information in a hierarchy of levels: lower layers in the 110 | stack are comparatively more sensitive to form, whereas 111 | higher layers are more sensitive to meaning.}, 112 | url = {http://aclweb.org/anthology/C16-1124} 113 | } 114 | 115 | @InProceedings{ adi2016fine, 116 | title = {{Fine-grained Analysis of Sentence Embeddings Using 117 | Auxiliary Prediction Tasks}}, 118 | author = {Adi, Yossi and Kermany, Einat and Belinkov, Yonatan and 119 | Lavi, Ofer and Goldberg, Yoav}, 120 | booktitle = {International Conference on Learning Representations 121 | (ICLR)}, 122 | year = {2017} 123 | } 124 | 125 | @InProceedings{ D15-1002, 126 | author = "Gupta, Abhijeet and Boleda, Gemma and Baroni, Marco and 127 | Pad{\'o}, Sebastian", 128 | title = "Distributional vectors encode referential attributes", 129 | booktitle = "Proceedings of the 2015 Conference on Empirical Methods in 130 | Natural Language Processing ", 131 | year = "2015", 132 | publisher = "Association for Computational Linguistics", 133 | pages = "12--21", 134 | location = "Lisbon, Portugal", 135 | doi = "10.18653/v1/D15-1002", 136 | url = "http://www.aclweb.org/anthology/D15-1002" 137 | } 138 | 139 | @Article{ 10.1371/journal.pone.0181142, 140 | author = {Arras, Leila and Horn, Franziska and Montavon, 141 | Gr{\'e}goire and M{\"u}ller, Klaus-Robert and Samek, 142 | Wojciech}, 143 | journal = {PLOS ONE}, 144 | publisher = {Public Library of Science}, 145 | title = {{"What is relevant in a text document?": An interpretable 146 | machine learning approach}}, 147 | year = {2017}, 148 | month = {08}, 149 | volume = {12}, 150 | url = {https://doi.org/10.1371/journal.pone.0181142}, 151 | pages = {1-23}, 152 | number = {8}, 153 | doi = {10.1371/journal.pone.0181142} 154 | } 155 | 156 | @Article{ li2016understanding, 157 | title = {{Understanding Neural Networks through Representation 158 | Erasure}}, 159 | author = {Li, Jiwei and Monroe, Will and Jurafsky, Dan}, 160 | journal = {arXiv preprint arXiv:1612.08220v3}, 161 | year = {2016} 162 | } 163 | 164 | @Article{ elman1990finding, 165 | title = {{Finding Structure in Time}}, 166 | author = {Elman, Jeffrey L.}, 167 | journal = {Cognitive science}, 168 | volume = {14}, 169 | number = {2}, 170 | pages = {179--211}, 171 | year = {1990}, 172 | publisher = {Wiley Online Library} 173 | } 174 | 175 | @TechReport{ elman1989representation, 176 | title = {{Representation and Structure in Connectionist Models}}, 177 | author = {Elman, Jeffrey L.}, 178 | year = {1989}, 179 | institution = {University of California, San Diego, Center for Research 180 | in Language}, 181 | location = {La Jolla, CA} 182 | } 183 | 184 | @InProceedings{ wang2017gate, 185 | title = {{Gate Activation Signal Analysis for Gated Recurrent 186 | Neural Networks and Its Correlation with Phoneme 187 | Boundaries}}, 188 | author = {Wang, Yu-Hsuan and Chung, Cheng-Tao and Lee, Hung-yi}, 189 | booktitle = {Interspeech 2017}, 190 | year = {2017} 191 | } 192 | 193 | @InProceedings{ chaabouni2017learning, 194 | title = {Learning weakly supervised multimodal phoneme embeddings}, 195 | author = {Chaabouni, Rahma and Dunbar, Ewan and Zeghidour, Neil and 196 | Dupoux, Emmanuel}, 197 | booktitle = {Interspeech 2017}, 198 | year = {2017} 199 | } 200 | 201 | @InProceedings{ wu2016investigating, 202 | title = {Investigating gated recurrent networks for speech 203 | synthesis}, 204 | author = {Wu, Zhizheng and King, Simon}, 205 | booktitle = {2016 IEEE International Conference on Acoustics, Speech 206 | and Signal Processing (ICASSP)}, 207 | pages = {5140--5144}, 208 | year = {2016}, 209 | organization = {IEEE} 210 | } 211 | 212 | @InProceedings{ Ganesh:2017:IST:3110025.3110083, 213 | author = {Ganesh, J. and Gupta, Manish and Varma, Vasudeva}, 214 | title = {{Interpretation of Semantic Tweet Representations}}, 215 | booktitle = {Proceedings of the 2017 IEEE/ACM International Conference 216 | on Advances in Social Networks Analysis and Mining 2017}, 217 | series = {ASONAM '17}, 218 | year = {2017}, 219 | isbn = {978-1-4503-4993-2}, 220 | location = {Sydney, Australia}, 221 | pages = {95--102}, 222 | numpages = {8}, 223 | url = {http://doi.acm.org/10.1145/3110025.3110083}, 224 | doi = {10.1145/3110025.3110083}, 225 | acmid = {3110083}, 226 | publisher = {ACM}, 227 | address = {New York, NY, USA} 228 | } 229 | 230 | @InProceedings{ W17-5221, 231 | author = "Arras, Leila and Montavon, Gr{\'e}goire and M{\"u}ller, 232 | Klaus-Robert and Samek, Wojciech", 233 | title = {{Explaining Recurrent Neural Network Predictions in 234 | Sentiment Analysis}}, 235 | booktitle = "Proceedings of the 8th Workshop on Computational 236 | Approaches to Subjectivity, Sentiment and Social Media 237 | Analysis", 238 | year = "2017", 239 | publisher = "Association for Computational Linguistics", 240 | pages = "159--168", 241 | location = "Copenhagen, Denmark", 242 | url = "http://aclweb.org/anthology/W17-5221" 243 | } 244 | 245 | @Article{ MONTAVON20181, 246 | title = "Methods for interpreting and understanding deep neural 247 | networks", 248 | journal = "Digital Signal Processing", 249 | volume = "73", 250 | pages = "1 - 15", 251 | year = "2018", 252 | issn = "1051-2004", 253 | doi = "https://doi.org/10.1016/j.dsp.2017.10.011", 254 | url = "http://www.sciencedirect.com/science/article/pii/S1051200417302385", 255 | author = "Gr{\'e}goire Montavon and Wojciech Samek and Klaus-Robert 256 | M{\"u}ller", 257 | keywords = "Deep neural networks, Activation maximization, Sensitivity 258 | analysis, Taylor decomposition, Layer-wise relevance 259 | propagation" 260 | } 261 | 262 | @Article{ brunner2018natural, 263 | title = {{Natural Language Multitasking: Analyzing and Improving 264 | Syntactic Saliency of Hidden Representations}}, 265 | author = {Brunner, Gino and Wang, Yuyi and Wattenhofer, Roger and 266 | Weigelt, Michael}, 267 | journal = {The 31st Annual Conference on Neural Information 268 | Processing (NIPS) - Workshop on Learning Disentangled 269 | Features: from Perception to Control}, 270 | year = {2017} 271 | } 272 | 273 | @Article{ Zhang2018, 274 | author = "Zhang, Quan-shi and Zhu, Song-chun", 275 | title = "Visual interpretability for deep learning: A survey", 276 | journal = "Frontiers of Information Technology {\&} Electronic 277 | Engineering", 278 | year = "2018", 279 | month = "Jan", 280 | day = "01", 281 | volume = "19", 282 | number = "1", 283 | pages = "27--39", 284 | issn = "2095-9230", 285 | doi = "10.1631/FITEE.1700808", 286 | url = "https://doi.org/10.1631/FITEE.1700808" 287 | } 288 | 289 | @InProceedings{ P17-1106, 290 | author = "Ding, Yanzhuo and Liu, Yang and Luan, Huanbo and Sun, 291 | Maosong", 292 | title = {{Visualizing and Understanding Neural Machine 293 | Translation}}, 294 | booktitle = "Proceedings of the 55th Annual Meeting of the Association 295 | for Computational Linguistics (Volume 1: Long Papers) ", 296 | year = "2017", 297 | publisher = "Association for Computational Linguistics", 298 | pages = "1150--1159", 299 | location = "Vancouver, Canada", 300 | doi = "10.18653/v1/P17-1106", 301 | url = "http://www.aclweb.org/anthology/P17-1106" 302 | } 303 | 304 | @Article{ adi2017analysis, 305 | title = {Analysis of sentence embedding models using prediction 306 | tasks in natural language processing}, 307 | author = {Adi, Yossi and Kermany, Einat and Belinkov, Yonatan and 308 | Lavi, Ofer and Goldberg, Yoav}, 309 | journal = {IBM Journal of Research and Development}, 310 | volume = {61}, 311 | number = {4}, 312 | pages = {3--1}, 313 | year = {2017}, 314 | publisher = {IBM} 315 | } 316 | 317 | @InCollection{ NIPS2009_3700, 318 | title = {{Reading Tea Leaves: How Humans Interpret Topic Models}}, 319 | author = {Chang, Jonathan and Sean Gerrish and Wang, Chong and 320 | Jordan L. Boyd-graber and David M. Blei}, 321 | booktitle = {Advances in Neural Information Processing Systems 22}, 322 | editor = {Y. Bengio and D. Schuurmans and J. D. Lafferty and C. K. 323 | I. Williams and A. Culotta}, 324 | pages = {288--296}, 325 | year = {2009}, 326 | publisher = {Curran Associates, Inc.}, 327 | url = {http://papers.nips.cc/paper/3700-reading-tea-leaves-how-humans-interpret-topic-models.pdf} 328 | } 329 | 330 | @InProceedings{ james2018beyond, 331 | title = {{Beyond Word Importance: Contextual Decomposition to 332 | Extract Interactions from {LSTM}s}}, 333 | author = {W. James Murdoch and Peter J. Liu and Bin Yu}, 334 | booktitle = {International Conference on Learning Representations}, 335 | year = {2018}, 336 | url = {https://openreview.net/forum?id=rkRwGg-0Z} 337 | } 338 | 339 | @InProceedings{ conneau2018you, 340 | author = "Conneau, Alexis and Kruszewski, Germ{\'a}n and Lample, 341 | Guillaume and Barrault, Lo{\"i}c and Baroni, Marco", 342 | title = "What you can cram into a single {\$}{\&}!{\#}* vector: 343 | Probing sentence embeddings for linguistic properties", 344 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 345 | for Computational Linguistics (Volume 1: Long Papers)", 346 | year = "2018", 347 | publisher = "Association for Computational Linguistics", 348 | pages = "2126--2136", 349 | location = "Melbourne, Australia", 350 | url = "http://aclweb.org/anthology/P18-1198" 351 | } 352 | 353 | @InProceedings{ D17-1042, 354 | author = "Alvarez-Melis, David and Jaakkola, Tommi", 355 | title = "A causal framework for explaining the predictions of 356 | black-box sequence-to-sequence models", 357 | booktitle = "Proceedings of the 2017 Conference on Empirical Methods in 358 | Natural Language Processing", 359 | year = "2017", 360 | publisher = "Association for Computational Linguistics", 361 | pages = "412--421", 362 | location = "Copenhagen, Denmark", 363 | url = "http://aclweb.org/anthology/D17-1042" 364 | } 365 | 366 | @InProceedings{ W16-2524, 367 | author = "Ettinger, Allyson and Elgohary, Ahmed and Resnik, Philip", 368 | title = "Probing for semantic evidence of composition by means of 369 | simple classification tasks", 370 | booktitle = "Proceedings of the 1st Workshop on Evaluating Vector-Space 371 | Representations for NLP ", 372 | year = "2016", 373 | publisher = "Association for Computational Linguistics", 374 | pages = "134--139", 375 | location = "Berlin, Germany", 376 | doi = "10.18653/v1/W16-2524", 377 | url = "http://www.aclweb.org/anthology/W16-2524" 378 | } 379 | 380 | @InProceedings{ nagamine2015exploring, 381 | title = {{Exploring How Deep Neural Networks Form Phonemic 382 | Categories}}, 383 | author = {Nagamine, Tasha and Seltzer, Michael L. and Mesgarani, 384 | Nima}, 385 | booktitle = {Interspeech 2015}, 386 | year = {2015} 387 | } 388 | 389 | @InProceedings{ Nagamine2016, 390 | author = {Tasha Nagamine and Michael L. Seltzer and Nima Mesgarani}, 391 | title = {{On the Role of Nonlinear Transformations in Deep Neural 392 | Network Acoustic Models}}, 393 | year = 2016, 394 | booktitle = {Interspeech 2016}, 395 | doi = {10.21437/Interspeech.2016-1406}, 396 | pages = {803--807} 397 | } 398 | 399 | @InProceedings{ Wang2017, 400 | author = {Shuai Wang and Yanmin Qian and Kai Yu}, 401 | title = {{What Does the Speaker Embedding Encode?}}, 402 | year = 2017, 403 | booktitle = {Interspeech 2017}, 404 | pages = {1497--1501}, 405 | doi = {10.21437/Interspeech.2017-1125}, 406 | url = {http://dx.doi.org/10.21437/Interspeech.2017-1125} 407 | } 408 | 409 | @InProceedings{ chrupala2017representations, 410 | author = "Chrupa{\l}a, Grzegorz and Gelderloos, Lieke and Alishahi, 411 | Afra", 412 | title = {{Representations of language in a model of visually 413 | grounded speech signal}}, 414 | booktitle = "Proceedings of the 55th Annual Meeting of the Association 415 | for Computational Linguistics (Volume 1: Long Papers) ", 416 | year = "2017", 417 | publisher = "Association for Computational Linguistics", 418 | pages = "613--622", 419 | location = "Vancouver, Canada", 420 | doi = "10.18653/v1/P17-1057", 421 | url = "http://aclweb.org/anthology/P17-1057" 422 | } 423 | 424 | @InProceedings{ harwath2017learning, 425 | author = "Harwath, David and Glass, James", 426 | title = {{Learning Word-Like Units from Joint Audio-Visual 427 | Analysis}}, 428 | booktitle = "Proceedings of the 55th Annual Meeting of the Association 429 | for Computational Linguistics (Volume 1: Long Papers) ", 430 | year = "2017", 431 | publisher = "Association for Computational Linguistics", 432 | pages = "506--517", 433 | location = "Vancouver, Canada", 434 | doi = "10.18653/v1/P17-1047" 435 | } 436 | 437 | @InProceedings{ K17-1037, 438 | author = "Alishahi, Afra and Barking, Marie and Chrupa{\l}a, 439 | Grzegorz", 440 | title = "Encoding of phonology in a recurrent neural model of 441 | grounded speech", 442 | booktitle = "Proceedings of the 21st Conference on Computational 443 | Natural Language Learning (CoNLL 2017) ", 444 | year = "2017", 445 | publisher = "Association for Computational Linguistics", 446 | pages = "368--378", 447 | location = "Vancouver, Canada", 448 | doi = "10.18653/v1/K17-1037" 449 | } 450 | 451 | @Article{ linzen2016assessing, 452 | author = "Linzen, Tal and Dupoux, Emmanuel and Goldberg, Yoav", 453 | title = {{Assessing the Ability of LSTMs to Learn Syntax-Sensitive 454 | Dependencies}}, 455 | journal = "Transactions of the Association for Computational 456 | Linguistics", 457 | year = "2016", 458 | volume = "4", 459 | pages = "521--535", 460 | url = "http://aclweb.org/anthology/Q16-1037" 461 | } 462 | 463 | @InProceedings{ szegedy2013intriguing, 464 | title = {Intriguing properties of neural networks}, 465 | author = {Szegedy, Christian and Zaremba, Wojciech and Sutskever, 466 | Ilya and Bruna, Joan and Erhan, Dumitru and Goodfellow, Ian 467 | and Fergus, Rob}, 468 | booktitle = {International Conference on Learning Representations 469 | (ICLR)}, 470 | year = {2014} 471 | } 472 | 473 | @InProceedings{ goodfellow2014explaining, 474 | title = {{Explaining and Harnessing Adversarial Examples}}, 475 | author = {Goodfellow, Ian J. and Shlens, Jonathon and Szegedy, 476 | Christian}, 477 | booktitle = {International Conference on Learning Representations 478 | (ICLR)}, 479 | year = {2015} 480 | } 481 | 482 | @InProceedings{ Papernot:2017:PBA:3052973.3053009, 483 | author = {Papernot, Nicolas and McDaniel, Patrick and Goodfellow, 484 | Ian and Jha, Somesh and Celik, Z. Berkay and Swami, 485 | Ananthram}, 486 | title = {{Practical Black-Box Attacks Against Machine Learning}}, 487 | booktitle = {Proceedings of the 2017 ACM on Asia Conference on Computer 488 | and Communications Security}, 489 | series = {ASIA CCS '17}, 490 | year = {2017}, 491 | isbn = {978-1-4503-4944-4}, 492 | location = {Abu Dhabi, United Arab Emirates}, 493 | pages = {506--519}, 494 | numpages = {14}, 495 | url = {http://doi.acm.org/10.1145/3052973.3053009}, 496 | doi = {10.1145/3052973.3053009}, 497 | acmid = {3053009}, 498 | publisher = {ACM}, 499 | address = {New York, NY, USA}, 500 | keywords = {adversarial machine learning, black-box attack, machine 501 | learning} 502 | } 503 | 504 | @Article{ papernot2016transferability, 505 | title = {{Transferability in Machine Learning: From Phenomena to 506 | Black-Box Attacks using Adversarial Samples}}, 507 | author = {Papernot, Nicolas and McDaniel, Patrick and Goodfellow, 508 | Ian}, 509 | journal = {arXiv preprint arXiv:1605.07277v1}, 510 | year = {2016} 511 | } 512 | 513 | @InProceedings{ liu2016delving, 514 | title = {{Delving into Transferable Adversarial Examples and 515 | Black-box Attacks}}, 516 | author = {Liu, Yanpei and Chen, Xinyun and Liu, Chang and Song, 517 | Dawn}, 518 | booktitle = {International Conference on Learning Representations 519 | (ICLR)}, 520 | year = {2017} 521 | } 522 | 523 | @InProceedings{ 8014906, 524 | author = {Nina Narodytska and Shiva Kasiviswanathan}, 525 | booktitle = {2017 IEEE Conference on Computer Vision and Pattern 526 | Recognition Workshops (CVPRW)}, 527 | title = {{Simple Black-Box Adversarial Attacks on Deep Neural 528 | Networks}}, 529 | year = {2017}, 530 | volume = {}, 531 | number = {}, 532 | pages = {1310-1318}, 533 | keywords = {approximation theory;feedforward neural nets;gradient 534 | methods;image classification;learning (artificial 535 | intelligence);search problems;black-box adversarial 536 | attacks;deep convolutional neural networks;deep neural 537 | networks;image pixels;learning models;local-search based 538 | technique;network gradient;numerical approximation;pattern 539 | recognition performance;Cats;Computer vision;Knowledge 540 | engineering;Network architecture;Neural 541 | networks;Robustness;Training}, 542 | doi = {10.1109/CVPRW.2017.172}, 543 | issn = {}, 544 | month = {July} 545 | } 546 | 547 | @InProceedings{ heigold2017robust, 548 | title = {{How Robust Are Character-Based Word Embeddings in Tagging 549 | and MT Against Wrod Scramlbing or Randdm Nouse?}}, 550 | author = {Heigold, Georg and Neumann, G{\"u}nter and van Genabith, 551 | Josef}, 552 | booktitle = {Proceedings of the 13th Conference of The Association for 553 | Machine Translation in the Americas (Volume 1: Research 554 | Track)}, 555 | year = {2018}, 556 | pages = {68--79}, 557 | month = {March}, 558 | location = {Boston} 559 | } 560 | 561 | @InProceedings{ DBLP:conf/aaai/SakaguchiDPD17, 562 | author = {Keisuke Sakaguchi and Kevin Duh and Matt Post and Benjamin 563 | Van Durme}, 564 | title = {{Robsut Wrod Reocginiton via Semi-Character Recurrent 565 | Neural Network}}, 566 | booktitle = {Proceedings of the Thirty-First {AAAI} Conference on 567 | Artificial Intelligence, February 4-9, 2017, San Francisco, 568 | California, {USA.}}, 569 | pages = {3281--3287}, 570 | year = {2017}, 571 | url = {http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14332}, 572 | publisher = {{AAAI} Press} 573 | } 574 | 575 | @InProceedings{ P18-2006, 576 | author = "Ebrahimi, Javid and Rao, Anyi and Lowd, Daniel and Dou, 577 | Dejing", 578 | title = {{HotFlip: White-Box Adversarial Examples for Text 579 | Classification}}, 580 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 581 | for Computational Linguistics (Volume 2: Short Papers)", 582 | year = "2018", 583 | publisher = "Association for Computational Linguistics", 584 | pages = "31--36", 585 | location = "Melbourne, Australia", 586 | url = "http://aclweb.org/anthology/P18-2006" 587 | } 588 | 589 | @Article{ gao2018black, 590 | title = {{Black-box Generation of Adversarial Text Sequences to 591 | Evade Deep Learning Classifiers}}, 592 | author = {Gao, Ji and Lanchantin, Jack and Soffa, Mary Lou and Qi, 593 | Yanjun}, 594 | journal = {arXiv preprint arXiv:1801.04354v5}, 595 | year = {2018} 596 | } 597 | 598 | @InProceedings{ papernot2016crafting, 599 | title = {{Crafting Adversarial Input Sequences for Recurrent Neural 600 | Networks}}, 601 | author = {Papernot, Nicolas and McDaniel, Patrick and Swami, 602 | Ananthram and Harang, Richard}, 603 | booktitle = {Military Communications Conference, MILCOM 2016-2016 604 | IEEE}, 605 | pages = {49--54}, 606 | year = {2016}, 607 | organization = {IEEE} 608 | } 609 | 610 | @Article{ samanta2017towards, 611 | title = {{Towards Crafting Text Adversarial Samples}}, 612 | author = {Samanta, Suranjana and Mehta, Sameep}, 613 | journal = {arXiv preprint arXiv:1707.02812v1}, 614 | year = {2017} 615 | } 616 | 617 | @InProceedings{ liang2017deep, 618 | title = {{Deep Text Classification Can be Fooled}}, 619 | author = {Bin Liang and Hongcheng Li and Miaoqiang Su and Pan Bian 620 | and Xirong Li and Wenchang Shi}, 621 | booktitle = {Proceedings of the Twenty-Seventh International Joint 622 | Conference on Artificial Intelligence, {IJCAI-18}}, 623 | publisher = {International Joint Conferences on Artificial Intelligence 624 | Organization}, 625 | pages = {4208--4215}, 626 | year = {2018}, 627 | month = {7}, 628 | doi = {10.24963/ijcai.2018/585}, 629 | url = {https://doi.org/10.24963/ijcai.2018/585} 630 | } 631 | 632 | @InProceedings{ tran2018importance, 633 | author = "Tran, Ke and Bisazza, Arianna and Monz, Christof", 634 | title = {{The Importance of Being Recurrent for Modeling 635 | Hierarchical Structure}}, 636 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 637 | Natural Language Processing", 638 | year = "2018", 639 | publisher = "Association for Computational Linguistics", 640 | pages = "4731--4736", 641 | location = "Brussels, Belgium", 642 | url = "http://aclweb.org/anthology/D18-1503" 643 | } 644 | 645 | @Article{ burchardt2017linguistic, 646 | title = {{A Linguistic Evaluation of Rule-Based, Phrase-Based, and 647 | Neural MT Engines}}, 648 | author = {Burchardt, Aljoscha and Macketanz, Vivien and Dehdari, Jon 649 | and Heigold, Georg and Peter, Jan-Thorsten and Williams, 650 | Philip}, 651 | journal = {The Prague Bulletin of Mathematical Linguistics}, 652 | volume = {108}, 653 | number = {1}, 654 | pages = {159--170}, 655 | year = {2017}, 656 | publisher = {De Gruyter Open} 657 | } 658 | 659 | @InProceedings{ D17-1263, 660 | author = "Isabelle, Pierre and Cherry, Colin and Foster, George", 661 | title = {{A Challenge Set Approach to Evaluating Machine 662 | Translation}}, 663 | booktitle = "Proceedings of the 2017 Conference on Empirical Methods in 664 | Natural Language Processing", 665 | year = "2017", 666 | publisher = "Association for Computational Linguistics", 667 | pages = "2486--2496", 668 | location = "Copenhagen, Denmark", 669 | url = "http://aclweb.org/anthology/D17-1263" 670 | } 671 | 672 | @InProceedings{ C90-2037, 673 | author = "King, Margaret and Falkedal, Kirsten", 674 | title = {{Using Test Suites in Evaluation of Machine Translation 675 | Systems}}, 676 | booktitle = "COLNG 1990 Volume 2: Papers presented to the 13th 677 | International Conference on Computational Linguistics", 678 | year = "1990", 679 | url = "http://www.aclweb.org/anthology/C90-2037" 680 | } 681 | 682 | @InProceedings{ C96-2120, 683 | author = "Lehmann, Sabine and Oepen, Stephan and Regnier-Prost, 684 | Sylvie and Netter, Klaus and Lux, Veronika and Klein, 685 | Judith and Falkedal, Kirsten and Fouvry, Frederik and 686 | Estival, Dominique and Dauphin, Eva and Compagnion, Herve 687 | and Baur, Judith and Balkan, Lorna and Arnold, Doug", 688 | title = {{TSNLP - Test Suites for Natural Language Processing}}, 689 | booktitle = "COLING 1996 Volume 2: The 16th International Conference on 690 | Computational Linguistics", 691 | year = "1996", 692 | url = "http://www.aclweb.org/anthology/C96-2120" 693 | } 694 | 695 | @InProceedings{ W17-4705, 696 | author = "Burlot, Franck and Yvon, Fran{\c{c}}ois", 697 | title = {{Evaluating the morphological competence of Machine 698 | Translation Systems}}, 699 | booktitle = "Proceedings of the Second Conference on Machine 700 | Translation", 701 | year = "2017", 702 | publisher = "Association for Computational Linguistics", 703 | pages = "43--55", 704 | location = "Copenhagen, Denmark", 705 | url = "http://aclweb.org/anthology/W17-4705" 706 | } 707 | 708 | @InProceedings{ E17-2060, 709 | author = "Sennrich, Rico", 710 | title = {{How Grammatical is Character-level Neural Machine 711 | Translation? Assessing MT Quality with Contrastive 712 | Translation Pairs}}, 713 | booktitle = "Proceedings of the 15th Conference of the European Chapter 714 | of the Association for Computational Linguistics: Volume 2, 715 | Short Papers", 716 | year = "2017", 717 | publisher = "Association for Computational Linguistics", 718 | pages = "376--382", 719 | location = "Valencia, Spain", 720 | url = "http://aclweb.org/anthology/E17-2060" 721 | } 722 | 723 | @Article{ hupkes2017visualisation, 724 | title = {Visualisation and 'diagnostic classifiers' reveal how 725 | recurrent and recursive neural networks process 726 | hierarchical structure}, 727 | author = {Hupkes, Dieuwke and Veldhoen, Sara and Zuidema, Willem}, 728 | journal = {Journal of Artificial Intelligence Research}, 729 | volume = {61}, 730 | pages = {907--926}, 731 | year = {2018} 732 | } 733 | 734 | @InProceedings{ veldhoen2016diagnostic, 735 | title = {{Diagnostic Classifiers: Revealing how Neural Networks 736 | Process Hierarchical Structure}}, 737 | author = {Veldhoen, Sara and Hupkes, Dieuwke and Zuidema, Willem}, 738 | booktitle = {CEUR Workshop Proceedings}, 739 | year = {2016} 740 | } 741 | 742 | @InProceedings{ N18-1118, 743 | author = "Bawden, Rachel and Sennrich, Rico and Birch, Alexandra and 744 | Haddow, Barry", 745 | title = {{Evaluating Discourse Phenomena in Neural Machine 746 | Translation}}, 747 | booktitle = "Proceedings of the 2018 Conference of the North American 748 | Chapter of the Association for Computational Linguistics: 749 | Human Language Technologies, Volume 1 (Long Papers)", 750 | year = "2018", 751 | publisher = "Association for Computational Linguistics", 752 | pages = "1304--1313", 753 | location = "New Orleans, Louisiana", 754 | url = "http://aclweb.org/anthology/N18-1118" 755 | } 756 | 757 | @InProceedings{ DoshiKim2017Interpretability, 758 | title = {{Towards A Rigorous Science of Interpretable Machine 759 | Learning}}, 760 | author = {Doshi-Velez, Finale and Kim, Been}, 761 | booktitle = {arXiv preprint arXiv:1702.08608v2}, 762 | year = {2017} 763 | } 764 | 765 | @Article{ dasgupta2018evaluating, 766 | title = {{Evaluating Compositionality in Sentence Embeddings}}, 767 | author = {Dasgupta, Ishita and Guo, Demi and Stuhlm{\"u}ller, 768 | Andreas and Gershman, Samuel J. and Goodman, Noah D.}, 769 | journal = {arXiv preprint arXiv:1802.04302v2}, 770 | year = {2018} 771 | } 772 | 773 | @InProceedings{ lake2018, 774 | title = {{Generalization without Systematicity: On the 775 | Compositional Skills of Sequence-to-Sequence Recurrent 776 | Networks}}, 777 | author = {Lake, Brenden and Baroni, Marco}, 778 | booktitle = {Proceedings of the 35th International Conference on 779 | Machine Learning}, 780 | pages = {2873--2882}, 781 | year = {2018}, 782 | editor = {Dy, Jennifer and Krause, Andreas}, 783 | volume = {80}, 784 | series = {Proceedings of Machine Learning Research}, 785 | address = {Stockholmsm{\"a}ssan, Stockholm, Sweden}, 786 | month = {10--15 Jul}, 787 | publisher = {PMLR}, 788 | pdf = {http://proceedings.mlr.press/v80/lake18a/lake18a.pdf}, 789 | url = {http://proceedings.mlr.press/v80/lake18a.html} 790 | } 791 | 792 | @Article{ doshi2017accountability, 793 | title = {{Accountability of AI Under the Law: The Role of 794 | Explanation}}, 795 | author = {Doshi-Velez, Finale and Kortz, Mason and Budish, Ryan and 796 | Bavitz, Chris and Gershman, Sam and O'Brien, David and 797 | Shieber, Stuart and Waldo, James and Weinberger, David and 798 | Wood, Alexandra}, 799 | journal = {Berkman Center Publication Forthcoming}, 800 | year = {2017} 801 | } 802 | 803 | @InProceedings{ N07-1033, 804 | author = "Zaidan, Omar and Eisner, Jason and Piatko, Christine", 805 | title = {{Using ``Annotator Rationales'' to Improve Machine 806 | Learning for Text Categorization}}, 807 | booktitle = "Human Language Technologies 2007: The Conference of the 808 | North American Chapter of the Association for Computational 809 | Linguistics; Proceedings of the Main Conference", 810 | year = "2007", 811 | publisher = "Association for Computational Linguistics", 812 | pages = "260--267", 813 | location = "Rochester, New York", 814 | url = "http://www.aclweb.org/anthology/N07-1033" 815 | } 816 | 817 | @InProceedings{ D16-1076, 818 | author = "Zhang, Ye and Marshall, Iain and Wallace, Byron C.", 819 | title = {{Rationale-Augmented Convolutional Neural Networks for 820 | Text Classification}}, 821 | booktitle = "Proceedings of the 2016 Conference on Empirical Methods in 822 | Natural Language Processing ", 823 | year = "2016", 824 | publisher = "Association for Computational Linguistics", 825 | pages = "795--804", 826 | location = "Austin, Texas", 827 | doi = "10.18653/v1/D16-1076", 828 | url = "http://www.aclweb.org/anthology/D16-1076" 829 | } 830 | 831 | @InProceedings{ belinkov:2018:ICLR, 832 | title = {{Synthetic and Natural Noise Both Break Neural Machine 833 | Translation}}, 834 | author = {Belinkov, Yonatan and Bisk, Yonatan}, 835 | booktitle = {International Conference on Learning Representations 836 | (ICLR)}, 837 | year = {2018}, 838 | month = {April} 839 | } 840 | 841 | @InProceedings{ jia-liang:2017:EMNLP2017, 842 | author = "Jia, Robin and Liang, Percy", 843 | title = "Adversarial Examples for Evaluating Reading Comprehension 844 | Systems", 845 | booktitle = "Proceedings of the 2017 Conference on Empirical Methods in 846 | Natural Language Processing", 847 | year = "2017", 848 | publisher = "Association for Computational Linguistics", 849 | pages = "2021--2031", 850 | location = "Copenhagen, Denmark", 851 | url = "http://aclweb.org/anthology/D17-1215" 852 | } 853 | 854 | @InProceedings{ gulordava2018colorless, 855 | author = "Gulordava, Kristina and Bojanowski, Piotr and Grave, 856 | Edouard and Linzen, Tal and Baroni, Marco", 857 | title = {{Colorless Green Recurrent Networks Dream 858 | Hierarchically}}, 859 | booktitle = "Proceedings of the 2018 Conference of the North American 860 | Chapter of the Association for Computational Linguistics: 861 | Human Language Technologies, Volume 1 (Long Papers)", 862 | year = "2018", 863 | publisher = "Association for Computational Linguistics", 864 | pages = "1195--1205", 865 | location = "New Orleans, Louisiana", 866 | url = "http://aclweb.org/anthology/N18-1108" 867 | } 868 | 869 | @Article{ elman1991distributed, 870 | title = {Distributed representations, simple recurrent networks, 871 | and grammatical structure}, 872 | author = {Elman, Jeffrey L.}, 873 | journal = {Machine learning}, 874 | volume = {7}, 875 | number = {2-3}, 876 | pages = {195--225}, 877 | year = {1991}, 878 | publisher = {Springer} 879 | } 880 | 881 | @Article{ bahdanau2014neural, 882 | title = {{Neural Machine Translation by Jointly Learning to Align 883 | and Translate}}, 884 | author = {Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua}, 885 | journal = {arXiv preprint arXiv:1409.0473v7}, 886 | year = {2014} 887 | } 888 | 889 | @InProceedings{ N16-1082, 890 | author = "Li, Jiwei and Chen, Xinlei and Hovy, Eduard and Jurafsky, 891 | Dan", 892 | title = {{Visualizing and Understanding Neural Models in NLP}}, 893 | booktitle = "Proceedings of the 2016 Conference of the North American 894 | Chapter of the Association for Computational Linguistics: 895 | Human Language Technologies ", 896 | year = "2016", 897 | publisher = "Association for Computational Linguistics", 898 | pages = "681--691", 899 | location = "San Diego, California", 900 | doi = "10.18653/v1/N16-1082", 901 | url = "http://www.aclweb.org/anthology/N16-1082" 902 | } 903 | 904 | @InProceedings{ D16-1216, 905 | author = "Aubakirova, Malika and Bansal, Mohit", 906 | title = {{Interpreting Neural Networks to Improve Politeness 907 | Comprehension}}, 908 | booktitle = "Proceedings of the 2016 Conference on Empirical Methods in 909 | Natural Language Processing ", 910 | year = "2016", 911 | publisher = "Association for Computational Linguistics", 912 | pages = "2035--2041", 913 | location = "Austin, Texas", 914 | doi = "10.18653/v1/D16-1216", 915 | url = "http://www.aclweb.org/anthology/D16-1216" 916 | } 917 | 918 | @InProceedings{ D16-1011, 919 | author = "Lei, Tao and Barzilay, Regina and Jaakkola, Tommi", 920 | title = {{Rationalizing Neural Predictions}}, 921 | booktitle = "Proceedings of the 2016 Conference on Empirical Methods in 922 | Natural Language Processing ", 923 | year = "2016", 924 | publisher = "Association for Computational Linguistics", 925 | pages = "107--117", 926 | location = "Austin, Texas", 927 | doi = "10.18653/v1/D16-1011", 928 | url = "http://www.aclweb.org/anthology/D16-1011" 929 | } 930 | 931 | @InProceedings{ W17-4702, 932 | author = "Rios Gonzales, Annette and Mascarell, Laura and Sennrich, 933 | Rico", 934 | title = {{Improving Word Sense Disambiguation in Neural Machine 935 | Translation with Sense Embeddings}}, 936 | booktitle = "Proceedings of the Second Conference on Machine 937 | Translation", 938 | year = "2017", 939 | publisher = "Association for Computational Linguistics", 940 | pages = "11--19", 941 | location = "Copenhagen, Denmark", 942 | url = "http://aclweb.org/anthology/W17-4702" 943 | } 944 | 945 | @InProceedings{ park2018multimodal, 946 | author = {Park, Dong Huk and Hendricks, Lisa Anne and Akata, Zeynep 947 | and Rohrbach, Anna and Schiele, Bernt and Darrell, Trevor 948 | and Rohrbach, Marcus}, 949 | title = {{Multimodal Explanations: Justifying Decisions and 950 | Pointing to the Evidence}}, 951 | booktitle = {The IEEE Conference on Computer Vision and Pattern 952 | Recognition (CVPR)}, 953 | month = {June}, 954 | year = {2018} 955 | } 956 | 957 | @InProceedings{ N15-1004, 958 | author = "Fyshe, Alona and Wehbe, Leila and Talukdar, Partha P. and 959 | Murphy, Brian and Mitchell, Tom M.", 960 | title = {{A Compositional and Interpretable Semantic Space}}, 961 | booktitle = "Proceedings of the 2015 Conference of the North American 962 | Chapter of the Association for Computational Linguistics: 963 | Human Language Technologies ", 964 | year = "2015", 965 | publisher = "Association for Computational Linguistics", 966 | pages = "32--41", 967 | location = "Denver, Colorado", 968 | doi = "10.3115/v1/N15-1004", 969 | url = "http://www.aclweb.org/anthology/N15-1004" 970 | } 971 | 972 | @InProceedings{ C12-1118, 973 | author = "Murphy, Brian and Talukdar, Partha and Mitchell, Tom", 974 | title = {{Learning Effective and Interpretable Semantic Models 975 | using Non-Negative Sparse Embedding}}, 976 | booktitle = "Proceedings of COLING 2012", 977 | year = "2012", 978 | publisher = "The COLING 2012 Organizing Committee", 979 | pages = "1933--1950", 980 | location = "Mumbai, India", 981 | url = "http://www.aclweb.org/anthology/C12-1118" 982 | } 983 | 984 | @InProceedings{ pmlr-v70-sundararajan17a, 985 | title = {{Axiomatic Attribution for Deep Networks}}, 986 | author = {Mukund Sundararajan and Ankur Taly and Qiqi Yan}, 987 | booktitle = {Proceedings of the 34th International Conference on 988 | Machine Learning}, 989 | pages = {3319--3328}, 990 | year = {2017}, 991 | editor = {Doina Precup and Yee Whye Teh}, 992 | volume = {70}, 993 | series = {Proceedings of Machine Learning Research}, 994 | address = {International Convention Centre, Sydney, Australia}, 995 | month = {06--11 Aug}, 996 | publisher = {PMLR}, 997 | pdf = {http://proceedings.mlr.press/v70/sundararajan17a/sundararajan17a.pdf}, 998 | url = {http://proceedings.mlr.press/v70/sundararajan17a.html} 999 | } 1000 | 1001 | @InProceedings{ Drexler2017AnalysisOA, 1002 | title = {{Analysis of Audio-Visual Features for Unsupervised Speech 1003 | Recognition}}, 1004 | author = {Jennifer Drexler and James Glass}, 1005 | year = {2017}, 1006 | booktitle = {International Workshop on Grounding Language 1007 | Understanding}, 1008 | location = {Stockholm, Sweden} 1009 | } 1010 | 1011 | @InProceedings{ mccoy2018revisiting, 1012 | title = {{Revisiting the poverty of the stimulus: Hierarchical 1013 | generalization without a hierarchical bias in recurrent 1014 | neural networks}}, 1015 | author = {McCoy, R. Thomas and Frank, Robert and Linzen, Tal}, 1016 | booktitle = {Proceedings of the 40th Annual Conference of the Cognitive 1017 | Science Society}, 1018 | year = {2018} 1019 | } 1020 | 1021 | @InProceedings{ N18-1170, 1022 | author = "Iyyer, Mohit and Wieting, John and Gimpel, Kevin and 1023 | Zettlemoyer, Luke", 1024 | title = {{Adversarial Example Generation with Syntactically 1025 | Controlled Paraphrase Networks}}, 1026 | booktitle = "Proceedings of the 2018 Conference of the North American 1027 | Chapter of the Association for Computational Linguistics: 1028 | Human Language Technologies, Volume 1 (Long Papers)", 1029 | year = "2018", 1030 | publisher = "Association for Computational Linguistics", 1031 | pages = "1875--1885", 1032 | location = "New Orleans, Louisiana", 1033 | url = "http://aclweb.org/anthology/N18-1170" 1034 | } 1035 | 1036 | @InProceedings{ N18-2122, 1037 | author = "Dharmaretnam, Dhanush and Fyshe, Alona", 1038 | title = {{The Emergence of Semantics in Neural Network 1039 | Representations of Visual Information}}, 1040 | booktitle = "Proceedings of the 2018 Conference of the North American 1041 | Chapter of the Association for Computational Linguistics: 1042 | Human Language Technologies, Volume 2 (Short Papers)", 1043 | year = "2018", 1044 | publisher = "Association for Computational Linguistics", 1045 | pages = "776--780", 1046 | location = "New Orleans, Louisiana", 1047 | url = "http://aclweb.org/anthology/N18-2122" 1048 | } 1049 | 1050 | @InProceedings{ W18-3024, 1051 | author = "Liu, Nelson F. and Levy, Omer and Schwartz, Roy and Tan, 1052 | Chenhao and Smith, Noah A.", 1053 | title = {{LSTMs Exploit Linguistic Attributes of Data}}, 1054 | booktitle = "Proceedings of The Third Workshop on Representation 1055 | Learning for NLP", 1056 | year = "2018", 1057 | publisher = "Association for Computational Linguistics", 1058 | pages = "180--186", 1059 | location = "Melbourne, Australia", 1060 | url = "http://aclweb.org/anthology/W18-3024" 1061 | } 1062 | 1063 | @InProceedings{ D16-1248, 1064 | author = "Shi, Xing and Knight, Kevin and Yuret, Deniz", 1065 | title = {{Why Neural Translations are the Right Length}}, 1066 | booktitle = "Proceedings of the 2016 Conference on Empirical Methods in 1067 | Natural Language Processing ", 1068 | year = "2016", 1069 | publisher = "Association for Computational Linguistics", 1070 | pages = "2278--2282", 1071 | location = "Austin, Texas", 1072 | doi = "10.18653/v1/D16-1248", 1073 | url = "http://www.aclweb.org/anthology/D16-1248" 1074 | } 1075 | 1076 | @InProceedings{ naik2018stress, 1077 | author = "Naik, Aakanksha and Ravichander, Abhilasha and Sadeh, 1078 | Norman and Rose, Carolyn and Neubig, Graham", 1079 | title = {{Stress Test Evaluation for Natural Language Inference}}, 1080 | booktitle = "Proceedings of the 27th International Conference on 1081 | Computational Linguistics", 1082 | year = "2018", 1083 | publisher = "Association for Computational Linguistics", 1084 | pages = "2340--2353", 1085 | location = "Santa Fe, New Mexico, USA", 1086 | url = "http://aclweb.org/anthology/C18-1198" 1087 | } 1088 | 1089 | @Article{ singh2018hierarchical, 1090 | title = {{Hierarchical interpretations for neural network 1091 | predictions}}, 1092 | author = {Singh, Chandan and Murdoch, W. James and Yu, Bin}, 1093 | journal = {arXiv preprint arXiv:1806.05337v1}, 1094 | year = {2018} 1095 | } 1096 | 1097 | @InProceedings{ N18-1091, 1098 | author = "Gaddy, David and Stern, Mitchell and Klein, Dan", 1099 | title = {{What's Going On in Neural Constituency Parsers? An 1100 | Analysis}}, 1101 | booktitle = "Proceedings of the 2018 Conference of the North American 1102 | Chapter of the Association for Computational Linguistics: 1103 | Human Language Technologies, Volume 1 (Long Papers)", 1104 | year = "2018", 1105 | publisher = "Association for Computational Linguistics", 1106 | pages = "999--1010", 1107 | location = "New Orleans, Louisiana", 1108 | url = "http://aclweb.org/anthology/N18-1091" 1109 | } 1110 | 1111 | @InProceedings{ P18-1079, 1112 | author = "Ribeiro, Marco Tulio and Singh, Sameer and Guestrin, 1113 | Carlos", 1114 | title = {{Semantically Equivalent Adversarial Rules for Debugging 1115 | NLP models}}, 1116 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1117 | for Computational Linguistics (Volume 1: Long Papers)", 1118 | year = "2018", 1119 | publisher = "Association for Computational Linguistics", 1120 | pages = "856--865", 1121 | location = "Melbourne, Australia", 1122 | url = "http://aclweb.org/anthology/P18-1079" 1123 | } 1124 | 1125 | @InProceedings{ unanue2018shared, 1126 | author = "Jauregi Unanue, Inigo and Zare Borzeshi, Ehsan and 1127 | Piccardi, Massimo", 1128 | title = {{A Shared Attention Mechanism for Interpretation of Neural 1129 | Automatic Post-Editing Systems}}, 1130 | booktitle = "Proceedings of the 2nd Workshop on Neural Machine 1131 | Translation and Generation", 1132 | year = "2018", 1133 | publisher = "Association for Computational Linguistics", 1134 | pages = "11--17", 1135 | location = "Melbourne, Australia", 1136 | url = "http://aclweb.org/anthology/W18-2702" 1137 | } 1138 | 1139 | @InProceedings{ shi2018learning, 1140 | author = "Shi, Haoyue and Mao, Jiayuan and Xiao, Tete and Jiang, 1141 | Yuning and Sun, Jian", 1142 | title = {{Learning Visually-Grounded Semantics from Contrastive 1143 | Adversarial Samples}}, 1144 | booktitle = "Proceedings of the 27th International Conference on 1145 | Computational Linguistics", 1146 | year = "2018", 1147 | publisher = "Association for Computational Linguistics", 1148 | pages = "3715--3727", 1149 | location = "Santa Fe, New Mexico, USA", 1150 | url = "http://aclweb.org/anthology/C18-1315" 1151 | } 1152 | 1153 | @Article{ isabelle2018challenge, 1154 | title = {{A Challenge Set for French--> English Machine 1155 | Translation}}, 1156 | author = {Isabelle, Pierre and Kuhn, Roland}, 1157 | journal = {arXiv preprint arXiv:1806.02725v2}, 1158 | year = {2018} 1159 | } 1160 | 1161 | @InProceedings{ W18-2903, 1162 | author = "Maillard, Jean and Clark, Stephen", 1163 | title = {{Latent Tree Learning with Differentiable Parsers: 1164 | Shift-Reduce Parsing and Chart Parsing}}, 1165 | booktitle = "Proceedings of the Workshop on the Relevance of Linguistic 1166 | Structure in Neural Architectures for NLP", 1167 | year = "2018", 1168 | publisher = "Association for Computational Linguistics", 1169 | pages = "13--18", 1170 | location = "Melbourne, Australia", 1171 | url = "http://aclweb.org/anthology/W18-2903" 1172 | } 1173 | 1174 | @Article{ yang2018greedy, 1175 | title = {{Greedy Attack and Gumbel Attack: Generating Adversarial 1176 | Examples for Discrete Data}}, 1177 | author = {Yang, Puyudi and Chen, Jianbo and Hsieh, Cho-Jui and Wang, 1178 | Jane-Ling and Jordan, Michael I.}, 1179 | journal = {arXiv preprint arXiv:1805.12316v1}, 1180 | year = {2018} 1181 | } 1182 | 1183 | @InProceedings{ tay2018couplenet, 1184 | title = {{CoupleNet: Paying Attention to Couples with Coupled 1185 | Attention for Relationship Recommendation}}, 1186 | author = {Tay, Yi and Luu, Anh Tuan and Hui, Siu Cheung}, 1187 | booktitle = {Proceedings of the Twelfth International AAAI Conference 1188 | on Web and Social Media (ICWSM)}, 1189 | year = {2018} 1190 | } 1191 | 1192 | @InProceedings{ P18-1117, 1193 | author = "Voita, Elena and Serdyukov, Pavel and Sennrich, Rico and 1194 | Titov, Ivan", 1195 | title = {{Context-Aware Neural Machine Translation Learns Anaphora 1196 | Resolution}}, 1197 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1198 | for Computational Linguistics (Volume 1: Long Papers)", 1199 | year = "2018", 1200 | publisher = "Association for Computational Linguistics", 1201 | pages = "1264--1274", 1202 | location = "Melbourne, Australia", 1203 | url = "http://aclweb.org/anthology/P18-1117" 1204 | } 1205 | 1206 | @InProceedings{ P18-1126, 1207 | author = "C{\'i}fka, Ond{\v{r}}ej and Bojar, Ond{\v{r}}ej", 1208 | title = {{Are BLEU and Meaning Representation in Opposition?}}, 1209 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1210 | for Computational Linguistics (Volume 1: Long Papers)", 1211 | year = "2018", 1212 | publisher = "Association for Computational Linguistics", 1213 | pages = "1362--1371", 1214 | location = "Melbourne, Australia", 1215 | url = "http://aclweb.org/anthology/P18-1126" 1216 | } 1217 | 1218 | @InProceedings{ P18-1176, 1219 | author = "Mudrakarta, Pramod Kaushik and Taly, Ankur and 1220 | Sundararajan, Mukund and Dhamdhere, Kedar", 1221 | title = {{Did the Model Understand the Question?}}, 1222 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1223 | for Computational Linguistics (Volume 1: Long Papers)", 1224 | year = "2018", 1225 | publisher = "Association for Computational Linguistics", 1226 | pages = "1896--1906", 1227 | location = "Melbourne, Australia", 1228 | url = "http://aclweb.org/anthology/P18-1176" 1229 | } 1230 | 1231 | @InProceedings{ P18-1027, 1232 | author = "Khandelwal, Urvashi and He, He and Qi, Peng and Jurafsky, 1233 | Dan", 1234 | title = {{Sharp Nearby, Fuzzy Far Away: How Neural Language Models 1235 | Use Context}}, 1236 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1237 | for Computational Linguistics (Volume 1: Long Papers)", 1238 | year = "2018", 1239 | publisher = "Association for Computational Linguistics", 1240 | pages = "284--294", 1241 | location = "Melbourne, Australia", 1242 | url = "http://aclweb.org/anthology/P18-1027" 1243 | } 1244 | 1245 | @InProceedings{ P18-2117, 1246 | author = "Weiss, Gail and Goldberg, Yoav and Yahav, Eran", 1247 | title = {{On the Practical Computational Power of Finite Precision 1248 | RNNs for Language Recognition}}, 1249 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1250 | for Computational Linguistics (Volume 2: Short Papers)", 1251 | year = "2018", 1252 | publisher = "Association for Computational Linguistics", 1253 | pages = "740--745", 1254 | location = "Melbourne, Australia", 1255 | url = "http://aclweb.org/anthology/P18-2117" 1256 | } 1257 | 1258 | @InProceedings{ N18-1179, 1259 | author = "Sanchez, Ivan and Mitchell, Jeff and Riedel, Sebastian", 1260 | title = {{Behavior Analysis of NLI Models: Uncovering the Influence 1261 | of Three Factors on Robustness}}, 1262 | booktitle = "Proceedings of the 2018 Conference of the North American 1263 | Chapter of the Association for Computational Linguistics: 1264 | Human Language Technologies, Volume 1 (Long Papers)", 1265 | year = "2018", 1266 | publisher = "Association for Computational Linguistics", 1267 | pages = "1975--1985", 1268 | location = "New Orleans, Louisiana", 1269 | url = "http://aclweb.org/anthology/N18-1179" 1270 | } 1271 | 1272 | @InProceedings{ P18-2003, 1273 | author = "Blevins, Terra and Levy, Omer and Zettlemoyer, Luke", 1274 | title = {{Deep RNNs Encode Soft Hierarchical Syntax}}, 1275 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1276 | for Computational Linguistics (Volume 2: Short Papers)", 1277 | year = "2018", 1278 | publisher = "Association for Computational Linguistics", 1279 | pages = "14--19", 1280 | location = "Melbourne, Australia", 1281 | url = "http://aclweb.org/anthology/P18-2003" 1282 | } 1283 | 1284 | @InProceedings{ ijcai2018-601, 1285 | title = {{Interpretable Adversarial Perturbation in Input Embedding 1286 | Space for Text}}, 1287 | author = {Motoki Sato and Jun Suzuki and Hiroyuki Shindo and Yuji 1288 | Matsumoto}, 1289 | booktitle = {Proceedings of the Twenty-Seventh International Joint 1290 | Conference on Artificial Intelligence, {IJCAI-18}}, 1291 | publisher = {International Joint Conferences on Artificial Intelligence 1292 | Organization}, 1293 | pages = {4323--4330}, 1294 | year = {2018}, 1295 | month = {7}, 1296 | doi = {10.24963/ijcai.2018/601}, 1297 | url = {https://doi.org/10.24963/ijcai.2018/601} 1298 | } 1299 | 1300 | @InProceedings{ P18-2103, 1301 | author = "Glockner, Max and Shwartz, Vered and Goldberg, Yoav", 1302 | title = {{Breaking NLI Systems with Sentences that Require Simple 1303 | Lexical Inferences}}, 1304 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1305 | for Computational Linguistics (Volume 2: Short Papers)", 1306 | year = "2018", 1307 | publisher = "Association for Computational Linguistics", 1308 | pages = "650--655", 1309 | location = "Melbourne, Australia", 1310 | url = "http://aclweb.org/anthology/P18-2103" 1311 | } 1312 | 1313 | @Article{ strobelt2018seq2seq, 1314 | title = {{Seq2Seq-Vis: A Visual Debugging Tool for 1315 | Sequence-to-Sequence Models}}, 1316 | author = {Strobelt, Hendrik and Gehrmann, Sebastian and Behrisch, 1317 | Michael and Perer, Adam and Pfister, Hanspeter and Rush, 1318 | Alexander M.}, 1319 | journal = {arXiv preprint arXiv:1804.09299v1}, 1320 | year = {2018} 1321 | } 1322 | 1323 | @InProceedings{ N18-2002, 1324 | author = "Rudinger, Rachel and Naradowsky, Jason and Leonard, Brian 1325 | and Van Durme, Benjamin", 1326 | title = {{Gender Bias in Coreference Resolution}}, 1327 | booktitle = "Proceedings of the 2018 Conference of the North American 1328 | Chapter of the Association for Computational Linguistics: 1329 | Human Language Technologies, Volume 2 (Short Papers)", 1330 | year = "2018", 1331 | publisher = "Association for Computational Linguistics", 1332 | pages = "8--14", 1333 | location = "New Orleans, Louisiana", 1334 | url = "http://aclweb.org/anthology/N18-2002" 1335 | } 1336 | 1337 | @InProceedings{ N18-2003, 1338 | author = "Zhao, Jieyu and Wang, Tianlu and Yatskar, Mark and 1339 | Ordonez, Vicente and Chang, Kai-Wei", 1340 | title = {{Gender Bias in Coreference Resolution: Evaluation and 1341 | Debiasing Methods}}, 1342 | booktitle = "Proceedings of the 2018 Conference of the North American 1343 | Chapter of the Association for Computational Linguistics: 1344 | Human Language Technologies, Volume 2 (Short Papers)", 1345 | year = "2018", 1346 | publisher = "Association for Computational Linguistics", 1347 | pages = "15--20", 1348 | location = "New Orleans, Louisiana", 1349 | url = "http://aclweb.org/anthology/N18-2003" 1350 | } 1351 | 1352 | @Article{ ahmad2018multi, 1353 | title = {{Multi-task Learning for Universal Sentence Embeddings: A 1354 | Thorough Evaluation using Transfer and Auxiliary Tasks}}, 1355 | author = {Ahmad, Wasi Uddin and Bai, Xueying and Huang, Zhechao and 1356 | Jiang, Chao and Peng, Nanyun and Chang, Kai-Wei}, 1357 | journal = {arXiv preprint arXiv:1804.07911v2}, 1358 | year = {2018} 1359 | } 1360 | 1361 | @InProceedings{ alzantot2018generating, 1362 | author = "Alzantot, Moustafa and Sharma, Yash and Elgohary, Ahmed 1363 | and Ho, Bo-Jhang and Srivastava, Mani and Chang, Kai-Wei", 1364 | title = {{Generating Natural Language Adversarial Examples}}, 1365 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 1366 | Natural Language Processing", 1367 | year = "2018", 1368 | publisher = "Association for Computational Linguistics", 1369 | pages = "2890--2896", 1370 | location = "Brussels, Belgium", 1371 | url = "http://aclweb.org/anthology/D18-1316" 1372 | } 1373 | 1374 | @InProceedings{ zhao2018generating, 1375 | title = {{Generating Natural Adversarial Examples}}, 1376 | author = {Zhengli Zhao and Dheeru Dua and Sameer Singh}, 1377 | booktitle = {International Conference on Learning Representations}, 1378 | year = {2018}, 1379 | url = {https://openreview.net/forum?id=H1BLjgZCb} 1380 | } 1381 | 1382 | @Article{ freeman2018paying, 1383 | title = {{Paying Attention to Attention: Highlighting Influential 1384 | Samples in Sequential Analysis}}, 1385 | author = {Freeman, Cynthia and Merriman, Jonathan and Aggarwal, 1386 | Abhinav and Beaver, Ian and Mueen, Abdullah}, 1387 | journal = {arXiv preprint arXiv:1808.02113v1}, 1388 | year = {2018} 1389 | } 1390 | 1391 | @Article{ wang2018glue, 1392 | title = {{GLUE: A Multi-Task Benchmark and Analysis Platform for 1393 | Natural Language Understanding}}, 1394 | author = {Wang, Alex and Singh, Amapreet and Michael, Julian and 1395 | Hill, Felix and Levy, Omer and Bowman, Samuel R.}, 1396 | journal = {arXiv preprint arXiv:1804.07461v1}, 1397 | year = {2018} 1398 | } 1399 | 1400 | @InProceedings{ N18-1100, 1401 | author = "Mullenbach, James and Wiegreffe, Sarah and Duke, Jon and 1402 | Sun, Jimeng and Eisenstein, Jacob", 1403 | title = {{Explainable Prediction of Medical Codes from Clinical 1404 | Text}}, 1405 | booktitle = "Proceedings of the 2018 Conference of the North American 1406 | Chapter of the Association for Computational Linguistics: 1407 | Human Language Technologies, Volume 1 (Long Papers)", 1408 | year = "2018", 1409 | publisher = "Association for Computational Linguistics", 1410 | pages = "1101--1111", 1411 | location = "New Orleans, Louisiana", 1412 | url = "http://aclweb.org/anthology/N18-1100" 1413 | } 1414 | 1415 | @Article{ senel2018semantic, 1416 | title = {{Semantic Structure and Interpretability of Word 1417 | Embeddings}}, 1418 | author = {Senel, Lutfi Kerem and Utlu, Ihsan and Yucesoy, Veysel and 1419 | Koc, Aykut and Cukur, Tolga}, 1420 | journal = {IEEE/ACM Transactions on Audio, Speech, and Language 1421 | Processing}, 1422 | year = {2018}, 1423 | publisher = {IEEE}, 1424 | doi = {10.1109/TASLP.2018.2837384} 1425 | } 1426 | 1427 | @InProceedings{ I17-1004, 1428 | author = "Ghader, Hamidreza and Monz, Christof", 1429 | title = {{What does Attention in Neural Machine Translation Pay 1430 | Attention to?}}, 1431 | booktitle = "Proceedings of the Eighth International Joint Conference 1432 | on Natural Language Processing (Volume 1: Long Papers)", 1433 | year = "2017", 1434 | publisher = "Asian Federation of Natural Language Processing", 1435 | pages = "30--39", 1436 | location = "Taipei, Taiwan", 1437 | url = "http://aclweb.org/anthology/I17-1004" 1438 | } 1439 | 1440 | @Article{ Q18-1019, 1441 | author = "Williams, Adina and Drozdov, Andrew and Bowman, Samuel R.", 1442 | title = "Do latent tree learning models identify meaningful 1443 | structure in sentences?", 1444 | journal = "Transactions of the Association for Computational 1445 | Linguistics", 1446 | year = "2018", 1447 | volume = "6", 1448 | pages = "253--267", 1449 | url = "http://aclweb.org/anthology/Q18-1019" 1450 | } 1451 | 1452 | @InProceedings{ P18-1241, 1453 | author = "Chen, Hongge and Zhang, Huan and Chen, Pin-Yu and Yi, 1454 | Jinfeng and Hsieh, Cho-Jui", 1455 | title = "Attacking Visual Language Grounding with Adversarial 1456 | Examples: A Case Study on Neural Image Captioning", 1457 | booktitle = "Proceedings of the 56th Annual Meeting of the Association 1458 | for Computational Linguistics (Volume 1: Long Papers)", 1459 | year = "2018", 1460 | publisher = "Association for Computational Linguistics", 1461 | pages = "2587--2597", 1462 | location = "Melbourne, Australia", 1463 | url = "http://aclweb.org/anthology/P18-1241" 1464 | } 1465 | 1466 | @Misc{ kuleshov2018adversarial, 1467 | title = {{Adversarial Examples for Natural Language Classification 1468 | Problems}}, 1469 | author = {Volodymyr Kuleshov and Shantanu Thakoor and Tingfung Lau 1470 | and Stefano Ermon}, 1471 | year = {2018}, 1472 | url = {https://openreview.net/forum?id=r1QZ3zbAZ} 1473 | } 1474 | 1475 | @Article{ cheng2018seq2sick, 1476 | title = {{Seq2Sick: Evaluating the Robustness of 1477 | Sequence-to-Sequence Models with Adversarial Examples}}, 1478 | author = {Cheng, Minhao and Yi, Jinfeng and Zhang, Huan and Chen, 1479 | Pin-Yu and Hsieh, Cho-Jui}, 1480 | journal = {arXiv preprint arXiv:1803.01128v1}, 1481 | year = {2018} 1482 | } 1483 | 1484 | @InProceedings{ C18-1055, 1485 | author = "Ebrahimi, Javid and Lowd, Daniel and Dou, Dejing", 1486 | title = {{On Adversarial Examples for Character-Level Neural 1487 | Machine Translation}}, 1488 | booktitle = "Proceedings of the 27th International Conference on 1489 | Computational Linguistics", 1490 | year = "2018", 1491 | publisher = "Association for Computational Linguistics", 1492 | pages = "653--663", 1493 | location = "Santa Fe, New Mexico, USA", 1494 | url = "http://aclweb.org/anthology/C18-1055" 1495 | } 1496 | 1497 | @Article{ vanmassenhoveinvestigating, 1498 | title = {{Investigating `Aspect' in NMT and SMT: Translating the 1499 | English Simple Past and Present Perfect}}, 1500 | journal = {Computational Linguistics in the Netherlands Journal}, 1501 | volume = {7}, 1502 | year = {2017}, 1503 | month = {12/2017}, 1504 | pages = {109-128}, 1505 | issn = {2211-4009}, 1506 | author = {Vanmassenhove, Eva and Du, Jinhua and Way, Andy} 1507 | } 1508 | 1509 | @InProceedings{ isahara1995jeida, 1510 | title = {{JEIDA's test-sets for quality evaluation of MT 1511 | systems-technical evaluation from the developer's point of 1512 | view}}, 1513 | author = {Isahara, Hitoshi}, 1514 | booktitle = {Proceedings of MT Summit V}, 1515 | year = {1995} 1516 | } 1517 | 1518 | @InProceedings{ koh2001test, 1519 | title = {{A test suite for evaluation of English-to-Korean machine 1520 | translation systems}}, 1521 | author = {Koh, Sungryong and Maeng, Jinee and Lee, Ji-Young and 1522 | Chae, Young-Sook and Choi, Key-Sun}, 1523 | booktitle = {MT Summit Conference}, 1524 | location = {Santiago de Compostela}, 1525 | year = {2001} 1526 | } 1527 | 1528 | @TechReport{ cooper1996using, 1529 | title = {Using the Framework}, 1530 | author = {Cooper, Robin and Crouch, Dick and van Eijck, Jan and Fox, 1531 | Chris and van Genabith, Josef and Jaspars, Jan and Kamp, 1532 | Hans and Milward, David and Pinkal, Manfred and Poesio, 1533 | Massimo and Pulman, Steve and Briscoe, Ted and Maier, 1534 | Holger and Konrad, Karsten}, 1535 | year = {1996}, 1536 | institution = {The FraCaS Consortium} 1537 | } 1538 | 1539 | @Article{ strobelt2018lstmvis, 1540 | title = {{LSTMVis: A Tool for Visual Analysis of Hidden State 1541 | Dynamics in Recurrent Neural Networks}}, 1542 | author = {Strobelt, Hendrik and Gehrmann, Sebastian and Pfister, 1543 | Hanspeter and Rush, Alexander M.}, 1544 | journal = {IEEE transactions on visualization and computer graphics}, 1545 | volume = {24}, 1546 | number = {1}, 1547 | pages = {667--676}, 1548 | year = {2018}, 1549 | url = {http://lstm.seas.harvard.edu}, 1550 | publisher = {IEEE} 1551 | } 1552 | 1553 | @Article{ rikters2018debugging, 1554 | title = {{Debugging Neural Machine Translations}}, 1555 | author = {Rikters, Mat{\=\i}ss}, 1556 | journal = {arXiv preprint arXiv:1808.02733v1}, 1557 | url = {http://attention.lielakeda.lv}, 1558 | year = {2018} 1559 | } 1560 | 1561 | @InProceedings{ rocktaschel2016reasoning, 1562 | title = {{Reasoning about Entailment with Neural Attention}}, 1563 | author = {Rockt{\"a}schel, Tim and Grefenstette, Edward and Hermann, 1564 | Karl Moritz and Ko{\v{c}}isk{\`y}, Tom{\'a}{\v{s}} and 1565 | Blunsom, Phil}, 1566 | booktitle = {International Conference on Learning Representations 1567 | (ICLR)}, 1568 | year = {2016} 1569 | } 1570 | 1571 | @Article{ Q16-1019, 1572 | author = "Yin, Wenpeng and Sch{\"u}tze, Hinrich and Xiang, Bing and 1573 | Zhou, Bowen", 1574 | title = {{ABCNN: Attention-Based Convolutional Neural Network for 1575 | Modeling Sentence Pairs}}, 1576 | journal = "Transactions of the Association for Computational 1577 | Linguistics", 1578 | year = "2016", 1579 | volume = "4", 1580 | pages = "259--272", 1581 | url = "http://aclweb.org/anthology/Q16-1019" 1582 | } 1583 | 1584 | @InProceedings{ D15-1044, 1585 | author = "Rush, Alexander M. and Chopra, Sumit and Weston, Jason", 1586 | title = {{A Neural Attention Model for Abstractive Sentence 1587 | Summarization}}, 1588 | booktitle = "Proceedings of the 2015 Conference on Empirical Methods in 1589 | Natural Language Processing ", 1590 | year = "2015", 1591 | publisher = "Association for Computational Linguistics", 1592 | pages = "379--389", 1593 | location = "Lisbon, Portugal", 1594 | doi = "10.18653/v1/D15-1044", 1595 | url = "http://www.aclweb.org/anthology/D15-1044" 1596 | } 1597 | 1598 | @InProceedings{ P17-1183, 1599 | author = "Aharoni, Roee and Goldberg, Yoav", 1600 | title = {{Morphological Inflection Generation with Hard Monotonic 1601 | Attention}}, 1602 | booktitle = "Proceedings of the 55th Annual Meeting of the Association 1603 | for Computational Linguistics (Volume 1: Long Papers) ", 1604 | year = "2017", 1605 | publisher = "Association for Computational Linguistics", 1606 | pages = "2004--2015", 1607 | location = "Vancouver, Canada", 1608 | doi = "10.18653/v1/P17-1183", 1609 | url = "http://www.aclweb.org/anthology/P17-1183" 1610 | } 1611 | 1612 | @InProceedings{ ghaeini2018interpreting, 1613 | author = "Ghaeini, Reza and Fern, Xiaoli and Tadepalli, Prasad", 1614 | title = {{Interpreting Recurrent and Attention-Based Neural Models: 1615 | A Case Study on Natural Language Inference}}, 1616 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 1617 | Natural Language Processing", 1618 | year = "2018", 1619 | publisher = "Association for Computational Linguistics", 1620 | pages = "4952--4957", 1621 | location = "Brussels, Belgium", 1622 | url = "http://aclweb.org/anthology/D18-1537" 1623 | } 1624 | 1625 | @InProceedings{ N18-2017, 1626 | author = "Gururangan, Suchin and Swayamdipta, Swabha and Levy, Omer 1627 | and Schwartz, Roy and Bowman, Samuel and Smith, Noah A.", 1628 | title = {{Annotation Artifacts in Natural Language Inference 1629 | Data}}, 1630 | booktitle = "Proceedings of the 2018 Conference of the North American 1631 | Chapter of the Association for Computational Linguistics: 1632 | Human Language Technologies, Volume 2 (Short Papers)", 1633 | year = "2018", 1634 | publisher = "Association for Computational Linguistics", 1635 | pages = "107--112", 1636 | location = "New Orleans, Louisiana", 1637 | url = "http://aclweb.org/anthology/N18-2017" 1638 | } 1639 | 1640 | @InProceedings{ S18-2023, 1641 | author = "Poliak, Adam and Naradowsky, Jason and Haldar, Aparajita 1642 | and Rudinger, Rachel and Van Durme, Benjamin", 1643 | title = {{Hypothesis Only Baselines in Natural Language 1644 | Inference}}, 1645 | booktitle = "Proceedings of the Seventh Joint Conference on Lexical and 1646 | Computational Semantics", 1647 | year = "2018", 1648 | publisher = "Association for Computational Linguistics", 1649 | pages = "180--191", 1650 | location = "New Orleans, Louisiana", 1651 | url = "http://aclweb.org/anthology/S18-2023" 1652 | } 1653 | 1654 | @Article{ yuan2017adversarial, 1655 | title = {{Adversarial Examples: Attacks and Defenses for Deep 1656 | Learning}}, 1657 | author = {Xiaoyong Yuan and Pan He and Qile Zhu and Xiaolin Li}, 1658 | journal = {arXiv preprint arXiv:1712.07107v3}, 1659 | year = {2017} 1660 | } 1661 | 1662 | @InProceedings{ lipton2016mythos, 1663 | title = {{The Mythos of Model Interpretability}}, 1664 | author = {Lipton, Zachary C.}, 1665 | booktitle = {ICML Workshop on Human Interpretability of Machine 1666 | Learning}, 1667 | year = {2016} 1668 | } 1669 | 1670 | @InProceedings{ ming2017understanding, 1671 | title = {{Understanding Hidden Memories of Recurrent Neural 1672 | Networks}}, 1673 | author = {Ming, Yao and Cao, Shaozu and Zhang, Ruixiang and Li, Zhen 1674 | and Chen, Yuanzhe and Song, Yangqiu and Qu, Huamin}, 1675 | booktitle = {IEEE Conference on Visual Analytics Science and Technology 1676 | (IEEE VAST 2017)}, 1677 | url = {http://www.myaooo.com/projects/rnnvis/}, 1678 | year = {2017} 1679 | } 1680 | 1681 | @InProceedings{ gupta2018lisa, 1682 | author = "Gupta, Pankaj and Sch{\"u}tze, Hinrich", 1683 | title = {{LISA: Explaining Recurrent Neural Network Judgments via 1684 | Layer-wIse Semantic Accumulation and Example to Pattern 1685 | Transformation}}, 1686 | booktitle = "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: 1687 | Analyzing and Interpreting Neural Networks for NLP", 1688 | year = "2018", 1689 | publisher = "Association for Computational Linguistics", 1690 | pages = "154--164", 1691 | location = "Brussels, Belgium", 1692 | url = "http://aclweb.org/anthology/W18-5418" 1693 | } 1694 | 1695 | @InProceedings{ wang18emnlptrdec, 1696 | title = {{A Tree-based Decoder for Neural Machine Translation}}, 1697 | author = {Xinyi Wang and Hieu Pham and Pengcheng Yin and Graham 1698 | Neubig}, 1699 | booktitle = {Conference on Empirical Methods in Natural Language 1700 | Processing (EMNLP)}, 1701 | address = {Brussels, Belgium}, 1702 | month = {November}, 1703 | url = {https://arxiv.org/abs/1808.09374}, 1704 | year = {2018} 1705 | } 1706 | 1707 | @InProceedings{ D15-1092, 1708 | author = "Chen, Xinchi and Qiu, Xipeng and Zhu, Chenxi and Wu, Shiyu 1709 | and Huang, Xuanjing", 1710 | title = {{Sentence Modeling with Gated Recursive Neural Network}}, 1711 | booktitle = "Proceedings of the 2015 Conference on Empirical Methods in 1712 | Natural Language Processing ", 1713 | year = "2015", 1714 | publisher = "Association for Computational Linguistics", 1715 | pages = "793--798", 1716 | location = "Lisbon, Portugal", 1717 | doi = "10.18653/v1/D15-1092", 1718 | url = "http://www.aclweb.org/anthology/D15-1092" 1719 | } 1720 | 1721 | @InProceedings{ giulianelli2018under, 1722 | author = "Giulianelli, Mario and Harding, Jack and Mohnert, Florian 1723 | and Hupkes, Dieuwke and Zuidema, Willem", 1724 | title = {{Under the Hood: Using Diagnostic Classifiers to 1725 | Investigate and Improve how Language Models Track Agreement 1726 | Information}}, 1727 | booktitle = "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: 1728 | Analyzing and Interpreting Neural Networks for NLP", 1729 | year = "2018", 1730 | publisher = "Association for Computational Linguistics", 1731 | pages = "240--248", 1732 | location = "Brussels, Belgium", 1733 | url = "http://aclweb.org/anthology/W18-5426" 1734 | } 1735 | 1736 | @InProceedings{ elloumi2018analyzing, 1737 | author = "Elloumi, Zied and Besacier, Laurent and Galibert, Olivier 1738 | and Lecouteux, Benjamin", 1739 | title = {{Analyzing Learned Representations of a Deep ASR 1740 | Performance Prediction Model}}, 1741 | booktitle = "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: 1742 | Analyzing and Interpreting Neural Networks for NLP", 1743 | year = "2018", 1744 | publisher = "Association for Computational Linguistics", 1745 | pages = "9--15", 1746 | location = "Brussels, Belgium", 1747 | url = "http://aclweb.org/anthology/W18-5402" 1748 | } 1749 | 1750 | @InCollection{ belinkov:2017:nips, 1751 | title = {{Analyzing Hidden Representations in End-to-End Automatic 1752 | Speech Recognition Systems}}, 1753 | author = {Belinkov, Yonatan and Glass, James}, 1754 | booktitle = {Advances in Neural Information Processing Systems 30}, 1755 | editor = {I. Guyon and U. V. Luxburg and S. Bengio and H. Wallach 1756 | and R. Fergus and S. Vishwanathan and R. Garnett}, 1757 | pages = {2441--2451}, 1758 | year = {2017}, 1759 | publisher = {Curran Associates, Inc.}, 1760 | url = {http://papers.nips.cc/paper/6838-analyzing-hidden-representations-in-end-to-end-automatic-speech-recognition-systems.pdf} 1761 | } 1762 | 1763 | @PhDThesis{ belinkov:2018:phdthesis, 1764 | title = {On Internal Language Representations in Deep Learning: An 1765 | Analysis of Machine Translation and Speech Recognition}, 1766 | author = {Yonatan Belinkov}, 1767 | school = {Massachusetts Institute of Technology}, 1768 | year = {2018}, 1769 | month = {May}, 1770 | pdf = {thesis2018.pdf} 1771 | } 1772 | 1773 | @InProceedings{ FengRAWR2018, 1774 | author = "Feng, Shi and Wallace, Eric and Grissom II, Alvin and 1775 | Iyyer, Mohit and Rodriguez, Pedro and Boyd-Graber, Jordan", 1776 | title = {{Pathologies of Neural Models Make Interpretations 1777 | Difficult}}, 1778 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 1779 | Natural Language Processing", 1780 | year = "2018", 1781 | publisher = "Association for Computational Linguistics", 1782 | pages = "3719--3728", 1783 | location = "Brussels, Belgium", 1784 | url = "http://aclweb.org/anthology/D18-1407" 1785 | } 1786 | 1787 | @InProceedings{ P15-1144, 1788 | author = "Faruqui, Manaal and Tsvetkov, Yulia and Yogatama, Dani and 1789 | Dyer, Chris and Smith, Noah A.", 1790 | title = {{Sparse Overcomplete Word Vector Representations}}, 1791 | booktitle = "Proceedings of the 53rd Annual Meeting of the Association 1792 | for Computational Linguistics and the 7th International 1793 | Joint Conference on Natural Language Processing (Volume 1: 1794 | Long Papers)", 1795 | year = "2015", 1796 | publisher = "Association for Computational Linguistics", 1797 | pages = "1491--1500", 1798 | location = "Beijing, China", 1799 | doi = "10.3115/v1/P15-1144", 1800 | url = "http://www.aclweb.org/anthology/P15-1144" 1801 | } 1802 | 1803 | @InProceedings{ D17-1041, 1804 | author = "Park, Sungjoon and Bak, JinYeong and Oh, Alice", 1805 | title = {{Rotated Word Vector Representations and their 1806 | Interpretability}}, 1807 | booktitle = "Proceedings of the 2017 Conference on Empirical Methods in 1808 | Natural Language Processing", 1809 | year = "2017", 1810 | publisher = "Association for Computational Linguistics", 1811 | pages = "401--411", 1812 | location = "Copenhagen, Denmark", 1813 | url = "http://aclweb.org/anthology/D17-1041" 1814 | } 1815 | 1816 | @InProceedings{ godin2018explaining, 1817 | author = "Godin, Fr{\'e}deric and Demuynck, Kris and Dambre, Joni 1818 | and De Neve, Wesley and Demeester, Thomas", 1819 | title = {{Explaining Character-Aware Neural Networks for Word-Level 1820 | Prediction: Do They Discover Linguistic Rules?}}, 1821 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 1822 | Natural Language Processing", 1823 | year = "2018", 1824 | publisher = "Association for Computational Linguistics", 1825 | pages = "3275--3284", 1826 | location = "Brussels, Belgium", 1827 | url = "http://aclweb.org/anthology/D18-1365" 1828 | } 1829 | 1830 | @InProceedings{ rozsa2016adversarial, 1831 | title = {{Adversarial Diversity and Hard Positive Generation}}, 1832 | author = {Rozsa, Andras and Rudd, Ethan M. and Boult, Terrance E.}, 1833 | booktitle = {Proceedings of the IEEE Conference on Computer Vision and 1834 | Pattern Recognition Workshops}, 1835 | pages = {25--32}, 1836 | year = {2016} 1837 | } 1838 | 1839 | @InProceedings{ pmlr-v80-zhao18b, 1840 | title = {{Adversarially Regularized Autoencoders}}, 1841 | author = {Zhao, Junbo and Kim, Yoon and Zhang, Kelly and Rush, 1842 | Alexander and LeCun, Yann}, 1843 | booktitle = {Proceedings of the 35th International Conference on 1844 | Machine Learning}, 1845 | pages = {5902--5911}, 1846 | year = {2018}, 1847 | editor = {Dy, Jennifer and Krause, Andreas}, 1848 | volume = {80}, 1849 | series = {Proceedings of Machine Learning Research}, 1850 | address = {Stockholmsm{\"a}ssan, Stockholm, Sweden}, 1851 | month = {10--15 Jul}, 1852 | publisher = {PMLR}, 1853 | pdf = {http://proceedings.mlr.press/v80/zhao18b/zhao18b.pdf}, 1854 | url = {http://proceedings.mlr.press/v80/zhao18b.html}, 1855 | abstract = {Deep latent variable models, trained using variational 1856 | autoencoders or generative adversarial networks, are now a 1857 | key technique for representation learning of continuous 1858 | structures. However, applying similar methods to discrete 1859 | structures, such as text sequences or discretized images, 1860 | has proven to be more challenging. In this work, we propose 1861 | a more flexible method for training deep latent variable 1862 | models of discrete structures. Our approach is based on the 1863 | recently proposed Wasserstein Autoencoder (WAE) which 1864 | formalizes adversarial autoencoders as an optimal transport 1865 | problem. We first extend this framework to model discrete 1866 | sequences, and then further explore different learned 1867 | priors targeting a controllable representation. Unlike many 1868 | other latent variable generative models for text, this 1869 | adversarially regularized autoencoder (ARAE) allows us to 1870 | generate fluent textual outputs as well as perform 1871 | manipulations in the latent space to induce change in the 1872 | output space. Finally we show that the latent 1873 | representation can be trained to perform unaligned textual 1874 | style transfer, giving improvements both in automatic 1875 | measures and human evaluation.} 1876 | } 1877 | 1878 | @InProceedings{ goodfellow2014generative, 1879 | title = {{Generative Adversarial Nets}}, 1880 | author = {Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi 1881 | and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and 1882 | Courville, Aaron and Bengio, Yoshua}, 1883 | booktitle = {Advances in neural information processing systems}, 1884 | pages = {2672--2680}, 1885 | year = {2014} 1886 | } 1887 | 1888 | @InCollection{ Rumelhart:1986:LPT:21935.42475, 1889 | author = {Rumelhart, D. E. and McClelland, J. L.}, 1890 | chapter = {{On Learning the Past Tenses of English Verbs}}, 1891 | title = {{Parallel Distributed Processing: Explorations in the 1892 | Microstructure of Cognition}}, 1893 | volume = {2}, 1894 | editor = {Rumelhart, David E. and McClelland, James L. and PDP 1895 | Research Group, CORPORATE}, 1896 | year = {1986}, 1897 | isbn = {0-262-13218-4}, 1898 | pages = {216--271}, 1899 | numpages = {56}, 1900 | url = {http://dl.acm.org/citation.cfm?id=21935.42475}, 1901 | acmid = {42475}, 1902 | publisher = {MIT Press}, 1903 | address = {Cambridge, MA, USA} 1904 | } 1905 | 1906 | @InProceedings{ Niu:2018, 1907 | author = "Niu, Tong and Bansal, Mohit", 1908 | title = {{Adversarial Over-Sensitivity and Over-Stability 1909 | Strategies for Dialogue Models}}, 1910 | booktitle = "Proceedings of the 22nd Conference on Computational 1911 | Natural Language Learning", 1912 | year = "2018", 1913 | publisher = "Association for Computational Linguistics", 1914 | pages = "486--496", 1915 | location = "Brussels, Belgium", 1916 | url = "http://aclweb.org/anthology/K18-1047" 1917 | } 1918 | 1919 | @Article{ doi:10.1080/09540099008915660, 1920 | author = { Catherine L. Harris }, 1921 | title = {{Connectionism and Cognitive Linguistics}}, 1922 | journal = {Connection Science}, 1923 | volume = {2}, 1924 | number = {1-2}, 1925 | pages = {7-33}, 1926 | year = {1990}, 1927 | publisher = {Taylor & Francis}, 1928 | doi = {10.1080/09540099008915660} 1929 | } 1930 | 1931 | @Article{ doi:10.1080/09540090010014070, 1932 | author = { Lars Niklasson and Fredrik Lin\r{a}ker }, 1933 | title = {Distributed representations for extended syntactic 1934 | transformation}, 1935 | journal = {Connection Science}, 1936 | volume = {12}, 1937 | number = {3-4}, 1938 | pages = {299-314}, 1939 | year = {2000}, 1940 | publisher = {Taylor & Francis}, 1941 | doi = {10.1080/09540090010014070}, 1942 | url = { https://doi.org/10.1080/09540090010014070 1943 | 1944 | }, 1945 | eprint = { https://doi.org/10.1080/09540090010014070 } 1946 | } 1947 | 1948 | @Article{ POLLACK199077, 1949 | title = "Recursive distributed representations", 1950 | journal = "Artificial Intelligence", 1951 | volume = "46", 1952 | number = "1", 1953 | pages = "77 - 105", 1954 | year = "1990", 1955 | issn = "0004-3702", 1956 | doi = "https://doi.org/10.1016/0004-3702(90)90005-K", 1957 | url = "http://www.sciencedirect.com/science/article/pii/000437029090005K", 1958 | author = "Jordan B. Pollack" 1959 | } 1960 | 1961 | @Article{ Miikkulainen:1991, 1962 | author = {Miikkulainen, Risto and Dyer, Michael G.}, 1963 | title = {{Natural Language Processing With Modular Pdp Networks and 1964 | Distributed Lexicon}}, 1965 | journal = {Cognitive Science}, 1966 | volume = {15}, 1967 | number = {3}, 1968 | pages = {343-399}, 1969 | doi = {10.1207/s15516709cog1503\_2}, 1970 | url = {https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1503_2}, 1971 | eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1207/s15516709cog1503_2}, 1972 | year = {1991} 1973 | } 1974 | 1975 | @Book{ Goodfellow-et-al-2016, 1976 | title = {Deep Learning}, 1977 | author = {Ian Goodfellow and Yoshua Bengio and Aaron Courville}, 1978 | publisher = {MIT Press}, 1979 | note = {\url{http://www.deeplearningbook.org}}, 1980 | year = {2016} 1981 | } 1982 | 1983 | @InProceedings{ sutskever2014sequence, 1984 | title = {{Sequence to Sequence Learning with Neural Networks}}, 1985 | author = {Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V.}, 1986 | booktitle = {Advances in neural information processing systems}, 1987 | pages = {3104--3112}, 1988 | year = {2014} 1989 | } 1990 | 1991 | @InProceedings{ mikolov2010recurrent, 1992 | title = {Recurrent neural network based language model}, 1993 | author = {Mikolov, Tom{\'a}{\v{s}} and Karafi{\'a}t, Martin and 1994 | Burget, Luk{\'a}{\v{s}} and {\v{C}}ernock{\`y}, Jan and 1995 | Khudanpur, Sanjeev}, 1996 | booktitle = {Eleventh Annual Conference of the International Speech 1997 | Communication Association}, 1998 | year = {2010} 1999 | } 2000 | 2001 | @Article{ jozefowicz2016exploring, 2002 | title = {{Exploring the Limits of Language Modeling}}, 2003 | author = {Jozefowicz, Rafal and Vinyals, Oriol and Schuster, Mike 2004 | and Shazeer, Noam and Wu, Yonghui}, 2005 | journal = {arXiv preprint arXiv:1602.02410v2}, 2006 | year = {2016} 2007 | } 2008 | 2009 | @Article{ Q16-1023, 2010 | author = "Kiperwasser, Eliyahu and Goldberg, Yoav", 2011 | title = {{Simple and Accurate Dependency Parsing Using 2012 | Bidirectional LSTM Feature Representations}}, 2013 | journal = "Transactions of the Association for Computational 2014 | Linguistics", 2015 | year = "2016", 2016 | volume = "4", 2017 | pages = "313--327", 2018 | url = "http://aclweb.org/anthology/Q16-1023" 2019 | } 2020 | 2021 | @Book{ goldberg2017neural, 2022 | title = {{Neural Network methods for Natural Language Processing}}, 2023 | author = {Goldberg, Yoav}, 2024 | series = {Synthesis Lectures on Human Language Technologies}, 2025 | volume = {10}, 2026 | number = {1}, 2027 | pages = {1--309}, 2028 | year = {2017}, 2029 | publisher = {Morgan \& Claypool Publishers} 2030 | } 2031 | 2032 | @Article{ doi:10.1080/10489223.2013.796950, 2033 | author = { Robert Frank and Donald Mathis and William Badecker }, 2034 | title = {{The Acquisition of Anaphora by Simple Recurrent 2035 | Networks}}, 2036 | journal = {Language Acquisition}, 2037 | volume = {20}, 2038 | number = {3}, 2039 | pages = {181-227}, 2040 | year = {2013}, 2041 | publisher = {Routledge}, 2042 | doi = {10.1080/10489223.2013.796950}, 2043 | url = {https://doi.org/10.1080/10489223.2013.796950 } 2044 | } 2045 | 2046 | @InProceedings{ N18-1205, 2047 | author = "Chen, Yining and Gilroy, Sorcha and Maletti, Andreas and 2048 | May, Jonathan and Knight, Kevin", 2049 | title = {{Recurrent Neural Networks as Weighted Language 2050 | Recognizers}}, 2051 | booktitle = "Proceedings of the 2018 Conference of the North American 2052 | Chapter of the Association for Computational Linguistics: 2053 | Human Language Technologies, Volume 1 (Long Papers)", 2054 | year = "2018", 2055 | publisher = "Association for Computational Linguistics", 2056 | pages = "2261--2271", 2057 | location = "New Orleans, Louisiana", 2058 | url = "http://aclweb.org/anthology/N18-1205" 2059 | } 2060 | 2061 | @Article{ gers2001lstm, 2062 | title = {{LSTM Recurrent Networks Learn Simple Context-Free and 2063 | Context-Sensitive Languages}}, 2064 | author = {Gers, Felix A. and Schmidhuber, J{\"u}rgen}, 2065 | journal = {IEEE Transactions on Neural Networks}, 2066 | volume = {12}, 2067 | number = {6}, 2068 | pages = {1333--1340}, 2069 | year = {2001}, 2070 | doi = {10.1109/72.963769}, 2071 | publisher = {IEEE} 2072 | } 2073 | 2074 | @InProceedings{ das1992learning, 2075 | title = {{Learning Context-free Grammars: Capabilities and 2076 | Limitations of a Recurrent Neural Network with an External 2077 | Stack Memory}}, 2078 | author = {Das, Sreerupa and Giles, C. Lee and Sun, Guo-Zheng}, 2079 | booktitle = {Proceedings of The Fourteenth Annual Conference of 2080 | Cognitive Science Society. Indiana University}, 2081 | pages = {14}, 2082 | year = {1992} 2083 | } 2084 | 2085 | @Article{ casey1996dynamics, 2086 | title = {{The Dynamics of Discrete-Time Computation, with 2087 | Application to Recurrent Neural Networks and Finite State 2088 | Machine Extraction}}, 2089 | author = {Casey, Mike}, 2090 | journal = {Neural computation}, 2091 | volume = {8}, 2092 | number = {6}, 2093 | pages = {1135--1178}, 2094 | year = {1996}, 2095 | doi = {10.1162/neco.1996.8.6.1135}, 2096 | publisher = {MIT Press} 2097 | } 2098 | 2099 | @Article{ bernardy2018can, 2100 | title = {{Can Recurrent Neural Networks Learn Nested Recursion?}}, 2101 | author = {Bernardy, Jean-Philippe}, 2102 | journal = {LiLT (Linguistic Issues in Language Technology)}, 2103 | volume = {16}, 2104 | number = {1}, 2105 | year = {2018} 2106 | } 2107 | 2108 | @Article{ boden2002learning, 2109 | title = {On learning context-free and context-sensitive languages}, 2110 | author = {Bod{\'e}n, Mikael and Wiles, Janet}, 2111 | journal = {IEEE Transactions on Neural Networks}, 2112 | volume = {13}, 2113 | number = {2}, 2114 | pages = {491--493}, 2115 | year = {2002}, 2116 | publisher = {IEEE} 2117 | } 2118 | 2119 | @Article{ chalup2003incremental, 2120 | title = {{Incremental Training of First Order Recurrent Neural 2121 | Networks to Predict a Context-sensitive Language}}, 2122 | author = {Chalup, Stephan K. and Blair, Alan D.}, 2123 | journal = {Neural Networks}, 2124 | volume = {16}, 2125 | number = {7}, 2126 | pages = {955--972}, 2127 | year = {2003}, 2128 | doi = {10.1016/S0893-6080(03)00054-6}, 2129 | publisher = {Elsevier} 2130 | } 2131 | 2132 | @InProceedings{ peters2018dissecting, 2133 | author = "Peters, Matthew and Neumann, Mark and Zettlemoyer, Luke 2134 | and Yih, Wen-tau", 2135 | title = {{Dissecting Contextual Word Embeddings: Architecture and 2136 | Representation}}, 2137 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 2138 | Natural Language Processing", 2139 | year = "2018", 2140 | publisher = "Association for Computational Linguistics", 2141 | pages = "1499--1509", 2142 | location = "Brussels, Belgium", 2143 | url = "http://aclweb.org/anthology/D18-1179" 2144 | } 2145 | 2146 | @InProceedings{ D15-1075, 2147 | author = "Bowman, Samuel R. and Angeli, Gabor and Potts, Christopher 2148 | and Manning, Christopher D.", 2149 | title = "A large annotated corpus for learning natural language 2150 | inference", 2151 | booktitle = "Proceedings of the 2015 Conference on Empirical Methods in 2152 | Natural Language Processing ", 2153 | year = "2015", 2154 | publisher = "Association for Computational Linguistics", 2155 | pages = "632--642", 2156 | location = "Lisbon, Portugal", 2157 | doi = "10.18653/v1/D15-1075", 2158 | url = "http://www.aclweb.org/anthology/D15-1075" 2159 | } 2160 | 2161 | @InProceedings{ belinkov:2017:acl, 2162 | author = "Belinkov, Yonatan and Durrani, Nadir and Dalvi, Fahim and 2163 | Sajjad, Hassan and Glass, James", 2164 | title = {{What do Neural Machine Translation Models Learn about 2165 | Morphology?}}, 2166 | booktitle = "Proceedings of the 55th Annual Meeting of the Association 2167 | for Computational Linguistics (Volume 1: Long Papers) ", 2168 | year = "2017", 2169 | publisher = "Association for Computational Linguistics", 2170 | pages = "861--872", 2171 | location = "Vancouver, Canada", 2172 | doi = "10.18653/v1/P17-1080", 2173 | url = "http://aclweb.org/anthology/P17-1080" 2174 | } 2175 | 2176 | @InProceedings{ belinkov:2017:ijcnlp, 2177 | author = "Belinkov, Yonatan and M{\`a}rquez, Llu{\'i}s and Sajjad, 2178 | Hassan and Durrani, Nadir and Dalvi, Fahim and Glass, 2179 | James", 2180 | title = {{Evaluating Layers of Representation in Neural Machine 2181 | Translation on Part-of-Speech and Semantic Tagging Tasks}}, 2182 | booktitle = "Proceedings of the Eighth International Joint Conference 2183 | on Natural Language Processing (Volume 1: Long Papers)", 2184 | year = "2017", 2185 | publisher = "Asian Federation of Natural Language Processing", 2186 | pages = "1--10", 2187 | location = "Taipei, Taiwan", 2188 | url = "http://aclweb.org/anthology/I17-1001" 2189 | } 2190 | 2191 | @InProceedings{ dalvi:2017:ijcnlp, 2192 | author = "Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and 2193 | Belinkov, Yonatan and Vogel, Stephan", 2194 | title = {{Understanding and Improving Morphological Learning in the 2195 | Neural Machine Translation Decoder}}, 2196 | booktitle = "Proceedings of the Eighth International Joint Conference 2197 | on Natural Language Processing (Volume 1: Long Papers)", 2198 | year = "2017", 2199 | publisher = "Asian Federation of Natural Language Processing", 2200 | pages = "142--151", 2201 | location = "Taipei, Taiwan", 2202 | url = "http://aclweb.org/anthology/I17-1015" 2203 | } 2204 | 2205 | @InProceedings{ D18-1313, 2206 | author = "Bisazza, Arianna and Tump, Clara", 2207 | title = {{The Lazy Encoder: A Fine-Grained Analysis of the Role of 2208 | Morphology in Neural Machine Translation}}, 2209 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 2210 | Natural Language Processing", 2211 | year = "2018", 2212 | publisher = "Association for Computational Linguistics", 2213 | pages = "2871--2876", 2214 | location = "Brussels, Belgium", 2215 | url = "http://aclweb.org/anthology/D18-1313" 2216 | } 2217 | 2218 | @InProceedings{ W18-5408, 2219 | author = "Jacovi, Alon and Sar Shalom, Oren and Goldberg, Yoav", 2220 | title = {{Understanding Convolutional Neural Networks for Text 2221 | Classification}}, 2222 | booktitle = "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: 2223 | Analyzing and Interpreting Neural Networks for NLP", 2224 | year = "2018", 2225 | publisher = "Association for Computational Linguistics", 2226 | pages = "56--65", 2227 | location = "Brussels, Belgium", 2228 | url = "http://aclweb.org/anthology/W18-5408" 2229 | } 2230 | 2231 | @InProceedings{ K18-1028, 2232 | author = "Artetxe, Mikel and Labaka, Gorka and Lopez-Gazpio, Inigo 2233 | and Agirre, Eneko", 2234 | title = {{Uncovering Divergent Linguistic Information in Word 2235 | Embeddings with Lessons for Intrinsic and Extrinsic 2236 | Evaluation}}, 2237 | booktitle = "Proceedings of the 22nd Conference on Computational 2238 | Natural Language Learning", 2239 | year = "2018", 2240 | publisher = "Association for Computational Linguistics", 2241 | pages = "282--291", 2242 | location = "Brussels, Belgium", 2243 | url = "http://aclweb.org/anthology/K18-1028" 2244 | } 2245 | 2246 | @InProceedings{ Faruqui:repeval:16, 2247 | author = {Faruqui, Manaal and Tsvetkov, Yulia and Rastogi, 2248 | Pushpendre and Dyer, Chris}, 2249 | title = {{Problems With Evaluation of Word Embeddings Using Word 2250 | Similarity Tasks}}, 2251 | booktitle = {Proc. of the 1st Workshop on Evaluating Vector Space 2252 | Representations for NLP}, 2253 | year = {2016}, 2254 | url = {http://arxiv.org/pdf/1605.02276v1.pdf} 2255 | } 2256 | 2257 | @Article{ J15-4004, 2258 | author = "Hill, Felix and Reichart, Roi and Korhonen, Anna", 2259 | title = {{SimLex-999: Evaluating Semantic Models With (Genuine) 2260 | Similarity Estimation}}, 2261 | journal = "Computational Linguistics", 2262 | year = "2015", 2263 | volume = "41", 2264 | number = "4", 2265 | pages = "665--695", 2266 | doi = "10.1162/COLI_a_00237", 2267 | url = "http://aclweb.org/anthology/J15-4004" 2268 | } 2269 | 2270 | @InProceedings{ P12-1015, 2271 | author = "Bruni, Elia and Boleda, Gemma and Baroni, Marco and Tran, 2272 | Nam Khanh", 2273 | title = {{Distributional Semantics in Technicolor}}, 2274 | booktitle = "Proceedings of the 50th Annual Meeting of the Association 2275 | for Computational Linguistics (Volume 1: Long Papers)", 2276 | year = "2012", 2277 | publisher = "Association for Computational Linguistics", 2278 | pages = "136--145", 2279 | location = "Jeju Island, Korea", 2280 | url = "http://aclweb.org/anthology/P12-1015" 2281 | } 2282 | 2283 | @Article{ finkelstein2002placing, 2284 | title = {{Placing Search in Context: The Concept Revisited}}, 2285 | author = {Finkelstein, Lev and Gabrilovich, Evgeniy and Matias, 2286 | Yossi and Rivlin, Ehud and Solan, Zach and Wolfman, Gadi 2287 | and Ruppin, Eytan}, 2288 | journal = {ACM Transactions on information systems}, 2289 | volume = {20}, 2290 | number = {1}, 2291 | doi = {10.1145/503104.503110}, 2292 | pages = {116--131}, 2293 | year = {2002} 2294 | } 2295 | 2296 | @InProceedings{ D16-1235, 2297 | author = "Gerz, Daniela and Vuli{\'{c}}, Ivan and Hill, Felix and 2298 | Reichart, Roi and Korhonen, Anna", 2299 | title = {{SimVerb-3500: A Large-Scale Evaluation Set of Verb 2300 | Similarity}}, 2301 | booktitle = "Proceedings of the 2016 Conference on Empirical Methods in 2302 | Natural Language Processing ", 2303 | year = "2016", 2304 | publisher = "Association for Computational Linguistics", 2305 | pages = "2173--2182", 2306 | location = "Austin, Texas", 2307 | doi = "10.18653/v1/D16-1235", 2308 | url = "http://aclweb.org/anthology/D16-1235" 2309 | } 2310 | 2311 | @InProceedings{ W13-3512, 2312 | author = "Luong, Thang and Socher, Richard and Manning, Christopher", 2313 | title = {{Better Word Representations with Recursive Neural 2314 | Networks for Morphology}}, 2315 | booktitle = "Proceedings of the Seventeenth Conference on Computational 2316 | Natural Language Learning", 2317 | year = "2013", 2318 | publisher = "Association for Computational Linguistics", 2319 | pages = "104--113", 2320 | location = "Sofia, Bulgaria", 2321 | url = "http://aclweb.org/anthology/W13-3512" 2322 | } 2323 | 2324 | @Article{ leviant2015separated, 2325 | title = {{Separated by an Un-common Language: Towards Judgment 2326 | Language Informed Vector Space Modeling}}, 2327 | author = {Leviant, Ira and Reichart, Roi}, 2328 | journal = {arXiv preprint arXiv:1508.00106v5}, 2329 | year = {2015} 2330 | } 2331 | 2332 | @InProceedings{ S17-2001, 2333 | author = "Cer, Daniel and Diab, Mona and Agirre, Eneko and 2334 | Lopez-Gazpio, Inigo and Specia, Lucia", 2335 | title = {{SemEval-2017 Task 1: Semantic Textual Similarity 2336 | Multilingual and Crosslingual Focused Evaluation}}, 2337 | booktitle = "Proceedings of the 11th International Workshop on Semantic 2338 | Evaluation (SemEval-2017) ", 2339 | year = "2017", 2340 | publisher = "Association for Computational Linguistics", 2341 | pages = "1--14", 2342 | location = "Vancouver, Canada", 2343 | doi = "10.18653/v1/S17-2001", 2344 | url = "http://aclweb.org/anthology/S17-2001" 2345 | } 2346 | 2347 | @InProceedings{ S14-2001, 2348 | author = "Marelli, Marco and Bentivogli, Luisa and Baroni, Marco and 2349 | Bernardi, Raffaella and Menini, Stefano and Zamparelli, 2350 | Roberto", 2351 | title = {{SemEval-2014 Task 1: Evaluation of Compositional 2352 | Distributional Semantic Models on Full Sentences through 2353 | Semantic Relatedness and Textual Entailment}}, 2354 | booktitle = "Proceedings of the 8th International Workshop on Semantic 2355 | Evaluation (SemEval 2014) ", 2356 | year = "2014", 2357 | publisher = "Association for Computational Linguistics", 2358 | pages = "1--8", 2359 | location = "Dublin, Ireland", 2360 | doi = "10.3115/v1/S14-2001", 2361 | url = "http://aclweb.org/anthology/S14-2001" 2362 | } 2363 | 2364 | @InProceedings{ suzgun:2019:SCiL, 2365 | title = {{On Evaluating the Generalization of LSTM Models in Formal 2366 | Languages}}, 2367 | author = {Suzgun, Mirac and Belinkov, Yonatan and Shieber, Stuart 2368 | M.}, 2369 | booktitle = {Proceedings of the Society for Computation in Linguistics 2370 | (SCiL)}, 2371 | year = {2019}, 2372 | month = {January} 2373 | } 2374 | 2375 | @InProceedings{ dalvi:2019:AAAI:demo, 2376 | title = {{NeuroX: A Toolkit for Analyzing Individual Neurons in 2377 | Neural Networks}}, 2378 | author = {Dalvi, Fahim and Nortonsmith, Avery and Bau, D. Anthony 2379 | and Belinkov, Yonatan and Sajjad, Hassan and Durrani, Nadir 2380 | and Glass, James}, 2381 | booktitle = {Proceedings of the Thirty-Third AAAI Conference on 2382 | Artificial Intelligence (AAAI): Demonstrations Track}, 2383 | year = {2019}, 2384 | month = {January} 2385 | } 2386 | 2387 | @InProceedings{ D18-1277, 2388 | author = "Elkahky, Ali and Webster, Kellie and Andor, Daniel and 2389 | Pitler, Emily", 2390 | title = {{A Challenge Set and Methods for Noun-Verb Ambiguity}}, 2391 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 2392 | Natural Language Processing", 2393 | year = "2018", 2394 | publisher = "Association for Computational Linguistics", 2395 | pages = "2562--2572", 2396 | location = "Brussels, Belgium", 2397 | url = "http://aclweb.org/anthology/D18-1277" 2398 | } 2399 | 2400 | @InProceedings{ W18-6304, 2401 | author = "Tang, Gongbo and Sennrich, Rico and Nivre, Joakim", 2402 | title = {{An Analysis of Attention Mechanisms: The Case of Word 2403 | Sense Disambiguation in Neural Machine Translation}}, 2404 | booktitle = "Proceedings of the Third Conference on Machine 2405 | Translation: Research Papers", 2406 | year = "2018", 2407 | publisher = "Association for Computational Linguistics", 2408 | pages = "26--35", 2409 | location = "Belgium, Brussels", 2410 | url = "http://aclweb.org/anthology/W18-6304" 2411 | } 2412 | 2413 | @InProceedings{ dalvi:2019:AAAI, 2414 | title = {{What Is One Grain of Sand in the Desert? Analyzing 2415 | Individual Neurons in Deep NLP Models}}, 2416 | author = {Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and 2417 | Belinkov, Yonatan and Bau, D. Anthony and Glass, James}, 2418 | booktitle = {Proceedings of the Thirty-Third AAAI Conference on 2419 | Artificial Intelligence (AAAI)}, 2420 | year = {2019}, 2421 | month = {January} 2422 | } 2423 | 2424 | @InProceedings{ D18-1007, 2425 | author = "Poliak, Adam and Haldar, Aparajita and Rudinger, Rachel 2426 | and Hu, J. Edward and Pavlick, Ellie and White, Aaron 2427 | Steven and Van Durme, Benjamin", 2428 | title = {{Collecting Diverse Natural Language Inference Problems 2429 | for Sentence Representation Evaluation}}, 2430 | booktitle = "Proceedings of the 2018 Conference on Empirical Methods in 2431 | Natural Language Processing", 2432 | year = "2018", 2433 | publisher = "Association for Computational Linguistics", 2434 | pages = "67--81", 2435 | location = "Brussels, Belgium", 2436 | url = "http://aclweb.org/anthology/D18-1007" 2437 | } 2438 | 2439 | @Article{ bau2018identifying, 2440 | title = {{Identifying and Controlling Important Neurons in Neural 2441 | Machine Translation}}, 2442 | author = {Bau, Anthony and Belinkov, Yonatan and Sajjad, Hassan and 2443 | Durrani, Nadir and Dalvi, Fahim and Glass, James}, 2444 | journal = {arXiv preprint arXiv:1811.01157v1}, 2445 | year = {2018} 2446 | } 2447 | 2448 | @inproceedings{lalor2016building, 2449 | title={Building an Evaluation Scale using Item Response Theory}, 2450 | author={Lalor, John and Wu, Hao and others}, 2451 | booktitle={Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}, 2452 | pages={648--657}, 2453 | year={2016} 2454 | } 2455 | 2456 | @inproceedings{lalor2018understanding, 2457 | title={Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study}, 2458 | author={Lalor, John and Wu, Hao and Munkhdalai, Tsendsuren and Yu, Hong}, 2459 | booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, 2460 | pages={4711--4716}, 2461 | year={2018} 2462 | } 2463 | 2464 | @inproceedings{Anastasopoulos2019, 2465 | title={Neural Machine Translation of Text from Non-Native Speakers}, 2466 | author={Anastasopoulos, Antonios and Lui, Alison and Nguyen, Toan and Chiang, David}, 2467 | booktitle={Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019)}, 2468 | year={2019} 2469 | } 2470 | 2471 | @inproceedings{blohm-etal-2018-comparing, 2472 | title = {Comparing Attention-Based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension}, 2473 | author = {Blohm, Matthias and Jagfeld, Glorianna and Sood, Ekta and Yu, Xiang and Vu, Ngoc Thang}, 2474 | booktitle = {Proceedings of the 22nd Conference on Computational Natural Language Learning}, 2475 | year = {2018}, 2476 | } 2477 | 2478 | @InProceedings{stojanovski-fraser:2018:WMT, 2479 | author = {Stojanovski, Dario and Fraser, Alexander}, 2480 | title = {Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments}, 2481 | booktitle = {Proceedings of the Third Conference on Machine Translation, Volume 1: Research Papers}, 2482 | month = {October}, 2483 | year = {2018}, 2484 | address = {Brussels, Belgium}, 2485 | publisher = {Association for Computational Linguistics}, 2486 | pages = {49--60}, 2487 | url = {http://www.aclweb.org/anthology/W18-6306} 2488 | } 2489 | 2490 | @InProceedings{mller-EtAl:2018:WMT, 2491 | author = {Müller, Mathias and Rios, Annette and Voita, Elena and Sennrich, Rico}, 2492 | title = {A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation}, 2493 | booktitle = {Proceedings of the Third Conference on Machine Translation, Volume 1: Research Papers}, 2494 | month = {October}, 2495 | year = {2018}, 2496 | address = {Belgium, Brussels}, 2497 | publisher = {Association for Computational Linguistics}, 2498 | pages = {61--72}, 2499 | url = {http://www.aclweb.org/anthology/W18-6307} 2500 | } 2501 | -------------------------------------------------------------------------------- /references.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
10 |
11 | , , . 12 | 13 | . 14 | 15 | 16 | (view online) 17 | 18 |
19 |
20 | 21 |
22 |
23 | 24 |
25 | 26 | 27 | -------------------------------------------------------------------------------- /table1.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 |
13 | , 14 | 15 |
16 |
17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 |
ReferenceComponentPropertyMethod
Word embeddingsPOS, head POS, dependency relation, gender, case, number, tense Classification
Word embeddingsPOS, dependency relations, morphological features, emotionsClassification
Word embeddingsReferential attributesClassification
RNN states/gatesPOS, syntactic role, gender, case, definiteness, verb form, moodClassification, correlation
RNN statesPOS, top syntactic sequence, smallest constituent, tense, voice Classification
RNN states/gatesAcoustic featuresCorrelation
RNN gatesPhoneme boundariesChange in activation signal
RNN statesPOS, ancestor label prediction, dependency relation predictionClassification
Sentence embeddingsSentence length, word presence, word orderClassification
Sentence embeddingsSentence length, word presence, word order; orthography; social tasksClassification
Sentence embeddingsSemantic role, word presenceClassification
Sentence embeddingsSynthetic syntactic patternsClustering
Sentence embeddingsSentence length, word presence, word order; tree depth, top constituent; main tense, subject/object number, semantic odd man out, coordinate inversionClassification
Sentence embeddingsSentence length, word presence, word order; POS, word sense disambiguation; sentence order Classification
Attention weightsPOS, word alignmentDistribution measures, match with alignments
Attention weightsAnaphoraAttention score
Attention weightsWord sense disambiguationDistribution measures
Hidden activations in feed-forward acoustic modelPhonemes, phonetic features, genderClustering, average activations by group/label
Hidden activations in feed-forward acoustic modelPhonemes, phonetic featuresClassification, clustering measures
CNN/RNN activationsPhonetic unitsClassification
Speaker embeddingsSpeaker, speech content, word order, utterance length, channel, gender, speaking rate Classification
CNN activationsStyle, accent, broadcast programClassification
Audio-visual RNN activationsUtterance length, word presence, homonym disambiguationClassification, regression, similarity measures
Audio-visual CNN embeddingsWord classesClustering
RNN states in audio-visual modelPhonemes, synonymsClassification, clustering, discrimination
Audio-visual CNN activationsPhonemes, speakers, word identityClustering, discrimination
Word embeddings, vision CNN ConceptsSimilarity measures
RNN statesWord presenceDirect classification
NMT encoder neuronsSentence lengthRegression
NMT word embeddingssynonyms, morphological featuresNearest neighbors
NMT statesPOS, morphologyClassification
NMT statesPOS, lexical semanticsClassification
NMT statesMorphologyClassification
NMT and LM neuronsPOS, morphology, lexical semanticsClassification
RNN statesSubject-verb agreementLikelihood comparison, direct classification
RNN / self-attention statesSubject-verb agreementLikelihood comparison, direct classification
RNN statesNumber agreementLikelihood comparison
Parser word embeddingsWord features (shape, etc.)Classification; also other methods
RNN sentence embeddingHierarchical structureClassification
biLM representations (RNN, Transformer, gated CNN) POS, constituency parsing, coreferenceClassification; similarity scores
RNN states in language-vision modelWord boundary, word similarityClassification
Hidden activations in feed-forward audio-visual model Phonetic featuresDiscrimination
280 | 289 | 290 | 291 | 292 | -------------------------------------------------------------------------------- /table2.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 |
13 | , 14 | 15 |
16 |
17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 |
ReferenceTaskPhenomenaLanguageSizeConstructionComments
NLIAntonyms, quantities, spelling, word overlap, negation, lengthEnglish7596Automatic
NLICompositionalityEnglish44010Automatic
NLIAntonyms, hyper/hyponymsEnglish6279Semi-automatic
NLIDiverse semanticsEnglish550Manual
NLILexical inferenceEnglish8193Semi-automatic
NLIDiverseEnglish570KManual, semi-automatic, automatic
MTWord sense disambiguationGerman→English/French13900Semi-automatic
MTMorphologyEnglish→Czech/Latvian18500Automatic
MTPolarity, verb-particle constructions, agreement, transliterationEnglish→German97000Automatic
MTDiscourseEnglish→French400Manual
MTMorpho-syntax, syntax, lexiconEnglish↔French108+506Manual
MTDiverseEnglish↔German10000Manual
MTDiscourseEnglish→German4627AutomaticTest sets created using oracles, an alternative to challenge sets. The method can be applied to different language pairs and datasets.
MTCoreference, pronounsEnglish→German12000Automatic
LMSubject-verb agreementEnglish∼1.35MAutomatic
LMNumber agreementEnglish, Russian, Hebrew, Italian∼10KAutomatic
CoreferenceGender biasEnglish720Semi-automatic
CoreferenceGender biasEnglish3160Semi-automatic
Seq2SeqCompositionalityEnglish20910Automatic
POS taggingNoun-verb ambiguityEnglish32654Semi-automatic
NLIPsychometric assessmentEnglish180Manual
SentimentPsychometric assessmentEnglish134Manual
229 | 238 | 239 | 240 | 241 | -------------------------------------------------------------------------------- /table3.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 |
13 | , 14 | 15 |
16 |
17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 |
MethodKnowledgeTargetedUnitTask
Black✗CharacterMT
Black✗CharacterMT, morphology
Black✗CharacterSpelling correction
Black✓, ✗WordMT, NLI
Black✗CharacterText classification, sentiment
Black✗SentenceReading comprehension
Black✗SyntaxSentiment, entailment
Black✗WordImage captioning
Black✗WordNLI, sentiment
Black✗WordText classification, sentiment
Black✗WordReading comprehension, visual QA, sentiment
Black✗WordDialogue
White✓PixelsImage captioning
White✓WordMT
White✓WordMT, summarization
White✗WordReading comprehension, visual and table QA
White✗WordSentiment
White✗WordSentiment, gender detection
White✗WordText classification, sentiment, grammatical error detection
White✓Word/CharacterText classification
White✗Word/CharacterText classification
White✗Word/CharacterText classification
Black✗CharacterMT
Black/White✗Sentence/WordReading comprehension
197 | 206 | 207 | 208 | 209 | --------------------------------------------------------------------------------