ā””ā”€ā”€ README.md /README.md: -------------------------------------------------------------------------------- 1 | Awesome Deep Learning for Natural Language Processing (NLP) [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome) 2 | ==== 3 | 4 | Table of Contents 5 | ---- 6 | 7 | - __[Courses](#courses)__ 8 | - __[Books](#books)__ 9 | - __[Tutorials](#tutorials)__ 10 | - __[Talks / Lectures](#talks)__ 11 | - __[Frameworks / Models](#frameworks)__ 12 | - __[Papers](#papers)__ 13 | - __[Blog Posts](#blog-posts)__ 14 | - __[Datasets](#datasets)__ 15 | - __[Word Embeddings / Word Vectors](#word-embeddings)__ 16 | - __[Contributing](#contributing)__ 17 | 18 | Courses 19 | ---- 20 | 1. NLP with Deep Learning / CS224N from Stanford (Winter 2019) 21 | - [Course homepage](http://web.stanford.edu/class/cs224n/index.html) A complete survey of the field with videos, lecture slides, and sample student projects. 22 | - [Course lectures](https://www.youtube.com/watch?v=8rXD5-xhemo&list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z) Video playlist. 23 | - [Previous course notes](https://github.com/stanfordnlp/cs224n-winter17-notes) Probably the best "book" on DL for NLP. 24 | - [Course code](https://github.com/DSKSD/DeepNLP-models-Pytorch) Pytorch implementations of various Deep NLP models in cs-224n. 25 | 1. Neural Networks for NLP from Carnegie Mellon University 26 | - [Course homepage](http://phontron.com/class/nn4nlp2017/) 27 | - [Course lectures](https://www.youtube.com/user/neubig/videos) 28 | - [Course code](https://github.com/neubig/nn4nlp2017-code/) 29 | 1. Deep Learning for Natural Language Processing from University of Oxford and DeepMind 30 | - [Course homepage](https://www.cs.ox.ac.uk/teaching/courses/2016-2017/dl/) 31 | - [Course slides](https://github.com/oxford-cs-deepnlp-2017/lectures) 32 | - [Course lectures](https://www.youtube.com/playlist?list=PL613dYIGMXoZBtZhbyiBqb0QtgK6oJbpm) 33 | 34 | Books 35 | ----- 36 | 1. [Deep Learning with Text: Natural Language Processing (Almost) from Scratch with Python and spaCy](https://www.amazon.com/Deep-Learning-Text-Approach-Processing/dp/1491984414) by Patrick Harrison and Matthew Honnibal 37 | 1. [Neural Network Methods in Natural Language Processing](https://www.amazon.com/gp/product/1627052984) by Yoav Goldberg and Graeme Hirst 38 | 1. [Deep Learning in Natural Language Processing](http://www.springer.com/us/book/9789811052088) by Li Deng and Yang Liu 39 | 1. [Natural Language Processing in Action](https://www.manning.com/books/natural-language-processing-in-action) by Hobson Lane, Cole Howard, and Hannes Hapke 40 | 1. Deep Learning: Natural Language Processing in Python by The LazyProgrammer (Kindle only) 41 | 1. [Word2Vec and Word Embeddings in Python and Theano](https://www.amazon.com/Deep-Learning-Language-Processing-Embeddings-ebook/dp/B01KQ0ZN0A) 42 | 1. [From Word2Vec to GLoVe in Python and Theano](https://www.amazon.com/Deep-Learning-Language-Processing-Word2Vec-ebook/dp/B01KRBOO4Y/) 43 | 1. [Recursive Neural Networks: Recursive Neural (Tensor) Networks in Theano](https://www.amazon.com/Deep-Learning-Language-Processing-Recursive-ebook/dp/B01KS5AEXO) 44 | 1. [Applied Natural Language Processing with Python](https://www.amazon.ca/Applied-Natural-Language-Processing-Python/dp/1484237323) by Taweh Beysolow II 45 | 1. [Deep Learning Cookbook](https://www.amazon.ca/Deep-Learning-Cookbook-Practical-Recipes/dp/149199584X) by Douwe Osinga 46 | 1. [Deep Learning for Natural Language Processing: Creating Neural Networks with Python](https://www.amazon.ca/Deep-Learning-Natural-Language-Processing/dp/148423684X) by Palash Goyal, Sumit Pandey, Karan Jain 47 | 1. [Machine Learning for Text](https://www.amazon.ca/Machine-Learning-Text-Charu-Aggarwal/dp/3319735306) by Charu C. Aggarwal 48 | 1. [Natural Language Processing with TensorFlow](https://www.amazon.ca/Natural-Language-Processing-TensorFlow-language-ebook/dp/B077Q3VZFR) by Thushan Ganegedara 49 | 1. [fastText Quick Start Guide: Get started with Facebook's library for text representation and classification](https://www.amazon.ca/fastText-Quick-Start-Guide-representation/dp/1789130999) 50 | 1. [Hands-On Natural Language Processing with Python](https://www.amazon.ca/Hands-Natural-Language-Processing-Python/dp/178913949X) 51 | 1. [Natural Language Processing in Action, Seond Edition](https://www.manning.com/books/natural-language-processing-in-action-second-edition) by Hobson Lane and Maria Dyshel 52 | 1. [Getting Started with Natural Language Processing in Action](https://www.manning.com/books/getting-started-with-natural-language-processing) by Ekaterina Kochmar 53 | 2. [Deep Learning for Natural Language Processing in Action](https://www.manning.com/books/deep-learning-for-natural-language-processing) by Stephan Raaijmakers 54 | 55 | Tutorials 56 | ----- 57 | 58 | 1. [Text classification guide from Google](https://developers.google.com/machine-learning/guides/text-classification/) 59 | 1. [Deep Learning for NLP with PyTorch](https://pytorch.org/tutorials/beginner/deep_learning_nlp_tutorial.html) 60 | 61 | Talks 62 | ---- 63 | 1. [Deep Learning for Natural Language Processing (without Magic)](http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial) 64 | 1. [A Primer on Neural Network Models for Natural Language Processing](https://arxiv.org/abs/1510.00726) 65 | 1. [Deep Learning for Natural Language Processing: Theory and Practice (Tutorial)](https://www.microsoft.com/en-us/research/publication/deep-learning-for-natural-language-processing-theory-and-practice-tutorial/) 66 | 1. [TensorFlow Tutorials](https://www.tensorflow.org/tutorials/mandelbrot) 67 | 1. [Practical Neural Networks for NLP](https://github.com/clab/dynet_tutorial_examples) from EMNLP 2016 using DyNet framework 68 | 1. [Recurrent Neural Networks with Word Embeddings](http://deeplearning.net/tutorial/rnnslu.html) 69 | 1. [LSTM Networks for Sentiment Analysis](http://deeplearning.net/tutorial/lstm.html) 70 | 1. [TensorFlow demo using the Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/) 71 | 1. [LSTMVis: Visual Analysis for Recurrent Neural Networks](http://lstm.seas.harvard.edu/client/index.html) 72 | 1. Using deep learning in natural language processing by Rob Romijnders from PyData Amsterdam 2017 73 | - [video](https://www.youtube.com/watch?v=HVdPWoZ_swY) 74 | - [slides](https://github.com/RobRomijnders/talks/blob/master/pydata_DL_NLP.pdf) 75 | 1. [Richard Socher's talk on sentiment analysis, question answering, and sentence-image embeddings](https://www.youtube.com/watch?v=tdLmf8t4oqM) 76 | 1. [Deep Learning, an interactive introduction for NLP-ers](http://www.slideshare.net/roelofp/220115dlmeetup) 77 | 1. [Deep Natural Language Understanding](http://videolectures.net/deeplearning2016_cho_language_understanding/) 78 | 1. [Deep Learning Summer School, Montreal 2016](http://videolectures.net/deeplearning2016_montreal/) Includes state-of-art language modeling. 79 | 1. Tackling the Limits of Deep Learning for NLP by Richard Socher 80 | - [video](https://www.youtube.com/watch?v=JYwNmSe4HqE) 81 | - [slides](https://berkeley-deep-learning.github.io/cs294-131-s17/slides/socher-talk.pdf) 82 | 83 | Frameworks 84 | ---- 85 | 1. [Overview of DL frameworks for NLP](https://medium.com/@datamonsters/13-deep-learning-frameworks-for-natural-language-processing-in-python-2b84a6b6cd98) 86 | 1. General Frameworks 87 | 1. [Keras](https://keras.io/) - _The Python Deep Learning library_ Emphasis on user friendliness, modularity, easy extensibility, and Pythonic. 88 | 1. [TensorFlow](https://www.tensorflow.org/) - A cross-platform, general purpose Machine Intelligence library with Python and C++ API. 89 | 1. [PyTorch](http://pytorch.org/) - PyTorch is a deep learning framework that puts Python first. "Tensors and Dynamic neural networks in Python with strong GPU acceleration." 90 | 91 | 1. Specific Frameworks 92 | 1. [SpaCy](https://spacy.io/) - A Python package designed for speed, getting things dones, and interoperates with other Deep Learning frameworks 93 | 1. [Genism: Topic modeling for humans](https://pypi.python.org/pypi/gensim) - A Python package that includes word2vec and doc2vec implementations. 94 | 1. [fasttext](https://github.com/facebookresearch/fastText) Facebook's library for fast text representation and classification. 95 | 1. Built on TensorFlow 96 | 1. [SyntaxNet](https://github.com/tensorflow/models/tree/master/research/syntaxnet) - A toolkit for natural language understanding (NLU). 97 | 1. [textsum](https://github.com/tensorflow/models/tree/master/research/textsum) - A Sequence-to-Sequence with Attention Model for Text Summarization. 98 | 1. [Skip-Thought Vectors](https://github.com/tensorflow/models/tree/master/research/skip_thoughts) implementation in TensorFlow. 99 | 1. [ActiveQA: Active Question Answering](https://github.com/google/active-qa) - Using reinforcement learning to train artificial agents for question answering 100 | 1. [BERT](https://github.com/google-research/bert) - Bidirectional Encoder Representations from Transformers for pre-trained models 101 | 1. Built on PyTorch 102 | 1. [PyText](https://github.com/facebookresearch/pytext) - A deep-learning based NLP modeling framework by Facebook 103 | 1. [AllenNLP](https://allennlp.org/) - An open-source NLP research library 104 | 1. [Flair](https://github.com/zalandoresearch/flair) - A very simple framework for state-of-the-art NLP 105 | 1. [fairseq](https://github.com/pytorch/fairseq) - A Sequence-to-Sequence Toolkit 106 | 1. [fastai](http://docs.fast.ai/text.html) - Simplifies training fast and accurate neural nets using modern best practices 107 | 1. [Transformer model](http://nlp.seas.harvard.edu/2018/04/03/attention.html) - Annotated notebook implementation 108 | 1. [Deeplearning4jā€™s NLP framework](http://deeplearning4j.org/nlp) - Java implementation. 109 | 1. [DyNet](https://github.com/clab/dynet) - _The Dynamic Neural Network Toolkit_ "work well with networks that have dynamic structures that change for every training instance". 110 | 1. [deepnl](https://github.com/attardi/deepnl) - A Python library for NLP based on Deep Learning neural network architecture. 111 | 112 | Papers 113 | ---- 114 | 1. [Deep or shallow, NLP is breaking out](http://dl.acm.org/citation.cfm?id=2874915) - General overview of how Deep Learning is impacting NLP. 115 | 1. [Natural Language Processing from Research at Google](http://research.google.com/pubs/NaturalLanguageProcessing.html) - Not all Deep Learning (but mostly). 116 | 1. [Context Dependent Recurrent Neural Network Language Model](http://www.msr-waypoint.com/pubs/176926/rnn_ctxt.pdf) 117 | 1. [Translation Modeling with Bidirectional Recurrent Neural Networks](https://www-i6.informatik.rwth-aachen.de/publications/download/936/SundermeyerMartinAlkhouliTamerWuebkerJoernNeyHermann--TranslationModelingwithBidirectionalRecurrentNeuralNetworks--2014.pdf) 118 | 1. [Contextual LSTM (CLSTM) models for Large scale NLP tasks](https://arxiv.org/abs/1602.06291) 119 | 1. [LSTM Neural Networks for Language Modeling](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.248.4448&rep=rep1&type=pdf) 120 | 1. [Exploring the Limits of Language Modeling](http://arxiv.org/pdf/1602.02410.pdf) 121 | 1. [Conversational Contextual Cues](https://arxiv.org/abs/1606.00372) - Models context and participants in conversations. 122 | 1. [Sequence to sequence learning with neural networks](http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf) 123 | 1. [Efficient Estimation of Word Representations in Vector Space](http://arxiv.org/pdf/1301.3781.pdf) 124 | 1. [Learning Character-level Representations for Part-of-Speech Tagging](http://jmlr.org/proceedings/papers/v32/santos14.pdf) 125 | 1. [Representation Learning for Text-level Discourse Parsing](http://www.cc.gatech.edu/~jeisenst/papers/ji-acl-2014.pdf) 126 | 1. [Fast and Robust Neural Network Joint Models for Statistical Machine Translation](http://acl2014.org/acl2014/P14-1/pdf/P14-1129.pdf) 127 | 1. [Parsing With Compositional Vector Grammars](http://www.socher.org/index.php/Main/ParsingWithCompositionalVectorGrammars) 128 | 1. [Smart Reply: Automated Response Suggestion for Email](https://arxiv.org/abs/1606.04870) 129 | 1. [Neural Architectures for Named Entity Recognition](https://arxiv.org/abs/1603.01360) - State-of-the-art performance in NER with bidirectional LSTM with a sequential conditional random layer and transition-based parsing with stack LSTMs. 130 | 1. [Grammar as a Foreign Language](https://arxiv.org/abs/1412.7449) - State-of-the-art syntactic constituency parsing using generic sequence-to-sequence approach. 131 | 132 | Blog Posts 133 | ---- 134 | 135 | 1. [Natural Language Processing (NLP) progress](https://nlpprogress.com/) Tracking the most common NLP tasks, including the datasets and the current state-of-the-art 136 | 1. [A Review of the Recent History of Natural Language Processing](http://blog.aylien.com/a-review-of-the-recent-history-of-natural-language-processing/) 137 | 1. [Deep Learning, NLP, and Representations](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/) 138 | 1. [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) 139 | 1. [Neural Language Modeling From Scratch](http://ofir.io/Neural-Language-Modeling-From-Scratch/?a=1) 140 | 1. [Machine Learning for Emoji Trends](http://instagram-engineering.tumblr.com/post/117889701472/emojineering-part-1-machine-learning-for-emoji) 141 | 1. [Teaching Robots to Feel: Emoji & Deep Learning](http://getdango.com/emoji-and-deep-learning.html) 142 | 1. [Computational Linguistics and Deep Learning](http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239) - Opinion piece on how Deep Learning fits into the broader picture of text processing. 143 | 1. [Deep Learning NLP Best Practices](http://ruder.io/deep-learning-nlp-best-practices/index.html) 144 | 1. [7 types of Artificial Neural Networks for Natural Language Processing](https://medium.com/@datamonsters/artificial-neural-networks-for-natural-language-processing-part-1-64ca9ebfa3b2) 145 | 1. [How to solve 90% of NLP problems: a step-by-step guide](https://blog.insightdatascience.com/how-to-solve-90-of-nlp-problems-a-step-by-step-guide-fda605278e4e) 146 | 2. [7 Applications of Deep Learning for Natural Language Processing](https://machinelearningmastery.com/applications-of-deep-learning-for-natural-language-processing/) 147 | 148 | Datasets 149 | ---- 150 | 1. [Dataset from "One Billion Word Language Modeling Benchmark"](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) - Almost 1B words, already pre-processed text. 151 | 1. [Stanford Sentiment Treebank](https://nlp.stanford.edu/sentiment/treebank.html) - Fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences. 152 | 1. [Chatbot data from Kaggle](https://www.kaggle.com/samdeeplearning/deepnlp) 153 | 1. [A list of text datasets that are free/public domain in alphabetical order](https://github.com/niderhoff/nlp-datasets) 154 | 1. [Another list of text datasets that are free/public domain in reverse chronological order](https://github.com/karthikncode/nlp-datasets) 155 | 1. Question Answering datasets 156 | 1. [Quora's Question Pairs Dataset](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs) - Identify question pairs that have the same intent. 157 | 1. [CMU's Wikipedia Factoid Question Answers](https://www.cs.cmu.edu/~ark/QA-data/) 158 | 1. [DeepMind's Algebra Question Answering](https://github.com/deepmind/AQuA) 159 | 1. [DeepMind's from CNN & DailyMail Question Answering](https://github.com/deepmind/rc-data) 160 | 1. [Microsoft's WikiQA Open Domain Question Answering](https://www.microsoft.com/en-us/research/publication/wikiqa-a-challenge-dataset-for-open-domain-question-answering/) 161 | 1. [Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer/) - covering reading comprehension 162 | 163 | Word Embeddings and friends 164 | ---- 165 | 1. [The amazing power of word vectors](https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/) from The Morning Paper blog 166 | 1. [Distributed Representations of Words and Phrases and their Compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) - The original word2vec paper. 167 | 1. [word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738) An elucidating explanation of word2vec training 168 | 1. [Word embeddings in 2017: Trends and future directions](http://ruder.io/word-embeddings-2017/) 169 | 1. [Learning Word Vectors for 157 Languages](https://arxiv.org/abs/1802.06893) 170 | 1. [GloVe: Global Vectors for Word Representation](http://www-nlp.stanford.edu/pubs/glove.pdf) - A "count-based"/co-occurrence model to learn word embeddings. 171 | 1. Doc2Vec 172 | - [A gentle introduction to Doc2Vec](https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e) 173 | - [Distributed Representations of Sentences and Documents](https://cs.stanford.edu/~quocle/paragraph_vector.pdf) 174 | 1. [Dynamic word embeddings for evolving semantic discovery](https://blog.acolyer.org/2018/02/22/dynamic-word-embeddings-for-evolving-semantic-discovery/) from The Morning Paper blog 175 | 1. Ali Ghodsi's lecture on word2vec: 176 | - [part 1](https://www.youtube.com/watch?v=TsEGsdVJjuA) 177 | - [part 2](https://www.youtube.com/watch?v=nuirUEmbaJU) 178 | 1. [word2vec analogy demo](http://deeplearner.fz-qqq.net/) 179 | 1. [TensorFlow Embedding Projector of word vectors](http://projector.tensorflow.org/) 180 | 1. Skip-Thought Vectors - "unsupervised learning of a generic, distributed sentence encoder" 181 | - [Paper](http://arxiv.org/abs/1506.06726) 182 | - [Code](https://github.com/ryankiros/skip-thoughts) 183 | 184 | ----- 185 | Contributing 186 | ---- 187 | Have anything in mind that you think is awesome and would fit in this list? Feel free to send me a pull request! 188 | 189 | ----- 190 | License 191 | ---- 192 | 193 | [![CC0](http://i.creativecommons.org/p/zero/1.0/88x31.png)](http://creativecommons.org/publicdomain/zero/1.0/) 194 | 195 | To the extent possible under law, [Dr. Brian J. Spiering](http://www.linkedin.com/in/brianspiering/) has waived all copyright and related or neighboring rights to this work. 196 | --------------------------------------------------------------------------------