├── LICENSE ├── README.md ├── Section 1 ├── Lemmatization.ipynb ├── NER.ipynb ├── Stemming.ipynb ├── Tokenization.ipynb ├── Vectorization.ipynb └── Word2vec.ipynb ├── Section 2 ├── Building your first bot.ipynb └── aws_faq.csv ├── Section 3 └── drive-download-20181122T043051Z-001.zip ├── Section 4 └── drive-download-20181122T042821Z-001.zip ├── Section 5 ├── Deploy_chatbot.ipynb └── Generative Chatbot.ipynb └── Section 6 └── Notebooks-20190104T075709Z-001.zip /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Packt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Interactive Chatbots with TensorFlow [Video] 2 | This is the code repository for [Interactive Chatbots with TensorFlow [Video]](https://www.packtpub.com/big-data-and-business-intelligence/interactive-chatbots-tensorflow-video?utm_source=github&utm_medium=repository&utm_campaign=9781789613308), published by [Packt](https://www.packtpub.com/?utm_source=github). It contains all the supporting project files necessary to work through the video course from start to finish. 3 | ## About the Video Course 4 | Create chatbots of the future using the principles of Deep Learning, NLP, and the advanced TensorFlow library. Build contextually aware models to make your chatbots learn from repeated conversations and respond appropriately. 5 | 6 |

What You Will Learn

7 |
8 |
16 | 17 | ## Instructions and Navigation 18 | ### Assumed Knowledge 19 | To fully benefit from the coverage included in this course, you will need:
20 | This course is targeted at developers, researchers or aspiring AI data scientists who are interested in building chatbots using the latest research techniques. If you're interested in understanding the technology involved in building chatbots of the future, then this course is for you. 21 | ### Technical Requirements 22 | This course has the following software requirements:
23 | Minimum Hardware Requirements 24 | 25 | 26 | For successful completion of this course, students will require the computer systems with at least the following: 27 | 28 | 29 | 30 | 31 | OS: Ubuntu 16.04 or newer 32 | 33 | 34 | 35 | Processor: Intel i5 or an AMD equivalent 36 | 37 | 38 | 39 | GPU: Nvidia Geforce 960 or newer 40 | 41 | 42 | 43 | Memory: 8GB 44 | 45 | 46 | 47 | 48 | 49 | Recommended Hardware Requirements 50 | 51 | 52 | For an optimal experience with hands-on labs and other practical activities, we recommend the following configuration: 53 | 54 | 55 | 56 | OS: Ubuntu 16.04 or newer 57 | 58 | 59 | 60 | Processor: Intel i7 or an AMD equivalent 61 | 62 | 63 | 64 | GPU: Nvidia Geforce 1080 65 | 66 | 67 | 68 | Memory: 32GB 69 | 70 | 71 | 72 | 73 | Software Requirements 74 | 75 | 76 | 77 | Tensorflow 1.8 78 | 79 | 80 | 81 | Cuda 9 82 | 83 | 84 | 85 | Cudnn 7 86 | 87 | ## Related Products 88 | * [Hands-On Neural Network Programming with TensorFlow [Video]](https://www.packtpub.com/application-development/hands-neural-network-programming-tensorflow-video?utm_source=github&utm_medium=repository&utm_campaign=9781789534900) 89 | 90 | * [Hands-On Neural Network Programming with TensorFlow [Video]](https://www.packtpub.com/application-development/hands-neural-network-programming-tensorflow-video?utm_source=github&utm_medium=repository&utm_campaign=9781789534900) 91 | 92 | * [Hands-On Neural Network Programming with TensorFlow [Video]](https://www.packtpub.com/application-development/hands-neural-network-programming-tensorflow-video?utm_source=github&utm_medium=repository&utm_campaign=9781789534900) 93 | 94 | -------------------------------------------------------------------------------- /Section 1/Lemmatization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Lemmatization" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## In this notebook we solve the examples from the slides using spaCy." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "import spacy" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "nlp = spacy.load('en_core_web_sm')" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": null, 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "example1 = nlp(\"Animals\")\n", 42 | "for token in example1:\n", 43 | " print(token.lemma_)" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "example2 = nlp(\"is am are\")\n", 53 | "for token in example2:\n", 54 | " print(token.lemma_)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "lyrics = \"You better lose yourself in the music, the moment \"\\\n", 64 | "+ \"You own it, you better never let it go \" \\\n", 65 | "+ \"You only get one shot, do not miss your chance to blow \"\\\n", 66 | "+ \"This opportunity comes once in a lifetime\"\n", 67 | "\n", 68 | "example3 = nlp(lyrics)\n", 69 | "\n", 70 | "for token in example3:\n", 71 | " print(token.lemma_)" 72 | ] 73 | } 74 | ], 75 | "metadata": { 76 | "kernelspec": { 77 | "display_name": "Python 3", 78 | "language": "python", 79 | "name": "python3" 80 | }, 81 | "language_info": { 82 | "codemirror_mode": { 83 | "name": "ipython", 84 | "version": 3 85 | }, 86 | "file_extension": ".py", 87 | "mimetype": "text/x-python", 88 | "name": "python", 89 | "nbconvert_exporter": "python", 90 | "pygments_lexer": "ipython3", 91 | "version": "3.5.4" 92 | } 93 | }, 94 | "nbformat": 4, 95 | "nbformat_minor": 2 96 | } 97 | -------------------------------------------------------------------------------- /Section 1/NER.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Named Entity Recognition\n", 8 | "## In this notebook we will explore spaCy's abilities at detecting named entities." 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 1, 14 | "metadata": {}, 15 | "outputs": [], 16 | "source": [ 17 | "import spacy\n", 18 | "\n", 19 | "nlp = spacy.load('en_core_web_sm')" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 3, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "Google ORG\n", 32 | "Larry Page PERSON\n", 33 | "Sergey Brin ORG\n", 34 | "the United States of America GPE\n", 35 | "one CARDINAL\n" 36 | ] 37 | } 38 | ], 39 | "source": [ 40 | "example = \"Google, a company founded by Larry Page and Sergey Brin in the United States of America \"\\\n", 41 | "+ \"has one of the world’s most advanced search engines.\"\n", 42 | "\n", 43 | "doc = nlp(example)\n", 44 | "\n", 45 | "for ent in doc.ents:\n", 46 | " print(ent.text, ent.label_)" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 4, 52 | "metadata": {}, 53 | "outputs": [ 54 | { 55 | "name": "stdout", 56 | "output_type": "stream", 57 | "text": [ 58 | "U.S. GPE\n", 59 | "Taliban ORG\n", 60 | "America GPE\n", 61 | "three CARDINAL\n", 62 | "NBC News ORG\n" 63 | ] 64 | } 65 | ], 66 | "source": [ 67 | "example = \"U.S. officials are meeting with former Taliban members \"\\\n", 68 | "+ \"amid intensifying efforts to wind down America's longest war, three of the \"\\\n", 69 | "+ \"militant group's commanders told NBC News.\"\n", 70 | "\n", 71 | "doc = nlp(example)\n", 72 | "\n", 73 | "for ent in doc.ents:\n", 74 | " print(ent.text, ent.label_)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 6, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "name": "stdout", 84 | "output_type": "stream", 85 | "text": [ 86 | "an arduous year DATE\n", 87 | "German NORP\n", 88 | "Angela Merkel PERSON\n", 89 | "the European Union ORG\n", 90 | "US GPE\n", 91 | "Donald Trump PERSON\n", 92 | "one CARDINAL\n", 93 | "summer DATE\n", 94 | "Berlin GPE\n", 95 | "today (July 20 DATE\n", 96 | "a few days holiday DATE\n" 97 | ] 98 | } 99 | ], 100 | "source": [ 101 | "example = \"It’s been an arduous year for German chancellor Angela Merkel, so far. \"\\\n", 102 | "+ \"She has battled through coalition negotiations to form a government, chivvied \"\\\n", 103 | "+ \"the European Union into a loose agreement on migrants, weathered insults from \"\\\n", 104 | "+ \"US president Donald Trump, and headed off a revolt from her interior minister. No wonder \"\\\n", 105 | "+ \"then one journalist at her summer news conference in Berlin today (July 20) asked if \"\\\n", 106 | "+ \"she was, honestly, just exhausted. “I can’t complain,” Merkel said, “I have a few days \"\\\n", 107 | "+ \"holiday now and am looking forward to sleeping a bit longer.”\"\n", 108 | "\n", 109 | "doc = nlp(example)\n", 110 | "\n", 111 | "for ent in doc.ents:\n", 112 | " print(ent.text, ent.label_)" 113 | ] 114 | } 115 | ], 116 | "metadata": { 117 | "kernelspec": { 118 | "display_name": "Python 3", 119 | "language": "python", 120 | "name": "python3" 121 | }, 122 | "language_info": { 123 | "codemirror_mode": { 124 | "name": "ipython", 125 | "version": 3 126 | }, 127 | "file_extension": ".py", 128 | "mimetype": "text/x-python", 129 | "name": "python", 130 | "nbconvert_exporter": "python", 131 | "pygments_lexer": "ipython3", 132 | "version": "3.5.4" 133 | } 134 | }, 135 | "nbformat": 4, 136 | "nbformat_minor": 2 137 | } 138 | -------------------------------------------------------------------------------- /Section 1/Stemming.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Stemming" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Solving the Stemming examples from the slides using NLTK\n", 15 | "## More examples added in this notebook\n", 16 | "### You can install NLTK from https://www.nltk.org/install.html" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "from nltk.stem import PorterStemmer\n", 26 | "\n", 27 | "stemmer = PorterStemmer()" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 12, 33 | "metadata": {}, 34 | "outputs": [ 35 | { 36 | "name": "stdout", 37 | "output_type": "stream", 38 | "text": [ 39 | "cat run wa\n" 40 | ] 41 | } 42 | ], 43 | "source": [ 44 | "example = \"Cats Running Was\"\n", 45 | "example = [stemmer.stem(token) for token in example.split(\" \")]\n", 46 | "print(\" \".join(example))" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 13, 52 | "metadata": {}, 53 | "outputs": [ 54 | { 55 | "name": "stdout", 56 | "output_type": "stream", 57 | "text": [ 58 | "you better lose yourself in the music, the moment you own it, you better never let it go you onli get one shot, do not miss your chanc to blow thi opportun come onc in a lifetim \n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "lyrics = \"You better lose yourself in the music, the moment \"\\\n", 64 | "+ \"You own it, you better never let it go \"\\\n", 65 | "+ \"You only get one shot, do not miss your chance to blow \"\\\n", 66 | "+ \"This opportunity comes once in a lifetime \"\n", 67 | "lyrics = [stemmer.stem(token) for token in lyrics.split(\" \")]\n", 68 | "print(\" \".join(lyrics))" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 14, 74 | "metadata": {}, 75 | "outputs": [ 76 | { 77 | "name": "stdout", 78 | "output_type": "stream", 79 | "text": [ 80 | "bromwel high is a cartoon comedy. It ran at the same time as some other program about school life, such as \"teachers\". My 35 year in the teach profess lead me to believ that bromwel high' satir is much closer to realiti than is \"teachers\". the scrambl to surviv financially, the insight student who can see right through their pathet teachers' pomp, the petti of the whole situat , all remind me of the school I knew and their students. when I saw the episod in which a student repeatedli tri to burn down the school, I immedi recal ......... at .......... high. A classic line: inspector: i'm here to sack one of your teachers. student: welcom to bromwel high. I expect that mani adult of my age think that bromwel high is far fetched. what a piti that it isn't!\n" 81 | ] 82 | } 83 | ], 84 | "source": [ 85 | "review = \"Bromwell High is a cartoon comedy. \"\\\n", 86 | "+ \"It ran at the same time as some other programs about school life, such as \\\"Teachers\\\". \"\\\n", 87 | "+ \"My 35 years in the teaching profession lead me to believe that Bromwell High's satire is much \"\\\n", 88 | "+ \"closer to reality than is \\\"Teachers\\\". The scramble to survive financially, the insightful \"\\\n", 89 | "+ \"students who can see right through their pathetic teachers' pomp, the pettiness of the whole situation \"\\\n", 90 | "+ \", all remind me of the schools I knew and their students. When I saw the episode in which a student \"\\\n", 91 | "+ \"repeatedly tried to burn down the school, I immediately recalled ......... at .......... High. \"\\\n", 92 | "+ \"A classic line: INSPECTOR: I'm here to sack one of your teachers. STUDENT: Welcome to Bromwell High. \"\\\n", 93 | "+ \"I expect that many adults of my age think that Bromwell High is far fetched. What a pity that it isn't!\"\n", 94 | "\n", 95 | "review= [stemmer.stem(token) for token in review.split(\" \")]\n", 96 | "print(\" \".join(review))" 97 | ] 98 | } 99 | ], 100 | "metadata": { 101 | "kernelspec": { 102 | "display_name": "Python 3", 103 | "language": "python", 104 | "name": "python3" 105 | }, 106 | "language_info": { 107 | "codemirror_mode": { 108 | "name": "ipython", 109 | "version": 3 110 | }, 111 | "file_extension": ".py", 112 | "mimetype": "text/x-python", 113 | "name": "python", 114 | "nbconvert_exporter": "python", 115 | "pygments_lexer": "ipython3", 116 | "version": "3.5.4" 117 | } 118 | }, 119 | "nbformat": 4, 120 | "nbformat_minor": 2 121 | } 122 | -------------------------------------------------------------------------------- /Section 1/Tokenization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Tokenization using spaCy" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Load spaCy\n", 15 | "### If you don't have spaCy, you can install it from https://spacy.io/usage/" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 7, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "import spacy" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## Solve the examples from the slides using spaCy" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 8, 37 | "metadata": {}, 38 | "outputs": [ 39 | { 40 | "name": "stdout", 41 | "output_type": "stream", 42 | "text": [ 43 | "This\n", 44 | "is\n", 45 | "an\n", 46 | "example\n", 47 | "of\n", 48 | "text\n", 49 | "tokenization\n" 50 | ] 51 | } 52 | ], 53 | "source": [ 54 | "nlp = spacy.load('en_core_web_sm')\n", 55 | "\n", 56 | "example1 = nlp(\"This is an example of text tokenization\")\n", 57 | "\n", 58 | "for token in example1:\n", 59 | " print(token.text)\n" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 9, 65 | "metadata": {}, 66 | "outputs": [ 67 | { 68 | "name": "stdout", 69 | "output_type": "stream", 70 | "text": [ 71 | "The\n", 72 | "quick\n", 73 | "brown\n", 74 | "fox\n", 75 | "jumps\n", 76 | "over\n", 77 | "the\n", 78 | "lazy\n", 79 | "dog\n" 80 | ] 81 | } 82 | ], 83 | "source": [ 84 | "example2 = nlp(\"The quick brown fox jumps over the lazy dog\")\n", 85 | "\n", 86 | "for token in example2:\n", 87 | " print(token.text)" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 10, 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "name": "stdout", 97 | "output_type": "stream", 98 | "text": [ 99 | "We\n", 100 | "’re\n", 101 | "the\n", 102 | "champions\n" 103 | ] 104 | } 105 | ], 106 | "source": [ 107 | "example3 = nlp(\"We’re the champions\")\n", 108 | "\n", 109 | "for token in example3:\n", 110 | " print(token.text)" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": 11, 116 | "metadata": {}, 117 | "outputs": [ 118 | { 119 | "name": "stdout", 120 | "output_type": "stream", 121 | "text": [ 122 | "Will\n", 123 | "we\n", 124 | "have\n", 125 | "dinner\n", 126 | "today\n", 127 | "?\n" 128 | ] 129 | } 130 | ], 131 | "source": [ 132 | "example4 = nlp(\"Will we have dinner today?\")\n", 133 | "\n", 134 | "for token in example4:\n", 135 | " print(token.text)" 136 | ] 137 | } 138 | ], 139 | "metadata": { 140 | "kernelspec": { 141 | "display_name": "Python 3", 142 | "language": "python", 143 | "name": "python3" 144 | }, 145 | "language_info": { 146 | "codemirror_mode": { 147 | "name": "ipython", 148 | "version": 3 149 | }, 150 | "file_extension": ".py", 151 | "mimetype": "text/x-python", 152 | "name": "python", 153 | "nbconvert_exporter": "python", 154 | "pygments_lexer": "ipython3", 155 | "version": "3.5.4" 156 | } 157 | }, 158 | "nbformat": 4, 159 | "nbformat_minor": 2 160 | } 161 | -------------------------------------------------------------------------------- /Section 1/Vectorization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Vectorization\n", 8 | "## In this notebook we solve the examples in the slides and more using scikit-learn\n", 9 | "## You can install scikit-learn from: http://scikit-learn.org/stable/install.html" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 35, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "from sklearn.feature_extraction.text import CountVectorizer\n", 19 | "\n", 20 | "vectorizer = CountVectorizer(binary=True, token_pattern=r'\\b[^\\d\\W]+\\b')" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 36, 26 | "metadata": {}, 27 | "outputs": [ 28 | { 29 | "name": "stdout", 30 | "output_type": "stream", 31 | "text": [ 32 | "[[0 0 1 1 0 1 1 1]]\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "corpus = [\"The dog is on the table\", \"the cats now are on the table\"]\n", 38 | "vectorizer.fit(corpus)\n", 39 | "print(vectorizer.transform([\"The dog is on the table\"]).toarray())" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 37, 45 | "metadata": {}, 46 | "outputs": [ 47 | { 48 | "name": "stdout", 49 | "output_type": "stream", 50 | "text": [ 51 | "are: 0\n", 52 | "cats: 1\n", 53 | "dog: 2\n", 54 | "is: 3\n", 55 | "now: 4\n", 56 | "on: 5\n", 57 | "table: 6\n", 58 | "the: 7\n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "vocab = vectorizer.vocabulary_\n", 64 | "\n", 65 | "for key in sorted(vocab.keys()):\n", 66 | " print(\"{}: {}\".format(key, vocab[key]))" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": 38, 72 | "metadata": {}, 73 | "outputs": [ 74 | { 75 | "name": "stdout", 76 | "output_type": "stream", 77 | "text": [ 78 | "[[1 0 1 1 0 0]\n", 79 | " [0 1 0 0 1 1]\n", 80 | " [1 0 1 0 1 0]]\n" 81 | ] 82 | } 83 | ], 84 | "source": [ 85 | "corpus2 = [\"I am jack\", \"You are john\", \"I am john\"]\n", 86 | "vectorizer.fit(corpus2)\n", 87 | "\n", 88 | "print(vectorizer.transform(corpus2).toarray())" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 39, 94 | "metadata": {}, 95 | "outputs": [ 96 | { 97 | "name": "stdout", 98 | "output_type": "stream", 99 | "text": [ 100 | "am: 0\n", 101 | "are: 1\n", 102 | "i: 2\n", 103 | "jack: 3\n", 104 | "john: 4\n", 105 | "you: 5\n" 106 | ] 107 | } 108 | ], 109 | "source": [ 110 | "vocab = vectorizer.vocabulary_\n", 111 | "\n", 112 | "for key in sorted(vocab.keys()):\n", 113 | " print(\"{}: {}\".format(key, vocab[key]))" 114 | ] 115 | } 116 | ], 117 | "metadata": { 118 | "kernelspec": { 119 | "display_name": "Python 3", 120 | "language": "python", 121 | "name": "python3" 122 | }, 123 | "language_info": { 124 | "codemirror_mode": { 125 | "name": "ipython", 126 | "version": 3 127 | }, 128 | "file_extension": ".py", 129 | "mimetype": "text/x-python", 130 | "name": "python", 131 | "nbconvert_exporter": "python", 132 | "pygments_lexer": "ipython3", 133 | "version": "3.5.4" 134 | } 135 | }, 136 | "nbformat": 4, 137 | "nbformat_minor": 2 138 | } 139 | -------------------------------------------------------------------------------- /Section 1/Word2vec.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Word2vec\n", 8 | "## In this notebook we will play with spaCy's word vectors" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 2, 14 | "metadata": {}, 15 | "outputs": [], 16 | "source": [ 17 | "import spacy\n", 18 | "\n", 19 | "nlp = spacy.load('en_core_web_lg')" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 3, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "man man 1.0\n", 32 | "man woman 0.5783229\n", 33 | "man king 0.491998\n", 34 | "man queen 0.09714928\n", 35 | "woman man 0.5783229\n", 36 | "woman woman 1.0\n", 37 | "woman king 0.67171866\n", 38 | "woman queen 0.34120008\n", 39 | "king man 0.491998\n", 40 | "king woman 0.67171866\n", 41 | "king king 1.0\n", 42 | "king queen 0.31878188\n", 43 | "queen man 0.09714928\n", 44 | "queen woman 0.34120008\n", 45 | "queen king 0.31878188\n", 46 | "queen queen 1.0\n" 47 | ] 48 | } 49 | ], 50 | "source": [ 51 | "example1 = \"man woman king queen\"\n", 52 | "tokens = nlp(example1)\n", 53 | "for token1 in tokens:\n", 54 | " for token2 in tokens:\n", 55 | " print(token1.text, token2.text, token1.similarity(token2))" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 7, 61 | "metadata": {}, 62 | "outputs": [ 63 | { 64 | "name": "stdout", 65 | "output_type": "stream", 66 | "text": [ 67 | "walking walked 0.03380737\n", 68 | "walking swimming 0.46878663\n", 69 | "walking swam 0.13061452\n", 70 | "walked walking 0.03380737\n", 71 | "walked swimming -0.07051619\n", 72 | "walked swam -0.04058967\n", 73 | "swimming walking 0.46878663\n", 74 | "swimming walked -0.07051619\n", 75 | "swimming swam 0.3709051\n", 76 | "swam walking 0.13061452\n", 77 | "swam walked -0.04058967\n", 78 | "swam swimming 0.3709051\n" 79 | ] 80 | } 81 | ], 82 | "source": [ 83 | "example1 = \"walking walked swimming swam\"\n", 84 | "tokens = nlp(example1)\n", 85 | "for token1 in tokens:\n", 86 | " for token2 in tokens:\n", 87 | " if(token1.text == token2.text):\n", 88 | " continue\n", 89 | " print(token1.text, token2.text, token1.similarity(token2))" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 8, 95 | "metadata": {}, 96 | "outputs": [ 97 | { 98 | "name": "stdout", 99 | "output_type": "stream", 100 | "text": [ 101 | "spain russia 0.3216639\n", 102 | "spain madrid 0.51518124\n", 103 | "spain moscow 0.23857422\n", 104 | "russia spain 0.3216639\n", 105 | "russia madrid 0.49487153\n", 106 | "russia moscow 0.29201046\n", 107 | "madrid spain 0.51518124\n", 108 | "madrid russia 0.49487153\n", 109 | "madrid moscow 0.29396096\n", 110 | "moscow spain 0.23857422\n", 111 | "moscow russia 0.29201046\n", 112 | "moscow madrid 0.29396096\n" 113 | ] 114 | } 115 | ], 116 | "source": [ 117 | "example1 = \"spain russia madrid moscow\"\n", 118 | "tokens = nlp(example1)\n", 119 | "for token1 in tokens:\n", 120 | " for token2 in tokens:\n", 121 | " if(token1.text == token2.text):\n", 122 | " continue\n", 123 | " print(token1.text, token2.text, token1.similarity(token2))" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 9, 129 | "metadata": {}, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "cat dog 0.3464731\n", 136 | "dog cat 0.3464731\n" 137 | ] 138 | } 139 | ], 140 | "source": [ 141 | "example1 = \"cat dog\"\n", 142 | "tokens = nlp(example1)\n", 143 | "for token1 in tokens:\n", 144 | " for token2 in tokens:\n", 145 | " if(token1.text == token2.text):\n", 146 | " continue \n", 147 | " print(token1.text, token2.text, token1.similarity(token2))" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 10, 153 | "metadata": {}, 154 | "outputs": [ 155 | { 156 | "name": "stdout", 157 | "output_type": "stream", 158 | "text": [ 159 | "cat pizza 0.40963048\n", 160 | "pizza cat 0.40963048\n" 161 | ] 162 | } 163 | ], 164 | "source": [ 165 | "example1 = \"cat pizza\"\n", 166 | "tokens = nlp(example1)\n", 167 | "for token1 in tokens:\n", 168 | " for token2 in tokens:\n", 169 | " if(token1.text == token2.text):\n", 170 | " continue \n", 171 | " print(token1.text, token2.text, token1.similarity(token2))" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 11, 177 | "metadata": {}, 178 | "outputs": [ 179 | { 180 | "name": "stdout", 181 | "output_type": "stream", 182 | "text": [ 183 | "flower pasta 0.51757944\n", 184 | "pasta flower 0.51757944\n" 185 | ] 186 | } 187 | ], 188 | "source": [ 189 | "example1 = \"flower pasta\"\n", 190 | "tokens = nlp(example1)\n", 191 | "for token1 in tokens:\n", 192 | " for token2 in tokens:\n", 193 | " if(token1.text == token2.text):\n", 194 | " continue \n", 195 | " print(token1.text, token2.text, token1.similarity(token2))" 196 | ] 197 | } 198 | ], 199 | "metadata": { 200 | "kernelspec": { 201 | "display_name": "Python 3", 202 | "language": "python", 203 | "name": "python3" 204 | }, 205 | "language_info": { 206 | "codemirror_mode": { 207 | "name": "ipython", 208 | "version": 3 209 | }, 210 | "file_extension": ".py", 211 | "mimetype": "text/x-python", 212 | "name": "python", 213 | "nbconvert_exporter": "python", 214 | "pygments_lexer": "ipython3", 215 | "version": "3.5.4" 216 | } 217 | }, 218 | "nbformat": 4, 219 | "nbformat_minor": 2 220 | } 221 | -------------------------------------------------------------------------------- /Section 2/Building your first bot.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true, 8 | "deletable": true, 9 | "editable": true 10 | }, 11 | "outputs": [], 12 | "source": [ 13 | "# Load data preprocessing libs\n", 14 | "import pandas as pd\n", 15 | "import numpy as np\n", 16 | "\n", 17 | "# Load vectorizer and similarity measure\n", 18 | "from sklearn.feature_extraction.text import TfidfVectorizer\n", 19 | "from sklearn.metrics.pairwise import cosine_similarity\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "deletable": true, 26 | "editable": true 27 | }, 28 | "source": [ 29 | "## Read the data" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 2, 35 | "metadata": { 36 | "collapsed": true, 37 | "deletable": true, 38 | "editable": true 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "# Read data and drop examples that has no answer\n", 43 | "df = pd.read_csv(\"aws_faq.csv\")\n", 44 | "df.dropna(inplace=True)" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": { 50 | "deletable": true, 51 | "editable": true 52 | }, 53 | "source": [ 54 | "## Train the vectorizer" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 3, 60 | "metadata": { 61 | "collapsed": false, 62 | "deletable": true, 63 | "editable": true 64 | }, 65 | "outputs": [ 66 | { 67 | "data": { 68 | "text/plain": [ 69 | "TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',\n", 70 | " dtype=, encoding='utf-8', input='content',\n", 71 | " lowercase=True, max_df=1.0, max_features=None, min_df=1,\n", 72 | " ngram_range=(1, 1), norm='l2', preprocessor=None, smooth_idf=True,\n", 73 | " stop_words=None, strip_accents=None, sublinear_tf=False,\n", 74 | " token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b', tokenizer=None, use_idf=True,\n", 75 | " vocabulary=None)" 76 | ] 77 | }, 78 | "execution_count": 3, 79 | "metadata": {}, 80 | "output_type": "execute_result" 81 | } 82 | ], 83 | "source": [ 84 | "vectorizer = TfidfVectorizer()\n", 85 | "vectorizer.fit(np.concatenate((df.Question, df.Answer)))" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": { 91 | "deletable": true, 92 | "editable": true 93 | }, 94 | "source": [ 95 | "## Vectorize questions" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 4, 101 | "metadata": { 102 | "collapsed": true, 103 | "deletable": true, 104 | "editable": true 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "Question_vectors = vectorizer.transform(df.Question)" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": { 114 | "deletable": true, 115 | "editable": true 116 | }, 117 | "source": [ 118 | "\n", 119 | "## Chat with the user" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": { 126 | "collapsed": false, 127 | "deletable": true, 128 | "editable": true, 129 | "scrolled": false 130 | }, 131 | "outputs": [ 132 | { 133 | "name": "stdout", 134 | "output_type": "stream", 135 | "text": [ 136 | "You can start chatting with me now.\n", 137 | "Hello\n", 138 | "BOT: Hello\n", 139 | "How are you?\n", 140 | "BOT: I’m fine thanks.\n", 141 | "I've heard about ec2, can you tell me what can I do with it?\n", 142 | "BOT: Just as Amazon Simple Storage Service (Amazon S3) enables storage in the cloud, Amazon EC2 enables “compute” in the cloud. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use.\n", 143 | "how quickly will ec2 systems run?\n", 144 | "BOT: It typically takes less than 10 minutes from the issue of the RunInstances call to the point where all requested instances begin their boot sequences. This time depends on a number of factors including: the size of your AMI, the number of instances you are launching, and how recently you have launched that AMI. Images launched for the first time may take slightly longer to boot.\n", 145 | "how will I be charged for my usage?\n", 146 | "BOT: You pay only for what you use. Displayed pricing is an hourly rate but depending on which instances you choose, you pay by the hour or second (minimum of 60 seconds) for each instance type. Partial instance-hours consumed are billed based on instance usage. Data transferred between AWS services in different regions will be charged as Internet Data Transfer on both sides of the transfer. Usage for other Amazon Web Services is billed separately from Amazon EC2.\n", 147 | "For EC2 pricing information, please visit the \n", 148 | "pricing section on the EC2 detail page\n", 149 | ".\n", 150 | "are taxes included in pricing?\n", 151 | "BOT: Except as otherwise noted, our prices are exclusive of applicable taxes and duties, including VAT and applicable sales tax. For customers with a Japanese billing address, use of AWS services is subject to Japanese Consumption Tax. \n", 152 | "Learn more\n", 153 | ".\n", 154 | "Thanks, bye\n", 155 | "BOT: Bye, glad to have helped.\n" 156 | ] 157 | } 158 | ], 159 | "source": [ 160 | "print(\"You can start chatting with me now.\")\n", 161 | "while True:\n", 162 | " # Read user input\n", 163 | " input_question = input()\n", 164 | "\n", 165 | " # Locate the closest question\n", 166 | " input_question_vector = vectorizer.transform([input_question])\n", 167 | "\n", 168 | " # Compute similarities\n", 169 | " similarities = cosine_similarity(input_question_vector, Question_vectors)\n", 170 | "\n", 171 | " # Find the closest question\n", 172 | " closest = np.argmax(similarities, axis=1)\n", 173 | "\n", 174 | " # Print the correct answer\n", 175 | " print(\"BOT: \" + df.Answer.iloc[closest].values[0])" 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": null, 181 | "metadata": { 182 | "collapsed": true, 183 | "deletable": true, 184 | "editable": true 185 | }, 186 | "outputs": [], 187 | "source": [] 188 | } 189 | ], 190 | "metadata": { 191 | "anaconda-cloud": {}, 192 | "kernelspec": { 193 | "display_name": "Python 3", 194 | "language": "python", 195 | "name": "python3" 196 | }, 197 | "language_info": { 198 | "codemirror_mode": { 199 | "name": "ipython", 200 | "version": 3 201 | }, 202 | "file_extension": ".py", 203 | "mimetype": "text/x-python", 204 | "name": "python", 205 | "nbconvert_exporter": "python", 206 | "pygments_lexer": "ipython3", 207 | "version": "3.5.5" 208 | } 209 | }, 210 | "nbformat": 4, 211 | "nbformat_minor": 2 212 | } 213 | -------------------------------------------------------------------------------- /Section 3/drive-download-20181122T043051Z-001.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Interactive-Chatbots-with-TensorFlow-/0e3a665e9a22cb852e7325420af3d0fd37fc7c34/Section 3/drive-download-20181122T043051Z-001.zip -------------------------------------------------------------------------------- /Section 4/drive-download-20181122T042821Z-001.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Interactive-Chatbots-with-TensorFlow-/0e3a665e9a22cb852e7325420af3d0fd37fc7c34/Section 4/drive-download-20181122T042821Z-001.zip -------------------------------------------------------------------------------- /Section 5/Deploy_chatbot.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false, 8 | "deletable": true, 9 | "editable": true 10 | }, 11 | "outputs": [ 12 | { 13 | "name": "stderr", 14 | "output_type": "stream", 15 | "text": [ 16 | "/home/omar/anaconda2/envs/py35/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 17 | " from ._conv import register_converters as _register_converters\n" 18 | ] 19 | } 20 | ], 21 | "source": [ 22 | "import tensorflow as tf\n", 23 | "import numpy as np\n", 24 | "import pandas as pd\n", 25 | "\n", 26 | "from sklearn.model_selection import train_test_split\n", 27 | "from sklearn.preprocessing import LabelEncoder\n", 28 | "from sklearn.feature_extraction.text import CountVectorizer\n", 29 | "\n", 30 | "from nltk.stem import PorterStemmer\n", 31 | "from autocorrect import spell\n", 32 | "\n", 33 | "import os\n", 34 | "from six.moves import cPickle\n", 35 | "import re\n" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 2, 41 | "metadata": { 42 | "collapsed": true, 43 | "deletable": true, 44 | "editable": true 45 | }, 46 | "outputs": [], 47 | "source": [ 48 | "MAX_LEN = 25\n", 49 | "BATCH_SIZE = 64\n", 50 | "\n", 51 | "stemmer = PorterStemmer()\n", 52 | "def process_str(string, bot_input=False, bot_output=False):\n", 53 | " string = string.strip().lower()\n", 54 | " string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`:]\", \" \", string)\n", 55 | " string = re.sub(r\"\\'s\", \" \\'s\", string)\n", 56 | " string = re.sub(r\"\\'ve\", \" \\'ve\", string)\n", 57 | " string = re.sub(r\"n\\'t\", \" n\\'t\", string)\n", 58 | " string = re.sub(r\"\\'re\", \" \\'re\", string)\n", 59 | " string = re.sub(r\"\\'d\", \" \\'d\", string)\n", 60 | " string = re.sub(r\"\\'ll\", \" \\'ll\", string)\n", 61 | " string = re.sub(r\",\", \" , \", string)\n", 62 | " string = re.sub(r\"!\", \" ! \", string)\n", 63 | " string = re.sub(r\"\\s{2,}\", \" \", string)\n", 64 | " string = string.split(\" \")\n", 65 | " string = [re.sub(r\"[0-9]+\", \"NUM\", token) for token in string]\n", 66 | " string = [stemmer.stem(re.sub(r'(.)\\1+', r'\\1\\1', token)) for token in string]\n", 67 | " string = [spell(token).lower() for token in string]\n", 68 | " # Truncate string\n", 69 | " while True:\n", 70 | " try:\n", 71 | " string.remove(\"\")\n", 72 | " except:\n", 73 | " break\n", 74 | " if(not bot_input and not bot_output):\n", 75 | " string = string[0:MAX_LEN]\n", 76 | " elif(bot_input):\n", 77 | " string = string[0:MAX_LEN-1]\n", 78 | " string.insert(0, \"\")\n", 79 | " else:\n", 80 | " string = string[0:MAX_LEN-1]\n", 81 | " string.insert(len(string), \"\")\n", 82 | " old_len = len(string)\n", 83 | " for i in range((MAX_LEN) - len(string)):\n", 84 | " string.append(\" \")\n", 85 | " string = re.sub(\"\\s+\", \" \", \" \".join(string)).strip()\n", 86 | " return string, old_len" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 3, 92 | "metadata": { 93 | "collapsed": false, 94 | "deletable": true, 95 | "editable": true 96 | }, 97 | "outputs": [ 98 | { 99 | "name": "stdout", 100 | "output_type": "stream", 101 | "text": [ 102 | "INFO:tensorflow:Restoring parameters from checkpoints/best_validation\n" 103 | ] 104 | } 105 | ], 106 | "source": [ 107 | "imported_graph = tf.train.import_meta_graph('checkpoints/best_validation.meta')\n", 108 | "sess = tf.InteractiveSession()\n", 109 | "imported_graph.restore(sess, \"checkpoints/best_validation\")\n", 110 | "\n", 111 | "sess.run(tf.tables_initializer())\n", 112 | "graph = tf.get_default_graph()" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 4, 118 | "metadata": { 119 | "collapsed": true, 120 | "deletable": true, 121 | "editable": true 122 | }, 123 | "outputs": [], 124 | "source": [ 125 | "def test(text):\n", 126 | " text, text_len = process_str(text)\n", 127 | " text = [text] + [\"hi\"] * (BATCH_SIZE-1)\n", 128 | " text_len = [text_len] + [1] * (BATCH_SIZE-1)\n", 129 | " return text, text_len" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 5, 135 | "metadata": { 136 | "collapsed": false, 137 | "deletable": true, 138 | "editable": true 139 | }, 140 | "outputs": [], 141 | "source": [ 142 | "test_init_op = graph.get_operation_by_name('data/dataset_init')\n", 143 | "\n", 144 | "user_ph = graph.get_tensor_by_name(\"user_placeholder:0\")\n", 145 | "bot_inp_ph = graph.get_tensor_by_name(\"bot_inp_placeholder:0\")\n", 146 | "bot_out_ph = graph.get_tensor_by_name(\"bot_out_placeholder:0\")\n", 147 | "\n", 148 | "user_lens_ph = graph.get_tensor_by_name(\"user_len_placeholder:0\")\n", 149 | "bot_inp_lens_ph = graph.get_tensor_by_name(\"bot_inp_lens_placeholder:0\")\n", 150 | "bot_out_lens_ph = graph.get_tensor_by_name(\"bot_out_lens_placeholder:0\")\n", 151 | "\n", 152 | "words = graph.get_tensor_by_name(\"inference/words:0\")\n" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "def chat(text):\n", 166 | " user, user_lens = test(text)\n", 167 | " sess.run(test_init_op, feed_dict={\n", 168 | " user_ph: user,\n", 169 | " bot_inp_ph: [\"hi\"] * BATCH_SIZE,\n", 170 | " bot_out_ph: [\"hi\"] * BATCH_SIZE,\n", 171 | " user_lens_ph: user_lens,\n", 172 | " bot_inp_lens_ph: [1] * BATCH_SIZE,\n", 173 | " bot_out_lens_ph: [1] * BATCH_SIZE\n", 174 | " })\n", 175 | " translations_text = sess.run(words)\n", 176 | " output = [item.decode() for item in translations_text[0]]\n", 177 | " if(\"\" in output):\n", 178 | " end_idx = output.index(\"\")\n", 179 | " output = output[0:end_idx]\n", 180 | " output = \" \".join(output)\n", 181 | " print(\"BOT: \" + output)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": { 188 | "collapsed": false, 189 | "deletable": true, 190 | "editable": true 191 | }, 192 | "outputs": [], 193 | "source": [ 194 | "while True:\n", 195 | " chat(input())" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": null, 201 | "metadata": { 202 | "collapsed": false, 203 | "deletable": true, 204 | "editable": true 205 | }, 206 | "outputs": [], 207 | "source": [] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": { 213 | "collapsed": true 214 | }, 215 | "outputs": [], 216 | "source": [] 217 | } 218 | ], 219 | "metadata": { 220 | "kernelspec": { 221 | "display_name": "Python 3", 222 | "language": "python", 223 | "name": "python3" 224 | }, 225 | "language_info": { 226 | "codemirror_mode": { 227 | "name": "ipython", 228 | "version": 3 229 | }, 230 | "file_extension": ".py", 231 | "mimetype": "text/x-python", 232 | "name": "python", 233 | "nbconvert_exporter": "python", 234 | "pygments_lexer": "ipython3", 235 | "version": "3.5.5" 236 | } 237 | }, 238 | "nbformat": 4, 239 | "nbformat_minor": 2 240 | } 241 | -------------------------------------------------------------------------------- /Section 5/Generative Chatbot.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "# Generative Chatbot" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "Code adapted from https://github.com/tensorflow/nmt#training--how-to-build-our-first-nmt-system\n", 21 | "\n", 22 | "Also more insights drawn from https://github.com/thushv89/exercises_thushv_dot_com/blob/master/nmt_tutorial.ipynb" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": { 29 | "collapsed": true, 30 | "deletable": true, 31 | "editable": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "import pandas as pd\n", 36 | "import numpy as np\n", 37 | "\n", 38 | "from sklearn.model_selection import train_test_split\n", 39 | "from sklearn.preprocessing import LabelEncoder\n", 40 | "from sklearn.feature_extraction.text import CountVectorizer\n", 41 | "\n", 42 | "from nltk.stem import PorterStemmer\n", 43 | "from autocorrect import spell\n", 44 | "\n", 45 | "import os\n", 46 | "from six.moves import cPickle\n", 47 | "import re" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": { 54 | "collapsed": false, 55 | "deletable": true, 56 | "editable": true 57 | }, 58 | "outputs": [], 59 | "source": [ 60 | "import tensorflow as tf\n", 61 | "from tensorflow.contrib.tensorboard.plugins import projector\n", 62 | "\n", 63 | "sconfig = tf.ConfigProto()\n", 64 | "# sconfig.gpu_options.per_process_gpu_memory_fraction = 0.45\n" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": { 71 | "collapsed": true, 72 | "deletable": true, 73 | "editable": true 74 | }, 75 | "outputs": [], 76 | "source": [ 77 | "MAX_LEN = 25\n", 78 | "BATCH_SIZE = 64\n", 79 | "NUM_EPOCHS = 10000\n", 80 | "\n", 81 | "stemmer = PorterStemmer()\n", 82 | "def process_str(string, bot_input=False, bot_output=False):\n", 83 | " string = string.strip().lower()\n", 84 | " string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`:]\", \" \", string)\n", 85 | " string = re.sub(r\"\\'s\", \" \\'s\", string)\n", 86 | " string = re.sub(r\"\\'ve\", \" \\'ve\", string)\n", 87 | " string = re.sub(r\"n\\'t\", \" n\\'t\", string)\n", 88 | " string = re.sub(r\"\\'re\", \" \\'re\", string)\n", 89 | " string = re.sub(r\"\\'d\", \" \\'d\", string)\n", 90 | " string = re.sub(r\"\\'ll\", \" \\'ll\", string)\n", 91 | " string = re.sub(r\",\", \" , \", string)\n", 92 | " string = re.sub(r\"!\", \" ! \", string)\n", 93 | " string = re.sub(r\"\\s{2,}\", \" \", string)\n", 94 | " string = string.split(\" \")\n", 95 | " string = [re.sub(r\"[0-9]+\", \"NUM\", token) for token in string]\n", 96 | " string = [stemmer.stem(re.sub(r'(.)\\1+', r'\\1\\1', token)) for token in string]\n", 97 | " string = [spell(token).lower() for token in string]\n", 98 | " # Truncate string\n", 99 | " while True:\n", 100 | " try:\n", 101 | " string.remove(\"\")\n", 102 | " except:\n", 103 | " break\n", 104 | " if(not bot_input and not bot_output):\n", 105 | " string = string[0:MAX_LEN]\n", 106 | " elif(bot_input):\n", 107 | " string = string[0:MAX_LEN-1]\n", 108 | " string.insert(0, \"\")\n", 109 | " else:\n", 110 | " string = string[0:MAX_LEN-1]\n", 111 | " string.insert(len(string), \"\")\n", 112 | " old_len = len(string)\n", 113 | " for i in range((MAX_LEN) - len(string)):\n", 114 | " string.append(\" \")\n", 115 | " string = re.sub(\"\\s+\", \" \", \" \".join(string)).strip()\n", 116 | " return string, old_len" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": { 122 | "deletable": true, 123 | "editable": true 124 | }, 125 | "source": [ 126 | "# Load data" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": false, 134 | "deletable": true, 135 | "editable": true 136 | }, 137 | "outputs": [], 138 | "source": [ 139 | "data = cPickle.load(open(\"all_convos.pkl\", \"rb\"))\n", 140 | "print(len(data))\n", 141 | "user = [item[0] for item in data]\n", 142 | "bot = [item[1] for item in data]" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": { 148 | "deletable": true, 149 | "editable": true 150 | }, 151 | "source": [ 152 | "## Preprocess data" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "if(os.path.isfile(\"user_processed.pkl\")):\n", 166 | " user = cPickle.load(open(\"user_processed.pkl\", \"rb\"))\n", 167 | "else:\n", 168 | " user = [process_str(item) for item in user]\n", 169 | " cPickle.dump(user, open(\"user_processed.pkl\", \"wb\"))\n", 170 | "\n", 171 | "if(os.path.isfile(\"bot_in_processed.pkl\")):\n", 172 | " bot_inputs = cPickle.load(open(\"bot_in_processed.pkl\", \"rb\"))\n", 173 | "else:\n", 174 | " bot_inputs = [process_str(item, bot_input=True) for item in bot]\n", 175 | " cPickle.dump(bot_inputs, open(\"bot_in_processed.pkl\", \"wb\"))\n", 176 | "\n", 177 | "if(os.path.isfile(\"bot_out_processed.pkl\")):\n", 178 | " bot_outputs = cPickle.load(open(\"bot_out_processed.pkl\", \"rb\"))\n", 179 | "else:\n", 180 | " bot_outputs = [process_str(item, bot_output=True) for item in bot]\n", 181 | " cPickle.dump(bot_outputs, open(\"bot_out_processed.pkl\", \"wb\"))\n", 182 | " \n", 183 | " \n", 184 | "user_lens = np.array([message[1] for message in user]).astype(np.int32)\n", 185 | "user = np.array([message[0] for message in user])\n", 186 | "\n", 187 | "bot_inp_lens = np.array([message[1] for message in bot_inputs]).astype(np.int32)\n", 188 | "bot_out_lens = np.array([message[1] for message in bot_outputs]).astype(np.int32)\n", 189 | "\n", 190 | "bot_inputs = np.array([message[0] for message in bot_inputs])\n", 191 | "bot_outputs = np.array([message[0] for message in bot_outputs])" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "deletable": true, 198 | "editable": true 199 | }, 200 | "source": [ 201 | "## Show statistics about length" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": { 208 | "collapsed": false, 209 | "deletable": true, 210 | "editable": true 211 | }, 212 | "outputs": [], 213 | "source": [ 214 | "print(\"Average user message: {}, average bot message: {}\".format(np.mean(user_lens), np.mean(bot_inp_lens)))\n", 215 | "print(\"80th percentile of user lengths: {}, 80th percentile of bot lengths: {}\".format(np.percentile(user_lens, 80), np.percentile(bot_inp_lens, 80)))" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": { 221 | "deletable": true, 222 | "editable": true 223 | }, 224 | "source": [ 225 | "## Extract vocabulary" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": null, 231 | "metadata": { 232 | "collapsed": true, 233 | "deletable": true, 234 | "editable": true 235 | }, 236 | "outputs": [], 237 | "source": [ 238 | "bow = CountVectorizer()\n", 239 | "\n", 240 | "bow.fit(user.tolist() + bot_inputs.tolist())\n", 241 | "vocab = list(bow.vocabulary_.keys())\n", 242 | "vocab.insert(0, \"NUM\")\n", 243 | "vocab.insert(0, \"UNK\")\n", 244 | "vocab.insert(0, \"\")\n", 245 | "vocab.insert(0, \"\")\n", 246 | "vocab.insert(0, \"\")\n", 247 | "cPickle.dump(vocab, open(\"vocab\", \"wb\"))" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": { 253 | "deletable": true, 254 | "editable": true 255 | }, 256 | "source": [ 257 | "## Placeholders" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": null, 263 | "metadata": { 264 | "collapsed": true, 265 | "deletable": true, 266 | "editable": true 267 | }, 268 | "outputs": [], 269 | "source": [ 270 | "user_ph = tf.placeholder(user.dtype, name=\"user_placeholder\")\n", 271 | "bot_inp_ph = tf.placeholder(bot_inputs.dtype, name=\"bot_inp_placeholder\")\n", 272 | "bot_out_ph = tf.placeholder(bot_outputs.dtype, name=\"bot_out_placeholder\")\n", 273 | "\n", 274 | "user_lens_ph = tf.placeholder(user_lens.dtype, shape=[None], name=\"user_len_placeholder\")\n", 275 | "bot_inp_lens_ph = tf.placeholder(bot_inp_lens.dtype, shape=[None], name=\"bot_inp_lens_placeholder\")\n", 276 | "bot_out_lens_ph = tf.placeholder(bot_out_lens.dtype, shape=[None], name=\"bot_out_lens_placeholder\")" 277 | ] 278 | }, 279 | { 280 | "cell_type": "markdown", 281 | "metadata": { 282 | "deletable": true, 283 | "editable": true 284 | }, 285 | "source": [ 286 | "## Datasets" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": null, 292 | "metadata": { 293 | "collapsed": true, 294 | "deletable": true, 295 | "editable": true 296 | }, 297 | "outputs": [], 298 | "source": [ 299 | "tf_user = tf.data.Dataset.from_tensor_slices(user_ph)\n", 300 | "tf_bot_inp = tf.data.Dataset.from_tensor_slices(bot_inp_ph)\n", 301 | "tf_bot_out = tf.data.Dataset.from_tensor_slices(bot_out_ph)\n", 302 | "\n", 303 | "tf_user_lens = tf.data.Dataset.from_tensor_slices(user_lens_ph)\n", 304 | "tf_bot_inp_lens = tf.data.Dataset.from_tensor_slices(bot_inp_lens_ph)\n", 305 | "tf_bot_out_lens = tf.data.Dataset.from_tensor_slices(bot_out_lens_ph)\n" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": { 311 | "deletable": true, 312 | "editable": true 313 | }, 314 | "source": [ 315 | "## Data/Iterators" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "metadata": { 322 | "collapsed": true, 323 | "deletable": true, 324 | "editable": true 325 | }, 326 | "outputs": [], 327 | "source": [ 328 | "with tf.device(\"/cpu:0\"), tf.name_scope(\"data\"):\n", 329 | " words = tf.contrib.lookup.index_table_from_tensor(mapping=tf.constant(vocab), default_value=3)\n", 330 | " inverse = tf.contrib.lookup.index_to_string_table_from_tensor(mapping=tf.constant(vocab), default_value=\"UNK\", name=\"inverse_op\")\n", 331 | "\n", 332 | " tf_user = tf_user.map(lambda string: tf.string_split([string])).map(lambda tokens: (words.lookup(tokens)))\n", 333 | " tf_bot_inp = tf_bot_inp.map(lambda string: tf.string_split([string])).map(lambda tokens: (words.lookup(tokens)))\n", 334 | " tf_bot_out = tf_bot_out.map(lambda string: tf.string_split([string])).map(lambda tokens: (words.lookup(tokens)))\n", 335 | " \n", 336 | " data = tf.data.Dataset.zip((tf_user, tf_bot_inp, tf_bot_out, tf_user_lens, tf_bot_inp_lens, tf_bot_out_lens))\n", 337 | " data = data.shuffle(buffer_size=256).batch(BATCH_SIZE)\n", 338 | " data = data.prefetch(10)\n", 339 | " data_iterator = tf.data.Iterator.from_structure(data.output_types, data.output_shapes,\n", 340 | " None, data.output_classes)\n", 341 | " train_init_op = data_iterator.make_initializer(data, name='dataset_init')\n", 342 | " user_doc, bot_inp_doc, bot_out_doc, user_len, bot_inp_len, bot_out_len = data_iterator.get_next()\n", 343 | " user_doc = tf.sparse_tensor_to_dense(user_doc)\n", 344 | " bot_inp_doc = tf.sparse_tensor_to_dense(bot_inp_doc)\n", 345 | " bot_out_doc = tf.sparse_tensor_to_dense(bot_out_doc)" 346 | ] 347 | }, 348 | { 349 | "cell_type": "markdown", 350 | "metadata": { 351 | "deletable": true, 352 | "editable": true 353 | }, 354 | "source": [ 355 | "## Embedding" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": null, 361 | "metadata": { 362 | "collapsed": true, 363 | "deletable": true, 364 | "editable": true 365 | }, 366 | "outputs": [], 367 | "source": [ 368 | "with tf.name_scope(\"embedding\"):\n", 369 | " embedding = tf.get_variable(\"embedding\", [len(vocab), 200], initializer=tf.glorot_uniform_initializer())\n", 370 | " \n", 371 | " embedded_user = tf.nn.embedding_lookup(embedding, user_doc)\n", 372 | " embedded_user_dropout = tf.nn.dropout(embedded_user, 0.7)\n", 373 | " \n", 374 | " embedded_bot_inp = tf.nn.embedding_lookup(embedding, bot_inp_doc)\n", 375 | " embedded_bot_inp_dropout = tf.nn.dropout(embedded_bot_inp, 0.7)\n", 376 | " \n", 377 | " embedded_user_dropout = tf.reshape(embedded_user_dropout, [-1, MAX_LEN, 200])\n", 378 | " embedded_bot_inp_dropout = tf.reshape(embedded_bot_inp_dropout, [-1, MAX_LEN, 200])" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "metadata": { 384 | "deletable": true, 385 | "editable": true 386 | }, 387 | "source": [ 388 | "## Encoder" 389 | ] 390 | }, 391 | { 392 | "cell_type": "code", 393 | "execution_count": null, 394 | "metadata": { 395 | "collapsed": true, 396 | "deletable": true, 397 | "editable": true 398 | }, 399 | "outputs": [], 400 | "source": [ 401 | "with tf.name_scope(\"encoder\"):\n", 402 | " # Build RNN cell\n", 403 | " encoder_GRU = tf.nn.rnn_cell.GRUCell(128)\n", 404 | " encoder_cell_fw = tf.nn.rnn_cell.DropoutWrapper(encoder_GRU, input_keep_prob=0.7, \n", 405 | " output_keep_prob=0.7, state_keep_prob=0.9)\n", 406 | " \n", 407 | " encoder_cell_bw = tf.nn.rnn_cell.DropoutWrapper(encoder_GRU, input_keep_prob=0.7, \n", 408 | " output_keep_prob=0.7, state_keep_prob=0.9)\n", 409 | " encoder_outputs, encoder_state = tf.nn.bidirectional_dynamic_rnn(\n", 410 | " encoder_cell_fw, encoder_cell_bw, embedded_user_dropout,\n", 411 | " sequence_length=user_len, dtype=tf.float32)\n", 412 | " encoder_state = tf.concat(encoder_state, -1)\n", 413 | " encoder_outputs = tf.concat(encoder_outputs, -1)" 414 | ] 415 | }, 416 | { 417 | "cell_type": "markdown", 418 | "metadata": { 419 | "deletable": true, 420 | "editable": true 421 | }, 422 | "source": [ 423 | "## Attention" 424 | ] 425 | }, 426 | { 427 | "cell_type": "code", 428 | "execution_count": null, 429 | "metadata": { 430 | "collapsed": false, 431 | "deletable": true, 432 | "editable": true 433 | }, 434 | "outputs": [], 435 | "source": [ 436 | "attention_mechanism = tf.contrib.seq2seq.LuongAttention(256, encoder_outputs,\n", 437 | " memory_sequence_length=user_len)" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": { 443 | "deletable": true, 444 | "editable": true 445 | }, 446 | "source": [ 447 | "## Projection layer (Output of decoder)" 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": null, 453 | "metadata": { 454 | "collapsed": true, 455 | "deletable": true, 456 | "editable": true 457 | }, 458 | "outputs": [], 459 | "source": [ 460 | "with tf.name_scope(\"projection\"):\n", 461 | " projection_layer = tf.layers.Dense(\n", 462 | " len(vocab), use_bias=False)" 463 | ] 464 | }, 465 | { 466 | "cell_type": "markdown", 467 | "metadata": { 468 | "deletable": true, 469 | "editable": true 470 | }, 471 | "source": [ 472 | "## Decoder" 473 | ] 474 | }, 475 | { 476 | "cell_type": "code", 477 | "execution_count": null, 478 | "metadata": { 479 | "collapsed": false, 480 | "deletable": true, 481 | "editable": true 482 | }, 483 | "outputs": [], 484 | "source": [ 485 | "with tf.name_scope(\"decoder\"):\n", 486 | " decoder_GRU = tf.nn.rnn_cell.GRUCell(256)\n", 487 | " decoder_cell = tf.nn.rnn_cell.DropoutWrapper(decoder_GRU, input_keep_prob=0.7, \n", 488 | " output_keep_prob=0.7, state_keep_prob=0.9)\n", 489 | " decoder_cell = tf.contrib.seq2seq.AttentionWrapper(decoder_cell, attention_mechanism,\n", 490 | " attention_layer_size=128)\n", 491 | " \n", 492 | " decoder_initial_state = decoder_cell.zero_state(BATCH_SIZE, tf.float32).clone(\n", 493 | " cell_state=encoder_state)\n", 494 | " # Helper for use during training\n", 495 | " # During training we feed the decoder\n", 496 | " # the target sequence\n", 497 | " # However, during testing we use the decoder's\n", 498 | " # last output\n", 499 | " helper = tf.contrib.seq2seq.TrainingHelper(\n", 500 | " embedded_bot_inp_dropout, bot_inp_len)\n", 501 | " # Decoder\n", 502 | " decoder = tf.contrib.seq2seq.BasicDecoder(\n", 503 | " decoder_cell, helper, decoder_initial_state,\n", 504 | " output_layer=projection_layer)\n", 505 | " # Dynamic decoding\n", 506 | " outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(decoder)\n", 507 | " logits = outputs.rnn_output\n", 508 | " translations = outputs.sample_id" 509 | ] 510 | }, 511 | { 512 | "cell_type": "markdown", 513 | "metadata": { 514 | "deletable": true, 515 | "editable": true 516 | }, 517 | "source": [ 518 | "## Loss computation normalized by batch size" 519 | ] 520 | }, 521 | { 522 | "cell_type": "code", 523 | "execution_count": null, 524 | "metadata": { 525 | "collapsed": true, 526 | "deletable": true, 527 | "editable": true 528 | }, 529 | "outputs": [], 530 | "source": [ 531 | "with tf.name_scope(\"loss\"):\n", 532 | " loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=tf.reshape(bot_out_doc,\n", 533 | " [-1, MAX_LEN]), logits=logits)\n", 534 | " mask = tf.sequence_mask(bot_out_len, dtype=tf.float32)\n", 535 | " train_loss = (tf.reduce_sum(loss * mask) / BATCH_SIZE)" 536 | ] 537 | }, 538 | { 539 | "cell_type": "markdown", 540 | "metadata": { 541 | "deletable": true, 542 | "editable": true 543 | }, 544 | "source": [ 545 | "## Adam with gradient clipping and learning rate scheduling using cosine decay + restarts" 546 | ] 547 | }, 548 | { 549 | "cell_type": "code", 550 | "execution_count": null, 551 | "metadata": { 552 | "collapsed": true, 553 | "deletable": true, 554 | "editable": true 555 | }, 556 | "outputs": [], 557 | "source": [ 558 | "with tf.variable_scope('Adam'):\n", 559 | " global_step = tf.Variable(0, trainable=False)\n", 560 | " inc_gstep = tf.assign(global_step,global_step + 1)\n", 561 | " learning_rate = tf.train.cosine_decay_restarts(0.001, global_step, 550, t_mul=1.1)\n", 562 | " adam_optimizer = tf.train.AdamOptimizer(learning_rate)\n", 563 | " adam_gradients, v = zip(*adam_optimizer.compute_gradients(train_loss))\n", 564 | " adam_gradients, _ = tf.clip_by_global_norm(adam_gradients, 10.0)\n", 565 | " adam_optimize = adam_optimizer.apply_gradients(zip(adam_gradients, v))" 566 | ] 567 | }, 568 | { 569 | "cell_type": "markdown", 570 | "metadata": { 571 | "deletable": true, 572 | "editable": true 573 | }, 574 | "source": [ 575 | "## Inference nodes" 576 | ] 577 | }, 578 | { 579 | "cell_type": "code", 580 | "execution_count": null, 581 | "metadata": { 582 | "collapsed": false, 583 | "deletable": true, 584 | "editable": true 585 | }, 586 | "outputs": [], 587 | "source": [ 588 | "with tf.variable_scope(\"inference\"):\n", 589 | " # Helper\n", 590 | " # Start token is 1, which is the token\n", 591 | " # End token is 2\n", 592 | " helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(\n", 593 | " embedding,\n", 594 | " tf.fill([BATCH_SIZE], 1), 2)\n", 595 | "\n", 596 | " # Decoder\n", 597 | " decoder = tf.contrib.seq2seq.BasicDecoder(\n", 598 | " decoder_cell, helper, decoder_initial_state,\n", 599 | " output_layer=projection_layer)\n", 600 | " # Dynamic decoding\n", 601 | " test_outputs, _, _ = tf.contrib.seq2seq.dynamic_decode(\n", 602 | " decoder, maximum_iterations=10)\n", 603 | " test_translations = tf.identity(test_outputs.sample_id, name=\"word_ids\")\n", 604 | " test_words = tf.identity(inverse.lookup(tf.cast(test_translations, tf.int64)), name=\"words\")" 605 | ] 606 | }, 607 | { 608 | "cell_type": "markdown", 609 | "metadata": { 610 | "deletable": true, 611 | "editable": true 612 | }, 613 | "source": [ 614 | "## A function for testing" 615 | ] 616 | }, 617 | { 618 | "cell_type": "code", 619 | "execution_count": null, 620 | "metadata": { 621 | "collapsed": false, 622 | "deletable": true, 623 | "editable": true 624 | }, 625 | "outputs": [], 626 | "source": [ 627 | "def testBot(sess):\n", 628 | " text = [\"Hello\"] + [\"\"] * (BATCH_SIZE - 1)\n", 629 | " num_text = len(text)\n", 630 | " text = [process_str(sentence) for sentence in text]\n", 631 | " text_len = np.array([item[1] for item in text]).astype(np.int32)\n", 632 | " text = np.array([item[0] for item in text])\n", 633 | " \n", 634 | " user_test_ph = tf.placeholder(text.dtype)\n", 635 | " user_test_lens_ph = tf.placeholder(text_len.dtype)\n", 636 | " \n", 637 | " tf_user_test = tf.data.Dataset.from_tensor_slices(user_test_ph).map(lambda string: tf.string_split([string])).map(lambda tokens: (words.lookup(tokens)))\n", 638 | " tf_user_test_lens = tf.data.Dataset.from_tensor_slices(user_test_lens_ph)\n", 639 | " \n", 640 | " test_data = tf.data.Dataset.zip((tf_user_test, tf_bot_inp, tf_bot_out,\n", 641 | " tf_user_test_lens, tf_bot_inp_lens, tf_bot_out_lens))\n", 642 | " \n", 643 | " test_data = test_data.batch(num_text).prefetch(1)\n", 644 | " test_init_op = data_iterator.make_initializer(test_data)\n", 645 | " \n", 646 | " sess.run(test_init_op, feed_dict={\n", 647 | " user_test_ph: user,\n", 648 | " bot_inp_ph: bot_inputs[0:num_text],\n", 649 | " bot_out_ph: bot_outputs[0:num_text],\n", 650 | " user_test_lens_ph: user_lens,\n", 651 | " bot_inp_lens_ph: bot_inp_lens[0:num_text],\n", 652 | " bot_out_lens_ph: bot_out_lens[0:num_text]\n", 653 | " })\n", 654 | " translations_text = sess.run(inverse.lookup(tf.cast(test_translations, tf.int64)))\n", 655 | " return translations_text" 656 | ] 657 | }, 658 | { 659 | "cell_type": "code", 660 | "execution_count": null, 661 | "metadata": { 662 | "collapsed": false, 663 | "deletable": true, 664 | "editable": true 665 | }, 666 | "outputs": [], 667 | "source": [ 668 | "with tf.name_scope('summaries'):\n", 669 | " tf.summary.scalar('Loss', train_loss)\n", 670 | " tf.summary.scalar('LR', learning_rate)\n", 671 | " merged = tf.summary.merge_all()\n", 672 | " config = projector.ProjectorConfig()\n", 673 | " embedding_vis = config.embeddings.add()\n", 674 | " embedding_vis.tensor_name = embedding.name\n", 675 | " vocab_str = '\\n'.join(vocab)\n", 676 | " metadata = pd.Series(vocab)\n", 677 | " metadata.name = \"label\"\n", 678 | " metadata.to_csv(\"checkpoints/metadata.tsv\", sep=\"\\t\", header=True, index_label=\"index\")\n", 679 | " embedding_vis.metadata_path = 'metadata.tsv'" 680 | ] 681 | }, 682 | { 683 | "cell_type": "code", 684 | "execution_count": null, 685 | "metadata": { 686 | "collapsed": false, 687 | "deletable": true, 688 | "editable": true 689 | }, 690 | "outputs": [], 691 | "source": [ 692 | "losses = []\n", 693 | "print(\"Started training\")\n", 694 | "\n", 695 | "saver = tf.train.Saver()\n", 696 | "save_dir = 'checkpoints/'\n", 697 | "if not os.path.exists(save_dir):\n", 698 | " os.makedirs(save_dir)\n", 699 | "save_path = os.path.join(save_dir, 'best_validation')\n", 700 | "\n", 701 | "sess = tf.InteractiveSession(config=sconfig)\n", 702 | "\n", 703 | "writer = tf.summary.FileWriter('./checkpoints', sess.graph)\n", 704 | "projector.visualize_embeddings(writer, config)\n", 705 | "\n", 706 | "\n", 707 | "sess.run([words.init, tf.global_variables_initializer(), inverse.init])\n", 708 | "step = 0\n", 709 | "\n", 710 | "for i in range(NUM_EPOCHS):\n", 711 | " if(i % 10 == 0):\n", 712 | " saver.save(sess=sess, save_path=save_path, write_meta_graph=True)\n", 713 | " sess.run(train_init_op, feed_dict={\n", 714 | " user_ph: user,\n", 715 | " bot_inp_ph: bot_inputs,\n", 716 | " bot_out_ph: bot_outputs,\n", 717 | " user_lens_ph: user_lens,\n", 718 | " bot_inp_lens_ph: bot_inp_lens,\n", 719 | " bot_out_lens_ph: bot_out_lens\n", 720 | " })\n", 721 | "\n", 722 | " while True:\n", 723 | " try:\n", 724 | " _, batch_loss, summary = sess.run([adam_optimize, train_loss, merged])\n", 725 | " writer.add_summary(summary, i)\n", 726 | " losses.append(batch_loss)\n", 727 | " except tf.errors.InvalidArgumentError:\n", 728 | " continue\n", 729 | " except tf.errors.OutOfRangeError:\n", 730 | " print(\"Epoch {}: Loss(Mean): {} Loss(Std): {}\".format(i, np.mean(losses), np.std(losses)))\n", 731 | " losses = []\n", 732 | " break\n", 733 | " sess.run(inc_gstep)\n", 734 | " step += 1\n", 735 | " print(testBot(sess)[0])" 736 | ] 737 | }, 738 | { 739 | "cell_type": "code", 740 | "execution_count": null, 741 | "metadata": { 742 | "collapsed": true, 743 | "deletable": true, 744 | "editable": true 745 | }, 746 | "outputs": [], 747 | "source": [] 748 | } 749 | ], 750 | "metadata": { 751 | "anaconda-cloud": {}, 752 | "kernelspec": { 753 | "display_name": "Python 3", 754 | "language": "python", 755 | "name": "python3" 756 | }, 757 | "language_info": { 758 | "codemirror_mode": { 759 | "name": "ipython", 760 | "version": 3 761 | }, 762 | "file_extension": ".py", 763 | "mimetype": "text/x-python", 764 | "name": "python", 765 | "nbconvert_exporter": "python", 766 | "pygments_lexer": "ipython3", 767 | "version": "3.5.5" 768 | } 769 | }, 770 | "nbformat": 4, 771 | "nbformat_minor": 2 772 | } 773 | -------------------------------------------------------------------------------- /Section 6/Notebooks-20190104T075709Z-001.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Interactive-Chatbots-with-TensorFlow-/0e3a665e9a22cb852e7325420af3d0fd37fc7c34/Section 6/Notebooks-20190104T075709Z-001.zip --------------------------------------------------------------------------------