├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Dockerfile ├── LICENSE ├── NOTICE ├── README.md ├── coreinfra_cfn.yml ├── lex_streaming_client.py ├── media ├── image1.jpeg ├── image2.png ├── image2.tif ├── image3.png ├── image3.tif ├── image4.png ├── image5.png ├── image6.png ├── image7.png ├── image8.png ├── image8.tif ├── image9.png └── image9.tif ├── requirements.txt ├── server.py ├── serviceinfra_cfn.yml ├── templates └── streams.xml └── voice_and_silence_detecting_lex_wrapper.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | .DS_Store 106 | .vscode/ 107 | 108 | .DS_Store 109 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:18.04 2 | 3 | RUN apt-get update -y && \ 4 | apt-get install -y python3 python3-dev python3-pip 5 | 6 | COPY ./requirements.txt /app/requirements.txt 7 | 8 | WORKDIR /app 9 | 10 | RUN pip3 install -r requirements.txt 11 | 12 | COPY . /app 13 | 14 | ENTRYPOINT [ "python3" ] 15 | 16 | CMD [ "server.py" ] 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Use Amazon Lex as a conversational interface with Twilio Media Streams 2 | 3 | Businesses use the Twilio platform to build the forms of communication 4 | they need to best serve their customers: whether it’s fully automating 5 | food orders of a restaurant with a conversational IVR or building the 6 | next generation advanced contact center. With the launch of [Media Streams](https://www.twilio.com/media-streams), Twilio is opening up their Voice 7 | platform by providing businesses access to the raw audio stream of their 8 | phone calls in real time. 9 | 10 | You can use Media Streams to increase productivity in the call center by 11 | transcribing speech in real time with [Amazon 12 | Transcribe Streaming 13 | WebSockets](https://aws.amazon.com/blogs/aws/amazon-transcribe-streaming-now-supports-websockets/) 14 | or to automate end-user interactions and make recommendations to agents 15 | based on the caller’s intent using Amazon Lex. 16 | 17 | In this blog post, we show you how to use [Amazon 18 | Lex](https://aws.amazon.com/lex/) to integrate conversational interfaces 19 | (chatbots) to your voice application. This enables you to recognize the 20 | intent of speech and build highly engaging user experiences and lifelike 21 | conversations. We use Twilio Media Streams to get access to the raw 22 | audio stream from a phone call in near real time. 23 | 24 | The solution follows these steps: 25 | 26 | 1. Receive audio stream from Twilio 27 | 28 | 2. Send the audio stream to a voice activity detection component to 29 | determine voice in audio 30 | 31 | 3. Start streaming the user data to Amazon Lex when voice is detected 32 | 33 | 4. Stop streaming the user data to Amazon Lex when silence is detected 34 | 35 | 5. Update the ongoing Twilio call based on the response from Amazon Lex 36 | 37 | The Voice activity detection (VAD) implementation provided in this 38 | sample is for reference/demo purpose only and uses a rudimentary 39 | approach to detect voice and silence by looking at amplitude. It is not 40 | recommended for production use. You will need to implement a robust form 41 | of VAD module as per your need for use in production scenarios. 42 | 43 | The diagram below describes the steps: 44 | 45 | ![](./media/image1.jpeg) 46 | 47 | The instructions for integrating an Amazon Lex Bot with the Twilio Voice 48 | Stream are provided in the following steps: 49 | 50 | > Step 1: Create an Amazon Lex Bot 51 | > 52 | > Step 2: Create a Twilio Account and Setup 53 | > Programmable Voice 54 | > 55 | > Step 3: Build and Deploy the Amazon Lex and 56 | > Twilio Voice Stream Integration code to Amazon ECS/Fargate 57 | > 58 | > Step 4: Test the deployed service 59 | > 60 | > As an optional next step, you can build and test the service locally. 61 | > For instructions see, Step 5(Optional): Build and Test the service 62 | > locally 63 | 64 | To build and deploy the service, the following pre-requisites are 65 | needed: 66 | 67 | 1. [Python ](https://www.python.org/downloads/)(The language used to 68 | build the service) 69 | 70 | 2. [Docker](https://www.docker.com/products/docker-desktop) (The tool 71 | used for packaging the service for deployment) 72 | 73 | 3. [AWS CLI](https://aws.amazon.com/cli/) installed and configured (For 74 | creating the required AWS services and deploying the service to 75 | AWS). For instructions see, [Configuring AWS 76 | CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration) 77 | 78 | In addition, you need a domain name for hosting your service and you 79 | must register an SSL certificate for the domain using [Amazon 80 | Certificate Manager](https://console.aws.amazon.com/acm/home). For 81 | instructions, see [Request a public 82 | Certificate](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html). 83 | Record the Certificate ARN from the console. 84 | 85 | An SSL certificate is needed to communicate securely over **wss** 86 | (WebSocket Secure), a persistent bidirectional communication protocol 87 | used by the Twilio voice stream. The \ instruction in the 88 | templates/streams.xml file allows you to receive raw audio streams from 89 | a live phone call over WebSockets in near real-time. On successful 90 | connection, a WebSocket connection to the service is established and 91 | audio will start streaming. 92 | 93 | # Step 1: Create an Amazon Lex Bot 94 | 95 | If you don’t already have an Amazon Lex Bot, create and deploy one. For 96 | instructions, see [Create an Amazon Lex Bot 97 | Using a Blueprint 98 | (Console)](https://docs.aws.amazon.com/lex/latest/dg/gs-bp.html). 99 | 100 | Once you’ve created the bot, deploy the bot and create an alias. For 101 | instructions, see [Publish a Version and Create 102 | an 103 | Alias](https://docs.aws.amazon.com/lex/latest/dg/gettingstarted-ex3.html). 104 | 105 | In order to call the Amazon Lex APIs from the service, you must create 106 | an IAM user with an access type “Programmatic Access” and attach the 107 | appropriate policies. 108 | 109 | For this, in the AWS Console, go to IAM-\>Users-\>Add user 110 | 111 | Provide a user name, select “Programmatic access” Access type, then 112 | click on “Next: Permissions” 113 | 114 | ![](./media/image2.png) 115 | 116 | Using the “Attach existing policies directly” option, filter for Amazon 117 | Lex policies and select AmazonLexReadOnly and AmazonLexRunBotsOnly 118 | policies. 119 | 120 | ![](./media/image3.png) 121 | 122 | Click “Next: Tags”, “Next: Review”, and “Create User” in the pages that 123 | follow to create the user. Record the access key ID and the secret 124 | access key. We use these credentials during the deployment of the stack. 125 | 126 | # Step 2: Create a Twilio account and setup programmable voice 127 | 128 | Sign up for a Twilio account and create a programmable voice project. 129 | 130 | For sign-up instructions, see **.** 131 | 132 | Record the “**AUTH TOKEN”.** You can find this information on the Twilio 133 | dashboard under Settings-\>General-\>API Credentials. 134 | 135 | You must also verify the caller ID by adding the phone number that you 136 | are using to make calls to the Twilio phone number. You can do this by 137 | clicking on the ![](./media/image4.png) button on the Verify caller IDs 138 | page. 139 | 140 | # 141 | 142 | # Step 3: Build and deploy the Amazon Lex and Twilio Stream Integration code to Amazon ECS 143 | 144 | In this section, we create a new service using AWS Fargate to host the 145 | integration code. [AWS 146 | Fargate](https://aws.amazon.com/fargate/) is a deployment option 147 | in [Amazon Elastic Container 148 | Service](https://aws.amazon.com/ecs/) (ECS) that allows you to 149 | deploy containers without worrying about provisioning or scaling 150 | servers. For our service, we use Python and 151 | [Flask](https://palletsprojects.com/p/flask/) 152 | in a Docker container behind an Application Load Balancer (ALB). 153 | 154 | Deploy the core infrastructure 155 | 156 | As the first step in creating the infrastructure, we deploy the core 157 | infrastructure components such as VPC, Subnets, Security Groups, ALB, 158 | ECS cluster, and IAM policies using a CloudFormation Template. 159 | 160 | Clicking on the “Launch Stack” button below takes you to the AWS 161 | CloudFormation Stack creation page. Click “Next” and fill in the 162 | parameters. Please note that you will be using the same 163 | “EnvironmentName” parameter later in the process where we will be 164 | launching the service on top of the core infrastructure. This allows us 165 | to reference the stack outputs from this 166 | deployment. 167 | 168 | [![Launch Stack](./media/image5.png)](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=lex-twiliovoice-core&templateURL=https://aws-ml-blog.s3.amazonaws.com/artifacts/lex-twilio/coreinfra_cfn.yml) 169 | 170 | 171 | Once the stack creation is complete, from the “outputs” tab, record the 172 | value of the “ExternalUrl” key. 173 | 174 | Package and deploy the code to AWS 175 | 176 | In order to deploy the code to Amazon ECS, we package the code in a 177 | Docker container and upload the Docker image to the Amazon Elastic 178 | Container Registry (ECR). 179 | 180 | The code for the service is available at the GitHub repository below. 181 | Clone the repository on your local machine. 182 | 183 | git clone https://github.com/aws-samples/amazon-lex-conversational-interface-for-twilio.git 184 | 185 | cd amazon-lex-conversational-interface-for-twilio 186 | 187 | Next, we update the URL for the Streams element inside 188 | templates/streams.xml to match the DNS name for your service that you 189 | configured with the SSL certificate in the pre-requisites section. 190 | 191 | \\ 192 | 193 | Now, run the following command to build the container image using the 194 | Dockerfile. 195 | 196 | docker build -t lex-twiliovoice . 197 | 198 | Next, we create the container registry using the AWS CLI by passing in 199 | the value for the repository name. Record the “repositoryUri” from the 200 | output. 201 | 202 | aws ecr create-repository --repository-name \ 203 | 204 | In order to push the container image to the registry, we must 205 | authenticate. Run the following command: 206 | 207 | aws ecr get-login --region us-west-2 --no-include-email 208 | 209 | Execute the output of the above command to complete the authentication 210 | process. 211 | 212 | Next, we tag and push the container image to ECR. 213 | 214 | docker tag lex-twiliovoice \/lex-twiliovoice:latest 215 | 216 | docker push \/lex-twiliovoice:latest 217 | 218 | We now deploy the rest of the infrastructure using a CloudFormation 219 | template. As part of this stack, we deploy components such as ECS 220 | Service, ALB Target groups, HTTP/HTTPS Listener rules, and Fargate Task. 221 | The environment variables are injected into the container using the task 222 | definition properties. 223 | 224 | Since we are working with WebSocket connections in our service, we 225 | enable stickiness with our load balancer using the target group 226 | attribute to allow for persistent connection with the same instance. 227 | 228 | TargetGroup: 229 | 230 | Type: AWS::ElasticLoadBalancingV2::TargetGroup 231 | 232 | Properties: 233 | 234 | …. 235 | 236 | …. 237 | 238 | TargetGroupAttributes: 239 | 240 | \- Key: **stickiness.enabled** 241 | 242 | Value: true 243 | 244 | … 245 | 246 | Clicking on the “Launch Stack” button below takes you to the AWS 247 | CloudFormation Stack creation page. Click “Next” and fill in the correct 248 | values for the following parameters that are collected from the previous 249 | steps. 250 | 251 | IAMAccessKeyId, IAMSecretAccessKey, ImageUrl, LexBotName, LexBotAlias, 252 | and TwilioAuthToken. You can use default values for all the other 253 | parameters. Make sure to use the same “EnvironmentName” from the 254 | previous stack deployment since we are referring to the outputs of that 255 | deployment. 256 | 257 | [![Launch Stack](./media/image5.png)](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=lex-twiliovoice-service&templateURL=https://aws-ml-blog.s3.amazonaws.com/artifacts/lex-twilio/serviceinfra_cfn.yml) 258 | 259 | Once the deployment is complete, we can test the service. However, 260 | before we do that, make sure to point your custom DNS to the Application 261 | Load Balancer URL. 262 | 263 | To do that, we create an “A Record” under Route 53, Hosted Zones to 264 | point your custom DNS to the ALB Url that was part of the core 265 | infrastructure stack deployment (“ExternalUrl” key from output). In the 266 | “Create Record Set” screen, in the name field use your DNS name, for 267 | type select “A - IPv4 address”, select “Yes” for Alias field, select the 268 | Alias target as the ALB Url, and click “Create”. 269 | 270 | # Step 4: Test the deployed service 271 | 272 | You can verify the deployment by navigating to the [Amazon ECS 273 | Console](https://console.aws.amazon.com/ecs/home) and clicking on the 274 | cluster name. You can see the AWS Fargate service under the “services” 275 | tab and the running task under the “tasks” tab. 276 | 277 | ![](./media/image6.png) 278 | 279 | To test the service, we will first update the Webhook url field under 280 | the “Voice & Fax” section in the Twilio console with the URL of the 281 | service that is running in AWS (http://\/twiml). You can now call 282 | the Twilio phone number to reach the Lex Bot. Make sure that the number 283 | you are calling from is verified using the Twilio console. Once 284 | connected, you will hear the prompt “You will be interacting with Lex 285 | bot in 3, 2, 1. Go.” that is configured in the templates/streams.xml 286 | file. You can now interact with the Amazon Lex bot. 287 | 288 | You can monitor the service using the “CloudWatch Log Groups” and 289 | troubleshoot any issues that may arise while the service is running. 290 | 291 | ![](./media/image7.png) 292 | 293 | # Step 5(Optional): Build and test the service locally 294 | 295 | Now that the service is deployed and tested, you may be interested in 296 | building and testing the code locally. For this, navigate to the cloned 297 | GitHub repository on your local machine and install all the dependencies 298 | using the following command: 299 | 300 | pip install -r requirements.txt 301 | 302 | You can test the service locally by installing “ngrok”. See 303 | for more details. This tool provides public 304 | URLs for exposing the local web server. Using this, you can test the 305 | Twilio webhook integration. 306 | 307 | Start the ngrok process by using the following command in another 308 | terminal window. The ngrok.io url can be used to access the web service 309 | from external applications. 310 | 311 | ngrok http 8080 312 | 313 | ![](./media/image8.png) 314 | 315 | Next, configure the “Stream” element inside the templates/streams.xml 316 | file with the correct ngrok url. 317 | 318 | \\ 319 | 320 | In addition, we also need to configure the environment variables used in 321 | the code. Run the following command after providing appropriate values 322 | for the environment variables: 323 | 324 | export AWS\_REGION=us-west-2 325 | 326 | export ACCESS\_KEY\_ID=\ 327 | 328 | export SECRET\_ACCESS\_KEY=\ 330 | 331 | export LEX\_BOT\_NAME=\ 332 | 333 | export LEX\_BOT\_ALIAS=\ 335 | 336 | export TWILIO\_AUTH\_TOKEN=\ 337 | 338 | export CONTAINER\_PORT=8080 339 | 340 | export URL=\ (update with the appropriate url 341 | from ngrok) 342 | 343 | Once the variables are set, you can start the service using the 344 | following command: 345 | 346 | python server.py 347 | 348 | To test, configure the Webhook field under “Voice & Fax” in the Twilio 349 | console with the correct url (http://\/twiml) as shown below. 350 | 351 | ![](./media/image9.png) 352 | 353 | Initiate a call to the Twilio phone number from a verified phone. Once 354 | connected, you hear the prompt “You will be interacting with Lex bot in 355 | 3, 2, 1. Go.” that is configured in the templates/streams.xml file. You 356 | are now able to interact with the Amazon Lex bot that you created in 357 | Step1. 358 | 359 | In this blog post, we showed you how to use 360 | [Amazon Lex](https://aws.amazon.com/lex/) 361 | to integrate your chatbot to your voice application. To learn how to 362 | build more with Amazon Lex, check out the 363 | [developer 364 | resources](https://aws.amazon.com/lex/resources/). 365 | 366 | 367 | ## License 368 | 369 | This project is licensed under the Apache-2.0 License. 370 | 371 | -------------------------------------------------------------------------------- /coreinfra_cfn.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: The core infrastructure resources used to create an ECS cluster 3 | for the lex/twilio voice integration service 4 | Parameters: 5 | EnvironmentName: 6 | Type: String 7 | Default: lex-twiliovoice-stage 8 | Description: Environment name. You will use the same name in the second part of the deployment to refer to the outputs 9 | CertificateArn: 10 | Type: String 11 | Description: ARN of the Amazon Certificate Manager SSL certificate for your domain to use for this service 12 | 13 | Mappings: 14 | SubnetConfig: 15 | VPC: 16 | CIDR: '10.0.0.0/16' 17 | PublicOne: 18 | CIDR: '10.0.0.0/24' 19 | PublicTwo: 20 | CIDR: '10.0.1.0/24' 21 | 22 | Resources: 23 | VPC: 24 | Type: AWS::EC2::VPC 25 | Properties: 26 | EnableDnsSupport: true 27 | EnableDnsHostnames: true 28 | CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR'] 29 | PublicSubnetOne: 30 | Type: AWS::EC2::Subnet 31 | Properties: 32 | AvailabilityZone: 33 | Fn::Select: 34 | - 0 35 | - Fn::GetAZs: {Ref: 'AWS::Region'} 36 | VpcId: !Ref 'VPC' 37 | CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR'] 38 | MapPublicIpOnLaunch: true 39 | PublicSubnetTwo: 40 | Type: AWS::EC2::Subnet 41 | Properties: 42 | AvailabilityZone: 43 | Fn::Select: 44 | - 1 45 | - Fn::GetAZs: {Ref: 'AWS::Region'} 46 | VpcId: !Ref 'VPC' 47 | CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR'] 48 | MapPublicIpOnLaunch: true 49 | InternetGateway: 50 | Type: AWS::EC2::InternetGateway 51 | GatewayAttachement: 52 | Type: AWS::EC2::VPCGatewayAttachment 53 | Properties: 54 | VpcId: !Ref 'VPC' 55 | InternetGatewayId: !Ref 'InternetGateway' 56 | PublicRouteTable: 57 | Type: AWS::EC2::RouteTable 58 | Properties: 59 | VpcId: !Ref 'VPC' 60 | PublicRoute: 61 | Type: AWS::EC2::Route 62 | DependsOn: GatewayAttachement 63 | Properties: 64 | RouteTableId: !Ref 'PublicRouteTable' 65 | DestinationCidrBlock: '0.0.0.0/0' 66 | GatewayId: !Ref 'InternetGateway' 67 | PublicSubnetOneRouteTableAssociation: 68 | Type: AWS::EC2::SubnetRouteTableAssociation 69 | Properties: 70 | SubnetId: !Ref PublicSubnetOne 71 | RouteTableId: !Ref PublicRouteTable 72 | PublicSubnetTwoRouteTableAssociation: 73 | Type: AWS::EC2::SubnetRouteTableAssociation 74 | Properties: 75 | SubnetId: !Ref PublicSubnetTwo 76 | RouteTableId: !Ref PublicRouteTable 77 | ECSCluster: 78 | Type: AWS::ECS::Cluster 79 | FargateContainerSecurityGroup: 80 | Type: AWS::EC2::SecurityGroup 81 | Properties: 82 | GroupDescription: Access to the Fargate containers 83 | VpcId: !Ref 'VPC' 84 | EcsSecurityGroupIngressFromPublicALB: 85 | Type: AWS::EC2::SecurityGroupIngress 86 | Properties: 87 | Description: Ingress from the public ALB 88 | GroupId: !Ref 'FargateContainerSecurityGroup' 89 | IpProtocol: -1 90 | SourceSecurityGroupId: !Ref 'PublicLoadBalancerSG' 91 | EcsSecurityGroupIngressFromSelf: 92 | Type: AWS::EC2::SecurityGroupIngress 93 | Properties: 94 | Description: Ingress from other containers in the same security group 95 | GroupId: !Ref 'FargateContainerSecurityGroup' 96 | IpProtocol: -1 97 | SourceSecurityGroupId: !Ref 'FargateContainerSecurityGroup' 98 | PublicLoadBalancerSG: 99 | Type: AWS::EC2::SecurityGroup 100 | Properties: 101 | GroupDescription: Access to the public facing load balancer 102 | VpcId: !Ref 'VPC' 103 | SecurityGroupIngress: 104 | - CidrIp: 0.0.0.0/0 105 | IpProtocol: -1 106 | PublicLoadBalancer: 107 | Type: AWS::ElasticLoadBalancingV2::LoadBalancer 108 | Properties: 109 | Scheme: internet-facing 110 | LoadBalancerAttributes: 111 | - Key: idle_timeout.timeout_seconds 112 | Value: '30' 113 | Subnets: 114 | - !Ref PublicSubnetOne 115 | - !Ref PublicSubnetTwo 116 | SecurityGroups: [!Ref 'PublicLoadBalancerSG'] 117 | DummyTargetGroupPublic: 118 | Type: AWS::ElasticLoadBalancingV2::TargetGroup 119 | Properties: 120 | HealthCheckIntervalSeconds: 6 121 | HealthCheckPath: / 122 | HealthCheckProtocol: HTTP 123 | HealthCheckTimeoutSeconds: 5 124 | HealthyThresholdCount: 2 125 | Name: !Join ['-', [!Ref 'EnvironmentName', 'temp']] 126 | Port: 80 127 | Protocol: HTTP 128 | UnhealthyThresholdCount: 2 129 | VpcId: !Ref 'VPC' 130 | PublicLoadBalancerListenerHTTP: 131 | Type: AWS::ElasticLoadBalancingV2::Listener 132 | DependsOn: 133 | - PublicLoadBalancer 134 | Properties: 135 | DefaultActions: 136 | - TargetGroupArn: !Ref 'DummyTargetGroupPublic' 137 | Type: 'forward' 138 | LoadBalancerArn: !Ref 'PublicLoadBalancer' 139 | Port: 80 140 | Protocol: HTTP 141 | PublicLoadBalancerListenerHTTPS: 142 | Type: AWS::ElasticLoadBalancingV2::Listener 143 | DependsOn: 144 | - PublicLoadBalancer 145 | Properties: 146 | DefaultActions: 147 | - TargetGroupArn: !Ref 'DummyTargetGroupPublic' 148 | Type: 'forward' 149 | LoadBalancerArn: !Ref 'PublicLoadBalancer' 150 | Port: 443 151 | Protocol: HTTPS 152 | Certificates: 153 | - CertificateArn: !Ref 'CertificateArn' 154 | # IAM Roles 155 | LexTwilioECSTaskExecutionRolePolicy: 156 | Type: AWS::IAM::ManagedPolicy 157 | Properties: 158 | PolicyDocument: 159 | Version: '2012-10-17' 160 | Statement: 161 | - Effect: Allow 162 | Action: 163 | # ECR 164 | - 'ecr:GetAuthorizationToken' 165 | - 'ecr:BatchCheckLayerAvailability' 166 | - 'ecr:GetDownloadUrlForLayer' 167 | - 'ecr:BatchGetImage' 168 | # CW 169 | - 'logs:CreateLogStream' 170 | - 'logs:PutLogEvents' 171 | Resource: '*' 172 | ECSTaskExecutionRole: 173 | Type: AWS::IAM::Role 174 | Properties: 175 | AssumeRolePolicyDocument: 176 | Version: '2012-10-17' 177 | Statement: 178 | - Effect: Allow 179 | Principal: 180 | Service: [ecs-tasks.amazonaws.com] 181 | Action: ['sts:AssumeRole'] 182 | Path: / 183 | ManagedPolicyArns: 184 | - !Ref LexTwilioECSTaskExecutionRolePolicy 185 | Outputs: 186 | ClusterName: 187 | Description: The name of the ECS cluster 188 | Value: !Ref 'ECSCluster' 189 | Export: 190 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'ClusterName' ] ] 191 | ExternalUrl: 192 | Description: The url of the ALB 193 | Value: !Join ['', ['http://', !GetAtt 'PublicLoadBalancer.DNSName']] 194 | Export: 195 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'ExternalUrl' ] ] 196 | ECSTaskExecutionRole: 197 | Description: The ARN of the ECS Task execurtion role 198 | Value: !GetAtt 'ECSTaskExecutionRole.Arn' 199 | Export: 200 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'ECSTaskExecutionRole' ] ] 201 | HTTPListener: 202 | Description: The ARN of the public load balancer's HTTP Listener 203 | Value: !Ref PublicLoadBalancerListenerHTTP 204 | Export: 205 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'PublicListenerHTTP' ] ] 206 | HTTPSListener: 207 | Description: The ARN of the public load balancer's HTTPS Listener 208 | Value: !Ref PublicLoadBalancerListenerHTTPS 209 | Export: 210 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'PublicListenerHTTPS' ] ] 211 | VPCId: 212 | Description: The ID of the VPC that this stack is deployed in 213 | Value: !Ref 'VPC' 214 | Export: 215 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'VPCId' ] ] 216 | PublicSubnetOne: 217 | Description: Public subnet 1 218 | Value: !Ref 'PublicSubnetOne' 219 | Export: 220 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'PublicSubnetOne' ] ] 221 | PublicSubnetTwo: 222 | Description: Public subnet 2 223 | Value: !Ref 'PublicSubnetTwo' 224 | Export: 225 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'PublicSubnetTwo' ] ] 226 | FargateContainerSecurityGroup: 227 | Description: A security group used to allow containers to receive traffic 228 | Value: !Ref 'FargateContainerSecurityGroup' 229 | Export: 230 | Name: !Join [ ':', [ !Ref 'EnvironmentName', 'FargateContainerSecurityGroup' ] ] -------------------------------------------------------------------------------- /lex_streaming_client.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module contains functionality to stream data to Amazon Lex 3 | 4 | The LexClientStreaming runs in its own thread. Class calls post content on Lex 5 | and uses chunked upload of data by passing in data iterator when creating the POST 6 | connection to server. Clients of this class can keep adding to data by calling 7 | add_to_stream(). Once done, clients need to call stop(), at which point the 8 | iterator finally finishes. Iterator checks if there is more data to send or not 9 | by periodically looking at the data input queue. 10 | 11 | Todo: 12 | * TBD 13 | """ 14 | 15 | import datetime 16 | import hashlib 17 | import hmac 18 | import logging 19 | import threading 20 | import time 21 | import requests 22 | import os 23 | 24 | class LexClientStreaming: 25 | AUDIO_CONTENT_TYPE = 'audio/lpcm; sample-rate=8000; sample-size-bits=16; channel-count=1; is-big-endian=false' 26 | TEXT_CONTENT_TYPE = 'text/plain; charset=utf-8' 27 | 28 | lex_config = { 29 | "AccessKeyId": os.environ.get('ACCESS_KEY_ID'), 30 | "SecretAccessKey": os.environ.get('SECRET_ACCESS_KEY'), 31 | "Region": os.environ.get('AWS_REGION'), 32 | "BotName": os.environ.get('LEX_BOT_NAME'), 33 | "BotAlias": os.environ.get('LEX_BOT_ALIAS') 34 | } 35 | 36 | def __init__(self, user_id, content_type=AUDIO_CONTENT_TYPE, stage="lex"): 37 | self.logger = logging.getLogger(__name__) 38 | self.region = self.lex_config["Region"] 39 | self.access_key = self.lex_config["AccessKeyId"] 40 | self.secret_key = self.lex_config["SecretAccessKey"] 41 | self.bot_name = self.lex_config["BotName"] 42 | self.bot_alias = self.lex_config["BotAlias"] 43 | self.host_name = "runtime.lex.{0}.amazonaws.com".format(self.region) 44 | self.endpoint = "https://runtime.lex.{0}.amazonaws.com".format(self.region) 45 | self.service = "lex" 46 | self.data = [] 47 | self.data_index = 0 48 | self.close_stream = False 49 | self.user_id = user_id 50 | self.lex_user_id = "{0}_{1}".format(stage, user_id) 51 | self.content_type = content_type 52 | self.response = None 53 | self.crashed = False 54 | self.request_thread = None 55 | 56 | # add data if stream is not closed. no-op otherwise 57 | def add_to_stream(self, data): 58 | 59 | # start a connection first time we see that data is added to stream 60 | if self.request_thread is None: 61 | self.request_thread = threading.Thread(target=self.run) 62 | self.request_thread.start() 63 | 64 | # python operation to append data to new list seems thread safe 65 | # http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm 66 | if self.close_stream is False: 67 | self.data.append(data) 68 | 69 | def is_alive(self): 70 | return self.request_thread is not None and self.request_thread.is_alive() 71 | 72 | # stop the stream and wait for this thread to finish. 73 | def stop(self): 74 | self.logger.debug("closing lex streaming client") 75 | self.close_stream = True 76 | if self.request_thread is not None: 77 | self.logger.debug("waiting for lex connection thread to stop") 78 | self.request_thread.join() 79 | self.logger.debug("lex connection thread stopped") 80 | 81 | # check (every X milliseconds) and return new chunk if there is data added to stream 82 | def stream_iterator(self): 83 | while not self.close_stream: 84 | if self.data_index < len(self.data): 85 | if self.content_type == LexClientStreaming.TEXT_CONTENT_TYPE: 86 | yield str.encode(self.data[self.data_index]) 87 | else: 88 | yield self.data[self.data_index] 89 | 90 | self.data_index = self.data_index + 1 91 | else: 92 | # sleeping for 100 ms to check if the stream is closed or not 93 | time.sleep(0.1) 94 | 95 | # check if anything is left to send as loop could have stopped if stream got closed and chunks have not yet been 96 | # sent. 97 | 98 | while self.data_index < len(self.data): 99 | if self.content_type == LexClientStreaming.TEXT_CONTENT_TYPE: 100 | yield str.encode(self.data[self.data_index]) 101 | else: 102 | yield self.data[self.data_index] 103 | self.data_index = self.data_index + 1 104 | 105 | def is_crashed(self): 106 | return self.crashed 107 | 108 | def run(self): 109 | try: 110 | self.__run() 111 | except Exception as e: 112 | self.logger.exception(e) 113 | self.crashed = True 114 | 115 | def __run(self): 116 | headers = {} 117 | user_id = self.lex_user_id 118 | content_type = self.content_type 119 | data_type = "content" 120 | payload_hash = "UNSIGNED-PAYLOAD" 121 | headers['x-amz-content-sha256'] = "UNSIGNED-PAYLOAD" 122 | 123 | t = datetime.datetime.utcnow() 124 | amz_date = t.strftime('%Y%m%dT%H%M%SZ') # '20170714T010101Z' 125 | date_stamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope '20170714' 126 | 127 | # ************* TASK 1: CREATE A CANONICAL REQUEST ************* 128 | # http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html 129 | 130 | # Step 1 is to define the verb (GET, POST, etc.) 131 | verb = "POST" 132 | 133 | # Step 2: Create canonical URI--the part of the URI from domain to query 134 | # string (use '/' if no path) 135 | canonical_uri = "/bot/{0}/alias/{1}/user/{2}/{3}".format(self.bot_name, self.bot_alias, user_id, data_type) 136 | 137 | # Step 3: Create the canonical query string. In this example, request 138 | # parameters are passed in the body of the request and the query string 139 | # is blank. 140 | canonical_query_string = "" 141 | 142 | # Step 4: Create the canonical headers. Header names must be trimmed 143 | # and lowercase, and sorted in code point order from low to high. 144 | # Note that there is a trailing \n. 145 | canonical_headers = 'content-type:' + content_type + '\n' + 'host:' + self.host_name + '\n' + 'x-amz-date:' + amz_date + '\n' 146 | 147 | # Step 5: Create the list of signed headers. This lists the headers 148 | # in the canonical_headers list, delimited with ";" and in alpha order. 149 | # Note: The request can include any headers; canonical_headers and 150 | # signed_headers include those that you want to be included in the 151 | # hash of the request. "Host" and "x-amz-date" are always required. 152 | # For Lex, content-type and x-amz-target are also required. 153 | signed_headers = 'content-type;host;x-amz-date' 154 | 155 | # Step 6: Create payload hash. In this example, the payload (body of 156 | # the request) contains the request parameters. 157 | 158 | # overwrite payload hash if it is post text call 159 | # if data_type == "text": 160 | # payload_hash = hashlib.sha256(request_parameters.encode('utf-8')).hexdigest() 161 | 162 | # Step 7: Combine elements to create create canonical request 163 | canonical_request = verb + '\n' + canonical_uri + '\n' + canonical_query_string + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash 164 | 165 | # ************* TASK 2: CREATE THE STRING TO SIGN ************* 166 | # Match the algorithm to the hashing algorithm you use, either SHA-1 or 167 | # SHA-256 (recommended) 168 | algorithm = 'AWS4-HMAC-SHA256' 169 | credential_scope = date_stamp + '/' + self.region + '/' + self.service + '/' + 'aws4_request' 170 | string_to_sign = algorithm + '\n' + amz_date + '\n' + credential_scope + '\n' + hashlib.sha256( 171 | canonical_request.encode('utf-8')).hexdigest() 172 | 173 | # ************* TASK 3: CALCULATE THE SIGNATURE ************* 174 | # Create the signing key using the function defined above. 175 | signing_key = self.__get_signature_key(self.secret_key, date_stamp, self.region, self.service) 176 | 177 | # Sign the string_to_sign using the signing_key 178 | signature = hmac.new(signing_key, string_to_sign.encode('utf-8'), hashlib.sha256).hexdigest() 179 | 180 | # ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST ************* 181 | # Put the signature information in a header named Authorization. 182 | authorization_header = algorithm + ' ' \ 183 | + 'Credential=' + self.access_key + '/' + credential_scope + ', ' \ 184 | + 'SignedHeaders=' + signed_headers + ', ' \ 185 | + 'Signature=' + signature 186 | 187 | # For Lex, the request can include any headers, but MUST include "host", "x-amz-date", 188 | # "x-amz-target", "content-type", and "Authorization". Except for the authorization 189 | # header, the headers must be included in the canonical_headers and signed_headers values, as 190 | # noted earlier. Order here is not significant. 191 | # Python note: The 'host' header is added automatically by the Python 'requests' library. 192 | headers['Content-Type'] = content_type 193 | headers['X-Amz-Date'] = amz_date 194 | headers['Authorization'] = authorization_header 195 | 196 | # ************* SEND THE REQUEST ************* 197 | self.logger.debug("Calling Lex to stream data, endpoint: %s", self.endpoint) 198 | self.response = requests.post(self.endpoint + canonical_uri, data=self.stream_iterator(), headers=headers) 199 | self.logger.info("Lex response headers %s ", self.response.headers) 200 | 201 | # Key derivation functions. 202 | # See: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python 203 | @staticmethod 204 | def __sign(key, msg): 205 | return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest() 206 | 207 | def __get_signature_key(self, key, date_stamp, region_name, service_name): 208 | 209 | k_date = self.__sign(('AWS4' + key).encode('utf-8'), date_stamp) 210 | k_region = self.__sign(k_date, region_name) 211 | k_service = self.__sign(k_region, service_name) 212 | k_signing = self.__sign(k_service, 'aws4_request') 213 | return k_signing 214 | 215 | def get_response(self): 216 | 217 | if self.response is None: 218 | raise Exception("Cannot normalize response as there is no response from lex yet. check if add_to_stream() has been called.") 219 | 220 | if self.response.status_code != 200: 221 | raise Exception("Cannot normalize response as call to Lex did not end with status code 200") 222 | 223 | return {"DialogState":self.response.headers.get("x-amz-lex-dialog-state"), 224 | "Message":self.response.headers.get("x-amz-lex-message"), 225 | "Utterance":self.response.headers.get("x-amz-lex-input-transcript"), 226 | "LexRequestId":self.response.headers.get("x-amzn-RequestId"), 227 | "IntentName": self.response.headers.get("x-amz-lex-intent-name")} 228 | -------------------------------------------------------------------------------- /media/image1.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image1.jpeg -------------------------------------------------------------------------------- /media/image2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image2.png -------------------------------------------------------------------------------- /media/image2.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image2.tif -------------------------------------------------------------------------------- /media/image3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image3.png -------------------------------------------------------------------------------- /media/image3.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image3.tif -------------------------------------------------------------------------------- /media/image4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image4.png -------------------------------------------------------------------------------- /media/image5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image5.png -------------------------------------------------------------------------------- /media/image6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image6.png -------------------------------------------------------------------------------- /media/image7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image7.png -------------------------------------------------------------------------------- /media/image8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image8.png -------------------------------------------------------------------------------- /media/image8.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image8.tif -------------------------------------------------------------------------------- /media/image9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image9.png -------------------------------------------------------------------------------- /media/image9.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-lex-conversational-interface-for-twilio/6aa0488390da95c7fa47daa1dbb431e6925ed70d/media/image9.tif -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Flask==1.0.2 2 | Flask-Sockets==0.2.1 3 | requests 4 | twilio 5 | 6 | -------------------------------------------------------------------------------- /server.py: -------------------------------------------------------------------------------- 1 | from flask import Flask, render_template, request, render_template_string, jsonify 2 | from flask_sockets import Sockets 3 | 4 | import os 5 | import json 6 | import uuid 7 | import logging 8 | import threading 9 | from twilio.twiml.voice_response import VoiceResponse 10 | from twilio.rest import Client 11 | from voice_and_silence_detecting_lex_wrapper import VoiceAndSilenceDetectingLexClient 12 | 13 | HTTP_SERVER_PORT = int(os.environ.get('CONTAINER_PORT')) 14 | 15 | app = Flask(__name__) 16 | sockets = Sockets(app) 17 | 18 | updated_twimls = {} 19 | 20 | def log(msg, *args): 21 | print("Media WS: ", msg, *args) 22 | 23 | @app.route("/ping") 24 | def healthCheckResponse(): 25 | return jsonify({"message" : "echo...health check...."}) 26 | 27 | @app.route('/updatecall', methods=['POST']) 28 | def returnTwimlForCallSid(): 29 | request_object = request.form.to_dict() 30 | response = updated_twimls[request_object["CallSid"]] 31 | 32 | updated_twimls.pop(request_object["CallSid"]) 33 | return response 34 | 35 | @app.route('/twiml', methods=['POST']) 36 | def return_twiml(): 37 | print("POST TwiML") 38 | return render_template('streams.xml') 39 | 40 | @sockets.route('/') 41 | def echo(ws): 42 | print("Connection accepted") 43 | client_data_processor = TwilioDataProcessor(ws) 44 | client_data_processor.start() 45 | 46 | class TwilioCall: 47 | AUTH_TOKEN = os.environ.get('TWILIO_AUTH_TOKEN') 48 | SERVICE_DNS = os.environ.get('URL') 49 | 50 | def __init__(self, account_sid, call_sid): 51 | self.account_sid = account_sid 52 | self.auth_token = self.AUTH_TOKEN 53 | self.service_dns = self.SERVICE_DNS 54 | self.call_sid = call_sid 55 | 56 | def update(self): 57 | update_url = "{0}/{1}".format(self.service_dns, "updatecall") 58 | rest_client = Client(self.account_sid, self.auth_token) 59 | rest_client.calls(self.call_sid).update(method="POST", url=update_url) 60 | 61 | def persist(self, response): 62 | updated_twimls[self.call_sid] = response 63 | 64 | 65 | class TwilioDataProcessor: 66 | def __init__(self, ws): 67 | self.logger = logging.getLogger(__name__) 68 | self.ws = ws 69 | raw_id = str(uuid.uuid4()) 70 | self.user_id = raw_id[0:24].replace("-", "").upper() 71 | self.lex_streaming_client = VoiceAndSilenceDetectingLexClient(self.user_id, [self], [self]) 72 | self.listen_switch = threading.Event() 73 | self.twilio_call = None 74 | 75 | def start(self): 76 | try: 77 | while not self.ws.closed: 78 | while not self.listen_switch.is_set(): 79 | message = self.ws.receive() 80 | if message is None: 81 | print('No message') 82 | break 83 | 84 | data = json.loads(message) 85 | if data['event'] == "connected": 86 | log("Connected Message received", message) 87 | if data['event'] == "start": 88 | log("Start Message received", message) 89 | print("Media WS: received media and metadata: " + str(data)) 90 | self.twilio_call = TwilioCall(data["start"]["accountSid"], data["start"]["callSid"]) 91 | 92 | if data['event'] == "media": 93 | self.lex_streaming_client.stream_to_lex(data["media"]["payload"]) 94 | if data['event'] == "closed": 95 | log("Closed Message received", message) 96 | break 97 | 98 | except Exception as e: 99 | self.logger.exception(e) 100 | 101 | def pause_listening(self): 102 | self.listen_switch.set() 103 | 104 | def reset(self): 105 | self.logger.info("recreating VAD lex client") 106 | self.lex_streaming_client = VoiceAndSilenceDetectingLexClient(self.user_id, [self], [self]) 107 | self.listen_switch.clear() 108 | 109 | def voice_detected(self): 110 | self.logger.info("voice detected in input stream passed to fancy lex client") 111 | self.pause_playback() 112 | 113 | def silence_detected(self, **kwargs): 114 | self.logger.info("silence detected in input stream passed. stop listening for additional data from client, process the collected data and send the result to play back") 115 | for key, value in kwargs.items(): 116 | self.logger.info("{0} = {1}".format(key, value)) 117 | self.listen_switch.set() 118 | self.process() 119 | self.send_data_to_client(kwargs.get("lex_response")) 120 | self.listen_switch.clear() 121 | self.reset() 122 | 123 | def pause_playback(self): 124 | self.logger.info("if something is being played back on connection, now is the time to stop it") 125 | # stop playback processing here 126 | 127 | def process(self): 128 | self.logger.info("processing data listened so far") 129 | 130 | # create a new TwiML, 131 | # update the call. 132 | def send_data_to_client(self, lex_response): 133 | self.logger.info("sending data to client {0}".format(lex_response)) 134 | response = VoiceResponse() 135 | response.say(lex_response.get("Message")) 136 | 137 | dialog_state = lex_response.get("DialogState") 138 | intent_name = lex_response.get("IntentName") 139 | 140 | if intent_name is not None and intent_name == "GoodbyeIntent" and dialog_state == "Fulfilled" : 141 | # hang up the call after this 142 | response.hangup() 143 | else: 144 | response.pause(40) 145 | 146 | self.logger.info("response is {0}".format(response)) 147 | self.twilio_call.persist(response.to_xml()) 148 | self.twilio_call.update() 149 | 150 | if __name__ == '__main__': 151 | from gevent import pywsgi 152 | from geventwebsocket.handler import WebSocketHandler 153 | 154 | server = pywsgi.WSGIServer(('', HTTP_SERVER_PORT), app, handler_class=WebSocketHandler) 155 | print("Server listening on: http://localhost:" + str(HTTP_SERVER_PORT)) 156 | server.serve_forever() 157 | -------------------------------------------------------------------------------- /serviceinfra_cfn.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: lex/twilio voice integration service 3 | 4 | Parameters: 5 | EnvironmentName: 6 | Type: String 7 | Default: lex-twiliovoice-stage 8 | Description: Environment name. Same name from the previous deployment to refer to the outputs 9 | ServiceName: 10 | Type: String 11 | Default: lexbot 12 | Description: Name of the service 13 | ImageUrl: 14 | Type: String 15 | Default: "lex-twiliovoice-image" 16 | Description: The url of a docker image that contains the service (example;xxxxxxx.dkr.ecr.us-west-2.amazonaws.com/lex-twiliovoice:latest) 17 | ContainerPort: 18 | Type: Number 19 | Default: 8080 20 | Description: Port number where the service is running 21 | ContainerCpu: 22 | Type: Number 23 | Default: 1024 24 | Description: How much CPU to give the container. 1024 is 1 CPU 25 | ContainerMemory: 26 | Type: Number 27 | Default: 2048 28 | Description: How much memory in MB to give the container 29 | DesiredCount: 30 | Type: Number 31 | Default: 1 32 | Description: How many copies of the service task to run. For this demo, we will use 1 33 | TwilioAuthToken: 34 | Type: String 35 | Default: "TwilioAuthToken" 36 | Description: Auth token from the twilio console 37 | IAMAccessKeyId: 38 | Type: String 39 | Default: "IAM User AccessKeyId Here" 40 | Description: IAM User AccessKeyId with the following permissions, AmazonLexReadOnly & AmazonLexRunBotsOnly 41 | IAMSecretAccessKey: 42 | Type: String 43 | Default: "IAM User SecretAccessKey Here" 44 | Description: IAM User SecretAccessKey with the following permissions, AmazonLexReadOnly & AmazonLexRunBotsOnly 45 | LexBotName: 46 | Type: String 47 | Default: "BookTrip" 48 | Description: Enter your Lex Bot name 49 | LexBotAlias: 50 | Type: String 51 | Default: "beta" 52 | Description: Enter your Lex Bot Alias 53 | 54 | 55 | Resources: 56 | LogGroup: 57 | Type: AWS::Logs::LogGroup 58 | Properties: 59 | LogGroupName: !Join ['-', [!Ref 'EnvironmentName', 'service', !Ref 'ServiceName']] 60 | #Task definition and container definitions 61 | TaskDefinition: 62 | Type: AWS::ECS::TaskDefinition 63 | Properties: 64 | Family: !Ref 'ServiceName' 65 | Cpu: !Ref 'ContainerCpu' 66 | Memory: !Ref 'ContainerMemory' 67 | NetworkMode: awsvpc 68 | RequiresCompatibilities: 69 | - FARGATE 70 | ExecutionRoleArn: 71 | Fn::ImportValue: 72 | !Join [':', [!Ref 'EnvironmentName', 'ECSTaskExecutionRole']] 73 | ContainerDefinitions: 74 | - Name: !Ref 'ServiceName' 75 | Cpu: !Ref 'ContainerCpu' 76 | Memory: !Ref 'ContainerMemory' 77 | Image: !Ref 'ImageUrl' 78 | Environment: 79 | - Name: AWS_REGION 80 | Value: !Ref 'AWS::Region' 81 | - Name: ACCESS_KEY_ID 82 | Value: !Ref 'IAMAccessKeyId' 83 | - Name: SECRET_ACCESS_KEY 84 | Value: !Ref 'IAMSecretAccessKey' 85 | - Name: LEX_BOT_NAME 86 | Value: !Ref 'LexBotName' 87 | - Name: LEX_BOT_ALIAS 88 | Value: !Ref 'LexBotAlias' 89 | - Name: TWILIO_AUTH_TOKEN 90 | Value: !Ref 'TwilioAuthToken' 91 | - Name: CONTAINER_PORT 92 | Value: !Ref 'ContainerPort' 93 | - Name: URL 94 | Value: 95 | ! Fn::ImportValue: 96 | !Join [':', [!Ref 'EnvironmentName', 'ExternalUrl']] 97 | PortMappings: 98 | - ContainerPort: !Ref 'ContainerPort' 99 | LogConfiguration: 100 | LogDriver: 'awslogs' 101 | Options: 102 | awslogs-group: !Join ['-', [!Ref 'EnvironmentName', 'service', !Ref 'ServiceName']] 103 | awslogs-region: !Ref 'AWS::Region' 104 | awslogs-stream-prefix: !Ref 'ServiceName' 105 | 106 | #ECS Service 107 | Service: 108 | Type: AWS::ECS::Service 109 | DependsOn: 110 | - HTTPRule 111 | Properties: 112 | ServiceName: !Ref 'ServiceName' 113 | Cluster: 114 | Fn::ImportValue: 115 | !Join [':', [!Ref 'EnvironmentName', 'ClusterName']] 116 | LaunchType: FARGATE 117 | DeploymentConfiguration: 118 | MaximumPercent: 200 119 | MinimumHealthyPercent: 75 120 | DesiredCount: !Ref 'DesiredCount' 121 | NetworkConfiguration: 122 | AwsvpcConfiguration: 123 | AssignPublicIp: ENABLED 124 | SecurityGroups: 125 | - Fn::ImportValue: 126 | !Join [':', [!Ref 'EnvironmentName', 'FargateContainerSecurityGroup']] 127 | Subnets: 128 | - Fn::ImportValue: 129 | !Join [':', [!Ref 'EnvironmentName', 'PublicSubnetOne']] 130 | - Fn::ImportValue: 131 | !Join [':', [!Ref 'EnvironmentName', 'PublicSubnetTwo']] 132 | TaskDefinition: !Ref 'TaskDefinition' 133 | LoadBalancers: 134 | - ContainerName: !Ref 'ServiceName' 135 | ContainerPort: !Ref 'ContainerPort' 136 | TargetGroupArn: !Ref 'TargetGroup' 137 | 138 | #Target group 139 | TargetGroup: 140 | Type: AWS::ElasticLoadBalancingV2::TargetGroup 141 | Properties: 142 | HealthCheckIntervalSeconds: 6 143 | HealthCheckPath: /ping 144 | Matcher: 145 | HttpCode: 200 146 | HealthCheckProtocol: HTTP 147 | HealthCheckTimeoutSeconds: 5 148 | HealthyThresholdCount: 2 149 | TargetType: ip 150 | Name: !Join ['-', [!Ref 'EnvironmentName', !Ref 'ServiceName']] 151 | Port: !Ref 'ContainerPort' 152 | Protocol: HTTP 153 | UnhealthyThresholdCount: 2 154 | TargetGroupAttributes: 155 | - Key: stickiness.enabled 156 | Value: true 157 | - Key: deregistration_delay.timeout_seconds 158 | Value: 30 159 | VpcId: 160 | Fn::ImportValue: 161 | !Join [':', [!Ref 'EnvironmentName', 'VPCId']] 162 | 163 | # Create rules to forward HTTP Traffic to the target group 164 | HTTPRule: 165 | Type: AWS::ElasticLoadBalancingV2::ListenerRule 166 | Properties: 167 | Actions: 168 | - TargetGroupArn: !Ref 'TargetGroup' 169 | Type: 'forward' 170 | Conditions: 171 | - Field: path-pattern 172 | Values: ['*'] 173 | ListenerArn: 174 | Fn::ImportValue: 175 | !Join [':', [!Ref 'EnvironmentName', 'PublicListenerHTTP']] 176 | Priority: 1 177 | HTTPSRule: 178 | Type: AWS::ElasticLoadBalancingV2::ListenerRule 179 | Properties: 180 | Actions: 181 | - TargetGroupArn: !Ref 'TargetGroup' 182 | Type: 'forward' 183 | Conditions: 184 | - Field: path-pattern 185 | Values: ['*'] 186 | ListenerArn: 187 | Fn::ImportValue: 188 | !Join [':', [!Ref 'EnvironmentName', 'PublicListenerHTTPS']] 189 | Priority: 1 190 | 191 | 192 | -------------------------------------------------------------------------------- /templates/streams.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | You will be interacting with Lex bot in 3, 2, 1. Go. 4 | 5 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /voice_and_silence_detecting_lex_wrapper.py: -------------------------------------------------------------------------------- 1 | import audioop 2 | import base64 3 | import logging 4 | import time 5 | import threading 6 | from datetime import datetime 7 | from lex_streaming_client import LexClientStreaming 8 | 9 | class VoiceAndSilenceDetectingLexClient: 10 | vad_sd_config = { 11 | "VoiceThreshold": 500, 12 | "SilenceDurationTimeInSecs": 2, 13 | "TwilioRate": 8000, 14 | "LexRate": 16000, 15 | "Width": 2, 16 | "Channels": 1 17 | } 18 | 19 | def __init__(self, user_id, voice_detected_call_backs=[], silence_detected_call_backs=[]): 20 | self.logger = logging.getLogger(__name__) 21 | self.voice_threshold = self.vad_sd_config["VoiceThreshold"] 22 | self.silence_duration_time = self.vad_sd_config["SilenceDurationTimeInSecs"] 23 | self.twilio_rate = self.vad_sd_config["TwilioRate"] 24 | self.lex_rate = self.vad_sd_config["LexRate"] 25 | self.width = self.vad_sd_config["Width"] 26 | self.channels = self.vad_sd_config["Channels"] 27 | 28 | 29 | self.lex_client = LexClientStreaming(user_id) 30 | self.rms_graph = [] 31 | self.rms_values = [] 32 | self.last_detected_voice_time = None 33 | 34 | self.voice_detected_call_backs = voice_detected_call_backs 35 | self.silence_detected_call_backs = silence_detected_call_backs 36 | 37 | self.stop_data_processing = threading.Event() 38 | self.logger.info("VoiceAndSilenceDetectingLexClient configured with voice threshold {0}, silence duration {1}, lex user id {2}" 39 | .format(self.voice_threshold, 40 | self.silence_duration_time, 41 | user_id)) 42 | 43 | 44 | def stream_to_lex(self, base_64_encoded_data): 45 | if self.stop_data_processing.is_set(): 46 | self.logger.warn("discarding the passed in data, as underlying lex stream has been stopped") 47 | return 48 | 49 | data = self.__decode_data(base_64_encoded_data) 50 | 51 | raw_audio_data = audioop.ulaw2lin(data, self.width) 52 | #raw_audio_data, state = audioop.lin2adpcm(raw_audio_data, self.width, None) 53 | 54 | # raw_audio_data, state = audioop.ratecv(raw_audio_data, 55 | # self.width, 56 | # self.channels, 57 | # self.twilio_rate, 58 | # self.lex_rate, 59 | # None) 60 | 61 | rms = audioop.rms(raw_audio_data, self.width) 62 | 63 | #self.logger.info("RMS value is {0}".format(rms)) 64 | self.rms_values.append(rms) 65 | if rms > self.voice_threshold: 66 | #self.logger.debug("voice detected in input data") 67 | self.rms_graph.append("^") 68 | 69 | if self.last_detected_voice_time is None: 70 | # voice detected for first time 71 | self.logger.debug("voice detected for first time") 72 | self.voice_detected() 73 | self.last_detected_voice_time = datetime.now() 74 | self.lex_client.add_to_stream(raw_audio_data) 75 | else: 76 | self.rms_graph.append(".") 77 | #self.logger.debug("silence detected in input data") 78 | 79 | if self.last_detected_voice_time: 80 | # check if elapsed time is greater than configured time for silence 81 | self.lex_client.add_to_stream(raw_audio_data) 82 | 83 | silence_time = (datetime.now() - self.last_detected_voice_time).total_seconds() 84 | if silence_time >= self.silence_duration_time: 85 | self.logger.debug("elapsed time {0} seconds since last detected voice time {1} is higher than configured time for silence {2} seconds. closing connection to lex." 86 | .format(silence_time, 87 | self.last_detected_voice_time, 88 | self.silence_duration_time)) 89 | 90 | # stop lex client now 91 | self.lex_client.stop() 92 | self.stop_data_processing.set() 93 | self.logger.info("Voice activity graph {0}".format("".join(self.rms_graph))) 94 | self.logger.info("RMS values {0}".format(self.rms_values)) 95 | self.silence_detected() 96 | #else: 97 | # self.logger.debug("voice has not been detected even once. not starting the silence detection counter") 98 | 99 | def __decode_data(self, data): 100 | return base64.b64decode(data) 101 | 102 | def voice_detected(self): 103 | self.logger.info("invoking voice detected callbacks") 104 | for voice_detected_call_back in self.voice_detected_call_backs: 105 | self.logger.info("invoking voice detected callback {0}".format(voice_detected_call_back)) 106 | voice_detected_call_back.voice_detected() 107 | 108 | def silence_detected(self): 109 | 110 | self.logger.info("invoking silence detected callbacks with lex data ") 111 | lex_response = self.lex_client.get_response() 112 | self.logger.info("lex response is {0}".format(lex_response)) 113 | 114 | for silence_detected_call_back in self.silence_detected_call_backs: 115 | self.logger.info("invoking silence detected callback {0}".format(silence_detected_call_back)) 116 | silence_detected_call_back.silence_detected(lex_response = lex_response) 117 | 118 | if __name__ == '__main__': 119 | data = "/v7+/v7+/n7+/v5+fn7+fn7+/v5+/n7+/v7+/v5+fn7+/H5+/vx+/vz+fv78fnz+/Hx+/P5+/P56fvp+evz6fHz6/nr++np6+P54/Ph6fPR+dvr2dnj2/HL+8Hpy9vJyeO/+b/zydnT2+nh++n5+/P5+fnz+/n56/Pp6evb6dn70fnT89Hx0+PZ4ePj6dnr2+nR+9H50+vR6ePz6enr8+g==" 120 | lex_client = VoiceAndSilenceDetectingLexClient() 121 | lex_client.stream_to_lex(data) 122 | time.sleep(2) 123 | lex_client.stream_to_lex("/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////w==") 124 | 125 | --------------------------------------------------------------------------------