├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── imgs ├── Detailed.png └── General.png ├── publisher ├── Dockerfile ├── publisher.py ├── requirements.txt └── templates │ ├── index.html │ └── order.html └── subscribers ├── fulfilment ├── Dockerfile ├── fulfilment.py └── requirements.txt └── promotion ├── Dockerfile ├── promotion.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # macOS Internals 2 | .DS_Store 3 | 4 | # VSCode 5 | .workspace 6 | 7 | # Copilot Generated files 8 | copilot/* 9 | 10 | # Dependencies 11 | node_modules/ -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Implementing a pub/sub architecture with AWS Copilot 2 | 3 | The [AWS Copilot CLI](https://aws.github.io/copilot-cli/) is a tool that since its [launch in 2020](https://aws.amazon.com/blogs/containers/introducing-aws-copilot/), developers have been using to build, manage, and operate Linux and Windows containers on [Amazon Elastic Container Service (Amazon ECS)](https://aws.amazon.com/ecs/), [AWS Fargate](https://aws.amazon.com/fargate/), and [AWS App Runner.](https://aws.amazon.com/apprunner/) 4 | 5 | This repository contains code for a publisher and two subscriber “worker” services that respectively publish and consume events in a [pub/sub architecture](https://aws.amazon.com/pub-sub-messaging/). 6 | 7 | The high-level architecture is shown below. 8 | ![image info](./imgs/General.png) 9 | 10 | You can easily build a sample pub/sub architecture by leveraging the AWS Copilot CLI. Rather than losing time creating all of the infrastructure and policies needed for your services, you can leverage some service templates and commands that help you deploy all the needed resources, allowing you to focus on what really matters. Creating publisher and subscribers has never been easier, as the AWS Copilot CLI can take care of creating SNS Topics, SQS Queues, Subscriptions, Filtering policies, DLQ, re-drive configuration and the URI injection into the service themselves. 11 | 12 | ![image info](./imgs/Detailed.png) 13 | 14 | A technical how-to blog tutorial on the deployment can be found here: 15 | 16 | - https://aws.amazon.com/blogs/containers/implementing-a-pub-sub-architecture-with-aws-copilot/ 17 | 18 | ## Releases 19 | 20 | - v1.0 : This repository has been published on as support material for the technical how-to blog tutorial. 21 | 22 | ## Authors and acknowledgment 23 | 24 | - Rafael Mosca 25 | 26 | ## Contributing 27 | 28 | AWS Copilot is an [open-source tool](https://github.com/aws/copilot-cli), and you can check out our incremental features on our [sprint board](https://github.com/aws/copilot-cli/projects/1). We encourage you to [get involved](https://aws.github.io/copilot-cli/community/get-involved/) by creating [GitHub issues](https://github.com/aws/copilot-cli/issues) or joining the conversation on [Gitter](https://gitter.im/aws/copilot-cli)! 29 | 30 | - [AWS Copilot Documentation](https://aws.github.io/copilot-cli) 31 | - [AWS Copilot Public Sprint Board on GitHub](https://github.com/aws/copilot-cli/projects/1) 32 | - [AWS Copilot Gitter Chat Room](https://gitter.im/aws/copilot-cli) 33 | 34 | For contributing to this specific project, please refer to the CONTRIBUTING file for more info. 35 | 36 | ## License 37 | 38 | This project is licensed under the MIT-0 license. See the LICENSE file for more info. 39 | -------------------------------------------------------------------------------- /imgs/Detailed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-copilot-pubsub/dffa1e7e3b1d8619156af2c788190f63147e3a2b/imgs/Detailed.png -------------------------------------------------------------------------------- /imgs/General.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-copilot-pubsub/dffa1e7e3b1d8619156af2c788190f63147e3a2b/imgs/General.png -------------------------------------------------------------------------------- /publisher/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.8-slim-buster 2 | 3 | WORKDIR /app 4 | 5 | # Copy dependency file 6 | COPY requirements.txt requirements.txt 7 | 8 | # Install dependencies 9 | RUN pip3 install -r requirements.txt 10 | 11 | # Add source code into container image in path /app 12 | COPY . /app 13 | 14 | # Start the service 15 | CMD ["python3", "publisher.py"] -------------------------------------------------------------------------------- /publisher/publisher.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | # ------------- Imports ------------------------ 5 | from flask import Flask, request, render_template, redirect, url_for 6 | import os 7 | import sys 8 | import json 9 | import uuid 10 | import boto3 11 | import names 12 | import random 13 | import logging 14 | 15 | # ------------ Global config ----------------- 16 | app = Flask(__name__) 17 | logging.basicConfig(stream=sys.stdout, level=logging.INFO) 18 | aws_region = os.getenv("AWS_DEFAULT_REGION", default='eu-west-1') 19 | 20 | # ------------ SNS (Message sending) --------------- 21 | sns_client = boto3.client('sns', region_name=aws_region) 22 | dest_topic_name = 'ordersTopic' 23 | sns_topics_arn = json.loads(os.getenv("COPILOT_SNS_TOPIC_ARNS")) 24 | topic_arn = sns_topics_arn[dest_topic_name] 25 | 26 | # --------- DynamoDB (NoSQL Database) -------------- 27 | dynamodb = boto3.resource('dynamodb', region_name=aws_region) 28 | table_name = os.getenv("ORDERS_TABLE_NAME") 29 | db_table = dynamodb.Table(table_name) 30 | 31 | # ---------- Main Page --------------------- 32 | @app.route('/', methods=["GET", "POST"]) 33 | def submit_order(): 34 | 35 | # When "Send" button is clicked 36 | if request.method == 'POST': 37 | 38 | # Generate an Id 39 | id = str(uuid.uuid4()) 40 | 41 | # Get data from form 42 | customer = request.form['customer'] 43 | amount = request.form['amount'] 44 | 45 | # Save the data to a DynamoDB Table 46 | db_table.put_item( 47 | Item={ 48 | 'id': id, 49 | 'customer': customer, 50 | 'amount': amount, 51 | } 52 | ) 53 | logging.info('Request saved in database') 54 | 55 | # Send a message to the SNS topic 56 | sns_client.publish( 57 | TargetArn=topic_arn, 58 | Message=json.dumps({ 59 | 'customer': customer, 60 | 'amount': amount, 61 | }), 62 | MessageAttributes={ 63 | 'amount': { 64 | 'DataType': 'Number', 65 | 'StringValue': str(amount) 66 | } 67 | } 68 | ) 69 | logging.info(f'Message sent to {topic_arn}') 70 | 71 | 72 | return redirect(url_for('request_page', request_id=id)) 73 | 74 | 75 | # Generate a random name and amount to prepopulate the text box 76 | name = names.get_full_name() 77 | amount = round(random.uniform(0, 100), 2) 78 | 79 | return render_template('index.html', customer=name, amount=amount) 80 | 81 | # ------------ Request Redirection Page ------------------- 82 | @app.route('/request/') 83 | def request_page(request_id): 84 | response = db_table.get_item( 85 | Key={ 'id': str(request_id) } 86 | ) 87 | return render_template('order.html', response=response['Item']) 88 | 89 | if __name__ == '__main__': 90 | app.run(host='0.0.0.0', port=5000, debug=False) -------------------------------------------------------------------------------- /publisher/requirements.txt: -------------------------------------------------------------------------------- 1 | flask==2.3.2 2 | boto3==1.25.4 3 | names==0.3.0 4 | -------------------------------------------------------------------------------- /publisher/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | Pub/Sub Demo 3 | 9 | 10 |
11 |
12 |
Send an order
13 |
14 |
15 | 16 | 22 |
23 |
24 | 25 |
26 |
27 | $ 28 |
29 | 30 |
31 |
32 |
33 | 34 |
35 |
36 |
37 |
38 | 39 | -------------------------------------------------------------------------------- /publisher/templates/order.html: -------------------------------------------------------------------------------- 1 | 2 | Pub/Sub Demo 3 | 9 | 10 |
11 | Order Sent!
12 | 
{{ response }}
13 |
14 | 15 | -------------------------------------------------------------------------------- /subscribers/fulfilment/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.8-slim-buster 2 | 3 | WORKDIR /app 4 | 5 | # Copy dependency file 6 | COPY requirements.txt requirements.txt 7 | 8 | # Install dependencies 9 | RUN pip3 install -r requirements.txt 10 | 11 | # Add source code into container image in path /app 12 | COPY . /app 13 | 14 | # Start the service 15 | CMD ["python3", "fulfilment.py"] -------------------------------------------------------------------------------- /subscribers/fulfilment/fulfilment.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | # ------------- Imports ------------------------ 5 | import os 6 | import sys 7 | import json 8 | import boto3 9 | import logging 10 | import pyfiglet 11 | 12 | # ------------ Global config ----------------- 13 | logging.basicConfig(stream=sys.stdout, level=logging.INFO) 14 | slant = pyfiglet.Figlet(font='slant') 15 | aws_region = os.getenv("AWS_DEFAULT_REGION", default='eu-west-1') 16 | sqs_client = boto3.client('sqs', region_name=aws_region) 17 | 18 | # ------------ Main functions ----------------- 19 | def get_messages(queue_uri): 20 | """ 21 | Long-polls the specified Queue, returning zero up to 10 messages. 22 | """ 23 | messages = sqs_client.receive_message(QueueUrl=queue_uri, 24 | WaitTimeSeconds=5) 25 | 26 | return messages.get('Messages', []) 27 | 28 | def delete_message(message, queue_uri): 29 | """Deletes a message from the specified queue""" 30 | try: 31 | return sqs_client.delete_message(QueueUrl=queue_uri, 32 | ReceiptHandle=message.get('ReceiptHandle')) 33 | except boto3.ClientError: 34 | logging.exception(f'Error deleting the message from {queue_uri}') 35 | 36 | def process_message(): 37 | queue_uri = os.environ.get('COPILOT_QUEUE_URI') 38 | 39 | # Process messages 40 | for message in get_messages(queue_uri): 41 | 42 | # Print out the name with a slanted effect 43 | message_body = json.loads(message['Body'])['Message'] 44 | customer_name = json.loads(message_body)['customer'] 45 | logging.info('Processing order for: \n') 46 | logging.info('\n' + slant.renderText(customer_name)) 47 | 48 | # Message is processed, delete it from the queue 49 | delete_message(message, queue_uri) 50 | 51 | 52 | # -------------- Entrypoint ------------------- 53 | if __name__ == '__main__': 54 | while True: 55 | process_message() -------------------------------------------------------------------------------- /subscribers/fulfilment/requirements.txt: -------------------------------------------------------------------------------- 1 | boto3==1.25.4 2 | pyfiglet==0.8.post1 3 | -------------------------------------------------------------------------------- /subscribers/promotion/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.8-slim-buster 2 | 3 | WORKDIR /app 4 | 5 | # Copy dependency file 6 | COPY requirements.txt requirements.txt 7 | 8 | # Install dependencies 9 | RUN pip3 install -r requirements.txt 10 | 11 | # Add source code into container image in path /app 12 | COPY . /app 13 | 14 | # Start the service 15 | CMD ["python3", "promotion.py"] -------------------------------------------------------------------------------- /subscribers/promotion/promotion.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | import os 5 | import sys 6 | import json 7 | import uuid 8 | import boto3 9 | import logging 10 | 11 | # ------------ Global config ----------------- 12 | logging.basicConfig(stream=sys.stdout, level=logging.INFO) 13 | aws_region = os.getenv("AWS_DEFAULT_REGION", default='eu-west-1') 14 | sqs_client = boto3.client('sqs', region_name=aws_region) 15 | 16 | # ------------ Main functions ----------------- 17 | def get_messages(queue_uri): 18 | """ 19 | Long-polls the specified Queue, returning zero up to 10 messages. 20 | """ 21 | messages = sqs_client.receive_message(QueueUrl=queue_uri, 22 | WaitTimeSeconds=5) 23 | 24 | return messages.get('Messages', []) 25 | 26 | def delete_message(message, queue_uri): 27 | """Deletes a message from the specified queue""" 28 | try: 29 | return sqs_client.delete_message(QueueUrl=queue_uri, 30 | ReceiptHandle=message.get('ReceiptHandle')) 31 | except boto3.ClientError: 32 | logging.exception(f'Error deleting the message from {queue_uri}') 33 | 34 | def process_message(): 35 | queue_uri = os.environ.get('COPILOT_QUEUE_URI') 36 | 37 | # Process messages 38 | for message in get_messages(queue_uri): 39 | logging.info('Processing message Received from SNS') 40 | 41 | # Print out the name with a slanted effect 42 | coupon_code = str(uuid.uuid4())[:8] 43 | message_body = json.loads(message['Body'])['Message'] 44 | customer_name = json.loads(message_body)['customer'] 45 | 46 | logging.info(f'\n Hey {customer_name}!\n Thanks for your order, you ' +\ 47 | "have qualified for a 20% off on your next purchase.\n" +\ 48 | f"Use coupon --- {coupon_code} --- on you next purchase.") 49 | 50 | # Message is processed, delete it from the queue 51 | delete_message(message, queue_uri) 52 | 53 | 54 | # -------------- Entrypoint ------------------- 55 | if __name__ == '__main__': 56 | while True: 57 | process_message() -------------------------------------------------------------------------------- /subscribers/promotion/requirements.txt: -------------------------------------------------------------------------------- 1 | boto3==1.25.4 2 | --------------------------------------------------------------------------------