├── .gitignore ├── LICENSE ├── makefile ├── template.yml ├── state-machine ├── state-machine.asl.json └── test │ └── MockConfigFile.json └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | samconfig.toml 2 | state-machine/test/commands.txt 3 | .DS_Store 4 | .vscode/settings.json 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Marcia Villalba 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /makefile: -------------------------------------------------------------------------------- 1 | ROOT_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST)))) 2 | 3 | run: 4 | docker run -p 8083:8083 \ 5 | --mount type=bind,readonly,source=$(ROOT_DIR)/state-machine/test/MockConfigFile.json,destination=/home/StepFunctionsLocal/MockConfigFile.json \ 6 | -e SFN_MOCK_CONFIG="/home/StepFunctionsLocal/MockConfigFile.json" \ 7 | amazon/aws-stepfunctions-local 8 | 9 | create: 10 | aws stepfunctions create-state-machine \ 11 | --endpoint-url http://localhost:8083 \ 12 | --definition file://state-machine/state-machine.asl.json \ 13 | --name "LocalTesting" \ 14 | --role-arn "arn:aws:iam::123456789012:role/DummyRole" \ 15 | --no-cli-pager 16 | 17 | happy: 18 | aws stepfunctions start-execution \ 19 | --endpoint http://localhost:8083 \ 20 | --name HappyPathTest \ 21 | --state-machine arn:aws:states:us-east-1:123456789012:stateMachine:LocalTesting#HappyPathTest \ 22 | --no-cli-pager 23 | 24 | fileNotFound: 25 | aws stepfunctions start-execution \ 26 | --endpoint http://localhost:8083 \ 27 | --name FileNotFoundTest \ 28 | --state-machine arn:aws:states:us-east-1:123456789012:stateMachine:LocalTesting#FileNotFoundTest \ 29 | --no-cli-pager 30 | 31 | 32 | all: create happy fileNotFound 33 | 34 | happy-h: 35 | aws stepfunctions get-execution-history \ 36 | --endpoint http://localhost:8083 \ 37 | --execution-arn arn:aws:states:us-east-1:123456789012:execution:LocalTesting:HappyPathTest \ 38 | --query 'events[?type==`ExecutionSucceeded`]' \ 39 | --no-cli-pager 40 | 41 | fileNotFound-h: 42 | aws stepfunctions get-execution-history \ 43 | --endpoint http://localhost:8083 \ 44 | --execution-arn arn:aws:states:us-east-1:123456789012:execution:LocalTesting:FileNotFoundTest \ 45 | --query 'events[?type==`ExecutionFailed` && executionFailedEventDetails.error==`S3.NoSuchBucketException`]' \ 46 | --no-cli-pager 47 | 48 | history: happy-h fileNotFound-h -------------------------------------------------------------------------------- /template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: "App-video-processing" 4 | 5 | Parameters: 6 | MySampleDataBucketName: 7 | Type: String 8 | MySampleDataInputKey: 9 | Type: String 10 | 11 | Resources: 12 | # Define a common IAM role to be used for all components of this app 13 | ApplicationRole: 14 | Type: AWS::IAM::Role 15 | Properties: 16 | AssumeRolePolicyDocument: 17 | Version: "2012-10-17" 18 | Statement: 19 | - 20 | Effect: "Allow" 21 | Principal: 22 | Service: 23 | - "states.amazonaws.com" 24 | Action: 25 | - "sts:AssumeRole" 26 | Policies: 27 | - PolicyName: AppPolicy 28 | PolicyDocument: 29 | Version: 2012-10-17 30 | Statement: 31 | - 32 | Effect: Allow 33 | Action: 34 | - xray:PutTraceSegments 35 | - xray:PutTelemetryRecords 36 | - xray:GetSamplingRules 37 | - xray:GetSamplingTargets 38 | - transcribe:StartTranscriptionJob 39 | - transcribe:GetTranscriptionJob 40 | - translate:TranslateText 41 | Resource: '*' 42 | - 43 | Effect: Allow 44 | Action: 45 | - s3:* 46 | Resource: '*' 47 | 48 | # An S3 Bucket for the data processing 49 | DataBucket: 50 | Type: AWS::S3::Bucket 51 | Properties: 52 | BucketName : !Join ["",[!Ref "AWS::AccountId","-",!Ref "AWS::Region","-video-processing-example2"]] 53 | 54 | StateMachineProcessVideoFile: 55 | Type: AWS::Serverless::StateMachine 56 | Properties: 57 | Type: "STANDARD" 58 | Role: !GetAtt ApplicationRole.Arn 59 | Name: !Join ["",[!Ref "AWS::AccountId","-",!Ref "AWS::Region","-video-processing-example1"]] 60 | DefinitionUri: state-machine/state-machine.asl.json 61 | DefinitionSubstitutions: 62 | SampleDataBucketName: !Ref MySampleDataBucketName 63 | SampleDataInputKey: !Ref MySampleDataInputKey 64 | S3BucketName: !Ref DataBucket 65 | ApplicationRoleArn: !GetAtt ApplicationRole.Arn 66 | Tracing: 67 | Enabled: true 68 | -------------------------------------------------------------------------------- /state-machine/state-machine.asl.json: -------------------------------------------------------------------------------- 1 | { 2 | "Comment": "A State Machine that process a video file", 3 | "StartAt": "GetSampleVideo", 4 | "States": { 5 | 6 | "GetSampleVideo": { 7 | "Type": "Task", 8 | "Resource": "arn:aws:states:::aws-sdk:s3:copyObject", 9 | "Parameters": { 10 | "Bucket": "${S3BucketName}", 11 | "Key": "${SampleDataInputKey}", 12 | "CopySource": "${SampleDataBucketName}/${SampleDataInputKey}" 13 | }, 14 | "Next": "StartTranscriptionJob" 15 | }, 16 | 17 | "StartTranscriptionJob": { 18 | "Type": "Task", 19 | "Resource": "arn:aws:states:::aws-sdk:transcribe:startTranscriptionJob", 20 | "Parameters": { 21 | "Media": { 22 | "MediaFileUri": "s3://${S3BucketName}/${SampleDataInputKey}" 23 | }, 24 | "TranscriptionJobName.$": "$$.Execution.Name", 25 | "LanguageCode": "en-US", 26 | "OutputBucketName": "${S3BucketName}", 27 | "OutputKey": "transcribe.json" 28 | }, 29 | "Next": "Wait20Seconds" 30 | }, 31 | 32 | "Wait20Seconds": { 33 | "Type": "Wait", 34 | "Seconds": 20, 35 | "OutputPath": "$.TranscriptionJob", 36 | "Next": "CheckIfTranscriptionDone" 37 | }, 38 | 39 | "CheckIfTranscriptionDone": { 40 | "Type": "Task", 41 | "Resource": "arn:aws:states:::aws-sdk:transcribe:getTranscriptionJob", 42 | "Parameters": { 43 | "TranscriptionJobName.$": "$.TranscriptionJobName" 44 | }, 45 | "Next": "IsTranscriptionDone?" 46 | }, 47 | 48 | "IsTranscriptionDone?": { 49 | "Type": "Choice", 50 | "Choices": [ 51 | { 52 | "Variable": "$.TranscriptionJob.TranscriptionJobStatus", 53 | "StringEquals": "COMPLETED", 54 | "Next": "GetTranscriptionText" 55 | } 56 | ], 57 | "Default": "Wait20Seconds" 58 | }, 59 | 60 | "GetTranscriptionText": { 61 | "Type": "Task", 62 | "Resource": "arn:aws:states:::aws-sdk:s3:getObject", 63 | "Parameters": { 64 | "Bucket": "${S3BucketName}", 65 | "Key": "transcribe.json" 66 | }, 67 | "ResultSelector": { 68 | "filecontent.$": "States.StringToJson($.Body)" 69 | }, 70 | "Next": "PrepareTranscriptTest" 71 | }, 72 | 73 | "PrepareTranscriptTest" : { 74 | "Type": "Pass", 75 | "Parameters": { 76 | "transcript.$": "$.filecontent.results.transcripts[0].transcript" 77 | }, 78 | "Next": "StartTextTranslation" 79 | }, 80 | 81 | "StartTextTranslation": { 82 | "Type": "Task", 83 | "Resource": "arn:aws:states:::aws-sdk:translate:translateText", 84 | "Parameters": { 85 | "SourceLanguageCode": "en", 86 | "TargetLanguageCode": "es", 87 | "Text.$": "$.transcript" 88 | }, 89 | "ResultPath": "$.translate", 90 | "End": true 91 | } 92 | } 93 | } -------------------------------------------------------------------------------- /state-machine/test/MockConfigFile.json: -------------------------------------------------------------------------------- 1 | { 2 | "StateMachines": { 3 | "LocalTesting": { 4 | "TestCases": { 5 | "HappyPathTest": { 6 | "GetSampleVideo": "GetSampleVideoMockedSuccess", 7 | "StartTranscriptionJob": "StartTranscriptionJobMockedSuccess", 8 | "CheckIfTranscriptionDone": "CheckIfTranscriptionDoneMockedSuccess", 9 | "GetTranscriptionText": "GetTranscriptionTextMockedSucess", 10 | "StartTextTranslation": "StartTextTranslationMockedSuccess" 11 | }, 12 | "FileNotFoundTest": { 13 | "GetSampleVideo": "GetSampleVideoMockedFailed" 14 | } 15 | } 16 | } 17 | }, 18 | "MockedResponses": { 19 | "GetSampleVideoMockedSuccess": { 20 | "0": { 21 | "Return": { 22 | "CopyObjectResult": { 23 | "ETag": "\"de03e636b2b6f2ed24b691817ba510a5\"", 24 | "LastModified": "2022-02-01T19:29:36Z" 25 | } 26 | } 27 | } 28 | }, 29 | "StartTranscriptionJobMockedSuccess": { 30 | "0": { 31 | "Return": { 32 | "TranscriptionJob": { 33 | "CreationTime": "2022-02-01T19:29:36.560Z", 34 | "LanguageCode": "en-US", 35 | "Media": { 36 | "MediaFileUri": "s3://144824907653-eu-west-1-video-processing-example2/intro-step-functions.mp4" 37 | }, 38 | "StartTime": "2022-02-01T19:29:36.591Z", 39 | "TranscriptionJobName": "7905e912-c3c2-574d-3cd4-4366d64bccbe", 40 | "TranscriptionJobStatus": "IN_PROGRESS" 41 | } 42 | } 43 | } 44 | }, 45 | "CheckIfTranscriptionDoneMockedSuccess": { 46 | "0": { 47 | "Return": { 48 | "TranscriptionJob": { 49 | "CompletionTime": "2022-02-01T19:30:12.317Z", 50 | "CreationTime": "2022-02-01T19:29:36.560Z", 51 | "LanguageCode": "en-US", 52 | "Media": { 53 | "MediaFileUri": "s3://144824907653-eu-west-1-video-processing-example2/intro-step-functions.mp4" 54 | }, 55 | "MediaFormat": "mp4", 56 | "MediaSampleRateHertz": 44100, 57 | "Settings": { 58 | "ChannelIdentification": false, 59 | "ShowAlternatives": false 60 | }, 61 | "StartTime": "2022-02-01T19:29:36.591Z", 62 | "Transcript": { 63 | "TranscriptFileUri": "https://s3.eu-west-1.amazonaws.com/144824907653-eu-west-1-video-processing-example2/transcribe.json" 64 | }, 65 | "TranscriptionJobName": "7905e912-c3c2-574d-3cd4-4366d64bccbe", 66 | "TranscriptionJobStatus": "COMPLETED" 67 | } 68 | } 69 | } 70 | }, 71 | "GetTranscriptionTextMockedSucess": { 72 | "0": { 73 | "Return": { 74 | "Body": "{\"jobName\":\"7905e912-c3c2-574d-3cd4-4366d64bccbe\",\"accountId\":\"144824907653\",\"results\":{\"transcripts\":[{\"transcript\":\"Hi my name is Mm hmm.\"}],\"items\":[{\"start_time\":\"6.54\",\"end_time\":\"7.3\",\"alternatives\":[{\"confidence\":\"1.0\",\"content\":\"Hi\"}],\"type\":\"pronunciation\"}]},\"status\":\"COMPLETED\"}" 75 | } 76 | } 77 | }, 78 | "StartTextTranslationMockedSuccess": { 79 | "0": { 80 | "Return": { 81 | "SourceLanguageCode": "en", 82 | "TargetLanguageCode": "es", 83 | "TranslatedText": "Hola, me llamo Mm hmm." 84 | } 85 | } 86 | }, 87 | "GetSampleVideoMockedFailed": { 88 | "0": { 89 | "Throw": { 90 | "Error": "S3.NoSuchBucketException", 91 | "Cause": "Server Exception while calling CopyObject API in S3 Service" 92 | } 93 | } 94 | } 95 | 96 | } 97 | } 98 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS Step Functions - SDK Integrations 2 | 3 | _Infrastructure as code framework used_: AWS SAM, ASL 4 | 5 | _AWS Services used_: AWS Step functions, Amazon S3, Amazon Transcribe, Amazon Transcribe, Amazon Translate 6 | 7 | ## Summary of the demo 8 | 9 | This is a sample application which showcases how to use Step Functions with AWS SAM, SDK integrations and 10 | local testing using mock configs. 11 | 12 | In this example you won't find any Lambda function, but instead direct calls from the state machine to the different AWS Services. 13 | 14 | We are also showing how to test locally the state machine, using mock configs. Mock configs allows a developer to mock the output of service integrations that are present in a state machine. 15 | 16 | Developers can provide a valid sample output from the service call API that is present in the state machine as mock data. This allows developers to test the behavior of the state machine in isolation. 17 | 18 | In this demo you will see: 19 | 20 | - How to create a State Machine using ASL and the AWS SDK integrations 21 | - How to define a state machine using AWS SAM 22 | - How to transcribe an existing video/audio file and translate it into spanish 23 | - How to test locally a State Machine 24 | - How to create a MockConfigFile 25 | - How to run this automatically 26 | 27 | This demo is part of a video series posted in FooBar Serverless channel. You can check the videos to see the whole demo. 28 | 29 | ## Overview of the state machine 30 | 31 | This state machine takes a video or audio in English, stored in an S3 bucket. Transcribe this file with Amazon Transcribe, then grabs the transcriptions and gets the transcript for the text using Amazon Transcribe. And finally with the transcribed text uses Amazon Translate to get the spanish version of the text. 32 | 33 | ![StateMachine](https://d2908q01vomqb2.cloudfront.net/da4b9237bacccdf19c0760cab7aec4a8359010b0/2021/09/08/07-statemachine-750x1024.png) 34 | 35 | ## Deploy this demo 36 | 37 | We will be using AWS SAM and make sure you are running the latest version - at the time of writing, this was 1.37.0 (sam --version). 38 | 39 | Deploy the project to the cloud: 40 | 41 | ``` 42 | sam deploy -g # Guided deployments 43 | ``` 44 | 45 | You will need to input the S3 bucket and object key where your audio/video file is stored. 46 | 47 | Next times, when you update the code, you can build and deploy with: 48 | 49 | ``` 50 | sam deploy 51 | ``` 52 | 53 | To delete the app: 54 | 55 | ``` 56 | sam delete 57 | ``` 58 | 59 | ## Test locally 60 | 61 | To test the state machine locally you will first need to download the step functions local docker image. 62 | 63 | Following the user guide, you can either run Step Functions local as a [JAR](https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-computer.html) or as a [Docker container](https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-docker.html). Steps below will focus on the docker container approach. 64 | 65 | ``` 66 | docker pull amazon/aws-stepfunctions-local 67 | ``` 68 | 69 | Then you can run all the tests in the `makefile` 70 | 71 | On a terminal window, first start with running docker: 72 | 73 | ```bash 74 | make run 75 | ``` 76 | 77 | On a different terminal window/tab, you can then run: 78 | 79 | ```bash 80 | make all 81 | ``` 82 | 83 | Finally, you can check history of each execution by running: 84 | 85 | ```bash 86 | make history 87 | ``` 88 | 89 | Check [`makefile`](./makefile) for details 90 | 91 | ## Links related to this code 92 | 93 | - Video about Step function Service Integration: https://youtu.be/jtmQJqaInT0 94 | - Video about testing Step functions using Mocked service integrations: https://youtu.be/P3hEqxKxZe8 95 | - Launch blog post of Step function SDK service integration: https://aws.amazon.com/blogs/aws/now-aws-step-functions-supports-200-aws-services-to-enable-easier-workflow-automation/ 96 | - Documentation on using Mocked Service Integration: https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-test-sm-exec.html#run-mocked-serv-integ-tests 97 | - Launch blog post of Mocked Service Integration: https://aws.amazon.com/blogs/compute/mocking-service-integrations-with-aws-step-functions-local/ 98 | --------------------------------------------------------------------------------