├── heaviside ├── tests │ ├── sfn │ │ ├── error_unterminated_quote.sfn │ │ ├── error_unexpected_token.sfn │ │ ├── error_unterminated_multiquote.sfn │ │ ├── error_invalid_task_arn.sfn │ │ ├── error_missing_task_function.sfn │ │ ├── error_invalid_task_service.sfn │ │ ├── error_invalid_wait_seconds.sfn │ │ ├── error_invalide_task_service.sfn │ │ ├── error_missing_task_function_argument.sfn │ │ ├── error_unexpected_task_function.sfn │ │ ├── error_invalid_task_function.sfn │ │ ├── error_invalid_timeout.sfn │ │ ├── error_unexpected_heartbeat.sfn │ │ ├── error_unexpected_retry.sfn │ │ ├── error_unexpected_timeout.sfn │ │ ├── error_invalid_heartbeat2.sfn │ │ ├── error_invalid_retry_delay.sfn │ │ ├── error_unexpected_task_argument.sfn │ │ ├── error_invalid_retry_backoff.sfn │ │ ├── error_unexpected_catch.sfn │ │ ├── error_unexpected_data.sfn │ │ ├── error_unexpected_input.sfn │ │ ├── error_unexpected_output.sfn │ │ ├── error_unexpected_result.sfn │ │ ├── error_duplicate_state_name.sfn │ │ ├── error_invalid_multiple_input.sfn │ │ ├── error_invalid_task_sync_value.sfn │ │ ├── error_missing_task_keyword_argument.sfn │ │ ├── error_invalid_task_keyword_argument.sfn │ │ ├── error_invalid_goto_target.sfn │ │ ├── error_unexpected_task_keyword_argument.sfn │ │ ├── error_invalid_heartbeat.sfn │ │ ├── error_invalid_state_name.sfn │ │ ├── error_iterator_used_by_non_map_state.sfn │ │ ├── error_map_has_no_iterator.sfn │ │ └── error_map_iterator_duplicate_state_name.sfn │ ├── __init__.py │ ├── utils.py │ ├── test_compile.py │ ├── test_utils.py │ └── test_statemachine.py ├── exceptions.py ├── lexer.py ├── aws_services.json ├── __main__.py ├── utils.py ├── __init__.py ├── sfn.py └── parser.py ├── requirements.txt ├── tests ├── unicode.hsd ├── nested_while.hsd ├── unicode.sfn ├── test.sh ├── nested_while.sfn ├── full.hsd └── full.sfn ├── coverage.sh ├── .gitignore ├── .circleci └── config.yml ├── pyproject.toml ├── CHANGELOG.md ├── examples ├── sfn_example.py └── definition.sfn ├── docs ├── CompilerPipeline.md ├── LibraryAPI.md └── StepFunctionDSL.md ├── CONTRIBUTING.md ├── README.md └── LICENSE /heaviside/tests/sfn/error_unterminated_quote.sfn: -------------------------------------------------------------------------------- 1 | " 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_token.sfn: -------------------------------------------------------------------------------- 1 | input: 'foo' 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unterminated_multiquote.sfn: -------------------------------------------------------------------------------- 1 | """ 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_task_arn.sfn: -------------------------------------------------------------------------------- 1 | Arn('not:an:arn') 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_missing_task_function.sfn: -------------------------------------------------------------------------------- 1 | DynamoDB() 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_task_service.sfn: -------------------------------------------------------------------------------- 1 | UnknownService() 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_wait_seconds.sfn: -------------------------------------------------------------------------------- 1 | Wait(seconds = 0) 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalide_task_service.sfn: -------------------------------------------------------------------------------- 1 | UnknownService() 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_missing_task_function_argument.sfn: -------------------------------------------------------------------------------- 1 | Lambda() 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_task_function.sfn: -------------------------------------------------------------------------------- 1 | Lambda.Task() 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_task_function.sfn: -------------------------------------------------------------------------------- 1 | DynamoDB.InvalidFunction() 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_timeout.sfn: -------------------------------------------------------------------------------- 1 | Lambda('Test') 2 | timeout: 0 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_heartbeat.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | heartbeat: 60 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_retry.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | retry [] 1 0 1.0 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_timeout.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | timeout: 10 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_heartbeat2.sfn: -------------------------------------------------------------------------------- 1 | Lambda('Test') 2 | heartbeat: 0 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_retry_delay.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | retry [] 0 0 1.0 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_task_argument.sfn: -------------------------------------------------------------------------------- 1 | DynamoDB.GetItem('unexpected') 2 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_retry_backoff.sfn: -------------------------------------------------------------------------------- 1 | Lambda('Test') 2 | retry [] 1 0 0.0 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_catch.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | catch []: 3 | Pass() 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_data.sfn: -------------------------------------------------------------------------------- 1 | Success() 2 | data: 3 | {'one': 1} 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_input.sfn: -------------------------------------------------------------------------------- 1 | Fail("ERROR", "Cause message") 2 | input: '$' 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_output.sfn: -------------------------------------------------------------------------------- 1 | Fail("ERROR", "Cause message") 2 | output: '$' 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_result.sfn: -------------------------------------------------------------------------------- 1 | Fail("ERROR", "Cause message") 2 | result: '$' 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_duplicate_state_name.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | "Test" 3 | 4 | Pass() 5 | "Test" 6 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_multiple_input.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | input: '$.foo' 3 | input: '$.foo' 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_task_sync_value.sfn: -------------------------------------------------------------------------------- 1 | DynamoDB.GetItem() 2 | parameters: 3 | sync: 'A String' 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_missing_task_keyword_argument.sfn: -------------------------------------------------------------------------------- 1 | Batch.SubmitJob() 2 | parameters: 3 | JobName: 'Name' 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_task_keyword_argument.sfn: -------------------------------------------------------------------------------- 1 | DynamoDB.GetItem() 2 | parameters: 3 | Invalid: 'Keyword Argument' 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_goto_target.sfn: -------------------------------------------------------------------------------- 1 | 2 | parallel: 3 | Pass() 4 | "Target" 5 | 6 | parallel: 7 | goto "Target" 8 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_unexpected_task_keyword_argument.sfn: -------------------------------------------------------------------------------- 1 | Lambda('function') 2 | parameters: 3 | Unexpected: 'Keyword Argument' 4 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_heartbeat.sfn: -------------------------------------------------------------------------------- 1 | Lambda('arn:aws:lambda:region:account:function:FUNCTION_NAME') 2 | timeout: 60 3 | heartbeat: 60 4 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Check 2 | hvac>=0.11.2, <1.0.0 3 | setuptools>=75.3.0 4 | funcparserlib==1.0.1 5 | iso8601 6 | boto3>=1.4.3 7 | importlib_resources 8 | -------------------------------------------------------------------------------- /tests/unicode.hsd: -------------------------------------------------------------------------------- 1 | """Unicode support test 2 | 3 | »©« 4 | """ 5 | 6 | Lambda('∑') 7 | 8 | if "$.£" == '√': 9 | Fail('!√', '$.→') 10 | """Ø""" 11 | -------------------------------------------------------------------------------- /tests/nested_while.hsd: -------------------------------------------------------------------------------- 1 | while '$.foo' == True: 2 | while '$.foo' == True: 3 | while '$.foo' == True: 4 | while '$.foo' == True: 5 | Pass() 6 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_invalid_state_name.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 3 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_iterator_used_by_non_map_state.sfn: -------------------------------------------------------------------------------- 1 | Pass() 2 | iterator: 3 | Pass() 4 | """MapPassState""" 5 | Wait(seconds=2) 6 | """WaitState""" 7 | -------------------------------------------------------------------------------- /coverage.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | coverage erase 4 | 5 | coverage2 run -a -m unittest heaviside.tests 6 | coverage3 run -a -m unittest heaviside.tests 7 | 8 | for input in `ls tests/*.hsd`; do 9 | coverage2 run -a bin/heaviside -r '' -a '' compile $input -o tmp.sfn 10 | coverage3 run -a bin/heaviside -r '' -a '' compile $input -o tmp.sfn 11 | done 12 | if [ -f tmp.sfn ] ; then 13 | rm tmp.sfn 14 | fi 15 | 16 | coverage annotate --include=heaviside/*,bin/* -d tmp 17 | coverage report --include=heaviside/*,bin/* 18 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_map_has_no_iterator.sfn: -------------------------------------------------------------------------------- 1 | version: "1.0" 2 | timeout: 60 3 | Pass() 4 | """CreateSomeInputs""" 5 | result: '$' 6 | data: 7 | { 8 | "the_array": [1, 2, 3, 4, 5], 9 | "foo": "bar" 10 | } 11 | map: 12 | """TransformInputsWithMap""" 13 | parameters: 14 | foo.$: "$.foo" 15 | element.$: "$$.Map.Item.Value" 16 | items_path: "$.the_array" 17 | result: "$.transformed" 18 | output: "$.transformed" 19 | max_concurrency: 4 20 | retry [] 1 0 1.0 21 | catch []: 22 | Pass() 23 | """SomeErrorHandler""" 24 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # Distribution / packaging 6 | .Python 7 | env/ 8 | build/ 9 | develop-eggs/ 10 | dist/ 11 | downloads/ 12 | eggs/ 13 | .eggs/ 14 | lib64/ 15 | parts/ 16 | sdist/ 17 | var/ 18 | *.egg-info/ 19 | .installed.cfg 20 | *.egg 21 | 22 | # PyInstaller 23 | # Usually these files are written by a python script from a template 24 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 25 | *.manifest 26 | *.spec 27 | 28 | # Installer logs 29 | pip-log.txt 30 | pip-delete-this-directory.txt 31 | 32 | # PyCharm Stuff 33 | .idea/ 34 | 35 | # Vim Swap Files 36 | .*.swp 37 | 38 | # VS Code Files 39 | .vscode/ 40 | -------------------------------------------------------------------------------- /tests/unicode.sfn: -------------------------------------------------------------------------------- 1 | { 2 | "States": { 3 | "Line6": { 4 | "Resource": "arn:aws:lambda:::function:∑", 5 | "Type": "Task", 6 | "Next": "Line8" 7 | }, 8 | "Line8": { 9 | "Default": "Line8Default", 10 | "Type": "Choice", 11 | "Choices": [ 12 | { 13 | "Variable": "$.£", 14 | "StringEquals": "√", 15 | "Next": "Ø" 16 | } 17 | ] 18 | }, 19 | "Line8Default": { 20 | "Type": "Succeed" 21 | }, 22 | "Ø": { 23 | "Cause": "$.→", 24 | "Type": "Fail", 25 | "Error": "!√" 26 | } 27 | }, 28 | "Comment": "Unicode support test\n\n»©«\n", 29 | "StartAt": "Line6" 30 | } -------------------------------------------------------------------------------- /tests/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | which statelint > /dev/null 4 | if [ $? -ne 0 ] ; then 5 | echo "Need statelint installed on system, cannot run" 6 | exit 1 7 | fi 8 | 9 | for input in `ls *.hsd`; do 10 | output="${input%.*}.sfn" 11 | tmp="tmp.sfn" 12 | ../bin/heaviside -r '' -a '' compile $input -o $tmp 2> /dev/null 13 | if [ $? -eq 0 ] ; then 14 | echo "Verifying results for ${input}" 15 | # if there is an error, the compiler prints the error message 16 | diff -u $tmp $output 17 | if [ $? -eq 0 ] ; then 18 | # if there is an error, the compiler prints the error message 19 | statelint $output 20 | fi 21 | fi 22 | if [ -f $tmp ] ; then 23 | rm $tmp 24 | fi 25 | done 26 | -------------------------------------------------------------------------------- /heaviside/tests/sfn/error_map_iterator_duplicate_state_name.sfn: -------------------------------------------------------------------------------- 1 | version: "1.0" 2 | timeout: 60 3 | Pass() 4 | """CreateSomeInputs""" 5 | result: '$' 6 | data: 7 | { 8 | "the_array": [1, 2, 3, 4, 5], 9 | "foo": "bar" 10 | } 11 | map: 12 | """TransformInputsWithMap""" 13 | iterator: 14 | Lambda('myfunc') 15 | """DuplicateName""" 16 | Success() 17 | """DuplicateName""" 18 | parameters: 19 | foo.$: "$.foo" 20 | element.$: "$$.Map.Item.Value" 21 | items_path: "$.the_array" 22 | result: "$.transformed" 23 | output: "$.transformed" 24 | max_concurrency: 4 25 | retry [] 1 0 1.0 26 | catch []: 27 | Pass() 28 | """SomeErrorHandler""" 29 | 30 | -------------------------------------------------------------------------------- /heaviside/tests/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from .test_activities import * 16 | from .test_compile import * 17 | from .test_statemachine import * 18 | from .test_utils import * 19 | -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2.1 2 | orbs: 3 | python: circleci/python@0.2.1 4 | 5 | commands: 6 | install: 7 | description: "Install Python dependencies" 8 | parameters: 9 | pyversion: 10 | type: string 11 | steps: 12 | - checkout 13 | - python/load-cache 14 | - python/install-deps 15 | - python/save-cache 16 | 17 | test_heaviside: 18 | description: "Test the step function activitities" 19 | steps: 20 | - run: python3 -m unittest discover 21 | 22 | jobs: 23 | test_py3_8: 24 | docker: 25 | - image: cimg/python:3.8 26 | steps: 27 | - install: 28 | pyversion: python3.8 29 | - test_heaviside 30 | 31 | test_py3_9: 32 | docker: 33 | - image: cimg/python:3.9 34 | steps: 35 | - install: 36 | pyversion: python3.9 37 | - test_heaviside 38 | 39 | test_py3_10: 40 | docker: 41 | - image: cimg/python:3.10 42 | steps: 43 | - install: 44 | pyversion: python3.10 45 | - test_heaviside 46 | 47 | test_py3_11: 48 | docker: 49 | - image: cimg/python:3.11 50 | steps: 51 | - install: 52 | pyversion: python3.11 53 | - test_heaviside 54 | 55 | workflows: 56 | test: 57 | jobs: 58 | - test_py3_8 59 | - test_py3_9 60 | - test_py3_10 61 | - test_py3_11 62 | 63 | -------------------------------------------------------------------------------- /tests/nested_while.sfn: -------------------------------------------------------------------------------- 1 | { 2 | "States": { 3 | "Line1": { 4 | "Default": "Line1Default", 5 | "Type": "Choice", 6 | "Choices": [ 7 | { 8 | "Variable": "$.foo", 9 | "BooleanEquals": true, 10 | "Next": "Line2" 11 | } 12 | ] 13 | }, 14 | "Line1Default": { 15 | "Type": "Succeed" 16 | }, 17 | "Line2": { 18 | "Default": "Line1Loop", 19 | "Type": "Choice", 20 | "Choices": [ 21 | { 22 | "Variable": "$.foo", 23 | "BooleanEquals": true, 24 | "Next": "Line3" 25 | } 26 | ] 27 | }, 28 | "Line3": { 29 | "Default": "Line2Loop", 30 | "Type": "Choice", 31 | "Choices": [ 32 | { 33 | "Variable": "$.foo", 34 | "BooleanEquals": true, 35 | "Next": "Line4" 36 | } 37 | ] 38 | }, 39 | "Line4": { 40 | "Default": "Line3Loop", 41 | "Type": "Choice", 42 | "Choices": [ 43 | { 44 | "Variable": "$.foo", 45 | "BooleanEquals": true, 46 | "Next": "Line5" 47 | } 48 | ] 49 | }, 50 | "Line5": { 51 | "Type": "Pass", 52 | "Next": "Line4Loop" 53 | }, 54 | "Line4Loop": { 55 | "Type": "Pass", 56 | "Next": "Line4" 57 | }, 58 | "Line3Loop": { 59 | "Type": "Pass", 60 | "Next": "Line3" 61 | }, 62 | "Line2Loop": { 63 | "Type": "Pass", 64 | "Next": "Line2" 65 | }, 66 | "Line1Loop": { 67 | "Type": "Pass", 68 | "Next": "Line1" 69 | } 70 | }, 71 | "StartAt": "Line1" 72 | } -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | # Copyright 2020 - 2204 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | [build_system] 16 | requires = ["setuptools>=75.3.0", "setuptools_scm[toml]>=8.1.0"] 17 | build_backend = "setuptools.build_meta" 18 | 19 | [project] 20 | name = "heaviside" 21 | version = "2.2.5" 22 | authors = [ 23 | { name="Derek Pryor" }, 24 | ] 25 | 26 | description = "Python library and DSL for working with AWS StepFunctions" 27 | readme = "README.md" 28 | license = {file = "LICENSE"} 29 | requires-python = ">=3.8" 30 | 31 | classifiers=[ 32 | 'Development Status :: 5 - Production/Stable', 33 | 'Intended Audience :: Developers', 34 | 'License :: OSI Approved :: Apache Software License', 35 | 'Programming Language :: Python :: 3', 36 | ] 37 | 38 | dependencies = [ 39 | "hvac>=0.11.2, <1.0.0", 40 | "setuptools>=75.3.0", 41 | "funcparserlib==1.0.1", 42 | "iso8601", 43 | "boto3>=1.4.3", 44 | "importlib_resources", 45 | ] 46 | 47 | keywords = [ 48 | "bossdb", 49 | "microns", 50 | "aws", 51 | "stepfunctions", 52 | "dsl" 53 | ] 54 | 55 | [project.scripts] 56 | heaviside = "heaviside:__main__.main" 57 | 58 | [tool.setuptools.package-data] 59 | heaviside = ["aws_services.json"] 60 | 61 | [project.urls] 62 | Homepage = "https://github.com/jhuapl-boss/heaviside" 63 | Issues = "https://github.com/jhuapl-boss/heaviside/issues" 64 | Changelog = "https://github.com/jhuapl-boss/heaviside/blob/master/CHANGELOG.md" 65 | -------------------------------------------------------------------------------- /heaviside/tests/utils.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | 16 | try: 17 | from unittest import mock 18 | except ImportError: 19 | import mock 20 | 21 | try: 22 | from pathlib import PosixPath 23 | except ImportError: 24 | from heaviside.utils import Path as PosixPath 25 | 26 | class MockPath(PosixPath): 27 | def __init__(self, path): 28 | super(MockPath, self).__init__() 29 | self.open = mock.MagicMock() 30 | # close() is done on the return of open() 31 | 32 | class MockSession(object): 33 | """Mock Boto3 Session object that creates unique mock objects 34 | for each client type requested""" 35 | 36 | def __init__(self, **kwargs): 37 | super(MockSession, self).__init__() 38 | 39 | self.clients = {} 40 | self.kwargs = kwargs 41 | self.region_name = "region" 42 | 43 | def client(self, name, config=None): 44 | """Create a new client session 45 | 46 | Args: 47 | name (string): Name of the client to connect to 48 | 49 | Returns: 50 | MagicMock: The unique MagicMock object for the requested client 51 | """ 52 | if name not in self.clients: 53 | self.clients[name] = mock.MagicMock() 54 | return self.clients[name] 55 | 56 | @property 57 | def _session(self): 58 | return self 59 | 60 | def set_config_variable(self, name, value): 61 | self.__dict__[name] = value 62 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Heaviside Changelog 2 | ## 2.2.5 3 | * upped setuptools to version 75.3.0 in requirements to avoid previous version's vulnerabilities 4 | * upped setuptools-scm to 8.1.0 5 | 6 | ## 2.2.4 7 | * Added support for Python 3.11 8 | * Switched from `setup.py` to `pyproject.toml` 9 | 10 | ## 2.2.3 11 | * Update funcparserlib version to ensure compatibility with Python 3.8 and above. 12 | * Dropped support for Python 3.7 13 | 14 | ## 2.2.2 15 | * Catch duplicate state names within a map's iterator 16 | 17 | ## 2.2.1 18 | * Fixed issues with Map state with while blocks and resolving ARNs 19 | 20 | ## 2.2.0 21 | * Added support for Map state 22 | 23 | ## 2.1.0 24 | * Fixed bug in packaging `aws_service.json` definition file 25 | * Fix bug in `heaviside.ast.StateVisitor` implementation 26 | * Added support for updating existing Step Functions 27 | * Added support for specifying parameters for Pass states 28 | 29 | ## 2.0 30 | * New features 31 | - Added support for `goto` control flow construct 32 | - Added support for AWS API tasks and passing parameters to them 33 | * Improvements 34 | - Cleaned up unicode handling so that any string can contain unicode characters 35 | - Created [Library API](docs/LibraryAPI.md) documentation that expands upon the older Activities documentation and contains all of the public API that Heaviside exposes 36 | 37 | ## 1.1 38 | * New features 39 | - Added new `activities.fanout_nonblocking` that allows the StepFunction to use a `Wait()` state instead of waiting within the Activity or Lambda 40 | - Integration tests that cover as many code paths as possible 41 | * Improvements 42 | - Unit tests 43 | - Documentation 44 | * Bug fixes 45 | - Nested `while` loop 46 | - `activities.fanout` error handling 47 | 48 | ## 1.0 49 | * Initial public release of Heaviside 50 | * Added AST into the compiler pipeline 51 | * Added `activities.fanout` 52 | * Added support for `switch`, `parallel`, `while` control flow constructs 53 | * Refactored activities code into mixins for better reuse 54 | * Multiple bug fixes 55 | 56 | ## 0.8 57 | * Initial release of Heaviside 58 | -------------------------------------------------------------------------------- /heaviside/exceptions.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # DP ???: Should this be a subclass of HeavisideError? 16 | class ActivityError(Exception): 17 | def __init__(self, error, cause): 18 | super(ActivityError, self).__init__(cause) 19 | self.error = error 20 | self.cause = cause 21 | 22 | def __str__(self): 23 | return "{}: {}".format(self.error, self.cause) 24 | 25 | class HeavisideError(Exception): 26 | """Base class for all Heaviside errors""" 27 | pass 28 | 29 | class CompileError(HeavisideError): 30 | """A syntax error was encountered when compiling a Heaviside file 31 | 32 | Printing a CompileError will produce a message similar to a 33 | Python SyntaxError. 34 | """ 35 | 36 | def __init__(self, lineno, pos, line, msg): 37 | """Args: 38 | lineno (int): Line number of the error 39 | pos (int): Position in the line where the error starts 40 | line (string): The line of the Heaviside file where the problem is 41 | msg (string): Syntax error message 42 | """ 43 | super(CompileError, self).__init__(msg) 44 | self.source = '' 45 | self.lineno = lineno 46 | self.pos = pos 47 | self.line = line 48 | self.msg = msg 49 | 50 | def __str__(self): 51 | # produce a Python style Syntax Error for display 52 | lines = [] 53 | lines.append('File "{}", line {}'.format(self.source, self.lineno)) 54 | # Note: Using format because of Python 2.7 string types 55 | lines.append('{}'.format(self.line)) 56 | lines.append((' ' * self.pos) + '^') 57 | lines.append('Syntax Error: {}'.format(self.msg)) 58 | return '\n'.join(lines) 59 | 60 | @classmethod 61 | def from_token(cls, tok, msg): 62 | lineno, pos = tok.start 63 | line = tok.line.rstrip() # remove newline 64 | return cls(lineno, pos, line, msg) 65 | 66 | @classmethod 67 | def from_tokenize(cls, error): 68 | msg, (lineno, pos) = error.args 69 | 70 | return cls(lineno, pos, '', msg) 71 | 72 | -------------------------------------------------------------------------------- /examples/sfn_example.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys 4 | import os 5 | import argparse 6 | from threading import Thread 7 | from pathlib import Path 8 | 9 | import alter_path 10 | from lib.stepfunctions import StateMachine, Activity 11 | 12 | sfn = """ 13 | Activity('Echo') 14 | Wait(seconds = 5) 15 | Activity('Echo') 16 | """ 17 | 18 | class BossStateMachine(StateMachine): 19 | def __init__(self, name, domain, *args, **kwargs): 20 | name_ = name + "." + domain 21 | name_ = ''.join([x.capitalize() for x in name_.split('.')]) 22 | super().__init__(name_, *args, **kwargs) 23 | self.domain = domain 24 | 25 | def _translate(self, function): 26 | return "{}.{}".format(function, self.domain) 27 | 28 | def run_activity(domain, count, credentials): 29 | activity = Activity('Echo.' + domain, credentials = credentials) 30 | 31 | activity.create() 32 | try: 33 | while count > 0: 34 | input_ = activity.task() 35 | if input_ is None: 36 | continue 37 | count -= 1 38 | 39 | print("Echo: {}".format(input_)) 40 | 41 | activity.success(input_) 42 | except Exception as e: 43 | print("Error: {}".format(e)) 44 | raise 45 | finally: 46 | activity.delete() 47 | 48 | if __name__ == '__main__': 49 | parser = argparse.ArgumentParser(description = "Example AWS Step Function script") 50 | parser.add_argument("--aws-credentials", "-a", 51 | metavar = "", 52 | default = os.environ.get("AWS_CREDENTIALS"), 53 | help = "File with credentials to use when connecting to AWS (default: AWS_CREDENTIALS)") 54 | parser.add_argument("domain_name", help="Domain in which to execute the configuration (example: subnet.vpc.boss)") 55 | 56 | args = parser.parse_args() 57 | 58 | if args.aws_credentials is None: 59 | parser.print_usage() 60 | print("Error: AWS credentials not provided and AWS_CREDENTIALS is not defined") 61 | sys.exit(1) 62 | 63 | credentials = Path(args.aws_credentials) 64 | domain = args.domain_name 65 | 66 | activity = Thread(target = run_activity, args = (domain, 2, credentials)) 67 | activity.start() 68 | 69 | machine = BossStateMachine('hello.world', domain, credentials = credentials) 70 | if machine.arn is None: 71 | role = "StatesExecutionRole-us-east-1" 72 | machine.create(sfn, role) 73 | else: 74 | for arn in machine.running_arns(): 75 | macine.stop(arn, "USER", "Script automatically stops old executions") 76 | 77 | args = {"input": "Hello World!"} 78 | print("Input: {}".format(args)) 79 | arn = machine.start(args) 80 | output = machine.wait(arn) 81 | print("Output: {}".format(output)) 82 | 83 | machine.delete() 84 | activity.join() 85 | -------------------------------------------------------------------------------- /heaviside/lexer.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import tokenize 16 | import token 17 | 18 | from .exceptions import CompileError 19 | 20 | USELESS = ['NEWLINE', 'NL', 'COMMENT'] 21 | 22 | # DP TODO: Use __slots__ and or a generator to reduce overhead for tokenizing large files 23 | class Token(object): 24 | """Custom class wrapping all token properties""" 25 | 26 | def __init__(self, code, value, start=(0,0), stop=(0,0), line=''): 27 | """ 28 | Args: 29 | code (string|int): Token code. Ints are translated using token.tok_name. 30 | value (string): Token value 31 | start (tuple): Pair of values describing token start line, start position 32 | stop (tuple): Pair of values describing token stop line, stop position 33 | line (string): String containing the line the token was parsed from 34 | """ 35 | try: 36 | self.code = token.tok_name[code] 37 | except: 38 | self.code = code 39 | self.value = value 40 | self.start = start 41 | self.stop = stop 42 | self.line = line 43 | 44 | def __str__(self): 45 | sl, sp = self.start 46 | el, ep = self.stop 47 | pos = '{},{}-{},{}'.format(sl, sp, el, ep) 48 | return '{}, {}, {!r}'.format(pos, self.code, self.value) 49 | 50 | def __repr__(self): 51 | return '{}({!r}, {!r}, {!r}, {!r}, {!r})'.format(self.__class__.__name__, 52 | self.code, 53 | self.value, 54 | self.start, 55 | self.stop, 56 | self.line) 57 | 58 | def __eq__(self, other): 59 | return (self.code, self.value) == (other.code, other.value) 60 | 61 | def tokenize_source(source): 62 | """Tokenize a source and convert into Token objects 63 | 64 | Results are filtered by through the list of USELESS tokens 65 | 66 | Args: 67 | source (callable): Callable object which provides the same interface as io.IOBase.readline 68 | Each call provides a line of input as bytes 69 | 70 | Returns: 71 | list: List of parsed tokens 72 | """ 73 | try: 74 | tokens = tokenize.generate_tokens(source) 75 | tokens = [Token(*t) for t in tokens] 76 | return [t for t in tokens if t.code not in USELESS] 77 | except tokenize.TokenError as e: 78 | raise CompileError.from_tokenize(e) 79 | 80 | -------------------------------------------------------------------------------- /docs/CompilerPipeline.md: -------------------------------------------------------------------------------- 1 | # Heaviside Compiler Pipeline 2 | 3 | This document describes the different stages of the Heaviside compiler. 4 | 5 | ## Table of Contents: 6 | 7 | * [Overview](#Overview) 8 | * [Lexer](#Lexer) 9 | * [Parser](#Parser) 10 | * [Abstract Syntax Tree](#Abstract-Syntax-Tree) 11 | - [Transformations](#Transformations) 12 | * [State Machine Generation](#State-Machine-Generation) 13 | 14 | ## Overview 15 | The design of the lexer and parser and heavily influenced by the tutorials from 16 | the [funcparserlib](https://github.com/vlasovskikh/funcparserlib) library used 17 | by Heaviside. 18 | 19 | Currently `compile` (`heaviside/__init__.py`) reads in the source file, 20 | tokenizes it, and then passes the results to the parser. The parser is then 21 | responsible for the rest of the pipeline. 22 | 23 | ## Lexer 24 | The lexer (`heaviside/lexer.py`) is just a wrapper around the Python tokenizer. 25 | This helps gives the Heaviside DSL a Python like style with minimal effort. The 26 | results of the Python tokenizer are wrapped in a custom class. 27 | 28 | ## Parser 29 | The parser (`heaviside/parser.py`) implements the Heaviside parsing logic 30 | using the [funcparserlib](https://github.com/vlasovskikh/funcparserlib) library. 31 | The parsed tokens are wrapped in Abstract Syntax Tree (AST) nodes that are built 32 | into a structure representing the source file. 33 | 34 | ## Abstract Syntax Tree 35 | The Abstract Syntax Tree (AST) (`heaviside/ast.py`) is used to represent the 36 | structure of the source file in memory. Each AST node contains the token from 37 | the source file that it represents. This allows raising an error anywhere 38 | during the compilation process and being able to attach the context of the 39 | error (location in the source and the text of the line with the error). 40 | 41 | ### Transformations 42 | After tokens are parsed into AST nodes several transformations are applied to 43 | the AST structure. The transformations are at the bottom of `heaviside/ast.py`. 44 | 45 | The most important transformation is linking the different states together. 46 | When parsed into AST nodes, they don't contain information about the next node 47 | to transition to (which has to be explicitly specified). Linking the nodes 48 | together adds information about the next state to transition to. It also moves 49 | the blocks of states for the different Choice State branches into the correct 50 | location in the AST structure (they exist at the same level as the Choice State). 51 | 52 | The other transforms are: 53 | * Checking state names to make sure they are not too long 54 | * Checking state names to make sure there are no duplicates in a branch 55 | * Constructing the full ARN for Task State (uses the AWS region and account ID) 56 | * Running any user provided `StateVisitor` based transformations 57 | 58 | ## State Machine Generation 59 | The final step is to convert the AST to the State Machine JSON representation. 60 | In `heaviside/sfn.py` are Python classes that represent each of the different 61 | State Machine structures. They are sub-classes of dict, so they can be 62 | serialized to JSON with minimal work. Each class takes the equivalent AST node 63 | as argument and creates the needed JSON entries. 64 | 65 | Because this is the final step, minimal checking is done to the data. There are 66 | a few checks that happen during this step, but the AST structure is expected to 67 | be mostly / completely compliant with the States Language specification. 68 | 69 | -------------------------------------------------------------------------------- /examples/definition.sfn: -------------------------------------------------------------------------------- 1 | """State machine comment""" 2 | 3 | version: "1.0" 4 | timeout: 60 5 | 6 | # ====== # 7 | # States # 8 | # ====== # 9 | 10 | 11 | Pass() # Do nothing state. Can be used to modify / inject data 12 | """Function Name 13 | Comments go here""" 14 | input: '$.input' 15 | result: '$.results' 16 | output: '$.output' 17 | data: # only for Pass states 18 | { 'a': 'a', 'b': true, 'c': False, 'd': null } 19 | 20 | Lambda('FUNCTION_NAME') 21 | """Lambda 22 | FUNCTION_NAME is used to create actual ARN 23 | input / results / output / data are all valid""" 24 | timeout: 60 # Lambda / Activity only 25 | heartbeat: 30 # Lambda / Activity only, must be less than timeout 26 | retry ["Error(s)"] 1 0 1.0 # retry interval (seconds), max attempts, backoff rate 27 | retry [] 1 0 1.0 # Empty error to match all errors (Same as State.ALL) 28 | catch ["Error(s)"]: 29 | Pass() 30 | catch []: # Same as above 31 | Pass() 32 | 33 | # Activity / Lambda ARNs can be just the function name, the whole, or 34 | # a partial end of the ARN (and the rest will be added) 35 | Activity('us-east-1:123456:activity:FUNCTION_NAME') 36 | # Activities are non-lambda functions that can run on EC2, ECS or anywhere else 37 | # The code just polls AWS to see if there is work available 38 | timeout: 2 39 | heartbeat: 1 40 | input: '$.foo' 41 | result: '$.foo' 42 | output: '$.foo' 43 | retry "one" 1 1 1 44 | retry ['two'] 1 1 1 45 | catch 'one': 46 | Pass() 47 | catch ['two']: '$.foo' 48 | Success() 49 | 50 | # Four different versions of a sleep function 51 | # input / output are valid 52 | Wait(seconds=30) 53 | #Wait(timestamp='yyyy-mm-ddThh:mm:ssZ') # RFC3339 formatted 54 | Wait(timestamp='1111-11-11T11:11:11Z') 55 | Wait(seconds_path='$.seconds') 56 | Wait(timestamp_path='$.timestamp') 57 | 58 | 59 | # ============ # 60 | # Flow Control # 61 | # ============ # 62 | 63 | # Comparison operators 64 | # ==, != for boolean 65 | # ==, !=, <, >, <=, >= for int/float/string/timestamp 66 | # Note: A timestamp is determined by trying to parse the string into the correct formatted 67 | # not, and, or are also supported with the written precedence 68 | # () are supported 69 | 70 | # if / elif / else becomes a Choice State 71 | if '$.a' == '1111-11-11T11:11:11Z' or \ 72 | '$.a' == 1: 73 | Pass() 74 | elif '$.foo' == 1: 75 | """If-Elif-Else""" 76 | Pass() 77 | elif '$.foo' <= 1: 78 | Pass() 79 | elif '$.foo' < 1: 80 | Pass() 81 | elif '$.foo' >= 1: 82 | Pass() 83 | elif '$.foo' > 1: 84 | Pass() 85 | elif '$.foo' != 1: 86 | Pass() 87 | elif '$.foo' == '1': 88 | Pass() 89 | elif '$.foo' <= '1': 90 | Pass() 91 | elif '$.foo' < '1': 92 | Pass() 93 | elif '$.foo' >= '1': 94 | Pass() 95 | elif '$.foo' > '1': 96 | Pass() 97 | elif '$.foo' != '1': 98 | Pass() 99 | elif '$.foo' == true: 100 | Pass() 101 | elif '$.foo' != true: 102 | Pass() 103 | elif '$.foo' == '1111-11-11T11:11:11Z': 104 | Pass() 105 | elif '$.foo' <= '1111-11-11T11:11:11Z': 106 | Pass() 107 | elif '$.foo' < '1111-11-11T11:11:11Z': 108 | Pass() 109 | elif '$.foo' >= '1111-11-11T11:11:11Z': 110 | Pass() 111 | elif '$.foo' > '1111-11-11T11:11:11Z': 112 | Pass() 113 | elif '$.foo' != '1111-11-11T11:11:11Z': 114 | Pass() 115 | else: 116 | Pass() 117 | transform: 118 | output: '$.foo' 119 | 120 | # while loop becomes a Choice State 121 | while '$.foo' == False: 122 | """While""" 123 | Pass() 124 | """While-Body""" 125 | transform: 126 | input: '$.foo' 127 | result: '$.foo' 128 | output: '$.foo' 129 | 130 | # switch statement becomes a Choice State 131 | # that only handles '==' cases 132 | switch '$.a': 133 | case 1: 134 | Pass() 135 | case 'foo': 136 | Pass() 137 | case '1111-11-11T11:11:11Z': 138 | Pass() 139 | default: 140 | Pass() 141 | transform: 142 | output: '$.foo' 143 | 144 | # parallel executes multiple branches in parallel 145 | # the errors block applies to the whole parallel 146 | # state (all of the branches together) 147 | parallel: 148 | """Parallel Name 149 | Only comments / step name from the first parallel block is used""" 150 | Success() 151 | """Success 152 | Comment""" 153 | input: '$.foo' 154 | output: '$.foo' 155 | 156 | parallel: 157 | Fail('error', 'cause') 158 | """Fail 159 | Comment""" 160 | transform: 161 | input: '$.foo' 162 | error: 163 | retry [] 1 0 0.0 164 | catch []: 165 | Pass() 166 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | To get started, [sign the Contributor License Agreement](https://www.clahub.com/agreements/jhuapl-boss/heaviside) 2 | 3 | 4 | ### The Johns Hopkins University Applied Physics Laboratory LLC 5 | 6 | Individual Contributor License Agreement ("Agreement") 7 | 8 | Thank you for your interest in open source software distributed and/or maintained by The Johns Hopkins University Applied Physics Laboratory LLC (“JHU/APL”). To provide consistent and clear intellectual property licenses to its users, JHU/APL must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor (whether an individual or entity), indicating agreement to the license terms below. This Agreement is for your protection as a Contributor as well as the protection of JHU/APL and its users/licensees; it does not change your rights to use your own Contributions for any other purpose. 9 | 10 | You accept and agree to the following terms and conditions for Your present and future Contributions submitted to JHU/APL. Except for the license granted herein to JHU/APL and recipients of software distributed by JHU/APL, You reserve all right, title, and interest in and to Your Contributions. 11 | 12 | 1. Definitions. 13 | 14 | "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with JHU/APL. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. 15 | 16 | "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to JHU/APL for inclusion in, or documentation of, any of the products owned or managed by JHU/APL (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to JHU/APL or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, JHU/APL for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." 17 | 18 | 2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You agree to grant and hereby grant to JHU/APL and to recipients of software distributed by JHU/APL a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. 19 | 20 | 3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You agree to grant and hereby grant to JHU/APL and to recipients of software distributed by JHU/APL a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. 21 | 22 | 4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to JHU/APL, or that your employer has executed a separate Corporate CLA with JHU/APL. 23 | 24 | 5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions. 25 | 26 | 6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. 27 | 28 | 7. Should You wish to submit work that is not Your original creation, You may submit it to JHU/APL separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]". 29 | 30 | 8. You agree to notify JHU/APL of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. 31 | 32 | 33 | To get started, [sign the Contributor License Agreement](https://www.clahub.com/agreements/jhuapl-boss/heaviside) 34 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Heaviside 2 | Heaviside is a domain specific language (DSL) and Python compiler / support 3 | libraries for working with [AWS StepFunctions]. 4 | 5 | ## Why 6 | The reason for a StepFunctions DSL is that the [state machine language], while 7 | flexible, is hard to write and maintain. The DSL provides a simplied format for 8 | writing StepFunction state machines and serveral of common flow control 9 | statements. 10 | 11 | ## DSL 12 | 13 | The [StepFunctionDSL] document describes the Heaviside DSL. 14 | 15 | ## Getting Started 16 | 17 | In this document `.hsd` will be used to denote a StepFunction file written in 18 | the Heaviside DSL. The extension `.sfn` will be used to denote a StepFunction 19 | file written in the AWS [state machine language]. 20 | 21 | ### Installing 22 | 23 | ``` 24 | pip install heaviside 25 | ``` 26 | 27 | ### CLI Script 28 | 29 | The Heaviside package installs a script called `heaviside` that provides a CLI 30 | to the library. Running the command without arguments or with the `--help` or 31 | `-h` flag will provide detailed help. 32 | 33 | #### AWS Credentials 34 | 35 | All sub-commands (except `compile`) connect to AWS to manipulate StepFunctions. 36 | There are multiple ways to define the AWS credentials, listed in the order of 37 | precedence. 38 | 39 | * Explicitly pass the values as command line arguments or environmental variables 40 | * Pass a file path to a JSON file with the arguments 41 | * Letting Boto3 try to resolve secret / access keys 42 | * Looking at EC2 meta data for current AWS region 43 | * Looking at current IAM user for AWS account_id 44 | 45 | The `compile` sub-command doesn't connect to AWS, but does use the region and 46 | account_id values when resolving Task ARNs. If the Heaviside DSL file has full 47 | Task ARNs or the compiled file will not be uploaded to AWS these values can be 48 | blank. 49 | 50 | **Note**: Since `compile` doesn't connect to AWS, only the first two options in 51 | the list above are valid for passing the region and account_id value. 52 | 53 | #### Compiling 54 | 55 | To compile a Heaviside file into a AWS StepFunction file use the `compile` 56 | sub-command. 57 | 58 | ``` 59 | $ heaviside compile state_machine.hsd -o state_machine.sfn 60 | ``` 61 | 62 | #### Creating a StepFunction 63 | 64 | The `heaviside` script can compile and upload the resulting file to AWS. 65 | 66 | ``` 67 | $ heaviside create state_machine.hsd AwsIamStepFunctionRole 68 | ``` 69 | 70 | Arguments: 71 | * `state_machine.hsd`: The path to the Step Function definition written in the 72 | Heaviside DSL. 73 | * `AwsIamStepFunctionRole`: The AWS IAM Role that the StepFunction will use 74 | when executing. Most often this will be used to 75 | control which Lambdas and Activities the 76 | StepFunction has permission to execute. 77 | 78 | #### Updating a StepFunction 79 | 80 | The `heaviside` script can update an existing Step Function definition and / or 81 | IAM role in AWS. 82 | 83 | ``` 84 | $ heaviside update state_machine --file state_machine.hsd --role AwsIamStepFunctionRole 85 | ``` 86 | 87 | Arguments: 88 | * `state_machine`: Name of the state machine to update 89 | * `state_machine.hsd`: The path to the Step Function definition written in the 90 | Heaviside DSL. 91 | * `AwsIamStepFunctionRole`: The AWS IAM Role that the StepFunction will use 92 | when executing. Most often this will be used to 93 | control which Lambdas and Activities the 94 | StepFunction has permission to execute. 95 | 96 | #### Deleting a StepFunction 97 | 98 | The `delete` sub-command can be used to delete an existing StepFunction 99 | 100 | ``` 101 | $ heaviside delete state_machine 102 | ``` 103 | 104 | #### Executing a StepFunction 105 | 106 | The `start` sub-command can be used to start executing a StepFunction. 107 | 108 | ``` 109 | $ heaviside start --json "{}" state_machine 110 | ``` 111 | 112 | **Note**: By default the `start` sub-command will wait until the 113 | execution has finished and will print the output of the StepFunction. 114 | 115 | ## Python Library 116 | 117 | The Heaviside package installs the Python library `heaviside`. The public 118 | API is documented in the [Library API](docs/LibraryAPI.md) file. 119 | 120 | ## Compatibility 121 | 122 | Currently, Heaviside has only been tested with Python 3.8 and 3.11 123 | 124 | ## Related Projects 125 | 126 | * [statelint]: A Ruby project that verifies a AWS StepFunction definition file. 127 | Includes checks like making sure that all states are reachable. 128 | Helpful when developing a new StepFunction to ensure everything 129 | is correct. It will catch structural problems that Heaviside 130 | doesn't check for. 131 | 132 | 133 | [AWS StepFunctions]: https://aws.amazon.com/step-functions/ 134 | [state machine language]: https://states-language.net/spec.html 135 | [StepFunctionDSL]: docs/StepFunctionDSL.md 136 | [Activities document]: docs/Activites.md 137 | [statelint]: https://github.com/awslabs/statelint 138 | 139 | ## Legal 140 | 141 | Use or redistribution of the Boss system in source and/or binary forms, with or without modification, are permitted provided that the following conditions are met: 142 | 143 | 1. Redistributions of source code or binary forms must adhere to the terms and conditions of any applicable software licenses. 144 | 2. End-user documentation or notices, whether included as part of a redistribution or disseminated as part of a legal or scientific disclosure (e.g. publication) or advertisement, must include the following acknowledgement: The Boss software system was designed and developed by the Johns Hopkins University Applied Physics Laboratory (JHU/APL). 145 | 3. The names "The Boss", "JHU/APL", "Johns Hopkins University", "Applied Physics Laboratory", "MICrONS", or "IARPA" must not be used to endorse or promote products derived from this software without prior written permission. For written permission, please contact BossAdmin@jhuapl.edu. 146 | 4. This source code and library is distributed in the hope that it will be useful, but is provided without any warranty of any kind. 147 | 148 | 149 | -------------------------------------------------------------------------------- /heaviside/aws_services.json: -------------------------------------------------------------------------------- 1 | { 2 | "Batch": { 3 | "SubmitJob": { 4 | "name": "submitJob", 5 | "sync": true, 6 | "required_keys": [ 7 | "JobDefinition", 8 | "JobName", 9 | "JobQueue" 10 | ], 11 | "optional_keys": [ 12 | "ArrayProperties", 13 | "ContainerOverrides", 14 | "DependsOn", 15 | "Parameters", 16 | "RetryStrategy", 17 | "Timeout" 18 | ] 19 | } 20 | }, 21 | "DynamoDB": { 22 | "GetItem": { 23 | "name": "getItem", 24 | "sync": null, 25 | "required_keys": [ 26 | "Key", 27 | "TableName" 28 | ], 29 | "optional_keys": [ 30 | "AttributesToGet", 31 | "ConsistentRead", 32 | "ExpressionAttributeNames", 33 | "ProjectionExpression", 34 | "ReturnConsumedCapacity" 35 | ] 36 | }, 37 | "PutItem": { 38 | "name": "putItem", 39 | "sync": null, 40 | "required_keys": [ 41 | "Item", 42 | "TableName" 43 | ], 44 | "optional_keys": [ 45 | "ConditionalOperator", 46 | "ConditionExpression", 47 | "Expected", 48 | "ExpressionAttributeNames", 49 | "ExpressionAttributeValues", 50 | "ReturnConsumedCapacity", 51 | "ReturnItemCollectionMetrics", 52 | "ReturnValues" 53 | ] 54 | }, 55 | "DeleteItem": { 56 | "name": "deleteItem", 57 | "sync": null, 58 | "required_keys": [ 59 | "Key", 60 | "TableName" 61 | ], 62 | "optional_keys": [ 63 | "ConditionalOperator", 64 | "ConditionExpression", 65 | "Expected", 66 | "ExpressionAttributeNames", 67 | "ExpressionAttributeValues", 68 | "ReturnConsumedCapacity", 69 | "ReturnItemCollectionMetrics", 70 | "ReturnValues" 71 | ] 72 | }, 73 | "UpdateItem": { 74 | "name": "updateItem", 75 | "sync": null, 76 | "required_keys": [ 77 | "Key", 78 | "TableName" 79 | ], 80 | "optional_keys": [ 81 | "AttributeUpdates", 82 | "ConditionalOperator", 83 | "ConditionExpression", 84 | "Expected", 85 | "ExpressionAttributeNames", 86 | "ExpressionAttributeValues", 87 | "ReturnConsumedCapacity", 88 | "ReturnItemCollectionMetrics", 89 | "ReturnValues", 90 | "UpdateExpression" 91 | ] 92 | } 93 | }, 94 | "ECS": { 95 | "RunTask": { 96 | "name": "runTask", 97 | "sync": true, 98 | "required_keys": [ 99 | "TaskDefinition" 100 | ], 101 | "optional_keys": [ 102 | "Cluster", 103 | "Group", 104 | "LaunchType", 105 | "NetworkConfiguration", 106 | "Overrides", 107 | "PlacementConstraints", 108 | "PlacementStrategy", 109 | "PlatformVersion" 110 | ] 111 | } 112 | }, 113 | "SNS": { 114 | "Publish": { 115 | "name": "publish", 116 | "sync": null, 117 | "required_keys": [ 118 | "Message" 119 | ], 120 | "optional_keys": [ 121 | "MessageAttributes", 122 | "MessageStructure", 123 | "PhoneNumber", 124 | "Subject", 125 | "TargetArn", 126 | "TopicArn" 127 | ] 128 | } 129 | }, 130 | "SQS": { 131 | "SendMessage": { 132 | "name": "sendMessage", 133 | "sync": null, 134 | "required_keys": [ 135 | "MessageBody", 136 | "QueueUrl" 137 | ], 138 | "optional_keys": [ 139 | "DelaySeconds", 140 | "MessageAttributes", 141 | "MessageDeduplicationId", 142 | "MessageGroupId" 143 | ] 144 | } 145 | }, 146 | "Glue": { 147 | "StartJobRun": { 148 | "name": "startJobRun", 149 | "sync": true, 150 | "required_keys": [ 151 | "JobName" 152 | ], 153 | "optional_keys": [ 154 | "JobRunId", 155 | "Arguments", 156 | "AllocatedCapacity", 157 | "Timeout", 158 | "SecurityConfiguration", 159 | "NotificationProperty" 160 | ] 161 | } 162 | }, 163 | "SageMaker": { 164 | "CreateTrainingJob": { 165 | "name": "createTrainingJob", 166 | "sync": true, 167 | "required_keys": [ 168 | "AlgorithmSpecification", 169 | "OutputDataConfig", 170 | "ResourceConfig", 171 | "RoleArn", 172 | "StoppingCondition", 173 | "TrainingJobName" 174 | ], 175 | "optional_keys": [ 176 | "HyperParameters", 177 | "InputDataConfig", 178 | "Tags", 179 | "VpcConfig" 180 | ] 181 | }, 182 | "CreateTransformJob": { 183 | "name": "createTransformJob", 184 | "sync": true, 185 | "required_keys": [ 186 | "ModelName", 187 | "TransformInput", 188 | "TransformJobName", 189 | "TransformOutput", 190 | "TransformResources" 191 | ], 192 | "optional_keys": [ 193 | "BatchStrategy", 194 | "Environment", 195 | "MaxConcurrentTransforms", 196 | "MaxPayloadInMB", 197 | "Tags" 198 | ] 199 | } 200 | } 201 | } 202 | -------------------------------------------------------------------------------- /heaviside/tests/test_compile.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import os 16 | import sys 17 | import json 18 | import unittest 19 | from io import StringIO 20 | 21 | try: 22 | from unittest import mock 23 | except ImportError: 24 | import mock 25 | 26 | import heaviside 27 | Path = heaviside.utils.Path 28 | 29 | cur_dir = Path(os.path.dirname(os.path.realpath(__file__))) 30 | 31 | class TestCompile(unittest.TestCase): 32 | def execute(self, filename, error_msg): 33 | filepath = cur_dir / 'sfn' / filename 34 | 35 | try: 36 | out = heaviside.compile(filepath) 37 | self.assertFalse(True, "compile() should result in an exception") 38 | except heaviside.exceptions.CompileError as ex: 39 | actual = str(ex).split('\n')[-1] 40 | expected = "Syntax Error: {}".format(error_msg) 41 | self.assertEqual(actual, expected) 42 | 43 | def test_unterminated_quote(self): 44 | self.execute('error_unterminated_quote.sfn', 'Unterminated quote') 45 | 46 | def test_unterminated_multiquote(self): 47 | self.execute('error_unterminated_multiquote.sfn', 'EOF in multi-line string') 48 | 49 | def test_invalid_heartbeat(self): 50 | self.execute('error_invalid_heartbeat.sfn', "Heartbeat must be less than timeout (defaults to 60)") 51 | 52 | def test_invalid_heartbeat2(self): 53 | self.execute('error_invalid_heartbeat2.sfn', "'0' is not a positive integer") 54 | 55 | def test_invalid_timeout(self): 56 | self.execute('error_invalid_timeout.sfn', "'0' is not a positive integer") 57 | 58 | def test_unexpected_catch(self): 59 | self.execute('error_unexpected_catch.sfn', "Pass state cannot contain a Catch modifier") 60 | 61 | def test_unexpected_data(self): 62 | self.execute('error_unexpected_data.sfn', "Succeed state cannot contain a Data modifier") 63 | 64 | def test_unexpected_heartbeat(self): 65 | self.execute('error_unexpected_heartbeat.sfn', "Pass state cannot contain a Heartbeat modifier") 66 | 67 | def test_unexpected_input(self): 68 | self.execute('error_unexpected_input.sfn', "Fail state cannot contain a Input modifier") 69 | 70 | def test_unexpected_output(self): 71 | self.execute('error_unexpected_output.sfn', "Fail state cannot contain a Output modifier") 72 | 73 | def test_unexpected_result(self): 74 | self.execute('error_unexpected_result.sfn', "Fail state cannot contain a Result modifier") 75 | 76 | def test_unexpected_retry(self): 77 | self.execute('error_unexpected_retry.sfn', "Pass state cannot contain a Retry modifier") 78 | 79 | def test_unexpected_timeout(self): 80 | self.execute('error_unexpected_timeout.sfn', "Pass state cannot contain a Timeout modifier") 81 | 82 | def test_unexpected_token(self): 83 | self.execute('error_unexpected_token.sfn', 'Invalid syntax') 84 | 85 | def test_invalid_retry_delay(self): 86 | self.execute('error_invalid_retry_delay.sfn', "'0' is not a positive integer") 87 | 88 | def test_invalid_retry_backoff(self): 89 | self.execute('error_invalid_retry_backoff.sfn', "Backoff rate should be >= 1.0") 90 | 91 | def test_invalid_wait_seconds(self): 92 | self.execute('error_invalid_wait_seconds.sfn', "'0' is not a positive integer") 93 | 94 | def test_invalid_multiple_input(self): 95 | self.execute('error_invalid_multiple_input.sfn', "Pass state can only contain one Input modifier") 96 | 97 | def test_invalid_state_name(self): 98 | self.execute('error_invalid_state_name.sfn', "Name exceedes 128 characters") 99 | 100 | def test_duplicate_state_name(self): 101 | self.execute('error_duplicate_state_name.sfn', "Duplicate state name 'Test'") 102 | 103 | def test_invalid_goto_target(self): 104 | self.execute('error_invalid_goto_target.sfn', "Goto target 'Target' doesn't exist") 105 | 106 | def test_invalid_task_service(self): 107 | self.execute('error_invalid_task_service.sfn', "Invalid Task service") 108 | 109 | def test_missing_task_function_argument(self): 110 | self.execute('error_missing_task_function_argument.sfn', "Lambda task requires a function name argument") 111 | 112 | def test_missing_task_function(self): 113 | self.execute('error_missing_task_function.sfn', "DynamoDB task requires a function to call") 114 | 115 | def test_unexpected_task_function(self): 116 | self.execute('error_unexpected_task_function.sfn', "Unexpected function name") 117 | 118 | def test_invalid_task_function(self): 119 | self.execute('error_invalid_task_function.sfn', "Invalid Task function") 120 | 121 | def test_invalid_task_arn(self): 122 | self.execute('error_invalid_task_arn.sfn', "ARN must start with 'arn:aws:'") 123 | 124 | def test_unexpected_task_argument(self): 125 | self.execute('error_unexpected_task_argument.sfn', "Unexpected argument") 126 | 127 | def test_unexpected_task_keyword_argument(self): 128 | self.execute('error_unexpected_task_keyword_argument.sfn', "Unexpected keyword argument") 129 | 130 | def test_invalid_task_sync_value(self): 131 | self.execute('error_invalid_task_sync_value.sfn', "Synchronous value must be a boolean") 132 | 133 | def test_invalid_task_keyword_argument(self): 134 | self.execute('error_invalid_task_keyword_argument.sfn', "Invalid keyword argument") 135 | 136 | def test_missing_task_keyword_argument(self): 137 | self.execute('error_missing_task_keyword_argument.sfn', "Missing required keyword arguments: JobDefinition, JobQueue") 138 | 139 | def test_visitor(self): 140 | class TestVisitor(heaviside.ast.StateVisitor): 141 | def handle_task(self, task): 142 | task.arn = 'modified' 143 | 144 | hsd = u"""Lambda('function')""" 145 | out = heaviside.compile(hsd, visitors=[TestVisitor()]) 146 | out = json.loads(out) 147 | 148 | self.assertEqual(out['States']['Line1']['Resource'], 'modified') 149 | 150 | def test_invalid_iterator(self): 151 | self.execute('error_iterator_used_by_non_map_state.sfn', "Pass state cannot contain a Iterator modifier") 152 | 153 | def test_map_without_iterator(self): 154 | self.execute('error_map_has_no_iterator.sfn', 'Map state must have an iterator') 155 | 156 | def test_map_iterator_has_duplicate_state_name(self): 157 | self.execute('error_map_iterator_duplicate_state_name.sfn', "Duplicate state name 'DuplicateName'") 158 | -------------------------------------------------------------------------------- /tests/full.hsd: -------------------------------------------------------------------------------- 1 | """State machine comment""" 2 | 3 | version: "1.0" 4 | timeout: 60 5 | 6 | # ====== # 7 | # States # 8 | # ====== # 9 | 10 | Pass() 11 | """Pass 12 | Comment""" 13 | input: '$.foo' 14 | result: '$.foo' 15 | output: '$.foo' 16 | data: 17 | {} 18 | 19 | Lambda('FUNCTION_NAME') 20 | '''Lambda 21 | Comment''' 22 | timeout: 2 23 | heartbeat: 1 24 | input: '$.foo' 25 | result: '$.foo' 26 | output: '$.foo' 27 | retry "one" 1 1 1 28 | retry ['two'] 1 1 1 29 | catch 'one': 30 | Pass() 31 | catch ['two']: '$.foo' 32 | Success() 33 | 34 | Activity('FUNCTION_NAME') 35 | """Activity""" 36 | 37 | Arn('arn:aws:service:region:account:task_type:name') 38 | """Raw ARN Task""" 39 | parameters: 40 | KeyOne: 'ValueOne' 41 | KeyTwo: 'ValueTwo' 42 | 43 | Batch.SubmitJob() 44 | """Batch.SubmitJob""" 45 | parameters: 46 | JobName: 'Name' 47 | JobDefinition: '' 48 | JobQueue: 'arn' 49 | 50 | # Optional keyword arguments 51 | ArrayProperties: {} 52 | ContainerOverrides: {} 53 | DependsOn: [] 54 | Parameters: {} 55 | RetryStrategy: {} 56 | Timeout: {} 57 | sync: False 58 | 59 | DynamoDB.GetItem() 60 | """DynamoDB.GetItem""" 61 | parameters: 62 | TableName: 'Table' 63 | Key: {} 64 | 65 | # Optional keyword arguments 66 | AttributesToGet: [] 67 | ConsistentRead: True 68 | ExpressionAttributeNames: {} 69 | ProjectionExpression: '' 70 | ReturnConsumedCapacity: '' 71 | 72 | DynamoDB.PutItem() 73 | """DynamoDB.PutItem""" 74 | parameters: 75 | TableName: 'Table' 76 | Item: {} 77 | 78 | # Optional keyword arguments 79 | ConditionalOperator: '' 80 | ConditionExpression: '' 81 | Expected: {} 82 | ExpressionAttributeNames: {} 83 | ExpressionAttributeValues: {} 84 | ReturnConsumedCapacity: '' 85 | ReturnItemCollectionMetrics: '' 86 | ReturnValues: '' 87 | 88 | DynamoDB.DeleteItem() 89 | """DynamoDB.DeleteItem""" 90 | parameters: 91 | TableName: 'Table' 92 | Key: {} 93 | 94 | # Optional keyword arguments 95 | ConditionalOperator: '' 96 | ConditionExpression: '' 97 | Expected: {} 98 | ExpressionAttributeNames: {} 99 | ExpressionAttributeValues: {} 100 | ReturnConsumedCapacity: '' 101 | ReturnItemCollectionMetrics: '' 102 | ReturnValues: '' 103 | 104 | DynamoDB.UpdateItem() 105 | """DynamoDB.UpdateItem""" 106 | parameters: 107 | TableName: 'Table' 108 | Key: {} 109 | 110 | # Optional keyword arguments 111 | AttributeUpdates: {} 112 | ConditionalOperator: '' 113 | ConditionExpression: '' 114 | Expected: {} 115 | ExpressionAttributeNames: {} 116 | ExpressionAttributeValues: {} 117 | ReturnConsumedCapacity: '' 118 | ReturnItemCollectionMetrics: '' 119 | ReturnValues: '' 120 | UpdateExpression: '' 121 | 122 | ECS.RunTask() 123 | """ECS.RunTask""" 124 | parameters: 125 | TaskDefinition: '' 126 | 127 | # Optional keyword arguments 128 | Cluster: '' 129 | Group: '' 130 | LaunchType: '' 131 | NetworkConfiguration: {} 132 | Overrides: {} 133 | PlacementConstraints: {} 134 | PlacementStrategy: {} 135 | PlatformVersion: '' 136 | 137 | SNS.Publish() 138 | """SNS.Publish""" 139 | parameters: 140 | Message: '' 141 | 142 | # Optional keyword arguments 143 | MessageAttributes: {} 144 | MessageStructure: '' 145 | PhoneNumber: '' 146 | Subject: '' 147 | TargetArn: '' 148 | TopicArn: '' 149 | 150 | SQS.SendMessage() 151 | """SQS.SendMessage""" 152 | parameters: 153 | QueueUrl: '' 154 | MessageBody: '' 155 | 156 | # Optional keyword arguments 157 | DelaySeconds: 0 158 | MessageAttributes: {} 159 | MessageDeduplicationId: '' 160 | MessageGroupId: '' 161 | 162 | Glue.StartJobRun() 163 | """Glue.StartJobRun""" 164 | parameters: 165 | JobName: '' 166 | 167 | # Optional keyword arguments 168 | JobRunId: '' 169 | Arguments: {} 170 | AllocatedCapacity: 0 171 | Timeout: 1 172 | SecurityConfiguration: '' 173 | NotificationProperty: {} 174 | 175 | SageMaker.CreateTrainingJob() 176 | """SageMaker.CreateTrainingJob""" 177 | parameters: 178 | TrainingJobName: '' 179 | AlgorithmSpecification: {} 180 | OutputDataConfig: {} 181 | ResourceConfig: {} 182 | RoleArn: '' 183 | StoppingCondition: {} 184 | 185 | # Optional keyword arguments 186 | HyperParameters: {} 187 | InputDataConfig: [] 188 | Tags: [] 189 | VpcConfig: {} 190 | 191 | SageMaker.CreateTransformJob() 192 | """SageMaker.CreateTransformJob""" 193 | parameters: 194 | TransformJobName: '' 195 | ModelName: '' 196 | TransformInput: {} 197 | TransformOutput: {} 198 | TransformResources: {} 199 | 200 | # Optional keyword arguments 201 | BatchStrategy: '' 202 | Environment: {} 203 | MaxConcurrentTransforms: 0 204 | MaxPayloadInMB: 0 205 | Tags: [] 206 | 207 | Wait(seconds=1) 208 | """Wait-Seconds""" 209 | Wait(timestamp='1111-11-11T11:11:11Z') 210 | """Wait-Timestamp""" 211 | Wait(seconds_path='$.foo') 212 | """Wait-Seconds-Path""" 213 | Wait(timestamp_path='$.foo') 214 | """Wait-Timestamp-Path 215 | Comment""" 216 | input: '$.foo' 217 | output: '$.foo' 218 | 219 | # ============ # 220 | # Flow Control # 221 | # ============ # 222 | 223 | while '$.foo' == 1: 224 | """While""" 225 | Pass() 226 | """While-Body""" 227 | transform: 228 | input: '$.foo' 229 | output: '$.foo' 230 | 231 | if '$.foo' == 1 or ('$.foo' >= 10 and (not '$.foo' < 20)): 232 | """If-Elif-Else""" 233 | Pass() 234 | elif '$.foo' <= 1: 235 | Pass() 236 | elif '$.foo' < 1: 237 | Pass() 238 | elif '$.foo' >= 1: 239 | Pass() 240 | elif '$.foo' > 1: 241 | Pass() 242 | elif '$.foo' != 1: 243 | Pass() 244 | elif '$.foo' == '1': 245 | Pass() 246 | elif '$.foo' <= '1': 247 | Pass() 248 | elif '$.foo' < '1': 249 | Pass() 250 | elif '$.foo' >= '1': 251 | Pass() 252 | elif '$.foo' > '1': 253 | Pass() 254 | elif '$.foo' != '1': 255 | Pass() 256 | elif '$.foo' == true: 257 | Pass() 258 | elif '$.foo' != true: 259 | Pass() 260 | elif '$.foo' == '1111-11-11T11:11:11Z': 261 | Pass() 262 | elif '$.foo' <= '1111-11-11T11:11:11Z': 263 | Pass() 264 | elif '$.foo' < '1111-11-11T11:11:11Z': 265 | Pass() 266 | elif '$.foo' >= '1111-11-11T11:11:11Z': 267 | Pass() 268 | elif '$.foo' > '1111-11-11T11:11:11Z': 269 | Pass() 270 | elif '$.foo' != '1111-11-11T11:11:11Z': 271 | Pass() 272 | else: 273 | Pass() 274 | transform: 275 | input: '$.foo' 276 | output: '$.foo' 277 | 278 | switch '$.a': 279 | """Switch""" 280 | case 1: 281 | Pass() 282 | case 'foo': 283 | Pass() 284 | case '1111-11-11T11:11:11Z': 285 | Pass() 286 | case false: 287 | Pass() 288 | default: 289 | Pass() 290 | transform: 291 | input: '$.foo' 292 | output: '$.foo' 293 | 294 | parallel: 295 | """Parallel""" 296 | Success() 297 | """Success 298 | Comment""" 299 | # DP NOTE 2019/05/19: AWS Console considers InputPath / OutputPath 300 | # invalid for the Succeed state, even though the language definition 301 | # says it is valid, but will allow creating a Step Function with them 302 | input: '$.foo' 303 | output: '$.foo' 304 | 305 | parallel: 306 | Fail('error', 'cause') 307 | """Fail 308 | Comment""" 309 | 310 | transform: 311 | input: '$.foo' 312 | result: '$.foo' 313 | output: '$.foo' 314 | error: 315 | retry [] 1 0 1.0 316 | catch []: 317 | goto 'Switch' 318 | -------------------------------------------------------------------------------- /heaviside/tests/test_utils.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import unicode_literals 16 | 17 | import unittest 18 | import sys 19 | from io import StringIO 20 | 21 | try: 22 | from unittest import mock 23 | except ImportError: 24 | import mock 25 | 26 | from .utils import MockPath, MockSession 27 | from heaviside import utils 28 | 29 | class TestRead(unittest.TestCase): 30 | def test_path(self): 31 | name = "/path/to/file" 32 | path = MockPath(name) 33 | 34 | with utils.read(path) as fh: 35 | pass 36 | 37 | # close() is on the fh returned by open() 38 | calls = [mock.call(), mock.call().close()] 39 | self.assertEqual(path.open.mock_calls, calls) 40 | 41 | def test_path_close(self): 42 | name = "/path/to/file" 43 | path = MockPath(name) 44 | 45 | with self.assertRaises(Exception): 46 | with utils.read(path) as fh: 47 | raise Exception() 48 | 49 | # close() is on the fh returned by open() 50 | calls = [mock.call(), mock.call().close()] 51 | self.assertEqual(path.open.mock_calls, calls) 52 | 53 | def test_string(self): 54 | data = "foo bar boo" 55 | 56 | with utils.read(data) as fh: 57 | self.assertEqual(data, fh.read()) 58 | self.assertEqual('', fh.name) 59 | 60 | def test_fileobj(self): 61 | data = "foo bar boo" 62 | obj = StringIO(data) 63 | 64 | with utils.read(obj) as fh: 65 | self.assertEqual(obj, fh) 66 | self.assertEqual(data, fh.read()) 67 | 68 | def test_unsuported(self): 69 | with self.assertRaises(ValueError): 70 | with utils.read(None) as fh: 71 | pass 72 | 73 | class TestWrite(unittest.TestCase): 74 | def test_stdout(self): 75 | with utils.write('-') as fh: 76 | self.assertEqual(sys.stdout, fh) 77 | 78 | @mock.patch.object(utils.Path, 'open') 79 | def test_string(self, mOpen): 80 | path = '/path/to/file' 81 | data = "foo bar boo" 82 | 83 | with utils.write(path) as fh: 84 | fh.write(data) 85 | 86 | expected = [mock.call('w'), 87 | mock.call().write(data), 88 | mock.call().close()] 89 | self.assertEqual(mOpen.mock_calls, expected) 90 | 91 | @mock.patch.object(utils.Path, 'open') 92 | def test_path(self, mOpen): 93 | path = utils.Path('/path/to/file') 94 | data = "foo bar boo" 95 | 96 | with utils.write(path) as fh: 97 | fh.write(data) 98 | 99 | expected = [mock.call('w'), 100 | mock.call().write(data), 101 | mock.call().close()] 102 | self.assertEqual(mOpen.mock_calls, expected) 103 | 104 | def test_fileobj(self): 105 | data = "foo bar boo" 106 | obj = StringIO() 107 | 108 | with utils.write(obj) as fh: 109 | self.assertEqual(obj, fh) 110 | fh.write(data) 111 | 112 | obj.seek(0) 113 | self.assertEqual(data, obj.read()) 114 | 115 | def test_unsuported(self): 116 | with self.assertRaises(ValueError): 117 | with utils.write(None) as fh: 118 | pass 119 | 120 | class TestCreateSession(unittest.TestCase): 121 | @mock.patch('heaviside.utils.Session', autospec=True) 122 | def test_noargs(self, mSession): 123 | iSession = mSession.return_value = MockSession() 124 | account = '123456' 125 | region = 'east' 126 | 127 | session, account_id = utils.create_session(account_id=account, region=region) 128 | 129 | self.assertEqual(session, iSession) 130 | self.assertEqual(account_id, account) 131 | self.assertEqual(iSession.region, region) 132 | 133 | calls = [mock.call()] 134 | self.assertEqual(mSession.mock_calls, calls) 135 | 136 | @mock.patch('heaviside.utils.Session', autospec=True) 137 | def test_session(self, mSession): 138 | iSession = MockSession() 139 | account = '123456' 140 | 141 | session, account_id = utils.create_session(account_id=account, session=iSession) 142 | 143 | self.assertEqual(session, iSession) 144 | self.assertEqual(account_id, account) 145 | self.assertEqual(mSession.mock_calls, []) 146 | 147 | @mock.patch('heaviside.utils.Session', autospec=True) 148 | def test_credentials(self, mSession): 149 | iSession = mSession.return_value = MockSession() 150 | access = 'XXX' 151 | secret = 'YYY' 152 | account = '123456' 153 | region = 'east' 154 | credentials = { 155 | 'aws_access_key': access, 156 | 'aws_secret_key': secret, 157 | 'aws_account_id': account, 158 | 'aws_region': region 159 | } 160 | 161 | session, account_id = utils.create_session(credentials = credentials) 162 | 163 | self.assertEqual(session, iSession) 164 | self.assertEqual(account_id, account) 165 | self.assertEqual(iSession.region, region) 166 | 167 | call = mock.call(aws_access_key_id = access, 168 | aws_secret_access_key = secret) 169 | calls = [call] 170 | self.assertEqual(mSession.mock_calls, calls) 171 | 172 | @mock.patch('heaviside.utils.Session', autospec=True) 173 | def test_keys(self, mSession): 174 | iSession = mSession.return_value = MockSession() 175 | access = 'XXX' 176 | secret = 'YYY' 177 | account = '123456' 178 | region = 'east' 179 | credentials = { 180 | 'aws_access_key': access, 181 | 'aws_secret_key': secret, 182 | 'aws_account_id': account, 183 | 'aws_region': region 184 | } 185 | 186 | # Note: Same as test_credentials, except passing the arguments are key word args 187 | session, account_id = utils.create_session(**credentials) 188 | 189 | self.assertEqual(session, iSession) 190 | self.assertEqual(account_id, account) 191 | self.assertEqual(iSession.region, region) 192 | 193 | call = mock.call(aws_access_key_id = access, 194 | aws_secret_access_key = secret) 195 | calls = [call] 196 | self.assertEqual(mSession.mock_calls, calls) 197 | 198 | def test_lookup_account(self): 199 | # only check the logic for looking up the account id 200 | # test_session will verify the rest of the logic is still working 201 | iSession = MockSession() 202 | client = iSession.client('iam') 203 | client.list_users.return_value = { 204 | 'Users': [{ 205 | 'Arn': '1:2:3:4:5:6' 206 | }] 207 | } 208 | 209 | session, account_id = utils.create_session(session=iSession) 210 | 211 | self.assertEqual(account_id, '5') 212 | 213 | calls = [mock.call.list_users(MaxItems=1)] 214 | self.assertEqual(client.mock_calls, calls) 215 | 216 | @mock.patch('heaviside.utils.urlopen', autospec=True) 217 | @mock.patch('heaviside.utils.Session', autospec=True) 218 | def test_lookup_region(self, mSession, mUrlOpen): 219 | # only check the logic for looking up the region 220 | # test_noargs will verify the rest of the logic is still working 221 | mSession.return_value = MockSession() 222 | region = 'east' 223 | az = region + 'a' 224 | mUrlOpen.return_value.read.return_value = az.encode() 225 | 226 | session, account_id = utils.create_session(account_id='12345') 227 | 228 | self.assertEqual(session.region, region) 229 | 230 | -------------------------------------------------------------------------------- /heaviside/__main__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2024 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import os 16 | import sys 17 | import json 18 | import argparse 19 | import heaviside 20 | 21 | # heaviside.utils handles either importing pathlib 22 | # or creating a custom Path object (for Python 2.7+) 23 | Path = heaviside.utils.Path 24 | 25 | 26 | def main(): 27 | parser = argparse.ArgumentParser( 28 | description="Heaviside CLI script for executing the Heaviside Python Library", 29 | formatter_class=argparse.RawDescriptionHelpFormatter, 30 | ) 31 | parser.add_argument( 32 | "--credentials", 33 | "-c", 34 | metavar="", 35 | default=os.environ.get("AWS_CREDENTIALS"), 36 | type=argparse.FileType("r"), 37 | help="File with credentials to use when connecting to AWS (default: AWS_CREDENTIALS)", 38 | ) 39 | parser.add_argument( 40 | "--region", 41 | "-r", 42 | metavar="", 43 | default=os.environ.get("AWS_REGION"), 44 | help="AWS Region (default: AWS_REGION)", 45 | ) 46 | parser.add_argument( 47 | "--account_id", 48 | "-a", 49 | metavar="", 50 | default=os.environ.get("AWS_ACCOUNT_ID"), 51 | help="AWS Account ID (default: AWS_ACCOUNT_ID)", 52 | ) 53 | parser.add_argument( 54 | "--secret_key", 55 | metavar="", 56 | default=os.environ.get("AWS_SECRET_KEY"), 57 | help="AWS Secret Key (default: AWS_SECRET_KEY)", 58 | ) 59 | parser.add_argument( 60 | "--access_key", 61 | metavar="", 62 | default=os.environ.get("AWS_ACCESS_KEY"), 63 | help="AWS Access Key (default: AWS_ACCESS_KEY)", 64 | ) 65 | 66 | subparsers = parser.add_subparsers( 67 | dest="command", metavar="", help="Command to execute" 68 | ) 69 | 70 | #### compile #### 71 | compile_parser = subparsers.add_parser( 72 | "compile", help="Compile a Heaviside file into a StepFunction State Machine" 73 | ) 74 | compile_parser.add_argument( 75 | "--output", 76 | "-o", 77 | metavar="", 78 | default="-", 79 | help="Location to save the StepFunction State Machine to (default: stdout)", 80 | ) 81 | compile_parser.add_argument("file", help="heaviside file to compile") 82 | 83 | #### create #### 84 | create_parser = subparsers.add_parser( 85 | "create", help="Compile a Heaviside file and create a AWS StepFunction" 86 | ) 87 | create_parser.add_argument( 88 | "--name", "-n", metavar="", help="StepFunction name (default: filename)" 89 | ) 90 | create_parser.add_argument("file", help="heaviside file to compile and create") 91 | create_parser.add_argument( 92 | "role", help="AWS IAM role name or full ARN of the role for the StepFunction" 93 | ) 94 | 95 | #### update #### 96 | update_parser = subparsers.add_parser( 97 | "update", help="Compile a Heaviside file and create a AWS StepFunction" 98 | ) 99 | update_parser.add_argument("name", help="Name of the StepFunction to update") 100 | update_parser.add_argument( 101 | "--file", "-f", help="heaviside file to compile and update" 102 | ) 103 | update_parser.add_argument( 104 | "--role", 105 | "-r", 106 | help="AWS IAM role name or full ARN of the role for the StepFunction", 107 | ) 108 | 109 | #### delete #### 110 | delete_parser = subparsers.add_parser("delete", help="Delete an AWS StepFunction") 111 | delete_parser.add_argument("name", help="Name of the StepFunction to delete") 112 | 113 | #### start #### 114 | start_parser = subparsers.add_parser( 115 | "start", help="Start executing a AWS StepFunction" 116 | ) 117 | start_parser.add_argument( 118 | "--no-wait", 119 | "-n", 120 | dest="wait", 121 | action="store_false", 122 | help="If the command should wait for the StepFunction to finish execution", 123 | ) 124 | start_parser.add_argument( 125 | "--input", 126 | "-i", 127 | help="Path to file containing input Json data for the StepFunction ('-' for stdin)", 128 | ) 129 | start_parser.add_argument( 130 | "--json", "-j", help="Json input data for the StepFunction" 131 | ) 132 | start_parser.add_argument( 133 | "name", help="Name of the StepFunction to start executing" 134 | ) 135 | 136 | args = parser.parse_args() 137 | 138 | if args.command is None: 139 | parser.print_usage() 140 | name = os.path.basename(sys.argv[0]) 141 | print("{} error: the following arguments are required: command".format(name)) 142 | return 1 143 | 144 | credentials = {} 145 | if args.credentials is not None: 146 | credentials["credentials"] = json.load(args.credentials) 147 | if args.region is not None: 148 | credentials["region"] = args.region 149 | if args.account_id is not None: 150 | credentials["account_id"] = args.account_id 151 | if args.secret_key is not None: 152 | credentials["secret_key"] = args.secret_key 153 | if args.access_key is not None: 154 | credentials["access_key"] = args.access_key 155 | 156 | try: 157 | if args.command == "compile": 158 | 159 | def find_(key): 160 | if key in credentials: 161 | return credentials[key] 162 | if "credentials" in credentials: 163 | for key_ in (key, "aws_" + key): 164 | if key_ in credentials["credentials"]: 165 | return credentials["credentials"][key_] 166 | print("Could not find {} value, using ''".format(key), file=sys.stderr) 167 | return "" 168 | 169 | region = find_("region") 170 | account = find_("account_id") 171 | 172 | machine = heaviside.compile(Path(args.file), region, account, indent=3) 173 | 174 | with heaviside.utils.write(args.output) as fh: 175 | fh.write(machine) 176 | elif args.command == "create": 177 | name = args.name 178 | if name is None: 179 | name = os.path.basename(args.file) 180 | name = os.path.splitext(name)[0] 181 | 182 | machine = heaviside.StateMachine(name, **credentials) 183 | machine.create(Path(args.file), args.role) 184 | elif args.command == "update": 185 | if args.file: 186 | args.file = Path(args.file) 187 | 188 | machine = heaviside.StateMachine(args.name, **credentials) 189 | machine.update(args.file, args.role) 190 | elif args.command == "delete": 191 | machine = heaviside.StateMachine(args.name, **credentials) 192 | machine.delete(True) 193 | elif args.command == "start": 194 | if args.json is not None: 195 | input_ = args.json 196 | elif args.input is not None: 197 | if args.input == "-": 198 | input_ = sys.stdin.read() 199 | else: 200 | with open(args.input, "r") as fh: 201 | input_ = fh.read() 202 | else: 203 | input_ = "{}" 204 | input_ = json.loads(input_) 205 | 206 | machine = heaviside.StateMachine(args.name, **credentials) 207 | arn = machine.start(input_) 208 | 209 | if args.wait: 210 | output = machine.wait(arn) 211 | print(json.dumps(output, indent=3)) 212 | else: 213 | print("Execution ARN: {}".format(arn)) 214 | else: 215 | raise heaviside.exceptions.HeavisideError( 216 | "Unsupported command: {}".format(args.command) 217 | ) 218 | except heaviside.exceptions.HeavisideError as e: 219 | print(e) 220 | return 1 221 | except Exception as e: 222 | print("Error: {}".format(e)) 223 | return 1 224 | return 0 225 | -------------------------------------------------------------------------------- /heaviside/utils.py: -------------------------------------------------------------------------------- 1 | # Copyright 2024 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import json 16 | import sys 17 | from io import IOBase, StringIO 18 | from contextlib import contextmanager 19 | 20 | from boto3.session import Session 21 | 22 | # With Python3.11 Mapping is imported from collections.abc 23 | # Try to import with the new method and if it fails fall back to old way for compatibility 24 | try: 25 | from collections.abc import Mapping 26 | except ImportError: 27 | from collections import Mapping 28 | 29 | 30 | try: 31 | from urllib.request import urlopen 32 | except ImportError: 33 | from urllib2 import urlopen 34 | 35 | try: 36 | from pathlib import Path 37 | except ImportError: 38 | import os 39 | import io 40 | 41 | class Path(object): 42 | """Stub Path object that implements only the features used by heaviside""" 43 | def __init__(self, path = '/'): 44 | self.path = path 45 | 46 | def open(self, *args, **kwargs): 47 | return io.open(self.path, *args, **kwargs) 48 | 49 | def __div__(self, path): 50 | return Path(os.path.join(self.path, path)) 51 | 52 | def isstr(obj): 53 | """Determine if the given object is a string 54 | 55 | This function support comparing against the Python2 type as well 56 | as the Python2/3 type. 57 | 58 | Args: 59 | obj (object) : Object to check 60 | 61 | Returns: 62 | bool : If the object is a str or unicode type 63 | """ 64 | try: # Python 2 compatibility 65 | is_unicode = isinstance(obj, unicode) 66 | except NameError: 67 | is_unicode = False 68 | 69 | return isinstance(obj, str) or is_unicode 70 | 71 | @contextmanager 72 | def read(obj): 73 | """Context manager for reading data from multiple sources as a file object 74 | 75 | Args: 76 | obj (string|Path|file object): Data to read / read from 77 | If obj is a file object, this is just a pass through 78 | If obj is a Path object, this is similar to obj.open() 79 | If obj is a string, this creates a StringIO so 80 | the data can be read like a file object 81 | 82 | Returns: 83 | file object: File handle containing data 84 | """ 85 | is_open = False 86 | if isinstance(obj, Path): 87 | fh = obj.open() 88 | is_open = True 89 | elif isstr(obj): 90 | fh = StringIO(obj) 91 | fh.name = '' 92 | elif isinstance(obj, IOBase): 93 | fh = obj 94 | else: 95 | raise ValueError("Unknown input type {}".format(type(obj).__name__)) 96 | 97 | try: 98 | yield fh 99 | finally: 100 | if is_open: 101 | fh.close() 102 | 103 | @contextmanager 104 | def write(obj): 105 | """Context manager for writing data to multiple sources as a file object 106 | 107 | Args: 108 | obj (string|Path|file object): Data to read / read from 109 | If obj is a file object, this is just a pass through 110 | If obj is a Path object, this is similar to obj.open() 111 | If obj is a string, and is '-' then sys.stdout is used 112 | else this is similar to Path(obj).open() 113 | 114 | Returns: 115 | file object: File handle ready to write data 116 | """ 117 | fh = None 118 | is_open = False 119 | if isstr(obj): 120 | if obj == '-': 121 | fh = sys.stdout 122 | else: 123 | obj = Path(obj) 124 | 125 | if fh is None: 126 | if isinstance(obj, Path): 127 | fh = obj.open('w') 128 | is_open = True 129 | elif isinstance(obj, IOBase): 130 | fh = obj 131 | else: 132 | raise ValueError("Unknown input type {}".format(type(obj).__name__)) 133 | 134 | try: 135 | yield fh 136 | finally: 137 | if is_open: 138 | fh.close() 139 | 140 | def create_session(**kwargs): 141 | """Create a Boto3 session from multiple different sources 142 | 143 | Basic file format / dictionary format: 144 | { 145 | 'aws_secret_key': '', 146 | 'aws_access_key': '', 147 | 'aws_region': '', 148 | 'aws_account_id': '' 149 | } 150 | 151 | Note: If no arguments are given, a Boto3 session is created and it will attempt 152 | to figure out this information for itself, from predefined locations. 153 | 154 | Args: 155 | session (Boto3 Session): Existing session to use. Only account id will be looked for 156 | credentials (dict|fh|Path|json string): source to load credentials from 157 | If a dict, used directly 158 | If a fh, read and parsed as a Json object 159 | If a Path, opened, read, and parsed as a Json object 160 | If a string, parsed as a Json object 161 | 162 | Note: The following will override the values in credentials if they exist 163 | secret_key / aws_secret_key (string): AWS Secret Key 164 | access_key / aws_access_key (string): AWS Access Key 165 | 166 | Note: The following will be derived from the AWS Session if not provided 167 | account_id / aws_account_id (string): AWS Account ID 168 | 169 | Note: The following will be pulled from EC2 Meta-data if not provided 170 | region / aws_region (string): AWS region to connect to 171 | 172 | Returns: 173 | (Boto3 Session, account_id) : Boto3 session populated with given credentials and 174 | AWS Account ID (either given or derived from session) 175 | """ 176 | credentials = kwargs.get('credentials', {}) 177 | if isinstance(credentials, Mapping): 178 | pass 179 | elif isinstance(credentials, Path): 180 | with credentials.open() as fh: 181 | credentials = json.load(fh) 182 | elif isinstance(credentials, str): 183 | credentials = json.loads(credentials) 184 | elif isinstance(credentials, IOBase): 185 | credentials = json.load(credentials) 186 | else: 187 | raise Exception("Unknown credentials type: {}".format(type(credentials).__name__)) 188 | 189 | def locate(names, locations): 190 | for location in locations: 191 | for name in names: 192 | if name in location: 193 | return location[name] 194 | names = " or ".join(names) 195 | raise Exception("Could not find credentials value for {}".format(names)) 196 | 197 | locate_region = True 198 | if 'session' in kwargs: 199 | session = kwargs['session'] 200 | locate_region = False 201 | else: 202 | try: 203 | access = locate(('access_key', 'aws_access_key'), (kwargs, credentials)) 204 | secret = locate(('secret_key', 'aws_secret_key'), (kwargs, credentials)) 205 | 206 | session = Session(aws_access_key_id = access, 207 | aws_secret_access_key = secret) 208 | except: 209 | session = Session() # Let boto3 try to resolve the keys iteself, potentially from EC2 meta data 210 | 211 | try: 212 | account_id = locate(('account_id', 'aws_account_id'), (kwargs, credentials)) 213 | except: 214 | # From boss-manage.git/lib/aws.py:get_account_id_from_session() 215 | account_id = session.client('iam').list_users(MaxItems=1)["Users"][0]["Arn"].split(':')[4] 216 | 217 | if locate_region: 218 | try: 219 | region = locate(('region', 'aws_region'), (kwargs, credentials)) 220 | except: 221 | try: 222 | region = urlopen('http://169.254.169.254/latest/meta-data/placement/availability-zone/') 223 | region = region.read().decode('utf-8')[:-1] # remove AZ character from region name 224 | except: 225 | raise Exception("Could not locate AWS region to connect to") 226 | 227 | # DP HACK: No good way to get the region after the session is created 228 | # Needed here to support loading keys from EC2 meta data 229 | session._session.set_config_variable('region', region) 230 | 231 | return session, account_id 232 | 233 | -------------------------------------------------------------------------------- /heaviside/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 - 2024 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import print_function, unicode_literals 16 | 17 | import time 18 | import json 19 | from datetime import datetime 20 | 21 | from .lexer import tokenize_source 22 | from .parser import parse 23 | from .exceptions import CompileError, HeavisideError 24 | from .utils import create_session, read 25 | 26 | 27 | def compile(source, region="", account_id="", visitors=[], **kwargs): 28 | """Compile a source step function dsl file into the AWS state machine definition 29 | 30 | Args: 31 | source (string|Path|file object): Source of step function dsl, passed to read() 32 | region (string): AWS Region where Lambdas and Activities are located 33 | account_id (string): AWS Account ID where where Lambdas and Activities are located 34 | visitors (list[ast.StateVisitor]): List of StateVisitors that can be used modify 35 | Task states 36 | kwargs (dict): Arguments to be passed to json.dumps() when creating the definition 37 | 38 | Returns: 39 | string: State machine definition 40 | """ 41 | try: 42 | with read(source) as fh: 43 | if hasattr(fh, "name"): 44 | source_name = fh.name 45 | else: 46 | source_name = "" 47 | tokens = tokenize_source(fh.readline) 48 | 49 | machine = parse(tokens, region=region, account_id=account_id, visitors=visitors) 50 | def_ = machine.definition(**kwargs) 51 | return def_ 52 | except CompileError as e: 53 | e.source = source_name 54 | raise e # DP ???: Should the original stacktrace be perserved? 55 | # except Exception as e: 56 | # print("Unhandled Error: {}".format(e)) 57 | 58 | 59 | class StateMachine(object): 60 | """Class for working with and executing AWS Step Function State Machines""" 61 | 62 | def __init__(self, name, **kwargs): 63 | """ 64 | Args: 65 | name (string): Name of the state machine 66 | kwargs (dict): Same arguments as create_session() 67 | """ 68 | self.name = name 69 | self.arn = None 70 | self.visitors = [] 71 | self.session, self.account_id = create_session(**kwargs) 72 | self.region = self.session.region_name 73 | self.client = self.session.client("stepfunctions") 74 | 75 | resp = self.client.list_state_machines() 76 | for machine in resp["stateMachines"]: 77 | if machine["name"] == name: 78 | self.arn = machine["stateMachineArn"] 79 | break 80 | 81 | def add_visitor(self, visitor): 82 | """Add a StateVisitor to be used when compiling 83 | 84 | Args: 85 | visitor (ast.StateVisitor): a Visitor to use when compiling 86 | """ 87 | self.visitors.append(visitor) 88 | 89 | def build(self, source, **kwargs): 90 | """Build the state machine definition from a source (file) 91 | 92 | Region and account id are determined from constructor arguments 93 | 94 | Args: 95 | source (string|file path|file object): Source of step function dsl 96 | kwargs (dict): Arguments to be passed to json.dumps() when creating the definition 97 | 98 | Returns: 99 | string: State machine definition 100 | 101 | Raises: 102 | CompileError: If the was a problem compiling the source 103 | """ 104 | return compile( 105 | source, 106 | region=self.region, 107 | account_id=self.account_id, 108 | visitors=self.visitors, 109 | **kwargs, 110 | ) 111 | 112 | def _resolve_role(self, role): 113 | role = role.strip() 114 | if not role.lower().startswith("arn:aws:iam"): 115 | client = self.session.client("iam") 116 | try: 117 | response = client.get_role(RoleName=role) 118 | role = response["Role"]["Arn"] 119 | except: 120 | raise HeavisideError("Could not lookup role '{}'".format(role)) 121 | 122 | return role 123 | 124 | def create(self, source, role): 125 | """Create the state machine in AWS from the give source 126 | 127 | If a state machine with the given name already exists an exception is thrown 128 | 129 | Args: 130 | source (string|file path|file object): Source of step function dsl 131 | role (string): AWS IAM role for the state machine to execute under 132 | 133 | Raises: 134 | CompileError: If the was a problem compiling the source 135 | """ 136 | if self.arn is not None: 137 | raise HeavisideError("State Machine {} already exists".format(self.arn)) 138 | 139 | role = self._resolve_role(role) 140 | definition = self.build(source) 141 | 142 | resp = self.client.create_state_machine( 143 | name=self.name, definition=definition, roleArn=role 144 | ) 145 | 146 | self.arn = resp["stateMachineArn"] 147 | 148 | def update(self, source=None, role=None): 149 | """Update the state machine definition and/or IAM role in AWS 150 | 151 | If the state machine doesn't exist an exception is thrown 152 | 153 | Args: 154 | source (string|file path|file object): Optional, source of step function dsl 155 | role (string): Optional, AWS IAM role for the state machine to execute under 156 | 157 | Raises: 158 | CompileError: If the was a problem compiling the source 159 | """ 160 | if source is None and role is None: 161 | raise ValueError("Either 'source' or 'role' need to be provided") 162 | if self.arn is None: 163 | raise HeavisideError("State Machine {} doesn't exist yet".format(self.arn)) 164 | 165 | args = {"stateMachineArn": self.arn} 166 | 167 | if source is not None: 168 | args["definition"] = self.build(source) 169 | 170 | if role is not None: 171 | args["roleArn"] = self._resolve_role(role) 172 | 173 | resp = self.client.update_state_machine(**args) 174 | 175 | def delete(self, exception=False): 176 | """Delete the state machine from AWS 177 | 178 | Args: 179 | exception (boolean): If an excpetion should be thrown if the machine doesn't exist (default: False) 180 | """ 181 | if self.arn is None: 182 | if exception: 183 | raise HeavisideError( 184 | "State Machine {} doesn't exist yet".format(self.name) 185 | ) 186 | else: 187 | resp = self.client.delete_state_machine(stateMachineArn=self.arn) 188 | self.arn = None 189 | 190 | def start(self, input_, name=None): 191 | """Start executing the state machine 192 | 193 | If the state machine doesn't exists an exception is thrown 194 | 195 | Args: 196 | input_ (Json): Json input data for the first state to process 197 | name (string|None): Name of the execution (default: Name of the state machine) 198 | 199 | Returns: 200 | string: ARN of the state machine execution, used to get status and output data 201 | """ 202 | if self.arn is None: 203 | raise HeavisideError("State Machine {} doesn't exist yet".format(self.name)) 204 | 205 | input_ = json.dumps(input_) 206 | 207 | if name is None: 208 | name = self.name + "-" + datetime.now().strftime("%Y%m%d%H%M%s%f") 209 | 210 | resp = self.client.start_execution( 211 | stateMachineArn=self.arn, name=name, input=input_ 212 | ) 213 | 214 | arn = resp["executionArn"] 215 | return ( 216 | arn # DP NOTE: Could store ARN in internal dict and return execution name 217 | ) 218 | 219 | def stop(self, arn, error, cause): 220 | """Stop an execution of the state machine 221 | 222 | Args: 223 | arn (string): ARN of the execution to stop 224 | error (string): Error for the stop 225 | cause (string): Error cause for the stop 226 | """ 227 | resp = self.client.stop_execution(executionArn=arn, error=error, cause=cause) 228 | 229 | def status(self, arn): 230 | """Get the status of an execution 231 | 232 | Args: 233 | arn (string): ARN of the execution to get the status of 234 | 235 | Returns: 236 | string: One of 'RUNNING', 'SUCCEEDED', 'FAILED', 'TIMED_OUT', 'ABORTED' 237 | """ 238 | resp = self.client.describe_execution(executionArn=arn) 239 | return resp["status"] 240 | 241 | def wait(self, arn, period=10): 242 | """Wait for an execution to finish and get the results 243 | 244 | Args: 245 | arn (string): ARN of the execution to get the status of 246 | period (int): Number of seconds to sleep between polls for status 247 | 248 | Returns: 249 | dict: Dict of Json data 250 | 251 | Exceptions: 252 | HeavisideError: If there was an error getting the failure message 253 | """ 254 | while True: 255 | resp = self.client.describe_execution(executionArn=arn) 256 | if resp["status"] != "RUNNING": 257 | if "output" in resp: 258 | return json.loads(resp["output"]) 259 | else: 260 | resp = self.client.get_execution_history( 261 | executionArn=arn, reverseOrder=True 262 | ) 263 | event = resp["events"][0] 264 | for key in ["Failed", "Aborted", "TimedOut"]: 265 | key = "execution{}EventDetails".format(key) 266 | if key in event: 267 | return event[key] 268 | raise HeavisideError( 269 | "Could not locate error output for execution '{}'".format(arn) 270 | ) 271 | else: 272 | time.sleep(period) 273 | 274 | def running_arns(self): 275 | """Query for the ARNs of running executions 276 | 277 | Returns: 278 | list: List of strings containing the ARNs of all running executions 279 | """ 280 | resp = self.client.list_executions( 281 | stateMachineArn=self.arn, statusFilter="RUNNING" 282 | ) 283 | return [ex["executionArn"] for ex in resp["executions"]] 284 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /heaviside/sfn.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import unicode_literals 16 | 17 | import json 18 | from collections import OrderedDict 19 | 20 | import iso8601 # parser for timestamp format 21 | 22 | from .ast import ASTStateChoice, ASTCompOp, ASTCompNot, ASTCompAndOr, ASTValue, ASTModNext, ASTStepFunction 23 | 24 | class Timestamp(object): 25 | """Wrapper around a timestamp string. 26 | 27 | Used to determine if a string is in a valid timestamp format and type it 28 | for the parser 29 | """ 30 | 31 | def __init__(self, timestamp): 32 | """ 33 | Args: 34 | timestamp (string): Timestamp string 35 | 36 | Exceptions: 37 | An exception is thrown if the string is not a valid timestamp 38 | """ 39 | iso8601.parse_date(timestamp) 40 | self.timestamp = timestamp 41 | 42 | def __str__(self): 43 | return self.timestamp 44 | 45 | def __repr__(self): 46 | return "Timestamp({!r})".format(self.timestamp) 47 | 48 | class _StateMachineEncoder(json.JSONEncoder): 49 | """Custom JSONEncoder that handles the Timestamp type""" 50 | def default(self, o): 51 | if type(o) == Timestamp: 52 | return str(o) 53 | return super(_StateMachineEncoder, self).default(o) 54 | 55 | class Branch(dict): 56 | def __init__(self, ast): 57 | super(Branch, self).__init__() 58 | 59 | # Makes states be dumped in the same order they were added 60 | # making it easier to read the output and match it to the input 61 | self['States'] = OrderedDict() 62 | for state in ast.states: 63 | self['States'][state.name] = State(state) 64 | 65 | self['StartAt'] = ast.states[0].name 66 | 67 | class StepFunction(Branch): 68 | def __init__(self, ast): 69 | super(StepFunction, self).__init__(ast) 70 | 71 | if ast.comment is not None: 72 | self['Comment'] = ast.comment.value 73 | 74 | if ast.version is not None: 75 | self['Version'] = ast.version.value.value 76 | 77 | if ast.timeout is not None: 78 | self['TimeoutSeconds'] = ast.timeout.value.value 79 | 80 | def definition(self, **kwargs): 81 | """Dump the state machine into the JSON format needed by AWS 82 | 83 | Args: 84 | kwargs (dict): Arguments passed to json.dumps() 85 | """ 86 | kwargs.setdefault('ensure_ascii', False) # Allow Unicode in the output by default 87 | 88 | return json.dumps(self, cls=_StateMachineEncoder, **kwargs) 89 | 90 | class State(dict): 91 | def __init__(self, ast): 92 | super(State, self).__init__() 93 | 94 | self['Type'] = ast.state_type 95 | 96 | # Generic Modifiers for all States 97 | if ast.comment is not None: 98 | # No longer a token, parsed by AST class into name/comment 99 | self['Comment'] = ast.comment 100 | 101 | if ast.timeout is not None: 102 | timeout = ast.timeout.value.value 103 | self['TimeoutSeconds'] = timeout 104 | else: 105 | timeout = 60 # default 106 | 107 | if ast.heartbeat is not None: 108 | heartbeat = ast.heartbeat.value.value 109 | if not heartbeat < timeout: 110 | ast.heartbeat.raise_error("Heartbeat must be less than timeout (defaults to 60)") 111 | self['HeartbeatSeconds'] = heartbeat 112 | 113 | if ast.input is not None: 114 | self['InputPath'] = ast.input.value.value 115 | 116 | if ast.result is not None: 117 | self['ResultPath'] = ast.result.value.value 118 | 119 | if ast.output is not None: 120 | self['OutputPath'] = ast.output.value.value 121 | 122 | if ast.data is not None: 123 | self['Result'] = ast.data.value 124 | 125 | if ast.catch is not None: 126 | self['Catch'] = [] 127 | for catch in ast.catch: 128 | self['Catch'].append(Catch(catch)) 129 | 130 | if ast.retry is not None: 131 | self['Retry'] = [] 132 | for retry in ast.retry: 133 | self['Retry'].append(Retry(retry)) 134 | 135 | if ast.iterator is not None: 136 | # The iterator contains a separate state machine that runs on each 137 | # element of the input array. 138 | substates = [b for b in ast.iterator.states] 139 | self['Iterator'] = StepFunction(ASTStepFunction(None, None, None, substates)) 140 | 141 | if ast.items_path is not None: 142 | self['ItemsPath'] = ast.items_path.value.value 143 | 144 | if ast.max_concurrency is not None: 145 | max_con = ast.max_concurrency.value.value 146 | if max_con < 0: 147 | ast.max_concurrency.raise_error("max_concurrency must be non-negative") 148 | self['MaxConcurrency'] = max_con 149 | 150 | # State specific arguments 151 | if ast.state_type == 'Fail': 152 | self['Error'] = ast.error.value 153 | self['Cause'] = ast.cause.value 154 | 155 | if ast.state_type == 'Pass' or ast.state_type == 'Map': 156 | if ast.parameters is not None: 157 | self['Parameters'] = Parameters(ast.parameters) 158 | 159 | if ast.state_type == 'Task': 160 | self['Resource'] = ast.arn 161 | if ast.parameters is not None: 162 | self['Parameters'] = Parameters(ast.parameters) 163 | 164 | if ast.state_type == 'Wait': 165 | key = ''.join([t.capitalize() for t in ast.type.value.split('_')]) 166 | self[key] = ast.val.value 167 | 168 | if ast.state_type == 'Choice': 169 | key = ASTStateChoice.DEFAULT 170 | if key in ast.branches: 171 | self['Default'] = ast.branches[key] 172 | del ast.branches[key] 173 | 174 | self['Choices'] = [] 175 | for comp in ast.branches: 176 | self['Choices'].append(Choice(comp, ast.branches[comp])) 177 | 178 | if ast.state_type == 'Parallel': 179 | self['Branches'] = [] 180 | for branch in ast.branches: 181 | self['Branches'].append(Branch(branch)) 182 | 183 | if ast.next is not None: 184 | if isinstance(ast.next, ASTModNext): 185 | self['Next'] = ast.next.value 186 | else: 187 | self['Next'] = ast.next 188 | 189 | if ast.end: 190 | self['End'] = ast.end 191 | 192 | class Catch(dict): 193 | def __init__(self, ast): 194 | super(Catch, self).__init__() 195 | 196 | errors = ast.errors 197 | 198 | # Support a single string for error type 199 | # ??? put this transformation in AST 200 | if type(errors) != list: 201 | errors = [errors] 202 | 203 | if len(errors) == 0: 204 | errors = [ASTValue('States.ALL', None)] 205 | 206 | self['ErrorEquals'] = [e.value for e in errors] 207 | self['Next'] = ast.next 208 | 209 | if ast.path is not None: 210 | self['ResultPath'] = ast.path.value 211 | 212 | class Retry(dict): 213 | def __init__(self, ast): 214 | super(Retry, self).__init__() 215 | 216 | errors = ast.errors 217 | 218 | # Support a single string for error type 219 | # ??? put this transformation in AST 220 | if type(errors) != list: 221 | errors = [errors] 222 | 223 | if len(errors) == 0: 224 | errors = [ASTValue('States.ALL', None)] 225 | 226 | if float(ast.backoff.value) < 1.0: 227 | ast.backoff.raise_error("Backoff rate should be >= 1.0") 228 | 229 | self['ErrorEquals'] = [e.value for e in errors] 230 | self['IntervalSeconds'] = ast.interval.value 231 | self['MaxAttempts'] = ast.max.value 232 | self['BackoffRate'] = float(ast.backoff.value) 233 | 234 | def Parameters(ast): 235 | rst = OrderedDict() 236 | for k,v in ast.items(): 237 | # Keys are ASTValues and need to have the actual value unwrapped 238 | # Values are already raw values as they are JSON Text 239 | rst[k.value] = v 240 | return rst 241 | 242 | COMPARISON = { 243 | '==': { 244 | str: 'StringEquals', 245 | int: 'NumericEquals', 246 | float: 'NumericEquals', 247 | bool: 'BooleanEquals', 248 | Timestamp: 'TimestampEquals', 249 | }, 250 | '<': { 251 | str: 'StringLessThan', 252 | int: 'NumericLessThan', 253 | float: 'NumericLessThan', 254 | Timestamp: 'TimestampLessThan', 255 | }, 256 | '>': { 257 | str: 'StringGreaterThan', 258 | int: 'NumericGreaterThan', 259 | float: 'NumericGreaterThan', 260 | Timestamp: 'TimestampGreaterThan', 261 | }, 262 | '<=': { 263 | str: 'StringLessThanEquals', 264 | int: 'NumericLessThanEquals', 265 | float: 'NumericLessThanEquals', 266 | Timestamp: 'TimestampLessThanEquals', 267 | }, 268 | '>=': { 269 | str: 'StringGreaterThanEquals', 270 | int: 'NumericGreaterThanEquals', 271 | float: 'NumericGreaterThanEquals', 272 | Timestamp: 'TimestampGreaterThanEquals', 273 | }, 274 | } 275 | 276 | try: 277 | for op in COMPARISON.keys(): 278 | COMPARISON[op][unicode] = COMPARISON[op][str] 279 | except NameError: 280 | pass # Support Python2 Unicode string type 281 | 282 | def Choice(ast, target=None): 283 | if type(ast) == ASTCompOp: 284 | var = ast.var.value 285 | val = ast.val.value 286 | op = ast.op.value 287 | op_type = type(val) # The type of the operator is based on the value type 288 | try: 289 | if op == '!=': 290 | op = COMPARISON['=='][op_type] 291 | choice = OpChoice(var, op, val) 292 | return NotChoice(choice, target) 293 | else: 294 | op = COMPARISON[op][op_type] 295 | return OpChoice(var, op, val, target) 296 | except KeyError: 297 | msg = "Cannot make '{}' comparison with type '{}'".format(op, op_type) 298 | ast.raise_error(msg) 299 | elif type(ast) == ASTCompNot: 300 | return NotChoice(Choice(ast.comp), target) 301 | elif isinstance(ast, ASTCompAndOr): 302 | return AndOrChoice(ast, target) 303 | else: 304 | ast.raise_error("Comparison support not implemented yet") 305 | 306 | class OpChoice(dict): 307 | """A ChoiceState Choice wrapping a comparison and reference to state to execute""" 308 | 309 | def __init__(self, var, op, val, target=None): 310 | super(OpChoice, self).__init__(Variable = var) 311 | 312 | self.op = op # for __str__ / __repr__ 313 | self[self.op] = val 314 | 315 | if target is not None: 316 | self['Next'] = target 317 | 318 | def __str__(self): 319 | return repr(self) 320 | 321 | def __repr__(self): 322 | return "({} {} {})".format(self['Variable'], self.op, self[self.op]) 323 | 324 | class NotChoice(dict): 325 | """Wraper around a Choice that negates the Choice""" 326 | 327 | def __init__(self, comp, target=None): 328 | super(NotChoice, self).__init__(Not = comp) 329 | 330 | if target is not None: 331 | self['Next'] = target 332 | 333 | def __str__(self): 334 | return repr(self) 335 | 336 | def __repr__(self): 337 | return "(Not {!r})".format(self['Not']) 338 | 339 | class AndOrChoice(dict): 340 | """Wrapper arounds a list of Choices and 'and's or 'or's the results together""" 341 | 342 | def __init__(self, ast, target=None): 343 | super(AndOrChoice, self).__init__() 344 | 345 | self.op = ast.op # for __str__ / __repr__ 346 | self[self.op] = [Choice(comp) for comp in ast.comps] 347 | 348 | if target is not None: 349 | self['Next'] = target 350 | 351 | def __str__(self): 352 | return repr(self) 353 | 354 | def __repr__(self): 355 | vals = map(repr, self[self.op]) 356 | return "(" + (" {} ".format(self.op.lower())).join(vals) + ")" 357 | 358 | -------------------------------------------------------------------------------- /heaviside/parser.py: -------------------------------------------------------------------------------- 1 | # Copyright 2020 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from funcparserlib.parser import (some, a, many, skip, maybe, forward_decl) 16 | from funcparserlib.parser import NoParseError, State 17 | 18 | from .lexer import Token 19 | from .exceptions import CompileError 20 | from .ast import * 21 | 22 | from .sfn import StepFunction, Timestamp 23 | 24 | # Helper functions 25 | # Used by the main parser logic 26 | 27 | def make(cls): 28 | """Helper that unpacks the tuple of arguments before creating a class""" 29 | def make_(args): 30 | return cls(*args) 31 | return make_ 32 | 33 | def debug(x): 34 | """Print the current object being parsed""" 35 | print(x) 36 | return x 37 | 38 | def debug_(m): 39 | """Print the current object with a prefix 40 | 41 | Args: 42 | m (string): Prefix to print before debuged object 43 | """ 44 | def debug__(a): 45 | print("{}: {!r}".format(m, a)) 46 | return a 47 | return debug__ 48 | 49 | def const(value): 50 | """Create an ASTValue with a constant value""" 51 | def const_(token): 52 | return ASTValue(value, token) 53 | return const_ 54 | 55 | def tok_to_value(token): 56 | """Wrap a token in an ASTValue""" 57 | return ASTValue(token.value, token) 58 | 59 | def toktype(code): 60 | """Get an ASTValue with the given token type""" 61 | return some(lambda x: x.code == code) >> tok_to_value 62 | 63 | def op(operator): 64 | """Get an ASTValue with the given operator value""" 65 | return a(Token('OP', operator)) >> tok_to_value 66 | 67 | def op_(operator): 68 | """Skip the operator with the given value""" 69 | return skip(op(operator)) 70 | 71 | def n(name): 72 | """Get an ASTValue with the given name value""" 73 | return a(Token('NAME', name)) >> tok_to_value 74 | 75 | def n_(name): 76 | """Skip the name with the given value""" 77 | return skip(n(name)) 78 | 79 | def e(name): 80 | """Get an ASTValue with the given error value 81 | 82 | An ERRORTOKEN is any unrecognized input (invalid Python value) 83 | or an unterminated single quote 84 | """ 85 | return a(Token('ERRORTOKEN', name)) >> tok_to_value 86 | 87 | def e_(name): 88 | """Skip the error with the given value""" 89 | return skip(e(name)) 90 | 91 | name = toktype('NAME') 92 | 93 | # Define true and false in terms of Python boolean values 94 | true = (n('true') | n('True')) >> const(True) 95 | false = (n('false') | n('False')) >> const(False) 96 | boolean = true | false 97 | 98 | def value_to_number(ast): 99 | """Convert the ASTValue.value into an int or float""" 100 | try: 101 | ast.value = int(ast.value) 102 | except ValueError: 103 | try: 104 | ast.value = float(ast.value) 105 | except ValueError: 106 | ast.raise_error("'{}' is not a valid number".format(ast.value)) 107 | return ast 108 | 109 | # Get an int or float as an ASTValue 110 | number = toktype('NUMBER') >> value_to_number 111 | 112 | def check(cond, msg): 113 | def check_(ast): 114 | if not cond(ast.value): 115 | ast.raise_error(msg.format(ast.value)) 116 | return ast 117 | return check_ 118 | 119 | # Get an integer, non-negative integer, positive integer as an ASTValue 120 | integer = number >> check(lambda val: isinstance(val, int), "'{}' is not a valid integer") 121 | integer_nn = integer >> check(lambda val: val >= 0, "'{}' is not a non-negative integer") 122 | integer_pos = integer >> check(lambda val: val > 0, "'{}' is not a positive integer") 123 | 124 | def value_to_string(ast): 125 | """Remove the quotes from around the string value""" 126 | if ast.value[:3] in ('"""', "'''"): 127 | ast.value = ast.value[3:-3] 128 | else: 129 | ast.value = ast.value[1:-1] 130 | return ast 131 | 132 | # Get a string as an ASTValue 133 | string = toktype('STRING') >> value_to_string 134 | 135 | def string_to_timestamp(ast): 136 | """Try to parse a string as a Timestamp""" 137 | try: 138 | ast.value = Timestamp(ast.value) 139 | except: 140 | pass 141 | #ast.raise_error("'{}' is not a valid timestamp".format(ast.value)) 142 | return ast 143 | 144 | # Get a string or timestamp as an ASTValue 145 | timestamp_or_string = string >> string_to_timestamp 146 | 147 | # Skip the end sequence token 148 | end = skip(a(Token('ENDMARKER', ''))) 149 | 150 | # Skip the indent / dedent tokens 151 | block_s = skip(toktype('INDENT')) 152 | block_e = skip(toktype('DEDENT')) 153 | 154 | def make_array(n): 155 | """Take the results of parsing an array and return an array 156 | 157 | Args: 158 | n (None|list): None for empty list 159 | list should be [head, [tail]] 160 | """ 161 | if n is None: 162 | return [] 163 | else: 164 | return [n[0]] + n[1] 165 | 166 | def make_object(n): 167 | """Take a list of pairs and create a dict 168 | 169 | NOTE: run through make_array to transform the results to an array 170 | """ 171 | return dict(make_array(n)) 172 | 173 | #============= 174 | # Parser Rules 175 | #============= 176 | def json_text_(): 177 | """Returns the parser for JSON Text""" 178 | # Taken from https://github.com/vlasovskikh/funcparserlib/blob/master/funcparserlib/tests/json.py 179 | # and modified slightly 180 | unwrap = lambda x: x.value 181 | 182 | null = (n('null') | n('Null')) >> const(None) >> unwrap 183 | 184 | value = forward_decl() 185 | member = (string >> unwrap) + op_(u':') + value >> tuple 186 | object = ( 187 | op_(u'{') + 188 | maybe(member + many(op_(u',') + member) + maybe(op_(','))) + 189 | op_(u'}') 190 | >> make_object) 191 | array = ( 192 | op_(u'[') + 193 | maybe(value + many(op_(u',') + value) + maybe(op_(','))) + 194 | op_(u']') 195 | >> make_array) 196 | 197 | value.define( 198 | null 199 | | (true >> unwrap) 200 | | (false >> unwrap) 201 | | object 202 | | array 203 | | (number >> unwrap) 204 | | (string >> unwrap)) 205 | 206 | return value 207 | json_text = json_text_() 208 | 209 | def comparison_(): 210 | """Returns the parse for a compound compare statement""" 211 | ops = op('==') | op('<') | op('>') | op('<=') | op('>=') | op('!=') 212 | op_vals = (boolean|number|timestamp_or_string) 213 | comp_op = string + ops + op_vals >> make(ASTCompOp) 214 | 215 | def multi(func): 216 | """For x + many(x) lists, call func only when there are multiple xs""" 217 | def multi_(args): 218 | x, xs = args 219 | if len(xs) == 0: 220 | return x 221 | return func(args) 222 | return multi_ 223 | 224 | comp_stmt = forward_decl() 225 | comp_base = forward_decl() 226 | comp_base.define((op_('(') + comp_stmt + op_(')')) | comp_op | ((n('not') + comp_base) >> make(ASTCompNot))) 227 | comp_and = comp_base + many(n_('and') + comp_base) >> multi(make(ASTCompAnd)) 228 | comp_or = comp_and + many(n_('or') + comp_and) >> multi(make(ASTCompOr)) 229 | comp_stmt.define(comp_or) 230 | 231 | return comp_stmt 232 | comparison = comparison_() 233 | 234 | def parse(seq, region = '', account_id = '', visitors=[]): 235 | """Parse the given sequence of tokens into a StateMachine object 236 | 237 | Args: 238 | seq (list): List of lexer.Token tokens to parse 239 | region (string): AWS Region where Lambdas and Activities are located 240 | account_id (string): AWS Account ID where where Lambdas and Activities are located 241 | visitors (list[ast.StateVisitor]): List of StateVisitors that can be used modify 242 | Task states 243 | 244 | Returns 245 | sfn.StateMachine: StateMachine object 246 | """ 247 | state = forward_decl() 248 | 249 | # Primitives 250 | array = op_('[') + maybe(string + many(op_(',') + string)) + op_(']') >> make_array 251 | 252 | block = block_s + many(state) + block_e 253 | comment_block = block_s + maybe(string) + many(state) + block_e 254 | parameter_kv = name + maybe(op_('.') + e('$')) + op_(':') + json_text 255 | parameter_block = n('parameters') + op_(':') + block_s + parameter_kv + many(parameter_kv) + block_e >> make(ASTModParameters) 256 | retry_block = n('retry') + (array|string) + integer_pos + integer_nn + number >> make(ASTModRetry) 257 | catch_block = n('catch') + (array|string) + op_(':') + maybe(string) + block >> make(ASTModCatch) 258 | iterator_block = n('iterator') + op_(':') + comment_block >> make(ASTModIterator) 259 | 260 | 261 | # Simple States 262 | # DP Note: The 'next' modifier is not allowed in general usage, must use the 'Goto' 263 | # state to create that modifier. If 'next' should be allowed from any state 264 | # just add it to 'state_modifier' and 'transform_modifier' 265 | state_modifier = ((n('timeout') + op_(':') + integer_pos >> make(ASTModTimeout)) | 266 | (n('heartbeat') + op_(':') + integer_pos >> make(ASTModHeartbeat)) | 267 | (n('input') + op_(':') + string >> make(ASTModInput)) | 268 | (n('result') + op_(':') + string >> make(ASTModResult)) | 269 | (n('output') + op_(':') + string >> make(ASTModOutput)) | 270 | (n('data') + op_(':') + block_s + json_text + block_e >> make(ASTModData)) | 271 | (n('max_concurrency') + op_(':') + integer_nn >> make(ASTModMaxConcurrency)) | 272 | (n('items_path') + op_(':') + string >> make(ASTModItemsPath)) | 273 | parameter_block | retry_block | catch_block | iterator_block) 274 | 275 | state_modifiers = state_modifier + many(state_modifier) >> make(ASTModifiers) 276 | state_block = maybe(block_s + maybe(string) + maybe(state_modifiers) + block_e) 277 | 278 | pass_ = n('Pass') + op_('(') + op_(')') + state_block >> make(ASTStatePass) 279 | success = n('Success') + op_('(') + op_(')') + state_block >> make(ASTStateSuccess) 280 | fail = n('Fail') + op_('(') + string + op_(',') + string + op_(')') + state_block >> make(ASTStateFail) 281 | wait_types = n('seconds') | n('seconds_path') | n('timestamp') | n('timestamp_path') 282 | wait = n('Wait') + op_('(') + wait_types + op_('=') + (integer_pos|timestamp_or_string) + op_(')') + state_block >> make(ASTStateWait) 283 | task = name + maybe(op_('.') + name) + op_('(') + maybe(string) + op_(')') + state_block >> make(ASTStateTask) 284 | simple_state = pass_ | success | fail | wait | task 285 | 286 | # Flow Control States 287 | transform_modifier = ((n('input') + op_(':') + string >> make(ASTModInput)) | 288 | (n('result') + op_(':') + string >> make(ASTModResult)) | 289 | (n('output') + op_(':') + string >> make(ASTModOutput))) 290 | transform_modifiers = transform_modifier + many(transform_modifier) >> make(ASTModifiers) 291 | transform_block = maybe(n_('transform') + op_(':') + block_s + maybe(transform_modifiers) + block_e) 292 | 293 | while_ = n('while') + comparison + op_(':') + comment_block + transform_block >> make(ASTStateWhile) 294 | if_else = (n('if') + comparison + op_(':') + comment_block + 295 | many(n_('elif') + comparison + op_(':') + block) + 296 | maybe(n_('else') + op_(':') + block) + transform_block) >> make(ASTStateIfElse) 297 | switch_case = n('case') + (boolean|number|timestamp_or_string) + op_(':') + block 298 | switch = (n('switch') + string + op_(':') + 299 | block_s + maybe(string) + many(switch_case) + 300 | maybe(n('default') + op_(':') + block) + 301 | block_e + transform_block) >> make(ASTStateSwitch) 302 | choice_state = while_ | if_else | switch 303 | 304 | error_modifier = (retry_block|catch_block) + many(retry_block|catch_block) >> make(ASTModifiers) 305 | error_block = maybe(n_('error') + op_(':') + block_s + maybe(error_modifier) + block_e) 306 | parallel = (n('parallel') + op_(':') + comment_block + 307 | many(n('parallel') + op_(':') + block) + 308 | transform_block + error_block) >> make(ASTStateParallel) 309 | 310 | goto = n('goto') + string >> make(ASTStateGoto) 311 | 312 | map_ = ((n('map') + op_(':') + state_block + transform_block + error_block) 313 | ) >> make(ASTStateMap) 314 | 315 | state.define(simple_state | choice_state | parallel | goto | map_) 316 | 317 | # State Machine 318 | version = maybe(n('version') + op_(':') + string >> make(ASTModVersion)) 319 | timeout = maybe(n('timeout') + op_(':') + integer_pos >> make(ASTModTimeout)) 320 | machine = maybe(string) + version + timeout + many(state) + end >> make(ASTStepFunction) 321 | 322 | try: 323 | # DP NOTE: calling run() directly to have better control of error handling 324 | (tree, _) = machine.run(seq, State(pos=0, max=0)) 325 | link_branch(tree) 326 | check_names(tree) 327 | resolve_arns(tree, region, account_id) 328 | verify_goto_targets(tree) 329 | for visitor in visitors: 330 | visitor.visit(tree) 331 | function = StepFunction(tree) 332 | #import code 333 | #code.interact(local=locals()) 334 | 335 | return function 336 | except NoParseError as ex: 337 | max = ex.state.max 338 | tok = seq[max] if len(seq) > max else Token('EOF', '') 339 | 340 | if tok.code == 'ERRORTOKEN': 341 | msg = "Unterminated quote" 342 | else: 343 | msg = "Invalid syntax" 344 | # DP ???: Should the actual token be used in the error message? 345 | 346 | raise CompileError.from_token(tok, msg) 347 | 348 | -------------------------------------------------------------------------------- /heaviside/tests/test_statemachine.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import unittest 16 | 17 | try: 18 | from unittest import mock 19 | except ImportError: 20 | import mock 21 | 22 | from .utils import MockSession 23 | 24 | from heaviside import StateMachine 25 | 26 | class TestStateMachine(unittest.TestCase): 27 | @mock.patch('heaviside.create_session', autospec=True) 28 | def test_constructor(self, mCreateSession): 29 | iSession = MockSession() 30 | mCreateSession.return_value = (iSession, '123456') 31 | client = iSession.client('stepfunctions') 32 | client.list_state_machines.return_value = { 33 | 'stateMachines':[{ 34 | 'name': 'name', 35 | 'stateMachineArn': 'XXX' 36 | }] 37 | } 38 | 39 | machine = StateMachine('name') 40 | 41 | self.assertEqual(machine.arn, 'XXX') 42 | 43 | calls = [ 44 | mock.call.list_state_machines() 45 | ] 46 | self.assertEqual(client.mock_calls, calls) 47 | 48 | @mock.patch('heaviside.compile', autospec=True) 49 | @mock.patch('heaviside.create_session', autospec=True) 50 | def test_build(self, mCreateSession, mCompile): 51 | iSession = MockSession() 52 | iSession.region_name = 'region' 53 | mCreateSession.return_value = (iSession, '123456') 54 | client = iSession.client('stepfunctions') 55 | client.list_state_machines.return_value = { 56 | 'stateMachines':[{ 57 | 'name': 'name', 58 | 'stateMachineArn': 'XXX' 59 | }] 60 | } 61 | 62 | mCompile.return_value = {} 63 | 64 | machine = StateMachine('name') 65 | 66 | sfn = "Pass()" 67 | actual = machine.build(sfn) 68 | expected = {} 69 | 70 | self.assertEqual(actual, expected) 71 | 72 | calls = [ 73 | mock.call(sfn, region=machine.region, account_id=machine.account_id, visitors=[]) 74 | ] 75 | self.assertEqual(mCompile.mock_calls, calls) 76 | 77 | @mock.patch('heaviside.create_session', autospec=True) 78 | def test_resolve_role(self, mCreateSession): 79 | iSession = MockSession() 80 | mCreateSession.return_value = (iSession, '123456') 81 | 82 | client = iSession.client('stepfunctions') 83 | client.list_state_machines.return_value = { 84 | 'stateMachines':[{ 85 | 'name': 'name', 86 | 'stateMachineArn': 'XXX' 87 | }] 88 | } 89 | 90 | iam = iSession.client('iam') 91 | iam.get_role.return_value = { 92 | 'Role': { 93 | 'Arn': 'YYY' 94 | } 95 | } 96 | 97 | machine = StateMachine('name') 98 | arn = machine._resolve_role('role') 99 | 100 | self.assertEqual(arn, 'YYY') 101 | 102 | calls = [ 103 | mock.call.get_role(RoleName = 'role') 104 | ] 105 | self.assertEqual(iam.mock_calls, calls) 106 | 107 | @mock.patch('heaviside.create_session', autospec=True) 108 | def test_create(self, mCreateSession): 109 | iSession = MockSession() 110 | mCreateSession.return_value = (iSession, '123456') 111 | 112 | client = iSession.client('stepfunctions') 113 | client.list_state_machines.return_value = { 114 | 'stateMachines':[] 115 | } 116 | 117 | client.create_state_machine.return_value = { 118 | 'stateMachineArn': 'XXX' 119 | } 120 | 121 | _resolve_role = mock.MagicMock() 122 | _resolve_role.return_value = 'arn' 123 | 124 | build = mock.MagicMock() 125 | build.return_value = {} 126 | 127 | machine = StateMachine('name') 128 | machine._resolve_role = _resolve_role 129 | machine.build = build 130 | 131 | machine.create('source', 'role') 132 | 133 | self.assertEqual(machine.arn, 'XXX') 134 | 135 | calls = [mock.call('role')] 136 | self.assertEqual(_resolve_role.mock_calls, calls) 137 | 138 | calls = [mock.call('source')] 139 | self.assertEqual(build.mock_calls, calls) 140 | 141 | calls = [ 142 | mock.call.list_state_machines(), 143 | mock.call.create_state_machine(name = 'name', 144 | definition = {}, 145 | roleArn = 'arn') 146 | ] 147 | self.assertEqual(client.mock_calls, calls) 148 | 149 | @mock.patch('heaviside.create_session', autospec=True) 150 | def test_create_exists(self, mCreateSession): 151 | iSession = MockSession() 152 | mCreateSession.return_value = (iSession, '123456') 153 | 154 | client = iSession.client('stepfunctions') 155 | client.list_state_machines.return_value = { 156 | 'stateMachines':[{ 157 | 'name': 'name', 158 | 'stateMachineArn': 'XXX' 159 | }] 160 | } 161 | 162 | machine = StateMachine('name') 163 | 164 | with self.assertRaises(Exception): 165 | machine.create('source', 'role') 166 | 167 | self.assertEqual(machine.arn, 'XXX') 168 | 169 | calls = [ 170 | mock.call.list_state_machines(), 171 | ] 172 | self.assertEqual(client.mock_calls, calls) 173 | 174 | @mock.patch('heaviside.create_session', autospec=True) 175 | def test_delete(self, mCreateSession): 176 | iSession = MockSession() 177 | mCreateSession.return_value = (iSession, '123456') 178 | 179 | client = iSession.client('stepfunctions') 180 | client.list_state_machines.return_value = { 181 | 'stateMachines':[{ 182 | 'name': 'name', 183 | 'stateMachineArn': 'XXX' 184 | }] 185 | } 186 | 187 | machine = StateMachine('name') 188 | 189 | machine.delete() 190 | 191 | self.assertEqual(machine.arn, None) 192 | 193 | calls = [ 194 | mock.call.list_state_machines(), 195 | mock.call.delete_state_machine(stateMachineArn = 'XXX') 196 | ] 197 | self.assertEqual(client.mock_calls, calls) 198 | 199 | @mock.patch('heaviside.create_session', autospec=True) 200 | def test_delete_exception(self, mCreateSession): 201 | iSession = MockSession() 202 | mCreateSession.return_value = (iSession, '123456') 203 | 204 | client = iSession.client('stepfunctions') 205 | client.list_state_machines.return_value = { 206 | 'stateMachines':[] 207 | } 208 | 209 | machine = StateMachine('name') 210 | 211 | with self.assertRaises(Exception): 212 | machine.delete(True) 213 | 214 | self.assertEqual(machine.arn, None) 215 | 216 | calls = [ 217 | mock.call.list_state_machines(), 218 | ] 219 | self.assertEqual(client.mock_calls, calls) 220 | 221 | @mock.patch('heaviside.create_session', autospec=True) 222 | def test_start(self, mCreateSession): 223 | iSession = MockSession() 224 | mCreateSession.return_value = (iSession, '123456') 225 | 226 | client = iSession.client('stepfunctions') 227 | client.list_state_machines.return_value = { 228 | 'stateMachines':[{ 229 | 'name': 'name', 230 | 'stateMachineArn': 'XXX' 231 | }] 232 | } 233 | 234 | client.start_execution.return_value = { 235 | 'executionArn': 'YYY' 236 | } 237 | 238 | machine = StateMachine('name') 239 | 240 | arn = machine.start({}, 'run') 241 | 242 | self.assertEqual(arn, 'YYY') 243 | 244 | calls = [ 245 | mock.call.list_state_machines(), 246 | mock.call.start_execution(stateMachineArn = 'XXX', 247 | name = 'run', 248 | input = '{}') 249 | 250 | ] 251 | self.assertEqual(client.mock_calls, calls) 252 | 253 | @mock.patch('heaviside.create_session', autospec=True) 254 | def test_start_doesnt_exist(self, mCreateSession): 255 | iSession = MockSession() 256 | mCreateSession.return_value = (iSession, '123456') 257 | 258 | client = iSession.client('stepfunctions') 259 | client.list_state_machines.return_value = { 260 | 'stateMachines':[] 261 | } 262 | 263 | machine = StateMachine('name') 264 | 265 | with self.assertRaises(Exception): 266 | machine.start({}, 'run') 267 | 268 | calls = [ 269 | mock.call.list_state_machines(), 270 | ] 271 | self.assertEqual(client.mock_calls, calls) 272 | 273 | @mock.patch('heaviside.create_session', autospec=True) 274 | def test_stop(self, mCreateSession): 275 | iSession = MockSession() 276 | mCreateSession.return_value = (iSession, '123456') 277 | 278 | client = iSession.client('stepfunctions') 279 | client.list_state_machines.return_value = { 280 | 'stateMachines':[{ 281 | 'name': 'name', 282 | 'stateMachineArn': 'XXX' 283 | }] 284 | } 285 | 286 | machine = StateMachine('name') 287 | machine.stop('arn', 'error', 'cause') 288 | 289 | calls = [ 290 | mock.call.list_state_machines(), 291 | mock.call.stop_execution(executionArn = 'arn', 292 | error = 'error', 293 | cause = 'cause') 294 | 295 | ] 296 | self.assertEqual(client.mock_calls, calls) 297 | 298 | @mock.patch('heaviside.create_session', autospec=True) 299 | def test_status(self, mCreateSession): 300 | iSession = MockSession() 301 | mCreateSession.return_value = (iSession, '123456') 302 | 303 | client = iSession.client('stepfunctions') 304 | client.list_state_machines.return_value = { 305 | 'stateMachines':[{ 306 | 'name': 'name', 307 | 'stateMachineArn': 'XXX' 308 | }] 309 | } 310 | 311 | client.describe_execution.return_value = { 312 | 'status': 'status' 313 | } 314 | 315 | machine = StateMachine('name') 316 | status = machine.status('arn') 317 | 318 | self.assertEqual(status, 'status') 319 | 320 | calls = [ 321 | mock.call.list_state_machines(), 322 | mock.call.describe_execution(executionArn = 'arn') 323 | 324 | ] 325 | self.assertEqual(client.mock_calls, calls) 326 | 327 | @mock.patch('heaviside.time.sleep', autospec=True) 328 | @mock.patch('heaviside.create_session', autospec=True) 329 | def test_wait_success(self, mCreateSession, mSleep): 330 | iSession = MockSession() 331 | mCreateSession.return_value = (iSession, '123456') 332 | 333 | client = iSession.client('stepfunctions') 334 | client.list_state_machines.return_value = { 335 | 'stateMachines':[{ 336 | 'name': 'name', 337 | 'stateMachineArn': 'XXX' 338 | }] 339 | } 340 | 341 | client.describe_execution.side_effect = [ 342 | {'status': 'RUNNING'}, 343 | {'status': 'SUCCESS', 'output': '{}'} 344 | ] 345 | 346 | machine = StateMachine('name') 347 | output = machine.wait('arn') 348 | 349 | self.assertEqual(output, {}) 350 | 351 | calls = [ 352 | mock.call.list_state_machines(), 353 | mock.call.describe_execution(executionArn = 'arn'), 354 | mock.call.describe_execution(executionArn = 'arn') 355 | ] 356 | self.assertEqual(client.mock_calls, calls) 357 | 358 | calls = [ 359 | mock.call(10) 360 | ] 361 | self.assertEqual(mSleep.mock_calls, calls) 362 | 363 | @mock.patch('heaviside.time.sleep', autospec=True) 364 | @mock.patch('heaviside.create_session', autospec=True) 365 | def _test_wait_xxx(self, error, mCreateSession, mSleep): 366 | iSession = MockSession() 367 | mCreateSession.return_value = (iSession, '123456') 368 | 369 | client = iSession.client('stepfunctions') 370 | client.list_state_machines.return_value = { 371 | 'stateMachines':[{ 372 | 'name': 'name', 373 | 'stateMachineArn': 'XXX' 374 | }] 375 | } 376 | 377 | client.describe_execution.side_effect = [ 378 | {'status': 'RUNNING'}, 379 | {'status': error} 380 | ] 381 | 382 | client.get_execution_history.return_value = { 383 | 'events':[{ 384 | 'execution{}EventDetails'.format(error): {} 385 | }] 386 | } 387 | 388 | machine = StateMachine('name') 389 | 390 | if error is None: 391 | with self.assertRaises(Exception): 392 | machine.wait('arn') 393 | else: 394 | output = machine.wait('arn') 395 | self.assertEqual(output, {}) 396 | 397 | calls = [ 398 | mock.call.list_state_machines(), 399 | mock.call.describe_execution(executionArn = 'arn'), 400 | mock.call.describe_execution(executionArn = 'arn'), 401 | mock.call.get_execution_history(executionArn = 'arn', 402 | reverseOrder = True) 403 | ] 404 | self.assertEqual(client.mock_calls, calls) 405 | 406 | calls = [ 407 | mock.call(10) 408 | ] 409 | self.assertEqual(mSleep.mock_calls, calls) 410 | 411 | def test_wait_failed(self): 412 | self._test_wait_xxx('Failed') 413 | 414 | def test_wait_aborted(self): 415 | self._test_wait_xxx('Aborted') 416 | 417 | def test_wait_timedout(self): 418 | self._test_wait_xxx('TimedOut') 419 | 420 | def test_wait_exception(self): 421 | self._test_wait_xxx(None) 422 | 423 | @mock.patch('heaviside.create_session', autospec=True) 424 | def test_running_arns(self, mCreateSession): 425 | iSession = MockSession() 426 | mCreateSession.return_value = (iSession, '123456') 427 | 428 | client = iSession.client('stepfunctions') 429 | client.list_state_machines.return_value = { 430 | 'stateMachines':[{ 431 | 'name': 'name', 432 | 'stateMachineArn': 'XXX' 433 | }] 434 | } 435 | 436 | client.list_executions.return_value = { 437 | 'executions': [ 438 | {'executionArn': 'arn1'}, 439 | {'executionArn': 'arn2'} 440 | ] 441 | } 442 | 443 | machine = StateMachine('name') 444 | arns = machine.running_arns() 445 | 446 | self.assertEqual(arns, ['arn1', 'arn2']) 447 | 448 | calls = [ 449 | mock.call.list_state_machines(), 450 | mock.call.list_executions(stateMachineArn = 'XXX', 451 | statusFilter = 'RUNNING') 452 | 453 | ] 454 | self.assertEqual(client.mock_calls, calls) 455 | 456 | 457 | -------------------------------------------------------------------------------- /docs/LibraryAPI.md: -------------------------------------------------------------------------------- 1 | # Heaviside Library API 2 | 3 | This document describes the public API available to other developers using the Heaviside library. 4 | 5 | Copyright 2019 The Johns Hopkins University Applied Physics Laboratory 6 | 7 | Licensed under the Apache License, Version 2.0 (the "License"); 8 | you may not use this file except in compliance with the License. 9 | You may obtain a copy of the License at 10 | 11 | http://www.apache.org/licenses/LICENSE-2.0 12 | 13 | Unless required by applicable law or agreed to in writing, software 14 | distributed under the License is distributed on an "AS IS" BASIS, 15 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | See the License for the specific language governing permissions and 17 | limitations under the License. 18 | 19 | 20 | ## Table of Contents: 21 | 22 | * [Overview](#Overview) 23 | * [Utilities](#Utilities) 24 | - [Path](#Path) 25 | * [Compiling](#Compiling) 26 | - [Visitors](#Visitors) 27 | * [State Machine](#State-Machine) 28 | - [Add Visitor](#Add-Visitor) 29 | - [Build](#Build) 30 | - [Create](#Create) 31 | - [Update](#Update) 32 | - [Delete](#Delete) 33 | - [Start](#Start) 34 | - [Stop](#Stop) 35 | - [Status](#Status) 36 | - [Wait](#Wait) 37 | - [Running ARNs](#Running-ARNs) 38 | * [Activities](#Activities) 39 | - [Activity Mixin](#Activity-Mixin) 40 | - [Task Mixin](#Task-Mixin) 41 | - [Activity Object](#Activity-Object) 42 | - [Task Process](#Task-Process) 43 | - [Activity Process](#Activity-Process) 44 | - [Activity Manager](#Activity-Manager) 45 | - [Fan-out](#Fan-out) 46 | 47 | ## Overview 48 | 49 | Heaviside provides both a command line interface as well as a Python library. More information on the command line interface is in the [README](../README.md). 50 | 51 | This document covers what is considered the public API for Heaviside, meaning that any changes to these APIs will either be backwards compatible or will result in a major version number change. All of the APIs described contain full Python documentation with more details about arguments and functionality. 52 | 53 | ## Utilities 54 | 55 | Heaviside utility functions. 56 | 57 | ### Path 58 | 59 | Heaviside makes use of `pathlib.Path` to be able to determine if the given argument is a file path reference or if it is a string containing the data to work with. Because `pathlib` was added in Python 3.4 Heaviside includes a minimal implementation that covers just the features used internally by Heaviside. 60 | 61 | This minimal implementation is only used if the current runtime doesn't have `pathlib`. If `pathlib.Path` is available it is exposed as `heaviside.utils.Path` so that code can be interoperable between different Python version. 62 | 63 | ```python 64 | source = heaviside.utils.Path('/path/to/file.hsd') 65 | # or 66 | source = heaviside.utils.Path() / 'path' / 'to' / 'file.hsd' 67 | ``` 68 | 69 | For more details refer to the [pathlib](https://docs.python.org/3.4/library/pathlib.html) documentation. 70 | 71 | ## Compiling 72 | 73 | The core of Heaviside is the [Heaviside DSL](StepFunctionDSL.md) to [Step Function Definition](https://aws.amazon.com/step-functions/) compiler. The `compile` function compiles the given DSL source and returns the Step Function result. 74 | 75 | `definition = heaviside.compile(heaviside.utils.Path('/path/to/hsd/file.hsd'))` 76 | 77 | ```python 78 | heaviside.compile(source : Union[str, IOBase, Path], 79 | region : str = '', 80 | account_id : str = '', 81 | visitors : List[Visitor] = [], 82 | **kwargs) -> str 83 | ``` 84 | 85 | * `source`: The Heaviside DSL description to compile 86 | `Path` is either `pathlib.Path` or `heaviside.utils.Path` 87 | * `region`: The name of the AWS region where Lambdas and Activities are located 88 | * `account_id`: The AWS account ID where Lambdas and Activities are located 89 | * `visitors`: A list of [Visitor](#Visitors) objects to apply during the compiling process 90 | * `kwargs`: Additional key-word arguments to be passed to `json.dumps` 91 | * Returns: String containing StepFunction Machine Definition 92 | 93 | ### Visitors 94 | 95 | To support programatic modification of the Heaviside AST during compilation Heaviside supports a visitor pattern. This allows modifications like modifying the task ARN, allowing the creation of a generic Heaviside file that can be used in development, testing, and production deployments. 96 | 97 | The `heaviside.ast.StateVisitor` class can serve as the base class for a custom visitor: 98 | 99 | ```python 100 | class CustomVisitor(heaviside.ast.StateVisitor): 101 | def handle_task(self, task: heaviside.ast.ASTStateTask): 102 | pass 103 | ``` 104 | 105 | It is also possible to create a completely custom visitor, giving more control and ability but at the expense of needing to walk the branches of execution manually: 106 | 107 | ```python 108 | class CustomVisitor(object): 109 | def visit(self, branch: List[heaviside.ast.ASTState]): 110 | # NOTE: only called one with the main branch of execution and it is the callee's reposibility 111 | # to decent into the nested branches of execution 112 | pass 113 | ``` 114 | 115 | ## State Machine 116 | 117 | A Step Function is more than then definition of the state machine but also includes the launching different executions and the lifecycle of the executions and the Step Function itself. The `heaviside.StateMachine` object makes managing this lifecycle easier and more convenient. 118 | 119 | Note that when initializing a new `heaviside.StateMachine` object it will query AWS to determine if the Step Function exists and if so the ARN of the Step Function. 120 | 121 | `machine = heaviside.StateMachine('StepFunction_Name')` 122 | 123 | ```python 124 | heaviside.StateMachine(name : str, 125 | 126 | # Only provide one of the following or allow Boto3 to 127 | # search for credentials on the system 128 | 129 | # Existing Boto3 Session object 130 | session : boto3.session.Session, 131 | 132 | # JSON object, String with JSON object, File with JSON object 133 | # containing a Secret Key and Access Key (and optional 134 | # Account ID and Region 135 | credentials : Union[dict, IOBase, Path, str], 136 | 137 | # Individual keyword arguments 138 | # AWS Secret Key 139 | secret_key : str, 140 | # or 141 | aws_secret_key: str, 142 | 143 | # AWS Access Key 144 | access_key: str, 145 | # or 146 | aws_access_key: str, 147 | 148 | # Account ID (will be looked up if not provided) 149 | account_id: str, 150 | # or 151 | aws_account_id: str, 152 | 153 | # Region (will be looked up from EC2 Meta-data if not provided) 154 | region: str, 155 | # or 156 | aws_region: str 157 | 158 | ) -> StateMachine 159 | ``` 160 | 161 | * `name`: The name of the Step Function to manage 162 | * `kwargs`: Arguments passed to `heaviside.utils.create_session` to setup a Boto3 163 | Session for communicating with AWS 164 | 165 | ### Add Visitor 166 | 167 | Add a custom `heaviside.ast.StateVisitor` to be used when building or creating a new Step Function. 168 | 169 | `machine.add_visitor(heaviside.ast.StateVisitor())` 170 | 171 | ```python 172 | StateMachine.add_visitor(visitor: heaviside.ast.StateVisitor) -> None 173 | ``` 174 | 175 | * `visitor`: A custom implementation / subclass of `heaviside.ast.StateVisitor` 176 | 177 | ### Build 178 | 179 | Compile a given Heaviside DSL definition. This is just a wrapper around `heaviside.compile` that passess the `heaviside.StateMachine` variables. This is normally used by `StateMachine.create`. 180 | 181 | `definition = machine.build(heaviside.utils.Path('/path/to/hsd/file.hsd'))` 182 | 183 | ```python 184 | StateMachine.build(source: Union[str, IOBase, Path], 185 | **kwargs) -> str 186 | ``` 187 | 188 | * `source`: The Heaviside DSL description to compile 189 | `Path` is either `pathlib.Path` or `heaviside.utils.Path` 190 | * `kwargs`: Additional key-word arguments to be passed to `json.dumps` 191 | 192 | ### Create 193 | 194 | Compile the given Heaviside DSL definition and then create the Step Function in AWS. 195 | 196 | `arn = machine.create(heaviside.utils.Path('/path/to/hsd/file.hsd'))` 197 | 198 | ```python 199 | StateMachine.create(source: Union[str, IOBase, Path], 200 | role: str) -> arn 201 | ``` 202 | 203 | * `source`: The Heaviside DSL description to compile 204 | `Path` is either `pathlib.Path` or `heaviside.utils.Path` 205 | * `role`: AWS IAM role for the State Machine to execute under 206 | This can either be the full ARN or just the IAM role name 207 | * Returns: ARN of the Step Function 208 | 209 | ### Update 210 | 211 | Update the Heaviside DSL definition and / or the IAM role of the Step Function in AWS 212 | 213 | `machine.update(heaviside.utils.Path('/path/to/hsd/file.hsd'))` 214 | 215 | ```python 216 | StateMachine.create(source: optional[Union[str, IOBase, Path]], 217 | role: optional[str]) -> None 218 | ``` 219 | 220 | * `source`: The Heaviside DSL description to compile 221 | `Path` is either `pathlib.Path` or `heaviside.utils.Path` 222 | * `role`: AWS IAM role for the State Machine to execute under 223 | This can either be the full ARN or just the IAM role name 224 | 225 | ### Delete 226 | 227 | Delete the AWS Step Function definition. After this the Step Function is no longer usable. 228 | 229 | `machine.delete()` 230 | 231 | ```python 232 | StateMachine.delete(exception: bool = False) -> None 233 | ``` 234 | 235 | * `exception`: If a `HeavisideError` should be raised if the Step Function doesn't exist 236 | 237 | ### Start 238 | 239 | Start the execution of a Step Function. 240 | 241 | `arn = machine.start({'key1': 'value1', 'key2': 'value2'})` 242 | 243 | ```python 244 | StateMachine.start(input_: object, 245 | name: Optional[str]) -> str 246 | ``` 247 | 248 | * `input_`: The input JSON text to pass to the start state 249 | * `name`: The name of the Step Function execution, if `None` then one will be generated 250 | * Returns: ARN of the Step Function execution 251 | 252 | ### Stop 253 | 254 | Stop the execution of an active Step Function execution. 255 | 256 | `machine.stop('arn:aws:...', 'Error Message', 'Cause Details')` 257 | 258 | ```python 259 | StateMachine.stop(arn: str, 260 | error: str, 261 | cause: str) -> None 262 | ``` 263 | 264 | * `arn`: The ARN of the running execution to terminate 265 | * `error`: The error message to report for the termination 266 | * `cause`: The error cause details to report for the termination 267 | 268 | ### Status 269 | 270 | Get the current status of an active Step Function execution. 271 | 272 | `status = machine.status('arn:aws:...')` 273 | 274 | ```python 275 | StateMachine.status(arn: str) -> str 276 | ``` 277 | 278 | * `arn`: The ARN of the running execution to get the status of 279 | * Returns: One of the following values: `RUNNING`, `SUCCEEDED`, `FAILED`, `TIMED_OUT`, `ABORTED` 280 | 281 | ### Wait 282 | 283 | Wait until an active Step Function execution has finished executing and get the output or the error information. 284 | 285 | `machine.wait(arn)` 286 | 287 | ```python 288 | StateMachine.wait(arn: str, 289 | period : int = 10) -> Dict 290 | ``` 291 | 292 | * `arn`: The ARN of the running execution to monitor 293 | * `period`: The number of seconds to sleep between polls for status 294 | * Returns: Step Function output or error event details 295 | 296 | ### Running ARNs 297 | 298 | Get a list of the ARNs for the running executions for the Step Function. 299 | 300 | `arns = machine.running_arns()` 301 | 302 | ```python 303 | StateMachine.running_args() -> List[str] 304 | ``` 305 | 306 | * Returns: List of ARNs for running executions of the named Step Function 307 | 308 | ## Activities 309 | 310 | AWS Step Function Activities are workers implemented and hosted by the user and are used to perfom a specific task. This gives the user a lot of flexibility, but with the trade off of more management. Activites are helpful if the task is longer running or requires more resources then a Lambda can handle. 311 | 312 | Heaviside provides several classes, detailed below, that support creating Activity workers and responding to Step Function requests. The majority of the logic is provided in the form of two mixin classes, one for activities (polling for tasking) and one for tasks (processing input and sending AWS the results or a failure), making it easier to integrate the functionality into other code. In addition to the mixins there are several classes, that use the mixins, that form the bases for a service that manages multiple activities and their worker processes. 313 | 314 | All of the Heaviside activity code is configured to use the Python `logging` module with the `heaviside.activities` prefix. 315 | 316 | ### Activity Mixin 317 | The main job of the ActivityMixin is to provide a method (`ActivityMixin.run()`) that polls the given Activity ARN for tasking. When received, it passes it to a method (`handle_task()`) that will actually process the request and send the response. The method handle_task is not defined in the mixin. 318 | 319 | If the method handle_task returns a non-None value, it is expected to be a Thread like object (with `start()` and `is_alive()` methods). ActivityMixin.run will start the Thread like object and keep track of how many objects are concurrently running. An upper bound can be placed on this number, and if it is reached, ActivityMixin.run will not accept any new tasking until a Thread like object has finished execution. 320 | 321 | ### Task Mixin 322 | The main job of the TaskMixin is to provide a wrapper around a method that processes the task input and produces an output. The wrapper takes care of sending the output back to AWS or capturing any exceptions and sending the failure back to AWS. 323 | 324 | The TaskMixin also supports sending a heartbeat when the Step Function expects it. The TaskMixin handles sending heartbeats by using generator coroutines to ceed control from the executing code back to the TaskMixin. The target code uses 'yield' to ceed execution and send a heartbeat. After the heartbeat has been sent, the coroutine resumes execution where it left offf. 325 | 326 | ```python 327 | def example_heartbeat(input_): 328 | for i in input_: 329 | yield # send heartbeat 330 | # process i 331 | 332 | # Python 3 333 | return output # whatever data to pass to the next step 334 | 335 | # Python 2 336 | yield output # whatever data to pass to the next step 337 | ``` 338 | 339 | ### Activity Object 340 | 341 | The Activity object (`heaviside.activities.Activity`) implements both mixins, making it easy to embed activity processing into existing code or framework. 342 | 343 | ### Task Process 344 | 345 | The TaskProcess object (`heaviside.activities.TaskProcess`) implements `heaviside.activities.TaskMixin` and `multiprocessing.Process` and is responsible for executing the given function and handling the results in a seperate Python process. 346 | 347 | ### Activity Process 348 | 349 | The ActivityProcess object (`heaviside.activities.ActivityProcess`) implements the `heaviside.activities.ActivityMixin` and `multiprocessing.Process` and is responsible for monitoring an Activity ARN for tasking and spawning a TaskProcess to handle the task, while it continues to monitor for more work. 350 | 351 | ### Activity Manager 352 | 353 | The ActivityManager (`heaviside.activities.ActivityManager`) is used to launch multiple ActivityProcesses and monitor them to ensure that they havn't crashed. If they have, it will start a new ActivityProcess. 354 | 355 | The class is designed to be subclassed to provide the list of ActivityProcesses that should be launched an monitored 356 | 357 | #### Example Activity Manager 358 | 359 | ```python 360 | from heaviside.activities import ActivityManager 361 | 362 | def example_activity(input_): 363 | # process input data 364 | return input_ 365 | 366 | class ExampleActivityManager(ActivityManager): 367 | def __init__(self, **kwargs): 368 | super(ExampleActivityManager, self).__init__(**kwargs) 369 | 370 | # Dictionary of activity name : activity function 371 | self.activities = { 372 | 'example': example_activity, 373 | } 374 | 375 | if __name__ == '__main__': 376 | manager = ExampleActivityManager() 377 | manager.run() 378 | ``` 379 | 380 | ### Fan-out 381 | Currently AWS Step Functions do not support dynamic parallel execution of 382 | tasks. As a temporary solution there is a function `heaviside.activities.fanout` 383 | that helps execute a dynamic number of step functions and collect the results. 384 | This requires splitting the desired functionality into a seperate step function, 385 | but provides an easy to implement solution. 386 | 387 | The fanout function will execute the given step function once per input argument 388 | and poll the execution ARN for the results. If successful, returns are 389 | aggregated into any array to be returned. If a failure occures, the failure 390 | information is extracted and raised as a `heaviside.exceptions.ActivityError`. 391 | If any exception is raised, fanout will attempt to stop any executing step 392 | functions before returning, so there are no orphaned executions running. 393 | 394 | The fanout function will also limit the number of concurrently executions and 395 | will slowly ramp up the execution of the step functions. The ramp up allows for 396 | other AWS resources to scale before the total number of executing processes 397 | are running. 398 | 399 | The fanout function also has two additional delay arguments that can be used to 400 | limit the rate at which AWS requests are made during status polling. These can be 401 | changed to keep the caller from exceeding the AWS throttling limits (200 unit 402 | bucket per account, refilling at 1 unit per second). 403 | 404 | **Note:** The fanout function currently maintains state in-memory, which means 405 | that if there is a failure, state / sub-state is lost. It also means that fanout 406 | works best when calling stateless / idempotent step functions or when the caller 407 | can clean up state before a retry is attempted 408 | -------------------------------------------------------------------------------- /tests/full.sfn: -------------------------------------------------------------------------------- 1 | { 2 | "States": { 3 | "Pass": { 4 | "Comment": "Comment", 5 | "ResultPath": "$.foo", 6 | "Next": "Lambda", 7 | "OutputPath": "$.foo", 8 | "Result": {}, 9 | "InputPath": "$.foo", 10 | "Type": "Pass" 11 | }, 12 | "Lambda": { 13 | "Comment": "Comment", 14 | "Retry": [ 15 | { 16 | "ErrorEquals": [ 17 | "one" 18 | ], 19 | "MaxAttempts": 1, 20 | "IntervalSeconds": 1, 21 | "BackoffRate": 1.0 22 | }, 23 | { 24 | "ErrorEquals": [ 25 | "two" 26 | ], 27 | "MaxAttempts": 1, 28 | "IntervalSeconds": 1, 29 | "BackoffRate": 1.0 30 | } 31 | ], 32 | "Resource": "arn:aws:lambda:::function:FUNCTION_NAME", 33 | "TimeoutSeconds": 2, 34 | "ResultPath": "$.foo", 35 | "HeartbeatSeconds": 1, 36 | "OutputPath": "$.foo", 37 | "Catch": [ 38 | { 39 | "ErrorEquals": [ 40 | "one" 41 | ], 42 | "Next": "Line30" 43 | }, 44 | { 45 | "ErrorEquals": [ 46 | "two" 47 | ], 48 | "ResultPath": "$.foo", 49 | "Next": "Line32" 50 | } 51 | ], 52 | "InputPath": "$.foo", 53 | "Next": "Activity", 54 | "Type": "Task" 55 | }, 56 | "Line30": { 57 | "Type": "Pass", 58 | "Next": "Activity" 59 | }, 60 | "Line32": { 61 | "Type": "Succeed" 62 | }, 63 | "Activity": { 64 | "Resource": "arn:aws:states:::activity:FUNCTION_NAME", 65 | "Type": "Task", 66 | "Next": "Raw ARN Task" 67 | }, 68 | "Raw ARN Task": { 69 | "Resource": "arn:aws:service:region:account:task_type:name", 70 | "Type": "Task", 71 | "Parameters": { 72 | "KeyOne": "ValueOne", 73 | "KeyTwo": "ValueTwo" 74 | }, 75 | "Next": "Batch.SubmitJob" 76 | }, 77 | "Batch.SubmitJob": { 78 | "Resource": "arn:aws:states:::batch:submitJob", 79 | "Type": "Task", 80 | "Parameters": { 81 | "JobName": "Name", 82 | "JobDefinition": "", 83 | "JobQueue": "arn", 84 | "ArrayProperties": {}, 85 | "ContainerOverrides": {}, 86 | "DependsOn": [], 87 | "Parameters": {}, 88 | "RetryStrategy": {}, 89 | "Timeout": {} 90 | }, 91 | "Next": "DynamoDB.GetItem" 92 | }, 93 | "DynamoDB.GetItem": { 94 | "Resource": "arn:aws:states:::dynamodb:getItem", 95 | "Type": "Task", 96 | "Parameters": { 97 | "TableName": "Table", 98 | "Key": {}, 99 | "AttributesToGet": [], 100 | "ConsistentRead": true, 101 | "ExpressionAttributeNames": {}, 102 | "ProjectionExpression": "", 103 | "ReturnConsumedCapacity": "" 104 | }, 105 | "Next": "DynamoDB.PutItem" 106 | }, 107 | "DynamoDB.PutItem": { 108 | "Resource": "arn:aws:states:::dynamodb:putItem", 109 | "Type": "Task", 110 | "Parameters": { 111 | "TableName": "Table", 112 | "Item": {}, 113 | "ConditionalOperator": "", 114 | "ConditionExpression": "", 115 | "Expected": {}, 116 | "ExpressionAttributeNames": {}, 117 | "ExpressionAttributeValues": {}, 118 | "ReturnConsumedCapacity": "", 119 | "ReturnItemCollectionMetrics": "", 120 | "ReturnValues": "" 121 | }, 122 | "Next": "DynamoDB.DeleteItem" 123 | }, 124 | "DynamoDB.DeleteItem": { 125 | "Resource": "arn:aws:states:::dynamodb:deleteItem", 126 | "Type": "Task", 127 | "Parameters": { 128 | "TableName": "Table", 129 | "Key": {}, 130 | "ConditionalOperator": "", 131 | "ConditionExpression": "", 132 | "Expected": {}, 133 | "ExpressionAttributeNames": {}, 134 | "ExpressionAttributeValues": {}, 135 | "ReturnConsumedCapacity": "", 136 | "ReturnItemCollectionMetrics": "", 137 | "ReturnValues": "" 138 | }, 139 | "Next": "DynamoDB.UpdateItem" 140 | }, 141 | "DynamoDB.UpdateItem": { 142 | "Resource": "arn:aws:states:::dynamodb:updateItem", 143 | "Type": "Task", 144 | "Parameters": { 145 | "TableName": "Table", 146 | "Key": {}, 147 | "AttributeUpdates": {}, 148 | "ConditionalOperator": "", 149 | "ConditionExpression": "", 150 | "Expected": {}, 151 | "ExpressionAttributeNames": {}, 152 | "ExpressionAttributeValues": {}, 153 | "ReturnConsumedCapacity": "", 154 | "ReturnItemCollectionMetrics": "", 155 | "ReturnValues": "", 156 | "UpdateExpression": "" 157 | }, 158 | "Next": "ECS.RunTask" 159 | }, 160 | "ECS.RunTask": { 161 | "Resource": "arn:aws:states:::ecs:runTask.sync", 162 | "Type": "Task", 163 | "Parameters": { 164 | "TaskDefinition": "", 165 | "Cluster": "", 166 | "Group": "", 167 | "LaunchType": "", 168 | "NetworkConfiguration": {}, 169 | "Overrides": {}, 170 | "PlacementConstraints": {}, 171 | "PlacementStrategy": {}, 172 | "PlatformVersion": "" 173 | }, 174 | "Next": "SNS.Publish" 175 | }, 176 | "SNS.Publish": { 177 | "Resource": "arn:aws:states:::sns:publish", 178 | "Type": "Task", 179 | "Parameters": { 180 | "Message": "", 181 | "MessageAttributes": {}, 182 | "MessageStructure": "", 183 | "PhoneNumber": "", 184 | "Subject": "", 185 | "TargetArn": "", 186 | "TopicArn": "" 187 | }, 188 | "Next": "SQS.SendMessage" 189 | }, 190 | "SQS.SendMessage": { 191 | "Resource": "arn:aws:states:::sqs:sendMessage", 192 | "Type": "Task", 193 | "Parameters": { 194 | "QueueUrl": "", 195 | "MessageBody": "", 196 | "DelaySeconds": 0, 197 | "MessageAttributes": {}, 198 | "MessageDeduplicationId": "", 199 | "MessageGroupId": "" 200 | }, 201 | "Next": "Glue.StartJobRun" 202 | }, 203 | "Glue.StartJobRun": { 204 | "Resource": "arn:aws:states:::glue:startJobRun.sync", 205 | "Type": "Task", 206 | "Parameters": { 207 | "JobName": "", 208 | "JobRunId": "", 209 | "Arguments": {}, 210 | "AllocatedCapacity": 0, 211 | "Timeout": 1, 212 | "SecurityConfiguration": "", 213 | "NotificationProperty": {} 214 | }, 215 | "Next": "SageMaker.CreateTrainingJob" 216 | }, 217 | "SageMaker.CreateTrainingJob": { 218 | "Resource": "arn:aws:states:::sagemaker:createTrainingJob.sync", 219 | "Type": "Task", 220 | "Parameters": { 221 | "TrainingJobName": "", 222 | "AlgorithmSpecification": {}, 223 | "OutputDataConfig": {}, 224 | "ResourceConfig": {}, 225 | "RoleArn": "", 226 | "StoppingCondition": {}, 227 | "HyperParameters": {}, 228 | "InputDataConfig": [], 229 | "Tags": [], 230 | "VpcConfig": {} 231 | }, 232 | "Next": "SageMaker.CreateTransformJob" 233 | }, 234 | "SageMaker.CreateTransformJob": { 235 | "Resource": "arn:aws:states:::sagemaker:createTransformJob.sync", 236 | "Type": "Task", 237 | "Parameters": { 238 | "TransformJobName": "", 239 | "ModelName": "", 240 | "TransformInput": {}, 241 | "TransformOutput": {}, 242 | "TransformResources": {}, 243 | "BatchStrategy": "", 244 | "Environment": {}, 245 | "MaxConcurrentTransforms": 0, 246 | "MaxPayloadInMB": 0, 247 | "Tags": [] 248 | }, 249 | "Next": "Wait-Seconds" 250 | }, 251 | "Wait-Seconds": { 252 | "Seconds": 1, 253 | "Type": "Wait", 254 | "Next": "Wait-Timestamp" 255 | }, 256 | "Wait-Timestamp": { 257 | "Timestamp": "1111-11-11T11:11:11Z", 258 | "Type": "Wait", 259 | "Next": "Wait-Seconds-Path" 260 | }, 261 | "Wait-Seconds-Path": { 262 | "SecondsPath": "$.foo", 263 | "Type": "Wait", 264 | "Next": "Wait-Timestamp-Path" 265 | }, 266 | "Wait-Timestamp-Path": { 267 | "Comment": "Comment", 268 | "OutputPath": "$.foo", 269 | "Next": "While", 270 | "TimestampPath": "$.foo", 271 | "InputPath": "$.foo", 272 | "Type": "Wait" 273 | }, 274 | "While": { 275 | "InputPath": "$.foo", 276 | "OutputPath": "$.foo", 277 | "Default": "If-Elif-Else", 278 | "Type": "Choice", 279 | "Choices": [ 280 | { 281 | "Variable": "$.foo", 282 | "Next": "While-Body", 283 | "NumericEquals": 1 284 | } 285 | ] 286 | }, 287 | "While-Body": { 288 | "Type": "Pass", 289 | "Next": "WhileLoop" 290 | }, 291 | "WhileLoop": { 292 | "Type": "Pass", 293 | "Next": "While" 294 | }, 295 | "If-Elif-Else": { 296 | "InputPath": "$.foo", 297 | "OutputPath": "$.foo", 298 | "Default": "Line273", 299 | "Type": "Choice", 300 | "Choices": [ 301 | { 302 | "Or": [ 303 | { 304 | "Variable": "$.foo", 305 | "NumericEquals": 1 306 | }, 307 | { 308 | "And": [ 309 | { 310 | "Variable": "$.foo", 311 | "NumericGreaterThanEquals": 10 312 | }, 313 | { 314 | "Not": { 315 | "Variable": "$.foo", 316 | "NumericLessThan": 20 317 | } 318 | } 319 | ] 320 | } 321 | ], 322 | "Next": "Line233" 323 | }, 324 | { 325 | "Variable": "$.foo", 326 | "NumericLessThanEquals": 1, 327 | "Next": "Line235" 328 | }, 329 | { 330 | "Variable": "$.foo", 331 | "NumericLessThan": 1, 332 | "Next": "Line237" 333 | }, 334 | { 335 | "Variable": "$.foo", 336 | "NumericGreaterThanEquals": 1, 337 | "Next": "Line239" 338 | }, 339 | { 340 | "Variable": "$.foo", 341 | "NumericGreaterThan": 1, 342 | "Next": "Line241" 343 | }, 344 | { 345 | "Not": { 346 | "Variable": "$.foo", 347 | "NumericEquals": 1 348 | }, 349 | "Next": "Line243" 350 | }, 351 | { 352 | "Variable": "$.foo", 353 | "StringEquals": "1", 354 | "Next": "Line245" 355 | }, 356 | { 357 | "Variable": "$.foo", 358 | "StringLessThanEquals": "1", 359 | "Next": "Line247" 360 | }, 361 | { 362 | "Variable": "$.foo", 363 | "StringLessThan": "1", 364 | "Next": "Line249" 365 | }, 366 | { 367 | "Variable": "$.foo", 368 | "StringGreaterThanEquals": "1", 369 | "Next": "Line251" 370 | }, 371 | { 372 | "Variable": "$.foo", 373 | "Next": "Line253", 374 | "StringGreaterThan": "1" 375 | }, 376 | { 377 | "Not": { 378 | "Variable": "$.foo", 379 | "StringEquals": "1" 380 | }, 381 | "Next": "Line255" 382 | }, 383 | { 384 | "Variable": "$.foo", 385 | "BooleanEquals": true, 386 | "Next": "Line257" 387 | }, 388 | { 389 | "Not": { 390 | "Variable": "$.foo", 391 | "BooleanEquals": true 392 | }, 393 | "Next": "Line259" 394 | }, 395 | { 396 | "Variable": "$.foo", 397 | "TimestampEquals": "1111-11-11T11:11:11Z", 398 | "Next": "Line261" 399 | }, 400 | { 401 | "Variable": "$.foo", 402 | "TimestampLessThanEquals": "1111-11-11T11:11:11Z", 403 | "Next": "Line263" 404 | }, 405 | { 406 | "Variable": "$.foo", 407 | "TimestampLessThan": "1111-11-11T11:11:11Z", 408 | "Next": "Line265" 409 | }, 410 | { 411 | "Variable": "$.foo", 412 | "TimestampGreaterThanEquals": "1111-11-11T11:11:11Z", 413 | "Next": "Line267" 414 | }, 415 | { 416 | "Variable": "$.foo", 417 | "TimestampGreaterThan": "1111-11-11T11:11:11Z", 418 | "Next": "Line269" 419 | }, 420 | { 421 | "Not": { 422 | "Variable": "$.foo", 423 | "TimestampEquals": "1111-11-11T11:11:11Z" 424 | }, 425 | "Next": "Line271" 426 | } 427 | ] 428 | }, 429 | "Line233": { 430 | "Type": "Pass", 431 | "Next": "Switch" 432 | }, 433 | "Line235": { 434 | "Type": "Pass", 435 | "Next": "Switch" 436 | }, 437 | "Line237": { 438 | "Type": "Pass", 439 | "Next": "Switch" 440 | }, 441 | "Line239": { 442 | "Type": "Pass", 443 | "Next": "Switch" 444 | }, 445 | "Line241": { 446 | "Type": "Pass", 447 | "Next": "Switch" 448 | }, 449 | "Line243": { 450 | "Type": "Pass", 451 | "Next": "Switch" 452 | }, 453 | "Line245": { 454 | "Type": "Pass", 455 | "Next": "Switch" 456 | }, 457 | "Line247": { 458 | "Type": "Pass", 459 | "Next": "Switch" 460 | }, 461 | "Line249": { 462 | "Type": "Pass", 463 | "Next": "Switch" 464 | }, 465 | "Line251": { 466 | "Type": "Pass", 467 | "Next": "Switch" 468 | }, 469 | "Line253": { 470 | "Type": "Pass", 471 | "Next": "Switch" 472 | }, 473 | "Line255": { 474 | "Type": "Pass", 475 | "Next": "Switch" 476 | }, 477 | "Line257": { 478 | "Type": "Pass", 479 | "Next": "Switch" 480 | }, 481 | "Line259": { 482 | "Type": "Pass", 483 | "Next": "Switch" 484 | }, 485 | "Line261": { 486 | "Type": "Pass", 487 | "Next": "Switch" 488 | }, 489 | "Line263": { 490 | "Type": "Pass", 491 | "Next": "Switch" 492 | }, 493 | "Line265": { 494 | "Type": "Pass", 495 | "Next": "Switch" 496 | }, 497 | "Line267": { 498 | "Type": "Pass", 499 | "Next": "Switch" 500 | }, 501 | "Line269": { 502 | "Type": "Pass", 503 | "Next": "Switch" 504 | }, 505 | "Line271": { 506 | "Type": "Pass", 507 | "Next": "Switch" 508 | }, 509 | "Line273": { 510 | "Type": "Pass", 511 | "Next": "Switch" 512 | }, 513 | "Switch": { 514 | "InputPath": "$.foo", 515 | "OutputPath": "$.foo", 516 | "Default": "Line289", 517 | "Type": "Choice", 518 | "Choices": [ 519 | { 520 | "Variable": "$.a", 521 | "Next": "Line281", 522 | "NumericEquals": 1 523 | }, 524 | { 525 | "Variable": "$.a", 526 | "StringEquals": "foo", 527 | "Next": "Line283" 528 | }, 529 | { 530 | "Variable": "$.a", 531 | "TimestampEquals": "1111-11-11T11:11:11Z", 532 | "Next": "Line285" 533 | }, 534 | { 535 | "Variable": "$.a", 536 | "BooleanEquals": false, 537 | "Next": "Line287" 538 | } 539 | ] 540 | }, 541 | "Line281": { 542 | "Type": "Pass", 543 | "Next": "Parallel" 544 | }, 545 | "Line283": { 546 | "Type": "Pass", 547 | "Next": "Parallel" 548 | }, 549 | "Line285": { 550 | "Type": "Pass", 551 | "Next": "Parallel" 552 | }, 553 | "Line287": { 554 | "Type": "Pass", 555 | "Next": "Parallel" 556 | }, 557 | "Line289": { 558 | "Type": "Pass", 559 | "Next": "Parallel" 560 | }, 561 | "Parallel": { 562 | "Retry": [ 563 | { 564 | "ErrorEquals": [ 565 | "States.ALL" 566 | ], 567 | "MaxAttempts": 0, 568 | "IntervalSeconds": 1, 569 | "BackoffRate": 1.0 570 | } 571 | ], 572 | "Branches": [ 573 | { 574 | "States": { 575 | "Success": { 576 | "Comment": "Comment", 577 | "InputPath": "$.foo", 578 | "Type": "Succeed", 579 | "OutputPath": "$.foo" 580 | } 581 | }, 582 | "StartAt": "Success" 583 | }, 584 | { 585 | "States": { 586 | "Fail": { 587 | "Comment": "Comment", 588 | "Cause": "cause", 589 | "Type": "Fail", 590 | "Error": "error" 591 | } 592 | }, 593 | "StartAt": "Fail" 594 | } 595 | ], 596 | "ResultPath": "$.foo", 597 | "OutputPath": "$.foo", 598 | "Catch": [ 599 | { 600 | "ErrorEquals": [ 601 | "States.ALL" 602 | ], 603 | "Next": "Line317" 604 | } 605 | ], 606 | "InputPath": "$.foo", 607 | "End": true, 608 | "Type": "Parallel" 609 | }, 610 | "Line317": { 611 | "Type": "Pass", 612 | "Next": "Switch" 613 | } 614 | }, 615 | "Comment": "State machine comment", 616 | "Version": "1.0", 617 | "StartAt": "Pass", 618 | "TimeoutSeconds": 60 619 | } -------------------------------------------------------------------------------- /docs/StepFunctionDSL.md: -------------------------------------------------------------------------------- 1 | # AWS Step Function DSL 2 | 3 | This document describes a domain specific language (DSL) for AWS Step Function 4 | (SFN) state machines. The using the Python stepfunctions library the DSL can be 5 | compiled down to the [AWS States Language][language definition]. 6 | 7 | For more information on the Python [stepfunctions library] or its use visit the 8 | libraries page. 9 | 10 | Copyright 2016 The Johns Hopkins University Applied Physics Laboratory 11 | 12 | Licensed under the Apache License, Version 2.0 (the "License"); 13 | you may not use this file except in compliance with the License. 14 | You may obtain a copy of the License at 15 | 16 | http://www.apache.org/licenses/LICENSE-2.0 17 | 18 | Unless required by applicable law or agreed to in writing, software 19 | distributed under the License is distributed on an "AS IS" BASIS, 20 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 21 | See the License for the specific language governing permissions and 22 | limitations under the License. 23 | 24 | 25 | ## Table of Contents: 26 | 27 | * [Why](#Why) 28 | * [Style](#Style) 29 | * [Structure](#Structure) 30 | * [Concepts](#Concepts) 31 | - [Error Names](#Error-Names) 32 | - [Timestamps](#Timestamps) 33 | - [JsonPath](#JsonPath) 34 | * [Top-Level Fields](#Top-Level-Fields) 35 | * [States](#States) 36 | - [Basic States](#Basic-States) 37 | - [Success State](#Success-State) 38 | - [Fail State](#Fail-State) 39 | - [Pass State](#Pass-State) 40 | - [Task State](#Task-State) 41 | - [Wait State](#Wait-State) 42 | - [Map State](#Map-State) 43 | - [Flow Control States](#Flow-Control-States) 44 | - [Comparison Operators](#Comparison-Operators) 45 | - [If](#If) 46 | - [Switch](#Switch) 47 | - [While Loop](#While-Loop) 48 | - [Parallel](#Parallel) 49 | - [Goto](#Goto) 50 | 51 | ## Why 52 | When Amazon released AWS Step Functions they provided a [language definition] 53 | for writing state machine in. While functional, it is cumbersome to write and 54 | maintain state machines in their Json format. This DSL is designed to make it 55 | easier to read, write, and maintain step function state machines. 56 | 57 | The biggest benefit of using the DSL for writing a state machine is that when 58 | compiled to the AWS Json format by a library like [stepfunctions library] the 59 | states can be automatically linked together, instead of manually having to 60 | specify the next state for each state. 61 | 62 | The single flow control state has be translated into two of the basic flow 63 | control operations used in programming (if/elif/else and while loop). 64 | 65 | ## Style 66 | The DSL's style is influenced by Python code style. 67 | 68 | * It is an indent based language where the level of indent is used to specify a 69 | block of code. 70 | * Strings can be defined using `'`, `"`, `'''`, `"""` 71 | * Doc String style comments for states 72 | 73 | ## Structure 74 | The SFN DSL format is an optional top level comment followed by a list of states. 75 | 76 | ### Example 77 | """Simple Example of the SFN DSL""" 78 | Lambda('HelloWorld') 79 | 80 | Execution of the state machine is started at the first state in the file and 81 | execution proceedes until the state at the end of the file is reached or until 82 | a state terminates execution. 83 | 84 | In this example there is one state. The full ARN for the Lambda will be determined 85 | when the DSL is compiled into the AWS Json format. The full ARN can be passed if 86 | the desired Lambda doesn't reside in the same account or region as the connection 87 | used to compile and create the state machine. 88 | 89 | ## Concepts 90 | ### Error Names 91 | There is a predefined set of basic errors that can happen. 92 | [State machine errors reference][language definition errors]. 93 | 94 | ### Timestamps 95 | The SFN DSL supports comparison against timestamp values. The way a timestamp is 96 | determined, compared to a regular string, is that it can be parsed as a timestamp 97 | according to RFC3339. This format often looks like `yyyy-mm-ddThh:mm:ssZ`. If a 98 | timestamp is not in the correct format the comparison will be performed as a 99 | string comparison. 100 | 101 | ### JsonPath 102 | State machines use a version of JsonPath for referencing data that is is being 103 | processed. [State machine path reference][language definition path]. 104 | 105 | ## Top-Level Fields 106 | A SFN DSL file consists of three optional fields followed by a list of States. 107 | 108 | """State machine comment""" 109 | version: "1.0" 110 | timeout: int 111 | 112 | The top level comment is the comment for the state machine that is created. 113 | 114 | * `version`: Is a string of the version number of the State Machine language to 115 | compile down to. Currently only `"1.0"` is supported. 116 | * `timeout`: The overall timeout in seconds for the whole state machine execution. 117 | If the state machine has not finished execution within this time 118 | the execution fails with a `States.Timeout` error. 119 | 120 | ## States 121 | The different types of state machine states are divided into two categories. 122 | Basic states are those that perform a single action and, potentially, link to 123 | another state. Flow control states are those that apply some flow control logic. 124 | 125 | ### Basic States 126 | #### Success State 127 | A terminal state, `Success()` will cause the state machine to terminate execution 128 | successfully and return a result value. 129 | 130 | Success() 131 | """State Name 132 | State Comment""" 133 | input: JsonPath 134 | output: JsonPath 135 | 136 | States can have a Python style doc string. If given, the first line of the doc 137 | string is the state's name and the rest if the states comment. If no name is given 138 | (or an empty name) the state's name is built from the line number. 139 | 140 | Modifiers: 141 | * `input`: JsonPath selecting a value from the input object to be passed to the 142 | current state (Default: `"$"`) 143 | * `output`: JsonPath selecting a value from the output object to be passed to the 144 | next state (Default: `"$"`) 145 | 146 | #### Fail State 147 | A terminal state, `Fail()` will cause the state machine to terminate execution 148 | unsuccessfully with the given error and cause values. 149 | 150 | Fail(error, cause) 151 | """State Name 152 | State Comment""" 153 | 154 | Arguments: 155 | * `error`: String containing the error value 156 | * `cause`: String containing the error's cause, a more readable value 157 | 158 | #### Pass State 159 | A state that does nothing `Pass()` can be used to modify the data being passed 160 | around or inject new data into the results. 161 | 162 | Pass() 163 | """State Name 164 | State Comment""" 165 | input: JsonPath 166 | result: JsonPath 167 | output: JsonPath 168 | parameters: 169 | Key1: {"JSON": "Text"} 170 | Key2: "string-value" 171 | data: 172 | Json 173 | 174 | Modifiers: 175 | * `result`: JsonPath of where to place the results of the state, relative to the 176 | raw input (before the `input` modifier was applied) (Default: `"$"`) 177 | * `parameters`: Keyword arguments to be passed in the API call. The value is 178 | JSON text. The parameters will override the the incoming 179 | input, but JsonPaths may be used to select from existing 180 | inputs. 181 | NOTE: If the value contains a JsonPath the key must end wit `.$` 182 | AWS documentation: https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html 183 | * `data`: A block of Json data that will be used as the result of the state 184 | 185 | #### Task State 186 | A task is a unit of work to be performed by the state machine and it is broken 187 | into two different categories. The first is AWS maintained APIs that can be used 188 | to execute a limited number of functions from other AWS services. The second is 189 | user created and/or maintained workers that perform whatever custom logic is 190 | coded. 191 | 192 | AWS maintained APIs are in the format `_Type_._Function_()` and use the 193 | `parameters` block to pass argument. Some of the APIs are for asynchronous 194 | functions. By default Heaviside is configured to make these API calls as 195 | synchronous calls, so that the Step Function waits for the function's return. 196 | __To disable this default behavior__ the `sync: False` parameter can be used in 197 | the `parameters` block to convert the API call back to a asynchronous call. 198 | 199 | User maintained workers are either `Lambda()` or `Activity()`, with the difference 200 | being where the code that will be executed is located. For `Lambda()` the code 201 | is an AWS Lambda function. For `Activity()` the code can be running anywhere, 202 | and is responsible for polling AWS to see if there is new work for it to perform. 203 | 204 | Activity ARNs are created in the Step Functions section of AWS (console or API). 205 | Once defined multiple workers can start polling for work and state machines can send 206 | data to the worker(s) for processing. 207 | 208 | _Type_('name') 209 | _Type_._Function_() 210 | Arn('arn') 211 | """State Name 212 | State Comment""" 213 | timeout: int 214 | heartbeat: int 215 | input: JsonPath 216 | result: JsonPath 217 | output: JsonPath 218 | parameters: 219 | Key1: {"JSON": "Text"} 220 | Key2.$: "$.json_path.to.input.data" 221 | retry error(s) retry interval (seconds), max attempts, backoff rate 222 | catch error(s): JsonPath 223 | State(s) 224 | 225 | Tasks: 226 | * `_Type_('name')`: Either `Lambda()` or `Activity()` 227 | * `_Type_._Function_()`: One of the following (follow the link for details about 228 | what arguments are valid and required for the `parameters` 229 | block. 230 | - [Batch.SubmitJob](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-batch.html) 231 | - [DynamoDB.GetItem](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ddb.html) 232 | - [DynamoDB.PutItem](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ddb.html) 233 | - [DynamoDB.DeleteItem](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ddb.html) 234 | - [DynamoDB.UpdateItem](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ddb.html) 235 | - [ECS.RunTask](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ecs.html) 236 | - [SNS.Publish](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-sns.html) 237 | - [SQS.SendMessage](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-sqs.html) 238 | - [Glue.StartJobRun](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-glue.html) 239 | - [SageMaker.CreateTrainingJob](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-sagemaker.html) 240 | - [SageMaker.CreateTransformJob](https://docs.aws.amazon.com/step-functions/latest/dg/connectors-sagemaker.html) 241 | * `Arn('arn')`: Raw access for specifying the full ARN of the task to execute 242 | 243 | Arguments: 244 | * `name`: The name of the Lambda or Activity. The full ARN will be created at 245 | compile time using the given AWS region and account information. 246 | * `arn`: The full ARN of an Activity, Lambda, or other AWS API to be executed. 247 | This allows calling Activities hosted in different regions / accounts 248 | or calling new AWS APIs before this library is updated to handle them. 249 | 250 | Modifiers: 251 | * `timeout`: Number of seconds before the task times out (Default: 60 seconds) 252 | * `heatbeat`: Number of seconds before the task times out if no heartbeat has been 253 | received from the task. Needs to be less than the `timeout` value. 254 | * `parameters`: Keyword arguments to be passed in the API call. The value is a 255 | JSON text, which may be a JsonPath referencing data from the 256 | state's input data object. 257 | NOTE: If the value contains a JsonPath the key must end with `.$` 258 | * `retry`: If the given error(s) were encountered, rerun the state 259 | - `error(s)`: A single string, array of strings, or empty array of errors to match 260 | against. An empty array matches against all errors. 261 | - `retry interval`: Number of seconds to wait before the first retry 262 | - `max attempts`: Number of retries to attempt before passing errors to `catch` 263 | modifiers. Zero (0) is a valid value, meaning don't retry. 264 | - `backoff rate`: The multipler that increases the `retry interval` on each attempt 265 | * `catch`: If the given error(s) were encountered and not handled by a `retry` 266 | then execute the given states. If the states in the catch block don't 267 | terminate, then execution will continue on the next valid state. 268 | - `error(s)`: A single string, array of strings, or empty array of errors to match 269 | against. An empty array matches against all errors. 270 | - JsonPath: An optional JsonPath string about where to place the error information 271 | in relationship to the errored state's input. By default the error 272 | information will replace the errored state's input. 273 | Error information is a dictionary containing the key 'Error' and 274 | possibly the key 'Cause'. 275 | 276 | Note: Ordering of everything besides `retry` and `catch` is currently fixed. There 277 | can be multiple `retry` and `catch` statements and there is no ordering of 278 | those modifiers. 279 | 280 | #### Wait State 281 | There are four different versions of the `Wait()` state, but each pauses execution 282 | for a given amount of time. 283 | 284 | Wait(seconds=int) 285 | Wait(timestamp='yyyy-mm-ddThh:mm:ssZ') 286 | Wait(seconds_path=JsonPath) 287 | Wait(timestamp_path=JsonPath) 288 | """State Name 289 | State Comment""" 290 | input: JsonPath 291 | output: JsonPath 292 | 293 | Arguments: 294 | * `seconds`: Number of seconds to wait 295 | * `timestamp`: Wait until the specified time 296 | * `seconds_path`: Read the number of seconds to wait from the given JsonPath 297 | * `timestamp_path`: Read the timestamp to wait until from the givne JsonPath 298 | 299 | #### Map State 300 | This state defines an independent state machine that runs on each element of 301 | the input array. 302 | 303 | Modifiers: 304 | * `iterator`: Required - the independent state machine is defined here. 305 | * `max_concurrency`: The maximum instances of the iterator that may run in 306 | parallel. 307 | * `items_path`: JsonPath to the array to run the map on. Defaults to `"$"`. 308 | * `parameters`: Keyword arguments to be passed in the API call. The value is a 309 | JSON text, which may be a JsonPath referencing data from the 310 | state's input data object. 311 | NOTE: If the value contains a JsonPath the key must end with `.$` 312 | * `result`: JsonPath of where to place the results of the state, relative to the 313 | raw input (before the `input` modifier was applied) (Default: `"$"`) 314 | * `output`: JsonPath selecting a value from the output object to be passed to the 315 | next state (Default: `"$"`) 316 | * `retry`: If the given error(s) were encountered, rerun the state 317 | - `error(s)`: A single string, array of strings, or empty array of errors to match 318 | against. An empty array matches against all errors. 319 | - `retry interval`: Number of seconds to wait before the first retry 320 | - `max attempts`: Number of retries to attempt before passing errors to `catch` 321 | modifiers. Zero (0) is a valid value, meaning don't retry. 322 | - `backoff rate`: The multipler that increases the `retry interval` on each attempt 323 | * `catch`: If the given error(s) were encountered and not handled by a `retry` 324 | then execute the given states. If the states in the catch block don't 325 | terminate, then execution will continue on the next valid state. 326 | - `error(s)`: A single string, array of strings, or empty array of errors to match 327 | against. An empty array matches against all errors. 328 | - JsonPath: An optional JsonPath string about where to place the error information 329 | in relationship to the errored state's input. By default the error 330 | information will replace the errored state's input. 331 | Error information is a dictionary containing the key 'Error' and 332 | possibly the key 'Cause'. 333 | 334 | Example showing `map` used modifiers: 335 | 336 | ``` 337 | version: "1.0" 338 | timeout: 60 339 | Pass() 340 | """CreateSomeInputs""" 341 | result: '$' 342 | data: 343 | { 344 | "the_array": [1, 2, 3, 4, 5], 345 | "foo": "bar" 346 | } 347 | map: 348 | """TransformInputsWithMap""" 349 | iterator: 350 | Pass() 351 | """MapPassState""" 352 | Wait(seconds=2) 353 | """WaitState""" 354 | parameters: 355 | foo.$: "$.foo" 356 | element.$: "$$.Map.Item.Value" 357 | items_path: "$.the_array" 358 | result: "$.transformed" 359 | output: "$.transformed" 360 | max_concurrency: 4 361 | retry [] 1 0 1.0 362 | catch []: 363 | Pass() 364 | """SomeErrorHandler""" 365 | ``` 366 | 367 | ### Flow Control States 368 | #### Comparison Operators 369 | 370 | Value Type | Supported Operators 371 | -----------|-------------------- 372 | Boolean | ==, != 373 | Integer | ==, !=, <, >, <=, >= 374 | Float | ==, !=, <, >, <=, >= 375 | String | ==, !=, <, >, <=, >= 376 | Timestamp | ==, !=, <, >, <=, >= 377 | 378 | Comparison operators can be composed using (order of list is order of precedence): 379 | * () 380 | * not 381 | * and 382 | * or 383 | 384 | #### If 385 | The basic `if` statement. Multiple (or no) `elif` statements can be included. The 386 | `else` statement is also optional. 387 | 388 | if JsonPath operator value: 389 | """State Name 390 | State Comment""" 391 | State(s) 392 | elif JsonPath operator value: 393 | State(s) 394 | else: 395 | State(s) 396 | transform: 397 | input: JsonPath 398 | output: JsonPath 399 | 400 | The `transform` block contains the same `input` and `output` modifiers that the 401 | simple states use. 402 | 403 | Modifiers: 404 | * `input`: JsonPath selecting a value from the input object to be passed to the 405 | current state, applied before performing the comparison (Default: 406 | `"$"`) 407 | * `output`: JsonPath selecting a value from the output object to be passed to the 408 | next state, applied before executing the first state in the selected 409 | branch (Default: `"$"`) 410 | 411 | #### Switch 412 | The basic `switch` statement. Multiple `case` statements are compared against the 413 | JsonPath variable in the `switch` statement. The `default` statement is optional. 414 | 415 | switch JsonPath: 416 | """State Name 417 | State Comment""" 418 | case value: 419 | State(s) 420 | default: 421 | State(s) 422 | transform: 423 | input: JsonPath 424 | output: JsonPath 425 | 426 | #### While Loop 427 | The basic `while` loop the continues to execute the given states until the condition 428 | is no longer true. 429 | 430 | while JsonPath operator value: 431 | """State Name 432 | State Comment""" 433 | State(s) 434 | transform: 435 | input: JsonPath 436 | output: JsonPath 437 | 438 | #### Parallel 439 | The `parallel` control structure allows running multiple branches of execution 440 | in parallel. The parallel state waits until all branches have finished before 441 | moving to the next state. If there is any unhandled error in any branch, the 442 | whole state is considered to have failed. 443 | 444 | The state's input is passed to the first state in each branch and the results of 445 | the parallel state is an array of outputs from each branch. 446 | 447 | parallel: 448 | """State Name 449 | State Comment""" 450 | State(s) 451 | parallel: 452 | State(s) 453 | transform: 454 | input: JsonPath 455 | result: JsonPath 456 | output: JsonPath 457 | error: 458 | retry error(s) retry interval (seconds), max attempts, backoff rate 459 | catch error(s): JsonPath 460 | State(s) 461 | 462 | The `transform` block contains the same `input`, `result`, and `output` modifiers 463 | that the simple states use. 464 | 465 | Modifiers: 466 | * `input`: JsonPath selecting a value from the input object to be passed to the 467 | first state in each parallel branch (Default: `"$"`) 468 | * `result`: JsonPath of where to place the array of results from each of the 469 | parallel branches (before the `input` modifier was applied) (Default: 470 | `"$"`) 471 | * `output`: JsonPath selecting a value from the output object to be passed to the 472 | next state (Default: `"$"`) 473 | 474 | The `error` block contains the same `retry` and `catch` modifiers as the task state. 475 | 476 | #### Goto 477 | The `goto` control statement allows jumping to another state. This can be used 478 | to create common error handling routines (among other uses). 479 | 480 | The state can only target states within the current branch of execution. This 481 | means either the main body of the Step Function or within a branch of a Parallel 482 | state. 483 | 484 | goto "State Name" 485 | 486 | [stepfunctions library]: https://github.com/jhuapl-boss/heaviside 487 | [language definition]: https://states-language.net/spec.html 488 | [language definition errors]: https://states-language.net/spec.html#appendix-a 489 | [language definition path]: https://states-language.net/spec.html#path 490 | --------------------------------------------------------------------------------