├── images ├── .gitkeep ├── appflow-console-home.png ├── appflow-console-create-flow.png ├── appflow-console-connector-label.png ├── appflow-console-connectors-page.png ├── appflow-console-lambda-function.png ├── appflow-console-test-connector.png ├── appflow-console-create-connection.png ├── appflow-console-connection-details.png ├── appflow-console-registered-connector.png ├── appflow-console-custom-connectors-dropdown.png └── appflow-console-custom-connectors-view-details.png ├── .gitignore ├── custom_connector_sdk ├── __init__.py ├── test │ ├── __init__.py │ └── resources │ │ ├── describe_connector_configuration_request_required.json │ │ ├── describe_connector_configuration_request_optional.json │ │ ├── validate_connector_runtime_settings_request_invalid.json │ │ ├── retrieve_data_response_required.json │ │ ├── validate_credentials_response_required.json │ │ ├── write_data_response_required.json │ │ ├── query_data_response_required.json │ │ ├── describe_entity_response_required.json │ │ ├── validate_credentials_request_required.json │ │ ├── list_entities_response_required.json │ │ ├── validate_connector_runtime_settings_response_required.json │ │ ├── validate_credentials_response_optional.json │ │ ├── validate_connector_runtime_settings_request_required.json │ │ ├── list_entities_request_required.json │ │ ├── write_data_request_invalid.json │ │ ├── retrieve_data_request_invalid.json │ │ ├── describe_entity_request_required.json │ │ ├── retrieve_data_response_optional.json │ │ ├── query_data_response_optional.json │ │ ├── validate_credentials_request_optional.json │ │ ├── validate_connector_runtime_settings_response_optional.json │ │ ├── write_data_response_optional.json │ │ ├── describe_entity_response_optional.json │ │ ├── list_entities_response_optional.json │ │ ├── query_data_request_required.json │ │ ├── write_data_request_required.json │ │ ├── retrieve_data_request_required.json │ │ ├── describe_entity_request_invalid.json │ │ ├── validate_credentials_request_invalid.json │ │ ├── describe_connector_configuration_response_required.json │ │ ├── describe_connector_configuration_response_optional.json │ │ ├── describe_entity_request_optional.json │ │ ├── retrieve_data_request_optional.json │ │ ├── query_data_request_invalid.json │ │ ├── write_data_request_optional.json │ │ ├── list_entities_request_optional.json │ │ ├── query_data_request_optional.json │ │ └── list_entities_request_invalid.json ├── connector │ ├── __init__.py │ ├── configuration.py │ ├── settings.py │ └── context.py ├── lambda_handler │ ├── __init__.py │ ├── lambda_handler.py │ └── handlers.py ├── marketplace │ ├── __init__.py │ └── entititlement_util.py └── .DS_Store ├── custom_connector_example ├── __init__.py ├── query │ ├── __init__.py │ └── builder.py ├── test │ ├── __init__.py │ ├── resources │ │ ├── null_sobjects_server_response.json │ │ ├── describe_connector_configuration_request_valid.json │ │ ├── get_sobjects_server_response.json │ │ ├── validate_connector_runtime_settings_request_invalid.json │ │ ├── validate_connector_runtime_settings_request_valid.json │ │ ├── validate_credentials_request_valid.json │ │ ├── list_entities_request_invalid.json │ │ ├── describe_entity_request_invalid.json │ │ ├── validate_credentials_request_invalid.json │ │ ├── list_entities_request_no_path.json │ │ ├── describe_entity_request_valid.json │ │ ├── retrieve_data_request_invalid.json │ │ ├── list_entities_request_valid.json │ │ ├── query_data_request_invalid.json │ │ ├── write_data_request_invalid.json │ │ ├── write_data_request_valid.json │ │ ├── describe_sobject_server_response.json │ │ ├── query_data_request_no_selected_fields.json │ │ ├── query_data_request_no_filter_expression.json │ │ ├── query_data_request_filter_expression.json │ │ └── retrieve_data_request_valid.json │ ├── configuration_test.py │ ├── metadata_test.py │ ├── query_test.py │ └── record_test.py ├── handlers │ ├── __init__.py │ ├── lambda_handler.py │ ├── validation.py │ ├── salesforce.py │ ├── client.py │ └── configuration.py ├── integ_test │ ├── __init__.py │ ├── sales_generator.py │ └── sales_test_case.py ├── salesforce-example-test-files │ ├── salesforce-insert-file.csv │ ├── describe-connector-entity-validation-file.json │ ├── test-file-3.json │ ├── list-entities-validation-file.json │ ├── test-file-2.json │ ├── test-file.json │ └── describe-connector-validation-file.json ├── constants.py └── template.yml ├── custom_connector_tests ├── __init__.py ├── invokers │ ├── __init__.py │ └── test_invoker.py ├── exceptions │ ├── __init__.py │ └── ValidationException.py ├── validation │ ├── __init__.py │ └── connector_configuration_validator.py ├── run_tests.py └── configuration │ ├── TestConfig.json │ └── ConfigSchema.json ├── custom_connector_integ_test ├── __init__.py ├── utils │ ├── __init__.py │ ├── s3_helper.py │ ├── flow_poller.py │ └── resource_info_provider.py ├── appflow_test │ └── __init__.py ├── configuration │ ├── __init__.py │ └── services.py ├── base-test-config.json ├── sample-test-config.json └── README.md ├── custom_connector_queryfilter ├── __init__.py ├── tests │ ├── __init__.py │ ├── parse_tree_test.py │ └── expression_visitor_test.py └── queryfilter │ ├── __init__.py │ ├── antlr │ ├── __init__.py │ ├── CustomConnectorQueryFilterLexer.tokens │ ├── CustomConnectorQueryFilterParser.tokens │ └── CustomConnectorQueryFilterParser.interp │ ├── errors.py │ ├── parse_tree_builder.py │ └── grammar │ ├── CustomConnectorQueryFilterLexer.g4 │ └── CustomConnectorQueryFilterParser.g4 ├── requirements.txt ├── CODE_OF_CONDUCT.md ├── setup.py ├── Dockerfile ├── MANIFEST.in ├── LICENSE ├── custom_connector_tools ├── logFetcher.sh ├── deploy.sh └── README.md ├── CONTRIBUTING.md ├── TroubleShootingGuide.md └── MarketplaceIntegration.md /images/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | **/__pycache__ -------------------------------------------------------------------------------- /custom_connector_sdk/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_example/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_example/query/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_example/test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_integ_test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_sdk/connector/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_tests/invokers/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_example/handlers/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_example/integ_test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_integ_test/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_sdk/lambda_handler/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_sdk/marketplace/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_tests/exceptions/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_tests/validation/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_integ_test/appflow_test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_integ_test/configuration/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/antlr/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /custom_connector_example/test/resources/null_sobjects_server_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "sobjects": null 3 | } -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/salesforce-insert-file.csv: -------------------------------------------------------------------------------- 1 | Name,Description 2 | aaron,first account! -------------------------------------------------------------------------------- /custom_connector_sdk/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/custom_connector_sdk/.DS_Store -------------------------------------------------------------------------------- /images/appflow-console-home.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-home.png -------------------------------------------------------------------------------- /custom_connector_example/test/resources/describe_connector_configuration_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeConnectorConfigurationRequest" 3 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_connector_configuration_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeConnectorConfigurationRequest" 3 | } -------------------------------------------------------------------------------- /images/appflow-console-create-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-create-flow.png -------------------------------------------------------------------------------- /images/appflow-console-connector-label.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-connector-label.png -------------------------------------------------------------------------------- /images/appflow-console-connectors-page.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-connectors-page.png -------------------------------------------------------------------------------- /images/appflow-console-lambda-function.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-lambda-function.png -------------------------------------------------------------------------------- /images/appflow-console-test-connector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-test-connector.png -------------------------------------------------------------------------------- /images/appflow-console-create-connection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-create-connection.png -------------------------------------------------------------------------------- /images/appflow-console-connection-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-connection-details.png -------------------------------------------------------------------------------- /images/appflow-console-registered-connector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-registered-connector.png -------------------------------------------------------------------------------- /custom_connector_tests/exceptions/ValidationException.py: -------------------------------------------------------------------------------- 1 | class ValidationException(BaseException): 2 | def __init__(self, message): 3 | super().__init__(message) 4 | -------------------------------------------------------------------------------- /custom_connector_example/integ_test/sales_generator.py: -------------------------------------------------------------------------------- 1 | class SalesForceTestData: 2 | 3 | def generate_data(self): 4 | return "Name,Description\naaron,\"first account!\"" -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_connector_configuration_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeConnectorConfigurationRequestRequired", 3 | "locale": "en-US" 4 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_connector_runtime_settings_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateConnectorRuntimeSettingsRequest", 3 | "scope": "DESTINATION" 4 | } -------------------------------------------------------------------------------- /images/appflow-console-custom-connectors-dropdown.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-custom-connectors-dropdown.png -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/retrieve_data_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataResponse", 3 | "isSuccess": false, 4 | "errorDetails": null, 5 | "records": null 6 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_credentials_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsResponse", 3 | "isSuccess": false, 4 | "errorDetails": null 5 | } 6 | -------------------------------------------------------------------------------- /images/appflow-console-custom-connectors-view-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/aws-appflow-custom-connector-python/HEAD/images/appflow-console-custom-connectors-view-details.png -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/write_data_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataResponse", 3 | "isSuccess": false, 4 | "errorDetails": null, 5 | "writeRecordResults": null 6 | } 7 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | antlr4-python3-runtime==4.9.2 2 | python-dateutil==2.8.2 3 | six==1.16.0 4 | urllib3==1.26.6 5 | awslambdaric==1.2.2 6 | setuptools~=65.6.3 7 | boto3~=1.20.47 8 | jsonschema==4.1.2 9 | -------------------------------------------------------------------------------- /custom_connector_example/integ_test/sales_test_case.py: -------------------------------------------------------------------------------- 1 | 2 | from custom_connector_integ_test.appflow_test.test_case import AppflowTestCase 3 | 4 | 5 | class SalesTest(AppflowTestCase): 6 | pass 7 | 8 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/query_data_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataResponse", 3 | "isSuccess": true, 4 | "errorDetails": null, 5 | "nextToken": null, 6 | "records": null 7 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_entity_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityResponse", 3 | "isSuccess": true, 4 | "errorDetails": null, 5 | "entityDefinition": null, 6 | "cacheControl": null 7 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_credentials_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsRequest", 3 | "credentials": { 4 | "secretArn": "TestSecretArn", 5 | "authenticationType": "ApiKey" 6 | } 7 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/get_sobjects_server_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "sobjects": [ 3 | { 4 | "label": "testLabel", 5 | "childRelationships": [ 6 | "relationship_a" 7 | ] 8 | } 9 | ] 10 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/list_entities_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesResponse", 3 | "isSuccess": true, 4 | "errorDetails": null, 5 | "entities": null, 6 | "nextToken": null, 7 | "cacheControl": null 8 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_connector_runtime_settings_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateConnectorRuntimeSettingsResponse", 3 | "isSuccess": false, 4 | "errorsByInputField": null, 5 | "errorDetails": null 6 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/validate_connector_runtime_settings_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateConnectorRuntimeSettingsRequest", 3 | "scope": "DESTINATION", 4 | "connectorRuntimeSettings": { 5 | "testSettingKey": "testSettingValue" 6 | } 7 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/validate_connector_runtime_settings_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateConnectorRuntimeSettingsRequest", 3 | "scope": "DESTINATION", 4 | "connectorRuntimeSettings": { 5 | "instanceUrl": "https://salesforce.amazon.com" 6 | } 7 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_credentials_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsResponse", 3 | "isSuccess": true, 4 | "errorDetails": { 5 | "errorCode": "ServerError", 6 | "errorMessage": "this is an error", 7 | "retryAfterSeconds": 20 8 | } 9 | } -------------------------------------------------------------------------------- /custom_connector_example/constants.py: -------------------------------------------------------------------------------- 1 | from custom_connector_sdk.connector.fields import WriteOperationType 2 | 3 | INSTANCE_URL_KEY = 'instanceUrl' 4 | SUPPORTED_WRITE_OPERATIONS = [WriteOperationType.INSERT, WriteOperationType.UPDATE, WriteOperationType.UPSERT] 5 | IS_SANDBOX_ACCOUNT = 'IsSandboxAccount' 6 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_connector_runtime_settings_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateConnectorRuntimeSettingsRequest", 3 | "scope": "DESTINATION", 4 | "connectorRuntimeSettings": { 5 | "testSettingKey": "testSettingValue", 6 | "testSettingKey2": "testSettingValue2" 7 | } 8 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/list_entities_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesRequest", 3 | "maxResult": null, 4 | "connectorContext": { 5 | "credentials": { 6 | "secretArn": "TestSecretArn", 7 | "authenticationType": "BasicAuth" 8 | }, 9 | "apiVersion": "v47.0" 10 | } 11 | } -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/write_data_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 7 | }, 8 | "apiVersion": "v47.0" 9 | } 10 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/validate_credentials_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsRequest", 3 | "credentials": { 4 | "oAuth2Credentials": { 5 | "accessToken": "testToken" 6 | } 7 | }, 8 | "connectorRuntimeSettings": { 9 | "instanceUrl": "https://amazon.salesforce.com" 10 | } 11 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/retrieve_data_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 7 | }, 8 | "apiVersion": "v47.0" 9 | } 10 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_entity_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityRequest", 3 | "entityIdentifier": "EmailMessage", 4 | "connectorContext": { 5 | "credentials": { 6 | "secretArn": "TestSecretArn", 7 | "authenticationType": "BasicAuth" 8 | }, 9 | "apiVersion": "v47.0" 10 | } 11 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/list_entities_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesRequest", 3 | "maxResult": null, 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "http://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | }, 10 | "apiVersion": "v47.0" 11 | } 12 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/retrieve_data_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataResponse", 3 | "isSuccess": false, 4 | "errorDetails": { 5 | "errorCode": "ClientError", 6 | "errorMessage": "test error", 7 | "retryAfterSeconds": 120 8 | }, 9 | "records": [ 10 | "recordA", 11 | "recordB" 12 | ] 13 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/describe_entity_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityRequest", 3 | "entityIdentifier": "testIdentifier", 4 | "connectorContext": { 5 | "credentials": { 6 | "oAuth2Credentials": { 7 | "accessToken": "testToken" 8 | } 9 | }, 10 | "apiVersion": "v47.0" 11 | } 12 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/validate_credentials_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsRequest", 3 | "credentials": { 4 | "BasicAuthCredentials": { 5 | "username": "username", 6 | "password": "password" 7 | } 8 | }, 9 | "connectorRuntimeSettings": { 10 | "instanceUrl": "https://amazon.salesforce.com" 11 | } 12 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/query_data_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataResponse", 3 | "isSuccess": false, 4 | "errorDetails": { 5 | "errorCode": "ClientError", 6 | "errorMessage": "test error", 7 | "retryAfterSeconds": 120 8 | }, 9 | "nextToken": "testToken", 10 | "records": [ 11 | "recordA", 12 | "recordB" 13 | ] 14 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_credentials_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsRequest", 3 | "credentials": { 4 | "secretArn": "TestSecretArn", 5 | "authenticationType": "CustomAuth" 6 | }, 7 | "connectorRuntimeSettings": { 8 | "testSettingKey": "testSettingValue", 9 | "testSettingKey2": "testSettingValue2" 10 | } 11 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_connector_runtime_settings_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateConnectorRuntimeSettingsResponse", 3 | "isSuccess": true, 4 | "errorsByInputField": { 5 | "field": "error" 6 | }, 7 | "errorDetails": { 8 | "errorCode": "ServerError", 9 | "errorMessage": "this is an error", 10 | "retryAfterSeconds": 20 11 | } 12 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/write_data_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataResponse", 3 | "isSuccess": false, 4 | "errorDetails": { 5 | "errorCode": "ClientError", 6 | "errorMessage": "test error", 7 | "retryAfterSeconds": 120 8 | }, 9 | "writeRecordResults": { 10 | "isSuccess": false, 11 | "recordId": "testId", 12 | "errorMessage": "write error" 13 | } 14 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_entity_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityResponse", 3 | "isSuccess": true, 4 | "errorDetails": { 5 | "errorCode": "ServerError", 6 | "errorMessage": "this is an error", 7 | "retryAfterSeconds": 20 8 | }, 9 | "entityDefinition": null, 10 | "cacheControl": { 11 | "timeToLive": 60, 12 | "timeToLiveUnit": "SECONDS" 13 | } 14 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/list_entities_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesResponse", 3 | "isSuccess": true, 4 | "errorDetails": { 5 | "errorCode": "ServerError", 6 | "errorMessage": "this is an error", 7 | "retryAfterSeconds": 20 8 | }, 9 | "entities": null, 10 | "nextToken": "2", 11 | "cacheControl": { 12 | "timeToLive": 60, 13 | "timeToLiveUnit": "SECONDS" 14 | } 15 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/list_entities_request_no_path.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesRequest", 3 | "maxResult": null, 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0" 14 | } 15 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/query_data_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 7 | }, 8 | "credentials": { 9 | "secretArn": "TestSecretArn", 10 | "authenticationType": "BasicAuth" 11 | }, 12 | "apiVersion": "v47.0" 13 | } 14 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/write_data_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 7 | }, 8 | "credentials": { 9 | "secretArn": "TestSecretArn", 10 | "authenticationType": "OAuth2" 11 | }, 12 | "apiVersion": "v47.0" 13 | } 14 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/describe_entity_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityRequest", 3 | "entityIdentifier": "testIdentifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "http://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0" 14 | } 15 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/retrieve_data_request_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 7 | }, 8 | "credentials": { 9 | "secretArn": "TestSecretArn", 10 | "authenticationType": "OAuth2" 11 | }, 12 | "apiVersion": "v47.0" 13 | } 14 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/retrieve_data_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | }, 10 | "apiVersion": "v47.0" 11 | }, 12 | "selectedFieldNames": [ 13 | "field_a", 14 | "field_b", 15 | "field_c" 16 | ] 17 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_entity_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityRequest", 3 | "connectorContext": { 4 | "credentials": { 5 | "basicAuthCredentials": { 6 | "userName": "testUsername", 7 | "password": "testPassword" 8 | }, 9 | "apiKeyCredentials": null, 10 | "oAuth2Credentials": null, 11 | "customAuthCredentials": null 12 | }, 13 | "apiVersion": "v47.0" 14 | } 15 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/list_entities_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesRequest", 3 | "maxResult": null, 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "http://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "entitiesPath": "path/to/entities", 14 | "apiVersion": "v47.0" 15 | } 16 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/query_data_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "credentials": { 6 | "oAuth2Credentials": { 7 | "accessToken": "testToken" 8 | } 9 | }, 10 | "apiVersion": "v47.0" 11 | }, 12 | "selectedFieldNames": [ 13 | "field_a", 14 | "field_b", 15 | "field_c" 16 | ], 17 | "filter_expression": "SELECT * FROM table" 18 | } -------------------------------------------------------------------------------- /custom_connector_integ_test/base-test-config.json: -------------------------------------------------------------------------------- 1 | { 2 | "resourcePrefix": "string", 3 | "customConnectorConfigurations": [], 4 | "customConnectorProfileConfigurations": [], 5 | "testBucketConfiguration": 6 | { 7 | "bucketName":"string", 8 | "bucketPrefix":"string" 9 | }, 10 | "listConnectorEntitiesTestConfigurations": [], 11 | "describeConnectorEntityTestConfigurations":[], 12 | "onDemandFromS3TestConfigurations": [], 13 | "onDemandToS3TestConfigurations": [] 14 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/validate_credentials_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ValidateCredentialsRequest", 3 | "credentials": { 4 | "basicAuthCredentials": null, 5 | "apiKeyCredentials": null, 6 | "oAuth2Credentials": null, 7 | "customAuthCredentials": { 8 | "authenticationType": "testCustomAuth" 9 | } 10 | }, 11 | "connectorRuntimeSettings": { 12 | "testSettingKey": "testSettingValue", 13 | "testSettingKey2": "testSettingValue2" 14 | } 15 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/write_data_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | }, 7 | "credentials": { 8 | "oAuth2Credentials": { 9 | "accessToken": "testToken" 10 | } 11 | }, 12 | "apiVersion": "v47.0" 13 | }, 14 | "operation": "INSERT", 15 | "records": [ 16 | "record_a", 17 | "record_b", 18 | "record_c" 19 | ] 20 | } -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | setup(name='custom_connector_sdk', 4 | version='1.0.5', 5 | description='Amazon AppFlow Custom Connector SDK', 6 | url='https://github.com/awslabs/aws-appflow-custom-connector-python', 7 | author='Amazon AppFlow', 8 | license="Apache License 2.0", 9 | python_requires=">= 3.6", 10 | packages=find_packages(include=['custom*']), 11 | include_package_data=True, 12 | zip_safe=False 13 | ) 14 | -------------------------------------------------------------------------------- /custom_connector_example/test/resources/write_data_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0" 14 | }, 15 | "operation": "INSERT", 16 | "records": [ 17 | "record_a", 18 | "record_b", 19 | "record_c" 20 | ] 21 | } -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/antlr/CustomConnectorQueryFilterLexer.tokens: -------------------------------------------------------------------------------- 1 | AND=1 2 | OR=2 3 | NOT=3 4 | TRUE=4 5 | FALSE=5 6 | GT=6 7 | GE=7 8 | LT=8 9 | LE=9 10 | EQ=10 11 | NE=11 12 | LIKE=12 13 | BETWEEN=13 14 | LPAREN=14 15 | RPAREN=15 16 | NULL=16 17 | IN=17 18 | LIMIT=18 19 | COMMA=19 20 | IDENTIFIER=20 21 | POS_INTEGER=21 22 | DECIMAL=22 23 | SINGLE_STRING=23 24 | DOUBLE_STRING=24 25 | EMPTY_SINGLE_STRING=25 26 | EMPTY_DOUBLE_STRING=26 27 | WS=27 28 | DATE=28 29 | DATETIME=29 30 | '>'=6 31 | '>='=7 32 | '<'=8 33 | '<='=9 34 | '='=10 35 | '!='=11 36 | '('=14 37 | ')'=15 38 | 'null'=16 39 | ','=19 40 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/antlr/CustomConnectorQueryFilterParser.tokens: -------------------------------------------------------------------------------- 1 | AND=1 2 | OR=2 3 | NOT=3 4 | TRUE=4 5 | FALSE=5 6 | GT=6 7 | GE=7 8 | LT=8 9 | LE=9 10 | EQ=10 11 | NE=11 12 | LIKE=12 13 | BETWEEN=13 14 | LPAREN=14 15 | RPAREN=15 16 | NULL=16 17 | IN=17 18 | LIMIT=18 19 | COMMA=19 20 | IDENTIFIER=20 21 | POS_INTEGER=21 22 | DECIMAL=22 23 | SINGLE_STRING=23 24 | DOUBLE_STRING=24 25 | EMPTY_SINGLE_STRING=25 26 | EMPTY_DOUBLE_STRING=26 27 | WS=27 28 | DATE=28 29 | DATETIME=29 30 | '>'=6 31 | '>='=7 32 | '<'=8 33 | '<='=9 34 | '='=10 35 | '!='=11 36 | '('=14 37 | ')'=15 38 | 'null'=16 39 | ','=19 40 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM public.ecr.aws/lambda/python:3.8 2 | 3 | # Copy function code 4 | COPY custom_connector_example ${LAMBDA_TASK_ROOT} 5 | COPY custom_connector_sdk ${LAMBDA_TASK_ROOT} 6 | COPY custom_connector_queryfilter ${LAMBDA_TASK_ROOT} 7 | 8 | # Install the function's dependencies using file requirements.txt 9 | # from your project folder. 10 | COPY requirements.txt . 11 | RUN pip3 install -r ./requirements.txt --target "${LAMBDA_TASK_ROOT}" 12 | 13 | # Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile) 14 | CMD [ "custom_connector_example.handlers.lambda_handler.salesforce_lambda_handler" ] -------------------------------------------------------------------------------- /custom_connector_integ_test/utils/s3_helper.py: -------------------------------------------------------------------------------- 1 | from custom_connector_integ_test.configuration import services 2 | from custom_connector_integ_test.configuration.test_configuration import TestBucketConfiguration 3 | 4 | 5 | class S3Helper: 6 | def __init__(self, 7 | bucket_config: TestBucketConfiguration): 8 | self.bucket_config = bucket_config 9 | 10 | def upload_file(self, file: str, file_name: str): 11 | services.get_s3().put_object(Bucket=self.bucket_config.bucket_name, 12 | Key=self.bucket_config.bucket_prefix + file_name, 13 | Body=str.encode(file)) 14 | -------------------------------------------------------------------------------- /custom_connector_integ_test/configuration/services.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | 3 | # File with amazon service singletons 4 | 5 | SECRETS_MANAGER = None 6 | 7 | APP_FLOW = None 8 | 9 | S3 = None 10 | 11 | 12 | def get_appflow(): 13 | global APP_FLOW 14 | if APP_FLOW is None: 15 | APP_FLOW = boto3.client('appflow') 16 | return APP_FLOW 17 | 18 | 19 | def get_s3(): 20 | global S3 21 | if S3 is None: 22 | S3 = boto3.client('s3') 23 | return S3 24 | 25 | 26 | def get_secrets_manager(): 27 | global SECRETS_MANAGER 28 | if SECRETS_MANAGER is None: 29 | SECRETS_MANAGER = boto3.client('secretsmanager') 30 | return SECRETS_MANAGER 31 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft custom_connector_example 2 | graft custom_connector_example/test/resources 3 | graft custom_connector_example/salesforce-example-test-files 4 | graft custom_connector_integ_test 5 | graft custom_connector_queryfilter 6 | graft custom_connector_queryfilter/grammar 7 | graft custom_connector_sdk/test/resources 8 | graft custom_connector_tests 9 | graft custom_connector_tests/configuration 10 | graft custom_connector_tools 11 | graft images 12 | include CODE_OF_CONDUCT.md 13 | include CONTRIBUTING.md 14 | include Dockerfile 15 | include LICENSE 16 | include MarketplaceIntegration.md 17 | include README.md 18 | include requirements.txt 19 | include ThirdPartyAttributions.txt 20 | include TroubleShootingGuide.md 21 | -------------------------------------------------------------------------------- /custom_connector_example/handlers/lambda_handler.py: -------------------------------------------------------------------------------- 1 | from custom_connector_example.handlers.record import SalesforceRecordHandler 2 | from custom_connector_example.handlers.metadata import SalesforceMetadataHandler 3 | from custom_connector_example.handlers.configuration import SalesforceConfigurationHandler 4 | from custom_connector_sdk.lambda_handler.lambda_handler import BaseLambdaConnectorHandler 5 | 6 | class SalesforceLambdaHandler(BaseLambdaConnectorHandler): 7 | def __init__(self): 8 | super().__init__(SalesforceMetadataHandler(), SalesforceRecordHandler(), SalesforceConfigurationHandler()) 9 | 10 | def salesforce_lambda_handler(event, context): 11 | """Lambda entry point.""" 12 | return SalesforceLambdaHandler().lambda_handler(event, context) 13 | -------------------------------------------------------------------------------- /custom_connector_example/test/resources/describe_sobject_server_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "fields": [ 3 | { 4 | "type": "string", 5 | "name": "testField", 6 | "unique": true, 7 | "label": "testLabel", 8 | "filterable": true, 9 | "idLookup": "testIdLookup", 10 | "createable": true, 11 | "updateable": false, 12 | "nillable": true, 13 | "defaultedOnCreate": false 14 | }, 15 | { 16 | "type": "struct", 17 | "name": "testField2", 18 | "unique": true, 19 | "label": "testLabel", 20 | "filterable": true, 21 | "externalId": "testId", 22 | "createable": true, 23 | "updateable": true, 24 | "nillable": false, 25 | "defaultedOnCreate": false 26 | } 27 | ] 28 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/query_data_request_no_selected_fields.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0", 14 | "entityDefinition": { 15 | "entity": { 16 | "entityIdentifier": "identifier", 17 | "hasNestedEntities": false, 18 | "label": "testLabel", 19 | "description": "testDescription" 20 | }, 21 | "fields": [ 22 | { 23 | "fieldName": "testField", 24 | "dataType": "String", 25 | "dataTypeLabel": "String" 26 | } 27 | ] 28 | } 29 | }, 30 | "filterExpression": "testField = 'abc'" 31 | } -------------------------------------------------------------------------------- /custom_connector_integ_test/utils/flow_poller.py: -------------------------------------------------------------------------------- 1 | from custom_connector_integ_test.configuration import services 2 | import time 3 | 4 | # Use this method to poll flow executions. 5 | def poll_for_execution_records_response(flow_name: str, execution_id: str, max_poll_time: int, time_between_polls: int): 6 | appflow = services.get_appflow() 7 | execution_record = None 8 | total_time = 0 9 | while True: 10 | time.sleep(time_between_polls) 11 | res = appflow.describe_flow_execution_records(flowName=flow_name) 12 | for record in res["flowExecutions"]: 13 | if record["executionId"] == execution_id: 14 | execution_record = record 15 | break 16 | total_time += time_between_polls 17 | if total_time > max_poll_time or (execution_record is not None and execution_record["executionStatus"] != "InProgress"): 18 | return execution_record 19 | -------------------------------------------------------------------------------- /custom_connector_example/test/resources/query_data_request_no_filter_expression.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0", 14 | "entityDefinition": { 15 | "entity": { 16 | "entityIdentifier": "identifier", 17 | "hasNestedEntities": false, 18 | "label": "testLabel", 19 | "description": "testDescription" 20 | }, 21 | "fields": [ 22 | { 23 | "fieldName": "testField", 24 | "dataType": "String", 25 | "dataTypeLabel": "String" 26 | } 27 | ] 28 | } 29 | }, 30 | "selectedFieldNames": [ 31 | "testField" 32 | ] 33 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/query_data_request_filter_expression.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0", 14 | "entityDefinition": { 15 | "entity": { 16 | "entityIdentifier": "identifier", 17 | "hasNestedEntities": false, 18 | "label": "testLabel", 19 | "description": "testDescription" 20 | }, 21 | "fields": [ 22 | { 23 | "fieldName": "testField", 24 | "dataType": "String", 25 | "dataTypeLabel": "String" 26 | } 27 | ] 28 | } 29 | }, 30 | "selectedFieldNames": [ 31 | "testField" 32 | ], 33 | "filterExpression": "testField = 'abc'" 34 | } -------------------------------------------------------------------------------- /custom_connector_example/test/resources/retrieve_data_request_valid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://salesforce.amazon.com" 7 | }, 8 | "credentials": { 9 | "oAuth2Credentials": { 10 | "accessToken": "testToken" 11 | } 12 | }, 13 | "apiVersion": "v47.0", 14 | "entityDefinition": { 15 | "entity": { 16 | "entityIdentifier": "identifier", 17 | "hasNestedEntities": false, 18 | "label": "testLabel", 19 | "description": "testDescription" 20 | }, 21 | "fields": [ 22 | { 23 | "fieldName": "testField", 24 | "dataType": "String", 25 | "dataTypeLabel": "String" 26 | } 27 | ] 28 | } 29 | }, 30 | "selectedFieldNames": [ 31 | "testField" 32 | ], 33 | "idFieldName": "testField", 34 | "ids": ["'id1'", "id2"] 35 | } -------------------------------------------------------------------------------- /custom_connector_tests/invokers/test_invoker.py: -------------------------------------------------------------------------------- 1 | import abc 2 | 3 | 4 | class ConnectorTestInvoker(metaclass=abc.ABCMeta): 5 | """ 6 | This interface defines the functionality that is required for testing custom connector implementation. 7 | The tests must verify behavior of all three handlers along with some implementation specific validations. 8 | """ 9 | 10 | def invoke_configuration_handler_tests(self): 11 | """ 12 | Tests for the ConfigurationHandler. This includes verifying behaviour of validate_credentials, 13 | validate_connector_runtime_settings and describe_connector_configuration. 14 | """ 15 | pass 16 | 17 | def invoke_metadata_handler_tests(self): 18 | """ 19 | Tests for the MetadataHandler. This includes verifying behaviour of list_entities and describe_entity. 20 | """ 21 | pass 22 | 23 | def invoke_record_handler_tests(self): 24 | """ 25 | Tests for the RecordHandler. This includes verifying behaviour of retrieve_data, write_data and query_data. 26 | """ 27 | pass 28 | -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/describe-connector-entity-validation-file.json: -------------------------------------------------------------------------------- 1 | { 2 | "connectorEntityFields": [ 3 | { 4 | "identifier": "Id", 5 | "label": "Account ID", 6 | "isPrimaryKey": false, 7 | "isDeprecated": false, 8 | "supportedFieldTypeDetails": { 9 | "v1": { 10 | "fieldType": "String", 11 | "filterOperators": [ 12 | "CONTAINS", 13 | "EQUAL_TO", 14 | "NOT_EQUAL_TO" 15 | ], 16 | "supportedValues": [ 17 | ] 18 | } 19 | }, 20 | "description": "Account ID", 21 | "sourceProperties": { 22 | "isRetrievable": true, 23 | "isQueryable": true, 24 | "isTimestampFieldForIncrementalQueries": false 25 | }, 26 | "destinationProperties": { 27 | "isCreatable": false, 28 | "isNullable": false, 29 | "isUpsertable": false, 30 | "isUpdatable": false, 31 | "isDefaultedOnCreate": true, 32 | "supportedWriteOperations": [ 33 | "UPSERT", 34 | "UPDATE", 35 | "INSERT" 36 | ] 37 | } 38 | } 39 | ] 40 | } -------------------------------------------------------------------------------- /custom_connector_sdk/marketplace/entititlement_util.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import boto3 3 | import uuid 4 | 5 | LOGGER = logging.getLogger() 6 | LOGGER.setLevel(logging.INFO) 7 | 8 | THE_KEY_FINGERPRINT = 'aws:294406891311:AWS/Marketplace:issuer-fingerprint' 9 | CHECKOUT_TYPE = "PROVISIONAL" 10 | MARKETPLACE_ENTITLEMENT_NAME = 'AWS::Marketplace::Usage' 11 | 12 | 13 | def check_entitlement(product_sku: str) -> bool: 14 | """Checks if the Connector subscribed from Marketplace has entitlement to use or not for an AWS account.""" 15 | client = boto3.client('license-manager') 16 | try: 17 | res = client.checkout_license( 18 | ProductSKU=product_sku, 19 | CheckoutType=CHECKOUT_TYPE, 20 | KeyFingerprint=THE_KEY_FINGERPRINT, 21 | Entitlements=[{'Name': MARKETPLACE_ENTITLEMENT_NAME, 'Unit': 'None'}], 22 | ClientToken=str(uuid.uuid4()) 23 | ) 24 | for entitlement in res['EntitlementsAllowed']: 25 | if MARKETPLACE_ENTITLEMENT_NAME == entitlement['Name']: 26 | return True 27 | except Exception as e: 28 | LOGGER.error('Entitlement check failed with exception ' + str(e)) 29 | return False 30 | -------------------------------------------------------------------------------- /custom_connector_tests/run_tests.py: -------------------------------------------------------------------------------- 1 | import json 2 | import sys 3 | import importlib 4 | from custom_connector_tests.invokers.base_test_invoker import BaseConnectorTestInvoker 5 | from jsonschema import validate 6 | 7 | def main(): 8 | module_name = sys.argv[1] 9 | class_name = sys.argv[2] 10 | config_file_path = sys.argv[3] 11 | 12 | module = importlib.import_module(module_name) 13 | class_ = getattr(module, class_name) 14 | connector_handler = class_() 15 | 16 | try: 17 | test_config = json.load(open(config_file_path)) 18 | except ValueError as e: 19 | print("Error while parsing Configuration File. Invalid JSON. " + str(e)) 20 | sys.exit(1) 21 | 22 | print("Validating the schema") 23 | schema = json.load(open("./custom_connector_tests/configuration/ConfigSchema.json")) 24 | validate(test_config, schema) 25 | test_invoker = BaseConnectorTestInvoker(connector_handler, test_config) 26 | 27 | test_invoker.invoke_configuration_handler_tests() 28 | test_invoker.invoke_metadata_handler_tests() 29 | test_invoker.invoke_record_handler_tests() 30 | 31 | print("Tests completed successfully, see above logs for results") 32 | 33 | if __name__ == '__main__': 34 | main() 35 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Amazon AppFlow Custom Connector Software Developer Kit License Agreement 2 | Copyright 2022 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. 3 | 4 | By installing or using this software development kit ("Software"), you agree to 5 | the AWS Customer Agreement (https://aws.amazon.com/agreement) or other agreement 6 | with Amazon Web Services, Inc. ("AWS") governing your use of services provided by 7 | AWS, including the AWS Service Terms (https://aws.amazon.com/service-terms). 8 | The Software is AWS Content, and you may install and use the Software solely for 9 | the purpose of use with Amazon AppFlow in accordance with the Documentation. 10 | 11 | Certain components of the Software contain third party software programs which 12 | are governed by separate licenses, including but not limited to open source 13 | software licenses, identified in the included "THIRD PARTY ATTRIBUTIONS" file. 14 | Your rights and obligations with respect to these components are defined by the 15 | applicable software license, and nothing in this agreement will restrict, limit, 16 | or otherwise affect your rights or obligations under such software licenses. 17 | 18 | 19 | To learn more about the Amazon AppFlow Custom Connector Software Developer Kit see: 20 | https://docs.aws.amazon.com/appflow/ 21 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/errors.py: -------------------------------------------------------------------------------- 1 | from antlr4.error.ErrorListener import ErrorListener 2 | from antlr4.error.Errors import RecognitionException 3 | from antlr4.Recognizer import Recognizer 4 | from antlr4.Parser import Parser 5 | 6 | from io import StringIO 7 | 8 | class InvalidFilterExpressionError(Exception): 9 | """This error is raised when an invalid filter expression is given as input to the query expression parser.""" 10 | pass 11 | 12 | class SyntaxErrorReporter(ErrorListener): 13 | """This class is responsible for collecting and reporting syntax errors in passed filter expression in the input 14 | request. 15 | Note: This class is not thread safe. 16 | 17 | """ 18 | def __init__(self): 19 | self.has_error = False 20 | self.syntax_errors = StringIO() 21 | 22 | def syntaxError(self, 23 | parser: Parser, 24 | offending_symbol: object, 25 | line: int, 26 | char_position_in_line: int, 27 | msg: str, 28 | e: RecognitionException): 29 | self.has_error = True 30 | stack = reversed(parser.getRuleInvocationStack()) 31 | error = f'rule stack: {stack} line {line}:{char_position_in_line} at {offending_symbol}:{msg}' 32 | self.syntax_errors.write(error) 33 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/parse_tree_builder.py: -------------------------------------------------------------------------------- 1 | from antlr4.tree.Tree import ParseTree 2 | from antlr4 import CommonTokenStream, InputStream 3 | from custom_connector_queryfilter.queryfilter.antlr.CustomConnectorQueryFilterLexer import CustomConnectorQueryFilterLexer 4 | from custom_connector_queryfilter.queryfilter.antlr.CustomConnectorQueryFilterParser import CustomConnectorQueryFilterParser 5 | from custom_connector_queryfilter.queryfilter.errors import SyntaxErrorReporter, InvalidFilterExpressionError 6 | 7 | def parse(filter_expression: str) -> ParseTree: 8 | """Helper function to validate and construct a parse tree for a given filter expression.""" 9 | assert filter_expression is not None, "Filter expression cannot be empty" 10 | 11 | input_stream = InputStream(filter_expression) 12 | lexer = CustomConnectorQueryFilterLexer(input_stream) 13 | common_token_stream = CommonTokenStream(lexer) 14 | parser = CustomConnectorQueryFilterParser(common_token_stream) 15 | 16 | parser.removeErrorListeners() # Remove any pre-existing error listeners and register custom listeners 17 | syntax_error_reporter = SyntaxErrorReporter() 18 | parser.addErrorListener(syntax_error_reporter) 19 | 20 | tree = parser.queryfilter() 21 | if syntax_error_reporter.has_error: 22 | raise InvalidFilterExpressionError('Filter expression has the following syntax errors :', 23 | syntax_error_reporter.syntax_errors.getvalue()) 24 | return tree 25 | -------------------------------------------------------------------------------- /custom_connector_example/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: 'AWS::Serverless-2016-10-31' 3 | Description: Template to deploy the lambda connector in your account. 4 | Resources: 5 | Function: 6 | Type: AWS::Serverless::Function 7 | Properties: 8 | Handler: custom_connector_example.handlers.lambda_handler.salesforce_lambda_handler 9 | Runtime: python3.8 10 | CodeUri: ../. 11 | Description: "Example for writing and deploying your AppFlow connector" 12 | Timeout: 30 13 | MemorySize: 256 14 | Policies: 15 | Version: '2012-10-17' 16 | Statement: 17 | Effect: Allow 18 | Action: 'secretsmanager:GetSecretValue' 19 | Resource: !Sub 'arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:appflow!${AWS::AccountId}--*' 20 | Layers: 21 | - !Ref Dependencies 22 | Dependencies: 23 | Type: AWS::Serverless::LayerVersion 24 | Properties: 25 | LayerName: custom-connector-dependency-lib 26 | Description: Dependencies for the sample custom connector 27 | ContentUri: ../../package/. 28 | CompatibleRuntimes: 29 | - python3.8 30 | PolicyPermission: 31 | Type: 'AWS::Lambda::Permission' 32 | Properties: 33 | FunctionName: !GetAtt Function.Arn 34 | Action: lambda:InvokeFunction 35 | Principal: 'appflow.amazonaws.com' 36 | SourceAccount: !Ref 'AWS::AccountId' 37 | SourceArn: !Sub 'arn:aws:appflow:${AWS::Region}:${AWS::AccountId}:*' -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_connector_configuration_response_required.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeConnectorConfigurationResponse", 3 | "connectorOwner": "owner", 4 | "connectorName": "name", 5 | "connectorVersion": "v1.0", 6 | "connectorModes": ["SOURCE"], 7 | "authenticationConfig": { 8 | "isBasicAuthSupported": true, 9 | "isApiKeyAuthSupported": null, 10 | "isOAuth2Supported": null, 11 | "isCustomAuthSupported": null, 12 | "oAuth2Defaults": null, 13 | "customAuthConfig": null 14 | }, 15 | "supportedApiVersions": ["v1.0", "v1.1"], 16 | "isSuccess": true, 17 | "connectorRuntimeSetting": null, 18 | "logoURL": null, 19 | "errorDetails": null, 20 | "operatorsSupported": [ 21 | "PROJECTION", 22 | "LESS_THAN", 23 | "GREATER_THAN", 24 | "BETWEEN", 25 | "LESS_THAN_OR_EQUAL_TO", 26 | "GREATER_THAN_OR_EQUAL_TO", 27 | "EQUAL_TO", 28 | "CONTAINS", 29 | "NOT_EQUAL_TO", 30 | "ADDITION", 31 | "SUBTRACTION", 32 | "MULTIPLICATION", 33 | "DIVISION", 34 | "MASK_ALL", 35 | "MASK_FIRST_N", 36 | "MASK_LAST_N", 37 | "VALIDATE_NON_NULL", 38 | "VALIDATE_NON_ZERO", 39 | "VALIDATE_NON_NEGATIVE", 40 | "VALIDATE_NUMERIC", 41 | "NO_OP" 42 | ], 43 | "triggerFrequenciesSupported": [ 44 | "Minutely", 45 | "Hourly", 46 | "Daily", 47 | "Weekly", 48 | "Monthly", 49 | "Once" 50 | ], 51 | "supportedWriteOperations": [ 52 | "INSERT", 53 | "UPDATE", 54 | "UPSERT", 55 | "DELETE" 56 | ] 57 | } 58 | -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/test-file-3.json: -------------------------------------------------------------------------------- 1 | { 2 | "customConnectorConfigurations": [ 3 | { 4 | "name": "connector1", 5 | "lambdaArn":"arn:aws:lambda:us-west-2:***********:function:custom-connector-Function-FpcRtBXSqTWy", 6 | "validationFileName": "custom_connector_example/salesforce-example-test-files/describe-connector-validation-file.json" 7 | } 8 | ], 9 | "customConnectorProfileConfigurations": [ 10 | { 11 | "connectorName": "connector1", 12 | "name": "profile1", 13 | "profileProperties": { 14 | "api_version": "v51.0", 15 | "instanceUrl": "https://***********.my.salesforce.com" 16 | }, 17 | "defaultApiVersion": "v51.0", 18 | "authenticationType": "OAUTH2", 19 | "oAuth2Properties": { 20 | "oAuth2GrantType": "CLIENT_CREDENTIALS", 21 | "tokenUrl": "https://login.salesforce.com/services/oauth2/token" 22 | }, 23 | "secretsManagerArn": "arn:aws:secretsmanager:us-west-2:***********:secret:custom-connector-qrSqOc" 24 | } 25 | ], 26 | "testBucketConfiguration": { 27 | "bucketName": "cvs-beta", 28 | "bucketPrefix": "" 29 | }, 30 | "listConnectorEntitiesTestConfigurations": [ 31 | { 32 | "validationFileName": "custom_connector_example/salesforce-example-test-files/list-entities-validation-file.json" 33 | } 34 | ], 35 | "describeConnectorEntityTestConfigurations": [ 36 | { 37 | "validationFileName": "custom_connector_example/salesforce-example-test-files/describe-connector-entity-validation-file.json", 38 | "entityName" : "Account" 39 | } 40 | ], 41 | "onDemandFromS3TestConfigurations": [ 42 | ], 43 | "onDemandToS3TestConfigurations": [ 44 | ] 45 | } -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/list-entities-validation-file.json: -------------------------------------------------------------------------------- 1 | { 2 | "connectorEntityMap": { 3 | "Objects": [ 4 | { 5 | "name": "AIApplication", 6 | "label": "AI Application", 7 | "hasNestedEntities": false 8 | }, 9 | { 10 | "name": "AIApplicationConfig", 11 | "label": "AI Application config", 12 | "hasNestedEntities": false 13 | }, 14 | { 15 | "name": "AIInsightAction", 16 | "label": "AI Insight Action", 17 | "hasNestedEntities": false 18 | }, 19 | { 20 | "name": "AIInsightFeedback", 21 | "label": "AI Insight Feedback", 22 | "hasNestedEntities": false 23 | }, 24 | { 25 | "name": "AIInsightReason", 26 | "label": "AI Insight Reason", 27 | "hasNestedEntities": false 28 | }, 29 | { 30 | "name": "AIInsightValue", 31 | "label": "AI Insight Value", 32 | "hasNestedEntities": false 33 | }, 34 | { 35 | "name": "AIPredictionEvent", 36 | "label": "AI Prediction Event", 37 | "hasNestedEntities": false 38 | }, 39 | { 40 | "name": "AIRecordInsight", 41 | "label": "AI Record Insight", 42 | "hasNestedEntities": false 43 | }, 44 | { 45 | "name": "AcceptedEventRelation", 46 | "label": "Accepted Event Relation", 47 | "hasNestedEntities": false 48 | }, 49 | { 50 | "name": "Account", 51 | "label": "Account", 52 | "hasNestedEntities": false 53 | }, 54 | { 55 | "name": "AccountChangeEvent", 56 | "label": "Account Change Event", 57 | "hasNestedEntities": false 58 | } 59 | ] 60 | } 61 | } -------------------------------------------------------------------------------- /custom_connector_sdk/connector/configuration.py: -------------------------------------------------------------------------------- 1 | from enum import Enum, auto 2 | 3 | class ConnectorModes(Enum): 4 | """Enumerates the supported connector Modes. Used by Connectors to declare the modes of operation a custom 5 | connector supports. 6 | 7 | """ 8 | SOURCE = auto() 9 | DESTINATION = auto() 10 | 11 | class ConnectorOperator(Enum): 12 | """Enumerates the set of operations that are allowed for constructing filter criteria against specific entity 13 | fields. 14 | 15 | """ 16 | # TODO: Need to add description for each of the following 17 | 18 | # Column Filter Operator 19 | PROJECTION = auto() 20 | 21 | # Row Filter Operators 22 | LESS_THAN = auto() 23 | GREATER_THAN = auto() 24 | BETWEEN = auto() 25 | LESS_THAN_OR_EQUAL_TO = auto() 26 | GREATER_THAN_OR_EQUAL_TO = auto() 27 | EQUAL_TO = auto() 28 | CONTAINS = auto() 29 | NOT_EQUAL_TO = auto() 30 | 31 | # Operators with a Destination Field 32 | ADDITION = auto() 33 | SUBTRACTION = auto() 34 | MULTIPLICATION = auto() 35 | DIVISION = auto() 36 | 37 | # Masking related operators 38 | MASK_ALL = auto() 39 | MASK_FIRST_N = auto() 40 | MASK_LAST_N = auto() 41 | 42 | # Validation specific operators 43 | VALIDATE_NON_NULL = auto() 44 | VALIDATE_NON_ZERO = auto() 45 | VALIDATE_NON_NEGATIVE = auto() 46 | VALIDATE_NUMERIC = auto() 47 | 48 | NO_OP = auto() 49 | 50 | class TriggerFrequency(Enum): 51 | """Enum for flow trigger frequency.""" 52 | BYMINUTE = auto() 53 | HOURLY = auto() 54 | DAILY = auto() 55 | WEEKLY = auto() 56 | MONTHLY = auto() 57 | ONCE = auto() 58 | 59 | class TriggerType(Enum): 60 | """Enum for flow trigger type.""" 61 | SCHEDULED = auto() 62 | ONDEMAND = auto() 63 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_connector_configuration_response_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeConnectorConfigurationResponse", 3 | "connectorOwner": "owner", 4 | "connectorName": "name", 5 | "connectorVersion": "v1.0", 6 | "connectorModes": ["SOURCE"], 7 | "authenticationConfig": { 8 | "isBasicAuthSupported": true, 9 | "isApiKeyAuthSupported": true, 10 | "isOAuth2Supported": false, 11 | "isCustomAuthSupported": false, 12 | "oAuth2Defaults": { 13 | "loginURL": ["a"], 14 | "authURL": ["b"], 15 | "refreshURL": ["c"], 16 | "oAuth2GrantTypesSupported": ["CLIENT_CREDENTIALS"], 17 | "oAuthScopes": ["e"], 18 | "redirectURL": ["f"] 19 | }, 20 | "customAuthConfig": [ 21 | { 22 | "authenticationType": "testType", 23 | "authParameters": [ 24 | { 25 | "key": "testKey", 26 | "required": true, 27 | "label": "testLabel", 28 | "description": "testDescription", 29 | "sensitiveField": true, 30 | "connectorSuppliedValues": [ 31 | "testValue" 32 | ] 33 | } 34 | ] 35 | } 36 | ] 37 | }, 38 | "supportedApiVersions": ["v1.0", "v1.1"], 39 | "isSuccess": true, 40 | "connectorRuntimeSetting": [ 41 | { 42 | "key": "key", 43 | "dataType": "String", 44 | "required": true, 45 | "label": "label", 46 | "description": "description", 47 | "scope": "SOURCE", 48 | "connectorSuppliedValueOptions": null 49 | } 50 | ], 51 | "logoURL": "test.amazon.com", 52 | "errorDetails": null, 53 | "operatorsSupported": ["ADDITION", "CONTAINS"], 54 | "triggerFrequenciesSupported": ["Once", "Daily"], 55 | "supportedWriteOperations": ["DELETE", "INSERT"] 56 | } 57 | -------------------------------------------------------------------------------- /custom_connector_tests/configuration/TestConfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "runtimeSettings": { 3 | "connectorProfile": { 4 | "instanceUrl": "https://amazon-1de-dev-ed.my.salesforce.com", 5 | "credentialsArn": "CredentialsArn", 6 | "apiVersion": "v51.0" 7 | } 8 | }, 9 | "credentials": { 10 | "secretArn": "", 11 | "authenticationType": "OAuth2" 12 | }, 13 | "testEntityIdentifier": "Account", 14 | "retrieveRecordConfigurations": [ 15 | { 16 | "entityIdentifier": "Account", 17 | "selectedFieldNames": [ 18 | "Name", 19 | "AccountNumber" 20 | ], 21 | "idFieldName": "AccountNumber", 22 | "ids": [ 23 | "12345" 24 | ] 25 | } 26 | ], 27 | "writeRecordConfigurations": [ 28 | { 29 | "entityIdentifier": "Account", 30 | "operation": "INSERT", 31 | "idFieldNames": [], 32 | "records": [ 33 | "{\"Name\": \"TestAccount\"}" 34 | ], 35 | "allOrNone": false 36 | }, 37 | { 38 | "entityIdentifier": "Account", 39 | "operation": "UPDATE", 40 | "idFieldNames": ["Id"], 41 | "records": [ 42 | "{\"Name\": \"NewNameUpdateTest\", \"Id\": \"0015e00000USsUgAAL\"}" 43 | ], 44 | "allOrNone": false 45 | }, 46 | { 47 | "entityIdentifier": "Account", 48 | "operation": "UPSERT", 49 | "idFieldNames": ["ExternalId__c"], 50 | "records": [ 51 | "{\"Name\": \"UpsertAccount\", \"ExternalId__c\": \"Identifier123\"}" 52 | ], 53 | "allOrNone": false 54 | } 55 | ], 56 | "queryRecordConfigurations": [ 57 | { 58 | "entityIdentifier": "Account", 59 | "selectedFieldNames": [ 60 | "Name", 61 | "AccountNumber" 62 | ], 63 | "filterExpression": "Name contains \"Test\"" 64 | } 65 | ] 66 | } 67 | -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/test-file-2.json: -------------------------------------------------------------------------------- 1 | { 2 | "customConnectorConfigurations":[ 3 | { 4 | "name":"connector1", 5 | "lambdaArn":"arn:aws:lambda:us-west-2:***********:function:custom-connector-Function-FpcRtBXSqTWy" 6 | } 7 | ], 8 | "customConnectorProfileConfigurations":[ 9 | { 10 | "connectorName":"connector1", 11 | "name":"profile1", 12 | "profileProperties":{ 13 | "api_version":"v51.0", 14 | "instanceUrl":"https://***********.my.salesforce.com" 15 | }, 16 | "defaultApiVersion": "v51.0", 17 | "authenticationType":"OAUTH2", 18 | "oAuth2Properties":{ 19 | "oAuth2GrantType":"CLIENT_CREDENTIALS", 20 | "tokenUrl":"https://login.salesforce.com/services/oauth2/token" 21 | }, 22 | "secretsManagerArn":"arn:aws:secretsmanager:us-west-2:***********:secret:custom-connector-qrSqOc" 23 | } 24 | ], 25 | "testBucketConfiguration": 26 | { 27 | "bucketName":"cvs-beta", 28 | "bucketPrefix":"" 29 | }, 30 | "listConnectorEntitiesTestConfigurations":[ 31 | 32 | ], 33 | "describeConnectorEntityTestConfigurations":[ 34 | 35 | ], 36 | "onDemandFromS3TestConfigurations":[ 37 | { 38 | "flowName": "flow4", 39 | "entityName":"Account", 40 | "writeOperationType": "INSERT", 41 | "dataGeneratorClassName":"custom_connector_example.integ_test.sales_generator.SalesForceTestData", 42 | "destinationRuntimeProperties": {} 43 | }, 44 | { 45 | "flowName": "flow2", 46 | "entityName":"Account", 47 | "writeOperationType": "INSERT", 48 | "sourceDataFile":"custom_connector_example/salesforce-example-test-files/salesforce-insert-file.csv", 49 | "destinationRuntimeProperties": {} 50 | } 51 | ], 52 | "onDemandToS3TestConfigurations":[ 53 | ] 54 | } -------------------------------------------------------------------------------- /custom_connector_tools/logFetcher.sh: -------------------------------------------------------------------------------- 1 | echo -e "\033[33m*********** Warning: Please make sure you give correct region, loggroup and Suffix ***********\033[0m" 2 | echo -n "Provide Region:" 3 | read -r 4 | region=$REPLY 5 | echo -n "Provide Loggroup (Should be in the format of /aws/lambda/custom-connector-logging-Aaw0rrvylsya):" 6 | read -r 7 | loggroup=$REPLY 8 | echo -n "Provide name for log file that will be generated:" 9 | read -r 10 | filePath=$REPLY 11 | echo -n "Provide start time (in epoc seconds) for log query:" 12 | read -r 13 | startTime=$REPLY 14 | echo -n "Provide End Time (in epoc seconds) for log query:" 15 | read -r 16 | endTime=$REPLY 17 | read -r -p "Provide Query String:" 18 | query=$REPLY 19 | echo -n "Provide Bucket for log file:" 20 | read -r 21 | bucket=$REPLY 22 | echo -n "Provide number of seconds until the pre-signed URL expires." 23 | read -r 24 | expiryInSeconds=$REPLY 25 | echo -n "Cloudwatch query takes time to execute. The time depends on the interval for which logs are being fetched." 26 | echo -n "Provide wait time (seconds) before query finished." 27 | sleepTime=$REPLY 28 | echo -n "Are above details correct, Please select y/n:" 29 | read -r 30 | if [[ "$REPLY" = "n" ]]; then 31 | echo -e "User chose to end the script.Exiting...." 32 | exit 1 33 | fi 34 | if [[ "$REPLY" != "y" ]]; then 35 | echo -e "Please type either 'y' on 'n'.Exiting...." 36 | exit 1 37 | fi 38 | query_response=$(aws logs start-query \ 39 | --region $region \ 40 | --log-group-name $loggroup \ 41 | --start-time $startTime \ 42 | --end-time $endTime \ 43 | --query-string "$query") 44 | query_id=$(echo $query_response | jq -r '.queryId') 45 | 46 | echo -e "\033[33m*********** Sleeping for someitme to make sure the query is in complete state ***********\033[0m" 47 | sleep $sleepTime 48 | 49 | aws logs get-query-results --query-id $query_id --region $region > /home/$USER/$filePath 50 | 51 | aws s3 cp $filePath s3://$bucket 52 | aws s3 presign s3://$bucket/$filePath --expires-in $expiryInSeconds 53 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/describe_entity_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "DescribeEntityRequest", 3 | "entityIdentifier": "EmailMessage", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 7 | }, 8 | "credentials": { 9 | "secretArn": "TestSecretArn", 10 | "authenticationType": "OAuth2" 11 | }, 12 | "entityDefinition": { 13 | "entity": { 14 | "entityIdentifier": "testIdentifier", 15 | "hasNestedEntities": true, 16 | "label": "testLabel", 17 | "description": "testDescription" 18 | }, 19 | "fields": [ 20 | { 21 | "fieldName": "testField", 22 | "dataType": "String", 23 | "label": "testFieldLabel", 24 | "description": "testFieldDescription", 25 | "isPrimaryKey": true, 26 | "defaultValue": "defaultValue", 27 | "isDeprecated": false, 28 | "constraints": { 29 | "allowedLengthRange": { 30 | "minRange": 0, 31 | "maxRange": 30 32 | }, 33 | "allowedValues": [ 34 | "defaultValue", 35 | "anotherValue", 36 | "someValue" 37 | ], 38 | "allowedValuesRegexPattern": "*Value" 39 | }, 40 | "readProperties": { 41 | "isRetrievable": true, 42 | "isNullable": true, 43 | "isQueryable": true, 44 | "isTimestampFieldForIncrementalQueries": false 45 | }, 46 | "writeProperties": { 47 | "isCreatable": true, 48 | "isUpdatable": true, 49 | "isNullable": true, 50 | "isUpsertable": true, 51 | "supportedWriteOperations": [ 52 | "INSERT", 53 | "UPDATE", 54 | "UPSERT" 55 | ] 56 | }, 57 | "customProperties": { 58 | "customProperty": "customPropertyValue" 59 | } 60 | } 61 | ], 62 | "customProperties": { 63 | "customProperty": "customPropertyValue" 64 | } 65 | }, 66 | "apiVersion": "v47.0" 67 | } 68 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/retrieve_data_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "RetrieveDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "test.amazon.com" 7 | }, 8 | "credentials": { 9 | "secretArn": "TestSecretArn", 10 | "authenticationType": "OAuth2" 11 | }, 12 | "entityDefinition": { 13 | "entity": { 14 | "entityIdentifier": "testIdentifier", 15 | "hasNestedEntities": true, 16 | "label": "testLabel", 17 | "description": "testDescription" 18 | }, 19 | "fields": [ 20 | { 21 | "fieldName": "testField", 22 | "dataType": "String", 23 | "label": "testFieldLabel", 24 | "description": "testFieldDescription", 25 | "isPrimaryKey": true, 26 | "defaultValue": "defaultValue", 27 | "isDeprecated": false, 28 | "constraints": { 29 | "allowedLengthRange": { 30 | "minRange": 0, 31 | "maxRange": 30 32 | }, 33 | "allowedValues": [ 34 | "defaultValue", 35 | "anotherValue", 36 | "someValue" 37 | ], 38 | "allowedValuesRegexPattern": "*Value" 39 | }, 40 | "readProperties": { 41 | "isRetrievable": true, 42 | "isNullable": true, 43 | "isQueryable": true, 44 | "isTimestampFieldForIncrementalQueries": false 45 | }, 46 | "writeProperties": { 47 | "isCreatable": true, 48 | "isUpdatable": true, 49 | "isNullable": true, 50 | "isUpsertable": true, 51 | "supportedWriteOperations": [ 52 | "INSERT", 53 | "UPDATE", 54 | "UPSERT" 55 | ] 56 | }, 57 | "customProperties": { 58 | "customProperty": "customPropertyValue" 59 | } 60 | } 61 | ], 62 | "customProperties": { 63 | "customProperty": "customPropertyValue" 64 | } 65 | }, 66 | "apiVersion": "v47.0" 67 | }, 68 | "selectedFieldNames": ["fieldA", "fieldB"], 69 | "idFieldName": "idField", 70 | "ids": ["idA", "idB"] 71 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/query_data_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "test.amazon.com" 7 | }, 8 | "credentials": { 9 | "basicAuthCredentials": null, 10 | "apiKeyCredentials": null, 11 | "oAuth2Credentials": { 12 | "accessToken": "accessTokenValue", 13 | "refreshToken": "refreshTokenValue" 14 | }, 15 | "customAuthCredentials": null 16 | }, 17 | "entityDefinition": { 18 | "fields": [ 19 | { 20 | "fieldName": "testField", 21 | "dataType": "String", 22 | "label": "testFieldLabel", 23 | "description": "testFieldDescription", 24 | "isPrimaryKey": true, 25 | "defaultValue": "defaultValue", 26 | "isDeprecated": false, 27 | "constraints": { 28 | "allowedLengthRange": { 29 | "minRange": 0, 30 | "maxRange": 30 31 | }, 32 | "allowedValues": [ 33 | "defaultValue", 34 | "anotherValue", 35 | "someValue" 36 | ], 37 | "allowedValuesRegexPattern": "*Value" 38 | }, 39 | "readProperties": { 40 | "isRetrievable": true, 41 | "isNullable": true, 42 | "isQueryable": true, 43 | "isTimestampFieldForIncrementalQueries": false 44 | }, 45 | "writeProperties": { 46 | "isCreatable": true, 47 | "isUpdatable": true, 48 | "isNullable": true, 49 | "isUpsertable": true, 50 | "supportedWriteOperations": [ 51 | "INSERT", 52 | "UPDATE", 53 | "UPSERT" 54 | ] 55 | }, 56 | "customProperties": { 57 | "customProperty": "customPropertyValue" 58 | } 59 | } 60 | ], 61 | "customProperties": { 62 | "customProperty": "customPropertyValue" 63 | } 64 | }, 65 | "apiVersion": "v47.0" 66 | }, 67 | "selectedFieldNames": ["fieldA", "fieldB"], 68 | "filterExpression": "testExpression", 69 | "nextToken": "testToken", 70 | "maxResults": 50 71 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/write_data_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "WriteDataRequest", 3 | "entityIdentifier": "identifier", 4 | "operation": "UPDATE", 5 | "connectorContext": { 6 | "connectorRuntimeSettings": { 7 | "instanceUrl": "test.amazon.com" 8 | }, 9 | "credentials": { 10 | "secretArn": "TestSecretArn", 11 | "authenticationType": "OAuth2" 12 | }, 13 | "entityDefinition": { 14 | "entity": { 15 | "entityIdentifier": "testIdentifier", 16 | "hasNestedEntities": true, 17 | "label": "testLabel", 18 | "description": "testDescription" 19 | }, 20 | "fields": [ 21 | { 22 | "fieldName": "testField", 23 | "dataType": "String", 24 | "label": "testFieldLabel", 25 | "description": "testFieldDescription", 26 | "isPrimaryKey": true, 27 | "defaultValue": "defaultValue", 28 | "isDeprecated": false, 29 | "constraints": { 30 | "allowedLengthRange": { 31 | "minRange": 0, 32 | "maxRange": 30 33 | }, 34 | "allowedValues": [ 35 | "defaultValue", 36 | "anotherValue", 37 | "someValue" 38 | ], 39 | "allowedValuesRegexPattern": "*Value" 40 | }, 41 | "readProperties": { 42 | "isRetrievable": true, 43 | "isNullable": true, 44 | "isQueryable": true, 45 | "isTimestampFieldForIncrementalQueries": false 46 | }, 47 | "writeProperties": { 48 | "isCreatable": true, 49 | "isUpdatable": true, 50 | "isNullable": true, 51 | "isUpsertable": true, 52 | "supportedWriteOperations": [ 53 | "INSERT", 54 | "UPDATE", 55 | "UPSERT" 56 | ] 57 | }, 58 | "customProperties": { 59 | "customProperty": "customPropertyValue" 60 | } 61 | } 62 | ], 63 | "customProperties": { 64 | "customProperty": "customPropertyValue" 65 | } 66 | }, 67 | "apiVersion": "v47.0" 68 | }, 69 | "idFieldNames": ["idFieldA", "idFieldB"], 70 | "records": ["recordA", "recordB"], 71 | "allOrNone": false 72 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/list_entities_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesRequest", 3 | "entitiesPath": "https://amazon141-dev-ed.my.salesforce.com", 4 | "maxResult": 500, 5 | "nextToken": "2", 6 | "connectorContext": { 7 | "connectorRuntimeSettings": { 8 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 9 | }, 10 | "credentials": { 11 | "secretArn": "TestSecretArn", 12 | "authenticationType": "OAuth2" 13 | }, 14 | "entityDefinition": { 15 | "entity": { 16 | "entityIdentifier": "testIdentifier", 17 | "hasNestedEntities": true, 18 | "label": "testLabel", 19 | "description": "testDescription" 20 | }, 21 | "fields": [ 22 | { 23 | "fieldName": "testField", 24 | "dataType": "Struct", 25 | "dataTypeLabel": "String", 26 | "label": "testFieldLabel", 27 | "description": "testFieldDescription", 28 | "isPrimaryKey": true, 29 | "defaultValue": "defaultValue", 30 | "isDeprecated": false, 31 | "constraints": { 32 | "allowedLengthRange": { 33 | "minRange": 0, 34 | "maxRange": 30 35 | }, 36 | "allowedValues": [ 37 | "defaultValue", 38 | "anotherValue", 39 | "someValue" 40 | ], 41 | "allowedValuesRegexPattern": "*Value" 42 | }, 43 | "readProperties": { 44 | "isRetrievable": true, 45 | "isNullable": true, 46 | "isQueryable": true, 47 | "isTimestampFieldForIncrementalQueries": false 48 | }, 49 | "writeProperties": { 50 | "isCreatable": true, 51 | "isUpdatable": true, 52 | "isNullable": true, 53 | "isUpsertable": true, 54 | "supportedWriteOperations": [ 55 | "INSERT", 56 | "UPDATE", 57 | "UPSERT" 58 | ] 59 | }, 60 | "customProperties": { 61 | "customProperty": "customPropertyValue" 62 | } 63 | } 64 | ], 65 | "customProperties": { 66 | "customProperty": "customPropertyValue" 67 | } 68 | }, 69 | "apiVersion": "v47.0" 70 | } 71 | } -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/query_data_request_optional.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "QueryDataRequest", 3 | "entityIdentifier": "identifier", 4 | "connectorContext": { 5 | "connectorRuntimeSettings": { 6 | "instanceUrl": "test.amazon.com" 7 | }, 8 | "credentials": { 9 | "secretArn": "TestSecretArn", 10 | "authenticationType": "OAuth2" 11 | }, 12 | "entityDefinition": { 13 | "entity": { 14 | "entityIdentifier": "testIdentifier", 15 | "hasNestedEntities": true, 16 | "label": "testLabel", 17 | "description": "testDescription" 18 | }, 19 | "fields": [ 20 | { 21 | "fieldName": "testField", 22 | "dataType": "String", 23 | "label": "testFieldLabel", 24 | "description": "testFieldDescription", 25 | "isPrimaryKey": true, 26 | "defaultValue": "defaultValue", 27 | "isDeprecated": false, 28 | "constraints": { 29 | "allowedLengthRange": { 30 | "minRange": 0, 31 | "maxRange": 30 32 | }, 33 | "allowedValues": [ 34 | "defaultValue", 35 | "anotherValue", 36 | "someValue" 37 | ], 38 | "allowedValuesRegexPattern": "*Value" 39 | }, 40 | "readProperties": { 41 | "isRetrievable": true, 42 | "isNullable": true, 43 | "isQueryable": true, 44 | "isTimestampFieldForIncrementalQueries": false 45 | }, 46 | "writeProperties": { 47 | "isCreatable": true, 48 | "isUpdatable": true, 49 | "isNullable": true, 50 | "isUpsertable": true, 51 | "supportedWriteOperations": [ 52 | "INSERT", 53 | "UPDATE", 54 | "UPSERT" 55 | ] 56 | }, 57 | "customProperties": { 58 | "customProperty": "customPropertyValue" 59 | } 60 | } 61 | ], 62 | "customProperties": { 63 | "customProperty": "customPropertyValue" 64 | } 65 | }, 66 | "apiVersion": "v47.0" 67 | }, 68 | "selectedFieldNames": ["fieldA", "fieldB"], 69 | "filterExpression": "testExpression", 70 | "nextToken": "testToken", 71 | "maxResults": 50 72 | } -------------------------------------------------------------------------------- /custom_connector_tools/deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -o pipefail 3 | 4 | cat << EOF 5 | # Run this script from the custom_connector_tools directory. 6 | # This script performs the following actions: 7 | # 1. Creates the bucket if you have provided empty string. 8 | # 2. Removes the package name directory from the directory where python sdk exists. 9 | # 3. Install the requirements in the package name directory. 10 | # 3. Uploads the packaged connector code to the S3 bucket you specified/or created if not specified. 11 | # 5. Deploys the connector. 12 | # 6. Describe the connector stack resources created. 13 | # 7. Please verify the policies in custom-connector-example/template.yml. 14 | EOF 15 | 16 | while true; do 17 | read -p "Do you wish to proceed? (yes or no) " yn 18 | case $yn in 19 | [Yy]* ) echo "Proceeding..."; break;; 20 | [Nn]* ) exit;; 21 | * ) echo "Please answer yes or no.";; 22 | esac 23 | done 24 | 25 | if [ "$#" -lt 4 ]; then 26 | echo "\n\nERROR: Script requires 4 arguments \n" 27 | echo "\n1. The AWS_REGION to target (e.g. us-east-1 or us-east-2) \n" 28 | echo "\n2. S3_BUCKET used for publishing artifacts.\n" 29 | echo "\n3. The STACK_NAME to create in cloudformation \n" 30 | echo "\n4. The PACKAGE_NAME to install the dependency libraries. \n" 31 | exit; 32 | fi 33 | 34 | AWS_REGION=$1 35 | if [ -z "$AWS_REGION" ] 36 | then 37 | AWS_REGION="us-east-1" 38 | fi 39 | echo "Using AWS Region $AWS_REGION" 40 | 41 | BUCKET_NAME=$2 42 | if [ -z "$BUCKET_NAME" ] 43 | then 44 | BUCKET_ID=$(dd if=/dev/random bs=8 count=1 2>/dev/null | od -An -tx1 | tr -d ' \t\n') 45 | BUCKET_NAME=customconnector-artifacts-$BUCKET_ID 46 | aws --region $AWS_REGION s3 mb s3://$BUCKET_NAME 47 | fi 48 | echo "Using Bucket Name $BUCKET_NAME" 49 | 50 | STACK_NAME=$3 51 | echo "Using Stack Name $STACK_NAME" 52 | 53 | PACKAGE_NAME=$4 54 | echo "Using Package Name $PACKAGE_NAME" 55 | 56 | rm -rf ../../"$PACKAGE_NAME" 57 | pip3 install --target ../../"$PACKAGE_NAME"/python -r ../requirements.txt 58 | 59 | aws --region $AWS_REGION cloudformation package --template-file template.yml --s3-bucket "$BUCKET_NAME" --output-template-file out.yml 60 | aws --region $AWS_REGION cloudformation deploy --template-file out.yml --stack-name "$STACK_NAME" --capabilities CAPABILITY_NAMED_IAM 61 | aws --region $AWS_REGION cloudformation describe-stack-resources --stack-name "$STACK_NAME" 62 | -------------------------------------------------------------------------------- /custom_connector_sdk/test/resources/list_entities_request_invalid.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "ListEntitiesRequest", 3 | "entitiesPath": "https://amazon141-dev-ed.my.salesforce.com", 4 | "maxResult": 500, 5 | "nextToken": "2", 6 | "connectorContext": { 7 | "connectorRuntimeSettings": { 8 | "instanceUrl": "https://amazon141-dev-ed.my.salesforce.com" 9 | }, 10 | "credentials": { 11 | "basicAuthCredentials": null, 12 | "apiKeyCredentials": null, 13 | "oAuth2Credentials": { 14 | "accessToken": "accessTokenValue", 15 | "refreshToken": "refreshTokenValue" 16 | }, 17 | "customAuthCredentials": null 18 | }, 19 | "entityDefinition": { 20 | "entity": { 21 | "entityIdentifier": "testIdentifier", 22 | "hasNestedEntities": true, 23 | "label": "testLabel", 24 | "description": "testDescription" 25 | }, 26 | "fields": [ 27 | { 28 | "dataType": "String", 29 | "label": "testFieldLabel", 30 | "description": "testFieldDescription", 31 | "isPrimaryKey": true, 32 | "defaultValue": "defaultValue", 33 | "isDeprecated": false, 34 | "constraints": { 35 | "allowedLengthRange": { 36 | "minRange": 0, 37 | "maxRange": 30 38 | }, 39 | "allowedValues": [ 40 | "defaultValue", 41 | "anotherValue", 42 | "someValue" 43 | ], 44 | "allowedValuesRegexPattern": "*Value" 45 | }, 46 | "readProperties": { 47 | "isRetrievable": true, 48 | "isNullable": true, 49 | "isQueryable": true, 50 | "isTimestampFieldForIncrementalQueries": false 51 | }, 52 | "writeProperties": { 53 | "isCreatable": true, 54 | "isUpdatable": true, 55 | "isNullable": true, 56 | "isUpsertable": true, 57 | "supportedWriteOperations": [ 58 | "INSERT", 59 | "UPDATE", 60 | "UPSERT" 61 | ] 62 | }, 63 | "customProperties": { 64 | "customProperty": "customPropertyValue" 65 | } 66 | } 67 | ], 68 | "customProperties": { 69 | "customProperty": "customPropertyValue" 70 | } 71 | }, 72 | "apiVersion": "v47.0" 73 | } 74 | } -------------------------------------------------------------------------------- /custom_connector_example/handlers/validation.py: -------------------------------------------------------------------------------- 1 | from typing import List, Optional 2 | 3 | from custom_connector_sdk.lambda_handler.responses import ErrorDetails, ErrorCode 4 | from custom_connector_sdk.connector.context import ConnectorContext, Credentials 5 | import custom_connector_sdk.lambda_handler.requests as requests 6 | import custom_connector_example.constants as constants 7 | 8 | def validate_write_data_request(request: requests.WriteDataRequest) -> Optional[ErrorDetails]: 9 | errors = [] 10 | if request.operation not in constants.SUPPORTED_WRITE_OPERATIONS: 11 | errors.append(f'Operation {request.operation.name} not supported by Salesforce') 12 | errors += check_connector_context_errors(request.connector_context) 13 | if not errors: 14 | return None 15 | return ErrorDetails(error_code=ErrorCode.InvalidArgument, error_message=','.join(errors)) 16 | 17 | def validate_credentials(request: requests.ValidateCredentialsRequest) -> Optional[ErrorDetails]: 18 | errors = check_credentials_input_errors(request.credentials) 19 | if not errors: 20 | return None 21 | return ErrorDetails(error_code=ErrorCode.InvalidArgument, error_message=','.join(errors)) 22 | 23 | def validate_connector_runtime_settings(request: requests.ValidateConnectorRuntimeSettingsRequest) -> \ 24 | Optional[ErrorDetails]: 25 | errors = check_connector_runtime_settings_errors(request.connector_runtime_settings) 26 | if not errors: 27 | return None 28 | return ErrorDetails(error_code=ErrorCode.InvalidArgument, error_message=','.join(errors)) 29 | 30 | def validate_request_connector_context(request) -> Optional[ErrorDetails]: 31 | errors = check_connector_context_errors(request.connector_context) 32 | if not errors: 33 | return None 34 | return ErrorDetails(error_code=ErrorCode.InvalidArgument, error_message=','.join(errors)) 35 | 36 | def check_connector_context_errors(connector_context: ConnectorContext) -> List[str]: 37 | errors = check_credentials_input_errors(connector_context.credentials) 38 | errors += check_connector_runtime_settings_errors(connector_context.connector_runtime_settings) 39 | return errors 40 | 41 | def check_credentials_input_errors(credentials: Credentials) -> List[str]: 42 | errors = [] 43 | if not credentials.secret_arn: 44 | errors.append('OAuth2 credentials should be provided using SecretsManager ARN') 45 | return errors 46 | 47 | def check_connector_runtime_settings_errors(connector_runtime_settings: dict) -> List[str]: 48 | errors = [] 49 | if not connector_runtime_settings or constants.INSTANCE_URL_KEY not in connector_runtime_settings: 50 | errors.append(f'{constants.INSTANCE_URL_KEY} should be provided as runtime setting for Salesforce connector') 51 | return errors 52 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/grammar/CustomConnectorQueryFilterLexer.g4: -------------------------------------------------------------------------------- 1 | lexer grammar CustomConnectorQueryFilterLexer; 2 | 3 | // logical operator tokens 4 | AND : 'AND' | 'and' ; 5 | OR : 'OR' | 'or' ; 6 | NOT : 'NOT' | 'not' ; 7 | TRUE : 'TRUE' |'True'|'true' ; 8 | FALSE : 'FALSE' | 'False'| 'false' ; 9 | GT : '>' ; 10 | GE : '>=' ; 11 | LT : '<' ; 12 | LE : '<=' ; 13 | EQ : '=' ; 14 | NE : '!='; 15 | LIKE : 'CONTAINS' | 'contains' ; 16 | BETWEEN : 'BETWEEN' | 'between' ; 17 | // string literals 18 | LPAREN : '(' ; 19 | RPAREN : ')' ; 20 | NULL : 'null'; 21 | IN : 'IN' | 'in'; 22 | LIMIT : 'LIMIT' | 'limit'; 23 | COMMA : ','; 24 | 25 | // represents identifier string in filter expression. 26 | IDENTIFIER : [a-zA-Z][A-Za-z0-9_.-]*; 27 | 28 | // represents a positive non-zero integer 29 | POS_INTEGER: [1-9][0-9]+ 30 | ; 31 | 32 | // represents decimal values like 5.0 or -5.0 etc 33 | DECIMAL :'-'? [0-9]+ ( '.' [0-9]+ )? 34 | ; 35 | 36 | // represents single quote string like 'grammar' etc 37 | SINGLE_STRING 38 | : '\'' (~('\'') | STR_ESC)+ '\'' 39 | ; 40 | 41 | // represents double quote string like "grammar" etc 42 | DOUBLE_STRING 43 | : '"'(~('"') | STR_ESC)+ '"' 44 | ; 45 | 46 | // represents empty single quote string like '' 47 | EMPTY_SINGLE_STRING 48 | : '\'''\'' 49 | ; 50 | 51 | // represents empty double quote string like "" 52 | EMPTY_DOUBLE_STRING 53 | : '"''"' 54 | ; 55 | 56 | fragment STR_ESC 57 | : '\\' ('"' | '\\'|'\''| 't' | 'n' | 'r') // add more: Unicode esapes, ... 58 | ; 59 | // represents white spaces 60 | WS : [ \r\t\u000C\n]+ -> skip; 61 | 62 | // ISO 8601 Date and Time format 63 | // Year: 64 | // YYYY (eg 1997) 65 | // Year and month: 66 | // YYYY-MM (eg 1997-07) 67 | // Complete date: 68 | // YYYY-MM-DD (eg 1997-07-16) 69 | // Complete date plus hours and minutes: 70 | // YYYY-MM-DDThh:mmTZD (eg 1997-07-16T19:20+01:00) 71 | // Complete date plus hours, minutes and seconds: 72 | // YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00) 73 | // Complete date plus hours, minutes, seconds and a decimal fraction of a 74 | //second 75 | // YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00) 76 | //where: 77 | // 78 | // YYYY = four-digit year 79 | // MM = two-digit month (01=January, etc.) 80 | // DD = two-digit day of month (01 through 31) 81 | // hh = two digits of hour (00 through 23) (am/pm NOT allowed) 82 | // mm = two digits of minute (00 through 59) 83 | // ss = two digits of second (00 through 59) 84 | // s = one or more digits representing a decimal fraction of a second 85 | // TZD = time zone designator (Z or +hh:mm or -hh:mm) 86 | 87 | // CustomConnector will either support ISO DATE or ISO DateTime 88 | 89 | DATE : [0-9] [0-9] [0-9] [0-9] '-' ('0'[1-9] | '1'[0-2]) '-' ('0'[0-9] | '1'[0-9] | '2'[0-9] | '3'[0-1]); 90 | DATETIME : DATE 'T' TIME; 91 | 92 | fragment TIME : ('0'[0-9] | '1'[0-9] | '2'[0-4]) ':' [0-5][0-9] ':' [0-5][0-9] ('.' [0-9]+)? 'Z'; 93 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/grammar/CustomConnectorQueryFilterParser.g4: -------------------------------------------------------------------------------- 1 | parser grammar CustomConnectorQueryFilterParser; 2 | 3 | options { tokenVocab=CustomConnectorQueryFilterLexer; } 4 | 5 | // 'queryfilter' is the root node for the filter expression 6 | queryfilter 7 | : expression EOF 8 | | limitexpression EOF 9 | ; 10 | 11 | limitexpression 12 | : op=limit right=count #limitExpression 13 | | left=expression op=limit right=count #limitExpression // SQL 'LIMIT xyz' operator 14 | ; 15 | 16 | expression 17 | : LPAREN expression RPAREN #parenExpression // supports parenthesis expressions 18 | // logical operators support 19 | | NOT expression #notExpression 20 | | left=expression op=andBinary right=expression #aNDBinaryExpression 21 | | left=expression op=orBinary right=expression #oRBinaryExpression 22 | | left=identifier op=gtComparator right=value #greaterThanComparatorExpression 23 | | left=identifier op=geComparator right=value #greaterThanEqualToComparatorExpression 24 | | left=identifier op=ltComparator right=value #lesserThanComparatorExpression 25 | | left=identifier op=leComparator right=value #lesserThanEqualToComparatorExpression 26 | | left=identifier op=eqComparator right=value #equalToComparatorExpression 27 | | left=identifier op=eqComparator right=boolean #booleanEqualToComparatorExpression 28 | | left=identifier op=neComparator right=value #notEqualToComparatorExpression 29 | | left=identifier op=neComparator right=boolean #booleanNotEqualToComparatorExpression 30 | | left=identifier op=likeComparator right=value #likeComparatorExpression 31 | | (left=identifier op=betweenComparator (l1=value op1=andBinary right=value)) #betweenExpression 32 | | identifier #identifierExpression 33 | // Following is a leaf node in the parse tree 34 | // This allows validation and transformations of values. 35 | | value #valueExpression 36 | | identifier op=inOperator LPAREN value (COMMA value)* RPAREN # inExpression // Supports SQL like 'IN' operator 37 | ; 38 | 39 | gtComparator 40 | : GT ; 41 | 42 | geComparator 43 | : GE ; 44 | 45 | ltComparator 46 | : LT ; 47 | 48 | leComparator 49 | : LE ; 50 | 51 | eqComparator 52 | : EQ ; 53 | 54 | neComparator 55 | : NE ; 56 | 57 | likeComparator 58 | : LIKE ; 59 | 60 | betweenComparator 61 | : BETWEEN ; 62 | 63 | andBinary 64 | : AND ; 65 | 66 | orBinary 67 | : OR 68 | ; 69 | 70 | boolean 71 | : TRUE | FALSE 72 | ; 73 | 74 | identifier 75 | :IDENTIFIER ; 76 | 77 | inOperator 78 | :IN ; 79 | 80 | limit 81 | :LIMIT ; 82 | 83 | // Following is to support different String formats in the value expression 84 | string 85 | : SINGLE_STRING 86 | | DOUBLE_STRING 87 | | EMPTY_DOUBLE_STRING 88 | | EMPTY_SINGLE_STRING 89 | | NULL 90 | ; 91 | 92 | value 93 | : string #stringValueExpression 94 | | POS_INTEGER #decimalValueExpression 95 | | DECIMAL #decimalValueExpression 96 | | DATE #isoDate 97 | | DATETIME #isoDateTime 98 | ; 99 | 100 | count 101 | : POS_INTEGER #countValueExpression 102 | ; 103 | -------------------------------------------------------------------------------- /custom_connector_example/handlers/salesforce.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | import boto3 4 | from typing import Optional 5 | import custom_connector_example.constants as constants 6 | from custom_connector_sdk.connector.context import ConnectorContext 7 | from custom_connector_sdk.lambda_handler.responses import ErrorDetails, ErrorCode 8 | from custom_connector_sdk.connector.auth import ACCESS_TOKEN 9 | from custom_connector_example.handlers.client import HttpsClient, SalesforceResponse 10 | 11 | HTTP_STATUS_CODE_RANGE = range(200, 300) 12 | LOGGER = logging.getLogger() 13 | 14 | def get_salesforce_client(connector_context: ConnectorContext): 15 | return HttpsClient(get_access_token_from_secret(connector_context.credentials.secret_arn)) 16 | 17 | def check_for_errors_in_salesforce_response(response: SalesforceResponse) -> Optional[ErrorDetails]: 18 | """Parse Salesforce response for errors and convert them to an ErrorDetails object.""" 19 | status_code = response.status_code 20 | 21 | if status_code in HTTP_STATUS_CODE_RANGE: 22 | return None 23 | 24 | if status_code == 401: 25 | error_code = ErrorCode.InvalidCredentials 26 | elif status_code == 400: 27 | error_code = ErrorCode.InvalidArgument 28 | else: 29 | error_code = ErrorCode.ServerError 30 | 31 | error_message = f'Request failed with status code {status_code} error reason {response.error_reason} and ' + \ 32 | f'Salesforce response is {response.response}' 33 | LOGGER.error(error_message) 34 | 35 | return ErrorDetails(error_code=error_code, error_message=error_message) 36 | 37 | def build_salesforce_request_uri(connector_context: ConnectorContext, url_format: str, request_path: str) -> str: 38 | connector_runtime_settings = connector_context.connector_runtime_settings 39 | instance_url = connector_runtime_settings.get(constants.INSTANCE_URL_KEY) 40 | instance_url = add_path(instance_url) 41 | api_version = connector_context.api_version 42 | 43 | request_uri = url_format.format(instance_url, api_version, request_path) 44 | 45 | return request_uri 46 | 47 | def get_access_token_from_secret(secret_arn: str) -> str: 48 | secrets_manager = boto3.client('secretsmanager') 49 | secret = secrets_manager.get_secret_value(SecretId=secret_arn) 50 | 51 | return json.loads(secret["SecretString"])[ACCESS_TOKEN] 52 | 53 | def get_string_value(response: dict, field_name: str) -> Optional[str]: 54 | if field_name is None or response.get(field_name) is None: 55 | return None 56 | elif isinstance(response.get(field_name), bool): 57 | return str(response.get(field_name)).lower() 58 | else: 59 | return str(response.get(field_name)) 60 | 61 | def get_boolean_value(response: dict, field_name: str) -> bool: 62 | if field_name is None: 63 | return False 64 | elif field_name == 'true': 65 | return True 66 | elif response.get(field_name) is None: 67 | return False 68 | else: 69 | return bool(response.get(field_name)) 70 | 71 | 72 | def add_path(url: str) -> str: 73 | if url.endswith('/'): 74 | return url 75 | return url + '/' 76 | -------------------------------------------------------------------------------- /custom_connector_example/test/configuration_test.py: -------------------------------------------------------------------------------- 1 | import json 2 | import unittest 3 | from unittest.mock import patch 4 | import custom_connector_example.handlers.lambda_handler as lambda_handler 5 | from custom_connector_example.handlers.salesforce import SalesforceResponse 6 | 7 | class SalesforceConfigurationHandlerTests(unittest.TestCase): 8 | """Test class to validate handling of Configuration requests by Salesforce connector.""" 9 | def test_validate_connector_runtime_settings_request_valid(self): 10 | with open('custom_connector_example/test/resources/validate_connector_runtime_settings_request_valid.json', 11 | 'r') as json_file: 12 | data = json.load(json_file) 13 | response = lambda_handler.salesforce_lambda_handler(data, None) 14 | self.assertEqual(response.get('isSuccess'), True) 15 | 16 | def test_validate_connector_runtime_settings_request_invalid(self): 17 | with open('custom_connector_example/test/resources/validate_connector_runtime_settings_request_invalid.json', 18 | 'r') as json_file: 19 | data = json.load(json_file) 20 | response = lambda_handler.salesforce_lambda_handler(data, None) 21 | self.assertEqual(response.get('isSuccess'), False) 22 | 23 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 24 | def test_validate_credentials_request_success(self, mock): 25 | mock.return_value = SalesforceResponse(200, '{}', '') 26 | 27 | with open('custom_connector_example/test/resources/validate_credentials_request_valid.json', 28 | 'r') as json_file: 29 | data = json.load(json_file) 30 | response = lambda_handler.salesforce_lambda_handler(data, None) 31 | self.assertEqual(response.get('isSuccess'), True) 32 | 33 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 34 | def test_validate_credentials_request_failure(self, mock): 35 | mock.return_value = SalesforceResponse(401, '{}', 'Invalid Credentials') 36 | 37 | with open('custom_connector_example/test/resources/validate_credentials_request_valid.json', 38 | 'r') as json_file: 39 | data = json.load(json_file) 40 | response = lambda_handler.salesforce_lambda_handler(data, None) 41 | self.assertEqual(response.get('isSuccess'), False) 42 | 43 | def test_validate_credentials_request_invalid(self): 44 | with open('custom_connector_example/test/resources/validate_credentials_request_invalid.json', 45 | 'r') as json_file: 46 | data = json.load(json_file) 47 | response = lambda_handler.salesforce_lambda_handler(data, None) 48 | self.assertEqual(response.get('isSuccess'), False) 49 | 50 | def test_describe_connector_configuration_request(self): 51 | with open('custom_connector_example/test/resources/describe_connector_configuration_request_valid.json', 52 | 'r') as json_file: 53 | data = json.load(json_file) 54 | response = lambda_handler.salesforce_lambda_handler(data, None) 55 | self.assertEqual(response.get('isSuccess'), True) 56 | -------------------------------------------------------------------------------- /custom_connector_tests/configuration/ConfigSchema.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "object", 3 | "properties": { 4 | "runtimeSettings": { 5 | "type": "object", 6 | "properties": { 7 | "connectorProfile": {"$ref": "#/definitions/map"}, 8 | "source": {"$ref": "#/definitions/map"}, 9 | "destination": {"$ref": "#/definitions/map"} 10 | }, 11 | "additionalProperties": false 12 | }, 13 | "credentials": {"$ref": "#definitions/credentials"}, 14 | "testEntityIdentifier": {"type": "string"}, 15 | "retrieveRecordConfigurations": {"$ref": "#/definitions/retrieveRecordConfigurations"}, 16 | "writeRecordConfigurations": {"$ref": "#/definitions/writeRecordConfigurations"}, 17 | "queryRecordConfigurations": {"$ref": "#/definitions/queryRecordConfigurations"} 18 | }, 19 | "required": ["runtimeSettings", "credentials", "testEntityIdentifier"], 20 | "additionalProperties": false, 21 | "definitions": { 22 | "map": { 23 | "type": "object", 24 | "additionalProperties": {"type": "string"} 25 | }, 26 | "credentials": { 27 | "type": "object", 28 | "properties": { 29 | "secretArn": {"type": "string"}, 30 | "authenticationType": {"type": "string"} 31 | }, 32 | "additionalProperties": false 33 | }, 34 | "retrieveRecordConfigurations": { 35 | "type": "array", 36 | "items": { 37 | "type": "object", 38 | "properties": { 39 | "entityIdentifier": {"type": "string"}, 40 | "selectedFieldNames": {"type": "array", "items": {"type": "string"}}, 41 | "idFieldName": {"type": "string"}, 42 | "ids": {"type": "array", "items": {"type": "string"}} 43 | }, 44 | "required": ["entityIdentifier", "selectedFieldNames"], 45 | "additionalProperties": false 46 | } 47 | }, 48 | "writeRecordConfigurations": { 49 | "type": "array", 50 | "items": { 51 | "type": "object", 52 | "properties": { 53 | "entityIdentifier": {"type": "string"}, 54 | "operation": {"enum": ["INSERT", "UPDATE", "UPSERT"]}, 55 | "idFieldNames": {"type": "array", "items": {"type": "string"}}, 56 | "records": {"type": "array", "items": {"type": "string"}}, 57 | "allOrNone": {"type": "boolean"} 58 | }, 59 | "required": ["entityIdentifier"], 60 | "additionalProperties": false 61 | } 62 | }, 63 | "queryRecordConfigurations": { 64 | "type": "array", 65 | "items": { 66 | "type": "object", 67 | "properties": { 68 | "entityIdentifier": {"type": "string"}, 69 | "selectedFieldNames": {"type": "array", "items": {"type": "string"}}, 70 | "filterExpression": {"type": "string"} 71 | }, 72 | "required": ["entityIdentifier", "selectedFieldNames"], 73 | "additionalProperties": false 74 | } 75 | } 76 | } 77 | } -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/test-file.json: -------------------------------------------------------------------------------- 1 | { 2 | "customConnectorConfigurations":[ 3 | { 4 | "name":"connector1", 5 | "lambdaArn":"arn:aws:lambda:us-west-2:*********:function:custom-connector-Function-FpcRtBXSqTWy" 6 | }, 7 | { 8 | "name":"connector2", 9 | "lambdaArn":"arn:aws:lambda:us-west-2:*********:function:custom-connector-Function-FpcRtBXSqTWy" 10 | } 11 | ], 12 | "customConnectorProfileConfigurations":[ 13 | { 14 | "connectorName":"connector1", 15 | "name":"profile1", 16 | "profileProperties":{ 17 | "api_version":"v51.0", 18 | "instanceUrl":"https://*********.my.salesforce.com" 19 | }, 20 | "defaultApiVersion": "v51.0", 21 | "authenticationType":"OAUTH2", 22 | "oAuth2Properties":{ 23 | "oAuth2GrantType":"CLIENT_CREDENTIALS", 24 | "tokenUrl":"https://login.salesforce.com/services/oauth2/token" 25 | }, 26 | "secretsManagerArn":"arn:aws:secretsmanager:us-west-2:*********:secret:custom-connector-qrSqOc" 27 | }, 28 | { 29 | "name":"profile2", 30 | "profileProperties":{ 31 | "api_version":"v51.0", 32 | "instanceUrl":"https://*********.my.salesforce.com" 33 | }, 34 | "defaultApiVersion": "v51.0", 35 | "authenticationType":"OAUTH2", 36 | "oAuth2Properties":{ 37 | "oAuth2GrantType":"CLIENT_CREDENTIALS", 38 | "tokenUrl":"https://login.salesforce.com/services/oauth2/token" 39 | }, 40 | "secretsManagerArn":"arn:aws:secretsmanager:us-west-2:*********:secret:custom-connector-qrSqOc" 41 | }, 42 | { 43 | "connectorName":"connector2", 44 | "name":"profile3", 45 | "profileProperties":{ 46 | "api_version":"v51.0", 47 | "instanceUrl":"https://*********.my.salesforce.com" 48 | }, 49 | "defaultApiVersion": "v51.0", 50 | "authenticationType":"OAUTH2", 51 | "oAuth2Properties":{ 52 | "oAuth2GrantType":"CLIENT_CREDENTIALS", 53 | "tokenUrl":"https://login.salesforce.com/services/oauth2/token" 54 | }, 55 | "secretsManagerArn":"arn:aws:secretsmanager:us-west-2:*********:secret:custom-connector-qrSqOc" 56 | } 57 | ], 58 | "testBucketConfiguration": 59 | { 60 | "bucketName":"cvs-beta", 61 | "bucketPrefix":"" 62 | }, 63 | "listConnectorEntitiesTestConfigurations":[ 64 | { 65 | "profileName":"profile1", 66 | "apiVersion": "v51.0" 67 | }, 68 | { 69 | "profileName":"profile2" 70 | }, 71 | { 72 | 73 | } 74 | ], 75 | "describeConnectorEntityTestConfigurations":[ 76 | { 77 | "entityName":"Account", 78 | "profileName": "profile3", 79 | "testName": "", 80 | "apiVersion": "v52.0" 81 | }, 82 | { 83 | "entityName":"Account" 84 | }, 85 | { 86 | "entityName":"Account", 87 | "profileName": "profile2", 88 | "testName": "" 89 | } 90 | ], 91 | "onDemandFromS3TestConfigurations":[ 92 | ], 93 | "onDemandToS3TestConfigurations":[ 94 | { 95 | "testName":"", 96 | "flowName": "flow1", 97 | "entityName":"Account", 98 | "flowTimeout":100, 99 | "entityFields":["LastActivityDate"], 100 | "outputSize":443, 101 | "sourceRuntimeProperties": {} 102 | } 103 | ] 104 | } -------------------------------------------------------------------------------- /custom_connector_integ_test/utils/resource_info_provider.py: -------------------------------------------------------------------------------- 1 | import time 2 | from unittest import SkipTest 3 | 4 | from custom_connector_integ_test.configuration.test_configuration import TestConfiguration 5 | 6 | 7 | # This class also helps in defaulting resource names if they are not explicitly provided in the configuration. 8 | 9 | class ResourceInfoProvider: 10 | versions = {} 11 | names = {} 12 | 13 | created_profiles = set() 14 | created_connectors = set() 15 | 16 | integ_connector = "Integ_Connector_" 17 | 18 | integ_profile = "Integ_Profile_" 19 | 20 | integ_flow = "Integ_Flow_" 21 | 22 | def __init__(self, 23 | config: TestConfiguration): 24 | self.config = config 25 | self.prefix = config.resource_prefix or "" 26 | self.default_connector_name = config.custom_connector_configurations[0].name 27 | self.default_profile_name = config.custom_connector_profile_configurations[0].name 28 | for profileConfig in config.custom_connector_profile_configurations: 29 | self.versions[profileConfig.name] = profileConfig.default_api_version or 1 30 | for profileConfig in config.custom_connector_profile_configurations: 31 | self.names[profileConfig.name] = profileConfig.connector_name or self.default_connector_name 32 | self.test_start_time = str(int(time.time())) 33 | 34 | def get_api_for_profile_name(self, profile_name, api_version): 35 | return api_version or self.versions[profile_name or self.default_profile_name] 36 | 37 | def get_version_for_profile(self, profile_name): 38 | return self.versions[profile_name] 39 | 40 | def get_connector_for_profile(self, profile_name): 41 | return self.names[profile_name] 42 | 43 | def generate_resource_name(self, name, resource_type): 44 | return resource_type + self.prefix + name + self.test_start_time 45 | 46 | def generate_profile_name(self, name): 47 | return self.generate_resource_name(name, self.integ_profile) 48 | 49 | def generate_flow_name(self, name): 50 | return self.generate_resource_name(name, self.integ_flow) 51 | 52 | def generate_connector_name(self, name): 53 | return self.generate_resource_name(name, self.integ_connector) 54 | 55 | def get_connector_for_profile_name(self, profile_name): 56 | return self.get_connector_name(self.names[profile_name or self.default_profile_name]) 57 | 58 | def get_connector_name(self, connector_name): 59 | return self.get_connector_name_if_created(connector_name or self.default_connector_name) 60 | 61 | def get_profile_name(self, profile_name): 62 | return self.get_profile_name_if_created(profile_name or self.default_profile_name) 63 | 64 | def get_connector_name_if_created(self, name): 65 | if not self.created_connectors.__contains__(name): 66 | raise SkipTest(f'Connector {name} failed to Create') 67 | return self.generate_connector_name(name) 68 | 69 | def get_profile_name_if_created(self, name): 70 | if not self.created_profiles.__contains__(name): 71 | raise SkipTest(f'Profile {name} failed to Create') 72 | return self.generate_profile_name(name) 73 | 74 | def add_to_created_profiles(self, name): 75 | self.created_profiles.add(name) 76 | 77 | def add_to_created_connectors(self, name): 78 | self.created_connectors.add(name) 79 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /custom_connector_example/handlers/client.py: -------------------------------------------------------------------------------- 1 | import urllib3 2 | 3 | CONNECTION_TIMEOUT_SECS = 30 4 | READ_TIMEOUT_SECS = 600 5 | 6 | class SalesforceResponse: 7 | def __init__(self, status_code: int, response: str, error_reason: str): 8 | self.status_code = status_code 9 | self.response = response 10 | self.error_reason = error_reason 11 | 12 | class HttpsClient: 13 | def __init__(self, access_token): 14 | timeout = urllib3.Timeout(connect=CONNECTION_TIMEOUT_SECS, read=READ_TIMEOUT_SECS) 15 | self.https_client = urllib3.PoolManager(timeout=timeout) 16 | self.access_token = access_token 17 | self.authorization_header = {'Authorization': 'Bearer ' + access_token} 18 | 19 | def rest_get(self, request_uri: str) -> SalesforceResponse: 20 | headers = self.authorization_header 21 | resp = self.https_client.request(method='GET', 22 | url=request_uri, 23 | headers=headers) 24 | return SalesforceResponse(status_code=resp.status, 25 | response=resp.data.decode('utf-8'), 26 | error_reason=resp.reason) 27 | 28 | def rest_post(self, request_uri: str, post_data: str) -> SalesforceResponse: 29 | headers = {**self.authorization_header, 'Accept-Encoding': 'gzip', 'Content-Type': 'application/json'} 30 | resp = self.https_client.request(method='POST', 31 | url=request_uri, 32 | headers=headers, 33 | body=post_data) 34 | return SalesforceResponse(status_code=resp.status, 35 | response=resp.data.decode('utf-8'), 36 | error_reason=resp.reason) 37 | 38 | def rest_patch(self, request_uri: str, patch_data: str) -> SalesforceResponse: 39 | headers = {**self.authorization_header, 'Accept-Encoding': 'gzip', 'Content-Type': 'application/json'} 40 | resp = self.https_client.request(method='PATCH', 41 | url=request_uri, 42 | headers=headers, 43 | body=patch_data) 44 | return SalesforceResponse(status_code=resp.status, 45 | response=resp.data.decode('utf-8'), 46 | error_reason=resp.reason) 47 | 48 | def rest_put(self, request_uri: str, put_data: str) -> SalesforceResponse: 49 | headers = {**self.authorization_header, 'Content-Type': 'text/csv'} 50 | resp = self.https_client.request(method='PUT', 51 | url=request_uri, 52 | headers=headers, 53 | body=put_data) 54 | return SalesforceResponse(status_code=resp.status, 55 | response=resp.data.decode('utf-8'), 56 | error_reason=resp.reason) 57 | 58 | def rest_delete(self, request_uri: str) -> SalesforceResponse: 59 | resp = self.https_client.request(method='DELETE', 60 | url=request_uri, 61 | headers=self.authorization_header) 62 | return SalesforceResponse(status_code=resp.status, 63 | response=resp.data.decode('utf-8'), 64 | error_reason=resp.reason) 65 | -------------------------------------------------------------------------------- /custom_connector_example/salesforce-example-test-files/describe-connector-validation-file.json: -------------------------------------------------------------------------------- 1 | { 2 | "connectorConfiguration": { 3 | "canUseAsSource":true, 4 | "canUseAsDestination":true, 5 | "supportedDestinationConnectors":[ 6 | "S3", 7 | "CustomConnector", 8 | "Salesforce" 9 | ], 10 | "supportedSchedulingFrequencies":[ 11 | "BYMINUTE", 12 | "HOURLY", 13 | "DAILY", 14 | "WEEKLY", 15 | "MONTHLY", 16 | "ONCE" 17 | ], 18 | "isPrivateLinkEnabled":false, 19 | "isPrivateLinkEndpointUrlRequired":false, 20 | "supportedTriggerTypes":[ 21 | "OnDemand", 22 | "Scheduled" 23 | ], 24 | "connectorType":"CUSTOMCONNECTOR", 25 | "connectorLabel":null, 26 | "connectorDescription":"SampleSalesforceConnector", 27 | "connectorOwner":"SampleConnector", 28 | "connectorName":"SampleSalesforceConnector", 29 | "connectorVersion":"1.0", 30 | "connectorModes":[ 31 | "SOURCE", 32 | "DESTINATION" 33 | ], 34 | "authenticationConfig":{ 35 | "isBasicAuthSupported":false, 36 | "isApiKeyAuthSupported":false, 37 | "isOAuth2Supported":true, 38 | "isCustomAuthSupported":false, 39 | "oAuth2Defaults":{ 40 | "oauthScopes":[ 41 | "api", 42 | "refresh_token" 43 | ], 44 | "tokenUrls":[ 45 | "https://login.salesforce.com/services/oauth2/token" 46 | ], 47 | "authCodeUrls":[ 48 | "https://login.salesforce.com/services/oauth2/authorize" 49 | ], 50 | "refreshUrls":[ 51 | "https://login.salesforce.com/services/oauth2/token" 52 | ], 53 | "redirectUris":[ 54 | "https://login.salesforce.com" 55 | ], 56 | "oauth2GrantTypesSupported":[ 57 | "AUTHORIZATION_CODE" 58 | ] 59 | } 60 | }, 61 | "connectorRuntimeSettings":[ 62 | { 63 | "key":"instanceUrl", 64 | "dataType":"String", 65 | "isRequired":true, 66 | "label":"Salesforce Instance URL", 67 | "description":"URL of the instance where user wants to tun the operations.", 68 | "scope":"CONNECTOR_PROFILE" 69 | }, 70 | { 71 | "key":"api_version", 72 | "dataType":"String", 73 | "isRequired":true, 74 | "label":"Salesforce API version", 75 | "description":"Salesforce API version to use.", 76 | "scope":"CONNECTOR_PROFILE" 77 | } 78 | ], 79 | "supportedApiVersions":[ 80 | "v51.0" 81 | ], 82 | "supportedOperators":[ 83 | "PROJECTION", 84 | "LESS_THAN", 85 | "GREATER_THAN", 86 | "BETWEEN", 87 | "LESS_THAN_OR_EQUAL_TO", 88 | "GREATER_THAN_OR_EQUAL_TO", 89 | "EQUAL_TO", 90 | "CONTAINS", 91 | "NOT_EQUAL_TO", 92 | "ADDITION", 93 | "SUBTRACTION", 94 | "MULTIPLICATION", 95 | "DIVISION", 96 | "MASK_ALL", 97 | "MASK_FIRST_N", 98 | "MASK_LAST_N", 99 | "VALIDATE_NON_NULL", 100 | "VALIDATE_NON_ZERO", 101 | "VALIDATE_NON_NEGATIVE", 102 | "VALIDATE_NUMERIC", 103 | "NO_OP" 104 | ], 105 | "supportedWriteOperations":[ 106 | "INSERT" 107 | ], 108 | "connectorProvisioningType":"LAMBDA", 109 | "connectorProvisioningConfig":{ 110 | "lambda":{ 111 | "lambdaArn":"arn:aws:lambda:us-west-2:201726789647:function:custom-connector-function-xomoZvWcPYLp" 112 | } 113 | }, 114 | "registeredAt":null, 115 | "registeredBy":null 116 | } 117 | } -------------------------------------------------------------------------------- /custom_connector_queryfilter/tests/parse_tree_test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import custom_connector_queryfilter.queryfilter.parse_tree_builder as parse_tree_builder 3 | from custom_connector_queryfilter.queryfilter.errors import InvalidFilterExpressionError 4 | 5 | 6 | class CustomConnectorParseTreeTest(unittest.TestCase): 7 | """Unit tests for parse_tree_builder.""" 8 | def _test_parse_tree_with_valid_filter_expression(self, filter_expression): 9 | parse_tree_builder.parse(filter_expression) # Should not raise an exception 10 | 11 | def _test_parse_tree_with_invalid_filter_expression(self, filter_expression): 12 | self.assertRaises(InvalidFilterExpressionError, parse_tree_builder.parse, filter_expression) 13 | 14 | def test_valid_expression(self): 15 | expressions = [ 16 | 'os = "mojave"', 17 | 'os != "mojave"', 18 | "accountId > 90", 19 | "LIMIT 100", 20 | "dateRange BETWEEN 1611639470000 AND 1611639476298", 21 | "date BETWEEN 1511630000000 AND 1611639476298", 22 | "time between 1511630000000 AND 1611639476298", 23 | "accountId < 100", 24 | "accountId >= 90", 25 | "accountId <= 100", 26 | "accountId <= 100 LIMIT 100", 27 | "accountId BETWEEN 90 AND 100", 28 | "accountId BETWEEN 90 AND 100 LIMIT 100", 29 | 'os CONTAINS "mojave"', 30 | 'os CONTAINS "moj%ave"', 31 | 'os = "mojave" and app = "mo"', 32 | 'os = "mojave" OR app = "mo"', 33 | '(os = "mojave" AND app = "mo") and (os = "mojave" OR app = "mo")', 34 | '(os = "mojave" AND app = "mo") or (os = "mojave" OR app = "mo")', 35 | "accountId in (100, 90, 70)", 36 | "date between 2021-04-20 and 2021-04-21", 37 | 'date between 2021-04-20T12:30:45Z and 2021-04-20T15:45:49.234Z', 38 | '(accountId > 100 and ((date < 2021-04-20T12:30:45Z and date > 2021-04-21T15:45:49.234Z) and ' + 39 | 'accountId < 200))', 40 | "overrides = true or accountFlag != false", 41 | "overrides != true", 42 | "date > 2020-10-05T12:05:34Z" 43 | ] 44 | for expr in expressions: 45 | with self.subTest(expression=expr): 46 | print(expr) 47 | self._test_parse_tree_with_valid_filter_expression(expr) 48 | 49 | def test_invalid_expression(self): 50 | expressions = [ 51 | 'os == "mojave"', 52 | 'os <> "mojave"', 53 | "LIMIT 100 LIMIT 100", 54 | "accountId => 90", 55 | "accountId => 90 LIMIT", 56 | "accountId => 90 LIMIT 0", 57 | "accountId => 90 LIMIT -1", 58 | "accountId => 90 LIMIT 1.5", 59 | "accountId => 90 LIMIT abc", 60 | "dateRange in 1611639470000 AND 1611639476298", 61 | "date FROM 1611639470000 TO 1611639476298", 62 | "time Between 1611639470000 and 1611639476298", 63 | 'os CONTAIN "mojave"', 64 | 'os CONTAINS "moj%ave', 65 | 'accountId in (id, "90", 70)', 66 | "accountId in (true)", 67 | "date > 2021-04-203", 68 | "date > 2021-04-20T20:30", 69 | "date > 2021-04_20T20:30:20.9999+26", 70 | "date > 2021-04_20T20:30:20.9999+12:23", 71 | "date > 2021-04-20T20:30:20.9999-12:23" 72 | ] 73 | for expr in expressions: 74 | with self.subTest(expression=expr): 75 | print(expr) 76 | self._test_parse_tree_with_invalid_filter_expression(expr) 77 | -------------------------------------------------------------------------------- /custom_connector_sdk/lambda_handler/lambda_handler.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import custom_connector_sdk.lambda_handler.requests as requests 3 | from custom_connector_sdk.lambda_handler.handlers import MetadataHandler, RecordHandler, ConfigurationHandler 4 | 5 | VALIDATE_CONNECTOR_RUNTIME_SETTINGS = 'ValidateConnectorRuntimeSettingsRequest' 6 | VALIDATE_CREDENTIALS = 'ValidateCredentialsRequest' 7 | DESCRIBE_CONNECTOR_CONFIGURATION = 'DescribeConnectorConfigurationRequest' 8 | LIST_ENTITIES = 'ListEntitiesRequest' 9 | DESCRIBE_ENTITY = 'DescribeEntityRequest' 10 | RETRIEVE_DATA = 'RetrieveDataRequest' 11 | WRITE_DATA = 'WriteDataRequest' 12 | QUERY_DATA = 'QueryDataRequest' 13 | 14 | class BaseLambdaConnectorHandler: 15 | """Base class for Lambda Connector handlers. It is recommended that all connectors extend this class for Lambda 16 | operations, though it is possible for you to write your own from the ground up.""" 17 | def __init__(self, 18 | metadata_handler: MetadataHandler, 19 | record_handler: RecordHandler, 20 | configuration_handler: ConfigurationHandler): 21 | self.metadata_handler = metadata_handler 22 | self.record_handler = record_handler 23 | self.configuration_handler = configuration_handler 24 | 25 | def lambda_handler(self, event, context): 26 | logger = logging.getLogger() 27 | logger.setLevel(logging.INFO) 28 | 29 | try: 30 | event_type = event['type'] 31 | logger.info('Handling request for requestType: ' + event_type) 32 | 33 | if event_type == VALIDATE_CONNECTOR_RUNTIME_SETTINGS: 34 | request = requests.ValidateConnectorRuntimeSettingsRequest.from_dict(event) 35 | response = self.configuration_handler.validate_connector_runtime_settings(request) 36 | elif event_type == VALIDATE_CREDENTIALS: 37 | request = requests.ValidateCredentialsRequest.from_dict(event) 38 | response = self.configuration_handler.validate_credentials(request) 39 | elif event_type == DESCRIBE_CONNECTOR_CONFIGURATION: 40 | request = requests.DescribeConnectorConfigurationRequest.from_dict(event) 41 | response = self.configuration_handler.describe_connector_configuration(request) 42 | elif event_type == LIST_ENTITIES: 43 | request = requests.ListEntitiesRequest.from_dict(event) 44 | response = self.metadata_handler.list_entities(request) 45 | elif event_type == DESCRIBE_ENTITY: 46 | request = requests.DescribeEntityRequest.from_dict(event) 47 | response = self.metadata_handler.describe_entity(request) 48 | elif event_type == RETRIEVE_DATA: 49 | request = requests.RetrieveDataRequest.from_dict(event) 50 | response = self.record_handler.retrieve_data(request) 51 | elif event_type == WRITE_DATA: 52 | request = requests.WriteDataRequest.from_dict(event) 53 | response = self.record_handler.write_data(request) 54 | elif event_type == QUERY_DATA: 55 | request = requests.QueryDataRequest.from_dict(event) 56 | response = self.record_handler.query_data(request) 57 | else: 58 | logger.exception(f'lambda_handler: Request type {event_type} is not supported') 59 | raise ValueError('No operation is defined for request type: ' + event_type) 60 | 61 | return response.to_dict() 62 | except Exception: 63 | logger.exception('lambda_handler: Completed with an exception') 64 | raise RuntimeError('Exception while processing the request') 65 | -------------------------------------------------------------------------------- /custom_connector_tests/validation/connector_configuration_validator.py: -------------------------------------------------------------------------------- 1 | import re 2 | import custom_connector_sdk.lambda_handler.responses as responses 3 | import custom_connector_sdk.connector.auth as auth 4 | from custom_connector_tests.exceptions.ValidationException import ValidationException 5 | from urllib.parse import urlparse 6 | 7 | pattern = re.compile("([Aa][Ww][Ss]|[Aa][Mm][Aa][Zz][Oo][Nn]|[Aa][Pp][Pp][Ff][Ll][Oo][Ww]).*") 8 | reserved_keywords = ["aws", "amazon", "appflow"] 9 | 10 | def validate_connector_configuration_response(response): 11 | validation_errors = [] 12 | 13 | if check_for_reserved_keywords(response[responses.CONNECTOR_NAME]): 14 | validation_errors.append(f'ConnectorName should not contain these reserved keywords {reserved_keywords}') 15 | if check_for_reserved_keywords(response[responses.CONNECTOR_OWNER]): 16 | validation_errors.append(f'ConnectorOwner should not contain these reserved keywords {reserved_keywords}') 17 | 18 | if not response[responses.CONNECTOR_MODES]: 19 | validation_errors.append(f'ConnectorModes cannot be null for Connector') 20 | 21 | validation_errors += validate_authentication_config(response[responses.AUTHENTICATION_CONFIG]) 22 | 23 | if validation_errors: 24 | error_message = f'ConnectorConfiguration from the connector failed with following validation validationErrors. {validation_errors}' 25 | raise ValidationException(error_message) 26 | 27 | def check_for_reserved_keywords(input_string): 28 | return pattern.match(input_string) 29 | 30 | def validate_authentication_config(auth_config): 31 | validation_errors = [] 32 | 33 | if auth_config[auth.IS_CUSTOM_AUTH_SUPPORTED] and not auth_config[auth.CUSTOM_AUTH_CONFIG]: 34 | validation_errors.append("For custom Authentication, CustomAuthConfig is required.") 35 | if not auth_config[auth.IS_CUSTOM_AUTH_SUPPORTED] and auth_config[auth.CUSTOM_AUTH_CONFIG]: 36 | validation_errors.append("CustomAuthConfig can only be provided for CustomAuthentication.") 37 | if auth_config[auth.IS_O_AUTH_2_SUPPORTED] and not auth_config[auth.O_AUTH_2_DEFAULTS]: 38 | validation_errors.append("For OAuth2 Authentication, OAuth2Defaults cannot be null.") 39 | if not auth_config[auth.IS_O_AUTH_2_SUPPORTED] and auth_config[auth.O_AUTH_2_DEFAULTS]: 40 | validation_errors.append("OAuth2Defaults can only be provided for OAuth2 Authentication.") 41 | 42 | if auth_config[auth.IS_O_AUTH_2_SUPPORTED] and auth_config[auth.O_AUTH_2_DEFAULTS]: 43 | o_auth_defaults = auth_config[auth.O_AUTH_2_DEFAULTS] 44 | validation_errors.extend(validate_urls(o_auth_defaults[auth.LOGIN_URL]) + 45 | validate_urls(o_auth_defaults[auth.AUTH_URL]) + 46 | validate_urls(o_auth_defaults[auth.REFRESH_URL])) 47 | if o_auth_defaults[auth.REDIRECT_URL]: 48 | validation_errors += validate_urls(o_auth_defaults[auth.REDIRECT_URL]) 49 | 50 | return validation_errors 51 | 52 | def validate_urls(urls): 53 | validation_errors = [] 54 | 55 | for url in urls: 56 | try: 57 | validate(url) 58 | except ValidationException as e: 59 | validation_errors.append(f'Validation failed for url {url} with error {e}') 60 | 61 | return validation_errors 62 | 63 | def validate(url): 64 | if not url: 65 | return 66 | 67 | try: 68 | url_parsed = urlparse(url) 69 | if not "https".__eq__(url_parsed.scheme): 70 | error_message = f'Invalid protocol in url {url}. Only https format is supported' 71 | raise ValidationException(error_message) 72 | except ValueError: 73 | error_message = f'Invalid format for url {url}. Please check if the url syntax is correct and resolves to a known host' 74 | raise ValidationException(error_message) 75 | -------------------------------------------------------------------------------- /custom_connector_sdk/connector/settings.py: -------------------------------------------------------------------------------- 1 | from enum import Enum, auto 2 | from typing import List 3 | 4 | TIME_TO_LIVE = 'timeToLive' 5 | TIME_TO_LIVE_UNIT = 'timeToLiveUnit' 6 | KEY = 'key' 7 | DATA_TYPE = 'dataType' 8 | REQUIRED = 'required' 9 | LABEL = 'label' 10 | DESCRIPTION = 'description' 11 | SCOPE = 'scope' 12 | CONNECTOR_SUPPLIED_VALUE_OPTIONS = 'connectorSuppliedValueOptions' 13 | 14 | class ConnectorRuntimeSettingDataType(Enum): 15 | """Enum for connector runtime setting data type.""" 16 | String = auto() 17 | Date = auto() 18 | DateTime = auto() 19 | Long = auto() 20 | Integer = auto() 21 | Boolean = auto() 22 | 23 | class ConnectorRuntimeSettingScope(Enum): 24 | """Defines the scope for a given connector runtime setting. All connector runtime settings will be aggregated 25 | and will be sent along with every function invocation on the connector. 26 | 27 | """ 28 | # Settings to be populated during connector profile creation. 29 | CONNECTOR_PROFILE = auto() 30 | 31 | # Setting to be populated during a flow creation if the connector is chosen as a source connector. 32 | SOURCE = auto() 33 | 34 | # Setting to be populated during a flow creation if the connector is chosen as a destination connector. 35 | DESTINATION = auto() 36 | 37 | # Setting to be populated during a flow creation if the connector is chosen either as a source or a destination 38 | # connector. 39 | SOURCE_AND_DESTINATION = auto() 40 | 41 | class TimeUnit(Enum): 42 | """Enum of time units.""" 43 | NANOSECONDS = auto() 44 | MICROSECONDS = auto() 45 | MILLISECONDS = auto() 46 | SECONDS = auto() 47 | MINUTES = auto() 48 | HOURS = auto() 49 | DAYS = auto() 50 | 51 | class CacheControl: 52 | """Represents the caching policy for metadata for the supported entities.""" 53 | def __init__(self, time_to_live: int = None, time_to_live_unit: TimeUnit = None): 54 | # Time to keep the metadata in cache. 55 | # Return a large number when entity metadata is not dynamic and can 56 | # be cached for long time. The minimum allowed value is 600 seconds. 57 | self.time_to_live = time_to_live 58 | 59 | # TimeUnit for the time_to_live 60 | self.time_to_live_unit = time_to_live_unit 61 | 62 | def to_dict(self): 63 | return {TIME_TO_LIVE: self.time_to_live, 64 | TIME_TO_LIVE_UNIT: self.time_to_live_unit and self.time_to_live_unit.name} 65 | 66 | class ConnectorRuntimeSetting: 67 | """Represents the setting that the connector needs at runtime and the input will be provided by the AppFlow user. 68 | For eg. instanceUrl, maxParallelism, etc. 69 | 70 | """ 71 | def __init__(self, 72 | key: str, 73 | data_type: ConnectorRuntimeSettingDataType, 74 | required: bool, 75 | label: str, 76 | description: str, 77 | scope: ConnectorRuntimeSettingScope, 78 | connector_supplied_value_options: List[str] = None): 79 | # Unique identifier for the connector runtime setting 80 | self.key = key 81 | 82 | # Data type for the connector runtime setting. 83 | self.data_type = data_type 84 | 85 | # Specifies if this setting is required or not. 86 | self.required = required 87 | 88 | # Label for the connector runtime setting. 89 | self.label = label 90 | 91 | # Description of the connector runtime setting. 92 | self.description = description 93 | 94 | # Scope of the runtime setting needed for CONNECTOR_PROFILE, SOURCE, DESTINATION, etc. 95 | self.scope = scope 96 | 97 | # Optional connector supplied value options (with matching data type) that the user can pick from as a value 98 | # for this runtime setting. 99 | self.connector_supplied_value_options = connector_supplied_value_options 100 | 101 | def to_dict(self): 102 | return {KEY: self.key, 103 | DATA_TYPE: self.data_type.name, 104 | REQUIRED: self.required, 105 | LABEL: self.label, 106 | DESCRIPTION: self.description, 107 | SCOPE: self.scope.name, 108 | CONNECTOR_SUPPLIED_VALUE_OPTIONS: self.connector_supplied_value_options} 109 | -------------------------------------------------------------------------------- /custom_connector_sdk/lambda_handler/handlers.py: -------------------------------------------------------------------------------- 1 | import abc 2 | import custom_connector_sdk.lambda_handler.requests as requests 3 | import custom_connector_sdk.lambda_handler.responses as responses 4 | 5 | class MetadataHandler(metaclass=abc.ABCMeta): 6 | """This abstract base class defines the functionality to be implemented by custom connectors for metadata 7 | operations. 8 | 9 | """ 10 | @abc.abstractmethod 11 | def list_entities(self, request: requests.ListEntitiesRequest) -> responses.ListEntitiesResponse: 12 | """Lists all the entities available in a paginated fashion. This API is recursive in nature 13 | and provides a heretical entity listing based on entityPath. If the ListEntitiesResponse 14 | returns has_children=true, that indicates that there are more entities in the next level. 15 | 16 | Parameters: 17 | request (ListEntitiesRequest) 18 | 19 | Return: 20 | ListEntitiesResponse 21 | 22 | """ 23 | pass 24 | 25 | @abc.abstractmethod 26 | def describe_entity(self, request: requests.DescribeEntityRequest) -> responses.DescribeEntityResponse: 27 | """Describes the entity definition with its field level metadata. 28 | 29 | Parameters: 30 | request (DescribeEntityRequest) 31 | 32 | Return: 33 | DescribeEntityResponse 34 | 35 | """ 36 | pass 37 | 38 | class ConfigurationHandler(metaclass=abc.ABCMeta): 39 | """This abstract base class defines the functionality to be implemented by custom connectors for configurations, 40 | credentials related operations. 41 | 42 | """ 43 | @abc.abstractmethod 44 | def validate_connector_runtime_settings(self, request: requests.ValidateConnectorRuntimeSettingsRequest) -> \ 45 | responses.ValidateConnectorRuntimeSettingsResponse: 46 | """Validates the user inputs corresponding to the connector settings for a given ConnectorRuntimeSettingScope 47 | 48 | Parameters: 49 | request (ValidateConnectorRuntimeSettingsRequest) 50 | 51 | Return: 52 | ValidateConnectorRuntimeSettingsResponse 53 | 54 | """ 55 | pass 56 | 57 | @abc.abstractmethod 58 | def validate_credentials(self, request: requests.ValidateCredentialsRequest) -> \ 59 | responses.ValidateCredentialsResponse: 60 | """Validates the user provided credentials. 61 | 62 | Parameters: 63 | request (ValidateCredentialsRequest) 64 | 65 | Return: 66 | ValidateCredentialsResponse 67 | 68 | """ 69 | pass 70 | 71 | @abc.abstractmethod 72 | def describe_connector_configuration(self, request: requests.DescribeConnectorConfigurationRequest) -> \ 73 | responses.DescribeConnectorConfigurationResponse: 74 | """Describes the Connector Configuration supported by the connector. 75 | 76 | Parameters: 77 | request (DescribeConnectorConfigurationRequest) 78 | 79 | Return: 80 | DescribeConnectorConfigurationResponse 81 | 82 | """ 83 | pass 84 | 85 | class RecordHandler(metaclass=abc.ABCMeta): 86 | """This abstract base class defines the functionality to be implemented by custom connectors for record related 87 | operations. 88 | 89 | """ 90 | @abc.abstractmethod 91 | def retrieve_data(self, request: requests.RetrieveDataRequest) -> responses.RetrieveDataResponse: 92 | """Retrieves the batch of records against a set of identifiers from the source application. 93 | 94 | Parameters: 95 | request (RetrieveDataRequest) 96 | 97 | Return: 98 | RetrieveDataResponse 99 | """ 100 | pass 101 | 102 | @abc.abstractmethod 103 | def write_data(self, request: requests.WriteDataRequest) -> responses.WriteDataResponse: 104 | """Writes batch of records to the destination application. 105 | 106 | Parameters: 107 | request (WriteDataRequest) 108 | 109 | Return: 110 | WriteDataResponse 111 | """ 112 | pass 113 | 114 | @abc.abstractmethod 115 | def query_data(self, request: requests.QueryDataRequest) -> responses.QueryDataResponse: 116 | """Queries the data from the source application against the supplied filter conditions. 117 | 118 | Parameters: 119 | request (QueryDataRequest) 120 | 121 | Return: 122 | QueryDataResponse 123 | """ 124 | pass 125 | -------------------------------------------------------------------------------- /custom_connector_example/query/builder.py: -------------------------------------------------------------------------------- 1 | from typing import List, Tuple 2 | 3 | from custom_connector_sdk.connector.fields import FieldDataType 4 | from custom_connector_sdk.connector.context import EntityDefinition 5 | from custom_connector_queryfilter.queryfilter.parse_tree_builder import parse 6 | from custom_connector_example.query.visitor import SalesforceQueryFilterExpressionVisitor 7 | 8 | CONDITION_FORMAT = '{} {} {}' 9 | WHERE_AND = ' and ' 10 | CLAUSE_STRING_FORMAT = '{} {}' 11 | WHERE_CLAUSE = 'where' 12 | FROM_CLAUSE = 'from' 13 | SELECT_CLAUSE = 'select' 14 | LIMIT_CLAUSE = 'limit' 15 | 16 | 17 | class QueryObject: 18 | """Stores parameters to be built into a Salesforce query.""" 19 | def __init__(self, 20 | s_object: str, 21 | selected_field_names: List[str] = None, 22 | filter_expression: str = None, 23 | id_field_name: str = None, 24 | fields: List[str] = None, 25 | data_type: str = None, 26 | entity_definition: EntityDefinition = None): 27 | self.s_object = s_object 28 | self.selected_field_names = selected_field_names 29 | self.filter_expression = filter_expression 30 | self.id_field_name = id_field_name 31 | self.fields = fields 32 | self.data_type = data_type 33 | self.entity_definition = entity_definition 34 | 35 | 36 | def build_query(query_object: QueryObject) -> str: 37 | """Build Salesforce specific query given a QueryObject.""" 38 | if not query_object.selected_field_names: 39 | raise ValueError('No fields were selected for Salesforce Query') 40 | 41 | clauses = [] 42 | conditions = [] 43 | 44 | select_fields = ', '.join(query_object.selected_field_names) 45 | clauses.append(CLAUSE_STRING_FORMAT.format(SELECT_CLAUSE, select_fields)) 46 | clauses.append(CLAUSE_STRING_FORMAT.format(FROM_CLAUSE, query_object.s_object)) 47 | 48 | # QueryData allows data filtering based on filter expression. 49 | if query_object.filter_expression: 50 | where_clause, limit_clause = translate_filter_expression(query_object.filter_expression, query_object.entity_definition) 51 | if where_clause: 52 | clauses.append(CLAUSE_STRING_FORMAT.format(WHERE_CLAUSE, where_clause)) 53 | if limit_clause: 54 | clauses.append(CLAUSE_STRING_FORMAT.format(LIMIT_CLAUSE, limit_clause)) 55 | # RetrieveData allows data filtering based on entity primary ID fields 56 | elif query_object.id_field_name and query_object.fields and query_object.data_type: 57 | conditions = add_or_conditions('=', 58 | conditions, 59 | query_object.id_field_name, 60 | query_object.fields, 61 | query_object.data_type) 62 | if conditions: 63 | where_clause = WHERE_AND.join(conditions) 64 | clauses.append(CLAUSE_STRING_FORMAT.format(WHERE_CLAUSE, where_clause)) 65 | 66 | return ' '.join(clauses) 67 | 68 | 69 | def add_or_conditions(operator: str, 70 | conditions: List[str], 71 | variable: str, 72 | values: List[str], 73 | value_type: str) -> List[str]: 74 | """Joins clauses with 'or'""" 75 | or_conditions = [] 76 | for value in values: 77 | add_condition(operator, or_conditions, variable, value, value_type) 78 | 79 | condition = '(' + ' or '.join(or_conditions) + ')' 80 | conditions.append(condition) 81 | return conditions 82 | 83 | 84 | def add_condition(operator: str, 85 | conditions: List[str], 86 | field_name: str, 87 | value: str, 88 | value_type: str) -> List[str]: 89 | """Builds an individual condition.""" 90 | value = format_quotes(value_type, value) 91 | conditions.append(CONDITION_FORMAT.format(field_name, operator, value)) 92 | return conditions 93 | 94 | 95 | def format_quotes(value_type: str, value: str) -> str: 96 | """Escapes customer-added quotes, and changes/adds outer quotes as single quotes.""" 97 | customer_quoted = False 98 | 99 | if (value.startswith("'") and value.endswith("'")) or (value.startswith('"') and value.endswith('"')): 100 | value = value[1:-1] 101 | customer_quoted = True 102 | 103 | value.replace("'", "\\'") 104 | 105 | if value_type == FieldDataType.Struct.name or customer_quoted: 106 | return "'" + value + "'" 107 | return value 108 | 109 | 110 | def translate_filter_expression(filter_expression: str, entity_definition: EntityDefinition) -> Tuple[str, str]: 111 | """Converts filter expression to Salesforce specific expression using FilterExpressionVisitor.""" 112 | if filter_expression: 113 | parse_tree = parse(filter_expression) 114 | filter_expression_visitor = SalesforceQueryFilterExpressionVisitor(entity_definition) 115 | filter_expression_visitor.visit(parse_tree) 116 | return filter_expression_visitor.get_result() 117 | return '', '' 118 | -------------------------------------------------------------------------------- /TroubleShootingGuide.md: -------------------------------------------------------------------------------- 1 | ### TroubleShooting Guide 2 | 3 | ##### 1. OOM from the lambda: 4 | Since your lambda connector will perform memory/CPU bound operation you might experience OOM exception (java.lang.OutOfMemoryError). Please go to your lambda log group and check for the OutOfMemoryError exception. Below is the Sample Stack Trace: 5 | 6 | ```` 7 | START RequestId: b86c93c6-e1d0-11e7-955b-539d8b965ff9 Version: $LATEST 8 | REPORT RequestId: b86c93c6-e1d0-11e7-955b-539d8b965ff9 9 | Duration: 122204.28 ms Billed Duration: 122300 ms Memory Size: 256 MB Max Memory Used: 256 MB 10 | RequestId: b86c93c6-e1d0-11e7-955b-539d8b965ff9 Process exited before completing request 11 | ```` 12 | 13 | To resolve this issue , Please increase the Memory allocation of your lambda. To know more please read 14 | 15 | 16 | - https://docs.aws.amazon.com/lambda/latest/operatorguide/configurations.html#memory-config 17 | 18 | ##### 2. Exception while processing the request due to AccessDeniedException 19 | ```` 20 | Traceback (most recent call last): 21 | File "/var/task/custom_connector_sdk/lambda_handler/lambda_handler.py", line 38, in lambda_handler 22 | response = self.configuration_handler.validate_credentials(request) 23 | File "/var/task/custom_connector_example/handlers/configuration.py", line 39, in validate_credentials 24 | list_entities_response = SalesforceMetadataHandler().list_entities(list_entities_request) 25 | File "/var/task/custom_connector_example/handlers/metadata.py", line 142, in list_entities 26 | salesforce_response = salesforce.get_salesforce_client(request.connector_context).rest_get(request_uri) 27 | File "/var/task/custom_connector_example/handlers/salesforce.py", line 15, in get_salesforce_client 28 | return HttpsClient(get_access_token_from_secret(connector_context.credentials.secret_arn)) 29 | File "/var/task/custom_connector_example/handlers/salesforce.py", line 49, in get_access_token_from_secret 30 | secret = secrets_manager.get_secret_value(SecretId=secret_arn) 31 | File "/opt/python/botocore/client.py", line 388, in _api_call 32 | return self._make_api_call(operation_name, kwargs) 33 | File "/opt/python/botocore/client.py", line 708, in _make_api_call 34 | raise error_class(parsed_response, operation_name) 35 | botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::*******:assumed-role/python-memory-FunctionRole-1B6GH25FAONJ3/python-memory-Function-mQamETBP6j7L is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:*********:secret:appflow!*************-python-memory-python-memory-profile-Qdj0Lu because no identity-based 36 | policy allows the secretsmanager:GetSecretValue action 37 | ```` 38 | 39 | Your lambda fetches the credentials from the secret created in the secret manager in your AWS account.To fetch the credentials it uses getSecrets function on the secret. This type of error indicates that 40 | your lambda do not have permission on getSecret on your secrets. Please add the required permissions to your lambda. 41 | 42 | ##### 3. AccessDeniedException from Secrets even after providing the correct permissions to my connector lambda. 43 | 44 | This happens when you try to access your secret right after making the changes in the permissions. AWS Secrets Manager uses a distributed computing model called eventual consistency. Any change that you make in Secrets Manager (or other AWS services) takes time to become visible from all possible endpoints. Some of the delay results from the time it takes to send the data from server to server, from replication zone to replication zone, and from region to region around the world. 45 | 46 | Please read https://docs.aws.amazon.com/secretsmanager/latest/userguide/troubleshoot_general.html#troubleshoot_general_eventual-consistency for more details. 47 | 48 | 49 | ##### 4. ResourceNotFoundException from Secret Manager. 50 | 51 | Amazon AppFlow creates the secret in AppFlow user account and then pass the secret arn to the connector. AWS Secrets Manager uses a distributed computing model called eventual consistency. Any change that you make in Secrets Manager (or other AWS services) takes time to become visible from all possible endpoints. Some of the delay results from the time it takes to send the data from server to server, from replication zone to replication zone, and from region to region around the world. 52 | 53 | Please read https://docs.aws.amazon.com/secretsmanager/latest/userguide/troubleshoot_general.html#troubleshoot_general_eventual-consistency for more details. 54 | 55 | #### 5. How do I put logs or find logs 56 | AppFlow provides `flowName/connectorProfileLabel/executionId` to the `ConnectorContext`. Please use these string to query the log group for your lambda in the cloudwatch. Please Read more here : https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html for more advanced cloudwatch queries. 57 | 58 | #### 6. Resolving issues becuase of connector lambda cold start. 59 | This can happen when connector lambda is being invoked after a long period of inactivity. This cold start can result in timeout issues. Please follow the below links to know more on how to avoid such issues: 60 | https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/ 61 | https://aws.amazon.com/blogs/compute/new-for-aws-lambda-predictable-start-up-times-with-provisioned-concurrency/ 62 | 63 | 64 | 65 | -------------------------------------------------------------------------------- /custom_connector_integ_test/sample-test-config.json: -------------------------------------------------------------------------------- 1 | { 2 | "resourcePrefix": "string", //Optional: Name that will be added to all resources created by integ test. 3 | "customConnectorConfigurations": [ // Required: List of connectors to created at the begining of the test run. 4 | { 5 | // Optional: This file is identical to the response provided by describe connector api. 6 | // We use this file to validate the api response. 7 | "validationFileName": "string", 8 | "name":"string", // Required: This name can be used in the profile configuration. 9 | "lambdaArn":"string" // Required: The arn of the lambda deployed in the users account. 10 | } 11 | ], 12 | "customConnectorProfileConfigurations": [ // Required: List of profiles to created at the begining of the test run. 13 | { 14 | "connectorName":"string", // Optional: Name of connector created above. Otherwise the first connector in the list. 15 | "name":"string", // Required: This name can referenced in other tests. 16 | "profileProperties":{ // Optional: The connecor specific properties used when creating the profile. 17 | 18 | }, 19 | "defaultApiVersion": "string", // Optional: An Api version that will be used with other test cases that use this profile. 20 | "authenticationType":"OAUTH2|BASIC|API_KEY|CUSTOM|NO_AUTH", // Required: the auth type used by the profile 21 | "oAuth2Properties": { // Optional: Required if authenticationType is OAUTH 2. 22 | 23 | // OAuth properties needed by the connector 24 | "oAuth2GrantType":"string", 25 | "tokenUrl":"string" 26 | }, 27 | "secretsManagerArn":"string" // Optional: Arn to the secrets manager secret where secrets are stored. 28 | } 29 | ], 30 | "testBucketConfiguration": // Required: Bucket information that will be used when running tests. 31 | { 32 | "bucketName":"string", // Required: Bucket name in the users account. 33 | "bucketPrefix":"string" // Required: Bucket Prefix. 34 | }, 35 | "listConnectorEntitiesTestConfigurations": [ // Test for the list connector 36 | { 37 | // Optional: This file is identical to the response provided by list connector entities api. 38 | // However, not all entities need to be provided. The test case will only evaluate equality between entities in validation file. 39 | "validationFileName": "string", 40 | "profileName":"string", // Optional: Profile used to run the test. Uses first profile otherwise. 41 | "apiVersion": "string", // Optional: Api version used in request input. Uses default api version from profile otherwise. 42 | "testName": "string", // Optional: Test name used to associate the test report with this test case. 43 | "entitiesPath": "string" // Optional: Paramater used in list entities request. 44 | } 45 | ], 46 | "describeConnectorEntityTestConfigurations":[ 47 | { 48 | // Optional: This file is identical to the response provided by describe connector entity api. 49 | // However, not all fields need to be provided. The test case will only evaluate equality between the fields in validation file. 50 | "validationFileName": "string", 51 | "profileName":"string", // Optional: Profile used to run the test. Uses first profile otherwise. 52 | "apiVersion": "string", // Optional: Api version used in request input. Uses default api version from profile otherwise. 53 | "testName": "string", // Optional: Test name used to associate the test report with this test case. 54 | "entitiesName": "string" // Required: Entity in connector. 55 | } 56 | ], 57 | "onDemandFromS3TestConfigurations": [ 58 | { 59 | "testName":"string", // Optional: Test name used to associate the test report with this test case. 60 | "profileName":"string", // Optional: Profile used to run the test. Uses first profile otherwise. 61 | "idFieldNames": ["string"], // Optional: Input is required for non-insert write operation. 62 | "flowName": "string", // Required: Name of flow. 63 | "apiVersion": "string", // Optional: Api version used in request input. Uses default api version from profile otherwise. 64 | "entityName":"string", // Required: Entity name that we will be creating in the connector. 65 | "sourceDataFile":"string", // Required: Entity name that we will be creating in the connector. 66 | "dataGeneratorClassName":"string", // Required: Entity name that we will be creating in the connector. 67 | "destinationRuntimeProperties": { // Optional: Connector Specific Properties. 68 | "key": "value" 69 | }, 70 | "flowTimeout":"Integer" //Maximum amount of time to run the flow before timing out. 71 | } 72 | ], 73 | "onDemandToS3TestConfigurations": [ 74 | { 75 | "testName":"string", // Optional: Test name used to associate the test report with this test case. 76 | "profileName":"string", // Optional: Profile used to run the test. Uses first profile otherwise. 77 | "flowName": "string", // Required: Name of flow. 78 | "entityName":"string", // Required: Entity name that we will be creating in the connector. 79 | "apiVersion": "string", // Optional: Api version used in request input. Uses default api version from profile otherwise. 80 | "query":"string", // Optional: Filter expression used with flow to test filter capability 81 | "flowTimeout":"number" // Maximum amount of time to run the flow before timing out. 82 | "entityFields":["LastActivityDate"], // Required: Fields that need to be retrievedr 83 | "outputSize":"number", // Optional: Output size of the flow. Used to validate that correct data was retrieved. 84 | "sourceRuntimeProperties": { // Optional: Connector Specific Properties. 85 | "key": "value" 86 | } 87 | } 88 | ] 89 | } -------------------------------------------------------------------------------- /custom_connector_integ_test/README.md: -------------------------------------------------------------------------------- 1 | # Integration Test FrameWork 2 | This document explains how to run the integration tests provided by AppFlow for a custom connector. 3 | 4 | ## Prerequisites: 5 | 6 | - python 7 | 8 | ## Test Case Overview: 9 | 10 | The framework provides the following test cases. 11 | 12 | 1. Register and Describing the connector. 13 | 2. Calling the metadata apis. 14 | 3. Running a flow from s3 -> connector 15 | 4. Running a flow from the connector -> s3 16 | 17 | Additionally, we provide a test class that cleans up all AppFlow resources created by the integration tests. 18 | This test case cleans up resources based on the prefix "Integ_{ResourceType}" e.g. Integ_Profile. 19 | You can also provide a prefix in the test configuration which will be appended to all resources. 20 | If you are running multiple test cases in the same aws account, be sure to use this prefix. Otherwise, 21 | one test run might delete resources deployed by another test run. 22 | 23 | All tests are written using unittest. 24 | 25 | ## Configuration File Overview 26 | 27 | The test cases are driven by a set of configuration files. The main configuration file location is set 28 | as an env variable. 29 | 30 | The test configuration allows you to specifiy any number of custom connectors and connector profiles. 31 | These resources are given a name. This name will be used as part of a randomly 32 | generated name. All tests cases take as input either a connector name or a connector profile name 33 | as an optinal parameter. If this name is not provided, then the first profile in the configuration 34 | list will be used. To make the configuration simple, it may be worth specifying one profile and 35 | connector per test file. Instead, you can use multiple configuration files. 36 | 37 | You are also free to include as many connectors as you want in one configuration file. In general, anything that 38 | can be done in multiple test configuration files, can be done in a single file. 39 | 40 | The configuration file is in json. It includes a json list for each test case. Each json object in the list 41 | is a test case. 42 | e.g. 43 | 44 | ``` 45 | "listConnectorEntitiesTestConfigurations": [ // Test for the list connector 46 | { 47 | // Optional: This file is identical to the response provided by list connector entities api. 48 | // However, not all entities need to be provided. The test case will only evaluate equality between entities in validation file. 49 | "validationFileName": "string", 50 | "profileName":"string", // Optional: Profile used to run the test. Uses first profile otherwise. 51 | "apiVersion": "string", // Optional: Api version used in request input. Uses default api version from profile otherwise. 52 | "testName": "string", // Optional: Test name used to associate the test report with this test case. 53 | "entitiesPath": "string" // Optional: Paramater used in list entities request. 54 | } 55 | ], 56 | ``` 57 | 58 | There is a test case called listConnectorEntitiesTest. 59 | This test takes in the following parameters shown in the sample-test-config.json 60 | You can specify multiple test cases per test by adding another json payload to the list. 61 | Alternatively, an empty list won't run any test cases for that test. 62 | 63 | An example set of configuration files can be found in the salesforce custom connector example. 64 | 65 | A base configuration file can be found in the current directory. 66 | 67 | 68 | ## Running the tests 69 | ###Resources 70 | The configuration file requires several aws resources to already exist in your account. The purpose of these 71 | resources is described in the sample-test-config.json file. 72 | 1. A S3 bucket with AppFlow bucket policy. 73 | 2. A Secrets Manager secret for each set of credentials. 74 | 3. A Lambda custom connector. 75 | 76 | Note: The integration tests will not delete files that are created during the test run. 77 | The test bucket you're using, should have an object expiration for the specific prefix used by the test cases. 78 | 79 | ###Environment Configuration 80 | The integration tests rely on both the AWS_DEFAULT_REGION environment variable, a TEST_CONFIG env variable 81 | and aws credentials environment variables. 82 | 83 | ``` 84 | TEST_CONFIG=custom_connector_example/salesforce-example-test-files/test-file.json; 85 | AWS_DEFAULT_REGION=us-west-2; 86 | AWS_ACCESS_KEY_ID=keyId; 87 | AWS_SECRET_ACCESS_KEY=key; 88 | ``` 89 | 90 | ###Running the test 91 | The test class can be run directly or subclassed. 92 | For example, in the salesforce example connector we sub-class the test class. 93 | ``` 94 | class SalesTest(AppflowTestCase): 95 | pass 96 | ``` 97 | The tests can then be run with the following command. 98 | ``` 99 | python -m unittest -v custom_connector_example.integ_test.sales_test_case.SalesTest 100 | ``` 101 | 102 | 103 | ##Writing your own test cases 104 | You can extend the class AppflowTestCase if you want to write your test cases using the utility 105 | classes used by the integration tests. 106 | 107 | For example, you can use the ResourceInfoProvider to get the name for a connector or profile generated during the test run. 108 | If the resource failed to create then this method will raise a skip exception. 109 | 110 | You can also use ServiceProvider class to get the amazon sdk clients used by the integration tests as well as a FlowPoller 111 | class which you can use to poll flow executions. 112 | 113 | Tests in unittest run in alphabetical order. Be sure to name your test so that it happens after the connector is 114 | registered and before the profile is created. 115 | 116 | 117 | This ensures that the AppFlow test cases always run, run after the connectors and connector profiles are created, 118 | and run before all resources are deleted. -------------------------------------------------------------------------------- /custom_connector_example/test/metadata_test.py: -------------------------------------------------------------------------------- 1 | import json 2 | import unittest 3 | from unittest.mock import patch 4 | import custom_connector_example.handlers.lambda_handler as lambda_handler 5 | from custom_connector_example.handlers.salesforce import SalesforceResponse 6 | 7 | class SalesforceMetadataHandlerTests(unittest.TestCase): 8 | """Test class to validate handling of Metadata requests by Salesforce connector.""" 9 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 10 | def test_list_entities_request_success(self, mock): 11 | with open('custom_connector_example/test/resources/get_sobjects_server_response.json', 'r') as json_response: 12 | server_response = json_response.read() 13 | mock.return_value = SalesforceResponse(200, server_response, '') 14 | 15 | with open('custom_connector_example/test/resources/list_entities_request_valid.json', 'r') as json_file: 16 | data = json.load(json_file) 17 | 18 | response = lambda_handler.salesforce_lambda_handler(data, None) 19 | self.assertEqual(response.get('isSuccess'), True) 20 | 21 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 22 | def test_list_entities_request_no_path(self, mock): 23 | with open('custom_connector_example/test/resources/get_sobjects_server_response.json', 'r') as json_response: 24 | server_response = json_response.read() 25 | mock.return_value = SalesforceResponse(200, server_response, '') 26 | 27 | with open('custom_connector_example/test/resources/list_entities_request_no_path.json', 'r') as json_file: 28 | data = json.load(json_file) 29 | 30 | response = lambda_handler.salesforce_lambda_handler(data, None) 31 | self.assertEqual(response.get('isSuccess'), True) 32 | 33 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 34 | def test_list_entities_request_no_sobjects(self, mock): 35 | with open('custom_connector_example/test/resources/null_sobjects_server_response.json', 'r') as json_response: 36 | server_response = json_response.read() 37 | mock.return_value = SalesforceResponse(200, server_response, '') 38 | 39 | with open('custom_connector_example/test/resources/list_entities_request_no_path.json', 'r') as json_file: 40 | data = json.load(json_file) 41 | 42 | response = lambda_handler.salesforce_lambda_handler(data, None) 43 | self.assertEqual(response.get('isSuccess'), True) 44 | 45 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 46 | def test_list_entities_request_server_error(self, mock): 47 | mock.return_value = SalesforceResponse(500, '', 'Internal server error') 48 | 49 | with open('custom_connector_example/test/resources/list_entities_request_valid.json', 'r') as json_file: 50 | data = json.load(json_file) 51 | 52 | response = lambda_handler.salesforce_lambda_handler(data, None) 53 | self.assertEqual(response.get('isSuccess'), False) 54 | 55 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 56 | def test_list_entities_request_invalid_context(self, mock): 57 | with open('custom_connector_example/test/resources/get_sobjects_server_response.json', 'r') as json_response: 58 | server_response = json_response.read() 59 | mock.return_value = SalesforceResponse(200, server_response, '') 60 | 61 | with open('custom_connector_example/test/resources/list_entities_request_invalid.json', 'r') as json_file: 62 | data = json.load(json_file) 63 | 64 | response = lambda_handler.salesforce_lambda_handler(data, None) 65 | self.assertEqual(response.get('isSuccess'), False) 66 | 67 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 68 | def test_describe_entity_request_success(self, mock): 69 | with open('custom_connector_example/test/resources/describe_sobject_server_response.json', 70 | 'r') as json_response: 71 | server_response = json_response.read() 72 | mock.return_value = SalesforceResponse(200, server_response, '') 73 | 74 | with open('custom_connector_example/test/resources/describe_entity_request_valid.json', 'r') as json_file: 75 | data = json.load(json_file) 76 | 77 | response = lambda_handler.salesforce_lambda_handler(data, None) 78 | self.assertEqual(response.get('isSuccess'), True) 79 | 80 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 81 | def test_describe_entity_request_server_error(self, mock): 82 | mock.return_value = SalesforceResponse(500, '{}', 'Internal server error') 83 | 84 | with open('custom_connector_example/test/resources/describe_entity_request_valid.json', 'r') as json_file: 85 | data = json.load(json_file) 86 | 87 | response = lambda_handler.salesforce_lambda_handler(data, None) 88 | self.assertEqual(response.get('isSuccess'), False) 89 | 90 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 91 | def test_describe_entity_request_invalid_context(self, mock): 92 | with open('custom_connector_example/test/resources/describe_sobject_server_response.json', 93 | 'r') as json_response: 94 | server_response = json_response.read() 95 | mock.return_value = SalesforceResponse(200, server_response, '') 96 | 97 | with open('custom_connector_example/test/resources/describe_entity_request_invalid.json', 'r') as json_file: 98 | data = json.load(json_file) 99 | 100 | response = lambda_handler.salesforce_lambda_handler(data, None) 101 | self.assertEqual(response.get('isSuccess'), False) 102 | -------------------------------------------------------------------------------- /custom_connector_sdk/connector/context.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | 3 | from custom_connector_sdk.connector.fields import FieldDefinition 4 | from custom_connector_sdk.connector.auth import Credentials 5 | 6 | ENTITY_IDENTIFIER = 'entityIdentifier' 7 | HAS_NESTED_ENTITIES = 'hasNestedEntities' 8 | IS_WRITABLE = 'isWritable' 9 | LABEL = 'label' 10 | DESCRIPTION = 'description' 11 | ENTITY = 'entity' 12 | FIELDS = 'fields' 13 | CUSTOM_PROPERTIES = 'customProperties' 14 | CREDENTIALS = 'credentials' 15 | API_VERSION = 'apiVersion' 16 | CONNECTOR_RUNTIME_SETTINGS = 'connectorRuntimeSettings' 17 | ENTITY_DEFINITION = 'entityDefinition' 18 | 19 | class Entity: 20 | """Represents the entity structure.""" 21 | def __init__(self, 22 | entity_identifier: str, 23 | has_nested_entities: bool, 24 | is_writable: bool, 25 | label: str = None, 26 | description: str = None): 27 | # Unique identifier for the entity. Can be entityId, entityName, entityPath+name, entityUrl, etc. 28 | self.entity_identifier = entity_identifier 29 | 30 | # Specifies whether the connector entity is a parent or a category and has more entities nested underneath it. 31 | self.has_nested_entities = has_nested_entities 32 | 33 | # Specifies if entity is writable 34 | self.is_writable = is_writable 35 | 36 | # Label of the entity. 37 | self.label = label 38 | 39 | # Description of the entity. 40 | self.description = description 41 | 42 | def to_dict(self): 43 | return {ENTITY_IDENTIFIER: self.entity_identifier, 44 | HAS_NESTED_ENTITIES: self.has_nested_entities, 45 | IS_WRITABLE: self.is_writable, 46 | LABEL: self.label, 47 | DESCRIPTION: self.description} 48 | 49 | @classmethod 50 | def from_dict(cls, entity: dict): 51 | if entity is None: 52 | return None 53 | 54 | required_keys = {ENTITY_IDENTIFIER, HAS_NESTED_ENTITIES} 55 | assert entity.keys() >= required_keys, f'{cls.__name__} is missing required parameters {required_keys}' 56 | 57 | entity_identifier = entity.get(ENTITY_IDENTIFIER) 58 | has_nested_entities = entity.get(HAS_NESTED_ENTITIES) 59 | is_writable = entity.get(IS_WRITABLE) 60 | label = entity.get(LABEL) 61 | description = entity.get(DESCRIPTION) 62 | 63 | return cls(entity_identifier, has_nested_entities, is_writable, label, description) 64 | 65 | class EntityDefinition: 66 | """Data model of the Entity.""" 67 | def __init__(self, entity: Entity, fields: List[FieldDefinition], custom_properties: dict = None): 68 | # Contains its name, description, label or if it has child properties or not. 69 | self.entity = entity 70 | 71 | # List of data models of the fields an Entity has. 72 | self.fields = fields 73 | 74 | # Custom properties defined for an Entity (str -> object) 75 | self.custom_properties = custom_properties 76 | 77 | @classmethod 78 | def from_dict(cls, definition: dict): 79 | if definition is None: 80 | return None 81 | 82 | required_keys = {ENTITY, FIELDS} 83 | assert definition.keys() >= required_keys, f'{cls.__name__} is missing required parameters {required_keys}' 84 | 85 | entity = Entity.from_dict(definition.get(ENTITY)) 86 | fields = [FieldDefinition.from_dict(field) for field in definition.get(FIELDS)] 87 | custom_properties = definition.get(CUSTOM_PROPERTIES) 88 | 89 | return cls(entity, fields, custom_properties) 90 | 91 | def to_dict(self): 92 | return {ENTITY: self.entity.to_dict(), 93 | FIELDS: [field.to_dict() for field in self.fields], 94 | CUSTOM_PROPERTIES: self.custom_properties} 95 | 96 | class ConnectorContext: 97 | """Represents the Connector Context which contains the connector runtime settings, credentials, api version, and 98 | entity metadata. 99 | 100 | """ 101 | def __init__(self, 102 | api_version: str, 103 | connector_runtime_settings: dict = None, 104 | entity_definition: EntityDefinition = None, 105 | credentials: Credentials = None): 106 | # Credentials which will be used to make API call 107 | self.credentials = credentials 108 | 109 | # API version to use. Value will be the API Version supported by Connector as part of Connector Configuration. 110 | self.api_version = api_version 111 | 112 | # Connector settings required for API call. For example, for the Read API it will contain all the 113 | # ConnectorSettingScope.SOURCE settings. Key will be Connector Setting (str) and value will be the input 114 | # provided by the user (object). 115 | self.connector_runtime_settings = connector_runtime_settings 116 | 117 | # Entity definition in compressed form, as it will be required by calling application as well as connector 118 | # metadata to serialize/deserialize request/response payload. 119 | self.entity_definition = entity_definition 120 | 121 | @classmethod 122 | def from_dict(cls, context: dict): 123 | if context is None: 124 | return None 125 | 126 | required_keys = {API_VERSION} 127 | assert context.keys() >= required_keys, f'{cls.__name__} is missing required parameters {required_keys}' 128 | 129 | credentials = Credentials.from_dict(context.get(CREDENTIALS)) 130 | api_version = context.get(API_VERSION) 131 | connector_runtime_settings = context.get(CONNECTOR_RUNTIME_SETTINGS) 132 | entity_definition = EntityDefinition.from_dict(context.get(ENTITY_DEFINITION)) 133 | 134 | return cls( 135 | credentials=credentials, 136 | api_version=api_version, 137 | connector_runtime_settings=connector_runtime_settings, 138 | entity_definition=entity_definition) 139 | -------------------------------------------------------------------------------- /custom_connector_queryfilter/queryfilter/antlr/CustomConnectorQueryFilterParser.interp: -------------------------------------------------------------------------------- 1 | token literal names: 2 | null 3 | null 4 | null 5 | null 6 | null 7 | null 8 | '>' 9 | '>=' 10 | '<' 11 | '<=' 12 | '=' 13 | '!=' 14 | null 15 | null 16 | '(' 17 | ')' 18 | 'null' 19 | null 20 | null 21 | ',' 22 | null 23 | null 24 | null 25 | null 26 | null 27 | null 28 | null 29 | null 30 | null 31 | null 32 | 33 | token symbolic names: 34 | null 35 | AND 36 | OR 37 | NOT 38 | TRUE 39 | FALSE 40 | GT 41 | GE 42 | LT 43 | LE 44 | EQ 45 | NE 46 | LIKE 47 | BETWEEN 48 | LPAREN 49 | RPAREN 50 | NULL 51 | IN 52 | LIMIT 53 | COMMA 54 | IDENTIFIER 55 | POS_INTEGER 56 | DECIMAL 57 | SINGLE_STRING 58 | DOUBLE_STRING 59 | EMPTY_SINGLE_STRING 60 | EMPTY_DOUBLE_STRING 61 | WS 62 | DATE 63 | DATETIME 64 | 65 | rule names: 66 | queryfilter 67 | limitexpression 68 | expression 69 | gtComparator 70 | geComparator 71 | ltComparator 72 | leComparator 73 | eqComparator 74 | neComparator 75 | likeComparator 76 | betweenComparator 77 | andBinary 78 | orBinary 79 | boolean 80 | identifier 81 | inOperator 82 | limit 83 | string 84 | value 85 | count 86 | 87 | 88 | atn: 89 | [3, 24715, 42794, 33075, 47597, 16764, 15335, 30598, 22884, 3, 31, 178, 4, 2, 9, 2, 4, 3, 9, 3, 4, 4, 9, 4, 4, 5, 9, 5, 4, 6, 9, 6, 4, 7, 9, 7, 4, 8, 9, 8, 4, 9, 9, 9, 4, 10, 9, 10, 4, 11, 9, 11, 4, 12, 9, 12, 4, 13, 9, 13, 4, 14, 9, 14, 4, 15, 9, 15, 4, 16, 9, 16, 4, 17, 9, 17, 4, 18, 9, 18, 4, 19, 9, 19, 4, 20, 9, 20, 4, 21, 9, 21, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 5, 2, 49, 10, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 5, 3, 58, 10, 3, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 7, 4, 117, 10, 4, 12, 4, 14, 4, 120, 11, 4, 3, 4, 3, 4, 5, 4, 124, 10, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 3, 4, 7, 4, 134, 10, 4, 12, 4, 14, 4, 137, 11, 4, 3, 5, 3, 5, 3, 6, 3, 6, 3, 7, 3, 7, 3, 8, 3, 8, 3, 9, 3, 9, 3, 10, 3, 10, 3, 11, 3, 11, 3, 12, 3, 12, 3, 13, 3, 13, 3, 14, 3, 14, 3, 15, 3, 15, 3, 16, 3, 16, 3, 17, 3, 17, 3, 18, 3, 18, 3, 19, 3, 19, 3, 20, 3, 20, 3, 20, 3, 20, 3, 20, 5, 20, 174, 10, 20, 3, 21, 3, 21, 3, 21, 2, 3, 6, 22, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 2, 4, 3, 2, 6, 7, 4, 2, 18, 18, 25, 28, 2, 180, 2, 48, 3, 2, 2, 2, 4, 57, 3, 2, 2, 2, 6, 123, 3, 2, 2, 2, 8, 138, 3, 2, 2, 2, 10, 140, 3, 2, 2, 2, 12, 142, 3, 2, 2, 2, 14, 144, 3, 2, 2, 2, 16, 146, 3, 2, 2, 2, 18, 148, 3, 2, 2, 2, 20, 150, 3, 2, 2, 2, 22, 152, 3, 2, 2, 2, 24, 154, 3, 2, 2, 2, 26, 156, 3, 2, 2, 2, 28, 158, 3, 2, 2, 2, 30, 160, 3, 2, 2, 2, 32, 162, 3, 2, 2, 2, 34, 164, 3, 2, 2, 2, 36, 166, 3, 2, 2, 2, 38, 173, 3, 2, 2, 2, 40, 175, 3, 2, 2, 2, 42, 43, 5, 6, 4, 2, 43, 44, 7, 2, 2, 3, 44, 49, 3, 2, 2, 2, 45, 46, 5, 4, 3, 2, 46, 47, 7, 2, 2, 3, 47, 49, 3, 2, 2, 2, 48, 42, 3, 2, 2, 2, 48, 45, 3, 2, 2, 2, 49, 3, 3, 2, 2, 2, 50, 51, 5, 34, 18, 2, 51, 52, 5, 40, 21, 2, 52, 58, 3, 2, 2, 2, 53, 54, 5, 6, 4, 2, 54, 55, 5, 34, 18, 2, 55, 56, 5, 40, 21, 2, 56, 58, 3, 2, 2, 2, 57, 50, 3, 2, 2, 2, 57, 53, 3, 2, 2, 2, 58, 5, 3, 2, 2, 2, 59, 60, 8, 4, 1, 2, 60, 61, 7, 16, 2, 2, 61, 62, 5, 6, 4, 2, 62, 63, 7, 17, 2, 2, 63, 124, 3, 2, 2, 2, 64, 65, 7, 5, 2, 2, 65, 124, 5, 6, 4, 18, 66, 67, 5, 30, 16, 2, 67, 68, 5, 8, 5, 2, 68, 69, 5, 38, 20, 2, 69, 124, 3, 2, 2, 2, 70, 71, 5, 30, 16, 2, 71, 72, 5, 10, 6, 2, 72, 73, 5, 38, 20, 2, 73, 124, 3, 2, 2, 2, 74, 75, 5, 30, 16, 2, 75, 76, 5, 12, 7, 2, 76, 77, 5, 38, 20, 2, 77, 124, 3, 2, 2, 2, 78, 79, 5, 30, 16, 2, 79, 80, 5, 14, 8, 2, 80, 81, 5, 38, 20, 2, 81, 124, 3, 2, 2, 2, 82, 83, 5, 30, 16, 2, 83, 84, 5, 16, 9, 2, 84, 85, 5, 38, 20, 2, 85, 124, 3, 2, 2, 2, 86, 87, 5, 30, 16, 2, 87, 88, 5, 16, 9, 2, 88, 89, 5, 28, 15, 2, 89, 124, 3, 2, 2, 2, 90, 91, 5, 30, 16, 2, 91, 92, 5, 18, 10, 2, 92, 93, 5, 38, 20, 2, 93, 124, 3, 2, 2, 2, 94, 95, 5, 30, 16, 2, 95, 96, 5, 18, 10, 2, 96, 97, 5, 28, 15, 2, 97, 124, 3, 2, 2, 2, 98, 99, 5, 30, 16, 2, 99, 100, 5, 20, 11, 2, 100, 101, 5, 38, 20, 2, 101, 124, 3, 2, 2, 2, 102, 103, 5, 30, 16, 2, 103, 104, 5, 22, 12, 2, 104, 105, 5, 38, 20, 2, 105, 106, 5, 24, 13, 2, 106, 107, 5, 38, 20, 2, 107, 124, 3, 2, 2, 2, 108, 124, 5, 30, 16, 2, 109, 124, 5, 38, 20, 2, 110, 111, 5, 30, 16, 2, 111, 112, 5, 32, 17, 2, 112, 113, 7, 16, 2, 2, 113, 118, 5, 38, 20, 2, 114, 115, 7, 21, 2, 2, 115, 117, 5, 38, 20, 2, 116, 114, 3, 2, 2, 2, 117, 120, 3, 2, 2, 2, 118, 116, 3, 2, 2, 2, 118, 119, 3, 2, 2, 2, 119, 121, 3, 2, 2, 2, 120, 118, 3, 2, 2, 2, 121, 122, 7, 17, 2, 2, 122, 124, 3, 2, 2, 2, 123, 59, 3, 2, 2, 2, 123, 64, 3, 2, 2, 2, 123, 66, 3, 2, 2, 2, 123, 70, 3, 2, 2, 2, 123, 74, 3, 2, 2, 2, 123, 78, 3, 2, 2, 2, 123, 82, 3, 2, 2, 2, 123, 86, 3, 2, 2, 2, 123, 90, 3, 2, 2, 2, 123, 94, 3, 2, 2, 2, 123, 98, 3, 2, 2, 2, 123, 102, 3, 2, 2, 2, 123, 108, 3, 2, 2, 2, 123, 109, 3, 2, 2, 2, 123, 110, 3, 2, 2, 2, 124, 135, 3, 2, 2, 2, 125, 126, 12, 17, 2, 2, 126, 127, 5, 24, 13, 2, 127, 128, 5, 6, 4, 18, 128, 134, 3, 2, 2, 2, 129, 130, 12, 16, 2, 2, 130, 131, 5, 26, 14, 2, 131, 132, 5, 6, 4, 17, 132, 134, 3, 2, 2, 2, 133, 125, 3, 2, 2, 2, 133, 129, 3, 2, 2, 2, 134, 137, 3, 2, 2, 2, 135, 133, 3, 2, 2, 2, 135, 136, 3, 2, 2, 2, 136, 7, 3, 2, 2, 2, 137, 135, 3, 2, 2, 2, 138, 139, 7, 8, 2, 2, 139, 9, 3, 2, 2, 2, 140, 141, 7, 9, 2, 2, 141, 11, 3, 2, 2, 2, 142, 143, 7, 10, 2, 2, 143, 13, 3, 2, 2, 2, 144, 145, 7, 11, 2, 2, 145, 15, 3, 2, 2, 2, 146, 147, 7, 12, 2, 2, 147, 17, 3, 2, 2, 2, 148, 149, 7, 13, 2, 2, 149, 19, 3, 2, 2, 2, 150, 151, 7, 14, 2, 2, 151, 21, 3, 2, 2, 2, 152, 153, 7, 15, 2, 2, 153, 23, 3, 2, 2, 2, 154, 155, 7, 3, 2, 2, 155, 25, 3, 2, 2, 2, 156, 157, 7, 4, 2, 2, 157, 27, 3, 2, 2, 2, 158, 159, 9, 2, 2, 2, 159, 29, 3, 2, 2, 2, 160, 161, 7, 22, 2, 2, 161, 31, 3, 2, 2, 2, 162, 163, 7, 19, 2, 2, 163, 33, 3, 2, 2, 2, 164, 165, 7, 20, 2, 2, 165, 35, 3, 2, 2, 2, 166, 167, 9, 3, 2, 2, 167, 37, 3, 2, 2, 2, 168, 174, 5, 36, 19, 2, 169, 174, 7, 23, 2, 2, 170, 174, 7, 24, 2, 2, 171, 174, 7, 30, 2, 2, 172, 174, 7, 31, 2, 2, 173, 168, 3, 2, 2, 2, 173, 169, 3, 2, 2, 2, 173, 170, 3, 2, 2, 2, 173, 171, 3, 2, 2, 2, 173, 172, 3, 2, 2, 2, 174, 39, 3, 2, 2, 2, 175, 176, 7, 23, 2, 2, 176, 41, 3, 2, 2, 2, 9, 48, 57, 118, 123, 133, 135, 173] -------------------------------------------------------------------------------- /custom_connector_queryfilter/tests/expression_visitor_test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from custom_connector_queryfilter.queryfilter.parse_tree_builder import parse 3 | from custom_connector_queryfilter.queryfilter.antlr.CustomConnectorQueryFilterParserVisitor import CustomConnectorQueryFilterParserVisitor 4 | 5 | 6 | class TestVisitor(CustomConnectorQueryFilterParserVisitor): 7 | def __init__(self): 8 | self.count_of_expressions_visited = 0 9 | 10 | def visit(self, tree): 11 | self.count_of_expressions_visited = 0 12 | super().visit(tree) 13 | return self.count_of_expressions_visited 14 | 15 | def visitLesserThanComparatorExpression(self, ctx): 16 | self.count_of_expressions_visited += 1 17 | return super().visitLesserThanComparatorExpression(ctx) 18 | 19 | def visitGreaterThanComparatorExpression(self, ctx): 20 | self.count_of_expressions_visited += 1 21 | return super().visitGreaterThanComparatorExpression(ctx) 22 | 23 | def visitValueExpression(self, ctx): 24 | self.count_of_expressions_visited += 1 25 | return super().visitValueExpression(ctx) 26 | 27 | def visitIdentifierExpression(self, ctx): 28 | return super().visitIdentifierExpression(ctx) 29 | 30 | def visitNotExpression(self, ctx): 31 | self.count_of_expressions_visited += 1 32 | return super().visitNotExpression(ctx) 33 | 34 | def visitParenExpression(self, ctx): 35 | return super().visitParenExpression(ctx) 36 | 37 | def visitORBinaryExpression(self, ctx): 38 | self.count_of_expressions_visited += 1 39 | return super().visitORBinaryExpression(ctx) 40 | 41 | def visitEqualToComparatorExpression(self, ctx): 42 | self.count_of_expressions_visited += 1 43 | return super().visitEqualToComparatorExpression(ctx) 44 | 45 | def visitBetweenExpression(self, ctx): 46 | self.count_of_expressions_visited += 1 47 | return super().visitBetweenExpression(ctx) 48 | 49 | def visitInExpression(self, ctx): 50 | self.count_of_expressions_visited += 1 51 | return super().visitInExpression(ctx) 52 | 53 | def visitGreaterThanEqualToComparatorExpression(self, ctx): 54 | self.count_of_expressions_visited += 1 55 | return super().visitGreaterThanEqualToComparatorExpression(ctx) 56 | 57 | def visitLikeComparatorExpression(self, ctx): 58 | self.count_of_expressions_visited += 1 59 | return super().visitLikeComparatorExpression(ctx) 60 | 61 | def visitLesserThanEqualToComparatorExpression(self, ctx): 62 | self.count_of_expressions_visited += 1 63 | return super().visitLesserThanEqualToComparatorExpression(ctx) 64 | 65 | def visitNotEqualToComparatorExpression(self, ctx): 66 | self.count_of_expressions_visited += 1 67 | return super().visitNotEqualToComparatorExpression(ctx) 68 | 69 | def visitANDBinaryExpression(self, ctx): 70 | self.count_of_expressions_visited += 1 71 | return super().visitANDBinaryExpression(ctx) 72 | 73 | def visitStringValueExpression(self, ctx): 74 | self.count_of_expressions_visited += 1 75 | return super().visitStringValueExpression(ctx) 76 | 77 | def visitDecimalValueExpression(self, ctx): 78 | self.count_of_expressions_visited += 1 79 | return super().visitDecimalValueExpression(ctx) 80 | 81 | def visitIsoDate(self, ctx): 82 | self.count_of_expressions_visited += 1 83 | return super().visitIsoDate(ctx) 84 | 85 | def visitIsoDateTime(self, ctx): 86 | self.count_of_expressions_visited += 1 87 | return super().visitIsoDateTime(ctx) 88 | 89 | def visitLimit(self, ctx): 90 | self.count_of_expressions_visited += 1 91 | return super().visitLimit(ctx) 92 | 93 | def visitCountValueExpression(self, ctx): 94 | self.count_of_expressions_visited += 1 95 | return super().visitCountValueExpression(ctx) 96 | 97 | 98 | class QueryFilterExpressionVisitorTest(unittest.TestCase): 99 | def __init__(self, method_name='runTest'): 100 | self.test_visitor = TestVisitor() 101 | super().__init__(method_name) 102 | 103 | def _test_count_of_expression_visited(self, filter_expression: str, expected_count_of_expression_visited: int): 104 | parse_tree = parse(filter_expression) 105 | visited = self.test_visitor.visit(parse_tree) 106 | self.assertEqual(expected_count_of_expression_visited, visited) 107 | 108 | def test_expressions_visited(self): 109 | cases = [ 110 | ('os = "mojave"', 2), 111 | ('os != "mojave"', 2), 112 | ('accountId > 90', 2), 113 | ('LIMIT 100', 2), 114 | ('dateRange BETWEEN 1611639470000 AND 1611639476298', 3), 115 | ('date BETWEEN 1511630000000 AND 1611639476298', 3), 116 | ('time between 1511630000000 AND 1611639476298', 3), 117 | ('accountId < 100', 2), 118 | ('accountId >= 90', 2), 119 | ('accountId >= 90 LIMIT 100', 4), 120 | ('accountId <= 100', 2), 121 | ('accountId BETWEEN 90 AND 100', 3), 122 | ('os CONTAINS "mojave"', 2), 123 | ('os CONTAINS "moj%ave"', 2), 124 | ('os = "mojave" and app = "mo"', 5), 125 | ('os = "mojave" OR app = "mo"', 5), 126 | ('(os = "mojave" AND app = "mo") and (os = "mojave" OR app = "mo")', 11), 127 | ('(os = "mojave" AND app = "mo") or (os = "mojave" OR app = "mo")', 11), 128 | ('accountId in (100, 90, 70)', 4), 129 | ('date between 2021-04-20 and 2021-04-21', 3), 130 | ('date between 2021-04-20T12:30:45Z and 2021-04-20T15:45:49.234Z', 3), 131 | ('(accountId > 100 and ((date < 2021-04-20T12:30:45Z and date > 2021-04-21T15:45:49.234Z) and ' + 132 | 'accountId < 200))', 11) 133 | ] 134 | for expression, count in cases: 135 | with self.subTest(expression=expression, count=count): 136 | print((expression, count)) 137 | self._test_count_of_expression_visited(expression, count) 138 | -------------------------------------------------------------------------------- /custom_connector_example/test/query_test.py: -------------------------------------------------------------------------------- 1 | import custom_connector_sdk.connector.context as context 2 | import custom_connector_sdk.connector.fields as fields 3 | import custom_connector_queryfilter.queryfilter.parse_tree_builder as parse_tree_builder 4 | import custom_connector_queryfilter.queryfilter.errors as errors 5 | from custom_connector_example.query.visitor import SalesforceQueryFilterExpressionVisitor 6 | import unittest 7 | 8 | CREATED_DATE = 'CreatedDate' 9 | UPDATED_DATE = 'UpdatedDate' 10 | ACCOUNT_NUMBER = 'AccountNumber' 11 | ID = 'Id' 12 | NAME = 'Name' 13 | 14 | 15 | class SalesforceQueryFilterExpressionVisitorTest(unittest.TestCase): 16 | """Test class to validate Salesforce queries built by SalesforceQueryFilterExpressionVisitor.""" 17 | def _test_conversion_from_filter_expression_to_salesforce_query(self, 18 | filter_expression: str, 19 | expected_where_clause: str, 20 | expected_limit_clause: str): 21 | entity = context.Entity(entity_identifier='Account', 22 | has_nested_entities=False, 23 | is_writable=False) 24 | field_definitions = [fields.FieldDefinition(field_name=CREATED_DATE, 25 | data_type=fields.FieldDataType.DateTime, 26 | data_type_label='DateTime'), 27 | fields.FieldDefinition(field_name=UPDATED_DATE, 28 | data_type=fields.FieldDataType.Date, 29 | data_type_label='Date'), 30 | fields.FieldDefinition(field_name=NAME, 31 | data_type=fields.FieldDataType.String, 32 | data_type_label='String'), 33 | fields.FieldDefinition(field_name=ID, 34 | data_type=fields.FieldDataType.String, 35 | data_type_label='String'), 36 | fields.FieldDefinition(field_name=ACCOUNT_NUMBER, 37 | data_type=fields.FieldDataType.Long, 38 | data_type_label='Long')] 39 | entity_definition = context.EntityDefinition(entity=entity, fields=field_definitions) 40 | salesforce_query_expression_visitor = SalesforceQueryFilterExpressionVisitor(entity_definition) 41 | 42 | salesforce_query_expression_visitor.visit(parse_tree_builder.parse(filter_expression)) 43 | where_clause, limit_clause = salesforce_query_expression_visitor.get_result() 44 | self.assertEqual(expected_where_clause, where_clause) 45 | self.assertEqual(expected_limit_clause, limit_clause) 46 | 47 | def test_query(self): 48 | expressions = [ 49 | ('Name = "TestAccountName"', 'Name = \'TestAccountName\'', ''), 50 | ('Id != \'0016g00001cyrfiAAA\' AND AccountNumber = 40', 51 | 'Id != \'0016g00001cyrfiAAA\' AND AccountNumber = 40', 52 | ''), 53 | ('limit 100', '', '100'), 54 | ('CreatedDate > 2021-04-20T10:30:35Z AND AccountNumber = 40', 55 | 'CreatedDate > 2021-04-20T10:30:35.000+0000 AND AccountNumber = 40', 56 | ''), 57 | ('CreatedDate between 2021-04-20T10:30:35Z and 2021-04-25T10:30:35Z', 58 | 'CreatedDate > 2021-04-20T10:30:35.000+0000 and CreatedDate < 2021-04-25T10:30:35.000+0000', 59 | ''), 60 | ('(AccountNumber > 100 and ((CreatedDate < 2021-04-20T12:30:45Z and CreatedDate > ' 61 | '2021-04-21T15:45:49.234Z) and Name contains \"TestAccountName\"))', 62 | 'AccountNumber > 100 and CreatedDate < 2021-04-20T12:30:45.000+0000 and CreatedDate > ' 63 | '2021-04-21T15:45:49.234+0000 and Name LIKE \'%TestAccountName%\'', 64 | ''), 65 | ('(AccountNumber >= 100 and ((CreatedDate <= 2021-04-20T12:30:45Z and CreatedDate >= ' 66 | '2021-04-21T15:45:49.234Z) and Name contains \"TestAccountName\"))', 67 | 'AccountNumber >= 100 and CreatedDate <= 2021-04-20T12:30:45.000+0000 and CreatedDate >= ' 68 | '2021-04-21T15:45:49.234+0000 and Name LIKE \'%TestAccountName%\'', 69 | ''), 70 | ('Date = 2018-05-03 OR Date = 2018-04-20', 'Date = 2018-05-03 OR Date = 2018-04-20', ''), 71 | ('AccountNumber between 1 AND 10', 'AccountNumber > 1 and AccountNumber < 10', ''), 72 | ('AccountNumber = 1 or (UpdatedDate between 2020-03-05 and 2020-03-07)', 73 | 'AccountNumber = 1 or UpdatedDate > 2020-03-05 and UpdatedDate < 2020-03-07', 74 | ''), 75 | ('AccountNumber = 1 or (UpdatedDate between 2020-03-05 and 2020-03-07) limit 100', 76 | 'AccountNumber = 1 or UpdatedDate > 2020-03-05 and UpdatedDate < 2020-03-07', 77 | '100'), 78 | ('AccountNumber in (3, 5, 9)', "AccountNumber IN (3,5,9)", '') 79 | ] 80 | for filter_expression, expected_where, expected_limit in expressions: 81 | with self.subTest(expression=filter_expression): 82 | self._test_conversion_from_filter_expression_to_salesforce_query(filter_expression, 83 | expected_where, 84 | expected_limit) 85 | 86 | def test_invalid_filter_expressions(self): 87 | expressions = [ 88 | ('AccountNumber BETWEEN 5', errors.InvalidFilterExpressionError), 89 | ('LIMIT 100 LIMIT 5', errors.InvalidFilterExpressionError), 90 | ('NotAField = 10', ValueError) 91 | ] 92 | for filter_expression, error in expressions: 93 | with self.subTest(expression=filter_expression, expected_error=error): 94 | self.assertRaises(error, 95 | self._test_conversion_from_filter_expression_to_salesforce_query, 96 | filter_expression, 97 | '', 98 | '') 99 | -------------------------------------------------------------------------------- /custom_connector_tools/README.md: -------------------------------------------------------------------------------- 1 | # Deploy a connector 2 | This document explains how you can deploy a lambda connector into your AWS account. This script uses the AWS CloudFormation template to deploy the custom connector lambda into the given AWS account. 3 | 4 | ## Prerequisites: 5 | 6 | - pip3 7 | - aws cli 8 | 9 | ## Command: 10 | 11 | Run this command from the connector module which you want to deploy. Make sure that cloudformation template exists for the connector inside the connector module. 12 | 13 | #### Command : ../custom-connector-tools/deploy.sh AWS_REGION BUCKET_NAME STACK_NAME 14 | 15 | - AWS_REGION : AWS region where you want to deploy your custom connector. By default it will use the us-east-1 region, if the value that is provided is an empty string (""). 16 | 17 | - BUCKET_NAME : S3 bucket where you want to upload artifacts to deploy the connector. By default it will create the S3 bucket, if the value that is provided is an empty string (""). 18 | 19 | - STACK_NAME : Name of the stack to be displayed in AWS CloudFormation. 20 | 21 | - PACKAGE_NAME : Name of the folder which will be created in the same directory as SDK and it will delete the existing folder if already exists. 22 | 23 | Sample command : `../custom_connector_tools/deploy.sh us-west-2 agrshubh-awsappflow-us-west-2-test custom-connector package` 24 | Sample command with region and S3 bucket as empty strings: `../custom_connector_tools/deploy.sh "" "" custom-connector package` 25 | 26 | ### Creating a template.yml for cloudformation. 27 | Define the AWS CloudFormation template by cloning the template named template.yaml for Lambda connector deployment. Alternatively, you can copy the AWS CloudFormation template from the example connector module and change the handler parameter appropriately. Please make sure to verify the policies. This template.yml will put a policy as below on the execution role of lambda 28 | 29 | if you want to restrict lambda access to the secrets. You can update the policy as below. 30 | ##### Please fill your accountId and connectorLabel in the policy before deploying. 31 | 32 | ```` 33 | { 34 | "Version": "2012-10-17", 35 | "Statement": { 36 | "Action": "secretsmanager:GetSecretValue", 37 | "Resource": "arn:aws:secretsmanager:us-west-2::secret:appflow!--*”, 38 | "Effect": "Allow" 39 | } 40 | 41 | } 42 | ```` 43 | This will allow lambda to access the secrets only for the connector profiles that were created using this lambda connector. 44 | 45 | 2. This script also attaches below policy 46 | ##### Please fill your accountId in the policy before deploying. 47 | ```` 48 | { 49 | "Version": "2012-10-17", 50 | "Id": "default", 51 | "Statement": [ 52 | { 53 | "Sid": "PolicyPermission-1CM96J5MWQY04", 54 | "Effect": "Allow", 55 | "Principal": { 56 | "Service": "appflow.amazonaws.com" 57 | }, 58 | "Action": "lambda:InvokeFunction", 59 | "Resource": "arn:aws:lambda:us-west-2::function:custom-connector-lN7JnmKTetlM", 60 | "Condition": { 61 | "StringEquals": { 62 | "AWS:SourceAccount": "" 63 | }, 64 | "ArnLike": { 65 | "AWS:SourceArn": "arn:aws:appflow:us-west-2::*" 66 | } 67 | } 68 | } 69 | ] 70 | } 71 | ```` 72 | 73 | ## Deploying the example connector: 74 | 75 | - Step 1 : Set your AWS account credentials to use aws cli using ‘aws configure’ command. Follow this link to get the creds: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds 76 | 77 | - Step 2: Run the script from custom-connector-example module and it will deploy the connector the given AWS account. 78 | 79 | ## Deploying the new connector: 80 | 81 | - Step 1: Define the AWS CloudFormation template by cloning the template named template.yaml for Lambda connector deployment. Alternatively, you can copy the AWS CloudFormation template from the example connector module and change the handler parameter appropriately. 82 | 83 | - Step 2 : Set your AWS account credentials to use aws cli using ‘aws configure’ command. Follow this link to get the creds: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds 84 | 85 | - Step 3: Run the script from newly built connector module and it will deploy the connector in the corresponding AWS account. 86 | 87 | # Sharing the logs with the connector developer 88 | There can be situation when as a AppFlow user you want to share the logs of the custom connector with the custom connector developer. Since lambda will be deployed in you account , AppFlow and Connector developer will be dependent on you to share the logs. We understand that you might not feel comfortable in providing the access to your log groups to either Connector Developer or AppFlow. Hence we have provided a script to fetch the selected logs. 89 | 90 | To use the script , please run the below command 91 | ```` 92 | ./logfetcher.sh 93 | ```` 94 | 95 | Sample Inputs to the command 96 | ```` 97 | Provide Region:us-east-1 98 | -------------- 99 | Provide Loggroup (Should be in the format of /aws/lambda/custom-connector-logging-Aaw0rrvylsya):/aws/lambda/custom-connector-logging-poc-function-Aaw0rrvylsya 100 | ------------------------------------------------------------------------------------------------ 101 | Provide name for log file that will be generated:debug-logs 102 | ------------------------------------------------- 103 | Provide start time (in epoc seconds) for log query:1635383800 104 | -------------------------------------------------- 105 | Provide End Time (in epoc seconds) for log query:1635384000 106 | ------------------------------------------------- 107 | Provide Query String:filter @message like /appflow/ | fields @timestamp,@message | sort @timestamp desc 108 | -------------------- 109 | Provide Bucket for log file:logbucket 110 | --------------------------- 111 | Provide number of seconds until the pre-signed URL expires. 2000 112 | ----------------------------------------------------------- 113 | Cloudwatch query takes time to execute. The time depends on the interval for which logs are being fetched.Provide wait time (seconds) before query finished.300 114 | Are above details correct, Please select y/n:y 115 | ```` 116 | 117 | Read more here : https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html for more advanced cloudwatch queries. 118 | -------------------------------------------------------------------------------- /custom_connector_example/test/record_test.py: -------------------------------------------------------------------------------- 1 | import json 2 | import unittest 3 | from unittest.mock import patch 4 | import custom_connector_example.handlers.lambda_handler as lambda_handler 5 | from custom_connector_example.handlers.salesforce import SalesforceResponse 6 | 7 | class SalesforceRecordHandlerTests(unittest.TestCase): 8 | """Test class to validate handling of Record requests by Salesforce connector.""" 9 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 10 | def test_retrieve_data_request_success(self, mock): 11 | with open('custom_connector_example/test/resources/describe_sobject_server_response.json', 12 | 'r') as json_response: 13 | server_response = json_response.read() 14 | mock.return_value = SalesforceResponse(200, server_response, '') 15 | 16 | with open('custom_connector_example/test/resources/retrieve_data_request_valid.json', 'r') as json_file: 17 | data = json.load(json_file) 18 | 19 | response = lambda_handler.salesforce_lambda_handler(data, None) 20 | self.assertEqual(response.get('isSuccess'), True) 21 | 22 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 23 | def test_retrieve_data_request_no_matches(self, mock): 24 | mock.return_value = SalesforceResponse(200, '{}', '') 25 | 26 | with open('custom_connector_example/test/resources/retrieve_data_request_valid.json', 'r') as json_file: 27 | data = json.load(json_file) 28 | 29 | response = lambda_handler.salesforce_lambda_handler(data, None) 30 | self.assertEqual(response.get('isSuccess'), True) 31 | 32 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 33 | def test_retrieve_data_request_server_error(self, mock): 34 | mock.return_value = SalesforceResponse(400, '{}', 'Invalid argument') 35 | 36 | with open('custom_connector_example/test/resources/retrieve_data_request_valid.json', 'r') as json_file: 37 | data = json.load(json_file) 38 | 39 | response = lambda_handler.salesforce_lambda_handler(data, None) 40 | self.assertEqual(response.get('isSuccess'), False) 41 | 42 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 43 | def test_retrieve_data_request_invalid_context(self, mock): 44 | with open('custom_connector_example/test/resources/describe_sobject_server_response.json', 45 | 'r') as json_response: 46 | server_response = json_response.read() 47 | mock.return_value = SalesforceResponse(200, server_response, '') 48 | 49 | with open('custom_connector_example/test/resources/retrieve_data_request_invalid.json', 'r') as json_file: 50 | data = json.load(json_file) 51 | 52 | response = lambda_handler.salesforce_lambda_handler(data, None) 53 | self.assertEqual(response.get('isSuccess'), False) 54 | 55 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_post') 56 | def test_write_data_request_success(self, mock): 57 | mock.return_value = SalesforceResponse(200, '{}', '') 58 | 59 | with open('custom_connector_example/test/resources/write_data_request_valid.json', 'r') as json_file: 60 | data = json.load(json_file) 61 | 62 | response = lambda_handler.salesforce_lambda_handler(data, None) 63 | self.assertEqual(response.get('isSuccess'), True) 64 | 65 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_post') 66 | def test_write_data_request_server_error(self, mock): 67 | mock.return_value = SalesforceResponse(401, '{}', 'Invalid credentials') 68 | 69 | with open('custom_connector_example/test/resources/write_data_request_valid.json', 'r') as json_file: 70 | data = json.load(json_file) 71 | 72 | response = lambda_handler.salesforce_lambda_handler(data, None) 73 | self.assertEqual(response.get('isSuccess'), False) 74 | 75 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_post') 76 | def test_write_data_request_invalid_context(self, mock): 77 | mock.return_value = SalesforceResponse(200, '{}', '') 78 | 79 | with open('custom_connector_example/test/resources/write_data_request_invalid.json', 'r') as json_file: 80 | data = json.load(json_file) 81 | 82 | response = lambda_handler.salesforce_lambda_handler(data, None) 83 | self.assertEqual(response.get('isSuccess'), False) 84 | 85 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 86 | def test_query_data_request_filter_expression(self, mock): 87 | mock.return_value = SalesforceResponse(200, '{}', '') 88 | 89 | with open('custom_connector_example/test/resources/query_data_request_filter_expression.json', 90 | 'r') as json_file: 91 | data = json.load(json_file) 92 | 93 | response = lambda_handler.salesforce_lambda_handler(data, None) 94 | self.assertEqual(response.get('isSuccess'), True) 95 | 96 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 97 | def test_query_data_request_no_filter_expression(self, mock): 98 | mock.return_value = SalesforceResponse(200, '{}', '') 99 | 100 | with open('custom_connector_example/test/resources/query_data_request_no_filter_expression.json', 101 | 'r') as json_file: 102 | data = json.load(json_file) 103 | 104 | response = lambda_handler.salesforce_lambda_handler(data, None) 105 | self.assertEqual(response.get('isSuccess'), True) 106 | 107 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 108 | def test_query_data_request_no_selected_fields(self, mock): 109 | mock.return_value = SalesforceResponse(200, '{}', '') 110 | 111 | with open('custom_connector_example/test/resources/query_data_request_no_selected_fields.json', 112 | 'r') as json_file: 113 | data = json.load(json_file) 114 | 115 | self.assertRaises(RuntimeError, lambda_handler.salesforce_lambda_handler, data, None) 116 | 117 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 118 | def test_query_data_request_server_error(self, mock): 119 | mock.return_value = SalesforceResponse(400, '{}', 'Argument is invalid') 120 | 121 | with open('custom_connector_example/test/resources/query_data_request_filter_expression.json', 122 | 'r') as json_file: 123 | data = json.load(json_file) 124 | 125 | response = lambda_handler.salesforce_lambda_handler(data, None) 126 | self.assertEqual(response.get('isSuccess'), False) 127 | 128 | @patch('custom_connector_example.handlers.client.HttpsClient.rest_get') 129 | def test_query_data_request_invalid_context(self, mock): 130 | mock.return_value = SalesforceResponse(200, '{}', '') 131 | 132 | with open('custom_connector_example/test/resources/query_data_request_invalid.json', 'r') as json_file: 133 | data = json.load(json_file) 134 | 135 | response = lambda_handler.salesforce_lambda_handler(data, None) 136 | self.assertEqual(response.get('isSuccess'), False) 137 | -------------------------------------------------------------------------------- /custom_connector_example/handlers/configuration.py: -------------------------------------------------------------------------------- 1 | import custom_connector_sdk.lambda_handler.requests as requests 2 | import custom_connector_sdk.lambda_handler.responses as responses 3 | import custom_connector_sdk.connector.settings as settings 4 | import custom_connector_sdk.connector.configuration as config 5 | import custom_connector_sdk.connector.context as context 6 | import custom_connector_sdk.connector.auth as auth 7 | import custom_connector_example.constants as constants 8 | import custom_connector_example.handlers.validation as validation 9 | import custom_connector_example.handlers.salesforce as salesforce 10 | from custom_connector_sdk.lambda_handler.handlers import ConfigurationHandler 11 | from custom_connector_example.handlers.client import HttpsClient, SalesforceResponse 12 | 13 | CONNECTOR_OWNER = 'SampleConnector' 14 | CONNECTOR_NAME = 'SamplePythonSalesforceConnector' 15 | CONNECTOR_VERSION = '1.0' 16 | 17 | API_VERSION = 'v51.0' 18 | API_VERSION_KEY = 'api_version' 19 | IS_SANDBOX_ACCOUNT = "is_sandbox_account" 20 | 21 | AUTH_URL = 'https://login.salesforce.com/services/oauth2/authorize' 22 | TOKEN_URL = 'https://login.salesforce.com/services/oauth2/token' 23 | SANDBOX_AUTH_URL = 'https://test.salesforce.com/services/oauth2/authorize' 24 | SANDBOX_TOKEN_URL = 'https://test.salesforce.com/services/oauth2/token' 25 | REFRESH_URL = 'https://login.salesforce.com/services/oauth2/token' 26 | REDIRECT_URL = 'https://login.salesforce.com' 27 | SALESFORCE_USERINFO_URL_FORMAT = 'https://login.salesforce.com/services/oauth2/userinfo' 28 | SALESFORCE_USERINFO_SANDBOX_URL_FORMAT = 'https://test.salesforce.com/services/oauth2/userinfo' 29 | TRUE = 'true' 30 | 31 | def buildSalesforceUserInfoRequest(connector_runtime_settings: dict) -> str: 32 | is_sandbox = connector_runtime_settings.get(IS_SANDBOX_ACCOUNT) 33 | if is_sandbox and is_sandbox.lower() == TRUE: 34 | return SALESFORCE_USERINFO_SANDBOX_URL_FORMAT 35 | return SALESFORCE_USERINFO_URL_FORMAT 36 | 37 | def get_salesforce_client(secret_arn: str): 38 | return HttpsClient(salesforce.get_access_token_from_secret(secret_arn)) 39 | 40 | class SalesforceConfigurationHandler(ConfigurationHandler): 41 | """Salesforce Configuration Handler.""" 42 | def validate_connector_runtime_settings(self, request: requests.ValidateConnectorRuntimeSettingsRequest) -> \ 43 | responses.ValidateConnectorRuntimeSettingsResponse: 44 | errors = validation.validate_connector_runtime_settings(request) 45 | if errors: 46 | return responses.ValidateConnectorRuntimeSettingsResponse(is_success=False, error_details=errors) 47 | return responses.ValidateConnectorRuntimeSettingsResponse(is_success=True) 48 | 49 | def validate_credentials(self, request: requests.ValidateCredentialsRequest) -> \ 50 | responses.ValidateCredentialsResponse: 51 | connector_context = context.ConnectorContext(credentials=request.credentials, 52 | api_version=API_VERSION, 53 | connector_runtime_settings=request.connector_runtime_settings) 54 | request_uri = buildSalesforceUserInfoRequest(request.connector_runtime_settings) 55 | salesforce_response = get_salesforce_client(request.credentials.secret_arn).rest_get(request_uri) 56 | error_details = salesforce.check_for_errors_in_salesforce_response(salesforce_response) 57 | if error_details: 58 | return responses.ValidateCredentialsResponse(is_success=False, 59 | error_details=error_details) 60 | return responses.ValidateCredentialsResponse(is_success=True) 61 | 62 | def describe_connector_configuration(self, request: requests.DescribeConnectorConfigurationRequest) -> \ 63 | responses.DescribeConnectorConfigurationResponse: 64 | connector_modes = [config.ConnectorModes.SOURCE, config.ConnectorModes.DESTINATION] 65 | 66 | instance_url_setting = settings.ConnectorRuntimeSetting(key=constants.INSTANCE_URL_KEY, 67 | data_type=settings.ConnectorRuntimeSettingDataType 68 | .String, 69 | required=True, 70 | label='Salesforce Instance URL', 71 | description='URL of the instance where user wants to ' + 72 | 'run the operations', 73 | scope=settings.ConnectorRuntimeSettingScope 74 | .CONNECTOR_PROFILE) 75 | is_sandbox_account_setting = settings.ConnectorRuntimeSetting(key=IS_SANDBOX_ACCOUNT, 76 | data_type=settings.ConnectorRuntimeSettingDataType 77 | .Boolean, 78 | required=True, 79 | label='Type of salesforce account', 80 | description='Is Salesforce account a sandbox account', 81 | scope=settings.ConnectorRuntimeSettingScope 82 | .CONNECTOR_PROFILE) 83 | connector_runtime_settings = [instance_url_setting, is_sandbox_account_setting] 84 | 85 | o_auth_2_defaults = auth.OAuth2Defaults(auth_url=[AUTH_URL, SANDBOX_AUTH_URL], 86 | token_url=[TOKEN_URL, SANDBOX_TOKEN_URL], 87 | o_auth_scopes=['api', 'refresh_token'], 88 | o_auth_2_grant_types_supported=[auth.OAuth2GrantType.AUTHORIZATION_CODE]) 89 | authentication_config = auth.AuthenticationConfig(is_oauth_2_supported=True, 90 | o_auth_2_defaults=o_auth_2_defaults) 91 | 92 | return responses.DescribeConnectorConfigurationResponse(is_success=True, 93 | connector_owner=CONNECTOR_OWNER, 94 | connector_name=CONNECTOR_NAME, 95 | connector_version=CONNECTOR_VERSION, 96 | connector_modes=connector_modes, 97 | connector_runtime_setting=connector_runtime_settings, 98 | authentication_config=authentication_config, 99 | supported_api_versions=[API_VERSION], 100 | supported_write_operations=constants 101 | .SUPPORTED_WRITE_OPERATIONS) 102 | -------------------------------------------------------------------------------- /MarketplaceIntegration.md: -------------------------------------------------------------------------------- 1 | # Sharing AppFlow connectors via AWS Marketplace 2 | 3 | Amazon AppFlow allows 3rd party developers and partners to author connectors and share them with other AppFlow users using AWS Marketplace. As a developer you can upload the Lambda as a “Container” product on AWS Marketplace to sell it to Amazon AppFlow customers or to share it for free. This guide provides the details on the development and publishing process. 4 | 5 | # Overview of creating connectors for AWS Marketplace 6 | 7 | Amazon AppFlow customers discover and subscribe to connectors of interest on AWS Marketplace. 8 | This section helps the connector developers to build and test custom connectors, and deploy them for integration testing with Amazon AppFlow. Summary of steps: 9 | 10 | 1. Develop a custom connector. 11 | 2. Create a new product in AWS Marketplace. 12 | 3. Create Repository for your container. 13 | 4. Test/Validate your connector. 14 | 5. After you've completed the above steps, post your product onto AWS Marketplace. 15 | 16 | ## Step 1: Developing Marketplace connectors 17 | 18 | Amazon AppFlow provides support for developing the connector for various SaaS applications in Java and Python. 19 | 20 | ## Step 2: Create your Connector product in AWS Marketplace 21 | 22 | Amazon AppFlow only supports the “Container” product type for publishing custom connectors on Marketplace. Creating a product in AWS Marketplace involves the following steps: 23 | 24 | 1. Create the product ID. https://docs.aws.amazon.com/marketplace/latest/userguide/container-product-getting-started.html#create-initial-container-product 25 | 2. Create the pricing details. https://docs.aws.amazon.com/marketplace/latest/userguide/container-product-getting-started.html#container-product-load-form 26 | Note : Amazon AppFlow only supports Free and Monthly subscription based model for pricing. It does not provide support for Custom/Hourly metering. 27 | 3. For paid products, integrate metering into your product. 28 | 1. To validate if the customer is entitled to use the paid product, you can add a validation in lambda to call the EntitlementUtil helper class provided with the SDK. 29 | 2. To call the Entitlement checker you need to provide the ProductId as input. 30 | 3. Instead of calling this entitlement check for every Lambda invocation, you are recommended to call it at a specific frequency of your choice from the Lambda code. e.g. once per hour or once per day 31 | 32 | Once your request is approved from the AWS Marketplace for Limited use, proceed to the next step. 33 | 34 | ## Step 3: Create Repository for your container. 35 | 36 | 1. Select your product: Choose `Server` from the `Products` drop down tab of your AWS Marketplace Management Portal (https://4hs3rzdz.r.us-east-1.awstrack.me/L0/https:%2F%2Faws.amazon.com%2Fmarketplace%2Fmanagement%2Fhomepage/1/0100017c18e8d88e-e5c55f5d-9c02-4aea-aaf8-313d15621a11-000000/a08CntXX0hvPwR0OBuFAHEtxlLM=237) to go to the Server Products (https://4hs3rzdz.r.us-east-1.awstrack.me/L0/https:%2F%2Faws.amazon.com%2Fmarketplace%2Fmanagement%2Fproducts%2Fserver/1/0100017c18e8d88e-e5c55f5d-9c02-4aea-aaf8-313d15621a11-000000/dHlwBH29mAefqfiNc4YhN0Os4vo=237) page. Select your product using the corresponding radio button or by clicking its Title. 37 | 38 | 2. Add repositories: From the `Request changes` drop down, choose `Add repositories` to create repositories that you can then push your product’s resources into. 39 | 40 | 3. Push images to your repository: On the `Add repositories` page, click on `View existing repositories` to see all available repositories that were successfully added. Select the repository name, and choose `View push commands` to view instructions to push your container images and resources to the selected repository. Follow the steps provided below in *Packaging and uploading Marketplace connectors section below. 41 | 42 | 4. Create a new version: On the `Products` page, select your product, and choose `Add new version` to create a version of your product using the container images and resources that you added to the repository. 43 | 44 | 5. Update product details: Select `Update product information` to edit the data that buyers will see when they select your product. 45 | 46 | Note: Currently, AWS marketplace does not support the Lambda product types. So, it is your responsibility to provide the usage instructions to the users. 47 | 48 | ## Step 4: Test/Validate your connector. 49 | 50 | Please Follow the steps provided in Unit testing and Integration testing guidelines. 51 | 52 | ## Step 5: Publish the product to the public* 53 | 54 | After completing all the previous steps, you can publish your product to make it visible to all AWS AppFlow customers. Follow the instructions in Publishing container products (https://docs.aws.amazon.com/marketplace/latest/userguide/container-product-getting-started.html#container-product-publishing) in the AWS Marketplace Seller Guide. 55 | 56 | # Packaging and uploading Marketplace connectors* 57 | 58 | This section describes how to create and publish a container product with the required connector JAR files to AWS Marketplace. 59 | 60 | Prerequisites: 61 | 62 | 1. Setup AWS Command Line Interface (AWSCLI). 63 | 2. Install Docker Engine. 64 | 65 | Steps: 66 | 67 | 1. Create a DockerFile in your connector directory. You can use the DockerFile provided in the example. 68 | 69 | FROM public.ecr.aws/lambda/python:3.8 70 | 71 | #Copy function code 72 | COPY custom_connector_example ${LAMBDA_TASK_ROOT} 73 | COPY custom_connector_sdk ${LAMBDA_TASK_ROOT} 74 | COPY custom_connector_queryfilter ${LAMBDA_TASK_ROOT} 75 | 76 | #Install the function's dependencies using file requirements.txt from your project folder. 77 | COPY requirements.txt . 78 | RUN pip3 install -r ../requirements.txt --target "${LAMBDA_TASK_ROOT}" 79 | 80 | #Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile) 81 | CMD [ "custom_connector_example.handlers.lambda_handler.salesforce_lambda_handler" ] 82 | 83 | 2. Build your docker image by using the following command. 84 | 85 | docker build -t salesforcepaid . 86 | 87 | 3. Authenticate to the registry created from step 2.4a. 88 | 89 | aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com (http://aws_account_id.dkr.ecr.region.amazonaws.com/) 90 | 91 | 4. Tag the image to push to repository. 92 | 93 | docker tag hello-world:1.0 aws_account_id.dkr.ecr.us-east-1.amazonaws.com/hello-world:1.0 94 | 95 | 5. Push the image 96 | 97 | docker push aws_account_id.dkr.ecr.us-east-1.amazonaws.com/hello-world:1.0 98 | 99 | 6. Validate the image is created in ECR or not. 100 | 101 | aws ecr describe-images --registry-id 709825985650 --repository-name test-product/salesforcepaid —region us-east-1 102 | 103 | # Usage Instruction for Connector Users: 104 | 105 | Ensure you have installed the latest version of the AWS CLI and Docker. For more information, see ECR documentation 106 | 107 | For macOS or Linux systems, use the AWS CLI: 108 | 109 | ## Step 1: Retrieve the login command to authenticate your Docker client to your registry. 110 | 111 | aws ecr get-login-password --region us-east-1 | docker login --username AWS —password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com 112 | 113 | For Windows systems, use AWS Tools for PowerShell: 114 | 115 | Invoke-Expression -Command (Get-ECRLoginCommand -Region us-east-1 -RegistryId "709825985650").Command 116 | 117 | Note: If you receive an 'Unknown options: -no-include-email' error when using the AWS CLI, ensure that you have the latest version installed. 118 | 119 | ## Step 2: Pull the docker images listed below. 120 | 121 | docker pull 709825985650.dkr.ecr.us-east-1.amazonaws.com/test-product/salesforcejava:1.0 (http://709825985650.dkr.ecr.us-east-1.amazonaws.com/test-product/salesforcejava:1.0) 122 | 123 | ## Step 3: Create an ECR repository in your account. Follow this to create repository https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html 124 | 125 | ## Step 4: Tag a new image based on the original source image. 126 | 127 | docker tag $SOURCE_IMAGE:$VERSION $TARGET_IMAGE:$VERSION 128 | 129 | ## Step 5: Run this for your AWS account where you want to deploy this. 130 | 131 | aws ecr get-login-password --region AWS_REGION | docker login --username AWS --password-stdin AWS_ACCOUNT.dkr.ecr.AWS_REGION.amazonaws.com (http://aws_account.dkr.ecr.us-east-1.amazonaws.com/) 132 | 133 | ## Step 6: Push the image into the repository you created. 134 | 135 | docker push docker push $TARGET_IMAGE:$VERSION 136 | 137 | ## Step 7: Create a lambda function in your account by using the container image. 138 | 139 | You are all set to use this Lambda Connector. 140 | --------------------------------------------------------------------------------