├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── env-setup ├── composer_variables.template ├── create_buckets.sh ├── set_composer_variables.sh └── set_env.sh ├── example └── 1.terraform-automation │ ├── README.md │ ├── composer.tf │ ├── csr-cloudbuildtrigger.tf │ ├── gcs.tf │ ├── iam.tf │ ├── main.tf │ ├── network.tf │ ├── output.tf │ ├── pubsub.tf │ ├── terraform.tfvars │ └── variables.tf └── source-code ├── build-pipeline ├── build_deploy_test.yaml ├── deploy_prod.yaml └── wait_for_dag_deployed.sh ├── data-processing-code ├── pom.xml └── src │ ├── main │ └── java │ │ └── org │ │ └── apache │ │ └── beam │ │ └── examples │ │ └── WordCount.java │ └── test │ └── java │ └── org │ └── apache │ └── beam │ └── examples │ └── WordCountTest.java └── workflow-dag ├── compare_xcom_maps.py ├── data-pipeline-prod.py ├── data-pipeline-test.py ├── support-files ├── input.txt └── ref.txt └── test_compare_xcom_maps.py /CONTRIBUTING.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/CONTRIBUTING.md -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/README.md -------------------------------------------------------------------------------- /env-setup/composer_variables.template: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/env-setup/composer_variables.template -------------------------------------------------------------------------------- /env-setup/create_buckets.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/env-setup/create_buckets.sh -------------------------------------------------------------------------------- /env-setup/set_composer_variables.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/env-setup/set_composer_variables.sh -------------------------------------------------------------------------------- /env-setup/set_env.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/env-setup/set_env.sh -------------------------------------------------------------------------------- /example/1.terraform-automation/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/README.md -------------------------------------------------------------------------------- /example/1.terraform-automation/composer.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/composer.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/csr-cloudbuildtrigger.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/csr-cloudbuildtrigger.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/gcs.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/gcs.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/iam.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/iam.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/main.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/main.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/network.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/network.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/output.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/output.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/pubsub.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/pubsub.tf -------------------------------------------------------------------------------- /example/1.terraform-automation/terraform.tfvars: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/terraform.tfvars -------------------------------------------------------------------------------- /example/1.terraform-automation/variables.tf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/example/1.terraform-automation/variables.tf -------------------------------------------------------------------------------- /source-code/build-pipeline/build_deploy_test.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/build-pipeline/build_deploy_test.yaml -------------------------------------------------------------------------------- /source-code/build-pipeline/deploy_prod.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/build-pipeline/deploy_prod.yaml -------------------------------------------------------------------------------- /source-code/build-pipeline/wait_for_dag_deployed.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/build-pipeline/wait_for_dag_deployed.sh -------------------------------------------------------------------------------- /source-code/data-processing-code/pom.xml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/data-processing-code/pom.xml -------------------------------------------------------------------------------- /source-code/data-processing-code/src/main/java/org/apache/beam/examples/WordCount.java: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/data-processing-code/src/main/java/org/apache/beam/examples/WordCount.java -------------------------------------------------------------------------------- /source-code/data-processing-code/src/test/java/org/apache/beam/examples/WordCountTest.java: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/data-processing-code/src/test/java/org/apache/beam/examples/WordCountTest.java -------------------------------------------------------------------------------- /source-code/workflow-dag/compare_xcom_maps.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/workflow-dag/compare_xcom_maps.py -------------------------------------------------------------------------------- /source-code/workflow-dag/data-pipeline-prod.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/workflow-dag/data-pipeline-prod.py -------------------------------------------------------------------------------- /source-code/workflow-dag/data-pipeline-test.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/workflow-dag/data-pipeline-test.py -------------------------------------------------------------------------------- /source-code/workflow-dag/support-files/input.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/workflow-dag/support-files/input.txt -------------------------------------------------------------------------------- /source-code/workflow-dag/support-files/ref.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/workflow-dag/support-files/ref.txt -------------------------------------------------------------------------------- /source-code/workflow-dag/test_compare_xcom_maps.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GoogleCloudPlatform/ci-cd-for-data-processing-workflow/HEAD/source-code/workflow-dag/test_compare_xcom_maps.py --------------------------------------------------------------------------------