├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── app.py ├── assets ├── amazon-kinesis-data-viewer.png ├── cdc-iceberg-data-level-01.png ├── cdc-iceberg-data-level-02.png ├── cdc-iceberg-data-level-03.png └── cdc-iceberg-table.png ├── cdk.json ├── cdk_stacks ├── __init__.py ├── aurora_mysql.py ├── bastion_host.py ├── dms_aurora_mysql_to_kinesis.py ├── dms_iam_roles.py ├── glue_job_role.py ├── glue_stream_data_schema.py ├── glue_streaming_job.py ├── kds.py ├── lakeformation_permissions.py ├── s3.py └── vpc.py ├── requirements-dev.txt ├── requirements.txt ├── source.bat ├── src ├── main │ └── python │ │ └── spark_sql_merge_into_iceberg.py └── utils │ ├── gen_fake_cdc_data.py │ └── gen_fake_mysql_data.py └── transactional-datalake-arch.svg /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/.gitignore -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/CODE_OF_CONDUCT.md -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/CONTRIBUTING.md -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/README.md -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/app.py -------------------------------------------------------------------------------- /assets/amazon-kinesis-data-viewer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/assets/amazon-kinesis-data-viewer.png -------------------------------------------------------------------------------- /assets/cdc-iceberg-data-level-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/assets/cdc-iceberg-data-level-01.png -------------------------------------------------------------------------------- /assets/cdc-iceberg-data-level-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/assets/cdc-iceberg-data-level-02.png -------------------------------------------------------------------------------- /assets/cdc-iceberg-data-level-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/assets/cdc-iceberg-data-level-03.png -------------------------------------------------------------------------------- /assets/cdc-iceberg-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/assets/cdc-iceberg-table.png -------------------------------------------------------------------------------- /cdk.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk.json -------------------------------------------------------------------------------- /cdk_stacks/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/__init__.py -------------------------------------------------------------------------------- /cdk_stacks/aurora_mysql.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/aurora_mysql.py -------------------------------------------------------------------------------- /cdk_stacks/bastion_host.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/bastion_host.py -------------------------------------------------------------------------------- /cdk_stacks/dms_aurora_mysql_to_kinesis.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/dms_aurora_mysql_to_kinesis.py -------------------------------------------------------------------------------- /cdk_stacks/dms_iam_roles.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/dms_iam_roles.py -------------------------------------------------------------------------------- /cdk_stacks/glue_job_role.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/glue_job_role.py -------------------------------------------------------------------------------- /cdk_stacks/glue_stream_data_schema.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/glue_stream_data_schema.py -------------------------------------------------------------------------------- /cdk_stacks/glue_streaming_job.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/glue_streaming_job.py -------------------------------------------------------------------------------- /cdk_stacks/kds.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/kds.py -------------------------------------------------------------------------------- /cdk_stacks/lakeformation_permissions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/lakeformation_permissions.py -------------------------------------------------------------------------------- /cdk_stacks/s3.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/s3.py -------------------------------------------------------------------------------- /cdk_stacks/vpc.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/cdk_stacks/vpc.py -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/requirements-dev.txt -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/requirements.txt -------------------------------------------------------------------------------- /source.bat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/source.bat -------------------------------------------------------------------------------- /src/main/python/spark_sql_merge_into_iceberg.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/src/main/python/spark_sql_merge_into_iceberg.py -------------------------------------------------------------------------------- /src/utils/gen_fake_cdc_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/src/utils/gen_fake_cdc_data.py -------------------------------------------------------------------------------- /src/utils/gen_fake_mysql_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/src/utils/gen_fake_mysql_data.py -------------------------------------------------------------------------------- /transactional-datalake-arch.svg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/transactional-datalake-using-apache-iceberg-on-aws-glue/HEAD/transactional-datalake-arch.svg --------------------------------------------------------------------------------