├── .flake8 ├── .github └── workflows │ └── ci.yml ├── .gitignore ├── Dockerfile ├── LICENSE ├── README.md ├── dags ├── .airflowignore ├── dag.py ├── dbt │ ├── .gitignore │ ├── dbt_project.yml │ ├── models │ │ ├── enriched_transactions.sql │ │ ├── filter_countries.sql │ │ └── schema.yml │ ├── profiles.yml │ └── tests │ │ ├── mock_pipeline │ │ ├── test_mock_enriched_transactions_beneficiary_name.sql │ │ ├── test_mock_enriched_transactions_no_name.sql │ │ ├── test_mock_enriched_transactions_payer_name.sql │ │ ├── test_mock_filter_countries_banned_beneficiary_countries.sql │ │ └── test_mock_filter_countries_banned_payer_countries.sql │ │ ├── test_enriched_transactions.sql │ │ └── test_filter_countries.sql └── spark │ ├── generate_data.py │ ├── tests │ ├── conftest.py │ └── test_union_transactions.py │ └── union_transactions.py ├── docker ├── add_spark_config.sh ├── entrypoint.sh ├── install_packages.sh └── setup_mockdata.sh ├── integrity_tests ├── conftest.py ├── requirements.txt ├── test_airflow_home │ └── airflow.cfg └── test_dag_integrity.py ├── mock_pipeline_requirements.txt └── slides └── PyData Amsterdam 2023 - Return to datas inferno.pdf /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | max-line-length = 120 3 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/.github/workflows/ci.yml -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/.gitignore -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/Dockerfile -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/README.md -------------------------------------------------------------------------------- /dags/.airflowignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/.airflowignore -------------------------------------------------------------------------------- /dags/dag.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dag.py -------------------------------------------------------------------------------- /dags/dbt/.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | dbt_modules 3 | logs 4 | target 5 | -------------------------------------------------------------------------------- /dags/dbt/dbt_project.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/dbt_project.yml -------------------------------------------------------------------------------- /dags/dbt/models/enriched_transactions.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/models/enriched_transactions.sql -------------------------------------------------------------------------------- /dags/dbt/models/filter_countries.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/models/filter_countries.sql -------------------------------------------------------------------------------- /dags/dbt/models/schema.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/models/schema.yml -------------------------------------------------------------------------------- /dags/dbt/profiles.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/profiles.yml -------------------------------------------------------------------------------- /dags/dbt/tests/mock_pipeline/test_mock_enriched_transactions_beneficiary_name.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/mock_pipeline/test_mock_enriched_transactions_beneficiary_name.sql -------------------------------------------------------------------------------- /dags/dbt/tests/mock_pipeline/test_mock_enriched_transactions_no_name.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/mock_pipeline/test_mock_enriched_transactions_no_name.sql -------------------------------------------------------------------------------- /dags/dbt/tests/mock_pipeline/test_mock_enriched_transactions_payer_name.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/mock_pipeline/test_mock_enriched_transactions_payer_name.sql -------------------------------------------------------------------------------- /dags/dbt/tests/mock_pipeline/test_mock_filter_countries_banned_beneficiary_countries.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/mock_pipeline/test_mock_filter_countries_banned_beneficiary_countries.sql -------------------------------------------------------------------------------- /dags/dbt/tests/mock_pipeline/test_mock_filter_countries_banned_payer_countries.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/mock_pipeline/test_mock_filter_countries_banned_payer_countries.sql -------------------------------------------------------------------------------- /dags/dbt/tests/test_enriched_transactions.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/test_enriched_transactions.sql -------------------------------------------------------------------------------- /dags/dbt/tests/test_filter_countries.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/dbt/tests/test_filter_countries.sql -------------------------------------------------------------------------------- /dags/spark/generate_data.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/spark/generate_data.py -------------------------------------------------------------------------------- /dags/spark/tests/conftest.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/spark/tests/conftest.py -------------------------------------------------------------------------------- /dags/spark/tests/test_union_transactions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/spark/tests/test_union_transactions.py -------------------------------------------------------------------------------- /dags/spark/union_transactions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/dags/spark/union_transactions.py -------------------------------------------------------------------------------- /docker/add_spark_config.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/docker/add_spark_config.sh -------------------------------------------------------------------------------- /docker/entrypoint.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/docker/entrypoint.sh -------------------------------------------------------------------------------- /docker/install_packages.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/docker/install_packages.sh -------------------------------------------------------------------------------- /docker/setup_mockdata.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/docker/setup_mockdata.sh -------------------------------------------------------------------------------- /integrity_tests/conftest.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/integrity_tests/conftest.py -------------------------------------------------------------------------------- /integrity_tests/requirements.txt: -------------------------------------------------------------------------------- 1 | pytest 2 | apache-airflow 3 | coverage 4 | -------------------------------------------------------------------------------- /integrity_tests/test_airflow_home/airflow.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/integrity_tests/test_airflow_home/airflow.cfg -------------------------------------------------------------------------------- /integrity_tests/test_dag_integrity.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/integrity_tests/test_dag_integrity.py -------------------------------------------------------------------------------- /mock_pipeline_requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/mock_pipeline_requirements.txt -------------------------------------------------------------------------------- /slides/PyData Amsterdam 2023 - Return to datas inferno.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/danielvdende/data-testing-with-airflow/HEAD/slides/PyData Amsterdam 2023 - Return to datas inferno.pdf --------------------------------------------------------------------------------