├── .gitignore ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── data ├── README.md └── sample_data.csv ├── data_quality ├── README.md ├── custom_expectations │ ├── README.md │ ├── __init__.py │ ├── expect_column_length_match_input_length.py │ ├── expect_column_pair_a_to_be_approximately_smaller_or_equal_than_b.py │ └── expect_multicolumn_customer_id_user_id_device_id.py ├── generate_data_doc │ └── generate_expectation_suite_doc_site.py ├── suite_dev_notebooks │ └── expectation_suite_template.ipynb └── validate_data │ ├── data_validation_with_checkpoints.py │ └── data_validation_with_checkpoints_template.ipynb ├── expectation_suites └── sample_data │ └── data_quality_check.json ├── images └── ge_steps.png └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/.gitignore -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/Dockerfile -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/LICENSE -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/Makefile -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/README.md -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data/README.md -------------------------------------------------------------------------------- /data/sample_data.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data/sample_data.csv -------------------------------------------------------------------------------- /data_quality/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/README.md -------------------------------------------------------------------------------- /data_quality/custom_expectations/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/custom_expectations/README.md -------------------------------------------------------------------------------- /data_quality/custom_expectations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/custom_expectations/__init__.py -------------------------------------------------------------------------------- /data_quality/custom_expectations/expect_column_length_match_input_length.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/custom_expectations/expect_column_length_match_input_length.py -------------------------------------------------------------------------------- /data_quality/custom_expectations/expect_column_pair_a_to_be_approximately_smaller_or_equal_than_b.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/custom_expectations/expect_column_pair_a_to_be_approximately_smaller_or_equal_than_b.py -------------------------------------------------------------------------------- /data_quality/custom_expectations/expect_multicolumn_customer_id_user_id_device_id.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/custom_expectations/expect_multicolumn_customer_id_user_id_device_id.py -------------------------------------------------------------------------------- /data_quality/generate_data_doc/generate_expectation_suite_doc_site.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/generate_data_doc/generate_expectation_suite_doc_site.py -------------------------------------------------------------------------------- /data_quality/suite_dev_notebooks/expectation_suite_template.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/suite_dev_notebooks/expectation_suite_template.ipynb -------------------------------------------------------------------------------- /data_quality/validate_data/data_validation_with_checkpoints.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/validate_data/data_validation_with_checkpoints.py -------------------------------------------------------------------------------- /data_quality/validate_data/data_validation_with_checkpoints_template.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/data_quality/validate_data/data_validation_with_checkpoints_template.ipynb -------------------------------------------------------------------------------- /expectation_suites/sample_data/data_quality_check.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/expectation_suites/sample_data/data_quality_check.json -------------------------------------------------------------------------------- /images/ge_steps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MDS-BD/hands-on-great-expectations-with-spark/HEAD/images/ge_steps.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | great-expectations==0.15.20 2 | pyspark==3.1.1 --------------------------------------------------------------------------------