├── .github ├── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── PULL_REQUEST_TEMPLATE.md └── workflows │ └── base.yml ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Dockerfile ├── LICENSE ├── README.md ├── docs ├── Makefile ├── analyzers.md ├── anomaly_detection.md ├── checks.md ├── make.bat ├── profiles.md ├── repository.md ├── source │ ├── README.md │ ├── api.rst │ ├── conf.py │ ├── index.rst │ ├── pydeequ.rst │ └── requirements.txt ├── suggestions.md └── verification.md ├── imgs ├── pydeequ_architecture.jpg └── pydeequ_deployment.png ├── poetry.lock ├── pydeequ ├── __init__.py ├── analyzers.py ├── anomaly_detection.py ├── check_functions.py ├── checks.py ├── configs.py ├── metrics.py ├── pandas_utils.py ├── profiles.py ├── repository.py ├── scala_utils.py ├── suggestions.py └── verification.py ├── pyproject.toml ├── tests ├── __init__.py ├── conftest.py ├── test_analyzers.py ├── test_anomaly_detection.py ├── test_checks.py ├── test_config.py ├── test_pandas_utils.py ├── test_profiles.py ├── test_repository.py ├── test_scala_utils.py └── test_suggestions.py └── tutorials ├── KLLCheckExample.ipynb ├── KLLExample.ipynb ├── analyzers.ipynb ├── anomaly_detection.ipynb ├── basic_example.ipynb ├── constraint_suggestion_example.ipynb ├── data_profiling_example.ipynb ├── glue_tutorials ├── advanced_anomaly_detection_tutorial.ipynb ├── basic_anomalies_script_demo.ipynb ├── data_cleaning_tutorial.ipynb ├── integrate_pydeequ_glue.md └── simple_metrics_repository_tutorial.ipynb ├── hasPattern_check.ipynb ├── profiles.ipynb ├── repository.ipynb ├── repository_file_dbfs.ipynb ├── suggestions.ipynb ├── synthetic_data ├── 01-synthetic-data-electronics.ipynb ├── 02-synthetic-data-jewelry.ipynb ├── 03-synthetic-data-other-products.ipynb └── review_generation_helpers.py ├── test_data_quality_at_scale.ipynb └── verification.ipynb /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/.github/ISSUE_TEMPLATE/bug_report.md -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/.github/ISSUE_TEMPLATE/feature_request.md -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/.github/PULL_REQUEST_TEMPLATE.md -------------------------------------------------------------------------------- /.github/workflows/base.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/.github/workflows/base.yml -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/.gitignore -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/CODE_OF_CONDUCT.md -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/CONTRIBUTING.md -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/Dockerfile -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/README.md -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/Makefile -------------------------------------------------------------------------------- /docs/analyzers.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/analyzers.md -------------------------------------------------------------------------------- /docs/anomaly_detection.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/anomaly_detection.md -------------------------------------------------------------------------------- /docs/checks.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/checks.md -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/make.bat -------------------------------------------------------------------------------- /docs/profiles.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/profiles.md -------------------------------------------------------------------------------- /docs/repository.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/repository.md -------------------------------------------------------------------------------- /docs/source/README.md: -------------------------------------------------------------------------------- 1 | ../../README.md -------------------------------------------------------------------------------- /docs/source/api.rst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/source/api.rst -------------------------------------------------------------------------------- /docs/source/conf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/source/conf.py -------------------------------------------------------------------------------- /docs/source/index.rst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/source/index.rst -------------------------------------------------------------------------------- /docs/source/pydeequ.rst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/source/pydeequ.rst -------------------------------------------------------------------------------- /docs/source/requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/source/requirements.txt -------------------------------------------------------------------------------- /docs/suggestions.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/suggestions.md -------------------------------------------------------------------------------- /docs/verification.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/docs/verification.md -------------------------------------------------------------------------------- /imgs/pydeequ_architecture.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/imgs/pydeequ_architecture.jpg -------------------------------------------------------------------------------- /imgs/pydeequ_deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/imgs/pydeequ_deployment.png -------------------------------------------------------------------------------- /poetry.lock: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/poetry.lock -------------------------------------------------------------------------------- /pydeequ/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/__init__.py -------------------------------------------------------------------------------- /pydeequ/analyzers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/analyzers.py -------------------------------------------------------------------------------- /pydeequ/anomaly_detection.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/anomaly_detection.py -------------------------------------------------------------------------------- /pydeequ/check_functions.py: -------------------------------------------------------------------------------- 1 | def is_one(x): 2 | return x == 1 / 1 3 | -------------------------------------------------------------------------------- /pydeequ/checks.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/checks.py -------------------------------------------------------------------------------- /pydeequ/configs.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/configs.py -------------------------------------------------------------------------------- /pydeequ/metrics.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/metrics.py -------------------------------------------------------------------------------- /pydeequ/pandas_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/pandas_utils.py -------------------------------------------------------------------------------- /pydeequ/profiles.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/profiles.py -------------------------------------------------------------------------------- /pydeequ/repository.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/repository.py -------------------------------------------------------------------------------- /pydeequ/scala_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/scala_utils.py -------------------------------------------------------------------------------- /pydeequ/suggestions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/suggestions.py -------------------------------------------------------------------------------- /pydeequ/verification.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pydeequ/verification.py -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/pyproject.toml -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/conftest.py -------------------------------------------------------------------------------- /tests/test_analyzers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_analyzers.py -------------------------------------------------------------------------------- /tests/test_anomaly_detection.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_anomaly_detection.py -------------------------------------------------------------------------------- /tests/test_checks.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_checks.py -------------------------------------------------------------------------------- /tests/test_config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_config.py -------------------------------------------------------------------------------- /tests/test_pandas_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_pandas_utils.py -------------------------------------------------------------------------------- /tests/test_profiles.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_profiles.py -------------------------------------------------------------------------------- /tests/test_repository.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_repository.py -------------------------------------------------------------------------------- /tests/test_scala_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_scala_utils.py -------------------------------------------------------------------------------- /tests/test_suggestions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tests/test_suggestions.py -------------------------------------------------------------------------------- /tutorials/KLLCheckExample.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/KLLCheckExample.ipynb -------------------------------------------------------------------------------- /tutorials/KLLExample.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/KLLExample.ipynb -------------------------------------------------------------------------------- /tutorials/analyzers.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/analyzers.ipynb -------------------------------------------------------------------------------- /tutorials/anomaly_detection.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/anomaly_detection.ipynb -------------------------------------------------------------------------------- /tutorials/basic_example.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/basic_example.ipynb -------------------------------------------------------------------------------- /tutorials/constraint_suggestion_example.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/constraint_suggestion_example.ipynb -------------------------------------------------------------------------------- /tutorials/data_profiling_example.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/data_profiling_example.ipynb -------------------------------------------------------------------------------- /tutorials/glue_tutorials/advanced_anomaly_detection_tutorial.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/glue_tutorials/advanced_anomaly_detection_tutorial.ipynb -------------------------------------------------------------------------------- /tutorials/glue_tutorials/basic_anomalies_script_demo.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/glue_tutorials/basic_anomalies_script_demo.ipynb -------------------------------------------------------------------------------- /tutorials/glue_tutorials/data_cleaning_tutorial.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/glue_tutorials/data_cleaning_tutorial.ipynb -------------------------------------------------------------------------------- /tutorials/glue_tutorials/integrate_pydeequ_glue.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/glue_tutorials/integrate_pydeequ_glue.md -------------------------------------------------------------------------------- /tutorials/glue_tutorials/simple_metrics_repository_tutorial.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/glue_tutorials/simple_metrics_repository_tutorial.ipynb -------------------------------------------------------------------------------- /tutorials/hasPattern_check.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/hasPattern_check.ipynb -------------------------------------------------------------------------------- /tutorials/profiles.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/profiles.ipynb -------------------------------------------------------------------------------- /tutorials/repository.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/repository.ipynb -------------------------------------------------------------------------------- /tutorials/repository_file_dbfs.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/repository_file_dbfs.ipynb -------------------------------------------------------------------------------- /tutorials/suggestions.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/suggestions.ipynb -------------------------------------------------------------------------------- /tutorials/synthetic_data/01-synthetic-data-electronics.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/synthetic_data/01-synthetic-data-electronics.ipynb -------------------------------------------------------------------------------- /tutorials/synthetic_data/02-synthetic-data-jewelry.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/synthetic_data/02-synthetic-data-jewelry.ipynb -------------------------------------------------------------------------------- /tutorials/synthetic_data/03-synthetic-data-other-products.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/synthetic_data/03-synthetic-data-other-products.ipynb -------------------------------------------------------------------------------- /tutorials/synthetic_data/review_generation_helpers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/synthetic_data/review_generation_helpers.py -------------------------------------------------------------------------------- /tutorials/test_data_quality_at_scale.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/test_data_quality_at_scale.ipynb -------------------------------------------------------------------------------- /tutorials/verification.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/python-deequ/HEAD/tutorials/verification.ipynb --------------------------------------------------------------------------------