Great Expectations Walkthrough
224 | 227 |How to work with Great Expectations
234 | 236 | Welcome! Now that you have initialized your project, the best way to work with Great Expectations is in this iterative dev loop: 237 |
238 |-
239 |
- Let Great Expectations create a simple first draft suite, by running
great_expectations suite new.
240 | - View the suite here in Data Docs. 241 |
- Edit the suite in a Jupyter notebook by running
great_expectations suite edit
242 | - Repeat Steps 2-3 until you are happy with your suite. 243 |
- Commit this suite to your source control repository. 244 |
What are Expectations?
-
271 |
- expect_column_to_exist 272 |
- expect_table_row_count_to_be_between 273 |
- expect_column_values_to_be_unique 274 |
- expect_column_values_to_not_be_null 275 |
- expect_column_values_to_be_between 276 |
- expect_column_values_to_match_regex 277 |
- expect_column_mean_to_be_between 278 |
- expect_column_kl_divergence_to_be_less_than 279 |
- ... and many more 280 |
An expectation is a falsifiable, verifiable statement about data.
282 |Expectations provide a language to talk about data characteristics and data quality - humans to humans, humans to machines and machines to machines.
283 |Expectations are both data tests and docs!
284 |Expectations can be presented in a machine-friendly JSON
310 | {
311 | "expectation_type": "expect_column_values_to_not_be_null",
312 | "kwargs": {
313 | "column": "user_id"
314 | }
315 | }
316 |
A machine can test if a dataset conforms to the expectation.
318 |Validation produces a validation result object
343 | {
344 | "success": false,
345 | "result": {
346 | "element_count": 253405,
347 | "unexpected_count": 7602,
348 | "unexpected_percent": 2.999
349 | },
350 | "expectation_config": {
351 | "expectation_type": "expect_column_values_to_not_be_null",
352 | "kwargs": {
353 | "column": "user_id"
354 | }
355 | }
356 |
Here's an example Validation Result (not from your data) in JSON format. This object has rich context about the test failure.
358 |Validation results save you time.
384 | This is an example of what a single failed Expectation looks like in Data Docs. Note the failure includes unexpected values from your data. This helps you debug pipelines faster.
386 |Great Expectations provides a large library of expectations.
412 | Nearly 50 built in expectations allow you to express how you understand your data, and you can add custom 414 | expectations if you need a new one. 415 |
416 |Now explore and edit the sample suite!
This sample suite shows you a few examples of expectations.
441 |Note this is not a production suite and was generated using only a small sample of your data.
442 |When you are ready, press the How to Edit button to kick off the iterative dev loop.
443 |
234 | 
384 | 
412 |