Data Quality Scores
- Dean Nguyen
- Chichi
DQ Score
All DQ test results are converted to a DQ score out of 100.
For example, if a Snowflake DMF detects 10 failed records out of 100 records checked, the DQ score for the Snowflake DMF is 90.
Some DQ tools which are integrated with K only provide PASS or FAILED. In this case the DQ core is either 100 (PASS) or 0 (FAILED).
DQ Score by Dimension
When a DQ Test is assigned to a DQ Dimension, the DQ Score is aggregated to that dimension.
If multiple tests are assigned to the same dimension, the dimension scores are averaged across the tests.
For example with the following tests:
Not null test with a score of 100 assigned to COMPLETENESS dimension
Value in list test with a score of 50 assigned to COMPLETENESS dimension
The COMPLETENESS dimension will have a score of 75 (150/2)
Overall DQ Score
The Overall DQ score for an asset is an average of recent DQ tests for that asset
It is the average of the most recent score for each DQ tests run against an asset.
For example with the following tests:
Not null test with a score of 100 assigned to COMPLETENESS dimension
Format test with a score of 50 assigned to CONSISTENCY dimension
Uniqueness test with a score of 0 assigned to NO dimension
The Overall DQ score is 50 (150/3)
DQ Score Delta
The DQ Score Delta is a score that tracks changes between the DQ score. If the DQ score increases or decreases between the start and end of the window defined by dq_lookback_window (days) setting, the DQ Score Delta tracks the difference (positive or negative).
For example
Day 1: Overall DQ score is 100
Day 7: Overall DQ score is 80
The DQ Score Delta calculated will be -20.
Use the DQ Score Delta to configure alerts to notify key users of significant changes to the DQ score.