What is an Anomaly?
- An
Anomaly
is an anomalous data set (record or column) that asserted false to applied data quality check(s). BothInferred
andAuthored
Checks generate Anomalies, and are batched together when applied through aScan Operation
to highlight anomalies:
There are two types of anomalies in Qualytics: Record
and Shape
:
- A
Record
Anomaly is at the record-level, where a specific field or a combination of fields are anomalous. - A
Shape
Anomaly is at the column-level, where a field's shapes and patterns are being impacted at a higher level.
Note
In either anomaly type, source records are exposed as part of Anomaly Details
. A Record anomaly will highlight the specific record, and a Shape anomaly will highlight 10 samples from underlying anomalous records.
- Shape Anomaly view
- Record Anomaly view
-
When a Scan is run, Qualytics will highlight anomalies with the following information:
Table
/File
: theTable
orFile
of the anomalyField
: the field(s) of the anomalyLocation
: fully qualified location of the anomalyRule
:Infered
and/orAuthored
checks that failed assertionsDescription
: human-readable, auto-generated description of theAnomaly
Status
: The status of the anomaly. If it'sactive
,acknowledged
,resolved
orinvalid
Type
:Record
orShape
Tag
: tag(s) / label(s) associated with an anomalyDate time
: date/time when the anomaly was found
Note
The New
Scan Operation
.
Status
-
The
Anomaly Status
can be set toActive
,Acknowledged
,Resolved
andInvalid
. While this enables users to better maintain a worklist through anomalies, it is also a mechanism of providing feedback to the system. Specifically, learning methods are tuned according to the feedback provided by the user - invalidating an anomaly will mean that the tolerances of the checks that caught the anomaly will be updated going forward.Active
: The anomaly is active and needs to be addressedAcknowledged
: The anomaly is valid, has been acknowledged but kept active in the Anomaly worklistResolved
: The anomaly is valid and has been resolved, therefore removed from the Anomaly worklistInvalid
: The anomaly is not valid, removed from the Anomaly worklist and rules updates are suggested to inference engine.