What Is a Good Data Check for Impossible Sensor Values?
From the earlier reference collection. Read the current software blog.
Use documented domain checks to flag values that cannot fit the measurement context, and route them for review rather than silently changing them.
Define what impossible means
A useful validation check begins with the quantity being measured and the conditions under which it is collected. The acceptable domain for one sensor may differ from that of another, so a generic cutoff can reject legitimate data or admit obvious errors. Write down the measurement definition, unit, instrument context, and the reason for each boundary before enforcing it.
For example, a check for a water-temperature feed might reject a value that cannot be represented in the documented unit or mark a value outside an instrument’s stated operating range for review. Those are distinct rules: one may indicate malformed data, while the other may indicate a value that needs contextual investigation.
Separate checking from correction
A validator should identify a suspect record without silently clipping it to a boundary. If a submitted value exceeds a configured limit, changing it to the nearest allowed number creates a measurement that was never reported. It also hides the original error from anyone trying to diagnose the source.
Retain the received value, the validation result, and the rule that was triggered. A structured record might include the original reading, a status such as “outside configured range,” and a review state. That allows a dashboard to suppress or visually mark the value while preserving an audit trail.
Use layered checks
Basic checks can confirm that a value is numeric, that the unit is recognized, and that required fields are present. Domain checks can compare the value with documented instrument or application limits. Context checks can flag a sudden change from nearby readings, but those should generally be treated as review prompts rather than hard impossibility rules: a real event can cause an abrupt change.
Consider a stream that suddenly reports a negative value where the documented sensor definition does not permit one. The system can flag the record, retain it, and show a clear status while an operator checks the source and device context. It should not automatically substitute zero.
Make rules reviewable
Keep checks versioned and explain them in plain language. When a threshold changes, record who or what changed it and when, so users can understand why older and newer records received different statuses. Include a way to distinguish “failed validation,” “needs review,” and “accepted” if those states lead to different display or analysis behavior.
A good check protects interpretation without erasing evidence. Document the domain, flag uncertain cases, and let corrections happen through an explicit process.