How Should Sensor Spikes Be Investigated?
From the earlier reference collection. Read the current software blog.
An unusual sensor value deserves investigation before removal. Compare nearby readings, source notes, and operating context, then document the decision.
Treat an outlier as a question
A sharp jump can signal a real event, an instrument problem, a transmission issue, or a transcription error. The shape of a plot alone cannot tell which explanation is correct. Removing every point that looks surprising risks deleting the very event a reader needs to understand; accepting every point without review can give a faulty measurement undue weight.
Start by describing the observation precisely. Record its station, timestamp, value, unit, and any quality flag. Then compare the point with observations immediately before and after it. A single isolated jump followed by a quick return is a reason to investigate, not proof that the value is wrong.
Compare the surrounding evidence
Check neighboring observations from the same instrument and, where useful, other nearby stations. Ask whether the change appears in more than one series and whether it persists. A rapid local change may be plausible under some conditions, while a one-record spike surrounded by steady readings may suggest a transient fault. Neither pattern is conclusive on its own.
Review the source notes and operating context. Maintenance records, sensor replacement, calibration notes, power interruptions, or known collection changes can explain discontinuities. If the data provider publishes quality flags, learn what each flag means before using it to filter values. A flag may identify a review condition without declaring a measurement invalid.
Make a reversible decision
Keep the original observation available even if an analysis excludes it. Store any review result separately, with the reason, date, and rule or evidence used. For example, an analyst might mark a point “suspect: isolated jump, provider review pending” rather than replacing it with the average of its neighbors. That preserves the evidence and avoids presenting an invented value as measured data.
If a downstream calculation needs a clean series, make the inclusion rule explicit. One chart might show all measurements with flagged points marked; a second analysis might omit a defined set of suspect records. Readers can then see how the decision affects the result.
Explain uncertainty plainly
When the evidence remains inconclusive, say so. A concise status note can identify the unusual reading, explain which checks were made, and state whether it remains included. If the provider later clarifies the record, update the note and keep a trace of the change.
Good spike investigation is disciplined documentation. Compare context, keep raw data, and make exclusions explainable and reversible.