What Does a Sensor’s Reported Precision Mean?
From the earlier reference collection. Read the current software blog.
Decimal places describe how a value is displayed; they do not by themselves establish accuracy or uncertainty.
Digits are a display choice
A reading shown as 18.427 may look exact, but the number of decimal places only tells you how the value is represented. The source may have produced those digits through a calculation or formatting rule, and the instrument may not distinguish changes that small. More displayed digits do not automatically mean a more trustworthy measurement.
Conversely, a value rounded to one decimal place may come from an instrument with finer internal resolution. The display alone cannot establish what the sensor measured or how it was processed. A responsible interface identifies the unit and avoids showing excessive digits that suggest unsupported certainty.
Precision and accuracy differ
Precision describes the fineness or repeatability of reported values in a particular context; accuracy concerns how close a measurement is to the quantity being measured. A sensor can repeatedly produce the same finely displayed number and still have a calibration offset. It can also show coarse decimals while measuring adequately for a practical purpose.
For example, two instruments could both display 20.0 °C while differing slightly in their underlying measurements. The shared decimal format does not prove agreement beyond that displayed rounding. Calibration records, instrument specifications, and uncertainty information provide context that decimal places alone cannot supply.
Read uncertainty in context
A stated uncertainty helps describe limits on interpretation, but its meaning depends on the source’s method and conditions. Check what quantity it applies to, the units, and the circumstances under which it was established. If no uncertainty is supplied, do not treat that absence as evidence of zero uncertainty.
When comparing values, consider both reported precision and uncertainty. A difference smaller than the meaningful resolution of the measurements may not support a strong conclusion. Keep calculations at sufficient precision to avoid rounding errors, then round the published result to a level justified by the inputs and method.
Communicate sensible precision
Choose display digits that aid readers without implying knowledge the measurement does not have. If a source gives 18.427 °C but the instrument’s documented uncertainty is much larger than the final decimal place, a shorter display may be clearer. Preserve the source value in the underlying record and document any rounding used for the interface.
Explain that the displayed format is not a guarantee of accuracy. A reader assessing a sensor should look at the measurement context, calibration information, stated uncertainty, and data processing. Precision is useful when described carefully; on its own, it cannot tell readers whether a sensor’s value is correct.