How Would You Build a Data Freshness Report?
From the earlier reference collection. Read the current software blog.
A freshness report summarizes observation and retrieval times separately so readers can see when data was measured and when it was obtained.
Give the timestamps distinct meanings
A reliable freshness report should not collapse different clocks into one “last updated” value. The observation time says when the source says a measurement or event occurred. The retrieval time says when your system obtained it. Those times can diverge because of delayed publication, polling intervals, or upstream outages. Label them separately so a reader can reason about the data’s age.
For each configured source, decide whether an observation timestamp is available and trustworthy. A feed may provide a timestamp per record, while another source may only expose a retrieval time. Do not manufacture an observation time from the retrieval time. If no observation timestamp exists, show “not provided” and explain that freshness can only be assessed from retrieval history.
Collect a compact record
Store a source key, the most recent valid observation time, the latest successful retrieval time, and the last retrieval outcome. If the source contains many records, define what “most recent observation” means: the maximum valid timestamp across the selected dataset, or the timestamp of a particular station or entity. State that scope near the report.
For example, suppose a station’s newest observation is 09:10 and the dashboard retrieved the source at 09:18. The report can show both values and calculate an eight-minute difference if their time zones are known. If the source timestamp lacks a zone, do not present a precise age as though it were comparable to local time. Preserve the raw value and mark the ambiguity.
Calculate status without hiding details
Set freshness thresholds that match the source’s expected update schedule. A source intended to update hourly needs a different threshold from one published daily. Display the configured threshold or the age used for a status such as “current,” “delayed,” or “stale.” These labels are summaries; the actual timestamps should remain visible so users can judge the situation themselves.
A failed retrieval must not replace the last successful retrieval time with the failure time. Instead, show the failure time and retain the last success. This distinction answers two separate questions: when did the system last check, and when did it last obtain usable data? Mark unavailable, invalid, and stale states separately when those distinctions can be established.
Make the report useful during incidents
Order sources consistently and include a generated-at time for the report itself. If a source is missing from the configuration or has never succeeded, display that state rather than omitting the row. A report that silently drops unhealthy sources can look reassuring precisely when it is least useful.
Test current data, delayed observation timestamps, a retrieval outage, a malformed timestamp, and a source with no observation field. Keep sample fixtures with explicit time zones so expected ages are deterministic. A freshness report cannot make upstream data newer, but it can make the age and limits of the available information visible at a glance.