What Should Be Logged When a Public Data Refresh Fails?
From the earlier reference collection. Read the current software blog.
Operational logs should make a failed refresh diagnosable while protecting credentials and other sensitive details.
Record what was attempted
A useful failure record begins with the identity of the source and the time the refresh was attempted. Include the job or dataset name, the requested date range or partition when relevant, and a request or run identifier that lets maintainers connect related events. These details help distinguish one feed’s failure from another without copying the entire payload into a log.
For example, a record might say that the hourly station feed refresh for 10:00 UTC began at 10:03 UTC and failed during response parsing. The attempted time and affected period are both useful: a delayed job can start late but still be processing the correct data window.
Describe the failure and recovery
Capture the failure stage and a concise reason, such as a connection timeout, invalid file structure, or rejected authentication. Preserve a safe error code or short message if it helps diagnosis, but do not dump unfiltered response bodies. A source response could contain personal data or credentials, and a debug log can accidentally retain more than the job needs.
Record whether a retry occurred, whether it succeeded, and when the data became current again. This creates a practical history of the outage and helps the dashboard report freshness honestly. If the refresh partially succeeded, identify which partitions or records were committed so maintainers do not assume the entire dataset was updated.
Protect secrets and keep logs actionable
Never log API keys, passwords, authorization headers, private tokens, or signed URLs. Redact secrets before recording exceptions, since libraries sometimes include request details in error messages. Limit access and retention to what operational support requires, and avoid storing full raw records when a count or identifier is enough.
Use consistent fields for timestamps, source names, outcome, and error class. A structured record is easier to search than a paragraph assembled differently by every job. Include a stable correlation ID for related retry attempts, while keeping each attempt’s timestamp and result separate.
Connect operations to reader-facing status
Logs are for diagnosis; a dashboard status is for readers. Translate the operational event into a simple message such as “Latest update delayed; last successful refresh at 09:00 UTC.” Include the recovery time once confirmed. This connects internal monitoring to a clear account of what the public data currently can and cannot show.