Writing a Runbook for Intermittent Public Data Feeds
From the earlier reference collection. Read the current software blog.
A useful runbook gives operators a short sequence for checking connectivity, freshness, response validity, and safe recovery while preserving evidence for follow-up.
Start with the visible symptom
Write the first steps around what an operator can observe: a dashboard has not changed, a scheduled import failed, or a result is older than expected. State where to find the latest successful retrieval time and how to distinguish a stale snapshot from an empty but valid response.
Name the system and its normal freshness target if one has been established. Avoid vague directions such as “check the feed.” A better instruction identifies the command, dashboard, or log field to inspect, then explains what outcome warrants the next step.
Check the failure boundary
Use a fixed order: confirm the host has network access, inspect the last request result, check whether the response is empty or malformed, and review the parser’s validation errors. This avoids restarting the entire pipeline before knowing whether the failure lies in connectivity, the source response, or local code.
For example, a timeout followed by a valid cached result points toward a transient retrieval problem. A successful response that lacks a required field points toward a content change or parsing assumption. Record the timestamp, status, and a sanitized error message before changing anything.
Recover without losing evidence
Document a safe restart command and what it changes. If the importer uses a temporary file, explain how to confirm that the previous complete output remains intact before replacing it. Prefer rerunning an idempotent fetch over deleting a cache or database whose contents may be the only usable snapshot.
Include stop conditions. If repeated attempts fail, preserve the last known good data, report its age, and escalate with the collected details. Do not advise operators to bypass validation merely to make a job appear green; invalid data can be more misleading than a visible failure.
Make the handoff concrete
A runbook should end with the information needed for another person to continue: run identifier, first failure time, last successful retrieval, affected output, actions taken, and current user impact. Keep credentials out of the handoff and provide a place to store restricted diagnostics if they are necessary.
An outage update is a helpful example because people may need to distinguish a real status change from a delayed refresh. The operator should be able to say whether the latest visible information is current, when it was retrieved, and what prevented a new result. A public outage page gives context for those questions without defining a particular feed implementation.