A Field Guide to Monitoring a Public Data Import Job
From the earlier reference collection. Read the current software blog.
Monitor an unattended importer with a compact record of fetch time, parsing outcome, row counts, and safe failure details that help establish whether its data is trustworthy.
Record the stages separately
An import job is a chain of steps: retrieve a response, decode it, validate its structure, transform records, and store the result. A single “job failed” message hides where the problem occurred. Emit a structured record for each run with a run identifier, start and finish times, and a stage outcome.
For example, a JSON importer might report retrieval success, decoding success, validation failure, and zero rows written. That tells an operator not to interpret the empty table as a valid empty dataset. A later run can use the same stages, making changes in failure patterns easier to spot.
Measure freshness and volume
Store the fetch completion time separately from the time the job ran. A process can start on schedule yet retrieve stale content from a cache or receive an old snapshot. If the source provides its own publication or update time, retain that as a separate value and avoid treating it as equivalent to retrieval time.
Track rows read, accepted, rejected, and written. Counts should reconcile: if 120 records were read, 117 accepted, and 3 rejected, an operator can understand the output. A sudden change from hundreds of rows to zero merits inspection, but do not make an arbitrary row threshold a universal failure rule; some datasets legitimately have no entries at a particular moment.
Keep logs safe and useful
Include an error class and a short sanitized explanation, such as “connection timed out” or “required field missing.” Exclude authorization headers, tokens, private keys, and full URLs if query parameters may contain secrets. Avoid dumping entire response bodies into routine logs; retain a carefully controlled diagnostic sample only when it is genuinely needed.
Use stable field names and machine-readable output such as one JSON object per run. A shell operator can then filter for failed stages or stale fetches without scraping prose. Set a distinct nonzero process status for failure so a scheduler can detect it, while preserving the detailed stage and cause in the log.
Use the information to assess trust
Define what a healthy run means: a recent retrieval, a valid parse, reasonable accounting of rows, and a successful commit. Show the latest successful import time even when a later attempt fails. That makes it possible to distinguish “the job is broken” from “the last known data is aging.”
For an outage information page, ask what an operator would need to know before trusting the imported result: when it was fetched, whether it parsed, and whether its size changed unexpectedly. The page offers a concrete subject for setting those questions; it does not establish an underlying API or promise a particular refresh schedule.