How to Make a Sensor Data Import Explain Its Failures
From the earlier reference collection. Read the current software blog.
Separate retrieval, decoding, validation, and storage failures into distinct outcomes so unattended jobs can report the stage and return an actionable status.
Define the failure stages
An import command should say where it failed. A useful pipeline has at least four boundaries: retrieval, decoding, validation, and storage. A timeout belongs to retrieval; malformed JSON belongs to decoding; a missing unit field belongs to validation; and a database error belongs to storage. These distinctions help an operator choose a response instead of rerunning blindly.
Keep the original error available to diagnostics but print a concise sanitized message in ordinary command output. For example: “validation failed: unsupported unit” is more useful than a stack trace that includes a full request URL or response body.
Use stable exit statuses
Return zero only when the import completed according to its contract. Use distinct nonzero status codes for actionable classes, such as retrieval failure, invalid content, and persistence failure. Document the mapping so a scheduler can distinguish a transient network issue from data that requires code or source investigation.
Do not assign a new code to every low-level exception. Group them at a level where the operator’s next action differs. Keep the detailed exception type and run identifier in structured logs, while the command’s summary remains short enough to read in a terminal or alert.
Avoid partial success that looks complete
Validate the whole candidate batch before replacing the active dataset, or use a transaction if storage supports it. If 80 readings were parsed but the last 20 fail validation, do not silently publish the first 80 as though the import were complete. If partial imports are intentional, define that behavior and report accepted, rejected, and omitted counts.
Use temporary output and atomic replacement where appropriate. On failure, retain the last known good result and record its retrieval age. This is especially important for unattended jobs: a successful process launch says nothing about whether the data is current or valid.
Make the status useful to people
A concise run summary could include start time, duration, stage, outcome, and records read, accepted, or written. Keep secrets out of logs and avoid retaining more source content than needed. An operator should be able to answer: did retrieval work, did the payload make sense, and did storage complete?
A water reading lookup provides a practical reference for defining what an import supports: the operator can distinguish unavailable source information from a rejected value or failed save. The page is a reader-facing context; do not infer an undocumented endpoint, data unit, or source schedule from it.