What Is a Safe Way to Download a Public Data File?
From the earlier reference collection. Read the current software blog.
Download to a temporary path, check the response and file contents, then replace the existing copy only after validation.
Keep the current copy until the new one is checked
A failed or interrupted download should not erase the last usable data file. Save a new response to a temporary filename in the same directory as the destination, such as feed.zip.part, and only move it into place after checks succeed. Keeping the temporary file on the same filesystem makes the final rename straightforward and generally atomic on local filesystems.
With curl, use failure handling and follow redirects deliberately: curl --fail --location --output feed.zip.part URL. A successful process exit is one check, not a complete validation. A server can return a successful status with an unexpected file, and a proxy or login page may still be saved under a .zip name.
Inspect response metadata and bytes
Request headers separately when investigating a source: curl --head --location URL can show the final response status and headers, including the declared content type. Some servers do not support HEAD consistently, and headers may describe a generic binary response rather than its true format. Treat content type as a clue, not proof.
Check the downloaded bytes with a format-aware tool. For a ZIP file, unzip -t feed.zip.part checks whether its archive members can be read. A plain-text table can be inspected for a plausible header. If the service publishes a checksum over a trusted channel, compute the local digest and compare it with that value. A checksum delivered alongside the file over the same compromised path cannot independently prove authenticity.
Replace only after validation
A safe sequence is: download to a temporary path, check the transfer exit status, inspect the response metadata, validate the file format, compare a trusted checksum when available, and then move the file to its final name. For example, after all checks pass, mv feed.zip.part feed.zip replaces the old path. If a check fails, retain the old feed.zip, report the failure, and decide whether to remove the partial file.
Avoid assuming that a fixed size proves completeness. Public datasets can legitimately change size, and an abruptly truncated file can sometimes still be syntactically readable. A known checksum or version identifier provides a stronger comparison when the publisher supplies one.
Record enough to diagnose changes
For recurring jobs, log the source URL, retrieval time, final status, content type, byte count, and checksum. Do not log credentials embedded in URLs or headers. These details help distinguish a publisher’s new release from a transient error page or a local write problem.
The goal is a controlled handoff: the program should expose the candidate file for inspection, preserve the known-good copy until acceptance, and make replacement a separate final step. That pattern is useful for feeds, archives, and other public downloads whose contents may change over time.