How Can You Make a Data Download Resumable?
From the earlier reference collection. Read the current software blog.
Resume from a partial file only when the server supports byte ranges, then verify the completed download before publishing it as usable data.
Confirm the server can resume
A resumable download continues from an existing partial file by requesting a byte range. The server must support that behavior and return a response consistent with the requested range. Do not assume that every endpoint supports ranges or that a client option alone guarantees a safe continuation.
With curl, -C - asks it to continue a transfer based on the local file size. Check the response and client result; if the server ignores the range or the resource changed, appending bytes may create a corrupt file.
Keep partial and complete states distinct
Download into a temporary name such as feed.zip.part, and reserve feed.zip for a verified result. The partial file is useful for a retry, but other jobs should not mistake it for a complete archive. Keep the URL and relevant metadata alongside it if the retry process needs to know which representation it belongs to.
A validator such as an entity tag can help detect whether the remote object changed between attempts, when the server and client support the relevant behavior. If the remote representation no longer matches the partial data, restart from the beginning instead of joining incompatible pieces.
Verify after transfer
A successful transfer status only says that the client completed its request; it does not prove that the file is the intended dataset. Check a trusted checksum if one is provided, test archive integrity for an archive, and run the format-specific validation before promoting the file to its final name. Each check answers a different question.
If there is no expected checksum, you can still verify basic properties such as a nonempty file, readable archive members, and expected schema. Be careful not to describe such checks as proof that the content is authentic or current.
Make retry behavior predictable
On a failed attempt, preserve the partial data only when the next run can safely identify and validate it. Otherwise discard it and begin again. Report whether the transfer resumed, restarted, or failed, and keep network errors distinct from validation errors so an operator knows which step to investigate.
Once the file passes the required checks, rename it into the completed location and allow ingestion to begin. The boundary between .part and final output keeps retries practical without letting an incomplete download masquerade as a finished feed.