How Do You Make a Long-Running Import Interruptible?
From the earlier reference collection. Read the current software blog.
Handle termination as a controlled stop, and keep partial output clearly separate from a completed import.
Design for interruption as a normal event
A long import may be stopped because an operator needs to deploy a fix, the input is wrong, or the job is taking too long. Treat an interrupt as an expected control path rather than an exceptional crash. Decide which signals the program handles, what cleanup it can safely perform, and how callers will distinguish a stopped import from a successful one.
The key invariant is simple: incomplete output must not look like a finished dataset. A reader should never see half-written files at the final destination merely because the process received a termination request.
Write into a separate working area
Build the import in a temporary directory or under a clearly marked temporary filename. Keep a manifest or state marker if the job may be resumed, and make it clear that the data is incomplete until final validation passes. For instance, dataset.building/ can hold intermediate tables while dataset/ remains the last completed result.
On an interrupt, stop accepting new work, finish or abandon the current record according to the format, close open files, and leave the temporary area recognizable. Avoid deleting evidence automatically if it could help diagnose a failure or support a deliberate resume.
Make cleanup bounded and safe
Signal handlers should do very little: record that shutdown was requested, then let the normal control flow clean up at a safe point. Complex file operations inside a handler can interrupt code while it holds locks or is updating internal state. In a worker-based importer, communicate the stop request to workers and wait for them to exit within a defined policy.
If an import writes a database, use transactions or staging tables where available. A file-based design can write temporary outputs, validate them, then promote them only when complete. The mechanism differs, but the goal is the same: readers see either the previous complete version or the new complete version.
Report the outcome precisely
Return a status that indicates interruption, and report which stage stopped. Do not print “import complete” from a generic cleanup block that runs on both success and termination. Log the temporary location if it remains, and state whether it is safe to resume or should be discarded; only promise resumability if the program tracks enough state to do so.
Try interrupting a small test import during reading, transformation, and final writing. Confirm that the prior completed output remains usable, the temporary output is marked incomplete, and a later run does not mistake it for success.