How Should an Outage Feed Consumer Handle Duplicate Updates?
From the earlier reference collection. Read the current software blog.
A consumer should define event identity, make processing idempotent, and retain enough state to recover safely when updates arrive more than once.
Assume delivery can repeat
A feed consumer may see the same update twice after a timeout, restart, or upstream replay. That does not necessarily mean two outages occurred. The safe starting assumption is that delivery can repeat, and that the consumer needs a stable way to decide whether an update has already been applied.
First identify what the source actually provides: a durable event identifier, a revision number, or only a collection of fields. Do not assume that two records with identical text are always the same event; two separate locations could report similar conditions. Identity must reflect the source’s semantics.
Choose a deduplication key
If an update has a stable event ID, use it as the primary key. Store the ID and the result of processing in one durable transaction when possible. On receiving an existing ID, skip the side effect and optionally record that a duplicate delivery occurred.
If no ID exists, a composite key can be a fallback: for example, source, affected area, event type, and source-provided update time. This can still collide or change when a correction arrives. A content hash is useful for exact repeated payloads, but a harmless formatting change creates a different hash. Document the compromise rather than treating the key as certainty.
Make effects idempotent
Deduplication at ingestion is not enough if a crash can occur between saving the update and sending a notification. Consider a consumer that inserts an event, then emails an operator, then crashes before recording the email. On restart, it may send the email again. Use an outbox table or another durable workflow: commit the event and a pending notification together, then mark the notification sent after delivery.
Keep corrections distinct from duplicates. If the source revises an event, preserve its identity and apply the newer revision, or record a new version linked to the original. A monotonic revision number makes this decision clearer than comparing arrival order, since delayed messages can arrive after newer ones.
Test restart and recovery assumptions
Build a small test sequence: deliver event A, deliver A again, deliver a revised A, then restart the consumer and replay the sequence. Verify that the event count stays stable, the correction is visible, and notifications follow the policy you intended. Also test two similar events for different areas to ensure the key does not merge them.
An outage information page gives readers a practical context for asking what counts as one update and how fresh it appears. Compare your consumer’s deduplication assumptions with the visible information, but do not infer an undocumented feed protocol from the page. The useful result is a policy that says what the system can safely count and what requires human review.