Can You Write a Minimal Activity Deduplicator?
From the earlier reference collection. Read the current software blog.
A minimal deduplicator records stable activity identifiers and treats repeated delivery as an expected retry while keeping processing safe.
Understand why deliveries repeat
Networked systems often retry after a timeout because the sender cannot know whether the receiver completed its work. The same activity can therefore arrive more than once. A deduplicator uses a stable identifier to recognize a repeat and avoid applying the same effect again. It should be based on the activity’s identity, not a guess that two similar-looking payloads are duplicates.
Choose the identifier according to the protocol and producer contract. Store it in a form that preserves its full value, and scope it appropriately if identifiers are only unique within a sender or account. For example, a key might combine an actor identifier with an activity identifier when the protocol’s uniqueness rules require that. Do not deduplicate by message text alone; two legitimate activities can have identical wording.
Record processing state
A minimal table can contain the deduplication key, first-seen time, and processing status. On receipt, attempt to insert the key with a uniqueness constraint. If the insert succeeds, process the activity. If the key already exists, treat the request as a repeat and return the response required by the surrounding protocol without repeating the side effect.
The order matters. If the system marks an identifier as complete before performing the work, a crash can lose the activity permanently. If it performs an irreversible action before recording the key, a crash can cause that action to happen twice. Where possible, put the deduplication record and application change in the same database transaction. For external effects, use an outbox or a separate retryable workflow.
Define safe repeat behavior
A repeated delivery is safe to ignore only if the earlier processing outcome is known. If the first attempt failed halfway through, a second attempt may need to resume or retry work. Keep statuses such as received, processing, completed, and failed only when the implementation can recover them reliably; a single boolean may hide important ambiguity.
Consider an activity that creates a notification. If delivery one commits both the deduplication key and notification record in one transaction, delivery two can return success without inserting another notification. If notification delivery depends on an external service, record a pending delivery and retry that effect independently while still preventing duplicate local records.
Set retention and observability rules
Deduplication records need a retention policy. Keep keys at least as long as the sender may retry or the protocol permits redelivery. Deleting them too early can make an old repeat appear new. If indefinite storage is impractical, document the window and consider compacting completed records while preserving enough identity to recognize likely retries.
Test same-key repeated delivery, distinct identifiers with identical content, a processing failure, and concurrent arrivals. Concurrent attempts should not both win the insert and perform the action. Log whether an activity was new, repeated, or retried after failure, while avoiding unnecessary sensitive payload data. The goal is reliable idempotent handling, not suppressing every event that looks familiar.