Skip to main content

Idempotency & duplicates

An idempotency key identifies "this exact logical step," independent of how many times the underlying message gets delivered. Attaching a policy means a re-delivered or replayed event is caught and skipped instead of running your handler twice.

Attaching a policy​

from pymidil.event.idempotency import IdempotencyPolicy, InMemoryIdempotencyStore

consumer.use_idempotency(IdempotencyPolicy(InMemoryIdempotencyStore()))

Choosing a key​

By default, the policy uses your message's idempotency_key if you set one, falling back to the message's own ID if you didn't:

message = Message(
    id="EVT-1",
    body={"amount": 50},
    idempotency_key="ORDER-42:PaymentAuthorized",  # your own logical key
)

A good key identifies the logical operation, not the delivery. For example, f"{order_id}:PaymentAuthorized" rather than a per-attempt message ID, so retries and redeliveries of the same logical step collide on purpose. If you need custom logic (composing a key from several fields, for example), pass a key_fn when constructing the policy:

IdempotencyPolicy(InMemoryIdempotencyStore(), key_fn=lambda msg: f"{msg.body['order_id']}:{msg.body['step']}")

Choosing a store​

  • InMemoryIdempotencyStore: process-local, no extra infrastructure. Fine for a single instance, or when you don't need protection to survive a restart.
  • RedisIdempotencyStore(url="redis://..."): shared across every instance of your service, so duplicates are caught even when different instances handle each delivery. Requires pymidil[redis].

Both accept an optional ttl_seconds, how long a key is remembered before it's forgotten and could be reprocessed again. Leave it unset to remember keys indefinitely.

What happens on a duplicate​

When the same key is seen again before its TTL expires, your handler doesn't run. The event is acknowledged as handled and Observatory records it with the duplicate status. Check the Idempotency Center to see which event types are currently covered and how many duplicates have been caught.