Idempotency in Distributed Systems: Building Safe Retries in C#

Purpose

I want you to walk away from this post able to make any message handler safe to run more than once. If a message gets redelivered, or a retry kicks in, the result should be exactly the same as if it had run a single time. No double charges, no duplicate rows, no counters quietly drifting off. Once you have that property, retries stop being scary and turn into something you actually lean on.

Background

A while back I wrote about Event-Driven Architecture with C#, where services talk to each other through events over a message broker. One thing I glossed over there is worth spelling out now. Almost every broker you will meet in real life, whether it is RabbitMQ, Azure Service Bus, Kafka or SQS, promises at-least-once delivery, not exactly-once.

In plain terms, the same message can land on your consumer more than once. The broker retries when it never sees an acknowledgement. A consumer finishes the work and then crashes before it acks, so the message comes back around. A network hiccup or a partition rebalance triggers a redelivery. None of this is a fault. It is simply how these systems behave, and any design that pretends otherwise will bite you sooner or later.

Problem statement

The trouble starts when a handler was written as if it only ever runs once. Replay the message and every side effect runs again. I have run into all three of these in production.

A payment handler processes the same OrderPlaced event twice, and the customer gets charged twice. An analytics handler bumps a counter on each delivery, and now the dashboard is quietly wrong. An insert runs a second time and either creates a duplicate row or throws a primary key error that sends a perfectly good message off to the dead-letter queue.

The usual reaction is "let me just turn on exactly-once delivery." I understand the instinct, but at the transport level exactly-once is mostly wishful thinking. The moment a network sits between two machines, you cannot reliably tell "the message never arrived" apart from "the message arrived but the ack got lost." What you can build instead is at-least-once delivery paired with idempotent processing. Put those two together and you get effectively-once behaviour, which is what you actually wanted all along. The idempotent half is the part you control, so that is where we spend our time.

Possible solutions

Here are three ways to get there, going from the simplest to the most general. In a real system you usually mix them.

1. Natural idempotency

The nicest handlers are safe to repeat because they set a state instead of changing it. Setting an order to Paid twice still leaves it Paid, so nothing bad happens on a replay.

// Safe to run twice: it just sets the final state.
public async Task Handle(OrderPlaced message)
{
    await _orders.SetStatus(message.OrderId, OrderStatus.Paid);
}

Compare that with an operation that adds something each time. This one is not safe, because a duplicate delivery counts twice.

// Not safe: every delivery increments again.
await _orders.IncrementRetryCount(message.OrderId);

So whenever the domain lets you phrase the work as an upsert, or a "set it to this value" operation, take it. You are deleting the problem instead of babysitting it.

2. Idempotency key and a dedup table

Most handlers do something you cannot make naturally repeatable, like charging a card. For those, give every message a stable idempotency key (its message id works fine) and keep a record of the keys you have already handled. This leans on the same PostgreSQL habits from my IoT database post.

CREATE TABLE processed_messages (
    message_id   UUID        NOT NULL,
    handler      TEXT        NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (message_id, handler)
);

That composite primary key on (message_id, handler) is quietly doing the heavy lifting, and I will come back to why in a second.

public class IdempotentOrderPlacedHandler
{
    private readonly IPaymentGateway _gateway;
    private readonly AppDbContext _db;

    public IdempotentOrderPlacedHandler(IPaymentGateway gateway, AppDbContext db)
    {
        _gateway = gateway;
        _db = db;
    }

    public async Task Handle(OrderPlaced message)
    {
        var handler = nameof(IdempotentOrderPlacedHandler);

        // Quick check: have we already handled this one?
        bool alreadyProcessed = await _db.ProcessedMessages
            .AnyAsync(p => p.MessageId == message.MessageId && p.Handler == handler);
        if (alreadyProcessed)
            return;

        await using var tx = await _db.Database.BeginTransactionAsync();
        try
        {
            // Pass the message id as an idempotency key so the payment
            // provider also refuses to charge the customer twice.
            await _gateway.Charge(
                message.CustomerId,
                message.Amount,
                idempotencyKey: message.MessageId.ToString());

            _db.ProcessedMessages.Add(new ProcessedMessage
            {
                MessageId = message.MessageId,
                Handler   = handler
            });

            await _db.SaveChangesAsync();   // this insert fails if a duplicate beat us to it
            await tx.CommitAsync();
        }
        catch (DbUpdateException)           // a concurrent duplicate hit the primary key first
        {
            await tx.RollbackAsync();
            // Someone else already processed it, so we quietly move on.
        }
    }
}

Two things keep this honest. First, that AnyAsync check at the top is only a shortcut. If two copies of the message arrive at the same instant, both of them can pass the check, so it cannot be the real defence. The primary key is. Only one insert wins, the other trips the unique violation, and we swallow it. Second, notice the charge carries an idempotency key of its own. The database commit and the call out to a third party cannot live in the same transaction, so the key is what lets the payment provider dedupe on its end. Stripe, Adyen and most gateways support exactly this, and it is a small habit that saves real money.

3. Database constraints as a backstop

Even with all of the above, I like to make the bad state impossible to store in the first place.

ALTER TABLE payments
    ADD CONSTRAINT uq_one_payment_per_order UNIQUE (order_id);

Now if the code ever slips, the database itself will not record a second payment for the same order. A constraint like this is about the cheapest insurance you will ever buy.

Conclusion

My advice is to layer these rather than pick a single favourite.

Start with natural idempotency and model side effects as absolute state, using SetStatus or upserts wherever the domain allows, because that route carries the least machinery. When you cannot, reach for the idempotency key and dedup table, and let a unique or primary key be the real guard instead of an if statement, since only the database can settle a race cleanly. Add constraints so a duplicate state simply cannot exist. And push idempotency keys out to your external calls so third parties dedupe as well.

The reason I keep steering people away from chasing exactly-once delivery is that at-least-once is what brokers actually give you, and at-least-once plus idempotent processing already equals effectively-once. That is the result everyone wanted in the first place. Idempotency is also the floor that everything sturdier stands on, from the outbox pattern to full sagas, which is exactly where I am headed next: publishing events reliably with the outbox pattern.