No. 003 · October 20, 2025

At Least Once Means At Least One

SNS and SQS deliver at-least-once. Not “usually once,” not “once unless something goes wrong” — at-least-once is the guarantee itself, stated up front, not a bug you eventually run into. A message can be delivered twice and the system is still working exactly as designed. Knowing that going in doesn’t make it any less strange the first time it actually happens to something you own.

I’ve written about the event backbone this sits on top of — services publishing to topics instead of calling each other directly, the notification service picking work up off a queue instead of being called synchronously. What I didn’t get into there is what that queue actually promises, and what it costs when you build as if it promised more.

The small case

The notification service reads messages off a queue and sends an email per message. Every so often, a user got the same email twice. Nobody had missed this in design — at-least-once delivery duplicating a message occasionally was a known, considered-unlikely failure mode. It happened anyway, because “unlikely” and “never” are different words for a reason.

The fix was to start tracking which notification had already been sent, and check that before sending again — tracking that hadn’t existed before, on purpose. The notification service’s first version didn’t store any “already sent” state at all: it went from being called synchronously to sitting on a queue and stopped there, which kept that step small and avoided a real storage cost for a failure mode that was, at the time, still theoretical. Normal startup sequencing — ship the version that solves the problem in front of you, mature it once the gap it left behind actually costs something.

Once it did start to cost something, the fix wasn’t complicated by missing data — there was already a record of every notification sent, on the analytics side, in the datawarehouse. That existed for a different purpose. What didn’t exist was a way for the notification service itself to check that record before sending. Preventing the duplicate meant standing up that same tracking operationally, in the request path, as its own piece of state — paying a real storage cost specifically to answer one question fast: has this notification gone out already?

Low stakes, and low stakes for a specific reason: at the time, the notification types this service sent didn’t include anything that would alarm someone if it showed up twice — nothing like “money was just deducted from your account.” Somebody got an email twice, said something, and it got fixed. What it was worth remembering wasn’t the incident, it was that “unlikely” had already stopped being good enough as an answer, well before anything expensive was on the line.

The case that actually hurts

The same root cause shows up wherever at-least-once delivery meets something you can only afford to count once. Money is the sharpest version of that: a queue that might redeliver a message is fine when the downstream effect is an extra email; it’s not fine when the downstream effect is a spend deduction, because a retry that gets treated as a new event is a double charge, silently, with nothing about the retry itself announcing that it’s not new.

The fix follows the same idea as the email case, with a harder requirement underneath it. The write itself was already conditional — that part didn’t change. What mattered here was consistency: the check for “has this request id been processed already” can’t tolerate eventual consistency, because eventual consistency means one node can answer “not present” while another already has the record — exactly the race that lets a retry slip through as new work. What matters isn’t that the database itself is “strongly consistent” as a general property; it’s that the specific operation deciding who gets to process a given request id is atomic, so two consumers can’t both land on “not present” for the same id. Here that’s a conditional write; elsewhere it might be a unique constraint, a transaction, or a lock — same requirement, different mechanism:

try:
    insert(spend_log, request_id, amount)  # fails if request_id already present
    deduct(account, amount)  # only runs if the insert above succeeded
except AlreadyExists:
    pass  # already processed this request_id — no-op, not an error

The database becomes the single source of truth for “did this already happen,” and the deduction only proceeds if the conditional write claims that request id first. A retry doesn’t get treated as new work; it gets recognized and dropped.

The example simplifies one thing worth naming: the insert and the deduction need to be coordinated so one can’t succeed without the other. Record the request id but never run the deduction, or run the deduction and fail to record the id, and the system isn’t actually idempotent — it just looks that way until the next retry lands on the gap. That’s the same coordination problem the outbox pattern exists to solve, from the producer side, which is part of why the two ideas keep showing up together.

I’m deliberately not narrating the specific incident behind this — it’s the kind of detail that belongs to the company it happened at, not to a blog post — but the pattern is the one that actually matters here, and it’s the same pattern as the email case, just enforced at a level where “occasionally” isn’t an acceptable rate anymore.

Idempotency as a default, not a patch

The lesson underneath both cases is the same: at-least-once delivery isn’t something you special-case around when it bites you, it’s a property of the architecture that every consumer has to be built for from the start. Dedupe-by-id and conditional writes aren’t a fix applied where it happened to hurt; once you’ve seen it hurt once, the same discipline belongs on anything else consuming off that same backbone, before it has its own incident to point to. And where it genuinely isn’t in place yet — the notification service’s first version, before the cost was worth paying — that has to be a decision made with eyes open: knowing exactly what a duplicate delivery does to that consumer, and having a fast way to react once it shows up, not finding out both at the same time.

A related failure mode is the opposite: a message that isn’t duplicated, but never finishes — an event processor crashes mid-stream and leaves it stuck, half-processed. Idempotency doesn’t solve that, since nothing was delivered twice; it needed its own background process, a reaper that detects stuck events and replays them.

What this doesn’t solve

Idempotent consumers solve one half of the problem: what happens when a message you received arrives more than once. They don’t solve the other half — whether the message gets published at all, in the first place, if the process publishing it dies at the wrong moment. That’s a producer-side guarantee, not a consumer-side one, and it’s a different mechanism entirely. That’s the outbox pattern, and it’s next.