Retries, idempotency and dead letters: the boring half of every automation

The fun half of an automation is the part that works. The half that determines whether anyone trusts it is what happens at the third failed call to a supplier's API at three in the morning.

This is true whether the workflow lives in n8n, Make, a Lambda or a Python script on a cron. The mechanisms are the same four, and most no-code platforms give you none of them by default.

Idempotency: the one that prevents the embarrassing failure

A retry that duplicates a side effect is worse than no retry. Two invoices, two welcome emails, two Slack notifications to a customer channel — these are the incidents people remember.

The fix is an idempotency key: a deterministic identifier derived from the business event, not from the run. invoice-2026-0918-ACME-4471, not a UUID generated at the top of the workflow, because a retried run generates a new UUID and defeats the whole thing.

Then one of two things has to honour it. Either the target API supports an idempotency header — Stripe, most payment providers and a growing number of SaaS APIs do, and you should be sending it — or you keep the ledger yourself: a table with the key as primary key, written before the side effect, checked at the start. A unique constraint violation is not an error, it is the system working.

Retries: bounded, backed off, and only for the right errors

Retrying a 500 is correct. Retrying a 400 is a loop that will run until your quota ends, because a malformed payload stays malformed however many times you send it.

The policy that covers almost everything: retry on 429, 502, 503, 504 and on network timeouts; never retry on 4xx except 429; three attempts with exponential backoff and jitter — one second, four, sixteen, each plus a random fraction. The jitter matters when a downstream API comes back from an outage and two hundred queued workflows all retry in the same second; without it you take the API down again yourself.

Respect Retry-After when the response carries it. It is the API telling you exactly what you want to know.

Dead letters: where a failure waits for a human

After the last retry, the run has to go somewhere that is not a log line. A dead letter queue can be an SQS queue, a database table, or a dedicated channel — what matters is that it holds the full input payload, the error, the timestamp and the idempotency key, and that a human can replay from it.

Replay is the part teams skip and then regret. If recovering from a two-hour outage means someone re-entering forty records by hand, the automation did not survive its first bad day. A replay script that reads the dead letter table and re-submits is an hour of work and it is the difference between an incident and a non-event.

The alert has to name the business event

"Workflow 47 failed" wakes someone who then has to open a browser to find out whether it matters. "12 supplier invoices failed to post to the accounting system since 02:10" gets triaged in the notification.

Alert on the aggregate, not on each failure — one message per failed run at 3am is how a channel gets muted, and a muted channel is how a three-day outage happens. Alert on rate and on age: more than N failures in ten minutes, or anything sitting in the dead letter queue for more than an hour.

And alert on silence. The failure mode nobody instruments is the workflow that stopped triggering — the webhook that got unregistered, the schedule that was disabled during a debugging session eight weeks ago. A check that fires when a workflow has not run in twice its expected interval catches a whole class of outages that produce no errors at all, because nothing ran.

Where this lives in each tool

In n8n, an Error Trigger workflow catches failures across the instance and is the natural home for the dead letter write and the alert. In Make, it is an explicit error route per fallible module with a Break directive. In your own code it is whatever queue you already run. The platform matters less than whether all four mechanisms exist; in most accounts we audit, zero or one of them do.

What to do this week

Pick the automation whose failure would most annoy a customer. Force a failure in staging — point it at a hostname that does not resolve — and watch what happens. Then answer three questions: did anyone find out, is the input still recoverable, and if you run it twice does the customer get two of something?