The boring backbone: webhooks, queues and workers behind every automation that survives

Automation platforms hide an architecture. That is their value — you get a webhook endpoint, a canvas and a runtime without building any of it. It is also why the first serious outage is confusing: the failure is in a layer nobody on the team has a mental model of.

The model is simple and worth having, whether you are running n8n, Make, or something you wrote.

The three pieces

An ingress that accepts and acknowledges fast. A webhook receiver's only job is to validate the request, write it somewhere durable, and return 200 in a few milliseconds. Nothing else. If your ingress does work before acknowledging, every slow step is visible to the sender as a timeout, and senders respond to timeouts by retrying — which means your slow endpoint now receives triple the traffic at its worst moment.

A queue in the middle. It absorbs bursts, decouples the sender's rate from your processing rate, and preserves work across a restart. This is the piece that turns a spike into a delay instead of a data loss.

Workers that process. Stateless, horizontally scalable, and — critically — with a concurrency limit set by what the downstream systems tolerate, not by your CPU. Most self-inflicted outages in automation are a worker pool succeeding too fast against an API that rate-limits.

Everything else — retries, dead letters, scheduling — hangs off these three.

The five decisions

1. Acknowledge before processing, always. The sender's contract is "I gave it to you". Anything you do before returning 200 is on the critical path of somebody else's system.

2. Verify the signature, and reject unsigned. A webhook endpoint is an unauthenticated public entry point into your business processes. Almost every provider signs its payloads; almost every integration we audit skips the verification because it worked without it during testing. Verify, and compare in constant time.

3. Decide your delivery semantics out loud. Queues give you at-least-once; exactly-once is a story people tell. Accept the duplicate and make the processing idempotent — the mechanics are here. Any design that depends on a message arriving precisely once is a design that will produce a duplicate invoice.

4. Order matters less than you think, until it does. If a "customer updated" event can overtake "customer created", you need a partition key — usually the entity id — so events for one entity stay ordered while different entities process in parallel. Global ordering is not worth it and destroys throughput; per-key ordering is cheap and is almost always what the business meant.

5. Bound everything. A maximum message size, a visibility timeout longer than your slowest realistic run, a maximum receive count that moves a poisoned message to the dead letter queue. An unbounded system fails in ways that need a database to diagnose.

Where the platforms put these

In n8n queue mode this is exactly the architecture — main instance as ingress, Redis as the queue, workers as workers — which is why queue mode is not optional in production; see self-hosting n8n. In Make and Zapier the pieces exist but are the vendor's, so your levers are the schedule, the filters and the error routes. In your own code it is SQS plus Lambda, or Pub/Sub plus Cloud Run, and the whole thing is forty lines of Terraform.

The one thing the platforms will not do for you

Backpressure toward the source. When your queue depth is growing faster than the workers drain it, something has to give: either you scale workers, or you shed and tell the sender to slow down. A system with no signal at all just accumulates until a timeout somewhere turns a delay into data loss.

Alert on queue depth and on the age of the oldest message. Age is the better metric — a depth of ten thousand that drains in a minute is fine, a depth of forty where the oldest is two hours old means processing has stopped.

What to do this week

Find your busiest webhook endpoint and measure how long it takes to return 200. If the answer is more than a couple of hundred milliseconds, work is happening before the acknowledgement, and the fix — accept, enqueue, return — is usually an afternoon and removes an entire category of incident.