A queue and an event stream are not the same thing
One distributes work and forgets it. The other is an ordered log several consumers can replay. Choosing the wrong one produces an architecture that fights you for years.
A team needs to decouple two services, so they put a queue between them. Six months later a second team wants the same events. They add a second queue and a fan-out. Then someone needs to reprocess last month's data after a bug, and discovers the messages are gone, because a queue deletes a message when it is consumed.
That is not a queue failing. That is a log-shaped problem solved with a queue.
The distinction that decides everything
A queue distributes work. A message goes in, one consumer takes it, processes it, acknowledges it, and it is deleted. Multiple consumers compete for messages, which is exactly what you want for scaling work across workers. Once consumed, the message is gone. The queue's job is to make sure each unit of work happens once and to absorb bursts.
A stream is an ordered, retained log. Events are appended and stay for a retention period. Consumers track their own position independently, so several can read the same events without interfering, a new consumer can start from the beginning, and an existing one can rewind and reprocess. The stream's job is to be the record of what happened.
The test to apply: does more than one thing care about this event, now or plausibly later, and would you ever want to replay it? If yes, stream. If it is a task for a worker to do once, queue.
Getting this wrong is expensive because it is hard to reverse. Retrofitting replay onto a queue-based architecture means rebuilding the transport and everything that depends on its semantics.
The managed options
Every cloud offers both, and the mapping is straightforward. There is a simple queue service, a topic or pub-sub mechanism for fan-out, and a stream or event-hub product for the retained log. Managed Kafka is available on all of them as a service, and the open source project remains the default when you need its ecosystem, its connectors or portability between clouds.
The choice between managed Kafka and the cloud-native stream product is mostly about ecosystem and operational model. Kafka brings a large connector ecosystem and stream-processing tooling, and brings operational weight even when managed. The native products integrate more cleanly with the rest of the cloud, particularly with functions, and have less surrounding machinery.
For most teams the native option is correct, and Kafka earns its place when you need the connectors, when you are genuinely multi-cloud, or when a team already knows it well.
Delivery guarantees, and the one that is not what you think
At most once means a message may be lost and never duplicated. Rarely what you want.
At least once is what you get in practice from nearly every system. A message is delivered one or more times, and duplicates happen because acknowledgement can fail after processing succeeded. This is the model to design for.
Exactly once is offered by some systems and is narrower than it sounds. It generally applies within the boundary of that system, for example a read-process-write cycle entirely inside one platform with transactional support. The moment your consumer calls an external API or writes to a different database, that guarantee does not extend to the side effect.
So the practical rule: assume at least once, and make consumers idempotent. Use an idempotency key derived from the message, record processed keys, and make the write safe to repeat. This is the boring half of every integration and it is the half that determines whether the system is trustworthy, as set out in retries, idempotency and dead letters.
Ordering costs you parallelism
Global ordering across a topic is generally unavailable at scale, because it means one consumer.
What you get instead is ordering within a partition or a group key. Choose the key so that events which must be ordered relative to each other share it: all events for one customer, one account, one document. Then parallelism equals the number of partitions, and events for different keys process concurrently.
Two consequences. A hot key, one customer generating far more events than others, creates a partition that lags while others idle. And partition count is difficult to change afterwards in most systems, because it changes the key-to-partition mapping, so choose it with growth in mind.
If you find yourself needing strict global ordering, that is usually a sign the design should be rethought rather than the transport.
Retries, dead letters and the poison message
A consumer that fails must retry, and a consumer that always fails must stop.
Use exponential backoff with jitter, not a tight loop. A failing downstream dependency plus aggressive retries turns one problem into a denial of service against yourself, which is one of the cost spike shapes in finding the runaway before the invoice does.
Set a maximum receive count and route exhausted messages to a dead letter queue. Without one, a single unprocessable message blocks its partition or cycles forever.
Then actually watch the dead letter queue. An unmonitored one is a silent data loss mechanism. Alert on depth above zero, and build the tooling to inspect and replay from it before you need it at three in the morning.
Distinguish retryable from permanent failures in the consumer. A network timeout deserves a retry; a malformed payload does not, and should go straight to the dead letter queue with its error.
Schemas, and the contract nobody wrote down
The event is an interface between teams, and an undocumented interface breaks.
Use a schema with an explicit definition, ideally in a registry that validates on publish. Then follow compatibility rules: additive changes with defaults are safe, removing or renaming a field is not, and changing a type is not. A consumer must tolerate fields it does not recognise, so that a producer can add one without coordinating a release.
Version the event type in its name or in an attribute, and when you need an incompatible change, publish a new version alongside the old one until consumers have migrated. The schema is the part of this architecture that outlives every service that touches it.
Event carries state, or event carries reference
Two shapes, and the choice has consequences.
Notification with a reference sends an identifier and lets consumers fetch what they need. Small messages, always current data, and it couples consumers to the producer's API and creates a load spike when many consumers fetch at once.
Event carries the state includes the relevant data in the message. Consumers are independent and can process historically without calling anyone, at the cost of larger messages and data that reflects the moment it was published.
Default to carrying state for events consumed across team boundaries, because independence is the reason you chose events. Use references for large payloads, putting the object in storage and the pointer in the event, and for anything where consumers must not see a stale value.
The things people forget
- Message size limits are lower than you think, and the claim-check pattern of storing the payload and sending a pointer is the standard answer.
- Consumer lag is the metric that matters. Alert on it. A consumer that is up but falling behind looks healthy and is failing.
- Retention is a cost and a safety net. Too short and you cannot replay through an incident; too long and you are paying to store events nobody will read.
- Ordering and retries interact badly. Retrying a failed message while continuing with the next one breaks ordering within the key, which may or may not matter.
- Backpressure has to exist somewhere. A producer that cannot be slowed will fill a queue faster than consumers drain it, and the failure surfaces as memory or cost rather than as an error.
What to do this week
Take your most important asynchronous integration and answer two questions: if the consumer processed a message twice, what would break, and if you needed to reprocess yesterday's events, could you. The first answers whether you have an idempotency problem, the second whether you chose the right transport. Both are worth knowing before the incident that asks them for you. We work through this in the architecture phase of a cloud engagement.