Stateful workloads in Kubernetes, and the question you should ask first
A StatefulSet guarantees less than people assume, a volume pins a pod to a zone, and the honest question is whether that database belongs in the cluster at all.
Articles about Data platform from the ConsultorIA blog: cloud, applied AI and security in production.
A StatefulSet guarantees less than people assume, a volume pins a pod to a zone, and the honest question is whether that database belongs in the cluster at all.
One distributes work and forgets it. The other is an ordered log several consumers can replay. Choosing the wrong one produces an architecture that fights you for years.
Nearly every performance incident we investigate is a missing index, a connection pool that is not there, or autovacuum sized for a database a tenth the size. Resizing the instance is the fix people try first and it is almost never the answer.
Open table formats changed the negotiation by separating storage from compute. What is left to decide is your workload mix, your governance boundary, and which cost model you can actually control.
Invalidation, stampede and the day the cache fills are the three. A time to live is the boring correct answer to the first, and most caching problems turn out to be a missing index.
Airflow should schedule and coordinate, never process. dbt should transform inside the warehouse with tests attached. The failure that costs most is data quality nobody checked until a dashboard was wrong for a quarter.