Observability for AI automations: what to log when the step is non-deterministic

A workflow with an AI step can return HTTP 200 on every single run for a week while quietly misclassifying one in eight inputs. Your uptime is perfect. Your automation is broken. No conventional monitoring will tell you, because nothing failed.

That gap is what observability for AI automations has to close, and it needs different instrumentation from the rest of your stack.

The record to write on every run

One row per execution, in a table you can query with SQL. Not a log line — a row, because you will want to aggregate it.

Capture: run id, workflow and version, the input identifier (not necessarily the input), a hash of the prompt template, the model and its exact version, input and output token counts, latency, the cost you computed from those counts, the decision or output in a structured form, and the outcome — auto-completed, sent to a human, failed.

The two fields people omit and later need are the prompt hash and the pinned model version. Without them, "quality dropped three weeks ago" is unanswerable, and it will be asked.

The three dashboards

Volume and outcome mix. Runs per day, split by outcome. The line that matters is the proportion going to human review. It is your quality metric, it is available immediately, it needs no labelling effort, and it is directly proportional to cost — see what an automation costs per run. A drift in this ratio is the earliest signal you get of anything changing: model, inputs, upstream system.

Cost and latency distributions, not averages. Average token count hides the pathological input that uses forty times the median. Plot p50, p95 and max. The p95 is where your timeout should be set, and the max is usually a bug.

Correction rate. Where a human reviews or edits the agent's output, log whether they changed it and, if you can capture it cheaply, what to. This is the only ground truth you get for free, and it accumulates into an evaluation set without anyone being asked to build one.

Sampling for actual quality

The dashboards tell you something moved. They do not tell you whether the output is correct, because for auto-completed runs nobody looked.

Sample. Pull twenty auto-completed runs a week at random, have a person score them against the rule you would apply, and record the result in the same table. Twenty a week is two hours a month and it produces something no dashboard can: a measured accuracy with a date attached, so when someone asks whether the automation is getting worse you answer with a trend instead of an impression.

Weight the sample toward low-confidence cases if your step emits a confidence, but keep a uniformly random portion, because a sample drawn only from suspected problems cannot measure the base rate.

Alert on ratios, not on errors

The alerts that matter here are statistical:

• Exception rate above its trailing average by more than a threshold, over a window long enough to be meaningful — a daily ratio, not an hourly one, or you will be woken by noise.
• Cost per run drifting up: usually a prompt got longer, an input source started sending more, or a retry loop appeared.
• Output distribution shifting: if a classifier that normally returns 60/30/10 starts returning 20/20/60, something changed even if every run succeeded. This single check catches model changes, upstream format changes and prompt regressions, and it is about fifteen lines of SQL.
• No runs at all, on the schedule it should be running. The silent stop is the outage nobody instruments.

Traces when there is a loop

For a single-call step, the row is enough. For an agent that calls tools in a loop, you want the sequence: each tool call, its arguments, its result size and the model's stated reason. OpenTelemetry works for this and there are LLM-specific tools that make the waterfall nicer, but the bar is low — the first time you debug a loop that ran eleven times you will build it regardless.

What to do this week

Add the run table to your highest-volume AI automation if it does not exist. Then write one query: outcomes grouped by day for the last thirty days. Almost every team that runs this query for the first time finds something in it — a step-change on a date that matches a deploy, or a slow drift nobody had noticed.