Guardrails that earn their latency

The first guardrail a team ships is usually a moderation call on the user's message. It catches the obvious abuse, everyone feels better, and it does almost nothing about the failures that actually reach customers: the assistant that confidently invents a refund policy, the one that quotes a competitor's pricing, the one that returns another tenant's data because the retrieval filter was wrong.

Guardrails are worth building. They are also a tax on every request, and the ones that pay for themselves are chosen deliberately rather than installed as a category.

Input and output are different jobs

Input checks happen before you spend a model call. They are cheap, they fail fast, and their ceiling is low. Useful ones: length and token limits so a pasted document does not cost twenty times a normal turn, rate limits per user and per tenant, obvious abuse detection, and personal data detection where you are contractually obliged not to forward it to a provider.

What input filtering cannot do is stop prompt injection. The attack surface is natural language, there is no grammar to reject, and every filter you ship is a public target. Treating an input classifier as an injection control is the mistake, and the architectural alternatives are the subject of prompt injection.

Output checks are where the value is, because the output is what reaches the user and what you are accountable for. The checks worth having:

• Schema validation. If you asked for structured output, validate it and retry on failure. This is the cheapest guardrail in existence and it catches a surprising share of real problems.
• Grounding. For a retrieval system, check that claims in the answer are supported by the retrieved context. This is the closest thing to a hallucination control that actually works.
• Policy checks. No competitor names, no legal or medical advice, no commitments about refunds or pricing, whatever your domain forbids.
• Leakage checks. No system prompt content, no API keys, no identifiers from another tenant.
• Format and tone, which matters more for brand than people admit.

Rules, classifiers and judges

Three mechanisms, in increasing order of cost and capability. Use the cheapest one that works.

Deterministic rules. Regular expressions, denylists, schema validation, identifier checks. Microseconds, free, completely predictable, and they only catch what you enumerated. A check that the answer contains no string matching another tenant's identifier is a rule, and it is better than any model at that job.

Small classifiers. A purpose-built moderation or topic model. Tens of milliseconds, cheap, reasonably robust, and they generalise beyond the exact phrasing. Good for abuse, toxicity and topic boundaries.

A model as judge. Another LLM call asking whether the answer is grounded, on-policy or complete. Hundreds of milliseconds to seconds, a full model call in cost, and the only thing that handles nuanced judgements like "is this claim supported by this document". Use a smaller, faster model than your generator where you can, and give it a narrow question rather than a general one. The calibration problem it introduces is the same one described in evaluating an LLM feature.

The pattern that works in practice: rules always, a classifier where the category is well-defined, and a judge only on the checks that matter most, often only on a sampled fraction of traffic for monitoring rather than blocking.

Deciding whether it is worth it

Every guardrail has three costs. Latency added to every request, including the ones that were fine. Money, if it is a model call. And false positives, which are the expensive one, because a guardrail that blocks legitimate answers erodes trust in the product faster than an occasional bad answer does.

The arithmetic to do before shipping one: how often does the failure it prevents actually occur, what does one occurrence cost, and what does the guardrail cost per thousand requests including the false positives. A grounding judge that adds 800 ms to every answer to catch a problem occurring in one response in two hundred is usually the wrong trade, and the right version is to run it asynchronously on a sample and alert when the rate moves.

Reserve synchronous blocking for the failures that are genuinely unacceptable: data leakage across tenants, output that creates a legal commitment, content that would end the customer relationship.

Streaming breaks output guardrails

This is the constraint nobody plans for. If you stream tokens to the user, you cannot check a complete answer before it is shown, because part of it is already on screen.

Three options, none free. Buffer the whole response, check it, then release it, which removes the perceived latency benefit that made you stream in the first place. Check in chunks as they arrive, which catches some classes of problem and cannot evaluate the answer as a whole. Or stream optimistically and retract, which is technically possible and looks alarming to a user.

Decide this before you build the interface, because retrofitting a guardrail onto a streaming product usually means changing the product.

What happens when it fires

The action matters as much as the detection, and there are four options.

Block and apologise. Safe, and the worst user experience. Reserve for genuinely unacceptable output.

Regenerate. Retry, often with an added instruction about what went wrong. Costs another call and works surprisingly often for schema and format failures. Cap the retries, because a loop here is expensive.

Degrade. Return a reduced answer: the retrieved sources without the synthesis, or a template response, or a clarifying question. Usually the best option for a grounding failure, because the user gets something useful and nothing false.

Escalate to a human. The right answer when the stakes are high and a person is available, and it needs the routing to exist rather than being aspirational. Where to place that boundary is the subject of human in the loop.

Whatever you choose, tell the user something honest. "I could not find support for that in your documents" is a better experience than a generic refusal, and it teaches people how the system works.

Measure the guardrail, not just the model

A guardrail is a classifier, and an unmeasured classifier drifts into either uselessness or obstruction.

Log every trigger with the input, the output, the check that fired and the action taken. Sample the triggers and label them: was it a true positive. Track the false positive rate as a headline metric, because that is the one that silently destroys the product. Track the trigger rate over time, since a sudden change usually means a model version changed or someone is probing you.

Version the guardrails alongside the prompts, in the same repository, with the same review and the same evaluation set, as in versioning prompts and evaluating automations. A guardrail change is a product change and deserves the same gate. And feed the triggers into your observability, because the trigger rate is one of the most useful signals an AI feature emits, which is the argument in observability for AI automations.

The things people forget

• Guardrails are not a permission model. No output check replaces filtering retrieval by the user's access rights.
• The system prompt is not a guardrail. It is an instruction to a component that can be talked out of following it.
• Language coverage. Moderation models are much weaker outside English. If you serve European or global users, test in the languages you actually receive.
• The judge can be attacked too. If untrusted content reaches the judging prompt, it can be persuaded to approve.
• Someone has to read the logs. A guardrail firing two thousand times a day that nobody has looked at is a signal being thrown away.

What to do this week

Take fifty real conversations from production and read the outputs looking for one thing: an answer that stated something not supported by the retrieved context. Count them. That rate is the argument for a grounding check, and if the rate is low, it is the argument for running one asynchronously instead of paying for it on every request. We size guardrails this way in an applied AI project.