Azure OpenAI in production: quotas, networking and the permission model

Azure OpenAI pilots almost never fail on answer quality. They fail on three operational things that nobody looks at during the demo: the deployment ran out of quota, the security team would not approve a public endpoint, and nobody could explain why a salesperson could retrieve an HR document.

Here is the shape that survives all three.

Quota is the first architecture constraint

Azure OpenAI capacity is allocated as tokens per minute per model per region, per subscription. Two consequences that shape the design:

• The quota you have in your preferred region may not be the quota you need. Plan for multiple regional deployments behind a router from the start, even if you only use one at first. Adding the second one later means changing every client.
• Pay-as-you-go quota is shared and can throttle. Provisioned throughput units give you dedicated, predictable capacity with a latency profile you can put in an SLA. The crossover is roughly steady high volume; below that, standard deployments with retry-and-backoff are cheaper.

Put Azure API Management in front of the deployments. It gives you per-consumer token quotas, retries across regions on 429, a single endpoint for clients, and — critically — per-team usage metering. Without metering you cannot answer "what does this cost per team", and that question always arrives.

Networking, which is what the security team actually asks about

Default Azure OpenAI endpoints are public with key auth. That is fine for a pilot and usually unacceptable for production with customer data. The production shape:

• Private endpoint into your VNet, public network access disabled.
• Entra ID authentication with managed identities, not API keys. Keys in a Key Vault are still keys; a managed identity has nothing to leak.
• Customer-managed keys for the data at rest in the resource, if your compliance line requires it.
• Diagnostic settings to Log Analytics, so prompts and completions metadata are auditable. Be deliberate about whether you log content — it may be exactly what you need for debugging and exactly what you must not retain under your privacy commitments.

Retrieval and the permission problem

Azure AI Search is the natural retrieval layer, and it supports the thing that matters: security filters at query time. Index each document with the security identifiers of the groups permitted to see it, and pass the caller's group membership as a filter on every query.

Two rules from projects that went wrong:

Filter at retrieval, never after. If you retrieve broadly and filter in application code before showing results, the model has already been given the text. Any prompt injection or summarisation step leaks it.

Re-sync permissions, not just content. Your ingestion pipeline probably runs when a document changes. Permissions change without the document changing — someone leaves a team — and a stale ACL in the index is a genuine data exposure. Schedule a permissions re-sync independently.

Beyond that, AI Search gives you hybrid retrieval (BM25 plus vectors) and semantic ranking. Enable both. Vector-only retrieval misses exact identifiers, which in an enterprise corpus means product codes, ticket numbers and names — the things people actually search for.

Content safety and grounding

Azure AI Content Safety filters apply on input and output, with configurable severity thresholds and a prompt shield for jailbreak and indirect injection attempts. Indirect injection matters more than people expect in a RAG system: the attacker's payload arrives inside a retrieved document, not from the user.

Add a groundedness check on the generated answer against the retrieved passages, and have the assistant decline rather than guess below a threshold. Saying "I could not find this" is a feature, and it is the single behaviour that most determines whether people keep using the assistant after week three.

Cost and evaluation

Track cost per question, per team, from the first week. Batch embeddings, cache aggressively, and use a small model for query rewriting and routing — the general pattern is in the four decisions that make or break a RAG project.

And write the evaluation set before you build: fifty real questions, each with the document that should answer it, run on every change. Azure AI Foundry's evaluation tooling will run groundedness, relevance and retrieval metrics over it, but the questions have to come from your users. That set is the artefact that stalled projects turn out not to have, which is why it is the first thing we build in an applied AI engagement.

What to do this week

Check your token-per-minute quota for the model you are using, in the region you are using, and divide by the tokens your average request consumes. That number is your real concurrency ceiling, and most teams have never calculated it.