Logs are the line that grows without anyone deciding it should

Nobody approves a log budget. A service ships with debug logging on because that is how it was during development. A new feature adds a line per iteration inside a loop. An engineer turns on verbose logging to chase an incident and the incident closes but the setting does not. Three years of that, and logging is one of the three largest lines on the bill on a platform nobody thinks of as data-heavy.

The frustrating part is that the volume and the usefulness are almost unrelated. The estates with the biggest log bills are usually the ones where an engineer still cannot answer "what happened to this specific request" without grepping.

Structured means fields, not a sentence

A log line like Failed to process order 8814 for user 22 after 3 retries is unqueryable. To count retries by service you need a regular expression, and the regular expression breaks the day someone rephrases the message.

The same event as structured fields is a record you can aggregate, filter and alert on. Emit JSON, or your platform's native structured format, with a stable event name and the variable parts as typed fields: order ID, user ID, retry count, duration, outcome. The message text becomes a constant, which is exactly what makes it groupable.

Two fields do most of the work. The trace ID on every line ties the log to the request and to the span, which is the handoff that makes an incident investigation take minutes; it is the same link described in traces you actually debug with. And a stable event name lets you count occurrences without pattern matching.

Levels used as they were designed

Most estates use two levels in practice, "on" and "off", and pay for it.

• Error means a human needs to know, eventually. If you emit an error for a condition you handle and retry successfully, it is not an error.
• Warn means something unexpected that the system absorbed. This is the level teams abuse most.
• Info means a business-significant event: an order placed, a user created, a job completed. One or two per request, not twenty.
• Debug is off in production, and can be turned on for a single service, or better a single tenant or request, without a redeploy. Dynamic log levels are worth building; they remove the incentive to leave debug on permanently.

The test for an info line is whether you would want it in a record of what the system did. Function entry and exit do not pass that test.

Sample the repetitive, keep every error

The instinct that filtering loses information is right in general and wrong for the specific shape of log data. Ten thousand identical successful health-check lines carry the same information as ten of them plus a count.

So sample by outcome. Keep 100 percent of errors and warnings, always. Sample successful, high-frequency, low-variance events aggressively, and record the sampling rate in the line so your counts can be corrected. Drop health checks and readiness probes entirely at the collector, along with the load balancer's own access log for those paths.

Do the filtering in the collection pipeline rather than in application code. That way a noisy deploy can be contained in minutes without waiting for a code change, the same lever that makes the Azure Monitor bill tractable.

What must never be in a log

This is a compliance control, not a style preference. Logs are copied to more places than any other data you hold, retained under different rules, and read by more people.

Never log: passwords, tokens, API keys, session cookies, authorisation headers, full payment card numbers, and personal data beyond an identifier you can resolve elsewhere. Full request and response bodies are the usual route by which all of these arrive at once.

Build redaction into the logging library so it is the default rather than a discipline. Allowlist the fields that may be logged instead of denylisting the ones that may not, because a denylist fails the first time someone adds a field. And scan a sample of production logs for secret patterns periodically; the tooling from secret scanning works on log archives too.

A leaked token in a log is a rotation event and, if personal data is involved, potentially a notifiable one under GDPR.

Retention per class, not one number for everything

A single workspace-wide retention setting means the strictest requirement is applied to every byte. That is the most expensive possible configuration.

Split by class:

• Audit and security logs, where a compliance framework sets the period. Often a year or more, and the archive tier is the right home for most of it.
• Application logs for operational debugging. Seven to thirty days interactive covers nearly every real investigation.
• Access and infrastructure logs. Short interactive retention, longer cheap retention if anything asks for it.
• Debug output. Days, not weeks.

Then use the tiers. Every platform now offers a cheaper class with limited query features for high-volume, rarely-read data, and an archive that is far cheaper still with a slower retrieval path. Verify nothing alerts on a table before you move it to a cheaper tier, because losing alerting is the way this change goes wrong.

The things people forget

• Multi-line stack traces multiply. One exception can be sixty lines and sixty billing units unless the collector joins them into one event.
• A log line costs more than its bytes. Serialising, shipping and indexing consume CPU and network on the node that is also serving traffic.
• Logs in a loop scale with traffic. A line per item in a batch is fine at a hundred items and ruinous at a million.
• Duplicate collection is common. A sidecar agent, a node agent and the platform's own collection all shipping the same file is a bill paid three times. Audit what is actually collecting.
• Metrics are cheaper than logs for counting. If you only ever aggregate a log line, it should have been a counter.

What to do this week

Pull the top ten log sources by volume for the last 30 days and, for each, find out when anyone last queried it. The gap between what arrives and what anyone reads is where the money is, and in most estates the largest single source is something no dashboard or alert has ever touched. We start the observability workstream of a cloud engagement with exactly that comparison.