Traces you actually debug with, not traces you collect

The tracing project finishes, the dashboards look impressive, and six months later nobody has opened the trace UI during an incident. When you look at why, it is almost always one of two things. Either the traces break at the third hop so you can never see the whole request, or the one trace you needed was not sampled.

Both are fixable, and neither is fixed by collecting more. Tracing is the only signal that answers "why was this specific request slow", and it earns its cost only when someone can follow a real request end to end at the moment they need to.

Context propagation is where it breaks

A trace is a tree of spans held together by a trace ID carried in the request. Auto-instrumentation handles this for ordinary HTTP and gRPC calls between instrumented services, which is why the first demo always works.

It breaks in the places auto-instrumentation cannot see:

• Queues and message brokers. The producer must write the trace context into the message headers and the consumer must read it back out. Without that, the work done in the consumer becomes an orphan trace and the causal link to the request that triggered it is gone. This is the most common break we find, and the most damaging, because asynchronous work is exactly where the latency hides.
• Batch jobs and scheduled work. There is no incoming request, so decide deliberately: one trace per run, or one per processed item, with a link back to the originating request when there is one.
• Third-party clients and SDKs. A vendor library that builds its own HTTP request will not carry your headers unless you hook it.
• Anything crossing a language or framework boundary where one side uses W3C traceparent and the other uses a legacy vendor format. Standardise on W3C Trace Context and configure the propagators explicitly rather than relying on defaults.
• Front ends. If the trace starts at your load balancer, you cannot see the time the user actually waited. Starting it in the browser or mobile client is more work and is the difference between "the API took 80ms" and "the page took 4 seconds".

Test propagation deliberately. Send one request through the longest path you have, including a queue hop, and confirm the trace has the span count you expect. Do it in CI if you can.

Tail sampling is what you want, and it costs more

Head sampling decides at the start of the request, before anything has happened, usually by keeping a fixed percentage. At 1 percent you keep a hundredth of the traffic and you will not have the trace for the incident, because errors are rare by definition and randomly discarding traces discards them at the same rate.

Tail sampling decides after the trace is complete, when you know whether it was slow or failed. That lets you write the policy you actually want: keep everything that errored, keep everything above a latency threshold, keep a small baseline percentage of successful requests for comparison, and keep everything from a specific customer or endpoint you are investigating.

The cost is that the collector must buffer all spans of a trace until it is complete, which needs memory and, critically, all spans of one trace must reach the same collector instance. That means a load-balancing exporter in front of a tail-sampling collector tier, keyed on trace ID. It is genuinely more infrastructure. It is also the difference between a trace store you use and one you pay for.

A reasonable middle path, if tail sampling is too much for now: keep 100 percent of errors via head sampling on the error path where your framework allows it, plus a higher baseline percentage than you think you need, and revisit once volume hurts. The OpenTelemetry collector is where all of this belongs, not in application code.

Attributes are what make a trace searchable

A trace with perfect structure and no attributes tells you a request was slow but not which request. The span attributes are the difference between browsing and finding.

Put on the root span, at minimum: the route template rather than the resolved URL, the HTTP method and status, the tenant or customer identifier, the user identifier, the deployment version, and the region or cell. On downstream spans: the database statement in its parameterised form, never with values interpolated, the target table or collection, the queue or topic name, the remote service name, and the retry attempt number.

Follow the OpenTelemetry semantic conventions for anything they cover. The payoff is that your backend can build service maps and latency breakdowns automatically instead of you writing bespoke queries per service.

This is also the right home for high-cardinality identifiers. A user ID in a Prometheus label is how you kill Prometheus, as covered in Prometheus at scale. The same ID on a span attribute is exactly correct, because trace backends are built to index high-cardinality fields and metrics stores are not. That split is the whole reason to run both.

Never put secrets, full request bodies, authorisation headers or personal data in attributes. Spans go to a store with different retention and different access control from your application database, and a trace payload is a very easy way to leak a token into a vendor's system.

The path from alert to span

The reason to wire all three signals together is a single incident workflow that takes minutes instead of an afternoon.

An alert fires on a symptom, typically error rate or latency on a user-facing endpoint. The alert links to a dashboard scoped to that service. From the dashboard you jump to exemplars, which are trace IDs attached to specific metric samples, so you land on a trace that produced the number you are looking at rather than a random slow request. In the trace you find the span that consumed the time. From that span you jump to the logs for that trace ID, filtered to that request, and read the error.

Each of those hops needs to be a link, not a copy-paste. Exemplars need to be enabled in your metrics pipeline. Logs need the trace ID as a structured field, which is the single most valuable field in a log line and costs nothing, as argued in structured logging and what it costs.

If your team cannot do that sequence today, practise it in a game day before you need it at 3am.

The things people forget

• Clock skew makes spans look impossible. Child spans appearing to start before their parent, or negative durations, are almost always NTP drift on one host.
• Sampling decisions must be consistent across the trace. If one service samples independently, you get half-traces that are worse than none.
• Trace storage is not cheap and is usually priced per span. Chatty instrumentation that emits a span per ORM call produces thousands of spans per request. Aggregate them.
• A trace without an error is still useful. Latency investigations are the majority of real usage, not error investigations.
• Retention should be short. Seven to thirty days covers nearly every investigation. Keeping traces for a year is paying to store data nobody will query.

What to do this week

Take one request that goes through a queue, run it end to end, and open the trace. Count the spans and compare against the number of services it really touched. If the trace stops at the queue, you have found your highest-value fix, and it is usually a dozen lines of producer and consumer code.