Adopting OpenTelemetry without stopping the team for a quarter

The expensive version of an OpenTelemetry migration looks like this: a platform team declares a vendor-neutral future, rips instrumentation out of forty services, and six months later is still running the old agent alongside the new pipeline because three teams never finished. You paid for two observability stacks for half a year and the dashboards are worse than when you started.

The cheap version treats OpenTelemetry as three separate adoptions on three timelines, starts with the collector rather than the SDKs, and never has more than a few weeks of overlap on any one signal.

The three signals are not at the same maturity

This is the most useful thing to internalise before planning anything.

Traces are the mature part. Tracing is what OpenTelemetry was designed around. The API and SDK are stable in every major language, auto-instrumentation covers the common frameworks, and the wire protocol is accepted by every commercial backend worth using. If you adopt one signal, adopt this one.

Metrics are stable but the migration is fiddly. You are almost certainly coming from Prometheus, which has its own scrape model, naming conventions and a decade of dashboards and alert rules built on them. Translating is mechanical but not free, and it is where naming collisions bite. Most teams keep Prometheus scraping as the metrics path for a long time and use OpenTelemetry for traces and logs first. That is a legitimate end state, not a failure.

Logs are the newest. The specification stabilised later, and the dominant pattern is not "instrument your logger with an SDK" but "keep writing structured logs and let the collector pick them up, correlate them and forward them". That is fine, and much less work.

Plan accordingly: traces now, logs through the collector next, metrics when there is a concrete reason.

Auto-instrumentation is the first step and it has a ceiling

Instrumentation libraries give you spans for HTTP servers and clients, database drivers, message brokers and gRPC with no code change. On the JVM and in .NET it is a flag on the process. In Python and Node it is close. In Go you write more by hand, because there is no runtime to hook.

Turn it on across a whole service tier in an afternoon and you get a call graph you did not have. Bank that win before doing anything more sophisticated.

The ceiling arrives quickly. Auto-instrumentation knows the framework boundary and nothing about your domain. It cannot tell you which tenant, which plan tier, which batch job, or which of the four code paths inside the handler you actually took. Those are manual span attributes, and they are what turns a trace from a picture into a search index. Budget that manual pass for the three or four services that matter, not for all of them.

The collector is the control point

Take one architectural decision from this: do not send telemetry from applications straight to a vendor. Send it to a collector you run, and let the collector decide what happens next. That gives you four things:

• Filtering before you pay for it. Drop health-check spans, high-cardinality debug attributes and chatty internal endpoints at the collector, not at the vendor's ingest meter.
• Enrichment. Attach Kubernetes pod, namespace, deployment, cluster, region and environment centrally, so no application has to know or care.
• Re-routing without redeploying anything. Backend change, dual-write during a migration, a copy of errors to a cheap archive: all collector configuration.
• A buffer. Batching and retry between your services and a backend having a bad day.

Run it as a DaemonSet for host and pod enrichment plus a gateway deployment for the heavier processors. The gateway is where the interesting policy lives, and it is what makes tail-based sampling for distributed traces possible at all, since the sampler needs every span of a trace in one place.

Semantic conventions matter more than they sound like they do

Semantic conventions are the agreed attribute names: http.request.method, server.address, db.system.name, service.name. They read like bureaucracy and they are the difference between telemetry that works and a pile of strings.

Two reasons. Every backend's out-of-the-box dashboards, service maps and RED-metric views are built on those names; deviate and you get an empty dashboard and a support ticket. And the conventions are what makes correlation possible: a trace, a log line and a metric series carrying the same service.name and deployment.environment.name can be joined, three different spellings cannot.

The catch is that some convention groups changed as they stabilised, and older instrumentation emits the previous names. Check which version yours emits, and if you are straddling a rename, normalise in the collector with a transform processor rather than asking every team to upgrade on the same day.

Migrating off a proprietary agent without paying twice for six months

1. Deploy the collector alongside the existing agent. Nothing changes for applications yet.
2. Point the collector at your existing backend over OTLP or an exporter. Most commercial vendors accept OTLP now. The new pipeline is proven against the old backend at zero migration risk.
3. Move one service tier to OpenTelemetry instrumentation, still exporting to the same backend. Compare dashboards side by side for a week.
4. Only then, if you are actually changing vendor, dual-export from the collector for a defined window. Two weeks, with a date on it.
5. Turn off the old agent per service as it is verified, not all at once at the end.

The double-cost months come from step 4 having no end date. Put the cutoff in the ticket.

The things people forget

• Resource attributes are set once and wrong forever. service.name, service.version and deployment.environment.name are the primary keys of your telemetry estate. Set them from the deployment, not a default that leaves half your fleet named unknown_service.
• Context propagation across queues. Auto-instrumentation propagates over HTTP and gRPC. Over SQS, Kafka or a database-backed job table it usually does not, unless the library supports it explicitly. That is where traces silently split in two.
• Sampling configured in the SDK is hard to undo. If applications drop spans before export, the collector cannot get them back. Sample at the collector, where you can change your mind.
• The collector needs monitoring too. Watch queue length, refused spans and exporter failures. A collector dropping data quietly is worse than no collector.
• Portability is real but partial. OTLP means instrumentation moves between backends without touching application code, and that genuinely changes your negotiating position. Dashboards, alert rules and saved queries do not move. Price the switch honestly.

This is usually a few weeks of platform work plus a light touch per team, and it is one of the pieces we set up early in a cloud engagement because everything else is easier to argue about once the data is neutral.

What to do this week

Deploy a collector in one non-production cluster with the Kubernetes attributes processor and an OTLP exporter pointed at the backend you already pay for, and enable auto-instrumentation on a single service. You will have a working end-to-end trace in under an hour, and something concrete to show the teams that have to do the rest.