Trace ID vs span ID hierarchy, W3C Trace Context propagation, OpenTelemetry as the vendor-neutral standard, Zipkin/Jaeger visualization, and sampling trade-offs at scale.
Published September 23, 2026
Centralized Logging's correlation ID tells you which log lines belong to the same request. Distributed tracing goes further — it captures the actual timing and hierarchy of every hop that request took.
Trace ID: abc-123 (the WHOLE request's journey, shared across every hop)
Span 1: Gateway routing [0ms - 5ms]
Span 2: Order Service processing [5ms - 850ms]
Span 3: Inventory Service call [50ms - 400ms]
Span 4: Payment Service call [410ms - 840ms]
A trace represents the entire request's end-to-end journey, identified by one trace ID shared across every service it touches. Each individual hop (a service processing the request, or a single downstream call) is a span, with its own span ID, start/end timestamps, and a reference to its parent span — this parent-child structure is what reconstructs the full call hierarchy (which call happened inside which other call) and exactly where time was actually spent, not just that the request was slow overall.
traceparent: 00-abc123def456-span789-01
A standardized HTTP header format (the traceparent header, per the W3C Trace Context specification) carries trace ID, parent span ID, and trace flags across service boundaries — standardizing this means traces can be correctly stitched together across services built with different tracing libraries or even different languages, as long as each respects the same header format, rather than requiring every service in an organization to use identical tracing tooling.
OpenTelemetry ("OTel") provides a single, vendor-neutral instrumentation API and SDK for generating traces (and metrics, and logs) — instrument your code once against OTel's API, and the actual backend you send that telemetry to (Jaeger, Zipkin, a commercial APM vendor) becomes a configuration choice, not a rewrite. This directly avoids vendor lock-in that instrumenting directly against one specific tracing vendor's proprietary SDK would create — a real, practical concern for any team choosing tracing infrastructure today, since switching backends later shouldn't require re-instrumenting the entire codebase.
Both are open-source trace visualization/storage backends (receiving OTel-instrumented data, among other sources) — the actual UI where a specific trace ID becomes a visual waterfall/Gantt-chart view of every span, showing exactly which hop consumed the most time. This is precisely the tool that turns the "Why Microservices Fail" latency-spike scenario from guesswork into a direct answer: open the trace for a slow request, and the widest span in the waterfall view is the bottleneck, visually obvious rather than inferred.
opentelemetry:
traces:
sampler: parentbased_traceidratio
sampler-arg: 0.1 # trace 10% of requests, not 100%
Tracing every single request at high scale generates enormous data volume and real overhead (each span adds a small amount of processing/network cost) — most production systems sample, tracing only a percentage of requests (10%, or a smaller/larger fraction depending on traffic volume and how much fidelity is needed). The tradeoff: a genuine, rare failure might occur in an un-sampled request and simply never get a trace captured — some systems use smarter sampling (always trace requests that end in an error, sample everything else at a lower rate) to bias toward capturing the traces most worth having, rather than pure random sampling treating every request identically.
Q: How does trace sampling interact with a request that spans MANY services? A: The sampling decision is typically made once, at the START of the trace (the first service, or the gateway), and propagated as a flag in the trace context to every downstream hop — this ensures a trace is either fully captured across ALL its spans or not captured at all, avoiding the useless scenario of a partially-sampled trace missing some of its own hops.
Q: Why would 'always trace errors, sample the rest' be better than pure random sampling? A: Random sampling at, say, 10% means a rare failure affecting 1% of requests has only a 10% chance of being among the sampled ones needing investigation — biasing sampling toward error cases specifically ensures you're far more likely to have a trace available exactly when you most need one (during an incident), at the cost of somewhat less visibility into the 'everything's fine' baseline case, which matters less anyway.
Q: How does a span's parent-child relationship get established across an asynchronous message, not just a synchronous call? A: Similar to correlation ID propagation for logging — trace context needs to travel as message metadata, and the consuming service creates a new span with the propagated trace ID and a reference to the parent span from the message, preserving the hierarchy even though the actual call wasn't a direct synchronous HTTP request.
Q: Could distributed tracing data be USED as an input to alerting (see Alerting Strategy), not just manual investigation? A: Yes — aggregate trace data (e.g. p99 span duration for a specific service-to-service call) is exactly the kind of metric an SLO-based alert (see Alerting Strategy's SLO-burn-rate discussion) could be built on, turning tracing from a purely reactive investigation tool into a proactive monitoring signal too.