Distributed Tracing

The Observability Gap in Distributed Systems

In a monolith, debugging is straightforward. A request enters the application, moves through functions, and you can trace the entire flow in a single stack trace. In a microservices architecture, a single user request might touch 6, 10, or 20 services. Each service has its own logs. If the response is slow, which service caused the delay? If there is an error, where did it originate? Without distributed tracing, you are left searching through logs from a dozen services, trying to correlate timestamps manually.

Traces, Spans, and Context Propagation

Distributed tracing relies on three concepts:

Trace. A trace represents the complete journey of a single request through the system. It has a unique trace ID (typically a 128-bit random value) that is propagated across every service the request touches. All work related to that request shares this trace ID.

Span. A span represents a single unit of work within a trace. Each service creates one or more spans. A span has a start time, duration, operation name, status, and optional metadata (tags/attributes). Spans have parent-child relationships, forming a tree. The root span is created by the first service. Each downstream call creates a child span.

Context propagation. When Service A calls Service B, the trace ID and the current span ID must travel with the request. This is context propagation. For HTTP calls, the trace context is passed in headers (the W3C traceparent header is the standard). For Kafka messages, it is embedded in message headers. Without context propagation, spans from different services cannot be linked into a single trace.

In this trace, the API Gateway receives the request (root span, 450ms total). It calls the Order Service (child span, 390ms). The Order Service calls Inventory (100ms) and then Payment (250ms). Payment calls Fraud Check (110ms). The waterfall view shows that Payment is the bottleneck: it accounts for more than half the total latency. Without tracing, you would only know the overall response was 450ms.

The Span Hierarchy as a Timeline

Tracing UIs display spans as a waterfall diagram. Each span is a horizontal bar. Nested spans are indented under their parent. The width of the bar represents duration. This immediately reveals where time is spent.

This Gantt-style view shows the same trace as a timeline. The Inventory check completes quickly (100ms). Payment takes 250ms, with the Fraud Check consuming most of that time. If you needed to reduce overall latency, you would focus on the Payment/Fraud Check path.

OpenTelemetry: The Standard

OpenTelemetry (OTel) is the industry-standard framework for distributed tracing, metrics, and logs. It was formed by merging two earlier projects: OpenTracing and OpenCensus. The value proposition is simple: instrument your code once with OpenTelemetry, then send the data to any compatible backend (Jaeger, Zipkin, AWS X-Ray, Datadog, Grafana Tempo).

OpenTelemetry provides:

• SDKs for Java, Python, Go, JavaScript, .NET, and more. Auto-instrumentation libraries hook into popular frameworks (Spring Boot, Express, Django) and create spans automatically for HTTP calls, database queries, and message broker interactions.
• The OpenTelemetry Collector, a proxy that receives telemetry data, processes it (filtering, sampling, batching), and exports it to one or more backends. This decouples your application from the tracing backend.
• W3C Trace Context propagation out of the box, ensuring interoperability across services written in different languages.

Tracing Tools Compared

Sampling Strategies

In a system processing millions of requests per second, storing a trace for every request is impractical. Storage costs would be enormous and most traces are uninteresting (successful requests with normal latency). Sampling decides which traces to keep.

In practice, many organizations use a combination: head-based sampling at 1-5% for general visibility, plus tail-based sampling to always capture errors and high-latency traces. This balances cost against the need to debug production issues.

Designing a Tracing Setup

A production tracing pipeline typically looks like this:

Each service instruments its code with the OpenTelemetry SDK. On every incoming request, the SDK extracts the trace context from headers (or creates a new trace if none exists). On every outgoing call, the SDK injects the trace context into headers. Spans are exported via OTLP (OpenTelemetry Protocol) to the OTel Collector. The collector applies sampling, batching, and filtering, then exports to the tracing backend.

What to Capture in Spans

Not all spans are equally useful. At minimum, capture:

• Operation name: GET /api/orders/{id}, db.query, kafka.produce
• Duration: Start and end timestamps.
• Status: OK, ERROR, with error message if applicable.
• Attributes: http.method, http.status_code, db.system, db.statement (sanitized), messaging.system, user.id (if relevant).
• Events: Log entries attached to the span (e.g., "retry attempt 2", "cache miss").

Avoid capturing sensitive data in span attributes: passwords, credit card numbers, PII. Sanitize database queries to remove parameter values. Set clear policies for what is and is not included in trace data.

Systems Thinking Lens

Distributed tracing introduces a meta-feedback loop. Without tracing, teams lack visibility into cross-service behavior, so they make local optimizations that may not improve overall system performance. Tracing closes this loop by making the global behavior visible. A delay in Service D is now traceable to a slow database query, which is traceable to a missing index. The systems thinker recognizes tracing as a leverage point: it does not fix problems directly, but it makes problems visible, which accelerates every other improvement.

However, tracing itself is a system with its own feedback dynamics. More services generate more spans. More spans require more collector capacity and storage. Costs grow with adoption. Sampling is the balancing loop that keeps the system sustainable. Without it, the observability infrastructure becomes a scaling problem of its own.

Further Reading

• OpenTelemetry, Traces (official documentation). Definitive reference for trace concepts, span structure, context propagation, and the OpenTelemetry data model.
• Jaeger, Architecture Overview (official documentation). How Jaeger collectors, agents, and storage backends work together.
• Charity Majors, Liz Fong-Jones, George Miranda, Observability Engineering (O'Reilly, 2022). Comprehensive treatment of tracing, metrics, and logs in production systems, with practical guidance on sampling and cost management.
• W3C, Trace Context Specification. The standard for the traceparent and tracestate HTTP headers used for context propagation.
• Ajit Singh, Distributed Tracing: Jaeger vs Tempo vs Zipkin. Side-by-side comparison of tracing backends with architecture diagrams and feature matrices.