Distributed tracing helps explain latency and failure across multiple components. It is especially useful when a user request triggers API calls, database work, queues, and external providers.
Create trace and span context
A trace represents an end-to-end operation; spans represent timed work within it. Propagate context across HTTP requests and message headers using a standard format such as W3C Trace Context where supported.
Name spans consistently
Use stable operation names such as catalog.lookup rather than values that include user IDs or full URLs with sensitive query parameters. Add attributes that help diagnose behavior without exposing private data.
Include asynchronous work
When a request enqueues a job, preserve trace linkage in message metadata. The worker can continue the trace or create a linked trace depending on the processing model.
Control volume and sampling
Tracing every request may be expensive. Use head-based or tail-based sampling according to traffic and diagnostic requirements. Retain error and high-latency traces at a higher rate where feasible.
Checklist
- Propagate trace context across boundaries.
- Use stable, low-cardinality span names.
- Link asynchronous jobs to originating work.
- Avoid sensitive span attributes.
- Define sampling and retention policies.