Incomplete traces – spanID not in trace and clock skew adjustment warnings

[thapaashesh]

Description

We are observing incomplete traces in Jaeger where some spans are missing their parent spans.

The Jaeger UI shows warnings such as:

1. invalid parent span ID
2. spanID not in trace
3. clock skew adjustment

Environment

1. Jaeger version: 2.15.1 (also observed in 2.14.x)
2. Storage backend: Elasticsearch
   3.Deployment: Kubernetes
3. Traces sent via: OpenTelemetry Collector (OTLP gRPC)
4. Elasticsearch is healthy and indexing new spans

Current Behavior

• Some services show incomplete trace trees.
• Child spans exist in Elasticsearch.
• Corresponding parent spans are not found in the same trace.
• Jaeger UI shows clock skew adjustment warnings.
• This does not affect all services — only a subset.

Elasticsearch storage instability occurred previously (disk full), but that has been resolved. New traces are flowing normally, but incomplete traces still appear for certain services.

Expected Behavior

• All spans belonging to a trace should be properly connected.
• No spanID not in trace errors.
• No clock skew adjustment warnings when system clocks are synchronized.
• Complete trace tree displayed in Jaeger UI.

What We Have Verified

• NTP/chrony synchronization across nodes (clock offset < 1ms).
• Elasticsearch cluster health is green.
• New spans are being indexed successfully.
• OTLP exporter is successfully sending spans (no exporter errors).
• No visible ingestion errors in Jaeger collector logs.

Questions

1. Under what conditions does Jaeger emit spanID not in trace?
2. Can Elasticsearch shard failures or partial indexing cause permanent missing parent spans?
3. Could trace context propagation mismatch (W3C vs B3) cause this behavior even if spans share the same traceID?
4. Is there any known issue in Jaeger 2.x related to parent span resolution with Elasticsearch storage?

[pratikrath126]

Hey! This is a classic distributed tracing headache. The 'spanID not in trace' and clock skew warnings often come down to a few patterns:

1. Inconsistent propagators: Even though spanIDs are unique within a trace, if some services use W3C Trace Context and others use B3 (or B3 multi-header), the trace ID might match but parent relationships can break. Check your OTLP collector's propagators config – ensure all services send the same format. For safety, explicitly set: to use defaults (W3C) across the board.

2. Indexing gaps: If a parent span arrived during the earlier ES disk-full period, it might have been dropped or partially indexed. Child spans that arrived later are now referencing a parent that never made it into ES. This would be permanent. Check collector logs from that period for 'failed to encode span' or 'failed to submit' errors.

3. Clock skew with large offsets can confuse the UI's search window. Even with NTP synced now, past skew could cause spans to be outside the query range for a trace. Verify that all hosts' clocks were in sync during the problematic period.

4. Jaeger 2.x known quirks: There have been reports of ES bulk responses causing 'partial' indexing acknowledgments where some shards failed silently. Check Elasticsearch indices health and check for any red shards.

Quick checks:

• Run a trace ID query directly against ES to see if parent span exists at all.
• • Enable debug logs on jaeger-query to see the exact lookup queries and responses.
• • Temporarily increase the 'max duration' in UI to widen the window.
    Hope this helps narrow the root cause!

[thapaashesh]

@pratikrath126

Thank you for the detailed explanation , this was very helpful.
I’ve added the following across services:

env:
  - name: OTEL_PROPAGATORS
    value: "tracecontext,baggage"

After this change, I’m no longer seeing new incomplete span warnings, so it looks promising. I’ll continue to monitor closely and will update if anything changes.

Thanks again!

[thapaashesh]

@pratikrath126 I forced W3C propagation (OTEL_PROPAGATORS=tracecontext,baggage), but incomplete spans came back — so it’s likely not just propagators.