ChatCompletion span exported with empty traceState, breaking W3C Trace Context propagation

[thspinto]

Description

The ChatCompletion span created by the AI Gateway extproc is exported with an empty traceState, even when the incoming request carries a non-empty W3C tracestate header. All Envoy-native spans in the same trace (ingress, router egress) correctly preserve it — only the extproc-created span loses it.

Per the W3C Trace Context spec, received tracestate fields must be forwarded.

Root cause

In internal/tracing/tracer.go, StartSpanAndInjectHeaders() calls tracer.Start() which creates a new SpanContext with empty TraceState — the parent's value is not copied:

parentCtx := t.propagator.Extract(ctx, propagation.MapCarrier(headers))
newCtx, span := t.tracer.Start(parentCtx, spanName, opts...)  // TraceState lost here
t.propagator.Inject(newCtx, carrier)

Repro steps

Send a request with both W3C trace context headers:

curl -X POST http://<gateway-host>/v1/chat/completions \
  -H "traceparent: 00-ff202ab855ba97c79471312ef973f469-e859612da0706e48-01" \
  -H "tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE" \
  -d '{"model": "<model>", "messages": [{"role": "user", "content": "hello"}]}'

Inspect the trace in any OTLP backend — the ChatCompletion span will have traceState: "" while all Envoy spans preserve the original value.

Environment

• AI Gateway: v0.5.0
• Envoy: v1.36.4
• Envoy Gateway: v1.6.3
• Go OTel SDK: v1.39.0
• OTel propagator: autoprop (defaults to tracecontext,baggage)

[nacx]

Hi! I've tried to reproduce the issue using the latest version from main:

# Build aigw to run it locally standalone
$ make build.aigw

# Configure the OTEL env vars to point to the locally running `otel-tui`
$ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
$ export OTEL_SERVICE_NAME=aigw-tracing-debug

# Export my OpenAI API key so that aigw auto-proxies traffic to OpenAI
$ export OPENAI_API_KEY=<my key>

# Start AIGW locally
$ ./out/aigw-darwin-arm64 run

AI Gateway External Processor is ready
[2026-04-13 20:16:00.150][22754465][warning][config] [source/server/options_impl_platform_default.cc:9] CPU number provided by HW thread count (instead of cpuset).
Envoy AI Gateway listening on http://localhost:1975 (admin http://localhost:59522) after 2s

Then send your request:

curl -X POST http://localhost:1975/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "traceparent: 00-ff202ab855ba97c79471312ef973f469-e859612da0706e48-01" \
  -H "tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE" \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hello"}]}'

And I can see the span and the tracestate set:

[thspinto]

Thanks for looking in to that @nacx.

I dug a little deeper an found there was another system duplicating the key in tracestate. It was sending, for example,

curl -X POST http://localhost:1975/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "traceparent: 00-ff202ab855ba97c79471312ef973f469-e859612da0706e48-01" \
  -H "tracestate: rojo=00f067aa0ba902b7,rojo=00f067aa0ba902b7" \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hello"}]}'

Envoy Library Name: envoy, Library Version: bad8280de85c25b147a90c1d9b8a8c67a13e7134/1.36.4/Clean/RELEASE/BoringSSL does not care about the duplication and sends the span with duplicated key to my collector. However, Library Name: envoyproxy/ai-gateway cares and discards the values.

I'm closing the issue as that is an accepted in behavior in the spec:

Delete a key/value pair. Any key/value pair MAY be deleted. Vendors SHOULD NOT delete keys that were not generated by them. The deletion of an unknown key/value pair will break correlation in other systems. This mutation enables three scenarios. The first is that proxies can block certain tracestate keys for privacy and security concerns. The second scenario is a truncation of long tracestates. Finally, vendors MAY also discard duplicate keys that were not generated by them.