Skip to content

feat: OpenTelemetry GenAI semantic conventions#681

Open
vasspilka wants to merge 14 commits into
agentjido:mainfrom
vasspilka:gen-ai-telemetry
Open

feat: OpenTelemetry GenAI semantic conventions#681
vasspilka wants to merge 14 commits into
agentjido:mainfrom
vasspilka:gen-ai-telemetry

Conversation

@vasspilka
Copy link
Copy Markdown
Contributor

@vasspilka vasspilka commented May 6, 2026

Summary

Brings the OpenTelemetry bridges into full conformance with the GenAI semantic conventions so spans, metrics, and events emitted by ReqLLM.OpenTelemetry (auto-attach handler) and ReqLLM.Telemetry.OpenTelemetry (dependency-free mapper) drop into spec-aware backends like Langfuse, Grafana, and Honeycomb without bespoke instrumentation glue.

Work is structured in four shippable phases (see OPENTELEMETRY_GENAI_PLAN.md):

  1. Phase 1 — span attribute completeness: request params, server, conversation id, reasoning/embedding tokens, error.type on stop spans.
  2. Phase 2 — opt-in content capture: gen_ai.input.messages, gen_ai.system_instructions, gen_ai.tool.definitions, gen_ai.output.messages (default :none; reasoning text stays redacted in every mode).
  3. Phase 3 — GenAI client metrics: operation.duration, token.usage, time_to_first_chunk, time_per_output_chunk with spec bucket boundaries; streaming TTFC also surfaced as a span attribute.
  4. Phase 4gen_ai.usage.cost, OpenAI extension attributes (openai.api.type, service_tier, system_fingerprint), Langfuse cost-details opt-in, docs.

The two surfaces (auto-bridge and dependency-free mapper) now share five internal modules (Attributes, Content, Metrics, SemConv, Shared) so they emit identical payloads.

How past integrations are affected

Auto-bridge users (ReqLLM.OpenTelemetry.attach()) — only the breaking changes below apply. Default options still emit no content payloads.

Dependency-free mapper users (ReqLLM.Telemetry.OpenTelemetry) — the breaking changes apply, plus the system-message split for content: :attributes consumers (see Breaking changes #4).

Native ReqLLM telemetry consumers ([:req_llm, :request, *], [:req_llm, :token_usage]) — fully back-compat. The lifecycle event metadata gained three new optional keys (request_options, server, streaming); every pre-existing key is preserved. [:req_llm, :token_usage] is unchanged. Public function signatures on ReqLLM.Telemetry, ReqLLM.OpenTelemetry, and ReqLLM.Telemetry.OpenTelemetry are unchanged.

Breaking changes

Downstream OTel dashboards/alerts that filtered on the old values need to be updated:

  1. gen_ai.provider.name normalized to spec enums: :amazon_bedrockaws.bedrock, :azureazure.ai.openai, :googlegcp.gen_ai, :google_vertexgcp.vertex_ai, :xaix_ai, :openai_codexopenai. Other providers keep their stringified atom name.
  2. gen_ai.operation.name for embeddings: embeddingembeddings (plural, per spec).
  3. error.type is now also set on :stop spans when http_status >= 400 (previously only on :exception spans).
  4. Dependency-free mapper, content: :attributes mode: system messages are now split out of gen_ai.input.messages into a new gen_ai.system_instructions attribute. Consumers iterating gen_ai.input.messages looking for role == "system" need to read gen_ai.system_instructions instead. Auto-bridge default users are not affected (default is still content: :none).

Test plan

  • mix test — 3,095 unit tests pass, 0 failures (11 skipped, 145 excluded).
  • New unit coverage in test/req_llm/open_telemetry/{attributes,content,metrics,sem_conv}_test.exs, plus extended test/req_llm/open_telemetry_test.exs, telemetry_open_telemetry_test.exs, telemetry_test.exs.
  • Cross-surface parity check (SemConv test) — auto-bridge and dependency-free mapper produce identical provider/operation/span names.
  • Manual: console-exporter run across against Azure OpenAI.
  • Manual: Langfuse cloud smoke test — generation shows model, cost, tokens, conversation id, structured input/output with tool calls.
  • Manual: Prometheus exporter — verify spec bucket boundaries on the four histograms.
  • Static: mix quality, mix dialyzer.

Related issues

Closes #

@vasspilka vasspilka changed the title Gen ai telemetry feat: OpenTelemetry GenAI semantic conventions May 7, 2026
@vasspilka vasspilka force-pushed the gen-ai-telemetry branch from 7c955c9 to 6dbe798 Compare May 7, 2026 21:12
@vasspilka vasspilka force-pushed the gen-ai-telemetry branch from 6dbe798 to 609bca8 Compare May 7, 2026 22:00
@mikehostetler mikehostetler added the needs_work Changes requested before merge label May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs_work Changes requested before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants