Add a Rhesis tracing integration (rhesis-haystack)

[Arman-Beykmohammadi]

Summary and motivation

I'd like to add rhesis-haystack, a tracing integration that exports Haystack pipeline, component and
Agent spans to Rhesis over OpenTelemetry. Implementation is in #3669.

What Rhesis is. An open-source platform for structured feedback and evaluation on LLM agents.
Domain experts review agent responses in a UI; the feedback stays attached to the test case and the
agent version that produced it, and recurring feedback becomes tests and metrics that run on every
change. Docs: https://docs.rhesis.ai

Why a fifth tracer. The repo already has four: datadog-haystack and opentelemetry-haystack
export to generic OTel/APM backends, langfuse-haystack and weave-haystack are LLM trace viewers.
What this one adds is correlation rather than another place to look at traces. Spans carry:

Attribute	Purpose
rhesis.test.run_id, rhesis.test.id, rhesis.test.result_id	joins a trace to the test execution that produced it
rhesis.conversation.id, rhesis.conversation.is_turn_root	groups a multi-turn conversation into one trace

So the use case is running a Haystack pipeline under a Rhesis test run and having a reviewer's
feedback land on the exact span tree that produced the answer.

Concretely, the pipeline it supports: run a Haystack agent against a Rhesis test set, have domain
experts review the responses in the Rhesis UI, and get the recurring failures back as tests and
metrics that run in CI on the next change — with each result linked to the trace it came from.

Adoption signals

Measured 14 Aug 2026.

• GitHub stars of the main repository: 387 on
  rhesis-ai/rhesis (31 forks, 31 contributors, first commit
  Oct 2024).
• PyPI downloads in the last 30 days: rhesis-sdk
  857, rhesis 553. These are small, and I would
  rather say so than dress them up — see the note at the end of this section.
• Release activity: latest release sdk-v0.12.0 on 2026-08-07. Cadence is roughly every two
  weeks and has been steady for months: 0.7.0 (Apr 23), 0.7.1 (May 7), 0.8.0 (May 21), 0.9.0
  (Jun 11), 0.9.1 (Jun 25), 0.10.0 (Jul 13), 0.11.0 (Jul 23), 0.12.0 (Aug 7). The platform is
  versioned per component; 22 SDK releases so far.
• Maintenance: actively maintained, commits daily, 31 contributors listed. Rhesis AI GmbH is the
  company behind it and employs the maintainers; I am one of them, which is also the conflict of
  interest I should declare here. Source is MIT except an ee/ directory and brand assets. Issues
  are triaged in the open on the repo.
• Haystack community demand: none that I can point to — I searched this repo and deepset-ai/haystack
  for existing requests and found none. The demand I can attest to is from the other direction: Rhesis
  users who build on Haystack and currently have no way to connect their pipeline runs to their test
  results. I would rather state that plainly than manufacture a thread.
• Anything else: the Rhesis SDK already ships tracing integrations for LangChain, LangGraph,
  Pydantic AI, Microsoft Agent Framework, AutoGen and the Google GenAI SDK
  (sdk/src/rhesis/sdk/telemetry/integrations/),
  so Haystack is the notable gap in a set we already maintain. The repo also carries a Haystack
  multi-agent reference application (agents/visit-prep) built on this integration, with its own
  test suite, so it has a real consumer we keep working from day one.

On the numbers being small. Rhesis is 22 months old and the download figures reflect that. I am
not going to argue they are large. What I can offer instead is that the maintenance cost to deepset
is low and bounded: the integration depends on rhesis[telemetry], which resolves to 9 packages on
top of haystack-ai (52 total, against 54 for langfuse-haystack) — the OpenTelemetry SDK and
exporter, protobuf, and the rhesis package itself, with no ML or GPU dependencies anywhere in the
tree. requires-python = ">=3.10". I intend to keep maintaining it here — including the
nightly Haystack-main job, which is the part that actually costs a maintainer time when Haystack's
tracing internals move. If you would rather see the adoption numbers grow first, I am happy to ship
it from the Rhesis repo and come back with the same proposal later; that is a reasonable answer and
I would rather hear it now than after a review.

Detailed design

Full implementation and discussion in #3669. In outline:

• RhesisConnector — a component you add to a pipeline without connecting it to anything, the
  same shape as LangfuseConnector. Constructing it registers the tracer with Haystack, so a
  standalone Agent needs the constructor and nothing else. Returns name, trace_url, trace_id.
• RhesisTracer / DefaultSpanHandler — bridges Haystack spans to OpenTelemetry, with the same
  SpanHandler extension point langfuse-haystack exposes. I followed that integration's file
  layout and class split deliberately (Connector / Tracer / Span / SpanContext /
  SpanHandler / DefaultSpanHandler), so it reads the same to whoever maintains that one.
• Semantic mapping — Haystack operation names and component types map to Rhesis span names and
  ai.operation.type, plus content and token attributes and invocation-context propagation. Both
  Haystack span shapes are covered: the 2.x batched ToolInvoker component span and the 3.0 agent
  loop (haystack.agent.step.*), including promoting a tool span to an agent handoff when a tool
  runs an Agent.
• RhesisTracing — a conversation-aware entry point for applications that drive Haystack from
  their own loop (chat servers, REPLs) rather than through Pipeline.run. This is the one piece with
  no precedent among the existing tracers, and I have flagged it in the PR as something I would drop
  if you would rather not have a non-component entry point in the repo.
• No process-wide side effects — the connector builds its own TracerProvider and never calls
  trace.set_tracer_provider, so a user who already runs their own OpenTelemetry pipeline keeps the
  global provider and every span it produces.

Usage is one component and no wiring:

import os

os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"

from haystack import Pipeline
from haystack_integrations.components.connectors.rhesis import RhesisConnector

pipe = Pipeline()
pipe.add_component("tracer", RhesisConnector("Chat example"))  # no connect() needed
...
result = pipe.run({..., "tracer": {"invocation_context": {"session_id": "demo-session"}}})
result["tracer"]["trace_url"]  # deep link to the trace

Checklist

If the request is accepted, ensure the following checklist is complete before closing this issue.
Follow the instructions in https://github.com/deepset-ai/haystack-core-integrations/blob/main/CONTRIBUTING.md#create-a-new-integration and use our scaffolding script for the implementation.

Tasks

• The code is documented with docstrings and was merged in the main branch
• Docs are published at https://docs.haystack.deepset.ai/
• There is a Github workflow running the tests for the integration nightly and at every PR
• A new label named like integration:<your integration name> has been added to the list of labels for this repository
• The labeler.yml file has been updated
• The package has been released on PyPI
• An integration tile with a usage example has been added to https://github.com/deepset-ai/haystack-integrations
• The integration has been listed in the Inventory section of this repo README
• The feature was announced through social media