Database connection latency increases when Beyla trace collection is enabled (v2.7.2)

[mehmetyazicioglu]

When I enable Grafana Beyla to collect traces, my database connection times increase significantly.
To confirm the correlation, I tested by toggling Beyla tracing on and off — the latency only occurs when Beyla trace collection is enabled.

Beyla version : 2.7.2
Kubernetes / EKS version : 1.32
Kernel version: 6.12
Application language : Java
Database driver : Microsoft SQL Server JDBC driver
Database: AWS RDS MS-SQL instance
Beyla DaemonSet CPU & memory usage: Normal (no spikes observed)

Steps to Reproduce

Deploy Grafana Beyla 2.7.2 as a DaemonSet on EKS 1.32.

Run a Java application that connects to an AWS RDS MS-SQL instance using the official Microsoft JDBC driver.

Enable Beyla trace collection.

Measure database connection or query latency — it increases noticeably.

Disable Beyla — latency returns to normal.

Expected Behavior

Enabling Beyla tracing should not introduce noticeable latency in database connections.

[grcevski]

Thanks for reporting this @mehmetyazicioglu. This is unusual and I'm wondering if it's somehow related to a new feature we introduced to fetch the HTTP routes of Java applications.

Do you mind trying with the following change to the Beyla configuration (also please try with 2.7.5 when enabling this option, I believe the feature was in 2.7.2, but better to be on the latest version):

discovery:
  disabled_route_harvesters:
    - java

If you already have something in the discovery section, just append the disabled_route_harvesters option.

What Java version are you using? If you can paste the output of java -version it would help a lot in us reproducing this issue.

[mehmetyazicioglu]

I am running Java 21 version,

openjdk version "21.0.9" 2025-10-21 LTS
OpenJDK Runtime Environment Corretto-21.0.9.10.1 (build 21.0.9+10-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.9.10.1 (build 21.0.9+10-LTS, mixed mode, sharing)

we are running "[opentelemetry-javaagent.jar]" within the pods.

I have rolled out the change

  data:
    discovery:
      disabled_route_harvesters:
        - java
      instrument:
        # Collect the traces from pods with label "instrument: beyla" 
        - k8s_pod_labels:
            instrument: beyla

i am still seeing the same output.

I am going to try to upgrade beyla from 2.7.2 to 2.7.5 and update here again.

[mehmetyazicioglu]

I have rolled out the beyla 2.7.5 but i am still seeing the same issue.

[mehmetyazicioglu]

@grcevski i checked opentelemetry-java instrumentation, i found an issue, does beyla use the same instrument for java applications?

open-telemetry/opentelemetry-java-instrumentation#8303

[grcevski]

Interesting... we don't use it unless you have enabled:

ebpf:
  use_otel_sdk_for_java: true

or via the environment variable: OTEL_EBPF_USE_OTEL_SDK_FOR_JAVA=true

If those are set, we'll try to inject the actual OTEL Java SDK agent into the process. We provide this option for better quality instrumentation in certain cases. Do you have it enabled?

[grcevski]

If you can share your config we can try to reproduce this ourselves.

[mehmetyazicioglu]

it is not enabled, and here is my configuration file.

config:
  # -- set to true, to skip the check around the ConfigMap creation
  skipConfigMapCheck: false
  # -- set to true, to use the below default configurations
  create: true
  ## -- Provide the name of the external configmap containing the beyla configuration.
  ## To create configmap from configuration file, user can use the below command. Note: The name 'beyla-config.yaml' is important.
  ## `kubectl create cm --from-file=beyla-config.yaml=<name-of-config-file> -n <namespace>`
  ## If empty, default configuration below is used.
  name: ""
  # -- default value of beyla configuration
  data:
    discovery:
      disabled_route_harvesters:
        - java
      instrument:
        # Collect the traces from pods with label "instrument: beyla" 
        - k8s_pod_labels:
            instrument: beyla
    # profile_port: 6060
    # open_port: 8443
    routes:
       ignored_patterns:
         - /health
         - /status
         - /status/*
         - /info
         - /api/*/schema/*/report/history/VehicleHistoryReportRecord/status
         - /stats/prometheus
         - /prometheus
         - /actuator/health/*
         - /opentelemetry.proto.collector.metrics*
    # log_level: info
    ebpf:
      context_propagation: "all"
      #wakeup_len: 100
    otel_traces_export:
      sampler:
        name: "traceidratio"
        arg: "0.1"
      endpoint: http://opentelemetry-collector.opentelemetry-collector.svc.cluster.local:4318
      insecure_skip_verify:
    otel_metrics_export:
      endpoint: http://opentelemetry-collector.opentelemetry-collector.svc.cluster.local:4318
      protocol: http/protobuf
      features:
        - application
        - application_service_graph
        - application_span
    ## or alternatively use
    # grafana:
    #   otlp:
    #     cloud_zone: prod-eu-west-0
    #     cloud_instance_id: 123456
    #     cloud_api_key:
    attributes:
      kubernetes:
        enable: true
    filter:   
      application:
        db_operation_name:
          not_match: "{PING,CLIENT,ismaster}"    
      network:
        k8s_dst_owner_name:
          not_match: '{kube*,*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
        k8s_src_owner_name:
          not_match: '{kube*,*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
    ## to enable network metrics
    # network:
    #   enable: true
    prometheus_export:
      port: 9090
      path: /metrics
    ## to enable internal metrics
    internal_metrics:
      prometheus:
        port: 6060
        path: /metrics

## Env variables that will override configmap values
## For example:
##   BEYLA_INTERNAL_METRICS_PROMETHEUS_PORT: 9090
# -- extra environment variables
env: {}
  # BEYLA_INTERNAL_METRICS_PROMETHEUS_PORT: 9090
  # BEYLA_TRACE_PRINTER: "text"

# -- extra environment variables to be set from resources such as k8s configMaps/secrets
envValueFrom: {}
  #  ENV_NAME:
  #    secretKeyRef:
  #      name: secret-name
  #      key: value_key

[grcevski]

Thanks for this @mehmetyazicioglu. I think the only thing that might potentially affect the database connections in this way is our context propagation, unless I'm missing something.

Do you mind trying the following:

1. Can you please retry by disabling the section:

    ebpf:
      context_propagation: "all"
      #wakeup_len: 100

Just comment it out. I suspect it will resolve the issue.

2. In case it does then can you check to see if it works with:

    ebpf:
      context_propagation: "headers"
      #wakeup_len: 100

I think our injection of the context in the IP packet does something with Microsoft SQL server and it's causing TCP resets and retries, which is why the connection is so slow.

In case none of it works please let us know too! :)

Thanks again!

[grcevski]

It might be possible it's a duplicate of #2293. This specific issue I mentioned here is concerning with context_propagation: "all" with IPv6. It's possible that the client is using IPv6 to communicate with the RDS instance, or it might be another similar problem.

[mehmetyazicioglu]

I have changed my context_propagation from "all" to "headers" and deployed on my cluster. the DB connection times reduced.
Our EKS cluster, we have only enabled IPV4, but i will check with development to make sure we don't use IPV6.
Update: we use RDS only "Network type = IPv4"

Update:
https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver17#properties

"
ipaddresspreferenceString[<=128 char]IPv4First | The IP preference used by the client application.With IPV4First, the driver traverses IPv4 addresses first. If no IPv4 addresses can be connected to successfully, the driver continues and tries IPv6 addresses, if there are any.With IPV6First, the driver traverses IPv6 addresses first. If no IPv6 addresses can be connected to successfully, the driver continues and tries IPv4 addresses, if there are any.With UsePlatformDefault, the driver traverses all IP addresses in their initial orders from DNS resolution."

"the default value for ipaddresspreference is IPv4First" so i do not think application sends IPV6 calls.

one more question context_propagation= all is better than headers option ?

"You can disable the TCP/IP level encoding and TC programs by setting context_propagation to "headers". This context propagation support is fully compatible with any OpenTelemetry distributed tracing library.

context_propagation can be set to either one of the following values:

all: Enable both HTTP and IP options context propagation.
headers: Enable context propagation via the HTTP headers only.
ip: Enable context propagation via the IP options field only.
disabled: Disable trace context propagation."

is there a planned fix for context_propagation option for "all" ?
Thank you very much for the help..

@grcevski

[grcevski]

Thanks for trying it out! In general the context propagation with "all" which includes the IP options field covers more cases, but I think it's causing some issue in this case. Based on what you've told us so far about your application, I don't think you'll see any difference with just "headers". We already started working on the fix for IPv6 and perhaps there's a similar issue with IPv4. Now that we know what it is, we can fix it. We are definitely fixing this :).

[mehmetyazicioglu]

I am going to remove this configuration which did not affect anything for my case.

disabled_route_harvesters: - java
Thank you very much for support again :)

[marctc]

@mehmetyazicioglu Is this problem solved? can we close the issue?

[mehmetyazicioglu]

@mehmetyazicioglu Is this problem solved? can we close the issue?

we can close this ticket, i am waiting for once "It might be possible it's a duplicate of #2293. " issue is resolved, i can change context probagation from "headers" to "all" again.