Java 25 Lambda with OTEL Metrics Exporter (Direct OTLP to New Relic) intermittently loses metric datapoints

[vidyarajsv]

We are observing intermittent metric loss when using the AWS OTel Lambda Layer with a Java 25 Lambda runtime and Micrometer instrumentation.

A custom Micrometer counter is incremented once per successfully processed business event. During testing, the number of metric datapoints received by New Relic does not match the actual number of processed events or Lambda invocations.

For one test window:

we lost 5 metrics

we receive below error in the log :

[otel.javaagent 2026-08-28 04:34:07:301 +0000] [OkHttp https://otlp.nr-data.net/...] ERROR io.opentelemetry.exporter.otlp.internal.HttpExporter - Failed to export metrics. The request could not be executed.

java.io.InterruptedIOException: timeout

Caused by: java.io.IOException: Canceled

      AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-secret-wrapper.sh"
      ENVIRONMENT_TAG: !Ref 'EnvironmentTag'
      #  Service metadata
      OTEL_SERVICE_NAME: !Sub '${FunctionName}-${EnvironmentTag}'
      OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED: "false"
      OTEL_INSTRUMENTATION_AWS_LAMBDA_ENABLED: "true"
      OTEL_INSTRUMENTATION_AWS_SDK_ENABLED: "false"
      OTEL_INSTRUMENTATION_MICROMETER_ENABLED: "true"
      OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED: "false"
      OTEL_METRICS_EXPORTER: "otlp"
      OTEL_TRACES_EXPORTER: "none"
      OTEL_LOGS_EXPORTER: "none"
      OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://otlp.nr-data.net"
      OTEL_RESOURCE_ATTRIBUTES: !Sub >-
        deployment.environment=${EnvironmentTag}
      # --- FIXED: FORCE SYNCHRONOUS AND INSTANT METRIC FLUSHES ---
      OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED: "true"
      # Export interval aligned to Lambda invocation cadence. A 1-second interval
      # causes overlapping OkHttp requests in a VPC (DNS+TLS overhead), leading to
      # InterruptedIOException/Canceled errors and metric loss. 60 seconds gives each
      # export request room to complete before the next one starts. Timeouts are set
      # generously above typical VPC round-trip to New Relic (~1-2s) so requests are
      # not prematurely canceled by the retry interceptor.
      OTEL_METRIC_EXPORT_INTERVAL: "60000"
      OTEL_METRIC_EXPORT_TIMEOUT: "30000"
      OTEL_EXPORTER_OTLP_TIMEOUT: "30000"
      OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: "delta"
      JDK_JAVA_OPTIONS: "-XX:AOTMode=off"

Let me know what configuration we need to change

[mxiamxia]

Can you share the exact layer ARN and version on the function? "AWS OTel Lambda Layer" maps to two different AWS-published Java layers, and they don't behave the same:

1. AWS OpenTelemetry layer with Application Signals support. The current one, built from this repo and shipped as layer.zip in the Releases here. Standard wrapper is /opt/otel-instrument. It turns on CloudWatch Application Signals by default but can also export plain OTLP to any backend, which is what you're doing with New Relic.
2. Classic ADOT Lambda layer, from aws-observability/aws-otel-lambda. Two Java variants: aws-otel-java-agent-<arch>-ver-x-y-z (auto-instrumentation) and aws-otel-java-wrapper-<arch>-ver-x-y-z (manual wrapper). It bundles the ADOT Collector for Lambda and typically uses /opt/otel-handler.

ADOT layer ARNs by region are listed at https://aws-otel.github.io/docs/getting-started/lambda, and the Application Signals layer comes from this repo's Releases. Knowing which one you're on tells me which defaults are in play.

Separately, your AWS_LAMBDA_EXEC_WRAPPER is /opt/otel-secret-wrapper.sh, not either standard wrapper. Can you paste that script (minus secrets)? I want to see whether it calls /opt/otel-instrument or sets up the agent itself. That decides whether the end-of-invocation flush is running, which is where the metric loss would come from.

[vidyarajsv]

As requested, please find the details :

Layer used :

arn:aws:lambda:eu-west-1:615299751070:layer:AWSOpenTelemetryDistroJava:17

wrapper script to load the secret

#!/bin/bash
set -euo pipefail

REGION="${AWS_REGION}"
SECRET_ID="/newrelic-credentials"
SERVICE="secretsmanager"
HOST="${SERVICE}.${REGION}.amazonaws.com"
CONTENT_TYPE="application/x-amz-json-1.1"
AMZTARGET="secretsmanager.GetSecretValue"
REQUEST_BODY="{"SecretId":"${SECRET_ID}"}"

AMZDATE=$(date -u +"%Y%m%dT%H%M%SZ")
DATESTAMP=$(date -u +"%Y%m%d")

PAYLOAD_HASH=$(printf '%s' "${REQUEST_BODY}" | openssl dgst -sha256 | awk '{print $NF}')

CANONICAL_REQUEST="POST
/

content-type:${CONTENT_TYPE}
host:${HOST}
x-amz-date:${AMZDATE}
x-amz-security-token:${AWS_SESSION_TOKEN}
x-amz-target:${AMZTARGET}

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
${PAYLOAD_HASH}"

CR_HASH=$(printf '%s' "${CANONICAL_REQUEST}" | openssl dgst -sha256 | awk '{print $NF}')

CREDENTIAL_SCOPE="${DATESTAMP}/${REGION}/${SERVICE}/aws4_request"
STRING_TO_SIGN="AWS4-HMAC-SHA256
${AMZDATE}
${CREDENTIAL_SCOPE}
${CR_HASH}"

hmac_hex() {
printf '%s' "$2" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$1" | awk '{print $NF}'
}

K_SECRET=$(printf '%s' "AWS4${AWS_SECRET_ACCESS_KEY}" | od -A n -t x1 | tr -d ' \n')
K_DATE=$(hmac_hex "$K_SECRET" "$DATESTAMP")
K_REGION=$(hmac_hex "$K_DATE" "$REGION")
K_SERVICE=$(hmac_hex "$K_REGION" "$SERVICE")
K_SIGNING=$(hmac_hex "$K_SERVICE" "aws4_request")
SIGNATURE=$(printf '%s' "$STRING_TO_SIGN" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${K_SIGNING}" | awk '{print $NF}')

AUTHORIZATION="AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY_ID}/${CREDENTIAL_SCOPE}, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=${SIGNATURE}"

RESPONSE=$(curl --fail --silent --show-error -X POST "https://${HOST}"
-H "Content-Type: ${CONTENT_TYPE}"
-H "X-Amz-Date: ${AMZDATE}"
-H "X-Amz-Security-Token: ${AWS_SESSION_TOKEN}"
-H "X-Amz-Target: ${AMZTARGET}"
-H "Authorization: ${AUTHORIZATION}"
-d "${REQUEST_BODY}")

NR_KEY=$(printf '%s' "$RESPONSE" | sed 's/."LICENSE_KEY\":\"//; s/\".//')

if [ -z "${NR_KEY}" ]; then
echo "Failed to extract LICENSE_KEY from secret ${SECRET_ID}" >&2
exit 1
fi

export OTEL_EXPORTER_OTLP_HEADERS="api-key=${NR_KEY}"

exec /opt/otel-instrument "$@"

in addtion i added below properties, still i see the error :

      OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_ON_EXIT: "true"
      OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_TIMEOUT: "30000"

Let me know whether below property help to reduce payload size

      OTEL_EXPORTER_OTLP_COMPRESSION: "gzip"

Error

  -- [otel.javaagent 2026-08-30 19:43:40:177 +0000] [OkHttp https://otlp.nr-data.net/...] ERROR io.opentelemetry.exporter.otlp.internal.HttpExporter - Failed to export metrics. The request could not be executed. java.io.InterruptedIOException: timeout at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:467) at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:429) at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:387) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:240) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:580) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Canceled at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:68) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:332) at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:96) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:332) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:232) ... 4 more [otel.javaagent 2026-08-30 19:43:40:180 +0000] [OkHttp https://otlp.nr-data.net/...] WARN io.opentelemetry.sdk.metrics.export.PeriodicMetricReader - Exporter failed

[vidyarajsv]

Below is my updated properties, and i filtered our all system level metric, still i see same error. Any pointer for resolution.

      AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-secret-wrapper.sh"
      ENVIRONMENT_TAG: !Ref 'EnvironmentTag'
      #  Service metadata
      OTEL_SERVICE_NAME: !Sub '${FunctionName}-${EnvironmentTag}'
      OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED: "false"
      OTEL_INSTRUMENTATION_AWS_LAMBDA_ENABLED: "true"
      OTEL_INSTRUMENTATION_AWS_SDK_ENABLED: "false"
      OTEL_INSTRUMENTATION_MICROMETER_ENABLED: "true"
      OTEL_INSTRUMENTATION_HTTPCLIENT_ENABLED: "false"
      OTEL_INSTRUMENTATION_JAVA_HTTP_CLIENT_ENABLED: "false"
      OTEL_INSTRUMENTATION_OKHTTP_ENABLED: "false"
      OTEL_INSTRUMENTATION_APACHE_HTTPCLIENT_ENABLED: "false"
      OTEL_INSTRUMENTATION_URLCONNECTION_ENABLED: "false"
      OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED: "false"
      OTEL_METRICS_EXPORTER: "otlp"
      OTEL_TRACES_EXPORTER: "none"
      OTEL_LOGS_EXPORTER: "none"
      OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://otlp.nr-data.net"
      OTEL_RESOURCE_ATTRIBUTES: !Sub >-
        deployment.environment=${EnvironmentTag}
      # --- FIXED: FORCE SYNCHRONOUS AND INSTANT METRIC FLUSHES ---
      OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED: "true"
      # Export interval aligned to Lambda invocation cadence. A 1-second interval
      # causes overlapping OkHttp requests in a VPC (DNS+TLS overhead), leading to
      # InterruptedIOException/Canceled errors and metric loss. 60 seconds gives each
      # export request room to complete before the next one starts. Timeouts are set
      # generously above typical VPC round-trip to New Relic (~1-2s) so requests are
      # not prematurely canceled by the retry interceptor.
      OTEL_METRIC_EXPORT_INTERVAL: "60000"
      OTEL_METRIC_EXPORT_TIMEOUT: "30000"
      OTEL_EXPORTER_OTLP_TIMEOUT: "30000"
      OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: "delta"
      OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_ON_EXIT: "true"
      OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_TIMEOUT: "30000"
      JDK_JAVA_OPTIONS: "-XX:AOTMode=off"
      OTEL_EXPORTER_OTLP_COMPRESSION: "gzip"
        # OTel agent debug logging (shows export attempts, payload size, response codes, retries)
      OTEL_JAVAAGENT_DEBUG: "true"
        # OTel SDK log level
      OTEL_LOG_LEVEL: "debug"

[srprash]

This is possibly because the lambda freeze may be causing the metric export to freeze and the freeze time is counted against the okhttp's timeout.
I am taking a look and will update shortly. @vidyarajsv in the meantime can you provide more logs, preferably across multiple invocations of your lambda function?