Component(s)

pkg/ottl

What happened?

Description

The truncate_all OTTL function truncates the string (UTF-8 encoded byte slice in Go) without checking if it breaks multi-byte UTF-8 codepoints in the middle. So using the transform processor with truncate_all often results in corrupted UTF-8 values.

Here's the code:

(It looks like opentelemetry-go SDK had a similar problem which is fixed now: open-telemetry/opentelemetry-go#3160)

Impact:
While the collector or a chain of collectors usually works fine with the corrupted UTF-8 down the line (open-telemetry/opentelemetry-collector#11449), some components fail:

• GCP Trace exporter failing due to "invalid utf-8" #35723
• Entire batch gets dropped: Invalid UTF-8 error handling policy oteps#257 (review)
  (We are seeing similar behavior with one of the vendors too. They drop the entire batch if anything contains an invalid UTF-8 string)

Steps to Reproduce

The truncate_all slicing code should be self-explanatory that it doesn't handle rune.

Expected Result

truncate_all will slice the string up to the given length. If truncating at exactly the length results in a broken UTF-8 encoding, it'll truncate before where the last rune started.

Actual Result

truncate_all slices UTF-8 encoded string without checking if it breaks a multi-byte character in the middle.
This results in the entire batch getting dropped on the vendor side.

Collector version

v0.111.0

Environment information

Environment

This applies to all environment

OpenTelemetry Collector configuration

extensions:
  health_check: {}
receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
  transform:
    trace_statements:
      - context: span
        statements:
        - truncate_all(attributes, 4095)
        - truncate_all(resource.attributes, 4095)
exporters:
  otlphttp:
    endpoint: https://otlp.nr-data.net
    headers:
      api-key: $NEW_RELIC_API_KEY
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform, batch]
      exporters: [otlphttp]