Easier Cloud Trace integration by automating project_id requirements

[simonbos]

Disclaimer: I am unsure if this is the correct place for the question or if there are already existing solutions.

TLDR: Looking at the docs, the requirements for GCM - Cloud Trace integration include having project_id as a label on the exemplar itself. This is not always an easy requirement, and having GCM add that automatically would be easier from a consumer point of view.

The following is a detailed description of the problem.

Using a custom application with OpenTelemetry metrics

I have a Go application on GKE which:

• Uses an OpenTelemetry TracerProvider exporting traces to a Google-Build OpenTelemetry Collector (docs).
• Uses an OpenTelemetry MeterProvider exporting metrics via a Prometheus HTTP server and Google Cloud Managed Service for Prometheus scraping the endpoint using a PodMonitoring CR.

Now I am looking into the Google Cloud Managed Service for Prometheus integration with Cloud Trace as described here. The exemplars in Cloud Monitoring, which I received using a ListTimeSeries RPC call look like this (for a metric prometheus/grpc_server_call_duration_seconds/histogram):

exemplars": [
  {
    "value": 0.00946708,
    "timestamp": "2025-09-11T08:01:22.777Z",
    "attachments": [
      {
        "@type": "type.googleapis.com/google.monitoring.v3.DroppedLabels",
        "label": {
          "resource.labels.namespace": "<redacted>",
          "resource.labels.location": "europe-west1-b",
          "resource.labels.cluster": "development",
          "resource.labels.job": "<redacted>",
          "project_id": "<redacted>",
          "resource.labels.instance": "<redacted>"
        }
      }
    ]
  }
]

Note that there is no SpanContext created. If I directly scrape the Prometheus endpoint using GKE port forwarding and curl -H "Accept: application/openmetrics-text" http://localhost:8080/metrics, I do see the trace_id and span_id showing up but no project_id (as expected).

A workaround is probably to

• Use the OpenTelemetry collector for metrics as well. This option I would like to avoid.
• Add a custom wrapping OpenTelemetry Gatherer which adds the project_id label on each exemplar. However it would require quite some application changes.

Using go-synthetic using Prometheus directly

Testing the go-synthetic example from this repository (which uses the Prometheus client directly), I do see the SpanContext created. This example correctly sets the project ID as an exemplar label here (it's of course not the correct project ID).

"exemplars": [
   {
     "value": -229.4003856706737,
     "timestamp": "2025-09-11T10:01:33.719Z",
     "attachments": [
       {
         "@type": "type.googleapis.com/google.monitoring.v3.SpanContext",
         "spanName": "projects/example-project/traces/c8e03aa73b21f6bb13f2716f179a6fa8/spans/838d17d28b312737"
       },
       {
         "@type": "type.googleapis.com/google.monitoring.v3.DroppedLabels",
         "label": {
           "resource.labels.location": "europe-west1-b",
           "resource.labels.job": "go-synthetic",
           "project_id": "<redacted>",
           "resource.labels.instance": "<redacted>",
           "resource.labels.cluster": "development",
           "resource.labels.namespace": "<redacted>"
         }
       }
     ]
   }
]