gnmic internal metrics questions

[pobk]

I'm not sure if this is intended or accidental, but I noticed the metrics for the nats-jetstream output encodes worker ID into the publisher_id label in gnmic internal metrics. With an output named nats-jetstream, I see bother publisher_id="nats-jetstream" and publisher_id="nats-jetstream-0".

This seems to be appended with the worker instance to the label in pkg/output/jetstream_output.go around line 550. That makes queries awkward and slightly increases cardinality because the output identity and worker identity are mixed in the label.

Is per-worker visibility important here? If so, would it make sense to keep publisher_id stable (the configured output name) and expose worker as a separate worker label instead?

On a separate point: some internal metric names look a bit repetitive, for example ...prometheus_write_output_number_of_prometheus_write.... Is there a naming convention doc for internal metrics, or appetite to simplify these names? Moreover the label naming doesn't seem to be consistent either (e.g. publisher_id used in nats-jetstream and name used in prometheus_write).

If this makes sense, I'll be happy to put a PR together?

Longer-term idea, (I'm happy to open a separate isue for this): a small consistent set of output metrics (msgs, bytes, success, failures, durations) with output_type, name, and optional worker labels?

[karimra]

Is per-worker visibility important here?

Probably not, I'm ok simplifying it.

On a separate point: some internal metric names look a bit repetitive

That's coming from the subsystem name. I'm fine changing it but that will break dashboards people already built, so I'm bit hesitant.

Moreover the label naming doesn't seem to be consistent either

Not sure it has to be.

Longer-term idea, (I'm happy to open a separate isue for this): a small consistent set of output metrics (msgs, bytes, success, failures, durations) with output_type, name, and optional worker labels?

This makes sense, add a separate set of metrics for (almost) all outputs and keep the current one as is to avoid breaking existing dashboards. I'd say go for it (no need for worker label).

[pobk]

Is per-worker visibility important here?

Probably not, I'm ok simplifying it.

I think moving the worker to a separate worker label makes more sense in case others might be using it as some kind of dimension to their visualisations.

On a separate point: some internal metric names look a bit repetitive

That's coming from the subsystem name. I'm fine changing it but that will break dashboards people already built, so I'm bit hesitant.

Moreover the label naming doesn't seem to be consistent either

Not sure it has to be.

Doesn't have to be, but would make building visualisations and dashboards easier. Think repeating rows based on output type, name, workers, etc

Longer-term idea, (I'm happy to open a separate isue for this): a small consistent set of output metrics (msgs, bytes, success, failures, durations) with output_type, name, and optional worker labels?

This makes sense, add a separate set of metrics for (almost) all outputs and keep the current one as is to avoid breaking existing dashboards. I'd say go for it (no need for worker label).

I'll write up another issue with my thoughts for discussion.