Add metrics

[nojnhuh]

• Determine which metrics are valuable for DRA drivers to emit.
  • Ideas: Kubernetes API calls, kubelet gRPC calls.
• Establish a pattern for recording and serving those metrics.

[nmn3m]

Metrics Proposal for DRA Drivers

Hey! I took a look at the codebase and wanted to share some thoughts on what metrics would be valuable for DRA drivers.

Proposed Metrics

High Priority - Kubelet gRPC Calls

These are the most important since they capture the core interactions between kubelet and the driver:

• dra_node_prepare_resources_total (counter) - How many times NodePrepareResources was called
  • Labels: result (success/error)
• dra_node_prepare_resources_duration_seconds (histogram) - How long prepare operations take
• dra_node_unprepare_resources_total (counter) - How many times NodeUnprepareResources was called
  • Labels: result (success/error)
• dra_node_unprepare_resources_duration_seconds (histogram) - How long unprepare operations take
• dra_prepared_claims (gauge) - Number of currently prepared resource claims

Medium Priority - Operational Health

These help understand the internal state of the driver:

• dra_allocatable_devices (gauge) - Total allocatable devices on the node
• dra_checkpoint_operations_total (counter) - Checkpoint save/load operations
  • Labels: operation (save/load), result (success/error)
• dra_cdi_spec_operations_total (counter) - CDI spec file create/delete operations
  • Labels: operation (create/delete), result (success/error)

Lower Priority - Extended Observability

Nice to have for debugging and deeper insights:

• dra_health_checks_total (counter) - Health check invocations
  • Labels: result (success/error)
• dra_webhook_validations_total (counter) - Admission webhook validation requests
  • Labels: result (success/error)
• dra_plugin_info (gauge) - Driver metadata, always set to 1
  • Labels: version, node

Serving Pattern

For exposing these metrics, I was thinking:

1. Use prometheus/client_golang for instrumentation
2. Expose a /metrics HTTP endpoint on the existing healthcheck port (51515)
3. Optionally add a ServiceMonitor to the Helm chart for Prometheus Operator users

Open Questions

A few things I'd love your input on:

1. Should metric names use dra_example_ prefix (driver-specific) or just dra_ (as a pattern other drivers could follow)?
2. Is it worth tracking device-level metrics like per-GPU sharing strategy usage?

Let me know what you think! Happy to work on a PR if this direction looks good.