[jmx-scraper] Support for kafka-connect metrics

[atoulme]

Component(s)

No response

Is your feature request related to a problem? Please describe.

See https://docs.confluent.io/platform/current/connect/monitoring.html#connector-metrics

Kafka Connect defines metrics useful for monitoring its usage.

Describe the solution you'd like

We have the following definitions, which we would need to translate over to the jmx scraper yaml syntax and semantic conventions naming:

mBeanDefinitions:
  kafka-connect-worker-metrics:
    objectName: kafka.connect:type=connect-worker-metrics
    values:
    - instancePrefix: kafka-connect-connector-count
      type: gauge
      table: false
      attribute: connector-count
    - instancePrefix: kafka-connect-task-count
      type: gauge
      table: false
      attribute: task-count
    - instancePrefix: kafka-connect-connector-startup-attempts-total
      type: counter
      table: false
      attribute: connector-startup-attempts-total
    - instancePrefix: kafka-connect-connector-startup-success-total
      type: counter
      table: false
      attribute: connector-startup-success-total
    - instancePrefix: kafka-connect-connector-startup-failure-total
      type: counter
      table: false
      attribute: connector-startup-failure-total
    - instancePrefix: kafka-connect-connector-startup-failure-percentage
      type: gauge
      table: false
      attribute: connector-startup-failure-percentage
    - instancePrefix: kafka-connect-connector-startup-success-percentage
      type: gauge
      table: false
      attribute: connector-startup-success-percentage
    - instancePrefix: kafka-connect-task-startup-attempts-total
      type: counter
      table: false
      attribute: task-startup-attempts-total
    - instancePrefix: kafka-connect-task-startup-success-total
      type: counter
      table: false
      attribute: task-startup-success-total
    - instancePrefix: kafka-connect-task-startup-failure-total
      type: counter
      table: false
      attribute: task-startup-failure-total
    - instancePrefix: kafka-connect-task-startup-failure-percentage
      type: gauge
      table: false
      attribute: task-startup-failure-percentage
    - instancePrefix: kafka-connect-task-startup-success-percentage
      type: gauge
      table: false
      attribute: task-startup-success-percentage

  kafka-connect-worker-connector-metrics:
    objectName: kafka.connect:type=connect-worker-metrics,connector=*
    instanceFrom:
      - connector
    values:
    - instancePrefix: kafka-connect-connector-destroyed-task-count
      type: gauge
      table: false
      attribute: connector-destroyed-task-count
    - instancePrefix: kafka-connect-connector-failed-task-count
      type: gauge
      table: false
      attribute: connector-failed-task-count
    - instancePrefix: kafka-connect-connector-paused-task-count
      type: gauge
      table: false
      attribute: connector-paused-task-count
    - instancePrefix: kafka-connect-connector-restarting-task-count
      type: gauge
      table: false
      attribute: connector-restarting-task-count
    - instancePrefix: kafka-connect-connector-running-task-count
      type: gauge
      table: false
      attribute: connector-running-task-count
    - instancePrefix: kafka-connect-connector-total-task-count
      type: gauge
      table: false
      attribute: connector-total-task-count
    - instancePrefix: kafka-connect-connector-unassigned-task-count
      type: gauge
      table: false
      attribute: connector-unassigned-task-count

  kafka-connect-connector-metrics:
    objectName: kafka.connect:type=connector-metrics,connector=*
    instanceFrom:
      - connector
    values:
    - instancePrefix: kafka-connect-connector-class
      type: gauge
      table: false
      attribute: connector-class
    - instancePrefix: kafka-connect-connector-version
      type: gauge
      table: false
      attribute: connector-version
    - instancePrefix: kafka-connect-connector-type
      type: gauge
      table: false
      attribute: connector-type
    - instancePrefix: kafka-connect-connector-status
      type: gauge
      table: false
      attribute: status

  kafka-connect-task-metrics:
    objectName: kafka.connect:type=connector-task-metrics,connector=*,task=*
    instanceFrom:
      - connector
      - task
    values:
    - instancePrefix: kafka-connect-task-batch-size-avg
      type: gauge
      table: false
      attribute: batch-size-avg
    - instancePrefix: kafka-connect-task-batch-size-max
      type: gauge
      table: false
      attribute: batch-size-max
    - instancePrefix: kafka-connect-task-offset-commit-failure-percentage
      type: gauge
      table: false
      attribute: offset-commit-failure-percentage
    - instancePrefix: kafka-connect-task-offset-commit-avg-time-ms
      type: gauge
      table: false
      attribute: offset-commit-avg-time-ms
    - instancePrefix: kafka-connect-task-running-ratio
      type: gauge
      table: false
      attribute: running-ratio
    - instancePrefix: kafka-connect-task-offset-commit-success-percentage
      type: gauge
      table: false
      attribute: offset-commit-success-percentage
    - instancePrefix: kafka-connect-task-offset-commit-max-time-ms
      type: gauge
      table: false
      attribute: offset-commit-max-time-ms
    - instancePrefix: kafka-connect-task-pause-ratio
      type: gauge
      table: false
      attribute: pause-ratio
    - instancePrefix: kafka-connect-task-status
      type: gauge
      table: false
      attribute: status

  kafka-connect-task-error-metrics:
    objectName: kafka.connect:type=task-error-metrics,connector=*,task=*
    instanceFrom:
      - connector
      - task
    values:
    - instancePrefix: kafka-connect-task-total-record-errors
      type: counter
      table: false
      attribute: total-record-errors
    - instancePrefix: kafka-connect-task-total-record-failures
      type: counter
      table: false
      attribute: total-record-failures
    - instancePrefix: kafka-connect-task-last-error-timestamp
      type: gauge
      table: false
      attribute: last-error-timestamp
    - instancePrefix: kafka-connect-task-deadletterqueue-produce-failures
      type: counter
      table: false
      attribute: deadletterqueue-produce-failures
    - instancePrefix: kafka-connect-task-deadletterqueue-produce-requests
      type: counter
      table: false
      attribute: deadletterqueue-produce-requests
    - instancePrefix: kafka-connect-task-total-errors-logged
      type: counter
      table: false
      attribute: total-errors-logged
    - instancePrefix: kafka-connect-task-total-records-skipped
      type: counter
      table: false
      attribute: total-records-skipped
    - instancePrefix: kafka-connect-task-total-retries
      type: counter
      table: false
      attribute: total-retries

  kafka-connect-worker-rebalance-metrics:
    objectName: kafka.connect:type=connect-worker-rebalance-metrics
    values:
    - instancePrefix: kafka-connect-completed-rebalances-total
      type: counter
      table: false
      attribute: completed-rebalances-total
    - instancePrefix: kafka-connect-epoch
      type: gauge
      table: false
      attribute: epoch
    - instancePrefix: kafka-connect-rebalance-avg-time-ms
      type: gauge
      table: false
      attribute: rebalance-avg-time-ms
    - instancePrefix: kafka-connect-rebalance-max-time-ms
      type: gauge
      table: false
      attribute: rebalance-max-time-ms
    - instancePrefix: kafka-connect-rebalancing
      type: gauge
      table: false
      attribute: rebalancing
    - instancePrefix: kafka-connect-time-since-last-rebalance-ms
      type: gauge
      table: false
      attribute: time-since-last-rebalance-ms
    - instancePrefix: kafka-connect-connect-protocol
      type: gauge
      table: false
      attribute: connect-protocol
    - instancePrefix: kafka-connect-leader-name
      type: gauge
      table: false
      attribute: leader-name

  kafka-connect-sink-task-metrics:
    objectName: kafka.connect:type=sink-task-metrics,connector=*,task=*
    instanceFrom:
      - connector
      - task
    values:
    - instancePrefix: kafka-connect-sink-task-offset-commit-completion-rate
      type: gauge
      table: false
      attribute: offset-commit-completion-rate
    - instancePrefix: kafka-connect-sink-task-offset-commit-completion-total
      type: counter
      table: false
      attribute: offset-commit-completion-total
    - instancePrefix: kafka-connect-sink-task-offset-commit-seq-no
      type: gauge
      table: false
      attribute: offset-commit-seq-no
    - instancePrefix: kafka-connect-sink-task-offset-commit-skip-rate
      type: gauge
      table: false
      attribute: offset-commit-skip-rate
    - instancePrefix: kafka-connect-sink-task-offset-commit-skip-total
      type: counter
      table: false
      attribute: offset-commit-skip-total
    - instancePrefix: kafka-connect-sink-task-partition-count
      type: gauge
      table: false
      attribute: partition-count
    - instancePrefix: kafka-connect-sink-task-put-batch-avg-time-ms
      type: gauge
      table: false
      attribute: put-batch-avg-time-ms
    - instancePrefix: kafka-connect-sink-task-put-batch-max-time-ms
      type: gauge
      table: false
      attribute: put-batch-max-time-ms
    - instancePrefix: kafka-connect-sink-task-sink-record-active-count
      type: gauge
      table: false
      attribute: sink-record-active-count
    - instancePrefix: kafka-connect-sink-task-sink-record-active-count-avg
      type: gauge
      table: false
      attribute: sink-record-active-count-avg
    - instancePrefix: kafka-connect-sink-task-sink-record-active-count-max
      type: gauge
      table: false
      attribute: sink-record-active-count-max
    - instancePrefix: kafka-connect-sink-task-sink-record-lag-max
      type: gauge
      table: false
      attribute: sink-record-lag-max
    - instancePrefix: kafka-connect-sink-task-sink-record-read-rate
      type: gauge
      table: false
      attribute: sink-record-read-rate
    - instancePrefix: kafka-connect-sink-task-sink-record-read-total
      type: counter
      table: false
      attribute: sink-record-read-total
    - instancePrefix: kafka-connect-sink-task-sink-record-send-rate
      type: gauge
      table: false
      attribute: sink-record-send-rate
    - instancePrefix: kafka-connect-sink-task-sink-record-send-total
      type: counter
      table: false
      attribute: sink-record-send-total

  kafka-connect-source-task-metrics:
    objectName: kafka.connect:type=source-task-metrics,connector=*,task=*
    instanceFrom:
      - connector
      - task
    values:
    - instancePrefix: kafka-connect-source-task-poll-batch-avg-time-ms
      type: gauge
      table: false
      attribute: poll-batch-avg-time-ms
    - instancePrefix: kafka-connect-source-task-poll-batch-max-time-ms
      type: gauge
      table: false
      attribute: poll-batch-max-time-ms
    - instancePrefix: kafka-connect-source-task-source-record-active-count
      type: gauge
      table: false
      attribute: source-record-active-count
    - instancePrefix: kafka-connect-source-task-source-record-active-count-avg
      type: gauge
      table: false
      attribute: source-record-active-count-avg
    - instancePrefix: kafka-connect-source-task-source-record-active-count-max
      type: gauge
      table: false
      attribute: source-record-active-count-max
    - instancePrefix: kafka-connect-source-task-source-record-poll-rate
      type: gauge
      table: false
      attribute: source-record-poll-rate
    - instancePrefix: kafka-connect-source-task-source-record-poll-total
      type: counter
      table: false
      attribute: source-record-poll-total
    - instancePrefix: kafka-connect-source-task-source-record-write-rate
      type: gauge
      table: false
      attribute: source-record-write-rate
    - instancePrefix: kafka-connect-source-task-source-record-write-total
      type: counter
      table: false
      attribute: source-record-write-total
    - instancePrefix: kafka-connect-source-task-transaction-size-avg
      type: gauge
      table: false
      attribute: transaction-size-avg
    - instancePrefix: kafka-connect-source-task-transaction-size-max
      type: gauge
      table: false
      attribute: transaction-size-max
    - instancePrefix: kafka-connect-source-task-transaction-size-min
      type: gauge
      table: false
      attribute: transaction-size-min

Describe alternatives you've considered

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[SylvainJuge]

A PR has been recently merged in instrumentation to add support for it https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/library/kafka-connect.md

We still need to wait for an instrumentation release, update dependency in contrib, then a contrib release before it gets available in jmx-scraper.

In the mean time you can use a copy of the kafka-connect.yaml file with otel.jmx.config=kafka-connect.yaml

[SylvainJuge]

Also I've opened open-telemetry/opentelemetry-java-instrumentation#16444 to add the missing link in instrumentation.

[atoulme]

Thanks Sylvain!

[atoulme]

In the mean time you can use a copy of the kafka-connect.yaml file with otel.jmx.config=kafka-connect.yaml
correct link appears to be https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/library/src/main/resources/jmx/rules/kafka-connect.yaml