This documentation shows you how to enable custom monitoring on a Kafka cluster installed using the Kafka operator.

Using Helm for Prometheus 🔗︎

By default operator installs Kafka Pods with the following annotations, also it opens port 9020 in all brokers to enable scraping.

    "": "true"
    "":   "9020"

Prometheus must be configured to recognize these annotations. The following example contains the required config.

# Example scrape config for pods
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
# * ``: Only scrape pods that have a value of `true`
# * ``: If the metrics path is not `/metrics` override this.
# * ``: Scrape the pod on the indicated port instead of the default of `9102`.
- job_name: 'kubernetes-pods'

    - role: pod

    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__

Using the provided CR, the operator installs the official jmx exporter for Prometheus.

To change this behavior, modify the following lines in the end of the CR.

   jmxImage describes the used prometheus jmx exporter agent container
    jmxImage: "banzaicloud/jmx-javaagent:0.12.0"
   pathToJar describes the path to the jar file in the given image
    pathToJar: "/opt/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar"
   kafkaJMXExporterConfig describes jmx exporter config for Kafka
    kafkaJMXExporterConfig: |
     lowercaseOutputName: true

Using the ServiceMonitors 🔗︎

To use ServiceMonitors, we recommend to use Kafka with unique service/broker instead of headless service.

Configure the CR the following way:

  # Specify if the cluster should use headlessService for Kafka or individual services
  # using service/broker may come in handy in case of service mesh
  headlessServiceEnabled: false

Disabling Headless service means the operator will set up Kafka with unique services per broker.

Once you have a cluster up and running create as many ServiceMonitors as brokers.

kind: ServiceMonitor
  name: kafka-0
      app: kafka
      brokerId: "0"
      kafka_cr: kafka
  - port: metrics
    interval: 10s