The following tips and commands can help you to troubleshoot your Koperator installation.
First things to do ๐︎
-
Verify that the Koperator pod is running. Issue the following command:
kubectl get pods -n kafka|grep kafka-operator
The output should include a running pod, for example:NAME READY STATUS RESTARTS AGE kafka-operator-operator-6968c67c7b-9d2xq 2/2 Running 0 10m
-
Verify that the Kafka broker pods are running. Issue the following command:
kubectl get pods -n kafka
The output should include a numbered running pod for each broker, with names like kafka-0-zcxk7, kafka-1-2nhj5, and so on, for example:NAME READY STATUS RESTARTS AGE kafka-0-zcxk7 1/1 Running 0 3h16m kafka-1-2nhj5 1/1 Running 0 3h15m kafka-2-z4t84 1/1 Running 0 3h15m kafka-cruisecontrol-7f77ccf997-cqhsw 1/1 Running 1 3h15m kafka-operator-operator-6968c67c7b-9d2xq 2/2 Running 0 3h17m prometheus-kafka-prometheus-0 2/2 Running 1 3h16m
-
If you see any problems, check the logs of the affected pod, for example:
kubectl logs kafka-0-zcxk7 -n kafka
-
Check the status (State) of your resources. For example:
kubectl get KafkaCluster kafka -n kafka -o jsonpath="{.status}" |jq
-
Check the status of your Zookeeper deployment, and the logs of the zookeeper-operator and zookeeper pods.
kubectl get pods -n zookeeper
Check the KafkaCluster configuration ๐︎
You can display the current configuration of your Kafka cluster using the following command:
kubectl describe KafkaCluster kafka -n kafka
The output looks like the following:
Name: kafka
Namespace: kafka
Labels: controller-tools.k8s.io=1.0
Annotations: <none>
API Version: kafka.banzaicloud.io/v1beta1
Kind: KafkaCluster
Metadata:
Creation Timestamp: 2021-02-15T09:46:02Z
Finalizers:
finalizer.kafkaclusters.kafka.banzaicloud.io
topics.kafkaclusters.kafka.banzaicloud.io
users.kafkaclusters.kafka.banzaicloud.io
Generation: 2
Spec:
Broker Config Groups:
Default:
Broker Annotations:
prometheus.io/port: 9020
prometheus.io/scrape: true
Storage Configs:
Mount Path: /kafka-logs
Pvc Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 10Gi
Brokers:
Broker Config Group: default
Id: 0
Broker Config Group: default
Id: 1
Broker Config Group: default
Id: 2
Cluster Image: ghcr.io/banzaicloud/kafka:2.13-2.6.0-bzc.1
Cruise Control Config:
Cluster Config: {
"min.insync.replicas": 3
}
...
Cruise Control Task Spec:
Retry Duration Minutes: 5
Topic Config:
Partitions: 12
Replication Factor: 3
Disruption Budget:
Envoy Config:
Headless Service Enabled: true
Istio Ingress Config:
Listeners Config:
Internal Listeners:
Container Port: 29092
Name: internal
Type: plaintext
Used For Inner Broker Communication: true
Container Port: 29093
Name: controller
Type: plaintext
Used For Controller Communication: true
Used For Inner Broker Communication: false
Monitoring Config:
Jmx Image:
Path To Jar:
One Broker Per Node: false
Read Only Config: auto.create.topics.enable=false
cruise.control.metrics.topic.auto.create=true
cruise.control.metrics.topic.num.partitions=1
cruise.control.metrics.topic.replication.factor=2
Rolling Upgrade Config:
Failure Threshold: 1
Zk Addresses:
zookeeper-client.zookeeper:2181
Status:
Alert Count: 0
Brokers State:
0:
Configuration State: ConfigInSync
Graceful Action State:
Cruise Control State: GracefulUpscaleSucceeded
Error Message: CruiseControl not yet ready
Rack Awareness State:
1:
Configuration State: ConfigInSync
Graceful Action State:
Cruise Control State: GracefulUpscaleSucceeded
Error Message: CruiseControl not yet ready
Rack Awareness State:
2:
Configuration State: ConfigInSync
Graceful Action State:
Cruise Control State: GracefulUpscaleSucceeded
Error Message: CruiseControl not yet ready
Rack Awareness State:
Cruise Control Topic Status: CruiseControlTopicReady
Rolling Upgrade Status:
Error Count: 0
Last Success:
State: ClusterRunning
Events: <none>
Getting Support ๐︎
If you encounter any problems that the documentation does not address, file an issue or talk to us on the Banzai Cloud Slack channel #kafka-operator.
Commercial support is also available for the Koperator .
Before asking for help, prepare the following information to make troubleshooting faster:
- Koperator version
- Kubernetes version (kubectl version)
- Helm/chart version (if you installed the Koperator with Helm)
- Koperator logs, for example kubectl logs kafka-operator-operator-6968c67c7b-9d2xq manager -n kafka and kubectl logs kafka-operator-operator-6968c67c7b-9d2xq kube-rbac-proxy -n kafka
- Kafka broker logs
- Koperator configuration
- Kafka cluster configuration (kubectl describe KafkaCluster kafka -n kafka)
- Zookeeper configuration (kubectl describe ZookeeperCluster zookeeper -n zookeeper)
- Zookeeper logs (kubectl logs zookeeper-operator-5c9b597bcc-vkdz9 -n zookeeper) Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.