The Banzai Cloud Kafka operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of Apache Kafka clusters deployed to K8s.

Overview 🔗︎

Apache Kafka is an open-source distributed streaming platform, and some of the main features of the Kafka-operator are:

  • the provisioning of secure and production-ready Kafka clusters
  • fine grained broker configuration support
  • advanced and highly configurable External Access via LoadBalancers using Envoy
  • graceful Kafka cluster scaling and rebalancing
  • monitoring via Prometheus
  • encrypted communication using SSL
  • automatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins) using Cruise Control
  • graceful rolling upgrade
  • advanced topic and user management via CRD

Kafka-operator architecture

We took a different approach to what’s out there - we believe for a good reason - please read on to understand more about our design motivations and some of the scenarios which were driving us to create the Banzai Cloud Kafka operator.

The Banzai Cloud Kafka operator is a core part of Banzai Cloud Supertubes that helps you create production-ready Kafka cluster on Kubernetes, with scaling, rebalancing, and alerts based self healing. While the Kafka operator itself is an open-source project, the Banzai Cloud Supertubes product extends the functionality of the Kafka operator with commercial features (for example, built-in monitoring and multiple ways of disaster recovery). Read a detailed comparison of Supertubes and the Kafka operator.

Motivation 🔗︎

At Banzai Cloud we are building a Kubernetes distribution, PKE, and a hybrid-cloud container management platform, Pipeline, that operate Kafka clusters (among other types) for our customers. Apache Kafka predates Kubernetes and was designed mostly for static on-premise environments. State management, node identity, failover, etc all come part and parcel with Kafka, so making it work properly on Kubernetes and on an underlying dynamic environment can be a challenge.

There are already several approaches to operating Kafka on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our customers’ needs. At the same time, there is substantial interest within the Kafka community for a solution which enables Kafka on Kubernetes, both in the open source and closed source space.

We took a different approach to what’s out there - we believe for a good reason - please read on to understand more about our design motivations and some of the scenarios which were driving us to create the Banzai Cloud Kafka operator.

Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator. We are long term contributors and active community members of both Apache Kafka and Kubernetes, and we hope to recreate a similar community around this operator.

If you are willing to kickstart your managed Apache Kafka experience on 5 cloud providers, on-premise or hybrid environments, check out the free developer beta of the Banzai Cloud Pipeline platform: