Banzai Cloud Blog

In-depth introduction to Kubernetes admission webhooks

Banzai Cloud’s Pipeline platform is an operating system which allows enterprises to develop, deploy and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operation teams alike. One of the main features of the Pipeline platform is that it allows enterprises to run workloads cost effectively by mixing spot instances with regular ones, without sacrificing overall reliability.

Read more...


Placeholder image

Balint Molnar

Fri, Sep 21, 2018

Satellite - simple Golang library to provide cloud agnosticity

Satellite is a Golang library and RESTful API that determines a host’s cloud provider with a simple HTTP call. Behind the scenes, it uses file systems and provider metadata to properly identify cloud providers. When we started to work on Pipeline and the Banzai Cloud Pipeline Platform Operators, we soon realized how frequently we would need to find out which cloud provider the service was actually running on. Note that Pipeline supports 6 different cloud providers

Read more...


Placeholder image

Toader Sebastian

Mon, Sep 17, 2018

Monitoring Spark with Prometheus, metric name preprocessing and customizable metric metadata

Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded Hands on Thanos Monitoring Vault on Kubernetes using Cloud Native technologies At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline.

Read more...


Placeholder image

Laszlo Puskas

Fri, Sep 14, 2018

Enhancing the cloud product information

The Banzai Cloud Productinfo service retrieves product and pricing information from cloud providers and exposes it through a RESTful API, and UI. Our Kubernetes based Pipeline platform and Telescopes recommendation engine make use of this information when they advise users on cluster layout and resourcing. Here’s a quick primer of how and why we utilize the Productinfo service: Pipeline platform users have the option of launching clusters or deploying applications based only on resource- and SLA-requirements (price, IO, memory, CPU, GPU, etc.

Read more...


Placeholder image

Attila Prekopcsak

Wed, Sep 12, 2018

Monitoring Golang web apps using Gin and Prometheus

Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded Hands on Thanos Monitoring Vault on Kubernetes using Cloud Native technologies In depth application observability is a must-have feature for us; we put significant effort into monitoring all our customers’ Kubernetes deployments on the Pipeline platform, as well as the platform itself.

Read more...


Placeholder image

Sandor Magyari

Mon, Sep 10, 2018

The Horizontal Pod Autoscaler operator reloaded

One of the main advantages of the Pipeline platform is that it allows users to use their infrastructure cost effectively; Telescopes helps with cluster and machine instance recommendations, Hollowtrees enables SLA-aware cost reduction using spot instances, and autoscalers allow for multi-dimensional autoscaling based on custom metrics. This post will highlight some new features of the Banzai Cloud Horizontal Pod Autoscaler Kubernetes Operator and the advanced automation supported by Pipeline - a new, forward-thinking way to operate Kubernetes clusters and autoscale deployments.

Read more...


Placeholder image

Ferenc Hernadi

Thu, Sep 6, 2018

Monitoring Vault on Kubernetes using Cloud Native technologies

Two core features of the Pipeline platform are advanced security and observability for all Kubernetes deployments. We make sure that all secrets are securely stored, transported, images scanned, deployments monitored, and logs centrally collected. As you might expect from us, we use the industry standard in security components, such as Vault, Prometheus, Grafana, Fluent and more. This post is about monitoring Vault with Prometheus (on Kubernetes) and displaying metrics on Grafana.

Read more...


Taints and tolerations, pod and node affinities demystified

Enterprises often use multi-tenant and heterogenous clusters to deploy their applications to Kubernetes. These applications usually have needs which require special scheduling constraints. Pods may require nodes with special hardware, isolation, or colocation with other pods running in the system. The Pipeline platform allows users to express their constraints in terms of resources (CPU, memory, network, IO, etc.). These requirements are turned into infrastructure specifications using Telescopes. Once the cluster nodes are created and properly labeled by Pipeline, deployments are run with the specified constraints automatically on top of Kubernetes.

Read more...


Placeholder image

Janos Matyas

Thu, Aug 30, 2018

Banzai Cloud announces collaboration with Oracle and becomes technology partner

Banzai Cloud announced today that it is collaborating with Oracle to bring its feature-rich application platform to Oracle Kubernetes Engine users. Banzai Cloud’s Pipeline deployment automation and execution engine enables developers to go from commit to scale in minutes by automating all the underlying tasks that provide convenient CI/CD flows, robust security, analytics and the ability to scale. The technology not only provides significant productivity gains to developers, but also increases operational efficiencies by aiding instance selection and automated introspection of large-scale workloads.

Read more...


Placeholder image

Márk Sági-Kazár

Mon, Aug 27, 2018

Error handling practices in Go

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. The platform itself consists of many building blocks - over 30 components - but they share one commonality: they are all developed in Golang. Obviously we are very fond of Go and like it quite a bit, so in this post we’d like to share the error handling practices that our team of 20+ developers adheres to while building the Pipeline platform.

Read more...


Placeholder image

Nandor Kracser

Thu, Aug 23, 2018

Alibaba Cloud Container Service - enhanced

Last year Alibaba joined CNCF and announced plans to create their own Kubernetes service - Alibaba CSK. The service was luanched more than a year ago, with its stated objective to make it easy to run Kubernetes on Alibaba Cloud without needing to install, operate, and maintain a Kubernetes control plane. At Banzai Cloud we are committed to providing support for Kubernetes on all major cloud providers, thus one of our priorities was to enable Alibaba Cloud’s Container Service for Kubernetes in Pipeline and take the DevOps experience to the next level by turning CSK into a feature-rich enterprise-grade application platform.

Read more...


Placeholder image

András Tóth

Mon, Aug 20, 2018

Prepare Node.js apps production ready for Kubernetes

At Banzai Cloud we are building an application-centric platform for containers - Pipeline - running on Kubernetes to allow developers to go from commit to scale in minutes. We support multiple development languages and frameworks to build applications, with one common goal: all Pipeline deployments receive integrated CI/CD, centralized logging, monitoring, enterprise-grade security, autoscaling, and spot price support automatically, out of the box. In most cases we accomplish this in a non-intrusive way (i.

Read more...


Placeholder image

Sandor Magyari

Mon, Aug 13, 2018

Deploying Zeppelin and Spark on Kubernetes using Helm charts

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Toader Sebastian

Wed, Aug 8, 2018

Amazon EKS - enhanced and simplified

Banzai Cloud is happy to announce that it is an Amazon EKS Platform Partner. Last year Amazon joined CNCF and announced plans to create their own Kubernetes service - Amazon EKS. The service has been launched this June, with the objective to make it easy for you to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. At Banzai Cloud we are committed to provide support for Kubernetes on all major cloud providers for our users thus one of our priority was to enable Amazon EKS in Pipeline and take the DevOps and user experience to the next level by turning EKS into a feature rich enterprise-grade application platform.

Read more...


Placeholder image

Janos Matyas

Mon, Aug 6, 2018

Cloud agnostic (aka multi-cloud) Kubernetes with Pipeline

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. We have always been committed to supporting Kubernetes and our container based application platform on all major providers, however, we are also committed to making portability between cloud vendors easy, seamless and automated. Accordingly, this post will highlight a few important aspects of a multi-cloud approach we learned from our users, and the open source code we developed and made part of the Pipeline platform.

Read more...


Placeholder image

Sandor Guba, Balint Molnar

Thu, Aug 2, 2018

Centralized logging on Kubernetes automated

At Banzai Cloud we put a lot of emphasis on observability, so we automatically provide centralized monitoring and log collection for all clusters and deployments done through Pipeline. Over the last few months we’ve been experimenting with different approaches - tailored and driven by our customers’ individual needs - the best of which are now coded into our open source Logging-Operator. Just to recap, here are our earlier posts about logging using the fluent ecosystem Centralized log collection on Kubernetes.

Read more...


Placeholder image

Ferenc Hernadi

Mon, Jul 30, 2018

Oracle Kubernetes Engine (OKE) - the easy way

Continuing our commitment to support all major cloud providers, today we are adding support for Oracle’s Kubernetes-managed cloud service, OKE – Oracle Kubernetes Engine in Pipeline. We are building a feature rich enterprise-grade application platform on top of Kubernetes - called Pipeline - to deliver a better DevOps experience by automating the lifecycle management of applications. This experience is now available to OKE users as well; here, we will guide you through the first steps of using OKE and summarize the benefits of using Pipeline.

Read more...


Placeholder image

Nandor Kracser

Thu, Jul 26, 2018

Kubernetes API proxy with Pipeline

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers, specifically AWS, GCP, Azure, AliCloud, Oracle and BYOC - on-premise and hybrid - and deploy all kinds of predefined or ad-hoc workloads to these clusters. For us and our enterprise users authentication and authorization is absolutely vital, thus, in order to access the Kubernetes API and the Services in an authenticated manner as defined within Kubernetes, we arrived at a simple but flexible solution.

Read more...


Placeholder image

Sandor Magyari

Tue, Jul 24, 2018

Horizontal Pod Autoscaler Kubernetes Operator

A few months ago the Kubernetes Operator SDK was released with one of its goals being the conversion of human operational knowledge into code. At Banzai Cloud we’ve been contributors and early adopters of this technology, since it provides a better standardized method of automating our processes and allows us to dramatically ease the lives of our customers. We are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline, wherein we endeavour to automate the DevOps experience and the lifecycle of deployments.

Read more...


Placeholder image

Balint Molnar

Thu, Jul 19, 2018

Apache Spark (Driver) resilience on Kubernetes - network partitioning

At Banzai Cloud we are building a feature-rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. Applications deployed to Pipeline automatically inherit the platform’s features: enterprise-grade security, observability (centralized log collection, monitoring and tracing), discovery, high availability and resiliency, just to name a few - encapsulated in spotguides. One of the most popular spotguides we deploy is Spark. In the past few months we’ve been working and pushing many pull requests to make Spark a first class player on Kubernetes and to make it resilient.

Read more...


Placeholder image

Laszlo Puskas

Mon, Jul 16, 2018

Cloud agnostic cluster recommendations for Kubernetes

A few weeks back we released Telescopes, our Kubernetes cluster layout recommender application. That application has evolved quite a bit, and in this post we’ll provide insight into some its new features and recent changes. Cloud cost management series: Overspending in the cloud Managing spot instance clusters on Kubernetes with Hollowtrees Monitor AWS spot instance terminations Diversifying AWS auto-scaling groups Draining Kubernetes nodes tl;dr: We added new features to Telescopes to provide support for blacklisting or whitelisting instance types Recommendation accuracies can now be checked There is now support that allows asking cloud instance types for CPU, memory and network performance.

Read more...


Placeholder image

Gabor Kozma

Thu, Jul 12, 2018

Running MySQL on Kubernetes using an operator

One of our goals at BanzaiCloud is to make our customers’ lives easier by providing low barrier to entry, easy to use solutions for running applications on Kubernetes. To achieve this, we often rely on Kubernetes Operators to provide comprehensive solutions over the course of an application’s lifecycle. Here is a list of our operators, which we have already open sourced: Vault Operator Prometheus JMX Exporter Operator

Read more...


Placeholder image

Janos Matyas

Mon, Jul 9, 2018

Vulnerability scans on Kubernetes with Pipeline

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. Security is one of our main areas of focus, and we strive to automate and enable those security patterns we consider essential, including tier zero features for all enterprises using the Pipeline Platform. We’ve blogged about how to handle security scenarios on several of our previous posts. This time we’d like to focus on a different aspect of securing Kubernetes deployments:

Read more...


Placeholder image

Janos Matyas

Fri, Jul 6, 2018

Banzai Cloud has joined CNCF

We are excited to announce that Banzai Cloud has joined the Cloud Native Computing Foundation! The CNCF and The Linux Foundation are expending extraordinary effort in helping to standardize open source technologies that enable the development, deployment, management and operation of next generation Cloud Native software stacks. Our mission is to bring Cloud Native to enterprises and with this announcement, we strive to help push container and cloud native technology standardization and interoperability forward.

Read more...


Placeholder image

Toader Sebastian

Thu, Jul 5, 2018

Kubernetes and external DNS services

At Banzai Cloud we are building a feature-rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. For an enterprise-grade application platform security is a must and it has many building blocks. Please read through the Security series on our blog to learn how we deal with a variety of security-related issues. Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS

Read more...


Placeholder image

Nandor Kracser

Mon, Jul 2, 2018

Kubernetes secret management with Pipeline

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers such as AWS, GCP, Azure, Oracle, Alibaba and BYOC, on-premise and hybrid, and deploy all kinds of predefined or ad-hoc workloads to these clusters. For us and our enterprise users, Kubernetes secret management (base 64) was not sufficient, so we chose Vault and added Kubernetes support to manage our secrets.

Read more...


Placeholder image

Sandor Magyari

Thu, Jun 28, 2018

Horizontally autoscale Kubernetes deployments on custom metrics

If you followed our blog series on Autoscaling on Kubernetes, you should already be familiar with Kubernetes’ Cluster autoscaler and the Vertical Pod Autoscaler used with Java 10 applications. This post will show you how to use the Horizontal Pod Autoscaler to autoscale your deployments based on custom metrics obtained from Prometheus. As a deployment example we’ve chosen our JEE Petstore example application on Wildfly to show that, beside metrics like cpu and memory, which are provided by default on Kubernetes, using our Wildfly Operator, all Java and Java Enterprise Edition / Wildfly specific metrics are automatically placed at your fingertips, available in Prometheus, allowing you to easily autoscale deployments.

Read more...


Placeholder image

Sandor Guba

Mon, Jun 25, 2018

Hands on Thanos

Here at Banzai Cloud we blog a lot about Prometheus and how to use it. One of the problems we have so far neglected to discuss is the inadequate long term storage capability of Prometheus. Luckily a new project called Thanos seeks to address this. If you are not familiar with Prometheus, or are interested in other monitoring related articles, check out our monitoring series, here: Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded

Read more...


Placeholder image

Marton Sereg

Thu, Jun 21, 2018

Cloud instance type and price information as a service

When we started to work on our cluster infrastructure recommender, Telescopes, we soon realized how difficult it was to get instance type attributes and pricing information from cloud providers programatically. While EC2, Google Cloud, and Azure all provide some kind of API from which to query this information, in some cases these APIs respond with partially inconsistent data, or their responses are large chunks of JSON files that are very cumbersome to parse.

Read more...


Placeholder image

Nandor Kracser

Mon, Jun 18, 2018

Vault unseal flow with KMS

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, called Pipeline. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers such as AWS, GCP, Azure and BYOC, on-premise and hybrid, and deploy all kinds of predefined or ad-hoc workloads to these clusters. For us and our enterprise users, Kubernetes secret management (Base64) was woefully inadequate, so we chose Vault with native Kubernetes support to manage our secrets.

Read more...


Placeholder image

Flora Piszker

Wed, Jun 13, 2018

Azure Managed Service Identity and Role Assign with AKS

This post highlights how the Pipeline Platform enables Managed Service Identity (MSI) and assigns the Storage Account Contributor role to AKS cluster Virtual Machines. But wait, why? At Banzai Cloud we have a PVC Operator, which makes using Kubernetes Persistent Volumes easier on cloud providers by dynamically creating the required accounts and storage classes. That operator allows us to use the same Helm chart on all supported providers, thus there is no need to create cloud specific Helm charts.

Read more...


Placeholder image

Toader Sebastian

Mon, Jun 11, 2018

Vertical pod autoscaler

At Banzai Cloud we provision all kinds of applications to Kubernetes and we try to autoscale these clusters with Pipeline and/or properly size application resources as needed. As promised in an earlier blog post, How to correctly size containers for Java 10 applications, we’ll share our findings on the Vertical Pod Autoscaler(VPA) used with Java 10. VPA sets resource requests on pod containers automatically, based on historical usage, thus ensuring that pods are scheduled onto nodes where appropriate resource amounts are available for each pod.

Read more...


Placeholder image

Janos Matyas

Fri, Jun 8, 2018

Running Infinispan on Kubernetes

One of our goals with Pipeline is to support Java and Java Enterprise Edition deployments, allowing developers to iterate fast while building and deploying safe, and also pushing code to production. In order to do that, we place a lot of importance on different aspects of a Java/JEE application’s lifecycle - we allow engineers: To continuously integrate and deploy their Java apps to Kubernetes To deploy Java Enterprise Edition applications to Kubernetes Once the Java containers are deployed to K8s, to avoid OOMKills To correctly size Java containers And, once deployments are done and sized, to monitor them without any code modification Enter Infinispan - a distributed cache and data grid.

Read more...


Placeholder image

Sandor Magyari

Wed, Jun 6, 2018

Autoscaling Kubernetes clusters

One of our goals at Banzai Cloud is to eliminate the concept of nodes, insofar as that is possible, so that users will only be aware of their applications and respective resource needs (cpu, gpu, memory, network, etc). Launching Telescopes was a first step in that direction - helping end users to select the right instance types for the job, through Telescopes infrastructure recommendations, then turning those recommendations into actual infrastructure with Pipeline.

Read more...


Placeholder image

Nandor Kracser

Mon, Jun 4, 2018

Golang runtime Logrus Formatter

For our Pipeline Platform, observability is an essential part of operating distributed applications in production. We put a great deal of effort into monitoring large and federated clusters, and automating the centralized log collection of these clusters with Pipeline. That way, all our users get out-of-the-box observability for free. Logging series: Centralized logging under Kubernetes Secure logging on Kubernetes with Fluentd and Fluent Bit Advanced logging on Kubernetes

Read more...


Placeholder image

Balint Molnar

Thu, May 31, 2018

Kafka on Kubernetes with Local Persistent Volumes

At Banzai Cloud we use Kafka internally a lot. We have some internal systems and customer reporting deployments where we rely heavily on Kafka deployed to Kubernetes. We practice what we preach and all these deployments (not just the external ones) are done using our application platform, Pipeline. There is one difference between regular Kafka deployments and ours (though it is not relevant to this post): we have removed Zookeeper and use etcd instead.

Read more...


Placeholder image

Laszlo Puskas

Wed, May 30, 2018

POC to production - transition of code

At Banzai Cloud we’re always open to experimenting with and integrating new software (tools, products). We also love to validate our new ideas by quickly implementing “proof of concept” projects. Even though we used five or so programming languages while building the Pipeline Platform, we love and use Golang the most. While these PoC projects are not intended for production use, they often serve as the basis for it. When this is the case, the PoC code needs to be refactored - or prepared for production.

Read more...


Effortless monitoring of Java applications on Kubernetes

At Banzai Cloud we place a lot of emphasis on the observability of applications deployed to the Pipeline Platform, which we built on top of Kubernetes. To this end, one of the key components we use is Prometheus. Just to recap, Prometheus is: an open source systems monitoring and alerting tool a powerful query language (PromQL) a pull based metrics gathering system a simple text format for metrics exposition Problem statement Usually, legacy applications are not exactly prepared for these last two, so we need a solution that bridges the gap between systems that do not speak the Prometheus metrics format: enter exporters.

Read more...


Placeholder image

Nandor Kracser

Thu, May 24, 2018

kurun

During the development of the Pipeline Platform all of its key building blocks such as Pipeline, Hollowtrees and Bank-Vaults have relied on making extensive Kubernetes API calls. Often, we tried a quick K8s API call or ran a small PoC inside a cluster, while also wanting to avoid the usual deployment process. We quickly realized that we needed a shortcut. There are tools like telepresence that support slightly more complex scenarios.

Read more...


Placeholder image

Flora Piszker

Wed, May 23, 2018

Create Kubernetes clusters in the cloud

Creating Kubernetes clusters in the cloud and deploying (or CI/CDing) applications to those clusters is not always simple. There are a few conventional options, but they are either cloud or distribution specific. While we were working on our open source Pipeline Platform, we needed a solution which covered (here follows an inclusive but not exhaustive list of requirements): provisioning of Kubernetes clusters on all major cloud providers (via REST, UI and CLI) using a unified interface application lifecycle management (on-demand deploy, CI/CD, dependency management, etc) preferably over a REST interface support for multi tenancy, and advanced security scenarios (app to app security with dynamic secrets, standards, multi-auth backends, and more) ability to build cross-cloud or hybrid Kubernetes environments This posts highlights the ease of creating Kubernetes clusters using the Pipeline API on the following providers:

Read more...


PVC Operator; Creating Persistent Volume on Kubernetes made simple

At Banzai Cloud we continue to work hard on the Pipeline platform we’re building on Kubernetes. We’ve open sourced quite a few operators already, and even recently teamed up with Red Hat and CoreOS to begin work on Kubernetes Operators using the new Operator SDK, and to help move human operational knowledge into code. The purpose of this blog will be to take a dive deep into the PVC Operator.

Read more...


Placeholder image

Nandor Kracser

Wed, May 16, 2018

The Banzai Cloud Vault Operator

At Banzai Cloud we’re building a feature rich platform, Pipeline, on top of Kubernetes. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers - AWS, GCP, Azure and BYOC - and deploy all kinds of predefined or ad-hoc workloads to these clusters. We wanted to set the industry standard for the way in which our users log in and interact with secure endpoints, and, at the same time, we wanted to provide dynamic secret management for each application we support.

Read more...


Placeholder image

Toader Sebastian

Mon, May 14, 2018

How to correctly size containers for Java 10 applications

At Banzai Cloud we run and deploy containerized applications to our PaaS, Pipeline. Java or JVM-based workloads, are among the notable workloads deployed to Pipeline, so getting them right is pretty important for us and our users. Java/JVM based workloads on Kubernetes with Pipeline Why my Java application is OOMKilled Deploying Java Enterprise Edition applications to Kubernetes A complete guide to Kubernetes Operator SDK Spark, Zeppelin, Kafka on Kubernetes

Read more...


Placeholder image

Sandor Magyari

Thu, May 10, 2018

Deploying Java Enterprise Edition applications to Kubernetes

A good number of years ago, back at beginning of this century, most of us here at Banzai Cloud were in the Java Enterprise business, building application servers (BEA Weblogic and JBoss) and JEE applications. Those days are gone; the technology stack and landscape has dramatically changed; monolithic applications are out of fashion, but we still have lots of them running in production. Because of our background, we have a personal investment in helping to shift Java enterprise edition business applications towards microservices, managed deployments, Kubernetes, and the cloud using Pipeline.

Read more...


Placeholder image

Janos Matyas

Tue, May 8, 2018

Banzai Cloud announces collaboration with Red Hat and becomes technology partner

Banzai Cloud announced today that it will collaborate with Red Hat in an effort to help standardize the management of complex stateful applications on Kubernetes. Simultaneously, Red Hat announced the Operator Framework, an open source toolkit designed to manage application instances on Kubernetes in a more effective, automated, and scalable way. Banzai Cloud will work collaboratively with Red Hat on this open source project, which will focus on the creation of a new Software Development Kit (SDK) for the “operators” pattern.

Read more...


Placeholder image

Laszlo Puskas

Mon, May 7, 2018

Deploy Node.js applications to Kubernetes

The Pipeline PaaS contains a complete CI/CD component to support developers building, deploying and operating applications in an automated way on Kubernetes. Most of our documentation, blog posts and how-tos have focused on Spark, Zeppelin and Tensorflow examples. However, it is possible to build and deploy any application with Pipeline’s CI/CD component. Our last post about the Banzai Cloud CI/CD flow described how to build/deploy a Spring Boot application on Kuberbetes.

Read more...


Placeholder image

Marton Sereg

Thu, May 3, 2018

Cloud instance type recommendation

Cloud cost management series: Overspending in the cloud Managing spot instance clusters on Kubernetes with Hollowtrees Monitor AWS spot instance terminations Diversifying AWS auto-scaling groups Draining Kubernetes nodes A few months ago we posted on this blog about overspending in the cloud. We discussed how difficult it is to keep track of the vast array of instance types and pricing options offered by cloud providers, especially on AWS with spot pricing.

Read more...


Placeholder image

Toader Sebastian

Tue, May 1, 2018

A complete guide to Kubernetes Operator SDK

At Banzai Cloud we are always looking for new and innovative technologies to support our users with their transition towards microservices deployed to Kubernetes, using Pipeline. In the recent months we have been partnered with CoreOS and RedHat to work on operators and the project it has just been made open source today and available on GitHub. If you read through this blog you’ll learn what is an operator, how to use the operator sdk to develop an operator through a concrete example that we developed and used here at Banzai Cloud.

Read more...


Placeholder image

Janos Matyas

Sun, Apr 29, 2018

Banzai Cloud @ KubeCon, Copenhagen

This week KubeCon + CloudNativeCon, Copanhagen is bringing together over 4000+ developers, architects and people from the cloud native open source communities. We are part of this community and the CNCF landscape as certified experts of these technologies running under the umbrella of the Cloud Naticve Computing Foundation so we could not miss the event. Join us to learn about Kubernetes and the related technologies directly from the the experts of the industry.

Read more...


Placeholder image

Janos Matyas

Fri, Apr 27, 2018

Banzai Cloud is now a Kubernetes Certified Service Provider

We are excited to announce that Banzai Cloud has become a Kubernetes Certified Service Provider (KCSP). The KCSP program was started by the Cloud Native Computing Foundation in collaboration with the Linux Foundation and it is a major milestone to help enterprises move to a cloud native platform. It provides a strict set of rules and certified experts to guarantee that only experienced partners are part of the program. This creates a trust relationship as enterprises can rely on Banzai Cloud and our flagship PaaS, Pipeline bringing in the necessary experience and guide them on their Kubernetes and microservices journey to cloud native application platforms and production usage.

Read more...


Placeholder image

Nandor Kracser

Thu, Apr 26, 2018

The Vault swiss-army knife

Bank Vaults is a thick, tricky, shifty right with a fast and intense tube for experienced surfers only, located on Mentawai. Think heavy steel doors, secret unlocking combinations and burly guards with smack-down attitude. Watch out for clean-up sets. Bank Vaults is a wrapper for the official Vault client with automatic token renewal, built in Kubernetes support, dynamic database credential management, multiple unseal options, automatic re/configuration and more.

Read more...


Placeholder image

Balint Molnar

Wed, Apr 25, 2018

Kubernetes persistent volume options

At Banzai Cloud we push different types of workload to Kubernetes with our open source PaaS, Pipeline. There are lots of deployments we support and have defined the Helm charts however Pipeline is able to deploy applications from any repository. These deployments are pushed on-prem or in the cloud but among many there is one common feature, the need for persistent volumes. The options provided by Kubernetes are abundant and every cloud provider has a custom/additional offering as well.

Read more...


Placeholder image

Sandor Guba

Mon, Apr 23, 2018

Advanced logging on Kubernetes

We continue our series about Kubernetes logging and this post will cover some advanced techniques and visualizations of the collected logs. Just to recap, with our open source PaaS, Pipeline we are monitoring and collecting/moveing large amounts of logs of the distributed applications we push to Kubernetes. We are putting huge efforts to monitor large and federated clusters and automating all these with Pipeline so all our users are getting out of the box monitoring and log collection for free.

Read more...


Placeholder image

Lajos Papp

Fri, Apr 20, 2018

Control your AWS spendings with ChatOps

While we are building our open source, cloud agnostic Heroku / Cloud Foundry like Paas, Pipeline built on top of Kubernetes we launch lots of clusters on different cloud providers. Most of these clusters are launched on spot or preemptible instances and managed by Hollowtrees, however there are many smaller development clusters, control planes, instances and proof of concepts we regularly do and they are marginally related or launched with Pipeline.

Read more...


Placeholder image

Janos Matyas

Wed, Apr 18, 2018

Policy enforcements on K8s with Pipeline

In the past few weeks we have been blogging about the advanced, enterprise-grade security features we are building into our open source PaaS, Pipeline. To recap these features please read the series here: Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

Read more...


Placeholder image

Sandor Magyari

Mon, Apr 16, 2018

Collecting Spark History Server event logs in the cloud

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes

Read more...


Placeholder image

Toader Sebastian

Fri, Apr 13, 2018

Apache Spark application resilience on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Collecting Spark History Server event logs in the cloud

Read more...


Secure logging on Kubernetes with Fluentd and Fluent Bit

As we concluded in the previous blog post we continue this series about centralized and secure Kubernetes logging/log collection.Log messages can contain sensitive information thus it is important to secure the transport between the distributed parts of the log flow. This post describes how we have secured moving log messages on our Kubernetes clusters provisioned by Pipeline. Logging series: Centralized logging under Kubernetes Secure logging on Kubernetes with Fluentd and Fluent Bit

Read more...


Placeholder image

Laszlo Puskas

Tue, Apr 10, 2018

Manage Helm repositories and deploy charts via REST

During the development of our open source Pipeline PaaS, we introduced some handy features to deal with deployments. Note that most of our applications are deployed as Helm releases and we needed a way to interact programatically (using gRPC) and using a UI (RESTful API) with Helm. In order to do that with Pipeline we have introduced a nice feature to be able to manage Helm repositories and deploy applications with Helm to Kubernetes using RESTful API calls.

Read more...


Placeholder image

Marton Sereg

Mon, Apr 9, 2018

Draining Kubernetes nodes

Cloud cost management series: Overspending in the cloud Managing spot instance clusters on Kubernetes with Hollowtrees Monitor AWS spot instance terminations Diversifying AWS auto-scaling groups Draining Kubernetes nodes Cluster recommender Cloud instance type and price information as a service Kubernetes was designed in a way to be fault tolerant to worker node failures. If a node goes missing because of a hardware problem, a cloud infrastructure problem, or in general Kubernetes simply no longer receives heartbeat messages from that node because of any reason, the Kubernetes control plane is clever enough to handle these failures.

Read more...


Placeholder image

Nandor Kracser

Thu, Apr 5, 2018

Dynamic SSH with Vault and Pipeline

At Banzai Cloud, we are building a feature rich platform as a service, Pipeline - built on Kubernetes. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers as AWS, GCP, Azure and BYOC, and deploy all kind of predefined or ad-hoc workloads to these clusters. We needed to find an industry standards-based way for our users to login and interact with protected endpoints and at the same time provide dynamic secrets management for all the different applications we support, all these with native Kubernetes support and we chose to standardize on Vault.

Read more...


Placeholder image

Laszlo Puskas

Tue, Apr 3, 2018

CI/CD for Kubernetes, through a Spring Boot example

The Pipeline PaaS contains a complete CI/CD component to support developers building, deploying and operating applications in an automated way, deployed to Kubernetes. Most of our documentation, blog posts and howtos were focusing on Spark, Zeppelin and Tensorflow examples, however we can actually build and deploy any application with Pipeline’s CI/CD component. This post showcases how to enable a simple Spring Boot application for the Banzai Cloud CI/CD flow, build and save the artifacts and deploy it to a Kubernetes cluster.

Read more...


Placeholder image

Ferenc Hernadi

Fri, Mar 30, 2018

Centralized logging under Kubernetes

For our Pipeline PaaS, monitoring is an essential part of operating distributed applications in production. We are placing large efforts to monitor large and federated clusters and automating all these with Pipeline so all our users are getting out of the box monitoring for free. You can read about our monitoring series below here: Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded

Read more...


Placeholder image

Janos Matyas

Tue, Mar 27, 2018

The future of big data is Kubernetes

For some time we’ve been evangelizing the idea that the runtime fabric for big data workloads should be Kubernetes. In this post I’d like to walk through the reasoning behind the change and discuss the benefits of it. Obliviously this is a pretty large topic and this post has no intentions to cover it all - also it’s an opinionated view, we at Banzai Cloud believe and push for.

Read more...


Placeholder image

Marton Sereg

Mon, Mar 26, 2018

Fn and Hollowtrees

Adoption of serverless technologies is quickly emerging. According to this survey it is on par with containers. And even though serverless is a very vague term and it can be argued that it is still rarely used in production especially in complex applications, it seems to be sure that it will be one of the most dominant trends in the near future in the cloud computing space. While a few years ago serverless only meant AWS Lambda in its early stages, nowadays the category is maturing rapidly.

Read more...


Placeholder image

Nandor Kracser

Sun, Mar 25, 2018

Secure Kubernetes Deployments with Vault and Pipeline

This is a copy of our guest post we published on the Hashicorp blog about how we use Vault with Kubernetes. At Banzai Cloud, we are building an open source next generation platform as a service, Pipeline - built on Kubernetes. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers and deploy different workloads to these clusters. We needed to find an industry standards-based way for our users to publish and interact with protected endpoints and at the same time provide dynamic secrets management for all the different applications we support, all these with native Kubernetes support.

Read more...


Placeholder image

Flora Piszker

Wed, Mar 21, 2018

Pipeline PaaS 0.3.0 - new release

Banzai Pipeline, or simply Pipeline is a tabletop reef break located in Hawaii, Oahu’s North Shore. The most famous and infamous reef on the planet is forming the benchmark by which all other waves are measured. Pipeline is a PaaS with a built in CI/CD engine to deploy cloud native microservices in public cloud and on-premise. It simplifies and abstracts all the details of provisioning the cloud infrastructure, installing or reusing the Kubernetes cluster and deploying the application.

Read more...


Placeholder image

Sandor Guba

Tue, Mar 20, 2018

Kubernetes port hunting

Part of the Debug 101 series, we are back with a small but annoying bug hunting. This kind of bug is not really a bug but the side effect of several tools working together. Here comes the trouble I was deploying a development version of Pipeline on a Kubernetes cluster running on top of an AWS infrastructure. To do this deployment I’ve used the following Helm chart command. $: helm install --name pipeline banzaicloud-stable/pipeline-cp \ --set=drone.

Read more...


Placeholder image

Balint Molnar

Mon, Mar 19, 2018

Kubeless using Kafka on etcd

Kubeless has been designed as a Kubernetes native serverless framework, and for PubSub functions is using Apache Kafka behind the scenes. At Banzai Cloud we like cloud-native technologies, however we were not happy about operating a Zookeeper cluster on Kubernetes, thus we have modified and open sourced a version for Kafka where we have replaced Zookeeper with etcd, which is a better fit. This post is part of the serverless series talking about how to deploy Kubeless using Kafka on etcd with Pipeline and deploy a so called PubSub function.

Read more...


Placeholder image

Janos Matyas

Thu, Mar 15, 2018

Monitoring Spark with Prometheus, reloaded

Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded At Banzai Cloud we deploy large distributed applications to Kubernetes and operate these clusters as well. We don’t like to get a PagerDuty notification during the night so we try to get ahead of these issues by operating these clusters as efficient as we can.

Read more...


Placeholder image

Nandor Kracser

Wed, Mar 14, 2018

Dynamic credentials with Vault using Kubernetes Service Accounts

Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

Read more...


Placeholder image

Gabor Kozma

Mon, Mar 12, 2018

Monitoring Apache Kafka with Prometheus

Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded At Banzai Cloud we provision and monitor large Kubernetes clusters deployed to multiple cloud/hybrid environments using Prometheus. The clusters and the applications or frameworks are all managed by our next generation PaaS, Pipeline.

Read more...


Placeholder image

Toader Sebastian

Wed, Mar 7, 2018

Fn - a container native serverless platform

At Banzai Cloud we are constantly searching products/frameworks to enable in our open source PaaS, Pipeline that adds value to businesses. Serverless frameworks are among those, thus today we are adding Fn as a supported spotguide to make it easy for users to deploy it with Pipeline on their preferred cloud provider. Before we dive into how to deploy and use Fn with Pipeline a few of the reasons why we thought that Fn should be supported by Pipeline:

Read more...


Placeholder image

Sandor Magyari

Mon, Mar 5, 2018

Distributed Tensorflow deployed to Azure AKS Kubernetes using GPU instances

In our last post about distributed TensorFlow we used a research example for distributed training of an Inception model. In this episode we will showcase how to run the same example on GPU instances, this time on Azure managed Kubernetes, AKS deployed with Pipeline. As you might be already familiar with the previous post, among the first things to consider when running distributed Tensorflow models is to have some shared storage available.

Read more...


Placeholder image

Ferenc Hernadi

Tue, Feb 27, 2018

Play With Ingress Authentication

At Banzai Cloud we secure our Kubernetes services using Vault and OAuth2 tokens. This has not always been the case, however we had authentication in the project (even though it was basic) from a very early PoC stage - and suggest all to do so. Usually, inbound connections to Kubernetes cluster services are done via ingress. Just to recap, all public services are typically accessed through a loadbalancer service, however, this can get quite expensive.

Read more...


Placeholder image

Sandor Guba

Mon, Feb 26, 2018

Application monitoring with Prometheus and Pipeline

Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded At Banzai Cloud we provision and monitor large Kubernetes clusters deployed to multiple cloud/hybrid environments. The clusters and the applications or frameworks are all managed by our next generation PaaS, Pipeline.

Read more...


Placeholder image

Balint Molnar

Wed, Feb 21, 2018

Spark Streaming Checkpointing on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Toader Sebastian

Mon, Feb 19, 2018

Function as a service with OpenFaaS on Banzai Cloud Pipeline

At Banzai Cloud we provision different frameworks and tools like Spark, Zeppelin, Kafka, Tensorflow, etc to our Pipeline PaaS (built on Kubernetes). Last week we have added serverless capabilities to Pipeline, using OpenFaas. This blog post explains how to deploy OpenFaaS to Kubernetes using Pipeline and invoke an example function running on it. We shall separate the provisioning of the serverless frameworks we support (this post is about OpenFaaS but Pipeline equally supports Kubeless as well) and the invocation of functions through the Pipeline API or CI/CD workflow dispatched to any of the supported serverless frameworks we deploy to Kubernetes.

Read more...


Placeholder image

Laszlo Puskas

Wed, Feb 14, 2018

CI/CD flow for Zeppelin notebooks

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Marton Sereg

Mon, Feb 12, 2018

Diversifying AWS auto-scaling groups, or how to write a Hollowtrees action plugin

Cloud cost management series: Overspending in the cloud Managing spot instance clusters on Kubernetes with Hollowtrees Monitor AWS spot instance terminations Diversifying AWS auto-scaling groups Draining Kubernetes nodes Cluster recommender Cloud instance type and price information as a service You may remember the Hollowtrees project we’ve open sourced a few weeks ago - a framework to manage AWS spot instance clusters with a few batteries included: Hollowtrees is an alert-react based framework part of the Pipeline PaaS which coordinates monitoring, applies rules and dispatches action chains towards plugins using standard CNCF interfaces AWS spot instance termination Prometheus exporter AWS autoscaling group Prometheus exporter AWS Spot Instance recommender Kubernetes action plugin to execute k8s operations (e.

Read more...


Placeholder image

Balint Molnar

Thu, Feb 8, 2018

Kafka on Kubernetes - using etcd

At Banzai Cloud we are building a cloud agnostic, open source next generation CloudFoundry/Heroku like PaaS - Pipeline and running several big data workloads natively on Kubernetes. Apache Kafka is one of those cloud native workloads we support out of the box - beside Apache Spark and Apache Zeppelin. In case you are interested in running big data workloads on Kubernetes please read the following blog series as well.

Read more...


Placeholder image

Gabor Kozma

Wed, Feb 7, 2018

Monitoring multiple federated clusters with Prometheus - the secure way

At Banzai Cloud we run multiple Kubernetes clusters deployed with our next generation PaaS, Pipeline and we deploy these clusters across different cloud providers like AWS, Azure, Google or on-prem. These clusters are usually launched using the same control plane deployed either to AWS as a CloudFormation template or Azure as an ARM template and they are running inside a Kubernetes cluster as well (we eat our own dog food).

Read more...


Placeholder image

Marton Sereg

Mon, Feb 5, 2018

Monitor AWS spot instance terminations

Cloud cost management series: Overspending in the cloud Managing spot instance clusters on Kubernetes with Hollowtrees Monitor AWS spot instance terminations Diversifying AWS auto-scaling groups Draining Kubernetes nodes Cluster recommender Cloud instance type and price information as a service Last week we have opensourced the Hollowtrees project - a framework to manage AWS spot instance clusters with a few batteries included: Hollowtrees is an alert-react based framework part of the Pipeline PaaS which coordinates monitoring, applies rules and dispatches action chains towards plugins using standard CNCF interfaces AWS spot instance termination Prometheus exporter AWS autoscaling group Prometheus exporter AWS Spot Instance recommender Kubernetes action plugin to execute k8s operations (e.

Read more...


Placeholder image

Toader Sebastian

Thu, Feb 1, 2018

Spark scheduling on Kubernetes demystified

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Janos Matyas

Tue, Jan 30, 2018

Pipeline PaaS 0.2.0 - new release

Banzai Pipeline, or simply Pipeline is a tabletop reef break located in Hawaii, Oahu’s North Shore. The most famous and infamous reef on the planet is forming the benchmark by which all other waves are measured. Pipeline is a PaaS with a built in CI/CD engine to deploy cloud native microservices in public cloud and on-premise. It simplifies and abstracts all the details of provisioning the cloud infrastructure, installing or reusing the Kubernetes cluster and deploying the application.

Read more...


Placeholder image

Marton Sereg

Mon, Jan 29, 2018

Managing spot instance clusters on Kubernetes with Hollowtrees

Hollowtrees is a wave for the highest level, the pin-up centerfold for the Mentawai islands bringing a new machine-like level to the word perfection. Watch out for the vigilant guardian aptly named The Surgeons Table, whose sole purpose is to take parts of you as a trophy. Hollowtrees, a ruleset based watch-guard is keeping spot instance based clusters safe and allows to use them in production. Handles spot price surges within one region or availability zone and reschedules applications before instances are taking down.

Read more...


Placeholder image

Janos Matyas

Fri, Jan 26, 2018

Authentication and authorization of Pipeline users with OAuth2 and Vault

Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

Read more...


Placeholder image

Sandor Magyari

Wed, Jan 24, 2018

Spark application logs - History Server setup on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Flora Piszker

Mon, Jan 22, 2018

The challenges (and resolutions) of working with Azure AKS

We are moving rather fast with new Pipeline features and releases, the second major one scheduled for this week. Among many new features we have added a new managed Kubernetes provider, Microsoft’s Azure AKS. Azure Container Service (AKS) is a preview feature of the Azure Cloud - and we are proud of being very early adopters of it. We can provision and deploy apps to Kubernetes on Azure VMs same as we do on EC2, however at Banzai Cloud we strongly believe that the future is in managed Kubernetes services, and most of our investment regarding cloud neutrality and provisioning is built on managed Kubernetes services both in the cloud (GKE, OCI and ACS in beta or under developent) and on-prem.

Read more...


Placeholder image

Sandor Magyari

Thu, Jan 18, 2018

Introduction to distributed TensorFlow on Kubernetes

Last time we were discussing about how our Pipeline PaaS is deploying and provisioning an AWS EFS filesystem on Kubernetes and what are the performance benefits for Spark or TensorFlow. This post is about: Introduction to TensorFlow on Kubernetes Benefits of EFS for TensorFlow (store image data for TensorFlow jobs) Pipeline uses the kubeflow framework to deploy: A JupyterHub to create & manage interactive Jupyter notebooks A TensorFlow Training Controller that can be configured to use CPUs or GPUs A TensorFlow Serving container Note that beside the ones above Pipeline also has default Spotguides for Spark and Zeppelin as well to support your datascience experience

Read more...


Placeholder image

Sandor Magyari

Mon, Jan 15, 2018

Amazon Elastic File System on Kubernetes

At Banzai Cloud we provision different frameworks and tools like Spark, Zeppelin and most recently Tensorflow, all running on our Pipeline PaaS (built on Kubernetes). One of Pipeline’s early adopter is running a Tensorflow Training Controller using GPUs on AWS EC2 wired into our CI/CD pipeline and needed significant parallelization for reading training data. We have introduced support for Amazon Elastic File System and will make it publicly available in the forthcoming release of Pipeline.

Read more...


Placeholder image

Janos Matyas

Thu, Jan 11, 2018

Running TiDB on Kubernetes

At Banzai Cloud we provision different applications or frameworks to our PaaS - Pipeline, built on Kubernetes. At the same time we eat our own dogfood and the PaaS’ control plane itself is running on Kubernetes and needs a data storage layer. So we needed to cover two use cases - deploy and run a distributed, scalable and fully SQL compliant DB to cover our client’s and our own internal needs.

Read more...


Placeholder image

Toader Sebastian

Tue, Jan 9, 2018

Why my Java application is OOMKilled

At Banzai Cloud we run and deploy containerized applications to our PaaS, Pipeline. Like us, those who already ran Java application inside Docker have probably came across the problem of the JVM incorrectly detecting the available memory when running inside of the container. The JVM rather sees the available memory of the machine instead of the memory available only to the Docker container. This can lead to cases where applications running inside the container is killed when tries to use more memory beyond the limits of the Docker container.

Read more...


Placeholder image

Sandor Magyari

Mon, Jan 8, 2018

Running Zeppelin Spark notebooks on Kubernetes - deep dive

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Laszlo Puskas

Thu, Jan 4, 2018

Take a rest - enjoy the REST

Modern applications and services usually expose their functionalities via REST; moreover modules and components also can make use of external services that again are exposed as REST. Thus developers often need to design RESTful services and write REST service clients. This kind of work implies calling these services thousands of times during the development process (developers need to understand the API, the messages and the resources involved) and even after it to make sure everything works as desired.

Read more...


Placeholder image

Toader Sebastian

Tue, Jan 2, 2018

The anatomy of Spark applications on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Janos Matyas

Wed, Dec 27, 2017

Top 3 blogs of 2017 and what’s next

As 2017 comes to an end, we are looking back at the top three blog posts that were most popular with our readers. We can’t really look too far back (though we had 13 posts and one release already) as we basically started our startup just a little bit over one month (November 20, 2017 to be more precise) but during this short time period we achieved quite a lot and laid the foundation to some exciting new projects we plan to ship early next year.

Read more...


Placeholder image

Balint Molnar

Thu, Dec 21, 2017

Debugging a jetcd Txn Bug

This post is part of the Debug 101 series - if you missed the previoius one check it here: Nodes successfully joined, not! We are in the middle of deploying Apache Kafka to Kubernetes the cloud native way - by totally removing the Zookeeper dependency and using etcd instead. All service registry/discovery and other internal Kafka to Zookeeper operations are dispatched to the already existing etcd cluster. Sweet, isn’t it - no need to yet another third party system as we already have etcd part of Kubernetes out of the box.

Read more...


Placeholder image

Miklos Csendes

Wed, Dec 20, 2017

Introduction to spotguides

Last week we have released the first version of Pipeline - with end to end support for cloud native apps starting from a GitHub commit hook deployed into the cloud in minutes using a fully customizable CI/CD workflow. The core part of the Pipeline PaaS is spotguides - a collection of workflow/pipeline steps defined in a .pipeline.yml file and a few Drone plugins. In this post we would like to demystify spotguides and describe step by step how they work; the next post will be a tutorial of how to write a custom spotguide and an associated plugin.

Read more...


Placeholder image

Balint Molnar

Mon, Dec 18, 2017

Monitoring Apache Spark with Prometheus on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Laszlo Puskas

Thu, Dec 14, 2017

Apache Spark CI/CD workflow howto

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Janos Matyas

Tue, Dec 12, 2017

Pipeline PaaS - the first release

Banzai Pipeline, or simply Pipeline is a tabletop reef break located in Hawaii, Oahu’s North Shore. The most famous and infamous reef on the planet is forming the benchmark by which all other waves are measured. Pipeline is a PaaS with a built in CI/CD engine to deploy cloud native microservices in public cloud and on-premise. It simplifies and abstracts all the details of provisioning the cloud infrastructure, installing or reusing the Kubernetes cluster and deploying the application.

Read more...


Placeholder image

Marton Sereg

Thu, Dec 7, 2017

Overspending in the cloud

Cloud cost management series: Overspending in the cloud Managing spot instance clusters on Kubernetes with Hollowtrees Monitor AWS spot instance terminations Diversifying AWS auto-scaling groups Draining Kubernetes nodes Cluster recommender Cloud instance type and price information as a service One of the main advantages that is always brought up when debating whether it’d be good to move a deployment to the cloud is cost. There are no upfront costs in the cloud because you don’t have to buy the hardware, and you’ll only pay for what you really use because you can scale your infrastructure based on your workloads.

Read more...


Placeholder image

Sandor Magyari

Tue, Dec 5, 2017

Running Zeppelin Spark notebooks on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Janos Matyas

Mon, Dec 4, 2017

Banzai Cloud @ KubeCon + CloudNativeCon, North America

This week KubeCon + CloudNativeCon, North America is bringing together over 2500+ developers, architects and people from the cloud native open source communities. We are part of this community by contributing and using these technologies running under the umbrella of the Cloud Naticve Computing Foundation so we could not miss the event. Join us to learn about Kubernetes and the related technologies directly from the the experts of the industry.

Read more...


Placeholder image

Toader Sebastian

Fri, Dec 1, 2017

Scaling Spark made simple on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Sandor Magyari

Wed, Nov 29, 2017

Nodes successfully joined, not!

Debug 101 Today we are starting a new series called Debug 101 - dealing with issues which gave us significant headaches and we spent lots of time to debug, understand and fix the problems. We strongly believe in open source software and open issue resolution and we try to describe the problems and suggest fixes, thus you don’t have to shave that yak. We already did, and it looks awesome.

Read more...


Placeholder image

Janos Matyas

Mon, Nov 27, 2017

Introduction to Spark on Kubernetes

Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Apache Spark CI/CD workflow howto Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive CI/CD flow for Zeppelin notebooks

Read more...


Placeholder image

Flora Piszker

Fri, Nov 24, 2017

Azure Managed Kubernetes (AKS) Go SDK

At Banzai Cloud we are using different cloud providers or managed Kubernetes offerings and one of these clusters we use is Microsoft Azure Managed Kubernetes. It is a pretty neat service and gives you a managed K8S cluster without the need of dealing with low level Kubernetes building blocks or tooling, nor starting with cloud infrastructure provisioning. However there is one temporary issue which is cornerstone for our PasS, Pipeline - the Azure Go-SDK does not contain the bindings for this new service.

Read more...


Placeholder image

Janos Matyas

Thu, Nov 23, 2017

The company I'd like to work for

While I had no intention to make or join a new startup (after a successful exit which was a good financial decision but turned out to be the worst professional one) a few former co-founders from SequenceIQ and friends I have been working together at Fathom Technology/Epam Systems approached me after I got back home from my pretty long surfing trip. Few of them moved to work on a project for a banking giant to do microservice based Java applications scheduled with Nomad.

Read more...