Placeholder image

Nandor Kracser

Wed, May 16, 2018

The Banzai Cloud Vault Operator

At Banzai Cloud we are building a feature rich platform, Pipeline - built on top of Kubernetes. With Pipeline we provision large, multi-tenant Kubernetes clusters on all major cloud providers such as AWS, GCP, Azure and BYOC and deploy all kind of predefined or ad-hoc workloads to these clusters. We had to find an industry-standard way for our users to login and interact with secured endpoints and the same time provide dynamic secret management for each application we support. With all these requirements in mind extended with native Kubernetes support we chose Vault.

This operator (and the Bank-Vaults project itself) are all part of the Pipeline Platform. We strive to offer enterprise grade security to our users and applications deployed to Kubernetes through Pipeline - this is a tier zero feature of our Pipeline Platform.

Security series:
Authentication and authorization of Pipeline users with OAuth2 and Vault
Dynamic credentials with Vault using Kubernetes Service Accounts
Dynamic SSH with Vault and Pipeline
Secure Kubernetes Deployments with Vault and Pipeline
Policy enforcement on K8s with Pipeline
The Vault swiss-army knife

In a previous blog post we have already described what Operators for Kubernetes are, in this blog post we explain how our Vault Operator has been born on top of the new SDK and our Bank Vaults project.

Banzai Cloud Vault Operator

There is already a Vault Operator developed by CoreOS.

The Vault Operator makes it easier to install, manage, and maintain instances of Vault – a tool designed for storing, managing, and controlling access to secrets, such as tokens, passwords, certificates, and API keys – on Kubernetes clusters.

A big thumbs up for release it into the wild! It is a great tool for installing a self contained instance of Vault and etcd on top of Kubernetes. With their inspirational work we have started to work our own solution. Although the idea is similar we started from scratch based on the brand new Operators SDK that really speeds up Operator development.

We have some additional features that we missed from the existing operators we could find as opensource projects on GitHub:

  • Automatic Vault initialization
  • Root Token and Unseal Keys encrypted and stored in cloud KMS systems (Azure Key Vault, AWS KMS, GCP KMS)
  • Also they can be stored in Kubernetes Secrets (however this is not supposed to be used in production, because the current limitations of Kubernetes Secrets, see this doc for more details)
  • Automated unsealing
  • Automated re/configuration of Vault based on a YAML/JSON file like: Auth backends, Secret backends and policies
  • It is not tied to etcd at all, you can choose your own storage backend (e.g. cloud provider storages)

These features are already embedded in our Bank-Vaults project (the heavy lifting is done there already) - the most comprehensive open source project built on top of Vault. This means the whole Vault experience is still cloud-agnostic as it was before but the Operator extends that with event handling.

All the above mentioned configuration is done in a single YAML file. An example Vault Custom Resource Definition looks like this:

 1apiVersion: ""
 2kind: "Vault"
 4  name: "vault"
 6  size: 1
 7  image: vault:0.10.1
 8  bankVaultsImage: banzaicloud/bank-vaults:latest
10  # Describe where you would like to store the Vault unseal keys and root token.
11  unsealConfig:
12    # In this case we are storing the root token and the unseal keys in Kubernetes secrets.
13    kubernetes:
14      secretNamespace: default
16  # A YAML representation of a final vault config file.
17  # See for more information.
18  config:
19    storage:
20      file:
21        path: "/vault/file"
22    listener:
23      tcp:
24        address: ""
25        tls_cert_file: /vault/tls/server.crt
26        tls_key_file: /vault/tls/server.key
27    ui: true
29  # See: for more details.
30  externalConfig:
31    policies:
32    - name: allow_secrets
33      rules: path "secret/*" {
34              capabilities = ["create", "read", "update", "delete", "list"]
35            }
37    auth:
38    - type: kubernetes
39      roles:
40        # Allow every pod in the default namespace to use the secret kv store
41        - name: default
42          bound_service_account_names: default
43          bound_service_account_namespaces: default
44          policies: allow_secrets
45          ttl: 1h

One might find strange where are the TLS certificates coming from in this configuration. Don’t worry, they are automatically generated by the Operator via Sprig functions, which are used in Helm as well.

The main benefit of this Operator in contrast to our Vault Helm chart is that we can react to certain Vault events in a standardized way (via the Operators framework). Vault is represented as a Kubernetes resource with its state stored from the Vault cluster.

Vault Operator flow

Local Vault operator

This is the simplest setup. The following commands will install a single node Vault instance that stores unseal and root tokens in Kubernetes secrets:

git clone
cd bank-vaults
kubectl apply -f deploy/rbac.yaml
kubectl apply -f deploy/operator.yaml
kubectl apply -f deploy/cr.yaml

A few seconds later you can check the existing operator and vault pods:

kubectl get pods

NAME                                                        READY     STATUS    RESTARTS   AGE
vault-66f484898d-lbltm                                      2/2       Running   0          10s
vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s

Vault HA in Google Cloud

To demonstrate the HA setup we deploy Vault on Google Cloud. For this example we assume you have a GKE Cluster and configured the Service Account with the following IAM Roles:

  • Cloud KMS Admin
  • Cloud KMS CryptoKey Encrypter/Decrypter
  • Storage Admin

If you don’t have a running GKE Cluster provision one with one click using Pipeline.

We assume that your kubectl is pointing to the GKE cluster, so you can continue deploying the operator:

git clone
cd bank-vaults

# You can't change RBAC if this command is not executed (you need to get the cluster-admin role):
kubectl create clusterrolebinding ${YOUR-NAME}-cluster-admin-binding --clusterrole=cluster-admin --user=${YOUR_EMAIL}
kubectl apply -f deploy/rbac.yaml
kubectl apply -f deploy/operator.yaml

# First edit this file to change the project settings, see below:
kubectl apply -f deploy/cr-gcs-ha.yaml

Here you can se an example HA configuration:

 1apiVersion: ""
 2kind: "Vault"
 4  name: "vault"
 6  size: 3
 7  image: vault:0.10.1
 8  bankVaultsImage: banzaicloud/bank-vaults:master
10  # Describe where you would like to store the Vault unseal keys and root token
11  # in GCS encrypted with KMS.
12  # NOTE: please use your own project values, otherwise this example won't work:
13  unsealConfig:
14    google:
15      kmsKeyRing: "vault" # CHANGEME
16      kmsCryptoKey: "vault-unsealer" # CHANGEME
17      kmsLocation: "global" # CHANGEME
18      kmsProject: "continual-air-196513" # CHANGEME
19      storageBucket: "vault-ha" # CHANGEME
21  # A YAML representation of a final vault config file, this config represents
22  # a HA config in Google Cloud.
23  # See for more information.
24  config:
25    storage:
26      gcs:
27        bucket: "vault-ha"
28        ha_enabled: "true"
29    listener:
30      tcp:
31        address: ""
32        tls_cert_file: /vault/tls/server.crt
33        tls_key_file: /vault/tls/server.key
34    api_addr: https://vault.default:8200
35    ui: true

Prometheus metrics are also in the making

We plan to add Vault Prometheus metrics also to the Operator’s feature set (a feature that the CoreOS operator already has), but first we would like to carefully analyze the available options, decide what would the best option, for more details please see the attached issue. We do place significant effort in observability and we do pretty advanced monitorings of our federated clusters, so stay tuned as this will land soon.

Unseal from mobile, through a quorum

Although there are options to unseal Vault (the most extensive options publicly available) already built in Bank-Vaults we don’t stop there. Since the operator is capable of listening to Kubernetes events - thus Vault restarts and statuses as well - we can simply notify alerting systems and notify operators to unseal Vault for example in case of Vaults has crashed. This opens up the possibility to unseal Vault with human intervention by unseal keys distributed amongst a group of people, with the keys will be stored on their mobile devices (or laptop). Those quorum of people would get a notification on their mobile devices and if they allow, the application will unseal Vault right from their phones. The Vault operator would open up an unseal port to minimize the attack window only for the limited time during Vault is sealed.

We will be releasing soon an iOS and Android app, and a Golang binary for different OS

Learn by the code

The project is open source of course and the code is tightly integrated into the Bank-Vaults project and can be found here in our GitHub repository.

We place significant effort to secure our users of the Pipeline Platform, as this is a core building block of it.

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn or Twitter:



comments powered by Disqus