Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact
Sign in
Author Adam Dudas

CI/CD secret management on Kubernetes

At Banzai Cloud we are building a managed Cloud Native application and devops platform, called Pipeline. Pipeline supercharges the development, deployment and scaling of container-based applications with native support for multi- and hybrid-cloud environments.

Pipeline’s built-in CI/CD solution is capable of creating Kubernetes clusters, running and testing builds, packaging and deploying applications as Helm charts, and lots more—while the secrets are stored and managed by Vault.

If you’d like to read more about the CI/CD system’s other features, such as native Kubernetes support, unprivileged builds and more, please read this post.

Secrets and CI

These days CI/CD systems are doing way more than just building code: they are accessing SCM repositories, pushing artifacts, creating clusters, and more—all these in multi-tenant and (in many cases) open source environments. Meanwhile, developers within an organization have different roles and authorizations, however all of them should be able to start builds and check their outcomes and logs. It’s fair to say that the above are hard to manage, requires lots of maintenance and are way beyond storing in environment variables, files or being stuck in CI descriptors.

Also, secrets should be short lived, or volatile—they should be easily and often revoked and replaced (especially when they are compromised), and all this should be automated.

We quickly realized these while working on Pipeline and:

  • we standardized on Vault and open sourced a Vault operator for Kubernetes
  • our enterprise customers were not happy with Kubernetes secrets (base64 encoding, that’s it) thus we allow them to inject secrets directly into pods from Vault
  • we run all CI builds in unprivileged mode within Kubernetes
  • secrets are pulled from Vault in ephemeral and persistent form
  • in Pipeline CI/CD descriptor secrets are referenced only, and binding is automatically done only when it’s required and for only as long as it’s required
  • additionally, build artifacts and images can be security scanned as part of the CI pipeline

Secret management in CI, the Pipeline way

In Pipeline, users are members of organizations. Users may invite other users into organizations, and a user can be a member of more than one organization at a time. These organizations can have different types of associated resources: Kubernetes clusters created inside organizations, object storage buckets, spotguide catalogs, etc. One feature that all of these have in common is that they require secrets to work properly. Usually, these secrets allow Pipeline to interact with your cloud provider account in order to provision and manage any underlying resources. Secrets may also be associated with applications running on your Kubernetes clusters, like a TLS certificate chain for MySQL, or a Kubernetes cluster’s own credentials. Secrets are stored in Vault with added layers of abstraction that allow them to be provided to applications and build pipelines.

Pipeline Secrets

Using secrets in build steps

At some point you’ll probably be executing a build step that needs to use a credential for its job. In the CI/CD flow, which runs natively on Kubernetes, Pipeline secrets can be injected directly into Pods that represent build steps. If you need one of your secrets for a specific build step, you can specify it in the step’s secretFrom parameter. This binds the value of a secret’s field to an environment variable that’s supplied to your build step. For example, you can use a secret that’s holding Docker Hub credentials to build and push an image:

  # ...
    image: plugins/docker
    repo: foo/bar
        name: my-docker-secret
        keyRef: username
        name: my-docker-secret
        keyRef: password

To use a Pipeline secret in the steps of a specific CI/CD project, you must first attach the secret. On the web interface, go to the CI/CD page, click on the project’s settings button, and attach the secret you want to use.

Attaching Pipeline Secrets

A plugin’s documentation will specify the name of the environment variables that the plugin will expect the credentials to be in, but you have the freedom to choose how you store them in Pipeline. Generally speaking, it’s possible to create a Generic-type secret, and add fields to it with the same names as the variables. Then the keys in the secretFrom object will be repeated as keyRefs, and the name parameter will be the same for all variables. However, the docker plugin example above uses a Password type secret, which has username and password fields. These are mapped to anticipated variable names.

Installing secrets to clusters

If your application (your Helm charts) need credentials in the form of Kubernetes Secrets, you can install them to the cluster in order to let the Pods of a given namespace consume them. The following code snippet is an example configuration for installing a secret:

    image: banzaicloud/ci-pipeline-client:0.7
    action: InstallSecret
      sourceSecretName: pipeline-secret-name
      name: installed-secret-name
      namespace: default

With the help of the Pipeline CI Client plugin, users can define how these secrets should be injected into the Kubernetes cluster their build is running on.

Learn more about the Banzai Cloud Pipeline platform CI system

In case you are interested in learning more about the built-in CI system, please follow the documentation.

If you’d like to learn more about Banzai Cloud check out our other posts in the blog, the Pipeline, Hollowtrees and Bank-Vaults projects on Github or follow us on our social channels.



comments powered by Disqus