Placeholder image

Janos Matyas

Mon, Jul 9, 2018


Vulnerability scans on Kubernetes with Pipeline

At Banzai Cloud we are building a feature rich enterprise-grade application platform, built for containers on top of Kubernetes, Pipeline. Security is one of our main focus thus we strive to automate and enable good security patterns we consider must haves - a tier zero feature for all enterprises using the Pipeline Platform.

We have been blogging about how we handle different security scenarios on several previous posts, and this time we’d like to focus on a different aspect of securing Kubernetes deployments:

  • static code scans part of our CI/CD Pipeline
  • container vulnerability scans part of the same CI/CD Pipeline
  • Kubernetes deployment scans/rescans at the time of deployment and post deployment

Security series:
Authentication and authorization of Pipeline users with OAuth2 and Vault
Dynamic credentials with Vault using Kubernetes Service Accounts
Dynamic SSH with Vault and Pipeline
Secure Kubernetes Deployments with Vault and Pipeline
Policy enforcement on K8s with Pipeline
The Vault swiss-army knife
The Banzai Cloud Vault Operator
Vault unseal flow with KMS

tl;dr:

We have released Helm charts to deploy Clair and Sonar to Kubernetes and Drone CI/CD plugins to run static code analysis and container vulnerability scans.

Secure CI/CD flow in Pipeline

The problem statement

A few weeks ago the news broke that a few container images available on Docker Hub (for over 8 months) were mining cryptocurrencies beside the original purpose of the image.

Backdoored images downloaded 5 million times finally removed from Docker Hub. 17 images posted by a single account over 10 months may have generated $90,000.

How was this possible, many might ask? The reason is pretty simple - we are living a paradigm shift where each and every developer is becoming a DevOps engineer but lacking the skills or not paying enough attention to this role. In the world of VMs, DevOps/infra/security teams used to validate the OS images, installed packages for vulnerabilities and released them. With the wide adoption of Kubernetes and containers in general, developers are building their own container images, however these images are rarely built from scratch. They are typically built on some base images, which are built on top of other base images and the layers below the actual application are built on layers from public third party sources.

Oversimplifying it, this brings in the following problems:

  • images and libraries may contain obsolete or vulnerable packages
  • many existing legacy vulnerability-scanning tools may not work with containers
  • existing tools are hard to integrate into new delivery models

Over 80% of the latest versions of official images publicly available on Docker Hub contained at least one high severity vulnerability!

How Pipeline automates vulnerability scans

Banzai Cloud is a Kubernetes Certified Solution Provider and one of our main mission statement is to Bring Cloud Native to the Enterprise. We have deployed and managed large Kubernetes deployments with Pipeline and during this process we have learned a few things that:

  • enterprises demands strict security (and they like Pipeline as we enforce that)
  • developers many times overlook security as the main focus is to make it work on my computer and this often ends up in production
  • they need a Platform where these strict security enforcements can’t be bypassed
  • the entry barrier should be as low as possible, ideally close to zero
  • security should a tier-zero feature

While we are building the Platform and adding new features we learned and listened and we are now enabling and automating the following steps for Pipeline users:

  • Static code analysis and vulnerability scans using our CI/CD pipeline plugin.
  • Once the build artifacts are created the CI/CD pipeline creates the containers. Once containers are created we scan each layer for vulnerabilities using data from the CVE databases.
  • If all scans pass Pipeline pushes the containers to a container registry or creates a Kubernetes deployment. Based on a configurable frequency we re-scan these Kubernetes deployment.
  • In case vulnerabilities are found during the deployment lifecycle we allow enterprises to apply several strategies as: notifications (default operation, we notify by email or Slack), gracefully remove the deployment, snapshot, fallback to previous deployment (less likely will pass the scan as usually the vulnerabilities are in 3rd party layers).

Under the hood - how it works

The Pipeline control plane hosts all the components for static code scans and container vulnerability checks (Sonar, Clair, databases, etc) and offers this as a service. There is hard (schema) isolation per organization or department, namespace separation for scans, OAuth2 JWT token based security and integration with Vault. As usual for any secrets in Pipeline (e.g. private container registry access) they are encrypted and stored inside Vault.

Static code analysis - Sonar

The CI/CD pipeline checks out the code inside the Kubernetes cluster owned by the organization/developer triggering the build via a GitHub webhook. The static code analysis runs inside the Kubernetes cluster, however the result is pushed back to the Pipeline control plane. Each organization has it’s own database schema, and for accessing the Sonar interface we use OAuth2 JWT tokens (as basically everywhere inside Pipeline). Depending on the organization ACLs and GitHub groups individuals might or might not see each other builds, or organization level results.

For our enterprise users we support LDAP and AD authentication and authorization based on roles. Pipeline is integrated with Dex to support multiple identity provider backends.

By default Pipeline is non intrusive - beside raising alerts - in a sense that does not break the build, however rules can be configured at organization level or profiles (e.g. test, UAT, production) and enforced to builds triggered by developers belonging to the organization.

Static code scans are enabled during. the CI/CD pipeline and is configured with an acceptable Sonar QualityGate status. Before running the sonar-scanner process, the plugin creates a sonar-scanner.properties file from a template. The properties file contains sonar-scanner configurations including the Sonar server host, sources, inclusions, exclusions, encoding, branch, etc… After the sonar-scan process is finished successfully the plugin will be waiting for the results of QualitiyGate and it breaks the build if is not passing it.

Build process failed due to Sonar Quality profile

An example CI/CD yaml:

workspace:
  base: /go
  path: src/github.com/example/app

pipeline:
  build:
    image: golang:1.9
    environment:
      - CGO_ENABLED=0
    commands:
      - go test -cover -coverprofile=coverage.out
      - go build -ldflags "-s -w -X main.revision=$(git rev-parse HEAD)" -a

  static_scan:
    image: registry/sonar-scanner
    quality: OK
    secrets: [ sonar_host, sonar_token ]

The Sonar server is running on the Pipeline control plane and it’s backed by an isolated database per organization. Every organization has it’s own user/password pair which is stored in Vault and managed with Pipeline. For further information of how we mange secrets on k8s please follow this post - Kubernetes secret management with Pipeline.

This is configurable per organization, the server and database can run inside the Kubernetes cluster as well

Container vulnerability checks - Clair

Pipeline is using Clair for vulnerability checks. The isolation of scans are per organization (or department) and it is running inside a separate namespace. We do not store the layers but only the hashes and the vulnerabilities associated to layers.

By default Pipeline is non intrusive - beside raising alerts - in a sense that does not break the builds, or remove deployments, however rules can be configured to trigger actions.

Vulnerability scans are enabled during the CI/CD pipeline with configs for images, severity and threshold. Multiple scans are allowed as during an image build we can use more then one image. For example if we are about to create an application written in Golang, first we’ll need to build it inside a container which contain all the necessary packages to compile but for running we won’t need to use whole build environment.

An example CI/CD yaml:

workspace:
  base: /go
  path: src/github.com/example/app

pipeline:
  build:
    image: golang:1.9
    environment:
      - CGO_ENABLED=0
    commands:
      - go test -cover -coverprofile=coverage.out
      - go build -ldflags "-s -w -X main.revision=$(git rev-parse HEAD)" -a

  scan_build_image:
    image: registry/drone-clair-scanner
    scan_image: golang:1.9
    severity: High
    treshold: 20
    secrets: [ docker_username, docker_password ]

  scan_base_image:
    image: registry/drone-clair-scanner
    scan_image: alpine:3.7
    severity: Medium
    treshold: 1
    secrets: [ docker_username, docker_password ]

  publish:
    image: plugins/docker
    repo: registry/app-docker-image
    tags: [ 1.3 ]
    secrets: [ docker_username, docker_password ]

  scan_created_image:
    image: registry/drone-clair-scanner
    scan_image: registry/app-docker-image
    severity: Medium
    treshold: 1
    secrets: [ docker_username, docker_password ]

When the configured treshold is exceeded, the build process is failed.

Build process failed due to vulnerability treshold

The Clair server running on the Pipeline control plane, so it’s crucial to isolate organization scans at database level. Same as for Sonar, every organization has it own database and Vault secret. When a vulnerability scan is initiated the control plane starts a Clair pod using the org’s own database.

Secure CI/CD flow in Pipeline

If a new organization is created, the control plane will start a new init pod within Pipeline K8s which connects to the DB service and creates new database owned by organization. This is configurable per organization, the server and database can run inside the Kubernetes cluster as well.

We take our users security seriously. If you believe that you have found a security vulnerability please contact us at security@banzaicloud.com. Thank you.

If you’d like to learn more about Banzai Cloud and our approach towards securing Kubernetes deployments check out our other posts in the blog, the Pipeline project on Github or follow us on our social channels.

Star



Comments

comments powered by Disqus