How the webhook works - overview 🔗︎
Kubernetes secrets are the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption eventually ends up as a Kubernetes secret. However, despite their name, Kubernetes secrets are not exactly secure, since they are only base64 encoded.
The mutating webhook of Bank-Vaults is a solution that bypasses the Kubernetes secrets mechanism and injects the secrets retrieved from Vault directly into the Pods. Specifically, the mutating admission webhook injects (in a very non-intrusive way) an executable into containers of Deployments and StatefulSets. This executable can request secrets from Vault through special environment variable definitions.
An important and unique aspect of the webhook is that it is a daemonless solution (although if you need it, you can deploy the webhook in daemon mode as well).
Why is this more secure than using Kubernetes secrets or any other custom sidecar container? 🔗︎
Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk or in etcd - not even temporarily. All secrets are stored in memory, and are only visible to the process that requested them. Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.
If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.
The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.
env: - name: AWS_SECRET_ACCESS_KEY value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY # or - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: aws-key-secret key: AWS_SECRET_ACCESS_KEY # or - name: AWS_SECRET_ACCESS_KEY valueFrom: configMapKeyRef: name: aws-key-configmap key: AWS_SECRET_ACCESS_KEY
The webhook checks if a container has envFrom and parses the defined ConfigMaps and Secrets:
envFrom: - secretRef: name: aws-key-secret # or - configMapRef: name: aws-key-configmap
Secret and ConfigMap examples 🔗︎
Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them.
The secret value should contain a base64 encoded template string referencing the vault path you want to insert.
echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.
apiVersion: v1 kind: Secret metadata: name: aws-key-secret data: AWS_SECRET_ACCESS_KEY: dmF1bHQ6c2VjcmV0L2RhdGEvYWNjb3VudHMvYXdzI0FXU19TRUNSRVRfQUNDRVNTX0tFWQ== type: Opaque
apiVersion: v1 kind: ConfigMap metadata: name: aws-key-configmap data: AWS_SECRET_ACCESSKEY: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
For further examples and use cases, see Configuration examples and scenarios.
Deploy the mutating webhook 🔗︎
You can deploy the Vault Secrets Webhook using Helm. Note that:
- The Helm chart of the vault-secrets-webhook contains the templates of the required permissions as well.
- The deployed RBAC objects contain the necessary permissions fo running the webhook.
- The user you use for deploying the chart to the Kubernetes cluster must have cluster-admin privileges.
- The chart requires Helm 3.
- To interact with Vault (for example, for testing), the vault command line client must be installed on your computer.
- You have deployed Vault with the operator and configured your Vault client to access it, as described in Deploy a local Vault operator.
Deploy the webhook 🔗︎
Create a namespace for the webhook and add a label to the namespace, for example, vault-infra:
kubectl create namespace vault-infra kubectl label namespace vault-infra name=vault-infra
Deploy the vault-secrets-webhook chart:
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook
For further details, see the webhook’s Helm chart repository.
Check that the pods are running:
kubectl get pods --namespace vault-infra NAME READY STATUS RESTARTS AGE vault-secrets-webhook-58b97c8d6d-qfx8c 1/1 Running 0 22s vault-secrets-webhook-58b97c8d6d-rthgd 1/1 Running 0 22s
Write a secret into Vault (the Vault CLI must be installed on your computer):
$ vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t Key Value --- ----- created_time 2020-11-04T11:39:01.863988395Z deletion_time n/a destroyed false version 1
Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:
apiVersion: apps/v1 kind: Deployment metadata: name: vault-test spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: vault template: metadata: labels: app.kubernetes.io/name: vault annotations: vault.security.banzaicloud.io/vault-addr: "https://vault:8200" # optional, the address of the Vault service, default values is https://vault:8200 vault.security.banzaicloud.io/vault-role: "default" # optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default" vault.security.banzaicloud.io/vault-skip-verify: "false" # optional, skip TLS verification of the Vault server certificate vault.security.banzaicloud.io/vault-tls-secret: "vault-tls" # optional, the name of the Secret where the Vault CA cert is, if not defined it is not mounted vault.security.banzaicloud.io/vault-agent: "false" # optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authentication vault.security.banzaicloud.io/vault-path: "kubernetes" # optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes" spec: serviceAccountName: default containers: - name: alpine image: alpine command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"] env: - name: AWS_SECRET_ACCESS_KEY value: vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
Check the mutated deployment.
kubectl describe deployment vault-test
The output should look similar to the following:
Name: vault-test Namespace: default CreationTimestamp: Wed, 04 Nov 2020 12:44:18 +0100 Labels: <none> Annotations: deployment.kubernetes.io/revision: 1 Selector: app.kubernetes.io/name=vault Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app.kubernetes.io/name=vault Annotations: vault.security.banzaicloud.io/vault-addr: https://vault:8200 vault.security.banzaicloud.io/vault-agent: false vault.security.banzaicloud.io/vault-path: kubernetes vault.security.banzaicloud.io/vault-role: default vault.security.banzaicloud.io/vault-skip-verify: false vault.security.banzaicloud.io/vault-tls-secret: vault-tls Service Account: default Containers: alpine: Image: alpine Port: <none> Host Port: <none> Command: sh -c echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000 Environment: AWS_SECRET_ACCESS_KEY: vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: vault-test-55c569f9 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 29s deployment-controller Scaled up replica set vault-test-55c569f9 to 1
As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.
Deploy the webhook from a private registry 🔗︎
If you are getting the x509: certificate signed by unknown authority app=vault-secrets-webhook error when the webhook is trying to download the manifest from a private image registry, you can:
- Build a docker image where the CA store of the OS layer of the image contains the CA certificate of the registry.
- Alternatively, you can disable certificate verification for the registry by using the REGISTRY_SKIP_VERIFY="true” environment variable in the deployment of the webhook.
Deploy in daemon mode 🔗︎
vault-env by default replaces itself with the original process of the Pod after reading the secrets from Vault, but with the
vault.security.banzaicloud.io/vault-env-daemon: "true" annotation this behavior can be changed. So
vault-env can change to
daemon mode, so
vault-env starts the original process as a child process and remains in memory, and renews the lease of the requested Vault token and of the dynamic secrets (if requested any) until their final expiration time.
You can find a full example using MySQL dynamic secrets in the Bank-Vaults project repository:
# Deploy MySQL first as the Vault storage backend and our application will request dynamic secrets for this database as well: helm upgrade --install mysql stable/mysql --set mysqlRootPassword=your-root-password --set mysqlDatabase=vault --set mysqlUser=vault --set mysqlPassword=secret --set 'initializationFiles.app-db\.sql=CREATE DATABASE IF NOT EXISTS app;' # Deploy the vault-operator and the vault-secerts-webhook kubectl create namespace vault-infra kubectl label namespace vault-infra name=vault-infra helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook # Create a Vault instance with MySQL storage and a configured dynamic database secrets backend kubectl apply -f operator/deploy/rbac.yaml kubectl apply -f operator/deploy/cr-mysql-ha.yaml # Deploy the example application requesting dynamic database credentials from the above Vault instance kubectl apply -f deploy/test-dynamic-env-vars.yaml kubectl logs -f deployment/hello-secrets