Banzai Cloud Logo Close
Home Benefits Blog Company Contact
Sign in
Author Nandor Kracser

Dynamic SSH with Vault and Pipeline

TRY PIPELINE FOR FREE

At Banzai Cloud, we’re building a feature rich platform as a service on Kubernetes, called Pipeline. With Pipeline, we provision large, multi-tenant Kubernetes clusters on all major cloud providers, such as AWS, GCP, Azure and BYOC, and deploy all kinds of predefined or ad-hoc workloads to these clusters. When we needed a way for our users to login and interact with protected endpoints and, at the same time, provide dynamic secrets management support, while simultaneously providing native Kubernetes support for all our applications, we turned to Vault.

Security series:
Authentication and authorization of Pipeline users with OAuth2 and Vault
Dynamic credentials with Vault using Kubernetes Service Accounts
Dynamic SSH with Vault and Pipeline
Secure Kubernetes Deployments with Vault and Pipeline
Policy enforcement on K8s with Pipeline
The Vault swiss-army knife
The Banzai Cloud Vault Operator
Vault unseal flow with KMS
Kubernetes secret management with Pipeline
Container vulnerability scans with Pipeline
Kubernetes API proxy with Pipeline

In this blog post we’ll explore one of the many workflows we support: authentication via GitHub using OAuth2 tokens, organization mapping to a Vault team, and seamless SSH in cloud instances belonging to that organization. The cloud instances/VMs become trusted by connecting to Vault via AppRole and signing host keys. This way we neither have to exchange keys with our end users, nor concern ourselves with manually managing or revoking them, all of which is automated by Pipeline and secured by Vault.

Vault SSH Login flow

All of the following operations are automated by Pipeline, thus, for clusters/control plains launched by Pipeline, these steps may not necessarily need to be taken.

Dynamic SSH credentials

Once we provision a Kubernetes cluster in the cloud or on-prem, our customers typically need to access the hosts, so we need a very dynamic method of distribution access. Static SSH keys that access these clusters (especially so that they can’t be dynamically revoked) are not an option for us and our customers. For this purpose, we’ve decided to use Vault’s SSH Secret backend, which does dynamic Client Key Signing and Host Key Signing for accessing our remote VMs.

Requirements:

  • A working Vault server accessible inside your infrastructure (automated by Pipeline)
  • A GitHub organization (can be GitHub Enterprise as well, but this demo uses the public version)
  • A VM (the examples below are based on Ubuntu)
  • An SSH keypair on your machine

Client Key Signing

With this method a user signs (requests a certificate for) his/her SSH public key with a CA key which is stored in Vault. The target machine has to be configured to trust users signed with this key. This is automated by Pipeline as well.

Administrative steps

First and foremost you have to log in as an administrator into Vault:

# Point your Vault CLI to the Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

vault login

Let’s configure a policy for the users who can SSH into the developer VMs. Configure the dev policy like this:

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }'

Configure the GitHub Auth method for the users. Everyone who logs in through the GitHub Auth method and is part of the dev team in the specified GitHub organization will be mapped to the dev policy in Vault:

vault auth enable github
vault write auth/github/config organization=myorganizationongithub
vault write auth/github/map/teams/dev value=dev

Enable the ssh secret engine for client key signing on the ssh-client-signer path (can be something else, but make it consistent):

vault secrets enable -path=ssh-client-signer ssh
vault write ssh-client-signer/config/ca generate_signing_key=true
# This policy makes the CA signings valid for 24h, and is acceptable for daily workflow
vault write ssh-client-signer/roles/dev -<<EOH
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "default_extensions": [
    {
      "permit-pty": ""
    }
  ],
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "24h"
}
EOH

Download the public key from Vault onto the machine you would like to SSH into:

curl https://${VAULT_ADDR}/v1/ssh-client-signer/public_key > /etc/ssh/trusted-user-ca-keys.pem

Change /etc/ssh/sshd_config and add the following line so it will accept users with signed keys:

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Restart sshd to realize changes:

service ssh restart

User steps

Using the following steps, a user mapped to the dev policy can request Vault to sign his/her SSH public key with the trusted CA key:

# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...

vault login -method github $GITHUB_TOKEN

vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/id_rsa-cert.pub

Now you can log in to the VM:

ssh ubuntu@xy.xyz.xyz.xyz

Host Key Signing

How many times have you seen this message?

The authenticity of host 'xy.xyz.xyz.xyz (xy.xyz.xyz.xyz)' can't be established.
ECDSA key fingerprint is SHA256:wu5UbyRRuKw+JwCotA2QzQa7dsgs627pzwIYpXhprqA.
Are you sure you want to continue connecting (yes/no)?

That means you’re unable to confirm that the machine you’re SSHing into is a machine you can trust. With an additional configuration mechanism, and with the help of Vault, you can make sure those VMs are trusted from the get go. This reduces the probability of a user accidentally SSHing into an unmanaged or malicious machine. This is a method wherein the VMs have to authenticate themselves and sign their own keys with Vault. The VMs will use Vault’s AppRole Auth Method which is an auth method oriented to automated workflows (machines and services), and is less useful for human operators.

Vault SSH flow

Administrative steps

Enable the ssh secret engine for host key signing on the ssh-host-signer path (can be something else, but make it consistent):

vault secrets enable -path=ssh-host-signer ssh
vault write ssh-host-signer/config/ca generate_signing_key=true
vault secrets tune -max-lease-ttl=87600h ssh-host-signer

Configure the hostsigning policy and hostrole role:

vault write ssh-host-signer/roles/hostrole \
    key_type=ca \
    ttl=87600h \
    allow_host_certificates=true \
    allowed_domains="*" \
    allow_subdomains=true
vault write sys/policy/hostsigning policy='
    path "ssh-host-signer/sign/hostrole" {
        capabilities = ["update"]
    }'

Let’s update the dev policy with two additional paths. The two additional AppRole based paths are needed so developers can request AppRole leased temporary tokens, and pass them to a newly started VM via User Data, for example:

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }
    path "auth/approle/role/hostrole/role-id" {
        capabilities = ["read"]
    }
    path "auth/approle/role/hostrole/secret-id" {
        capabilities = ["update"]
    }'

Setup AppRole auth method so that hosts will be able to lease Vault tokens for host key (self) signing:

vault write auth/approle/role/hostrole policies=hostsigning \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40

Sign the key on the host (log in with AppRole auth method first):

# Get a token with Vault AppRole
VAULT_TOKEN=$(vault write -field token auth/approle/login role_id=${VAULT_ROLE_ID} secret_id=${VAULT_SECRET_ID})

# Log in with that token
vault login ${VAULT_TOKEN}

# Sign the key of this host
vault write -field=signed_key ssh-host-signer/sign/hostrole \
    cert_type=host \
    public_key=@/etc/ssh/ssh_host_rsa_key.pub > /etc/ssh/ssh_host_rsa_key-cert.pub

# Set permissions on the certificate to be 0640:
chmod 0640 /etc/ssh/ssh_host_rsa_key-cert.pub

Change /etc/ssh/sshd_config and add the host certificate path to it:

# For host keys
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

Restart SSH to realize the changes:

service ssh restart

User steps

When starting a machine:

# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...

vault login -method github $GITHUB_TOKEN

VAULT_ROLE_ID=$(vault read -field role_id auth/approle/role/hostrole/role-id)
VAULT_SECRET_ID=$(vault write -f -field secret_id auth/approle/role/hostrole/secret-id)
# Pass VAULT_ROLE_ID and VAULT_SECRET_ID to the newly started VM.

When logging into that machine:

# Setup your local ssh configuration to allow hosts signed by this CA (coming from Vault)
CERT_AUTHORITY=$(vault read -field=public_key ssh-host-signer/config/ca)
echo "@cert-authority * $CERT_AUTHORITY" >> ~/.ssh/known_hosts

# If you are using client-side signed keys, sign your key temporarily
vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/id_rsa-cert.pub

# SSH into target machines as usual
ssh ubuntu@xy.xyz.xyz.xyz

Learn by the code

We already start our Pipeline Control Plane instances with this method, which you can check out, here, in our GitHub repository.

As we’ve just demonstrated, Vault has a superb engine for Dynamic SSH credentials and Host Key signing, and, with simple configurations and some code, most security features required by enterprises can be quickly and effectively implemented. The whole workflow can be automated with the help of Vault’s dynamic nature and the help of AppRole. For simplicity’s sake we’ll end this post here. However, we’ll keep posting about how we seal and unseal Vault and continue to explore other advanced scenarios in our forthcoming posts. Meanwhile, please check out the open source code on our GitHub and make sure to go through the excellent tutorials and use cases made available by Vault.

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn or Twitter:

Star

TRY PIPELINE FOR FREE

Comments

comments powered by Disqus