Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact
Author Szabolcs Berecz

Certificate management on Kubernetes

When exposing services it’s generally a good idea to follow the industry standard and use HTTPS protocol. HTTPS requires a certificate issued by a trusted third party, called a Certificate Authority (or CA for short).

There are several ways to acquire one, but a simple and effective method is to use Let’s Encrypt (a CA) by way of the ACME protocol. The ACME protocol is a communication protocol for interacting with CAs that makes it possible to automate the request and issuance of certificates. The idea is that manual certificate management can easily result in expired certificates, which usually translate to a non-working website and/or services. For this reason, you should be able to configure your infrastructure once, and let it handle certificate renewals automatically. To ensure that you automate this process, certificates issued by Let’s Encrypt are valid for only 90 days, since it’s widely believed that 90 days is frequent enough for users to automate the handling of renewals .

The ACME protocol 🔗︎

What you need to know about the ACME protocol is that it involves proving that you control the domains present in the Certificate Signing Request (CSR). This is done by solving challenges (one for each domain). You can usually choose between several challenge types, which vary depending on the CA and the domains involved. The two most common challenge types are HTTP-01 and DNS-01.

HTTP-01 challenge 🔗︎

This challenge type is solved by replying to a specific HTTP request with an appropriate response. The request is a GET request to a url of the form http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> and the response is a combination of the token and your ACME account key.

This is probably the simplest challenge to automate, but it can’t be used for wildcard domains (for example *.example.com). For wildcard domains you have to solve the DNS-01 challenge.

DNS-01 challenge 🔗︎

Solving the DNS-01 challenge can be done by putting a specific value in a TXT record for the given domain. As the main idea behind the ACME protocol is automation, this challenge type only makes sense if your DNS provider has an API.

There are several ACME clients which can handle the submitting of CSRs as well as solving the required challenges. One such client is certbot which can handle “legacy” environments (Apache, Nginx, etc.). If you are running your services in a Kubernetes cluster, your best bet is to use cert-manager. Let’s take a look at how it works.

cert-manager 🔗︎

Cert-manager is the complete package when it comes to handling multiple certificate issuer types (ACME, self-signed, CA among others). It can acquire and automatically renew certificates before expiry. If you are using Kubernetes Ingress to route your ingress traffic, cert-manager can automatically solve HTTP-01 challenges. It does this by spinning up a pod for each challenge, then applying the necessary routing changes to your Ingress config.

When you interact with cert-manager you’ll be using a couple of custom resource types: Issuer, ClusterIssuer, and Certificate. There are a few other custom resource types involved, which aren’t necessary to know about but which can be helpful in tracking down errors. These are the CertificateRequest, Order and Challenge custom resources.

Let’s see what each one is used for.

cert-manager

User facing custom resources 🔗︎

Issuer, ClusterIssuer resources 🔗︎

An issuer is an entity that can generate signed certificates. There are several supported issuers built into cert-manager, and it can be extended with new ones if necessary. An Issuer or ClusterIssuer resource describes one issuer entity. You will need at least one such resource in your cluster. We will be focusing on the ACME Issuer type. ACME is the protocol implemented by Let’s Encrypt.

The difference between Issuer and ClusterIssuer is that the Issuer's use is restricted to the namespace it’s created in; it’s not possible to reference it from a Certificate resource in another namespace. A ClusterIssuer, however, is global, usable from any Certificate resource in the cluster.

Certificate resource 🔗︎

A Certificate resource describes the intention to acquire a certificate for one or more of your domains. Each certificate must reference an issuer resource. This will be the issuer used for acquiring the desired certificate.

When cert-manager notices a new Certificate resource it proceeds with the creation of a CertificateRequest.

Internal custom resources 🔗︎

CertificateRequest resource 🔗︎

This resource represents one request for a certificate. It contains the certificate signing request, which is itself encoded in PEM format, and the certificate if it was already received, in which case Ready will be set to True.

The absence of this resource doesn’t mean the certificate hasn’t yet been requested. Cert-manager will not recreate this resource if the certificate has already been acquired.

Order resource 🔗︎

This resource type comes into play when a certificate is requested from an ACME issuer. An Order resource represents the order for a certificate to be issued.

In its status you can find the challenges to be solved. There can be several challenge types for a domain, but only one of those need be solved. Cert-manager will select one challenge type for each of the domains, then proceed with the creation of Challenge resources.

Challenge resource 🔗︎

Not unlike the Order resource type, this resource type plays a role when the certificate is requested from an ACME issuer. These resources describe the challenges cert-manager selected to solve.

Now that we’ve covered the parts of cert-manager, let’s see it in action!

Try it out! 🔗︎

You will need a kubernetes cluster and domain to play with.

Create cluster 🔗︎

  1. Create a Kubernetes cluster.

    If you need a hand with that, you can create a cluster with the Banzai Clouds Pipeline platform on five different clouds or on-premise. Pipeline is available online, for free at Try Pipeline.

  2. Point KUBECONFIG at your cluster.

  3. Make sure you have an Ingress controller in your cluster. If you used Banzai Cloud’s Pipeline platform to create the cluster, you already have one along with a LoadBalancer-type Service:

    $ kubectl get -n pipeline-system deploy/ingress-traefik svc/ingress-traefik
      NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
      deployment.apps/ingress-traefik   1/1     1            1           3m19s
    
      NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)                      AGE
      service/ingress-traefik   LoadBalancer   10.10.231.18   a015671f691794291bd991eb635ed907-1982155256.us-east-2.elb.amazonaws.com   443:32271/TCP,80:30543/TCP   3m20s
    

    The service should have port 80 and 443 open. Port 80 is for the ACME HTTP-01 challenge, and port 443 is for HTTPS traffic.

  4. Install cert-manager

    It can be as simple as:

    kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml
    

    There are other options and more details at https://cert-manager.io/docs/installation/kubernetes/

  5. Point your domain to the external IP of your LoadBalancer typed Service on which you want external traffic to enter your cluster. “External IP” might be a DNS name (as it is in the example above), in which case you will have to create a CNAME record instead of an A record.

    Set the DOMAIN environment variable to your domain (example.com will not work):

    $ DOMAIN=example.com
    
  6. Deploy httpbin for testing purposes

    $ kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Service
    metadata:
      name: httpbin
      namespace: default
    spec:
      selector:
        app: httpbin
      ports:
        - port: 8080
          protocol: TCP
          targetPort: 80
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: httpbin
      namespace: default
      labels:
        app: httpbin
    spec:
      containers:
        - image: kennethreitz/httpbin:latest
          name: httpbin
          ports:
            - containerPort: 80
              protocol: TCP
    EOF
    

    You can check if it’s up and running through port-forwarding and curl:

    $ kubectl port-forward svc/httpbin 8080 &
    $ curl localhost:8080/get
    
  7. Create an issuer

    Note: we will be using the staging provider to avoid hitting rate limits.

    First, set the EMAIL environment variable to your email address. Let’s Encrypt will use this to contact you about expiring certificates and other issues related to your account.

    $ EMAIL=some@email.address
    

    Now, you’re ready to create the issuer for Let’s Encrypt:

    $ kubectl apply -f - <<EOF
    apiVersion: cert-manager.io/v1alpha2
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-staging
    spec:
      acme:
        email: ${EMAIL}
        server: https://acme-staging-v02.api.letsencrypt.org/directory
        privateKeySecretRef:
          # Secret resource that will be used to store the account's private key.
          name: example-issuer-account-key
        solvers:
          - http01:
              ingress:
                name: test-ingress
    EOF
    
  8. Create an ingress

    $ kubectl apply -f - <<EOF
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: test-ingress
      namespace: default
    spec:
      backend:
        serviceName: httpbin
        servicePort: 8080
      tls:
        - hosts:
            - ${DOMAIN}
          secretName: test-ingress-cert
    EOF
    

    This ingress is not functional yet, because the secret being referenced does not exist.

    If you remove the tls config, the service should be reachable on HTTP protocol:

    $ curl ${DOMAIN}/get
    

    If it’s not working, try the external IP of your ingress service to see if the problem is in your cluster or with the DNS resolution. If it works with the external IP, you might need to wait a couple of minutes for the DNS changes to propagate.

  9. Acquire a certificate

    As I have mentioned, we have a non-working ingress because of the missing secret. We have a couple options to acquire a certificate and make it work:

    1. Create the ingress without tls config. This way, the ingress can serve HTTP traffic to complete the ACME HTTP-01 challenge. After the certificate is issued, the tls config can be added back in.
    2. Create a self-signed certificate initially, and then acquire a properly signed one:
      1. Create a self-signed Issuer and use it in your Certificate resource
      2. Wait until the certificate is ready (check it using curl --insecure https://${DOMAIN}/get)
      3. Edit your Certificate resource and change the issuer to the Let’s Encrypt Issuer
      4. The self-signed certificate will then be replaced by a newly acquired, properly signed certificate
    3. The easiest solution: annotate the Certificate resource to instruct cert-manager to create a self signed certificate before attempting to request the properly signed one. This is basically the same as method 2, but automated.

    Let’s go with method 3 and annotate the Certificate resource with cert-manager.io/issue-temporary-certificate: "true".

    $ kubectl apply -f - <<EOF
    apiVersion: cert-manager.io/v1alpha2
    kind: Certificate
    metadata:
      name: test-ingress
      namespace: default
      labels:
        cert: test-ingress
      annotations:
        cert-manager.io/issue-temporary-certificate: "true"
    spec:
      secretName: test-ingress-cert
      dnsNames:
        - ${DOMAIN}
      issuerRef:
        name: letsencrypt-staging
        kind: ClusterIssuer
    EOF
    

    If everything is set up correctly, you should receive a certificate in a few minutes. You can check its progress by inspecting the Certificate, CertificateRequest, Order and Challenge resources. Take a look at the status and events sections.

    $ kubectl describe certificate -l cert=test-ingress
    $ kubectl describe certificaterequest -l cert=test-ingress
    $ kubectl describe order -l cert=test-ingress
    $ kubectl describe challenge
    

    Note: as of cert-manager v0.15.1, Challenge resources don’t inherit the labels the way other resources do, so it’s not as easy to select the challenges that correspond to your Certificate resource.

    Try adding one or more entries (for example foo.) to dnsNames in your Certificate resource, which you have not pointed to your ingress. Challenges for these domains won’t be solvable, so you’ll have time to inspect the Challenge resources. You can also see the changes it made to your Ingress resource.

    The received certificate will be placed in the secret with key tls.crt and the corresponding private key with key tls.key. You can check it with the following command:

    $ kubectl describe secret test-ingress-cert
    

    Now we are ready to see the issued certificate in action (the --insecure flag is necessary when using the staging provider, which we did):

    $ curl -v --insecure https://${DOMAIN}/get
    ...
    * Server certificate:
    *  subject: CN=<your-domain>
    *  start date: Jul  3 19:57:58 2020 GMT
    *  expire date: Oct  1 19:57:58 2020 GMT
    *  issuer: CN=Fake LE Intermediate X1
    ...
    

    Keeping cert-manager and your Certificate resources in your cluster will ensure that all your certificates are renewed before expiry.

Wrap-up 🔗︎

It’s important to secure your services with TLS; cert-manager makes it easy to do that in a Kubernetes environment. You should use it! If you are interested in certificate management on Istio, check out our next post, Certificate management on Istio as well.

Never miss a post again!

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn, or Twitter, or get in touch on Slack: