Recently, we blogged about certificate management on Kubernetes. Today, we’ll be returning to that topic, but we’ll be focusing on the differences an Istio service mesh makes.
The primary difference is the method of solving the ACME
HTTP-01 challenge. Solving this challenge involves routing an HTTP request from the ACME server (the Certificate Authority) to the cert-manager challenge solver pod.
Cert management with Istio
Ingress support 🔗︎
As we saw in our previous blog post, you can route such a challenge request by using a Kubernetes Ingress gateway. When using Istio, that same method is still viable, provided that Kubernetes Ingress support is enabled in your Istio config. If it’s enabled, and you’re using Kubernetes
Ingress resources for routing the traffic entering your cluster, everything should work just fine. However, you will be limited to using one ingress gateway for all your ingress traffic. This can be a security issue, and it also prevents you from setting up separate ingress gateways for your different teams or products. Additionally, this will cause you to miss out on some Istio features.
Considering these limitations, it usually makes sense to use native Istio routing.
Cert management using Istio 🔗︎
Using native Istio features for ingress gives you more freedom and flexibility:
- you can direct traffic for different domains to separate ingress gateways
- you can set up routing for HTTP and HTTPS traffic on non-standard ports
- you also get to use the familiar Istio features (circuit breaker, metrics, etc.)
While technically it’s possible to use Kubernetes
Ingress and Istio resources together to configure the same Istio ingress gateway, conflicts between them can easily cause difficult to diagnose problems; you should stick to using one or the other for a given ingress gateway.
All in all, if you have Istio running in your cluster, it’s a good idea to use it for your ingress gateways, too.
So, what does this mean in terms of cert management automation? Well, cert-manager doesn’t support Istio at the moment. There are solutions, though. When using an ACME issuer with challenge type
HTTP-01, you will end up with an
Ingress resource for routing the ACME challenge request. If you disabled Kubernetes
Ingress support in Istio (or you specified a non-existent ingress class in the
Issuer resource), then no ingress controller will react to the
Ingress resource, therefore, no HTTP request will be routed to the ACME solver pod(s).
What can we do? Two possible solutions: translate the
Ingress resource into
VirtualService resources, or add Istio support to cert-manager.
Obviously, adding Istio support to cert-manager would be the perfect solution, but it’s a more involved process. Nonetheless, at Banzai Cloud, we’re already working on just such an implementation. In fact, we have a functional solution, although it needs some more love before it’s production ready (cert-manager#3011).
Ingress resources 🔗︎
While we’re waiting for direct Istio support in cert-manager to be released, we can make do with another solution: replicating the
Ingress resource with Istio resources. It’s possible to do this by hand, but usually the reason we use the ACME protocol is automation, so introducing an error prone and repetitive manual step into that process sort of defeats the point.
When properly automated, this can serve as an effective solution, though. The good news is, if you’re a Backyards (now Cisco Service Mesh Manager) user, you can try it out right now, because Backyards 1.3 includes this feature. If not, then feel free to use the evaluation version!
Check out Backyards in action on your own clusters!
Try it out! 🔗︎
open the Backyards dashboard:
$ backyards dashboard
Gatewaysfrom the menu on the left
select the gateway you want secured. You can select whichever you like, but make sure the “Service Type” is “LoadBalancer”. This selection determines the load balancer and, therefore, the IP address(es) that will be used for the ACME
HTTP-01challenge. In our case, it’s
point your domain name to the IP address or DNS name found in the
Ports & Hostssection, click on
create newin the upper right corner
fill in the fields:
you will probably want HTTPS protocol on the standard port
enter your domain name(s) for
hosts. You can enter several by hitting enter after each one.
You can also set up a domain under
.banzaicloud.iofor testing purposes if you don’t want to use your own domain name. Just click on the checkbox to the right of
use .banzaicloud.ioand a new domain name will be generated for you, then add a subdomain under the generated domain name.
click on the checkbox
Use Let's Encrypt for TLSto get a certificate for your domain from Let’s Encrypt
enter your email address. This address will be for ACME account management, so will be sent to Let’s Encrypt.
As you can see, two more items have appeared in the
Ports & Hosts list, both for the host
frontpage.qahqab.backyards.banzaicloud.io. One of them on port 443, and the other on port 80. The latter one is for solving the ACME
HTTP-01 challenge created by translating the
While the certificate is being acquired, you can look up and see the contents of related
Ingressresource created by cert-manager:
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: istio labels: acme.cert-manager.io/http-domain: "1270878147" acme.cert-manager.io/http-token: "1101532839" acme.cert-manager.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2 namespace: istio-system spec: rules: - host: frontpage.qahqab.backyards.banzaicloud.io http: paths: - backend: serviceName: cm-acme-http-solver-rrg8c servicePort: 8089 path: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s
VirtualServicecreated by Backyards from the above
apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: labels: acme.backyards.banzaicloud.io/http-domain: "1270878147" acme.backyards.banzaicloud.io/http-token: "1101532839" acme.backyards.banzaicloud.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2-czvnv namespace: istio-system spec: selector: app: istio-ingressgateway istio: ingressgateway servers: - hosts: - istio-system/frontpage.qahqab.backyards.banzaicloud.io port: name: http number: 80 protocol: HTTP
apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: labels: acme.backyards.banzaicloud.io/http-domain: "1270878147" acme.backyards.banzaicloud.io/http-token: "1101532839" acme.backyards.banzaicloud.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2-tv22w namespace: istio-system spec: exportTo: - '*' gateways: - istio-system/cm-acme-http-solver-nnpg2-czvnv hosts: - frontpage.qahqab.backyards.banzaicloud.io http: - match: - uri: exact: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s route: - destination: host: cm-acme-http-solver-rrg8c.istio-system.svc.cluster.local port: number: 8089
As you can see, these two resources describe the same routing path as the
After a short while the item with port 80 and protocol HTTP will disappear, and a green checkmark will appear next to HTTPS. This signals that the certificate was issued and is being used for securing your domain:
Let’s see if that’s really the case:
$ curl -v https://frontpage.qahqab.backyards.banzaicloud.io ... * Server certificate: * subject: CN=frontpage.qahqab.backyards.banzaicloud.io * start date: Jul 14 10:42:43 2020 GMT * expire date: Oct 12 10:42:43 2020 GMT * subjectAltName: host "frontpage.qahqab.backyards.banzaicloud.io" matched cert's "frontpage.qahqab.backyards.banzaicloud.io" * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok. ... < HTTP/2 404 < date: Tue, 14 Jul 2020 11:44:09 GMT < server: istio-envoy
As you can see, the connection is secured with a certificate from Let’s Encrypt. Did you notice that 404, though? That’s because we haven’t set up any routing yet. We can do that now, so head over to the
routes tab, click
create new in the upper right corner and fill in the fields. At the very least, the gateway, host and port number need to be filled in, but, depending on your requirements, you might want to specify other values and settings, too.
When you’re ready, click
Create. Now that routing is set up for our demo service, let’s see if it works:
$ curl https://frontpage.qahqab.backyards.banzaicloud.io frontpage
You can also try it in your browser to inspect the certificate more easily.
Wrap up 🔗︎
As we have seen in this blog post, while cert-manager doesn’t directly support Istio, it’s possible to make it work. And with Backyards, it’s not only possible but seamless, too.