Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact

Hardened FIPS compliant Istio with Backyards

Author Janos Matyas

Companies providing services for government, healthcare, or finance sectors must ensure that their products adhere to government approved security requirements that ensure the protection of sensitive data. The Federal Information Processing Standards (FIPS) outlines the rules for cryptographic modules employed as mechanism that implement security. A cryptographic module is a combination of hardware and/or software that handles cryptographic functions such as encryption, decryption, digital signatures, authentication techniques and random number generation.

The FIPS 140-2 standard provides four increasing, qualitative levels of security: Level 1, Level 2, Level 3, and Level 4. These levels are intended to cover a wide range of potential applications and environments into which cryptographic modules may be employed.

In this post we’ll discuss the approach we took to reaching FIPS 140-2 Security Level 1 compliance for our Istio distribution, Backyards.

Backyards logo

Note that there is a newer/updated FIPS version, FIPS 140-3. FIPS 140-3 testing will begin on September 22, 2020.

tl;dr πŸ”—︎

  • we are using a Go build where the standard crypto library is replaced with an FIPS compliant library (BoringCrypto)
  • we are building Envoy with the same FIPS compliant library (BoringCrypto)
  • Backyards delivers a custom Istio build, using the same FIPS compliant library (BoringCrypto)
  • Backyards includes lots of automations and convenience features. One is certificate management which is closely related to FIPS. Backyards now comes with a version of cert-manager built with the same FIPS compliant library (BoringCrypto)
  • as of now, the hardened FIPS compliant version of Backyards is tested with FIPS 140-2 compliant cipher suites (and rejects anything else)
  • although FIPS 140 allows other ciphers, we only have GCM ciphers enabled, because only those can prevent the SSL LUCKY13 timing attack

Flow

FIPS at Banzai Cloud πŸ”—︎

While, for now, we’re discussing how we built a FIPS 140-2 compliant Istio with Backyards, several of the steps we’ll be covering are shared by other products we are working to make FIPS compliant:

  • PKE, our CNCF certified Kubernetes distribution,
  • Pipeline, the container management platform for hybrid clouds (supporting 5 major cloud providers, bare metal and VMware),
  • Supertubes, production ready Kafka on Kubernetes,
  • Bank-Vaults, the Vault Swiss Army knife and secret injection webhook,
  • and One Eye, the ultimate observability tool for Kubernetes.

You might want to check out how we manage AWS GovCloud Kubernetes clusters with Pipeline

FIPS challenges πŸ”—︎

FIPS Security Level 1 mandates the usage of approved security algorithms or functions used by cryptographic modules for generating keys that are used to protect sensitive data. This means that any exchange of sensitive data between applications must occur over secure channels, using only FIPS validated cryptographic modules. For the list of FIPS validated cryptographic modules see Cryptographic Module Validation Program (using Google as a vendor example).

For a guide on how to read an FIPS Validation Listing, follow this link

In microservices architecture, as its name implies, multiple services (applications) are employed to implement scalable, flexible, fault-tolerant and secure products that serve complex business requirements. The components that such a system comprise are often written in different programming languages using different technologies. Ensuring that each of these components use an FIPS validated cryptographic module can be easily complicated:

  • there might be no FIPS validated library (cryptographic module) available that is compatible with the component
  • third party components may not allow replacing the used cryptography with a FIPS validated one
  • each component may require its own custom solution which increases the maintenance burden

FIPS 140-2 encryption using Istio πŸ”—︎

If we take a look at a typical microservice deployment running on our Istio distribution, Backyards, we can see that the above mentioned challenges apply to it as well.

Also, in the Istio community there is a longstanding feature request of using FIPS 140-2 compliant versions of Istio (which has resulted in lots of discussion on forums, internal WG or Slack). It is our pleasure, then, to announce that the wait is over: Backyards and all the components that ship as part of the Banzai Cloud Istio distribution are now built with FIPS compliant crypto modules.

In a nutshell πŸ”—︎

So what steps did we take? What details make Istio FIPS compliant? As discussed in the tl;dr: section:

  • we are using a Go build where the standard crypto library is replaced with an FIPS compliant library (BoringCrypto)
  • we are building Envoy with the same FIPS compliant library (BoringCrypto)
  • Backyards delivers a custom Istio build, using the same FIPS compliant library (BoringCrypto)
  • Backyards includes lots of automations and convenience features, including certificate management. Backyards now comes with a version of cert-manager built with the same FIPS compliant library (BoringCrypto)
  • the hardened FIPS compliant version of Backyards is now tested with FIPS 140-2 compliant cipher suites (and rejects anything else)
  • although FIPS 140 allows for other ciphers, we only have GCM ciphers enabled, since only they can prevent an SSL LUCKY13 timing attack

Note: as FIPS introduces lots of restrictions on the accepted cipher suites and can introduce cryptographic incompatibilities, the default version of Backyards is still built with the standard SSL libraries. If you’d like to use an FIPS compliant version of Istio/Backyards, get in touch with us.

Flow2

Under the hood πŸ”—︎

The bottom line for FIPS compliance from the application standpoint is that it must use FIPS verified crypto libraries.

Go-based applications πŸ”—︎

Unfortunately, the native crypto in Go is not FIPS validated, so we need a way to use a validated crypto library with Go. Luckily, it’s Google’s BoringSSL to the rescue.

BoringSSL is a fork of OpenSSL that is designed to meet Google’s needs.

BoringSSL as a whole is not FIPS validated. However, there is a core library (called BoringCrypto) that has been FIPS validated. Google also maintains a Go fork that can use BoringCrypto for cryptographic functions.

The distribution name for a Go+BoringCrypto release has the form b, where is the Go version the release is based on, and is an integer that increments each time there is a new release with different BoringCrypto bits.

You can use the Go+BoringCrypto distribution as a drop-in replacement, no code changes usually needed for simple compilation.

Indeed, it takes more than just a plain re-compile to get our application to make use of the BoringCrypto. The application has to be re-compiled with specific build configuration:

  • The build must be GOOS=linux, GOARCH=amd64.
  • The build must have cgo enabled.
  • The android build tag must not be specified.
  • The cmd_go_bootstrap build tag must not be specified.

To check whether a given binary is using BoringCrypto, run go tool nm on it and check that it has symbols named *_Cfunc__goboringcrypto_*. The program rsc.io/goversion will report the crypto implementation used by a given binary when invoked with the -crypto flag.

Furthermore, we made changes in the Istio codebase to make sure that FIPS is enforced properly in each and every binary. Our build has a more strict TLS configuration than FIPS 140-2 requires, and places similar limitations on TLS versions and used ciphers. This required additional changes in the codebase, but it further enhanced application security.

Compare the built binaries

The compiled binaries can be verified by the aforementioned goversion tool.

If we check the upstream binary of Istiod, for example, we’ll find that it uses the standard non-FIPS compliant standard Go crypto:

❯ goversion ./pilot-discovery
pilot-discovery go1.14.2 (standard crypto)

While doing the same check with the Backyards FIPS 140-2 compliant version, we can see that it uses the validated BoringCrypto library:

❯ goversion ./pilot-discovery
pilot-discovery go1.14.6b4 (boring crypto) +crypto/tls/fipsonly

Every Istio component the FIPS compliant Backyards distribution deploys uses custom built binaries that are using FIPS 140-2 validated BoringCrypto library.

Envoy πŸ”—︎

Istio uses Envoy in its data-plane, so the next step is to add an FIPS compliant version of it as well. Envoy uses BoringSSL already and it can be built in an FIPS 140-2 compliant mode by using the --define boringssl=fips build option.

The correctness of the resulting FIPS build can be verified by checking the presence of BoringSSL-FIPS in the --version output.

Besides the FIPS compliant build of Envoy, we’ve also made changes in Istio to ensure that the generated Envoy configuration distributed through XDS has strict and compliant TLS settings.

Compare the Envoy builds

The upstream Istio proxy image contains a non-FIPS 140-2 compliant Envoy build:

~ ❯ docker run -ti --rm --entrypoint envoy istio/proxyv2:1.6.7 --version

envoy  version: ccae1bd37085ecd78415dc06b50233b3b97e30c0/1.14.4/Clean/RELEASE/BoringSSL

While the FIPS 140-2 compliant Backyards Istio proxy image contains a custom built compliant version of Envoy:

~ ❯ docker run -ti --rm --entrypoint envoy banzaicloud/istio-proxyv2:1.6.7-bzc-fips --version

envoy  version: 136ca7b03ff5821697d68b77e7411dbc6bfdd5ef/1.14.4/Clean/RELEASE/BoringSSL-FIPS

cert-manager πŸ”—︎

Backyards installs cert-manager and relies on it for various certificate management related tasks (webhook and ingress certs for example). This was another stumbling block on our path to FIPS 14 0-2 compliance. We used the same tooling, configuration and code base changes as with Istio’s components.

If you are interested in cert-manager on Kubernetes, we recently blogged about it, specifically about cert management on Istio as well.

Compare the built cert-manager binaries

If we check the upstream version of the cert manager controller we’ll find that it is uses standard Go crypto:

❯ goversion -crypto ./controller
./controller go1.14.2 (standard crypto)

While the FIPS 140-2 compliant Backyards version uses BoringCrpto with strict FIPS settings:

❯ goversion ./controller
./controller go1.14.6b4 (boring crypto) +crypto/tls/fipsonly

Showtime πŸ”—︎

We are going to use two Kubernetes clusters that were created with Banzai Cloud Pipeline. One cluster has an Istio mesh installed and configured using our FIPS 140-2 compliant Backyards version, while the other cluster has a non-compliant mesh installed, which will allow us to demonstrate the differences between the two.

Backyards contains a microservice-based demo application which is also installed on both clusters and will be used in the following steps. We are going to use the frontpage service which serves requests on port 8080. The Istio mesh is configured with strict auto mTLS on both clusters.

Since the most visible aspects of FIPS 140-2 compliance are the TLS settings, the last item in the demo toolbox today is a handy TLS/SSL testing tool called testssl.sh.

Deploy testssl.sh pod on both clusters πŸ”—︎

We deployed the testssl pods on each cluster into the default namespace using the following snippet:

❯ kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: testssl
  namespace: default
spec:
  containers:
  - name: app
    image: drwetter/testssl.sh
    command: [ "/bin/sh", "-c", "--" ]
    args: [ "while true; do sleep 3000; done;" ]
EOF

Check the frontpage service endpoint πŸ”—︎

We can access detailed information about a TLS endpoint by running testssl.sh.

The following is the output of such a check against the frontpage service on the cluster that uses the non-FIPS compliant build. The workload runs inside an Istio mesh, thus the TLS here is provided by Istio sidecar Envoy proxy.

non FIPS compliant cluster ❯ kubectl exec -ti testssl -- testssl.sh https://frontpage.backyards-demo.svc.cluster.local.:8080
 Start 2020-08-06 13:46:47        -->> 10.11.241.177:8080 (frontpage.backyards-demo.svc.cluster.local.) <<--

 rDNS (10.11.241.177):   frontpage.backyards-demo.svc.cluster.local.
 Service detected:       certificate-based authentication => skipping all HTTP checks


 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2 (offered)

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)


 Testing server's cipher preferences

 Has server cipher order?     yes (OK) -- only for < TLS 1.3
 Negotiated protocol          TLSv1.3
 Negotiated cipher            TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Cipher per protocol

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 -
SSLv3
 -
TLSv1 (server order)
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128
 x2f     AES128-SHA                        RSA        AES         128
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256
 x35     AES256-SHA                        RSA        AES         256
TLSv1.1 (server order)
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128
 x2f     AES128-SHA                        RSA        AES         128
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256
 x35     AES256-SHA                        RSA        AES         256
TLSv1.2 (server order)
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 253   ChaCha20    256
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 253   AESGCM      128
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128
 x2f     AES128-SHA                        RSA        AES         128
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 253   AESGCM      256
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256
 x35     AES256-SHA                        RSA        AES         256
TLSv1.3 (no server order, thus listed by strength)
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128


 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

 FS is offered (OK)           TLS_AES_256_GCM_SHA384
                              TLS_CHACHA20_POLY1305_SHA256
                              ECDHE-RSA-AES256-GCM-SHA384
                              ECDHE-RSA-AES256-SHA
                              ECDHE-RSA-CHACHA20-POLY1305
                              TLS_AES_128_GCM_SHA256
                              ECDHE-RSA-AES128-GCM-SHA256
                              ECDHE-RSA-AES128-SHA
 Elliptic curves offered:     prime256v1 X25519

If you do the same thing on the cluster with the FIPS 140-2 compliant Backyards mesh, the outputs will look totally different.

FIPS compliant cluster ❯ kubectl exec -ti testssl -- testssl.sh https://frontpage.backyards-demo.svc.cluster.local.:8080
 Start 2020-08-06 14:32:22        -->> 10.63.243.56:8080 (frontpage.backyards-demo.svc.cluster.local.) <<--

 rDNS (10.63.243.56):    frontpage.backyards-demo.svc.cluster.local.
 Service detected:       certificate-based authentication => skipping all HTTP checks


 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2 (offered)

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            not offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)


 Testing server's cipher preferences

 Has server cipher order?     yes (OK) -- only for < TLS 1.3
 Negotiated protocol          TLSv1.3
 Negotiated cipher            TLS_AES_256_GCM_SHA384, 256 bit ECDH (P-256)
 Cipher per protocol

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 -
SSLv3
 -
TLSv1
 -
TLSv1.1
 -
TLSv1.2 (server order)
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 256   AESGCM      128
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 256   AESGCM      256
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256
TLSv1.3 (no server order, thus listed by strength)
 x1302   TLS_AES_256_GCM_SHA384            ECDH 256   AESGCM      256
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 256   ChaCha20    256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 256   AESGCM      128


 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

 FS is offered (OK)           TLS_AES_256_GCM_SHA384
                              TLS_CHACHA20_POLY1305_SHA256
                              ECDHE-RSA-AES256-GCM-SHA384
                              TLS_AES_128_GCM_SHA256
                              ECDHE-RSA-AES128-GCM-SHA256
 Elliptic curves offered:     prime256v1

As you can see our FIPS compliant Istio build only allows connection with TLS1.2+ and only with FIPS compliant ciphers. Also, the only allowed elliptic curve algorithm is the FIPS compliant P-256/prime256v1.

Backyards provides gateway configuration features with Let’s Encrypt support utilizing an FIPS 140-2 compliant build of cert-manager, so the same TLS settings apply not just in east-west mTLS communication, but for ingress TLS traffic as well.

FIPS 140-2 compliant Backyards TLS settings πŸ”—︎

Allowed TLS versions

  • TLS v1.2
  • TLS v1.3

Although FIPS 140-2 would allow lower TLS versions under some circumstances, we disabled them for security reasons. TLS 1.0 and 1.1 are out-of-date protocols that do not support modern cryptographic algorithms, and they contain security vulnerabilities that may be exploited by attackers. The IETF is also planning to officially deprecate both protocols. In addition, the vast majority of encrypted Internet traffic is now over TLS 1.2, which was introduced over a decade ago.

Allowed FIPS compatible ciphers

  • ECDHE-RSA-AES128-GCM-SHA256
  • ECDHE-RSA-AES256-GCM-SHA384
  • ECDHE-ECDSA-AES128-GCM-SHA256
  • ECDHE-ECDSA-AES256-GCM-SHA384
  • AES128-GCM-SHA256
  • AES256-GCM-SHA384

There are more ciphers allowed by FIPS 140-2. We only enable GCM ciphers, because only those ciphers can prevent a LUCKY13 timing attack

Allowed Elliptic-curve algorithm

  • P-256

Summary πŸ”—︎

As was mentioned in the beginning, ensuring that each of the components are using an FIPS validated cryptographic module is easily complicated if:

  • there might be no FIPS validated library (cryptographic module) available that is compatible with the component
  • third party components may not allow replacing the used cryptography with an FIPS-validated one
  • each component requires its own custom solution which increases the maintenance burden

Having an FIPS 140-2 compliant service mesh can help you a great deal. It gives you a unified layer in front of your applications and in this particular case, provides the encryption settings and enforcement that your environment requires.

Flow

Check out Backyards in action on your own clusters!

Register for an evaluation version and run a simple install command!

If you’d like to use a FIPS 140-2 compliant version of Istio/Backyards, get in touch with us

Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we’ve already blogged about. .