Secure logging on Kubernetes with Fluentd and Fluent Bit

As we concluded in the previous blog post we continue this series about centralized and secure Kubernetes logging/log collection.Log messages can contain sensitive information thus it is important to secure the transport between the distributed parts of the log flow. This post describes how we have secured moving log messages on our Kubernetes clusters provisioned by Pipeline.

Logging series:
Centralized logging under Kubernetes
Secure logging on Kubernetes with Fluentd and Fluent Bit

Client Certificate Authentication

Most of us is familiar with the TLS protocol for securing connections like HTTPS. We will just discuss the basics of the TLS handshake and certificate management, to make sure we are not avoiding any details of securing the transport of log messages.

In a simple use-case the server owns its private key and a certificate that is signed by a third-party trusted authority. When the client connects and receives the server certificate it verifies the signature. After that it opens an encrypted channel that is secured by one of the symmetric cipher protocols.

To provide client authentication the protocol is extended with mTLS. You can read more about mTLS here. To enable mutual authentication a common way is to use client certificates. In this scenario the client also has a private key and certificate (signed by the third-party CA) and the server verifies it. To get a more detailed overview there is a great article by CloudFlare.


Setting up the Certificates

Following these easy steps you can set-up your own environment for securing your logging infrastructure. If you want a detailed how-to check this OpenSSL Certificate Authority article.

We will follow three easy steps:

  • Generate the Root CA (for self-signed certificates)
  • Generate the Intermediate Server Cert (Singed with CA)
  • Generate the Client certificate (Signed with CA)

Prepare the working directory

mkdir certs csr private
touch index.txt
echo "1000" > serial

You need a specific openssl configuration file openssl.cnf

To eliminate problems due to different OpenSSL settings we should ensure to use custom configuration with explicit settings (not falling back to defaults).

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.crt.pem

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 365
preserve          = no
policy            = policy_strict

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name (required)
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = US
stateOrProvinceName_default     = CA
#localityName_default           = Mountain View
0.organizationName_default      = Your company name
#organizationalUnitName_default =
emailAddress_default            =

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer
basicConstraints = critical,CA:true
keyUsage = critical, cRLSign, digitalSignature, keyCertSign

[ client_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

Generate files for Certification Authority (CA)

To start we first need a CA private key file: ca.key.pem . We will use this to sign the certificates.

openssl genrsa -aes256 -out private/ca.key.pem 4096

It’s highly recommended to protect keys with password, although you can skip it by removeing the -aes256 option.

Generate cert for Certification Authority (CA)

Using the key we generate the CA certificate: ca.crt.pem.

openssl req -config openssl.cnf \
  -key private/ca.key.pem \
  -new -x509 -days 365 -sha256 -extensions v3_ca \
  -out certs/ca.crt.pem
Generate Server Private Key

For the server component we also need a private key: server.key.pem

openssl genrsa -aes256 -out private/server.key.pem 4096
Create Server Certificate Sign Request (csr)

To produce a signed certificate first we need to generate a request: server.csr

openssl req -config openssl.cnf \
  -key private/server.key.pem \
  -new -sha256 -out csr/server.csr.pem
Self Sign Server CSR with CA

As soon as we get the CSR we can sign it with our CA: server.crt.pem

openssl ca -config openssl.cnf -outdir certs  \
  -cert certs/ca.crt.pem \
  -keyfile private/ca.key.pem \
  -extensions server_cert -days 365 -notext -md sha256 \
  -in csr/server.csr.pem \
  -out certs/server.crt.pem

Fill the Common Name like a hostname or fqdn, ie: server

Create the Client certificates

We follow similar steps like in case of the server certificate.

  • Create private key: client.key.pem
openssl genrsa -aes256 -out private/client.key.pem 4096
  • Create certification request: client.csr.pem
openssl req -config openssl.cnf \
  -key private/client.key.pem \
  -new -sha256 \
  -out csr/client.csr.pem
  • Sign Client CSR with CA: client.crt.pem
openssl ca -config openssl.cnf -outdir certs  \
  -cert certs/ca.crt.pem \
  -keyfile private/ca.key.pem \
  -extensions client_cert -days 365 -notext -md sha256 \
  -in csr/client.csr.pem \
  -out certs/client.crt.pem

Fill the Common Name like a hostname or fqdn, ie: client

Check the certificates

We are ready to use your fresh certificates. However it is highly advised to test if you did everything fine. Please run the following OpenSSL commands in two difference shells to check the two-way authentication.

Secure logging in action

To use these above we have to enable the TLS auth for Fluent-bit and Fluentd

Enable TLS on Fluent-bit

As the documentation says "Each output plugin that requires to perform Network I/O can optionally enable TLS...". Great, we already generated the needed certificates, so now we need to modify the configuration. For further information please read the official manual.

This is a snippet from our Fluent-bit Helm chart where the configuration looks like this:

    Name          forward
    Match         *
    Host          {{ }}
    Port          {{ .Values.backend.forward.port }}
    Retry_Limit False
    tls               On
    tls.verify        {{ .Values.backend.forward.tls.verify }}
    tls.ca_file       /fluent-bit/ssl/ca.crt.pem
    tls.crt_file      /fluent-bit/ssl/client.crt.pem
    tls.key_file      /fluent-bit/ssl/client.key.pem
    tls.key_passwd    ${TLS_PRIVATE_KEY_PASSPHRASE}
    Shared_Key        ${SHARED_KEY}

The ${VARABLE_NAME} values indicates that the values come from Environment variables.

To use the certificates as Secrets in Kubernetes we have to upload them to the cluster.

kubectl create secret generic fluentbit-tls \
    --from-file=ca.crt.pem=./certs/ca.crt.pem \
    --from-file=client.crt.pem=./certs/client.crt.pem \
    --from-file=client.key.pem=./private/client.key.pem \
    --from-literal 'server.key.passphrase=1111' \
    --from-literal 'fluent.shared.key=shared1234'

After this we only need to attach the secret to the pod with the /fluent-bit/ssl/ path.

Enable TLS on Fluentd

Luckily with the latest Fluentd we don’t need the secure_input plugin as it’s already bundled with the core. For more information you can check the official documentation.

This as a snippet from our custom Fluentd chart:

  @type   forward
  port    24231
  @log_level debug
    self_hostname fluentd
    shared_key fluentd
  <transport tls>
    version                TLSv1_2
    ca_path                /fluentd/etc/ssl/ca.crt.pem
    cert_path              /fluentd/etc/ssl/server.crt.pem
    private_key_path       /fluentd/etc/ssl/server.key.pem
    private_key_passphrase "#{ENV["TLS_PRIVATE_KEY_PASSPHRASE"]}"
    client_cert_auth       true

The #{ENV["VARABLE_NAME"]} values indicates that the values come from Environment variables.

We need to upload to server secret to the cluster as seen above.

kubectl create secret generic fluentd-tls \
    --from-file=ca.crt.pem=./certs/ca.crt.pem \
    --from-file=server.crt.pem=./certs/server.crt.pem \
    --from-file=server.key.pem=./private/server.key.pem \
    --from-literal 'server.key.passphrase=1234' \
    --from-literal 'fluent.shared.key=shared1234'

Put it together

In the first post we familiarized with the tools used for log collecting, aggregating and storing. In this article we extended that solution with secure communication channels and authentication.


In the following post we will simplify all of this in a Pipeline spotguide (publishing the Helm charts as well) and show some advanced log parsing and monitoring technics. See you later guys!



comments powered by Disqus