The following tips and commands can help you to troubleshoot your Logging operator installation.

Running on Kind 🔗︎

Persistent Volumes do not respect the fsGroup value on Kind so please disable using a PVC for fluentd:

kind: Logging
  name: example-on-kind
    disablePvc: true

First things to do 🔗︎

  1. Check that the necessary CRDs are installed. Issue the following command: kubectl get crd The output should include the following CRDs:     2019-12-05T15:11:48Z   2019-12-05T15:11:48Z            2019-12-05T15:11:48Z         2019-12-05T15:11:48Z          2019-12-05T15:11:48Z
  2. Verify that the Logging operator pod is running. Issue the following command: kubectl get pods |grep logging-operator The output should include the a running pod, for example:

    NAME                                          READY   STATUS      RESTARTS   AGE
    logging-demo-log-generator-6448d45cd9-z7zk8   1/1     Running     0          24m

Troubleshooting Fluent Bit 🔗︎

The following sections help you troubleshoot the Fluent Bit component of the Logging operator.

Check the Fluent Bit daemonset 🔗︎

Verify that the Fluent Bit daemonset is available. Issue the following command: kubectl get daemonsets The output should include a Fluent Bit daemonset, for example:

logging-demo-fluentbit   1         1         1       1            1           <none>          110s

Check the Fluent Bit configuration 🔗︎

You can display the current configuration of the Fluent Bit daemonset using the following command: kubectl get secret logging-demo-fluentbit -o jsonpath="{.data['fluent-bit\.conf']}" | base64 --decode

The output looks like the following:

    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
    storage.path  /buffers

    Name         tail
    DB  /tail-db/tail-containers-state.db
    Mem_Buf_Limit  5MB
    Parser  docker
    Path  /var/log/containers/*.log
    Refresh_Interval  5
    Skip_Long_Lines  On
    Tag  kubernetes.*

    Name        kubernetes
    Kube_CA_File  /var/run/secrets/
    Kube_Tag_Prefix  kubernetes.var.log.containers
    Kube_Token_File  /var/run/secrets/
    Kube_URL  https://kubernetes.default.svc:443
    Match  kubernetes.*
    Merge_Log  On

    Name          forward
    Match         *
    Host          logging-demo-fluentd.logging.svc
    Port          24240

    tls           On
    tls.verify    Off
    tls.ca_file   /fluent-bit/tls/ca.crt
    tls.crt_file  /fluent-bit/tls/tls.crt
    tls.key_file  /fluent-bit/tls/tls.key
    Shared_Key    Kamk2_SukuWenk
    Retry_Limit   False

Debug version of the fluentbit container 🔗︎

All Fluent Bit image tags have a debug version marked with the -debug suffix. You can install this debug version using the following command: kubectl edit logging-demo

      pullPolicy: Always
      repository: fluent/fluent-bit
      tag: 1.3.2-debug

After deploying the debug version, you can kubectl exec into the pod using sh and look around. For example: kubectl exec -it logging-demo-fluentbit-778zg sh

Check the queued log messages 🔗︎

You can check the buffer directory if Fluent Bit is configured to buffer queued log messages to disk instead of in memory. (You can configure it through the InputTail fluentbit config, by setting the storage.type field to filesystem.)

kubectl exec -it logging-demo-fluentbit-9dpzg ls /buffers

Troubleshooting Fluentd 🔗︎

The following sections help you troubleshoot the Fluentd statefulset component of the Logging operator.

Check Fluentd pod status (statefulset) 🔗︎

Verify that the Fluentd statefulset is available using the following command: kubectl get statefulsets

Expected output:

NAME                   READY   AGE
logging-demo-fluentd   1/1     1m

ConfigCheck 🔗︎

The Logging operator has a builtin mechanism that validates the generated fluentd configuration before applying it to fluentd. You should be able to see the configcheck pod and it’s log output. The result of the check is written into the status field of the corresponding Logging resource.

In case the operator is stuck in an error state caused by a failed configcheck, restore the previous configuration by modifying or removing the invalid resources to the point where the configcheck pod is finally able to complete successfully.

Check Fluentd configuration 🔗︎

Use the following command to display the configuration of Fluentd: kubectl get secret logging-demo-fluentd-app -o jsonpath="{.data['fluentd\.conf']}" | base64 --decode

The output should be similar to the following:

  @type forward
  @id main_forward
  port 24240
  <transport tls>
    ca_path /fluentd/tls/ca.crt
    cert_path /fluentd/tls/tls.crt
    client_cert_auth true
    private_key_path /fluentd/tls/tls.key
    version TLSv1_2
    self_hostname fluentd
    shared_key Kamk2_SukuWenk
<match **>
  @type label_router
  @id main_label_router
    @label @427b3e18f3a3bc3f37643c54e9fc960b
    namespace logging
<label @427b3e18f3a3bc3f37643c54e9fc960b>
  <match kubernetes.**>
    @type tag_normaliser
    @id logging-demo-flow_0_tag_normaliser
    format ${namespace_name}.${pod_name}.${container_name}
  <filter **>
    @type parser
    @id logging-demo-flow_1_parser
    key_name log
    remove_key_name_field true
    reserve_data true
      @type nginx
  <match **>
    @type s3
    @id logging_logging-demo-flow_logging-demo-output-minio_s3
    aws_key_id WVKblQelkDTSKTn4aaef
    aws_sec_key LAmjIah4MTKTM3XGrDxuD2dTLLmysVHvZrtxpzK6
    force_path_style true
    path logs/${tag}/%Y/%m/%d/
    s3_bucket demo
    s3_endpoint http://logging-demo-minio.logging.svc.cluster.local:9000
    s3_region test_region
    <buffer tag,time>
      @type file
      path /buffers/logging_logging-demo-flow_logging-demo-output-minio_s3.*.buffer
      retry_forever true
      timekey 10s
      timekey_use_utc true
      timekey_wait 0s

Set Fluentd log Level 🔗︎

Use the following command to change the log level of Fluentd. kubectl edit logging-demo

  logLevel: debug

Get Fluentd logs 🔗︎

The following command displays the logs of the Fluentd container. kubectl exec -it logging-demo-fluentd-0 cat /fluentd/log/out

The One Eye observability tool can display Fluentd logs on its web UI, where you can select which replica to inspect, search the logs, and use other ways to monitor and troubleshoot your logging infrastructure.

Set stdout as an output 🔗︎

You can use an stdout filter at any point in the flow to dump the log messages to the stdout of the Fluentd container. For example: kubectl edit logging-demo

kind: Flow
  name: exchange
  namespace: logging
    - stdout: {}
    - exchange
    application: exchange

Check the buffer path in the fluentd container 🔗︎

kubectl exec -it logging-demo-fluentd-0 ls /buffers

Defaulting container name to fluentd.
Use 'kubectl describe pod/logging-demo-fluentd-0 -n logging' to see all of the containers in this pod.

Getting Support 🔗︎

If you encounter any problems that the documentation does not address, file an issue or talk to us on the Banzai Cloud Slack channel #logging-operator.

Commercial support is also available for the Logging operator.

Before asking for help, prepare the following information to make troubleshooting faster:

Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.