This guide describes how to collect application and container logs in Kubernetes using the Logging operator, and how to send them to Splunk.

Logging operator collects the logs from the application, selects which logs to forward to the output, and sends the selected log messages to the output (in this case, to Splunk). For more details about the Logging operator, see the Logging operator overview.

Deploy Splunk 🔗︎

First, deploy Splunk Standalone in your Kubernetes cluster. The following procedure is based on the Splunk on Kubernetes quickstart.

  1. Create the logging Namespace.

    kubectl create ns logging
    
  2. Install the Splunk operator.

    kubectl apply -n logging -f https://tiny.cc/splunk-operator-install
    
  3. Install the Splunk cluster.

    kubectl apply -n logging -f - <<"EOF" 
    apiVersion: enterprise.splunk.com/v1alpha2
    kind: Standalone
    metadata:
      name: single
    spec:
      config:
        splunkPassword: helloworld456
        splunkStartArgs: --accept-license
      topology:
        standalones: 1
    EOF
    

Deploy the Logging operator and a demo Application 🔗︎

Install the Logging operator and a demo application to provide sample log messages.

Deploy the Logging operator with Kubernetes manifests 🔗︎

To deploy the Logging operator using Kubernetes manifests, complete these steps. If you want to install the Logging operator using Helm, see Deploy the Logging operator with Helm.

  1. Install the Logging operator. For details, see Deploy the Logging operator with Kubernetes manifests.

  2. Create the logging resource.

    kubectl -n logging apply -f - <<"EOF" 
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Logging
    metadata:
      name: default-logging-simple
    spec:
      fluentd: {}
      fluentbit: {}
      controlNamespace: logging
    EOF
    

    Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

  3. Get a Splunk HEC Token.

    HEC_TOKEN=$(kubectl get secret -n logging  splunk-single-standalone-secrets -o jsonpath='{.data.hec_token}' | base64 --decode)
    
  4. Create a Splunk output secret from the token.

    kubectl  create secret generic splunk-token -n logging --from-literal "SplunkHecToken=${HEC_TOKEN}"
    
  5. Define a Splunk output.

    kubectl -n logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Output
    metadata:
     name: splunk-output
    spec:
     splunkHec:
        hec_host: splunk-single-standalone-headless
        insecure_ssl: true
        hec_port: 8088
        hec_token: 
            valueFrom:
               secretKeyRef:
                  name:  splunk-token
                  key: SplunkHecToken
        index: main
        format:
          type: json 
    EOF
    
  6. Create a flow resource.

    kubectl -n logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Flow
    metadata:
      name: splunk-flow
    spec:
      filters:
        - tag_normaliser: {}
        - parser:
            remove_key_name_field: true
            reserve_data: true
            parse:
              type: nginx
      match:
        - select:
            labels:
              app.kubernetes.io/name: log-generator
      localOutputRefs:
        - splunk-output
    EOF
    
  7. Install the demo application.

    kubectl -n logging apply -f - <<"EOF" 
    apiVersion: apps/v1
    kind: Deployment
    metadata:
     name: log-generator
     labels:
       app.kubernetes.io/name: log-generator
    spec:
     replicas: 3
     selector:
       matchLabels:
         app.kubernetes.io/name: log-generator
     template:
       metadata:
         labels:
           app.kubernetes.io/name: log-generator
       spec:
         containers:
         - name: log-generator
           image: banzaicloud/log-generator:0.3.2
    EOF
    

Validate the deployment 🔗︎

To validate that the deployment was successful, complete the following steps.

  1. Use the following command to retrieve the password of the admin user:

    kubectl -n logging get secret splunk-single-standalone-secrets -o jsonpath='{.data.password}' | base64 --decode
    
  2. Enable port forwarding to the Splunk Dashboard Service.

    kubectl -n logging port-forward svc/splunk-single-standalone-headless 8000
    
  3. Open the Splunk dashboard in your browser: http://localhost:8000. You should see the dashboard and some sample log messages from the demo application.

If you don’t get the expected result you can find help in the troubleshooting section.