Kafka output plugin for Fluentd 🔗︎

Overview 🔗︎

More info at https://github.com/fluent/fluent-plugin-kafka

Example Deployment: Transport Nginx Access Logs into Kafka with Logging Operator

Example output configurations 🔗︎

    brokers: kafka-headless.kafka.svc.cluster.local:29092
    default_topic: topic
    sasl_over_ssl: false
      type: json
      tags: topic
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true

Configuration 🔗︎

Kafka 🔗︎

Send your logs to Kafka 🔗︎

Variable Name Type Required Default Description
brokers string Yes - The list of all seed brokers, with their host and port information.
topic_key string No “topic” Topic Key
partition_key string No “partition” Partition
partition_key_key string No “partition_key” Partition Key
message_key_key string No “message_key” Message Key
default_topic string No nil The name of default topic .
default_partition_key string No nil The name of default partition key .
default_message_key string No nil The name of default message key .
exclude_topic_key bool No false Exclude Topic key
exclude_partion_key bool No false Exclude Partition key
get_kafka_client_log bool No false Get Kafka Client log
headers map[string]string No {} Headers
headers_from_record map[string]string No {} Headers from Record
use_default_for_unknown_topic bool No false Use default for unknown topics
idempotent bool No false Idempotent
sasl_over_ssl bool Yes true SASL over SSL
username *secret.Secret No - Username when using PLAIN/SCRAM SASL authentication
password *secret.Secret No - Password when using PLAIN/SCRAM SASL authentication
scram_mechanism string No - If set, use SCRAM authentication with specified mechanism. When unset, default to PLAIN authentication
max_send_retries int No 1 Number of times to retry sending of messages to a leader
required_acks int No -1 The number of acks required per request .
ack_timeout int No nil => Uses default of ruby-kafka library How long the producer waits for acks. The unit is seconds
compression_codec string No nil The codec the producer uses to compress messages . The available options are gzip and snappy.
ssl_ca_certs_from_system *bool No false System’s CA cert store
ssl_ca_cert *secret.Secret No - CA certificate
ssl_client_cert *secret.Secret No - Client certificate
ssl_client_cert_chain *secret.Secret No - Client certificate chain
ssl_client_cert_key *secret.Secret No - Client certificate key
ssl_verify_hostname *bool No - Verify certificate hostname
format *Format Yes - Format
buffer *Buffer No - Buffer