Overview 🔗︎

More info at https://github.com/fluent/fluent-plugin-kafka

Example Deployment: Transport Nginx Access Logs into Kafka with Logging Operator

Example output configurations 🔗︎

     brokers: kafka-headless.kafka.svc.cluster.local:29092
     default_topic: topic
     sasl_over_ssl: false
       type: json
       tags: topic
       timekey: 1m
       timekey_wait: 30s
       timekey_use_utc: true

Configuration 🔗︎

Kafka 🔗︎

Send your logs to Kafka 🔗︎

Variable Name Type Required Default Description
brokers string Yes - The list of all seed brokers, with their host and port information.
topic_key string No “topic” Topic Key
partition_key string No “partition” Partition
partition_key_key string No “partition_key” Partition Key
message_key_key string No “message_key” Message Key
default_topic string No nil The name of default topic .
default_partition_key string No nil The name of default partition key .
default_message_key string No nil The name of default message key .
exclude_topic_key bool No false Exclude Topic key
exclude_partion_key bool No false Exclude Partition key
get_kafka_client_log bool No false Get Kafka Client log
headers map[string]string No {} Headers
headers_from_record map[string]string No {} Headers from Record
use_default_for_unknown_topic bool No false Use default for unknown topics
idempotent bool No false Idempotent
sasl_over_ssl bool Yes true SASL over SSL
max_send_retries int No 1 Number of times to retry sending of messages to a leader
required_acks int No -1 The number of acks required per request .
ack_timeout int No nil => Uses default of ruby-kafka library How long the producer waits for acks. The unit is seconds
compression_codec string No nil The codec the producer uses to compress messages . The available options are gzip and snappy.
ssl_ca_cert *secret.Secret No - CA certificate
ssl_client_cert *secret.Secret No - Client certificate
ssl_client_cert_chain *secret.Secret No - Client certificate chain
ssl_client_cert_key *secret.Secret No - Client certificate key
ssl_verify_hostname *bool No - Verify certificate hostname
format *Format Yes - Format
buffer *Buffer No - Buffer