Elasticsearch output plugin for Fluentd 🔗︎

Overview 🔗︎

More info at https://github.com/uken/fluent-plugin-elasticsearch

Example Deployment: Save all logs to ElasticSearch

Example output configurations 🔗︎

spec:
  elasticsearch:
    host: elasticsearch-elasticsearch-cluster.default.svc.cluster.local
    port: 9200
    scheme: https
    ssl_verify: false
    ssl_version: TLSv1_2
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true

Configuration 🔗︎

Elasticsearch 🔗︎

Send your logs to Elasticsearch 🔗︎

Variable Name Type Required Default Description
host string No localhost You can specify Elasticsearch host by this parameter.
port int No 9200 You can specify Elasticsearch port by this parameter.
hosts string No - You can specify multiple Elasticsearch hosts with separator “,”. If you specify hosts option, host and port options are ignored.
user string No - User for HTTP Basic authentication. This plugin will escape required URL encoded characters within %{} placeholders. e.g. %{demo+}
password *secret.Secret No - Password for HTTP Basic authentication.
Secret
path string No - Path for HTTP Basic authentication.
scheme string No http Connection scheme
ssl_verify *bool No true Skip ssl verification (default: true)
ssl_version string No - If you want to configure SSL/TLS version, you can specify ssl_version parameter. [SSLv23, TLSv1, TLSv1_1, TLSv1_2]
ssl_max_version string No - Specify min/max SSL/TLS version
ssl_min_version string No -
ca_file *secret.Secret No - CA certificate
client_cert *secret.Secret No - Client certificate
client_key *secret.Secret No - Client certificate key
client_key_pass *secret.Secret No - Client key password
logstash_format bool No false Enable Logstash log format.
include_timestamp bool No false Adds a @timestamp field to the log, following all settings logstash_format does, except without the restrictions on index_name. This allows one to log to an alias in Elasticsearch and utilize the rollover API.
logstash_prefix string No logstash Set the Logstash prefix.
logstash_prefix_separator string No - Set the Logstash prefix separator.
logstash_dateformat string No %Y.%m.%d Set the Logstash date format.
index_name string No fluentd The index name to write events to
type_name string No fluentd Set the index type for elasticsearch. This is the fallback if target_type_key is missing.
pipeline string No - This param is to set a pipeline id of your elasticsearch to be added into the request, you can configure ingest node.
time_key_format string No - The format of the time stamp field (@timestamp or what you specify with time_key). This parameter only has an effect when logstash_format is true as it only affects the name of the index we write to.
time_precision string No - Should the record not include a time_key, define the degree of sub-second time precision to preserve from the time portion of the routed event.
time_key string No - By default, when inserting records in Logstash format, @timestamp is dynamically created with the time at log ingestion. If you’d like to use a custom time, include an @timestamp with your record.
utc_index *bool No true By default, the records inserted into index logstash-YYMMDD with UTC (Coordinated Universal Time). This option allows to use local time if you describe utc_index to false.(default: true)
target_index_key string No - Tell this plugin to find the index name to write to in the record under this key in preference to other mechanisms. Key can be specified as path to nested record using dot (’.') as a separator. https://github.com/uken/fluent-plugin-elasticsearch#target_index_key
target_type_key string No fluentd Similar to target_index_key config, find the type name to write to in the record under this key (or nested record). If key not found in record - fallback to type_name.
template_name string No - The name of the template to define. If a template by the name given is already present, it will be left unchanged, unless template_overwrite is set, in which case the template will be updated.
template_file *secret.Secret No - The path to the file containing the template to install.
Secret
templates string No - Specify index templates in form of hash. Can contain multiple templates.
customize_template string No - Specify the string and its value to be replaced in form of hash. Can contain multiple key value pair that would be replaced in the specified template_file. This setting only creates template and to add rollover index please check the rollover_index configuration.
rollover_index bool No false Specify this as true when an index with rollover capability needs to be created. https://github.com/uken/fluent-plugin-elasticsearch#rollover_index
index_date_pattern *string No now/d Specify this to override the index date pattern for creating a rollover index.
deflector_alias string No - Specify the deflector alias which would be assigned to the rollover index created. This is useful in case of using the Elasticsearch rollover API
index_prefix string No logstash Specify the index prefix for the rollover index to be created.
application_name *string No default Specify the application name for the rollover index to be created.
template_overwrite bool No false Always update the template, even if it already exists.
max_retry_putting_template string No 10 You can specify times of retry putting template.
fail_on_putting_template_retry_exceed *bool No true Indicates whether to fail when max_retry_putting_template is exceeded. If you have multiple output plugin, you could use this property to do not fail on fluentd statup.(default: true)
max_retry_get_es_version string No 15 You can specify times of retry obtaining Elasticsearch version.
request_timeout string No 5s You can specify HTTP request timeout.
reload_connections *bool No true You can tune how the elasticsearch-transport host reloading feature works.(default: true)
reload_on_failure bool No false Indicates that the elasticsearch-transport will try to reload the nodes addresses if there is a failure while making the request, this can be useful to quickly remove a dead node from the list of addresses.
reload_after string No - When reload_connections true, this is the integer number of operations after which the plugin will reload the connections. The default value is 10000.
resurrect_after string No 60s You can set in the elasticsearch-transport how often dead connections from the elasticsearch-transport’s pool will be resurrected.
include_tag_key bool No false This will add the Fluentd tag in the JSON record.
tag_key string No tag This will add the Fluentd tag in the JSON record.
id_key string No - https://github.com/uken/fluent-plugin-elasticsearch#id_key
routing_key string No - Similar to parent_key config, will add _routing into elasticsearch command if routing_key is set and the field does exist in input event.
remove_keys_on_update string No - Remove keys on update will not update the configured keys in elasticsearch when a record is being updated. This setting only has any effect if the write operation is update or upsert.
remove_keys_on_update_key string No - This setting allows remove_keys_on_update to be configured with a key in each record, in much the same way as target_index_key works.
retry_tag string No - This setting allows custom routing of messages in response to bulk request failures. The default behavior is to emit failed records using the same tag that was provided.
write_operation string No index The write_operation can be any of: (index,create,update,upsert)
reconnect_on_error bool No false Indicates that the plugin should reset connection on any error (reconnect on next send). By default it will reconnect only on “host unreachable exceptions”. We recommended to set this true in the presence of elasticsearch shield.
with_transporter_log bool No false This is debugging purpose option to enable to obtain transporter layer log.
content_type string No application/json With content_type application/x-ndjson, elasticsearch plugin adds application/x-ndjson as Content-Type in payload.
include_index_in_url bool No - With this option set to true, Fluentd manifests the index name in the request URL (rather than in the request body). You can use this option to enforce an URL-based access control.
time_parse_error_tag string No - With logstash_format true, elasticsearch plugin parses timestamp field for generating index name. If the record has invalid timestamp value, this plugin emits an error event to @ERROR label with time_parse_error_tag configured tag.
http_backend string No excon With http_backend typhoeus, elasticsearch plugin uses typhoeus faraday http backend. Typhoeus can handle HTTP keepalive.
prefer_oj_serializer bool No false With default behavior, Elasticsearch client uses Yajl as JSON encoder/decoder. Oj is the alternative high performance JSON encoder/decoder. When this parameter sets as true, Elasticsearch client uses Oj as JSON encoder/decoder.
flatten_hashes bool No - Elasticsearch will complain if you send object and concrete values to the same field. For example, you might have logs that look this, from different places:
{“people” => 100} {“people” => {“some” => “thing”}}
The second log line will be rejected by the Elasticsearch parser because objects and concrete values can’t live in the same field. To combat this, you can enable hash flattening.
flatten_hashes_separator string No - Flatten separator
validate_client_version bool No false When you use mismatched Elasticsearch server and client libraries, fluent-plugin-elasticsearch cannot send data into Elasticsearch.
unrecoverable_error_types string No - Default unrecoverable_error_types parameter is set up strictly. Because es_rejected_execution_exception is caused by exceeding Elasticsearch’s thread pool capacity. Advanced users can increase its capacity, but normal users should follow default behavior.
If you want to increase it and forcibly retrying bulk request, please consider to change unrecoverable_error_types parameter from default value.
Change default value of thread_pool.bulk.queue_size in elasticsearch.yml)
verify_es_version_at_startup *bool No true Because Elasticsearch plugin should change behavior each of Elasticsearch major versions.
For example, Elasticsearch 6 starts to prohibit multiple type_names in one index, and Elasticsearch 7 will handle only _doc type_name in index.
If you want to disable to verify Elasticsearch version at start up, set it as false.
When using the following configuration, ES plugin intends to communicate into Elasticsearch 6. (default: true)
default_elasticsearch_version string No 5 This parameter changes that ES plugin assumes default Elasticsearch version.
custom_headers string No {} This parameter adds additional headers to request. Example: {“token”:“secret”}
log_es_400_reason bool No false By default, the error logger won’t record the reason for a 400 error from the Elasticsearch API unless you set log_level to debug. However, this results in a lot of log spam, which isn’t desirable if all you want is the 400 error reasons. You can set this true to capture the 400 error reasons without all the other debug logs.
suppress_doc_wrap bool No false By default, record body is wrapped by ‘doc’. This behavior can not handle update script requests. You can set this to suppress doc wrapping and allow record body to be untouched.
ignore_exceptions string No - A list of exception that will be ignored - when the exception occurs the chunk will be discarded and the buffer retry mechanism won’t be called. It is possible also to specify classes at higher level in the hierarchy. For example
ignore_exceptions ["Elasticsearch::Transport::Transport::ServerError"]
will match all subclasses of ServerError - Elasticsearch::Transport::Transport::Errors::BadRequest, Elasticsearch::Transport::Transport::Errors::ServiceUnavailable, etc.
exception_backup *bool No true Indicates whether to backup chunk when ignore exception occurs. (default: true)
bulk_message_request_threshold string No 20MB Configure bulk_message request splitting threshold size.
Default value is 20MB. (20 * 1024 * 1024)
If you specify this size as negative number, bulk_message request splitting feature will be disabled.
sniffer_class_name string No - The default Sniffer used by the Elasticsearch::Transport class works well when Fluentd has a direct connection to all of the Elasticsearch servers and can make effective use of the _nodes API. This doesn’t work well when Fluentd must connect through a load balancer or proxy. The parameter sniffer_class_name gives you the ability to provide your own Sniffer class to implement whatever connection reload logic you require. In addition, there is a new Fluent::Plugin::ElasticsearchSimpleSniffer class which reuses the hosts given in the configuration, which is typically the hostname of the load balancer or proxy. https://github.com/uken/fluent-plugin-elasticsearch#sniffer-class-name
buffer *Buffer No - Buffer
enable_ilm bool No - Enable Index Lifecycle Management (ILM).
ilm_policy_id string No - Specify ILM policy id.
ilm_policy string No - Specify ILM policy contents as Hash.
ilm_policy_overwrite bool No - Specify whether overwriting ilm policy or not.