Customize Fluentd docker image

What is Fluentd

Briefly speaking, Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. Click here for more details.

You could use Fluentd by either installing the Fluentd agent manually as the daemon process or running a docker image. This article will mainly focus on how to build you own Fluentd docker image which fits your use cases.

Get started

• Create folder with the following skeleton

#$:~/fluentd-docker$ tree .
.
├── Dockerfile
├── fluent.conf
└── plugins

1 directory, 2 files

• Edit the Dockerfile

FROM fluent/fluentd:v1.4-debian-onbuild

# Use root account to use apt
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && apt-get install -y libsystemd0 \
 && sudo gem install fluent-plugin-vmware-loginsight \
 && sudo gem install fluent-plugin-systemd -v 1.0.1 \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem

USER fluent

• apt-get install -y libsystemd0 and sudo gem install fluent-plugin-systemd -v 1.0.1 are specificly for fetching systemd logs

• Refer to the official doc for more details

• All available fluentd plugins could be found here

• fluent.conf is required to build the custom docker image. It could be a default conf file. When you run the docker image the config file could be replaced from the docker run command.

The following fluent.conf is just an example for fetching the systemd log:

<source>
  @type systemd
  tag pks-mgmt-server
  path /var/log/journal
  matches [{ "_SYSTEMD_UNIT": "pks-mgmt-server.service" }]
  read_from_head true
</source>

<source>
  @type systemd
  tag pks-ui-server
  path /var/log/journal
  matches [{ "_SYSTEMD_UNIT": "pks-ui-server.service" }]
  read_from_head true
</source>

<source>
  @type systemd
  tag nginx
  path /var/log/journal
  matches [{ "_SYSTEMD_UNIT": "nginx.service" }]
  read_from_head true
</source>

<filter **>
  @type systemd_entry
  fields_lowercase true
  fields_strip_underscores true
</filter>

<match **>
  @type vmware_loginsight
  host <log-server-ip>
  port 9543
  scheme https
    ssl_verify false
  path api/v1/events/ingest
  agent_id xxxxxx
  http_method post
  serializer json
  rate_limit_msec 0
  raise_on_error false
  flush_mode interval
  flush_interval 20
  include_tag_key true
  tag_key tag
</match>

• You could leave plugins folder empty in this example

• Build the new image by using docker build command

• Refer to the official doc for running the container with your own configuration files and other additional options.