Skip to content

How to set up fluentd.md

Latest commit

 

History

History
117 lines (88 loc) · 2.96 KB

How to set up fluentd.md

File metadata and controls

117 lines (88 loc) · 2.96 KB

Installing and Configuring Fluentd

Step 1: Install Dependencies and Ruby

Fluentd requires Ruby 1.9.3 or newer, along with bundler. We'll install Ruby and additional dependencies first.

Update the package list:

sudo apt update

Install prerequisites:

sudo apt install -y git build-essential libssl-dev zlib1g-dev libjemalloc-dev

Install Ruby: To get the latest stable Ruby version, use the following:

sudo apt install -y ruby-full

Verify the installation by running:

ruby -v

Install Bundler:

sudo gem install bundler

Step 2: Fetch Fluentd Source Code

Clone the Fluentd GitHub repository:

git clone https://github.com/fluent/fluentd.git
cd fluentd

Check out the stable branch (the latest stable version is in v1.x rather than v0.12, which is outdated):

git checkout v1.17

Step 3: Build and Install Fluentd

Install gem dependencies:

bundle install

Troubleshooting Installation

If you get any errors, you can configure Bundler to install gems to a directory in your home folder instead. This avoids permission issues by keeping the installation local to your user account.

Set up Bundler to install gems in your home directory:

bundle config set path 'vendor/bundle'

Install the gems:

bundle install --path vendor/bundle

Run the build command to create the Fluentd gem package:

bundle exec rake build

This should create a .gem file in the pkg directory (e.g., pkg/fluentd-xxx.gem).

Install the Fluentd gem:

gem install pkg/fluentd-xxx.gem

Replace xxx with the specific version number of the Fluentd gem file that was created.

Step 4: Verify the Installation

Set up a sample configuration:

fluentd --setup ./fluent

This creates a basic directory structure with a sample configuration file in ./fluent/fluent.conf.

Run Fluentd with the sample configuration:

fluentd -c ./fluent/fluent.conf -vv &

The -vv flag enables verbose output, which is useful for verifying that Fluentd runs without issues.

Send a test message to Fluentd:

echo '{"json":"message"}' | fluent-cat debug.test

If everything is working, Fluentd should output a log message similar to:

[timestamp] debug.test: {"json":"message"}

Step 5: Configure Fluentd for Your Logging System

After confirming Fluentd is working, the next steps are to set it up as your log accumulator:

Create a Fluentd Configuration for Log Aggregation

Edit the fluent.conf file to set up Fluentd to accept logs from your microservices, and configure it to forward logs to Apache Kafka. Fluentd has a Kafka output plugin that you can use to send logs to Kafka. You can install it by running:

sudo fluent-gem install fluent-plugin-kafka

Configure Input and Output Plugins

  • Set up an input plugin to receive logs (e.g., in_forward for receiving JSON logs from each microservice)
  • Configure an output plugin to send logs to Kafka