Log Collection and Ingestion

Overview

Log Ingestion provides a way to transform unstructured log data into structured data and ingest it using Circonus. Structured log data allows for improved queries and filtering based on the data format when searching logs for an event.

Circonus log ingestion requires either Fluent-bit or Data-prepper to be deployed to collect logs, which are then passed to the Circonus Pipeline Exporter for ingestion into Circonus.

Basic Log Ingestion Data Flow into Circonus

Data flow diagram from a distributed application to Circonus

  1. Log Ingestion relies on you adding log collection to your application’s environment to gather and send log data.

  2. Fluent-bit is used as the log collector that collects log data from a file and sends the log data to Circonus Pipeline Exporter which encrypts, compresses and transports to the Circonus platform).

  3. The data can then be explored through Circonus search queries or the Logs Explorer in Circonus.

Quick Installation Guide

Overview

Let’s get some log data flowing into your new Circonus account. This quick install guide will get you up and running by shipping your linux or Windows logs to your Circonus account using Fluent-bit and the Circonus pipeline exporter.

After, you logs have been setup to be shipped to Circonus, we will then create an index pattern to then visualize your data.

Installation Steps

  1. Fluent-bit
  2. Circonus Pipeline Exporter
  3. Circonus index pattern creation

Fluent-bit for Linux

Install

A simple installation script is provided to be used for most Linux targets. This will by default install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Configure

Modify the Fluent-bit configuration file to collect logs and sent them to the c3-exporter found at /etc/fluent-bit/fluent-bit.conf

Using nano as an example to edit the file:

sudo nano /etc/fluent-bit/fluent-bit.conf

# Example config

[SERVICE]
    # Flush
    # =====
    # set an interval of seconds before to flush records to a destination
    flush        15

    # Daemon
    # ======
    # instruct Fluent Bit to run in foreground or background mode.
    daemon       Off

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level    info

    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf

    # Plugins File
    # ============
    # specify an optional 'Plugins' configuration file to load external plugins.
    plugins_file plugins.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    
    # Storage
    # =======
    # Fluent Bit can use memory and filesystem buffering based mechanisms
    #
    # - https://docs.fluentbit.io/manual/administration/buffering-and-storage
    #
    # storage metrics
    # ---------------
    # publish storage pipeline metrics in '/api/v1/storage'. The metrics are
    # exported only if the 'http_server' option is enabled.
    #
    storage.metrics on

    # storage.path
    # ------------
    # absolute file system path to store filesystem data buffers (chunks).
    #
    # storage.path /tmp/storage

    # storage.sync
    # ------------
    # configure the synchronization mode used to store the data into the
    # filesystem. It can take the values normal or full.
    #
    # storage.sync normal

    # storage.checksum
    # ----------------
    # enable the data integrity check when writing and reading data from the
    # filesystem. The storage layer uses the CRC32 algorithm.
    #
    # storage.checksum off

    # storage.backlog.mem_limit
    # -------------------------
    # if storage.path is set, Fluent Bit will look for data chunks that were
    # not delivered and are still in the storage layer, these are called
    # backlog data. This option configure a hint of maximum value of memory
    # to use when processing these records.
    #
    # storage.backlog.mem_limit 5M

[INPUT]
    name           systemd
    tag            systemd
    strip_underscores on
    read_from_tail on

# Optional: Comment out if you don't setup syslog
[INPUT]
    name   syslog
    parser syslog-rfc3164
    listen 127.0.0.1
    port   5140
    mode   tcp
    tag    syslog

# Add your specific Circonus host, user, and password to the following
[OUTPUT]
    name         opensearch
    host         <hostIpHere>
    port         9200
    http_user    <userNameHere>
    http_passwd  <passWordHere>
    generate_id  on
    logstash_format on
    logstash_prefix logs-systemd
    logstash_dateformat %Y-%m-%d
    time_key on @timestamp
    time_key_Format %Y-%m-%dT%H:%M:%S
    tls          on
    match        systemd

# Optional: Comment out if you don't setup syslog
[OUTPUT]
    name         opensearch
    host         <hostIpHere>
    port         9200
    http_user    <userNameHere>
    http_passwd  <passWordHere>
    generate_id  on
    logstash_format on
    logstash_prefix logs-syslog
    logstash_dateformat %Y-%m-%d
    time_key on @timestamp
    time_key_Format %Y-%m-%dT%H:%M:%S
    tls          on
    match        syslog

# Testing purposes only
#[OUTPUT]
#    name  stdout
#    match *

Optional: Adding syslogs as an [INPUT] collection

In the following directory /etc/rsyslog.d/ , create a file called 60-c3opensearch.conf with the following content action(type="omfwd" target="127.0.0.1" port="5140" protocol="TCP") to start collecting syslogs using the configuration example from above.

Run the following command to create the file.

$ sudo echo 'action(type="omfwd" target="127.0.0.1" port="5140" protocol="TCP")' > /etc/rsyslog.d/60-c3opensearch.conf

Check the contents of the newly created file

$ cat /etc/rsyslog.d/60-c3opensearch.conf 
action(type="omfwd" target="127.0.0.1" port="5140" protocol="tcp")

After creating the file, restart the rsyslog service

sudo systemctl restart rsyslog

Start the service

Now the following step is to instruct systemd to enable the service:

sudo systemctl start fluent-bit

Checking Fluent-bit’s status

sudo systemctl status fluent-bit.service
● fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-02-13 15:27:25 UTC; 1 week 1 day ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 1265874 (fluent-bit)
      Tasks: 3 (limit: 19187)
     Memory: 6.9M
        CPU: 5min 11.783s
     CGroup: /system.slice/fluent-bit.service
             └─1265874 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Feb 13 15:27:25 ubuntu-c3playground-observability-v1 fluent-bit[1265874]: [2023/02/13 15:27:25] [ info] [fluent bit] version=2.0.9, commit=, pid=1265874
Feb 13 15:27:25 ubuntu-c3playground-observability-v1 fluent-bit[1265874]: [2023/02/13 15:27:25] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
...

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/syslog file.

Fluent-bit for Windows

Download and Install

Download the Windows installer and follow the installation wizard instructions.

Configure

Modify the Fluent-bit configuration file to collect logs and sent them to the c3-exporter found at C:\Program Files\fluent-bit\conf\fluent-bit.conf

[SERVICE]
    # Flush
    # =====
    # set an interval of seconds before to flush records to a destination
    flush        15

    # Daemon
    # ======
    # instruct Fluent Bit to run in foreground or background mode.
    daemon       Off

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level    info

    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf

    # Plugins File
    # ============
    # specify an optional 'Plugins' configuration file to load external plugins.
    plugins_file plugins.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020

    # Storage
    # =======
    # Fluent Bit can use memory and filesystem buffering based mechanisms
    #
    # - https://docs.fluentbit.io/manual/administration/buffering-and-storage
    #
    # storage metrics
    # ---------------
    # publish storage pipeline metrics in '/api/v1/storage'. The metrics are
    # exported only if the 'http_server' option is enabled.
    #
    storage.metrics on

    # storage.path
    # ------------
    # absolute file system path to store filesystem data buffers (chunks).
    #
    # storage.path /tmp/storage

    # storage.sync
    # ------------
    # configure the synchronization mode used to store the data into the
    # filesystem. It can take the values normal or full.
    #
    # storage.sync normal

    # storage.checksum
    # ----------------
    # enable the data integrity check when writing and reading data from the
    # filesystem. The storage layer uses the CRC32 algorithm.
    #
    # storage.checksum off

    # storage.backlog.mem_limit
    # -------------------------
    # if storage.path is set, Fluent Bit will look for data chunks that were
    # not delivered and are still in the storage layer, these are called
    # backlog data. This option configure a hint of maximum value of memory
    # to use when processing these records.
    #
    # storage.backlog.mem_limit 5M

[INPUT]
    Name         winlog
    Channels     Setup,Application,Security,System,Windows PowerShell,Microsoft-Windows-sysmon/Operational
    Interval_Sec 1
    tag          winlog

# Adds a tag field named host.name to correlate metrics and logs
[FILTER]
    Name record_modifier
    Match *
    Record host.name ComputerName

[OUTPUT]
    name         opensearch
    # IP address of Circonus Pipeline Exporter
    host         <circonusPipelineExporterIp>
    port         9200
    # Credentials for your customer_ingest user
    http_user    <ingestUserName>
    http_passwd  <ingestUserPassword>
    generate_id  on
    logstash_format on
    logstash_prefix logs-winlog
    logstash_dateformat %Y-%m-%d
    time_key on @timestamp
    time_key_Format %Y-%m-%dT%H:%M:%S
    tls          on
    match        winlog

Create and Start the Service

Using Windows Powershell, you can manage the Fluent-bit service

Create and start a new Fluent-bit service, and check its status.

PS> New-Service fluent-bit -BinaryPathName "C:\PROGRA~1\fluent-bit\bin\fluent-bit.exe -c C:\PROGRA~1\fluent-bit\conf\fluent-bit.conf" -StartupType Automatic && Start-Service fluent-bit && get-service fluent-bit

Fluent-bit’s service can also be managed with the native Windows Services Manager.

  1. Right-click on the Start button to open the WinX Menu
  2. Select Run
  3. Type services.msc in the Run box which opens
  4. Windows Services Manager will open.
  5. Search for fluent-bit under the Name column

Additional information can be found on Fluent-bit’s website.

Circonus Pipeline Exporter

The Circonus pipeline exporter acts as a proxy between internal data collectors and data ingestion into the Circonus infrastructure. It offers a lower number of egress points, payload compression to reduce egress bandwidth costs, and enhanced security.

The pipeline exporter can be installed with a package or using a Docker Container. The following example will use a package.

Note: Reference example OS Linux Ubuntu 22.04

Download

Choose the Debian package for your platform and download it to your host.

Install

Once the package is downloaded on your host, Install the package

sudo apt install /path/to/exporter/c3-exporter_0.0.9_amd64.deb

Configure

Next, set up the exporter to send data to your account. The configuration file is located at /opt/circonus/c3/etc/c3-exporter.yaml on your host after installation.

Example c3-exporter.yaml file

server:
  listen_address: ":9200"
  cert_file: ""
  key_file: ""
  read_timeout: "60s"
  write_timeout: "60s"
  idle_timeout: "30s"
  read_header_timeout: "5s"
  handler_timeout: "30s"

destination:
  # Circonus host name
  host: "<yourAccountName>.c3.circonus.com"
  port: "9200"
  ca_file: ""
  enable_tls: true
  tls_skip_verify: false

circonus:
  check_target: ""
  # Circonus Data Source API Token
  api_key: "<apiToken>"
  api_url: "https://api.circonus.com/"
  flush_interval: "60s"

A full list of configuration options can be found at the Circonus exporter GitHub repo.

Restart

Restart the exporter so the configuration file changes will take effect and data will start being sent to Circonus.

sudo systemctl restart circonus-c3-exporter.service

Troubleshooting

Fluent-bit

Linux

Check if the Fluent-bit service is running

~$ sudo systemctl status fluent-bit.service
● fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-02-13 15:27:25 UTC; 1 week 1 day ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 1265874 (fluent-bit)
      Tasks: 3 (limit: 19187)
     Memory: 6.9M
        CPU: 5min 11.783s
     CGroup: /system.slice/fluent-bit.service
             └─1265874 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Feb 13 15:27:25 ubuntu-c3playground-observability-v1 fluent-bit[1265874]: [2023/02/13 15:27:25] [ info] [fluent bit] version=2.0.9, commit=, pid=1265874
...

View the Fluent-bit logs while the service is running

~$ sudo journalctl -u fluent-bit.service -f
Apr 05 19:52:04 ubuntu-c3-staging-observability-v1 fluent-bit[688648]: [2023/04/05 19:52:04] [ info] [fluent bit] version=2.0.9, commit=, pid=688648
Apr 05 19:52:04 ubuntu-c3-staging-observability-v1 fluent-bit[688648]: [2023/04/05 19:52:04] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
...

Additional information can be found on Fluent-bit’s website.

Windows

Check if the Fluent-bit service is running

PS> get-service fluent-bit

Status   Name               DisplayName
------   ----               -----------
Running  fluent-bit         fluent-bit

Start the service

PS> Start-Service fluent-bit

Stop the service

PS> Stop-Service fluent-bit

Remove the service

PS> Remove-Service fluent-bit

Fluent-bit’s service can also be managed with the native Windows Services Manager.

  1. Right-click on the Start button to open the WinX Menu
  2. Select Run
  3. Type services.msc in the Run box which opens
  4. Windows Services Manager will open.
  5. Search for fluent-bit under the Name column

Additional information can be found on Fluent-bit’s website.

Circonus Pipeline Exporter

~$ sudo systemctl status circonus-c3-exporter.service
● circonus-c3-exporter.service - Circonus C3 Data Exporter
     Loaded: loaded (/lib/systemd/system/circonus-c3-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-29 19:18:09 UTC; 1 week 0 days ago
       Docs: https://github.com/circonus/c3-exporter
   Main PID: 615963 (c3-exporterd)
      Tasks: 10 (limit: 9525)
     Memory: 11.2M
        CPU: 13min 17.823s
     CGroup: /system.slice/circonus-c3-exporter.service
             └─615963 /opt/circonus/c3/sbin/c3-exporterd --config=/opt/circonus/c3/etc/c3-exporter.yaml

Apr 06 13:51:13 ubuntu-c3-staging-observability-v1 c3-exporterd[615963]: {"level":"info","req_id":"d2700936-301b-4aac-892e-4a95853e0765","url":"https://c3-staging.c3.circonus.com:9>
Apr 06 13:51:19 ubuntu-c3-staging-observability-v1 c3-exporterd[615963]: {"level":"info","req_id":"40214b01-85c9-4380-b290-e7d35e07126b","url":"https://c3-staging.c3.circonus.com:9>
...

View the circonus-c3-exporter logs while the service is running

sudo journalctl -u circonus-c3-exporter -f
Feb 08 22:20:02 ubuntu-c3playground-observability systemd[1]: circonus-c3-exporter.service: Main process exited, code=exited, status=1/FAILURE
Feb 08 22:20:02 ubuntu-c3playground-observability systemd[1]: circonus-c3-exporter.service: Failed with result 'exit-code'.
Feb 08 22:20:03 ubuntu-c3playground-observability systemd[1]: circonus-c3-exporter.service: Scheduled restart job, restart counter is at 5.
Feb 08 22:20:03 ubuntu-c3playground-observability systemd[1]: Stopped Circonus C3 Data Exporter.
...

Example of data being processed and sent to Circonus.

Feb 08 23:01:02 ubuntu-c3playground-observability c3-exporterd[20077]: {"level":"info","req_id":"ec6ed8da-8b49-4488-9634-a5f7c12d6308","remote":"127.0.0.1:56120","proto":"HTTP/1.1","method":"POST","URI":"/_bulk","upstream_resp_code":200,"handle_dur":"283.960295ms","ups"

Create a Circonus Account Index Pattern

If you collected data using one of the Circonus ingest options or added sample data you get an index pattern, and you can start exploring your data. If you loaded your own data, follow these steps to create an index pattern.

  1. Log into your Circonus account and navigate to Management > Stack Management > Index Patterns > Create index pattern.

  2. Start typing in the Index pattern name field, and Circonus looks for the names of indices, data streams, and aliases that match your input.

    • To match multiple sources, use a wildcard (*). For example, filebeat-* matches filebeat-apache-a, filebeat-apache-b, and so on.
    • To match multiple single sources, enter their names, separated by a comma. Do not include a space after the comma. filebeat-a,filebeat-b matches two indices, but does not match filebeat-c.
    • To exclude a source, use a minus sign (-), for example, -test3.

  3. If Circonus detects an index with a timestamp, expand the Timestamp field menu, and then select the default field for filtering your data by time.

    • If your index doesn’t have time-based data, choose I don’t want to use the time filter.
    • If you don’t set a default time field, you can’t use global time filters on your dashboards. This is useful if you have multiple time fields and want to create dashboards that combine visualizations based on different timestamps.

  4. Click Create index pattern.

    • Circonus is now configured to use your data. When a new field is added to an index, the index pattern field list is updated the next time the index pattern is loaded.

  5. Select this newly created index pattern when you search and visualize your data.