Log Collection and Ingestion

Log Ingestion provides a way to transform unstructured log data into structured data and ingest it using Circonus. Structured log data allows for improved queries and filtering based on the data format when searching logs for an event.

Circonus log ingestion requires either Fluent Bit or Data Prepper to be deployed to collect logs, which are then passed to the Circonus Pipeline Exporter for ingestion into Circonus.

Basic Log Ingestion Data Flow into Circonus

Data flow diagram from a distributed application to Circonus

  1. Log Ingestion relies on you adding log collection to your application’s environment to gather and send log data.

  2. FluentBit is used as the log collector that collects log data from a file and sends the log data to Circonus Pipeline Exporter which encrypts, compresses and transports to the Circonus platform).

  3. The data can then be explored through Circonus search queries or the Logs Explorer in Circonus.

Next: Trace Ingestion