fluent/fluentd

Fluentd is a high-performance, extensible log and data collection engine designed for building robust data pipelines. It provides a modular architecture for ingesting, filtering, and routing events, supported by a comprehensive plugin system that includes HTTP servers, buffer management, and service discovery. The system features a modernized configuration interface supporting both XML and YAML, alongside a suite of CLI tools for debugging, certificate management, and plugin generation. Additionally, it offers a robust testing framework and a compatibility layer to ensure seamless operation of legacy plugins.

15 changes.

This period focused on a comprehensive architectural overhaul, introducing the v0.14 plugin API and a new modular command-line interface. The configuration system was significantly refactored with a new DSL and parser infrastructure, while the codebase expanded test coverage and updated dependencies to support these internal changes.

10 changes.

This period focused on introducing the v0.14 API, featuring new plugin helpers for metrics, event emission, and configuration compatibility. The buffer subsystem was refactored to support a new chunk-based API, while a comprehensive compatibility layer ensured legacy v0.12 plugins continued to function. Extensive test coverage and updated plugin templates were added to support the new architecture.

10 changes.

This period focused on modularizing the codebase by introducing a new Counter subsystem and refactoring the HTTP server, out\_forward, and in\_tail plugins into distinct, testable components. Comprehensive test coverage was added for all new and refactored modules to ensure stability and correctness.

5 changes.

This period focused on expanding configuration flexibility by introducing YAML support and a new logging adapter for the Console gem. Development efforts also included adding Rake tasks to automate backporting and benchmarking workflows, alongside corresponding test coverage.

Added new ERB templates for the plugin configuration formatter to generate Markdown output. Specifically, the system now supports three distinct Markdown layouts for plugin parameters: a compact list format (param.md-compact.erb), a table-based format (param.md-table.erb), and a standard header-based list format (param.md.erb). Additionally, a new template for rendering section headers and sub-sections in Markdown (section.md.erb) was introduced, enabling structured documentation of plugin configuration sections with support for required/optional, multiple/single, and alias metadata.

_templates/plugin\_config\formatter · high confidence

A new Rake task has been introduced to automate the process of backporting pull requests from an upstream repository to a specified branch. The script identifies closed pull requests labeled for backporting, creates corresponding pull requests on the target branch, and handles cherry-picking commits. This feature streamlines the maintenance workflow by reducing manual effort in managing backports.

tasks/backport · high confidence

A new \ConsoleAdapter\ class has been introduced in \lib/fluent/log/console\_adapter.rb\ to adapt the \Console\ gem's logger mechanism to Fluentd's logging system. This adapter handles the incompatibility between the \Console\ gem (used by the \http\_server\ helper) and Ruby's standard \Logger\, ensuring that log levels and messages are correctly mapped and output. This change supports the \Console\ gem, allowing for more flexible log formatting and output handling in asynchronous contexts.

lib/fluent/log · high confidence

New Rake tasks have been introduced to support development workflows. A backporting task is now available under the :backport namespace, allowing users to manually execute backporting processes via environment variables. Additionally, a :benchmark namespace has been added with tasks to initialize, prepare a 1GB data file, and run the in\_tail benchmark, including a Ruby patch to measure performance. Configuration for the benchmark is also provided.

tasks · high confidence

Added numerous example configuration files in the example/ directory to demonstrate usage of core plugins including copy, roundrobin, exec\_filter, file, forward (with TLS, shared key, user auth, and service discovery), http, syslog, tcp, udp, and tail. The examples also cover advanced features such as worker-specific configurations, buffer compression, and literal string escaping in config values.

example · high confidence

Fluentd can now be configured using YAML files in addition to the existing XML format. This new parser supports advanced YAML features including array formats for tags, glob-based file inclusion via the \!include\ directive, and special handling for \$id\, \$log\_level\, and \$tag\ elements. Users can now write more concise and readable configuration files using the \lib/fluent/config/yaml\_parser\ module.

_lib/fluent/config/yaml\parser · high confidence

The buffer subsystem has been refactored to support a new v0.14 Buffer API, introducing a \Chunk\ base class and specific implementations for memory (\MemoryChunk\) and file-based (\FileChunk\, \FileSingleChunk\) storage. This change adds new buffer types that allow users to configure in-memory or file-backed buffering with improved state management (unstaged, staged, queued, closed) and metadata handling. The \FileSingleChunk\ implementation consolidates chunk data into a single file per chunk, while \FileChunk\ maintains the previous multi-file (data + metadata) structure. These changes enable more flexible buffer configurations and better handling of chunk lifecycle states.

lib/fluent/plugin/buffer · high confidence

The configuration system has been refactored with a new DSL and parser architecture. A new \Fluent::Config::DSL\ module provides a Ruby-based configuration syntax, while \Fluent::Config::ConfigureProxy\ and \Fluent::Config::Element\ manage the configuration tree and metadata. The \Fluent::Config::Section\ class now handles section initialization and parameter binding. Additionally, \Fluent::Config::YamlParser\ has been added to support YAML configuration files, and \Fluent::Config::V1Parser\ handles the legacy V1 format. These changes enable more robust configuration validation, support for embedded Ruby code in config files, and improved error reporting.

lib/fluent/config · high confidence

Adds a new Counter subsystem to Fluentd, providing a client and server implementation for managing named counters. The client (lib/fluent/counter/client.rb) supports both synchronous and asynchronous (block) result handling, with a default port of 24321 and a 5-second timeout. The server (lib/fluent/counter/server.rb) exposes methods for initializing, incrementing, getting, resetting, and deleting counters, utilizing a MutexHash for thread-safe operations and a CleanupThread to periodically remove unused mutexes. The store layer (lib/fluent/counter/store.rb) persists counter data using the local storage plugin, and validation is handled by dedicated validator classes.

lib/fluent/counter · high confidence

The codebase introduces several new foundational classes: \Fluent::Agent\ manages emittable plugins and their lifecycle; \Fluent::Capability\ abstracts Linux capability handling (using \capng\ when available); \Fluent::Clock\ provides a monotonic clock with test-frozen capabilities; and \Fluent::Configurable\ standardizes configuration parameter handling. These changes establish the internal architecture for plugin management, timekeeping, and system capabilities.

lib/fluent · high confidence

The internal command-line interface for Fluentd has been refactored from a monolithic script into a modular, subcommand-based architecture. This change introduces distinct Ruby classes for each command (e.g., \FluentBinlogReader\, \FluentPluginConfigFormatter\, \FluentPluginGenerator\, \FluentCtl\, \FluentDebug\), each handling its own argument parsing and execution logic. This restructuring improves maintainability and allows for more complex command-line interactions, such as the new \fluent-binlog-reader\ for inspecting binary log files, \fluent-ca-generate\ for creating CA certificates, and \fluent-cap-ctl\ for managing Linux capabilities. The \fluentd\ command itself now supports a wider range of configuration and runtime options, including \--dry-run\ for configuration validation, \--inline-config\ for temporary configuration overrides, and granular control over logging, metrics, and worker processes.

lib/fluent/command · high confidence

Several new command-line tools are introduced: fluent-binlog-reader, fluent-ca-generate, fluent-cap-ctl, fluent-ctl, fluent-debug, fluent-plugin-config-format, and fluent-plugin-generate. The existing fluentd binary is optimized to disable trace\_instruction for RubyVM::InstructionSequence, reducing startup memory usage by approximately 0.5 MB. Additionally, the bin/fluent-cat script is simplified by removing the explicit require 'rubygems' call, as it is already required by Ruby.

bin · high confidence

The plugin\_helper directory introduces several new plugin helpers to support Fluentd's v0.14 API. The Metrics helper allows plugins to create and manage Prometheus-style metrics. The EventEmitter helper provides a standardized way for plugins to emit events to the router. The CompatParameters helper enables the conversion of legacy v0.12 configuration parameters (like buffer, parser, and formatter settings) into the new v0.14 format, ensuring backward compatibility. Additionally, the Extract and Inject helpers allow for the extraction and injection of time and tag data from/to records. These changes provide a more modular and consistent way to implement common plugin functionalities.

_lib/fluent/plugin\helper · high confidence

Adds a comprehensive compatibility layer for v0.12-style plugins, including new wrapper classes for Input, Filter, Output, and Parser, along with utility modules for handling configuration conversion, tag/time injection, and process detachment. This ensures that older plugins relying on the legacy API continue to function correctly within the newer architecture.

lib/fluent/compat · high confidence

The \lib/fluent/plugin\ directory has been updated to use the new v0.14 plugin API, which introduces plugin helpers for tasks like timers, event loops, and child processes. This migration simplifies plugin code, improves performance, and provides a more consistent interface for input, output, and filter plugins. The changes include the addition of new configuration parameters, improved error handling, and better integration with the Fluentd core.

lib/fluent/plugin · high confidence

The out\_forward plugin has been refactored into separate, modular files including AckHandler, ConnectionManager, FailureDetector, HandshakeProtocol, LoadBalancer, and SocketCache. This change improves code organization, thread safety, and maintainability of the forward output plugin's internal logic.

_lib/fluent/plugin/out\forward · high confidence

The in\_tail plugin was restructured by extracting the group watching logic into a new \GroupWatch\ module and the position file handling into a new \PositionFile\ class. This refactoring introduces a \GroupWatcher\ to manage file groups and rate limits, and a \PositionFile\ class to manage file positions and inodes, improving code organization and maintainability.

_lib/fluent/plugin/in\tail · high confidence

The HTTP server plugin helper has been rewritten to use the \async\ and \async-http\ gems, replacing the previous synchronous or CGI-based implementation. This change introduces new internal components (\App\, \Request\, \Router\, \Server\, and \SSLContextBuilder\) to handle HTTP requests, routing, and SSL/TLS configuration. Users benefit from improved performance and reliability, including proper handling of request bodies to prevent socket leaks, support for IPv6 addresses, and the ability to configure HTTPS via a TLS context.

_lib/fluent/plugin\_helper/http\server · high confidence

The templates for generating new Fluentd plugins have been reorganized and expanded. A .gitignore file was added to the template structure. The README template now includes an installation section and uses HTTPS for the Fluentd link. The gemspec template was updated to remove unnecessary files and specify version constraints for dependencies like bundler, rake, and test-unit. New templates were added for filter, formatter, input, output, parser, and storage plugins, each with appropriate base classes and registration calls. The Rakefile template was added to support testing tasks.

_templates/new\gem · high confidence

The http\_server helper now correctly constructs URIs for IPv6 bind addresses (e.g., \::\, \::1\) by properly bracketing them per RFC 3986, fixing a \URI::InvalidURIError\ that previously occurred when binding to IPv6 addresses. This affects all plugins using the http\_server helper with IPv6 bind addresses.

(repo-wide) · high confidence

Add tests for in\_tail plugin components; Added automated tests for Windows service lifecycle and script execution; Added compatibility tests for plugin lifecycle and parser behavior; Added comprehensive test coverage for plugin helpers; Added comprehensive test coverage for the configuration system; Added test fixtures for certificate handling; Added test fixtures for sd\_file plugin configuration; Added test plugins for Fluentd; Added test templates for new gem plugins; Added tests for Fluent::Log::ConsoleAdapter; Added tests for out\_forward plugin components; Added tests for service discovery manager and round-robin balancer; Added tests for the Counter plugin components; Added tests for the HTTP server plugin helper; Added tests for the renamed binlog-reader command and other command-line utilities; Added tests for v0.14 plugin base classes and buffer implementations; Expanded test coverage for core components; Introduce new test driver architecture for Fluentd plugins; New test driver framework for Fluentd plugins.

The project has updated its gem dependencies to align with modern Ruby versions, raising the minimum required Ruby version to 3.2. Key runtime dependencies have been updated or added, including msgpack (\>= 1.3.1), cool.io (\>= 1.4.5), serverengine (\>= 2.3.2), and webrick (\~\> 1.4). Additional dependencies such as zstd-ruby, uri, net-http, and async-http have been specified. Platform-specific dependencies for Windows (win32-service, win32-ipc, etc.) and Ruby 3.5/4.0/4.1 specific gems (logger, ostruct, fiddle) are now explicitly declared. Development dependencies like flexmock, rr, and test-unit have also been updated.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 69 → 69 (-0.3)
• Rubric changed (rubric-2026.08.17 → rubric-2026.08.19) — scores are not directly comparable.

• Code Health 75 → 74 (-1.7)
• Architecture 97 → 95 (-1.9)
• Maturity 59 → 59 (-0.1)
• Readiness 84 → 84 (+0.0)
• Security 75 → 77 (+1.6)

• Change coupling: engine.rb ↔ match.rb (lib/fluent/engine.rb)
• Change coupling: formatter_ltsv.rb ↔ formatter_out_file.rb (lib/fluent/plugin/formatter_ltsv.rb)
• Change coupling: in_monitor_agent.rb ↔ http_server.rb (lib/fluent/plugin/in_monitor_agent.rb)
• Change coupling: input.rb ↔ output.rb (lib/fluent/input.rb)
• Dependency hygiene not measured — no supported dependency manifest was read
• Duplicated block (11 lines × 2) (lib/fluent/command/ca_generate.rb)
• Duplicated block (11 lines × 2) (lib/fluent/plugin/in_exec.rb)
• Duplicated block (12 lines × 2) (lib/fluent/plugin/buffer/file_chunk.rb)
• Duplicated block (12 lines × 2) (lib/fluent/tls.rb)
• Duplicated block (14 lines × 2) (lib/fluent/compat/formatter_utils.rb)
• Duplicated block (14 lines × 2) (lib/fluent/plugin/buf_file.rb)
• Duplicated block (14 lines × 2) (lib/fluent/plugin_helper/compat_parameters.rb)
• Duplicated block (14 lines × 2) (lib/fluent/plugin_helper/formatter.rb)
• Duplicated block (15 lines × 2) (lib/fluent/counter/server.rb)
• Duplicated block (15 lines × 2) (lib/fluent/plugin/buf_file.rb)
• Duplicated block (15 lines × 2) (lib/fluent/plugin/in_tcp.rb)
• Duplicated block (15 lines × 2) (lib/fluent/plugin/parser_syslog.rb)
• Duplicated block (16 lines × 2) (lib/fluent/compat/output.rb)
• Duplicated block (17 lines × 2) (lib/fluent/plugin/in_forward.rb)
• Duplicated block (17 lines × 2) (lib/fluent/plugin/parser_syslog.rb)
• …and 12 more

• Change coupling: bare_output.rb ↔ filter.rb (lib/fluent/plugin/bare_output.rb)
• Change coupling: bare_output.rb ↔ input.rb (lib/fluent/plugin/bare_output.rb)
• Change coupling: bare_output.rb ↔ multi_output.rb (lib/fluent/plugin/bare_output.rb)
• Change coupling: bare_output.rb ↔ output.rb (lib/fluent/plugin/bare_output.rb)
• Change coupling: buf_file.rb ↔ buf_file_single.rb (lib/fluent/plugin/buf_file.rb)
• Change coupling: input.rb ↔ output.rb (lib/fluent/plugin/input.rb)
• Dependency hygiene not measured — dependency manifest found but not parsed for hygiene
• Duplicated block (10 lines × 2) (lib/fluent/plugin/in_exec.rb)
• Duplicated block (11 lines × 2) (lib/fluent/plugin/buffer/file_chunk.rb)
• Duplicated block (11 lines × 2) (lib/fluent/tls.rb)
• Duplicated block (12 lines × 2) (lib/fluent/plugin/parser_syslog.rb)
• Duplicated block (12 lines × 2) (lib/fluent/plugin_helper/formatter.rb)
• Duplicated block (13 lines × 2) (lib/fluent/compat/formatter_utils.rb)
• Duplicated block (13 lines × 2) (lib/fluent/plugin/buf_file.rb)
• Duplicated block (13 lines × 2) (lib/fluent/plugin_helper/compat_parameters.rb)
• Duplicated block (14 lines × 2) (lib/fluent/counter/server.rb)
• Duplicated block (14 lines × 2) (lib/fluent/plugin/buf_file.rb)
• Duplicated block (14 lines × 2) (lib/fluent/plugin/in_tcp.rb)
• Duplicated block (14 lines × 2) (lib/fluent/plugin_helper/formatter.rb)
• Duplicated block (15 lines × 2) (lib/fluent/compat/output.rb)
• …and 21 more

• 9 commits — 4 feature/other, 5 fixes

• test/command — 3 commits
• lib/fluent — 2 commits
• test/plugin — 2 commits
• .github/workflows — 1 commit
• test/helper.rb — 1 commit

• fix: buffer: fix stage_size leak when a staged chunk is unstaged (#5440)
• fix: test: fix unused ports outside the OS dynamic port range (#5451)
• fix: test_fluentd: fix flaky zero_downtime_restart from per-chunk log parsing (#5450)
• fix: test_out_buffer: fix flaky BufferOutputTest#write (#5443)
• fix: time: fix TimeFormatter returning nil for the Unix epoch (#5420)
• change: build(deps): bump the monthly-updates group with 5 updates (#5458)
• change: test_fluentd: read until EOF in assert_fluentd_fails_to_start (#5447)
• change: test_in_tail: relax too-tight timeouts in file-appended-after-start tests (#5446)
• change: test_plugin_generator: stub the Apache License download (#5461)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

fluent/fluentd was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.