silviucpp/erlkaf

erlkaf is an Erlang Kafka client library that wraps librdkafka to provide high-performance message production and consumption. It supports modern authentication methods like SASL/OAuth2, consumer group management, and local persistent disk queuing for reliability. The system is designed with a refactored C++ NIF architecture to efficiently handle events and configuration, including comprehensive logging and statistics callbacks.

This entry introduces the erlkaf Erlang Kafka client, bundling librdkafka v2.15.1 and adding support for OTP 29 and OpenSSL 3.x. The library provides comprehensive configuration options for both producers and consumers, including support for SASL/OAuthBearer authentication, Dead Letter Queue callbacks, and local persistent disk queues. It also includes a new callback trigger system that uses a single thread for all consumers and producers, improving efficiency over previous multi-threaded polling models.

(repo-wide) · high confidence

This release adds consumer group management via the new \create\_consumer\_group/5\ API, supporting both callback-driven (\erlkaf\_consumer\) and explicit polling (\erlkaf\_poll\_consumer\) consumption modes. It introduces a local persistent disk queue (using the \esq\ library) to buffer messages when the Kafka broker queue is full, configurable via \queue\_buffering\_overflow\_strategy\. Additionally, it adds support for SASL/OAuth2 authentication with token refresh callbacks, exposes client statistics via a \stats\_callback\, and provides a utility to map internal error codes to readable atoms.

src · high confidence

The C source layer has been restructured from a monolithic \erlkaf\_rd\_kafka\ module into distinct \erlkaf\_producer\ and \erlkaf\_consumer\ components, introducing a new \QueueCallbacksDispatcher\ to manage Kafka queue events on a dedicated thread. This change adds native support for OAuth2 token refresh callbacks and message headers in delivery reports, while updating the build system to link against OpenSSL 3, LZ4, and Zstd.

_c\src · high confidence

The erlkaf headers have been updated to support a significantly broader set of librdkafka configuration options, including new SSL settings (such as keystore and sigalgs), consumer group management parameters (like group\_instance\_id and max\_poll\_interval\_ms), and producer features (such as idempotence and gapless guarantees). The erlkaf\_msg record has been extended to include headers and timestamp fields, and new types for partition assignment strategies and overflow handling have been introduced. Additionally, the private header has migrated logging macros from the lager library to the standard Erlang/OTP logger, aligning with modern OTP practices.

include · high confidence

Added new test files (\benchmark\_consumer.erl\, \benchmark\_producer.erl\, \test\_consumer.erl\, \test\_producer.erl\, \test\_consume\_produce.erl\) and a \sys.config\ to provide benchmarking scripts and usage examples for both the native erlkaf client and the brod-based client. These tests demonstrate how to configure and run producers and consumers, including handling delivery reports, stats callbacks, and message dispatch modes.

test · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 63 → 66 (+3.1)
• Rubric changed (rubric-2026.09.12 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 98 → 98 (-0.1)
• Architecture 100 → 100 (+0.0)
• Maturity 56 → 56 (+0.0)
• Readiness 55 → 60 (+4.6)
• Security 67 → 80 (+13.3)
• Event Sourcing 100 → 100 (+0.0)

• Coverage not measured — no coverage collector is wired up
• Documentation: no installation or build instructions (README.MD)
• Documentation: no usage examples (README.MD)

• erlkaf_consumer.handle_failed_message (cognitive 26) (src/erlkaf_consumer.erl)
• erlkaf_consumer.process_events_batch (cognitive 24) (src/erlkaf_consumer.erl)
• erlkaf_consumer.process_events_one_by_one (cognitive 22) (src/erlkaf_consumer.erl)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

silviucpp/erlkaf was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.