kafka4beam/kafka_protocol

This system is a Kafka protocol library for Erlang and Elixir that manages connections, encodes and decodes wire formats, and handles RPC primitives. It supports a wide range of Kafka features including SASL authentication, multiple compression codecs, and transactional operations, with protocol schemas generated from official definitions. The library includes tooling for local testing and benchmarking, ensuring compatibility with Kafka versions up to 4.0.0.

The scripts directory now includes Docker Compose configurations for running local Kafka clusters, supporting both Zookeeper-based setups (for versions 0.9 through 2.x) and KRaft mode (for versions 3.x and 4.x). A new setup script automates the provisioning of these environments, including topic creation and SASL/SSL credential configuration. Additionally, a new Erlang benchmark module and shell script have been added to measure produce request encoding performance, and a helper script for running EUnit tests has been introduced.

scripts · high confidence

This entry introduces the \kafka\_protocol\ library, providing basic Kafka connection management, wire format encode/decode functions, and RPC primitives for Erlang and Elixir applications. The library now requires users to explicitly add compression dependencies (such as \snappyer\, \lz4b\, or \ezstd\) to their own project configurations, as these are no longer bundled as hard dependencies. It includes code generation from Kafka protocol schemas, support for SASL authentication, and compatibility with Kafka versions up to 4.0.0.

(repo-wide) · high confidence

This change introduces the kafka\_protocol library for Erlang/Elixir, providing a complete implementation of the Kafka protocol. It includes core modules for connection management (kpro\_connection), broker discovery (kpro\_brokers), and batch encoding/decoding (kpro\_batch, kpro\_batch\_v01). The library supports multiple compression codecs (gzip, snappy, lz4, zstd) via a pluggable backend (kpro\_compress), handles SASL authentication (kpro\_auth\_backend), and exposes APIs for transactional operations, consumer group metadata decoding, and API version negotiation (kpro\_api\_vsn). The application structure is defined in kafka\_protocol.app.src, specifying dependencies on kernel, stdlib, ssl, and crc32cer.

src · high confidence

The library now includes a code generator (priv/kpro\_gen.escript) that reads the Kafka protocol definition (priv/kafka.bnf) to produce the schema module (src/kpro\_schema.erl) and associated API key and error code mappings. This update introduces support for the full range of Kafka API keys and error codes, including newer features like transactional operations, ACLs, and delegation tokens, ensuring the client can correctly encode and decode requests and responses for these protocols.

priv · high confidence

The code generator for Kafka protocol BNF files has been replaced with a new Java application (KafkaProtocolBnf.java) that writes directly to the output file. This tool now explicitly handles Kafka's compact arrays by using curly brackets in the generated BNF syntax, aligning with the Kafka documentation, and includes logic to generate API keys while discarding broker-only APIs.

_priv/kafka\_protocol\bnf · high confidence

The library's include directory has been restructured into four distinct header files to better organize the Kafka client definitions. kpro.hrl now defines the core request and response records, including a new no\_ack field for fire-and-forget requests. kpro\_error\_codes.hrl provides a comprehensive, generated list of Kafka error codes. kpro\_private.hrl consolidates internal constants, adding support for zstd compression, SCRAM-SHA-256/512 SASL authentication mechanisms, and timestamp type handling. kpro\_public.hrl defines the kafka\_message record, extending it with timestamp and headers fields to support newer Kafka message formats.

include · high confidence

Added new test suites covering API version negotiation, batch encoding (including large batches and multiple compression algorithms like zstd), connection handling (SSL, SASL, IPv6), fetch operations (including incremental fetch), consumer group lifecycle, produce requests (including pre-encoded batches and async send), list offsets (with leader epoch), and topic management. These tests ensure correct behavior across various Kafka versions and configuration scenarios.

test · high confidence

A new Gradle-based Java application has been added to generate Backus-Naur Form (BNF) specifications for the Kafka protocol. This tool, located in priv/kafka\_protocol\_bnf, introduces a dependency on the Apache Kafka clients library version 2.4.0 and is configured to output the generated BNF file directly.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 70 → 69 (-1.4)
• Rubric changed (rubric-2026.09.12 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 96 → 96 (-0.1)
• Architecture 100 → 100 (+0.0)
• Maturity 58 → 58 (+0.0)
• Readiness 68 → 63 (-5.4)
• Security 87 → 87 (+0.0)
• Event Sourcing 100 → 100 (+0.0)

• Coverage not measured — no coverage collector is wired up
• Documentation: no installation or build instructions (README.md)
• Documentation: no licence statement (README.md)
• Documentation: no usage examples (README.md)
• Off-boarding risk: anonymized user #1

• Duplicated block (12 lines × 2) (priv/kafka_protocol_bnf/KafkaProtocolBnf.java)
• No dependency lockfile committed (rebar.config)
• Off-boarding risk: anonymized user #1

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

kafka4beam/kafka_protocol was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.