zendesk/ruby-kafka

This system is the ruby-kafka library, a Ruby client for interacting with Apache Kafka brokers. It provides core capabilities for producing and consuming messages, including asynchronous delivery, compression, and SSL/TLS support. The library also implements administrative APIs for cluster management and supports various SASL authentication mechanisms for secure connections.

The library now implements the Kafka protocol for a wide range of new and upgraded operations, enabling support for Kafka 0.10+ features and administrative tasks. This includes new request/response classes for transactional messaging (InitProducerID, AddPartitionsToTxn, AddOffsetsToTxn, EndTxn), consumer group coordination (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, FindCoordinator), and cluster administration (CreateTopics, CreatePartitions, DeleteTopics, AlterConfigs, DescribeConfigs, DescribeGroups, ListGroups). The Fetch API is upgraded to version 4 to support RecordBatches and ReadCommitted isolation levels, and the ListOffset API is upgraded to version 2. Additionally, an ApiVersions API is added to allow clients to discover supported broker versions, and the core Encoder/Decoder classes are enhanced with varint support for efficient serialization.

lib/kafka/protocol · high confidence

A new benchmark file has been added to measure the performance of Kafka message serialization, specifically focusing on the encoding and decoding of messages with string values and keys.

benchmarks · high confidence

The library now includes an \AsyncProducer\ that delivers messages in the background using a configurable queue with backpressure (raising \BufferOverflow\ when full) and optional automatic delivery thresholds or intervals. The underlying connection management has been refactored to use a \BrokerPool\ and \ConnectionBuilder\, centralizing broker connections and SASL authentication logic, while the \Cluster\ class now manages broker discovery and metadata refreshes. Additionally, compression support has been expanded to include Gzip, Snappy, LZ4, and Zstd codecs via a new \Compressor\ class.

lib/kafka · high confidence

The library now supports four additional SASL authentication mechanisms for connecting to Kafka brokers. Users can authenticate using AWS MSK IAM (including optional session tokens), GSSAPI (Kerberos) via the \gssapi\ gem, OAuth/OAuthBearer with a custom token provider, and SCRAM-SHA-256/512 with configurable mechanisms. These new classes are located in \lib/kafka/sasl/\ and integrate with the existing SASL framework, allowing users to specify the mechanism via the \sasl\_mechanism\ configuration option.

lib/kafka/sasl · high confidence

Added a set of example scripts in the \examples/\ directory to demonstrate various usage patterns of the Kafka client library. These include \simple-consumer.rb\ and \simple-producer.rb\ for basic message consumption and production, \consumer-group.rb\ for handling consumer groups with error recovery, \firehose-consumer.rb\ and \firehose-producer.rb\ for high-throughput scenarios using multiple threads and async producers, and \ssl-producer.rb\ for demonstrating SSL/TLS configuration with client certificates. The examples also show how to load environment variables using \dotenv\ and handle graceful shutdowns.

examples · high confidence

The ruby-kafka library is no longer actively developed and has been superseded by rdkafka-ruby; users are advised to migrate. Additionally, the project license has changed from MIT to Apache 2.0, and the gem name in installation instructions has been updated to ruby-kafka.

(repo-wide) · high confidence

The library now provides a structured hierarchy of exceptions for Kafka protocol errors, replacing generic error handling. All protocol-specific errors (such as OffsetOutOfRange, NotLeaderForPartition, and RebalanceInProgress) are now subclasses of Kafka::ProtocolError, which itself inherits from Kafka::Error. This allows users to catch specific Kafka protocol violations or handle all protocol errors collectively, improving reliability and debugging for applications interacting with Kafka brokers.

lib · high confidence

The development console (bin/console) now uses Pry instead of IRB, providing an enhanced interactive Ruby experience for experimentation. Additionally, the bin/setup script has been cleaned up by removing an unused comment placeholder, and a frozen\_string\_literal pragma has been added to bin/console for better string performance.

bin · high confidence

Added comprehensive test suite for Kafka client components; Added functional tests for Kafka client capabilities; Added fuzz test for consumer group message delivery; Added protocol unit tests for decoder, encoder, and message formats; Added test fixtures for client certificate chain validation.

The gem has been renamed from 'kafka' to 'ruby-kafka', requiring users to update their Gemfile references. The gemspec now explicitly lists runtime dependencies such as 'digest-crc' and a comprehensive set of development dependencies including 'dogstatsd-ruby' (\>= 4.0.0, \< 5.0.0), 'gssapi' (\>= 1.2.0), and various compression libraries like 'snappy', 'extlz4', and 'zstd-ruby'. A post-install message informs users that the library is no longer actively developed and has been superseded by rdkafka-ruby.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 66.

• Code Health 95
• Architecture 100
• Maturity 50
• Readiness 65
• Security 94

• 300 commits — 273 feature/other, 27 fixes

• lib/kafka — 119 commits
• (repo) — 84 commits
• (root) — 62 commits
• .circleci/config.yml — 10 commits
• spec/functional — 8 commits
• .github/workflows — 4 commits
• spec/async_producer_spec.rb — 2 commits
• spec/consumer_spec.rb — 2 commits
• spec/prometheus_spec.rb — 2 commits
• spec/spec_helper.rb — 2 commits
• .github/CODEOWNERS — 1 commit
• spec/fake_server.rb — 1 commit
• spec/offset_manager_spec.rb — 1 commit
• spec/sasl_authenticator_spec.rb — 1 commit
• spec/transaction_manager_spec.rb — 1 commit

• fix: #756: Fix logger again
• fix: Fix "@param tag has unknown parameter name"
• fix: Fix SSL authentication for ruby < 2.4.0
• fix: Fix getting last message time
• fix: Fix multiple [Producer name] tags on failures
• fix: Fixed gitignore
• fix: Fixed unexpected token - RESCUE
• fix: Merge pull request #706 from Elyahou/fix-record-timestamp
• fix: Merge pull request #742 from Elyahou/fix-ssl-verify
• fix: Merge pull request #746 from brettburley/fix-offset-handling
• fix: Merge pull request #762 from wishabi/fix-logger-again
• fix: Merge pull request #810 from jonmagic/fix-dependencies
• fix: Merge pull request #841 from Darhazer/fix-checking-last-message-time
• fix: Merge pull request #866 from stasCF/fix-send-offsets-to-txn
• fix: Merge pull request #897 from abicky/fix-yard-annotation
• fix: Merge pull request #989 from zendesk/nony/fix-fqdn
• fix: Revert client API change
• fix: Rubocop fixes
• fix: Rubocop fixes
• fix: Some fixes, changelog, readme
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

zendesk/ruby-kafka was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.