zio/zio-kafka

This system is a ZIO-based client library for Apache Kafka, providing type-safe, effectful abstractions for producing and consuming messages. It supports advanced operational features including transactional messaging, configurable metrics and diagnostics, and secure SSL/TLS authentication. The library also includes a test kit with embedded Kafka support and comprehensive benchmarks to facilitate integration testing and performance validation.

9 changes.

This period focused on deeply integrating the library with the ZIO ecosystem by introducing ZIO-native Serde abstractions, configurable producer metrics, and transactional APIs. Significant architectural changes included refactoring the consumer runloop for thread safety and adding embedded Kafka support via a new testkit module. The work was supported by comprehensive test coverage, benchmarking, and the establishment of binary compatibility checks within the build infrastructure.

9 changes.

This period focused on establishing robust testing capabilities through the release of the zio-kafka-testkit module and extensive unit and integration tests for consumer internals and configuration validation. Concurrently, the project introduced JMH benchmarks to measure ZIO-Kafka performance and compare it against the native Apache Kafka client, alongside new example applications to demonstrate core features.

5 changes.

This period focused on enhancing observability and reliability by introducing diagnostic event infrastructure for producers and exposing consumer metrics via zio-metrics. Significant effort was also dedicated to improving test coverage through comprehensive unit tests for the producer and admin client, alongside implementing graceful shutdown mechanisms for consumer subscriptions.

New JMH benchmarks have been added to measure the performance of ZIO-Kafka consumer and producer operations. The consumer benchmarks evaluate throughput with and without offset commits, while the producer benchmarks cover sequential and parallel single-record and chunk-based sending, providing users with standardized metrics to assess library performance.

zio-kafka-bench/src/main/scala/zio/kafka/bench · high confidence

A new example application has been added to help users manually test and understand zio-kafka features. It includes a basic consumer example (Main.scala) with SLF4J logging integration, a README example (ReadmeExample.scala) demonstrating both producer and consumer usage including offset collection, and a transactional example (Transactional.scala) showing how to perform transactional producing with safe commits. A Logback configuration (logback.xml) is also provided to set up appropriate logging levels for Kafka and ZIO components.

zio-kafka-example/src/main · high confidence

New JMH benchmarks have been added to compare the performance of the native Apache Kafka Java client against ZIO-Kafka. The suite includes \KafkaClientBenchmarks\ for the native client (both auto-subscribe and manual partition assignment modes) and \ZioKafkaBenchmarks\ for ZIO-Kafka (both topic-based and manual partition subscription modes), allowing users to evaluate throughput and latency differences between the two implementations.

zio-kafka-bench/src/main/scala/zio/kafka/bench/comparison · high confidence

The library now includes a new \Diagnostics\ trait and a \ConcurrentDiagnostics\ implementation within the \zio-kafka.diagnostics\ package. This introduces a callback interface for emitting diagnostic events, allowing users to hook into producer lifecycle and performance data. The \ConcurrentDiagnostics\ helper ensures these events are processed in a separate fiber via an unbounded queue, guaranteeing that diagnostic emission does not block the main Kafka operations, and supports a final event upon cleanup.

zio-kafka/src/main/scala/zio/kafka/diagnostics · high confidence

Users can now customize how producer metrics are collected by providing a custom \ProducerMetricsObserver\ via \ProducerSettings.withMetricsObserver\, with a default \ZioMetricsProducerMetricsObserver\ implementation available for ZIO metrics integration. The library introduces a new \Transaction\ trait and \TransactionImpl\ for managing transactional producer operations, including offset batching and abort handling. Additionally, a \KafkaCredentialStore\ is added to simplify SSL/TLS configuration using PEM strings, and a \ByteRecord\ type alias is exposed for convenience.

zio-kafka/src/main/scala/zio/kafka/producer · high confidence

A new \ZioMetricsConsumerMetricsObserver\ implementation has been added to the consumer metrics module, enabling the collection of consumer-side observations (such as poll, commit, rebalance, and runloop metrics) using the zio-metrics library. This change makes consumer metric data available for monitoring and observability, complementing the existing producer metric collection capabilities.

zio-kafka/src/main/scala/zio/kafka/consumer/metrics · high confidence

The repository has been initialized with the ZIO Kafka library (version 3.6.0), including a comprehensive README with usage examples, a Contributor License Agreement (CLA), and configuration files for the build system (SBT), code formatting (Scalafmt), and static analysis (Scalafix).

(repo-wide) · high confidence

The \zio-kafka-testkit\ module has been introduced (renamed from \zio-kafka-test-utils\ with a base package change to \zio.kafka.testkit\) to provide utilities for testing Kafka applications. This includes a \ZIOSpecWithKafka\ trait for running tests against a shared embedded Kafka cluster, helper utilities in \KafkaTestUtils\ for creating producers and transactional producers, and a \KafkaRandom\ trait for generating unique topics, groups, and client IDs to ensure test isolation. The module also ships with necessary SSL keystore and truststore resources for secure testing.

zio-kafka-testkit · high confidence

Added the \zio-kafka-testkit\ module and a new \zio-kafka-example\ module to simplify testing. The testkit provides \Kafka\ ZLayers for spinning up embedded Kafka brokers, including support for SASL and SSL configurations, allowing users to run integration tests against a real Kafka instance without external infrastructure. The example module demonstrates how to integrate this embedded Kafka service into a ZIO application.

repository · high confidence

The internal consumer architecture has been restructured to improve thread safety and isolation. The Kafka consumer runloop now executes on a dedicated single-threaded executor (named 'zio-kafka-runloop-thread-N') via the new RunloopExecutor, ensuring the runloop is isolated from other ZIO effects. Access to the underlying Java Kafka consumer is now strictly controlled: general access uses a Semaphore to prevent concurrent modification, while the runloop and rebalance listener have specific access paths (runloopAccess and rebalanceListenerAccess) defined in the new ConsumerAccess class. Additionally, a SameThreadRuntimeLayer is provided for internal operations that must run on the calling thread, though users should note that some ZIO operations like timeouts may still shift fibers to other threads.

zio-kafka/src/main/scala/zio/kafka/consumer/internal · high confidence

A new \StreamControl\ trait has been introduced in the consumer package to manage the lifecycle of a Kafka subscription stream. This addition allows users to explicitly stop fetching data for all partitions associated with a subscription via the \end\ method, ensuring the stream terminates gracefully. The trait also provides a \map\ operation to transform the stream while preserving the ability to control its lifecycle, enabling cleaner resource management and shutdown behavior for single subscriptions.

zio-kafka/src/main/scala/zio/kafka/consumer · high confidence

The \zio-kafka\ serde module now provides ZIO-based \Serializer\, \Deserializer\, and \Serde\ traits that replace the previous Kafka-native interfaces. These new abstractions allow serialization and deserialization logic to be expressed as ZIO effects (\RIO\), enabling seamless integration with ZIO's environment and error handling. The module includes built-in support for common types (String, Int, Long, etc.) via the \Serdes\ trait, offers utility methods for mapping and blocking execution, and provides factory methods to wrap existing Kafka serializers and deserializers.

zio-kafka/src/main/scala/zio/kafka/serde · high confidence

The project build has been upgraded to SBT 1.13.0 and the ZIO SBT ecosystem plugins to version 0.8.0, replacing previous CI and website tooling. Additionally, the build now includes the sbt-mima-plugin (v1.2.0) with a new configuration file (MimaSettings.scala) that enforces binary compatibility checks against a specified previous version, filtering out internal API changes in consumer and diagnostics packages to prevent breaking public contracts.

project · high confidence

Added benchmark suite and comprehensive producer tests; Added comprehensive admin client test suite; Added integration tests for example consumer, producer, and service components; Added producer unit tests and test infrastructure; Added test coverage for Serde, Serializer, Deserializer, and KafkaCredentialStore; Added tests for SSL endpoint validation resilience; Added unit tests for internal consumer components; Added unit tests for queue size-based fetch strategies; Added validation tests for ConsumerSettings.

This release updates the core ZIO Kafka library and its test kit to use ZIO 2.1.26, Kafka clients 4.3.1, and embedded-kafka 4.3.1. It also upgrades the logging infrastructure to logback-classic 1.6.3 and zio-logging-slf4j 2.5.3, ensuring compatibility with the latest ZIO ecosystem and Kafka features.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 60.

• Code Health 88
• Architecture 99
• Maturity 52
• Readiness 73
• Security 53

• 300 commits — 287 feature/other, 13 fixes

• (root) — 102 commits
• zio-kafka/src — 42 commits
• project/plugins.sbt — 35 commits
• .github/workflows — 33 commits
• project/build.properties — 31 commits
• zio-kafka-test/src — 22 commits
• docs/index.md — 7 commits
• docs/migrating-to-zio-kafka-3.md — 5 commits
• zio-kafka-bench/src — 5 commits
• zio-kafka-testkit/src — 4 commits
• docs/avoiding-chunk-breakers.md — 3 commits
• (repo) — 2 commits
• docs/consuming-kafka-topics-using-zio-streams.md — 2 commits
• docs/consumer-tuning.md — 1 commit
• docs/example-of-consuming-producing-and-committing-offsets.md — 1 commit
• docs/partition-assignment-and-offset-retrieval.md — 1 commit
• docs/preventing-duplicates.md — 1 commit
• docs/serialization-and-deserialization.md — 1 commit
• zio-kafka-bench/README.md — 1 commit
• zio-kafka-example/src — 1 commit

• fix: Fix chunk-breaking docs (#1513)
• fix: Fix consumer hang when Runloop crashes with empty dataQueue (#1707)
• fix: Fix embedded-kafka dependencies for scala 3 (#1607)
• fix: Fix flaky consumer group admin tests (#1729)
• fix: Fix flaky resume test (#1737)
• fix: Fix rebalance coordinator spec (#1485)
• fix: Fix scala-steward flow (#1418)
• fix: Fix scalafmt corrupted class path issue (#1687)
• fix: Fix small stuff (#1577)
• fix: Fix: removeSubscription hangs after Runloop crashes (#1712)
• fix: Fix: removeSubscription hangs after Runloop crashes - followup (#1733)
• fix: Fix: removeSubscription hangs after Runloop crashes - followup 3 (#1734)
• fix: Revert "Update sbt to 2.0.7 (#1763)" (#1764)
• change: Add comment motivating use of lazy in publisher (#1527)
• change: Add config for transactional consuming (#1433)
• change: Add even more zio-kafka resources (#1415)
• change: Add features to readme (#1466)
• change: Add low level unit test for produceAsync (#1519)
• change: Add more zio-kafka resources (#1414)
• change: Add new zio-kafka course by Alvin Alexander (#1537)
• …and 280 more

• 0 containers · 1 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

zio/zio-kafka was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.