apache/kafka

This system is a distributed event streaming platform that manages high-throughput, low-latency data pipelines. It provides a Java-based client library for producing and consuming messages with advanced features like transactions, share groups, and pluggable security mechanisms. The platform also includes a connector framework for integrating external systems and a comprehensive administrative API for cluster management, authorization, and configuration.

85 changes.

This period focused on the architectural shift from ZooKeeper to the KRaft metadata quorum, which required removing the legacy Scala-based server, client, and tooling code. The team simultaneously modernized the Java client API, introduced the Share Consumer, and cleaned up the codebase by eliminating obsolete integrations and test suites.

105 changes.

This period focused on significantly expanding the Kafka client's security capabilities by introducing pluggable authentication mechanisms including SCRAM, PLAIN, and OAuth/OIDC, alongside a new ConfigProvider interface for dynamic configuration. Concurrently, the Kafka Connect API was substantially extended with new interfaces for data models, transformations, and health monitoring, while the AdminClient received a major internal refactoring to support complex administrative operations.

90 changes.

This period was defined by the extensive migration to the KRaft metadata quorum, introducing a new metadata module and refactoring the server architecture to support a controller-based cluster without ZooKeeper. Concurrently, the codebase underwent significant modularization, splitting the monolithic core into dedicated modules for transaction, group, and share coordination, while introducing pluggable APIs for authorization, client quotas, and Connect predicates.

22 changes.

This period focused on expanding test coverage and infrastructure, introducing new classes for cluster management, selective test execution, and mock implementations. Simultaneously, the project refactored internal package structures to restrict visibility of non-public APIs and enhanced automated checks to prevent internal API leaks.

A new Basic Auth REST extension is introduced for Kafka Connect, providing a reference implementation for authenticating incoming REST requests using JAAS. The change adds three new classes: \BasicAuthSecurityRestExtension\, which registers the authentication filter; \JaasBasicAuthFilter\, which handles the actual JAAS login and sets the security context; and \PropertyFileLoginModule\, a sample LoginModule that authenticates against a plaintext properties file. This allows users to enable basic authentication for the Connect REST API by configuring the extension class and providing a JAAS configuration.

connect/basic-auth-extension/src/main/java · high confidence

The \docker\ directory now includes a comprehensive set of Python scripts (\docker\_build\_test.py\, \docker\_release.py\, \docker\_official\_image\_build\_test.py\) and supporting utilities (\common.py\) to build, test, and push both JVM and native GraalVM-based Docker images. These scripts support building multi-architecture images via \docker buildx\, running sanity tests, and promoting release candidates. The change also adds the \docker/README.md\ documentation and example configurations, providing users with the tools and guidance needed to build and manage Apache Kafka Docker images locally or via GitHub Actions.

docker · high confidence

The raft module now includes a standalone test server for performance testing and development, accessible via the new \bin/test-kraft-server-start.sh\ script. This addition is accompanied by a \README.md\ with setup instructions, a \.gitignore\ for build artifacts, and dedicated configuration files (\kraft.properties\, \kraft-log4j2.yaml\) to support running single or multi-node KRaft quorums locally.

raft · high confidence

The Kafka Streams Java quickstart is now available as a Maven archetype, allowing users to quickly scaffold a new project. The generated template includes three example applications—WordCount, Pipe, and LineSplit—along with a log4j2 configuration file and the necessary Maven archetype metadata and test resources.

streams/quickstart/java · high confidence

New classes have been added to the \org.apache.kafka.common.security.scram\ package to enable Salted Challenge Response Authentication Mechanism (SCRAM) for securing Kafka clusters. This includes \ScramCredential\ to hold credential data, \ScramCredentialCallback\ and \ScramExtensionsCallback\ for handling authentication callbacks, and \ScramLoginModule\ to manage the login and authentication process using the SCRAM algorithm.

clients/src/main/java/org/apache/kafka/common/security/scram · high confidence

The Kafka client library now includes a new \TimestampType\ enum in the \org.apache.kafka.common.record\ package, defining constants for message timestamp types (NoTimestampType, CreateTime, LogAppendTime) with their corresponding IDs and names. Additionally, package-level Javadoc has been added to the \record\ package to describe its role in providing utility components for Kafka records.

clients/src/main/java/org/apache/kafka/common/record · high confidence

A large number of new shell scripts are added to the bin directory, providing a consistent way to invoke Java-based command-line tools (e.g., kafka-topics.sh, kafka-consumer-groups.sh, kafka-cluster.sh). The existing kafka-run-class.sh script is significantly updated to support these tools, including adding a -daemon flag, setting default heap and log4j2 options, and updating the classpath logic to include the new 'tools' and 'connect' build directories. Additionally, several legacy scripts (such as kafka-consumer-shell.sh and kafka-producer-shell.sh) are removed, and the kafka-console-consumer.sh script is updated to use the new tools module.

bin · high confidence

The Kafka client library now includes the internal implementation for the SASL/PLAIN authentication mechanism. This change introduces three new classes in the \org.apache.kafka.common.security.plain.internals\ package: \PlainSaslServer\ handles the SASL protocol logic, \PlainServerCallbackHandler\ manages the callback interaction with JAAS configurations, and \PlainSaslServerProvider\ registers the server factory with the Java Security provider. This enables the Kafka client to support PLAIN authentication for server-side SASL connections.

clients/src/main/java/org/apache/kafka/common/security/plain/internals · high confidence

The jmh-benchmarks module was introduced to provide a dedicated space for micro-benchmarks using the Java Microbenchmark Harness (JMH). This includes new benchmark classes for the StandardAuthorizer (AuthorizerBenchmark, StandardAuthorizerUpdateBenchmark) to measure ACL evaluation and update performance, as well as benchmarks for client-side and server-side partition assignors (ClientSideAssignorBenchmark, ServerSideAssignorBenchmark, CurrentAssignmentBuilderBenchmark) to evaluate the performance of consumer group and share group rebalancing logic. The module also includes utility classes and a shell script to facilitate running these benchmarks outside of the standard Gradle build process.

jmh-benchmarks · high confidence

Added new public Java classes, DelegationToken and TokenInformation, to the Kafka clients library, providing the data structures required to represent and manage delegation tokens for authentication and authorization.

clients/src/main/java/org/apache/kafka/common/security/token/delegation · high confidence

The Kafka client now includes a PLAIN authentication mechanism for SASL, allowing users to secure connections using plaintext credentials. This change introduces the PlainAuthenticateCallback to handle password verification and a PlainLoginModule to manage subject credentials, enabling PLAIN as a configurable SASL mechanism.

clients/src/main/java/org/apache/kafka/common/security/plain · high confidence

A comprehensive set of Windows batch (.bat) scripts has been added to the bin/windows directory, providing Windows-native equivalents for the majority of Kafka command-line tools. This includes scripts for running the Kafka broker (kafka-server-start.bat, kafka-server-stop.bat), managing topics, configurations, ACLs, consumer groups, and various administrative tasks (kafka-topics.bat, kafka-configs.bat, kafka-acls.bat, kafka-consumer-groups.bat, etc.). Additionally, utility and performance testing scripts are now available for Windows users, such as kafka-run-class.bat, kafka-console-producer.bat, kafka-console-consumer.bat, kafka-producer-perf-test.bat, and others. This change ensures that Windows users have full access to the Kafka tooling suite without relying on Unix-style shell scripts.

bin/windows · high confidence

Introduced the \DelegationTokenCache\ class to manage the lifecycle of delegation tokens, including caching token information, HMAC-to-token-ID mappings, and SCRAM credentials. Added \DelegationTokenCredentialCallback\ to handle token owner and expiry timestamp data. These internal components support the broader KIP-368 feature for periodic re-authentication of SASL connections.

clients/src/main/java/org/apache/kafka/common/security/token/delegation/internals · high confidence

The Kafka client library introduces a new internal package, \org.apache.kafka.common.security.oauthbearer.internals.expiring\, containing three new classes: \ExpiringCredential\ (an interface for credentials with expiration times), \ExpiringCredentialRefreshConfig\ (configuration for refresh timing and jitter), and \ExpiringCredentialRefreshingLogin\ (a base class that manages background threads to automatically refresh expiring credentials). These changes provide the internal infrastructure for automatic credential refresh, supporting the OAuth/OIDC authentication flow.

clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/expiring · high confidence

Added new internal classes in the Kafka clients metrics package: a utility class for time conversion and tag creation, an implementation of the PluginMetrics interface for managing plugin-specific metrics and sensors, and a suite for managing integer gauges. These changes support making producer and consumer plugins monitorable by providing the underlying infrastructure for tracking metrics and sensors within the client library.

clients/src/main/java/org/apache/kafka/common/metrics/internals · medium confidence

The api-checker tool now automatically detects internal API usage and validates the public API surface. This includes a new cascade validator that checks for internal type leaks in public method signatures, a javadoc consistency validator to ensure all public classes have documentation, and a plugin developer API usage scanner to catch internal references in compiled bytecode. These checks are exposed via the PublicApiChecker facade, which coordinates the scanning and validation processes.

api-checker/core, api-checker/gradle-plugins, api-checker/maven-plugin · high confidence

The Kafka Java client now includes the full internal implementation for SASL/OAUTHBEARER authentication. This adds the \OAuthBearerClientInitialResponse\ to parse and build client handshake messages, \OAuthBearerSaslClient\ and \OAuthBearerSaslServer\ to handle the SASL protocol state machines for client and server respectively, and \OAuthBearerRefreshingLogin\ to manage automatic token refresh. The \OAuthBearerSaslClientProvider\ and \OAuthBearerSaslServerProvider\ register these implementations with the Java Security framework, enabling end-to-end OAuth 2.0 bearer token authentication for Kafka clients and brokers.

clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals · high confidence

The repository was initialized with essential project metadata and contributor information. This includes the addition of the \.asf.yaml\ file to configure GitHub notifications and merge settings, a \.gitignore\ file to exclude build artifacts and IDE directories, and an \AGENTS.md\ file providing guidance for automated agents and AI tools. Additionally, a \CONTRIBUTING.md\ file was added to outline the process for submitting pull requests and handling AI-generated contributions, while the \vagrant\ directory was introduced to support local development environments.

(repo-wide) · high confidence

The new \AbstractCoordinator\ class has been added to the \clients\ module to handle consumer group coordination logic, including heartbeat management, group joining, and state transitions. This refactoring extracts the core coordination responsibilities from the existing consumer implementation into a reusable base class, enabling shared group management features for both the classic and async consumers.

clients/src/main/java/org/apache/kafka/clients/consumer/internals · high confidence

Adds the \org.apache.kafka.connect.header\ package containing the \Header\ and \Headers\ interfaces and their implementations (\ConnectHeader\, \ConnectHeaders\). This provides a new API for attaching typed metadata to Connect records, allowing users to add, retrieve, and manipulate headers with associated schemas.

connect/api/src/main/java/org/apache/kafka/connect/header · high confidence

A new \CoordinatorPartitionWriter\ implementation is added to the group coordinator package. This class implements the \PartitionWriter\ interface, enabling the group coordinator to write records directly to the leader replica of a partition. It handles transaction verification, high watermark updates, and append operations, effectively bridging the group coordinator with the underlying storage layer for transactional writes.

core/src/main/scala/kafka/coordinator/group · high confidence

A new \KafkaDockerWrapper\ component has been added to the core module, providing a Java-based entry point for managing the Kafka Docker image. This wrapper handles the 'setup' and 'start' commands, managing configuration file preparation (server, log4j2, and tools log4j2) and invoking the \StorageTool\ to format storage with a specified cluster ID. It also integrates with the native image build process, ensuring that environment variables and mounted configurations are correctly merged into the final configuration files used to boot the Kafka server.

core/src/main/scala/kafka/docker · high confidence

Added RuntimeLoggerManager in the kafka.server.logger package to manage runtime changes to slf4j settings. This new component handles validation and application of log level configurations for specific nodes, supporting SET and DELETE operations while preventing unsupported operations like APPEND or SUBTRACT. It enforces authorization checks and validates that logger names exist and use supported log levels, providing a structured way to modify logging behavior at runtime.

core/src/main/java/kafka/server/logger · high confidence

The Kafka client and server now support SCRAM-SHA-256 and SCRAM-SHA-512 as SASL authentication mechanisms. This change adds the internal implementation classes for the SCRAM protocol, including message formatting, credential handling, and SASL client/server providers. Users can now configure their Kafka clients and brokers to use SCRAM-SHA-256 or SCRAM-SHA-512 for secure, password-based authentication.

clients/src/main/java/org/apache/kafka/common/security/scram/internals · high confidence

Added a new Vagrant-based setup for running a local Kafka cluster, including a comprehensive README, provisioning scripts (base.sh, broker.sh, zk.sh), and automation scripts for building the base box and managing the cluster lifecycle. The configuration supports running ZooKeeper and Broker nodes on local VMs or AWS EC2 instances, with support for parallel cluster bringup and JMX monitoring.

vagrant · high confidence

Added the ClientTelemetryState enum to define and validate state transitions for the client telemetry system, ensuring that the client only moves between valid states such as SUBSCRIPTION\_NEEDED, PUSH\_NEEDED, and TERMINATED.

clients/src/main/java/org/apache/kafka/common/telemetry · high confidence

Added new classes (JaasConfig, JaasContext, JaasUtils) in the Kafka clients security package to support dynamic JAAS configuration for SASL authentication. This allows SASL configuration to be specified directly via the SASL\_JAAS\_CONFIG property, enabling per-listener and per-mechanism security settings. The implementation includes parsing of JAAS configuration strings, validation of login modules against allow/deny lists, and support for both client and server contexts. The deprecated system property for disallowed login modules is retained but marked for removal.

clients/src/main/java/org/apache/kafka/common/security · high confidence

Added a new internal package \org.apache.kafka.common.telemetry.internals\ containing the core classes for client-side telemetry. This includes \ClientTelemetryReporter\ to manage the lifecycle of collecting and pushing metrics to the broker, \ClientTelemetryProvider\ to map client configuration (like group ID, transactional ID) to telemetry resource labels, and \ClientTelemetryEmitter\ to filter and emit metrics. The change also introduces supporting classes such as \KafkaMetricsCollector\ to gather Kafka-specific metrics, \MetricKey\ and \MetricKeyable\ for metric identification, \MetricNamingStrategy\ for canonical naming, and utility classes like \ClientTelemetryUtils\ for error handling and compression type selection. These components work together to enable the client to subscribe to telemetry subscriptions from the broker and push collected metrics.

clients/src/main/java/org/apache/kafka/common/telemetry/internals · high confidence

Added new classes in the \org.apache.kafka.common.feature\ package (\BaseVersionRange\, \Features\, \SupportedVersionRange\) to represent feature version ranges. These classes provide the underlying data structures for the feature versioning system, allowing the client to track supported and finalized feature versions.

clients/src/main/java/org/apache/kafka/common/feature · high confidence

Added new classes to the \org.apache.kafka.common.security.kerberos\ package to support SASL/GSSAPI authentication. This includes \KerberosClientCallbackHandler\ for handling SASL callbacks, \KerberosLogin\ for managing ticket renewal, \KerberosError\ for handling transient Kerberos errors, \KerberosName\ and \KerberosRule\ for parsing and applying principal name translation rules, and \KerberosShortNamer\ for mapping Kerberos principals to local OS users. Custom exception classes \BadFormatString\ and \NoMatchingRule\ are also introduced to handle specific error conditions during name parsing and rule application.

clients/src/main/java/org/apache/kafka/common/security/kerberos · high confidence

The OAuthBearer security package now includes new public classes to handle JWT lifecycle and validation: \JwtRetriever\ and \JwtValidator\ interfaces define the core abstractions; \DefaultJwtRetriever\ and \DefaultJwtValidator\ provide factory-style delegation; \ClientJwtValidator\ and \BrokerJwtValidator\ implement specific validation logic for clients and brokers respectively; \ClientCredentialsJwtRetriever\ and \JwtBearerJwtRetriever\ implement HTTP-based retrieval for different grant types; \FileJwtRetriever\ handles static file-based tokens; and \OAuthBearerExtensionsValidatorCallback\ supports SASL extension validation. These changes restructure and expand the OAuthBearer mechanism's capabilities.

clients/src/main/java/org/apache/kafka/common/security/oauthbearer · high confidence

Added a new set of public interfaces in the \org.apache.kafka.server.telemetry\ package to support server-side collection of client telemetry metrics. The new \ClientTelemetryExporter\ and \ClientTelemetryExporterProvider\ interfaces provide enhanced context, including the client's push interval and authorization details, allowing for better metric lifecycle management. Concurrently, the legacy \ClientTelemetry\ and \ClientTelemetryReceiver\ interfaces are marked as deprecated since version 4.2.0 and are scheduled for removal in Kafka 5.0.0.

clients/src/main/java/org/apache/kafka/server/telemetry · high confidence

The coordinator-common module now includes a new set of shared runtime classes that provide a common foundation for coordinator implementations. This includes the CoordinatorExecutor interface and its implementation for scheduling and managing asynchronous tasks, a CoordinatorLoader interface and implementation for reading and replaying records from a partition, and a new CoordinatorRecord class to represent key-value pairs in the coordinator log. Additionally, the change introduces a background thread pool executor for metrics reporting, a base interface for coordinator events, and a helper class for standardizing exception handling and error mapping.

coordinator-common/src/main · high confidence

The Kafka consumer client now uses a structured event system to manage internal operations, replacing ad-hoc callback and blocking mechanisms with a unified event queue. This change introduces new classes in the \internals.events\ package, including \ApplicationEvent\ and \BackgroundEvent\ hierarchies, which allow the application thread and the background network thread to communicate via a shared queue. This enables more robust handling of asynchronous operations like polling, committing offsets, and managing subscriptions, ensuring that background tasks such as rebalance listeners and position updates are processed consistently without blocking the application thread.

clients/src/main/java/org/apache/kafka/clients/consumer/internals/events · high confidence

The Kafka metadata shell is now available as a new tool for inspecting and navigating the Kafka metadata tree. Users can run the shell interactively to explore metadata nodes, change directories, list contents, search for specific nodes, and view file contents. The shell supports both interactive mode with command history and tab completion, and non-interactive mode for scripting. Key commands include \cat\ to display metadata file contents, \cd\ to change the working directory, \find\ to search the metadata tree, \ls\ to list nodes, \history\ to view past commands, and \help\ for documentation. This provides a powerful way to debug and understand the internal state of a Kafka cluster's metadata without needing to parse raw snapshot files manually.

shell · high confidence

The share-coordinator module is introduced to manage the state of share groups, including the implementation of the ShareCoordinator interface, the ShareCoordinatorService, and the ShareCoordinatorShard. The module adds configuration options for the share-group state topic, such as partition count, replication factor, and segment size. It also includes a PersisterStateBatchCombiner for merging state batches and a ShareCoordinatorOffsetsManager to track redundant offsets. This change adds the core infrastructure for handling share group state, including record serialization, metadata updates, and periodic snapshotting.

share-coordinator · high confidence

Added a new set of classes in the \clients\ module to support unsecured JWT parsing and validation for the OAuthBearer SASL mechanism. This includes \OAuthBearerUnsecuredJws\ for parsing, \OAuthBearerUnsecuredLoginCallbackHandler\ for generating test tokens, and \OAuthBearerUnsecuredValidatorCallbackHandler\ for validating them. The update also introduces supporting utilities (\OAuthBearerValidationUtils\, \OAuthBearerScopeUtils\) and specific exception types (\OAuthBearerConfigException\, \OAuthBearerIllegalTokenException\, \OAuthBearerValidationResult\) to handle configuration and validation errors. These components allow developers to use simple, unsecured tokens for local testing and development without requiring a real authorization server.

clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/unsecured · high confidence

A new internal framework for the Admin client has been introduced, centered around the \AdminApiDriver\ and \AdminApiHandler\ interfaces. This refactors the internal implementation of the Admin client to use a two-stage request workflow (Lookup and Fulfillment) that better handles multi-stage requests like partition leader lookups or group coordinator routing. The new structure includes \AdminApiFuture\ for managing asynchronous results, \AdminApiLookupStrategy\ for discovering broker targets, and specialized handlers like \AbortTransactionHandler\ to manage specific API requests. This change simplifies the handling of batched and unbatched admin requests, improving how the client manages metadata caching, retries, and error handling for administrative operations.

clients/src/main/java/org/apache/kafka/clients/admin/internals · high confidence

The broker now includes the primary implementation classes for the Share Group feature, including SharePartition to manage partition state and record acquisition, DelayedShareFetch to handle delayed fetch requests, and supporting utilities like ShareFetchUtils and ReplicaManagerLogReader. These components enable the broker to process share fetch requests, manage in-flight records, and interact with the log reader and metadata cache, forming the backend foundation for the new share consumer capability.

core/src/main/java/kafka/server/share · high confidence

The Kafka Connect API introduces a new \ConnectSchema\ implementation and supporting classes (\Schema\, \SchemaBuilder\, \Struct\, \Field\, \SchemaProjector\, \SchemaAndValue\) that define the core data model. This change adds logical type support for \Date\, \Time\, \Timestamp\, and \Decimal\, each with dedicated conversion utilities. The \Struct\ class provides a typed container for structured records, while \SchemaProjector\ enables safe projection between compatible schemas. These classes form the foundation for schema validation, value conversion, and type promotion within the Connect framework.

connect/api/src/main/java/org/apache/kafka/connect/data · high confidence

The Connect worker now automatically discovers and registers built-in components through Java service loader files. This includes converters (StringConverter, JsonConverter), header converters (SimpleHeaderConverter, JsonConverter), source connectors (FileStreamSourceConnector, MirrorCheckpointConnector, MirrorHeartbeatConnector, MirrorSourceConnector), sink connectors (FileStreamSinkConnector), transformations (Cast, DropHeaders, ExtractField, Filter, Flatten, HeaderFrom, HoistField, InsertField, InsertHeader, MaskField, RegexRouter, ReplaceField, SetSchemaMetadata, TimestampConverter, TimestampRouter, ValueToKey), and predicates (HasHeaderKey, RecordIsTombstone, TopicNameMatches). Users no longer need to manually configure these standard components in their worker configuration.

(repo-wide) · high confidence

The \Bytes\ class is now part of the public API, allowing users to wrap and manage byte arrays in Kafka Streams and serialization contexts. Additionally, the \Time\ interface and \Timer\ helper class are exposed in the \org.apache.kafka.common.utils\ package, enabling developers to abstract clock time for testing and manage timeouts in blocking operations, though their public API status is preliminary pending a future KIP.

clients/src/main/java/org/apache/kafka/common/utils · high confidence

The \tools\ module has been significantly expanded to include a wide range of command-line utilities and administrative commands, such as \AclCommand\, \BrokerApiVersionsCommand\, and \ClientCompatibilityTest\. This change consolidates these tools into a single module, moving them from the core or client modules to improve modularity and separation of concerns. Users will find these administrative and compatibility testing utilities more easily accessible within the \tools\ package.

tools · high confidence

The examples directory was modernized to use the current Java client API, replacing the legacy Scala-based and ZooKeeper-dependent code with the new KafkaConsumer and KafkaProducer implementations. This includes updating the standard producer-consumer demo to use the new API, adding a new exactly-once processing demo (KafkaExactlyOnceDemo) that demonstrates transactional processing, and introducing a new TransactionalClientDemo that showcases word-count processing with transactional guarantees. The old SimpleConsumerDemo and related utility classes were removed.

examples · high confidence

The Kafka client library introduces new Java annotations to clarify API stability and intended audience. \InterfaceStability\ marks public interfaces as Stable, Evolving, or Unstable, while \InterfaceAudience\ distinguishes between Public and Private classes. A \SuppressKafkaInternalApiUsage\ annotation allows controlled access to internal APIs. These annotations provide clearer compatibility guarantees and help enforce API boundaries.

clients/src/main/java/org/apache/kafka/common/annotation · high confidence

The AdminClient now exposes a new public Java interface, Admin, which serves as the primary entry point for administrative operations. This interface introduces methods for aborting transactions (AbortTransactionSpec, AbortTransactionOptions, AbortTransactionResult), adding Raft voters to the cluster (AddRaftVoterOptions, AddRaftVoterResult), and altering client quotas (AlterClientQuotasOptions, AlterClientQuotasResult). Additionally, a base class, AbstractOptions, is introduced to standardize timeout handling across all AdminClient options classes, and the legacy AdminClient class is retained for backward compatibility but marked as superseded by the new Admin interface.

clients/src/main/java/org/apache/kafka/clients/admin · high confidence

The Kafka client now supports loading configuration from files, directories, and environment variables. This is enabled by adding FileConfigProvider, DirectoryConfigProvider, and EnvVarConfigProvider to the service provider list, allowing users to reference these sources in their configuration.

clients/src/main/resources/META-INF · high confidence

The Kafka client library introduces a new \ConfigProvider\ interface in the \org.apache.kafka.common.config.provider\ package, enabling pluggable, late-binding of configuration values. This change adds three concrete implementations: \FileConfigProvider\ for loading properties from files, \DirectoryConfigProvider\ for loading properties from all files in a directory, and \EnvVarConfigProvider\ for reading configuration from environment variables. Each provider supports an allowlist of permitted paths or patterns to restrict access, and they are discoverable via Java's ServiceLoader mechanism.

clients/src/main/java/org/apache/kafka/common/config/provider · high confidence

The Kafka Connect API now includes new types to support exactly-once delivery guarantees and custom transaction boundaries. Source connectors can declare their exactly-once support via the new \ExactlyOnceSupport\ enum and the \SourceConnector.exactlyOnceSupport()\ method. Additionally, connectors can define their own transaction boundaries using the \ConnectorTransactionBoundaries\ enum and the \SourceConnector.canDefineTransactionBoundaries()\ method. The \TransactionContext\ interface and \SourceTask.transactionContext()\ method provide the runtime context needed to commit or abort transactions. Source tasks can also access the \OffsetStorageReader\ via \SourceTaskContext.offsetStorageReader()\.

connect/api/src/main/java/org/apache/kafka/connect/source · high confidence

A new \ConnectorClientConfigOverridePolicy\ interface and supporting \ConnectorClientConfigRequest\ class have been added to the Kafka Connect API. This introduces a pluggable mechanism for enforcing policies on overriding Kafka client configurations (such as SASL or optimization settings) on a per-connector basis. Users can now implement this interface to validate or modify client properties before connectors are started, with the framework automatically discovering implementations via the Java ServiceLoader mechanism.

connect/api/src/main/java/org/apache/kafka/connect/connector/policy · high confidence

Kafka Connect now exposes a new public API in the \org.apache.kafka.connect.rest\ package, allowing developers to register custom JAX-RS resources (such as filters, providers, and endpoints) via the \ConnectRestExtension\ interface. This change introduces the \ConnectRestExtension\ interface and its associated context (\ConnectRestExtensionContext\) to enable third-party plugins to extend the Connect REST API. The diff also adds the \ConnectPlugin\ and \Versioned\ interfaces in the \org.apache.kafka.connect.components\ package, which provide a standard contract for components to declare their version and configuration requirements, facilitating uniform discovery and introspection of these pluggable components.

connect/api/src/main/java/org/apache/kafka/connect/rest · high confidence

The Kafka Connect API now includes a dedicated \org.apache.kafka.connect.errors\ package containing new exception classes: \ConnectException\ as the base, along with \AlreadyExistsException\, \DataException\, \IllegalWorkerStateException\, \NotFoundException\, \RetriableException\, \SchemaBuilderException\, and \SchemaProjectorException\. These provide structured, public-facing error types for connector and task implementations to signal specific failure modes such as data issues, state violations, or schema problems.

connect/api/src/main/java/org/apache/kafka/connect/errors · high confidence

A new \org.apache.kafka.connect.health\ package is introduced, providing a public API for inspecting the state of the Connect cluster and individual connectors. The \ConnectClusterState\ interface allows REST extensions to retrieve a list of connectors, their health, and cluster details (including the backing Kafka cluster ID). Health information is exposed via \ConnectorHealth\ (containing state, task states, and type) and \ConnectorState\/\TaskState\ classes that track status, worker IDs, and error traces. This enables external tools and REST endpoints to query real-time health and configuration data for all connectors and tasks in the cluster.

connect/api/src/main/java/org/apache/kafka/connect/health · high confidence

Three new built-in predicates are now available for use in Connect transforms: HasHeaderKey, which filters records that contain a specific header name; RecordIsTombstone, which identifies records with a null value; and TopicNameMatches, which filters records based on a Java regular expression applied to the topic name. These additions expand the options for conditional routing and filtering within the Connect framework.

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/predicates · high confidence

The Kafka Connect API introduces a new \ConnectRecord\ class to represent data records with topic, partition, key, value, timestamp, and headers, alongside updated \Connector\ and \Task\ interfaces that define the lifecycle and configuration management for connectors and their associated tasks. A \ConnectorContext\ interface is added to allow connectors to interact with the runtime, and a \ConnectorUtils\ class provides helper methods for partitioning elements.

connect/api/src/main/java/org/apache/kafka/connect/connector · high confidence

The Kafka client library introduces a new \org.apache.kafka.common.header\ package containing the \Headers\ and \Header\ interfaces. The \Headers\ interface provides a mutable, ordered collection for attaching key-value metadata to Kafka records, supporting operations like \add\, \remove\, and retrieval of headers by key. The \Header\ interface defines the structure for individual header entries with \key()\ and \value()\ methods. This change enables developers to attach and manage custom metadata on Kafka messages.

clients/src/main/java/org/apache/kafka/common/header · high confidence

A new public \org.apache.kafka.server.authorizer\ package is introduced, providing a pluggable interface for performing authorization on a Kafka server. This includes the \Authorizer\ interface with methods for synchronously authorizing actions, asynchronously creating and deleting ACLs, and querying ACLs. Supporting classes include \Action\ to represent authorization requests, \AuthorizationResult\ for outcomes, \AuthorizableRequestContext\ for request metadata, and result wrappers (\AclCreateResult\, \AclDeleteResult\) that return exceptions as \Optional\<ApiException\>\.

clients/src/main/java/org/apache/kafka/server/authorizer · high confidence

The Kafka Connect API now includes a dedicated \org.apache.kafka.connect.sink\ package containing new interfaces and classes to support sink connectors. This includes the \ErrantRecordReporter\ interface, which allows sink tasks to report problematic records to a dead letter queue, and the \SinkTask\ abstract class, which defines the lifecycle methods (\open\, \close\, \put\, \flush\, \preCommit\) for processing records. The \SinkRecord\ class has been introduced to carry original Kafka metadata (topic, partition, offset) before transformations, and \SinkConnector\ provides a base class for sink connectors with support for topic regex configuration and manual offset alteration via the \alterOffsets\ method. Additionally, \SinkTaskContext\ and \SinkConnectorContext\ interfaces provide runtime utilities like offset management, partition assignment, and plugin metrics access.

connect/api/src/main/java/org/apache/kafka/connect/sink · high confidence

The Kafka Java client now exposes a new \org.apache.kafka.clients.producer\ package containing the \KafkaProducer\ implementation, the \Producer\ interface, and supporting classes such as \BufferExhaustedException\, \Callback\, \Partitioner\, and \PreparedTxnState\. This provides a public API for sending records, managing transactions, and configuring the producer, alongside a \MockProducer\ for testing.

clients/src/main/java/org/apache/kafka/clients/producer · high confidence

Added new example applications for Kafka Streams: a typed and untyped pageview join demo, a simple pipe demo, an IoT temperature monitoring demo, and word count demos using both the high-level DSL and low-level Processor API. Included corresponding unit tests for the wordcount example and developer guide testing code.

streams/examples · high confidence

MirrorMaker 2 now includes a new MirrorCheckpointConnector and CheckpointStore to replicate consumer group state between clusters. The connector reads checkpoints from a Kafka log, populates an in-memory map of checkpoints per consumer group, and emits checkpoint records. This enables offset synchronization and state replication for consumer groups, with configurable filters for topics, groups, and config properties. The implementation includes default filters (DefaultTopicFilter, DefaultGroupFilter, DefaultConfigPropertyFilter) and metrics tracking for checkpoint latency.

connect/mirror/src/main/java/org/apache/kafka/connect/mirror · high confidence

Added a new public API for managing MirrorMaker 2 internal resources, including the \MirrorClient\ and \RemoteClusterUtils\ for discovering and querying topics like heartbeats, checkpoints, and remote topics. Introduced the \ReplicationPolicy\ interface with two default implementations: \DefaultReplicationPolicy\, which prefixes remote topics with the source cluster alias, and \IdentityReplicationPolicy\, which preserves original topic names. The \MirrorClientConfig\ now supports configuring the replication policy class and separator, allowing users to customize how topics are named and identified across clusters.

connect/mirror-client/src/main · high confidence

Kafka Connect now exposes a new \Predicate\ interface in the \org.apache.kafka.connect.transforms.predicates\ package, allowing users to define custom conditions for applying transformations. This change enables the \Filter\ transformation to be applied conditionally based on record content, with automatic metric tagging for observability.

connect/api/src/main/java/org/apache/kafka/connect/transforms/predicates · high confidence

The Kafka core module now includes a new Scala-based metrics reporting framework. This introduces a \KafkaMetricsReporter\ trait and a \KafkaMetricsConfig\ class to manage reporter initialization and configuration. A concrete \KafkaCSVMetricsReporter\ implementation is added to support CSV-based metrics output, allowing users to configure and enable CSV reporting via system test configurations.

core/src/main/scala/kafka/metrics · medium confidence

The Kafka client introduces a new Share Consumer for reliable, single-consumer message delivery, accompanied by an acknowledgement model that allows consumers to explicitly accept, release, or reject records. This change adds the \AcknowledgeType\ enum (ACCEPT, RELEASE, REJECT, RENEW) and the \AcknowledgementCommitCallback\ interface to handle asynchronous acknowledgement results. The \Consumer\ interface is updated to include methods for acknowledging records, while \CloseOptions\ and \CommitFailedException\ are introduced to manage shutdown behavior and error states. These additions support the new \KafkaShareConsumer\ and \ShareConsumer\ interfaces, enabling applications to implement at-least-once or exactly-once processing semantics with explicit record lifecycle management.

clients/src/main/java/org/apache/kafka/clients/consumer · high confidence

A new \Transformation\ interface has been added to the Kafka Connect API, defining the contract for single message transforms that can be discovered via the Java ServiceLoader mechanism. This interface enforces that transformations must not mutate the input record or its reachable objects (such as headers or structs), requiring implementations to return a new record or null instead. The change also includes a package-info file for the \org.apache.kafka.connect.transforms\ package.

connect/api/src/main/java/org/apache/kafka/connect/transforms · high confidence

The storage module introduces new classes to manage remote log metadata, including a \TopicPartitionLog\ interface to decouple the \RemoteLogManager\ from the \Partition\, and a \BrokerReadyCallback\ interface for broker startup synchronization. Additionally, the \TopicBasedRemoteLogMetadataManager\ implementation is refactored to use dedicated \ConsumerManager\ and \ProducerManager\ classes, which encapsulate the logic for polling and publishing messages to the remote log metadata topic, improving separation of concerns and testability.

storage · high confidence

The \org.apache.kafka.common.security.auth\ package introduces several new public interfaces and classes to support pluggable authentication mechanisms. This includes the \AuthenticateCallbackHandler\ interface for SASL-based authentication, the \AuthenticationContext\ interface and its implementations (\PlaintextAuthenticationContext\, \SaslAuthenticationContext\, \SslAuthenticationContext\) to represent session context, and the \KafkaPrincipalBuilder\ interface for building principals from authentication context. Additionally, the \KafkaPrincipal\ class represents a principal with a type and name, while \SaslExtensions\ and \SaslExtensionsCallback\ support customizable SASL extensions. The \SecurityProtocol\ enum defines the supported security protocols (PLAINTEXT, SSL, SASL\_PLAINTEXT, SASL\_SSL), and \SecurityProviderCreator\ allows for custom security provider generation. These changes provide a more structured and extensible way to handle authentication and authorization in Kafka.

clients/src/main/java/org/apache/kafka/common/security/auth · high confidence

A new \org.apache.kafka.common.cache\ package has been added to the Kafka clients, introducing a \Cache\ interface alongside concrete implementations: an LRU (Least Recently Used) cache (\LRUCache\) and a thread-safe wrapper (\SynchronizedCache\). These components provide reusable caching primitives for internal use, enabling features like JSON schema conversion caching in the \JsonConverter\.

clients/src/main/java/org/apache/kafka/common/cache · high confidence

The Java client introduces new classes to manage connection states and DNS resolution, including \ApiVersions\ to track node API versions, \ClientDnsLookup\ to control how bootstrap servers are resolved, and \ClientRequest\/\ClientResponse\ to encapsulate request metadata and response details. Additionally, \CommonClientConfigs\ defines new configuration options such as \client.dns.lookup\, \socket.connection.setup.timeout.ms\, and \reconnect.backoff.max.ms\, allowing users to fine-tune connection behavior, timeout handling, and DNS resolution strategies.

clients/src/main/java/org/apache/kafka/clients · high confidence

The client library introduces several new public types to support updated cluster metadata and group management. A new \Cluster\ class and \ClusterResource\/\ClusterResourceListener\ interfaces allow clients to access and react to cluster metadata changes. Additionally, new enums \GroupState\, \ClassicGroupState\, and \ConsumerGroupState\ are added to represent the state of different group types, with \ConsumerGroupState\ marked as deprecated in favor of the unified \GroupState\. The \KafkaFuture\ class is also updated to expose a \toCompletionStage()\ method for interoperability with Java's \CompletionStage\ API.

clients/src/main/java/org/apache/kafka/common · high confidence

Added new Java classes in the \org.apache.kafka.common.quota\ package to support the \DescribeClientQuotas\ and \AlterClientQuotas\ APIs. The \ClientQuotaEntity\ class represents a client's identity (user, client-id, or IP), \ClientQuotaAlteration\ defines the configuration changes to apply, \ClientQuotaFilter\ and \ClientQuotaFilterComponent\ provide mechanisms to filter and match specific quota entities, and \ClientQuotaAlteration.Op\ defines individual quota operations.

clients/src/main/java/org/apache/kafka/common/quota · high confidence

The committer-tools directory now includes several new Python scripts and a shell script to automate common committer tasks. The \reviewers.py\ script helps generate and append a 'Reviewers:' trailer to pull request bodies by analyzing Git history. The \refresh\_collaborators.py\ script fetches top non-committer contributors from GitHub and updates the \.asf.yaml\ file with new collaborators. The \kafka-merge-pr.py\ script assists in merging pull requests and managing Git branches. Additionally, \find-unfinished-test.py\ identifies hanging tests in Gradle output, \update-cache.sh\ updates a local branch to the latest cached Gradle build, and \verify\_license.py\ checks that all JARs in the distribution are listed in the LICENSE file.

committer-tools · high confidence

The Kafka Java client introduces a new configuration framework centered on the \AbstractConfig\ base class, which supports external configuration providers via the \ConfigProvider\ interface. This allows configuration values to be resolved from external sources (such as files or secret managers) using variable substitution patterns (e.g., \${provider:path:key}\). The change includes new classes for managing configuration definitions (\ConfigDef\), transformation (\ConfigTransformer\), and validation (\ConfigValue\, \ConfigException\), enabling dynamic and secure configuration management for clients.

clients/src/main/java/org/apache/kafka/common/config · high confidence

The Kafka Java client now provides a comprehensive set of new exception classes in the \org.apache.kafka.common.errors\ package, including \ApiException\, \ApplicationRecoverableException\, \AuthenticationException\, \AuthorizationException\, and numerous specific error types (e.g., \BrokerNotAvailableException\, \CoordinatorNotAvailableException\, \FencedInstanceIdException\). These classes, marked as public API, allow developers to catch and handle specific client-side errors more precisely, improving error management and recovery strategies in Kafka applications.

clients/src/main/java/org/apache/kafka/common/errors · high confidence

The group-coordinator module introduces a new, structured approach to managing consumer, share, and streams groups, including the introduction of a GroupConfig class to handle group-specific settings and a GroupConfigManager to manage these configurations. A new CommitPartitionValidator interface and ConsumerGroupMigrationPolicy enum are added to support flexible group migration and validation. The GroupCoordinator interface is expanded to handle a wide range of group-specific RPCs, including Share, Streams, and Consumer group heartbeats, offsets, and descriptions, enabling the new coordinator to manage these distinct group types with specific behaviors and configurations.

group-coordinator · high confidence

The Kafka client introduces several new internal classes to improve security handling and asynchronous operations. A new \SecurityManagerCompatibility\ interface and its implementations (\LegacyStrategy\, \ModernStrategy\, \CompositeStrategy\) provide a unified way to access deprecated Java security APIs, ensuring compatibility across different Java versions. Additionally, \KafkaCompletableFuture\ and \KafkaFutureImpl\ are added to manage asynchronous operations while preventing user code from incorrectly completing futures. The \IdempotentCloser\ class is introduced to safely handle resource closing, and \Plugin\ wraps instances for metrics and lifecycle management. Finally, \PartitionStates\ and \Topic\ classes provide internal state management and validation logic.

clients/src/main/java/org/apache/kafka/common/internals · high confidence

The clients module introduces a new \org.apache.kafka.common.memory\ package containing a \MemoryPool\ interface and a \SimpleMemoryPool\ implementation for explicit memory management, alongside a new \Authenticator\ interface and related network classes (e.g., \ChannelBuilder\, \ChannelState\) that define the structure for network channels and authentication. These changes provide the foundational abstractions for managing client-side memory and network channel states.

clients/src/main/java/org/apache/kafka/common/network · high confidence

The metadata module introduces several new classes to manage controller and broker state. A new InvalidReplicaDirectoriesException is added to handle mismatches between replica and directory counts. The AclControlManager is introduced to manage ACLs stored in the \_\_cluster\_metadata topic, including validation for CIDR-based host patterns. New classes such as ActivationRecordsGenerator, BrokerControlState, BrokerControlStates, BrokerHeartbeatManager, BrokerHeartbeatTracker, BrokerIdAndEpoch, and BrokersToElrs are added to handle controller activation, broker heartbeat tracking, and leader replica management within the metadata module.

metadata · high confidence

The broker now uses new \BrokerMetadataPublisher\, \DynamicConfigPublisher\, and \ClientQuotaMetadataManager\ classes in the \kafka.server.metadata\ package to handle metadata updates. \BrokerMetadataPublisher\ coordinates the initial setup and ongoing updates for topics, group coordinators, transaction coordinators, and share coordinators. \DynamicConfigPublisher\ applies dynamic configuration changes for topics, brokers, client metrics, and groups. \ClientQuotaMetadataManager\ processes quota metadata records to update IP, user, and client ID quotas. These components centralize the logic for applying metadata deltas to the broker's internal state.

core/src/main/scala/kafka/server/metadata · high confidence

The Kafka client metrics library introduces a new set of statistical classes in the \org.apache.kafka.common.metrics.stats\ package to support richer metric reporting. This includes \Avg\, \Max\, and \Min\ for tracking simple aggregates; \CumulativeCount\ and \CumulativeSum\ for tracking totals over time; \Frequencies\ and \Percentiles\ for distribution analysis using histograms; \Meter\ for rate and total tracking; \Rate\ and \SimpleRate\ for calculating rates over time windows; and \TokenBucket\ for implementing token-bucket-based quotas. These classes provide the underlying mechanics for measuring and combining samples, enabling more granular and accurate metric collection for users.

clients/src/main/java/org/apache/kafka/common/metrics/stats · high confidence

The \clients\ module now includes a new \org.apache.kafka.server.quota\ package containing the \ClientQuotaCallback\ interface, along with supporting types \ClientQuotaEntity\ and \ClientQuotaType\. This introduces a public API that allows users to implement custom logic for computing quota metric tags and limits for client requests, enabling more flexible quota management on the broker side.

clients/src/main/java/org/apache/kafka/server/quota · high confidence

The Kafka client library now includes new public interfaces, \CreateTopicPolicy\ and \AlterConfigPolicy\, which allow administrators to enforce custom validation rules on topic creation and configuration changes. These interfaces, located in \org.apache.kafka.server.policy\, enable the implementation of policies that can reject requests that violate specific constraints, such as replication factors or retention settings, by throwing a \PolicyViolationException\.

clients/src/main/java/org/apache/kafka/server/policy · high confidence

The Kafka client library now includes built-in serializers and deserializers for primitive types (Boolean, Byte, Short, Int, Long, Float, Double), as well as for raw byte arrays, ByteBuffers, and Java Lists. This introduces a new \Serde\ interface and a \Serdes\ factory class that bundle serializers and deserializers together, making it easier to configure data formats for topics. Users can now easily serialize and deserialize common Java types without writing custom code.

clients/src/main/java/org/apache/kafka/common/serialization · high confidence

The Kafka Java client introduces a new, fully-featured producer implementation located in the \org.apache.kafka.clients.producer.internals\ package. This update replaces the legacy producer with a new architecture that includes a \BufferPool\ for efficient memory management, a \BuiltInPartitioner\ supporting sticky partitioning and rack-awareness, and a \TransactionManager\ for robust transactional guarantees. The new producer supports idempency, automatic batch splitting for large records, and provides detailed metrics for monitoring producer performance and latency.

clients/src/main/java/org/apache/kafka/clients/producer/internals · high confidence

The \org.apache.kafka.common.protocol.types\ package now provides a comprehensive, object-oriented framework for defining and serializing Kafka protocol messages. This includes new classes for handling structured data (\Schema\, \Struct\, \Field\, \BoundField\), array types (\ArrayOf\, \CompactArrayOf\), and tagged fields (\TaggedFields\, \RawTaggedField\, \RawTaggedFieldWriter\). The system supports nullable types, default values, and flexible versioning, enabling more robust and maintainable protocol definitions for Kafka requests and responses.

clients/src/main/java/org/apache/kafka/common/protocol/types · high confidence

The group-coordinator-api module introduces a new public API for consumer group partition assignment, including interfaces and classes such as PartitionAssignor, ConsumerGroupPartitionAssignor, ShareGroupPartitionAssignor, GroupSpec, GroupAssignment, MemberAssignment, MemberSubscription, SubscribedTopicDescriber, and SubscriptionType. Additionally, it adds the SPI for the Kafka Streams group topology description plugin, including the StreamsGroupTopologyDescription data model, the StreamsGroupTopologyDescriptionPlugin interface, and associated exception types for permanent and transient failures. These additions provide the core functionality and metadata management for consumer group partition assignment and expose the broker-side plugin interface for storing and retrieving topology descriptions.

group-coordinator/group-coordinator-api · high confidence

The \storage/api\ module now exposes a new set of public classes and interfaces for managing remote log segments, including \RemoteLogSegmentMetadata\, \RemoteLogSegmentMetadataUpdate\, \RemoteLogSegmentState\, \RemoteLogMetadataManager\, and \RemoteStorageManager\. These additions provide the core data structures and interfaces required to track the lifecycle and state of log segments in remote storage, enabling implementations to handle segment copying, updates, and deletion states asynchronously.

storage/api · high confidence

The Kafka client library now exposes a new public API for managing Access Control Lists (ACLs). This includes the \AccessControlEntry\ and \AccessControlEntryFilter\ classes to represent and filter individual ACL entries, as well as \AclBinding\ and \AclBindingFilter\ to bind resource patterns to access control entries. The change also introduces the \AclOperation\ and \AclPermissionType\ enums to define supported operations and permission types, providing a structured way to interact with Kafka's authorization system.

clients/src/main/java/org/apache/kafka/common/acl · high confidence

The release directory now contains a comprehensive set of Python scripts (release.py, git.py, gpg.py, notes.py, preferences.py, runtime.py, svn.py, templates.py, textfiles.py) and a README that automate the Apache Kafka release process. These tools handle Git operations, GPG signing, JIRA integration for release notes, and email template generation, providing a structured workflow for creating release candidates and final releases.

release · high confidence

Added new interfaces and classes in the \org.apache.kafka.common.replica\ package to support flexible replica selection. This includes the \ReplicaSelector\ interface for pluggable selection logic, along with \ClientMetadata\ and \PartitionView\ interfaces to pass client and partition state to selectors. A concrete \RackAwareReplicaSelector\ implementation is also provided, which selects a replica from the same rack as the client when possible, falling back to the leader. These changes enable more sophisticated read routing strategies based on client metadata and replica state.

clients/src/main/java/org/apache/kafka/common/replica · high confidence

The Kafka client now exposes a new set of public classes in the \org.apache.kafka.common.resource\ package to represent resources and patterns for Access Control Lists (ACLs). This includes \ResourceType\ (defining types like TOPIC, GROUP, CLUSTER, TRANSACTIONAL\_ID, and DElegation\_TOKEN), \Resource\ (a tuple of type and name), \ResourcePattern\ (combining a resource with a pattern type), and \ResourcePatternFilter\ (for matching patterns). These classes provide the foundational data structures for managing and filtering ACLs in the AdminClient API.

clients/src/main/java/org/apache/kafka/common/resource · high confidence

Added new single-message transforms to the Kafka Connect transforms module, including Cast, ExtractField, Filter, Flatten, HeaderFrom, HoistField, InsertField, InsertHeader, and MaskField. These transforms enable users to cast field types, extract or flatten nested data structures, conditionally filter records, move or copy fields to headers, and insert metadata or static values into records.

connect/transforms/src/main/java/org/apache/kafka/connect/transforms · high confidence

The Kafka Connect API introduces new interfaces for data conversion: \Converter\ and \HeaderConverter\ replace the previous single \Converter\ interface, allowing separate handling of record headers. A new \ConverterType\ enum distinguishes between key, value, and header conversions. To support immediate usage, the API now includes default implementations: \StringConverter\ for serializing data to/from strings, and \SimpleHeaderConverter\ for basic header serialization. These changes provide a more flexible and extensible foundation for custom data format plugins.

connect/api/src/main/java/org/apache/kafka/connect/storage · high confidence

The \connect/test-plugins\ module now includes new test connector implementations: \MockConnector\, \MockSinkConnector\, \MockSinkTask\, \MockSourceConnector\, \MockSourceTask\, \SchemaSourceConnector\, \SchemaSourceTask\, \VerifiableSinkConnector\, \VerifiableSinkTask\, \VerifiableSourceConnector\, and \VerifiableSourceTask\. These connectors are registered via the \META-INF/services\ files, making them available for use in system tests to simulate failures, generate verifiable output, or produce static data with schemas.

connect/test-plugins · high confidence

Added new test framework components in the test-common-internal-api module, including the @ClusterTest, @ClusterTests, @ClusterTemplate, and @ClusterTestDefaults annotations, along with supporting classes like ClusterConfig, ClusterFeature, and DetectThreadLeak to enable declarative and template-based Kafka cluster testing.

test-common/test-common-internal-api · high confidence

The test-common-runtime module introduces a suite of new classes to support the refactored test infrastructure, including \KafkaClusterTestKit\ for managing cluster lifecycles, \ClusterInstance\ as the primary interface for accessing cluster state, and \AdminUtils\ to simplify admin client operations. The update also adds \MockController\ for headless controller testing, \PreboundSocketFactoryManager\ to ensure stable port binding during test restarts, and \SslManager\/\JaasModule\ to streamline SSL and JAAS configuration for tests.

test-common/test-common-runtime · high confidence

The \tools-api\ module now exposes a public \Decoder\ interface and several concrete implementations (\DefaultDecoder\, \IntegerDecoder\, \LongDecoder\, \StringDecoder\) for converting byte arrays into typed objects. Additionally, the \RecordReader\ interface is introduced to handle reading records from an \InputStream\ into \ProducerRecord\ instances, with a corresponding unit test added for the \RecordReader\ default methods.

tools/tools-api · high confidence

Added new utility classes in the Kafka Connect transforms package to support single-message transforms. This includes validators for non-empty lists and regex patterns, helper methods for requiring specific record types (Map, Struct, SinkRecord) and schemas, and a simple configuration base class. These utilities provide common validation and type-checking logic used by transform implementations.

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util · high confidence

The \streams/test-utils\ module now provides a public API for testing Kafka Streams applications. This includes the \TopologyTestDriver\ and its builder, along with \TestInputTopic\ and \TestOutputTopic\ for piping and reading records, \TestRecord\ for constructing test data, and \MockProcessorContext\ for unit testing processors. These utilities allow developers to write tests without a real Kafka broker, enabling fast and lightweight verification of stream processing logic.

streams/test-utils · high confidence

The command-line entry points for running Kafka Connect in standalone and distributed modes have been refactored to share common initialization logic. A new \AbstractConnectCli\ base class now handles argument parsing, worker property loading, plugin scanning, and Connect worker startup, while \ConnectDistributed\ and \ConnectStandalone\ provide mode-specific herder and configuration creation. This change simplifies the CLI codebase, reduces duplication, and makes it easier to add new Connect CLI utilities in the future.

connect/runtime/src/main · high confidence

The metrics subsystem has been refactored to introduce a more flexible and type-safe API. A new \MetricValueProvider\<T\>\ interface allows metrics to provide typed values, replacing the previous generic \Metric.value()\ approach. This is supported by new interfaces like \Measurable\ for single-value statistics and \Gauge\ for instantaneous readings. The \Sensor\ class now supports hierarchical sensors with parent-child relationships and configurable recording levels (INFO, DEBUG, TRACE). Additionally, the \Metrics\ registry now supports dynamic reconfiguration of reporters and automatic expiration of inactive sensors. These changes provide a more robust foundation for adding new metrics and improving performance.

clients/src/main/java/org/apache/kafka/common/metrics · high confidence

The \core\ module now includes new builder classes (\KafkaApisBuilder\, \LogManagerBuilder\, \ReplicaManagerBuilder\) that provide a fluent, step-by-step API for constructing \KafkaApis\, \LogManager\, and \ReplicaManager\ instances. This change streamlines the initialization of these critical server components, making it easier to configure and wire them together during broker startup.

core/src/main/java/kafka/server/builders · high confidence

The Trogdor test framework is now available as a standalone Gradle module. This includes the core agent and coordinator components, REST interfaces, and basic platform implementations, enabling users to run benchmarks, inject faults, and stress test Kafka clusters.

trogdor · high confidence

The Hadoop consumer module, which provided a Hadoop job to pull data from Kafka into HDFS, has been removed. This includes the deletion of the README documentation, shell scripts for setup and execution (copy-jars.sh, hadoop-setup.sh, run-class.sh), and the test configuration file (test.properties).

contrib/hadoop-consumer · high confidence

The Hadoop-based Kafka consumer modules have been removed from the codebase. This includes the deletion of the \kafka.etl\ package, which contained the \KafkaETLContext\, \KafkaETLInputFormat\, \KafkaETLJob\, \KafkaETLKey\, \KafkaETLRecordReader\, \KafkaETLRequest\, \KafkaETLUtils\, and \Props\ classes. These components previously facilitated the integration between Kafka and Hadoop MapReduce jobs for ETL processes.

contrib/hadoop-consumer/src/main/java/kafka/etl · high confidence

The Java API classes \kafka.javaapi.message.ByteBufferMessageSet\ and \kafka.javaapi.message.MessageSet\ have been removed. Users relying on these specific Java API components for message set handling will need to migrate to the current message handling APIs.

core/src/main/scala/kafka/javaapi/message · high confidence

The Hadoop producer contrib module has been removed, specifically deleting the \TextPublisher\ example, \KafkaOutputFormat\, \KafkaRecordWriter\, and \AvroKafkaStorage\ classes. This eliminates the legacy Hadoop MapReduce and Pig integration points for the Kafka producer.

contrib/hadoop-producer/src · high confidence

The Hadoop consumer example code, including the DataGenerator, SimpleKafkaETLJob, and SimpleKafkaETLMapper classes, has been removed from the contrib/hadoop-consumer module. This change eliminates the legacy Hadoop-based ETL job implementations that were previously provided as examples for integrating Kafka with Hadoop MapReduce.

contrib/hadoop-consumer/src/main/java/kafka/etl/impl · high confidence

The legacy \kafka.serializer\ package has been removed, specifically deleting \Decoder.scala\ and \Encoder.scala\ and their associated classes (\DefaultDecoder\, \StringDecoder\, \DefaultEncoder\, \StringEncoder\). This change eliminates deprecated serialization utilities in favor of newer APIs, requiring users to migrate to the updated serialization mechanisms.

core/src/main/scala/kafka/serializer · high confidence

The \SimpleConsumer\ and \ZookeeperConsumerConnector\ classes in the \kafka.javaapi.consumer\ package have been removed. These deprecated Scala-based consumer implementations are no longer available for use in Java applications.

core/src/main/scala/kafka/javaapi/consumer · high confidence

The Ruby client library (kafka-rb) has been removed from the codebase. This includes the client source files (lib/kafka.rb, producer, consumer, and related utilities), the Rakefile, the README, and the LICENSE file. Users relying on this Ruby client will no longer have access to these components.

clients/ruby · high confidence

The C\# client library directory and all associated source files have been removed from the repository. This includes the \Kafka.Client\ namespace containing the \Producer\, \Consumer\, and \KafkaConnection\ classes, as well as request models like \FetchRequest\ and \ProducerRequest\. The removal also includes the \clients/csharp\ directory's metadata files, such as the \LICENSE\, \.gitignore\, and \README.md\. This change eliminates the original C\# client implementation from the codebase.

clients/cpp, clients/csharp · high confidence

The entire PHP client library directory has been removed from the repository. This includes all core library files (such as \Kafka\_Producer\, \Kafka\_SimpleConsumer\, \Kafka\_FetchRequest\, and \Kafka\_Encoder\), example scripts (\produce.php\, \consume.php\), and all associated PHPUnit test cases. Users relying on this specific PHP client implementation will no longer have access to these files.

clients/go, clients/php · high confidence

The Python client library (kafka.py) and its associated setup configuration (setup.py) have been removed from the codebase. This eliminates the older, simple socket-based producer implementation that encoded messages using a specific request format, requiring users to migrate to the current client implementation.

clients/python · high confidence

The legacy Scala-based asynchronous producer implementation, including \AsyncProducer\, \DefaultEventHandler\, and related configuration and callback handler classes, has been removed from the codebase. Users relying on the old \kafka.producer.async\ package will need to migrate to the newer producer API.

core/src/main/scala/kafka/producer/async · high confidence

The legacy Scala-based producer, including the \Producer\ class, \ProducerPool\, \SyncProducer\, and related configuration and partitioning classes, has been removed from the \kafka.producer\ package. This change eliminates the old producer implementation, requiring users to migrate to the newer Java-based client for publishing messages to Kafka.

core/src/main/scala/kafka/producer · high confidence

The legacy Scala-based Kafka consumer implementation has been removed from the codebase. This includes the deletion of core classes such as ZookeeperConsumerConnector, ConsumerConnector, and related utilities like Fetcher and ConsumerIterator. Users relying on the old Scala consumer API will need to migrate to the newer Java-based or high-level consumer implementations.

core/src/main/scala/kafka/consumer · high confidence

The legacy Scala-based producer classes—specifically \Producer\, \ProducerData\, and \SyncProducer\ from the \kafka.javaapi.producer\ package—have been removed. This change eliminates the older, non-Java-compatible producer interface, requiring users to migrate to the current Java API for publishing messages.

core/src/main/scala/kafka/javaapi/producer · high confidence

The deprecated Scala API classes for handling Produce, Fetch, and Offset requests and responses have been removed from the Kafka server. This includes the deletion of \FetchRequest\, \MultiFetchRequest\, \MultiFetchResponse\, \MultiProducerRequest\, \OffsetRequest\, \ProducerRequest\, and \RequestKeys\. As a result, clients using the old Scala-based protocol will no longer be able to communicate with this version of the broker, as these legacy request types are no longer supported.

core/src/main/scala/kafka/api · high confidence

Removed the legacy message format (magic byte 0) and associated classes including ByteBufferBackedInputStream, ByteBufferMessageSet, CompressionCodec, CompressionUtils, FileMessageSet, Message, MessageLengthException, and MessageSet. This change eliminates support for the older message format, requiring clients to use the current message format.

core/src/main/scala/kafka/message · high confidence

The \system\_test/embedded\_consumer\ and \system\_test/producer\_perf\ directories have been removed. These contained the test scripts, configuration files, and expected output files for the embedded consumer replication test and the producer performance/compression tests. This change eliminates these specific system tests from the repository.

_system\test · high confidence

The Jetty dependency has been upgraded to version 12.0.34 to address security vulnerabilities, specifically [CVE redacted]. This update ensures that the embedded HTTP server used by Kafka Connect and other components is protected against known security risks.

(dependencies) · high confidence

The server module's network package has been refactored to use Java sealed interfaces and records for request and response types. This introduces a strict type hierarchy for network operations, including new sealed interfaces for \BaseRequest\ and \Response\, and corresponding record classes like \CallbackRequest\, \SendResponse\, and \CloseConnectionResponse\. This change enforces a closed set of request and response types, improving type safety and clarity in the network handling logic.

server · high confidence

The broker server implementation has been significantly refactored to support the KRaft (Kafka Raft) metadata quorum, removing the dependency on Apache ZooKeeper for cluster coordination. This change introduces a new server module structure, moving core components such as the \ReplicaManager\, \LogManager\, and \DynamicConfig\ into dedicated modules (e.g., \server\, \storage\, \metadata\). As a result, brokers can now run in a standalone mode without ZooKeeper, relying on an internal Raft-based controller for metadata management. This architectural shift enables a simpler deployment topology and improves the reliability of the control plane by eliminating the external ZooKeeper dependency.

core/src/main/scala/kafka/server · high confidence

The transaction coordinator logic has been moved from the core Scala module into a new, separate \transaction-coordinator\ Java module. This refactoring includes moving key classes such as \TransactionMetadata\, \TransactionLog\, \ProducerIdManager\, and configuration classes like \TransactionLogConfig\ and \AddPartitionsToTxnConfig\ into this new module. This change improves modularity by separating transaction management from the main broker core, providing a cleaner architecture for transaction state management and producer ID allocation.

transaction-coordinator · high confidence

The generator module was refactored to improve code organization and maintainability. New classes were introduced to handle specific code generation tasks: ApiMessageTypeGenerator for generating the ApiMessageType enum, CoordinatorRecordTypeGenerator and CoordinatorRecordJsonConvertersGenerator for handling coordinator record serialization, and supporting utilities like CodeBuffer, HeaderGenerator, and IsNullConditional. This modularization separates concerns within the code generation pipeline, making it easier to extend and maintain the protocol schema processing.

generator · high confidence

A new handler for the DescribeTopicPartitions RPC is introduced in the server module, implementing the broker-side logic for describing topic partitions. The handler processes requests to list and describe partition details, applying authorization checks via AuthHelper and filtering topics based on permissions. It supports both explicit topic lists and fetching all topics with cursor-based pagination, returning structured response data including topic metadata and authorized operations.

core/src/main/java/kafka/server/handlers · high confidence

The Kafka client library now includes new internal classes to manage broker-side security settings and path validation. A new \AllowedPaths\ class validates that file and directory configuration providers are restricted to specific, absolute, and existing paths, resolving symlinks before validation. Additionally, \BrokerSecurityConfigs\ consolidates broker security configuration definitions, including settings for SSL client authentication, principal building, SASL mechanisms, and dynamic certificate reloading. This centralizes security configuration handling within the shared client library, ensuring consistent validation and access control for broker security features.

clients/src/main/java/org/apache/kafka/common/config/internals · high confidence

The \streams/streams-scala\ module, which provides a Scala wrapper for the Kafka Streams DSL, is now deprecated as of version 4.3.0. All classes and objects in this module—including \StreamsBuilder\, \KStream\, \KTable\, and related configuration wrappers like \Consumed\, \Grouped\, and \Joined\—are marked with \@deprecated\ annotations, directing users to migrate to the corresponding Java classes in \org.apache.kafka.streams\ and \org.apache.kafka.streams.kstream\.

streams/streams-scala · high confidence

The FileStreamSourceConnector and FileStreamSinkConnector now implement the alterOffsets method, enabling the Connect framework to manage and validate offsets for these connectors. This change allows the source connector to support exactly-once delivery semantics when reading from a file (while remaining unsupported for stdin), and ensures that offset modifications are properly validated against the configured file path. Additionally, the source connector introduces a 'batch.size' configuration to control the number of records read per poll, helping to prevent out-of-memory issues.

connect/file/src/main/java · high confidence

The state updater now correctly excludes failed tasks from the set of tasks to assign or process, preventing the system from attempting to handle or restore tasks that have encountered errors. This change ensures that transient failures do not block or corrupt the state update process, leading to more stable stream processing during error conditions.

streams · high confidence

The SSL client authentication process now logs the common name of expired client certificates before rejecting the connection. This change introduces a new logging mechanism in the SSL client authentication flow, allowing administrators to identify misconfigured clients in complex network environments where the IP address is not sufficient for troubleshooting.

clients/src/main/java/org/apache/kafka/common/security/ssl · high confidence

Added new internal REST endpoints for Mirror Maker 2, enabling intra-cluster communication. The changes introduce MirrorRestServer and InternalMirrorResource, which expose an internal API for managing connectors across source and target clusters. This supports the broader KIP-1032 effort to upgrade to Jakarta and Jetty 12, providing a structured way for Mirror Maker 2 to interact with Connect instances internally.

connect/mirror/src/main/java/org/apache/kafka/connect/mirror/rest · medium confidence

A collection of internal utility classes—including \AbstractIterator\, \AppInfoParser\, \BufferSupplier\, \ByteBufferInputStream\, \ByteBufferOutputStream\, \ByteBufferUnmapper\, \ByteUtils\, \Checksums\, \ChildFirstClassLoader\, \ChunkedBytesStream\, \CircularIterator\, \CloseableIterator\, \ConfigUtils\, \LogContext\, \OperatingSystem\, \Sanitizer\, and \ThreadUtils\—have been relocated from their previous locations in the public \org.apache.kafka.common.utils\ package to the new \org.apache.kafka.common.utils.internals\ package. This change restricts access to these implementation details, signaling that they are not part of the stable public API and may change without notice.

clients/src/main/java/org/apache/kafka/common/utils/internals · high confidence

Added the new StorageTool for managing storage directories and metadata in KRaft mode, supporting commands like format, info, version-mapping, and feature-dependencies. Simultaneously, removed the legacy Scala-based tools (ConsumerPerformance, ConsumerShell, GetOffsetShell, ProducerPerformance, ProducerShell, and ReplayLogProducer) in favor of their Java-based or modernized equivalents.

core/src/main/scala/kafka/tools · high confidence

Added FieldSyntaxVersion and SingleFieldPath classes to support nested field access via dotted notation (V2) while maintaining backward compatibility with root-level access (V1). This enables transforms like ExtractField and InsertField to navigate deeper into nested structures using a configurable syntax version.

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/field · high confidence

The Kafka Java client now uses new internal classes, RecordHeader and RecordHeaders, to manage record headers. RecordHeader implements lazy initialization for header values to reduce memory usage, while RecordHeaders provides a thread-safe, mutable collection of headers with read-only protection. This change improves memory efficiency and ensures safe concurrent access when manipulating record headers in the Kafka client.

clients/src/main/java/org/apache/kafka/common/header/internals · high confidence

Added NetworkUtils to centralize NetworkClient construction, QuotaFactory to instantiate quota managers (Client, Controller, Replication), and TierStateMachine to handle tiered storage state transitions. These new classes in the core server module support the ongoing modularization of the Kafka server, moving quota and network logic out of the core module into more appropriate modules as part of the server architecture refactoring.

core/src/main/java/kafka/server · medium confidence

The Kafka Connect JSON Converter now supports deserializing JSON messages with leading zeros in numeric fields, resolving parsing issues with certain JSON payloads. Additionally, users can now configure the serialization format for decimal types via the new 'decimal.format' configuration option, allowing them to choose between BASE64 (default) and NUMERIC formats. The converter also introduces a 'schema.content' configuration to provide a default schema for all messages, and enables the Blackbird module for improved performance.

connect/json/src/main/java · high confidence

The Kafka server entry point now initializes a KRaft-based server (KafkaRaftServer) instead of the legacy ZooKeeper-based server, and uses a new Exit utility for process termination. Startup failures now result in a non-zero exit code (1) rather than a generic System.exit(0) or unhandled exception, and the shutdown hook is managed via the Exit utility to ensure proper cleanup. Signal handlers for logging termination messages are now conditionally registered only on non-Windows, non-IBM JDK environments.

core/src/main/scala/kafka · high confidence

The log management system has been migrated from the \core\ module to the \storage\ module, with the \Log\, \LogManager\, and \LogStats\ classes rewritten in Java. This change removes the Scala-based log implementation from the core package, replacing it with a Java-based equivalent that provides the same functionality for managing log segments, flushing, and retention policies.

core/src/main/scala/kafka/log · high confidence

The \org.apache.kafka.common.protocol\ package has been significantly refactored to support an automated RPC generation system. This introduces new core interfaces and classes such as \ApiMessage\, \Message\, and \ApiKeys\ to manage API identifiers and message types. A new \Errors\ enum provides a comprehensive mapping of server-side error codes to specific Java exceptions, improving error handling for clients. Additionally, new utility classes like \ByteBufferAccessor\ and \DataOutputStreamWritable\ are introduced to handle data serialization and deserialization, while \MessageUtil\ and \ObjectSerializationCache\ support efficient message size calculation and caching. These changes underpin the transition to a more robust and automated protocol handling mechanism in the Kafka Java client.

clients/src/main/java/org/apache/kafka/common/protocol · high confidence

The \config\ directory has been updated to support the new KRaft (Kafka Raft) mode, introducing new configuration files for the controller (\controller.properties\), broker (\broker.properties\), and combined server (\server.properties\) that replace the previous ZooKeeper-centric setup. Additionally, the legacy \log4j.properties\ file has been removed and replaced with \log4j2.yaml\ and \tools-log4j2.yaml\ files to support the Log4j2 logging framework, while Connect-specific configuration files (e.g., \connect-standalone.properties\, \connect-distributed.properties\) have been updated to reflect modern defaults and internal topic settings.

config · high confidence

The Kafka client-side request and response classes have been refactored to use the automated protocol generation system. This change introduces new base classes, AbstractRequest and AbstractResponse, which standardize how requests and responses are built, serialized, and parsed. Specific request and response classes (such as AddOffsetsToTxn, AddPartitionsToTxn, and others) now extend these base classes and utilize generated data classes, replacing the previous manual serialization and parsing logic. This results in a more consistent and maintainable codebase for handling Kafka protocol messages.

clients/src/main/java/org/apache/kafka/common/requests · high confidence

The KafkaDockerWrapper now uses a new Java-based configuration model for Log4j2, replacing the previous Log4j1 setup. This change updates how logging is configured within the Docker environment, ensuring compatibility with the newer logging framework.

core/src/main/java/kafka/docker · high confidence

The Kafka client library has reorganized its internal implementation details by moving classes such as AbstractRecordBatch, DefaultRecordBatch, ControlRecordType, and various utility classes (e.g., CompressionRatioEstimator, ControlRecordUtils) from their previous locations (such as the main record package or utils) into the new org.apache.kafka.common.record.internal package. This change restricts the visibility of these classes, signaling that they are not part of the public API and should not be relied upon by external users.

clients/src/main/java/org/apache/kafka/common/record/internal · high confidence

The project introduces new checkstyle configuration files, including a central checkstyle.xml and module-specific import-control files (e.g., import-control-core.xml, import-control-api-checker.xml), alongside a new .scalafmt.conf for Scala code formatting. These changes enforce stricter code quality, import ordering, and style rules across the codebase.

checkstyle · high confidence

The test-common-util module now includes new JUnit 5 filters that allow CI pipelines to run only newly added tests or only flaky tests. The \KafkaPostDiscoveryFilter\ and \CatalogTestFilter\ classes, along with the \@Flaky\ annotation, enable selective test execution based on system properties (\kafka.test.run.new\, \kafka.test.run.flaky\) and external test catalogs. This change modifies test execution behavior by introducing new filtering logic that excludes flaky tests by default and can isolate new tests against a known catalog.

test-common/test-common-util · high confidence

The transaction coordinator implementation, including the main coordinator, state manager, and marker channel manager, has been moved from the core Scala module to the new transaction-coordinator module. This change restructures the codebase by migrating the transactional logic to Java, which improves maintainability and aligns with the project's ongoing effort to reduce Scala dependencies.

core/src/main/scala/kafka/coordinator/transaction · high confidence

The OAuth bearer security internals have been refactored to support client assertion-based authentication (RFC 7521/7523) alongside the existing client secret method. This change introduces new classes including \ClientAssertionRequestFormatter\ and \ClientSecretRequestFormatter\ to handle HTTP request formatting for token requests, a \ClientCredentialsRequestFormatterFactory\ to manage the fallback logic between file-based, locally-generated, and client secret authentication methods, and utility classes like \ClaimValidationUtils\ and \ConfigOrJaas\ to standardize configuration retrieval and validation. Users connecting with client assertions will now use the new assertion-based flow, while those using client secrets continue to use the updated secret-based formatter.

clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured · medium confidence

The SASL authentication logic has been reorganized into the \org.apache.kafka.common.security.authenticator\ package. This includes new or refactored classes such as \AbstractLogin\, \CredentialCache\, \DefaultKafkaPrincipalBuilder\, \DefaultLogin\, \LoginManager\, \SaslClientAuthenticator\, \SaslClientCallbackHandler\, \SaslInternalConfigs\, \SaslServerAuthenticator\, and \SaslServerCallbackHandler\. These changes consolidate the implementation of SASL client and server authentication, principal building, and login management within this specific directory, supporting features like dynamic JAAS configuration and periodic re-authentication.

clients/src/main/java/org/apache/kafka/common/security/authenticator · high confidence

The compression implementation in the \org.apache.kafka.common.compress\ package has been refactored to support configurable compression levels for Gzip and LZ4 codecs. The \Compression\ interface and its implementations (e.g., \GzipCompression\, \Lz4Compression\) now accept a \level\ parameter via a Builder pattern, allowing users to adjust the trade-off between compression speed and ratio. For Gzip, the level is validated against the standard min/max range. For LZ4, the implementation selects between a fast compressor (default level) and a high compressor (other levels). This change aligns with KIP-390 and provides users with more control over the performance characteristics of their compressed data.

clients/src/main/java/org/apache/kafka/common/compress · high confidence

The internal metrics system for Kafka consumers has been refactored to use a new \AbstractConsumerMetricsManager\ base class and a \MetricsLedger\ helper. This change introduces dedicated metrics managers for heartbeat, offset commits, and rebalance callbacks, while also adding new JMX metrics for the \AsyncKafkaConsumer\ (e.g., queue sizes, processing times, and network poll delays). Crucially, the new \MetricsLedger\ automatically tracks and removes all registered sensors and metric names when the consumer is closed, preventing metric leaks.

clients/src/main/java/org/apache/kafka/clients/consumer/internals/metrics · high confidence

The \kafka.utils\ package was refactored to modernize the codebase and remove legacy components. Several utility classes including \Annotations\, \DelayedItem\, \DumpLogSegments\, \IteratorTemplate\, \KafkaScheduler\, \MockTime\, \Pool\, \Range\, \Throttler\, \Time\, \UpdateOffsetsInZK\, \Utils\, and \ZkUtils\ were removed. New utility files such as \Implicits\, \Logging\, \Mx4jLoader\, and \VerifiableProperties\ were added to replace or supplement the old implementations.

core/src/main/scala/kafka/utils · high confidence

The network layer has been refactored to replace legacy Scala-specific request and response classes with the standard \org.apache.kafka.common.requests\ equivalents, improving consistency across the codebase. The \SocketServer\ implementation was significantly rewritten to use Java-based network abstractions, introducing a new \RequestChannel\ for managing request queues and processor threads. Additionally, obsolete classes such as \BoundedByteBufferReceive\, \BoundedByteBufferSend\, and \SocketServerStats\ were removed, while new metrics and configuration handling were integrated to support modern broker features like connection quotas and dynamic listener configuration.

core/src/main/scala/kafka/network · high confidence

A new service provider file has been added to register the BasicAuthSecurityRestExtension implementation. This change ensures that the extension is correctly discovered and loaded by the Kafka Connect REST API integration, addressing the ServiceLoader issue previously reported in KAFKA-6991.

connect/basic-auth-extension/src/main/resources · medium confidence

A significant number of classes have been moved into the server-common module to consolidate shared server-side logic. This includes the DeferredEvent and DeferredEventQueue for managing deferred operations, the StateChangeLogger for standardized logging, and various metadata and configuration utilities such as ConfigRepository, MetaProperties, and AssignmentsHelper. Additionally, core infrastructure components like the EventQueue and DirectoryId have been relocated to this shared module, improving code organization and reducing duplication across server components.

server-common · high confidence

The \CallbackHandler\ and \EventHandler\ interfaces in the asynchronous producer package have been removed. These interfaces previously allowed users to inject custom logic for processing data before and after queueing, as well as to define how batched data is dispatched to Kafka servers. With their removal, the asynchronous producer no longer supports these specific callback and event handling extension points.

core/src/main/scala/kafka/javaapi/producer/async · high confidence

The \kafka.javaapi\ package has removed several legacy components, specifically the \Implicits\ helper object, the \MultiFetchResponse\ class, and the \ProducerRequest\ class. These files have been deleted from the codebase, meaning users relying on these specific Java API classes for producer requests, multi-fetch responses, or implicit conversions will need to migrate to the updated API.

core/src/main/scala/kafka/javaapi · high confidence

The legacy Scala classes \kafka.cluster.Broker\ and \kafka.cluster.Cluster\ have been removed from the core module. This change eliminates outdated data structures that were previously used to manage broker metadata and cluster topology, reflecting a shift towards modern Java-based implementations and improved internal architecture.

core/src/main/scala/kafka/cluster · high confidence

The \ErrorMapping\ utility object and several associated exception classes (\InvalidMessageSizeException\, \NoBrokersForPartitionException\) have been removed from the \kafka.common\ package. This change eliminates the previous bi-directional mapping between numeric error codes and specific Scala exceptions, indicating a shift away from that error-handling pattern in the Kafka server codebase.

core/src/main/scala/kafka/common · high confidence

The project's sbt 0.7.5 build configuration files (build.properties, KafkaProject.scala, Plugins.scala) have been removed. This change eliminates the legacy build system setup, which previously defined project structure, dependencies, and release tasks for the core, examples, perf, and contrib modules.

project · high confidence

The entire Clojure client implementation has been removed from the codebase. This includes the core client code (kafka.clj, buffer.clj, types.clj), serialization helpers (print.clj, serializable.clj), example scripts, project configuration, and associated unit tests. Users relying on the Clojure client will no longer have access to these features.

clients/clojure · high confidence

The build system for the C++ client has been cleaned up by removing several legacy build auxiliary scripts and Autoconf macros. Specifically, the files \config.guess\, \config.sub\, \depcomp\, \install-sh\, \ltmain.sh\, \missing\, and the \m4/\ directory containing \libtool.m4\, \ltoptions.m4\, \ltsugar.m4\, \ltversion.m4\, and \lt\~obsolete.m4\ have been deleted. These files were part of the older GNU build infrastructure (Autoconf/Automake/Libtool) and their removal indicates a shift away from these legacy build tools.

clients/cpp/build-aux · high confidence

The \perf\ directory has been cleaned up by deleting legacy configuration files (\.classpath\, \.project\) and the \report-html\ directory, which contained the Highcharts-based reporting tools and scripts. This removes the ability to generate HTML performance reports and run the associated simulator tests, indicating a shift away from this specific performance testing workflow.

perf · high confidence

The Gradle wrapper has been upgraded to version 9.4.1. This update ensures the build system uses the specified Gradle distribution, with the wrapper properties and checksums updated to match the new version. Additionally, new files have been added to the build configuration: a license header file for OpenAPI templates, a dependency check suppression file for specific CVEs ([CVE redacted], [CVE redacted]), a stylesheet for RAT output, a SpotBugs exclusion filter, and a README for upgrading the Gradle version.

gradle · high confidence

The project's \licenses\ directory has been updated to include or refresh license text files for several third-party dependencies, including CDDL, MIT, BSD, and Go licenses. Specifically, new license files were added for argparse, classgraph, eclipse-distribution, eclipse-public, hdrHistogram, jline, jopt-simple, jsr305, mock-oauth2-server, paranamer, pcollections, protobuf-java, re2j, slf4j, and zstd-jni. This ensures that all binary distributions of the software correctly attribute the open-source components they contain.

licenses · high confidence

2 commits adding/updating tests in clients/src/test/resources/serializedData; Add Kafka Streams 3.6 upgrade system tests; Add SmokeTest framework for Kafka Streams upgrade system tests; Add system test infrastructure and documentation; Add system tests for Kafka Streams upgrade to version 3.2; Add unit tests for RecordHeaders; Added 3.3.x Streams upgrade system tests; Added Java test utilities for Kafka security configurations; Added Kafka Streams smoke tests for version 3.7; Added Kafka Streams upgrade system tests for version 3.4; Added RemoteTopicCrudTest for tiered storage validation; Added Smoke Test Client and Driver for Streams Upgrade System Tests; Added SmokeTestClient, SmokeTestDriver, and related utilities for Streams upgrade system tests; Added and updated integration tests for Kafka server components; Added comprehensive test coverage for Kafka message serialization and protocol consistency; Added comprehensive unit tests for StorageTool; Added concurrency tests for the coordinator; Added integration test for group authorizer; Added integration tests for File Stream connectors; Added integration tests for Kafka Streams features; Added integration tests for SocketServer memory pool behavior; Added integration tests for client rebootstrap, topic creation, and metadata version upgrades; Added integration tests for consumer behavior; Added integration tests for custom quota callbacks; Added integration tests for dynamic connection and network thread configuration; Added integration tests for the AdminClient; Added mock security providers for testing; Added new Streams upgrade system tests for cooperative rebalance; Added smoke test infrastructure for Kafka Streams upgrade system tests; Added smoke test infrastructure for Kafka Streams upgrade testing; Added smoke test suite for Kafka Streams upgrade scenarios; Added smoke test suite for Kafka Streams upgrade testing; Added smoke test suite for Kafka Streams upgrade to version 3.9; Added system tests for Kafka Streams upgrade paths 4.2 and 4.3; Added system tests for Kafka Streams upgrade scenarios; Added test configuration for JSON converter; Added test coverage for new admin client internals; Added test fixtures for coordinator runtime; Added test service provider files for Connect connectors; Added test utilities for Kafka client testing; Added tests for ClientQuotaMetadataManager; Added tests for Connect CLI, connector client config override policies, and new BooleanConverter; Added tests for ConnectorUtils.groupPartitions; Added tests for JSON converter configuration and data conversion; Added tests for Kafka Resource types; Added tests for Kafka client internals; Added tests for Kafka clients metrics internals; Added tests for Kafka exception hierarchy; Added tests for Kerberos name parsing and rule processing; Added tests for List serialization and deserialization; Added tests for NonEmptyListValidator; Added tests for OAuth 2.0 client credentials and assertion-based authentication; Added tests for OAuth/Bearer authentication components; Added tests for SASL authentication failure handling and login manager caching; Added tests for SSL factory components; Added tests for config providers and path validation; Added tests for consumer configuration, metadata, and assignment logic; Added tests for consumer metrics managers; Added tests for internal record and batch components; Added tests for internal utility classes; Added tests for new and updated RPC request/response classes; Added tests for the new DescribeTopicPartitions RPC handler; Added tests for the new consumer event processing and reaper logic; Added tests for the rewritten MiniKDC implementation; Added tests for unsecured OAuth 2.0/Bearer authentication internals; Added unit and integration tests for leader epoch functionality; Added unit tests for Admin client result and configuration classes; Added unit tests for AlterConfigPolicy; Added unit tests for ApiKeyVersionsProvider; Added unit tests for BrokerMetadataPublisher; Added unit tests for Connect data types and schema validation; Added unit tests for Connect predicate transforms; Added unit tests for Connect storage converters; Added unit tests for Connector reconfiguration and initialization; Added unit tests for FileStream connectors and tasks; Added unit tests for JAAS context parsing and SASL extensions; Added unit tests for KRaft configuration validation; Added unit tests for KRaft manager initialization and configuration; Added unit tests for Kafka Connect header classes; Added unit tests for Kafka Connect single message transforms; Added unit tests for Kafka client common classes; Added unit tests for Kafka client metrics stats; Added unit tests for Kafka feature versioning; Added unit tests for Kafka metrics subsystem; Added unit tests for Kafka producer client components; Added unit tests for Kafka protocol serialization and type handling; Added unit tests for Kafka security authentication components; Added unit tests for KafkaDockerWrapper; Added unit tests for MirrorClient, ReplicationPolicy, and MirrorRecordVersion; Added unit tests for MirrorMaker 2 checkpoint and heartbeat components; Added unit tests for OAuth 2.0 Bearer authentication internals; Added unit tests for OAuth2/OAuthBearer credential refresh logic; Added unit tests for PLAIN SASL server authentication; Added unit tests for RuntimeLoggerManager; Added unit tests for SCRAM authentication internals; Added unit tests for SinkConnector and SinkRecord; Added unit tests for all compression algorithms; Added unit tests for client telemetry state transitions; Added unit tests for common utility classes; Added unit tests for logging utilities and test info helpers; Added unit tests for partition and cluster components; Added unit tests for producer internals components; Added unit tests for request handling and leader endpoint operations; Added unit tests for the Connect basic-auth extension; Added unit tests for the Group Coordinator Partition Writer; Added unit tests for the Kafka clients network module; Added unit tests for the Kafka protocol client library; Added unit tests for the Kafka share group broker components; Added unit tests for the LRU cache implementation; Added unit tests for the RackAwareReplicaSelector; Added unit tests for the coordinator runtime components; Added unit tests for the new FieldPath abstraction and FieldSyntaxVersion configuration; Added unit tests for transaction coordinator components; Expanded test coverage for Kafka client components; Expanded test coverage for consumer internals; Expanded test coverage for the Kafka server module; Introduce shared test base classes for authorizer and consumer integration tests; Migrated log subsystem unit tests to the storage module; New test fixtures for Connect integration testing; New test fixtures for client and admin testing; Removed KafkaLog4jAppender test suite; Removed Ruby client test suite; Removed ZooKeeper test infrastructure; Removed ZookeeperConsumerConnector test suite; Removed deprecated Scala JavaConversions usage in message set tests; Removed deprecated Scala producer tests; Removed deprecated Scala test utilities; Removed deprecated Scala-based consumer tests; Removed deprecated Scala-based producer and consumer integration tests; Removed obsolete Java API producer tests; Removed obsolete message set and message unit tests; Removed obsolete test utilities and scripts; Removed test log4j configuration file; Updated test logging configuration and added KDC test resources.

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 63.

• Code Health 75
• Architecture 99
• Maturity 75
• Readiness 67
• Security 52

• 300 commits — 271 feature/other, 29 fixes

• clients/src — 69 commits
• streams/src — 51 commits
• core/src — 35 commits
• group-coordinator/src — 29 commits
• streams/integration-tests — 10 commits
• docs/streams — 9 commits
• server/src — 9 commits
• (root) — 8 commits
• tests/kafkatest — 8 commits
• storage/src — 7 commits
• tools/src — 7 commits
• metadata/src — 5 commits
• raft/src — 5 commits
• share-coordinator/src — 5 commits
• clients/clients-integration-tests — 4 commits
• server-common/src — 4 commits
• streams/test-utils — 4 commits
• .github/workflows — 3 commits
• docs/security — 3 commits
• tests/docker — 3 commits

• fix: HOTFIX: fix compilation error (#22646)
• fix: KAFKA-16630: Fix flaky classic consumer poll test (#22787)
• fix: KAFKA-20253 Heartbeat CPU spin fix for Consumer protocol (#22836)
• fix: KAFKA-20292 [5/N]: Fix UpdatedMembersAndTargetAssignmentView to handle instance id changes (#22713)
• fix: KAFKA-20403 : streams - Fix stream threads interruptions (#21970)
• fix: KAFKA-20423: Fix flakiness of testWakeupWithFetchDataAvailable (#22364)
• fix: KAFKA-20663: Fix startup state manager close to release StateDirectory task lock (#22490)
• fix: KAFKA-20688: Fix RocksDB memory leak (#22557)
• fix: KAFKA-20694: Fix share read leader epoch sentinel (#22581)
• fix: KAFKA-20712 Fix transaction state name in log decoder output (#22615)
• fix: KAFKA-20733: Fix fetchResponseWithUnexpectedPartitionIsIgnored passing for wrong reason with CONSUMER protocol (#22651)
• fix: KAFKA-20753: Fix transactional status marker writes (#22716)
• fix: KAFKA-20760: Fix ClassCastException in KeyValueToTimestampedKeyValueByteStoreAdapter (#22742)
• fix: KAFKA-20781: Fix flaky testConsumerGroupHeartbeatWithRegexWithDifferentMemberAcls (#22816)
• fix: MINOR: Fix "does not exists" grammatical typos in messages and Javadoc (#22760)
• fix: MINOR: Fix DLQ records using a non-epoch clock for their timestamp (#22849)
• fix: MINOR: Fix ShareGroupCommandTest reset-offsets tests passing invalid --from-file scope combination (#22679)
• fix: MINOR: Fix TargetAssignmentBuilderBenchmark (#22484)
• fix: MINOR: Fix broken relative links in docs/ (#22813)
• fix: MINOR: Fix error message for streams group reset (#22600)
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

apache/kafka was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.