apache/pulsar
51.8
Adequate · 13 November 2026
513.3k
lines of production code
Java
primary language
4
measurements over time
What this system is
Features
Add BookKeeper-based storage backend for Pulsar Package Management
This change introduces a new storage implementation for the Pulsar Package Management service that persists package data using Apache BookKeeper via DistributedLog. It adds the \BookKeeperPackagesStorage\ class along with its configuration (\BookKeeperPackagesStorageConfiguration\) and provider (\BookKeeperPackagesStorageProvider\), enabling users to store and retrieve packages by leveraging BookKeeper's durability and replication features. The implementation handles initialization of the DistributedLog namespace, supports configurable authentication plugins and parameters for the BookKeeper client, and allows mapping arbitrary BookKeeper client properties, providing a robust, distributed backend option for package management.
pulsar-package-management/bookkeeper-storage/src/main/java/org/apache/pulsar/packages/management · high confidence
Add JClouds utility for selecting HTTP client implementations
A new utility class, ShadedJCloudsUtils, has been added to the jclouds-shaded module to manage Guice module configuration for JClouds. This change introduces support for selecting between the Apache HTTP Client and OkHttp implementations via system properties (pulsar.jclouds.use\_apache\_hc and pulsar.jclouds.use\_okhttp), allowing users to override the default HTTP driver behavior for tiered storage operations.
jclouds-shaded · high confidence
Add OpenID Connect (OIDC) authentication provider for the Pulsar broker
Introduces the \AuthenticationProviderOpenID\ component in the \pulsar-broker-auth-oidc\ module, enabling clients to authenticate using JSON Web Tokens (JWT) issued by OpenID Connect providers. The implementation verifies JWT signatures by retrieving public keys from the issuer's JWKS endpoint (or via Kubernetes API fallback), supports configurable allowed issuers and audiences, and handles algorithm validation for RS256/ES256 families. It includes caching for provider metadata and public keys to optimize performance, and exposes authentication metrics for monitoring.
pulsar-broker-auth-oidc · high confidence
Add TransactionConflictException and enable SpotBugs checks
The pulsar-transaction/common module now includes a new TransactionConflictException, which is thrown when a transaction attempts to acknowledge a message in an invalid state. Additionally, SpotBugs static analysis has been enabled for this module via a new findbugsExclude.xml configuration file, ensuring code quality checks are applied to transaction-related code.
pulsar-transaction/common · high confidence
Add Wireshark Lua plugin for Pulsar protocol dissection
Users can now dissect Apache Pulsar traffic in Wireshark by installing the new pulsar.lua plugin and PulsarApi.proto definition. The plugin automatically registers a dissector for TCP port 6650, parsing the Pulsar binary protocol (including command headers, checksums, and metadata) using the Protobuf dissector. A README provides setup instructions, including configuring Protobuf search paths and using the recommended display filter to isolate Pulsar packets while excluding ping/pong messages. Note: The plugin may cause crashes in Wireshark versions 4.0.0 and later.
wireshark · high confidence
Add Zstd compression support and data URL handling
The pulsar-common module now includes support for Zstandard (Zstd) compression via new wrapper classes (ZStdRawCompressor, ZStdRawDecompressor) that expose the underlying AirCompressor implementation. Additionally, a new data URL stream handler (DataURLStreamHandler) and factory (PulsarURLStreamHandlerFactory) have been added to the client API, enabling the use of data: URIs for inline content. The module also introduces an AllocatorType enum to support configurable Netty buffer allocation strategies (pooled, unpooled, adaptive) and exposes allocator statistics via ByteBufAllocatorStats.
pulsar-common · high confidence
Add batch data generator source and print sink connectors
The batch-data-generator module now includes a BatchDataGeneratorSource that generates synthetic person data using the jfairy library and a BatchDataGeneratorPrintSink that logs received records. This enables users to test batch source functionality by generating and printing sample data within Pulsar IO.
pulsar-io/batch-data-generator · high confidence
Add default configuration templates for Pulsar components
This change introduces a comprehensive set of default configuration templates for the Pulsar deployment stack within the Terraform-Ansible module. It adds templates for the Broker, Proxy, BookKeeper, ZooKeeper, and Functions Worker, providing standard settings for ports, TLS, authentication, and JVM options (including ZGC). Additionally, it includes systemd service unit files for each component and environment scripts, establishing the baseline configuration structure for new deployments.
deployment/terraform-ansible/templates · high confidence
Add kitchen-sink Docker Compose setup for learning and testing
A new 'kitchen-sink' example has been added to the docker-compose directory, providing a complete, multi-container Apache Pulsar cluster configuration. This setup includes three ZooKeeper nodes, three BookKeeper nodes, three Pulsar brokers, a proxy, a WebSocket service, and a function worker, allowing users to run a full cluster locally via \docker-compose up\. It is designed to help users understand how the individual components fit together, exposing specific endpoints like the broker REST interface on localhost:8080 and the broker service URL on localhost:6650, along with a script to initialize cluster metadata.
docker-compose · high confidence
Add local filesystem backend for package storage
Introduces a new \FileSystemPackagesStorage\ implementation for the Pulsar package management system, allowing users to store package artifacts on the local filesystem. The component respects a configurable storage path (defaulting to \packages-storage\) and includes path normalization and validation to prevent directory traversal attacks, ensuring that resolved paths remain within the designated storage directory.
filesystem-storage · high confidence
Add managed-ledger admin tool to access ml binary information
This change introduces a new managed-ledger admin tool that allows users to access managed ledger binary information. The implementation adds the \bin/proto\ directory containing the \MLDataFormats.proto\ definitions and the generated Python protobuf code (\MLDataFormats\_pb2.py\), enabling the tool to parse and display internal managed ledger data structures.
bin/proto · high confidence
Added Python function examples and packaging scripts
This change introduces a collection of Python example functions for Pulsar Functions, including Avro schema handling, custom object serialization, logging, publishing, and word counting, along with shell scripts to run and submit these functions. It also adds scripts and templates to generate Python packages for the Pulsar Function API.
pulsar-functions · high confidence
Added custom command example for Pulsar Admin CLI extensions
The pulsar-client-tools-customcommand-example module now includes a concrete implementation of the PIP-201 extensions mechanism, demonstrating how users can register custom CLI commands. The example provides a command factory that registers a 'customgroup' containing two commands: 'command1', which retrieves topic statistics via the admin client, and 'command2', which accepts various parameter types (string, integer, boolean, flag). This serves as a reference for extending the Pulsar Admin CLI with custom functionality.
pulsar-client-tools-customcommand-example · high confidence
Added local filesystem storage backend for package management
Users can now store Pulsar packages on the local filesystem. This change introduces the \FileSystemPackagesStorageProvider\ and its corresponding \FileSystemPackagesStorage\ implementation, allowing the package manager to persist package data directly to the local file system rather than relying on other storage backends.
pulsar-package-management/filesystem-storage/src/main/java/org/apache/pulsar/packages/management · high confidence
Athenz authentication now supports the v5 async binary protocol
The Athenz client authentication plugin now exposes a v5-native authentication body (AthenzAuthenticationV5) that allows the Pulsar client to drive Athenz credentials over the non-blocking binary protocol path. The existing synchronous v4 surface is preserved as a shim that supplies the current ZTS role token and HTTP header to the v5 body, ensuring backward compatibility while enabling the new async transport.
pulsar-client-auth-athenz · high confidence
Go Functions runtime restructured with new configuration, context, and example APIs
The Go Functions runtime has been restructured to improve usability and production readiness. A new configuration system in \conf/conf.go\ and \conf/conf.yaml\ allows functions to be initialized via YAML files or command-line flags, supporting settings for authentication, metrics, and processing guarantees. The runtime now exposes a \FunctionContext\ via Go's \context.Context\, enabling functions to access metadata (such as function ID and version), record custom metrics, and create output messages directly. Several example functions (\hello\, \exclamation\, \contextFunc\, etc.) demonstrate these new patterns. Additionally, generated protobuf code in \pb/\ has been updated to support the latest Pulsar wire protocol, including new processing guarantees like \MANUAL\ and subscription positions.
pulsar-function-go · high confidence
IOConfigUtils supports required, default, and sensitive field annotations
The pulsar-io/common module introduces IOConfigUtils, a new utility class that parses connector configuration maps into strongly-typed config objects while respecting FieldDoc annotations. It now enforces required fields, applies default values when present, and automatically resolves sensitive fields by fetching secrets via SourceContext or SinkContext, allowing connectors to securely load credentials without hardcoding them.
pulsar-io/common · high confidence
Initial AWS Terraform and Ansible deployment scripts
This change introduces the complete set of Terraform and Ansible configuration files for deploying Pulsar on AWS. The Terraform scripts (instances.tf, network.tf, security.tf, etc.) provision the necessary AWS infrastructure, including a VPC, subnets, NAT gateway, security groups, and EC2 instances for ZooKeeper, BookKeeper, Brokers, and Proxies, using the AWS provider version \>=3.3.0 and Terraform \>=0.13. An Ansible playbook (setup-disk.yaml) is included to configure disk performance and formatting on the BookKeeper nodes. Default variable values in terraform.tfvars configure a specific instance topology and region (us-west-2).
deployment/terraform-ansible/aws · high confidence
Initial release of the pulsar-client-api module
This change introduces the \pulsar-client-api\ module, establishing the public Java client API for Apache Pulsar. It defines the core interfaces and configuration classes users interact with, including \PulsarClient\, \Producer\, \Consumer\, and \Reader\, along with their respective builders (\ClientBuilder\, \ProducerBuilder\, \ConsumerBuilder\). The module also exposes essential configuration types such as \BatchReceivePolicy\, \CompressionType\, and \ConsumerCryptoFailureAction\, as well as the \Authentication\ and \AuthenticationFactory\ interfaces for configuring client security.
pulsar-client-api · high confidence
Initial repository structure and contributor documentation
Establishes the foundational repository configuration and documentation for Apache Pulsar. Adds \.asf.yaml\ to configure GitHub project settings, including enabling discussions, enforcing required pull request reviews and conversation resolution on the master branch, and protecting release branches. Introduces comprehensive contributor guides: \AGENTS.md\ for AI coding assistant guardrails, \ARCHITECTURE.md\ for module mapping and build infrastructure details, \CODING.md\ for Java style and concurrency conventions, \CONTRIBUTING.md\ for local development and testing workflows, and \SECURITY.md\ for vulnerability reporting procedures. Also includes \.gitattributes\ for line ending normalization, \.ratignore\ for license check exclusions, \codecov.yml\ for code coverage reporting configuration, and \doap\_Pulsar.rdf\ for project metadata.
(repo-wide) · high confidence
Introduce Athenz authentication provider for Pulsar brokers
This change adds the \AuthenticationProviderAthenz\ component to the Pulsar broker, enabling clients to authenticate using Athenz RoleTokens. The provider validates tokens against configured Athenz domains, verifies signatures using ZTS public keys, and supports a configurable time offset for token validity. It also integrates with OpenTelemetry to record authentication success and failure metrics, and includes unit tests to verify token validation logic.
pulsar-broker-auth-athenz · high confidence
Introduce BitRateUnit enum and authentication data classes for broker-common
The pulsar-broker-common module adds a BitRateUnit enum to standardize bitrate calculations across the broker, and introduces several new authentication data classes (AuthenticationDataAnonymous, AuthenticationDataForwarded, AuthenticationDataSubscription, and AuthenticationParameters) to support more granular and flexible authentication handling for HTTP and subscription-based requests.
pulsar-broker-common · high confidence
Introduce BouncyCastle-based message encryption and decryption
The Pulsar client now includes a new BouncyCastle implementation (MessageCryptoBc) for encrypting and decrypting messages. This change adds support for RSA, ECDSA, and ECIES algorithms, enabling end-to-end encryption for Pulsar messages. The implementation is designed to be FIPS-compliant by leveraging the BouncyCastle FIPS provider when available, ensuring that random number generation and cryptographic operations adhere to FIPS 140-2 standards. This provides users with a robust, standards-compliant option for securing message payloads in transit.
pulsar-client-messagecrypto-bc · high confidence
Introduce BrokerInterceptor plugin interface and NAR-based loading
Operators can now extend the Pulsar broker by installing custom interceptor plugins packaged as NAR archives. The new \BrokerInterceptor\ interface defines lifecycle hooks for client connections, message publishing, dispatching, acknowledgments, and transactions, while \BrokerInterceptorUtils\ and \BrokerInterceptors\ handle discovering, loading, and chaining these interceptors from a configured directory using isolated \NarClassLoader\ instances. A \ManagedLedgerInterceptorImpl\ is also provided to bridge broker-level interceptors with the managed ledger layer for entry metadata and payload processing.
pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept · high confidence
Introduce Extensible Load Manager framework and Broker Registry
This change introduces the foundational components for the new Extensible Load Manager (PIP-192) within the broker's load balancing extension package. It adds the \BrokerRegistry\ interface and its \BrokerRegistryImpl\ implementation, which handles registering broker lookup information to the distributed metadata store for discovery. It also defines the \ExtensibleLoadManager\ interface and its \ExtensibleLoadManagerImpl\ class, which serves as the core orchestrator for load balancing logic, including broker selection, ownership checks, and integration with state channels. Additionally, the \ExtensibleLoadManagerWrapper\ is added to bridge the new implementation with the existing \LoadManager\ interface, and \LoadManagerContext\ classes are introduced to provide shared configuration and data stores to load balancing components.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions · high confidence
Introduce LocalRunner for executing Pulsar Functions locally
This change introduces the LocalRunner component, enabling users to execute Pulsar Functions, Sources, and Sinks locally without a full Pulsar cluster. The implementation provides a CLI interface (via picocli) to configure function definitions, broker connections, and authentication, while managing the function lifecycle, runtime spawning, and Prometheus metrics export. It also includes a default log4j2 configuration and FindBugs exclusions to support local development and debugging workflows.
pulsar-functions/localrun · high confidence
Introduce Protocol Handler API for extensible broker protocols
The broker now supports loading additional messaging protocols via a new extensible API. This change introduces the \ProtocolHandler\ interface and supporting classes (\ProtocolHandlers\, \ProtocolHandlerUtils\) that scan a configured directory for NAR packages, load them using isolated \NarClassLoader\ instances, and initialize them to provide custom channel initializers. This allows users to integrate non-native protocols (such as Kafka) into the Pulsar broker without modifying core broker code.
pulsar-broker/src/main/java/org/apache/pulsar/broker/protocol · high confidence
Introduce RawReader API for accessing raw topic messages
The Pulsar client API now includes a new \RawReader\ interface and \RawMessage\ type, allowing users to read messages in their raw, uninterpreted form as stored in the managed ledger. This new capability provides methods to read the next raw message, seek to specific message IDs, check for message availability, and acknowledge messages cumulatively with optional properties. The \RawReader\ also supports pausing and resuming message consumption, giving users finer control over the reading process when interacting with raw topic data.
pulsar-broker/src/main/java/org/apache/pulsar/client/api · high confidence
Introduce SASL authentication provider with Kerberos support and HTTP role tokens
The broker now includes a new SASL authentication provider (\AuthenticationProviderSasl\) that enables multi-stage SASL authentication, primarily for Kerberos (GSSAPI) mechanisms. This implementation introduces a \PulsarSaslServer\ to handle the SASL handshake and a \SaslAuthenticationState\ to manage the session. For HTTP-based interactions, the broker can now validate role tokens via \SaslRoleToken\ and \SaslRoleTokenSigner\, which uses SHA-512 for signature verification to ensure token integrity. The provider also supports configuring allowed client IDs via a regex pattern and manages authentication state caching using Caffeine.
pulsar-broker-auth-sasl · high confidence
Introduce SchedulerManager and DistributedLog support in the worker
The worker now includes a new SchedulerManager component that handles the scheduling of function instances, managing the exclusive producer for assignment topics, and coordinating with the leader service and membership manager. This change also adds DistributedLog (DL) output stream support via DLOutputStream, enabling the worker to write to DistributedLog streams. These additions enhance the worker's ability to manage function assignments and integrate with DistributedLog for data persistence.
worker · high confidence
Introduce ServiceUnitStateChannel for extensible load manager bundle ownership
The extensible load manager now uses a new ServiceUnitStateChannel to track and coordinate namespace bundle ownership across brokers. This channel defines a strict state machine (Init, Free, Owned, Assigning, Releasing, Splitting, Deleted) with valid transitions for both SystemTopic and MetadataStore backends, and provides an interface for publishing assignment, unload, and split events. Two table-view implementations are included: one backed by a Pulsar system topic (ServiceUnitStateTableViewImpl) using ZSTD compression and a conflict resolver, and another backed by the local metadata store (ServiceUnitStateMetadataStoreTableViewImpl) with path validation and session-expired handling. The channel also exposes ownership queries, listener registration, and a scheduled ownership monitor to keep the local view consistent with the remote store.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel · high confidence
Introduce V5 async client API and scalable topic consumer types
The V5 client now exposes a new asynchronous API surface, including \AsyncProducer\, \AsyncQueueConsumer\, \AsyncStreamConsumer\, and \AsyncCheckpointConsumer\, which wrap the underlying scalable topic implementations to return \CompletableFuture\-based operations. This change also introduces the \CheckpointConsumerBuilder\ for creating checkpoint-based consumers and the \CheckpointV5\ class to manage position vectors across topic segments, enabling users to build non-blocking, scalable consumer applications with explicit checkpointing capabilities.
pulsar-client-v5 · high confidence
Introduce V5 client API with new consumer and producer interfaces
The V5 client API now provides a new set of interfaces for interacting with Pulsar topics, including PulsarClient, Producer, and Consumer types. A new CheckpointConsumer is introduced for connector frameworks (like Flink and Spark) that require external state management, allowing them to create atomic checkpoints of read positions across topic segments. The API also defines core message types (Message, MessageId, Messages) and builders (ProducerBuilder, CheckpointConsumerBuilder) with support for configuration options such as encryption policies, connection backoff, and authentication.
pulsar-client-api-v5 · high confidence
Introduce WebSocket-based performance testing and modernize CLI infrastructure
The testclient now supports WebSocket-based performance benchmarking via the new \websocket-producer\ subcommand, allowing users to measure producer latency and throughput over the WebSocket protocol. The CLI framework has been migrated to picocli, introducing a \gen-doc\ subcommand that automatically generates documentation for all perf tools, and the base command handler now supports case-insensitive enum values to maintain compatibility with existing v4-style flags. Additionally, the \BrokerMonitor\ tool has been updated to read from the new scalable load-manager metadata paths, and the \ManagedLedgerWriter\ now uses the modern metadata store API instead of the deprecated ZooKeeper connection string.
pulsar-testclient · high confidence
Introduce bucket-based delayed message index snapshotting to BookKeeper
The broker now persists delayed message index buckets as snapshots in BookKeeper ledgers, enabling recovery of delayed delivery state across broker restarts. This change adds the \BucketDelayedDeliveryTracker\ and supporting classes (\BucketSnapshotStorage\, \ImmutableBucket\, \MutableBucket\, \BucketDelayedMessageIndex\) to manage, serialize, and store delayed message indices as snapshot segments. It introduces new metrics (\pulsar\_delayed\_message\_index\_bucket\_total\, \pulsar\_delayed\_message\_index\_loaded\, \pulsar\_delayed\_message\_index\_bucket\_snapshot\_size\_bytes\, and operation counts/latencies) to monitor snapshot health and performance, and ensures delayed message tracking remains consistent during recovery and cleanup operations.
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket · high confidence
Introduce configurable ManagedLedger storage abstraction
The broker's storage layer now supports multiple ManagedLedger storage classes, allowing users to configure and select different storage implementations per topic or namespace via persistence policies. This change introduces new interfaces (\ManagedLedgerStorage\, \ManagedLedgerStorageClass\, and \BookkeeperManagedLedgerStorageClass\) that decouple the BookKeeper client from the ManagedLedger logic, enabling the initialization of multiple ManagedLedgerFactory instances and providing access to the underlying BookKeeper client and stats provider for advanced configuration and monitoring.
pulsar-broker/src/main/java/org/apache/pulsar/broker/storage · high confidence
Introduce core package management service and storage abstraction
This change introduces the core interfaces and implementation for the Pulsar package management service, enabling users to manage packages for functions, sinks, and sources. It defines the \PackagesManagement\ API for uploading, downloading, listing, and deleting packages, along with a pluggable \PackagesStorage\ abstraction that allows backing storage to be configured via providers. The implementation includes \PackagesManagementImpl\ and \PackageMetadataUtil\, which support both JSON and legacy Java serialization for package metadata, with a default preference for JSON to improve interoperability and security.
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management · high confidence
Introduce extensible load data store backed by Pulsar TableView
Added the \LoadDataStore\ interface and its \TableViewLoadDataStoreImpl\ implementation in the broker's load balance extensions store package, enabling the extensible load manager to persist and sync load metrics via a Pulsar topic. The new store supports async push/remove operations, local reads, and iteration, while using asynchronous close calls for the underlying producer and table view to prevent shutdown deadlocks.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/store · high confidence
Introduce extensible load manager framework with coordination-based leader election
The broker's load balancing system now supports a pluggable, extensible architecture. A new \LoadManager\ interface serves as the unified entry point, automatically instantiating either the legacy \SimpleLoadManagerImpl\, the \ModularLoadManager\, or the new \ExtensibleLoadManager\ based on configuration. The \ExtensibleLoadManager\ introduces a modular pipeline with dedicated interfaces for filtering brokers (\BrokerFilter\), ranking resources (\LoadRanker\), and determining split/shedding strategies. Crucially, it replaces the previous ZooKeeper-based leader election with a new \LeaderElectionService\ that uses the generic \CoordinationService\ interface, enabling more robust leader selection and allowing the load manager to function correctly even when the metadata service is temporarily unavailable.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance · high confidence
Introduce filesystem-based tiered storage offloader
Adds a new filesystem offloader implementation for tiered storage, allowing managed ledger data to be offloaded to and read from a generic filesystem (such as HDFS or local file systems) via the \FileSystemManagedLedgerOffloader\. This change introduces the \FileSystemLedgerOffloaderFactory\ and supporting classes to handle offloading operations, including metrics recording and read-back capabilities, along with the necessary service registration and test coverage.
tiered-storage/file-system · high confidence
Introduce metadata store migration admin API and new broker stats endpoints
The broker admin API now exposes a new \MetadataMigrationBase\ resource with endpoints to check migration status (\/status\) and initiate migration to a target store (\/start\), enabling administrators to manage the transition from ZooKeeper to a local metadata store. Additionally, a new \BrokerStatsBase\ resource provides REST endpoints to retrieve broker metrics, JVM MBeans, topic destinations, allocator stats, pending bookie operations, and load reports, offering deeper visibility into broker health and resource usage.
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl · high confidence
Introduce new Java Functions API interfaces and state store abstractions
The \pulsar-functions/api-java\ module now exposes a comprehensive set of core interfaces for building Pulsar Functions in Java. This includes the \Function\ and \WindowFunction\ interfaces for processing logic, along with \Context\ and \WindowContext\ to provide runtime information such as tenant, namespace, instance details, and user configuration. The update introduces a robust \Record\ interface for handling input data, including partition and event time metadata, and adds a pluggable state management system via \StateStore\, \ByteBufferStateStore\, and \CounterStateStore\ interfaces, enabling functions to persist and retrieve state asynchronously. Utility classes like \FunctionRecord\ and \IdentityFunction\ are also provided to simplify record construction and common use cases.
pulsar-functions/api-java · high confidence
Introduce new MetadataStore API and MetadataCache layer
This change introduces a new, evolving \MetadataStore\ API in the \pulsar-metadata\ module, providing a unified interface for metadata operations including reads, writes, and atomic read-modify-update actions with support for options like ephemeral nodes, sequential keys, secondary indexes, and partition keys. Alongside the store interface, a \MetadataCache\ layer is added to cache metadata entries with configurable refresh and expiration policies, exposing results via new \GetResult\ and \CacheGetResult\ types that include associated \Stat\ information. The update also adds supporting infrastructure such as \MetadataSerde\ for serialization, \MetadataEvent\ and \MetadataEventSynchronizer\ for cross-cluster event synchronization, \MetadataNodeSizeStats\ for payload size tracking, and \MetadataStoreTableView\ for in-memory table views that sync with the remote store.
pulsar-metadata · high confidence
Introduce new Prometheus metrics implementation in broker stats
The broker's Prometheus stats module now includes a new set of metric collection and export components, including \PrometheusMetricsProvider\, \PrometheusStatsLogger\, and specialized loggers like \DataSketchesOpStatsLogger\ and \DataSketchesSummaryLogger\. These classes enable the collection of operation statistics and summaries using Apache DataSketches for approximate quantile calculations, and expose them via the Prometheus text format. The implementation also adds support for custom gauges (\ObserverGauge\) and counters (\LongAdderCounter\), along with utility classes for label handling and text formatting, providing a more robust and performant foundation for Prometheus metric exposure in the Pulsar broker.
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics · high confidence
Introduce new protobuf definitions for Pulsar Functions configuration and status
This change adds new Protocol Buffer schema files (Function.proto, InstanceCommunication.proto, Request.proto) to define the data structures for Pulsar Functions, Sources, and Sinks. For users, this establishes the standardized serialization format for function details (including runtime, resources, and processing guarantees), instance communication (status, metrics, and health checks), and service requests. It also includes a unit test to verify that the default processing guarantee is set to AT\_LEAST\_ONCE.
pulsar-functions/proto · high confidence
Introduce non-persistent topic support in the broker
The broker now supports non-persistent topics, which deliver messages in-memory without durable storage. This change adds the core service components for this capability in the \pulsar-broker\ module, including \NonPersistentTopic\ for topic management, \NonPersistentSubscription\ for handling subscription logic, and specific dispatchers (\NonPersistentDispatcherMultipleConsumers\, \NonPersistentDispatcherSingleActiveConsumer\, and \NonPersistentStickyKeyDispatcherMultipleConsumers\) to manage message delivery for Shared, Exclusive/Failover, and Key\_Shared subscription types respectively. It also introduces \NonPersistentReplicator\ for cross-cluster replication and \NonPersistentSystemTopic\ for internal system topics, enabling users to utilize non-persistent topics with the same subscription semantics as persistent ones.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent · high confidence
Introduce pluggable broker selection and namespace bundle split strategies
The load balancer extensions now support configurable strategies for broker selection and namespace bundle splitting. A new \BrokerSelectionStrategy\ interface and factory allow administrators to choose how brokers are picked for bundle assignment, with \LeastResourceUsageWithWeight\ (which considers historical and short-term load with weights) and \RoundRobinBrokerSelectionStrategy\ provided as implementations. Additionally, a \NamespaceBundleSplitStrategy\ interface and its \DefaultNamespaceBundleSplitStrategyImpl\ implementation replace the previous bundle splitting logic, determining which bundles to split based on configurable thresholds for topics, sessions, message rate, and bandwidth.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy · high confidence
Introduce pluggable entry filter plugin framework
Added the \org.apache.pulsar.broker.service.plugin\ package, which provides the core infrastructure for loading and managing entry filters. This includes the \EntryFilter\ interface for defining filter logic, \EntryFilterProvider\ for discovering and loading filter NAR archives from a configured directory, and \EntryFilterWithClassLoader\ to ensure proper isolation and lifecycle management of filter classes. This change establishes the foundation for applying filters to message entries before they are dispatched to consumers.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/plugin · high confidence
Introduce pluggable proxy extension framework
The Pulsar Proxy now supports a pluggable extension mechanism that allows third-party protocol handlers and custom logic to be loaded dynamically via NAR packages. This change adds the \ProxyExtension\ interface and supporting classes in the \pulsar.proxy.extensions\ package, enabling extensions to define their own listening ports, channel initializers, and lifecycle hooks. The \ProxyExtensions\ class handles loading, validation, and initialization of these extensions from a configured directory, while \BrokerDiscoveryProvider\ exposes broker discovery capabilities to extensions. This allows users to extend the proxy's functionality without modifying core Pulsar code.
pulsar-proxy · high confidence
Introduce secrets provider framework for Pulsar Functions
This change adds the \pulsar-functions/secrets\ module, establishing a new extensible secrets management system for Pulsar Functions. It introduces the \SecretsProvider\ interface and two default implementations: \ClearTextSecretsProvider\ for basic cleartext handling and \EnvironmentBasedSecretsProvider\ which supports secret interpolation (e.g., \${secretName}\) by reading from environment variables. Additionally, it provides \SecretsProviderConfigurator\ implementations, including \KubernetesSecretsProviderConfigurator\ to automatically bind Kubernetes secrets as environment variables in function pods, and \DefaultSecretsProviderConfigurator\ for non-Kubernetes runtimes.
pulsar-functions/secrets · high confidence
Introduce tenant and namespace-level resource groups with rate limiting and quota management
The broker now supports Resource Groups, allowing tenants and namespaces to be assigned to named groups for centralized rate limiting and quota management. This change adds the core \ResourceGroupService\ and \ResourceGroup\ data structures, along with \ResourceGroupPublishLimiter\ to enforce publish rate limits per group. Configuration changes are handled by \ResourceGroupConfigListener\ and \ResourceGroupNamespaceConfigListener\, which sync group definitions and namespace assignments from the metadata store. Usage data is gathered and exchanged between brokers via \ResourceUsageTopicTransportManager\ using a system topic, and \ResourceQuotaCalculatorImpl\ computes local quotas based on global usage to ensure fair distribution of resources.
pulsar-broker/src/main/java/org/apache/pulsar/broker/resourcegroup · high confidence
Introduce transaction pending-ack store interfaces and OpenTelemetry metrics
The broker now exposes a new set of interfaces in the \pulsar-broker/transaction/pendingack\ package to manage transaction pending acknowledgments, including \PendingAckHandle\ for individual and cumulative acks, \PendingAckStore\ for persisting ack and commit/abort marks, and \TransactionPendingAckStoreProvider\ for instantiating store implementations. To support observability, \PendingAckHandleStats\ and \PendingAckHandleAttributes\ have been added to record commit/abort success and failure counts and to emit OpenTelemetry attributes (tenant, namespace, topic, subscription, partition) for transaction pending-ack operations.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack · high confidence
Introduces OpenTelemetry metrics for managed ledger cache and cursor operations
The managed-ledger module now exposes new OpenTelemetry metrics to replace legacy JMX counters. This includes \pulsar.broker.managed\ledger.cache.\\ metrics for cache entry counts, sizes, and operations, as well as \pulsar.broker.managed\ledger.cursor.\\ metrics for cursor operations. These metrics provide standardized observability for cache performance and cursor activity.
managed-ledger/src/main/java · high confidence
Introduces Pulsar IO core interfaces and annotations for connectors
This change adds the foundational API for Pulsar IO connectors in the \pulsar-io/core\ module. It defines the core interfaces \Source\, \Sink\, \BatchSource\, and \PushSource\ (along with their abstract implementations like \AbstractPushSource\) that connector developers implement to ingest or export data. It also introduces \SourceContext\ and \SinkContext\ to provide connectors with environment information and capabilities such as publishing messages, creating consumers, and managing state. Additionally, it adds the \BatchSourceTriggerer\ interface for batch source discovery and a set of annotations (\@Connector\, \@FieldDoc\, \@IOType\) to support metadata documentation and configuration generation for connectors.
pulsar-io/core · high confidence
Introduces auto split/merge policy and scalable topic consumer coordination infrastructure
This change adds the broker-side configuration and evaluation logic for PIP-483 auto split/merge policies, including the \AutoScaleConfig\ record for resolving namespace and topic overrides, the \AutoScalePolicyEvaluator\ for deciding when to split, merge, or rebucket segments based on load and consumer counts, and the \AutoScaleDecision\ types representing those actions. It also introduces the \ConsumerSession\ and \ConsumerAssignment\ classes that manage the lifecycle, connection state, and segment/bucket assignments for scalable topic consumers, along with \DagWatchSession\ for handling client-side metadata watches and synthetic layouts for legacy topics.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable · high confidence
Introduces broker-side transaction timeout tracking
The broker now includes a new implementation for tracking transaction timeouts via \TransactionTimeoutTrackerImpl\ and its factory. This component uses a Netty \HashedWheelTimer\ to monitor transaction lifecycles, automatically ending transactions that exceed their configured timeout by calling the \TransactionMetadataStoreService\. This adds the core scheduling logic required for the transaction timeout feature within the broker module.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/timeout · high confidence
Introduces pluggable BookKeeper client and extended Pulsar resources abstractions
The broker now uses a new \BookKeeperClientFactory\ interface and its \BookKeeperClientFactoryImpl\ to create BookKeeper clients, allowing for more flexible client initialization and configuration. Additionally, a \PulsarResourcesExtended\ interface and its \DefaultPulsarResourcesExtended\ implementation are introduced to provide pluggable extended resource management capabilities, such as custom topic listing strategies with property-based filtering.
pulsar-broker/src/main/java/org/apache/pulsar/broker · high confidence
Introduces protobuf definitions for delayed messages, schema registry, resource usage, and transaction pending acks
This change adds six new Protocol Buffer schema files to the broker's proto directory, establishing the serialization formats for several internal broker capabilities. Specifically, it defines the data structures for delayed message index buckets (DelayedMessageIndexBucketMetadata.proto and DelayedMessageIndexBucketSegment.proto), the schema registry's storage and format specifications (SchemaRegistryFormat.proto and SchemaStorageFormat.proto), resource usage reporting (ResourceUsage.proto), and transaction pending acknowledgments (TransactionPendingAck.proto). These definitions enable the broker to correctly serialize and deserialize metadata for delayed delivery, schema versioning and storage, tenant/namespace resource tracking, and transactional message acknowledgment states.
pulsar-broker/src/main/proto · high confidence
Introduces v2 metadata models for segmented transaction buffer snapshots
This change adds a new set of metadata classes in the \pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/metadata/v2\ package to support the segmented snapshot system. Specifically, it introduces \TransactionBufferSnapshotIndex\ and \TransactionBufferSnapshotIndexes\ to track ledger and entry locations for both aborted transaction markers and snapshot segment data, alongside \TransactionBufferSnapshotSegment\ and \TxnIDData\ to represent segment content and transaction identifiers. These models provide the structural foundation for the new segmented index system topic (\\_\_transaction\_buffer\_snapshot\_segments\), enabling more granular storage and retrieval of transaction buffer state.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/metadata · high confidence
New BundlesQuotas cache class for managing namespace bundle resource quotas
A new \BundlesQuotas\ class has been added to the broker cache package to handle resource quota operations for namespace bundles. This component provides methods to get, set, and reset resource quotas (such as message rates and bandwidth) for specific bundles, falling back to a defined initial default quota when no specific quota is set. It acts as a cache layer that delegates quota persistence and retrieval to the underlying \LoadBalanceResources\ service.
pulsar-broker/src/main/java/org/apache/pulsar/broker/cache · high confidence
New CLI utility for managing authentication tokens
A new \TokensCliUtils\ command-line interface has been added to the broker module, enabling users to create secret keys, generate public/private key pairs, and issue or inspect JWT tokens directly from the command line. This tool supports multiple signature algorithms (such as HS256 and RS256) and allows specifying token subjects, expiry times, and custom headers, streamlining the setup and management of token-based authentication.
pulsar-broker/src/main/java/org/apache/pulsar/utils/auth · high confidence
New Data Generator Source and Print Sink connectors
The data-generator module now includes a new source connector that generates synthetic test data (using the jfairy library) and a new print sink connector that logs received records and acknowledges them. This adds the \data-generator\ source and \DataGeneratorPrintSink\ sink to the available Pulsar IO connectors, allowing users to easily generate and inspect test data streams.
pulsar-io/data-generator · high confidence
New Java function examples for windowing, async processing, schemas, and admin context
The java-examples module now includes a comprehensive set of new reference implementations demonstrating key Pulsar Functions capabilities. Users can now study windowing aggregation with AddWindowFunction and CommaWindowFunction, asynchronous processing patterns via AsyncContextFunction and JavaNativeAsyncExclamationFunction, and custom serialization/deserialization using ByteBufferSerDe, CustomBaseSerDe, and CustomDerivedSerDe. The examples also cover schema handling with AutoSchemaFunction and AvroSchemaTestFunction, configuration-driven behavior in ConfigBasedAppendFunction, and advanced context usage such as exposing the PulsarAdmin client in CursorManagementFunction for subscription cursor reset. Additional examples illustrate initialization lifecycle (InitializableFunction), logging (LoggingFunction), host/instance ID appending, integer addition, custom object transformations, and publishing to topics with MergeTopicFunction and PublishFunction.
pulsar-functions/java-examples · high confidence
New LogIndexLagBackoff utility for transaction backoff calculations
A new \LogIndexLagBackoff\ utility class has been added to the transaction utilities package. This class implements an exponential backoff strategy for transaction metadata positions, allowing users to configure minimum lag, maximum lag, and an exponent to calculate appropriate backoff delays based on index counts. This supports the optimization of pending acknowledgment stores in Multi-Leader Permits (MLP) scenarios by providing a configurable mechanism to manage retry delays.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/util · high confidence
New OpenTelemetry-based broker stats classes for consumer, producer, replication, and topic metrics
The broker stats module now includes several new OpenTelemetry stats generators that expose detailed metrics for consumers, producers, replication, and topics. These new classes (OpenTelemetryConsumerStats, OpenTelemetryProducerStats, OpenTelemetryReplicatedSubscriptionStats, OpenTelemetryReplicatorStats, and OpenTelemetryTopicStats) provide granular, attribute-rich metrics for monitoring and observability. Additionally, BrokerOperabilityMetrics introduces new counters for topic load failures, non-recoverable ledgers/entries skipped, and connection metrics, while OpenTelemetryMessageFinderStats adds metrics for timestamp-based message position searches. These changes enhance the broker's observability by providing more detailed and structured metrics for various broker operations.
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats · high confidence
New Prometheus metrics provider for ZooKeeper using Jetty 12
This location introduces a new Prometheus metrics provider for ZooKeeper that leverages Jetty 12, including the provider implementation, package documentation, and comprehensive test coverage for configuration validation, JVM info export, and metric collection behavior. Also in this area: 1 commit adding/updating tests in jetty-upgrade/zookeeper-prometheus-metrics.
jetty-upgrade/zookeeper-prometheus-metrics · high confidence · unverified
New REST API endpoints for producing messages to topics
The broker now exposes new REST endpoints under the \/rest/v1\ path (specifically \Topics.java\) allowing clients to produce messages to persistent and non-persistent topics, including specific partitions. This implementation introduces \TopicsBase\ to handle the core publishing logic, schema registration, and authorization checks, and utilizes a new \RestMessagePublishContext\ to manage publish latency recording and memory recycling via Netty's Recycler, replacing previous ad-hoc context handling.
pulsar-broker/src/main/java/org/apache/pulsar/broker/rest · high confidence
New SPI for framework-managed HTTP clients in authentication plugins
This change introduces the \pulsar-http-client-api\ module, which defines the SPI for authentication plugins to perform HTTP requests (such as OAuth2 token exchanges) using a framework-managed client. Instead of constructing private HTTP clients, plugins now obtain instances via \AuthenticationInitContext.httpClientFactory()\, ensuring shared event-loop, DNS, and TLS-resource integration. The API provides immutable \HttpRequest\ and \HttpResponse\ models, a \PulsarHttpClient\ interface for asynchronous execution, and a \PulsarHttpClientConfig\ builder that allows plugins to specify timeouts, user-agent, and a \TlsPurpose\ for TLS material resolution without handling raw certificates.
pulsar-http-client-api · high confidence
New StreamOutputter class for ZooKeeper admin streaming
A new StreamOutputter class has been added to the ZooKeeper admin module to handle streaming data output. This class implements the CommandOutputter interface, setting the content type to application/octet-stream and copying response data to the output stream. It includes logging for streaming exceptions, capturing the client IP address for context.
zookeeper-with-patched-admin · high confidence
New admin API interfaces and options for bookies, brokers, clusters, functions, namespaces, and topic listing
The pulsar-client-admin-api module now exposes a comprehensive set of new Java interfaces and configuration options for administrative operations. New interfaces include Bookies (for rack placement management), BrokerStats (for broker metrics and allocator stats), Brokers (for active broker lists and leader info), Clusters (for cluster configuration and peer management), Functions (for function lifecycle and stats), MetadataMigration (for cluster metadata migration status), and Lookup (for topic-to-broker resolution). Additionally, the Namespaces interface has been expanded with topic listing options (ListNamespaceTopicsOptions, ListTopicsOptions) and detailed namespace policy management. A new GetStatsOptions class allows fine-grained control over topic statistics retrieval (e.g., precise backlog, subscription backlog size, excluding publishers/consumers). These changes provide a more structured and feature-rich programmatic admin API for managing Pulsar cluster components.
pulsar-client-admin-api · high confidence
New broker and top-bundles load reporters for the extensible load manager
Added the \BrokerLoadDataReporter\ and \TopBundleLoadDataReporter\ components in the \pulsar-broker/loadbalance/extensions/reporter\ package to support the new extensible load manager. The broker reporter collects and publishes system resource usage, message rates, and bundle counts to the load data store, applying configurable thresholds for update frequency and change magnitude. The top-bundles reporter identifies and reports the highest-loaded bundles (top-K) and handles tombstoning of load data when bundles are released, split, or owned by the broker, ensuring the load balancer has accurate, up-to-date metrics for scheduling decisions.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter · high confidence
New broker-tool CLI for broker-specific operations
A new \broker-tool\ command-line interface has been added to the Pulsar broker, allowing operators to perform specific operations on a single broker instance. This tool, built using picocli, includes a \load-report\ subcommand that collects and displays real-time resource usage metrics (CPU, memory, direct memory, and bandwidth) for the broker, and a \gen-doc\ subcommand to generate documentation for the tool's commands.
pulsar-broker/src/main/java/org/apache/pulsar/broker/tools · high confidence
New cron-based and immediate batch source triggerers
The batch-discovery-triggerers module now includes two new trigger implementations: CronTriggerer, which schedules batch source executions based on a cron expression (supporting standard six-field syntax and Spring-style macros like @daily), and ImmediateTriggerer, which fires a single trigger event upon start. These replace the previous Spring Framework-based implementation, ensuring backward-compatible firing times while improving thread naming and error handling for batch source configurations.
pulsar-io/batch-discovery-triggerers · high confidence
New environment-driven configuration scripts for Docker containers
The \docker/pulsar/scripts\ directory now includes a suite of new Python and Bash scripts to manage container configuration via environment variables. \apply-config-from-env.py\ and its wrapper \apply-config-from-env-with-prefix.py\ allow users to update properties files by setting prefixed environment variables (defaulting to \PULSAR\PREFIX\\), with built-in sanitization for sensitive keys like passwords. \gen-yml-from-env.py\ generates YAML configuration from environment variables, supporting type conversion for integers and sets, and allows overriding the container factory (k8s, process, or thread). \update-ini-from-env.py\ and \update-rocksdb-conf-from-env.py\ enable tuning INI-style configuration files, specifically mapping environment variables to RocksDB settings such as log paths, compression, and cache sizes. Additionally, \generate-zookeeper-config.sh\ automates ZooKeeper server ID generation based on hostname, \pulsar-zookeeper-ruok.sh\ provides a health check for ZooKeeper, and \watch-znode.py\ offers a utility to wait for, create, or check the existence of ZooKeeper znodes.
docker/pulsar/scripts · high confidence
New extensible broker filters for load balancing decisions
The broker load balancer now uses a new extensible filter chain (BrokerFilter) to select candidate brokers, introducing several new filtering strategies: AntiAffinityGroupPolicyFilter respects anti-affinity group policies, BrokerIsolationPoliciesFilter enforces tenant isolation policies, BrokerLoadManagerClassFilter ensures brokers share the same load manager class, BrokerMaxTopicCountFilter excludes brokers exceeding the configured topic limit, and BrokerVersionFilter (when PreferLaterVersions is enabled) excludes older broker versions to favor the latest release. These filters run asynchronously to refine the set of available brokers before load-balancing decisions are made.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/filter · high confidence
New extensible load balancer data models for broker metrics and lookups
The broker now introduces new data structures in the \pulsar.broker.loadbalance.extensions.data\ package to support the extensible load manager. \BrokerLoadData\ replaces the legacy \LocalBrokerData\ for tracking local broker metrics, including system resource usage (CPU, memory, direct memory, bandwidth) and message throughput, while introducing computed features like weighted maximum exponential moving average (EMA) for more nuanced load decisions. \BrokerLookupData\ is a new record that consolidates broker connection details, advertised listeners, and protocol information, enabling lookup operations based on specific listener names and properties. Additionally, \TopBundlesLoadData\ provides a container for tracking the load statistics of the top bundles on a broker, supporting finer-grained bundle-level load balancing.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/data · high confidence
New extensible load balancing scheduler framework for bundle splits and unloads
The broker now introduces a new scheduling package (\org.apache.pulsar.broker.loadbalance.extensions.scheduler\) that decouples load management tasks into pluggable schedulers. This includes \SplitScheduler\ for handling automatic namespace bundle splits and \UnloadScheduler\ for managing load shedding, which utilizes the new \NamespaceUnloadStrategy\ interface (implemented by \TransferShedder\) to determine which bundles to unload. This change provides the structural foundation for the PIP-192 extensible load manager, allowing for more flexible and configurable load balancing behaviors compared to the previous monolithic approach.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler · high confidence
New extension API interfaces for custom Pulsar Admin CLI commands
This change introduces the \org.apache.pulsar.admin.cli.extensions\ package in the \pulsar-client-tools-api\ module, providing the core interfaces that enable third-party extensions to the Pulsar Admin CLI. It adds \CustomCommandFactory\ as the entry point for generating command groups, \CustomCommandGroup\ for organizing commands, and \CustomCommand\ for defining individual command logic, parameters, and execution context. Supporting types include \CommandExecutionContext\ for accessing the PulsarAdmin instance and configuration, \ParameterDescriptor\ for defining command arguments, and \ParameterType\ for specifying argument data types (STRING, INTEGER, BOOLEAN, BOOLEAN\_FLAG).
pulsar-client-tools-api · high confidence
New in-memory BookKeeper and ZooKeeper test mocks
The testmocks module now includes a suite of in-memory mocks for BookKeeper and ZooKeeper to simplify unit testing. This adds PulsarMockBookKeeper, which keeps all ledger data in memory and supports configurable entry delays and read handle interception via PulsarMockReadHandleInterceptor, along with PulsarBookKeeperTestClient and PulsarBookKeeperTestStatsProvider for accessing internal members and tracking in-memory statistics. It also introduces MockZooKeeper, a ZooKeeper implementation that supports ephemeral nodes, persistent watches, and different sessions, and ServerTester to encapsulate Bookie test objects with mock unclean shutdown detection.
testmocks · high confidence
New instance components for windowing, state, and V5 client support
The instance module introduces several new capabilities: a bounded, expiring ProducerCache that flushes and closes producers asynchronously to manage resources; a BookKeeper-backed state store implementation (BKStateStoreImpl) for function state persistence; a Prometheus-compatible metric registry (FunctionCollectorRegistryImpl); a V5MessageIdAdapter to bridge V5 client message IDs with the v4 API; and core windowing infrastructure including WindowImpl and WatermarkTimeEvictionPolicy for time-based event windowing.
instance · high confidence
New internal client implementation classes for strategic two-phase compaction
The broker module now includes several new internal client implementation classes in \org.apache.pulsar.client.impl\ to support the StrategicTwoPhaseCompactor. These include \CompactionReaderImpl\ for durable consumption of compacted messages, \RawReaderImpl\ for raw message reading with pause/resume capabilities, \RawMessageImpl\ for serializing/deserializing raw messages, \RawBatchConverter\ for extracting and rebatching raw batch messages, and \RawBatchMessageContainerImpl\ for handling raw batch encryption and serialization. These classes provide the underlying client-side infrastructure required for the new compaction strategy.
pulsar-broker/src/main/java/org/apache/pulsar/client/impl · high confidence
New load balancer extension managers for bundle splitting, unloading, and migration redirection
The broker now includes dedicated managers in the load balance extensions package to handle specific lifecycle events for namespace bundles. The SplitManager tracks in-flight bundle split requests with configurable timeouts and failure counters, while the UnloadManager manages bundle unloading operations, tracking latency metrics for assign, release, disconnect, and unload states via Prometheus histograms. Additionally, the RedirectManagerForLoadManagerMigration enables automatic redirection of lookup requests to brokers running the expected load manager class during migration scenarios, and the StateChangeListener interface provides a standardized hook for handling service unit state changes.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/manager · high confidence
New load balancing components and strategies introduced
The load balancing module now includes several new implementation classes: AvgShedder, a new load shedding strategy; LeastResourceUsageWithWeight and LeastLongTermMessageRate, new broker placement strategies; BrokerLoadManagerClassFilter and BrokerVersionFilter, new broker filters for version and class compatibility; BundleRangeCache and BundleSplitterTask, supporting cache and splitting logic; GenericBrokerHostUsageImpl and LinuxBrokerHostUsageImpl, implementations for gathering host resource usage metrics.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl · high confidence
New load-balancing model classes for bundle splits and unloads
The broker's load-balancing extension now includes new data models and metrics counters to track bundle split and unload operations. Specifically, the \SplitDecision\, \SplitCounter\, \Split\, and \AssignCounter\ classes provide structured tracking of split outcomes (success/failure) and reasons (topics, sessions, bandwidth, admin), while \UnloadDecision\, \UnloadCounter\, and \Unload\ handle unload lifecycle metrics (success, skip, failure) with detailed breakdowns by reason (overloaded, underloaded, admin, cooldown, etc.). These models expose granular metrics such as \brk\_lb\_bundles\_split\_total\, \brk\_lb\_unload\_bundle\_total\, and resource usage statistics, enabling better observability into the extensible load manager's decision-making process.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/models · high confidence
New managed ledger metrics classes expose cursor, cache, and ledger statistics
The broker now exposes detailed managed-ledger metrics through three new classes in the \pulsar-broker/stats/metrics\ package: \ManagedCursorMetrics\ (tracking per-cursor stats like non-contiguous deleted message ranges and persistence success/error counts), \ManagedLedgerCacheMetrics\ (exposing cache hit/miss rates, throughput, entry counts, and allocator pool usage), and \ManagedLedgerMetrics\ (aggregating ledger-level stats such as add/read entry rates, errors, backlog size, and latency/size buckets by namespace). These classes replace direct casts to internal implementation classes with interface-based access via \PulsarService\ and \ManagedLedgerFactoryMXBean\, and include a refactored \AbstractMetrics\ base class with pre-calculated bucket keys to reduce object allocation during metric collection.
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/metrics · high confidence
New metadata node size statistics and local bookkeeper ensemble components
The broker now includes \DefaultMetadataNodeSizeStats\ to track and report maximum sizes and child counts for specific ZooKeeper metadata path types (such as clusters, tenants, topics, and schemas), enabling better visibility into metadata store usage. Additionally, \LocalBookkeeperEnsemble\ is introduced to manage local BookKeeper instances for testing and standalone modes, supporting configurable stream storage ports and improved resource cleanup.
pulsar-broker/src/main/java/org/apache/pulsar/zookeeper · high confidence
New metadata-driven transaction buffer for segment topics
The broker now uses a new \MetadataTransactionBuffer\ implementation for \segment://\ topics (PIP-473), which stores transaction state in the local metadata store instead of relying solely on the transaction buffer ledger. A \DispatchingTransactionBufferProvider\ automatically routes segment topics to this new implementation while keeping persistent topics on the legacy \TopicTransactionBuffer\. This change introduces a new transaction layout under \/txn/\ in the metadata store, including per-segment watermark and aborted-transaction records, to improve durability and recovery for scalable topics.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl · high confidence
New metadata-store-backed transaction data layer for scalable topics
Introduces the PIP-473 transaction metadata data layer, which replaces the legacy transaction metadata storage with a schema-versioned, JSON-based layout persisted in the metadata store. This change adds a typed façade (TxnMetadataStore) and supporting record types (TxnHeader, TxnOp, TxnEvent, AbortedTxnRecord, SegmentWatermark) that use secondary indexes for efficient lookups (e.g., by segment, subscription, deadline, and final state) and partition keys for data locality. It also defines the new path conventions under /txn/ and the TxnIds encoding to ensure path-safe transaction ID representation, laying the foundation for the metadata-driven transaction coordinator. Also in this area: 7 commits (6 fixes) fixing pulsar-broker/src/main.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/coordinator, pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/metadata · medium confidence
New pulsar-cli-utils module with picocli-based CLI utilities
The new pulsar-cli-utils module provides shared utilities for the Pulsar CLI tools, including client API selection logic (V4 vs V5) based on topic domains, picocli type converters for byte units and relative time, value validation helpers, and option group management for client-specific help sections.
pulsar-cli-utils · high confidence
New release, versioning, and code-style tooling scripts
The \src\ directory now includes a set of new shell scripts and configuration files to streamline development and release workflows. \set-project-version.sh\ and \get-pulsar-version.sh\ manage the project version in \gradle.properties\ and update the Terraform deployment configuration. \stage-release.sh\ automates the creation of source archives via \git archive\ and copies binary distributions, while \sign-release.sh\ handles GPG and SHA-512 signing. Additionally, \update\_python\_protobuf\_stubs.sh\ (and its Docker wrapper) generates Python gRPC/Protobuf stubs, and \idea-code-style.xml\ provides an IntelliJ IDEA code style profile, replacing the previous Eclipse \formatter.xml\.
src · high confidence
New servlet plugin interface for broker service access
The broker now exposes a new \AdditionalServletWithPulsarService\ interface within the web plugin servlet package. This interface extends the existing \AdditionalServlet\ contract and adds a \setPulsarService\ method, allowing custom servlets to receive a reference to the running \PulsarService\ instance for deeper integration with broker internals.
pulsar-broker/src/main/java/org/apache/pulsar/broker/web/plugin · high confidence
New transaction pending-ack store implementations and dispatcher
The broker now includes new implementations for transaction pending-ack storage: \MetadataPendingAckStore\ (for segment topics, using the metadata store), \MLPendingAckStore\ (the legacy ManagedLedger-based store), and \InMemoryPendingAckStore\ (a no-op stub). A new \DispatchingTransactionPendingAckStoreProvider\ routes requests to the appropriate store based on the topic type, enabling segment subscriptions to use the metadata-driven approach while keeping persistent topics on the legacy path.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl · high confidence
New transaction recovery tracker implementation
The broker now includes a new \TransactionRecoverTrackerImpl\ class within the \pulsar.broker.transaction.recover\ package. This component tracks transactions in open, committing, and aborting states during metadata store recovery, ensuring that pending transactions are correctly finalized (committed or aborted) when the system restarts or recovers from a failure.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/recover · high confidence
New utility classes for Pulsar Functions runtime configuration and execution
This change introduces a suite of new utility classes in the pulsar-functions/utils module to support the runtime execution and configuration of Pulsar Functions. Actions.java provides a framework for defining and executing chained actions with retry logic and success/failure callbacks. BatchingUtils.java and CryptoUtils.java handle the conversion of batching and encryption configurations between internal objects and protobuf specs. ClientApiResolver.java determines whether to use the V4 or V5 Pulsar client based on topic types (e.g., scalable topic:// vs persistent://) and validates compatibility. FunctionCommon.java and FunctionConfigUtils.java provide core logic for resolving function types, converting runtime configurations, and validating function packages. FunctionFilePackage.java manages the loading of function code from NAR archives or JAR files, including classloader creation. ComponentTypeUtils.java and Exceptions.java offer helper methods for component type identification and exception handling. FunctionInstanceId.java parses fully qualified instance names. These utilities collectively enable the runtime to correctly configure, load, and execute functions with support for new client APIs, batching, and encryption features.
pulsar-functions/utils · high confidence
New utility classes for caching, rate limiting, and statistics output
The broker now includes several new utility classes in the \org.apache.pulsar.utils\ package to support internal operations. \SimpleCache\ provides a scheduled, time-expiring cache with configurable timeouts and expiration callbacks. \TimedSingleThreadRateLimiter\ implements a time-windowed rate limiter that can be opened or closed for specific durations. \StatsOutputStream\ improves the performance of statistics serialization by using \ArrayDeque\ instead of \Stack\ for managing separators. Additionally, \ConcurrentBitmapSortedLongPairSet\ offers a concurrent set for long pairs, supporting unsigned values up to 2^32 on the right side. These utilities are foundational components for broker performance and state management.
pulsar-broker/src/main/java/org/apache/pulsar/utils · high confidence
New v2 Admin REST API endpoints for bookies, brokers, clusters, functions, and metadata migration
The v2 admin REST API surface in \pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2\ is expanded with new resource classes that expose administrative capabilities for managing bookie rack placement (\Bookies\), broker stats and resource availability (\BrokerStats\), cluster configuration (\Clusters\), Pulsar Functions lifecycle (\Functions\), and metadata store migration (\MetadataMigration\). These endpoints allow administrators to query and update bookie rack configurations, retrieve broker resource availability reports, manage cluster settings, deploy and monitor functions, and initiate metadata migrations, all accessible via the v2 admin path structure.
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2 · high confidence
New v2 REST API endpoints for topic lookup and bundle resolution
The broker now exposes a new v2 REST API under the \/v2/topic\ path, providing endpoints to retrieve the owner broker for a specific topic and to identify the namespace bundle a topic belongs to. These endpoints support asynchronous responses and allow clients to specify the target listener via the \listenerName\ query parameter or the \X-Pulsar-ListenerName\ header, enabling more flexible routing in multi-listener configurations.
pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup/v2 · high confidence
New v3 Admin REST API endpoints for Functions, Sources, Sinks, Packages, and Transactions
This change introduces the \pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3\ package, providing a new set of admin REST API endpoints. The \Functions\, \Sources\, and \Sinks\ endpoints are added, with the singular \Source\ and \Sink\ paths marked as deprecated in favor of the plural \Sources\ and \Sinks\ paths. Additionally, new endpoints are exposed for managing packages (\Packages\) and for transaction administration (\Transactions\), including operations to list coordinators, retrieve coordinator and buffer stats, and manage transaction states. These endpoints utilize the \jakarta.ws.rs\ API and Swagger v3 annotations for documentation.
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v3 · high confidence
New v5 TLS SPI module introduces purpose-driven factory and configuration model
The pulsar-tls-factory-api module introduces a new Service Provider Interface (SPI) for TLS, replacing the legacy PIP-337 SSL factory with a purpose-driven model. This change adds the PulsarTlsFactory interface, which allows components to request fully configured TLS objects (such as Netty's SslContext or Jetty's SslContextFactory) based on a specific TlsPurpose (e.g., client, broker, web). It also introduces TlsPolicy as the unified, user-facing configuration value supporting both PEM and keystore formats, along with supporting types like TlsPurpose, TlsEndpoint, and TlsHandle to manage lifecycle and reloads.
pulsar-tls-factory-api · high confidence
Pluggable authentication providers for Pulsar Functions
The Functions runtime now supports pluggable authentication providers, allowing users to configure how function instances authenticate with the Pulsar cluster. This change introduces a new \FunctionAuthProvider\ interface and specific implementations for Kubernetes environments, including \KubernetesSecretsTokenAuthProvider\ (which mounts token secrets to function pods) and \KubernetesServiceAccountTokenAuthProvider\ (which supports OIDC via Kubernetes Service Account token projection). This enables more flexible and secure authentication strategies for functions running on Kubernetes.
pulsar-functions/runtime · high confidence
Pluggable topic bundle assignment and bundle split algorithms
The broker now supports configurable strategies for assigning topics to bundles and splitting overloaded bundles. A new \TopicBundleAssignmentStrategy\ interface and factory allow users to select different assignment algorithms (defaulting to consistent hashing) via configuration, while the \NamespaceBundleSplitAlgorithm\ interface introduces multiple split strategies—including range-based, topic-count-based, specified positions, and flow/QPS-based splitting—enabling more granular control over namespace bundle distribution and load balancing.
pulsar-broker/src/main/java/org/apache/pulsar/common · high confidence
Transaction coordinator adds buffered writer metrics and batch position support
The transaction coordinator now exposes Prometheus metrics for the transaction log buffered writer, including histograms for batch record counts, batch sizes, and oldest-record delay, plus counters for flush triggers (max records, max size, max delay, and large single data). This allows users to monitor and tune batch flush behavior. Additionally, the coordinator introduces a new sequence ID generator that persists and recovers the maximum local transaction ID via ManagedLedger properties and last-ledger metadata, and adds a TxnBatchedPositionImpl class to track batch index and size within transaction positions, supporting the new batched write feature.
coordinator · high confidence
Transaction coordinator interfaces and in-memory store implementation
The transaction coordinator module introduces the core abstractions for transaction metadata management, including the \TransactionMetadataStore\ interface for storing and querying transaction state, the \TransactionLog\ interface for persisting transaction operations, and supporting components like \TransactionCoordinatorID\, \TransactionTimeoutTracker\, and \TransactionRecoverTracker\. It also provides an in-memory implementation (\InMemTransactionMetadataStore\) and its provider, enabling the transaction coordinator to function with a lightweight, non-persistent backend for metadata storage.
pulsar-transaction/coordinator · high confidence
Removals
Removal of Apache Storm integration adapters
The \pulsar-storm\ module has been removed, deleting the \PulsarSpout\, \PulsarBolt\, and their associated configuration classes (\PulsarSpoutConfiguration\, \PulsarBoltConfiguration\), mappers, and shared client utilities. This change eliminates the ability to consume from or produce to Apache Pulsar topics within Apache Storm topologies, effectively removing the Storm adapter from the product.
pulsar-storm · high confidence
Removal of SimpleResourceAllocationPolicies class
The SimpleResourceAllocationPolicies class, which previously handled namespace isolation policy checks for broker assignment and failover decisions, has been removed from the broker. This eliminates the legacy synchronous ZooKeeper-based policy evaluation logic from the load balancing implementation.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/loadbalance/impl · high confidence
Removal of legacy HTTP lookup REST endpoints and supporting classes
The HTTP-based topic lookup service has been removed from the broker. Specifically, the \DestinationLookup\ REST resource (exposed at \/v2/destination/\) and its base class \LookupResource\ have been deleted, along with the \LookupResult\, \NamespaceData\, and \RedirectData\ data models that supported this HTTP interface. Users relying on these HTTP endpoints for topic location will no longer be able to perform lookups through this mechanism.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/lookup · high confidence
Removal of legacy Pulsar starter and metadata setup classes
The \PulsarBrokerStarter\, \PulsarClusterMetadataSetup\, \PulsarStandaloneStarter\, and \LoadReport\ classes have been removed from the broker module. This eliminates the legacy entry points for starting the broker and standalone mode, as well as the command-line tool for initial cluster metadata setup, and removes the broker load reporting data structure used for load balancing decisions.
pulsar-broker/src/main/java/com/yahoo/pulsar · high confidence
Removal of legacy REST admin resource classes
The legacy admin REST resource classes (AdminResource, BrokerStats, Brokers, Clusters, Namespaces, PersistentTopics, Properties) have been removed from the broker. This eliminates the corresponding REST API endpoints for managing clusters, properties, namespaces, and persistent topics, as well as broker statistics and load-reporting endpoints.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/admin · high confidence
Removal of legacy ZooKeeper utility classes
The \pulsar-zookeeper-utils\ module has removed a suite of legacy classes, including \BookieInfo\, \Deserializers\, \GlobalZooKeeperCache\, \LocalBookkeeperEnsemble\, \LocalZooKeeperCache\, \LocalZooKeeperConnectionService\, \ZkBookieRackAffinityMapping\, \ZkIsolatedBookieEnsemblePlacementPolicy\, \ZooKeeperCache\, \ZooKeeperCacheListener\, \ZooKeeperCacheWatcher\, \ZooKeeperChildrenCache\, \ZooKeeperClientFactory\, and \ZooKeeperDataCache\. This change eliminates the old synchronous and local ZooKeeper caching implementations, rack affinity mappings, and bookie ensemble placement policies that were previously used for managing BookKeeper interactions and metadata.
pulsar-zookeeper-utils · high confidence
Removal of legacy broker infrastructure and utility classes
The broker module has removed several core components and utilities, including the \PulsarService\ main entry point, the \MessagingServiceShutdownHook\ for graceful shutdown, and the \BookKeeperClientFactory\ along with its implementation. Additionally, the \ManagedLedgerClientFactory\, \NamespaceBundleFactory\ for namespace partitioning, and various utility classes such as \CmdUtility\, \CopyOnWriteArrayList\, and \StatsOutputStream\ have been deleted from the codebase.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker · high confidence
Removal of legacy broker stats generators
The broker stats module has removed several legacy classes responsible for generating specific metrics: BookieClientStatsGenerator, BrokerOperabilityMetrics, TopicLoadStats, ManagedLedgerCacheMetrics, and ManagedLedgerMetrics. This change eliminates the code that previously collected and exposed broker-level statistics such as topic load times, managed ledger cache performance, and per-namespace ledger aggregation, effectively removing these specific monitoring data points from the broker's output.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/stats · high confidence
Removal of legacy load-balancing core interfaces and tasks
The legacy load-balancing implementation in the broker has been removed. This change deletes the core \LoadManager\ interface and its associated supporting types (\LoadReport\, \ResourceUnit\, \ServiceRequest\, \LoadCalculator\, \LoadRanker\, \PlacementStrategy\), as well as the legacy leader election logic (\LeaderElectionService\, \LeaderBroker\) and the scheduled tasks that drove them (\LoadReportUpdaterTask\, \LoadSheddingTask\, \NamespaceBundleSplitTask\, \LoadResourceQuotaUpdaterTask\). The \BrokerHostUsage\ class, which previously relied on an external Python script, is also removed. These files are no longer part of the broker's load-balancing logic.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/loadbalance · high confidence
Removal of legacy namespace ownership and service unit classes
The \NamespaceEphemeralData\, \NamespaceService\, \OwnedServiceUnit\, \OwnershipCache\, and \ServiceUnitZkUtils\ classes in the \pulsar-broker/namespace\ package have been removed. These classes previously handled broker-side namespace ownership tracking, ZooKeeper-based ephemeral node management, and service unit lookup logic. Their deletion indicates a shift away from this specific implementation of namespace ownership and lookup, likely replaced by a newer mechanism (such as the async ZooKeeper cache or new load manager API mentioned in related changes).
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/namespace · high confidence
Removal of legacy web service infrastructure
The broker's legacy web service implementation has been removed, including the \WebService\ Jetty server setup, \PulsarServerConnector\ for connection throttling, and servlet filters for API versioning (\ApiVersionFilter\) and authentication (\AuthenticationFilter\). This eliminates the deprecated HTTP-based admin and lookup endpoints that relied on these components.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/web · high confidence
Removal of local ZooKeeper cache services
The \LocalZooKeeperCacheService\ and \ResourceQuotaCache\ classes in the broker cache package have been removed. This eliminates the broker's previous mechanism for maintaining local caches of ZooKeeper data, including namespace owner information, local policies, managed ledger lists, and resource quotas, shifting the system away from this specific local caching implementation.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/cache · high confidence
Removal of persistent topic and subscription dispatcher components
The broker's persistent topic handling logic has been removed, specifically deleting the core dispatcher implementations (\PersistentDispatcherMultipleConsumers\, \PersistentDispatcherSingleActiveConsumer\), the replication engine (\PersistentReplicator\), and the subscription/topic management classes (\PersistentSubscription\, \PersistentTopic\). This also removes the message expiry monitoring (\PersistentMessageExpiryMonitor\) and message position finding (\PersistentMessageFinder\) utilities. These changes eliminate the broker's ability to manage persistent subscriptions, replicate messages between clusters, and track message positions or expiry for stored topics.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/persistent · high confidence
Removal of the pulsar-checksum module
The pulsar-checksum module, which provided CRC-32C checksum computation using a native SSE 4.2 library, has been removed. This change deletes the module's Java classes (including the checksum utility and JNI bindings), the C++ source code for the native library, and the associated Maven assembly and test files, eliminating the local native dependency for this functionality.
pulsar-checksum, pulsar-discovery-service · high confidence
Security
Validate package name components and prevent path traversal in storage paths
The PackageName class now enforces strict validation on all package name components (tenant, namespace, name, version), rejecting empty strings, relative path references ("." and ".."), and path separators (/ and \\). Additionally, the toRestPath method uses URL path segment escaping to ensure that package names are safely encoded when converted to storage paths, preventing path traversal vulnerabilities.
pulsar-package-management/core · high confidence
Architecture
Extract config validation into a standalone module
The config validation logic has been extracted from the main codebase into a new, separate \pulsar-config-validation\ module. This change introduces a dedicated package (\org.apache.pulsar.config.validation\) containing the core \ConfigValidation\ engine, a set of annotation-based validation rules (such as \@NotNull\, \@PositiveNumber\, \@List\, and \@Map\), and a collection of built-in validator implementations. By decoupling this functionality, the module provides a reusable, self-contained library for validating configuration objects across the platform.
pulsar-config-validation · high confidence
Refactored Function Worker core into new service classes
The Function Worker core logic has been refactored into a new set of classes including ClusterServiceCoordinator, ErrorNotifier, FunctionActioner, FunctionAssignmentTailer, FunctionMetaDataManager, and FunctionRuntimeManager. This change reorganizes how function metadata is tracked, assignments are processed, and runtime instances are managed, providing a cleaner separation of concerns within the worker service.
pulsar-functions/worker · high confidence
Refactored system topic client architecture with new factory and base classes
The \pulsar-broker/systopic\ package has been restructured to introduce a cleaner, reusable architecture for system topic interactions. A new \NamespaceEventsSystemTopicFactory\ now centralizes the creation of system topic clients, including specific implementations for topic policies (\TopicPoliciesSystemTopicClient\) and transaction buffer snapshots (\TransactionBufferSnapshotBaseSystemTopicClient\). These implementations extend a new abstract \SystemTopicClientBase\ which manages the lifecycle of writers and readers, while a new \SystemTopicClient\ interface defines the standard contract for reading, writing, and closing system topic resources. This change improves code modularity and reduces duplication in how the broker handles internal system topic events.
pulsar-broker/src/main/java/org/apache/pulsar/broker/systopic · high confidence
Transaction buffer and exception interfaces moved to the broker module
The core transaction buffer interfaces (TransactionBuffer, AbortedTxnProcessor, TransactionCursor, TransactionMeta, TransactionEntry, TransactionBufferReader, TransactionBufferProvider, and TransactionBufferClientStats) and the associated exception hierarchy (TransactionException, TransactionBufferException, TransactionCoordinatorException, TransactionPendingAckException) have been relocated to pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer. This change consolidates the transaction buffer contract and error types within the broker module, establishing the API surface that broker implementations will use to manage transactional message buffering, commit/abort semantics, and snapshot recovery.
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer · high confidence
Behavioural changes
1 commit (0 fixes) modifying gradle
A change to existing behaviour in gradle — 1 commit, 1 file: “\[improve\]\[build\] Add Maven publishing conventions for ASF release (\#25457)” (refs \#25457).
gradle · medium confidence · unverified
1 commit (0 fixes) modifying microbench, pulsar-common
A change to existing behaviour in microbench, pulsar-common — 1 commit, 6 files: “\[improve\]\[client\] Reduce ZLIB input copying for direct and composite buffers (\#27420)” (refs \#27420).
microbench/src/main/java/org/apache/pulsar/common/compression · medium confidence · unverified
1 commit (0 fixes) modifying microbench/src/main/java/org/apache/pulsar/broker/stats
A change to existing behaviour in microbench/src/main/java/org/apache/pulsar/broker/stats — 1 commit, 2 files: “\[improve\]\[broker\] Reduce memory allocations when exporting managed ledger and cursor metrics to Prometheus (\#27417)” (refs \#27417).
microbench/src/main/java/org/apache/pulsar/broker/stats · medium confidence · unverified
1 commit (0 fixes) modifying microbench/src/main/java/org/apache/pulsar/common/topics
A change to existing behaviour in microbench/src/main/java/org/apache/pulsar/common/topics — 1 commit, 2 files: “\[improve\]\[broker\] Avoid regex splitting in topic-list filtering (\#27143)” (refs \#27143).
microbench/src/main/java/org/apache/pulsar/common/topics · medium confidence · unverified
1433 commits (528 fixes) modifying pulsar-broker/src/main/java/org/apache/pulsar/broker/service
A change to existing behaviour in pulsar-broker/src/main/java/org/apache/pulsar/broker/service — 1433 commits (528 fixs), 78 files: “\[fix\]\[broker\] Fix Shared and Key\_Shared consumer stalls during FLOW and unblock races (\#26797)”; “Optimize topic policy with HierarchyTopicPolicies about dispatchRate (\#14038)”; “\[improve\]\[broker\] Change partition metadata not-found log to warn (\#26381)”; and 1430 more (refs \#26797, \#14038, \#26381, \#27314, \#3553, …).
pulsar-broker/src/main/java/org/apache/pulsar/broker/service · medium confidence · unverified
1685 commits (681 fixes) modifying pulsar-client
A change to existing behaviour in pulsar-client — 1685 commits (681 fixs), 562 files: “\[ Java Client \] fix parseProtobufSchema method will be called two times (\#13163)”; “\[fix\]\[client\] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block (\#18633)”; “Support Topic lookup using Pulsar binary protocol (\#5)”; and 1682 more (refs \#13163, \#18633, \#5, \#18635, \#517, …).
pulsar-client · medium confidence · unverified
2 commits (0 fixes) modifying microbench/src/main/java/org/apache/pulsar/common/util
A change to existing behaviour in microbench/src/main/java/org/apache/pulsar/common/util — 2 commits, 5 files: “\[improve\]\[misc\] Trim executor queues using a shared capacity budget (\#26589)”; “\[improve\]\[broker\] Hash Key\_Shared sticky keys without allocation (\#26968)” (refs \#26589, \#26968).
microbench/src/main/java/org/apache/pulsar/common/util · medium confidence · unverified
2 commits (1 fix) modifying microbench/src/main/java/org/apache/pulsar/broker/service/persistent
A change to existing behaviour in microbench/src/main/java/org/apache/pulsar/broker/service/persistent — 2 commits (1 fix), 3 files: “\[improve\]\[broker\] Skip unused replicated subscription timestamp updates (\#26655)”; “\[fix\]\[broker\] Reduce Prometheus scrape latency after topic reload by avoiding unnecessary BookKeeper reads (\#27419)” (refs \#26655, \#27419).
microbench/src/main/java/org/apache/pulsar/broker/service/persistent · medium confidence · unverified
23 commits (5 fixes) modifying pulsar-build
A change to existing behaviour in pulsar-build — 23 commits (5 fixs), 7 files: “\[feat\]\[client\] Add combined unshaded and shaded v4/v5 client artifacts (\#26545)”; “\[improve\]\[build\] Support building and testing with Java 25 (\#25453)”; “\[improve\]\[ci\] Report the tests that pass only on their retry in every test job (\#27346)”; and 20 more (refs \#26545, \#25453, \#27346, \#27305, \#25408, …).
pulsar-build · medium confidence · unverified
3 commits (0 fixes) modifying conventions
A change to existing behaviour in conventions — 3 commits, 3 files: “\[improve\]\[build\] Add Dependabot configuration and an updateBinaryLicense task (\#27485)”; “\[improve\]\[build\] Add validated API/SPI publication subset (\#26579)”; “\[improve\]\[build\] Upgrade Gradle to 9.9.0 and its plugins, and follow Gradle's best practices (\#27493)” (refs \#27485, \#26579, \#27493).
conventions · medium confidence · unverified
6 commits (3 fixes) modifying microbench/src/main/java/org/apache/pulsar/common/util/collections
A change to existing behaviour in microbench/src/main/java/org/apache/pulsar/common/util/collections — 6 commits (3 fixs), 4 files: “\[improve\]\[ml\] Back LongBitmap with the on-heap RoaringBitmap to fix ack allocation hotspot (\#26500)”; “\[fix\]\[broker\] ConcurrentLongHashMap throw ArrayIndexOutOfBoundsException (\#25644)”; “\[fix\]\[sec\] Upgrade Checkstyle to 14.3.0 and use Guava 33.7.2-jre on its classpath (\#27178)”; and 3 more (refs \#26500, \#25644, \#27178, \#26010, \#27113, …).
microbench/src/main/java/org/apache/pulsar/common/util/collections · medium confidence · unverified
7 commits (3 fixes) modifying pulsar-docs-tools
A change to existing behaviour in pulsar-docs-tools — 7 commits (3 fixs), 4 files: “\[improve\] PIP-467: Convert remaining modules logging from SLF4J to slog (\#25511)”; “\[fix\]\[test\] Fixed Nondeterministic Ordering in Generated Docs Topics (\#24973)”; “\[improve\]\[cli\] PIP-343: Use picocli instead of jcommander in bin/pulsar (\#22288)”; and 4 more (refs \#25511, \#24973, \#22288, \#25937, \#25417, …).
pulsar-docs-tools · medium confidence · unverified
Admin client authentication offloaded to prevent caller-thread blocking
The PulsarAdmin client now offloads synchronous v4 authentication credential composition to a dedicated bounded executor pool, preventing stalled identity providers from blocking the caller's thread. This change, part of PIP-478, ensures that authentication operations do not deadlock or throttle other clients in the JVM by isolating blocking I/O to a per-admin pool rather than the process-wide shared pool.
pulsar-client-admin · high confidence
Admin server implementation upgraded to Jetty 12
The embedded ZooKeeper admin server implementation in this module has been updated to use Jetty 12 (specifically Jetty EE8 classes). This change introduces a new set of core classes for the admin server, including the \JettyAdminServer\ implementation, command interfaces (\Command\, \CommandBase\), and response handling (\CommandResponse\, \JsonOutputter\). Users relying on the admin server will now benefit from the underlying Jetty 12 runtime, which may affect HTTP handling and SSL/TLS configuration details, while the public API for running admin commands remains consistent.
jetty-upgrade/zookeeper-with-patched-admin · high confidence
Broker-side schema data validation for Avro, JSON, Protobuf, and primitives
The broker now validates schema definitions at registration and compatibility-check time to ensure they are well-formed before storage. A new \SchemaDataValidator\ framework enforces type-specific rules: primitive schemas (STRING, INT, etc.) must have empty data, STRING schemas accept only the Python 'null' sentinel, Protobuf Native schemas must deserialize to a valid descriptor, and Avro/JSON schemas are parsed with a compatible name validator that allows '$' in record names. For JSON schemas, strict Avro-only validation is the default (PIP-464), with an optional legacy Jackson JsonSchema fallback for backward compatibility. This validation is applied via a \SchemaRegistryService\ wrapper that intercepts \putSchemaIfAbsent\, \isCompatible\, and \checkCompatible\ calls.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/validator · high confidence
CLI commands migrated to picocli
The broker CLI commands (broker, standalone, initialize-cluster-metadata, delete-cluster-metadata, initialize-namespace, initialize-transaction-coordinator-metadata, and version) now use picocli instead of the previous argument parsing library. This change standardizes command-line argument handling, help output, and documentation generation across all broker subcommands.
pulsar-broker/src/main/java/org/apache/pulsar · high confidence
Introduce isolated Java function instance runtime with strict dependency boundaries
The \pulsar-functions/runtime-all\ module now provides the entry point for executing Java functions via \JavaInstanceMain\, which establishes a three-tier classloader hierarchy (root, function instance, and user code) to strictly isolate user function dependencies from the Pulsar framework. This change ensures that the resulting \java-instance.jar\ contains only the necessary framework interfaces and shared libraries (such as Pulsar IO, SLF4J, Log4j, and Avro), preventing dependency conflicts. Additionally, dedicated Log4j2 configuration files are introduced for standard and Kubernetes environments to standardize logging behavior, and a new test suite verifies that no extraneous classes leak into the instance JAR.
pulsar-functions/runtime-all · high confidence
Introduce reusable data memory mode for OpenTelemetry metrics
The OpenTelemetry service in Pulsar now enables the OpenTelemetry SDK's reusable data memory mode (MemoryMode.REUSABLE\_DATA) to reduce memory allocations during metric export. This change is part of the PIP-264 initiative to improve observability performance and resource efficiency.
pulsar-opentelemetry · high confidence
Introduces streaming offload support and bounded offsets cache in JCloud tiered storage
The JCloud tiered storage implementation now supports streaming offload via a new \StreamingDataBlockHeaderImpl\ that includes a ledger ID in its header, alongside a standard \DataBlockHeaderImpl\. To prevent OutOfMemory errors during read operations, the \OffsetsCache\ has been implemented with a configurable maximum size (defaulting to 1 million entries) and time-to-live expiration, replacing the previous unbounded caching behavior.
jcloud · high confidence
Introduction of AdminResource base class and Jakarta EE migration
The admin API layer now introduces a new \AdminResource\ base class in \pulsar-broker/src/main/java/org/apache/pulsar/broker/admin\ that extends \PulsarWebResource\. This change centralizes core administrative logic, including topic partition creation, policy validation, and domain detection, while migrating the package to use \jakarta.\\ APIs (e.g., \jakarta.servlet\, \jakarta.ws.rs\) instead of the legacy \javax.\\ namespace. The new base class also integrates with the \TopicEventsListener\ to notify on topic creation events and utilizes the \TopicPoliciesService\ for policy management.
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin · high confidence
Managed ledger protocol expanded with offload metadata, compression, and cursor properties
The MLDataFormats.proto schema has been significantly updated to support advanced managed ledger features. It now includes OffloadContext and OffloadDriverMetadata messages to track tiered storage offloading details within ledger info. Compression support is introduced via new ManagedLedgerInfoMetadata and ManagedCursorInfoMetadata messages using a CompressionType enum. Cursor and position tracking are enhanced with individualDeletedMessages, batched entry deletion indices, and custom properties (LongProperty and StringProperty). Additionally, the schema adds support for topic termination via a terminatedPosition field and tracks cursor idle time.
managed-ledger/src/main/proto · high confidence
Migrate build system from Maven to Gradle
The project's build system has been migrated from Maven to Gradle. This change introduces the Gradle wrapper (version 9.8.0), adds conventions for Maven publishing to support ASF releases, and includes new build scripts such as a GPG key setup utility for testing artifact signing and a verification task to ensure all test classes are properly assigned to TestNG groups for CI execution. Also in this area: 1 commit (0 fixes) modifying build-logic/conventions, conventions, gradle.
gradle/wrapper · low confidence · unverified
Migrate logging to Log4j2 and introduce OpenTelemetry JSON layout
The logging infrastructure has been upgraded from Log4j 1.x to Log4j2, replacing the legacy log4j.properties file with a new log4j2.yaml configuration. This change introduces support for structured JSON logging via a new OtelLogLayout.json template, allowing users to emit flat OpenTelemetry-compatible logs. Additionally, the functions\_worker.yml configuration has been updated to reflect the new logging and worker settings, and the bkenv.sh script has been modernized to support Java 21+ ZGC and adaptive Netty allocators.
conf · high confidence
Migrate shaded JAX-RS services to Jakarta namespace
The shaded client modules (pulsar-client-admin-shaded and pulsar-client-all) now register their JAX-RS service providers under the jakarta.\* namespace. Specifically, new META-INF/services files point the shaded ClientBuilder and RuntimeDelegate to the corresponding org.glassfish.jersey implementations, ensuring compatibility with the Jakarta EE migration.
pulsar-client-admin-shaded, pulsar-client-all · high confidence
New Docker build and publish scripts with non-root security defaults
The docker directory now includes a build script (build.sh) that invokes the Gradle task to create the image, a publish script (publish.sh) that tags and pushes images to Docker Hub based on version and git commit, and a README documenting the new security posture. Starting with version 2.10.0, the official Apache Pulsar Docker images run as a non-root user (UID 10000) by default to enhance container security and ensure OpenShift compliance; users requiring root access for debugging or custom tool installation must explicitly override the user context in their deployment configuration.
docker · high confidence
New load-balancing policy helpers for anti-affinity and isolation
The broker now includes dedicated helper classes for evaluating anti-affinity group and namespace isolation policies during load balancing. AntiAffinityGroupPolicyHelper filters brokers based on anti-affinity constraints and tracks failure domain changes, while IsolationPoliciesHelper applies namespace-level isolation rules to restrict which brokers can host specific service units. These helpers enable the extensible load manager to respect these policies when selecting brokers for bundle assignment and unloading.
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/policies · high confidence
Pulsar Docker images now use Alpine 3.24 and support Wolfi base images
The Pulsar Docker images have been updated to use Alpine 3.24 as the default base OS, replacing previous versions like 3.20 or 3.22. A new Wolfi-based Dockerfile (Dockerfile.wolfi) is introduced, allowing users to build images using the glibc-based Chainguard Wolfi base image, which is useful for environments requiring glibc compatibility. The Alpine-based images now use Amazon Corretto JDK 25 (configurable via IMAGE\_JDK\_MAJOR\_VERSION) and employ jlink to create a slimmed-down JVM runtime, significantly reducing image size. Additionally, the build process now excludes the filesystem offloader NAR by default to reduce bloat, and includes a script to remove unnecessary native binaries for Netty and RocksDB, optimizing the final image footprint.
docker/pulsar · high confidence
Refactor CLI tools to use picocli and new base classes
The Pulsar CLI tools have been refactored to use the picocli library instead of JCommander for command-line parsing. This change introduces new base classes, \CliCommand\ and \CmdBase\, which provide a unified structure for all CLI commands, including validation helpers, JSON pretty-printing, and admin client management. The refactoring affects core command modules such as \CmdBookies\, \CmdBrokerStats\, \CmdBrokers\, \CmdClusters\, \CmdFunctionWorker\, and \CmdFunctions\, standardizing how they define subcommands, handle options, and interact with the Pulsar Admin API.
pulsar-client-tools · high confidence
Refactor Java function instance to use pull model and support V5 client
The Java function instance runtime has been refactored to use a pull model for message ingestion, replacing the previous push-based approach. This change introduces support for the V5 Pulsar client API, allowing functions to publish messages using the new client when configured. The refactoring also includes a new \FunctionResultRouter\ that ensures deterministic partition routing for effectively-once processing by using sequence IDs when message keys are absent, and updates the \ContextImpl\ to better manage producer caching and schema handling for sinks.
pulsar-functions/instance · high confidence
Refactor Prometheus metrics generation with new aggregation classes and streaming
The Prometheus metrics generation logic in the broker has been refactored to improve performance and correctness. New aggregation classes (AggregatedBrokerStats, AggregatedNamespaceStats, AggregatedSubscriptionStats, etc.) are introduced to efficiently accumulate statistics at the broker, namespace, and subscription levels. The metrics generation now uses PrometheusMetricStreams to ensure metrics of the same name are grouped together under the correct TYPE header, complying with the Prometheus exposition format. The PulsarPrometheusMetricsServlet has been updated to use this new generator, supporting async processing and optional gzip compression for the /metrics endpoint.
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus · high confidence
Refactor broker web service with new servlet filters and async response handling
The broker's HTTP admin API now uses a new set of servlet filters (PreInterceptFilter, ProcessHandlerFilter, ResponseHandlerFilter) and an ExceptionHandler to manage request interception, response writing, and error handling. This change introduces a RequestWrapper to allow the request body to be read multiple times by interceptors without consuming the stream prematurely. Additionally, a WebExecutorAsyncResponse wrapper ensures that asynchronous REST responses are written on the web service executor thread pool, preventing blocking of metadata store callback threads during slow client reads. A MaxRequestSizeFilter is also added to enforce request size limits.
pulsar-broker/src/main/java/org/apache/pulsar/broker/web · high confidence
Refactor dispatcher read logic and introduce pluggable dispatch rate limiters
The broker's message dispatching behavior has been refactored to improve reliability and configurability. The \readMoreEntries\ logic in \AbstractPersistentDispatcherMultipleConsumers\ has been replaced with a state-machine approach that conflates concurrent and reentrant read requests into sequential passes, preventing dispatcher stalls and race conditions during unblock events. Additionally, the dispatch rate limiting mechanism has been decoupled into a pluggable factory pattern (\DispatchRateLimiterFactory\), introducing an \AsyncTokenBucket\ implementation alongside the existing classic limiter to allow for more flexible and potentially higher-performance rate limiting at the broker, topic, subscription, and replicator levels.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent · high confidence
Refactor startup scripts to Gradle classpath and modernize JVM options
The bin/pulsar, bin/bookkeeper, and CLI scripts (pulsar-admin, pulsar-client, pulsar-perf, pulsar-shell) have been refactored to replace Maven-based classpath generation with Gradle (exporting classpath.txt via Gradle tasks). This change also updates the default logging configuration from log4j.properties to log4j2.yaml, bridges java.util.logging to Log4j2, and adds JVM flags required for Java 17+ compatibility (e.g., --add-opens, --enable-native-access) and performance tuning (e.g., ZGC for Java 21+, compact object headers for Java 25/26).
bin · high confidence
Refactored JCloud tiered storage offloader implementation
The JCloud-based tiered storage offloader in the \tiered-storage/jcloud\ module has been refactored to use a new internal package structure (\org.apache.bookkeeper.mledger.offload.jcloud\). This change introduces new core abstractions for managing offloaded data, including \BackedInputStream\ for seekable blob reads, \DataBlockHeader\ for block metadata, and \OffloadIndexBlock\/\OffloadIndexBlockV2\ for index management. The implementation now utilizes \BlobStoreBackedReadHandleImpl\ and \BlobStoreBackedReadHandleImplV2\ to handle ledger reads from cold storage, and \JCloudLedgerOffloaderFactory\ to instantiate the offloader. This restructuring supports the underlying streaming offload capabilities and improves the modularity of the JCloud provider integration.
tiered-storage/jcloud · high confidence
Refactored delayed delivery tracker architecture with strict delivery mode support
The delayed delivery subsystem has been restructured to decouple tracker implementations from the dispatcher via new interfaces (DelayedDeliveryTracker, DelayedDeliveryTrackerFactory) and a loader mechanism. This introduces an abstract base class (AbstractDelayedDeliveryTracker) that centralizes timer management and adds support for strict delivery timing via the isDelayedDeliveryDeliverAtTimeStrict configuration; when enabled, messages are delivered exactly at or after their scheduled time, preventing the early delivery that was previously possible in non-strict mode. The InMemoryDelayedDeliveryTracker implementation now uses timestamp bucketing to reduce memory usage, and the new BucketDelayedDeliveryTrackerFactory provides a persistent, bucket-based alternative with automatic fallback to the in-memory tracker if recovery fails.
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed · high confidence
Refactored lookup result handling and migrated to Jakarta APIs
The broker's lookup logic in the \org.apache.pulsar.broker.lookup\ package has been refactored to support multi-listener configurations and modern Java EE standards. New classes \LookupResult\, \NamespaceData\, and \RedirectData\ replace previous ad-hoc return types, enabling the broker to correctly resolve and return specific broker and web service URLs based on advertised listener names (e.g., \advertisedListenerName\, \webServiceAdvertisedListenerName\) rather than defaulting to a single address. This allows clients to connect via the correct listener when multiple listeners are configured. Additionally, the codebase has migrated from \javax.ws.rs\ to \jakarta.ws.rs\ APIs, and \TopicLookupBase\ now uses the \slog\ logging framework. These changes improve lookup accuracy in multi-listener setups and align the broker with the Jakarta EE namespace.
pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup · high confidence
Refactored namespace bundle ownership and lookup mechanics
The broker's namespace service now uses a new \LookupOptions\ object to carry lookup context (such as authoritative flags and advertised listener names) and introduces \NamespaceBundleOwnershipListener\ and \NamespaceBundleSplitListener\ interfaces to decouple ownership and split event handling. The \OwnershipCache\ has been rewritten to use a Caffeine async loading cache backed by a coordination \ResourceLock\, ensuring that ownership generations are strictly tracked and that lock expiry triggers unloads for the correct generation. Additionally, \OwnedBundle\ now ties its lifecycle to a specific \ResourceLock\ to prevent race conditions during unloads, and \NamespaceEphemeralData\ excludes the \brokerId\ from equality checks to handle rolling upgrades gracefully.
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace · high confidence
Refactored schema compatibility checking and storage infrastructure
The schema compatibility checking logic has been refactored to use a new, cleaner architecture. A new \AvroSchemaBasedCompatibilityCheck\ abstract class centralizes Avro schema validation, introducing dedicated \AlwaysSchemaValidator\ and \NeverSchemaValidator\ classes to handle \ALWAYS\_COMPATIBLE\ and \ALWAYS\_INCOMPATIBLE\ strategies more explicitly. \JsonSchemaCompatibilityCheck\ now enforces strict Avro schema format for \SchemaType.JSON\ by default (per PIP-464), while retaining backward compatibility for legacy Jackson formats if explicitly enabled. \ProtobufNativeSchemaAdvancedCompatibilityCheck\ provides opt-in, field-aware compatibility checking for Protobuf native schemas, supporting specific editions and rejecting unsupported features. \KeyValueSchemaCompatibilityCheck\ now delegates to underlying key and value schema checkers. Additionally, \BookkeeperSchemaStorage\ has been updated to use \MetadataStore\ for schema locators and includes a Caffeine-based cache for schema entry reads to improve performance.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema · high confidence
Refactored topic compaction with pluggable service and improved read consistency
The compaction logic in the broker has been restructured to support a pluggable \TopicCompactionService\ and a new two-phase compactor (\AbstractTwoPhaseCompactor\) that separates key selection from ledger writing to reduce memory usage. A new \CompactedTopic\ interface and \CompactedTopicUtils\ now handle reading from compacted ledgers, ensuring that readers correctly transition from the compacted snapshot to the original ledger and properly handle tombstones and individually acknowledged messages. The \CompactorTool\ CLI has been updated to support different compactor types (Publishing and Event Time) and uses dedicated TLS configuration for the compactor client. Additionally, \CompactionRecord\ and \CompactorMXBean\ provide detailed metrics for compaction operations.
pulsar-broker/src/main/java/org/apache/pulsar/compaction · high confidence
Removal of custom protobuf code generation patch and documentation
The custom patch used to modify the Google Protocol Buffer generator for zero-allocation serialization and Netty ByteBuf integration has been removed, along with the associated README documentation. This indicates that the project is no longer maintaining or using this specific custom code generation approach for its wire protocol.
protobuf · high confidence
Removal of legacy broker service components
The broker service implementation in \pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service\ has been significantly refactored by removing several core classes, including \BacklogQuotaManager\, \BrokerService\, \BrokerServiceException\, \Consumer\, \Dispatcher\, \DistributedIdGenerator\, \Producer\, \PulsarChannelInitializer\, \PulsarStats\, and \ServerCnx\. This change removes the previous internal mechanisms for managing topic backlogs, handling client connections, and tracking message statistics, indicating a major architectural shift in how the broker handles these responsibilities.
pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service · high confidence
Replace shell-based license check with Gradle task and enhance build conventions
The build system replaces the legacy \check-binary-license.sh\ script with a new \CheckBinaryLicenseTask\ Gradle task that validates LICENSE/NOTICE coverage in binary distribution tarballs. This change is part of a broader update to the \build-logic/conventions\ module that introduces several new Gradle convention plugins and tasks: \JfrFlamegraphsTask\ for converting JFR recordings to flame graphs, \DockerImageBuild\ helpers for incremental Docker builds, \ValidateApiSpiPublication\ for verifying API/SPI publication integrity, \VerifyJavaCompatibility\ for checking bytecode compatibility, and \MinimizedDependenciesExtension\ for managing minimized shaded jars. The \pulsar.client-shade-conventions\ plugin is updated to support a new \shadowApi\ configuration that preserves compile-scope dependencies in shaded artifacts, and \pulsar.code-quality-conventions\ is modified to run checkstyle and spotless without compiling source code by excluding generated sources and clearing the classpath.
build-logic/conventions · high confidence
Restore BookKeeper Prometheus stats provider using Prometheus Java client 1.x
The BookKeeper Prometheus metrics provider has been re-implemented to use the Prometheus Java client 1.x library. This change introduces a new HTTP endpoint (defaulting to port 8000) served by Jetty 12, which exports BookKeeper metrics in Prometheus text format. The implementation replaces previous counting mechanisms with high-performance \LongAdder\-based counters and uses the DataSketches library to calculate approximate latency quantiles for operation statistics. It also includes support for thread-scoped metrics and exposes standard JVM metrics via the new client library.
bookkeeper-prometheus-metrics-provider · high confidence
Restore Pulsar 4.x compatible serialization for PackageMetadata
The \PackageMetadata\ class in the packages management API now explicitly defines a \serialVersionUID\ matching the default value computed in Pulsar 4.x. This change ensures that package metadata objects serialized by older Pulsar 4.x instances can still be correctly deserialized by the current version, preventing compatibility errors when reading legacy package data.
pulsar-client-admin-api/src/main/java/org/apache/pulsar/packages/management · high confidence
SASL authentication plugin refactored to v5-native implementation
The SASL authentication plugin has been rewritten to use the v5-native authentication framework (PIP-478), introducing a new \SaslAuthenticationV5\ class that handles both the binary protocol and SASL-over-HTTP flows. This change ports the existing SASL logic to the modern v5 SPI, ensuring that blocking GSSAPI/Kerberos operations are offloaded to a dedicated executor to prevent event-loop contention, and implements a robust role-token caching mechanism for HTTP requests to avoid unnecessary renegotiations. The legacy synchronous \AuthenticationSasl\ class now acts as a shim that routes calls to the new v5 implementation, maintaining backward compatibility for third-party integrations while improving performance and reliability for both client and admin HTTP connections.
pulsar-client-auth-sasl · high confidence
WebSocket service rewritten for Jetty 12 and Jakarta EE 10
The pulsar-websocket module has been completely rewritten to upgrade the underlying WebSocket server from Jetty 9 to Jetty 12 and migrate from Java EE (\javax.servlet\) to Jakarta EE 10 (\jakarta.servlet\). This change introduces new servlet entry points (\WebSocketConsumerServlet\, \WebSocketProducerServlet\, \WebSocketReaderServlet\, and \WebSocketMultiTopicConsumerServlet\) that handle the HTTP-to-WebSocket upgrade process using the modern Jetty factory API. The internal handler classes (\AbstractWebSocketHandler\, \ConsumerHandler\, \ProducerHandler\, \ReaderHandler\, and \MultiTopicConsumerHandler\) have been refactored to support these new servlets, including updated authentication and authorization logic that leverages the new Jakarta request wrappers and async authorization checks. The multi-topic consumer endpoint is now exposed at \/ws/v3/consumer\ to support pattern-based subscriptions, while single-topic endpoints remain at \/ws/v2/\. This rewrite ensures compatibility with the latest Jetty and Jakarta standards while maintaining the existing WebSocket proxy functionality for producers, consumers, and readers.
pulsar-websocket · high confidence
ZooKeeper Prometheus metrics provider now uses Jetty 12
The Prometheus metrics HTTP endpoint in the ZooKeeper metrics provider has been updated to run on Jetty 12 (specifically the EE8 servlet API). This change replaces the previous Jetty implementation, meaning the internal HTTP server handling the /metrics endpoint now relies on Jetty 12.1.x classes and APIs. Users relying on the metrics endpoint will see this underlying server technology change, which may affect compatibility with custom servlet filters or security constraints if they were tightly coupled to the previous Jetty version.
zookeeper-prometheus-metrics · high confidence
Fixes
1 commit (1 fix) fixing microbench/src/main/java/org/apache/pulsar/broker/topiclistlimit
A fix in microbench/src/main/java/org/apache/pulsar/broker/topiclistlimit — 1 commit (1 fix), 2 files: “\[fix\]\[broker\] PIP-442: Fix Netty IO threads blocking on the topic list memory limiter's wait time metrics (\#27359)” (refs \#27359).
microbench/src/main/java/org/apache/pulsar/broker/topiclistlimit · medium confidence · unverified
Introduce high-performance asynchronous token bucket rate limiter
The broker now includes a new, highly concurrent asynchronous token bucket implementation in the \org.apache.pulsar.broker.qos\ package to power rate limiting. This new \AsyncTokenBucket\ uses \LongAdder\ and CAS operations to minimize contention, supporting both fixed-rate and dynamic-rate configurations via dedicated builders. This change replaces the previous rate limiter logic to address issues where rates could initially exceed limits, causing excessive throttling and connection timeouts.
pulsar-broker/src/main/java/org/apache/pulsar/broker/qos · high confidence
Test coverage
10 commits adding/updating tests in pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/reporter; 10 commits adding/updating tests in pulsar-broker/src/test/resources; 11 commits adding/updating tests in buildtools/src/test; 1265 commits adding/updating tests in pulsar-broker/src/test/java/org/apache/pulsar/broker/service; 2 commits adding/updating tests in tests/pulsar-client-test-bcfips; 57 commits adding/updating tests in buildtools/src/main/java; Add test certificate authority assets and utilities; Added ClientChannelHelper test utility for broker service tests; Added JMH benchmark for TopicName.get() cold-start performance; Added JMH benchmarks for BucketDelayedDeliveryTracker; Added JMH benchmarks for QoS rate limiting components; Added JMH microbenchmarks for Managed Ledger cache operations; Added JMH microbenchmarks for broker service performance tuning; Added Protobuf schema test fixtures; Added SOCKS5 proxy test infrastructure; Added V5 client API test coverage; Added V5 receive-loop allocation benchmark; Added WebSocket proxy tests for client-side encryption, authentication, and proxy configuration; Added broker authentication and authorization test suite; Added broker-level tests for external schemas and partitioned topic schema handling; Added broker-side integration tests for client API behaviors; Added broker-side transaction tests for authentication, recovery, and snapshot processing; Added client-impl microbenchmarks for performance profiling; Added end-to-end tests for Pulsar Functions, Sources, and Sinks; Added integration tests for shaded Pulsar client and admin APIs; Added integration tests for the shaded Pulsar client; Added microbenchmark for perf-producer send completion overhead; Added microbenchmark for topic resource notification dispatch; Added microbenchmarks for Managed Ledger cursor and read-path performance; Added test certificates for client-side encryption scenarios; Added test configuration files for RocksDB and ZooKeeper SASL; Added test for batch send failure recovery with compression; Added test harness functions and IO connectors for integration testing; Added test infrastructure and coverage for broker startup, standalone mode, and CLI metadata commands; Added test infrastructure for intercepting broker components; Added test plugins for integration testing broker interceptors, entry filters, protocol handlers, and additional servlets; Added test resources for authentication scenarios; Added test utilities for mocking transaction buffer behavior; Added test utility for iterating over cached entries; Added tests for BookKeeper-based package storage and I/O streams; Added tests for Conscrypt SSL engine cleanup on connection reset; Added tests for GraalVM native image compatibility; Added tests for Java 17 client compatibility; Added tests for LoadDataStore functionality; Added tests for PIP-478 broker-client TLS factory selection and provider pin propagation; Added tests for Pulsar client OpenTelemetry metrics; Added tests for PulsarBrokerStatsClient and topic internal stats; Added tests for Resource Group service, configuration, and usage reporting; Added tests for TLS authentication with Elliptic Curve certificates; Added tests for ThreadLocalAccessor memory leak fix; Added tests for TopKBundles load balancing logic; Added tests for Transaction Coordinator client and configuration; Added tests for URL decoding in scalable topic migration; Added tests for V5 client consumer and producer behaviors; Added tests for ZooKeeper metadata stats and test utilities; Added tests for additional servlet plugin loading and class loader isolation; Added tests for backlog estimation and topic listing memory limits; Added tests for broker entry cache strategies and multi-broker caching scenarios; Added tests for broker interceptor lifecycle, class loading, and filtering; Added tests for broker stats generators and metrics; Added tests for broker test utilities and BookKeeper client configuration; Added tests for broker web service configuration and handling; Added tests for broker-tool documentation generation; Added tests for client connection auto-release and TLS authentication; Added tests for cluster metadata setup, teardown, and ZooKeeper session handling; Added tests for delayed message delivery tracking and bucket snapshot storage; Added tests for entry filter plugin implementations and configuration loading; Added tests for load balancing strategy components; Added tests for managed ledger cache components; Added tests for managed ledger cursor recovery, message expiry, and subscription expiration; Added tests for managed-ledger backlog estimation and replay tasks; Added tests for managed-ledger cursor containers, batching, and entry estimation; Added tests for managed-ledger utility classes; Added tests for namespace bundle management and ownership listeners; Added tests for offload driver metadata handling and offloader caching; Added tests for package management core storage and metadata handling; Added tests for persistent topic service reliability and policy inheritance; Added tests for scalable topic auto-scale configuration and policy evaluation; Added tests for scalable topic resources; Added tests for schema compatibility and storage reliability; Added tests for schema compatibility strategies; Added tests for schema data validation and registry service integration; Added tests for system topic services and partitioned system topics; Added tests for the Extensible Load Manager extension; Added tests for the MessagePayloadProcessor API; Added tests for the bucket delayed message delivery tracker; Added tests for the local filesystem package storage backend; Added tests for token CLI utility functions; Added tests for transaction admin APIs and package management service; Added tests for transaction buffer implementation details; Added tests for transaction metadata store and path utilities; Added tests for transaction pending-ack store implementations; Added tests for transaction pending-ack store initialization and in-memory deletion; Added tests for transaction recovery tracker; Added tests for v4/v5 client classpath isolation; Added unit and integration tests for the Protocol Handler system; Added unit tests for AsyncTokenBucket rate limiting logic; Added unit tests for BrokerLoadData and BrokerLookupData; Added unit tests for HTTP v2 topic lookup behavior; Added unit tests for HTTP v2 topic lookup listener name handling; Added unit tests for Prometheus metrics aggregation and generation; Added unit tests for Pulsar Admin CLI and client tools; Added unit tests for TransactionCoordinatorV5; Added unit tests for broker utility classes; Added unit tests for compaction internals and metrics; Added unit tests for extensible load balancer broker filters; Added unit tests for load balancer and leader election components; Added unit tests for load balancer extension managers; Added unit tests for load balancer implementation classes; Added unit tests for load balancing schedulers; Added unit tests for naming and bundle split components; Added unit tests for non-persistent topic dispatchers and subscription behavior; Added unit tests for the ServiceUnitStateChannel load balancing component; Enhanced performance testing framework with infrastructure monitoring and diagnostics; New admin API test coverage for clusters, delayed delivery, dynamic configs, and health checks; New integration tests for admin, token auth, and backward compatibility; New test infrastructure for Pulsar Functions worker scenarios; New test infrastructure for Pulsar broker unit tests; Refactor managed-ledger test infrastructure to use Pulsar test utilities; Removal of MockZooKeeper test utility; Removal of WebSocket proxy test suite; Removal of broker service test suite; Removal of custom TestNG test listener; Removal of legacy Pulsar client integration and unit tests; Removal of legacy Pulsar common and naming unit tests; Removal of legacy admin API test suite and test infrastructure; Removal of legacy authentication and authorization test suite; Removal of legacy client API test suite and mock infrastructure; Removal of legacy load balancer and broker starter unit tests; Removal of legacy namespace and cache unit tests; Removed BookieClientsStatsGeneratorTest; Removed Discovery Service unit tests and test utilities; Removed HTTP v2 lookup test suite; Removed legacy log4j test configuration; Removed test for StatsOutputStream utility; Updated and expanded broker test configuration files.
Dependencies
1470 commits updating dependencies (120 manifests)
A dependency / build maintenance change in (dependencies) — 1470 commits (380 fixs), 120 files: “Bump log4j to 2.15.0 (\#13226)”; “\[improve\]\[misc\] Upgrade Lombok to 1.18.24 (\#18604)”; “Upgrade dependencies for security fixes (\#5232)”; and 1467 more (refs \#13226, \#18604, \#5232, \#20704, \#27485, …).
(dependencies) · high confidence · unverified
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 59 → 52 (-7.1)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.3) — scores are not directly comparable.
Lenses
- Code Health 60 → 58 (-2.1)
- Architecture 73 → 73 (+0.0)
- Maturity 73 → 77 (+3.6)
- Readiness 60 → 57 (-2.7)
- Security 53 → 39 (-14.5)
- Performance 77 → 95 (+17.7)
Resolved (178)
- BrokerService.unloadServiceUnit (cognitive 16) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java)
- Change coupling: ConsistentHashingStickyKeyConsumerSelector.java ↔ HashRangeAutoSplitStickyKeyConsumerSelector.java (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ConsistentHashingStickyKeyConsumerSelector.java)
- Change coupling: MultiTopicsReaderImpl.java ↔ ReaderBuilderImpl.java (pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsReaderImpl.java)
- Change coupling: PersistentSubscription.java ↔ AggregatedNamespaceStats.java (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java)
- CheckBinaryLicenseTask.check (cyclomatic 17) (build-logic/conventions/src/main/kotlin/CheckBinaryLicenseTask.kt)
- Consumer.sendMessages (cyclomatic 16) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java)
- Duplicated block (10 lines × 2) (pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/filter/BrokerVersionFilter.java)
- Duplicated block (10–11 lines × 2) (managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java)
- Duplicated block (10–11 lines × 2) (pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LoadManagerShared.java)
- Duplicated block (10–11 lines × 2) (pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/coordinator/v5/TransactionCoordinatorV5.java)
- Duplicated block (11 lines × 2) (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/ClustersBase.java)
- Duplicated block (11 lines × 2) (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java)
- Duplicated block (11 lines × 2) (pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/PulsarAdminBuilderImpl.java)
- Duplicated block (12 lines × 10) (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java)
- Duplicated block (13–14 lines × 3) (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java)
- Duplicated block (13–15 lines × 3) (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TenantsBase.java)
- Duplicated block (14 lines × 2) (pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java)
- Duplicated block (14 lines × 2) (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/Namespaces.java)
- Duplicated block (14 lines × 2) (pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/ProduceV4.java)
- Duplicated block (14 lines × 3) (pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/FunctionsImpl.java)
- …and 158 more
New (878)
- AbstractBaseDispatcher.applyDispatchRateLimitsToReadLimits (cognitive 20) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java)
- AbstractBaseDispatcher.applyDispatchRateLimitsToReadLimits (cyclomatic 16) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java)
- AbstractBaseDispatcher.initConsumerAvgMessagesPerEntry (cognitive 23) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java)
- Backoff.next (cognitive 17) (pulsar-common/src/main/java/org/apache/pulsar/common/util/Backoff.java)
- Banned license: org.openjdk.jmh:jmh-core
- Banned license: org.openjdk.jmh:jmh-generator-annprocess
- Boundary-crossing change coupling: WebService.java ↔ WebServer.java (pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java)
- BrokerService.deleteTopicInternal (cognitive 21) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java)
- BrokerService.deleteTopicInternal (cyclomatic 17) (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java)
- Change coupling: LocalRunner.java ↔ JavaInstanceStarter.java (pulsar-functions/localrun/src/main/java/org/apache/pulsar/functions/LocalRunner.java)
- ClassTooLong: AbstractBaseDispatcher (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java)
- ClassTooLong: DualMetadataStore (pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/DualMetadataStore.java)
- ClassTooLong: MLPendingAckStore (pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java)
- ClassTooLong: PersistentDispatcherSingleActiveConsumer (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java)
- ClassTooLong: PrometheusMetricsProvider (jetty-upgrade/zookeeper-prometheus-metrics/src/main/java/org/apache/pulsar/metrics/prometheus/zookeeper/PrometheusMetricsProvider.java)
- ClassTooLong: PulsarShell (pulsar-client-tools/src/main/java/org/apache/pulsar/shell/PulsarShell.java)
- ClassTooLong: ReplicatedSubscriptionsController (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/ReplicatedSubscriptionsController.java)
- ClassTooLong: ScalableTopicPolicies (pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/ScalableTopicPolicies.java)
- ClassTooLong: ScalableTopicResources (pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java)
- CmdConsume.run (cognitive 18) (pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/CmdConsume.java)
- …and 858 more
Changes since last survey
- 300 commits — 94 feature/other, 206 fixes
By area
- pulsar-broker/src — 161 commits
- pulsar-client/src — 23 commits
- tests/integration — 21 commits
- managed-ledger/src — 14 commits
- pulsar-metadata/src — 10 commits
- .github/workflows — 8 commits
- gradle/libs.versions.toml — 7 commits
- distribution/server — 6 commits
- pulsar-broker-common/src — 5 commits
- (root) — 4 commits
- build-logic/conventions — 4 commits
- pulsar-common/src — 4 commits
- pulsar-functions/worker — 4 commits
- pulsar-proxy/src — 3 commits
- .github/actions — 2 commits
- conf/broker.conf — 2 commits
- docker/pulsar — 2 commits
- pulsar-client-tools-test/src — 2 commits
- pulsar-client-tools/src — 2 commits
- pulsar-functions/runtime — 2 commits
Notable commits
- fix: [fix][broker] Accept v5 transaction coordinator handshakes only on the elected leader (#27580)
- fix: [fix][broker] Answer lookups waiting for a bundle owner when the table view refills after a session is reestablished (#27317)
- fix: [fix][broker] Avoid waiting for bucket recovery when clearing backlog (#27199)
- fix: [fix][broker] Close and evict topics that failed to unload when the bundle is released (#27291)
- fix: [fix][broker] Close the broker's own admin client when it is created while the broker closes (#27395)
- fix: [fix][broker] Close the producer of a geo-replicator stopped while it starts (#27273)
- fix: [fix][broker] Correct the error message for missing partitions (#27351)
- fix: [fix][broker] Delete the schema and topic policies when another cluster deletes the partitioned topic metadata first during GC (#27588)
- fix: [fix][broker] Don't block the load manager channel's event thread on load data tombstones (#27519)
- fix: [fix][broker] Don't let a segment load report bring a deleted scalable topic back (#27560)
- fix: [fix][broker] Don't let the leader's bundle-data write recreate the data of a bundle it just removed (#27521)
- fix: [fix][broker] Don't lose the topic changes made while a topic list watcher lists its initial topics (#27525)
- fix: [fix][broker] Don't make clients retry a partitioned topic auto-creation that exceeds maxNumPartitionsPerPartitionedTopic (#27320)
- fix: [fix][broker] Don't throw from ServerCnx.toString() when the channel is null (#27440)
- fix: [fix][broker] Expire every subscription directly when the message TTL expires messages up to the current ledger (#27548)
- fix: [fix][broker] Fail a schema-less AUTO_PRODUCE_BYTES send at once instead of retrying it until the send timeout (#27312)
- fix: [fix][broker] Fail topic policy reads after the service closes (#27225)
- fix: [fix][broker] Fix REST produce failing every message on topics with deduplication enabled (#27424)
- fix: [fix][broker] Fix deadlock between ExclusiveWithFencing fencing and a concurrent producer close (#27410)
- fix: [fix][broker] Fix replication stuck on a schema-less message to a remote topic with an enforced schema (#27314)
- …and 280 more
Architecture
- Containers 0 added · 0 removed · contexts 1 added · 1 removed · edges 3 added · 6 removed
Added bounded contexts (1)
- pulsar-package-management/core
Removed bounded contexts (1)
- pulsar-package-management/core
Added dependency edges (3)
- pulsar-package-management/core → pulsar-client-admin-api
- filesystem-storage → pulsar-package-management/core
- pulsar-broker → pulsar-package-management/core
Removed dependency edges (was a dependency, now gone) (6)
- coordinator → jcloud
- filesystem-storage → pulsar-package-management/core
- managed-ledger → jcloud
- pulsar-broker → jcloud
- pulsar-broker → pulsar-package-management/core
- pulsar-package-management/core → pulsar-client-admin-api
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
apache/pulsar was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 13 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit f95caa2c17d6fe7b1424afa8ae8177e33fed8a8d — the exact code this score is about.
- Scored under rubric-2026.11.3 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.63+preprod-4a8cc97a9484.