apache/spark
43.3
Weak · 14 November 2026
1257.1k
lines of production code
Scala
with Python, Java
6
measurements over time
What this system is
Features
Add API v1 status enums for application, stage, and task states
The Spark Web UI and SHS REST API now expose dedicated status enums (ApplicationStatus, StageStatus, TaskStatus) and a TaskSorting enum in the api/v1 package, enabling clients to query and sort task lists by status and runtime. These enums provide case-insensitive string parsing and alternate names (e.g., 'runtime'/'-runtime') to support consistent status reporting and sorting in the UI and REST API.
core/src/main/java/org/apache/spark/status · high confidence
Add Binder integration for PySpark documentation
Added a Dockerfile and postBuild script to the binder directory, enabling users to launch a live Jupyter environment for PySpark documentation directly via Binder. The setup installs PySpark (pinned to the current version for tagged commits, or compatible versions otherwise), along with pandas (pinned below 2.0.0) and plotly, and configures the environment to suppress console progress bars and UI warnings for a cleaner notebook experience.
binder · high confidence
Add ByteArrayMethods and LongArray to common/unsafe
This change introduces two new classes to the \org.apache.spark.unsafe.array\ package. \ByteArrayMethods\ provides optimized static utilities for byte array operations, including \arrayEquals\ (which handles platform-specific unaligned memory access for performance), \contains\, \startsWith\, and \endsWith\. \LongArray\ offers a low-level, bounds-check-free wrapper for long arrays that supports both on-heap and off-heap memory blocks via \MemoryBlock\, allowing direct manipulation of underlying memory for performance-critical internal operations.
common/unsafe/src/main/java/org/apache/spark/unsafe/array · high confidence
Add DummySerializerInstance for shuffle write path
A new DummySerializerInstance class has been added to the serializer package to serve as a no-op serializer for DiskBlockObjectWriter. This allows the shuffle write path to construct the writer without relying on a functional serializer, as the write path primarily uses OutputStream methods. The dummy instance implements necessary methods like flush and close to support compression stream handling, while other serialization methods throw UnsupportedOperationException.
core/src/main/java/org/apache/spark/serializer · high confidence
Add Executor Heap Histogram and Flame Graph support to Web UI
The Executors tab in the Spark Web UI now includes optional views for heap histogram analysis and flame graph visualization of executor thread dumps. When enabled via configuration, users can view a breakdown of object instances and memory usage per class in the Heap Histogram page, and visualize thread call stacks as interactive flame graphs in the Thread Dump page. These features are controlled by the spark.ui.heapHistogram.enabled and spark.ui.threadDump.flamegraphEnabled configuration options, respectively, and are integrated into the existing Executors tab alongside the standard thread dump and summary views.
core/src/main/scala/org/apache/spark/ui/exec · high confidence
Add Ganglia metrics sink connector
The Spark Ganglia connector has been added, introducing a new GangliaSink that allows users to export Spark metrics to a Ganglia monitoring system via UDP. Users can configure the sink by specifying the Ganglia host and port, as well as optional parameters for polling period, time unit, multicast mode, TTL, and dmax.
connector/spark-ganglia-lgpl · high confidence
Add HadoopConfigProvider for YARN network module
The YARN network module now includes a HadoopConfigProvider that bridges Spark's configuration system with Hadoop's Configuration object, allowing the network layer to read settings directly from the Hadoop environment.
common/network-yarn/src/main/java/org/apache/spark/network/yarn/util · high confidence
Add Java UDF interfaces and catalog Identifier to sql/api
The sql/api module now exposes a complete set of Java UDF interfaces (UDF0 through UDF22) in the org.apache.spark.sql.api.java package, enabling Java users to define custom functions with up to 22 arguments. Additionally, the module introduces the catalog Identifier interface and its implementation, providing a standard way to reference catalog objects by namespace and name.
sql/api · high confidence
Add Java examples for MLlib and core Spark algorithms
New Java example programs have been added to the examples directory, providing reference implementations for core Spark operations and machine learning algorithms. The core examples include JavaPageRank, JavaSparkPi, JavaStatusTrackerDemo, JavaTC (transitive closure), and JavaWordCount. Additionally, a suite of ML examples has been introduced in the ml sub-package, covering algorithms such as AFT Survival Regression, ALS, Bisecting K-Means, Bucketed Random Projection LSH, Bucketizer, Chi-Square Selector, Chi-Square Test, Correlation, Count Vectorizer, Discrete Cosine Transform (DCT), and Decision Trees for both classification and regression.
examples · high confidence
Add Kafka 0.10 connector implementation
This change introduces the core implementation files for the Kafka 0.10 integration in Spark Streaming. It adds the \DirectKafkaInputDStream\ for continuous streaming consumption, the \KafkaRDD\ for batch-oriented consumption with explicit offset control, and supporting classes like \KafkaRDDPartition\, \OffsetRange\, and \PerPartitionConfig\. The \KafkaUtils\ object provides the public API entry points (\createRDD\ and \createDirectStream\) for both Scala and Java users, while the \package.scala\ defines the necessary configuration properties for consumer caching and rate limiting.
connector/kafka-0-10/src/main · high confidence
Added BitSetMethods utility for fixed-size bitsets
The \BitSetMethods\ class has been added to the \org.apache.spark.unsafe.bitset\ package within the \common/unsafe\ module. This utility provides static methods for manipulating fixed-size, uncompressed bitsets stored in off-heap or primitive arrays, assuming word-aligned data. It exposes operations to set and unset individual bits, check if any bits are set, and efficiently find the next set bit starting from a given index, leveraging direct memory access via \Platform\ for performance.
common/unsafe/src/main/java/org/apache/spark/unsafe/bitset · high confidence
Added Hive-compatible hashing implementation
A new HiveHasher class has been added to provide hashing functions that are semantically equivalent to Hive v1.2.1. This enables users to generate hash values for integers, longs, and unsafe byte arrays that match Hive's behavior, ensuring compatibility when migrating or integrating with Hive-based systems.
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/expressions · high confidence
Added Kinesis word count example script
A new Python example script, kinesis\_wordcount\_asl.py, has been added to the connector's examples directory. This script demonstrates how to consume messages from an Amazon Kinesis stream and perform a word count using PySpark Streaming, including usage instructions and credential handling details.
connector/kinesis-asl/src/main/python · high confidence
Added sample data files for MLlib and GraphX algorithms
Added sample data files for various MLlib algorithms (including ALS, Gaussian Mixture, K-Means, Ridge Regression, and Isotonic Regression) and GraphX (followers and users), enabling users to run examples and tutorials without needing to download external datasets.
data · high confidence
Executor metrics and isolation infrastructure introduced
The executor module now exposes detailed runtime metrics and supports isolated session state management. Executor-level metrics (memory, CPU, thread pool stats) are registered with the metrics system and sent to the driver via heartbeats, while task-level metrics (shuffle read/write, input/output, serialization, spill) are tracked via accumulators and exposed through the metrics system. A new IsolatedSessionState class manages session-specific classloaders and resources with LRU eviction and reference counting to prevent race conditions. Additionally, new exception types (CommitDeniedException) and exit codes (ExecutorExitCode) provide better error handling and diagnostics for executor failures.
core/src/main/scala/org/apache/spark/executor · high confidence
Executor plugins now receive task lifecycle callbacks and resource information
The plugin system in Spark now supports task-level events for executor plugins. The \ExecutorPluginContainer\ invokes \onTaskStart\, \onTaskSucceeded\, and \onTaskFailed\ callbacks on registered executor plugins, allowing them to react to individual task execution phases. Additionally, the \PluginContext\ passed to plugins now exposes allocated resources via the \resources\ field, enabling plugins to monitor or utilize specific resource allocations during task execution.
core/src/main/scala/org/apache/spark/internal/plugin · high confidence
Executor-side S3A access via OIDC credentials
Executors can now access S3A storage using OIDC-derived credentials by configuring \fs.s3a.aws.credentials.provider\ to \org.apache.spark.security.aws.SparkOidcAwsCredentialsProvider\. This new provider reads session credentials from Spark's internal credential store, using version-based caching to minimize deserialization overhead and ensure thread-safe access during concurrent S3A operations.
_spark-credential-aws\2.13 · high confidence
Expose YARN Shuffle Service metrics to Hadoop metrics system
Added YarnShuffleServiceMetrics to forward shuffle service metrics (counters, meters, gauges, and timers) to the Hadoop metrics system, enabling NodeManager JMX monitoring of YARN shuffle service performance.
_spark-network-yarn\2.13 · high confidence
History Server event log processing and storage architecture refactored
The Spark History Server's internal event log handling has been restructured to improve performance and maintainability. A new \ApplicationHistoryProvider\ abstraction decouples the UI and caching logic from the underlying storage implementation. Event log reading and writing are now managed by dedicated \EventLogFileReaders\ and \EventLogFileWriters\ classes, which support rolling logs and various compression codecs. Additionally, a new \EventLogFileCompactor\ has been introduced to automatically compact old event logs into single files based on configurable thresholds, reducing storage overhead and improving load times for historical applications.
core/src/main/scala/org/apache/spark/deploy/history · high confidence
Incorporate Hive service infrastructure and authentication components
The hive-thriftserver module now includes core Hive service classes (AbstractService, BreakableService, CompositeService, FilterService, Service, ServiceException, ServiceOperations, ServiceStateChangeListener, ServiceUtils) and authentication providers (AnonymousAuthenticationProviderImpl, AuthenticationProviderFactory, CustomAuthenticationProviderImpl, HttpAuthenticationException). This brings in the underlying service lifecycle management and authentication logic required for the Thrift Server's operation, including cookie signing via CookieSigner.
sql/hive-thriftserver · high confidence
Initial GraphX API introduction
This release introduces the GraphX library for distributed graph processing in Spark. It provides core abstractions including Graph, VertexRDD, and EdgeRDD, along with utilities for loading graphs from edge-list files. The API supports iterative computation via a Pregel-like operator, offers graph algorithms such as PageRank and Connected Components through GraphOps, and allows efficient message passing and aggregation over edge triplets.
graphx · high confidence
Initial Java API for Spark RDDs and Context
This change introduces the initial Java programming API for Apache Spark, providing Java-friendly wrappers for core data structures and operations. It adds \JavaSparkContext\ to manage the Spark environment and create RDDs, alongside \JavaRDD\, \JavaPairRDD\, and \JavaDoubleRDD\ classes that expose transformations (like \map\, \filter\, \flatMap\) and actions (like \collect\, \count\) using Java collections and types. The package also includes \JavaHadoopRDD\ and \JavaNewHadoopRDD\ for Hadoop integration, \JavaSparkStatusTracker\ for monitoring job and stage progress, and utility classes like \JavaUtils\ to handle type conversions and serialization issues specific to the Java/Scala interop.
core/src/main/scala/org/apache/spark/api/java · high confidence
Initial implementation of Bloom filter and Count-Min Sketch data structures
The spark-sketch module now includes core probabilistic data structures: a Bloom filter (with V1 and V2 serialization formats, bitwise AND/OR merge operations, and support for strings, longs, and binary data) and a Count-Min Sketch (supporting frequency estimation for strings, longs, and binary data with configurable depth/width or error/confidence parameters). These classes provide the underlying serialization, hashing (Murmur3), and bit-array utilities required for these sketches, laying the groundwork for higher-level SQL and DataFrame APIs.
common/sketch/src/main · high confidence
Initial project scaffolding and developer workflow configuration
Establishes the foundational configuration files for the Apache Spark repository, including \.asf.yaml\ to enforce squash-only merging and configure GitHub Pages, \.gitattributes\ to standardize line endings across platforms, and \.pre-commit-config.yaml\ to integrate local Python formatting and linting. Adds \AGENTS.md\ to document development workflows, test base class hierarchies, and CI failure investigation procedures, alongside \SECURITY.md\ for vulnerability reporting guidelines and \scalastyle-config.xml\ to enforce code style rules.
(repo-wide) · high confidence
Introduce BinaryView and nanosecond timestamp support in unsafe types
This change adds the BinaryView class, a non-owning zero-copy view over byte arrays for opaque binary SQL types, and the TimestampNanosVal class, which provides a physical representation for nanosecond-precision timestamps. It also includes updates to UTF8String and ByteArray to support these new types and improve collation-aware string handling.
common/unsafe/src/main/java/org/apache/spark/unsafe/types · high confidence
Introduce BytesToBytesMap for off-heap key-value storage
Added the BytesToBytesMap class and its supporting HashMapGrowthStrategy to the Spark unsafe package. This new component provides an append-only, off-heap hash map for storing contiguous byte regions as keys and values, designed to integrate with Spark's memory management and external sorters for improved performance in aggregation and join operations.
core/src/main/java/org/apache/spark/unsafe · high confidence
Introduce KVStore API with LevelDB and RocksDB backends
Adds a new \KVStore\ abstraction for local key/value storage, featuring an in-memory implementation (\InMemoryStore\) and persistent backends using LevelDB and RocksDB. The API supports automatic key management, Jackson-based serialization with GZIP compression, and field indexing via the \@KVIndex\ annotation to enable efficient sorting and querying. This provides a unified interface for storing and retrieving application data with configurable persistence and indexing strategies.
common/kvstore/src/main · high confidence
Introduce Kubernetes-specific service providers and configuration classes
Added new service provider files for \SparkDiagnosticsSetter\, \SparkSubmitOperation\, and \ExternalClusterManager\ to register the Kubernetes implementations (\SparkKubernetesDiagnosticsSetter\, \K8SSparkSubmitOperation\, \KubernetesClusterManager\). Introduced core configuration and utility classes including \Config\, \Constants\, \KubernetesConf\, \KubernetesDriverConf\, \KubernetesUtils\, \KubernetesVolumeUtils\, and \SparkKubernetesClientFactory\ to handle Kubernetes client creation, volume parsing, and driver/executor configuration.
resource-managers/kubernetes/core · high confidence
Introduce ManagedBuffer interface and FileSegmentManagedBuffer implementation
This change adds the \ManagedBuffer\ abstract class and its \FileSegmentManagedBuffer\ implementation to the \common/network-common\ module. \ManagedBuffer\ defines a standard interface for immutable byte data views, supporting conversion to NIO ByteBuffers, Java InputStreams, and Netty objects (including specific handling for SSL via \convertToNettyForSsl\). The new \FileSegmentManagedBuffer\ allows Spark to efficiently manage file-backed data segments, utilizing memory mapping for large files and Netty's \DefaultFileRegion\ for zero-copy transfers when appropriate.
common/network-common/src/main/java/org/apache/spark/network/buffer · high confidence
Introduce Netty-based BlockTransferService and NettyBlockRpcServer
This change introduces the Netty-based implementation for Spark's block transfer and RPC layers, replacing the previous architecture. The new \NettyBlockTransferService\ handles block fetching and uploading via Netty, including authentication bootstraps and retry logic that checks executor liveness on I/O errors. The \NettyBlockRpcServer\ serves as the RPC handler for block operations (open, fetch, upload, corruption diagnosis) using a one-for-one stream management strategy. Configuration is centralized in \SparkTransportConf\, which now supports role-specific thread configurations and SSL options for shuffle and RPC modules.
core/src/main/scala/org/apache/spark/network/netty · high confidence
Introduce Netty-based RPC implementation
This change introduces a new RPC implementation based on the Netty network module, replacing the previous version. The new system includes a Dispatcher for routing messages, MessageLoop classes (Shared and Dedicated) for handling message delivery with configurable thread pools, and NettyStreamManager for serving files and jars over the network. This provides a more robust and performant communication layer for Spark's distributed computing tasks.
core/src/main/scala/org/apache/spark/rpc/netty · high confidence
Introduce OIDC credential provider SPI and core types
Spark 4.4.0 adds a new Service Provider Interface (SPI) for OIDC-based credential propagation, allowing third-party providers to resolve short-lived service credentials for specific URI schemes (e.g., S3, HDFS). The change introduces the \CredentialProvider\ interface with \ServiceLoader\ discovery, a \CredentialProviderLoader\ for scheme-based selection and configuration scoping, and core data types including \UserContext\ (driver-side identity), \ServiceCredential\ (serializable credentials sent to executors), and \TokenIngestor\ (with a \FileTokenIngestor\ implementation that detects file rotation by content). Configuration is now namespaced under \spark.security.oidc.\*\ to prevent secret leakage to provider implementations.
core/src/main/java/org/apache/spark/security · high confidence
Introduce Protobuf schema for Spark UI status data
Added \store\_types.proto\, a new Protobuf definition file that establishes the serialization format for Spark UI status data. This schema defines message types for core UI entities such as \JobData\, \TaskDataWrapper\, \ExecutorStageSummary\, and \AccumulableInfo\, along with an enum for \JobExecutionStatus\. It also includes developer guidelines for handling null strings and using \int64\ for IDs to ensure future extensibility.
core/src/main/protobuf/org · high confidence
Introduce Protobuf serialization and deserialization connectors
Adds the core Scala implementation for converting between Spark Catalyst data and Protobuf format. This change introduces the \to\_protobuf\ and \from\_protobuf\ expressions, along with the underlying \ProtobufSerializer\ and \ProtobufDeserializer\ classes, enabling users to serialize Spark DataFrames into Protobuf binary data and deserialize Protobuf data back into Spark tables.
connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf · high confidence
Introduce REST-based application submission for Standalone clusters
The Standalone mode now supports submitting, managing, and monitoring applications via a new REST API. This change adds a \RestSubmissionClient\ and \RestSubmissionServer\ (with \StandaloneRestServer\ implementation) that expose endpoints for creating submissions, killing drivers or all applications, holding/resuming applications, clearing completed state, and checking readiness. The server supports configurable thread pools (including Java 21 virtual threads), request body size limits, and custom filters, while the client handles connection failures and retries across multiple masters.
core/src/main/scala/org/apache/spark/deploy/rest · high confidence
Introduce SCD1 and SCD2 AutoCDC batch processors and auxiliary table management
Declarative Pipelines now supports automatic Change Data Capture (AutoCDC) for Slowly Changing Dimensions (SCD) Type 1 and Type 2. This change introduces the core batch processing logic for both SCD strategies, including microbatch reconciliation, deduplication, and merging into target tables. It also adds the infrastructure for managing auxiliary state tables (used for tombstones in SCD1 and version tracking in SCD2) and defines reserved naming conventions (prefixed with \\_\_spark\autocdc\\) to distinguish internal AutoCDC columns and tables from user-defined ones. Users can now define AutoCDC flows that automatically maintain historical versions (SCD2) or latest-state snapshots (SCD1) without manual merge logic.
sql/pipelines · high confidence
Introduce ShuffleSecretManager for external shuffle service authentication
Added the ShuffleSecretManager class to manage shuffle secrets for the external shuffle service. This component allows applications to register and unregister their secrets, which are then used to authenticate executors before they can fetch shuffle files. It maintains a concurrent map of application IDs to their corresponding secrets and provides the SASL user for authentication.
common/network-shuffle/src/main/java/org/apache/spark/network/sasl · high confidence
Introduce Spark Connect JDBC driver
Adds a new JDBC driver module that allows connecting to a Spark Connect server using the standard JDBC URL format (jdbc:sc://). The driver registers itself via the Java SPI service file and implements core JDBC interfaces including Connection, Statement, ResultSet, DatabaseMetaData, and type-specific wrappers for complex data (Array, Map, Struct). It maps Spark Connect client errors to appropriate SQLExceptions and supports basic catalog/schema operations and type metadata.
sql/connect/client/jdbc · high confidence
Introduce Spark Connect REPL and client-side test infrastructure
Adds a new interactive REPL for Spark Connect (ConnectRepl) that launches a local server and provides an Ammonite-based shell with automatic class-finding for UDFs, alongside a suite of new client-side tests covering encoders, nearest-by joins, subqueries, table-valued functions, and geospatial data types.
sql/connect/client/jvm · high confidence
Introduce Streaming Shuffle Manager and Netty-based data transfer
This change adds the core components for Spark's new Streaming Shuffle feature, enabling pipelined data transfer between shuffle writers and readers via Netty. It introduces the StreamingShuffleManager and a MultiShuffleManager (which routes shuffles to either the new streaming manager or the existing SortShuffleManager based on a query property) to handle shuffle registration and delegation. The implementation includes a StreamingShuffleReader and StreamingShuffleClientHandler on the consumer side, and a StreamingShuffleWriter with a StreamingShuffleServerHandler on the producer side, facilitating a push-based protocol with credit control and termination signaling. Additionally, a TaskContextAwareLogging trait is added to enrich logs with streaming-specific identifiers like query ID and shuffle ID.
core/src/main/scala/org/apache/spark/shuffle/streaming · high confidence
Introduce Variant common module with shredding, manipulation, and strict parsing
This change adds the \common/variant\ module, providing the core Java implementation for the Apache Parquet Variant type. It includes utilities to parse JSON into Variant binaries (with strict UTF-8 validation and support for duplicate keys), rebuild Variant values from shredded Parquet data, and manipulate Variant structures via expressions like \variant\_set\, \variant\_delete\, \variant\_insert\, \variant\_array\_append\, \variant\_pick\, and \variant\_strip\_nulls\. The module also defines the \VariantSchema\ for shredding, a \VariantShreddingWriter\ for converting Variants to shredded columns, and enforces a 128 MiB size limit on Variant values and metadata.
common/variant · high confidence
Introduce YarnShuffleService as the external shuffle service for YARN deployments
This change adds the \YarnShuffleService\ class, a long-running auxiliary service that runs within the YARN NodeManager process to handle shuffle data for Spark applications. It allows executors to register and fetch shuffle blocks via a dedicated transport server, supporting optional authentication to isolate data between applications. The service manages state recovery (including executor registration and shuffle merge metadata) using a configurable database backend, exposes metrics to the YARN NodeManager's metrics system, and can be configured via Hadoop configuration or a classpath overlay resource.
common/network-yarn/src/main/java/org/apache/spark/network/yarn · high confidence
Introduce approximate result support for Spark actions
This change adds the \org.apache.spark.partial\ package, introducing a new capability for approximate single-result actions like \countApprox\. It includes \ApproximateActionListener\ to handle job completion with timeouts, \PartialResult\ for managing async results and callbacks, and specific evaluators (\CountEvaluator\, \MeanEvaluator\, \SumEvaluator\, \GroupedCountEvaluator\) that compute confidence intervals using statistical distributions (Poisson, Normal, t-distribution) from the Apache Commons Math library.
core/src/main/scala/org/apache/spark/partial · high confidence
Introduce asynchronous RDD actions and new RDD implementations
This change introduces the \AsyncRDDActions\ class, providing non-blocking \countAsync\, \collectAsync\, \takeAsync\, \foreachAsync\, and \foreachPartitionAsync\ methods that return Scala \FutureAction\ results, allowing users to execute RDD actions without blocking the calling thread. Additionally, it adds several new RDD types to the core API: \BinaryFileRDD\ for generic binary file input, \BlockRDD\ for direct access to block manager storage, \CartesianRDD\ for computing cross products of two RDDs, \CheckpointRDD\ as a base for recovering checkpointed data, and \EmptyRDD\ for representing RDDs with no partitions. The \CoGroupedRDD\ implementation is also updated to support configurable serializers for shuffle operations.
core/src/main/scala/org/apache/spark/rdd · high confidence
Introduce collation-aware string processing utilities
Added new utility classes in the catalyst util package to enable collation-aware string operations. CollationFactory provides the central entry point for managing collation settings, including identifiers, metadata, and comparison/sorting rules for UTF8\_BINARY, UTF8\_LCASE, and ICU-based collations. CollationAwareUTF8String implements the low-level logic for case-insensitive matching and Unicode handling. CollationSupport exposes static methods for common string expressions (such as Contains, StartsWith, EndsWith, and SplitSQL) to delegate to the appropriate collation-specific implementation (binary, lowercase, or ICU). CollationNames centralizes the definitions of supported collation identifiers.
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util · high confidence
Introduce gRPC-based UDF worker dispatch over Unix domain sockets
Spark now supports a new \Direct\ dispatch mode for external UDF workers, enabling communication via the gRPC protocol over Unix domain sockets. This change adds the \DirectGrpcDispatcher\ and \GrpcWorkerChannel\ components, which manage worker lifecycle, allocate private socket directories, and handle connection readiness with proper backoff reset logic. To ensure protocol compatibility, the gRPC channel overrides the HTTP/2 authority to \localhost\, preventing connection failures caused by invalid authority headers derived from socket paths. The implementation requires workers to declare \BIDIRECTIONAL\_STREAMING\ capability and is accompanied by integration and concurrency tests to validate the execution protocol and session management.
udf/worker/grpc · high confidence
Introduce internal Serializer API for object relocation support
The \Serializer\ base class now includes a private \supportsRelocationOfSerializedObjects\ method, allowing the framework to determine if a serializer's output can be safely reordered in the stream. This capability is required for optimizing shuffle fetches of continuous partition IDs, enabling more efficient data transfer by reordering serialized objects without deserializing them first.
core/src/main/scala/org/apache/spark/serializer · high confidence
Introduce language-agnostic UDF worker protocol definitions
This change adds the core Protocol Buffers definitions for the new language-agnostic UDF worker architecture. It introduces the \UdfWorker\ gRPC service with \Execute\ and \Manage\ RPCs, defines the bidirectional streaming message protocol for UDF execution (including \Init\, \PayloadChunk\, \DataRequest\, and \Finish\ phases), and specifies the worker capability negotiation schema. This establishes the wire format and service contract that allows the Spark engine to communicate with external UDF workers in any language.
udf/worker/proto · high confidence
Introduce local-disk shuffle I/O components via new Shuffle API
The local-disk shuffle implementation now uses the new Shuffle Driver and Executor Components API. This adds LocalDiskShuffleDriverComponents for driver-side lifecycle management (initialization and shuffle removal via BlockManagerMaster) and LocalDiskShuffleExecutorComponents for executor-side setup, including creating LocalDiskShuffleMapOutputWriter and LocalDiskSingleSpillMapOutputWriter instances. The writers persist shuffle data to local disk alongside index files, support configurable output buffer sizes, and return custom map output metadata (partition lengths) upon commit.
core/src/main/java/org/apache/spark/shuffle/sort/io · high confidence
Introduce new AppStatusStore and plugin architecture for Spark History Server
The Spark History Server now uses a new \AppStatusStore\ backed by a \KVStore\ (supporting both LevelDB and RocksDB backends) to persist and serve application state, replacing the previous \JobProgressListener\-based approach. This change introduces the \AppHistoryServerPlugin\ interface, allowing modules like SQL to provide their own listeners and UI setup for history replay. The new store exposes richer data via REST APIs, including stage-level task filtering by status, task metrics distributions, and executor memory metrics, while also improving performance through protobuf serialization for the live UI and optimized iterator handling.
core/src/main/scala/org/apache/spark/status · high confidence
Introduce new network client callback interfaces and core transport classes
The network client layer now defines a set of new interfaces and classes to handle RPC, chunk fetching, and streaming responses. This includes the \BaseResponseCallback\ interface (extended by \RpcResponseCallback\ and \MergedBlockMetaResponseCallback\) for handling RPC results, \ChunkReceivedCallback\ for chunk fetch results, and \StreamCallback\/\StreamCallbackWithID\ for streaming data. Additionally, \ChunkFetchFailureException\ is introduced to represent fetch errors, \StreamInterceptor\ is added to feed stream data to callbacks, and the core \TransportClient\, \TransportClientFactory\, \TransportResponseHandler\, and \TransportClientBootstrap\ classes are implemented to manage connections, request handling, and bootstrapping.
common/network-common/src/main/java/org/apache/spark/network/client · high confidence
Introduce new network-shuffle protocol messages for corruption diagnosis and merge management
The network-shuffle protocol now includes new message types to support shuffle data corruption diagnosis and shuffle merge lifecycle management. Specifically, the \DiagnoseCorruption\ and \CorruptionCause\ messages allow the system to identify and report the specific cause of shuffle data corruption, while \FinalizeShuffleMerge\, \MergeStatuses\, and \RemoveShuffleMerge\ messages facilitate the handling of merged shuffle blocks. These changes are implemented in the \BlockTransferMessage\ protocol class, which now handles these new message IDs and updates the decoder to route them to the appropriate handlers.
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol · high confidence
Introduce new random sampling utilities and RNG implementation
This change adds a new \org.apache.spark.util.random\ package containing core random number generation and sampling infrastructure. It introduces the \XORShiftRandom\ class, a faster alternative to \java.util.Random\ that uses the XORShift algorithm and MurmurHash3 for seed hashing. The package also provides a \RandomSampler\ trait hierarchy with \BernoulliSampler\ and \PoissonSampler\ implementations, supporting gap-sampling optimizations for efficiency. Additionally, it includes \SamplingUtils\ for reservoir sampling and statistical bounds calculation, and \StratifiedSamplingUtils\ to enable exact sample size guarantees for stratified sampling in \PairRDDFunctions\.
core/src/main/scala/org/apache/spark/util/random · high confidence
Introduce new security utilities and socket authentication infrastructure
This change adds several new components to the Spark security module to support encrypted I/O and robust inter-process communication. CryptoStreamUtils provides utilities for wrapping streams and channels with AES encryption using commons-crypto, including initialization vector generation and key creation. SocketAuthServer and SocketAuthHelper implement a new authentication protocol for socket-based communication, supporting both TCP/IP and Unix Domain Sockets to securely connect external processes (like Python) with the JVM. Additionally, the module introduces GroupMappingServiceProvider and ShellBasedGroupsMappingProvider for user group resolution, HadoopDelegationTokenProvider for delegation token management, and SecurityConfigurationLock to synchronize global JVM security configuration modifications.
core/src/main/scala/org/apache/spark/security · high confidence
Introduce pluggable BlockTransferService and BlockDataManager interfaces
This change introduces two new core interfaces in the network layer: \BlockTransferService\ and \BlockDataManager\. \BlockTransferService\ provides a pluggable abstraction for remote block transfer, including methods to upload and fetch blocks (both synchronously and asynchronously) and exposes the service's host and port. \BlockDataManager\ defines the local block management capabilities, such as storing blocks, retrieving local or host-local shuffle data, and diagnosing shuffle data corruption via checksums. These interfaces decouple the network transport mechanism from the block storage logic, allowing for different implementations (e.g., Netty-based) while standardizing how blocks are moved between nodes and managed locally.
core/src/main/scala/org/apache/spark/network · high confidence
Introduce pluggable fault-tolerance and recovery for the Standalone Master
The Standalone Master now supports configurable fault tolerance via a pluggable recovery architecture. Users can choose between ZooKeeper, a local Filesystem, or a new RocksDB persistence engine to store cluster state (applications, workers, drivers) for recovery after master failure. This is driven by the \RECOVERY\_MODE\ configuration and implemented through the \StandaloneRecoveryModeFactory\, which creates the specific \PersistenceEngine\ and \LeaderElectionAgent\ (e.g., \ZooKeeperLeaderElectionAgent\ or \MonarchyLeaderAgent\) required for the chosen mode.
core/src/main/scala/org/apache/spark/deploy/master · high confidence
Introduce pluggable shuffle data I/O and checksum verification
The shuffle subsystem now supports custom data I/O plugins via the new \ShuffleDataIO\ interface and \ShuffleDataIOUtils\ loader, allowing users to replace the default shuffle transport. Additionally, \ShuffleChecksumUtils\ has been added to enable verification of shuffle data integrity by comparing checksums against data and index files, improving fault detection for corrupted shuffle blocks.
core/src/main/scala/org/apache/spark/shuffle · high confidence
Introduce programmatic Spark launcher API with in-process and child-process support
The launcher module now provides a new Java API for programmatically launching Spark applications. Users can choose between launching as a separate child process (via \SparkLauncher\) or running directly in the current JVM (via \InProcessLauncher\). This new infrastructure introduces a \LauncherServer\ that manages communication between the launcher and the launched application, using a secret-based protocol defined in \LauncherProtocol\. The API includes \AbstractAppHandle\ for tracking application state and listeners, \AbstractCommandBuilder\ for constructing the execution command, and \JavaModuleOptions\ to handle JVM-specific flags required for modern Java versions (e.g., Java 17+ module opens). Additionally, \FilteredObjectInputStream\ is introduced to restrict deserialized classes for security, and \Main\ provides a CLI interface for internal script usage.
launcher · high confidence
Introduce proto-based configuration system with textproto definitions
The \common/config\ module now provides a new configuration framework where Spark settings are defined in Protocol Buffers text format (\.textproto\) files and loaded at runtime by \ConfigRegistry\. This change introduces a schema (\config\_schema.proto\) that enforces strict typing, scope (CLUSTER vs. SESSION), mutability (STATIC vs. DYNAMIC), and visibility rules for configuration entries. The \ConfigRegistry\ class loads these definitions from classpath resources, validating required fields, alphabetical ordering, and scope/mutability consistency. Initial SQL configurations are migrated into this system, with cluster-scoped configs like \spark.sql.ui.retainedExecutions\ and session-scoped configs like \spark.sql.optimizer.maxIterations\ now defined in \cluster\_configs/sql.textproto\ and \session\_configs/sql.textproto\ respectively.
common/config · high confidence
Introduce push-based shuffle network protocol and resolver
The external shuffle service now supports a push-based shuffle model, introducing new network protocol messages (such as FetchShuffleBlockChunks, FinalizeShuffleMerge, and BlockPushReturnCode) and a new RemoteBlockPushResolver implementation to manage merged shuffle files and state. This change adds the necessary network-layer infrastructure to allow executors to push shuffle blocks directly to the shuffle service, enabling features like work-preserving restarts and improved handling of stage retries.
_spark-network-shuffle\2.13 · high confidence
Introduce standalone Worker Web UI with log viewing and resource monitoring
This change adds the initial implementation of the Web UI for the Spark Standalone Worker, providing operators with a browser-based interface to monitor cluster health and debug issues. The new UI includes a Worker Page that displays the worker's ID, resource utilization (cores and memory), and collapsible tables listing running and finished executors and drivers, with links to application UIs and logs. It also introduces a Log Page that allows users to view and paginate through executor, driver, and worker logs (stdout, stderr, out) with support for offset-based loading and dynamic 'Load More'/'Load New' buttons, while ensuring path safety via directory validation and URL encoding.
core/src/main/scala/org/apache/spark/deploy/worker/ui · high confidence
Introduce standalone driver client and external shuffle service
This change adds the standalone driver client infrastructure, including the \Client\ and \ClientArguments\ classes that handle launching and killing drivers in standalone cluster mode, and introduces the \ExternalShuffleService\ daemon which allows executors to read shuffle files from a dedicated service rather than directly from each other, improving robustness against executor failures. It also adds supporting components such as \DriverEnvironment\ for filtering environment variables passed to drivers, \DriverTimeoutPlugin\ for configurable driver timeouts, \ExecutorFailureTracker\ for tracking executor failures, and \LocalSparkCluster\ for testing standalone cluster operations.
core/src/main/scala/org/apache/spark/deploy · high confidence
Introduce standalone worker process management components
This change adds the core Scala classes for the Spark Standalone Worker subsystem, including the Worker daemon itself, the WorkerArguments parser, and the WorkerWatcher for fate-sharing. It introduces DriverRunner and ExecutorRunner to manage the lifecycle of driver and executor processes, along with CommandUtils to handle process building and environment setup. These components enable the worker to launch, monitor, and terminate child processes while handling configuration, security, and resource management.
core/src/main/scala/org/apache/spark/deploy/worker · high confidence
Introduce streaming shuffle wire protocol and AES-GCM RPC encryption
This change adds the core message types for the new streaming shuffle wire protocol (including DataMessage, CreditControlMessage, and TerminationAckMessage) and introduces AES-GCM encryption for RPC traffic via the new GcmTransportCipher, alongside the existing CTR cipher. It also adds Java record-based protocol messages (AuthMessage, StreamChunkId) and new ManagedBuffer implementations (NettyManagedBuffer, NioManagedBuffer) to support these capabilities.
_spark-network-common\2.13 · high confidence
Introduce streaming shuffle wire protocol message types
Added the base \StreamingShuffleMessage\ class and the \StreamingShuffleMessageType\ enum to define the wire protocol for the new streaming shuffle feature. This change establishes the message structure, including sequence number tracking for ordering and reliability, and defines the four supported message types: data messages, credit control, termination control, and termination acknowledgment.
common/network-common/src/main/java/org/apache/spark/network/shuffle · high confidence
Introduce structured logging framework in Scala Logging trait
The \Logging\ trait in \common/utils\ has been updated to support a new structured logging framework. Users can now use the \log"..."\ string interpolator with \MDC\ wrappers to automatically capture variables as structured metadata in the Mapped Diagnostic Context (MDC), enabling better log analysis. The trait also introduces \LogEntry\ for lazy evaluation of log messages with context, ensures that \MessageWithContext\ values are not used as MDC values, and respects the \spark.log.structuredLogging.enabled\ configuration to conditionally apply structured logging behavior.
common/utils/src/main/scala/org/apache/spark/internal · high confidence
Introduce unified memory manager with dynamic storage-execution borrowing
Spark now uses a new UnifiedMemoryManager that replaces the legacy static memory model, allowing execution and storage memory pools to dynamically borrow from each other. This change introduces dedicated ExecutionMemoryPool and StorageMemoryPool classes to manage on-heap and off-heap memory separately, enabling execution tasks to evict cached blocks to reclaim storage memory when needed, while preventing storage from evicting execution memory. The system also integrates unmanaged memory tracking (e.g., for RocksDB) into memory allocation decisions to prevent OOM errors, and enforces fair memory distribution among active tasks by limiting each task's share to a fraction of the total pool size.
core/src/main/scala/org/apache/spark/memory · high confidence
Introduce unsafe in-memory and external sorters with radix sort support
This change adds a new set of classes in the \org.apache.spark.util.collection.unsafe.sort\ package to handle sorting operations using off-heap memory. It introduces \UnsafeInMemorySorter\ and \UnsafeExternalSorter\ which utilize key-prefix sorting to improve cache efficiency, and includes a \RadixSort\ implementation for faster sorting of primitive types like longs and doubles. The package also provides \PrefixComparators\ for handling various data types (String, Binary, Long, Double) with support for ascending/descending order and null handling, as well as bounded spill merging via \UnsafeSorterBoundedSpillMerger\ to prevent OOM during the merge phase of external sorting.
core/src/main/java/org/apache/spark/util/collection/unsafe · high confidence
Introduce v1 REST API for Spark status and metrics
This change introduces a new v1 REST API under the /api/v1 path to expose Spark application status, metrics, and configuration data as JSON. It provides endpoints to list applications with filtering by status and date ranges, retrieve detailed information for individual applications (including jobs, executors, stages, and environment details), and download event logs. Additionally, it adds a separate /metrics endpoint to export executor metrics in Prometheus format, enabling integration with external monitoring systems.
core/src/main/scala/org/apache/spark/status/api · high confidence
Introduces abstraction for shuffle state database backends
The shuffle service's local state storage is now abstracted via a new \DB\ interface and a \DBBackend\ enum, enabling support for both LevelDB and RocksDB. This change adds specific iterator implementations (\LevelDBIterator\, \RocksDBIterator\) to wrap the underlying database libraries and a \StoreVersion\ class to track data format versions, laying the groundwork for pluggable storage backends in the shuffle service.
common/network-common/src/main/java/org/apache/spark/network/shuffledb · high confidence
Introduces core abstractions for language-agnostic UDF worker management
This change adds the foundational Scala interfaces and implementations in \udf/worker/core\ that enable Spark to spawn and manage external UDF worker processes. It introduces \UDFDispatcherManager\ to cache and reuse \WorkerDispatcher\ instances based on worker specifications, and \WorkerDispatcher\ to handle the lifecycle of worker processes and sessions. The \DirectWorkerDispatcher\ implementation allows spawning workers as local callable processes, communicating via Unix domain sockets (with path-length handling for macOS/Linux) or TCP. New types like \WorkerConnection\, \WorkerSession\, \WorkerHandle\, and \WorkerSecurityScope\ decouple the transport and session logic from the provisioning model, while \WorkerLogger\ provides a minimal, backend-agnostic logging surface. Tests verify the dispatcher lifecycle, caching behavior, and socket management.
udf/worker/core · high confidence
Introduces network-common utility classes for I/O, configuration, and state storage
This change adds a suite of new utility classes to the network-common module to support Spark's transport and shuffle infrastructure. It introduces I/O abstractions including AbstractFileRegion, ByteArrayReadableChannel, ByteArrayWritableChannel, and ByteBufferWriteableChannel for efficient data handling. Configuration is abstracted via ConfigProvider and MapConfigProvider, while crypto settings are managed by CryptoUtils. State persistence is supported through DBProvider, LevelDBProvider, and RocksDBProvider, which handle initialization and versioning for shuffle state stores. Additionally, NettyUtils provides platform-aware transport selection (NIO, EPOLL, KQUEUE, AUTO), NettyMemoryMetrics exposes allocator metrics, NettyLogger integrates structured logging, and TimerWithCustomTimeUnit allows flexible timing units for metrics.
common/network-common/src/main/java/org/apache/spark/network/util · high confidence
Introduces new core classes for barrier execution and partition evaluation
Adds new Scala classes to the Spark core API to support barrier execution mode and SQL partition evaluation. This includes \BarrierTaskInfo\ to carry task information for barrier tasks, \PartitionEvaluator\ and \PartitionEvaluatorFactory\ to enable in-process pipelined channel shuffles for SQL operators, and \Heartbeater\ to manage periodic heartbeat reporting. Additionally, \ContextAwareIterator\ is deprecated as its usage for Python evaluation is no longer needed.
core/src/main/scala/org/apache/spark · high confidence
Introduces unsafe implementations for Murmur3 hashing, heap memory pooling, CalendarInterval, and Variant types
This change adds four new classes to the \spark-unsafe\ module to support core SQL and memory features. \Murmur3\_x86\_32\ provides a 32-bit Murmur3 hasher implementation that handles big-endian platforms and maintains backward compatibility with pre-2.3 hash outputs. \HeapMemoryAllocator\ introduces a pooling mechanism for on-heap memory allocations (threshold 1MB) and fills freed memory with debug values to help catch use-after-free bugs. \CalendarInterval\ defines the internal representation for calendar intervals using months, days, and microseconds, including methods to extract \java.time.Period\ and \Duration\ objects. \VariantVal\ implements the physical binary representation for the Variant data type, storing value and metadata bytes to support semi-structured data handling.
_spark-unsafe\2.13 · high confidence
Introduction of Coarse-Grained Scheduler Backend and Cluster Package Structure
This change introduces the \CoarseGrainedSchedulerBackend\ class and its supporting data structures (\ExecutorData\, \ExecutorInfo\) into the new \org.apache.spark.scheduler.cluster\ package, establishing the core infrastructure for managing executor lifecycle, resource tracking, and task scheduling in Spark's standalone deploy mode. It also adds \StandaloneSchedulerBackend\ as the specific implementation for the standalone cluster manager, which extends the coarse-grained backend to handle communication with the Spark Master, and includes \SchedulerBackendUtils\ to determine initial executor counts based on dynamic allocation settings.
core/src/main/scala/org/apache/spark/scheduler/cluster · high confidence
Introduction of SQL Connect Shims for API Compatibility
A new shim layer has been added to the SQL Connect module to provide stub classes for core Spark components, including SparkContext, SparkConf, RDD, and various SQL execution internals (such as SparkSessionExtensions, QueryExecution, and SessionState). This change establishes a unified Scala interface that allows the Connect server to interact with classic Spark APIs by defining the necessary structural types, facilitating the migration and compatibility between the Connect architecture and existing SQL/RDD functionality.
sql/connect/shims · high confidence
Introduction of Sort-based Shuffle Manager and Writer
This change introduces the core components for Spark's sort-based shuffle mechanism, specifically the SortShuffleManager and SortShuffleWriter. The SortShuffleManager handles the registration and lifecycle of shuffles, supporting three distinct write paths: bypass merge sort for small partition counts, serialized sorting for efficient memory usage when possible, and standard deserialized sorting. The SortShuffleWriter implements the actual data writing logic, utilizing an ExternalSorter to handle in-memory and on-disk sorting, and integrates with the new ShuffleExecutorComponents API for file creation. It also includes logic to track and report checksums for data integrity verification during the shuffle write process.
core/src/main/scala/org/apache/spark/shuffle/sort · high confidence
Introduction of SparkListenerEvent trait for event serialization
A new \SparkListenerEvent\ trait has been added to the \org.apache.spark.scheduler\ package. This trait serves as a base for Spark listener events, providing Jackson annotations (\@JsonTypeInfo\) to facilitate class-based serialization and a \logEvent\ method to control whether specific events are written to the event log.
common/utils/src/main/scala/org/apache/spark/scheduler · high confidence
Introduction of core in-memory and spilling collection data structures
Spark now includes a suite of optimized collection utilities in the \org.apache.spark.util.collection\ package to power shuffle and aggregation operations. This adds \AppendOnlyMap\ and \OpenHashMap\ for efficient in-memory key-value storage with quadratic probing, \OpenHashSet\ for fast set operations, and \BitSet\/\ImmutableBitSet\ for compact bit manipulation. For large-scale data that exceeds memory, \ExternalAppendOnlyMap\ and \ExternalSorter\ provide spilling capabilities to disk, while \CompactBuffer\ offers a memory-efficient alternative to \ArrayBuffer\ for small sequences. These components form the foundation for the new sort-based shuffle and cooperative memory management features.
core/src/main/scala/org/apache/spark/util/collection · high confidence
Introduction of new StreamingContext lifecycle and API structures
This change introduces several new core classes and enums for the Spark Streaming module, including \StreamingContextState\ to explicitly track context states (INITIALIZED, ACTIVE, STOPPED), \BatchStatus\ for the v1 REST API, and \ContextWaiter\ to manage stop/error notifications. It also adds the \WriteAheadLog\ abstract class and its record handle, alongside \DStreamGraph\, \Interval\, and \State\ implementations, establishing the foundational data structures and lifecycle management for the streaming engine.
streaming · high confidence
Introduction of the org.apache.spark.io compression codec package
This change introduces the \org.apache.spark.io\ package, establishing the core infrastructure for internal data compression. It defines the \CompressionCodec\ trait and provides concrete implementations for LZ4, LZF, Snappy, and ZStandard codecs, allowing users to configure compression via \SparkConf\. The package also includes \HadoopCodecStreams\ to handle decompression for Hadoop-compatible files (including ZStandard), and utility classes like \MutableCheckedOutputStream\ for checksumming.
core/src/main/scala/org/apache/spark/io · high confidence
Kafka source now supports topic-level earliest/latest offsets in startingOffsets and endingOffsets
Users can now specify \earliest\ or \latest\ as topic-level values in the \startingOffsets\ and \endingOffsets\ JSON configuration (e.g., \{"topicA":"earliest","topicB":{"0":23}}\), in addition to the existing per-partition offset specifications. This change is implemented in the \JsonUtils\ component of the Kafka connector, which parses these topic-level limits and defers their expansion to specific partition offsets until the actual partitions are known during query execution.
connector/kafka-0-10-sql/src/main · high confidence
New AWS STS OIDC credential provider for S3 access
The connector/credential-aws module introduces AwsStsCredentialProvider, which exchanges an OIDC identity token for temporary AWS credentials via the STS AssumeRoleWithWebIdentity API. It configures S3A-compatible Hadoop properties (fs.s3a.access.key, fs.s3a.secret.key, fs.s3a.session.token) for executor-side access, with configuration keys under spark.security.oidc.aws (roleArn required; sessionName, durationSeconds, region, and stsEndpoint optional). The provider is registered via META-INF/services for ServiceLoader discovery, and tests verify initialization validation, credential resolution, and refresh behavior.
connector/credential-aws · high confidence
New ConsoleProgressBar and enhanced UI security features
The Spark Web UI now includes a ConsoleProgressBar that displays stage progress in the console output, improving visibility for interactive sessions. Additionally, the UI introduces a new HttpSecurityFilter that enforces Content-Security-Policy headers with nonces for inline scripts, adds JWSFilter support for JWT-based authentication, and implements XSS protection by sanitizing request parameters. A new DriverLogPage provides a dedicated UI tab for viewing driver logs with pagination and dynamic loading capabilities.
core/src/main/scala/org/apache/spark/ui · high confidence
New Java API classes for async actions, optional values, and storage levels
The Java API package now includes three new classes to improve usability and compatibility. JavaFutureAction provides a Java-friendly interface for asynchronous operations, exposing the list of job IDs associated with the underlying async task. A new Optional class is introduced to replace Guava's Optional in the public API, offering a work-alike implementation that supports both java.util.Optional and Guava-style methods (such as absent, fromNullable, or, and orNull) to avoid library version conflicts. Additionally, StorageLevels exposes commonly used storage level constants (including DISK\_ONLY\_3 and OFF\_HEAP) as static fields for easier access in Java code.
core/src/main/java/org/apache/spark/api/java · high confidence
New Java API classes for job/stage info and event listening
The core package now exposes new Java-facing types: \SparkJobInfo\ and \SparkStageInfo\ interfaces to expose job and stage metadata (including job status via the new \JobExecutionStatus\ enum), \SparkExecutorInfo\ to expose executor details, and \SparkFirehoseListener\ to allow users to receive all SparkListener events in Java. These additions provide a stable Java API surface for monitoring and reacting to Spark execution state.
core/src/main/java/org/apache/spark · high confidence
New Java examples for Logistic Regression and Log Querying
Added two new Java example programs to the Spark examples package: JavaHdfsLR, which demonstrates a naive implementation of logistic regression using SparkSession and RDDs, and JavaLogQuery, which performs roll-up style queries against Apache log files. These examples provide users with reference implementations for common data processing tasks in Java.
_spark-examples\2.13 · high confidence
New Kafka delegation token provider with secure logging and proxy user support
This change introduces the \KafkaDelegationTokenProvider\ component in the \connector/kafka-0-10-token-provider\ module, enabling Spark to obtain Kafka delegation tokens for secure cluster communication. The provider now supports obtaining tokens for proxy users (requiring Kafka 3.3.0+ brokers) and includes robust error handling, such as classifying timestamp offset mismatches and preventing AdminClient leaks. Additionally, it implements comprehensive redaction of JAAS credentials (passwords, client secrets) in logs to prevent credential exposure, and uses structured logging for better observability.
connector/kafka-0-10-token-provider/src/main · high confidence
New PeriodicRDDCheckpointer utility for automatic RDD persistence and checkpointing
A new \PeriodicRDDCheckpointer\ class has been added to the Spark core utilities, allowing users to automatically manage the persistence and checkpointing of RDDs over time. This utility helps prevent memory overflow by persisting new RDDs and unpersisting older ones, while also periodically checkpointing data to durable storage. Users can configure the checkpoint interval and storage level, with the default storage level set to \MEMORY\_ONLY\. This tool is particularly useful for long-running applications that generate a sequence of RDDs and need to manage resource usage efficiently.
core/src/main/scala/org/apache/spark/rdd/util · high confidence
New RDD operation graph model for UI visualization
Introduces a new internal data model (RDDOperationGraph, RDDOperationNode, RDDOperationCluster) in the Spark UI scope to represent RDD dependencies and operation scopes. This model structures the graph with explicit edges, nodes, and nested clusters, enabling the Web UI to display detailed DAG visualizations including cached status, barrier execution flags, callsites, and deterministic levels for RDDs.
core/src/main/scala/org/apache/spark/ui/scope · high confidence
New ResourceDiscoveryPlugin API for custom resource discovery
A new \ResourceDiscoveryPlugin\ interface has been added to the Spark core API, allowing developers to create custom plugins that dynamically discover addresses for custom resources (such as GPUs or specialized hardware) during application initialization. This interface enables chaining of multiple plugins to handle different resource types, with Spark falling back to default discovery scripts if a plugin does not handle a specific request, thereby supporting advanced resource scheduling scenarios like those required by YARN.
core/src/main/java/org/apache/spark/api/resource · high confidence
New SASL-based authentication and encryption for Spark network transport
Spark now supports SASL DIGEST-MD5 authentication and optional AES encryption for all network-layer communication. This change introduces a new security mechanism in the transport library where clients and servers perform a challenge-response handshake before any data exchange; upon successful authentication, the connection can be configured to encrypt all subsequent traffic using SASL encryption handlers. This ensures that RPCs and data streams are authenticated and, when enabled, encrypted over the wire, protecting against eavesdropping and unauthorized access.
common/network-common/src/main/java/org/apache/spark/network/sasl · high confidence
New SSLFactory and ReloadingX509TrustManager for RPC SSL
The RPC SSL subsystem now uses a new \SSLFactory\ class to manage SSL contexts, supporting both JDK-based keystores/truststores and Netty-based PEM certificates. A key addition is the \ReloadingX509TrustManager\, which automatically reloads the truststore configuration when the underlying file on disk changes, enabling dynamic truststore updates without restarts. The factory also defaults to TLS 1.3 if no protocol is specified and supports separate key passwords for JKS keystores.
common/network-common/src/main/java/org/apache/spark/network/ssl · high confidence
New Spark JVM Profiler plugin for driver and executor code profiling
The connector/profiler module introduces a new Spark plugin that enables low-overhead JVM code profiling on both drivers and executors using Async Profiler. Users can enable driver profiling via \spark.profiler.driver.enabled\ and select a random fraction of executors for profiling via \spark.profiler.executor.enabled\ and \spark.profiler.executor.fraction\. The plugin captures Java Flight Recorder (JFR) files locally, with an optional periodic sync to an HDFS path (\spark.profiler.dfsDir\). A new \spark.profiler.executor.taskSpan.enabled\ configuration (since 5.0.0) allows recording task attempt spans in the JFR output to associate samples with specific task stages and partitions.
connector/profiler · high confidence
New Spark Plugin API for driver and executor components
Spark introduces a new plugin architecture allowing developers to create dynamic plugins with both driver-side and executor-side components. The \SparkPlugin\ interface serves as the entry point, returning instances of \DriverPlugin\ and \ExecutorPlugin\. \DriverPlugin\ provides lifecycle hooks for initialization, metrics registration, RPC message handling, and shutdown. \ExecutorPlugin\ includes initialization and shutdown hooks, plus new task-level callbacks (\onTaskStart\, \onTaskSucceeded\, \onTaskFailed\) available since version 3.1.0, enabling plugins to react to task execution events. \PluginContext\ provides plugins with access to configuration, metrics registry, executor/driver ID, hostname, and allocated resources, as well as methods to send messages or make RPC calls to the driver component.
core/src/main/java/org/apache/spark/api/plugin · high confidence
New asynchronous read-ahead input stream for improved I/O performance
Spark introduces a new \ReadAheadInputStream\ class that asynchronously prefetches data from the underlying input stream into a separate buffer while the current buffer is being consumed. This design amortizes disk I/O costs by overlapping read operations with data processing, resulting in smoother and faster hot reading paths for applications relying on sequential input streams.
core/src/main/java/org/apache/spark/io · high confidence
New build helper scripts for Maven and SBT
The build directory now includes \build/mvn\ and \build/sbt\ wrappers that automatically download and manage the required versions of Maven and SBT if they are not already present in the system path. These scripts also handle environment setup, such as detecting \JAVA\_HOME\ on macOS, configuring JVM memory and stack size defaults, and respecting \MAVEN\_MIRROR\_URL\ for internal repository mirrors. Additionally, \build/spark-build-info\ and \build/spark-build-info.ps1\ are introduced to generate the \spark-version-info.properties\ file, and \build/util.sh\ provides shared utility functions like \realpath\.
build · high confidence
New catalyst expression and catalog connector classes
The spark-catalyst module adds several new classes to support advanced SQL features. In expressions, it introduces \ArrayOfCollatedStringsSerDe\, \CollatedString\, and \XXH64\ to enable correct serialization and hashing for collated strings in the \approx\_top\_k\ function, alongside \FixedLengthRowBasedKeyValueBatch\ and \VariableLengthRowBasedKeyValueBatch\ to optimize row-based key-value processing. In the connector catalog package, it adds \ChangelogRange\ (with \VersionRange\, \TimestampRange\, and \UnboundedRange\ records) to define CDC query ranges, \DelegatingTable\ to adapt table metadata for Spark-managed data sources, \DependencyList\, \FunctionDependency\, and \TableDependency\ to track SQL object dependencies, and updates \NamespaceChange\ and \TableChange\ interfaces to support property alterations and column modifications.
_spark-catalyst\2.13 · high confidence
New common utilities: SparkThrowable interface and StorageLevelMapper
This change introduces two new components to the common utils module. First, it adds the \SparkThrowable\ interface, which standardizes error handling by providing methods to retrieve error conditions, SQL states, breaking change information, and message templates, while deprecating the older \getErrorClass\ method in favor of \getCondition\. Second, it adds the \StorageLevelMapper\ enum, which allows users to easily retrieve \StorageLevel\ instances by their string names via the \fromString\ method.
common/utils/src/main/java · high confidence
New common-utils Java module with core utility interfaces and classes
The common/utils-java location now contains a new module that consolidates core Java utility code previously scattered across the codebase. This includes the Java API function interfaces (such as MapFunction, FlatMapFunction, FilterFunction, and PairFunction) used in the Java Dataset and RDD APIs, the QueryContext and QueryContextType interfaces for providing error context in query execution, and the structured logging infrastructure (LogKey, LogKeys, MDC, SparkLoggerFactory). Additionally, it introduces the ByteUnit enum for safe byte-size conversions with overflow protection, the MemoryMode enum, and general-purpose utilities in JavaUtils for file operations and other common tasks.
common/utils-java · high confidence
New configuration templates for logging, metrics, and scheduling
Added new template files to the conf directory to provide out-of-the-box configuration examples for key subsystems. This includes log4j2 templates (log4j2.properties.template and log4j2-json-layout.properties.template) that configure default logging levels and disable verbose stacktrace formatting for performance, a metrics configuration template (metrics.properties.template) documenting available sinks like Console, Graphite, StatsD, and Prometheus, a fair scheduler allocation template (fairscheduler.xml.template) with example pool definitions, and updated templates for spark-defaults, spark-env, and workers to reflect current deployment options.
conf · high confidence
New delegation token providers and credential managers for HBase, HDFS, and OIDC
Spark introduces dedicated delegation token providers for HBase and Hadoop FileSystems, alongside a new UserCredentialManager for OIDC-based credential propagation. The HBase provider now supports both legacy and HBase 2.x+ APIs for token acquisition, while the HadoopFS provider handles token renewal intervals and exclusion lists. The UserCredentialManager independently manages OIDC identity tokens and service credentials, allowing clusters to access services secured by OIDC (e.g., S3) alongside Kerberos-secured services (e.g., HDFS) simultaneously.
core/src/main/scala/org/apache/spark/deploy/security · high confidence
New deployment scripts for Spark Connect, decommissioning, and daemon management
This change introduces several new shell scripts in the sbin directory to manage Spark services. It adds start-connect-server.sh and stop-connect-server.sh to launch and terminate the Spark Connect server, and decommission-worker.sh to gracefully remove workers from the cluster. The core spark-daemon.sh script is updated to support a new 'decommission' command and a 'status' command for checking daemon states. Additionally, spark-config.sh is introduced to centralize configuration loading, and spark-daemons.sh is renamed from the legacy hadoop-daemons.sh to align with Spark's deployment model.
sbin · high confidence
New file input formats for binary and whole-text files
Added \FixedLengthBinaryInputFormat\ and \WholeTextFileInputFormat\ (along with their record readers) to the \org.apache.spark.input\ package. The binary input format enables reading flat binary files where records have a fixed size, ensuring splits align with record boundaries. The whole-text input format allows reading entire text files as key-value pairs (path, content) using Hadoop's CombineFileInputFormat, supporting small-file scenarios with configurable partitioning.
core/src/main/scala/org/apache/spark/input · high confidence
New forward-secure authentication protocol for network encryption
Spark introduces a new transport-layer authentication protocol (v2.0) that uses X25519 Diffie-Hellman key exchange and AES-GCM encryption, replacing the legacy SASL-based auth path when encryption is enabled. This change adds AuthClientBootstrap, AuthServerBootstrap, AuthRpcHandler, and AuthEngine to handle the challenge-response handshake, derive session keys via HKDF, and attach the resulting cipher to the Netty channel. The implementation supports backward compatibility by falling back to SASL if the new protocol fails or is disabled, and it sets the client ID on the TransportClient after successful authentication. Note that v1.0 and v2.0 are cryptographically incompatible for encrypted RPCs, so v1.0 remains the default to preserve backward compatibility.
common/network-common/src/main/java/org/apache/spark/network/crypto · high confidence
New hadoop-cloud module for cloud storage integration components
This change introduces the \hadoop-cloud\ module, which consolidates Spark's integration with cloud storage systems. It adds the \AbortableStreamBasedCheckpointFileManager\ to support abortable streams for streaming checkpoints, the \BindingParquetOutputCommitter\ to dynamically bind Parquet writes to the configured output committer, and the \PathOutputCommitProtocol\ to manage commit protocols using Hadoop's \PathOutputCommitterFactory\. The module also includes a standard Maven project structure, a README for running integration tests against external systems like AWS S3, and a suite of tests to verify committer binding, serialization, and dynamic partition overwrite support.
hadoop-cloud · high confidence
New launch scripts for Beeline, Spark Connect Shell, and Pipelines
The bin directory now includes new entry-point scripts: beeline and beeline.cmd for running the Hive JDBC shell, spark-connect-shell for launching the Spark shell with the Spark Connect plugin enabled, and spark-pipelines for executing the Declarative Pipelines CLI. These scripts handle environment discovery and delegate to the core launchers, extending the existing set of user-facing tools.
bin · high confidence
New launcher infrastructure for in-process and remote Spark app communication
This change introduces new internal components in the \launcher\ package to support programmatic launching and communication with Spark applications. \LauncherBackend\ provides a client-side abstraction for connecting to a launcher server via a socket, handling state synchronization (such as app ID and execution state) and graceful stop requests. \SparkSubmitArgumentsParser\ exposes the argument parsing logic to other Spark code while keeping the underlying \SparkSubmitOptionParser\ package-private. \WorkerCommandBuilder\ constructs the command line for worker processes, specifically setting memory limits and environment variables. These additions form the foundation for the in-process launcher and remote launcher backend features.
core/src/main/scala/org/apache/spark/launcher · high confidence
New local linear algebra primitives and vectorized BLAS implementation
The \mllib-local\ module now includes the core local linear algebra types—\Vector\, \Matrix\, \DenseVector\, \DenseMatrix\, \SparseVector\, and \SparseMatrix\—along with a new \BLAS\ object that provides vectorized, native-accelerated implementations for linear algebra routines (such as \axpy\, \dot\, \gemv\, and \gemm\) using the \dev.ludovic.netlib\ library. This change also introduces utility functions for symmetric matrix handling, numerical stability (log1pExp), and softmax operations, establishing the foundational local math layer for MLlib algorithms.
mllib-local · high confidence
New metrics sinks and servlets for console, CSV, JMX, Slf4j, StatsD, and Prometheus
This change introduces a suite of new metrics sinks and servlets in the \org.apache.spark.metrics.sink\ package, enabling users to export Spark metrics to various destinations. New capabilities include \ConsoleSink\ and \CsvSink\ for local debugging and file-based storage, \JmxSink\ for JMX exposure, \Slf4jSink\ for logging metrics to the application log, and \StatsdSink\ (backed by \StatsdReporter\) for sending metrics to StatsD-compatible collectors with optional regex-based filtering. Additionally, \PrometheusServlet\ and \MetricsServlet\ provide HTTP endpoints to expose metrics in Prometheus text format and JSON format respectively, with the Prometheus output aligned to DropWizard 4.1 standards.
core/src/main/scala/org/apache/spark/metrics/sink · high confidence
New metrics sources for accumulators, JVM resources, and code generation
This change introduces several new metric sources in the Spark metrics system. AccumulatorSource now exposes gauge metrics for the current values of LongAccumulator and DoubleAccumulator instances, allowing users to monitor accumulator state. JVM monitoring is enhanced with JVMCPUSource, which reports JVM process CPU time in nanoseconds, and JvmSource, which exposes garbage collector, memory usage, and java.nio buffer pool metrics. Additionally, StaticSources provides new metrics for code generation (CodegenMetrics), tracking source code size, compilation time, and generated bytecode sizes for classes and methods, as well as metrics for Hive catalog operations (HiveCatalogMetrics) such as partition fetches, file discoveries, cache hits, and client calls.
core/src/main/scala/org/apache/spark/metrics/source · high confidence
New private Shuffle API interfaces for pluggable storage
Spark 3.0 introduces a new private API in the \org.apache.spark.shuffle.api\ package that enables pluggable modules for storing and reading temporary shuffle data. The diff adds interfaces such as \ShuffleDataIO\, \ShuffleDriverComponents\, and \ShuffleExecutorComponents\ to bootstrap shuffle support on the driver and executor sides, alongside \ShuffleMapOutputWriter\ and \ShufflePartitionWriter\ to manage the persistence of map task output. This change also includes \MapOutputCommitMessage\ and \MapOutputMetadata\ to return custom metadata and partition lengths from writers, allowing for more flexible shuffle storage backends beyond the default local disk implementation.
core/src/main/java/org/apache/spark/shuffle/api · high confidence
New shuffle I/O API, credential types, and NIO-based file input stream
Spark introduces a new shuffle driver/executor component API (ShuffleDataIO, ShuffleDriverComponents, ShuffleExecutorComponents) with a LocalDiskShuffleDataIO implementation that replicates the legacy local shuffle storage and index file behavior. A new UserCredentials class bundles service credentials by scheme for transmission to executors, and NioBufferedFileInputStream replaces BufferedInputStream for file reads by using direct buffers and Java's Cleaner for resource management. Additionally, TimeTrackingOutputStream is added to intercept write calls and track time for shuffle write metrics.
_spark-core\2.13 · high confidence
New structured logging infrastructure for driver and executor logs
This change introduces new Scala classes in the logging package to modernize how Spark handles log persistence and rotation. DriverLogger now manages a local log appender and asynchronously syncs driver logs to HDFS, ensuring logs are available in the UI even in client mode. For executors, FileAppender, RollingFileAppender, and associated policies (TimeBasedRollingPolicy, SizeBasedRollingPolicy) replace previous implementations, providing configurable time-based or size-based log rolling with optional GZIP compression and automatic cleanup of old files. These components use the new structured logging framework (MDC) for better log context and error handling.
core/src/main/scala/org/apache/spark/util/logging · high confidence
New tool to generate MIMA exclusions for package-private symbols
A new Scala tool, GenerateMIMAIgnore, has been added to the Spark tools module to help maintain binary compatibility checks. This utility scans the Spark codebase to identify package-private classes and members, generating exclusion lists for the MiMa binary compatibility checker. The tool specifically handles Scala reflection nuances, such as detecting package-private modules and inner functions, and includes heuristics to filter out JVM-generated classes (like anonymous classes or Hive-related wrappers) to reduce false positives in compatibility reports.
tools · high confidence
New unsafe memory allocation API with debug fill and double-free detection
The \common/unsafe/memory\ module now exposes a new memory management API consisting of \MemoryAllocator\, \MemoryBlock\, \MemoryLocation\, and \UnsafeMemoryAllocator\. This change introduces a \MEMORY\_DEBUG\_FILL\ feature (controlled by the \spark.memory.debugFill\ system property) that fills newly allocated memory with \0xa5\ and freed memory with \0x5a\ to help catch use of uninitialized or freed memory. Additionally, \UnsafeMemoryAllocator\ now detects double-frees and misuse of TaskMemoryManager-allocated pages by tracking page numbers and resetting pointers upon deallocation.
common/unsafe/src/main/java/org/apache/spark/unsafe/memory · high confidence
New utility classes for lazy initialization and class loading
Added several new utility classes to the core Spark utilities package: BestEffortLazyVal and TransientBestEffortLazyVal provide lock-free, lazily-initialized variables that help avoid deadlocks during concurrent initialization; ChildFirstURLClassLoader, MutableURLClassLoader, and ParentClassLoader introduce flexible, parallel-capable class loading mechanisms with child-first resolution and dynamic URL management; EnumUtil offers case-insensitive enum parsing; and ExposedBufferByteArrayOutputStream exposes the internal byte buffer for performance-sensitive serialization scenarios.
core/src/main/java/org/apache/spark/util · high confidence
Protobuf connector introduces configurable deserialization options
The Protobuf connector now supports several new configuration options that change how Protobuf data is read into Spark SQL. Users can enable \convert.any.fields.to.json\ to convert \google.protobuf.Any\ fields into readable JSON strings instead of binary blobs, and \emit.default.values\ to include zero-value fields in the output schema rather than omitting them. Additionally, \enums.as.ints\ allows enum fields to be returned as integers instead of strings, \upcast.unsigned.ints\ maps unsigned 32/64-bit integers to Long/Decimal types for better precision, and \unwrap.well.known.types\ simplifies schemas by unwrapping standard wrapper types (like \StringValue\) into their primitive equivalents. The connector also adds \recursive.fields.max.depth\ to control the depth of nested recursive messages and \retain.empty.message\ to preserve empty messages as structs with a dummy field.
connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils · high confidence
Protobuf serialization for Spark UI status data
The Spark History Server now uses Protobuf for serializing UI status data stored in the KV store, replacing the previous JSON-based approach. This change introduces a new \ProtobufSerDe\ trait and a \KVStoreProtobufSerializer\ that leverages Java's ServiceLoader to automatically discover and register serializers for various UI data types, including application info, executor summaries, job data, and stage details. This shift aims to improve storage efficiency and serialization performance for the UI backend.
core/src/main/scala/org/apache/spark/status/protobuf · high confidence
Refactor core utility classes and introduce new low-level I/O components
This change refactors several core utility classes in \org.apache.spark.util\ and introduces new low-level I/O components. Specifically, it adds \ByteBufferInputStream\ and \ByteBufferOutputStream\ to provide zero-copy conversion between \ByteArrayOutputStream\ and \ByteBuffer\, and \DirectByteBufferOutputStream\ for efficient direct buffer management. It also introduces \BoundedPriorityQueue\ to wrap Java's \PriorityQueue\ with a size limit, \CausedBy\ for easier exception root-cause extraction, and \CollectionsUtils\ for optimized binary search on primitive arrays. Additionally, \Clock\ is refactored to abstract time sources for better testability, and \CommandLineUtils\ is updated to standardize CLI exit behavior.
core/src/main/scala/org/apache/spark/util · high confidence
Shuffle data corruption diagnosis with CRC32C support
The shuffle checksum module now supports the CRC32C algorithm alongside existing ADLER32 and CRC32 options, and includes a new diagnostic capability that identifies the root cause of shuffle data corruption. Users can now distinguish between disk issues, network issues, or verification passes when corruption is detected, with detailed logging provided via the structured logging framework to aid in troubleshooting.
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/checksum · high confidence
Shuffle writers now compute and expose checksums for data integrity verification
Sort-based shuffle writers (BypassMergeSortShuffleWriter, ShuffleExternalSorter, and UnsafeShuffleWriter) now calculate checksums for shuffle output data. This includes partition-level checksums for standard verification and row-based checksums that are order-independent, allowing Spark to detect if different task attempts produce different data for the same partition. These checksums are integrated into the MapStatus metadata, enabling downstream consumers to verify data integrity during shuffle reads.
core/src/main/java/org/apache/spark/shuffle/sort · high confidence
Spark Connect server is now loadable as a DriverPlugin
The Spark Connect server is now packaged as a \DriverPlugin\ (\SparkConnectPlugin\), allowing it to be enabled by simply adding the JAR to the classpath and configuring \spark.plugins\. This decouples the Connect build from the core Spark distribution. The server lifecycle is managed by the plugin's \init\ and \shutdown\ methods, which start and stop the \SparkConnectService\. A new \SimpleSparkConnectService\ main class is provided for testing, and a set of new plugin interfaces (\CommandPlugin\, \ExpressionPlugin\, \RelationPlugin\, \GetStatusPlugin\, \MLBackendPlugin\) are introduced to allow extensibility of the Connect planner and status reporting.
sql/connect/server · high confidence
SparkR 5.0.0 initial package structure and build tooling
This change introduces the foundational structure for the SparkR 5.0.0 package. It adds the R package \DESCRIPTION\ file, which sets the version to 5.0.0, requires R \>= 4.0, and specifies Java 17–25 as a system requirement. The commit also adds essential build and development scripts (\install-dev.sh\, \check-cran.sh\, \create-docs.sh\, etc.) for installing the package, generating documentation, and running CRAN checks, alongside configuration files like \.lintr\ for code linting and \.gitignore\ files to manage build artifacts.
R · high confidence
Standalone Master Web UI rewritten with new pages and security controls
The Spark Standalone Master Web UI has been completely rewritten, introducing new pages for Application details, Environment inspection, and Log viewing, while updating the Master index page to show driver states (including SUBMITTED and RELAUNCHING) and application duration. The UI now supports holding and resuming applications, enforces modify ACLs on kill and decommission endpoints, and restricts the /workers/kill endpoint to POST requests only when decommissioning is enabled. The Environment page allows filtering environment variables by configurable prefixes, and the Application page displays executor limits, resource profiles, and links to application or history UIs when available.
core/src/main/scala/org/apache/spark/deploy/master/ui · high confidence
StandaloneAppClient now supports application hold/resume and executor decommissioning callbacks
The StandaloneAppClient and its listener interface have been updated to support new cluster management capabilities. The listener now includes callbacks for executor decommissioning (executorDecommissioned) and for holding or resuming the application (holdApplication), allowing the client to react to master-initiated state changes. Additionally, the client implementation now reports its hold status to the Master upon successful registration, enabling features like draining executors and managing application lifecycle via the Master UI.
core/src/main/scala/org/apache/spark/deploy/client · high confidence
Support Volcano PodGroup configuration via JSON template for Spark drivers
Users can now configure Volcano PodGroup settings for Spark driver pods using the new \spark.kubernetes.scheduler.volcano.podGroupTemplateJson\ configuration property, which accepts a JSON string defining attributes like minimum members, resource requests, priority class, and queue. This feature allows for fine-grained scheduling control via the Volcano scheduler without requiring external template files, while executor pods remain unaffected by PodGroup resource creation.
resource-managers/kubernetes/core/volcano · high confidence
Support for fractional CPU resources in stage-level scheduling
Spark now allows specifying fractional CPU amounts (e.g., 0.2 cores) for tasks via the \spark.task.cpus\ configuration and the \TaskResourceRequests.cpus()\ API. To prevent precision loss inherent in floating-point arithmetic, the \core/src/main/scala/org/apache/spark/resource\ package introduces \CpuAmount\, which uses a fixed-scale \BigDecimal\ (9 decimal places) for all internal CPU accounting. This ensures that fractional CPU allocations are handled accurately during scheduling and resource allocation, enabling more granular resource utilization across stages.
core/src/main/scala/org/apache/spark/resource · high confidence
Support for handling data blocks larger than 2GB via chunked buffers
Spark now introduces \ChunkedByteBuffer\, \ChunkedByteBufferOutputStream\, and \ChunkedByteBufferFileRegion\ to manage byte buffers split into multiple chunks. This change enables the storage, serialization, and network transfer of data exceeding the 2GB limit (such as large task results or remote cached partitions) by avoiding contiguous array allocations and leveraging Netty's FileRegion for efficient, zero-copy transfers.
core/src/main/scala/org/apache/spark/util/io · high confidence
Removals
Removal of ASM 3.2 Maven POMs and Hadoop C++ libtool metadata
This change removes the Maven POM files for the ASM 3.2 library (including core, tree, analysis, commons, util, xml, and the all-in-one jar) and the libtool archive metadata for the Hadoop C++ HDFS client library (libhdfs). These files are no longer included in the distribution, which may affect build systems that relied on these specific third-party metadata files for dependency resolution or linking.
_third\_party/asm-3.2, third\_party/hadoop-0.20.0/c++/Linux-amd64-64, third\party/hadoop-0.20.0/c++/Linux-i386-32 · high confidence
Removal of example applications and core Spark runtime components
This change removes a large set of example applications (including CpuHog, LocalALS, SparkALS, SparkLR, and SparkPi) and core Spark runtime source files (such as SparkContext, Executor, NexusScheduler, LocalScheduler, Accumulators, and HdfsFile) from the src/scala directory. For users, this means the provided runnable examples and the underlying execution engine code are no longer present in this location, effectively stripping the project of its immediate usability and core functionality in this module.
src/scala · high confidence
Removal of legacy Hadoop third-party artifacts
The Hadoop 0.20.0 third-party library bundle has been cleaned up by removing the Hypersonic SQL (HSQLDB) license file and a series of obsolete JDiff API definition XML files for Hadoop versions 0.17.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, and 0.19.1. This reduces the footprint of the bundled dependencies by discarding documentation and license metadata for versions no longer actively supported.
_third\party/hadoop-0.20.0 · high confidence
Removed bundled Hadoop 0.20.0 third-party dependency
The bundled Hadoop 0.20.0 distribution has been removed from the third\_party directory. This deletion includes the core binaries, configuration scripts, build files, and C++ Pipes interface headers, effectively stripping the project of this specific Hadoop version.
_third\party · high confidence
Architecture
Centralized configuration definitions for Spark subsystems
Spark now defines its configuration entries in the \org.apache.spark.internal.config\ package (e.g., \Deploy.scala\, \History.scala\, \Kryo.scala\, \Network.scala\, \Python.scala\, \R.scala\, \Status.scala\, \Streaming.scala\, \Tests.scala\, \UI.scala\, \Worker.scala\) using the \ConfigBuilder\ framework. This change provides a single, type-safe source of truth for configuration keys, default values, versioning, and validation rules, replacing scattered hardcoded strings and improving consistency across the platform.
core/src/main/scala/org/apache/spark/internal/config · high confidence
Consolidate core utility classes into the common/utils module
A large set of internal utility classes—including ClosureCleaner, MavenUtils, SparkFileUtils, SparkErrorUtils, SparkCollectionUtils, and others—have been moved into the common/utils module. This refactoring centralizes shared infrastructure, making these utilities available to other Spark components (such as Spark Connect and SQL) without requiring dependencies on the core module, and ensures consistent behavior for file operations, dependency resolution, and error handling across the platform.
common/utils/src/main/scala/org/apache/spark/util · high confidence
Move Hadoop write commit protocol and writer to internal.io package
The Hadoop write commit protocol implementation and the SparkHadoopWriter helper have been relocated from the sql/core module into the core module's internal.io package. This change consolidates the internal APIs for saving RDDs using Hadoop OutputFormats (supporting both the legacy mapred and newer mapreduce APIs) into a single, shared location, making the commit protocol and writer utilities available to other core components without depending on the SQL module.
core/src/main/scala/org/apache/spark/internal/io · high confidence
Spark Connect protocol definitions and code generation moved to sql/connect/common
The Spark Connect protocol definition files (\.proto\) and the \buf\ code-generation configuration (\buf.gen.yaml\, \buf.yaml\) have been relocated to the \sql/connect/common\ module. This change centralizes the shared protocol schema and build tooling, establishing a single source of truth for the Spark Connect interface that is used to generate client and server stubs for multiple languages (Java, Python, C++, C\#, Ruby, Go).
sql/connect/common · high confidence
Behavioural changes
12104 commits (1432 fixes) modifying repository
A change to existing behaviour in repository — 12104 commits (1432 fixs), 626 files: “Revert "\[SPARK-24250\]\[SQL\] support accessing SQLConf inside tasks"”; “\[SPARK-60423\]\[SQL\] Fix Hive ORC filter pushdown on timestamp literals with sub-millisecond digits”; “\[SPARK-42845\]\[SQL\] Update the error class \_LEGACY\_ERROR\_TEMP\_2010 to InternalError”; and 12018 more (refs \#450, \#6457, \#8905, \#12178, \#55768, …).
repository · medium confidence · unverified
12905 commits (1741 fixes) modifying sql/core
A change to existing behaviour in sql/core — 12905 commits (1741 fixs), 6598 files: “\[SPARK-58232\]\[SQL\] Simplify cached-batch column-index resolution with a map lookup”; “Revert "\[SPARK-24250\]\[SQL\] support accessing SQLConf inside tasks"”; “\[SPARK-42845\]\[SQL\] Update the error class \_LEGACY\_ERROR\_TEMP\_2010 to InternalError”; and 12820 more (refs \#12178, \#55768, \#978, \#2, \#8494, …).
sql/core · medium confidence · unverified
1651 commits (187 fixes) modifying project
A change to existing behaviour in project — 1651 commits (187 fixs), 5 files: “\[SPARK-25765\]\[ML\] Add training cost to BisectingKMeans summary”; “\[SPARK-25908\]\[CORE\]\[SQL\] Remove old deprecated items in Spark 3”; “\[SPARK-35488\]\[BUILD\] Upgrade ASM to 7.3.1”; and 1634 more (refs \#596, \#12178, \#2, \#31, \#69, …).
project · medium confidence · unverified
1972 commits (214 fixes) modifying dev
A change to existing behaviour in dev — 1972 commits (214 fixs), 83 files: “\[SPARK-44813\]\[INFRA\] The Jira Python misses our assignee when it searches users again”; “\[SPARK-8495\] \[SPARKR\] Add a \.lintr\ file to validate the SparkR files and the \lint-r\ script”; “\[SPARK-16484\]\[SQL\] Add support for Datasketches HllSketch”; and 1946 more (refs \#6967, \#592, \#569, \#565, \#13286, …).
dev · medium confidence · unverified
2393 commits (290 fixes) modifying mllib
A change to existing behaviour in mllib — 2393 commits (290 fixs), 685 files: “\[SPARK-25765\]\[ML\] Add training cost to BisectingKMeans summary”; “\[SPARK-25908\]\[CORE\]\[SQL\] Remove old deprecated items in Spark 3”; “\[SPARK-30818\]\[SPARKR\]\[ML\] Add SparkR LinearRegression wrapper”; and 2380 more (refs \#566, \#528, \#12625, \#8271, \#30327, …).
mllib · medium confidence · unverified
2728 commits (323 fixes) modifying sql/hive
A change to existing behaviour in sql/hive — 2728 commits (323 fixs), 9483 files: “\[SPARK-60423\]\[SQL\] Fix Hive ORC filter pushdown on timestamp literals with sub-millisecond digits”; “\[SPARK-29968\]\[SQL\] Remove the Predicate code from SparkPlan”; “\[SPARK-20202\]\[BUILD\]\[SQL\] Remove references to org.spark-project.hive (Hive 1.2.1)”; and 2703 more (refs \#12178, \#2, \#1071, \#17311, \#6882, …).
sql/hive · medium confidence · unverified
6870 commits (850 fixes) modifying python
A change to existing behaviour in python — 6870 commits (850 fixs), 1478 files: “\[SPARK-38400\]\[PYTHON\] Enable Series.rename to change index labels”; “\[SPARK-25765\]\[ML\] Add training cost to BisectingKMeans summary”; “\[SPARK-43671\]\[PS\]\[FOLLOWUP\] Refine \CategoricalOps\ functions”; and 6823 more (refs \#9527, \#2, \#30327, \#462, \#542).
python · medium confidence · unverified
7866 commits (961 fixes) modifying sql/catalyst
A change to existing behaviour in sql/catalyst — 7866 commits (961 fixs), 1341 files: “\[SPARK-42746\]\[SQL\]\[FOLLOWUP\] Correct the comments for SupportsOrderingWithinGroup and Mode”; “\[SPARK-30554\]\[SQL\] Return \Iterable\ from \FailureSafeParser.rawParser\”; “Revert "\[SPARK-24250\]\[SQL\] support accessing SQLConf inside tasks"”; and 7819 more (refs \#55768, \#7920, \#2, \#7506, \#7456, …).
sql/catalyst · medium confidence · unverified
Avro connector migrated to the DataSource V2 API
The Avro data source implementation has been rewritten to use the DataSource V2 API, replacing the legacy V1 file source. This migration introduces new internal components such as AvroDataSourceV2, AvroTable, AvroScan, and AvroPartitionReaderFactory to handle reading and writing. For users, this change provides improved performance and feature parity with other V2 sources, including support for scan merging and more robust handling of Avro schemas and data types, while maintaining backward compatibility through the existing \avro\ data source short name.
connector/avro/src/main · high confidence
Centralized API stability annotations in spark-tags
Spark has moved API stability and developer annotations (such as @Stable, @Experimental, @Unstable, @Private, @DeveloperApi, @Since, and the new @ClassicOnly and @AlphaComponent) into the common/tags module. These annotations are now retained at runtime and are consistently applied across the project to mark API stability levels, ensuring that documentation tools and runtime checks can reliably identify which interfaces are stable, evolving, experimental, or restricted to specific implementations like the Classic SQL engine.
common/tags/src/main · high confidence
Centralized error handling and build info in common utils
The error framework and build information utilities have been moved to the common/utils module to improve code reuse and consistency. Error messages are now loaded from JSON definitions via ErrorClassesJsonReader, which supports structured error classes, SQLSTATEs, and breaking-change metadata, while SparkBuildInfo now loads version details from its own classloader to avoid classpath conflicts. Additionally, a new StringSubstitutor utility is available for safe template variable replacement in error messages.
common/utils/src/main/scala/org/apache/spark · high confidence
Centralized error handling with structured error classes
Spark now uses a centralized \SparkCoreErrors\ object to generate exceptions, replacing ad-hoc error messages with structured error classes (e.g., \CHECKPOINT\_RDD\_BLOCK\_ID\_NOT\_FOUND\, \UNSUPPORTED\_ARRAY\_KEY.MAP\_SIDE\_COMBINE\). This change standardizes error reporting by attaching specific error class identifiers and parameter maps to exceptions, enabling more consistent error handling and better diagnostic information for users.
core/src/main/scala/org/apache/spark/errors · high confidence
Default logging configuration for Kinesis connector updated
The Kinesis ASL connector now includes a default log4j2.properties file that sets the root logging level to warn and configures console and file appenders. This change aligns with the broader effort to remove Log4j2's extended throwable conversion pattern from default logging configurations, ensuring consistent and less verbose logging behavior for users of the connector.
connector/kinesis-asl/src/main/resources · high confidence
Environment page now uses client-side rendering with Bootstrap 5 tabs
The Environment page in the Spark Web UI has been migrated to a new client-side rendering architecture. Instead of server-side generated tables, the page now loads a Bootstrap 5 tabbed interface (Runtime Information, Spark Properties, Resource Profiles, Hadoop Properties, System Properties, Metrics Properties, and Classpath Entries) and fetches data dynamically via JavaScript modules. This change improves interactivity and allows for client-side filtering and sorting of configuration data, while also preparing the UI for better handling of sensitive information redaction.
core/src/main/scala/org/apache/spark/ui/env · high confidence
Ganglia metrics reporter now uses structured logging
The GangliaReporter in the connector module has been updated to use Spark's structured logging framework (SparkLogger and LogKeys) instead of standard Java logging. This change aligns the reporter's log output with Spark's internal logging conventions, ensuring consistent formatting and context (MDC) for metrics-related events.
_spark-ganglia-lgpl\2.13 · high confidence
Introduce UnsafeAlignedOffset and update Platform for modern JDKs
The unsafe module now includes a new UnsafeAlignedOffset utility that dynamically selects between 4-byte and 8-byte record length offsets based on the platform's unaligned-access capability, ensuring correct memory alignment on architectures like SPARC. The Platform class has been updated to support Java 26 by switching from jdk.internal.ref.Cleaner to java.nio.BufferCleaner, and it now uses trySetAccessible for safer reflection on DirectByteBuffer constructors and cleaner fields. Additionally, new utility classes KVIterator and UTF8StringBuilder have been added to support key-value iteration and efficient UTF-8 string building using off-heap memory operations.
common/unsafe/src/main/java/org/apache/spark/unsafe · high confidence
Introduce new standard RPC interface and Netty-based implementation
Spark replaces the previous Akka-based RPC system with a new, standard RPC interface built on Netty. This change introduces core abstractions such as \RpcEnv\, \RpcEndpoint\, and \RpcEndpointRef\, along with new configuration properties under \spark.rpc.\\ (deprecating the old \spark.akka.\\ settings). Users benefit from a more robust and configurable communication layer, including support for dedicated thread pools for specific endpoints, improved timeout handling with descriptive error messages, and better integration with Spark's internal network module.
core/src/main/scala/org/apache/spark/rpc · high confidence
Introduce pipelined shuffle dependency tracking in ActiveJob
The scheduler now tracks whether a job contains a \PipelinedShuffleDependency\ via a new \hasPipelinedDependency\ flag on \ActiveJob\. This flag allows the scheduler to short-circuit group-membership graph walks for regular jobs, optimizing the scheduling path for pipelined shuffle stages.
core/src/main/scala/org/apache/spark/scheduler · high confidence
Introduce unified block transfer listener interfaces and app recovery control
The external shuffle service now uses a unified \BlockTransferListener\ interface to handle both block fetching and pushing, with specific \BlockFetchingListener\ and \BlockPushingListener\ implementations delegating to it for code reuse. Additionally, a new \AppsWithRecoveryDisabled\ singleton allows the service to selectively disable state recovery for specific applications, and \ErrorHandler\ implementations now prevent retries for \FileNotFoundException\ and stale shuffle finalize requests to improve failure handling.
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle · high confidence
Introduction of Torrent-based broadcast implementation
Spark now uses a BitTorrent-like protocol for distributing broadcast variables, replacing the previous HTTP-based approach. This change introduces a new \TorrentBroadcast\ implementation that splits serialized data into chunks and distributes them via the BlockManager, allowing executors to fetch pieces from the driver and other peers to reduce driver bottleneck. The \BroadcastManager\ now defaults to \TorrentBroadcastFactory\, and the system supports configuration for compression, block size, and checksums to ensure data integrity and efficiency.
core/src/main/scala/org/apache/spark/broadcast · high confidence
Kinesis connector refactored to use structured logging and updated KCL dependencies
The Kinesis connector has been updated to use the new structured logging framework, replacing legacy string interpolation with MDC-based logging for better log parsing and correlation. Additionally, the connector now supports configuring CloudWatch metrics levels via the builder API, allowing users to control the verbosity of metrics sent to CloudWatch. The underlying Kinesis Client Library (KCL) has been upgraded to version 2.7.2, and the dependency on AWS SDK for Java 1.x has been removed in favor of the AWS SDK for Java 2.x, improving compatibility and security.
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming · high confidence
Kinesis connector upgraded to AWS SDK v2 and KCL 2.7.2
The Kinesis connector now uses the AWS SDK for Java v2 and the Kinesis Client Library (KCL) version 2.7.2, replacing the previous AWS SDK v1 dependency. This change updates the underlying client implementation (e.g., using \software.amazon.awssdk.services.kinesis.KinesisClient\) and introduces new Java wrapper classes like \KinesisInitialPositions\ to expose KCL initial position options (Latest, TrimHorizon, AtTimestamp) to Spark Streaming users.
connector/kinesis-asl/src/main/java · high confidence
LZF compression now uses pure Java instead of native libraries
The native LZF compression implementation has been removed from the Spark Java library. The previous code relied on a native library (spark\_native) loaded via JNI for compressing and decompressing data, which required external dependencies and platform-specific binaries. This change eliminates that native dependency, simplifying deployment and improving portability by relying solely on Java code for LZF stream handling.
src/java · high confidence
Maven build configuration updates for JVM and parallelism
The Maven build now uses a new \.mvn/jvm.config\ file to configure the JVM, enabling diagnostic options, increasing the GCLocker retry allocation count to 100, and enabling native access for all unnamed modules to suppress warnings. Additionally, a new \.mvn/maven.config\ file disables parallel PUT operations and enables compatibility mode for the OS detector plugin.
.mvn · high confidence
Metrics system configuration via SparkConf and structured logging
The metrics system can now be configured directly through SparkConf properties (prefixed with spark.metrics.conf.), allowing users to set metrics options without relying solely on external property files. Additionally, internal logging within the metrics subsystem has been migrated to use the structured logging framework, improving log consistency and parseability.
core/src/main/scala/org/apache/spark/metrics · high confidence
Migrate YARN web proxy filter to Jakarta Servlet API
The YARN ApplicationMaster web proxy filter has been replaced with a forked implementation that uses the \jakarta.servlet\ API instead of the deprecated \javax.servlet\ API. This change removes the dependency on the external Hadoop \AmIpFilter\ class by copying and adapting the necessary proxy logic (including \AmIpPrincipal\ and \ProxyUtils\) into the Spark codebase, ensuring compatibility with newer servlet containers and Jetty versions.
resource-managers/yarn · high confidence
New Avro metadata injection and Parquet vectorized reader refactoring
The Avro data source now supports writing custom file metadata by introducing SparkAvroKeyOutputFormat and SparkAvroKeyRecordWriter, which wrap the standard Avro writers to inject key-value pairs into the output file footer. In the Parquet data source, the vectorized reader has been refactored to improve performance and maintainability: a new ParquetReadState class centralizes row-range and batch tracking logic, a new OpenedParquetFooter record simplifies footer management, and a comprehensive ParquetVectorUpdaterFactory replaces the previous updater selection logic to support a wider range of type conversions and bulk-read paths.
_spark-sql\2.13 · high confidence
New internal configuration binding policy for SQL views and UDFs
This change introduces a new \ConfigBindingPolicy\ enum and associated internal configuration infrastructure (including \ConfigBuilder\, \ConfigEntry\, and \ConfigReader\) to control how configuration values propagate to SQL views, UDFs, and procedures. Users can now define whether a configuration should use the active session value (\SESSION\), persist the value from creation time (\PERSISTED\), or remain not applicable (\NOT\_APPLICABLE\) within these SQL objects, ensuring consistent query behavior and preventing unintended side effects from session-level config changes.
common/utils/src/main/scala/org/apache/spark/internal/config · high confidence
New unified Spark-on-Kubernetes Docker images and entrypoints
The Spark-on-Kubernetes Dockerfiles have been consolidated into a single base image (defaulting to Eclipse Temurin Java 25 JRE on Ubuntu) with separate binding images for Python and R. The new entrypoint.sh script now supports arbitrary UIDs, passes executor library paths via SPARK\_EXECUTOR\_EXTRA\_LIBRARY\_PATH, includes the current working directory in the classpath, and injects the SPARK\_DRIVER\_POD\_IP environment variable into executor pods. A new decom.sh script enables graceful decommissioning by signaling the Java executor process, and the images include standard utilities like logrotate, net-tools, and jemalloc support.
resource-managers/kubernetes/docker · high confidence
Port Jobs and Stages tabs to the new UI backend
The Jobs and Stages tabs in the Spark Web UI have been rewritten to use the new \AppStatusStore\-based backend, replacing the legacy \JobProgressListener\ architecture. This change introduces new Scala UI page classes (e.g., \AllJobsPage\, \JobPage\, \AllStagesPage\, \StagePage\) that render job and stage lists, timelines, and details using the modern API. The implementation includes support for event timelines with configurable limits, fair scheduler pool views, task thread dumps (when enabled), and consistent handling of stage/job statuses, submission times, and durations. It also integrates with the new pagination and sorting frameworks for tables and ensures proper escaping and security checks for UI content.
core/src/main/scala/org/apache/spark/ui/jobs · high confidence
Port the Storage UI tab to the new UI backend
The Storage tab in the Spark Web UI has been ported to the new UI backend, introducing new Scala components (StorageTab, StoragePage, RDDPage) that render the list of persisted RDDs, their storage details, and data distribution across executors. This change updates how the Storage tab displays RDD information, including storage level, cached partitions, memory/disk usage, and executor addresses, while maintaining the existing functionality of the UI.
core/src/main/scala/org/apache/spark/ui/storage · high confidence
PySpark worker infrastructure and communication overhaul
This change introduces a comprehensive refactoring of the PySpark worker lifecycle and JVM-Python communication layer. It adds a new \CloseShieldOutputStream\ to prevent native buffer leaks during interrupted Arrow writes, and introduces \PythonWorkerHandle\ and \LocalPythonWorkerHandle\ to abstract worker process management, enabling faulthainer-based post-mortem diagnostics on worker crashes. The \Py4JServer\ is updated to support a pinned-thread mode via \ClientServer\ when \PYSPARK\_PIN\_THREAD\ is enabled, and \PythonWorkerFactory\ is enhanced to manage idle worker pools, enforce non-blocking channel modes, and support Unix Domain Sockets. Additionally, \PythonWorkerLogCapture\ is added to route Python UDF logs to per-worker rolling log writers, and \PythonWorkerUtils\ is expanded to serialize detailed \TaskContext\ information (including fractional CPU amounts and resources) to workers.
core/src/main/scala/org/apache/spark/api/python · high confidence
Refactor TransportContext to support SSL and separate chunk-fetch thread pools
The TransportContext class has been refactored to integrate SSL support via an SSLFactory and to allow separate thread pools for handling chunk fetch requests. This change introduces a new constructor parameter \isClientOnly\ to control whether a dedicated event loop group for shuffle chunk fetches is created, addressing performance and stability issues in the external shuffle server. Additionally, static initialization of worker event loops has been removed to fix test suite failures, and logging has been migrated to the structured SparkLogger framework.
common/network-common/src/main/java/org/apache/spark/network · high confidence
Refactored block metadata tracking and locking into BlockInfoManager
The block metadata tracking and locking logic has been extracted from BlockManager into a new, dedicated BlockInfoManager component. This change introduces fine-grained readers-writer locking per block, automatically associates locks with the running task, and ensures locks are released upon task completion or failure. It also adds support for tracking block checksums and sealed checksums for data integrity verification, and improves the handling of block visibility and removal operations.
core/src/main/scala/org/apache/spark/storage · high confidence
Refactored network protocol layer with new message types and zero-copy optimizations
The network protocol implementation in \common/network-common\ has been restructured to support new streaming and shuffle capabilities. New message types have been added to the wire protocol, including \ChunkFetchRequest\ for stream uploads and \MergedBlockMetaRequest\ for push-based shuffle metadata. The encoding and decoding logic has been refactored to enable zero-copy transfers via Netty's \FileRegion\ for native transports, significantly reducing memory copies for large data blocks. Additionally, the \Encoders\ utility class now validates array lengths before allocation to prevent potential issues, and the \Message.Type\ enum has been updated to include the new message identifiers.
common/network-common/src/main/java/org/apache/spark/network/protocol · high confidence
Registration of Protobuf serializers and service providers in core
This change registers several new service provider implementations via META-INF/services files. It introduces a ProtobufSerDe service provider listing 17 specific serializers (including AppSummary, PoolData, and various UI wrappers) to enable protobuf-based serialization for Spark status data. It also registers the HadoopFSDelegationTokenProvider and HBaseDelegationTokenProvider as the active HadoopDelegationTokenProvider implementations, and adds a BasicEventFilterBuilder for event log filtering.
core/src/main/resources/META-INF/services · high confidence
Removal of native LZF compression implementation
The native C/JNI implementation for LZF compression and decompression has been removed from the project. This change deletes the \src/native/Makefile\ and the \spark\_compress\_lzf\_LZF.c\ source file, eliminating the custom native code that previously handled LZF operations via JNI. As noted in the commit history, this native code is being replaced by the open-source \compress-lzf\ library, meaning users will no longer rely on this specific native build path for LZF functionality.
src/native · high confidence
Restructured network server components and enforced authentication on stream access
The network server layer has been refactored to improve security and performance. Authentication is now strictly enforced for stream operations (chunk and stream fetches) via a new \AuthCheckingStreamManager\ wrapper in \AbstractAuthRpcHandler\, preventing unauthenticated access to file-distribution channels. Performance is improved by introducing a dedicated \ChunkFetchRequestHandler\ to offload potentially blocking disk I/O from the main event loop, and by adding \BlockPushNonFatalFailure\ to handle push-based shuffle errors more efficiently without retrying on specific non-fatal conditions.
common/network-common/src/main/java/org/apache/spark/network/server · high confidence
Spark REPL now fails the shell if SparkSession initialization fails
The Spark REPL (spark-shell) now exits with an error code if it cannot successfully create the SparkSession, rather than continuing in a broken state. This change ensures that users are immediately notified of startup failures, preventing confusing errors later when trying to use \sc\ or \spark\.
repl · high confidence
SparkLogger now safely handles null or empty MDC arrays
The SparkLogger implementation in the new common-utils-java module has been updated to explicitly check for null or empty MDC (Mapped Diagnostic Context) arrays before processing log messages. This change ensures that logging methods accepting MDC arguments behave correctly and do not encounter errors when no context data is provided, improving the robustness of structured logging in Java applications.
_spark-common-utils-java\2.13 · high confidence
SparkR communication protocol refactored to use Netty-based RBackend
The SparkR integration has been refactored to replace the previous socket-based communication with a new Netty-based backend server (RBackend). This change introduces a structured authentication mechanism using RAuthHelper and RBackendAuthHandler to secure the connection between the R process and the JVM, and adds a JVMObjectTracker to manage object references across the boundary. The new architecture supports configurable backend threads and connection timeouts, and provides a more robust foundation for executing R UDFs and handling data serialization (SerDe) between Spark and R.
core/src/main/scala/org/apache/spark/api/r · high confidence
StorageLevel relocated to common/utils
The StorageLevel class and its companion object have been moved from their previous location to common/utils/src/main/scala/org/apache/spark/storage. This relocation makes the storage level definitions available in the common utilities module, potentially allowing other components outside the core storage package to reference these constants and configuration flags without depending on the full storage subsystem.
common/utils/src/main/scala/org/apache/spark/storage · high confidence
StreamingQueryException now supports error framework parameters and SQL state
The StreamingQueryException class has been updated to integrate with the Spark error framework, allowing it to carry structured error details such as error classes, message parameters, and optional SQL states. This change improves how streaming query failures are reported and handled, particularly in Spark Connect clients, by providing more context about the failure cause and enabling better error classification and debugging for users.
common/utils/src/main/scala/org/apache/spark/sql · high confidence
Structured logging and output commit coordination in Hadoop MapReduce utilities
The SparkHadoopMapRedUtil now uses the structured logging framework for task commit operations, including logging elapsed time and task attempt IDs via MDC keys. It also enforces output commit coordination by default (via spark.hadoop.outputCommitCoordination.enabled) to prevent race conditions when multiple task attempts try to commit the same output partition, coordinating with the driver to authorize commits or aborting tasks if denied.
core/src/main/scala/org/apache/spark/mapred · high confidence
Structured memory error handling and atomic usage tracking in core memory classes
The core memory management classes in \org.apache.spark.memory\ have been updated to improve error diagnostics and concurrency safety. \SparkOutOfMemoryError\ now extends \SparkThrowable\, allowing it to carry structured error classes, message parameters, and SQL states instead of relying on plain \OutOfMemoryError\ messages. \MemoryConsumer\ now tracks its used memory via an \AtomicLong\, preventing potential deadlocks caused by user code during usage updates, and its \throwOom\ method attaches a bounded per-consumer memory breakdown to the error for better debugging. Additionally, \TaskMemoryManager\ exposes peak on-heap and off-heap memory usage via volatile fields, and \TooLargePageException\ is introduced to handle cases where requested pages exceed supported sizes.
core/src/main/java/org/apache/spark/memory · high confidence
Updated Kinesis examples to use AWS SDK for Java v2
The Kinesis streaming examples (KinesisWordCountASL and KinesisExampleUtils) have been migrated from the AWS SDK for Java 1.x to the AWS SDK for Java 2.x. This change updates the underlying client libraries and credential providers used by the examples, aligning them with the connector's upgrade to KCL 2.7.2 and removing the legacy AWS SDK 1.x dependency.
connector/kinesis-asl/src/main/scala/org/apache/spark/examples · high confidence
Updated internal TimSort implementation to align with JDK 11
The internal TimSort algorithm used for sorting collections has been updated to match the JDK 11 implementation. This change ensures consistent sorting behavior and stability across different Java versions, particularly affecting internal operations like those in AppendOnlyMap that rely on this sorting mechanism.
core/src/main/java/org/apache/spark/util/collection · high confidence
Upgrade Bootstrap to v5.3.8
The Spark Web UI now uses Bootstrap v5.3.8, replacing the previous version. This update brings modern CSS utility classes, improved component behavior (such as the new Offcanvas and Collapse APIs), and enhanced accessibility features to the UI pages.
core/src/main/resources/org · high confidence
Test coverage
Add Jest-based UI test suite for Spark Web UI; Add scheduler test infrastructure and suites for AQE, barrier execution, and resource management; Add unit tests for BytesToBytesMap; Added Avro read and write performance benchmarks; Added Avro test resources and test logging configuration; Added BroadcastSuite tests for TorrentBroadcast behavior; Added Docker-based integration tests for DB2, MariaDB, and SQL Server JDBC connectors; Added Hive compatibility test suites; Added JUnit 5 test suite for Java Avro functions; Added JUnit 5 tests for Spark network client transport and SSL factory; Added Java API and TaskContext test suites; Added Java API test suite for JdbcRDD; Added Java tests for Kinesis DStream builder API; Added Java unit tests for Kafka 0-10 connector; Added Java unit tests for ResourceProfile API; Added Kerberos setup scripts and logging configuration for database integration tests; Added OIDC credential propagation E2E integration tests for AWS; Added Protobuf test schemas for the connector; Added Real-Time Mode benchmarks for the Kafka 0-10 connector; Added SSL test resources and test logging configuration; Added Spark event log fixture for executor blacklist testing; Added Volcano integration tests for Spark on Kubernetes; Added and updated test suites for Spark Core; Added comprehensive test suites for Spark storage components; Added comprehensive tests for the Spark Plugin Container; Added executor-side unit tests for classloader isolation, resource parsing, and session lifecycle; Added integration and unit tests for SASL authentication in the shuffle service; Added integration tests for Spark SASL authentication and encryption; Added integration tests for the Kinesis connector; Added log4j2 configuration for protobuf module tests; Added microbenchmark for Platform memory operations; Added microbenchmark suite for RocksDB KVStore operations; Added network-common test suites for chunk fetch, RPC, and SSL integration; Added new test tags for Ammonite, Web Browser, Docker, and extended SQL/Hive/YARN/LevelDB tests; Added security tests for delegation token and OIDC credential management; Added serializer test suites and benchmarks; Added test coverage for Java Optional API; Added test coverage for Netty RPC components; Added test data for XSS escaping in history page; Added test fixtures for ProcfsMetrics parsing; Added test for SerializableConfiguration serialization; Added test resource for incomplete last attempt application; Added test resources for Kafka and Kinesis connectors; Added test resources for fair scheduler, metrics, and SSL validation; Added test service provider configurations for cluster managers and credential providers; Added test suite for SparkLauncher configuration and in-process execution; Added test suite for StringSubstitutor; Added test suite for TimSort edge cases; Added test suite for implicit conversion APIs; Added test suite for the Unified Memory Manager; Added test suites and benchmarks for Spark shuffle components; Added test suites for Kafka 0-10 connector; Added test suites for NioBufferedFileInputStream and ReadAheadInputStream; Added test suites for PySpark Python runner infrastructure; Added test suites for Spark deploy components; Added test suites for Spark metrics system configuration and input/output tracking; Added test suites for UnsafeExternalSorter and UnsafeInMemorySorter with radix sort support; Added test suites for external shuffle service components; Added test suites for network-common utilities; Added test suites for sort-based shuffle components; Added test suites for the Protobuf SQL connector; Added test suites for the REST application submission protocol; Added test suites for the new network authentication and encryption engine; Added test suites for unsafe prefix comparators and radix sort; Added test utilities for Platform memory copy benchmarks; Added tests and benchmarks for Spark IO compression codecs; Added tests and benchmarks for random sampling utilities; Added tests for AppClient standalone mode and a test utility executor; Added tests for Avro archive reading and schema handling; Added tests for BlockTransferService hang prevention and Netty transport benchmarks; Added tests for CollationFactory and UTF8String utilities; Added tests for DriverLogger persistence and configuration behavior; Added tests for EnvironmentPage handling of missing properties and config defaults rendering; Added tests for FileCommitProtocol instantiation and JobID generation; Added tests for Java SerializableMapWrapper containsKey behavior; Added tests for Java-friendly Serializer API and proxy class deserialization; Added tests for KVStoreProtobufSerializer null handling and fallback logging; Added tests for Kafka token provider configuration and credential redaction; Added tests for LocalDiskShuffleMapOutputWriter; Added tests for LocalSchedulerBackend OIDC credential management and lifecycle; Added tests for Master Web UI application and environment pages; Added tests for Murmur3\_x86\_32 hash implementation; Added tests for R backend components; Added tests for RDD operation graph visualization logic; Added tests for RPC SSL configuration and trust manager reloading; Added tests for Spark Status listener, store, and UI backend components; Added tests for SparkBloomFilterSuite with controlled execution; Added tests for SparkShellLoggingFilter behavior; Added tests for StreamingQueryException.toString() behavior; Added tests for UI escaping of executor and task timeline data; Added tests for WholeTextFile input handling and compression support; Added tests for accumulator metrics and source registration configuration; Added tests for authentication-gated RPC handlers and stream manager lifecycle; Added tests for browser-side executor thread dump download; Added tests for pipelined channel shuffle behavior; Added tests for resource discovery plugins and resource profiles; Added tests for the Spark Launcher backend; Added tests for the new credential provider SPI and token ingestion components; Added tests for the new internal configuration API; Added tests for unsafe memory allocation and platform utilities; Added tests for worker log page security and validation; Added unit and integration tests for the Kafka SQL connector; Added unit tests for ByteArray and LongArray utilities; Added unit tests for ChunkedByteBufferOutputStream; Added unit tests for ExecutorMonitor dynamic allocation logic; Added unit tests for ExecutorSummary serialization and SimpleDateParam parsing; Added unit tests for Graphite, StatsD, and Prometheus metric sinks; Added unit tests for Kafka consumer pooling and data fetching; Added unit tests for Netty block transfer and RPC components; Added unit tests for RpcAddress and RpcEnv; Added unit tests for Spark History Server core components; Added unit tests for Spark Standalone Worker components; Added unit tests for Spark utility classes and logging infrastructure; Added unit tests for TaskMemoryManager and memory consumer utilities; Added unit tests for core utility classes; Added unit tests for network protocol encoders and message handling; Added unit tests for new unsafe types and collation support; Added unit tests for partial Sum, Count, and Mean evaluators; Added unit tests for shuffle protocol message encoding and validation; Added unit tests for shuffle sort components; Added unit tests for sketch data structures; Added unit tests for streaming shuffle message encoding and decoding; Added unit tests for the KVStore utility package; Added unit tests for the Storage Page UI component; Added unit tests for the new Streaming Shuffle components; Expanded test coverage for Avro connector data conversion and serialization; Expanded test coverage for Spark Standalone Master; Expanded test coverage for Spark Web UI components; Initial Kubernetes integration test suite; Initial Kubernetes integration test suite and tooling; New benchmark infrastructure for running and generating results; Removed ParallelArraySplitSuite and ReplSuite tests; Test logging configuration for structured and pattern loggers; Updated SQL benchmark results and documentation.
Dependencies
2811 commits updating dependencies (64 manifests)
A dependency / build maintenance change in (dependencies) — 2811 commits (271 fixs), 64 files: “\[SPARK-7895\] \[STREAMING\] \[EXAMPLES\] Move Kafka examples from scala-2.10/src to src”; “\[SPARK-35488\]\[BUILD\] Upgrade ASM to 7.3.1”; “\[SPARK-33843\]\[BUILD\] Upgrade to Zstd 1.4.8”; and 2775 more (refs \#3837, \#221, \#211, \#841, \#527, …).
(dependencies) · high confidence · unverified
Introduce dedicated setup.py for PySpark Classic package
A new \setup.py\ file has been added to the \python/packaging/classic/\ directory to handle the packaging and installation of the PySpark Classic distribution. This script manages the inclusion of necessary JARs, scripts, examples, and license files, and enforces minimum version requirements for dependencies such as Pandas (2.2.0), NumPy (1.23.2), PyArrow (18.0.0), gRPC (1.76.0), and PyYAML (3.11).
pyspark · high confidence
Updated binary license and notice files
The \licenses-binary\ directory has been updated to include the license texts for numerous third-party dependencies, such as AnchorJS, CC0, JLargeArrays, JTransforms, ANTLR, ARPACK, Automaton, BLAS, Bootstrap, Cloudpickle, D3, Dagre-D3, DataTables, DNSJava, F2J, Graphlib-Dot, ICU4J, istack-commons-runtime, Jakarta Annotation API, and Jakarta Servlet API. This ensures that the binary distribution includes the required legal notices for all included libraries.
licenses-binary · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 43 → 43 (-0.1)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 34 → 31 (-2.3)
- Architecture 95 → 95 (+0.0)
- Maturity 74 → 75 (+0.9)
- Readiness 48 → 51 (+2.3)
- Security 73 → 69 (-3.8)
- Accessibility 38 → 40 (+2.3)
- Performance 84 → 100 (+15.6)
Resolved (264)
- ArrowBatchedUDFArrowHandler.run (cyclomatic 17) (python/pyspark/eval_handlers/_arrow.py)
- Change coupling: CTESubstitution.scala ↔ PushdownPredicatesAndPruneColumnsForCTEDef.scala (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala)
- Change coupling: DataSourceV2ScanExecBase.scala ↔ GroupPartitionsExec.scala (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala)
- CheckAnalysis.checkAlterTableCommand (cognitive 32) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala)
- CheckAnalysis.checkAlterTableCommand (cyclomatic 21) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala)
- ClassTooLong: ResolveSessionCatalog (sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala)
- Critical CVE: [GHSA redacted] (core/pom.xml)
- DataStreamWriter.startInternal (cognitive 29) (sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala)
- Duplicated block (11 lines × 2) (common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java)
- Duplicated block (11 lines × 2) (examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java)
- Duplicated block (11 lines × 2) (examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java)
- Duplicated block (11 lines × 2) (mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala)
- Duplicated block (11 lines × 2) (mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala)
- Duplicated block (11 lines × 2) (python/pyspark/ml/evaluation.py)
- Duplicated block (11 lines × 2) (python/pyspark/sql/connect/plan.py)
- Duplicated block (12 lines × 2) (core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala)
- Duplicated block (12 lines × 2) (sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala)
- Duplicated block (12 lines × 3) (sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala)
- Duplicated block (13–14 lines × 2) (examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala)
- Duplicated block (13–14 lines × 3) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala)
- …and 244 more
New (1418)
- AlterTableValidation.validate (cognitive 32) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlterTableValidation.scala)
- AlterTableValidation.validate (cyclomatic 22) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlterTableValidation.scala)
- AsOfJoinMatchConditionResolution.coerceNearestKeys (cognitive 16) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AsOfJoinMatchConditionResolution.scala)
- AstBuilder.visitJsonQuery (cognitive 18) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala)
- BasicExecutorFeatureStep.configurePod (cyclomatic 18) (resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala)
- BroadcastRangeJoinExec.makeCandidateFn (cognitive 31) (sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastRangeJoinExec.scala)
- BroadcastRangeJoinExec.makeCandidateFn (cyclomatic 17) (sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastRangeJoinExec.scala)
- Change coupling: RewriteMergeIntoTable.scala ↔ RewriteUpdateTable.scala (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala)
- ClassTooLong: NativeLibrary (sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java)
- ClassTooLong: V2ExpressionBuilder (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala)
- ColumnDomain.toString (cognitive 18) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnDomains.scala)
- ColumnDomains.compare (cognitive 17) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnDomains.scala)
- ColumnDomains.compare (cyclomatic 16) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnDomains.scala)
- DataSourceV2Utils.loadV2Source (cognitive 17) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala)
- Duplicated block (10 lines × 2) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala)
- Duplicated block (11 lines × 2) (mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala)
- Duplicated block (11 lines × 2) (python/pyspark/eval_handlers/_pandas.py)
- Duplicated block (11 lines × 3) (python/pyspark/eval_handlers/_pandas.py)
- Duplicated block (12 lines × 2) (mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala)
- Duplicated block (12 lines × 2) (sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/PartitionReaderFactory.java)
- …and 1398 more
Changes since last survey
- 300 commits — 209 feature/other, 91 fixes
By area
- sql/core — 102 commits
- sql/catalyst — 70 commits
- python/pyspark — 36 commits
- .github/workflows — 16 commits
- core/src — 13 commits
- (root) — 12 commits
- docs/sql-migration-guide.md — 6 commits
- sql/hive — 6 commits
- dev/merge_spark_pr.py — 4 commits
- sql/connect — 4 commits
- common/variant — 3 commits
- dev/sparktestsupport — 3 commits
- mllib/src — 3 commits
- sql/pipelines — 3 commits
- common/utils — 2 commits
- .github/labeler.yml — 1 commit
- binder/Dockerfile — 1 commit
- build/mvn — 1 commit
- connector/avro — 1 commit
- connector/kafka-0-10-sql — 1 commit
Notable commits
- fix: [MINOR][PYTHON] Fix framed serializer length limit at 2 GiB
- fix: [SPARK-33744][SQL] Fix canonicalization of SortAggregateExec
- fix: [SPARK-48995][PYTHON] Fix intermittent NullPointerException in Column.startswith/endswith with None
- fix: [SPARK-55014][SQL][FOLLOWUP] Fix the version of spark.sql.optimizer.splitProjectionForExpensiveFilters to 4.4.0
- fix: [SPARK-60124][SQL] Fix RewriteDistinctAggregates failure on ListAgg with DISTINCT and FILTER
- fix: [SPARK-60421][SQL] Fix overflow and non-finite handling in vector_avg for float vectors
- fix: [SPARK-60522][SPARK-60027][SPARK-60539][SQL] Fix storage-partitioned join failures and wrong results when Spark cannot evaluate a partition transform as is
- fix: [SPARK-60608][ML][CONNECT][TEST][FOLLOWUP] Strengthen nested model allowlist regression coverage
- fix: [SPARK-60651][SQL] Expand decimal remainder regression coverage
- fix: [SPARK-60655][PS][SQL] Fix modulo with an infinite divisor in pandas API on Spark
- fix: [SPARK-60656][PS] Fix NumPy maximum/minimum NaN propagation
- fix: [SPARK-60743][SQL] Fix make_timestamp silently wrapping an out-of-range seconds value
- fix: [SPARK-60780][SQL] Fix internal error when table OPTIONS contain a subquery
- fix: [SPARK-60797][SQL] Fix wrong decimal partition values when the scale differs from the schema
- fix: [SPARK-60801][SQL] Fix wrong results by reading the Hive default partition as NULL
- fix: [SPARK-60803][SQL] Fix auto-filled DSv2 generated columns reading a struct field before the store assignment cast
- fix: [SPARK-60808][SQL] Fix lost join rows when a runtime bloom filter's creation side is non-deterministic
- fix: [SPARK-60814][INFRA] Determine revert vs reapply by wrapper parity in merge_spark_pr.py
- fix: [SPARK-60817][SQL] Fix auto-filled DSv2 generated columns reading a column in a lambda function before the store assignment cast
- fix: [SPARK-60818][SQL] Fix wrong TINYINT and SMALLINT partition values when they are out of range
- …and 280 more
Architecture
- Unchanged — 0 containers · 14 contexts · 18 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
apache/spark was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit d049f2a96c2b9c6ed87b707996ac84495ad7c39b — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.