Skip to content
CAI
Software that uses CAICheck a score

delta-io/delta

64.8

Adequate · 14 November 2026

279.7k

lines of production code

Scala

with Java

6

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add Delta Connect Python Protobuf definitions

The \python/delta/connect/proto\ package now includes the generated Python Protocol Buffer code (\.py\ and \.pyi\ stubs) for the Delta Connect interface. This adds the foundational message types for accessing Delta tables (by path or name) and defines the command and relation structures required for operations such as cloning, vacuuming, upgrading protocol, generating manifests, creating tables, managing feature support, scanning, describing history/detail, converting to Delta, restoring, checking table status, deleting, updating, merging, and optimizing tables.

python/delta/connect/proto · high confidence

Add Delta Connect protocol definitions and Python code generation configuration

This change introduces the core Protobuf schema definitions for Delta Connect in the \spark-connect/common\ module, establishing the contract for remote execution of Delta Lake operations. The new files define the \delta.connect\ package with base messages for table access (\DeltaTable\) and a comprehensive set of commands and relations, including \CloneTable\, \VacuumTable\, \UpgradeTableProtocol\, \Generate\, \CreateDeltaTable\, \AddFeatureSupport\, \DropFeatureSupport\, \Scan\, \DescribeHistory\, \DescribeDetail\, \ConvertToDelta\, \RestoreTable\, \IsDeltaTable\, \DeleteFromTable\, \UpdateTable\, and \MergeIntoTable\. Additionally, it adds the \buf.gen.yaml\ configuration to enable Python Protobuf code generation for these definitions, ensuring that Python clients can generate the necessary stubs to interact with the Delta Connect server.

spark-connect/common · high confidence

Add DynamoDB as a Commit Coordinator for Delta tables

Users can now use Amazon DynamoDB as a backend for Delta's coordinated commits feature, enabling distributed commit coordination for Delta tables. This change introduces the \DynamoDBCommitCoordinatorClientBuilder\ and supporting classes in the \io.delta.dynamodbcommitcoordinator\ package, which configure the DynamoDB table, endpoint, and AWS credentials provider to manage commit state. An integration test is also included to verify concurrent write and read operations against this coordinator.

spark/src/main/java/io · high confidence

Add GoldenTableUtils for discovering and accessing golden test tables

A new utility object, GoldenTableUtils, has been added to the golden-tables connector to support the consolidation of testing infrastructure. This utility provides methods to locate the root 'golden' resource directory, retrieve specific table paths or files by name, and automatically discover all available golden tables by scanning for directories containing a '\_delta\_log' folder. This simplifies how tests access and iterate over the set of golden table fixtures.

connectors/golden-tables/src/main/scala · high confidence

Add Hilbert clustering support and JoinedProjection helper

This change introduces new internal components to support Hilbert clustering and optimize statistics collection. It adds HilbertStates.java, which provides pre-computed state lists for mapping n-dimensional points (2-9 dimensions) to Hilbert distance keys, enabling efficient clustering operations. Additionally, it adds JoinedProjection.scala, a helper class that generates joined projections by wrapping two rows into a single concatenated view using nested access, primarily used to efficiently update per-column aggregate buffers during statistics collection without the overhead of size checks on every access.

spark/src/main/scala/org/apache/spark/sql/delta/expressions · high confidence

Add Hilbert curve clustering and sort-within-files support for multi-dimensional clustering

Delta Lake now supports Hilbert curve-based multi-dimensional clustering alongside the existing Z-order method, allowing users to choose the clustering algorithm via the \curve\ parameter. The implementation introduces new internal functions (\hilbert\_index\, \range\_partition\_id\, \interleave\_bits\) to compute clustering keys. Additionally, a new configuration option \MDC\_SORT\_WITHIN\_FILES\ enables sorting data within partitions after clustering, which can improve query performance by keeping related rows physically closer together on disk.

spark/src/main/scala/org/apache/spark/sql/delta/skipping · high confidence

Add JSON serialization utilities for Delta Kernel rows

The kernel-defaults module now includes a new \JsonUtils\ class that provides methods to serialize and deserialize Delta \Row\ objects to and from JSON strings. This utility supports a specific set of data types (including primitives, strings, structs, arrays, and maps with string keys) and is currently utilized in tests, with plans to support broader refactoring efforts for JSON file handling.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/json · high confidence

Add Terraform infrastructure for AWS and GCP benchmarks

New Terraform configurations have been added to provision the cloud infrastructure required for running benchmarks. The AWS module (in \benchmarks/infrastructure/aws/terraform\) creates a VPC, S3 bucket, MySQL metastore, and EMR cluster using the \hashicorp/aws\ provider (v4.15.1). The GCP module (in \benchmarks/infrastructure/gcp/terraform\) provisions a Google Storage bucket, Dataproc Metastore, and Dataproc cluster using the \hashicorp/google\ and \hashicorp/google-beta\ providers (v4.22.0). Both sets of files include README documentation and variable definitions to guide users through setup and teardown.

(repo-wide) · high confidence

Add TestBarrier utility and enhance DatabricksLogging with usage tracking

This change introduces a new TestBarrier utility in spark/src/main/scala/com/databricks/spark/util/TestBarrier.scala to support concurrency testing, specifically for AMT conflict-resolution scenarios. It also significantly updates DatabricksLogging.scala (renamed from the root src directory) by adding a Log4jUsageLogger that captures usage records during test execution, enhancing the logging trait with methods like recordProductUsage and recordProductEvent, and introducing new metric and tag definitions for better observability.

spark/src/main/scala/com · high confidence

Add script to run Delta Kernel examples locally or against staged artifacts

Users can now execute Delta Kernel example programs (single-threaded and multi-threaded table readers) and integration tests via a new Python runner script. The script supports running examples against locally built JARs or against specific versions from a Maven repository, automatically managing artifact caches and invoking the necessary Maven commands to demonstrate API usage.

kernel/examples · high confidence

Add support for Apache Spark 4.3.0

The client now supports building against Apache Spark 4.3.0 (specifically the 4.3.0-SNAPSHOT version). This update includes new Scala shim directories for version 4.3 and cross-version compatibility (4.0-4.3, 4.1-4.3, 4.2-4.3), renames existing shims to cover the 4.1-4.3 range, and updates the build configuration and artifact tests to recognize and validate the new Spark version. Also in this area: Added tests for published artifact naming conventions across Spark and Flink versions.

project/tests · high confidence

Add support for IBM Cloud Object Storage and Oracle Cloud Infrastructure

Delta Lake now includes new LogStore implementations for IBM Cloud Object Storage (IBMCOSLogStore) and Oracle Cloud Infrastructure (OracleCloudLogStore), enabling users to store transaction logs in these cloud environments. The IBM Cloud Object Storage implementation requires the Hadoop configuration \fs.cos.atomic.write\ to be set to true to ensure atomic writes, while the Oracle Cloud Infrastructure implementation utilizes atomic rename operations for non-overwrite writes. Both components are marked as unstable and include corresponding test suites to verify their behavior.

contribs · high confidence

Add utility to validate Parquet files for Iceberg Compatibility V2

A new utility class, ParquetIcebergCompatV2Utils, has been added to the Delta Lake UniForm module to help users verify whether existing Parquet data files meet the requirements for Iceberg Compatibility V2. This tool inspects Parquet file footers to ensure that timestamps are not stored as INT96 and that all fields (including nested types like LIST and MAP) possess the required field IDs, which are mandatory for Iceberg V2 compatibility. This enables users to check data readiness without performing a full table rewrite.

spark/src/main/scala/org/apache/spark/sql/delta/uniform · high confidence

A new Docker Compose setup for Flink version 2.0.1 is now available in the flink/docker/2.0 directory. This configuration defines a JobManager and two TaskManagers, enabling users to run a local Flink cluster. The setup includes an initialization script that copies user-provided JARs and an S3 configuration file (core-site.xml) into the appropriate Flink directories, allowing for S3 integration out of the box.

flink/docker · high confidence

Added Scala build launcher and configuration for the Scala example

The Scala example now includes a dedicated build environment with an sbt launcher script, a library for managing sbt execution, and a repository configuration file. This setup allows users to build and run the Scala example using sbt, with repository sources configured to use Maven Central, Typesafe, and the Spark Packages repository (repos.spark-packages.org) instead of the deprecated Bintray service.

examples/scala/build · high confidence

Added internal utility classes for lazy evaluation and list operations

The kernel API now includes two new internal utility classes: \Lazy\, which provides a simple lazy-initialization wrapper using a supplier and optional, and \ListUtils\, which offers helper methods for partitioning lists and retrieving the first or last element (with a note to remove the latter once JDK 21+ is the minimum supported version). These additions support internal implementation details but do not expose new public APIs to users.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/lang · high confidence

Adds Mumbling deletion vector bitmap implementation

Delta Lake now includes a Java implementation of the Mumbling bitmap format for deletion vectors, enabling both reading and writing of this compressed representation. The change introduces core components including a read-only \MumblingBitmap\ class, a \MumblingBitmapWriter\ for serializing and merging bitmaps, and supporting utilities for PFOR encoding, bit packing, and byte buffer access. This provides the underlying infrastructure for more efficient storage and processing of deletion vectors.

spark/src/main/java/org/apache/spark/sql/delta/deletionvectors · high confidence

Automated suite generation with formatting and validation

The Delta Suite Generator now automatically writes generated test suites to files, organizing them by package and base class while applying consistent code formatting via Scalafmt. A new validation test ensures that generated files are not manually modified, preserving the integrity of the automated workflow.

spark/delta-suite-generator · high confidence

Concurrent Identity Columns: sequence-service abstraction and in-memory backend

This change introduces the driver-side abstraction for the Unity Catalog Identity Sequence Service, enabling Concurrent Identity Columns (CIC) to reserve non-overlapping ID ranges. It defines the \IdentitySequenceService\ trait with operations to create, reserve, and drop sequences, along with request/response models for managing sequence state. A new \IdentitySequenceServices\ resolver allows users to configure a custom backend implementation via the \spark.databricks.delta.identityColumn.concurrent.serviceClassName\ configuration; if no backend is configured, the system fails with a clear error directing users to set this property.

spark/src/main/scala/org/apache/spark/sql/delta/cic · high confidence

Convenience Delta implicits for Spark DataFrames and Streams

Users can now use concise \.delta(path)\ methods directly on Spark's DataFrameReader, DataFrameWriter, DataStreamReader, and DataStreamWriter APIs to read from and write to Delta tables, eliminating the need to explicitly call \.format("delta")\.

spark/src/main/resources/META-INF, spark/src/main/scala/io/delta/implicits · high confidence

Delta Lake Spark Connect client API for table operations

The Spark Connect thin client now includes a full set of Scala APIs for managing Delta tables remotely. This adds a \DeltaTable\ class for standard operations like vacuum, history, and delete, alongside builder classes (\DeltaTableBuilder\, \DeltaColumnBuilder\) for creating and replacing tables with support for generated and identity columns. It also introduces \DeltaMergeBuilder\ for upserts (including \whenNotMatchedBySource\ updates) and \DeltaOptimizeBuilder\ for compaction and Z-Ordering. To ensure compatibility across Spark 4.0, 4.1, and 4.2, the client uses version-specific shims to handle the relocation of the \SparkStringUtils\ class.

spark-connect/client · high confidence

Delta format sharing support for Spark queries

Delta Sharing now supports reading shared tables using the native Delta format instead of only Parquet. This enables users to leverage Delta-specific capabilities such as Change Data Feed (CDF) for streaming and historical queries, limit pushdown for more efficient file fetching, and server-side file skipping via JSON predicate hints. The change introduces a new data source implementation and query planning rules that construct a local Delta log from sharing metadata, allowing Spark to execute these queries with improved performance and feature parity with local Delta tables.

sharing/src/main · high confidence

Users can now utilize Delta Lake with Flink SQL and external catalogs. This change introduces the \org.apache.flink.table.factories.Factory\ service provider file, registering \DeltaDynamicTableSinkFactory\ and \FlinkUnityCatalogFactory\ to enable automatic discovery of Delta sinks and Unity Catalog support in Flink SQL sessions. Additionally, a new \delta-flink.properties\ configuration file is added, providing default settings for sink retry attempts, table caching behavior, and credential refresh parameters, allowing users to tune performance and reliability without code changes.

flink/src/main/resources · high confidence

The Flink connector now supports upsert writes (INSERT, UPDATE\_AFTER, DELETE) via the \write.mode='upsert'\ SQL DDL option, enabling row-level updates and deletes in Delta tables. This change introduces new Flink sink components (\DeltaWriterResult\, \DeltaWriterTask\, \DeltaDynamicTableSink\) to handle checkpointing and file writing, and adds a \FlinkUnityCatalog\ implementation to allow writing to tables managed by Unity Catalog using endpoint and token authentication. Also in this area: Delta V2 connector classes relocated to Spark namespace.

kernel-examples · high confidence

The Flink kernel module now includes new classes that enable incremental v2 checkpointing for Flink sink tables (CheckpointActionRow and CheckpointWriter), provide expression utilities for building row-IN predicates used in upsert merge paths (ExpressionUtils), and implement deletion-vector read/write/merge support via a new DVAccess abstraction and BinDVAccess file format (along with RoaringBitmapArray serialization). These changes add the underlying kernel capabilities that Flink sink tables rely on for efficient checkpointing, upserts, and deletion-vector operations.

flink/src/main/java/io/delta/flink/kernel · high confidence

Initial Python package structure and configuration utility

This change introduces the foundational structure for the Delta Lake Python package. It adds a \configure\_spark\_with\_delta\pip\ utility that automatically resolves and configures the correct Delta Lake Maven artifact (including Spark and Scala version suffixes) for a SparkSession builder, simplifying local pip-based installations. The package also exposes the library version via \delta.\\version\\_\, provides type definitions in \\_typing.py\, and includes a \py.typed\ marker file to indicate PEP 561 compliance for static type checking.

python/delta · high confidence

Initial release of the Python DeltaTable API

Introduces the \python/delta/tables.py\ module, providing the primary Python interface for interacting with Delta Lake tables. This includes the \DeltaTable\ class for operations such as converting Parquet tables, reading data as DataFrames, and performing schema-agnostic mutations like \delete\, \update\, and \merge\. It also introduces the \DeltaMergeBuilder\ to construct complex merge operations with conditional updates, inserts, and deletes, along with utility methods for generating manifest files and applying aliases.

_delta\spark · high confidence

Introduce Catalog-Owned table feature with Unity Catalog commit coordination

This change introduces the Catalog-Owned table feature, enabling Delta tables to be managed by Unity Catalog as their commit coordinator. It adds a new \CatalogOwnedCommitCoordinatorBuilder\ interface and a dedicated provider to register and retrieve catalog-specific commit coordinators, allowing tables to leverage Unity Catalog for transaction coordination. The implementation includes an in-memory mock (\InMemoryUCCommitCoordinator\ and \InMemoryUCClient\) to support testing of catalog-owned table operations without requiring a live Unity Catalog service. Additionally, it adds usage logging for catalog-owned tables and coordinated commits to track operational metrics and errors, such as invalid path-based access attempts or missing coordinator implementations.

spark/src/main/scala/org/apache/spark/sql/delta/coordinatedcommits · high confidence

Introduce CommitRange API for incremental table changes

The Delta Kernel now exposes a new CommitRange API that allows users to retrieve actions and commit metadata for a specific range of table versions. This feature enables incremental processing by letting callers define start and end boundaries (by version or timestamp) and retrieve the corresponding delta log files and actions. The implementation includes a builder pattern for configuring the range, validation logic to ensure version consistency, and factory methods to resolve boundaries and fetch the necessary data from the log.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/commitrange · high confidence

Introduce DefaultEngine and Hadoop-based handler implementations

The kernel-defaults module now provides a concrete DefaultEngine implementation that serves as the entry point for Delta Lake operations, replacing the previous direct Hadoop coupling with a generic FileIO abstraction. This engine wires together default implementations for expression evaluation, JSON parsing and writing, Parquet reading and writing, and file system operations, all backed by Hadoop APIs. Additionally, a LoggingMetricsReporter is included to output metrics reports as JSON via SLF4J, giving users visibility into engine performance and operation details.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/engine · high confidence

Introduce DefaultJsonRow for parsing Delta transaction log JSON

The kernel-defaults module now includes a new DefaultJsonRow class that implements the Row interface to decode JSON nodes from Delta transaction logs into strongly-typed Java objects. This implementation handles decoding for primitive types (boolean, byte, short, int, long, float, double, string), decimals, and complex structures (struct, array, map), enabling the kernel to correctly read and interpret the JSON-based metadata in Delta log files.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/data · high confidence

Introduce Delta Connect Python client for remote Spark sessions

This change adds the initial implementation of the Delta Connect Python client in the \python/delta/connect\ module, enabling Delta Lake operations over Spark Connect. It introduces a \DeltaTable\ class compatible with the remote execution model, supporting core table operations such as scan, delete, update, merge, vacuum, history, and detail. The module also includes the necessary logical plan definitions to serialize these operations to the server and registers custom exception classes to ensure Delta-specific errors are correctly converted and surfaced to Python users.

python/delta/connect · high confidence

Introduce Delta Kernel Java data type API

This change adds the core data type definitions for the Delta Kernel Java API in the \io.delta.kernel.types\ package, enabling connectors to exchange type information with the kernel. It introduces a class hierarchy with \DataType\ as the base, \BasePrimitiveType\ for primitives, and specific types including \BooleanType\, \ByteType\, \ShortType\, \IntegerType\, \LongType\, \FloatType\, \DoubleType\, \DateType\, \TimestampType\, \TimestampNTZType\, \BinaryType\, \StringType\, \DecimalType\, and \VariantType\. The update also adds \StructField\ to represent schema fields with metadata and type changes, \FieldMetadata\ for storing field-level metadata, \TypeChange\ to track schema evolution, and \MetadataColumnSpec\ to define internal metadata columns like \row\_index\, \row\_id\, and \row\_commit\_version\. All public API classes are marked with \@Evolving\ and \@since\ tags to indicate their stability status.

kernel/kernel-api/src/main/java/io/delta/kernel/types · high confidence

Introduce Delta Kernel Unity Catalog integration layer

This change adds the \kernel/unitycatalog\ module, providing the core Java components for Delta Kernel to interact with Unity Catalog-managed tables. It introduces \UCCatalogManagedClient\ for loading snapshots and \UCCatalogManagedCommitter\ for handling catalog create and write operations, including support for Uniform metadata and domain metadata (e.g., clustering). The implementation includes utility classes like \UnityCatalogUtils\ for parsing delta files and extracting table properties, adapters for protocol and metadata actions, and a telemetry framework (\UcCommitTelemetry\, \UcLoadSnapshotTelemetry\, \UcPublishTelemetry\) to track performance metrics for these operations.

kernel/unitycatalog/src/main · high confidence

Introduce Delta Kernel checksum computation and storage utilities

Added new internal utility classes (ChecksumReader, ChecksumUtils, ChecksumWriter) to the Delta Kernel API that enable computing table state checksums and writing them as files. This allows the kernel to generate and persist CRC information—including table size, file counts, histograms, and metadata—which supports incremental CRC construction and improves compatibility with Delta-Spark checksum expectations.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/checksum · high confidence

Introduce Delta Kernel data representation interfaces

The \io.delta.kernel.data\ package now provides the core Java interfaces for representing data within the Delta Kernel, including \ColumnVector\ for columnar access, \Row\ for record-level access, and \ColumnarBatch\ for batch operations. It also introduces \FilteredColumnarBatch\ to handle row selection via optional selection vectors, and \ArrayValue\/\MapValue\ to support complex nested types. These interfaces, marked as \@Evolving\ since version 3.0.0, form the foundation for reading and manipulating Delta table data in a columnar format.

kernel/kernel-api/src/main/java/io/delta/kernel/data · high confidence

Introduce Delta Kernel internal action models for log replay and writes

The Delta Kernel module now includes a set of internal Java classes in \io.delta.kernel.internal.actions\ to represent Delta Log actions (\AddFile\, \RemoveFile\, \CommitInfo\, \Metadata\, \SingleAction\, etc.). These classes provide structured, row-backed models for parsing and generating Delta log entries, enabling features like conflict detection, Iceberg compatibility utilities, and transaction building within the Kernel engine.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/actions · high confidence

Introduce Delta Storage module with cloud-specific LogStore implementations

The new \delta-storage\ module provides a pluggable \LogStore\ interface and concrete implementations for Azure, GCS, HDFS, and local file systems, enabling Delta Lake to write transaction logs with storage-system-specific atomicity and consistency guarantees. Azure and HDFS implementations use atomic rename operations to ensure files are visible only when fully written, while the GCS implementation handles GCS's strong consistency and precondition failures by writing in a separate thread, and the S3 implementation uses a per-JVM path lock to prevent concurrent writes to the same commit file. This modularization allows users to configure the appropriate storage backend via Hadoop configuration properties.

storage · high confidence

Introduce DeltaSharingFileIndex for batch query file resolution

Added a new \DeltaSharingFileIndex\ class to handle file listing for Delta Sharing batch queries. This component wraps the Delta Sharing client to fetch pre-signed URLs, constructs a synthetic Delta log, and delegates file listing to a \TahoeLogFileIndex\, enabling the V2 scan path to build snapshots consistently with the shared log builder. Also in this area: Support for Apache Spark 4.3 and improved streaming and bitmap filtering; New mutable bitmap and 64-bit bitmap implementations for deletion vectors.

repository, spark/src/main/scala/org/apache/spark/sql/delta/deletionvectors · high confidence

The Flink Delta sink now supports configurable merge strategies via the new \mergestrategy\ package. For append-only sinks, the \AppendOnly\ strategy writes rows directly without modifying existing files. For upsert sinks, the system now implements \MoRUpsert\ (Merge-on-Read) as the default, which updates data by writing deletion vectors to existing Parquet files, and \CoWUpsert\ (Copy-on-Write), which rewrites affected files entirely. These strategies are coordinated by a \RowLocator\ interface (with a default \ScanLocator\ implementation) to identify candidate files based on primary keys, enabling efficient row-level updates in Flink Delta sinks.

flink/src/main/java/io/delta/flink/sink/mergestrategy · high confidence

Introduce Hudi Universal Format support for Delta tables

Delta Lake now supports Hudi as a Universal Format, allowing Delta tables to be automatically converted into Hudi tables. This change introduces the core conversion engine, including schema translation utilities that map Delta data types to Hudi Avro schemas, and transaction handling that commits Delta actions as Hudi replace commits. The implementation includes an asynchronous conversion thread to manage the transformation of Delta snapshots into Hudi format in the background, ensuring that the Hudi table at the same path stays synchronized with Delta commits.

hudi/src/main · high confidence

Introduce Iceberg-to-Delta conversion infrastructure and utilities

This change adds the core implementation for converting Iceberg tables to Delta format within the \iceberg\ module. It introduces a suite of Scala utilities to handle schema translation (\IcebergSchemaUtils\), partition value conversion (\IcebergPartitionUtil\, \IcebergPartitionConverter\), and statistics mapping (\IcebergStatsUtils\), alongside serialization wrappers (\IcebergSparkWrappers\) to ensure compatibility with Spark's distributed execution. A new integration test script (\iceberg\_converter.py\) is also included to demonstrate the end-to-end workflow of creating an Iceberg table, converting it to Delta, and verifying data integrity.

iceberg · high confidence

Introduce JsonMetadataDomain base class for structured metadata storage

The Delta Kernel now includes a new abstract base class, JsonMetadataDomain, which provides a standardized mechanism for serializing and deserializing metadata domain configurations to and from JSON strings using Jackson. This class simplifies the implementation of specific metadata domains (such as RowTrackingMetadataDomain) by handling the JSON conversion logic, allowing features to store structured configuration data within table metadata more reliably.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/metadatadomain · high confidence

The Flink connector now supports a new table implementation backed by the Delta Kernel, providing a consistent foundation for interacting with Delta tables. This change introduces a new \AbstractKernelTable\ base class and specific implementations like \HadoopTable\ for filesystem-based tables and \CatalogManagedTable\ for Unity Catalog-managed tables, enabling unified handling of metadata, schema, and commit operations. It also adds a new \DeltaCatalog\ interface to abstract table discovery and credential vending, a \CredentialManager\ for proactive credential refresh, and a global \Conf\ class for operational tuning of sinks (e.g., retry policies, thread pools, caching).

flink/src/main/java/io/delta/flink/table · high confidence

Introduce Row Tracking Backfill and Unbackfill Commands

Delta Lake now provides commands to backfill and unbackfill row tracking metadata for existing tables. The new RowTrackingBackfillCommand re-commits files that lack base row IDs, enabling row tracking on tables that were created before this feature was available, while RowTrackingUnBackfillCommand removes this metadata to disable row tracking. These operations are implemented via a new backfill framework (BackfillCommand, BackfillExecutor, BackfillBatch) that processes files in configurable batches to avoid large commits, and includes protocol upgrades to support the RowTrackingFeature when necessary.

spark/src/main/scala/org/apache/spark/sql/delta/commands/backfill · high confidence

Introduce S3DynamoDBLogStore for concurrent Delta Lake commits on S3

This change adds the S3DynamoDBLogStore implementation, enabling Delta Lake to use an external DynamoDB table as a coordination layer for commit operations on S3. This allows multiple concurrent writers to safely commit to the same Delta table by using DynamoDB for mutual exclusion, preventing data loss in eventually consistent or non-mutually-exclusive storage environments. The implementation includes a base class (BaseExternalLogStore) for external log stores, a retryable iterator to handle transient S3 errors, and configuration options for DynamoDB table settings, credentials, and retry behavior.

storage-s3-dynamodb · high confidence

Introduce UnityCatalog as the committing catalog for Delta UniForm tables

Added \UnityCatalog\ and \UnityCatalogTableOperations\ to enable Iceberg metadata commits for Delta UniForm tables via Unity Catalog instead of HiveCatalog. This change allows Delta Lake tables to maintain their metadata in Unity Catalog while using Iceberg for storage, ensuring that schema and partition spec field IDs assigned by Delta are preserved during conversion to Iceberg V3 format.

icebergShaded/src/main/java/org/apache/iceberg/unityCatalog · high confidence

Introduce V2 checkpoint support and checkpoint protection in the Kernel

The Kernel checkpoint subsystem now supports V2 checkpoints, which can include sidecar files for large actions, and introduces checkpoint protection to prevent the cleanup of expired transaction logs when a checkpoint is in use. This change adds new internal classes to parse and write V2 checkpoint formats, handle sidecar file metadata, and manage the \\_last\_checkpoint\ hint file, ensuring compatibility with newer Delta Lake features while maintaining support for classic and multi-part checkpoints.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/checkpoints · high confidence

Introduce collation-aware data skipping in the Kernel

The Kernel's data skipping engine now supports collation identifiers, allowing it to correctly prune files for string columns that use specific collations. This change introduces a new \DataSkippingPredicate\ class to track referenced columns and collations, updates \StatsSchemaHelper\ to include a \statsWithCollation\ field in the statistics schema, and modifies \DataSkippingUtils\ to construct and apply skipping filters that respect these collation settings.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/skipping · high confidence

Introduce default ColumnVector implementations for the Delta Kernel

The \kernel-defaults\ module now provides a complete set of default \ColumnVector\ implementations for reading data, including scalar types (boolean, byte, short, int, long, float, double, decimal, string, binary), complex types (array, map, struct), and generic views. This change introduces \AbstractColumnVector\ as a base class, specific implementations like \DefaultIntVector\ and \DefaultStructVector\, and utility vectors such as \DefaultViewVector\ for sub-setting data and \DefaultSubFieldVector\ for accessing nested struct fields. These components form the internal data access layer for the kernel's default reader, enabling efficient columnar data retrieval.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/data/vector · high confidence

Introduce internal Deletion Vectors implementation for Delta Kernel

Added a new internal implementation for Deletion Vectors in the Delta Kernel, including classes for Base85 encoding/decoding, RoaringBitmapArray handling, and loading deletion vector bitmaps from inline or on-disk storage. This enables the Kernel to read and process deletion vectors, which are used to track deleted rows in Delta tables.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/deletionvectors · high confidence

Introduce internal Path utility for file system operations

The Delta Kernel now includes an internal \Path\ class in the \io.delta.kernel.internal.fs\ package to handle file and directory naming. This utility, adapted from Apache Hadoop, provides standardized path construction, resolution against parent paths, and normalization of URI components, including specific handling for Windows drive letters and directory separators.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/fs · high confidence

Introduces AMT-specific Parquet file format and typed partition handling

This change adds the AMTParquetFileFormat class to handle writing and reading Adaptive Metadata Tree (AMT) manifests. On the write side, it ensures manifests are written as Iceberg V4-compatible Parquet files by enforcing TIMESTAMP\_MICROS output and including field IDs. On the read side, it disables file splitting and pushed filters to correctly compute row-ID inheritance over manifest deletion vectors before filtering. Additionally, it introduces AMTPartitionValues to convert between Delta's string-based partition maps and Iceberg's typed partition structs, ensuring consistent serialization and deserialization of partition data for AMT manifests. Also in this area: 7 commits (0 fixes) modifying spark; Pinned statement time for INSERT REPLACE ON/USING operations.

spark/src/main/scala/org/apache/spark/sql/delta/amt · medium confidence · unverified

Introduces internal metrics reporting infrastructure for Delta Kernel operations

This change adds the internal implementation classes required to capture and report metrics for Snapshot, Scan, and Transaction operations within the Delta Kernel. It introduces report implementations (SnapshotReportImpl, ScanReportImpl, TransactionReportImpl) that track operational details such as table version, schema, file counts, and file size histograms, along with a JSON serialization mechanism (MetricsReportSerializer) to output these reports. A SnapshotQueryContext class is also added to manage query state and trigger metric reporting upon completion or failure, enabling observability into kernel-level data operations.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/metrics · high confidence

Introduction of AdmittableFile interface for Delta streaming admission control

A new AdmittableFile interface has been added to the Delta Lake streaming sources to support admission control logic. This interface defines methods to check if a file has an associated action and to retrieve its size, providing a unified abstraction that allows both DSv1 and DSv2 IndexedFile implementations to be used with the new admission control mechanism.

spark/src/main/java/org/apache/spark/sql/delta/sources · high confidence

Introduction of LogStoreProvider for scheme-based storage selection

The Delta Kernel now includes a new LogStoreProvider utility that automatically selects the appropriate LogStore implementation based on the file scheme (e.g., S3, Azure, GCS, HDFS) of the path being accessed. This allows the kernel to dynamically instantiate the correct storage handler (such as S3SingleDriverLogStore or AzureLogStore) either via configuration properties or by default, enabling seamless support for multiple cloud storage backends without manual configuration for standard schemes.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/logstore · high confidence

Kernel implementation of row tracking metadata and materialized columns

The Delta Kernel now includes internal support for row tracking, enabling tables to maintain unique row identifiers and commit versions. This change introduces the \RowTrackingMetadataDomain\ to persist the row ID high watermark as domain metadata, and the \MaterializedRowTrackingColumn\ class to manage the configuration and physical mapping of materialized \\_row-id\ and \\_row-commit-version\ columns. Additionally, the \RowTracking\ utility handles the assignment of base row IDs and default commit versions to new files, including logic for conflict resolution during concurrent transactions. For users, this provides the foundational kernel capability to read and write row tracking metadata, ensuring consistent row identity across table operations.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/rowtracking · high confidence

Kernel support for clustering column metadata storage

The Delta Kernel now includes internal infrastructure to persist and retrieve clustering configuration. A new \ClusteringMetadataDomain\ class handles serialization of clustering column names (supporting both logical and physical names depending on column mapping settings) into the \delta.clustering\ metadata domain, while \ClusteringUtils\ provides a helper to generate the corresponding domain metadata. This enables the kernel to store and read clustering definitions as part of the transaction log.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/clustering · high confidence

New Delta Kernel Java interfaces for table operations

The Delta Kernel module introduces a set of new Java interfaces to standardize table interactions. This includes \CommitActions\ and \CommitRange\ for iterating over specific commit versions and their actions, \Snapshot\ and \SnapshotBuilder\ for resolving table states by version or timestamp, and \Scan\ and \ScanBuilder\ for reading data with filtering and pagination support. Additionally, \Table\ serves as the entry point for resolving tables, while \Operation\ defines standard transaction types and \DataWriteContext\ provides context for data writes.

kernel/kernel-api/src/main/java/io/delta/kernel · high confidence

New Delta Kernel example programs for table reads and writes

Added a set of Java example programs in the \kernel-examples\ module that demonstrate how to use the Delta Kernel APIs for both reading and writing Delta tables. The new code includes \BaseTableReader\ and \BaseTableWriter\ as base utilities, along with concrete examples: \SingleThreadedTableReader\ and \MultiThreadedTableReader\ for querying tables with column pruning and filtering, and \CreateTable\, \CreateTableAndInsertData\ for creating unpartitioned and partitioned tables and inserting data (including CTAS and idempotent writes). A \RowSerDe\ utility is also provided to support serialization of scan state and files for distributed reading scenarios.

kernel/examples/kernel-examples · high confidence

New Delta Kernel expressions API for predicates and scalars

The \io.delta.kernel.expressions\ package now provides a structured API for defining and evaluating filter predicates and scalar expressions within Delta Kernel. This includes a base \Expression\ interface, \ScalarExpression\ for value transformations (such as \ELEMENT\_AT\, \COALESCE\, \ADD\, \TIMEADD\, and \SUBSTRING\), and \Predicate\ for boolean filters (including \AND\, \OR\, \NOT\, \IS\_NULL\, \IS\_NOT\_NULL\, \LIKE\, \IN\, and \StGeometryBoxesIntersect\). The API supports advanced features like nested column references via \Column\, literal values via \Literal\, and string collation for consistent comparison semantics. Connectors can implement \ExpressionEvaluator\ and \PredicateEvaluator\ to optimize these expressions against columnar data batches.

kernel/kernel-api/src/main/java/io/delta/kernel/expressions · high confidence

New Delta benchmark framework with TPC-DS and Merge workload support

The \benchmarks/src\ directory now contains a new open-source benchmarking framework for Delta Lake workloads. This includes a base \Benchmark\ class for measuring SQL query durations and generating JSON reports, alongside specific implementations for TPC-DS performance benchmarks (\TPCDSBenchmark\, \TPCDSDataLoad\) and Delta \MERGE\ operation benchmarks (\MergeBenchmark\, \MergeDataLoad\, \MergeTestCases\). The framework supports configurable scale factors (1GB and 3TB for TPC-DS), multiple iterations, and cloud storage paths for data and reports. A utility object \SparkUtils\ is also added to capture Spark environment information and provide a median calculation helper.

benchmarks/src · high confidence

New Delta performance benchmark framework with TPC-DS and Merge workloads

A new benchmarking framework has been introduced to measure Delta Lake performance on Spark clusters running on AWS EMR or Google Cloud Dataproc. The framework includes predefined benchmarks for TPC-DS (at 1GB and 3TB scales) and Merge operations, allowing users to load data and run 99 standard queries against Delta tables. It supports configuration for external Hive Metastores (RDS MySQL or Dataproc Metastore) and provides Terraform templates for infrastructure setup. The tooling includes a Python runner (\run-benchmark.py\) and SBT build scripts, targeting Delta version 2.3.0.

benchmarks · high confidence

New Iceberg compatibility metadata validators and updaters

The Delta Kernel now includes a new set of classes in the \icebergcompat\ package to validate and update table metadata for Iceberg compatibility. This includes \IcebergCompatMetadataValidatorAndUpdater\ as a base class, with specific implementations for \IcebergCompatV2\, \IcebergCompatV3\, \IcebergWriterCompatV1\, and \IcebergWriterCompatV3\. These validators enforce required table properties (like column mapping modes), check for supported data types and table features, and ensure metadata consistency when Iceberg compatibility features are enabled. Additionally, \IcebergUniversalFormatMetadataValidatorAndUpdater\ ensures that Universal Format settings are consistent with Iceberg compatibility settings.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/icebergcompat · high confidence

New JSON serialization and type widening validation in Kernel API

The Kernel API now includes a dedicated \DataTypeJsonSerDe\ class for serializing and deserializing Delta data types to and from JSON, replacing previous ad-hoc handling. Additionally, a new \TypeWideningChecker\ utility has been added to validate schema evolution operations, enforcing protocol rules that allow widening integer, floating-point, date, and decimal types (e.g., Byte to Long, Float to Double, Date to TimestampNTZ, and specific precision/scale increases for Decimals).

kernel/kernel-api/src/main/java/io/delta/kernel/internal/types · high confidence

New Scala APIs for Identity Columns, Merge Execution, and Table Optimization

The Delta Lake Scala API in \io.delta.tables\ has been expanded with new capabilities. \DeltaColumnBuilder\ now supports defining identity columns via \generatedAlwaysAsIdentity\ and \generatedByDefaultAsIdentity\, allowing users to create auto-incrementing columns. The \DeltaMergeBuilder\ has been refactored to execute MERGE operations using the DataFrame API, returning a \DataFrame\ instead of \Unit\ to provide execution metrics and enable chaining. Additionally, \DeltaOptimizeBuilder\ introduces \executeCompaction\ and \executeZOrderBy\ methods, giving users programmatic control over file compaction and Z-Order optimization directly from Scala code.

spark/src/main/scala/io/delta/tables · high confidence

New Scala examples for Delta Lake features

The examples/scala directory now includes a comprehensive set of runnable Scala programs demonstrating core and advanced Delta Lake capabilities. Users can explore Change Data Feed (ChangeDataFeed.scala), table clustering (Clustering.scala), schema evolution with map types (EvolutionWithMap.scala), Iceberg compatibility via UniForm (IcebergCompatV2.scala, UniForm.scala), and streaming upserts (Streaming.scala). The suite also provides quickstart guides for SQL and path-based operations (Quickstart.scala, QuickstartSQL.scala, QuickstartSQLOnPaths.scala), utility commands like vacuum and history (Utilities.scala), integration with Unity Catalog (UnityCatalogQuickstart.scala), and support for the VARIANT data type (Variant.scala). A build configuration (build.sbt) and README are included to guide execution.

examples/python, examples/scala/src/main/scala · high confidence

New Scala extension utilities for conditional option handling

A new ScalaExtensions utility object has been added to the Spark SQL utilities package, providing implicit classes that enhance standard Scala types. For Option values, it introduces ifDefined for clearer conditional execution, and for the Option companion object, it adds when and whenNot for conditional value creation, as well as sum for aggregating optional numeric values. Additionally, it provides condDo on Any to apply a partial function conditionally, improving code readability for conditional logic.

spark/src/main/scala/org/apache/spark/sql/util · high confidence

New benchmarking framework for Delta Lake workloads

This change introduces a new Python-based benchmarking framework located in \benchmarks/scripts\ to evaluate Delta Lake performance. It provides a structured way to define and run benchmarks, specifically adding support for TPC-DS and Merge workloads. The framework includes utility scripts for command execution and SSH operations, along with specification classes that configure Spark sessions with the necessary Delta extensions and Maven artifacts for running these specific benchmark types.

benchmarks/scripts · high confidence

New build and CI helper scripts for Spark, Unity Catalog, and test analytics

Added several new scripts to the project to streamline local development and CI workflows. The \build\_spark.sh\ script automates building a specific Apache Spark source reference and publishing it to the local Maven repository for Delta tests. The \setup\_unitycatalog\_main.sh\ script clones and publishes Unity Catalog artifacts to local Ivy/Maven repositories, supporting pinned SHAs, floating main canaries, and release pipelines. The \get\_spark\_version\_info.py\ script provides utilities for CI/CD to resolve Spark version metadata and compute artifact versions. Additionally, \install-buf.sh\ and \install-uv.sh\ provide SHA-verified installation of the Buf and uv tools, while \collect\_test\_durations.py\ aggregates test duration data from CI runs to update a project-level CSV file.

project/scripts · high confidence

New char/varchar length constraint and invariant checker implementation

This change introduces two new Scala files in the Delta constraints package: CharVarcharConstraint.scala and CheckDeltaInvariant.scala. CharVarcharConstraint.scala adds logic to generate constraints for checking string length in CHAR and VARCHAR columns, handling nested structures like arrays and maps. CheckDeltaInvariant.scala introduces a new expression type that validates invariants (such as NOT NULL and CHECK constraints) before writing data to Delta, including code generation for efficient evaluation and detailed error reporting when violations occur.

spark/src/main/scala/org/apache/spark/sql/delta/constraints · high confidence

New default Parquet reader and writer implementations in Delta Kernel

The Delta Kernel default module now includes a complete, new implementation for reading and writing Parquet files, replacing previous approaches. The new reader (ParquetFileReader, ParquetBatchRowReader) supports column projection, row-group level predicate pushdown, and materializes data into columnar batches using dedicated type-specific readers (ArrayColumnReader, DecimalColumnReader, MapColumnReader, etc.). The new writer (ParquetFileWriter, ParquetColumnWriters) handles writing columnar batches to Parquet files, supporting multiple output files, atomic writes, and statistics collection. This change enables more efficient and flexible Parquet I/O operations within the Delta Kernel.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/parquet · high confidence

New development tooling for code quality and Delta Connect code generation

The \dev\ directory now includes a suite of new scripts and configuration files to enforce code standards and manage generated assets. A new Python script (\check-delta-connect-codegen-python.py\) and shell script (\delta-connect-gen-protos.sh\) have been added to generate and verify the synchronization of Delta Connect Python protobuf code. Additionally, Checkstyle configurations (\connectors-checkstyle.xml\, \kernel-checkstyle.xml\) and a suppression file (\checkstyle-suppressions.xml\) are introduced to standardize Java code formatting, while a new Python linting script (\lint-python\) and configuration (\tox.ini\) enforce style checks using tools like flake8 and pycodestyle. A structured logging style checker (\spark\_structured\_logging\_style.py\) and a copyright header template (\copyrightHeader\) are also added to the development workflow.

dev · high confidence

New experimental commit APIs for catalog-managed tables

The Delta Kernel now exposes a new set of experimental APIs in the \io.delta.kernel.commit\ package to support catalog-managed tables. This includes the \CatalogCommitter\ interface, which extends the standard \Committer\ to handle catalog-specific operations like publishing ratified commits to the Delta log and managing required table properties. Supporting this are new classes such as \PublishMetadata\ for defining commit publication scope, \CommitFailedException\ with retry/conflict semantics, and \CatalogCommitterUtils\ for extracting protocol properties. These changes provide the foundational infrastructure for implementing catalog-managed transaction and commit workflows.

kernel/kernel-api/src/main/java/io/delta/kernel/commit · high confidence

New expression evaluators and infrastructure in the default kernel

The default kernel implementation now includes a comprehensive set of new expression evaluators and supporting utilities, enabling users to query data using a wider range of SQL-like operations. This change adds support for the \element\_at\ expression to look up values in \map(string, string)\ columns, the \IN\ expression for checking membership in a list of values, the \LIKE\ expression for pattern matching with optional escape characters, the \STARTS\_WITH\ expression for prefix checks, and the \SUBSTRING\ expression for extracting parts of strings. It also introduces the \partition\_value\ expression to decode partition column values into their native types (including handling ISO8601 timestamps). To support these and future expressions, the change adds an \ExpressionVisitor\ to dispatch expression trees, an \ImplicitCastExpression\ to handle automatic up-casting of numeric types (e.g., byte to int), and utility classes like \DefaultExpressionUtils\ for common comparison logic and lazy evaluation of comparator vectors.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/expressions · high confidence

New implicit conversion methods for Spark data structures

The Delta Lake Spark module now provides new implicit classes in the \org.apache.spark.sql.delta.implicits\ package to simplify working with common data structures. Users can now call \toDF()\ directly on \Seq\[AddFile\]\, \Seq\[String\]\, and \Seq\[Int\]\ to convert them into DataFrames. Additionally, \StructType\ gains a \findNestedFieldIgnoreCase\ method for case-insensitive field lookup within nested maps and arrays, and \LogicalPlan\ gains \transformAllExpressionsUp\ and \transformAllExpressionsUpWithPruning\ methods for more flexible expression transformation during query planning.

spark/src/main/scala/org/apache/spark/sql/delta/implicits · high confidence

New intent-based transaction builders and commit action handling in Delta Kernel

Delta Kernel introduces intent-based builders for table operations, specifically the \CreateTableTransactionBuilder\ (implemented by \CreateTableTransactionBuilderImpl\) and the \CommitActions\ interface (implemented by \CommitActionsImpl\). The new builder allows users to construct and execute \CREATE TABLE\ transactions with support for table properties, data layout specs (partitioning and clustering), custom committers, and retry logic, while ensuring the table does not already exist. The \CommitActions\ implementation provides a memory-efficient way to read and iterate over Delta commit log files, handling protocol validation and column stripping. These changes are accompanied by new internal utilities for data write contexts (\DataWriteContextImpl\), error handling (\DeltaErrors\, \DeltaErrorsInternal\), and log action utilities (\DeltaLogActionUtils\), establishing the foundation for intent-driven write operations in the kernel.

kernel/kernel-api/src/main/java/io/delta/kernel/internal · high confidence

New internal utility classes for Delta Kernel

The kernel's internal utility package now includes several new classes to support core Delta features: CaseInsensitiveMap for case-insensitive key lookups, Clock for testable time operations, ColumnMapping for handling logical-to-physical schema conversions and column ID/name mapping modes, DateTimeConstants for time unit conversions, DirectoryCreationUtils for managing Delta log directories (including staged commits and sidecars), DomainMetadataUtils for validating and populating domain metadata, ExpressionUtils for predicate and collation handling, FileNames for parsing Delta log file patterns and versions, GeometryUtils for parsing WKT geospatial stats, InCommitTimestampUtils for ICT enablement and extraction, InternalUtils for common data and path operations, and IntervalParserUtils for parsing Delta table interval configurations.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/util · high confidence

New metrics reporting interfaces for Delta operations

The Delta Kernel API now exposes structured metrics reporting interfaces for Snapshot, Scan, and Transaction operations. Users can access detailed performance data through new report types (SnapshotReport, ScanReport, TransactionReport) and their corresponding metrics result interfaces (SnapshotMetricsResult, ScanMetricsResult, TransactionMetricsResult). These interfaces provide granular timing information (e.g., snapshot construction duration, scan planning time, commit duration), file counts and sizes (add/remove files, total bytes), and operational metadata (table version, schema, filters, clustering columns). All reports implement a common MetricsReport interface with JSON serialization support, enabling monitoring and debugging of Delta table operations.

kernel/kernel-api/src/main/java/io/delta/kernel/metrics · high confidence

New stats collection and data skipping infrastructure for auto-compaction and variant formats

This change introduces new components in the stats package to support auto-compaction and variant data formats. It adds AutoCompactPartitionStats to track partition-level file counts for triggering auto-compaction, and VariantStatsHelper to handle fixed-precision encoding for Variant types. Additionally, it refactors data skipping logic by extracting DataSkippingPredicateBuilder and StatsProvider to standardize predicate generation, and introduces FileStatsHistogram for tracking file size distributions. These changes enable more efficient file skipping and better support for new storage formats.

spark/src/main/scala/org/apache/spark/sql/delta/stats · high confidence

New telemetry structures and logging abstraction for Delta Lake

This change introduces a new \DeltaLogging\ trait that decouples logging call sites from \DeltaLog\ by using a \DeltaLoggingProvider\ interface, allowing telemetry to be recorded without direct dependencies on the log instance. It also adds a \ScanReport\ case class to capture detailed metrics about data scans, including partition filters, data filters, and data sizes, which enhances observability into scan performance and filter usage.

spark/src/main/scala/org/apache/spark/sql/delta/metering · high confidence

New utility classes for Delta Lake internal operations

This change introduces a suite of new utility classes in the \spark/src/main/scala/org/apache/spark/sql/delta/util\ package to support internal Delta Lake operations. Key additions include \DeltaCommitFileProvider\ to resolve commit file paths for coordinated commits, \DatasetRefCache\ to manage Spark Dataset references across sessions, \BinPackingIterator\ and \BinPackingUtils\ for grouping files by size, and \DeltaEncoders\ for reusable Spark encoders. It also adds \AnalysisHelper\ for expression resolution, \Codec\ for UUID and Base85 encoding, \DeltaFileOperations\ for file system interactions, \DeltaFileSystemOptions\ for Hadoop configuration, \DeltaLogGroupingIterator\ for grouping log files, \DeltaProgressReporter\ for job status reporting, \DeltaSparkPlanUtils\ for logical plan analysis, \DeltaSqlParserUtils\ for SQL parsing, and \DeltaStatsJsonUtils\ for Variant statistics encoding.

spark/src/main/scala/org/apache/spark/sql/delta/util · high confidence

New utility classes for Unity Catalog detection and Parquet format versioning

Added two new utility classes to the Delta Spark connector: CatalogTableUtils, which provides methods to detect whether a table is managed by Unity Catalog by inspecting specific storage properties (delta.feature.catalogManaged, delta.feature.catalogOwned-preview, and the UC table ID), and ParquetFormatVersion, an enum that maps Parquet format versions (V1\_0\_0 and V2\_12\_0) to their corresponding Parquet WriterVersion and supports semantic version resolution. These utilities support the connector's ability to handle catalog-managed tables and configure Parquet output formats correctly.

spark/src/main/java/org/apache/spark/sql/delta/util · high confidence

New utility classes for resource-safe iteration and file metadata in Delta Kernel

The \io.delta.kernel.utils\ package now includes \CloseableIterator\ and \CloseableIterable\ interfaces, which extend standard Java iteration with explicit \close()\ methods to ensure resources are released safely during Delta log and data file processing. Additionally, \FileStatus\ and \DataFileStatus\ classes have been added to encapsulate file metadata, with the latter supporting optional column-level statistics, and \PartitionUtils\ provides a helper to check if a partition contains data.

kernel/kernel-api/src/main/java/io/delta/kernel/utils · high confidence

Users can now define Delta Lake sinks and Unity Catalog connections directly via Flink SQL DDL statements. The new \DeltaDynamicTableSinkFactory\ enables the \write.mode='upsert'\ option, which requires a \PRIMARY KEY\ clause in the table definition to correctly map primary key columns to internal ordinals for upsert operations. Additionally, the factory validates that only physical columns are used, rejecting computed or metadata columns to prevent ordinal shifting. A new \FlinkUnityCatalogFactory\ allows users to register a Unity Catalog catalog in Flink SQL by specifying the REST endpoint and authentication token, facilitating seamless integration with Unity Catalog-managed tables.

flink/src/main/java/io/delta/flink/sink/sql · high confidence

Support for Spark 4.0 compatibility and catalog-managed table maintenance

This change introduces support for Apache Spark 4.0 by implementing the \ClassicColumnConversions\ trait and updating \DeltaAnalysis\ to handle Spark 4.0's \LogicalRelationWithTable\. It also adds the \CatalogManagedTableMaintenanceOperation\ object, which defines the \delta.clientMaintenanceOperations\ property to control catalog-permitted maintenance operations (such as data cleanup, reorganization, and metadata cleanup) for catalog-managed tables.

spark/src/main/scala/org/apache/spark/sql/delta · high confidence

Support for removing column mapping from Delta tables

Users can now remove column mapping from Delta tables using the new RemoveColumnMappingCommand. This operation rewrites table data to eliminate column mapping metadata, updates the table schema and configuration to disable column mapping, and preserves row tracking information. The command also validates that schema field names are valid before proceeding and prevents execution on catalog-managed tables.

spark/src/main/scala/org/apache/spark/sql/delta/commands/columnmapping · high confidence

Support for type widening and portable row-index filter providers

This change introduces two key capabilities to the Delta Lake Spark connector. First, it adds support for type widening during schema evolution in the DSv2 write path, allowing columns to be automatically widened to compatible types (e.g., INT to LONG) based on configuration settings like \DELTA\_ALLOW\_AUTOMATIC\_WIDENING\, while checking for blocking dependencies such as generated columns or constraints. Second, it integrates portable manifest row-index filter providers, introducing new interfaces (\RowIndexFilter\, \RowIndexFilterProvider\) and support classes (\ColumnTypeChangeSupport\) to enable efficient filtering of rows based on index ranges, which improves scan performance by allowing filters to be loaded and applied directly from storage formats.

spark/src/main/java/org/apache/spark/sql/delta · high confidence

Support for writing log compaction files

The Delta Kernel now includes the capability to write log compaction files. A new \LogCompactionWriter\ utility has been added to the \kernel-api\ module, which compacts commit logs between specified versions into a single file. This feature allows the system to reduce the number of individual log files by generating a consolidated checkpoint-like structure for a range of versions, improving efficiency in log management.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/compaction · high confidence

Unified Delta connector introduces Kernel-backed V2 streaming and Change Data Capture support

The new spark-unified module merges the legacy V1 connector and the new Kernel-backed V2 connector into a single published artifact, providing a unified entry point via \DeltaCatalog\ and \DeltaSparkSessionExtension\. This change enables V2 streaming reads for catalog-owned tables by rewriting V1 \StreamingRelation\ plans to \StreamingRelationV2\ backed by \DeltaV2Table\, and adds support for read-time Change Data Capture (CDF) on V2 tables through the \ResolveTableChangesV2\ rule and \ChangelogSupport\ trait (gated by \DELTA\_CHANGELOG\_V2\_ENABLED\). The implementation also includes a Kernel-backed \DeltaV2OptimisticTransaction\ for writes and version-specific shims to ensure compatibility across Spark 4.0, 4.1, and 4.2. Also in this area: Delta V2 optimistic transactions now use the table manager; Delta V2 connector classes relocated to Spark namespace; Updated DeltaV2SinkSuite test configuration; Added tests for Delta V2 optimistic transaction column mapping support; Added test for variant column shredding in Delta V2 sink; and 1 more.

spark-unified · high confidence

Removals

Removal of legacy Delta Lake core implementation files

The legacy core implementation files for the Delta Lake transaction log system have been removed from the source code. This includes the deletion of Checkpoints.scala, Checksum.scala, DeltaConfig.scala, DeltaErrors.scala, DeltaFileFormat.scala, DeltaHistoryManager.scala, DeltaLog.scala, DeltaOperations.scala, and DeltaOptions.scala. These files contained the previous logic for managing checkpoints, checksums, configuration, error handling, file formats, history, logging, operations, and options.

src/main · high confidence

API

New Engine API for connector-provided file system and data handlers

The Delta Kernel now exposes a new \Engine\ interface in the \io.delta.kernel.engine\ package, allowing connectors to provide their own implementations for core operations. This interface aggregates handlers for expression evaluation (\ExpressionHandler\), JSON parsing and writing (\JsonHandler\), Parquet file reading and writing (\ParquetHandler\), and file system access (\FileSystemClient\). Connectors must implement these interfaces to support reading Delta tables, enabling custom logic for file listing, metadata retrieval, atomic file operations, and data format processing.

kernel/kernel-api/src/main/java/io/delta/kernel/engine · high confidence

Architecture

CDC reader refactored into modular base and implementation traits

The Change Data Capture (CDC) reading logic has been restructured to improve maintainability and extensibility. The previous monolithic \CDCReader\ implementation has been split into a new \CDCReaderBase\ trait containing shared functionality (such as version resolution, schema handling, and filter construction) and a \CDCReaderImpl\ trait that extends it. The \CDCReader\ object now extends \CDCReaderImpl\, and the \DeltaCDFRelation\ class has been moved into the implementation layer while \DeltaCDFRelationBase\ handles the common Spark \BaseRelation\ logic. This change isolates the core CDC reading mechanics, making it easier to support different CDC reader implementations or configurations in the future without duplicating code.

spark/src/main/scala/org/apache/spark/sql/delta/commands/cdc · high confidence

Introduces abstract traits for Delta metadata, protocol, and commit info to enable V2 connector interoperability

The Delta Lake Spark connector now provides a set of abstract traits (AbstractMetadata, AbstractProtocol, AbstractCommitInfo) in the v2.interop package. These abstractions allow the V2 connector to reuse existing V1 utilities by implementing adapters that wrap Kernel's action types, ensuring that V1 code depends on these interfaces rather than concrete implementations. This change facilitates code reuse and smoother integration between the V1 and V2 connector codebases.

spark/src/main/scala/org/apache/spark/sql/delta/v2 · high confidence

Introduces comprehensive SBT build infrastructure for multi-version support and CI automation

The build system is restructured with new SBT plugins to enable cross-compilation and publishing for multiple Spark and Flink versions, enforce Java and Scala style checks during compilation and testing, and parallelize test execution across shards and JVMs. This change also centralizes binary compatibility checks via MiMa and manages shaded Iceberg dependencies, providing a unified source of truth for version matrices and release workflows.

project · high confidence

Refactored Delta table command execution into modular command classes

The code in the \spark/src/main/scala/org/apache/spark/sql/delta/commands\ directory has been restructured to replace monolithic command implementations with a modular, trait-based architecture. This change introduces dedicated base classes and traits—such as \CloneTableBase\ for cloning logic, \CreateDeltaTableLike\ for shared table creation utilities, and \DMLWithDeletionVectorsHelper\ for deletion vector handling—which are now used by specific commands like \CloneTableCommand\, \CreateDeltaTableCommand\, and \ConvertToDeltaCommand\. This refactoring centralizes common behaviors (like catalog updates, metric reporting, and transaction handling) into reusable components, improving code maintainability and separation of concerns across Delta Lake's DDL and DML operations.

spark/src/main/scala/org/apache/spark/sql/delta/commands · high confidence

Refactored MERGE execution into modular executor traits

The MERGE command implementation has been restructured into distinct Scala traits to improve code organization and support specialized execution paths. The new \ClassicMergeExecutor\ trait handles the standard two-phase MERGE logic (finding touched files and writing changes), while \InsertOnlyMergeExecutor\ provides an optimized path for MERGE operations that only insert new data, avoiding unnecessary joins. \MergeIntoMaterializeSource\ manages the materialization of the source DataFrame to ensure deterministic reads, including retry logic for RDD block loss. \MergeOutputGeneration\ centralizes the logic for transforming MERGE clauses into output expressions, and \MergeStats\ defines the comprehensive metrics and statistics collected during the operation. This refactoring isolates specific concerns like source materialization, insert-only optimization, and output generation within the \spark/src/main/scala/org/apache/spark/sql/delta/commands/merge\ directory.

spark/src/main/scala/org/apache/spark/sql/delta/commands/merge · high confidence

Behavioural changes

72 commits (3 fixes) modifying spark/v2

A change to existing behaviour in spark/v2 — 72 commits (3 fixs), 136 files: “\[Spark\] Stream V2 commit actions without eager materialization (\#8010)”; “\[DEDUP\] Share action extraction in StreamingHelper (\#8100)”; “\[Spark\] Revert "Pass the table manager to optimistic transactions" (\#8003)”; and 69 more (refs \#8010, \#8100, \#8003, \#7942, \#7513, …).

spark/v2 · medium confidence · unverified

Add Spark 4.0–4.2 compatibility shims for catalog, expressions, and streaming APIs

This change introduces version-specific shim implementations in spark/src/main/scala-shims to keep Delta Lake compatible with Spark 4.0, 4.1, and 4.2. The shims adapt to API changes across these versions, including: DataSourceV2Relation constructor differences (5 params in 4.0 vs 6 in 4.1+), ParseException handling (separate start/stop origins in 4.0 vs single origin in 4.1+), LogKey trait vs interface differences, QualifiedColType default value representation (String in 4.0 vs DefaultValueExpression in 4.1+), View constructor parameter changes, streaming class relocation (e.g., CheckpointFileManager, StreamExecution moved to new packages in 4.1), geospatial type support (no-op in 4.0, full implementation in 4.1+), variant shredding/stats collection (no-op in 4.0, active in 4.1+), schema evolution propagation rules (no-op in 4.0/4.1, active in 4.2), and table creation overwrite behavior differentiation between V1 and V2 writers. Also in this area: Support for Apache Spark 4.3 and improved streaming and bitmap filtering.

spark/src/main/scala-shims · high confidence

Added default logging configuration for Scala examples

A new log4j2.properties file has been added to the Scala examples resources, configuring the root logger to output only error-level messages to the console. This ensures that example runs are less verbose by default, suppressing informational and debug logs that might clutter the output for users running the Delta Lake examples.

examples/scala/src/main/resources · high confidence

Added source-compatibility shims for Spark 4.2 API changes

New shim files have been introduced to maintain source compatibility with Spark 4.2, which introduced breaking API changes. The \StreamingRelationV2Shim\ provides an 8-field pattern extractor for \StreamingRelationV2\, bypassing the new 9th parameter (\sourceIdentifyingName\) added in Spark 4.2. Similarly, \CharVarcharShims\ offers extractors for \CharType\ and \VarcharType\ that expose only the length, ignoring the new \collation\ parameter introduced in Spark 4.2, ensuring Delta Lake's internal callsites continue to function without modification.

spark/src/main/scala/org/apache/spark/sql/delta/shims, spark/src/main/scala/org/apache/spark/sql/types · high confidence

Centralized Spark session extension logic for Delta connector

The Delta Spark connector now uses a new \AbstractDeltaSparkSessionExtension\ class to centralize the registration of SQL parsers, optimizer rules, and analysis rules. This change consolidates the extension logic previously scattered across V1 and V2 implementations, ensuring consistent behavior for features like implicit casting, schema evolution, and time travel preprocessing. Users benefit from a unified extension point that supports both legacy (DSv1) and modern (DSv2) connector modes without requiring changes to their existing configurations.

spark/src/main/scala/io/delta/sql · high confidence

Delta Source V1 integration and CDC streaming refinements

The Delta source implementation in this location now supports the V1 Overwrite with SaveAsTable capability, allowing users to overwrite tables using the standard Spark V1 save API while preserving table metadata. For streaming queries, the change-data capture (CDC) reader has been refined to correctly handle end offsets by skipping log entries beyond the specified limit and preserving BEGIN/END index markers for AddCDCFile commits, ensuring streams do not stall. Additionally, literal string predicates (such as startsWith, endsWith, and contains) are now preserved in Delta overwrite filters when the relevant configuration is enabled, improving filter pushdown accuracy.

spark/src/main/scala/org/apache/spark/sql/delta/sources · high confidence

Enhanced SBT launcher with proxy support, mirror fallbacks, and reduced memory usage

The SBT build launcher now supports dependency resolution through a configurable Maven proxy via the MAVEN\_PROXY\_URL environment variable, allowing users to route requests through an internal mirror. It also introduces a fallback mechanism for downloading the SBT launch JAR, prioritizing specific mirror URLs (for versions 0.13.18 and 1.5.5) or a Google-hosted Maven Central mirror before falling back to the official Maven repository. Additionally, the default memory allocation for SBT has been reduced from 4000MB to 1000MB to lower resource consumption during builds.

build · high confidence

Expanded SQL grammar for Delta Lake table management commands

The Delta SQL parser grammar has been updated to support a broader range of table management operations. Users can now use SQL syntax to drop table features (\ALTER TABLE ... DROP FEATURE\), manage clustering (\ALTER TABLE ... CLUSTER BY\ and \CLUSTER BY NONE\), and synchronize identity columns (\ALTER TABLE ... SYNC IDENTITY\). Additionally, the parser now handles \OPTIMIZE ... FULL\ for clustered tables, \REORG TABLE\ commands for purging or upgrading to Iceberg compatibility, and string coalescing in identifiers.

spark/src/main/antlr4 · high confidence

Improved error handling for empty schema writes and schema merging logic

This change introduces \EmptySchemaWriteException\ to provide specific, user-friendly error messages when attempting to write to a Delta table with no columns or all \NullType\ (void) columns, replacing generic Parquet exceptions. It also refactors \ImplicitMetadataOperation\ and \SchemaMergingUtils\ to centralize schema merging logic, explicitly handling type widening metadata recording and ensuring that generated metadata columns are stripped before schema updates. These updates ensure that schema evolution operations fail gracefully with clear diagnostics rather than cryptic internal errors.

spark/src/main/scala/org/apache/spark/sql/delta/schema · high confidence

Introduce DefaultFileSystemManagedTableOnlyCommitter with catalog-managed table protection

The kernel now includes a new \DefaultFileSystemManagedTableOnlyCommitter\ implementation that handles commit logic for file-system managed tables. This component validates protocol versions before writing the JSON commit file atomically and explicitly rejects attempts to use this committer on catalog-managed tables by throwing an error if such a protocol is detected. It also exposes a protected constructor, allowing subclasses to extend its behavior while ensuring the default instance remains restricted to file-system managed scenarios.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/commit · high confidence

The Delta Flink connector now implements the Flink Sink V2 API, replacing the previous sink implementation. This new architecture introduces a two-phase commit protocol with dedicated writer and committer components, enabling exactly-once semantics and robust checkpointing. Users can now configure the sink to operate in 'upsert' mode, which merges incoming changes into the table based on a declared primary key using a merge-on-read strategy, or stick to the default 'append' mode. The update also includes new configuration options for file rolling strategies (by size or count), schema evolution policies, and primary key definitions, along with comprehensive data type conversions between Flink and Delta structures.

flink/src/main/java/io/delta/flink/sink · high confidence

Introduces a generic FileIO interface for the Default Engine

The Default Engine now uses a new, generic \FileIO\ interface for all file operations, removing the previous direct dependency on Hadoop. This interface defines methods for listing files, resolving paths, creating directories, deleting files, and handling atomic copy operations. It also introduces \InputFile\ and \OutputFile\ abstractions for reading and writing data, along with \SeekableInputStream\ and \PositionOutputStream\ to support position-aware I/O. This change allows connectors to provide their own file system implementations by implementing the \FileIO\ interface, decoupling the engine from specific Hadoop APIs.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/engine/fileio · high confidence

New internal data structures for row and column vector handling in the Kernel API

The \kernel/kernel-api\ module introduces a suite of new internal classes to manage row and columnar data representations. \ChildVectorBasedRow\ and its subclasses (\ColumnarBatchRow\, \StructRow\) provide abstractions for accessing data from column vectors by row ID, while \GenericRow\ and \GenericColumnVector\ offer flexible implementations wrapping maps and lists of values. Additionally, \DelegateRow\ allows creating modified views of existing rows without mutation, and \SelectionColumnVector\ handles boolean selection masks. Specific state containers \ScanStateRow\ and \TransactionStateRow\ encapsulate table metadata, schemas, and protocol information for scan and transaction operations, providing utility methods to extract logical/physical schemas, partition columns, and configuration.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/data · high confidence

New internal utility classes for error handling and timestamp conversion

The default kernel implementation introduces two new internal classes: \DefaultEngineErrors\ and \DefaultKernelUtils\. \DefaultEngineErrors\ provides factory methods for specific exceptions, including support for invalid escape sequences in LIKE expressions and errors for unsupported expression evaluation. \DefaultKernelUtils\ adds utility methods for converting between Julian days and microseconds, parsing TimestampNTZ strings, and resolving column data types within a schema, supporting more robust timestamp handling and expression evaluation in the default engine.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal · high confidence

Parallelized Delta operations with thread-local context propagation

Delta Lake now executes listFrom and commitStore getCommits calls in parallel using a new thread pool infrastructure. This change introduces a custom thread pool that captures and forwards Spark thread-local properties (such as TaskContext and local properties) to worker threads, ensuring that parallel tasks run with the correct caller's SparkSession context. It also includes a NonFateSharingFuture mechanism to prevent error propagation across multiple waiting threads, improving reliability and performance for concurrent Delta operations.

spark/src/main/scala/org/apache/spark/sql/delta/util/threads · high confidence

Post-commit hooks now use CommittedTransaction and support configurable error handling

Post-commit hooks (Checkpoint, IcebergConverter, HudiConverter) now receive a CommittedTransaction instead of a generic transaction, ensuring they operate on the final committed state. Additionally, a new configuration (DELTA\_POST\_COMMIT\_HOOK\_THROW\_ON\_ERROR) allows users to control whether hook failures should abort the commit or be silently ignored, improving reliability for critical operations like UniForm metadata conversion.

spark/src/main/scala/org/apache/spark/sql/delta/hooks · high confidence

Refactor Delta LogStore to use Java-based implementations and lazy ClosableIterator

The Delta Spark storage layer has been restructured to delegate file system operations to Java-based classes from the Delta Kernel project (io.delta.storage) for S3, Azure, HDFS, and GCS, with DelegatingLogStore now routing requests to these implementations based on the file scheme. Additionally, the new ClosableIterator utility ensures that inner iterators are opened lazily only when hasNext() is called, preventing premature resource acquisition and improving efficiency during log reading.

spark/src/main/scala/org/apache/spark/sql/delta/storage · high confidence

Refactor DeltaTable execution layer to use new command-based operations

The \spark/src/main/scala/io/delta/tables/execution\ package has been restructured to decouple the Scala \DeltaTable\ API from its underlying Spark SQL command implementations. New files introduce \DeltaTableOperations\ as the central trait executing operations like \delete\, \update\, \vacuum\, \restore\, and \clone\ via specific logical plan commands (e.g., \VacuumTableCommand\, \CloneTableStatement\). This includes adding \DeltaConvert\ for converting external tables to Delta and \DeltaTableBuilderOptions\ for create/replace semantics. The \VacuumTableCommand\ specifically supports external inventory sources for garbage collection. This change standardizes how DeltaTable API calls are translated into Spark execution plans, improving modularity and enabling features like shallow cloning and inventory-based vacuuming.

spark/src/main/scala/io/delta/tables/execution · high confidence

Refactor Kernel exceptions into a dedicated hierarchy

The Delta Kernel API now provides a structured set of specific exception classes under \io.delta.kernel.exceptions\ to replace generic error handling. This change introduces a base \KernelException\ and granular types for common scenarios, including \TableNotFoundException\, \TableAlreadyExistsException\, \InvalidTableException\, and \ConcurrentWriteException\ (with subclasses like \ConcurrentTransactionException\ and \MetadataChangedException\). It also adds new exceptions for versioning and protocol issues, such as \StartVersionNotFoundException\, \CommitRangeNotFoundException\, \UnsupportedProtocolVersionException\, and \UnsupportedTableFeatureException\, along with \CommitStateUnknownException\ to handle ambiguous commit states during retries. This allows users to catch and handle specific failure modes more precisely.

kernel/kernel-api/src/main/java/io/delta/kernel/exceptions · high confidence

Refactored CONVERT TO DELTA file listing and schema inference

The internal implementation of the CONVERT TO DELTA command has been restructured to improve performance and reliability. A new \ConvertUtils\ module centralizes utility logic, while file manifest handling is split into specialized classes (\ManualListingFileManifest\, \CatalogFileManifest\) that support rebalancing file listings across tasks to prevent skewed processing during schema inference. Additionally, the \ParquetTable\ implementation now exposes \numFiles\ and \sizeInBytes\ properties, providing users with accurate metadata about the source table during the conversion process.

spark/src/main/scala/org/apache/spark/sql/delta/commands/convert · high confidence

Refactored Delta log file parsing into a typed class hierarchy

The internal log-file parsing logic in the Kernel API has been restructured from a monolithic approach into a typed class hierarchy under \io.delta.kernel.internal.files\. New classes now represent specific Delta log components—\ParsedPublishedDeltaData\ and \ParsedCatalogCommitData\ for commits, \ParsedClassicCheckpointData\, \ParsedV2CheckpointData\, and \ParsedMultiPartCheckpointData\ for checkpoints, plus \ParsedLogCompactionData\ and \ParsedChecksumData\—all extending a common \ParsedLogData\ base. This change introduces explicit type prioritization for checkpoints (V2 \> MultiPart \> Classic) and adds utility methods in \LogDataUtils\ to validate log data ordering and merge published versus ratified commits, ensuring the kernel correctly handles and prioritizes different log file types.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/files · high confidence

Refactored Delta log replay to use columnar iterators and pagination support

The log replay mechanism in the Delta Kernel has been rewritten to use a columnar, iterator-based design for improved performance and memory efficiency. This change introduces new internal classes such as \ActionsIterator\, \ActiveAddFilesIterator\, and \CreateCheckpointIterator\ to process Delta log actions (adds, removes, metadata) in batches. It also adds support for paginated scans via the \PageToken\ class, allowing large log segments to be read in chunks. Additionally, the \ConflictChecker\ has been updated to handle concurrent delete-vs-delete conflicts and row tracking metadata during conflict resolution.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay · high confidence

Refactored log replay and protocol feature handling in Delta Lake actions

The Delta Lake actions module has been refactored to improve how table state is reconstructed and how protocol features are managed. A new \LogReplay\ trait and its \InMemoryLogReplay\ implementation now handle the replay of commit history, introducing object-identity resolution for deletion vectors and the ability to retain the first back-reference for file actions during compaction. Additionally, the \TableFeatureSupport\ trait has been expanded to provide more granular control over protocol transitions, allowing explicit addition and removal of reader and writer features while ensuring legacy features are handled correctly during protocol upgrades and downgrades.

spark/src/main/scala/org/apache/spark/sql/delta/actions · high confidence

Refactored snapshot construction with intent-based builders and catalog version support

The internal snapshot building process has been restructured to use a new intent-based builder pattern (SnapshotBuilderImpl) and a dedicated factory (SnapshotFactory). This change introduces support for catalog-managed tables by allowing users to specify a maximum catalog version, which is enforced during snapshot resolution to prevent time-travel beyond allowed limits. The refactoring also improves timestamp-based time travel by correctly handling catalog commits and reading the \_last\_checkpoint file for latest queries, ensuring consistent snapshot state across different table management modes.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/table · high confidence

Refactored snapshot construction with new LogSegment and MetadataCleanup components

The snapshot loading logic in the kernel has been restructured to improve clarity and maintainability. A new \LogSegment\ class now encapsulates the validation and representation of the transaction log files (deltas, checkpoints, and compactions) required to build a snapshot, replacing previous inline logic. \SnapshotManager\ has been updated to utilize this new structure, delegating log segment generation and validation to \LogSegment\. Additionally, a new \MetadataCleanup\ class has been introduced to handle the deletion of expired delta log files based on retention policies, separating this concern from the snapshot building process. These changes support features like time-travel by timestamp and version-specific snapshot retrieval while ensuring robust validation of the Delta log state.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/snapshot · high confidence

Refactored streaming snapshot initialization for improved performance

The DeltaSourceSnapshot component now uses range partitioning and caching to optimize the initial file listing for streaming queries. By replacing the previous shuffle-based approach with \repartitionByRange\ and caching the resulting dataset, the system reduces overhead and improves the efficiency of the initial snapshot read.

spark/src/main/scala/org/apache/spark/sql/delta/files · high confidence

Route managed Delta table operations through the Unity Catalog Delta REST API

Managed Delta table operations (CREATE, REPLACE, and table loading) are now routed through the Unity Catalog Delta REST API by default, replacing the legacy Spark catalog delegation path. This change introduces a new client interface and implementation that handles staging new tables, applying catalog-supplied storage credentials, and registering table metadata, ensuring that catalog-managed properties are correctly carried forward and that table existence checks and loads utilize the UC endpoint. The feature can be disabled by setting deltaRestApi.enabled=false.

spark/src/main/scala/org/apache/spark/sql/delta/catalog · high confidence

Structured error classes for Delta concurrent modification exceptions

The Delta Spark library now exposes a dedicated set of exception classes in the \io.delta.exceptions\ package for handling concurrent modification conflicts. These include \ConcurrentWriteException\, \MetadataChangedException\, \ProtocolChangedException\, \ConcurrentAppendException\, and \ConcurrentDeleteReadException\. Each class implements the \DeltaThrowable\ interface, providing structured error codes (such as \DELTA\_CONCURRENT\_WRITE\ and \DELTA\_METADATA\_CHANGED\) and message parameters, allowing applications to programmatically detect and handle specific types of commit conflicts rather than relying on generic exception messages.

spark/src/main/scala/io/delta/exceptions · high confidence

Support for Apache Spark 4.3 and improved streaming and bitmap filtering

This update adds support for Apache Spark 4.3, including shims for the new \DeltaSqlParser\ and updated expression validation rules. It introduces a \DeltaLogStreamingBackend\ to standardize how streaming sources access table state and commit history, and enhances deletion vector filtering by allowing \ManifestBitmap\ to handle multiple bitmaps and support inverse row index filtering for more efficient data skipping. Also in this area: Preserve Variant stats during checkpointing and state reconstruction.

spark/src/main/java-shims · high confidence

Support for Delta Lake-assigned field IDs in Hive tables via metadata updates

The Hive catalog implementation has been modified to accept and apply a list of MetadataUpdates during initialization and table operations. This allows Iceberg tables managed through Hive to utilize schema and partition specifications with field IDs assigned by Delta Lake, enabling interoperability in UniForm scenarios. Additionally, the table refresh logic now gracefully handles cases where a Hive Metastore entry exists but lacks the Iceberg table type marker, allowing Delta Lake to correctly initiate table creation transactions without conflict.

icebergShaded/src/main/java/org/apache/iceberg/hive · high confidence

Unified table feature framework in Delta Kernel

The Delta Kernel now uses a structured, unified framework for managing table features, replacing the previous ad-hoc logic. This change introduces a base \TableFeature\ class and a \FeatureAutoEnabledByMetadata\ interface, allowing features like Change Data Feed, Column Mapping, and Invariants to be explicitly defined with their protocol requirements (reader/writer versions) and metadata triggers. This provides a consistent mechanism for the Kernel to determine feature support, handle legacy features, and automatically enable features based on table properties.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/tablefeatures · high confidence

Test coverage

Add integration test for Delta Uniform Hudi support; Added DefaultKernelTestUtils for test data extraction; Added Delta Kernel benchmark suite with JMH infrastructure and workload models; Added Delta log fixtures for identity columns and deletion vectors; Added GoldenTables test infrastructure for generating reference table states; Added Hive Metastore schema resources for embedded HMS tests; Added Java unit tests for Delta Lake SQL commands and deletion vector encodings; Added Spark 4.0 test shims; Added Spark 4.1 test shims for GeoSpatial and schema evolution support; Added comprehensive test suites for Delta Format Sharing; Added golden table test fixtures for decimal decoding, iterator handling, and transactional operations; Added integration tests for Flink Delta Lake features; Added integration tests for Unity Catalog commit coordinator and managed tables; Added log4j 2.x test configuration files; Added parser tests for Delta SQL commands; Added shared test coverage for Delta table refresh, caching, and self-join behaviors; Added test coverage for Delta Kernel Parquet I/O; Added test coverage for Delta Kernel checksum and checkpoint features; Added test coverage for Delta Kernel type system features; Added test coverage for Flink SQL Delta sink and Unity Catalog integration; Added test coverage for Kernel expressions and predicates; Added test coverage for commit metadata, default committer, and publish metadata validation; Added test coverage for the Flink Delta Sink implementation; Added test coverage for the new Flink Kernel-backed table implementation; Added test fixtures for deletion vectors with and without checkpoints; Added test infrastructure for Delta Connect; Added test infrastructure for Unity Catalog and Delta-Tables API integration; Added test logging configuration files for Flink; Added test logging configuration for structured logging support; Added test shims and suites for Apache Spark 4.1–4.3 compatibility; Added test shims for Spark 4.0 and 4.1+ streaming classes; Added test shims for Spark 4.0/4.1 compatibility; Added test suite for Delta Kernel table feature validation; Added test suites for Delta Kernel internal action classes; Added test suites for DeltaTable builder, forName resolution, and core table operations; Added test suites for Iceberg compatibility and column defaults validation; Added test suites for kernel expression evaluation; Added test suites for the Default Engine's core components; Added test to audit delta-iceberg JAR contents; Added test utilities for building columnar batches and rows; Added test utilities for configuring Delta's Spark extensions; Added tests and benchmarks for the Path utility class; Added tests for ActionsIterator resource cleanup and interrupt handling; Added tests for CRCInfo serialization, backward compatibility, and checksum state tracking; Added tests for CatalogCommitterUtils.extractProtocolProperties; Added tests for CloseableIterator utilities and Transaction partition handling; Added tests for ClusteringMetadataDomain serialization and column handling; Added tests for DataLayoutSpec in transaction module; Added tests for DataType JSON serialization and Type Widening logic; Added tests for Deletion Vectors Base85 codec and RoaringBitmapArray; Added tests for Delta Connect Python client exception handling and API coverage; Added tests for Delta Kernel checkpoint instance logic and checkpointer behavior; Added tests for Delta Kernel log data parsing and validation utilities; Added tests for Delta SQL extension and catalog configuration; Added tests for Delta concurrent exception handling; Added tests for Delta-to-Hudi conversion; Added tests for FileSizeHistogram utility; Added tests for Flink kernel checkpointing and deletion vector access; Added tests for Hadoop input and output file handling; Added tests for JSON serialization and deserialization utilities; Added tests for JsonMetadataDomain serialization and deserialization; Added tests for Kernel metrics reporting and collection; Added tests for Kernel metrics serialization and timing utilities; Added tests for LogStore provider and listFrom behavior; Added tests for catalog-managed log segment construction and snapshot builder validation; Added tests for catalog-managed table E2E reads and property validation; Added tests for clustering column info resolution; Added tests for data skipping utilities and stats schema handling; Added unit and end-to-end tests for the Unity Catalog kernel integration; Added unit tests for Counter and StringType in kernel-api; Added unit tests for Delta Kernel internal components; Added unit tests for Flink upsert merge strategies and scan locator; Added unit tests for Kernel exception handling and error classes; Added unit tests for LogSegment validation and metadata cleanup logic; Added unit tests for kernel internal utility classes; Expanded Unity Catalog integration test coverage for Spark; Expanded test coverage for Delta Lake Spark integration; Initial Python test suite for Delta Lake; New Python test infrastructure and configuration for Delta Lake; New concurrency testing framework for Delta Lake transactions; New test utilities and mock infrastructure for Delta Kernel; New test utilities for Delta Kernel defaults; Removed Delta test suites; Shaded Iceberg PartitionSpec with Delta field ID compatibility.

Dependencies

Configures additional Maven and SBT repository mirrors for dependency resolution

The build system now explicitly defines a prioritized list of artifact repositories in the new \build/sbt-config/repositories\ file. This configuration adds a Google-hosted mirror of Maven Central as the primary remote source to reduce load on the central repository, includes MuleSoft as a fallback, and integrates specific repositories for Typesafe, Spark packages, and Apache snapshots. This change ensures more reliable and faster dependency resolution by distributing requests across multiple mirrors and handling specific plugin repositories.

build/sbt-config · high confidence

New build manifests for benchmarks, dev tools, docs, and examples

Added build configuration files for several project areas: benchmarks (Scala 2.12.18, Spark 3.5.3, scopt 4.0.1, jackson-module-scala 2.13.1), dev tooling (mypy 1.8.0, flake8 3.9.0, black 23.12.1, grpcio \>=1.67.0, mypy-protobuf 3.3.0), documentation site (Node 22.18.0, Astro 5.15.9, Starlight 0.36.2, TypeScript 5.9.3, ESLint 9.39.1, Prettier 3.6.2), and Scala examples (Scala 2.13.17, Iceberg 1.4.1, Jackson 2.15.4, with dynamic Spark/Delta version resolution). Also added a Maven POM for kernel examples (delta-kernel 3.2.0-SNAPSHOT, Hadoop 3.3.1, Jackson 2.13.5).

(dependencies) · high confidence

Housekeeping

Added placeholder for Flink Java source directory; Initial repository structure and documentation.

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 66 → 65 (-0.8)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 62 → 62 (+0.0)
  • Architecture 87 → 87 (+0.0)
  • Maturity 71 → 75 (+3.7)
  • Readiness 62 → 59 (-3.7)
  • Security 77 → 74 (-2.8)
  • Performance 78 → 89 (+11.1)

Resolved (70)

  • ClassTooLong: DeltaV2MicroBatchStream (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.applyPerCommitCDCAdmission (cognitive 20) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.checkReadIncompatibleSchemaChangeOnStreamStartOnce (cognitive 16) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.collectAndBuildCDCIndexedFiles (cognitive 56) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.collectAndBuildCDCIndexedFiles (cyclomatic 25) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.getCommitsFromRange (cognitive 19) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.validateCommitAndDecideSkipping (cognitive 38) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • DeltaV2MicroBatchStream.validateCommitAndDecideSkipping (cyclomatic 21) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • Duplicated block (10 lines × 2) (storage/src/main/java/io/delta/storage/commit/uccommitcoordinator/UCDeltaTokenBasedRestClient.java)
  • Duplicated block (12–13 lines × 2) (icebergShaded/src/main/java/org/apache/iceberg/hive/HiveCatalog.java)
  • Duplicated block (12–14 lines × 2) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • Duplicated block (14–15 lines × 2) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/utils/PartitionUtils.java)
  • Duplicated block (23 lines × 2) (kernel/examples/kernel-examples/src/main/java/io/delta/kernel/integration/ReadIntegrationTestSuite.java)
  • Duplicated block (5 lines × 2) (kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/internal/expressions/DefaultExpressionEvaluator.java)
  • Duplicated block (5 lines × 2) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.java)
  • Duplicated block (5 lines × 2) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/utils/SparkRowToKernelRow.java)
  • Duplicated block (58 lines × 2) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/utils/KernelRowToSparkRow.java)
  • Duplicated block (6 lines × 2) (iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/DeltaToIcebergConvert.scala)
  • Duplicated block (7 lines × 2) (iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/DeltaToIcebergConvert.scala)
  • Duplicated block (7 lines × 2) (storage/src/main/java/io/delta/storage/commit/uccommitcoordinator/UCDeltaTokenBasedRestClient.java)
  • …and 50 more

New (213)

  • DeltaV2MicroBatchStream.applyPerCommitCDCAdmission (cognitive 24) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • DeltaV2MicroBatchStream.checkReadIncompatibleSchemaChangeOnStreamStartOnce (cognitive 16) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • DeltaV2MicroBatchStream.collectAndBuildCDCIndexedFiles (cognitive 61) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • DeltaV2MicroBatchStream.collectAndBuildCDCIndexedFiles (cyclomatic 24) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • DeltaV2MicroBatchStream.getCommitsFromRange (cognitive 19) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • DeltaV2MicroBatchStream.validateCommitAndDecideSkipping (cognitive 33) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • DeltaV2MicroBatchStream.validateCommitAndDecideSkipping (cyclomatic 23) (spark/v2/src/main/scala/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStream.scala)
  • Duplicated block (10–11 lines × 2) (storage/src/main/java/io/delta/storage/commit/uccommitcoordinator/UCDeltaTokenBasedRestClient.java)
  • Duplicated block (9 lines × 3) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/utils/StreamingHelper.java)
  • Duplicated block (9–11 lines × 2) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/utils/StreamingHelper.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (python/delta/connect/tables.py)
  • Edited copies of one member (3 members, 10+ corresponding lines) (spark/src/main/java/org/apache/spark/sql/delta/deletionvectors/mumbling/BitPacking.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/utils/SchemaUtils.java)
  • Edited copies of one member (3 members, 13+ corresponding lines) (spark-connect/client/src/main/scala/io/delta/connect/tables/DeltaMergeBuilder.scala)
  • Edited copies of one member (3 members, 15+ corresponding lines) (spark/src/main/scala/org/apache/spark/sql/delta/catalog/AbstractDeltaCatalog.scala)
  • Edited copies of one member (3 members, 15+ corresponding lines) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/ColumnReorderReadFunction.java)
  • Edited copies of one member (3 members, 16+ corresponding lines) (spark/src/main/scala/org/apache/spark/sql/delta/DeltaErrors.scala)
  • Edited copies of one member (3 members, 18+ corresponding lines) (spark/v2/src/main/java/org/apache/spark/sql/delta/v2/read/DeltaV2MicroBatchStreamUtils.java)
  • Edited copies of one member (3 members, 24+ corresponding lines) (docs/apis/generate_api_docs.py)
  • Edited copies of one member (3 members, 26+ corresponding lines) (python/delta/tables.py)
  • …and 193 more

Changes since last survey

  • 132 commits — 126 feature/other, 6 fixes

By area

  • spark/src — 58 commits
  • spark/v2 — 39 commits
  • kernel/kernel-api — 7 commits
  • protocol_rfcs/iceberg-v4-metadata.md — 7 commits
  • protocol_rfcs/file-type.md — 4 commits
  • spark-unified/src — 4 commits
  • docs/src — 2 commits
  • storage/src — 2 commits
  • (root) — 1 commit
  • .github/CODEOWNERS — 1 commit
  • .github/ci-requirements — 1 commit
  • .github/workflows — 1 commit
  • iceberg/src — 1 commit
  • kernel/kernel-defaults — 1 commit
  • kernel/unitycatalog — 1 commit
  • protocol_rfcs/interval-types.md — 1 commit
  • spark/unitycatalog — 1 commit

Notable commits

  • fix: [PROTOCOL] File data type: fix the schema example and three wording errors (#8088)
  • fix: [Spark] Fix a losing OPTIMIZE CHECKPOINT hiding a concurrent SetTransaction commit (#8015)
  • fix: [Spark] Revert "Convert DeltaV2Scan from Java to Scala" (#8052)
  • fix: [Spark] Revert "Pass the table manager to optimistic transactions" (#8003)
  • fix: [Spark] Revert "Read V2 single commits through LogStore" (#8133)
  • fix: fix: allow large JSON strings in the kernel default reader (#7943)
  • change: Clarify DeltaV2Table references in connector tests (#8051)
  • change: Convert UCManagedTableSnapshotManager from Java to Scala (#8140)
  • change: Count files to delete before starting the VACUUM START transaction (#7961)
  • change: Refactor Relativize path code to pass boolean param using their names (#7974)
  • change: Remove unused coordinated commits validation error classes (#8108)
  • change: Retry Vacuum commit failures (#7988)
  • change: Skip incremental checksum when a derived field is negative (#8084)
  • change: [Build] Add murali-db as project code owner (#8106)
  • change: [DEDUP] Share action extraction in StreamingHelper (#8100)
  • change: [Flink] Update Unity Catalog calls for include_browse (#8104)
  • change: [Kernel] Accept published commits returned by the catalog (#7922)
  • change: [Kernel] Add hashCode to CollationIdentifier (#7957)
  • change: [Kernel] Add relaxed CommitRange validation (#8001)
  • change: [Kernel] Expose commit range validation options in UC client (#8029)
  • …and 112 more

Architecture

  • Containers 0 added · 0 removed · contexts 0 added · 0 removed · edges 0 added · 1 removed

Removed dependency edges (was a dependency, now gone) (1)

  • benchmarks → repository

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

delta-io/delta was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 70db69e228d81c9b66c6fe0e21fc46be90b898c9 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.