delta-io/delta-sharing

This system is the Delta Sharing protocol implementation, providing a standardized way to share Delta Lake data across cloud platforms. It consists of a server component that exposes shares and tables via a REST API with support for pagination, filtering, and change data feeds, alongside client libraries for Scala/Spark and Python. The clients enable secure data access using OAuth authentication and efficient scanning via Delta Kernel, allowing users to load shared data into Pandas, Spark, or Arrow formats.

17 changes.

The project migrated from the legacy Delta Exchange implementation to the Delta Sharing protocol, extracting the client into a standalone module and renaming the Spark connector accordingly. This period involved significant refactoring of the server to use Delta Kernel, adding support for pagination, JSON predicates, and Change Data Feed, while establishing comprehensive test suites and release automation for the new architecture.

4 changes.

The Delta Sharing client was refactored to support Spark 4.0 compatibility, introducing limit pushdown optimization and enhanced network resilience. Concurrently, new Rust-based Python bindings were added to expose core Delta Kernel capabilities, including table snapshot inspection and continuous data feed support. Comprehensive test suites were implemented across the client, CLI, and new wrapper modules to ensure robustness and correct configuration handling.

The \python/dev\ directory now includes helper scripts for linting (\lint-python\), formatting (\reformat\), and testing (\pytest\), along with a \.gitignore\ and \tox.ini\ configuration. These tools allow developers to run pycodestyle, flake8, mypy, and Black checks, as well as execute unit and doctests for the \delta\_sharing\ package, with support for specifying a custom Python executable via the \PYTHON\_EXECUTABLE\ environment variable.

python/dev · high confidence

New Python examples have been added to the \examples\ directory, including quickstart scripts for loading data into Pandas and Spark DataFrames, along with a configuration file for the public Delta Sharing server. Additionally, an OAuth debugging utility (\u2m-entraid-capture-jwt-token.py\) is now available in \examples/oauth\ to simulate User-to-Machine OAuth flows for troubleshooting authentication with Entra ID or Delta Sharing OIDC endpoints.

examples · high confidence

The server implementation for Delta Sharing has been refactored to use the Delta Kernel library for reading table data, replacing the previous standalone reader. This change introduces support for JSON predicate hints, allowing clients to push down filtering conditions to the server for more efficient data skipping. The update also includes new internal components for handling Azure User Delegation SAS tokens, cloud credential vending across AWS, Azure, and GCP, and robust exception handling to provide clearer error messages to API consumers.

server/src/main/scala/io/delta/sharing · high confidence

The Delta Sharing client implementation has been moved from the Spark connector into a new, standalone \client\ module. This change introduces a dedicated \DeltaSharingClient\ interface and REST implementation, a new \DeltaSharingFileSystem\ for reading shared data via a custom Hadoop filesystem scheme, and a robust authentication system supporting both Bearer tokens and OAuth 2.0 Client Credentials with automatic token refresh. The new client also includes support for query pagination, JSON predicate filtering, and configurable HTTP proxy settings.

repository · high confidence

The server's standalone module now includes new internal components to support advanced query capabilities. A new \ColumnRange\ class enables data skipping by evaluating value ranges against file statistics, while \PartitionFilterUtils\ allows the server to evaluate Spark SQL-style predicates against partition columns to filter data files. Additionally, \DeltaDataSource\ defines constants for Change Data Feed (CDF) parameters (starting/ending version and timestamp), and \DeltaCDFErrors\ provides specific validation errors for CDF reads. These changes replace the legacy \DeltaTableHelper\ which previously handled table info and file retrieval via direct S3 signing, shifting the logic toward a more robust, filter-aware architecture.

server/src/main/scala/io/delta/standalone · high confidence

A new \delta-kernel-rust-sharing-wrapper\ module has been added, providing thin Python bindings for \delta-kernel-rust\ via Maturin and PyO3. This introduces a \delta\_sharing\_kernel\_rust\_wrapper\ library that exposes core Delta Kernel capabilities to Python, including table snapshot inspection (version retrieval), standard scan execution returning Arrow RecordBatches, and Continuous Data Feed (CDF) support via \TableChangesScan\ for querying changes between specific versions.

python/delta-kernel-rust-sharing-wrapper · high confidence

Added three executable shell scripts in the dev directory to streamline the release process: python\_release.sh handles Python connector versioning and artifact building using uv; release.sh manages the full release lifecycle including SBT builds, GPG signing, SHA256 verification, and Docker image publishing; and spark\_release.sh automates the SBT release process and transfers deployment artifacts to the Central Publisher Portal via Sonatype API.

dev · high confidence

A new configuration template for the Delta Sharing server has been added, providing a structured way to define shares, schemas, and tables across various cloud storage providers (S3, Azure, GCS, R2). This template introduces support for pagination in query operations via configurable page size limits and token TTLs, as well as the ability to evaluate V2 JSON predicate hints. It also includes settings for pre-signed URL timeouts, temporary credential validity, and table caching, allowing administrators to fine-tune server behavior and security parameters.

server/src/universal · high confidence

The Python connector now exposes a refreshed public API via \delta\sharing/\\init\\_.py\, including \load\_as\_arrow\, \load\_as\_pandas\, \load\_as\_spark\, and dedicated functions for table metadata and protocol queries. It adds support for OAuth 2.0 authentication, specifically \client\_credentials\ and private key JWT flows, allowing users to configure profiles with token endpoints and client secrets. Additionally, the connector now integrates with \delta-kernel-rust\ for efficient snapshot scanning and includes a patch for \yarl\ to correctly handle GCS pre-signed URLs.

_python/delta\sharing · high confidence

The legacy Delta Exchange server components have been removed from the codebase. This includes the deletion of the \CloudFileSigner\ trait and its S3 implementation, the \DeltaExchangeService\ which exposed the \/api/2.0/s3commit/table\ endpoints for table info, metadata, and files, and the \TestResource\ helper used for AWS credential management. Users relying on this specific server implementation for file signing and data access via the old protocol will no longer have this functionality available in this module.

server/src/main/scala/io/delta/exchange · high confidence

The Delta Sharing client has been refactored to support Spark 4.0, introducing version-specific implementations for limit pushdown optimization in \DeltaSharingLimitPushDown\ (handling distinct logical plan structures for SQL vs. Spark queries) and scan utilities in \DeltaSharingScanUtils\. The migration includes new HTTP client builders for proxy and HTTPS configuration, enhanced retry logic for transient network errors in \RandomAccessHttpInputStream\, and improved error handling with specific exception types like \DeltaSharingConnectionException\. Additionally, the client now supports partial filter evaluation via \UnsupportedOp\ for unsupported predicate sub-expressions, allowing more robust query pruning.

client/src/main · high confidence

The \protocol.proto\ definition has been significantly expanded to support the latest Delta Sharing protocol features. Key additions include pagination support for \QueryTable\ and list operations (Shares, Schemas, Tables) via \maxFiles\, \pageToken\, and dedicated page token messages. The \QueryTableRequest\ now supports querying specific table versions or timestamps, as well as version-range queries using \startingVersion\ and \endingVersion\. New metadata APIs are defined for listing and getting shares, schemas, and tables. Additional capabilities include JSON predicate hints, refresh token support for pre-signed URL renewal, historical protocol action inclusion for streaming queries, and idempotency keys for async queries. The package namespace has also been updated from \io.delta.exchange.protocol\ to \io.delta.sharing.server.protocol\.

server/src/main/protobuf · high confidence

The Delta Sharing Python client has raised its minimum supported Python version to 3.10, dropping support for earlier versions. To streamline installation and ensure reproducible builds, the project has migrated its dependency management to \uv\, introducing a \pyproject.toml\ and a locked \uv.lock\ file that pins specific versions for all dependencies, including \aiohttp\ and \aiobotocore\.

python · high confidence

The Spark connector has been renamed from Delta Exchange to Delta Sharing. This change updates the Spark DataSource registration in \META-INF/services/org.apache.spark.sql.sources.DataSourceRegister\ to point to the new \io.delta.sharing.spark.DeltaSharingDataSource\ class, replacing the old \io.delta.exchange.spark.DeltaExchangeDataSource\. Additionally, the Hadoop FileSystem service registration in \META-INF/services/org.apache.hadoop.fs.FileSystem\ is updated to use \io.delta.sharing.client.DeltaSharingFileSystem\ instead of the previous \io.delta.exchange.spark.DeltaFileSystem\. The old implementation classes in the \io.delta.exchange.spark\ package (including \CloudFileSigner\, \DeltaFileSystem\, \DeltaLogClient\, \RemoteDeltaLog\, and \TestResource\) have been removed.

spark/src/main · high confidence

The build system now allows configuring a single Maven proxy URL via the MAVEN\_PROXY\_URL environment variable to serve as the sole repository for all SBT dependencies, overriding the default committed repository list. Additionally, the SBT launcher JAR download process has been enhanced to fetch and verify a SHA1 checksum, ensuring the integrity of the downloaded binary before installation.

build · high confidence

Added integration tests for Delta Sharing Spark connector; Added server-side test infrastructure and unit tests for Delta Sharing; Added unit and integration tests for the Delta Sharing Scala client; Added unit tests for CLI URL construction, credential masking, and configuration handling; Initial test suite for Python Delta Sharing client.

The Delta Sharing Spark connector now supports building against both Spark 3.5.3 (via Scala 2.12) and the official Spark 4.0.0 release (via Scala 2.13). This change updates the build configuration to target the latest Spark version for Scala 2.13 builds, ensuring compatibility with newer Spark ecosystems while maintaining backward compatibility for existing Scala 2.12 users. Additionally, the Rust-based Python wrapper dependencies have been updated, including delta-kernel to version 0.22.0 and Arrow to 57.3.0, to support the new Python connector architecture.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 62 → 71 (+8.9)
• Rubric changed (rubric-2026.09.8 → rubric-2026.09.16) — scores are not directly comparable.

• Code Health 91 → 91 (-0.1)
• Architecture 99 → 85 (-14.1)
• Maturity 81 → 78 (-3.1)
• Readiness 71 → 76 (+5.6)
• Security 44 → 62 (+18.4)

• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Members sharing a duplicated core (4 members, 50+ identical tokens) (client/src/main/scala/io/delta/sharing/filters/JsonPredicates.scala)
• Off-boarding risk: anonymized user #1

• Duplicated block (12 lines × 2) (python/delta_sharing/protocol.py)
• Duplicated block (26–28 lines × 2) (python/delta_sharing/delta_sharing.py)
• Duplicated block (8 lines × 2) (python/delta_sharing/rest_client.py)
• Hotspot: python/delta_sharing/reader.py (python/delta_sharing/reader.py)
• Medium vulnerability: RUSTSEC-2026-0285 (python/delta-kernel-rust-sharing-wrapper/Cargo.lock)
• Members sharing a duplicated core (4 members, 50+ identical tokens) (client/src/main/scala/io/delta/sharing/filters/JsonPredicates.scala)
• Off-boarding risk: anonymized user #1
• Outdated: com.amazonaws:aws-java-sdk-bundle
• Outdated: com.azure:azure-storage-blob
• Outdated: com.fasterxml.jackson.core:jackson-core
• Outdated: com.fasterxml.jackson.core:jackson-databind
• Outdated: com.fasterxml.jackson.dataformat:jackson-dataformat-yaml
• Outdated: com.google.auth:google-auth-library-oauth2-http
• Outdated: com.google.cloud:google-cloud-storage
• Outdated: com.google.guava:guava
• Outdated: io.delta:delta-kernel-api
• Outdated: io.delta:delta-kernel-defaults
• Outdated: org.apache.hadoop:hadoop-aws
• Outdated: org.apache.hadoop:hadoop-azure
• Outdated: org.apache.hadoop:hadoop-client
• …and 5 more

• Unchanged — 0 containers · 2 contexts · 0 edges

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

delta-io/delta-sharing was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.