Skip to content
CAI
Software that uses CAICheck a score

apache/gluten

53.1

Adequate · 14 November 2026

180.9k

lines of production code

Scala

with C++, Java, C

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add Bolt backend support for Celeborn shuffle and Delta Lake operations

This change introduces the Bolt backend implementation for Apache Celeborn shuffle and Delta Lake integration. For Celeborn, it adds a columnar batch serializer and shuffle writer that offload data processing to native code via JNI wrappers, enabling high-performance columnar shuffle operations. For Delta Lake, it registers a component that injects optimization rules to offload Delta scans, projects, and filters to the columnar engine, along with test suites to validate Delta SQL operations including deletes and deletion vectors. Also in this area: 2 commits (0 fixes) modifying backends-bolt; Enable sorted writes for Iceberg tables in the Bolt backend; Removal of unused veloxToArrow metric and backend documentation reorganization.

backends-bolt · medium confidence · unverified

Add ClickHouse support for WindowGroupLimit operations

Introduces the CHWindowGroupLimitExecTransformer, enabling the ClickHouse backend to execute WindowGroupLimit plans. This new transformer handles partitioning, sorting, and ranking functions (such as ROW\_NUMBER, RANK, and DENSE\_RANK) by converting them into the appropriate Substrait WindowGroupLimit relational nodes, allowing users to utilize window group limit functionality on ClickHouse.

backends-clickhouse/src/main/scala/org/apache/gluten/execution · high confidence

Add ClickHouse-specific shuffle utilities and range partitioning support

This change introduces two new Scala files in the ClickHouse backend's shuffle utilities: CHShuffleUtil and RangePartitionerBoundsGenerator. CHShuffleUtil provides factory methods to generate columnar shuffle writers and readers, specifically handling the selection of the BlockStoreShuffleReader with bypass decompression for columnar handles. RangePartitionerBoundsGenerator implements a custom bounds generation logic for RangePartitioning, converting Spark sort orders and attributes into Substrait expressions and JSON structures to support range partitioning operations within the ClickHouse backend.

backends-clickhouse/src/main/scala/org/apache/spark/shuffle/utils · high confidence

Add Delta Lake 3.3 integration support for ClickHouse backend

This change introduces the source folder and implementation files for Delta Lake 3.3 compatibility within the ClickHouse backend. It includes a new \Delta33ShimProvider\ registration, a \ClickhouseTable\ wrapper for Delta table access, and custom SQL parsers (\GlutenCacheFilesSqlParser\, \GlutenClickhouseSqlParser\) to handle Delta-specific syntax. The update also provides a \ClickhouseOptimisticTransaction\ to manage write operations (including native MergeTree writes and deletion vectors) and a \Delta33Shims\ implementation to handle row index filters and optimized writer execution, enabling the backend to correctly read, write, and manage Delta 3.3 tables. Also in this area: Added test coverage for Delta Lake integration with ClickHouse backend.

backends-clickhouse/src-delta33 · high confidence

Add GPU plan validation and Parquet map-key pruning for Velox

This change introduces two new optimization and safety mechanisms for the Velox backend. First, it adds a CudfNodeValidationRule that validates the execution plan before running on GPU (when adaptive execution is enabled and validation is configured), ensuring plan compatibility and preventing runtime errors. Second, it implements ScanMapKeyPruning, a physical plan rule that analyzes Parquet scans to identify specific map keys accessed by filters, projections, or generators, allowing the Velox connector to read only the necessary map entries rather than decoding entire map columns, which improves scan performance for queries accessing specific map fields.

backends-velox/src/main/scala/org/apache/gluten/extension · high confidence

Add Gluten Kafka source builder and scan tests

This change introduces the \StreamKafkaSourceBuilder\ Java class and the \GlutenStreamKafkaSourceUtil\ Scala utility within the gluten-kafka module, which construct Substrait stream Kafka source nodes and handle split info generation for Kafka scans. It also adds a test suite (\GlutenKafkaScanSuite\) and logging configuration to verify that MicroBatchScanExecTransformer operations on Kafka sources do not fallback, ensuring the new builder logic is exercised in streaming scenarios.

gluten-kafka · high confidence

Add Gluten support for reading Apache Paimon tables

This change introduces the PaimonLocalFilesNode and PaimonLocalFilesBuilder classes to enable Gluten to read Apache Paimon tables. It also adds the PaimonSuite test suite, which verifies that the PaimonScanTransformer is correctly utilized for reading Paimon tables in Parquet and ORC formats, including partitioned and bucketed tables.

gluten-paimon · high confidence

Add Iceberg-specific protocol buffer definitions for read and partition handling

New protocol buffer definitions have been added to the Gluten backend to support Iceberg table operations. These include IcebergNestedField for schema structure, IcebergPartitionSpec for defining partition transforms (identity, year, month, day, hour, bucket, truncate), IcebergReadExtension for carrying column field IDs and default values during scans, and RequiredSubfieldsExtension to enable pruning of map entries and struct fields based on query access patterns.

backends-velox/src/main/resources/org · high confidence

Add QAT-accelerated Zstandard compression codec

Users can now leverage Intel QuickAssist Technology (QAT) for Zstandard (zstd) compression and decompression in addition to the existing GZIP support. This change introduces the QatZstdCodec implementation and its factory functions, enabling hardware-accelerated zstd operations for Arrow data processing when QAT hardware is available.

cpp/core/utils/qat · high confidence

Add SQL grammar for CACHE FILES and CACHE DATA commands

The ClickHouse backend now supports parsing two new SQL commands: \CACHE FILES\ and \CACHE DATA\. The \CACHE FILES\ command allows users to cache file contents with optional asynchronous execution and custom properties. The \CACHE DATA\ command enables caching data from MergeTree tables, supporting optional metadata, asynchronous execution, column selection, source specification (via file path or qualified table name), time-travel filtering (using \TIMESTAMP AS OF\ or date partition), and custom properties. These changes extend the SQL parser to recognize these new syntax structures.

backends-clickhouse/src/main/antlr4 · high confidence

Add Snowflake ID generator and in-memory filesystem abstraction for Velox backend

This change introduces a new Snowflake ID worker utility in the ClickHouse backend to generate unique IDs based on a configurable worker ID from Spark configuration. Simultaneously, it adds a Java JNI filesystem abstraction layer for the Velox backend, defining interfaces for file operations (read, write, delete, rename, etc.) and providing an in-memory implementation backed by Jimfs to mirror C++ side functionality.

backends-clickhouse/src/main/java/org/apache/gluten/utils, backends-velox/src/main/java/org/apache/gluten/fs · high confidence

Add Spark 3.5 shim layer

This release introduces a new shim layer for Apache Spark 3.5, enabling Gluten to run on this version. The shim includes a \Spark35Shims\ implementation that maps scalar, aggregate, and runtime-replaceable expressions (such as \mask\, \array\_insert\, \luhn\_check\, and \regexp\_instr\) to their native Velox equivalents. It also provides version-specific adapters for file scanning (\FileSourceScanExecShim\, \PartitionedFileUtilShim\), query plan tree generation (\GenerateTreeStringShim\), and shuffle operations (\ShuffleUtils\, \SparkSortShuffleWriterUtil\), ensuring compatibility with Spark 3.5's internal APIs while maintaining support for columnar execution.

shims/spark34, shims/spark35, shims/spark40 · high confidence

Add Velox Delta Lake component with configurable offload rules

Introduces the Velox Delta component, enabling native offloading of Delta Lake operations to the Velox backend. This includes support for Change Data Feed (CDF) scans, deletion-vector scans, and native writes. Users can control specific behaviors via new experimental configuration options: \spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite\ (disabled by default), \spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan\ (enabled by default, allowing an opt-out for DML row-index scans), and \spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan\ (enabled by default).

backends-velox/src-delta/main · high confidence

Add Velox-Paimon integration component

Introduces the VeloxPaimonComponent, which enables Gluten to offload Paimon table scans to the Velox backend. The component includes a runtime compatibility check that verifies the presence of Paimon Spark session extensions before activation, and registers the necessary transformation rules to handle Paimon scan operations within the Velox execution engine.

backends-velox/src-hudi/main, backends-velox/src-paimon · high confidence

Add Velox-specific SQL lifecycle listener and hash join utility

This change introduces two new Scala files for the Velox backend: a \VeloxGlutenSQLAppStatusListener\ that monitors Spark SQL execution start/end and executor removal events to notify the Gluten driver endpoint, and a \SparkHashJoinUtils\ object providing utilities for hash join key handling, specifically \canRewriteAsLongType\ and \getOriginalKeysFromPacked\ to support LongHashRelation identification.

backends-velox/src/main/scala/org/apache/spark/listener, backends-velox/src/main/scala/org/apache/spark/sql/execution/joins · high confidence

Add VeloxDataSourceJniWrapper for native data source operations

A new Java wrapper class, VeloxDataSourceJniWrapper, has been added to the Gluten Velox backend. This class provides the interface for native data source operations, including initialization, schema inspection, batch writing, and block splitting, by bridging Java calls to the underlying C++ JNI implementation.

backends-velox/src/main/java/org/apache/gluten/datasource · high confidence

Add native UDF/UDAF example library for Velox integration

The cpp/velox/udf/examples directory now provides a buildable shared library (myudf, myudaf) containing reference implementations of native User-Defined Functions and Aggregates for Velox. This includes a Hive-compatible string-concatenation UDF (UDFStringString) and a custom average aggregation (MyAvg) that demonstrates intermediate state handling, alongside the UdfCommon.h header defining the Gluten registerer interfaces used to expose these functions.

cpp/velox/udf/examples · high confidence

Added 128-bit (INT128/HUGEINT) support to TypeAwareCompress for shuffle

The TypeAwareCompressCodec in the TAC utility layer now supports column-wise compression for 128-bit integer types (INT128/HUGEINT/DECIMAL128) in addition to the existing 64-bit support. This is implemented via a new FForCodec that handles 128-bit values by splitting them into high and low 64-bit halves and applying 4-lane FFOR compression to each sub-stream. Users can now benefit from efficient compression of large decimal and 128-bit integer columns during shuffle operations.

cpp/core/utils/tac · high confidence

Added JNI wrapper for GPU plan validation

A new Java class, VeloxCudfPlanValidatorJniWrapper, has been added to provide a native interface for validating execution plans on the GPU. This wrapper exposes a static native method, validate, which accepts a byte array representing the plan, enabling the Gluten backend to verify plan correctness before execution.

backends-velox/src/main/java/org/apache/gluten/cudf · high confidence

Added JNI wrapper for enhanced features detection

A new Java class, ConfigJniWrapper, has been introduced in the Gluten configuration package to expose a native method, isEnhancedFeaturesEnabled(). This change enables the Java layer to query the underlying native code regarding the status of enhanced features, facilitating the activation of new capabilities within the Velox backend.

backends-velox/src/main/java/org/apache/gluten/config · high confidence

Added Velox Iceberg component registration file

A new component file for the Velox Iceberg integration has been added to the resources directory. This file registers the component within the Gluten framework's component discovery mechanism, enabling the system to identify and load the Velox Iceberg backend capabilities.

backends-velox/src-iceberg/main/resources · high confidence

Added Velox memory profiler JNI bindings

A new Java class, VeloxMemoryProfiler, has been added to the Gluten monitor package to provide JNI bindings for controlling the native memory profiler. This class exposes methods to start, dump, and stop the profiler, which utilizes jemalloc for memory profiling. To use this feature, users must build Gluten with jemalloc stats enabled and configure the environment to preload the jemalloc library and set specific malloc configuration options.

backends-velox/src/main/java/org/apache/gluten/monitor · high confidence

Added Velox-specific data source utility for schema inspection

A new \VeloxDataSourceUtil\ object has been added to the \backends-velox\ module to provide utilities for reading schemas from input files using the Velox backend. This utility leverages the Velox JNI wrapper to inspect file schemas via Arrow, enabling the backend to correctly handle schema detection for data sources.

backends-velox/src/main/scala/org/apache/gluten/datasource · high confidence

Added build/mvn and build/mvnd wrapper scripts for consistent builds

Developers can now use the new \build/mvn\ and \build/mvnd\ scripts to invoke Maven and Maven Daemon (mvnd) with standardized JVM settings and automatic dependency management. The \build/mvn\ script ensures the correct Maven version (read from \pom.xml\) is available by downloading it if necessary, while \build/mvnd\ installs and configures mvnd version 1.0.3 with JVM arguments tuned for Scala compilation workloads (e.g., increased heap and code cache size). These wrappers provide a consistent build environment across different local setups.

build · high confidence

Added support for tracking unsupported Gluten operators

The qualification tool now includes a new \NotSupported\ class that allows the system to record and categorize reasons why specific operators or nodes are not supported by Gluten. This change enables the tool to provide users with detailed, categorized feedback on compatibility issues, helping them understand which parts of their workload may not be accelerated.

qualification-tool · high confidence

ClickHouse backend adds Celeborn shuffle integration

The ClickHouse backend now includes native support for the Celeborn shuffle manager. This change introduces the \CHCelebornColumnarBatchSerializer\ and \CHCelebornColumnarShuffleWriter\ components, which handle columnar data serialization and writing to Celeborn, along with the necessary service provider files to register these factories with the Gluten shuffle infrastructure.

backends-clickhouse/src-celeborn/main · high confidence

ClickHouse backend adds expression transformer utility

The ClickHouse backend now includes an ExpressionUtil object that dynamically loads and instantiates custom expression transformers based on configuration. This allows the backend to extend its expression handling capabilities by specifying a custom class name, falling back to a default transformer if the specified class is unavailable or invalid.

backends-clickhouse/src/main/scala/org/apache/spark/sql/utils · high confidence

ClickHouse backend adds wrapper for hash partitioning with non-field expressions

The ClickHouse Spark backend now supports shuffle operations where hash expressions include non-field expressions. A new \HashPartitioningWrapper\ class preserves original hash expressions while allowing new expressions to be used, ensuring correct partitioning behavior in these specific scenarios.

backends-clickhouse/src/main/scala/org/apache/spark/shuffle · high confidence

ClickHouse backend component registration via service files

The ClickHouse backend is now registered with the Gluten component discovery system and Spark's data source registry. A new component file defines the CHBackend, and a service provider file registers the ClickHouseDataSource, enabling the backend to be automatically discovered and used as a data source.

backends-clickhouse/src/main/resources/META-INF · high confidence

Enable ClickHouse Kafka scan offloading

The ClickHouse backend now supports offloading Kafka scan operations. This is achieved by introducing the CHKafkaComponent, which registers the OffloadKafkaScan rule, allowing Kafka data ingestion to be processed by the ClickHouse engine rather than the default Spark execution.

backends-clickhouse/src-kafka/main · high confidence

Expanded native metrics reporting for Velox operators

The Velox backend now exposes a significantly broader set of execution metrics for scan, join, aggregation, and transformation operators. New metrics include storage-level details (storageReads, storageReadBytes, localReadBytes, ramReadBytes, pageLoadTime, dataSourceAddSplitTime, dataSourceReadTime), aggregation internals (abandonedPartialAggregationRows, toIntermediateFastPathCalls, extraction/rowConstruction CPU and wall time), and join-specific data (bloom filter statistics, dynamic filter counts, post-projection timing). This provides deeper visibility into query performance and resource usage.

backends-velox/src/main/scala/org/apache/gluten/metrics · high confidence

GPU concurrency control and plan validation for cuDF operations

This change introduces two new capabilities in the cuDF backend: a configurable GPU task concurrency limiter and a pre-execution plan validator. The new GpuLock mechanism allows users to control the maximum number of concurrent GPU tasks via \configureGpuTaskConcurrency\, preventing resource exhaustion by serializing or limiting parallel GPU execution. Additionally, the new CudfPlanValidator checks the execution plan before running it, ensuring that only supported cuDF operators are present (excluding TableScan and ValueStream), and cancels the task if unsupported operators are detected, thereby failing fast with a clear log message rather than encountering runtime errors during execution.

cpp/velox/cudf · high confidence

The Gluten Flink planner now executes stream operations—including Calc, Deduplicate, Exchange, Group Aggregate, and Window Aggregate—by translating them into Velox plan nodes (such as Project, Filter, LocalPartition, and Aggregation) and running them via Gluten-specific operators like GlutenOneInputOperator. This change shifts these core stream transformations from Flink's default row-based processing to a columnar execution engine, enabling users to benefit from Velox's optimized processing for these common stream patterns.

gluten-flink/planner · high confidence

Gluten SQL execution history is now available in the Spark History Server UI

Users can now view Gluten SQL and DataFrame execution details in the Spark History Server. This change introduces a new UI tab labeled "Gluten SQL / DataFrame" which lists past executions, showing metrics such as the number of Gluten nodes, fallback nodes, and fallback reasons. It also displays Gluten build information in a collapsible section, allowing users to audit and debug Gluten-accelerated queries after the application has finished running.

gluten-ui · high confidence

This change introduces the initial Gluten for Flink module, providing the foundational structure for an experimental integration of the Velox backend with Apache Flink. It includes comprehensive documentation in \docs/Flink.md\ detailing supported versions (Flink 1.19.3, JDK 11/17), build instructions, and usage examples, alongside a troubleshooting guide. The module also adds build configuration files such as \.scalafmt.conf\ for code formatting, \.tool-versions\ specifying Zulu JDK 17, and a patch file to fix Velox4j dependency resolution during the build process.

gluten-flink · high confidence

Initial support for Spark 4.1 shim layer

This change introduces the initial shim layer for Apache Spark 4.1, enabling the Gluten columnar engine to operate on this new Spark version. The implementation includes the \Spark41Shims\ class to map expressions and execution nodes, service provider registration for the shim, and compatibility adapters for internal Spark APIs that have changed in version 4.0 and 4.1 (such as \WritableColumnVector\, \ArrowColumnarArray\, and \ClassicDataset\).

shims/spark41 · high confidence

Introduce Bolt backend with Iceberg write and execution support

This change adds a new 'Bolt' backend to the Gluten project, providing the foundational execution components and Iceberg write integration for this backend. It introduces specific execution transformers for Iceberg operations (Append, Replace, Overwrite, WriteToDataSourceV2) and a dedicated data writer for columnar batches. Additionally, it implements core execution operators required for the Bolt backend, including BroadcastNestedLoopJoin, batch resizing, and limit/tail collection, along with the necessary columnar-to-row and row-to-columnar transitions.

repository · high confidence

Introduce CHDatasourceJniWrapper for native ClickHouse write operations

Added CHDatasourceJniWrapper, a new JNI wrapper class that exposes native methods for writing data to ClickHouse MergeTree tables and general file paths. This component enables the backend to handle native write operations by creating specific writers (MergeTreeWriter and FileWriter) via Substrait WriteRel protocols, and provides utilities for splitting data blocks by partition and bucket on the driver side before sending them to the native layer.

backends-clickhouse/src/main/java/org/apache/spark/sql/execution/datasources · high confidence

Introduce ClickHouse Iceberg component with runtime compatibility checks

The ClickHouse backend now registers a dedicated Iceberg component that automatically enables Iceberg scan offloading when the Iceberg Spark extensions are present in the runtime classpath. This change adds a new component definition and its corresponding service provider configuration, ensuring that Iceberg-specific optimizations are only activated if the required Iceberg libraries are available, thereby preventing runtime errors in environments where Iceberg is not installed.

backends-clickhouse/src-iceberg/main · high confidence

Introduce ClickHouse backend API layer

This change adds the initial API implementation for the ClickHouse backend in Gluten. It introduces the core backend registration class (CHBackend), configuration management (CHConfig), and the full set of backend-specific APIs including iterator handling, metrics collection, rule injection, and listener lifecycle management. This establishes the foundation for executing Spark queries against ClickHouse.

backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi · high confidence

Introduce ClickHouse configuration management for MergeTree data sources

Added ClickHouseConfig.scala to centralize configuration handling for the ClickHouse backend's MergeTree data source. This component normalizes properties for table creation and reading, enforcing defaults such as the 'mergetree' engine, '\_delta\_log' metadata directory, and 'default' storage policy. It also integrates bucket scan support by mapping Spark BucketSpec details (numBuckets, bucketColumnNames, orderByKey) into the ClickHouse configuration map, and provides utilities to filter and extract ClickHouse-specific options from DataFrameReader/Writer properties.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/datasources/v2/clickhouse · high confidence

Introduce ClickHouseTableV2Base trait for Delta-Catalog integration

A new base trait, ClickHouseTableV2Base, has been added to the Delta catalog layer to unify how ClickHouse-backed Delta tables resolve metadata. This trait implements the TablePropertiesReader interface and provides standardized logic for deriving the database name and table name from either the Delta catalog metadata or the underlying file path, ensuring consistent naming behavior for both table-based and path-based operations. It also exposes the Delta snapshot's partition columns and normalizes bucket specifications using the appropriate resolver, laying the groundwork for more robust table creation and configuration handling.

backends-clickhouse/src/main/scala/org/apache/spark/sql/delta/catalog · high confidence

Introduce Delta Lake integration component for the ClickHouse backend

This change adds the CHDeltaComponent, which enables the ClickHouse backend to offload Delta Lake operations (such as filtering and projection) to the columnar engine. It introduces a shim layer (DeltaShimLoader, DeltaShims, DeltaShimProvider) to handle version-specific Delta Lake APIs and runtime compatibility checks, allowing users to query Delta Lake tables with improved performance via the ClickHouse backend.

backends-clickhouse/src-delta · high confidence

Introduce Gluten function overlay with Spark-compatible get\_map\_value and round implementations

This change adds a new function overlay mechanism in Gluten that allows Spark-specific function implementations to override or supplement Velox's built-in functions without requiring upstream Velox changes. Specifically, it introduces a Spark-compatible \get\_map\_value\ function that enables map-key pruning during Parquet scans by supporting pushdown, and a \round\ function that adheres to Spark's rounding semantics for negative decimals and floating-point values. These functions are registered after Velox's default implementations, ensuring they take precedence when invoked via Spark SQL.

cpp/velox/operators/functions/overlay · high confidence

This change adds the core runtime classes required to integrate Gluten with Apache Flink's DataStream API. It introduces StreamGraphTranslator to intercept and translate Flink StreamGraphs into JobGraphs via the OffloadedJobGraphGenerator, enabling Gluten to take over execution. Additionally, it provides GlutenOutputCollector to manage record output and watermark propagation within Gluten tasks, GlutenStatefulRecordSerializer to handle serialization of stateful records (RowVectors) for checkpointing, and ArrowVectorAccessor/Writer classes to facilitate conversion between Flink's generic row data and Arrow/Velox vectorized formats.

gluten-flink-runtime · high confidence

Introduce HashTableBuilder and HashTableSerializer for hash join operations

Added the HashTableBuilder component to manage the construction of hash tables for join operations, including configuration for join types, null awareness, filtering, and memory usage tracking. Introduced the HashTableSerializer to provide serialization and deserialization capabilities for Velox hash tables, enabling the persistence and transfer of hash table state.

cpp/velox/operators/hashjoin · high confidence

Introduce MergeTree-specific Delta Lake integration components for ClickHouse

This change adds new Scala files in the ClickHouse backend's Delta adapter layer to support the MergeTree storage engine. Specifically, it introduces a \DeltaAdapterTrait\ to abstract Delta snapshot and scan operations, \MergeTreeConf\ to define backend-specific configuration options (such as UUID assignment and compaction settings), \MergeTreeFileFormat\ to handle data writing and metadata extraction for MergeTree tables, and \MergeTreePartitionUtils\ to parse partition directories. These components provide the foundational infrastructure required for the ClickHouse backend to interact with Delta Lake tables using MergeTree semantics.

backends-clickhouse/src/main/scala/org/apache/spark/sql/delta · high confidence

Introduce MergeTreeDelayedCommitProtocol for ClickHouse writes

A new \MergeTreeFileCommitProtocol\ trait is added to handle file commit operations for ClickHouse MergeTree tables. This implementation manages task setup by registering thread groups and configuring native write settings, generates unique part prefixes for temporary files based on partition and bucket IDs, and commits task results by aggregating write metrics into file statuses. This enables the backend to correctly stage and commit data parts to ClickHouse during Spark write operations.

backends-clickhouse/src/main/scala/org/apache/spark/sql/delta/files · high confidence

Introduce OS-specific shared library loaders in the SPI layer

The Velox backend now includes a new \SharedLibraryLoader\ SPI interface and concrete implementations for various operating systems (CentOS 7/8/9, Debian 11/12, Ubuntu 20.04/22.04, openEuler 24.03, and macOS). These loaders automatically detect the user's OS and load the required native shared libraries (such as Boost, ICU, glog, and Thrift) with the correct versioned filenames, ensuring compatibility across different Linux distributions and macOS. This change also removes the old \RoundOperatorTransformer\ from this location, as it was renamed and moved to this new SPI package.

backends-velox/src/main/scala/org/apache/gluten/spi · high confidence

Introduce RSS-based shuffle partition writer

Added the RssPartitionWriter component, which enables writing shuffle data to a Remote Shuffle Service (RSS) client. This writer supports both hash and sort-based eviction strategies, handling data serialization, optional compression, and metric tracking (such as compression time and bytes evicted) before pushing complete row batches to the RSS client.

cpp/core/shuffle/rss · high confidence

Introduce Velox backend API implementation

The Velox backend is now available as a first-class component in Apache Gluten. This change introduces the core API implementations (including backend registration, iterator handling, rule injection, metrics, and Spark plan execution) that enable Gluten to offload Spark query execution to the Velox engine, providing users with a high-performance columnar execution path.

backends-velox/src/main/scala/org/apache/gluten/backendsapi · high confidence

Introduce Velox backend RPC endpoints for driver-executor communication

This change adds the RPC infrastructure for the Velox backend, introducing \GlutenDriverEndpoint\ and \GlutenExecutorEndpoint\ classes along with their communication constants. The driver endpoint manages executor registration and tracks execution resources using a Caffeine cache, while the executor endpoint handles the initial handshake and receives cleanup instructions to invalidate broadcast build-side caches. This enables the Velox backend to coordinate lifecycle events and resource management between the Spark driver and executors.

backends-velox/src/main/scala/org/apache/spark/rpc · high confidence

Introduce Velox-backed Celeborn columnar shuffle writer and reader

The Velox backend now provides native columnar shuffle support for Celeborn. This change adds the \VeloxCelebornColumnarShuffleWriter\ and \CelebornColumnarBatchSerializer\ components, which leverage JNI wrappers (\ShuffleWriterJniWrapper\, \CelebornPartitionWriterJniWrapper\) to perform shuffle writes and reads using Velox's native execution engine. This replaces or augments the previous Scala-only implementation, enabling columnar data transfer between Spark tasks via Celeborn with improved performance and memory management through native buffers and compression.

backends-velox/src-celeborn/main/scala · high confidence

Introduce Velox-based Uniffle columnar shuffle writer

Adds a new columnar shuffle writer implementation for Uniffle that leverages the Velox native engine. The change introduces a JNI wrapper (UnifflePartitionWriterJniWrapper) and a Java/Scala writer (VeloxUniffleColumnarShuffleWriter) that offloads shuffle partitioning and data pushing to native code, supporting both hash and sort modes with configurable compression and buffer sizes. A dedicated UniffleShuffleManager intercepts columnar shuffle dependencies to route them to this new writer, enabling columnar shuffle execution via Uniffle with improved memory management and native performance.

backends-velox/src-uniffle · high confidence

Introduce Velox-based columnar shuffle reader and writer

This change adds the \ColumnarShuffleReader\ and \ColumnarShuffleWriter\ classes, along with \VeloxShuffleUtils\, to the Velox backend. The writer now handles both hash and sort shuffle modes via native JNI wrappers, supporting features like dictionary encoding and type-aware compression. The reader dispatches between batch deserialization for Gluten's columnar serializer and per-stream fallback for other serializers (such as Celeborn), preventing ClassCastExceptions during fallback scenarios. These components replace the previous shuffle implementation in this package, enabling end-to-end columnar shuffle execution for Velox.

backends-velox/src/main/scala/org/apache/spark/shuffle · high confidence

Introduce async multipart upload support for S3 writes in Gluten

The Gluten S3 filesystem implementation now supports asynchronous multipart uploads for writing data to S3. This change introduces the \GlutenS3FileSystem\ component, which manages a dedicated thread pool for handling upload parts concurrently. Users can configure this behavior via the \part-upload-async\ (legacy: \uploadPartAsync\) setting to enable async uploads, and tune performance using \max-concurrent-upload-num\ and \upload-threads\ (defaulting to 4 and 16 threads respectively). The implementation also includes specific retry strategy handling (standard, adaptive, legacy) and payload signing policy inference for S3 requests.

cpp/velox/filesystem · high confidence

Introduce columnar data writer factories and Substrait expression node infrastructure

This change adds the \ColumnarBatchDataWriterFactory\ and \ColumnarStreamingDataWriterFactory\ interfaces, along with their implementations (\ColumnarMicroBatchWriterFactory\), to support native columnar writes in micro-batch and streaming modes. It also introduces a comprehensive set of Substrait expression node classes (e.g., \CastNode\, \IfThenNode\, \SelectionNode\, and various literal nodes) to handle expression conversion, and adds \GlutenNotSupportException\ and \GlutenExceptionUtil\ to manage unsupported function fallbacks.

gluten-substrait · high confidence

Introduce columnar execution path for Arrow Python UDFs

Adds a new \ColumnarArrowEvalPythonExec\ execution node and its associated \ColumnarArrowPythonRunner\ to enable columnar data transfer for Python UDFs via Arrow. This change allows Spark to pass data between the JVM and Python workers in a columnar format rather than row-based, which is expected to improve performance for vectorized Python operations. The implementation includes version-specific logic to handle configuration and schema passing differences between Spark versions prior to 4.2 and 4.2+, ensuring compatibility across supported Spark releases.

backends-velox/src/main/scala/org/apache/spark/api · high confidence

Introduce columnar shuffle dictionary encoding support

The shuffle subsystem now supports dictionary encoding for hash-based shuffles. This location introduces the core infrastructure for this capability: a new \Dictionary\ module with a factory pattern for creating \ShuffleDictionaryWriter\ instances, and a \BlockType\ enum that defines new wire-format markers (\kDictionary\ and \kDictionaryPayload\). The \LocalPartitionWriter\ and \Payload\ classes have been updated to integrate this dictionary writer, allowing columnar data to be encoded with dictionaries during the shuffle write process to improve compression and reduce memory usage.

cpp/core/shuffle · high confidence

Introduce gluten-core module with native build and symbol isolation

The cpp/core directory now contains the new gluten-core module, including a CMakeLists.txt that builds the shared libgluten library from core C++ sources (compute, memory, shuffle, jni, utils). To prevent runtime crashes caused by symbol collisions between the bundled libstdc++ and the system C++ runtime, the build now applies linker options (--version-script and --exclude-libs) to hide internal static dependency symbols on non-Darwin platforms. The module also defines the ColumnarBatchWriter interface for dumping columnar batches to Parquet and provides a symbols.map file to control symbol visibility for protobuf, substrait, glog, and gflags.

cpp/core · high confidence

Introduce gluten-it CLI with structured benchmarking and parameterized testing

The gluten-it integration testing tool now provides a structured CLI (built with picocli) that supports multiple subcommands: \data-gen-only\ for data generation, \queries\ for running SQL queries with optional SQL metrics collection (e.g., execution-time), \queries-compare\ for comparing results against a baseline preset, \parameterized\ for running queries across multiple configuration dimensions, and \spark-shell\ for interactive use. Users can configure run modes (local or local-cluster with auto/manual resource settings), select presets (vanilla, velox, velox-with-celeborn, velox-with-uniffle), control data generation strategies, and enable features like the Spark history server UI. The tool also includes ClickBench SQL queries and a history server plugin for better observability.

tools/gluten-it/common · high confidence

Introduce native Delta bitmap aggregation support in the Velox backend

The cpp/velox module now includes a new native aggregation function, \bitmapaggregator\, which leverages Roaring bitmaps to efficiently aggregate row indices for Delta Lake deletion vectors. This change adds the implementation files (DeltaBitmapAggregator.cc/h), registers the function within the Velox backend's operator function registry, and updates the CMake build configuration and symbol map to include the new code and expose the necessary JNI symbols.

cpp/velox · high confidence

Introduce native Iceberg read and write offload capabilities

This change adds the core components for offloading Iceberg table scans and writes to the native Gluten backend. For reads, it introduces a new Iceberg-specific local files node that propagates delete file bounds, field IDs, and initial default values to the backend, while also handling fallback for scans using vended credentials. For writes, it adds configuration switches to enable native execution, a validation layer to check for unsupported data types and compression codecs, and utility classes to serialize partition data and metrics for the write path.

gluten-iceberg · high confidence

Introduce native Parquet file reading with stream and buffered modes

Added new C++ components in the reader operator directory to support reading Parquet files directly via the Velox ParquetReader. This introduces a factory method in FileReaderIterator to select between streaming and buffered reading modes based on file extension, and implements ParquetReaderIterator with two concrete variants: ParquetStreamReaderIterator for on-demand row batch retrieval and ParquetBufferedReaderIterator for pre-loading batches into memory. These changes provide the underlying file-reading capability for Parquet data sources within the Gluten execution engine. Also in this area: Support for Kafka reads and robustness improvements in benchmarks and GPU streams.

cpp/velox/operators/reader · high confidence

Introduce native Velox Parquet writer with configurable block settings

The Velox backend now provides a native implementation for writing Parquet files via new \VeloxParquetWriterInjects\ and \VeloxFormatWriterInjects\ classes. This change enables users to control native write parameters such as block size, block rows, compression codec, legacy format mode, and gzip window size through Spark configuration options. It also ensures that HDFS parent directories are automatically created when writing to HDFS paths and passes the session local timezone to the native writer for correct timestamp handling.

backends-velox/src/main/scala/org/apache/spark/sql/execution/datasources · high confidence

Introduce native columnar batch serializer for shuffle reads

Added \ColumnarBatchSerializer\ and \ColumnarBatchSerializerInstance\ to the Velox backend, enabling native deserialization of shuffle data via JNI. This change allows users to leverage Velox's optimized shuffle reader capabilities, including support for configurable compression codecs, stream merging, and GPU async reading, by replacing the previous serialization path with a native implementation that handles schema export, buffer allocation, and metric collection directly.

backends-velox/src/main/scala/org/apache/gluten/vectorized · high confidence

Introduce new aggregate functions and update ClickHouse backend version

The ClickHouse backend now includes new aggregate functions: \bitmapaggregator\ (DVRoaringBitmap), \uniqHLLPP\ (HyperLogLog++), and support for partial merge and group bloom filters. The underlying ClickHouse version has been updated to commit 54c5bf9a97b on the rebase\_ch/20260425-25.12.10.7 branch. Additionally, a \.clang-format\ configuration file has been added to standardize C++ code formatting.

cpp-ch · high confidence

Introduce off-heap UnsafeColumnarBuildSideRelation for broadcast hash joins

Added UnsafeColumnarBuildSideRelation, a new broadcast relation implementation that stores build-side data in off-heap memory to prevent on-heap OutOfMemory errors. This component supports configurable parallel build threads and offload settings, integrating with the Velox backend's native hash join builder to construct hash tables directly from off-heap batches, thereby improving memory safety and performance for large broadcast joins.

backends-velox/src/main/scala/org/apache/spark/sql/execution/unsafe · high confidence

Introduces ClickHouse-specific off-heap memory tracking per task

The ClickHouse backend now registers a dedicated memory consumer (CHThreadGroup) for each Spark task to track and report peak off-heap execution memory. This new component integrates with Spark's task resource lifecycle, ensuring that memory metrics are accurately captured and reported as part of the task's execution metrics.

backends-clickhouse/src/main/java/org/apache/gluten/memory · high confidence

This change adds the Gluten Flink runtime module, providing the core infrastructure to offload Flink streaming operators to the Velox engine. It introduces a JobGraph transformation layer (OffloadedJobGraphGenerator) that identifies offloadable operators and splits operator chains into slices, along with a set of Gluten-specific stream operators (GlutenAbstractStreamOperator, GlutenOneInputOperatorFactory, etc.) that wrap Velox execution. The runtime also includes custom network input handling to support WatermarkStatus propagation and implements native checkpoint snapshot and restore for source operators, enabling end-to-end state management for offloaded tasks.

gluten-flink/runtime · high confidence

Adds a new \PlannerModule\ class in the Flink loader that overrides default class loading behavior to prioritize Gluten classes. The module loads both the standard Flink table planner JAR and the \gluten-flink-planner.jar\, configuring a custom class loader to ensure Gluten packages (e.g., \org.apache.gluten\) are resolved from the component classpath while specific owner packages (including \com.fasterxml.jackson\) are delegated to the owner classloader. This enables Gluten to intercept and replace Flink planner components during execution.

gluten-flink/loader · high confidence

Introduces MergeTree storage metadata and configuration handling

Adds a new \StorageMeta\ object and \TablePropertiesReader\ trait to centralize MergeTree table configuration. This change enables the system to read storage properties (such as database, table name, snapshot ID, and various index keys) from Delta Lake metadata, while also implementing case-insensitive column name mapping and validation for partition, sort, and primary keys.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/datasources/mergetree · high confidence

Introduces driver/executor RPC endpoints for the ClickHouse backend

This change adds the RPC infrastructure required for the ClickHouse backend to communicate with Spark executors. It introduces \GlutenDriverEndpoint\ and \GlutenExecutorEndpoint\ to handle executor registration, lifecycle events, and resource cleanup, along with a \CHGlutenSQLAppStatusListener\ to bridge Spark SQL execution events to the backend. This enables the backend to manage execution resources and coordinate cache operations (such as MergeTree cache loading) between the driver and executors.

backends-clickhouse/src/main/scala/org/apache/spark/rpc · high confidence

Introduces native JNI utilities for Java-to-C++ interop and task context access

This change adds new C++ JNI wrapper files (JniCommon, JniError, JniWrapper, TaskContextJniWrapper) to the core native module. It provides a centralized, thread-safe mechanism for the native side to interact with Java classes—specifically \RuntimeAware\, \JniByteInputStream\, and \ShuffleStreamReader\—and introduces a \JavaInputStreamAdaptor\ to allow native Arrow code to read from Java input streams. Additionally, it exposes a \getCurrentSparkTaskAttemptId\ function, enabling native components to retrieve the current Spark task attempt ID for logging and metrics.

cpp/core/jni · high confidence

Introduces native memory management and columnar batch implementations for Velox

This change adds the core C++ memory management and data batch infrastructure for the Velox backend. It introduces the VeloxMemoryManager, which wraps the Velox memory system and includes a ListenableArbitrator to coordinate memory growth and shrinkage with an AllocationListener. It also adds specific columnar batch types—VeloxColumnarBatch for standard Velox RowVectors and GpuBufferColumnarBatch for GPU-backed buffers—along with supporting utilities like BufferOutputStream for memory-backed output streams and GlutenDirectBufferedInput for optimized file reading.

cpp/velox/memory · high confidence

Introduces native shuffle utilities in the Velox backend

A new \ExecUtil\ object has been added to the Velox backend to handle core execution utilities, specifically focusing on native shuffle operations. This component provides the implementation for converting columnar batches to rows (required for fallback range partitioning) and constructing \ShuffleDependency\ objects that support native partitioning schemes such as Single, RoundRobin, Hash, and Range. It also includes logic to handle partition ID assignment for range-partitioned data by falling back to row-based computation when necessary.

backends-velox/src/main/scala/org/apache/spark/sql/execution/utils · high confidence

Introduces native vectorized execution and shuffle I/O for the ClickHouse backend

The ClickHouse backend now includes a new set of Java classes in the \org.apache.gluten.vectorized\ package that bridge Spark's columnar execution model with native code. This adds \BatchIterator\ and \CHNativeBlock\ to handle native columnar batch iteration and conversion, \BlockOutputStream\ and \CHShuffleSplitterJniWrapper\ to manage native shuffle writing with compression support, and \CHStreamReader\ along with \LowCopyFileSegmentShuffleInputStream\/\LowCopyNettyShuffleInputStream\ to optimize shuffle reading by reducing memory copies. It also introduces \CHColumnVector\ for native column access, \StorageJoinBuilder\ for native storage join operations, and \CHNativeExpressionEvaluator\ to initialize and manage the native computing kernel.

backends-clickhouse/src/main/java/org/apache/gluten/vectorized · high confidence

Introduces new native batch lifecycle and memory management APIs in gluten-arrow

The gluten-arrow module now exposes a new set of Java classes to manage the lifecycle of native columnar batches and their associated memory. This includes the \ColumnarBatchJniWrapper\ for JNI access to native batch operations, and a new batch typing system (\IndicatorVector\, \PlaceholderVector\, \ColumnarBatches\) that distinguishes between 'light' (native-only) and 'heavy' (JVM-readable Arrow) batches. It also introduces \NativeBackendInitializer\ for managing native backend initialization and shutdown, \NativeMemoryManagerJniWrapper\ for native memory tracking, and \ArrowBufferAllocators\ to manage Arrow buffer allocation within Spark's memory manager. These changes provide a more robust and explicit way to handle native memory, batch offloading, and thread context propagation for native workers. Also in this area: Replace custom ArrowWritableColumnVector with Spark's ArrowColumnVector and ArrowWriter.

gluten-arrow · high confidence

Introduction of ColumnarToRowConverter base class

A new base class, ColumnarToRowConverter, has been added to the core operators module to define the interface for converting columnar data to row format. This class establishes the contract for conversion operations, including methods to initiate the process from a specific row, retrieve the number of converted rows, and access internal buffers and offsets. It serves as the foundational component for implementing specific columnar-to-row conversion logic within the Gluten framework.

cpp/core/operators/c2r · high confidence

Introduction of RowToColumnarConverter interface

A new abstract base class, RowToColumnarConverter, has been added to the core operators module to define the contract for converting row-based data into columnar format. This interface establishes the foundational structure for native Velox row-to-column conversion capabilities, allowing specific implementations to handle the transformation logic while providing a consistent API for the rest of the system.

cpp/core/operators/r2c · high confidence

Introduction of centralized Velox configuration API

The Velox backend now exposes a dedicated \VeloxConfig\ class that consolidates backend-specific settings into a single, type-safe interface. This change introduces accessors for a wide range of operational parameters, including memory management (cache size, SSD cache paths, memory pool transfers), shuffle behavior (batch resizing, stream merging, GPU async shuffle), and query execution optimizations (broadcast hash table builds, bloom filter pushdown, dynamic filters). Users can now configure these aspects through the standard Spark configuration system, with the new API ensuring consistent access to these settings across the backend.

backends-velox/src/main/scala/org/apache/gluten/config · high confidence

Native Delta Lake deletion vector support in Velox connector

The Gluten Velox connector now natively reads Delta Lake deletion vectors, allowing it to correctly filter out logically deleted rows during table scans. This change introduces a new connector and data source in cpp/velox/compute/delta that integrates with the JVM Delta scan handoff to receive serialized bitmap payloads. The native layer deserializes these payloads using Roaring bitmaps, validates file statistics against deletion vector cardinality, and applies an efficient iterator-based bulk lookup to mark deleted rows in the output, ensuring data consistency and improved scan performance for Delta tables.

cpp/velox/compute/delta · high confidence

Native Delta scan offload with deletion vector and column mapping support

This change introduces the core infrastructure for offloading Delta Lake scans to the native Gluten execution engine. It adds post-transform rules that strip synthetic deletion-vector predicates and columns injected by Spark, allowing Velox to apply deletion-vector masks directly from split metadata and preventing double-filtering. It also includes logic to translate Delta column-mapping metadata into Parquet-compatible formats so the native reader can access files correctly. Additionally, it provides helper utilities for Change Data Feed (CDF) batch reads and a driver-side lookup index to efficiently map partitioned files to Delta AddFile metadata, ensuring accurate partition pruning and file selection during native scans.

gluten-delta · high confidence

Native Delta write offload for Velox backend

This change introduces a new Velox-specific write component for Delta Lake 3.3 that offloads Delta write commands (including INSERT, UPDATE, DELETE, and OPTIMIZE) to the native Velox engine. It replaces the standard Spark file format writer with a custom Gluten implementation that supports columnar execution, native Parquet output, and optimized statistics tracking to reduce conversion overhead. Users benefit from faster Delta write performance when using the Velox backend, with automatic fallback to row-based execution if native offloading is not possible.

backends-velox/src-delta33/main · high confidence

Native Iceberg read and write support in Velox backend

This change introduces the core C++ implementation for reading and writing Iceberg tables directly within the Velox backend. For reads, the new IcebergPlanConverter parses Substrait extensions to extract file formats (Parquet/ORC), column metadata, and delete file bounds (position and equality deletes), enabling the backend to handle Iceberg's merge-on-read semantics. For writes, the IcebergWriter component manages data sinking, including a custom filename generator that aligns output filenames with Java Iceberg conventions, and supports partitioned and unpartitioned table writes. This provides the foundational capability for native Iceberg table operations without relying on external connectors.

cpp/velox/compute/iceberg · high confidence

Native Iceberg write support for Velox backend

This change introduces the core components for writing to Apache Iceberg tables using the Velox backend, enabling columnar write offloading. It adds a new VeloxIcebergComponent that automatically registers when Iceberg session extensions are present, injecting rules to offload both scan and write operations. The implementation includes an IcebergDataWriteFactory and AbstractIcebergWriteExec that handle batch and streaming writes, passing schema, partitioning, and sorting information to the native layer via IcebergWriteJniWrapper. Additionally, it propagates Parquet configuration settings (page size, row group size, target file size, dictionary size, page row limit, page version, and compression level) to the native writer and provides utilities to convert Iceberg partition transforms for the native engine.

backends-velox/src-iceberg/main/scala · high confidence

Native ORC and Parquet write support for ClickHouse backend

The ClickHouse backend now supports native writing to ORC and Parquet file formats via the Spark V1 data source API. This change introduces new writer injects (CHOrcWriterInjects, CHParquetWriterInjects) that configure native write operations, including passing user-specified Parquet settings like compression and block size to the underlying engine. It also adds CHBlockStripes and CHRowSplitter to handle partitioning and bucketing of columnar batches, and replaces the previous FakeRow mechanism with BatchCarrierRow for more efficient data handling during the write process.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/datasources/v1 · high confidence

Native Parquet writer support for distributed cloud file systems

The Velox-based Parquet writer now supports writing to remote storage systems including S3, HDFS, GCS, and Azure Blob Storage (ABFS). Previously limited to local file paths, the writer now initializes the appropriate cloud file system client and sink during the write operation, enabling direct Parquet output to cloud object stores and distributed file systems.

cpp/velox/operators/writer · high confidence

Native Substrait-to-Velox plan conversion engine

The Velox backend now executes queries natively by converting Substrait plans into Velox execution plans directly in C++. This change introduces a new conversion layer (SubstraitToVeloxPlan) that maps Substrait relational operators (such as joins, aggregates, filters, and scans) to corresponding Velox plan nodes, along with expression conversion (SubstraitToVeloxExpr) and type parsing (SubstraitParser). It also includes a validation component (SubstraitToVeloxPlanValidator) to check plan correctness and an extension collector to manage function signatures. This enables the Velox backend to process Substrait plans without falling back to the Spark JVM for plan interpretation, improving performance and reducing serialization overhead.

cpp/velox/substrait · high confidence

Native UDF/UDAF library loading with variable arity and type conversion support

The \cpp/velox/udf\ module now provides the infrastructure to load native UDF and UDAF libraries at runtime. This change introduces \Udaf.h\ and \Udf.h\ to define entry structures and registration macros, and implements \UdfLoader\ to dynamically load these libraries, parse their signatures, and register them with Velox. The loader supports variable-arity functions and type conversion flags, and includes error handling for library load failures and memory allocation.

cpp/velox/udf · high confidence

Native metrics collection for ClickHouse backend operators

The ClickHouse backend now collects detailed execution metrics for a wide range of operators, including scans (BatchScan, FileSourceScan, HiveTableScan), joins (HashJoin, SortMergeJoin, BroadcastNestedLoopJoin), aggregations (HashAggregate), and transformations (Filter, Project, Sort, Window, Limit, Expand, Generate, Range). MetricsUpdater classes in this package map native execution data to Spark SQL metrics, providing visibility into timing, row counts, byte volumes, and specific operational details like cache hits for Parquet metadata and FillingRightJoinSide time for joins.

backends-clickhouse/src/main/scala/org/apache/gluten/metrics · high confidence

Native offloading support for Hive UDFs in Velox backend

The Velox backend now includes a dedicated transformer (VeloxHiveUDFTransformer) that enables the native execution of supported Hive User-Defined Functions. This change allows queries using these UDFs to be offloaded to the Velox engine, improving performance by avoiding the overhead of the JVM-based Spark execution path for these specific functions.

backends-velox/src/main/scala/org/apache/spark/sql/hive · high confidence

New Bloop-based Scala development workflow

Developers can now use Bloop for faster incremental Scala compilation and testing. The new dev/bloop-setup.sh script generates Bloop configuration files from Maven profiles, and dev/bloop-test.sh provides a wrapper to run Scala tests via Bloop instead of Maven, significantly reducing iteration time during development.

dev · high confidence

New C++ core utility library for Gluten

This change introduces a new set of C++ utility headers and source files in the \cpp/core/utils\ directory, establishing the foundational runtime support for the Gluten project. The new code provides essential infrastructure including an \ObjectStore\ for managing native object lifecycles across JNI boundaries, a \Registry\ for component registration, and \ResourceMap\ for handle management. It also adds robust error handling via \GlutenException\ and \ArrowException\, comprehensive timing utilities (\Timer\, \ScopedTimer\), and helper functions for string manipulation and UUID generation. Additionally, it includes a compression codec factory supporting QAT backends and a \WholeStageDumper\ interface for debugging, effectively consolidating core utilities previously scattered or missing into a unified, reusable library.

cpp/core/utils · high confidence

New C++ memory management and columnar batch abstractions

The \cpp/core/memory\ module introduces a new memory management layer and data structures. It adds an \AllocationListener\ interface and a \ListenableMemoryAllocator\ to track memory usage changes, and a \BlockAllocationListener\ to batch these updates. A new \MemoryManager\ registry allows for pluggable memory manager implementations. The module also defines \ColumnarBatch\ and \ArrowColumnarBatch\ classes to wrap Arrow record batches, and a \SplitAwareColumnarBatchIterator\ to support dynamic split management in task execution.

cpp/core/memory · high confidence

New ClickHouse backend build and runtime scripts

The \ep/build-clickhouse\ area now includes a complete set of shell scripts and configuration files to build and run the ClickHouse backend. \package.sh\ automates the build process for Spark 3.5 with Scala 2.12, packaging Gluten, Delta Lake, and Celeborn dependencies into a distributable archive. \build-clickhouse.sh\ handles the CMake-based compilation of the native \libch.so\ library using Clang 18. Runtime support is provided by \gluten.sh\ to start/stop the Spark ThriftServer with the necessary native library preload and classpath settings, while \check-env.sh\ validates the OS, Java, and Spark 3.5 versions. Default configurations in \gluten.properties\ and \spark-default.conf\ are included to enable columnar execution and native validation for the ClickHouse backend.

ep/build-clickhouse · high confidence

New ClickHouse backend utility modules for validation, partitioning, and plan generation

This change introduces a suite of new utility objects in the ClickHouse backend to enhance execution control and optimization. CHExpressionUtil adds a comprehensive validation framework with specific validators for scalar functions (such as date/time, string, and JSON operations) and aggregate functions, allowing the backend to safely reject unsupported or incompatible expressions. CHJoinValidateUtil provides logic to determine when join operations (like Sort Merge Join) must fallback to Spark due to complex conditions or join types. CHPartitionsUtil implements custom partition generation logic, including support for bucketed scans and file-count-based partitioning to optimize data reading. Additionally, CHAQEUtil aids in retrieving runtime statistics for adaptive query execution, IteratorUtil facilitates columnar batch creation from native blocks, and PlanNodesUtil assists in generating Substrait plan nodes for build-side relations.

backends-clickhouse/src/main/scala/org/apache/gluten/utils · high confidence

New Gluten tooling for integration testing, workload qualification, and performance benchmarking

This release introduces several new tools in the \tools/\ directory to streamline development and evaluation workflows. The \gluten-it\ integration test suite is now runnable via a dedicated \gluten-it.sh\ script, supporting commands for data generation, query execution, and a standard Spark shell for interactive testing. A new Qualification Tool analyzes Spark event logs to assess SQL workload compatibility with Gluten, identifying unsupported operators and estimating performance boosts. Additionally, a comprehensive benchmarking suite under \tools/workload/benchmark\_velox\ provides Jupyter notebooks and scripts to set up environments, run TPC-H/TPC-DS workloads, and perform detailed performance analysis with hardware metrics. Finally, automation scripts for generating and diffing function support documentation have been added to keep documentation aligned with code changes.

tools · high confidence

New JNI components for file system, hash table, and UDF integration

The \cpp/velox/jni\ area introduces three new native modules to bridge Java and C++ capabilities: \JniFileSystem\ registers a \jni:\/\jol:\ file system scheme allowing Velox to read and write files via Java objects (including spill-to-heap support); \JniHashTable\ provides JNI wrappers to build, serialize, and deserialize broadcast hash tables using the Java-side \VeloxBroadcastBuildSideCache\; and \JniUdf\ enables dynamic registration of user-defined functions and aggregate functions by passing signature metadata to the Java \UDFResolver\. These components are initialized and finalized within \VeloxJniWrapper\ during backend load/unload, extending the native backend's ability to interact with JVM resources for I/O, join optimization, and custom functions.

cpp/velox/jni · high confidence

New Java metrics classes for Velox operator and batch write tracking

The \backends-velox/src/main/java/org/apache/gluten/metrics\ package now includes new Java classes (\BatchWriteMetrics\, \IteratorMetricsJniWrapper\, \Metrics\, and \OperatorMetrics\) that expose detailed execution statistics from the Velox backend. \OperatorMetrics\ aggregates granular counters for data processing (rows, bytes, vectors), resource usage (CPU, memory, spills), and advanced features like bloom filter performance, dynamic filters, and storage I/O. \BatchWriteMetrics\ provides specific write-related counters (bytes, files, I/O time, wall time), while \IteratorMetricsJniWrapper\ and \Metrics\ facilitate the retrieval of native iterator metrics via JNI. These changes enable users to monitor fine-grained performance characteristics of Velox-backed operations.

backends-velox/src/main/java/org/apache/gluten/metrics · high confidence

New Velox build and setup infrastructure for ep/build-velox

This change introduces a new build environment for Velox in the ep/build-velox location. It adds a comprehensive set of shell scripts (build-velox.sh, get-velox.sh, and platform-specific setup scripts for CentOS 7/8, RHEL, openEuler 24, and macOS) to automate dependency installation and compilation. The build system now supports configurable features such as S3, GCS, HDFS, ABFS, GPU (cudf), and LTO. It also includes patches to fix compatibility issues with Arrow (protobuf, zstd, JNI wrappers) and adds support for IBM Power (ppc64le) architectures.

ep/build-velox · high confidence

New Velox utility classes for batch resizing, bloom filters, and file system validation

This change introduces several new utility classes in the Gluten Velox backend to support native integration. It adds GpuBufferBatchResizer and VeloxBatchResizer (with their JNI wrappers) to handle GPU buffer batch resizing with configurable size limits and prefetch bytes. It also introduces VeloxBloomFilter and its JNI wrapper, providing native implementations for inserting longs, checking membership, merging filters, and serializing bloom filters for zero-copy evaluation. Additionally, a new VeloxFileSystemValidationJniWrapper is added to check if file paths are supported by registered file systems.

backends-velox/src/main/java/org/apache/gluten/utils · high confidence

New Velox utility components for batch resizing, GPU shuffling, and configuration extraction

This change introduces a suite of new C++ utility classes in cpp/velox/utils to support advanced data processing capabilities. It adds VeloxBatchResizer and GpuBufferBatchResizer to manage and resize columnar data batches, with the GPU variant enabling efficient data movement to CUDA devices for shuffle operations. It also introduces CachedBatchQueue for thread-safe batch buffering, ConfigExtractor to map Spark and Gluten configuration settings to Velox connector configs (including S3, GCS, and Hive), and VeloxWholeStageDumper to persist execution plans, configurations, and input data for debugging. Additionally, it provides Common utilities for RE2 regex validation and a fast memory copy macro, along with VeloxArrowUtils for seamless schema and buffer conversion between Velox and Apache Arrow.

cpp/velox/utils · high confidence

New Velox-based columnar serialization and row/column conversion operators

The serializer directory now introduces a complete set of C++ operators for serializing and deserializing columnar batches using the Velox Presto serializer, alongside converters for transforming between columnar and row formats. Users benefit from a new \VeloxColumnarBatchSerializer\ that supports configurable compression, lossless timestamp handling, and per-column min/max/null statistics for partition pruning. The addition of \VeloxColumnarToRowConverter\ and \VeloxRowToColumnarConverter\ enables efficient in-memory conversion between columnar and row-based representations, while \VeloxGpuColumnarBatchSerializer\ extends serialization to support GPU-accelerated deserialization into cuDF tables.

cpp/velox/operators/serializer · high confidence

New benchmark utility library for Velox benchmarks

Added BenchmarkUtils.cc and BenchmarkUtils.h in the benchmarks/common directory, providing shared helper functions for initializing the Velox backend, reading Substrait plans from JSON files, constructing split information for ORC and Parquet datasets, and managing benchmark execution parameters such as CPU affinity, thread count, and iteration counts.

cpp/velox/benchmarks/common · high confidence

New columnar batch serialization interface with V3 per-column support

A new \ColumnarBatchSerializer\ interface has been introduced in the C++ core operators, defining a base class for serializing and deserializing columnar batches. This interface adds V2 and V3 serialization methods (\framedSerializeWithStats\, \framedSerializeV3\, \framedSerializeWithStatsV3\) that support per-column framing and statistics, enabling features like lazy deserialization and partition pruning. It also introduces \deserializeV3\, which allows for column projection during deserialization, allowing users to request specific columns rather than the entire batch.

cpp/core/operators/serializer · high confidence

New columnar shuffle components for dictionary encoding, GPU async reading, and reader threading

The shuffle module introduces several new capabilities: ArrowShuffleDictionaryWriter and its header add support for dictionary-encoded columns in hash-based shuffle, enabling more efficient serialization for repeated string and primitive values. VeloxGpuAsyncShuffleReader and VeloxGpuShuffleReader provide GPU-accelerated shuffle reading, with the async variant using a multi-threaded deserializer to prefetch and decompress batches concurrently, improving throughput for GPU workloads. ReaderThreadPool and its header introduce a configurable, priority-aware thread pool dedicated to shuffle reader tasks, allowing parallelized batch processing. RadixSort adds a lightweight, Spark-compatible radix sort implementation for shuffle-specific sorting needs. These changes collectively enhance shuffle performance, memory efficiency, and GPU integration.

cpp/velox/shuffle · high confidence

New dev container configurations for Gluten Velox Backend

Developers can now use pre-configured VS Code Dev Containers to set up the Gluten Velox development environment. Two variants are provided: a dynamic-link configuration (using the apache/gluten:centos-9-jdk8 image) for daily development, and a static-link configuration (using apache/gluten:vcpkg-centos-9) for packaging and vcpkg issue reproduction. Both configurations include necessary toolchains (GCC 12), dependencies (Arrow, JDK 8/17), and VS Code extensions (Metals, C/C++), and automatically handle build parallelism to prevent out-of-memory errors during native compilation.

.devcontainer · high confidence

New execution utilities and protocols for broadcast joins and file writes

This change introduces several new Scala components in the Velox execution layer to improve broadcast join handling and file write reliability. It adds \AdjustStageExecutionMode\ to manage GPU offloading for adaptive query execution stages, and \BroadcastUtils\ along with \ColumnarBuildSideRelation\ and \SerializedHashTableBroadcastRelation\ to handle the conversion, serialization, and caching of broadcast build-side data for hash joins. Additionally, it includes \SparkWriteFilesCommitProtocol\ to wrap Hadoop's commit protocol, ensuring that files written by a task are properly deleted if the task fails.

backends-velox/src/main/scala/org/apache/spark/sql/execution · high confidence

New execution utilities for the ClickHouse backend

Added two new utility objects, CHExecUtil and PushDownUtil, to the ClickHouse backend's execution package. CHExecUtil provides core functionality for converting between columnar and row formats, handling buffer compression, and managing partitioned block iteration during shuffle operations. PushDownUtil introduces logic to determine whether specific filter expressions can be pushed down to the underlying storage layer, enabling more efficient query execution by reducing data transfer.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/utils · high confidence

New internal data structures for ClickHouse MergeTree range filtering and write metrics

This change introduces four new Java classes within the ClickHouse datasource package to support internal data exchange: ExtensionTableBuilder and ExtensionTableNode handle the construction and serialization of table metadata (including schema, indexes, and part lists) for Substrait extension tables; MergeTreePartFilterReturnedRange defines the structure for returning filtered part ranges (part name, begin, end) from the driver; and WriteReturnedMetric captures write outcomes (part name, mark count, disk size, row count, bucket ID, partition values). These classes provide the necessary data models for the backend to communicate table structure and write statistics, supporting the broader MergeTree read/write and range execution capabilities.

backends-clickhouse/src/main/java/org/apache/spark/sql/execution/datasources/clickhouse · high confidence

New memory management and JNI loading infrastructure in gluten-core

This change introduces a new, unified memory-target abstraction in gluten-core, replacing previous ad-hoc memory handling with a composable chain of targets (e.g., DynamicOffHeapSizing, OverAcquire, RetryOnOom, Logging) that support borrowing, repaying, and spilling. It also adds a new JNI library loading mechanism (JniLibLoader/JniWorkspace) that extracts native libraries to a work directory, resolves symlinks safely via Path.toRealPath, and prevents duplicate loads, alongside a new ClosableIterator base class for safe resource cleanup and a ConsistentHash utility for node allocation.

gluten-core · high confidence

New metrics data models for ClickHouse backend observability

The ClickHouse backend now includes a new set of Java classes in the \org.apache.gluten.metrics\ package to structure and expose execution metrics. \MetricsData\, \MetricsProcessor\, and \MetricsStep\ define the schema for tracking input/output rows, bytes, vectors, and wait times, along with specific Parquet reader details like cache hits/misses and reader version. \NativeMetrics\ handles the deserialization of JSON metrics from the native layer, while \OperatorMetrics\ aggregates these data points alongside join and aggregation parameters, enabling more detailed performance monitoring for users.

backends-clickhouse/src/main/java/org/apache/gluten/metrics · high confidence

New native benchmark suite for Velox shuffle, I/O, and memory operations

The \cpp/velox/benchmarks\ directory now provides a dedicated CMake build and a set of native executables for performance testing. Users can run the \generic\_benchmark\ to simulate Spark-like query stages (including shuffle, spill, and parquet I/O), the \parquet\_write\_benchmark\ to measure write throughput, the \delta\_bitmap\_benchmark\ to profile Delta deletion vector handling, and the \s3\_async\_upload\_benchmark\ to test S3 multipart uploads. A \plan\_validator\_util\ is also included to debug Substrait plan validation, and \VeloxBatchResizerBenchmark\ covers memory management efficiency.

cpp/velox/benchmarks · high confidence

New native configuration module for Gluten settings

The C++ core now includes a dedicated configuration module (GlutenConfig) that defines constants for Spark configuration keys (such as debug mode, ANSI mode, session timezone, and memory settings) and provides utilities to parse configuration maps from the JVM and print configuration logs with sensitive data redaction.

cpp/core/config · high confidence

New native execution runtime and thread management infrastructure

The native compute layer introduces a pluggable execution runtime and thread management system. A new Runtime class provides a factory-based registry for backend implementations, allowing different execution engines to be registered and created dynamically. It also manages thread-local state for file writing operations. A new ThreadManager and ThreadInitializer framework enables per-task context propagation (such as JNI attachment) around native task execution, with a built-in noop initializer for environments like benchmarks where no JVM context is available. Supporting utilities include ProtobufUtils for handling Substrait plan serialization and a ResultIterator for managing execution output and metrics.

cpp/core/compute · high confidence

New protocol buffer definition for ClickHouse MergeTree write optimization

A new \write\_optimization.proto\ file has been added to define the data structures for write operations, specifically introducing a \MergeTreeWrite\ message. This message captures ClickHouse-specific metadata such as database, table, snapshot ID, and various index keys (order-by, low-cardinality, minmax, bloom filter, set, and primary keys), alongside storage policy and path information. This schema supports the underlying implementation of optimized write paths for the ClickHouse backend.

backends-clickhouse/src/main/resources/org · high confidence

New utility object for ClickHouse table identification and path resolution

A new \CHDataSourceUtils\ object has been added to the ClickHouse backend to provide helper methods for identifying ClickHouse tables and resolving their storage paths. This utility introduces functions to check if a table is a ClickHouse table based on catalog provider information, verifies if a table identifier represents a ClickHouse path (including handling Glue permission errors), finds the root directory of a ClickHouse table by scanning for metadata directories, and ensures the \ClickHouseTableV2\ is loaded for a given table identifier. These changes support the backend's ability to correctly recognize and interact with ClickHouse tables within the Spark SQL execution engine.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/datasources/v2/clickhouse/utils · high confidence

Placeholder added for Uniffle support module

A placeholder file has been added to the gluten-uniffle directory to reserve the location for common code that will be shared by backends for Uniffle support in Gluten.

gluten-uniffle · high confidence

Project initialization and infrastructure setup for Apache Gluten

This change establishes the foundational repository structure for the Apache Gluten project. It introduces essential configuration files including \.asf.yaml\ for GitHub integration (enabling discussions, Copilot code reviews, and branch protection), \AGENTS.md\ and \CONTRIBUTING.md\ to guide developers and AI agents on build workflows, code style, and PR conventions. It also adds standard Apache licensing files (\LICENSE\, \NOTICE\, \LICENSE-binary\, \NOTICE-binary\) and updates \.gitignore\ to exclude build artifacts, IDE configs, and generated files. The \Makefile\ provides targets for building native components (Velox, ClickHouse, Bolt) and packaging JARs, while \.scalafmt.conf\ enforces code formatting standards.

(repo-wide) · high confidence

Qualification tool adds lakehouse format detection and execution support analysis

The qualification tool now detects lakehouse table formats (Iceberg, Delta Lake, Hudi, Paimon) from Spark execution plan scan nodes using provider class, location path, and node name patterns. It also analyzes SQL workload support by classifying operators as supported or not, calculating total versus supported execution time, and identifying unsupported operators that require implementation for full Gluten compatibility.

tools/qualification-tool · high confidence

Support for caching Delta Lake data and local files in ClickHouse backend

Users can now cache Delta Lake table data and local file paths directly into the ClickHouse backend using new CACHE DATA and CACHE FILES commands. The implementation introduces GlutenCHCacheDataCommand to handle Delta table snapshots (including partition and timestamp filtering) and GlutenCacheFilesCommand to cache Parquet files from HDFS/local storage, leveraging executor affinity for efficient data placement.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/commands · high confidence

Support for reading MergeTree tables with Delta Deletion Vectors

The ClickHouse backend now supports reading MergeTree tables that utilize Delta's Deletion Vectors. A new utility class, MergeTreePartsPartitionsUtil, has been added to handle partition generation for these tables, integrating with the DeltaMergeTreeFileFormat and TahoeFileIndex to correctly process file paths and partition metadata when deletion vectors are present.

backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/datasources/clickhouse · high confidence

Support for writing task statistics to the event log

A new \TaskStatsAccumulator\ class has been added to the Spark metrics package to enable the collection and serialization of task-level statistics. This accumulator allows Velox-backed tasks to record specific performance or execution stats as strings, which are then merged and made available via the standard Spark AccumulableInfo interface, facilitating their inclusion in the Spark event log for downstream analysis.

backends-velox/src/main/scala/org/apache/spark/metrics · high confidence

Velox backend now supports offloading Lance table scans via Arrow C stream

Users can now execute Lance table scans through the Velox engine, leveraging the Arrow C stream for efficient data transfer. This change introduces a new component that registers the offload logic, allowing columnar Lance scans to be processed by Velox operators (such as aggregations) without converting data to rows first. The feature is inert unless the lance-spark runtime is present on the classpath, and it gracefully falls back to vanilla Spark for operations that cannot be served by the export path, such as pushed aggregations.

backends-velox/src-lance · high confidence

Removals

Removal of Arrow Data Source module

The Arrow Data Source module has been removed from the product. This change deletes the \arrow-data-source\ directory and all its contents, including the Parquet file format override (\ParquetFileFormat.scala\), the standard Arrow data source implementation (\ArrowFileFormat.scala\), the write extension logic (\ArrowWriteExtension.scala\), and the associated build scripts and CI configuration (\.travis.yml\). Users will no longer be able to use this specific Arrow-based data source for reading or writing Parquet files in Spark.

arrow-data-source, arrow-data-source/parquet, arrow-data-source/standard · high confidence

Removal of Spark 3.1.1 shim implementation

The Spark 3.1.1 shim provider, its corresponding service registration file, and the associated unit tests have been removed from the codebase. This eliminates the specific shim implementation for Spark version 3.1.1, meaning this version is no longer supported or shimmed by the library.

shims/spark311 · high confidence

Removal of legacy Arrow data source and native SQL engine components

The \arrow-data-source/common\ module has been removed, deleting the Java and Scala classes responsible for the legacy Arrow data source integration, including \SparkManagedReservationListener\, \RowToArrowColumnarExec\, and \SparkMemoryUtils\. Additionally, the \native-sql-engine\ C++ implementation and its build infrastructure (CMakeLists, compile scripts) have been deleted, along with the C++ source files for JNI wrappers, result iterators, and columnar-to-row conversion. This change eliminates the previous native execution path and its associated memory management and data conversion logic.

arrow-data-source/common, native-sql-engine, native-sql-engine/cpp · high confidence

Removal of legacy Parquet Arrow reader, Gandiva arithmetic, and Substrait binary protocol definitions

The legacy Parquet Arrow data source implementation has been removed, including the \VectorizedParquetArrowReader\ and its underlying native JNI wrappers (\ParquetReader\, \ParquetReaderJniWrapper\, \ParquetWriter\). Additionally, the \ColumnarArithmeticWithGandiva\ component for Gandiva-based columnar arithmetic expressions has been deleted. The engine also no longer includes the Substrait binary protocol definitions, as all related \.proto\ files (covering expressions, functions, plans, relations, and types) have been removed from the codebase.

native-sql-engine/core · high confidence

Architecture

Introduce new Velox compute backend implementation in cpp/velox/compute

The Velox backend's core C++ implementation has been reorganized into the new \cpp/velox/compute\ directory, introducing a dedicated \VeloxBackend\ singleton for initialization and configuration, a \VeloxPlanConverter\ to translate Substrait plans into Velox execution plans, a \VeloxRuntime\ to manage task execution and connectors (Hive, Iceberg, Delta, cuDF), and a \WholeStageResultIterator\ to drive the actual query execution. This change consolidates the backend's lifecycle management, plan conversion, and runtime execution logic into a cohesive, modular structure within the Gluten project.

cpp/velox/compute · high confidence

Refactor shim layer with common utilities and Spark 3.x/4.x compatibility

The shims/common module has been restructured to consolidate shared infrastructure and improve compatibility across Spark versions. A new AbstractColumnarArray class provides a safe workaround for accessing null values in Arrow vectors, while BatchCarrierRow and InternalRowSparkCompatible introduce zero-copy row abstractions and stub implementations for Spark 4.x-specific methods (getVariant, getGeography, getGeometry, getBinaryView) to ensure compatibility with both Spark 3.x and 4.x. The module also introduces GlutenFormatFactory for managing format writer injections, a centralized ExpressionNames object for function mapping, and a new SparkShimLoader that uses ServiceLoader for dynamic provider discovery. Legacy Intel OAP classes have been removed, and utility classes like Arm, ExceptionUtils, and GlutenBuildInfo have been added to support build-time versioning and resource management.

shims/common · high confidence

Behavioural changes

1 commit (0 fixes) modifying backends-clickhouse/src-paimon

A change to existing behaviour in backends-clickhouse/src-paimon — 1 commit, 3 files: “\[GLUTEN-13256\]\[CH\] Support reading Paimon tables with paimon-cpp (\#13334)” (refs \#13334).

backends-clickhouse/src-paimon · medium confidence · unverified

1 commit (0 fixes) modifying backends-velox/src-kafka

A change to existing behaviour in backends-velox/src-kafka — 1 commit, 3 files: “\[VL\] Offload Kafka micro-batch scans (\#13237)” (refs \#13237).

backends-velox/src-kafka · medium confidence · unverified

1 commit (0 fixes) modifying cpp/velox/benchmarks/data/tpch\_sf10m

A change to existing behaviour in cpp/velox/benchmarks/data/tpch\_sf10m — 1 commit, 8 files: “\[VL\] Add shuffle split to generic benchmark (\#2480)” (refs \#2480).

_cpp/velox/benchmarks/data/tpch\sf10m · medium confidence · unverified

2 commits (0 fixes) modifying backends-velox, repository

A change to existing behaviour in backends-velox, repository — 2 commits, 12 files: “\[GLUTEN-9003\]\[VL\] Count Delta's IncrementMetric natively (\#13165)”; “\[VL\]\[Iceberg\] Support SQL MERGE INTO for parquet(\#12964)” (refs \#13165, \#12964). Also in this area: 4 commits (1 fix) modifying backends-velox; Enable native Velox writes for Delta Lake on Spark 4.2.

backends-velox/src-delta4 · medium confidence · unverified

4 commits (2 fixes) modifying cpp/velox/benchmarks/data/bm\_lineitem

A change to existing behaviour in cpp/velox/benchmarks/data/bm\_lineitem — 4 commits (2 fixs), 5 files: “(VL) bugfix for query benchmark and test for orc/decimal reader (\#1613)”; “fix query benchmark (\#1189)”; “\[VL\] Add support for reading ORC (\#1513)”; and 1 more (refs \#1613, \#1189, \#1513, \#1726).

_cpp/velox/benchmarks/data/bm\lineitem · medium confidence · unverified

8 commits (1 fix) modifying backends-velox/src/main/scala/org/apache/gluten/execution

A change to existing behaviour in backends-velox/src/main/scala/org/apache/gluten/execution — 8 commits (1 fix), 2 files: “\[VL\]Use Velox's HashTableCache to cache the BHJ's HashTable (\#12163)”; “\[MINOR\] Decouple BHJ/BNLJ type flag from broadcast build-side cache key (\#12309)”; “\[GLUTEN-7548\]\[VL\] Optimize BHJ in velox backend (\#8931)”; and 5 more (refs \#12163, \#12309, \#8931, \#12264, \#11834, …).

backends-velox/src/main/scala/org/apache/gluten/execution · medium confidence · unverified

Align Velox Bloom Filter aggregate buffer capacity with native implementation

The Velox backend now uses a new \VeloxBloomFilterAggregate\ expression that calculates the bloom filter buffer capacity based on the \numBits\ parameter (divided by 16) rather than the estimated number of items. This change aligns the Scala-side capacity calculation with the native Velox \bloom\_filter\_agg\ implementation, preventing silent data corruption during two-phase aggregations where partial and final stages might execute on different engines with mismatched buffer sizes.

backends-velox/src/main/scala/org/apache/gluten/expression/aggregate · high confidence

Centralized Velox configuration constants in VeloxConfig.h

The Velox backend configuration options are now consolidated into a single header file, \cpp/velox/config/VeloxConfig.h\. This change introduces a comprehensive set of constants for Spark configuration keys covering spill strategies, memory management (including init capacity and reclaim wait times), hash map build parameters, bloom filter pushdown settings, file handle caching, and SSD cache tuning. Users benefit from a unified location for these settings, ensuring consistent defaults and enabling finer control over performance-critical behaviors like spilling, memory arbitration, and I/O optimization.

cpp/velox/config · high confidence

ClickHouse backend introduces dedicated vectorized serialization and iterator components

The ClickHouse backend now includes a new \CHColumnarBatchSerializer\ for handling shuffle serialization with configurable compression, alongside updated iterator classes (\CloseableCHColumnBatchIterator\, \CloseablePartitionedBlockIterator\) that ensure proper resource cleanup of columnar batches. Additionally, internal data structures like \SplitResult\ (renamed to \CHSplitResult\) and \LiteralTransformer\ (renamed to \BlockStats\) have been refactored and moved to the \org.apache.gluten.vectorized\ package to better align with the ClickHouse-specific pipeline.

backends-clickhouse/src/main/scala/org/apache/gluten/vectorized · high confidence

ClickHouse backend introduces soft and forced host affinity for MergeTree partitions

The ClickHouse backend now supports two modes of host affinity when writing to or reading from MergeTree tables: soft affinity, which suggests optimal hosts based on data locality, and forced affinity, which strictly pins tasks to specific hosts. This is implemented via the new \CHAffinity\ class, which integrates with the existing \SoftAffinityManager\ and respects the \affinityMode\ setting from \CHBackendSettings\. Users benefit from improved data placement and query performance by leveraging these affinity strategies for MergeTree operations.

backends-clickhouse/src/main/scala/org/apache/spark/affinity · high confidence

Explicit Arrow-to-Velox batch transition and new batch manipulation APIs

The Velox backend now provides explicit APIs to convert Arrow columnar batches into the native Velox format and to manipulate Velox batches directly. New methods in \VeloxColumnarBatches\ allow users to ensure a batch is in Velox format via \ensureVeloxBatch\, convert batches explicitly with \toVeloxBatch\, and perform operations like horizontal composition (\compose\), row slicing (\slice\), and repeated composition (\repeatedThenCompose\). These changes support explicit Arrow-to-Velox transitions, enabling better integration for specific use cases such as cached batch serialization where automatic planner insertion is not possible.

backends-velox/src/main/java/org/apache/gluten/columnarbatch · high confidence

Explicit restriction definitions for Velox expression support

The Velox backend now explicitly defines restriction rules for several SQL functions, clarifying which configurations and input types are unsupported. Specifically, \str\_to\_map\ is limited to the EXCEPTION key dedup policy; \from\_json\ requires partial results to be enabled and does not support options, case-sensitive mode, duplicate keys, or corrupt record columns; \to\_json\ does not support options and produces unexpected results for struct fields with uppercase names when case sensitivity is disabled; \unbase64\ does not support the failOnError flag; \base64\ does not support disabling chunked string output; \raise\_error\ only supports the errorMessage argument; and \format\_number\ is restricted to numeric inputs (excluding DecimalType) and integer decimal place counts (excluding string format patterns). These restrictions ensure users receive clear error messages when attempting unsupported configurations.

backends-velox/src/main/scala/org/apache/gluten/expression · high confidence

Introduces HashJoinBuilder for optimized hash table management

A new HashJoinBuilder class has been added to the Velox backend to facilitate optimized build-side hash join (BHJ) operations. This component exposes native methods for building, serializing, deserializing, cloning, and clearing hash tables, as well as retrieving metrics such as memory usage and bloom filter size. This change supports the broader effort to optimize hash join performance by allowing the hash table to be built and managed more efficiently on the driver side.

backends-velox/src/main/java/org/apache/gluten/vectorized · high confidence

Introduces iterator-based data source with task suspension for spill safety

This change adds new plan nodes and connector infrastructure (ValueStream, CudfValueStream, and IteratorSplit) to allow data from ResultIterators to be consumed via Velox's standard connector/split mechanism. Crucially, the RowVectorStream and CudfVectorStream implementations now suspend the Velox driver when crossing into Spark code via JNI. This ensures the task has zero running threads, preventing hangs during memory spilling that could occur if the task remained active while external code accessed its buffers.

cpp/velox/operators/plannodes · high confidence

Native shuffle stream handling for ClickHouse backend

The ClickHouse backend now includes dedicated factories for shuffle read and write streams that optimize data transfer by bypassing unnecessary memory copies. The new CHShuffleReadStreamFactory detects when shuffle data is available via Netty direct buffers or local file segments and routes it through low-copy input streams, falling back to on-heap copying only when necessary. Similarly, the CHShuffleWriteStreamFactory unwraps Spark compression output streams to support native writing paths for codecs like Zstd, LZ4, and LZF. This change improves shuffle performance by reducing data movement overhead between the JVM heap and native memory during query execution.

backends-clickhouse/src/main/java/org/apache/spark/storage · high confidence

New UDF JNI wrapper for function signature registration

A new UdfJniWrapper class has been added to the Gluten UDF backend, providing a native method to register function signatures. This component serves as the Java-side interface for binding UDF native registration logic, addressing previous JNI signature mismatches and replacing the prior native registration approach.

backends-velox/src/main/java/org/apache/gluten/udf · high confidence

New utility classes for Parquet metadata validation, shared library loading, and aggregation intermediate data mapping

This change introduces three new utility objects in the Gluten Velox backend to support specific runtime behaviors. ParquetMetadataUtils adds logic to validate Parquet file metadata (checking for encryption, unsupported codecs, legacy timezones, and variant annotations) and implements sampling to optimize validation performance, allowing scans to fall back to vanilla Spark when unsupported features are detected. SharedLibraryLoaderUtils promotes the shared library loading mechanism to an SPI-based approach, enabling OS-specific library loading via ServiceLoader while stripping quotes from OS identifiers for robustness. VeloxIntermediateData provides mappings for intermediate data ordering and types for various aggregate functions (such as Corr, Covariance, Skewness, Kurtosis, and Regr\*) to resolve inconsistencies between Spark and Velox accumulator structures, and fixes min\_by/max\_by result handling by adjusting the row constructor logic for null values.

backends-velox/src/main/scala/org/apache/gluten/utils · high confidence

Refactor Celeborn integration into an independent module with extensible serialization

The Gluten Celeborn integration has been refactored into an independent module, introducing new Java interfaces (CelebornShuffleWriterFactory, CelebornColumnarBatchSerializerFactory) and a unified CelebornShuffleManager that uses ServiceLoader to discover writer and serializer implementations. This change decouples Celeborn-specific code from the CH backend, adds support for both sort and RSS shuffle writers, and includes compatibility adjustments for various Celeborn versions (0.3.0 through 0.6.3) via reflection-based method dispatching in CelebornUtils.

gluten-celeborn · high confidence

Refactored native iterator to use ClickHouse column vectors and added cache management classes

The \ColumnarNativeIterator\ in the ClickHouse backend has been refactored to implement \Iterator\<byte\[\]\>\ instead of \NativeSerializedRecordBatchIterator\, switching from Arrow serialization to directly returning the block address of a \CHColumnVector\ via a new \longtoBytes\ helper. This change is accompanied by the introduction of \CHNativeCacheManager\ for managing cache operations (parts, files, status) and \CacheResult\ to represent cache status, as well as a new \SparkRowIterator\ for handling row iteration with byte buffers.

backends-clickhouse/src/main/java/org/apache/gluten/execution · high confidence

Register Velox-specific Celeborn serializer and writer factories

The Celeborn backend for Velox now registers its specific implementation classes via Java SPI service files. Users relying on the Velox backend with Celeborn shuffle will now have the shuffle operations routed through \VeloxCelebornColumnarBatchSerializerFactory\ and \VeloxCelebornColumnarShuffleWriterFactory\, ensuring the correct columnar batch serialization and shuffle writing logic is used instead of generic or previous defaults.

backends-velox/src-celeborn/main/resources · high confidence

The \veloxToArrow\ metric field has been removed from the \Metrics\ struct, and the associated \exportNanos\ tracking logic has been stripped from the \ColumnarBatch\, \ResultIterator\, and runtime implementations across both Bolt and Velox backends. This change simplifies the metrics payload by eliminating the now-unused timing data. Additionally, the unused \ShuffleSchema.h\ header and the \veloxToArrow\ metric have been removed, and benchmark error messages have been updated to point to the new documentation path under \docs/backends/velox/developers/\. Also in this area: Introduce C++ Bolt backend implementation and benchmarking suite. Also in this area: 3 commits (0 fixes) modifying cpp.

cpp/bolt · medium confidence · unverified

Standardized C++ code style and linting configuration

Developers now have a unified code style and static analysis setup for the C++ codebase. A new .clang-format file enforces a 120-column limit, 2-space indentation, and specific brace/spacing rules, while .clang-tidy introduces a diff-scoped check list (including naming conventions and performance checks) to catch issues early without flagging pre-existing header noise.

cpp · high confidence

Standardized C++ dependency management via FetchContent

The CMake build system now manages C++ dependencies (gflags, glog, gtest, google benchmark, jemalloc, roaring, QATzip, QAT-ZSTD, Arrow, Thrift, and Zstd) through a unified set of scripts in cpp/CMake. These scripts replace previous ad-hoc or external dependency resolution methods by downloading sources, verifying SHA256 checksums, and building them locally via FetchContent or ExternalProject. This ensures reproducible builds by bundling specific versions (e.g., gtest 1.13.0, jemalloc 5.3.0, roaring 4.3.11) and handling platform-specific requirements, such as glog namespace compatibility and QAT hardware library linking.

cpp/CMake · high confidence

Unified function registration and enhanced filter pushdown support

This change introduces a centralized function registration entry point (registerAllFunctions) that consolidates the loading of Spark SQL, Presto SQL, Iceberg, and Delta functions, while adding a new row constructor with null-handling logic and a custom Spark expression-to-subfield filter parser that enables filter pushdown for nested fields and Bloom filter might\_contain checks.

cpp/velox/operators/functions · high confidence

Velox backend component registration and updated shared library loader support

The Velox backend is now registered as a discoverable component via a new service file, enabling automatic detection. Additionally, the shared library loader configuration has been updated to support a broader range of operating systems, including CentOS 9, Debian 11/12, OpenEuler 24.03, and macOS, while removing support for CentOS 7.

backends-velox/src/main/resources/META-INF · high confidence

Fixes

Fix NaN equality comparison in ClickHouse backend

The ClickHouse backend now correctly handles comparisons where a column is compared against NaN. Previously, using \= NaN\ produced different results than expected; this change introduces an \EqualToRewrite\ rule that automatically transforms \EqualTo(left, NaN)\ or \EqualTo(NaN, right)\ expressions into \IsNaN(left)\ or \IsNaN(right)\, ensuring consistent behavior with the ClickHouse engine.

backends-clickhouse/src/main/scala/org/apache/spark/sql/catalyst · high confidence

Test coverage

1 commit adding/updating tests in backends-clickhouse/src/test/resources/text-data/compressed; 1 commit adding/updating tests in backends-clickhouse/src/test/resources/tpch-data-bucket; 1 commit adding/updating tests in backends-velox/src/test/resources/data-type-validation-data; 1 commit adding/updating tests in backends-velox/src/test/resources/parquet-for-read; 1 commit adding/updating tests in backends-velox/src/test/resources/tpch-data-parquet; 1 commit adding/updating tests in backends-velox/src/test/scala/org/apache/gluten/metrics; 3 commits adding/updating tests in backends-clickhouse/src/test/resources/orc-data; 41 commits adding/updating tests in gluten-ut/spark42; Add ClickHouse backend test suite for Spark 4.0; Add Spark 3.4 test resources and configuration for the Bolt backend; Add test coverage for the new Bolt backend in Spark 3.5; Added CSV test fixtures for Excel format parsing edge cases; Added CSV test resources for Velox datasource tests; Added ClickHouse Kafka scan test suite; Added ClickHouse backend test suites for TPCH queries and datetime expressions; Added ClickHouse compatibility test suites; Added ClickHouse-specific TPC-H query resources for testing; Added ClickHouse-specific benchmarks and test utilities; Added ClickHouse-specific test infrastructure and utilities; Added ClickHouse-specific test suites for Parquet column index and filter pushdown; Added Delta Lake test suites for Spark 3.5 and Delta 3.3; Added Gluten-specific test suites for Iceberg Spark 3.4 source operations; Added GlutenSuiteUtils test helper for fallback event collection; Added Hive table integration tests for the ClickHouse backend; Added Hudi integration tests for time travel, soft delete, and partition filtering; Added Iceberg integration tests for ClickHouse backend; Added NativeWriteChecker test utility for validating ClickHouse native write execution; Added S3 authentication test suite for ClickHouse backend; Added Spark 4.1 test suite for ClickHouse backend; Added TPC-DS and TPC-H benchmark schema generation scripts for ClickHouse; Added TPC-DS decimal query test outputs; Added TPC-DS query output fixtures for the ClickHouse backend; Added TPC-DS test suites for ClickHouse backend shuffle and join strategies; Added UDTF test fixtures for Gluten integration testing; Added Velox-specific test suites for Hudi and TPCH integration; Added benchmark for Delta Lake planning performance; Added benchmarks for columnar table cache, streaming aggregates, and broadcast joins; Added regression tests for Bloom Filter whole-stage fallback consistency; Added test UDFs for link ID generation and array duplication; Added test coverage for Gluten configuration, expression casting, and aggregate behavior; Added test coverage for Iceberg integration and Delta native writes; Added test coverage for existence join with broadcast nested loop join; Added test data files for ClickHouse backend tests; Added test data for CSV files with carriage return line endings; Added test for Celeborn-to-local shuffle fallback; Added test for OnHeapFileSystem round-trip operations; Added test for Velox configuration parsing; Added test for position deletes table support in Iceberg for Spark 3.5; Added test helper for DataFrame creation in ClickHouse backend tests; Added test infrastructure for Iceberg Spark 3.5 integration; Added test infrastructure for Velox backend integration; Added test logging configuration for Velox backend tests; Added test service provider configuration for SharedLibraryLoader; Added test suite for native Delta Lake support; Added test suites for ClickHouse backend execution features; Added test suites for MergeTree data format operations; Added test suites for Velox execution features and fixes; Added test suites for Velox-backed Iceberg operations; Added test suites for struct field binding, UDF partial projects, and UDF resolution; Added test tag annotations for test categorization; Added test utilities and benchmark helpers for the ClickHouse backend; Added test utility for ClickHouse soft affinity configuration; Added test utility helpers for ClickHouse backend storage and cache testing; Added tests for Arrow Python UDF execution in Velox backend; Added tests for Celeborn shuffle integration with ClickHouse backend; Added tests for ClickHouse-specific expression and columnar extension transformers; Added tests for ColumnarCachedBatch serialization and filter pruning; Added tests for Delta Deletion Vector scan info extraction; Added tests for Iceberg nested field visitor; Added tests for Iceberg write execution capacity and table property handling; Added tests for Iceberg write functionality and Parquet configuration; Added tests for MixedAffinity partition location resolution; Added tests for NativeBenchmarkPlanGenerator; Added tests for Parquet encryption detection and SharedLibraryLoader SPI integration; Added tests for TencentOS Server compatibility in CentOS 8 shared library loader; Added tests for UnsafeColumnarBuildSideRelation serialization and memory management; Added tests for Velox Bloom Filter utility; Added tests for Velox ColumnarBatch lifecycle and data integrity; Added tests for Velox cast expressions; Added tests for Velox columnar transition logic; Added tests for Velox configuration documentation generation and default values; Added tests for Velox schema validation of VOID and empty struct types; Added tests for custom aggregate expression transformation; Added tests for custom aggregate function support; Added tests for decimal write support in ArrowWritableColumnVector; Added tests for global off-heap memory management; Added unit tests for ClickHouse backend native metrics collection; Added unit tests for ClickHouse file cache functionality; Added unit tests for Gluten Flink streaming operators and RexNode conversion; Added unit tests for core components; Added validation test suites for Velox scalar, date, math, JSON, and window functions; New C++ test suite for Velox integration components; New common test infrastructure and logging configuration; Updated expected results for ClickHouse TPCH queries; Updated test logging configuration for ClickHouse backend.

Dependencies

The build structure now includes several new Maven modules: \backends-bolt\ (a new backend), \gluten-celeborn\ (extracted Celeborn integration), \gluten-arrow\ (renamed from \gluten-data\), and \gluten-flink\ (split into \loader\, \planner\, and \runtime\ submodules). Additionally, the Flink dependency has been upgraded to version 1.19.3.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 54 → 53 (-0.7)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 55 → 51 (-3.9)
  • Architecture 63 → 63 (+0.0)
  • Maturity 73 → 76 (+2.8)
  • Readiness 63 → 66 (+2.9)
  • Security 46 → 45 (-1.4)
  • Domain Modelling 100 → 100 (+0.0)

Resolved (112)

  • Change coupling: CHColumnarShuffleWriter.scala ↔ ColumnarShuffleWriter.scala (backends-clickhouse/src/main/scala/org/apache/spark/shuffle/CHColumnarShuffleWriter.scala)
  • Change coupling: ColumnarBuildSideRelation.scala ↔ ExecUtil.scala (backends-velox/src/main/scala/org/apache/spark/sql/execution/ColumnarBuildSideRelation.scala)
  • Duplicated block (10 lines × 2) (backends-bolt/src-paimon/main/scala/org/apache/gluten/component/BoltPaimonComponent.scala)
  • Duplicated block (10 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltSparkPlanExecApi.scala)
  • Duplicated block (10 lines × 3) (tools/gluten-it/common/src/main/scala/org/apache/gluten/integration/ds/TpcdsTableLayout.scala)
  • Duplicated block (10 lines × 6) (tools/gluten-it/common/src/main/scala/org/apache/gluten/integration/ds/TpcdsTableLayout.scala)
  • Duplicated block (10 lines × 6) (tools/gluten-it/common/src/main/scala/org/apache/gluten/integration/ds/TpcdsTableLayout.scala)
  • Duplicated block (11 lines × 2) (gluten-delta/src-delta33/main/scala/org/apache/gluten/execution/DeltaProjectExecTransformer.scala)
  • Duplicated block (11 lines × 3) (tools/gluten-it/common/src/main/scala/org/apache/gluten/integration/ds/TpcdsTableLayout.scala)
  • Duplicated block (12 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltSparkPlanExecApi.scala)
  • Duplicated block (12 lines × 2) (tools/gluten-it/common/src/main/scala/org/apache/gluten/integration/ds/TpcdsTableLayout.scala)
  • Duplicated block (13 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltSparkPlanExecApi.scala)
  • Duplicated block (13 lines × 6) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala)
  • Duplicated block (14–15 lines × 2) (backends-bolt/src/main/scala/org/apache/spark/api/python/ColumnarArrowEvalPythonExec.scala)
  • Duplicated block (15 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala)
  • Duplicated block (15 lines × 4) (backends-bolt/src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala)
  • Duplicated block (15–17 lines × 2) (tools/gluten-it/common/src/main/scala/org/apache/gluten/integration/ds/TpcdsTableLayout.scala)
  • Duplicated block (17 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltSparkPlanExecApi.scala)
  • Duplicated block (17 lines × 2) (backends-bolt/src/main/scala/org/apache/spark/api/python/ColumnarArrowEvalPythonExec.scala)
  • Duplicated block (18 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltIteratorApi.scala)
  • …and 92 more

New (303)

  • Duplicated block (10 lines × 2) (gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala)
  • Duplicated block (10–12 lines × 4) (backends-bolt/src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala)
  • Duplicated block (10–13 lines × 2) (gluten-flink/planner/src/main/java/org/apache/gluten/rexnode/functions/IntervalArithmeticRexCallConverter.java)
  • Duplicated block (12–13 lines × 2) (backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenOptimisticTransaction.scala)
  • Duplicated block (13 lines × 6) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala)
  • Duplicated block (15 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala)
  • Duplicated block (16 lines × 2) (gluten-delta/src-delta33/main/scala/org/apache/gluten/execution/DeltaFilterExecTransformer.scala)
  • Duplicated block (18 lines × 2) (gluten-delta/src-delta33/main/scala/org/apache/gluten/execution/DeltaFilterExecTransformer.scala)
  • Duplicated block (23 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/execution/ColumnarPartialGenerateExec.scala)
  • Duplicated block (23 lines × 2) (gluten-delta/src-delta33/main/scala/org/apache/gluten/execution/DeltaProjectExecTransformer.scala)
  • Duplicated block (25–26 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltBackend.scala)
  • Duplicated block (26 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala)
  • Duplicated block (26–29 lines × 2) (backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenOptimisticTransaction.scala)
  • Duplicated block (27 lines × 2) (gluten-delta/src-delta33/main/scala/org/apache/gluten/execution/DeltaProjectExecTransformer.scala)
  • Duplicated block (28 lines × 2) (backends-bolt/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala)
  • Duplicated block (5 lines × 3) (gluten-flink/planner/src/main/java/org/apache/gluten/rexnode/functions/BaseRexCallConverters.java)
  • Duplicated block (6 lines × 2) (backends-clickhouse/src-paimon/main/scala/org/apache/gluten/execution/CHPaimonScanTransformer.scala)
  • Duplicated block (6 lines × 2) (backends-clickhouse/src-paimon/main/scala/org/apache/gluten/execution/CHPaimonScanTransformer.scala)
  • Duplicated block (7 lines × 2) (backends-bolt/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleReader.scala)
  • Edited copies of one member (3 members, 11+ corresponding lines) (backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltMetricsApi.scala)
  • …and 283 more

Changes since last survey

  • 99 commits — 87 feature/other, 12 fixes

By area

  • cpp/velox — 18 commits
  • backends-velox/src — 15 commits
  • .github/workflows — 10 commits
  • gluten-flink/planner — 8 commits
  • gluten-ut/spark42 — 7 commits
  • tools/workload — 7 commits
  • gluten-flink/ut — 4 commits
  • gluten-substrait/src — 4 commits
  • cpp/core — 3 commits
  • gluten-ut/spark34 — 3 commits
  • ep/build-velox — 2 commits
  • gluten-ut/spark35 — 2 commits
  • gluten-ut/spark40 — 2 commits
  • (repo) — 1 commit
  • backends-bolt/src — 1 commit
  • backends-clickhouse/src — 1 commit
  • backends-velox/src-kafka — 1 commit
  • backends-velox/src-lance — 1 commit
  • cpp-ch/local-engine — 1 commit
  • cpp/bolt — 1 commit

Notable commits

  • fix: [CORE] Fix nested field rebinding after schema pruning (#13478)
  • fix: [GLUTEN-12857][TEST] Fix V1 DPP suites extending the V2 base (#13620)
  • fix: [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 (#13287)
  • fix: [GLUTEN-13179][VL] Fix Spark 4.2 function-registry and error-wrapping test failures (#13442)
  • fix: [GLUTEN-13276][VL] Fix CelebornShuffleManager init fails with NoSuchMethodException: FailedShuffleCleaner.<init>() (#13283)
  • fix: [GLUTEN-13593][CH] Fix regexp_extract for patterns with escaped brackets in character class (#13594)
  • fix: [MINOR] fix release build pipeline (#13632)
  • fix: [MINOR][CI] Fix the test report workflow (#13611)
  • fix: [MINOR][DOC] Fix Delta query flow image link (#13498)
  • fix: [MINOR][VL] Fix integer overflow in shuffle buffer offsets and shrink size (#13448)
  • fix: [VL] Fix Parquet metadata scans with same-named data columns (#13136)
  • fix: [VL] Fix and enable native UDF cluster suite (#13530)
  • change: Bump black in /tools/workload/benchmark_velox/analysis (#13465)
  • change: Bump gitpython in /tools/workload/benchmark_velox/analysis (#13464)
  • change: Bump nbconvert in /tools/workload/benchmark_velox/analysis (#13466)
  • change: Bump pyasn1 in /tools/workload/benchmark_velox/analysis (#13536)
  • change: Bump setuptools in /tools/workload/benchmark_velox/analysis (#13562)
  • change: Bump soupsieve in /tools/workload/benchmark_velox/analysis (#13395)
  • change: Bump soupsieve in /tools/workload/benchmark_velox/analysis (#13638)
  • change: [CORE] Bump version to 1.9.0-SNAPSHOT (#13475)
  • …and 79 more

Architecture

  • Unchanged — 0 containers · 10 contexts · 10 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

apache/gluten was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 3342737a427bac513dfd5d931ed93619110da215 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.