Skip to content
CAI
Software that uses CAICheck a score

apache/datafusion-comet

66.1

Adequate · 14 November 2026

280.2k

lines of production code

Rust

with Scala

7

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add TPC-DS benchmark queries to repository

Added a collection of SQL benchmark queries (q1 through q37) derived from the TPC-DS standard to the \benchmarks/tpc/queries/tpcds\ directory. These files provide the specific workload definitions required to run the CometBench-DS performance benchmarks.

benchmarks/tpc/queries · high confidence

Add native RSS partition writer with task-owned JNI callbacks

Introduces a new native RSS (Remote Shuffle Service) partition writer that encodes shuffle blocks into Arrow IPC format and pushes them to a task-owned remote pusher via JNI callbacks. This implementation enforces JVM partition limits (i32), validates frame sizes against configurable reservation limits, and ensures block boundaries are preserved for remote concatenation, enabling efficient map-side shuffle push to external shuffle services.

native/shuffle/src/writers/rss · high confidence

Add nested TPC-H benchmark for testing nested data handling

Added a new benchmark suite in benchmarks/nested that transforms flat TPC-H datasets into a nested schema, wrapping non-key columns into single-element arrays of structs. This includes a Spark conversion script (scripts/nest\_tpch.scala) to generate the nested Parquet or Iceberg tables, 22 SQL queries (queries/q01.sql–q22.sql) that unnest these arrays to exercise nested data operations like pruning and pushdown, and comprehensive documentation (README.md) explaining the schema changes and usage.

benchmarks/nested · high confidence

Add standalone shuffle benchmark tool

A new standalone binary tool (\shuffle\_bench\) has been added to profile Comet shuffle write performance independently of Spark. It reads input directly from Parquet files and allows users to configure partitioning schemes (hash, single, round-robin), compression codecs, memory limits, and concurrent task simulation to measure write throughput and metrics.

native/shuffle/src/bin · high confidence

Add to\_csv function for converting structs to CSV strings

Users can now convert struct data into CSV-formatted strings using the new \to\_csv\ function. This feature allows customization of the output format via \CsvWriteOptions\, including configuration for delimiters, quote characters, escape characters, null value representation, and whitespace handling (ignoring leading or trailing spaces). The implementation handles type casting to UTF-8 and properly quotes/escapes values based on the specified options.

_native/spark-expr/src/csv\funcs · high confidence

Added JNI utility for Parquet schema deserialization

A new \jni\ module has been added to the Parquet utilities, providing a \deserialize\_schema\ function that allows the native layer to parse Arrow IPC byte streams into Arrow schemas. This enables Java/Python bindings to efficiently transfer schema definitions from the native Rust core without requiring full serialization round-trips.

native/core/src/parquet/util · high confidence

Added build gate stub for native Lance integration

A new native Rust module for the Lance contrib component has been introduced, establishing a build-time gate for the feature. This location provides the core library structure and a planner stub that currently returns a 'not implemented' error, serving as the foundational placeholder for the native Lance read path within the DataFusion Comet integration.

contrib/lance/native · high confidence

Added build-gate stubs and configuration for native Delta scan integration

This change introduces the initial scaffolding for the native Delta scan feature within the \contrib/delta\ module. It adds a Rust build-gate stub (\contrib/delta/native\) that defines the \plan\_delta\_scan\ entry point, which currently returns a 'not implemented' error to allow compilation without the full implementation. Additionally, it registers new Spark configuration options under the \spark.comet.scan.deltaNative\ prefix (such as \enabled\, \fallbackOnUnsupportedFeature\, and \dataFileConcurrencyLimit\) to control the behavior of the future native Delta read path, ensuring these settings are discoverable in documentation and do not affect default builds.

contrib/delta, contrib/delta/native · high confidence

Added local HDFS development environment configuration

Users can now run a local Hadoop Distributed File System (HDFS) cluster for development and testing purposes. This change introduces a Docker Compose setup (\hdfs-docker-compose.yml\) and its associated environment configuration (\hadoop.env\) within the \kube/local\ directory. The configuration defines a cluster consisting of a NameNode, three DataNodes, a ResourceManager, and a NodeManager, allowing developers to spin up a complete HDFS environment locally using the specified BDE2020 Hadoop images.

kube/local · high confidence

Comet integration for Spark 3.4, 3.5, 4.0, and 4.1

This change integrates the Apache DataFusion Comet engine into Spark versions 3.4.3, 3.5.9, 4.0.4, and 4.1.3. It adds the \comet-spark\ dependency to the build, modifies \SparkSession\ to automatically load the \CometSparkSessionExtensions\ when enabled, and updates \SparkPlanInfo\ to include metadata for Comet scans. Additionally, it disables Comet in specific test suites (e.g., fallback storage, explain plans) where it is incompatible or causes precision differences.

dev/diffs · high confidence

Consolidated TPC benchmark suite with Iceberg and profiling support

The TPC benchmarking scripts have been consolidated into a unified runner (\run.py\) that supports TPC-H and TPC-DS benchmarks across multiple engines (Spark, Comet, Comet-Iceberg, Gluten). This update introduces native Iceberg table support, allowing users to benchmark Comet's native Iceberg scan acceleration by converting Parquet data to Iceberg format and running queries against the resulting tables. Additionally, the benchmark suite now includes built-in profiling capabilities via Java Flight Recorder (JFR) and async-profiler, enabling detailed performance analysis of Java and native code during execution. The suite also features consistency checks with result hashing to detect regressions and a new script to generate comparative speedup charts from benchmark results.

benchmarks/tpc · high confidence

Docker-based infrastructure for TPC benchmarks

Added Dockerfiles and Docker Compose configurations to containerize the TPC benchmark environment. The setup includes a base benchmark image with Java 8/17, Python, and async-profiler, a dedicated builder image for compiling Comet native libraries, and two Docker Compose profiles (laptop and cluster) to orchestrate Spark standalone clusters for running benchmarks.

benchmarks/tpc/infra · high confidence

Experimental native Delta Lake scan support in contrib module

A new experimental contrib module (\comet-contrib-delta-spark\) enables native scanning of Delta Lake tables through Comet's existing native Parquet reader, providing row-group pruning, page-index pruning, filter pushdown, and deletion vector support. This feature is opt-in and requires adding the module's JAR to the classpath alongside \delta-spark\ and setting \spark.comet.scan.delta.enabled=true\. It is supported on Spark 3.5/4.0/4.1 with corresponding Delta versions, while unsupported configurations fall back to Spark's reader.

contrib · high confidence

Experimental support for custom Rust UDFs via Arrow FFI

This change introduces an experimental mechanism for loading and executing custom scalar UDFs written in Rust as native shared libraries (cdylibs). It adds the \comet-udf-sdk\ crate, which defines a C ABI based on Arrow's stable C Data Interface to allow user libraries to interact with Comet without being coupled to its DataFusion version. The \native/core/src/execution/c\_udf\ module provides the host-side infrastructure to load these libraries, cache them process-wide, and adapt them into DataFusion \ScalarUDFImpl\ instances for execution. A test crate (\comet-test-udfs\) is also added to validate the ABI round-trip for various data types.

native/comet-test-udfs, native/comet-udf-sdk · high confidence

Experimental support for loading native UDF libraries via C-ABI

The native core now introduces an experimental mechanism to load user-supplied native UDFs (compiled as cdylibs) at runtime. This change adds a process-wide library cache to prevent use-after-free issues and ensure concurrent tasks share loaded libraries safely, a loader that validates the ABI version and discovers kernels via the C Data Interface, and an adapter that wraps the C-ABI kernels as DataFusion ScalarUDFImpls. It also includes a configurable security policy (controlled by spark.comet.nativeUdf.enabled and spark.comet.nativeUdf.allowedPaths) to restrict which libraries can be loaded, and type normalization logic to reconcile Spark and Arrow type differences for lists and maps during validation.

_native/core/src/execution/c\udf · high confidence

Initial project scaffolding and repository configuration

The repository has been initialized with the foundational structure for Apache DataFusion Comet, including the Apache License 2.0, NOTICE file, and a Maven wrapper (version 3.2.0) to ensure consistent builds. A Makefile is provided to manage the native Rust and JVM build lifecycle, alongside configuration files for code formatting (scalafmt, prettier) and linting (scalafix). The project is configured as an Apache Software Foundation repository via \.asf.yaml\, which enforces a squash-merge workflow and sets up branch protection for release branches. Additionally, developer guidelines are established through \AGENTS.md\ and \CONTRIBUTING.md\, and the README is updated to reflect the project's Arrow-native acceleration capabilities.

(repo-wide) · high confidence

Introduce PySpark benchmark suite for shuffle performance comparison

A new PySpark benchmark framework has been added to the \benchmarks/pyspark\ directory, enabling users to compare shuffle performance across Spark, Comet JVM, and Comet Native implementations. The suite includes a data generation script (\generate\data.py\) that creates realistic test datasets with complex nested schemas, a base benchmark class (\benchmarks/base.py\) for standardized timing and configuration reporting, and a registry (\benchmarks/\\init\\_.py\) that currently supports \shuffle-hash\ and \shuffle-roundrobin\ benchmarks. Users can run individual benchmarks or execute the full suite via \run\_all\_benchmarks.sh\, which automatically resolves the Comet JAR and configures the necessary Spark settings for each mode.

benchmarks/pyspark · high confidence

Introduce micro benchmark runner for standalone execution

Added a new Python-based micro benchmark runner (\benchmarks/micro/run.py\) and its documentation, enabling users to execute Comet micro benchmarks on a dedicated machine (such as an EC2 instance) without cloning the entire repository. The script automates the full workflow—including environment setup, building Comet in release mode, running specific benchmark suites, collecting results, and publishing them via pull requests—while discovering test suites directly from the source tree.

benchmarks/micro · high confidence

Introduce native Comet execution engine and codegen dispatcher

This change introduces the core native execution engine for Comet, replacing the previous scan-only approach with a full query execution path. It adds the \CometExecIterator\ to orchestrate native plan execution, integrates a Janino-based codegen dispatcher for JVM UDFs and expressions, and implements native shuffle and columnar-to-row conversion. The update also includes new memory management via \CometTaskMemoryManager\, enhanced error conversion through \SparkErrorConverter\, and expanded documentation generation for expression compatibility.

repository · high confidence

Introduce native columnar-to-row conversion and aggregate merge support

This change adds the native implementation for converting Arrow columnar data into Spark's UnsafeRow format (columnar\_to\_row.rs), enabling faster row-based operations in Spark. It also introduces a MergeAsPartial wrapper (merge\_as\_partial.rs) that bridges DataFusion's aggregate modes to support Spark's PartialMerge aggregation semantics, allowing intermediate state merging to be executed natively.

native/core/src/execution · high confidence

Introduce native common library for error handling, schema reconciliation, and UTF-8 decoding

A new \native/common\ crate has been added to centralize shared native logic. It introduces a comprehensive \SparkError\ enum that maps native failures to specific Spark error codes (such as \CAST\_INVALID\_INPUT\, \ARITHMETIC\_OVERFLOW\, and \MALFORMED\_VARIANT\) to ensure consistent exception reporting. The library includes schema reconciliation helpers (\cast\_and\_stamp\_schema\, \widen\_nested\_nullability\) that normalize Arrow type drift—particularly nested nullability mismatches—before stamping schemas onto record batches. It also provides UTF-8 decoding utilities (\decode\_string\_arrays\, \decode\_utf8\_spark\_lossy\) that validate and decode string data at the JVM-to-native FFI boundary, matching the JDK's replacement behavior for invalid sequences. Additionally, the crate contains utilities for pushing struct null masks into children to prevent silent data corruption and a tracing recorder for performance profiling. Also in this area: Fix Parquet timestamp overflow errors and handle NullType in struct null propagation; Native Python UDF errors are surfaced as Spark PythonException; Fixes JVM data corruption and memory log inaccuracies in native exports.

native/common/src · high confidence

Introduce official release process scripts for Apache DataFusion Comet

The \dev/release\ directory now contains a complete set of scripts to automate the official Apache Software Foundation source release process. This includes \build-release-comet.sh\ for building native binaries in Docker containers, \create-tarball.sh\ for generating signed tarballs and drafting the release vote email, \publish-to-maven.sh\ for uploading artifacts to a Nexus staging repository, and \verify-release-candidate.sh\ for validating release candidates. Supporting tools include \generate-changelog.py\ for automated changelog generation, \run-rat.sh\ and \check-rat-report.py\ for license compliance checking, and \release-tarball.sh\ for promoting approved candidates to the final distribution area.

dev/release · high confidence

Native Iceberg partitioning transforms and CSV struct serialization

This release adds native support for Iceberg's partitioning system functions—\iceberg\_bucket\, \iceberg\_truncate\, \iceberg\_years\, \iceberg\_months\, \iceberg\_days\, and \iceberg\_hours\—ensuring that hidden partitioning logic (hash distribution, local sort, and row-level filters) produces identical results to the Java implementation. It also introduces a native \to\_csv\ expression that converts struct columns to CSV strings, configurable via \CsvWriteOptions\ (delimiter, quote, escape, null value, and whitespace handling).

native/spark-expr/src · high confidence

Native Iceberg scan and write operators with CSV support

This change introduces the native execution operators for Iceberg table scans and writes, along with a new native CSV scan operator. The Iceberg scan operator (\IcebergScanExec\) leverages \iceberg-rust\ to execute pre-planned file scan tasks, handling Merge-On-Read delete application and exposing scan metrics. The Iceberg write operator (\IcebergWriteExec\) includes a custom location generator to ensure partition paths match \iceberg-java\ formatting, a dictionary encoding chooser to align with \parquet-mr\'s first-page decisions, and a partition value calculator that uses Comet's temporal kernels to match \iceberg-java\'s time transform behavior. Additionally, the CSV scan operator (\CsvScanExec\) provides native CSV reading capabilities by configuring a DataFusion \CsvSource\ with options like delimiter, quote, and escape characters.

native/core/src/execution/operators · high confidence

Native Parquet scans now support Azure Blob Storage and location-scoped S3 credentials

Native Parquet scans can now read from Azure Blob Storage by translating Hadoop ABFS configurations (e.g., \fs.azure.\*\) into \object\_store\ options, supporting authentication via environment variables, account-scoped keys, and SAS tokens. For S3, the system now supports location-scoped credentials, allowing different paths within a bucket to use distinct credentials via a \LocationScopedObjectStore\ that routes requests based on policy locations. Additionally, S3-compliant filesystem aliases (like \blob://\) can be configured to route through the native S3 object store, with support for hostless URLs where the bucket is in the path. Also in this area: Improved S3 credential forwarding and JVM UDF filter handling in native Parquet scans.

native/core/src/parquet/objectstore · high confidence

Native Spark array functions now support insertion, position lookup, slicing, overlap checks, zipping, and flattening

The native execution engine now implements several Spark-compatible array functions in \native/spark-expr/src/array\_funcs\. Users can now insert items into arrays (\array\_insert\), find the position of an element (\array\_position\), slice arrays (\array\_slice\), check for overlapping elements between two arrays (\arrays\_overlap\), zip multiple arrays into structs (\arrays\_zip\), and flatten nested arrays (\flatten\). These additions also include extracting fields from struct arrays (\get\_array\_struct\_fields\) and extracting elements by index (\list\_extract\). The implementations handle Spark-specific behaviors such as three-valued null logic for overlaps, correct handling of negative indices for slices, and proper null propagation for nested structures.

_native/spark-expr/src/array\funcs · high confidence

Native Spark-compatible map lookup and sorting functions

This change introduces native Rust implementations for Spark's \GetMapValue\ (map lookup) and \MapSort\ expressions in the \map\_funcs\ module. The new \SparkMapExtract\ function replaces the default DataFusion behavior for map lookups (\element\_at\ and \m\[k\]\) by returning the matched value directly instead of a one-element list, and uses vectorized Arrow equality checks to significantly improve performance for constant-key lookups. Additionally, the \spark\_map\_sort\ function provides a native implementation to sort map entries by key in ascending order, featuring optimized paths for single-entry maps and string-keyed maps to reduce overhead. These additions allow Comet to execute these map operations natively without falling back to Spark, improving both correctness alignment with Spark's semantics and execution speed.

_native/spark-expr/src/map\funcs · high confidence

Native dynamic filter pushdown for hash joins into Parquet scans

This change introduces native dynamic filter pushdown for hash joins into Parquet scans. It adds new execution operators (\DynamicFilterJoinExec\, \DynamicFilterExec\, \TopKReaderFilterExec\) and supporting modules (\early\, \parquet\_reader\) that connect a hash join's completed build domain to its probe input. The implementation filters probe batches before intermediate joins and attaches live predicates directly to the Parquet reader for row-group pruning, bypassing the need for DataFusion's physical optimizer. It also includes logic to preserve Parquet conversion errors (such as schema mismatches or timestamp overflows) even when filters prune rows, ensuring that invalid data is still reported correctly. Also in this area: Fixes for Parquet timestamp handling, error context, and deletion vector compression.

_native/core/src/execution/operators/dynamic\_filter, native/core/src/execution/operators/dynamic\filter/join · high confidence

Native expression builders for arithmetic, bitwise, temporal, and random functions

The \native/core/src/execution/expressions\ module now provides native builders for a broad set of Spark expressions, including arithmetic (add, subtract, multiply, divide, remainder, unary minus), bitwise (and, or, xor, shifts), comparison (eq, neq, lt, gt, null-safe variants), logical (and, or, not), null checks (is null, is not null), string operations (like, rlike, from\_json), temporal functions (hour, minute, second, unix\_timestamp, date\_trunc), and random generators (rand, shuffle, uuid, randstr, randn). These builders translate Spark protobuf expressions into DataFusion physical expressions, enabling native execution of these operations and improving query performance.

native/core/src/execution/expressions · high confidence

Native implementation of Iceberg system functions (bucket, truncate, years, months, days, hours)

This change introduces native Rust implementations for Iceberg's Spark system functions—\bucket\, \truncate\, \years\, \months\, \days\, and \hours\—within the \native/spark-expr/src/iceberg\_funcs\ module. By reproducing Iceberg's Java partition transform logic exactly, these functions ensure that row partitioning (hash distribution, sort orders, and filters) produces identical results whether computed by Comet or the standard Iceberg writer, maintaining compatibility for hidden partitioning scenarios.

_native/spark-expr/src/iceberg\funcs · high confidence

Native implementation of LPAD, RPAD, and read-side padding functions

This change introduces native Rust implementations for the Spark \lpad\, \rpad\, and \read\_side\_padding\ string functions within the \static\_invoke\ module. These functions replace the previous DataFusion defaults to provide correct Spark-compatible behavior, specifically handling Unicode padding, supporting dictionary-encoded arrays, and allowing the length argument to be a column rather than just a literal. The implementation is organized under \char\_varchar\_utils\ and exposes \spark\_lpad\, \spark\_rpad\, and \spark\_read\_side\_padding\ for use by the query engine.

_native/spark-expr/src/static\invoke · high confidence

Native implementation of Spark JSON functions

The \native/spark-expr/src/json\_funcs\ module now provides native Rust implementations for key Spark JSON functions, replacing or supplementing previous logic. This includes \from\_json\ for parsing JSON strings into structured types, \json\_array\_length\ for determining the size of JSON arrays, and \to\_json\ for converting structured data into JSON strings. These changes enable more efficient, native execution of JSON operations within the DataFusion-based engine.

_native/spark-expr/src/json\funcs · high confidence

Native implementation of Spark-compatible ISNAN and RLIKE predicate functions

The native execution engine now includes dedicated Rust implementations for the Spark \ISNAN\ and \RLIKE\ predicate functions, replacing previous fallbacks or incomplete support. The \ISNAN\ function correctly handles Float32 and Float64 arrays and scalars, ensuring that null values are treated as false in accordance with Spark semantics. The \RLIKE\ function provides a physical expression that supports Utf8, LargeUtf8, and Utf8View string types, as well as dictionary-encoded arrays, using a pre-compiled regex pattern for performance. These changes are located in the \native/spark-expr/src/predicate\_funcs\ module, which organizes these specific Spark grouping expressions.

_native/spark-expr/src/predicate\funcs · high confidence

Native implementation of non-deterministic functions (UUID, shuffle, randn)

This change introduces the internal Rust infrastructure required to support native, Spark-compatible non-deterministic functions. It adds a bit-for-bit port of the Mersenne Twister PRNG (seeded per partition to match Spark's behavior) and a utility framework for evaluating batch random values while maintaining state across partitions. This foundation enables the native implementation of \uuid()\, \shuffle\, and \randn\ expressions, ensuring their output matches Apache Spark's determinism and distribution guarantees.

_native/spark-expr/src/nondetermenistic\funcs/internal · high confidence

Native implementations for Spark datetime functions

The native engine now includes native implementations for several Spark datetime expressions, including \date\_diff\, \date\_from\_unix\_date\, \date\_trunc\, \dayname\, \monthname\, \dayofweek\, \weekday\, \hour\, \minute\, \second\, \hours\ (V2 partition transform), \make\_date\, \make\_interval\, and \make\_time\. These additions replace previous fallback or less optimized paths, providing direct Spark-compatible logic for date arithmetic, truncation, name extraction, and time construction.

_native/spark-expr/src/datetime\funcs · high confidence

Native implementations for base64, JSON extraction, Levenshtein, and regex functions

This change introduces native Rust implementations for several Spark string functions in the \spark-expr\ module, replacing slower or non-native paths. Users gain support for \base64\ encoding (with optional MIME chunking) and \get\_json\_object\ (extracting values via JSONPath). The \levenshtein\ function now supports an optional distance threshold for optimized computation, while \regexp\_extract\ and \regexp\_extract\_all\ provide native regex extraction with a pattern cache to avoid recompiling regular expressions per batch. Additionally, \concat\_ws\ is implemented to handle scalar subqueries correctly, and the \contains\ function is optimized for scalar needle patterns.

_native/spark-expr/src/string\funcs · high confidence

Native implementations for new and existing aggregate functions

This change adds native Rust implementations for several Spark-compatible aggregate functions in the \agg\_funcs\ module, including \approx\_percentile\ (using a Greenwald-Khanna port), \approx\_count\_distinct\ (using a HyperLogLog++ port with Spark-compatible hashing and bias tables), and \correlation\/\covariance\ (with Spark-compatible state fields and null-on-divide-by-zero behavior). It also introduces native \GroupsAccumulator\ implementations for \collect\_list\ and \collect\_set\ to optimize grouped aggregation performance, and adds native \avg\ and \avg\_decimal\ accumulators that handle type casting and decimal precision scaling consistent with Spark.

_native/spark-expr/src/agg\funcs · high confidence

Native implementations of Spark nondeterministic functions

This change introduces native Rust implementations for several Spark nondeterministic functions within the \nondetermenistic\_funcs\ module, ensuring bit-for-bit compatibility with Spark's random number generators. The update adds support for \monotonically\_increasing\_id\ (using atomic offsets), \rand\ and \randn\ (using Spark's XORShift and Marsaglia polar methods), \randstr\ (alphanumeric string generation), \shuffle\ (list shuffling via Mersenne Twister), \uuid\ (RFC 4122 v4 generation), and \BernoulliCellSampler\ (for sampling without replacement). These expressions are designed to maintain stateful, per-partition evaluation semantics consistent with Spark, allowing users to run these nondeterministic operations natively with predictable, Spark-compatible results.

_native/spark-expr/src/nondetermenistic\funcs · high confidence

Native memory usage observability and allocator selection

The native core library now provides process-wide accounting of bytes allocated by the Rust global allocator, exposing the current balance via \current\_balance()\ for integration with executor memory usage logs and tracing. This is achieved by wrapping the selected backend allocator (jemalloc, mimalloc, or system) in an \AccountingAllocator\ that tracks allocations with minimal thread-local overhead. Additionally, the library now exposes a JNI method \isFeatureEnabled\ to check if specific features like jemalloc or hdfs-opendal are compiled in, and prints the native library version upon initialization.

native/core/src · high confidence

Native operators moved to dedicated crate with expanded coverage and benchmarks

The self-contained native execution operators have been moved from the core crate into a new \native/operators\ crate. This location now provides the native implementations for \ExpandExec\, \ExplodeExec\, \CometFilterExec\, \CometProjectionExec\, \RangeExec\, \PartitionedRankLimitExec\, and \SampleExec\. The \ExplodeExec\ implementation includes specialized performance paths for unnesting, while \ExpandExec\ now correctly derives output schemas by widening nested nullability across all projections. Additionally, micro-benchmarks have been added for the \ExplodeExec\ and window operators to measure their specific costs.

native/operators · high confidence

Native support for Spark UnsafeMap and UnsafeArray in columnar shuffle

The native shuffle implementation now supports converting complex nested types—specifically Maps and Arrays—stored in JVM Unsafe memory directly to Arrow structures. This change introduces new Rust modules (\list.rs\ and \map.rs\) that handle the parsing of Spark's UnsafeArray and UnsafeMap layouts, enabling efficient bulk appending of these types during JVM columnar shuffle operations without requiring intermediate serialization.

_native/shuffle/src/spark\unsafe · high confidence

Native temporal kernel implementations for date and timestamp operations

The \native/spark-expr/src/kernels\ module now contains native Rust implementations for temporal operations, including optimized date truncation (year, quarter, month, week) and timezone-aware timestamp conversions. These kernels handle edge cases such as Daylight Saving Time transitions and schema consistency when converting between timezones, providing the underlying logic for date and timestamp expressions in the Spark engine.

native/spark-expr/src/kernels · high confidence

New CI validation scripts for benchmarks, configuration, and documentation

Added a suite of Python and shell scripts in dev/ci to enforce CI invariants and prevent silent failures. check-benchmark-runner.py validates that the benchmark runner discovers the expected suites and correctly handles selection filters. check-ci-config.py verifies that the change-routing policy, event policies, required-check aggregation, and artifact naming remain consistent with the workflow definitions. check-mermaid.py ensures that documentation diagrams render correctly and are present in the built site, preventing missing diagrams from going unnoticed. check-suites.py confirms that all Scala test suites are included in the Linux and macOS PR workflows. check-working-tree-clean.sh ensures the repository state is clean before proceeding. compute-changes.py replaces the external paths-filter action with an internal script to determine which jobs should run based on changed files. linux-test-profiles.py and local-ci-config.py support local CI execution and profile selection. These changes improve CI reliability by catching configuration drift and broken pipelines early. Also in this area: CI harness updates for native UDFs and Iceberg scheduler failure testing; CI pipeline optimization and test stability improvements; Enable Spark 4.2 testing and update version snapshots.

dev/ci · high confidence

New analyze\_trace binary for native memory leak detection

A new command-line tool, \analyze\_trace\, has been added to the native common module. It analyzes Comet Chrome trace event logs to detect native memory leaks by comparing process-wide allocation counters (such as \native\_allocated\ or \jemalloc\_allocated\) against the total memory reserved by Comet's memory pools. The tool identifies points where allocated bytes exceed pool reservations, reporting peak excess and sample violations to help users diagnose memory issues.

native/common/src/bin · high confidence

New developer tooling and build configuration scripts

Added several new scripts and configuration files to the \dev/\ directory to support local development and release processes. This includes \local-ci.sh\ for running Spark SQL and Iceberg CI workflows locally, \generate-release-docs.sh\ for freezing documentation content onto release branches, and \regenerate-golden-files.sh\ for updating plan stability test fixtures. Additionally, \cargo.config\ configures specific linkers for Apple Silicon and Intel Macs, \checkstyle-suppressions.xml\ provides an empty suppression list for Java style checks, \ensure-jars-have-correct-contents.sh\ validates JAR artifacts against an allowed class list, and \verify-contrib-delta-gate.sh\ ensures Delta dependencies do not leak into default builds.

dev · high confidence

New exception hierarchy for Comet native and runtime errors

The common module now introduces a dedicated exception hierarchy to better distinguish and handle errors originating from the Comet native execution layer. New classes include CometRuntimeException as the base for general runtime issues, CometNativeException for native-side errors, and specific subclasses like CometOutOfMemoryError, CometShuffleSizeLimitException, and ParquetRuntimeException for targeted error handling. Additionally, CometQueryExecutionException is added to wrap JSON-encoded error details from native execution, enabling more precise error classification and recovery in Spark applications.

common · high confidence

New native Delta Lake scan contrib module for Comet

Adds a new \contrib/delta-spark\ module that enables native execution of Delta Lake table scans via the \spark.comet.scan.delta.enabled\ configuration flag. This feature allows Delta reads to leverage Comet's native Parquet scan capabilities, including row-group and page-index pruning, filter pushdown, and deletion vector support, while ensuring that scans spanning multiple object stores or using unsupported metadata columns correctly fall back to Spark's reader. The module includes the necessary service provider registrations, configuration definitions, and a comprehensive test suite covering basic reads, partitioned tables, time travel, and multi-bucket scenarios.

contrib/delta-spark · high confidence

New native JNI bridge crate for Apache DataFusion Comet

A new \native/jni-bridge\ Rust crate has been introduced to centralize the JNI interaction layer for Apache DataFusion Comet. This crate provides the specific bindings and helper structures required for native execution to communicate with the JVM, including handles for \CometExec\ (scalar subqueries), \CometMetricNode\, \CometTaskMemoryManager\, \CometUdfBridge\, and \CometShuffleBlockIterator\. It also introduces support for pluggable S3 credentials via \CometS3CredentialDispatcher\, case-insensitive schema resolution through \CometSchemaUtils\, and task-owned remote shuffle callbacks via \JavaShufflePartitionPusher\. Additionally, it standardizes error handling with a comprehensive \CometError\ enum and ensures reliable panic backtraces are captured and propagated across the FFI boundary.

native/jni-bridge · high confidence

Support for JVM User-Defined Functions in native execution

Users can now execute Scala and Java UDFs using the native Comet execution engine instead of falling back to the JVM. This change introduces a new \JvmScalarUdfExpr\ component that delegates evaluation to JVM-side \CometUDF\ implementations via JNI. It ensures correct behavior by propagating the Spark task's ClassLoader and TaskContext (including partition and execution context IDs) to the JVM bridge, allowing UDFs to access partition-sensitive built-ins like \Rand\ or \Uuid\. Additionally, it optimizes performance by sending literal arguments as length-1 arrays to avoid unnecessary copying and by zeroing sliced boolean offsets before exporting data to the JVM.

_native/spark-expr/src/jvm\udf · high confidence

Support for shuffling empty-schema record batches

The native shuffle partitioners now handle zero-column schemas (such as those produced by COUNT(\*) after column pruning) via a new EmptySchemaShufflePartitioner. This partitioner accumulates the total row count from input batches and writes a single zero-column IPC batch to partition 0, ensuring that shuffle operations on empty-schema data complete correctly without errors.

native/shuffle/src/partitioners · high confidence

Architecture

Native planner refactored into modular operator and expression registries

The native query planner has been restructured to use dedicated modules and registries for planning operators and expressions. This change introduces an expression registry to dispatch Spark expression creation, separate builders for operators like HashAgg, Explode, MergeRows, and joins, and contrib scan handlers for Delta and Lance formats. This modularization improves code organization and maintainability while preserving existing native execution capabilities.

native/core/src/execution/planner · high confidence

Protobuf definitions moved to a separate crate with dedicated build script

The protocol buffer definitions for Spark physical query plans (expr, metric, partitioning, operator, and config) have been moved into a dedicated native/proto crate. This change introduces a build script that automatically generates Rust code from these .proto files into a src/generated directory during compilation, streamlining the integration of query plan intermediates for the Apache DataFusion Comet project.

native/proto · high confidence

Behavioural changes

11 commits (5 fixes) modifying native/spark-expr/src/bloom\_filter

A change to existing behaviour in native/spark-expr/src/bloom\_filter — 11 commits (5 fixs), 6 files: “chore: Implement BloomFilterMightContain as a ScalarUDFImpl (\#1954)”; “feat: convert OneRowRelation to Arrow by default (\#6666)”; “chore: upgrade to DataFusion 50.0.0, Arrow 56.1.0, Parquet 56.0.0 among others (\#2286)”; and 8 more (refs \#1954, \#6666, \#2286, \#7173, \#4390, …).

_native/spark-expr/src/bloom\filter · medium confidence · unverified

18 commits (8 fixes) modifying native/spark-expr/src/conditional\_funcs

A change to existing behaviour in native/spark-expr/src/conditional\_funcs — 18 commits (8 fixs), 5 files: “fix: if expr nullable (\#2217)”; “fix: evaluate the array\_append item on rows whose array is NULL (\#7025)”; “perf: compare floats in Spark's order without normalizing copies of the operands (\#6691)”; and 15 more (refs \#2217, \#7025, \#6691, \#6763, \#1514, …).

_native/spark-expr/src/conditional\funcs · medium confidence · unverified

19 commits (8 fixes) modifying comet-spark-spark${spark.version.short}\_${scala.binary.version}

A change to existing behaviour in comet-spark-spark${spark.version.short}\_${scala.binary.version} — 19 commits (8 fixs), 10 files: “chore: remove dead native\_iceberg\_compat code path (\#4363)”; “perf: write every partition of a JVM shuffle spill in one native call (\#7065)”; “chore: remove dead vector classes left over from native\_iceberg\_compat removal (\#4416)”; and 16 more (refs \#4363, \#7065, \#4416, \#5560, \#4382, …).

comet-spark-spark${spark.version.short}\${scala.binary.version} · medium confidence · unverified_

2 commits (0 fixes) modifying native/native-executors

A change to existing behaviour in native/native-executors — 2 commits, 10 files: “feat: add comet-scheduler and comet-executor for Comet Native Executors (\#7299)”; “feat: add cluster settings to the Comet Native Executor binaries (\#7420)” (refs \#7299, \#7420).

native/native-executors · medium confidence · unverified

5 commits (0 fixes) modifying native/spark-expr/src/url\_funcs

A change to existing behaviour in native/spark-expr/src/url\_funcs — 5 commits, 2 files: “perf: optimize native parse\_url extraction (\#7329)”; “feat: convert OneRowRelation to Arrow by default (\#6666)”; “feat: make parse\_url compatible (\#4413)”; and 2 more (refs \#7329, \#6666, \#4413, \#4893, \#4062).

_native/spark-expr/src/url\funcs · medium confidence · unverified

5 commits (2 fixes) modifying dev/release/comet-rm

A change to existing behaviour in dev/release/comet-rm — 5 commits (2 fixs), 2 files: “chore: remove dead code from the release build (\#7524)”; “fix: build release jars from the same commit as the native libraries (\#6377)”; “Add JDK to Docker image for release build (\#3262)”; and 2 more (refs \#7524, \#6377, \#3262, \#1960, \#932).

dev/release/comet-rm · medium confidence · unverified

Enable Comet native execution for Iceberg tests and benchmarks

The Iceberg test suites and benchmarks now run with the Apache DataFusion Comet plugin enabled. This integrates Comet's native scan, shuffle, and write capabilities into the Iceberg test infrastructure, allowing users to validate Iceberg operations against Comet's accelerated execution engine. The changes update build dependencies to include the Comet Spark module and configure Spark sessions in test bases and benchmarks to use the Comet shuffle manager and native Iceberg features.

dev/diffs/iceberg · high confidence

Maven wrapper and network retry configuration added

The project now includes a Maven wrapper (version 3.2.0) using Maven 3.9.6, ensuring consistent builds across environments. Additionally, a new \.mvn/maven.config\ file configures HTTP retry behavior to handle transient network issues more robustly: it increases the retry count to 6, expands retryable status codes to include 408, 500, 502, and 504 (in addition to 429 and 503), sets a TCP connect timeout of 30 seconds, and sets a request timeout of 600 seconds to prevent dead connections from consuming excessive build time.

.mvn · high confidence

Native Parquet scan performance and compatibility improvements

This release introduces an eager page-index reader factory that forces loading the Parquet page index on the first metadata fetch and caching it, eliminating repeated uncached I/O for predicates that do not fully match row-group statistics. It also adds a unified field-name folding module to ensure case-insensitive Parquet reads match Spark's \toLowerCase(Locale.ROOT)\ behavior consistently across the schema adapter, nested conversions, and plan-time projection, while introducing a debug batch stream and execution plan wrapper to print batch details for troubleshooting. Additionally, the scan now supports Parquet Modular Encryption with a Spark KMS integration via a JNI-based key retriever, and improves type conversion by aligning struct casting to Spark's behavior and rejecting duplicate Parquet field names before decoding.

native/core/src/parquet · high confidence

Native S3 credential handling gains location-scoped routing and IRSA throttle resilience

The native S3 layer now supports per-location credential routing, allowing distinct S3 policy locations to map to specific credentials for both native Parquet and Iceberg scans. It introduces a credential bridge that caches credentials until shortly before their reported expiry, reducing STS call frequency. Additionally, the new IRSA web-identity provider prevents transient STS throttles from causing hard 403 errors by implementing retry logic, avoiding silent fallback to node instance roles, and sharing cached credentials across concurrent executors.

native/core/src/cloud/s3 · high confidence

Native cast module refactored into typed submodules

The native cast implementation in \native/spark-expr/src/conversion\_funcs\ has been reorganized from a single monolithic file into distinct modules (\boolean\, \numeric\, \string\, \temporal\, \trim\, \utils\) to improve maintainability and performance. This change introduces specialized casting logic for each data type category, including Spark-compatible boolean-to-timestamp conversion, precise Java-style float/double string formatting, and strict whitespace trimming rules that match Spark's \trimAll\ and \String.trim\ behaviors. The refactoring also adds dedicated handling for date-to-timestamp conversions with correct timezone and DST resolution, and ensures decimal-to-string output matches Spark's legacy and ANSI modes.

_native/spark-expr/src/conversion\funcs · high confidence

Native math functions now support ANSI mode error handling

The native math expression implementations in \native/spark-expr/src/math\_funcs\ have been updated to respect the ANSI evaluation mode. Functions such as \abs\, \ceil\, \floor\, \log\, \modulo\, and arithmetic operations now throw specific errors (e.g., \ARITHMETIC\_OVERFLOW\, \DIVIDE\_BY\_ZERO\, \REMAINDER\_BY\_ZERO\) when \fail\_on\_error\ is enabled, matching Spark's ANSI-compliant behavior. In legacy mode, these functions continue to use wrapping arithmetic or return nulls for invalid operations, ensuring backward compatibility while providing strict error reporting for users who require ANSI SQL compliance.

_native/spark-expr/src/math\funcs · high confidence

Native query plans are now serialized via a new set of Protobuf schemas

The native execution engine now uses a dedicated set of Protobuf definitions (config.proto, expr.proto, literal.proto, metric.proto, operator.proto, partitioning.proto, and types.proto) to serialize Spark plans, expressions, and types to the native side. This replaces the previous serialization approach, enabling richer expression support (including new array, JSON, and aggregate functions), more efficient plan transmission via SQL text interning, and better error reporting with query context.

native/proto/src/proto · high confidence

Native shuffle module restructured with performance and reliability improvements

The native shuffle module has been reorganized into a separate crate, introducing a \PartitionWriter\ interface to decouple partitioning logic from storage. This refactor includes significant performance optimizations: zstd and Arrow IPC compression contexts are now reused across shuffle blocks to reduce allocation overhead, and the IPC schema is encoded once per writer instead of per block. A new \ShuffleCodecContext\ manages these reusable contexts, while a per-thread schema cache with bounded retention limits memory usage during decoding. The module also adds a \bench\_support\ seam for benchmarking partitioning without the write path, implements a \RoundRobinStrategy\ with both hash-based and positional row-group placement, and introduces \SchemaAlignExec\ to correct type and nullability drift before partitioning to ensure consistent hash-based routing. Remote shuffle support includes dictionary unpacking and schema validation to safely handle data from the JVM.

native/shuffle/src · high confidence

Native shuffle plans now support partition writer destinations

The native protocol layer has been updated to include partition writer destinations within shuffle plans. This change allows the system to specify where shuffle data should be written (e.g., local files or remote shuffle services) directly in the plan structure, replacing the previous mechanism that relied on separate index paths. This enables more flexible and efficient data shuffling strategies by explicitly defining output destinations at the protocol level.

native/proto/src · high confidence

Native struct field extraction now correctly propagates parent nullability

The native struct expression implementation has been reorganized into a dedicated \struct\_funcs\ module, introducing \CreateNamedStruct\ and \GetStructField\ expressions. A key behavioral fix ensures that extracting a field from a struct correctly reports the field as nullable if the parent struct is nullable, even when the field itself is defined as non-nullable. This aligns with Spark semantics and prevents downstream Arrow validation errors during shuffles or sorts where a non-nullable field of a nullable struct would otherwise incorrectly claim to contain no nulls.

_native/spark-expr/src/struct\funcs · high confidence

Normalize Variant arrays at the native Parquet boundary

The native Parquet scan layer now normalizes Variant arrays before they are exposed to the query engine. This ensures that storage types accepted by Spark's Parquet reader (such as unsigned integers, Decimal256, millisecond timestamps, and fixed-size binary/list types) are converted to Arrow types supported by the Variant compute layer, and that UUID extension types are explicitly rejected. This change allows direct Variant projection in native Parquet scans without requiring the query engine to handle these storage-level discrepancies.

_native/core/src/parquet/cast\column · high confidence

Optimized decimal overflow handling and fused rescale checks

The native decimal math expressions now use a shared fast path that skips heavy validation when no overflow occurs, significantly improving performance for typical decimal arithmetic. Additionally, the \DecimalRescaleCheckOverflow\ expression fuses scale conversion and precision validation into a single pass, replacing the previous two-step pattern. Overflow behavior in ANSI mode is now more precise, correctly identifying and reporting the first offending value rather than a generic error, and the \MakeDecimal\ expression properly honors the \fail\_on\_error\ flag to either raise an error or return null on overflow.

_native/spark-expr/src/math\funcs/internal · high confidence

Refactored hash function implementation into a modular structure

The hash function logic has been reorganized into a dedicated \hash\_funcs\ module, splitting the code into separate files for Murmur3, xxHash64, and utilities. This change introduces a new \spark\_xxhash64\ entry point that delegates to DataFusion's \SparkXxhash64\ implementation when arguments are compatible (e.g., default seed 42, supported types), while retaining a native kernel for cases where the upstream implementation is incompatible (e.g., non-default seeds, structs with null masks, dictionaries in lists). The Murmur3 implementation remains native but is now structured to handle dictionary arrays and null masks more explicitly.

_native/spark-expr/src/hash\funcs · high confidence

Refactored memory pool architecture with new fair pool and task sharing

The memory pool subsystem has been refactored to introduce a new \fair\_unified\ pool type that enforces per-consumer memory limits, replacing the previous default \greedy\_unified\ strategy. The implementation now supports task-shared memory pools via an RAII guard, ensuring pools are shared across native execution contexts within the same task attempt and cleaned up automatically. Additionally, the codebase now includes a logging pool wrapper for debugging, improved overcommit handling to prevent panics during memory growth, and stricter checks for fair pool reservations.

_native/core/src/execution/memory\pools · high confidence

Refactored shuffle writers with partition abstraction and performance optimizations

The shuffle write path has been restructured to decouple partitioning logic from storage backends via a new PartitionWriter interface, introducing dedicated writers for local and RSS destinations. Performance is improved by reusing per-partition scratch buffers to reduce allocations, bypassing the BatchCoalescer for already-sized batches to avoid unnecessary copying, and encoding the Arrow IPC schema once per writer instead of per block. Additionally, a new Checksum module supports CRC32, Adler32, and CRC32C algorithms for data integrity verification during shuffle writes.

native/shuffle/src/writers · high confidence

Safer runtime filter application during Parquet schema adaptation

The Parquet reader now validates that runtime filter columns can be safely resolved against the file's physical schema before applying the filter. If a column reference is unresolved, stale, or otherwise unsafe to adapt, the runtime filter is disabled for that file while static predicates continue to execute. This prevents conversion errors and incorrect row filtering when schema mappings (such as case or field-ID remapping) do not align with the filter's expectations.

_native/core/src/execution/operators/dynamic\_filter/parquet\reader · high confidence

A new symbolic link at .claude/skills has been added, redirecting to the ../.ai/skills directory. This change establishes a reference from the .claude location to the .ai skills, likely supporting the transition or coexistence of vendor-neutral configurations.

.claude · medium confidence

Fixes

Fixes missing PruningMetrics and Ratio values in native scan metrics

The native metrics collection now correctly exposes DataFusion 54's PruningMetrics and Ratio types to the Spark side. Previously, these metrics were surfaced as zero because DataFusion reserves their aggregation to MetricsSet; the new utils module expands them into individual integer fields (e.g., pruned/matched for pruning, part/total for ratios) so that parquet scan filtering statistics are visible in Spark metrics.

native/core/src/execution/metrics · high confidence

Fixes native linking failures caused by stale JDK library paths

The native core build script now explicitly emits the JDK's libjvm search path during compilation. This ensures that the linker can always locate libjvm, preventing build failures on CI runners or environments where the JDK installation path changes or is cached incorrectly from previous builds. Also in this area: 1 commit (1 fix) fixing native.

native/core · medium confidence · unverified

Native float comparisons now match Spark's SQL ordering for NaN and -0.0

The native engine now handles -0.0 and NaN values in comparisons, sorting, and aggregation functions (such as min, max, greatest, least, array\_remove, and sort\_array) consistently with Spark SQL. Previously, native code may have used standard IEEE 754 ordering, causing discrepancies where -0.0 and 0.0 were treated as distinct or NaN values were ordered incorrectly. This change introduces a dedicated \float\_semantics\ module that normalizes floats (converting -0.0 to 0.0 and canonicalizing NaN) for grouping and hashing, and uses Spark-specific comparison logic for sorting and equality checks, ensuring that native results match Spark's behavior.

_native/spark-expr/src/float\semantics · high confidence

Native shuffle now reports spill metrics correctly

Users will now see accurate native shuffle read and write metrics, including spill information, in the Spark UI and logs. Previously, these metrics were either missing or incorrect, making it difficult to monitor shuffle performance and identify spill-related bottlenecks during query execution.

spark · high confidence

Test coverage

Added benchmarks for native shuffle read and write performance; Added native benchmarks for scalar and aggregate functions; Added test utility for generating unique temporary files; Added tests for Parquet Variant projection support; Added tests for Spark-compatible expression registration and behavior; Added tests for native scan planner edge cases and multi-store partition validation; New benchmark suite for native performance and allocation overhead.

Dependencies

Initial release scaffolding and dependency configuration for Comet 1.1.0

This change introduces the foundational build configuration for the 1.1.0 release, establishing the Maven parent POM and the Rust workspace structure. It defines the core native crates (core, spark-expr, common, proto, jni-bridge, shuffle) and pins key dependencies, including DataFusion 55.1.0, Arrow 59.2.0, and Iceberg-rust (pinned to revision bb1e4a4). The diff also adds build-gate stubs for optional contrib integrations (Delta, Lance) and configures CI tooling, such as Gradle-based Iceberg test sharding and documentation requirements.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 68 → 66 (-2.1)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 60 → 61 (+1.3)
  • Architecture 97 → 96 (-0.6)
  • Maturity 77 → 78 (+1.0)
  • Readiness 74 → 63 (-11.9)
  • Security 73 → 69 (-3.7)
  • Performance 81 → 94 (+13.5)

Resolved (121)

  • Boundary-crossing change coupling: error.rs ↔ ShimSparkErrorConverter.scala (native/common/src/error.rs)
  • Boundary-crossing change coupling: error.rs ↔ ShimSparkErrorConverter.scala (native/common/src/error.rs)
  • Boundary-crossing change coupling: lib.rs ↔ NativeBase.java (native/core/src/lib.rs)
  • ColumnarToRowContext::write_column_fixed_width (cognitive 17) (native/core/src/execution/columnar_to_row.rs)
  • ColumnarToRowContext::write_column_fixed_width (cyclomatic 18) (native/core/src/execution/columnar_to_row.rs)
  • CometDataWritingCommand.getSupportLevel (cognitive 19) (spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala)
  • Duplicated block (10 lines × 2) (native/common/src/utf8.rs)
  • Duplicated block (10 lines × 2) (native/shuffle/src/remote_schema.rs)
  • Duplicated block (10 lines × 2) (native/spark-expr/src/array_funcs/array_contains.rs)
  • Duplicated block (10 lines × 2) (spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenInput.scala)
  • Duplicated block (10 lines × 3) (spark/src/main/spark-3.5/org/apache/comet/serde/operator/CometMergeRows.scala)
  • Duplicated block (11 lines × 2) (native/core/src/parquet/cast_column/variant.rs)
  • Duplicated block (11 lines × 2) (native/spark-expr/src/agg_funcs/sum_int.rs)
  • Duplicated block (11 lines × 2) (native/spark-expr/src/datetime_funcs/make_date.rs)
  • Duplicated block (11 lines × 2) (spark/src/main/spark-3.4/org/apache/comet/shims/ShimCometSparkSessionExtensions.scala)
  • Duplicated block (11 lines × 3) (native/spark-expr/src/agg_funcs/sum_int.rs)
  • Duplicated block (11–14 lines × 3) (spark/src/main/scala/org/apache/spark/sql/comet/CometMetricNode.scala)
  • Duplicated block (11–15 lines × 3) (spark/src/main/scala/org/apache/spark/sql/comet/CometMetricNode.scala)
  • Duplicated block (12 lines × 2) (native/core/src/execution/columnar_to_row.rs)
  • Duplicated block (12 lines × 2) (native/core/src/parquet/cast_column/variant.rs)
  • …and 101 more

New (315)

  • Boundary-crossing change coupling: error.rs ↔ ShimSparkErrorConverter.scala (native/common/src/error.rs)
  • Boundary-crossing change coupling: parquet_writer.rs ↔ CometNativeWriteExec.scala (native/core/src/execution/operators/parquet_writer.rs)
  • CometIcebergNativeScan.hdfsEncryptionZoneFallback (cognitive 20) (spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala)
  • CometMakeDate.unsupportedParentReason (cognitive 32) (spark/src/main/scala/org/apache/comet/serde/datetime.scala)
  • CometMakeDate.unsupportedParentReason (cyclomatic 20) (spark/src/main/scala/org/apache/comet/serde/datetime.scala)
  • CometNativeShuffleWriter.writeInternal (cyclomatic 16) (spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala)
  • CometScanRule.validateIcebergFileScanTasks (cyclomatic 17) (spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala)
  • CometWindowExec.convert (cognitive 17) (spark/src/main/scala/org/apache/spark/sql/comet/CometWindowExec.scala)
  • Duplicated block (10 lines × 2) (native/common/src/utf8.rs)
  • Duplicated block (10 lines × 2) (native/shuffle/src/remote_schema.rs)
  • Duplicated block (10 lines × 2) (native/spark-expr/src/array_funcs/array_contains.rs)
  • Duplicated block (11 lines × 2) (native/core/src/parquet/cast_column/variant.rs)
  • Duplicated block (11 lines × 2) (native/shuffle/src/spark_unsafe/list.rs)
  • Duplicated block (11 lines × 2) (native/spark-expr/src/agg_funcs/covariance.rs)
  • Duplicated block (11 lines × 3) (native/spark-expr/src/agg_funcs/sum_int.rs)
  • Duplicated block (12 lines × 2) (native/core/src/parquet/cast_column/variant.rs)
  • Duplicated block (12 lines × 2) (native/spark-expr/src/agg_funcs/sum_int.rs)
  • Duplicated block (12–13 lines × 2) (spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala)
  • Duplicated block (12–15 lines × 2) (native/core/src/execution/planner/window.rs)
  • Duplicated block (12–19 lines × 2) (spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala)
  • …and 295 more

Changes since last survey

  • 97 commits — 62 feature/other, 35 fixes

By area

  • spark/src — 42 commits
  • docs/source — 16 commits
  • native/core — 7 commits
  • native/spark-expr — 7 commits
  • .github/workflows — 6 commits
  • native/native-executors — 6 commits
  • (root) — 4 commits
  • native/shuffle — 2 commits
  • spark/pom.xml — 2 commits
  • .ai/skills — 1 commit
  • contrib/delta — 1 commit
  • dev/generate-release-docs.sh — 1 commit
  • native/.cargo — 1 commit
  • native/jni-bridge — 1 commit

Notable commits

  • fix: fix: align nested collection buffer nullability before spilling (#5903)
  • fix: fix: apply the session's Comet configs in map stages of jobs outside a SQL execution (#7302)
  • fix: fix: copy rows retained by identity broadcasts (#7260)
  • fix: fix: declare required child distribution on aggregate, window and join operators (#7231)
  • fix: fix: declare the input order of a native window over a sorted cache (#7256)
  • fix: fix: decode invalid UTF-8 in nested Parquet BINARY values read as strings (#7106)
  • fix: fix: don't treat planner-inserted casts as Parquet schema conversions (#7159)
  • fix: fix: evaluate the right operand of AND, OR and IN only for rows the left side has not decided (#7326)
  • fix: fix: fall back for brotli Iceberg writes when Hadoop's BrotliCodec cannot be loaded (#6996)
  • fix: fix: fall back to Spark when the driver cannot use the ABFS authentication (#7221)
  • fix: fix: give zstd the real destination capacity when compressing cached batches (#7307)
  • fix: fix: include licenses for bundled lz4_flex and Zstandard (#7382)
  • fix: fix: keep URL passwords and SAS queries out of Azure and object store errors, keys and logs (#7127)
  • fix: fix: locate the test UDF library under Cargo's Rust 1.100 build layout (#7617)
  • fix: fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171)
  • fix: fix: merge all Bloom filter states and respect sliced offsets (#7173)
  • fix: fix: pass the session batch size to the native FilterExec (#7280)
  • fix: fix: pick the signed zero Spark ranks first in native percentile (#6928)
  • fix: fix: point the external sort OOM error at Comet's memory settings (#7104)
  • fix: fix: prefer Unsupported when combining nested cast support (#6501)
  • …and 77 more

Architecture

  • Unchanged — 0 containers · 2 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

apache/datafusion-comet was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit fd82542adf69b64ed20ecb82c73493b36273c8e4 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.