Skip to content
CAI
Software that uses CAICheck a score

G-Research/spark-extension

65.6

Adequate · 6 November 2026

7.1k

lines of production code

Scala

with Python

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

This system is a Spark extension library that provides utilities for data comparison, metadata inspection, and transformation. It enables users to diff DataFrames with customizable comparators, compute histograms, generate global row numbers, and read Parquet file metadata. The project supports Scala, Java, and Python, with the Python package bundling the necessary Scala artifacts for seamless integration.

How it got here

2020 — Spark 3.5 upgrade and Python API launch

7 changes.

The project upgraded its core dependencies to Spark 3.5 and Scala 2.13 while introducing a new Python extension API for DataFrame diffing and Parquet metadata inspection. This period also established comprehensive documentation, formatting standards, and test suites for both Scala and Python components to support the new features and build systems.

2022–2026 — Python packaging and multi-version testing

5 changes.

This period focused on releasing the official Python package with integrated build processes and documentation, alongside adding Java integration tests for the Spark Diff library. It also involved establishing test compatibility helpers and directory structures for upcoming Spark 4 and 5 versions, as well as providing examples for installing Python dependencies in Spark environments.

Features

Add example for installing Python dependencies in Spark

A new example located in the \examples/python-deps\ directory demonstrates how to install Python packages (specifically \pandas\ and \pyarrow\) into a Spark environment using the \install\_pip\_package\ API. The example includes a Dockerfile based on Apache Spark 3.5.0, a docker-compose configuration for a local Spark master and worker setup, and a Python script (\example.py\) that initializes a Spark session, installs the dependencies, and executes a pandas UDF.

examples · high confidence

Initial Python package release with integrated build and documentation

The \python/\ directory now contains the official \pyspark-extension\ package, including \setup.py\ and \README.md\. The \setup.py\ script integrates the Scala JAR build process, automatically compiling the Spark extension JAR and bundling it into the Python distribution, while declaring \typing\_extensions\ as a runtime dependency and supporting Python 3.7 through 3.13. The accompanying \README.md\ provides comprehensive installation instructions for PyPI, PySpark API, REPL, and \spark-submit\, and documents available features such as dataset diffing, histogram computation, global row numbering, Parquet metadata inspection, and .Net DateTime conversion.

python · high confidence

Initial release of Python Spark extension API

This change introduces the initial Python API for the G-Research Spark extension, providing new capabilities for data comparison and metadata inspection. Users can now compare DataFrames using the \diff\ module, which supports various modes (ColumnByColumn, SideBySide, etc.), sparse output, and custom comparators for strings, maps, and numeric values with epsilon tolerance. Additionally, the \parquet\ module exposes functions to read detailed metadata from Parquet files, including file-level statistics, schema details, block information, and partition data, with support for controlling parallelism during metadata reads.

python/gresearch · high confidence

New utility functions and diffing capabilities

This release introduces several new capabilities for data processing and comparison. Users can now generate global row numbers with the new \RowNumbers\ transformation, which supports custom ordering and storage levels. A \Histogram\ transformation is added to compute column histograms aggregated by other columns. The diffing API is expanded with new \DiffComparators\ for epsilon, duration, whitespace-agnostic strings, and maps, alongside a new \App\ for command-line diffing of datasets. Additionally, the \Backticks\ utility simplifies quoting column names with special characters, and a \ConditionalCall\ API enables fluent conditional execution of transformations.

src/main · high confidence

Project documentation and configuration files added

The repository now includes a \.scalafmt.conf\ file to enforce code formatting rules (Scala 2.13, max column 120), a \LICENSE\ file adopting the Apache License 2.0, and a \CHANGELOG.md\ documenting the project's history and upcoming changes. Additionally, several documentation files have been added to explain library features: \DIFF.md\ for the dataset diffing API, \CONDITIONAL.md\ for fluent conditional transformations, \GROUPS.md\ for sorted group operations, \HISTOGRAM.md\ for histogram computations, \PARQUET.md\ for reading Parquet metadata, \PARTITIONING.md\ for optimized partitioned writing, \PYSPARK-DEPS.md\ for installing Python dependencies in PySpark jobs, \MAINTAINERS.md\ listing project maintainers, and \RELEASE.md\ detailing the release process.

(repo-wide) · high confidence

Test coverage

Added Java integration tests for Spark Diff and utility functions; Added Python test suite for Spark extension features; Added Scala test suites for new group-by, histogram, partitioned write, and diff features; Added Spark 4 test compatibility helpers; Added Spark 5 test symlink; Added test logging configuration files.

Dependencies

Upgrade to Spark 3.5 and Scala 2.13 with new Python build system

The project has upgraded its core dependencies to Apache Spark 3.5.1 and Scala 2.13.8, replacing the previous Spark 2.4/Scala 2.12 baseline. This change includes adding explicit support for Spark Hive and managing Parquet Hadoop dependencies (version 1.16.0) to ensure compatibility. Additionally, a new Python build configuration (pyproject.toml) using setuptools has been introduced alongside the existing Maven build, and test dependencies have been updated to ScalaTest 3.3.0-SNAP4 and JUnit 4.13.2.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 64 → 66 (+1.6)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 60 → 60 (+0.0)
  • Architecture 100 → 100 (+0.0)
  • Maturity 62 → 66 (+4.0)
  • Readiness 65 → 66 (+1.4)
  • Security 76 → 80 (+4.3)

Resolved (171)

  • Four distinct method names perform the same core operation (comparing two DataFrames) with varying levels of configurability. The naming convention is inconsistent ('diff' vs 'diffwith', 'with_options' suffix vs no suffix), and the distinction between 'diff' and 'diffwith' is unclear (likely related to column naming strategies, but not obvious from signatures). Similarly, 'diff_with_options' and 'diffwith_with_options' are redundant variations.
  • Medium CVE: [GHSA redacted] (pom.xml)
  • Medium CVE: [GHSA redacted] (pom.xml)
  • Outdated (compile): com.github.scopt:scopt_2.13
  • Outdated (compile): org.scala-lang:scala-library
  • Outdated (provided): com.clearspring.analytics:stream
  • Outdated (provided): com.esotericsoftware:kryo-shaded
  • Outdated (provided): com.esotericsoftware:minlog
  • Outdated (provided): com.fasterxml.jackson.core:jackson-annotations
  • Outdated (provided): com.fasterxml.jackson.core:jackson-core
  • Outdated (provided): com.fasterxml.jackson.core:jackson-databind
  • Outdated (provided): com.fasterxml.jackson.datatype:jackson-datatype-jsr310
  • Outdated (provided): com.fasterxml.jackson.module:jackson-module-scala_2.13
  • Outdated (provided): com.github.luben:zstd-jni
  • Outdated (provided): com.google.code.findbugs:jsr305
  • Outdated (provided): com.google.code.gson:gson
  • Outdated (provided): com.google.crypto.tink:tink
  • Outdated (provided): com.google.flatbuffers:flatbuffers-java
  • Outdated (provided): com.google.guava:guava
  • Outdated (provided): com.google.protobuf:protobuf-java
  • …and 151 more

New (3)

  • Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)

Architecture

  • Unchanged — 0 containers · 2 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

G-Research/spark-extension was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 6 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 24fc7308d289e53e0cff2c3793ed4307fbf33b2f — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-410a2463fa58.