Skip to content
CAI
Software that uses CAICheck a score

holdenk/spark-testing-base

61.8

Adequate · 14 November 2026

4.2k

lines of production code

Scala

with Python

7

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Added local sbt launch scripts for CI usage

The build directory now includes the \sbt\ launcher script and its supporting library \sbt-launch-lib.bash\. This allows the project to download and run the specific sbt version defined in \project/build.properties\ without requiring sbt to be pre-installed on the system, facilitating consistent builds in CI environments.

build · high confidence

Initial release of spark-testing-base Python module

This change introduces the initial version (0.11.1) of the \spark-testing-base\ Python library, providing a framework for testing Spark applications. The package includes a \setup.py\ configuration that declares dependencies on \findspark\ and \pytest\, along with test requirements for \nose\ and \coverage\. It also adds a \run-tests\ script to facilitate test execution by automatically locating the Spark home directory and running tests with coverage reporting, alongside standard project files like the Apache 2.0 license and manifest template.

python · high confidence

Initial release of the SparkTestingBase Python testing library

This change introduces the \sparktestingbase\ Python package, providing a suite of base test classes to simplify writing tests for Apache Spark applications. It includes \SparkTestingBaseTestCase\ and \SparkTestingBaseReuse\ for managing SparkContext lifecycles, \SQLTestCase\ for testing Spark SQL with DataFrame comparison helpers, and \StreamingTestCase\ for testing Spark Streaming with utilities to queue input and collect results. The library also handles PySpark path configuration via \findspark\ or \SPARK\_HOME\ and reduces log noise from py4j.

python/sparktestingbase · high confidence

Behavioural changes

Added default log4j configuration for Spark environments

A new log4j.properties file has been introduced to standardize logging behavior for Spark applications. By default, the root logging level is set to WARN, and output is directed to the console error stream. The configuration also explicitly sets specific log levels for third-party libraries (such as Jetty and Parquet) and Spark REPL components to reduce noise, and addresses SPARK-9183 by setting the Hive metastore handler to FATAL to suppress messages regarding nonexistent UDFs.

log4j · high confidence

Project documentation and configuration updates

The repository now includes standard governance and contribution files (CODE\_OF\_CONDUCT.md, CONTRIBUTING.md, SECURITY.md) and a dedicated release notes file (RELEASE\_NOTES.md). The README has been expanded with installation instructions for Scala and Python, memory configuration guidance for JDK 17+, and notes on disabling parallel test execution. Additionally, a \.codecov.yml\ file has been added to exclude specific source directories and tests from coverage reports, and a \.scala-steward.conf\ file has been created to ignore updates for the \jetty-util\ dependency.

(repo-wide) · high confidence

Unified Spark 2.4 testing implementation in core

The core testing library now consolidates its Spark 2.x support into a single 2.4 implementation, replacing previous fragmented 2.2 and 2.3 versions. This change introduces a unified set of Scala and Java source files under the 2.4 directory structure, including new generators for DataFrames, Datasets, and RDDs, as well as updated base traits for streaming and dataset assertions, ensuring consistent testing capabilities across the supported 2.4.x Spark versions.

core/src/main · high confidence

Test coverage

Added Java test fixtures and dataset/RDD comparison tests for Spark 3.0; Added Kafka 0.8 test utilities; Added Spark 2.4+ testing infrastructure and Java streaming test coverage; Added Spark 2.4-specific test suite; Added Spark 4.0 preview test suite for expression codegen; Added tests for Spark Connect client integration; Added unit tests for RDD, SQL, and streaming assertions.

Dependencies

Add Spark 4 and Spark Connect support with JDK 17 compatibility

The build configuration now supports Spark 4.x (including preview versions) and integrates the Spark Connect client for distributed DataFrame testing. To ensure compatibility with modern Java runtimes, the build explicitly configures JVM arguments (via --add-opens) for JDK 17 and later, and updates specific dependencies like Netty and zstd-jni for Spark 4. Additionally, a shaded sub-project is introduced to relocate Spark SQL classes from the Connect client, preventing classpath conflicts when used alongside standard Spark SQL components.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 63 → 62 (-1.3)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 65 → 62 (-3.3)
  • Architecture 98 → 98 (+0.0)
  • Maturity 50 → 50 (+0.0)
  • Readiness 79 → 78 (-1.1)
  • Security 87 → 82 (-5.2)
  • Performance 85 → 85 (+0.0)

New (4)

  • Edited copies of one member (3 members, 14+ corresponding lines) (core/src/main/2.4/scala/com/holdenkarau/spark/testing/StreamingSuiteCommon.scala)
  • Edited copy of a member (13 corresponding lines) (core/src/main/2.4/scala/com/holdenkarau/spark/testing/StreamingSuiteBase.scala)
  • Edited copy of a member (14 corresponding lines) (python/sparktestingbase/streamingtestcase.py)
  • Edited copy of a member (34 corresponding lines) (core/src/main/2.4/scala/com/holdenkarau/spark/testing/StreamingActionBase.scala)

Changes since last survey

  • 2 commits — 2 feature/other, 0 fixes

By area

  • (root) — 1 commit
  • project/build.properties — 1 commit

Notable commits

  • change: Add Spark 4.1.1 support (#512)
  • change: Update sbt, sbt-dependency-tree, ... to 1.13.1 (#513)

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

holdenk/spark-testing-base was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 649d3c501facc2704027c52bede940a932fcaff7 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.