Skip to content
CAI
Software that uses CAICheck a score

LucaCanali/sparkMeasure

60.6

Adequate · 14 November 2026

2.8k

lines of production code

Scala

with Python

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

SparkMeasure is a library and tool for measuring and troubleshooting Apache Spark job performance, providing both Scala and Python APIs to collect stage and task metrics. It enables users to instrument Spark workloads, generate reports, and export data to various sinks including Kafka, InfluxDB, Prometheus, and local files. The system supports detailed monitoring capabilities such as flight recording, JMX export, and end-to-end testing in containerized environments.

Features

Added Scala example for instrumenting Spark workloads with sparkMeasure

A new example project, testSparkMeasureScala, has been added to demonstrate how to use the sparkMeasure library to instrument Scala code running on Apache Spark. The example includes source code and documentation showing how to measure stage and task metrics, print reports, and save metric data to disk using Spark 4 and Scala 2.13.

examples/testSparkMeasureScala · high confidence

Initial release of Python API for Spark performance metrics

This change introduces the Python wrapper for the sparkMeasure library, enabling users to collect and analyze Apache Spark performance data directly from Python/PySpark applications. The new API includes \StageMetrics\ and \TaskMetrics\ classes, which allow users to begin and end metric collection, generate reports, and export aggregated metrics as Spark DataFrames. Additionally, a new \jmxexport\ function is provided to publish these metrics to Dropwizard via JMX, and the package is configured as a universal wheel for broader Python compatibility.

python · high confidence

Initial release of SparkMeasure performance tool

Introduces SparkMeasure, a library and tool for measuring and troubleshooting Apache Spark job performance. The release includes Scala and Python implementations for collecting stage and task metrics, a Dockerfile for containerized testing, and documentation with examples for Jupyter notebooks and CLI usage.

(repo-wide) · high confidence

New Python examples and documentation for sparkMeasure

The examples directory now includes a comprehensive set of Python-focused resources to help users instrument and analyze Spark workloads. This includes a new README.md that serves as a central index, a dedicated Databricks example page linking to Scala and Python notebooks, and a new Colab-compatible Jupyter notebook for quick start. Additionally, a standard Python script example (test\_sparkmeasure\_python.py) and a task metrics analysis notebook are provided, all demonstrating the use of sparkMeasure version 0.28 with Scala 2.13.

examples · high confidence

New end-to-end test suite for SparkMeasure on Kubernetes

Added a comprehensive end-to-end testing workflow for SparkMeasure on Kubernetes, including scripts to build test images, provision a Kind cluster with ArgoCD, deploy Spark jobs, and verify metrics availability via a Prometheus exporter. The suite includes example PySpark applications (spark-pi, spark-sql) that demonstrate StageMetrics instrumentation and JMX-based metric export, enabling validation of the library's functionality in a realistic containerized environment.

e2e · high confidence

New metrics sinks and flight recorder mode for Spark monitoring

This release adds several new ways to export Spark metrics: a Flight Recorder mode (FlightRecorderStageMetrics and FlightRecorderTaskMetrics) that records stage and task metrics to local files, HDFS, or stdout in JSON or Java-serialized formats; a Kafka sink (KafkaSink and KafkaSinkV2) for shipping metrics to Kafka streams, with V2 adding application-level aggregates and custom labels; an InfluxDB sink (InfluxDBSink) for near real-time writing to InfluxDB v1.x; a Prometheus Push Gateway sink (PushGatewaySink) with configurable connection and read timeouts; and a Dropwizard/JMX reporter that exposes metrics via JMX in the sparkmeasure.metrics domain. Utility support includes JSON serialization via Jackson (IOUtils.writeToStringSerializedJSON) and improved formatting for durations and bytes.

src/main · high confidence

Behavioural changes

Refactored Stage and Task metrics collection with new listener architecture

The Stage and Task metrics collection logic has been refactored to use new, more granular Spark Listener implementations (StageInfoRecorderListener and TaskInfoRecorderListener) that replace the previous monolithic listeners. This change introduces several behavioral updates: stage metrics now include the job group ID and detailed executor metrics (such as peak memory usage via Spark 3.1+ features), while task metrics now explicitly track speculative tasks and calculate scheduler delay more accurately. The reporting API has also changed; the previous SQL-based aggregation and DataFrame views are now supplemented or replaced by in-memory aggregation and direct reporting methods, and the output format for metrics like CPU time and shuffle write time is normalized to milliseconds. Additionally, the new implementation clears metrics data at the start of a new collection period to reduce memory footprint and handles null values in stage submission/completion times more robustly.

repository · high confidence

Test coverage

Added test coverage for KafkaSinkV2, StageMetrics, TaskMetrics, Utils, and PushGateway

New test suites have been added to verify the correctness of the KafkaSinkV2 implementation (including application start/end events, custom labels, and Spark configuration extraction), the aggregation of Stage and Task metrics, the serialization and JSON writing capabilities in Utils, and the PushGateway integration with label/metric validation and HTTP posting.

src/test · high confidence

Dependencies

Updated build configuration to Spark 3.5.8 and Scala 2.12/2.13

The main Scala build (build.sbt) now targets Apache Spark 3.5.8 and supports Scala 2.12.18 and 2.13.16, replacing the previous Spark 2.1.0 and Scala 2.11.8 setup. Jackson dependencies are pinned to version 2.15.2 to ensure compatibility with Spark 3.5.x, and the project publishes to Central Sonatype. A new example project (examples/testSparkMeasureScala) demonstrates usage with Spark 4.0.1 and spark-measure 0.28, while the Python package (python/pyproject.toml) is updated to version 0.28.0 with optional PySpark support (\>=3.0.0) and requirements.txt specifies PySpark 3.5.8.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 63 → 61 (-2.9)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 61 → 55 (-5.8)
  • Architecture 100 → 100 (+0.0)
  • Maturity 79 → 79 (+0.0)
  • Readiness 54 → 52 (-1.8)
  • Security 80 → 76 (-4.1)

New (13)

  • Edited copies of one member (4 members, 14+ corresponding lines) (src/main/scala/ch/cern/sparkmeasure/KafkaSink.scala)
  • Edited copy of a member (10 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/StageMetrics.scala)
  • Edited copy of a member (13 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/InfluxDBSink.scala)
  • Edited copy of a member (16 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/KafkaSink.scala)
  • Edited copy of a member (20 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/TaskMetrics.scala)
  • Edited copy of a member (22 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/StageMetrics.scala)
  • Edited copy of a member (22 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/StageMetrics.scala)
  • Edited copy of a member (32 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/KafkaSink.scala)
  • Edited copy of a member (4 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/StageMetrics.scala)
  • Edited copy of a member (43 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/StageMetrics.scala)
  • Edited copy of a member (58 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/InfluxDBSink.scala)
  • Edited copy of a member (59 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/KafkaSink.scala)
  • Edited copy of a member (7 corresponding lines) (src/main/scala/ch/cern/sparkmeasure/StageInfoRecorderListener.scala)

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

LucaCanali/sparkMeasure was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 9ab678a4d4c58128c090c87650706be926bfbc11 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.