Skip to content
CAI
Software that uses CAICheck a score

spotify/ratatool

59.8

Adequate · 6 November 2026

7.3k

lines of production code

Scala

with Java

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add CaseClassDiffy for comparing Scala case classes

Introduces a new CaseClassDiffy component in the shapeless module that enables diffing of Scala case classes. It leverages Shapeless to automatically derive encoders for primitive types, options, lists, and nested structures, converting case class instances into maps for comparison. This allows users to compare collections of case class objects using the existing BigDiffy infrastructure, supporting ignore lists and unordered field comparisons.

ratatool-shapeless/src/main · high confidence

Add directSampler command for local Avro, Parquet, and BigQuery sampling

The CLI now includes a new \directSampler\ command that allows users to perform random or head-based sampling directly on local Avro and Parquet files, as well as BigQuery tables, without requiring a distributed runner. This command accepts input and output paths for the respective formats and supports a \--head\ flag to sample from the beginning of the dataset instead of randomly. The BigQuery sampling mode is marked as deprecated in favor of using \bigSampler\ with the DirectRunner.

ratatool-cli/src/main · high confidence

Add field-level diffing support for Avro, ProtoBuf, and TableRow records

The diffy module now includes dedicated diffing implementations for Avro, Protocol Buffers, and BigQuery TableRow records. These new components enable field-level comparison of structured data, supporting features such as ignoring specific fields, treating repeated fields as unordered sets, and handling nested repeated records with specific key fields for accurate matching. This expands the tool's capability to detect differences across these common data formats beyond its previous scope.

ratatool-diffy/src/main · high confidence

Initial project scaffolding and configuration

The repository is initialized with essential configuration files including an Apache 2.0 license, a NOTICE file, and a \.scalafmt.conf\ defining code formatting rules (version 3.9.5, Scala 2.12). A \.scala-steward.conf\ is added to manage automated dependency updates by ignoring specific shared libraries (e.g., shapeless, Guava, Avro). The project structure is defined via \version.sbt\ (set to 0.4.14-SNAPSHOT) and \catalog-info.yaml\ for Backstage integration. The README is significantly expanded to document features like ScalaCheck generators, IO utilities, samplers, Diffy, and BigDiffy, along with usage instructions for sbt, Homebrew, and CLI commands.

(repo-wide) · high confidence

Initial release of Ratatool data comparison and sampling toolkit

This change introduces the core Ratatool library, providing tools for comparing and sampling data across multiple formats. The BigDiffy module enables field-level and global statistics for comparing datasets (supporting Avro, Parquet, and BigQuery), while the sampling module offers stratified and uniform distribution strategies with configurable hash algorithms (Murmur, Farm). Additionally, the ratatool-scalacheck module adds generators for creating test data from Protocol Buffer schemas.

repository · high confidence

Introduce ratatool-scalacheck for generating test data from Avro, BigQuery, and Protobuf schemas

This change adds the ratatool-scalacheck module, providing Scalacheck generators for Apache Avro records (both specific and generic), Google BigQuery TableRows, and Protobuf messages. It includes utility classes like HashMapBuilder to support collection generation and GenTestUtils to help execute tests with reproducible random seeds. Users can now easily create randomized but schema-compliant test data for pipelines and components that process these formats.

ratatool-scalacheck/src/main · high confidence

Introduces standalone sampling and IO utilities for Avro, Parquet, and BigQuery

The \ratatool-sampling\ module now includes its own self-contained implementations for reading and writing data, decoupling from external Scio dependencies. This adds \AvroIO\ and \ParquetIO\ for file-based formats, \BigQueryIO\ for table operations, and \TableRowJsonIO\ for JSON serialization. It also introduces \AvroSampler\ and \BigQuerySampler\ for local file/table sampling, and \BigSampler\ for distributed sampling via Scio. To support BigQuery access without the original Scio internals, the module includes \PatchedBigQueryServices\ and \PatchedBigQueryTableRowIterator\, which are copied and adapted from Scio to handle table iteration and service interactions directly.

ratatool-sampling/src/main · high confidence

New example code and schemas for BigDiffy and data generation

Added example implementations for running BigDiffy on Avro and Protobuf data, including a pre-processing step for Avro records. Included new Avro and Protobuf schema definitions, along with BigQuery JSON schemas, to support these examples. Also added Scala generators for creating test data for Avro, Protobuf, and BigQuery TableRows, demonstrating how to use the library's amend capabilities for complex field dependencies.

ratatool-examples/src/main · high confidence

Behavioural changes

Introduce BigQuery field mode normalization and Command trait

Added a new BigQueryUtil object that normalizes null TableFieldSchema modes to 'NULLABLE' (the default value), ensuring consistent handling of missing mode information. Also introduced a Command trait to standardize command identification across the application.

ratatool-common/src/main · high confidence

Test coverage

Added Avro and BigQuery test schemas and test logging configuration; Added Avro and Protobuf test schemas for complex data structures; Added property-based tests for example generators and BigDiffy pre-processing; Added test coverage for Avro, BigQuery, Protobuf, and TableRow diffing; Added test coverage for samplers and hashing utilities; Added test utilities for Avro and BigQuery schema loading; Added tests for Avro, Parquet, and TableRow JSON I/O operations; Added tests for CaseClassDiffy; Added tests for DirectSamplerParser command-line argument handling; Added unit tests for ratatool-scalacheck generators.

Dependencies

Migrate build infrastructure to SBT 1.12.8 and modernize plugin ecosystem

The project has upgraded the SBT version from 0.13.11 to 1.12.8, requiring a significant overhaul of the build configuration. The legacy sbt-assembly plugin has been replaced with sbt-pack for packaging, and the plugin suite has been updated to include sbt-avro (3.5.1), sbt-protoc (1.0.7), sbt-release (1.4.0), sbt-pgp (2.3.1), sbt-scoverage (2.4.4), sbt-sonatype (3.12.2), sbt-scalafmt (2.5.6), and sbt-scalafix (0.14.6). A new JavaOptions helper has been introduced to manage JVM arguments with version-specific support, notably adding necessary --add-opens flags for Java 17+ compatibility. Additionally, direct dependencies on avro-compiler and scalac-options have been added to the build definition.

project · high confidence

Upgrade to Scio 0.15.4 and modernize build configuration

The build system has been significantly updated to align with Scio 0.15.4, which drives upgrades to core dependencies including Apache Beam 2.71.0, Protocol Buffers 4.33.0, and Parquet 1.17.0. The project has migrated from Scala 2.11 to 2.12 (with 2.13 support), updated the Java target to version 8, and switched publishing credentials to Central Portal. Additionally, the build now uses Magnolify for BigQuery types instead of the deprecated BigQueryType and excludes the Kafka SDK to reduce dependency conflicts.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 60 → 60 (-0.1)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 63 → 63 (+0.0)
  • Architecture 91 → 91 (+0.0)
  • Maturity 52 → 52 (+0.2)
  • Readiness 64 → 64 (+0.3)
  • Security 84 → 79 (-5.2)
  • Performance 70 → 70 (+0.0)

Resolved (12)

  • High CVE: [GHSA redacted] (build.sbt)
  • Off-boarding risk: anonymized user #1
  • Orphaned file (ratatool-scalacheck/src/main/scala/com/spotify/ratatool/scalacheck/AvroGenerator.scala)
  • Outdated: com.fasterxml.jackson.core:jackson-annotations
  • Outdated: com.fasterxml.jackson.core:jackson-databind
  • Outdated: com.fasterxml.jackson.module:jackson-module-scala_2.12
  • Outdated: com.google.guava:guava
  • Outdated: com.google.protobuf:protobuf-java
  • Outdated: joda-time:joda-time
  • Outdated: org.apache.beam:beam-runners-direct-java
  • Outdated: org.apache.beam:beam-runners-google-cloud-dataflow-java
  • Outdated: org.scalacheck:scalacheck_2.12

New (3)

  • Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
  • High CVE: [GHSA redacted] (build.sbt)
  • Off-boarding risk: anonymized user #1

Changes since last survey

  • 7 commits — 6 feature/other, 1 fixes

By area

  • (repo) — 6 commits
  • project/build.properties — 1 commit

Notable commits

  • fix: Merge pull request #920 from spotify/jpark/fix-logical-type-conversion
  • change: Merge pull request #921 from spotify/jpark/scalacheck-1.20.0
  • change: Merge pull request #923 from scala-steward/update/scalafmt-core-3.11.5
  • change: Merge pull request #926 from scala-steward/update/jackson-annotations-2.18.12
  • change: Merge pull request #927 from scala-steward/update/jackson-module-scala-2.18.12
  • change: Merge pull request #929 from scala-steward/update/sbt-1.13.1
  • change: Update sbt, sbt-dependency-tree, ... to 1.13.1

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

spotify/ratatool was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 6 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 561826be77fdf16349ec1bfd332424ed0d31af29 — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-60d1b390edf0.