spotify/ratatool
59.8
Adequate · 6 November 2026
7.3k
lines of production code
Scala
with Java
4
measurements over time
What this system is
Features
Add CaseClassDiffy for comparing Scala case classes
Introduces a new CaseClassDiffy component in the shapeless module that enables diffing of Scala case classes. It leverages Shapeless to automatically derive encoders for primitive types, options, lists, and nested structures, converting case class instances into maps for comparison. This allows users to compare collections of case class objects using the existing BigDiffy infrastructure, supporting ignore lists and unordered field comparisons.
ratatool-shapeless/src/main · high confidence
Add directSampler command for local Avro, Parquet, and BigQuery sampling
The CLI now includes a new \directSampler\ command that allows users to perform random or head-based sampling directly on local Avro and Parquet files, as well as BigQuery tables, without requiring a distributed runner. This command accepts input and output paths for the respective formats and supports a \--head\ flag to sample from the beginning of the dataset instead of randomly. The BigQuery sampling mode is marked as deprecated in favor of using \bigSampler\ with the DirectRunner.
ratatool-cli/src/main · high confidence
Add field-level diffing support for Avro, ProtoBuf, and TableRow records
The diffy module now includes dedicated diffing implementations for Avro, Protocol Buffers, and BigQuery TableRow records. These new components enable field-level comparison of structured data, supporting features such as ignoring specific fields, treating repeated fields as unordered sets, and handling nested repeated records with specific key fields for accurate matching. This expands the tool's capability to detect differences across these common data formats beyond its previous scope.
ratatool-diffy/src/main · high confidence
Initial project scaffolding and configuration
The repository is initialized with essential configuration files including an Apache 2.0 license, a NOTICE file, and a \.scalafmt.conf\ defining code formatting rules (version 3.9.5, Scala 2.12). A \.scala-steward.conf\ is added to manage automated dependency updates by ignoring specific shared libraries (e.g., shapeless, Guava, Avro). The project structure is defined via \version.sbt\ (set to 0.4.14-SNAPSHOT) and \catalog-info.yaml\ for Backstage integration. The README is significantly expanded to document features like ScalaCheck generators, IO utilities, samplers, Diffy, and BigDiffy, along with usage instructions for sbt, Homebrew, and CLI commands.
(repo-wide) · high confidence
Initial release of Ratatool data comparison and sampling toolkit
This change introduces the core Ratatool library, providing tools for comparing and sampling data across multiple formats. The BigDiffy module enables field-level and global statistics for comparing datasets (supporting Avro, Parquet, and BigQuery), while the sampling module offers stratified and uniform distribution strategies with configurable hash algorithms (Murmur, Farm). Additionally, the ratatool-scalacheck module adds generators for creating test data from Protocol Buffer schemas.
repository · high confidence
Introduce ratatool-scalacheck for generating test data from Avro, BigQuery, and Protobuf schemas
This change adds the ratatool-scalacheck module, providing Scalacheck generators for Apache Avro records (both specific and generic), Google BigQuery TableRows, and Protobuf messages. It includes utility classes like HashMapBuilder to support collection generation and GenTestUtils to help execute tests with reproducible random seeds. Users can now easily create randomized but schema-compliant test data for pipelines and components that process these formats.
ratatool-scalacheck/src/main · high confidence
Introduces standalone sampling and IO utilities for Avro, Parquet, and BigQuery
The \ratatool-sampling\ module now includes its own self-contained implementations for reading and writing data, decoupling from external Scio dependencies. This adds \AvroIO\ and \ParquetIO\ for file-based formats, \BigQueryIO\ for table operations, and \TableRowJsonIO\ for JSON serialization. It also introduces \AvroSampler\ and \BigQuerySampler\ for local file/table sampling, and \BigSampler\ for distributed sampling via Scio. To support BigQuery access without the original Scio internals, the module includes \PatchedBigQueryServices\ and \PatchedBigQueryTableRowIterator\, which are copied and adapted from Scio to handle table iteration and service interactions directly.
ratatool-sampling/src/main · high confidence
New example code and schemas for BigDiffy and data generation
Added example implementations for running BigDiffy on Avro and Protobuf data, including a pre-processing step for Avro records. Included new Avro and Protobuf schema definitions, along with BigQuery JSON schemas, to support these examples. Also added Scala generators for creating test data for Avro, Protobuf, and BigQuery TableRows, demonstrating how to use the library's amend capabilities for complex field dependencies.
ratatool-examples/src/main · high confidence
Behavioural changes
Introduce BigQuery field mode normalization and Command trait
Added a new BigQueryUtil object that normalizes null TableFieldSchema modes to 'NULLABLE' (the default value), ensuring consistent handling of missing mode information. Also introduced a Command trait to standardize command identification across the application.
ratatool-common/src/main · high confidence
Test coverage
Added Avro and BigQuery test schemas and test logging configuration; Added Avro and Protobuf test schemas for complex data structures; Added property-based tests for example generators and BigDiffy pre-processing; Added test coverage for Avro, BigQuery, Protobuf, and TableRow diffing; Added test coverage for samplers and hashing utilities; Added test utilities for Avro and BigQuery schema loading; Added tests for Avro, Parquet, and TableRow JSON I/O operations; Added tests for CaseClassDiffy; Added tests for DirectSamplerParser command-line argument handling; Added unit tests for ratatool-scalacheck generators.
Dependencies
Migrate build infrastructure to SBT 1.12.8 and modernize plugin ecosystem
The project has upgraded the SBT version from 0.13.11 to 1.12.8, requiring a significant overhaul of the build configuration. The legacy sbt-assembly plugin has been replaced with sbt-pack for packaging, and the plugin suite has been updated to include sbt-avro (3.5.1), sbt-protoc (1.0.7), sbt-release (1.4.0), sbt-pgp (2.3.1), sbt-scoverage (2.4.4), sbt-sonatype (3.12.2), sbt-scalafmt (2.5.6), and sbt-scalafix (0.14.6). A new JavaOptions helper has been introduced to manage JVM arguments with version-specific support, notably adding necessary --add-opens flags for Java 17+ compatibility. Additionally, direct dependencies on avro-compiler and scalac-options have been added to the build definition.
project · high confidence
Upgrade to Scio 0.15.4 and modernize build configuration
The build system has been significantly updated to align with Scio 0.15.4, which drives upgrades to core dependencies including Apache Beam 2.71.0, Protocol Buffers 4.33.0, and Parquet 1.17.0. The project has migrated from Scala 2.11 to 2.12 (with 2.13 support), updated the Java target to version 8, and switched publishing credentials to Central Portal. Additionally, the build now uses Magnolify for BigQuery types instead of the deprecated BigQueryType and excludes the Kafka SDK to reduce dependency conflicts.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 60 → 60 (-0.1)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 63 → 63 (+0.0)
- Architecture 91 → 91 (+0.0)
- Maturity 52 → 52 (+0.2)
- Readiness 64 → 64 (+0.3)
- Security 84 → 79 (-5.2)
- Performance 70 → 70 (+0.0)
Resolved (12)
- High CVE: [GHSA redacted] (build.sbt)
- Off-boarding risk: anonymized user #1
- Orphaned file (ratatool-scalacheck/src/main/scala/com/spotify/ratatool/scalacheck/AvroGenerator.scala)
- Outdated: com.fasterxml.jackson.core:jackson-annotations
- Outdated: com.fasterxml.jackson.core:jackson-databind
- Outdated: com.fasterxml.jackson.module:jackson-module-scala_2.12
- Outdated: com.google.guava:guava
- Outdated: com.google.protobuf:protobuf-java
- Outdated: joda-time:joda-time
- Outdated: org.apache.beam:beam-runners-direct-java
- Outdated: org.apache.beam:beam-runners-google-cloud-dataflow-java
- Outdated: org.scalacheck:scalacheck_2.12
New (3)
- Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
- High CVE: [GHSA redacted] (build.sbt)
- Off-boarding risk: anonymized user #1
Changes since last survey
- 7 commits — 6 feature/other, 1 fixes
By area
- (repo) — 6 commits
- project/build.properties — 1 commit
Notable commits
- fix: Merge pull request #920 from spotify/jpark/fix-logical-type-conversion
- change: Merge pull request #921 from spotify/jpark/scalacheck-1.20.0
- change: Merge pull request #923 from scala-steward/update/scalafmt-core-3.11.5
- change: Merge pull request #926 from scala-steward/update/jackson-annotations-2.18.12
- change: Merge pull request #927 from scala-steward/update/jackson-module-scala-2.18.12
- change: Merge pull request #929 from scala-steward/update/sbt-1.13.1
- change: Update sbt, sbt-dependency-tree, ... to 1.13.1
Architecture
- Unchanged — 0 containers · 1 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
spotify/ratatool was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 6 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 561826be77fdf16349ec1bfd332424ed0d31af29 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-60d1b390edf0.