spotify/ratatool

Ratatool is a data toolkit for comparing and sampling datasets across Avro, Parquet, Protocol Buffers, and BigQuery formats. It provides field-level diffing capabilities to detect structural and value differences, alongside utilities for local and distributed data sampling. The system also includes generators for creating schema-compliant test data and tools for comparing Scala case classes.

Introduces a new CaseClassDiffy component in the shapeless module that enables diffing of Scala case classes. It leverages Shapeless to automatically derive encoders for primitive types, options, lists, and nested structures, converting case class instances into maps for comparison. This allows users to compare collections of case class objects using the existing BigDiffy infrastructure, supporting ignore lists and unordered field comparisons.

ratatool-shapeless/src/main · high confidence

The CLI now includes a new \directSampler\ command that allows users to perform random or head-based sampling directly on local Avro and Parquet files, as well as BigQuery tables, without requiring a distributed runner. This command accepts input and output paths for the respective formats and supports a \--head\ flag to sample from the beginning of the dataset instead of randomly. The BigQuery sampling mode is marked as deprecated in favor of using \bigSampler\ with the DirectRunner.

ratatool-cli/src/main · high confidence

The diffy module now includes dedicated diffing implementations for Avro, Protocol Buffers, and BigQuery TableRow records. These new components enable field-level comparison of structured data, supporting features such as ignoring specific fields, treating repeated fields as unordered sets, and handling nested repeated records with specific key fields for accurate matching. This expands the tool's capability to detect differences across these common data formats beyond its previous scope.

ratatool-diffy/src/main · high confidence

The repository is initialized with essential configuration files including an Apache 2.0 license, a NOTICE file, and a \.scalafmt.conf\ defining code formatting rules (version 3.9.5, Scala 2.12). A \.scala-steward.conf\ is added to manage automated dependency updates by ignoring specific shared libraries (e.g., shapeless, Guava, Avro). The project structure is defined via \version.sbt\ (set to 0.4.14-SNAPSHOT) and \catalog-info.yaml\ for Backstage integration. The README is significantly expanded to document features like ScalaCheck generators, IO utilities, samplers, Diffy, and BigDiffy, along with usage instructions for sbt, Homebrew, and CLI commands.

(repo-wide) · high confidence

This change introduces the core Ratatool library, providing tools for comparing and sampling data across multiple formats. The BigDiffy module enables field-level and global statistics for comparing datasets (supporting Avro, Parquet, and BigQuery), while the sampling module offers stratified and uniform distribution strategies with configurable hash algorithms (Murmur, Farm). Additionally, the ratatool-scalacheck module adds generators for creating test data from Protocol Buffer schemas.

repository · high confidence

This change adds the ratatool-scalacheck module, providing Scalacheck generators for Apache Avro records (both specific and generic), Google BigQuery TableRows, and Protobuf messages. It includes utility classes like HashMapBuilder to support collection generation and GenTestUtils to help execute tests with reproducible random seeds. Users can now easily create randomized but schema-compliant test data for pipelines and components that process these formats.

ratatool-scalacheck/src/main · high confidence

The \ratatool-sampling\ module now includes its own self-contained implementations for reading and writing data, decoupling from external Scio dependencies. This adds \AvroIO\ and \ParquetIO\ for file-based formats, \BigQueryIO\ for table operations, and \TableRowJsonIO\ for JSON serialization. It also introduces \AvroSampler\ and \BigQuerySampler\ for local file/table sampling, and \BigSampler\ for distributed sampling via Scio. To support BigQuery access without the original Scio internals, the module includes \PatchedBigQueryServices\ and \PatchedBigQueryTableRowIterator\, which are copied and adapted from Scio to handle table iteration and service interactions directly.

ratatool-sampling/src/main · high confidence

Added example implementations for running BigDiffy on Avro and Protobuf data, including a pre-processing step for Avro records. Included new Avro and Protobuf schema definitions, along with BigQuery JSON schemas, to support these examples. Also added Scala generators for creating test data for Avro, Protobuf, and BigQuery TableRows, demonstrating how to use the library's amend capabilities for complex field dependencies.

ratatool-examples/src/main · high confidence

Added a new BigQueryUtil object that normalizes null TableFieldSchema modes to 'NULLABLE' (the default value), ensuring consistent handling of missing mode information. Also introduced a Command trait to standardize command identification across the application.

ratatool-common/src/main · high confidence

Added Avro and BigQuery test schemas and test logging configuration; Added Avro and Protobuf test schemas for complex data structures; Added property-based tests for example generators and BigDiffy pre-processing; Added test coverage for Avro, BigQuery, Protobuf, and TableRow diffing; Added test coverage for samplers and hashing utilities; Added test utilities for Avro and BigQuery schema loading; Added tests for Avro, Parquet, and TableRow JSON I/O operations; Added tests for CaseClassDiffy; Added tests for DirectSamplerParser command-line argument handling; Added unit tests for ratatool-scalacheck generators.

The project has upgraded the SBT version from 0.13.11 to 1.12.8, requiring a significant overhaul of the build configuration. The legacy sbt-assembly plugin has been replaced with sbt-pack for packaging, and the plugin suite has been updated to include sbt-avro (3.5.1), sbt-protoc (1.0.7), sbt-release (1.4.0), sbt-pgp (2.3.1), sbt-scoverage (2.4.4), sbt-sonatype (3.12.2), sbt-scalafmt (2.5.6), and sbt-scalafix (0.14.6). A new JavaOptions helper has been introduced to manage JVM arguments with version-specific support, notably adding necessary --add-opens flags for Java 17+ compatibility. Additionally, direct dependencies on avro-compiler and scalac-options have been added to the build definition.

project · high confidence

The build system has been significantly updated to align with Scio 0.15.4, which drives upgrades to core dependencies including Apache Beam 2.71.0, Protocol Buffers 4.33.0, and Parquet 1.17.0. The project has migrated from Scala 2.11 to 2.12 (with 2.13 support), updated the Java target to version 8, and switched publishing credentials to Central Portal. Additionally, the build now uses Magnolify for BigQuery types instead of the deprecated BigQueryType and excludes the Kafka SDK to reduce dependency conflicts.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 63.

• Code Health 91
• Architecture 100
• Maturity 52
• Readiness 65
• Security 73

• 300 commits — 282 feature/other, 18 fixes

• (root) — 131 commits
• (repo) — 79 commits
• project/plugins.sbt — 42 commits
• ratatool-sampling/src — 21 commits
• ratatool-scalacheck/src — 10 commits
• project/build.properties — 5 commits
• .circleci/config.yml — 4 commits
• ratatool-diffy/src — 4 commits
• ratatool-diffy/README.md — 2 commits
• ratatool-sampling/README.md — 2 commits

• fix: (fix) Respect avro.java.string type (#709)
• fix: Add missing Log4J dependency due to Scio 0.13.3 bug
• fix: Fix filepath wildcard for avro/parquet (#564)
• fix: Merge pull request #635 from spotify/revert-628-update/joda-time-2.12.5
• fix: Merge pull request #690 from spotify/fix-cross-compilation
• fix: Move fix to object initializer
• fix: Revert "Update joda-time to 2.12.5"
• fix: Revert "Update scalatest to 3.2.19 (#741)" (#772)
• fix: Revert "Upgrade Scio to 0.12.1"
• fix: Revert commit(s) 2886b96
• fix: Revert commit(s) b1ff98d
• fix: Revert commit(s) ef190b4
• fix: Revert commit(s) f4c3434
• fix: fix flaky test
• fix: fix test
• fix: fix test for avro 1.8.2
• fix: fix+debug flaky test
• fix: revert debug logging
• change: 0.4.1-SNAPSHOT
• change: Add 'Reformat with scalafmt 3.8.0' to .git-blame-ignore-revs
• …and 280 more

• 0 containers · 1 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

spotify/ratatool was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.