Shopify/job-iteration

JobIteration is a Ruby gem that enables the safe, resumable iteration over large datasets within ActiveJob background jobs. It provides built-in support for processing ActiveRecord relations in batches, reading CSV files, and integrating with major queue adapters like Sidekiq and Resque to handle graceful interruptions. The library also includes tooling for type signature generation via Tapioca and ensures compatibility across various Rails and Ruby versions.

10 changes.

This period focused on releasing version 1.15.1, which introduced batch processing, CSV support, and comprehensive interruption adapters for various queue systems. The library underwent significant restructuring, moving to a top-level module and adding global configuration options, while simultaneously modernizing the development environment to require Ruby 3.1+ and ActiveJob 7.1+. Extensive test coverage was added to validate the new enumerators, integration behaviors, and CI matrix consistency.

4 changes.

This period focused on enhancing type safety and database compatibility by introducing a Tapioca compiler for JobIteration jobs and adding support for composite primary keys in batch enumeration. The work included expanding CI testing infrastructure to cover multiple Rails versions and implementing comprehensive test coverage for the new type generation features.

Introduces a new DSL compiler that automatically generates RBI type signatures for Active Job classes inheriting from JobIteration::Iteration. This compiler extracts method parameters from the \build\_enumerator\ method, expands FixedHash types into individual keyword arguments, and generates signatures for \perform\_later\, \perform\_now\, and \perform\ methods, ensuring correct parameter ordering and handling of generic type members.

lib/tapioca · high confidence

This release adds the ability to iterate over ActiveRecord relations in batches (yielding relations rather than individual records) via \ActiveRecordBatchEnumerator\, and introduces a \CsvEnumerator\ for processing CSV files. It also adds built-in interruption adapters for Sidekiq, GoodJob, DelayedJob, Resque, SolidQueue, and AWS SQS, allowing long-running iteration jobs to pause and re-enqueue gracefully during worker shutdowns. The library now supports composite primary keys, parallel iteration across multiple job instances, and custom timezones for date/datetime serialization. Version is bumped to 1.15.1.

lib/job-iteration · high confidence

The batch enumerator now correctly handles models with composite primary keys. A new ColumnManager utility ensures that all primary key columns are included in the plucked data, allowing records to be efficiently queried and processed even when the primary key consists of multiple columns.

_lib/job-iteration/active\_record\_batch\enumerator · high confidence

The library has been refactored from the nested Job::Iteration namespace to the top-level JobIteration module, introducing a new Railtie and dedicated components for enumeration, interruption handling, and logging. Users gain new global configuration options, including max\_job\_runtime for limiting job execution time, default\_retry\_backoff for controlling resume delays, and the ability to register custom interruption adapters or replace the default enumerator builder. The module now integrates with ActiveJob's logging infrastructure via ActiveSupport::Notifications and includes a deprecation mechanism for legacy settings.

lib · high confidence

The repository now includes a new \bin/test\ script that allows users to run tests with flexible options: running all tests, targeting a specific file, or filtering by test name. The previous \bin/console\ script has been removed. Additionally, \bin/setup\ has been updated to automatically check for MySQL installation, start the MySQL server if needed, and create the \job\_iteration\_test\ database if it does not already exist, streamlining the local development environment setup.

bin · high confidence

The development environment now targets Ruby 4.0.6 and enforces the Shopify Ruby style guide via RuboCop. Configuration files have been updated to include RuboCop settings (inheriting from rubocop-shopify, including gemfiles, excluding specific files), YARD documentation options, and a new dev.yml for local setup. The legacy Travis CI configuration has been removed, and the README has been expanded to document the library's features, supported queue adapters, and usage examples.

(repo-wide) · high confidence

Added CI matrix consistency tests and comprehensive test suite setup; Added Sidekiq test support initialization; Added integration tests for Resque and Sidekiq job interruption; Added test support fixtures for job iteration and CSV processing; Added tests for the JobIteration Tapioca compiler; Added unit tests for JobIteration enumerators and helpers.

A new gemfile (gemfiles/rails\_gems.gemfile) has been added to support CI testing across different Rails versions. It dynamically resolves gem requirements based on the RAILS\_VERSION environment variable, supporting edge versions from the main branch and specific minor versions (e.g., 7.1, 7.2). The file evaluates the main Gemfile after setting up these requirements, enabling flexible testing of the gem against various Rails versions.

gemfiles · high confidence

The gem now requires Ruby 3.1 or later and ActiveJob 7.1 or later, dropping support for older versions. The Gemfile has been refactored to conditionally pin versions for Sidekiq, Resque, connection\_pool, Sorbet, and Tapioca based on the Ruby and Rails versions in use, ensuring compatibility with newer dependency releases (e.g., Sidekiq 8.0.9+ for Ruby 3.2+). Development dependencies have been updated to include the latest versions of tools like RuboCop, Mocha, and RSpec-compatible testing libraries, and the Gemfile.lock is now committed to the repository.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 75.

• Code Health 100
• Architecture 100
• Maturity 70
• Readiness 73
• Security 75

• 300 commits — 277 feature/other, 23 fixes

• (repo) — 130 commits
• (root) — 70 commits
• .github/workflows — 53 commits
• lib/job-iteration — 23 commits
• lib/tapioca — 8 commits
• test/unit — 5 commits
• test/integration — 4 commits
• test/test_helper.rb — 2 commits
• .github/CODEOWNERS — 1 commit
• guides/custom-enumerator.md — 1 commit
• guides/iteration-how-it-works.md — 1 commit
• guides/throttling.md — 1 commit
• test/tapioca — 1 commit

• fix: Bump tapioca to >= 0.19.1 to fix has_rest NoMethodError
• fix: Fix Resque-Rails compatibility
• fix: Fix anonymous block handling in DSL compiler
• fix: Fix dev database issues
• fix: Fix empty FixedHash expansion to preserve required positional param
• fix: Fix integration load test
• fix: Fix linter error
• fix: Fix linter error
• fix: Fix max_job_runtime test by removing freeze_time
• fix: Fix rubydoc link
• fix: Fix tapioca compiler to preserve params for empty FixedHash
• fix: Fix the UTC timezone issue with cursor resume
• fix: Fix timezone serialisation on ActiveRecordEnumerator too
• fix: Fix uninitialized constant Sidekiq::ActiveJob
• fix: Merge pull request #512 from yigitozkavci/fix-utc-timezone-in-cursor-resume
• fix: Merge pull request #534 from Shopify/fix-ci
• fix: Merge pull request #537 from Shopify/fix-perform-now-return-type
• fix: Merge pull request #543 from Shopify/mangara-fix-sidekiq-7-3-9-integration-test
• fix: Merge pull request #556 from Shopify/el-fix-sorbet-signature-for-generic-jobs
• fix: Merge pull request #593 from Shopify/mangara-fix-positional-args-compiler
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Shopify/job-iteration was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.