evanphx/benchmark-ips

This system is a Ruby benchmarking library that measures and compares code execution performance with statistical rigor. It provides capabilities for running iterative benchmarks, calculating confidence intervals, and persisting or sharing results via JSON and online services. The tool supports flexible reporting modes, quiet execution, and comparison against baselines to determine statistical significance.

The benchmarking output now supports a new statistical model using bootstrap confidence intervals (via the \kalibera\ gem) in addition to the existing standard deviation method. This change introduces a \Bootstrap\ stats class that calculates central tendency and error using 10,000 iterations and a configurable confidence level, providing a 'with X% confidence' footer in results. It also adds a \StatsMetric\ module shared by both \Bootstrap\ and \SD\ classes, implementing \error\_percentage\ and \overlaps?\ methods to standardize how statistical overlap and error percentages are reported for comparisons.

lib/benchmark/ips/stats · high confidence

This update introduces several new capabilities to the benchmarking workflow. Users can now persist results between separate Ruby invocations using the new \hold!\ and \save!\ methods, which store data to a specified file path. Benchmark results can be exported to JSON files via the \json!\ method or printed to standard output. Additionally, a new \Share\ class allows users to upload their benchmark reports to an online service (defaulting to ips.fyi) for easy comparison and sharing. The underlying \Job\ and \Report\ classes have been refactored to support these features, including improved statistical handling and label width auto-computation.

lib/benchmark/ips · high confidence

Added five new example scripts to the examples directory to demonstrate specific benchmark-ips capabilities. advanced.rb shows how to configure bootstrap confidence intervals and iteration counts. simple.rb illustrates various reporting modes, including standard blocks, iteration-count blocks, and string-grafted code. hold.rb and save.rb demonstrate how to persist benchmark results between runs to compare different Ruby versions or code changes. quick.rb provides a shortcut for comparing methods on objects without defining a full benchmark block.

examples · high confidence

The bin/benchmark\_ips executable, which previously contained only a placeholder script aborting with a message to write the implementation, has been removed from the repository.

bin · high confidence

The benchmark comparison output now uses statistical overlap checks to report results as 'same-ish' when differences fall within the error margin, rather than always declaring a speedup or slowdown. The comparison logic supports an \order: :baseline\ option to compare against the first report instead of the fastest. Internally, timing measurements now use the monotonic clock for greater accuracy, and the library has been restructured with modularized stats and report classes.

lib/benchmark · high confidence

The benchmarking output and execution logic have been restructured into specific components: \Benchmark::IPS::Job::Entry\ now compiles a specialized \call\_times\ method for each benchmark action to optimize performance, while \StreamReport\ and \MultiReport\ handle the formatting and routing of benchmark results to output streams. This change separates the concerns of code execution and report generation, allowing for more flexible output handling and improved benchmarking speed.

lib/benchmark/ips/job · high confidence

The test suite has been migrated from Test::Unit to Minitest. This change adds comprehensive tests for the reporting system, verifying that custom suites receive the correct lifecycle callbacks (warming, warmup\_stats, running, add\_report) and that multi-report configurations work as expected. It also adds extensive tests for the quiet mode feature, ensuring that the quiet flag correctly suppresses output when set via configuration options, accessors, or runtime changes, and verifies that the last value set wins when the flag is toggled during block definition.

test · high confidence

The project now includes a Gemfile and a benchmark-ips.gemspec to manage dependencies and packaging. The Gemfile specifies rake (\>= 10.5) as a runtime dependency, minitest for testing, and conditionally includes the json gem for Ruby versions older than 1.9. The gemspec defines the library version dynamically from the source code and adds minitest (\~\> 5.4) and rdoc (\~\> 4.0) as dependencies, ensuring compatibility with older Ruby versions.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 66.

• Code Health 100
• Architecture 69
• Maturity 54
• Readiness 72
• Security 96

• 299 commits — 272 feature/other, 27 fixes

• lib/benchmark — 134 commits
• (root) — 75 commits
• (repo) — 72 commits
• test/test_benchmark_ips.rb — 10 commits
• .github/workflows — 3 commits
• examples/simple.rb — 2 commits
• bin/benchmark_ips — 1 commit
• examples/advanced.rb — 1 commit
• examples/hold.rb — 1 commit

• fix: ! Fix quiet mode broken when config called without key quiet
• fix: Add gemspec. Fixes #23
• fix: Add missing file. Fixes #66
• fix: Add missing files. Fixes #74
• fix: Bump to 2.4.1. Fixes #59
• fix: Don't send the label through printf. Fixes #18
• fix: Exclude warmup if set to 0. Fixes #77.
• fix: Fix Job & Report class instance variables incorrect documentations.
• fix: Fix Ruby 1.8 support.
• fix: Fix Ruby 4.0 warning: use require_relative for internal files
• fix: Fix broken hold! when results file didn't exist
• fix: Fix missing noop_suite and job/noop_report files in v2.9.0 Manifest
• fix: Fix raise message, maximum => minimum.
• fix: Fix run method
• fix: Fix runtime comparison
• fix: Fix speeling
• fix: Fix tests I broke.
• fix: Fix the interaction between iterations and holding.
• fix: Fix travis targets
• fix: Ignore iterations that happen instantaneously. Fixes #1
• …and 279 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

evanphx/benchmark-ips was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.