Skip to content
CAI
Software that uses CAICheck a score

bencheeorg/benchee

76.3

Strong · 13 November 2026

7.3k

lines of production code

Elixir

primary language

8

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add development tooling configuration for Credo, ExGuard, and Elixir formatter

The project now includes configuration files for its development toolchain: \.credo.exs\ sets up static code analysis rules (including exclusions for test files and formatters), \.exguard.exs\ configures automatic re-running of compilation, Credo, formatting, dialyzer, and tests on file changes, and \.formatter.exs\ defines the Elixir formatter inputs and registers the \DoctestFormatter\ plugin. These changes standardize code quality checks and developer workflow automation.

(repo-wide) · high confidence

Add memory and reduction metrics collection to benchmarks

Benchee now supports collecting memory consumption and reduction count metrics in addition to execution time. New collector modules (\Benchee.Benchmark.Collect.Memory\ and \Benchee.Benchmark.Collect.Reductions\) implement the \Benchee.Benchmark.Collect\ behavior to measure heap usage via Erlang tracing and process reductions, respectively, allowing users to benchmark these additional performance dimensions alongside the existing time-based measurements.

lib/benchee/benchmark/collect · high confidence

Added architecture documentation with C4 diagrams

The \architecture\ folder now contains a comprehensive README and C4 diagrams (Context, Container, and Component levels) that visualize Benchee's internal structure. These diagrams illustrate the 'pipes' architecture, showing how the \Benchee.Suite\ is progressively enhanced through exchangeable steps like configuration, system data gathering, benchmarking, statistics calculation, and output formatting.

architecture · high confidence

Expanded sample library for advanced benchmarking scenarios

The samples directory has been significantly expanded with new examples demonstrating advanced Benchee capabilities. Users can now see how to deactivate specific console outputs (like fast warnings or comparison reports) via the \print\ and formatter options, run benchmarks with profiling enabled using \profile\_after\ (supporting both \eprof\ and \tprof\), and measure BEAM reductions using \reduction\_time\. Additional samples illustrate how to handle fast functions that trigger compiler optimizations, benchmark memory usage with varying input sizes, remove outliers from statistics, and run benchmarks in parallel. The collection also includes examples for saving and loading benchmark suites, using custom formatters, and performing pre-checks to ensure benchmark stability.

samples · high confidence

New utility modules for file creation, version parsing, and parallel processing

Added several new internal utility modules to the \lib/benchee/utility\ directory: \FileCreation\ handles writing benchmark results to files with automatic directory creation and filename interleaving; \ErlangVersion\ provides robust parsing and comparison of Erlang/OTP version strings, including handling of non-standard formats and release candidates; \DeepConvert\ converts deep keyword lists to maps with optional key exclusions; \Parallel\ offers a simple interface for mapping functions over collections using spawned tasks; and \RepeatN\ allows calling a function a specified number of times.

lib/benchee/utility · high confidence

Behavioural changes

Benchee benchmarking framework major refactoring and new features

This change introduces a comprehensive overhaul of the Benchee benchmarking library, restructuring core data models and adding significant new capabilities. The \Suite\, \Scenario\, and \Configuration\ structs have been redesigned to support more flexible input handling, parallel execution, and detailed configuration options like \pre\_check\, \profile\_after\, and \exclude\_outliers\. New modules such as \Benchee.Benchmark\, \Benchee.CollectionData\, and \Benchee.RelativeStatistics\ provide a more robust foundation for running benchmarks, collecting data, and calculating relative performance metrics between scenarios. The framework now supports built-in profilers (\:cprof\, \:eprof\, \:fprof\, and conditionally \:tprof\), allows disabling function call overhead measurement by default, and exposes JIT status in system information. Additionally, the formatter system has been refactored to support parallel output generation and scrubbing of heavy input data to reduce memory consumption during formatting.

lib/benchee · high confidence

Console formatter splits into dedicated modules for runtime, memory, and reductions

The console formatter's output logic has been reorganized into separate modules (\Benchee.Formatters.Console.RunTime\, \Memory\, and \Reductions\) to handle each metric type independently, while common formatting utilities are shared via a new \Helpers\ module. This change enables the console reports to display reduction counts alongside runtime and memory statistics, and ensures that extended statistics (such as minimum, maximum, sample size, and mode) are only shown when explicitly requested via configuration.

lib/benchee/formatters/console · high confidence

New Console and Tagged Save formatters with reduction counting

The formatter system has been refactored to introduce two new formatters: \Benchee.Formatters.Console\ and \Benchee.Formatters.TaggedSave\. The Console formatter now provides a structured CLI output that includes a comparison report, memory usage statistics, and, for the first time, reduction count statistics alongside standard run-time metrics. The TaggedSave formatter replaces the previous ExternalTermFormat approach, allowing users to save benchmark suites with auto-incrementing tags for easier versioning and later loading. This change also introduces a new \Benchee.Formatter\ behavior that formatters must implement, replacing the older module-based output methods.

lib/benchee/formatters · high confidence

New high-level Benchee.run and Benchee.report APIs

The library now exposes a simplified high-level interface for running benchmarks and reporting on saved data. Users can call \Benchee.run/2\ with a map of jobs and configuration options to execute benchmarks and output results in a single step, or use \Benchee.report/1\ to load previously saved benchmark scenarios and generate reports without re-running the benchmarks. This replaces the previous lower-level workflow, streamlining the process for both Elixir and Erlang users.

lib · high confidence

New structured output printers for benchmarking and profiling

Benchee now uses dedicated modules (BenchmarkPrinter, ProfilePrinter, ProgressPrinter) to handle console output, introducing several user-visible changes. The configuration summary now explicitly lists memory and reduction measurement times alongside warmup and time settings, and indicates whether outliers are excluded. System information output has been updated to include JIT status and available memory. Users will now see a warning if they benchmark evaluated functions instead of compiled ones, and a notice displaying the measured function call overhead. Additionally, the 'benchmarking' status message can be suppressed via configuration, and the progress output now clearly distinguishes between calculating statistics and formatting results.

lib/benchee/output · high confidence

Refactored benchmark execution with dedicated configuration and hook support

The benchmarking engine has been restructured to improve testability and introduce new control features. A new \Benchee.Benchmark.BenchmarkConfig\ struct now isolates the specific settings required for running benchmarks, separating them from the broader application configuration. This change enables the implementation of \pre\_check\ (validating that functions return consistent results before benchmarking) and \measure\_function\_call\_overhead\ (allowing users to opt-in to measuring system overhead, which is now disabled by default to keep test times low). Additionally, the runner now fully supports \before\_each\, \after\_each\, \before\_scenario\, and \after\_scenario\ hooks, allowing users to inject setup and teardown logic that runs with every iteration or scenario, and passing input data and return values through these hooks.

lib/benchee/benchmark · high confidence

Refactored unit conversion with new behaviors and struct

The conversion logic has been restructured to use a \Benchee.Conversion.Unit\ struct and separate \Scale\ and \Format\ behaviors. This change introduces a \DeviationPercent\ converter for standard deviation display, implements \format\_human\ for more readable time outputs (e.g., "1h 30min"), and adds support for scaling memory units (bytes to terabytes).

lib/benchee/conversion · high confidence

Removed default application configuration file

The default configuration file (config/config.exs) has been removed from the project. This eliminates the pre-configured environment for the application and its dependencies, meaning users will no longer have default configuration values available via the standard configuration module unless they provide their own.

config · high confidence

Test coverage

3 commits adding/updating tests in test; Added test fixture for escript building; Added test support utilities and mock implementations; Added tests for benchmark runner, repeated measurements, and scenario handling; Added tests for console and tagged save formatters; Added tests for console formatter helpers and metric-specific output modules; Added tests for memory and reduction collection benchmarks; Added tests for the benchmark output printer; Added unit and integration tests for utility modules; Added unit tests for Benchee conversion modules; Added unit tests for Benchee core modules; Expanded integration tests and platform-specific test exclusions.

Dependencies

Benchee 1.5.1 release with Elixir 1.6+ requirement and expanded tooling

This update releases Benchee version 1.5.1, raising the minimum supported Elixir version from 1.2 to 1.6. It introduces several new development and testing dependencies, including credo (\~\> 1.7.13), ex\_doc, dialyxir, doctest\_formatter, and ex\_guard, while updating core dependencies like deep\_merge to 1.0.2 and statistex to \~1.1. The project now explicitly enables protocol consolidation, configures dialyzer with a custom PLT path, and adds ExCoveralls for test coverage reporting. Additionally, the \table\ dependency is now optional and only included for Elixir versions 1.7 and above, and the package metadata has been updated with maintainer information and license details.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 77 → 76 (-0.8)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.3) — scores are not directly comparable.

Lenses

  • Code Health 100 → 100 (+0.0)
  • Architecture 98 → 97 (-0.7)
  • Maturity 61 → 65 (+4.0)
  • Readiness 84 → 75 (-9.7)
  • Security 95 → 94 (-1.4)
  • Performance 100 → 100 (+0.0)

Resolved (3)

  • Outdated: credo
  • Outdated: ex_doc
  • Outdated: ex_guard

New (6)

  • Edited copies of one member (3 members, 11+ corresponding lines) (lib/benchee/formatters/console/memory.ex)
  • Edited copies of one member (6 members, 26+ corresponding lines) (lib/benchee/formatters/console/memory.ex)
  • Edited copy of a member (5 corresponding lines) (lib/benchee/formatters/console/memory.ex)
  • Low IaC: DS-0002 (.devcontainer/Dockerfile)
  • Low IaC: DS-0029 (.devcontainer/Dockerfile)
  • Low IaC: WD-DOCKER-0003 (.devcontainer/Dockerfile)

Changes since last survey

  • 8 commits — 7 feature/other, 1 fixes

By area

  • (root) — 3 commits
  • test/benchee — 2 commits
  • .devcontainer/Dockerfile — 1 commit
  • .devcontainer/devcontainer.json — 1 commit
  • .github/workflows — 1 commit

Notable commits

  • fix: Whoopsie, credo fixes
  • change: Combat a bunch of flakies
  • change: Devcontainer setup
  • change: Improve iex shutdown in test
  • change: Upgrade deps
  • change: elixir 1.20 and erlang 29.1 as well as prettier string quote updates
  • change: introduce my beloved mix check
  • change: remove prettier, I hate it touching my markdown

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

bencheeorg/benchee was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 13 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit f3ebaf9e9e01d97e04b400f063ade745163c3913 — the exact code this score is about.
  • Scored under rubric-2026.11.3 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.57+preprod-8ebfb01cf4c0.