bencheeorg/benchee
76.3
Strong · 13 November 2026
7.3k
lines of production code
Elixir
primary language
8
measurements over time
What this system is
Features
Add development tooling configuration for Credo, ExGuard, and Elixir formatter
The project now includes configuration files for its development toolchain: \.credo.exs\ sets up static code analysis rules (including exclusions for test files and formatters), \.exguard.exs\ configures automatic re-running of compilation, Credo, formatting, dialyzer, and tests on file changes, and \.formatter.exs\ defines the Elixir formatter inputs and registers the \DoctestFormatter\ plugin. These changes standardize code quality checks and developer workflow automation.
(repo-wide) · high confidence
Add memory and reduction metrics collection to benchmarks
Benchee now supports collecting memory consumption and reduction count metrics in addition to execution time. New collector modules (\Benchee.Benchmark.Collect.Memory\ and \Benchee.Benchmark.Collect.Reductions\) implement the \Benchee.Benchmark.Collect\ behavior to measure heap usage via Erlang tracing and process reductions, respectively, allowing users to benchmark these additional performance dimensions alongside the existing time-based measurements.
lib/benchee/benchmark/collect · high confidence
Added architecture documentation with C4 diagrams
The \architecture\ folder now contains a comprehensive README and C4 diagrams (Context, Container, and Component levels) that visualize Benchee's internal structure. These diagrams illustrate the 'pipes' architecture, showing how the \Benchee.Suite\ is progressively enhanced through exchangeable steps like configuration, system data gathering, benchmarking, statistics calculation, and output formatting.
architecture · high confidence
Expanded sample library for advanced benchmarking scenarios
The samples directory has been significantly expanded with new examples demonstrating advanced Benchee capabilities. Users can now see how to deactivate specific console outputs (like fast warnings or comparison reports) via the \print\ and formatter options, run benchmarks with profiling enabled using \profile\_after\ (supporting both \eprof\ and \tprof\), and measure BEAM reductions using \reduction\_time\. Additional samples illustrate how to handle fast functions that trigger compiler optimizations, benchmark memory usage with varying input sizes, remove outliers from statistics, and run benchmarks in parallel. The collection also includes examples for saving and loading benchmark suites, using custom formatters, and performing pre-checks to ensure benchmark stability.
samples · high confidence
New utility modules for file creation, version parsing, and parallel processing
Added several new internal utility modules to the \lib/benchee/utility\ directory: \FileCreation\ handles writing benchmark results to files with automatic directory creation and filename interleaving; \ErlangVersion\ provides robust parsing and comparison of Erlang/OTP version strings, including handling of non-standard formats and release candidates; \DeepConvert\ converts deep keyword lists to maps with optional key exclusions; \Parallel\ offers a simple interface for mapping functions over collections using spawned tasks; and \RepeatN\ allows calling a function a specified number of times.
lib/benchee/utility · high confidence
Behavioural changes
Benchee benchmarking framework major refactoring and new features
This change introduces a comprehensive overhaul of the Benchee benchmarking library, restructuring core data models and adding significant new capabilities. The \Suite\, \Scenario\, and \Configuration\ structs have been redesigned to support more flexible input handling, parallel execution, and detailed configuration options like \pre\_check\, \profile\_after\, and \exclude\_outliers\. New modules such as \Benchee.Benchmark\, \Benchee.CollectionData\, and \Benchee.RelativeStatistics\ provide a more robust foundation for running benchmarks, collecting data, and calculating relative performance metrics between scenarios. The framework now supports built-in profilers (\:cprof\, \:eprof\, \:fprof\, and conditionally \:tprof\), allows disabling function call overhead measurement by default, and exposes JIT status in system information. Additionally, the formatter system has been refactored to support parallel output generation and scrubbing of heavy input data to reduce memory consumption during formatting.
lib/benchee · high confidence
Console formatter splits into dedicated modules for runtime, memory, and reductions
The console formatter's output logic has been reorganized into separate modules (\Benchee.Formatters.Console.RunTime\, \Memory\, and \Reductions\) to handle each metric type independently, while common formatting utilities are shared via a new \Helpers\ module. This change enables the console reports to display reduction counts alongside runtime and memory statistics, and ensures that extended statistics (such as minimum, maximum, sample size, and mode) are only shown when explicitly requested via configuration.
lib/benchee/formatters/console · high confidence
New Console and Tagged Save formatters with reduction counting
The formatter system has been refactored to introduce two new formatters: \Benchee.Formatters.Console\ and \Benchee.Formatters.TaggedSave\. The Console formatter now provides a structured CLI output that includes a comparison report, memory usage statistics, and, for the first time, reduction count statistics alongside standard run-time metrics. The TaggedSave formatter replaces the previous ExternalTermFormat approach, allowing users to save benchmark suites with auto-incrementing tags for easier versioning and later loading. This change also introduces a new \Benchee.Formatter\ behavior that formatters must implement, replacing the older module-based output methods.
lib/benchee/formatters · high confidence
New high-level Benchee.run and Benchee.report APIs
The library now exposes a simplified high-level interface for running benchmarks and reporting on saved data. Users can call \Benchee.run/2\ with a map of jobs and configuration options to execute benchmarks and output results in a single step, or use \Benchee.report/1\ to load previously saved benchmark scenarios and generate reports without re-running the benchmarks. This replaces the previous lower-level workflow, streamlining the process for both Elixir and Erlang users.
lib · high confidence
New structured output printers for benchmarking and profiling
Benchee now uses dedicated modules (BenchmarkPrinter, ProfilePrinter, ProgressPrinter) to handle console output, introducing several user-visible changes. The configuration summary now explicitly lists memory and reduction measurement times alongside warmup and time settings, and indicates whether outliers are excluded. System information output has been updated to include JIT status and available memory. Users will now see a warning if they benchmark evaluated functions instead of compiled ones, and a notice displaying the measured function call overhead. Additionally, the 'benchmarking' status message can be suppressed via configuration, and the progress output now clearly distinguishes between calculating statistics and formatting results.
lib/benchee/output · high confidence
Refactored benchmark execution with dedicated configuration and hook support
The benchmarking engine has been restructured to improve testability and introduce new control features. A new \Benchee.Benchmark.BenchmarkConfig\ struct now isolates the specific settings required for running benchmarks, separating them from the broader application configuration. This change enables the implementation of \pre\_check\ (validating that functions return consistent results before benchmarking) and \measure\_function\_call\_overhead\ (allowing users to opt-in to measuring system overhead, which is now disabled by default to keep test times low). Additionally, the runner now fully supports \before\_each\, \after\_each\, \before\_scenario\, and \after\_scenario\ hooks, allowing users to inject setup and teardown logic that runs with every iteration or scenario, and passing input data and return values through these hooks.
lib/benchee/benchmark · high confidence
Refactored unit conversion with new behaviors and struct
The conversion logic has been restructured to use a \Benchee.Conversion.Unit\ struct and separate \Scale\ and \Format\ behaviors. This change introduces a \DeviationPercent\ converter for standard deviation display, implements \format\_human\ for more readable time outputs (e.g., "1h 30min"), and adds support for scaling memory units (bytes to terabytes).
lib/benchee/conversion · high confidence
Removed default application configuration file
The default configuration file (config/config.exs) has been removed from the project. This eliminates the pre-configured environment for the application and its dependencies, meaning users will no longer have default configuration values available via the standard configuration module unless they provide their own.
config · high confidence
Test coverage
3 commits adding/updating tests in test; Added test fixture for escript building; Added test support utilities and mock implementations; Added tests for benchmark runner, repeated measurements, and scenario handling; Added tests for console and tagged save formatters; Added tests for console formatter helpers and metric-specific output modules; Added tests for memory and reduction collection benchmarks; Added tests for the benchmark output printer; Added unit and integration tests for utility modules; Added unit tests for Benchee conversion modules; Added unit tests for Benchee core modules; Expanded integration tests and platform-specific test exclusions.
Dependencies
Benchee 1.5.1 release with Elixir 1.6+ requirement and expanded tooling
This update releases Benchee version 1.5.1, raising the minimum supported Elixir version from 1.2 to 1.6. It introduces several new development and testing dependencies, including credo (\~\> 1.7.13), ex\_doc, dialyxir, doctest\_formatter, and ex\_guard, while updating core dependencies like deep\_merge to 1.0.2 and statistex to \~1.1. The project now explicitly enables protocol consolidation, configures dialyzer with a custom PLT path, and adds ExCoveralls for test coverage reporting. Additionally, the \table\ dependency is now optional and only included for Elixir versions 1.7 and above, and the package metadata has been updated with maintainer information and license details.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 77 → 76 (-0.8)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.3) — scores are not directly comparable.
Lenses
- Code Health 100 → 100 (+0.0)
- Architecture 98 → 97 (-0.7)
- Maturity 61 → 65 (+4.0)
- Readiness 84 → 75 (-9.7)
- Security 95 → 94 (-1.4)
- Performance 100 → 100 (+0.0)
Resolved (3)
- Outdated: credo
- Outdated: ex_doc
- Outdated: ex_guard
New (6)
- Edited copies of one member (3 members, 11+ corresponding lines) (lib/benchee/formatters/console/memory.ex)
- Edited copies of one member (6 members, 26+ corresponding lines) (lib/benchee/formatters/console/memory.ex)
- Edited copy of a member (5 corresponding lines) (lib/benchee/formatters/console/memory.ex)
- Low IaC: DS-0002 (.devcontainer/Dockerfile)
- Low IaC: DS-0029 (.devcontainer/Dockerfile)
- Low IaC: WD-DOCKER-0003 (.devcontainer/Dockerfile)
Changes since last survey
- 8 commits — 7 feature/other, 1 fixes
By area
- (root) — 3 commits
- test/benchee — 2 commits
- .devcontainer/Dockerfile — 1 commit
- .devcontainer/devcontainer.json — 1 commit
- .github/workflows — 1 commit
Notable commits
- fix: Whoopsie, credo fixes
- change: Combat a bunch of flakies
- change: Devcontainer setup
- change: Improve iex shutdown in test
- change: Upgrade deps
- change: elixir 1.20 and erlang 29.1 as well as prettier string quote updates
- change: introduce my beloved mix check
- change: remove prettier, I hate it touching my markdown
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
bencheeorg/benchee was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 13 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit f3ebaf9e9e01d97e04b400f063ade745163c3913 — the exact code this score is about.
- Scored under rubric-2026.11.3 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.57+preprod-8ebfb01cf4c0.