bencheeorg/benchee

Benchee is a benchmarking library for Elixir that provides a high-level API for running and reporting on performance tests. It measures execution time, memory usage, and function call overhead, while also supporting parallel execution and statistical outlier removal. The system features a modular architecture for collecting metrics, converting units, and formatting results to the console or save files. Comprehensive test coverage ensures the reliability of its benchmarking, profiling, and output formatting capabilities.

15 changes.

This period focused on modernizing Benchee's public API with high-level convenience functions and a pluggable unit conversion system. The internal architecture was significantly refactored to support new benchmarking features like parallel execution, memory measurement, and profiling, accompanied by comprehensive test coverage for the new modules.

11 changes.

This period focused on restructuring Benchee's internal architecture to support new measurement capabilities, specifically tracking memory usage, function call overhead, and BEAM reductions. The codebase was refactored into modular components for benchmarking, collection, and console formatting, accompanied by comprehensive test coverage for these new features.

The \architecture\ directory now contains C4 diagrams (context, container, and component levels) along with a README that explains Benchee's pipe-based architecture. The diagrams are rendered as SVGs with a purple color scheme, providing a visual overview of how the \Benchee.Suite\ is progressively enhanced through exchangeable steps.

architecture · high confidence

The repository now includes configuration files for development tooling: \.credo.exs\ for static code analysis, \.exguard.exs\ for automated task execution on file changes, and \.formatter.exs\ for Elixir code formatting. These files establish the project's coding standards, automated checks, and formatting rules.

(repo-wide) · high confidence

The benchmarking tool now supports collecting memory usage, reduction counts, and high-precision execution time. New modules \Benchee.Benchmark.Collect.Memory\, \.Reductions\, and \.Time\ implement the \Benchee.Benchmark.Collect\ behaviour, enabling users to measure memory consumption, track function reductions, and measure execution time in nanoseconds.

lib/benchee/benchmark/collect · high confidence

The samples directory has been significantly expanded to demonstrate a wider range of Benchee features. New examples include outlier removal, parallel process benchmarking, pre-checks for fast functions, memory measurement, reduction counting, and profiling with eprof and tprof. Existing samples have been updated to reflect current API usage, such as the keyword list options style, and various bugs related to output formatting and configuration have been fixed.

samples · high confidence

The library now provides a simplified, high-level interface for running benchmarks and reporting on saved data. Users can execute benchmarks using \Benchee.run/2\ with a map of job names to functions, or load and report on previously saved benchmark data using the new \Benchee.report/1\ function. These functions wrap the internal suite construction and formatting steps, making the common use cases more accessible.

lib · high confidence

Benchee now uses dedicated printer modules (BenchmarkPrinter, ProfilePrinter, ProgressPrinter) to handle output. Users will see system information (OS, CPU, memory, Elixir/Erlang versions, JIT status) and configuration details (warmup, time, memory/reduction times, parallelism, inputs, estimated total time, and outlier exclusion status) printed before benchmarks run. The tool also prints progress updates for calculating statistics and formatting results, warns about fast functions and evaluated functions, and respects print configuration flags (e.g., suppressing benchmarking messages if print.benchmarking is false).

lib/benchee/output · high confidence

The library introduces several new utility modules to support benchmarking workflows. Benchee.Utility.DeepConvert provides a \to\_map/2\ function to recursively convert keyword lists to maps with optional key exclusions. Benchee.Utility.ErlangVersion adds logic to parse and compare Erlang/OTP versions, handling non-standard version formats to check for specific bug fixes. Benchee.Utility.FileCreation offers an \each/3\ function to write content to files with generated names and automatic directory creation, supporting interleaved naming and UTF-8 encoding. Benchee.Utility.Parallel provides a \map/2\ function for parallel execution of tasks. Benchee.Utility.RepeatN adds a \repeat\_n/2\ function to execute a function a specified number of times.

lib/benchee/utility · high confidence

Benchee has been refactored to use new internal data structures, including Benchee.Suite, Benchee.Scenario, and Benchee.Configuration, replacing the previous flat data model. This change introduces support for measuring function call overhead (disabled by default), counting reductions, and excluding statistical outliers. The configuration system now supports customizable percentiles, parallel benchmarking, and pre-checks. Additionally, the library now exposes system information (Elixir/Erang versions, CPU, memory) and supports built-in profilers (eprof, cprof, fprof, tprof).

lib/benchee · high confidence

The benchmarking logic has been refactored into new internal modules: \Benchee.Benchmark.Runner\, \Benchee.Benchmark.BenchmarkConfig\, \Benchee.Benchmark.ScenarioContext\, and \Benchee.Benchmark.Hooks\. This restructuring isolates the execution flow, configuration, and hook management. Additionally, the library now supports measuring function call overhead (disabled by default) and includes a new \reduction\_time\ measurement capability, allowing users to track BEAM reductions alongside runtime and memory usage.

lib/benchee/benchmark · high confidence

The console formatter has been restructured to support multiple inputs, displaying separate sections for run time, memory usage, and reduction counts, with headers for each input. The save formatter was renamed from ExternalTermFormat to TaggedSave, now automatically tagging scenarios with a specified tag for later loading. Both formatters now implement the Benchee.Formatter behaviour, with the console formatter handling title headers, input headers, and separate statistics for run time, memory, and reductions.

lib/benchee/formatters · high confidence

The \Benchee.Conversion\ module has been restructured to support a pluggable unit system. A new \Unit\ struct and \Scale\ and \Format\ behaviours allow different domains (Count, Duration, Memory, DeviationPercent) to define their own scaling and formatting logic. This change introduces a more robust and extensible way to handle unit conversions, including support for human-friendly formatting and multiple scaling strategies.

lib/benchee/conversion · high confidence

The default configuration file (config/config.exs) has been removed from the project. This file previously contained placeholder comments and example configurations for the Benchee library. Its removal indicates a shift away from maintaining a static configuration file, likely relying on environment-specific configurations or in-memory defaults instead.

config · high confidence

The console output logic has been refactored into distinct modules for each metric type: \Benchee.Formatters.Console.RunTime\, \Benchee.Formatters.Console.Memory\, and \Benchee.Formatters.Console.Reductions\. This separation allows each formatter to handle its specific data structure and display requirements without code duplication or confusion, while sharing common helper functions in \Benchee.Formatters.Console.Helpers\. Users will see a cleaner, more maintainable codebase that supports independent formatting of run time, memory usage, and reduction counts in the CLI output.

lib/benchee/formatters/console · high confidence

Added comprehensive test coverage for Benchee core modules; Added comprehensive test coverage for benchmarking logic; Added integration tests for Benchee's high-level and Erlang-style APIs; Added integration tests for benchmarking features; Added test fixture for escript building; Added test support modules and helpers; Added tests for console and tagged save formatters; Added tests for console formatters; Added tests for memory and reduction collection; Added tests for the Benchee output printer; Added tests for utility modules; Added unit tests for Benchee conversion modules.

The project's dependencies have been updated, including credo 1.7.18, ex\_doc 0.40.3, and statistex 1.1.1. The main \mix.exs\ file was significantly expanded to include package metadata (maintainers, licenses, links), documentation configuration, and dev/test tooling (dialyzer, excoveralls, doctest\_formatter). Additionally, a new fixture for an escript build was added, containing its own \mix.exs\ and \mix.lock\ files.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 70 → 77 (+7.3)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 100 → 100 (-0.2)
• Architecture 100 → 97 (-2.9)
• Maturity 63 → 64 (+1.4)
• Readiness 68 → 83 (+15.1)
• Security 74 → 94 (+20.3)

• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — no supported dependency manifest was read
• Duplicated block (10 lines × 2) (lib/benchee/formatters/console/memory.ex)
• Duplicated block (18 lines × 2) (lib/benchee/formatters/console/memory.ex)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• No exposed public API
• Off-boarding risk: anonymized user #1
• Test reliability not included
• The README describes only 'TODO: Add description' for the Escript package; no real content, installation, or usage. (test/fixtures/escript/README.md)

• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (11 lines × 2) (lib/benchee/formatters/console/memory.ex)
• Duplicated block (13 lines × 2) (lib/benchee/formatters/console/memory.ex)
• Duplicated block (13 lines × 3) (lib/benchee/formatters/console/memory.ex)
• Duplicated block (16 lines × 2) (lib/benchee/formatters/console/memory.ex)
• Duplicated block (23 lines × 2) (lib/benchee/formatters/console/memory.ex)
• Duplicated block (5 lines × 2) (lib/benchee/formatters/console/reductions.ex)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• No dependency advisory monitoring
• Outdated: credo
• …and 5 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

bencheeorg/benchee was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.