elixir-broadway/broadway

This system is the Broadway library, a concurrent and multi-stage data collection and processing framework for Elixir. It manages asynchronous message pipelines by orchestrating producers, processors, and batchers within a supervised topology, supporting flexible configuration storage and structured telemetry. The codebase focuses on robust fault tolerance, granular acknowledgment handling, and extensibility for various messaging backends.

The internal pipeline structure has been significantly reorganized: the previous monolithic stages (Batcher, Consumer, Processor) have been removed and replaced with a modular topology system (Broadway.Topology) that manages distinct Producer, Processor, Batcher, and Terminator stages. Configuration is now validated via Broadway.Options and persisted using a pluggable ConfigStorage (supporting ETS or persistent\_term). The message model has expanded to include batch\_key, batch\_mode, and metadata fields, while the acknowledgement system now uses a triplet {module, ack\_ref, data} and supports a configure callback, enabling more granular control over message acknowledgment and failure handling.

lib/broadway · high confidence

The Broadway module has been refactored from a simple GenServer into a supervisor-based architecture. This change introduces a new \start\_link/2\ API that accepts a configuration keyword list (defining producers, processors, etc.) instead of separate module and context arguments. Internally, Broadway now manages a child supervision tree for producers, processors, and batchers, enabling features like graceful shutdown and improved fault tolerance where producers are isolated from the rest of the pipeline.

lib · high confidence

The internal Broadway topology stages (ProducerStage, ProcessorStage, BatchProcessorStage, BatcherStage, RateLimiter, Subscriber, and Terminator) have been restructured to replace manual span tracking with the standard :telemetry.span/3 API. This change enriches telemetry events with consistent, structured metadata—including topology\_name, stage name, partition index, context, and producer information—allowing users to correlate metrics and traces across the entire processing pipeline more effectively.

lib/broadway/topology · high confidence

The library now provides two concrete implementations for the \Broadway.ConfigStorage\ behaviour: \Broadway.ConfigStorage.ETS\, which stores topology data in a named ETS table, and \Broadway.ConfigStorage.PersistentTerm\, which uses Erlang's persistent term storage. These modules replace the previous ad-hoc persistent term usage with a structured, behaviour-based approach, allowing users to choose the storage mechanism that best fits their runtime environment and requirements.

_lib/broadway/config\storage · high confidence

The application configuration file (config/config.exs) has been removed, eliminating the use of the deprecated Mix.Config module. This change aligns the configuration setup with modern Elixir standards, likely requiring users to migrate to the newer Config module or adjust their build environment to accommodate the removal of this legacy dependency.

config · medium confidence

Added tests for Broadway topology stage behaviors; Added tests for acknowledger crash handling, ETS config storage, and dummy producer; Expanded test suite and test infrastructure updates.

This release updates the project to version 1.3.0 and refreshes core dependencies to support modern Elixir/OTP versions: GenStage is bumped to \~\> 1.0, while nimble\_options and telemetry now accept \~\> 0.3.7, \~\> 0.4, or \~\> 1.0 ranges. The application configuration now uses persistent\_term for config storage and explicitly registers the Broadway.Application module. Documentation tooling is updated to ExDoc 0.40.1 with new guides for Amazon SQS, Apache Kafka, Google Cloud PubSub, RabbitMQ, and custom producers, and test coverage reporting is configured via ExCoveralls.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 64 → 74 (+9.3)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 98 → 98 (+0.7)
• Architecture 100 → 100 (+0.0)
• Maturity 63 → 63 (+0.5)
• Readiness 50 → 67 (+16.9)
• Security 87 → 100 (+13.5)

• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — no supported dependency manifest was read
• Duplicated block (9 lines × 2) (lib/broadway/topology/batch_processor_stage.ex)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• No exposed public API
• Off-boarding risk: anonymized user #1
• Test reliability not included

• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (9 lines × 2) (lib/broadway/topology/batch_processor_stage.ex)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• No dependency advisory monitoring
• Off-boarding risk: anonymized user #1
• Outdated: castore
• Outdated: ex_doc
• Outdated: excoveralls
• Outdated: gen_stage
• Outdated: telemetry
• TodoComment (lib/broadway/process.ex)
• TodoComment (lib/broadway/topology.ex)
• Workflow token permissions not restricted

• 2 commits — 0 feature/other, 2 fixes

• guides/examples — 1 commit
• lib/broadway.ex — 1 commit

• fix: Fix off-Broadway list ordering and minor documentation errors (#379)
• fix: Fix stale test_messages/3 reference in testing docs (#378)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

elixir-broadway/broadway was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.