jet/propulsion

Propulsion is an event sourcing and streaming infrastructure library that ingests, indexes, and publishes events from diverse sources such as Kafka, Cosmos DB, DynamoDB, and SQL databases. It provides a modular pipeline architecture with configurable backends, checkpointing mechanisms, and monitoring capabilities via Prometheus metrics. The system supports serverless deployment through AWS CDK constructs and includes CLI tools for store management and event synchronization.

10 changes.

This period focused on establishing the foundational repository structure and removing legacy F\# wrappers while introducing core Kafka ingestion and production pipelines with canonical event serialization. Significant development included adding EventStore integration, a CLI tool for store management, and Prometheus metrics for monitoring feed and scheduler health. The work was supported by comprehensive unit and integration tests to validate ingestion logic, checkpointing, and consumer behavior.

13 changes.

This period focused on significantly expanding Propulsion's event sourcing capabilities by introducing new feed source and checkpoint store implementations for SQL Stream Store, EventStoreDB, MessageDb, and DynamoDB. The work also included porting the Cosmos Store to the v3 SDK, adding AWS CDK constructs for serverless DynamoDB components, and enhancing the core Feed module with FsCodec-based source abstractions.

A new Lambda function has been added to the Propulsion.DynamoStore.Notifier module to monitor DynamoDB streams and notify downstream systems of feed position changes. When a DynamoDB record is inserted or modified, the function parses the stream event to extract the partition ID, epoch ID, and event count, then publishes a batched message to an SNS topic containing the new feed position. This enables real-time propagation of stream updates to other services.

src/Propulsion.DynamoStore.Notifier · high confidence

The repository is initialized with the core Propulsion library source, test projects, and helper tooling. This entry establishes the project's foundational documentation, including the Apache 2.0 license, contributor guidelines, and in-depth system architecture documentation (C4 diagrams and glossary). It also sets up the build configuration via Directory.Build.props and targets, enabling versioning with MinVer and reproducible builds, alongside an .editorconfig to enforce consistent F\# coding standards (4-space indentation, LF line endings).

(repo-wide) · high confidence

A new AWS Lambda function has been added to automatically index events from the DynamoDB stream. This component listens for INSERT and MODIFY operations on the store, filters out system streams and empty batches, and ingests the resulting event spans into the indexer service. It supports both NEW\_IMAGE and NEW\_AND\_OLD\_IMAGES stream view types and handles the 'a' field count to determine batch sizes, ensuring that events are correctly mapped to their respective partitions.

src/Propulsion.DynamoStore.Indexer · high confidence

Adds a new DynamoDB-backed store implementation for Propulsion, enabling event ingestion, indexing, and streaming via the \DynamoStoreSource\. This includes the \DynamoStoreIndexer\ for exactly-once ingestion into epoch-based streams, an \AppendsIndex\ to track active epochs per partition, and a \DynamoStoreSource\ that feeds events to sinks with configurable load modes (index-only or with data hydration). A utility for importing data from DynamoDB JSON exports is also provided.

src/Propulsion.DynamoStore · high confidence

Adds a new EventStore source and sink implementation to Propulsion, enabling event ingestion from and writing to EventStore. The source supports reading from the $all stream using striped readers to balance load across 256MB chunks, with configurable start positions (absolute, chunk, percentage, or tail) and automatic checkpointing via Equinox Deciders. The sink handles batched writes to EventStore streams, managing conflicts, duplicates, and prefix gaps, while providing detailed throughput and error statistics.

src/Propulsion.EventStore · high confidence

Added a new EventStoreDb source component that reads events from an EventStoreDB instance using the \EventStore.Client\. The implementation supports filtering events by stream name via a \streamFilter\ predicate, allows disabling data payload retrieval with \withData\ for performance optimization, and enables starting consumption from the current tail of the stream. It integrates with the existing Propulsion feed infrastructure for checkpointing and sink delivery.

src/Propulsion.EventStoreDb · high confidence

Added the MemoryStoreSource component, which forwards committed events from an Equinox VolatileStore to a Sink pipeline. This source now supports AwaitCompletion, allowing users to wait until all submitted batches have been processed by the sink, and includes a Monitor for tracking prepared and completed epochs. The implementation uses FsCodec.Encoded bodies and integrates with Serilog for detailed logging of submission and completion states.

src/Propulsion.MemoryStore · high confidence

Added a new Propulsion source for MessageDb that reads events from the database via Npgsql and supports checkpointing positions to a dedicated table. Users can now subscribe to MessageDb categories using \MessageDbSource\ and manage read progress with \ReaderCheckpoint.CheckpointStore\, which persists offsets to ensure reliable event processing and recovery.

src/Propulsion.MessageDb · high confidence

The new Propulsion.Tool provides a command-line interface for managing and synchronizing event stores. It supports initialization and checkpoint management for Cosmos DB and DynamoDB, as well as a sync command to project events to Kafka or Cosmos DB. The tool allows filtering events by category, stream name, and event type, and supports reading from various sources including Cosmos DB, DynamoDB, MessageDb, and JSON files.

tools · high confidence

Added a new SQL Stream Store integration that enables reading event streams from a SQL database and persisting consumer group checkpoints. The implementation includes a checkpoint store backed by a dedicated 'Checkpoints' table (using Dapper and Microsoft.Data.SqlClient) to track stream positions, and a feed source that reads batches of events, supports filtering by stream name or category, and allows starting from the tail of the stream.

src/Propulsion.SqlStreamStore · high confidence

This change adds the \Propulsion.Codec.NewtonsoftJson\ module with \RenderedSpan\ and \RenderedSummary\ types to define a canonical JSON format for serializing and deserializing event spans and stream summaries. It also introduces the core Kafka integration components: \ProducerSinks.fs\ and \Producers.fs\ provide the \ParallelProducerSink\ and \StreamsProducerSink\ for writing events to Kafka with configurable batching and compression, while \Consumers.fs\ implements the \KafkaIngestionEngine\ and \ParallelConsumer\ to continuously poll Kafka partitions, manage back-pressure via in-flight message limits, and submit batches for processing.

src/Propulsion.Kafka · high confidence

Added three new AWS CDK constructs in the Propulsion.DynamoStore.Constructs library to define serverless infrastructure: DynamoStoreIndexerLambda for indexing DynamoDB streams, DynamoStoreNotifierLambda for publishing updates via SNS, and DynamoStoreReactorLambda for processing notifications via SQS. These constructs handle IAM role configuration, event source mappings, and runtime settings (targeting .NET 10 and ARM64) for the respective Lambda functions.

src/Propulsion.DynamoStore.Constructs · high confidence

The Feed module now includes new source implementations (FeedSource, TailingFeedSource, PeriodicSource, SinglePassFeedSource, and JsonSource) that drive reading and checkpointing for data sources using FsCodec.ITimelineEvent bodies. These sources manage tranche-based reading, checkpoint persistence, and batch ingestion into the sink pipeline, supporting both tailing and single-pass consumption patterns.

src/Propulsion/Feed · high confidence

The Notifier Lambda now processes SQS notification batches to track work completion per index partition. It parses incoming messages to identify the target partition and position, then compares these against the processed checkpoints to determine if any work remains incomplete. If a partition's position has not been reached, the corresponding message is marked for re-queue via SQS batch item failures, ensuring reliable processing of index updates.

src/Propulsion.DynamoStore.Lambda · high confidence

Propulsion now includes a new \Propulsion.Prometheus\ package that exposes feed and scheduler metrics to Prometheus via Serilog sinks. The \FeedPrometheus\ sink publishes read/ingest latencies, queue lengths, and page/item counts, while the \PropulsionPrometheus\ sink exposes scheduler state (categories, streams, events, bytes), CPU usage breakdowns, handler latencies, and busy stream statistics. This allows users to monitor the internal performance and health of Propulsion's ingestion and scheduling pipelines using standard Prometheus tooling.

src/Propulsion · high confidence

The F\# wrapper library for Confluent.Kafka has been removed from the repository. This change deletes the core implementation files (AssemblyInfo.fs, ConfluentKafka.fs, Script.fsx) and the dependency manifest (paket.references), eliminating the lightweight F\# wrapper, its configuration DSL, and associated libraries such as NLog, System.Reactive, and Confluent.Kafka from this project area.

src/Confluent.Kafka.FSharp · high confidence

The Propulsion.CosmosStore module has been rewritten to target the Microsoft.Azure.Cosmos SDK v3 (replacing the legacy v2 SDK). This update introduces a new ChangeFeedProcessor implementation, a CosmosStorePruner for event deletion, and updated parsers (EquinoxNewtonsoftParser for V3, EquinoxSystemTextJsonParser for legacy) to handle the new SDK's JSON serialization and change feed APIs. Users benefit from improved performance, better lease management, and access to modern Cosmos DB features, though it requires the underlying Equinox.Cosmos dependency to be compatible with the v3 SDK.

src/Propulsion.CosmosStore · high confidence

Added integration tests for Kafka consumers and monitoring; Added integration tests for MessageDb source and checkpointing; Added test coverage for Propulsion ingestion, indexing, and streaming components; Removed F\# integration tests for Confluent.Kafka.

This change updates core dependencies across the project, including Equinox.CosmosStore to 4.1.0, FsCodec.SystemTextJson to 3.1.0, and MinVer to 5.0.0. It also migrates the AWS Lambda components (Propulsion.DynamoStore.Constructs, Indexer, Lambda, Notifier) from .NET 6 to .NET 10, while retaining .NET 6 for the core store implementations (Cosmos, Dynamo, EventStore, Kafka, Memory, MessageDb, SqlStreamStore) and the CLI tool.

(dependencies) · high confidence

New PlantUML diagrams have been added to the \diagrams\ directory to document the architecture of the Propulsion system. These include a context diagram showing the high-level organization of Ingesters, Reactors, and Publishers alongside Equinox and external feeds, as well as detailed container diagrams for the Ingestion, Publishing, and Reactor scenarios. These visuals clarify data flows, component interactions (such as the Scheduler, Dispatcher, and Handler), and dependencies on external systems like event stores and read-model caches.

diagrams · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 64 → 64 (-0.2)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 88 → 88 (+0.2)
• Architecture 100 → 98 (-1.7)
• Maturity 68 → 63 (-5.4)
• Readiness 52 → 53 (+1.5)
• Security 68 → 71 (+3.5)
• Domain Modelling 100 → 100 (+0.0)

• Duplicated block (11 lines × 2) (src/Propulsion.EventStoreDb/EventStoreSource.fs)
• Duplicated block (12 lines × 2) (src/Propulsion.CosmosStore/PropulsionCosmosStorePrometheus.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.CosmosStore/FeedObserver.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.DynamoStore.Indexer/Handler.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.DynamoStore.Indexer/Handler.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.Kafka/Consumers.fs)
• Duplicated block (5 lines × 2) (src/Propulsion/Streams.fs)
• Duplicated block (6 lines × 2) (src/Propulsion.CosmosStore/CosmosStoreSink.fs)
• Duplicated block (6 lines × 2) (src/Propulsion.CosmosStore/ReaderCheckpoint.fs)
• Duplicated block (6 lines × 2) (src/Propulsion.CosmosStore/ReaderCheckpoint.fs)
• Duplicated block (6 lines × 2) (src/Propulsion/Streams.fs)
• Duplicated block (7 lines × 2) (src/Propulsion.CosmosStore/CosmosStoreSink.fs)
• Duplicated block (7 lines × 2) (src/Propulsion.CosmosStore/CosmosStoreSink.fs)
• Duplicated block (7 lines × 2) (src/Propulsion.DynamoStore.Constructs/DynamoStoreIndexerLambda.fs)
• Duplicated block (7 lines × 2) (src/Propulsion.DynamoStore/DynamoStoreSource.fs)
• Duplicated block (7 lines × 2) (src/Propulsion/Pipeline.fs)
• Duplicated block (8 lines × 2) (src/Propulsion.CosmosStore/PropulsionCosmosStorePrometheus.fs)
• Duplicated block (8 lines × 2) (src/Propulsion.DynamoStore.Indexer/Function.fs)
• Further sole-owners (lower concentration)
• High: security finding (details withheld)
• …and 8 more

• Concentrated knowledge decay
• Critical CVE: System.Drawing.Common 4.7.0
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (11 lines × 2) (src/Propulsion.CosmosStore/PropulsionCosmosStorePrometheus.fs)
• Duplicated block (12–13 lines × 2) (src/Propulsion.EventStoreDb/EventStoreSource.fs)
• Duplicated block (18–19 lines × 2) (src/Propulsion.CosmosStore/CosmosStoreSink.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.CosmosStore/CosmosStoreSink.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.CosmosStore/FeedObserver.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.CosmosStore/ReaderCheckpoint.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.DynamoStore.Indexer/Handler.fs)
• Duplicated block (5 lines × 2) (src/Propulsion.DynamoStore/DynamoStoreSource.fs)
• Duplicated block (5–6 lines × 2) (src/Propulsion.Kafka/Consumers.fs)
• Duplicated block (6 lines × 2) (src/Propulsion.CosmosStore/CosmosStoreSink.fs)
• Duplicated block (6 lines × 2) (src/Propulsion.CosmosStore/ReaderCheckpoint.fs)
• Duplicated block (6 lines × 2) (src/Propulsion/Pipeline.fs)
• Duplicated block (6 lines × 2) (src/Propulsion/Streams.fs)
• Duplicated block (7 lines × 2) (src/Propulsion.DynamoStore.Indexer/Function.fs)
• Duplicated block (8 lines × 2) (src/Propulsion.CosmosStore/PropulsionCosmosStorePrometheus.fs)
• Duplicated block (8–10 lines × 2) (src/Propulsion.DynamoStore.Constructs/DynamoStoreIndexerLambda.fs)
• …and 17 more

• 1 commits — 1 feature/other, 0 fixes

• src/Propulsion.DynamoStore.Constructs — 1 commit

• change: deps(DynamoStore): Up AWS/Lambda to net10 (#277)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

jet/propulsion was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.