evolution-gaming/kafka-journal

This system is a Kafka-backed event sourcing journal that enables Akka and Pekko persistence plugins to store and replay events in Kafka, with optional eventual consistency via Cassandra. It provides core infrastructure for serializing, batching, and replicating journal entries, including support for snapshots and hierarchical replication from Kafka to Cassandra. The library also includes utilities for JSON serialization, configuration management, and local development environments.

Users can now serialize and deserialize Akka Persistence events using Circe JSON codecs. This change introduces a new \KafkaJournalCirce\ component that provides a \JsonEventSerializer\, enabling support for both raw JSON and text payloads within persistent journal entries, along with corresponding test coverage for serialization round-trips and error handling.

akka/persistence-circe · high confidence

New shell scripts and configuration files have been added to the scripts directory to facilitate running a local development environment. This includes a Zookeeper instance and three Kafka broker instances (broker.0, broker.1, broker.2), each with distinct IDs, log directories, and listening ports (9092, 9093, 9094). The configurations are tuned for development, such as setting the initial consumer rebalance delay to 0 and disabling per-IP connection limits on Zookeeper.

scripts · high confidence

The core library now includes a comprehensive set of domain types for the journal, including \SeqNr\, \Key\, \Payload\, \PayloadType\, \DeleteTo\, \ExpireAfter\, \Origin\, and \Version\, along with their serialization codecs and JSON formats. New abstractions for byte-level encoding and decoding (\ToBytes\, \FromBytes\, \JsonCodec\) and error handling (\Fail\, \JournalError\) have been added to support the journal's internal data flow. Additionally, execution infrastructure for managing thread pools (\ForkJoinPoolOf\, \ThreadPoolOf\, \ScheduledExecutorServiceOf\) and resource references (\ResourceRef\) is now available in the \execution\ and \util\ packages.

core · high confidence

The \pekko/persistence-circe\ module now provides a \KafkaJournalCirce\ implementation that serializes persistence events to and from JSON using the Circe library. This allows users to leverage Circe's type-class-based encoding and decoding for event payloads, supporting both JSON and text payload types, and includes corresponding tests to verify serialization round-trips and error handling for unsupported payload types.

pekko/persistence-circe · high confidence

Added core types and interfaces for managing journal snapshots, including the \SnapshotStore\ trait for saving, loading, and deleting snapshots by key and selection criteria, along with supporting data models like \Snapshot\, \SnapshotRecord\, \SnapshotSelectionCriteria\, and \BufferNr\. These changes provide the foundational API for persisting and retrieving state snapshots within the Kafka Journal system.

snapshot · high confidence

The journal now supports a new 'mark' operation, allowing users to attach a unique identifier to a key for tracking or synchronization purposes alongside existing append, delete, and purge actions. This capability is introduced via new \Action.Mark\ and \ActionHeader.Mark\ types and the \AppendMarker\ trait, which generates random IDs and produces marker records to Kafka. The underlying data model has been refactored to use a unified \Action\ sealed hierarchy with explicit \ActionHeader\ variants, replacing previous implicit or less structured representations. Additionally, the package has been moved to \com.evolution.kafka.journal\, and internal components like \HeadCache\ and \ConsumerPool\ have been restructured to support these changes with improved metrics and resource management.

journal/src/main/scala/com/evolution/kafka/journal · high confidence

This release introduces a new \KafkaJournal\ plugin for Akka Persistence, allowing users to store and replay persistent events in Kafka instead of traditional databases. The plugin is configured via \reference.conf\ under \evolutiongaming.kafka-journal.persistence\, supporting both constant-topic and split-topic key strategies, configurable batching, and optional Cassandra eventual consistency. It includes a \Journals\ extension to share the underlying Kafka consumer/producer state with external components, ensuring visibility for actions like purge or mark outside the standard persistence flow.

akka/persistence/src/main · high confidence

The Cassandra module has been rewritten to introduce a new persistence backend for the journal. This change adds explicit configuration for Cassandra read and write consistency levels (defaulting to LOCAL\_QUORUM), enabling users to tune durability and performance. It also introduces automated schema management, including the ability to auto-create keyspaces and tables, and a migration system that applies CQL statements to evolve the schema version over time. The implementation includes new internal components for handling Cassandra sessions, metadata, and synchronization, along with comprehensive tests for the consistency configuration and table creation logic.

cassandra · high confidence

The library now provides a dedicated Circe integration for serializing and deserializing JSON payloads. This adds new codecs for internal types (such as FiniteDuration, SeqNr, and PayloadType) and enables Kafka and Eventual storage read/write operations using io.circe.Json instead of the previous Play JSON format. It also includes utilities for converting between Play JSON and Circe JSON, allowing for interoperability between the two formats during serialization and deserialization.

(repo-wide) · high confidence

This change introduces a new set of conversion utilities in the journal module to handle the serialization and deserialization of Kafka records, including converting between internal Action types and Kafka Producer/Consumer records, as well as metrics tracking for read and write operations. Additionally, it adds adapters in the Pekko persistence module to bridge the application's logging interface with Pekko's LoggingAdapter, enabling structured logging within the journal system.

journal/src/main/scala/com/evolution/kafka/journal/conversions, pekko/persistence/src/main/scala/com/evolution/kafka/journal · high confidence

This change introduces the \replicator\ module, which subscribes to Kafka topics and replicates records into a Cassandra-backed eventual journal. It includes a \ConsumeTopic\ component that ensures strict record ordering and completeness by stalling on errors rather than skipping records, a \Batch\ processor that optimizes write loads on Cassandra by merging sequential appends and deletes, and a \DistributeJob\ service for coordinating work across application nodes using Kafka partition assignments. The module also provides configuration via \ReplicatorConfig\, caching through \CacheOf\, and metrics for purge and replication operations.

replicator/src/main/scala · high confidence

A new reference configuration file has been added for the Pekko persistence module, establishing default settings for the Kafka journal. This includes the fully qualified class name for the journal implementation, Kafka producer and consumer parameters (such as LZ4 compression and idempotence), Cassandra query consistency levels, circuit breaker thresholds, and a dynamic consumer pool configuration for recovery scenarios.

pekko/persistence/src/main/resources · high confidence

A new default configuration file (reference.conf) is introduced for the replicator module, defining baseline settings for Kafka and Cassandra interactions. For Kafka, the consumer is configured to use the CooperativeStickyAssignor to minimize latency during redeployments, with specific buffer sizes, LZ4 compression for producers, and a group ID of 'replicator'. For Cassandra, the client is named 'replicator' and configured with LOCAL\_QUORUM consistency and default idempotence enabled.

replicator/src/main/resources · high confidence

The demo applications (AppendReplicateApp and ReadEventsApp) have been relocated into a dedicated demo module. This change includes the addition of specific configuration files (AppendReplicateApp.conf and logback.xml) to support the demos in their new location, ensuring they retain their previous logging levels and Kafka/Cassandra connection settings.

demo · high confidence

The eventual-cassandra module has been completely rewritten for the 5.0.0 release, introducing a new schema structure that replaces the legacy \pointer\ table with a new \pointer2\ table and adds a \meta\_journal\ table for journal metadata. This change includes new encoding/decoding extensions for \PartitionOffset\, \RecordMetadata\, and other core types, along with new configuration classes (\EventualCassandraConfig\, \SchemaConfig\) and services (\ExpiryService\) to manage journal lifecycle, consistency, and data integrity. The implementation now supports automatic schema creation and migration, and exposes the schema version in metrics.

eventual-cassandra/src/main · high confidence

The eventual journal implementation has been restructured to use a hierarchical API for write operations, replacing the previous flat interface. The new \ReplicatedJournal\ trait acts as a factory for topic-specific \ReplicatedTopicJournal\ instances, which in turn provide partition-specific \ReplicatedPartitionJournal\ resources, finally yielding \ReplicatedKeyJournal\ instances for individual key operations. This change improves resource management and modularity for replicating events from Kafka to eventual storage (Cassandra). The legacy \ReplicatedJournalFlat\ interface is retained only for backward compatibility with existing unit tests.

journal/src/main/scala/com/evolution/kafka/journal/eventual · high confidence

This release introduces version 5.0.0, which migrates the project's package namespace from \com.evolutiongaming\ to \com.evolution\ and renames the Akka persistence module to \kafka-journal-akka-persistence\. It also adds a new \kafka-journal-pekko-persistence\ module to support Pekko, updates the README with a migration guide and new setup instructions, and switches the CI pipeline from Travis CI to GitHub Actions.

(repo-wide) · high confidence

The Pekko persistence plugin has been restructured to use a new \ActorSystemRef\ extension mechanism, making the underlying \Journals\ instance directly accessible to external components via the actor system. This change introduces a new base package (\com.evolution.kafka.journal.pekko.persistence\) and refactors core components like \KafkaJournal\, \JournalAdapter\, and \EventSerializer\ to support this extension-based wiring, while also adding configuration options for JSON codecs and batch sizes.

pekko/persistence/src/main/scala/com/evolution/kafka/journal/pekko/persistence · high confidence

Added benchmarks for journal read worst-case scenarios and replicator batch processing; Added test coverage for journal serialization, actions, and concurrency primitives; Added test coverage for the Kafka journal replicator; Added test suite for Akka Persistence Kafka Journal; Added tests for Cassandra journal schema, configuration, and replication logic; Added tests for Pekko persistence journal components; Refactored Kafka-Journal integration test infrastructure.

This change significantly updates the project's dependency versions and build tooling. Key library upgrades include Kafka clients to 4.3.1, scassandra to 5.7.0, skafka to 21.0.3, and Pekko to 1.7.0, alongside updates to Cats Effect (3.7.1), Logback (1.6.3), and Play JSON (3.0.6). The build system has been upgraded from sbt 1.1.6 to sbt 2.0.8, and plugins have been refreshed, including sbt-scoverage to 2.4.4 and sbt-scalafmt to 2.6.2, with the addition of sbt-version-policy for compatibility checks.

project · high confidence

The project's build infrastructure has been upgraded to sbt 2, which includes significant changes to how dependencies and compiler settings are managed. This update raises the supported Scala versions to 2.13.18 and 3.3.8, replacing the previous 2.12.x support. The migration also introduces stricter binary compatibility checks via sbt-version-policy and updates various internal library dependencies to align with the new Scala versions and build tooling.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 49 → 49 (-0.5)
• Rubric changed (rubric-2026.08.17 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 100 → 80 (-20.3)
• Architecture 100 → 94 (-6.5)
• Maturity 48 → 46 (-1.6)
• Readiness 61 → 46 (-14.9)
• Security 33 → 42 (+8.4)

• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — no supported dependency manifest was read
• Further sole-owners (lower concentration)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• …and 32 more

• Change coupling: PlayJsonHelper.scala ↔ ScodecHelper.scala (core/src/main/scala/com/evolution/kafka/journal/util/PlayJsonHelper.scala)
• ConsumeActionRecords.apply (cognitive 22) (journal/src/main/scala/com/evolution/kafka/journal/ConsumeActionRecords.scala)
• ConsumerPool.make (cognitive 66) (journal/src/main/scala/com/evolution/kafka/journal/ConsumerPool.scala)
• CreateKeyspace.apply (cognitive 20) (cassandra/src/main/scala/com/evolution/kafka/journal/cassandra/CreateKeyspace.scala)
• CreateTables.apply (cognitive 20) (cassandra/src/main/scala/com/evolution/kafka/journal/cassandra/CreateTables.scala)
• Damper.of (cognitive 47) (journal/src/main/scala/com/evolution/kafka/journal/Damper.scala)
• Damper.of (cyclomatic 26) (journal/src/main/scala/com/evolution/kafka/journal/Damper.scala)
• DistributeJob.apply (cognitive 368) (replicator/src/main/scala/com/evolution/kafka/journal/replicator/DistributeJob.scala)
• DistributeJob.apply (cyclomatic 65) (replicator/src/main/scala/com/evolution/kafka/journal/replicator/DistributeJob.scala)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (10 lines × 2) (akka/persistence/src/main/scala/com/evolution/kafka/journal/akka/persistence/EventSerializer.scala)
• Duplicated block (10 lines × 2) (akka/persistence/src/main/scala/com/evolution/kafka/journal/akka/persistence/KafkaJournal.scala)
• Duplicated block (10 lines × 2) (journal/src/main/scala/com/evolution/kafka/journal/eventual/ReplicatedKeyJournal.scala)
• Duplicated block (10 lines × 3) (core/src/main/scala/com/evolution/kafka/journal/Settings.scala)
• Duplicated block (10–12 lines × 2) (replicator/src/main/scala/com/evolution/kafka/journal/replicator/ReplicateRecords.scala)
• Duplicated block (11 lines × 2) (akka/persistence/src/main/scala/com/evolution/kafka/journal/akka/persistence/Batching.scala)
• Duplicated block (11 lines × 2) (akka/persistence/src/main/scala/com/evolution/kafka/journal/akka/persistence/ToKey.scala)
• Duplicated block (12 lines × 3) (journal/src/main/scala/com/evolution/kafka/journal/eventual/EventualJournal.scala)
• Duplicated block (13 lines × 2) (akka/persistence/src/main/scala/com/evolution/kafka/journal/akka/persistence/JournalAdapter.scala)
• …and 163 more

• 59 commits — 55 feature/other, 4 fixes

• project/Dependencies.scala — 19 commits
• (root) — 11 commits
• .github/workflows — 8 commits
• replicator/src — 5 commits
• akka/integration-tests — 4 commits
• project/build.properties — 4 commits
• akka/tests — 3 commits
• eventual-cassandra/src — 2 commits
• benchmark/src — 1 commit
• demo/src — 1 commit
• project/plugins.sbt — 1 commit

• fix: Integration test code style fix
• fix: Revert "Bound TopicReplicator key replication concurrency" (#936) (#956)
• fix: Revert "Cover TopicReplicator cancellation semantics with tests (#960)" (#990)
• fix: Revert "temporary disable library version compatibility checks for v10.1.0 (play-json organization name change)"
• change: Adopt shared Scala CI workflow (#967)
• change: Bound TopicReplicator key replication concurrency (#936)
• change: Clean-up missed redundant Akka/Pekko usage in int tests
• change: Cover TopicReplicator cancellation semantics with tests (#960)
• change: Cover worst-case journal read with tests (#938)
• change: Do not cancel processing of the whole records batch if one of keys failed (#957)
• change: Integration test refactoring
• change: Integration test refactoring
• change: Integration test refactoring (#992)
• change: Integration tests - 3x speed up, maintenance refactoring (#994)
• change: Migrate build to sbt 2 (#919)
• change: Move demo apps into a separate module (#991)
• change: Set up single status check for all CI action sub-jobs (#989)
• change: Update cats-helper to 3.13.1 (#983)
• change: Update log4j-over-slf4j, slf4j-api to 2.0.19 (#997)
• change: Update logback-classic, logback-core to 1.6.0 (#951)
• …and 39 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

evolution-gaming/kafka-journal was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.