gazette/core

Gazette is a distributed, append-only journaling system designed for building reliable, stateful data pipelines. It provides a broker for high-throughput, fault-tolerant journal storage with support for multiple cloud backends and fine-grained authorization, alongside a consumer framework that ensures exactly-once processing semantics through crash-recovery logs. The system includes comprehensive tooling for cluster management, hierarchical configuration, and integration testing to facilitate robust deployment on Kubernetes.

39 changes.

This period focused on the comprehensive transition from the legacy V1 codebase to a modular V2 architecture, introducing interface-driven consumer frameworks, hierarchical specification management, and robust fault-tolerance mechanisms. The project expanded its operational capabilities with fine-grained authorization, TLS support, and multi-backend storage options, while establishing standardized deployment patterns through Kustomize and Kubernetes manifests.

4 changes.

This period focused on decoupling etcd-dependent protocol extensions into a separate package to reduce client dependencies and introducing a modular cloud storage backend abstraction for journal fragments. Significant CLI capabilities were added to gazctl for comprehensive journal and shard management, while authentication was expanded to support pre-shared keys and tokens.

The auth package now supports two new authentication mechanisms: KeyedAuth, which uses symmetric pre-shared secrets (base64-encoded) to sign and verify JWTs using HS256/HS384, and BearerAuth, which uses a static pre-shared token. KeyedAuth allows multiple keys for rotation and includes a special 'AA==' key value to temporarily allow requests without an authorization header during rollout. Both implementations integrate with the broker's protocol Authorizer and Verifier interfaces, enforcing capability checks (LIST, APPLY, READ, APPEND, REPLICATE) and claim verification.

auth · high confidence

Introduces the stream-sum example application, which serves as both a demonstration of Gazette's processing capabilities and an active integration test. The application consists of a 'chunker' job that generates random data streams and a 'summer' consumer that aggregates these chunks to compute running sums, verifying exactly-once semantics and end-to-end latency bounds. It supports configurable storage backends (RocksDB or SQLite) and includes comprehensive unit and end-to-end tests to validate processing guarantees.

examples/stream-sum · high confidence

The stream-sum example now includes a chunker application that generates pseudo-random stream chunks and verifies their sums against a final journal, alongside a summer consumer application that processes these streams. The chunker component initializes diagnostics, registers a gRPC dispatcher, and serves diagnostics on port 8080, while the summer component runs as a standard consumer using the mainboilerplate framework.

examples/stream-sum/chunker · high confidence

A new example application has been added at examples/word-count/counter that demonstrates how to run a word-count consumer. The entry point initializes and runs the word\_count.Counter consumer using the runconsumer framework, relying on the updated go.gazette.dev/core import paths.

examples/word-count/counter · high confidence

The broker now exposes an HTTP gateway that maps GET and HEAD requests to journal reads and PUT requests to journal appends. This allows clients to interact with Gazette brokers using standard HTTP methods, with the gateway translating these into equivalent gRPC Read and Append RPCs. The implementation handles request parsing for journal names, offsets, and block modes, manages authorization headers, and returns appropriate HTTP status codes and headers (such as Content-Range and X-Write-Head) based on the broker's response.

_broker/http\gateway · high confidence

This change establishes the foundational project structure for Gazette, introducing a Makefile-based build system that generates protobuf sources and compiles Go binaries, alongside Dockerfiles for the broker and example images. It adds configuration for Read the Docs to build the Sphinx-based documentation, a CHANGELOG documenting releases from v0.83.2 through v0.89.0, and standard repository files including an MIT license, .gitignore, and contributor guidelines. The commit also removes legacy Go source files (config.go, journal.go, service.go, spool.go, to\_refactor.go) that were part of the initial V1 codebase, signaling a shift to the new V2 architecture.

(repo-wide) · high confidence

The message package now includes a Framing registry that allows applications to register and use custom message serialization formats. This change adds built-in support for CSV framing (via the \CSVFrameable\ interface and \labels.ContentType\_CSV\) and fixed-length Protobuf framing (via the \ProtoFrameable\ interface and \labels.ContentType\_ProtoFixed\), alongside the existing JSON Lines framing. Users can now serialize and deserialize messages using these new formats by ensuring their message types implement the required interfaces, enabling more flexible data exchange patterns in Gazette journals.

message · high confidence

Added a new \store-sqlite\ package that implements the \consumer.Store\ interface using an embedded SQLite database. This store hooks into SQLite's VFS layer to record all file mutations into a recovery log, ensuring fault tolerance and crash recovery. It leverages RocksDB to manage SQLite page data, allowing for efficient compaction and tighter recovery horizons. The implementation supports multiple attached databases, batch atomic writes, and configurable journal modes (WAL or TRUNCATE) to optimize performance and durability. Comprehensive tests verify SQL logic, multi-database transactions, and recovery after simulated crashes.

consumer/store-sqlite · high confidence

The consumer/recoverylog package has been introduced to provide a finite state machine for recording and replaying observed filesystem operations into a Gazette journal. This change adds core components including an FSM to validate operation sequence numbers and checksums, a Recorder to observe and write filesystem changes (creates, links, writes, unlinks) to a journal, and a Player to replay these operations back onto the local filesystem. The package also includes protocol buffer definitions for RecordedOps, Segments, and FSMHints, along with an afero adapter to integrate the recovery log with standard filesystem interfaces.

consumer/recoverylog · high confidence

A new command-line interface, gazctl, is introduced to allow users to interact with Gazette brokers and consumer applications. The tool provides subcommands for managing broker journals and consumer shards, as well as a specific command to attach UUIDs to consumers. Configuration can be managed via a local ini file or the standard config directory, and users can inspect their current settings using the print-config subcommand.

cmd/gazctl · high confidence

The broker now includes a new \journalspace\ package that maps collections of JournalSpecs into a hierarchical tree structure, enabling operators to manage journal configurations via a simplified, shared-parent model. This change introduces logic to hoist common specification fields (such as replication and flags) to parent directory nodes and push them down to children, reducing configuration redundancy. It also provides utilities to patch, validate, and flatten these hierarchical trees back into explicit JournalSpecs for storage in Etcd, streamlining the tooling experience for cluster operators.

broker/journalspace · high confidence

The new mainboilerplate package provides a unified foundation for Gazette services, handling command-line argument parsing, structured logging, and optional private diagnostics endpoints. It enables secure client connections with configurable TLS and supports both pre-shared keyed authentication and bearer tokens for gRPC communication. Additionally, it standardizes service registration with etcd, allowing for Unix domain socket binding and configurable gRPC message size limits.

mainboilerplate · high confidence

The broker/protocol package now includes a new authorization framework that enables fine-grained access control for broker operations. Clients can attach JWT-based claims to their contexts, which are automatically verified by the server to enforce specific capabilities (such as List, Read, Append, or Replicate) against targeted resources using label selectors. Additionally, the package introduces a custom gRPC dispatcher balancer that allows clients to route requests intelligently based on journal topology, preferring local zone members and handling connection states to optimize latency and network costs.

broker/protocol · high confidence

The labels package now defines a set of well-known label names and values for Gazette journals, adopting Kubernetes-style labels under the \app.gazette.dev\ namespace. This includes constants for content types (CSV, JSON Lines, Protobuf Fixed, and Recovery Log), message types, and operational metadata such as managed-by, instance, and region. The package also provides a map of single-value labels to enforce that certain attributes like content type and region are unique within a specification.

labels · high confidence

The broker now uses a new \broker/stores\ package to manage fragment persistence across S3, GCS, Azure, and local filesystems through a unified \Store\ interface. This change introduces \ActiveStore\ wrappers that provide continuous health monitoring with exponential backoff, Prometheus metrics for operation tracking, and mark-and-sweep garbage collection for unused stores. It also adds an opt-in delete-permission probe to \FragmentStoreHealth\ to verify write/delete capabilities, and improves GCS error handling to use \errors.Is\ for robust authentication error detection.

broker/stores · high confidence

This change introduces a comprehensive set of Kustomize base manifests for deploying the Gazette ecosystem on Kubernetes. It adds base definitions for core infrastructure components, including a 3-replica Broker Deployment, a generic Consumer Deployment, an Etcd StatefulSet with custom bootstrap/shutdown scripts, and optional Minio and Postgres bases. Additionally, it provides complete example deployments for 'bike-share', 'stream-sum', and 'word-count' scenarios, which wire these components together using Jobs for data ingestion and shard specification management.

kustomize/bases · high confidence

The consumer/store-rocksdb package introduces a new implementation of the consumer.Store interface backed by an embedded RocksDB instance. This store integrates with the recovery log to record all mutating file operations (such as appends, syncs, and deletions) via a custom C++ environment hook, ensuring that database state changes are faithfully captured in the journal for reliable recovery and hand-off between replicas. To improve performance, the package includes an ArenaIterator that batches key-value retrieval into a memory arena, significantly reducing the overhead of CGO calls during iteration. The implementation also handles the migration of legacy offset storage formats into the new checkpoint structure.

consumer/store-rocksdb · high confidence

Added a new example application in \examples/bike-share\ that demonstrates a Gazette consumer processing streaming Citi Bike data. The application indexes recent rides in a PostgreSQL or embedded SQLite database, detects completed graph cycles (rides of length \>= 10 that start and end at the same station without relocation), and exposes an HTTP API at \/api/bikes\ to query the history of specific bikes.

examples/bike-share · high confidence

The \broker/client\ package has been introduced to provide a comprehensive Go client for interacting with the Gazette broker's gRPC Journal service. This module consolidates core I/O primitives, including \Reader\ and \RetryReader\ for resilient journal reads, and \Appender\ and \AppendService\ for high-throughput, batched, and asynchronous writes. It also adds \WatchedList\ for dynamic journal discovery and a new \FragmentStoreHealth\ RPC to monitor storage backend status and probe delete permissions.

broker/client · high confidence

The new \brokertest\ package provides utilities for testing components that require a live Gazette broker, including an embedded \Broker\ struct that can be instantiated in tests with configurable keys and zones. It includes helper functions like \NewBroker\, \CreateJournals\, and \WaitForConsistency\ to simplify setting up test environments, and contains tests for simple and replicated read/write operations, reassignment scenarios, and graceful stop timeouts with proxied HTTP requests.

brokertest · high confidence

The task package now includes a new Group type that allows multiple functions to be executed concurrently. The first task to return an error cancels the entire group and causes Wait to return that error. Additionally, task completion and errors are now logged with descriptive fields for easier debugging.

task · high confidence

The consumer/protocol package has been introduced to centralize the datamodel, validation, and gRPC APIs for Gazette consumers. This change adds fine-grained authorization support via AuthShardClient and VerifiedAuthServer, which attach and verify bearer tokens for shard operations. It also introduces a new Checkpoint message with helper functions to build and flatten producer states and read-through offsets, ensuring wire-compatibility with previous implementations while avoiding non-UTF-8 string violations. Additionally, the package provides comprehensive validation logic for all RPC requests and responses (Stat, List, Apply, Unassign) and defines the ShardSpec structure with fields for sources, recovery logs, transaction durations, and hot standbys, along with generated protobuf code and gRPC-Gateway REST handlers for Shard.List and Shard.Stat.

consumer/protocol · high confidence

Added new example files to the \examples\ directory: \batch\_download.py\, a Python script for streaming and batching data downloads with incremental offset tracking, and \generate-zonemap-gke.sh\, a shell script that generates a Kubernetes ConfigMap for mapping GKE node names to availability zones.

examples · high confidence

A new \broker/protocol/ext\ package has been introduced to provide extensions for the broker protocol that rely on the keyspace and allocator (which depend on etcd). This separation allows client users to avoid pulling in the etcd dependency. The package includes utilities for converting etcd response headers to protocol headers, generating unrouted headers from allocator state, initializing routes from allocator assignments, and attaching endpoints to routes based on keyspace data.

broker/protocol/ext · high confidence

The broker now includes a dedicated \broker/fragment\ package that manages the lifecycle of journal fragments, handling the mapping of journal offsets to protocol fragments and managing local and remote journal content. This package introduces a unified storage abstraction supporting file, S3, GCS, and Azure Blob storage backends, allowing fragments to be persisted to various cloud providers. It implements a \CoverSet\ for efficient offset-to-fragment mapping, an \Index\ for querying fragment availability with support for offset jumping over missing data, and a \Persister\ for asynchronous background persistence of completed spools. The \Spool\ type handles transactional fragment construction during replication, including incremental compression and checksumming, while the \Index\ supports optional \BeginModTime\ constraints for read requests to ensure data consistency.

broker/fragment · high confidence

The \gazctl\ CLI now includes a comprehensive set of new subcommands for managing journals and shards, including \attach-uuids\ for generating RFC 4122 v1 UUIDs for input records, \append\ for writing content to journals with various framing and mapping strategies, \apply\ and \edit\ for creating, updating, and deleting journal specifications via an interactive editor, \list\ and \fragments\ for inspecting journal status and persisted data, \prune\ for deleting old fragments based on retention policies, \read\ for streaming journal contents, and \reset-head\ for disaster recovery by resetting append offsets. These commands are implemented in the \cmd/gazctl/gazctlcmd\ package, which also introduces a reusable \editor\ package to handle interactive specification editing with retry loops and validation.

cmd/gazctl/gazctlcmd · high confidence

The consumertest package now provides a lightweight, embedded consumer runtime suitable for in-process unit testing of Gazette consumer applications. It includes a Consumer struct that wraps a loopback server, consumer service, and task group, along with helper functions like NewConsumer, CreateShards, and WaitForShards to simplify test setup, shard creation, and synchronization. The package also supports testing consumer handoff and hot standby scenarios, allowing developers to verify consumer behavior in a controlled, in-memory environment without needing external infrastructure.

consumertest · high confidence

The new \keyspace\ package provides a local \KeySpace\ that mirrors a decoded portion of an Etcd key/value space, keeping it in sync via long-lived Watch operations. It supports user-provided decoders, validates keys and values, and handles decoding errors gracefully by logging them and skipping invalid entries while maintaining consistency. The implementation includes efficient range and point queries, observer hooks for derived state updates, and robust watch retry logic with leader requirement checks to handle Etcd partition scenarios.

keyspace · high confidence

The mainboilerplate/runconsumer package now provides a structured entry point for building Gazette consumers, exposing a Cmd.Execute method that handles configuration parsing, service bootstrap, and lifecycle management. This change introduces support for configurable Etcd watch delays, fine-grained authorization via signed keys, optional TLS for server and peer connections, and the ability to wrap the net.Listener for custom handling. It also integrates diagnostics configuration and supports file-based broker roots, providing a standardized foundation for consumer applications to extend.

mainboilerplate/runconsumer · high confidence

The consumer/shardspace package introduces a new mechanism for managing collections of ShardSpecs in a semi-hierarchical structure, primarily to simplify tooling interactions with Etcd. It provides a Set abstraction that allows common ShardSpec configuration to be hoisted into a shared parent and pushed down to individual shards, enabling users to apply edits in a simplified space before flattening changes back to the cluster. The package includes utilities for patching individual shards, marking unpatched shards for deletion, and performing round-trip serialization to YAML, while also reflecting the underlying protocol support for the DisableWaitForAck field.

consumer/shardspace · high confidence

The examples/word-count directory now includes a complete example application that demonstrates how to build a service for publishing text and querying running counts of NGrams. The implementation includes a gRPC service (NGram) with Publish and Query RPCs, a consumer application that aggregates NGram counts using a RocksDB-backed store, and a command-line client (wordcountctl) for interacting with the service. The example also provides unit tests validating the publish and query functionality against a test broker and consumer environment.

examples/word-count · high confidence

The \gazetted\ binary entry point (\main.go\) has been removed. This file previously contained the service initialization logic, including configuration via etcd, logging setup with varz hooks, and a subscription to the config journal path. Its deletion indicates that the gazetted service is no longer built or deployed as a standalone component in this location.

gazetted · high confidence

The allocator now uses a sparse maximum flow solver (push/relabel algorithm) to compute optimal item-to-member assignments. This change improves performance and scalability for large clusters by modeling the allocation problem as a maximum flow network and solving it efficiently, while maintaining the same leader-follower coordination and consistency guarantees.

allocator · high confidence

The broker's append handling has been restructured from a linear flow into an explicit \appendFSM\ state machine, which manages resolution, pipeline acquisition, synchronization, and content streaming as distinct states. This change introduces granular flow control for append operations, allowing clients to bank unused rate credit and enforcing both minimum and maximum data rates to prevent slow clients from blocking the pipeline. Additionally, the broker now supports fine-grained authorization via journal registers, enabling clients to atomically check, add, or remove labels during appends, and includes a new \FragmentStoreHealth\ RPC to monitor storage backend health and verify delete permissions.

broker · high confidence

The broker test stub has been updated to use the new server loopback mechanism and the testify/require assertion library. This change ensures that test environments correctly simulate broker RPCs (Replicate, Read, Append, List) via channels, while also improving test reliability by enforcing stricter assertion requirements and properly handling server shutdown via BoundedGracefulStop.

broker/teststub · high confidence

The consumer package has been restructured to expose a clean, interface-based API for building distributed, stateful journal consumers. Users now interact with the framework through defined contracts: the \Application\ interface for event-driven logic, the \Shard\ interface for processing context and progress tracking, and the \Store\ interface for durable, transactional state persistence. This change introduces a \Resolver\ to manage shard assignments and a \Service\ as the top-level runtime, enabling better modularity, testability, and support for exactly-once semantics through integrated recovery logs and hint management.

consumer · high confidence

The Gazette broker command-line interface has been updated to support secure and controlled operation. Users can now enable fine-grained authorization by providing base64-encoded keys via the --auth-keys flag, and secure communication using optional TLS certificates for both server and peer connections. Additionally, the broker introduces configurable flow control limits, including maximum append rates, minimum stream rates, and maximum replication factors, as well as a new --file-only mode to restrict storage to local file systems. The entry point also exposes a --watch-delay flag to tune Etcd event processing and supports optional private diagnostics ports.

cmd/gazette · high confidence

The etcdtest package now ensures that the embedded etcd server is properly terminated when the test process dies, preventing hanging tests on Linux by setting a death signal (SIGTERM) via Pdeathsig. It also standardizes logging configuration by using the ETCD\_LOG\_LEVEL environment variable for broader compatibility with older etcd binaries and introduces a cleanup function to remove key/value fixtures after each test, ensuring a clean state for subsequent tests.

etcdtest · high confidence

The server now allows binding to Unix domain sockets in addition to TCP, enabling local-only communication without TLS overhead. Users can configure the maximum accepted gRPC message size via the new \maxGRPCRecvSize\ parameter. When TLS is enabled, the server advertises both "http/1.1" and "h2" via ALPN to support mixed HTTP/1 and HTTP/2 clients, while gRPC loopback connections use aggressive back-off for faster cluster recovery. Additionally, the server exposes its underlying CMux and listeners, allowing consumers to wrap the net.Listener for custom handling or add additional protocol listeners.

server · high confidence

The broker's codec layer now supports Gzip multi-member compression, allowing multiple gzip members to be concatenated into a single stream for more efficient batching of writes. Additionally, Zstandard (zstd) compression is now available as an optional codec, enabled at compile time unless the 'nozstd' build tag is specified.

broker/codecs · high confidence

Added integration tests for fault tolerance and network partitions; Expanded kustomize test infrastructure for integration and crash testing.

The project's Go dependencies (go.mod) and Python documentation requirements (docs/requirements.txt) have been updated to their latest versions. Key Go upgrades include gRPC to v1.83.1, etcd client to v3.6.5, and Azure SDK packages. Documentation tools such as Sphinx, Jinja2, and Babel have also been updated to ensure compatibility and security for the documentation build process.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 53 → 59 (+6.2)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 73 → 82 (+9.7)
• Architecture 100 → 99 (-0.2)
• Maturity 59 → 59 (-0.6)
• Readiness 52 → 64 (+12.0)
• Security 44 → 51 (+6.6)

• Application.ServeBikeHistory (cognitive 16) (examples/bike-share/api.go)
• Application.ServeBikeHistory (cyclomatic 16) (examples/bike-share/api.go)
• Change coupling: doc.go ↔ stores.go (broker/fragment/doc.go)
• Counter.Publish (cognitive 16) (examples/word-count/word_count.go)
• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — dependency manifest found but not parsed for hygiene
• Duplicated block (10 lines × 2) (broker/list_apply_api.go)
• Duplicated block (10 lines × 2) (broker/list_apply_api.go)
• Duplicated block (10 lines × 2) (cmd/gazctl/gazctlcmd/journals_reset_head.go)
• Duplicated block (10 lines × 2) (consumer/protocol/auth.go)
• Duplicated block (11 lines × 2) (broker/protocol/auth.go)
• Duplicated block (12 lines × 2) (mainboilerplate/client.go)
• Duplicated block (15 lines × 2) (broker/teststub/broker.go)
• Duplicated block (5 lines × 2) (broker/client/append_service_win.go)
• Duplicated block (6 lines × 2) (allocator/alloc_state.go)
• Duplicated block (6 lines × 2) (broker/client/list.go)
• Duplicated block (6 lines × 2) (broker/stores/azure/account.go)
• Duplicated block (7 lines × 2) (broker/list_apply_api.go)
• Duplicated block (7 lines × 3) (broker/append_api.go)
• Duplicated block (8 lines × 2) (broker/replica.go)
• …and 55 more

• ClassTooLong: appendFSM (broker/append_fsm.go)
• Concentrated knowledge decay
• Dependency pinned to a stale untagged commit: gopkg.in/check.v1
• Deprecated module: github.com/aws/aws-sdk-go
• Deprecated module: github.com/golang/protobuf
• Documentation: no installation or build instructions (README.rst)
• Duplicated block (10 lines × 4) (broker/stores/azure/common.go)
• Duplicated block (11 lines × 2) (broker/protocol/fragment_store.go)
• Duplicated block (12 lines × 2) (consumer/protocol/auth.go)
• Duplicated block (13 lines × 2) (broker/protocol/auth.go)
• Duplicated block (13 lines × 2) (cmd/gazctl/gazctlcmd/journals_reset_head.go)
• Duplicated block (13–14 lines × 2) (broker/list_apply_api.go)
• Duplicated block (16 lines × 2) (broker/teststub/broker.go)
• Duplicated block (17 lines × 2) (mainboilerplate/client.go)
• Duplicated block (5 lines × 2) (broker/client/list.go)
• Duplicated block (5 lines × 2) (broker/key_space.go)
• Duplicated block (6 lines × 2) (allocator/alloc_state.go)
• Duplicated block (6 lines × 2) (broker/stores/azure/account.go)
• Duplicated block (7 lines × 2) (broker/client/append_service_win.go)
• Duplicated block (7 lines × 2) (broker/protocol/endpoint.go)
• …and 394 more

• 5 commits — 4 feature/other, 1 fixes

• (root) — 1 commit
• broker/client — 1 commit
• cmd/gazctl — 1 commit
• consumer/store-rocksdb — 1 commit
• docs/architecture-design-goals-and-nongoals.rst — 1 commit

• fix: Revert "client: retry all append errors when CheckRegisters is set"
• change: Fixed typo
• change: build(deps): bump google.golang.org/grpc from 1.79.3 to 1.83.1
• change: gazctl shards prune: retry transient fragment removal errors
• change: store-rocksdb: return barrier errors from the sync hook

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

gazette/core was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.