apache/iggy

This system is a distributed, high-performance messaging platform designed for low-latency data ingestion and retrieval. It manages streams, topics, and partitions with durable storage and role-based access control, supporting multiple transport protocols and offering SDKs for various programming languages. The platform facilitates integration with external systems through a connector framework and provides deployment tools including a web UI and Helm charts.

116 changes.

The project migrated to the VSR consensus protocol and replaced legacy UDP transport with robust TCP, QUIC, HTTP, and WebSocket implementations across the Rust SDK and server. This involved restructuring the codebase with a new binary wire protocol, a connectors runtime, and modernized tooling, while expanding SDK support with sharded connection caching and comprehensive examples for TypeScript and Java.

55 changes.

This period focused on building the foundational infrastructure for the new server architecture, including a centralized configuration system, a new VSR consensus wire protocol, and a multi-shard routing layer. It also involved extensive SDK updates to support the new binary protocol, the introduction of RBAC permissions, and the expansion of the connector ecosystem with numerous sink and source integrations.

Introduces a new external connector that enables Apache Pinot to consume data directly from Iggy streams. This change adds the necessary Java components to bridge the two systems, including an \IggyConsumerFactory\ to initialize the connection, an \IggyJsonMessageDecoder\ to parse incoming JSON payloads into Pinot's \GenericRow\ format, and configuration classes (\IggyStreamConfig\) to manage connection details such as host, port, credentials, and stream/topic identifiers. The connector is registered via the standard Pinot SPI service loader and plugin properties, allowing users to configure Pinot tables to read from Iggy sources.

foreign/java/external-processors/iggy-connector-pinot · high confidence

New C\# example applications are provided to demonstrate how to use the Iggy SDK for common messaging patterns. The Basic examples show simple TCP producer and consumer clients with configurable settings. The Getting Started examples provide a quick-start guide for connecting, creating streams/topics, and sending/receiving messages. The Message Envelope example demonstrates serializing and deserializing structured messages (like OrderCreated, OrderConfirmed, OrderRejected) using JSON envelopes. The Message Headers example shows how to attach and read custom headers on messages. Shared helper code handles stream/topic creation and message generation.

examples/csharp · high confidence

New example projects are now available for C\#, Go, and Java, providing comprehensive samples for basic operations, message headers, message envelopes, and security configurations. The C\# examples include a solution file with projects for getting started, basic usage, message headers, message envelopes, and TCP/TLS security, along with an .editorconfig for consistent code formatting. The Go examples demonstrate getting-started and TLS-enabled producer/consumer patterns using the VSR wire protocol. The Java examples showcase basic operations, message headers, message envelopes, multi-tenant architecture, high-volume data generation, stream builder, and async client patterns, supported by a Gradle wrapper and .gitignore. Each language directory includes a README with instructions for running the examples against a local Iggy server.

examples · high confidence

A new sink connector is available to write Iggy stream messages to a Meilisearch index. It supports configurable indexing behaviors (replace or update), automatic index creation, and optional inclusion of Iggy provenance metadata in the indexed documents. The connector handles various payload formats (JSON, text, base64) and provides robust retry logic for transient errors, with options to wait for indexing tasks to complete or operate in a fire-and-forget mode.

_core/connectors/sinks/meilisearch\_sink, core/connectors/sinks/rabbitmq\sink · high confidence

A new Stdout Sink connector is now available, allowing users to print consumed message data to the standard output. This is primarily useful for debugging or testing connector pipelines. The sink includes a configuration option, \print\_payload\, which defaults to \false\; when enabled, it logs the full message payload alongside metadata such as stream, topic, partition, and offset.

_core/connectors/sinks/stdout\sink · high confidence

A new \data-seeder\ utility has been added to \core/tools\ to automatically provision and populate test environments. When executed, it connects to an Iggy instance, creates three streams (\prod\, \test\, \dev\), and populates each with specific topics (\orders\, \users\, \notifications\, \payments\, \deliveries\) configured with varying partition counts. It then generates and sends randomized message batches, including optional user headers with mixed types (strings, booleans, integers), to simulate realistic traffic patterns for development and testing.

core/tools · high confidence

The benchmarking tool now supports a high-level consumer API alongside the existing low-level implementation. This change introduces a \TypedBenchmarkConsumer\ that allows users to select between high-level and low-level client implementations via a configuration flag, enabling more abstract and potentially easier-to-use benchmarking workflows while maintaining backward compatibility with the previous low-level approach.

core/bench/src/actors/consumer · high confidence

The benchmark tool now includes a high-level consumer client implementation alongside the existing low-level one. This new client, located in \core/bench/src/actors/consumer/client/high\_level.rs\, utilizes the Iggy SDK's high-level consumer APIs (such as \consumer\_group\ and \IggyConsumer\) to handle message consumption, offset storage, and batch metrics collection. It is exposed via a new \ConsumerClient\ trait in \interface.rs\, allowing users to select between high-level and low-level consumption strategies in their benchmarks.

core/bench/src/actors/consumer/client, core/bench/src/actors/producer/client · high confidence

The benchmarking tool now includes a new producing-consumer actor that runs a producer and consumer concurrently within a single benchmark run. This actor supports both low-level and high-level client APIs, allowing users to choose the appropriate level of abstraction via a configuration flag. The implementation includes a typed wrapper to select the client type, a core benchmark runner that handles warmup, rate limiting, and finish conditions, and integration with the existing metrics and reporting infrastructure.

_core/bench/src/actors/producing\consumer · high confidence

The core/certs directory now includes pre-generated self-signed certificates (iggy\_ca\_cert.pem, iggy\_cert.pem, and iggy\_key.pem) for the 'localhost' domain. This provides a ready-to-use TLS configuration for local development and testing, ensuring that self-signed certificate generation is consistent and available out-of-the-box without requiring manual setup.

core/certs · high confidence

A new \IggyBenchRunner\ CLI tool is now available in \core/bench/runner\ to automate local performance testing. It allows users to specify a local Iggy repository, a starting git reference, and a commit count; the tool then iterates through those commits, builds the \iggy-bench\ binary, runs the benchmarks, and copies the resulting \performance\_results\ to a user-defined output directory.

core/bench/runner · high confidence

The benchmarking module now includes a new \TimeSeriesProcessor\ trait and a concrete \MovingAverageProcessor\ implementation within the analytics time-series pipeline. This processor applies a sliding-window moving average to time series data points, caching the window sum for efficiency and rounding results to three decimal places, allowing benchmark reports to smooth out noise in performance metrics.

_core/bench/src/analytics/time\series/processors · high confidence

The server-ng implementation is now the default Apache Iggy server, replacing the legacy server crate. This change introduces a new command-line interface with \--fresh\ and \--with-default-root-credentials\ flags, an ASCII art startup banner, and a new \GetClusterMetadata\ command that exposes cluster topology and node roles. The server also writes a \current\_config.toml\ file at runtime with bound ports, enriches consumer group join/leave requests with client IDs for replication, and enforces namespace caps to prevent aliasing.

core/server/src · high confidence

The benchmark reporting module in \core/bench/src/analytics/metrics\ has been expanded to provide deeper visibility into performance characteristics. Individual and group metrics now compute and expose a latency distribution, which includes a histogram of latency samples and fitted log-normal parameters (mu and sigma). This data enables users to visualize the shape of latency tails in benchmark dashboards, moving beyond simple percentile summaries to understand the full distribution of response times.

core/bench/src/analytics/metrics · high confidence

The benchmarking tool's producer actor now allows users to choose between the high-level and low-level Iggy client APIs via a \use\_high\_level\_api\ flag. This is implemented through a new \TypedBenchmarkProducer\ enum that wraps the existing \BenchmarkProducer\ with either \HighLevelProducerClient\ or \LowLevelProducerClient\, enabling direct comparison of performance characteristics between the two API layers.

core/bench/src/actors/producer · high confidence

The CLI argument definitions in \core/cli/src/args\ have been reorganized from a single monolithic file into separate modules for each resource (e.g., \client.rs\, \cluster.rs\, \consumer\_group.rs\, \context.rs\, \message.rs\, \partition.rs\, \personal\_access\_token.rs\, \segment.rs\, \session.rs\, \stream.rs\, \system.rs\, \topic.rs\, \user.rs\). This change introduces a unified \ListMode\ enum for consistent table/list output formatting across commands, adds new CLI commands for client management, cluster metadata, consumer groups, contexts, and sessions, and standardizes identifier parsing using \clap\'s \Identifier\ type. The \common.rs\ module now provides shared parsing logic for key-value pairs and list modes, while \mod.rs\ wires these subcommands into the main CLI interface.

core/cli/src/args · high confidence

The core/cli location now includes a build script (build.rs) that configures compile-time feature flags for keyring support, distinguishing between systems using the D-Bus Secret Service backend (Linux and BSDs) and those with native keyring backends (macOS, Windows, and others). Additionally, a README.md file has been added to document the Apache Iggy CLI, providing installation instructions via Cargo and links to project resources.

core/cli · high confidence

The core/configs/src module has been introduced to centralize configuration management, exposing a structured set of server settings including cluster, message bus, metadata, partition, QUIC, server, sharding, TCP, and WebSocket configurations. This change consolidates previously scattered config types into a single crate, providing a unified interface for environment variable mappings and file-based configuration providers, which simplifies how users define and validate server parameters.

core/configs/src · high confidence

The connectors runtime now supports storing source checkpoints on a remote HTTP state server, allowing state to be shared across runtime instances and ensuring durability depends on the backing store rather than local disk. It also adds an HTTP configuration provider, enabling the runtime to fetch and manage connector configurations via a remote REST API instead of relying solely on local files. These changes are accompanied by a dedicated Dockerfile for building the runtime binary and a new HTTP API for managing connector configurations and inspecting runtime state.

core/connectors/runtime · high confidence

The simulator now uses a deterministic, seed-based workload generator to drive test runs. This introduces a structured set of server commands (actions) such as creating streams, sending messages, and managing users, along with a shadow state model that predicts server-side entity mutations. An auditor tracks in-flight requests and reply outcomes to ensure correctness, while invariants and a quiesce-time oracle verify that replicas converge and agree on committed state. This allows test scenarios to be reproduced bit-for-bit by reusing the same seed.

core/simulator/src/workload · high confidence

The simulator now includes a deterministic execution engine and CLI tools for testing. A new \simulator-ui\ binary provides a scripted demo of partition operations, client registration, and crash recovery. A \workload-fuzz\ binary drives a seeded, deterministic fuzzer that injects replica crashes, network faults, and WAL limits to validate invariants. The underlying simulator core now features a deterministic single-threaded executor with a virtual clock, an in-memory journal, and a message bus that stages outbound traffic into per-replica outboxes for controlled delivery.

core/simulator/src · high confidence

The dashboard server now includes a background poller that periodically checks the Apache Iggy repository's \performance.yml\ workflow for successful runs. When new runs are detected, the server authenticates via the \GITHUB\_TOKEN\ environment variable, downloads the associated artifacts, extracts them, and copies the benchmark results to a local directory, updating the internal cache to avoid reprocessing.

core/bench/dashboard/server/src/github · high confidence

The HTTP connector runtime now includes a configuration provider that supports dynamic URL construction and structured JSON response parsing. Users can define custom URL templates for connector operations (such as creating or retrieving source and sink configurations) using variable placeholders like {key} and {version}, which are resolved against a base URL. Additionally, the provider introduces a response extractor that allows users to specify dot-notation paths to locate specific data or error messages within JSON API responses, enabling more precise handling of nested API structures.

_core/connectors/runtime/src/configs/connectors/http\provider · high confidence

This change introduces the first C++ client library for Iggy, implemented as a C++ wrapper around the existing Rust SDK via CXX FFI. It provides a high-level \IggyBlockingClient\ for C++ developers to manage the full resource lifecycle—including creating and deleting streams, topics, and users, as well as sending and polling messages—and includes the necessary Rust-side FFI glue code, type conversions, and a shared Tokio runtime to handle asynchronous operations.

foreign/cpp · high confidence

Added a new Apache Doris sink connector that writes JSON or CSV messages from Iggy streams to a pre-created Doris table via the Stream Load HTTP API. The connector supports configurable output formats (JSON by default, CSV opt-in), in-request retries for transient failures with exponential backoff, and security controls for FE-to-BE redirects including scheme-downgrade refusal and host allowlisting. Configuration requires specifying the Doris frontend URL, database, table, and credentials, with validation at startup for database/table names and CSV column mappings.

_core/connectors/sinks/doris\sink · high confidence

The C++ SDK is now available in the foreign directory, built via Bazel using the Rust toolchain (rules\_rust) and cxxbridge to generate bindings. The SDK exposes a high-level C++ API (iggy.hpp) for client operations, including user management, consumer groups, and messaging. The build configuration pins Bazel 9.2.0 and Rust 1.98.0, and includes BDD tests that run against the VSR server to validate the client's behavior.

foreign · high confidence

The \core/configs\_derive\ crate now provides the \\#\[derive(ConfigEnv)\]\ macro, which generates type-safe environment variable mappings for configuration structs at compile time. This eliminates runtime ambiguity by producing compile-time constants for valid environment variable names and a builder API for tests. The macro supports nested structs, \Vec\ fields with indexed expansion (up to 256 elements by default), and internally-tagged enums via a \tag\ attribute. Users can control mapping behavior with field attributes such as \skip\, \secret\, \leaf\, and \max\_elements\, ensuring that configuration overrides are strictly typed and discoverable.

_core/configs\derive · high confidence

The Connectors SDK now provides the foundational runtime infrastructure for building and managing source and sink connectors. It introduces a unified Payload type supporting JSON, Raw, Text, Protobuf, FlatBuffers, and Avro formats, enabling connectors to handle diverse data schemas. The SDK includes an HTTP API for monitoring connector status and statistics, a shared retry and resilience library with circuit breakers and exponential backoff, and a logging layer that forwards plugin logs to the runtime's OpenTelemetry subscriber. Additionally, it defines the Source and Sink traits with batch acknowledgment mechanisms and state management via MessagePack serialization, allowing plugins to persist and restore their internal state across restarts.

core/connectors/sdk/src · high confidence

A new Delta Lake sink connector is now available, allowing you to consume messages from Iggy topics and store them in Delta Lake tables on local filesystems, AWS S3, Azure Blob Storage, or Google Cloud Storage. The connector requires the target Delta table to exist beforehand and supports intelligent type coercion, automatically converting ISO 8601/RFC 3339 timestamp strings to microsecond timestamps and non-string values to strings to match the table schema. It performs transactional writes with atomic flush-and-commit operations, though it does not guarantee end-to-end at-least-once delivery as failed batches are not retried.

_core/connectors/sinks/delta\sink · high confidence

Added a new Elasticsearch sink connector that consumes messages from Iggy streams and indexes them into Elasticsearch. The connector supports configurable cluster URLs, target index names, and optional basic authentication (username/password). It includes features such as automatic index creation, bulk indexing optimization, and configurable retry policies with exponential backoff for rejected items (HTTP 429 or 5xx). Users can tune HTTP timeouts, maximum retries, and retry delays to handle slow bulk workloads or transient network issues.

_core/connectors/sinks/elasticsearch\_sink, core/connectors/sinks/mongodb\sink · high confidence

A new HTTP Sink connector is now available, allowing Iggy streams to deliver messages to any HTTP endpoint such as webhooks, REST APIs, or SaaS integrations. The connector supports multiple payload formatting modes—including individual requests, newline-delimited JSON (NDJSON), JSON arrays, and raw bytes—and automatically wraps payloads in a metadata envelope containing Iggy-specific fields like offset, timestamp, and stream/topic identifiers. It includes robust configuration options for HTTP methods, timeouts, retry policies with exponential backoff, TLS settings, and custom headers, enabling reliable delivery to external services.

_core/connectors/sinks/http\sink · high confidence

Users can now stream data to Apache Iceberg tables via a new sink connector. The connector initializes a REST-based Iceberg catalog and supports S3-compatible object storage (including MinIO) with configurable path-style access. It offers both static routing to predefined tables and dynamic routing based on a message field, while enforcing strict credential validation (requiring both access key and secret key or neither) and logging redacted credentials for security.

_core/connectors/sinks/iceberg\sink/src · high confidence

The Iceberg sink connector is now available, allowing users to route incoming messages to Apache Iceberg tables. This location implements the routing logic via two strategies: a StaticRouter that writes all messages to a pre-configured list of tables, and a DynamicRouter that inspects a configurable JSON field in each message to determine the target table at runtime. Both routers validate table names, handle schema mapping to Arrow/Parquet formats, and commit data files to the Iceberg catalog.

_core/connectors/sinks/iceberg\sink/src/router · high confidence

Users can now run benchmarks via the new \iggy-bench\ CLI tool and visualize results in a modern web dashboard. The CLI supports multiple benchmark types (producer, consumer, combined, consumer group) and transports (TCP, WebSocket), with configurable topic durability policies and host-preparation guidance for accurate measurements. The dashboard provides interactive trend visualizations, hardware/version filtering, and a responsive interface built with a Rust backend and Yew frontend, featuring a redesigned dark mode that applies the Iggy brand colors.

core/bench · high confidence

The Kafka gateway now bridges requests to an Iggy backend, introducing SASL/PLAIN authentication that forwards Kafka credentials to Iggy for verification, along with a configurable topic mapping system (via \IGGY\_KAFKA\_TOPIC\_MAP\_PATH\ or \IGGY\_KAFKA\_IGGY\_STREAM\) to resolve Kafka topics to Iggy streams and topics. The bridge enforces strict configuration requirements, such as mandating the \IGGY\_KAFKA\_IGGY\_PASSWORD\ environment variable, and implements error mapping to translate Iggy SDK errors into appropriate Kafka protocol error codes for operations like produce, fetch, and topic management.

gateways/kafka · high confidence

The Iggy Model Context Protocol (MCP) server is now available in core/ai/mcp/src, enabling AI agents to interact with the Iggy streaming platform. The server supports both HTTP and stdio transports; the HTTP mode allows binding to a configurable address and path, with optional TLS encryption via certificate files and CORS configuration. It connects to Iggy using either username/password or a token (which can be read from a file prefixed with 'file:'), and supports Iggy-side TLS. Telemetry (logs and traces) can be enabled via OpenTelemetry OTLP (gRPC or HTTP). On Unix systems, the server handles SIGINT/SIGTERM for graceful shutdown, and when the 'systemd' feature is enabled, it integrates with systemd's watchdog and readiness notifications.

core/ai/mcp/src · high confidence

A new InfluxDB sink connector has been added to the Iggy connectors, allowing messages from Iggy streams to be written to InfluxDB as line-protocol points. The connector supports both InfluxDB V2 (OSS/Cloud) and V3 (Core/Enterprise), distinguished by a \version\ configuration key (defaulting to V2 for backward compatibility). It handles payload serialization (JSON, text, base64), configurable metadata tagging, and includes resilience features such as exponential backoff retries, circuit breaking, and startup health checks.

_core/connectors/sinks/influxdb\sink · high confidence

This change adds the core library components for a new Apache Flink connector, enabling users to read from and write to Iggy streams. It introduces configuration classes for connection details (IggyConnectionConfig), offset management strategies (OffsetConfig), and TCP endpoint parsing (TcpEndpoint), alongside a custom exception hierarchy (ConnectorException). The library provides an IggySink for writing records with configurable batching, flushing, and partitioning strategies, and an IggySource for reading messages using consumer groups with automatic offset tracking.

foreign/java/external-processors/iggy-connector-flink/iggy-connector-library · high confidence

Added a new Model Context Protocol (MCP) server implementation in \core/ai/mcp/src/service\ that exposes Iggy cluster operations as AI tools. This service allows AI agents to manage streams, topics, and partitions through defined tools such as \create\_stream\, \create\_topic\, \get\_topics\, and \poll\_messages\, with access control enforced via a \Permissions\ struct that validates read, create, update, and delete actions before executing the underlying Iggy client requests.

core/ai/mcp/src/service · high confidence

Adds a new PostgreSQL sink connector that consumes messages from Iggy topics and persists them into a PostgreSQL database. The connector supports configurable payload storage formats (BYTEA, JSONB, or TEXT), automatic table creation, and optional inclusion of message metadata (offset, timestamp, checksum, origin timestamp). It features connection pooling, batch processing with configurable sizes, and retry logic for transient insertion failures.

_core/connectors/sinks/postgres\sink · high confidence

The SDK now includes a new QUIC-based client implementation located in core/sdk/src/quic. This addition provides a high-performance transport layer alongside the existing TCP client, featuring leader-aware connection routing, automatic failover to surviving nodes when the current one dies, and support for consumer groups. The implementation includes a mechanism to skip server certificate verification for development scenarios and handles transient errors with specific retry intervals and roster walking to ensure reliable message delivery.

core/sdk/src/quic · high confidence

A new Random Source connector is available, designed to generate random data and send it to configured streams and topics. Users can configure the generation interval, maximum message count, message size range, and payload size via the plugin configuration. The connector supports state persistence, ensuring that the message count is preserved across restarts and committed only upon successful acknowledgement.

_core/connectors/sources/random\source · high confidence

Adds a new S3 sink connector that writes Iggy stream messages to Amazon S3 and S3-compatible stores (MinIO, Cloudflare R2, DigitalOcean Spaces, Backblaze B2). The connector buffers messages and uploads them via single PutObject calls, supporting configurable file rotation by size or message count, and output formats including JSON Lines, JSON Array, and Raw. It allows custom S3 endpoints, path-style addressing, and configurable S3 key paths using variables like stream, topic, date, and hour. Users can include message metadata and headers in the output, and the connector handles transient upload failures with exponential backoff and jitter. Startup validates write access to the target bucket, and the connector supports standard AWS credential sources including explicit keys, environment variables, shared credentials files, STS web identity, and instance metadata.

_core/connectors/sinks/s3\sink · high confidence

The \core/consensus\ module now implements the VSR (View-Stable Replication) consensus protocol, providing the foundational logic for cluster coordination. This includes a \ClientTable\ that manages session lifecycles, request deduplication, and reply caching to ensure at-most-once delivery semantics. The module introduces a \DoViewChange\ (DVC) merge mechanism to safely transition between views, ensuring committed operations are preserved and uncommitted ones are discarded based on quorum evidence. Additionally, it provides state transfer capabilities for bootstrapping new replicas and a \Clock\ abstraction to support deterministic simulation and testing.

core/consensus · high confidence

The \core/common/src/types/cluster\ module now exposes structured types for cluster introspection, including \ClusterMetadata\ (holding the cluster name and a list of nodes), \ClusterNode\ (containing name, IP, role, status, and transport endpoints), and supporting enums for \ClusterNodeRole\ (Leader, Follower) and \ClusterNodeStatus\ (Healthy, Starting, Stopping, Unreachable, Maintenance, Unknown). A new \TransportEndpoints\ struct standardizes port definitions for TCP, QUIC, HTTP, and WebSocket connections per node. These types provide the foundational data structures for retrieving and displaying cluster topology and node health information.

core/common/src/types/cluster · high confidence

The server component is now a distinct module under core/server, providing the foundational build and deployment artifacts. This includes a Dockerfile and .dockerignore for containerized builds (supporting Rust 1.98, Alpine 3.23, and cross-compilation via zigbuild), a build script (build.rs) that embeds the Web UI and handles musl linking stubs, and a comprehensive config.toml defining HTTP, JWT, and consumer group settings. Additionally, a README.md documents the server's architecture (io\_uring, VSR), configuration, and systemd integration, while a server.http file provides a reference for API endpoints.

core/server · high confidence

The benchmarking tool now generates reports using a new analytics module that records cluster topology and parallelizes the report-building process. The \BenchmarkReportBuilder\ fetches cluster metadata from the server to distinguish between single-node and multi-node runs, ensuring that trend data is grouped correctly in the dashboard. Metric aggregation (throughput, latency) is now executed in parallel threads, and the resulting reports include detailed cluster node information alongside standard performance metrics.

core/bench/src/analytics · high confidence

The connectors runtime is now available as a standalone executable that manages the lifecycle of source and sink connectors. It loads connector plugins via dynamic linking, initializes source and sink managers, and exposes an HTTP API for configuration and status reporting. The runtime supports OpenTelemetry telemetry (logs and traces via gRPC or HTTP), configurable logging formats (text or JSON), and Prometheus metrics for monitoring connector health and throughput. It also includes benchmarking capabilities to track per-batch processing times for both source and sink connectors.

core/connectors/runtime/src · high confidence

The new \core/shard\ crate provides the infrastructure for a multi-shard server architecture. It introduces the \IggyShard\ component, which handles frame routing between shards and internal planes, and a \ShardZeroCoordinator\ that manages round-robin delegation of incoming client and replica connections to other shards. The crate also implements a \ShardHost\ trait, allowing the embedding server to handle business logic (such as client requests and metadata submissions) dispatched from the shard layer, and adds comprehensive metrics for tracking inter-shard frame drops.

core/shard · high confidence

The \helm/charts/iggy\ directory now contains a complete, unified Helm chart (version 0.6.0, app version 0.9.0-edge.6) for deploying the Iggy server and web UI. This chart introduces native support for cluster mode, allowing users to deploy a replicated cluster by creating one Helm release per node with a shared roster configuration. It includes validation logic to prevent common misconfigurations (such as wildcard binds or missing replica IDs) at render time, handles \io\_uring\ requirements via specific security contexts, and provides examples for multi-node setups. The chart also standardizes the deployment of server secrets, root user credentials, and ingress resources for both the server and UI components.

helm/charts/iggy · high confidence

The core/common/locking module now provides an async-compatible read-write lock wrapper (IggyTokioRwLock) built on top of Tokio's RwLock. This change introduces a new trait (IggyRwLockFn) and implementation that allows code to perform async read and write operations on shared data, replacing or supplementing previous synchronous locking mechanisms in this location.

core/common/src/locking · high confidence

Adds the \connectors.rs\ module defining the configuration schema for connectors, including the \ConnectorKey\ validation logic (enforcing alphanumeric start and length limits to prevent path traversal), \ConfigFormat\ enum, and \SinkConfig\/\SourceConfig\ structures with \ConfigEnv\ derive macros for environment variable mapping. Additionally, introduces \runtime.rs\ which defines the \ConnectorsRuntimeConfig\ struct, bringing together HTTP, Iggy, state, telemetry, and logging configurations, including \TelemetryConfig\ with gRPC/HTTP transport support and \LoggingConfig\ with text/JSON format options.

core/connectors/runtime/src/configs · high confidence

The metadata module now persists its state to disk using a combination of snapshots and a Write-Ahead Log (WAL). On startup, the system recovers the last known state by loading the latest snapshot and replaying subsequent journal entries, ensuring no committed operations are lost after a restart. The implementation includes robust error handling for corrupted or mismatched superblock records, preventing the node from booting with inconsistent consensus state.

core/metadata/src/impls · high confidence

Users can now attach arbitrary key-value metadata to topics, streams, and users at creation. This change introduces a new \ResourceOptions\ type backed by a \BTreeMap\ to ensure deterministic ordering across replicas, and defines a \Durability\ enum allowing users to specify whether data should be \Replicated\ (quorum commit) or \Persisted\ (quorum commit with stable-storage barrier). The implementation includes JSON serialization support for HTTP responses, distinguishing between explicit client-provided values and derived server defaults.

core/common/src/types/options · high confidence

The dashboard server now maintains an in-memory cache of benchmark reports to improve performance and enable new navigation features. The cache loads lightweight JSON reports from the results directory, mapping them by hardware identifier and git reference, and automatically reloads when files change via a background watcher. This infrastructure supports deep linking to specific benchmarks and a 'Recent' tab that displays the newest reports sorted by timestamp, while also stripping HTML files to reduce storage overhead.

core/bench/dashboard/server/src/cache · high confidence

The SDK now provides dedicated configuration structs—\IggyConsumerConfig\, \IggyProducerConfig\, and \IggyStreamConfig\—in the \stream\_builder\ module, each equipped with a builder pattern for easier setup. Users can now explicitly configure consumer behaviors such as auto-commit strategies, polling intervals, and retry logic, as well as producer settings like batch lengths, linger times, and partitioning. The \IggyStreamConfig\ aggregates these to manage both sides of a stream, supporting convenient initialization from stream and topic names while exposing identifiers and settings for fine-grained control.

_core/sdk/src/stream\builder/config · high confidence

The Java SDK has been migrated to the Apache Iggy project, introducing a new fluent builder API for creating both blocking and async TCP clients as well as HTTP clients. The SDK now requires Java 17, enforces non-null API contracts via a custom annotation, and provides detailed version information including build time and Git commit hash. The new architecture separates client implementations into distinct async and blocking packages, with comprehensive interfaces for streams, topics, messages, consumer groups, and system operations.

foreign/java/java-sdk · high confidence

A new Kafka wire protocol gateway is introduced, allowing existing Kafka clients to connect to Iggy via a TCP listener (default port 9093). The gateway supports SASL/PLAIN authentication, verifying credentials against Iggy users, and provides a read-only view of Iggy permissions mapped to Kafka ACLs via DescribeAcls. It implements core coordination features including consumer group management (JoinGroup, Heartbeat, LeaveGroup, SyncGroup) and producer identity (InitProducerId), while mapping Kafka records to Iggy messages for Produce and ListOffsets operations when the Iggy bridge is enabled.

gateways · high confidence

The web interface has been migrated to SvelteKit, introducing a new build pipeline, Dockerfile, and development configuration. This change adds a \justfile\ for simplified local development, a \docker-compose.yml\ for running the server and UI together, and a \Dockerfile\ that builds the SvelteKit app and bundles third-party license compliance files. The UI now uses Tailwind CSS for styling, \@floating-ui/dom\ for positioning tooltips, and the \uuid\ npm library for generating identifiers. API interactions are standardized through a typed schema (\ApiSchema.ts\) and client libraries that handle authentication, error redirection, and safe parsing of large integers (BigInt) to strings.

web · high confidence

This change introduces a suite of new shell and Node.js scripts under scripts/ci/ to standardize and harden the development workflow. The bump-version.sh script provides a unified tool for managing version numbers across Rust crates, SDKs, and the web UI, supporting patch/minor/major bumps and edge-tagging. Safety and compliance are improved with binary-artifacts.sh (detecting compiled binaries in the repo), license-headers.sh (enforcing Apache headers via HawkEye), and render-node-licenses.mjs (validating npm license compliance). Additionally, apt-install.sh prevents CI jobs from hanging on dead mirrors, while coverage-baseline-affected.sh and edge-affected-images.sh optimize CI runs by only triggering jobs for changed code surfaces. The lib/init.sh script enforces a minimum Bash version (4.2) to ensure script portability.

scripts · high confidence

The \core/cpu\_allocation\ crate now provides the configuration types (\CpuAllocation\, \NumaConfig\) and the \allowed\_cpus\ probe that the server uses to determine which CPU cores are available. The \allowed\_cpus\ function respects environment-imposed restrictions (such as systemd \AllowedCPUs\ or container cpusets) by querying the process's affinity mask, falling back to available parallelism if the mask is unavailable. The new \CpuAllocation\ enum supports parsing from TOML strings for \all\, specific counts, ranges (e.g., \2..8\), and NUMA-aware configurations (e.g., \numa:auto\ or \numa:nodes=0,1;cores=4;no\_ht=true\), enabling the server to correctly pin shards to cores within the allowed set.

_core/cpu\allocation · high confidence

A new ClickHouse sink connector is available, allowing you to consume messages from Iggy topics and insert them into ClickHouse tables. It supports three insert formats: \json\_each\_row\ (default), \row\_binary\ (for high-throughput binary serialization with startup schema validation), and \string\ passthrough (for raw CSV/TSV/JSON). The connector includes configurable batch processing, automatic retries with exponential backoff for transient errors, and startup readiness checks to ensure the target table and schema are valid before processing begins.

_core/connectors/sinks/clickhouse\sink · high confidence

Added a new Elasticsearch source connector that polls documents from an Elasticsearch index and delivers them as JSON messages to the Iggy runtime. The connector supports incremental processing using an RFC3339 timestamp field as a watermark to track progress, and includes persistent state storage via MessagePack checkpoints to the runtime's file or HTTP backend. It also offers an optional plugin-level JSON snapshot for state override, tracks processing statistics (documents fetched, payload bytes, poll durations), and handles errors by retrying after a configurable polling interval without advancing the watermark on failures.

_core/connectors/sources/elasticsearch\source · high confidence

The connectors runtime now exposes an HTTP API (powered by Axum) for managing sources and sinks. This API provides endpoints to list, retrieve, and update configurations for sources and sinks, view their status and transforms, and restart them. It includes built-in authentication via an API key header, configurable CORS policies, optional TLS support, and health/stats endpoints. The API also validates connector keys to prevent path injection and warns operators if the API is exposed without proper security controls (e.g., no API key, open CORS, or no TLS).

core/connectors/runtime/src/api · high confidence

The connectors runtime now supports two new methods for loading connector configurations: an HTTP-based provider and a local file-based provider. The HTTP provider (\HttpConnectorsConfigProvider\) fetches configuration via REST API calls, supporting custom headers, timeouts, URL templating, response extraction, and automatic retry logic for transient failures. The local provider (\LocalConnectorsConfigProvider\) reads connector definitions from TOML files in a specified directory, caching them in memory with support for versioned configurations and thread-safe access. These providers implement the \ConnectorsConfigProvider\ trait, allowing the system to dynamically load and manage sink and source connector settings from either remote or local sources.

core/connectors/runtime/src/configs/connectors · high confidence

The \core/common/src/http\ module now exposes a comprehensive set of serializable command structs for managing streams, topics, partitions, consumer groups, consumer offsets, segments, and personal access tokens. This includes definitions for creating, deleting, and updating resources (e.g., \CreateTopic\, \UpdateStream\, \CreateConsumerGroup\), as well as operational commands like \PollMessages\, \SendMessages\, and \StoreConsumerOffset\. These models define the expected JSON payloads and validation rules for the HTTP API, enabling clients to interact with these core resources.

core/common/src/http · high confidence

The HTTP source connector is now available, allowing Apache Iggy to act as a webhook receiver. It runs an embedded HTTP server that accepts POST bodies on authenticated paths (via Bearer tokens or HMAC signatures) and produces them as raw bytes to a configured stream and topic. The connector supports multiple instances sharing a single listener, each routing to its own topic, and includes a management API for dynamic endpoint registration, revocation, and secret rotation. It provides best-effort delivery semantics with configurable buffer capacities, batch sizes, and body size limits, and exposes Prometheus metrics for monitoring request handling and buffer usage.

_core/connectors/sources/http\source · high confidence

The Rust SDK now includes a new HTTP transport layer, allowing clients to communicate with the Iggy server via HTTP/REST in addition to existing protocols. This change introduces the \HttpClient\ struct and implements the common SDK client interfaces (such as \MessageClient\, \StreamClient\, \TopicClient\, \UserClient\, and \SystemClient\) to expose full CRUD and messaging capabilities over HTTP. Users can now connect to the server using standard HTTP endpoints, enabling easier integration in environments where TCP/QUIC protocols are restricted or where HTTP-based tooling is preferred.

core/sdk/src/http · high confidence

A new source connector has been added for InfluxDB, enabling polling of data into Iggy streams. It supports both InfluxDB V2 (using Flux queries and annotated-CSV responses) and V3 (using SQL queries and JSONL responses). The connector handles cursor-based polling with version-specific semantics, including skip-based deduplication for V2 and offset-based pagination with stuck-batch handling for V3. Configuration allows specifying the version, query, authentication, and polling parameters, with backward compatibility for existing V2 configurations that omit the version field.

_core/connectors/sources/influxdb\source · high confidence

The connectors SDK now includes JSON-specific implementations for several field transforms, allowing users to add, delete, update, and filter fields within JSON payloads. The \add\_fields\ transform supports static values and computed fields (timestamps, UUIDs), while \update\_fields\ allows conditional updates based on key existence. The \delete\_fields\ transform removes specified keys, and \filter\_fields\ enables inclusion or exclusion of fields based on key patterns or value conditions (e.g., numeric ranges). Additionally, the \unwrap\_envelope\ transform extracts data from a specified nested field, facilitating the handling of database change records or similar envelope formats.

core/connectors/sdk/src/transforms/json · high confidence

Added a suite of Java example applications in the \examples/java\ directory to demonstrate core and advanced usage of the Iggy client. The new examples include \GettingStartedProducer\ and \GettingStartedConsumer\ for basic blocking TCP operations, \AsyncProducer\ and \AsyncConsumer\ for non-blocking async patterns with backpressure and error handling, \MessageHeadersProducer\ and \MessageHeadersConsumer\ for sending and reading custom message headers, \MessageEnvelopeProducer\ and \MessageEnvelopeConsumer\ for structured JSON payload serialization, and \MultiTenantConsumer\ for demonstrating multi-tenant isolation and permissions.

examples/java · high confidence

The \foreign/java/bench\ directory now includes a complete Java-based benchmarking tool, starting with a \pinned-producer\ subcommand. This CLI (powered by picocli) allows users to configure and run asynchronous TCP producer benchmarks against an Iggy server, handling resource provisioning (creating streams and topics), executing the load with configurable message sizes and batch counts, and generating detailed JSON reports with latency and throughput metrics.

foreign/java/bench · high confidence

A new CLI utility, \kafka-message-gen\, has been added to the \gateways/kafka/tools/kafka-tool\ package to assist with testing and debugging Kafka protocol interactions. The tool can generate binary \.bin\ files for all supported Kafka API keys and versions (based on the Kafka 4.1.0 schema) and send them to a live Kafka-compatible server to inspect responses. It includes a \verify\ command to validate that the Iggy gateway correctly handles these requests, checking for correlation ID matches, schema decode success, and acceptable error codes (such as stub-specific errors like \NOT\_LEADER\_OR\_FOLLOWER\). This tool provides a concrete way to validate the gateway's protocol compliance and response handling without needing a full broker.

gateways/kafka/tools/kafka-tool · high confidence

Added a comprehensive set of TypeScript examples for the Node.js SDK, including basic producer/consumer scripts, a getting-started guide, and advanced patterns such as message envelopes, message headers, multi-tenant streaming, and sink data production. These examples demonstrate how to connect via TCP, handle authentication, manage streams and topics, and process messages with various strategies.

examples/node · high confidence

A new Quickwit sink connector has been added, allowing users to stream data to a Quickwit search engine via HTTP. The sink automatically checks service readiness and creates the target index if it does not exist, then appends messages as NDJSON. It supports configurable HTTP retry policies, timeouts, and verbose logging, and handles various payload shapes (JSON objects, arrays, raw text, and base64-encoded data) by wrapping them appropriately for Quickwit's ingestion API.

_core/connectors/sinks/quickwit\sink · high confidence

A new Redshift sink connector has been added to the system, enabling users to write Apache Iggy stream messages into Amazon Redshift. The connector stages data by serializing batches into Parquet files and uploading them to a configured S3 bucket (or S3-compatible store like MinIO), then loads the data into a target Redshift table using the \COPY\ command. Users can configure connection details, AWS credentials or IAM roles, S3 bucket/prefix settings, and options such as including metadata, checksums, and origin timestamps. The connector supports configurable payload formats (varbyte or text) and offers an archiving option to retain staged Parquet files after successful loading.

_core/connectors/sinks/redshift\sink · high confidence

The Rust SDK now includes a new TCP client implementation in \core/sdk/src/tcp\ that provides robust connection management. This client automatically fails over to a surviving cluster node when the current one dies, using a leader-aware connect mechanism to maintain stability. It also adds support for TLS connections via \rustls\ and includes a \NoServerVerification\ mode for development or testing environments.

core/sdk/src/tcp · high confidence

The Rust examples directory now includes dedicated starter and advanced usage guides. Basic and getting-started examples demonstrate simple TCP producer and consumer workflows, including stream/topic creation and message polling. Additional examples illustrate advanced message header usage: a message-envelope pattern for routing different JSON types, a message-type pattern using custom headers, and a message-compression example using Lz4 and user headers to signal compression algorithms. A typed-headers example shows how to attach and read heterogeneous header types (strings, integers, floats, booleans) on messages.

examples/rust · high confidence

A new sink connector has been added that writes Apache Iggy stream messages into SurrealDB via its HTTP API. The connector chunks messages into configurable batch sizes (default 1000) and performs bulk inserts using deterministic record IDs to ensure idempotency on replay. It supports flexible payload formatting (auto, JSON, text, base64) and optional inclusion of metadata, headers, checksums, and origin timestamps. Configuration allows for custom endpoints, authentication scopes, TLS, automatic table definition, and retry policies with exponential backoff for transient failures.

_core/connectors/sinks/surrealdb\sink · high confidence

A new configuration module has been added to the common crate to manage WebSocket client settings. This introduces a builder pattern for configuring the server address, auto-login, reconnection behavior (including retries and intervals), and heartbeat intervals. It also exposes fine-grained control over the underlying tungstenite library, allowing users to adjust read/write buffer sizes, maximum message and frame sizes, and TLS options such as domain, CA file path, and certificate validation.

_core/common/src/types/configuration/websocket\config · high confidence

The \iggy-bench\ tool now includes a comprehensive analytics and reporting system. It calculates time-series data for latency and throughput (in MB/s and messages/s) using configurable bucket sizes, and generates HTML plots for throughput, latency, and latency distribution (with log-normal fit) that can be saved to a directory and optionally opened in a browser. The benchmark runner now automatically collects system hardware information, generates a JSON report, and saves server logs alongside the results.

core/bench/src · high confidence

The benchmark dashboard frontend has been rebuilt to support deep linking and a 'Recent' tab, allowing users to navigate directly to specific benchmarks via URL. A new compare mode enables side-by-side analysis of two benchmarks, while the UI now displays cluster topology and detailed benchmark metadata (such as streams, partitions, and batch configuration). The dashboard also features redesigned latency and throughput charts, including a new latency distribution chart with log-normal fit, and provides embeddable chart endpoints that generate PNG images and iframe snippets for external use.

core/bench/dashboard/frontend · high confidence

The IggyBench Dashboard server is now available in the monorepo, providing a web interface and REST API to browse benchmark results. Users can query hardware configurations, git refs, and specific benchmark reports via endpoints like /api/hardware and /api/benchmark/full/{unique\_id}. The server includes a 'Recent' tab for quick access to recent benchmarks, supports deep linking, and offers embeddable chart endpoints that render charts as PNG images for external use. It also features a GitHub artifact poller to automatically fetch new benchmark data, configurable via command-line arguments for host, port, results directory, and CORS origins.

core/bench/dashboard/server/src · high confidence

The benchmark reporting system now includes a new plotting module that generates interactive charts for throughput and latency metrics. This addition introduces support for visualizing performance data with configurable axes (time or category), dual Y-axes for combined metrics, and interactive features like tooltips, legends, and data zooming. The module supports both light and dark themes and provides a structured way to render benchmark results as charts.

core/bench/report/src/plotting · high confidence

The benchmark report module now defines a comprehensive set of data structures for serializing and deserializing benchmark results. This includes the root \BenchmarkReport\ model, which aggregates hardware details, server stats, cluster topology, and per-actor/group metrics. New types support detailed metric tracking, including \BenchmarkGroupMetrics\ and \BenchmarkIndividualMetrics\ with custom deserializers that backfill missing latency statistics from time-series data for backward compatibility. The schema also introduces \LatencyDistribution\ for log-normal fit data, \BenchmarkClusterInfo\ for multi-node topology, and \BenchmarkHardware\ for system specs. Additionally, \BenchmarkNumericParameter\ allows benchmark arguments to be specified as either fixed values or ranges, and \TimeSeries\ includes LTTB downsampling logic to optimize chart rendering performance.

core/bench/report/src/types · high confidence

The benchmark reporting system now includes a new \core/bench/report\ module that generates visual charts and formatted text summaries. Time-series charts for throughput and latency use LTTB downsampling to reduce data points while preserving visual shape, ensuring readable graphs even with high-frequency data. A new latency distribution chart displays density, PDF, and percentile curves for each actor group. Additionally, the text-based summary output has been enhanced with a 'pretty' mode that uses colored, wide/narrow terminal-aware tables to display detailed metrics like throughput, latency percentiles, and standard deviation.

core/bench/report/src · high confidence

The benchmark CLI now supports additional workload profiles—balanced, end-to-end, and pinned—each with dedicated argument structures for producers, consumers, and consumer groups. Users can configure stream, partition, producer, and consumer counts, as well as topic size limits and message expiry times. The balanced and pinned producer-consumer profiles introduce a configurable read-amplification multiplier (default 1.05) to adjust consumer throughput relative to producers, helping prevent queue buildup during end-to-end latency measurements. Validation rules ensure logical consistency, such as requiring consumer groups to be at least equal to the number of streams in balanced consumer group scenarios, and enforcing a single partition for pinned producer-consumer workloads.

core/bench/src/args/kinds · high confidence

The CLI now includes a suite of new commands under the binary protocol implementation: \get-client\ and \get-clients\ to inspect client details and lists; \get-cluster-metadata\ to display cluster node information; \create-consumer-group\, \delete-consumer-group\, \get-consumer-group\, and \get-consumer-groups\ to manage consumer group lifecycle and status; \get-consumer-offset\ and \set-consumer-offset\ to view and update consumer offsets; and a full context management set (\create-context\, \delete-context\, \get-contexts\, \show-context\, \use-context\) to handle multiple connection profiles and authentication states locally.

core/cli/src/commands · high confidence

The binary protocol layer now includes a new set of wire primitives in \core/binary\_protocol/src/primitives\ to support the updated consumer-offset commands and extensible configuration. This adds \AckLevel\ to allow clients to specify \NoAck\ or \Quorum\ acknowledgement policies for offset writes, \WireConsumer\ to distinguish between single consumers and consumer groups, and \WireIdentifier\ to support both numeric and string-based identifiers. It also introduces \WireOptions\ for validating key-value configuration blocks, \WirePartitioning\ for message routing strategies, \WirePermissions\ for granular user/stream/topic access control, \WirePollingStrategy\ for consumer resume points, and \WireUserHeaders\ for extensible message metadata. These types define the wire format and validation logic for the new protocol features.

_core/binary\protocol/src/primitives · high confidence

The binary protocol crate now includes wire-format definitions (encode/decode) and roundtrip tests for consumer group management (create, delete, get, join, leave, sync) and consumer offset operations (get, store, delete). These new request structures enable clients to interact with the server's consumer group coordination and offset tracking capabilities using the updated binary wire format.

_core/binary\protocol/src/requests · high confidence

The binary protocol layer now includes explicit wire-format definitions and decoders for three new response types: \PollMessagesResponse\ (which decodes a 16-byte header and iterates over batched message records with resolved offsets and timestamps), \PollRoutingResponse\ (which encodes consumer session and primary node metadata for routing decisions), and \SendMessagesResponse\ (which returns a list of partition-level commit confirmations containing stream, topic, partition IDs, and the assigned base offset). These changes introduce the concrete serialization logic for these protocol messages, enabling clients to correctly parse server responses for polling, routing, and message sending operations.

_core/binary\protocol/src/responses/messages · high confidence

The binary protocol now includes structured response definitions for user operations, enabling clients to correctly parse server replies. This adds wire-format encoders and decoders for user details (including optional permissions), user lists, and authentication flows (login, combined login/register, and logout). It also defines response types for administrative actions like creating, updating, and deleting users, as well as changing passwords and updating permissions.

_core/binary\protocol/src/responses/users · high confidence

The binary protocol now supports wire-format encoding and decoding for several new system-level responses. Clients can retrieve cluster topology via GetClusterMetadata (including node names, IPs, ports, roles, and statuses), inspect server health and resource usage via GetStats (covering CPU, memory, disk, runtime, and per-partition cache metrics), and query available configuration options via DescribeOptions. Additionally, the protocol defines GetSnapshot for exporting diagnostic ZIP archives, GetMe for retrieving the authenticated client's own details, and Ping for basic connectivity checks.

_core/binary\protocol/src/responses/system · high confidence

The \core/binary\_protocol\ crate now serves as the single source of truth for the Iggy binary wire format, replacing the previous \ServerCommand\/\BytesSerializable\ approach. It introduces a sans-IO codec (\WireEncode\/\WireDecode\) and defines the exact request/response frame layouts (\\[length\]\[code\]\[request\_id\]\[payload\]\ and \\[status\]\[length\]\[request\_id\]\[payload\]\). The crate also introduces a new 256-byte batch header structure for message ingestion and replication, a command dispatch table mapping codes to VSR consensus operations, and a packed namespace layout for routing. This change standardizes the wire format shared between the server and SDK, enabling zero-copy parsing and stricter validation of batch checksums and frame lengths.

_core/binary\protocol/src · high confidence

A new \ArgsOptional\ struct has been introduced in \core/common/src/types/args/mod.rs\ to define command-line arguments for client configuration. This change adds support for configuring transport protocols including HTTP, TCP, and QUIC, allowing users to specify connection details such as API URLs, server addresses, and TLS settings. It also introduces options for managing credentials, encryption keys, and retry/reconnection behaviors specific to each transport type, enabling more granular control over client connectivity and security.

core/common/src/types/args · high confidence

The \core/common\ library now includes \ConsumerGroupClientState\ to manage client-side caching of consumer group assignments, partition counts, and round-robin cursors for the VSR transport, enabling local partition resolution and rebalance detection via generation fences. It also introduces \PersonalAccessToken\ with secure random token generation and hashing, and exposes \ClientState\ for tracking connection lifecycle. These changes support improved consumer group coordination and secure token management within the SDK.

core/common/src · high confidence

The \core/common/src/traits\ module now exposes a comprehensive set of Rust traits that define the client interface and transport layer. This includes \BinaryTransport\ and \BinaryClient\ for low-level message handling, \Client\ as the main entry point, and specific traits for every server module (\ClusterClient\, \ConsumerGroupClient\, \ConsumerOffsetClient\, \MessageClient\, \PartitionClient\, \PersonalAccessTokenClient\, \SegmentClient\, \StreamClient\, \SystemClient\, \TopicClient\, \UserClient\). It also introduces \VsrSessionControl\ for managing consensus session state and \fail\_if\_not\_authenticated\ for enforcing authentication checks.

core/common/src/traits · high confidence

The \core/common/src/utils\ module now includes new types and helpers to improve configuration handling and data security. \IggyByteSize\ and \MaxTopicSize\ allow users to define and parse human-readable storage limits (e.g., "1 GB") for topics. \IggyDuration\ and \IggyExpiry\ provide standardized, human-readable time parsing for message retention and expiration. Additionally, \crypto.rs\ introduces AES-256-GCM encryption for message payloads and headers, while \checksum.rs\ and \hash.rs\ provide xxHash and Blake3 utilities for data integrity.

core/common/src/utils · high confidence

The \auth\_config\ module now includes new types to support parsing connection strings and managing auto-login credentials. Users can now configure authentication via \AutoLogin\ (enabling or disabling automatic login with \Credentials\ such as username/password or personal access tokens) and parse connection strings using \ConnectionString\ and \ConnectionStringUtils\. The \Credentials\ type wraps sensitive data in \SecretString\ to prevent accidental leaks, and \ConnectionStringOptions\ provides a trait for parsing connection-specific parameters like retries and heartbeat intervals.

_core/common/src/types/configuration/auth\config · high confidence

The connectors SDK now includes a comprehensive set of new transform and encoder modules that enable data format conversion and manipulation. Users can now convert payloads between JSON, Avro, FlatBuffers, and Protobuf formats using dedicated \AvroConvert\, \FlatBufferConvert\, and \ProtoConvert\ transforms. The SDK also introduces field manipulation capabilities via \AddFields\, \DeleteFields\, \FilterFields\, and \UpdateFields\ transforms, allowing for dynamic modification of message content. Additionally, new encoders (\AvroStreamEncoder\, \FlatBufferStreamEncoder\, \ProtoStreamEncoder\) and decoders are provided to handle serialization and deserialization for these binary formats, supporting schema loading from files or inline JSON.

core/connectors/sdk/src/transforms · high confidence

The integration test infrastructure has been replaced with a new \\#\[iggy\_harness\]\ procedural macro that generates test variants automatically. Tests can now declare a matrix of client transports (TCP, HTTP, QUIC, WebSocket, and their TLS variants) and server configuration overrides (such as heartbeat intervals or checksum validation) using a declarative DSL. The harness also supports multi-node cluster testing, automatic TLS certificate generation, and fixture injection, significantly reducing boilerplate and enabling broader coverage of transport and configuration combinations.

core/integration · high confidence

The connectors SDK now includes dedicated stream decoders for multiple data formats, allowing users to ingest and process Avro, FlatBuffers, Protobuf, JSON, raw binary, and text payloads. Each decoder (Avro, FlatBuffer, Proto, Json, Raw, Text) implements the StreamDecoder trait and supports configurable schema loading (via file path, JSON string, or descriptor set), field name mappings, and options to extract data as JSON or return the raw payload. This expands the range of message types the connectors runtime can natively decode.

core/connectors/sdk/src/decoders · high confidence

The integration test infrastructure now includes a new \harness/handle\ module that provides structured handles for managing test binaries. This adds \ServerHandle\ to launch and monitor the Iggy server (including TLS and cluster readiness checks), \ClientHandle\ and \ClientBuilder\ to connect via TCP, QUIC, HTTP, or WebSocket with optional auto-login, and dedicated handles for the \iggy-connectors\ and \iggy-mcp\ services. These components centralize process lifecycle, log collection, and health checks to make integration tests more robust and easier to write.

core/integration/src/harness/handle · high confidence

The metadata module now uses a new state machine (STM) implementation that enforces deterministic key allocation via \IdSlab\ to prevent replica divergence during snapshot restores, and gates all control-plane operations with role-based access control (RBAC) before state mutation. Consumer groups are now managed within the streams STM with monotonic IDs that are never reused, ensuring offset correctness across restarts. The snapshot format has been updated to version 5 to support new fields like \created\_view\ in partitions, and the system now supports state transfer between shards using factory bundles for read-side handoffs.

core/metadata/src/stm · high confidence

The \core/configs/src/server\_config\ module introduces the formal on-disk configuration schema for the new server implementation. This adds structured configuration sections for cluster topology and VSR consensus tunables (including heartbeat, view-change, and repair parameters), per-partition consensus settings (WAL capacity, group-commit delays, and eviction rings), and the inter-shard message bus (batch sizing, queue depths, and handshake timeouts). It also defines the metadata consensus plane schema (prepare queue depth, journal slots, and client table limits) and validates node advertised addresses, ensuring that previously hardcoded runtime constants are now exposed as configurable, validated server settings.

_core/configs/src/server\config · high confidence

Developers can now use the provided .devcontainer configuration to set up a consistent, cross-platform development environment in VS Code. The setup includes a Dockerfile based on Rust 1.98 and Node.js 22, pre-installing necessary build tools and dependencies. It configures the container to run Docker commands via the host's daemon (Docker-outside-of-Docker) and sets up specific port mappings for HTTP, QUIC, TCP, and WebSocket services. A post-create script initializes the environment, and a notice script warns about known limitations with bind-mount tests due to the Docker-in-Docker setup.

.devcontainer · high confidence

The message bus now includes a sharded connection cache with a round-robin allocation strategy to distribute SDK client connections across shards, alongside dedicated listener modules for TCP, TCP-TLS, WebSocket, WSS, and QUIC transports. These listeners run on shard 0 and delegate accepted connections to owning shards, with TLS and WebSocket handshakes performed in the install path to prevent slow peers from blocking the accept loop. The implementation also introduces runtime configuration for QUIC tuning and WebSocket frame layers, ensuring that socket options like TCP\_NODELAY are correctly applied and that handshake timeouts are enforced to mitigate slowloris-style attacks.

_core/message\bus · high confidence

A new \stream\_builder::build\ module has been added to the SDK, providing internal helper functions to construct \IggyClient\, \IggyConsumer\, \IggyProducer\, and stream/topic resources. This module centralizes the logic for initializing the client, configuring consumer groups and polling strategies, setting up producer batching and retry behaviors, and automatically creating streams and topics if they do not already exist based on configuration flags.

_core/sdk/src/stream\builder/build · high confidence

The Node.js client now accepts a connection string (e.g., \iggy://user:pass@host:port\) for initialization, parsing transport, credentials, TLS, and reconnection options. It enforces stricter client configuration validation, including bounds on heartbeat and reconnect intervals, and a max response frame size. The client now supports TLS transport, automatic reconnection with configurable retries and intervals, and heartbeats to maintain connection health. Additionally, it includes BDD tests using Cucumber for end-to-end verification of core operations like stream/topic creation, message send/poll, and raw command execution.

foreign/node · high confidence

A new PostgreSQL source connector has been added to stream data from PostgreSQL databases to Iggy topics. It supports two modes: table polling, which incrementally fetches data using configurable batch sizes and tracking columns, and Change Data Capture (CDC), which monitors database changes via logical replication. The connector allows flexible payload extraction from specific columns (BYTEA, TEXT, JSONB) and includes features such as delete-after-read, mark-as-processed, custom SQL queries, and retry logic for transient errors.

_core/connectors/sources/postgres\source · high confidence

The Python SDK has been renamed from \iggy\_py\ to \apache-iggy\ and rewritten to align with the Rust SDK's architecture. This update introduces a comprehensive client API including TCP, QUIC, HTTP, and WebSocket transport configurations, high-level producer and consumer interfaces with partitioning strategies, and full user and permission management. It also adds support for consumer groups, message partitioning, and async iteration, providing a complete, modern Python client for Apache Iggy.

foreign/python · high confidence

The repository now includes foundational configuration files for Apache Software Foundation (ASF) governance and developer workflows. This adds \.asf.yaml\ to define GitHub repository settings, branch protection, and mailing list notifications; \.pre-commit-config.yaml\ to enforce code quality via hooks for Rust, Python, TOML, and Markdown; and \.typos.toml\ to configure spelling checks. Additionally, new files such as \AGENTS.md\ (AI agent guidelines), \CONTRIBUTING.md\ (contribution standards), \SECURITY.md\ (vulnerability reporting), and license/notice files (\ASF\_LICENSE.txt\, \NOTICE\) have been added to standardize project operations and compliance.

(repo-wide) · high confidence

The Rust SDK has been reorganized into smaller packages and now features a new \ClientProvider\ that centralizes configuration for TCP, QUIC, HTTP, and WebSocket transports, including dedicated reconnection and heartbeat settings for each. A new leader-aware connection mechanism automatically detects the cluster leader and redirects client requests, improving availability during node failures. Additionally, the SDK introduces a \consume\_messages\ extension for \IggyConsumer\ that simplifies message processing loops with automatic offset committing and graceful shutdown support.

core/sdk/src · high confidence

The simulator's workload operations (such as creating or deleting streams, topics, users, and consumer groups) have been updated to explicitly target specific server-side outcomes (e.g., \Ok\, \NotFound\, \AlreadyExists\). Each operation now includes logic to sample inputs that reliably trigger these outcomes and to predict the resulting state changes, enabling more deterministic and comprehensive testing of the server's response handling.

core/simulator/src/workload/ops · high confidence

The benchmark reporting module now includes dedicated text-based formatting for report titles and subtext. The new \title\ module generates report headers that optionally include a user-provided remark, while the \subtext\ module constructs detailed summary lines displaying benchmark parameters (such as actors, streams, partitions, and message sizes) alongside key performance metrics including latency percentiles (P95, P99, etc.) and throughput statistics. This change provides a structured, human-readable text representation of benchmark results, specifically handling total system throughput for producer/consumer scenarios.

core/bench/report/src/plotting/text · high confidence

The SDK now exposes a single \ClientWrapper\ enum that unifies access to Iggy, HTTP, TCP, QUIC, and WebSocket transports. This wrapper implements all common client interfaces (system, stream, topic, partition, message, consumer group, user, personal access token, segment, cluster, and consumer offset), allowing users to interact with the same API surface regardless of the underlying connection protocol.

core/sdk/src/clients · high confidence

The Rust SDK now includes a new WebSocket client implementation, allowing users to connect to Iggy clusters via WebSocket (both plain and TLS) in addition to the existing TCP transport. This change introduces the \WebSocketClient\ and associated stream types (\WebSocketConnectionStream\, \WebSocketTlsConnectionStream\) within the \core/sdk/src/websocket\ module, enabling WebSocket-based communication for standard client operations.

core/sdk/src/websocket · high confidence

The server's legacy UDP transport layer has been removed. This change deletes the \args.rs\ configuration, the \command.rs\ dispatcher, and all specific protocol handlers (ping, poll, send, create/delete/get topics) that previously processed raw binary frames over UDP. Consequently, the server no longer supports this specific binary protocol, which was bound to \127.0.0.1:8080\ by default.

server · high confidence

The original UDP-based client application has been removed from the codebase. This change deletes the main entry point, the command parsing logic, and all specific command handlers (such as ping, send, poll, and topic management) that previously communicated with the server over UDP. This clears the way for the new SDK and transport implementations (HTTP/QUIC) referenced in the commit history.

client · high confidence

The placeholder implementation in the SDK library (specifically the \add\ function and its associated test) has been removed. This cleanup eliminates unused boilerplate code from the public API surface.

sdk · high confidence

The CLI's credential handling in core/cli/src has been refactored to store sensitive data (passwords, tokens) using the SecretString type from the secrecy crate. This change ensures that credentials are not accidentally exposed in logs or memory dumps, improving security for login, context, and personal access token operations.

core/cli/src · high confidence

The \server\_common\ crate has been introduced to centralize server-side components previously scattered across the codebase. This new module provides shared implementations for directory bootstrapping and cleanup, memory-pooled buffers, self-signed certificate generation, password hashing, and shard executor configuration. It also includes low-level I/O utilities such as aligned I/O buffers, consensus message framing, and file preallocation helpers, along with comprehensive diagnostics for Linux io\_uring runtime errors. This consolidation supports the ongoing migration to the new server architecture by providing a common foundation for these core services.

_core/server\common/src · high confidence

The partition journal implementation has been refactored to depend on a new \DurableStorage\ trait rather than a concrete file-storage implementation. This change introduces a \durable\_storage\ module defining the storage interface and a \file\_storage\ module providing the default file-backed implementation, allowing the journal to be decoupled from specific I/O mechanisms and enabling easier testing and potential future storage backends.

core/journal · high confidence

The server startup sequence has been refactored by splitting the previous monolithic bootstrap.rs file into a structured boot module (core/server/src/boot). This new layout separates concerns into distinct files: credentials.rs handles root user setup and TLS/PSK material, handoff.rs manages cross-shard metadata bundle broadcasting and listener barriers, listeners.rs controls TCP, WebSocket, QUIC, and HTTP listener startup, recovery.rs drives partition recovery and shard construction, threads.rs manages per-shard OS threads and shutdown coordination, and topology.rs resolves cluster roster and listener addresses. This change improves code maintainability and clarity of the boot process without altering the external server behavior.

core/server/src/boot · high confidence

The \core/bench/src/args\ module has been completely rewritten to replace the previous ad-hoc argument handling with a structured, modular approach using \clap\. This change introduces dedicated modules for common arguments (\common.rs\), default values (\defaults.rs\), benchmark kinds (\kind.rs\), and transport protocols (\transport.rs\). Users benefit from a more robust CLI that enforces validation rules (such as mutual exclusivity between \--message-batches\ and \--total-data\), provides clearer help text and examples (\examples.rs\), and supports a wider range of configuration options including TLS for TCP, QUIC client binding, and detailed output management. The new structure also adds support for specific benchmark kinds like \BalancedConsumerGroup\ and \EndToEndProducingConsumer\ with their own argument sets, ensuring that transport-specific options (like \nodelay\ for TCP) are correctly scoped and validated.

core/bench/src/args · high confidence

The benchmarking tool in core/bench has been restructured to support the Iggy high-level API for producers and consumers, introducing new benchmark types such as BalancedProducer, BalancedConsumerGroup, and EndToEndProducingConsumer alongside the existing Pinned variants. This change replaces the previous TestServer startup capability with a ClientFactory abstraction, allowing benchmarks to run against live servers. It also adds configurable topic options including message expiry, durability, and read amplification to better simulate real-world conditions and prevent corrupted latency measurements.

core/bench/src/benchmarks · high confidence

The benchmarking tool's internal utilities have been restructured into distinct, reusable modules to improve code organization and maintainability. The new layout includes \batch\_generator\ for constructing message payloads, \client\_factory\ for abstracting client creation across different transport protocols (HTTP, TCP, QUIC, WebSocket), \cpu\_name\ for hardware identification, \finish\_condition\ for managing benchmark termination logic, \rate\_limiter\ for controlling throughput, and a consolidated \mod.rs\ for shared helpers. This change isolates specific concerns within the benchmarking infrastructure, making it easier to extend or modify individual aspects of the benchmarking process without affecting the entire system.

core/bench/src/utils · high confidence

The SDK's binary client implementations in \core/common/src/traits/binary\_impls\ have been rewritten to use the new \iggy\_binary\_protocol\ wire types and codes. This change replaces the legacy message wire format with the new VSR (Versioned Session Request) framing, ensuring compatibility with the updated server protocol. The diff shows new implementations for cluster, consumer groups, consumer offsets, messages, partitions, personal access tokens, segments, streams, system, topics, and users, all utilizing the new request/response structures from \iggy\_binary\_protocol\. This migration is a prerequisite for the server-ng upgrade and ensures consistent behavior across the SDK.

_core/common/src/traits/binary\impls · high confidence

The CLI permission argument handling code has been moved from the \cmd\ crate to \core/cli/src/args/permissions\, and the internal data structures for stream and topic IDs have been changed from \u32\ to \usize\. Additionally, the implementation now uses \BTreeMap\ instead of \HashMap\ for storing stream and topic permissions, and imports for \GlobalPermissions\, \StreamPermissions\, \TopicPermissions\, and \UserStatus\ have been updated to use the \iggy::prelude\ module.

core/cli/src/args/permissions · high confidence

The error types have been reorganized into the \core/common/src/error\ module, moving \ClientError\ from the SDK to the common crate and introducing a new \eviction.rs\ module. This module provides a unified \eviction\_reason\_to\_error\ function that maps wire-level \EvictionReason\ codes (such as \InvalidCredentials\, \IncompatibleProtocol\, and \StaleClient\) to specific \IggyError\ variants, ensuring consistent error reporting across TCP, QUIC, WebSocket, and HTTP transports. The \IggyError\ enum itself has been expanded with new variants like \TransientNotCommitted\, \TransientNotAccepted\, and \RequestAlreadyApplied\ to support improved commit-time metadata validation and retry semantics.

core/common/src/error · high confidence

The server configuration module now centralizes default values and validation logic for core settings. Users benefit from stricter validation on critical parameters, such as enforcing a minimum 512 MiB memory pool size and requiring non-zero intervals for maintenance cleaners. Additionally, sensitive data like JWT secrets and encryption keys are now explicitly excluded from serialized configuration snapshots to prevent accidental leakage, and the HTTP layer is restricted to HMAC-based JWT algorithms for security.

core/configs/src/common · high confidence

The configuration system now uses compile-time generated mappings to resolve environment variables, eliminating the previous runtime ambiguity in path resolution. This change introduces a \ConfigEnv\ derive macro that produces static mappings, a \TypedEnvProvider\ that validates against known variable names (refusing to boot on unknowns in debug builds), and a \FileConfigProvider\ that explicitly rejects relocated or removed configuration keys with clear guidance. Users benefit from stricter validation, deterministic error messages, and the ability to migrate obsolete settings before they cause silent failures.

_core/configs/src/configs\impl · high confidence

The server's stats response now includes detailed system resource information, specifically the number of threads, free disk space, and total disk space for the data directory. Additionally, per-partition cache metrics (hits, misses, and hit ratio) are exposed, allowing users to monitor cache performance. The stats structure also now respects CPU affinity and cgroup limits for accurate CPU and memory usage reporting.

core/common/src/types/stats · high confidence

The server's request dispatch logic has been restructured into a modular, plane-named architecture to make routing and error paths explicit. Authorization is now enforced at dispatch time for partition-plane and non-replicated operations via a dedicated \authz\ module, ensuring consistent RBAC checks before requests reach their respective planes. The failure handling system has been standardized with a unified \failure\ module that defines specific wire channels (such as \TypedDeny\ and \Eviction\) to ensure clients receive immediate, typed error responses rather than wedging on silent drops or ambiguous status codes. Additionally, the \host\ module now centrally manages per-client request queues and session lifecycles, improving connection stability and cleanup.

core/server/src/dispatch · high confidence

The HTTP client configuration module has been restructured to include stricter validation and easier setup. The \HttpClientConfigBuilder\ now automatically trims whitespace and validates the API URL before building the configuration, rejecting invalid addresses. Additionally, a new \HttpConnectionStringOptions\ implementation allows users to configure retry counts and heartbeat intervals by parsing them directly from a connection string, simplifying initialization for these specific settings.

_core/common/src/types/configuration/http\config · high confidence

The Identifier type has been moved to core/common/src/types/identifier and refactored to use the specific IggyError::InvalidIdentifier error instead of the generic Error::InvalidCommand. This change allows numeric identifiers to accept zero as a valid value (previously rejected), supporting 0-based consumer groups and slab keys. Additionally, the type now implements Eq and provides new methods like get\_cow\_str\_value and as\_cow\_str for more efficient string handling.

core/common/src/types/identifier · high confidence

The metadata module now exposes an \AppliedFrontier\ that tracks the highest metadata operation applied and published across the entire node. This allows reads served by any shard to reliably determine if they have seen the latest committed state, ensuring that clients receive data consistent with their own previous writes even when routed to peer shards.

core/metadata/src · high confidence

The \core/server\_common/src/sharding\ module now provides the foundational types for server-side partition routing, including \ShardId\, \LocalIdx\, \PartitionLocation\, and \IggyNamespace\. A key behavioral change is in \IggyNamespace::new\, which now strictly enforces stream, topic, and partition limits at admission; previously, out-of-range values were silently masked, causing different partitions to alias onto the same shard and consensus group. This fix ensures that metadata creation fails if limits are exceeded, preventing silent data corruption and routing errors.

_core/server\common/src/sharding · high confidence

The HTTP server module has been replaced with a new implementation that introduces per-session and global admission controls for partition writes (capping concurrent in-flight writes to prevent resource starvation) and adds a follower-side forwarding mechanism that transparently relays consensus-needing requests to the metadata primary, ensuring any cluster node can answer any request without returning transient errors.

core/server/src/http · high confidence

The TCP client configuration has been restructured into a dedicated module (\core/common/src/types/configuration/tcp\_config\) containing a builder, reconnection settings, and connection-string parsing. Users can now configure TLS (domain, CA file, validation), automatic reconnection (retries, interval, cooldown), heartbeats, and Nagle's algorithm via a builder or a connection string. The builder validates server addresses (IP, hostname, port) at build time, rejecting invalid formats like unbracketed IPv6 or missing ports.

_core/common/src/types/configuration/tcp\config · high confidence

The binary protocol crate now defines the complete VSR (Viewstamped Replication) consensus wire format, introducing a new \Command\ enum with 31 message types (including replica auth, state transfer, and session heartbeats) and an \Operation\ enum for replicated state-machine actions. All consensus headers are fixed at 256 bytes with zero-copy deserialization and, critically, a new frame-sealing mechanism that cryptographically verifies header integrity on the wire. This is a breaking change for the replica-to-replica control plane: mixed-version clusters will drop all control frames and halt, so replicas must be upgraded together with the cluster down. Client-facing headers (Request, Reply, Eviction) and legacy Prepare frames are unaffected and remain backward-compatible with older SDKs.

_core/binary\protocol/src/consensus · high confidence

The binary protocol module now includes explicit response definitions for topic management commands. \GetTopicResponse\ and \GetTopicsResponse\ provide structured decoding for topic metadata and partition details, replacing previous implicit handling. \CreateTopicResponse\ reuses the existing \GetTopicResponse\ structure to return the newly created topic's header. \DeleteTopicResponse\, \PurgeTopicResponse\, and \UpdateTopicResponse\ are defined as empty responses, reflecting that these operations do not return payload data. These changes standardize how the server communicates topic state and operation results to clients.

_core/binary\protocol/src/responses/topics · high confidence

The \core/common/src/types\ module now includes new type definitions for \Either\, HTTP methods (\HttpMethod\), and \Segment\ state, alongside a comprehensive \streaming\_stats\ implementation. This implementation introduces atomic statistics counters for streams, topics, and partitions that clamp to zero instead of wrapping on underflow, preventing incorrect metric values in \/stats\ and \/metrics\ endpoints when rollup decrements exceed totals. A process-wide counter tracks these underflows, logging warnings to alert operators that a rebuild is needed.

core/common/src/types · high confidence

This change introduces new type definitions for consumer groups and offsets in the common module, including \ConsumerGroupId\, \ConsumerGroupOffsets\, \ConsumerOffset\, and \ConsumerOffsets\, which utilize \papaya::HashMap\ for concurrent access. It also refactors the existing \Consumer\ and \ConsumerKind\ types by removing the \BytesSerializable\ trait and \FromStr\ implementations, replacing them with custom serde serialization logic for identifiers and adding \ValueEnum\ support for CLI integration.

core/common/src/types/consumer · high confidence

The \core/common/src/types/message\ module has been restructured to introduce a new, explicit message wire format. This includes the \IggyMessage\ struct with defined payload and user-header size limits (64 MB and 100 KB respectively), and a 64-byte \IggyMessageHeader\ containing fields for checksum, ID, offset, timestamps, and header/payload lengths. To support efficient I/O, the update adds zero-copy view types (\IggyMessageView\, \IggyMessageHeaderView\, \IggyIndexView\) that allow reading message data directly from buffers without allocation. It also introduces \IggyIndexes\ for binary-encoded message indexing with timestamp-based binary search, and \IggyMessagesBatch\ to manage collections of messages with associated index data.

core/common/src/types/message · high confidence

The partitions module has been reworked to introduce a new storage layer and offset management system. This includes a new sparse index format (24-byte entries for offset, timestamp, and position) with dedicated readers and writers, replacing the legacy dense index. Consumer offset tracking is now handled by a new \DurableConsumerOffsets\ and \ConsumerOffsetCapacity\ system that enforces limits and manages auto-commit reservations. The core \IggyPartition\ struct has been updated to use this new journal and storage infrastructure, and a new \install\_backup\ mechanism ensures durable, atomic updates to partition files during state transfers or materialization.

core/partitions · high confidence

The \core/metadata/src/permissioner\ module has been introduced to centralize role-based access control (RBAC) logic. This change adds a \Permissioner\ struct that manages user permissions and implements specific authorization rules for streams, topics, partitions, segments, consumer groups, consumer offsets, and system operations. The module enforces permission inheritance (e.g., \manage\_streams\ implies \read\_streams\) and checks both global and stream-specific permissions, replacing previous scattered permission checks with a unified, auditable permissioning layer.

core/metadata/src/permissioner · high confidence

The \core/server\_common/src/segment\_storage\ module now provides dedicated \IndexReader\, \IndexWriter\, \MessagesReader\, and \MessagesWriter\ structs, along with a \SegmentStorage\ container. This change introduces strict size validation when opening existing segment files: if the on-disk size of a messages or index file does not match the expected size, the open operation fails with \SegmentSizeMismatchAtOpen\ rather than allowing potentially corrupt or stale data to be appended. The storage layer also supports preallocation for new message files and ensures file system synchronization (fsync) upon opening existing segments, providing a more robust foundation for segment lifecycle management.

_core/server\_common/src/segment\storage · high confidence

The Partition struct in the core common types has been updated to use strongly-typed wrapper types for its time and size data: \created\_at\ is now an \IggyTimestamp\ instead of a raw \u64\, and \size\_bytes\ is now an \IggyByteSize\ instead of a raw \u64\. This change improves type safety and consistency for users interacting with partition information, ensuring that timestamp and byte-size values are handled through dedicated types rather than primitive integers.

core/common/src/types/partition · high confidence

The permissions module has been reorganized under \core/common/src/types/permissions\, moving global and personal access token definitions into a shared common location. The \Permissions\ struct now uses \BTreeMap\ instead of \HashMap\ for stream and topic collections to ensure deterministic ordering. Documentation and internal logic have been updated to reflect that \read\_topics\ and \read\_topic\ permissions now include consumer group management capabilities (create, join, leave, get). Additionally, the \PersonalAccessTokenInfo\ expiry field has been renamed to \expiry\_at\ and changed from a raw \u64\ to an \IggyTimestamp\ type, and the \GlobalPermissions\ struct now implements \Display\ via a formatted table for better readability.

core/common/src/types/permissions · high confidence

The QUIC client configuration module has been reorganized under \core/common/src/types/configuration/quic\_config\, introducing a builder pattern (\QuicClientConfigBuilder\) that validates and trims the server address upon build. The configuration structure now groups reconnection settings into a dedicated \QuicClientReconnectionConfig\ (supporting max retries, interval, and re-establish delay) and adds a \heartbeat\_interval\ field. Additionally, the module provides \QuicConnectionStringOptions\ to parse connection-string parameters for these settings and implements \From\<ConnectionString\>\ to convert parsed connection strings into the new configuration struct.

_core/common/src/types/configuration/quic\config · high confidence

The \streaming\ module's core implementation files—including \index.rs\, \lib.rs\, \message.rs\, \partition.rs\, \segment.rs\, \serialization.rs\, \stream.rs\, \stream\_error.rs\, \system.rs\, \timestamp.rs\, and \topic.rs\—have been deleted. This removes the previous monolithic definitions for streams, topics, partitions, segments, and message handling from this location, indicating a structural refactor or migration to a different module or implementation.

streaming · high confidence

The Rust SDK has been restructured into smaller packages, moving core types to the common module. This change introduces a new \DiagnosticEvent\ enum to track client lifecycle states (shutdown, disconnected, connected, signed\_in, signed\_out) and relocates \ClientInfo\ types. Additionally, the \ConsumerGroupInfo\ struct now uses \group\_id\ instead of \consumer\_group\_id\ for the consumer group identifier.

core/common/src/types/client, core/common/src/types/diagnostic · high confidence

The system statistics endpoint now reports resource usage scoped to the process's actual constraints rather than the host-wide totals. When running inside a container or cgroup with CPU pinning or memory caps, the reported total CPU usage reflects only the allowed cores, and memory figures account for the effective cgroup limit (including reclaimable file cache) instead of the full host memory. If no such limits are detected, the behavior falls back to the previous host-wide reporting.

_core/system\stats · high confidence

The configuration module now centralizes transport protocol definitions in a new \TransportProtocol\ enum, supporting TCP, QUIC, HTTP, and WebSocket. This change allows users to specify the transport protocol in configuration files using either human-readable strings (e.g., "tcp", "quic") or numeric codes (1–4), with custom serialization logic ensuring compatibility across JSON, TOML, and HTTP interfaces.

core/common/src/types/configuration · high confidence

The user type definitions have been reorganized into a dedicated module, introducing an extensible \options\ field to both \UserInfo\ and \UserInfoDetails\ that allows clients to store custom key-value data. The \IdentityInfo\ structure has been simplified to expose only the \access\_token\ directly, removing the nested \IdentityTokens\ wrapper. Additionally, the \created\_at\ timestamp is now represented by the \IggyTimestamp\ type instead of a raw \u64\, and error handling in \UserStatus\ has been updated to use the \IggyError\ type.

core/common/src/types/user · high confidence

Add C++ BDD test harness with Dockerized wire server; Add C++ SDK BDD test suite for messaging and raw commands; Add PHP BDD test suite for basic messaging and raw commands; Added BDD test scenarios for basic messaging, stream CRUD, raw commands, and leader redirection; Added Go BDD test suite for messaging, clustering, and raw commands; Added Python BDD test suite for basic messaging and raw commands; Added Rust BDD test runners for basic messaging, stream CRUD, and leader redirection; Added integration tests for FlatBuffers and Protobuf connector support; Added simulator tests for storage purge and crash recovery; Added tests for module graph integrity and server bootstrap; Go SDK BDD test coverage for consumer groups, offsets, and partitions; Java BDD test suite with Gradle wrapper and Cucumber integration; New Rust BDD test suite for cluster, messaging, and resource management.

A dependency / build maintenance change in (dependencies) — 1189 commits (170 fixs), 122 files.

(dependencies) · high confidence · unverified

The Python BDD test environment now uses the \uv\ package manager for dependency resolution and installation, replacing the previous tooling. The environment is explicitly pinned to Python 3.10 via a new \.python-version\ file, and the Dockerfile has been updated to install \uv\ and use it for syncing dependencies. A new \pylock.toml\ file provides a deterministic, hashed lockfile for all Python dependencies, ensuring reproducible test runs.

bdd/python · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 52 → 55 (+2.5)
• Rubric changed (rubric-2026.09.11 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 65 → 65 (+0.7)
• Architecture 96 → 98 (+2.2)
• Maturity 77 → 76 (-0.3)
• Readiness 43 → 47 (+3.1)
• Security 44 → 55 (+11.0)
• Event Sourcing 100 → 100 (+0.0)
• Performance 70 → 57 (-13.0)

• Change coupling: metadata.rs ↔ iggy_partitions.rs (core/metadata/src/impls/metadata.rs)
• Change coupling: plane_helpers.rs ↔ lib.rs (core/consensus/src/plane_helpers.rs)
• DiskReadPlan::read_disk (cyclomatic 20) (core/partitions/src/poll_plan.rs)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (10 lines × 2) (core/connectors/sources/postgres_source/src/lib.rs)
• Duplicated block (10 lines × 2) (core/partitions/src/iggy_partition.rs)
• Duplicated block (10 lines × 3) (core/server/src/segment_recovery.rs)
• Duplicated block (10–13 lines × 2) (core/consensus/src/plane_helpers.rs)
• Duplicated block (11 lines × 2) (core/binary_protocol/src/requests/messages/flush_unsaved_buffer.rs)
• Duplicated block (11 lines × 2) (core/consensus/src/plane_helpers.rs)
• Duplicated block (11 lines × 2) (core/server/src/partition_helpers.rs)
• Duplicated block (11 lines × 3) (core/server/src/partition_helpers.rs)
• Duplicated block (14 lines × 2) (core/integration/src/harness/seeds.rs)
• Duplicated block (14 lines × 2) (core/message_bus/src/transports/tcp_tls.rs)
• Duplicated block (14 lines × 2) (core/sdk/src/quic/quic_client.rs)
• Duplicated block (15 lines × 2) (core/integration/src/harness/handle/connectors_runtime.rs)
• Duplicated block (15 lines × 2) (core/server/src/dispatch/partition.rs)
• Duplicated block (17 lines × 2) (core/message_bus/src/transports/wss.rs)
• Duplicated block (20 lines × 2) (core/connectors/sinks/postgres_sink/src/lib.rs)
• …and 80 more

• Ambiguous naming for similar operations. ReceiveAsync returns ReceivedMessage (which wraps MessageResponse), while ReceiveRentedAsync returns ReceivedRentedMessage (which wraps RentedMessageResponse). The distinction between 'Message' and 'RentedMessage' is not immediately obvious to a user and suggests a potential naming inconsistency regarding memory ownership semantics.
• Change coupling: iggy_partition.rs ↔ metrics.rs (core/partitions/src/iggy_partition.rs)
• CommandResponseStream._pollOnPrimary (cognitive 86) (foreign/node/src/client/client.socket.ts)
• CommandResponseStream._pollOnPrimary (cyclomatic 40) (foreign/node/src/client/client.socket.ts)
• CommandResponseStream._rememberCredentials (cognitive 18) (foreign/node/src/client/client.socket.ts)
• DiskReadPlan::walk_segment (cognitive 23) (core/partitions/src/poll_plan.rs)
• Documentation: no architecture or design documentation (core/server/README.md)
• Duplicate intent across generic and non-generic builders. The non-generic IggyConsumerBuilder and generic IggyConsumerBuilder<T> have nearly identical Create signatures, differing only by the presence of a deserializer. This forces users to choose between two parallel builder hierarchies for essentially the same operation.
• Duplicated block (10 lines × 2) (core/connectors/sources/postgres_source/src/lib.rs)
• Duplicated block (10 lines × 2) (core/partitions/src/partition_storage.rs)
• Duplicated block (10 lines × 2) (examples/csharp/src/Basic/Iggy_SDK.Examples.Basic.Consumer/Settings.cs)
• Duplicated block (10 lines × 2) (examples/python/high-level/background_producer.py)
• Duplicated block (10 lines × 2) (foreign/java/bench/src/main/java/org/apache/iggy/bench/report/FinalReportBuilder.java)
• Duplicated block (10 lines × 3) (core/partitions/src/segment_recovery.rs)
• Duplicated block (11 lines × 2) (core/partitions/src/partition_storage.rs)
• Duplicated block (11 lines × 2) (examples/java/src/main/java/org/apache/iggy/examples/multitenant/consumer/MultiTenantConsumer.java)
• Duplicated block (11 lines × 2) (examples/java/src/main/java/org/apache/iggy/examples/multitenant/consumer/MultiTenantConsumer.java)
• Duplicated block (11 lines × 2) (examples/python/basic/consumer.py)
• Duplicated block (11 lines × 2) (foreign/go/contracts/node_status.go)
• Duplicated block (11 lines × 3) (core/partitions/src/partition_storage.rs)
• …and 281 more

• 87 commits — 63 feature/other, 24 fixes

• gateways/kafka — 13 commits
• .github/workflows — 8 commits
• core/connectors — 7 commits
• core/server — 6 commits
• core/integration — 5 commits
• (root) — 4 commits
• core/partitions — 4 commits
• core/message_bus — 3 commits
• foreign/cpp — 3 commits
• foreign/csharp — 3 commits
• foreign/go — 3 commits
• foreign/java — 3 commits
• foreign/node — 3 commits
• foreign/swift — 3 commits
• web/package-lock.json — 3 commits
• .github/review-bot — 2 commits
• bdd/go — 2 commits
• core/configs — 2 commits
• examples/python — 2 commits
• foreign/python — 2 commits

• fix: fix(ci): don't mark PRs stale while they wait on review (#4172)
• fix: fix(ci): edit the Codecov PR comment once per run, not per test leg (#4326)
• fix: fix(ci): let committers with private org membership run PR commands (#4319)
• fix: fix(ci): retry flaky downloads and stop dry runs promising new tags (#4216)
• fix: fix(cluster): preserve group membership and reduce produce/poll overhead (#4169)
• fix: fix(cluster): prevent spurious heartbeat elections (#4194)
• fix: fix(cluster): stop partition repair asking for an inverted op range (#4246)
• fix: fix(configs): check server env vars once and refuse boot only in debug (#4200)
• fix: fix(connectors): defer postgres source progress until ack (#3957)
• fix: fix(connectors): defer source checkpoints and surface sink failures (#4153)
• fix: fix(connectors): disable SELinux label checks for WireMock test containers (#4265)
• fix: fix(connectors): meilisearch_sink URL scheme check is case-sensitive (#4158)
• fix: fix(connectors): refuse http_source bodies Iggy cannot store (#4302)
• fix: fix(connectors): tag sink batches with the payload's schema (#4204)
• fix: fix(connectors): validate sink writes and startup readiness (#4154)
• fix: fix(node): stop dropping and corrupting tokens in list responses (#4171)
• fix: fix(partitions): account for segment-backed append batches (#4202)
• fix: fix(partitions): handle clippy with poll diagnostics disabled (#4203)
• fix: fix(partitions): reject queued offsets after history resets (#4325)
• fix: fix(partitions): sync checkpoints through original writers (#4253)
• …and 67 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

apache/iggy was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.