Skip to content
CAI
Software that uses CAICheck a score

fluvio-community/fluvio

61.0

Adequate · 9 November 2026

97.8k

lines of production code

Rust

primary language

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

This system is Fluvio, a distributed streaming data platform that manages data ingestion, storage, and processing through a cluster architecture of Storage Controllers and Processing Units. It provides a Rust-based client library and CLI for producing and consuming records, supporting features like compression, smart module transformations, and consumer offset management. The platform also includes a Connector Development Kit for building data integrations and a SmartModule Development Kit for writing custom data processing logic, all managed via a version manager and extensive CI/CD tooling.

How it got here

2019–2021 — Platform architecture rewrite and SDK introduction

137 changes.

The project underwent a comprehensive architectural overhaul, stripping out legacy async runtimes, Kafka compatibility layers, and monolithic server components to establish a modular, Rust-native core. This period focused on rebuilding the Streaming Controller and Processing Unit with new protocol definitions, Kubernetes operators, and a unified public API. Simultaneously, it introduced the Connector and SmartModule Development Kits to standardize extensibility and significantly expanded the test harness and developer tooling.

2022–2023 — Connector SDK and CLI expansion

62 changes.

This period focused on establishing a comprehensive Connector Development Kit (CDK) and SmartModule ecosystem, introducing new crates for connector packaging, deployment, and derive macros alongside extensive example libraries. The Fluvio CLI was significantly restructured to support granular management of topics, partitions, SmartModules, and consumer offsets, while the release infrastructure was modernized with modular Makefiles and Rust-based validation tools.

2024–2026 — cluster mirroring and consumer offset management

19 changes.

This period focused on implementing comprehensive cluster mirroring capabilities, enabling data replication between home and edge clusters via new SC and SPU controllers. Concurrently, the codebase introduced robust consumer offset persistence and management strategies, supported by a new key-value storage layer and extensive test coverage.

Features

Add CLI commands to manage SPU Groups

Users can now create, delete, and list managed SPU Groups via the Fluvio CLI. The new \create\ command accepts options for the group name, number of replicas, minimum SPU ID, rack assignment, and storage size. The \list\ command displays SPU Group details in a formatted table (using comfy-table) or other output formats, showing name, replicas, min ID, rack, size, and status. The \delete\ command removes a specified SPU Group by name.

crates/fluvio-cluster/src/cli/group · high confidence

Add CLI commands to manage TableFormat specifications

The Fluvio CLI now supports creating, listing, and deleting TableFormat display specifications. Users can create a new TableFormat by providing a YAML configuration file via the \fluvio tableformat create\ command, list existing formats with \fluvio tableformat list\ (supporting standard output formats), and remove them using \fluvio tableformat delete\. This change introduces the \TableFormatCmd\ abstraction and its subcommands within the CLI client module.

crates/fluvio-cli/src/client/tableformat · high confidence

Add CLI commands to manage release channels

Users can now create, switch between, list, and delete release channels (stable, latest, dev) via the Fluvio CLI. The new \create\ command allows configuring channel-specific binary and extension paths, while \switch\ automatically installs the corresponding Fluvio binary for the selected channel if it is not already present. The \list\ command displays available channels with the active one marked, and \delete\ removes a channel's configuration along with its associated binary and extension files.

crates/fluvio-channel-cli/src/cli · high confidence

Add CLI library for installing Fluvio binaries via channels

The \fluvio-channel-cli\ crate introduces a new library component that enables the Fluvio CLI to install or update its binary based on a specified channel configuration. This addition allows users to fetch and install the Fluvio binary from a registry, supporting stable, latest, and specific tag versions, while handling channel-specific binary locations and error states such as missing channel info or unpublished versions.

crates/fluvio-channel-cli/src · high confidence

Add JSON test connector for generating configurable JSON records

A new JSON test connector has been added to the platform, allowing users to produce JSON records to a Fluvio topic using a configurable template and interval. The connector is defined as a source that reads a JSON template string and an interval (in seconds) from its custom configuration, emitting the templated JSON at the specified rate. Users can deploy this connector using the provided Connector.toml metadata and sample configuration files (sample-config.yaml and sample-config-v2.yaml), which demonstrate how to set the template, interval, and optional timeout parameters for testing data ingestion pipelines.

connector/json-test-connector · high confidence

Add Rust produce example demonstrating keyless record sending

A new Rust example (examples/00-produce) has been added to demonstrate producing messages to a Fluvio topic. The example shows how to initialize a producer, send a record with a NULL key using the RecordKey API, and flush the producer, providing a minimal starting point for users integrating with Fluvio via the Rust client.

examples/00-produce · high confidence

Add \`MyTestCase\` derive macro for test case definitions

The \fluvio-test-case-derive\ crate now provides a \\#\[derive(MyTestCase)\]\ macro that automatically generates a test case struct and implements the necessary \TestOption\ trait for the annotated struct. This allows users to define custom test options with minimal boilerplate, as the macro handles the conversion and metadata registration required by the \fluvio\_test\_util\ framework. The change includes UI tests to verify correct compilation for valid structs and failure for invalid inputs like enums.

crates/fluvio-test-case-derive · high confidence

Add build configuration and documentation for fluvio-spu

The fluvio-spu crate now includes a build script that embeds the current git commit hash into the binary at compile time, along with an Apache 2.0 license file, a Makefile for local development testing, and a README explaining how to run the SPU in development mode.

crates/fluvio-spu · high confidence

Add example of custom string-length-based partitioning

The \examples/partitioning-simple\ crate now provides a \AlphabetPartitioning\ implementation that maps message keys to partitions based on the character count of the key string (modulo the partition count). This example demonstrates how to implement the \Partitioner\ trait to create a custom partitioning strategy for Fluvio producers.

examples/partitioning-simple · high confidence

Add explicit cluster type detection and standardized reset/start scripts for local Kubernetes environments

The k8-util/cluster directory now includes a new cluster-type.sh script that automatically detects the local Kubernetes cluster type (minikube, k3d, kind, microk8s, or lima) by inspecting node names, alongside dedicated reset and start scripts for each supported environment. These scripts standardize the setup process, with minikube reset/start scripts now defaulting to Kubernetes v1.26.3 and using hyperkit as the driver on macOS, while kind and k3d scripts provide clean cluster recreation workflows. This ensures consistent cluster initialization and teardown across different local development environments.

k8-util/cluster · high confidence

Add filter\_odd\_key smartmodule example

A new example smartmodule named filter\_odd\_key has been added to the examples directory. This filter processes records by parsing their key as an integer and allows only records with even keys to pass through, demonstrating how to handle key-value pairs and custom error wrapping in a filter smartmodule.

_smartmodule/examples/filter\_odd\key · high confidence

Add fluvio-service crate with TCP server and request handling macros

The new \fluvio-service\ crate introduces the core server infrastructure for handling Fluvio API requests. It provides the \FluvioApiServer\ to accept incoming TCP connections and dispatch them to service implementations via the \FluvioService\ trait. The crate also includes \call\_service\, \api\_loop\, and \wait\_for\_request\ macros to standardize request processing, ensuring that incoming requests are served even if the write socket is closed, and includes a test module with example request/response types to validate the server logic.

crates/fluvio-service · high confidence

Add partition listing command to CLI

Users can now list all partitions in the cluster using the new \fluvio partition list\ command. This feature displays partition details including topic, leader, mirror status, replicas, resolution, size, base offset, high watermark, and follower offsets, with support for filtering system partitions and output formatting.

crates/fluvio-cli/src/client/partition · high confidence

Add profile sync command for local and Kubernetes contexts

The CLI now includes a new \sync\ command that allows users to automatically configure their local Fluvio profile to connect to either a local cluster or a Kubernetes-deployed Fluvio cluster. The \sync local\ subcommand sets the profile to point at a local endpoint (defaulting to localhost:9003), while the \sync k8\ subcommand discovers the Fluvio service address in a Kubernetes cluster and updates the profile accordingly, handling TLS configuration and profile creation as needed.

crates/fluvio-cli/src/profile/sync · high confidence

Add public API handlers for creating and deleting TableFormat resources

The service control component now exposes public API endpoints to manage TableFormat definitions. New handlers in the \tableformat\ module allow users to create and delete TableFormat resources, enforcing authorization checks for create and delete actions respectively, and returning appropriate status codes for existing resources, permission denials, or not-found errors.

_crates/fluvio-sc/src/services/public\api/tableformat · high confidence

Add sink-test-connector SDK example with custom config and stream lifetime testing

A new test sink connector has been added to the SDK distribution, providing a concrete example of how to implement a sink connector using the \ConsumerStream\ trait and custom configuration. The connector demonstrates support for secret resolution (via environment variables or files) and string interpolation in configuration values. It also includes a specific test mode (\test\_stream\_lifetime\) designed to verify that the connector SDK correctly handles stream shutdown and resource cleanup, addressing previous issues where streams could remain stuck in a state preventing proper dropping.

connector/sink-test-connector · high confidence

Add storage CLI utility for log, index, and segment validation

A new command-line interface (CLI) tool is introduced to help users inspect and validate Fluvio storage components. Users can now run commands to dump log batches with optional offset filtering, inspect index entries, validate segment integrity, and retrieve replica information. This tool leverages the Clap library for argument parsing and provides asynchronous operations for efficient handling of storage files.

crates/fluvio-storage/src/bin · high confidence

Added Cargo template for generating SmartModules

A new Cargo-based template has been added to the \smartmodule/cargo\_template\ directory, providing starter code for creating Fluvio SmartModules. The template supports multiple SmartModule types—filter, map, filter-map, array-map, and aggregate—each with a corresponding function signature and example logic. It also includes an initialization block for handling extra parameters, specifically demonstrating how to parse a 'key' parameter for filter modules using \OnceLock\ and \SmartModuleExtraParams\. This allows users to generate new SmartModule projects with a standardized, type-safe structure.

_smartmodule/cargo\template · high confidence

Added Kubernetes dashboard installation instructions and configuration files

Users can now install and access the Kubernetes dashboard in a Kind cluster by applying the provided metric-server, user, and token scripts. The new metric-server.yaml configures the metrics server with insecure TLS to work with Kind, while user.yaml creates an admin ServiceAccount with cluster-admin privileges, and token.sh helps retrieve the authentication token for login.

k8-util/dashboard · high confidence

Added SmartModule example library

The repository now includes a new \smartmodule/examples\ library containing Rust source code for various SmartModule patterns, including map, filter, aggregate, and array\_map operations. These examples demonstrate how to implement specific data transformations such as incremental averaging, JSON aggregation, string filtering, and regex-based mapping within the Fluvio SmartModule framework.

(repo-wide) · high confidence

Added TLS certificate generation and management tooling

Developers can now use the new tls/ directory to generate and manage TLS certificates for local and Kubernetes development environments. The included Makefile provides targets to generate a Certificate Authority (CA), server, and client certificates (root and user1), and to create Kubernetes secrets from these certificates. This simplifies setting up secure connections for Fluvio clusters in development.

tls · high confidence

Added development certificate generation tools and test infrastructure

The \crates/fluvio-socket/certs\ directory now includes a Makefile, OpenSSL configuration files, and pre-generated CA, server, and client certificates to streamline local development and testing. This setup allows developers to easily regenerate TLS credentials and run an Nginx-based test proxy for validating SSL/TLS connections. Additionally, the \crates/fluvio-sc/src/services\ module has been reorganized to explicitly expose public and internal server start functions, clarifying the service entry points.

crates/fluvio-sc/src/services, crates/fluvio-socket/certs · high confidence

Added example smart module for filtering records by parameter

A new example smart module has been added at \smartmodule/examples/filter\_with\_param\_v1\ that demonstrates how to implement a filter using parameters. This module defines a \FilterOpt\ struct with a \key\ field (defaulting to "a") and applies it to filter records based on whether their value contains the specified key string.

_smartmodule/examples/filter\_with\_param\v1 · high confidence

Added fluvio-test-util crate with TLS helpers and version metadata

The new \fluvio-test-util\ crate provides utilities for testing Fluvio components, including TLS certificate management and version string handling. It exposes modules for setup, test running, and TLS configuration, allowing tests to load client and server certificates from a local \tls/certs\ directory and construct \TlsPolicy\ objects. The crate also includes a \VERSION\ constant that reads the project version and optionally appends a suffix defined by the \FLUVIO\_VERSION\_SUFFIX\ environment variable.

crates/fluvio-test-util · high confidence

Added topic-level deduplication smartmodule example

A new example smartmodule is available at \smartmodule/examples/filter\_hashset\ that implements topic-level deduplication. It maintains a bounded set of seen record values in memory, allowing users to filter out duplicate records based on their content. The module supports configuration via an initialization parameter to set the maximum size of the deduplication cache.

_smartmodule/examples/filter\hashset · high confidence

Automated Docker image build and distribution for Fluvio runner

Added a new build script and Dockerfile to streamline the creation of the Fluvio runner container image. The build process now supports cross-compilation for ARM64 architectures and automatically loads the resulting image into various Kubernetes environments, including Minikube, k3d, Kind, and MicroK8s, ensuring the image is ready for immediate use in these clusters.

k8-util/docker · high confidence

Client metrics tracking for consumer and producer activity

The Fluvio client now exposes a \ClientMetrics\ struct that tracks record and byte counts for consumer operations and producer connections. This allows users to monitor data throughput and activity levels directly from the client side, with support for serializing and deserializing these metrics for external reporting or debugging.

crates/fluvio/src/metrics · high confidence

Connector config templates now support string interpolation with secrets

The connector package renderer now uses MiniJinja to allow string interpolation in connector configuration files. Users can inject secret values into their configs using the \${{ secrets.\<key\> }}\ syntax, and the renderer enforces strict mode to fail clearly if an undefined variable is referenced. This change also introduces a custom template syntax with block, variable, and comment delimiters, and upgrades the underlying MiniJinja library to version 2.6.0.

crates/fluvio-connector-package/src/render · high confidence

Consumer offset persistence in SPU

The SPU now persists consumer offsets to local storage, allowing consumers to resume from their last committed position after restarts. This change introduces a new key-value storage layer in the SPU's KV module to track and manage consumer offset data.

crates/fluvio-spu/src/kv · high confidence

FVM core infrastructure for version management and self-updates

The Fluvio Version Manager now includes the foundational components required to install, switch, and update Fluvio versions. This change introduces a structured storage system using \settings.toml\ to track the active channel and version, and \manifest.json\ files to record artifact details for each installed version. It adds logic to download and store binaries in \\~/.fvm/versions\, manage executable permissions on Unix systems, and handle the self-update process by downloading, verifying SHA-256 digests, and replacing the FVM binary. Additionally, it provides a colored notification system for user feedback during these operations.

crates/fluvio-version-manager/src/common · high confidence

FVM self-management commands (install, uninstall, update)

The Fluvio Version Manager now includes a \self\ subcommand group that allows users to manage the FVM binary itself. The \install\ command copies the current binary to the \\~/.fvm/bin\ directory and sets up the workspace environment. The \uninstall\ command removes the FVM installation directory after confirmation. The \update\ command checks for the latest stable release on GitHub and updates the FVM binary if a newer version is available, supporting an optional \FVM\_UPDATE\_VERSION\ environment variable for specific version targeting.

crates/fluvio-version-manager/src/command/itself · high confidence

Fluvio Run now embeds the current git commit hash at compile time

The build process for the fluvio-run crate has been updated to automatically capture the current git commit hash and expose it as a compile-time environment variable. This allows the application to report its specific build version (commit ID) in its output, aiding in debugging and version tracking, while safely handling cases where the VERSION file might not be present (such as during cargo publish).

crates/fluvio-run · high confidence

Fluvio Version Manager (FVM) artifact retrieval and download

The \fluvio-artifacts-util\ crate now includes a new FVM module that fetches and downloads Fluvio binaries (fluvio, fluvio-run, cdk, smdk) directly from GitHub releases. It resolves versions for stable, latest, and specific tags, downloads the corresponding architecture-specific zip archives, validates their SHA-256 checksums, and extracts the binaries to the target directory.

crates/fluvio-artifacts-util · high confidence

Fluvio crate initialization with documentation and build scaffolding

The \crates/fluvio\ directory has been added, establishing the root Rust client crate for the Fluvio platform. This includes a comprehensive README with getting-started examples for producing and consuming records, an Apache 2.0 license file, and a build script that manages version information. This change provides the foundational structure and public documentation for the client library.

crates/fluvio · high confidence

Home-to-edge mirroring connection handling

The service controller now supports home-to-edge mirroring by introducing a new connection handler that accepts remote mirroring requests. This component validates the remote cluster's authorization, monitors local topic and SPU changes, and synchronizes mirror status updates back to the remote edge cluster, enabling data replication from the home cluster to edge locations.

_crates/fluvio-sc/src/services/public\api/mirroring · high confidence

Initial release of fluvio-sys Helm chart

A new Helm chart named 'fluvio-sys' (version 0.9.20) has been added to the k8-util/helm directory. This chart, identified as an application type, provides the initial structure for deploying Fluvio system components, including a .helmignore file to exclude common development artifacts and a values.yaml file for default configuration.

k8-util/helm/fluvio-sys · high confidence

Initial release of the SPU schema crate

The \crates/fluvio-spu-schema\ crate has been added to the repository, establishing the protocol definitions for the Storage Processing Unit (SPU). This includes the schema for client-to-server requests (such as \ReplicaOffsetUpdateRequest\) and server-side API endpoints (including \ProduceRequest\, \FetchOffsetsRequest\, and \UpdateOffsetsRequest\). The crate also defines the data structures for fetch and produce operations, including support for transactional isolation levels and smart module invocations, along with standard project documentation and licensing files.

crates/fluvio-spu-schema · high confidence

Internal SPU API for consumer offset management

The SPU now exposes an internal API service that allows peer SPUs to fetch and update consumer offsets. This change introduces new request/response types (\FetchConsumerOffsetRequest\, \UpdateConsumerOffsetRequest\) and their corresponding handlers, enabling the SPU to store and retrieve consumer offset data via a dedicated internal communication channel.

crates/fluvio-spu/src/services/internal · high confidence

Introduce Connector Development Kit (CDK) and SmartModule Development Kit (SMDK)

This change adds the Connector Development Kit (CDK) and SmartModule Development Kit (SMDK) to the Fluvio platform, providing developers with standardized tools to create, build, and publish connectors and smart modules. The diff introduces new crates for the CDK (\crates/cdk\), connector packaging (\fluvio-connector-package\), and connector deployment (\fluvio-connector-deployer\), alongside test connectors (\json-test-connector\, \sink-test-connector\) and the SMDK infrastructure. These additions enable a streamlined workflow for extending Fluvio with custom data processing logic and external system integrations.

(dependencies) · high confidence

Introduce Fluvio Version Manager (FVM) CLI

The Fluvio Version Manager (FVM) CLI is now available, providing commands to manage multiple Fluvio versions. Users can install specific versions or channels (stable, latest) using \fvm install\, switch the active version with \fvm switch\, and view installed versions with \fvm list\ (which can also show included artifacts for a specific version). The tool supports updating the current channel to the most recent release via \fvm update\, uninstalling versions with \fvm uninstall\, and checking the currently active version with \fvm current\. Additionally, \fvm version\ displays detailed CLI and system information, and \fvm self\ manages FVM itself.

crates/fluvio-version-manager/src/command · high confidence

Introduce Fluvio channel configuration management

Added a new \fluvio-channel\ crate that provides the core logic for managing Fluvio release channels (such as stable, dev, and latest). This component handles reading and writing channel configurations from the user's home directory, allowing users to switch between different release tracks and manage associated binary and extension paths via a TOML-based config file.

crates/fluvio-channel · high confidence

Introduce Fluvio test harness with environment customization and process forking

The \fluvio-test-derive\ crate is introduced, providing a \\#\[fluvio\_test\]\ procedural macro that allows test writers to define minimum cluster requirements (such as \min\_spu\, \topic\_name\, \timeout\, and \cluster\_type\). The macro automatically generates test drivers that validate these requirements, customize the test environment, and create topics. To prevent async blocking issues in child processes, the macro utilizes a \fork\_and\_wait\ mechanism for setup tasks like topic creation, ensuring stable execution for both synchronous and asynchronous test cases.

crates/fluvio-test-derive/src · high confidence

Introduce SPU-to-SPU mirroring protocol and controller

Adds the remote-side implementation for SPU-to-SPU mirroring, including a new \MirrorRemoteToHomeController\ that manages the connection from a remote SPU to the home cluster. This change introduces the \MirrorRemoteApiEnum\ protocol definitions (\SyncRecords\, \UpdateEdgeOffset\) and the corresponding request/response types (\MirrorPartitionSyncRequest\, \UpdateRemoteOffsetRequest\) required for the remote SPU to synchronize records and update edge offsets with the home cluster.

crates/fluvio-spu/src/mirroring/remote · high confidence

Introduce TestDriver for Fluvio test execution

Added a new TestDriver component in the test utility crate to manage test cluster connections and provide standardized wrappers for creating Fluvio producers and consumers. This change enables tests to easily initialize clients, configure topic producers with specific settings, and manage consumer streams with defined offsets, centralizing the setup logic required for running integration tests against a Fluvio cluster.

_crates/fluvio-test-util/test\_runner/test\driver · high confidence

Introduce connector common library with unified configuration and I/O abstractions

This change introduces the \fluvio-connector-common\ crate, providing a shared foundation for connector development. It adds a configuration parser (\config.rs\) that handles YAML-based connector settings, including support for custom config roots and meta-data. The library implements standardized consumer and producer initialization (\consumer.rs\, \producer.rs\) that automatically ensures target topics exist and applies connector-specific settings like partition strategies, offset management, and smartmodule chains. Additionally, it includes a monitoring module (\monitoring.rs\) that exposes connector and smartmodule metrics via a Unix socket, and a smartmodule helper (\smartmodule.rs\) to build transformation chains from configuration.

crates/fluvio-connector-common/src · high confidence

Introduce connector package metadata, configuration, and secret handling

The \fluvio-connector-package\ crate is introduced, providing the core data structures and logic for defining connector packages. This includes \ConnectorMetadata\ for package details (name, version, visibility, direction, deployment image/binary) and \ConnectorConfig\ for validation. A new secret management system is added, supporting both plain strings and resolved secrets via a pluggable \SecretStore\ (defaulting to environment variables or file-based stores), with sensitive values masked in debug output.

crates/fluvio-connector-package/src · high confidence

Introduce core type definitions and configuration utilities

This change establishes the \fluvio-types\ crate, providing foundational types and utilities for the Fluvio system. It introduces a \Compression\ enum (supporting None, Gzip, Snappy, Lz4, and Zstd) for record compression, a \SaveLoadConfig\ trait for serializing and deserializing configuration files to/from TOML, and a set of default constants for SPU and SC server paths, ports, and storage limits. Additionally, it includes an event handling module with \StickyEvent\ and \OffsetPublisher\ for asynchronous state notifications, and utility functions for partition name parsing.

crates/fluvio-types · high confidence

Introduce derive macro for source and sink connectors

The \fluvio-connector-derive\ crate now provides a \\#\[connector\]\ procedural macro that simplifies building connectors. Users can annotate an async function with \\#\[connector(source)\]\ or \\#\[connector(sink)\]\ to automatically generate the \main\ entry point, including configuration parsing, secret management (file or environment), and graceful shutdown handling. Additionally, structs annotated with \\#\[connector(config)\]\ are supported to define custom configuration schemas, which are validated against reserved names like 'meta' and 'transforms'.

crates/fluvio-connector-derive · high confidence

Introduce environment drivers for local and Kubernetes cluster setup

The test utility now provides a unified interface for setting up test clusters in both local and Kubernetes environments. A new \EnvironmentDriver\ trait defines common operations like starting and removing clusters, implemented by \LocalEnvDriver\ for local setups and \K8EnvironmentDriver\ for Kubernetes. The \TestEnvironmentDriver\ enum wraps these implementations, allowing the test harness to abstract away the underlying infrastructure details when initializing the cluster for tests.

crates/fluvio-test-util/setup/environment · high confidence

Introduce fluvio-auth crate for X.509 authorization

The new \fluvio-auth\ crate provides the core authorization framework for Fluvio, defining the \Authorization\ and \AuthContext\ traits that govern type-level and instance-level access checks. It includes a \RootAuthorization\ implementation that permits all actions, serving as a permissive baseline, and an \X509Authenticator\ that extracts the principal from TLS client certificates and validates them against a scope-binding configuration file. The crate also defines the \AuthRequest\/\AuthResponse\ protocol messages used to communicate authorization decisions over the socket.

crates/fluvio-auth · high confidence

Introduce fluvio-channel CLI for managing release channels

Adds a new \fluvio-channel\ binary that provides a command-line interface for managing Fluvio release channels. Users can now create, delete, list, and switch between local release channels (such as stable or dev) using subcommands like \create\, \delete\, \list\, and \switch\. The tool acts as a frontend to support channel-based versioning, integrating with the existing Fluvio ecosystem to allow users to manage which channel's binaries are active.

crates/fluvio-channel-cli/src/bin · high confidence

Introduce fluvio-cli-common library for shared CLI infrastructure

The \fluvio-cli-common\ crate has been added to provide shared utilities for the Fluvio CLI. This includes HTTP helpers for fetching package data, installation logic for downloading and verifying smartmodule packages with checksums, and support for managing the Fluvio extensions directory. It also introduces a standardized way to handle user input for smartmodule testing, allowing records to be provided via text, file, or stdin, and exposes environment variables for release channels and update checks.

crates/fluvio-cli-common/src · high confidence

Introduce fluvio-compression library with Gzip, Snappy, LZ4, and Zstd support

The new \fluvio-compression\ crate provides a unified library for compressing and decompressing data within the Fluvio protocol. It introduces a \Compression\ enum supporting four codecs—Gzip, Snappy, LZ4, and Zstd—each gated behind its own Cargo feature flag (e.g., \gzip\, \snap\, \lz4\, \zstd\). The library exposes \compress\ and \uncompress\ methods on the enum, allowing producers to compress data and consumers/SPUs to decompress it based on batch attributes. It also includes error handling via \CompressionError\ and conversion logic from \fluvio\_types::compression::Compression\.

crates/fluvio-compression · high confidence

Introduce fluvio-hub-protocol crate for shared hub types and authentication

A new \fluvio-hub-protocol\ crate has been added to centralize reusable types and logic for the Fluvio Package Hub. It defines the \PackageMeta\ structure, which now supports \repository\_url\ and \tags\ fields, along with \PkgTag\ and \PkgVisibility\ enums. The crate also introduces a new authentication module (\infinyon\_tok\) that supports optional v4 multi-org access tokens alongside the existing v3 token format, providing a unified \read\_access\_token\ interface. Additionally, it exports common constants for hub configuration, package extensions, and metadata file names.

crates/fluvio-hub-protocol · high confidence

Introduce fluvio-package-index crate for package registry management

The \fluvio-package-index\ crate has been added to provide the core data structures and domain logic for reading the Fluvio Package Index. This library enables the CLI to install plugins (such as \fluvio-cloud\) and allows users to see new releases for the CLI and its plugins. It defines types for packages, releases, and targets, and includes an HTTP agent to fetch index metadata and package artifacts from the registry at \packages.fluvio.io\.

crates/fluvio-package-index · high confidence

Introduce fluvio-socket crate with versioned connections and multiplexing support

The new \fluvio-socket\ crate provides the foundational networking layer for Fluvio clients, introducing \VersionedSocket\ to automatically negotiate API versions with the server upon connection. It includes \MultiplexerSocket\ for handling multiple concurrent requests over a single connection, \StreamSocket\ for managing persistent streams, and a \ClientConfig\ for flexible connection setup. The module also exposes a \SocketError\ enum to clearly distinguish between I/O errors, closed sockets, and stale connections, ensuring more robust error handling for users interacting with the Fluvio cluster.

crates/fluvio-socket/src · high confidence

Introduce generic metadata dispatcher with local and Kubernetes backends

The stream dispatcher now supports a unified, generic architecture for managing metadata through a pluggable \MetadataClient\ trait. This change adds a \MetadataDispatcher\ that reconciles state from Kubernetes watch streams and a \MemoryDispatcher\ for local in-memory stores, both driven by a common \WSAction\ enum (Apply, UpdateSpec, UpdateStatus, Delete). It introduces a \LocalMetadataStorage\ implementation for local clusters, enabling cascade deletion and parent-child linking without requiring the Kubernetes flag, while retaining the existing K8 integration behind the \k8\ feature flag. Configuration for dispatcher wait times and reconciliation intervals is now exposed via environment variables (\FLV\_DISPATCHER\_WAIT\, \FLV\_SC\_RECONCILIATION\_INTERVAL\).

crates/fluvio-stream-dispatcher · high confidence

Introduce internal private API server for SPU communication

A new private API service has been added to the Storage Controller (SC) to handle internal communication with SmartProcessing Units (SPUs). This service exposes an internal endpoint where SPUs register and establish a persistent connection for health checks and state synchronization. The implementation includes a dispatch loop that listens for SPU requests—such as updates to logical replicas, partition status, and mirror statistics—and processes them, while also pushing local metadata changes (SPU specs, SmartModules, partitions, mirrors) to the connected SPU.

_crates/fluvio-sc/src/services/private\api · high confidence

Introduce local connector deployment capability

The connector deployer now supports deploying connectors locally by executing a specified binary with provided configuration and optional secrets files. Users can control the logging verbosity via a configurable log level (defaulting to Info) and choose to redirect output to a specific log file or inherit standard streams. This change establishes the foundational logic for running connector executables directly on the host machine.

crates/fluvio-connector-deployer · high confidence

Introduce local metadata store with epoch-based change tracking

The \fluvio-stream-model\ crate now provides a local metadata store (\LocalStore\) backed by a \DualEpochMap\ that tracks changes to spec, status, and metadata separately using epoch counters. This enables efficient, idempotent synchronization and change notification via an \EventPublisher\, allowing consumers to wait for and react to specific types of updates without polling.

crates/fluvio-stream-model · high confidence

Introduce local metadata synchronization for SPUs, partitions, and topics

The \crates/fluvio/src/sync\ module now provides a new \MetadataStores\ component that maintains a local cache of SPU, partition, and topic metadata. It establishes watch streams against the Service Controller to receive updates, which are processed by a \MetadataSyncController\ and stored in a \StoreContext\. This allows the client to resolve metadata locally with a configurable timeout (via \FLV\_METADATA\_TIMEOUT\) rather than relying solely on immediate remote queries.

crates/fluvio/src/sync · high confidence

Introduce local runtime module for managing Fluvio cluster processes

A new \local\ runtime module has been added to \crates/fluvio-cluster/src/runtime\, providing the infrastructure to launch and manage local Storage Controller (SC) and SPU processes. This includes the \ScProcess\ struct which supports starting the SC in local, read-only, or Kubernetes modes, and the \LocalSpuProcess\ struct for managing individual SPU instances. The module also defines the \FluvioLocalProcess\ trait for handling TLS configuration and the \LocalSpuProcessClusterManager\ for orchestrating multiple SPUs, effectively establishing the foundation for local cluster execution within the runtime layer.

crates/fluvio-cluster/src/runtime · high confidence

Introduce new fluvio-storage crate with core storage components

The \crates/fluvio-storage\ crate has been added, providing the foundational storage layer for Fluvio. This includes new modules for managing batch records (\batch.rs\, \batch\_header.rs\), file-based iteration (\file.rs\), memory-mapped segment indexing (\index.rs\), and checkpoint persistence (\checkpoint.rs\). It also introduces a background cleaner (\cleaner.rs\) that enforces partition size limits and retention policies, along with configuration structures (\config.rs\) and error handling (\error.rs\). A CLI tool for inspecting log files is also included via the Makefile and README.

crates/fluvio-storage/src · high confidence

Introduce standalone fluvio-benchmark tool for producer performance testing

A new \fluvio-benchmark\ crate has been added to provide a dedicated command-line tool for measuring producer throughput and latency. The tool supports running single producer benchmarks or executing matrix tests defined via YAML configuration files, allowing users to vary parameters such as batch size, record size, compression, and key allocation strategies. It automatically manages topic creation and cleanup, runs multiple producer workers concurrently, and outputs detailed results including per-second throughput, average and maximum latency, and percentile-based latency distributions in both console and Markdown table formats.

crates/fluvio-benchmark · high confidence

Introduce structured Helm chart installation and configuration API

The \fluvio-cluster\ crate now exposes a dedicated \charts\ module that provides a builder-based API for configuring and installing Fluvio Helm charts. This change introduces \ChartConfig\ to manage installation parameters such as namespace, chart version, and value overrides, and \ChartInstaller\ to handle the actual Helm install or upgrade operations. It supports three chart sources: inline charts (bundled within the binary via \include\_dir\), local filesystem paths, and remote Helm repositories, enabling users to install or upgrade the Fluvio system and application charts with greater flexibility and configurability.

crates/fluvio-cluster/src/charts · high confidence

Introduce structured pre-flight cluster checks with auto-fix suggestions

The \fluvio-cluster\ crate now includes a new \check\ module that performs pre-flight validation before cluster operations. This module defines a structured result type (\CheckStatus\) distinguishing between passed checks, auto-fixable errors (such as missing or outdated system charts), and unrecoverable errors (such as permission denials or incompatible Helm/Kubectl versions). For auto-fixable issues, the system provides specific remediation commands (e.g., \fluvio cluster start --sys\), improving user guidance during installation or startup failures.

crates/fluvio-cluster/src/check · high confidence

Introduce structured record, batch, and replica key types in the protocol crate

The \fluvio-protocol\ crate now exposes a dedicated \record\ module containing core data structures for message handling. Users can now work with \Batch\ objects that encapsulate base offsets, headers, and record collections, along with \RecordKey\ and \RecordData\ types that provide flexible conversion from strings, vectors, or bytes into record keys and payloads. The module also introduces \ReplicaKey\ for identifying topic-partition pairs and \PartitionOffset\ traits for tracking stable and start offsets, forming the foundational types for record encoding, decoding, and replication logic within the protocol layer.

crates/fluvio-protocol/src/record · high confidence

Introduce structured test runner with CLI configuration and process forking

The test harness now uses a new \test\_meta\ module to centralize test configuration, allowing users to control cluster lifecycle (start/delete), topic settings (name, replication, partitions, retention, segment size), and timeouts via command-line arguments. Tests can now be executed in isolated child processes using the new \async\_process\ and \fork\_and\_wait\ macros, and test requirements such as minimum SPUs, cluster type, and timeouts can be declared directly on test functions via the \\#\[fluvio\_test()\]\ attribute. Test results are displayed in a formatted table using \comfy-table\.

_crates/fluvio-test-util/test\meta · high confidence

Introduce the Connector Development Kit (CDK) CLI

The Connector Development Kit (CDK) is now available as a standalone CLI tool to streamline connector development. It provides commands to generate new connector projects from templates, build connectors with optional extra Cargo arguments, and deploy them locally using a YAML configuration file and optional secrets. The CDK also supports running connectors in a test mode, listing currently deployed connectors, viewing their logs, and shutting them down. Additionally, it includes a version command that displays the current git hash, captured at compile time.

crates/cdk · high confidence

Introduces Kubernetes controllers for SPU, SPG, and SPU services

The storage controller now includes a new module that registers and starts Kubernetes operators for managing SPU (Storage Processing Unit) and SPG (Storage Processing Group) resources. This adds three specific controllers: \SpgStatefulSetController\ to sync SPG definitions to StatefulSets, \K8SpuController\ to manage individual SPU instances based on SPG groups, and \SpuServiceController\ to handle SPU service creation. These components work together to automate the lifecycle of Fluvio storage nodes within a Kubernetes cluster.

crates/fluvio-sc/src/k8/controllers · high confidence

Introduces SC-to-SC mirroring controller with TLS support

The system now includes a new mirroring controller that enables data replication between Stream Controllers (SCs). This component establishes a connection to a 'home' cluster, retrieves topic metadata, and synchronizes local topics accordingly. The implementation supports TLS encryption for secure communication and includes logic to delete local topics that are no longer present in the source cluster, ensuring consistency across mirrored environments.

crates/fluvio-sc/src/controllers/mirroring · high confidence

Introduces core protocol encoding/decoding infrastructure

The \fluvio-protocol\ crate now includes a new \core\ module providing the foundational types and traits for Fluvio protocol serialization. This adds a \ByteBuf\ type for optimized immutable byte handling, implements \Encoder\ and \Decoder\ traits for standard Rust types (including \Vec\, \Option\, \BTreeMap\, and primitive integers), and provides variable-length integer (varint) encoding/decoding utilities. These components replace the previous \kf-protocol\ core implementation and serve as the basis for serializing Fluvio API messages.

crates/fluvio-protocol/src/core · high confidence

Introduces thread-safe replica storage with isolation-aware offset tracking

The storage module now provides a \SharableReplicaStorage\ wrapper that manages replica state using a read-write lock, exposing Log End Offset (LEO) and High Watermark (HW) via an event-publisher pattern. This enables consumers to listen for offset changes based on the configured isolation level: \ReadCommitted\ listeners track the high watermark, while \ReadUncommitted\ listeners track the log end offset, ensuring correct visibility semantics for produce and consume operations.

crates/fluvio-spu/src/storage · high confidence

Introduction of FluvioCodec for message framing

The codec module now includes a new FluvioCodec implementation that handles network message framing. This codec encodes messages by prepending a 4-byte length header to the payload and decodes incoming streams by reading this header to determine message boundaries, ensuring complete messages are processed before returning data to the application layer.

crates/fluvio-protocol/src/codec · high confidence

Introduction of Streaming Coordinator metadata store and global context

The Streaming Coordinator (SC) now maintains a local in-memory copy of metadata from the KV store, accessible via a new global \Context\ struct. This context aggregates references to core resources including SPUs, partitions, topics, SPGs, smartmodules, table formats, and mirrors, providing a unified entry point for the SC to manage and access these entities.

crates/fluvio-sc/src/core · high confidence

Introduction of a new partition scheduling module

A new scheduler module has been added to the storage controller to handle the allocation of topic partitions to SPUs. This change introduces a \PartitionScheduler\ and a \ReplicaPartitionMap\ wrapper, which implement logic to generate replica maps by selecting suitable SPUs based on weight and anti-affinity rules, while preserving existing assignments where possible. This new component replaces or supplements the previous scheduling logic within the controller.

crates/fluvio-sc/src/controllers/scheduler · high confidence

Introduction of basic RBAC and read-only authorization modes

The authentication service now supports two new authorization modes: a basic RBAC mode and a read-only mode. The basic authorization (\BasicAuthorization\) enforces permissions defined in a JSON policy file, allowing administrators to control access via roles and specific actions (Create, Read, Update, Delete) on object types. Additionally, a read-only authorization mode (\ReadOnlyAuthorization\) has been introduced, which restricts users to read-only operations on most object types while permitting specific actions like creation on SPUs. These changes provide a foundation for finer-grained access control within the Stream Controller.

crates/fluvio-sc/src/services/auth · high confidence

Introduction of fluvio-socket crate with build-time versioning

The new \fluvio-socket\ crate provides a TCP socket wrapper for the Fluvio protocol. It includes a \build.rs\ script that integrates the \built\ crate to embed build-time information and copies the project \VERSION\ file, ensuring the crate carries accurate version metadata at compile time. The crate is licensed under Apache 2.0.

crates/fluvio-socket · high confidence

Introduction of regex-filter smartmodule

A new regex-filter smartmodule has been added to the codebase, enabling users to filter records based on a regular expression pattern. The module initializes a regex pattern from the 'regex' parameter during setup and applies it to the UTF-8 decoded value of each record, allowing for pattern-based filtering logic.

smartmodule/regex-filter · high confidence

Mirror cluster management API handlers

The service controller now exposes handlers for managing mirror clusters, allowing users to register, unregister, and list remote mirror connections. The new list endpoint retrieves all configured mirrors, while the register endpoint creates new remote mirrors (preventing duplicates for remote types) and the unregister endpoint removes existing mirrors. All operations respect read-only metadata configurations, returning appropriate errors if changes are attempted in a read-only state.

_crates/fluvio-sc/src/services/public\api/mirror · high confidence

New 'fluvio home' commands for cluster mirroring management

Users can now use the new \fluvio home\ command group to manage home cluster mirroring. The \fluvio home connect\ command allows connecting to a home cluster by importing metadata from a specified file, while \fluvio home status\ displays the current state of home mirrors, including source ID, route, SC/SPU status, last seen time, and any errors.

crates/fluvio-cli/src/client/home · high confidence

New CI/CD action scripts for Fluvio cluster installation and package publishing

The repository now includes a set of new shell scripts in the \actions/\ directory to support CI/CD workflows. These scripts automate the installation of Fluvio local and Kubernetes clusters (\action-install-fluvio-cluster.sh\, \action-install-minikube.sh\), manage Helm binary replacement (\ci-replace-helm.sh\), and handle publishing package sets and binary packages to the Fluvio Hub (\publish-pkgset.sh\, \upload-bpkg.sh\). Additional utilities are provided for managing the Zig compiler (\zig-install.sh\, \zig-uninstall.sh\), generating changelogs (\post-release-changelog.sh\, \view-changelog.sh\), and managing the sccache server (\sccache-terminate.sh\).

actions · high confidence

New CLI commands for cluster lifecycle management and diagnostics

The \fluvio cluster\ command now includes several new subcommands to manage the cluster lifecycle and troubleshoot issues. Users can run \fluvio cluster check\ to verify prerequisites before starting, and \fluvio cluster diagnostics\ to collect logs and metadata into a tarball for debugging. Cluster shutdown is now handled by \fluvio cluster shutdown\ (aliased as \stop\), which stops processes without deleting data, while \fluvio cluster delete\ handles full uninstallation with optional flags for K8 or system chart removal. Additionally, \fluvio cluster resume\ allows restarting a previously stopped local cluster, and \fluvio cluster upgrade\ manages version updates with a confirmation prompt.

crates/fluvio-cluster/src/cli · high confidence

New CLI commands for managing topic partitions and mirrors

The Fluvio CLI now includes \fluvio topic add-partition\ to increment the number of partitions on an existing topic and \fluvio topic add-mirror\ to add a remote cluster as a mirror target for a topic. These new subcommands are integrated into the existing \fluvio topic\ command group alongside the standard create, delete, describe, and list operations, allowing users to dynamically scale topic partitioning and configure mirroring without recreating topics.

crates/fluvio-cli/src/client/topic · high confidence

New Kubernetes object definitions for SPU groups and services

The \crates/fluvio-sc/src/k8/objects\ module now contains the core type definitions and conversion logic for managing SPU (Storage Processing Unit) groups on Kubernetes. This includes \SpuGroupObj\ for handling group-level statefulsets and services, \SpuServiceSpec\ and \SpgServiceSpec\ for service resources, \StatefulsetSpec\ for statefulset management, and \ScK8Config\ for parsing pod configuration from the \spu-k8\ ConfigMap. These changes establish the structural foundation for how the storage controller creates and manages SPU pods, services, and associated configurations in a Kubernetes environment.

crates/fluvio-sc/src/k8/objects · high confidence

New SPU controller for health status reconciliation

The storage controller now includes a dedicated SPU controller that automatically reconciles SPU status with health check data. This component monitors SPU heartbeats and updates their online/offline state in the metadata store, ensuring that the system accurately reflects the current health of each SPU without manual intervention.

crates/fluvio-sc/src/controllers/spus · high confidence

New SmartModule derive macro generators for Filter, Map, Aggregate, and more

The \fluvio-smartmodule-derive\ crate now includes a complete set of new code generators for SmartModule types, including \filter\, \map\, \array\_map\, \filter\_map\, \aggregate\, \init\, and \look\_back\. These generators handle the low-level WASM entry points, input decoding, record processing, and output encoding for each SmartModule kind. This change introduces the \look\_back\ SmartModule type and the \SmartOpt\ derive macro for parsing extra parameters, replacing previous implementations with a consolidated, modular generator architecture.

crates/fluvio-smartmodule-derive/src/generator · high confidence

New SmartModule examples demonstrating timestamp access

Added three new example SmartModules that utilize the record timestamp feature: \aggregate\_with\_timestamp\ concatenates values with their timestamps, \map\_with\_timestamp\ outputs offset, timestamp, and value, and \filter\_look\_back\_with\_timestamps\ provides a look-back filter example (though it currently uses a static atomic variable rather than explicit timestamp logic in the diff). These examples illustrate how to access and format timestamps within SmartModule logic using the \chrono\ crate.

_smartmodule/examples/aggregate\_with\_timestamp, smartmodule/examples/filter\_look\_back\_with\_timestamps, smartmodule/examples/map\_with\timestamp · high confidence

New SmartModule examples for JSON array/object mapping and log filtering

Added three new SmartModule examples in the smartmodule/examples directory to demonstrate advanced data processing capabilities. The array\_map\_json\_array example shows how to flatten JSON arrays into individual records, while array\_map\_json\_object demonstrates extracting key-value pairs from JSON objects as separate records. Additionally, the filter\_json example provides a pattern for filtering records based on structured JSON fields, specifically illustrated by filtering application logs by severity level.

_smartmodule/examples/array\_map\_json\_array, smartmodule/examples/array\_map\_json\_object, smartmodule/examples/filter\json · high confidence

New TOML diff utility for release tooling

The \release-tools/check-crate-version/toml-diff\ crate has been added to the workspace, providing a Rust-based library to compute and display differences between TOML configuration files. It identifies added or deleted keys and values in tables and arrays, outputting the results with color-coded formatting (green for additions, red for deletions) to help users quickly spot changes in crate metadata or configuration during the release process.

release-tools/check-crate-version/toml-diff · high confidence

New admin watch example for topics, partitions, and SPUs

Added a new example demonstrating the Admin Watch API, showing how to asynchronously watch for updates on topics, partitions, and SPUs using the Fluvio client.

examples/04-admin-watch · high confidence

New aggregate smartmodule example with configurable initial value

Added a new example smartmodule in \smartmodule/examples/aggregate-init\ that demonstrates how to initialize an aggregation with a specific starting value. The module uses the \init\ hook to parse an \initial\_value\ parameter and stores it using \OnceLock\, ensuring it is applied only to the first record processed. Subsequent records are summed with this initial value (or the first record's value if no parameter is provided), allowing users to start their aggregation from a custom baseline rather than zero or the first input.

smartmodule/examples/aggregate-init · high confidence

New array\_map example that appends record timestamps to JSON values

Added a new SmartModule example (\array\_map\_json\_array\_with\_timestamp\) that demonstrates how to access the record timestamp within a SmartModule. This example takes a stream of JSON arrays as input, expands them into individual values, and appends the original record's timestamp to each output value, providing a concrete pattern for users who need to enrich data with metadata during transformation.

_smartmodule/examples/array\_map\_json\_array\_with\timestamp · high confidence

New batch produce example for Fluvio

Added a new example demonstrating how to produce batches of records to a Fluvio topic. The example shows creating a producer, sending multiple records in a loop, and flushing the batch at the end to optimize throughput.

examples/01-produce-batch · high confidence

New cargo-builder crate for managing Cargo build and clean operations

The \crates/cargo-builder\ crate introduces a new library for invoking the \cargo\ CLI tool, supporting both \build\ and \clean\ commands. It allows users to specify build profiles (defaulting to \release\), target packages via the \-p\ flag, and explicit compilation targets. The crate includes a builder pattern for constructing these commands and validates that the \--target\ argument is not provided redundantly in extra arguments. It also defines a default WASM target (\wasm32-wasip1\) and provides utilities to resolve package metadata and locate output binaries for various architectures, including \wasm32-unknown-unknown\ and \wasm32-wasip1\.

crates/cargo-builder · high confidence

New cluster uninstall and progress reporting infrastructure

The \fluvio-cluster\ crate now includes a dedicated uninstall workflow (\delete.rs\) that removes Kubernetes components, local data, and associated custom resources (such as SPUs, topics, and managed connectors) via Helm and direct API calls. This change introduces a structured error model (\error.rs\) to provide specific, user-friendly messages for installation and uninstallation failures, and adds a progress reporting system (\progress.rs\, \render.rs\) that displays animated spinners during operations while suppressing them in CI environments or when explicitly hidden.

crates/fluvio-cluster/src · high confidence

New consumer offset management commands and consume subcommand restructuring

The CLI now includes a new \fluvio consumer\ command group with \list\ and \delete\ subcommands, allowing users to view and remove stored consumer offsets for topics and partitions. Additionally, the \fluvio consume\ command has been restructured into a modular layout (\consume/mod.rs\, \record\_format.rs\, \table\_format.rs\) to support enhanced output formatting, including JSON pretty-printing, table formats with primary key-based row updates, and smartmodule chaining support.

crates/fluvio-cli/src/client/consume · high confidence

New echo example demonstrating key/value record handling

Added a new \03-echo\ example that demonstrates producing and consuming messages with explicit keys and values. The example shows how to send records using \producer.send(key, value)\ and how to retrieve the key via \record.get\_key()\ during consumption, illustrating the key/value API for streaming applications.

examples/03-echo · high confidence

New example for producing key-value records

Added a new Rust example demonstrating how to produce key-value records to a Fluvio topic. The example shows how to initialize a producer, send a record with a specific key and value, and flush the producer, allowing users to see the sent messages via the consume command.

examples/01-produce-key-value · high confidence

New fluvio-controlplane-metadata crate for cluster metadata definitions

The fluvio-controlplane-metadata crate has been introduced to centralize metadata definitions for the Fluvio control plane. It provides structured specs, statuses, and Kubernetes CRD mappings for core resources including Topics, Partitions, SPUs, SmartModules, and Mirrors. This change establishes the foundational data models and serialization logic used by the control plane to manage cluster state.

crates/fluvio-controlplane-metadata · high confidence

New fluvio-extension-common crate for shared CLI infrastructure

This change introduces the \fluvio-extension-common\ crate, consolidating shared CLI components previously located in the \cli\ crate. It provides a unified \InstallationType\ enum (K8, Local, Docker, Cloud, etc.) for detecting and managing cluster installation modes, a \ClusterTarget\ struct for handling connection profiles and TLS settings, and a comprehensive output system using \comfy-table\ for rendering tables, JSON, YAML, and TOML. Additionally, it includes a \time\_elapsed\ utility for formatting dates as relative time expressions and a \hex\_dump\ helper for debugging binary data.

crates/fluvio-extension-common · high confidence

New follower replication protocol and state management

The SPU now includes a dedicated follower replication subsystem that manages replica state and coordinates synchronization with leader SPUs. This change introduces a new peer-to-peer API (\FollowerPeerApiEnum\) supporting \SyncRecords\ and \RejectedOffsetRequest\ messages, allowing followers to receive data updates and handle offset rejections. A \FollowersState\ manager tracks individual replica states and groups them by leader SPU, while a \FollowGroupController\ handles the lifecycle of these groups, including initial offset synchronization and periodic reconciliation with the leader. This provides the foundational infrastructure for follower replicas to maintain consistency with their leaders.

crates/fluvio-spu/src/replication/follower, crates/fluvio-spu/src/replication/leader · high confidence

New key-value storage crate with log-backed implementation

The \fluvio-kv-storage\ crate has been added, providing a \KVStorage\ trait and a \LogBasedKVStorage\ implementation that maintains an in-memory cache synchronized with an underlying append-only log. This allows users to perform standard key-value operations (get, put, delete, entries) while ensuring durability and consistency through log replay and checkpointing mechanisms.

crates/fluvio-kv-storage · high confidence

New look\_back smartmodule example for stateful filtering

Added a new example smartmodule in \smartmodule/examples/filter\_look\_back\ that demonstrates the \look\_back\ capability. This example implements a filter that only passes records with values greater than the previously seen maximum, using a \look\_back\ function to update the stored state with each record's value.

_smartmodule/examples/filter\_look\back · high confidence

New modular Makefile structure for build, check, release, and test workflows

The project introduces a new set of Makefiles (build.mk, check.mk, release.mk, test.mk) that reorganize and expand the build and release infrastructure. Users can now build specific binaries like the CLI, SMDK, CDK, and benchmarking tools via dedicated targets, while the check suite includes formatting, documentation, clippy, unused dependency, and crate audit checks. The release workflow supports publishing binaries to the Fluvio Hub and package registry, handling versioning for stable and latest channels, and managing Docker image manifests. Additionally, the test suite provides granular targets for smoke tests (including TLS, RBAC, and Kubernetes variants), consumer offset tests, and validation of the test harness itself.

makefiles · high confidence

New profile-based cluster configuration with custom metadata and TLS support

The Fluvio client now uses a structured profile system to manage cluster connections, allowing users to define named profiles (e.g., 'local') that store cluster endpoints, TLS policies, and custom metadata. This change introduces support for custom metadata in cluster configurations, enabling users to store and retrieve arbitrary key-value pairs for specific clusters. Additionally, it provides flexible TLS configuration options, including disabled, anonymous, and verified modes with support for both inline certificates and file paths. The configuration is persisted in a TOML file at the default location (\~/.fluvio/config) or a custom path specified via the FLV\_PROFILE\_PATH environment variable.

crates/fluvio/src/config · high confidence

The \fluvio-protocol\ crate now includes a new \link\ module that defines a structured set of error codes for topics, partitions, SPUs, SmartModules, and connectors, along with an \ApiVersions\ request/response mechanism for client-server capability negotiation. SmartModule errors are now more specific, covering initialization, runtime, look-back, and memory limit issues, which helps users diagnose processing failures more accurately.

crates/fluvio-protocol/src/link · high confidence

New protocol testing and storage utilities

The \fluvio-protocol\ crate now includes a \fixture\ module to help generate test batches and records, and a \store\ module that adds \FileWrite\ implementations for encoding request and response messages to files, supporting both in-memory bytes and async file slices.

crates/fluvio-protocol/src · high confidence

New protocol-derive macro crate for Fluvio protocol encoding and decoding

A new \fluvio-protocol-derive\ crate has been added to provide procedural macros for the Fluvio protocol. This crate introduces \\#\[derive(Encoder)\]\ and \\#\[derive(Decoder)\]\ attributes that automatically generate serialization and deserialization logic for structs and enums, supporting named fields, tuple structs, and unit variants. It also includes a \\#\[derive(RequestApi)\]\ macro to generate API request traits based on \\#\[fluvio\]\ attributes (such as \api\_key\, \api\_min\_version\, and \api\_max\_version\). The implementation enforces that enums must have explicit tags or discriminants and supports version-based field filtering during encode/decode operations.

crates/fluvio-protocol-derive · high confidence

New public API handlers for SpuGroup creation, deletion, and listing

The service controller now exposes dedicated request handlers for managing SpuGroups via the public API. This change adds \create.rs\, \delete.rs\, and \fetch.rs\ modules that process create, delete, and list requests respectively. Each handler enforces authorization checks (using \TypeAction\ for create/list and \InstanceAction\ for delete) before interacting with the underlying KV store, ensuring that users can only perform actions they are permitted to. The create handler supports a configurable timeout for the operation, while the delete handler returns a specific 'not found' error if the SpuGroup does not exist.

_crates/fluvio-sc/src/services/public\api/spg · high confidence

New public API handlers for topic creation, deletion, and listing

The service layer in \crates/fluvio-sc/src/services/public\_api/topic\ now includes dedicated request handlers for managing topics. The new \create.rs\ handler validates topic names, checks for duplicates, verifies configuration, and ensures deduplication smart modules are loaded before creating computed, assigned, or mirror topics. The \delete.rs\ handler enforces safety by preventing the deletion of system topics unless the \force\ flag is set, and blocks deletion of mirror topics from remote locations. The \fetch.rs\ handler allows listing topics with filtering options, including the ability to toggle visibility of system topics. These changes implement the core CRUD operations for topics within the public API service.

_crates/fluvio-sc/src/services/public\api/topic · high confidence

New public API service for cluster administration

A new public API service has been added to the Streaming Controller, exposing a unified set of endpoints for managing cluster resources. This service handles Create, Read (List), Update, Delete, and Watch operations for Topics, SPUs, SPGs, Partitions, SmartModules, TableFormats, and Mirrors. It includes version negotiation via an ApiVersions endpoint and enforces authorization checks for all administrative actions, allowing external clients to programmatically manage the cluster state through a structured protocol.

_crates/fluvio-sc/src/services/public\api · high confidence

New public SPU API handlers for offset management and consumer lifecycle

The SPU now exposes dedicated public API handlers for managing consumer offsets and stream sessions. This includes new endpoints to fetch and update consumer offsets (replacing older internal mechanisms), handle API version negotiation to support client compatibility, and manage persistent stream fetch sessions with proper connection context tracking. These changes enable more robust consumer offset persistence and cleaner separation of stream fetch logic from standard fetch requests.

crates/fluvio-spu/src/services/public · high confidence

New release tooling for validating and publishing Fluvio crates

Added a suite of release automation scripts in the release-tools directory to streamline the publishing process for Fluvio crates. This includes check-publish-crates.sh to validate that dependencies are correctly pinned before publishing, publish-crates.sh to handle the actual upload with retry logic, and test-crate-version.sh to detect version mismatches between the repository and crates.io. A shared publish-list file defines the ordered list of crates to be processed, and a new Rust-based toml-diff utility with comprehensive test data is introduced to assist in comparing TOML configurations.

release-tools · high confidence

New remote cluster management commands

The CLI now includes a new set of commands under the 'remote' namespace to manage remote cluster connections: 'register' to add a new remote cluster, 'list' to view their status (including SC/SPU status and errors), 'unregister' to remove a remote cluster, and 'export' to generate a metadata file for a remote cluster, optionally including TLS certificate and key details for secure configuration.

crates/fluvio-cli/src/client/remote · high confidence

New smartmodule development container and publishing documentation

The dev-tools directory now includes a Dockerfile for a Rust-based smartmodule development environment, which installs Rust, Fluvio CLI, and necessary WASM targets, and pre-generates an example project. A new PUBLISH.md file documents the steps to update the version and release crates. Additionally, a script to retrieve Service Controller logs from Kubernetes pods has been added, while the legacy minikube configuration script has been removed.

dev-tools · high confidence

New test cluster setup utility for integration tests

A new \TestCluster\ struct has been added to the test utilities, providing a standardized way to initialize and manage Fluvio clusters for testing. This component handles cluster lifecycle management, including optional cleanup before initialization, starting the cluster, and establishing a Fluvio connection with optional TLS configuration based on the test environment settings.

crates/fluvio-test-util/setup · high confidence

New topic store module with partition creation and admin formatting

The topic store module in the service controller has been restructured to expose a dedicated \TopicMd\ trait that automatically creates new partitions based on a topic's replica map when they do not already exist in the partition store. Additionally, the \TopicLocalStore\ now implements a \table\_fmt\ policy, enabling users to view topic details (name, type, partitions, replication factor, resolution, etc.) in a formatted table output.

crates/fluvio-sc/src/stores/topic · high confidence

SPU SmartEngine batch processing and lookback implementation

The SPU now implements the core SmartEngine logic for processing record batches and supporting lookback queries. This change introduces batch processing logic that applies SmartModule chains to produce records, handling compression, offset adjustments, and memory limits. It also adds lookback functionality, allowing SmartModules to read historical records from the replica storage based on count or age constraints. The implementation includes context management for SmartModule invocations, chain building, and metrics export, with stub implementations for when the SmartEngine feature is disabled.

crates/fluvio-spu/src/smartengine · high confidence

SPU control-plane communication and mirroring infrastructure established

The SPU now includes a dedicated control-plane module to manage its connection to the SmartConnector (SC). This introduces a dispatcher that handles registration, reconnects on failure, and processes incoming commands such as replica, SPU, SmartModule, and mirror updates. It also implements a status-sink mechanism that batches and periodically sends LRS, partition, and mirror status back to the SC. Additionally, a mirroring module is added to structure remote and home mirroring logic, laying the groundwork for SPU-to-SPU mirroring capabilities.

_crates/fluvio-spu/src/control\plane · high confidence

SPU-to-SPU mirroring connection and authorization

The SPU now supports direct mirroring connections between clusters, including the necessary API definitions (MirrorHomeApiEnum) and request handlers for syncing records and updating offsets. This change introduces authorization checks to ensure that only permitted remote clusters can initiate mirroring, and adds logic to update mirror status when errors occur during the synchronization process.

crates/fluvio-spu/src/mirroring/home · high confidence

SmartEngine v2 introduces structured metrics, lookback support, and memory limits

The \fluvio-smartengine\ crate has been restructured to support SmartEngine v2, introducing a new \SmartModuleChainMetrics\ system that tracks bytes in, records out/err, invocation counts, fuel used, and CPU milliseconds for smartmodule chains. The engine now supports \look\_back\ calls with time-bound and record-count configurations, allowing smartmodules to access historical data. Additionally, a configurable memory limit (defaulting to 1 GB) is enforced via a \StoreResourceLimiter\ to prevent excessive memory usage, and WASI is explicitly enabled for smartmodule execution.

crates/fluvio-smartengine · high confidence

SmartModule SDK introduces lookback support and timestamp-aware record context

The SmartModule crate now supports the \look\_back\ feature, allowing SmartModules to access historical records via a new \Lookback\ struct in \SmartModuleExtraParams\ (encoded at version 22). Additionally, the \SmartModuleRecord\ type now exposes \offset()\ and \timestamp()\ methods, enabling modules to determine the absolute position and time of records within a stream. The crate also includes new error types for lookback operations (\SmartModuleLookbackErrorStatus\) and a benchmark for \SmartModuleInput\ encoding.

crates/fluvio-smartmodule · high confidence

SmartModule derive macro now supports look\_back and SmartModuleRecord types

The \fluvio-smartmodule-derive\ crate has been refactored to support new SmartModule kinds, specifically adding \look\_back\ alongside existing types like \init\, \aggregate\, \filter\, \map\, \array\_map\, and \filter\_map\. The macro now detects whether a function uses the new \SmartModuleRecord\ type or the legacy record format, enabling compatibility with SmartModule Engine v21 and later. Additionally, a new \SmartOpt\ derive macro is introduced to allow structs to be used as extra parameters in smartmodule functions, provided they implement \Default\ and their fields implement \FromStr\.

crates/fluvio-smartmodule-derive/src · high confidence

SmartModule development template and examples updated for WASI and SMDK

The \smartmodule\ directory now provides a standardized \cargo\_template\ for generating new SmartModules, featuring a \Cargo.toml.liquid\ template configured for the \wasm32-wasip1\ target and Rust edition 2024, along with a \SmartModule.toml\ for package metadata (including kebab-case group names and public/private visibility). The \smdk\ build tool is now the default workflow, with examples like \aggregate-init\, \filter\_hashset\, and \regex-filter\ updated to use \smdk build\ and \smdk load\, and \transforms.yaml\ files demonstrating new capabilities such as \lookback\ for filters and initial values for aggregations.

smartmodule · high confidence

SmartModule management commands added to CLI

The Fluvio CLI now includes dedicated commands for managing SmartModules. Users can create new SmartModules by specifying a name and a WASM binary file, list existing SmartModules with optional filtering and output formatting, delete one or more SmartModules (with an option to continue on error), watch for real-time changes to SmartModules, and test SmartModules locally (available on non-ARM architectures). These commands are implemented as a new \smartmodule\ subcommand group within the CLI client.

crates/fluvio-cli/src/client/smartmodule · high confidence

SmartModule public API handlers for create, delete, and list

The SmartModule service now exposes public API endpoints for creating, deleting, and listing SmartModules. The create handler validates authorization and processes the SmartModule specification, including package metadata, before storing it. The delete handler checks permissions, resolves the qualified name to a store ID, and removes the SmartModule if it exists, returning a not-found error otherwise. The list handler supports filtering and a summary mode, returning matching SmartModules or an empty list if authorization fails. These changes implement the core CRUD operations for SmartModules in the public API layer.

_crates/fluvio-sc/src/services/public\api/smartmodule · high confidence

Streaming Controller configuration module introduced

A new configuration module has been added to the Streaming Controller, defining the \ScConfig\ struct and its builder. This introduces support for read-only metadata mode, configurable public and private endpoints, namespace settings, X.509 authentication scopes, and a whitelist mechanism for controller capabilities.

crates/fluvio-sc/src/config · high confidence

Support for adding mirrors and partitions to existing topics

The topic update service now handles two new actions: adding a mirror to a home-partitioned topic and adding partitions to a computed-replica topic. Users can now dynamically extend a home-partitioned topic by assigning a remote cluster as a mirror (with validation to prevent duplicate assignments) and increase the partition count of computed-replica topics, subject to standard authorization checks.

_crates/fluvio-sc/src/services/public\api/topic/update · high confidence

Support for custom SPU registration and management

The SmartConnect now allows administrators to register, list, and unregister custom SPUs via the public API. This change introduces handlers for creating custom SPU specifications (which are validated and converted to standard SPU specs before storage), fetching lists of both standard and custom SPUs with appropriate authorization checks, and deleting custom SPUs by name or ID. Users can now manage custom SPU definitions through the standard SPU endpoints, with specific validation ensuring that only custom SPUs can be deleted via the custom SPU API and that duplicate names or IDs are rejected during creation.

_crates/fluvio-sc/src/services/public\api/spu · high confidence

Removals

Removal of Kubernetes configuration and metadata core modules

The \k8-config\ crate, which previously handled reading Kubernetes configuration files and pod service account credentials, has been removed along with its associated YAML data and documentation. Additionally, the \k8-metadata/metadata-core\ library, which provided core traits and structures for Kubernetes Custom Resource Definitions (CRDs) and object metadata, has been deleted. The \spu-server\ component has also been removed, including its CLI configuration, config file parsing, and follower replica controller logic, effectively stripping out the local Kubernetes configuration handling and SPU server components from this area.

k8-config, k8-metadata/metadata-core, spu-server · high confidence

Removal of advanced CLI commands and auth token management

The CLI no longer supports the \advanced\ subcommands (such as \generate\ and \run\ for sending raw Kafka protocol requests) and the entire \auth-token\ command group (including \create\, \list\, and \delete\ operations for Streaming Controller auth tokens). This change removes the associated CLI argument definitions, request processing logic, and metadata query modules, effectively stripping these capabilities from the user-facing interface.

cli · high confidence

Removal of core metadata and types modules

The \metadata\ and \types\ modules have been removed from the codebase. This deletes the local definitions and conversion logic for SPU, Auth Token, Partition, and Topic specs and statuses, along with the shared type definitions, default configuration constants, and socket helper utilities that supported them.

k8-metadata/metadata-spu, metadata, types · high confidence

Removal of internal and public API protocol definitions

The internal communication protocol definitions between the Stream Controller (SC) and SPU (Stream Processing Unit) have been removed, including the \internal-api\ crate and its request types for updating SPUs, replicas, and live replica status. Additionally, the public-facing API definitions for the SC and SPU have been deleted, removing support for Fluvio-specific operations such as creating/deleting topics and SPUs, fetching metadata, and managing SPU groups, as well as standard Kafka protocol requests like Produce and Fetch.

(repo-wide) · high confidence

Removal of legacy Docker build scripts for SC and SPU

The legacy Makefiles and Dockerfiles used to build the \sc-server\ and \spu-server\ Docker images have been removed. This eliminates the manual build targets (such as \build\, \push\, and \pull\_minikube\) and associated configuration files (including \.gitignore\ entries for the \target\ directory) that previously facilitated local Docker image creation and deployment for these components.

k8-util/docker/sc · high confidence

Removal of legacy Kubernetes CRD definitions and storage configuration

The \k8-util/crd\ directory has removed the legacy Custom Resource Definition (CRD) YAML files for \Spu\, \SpuGroup\, \Partition\, and \Topic\, along with the \minikube-storageclass-spu.yaml\ storage class and the \init.sh\ script used to apply them. This change eliminates the deprecated \fluvio.infinyon.com\ API group resources and the associated minikube-specific storage provisioning setup from this utility location.

k8-util/crd · high confidence

Removal of legacy async runtime and socket infrastructure

This change removes the \future-helper\ crate (including its \test\_async\ derive macro and Tokio-based runtime helpers), the \kf-protocol-transport\ codec, and the entire \kf-socket\ library (covering socket pooling, sinks, and file-based fetch/produce logic). It also deletes associated test fixtures and toolchain symlinks, effectively stripping out the previous asynchronous networking and protocol transport stack from the codebase.

(repo-wide) · high confidence

Removal of legacy async runtime support and compatibility layers

The \future-aio\ and \future-helper-03\ modules have removed all code supporting the legacy \tokio\_1\ runtime and \futures\_0.1\ ecosystem. This includes the deletion of the \compat\ module (which provided \Compat01As03Sink\ for bridging old sinks), the removal of \io\_util\_1.rs\ (which handled blocking I/O for the old runtime), and the deletion of \tcp\_stream\_1.rs\ (the TCP stream implementation for \tokio\_1\). Additionally, the \rust-toolchain\ symlinks and files in both \future-aio\ and \future-helper-03\ have been removed, indicating a shift away from the previous nightly toolchain requirements and legacy runtime configurations.

future-aio, future-helper-03, kf-protocol · high confidence

Removal of sc-server legacy codebase and configuration

The sc-server module has been removed from the product. This change deletes the server's source code (including CLI, connection management, and core logic), configuration files, and documentation, indicating that this component is no longer part of the system.

sc-server · high confidence

Removal of the k8-client crate

The k8-client crate has been completely removed from the codebase. This deletion eliminates the Kubernetes Rust client implementation, including its HTTP client logic, authentication helpers, and support for Kubernetes resources such as Pods, Services, StatefulSets, ConfigMaps, and Secrets, along with all associated integration tests and build artifacts.

k8-client · high confidence

Removed legacy 'flv' Kubernetes deployment manifests

The legacy Kubernetes deployment files for the storage controller (previously identified by the 'flv' prefix) have been removed. This includes the namespace definition, the main and debug deployment configurations, the internal and public service definitions, and the Minikube context setup script. Users relying on these specific files for deployment should update their configuration to use the current deployment artifacts.

k8-util/sc-deployment · high confidence

Architecture

Refactored cluster startup logic into modular components

The cluster startup implementation has been reorganized into distinct modules (\k8.rs\, \local.rs\, \common.rs\) to separate concerns between Kubernetes and local installation paths. This change introduces a shared \common.rs\ module containing retry logic for connecting to the SmartConnect (SC) service and checking Custom Resource Definition (CRD) readiness, which is now reused across both installation types. The \LocalConfig\ and \ClusterConfig\ structs are now defined within their respective modules, providing a cleaner structure for configuration and installation flows.

crates/fluvio-cluster/src/start · high confidence

Refactored topic controller into modular components

The topic controller logic in the storage coordinator has been restructured into distinct modules: \actions.rs\ defines the \TopicActions\ struct for batching topic and partition operations, \controller.rs\ implements the main \TopicController\ and \SystemTopicController\ with their respective dispatch loops for syncing topics and SPUs, \policy.rs\ handles validation of topic parameters and mirror configurations, and \reducer.rs\ manages the state transitions and partition generation logic. This change improves testability and separation of concerns within the topic management subsystem.

crates/fluvio-sc/src/controllers/topics · high confidence

SPU core refactored into modular local stores and a global context

The SPU's internal architecture has been reorganized to improve modularity and state management. A new \GlobalContext\ now serves as the central hub, aggregating shared state such as configuration, metrics, the SmartEngine, and leader connections. This context relies on a set of new, specialized local stores (\SpuLocalStore\, \ReplicaStore\, \SmartModuleLocalStore\, \MirrorLocalStore\) that manage their respective specifications (SPUs, replicas, smart modules, mirrors) with a unified \Spec\ trait and \LocalStore\ implementation. Additionally, a dedicated \LeaderConnections\ component handles the lifecycle of connections to leader SPUs, and a new \SpuMetrics\ module provides granular tracking of inbound/outbound activity and smart module performance.

crates/fluvio-spu/src/core · high confidence

Behavioural changes

Adopts Zig as the C compiler for cross-platform builds

The build system now uses Zig to compile C code for various target architectures, including aarch64-linux-musl, x86\_64-linux-musl, aarch64-macos-gnu, and arm-linux-gnueabihf. New wrapper scripts in the build-scripts directory configure the Zig compiler with appropriate targets and filter out specific flags (like self-contained crt paths or -latomic) to ensure successful cross-compilation.

build-scripts · high confidence

Automated chart packaging and versioning during build

The Fluvio cluster crate now automatically packages Helm charts and embeds the current Git commit hash at compile time. The new build script triggers the chart packaging process via the Makefile and sets the GIT\_HASH environment variable, ensuring that the installed cluster binaries accurately reflect the source version and include the necessary chart artifacts.

crates/fluvio-cluster · high confidence

Build-time version and git hash integration for version command

The crate now includes a build script that activates when the \version-cmd\ feature is enabled. This script embeds the current git commit hash into the binary at compile time and ensures the build re-runs if the source \VERSION\ file or the build script itself changes, providing the necessary metadata for the newly added \version\ command in \smdk\ and \cdk\.

crates/fluvio-cli-common · high confidence

CLI entry point now uses mimalloc and Clap with improved help handling

The main entry point for the Fluvio CLI has been updated to use the mimalloc allocator for improved performance and switched from StructOpt to Clap for command-line argument parsing. Additionally, the application now includes a custom 'help hack' that intercepts early arguments like -h, --help, or help to display usage information immediately, even before full command parsing occurs, ensuring a more responsive user experience when seeking help.

crates/fluvio-cli/src/bin · high confidence

CLI restructured with modular source files and enhanced version reporting

The CLI source code has been reorganized into distinct modules (error, client, profile, version, metadata, render, monitoring) to improve maintainability. The \fluvio version\ command now provides richer output, including the release channel, a SHA256 hash of the frontend binary, OS details, and a list of installed plugin versions. Additionally, a new \fluvio metadata\ command (hidden by default) allows users to retrieve JSON-formatted metadata for the CLI and all installed extensions, and Unix-based systems now support client metrics monitoring via a Unix socket when the \FLUVIO\_METRIC\_CLIENT\_DIR\ environment variable is set.

crates/fluvio-cli/src · high confidence

CLI versioning and documentation updates

The Fluvio CLI now includes a build script that embeds the current git hash and OS information into the binary at compile time, enabling more accurate version reporting. Additionally, the CLI documentation has been updated to reflect the current cluster architecture (Streaming Controller and Streaming Processing Unit) and remove outdated references to Kafka compatibility.

crates/fluvio-cli · high confidence

Control plane internal API restructured and migrated to fluvio-protocol

The internal control plane crate has been reorganized into a new \crates/fluvio-controlplane\ structure, consolidating request and message types for SPU and SC communication. The implementation has migrated from the legacy \kf\_protocol\ library to \fluvio\_protocol\, updating encoding/decoding traits and error codes accordingly. This change introduces specific internal API endpoints for registering SPUs, updating live replica status (LRS), handling replica removal, updating partition status, and managing mirror status, alongside dedicated request types for updating SPUs, replicas, smart modules, and mirrors.

crates/fluvio-controlplane · high confidence

Fluvio Run CLI restructured with Clap and version support

The fluvio-run crate has been refactored to use the Clap library for command-line argument parsing, replacing the previous implementation. This change introduces a unified RunCmd interface that manages subcommands for running Streaming Processing Units (SPU) and Streaming Controllers (SC), alongside new metadata and version reporting capabilities. Users can now access version information via the 'version' subcommand, which displays the Git commit hash and platform version, and retrieve plugin metadata in JSON format.

crates/fluvio-run/src · high confidence

Introduce new test runner with process forking and signal-based completion

The test harness in \crates/fluvio-test\ has been rewritten to support running individual tests in separate child processes via OS forking. The runner now waits for test completion by listening for SIGUSR1 (success) or SIGUSR2 (failure) signals sent from the child process, replacing the previous synchronous or panic-based completion model. This change also introduces configurable test timeouts and a spinning progress indicator for better user feedback during long-running tests.

crates/fluvio-test/src · high confidence

Introduce structured API request/response types and Clone support for Produce requests

The \fluvio-protocol\ crate now exposes a structured \api\ module containing \RequestHeader\, \RequestKind\, \RequestMessage\, and \ResponseMessage\ types to standardize how API messages are encoded, decoded, and handled. A key behavioral change is that \RequestMessage\ (including the \Produce\ request) now implements the \Clone\ trait, allowing these message structures to be duplicated for retry or inspection logic without consuming the original. This refactors the previous ad-hoc protocol handling into a more explicit, type-safe API surface.

crates/fluvio-protocol/src/api · high confidence

Introduces new batching producer with configurable delivery semantics and compression

The producer module has been rewritten to use a new \RecordAccumulator\ that buffers records into batches before sending them to the server. This change introduces \TopicProducerConfig\ with new options for \DeliverySemantic\ (allowing users to choose between 'AtMostOnce' fire-and-forget or 'AtLeastOnce' retry-on-error guarantees), \Isolation\ levels, and optional \Compression\. The default \linger\ time is now 0ms, meaning records are sent immediately unless a batch is full, and the system now supports SmartModule invocations and callbacks for post-send processing.

crates/fluvio/src/producer · high confidence

Introduction of fair replica scheduling and election policies

The partition store now includes a new election policy module that implements a fair replica scheduler. This change introduces a scoring mechanism for leader election, where replicas are evaluated based on their log end offset (LEO) lag relative to the current leader; replicas with a lag of less than 4 are considered suitable candidates, while those with higher lag are marked as unsuitable. Additionally, the store now tracks replica scheduling groups to help identify suitable SPUs for leadership based on weight and anti-affinity rules, improving the distribution of leader responsibilities across the cluster.

crates/fluvio-sc/src/stores/partition · high confidence

Major developer experience overhaul and Rust toolchain upgrade

The project has significantly updated its developer documentation (DEVELOPER.md, README.md, CONTRIBUTING.md) with modernized instructions for building and running the cluster locally and on Kubernetes. The build system has been refactored, replacing the legacy Makefile with a modular structure (build.mk, test.mk, etc.) and updating the Rust toolchain from an old nightly version to stable 1.93.0. Additionally, the repository now includes a Code of Conduct, a detailed Release process guide, and a new GitHub Action for setting up the development environment.

(repo-wide) · high confidence

New SPU store module with health tracking and metadata policies

The storage layer for Storage Processing Units (SPUs) has been reorganized into a new \stores/spu\ module. This change introduces a \HealthCheck\ component that tracks the online status of SPUs and publishes events when health states change, allowing consumers to react to SPU availability updates. Additionally, the module defines \SpuLocalStorePolicy\ traits and implementations that provide standardized methods for querying SPU metadata, such as retrieving online IDs, counting active units, and validating registrations, replacing the previous ad-hoc storage logic.

crates/fluvio-sc/src/stores/spu · high confidence

New consumer API with configurable offset management and retry logic

The consumer module has been refactored to introduce a new \ConsumerConfigExt\ configuration structure and a \ConsumerStream\ trait that supports explicit offset management. Users can now choose between \None\, \Manual\, or \Auto\ strategies for tracking consumed offsets, with \Auto\ mode automatically committing and flushing offsets based on configurable periods. The new API also includes a \ConsumerRetryStream\ that handles reconnection and retry logic using exponential backoff, ensuring more robust consumption in unstable network conditions. The previous \PartitionConsumer::stream\ method is now deprecated in favor of the new \Fluvio::consumer\_with\_config()\ approach.

crates/fluvio/src/consumer · high confidence

Partition controller introduces generic metadata context and election logic

The partitions controller module has been restructured to support a generic metadata context (C: MetadataItem), allowing the same controller logic to operate with different storage backends (e.g., Kubernetes vs. in-memory). This change introduces a \PartitionController\ that dispatches loops to sync SPU status and partition metadata changes, delegating state transitions to a \PartitionReducer\. The reducer now handles leader election logic, specifically detecting offline SPUs and triggering leader re-election or marking partitions as offline if no suitable candidate is found. A test plan document is also added to outline expected behaviors for topic creation, SPU status changes, and leader elections.

crates/fluvio-sc/src/controllers/partitions · high confidence

Producer command modularized with new stats reporting and SmartModule transformation support

The \fluvio produce\ CLI command has been restructured into a modular layout (\src/client/produce\), introducing dedicated stats reporting that outputs throughput, latency, and system metrics to CSV files or formatted tables. This change also enables SmartModule transformations for the producer via \--transforms\ and \--transforms-line\ arguments, allowing users to apply data transformations during record production.

crates/fluvio-cli/src/client/produce · high confidence

Refactor SPU CLI to support custom SPUs and improved table output

The SPU CLI module has been restructured to explicitly support custom SPUs alongside managed ones. The \list\ command now accepts a \--custom\ flag to filter for custom SPUs only, and the output formatting has been migrated to \comfy-table\ to provide a consistent tabular view (ID, NAME, STATUS, TYPE, RACK, PUBLIC, PRIVATE) for SPU listings. Additionally, the \register\ command now supports specifying a local public server address via the \--public-server-local\ flag, allowing users to configure custom SPUs with specific local endpoint bindings.

crates/fluvio-cluster/src/cli/spu · high confidence

Refactor cluster start command into modular installer components

The \fluvio cluster start\ command has been restructured to separate concerns by installation target. New modules handle Kubernetes (\k8.rs\), local (\local.rs\), and system chart (\sys.rs\) installations independently, while \tls.rs\ centralizes TLS configuration parsing. This change introduces specific CLI options for each target, such as \--use-k8-port-forwarding\ and \--use-cluster-ip\ for Kubernetes, and \--data-dir\ and \--sc-pub-addr\ for local clusters, allowing users to configure their cluster startup with greater precision.

crates/fluvio-cluster/src/cli/start · high confidence

Refactor profile management commands to use Clap v4 and anyhow

The profile subcommands (add, current, delete, list, export, rename, switch) have been rewritten to use the Clap v4 argument parser and the anyhow error-handling library. This change standardizes how profile configuration is loaded and saved, ensuring consistent error reporting and command-line interface behavior across all profile operations.

crates/fluvio-cli/src/profile · high confidence

Refactored CLI command structure and SmartModule invocation logic

The CLI client module has been reorganized to introduce a \ClientCmd\ trait and a unified \FluvioCmd\ enum, which now explicitly manages subcommands for topics, partitions, consumers, smartmodules, table formats, and remote/home cluster interactions. This structural change includes the addition of a new \smartmodule\_invocation\ module that standardizes how SmartModules are created from predefined names, local WASM files, or transformation configurations, ensuring consistent handling of context data and parameters across producer and consumer operations.

crates/fluvio-cli/src/client · high confidence

Reintroduce tracing initialization in the run binary

The main entry point for the fluvio-run binary now explicitly initializes the tracing subscriber via \fluvio\_future::subscriber::init\_tracer(None)\ before processing commands. This restores the ability to capture and observe runtime traces and logs, which were previously missing from this execution path.

crates/fluvio-run/src/bin · high confidence

Removal of legacy shell scripts and addition of developer documentation

The k8-util directory has been cleaned up by removing several legacy shell scripts (deploy.sh, install.sh, minikube-start.sh, minikube-tunnel.sh) that previously handled deployment, cluster role bindings, storage class creation, and minikube tunneling. In their place, a new DEVELOP.md file has been added to provide instructions for setting up a local Kubernetes proxy to minikube for API tracing and inspection.

k8-util · high confidence

Removed custom x86\_64 musl linker configuration

The custom linker configuration for the x86\_64-unknown-linux-musl target has been removed from the Cargo configuration. This means the project no longer explicitly specifies x86\_64-linux-musl-gcc as the linker for musl-based builds, reverting to Cargo's default linker selection for this target.

.cargo · high confidence

Renamed smartstream to derivedstream

The smartstream service has been renamed to derivedstream. This change updates the internal service location and module structure to reflect the new naming convention, ensuring consistency with the broader platform terminology.

_crates/fluvio-sc/src/services/public\api/derivedstream · high confidence

Reorganize store modules and rename table format module

The store module structure in the smart connector has been reorganized to improve clarity and alignment with package naming conventions. The \tableformat\ module has been renamed from its previous identifier to better reflect its purpose, and the module hierarchy has been updated to expose specific sub-modules for SPUs, topics, partitions, SPGs, smartmodules, and the newly named table format. This change primarily affects internal module organization and does not alter external APIs or user-facing behavior.

crates/fluvio-sc/src/stores · high confidence

Replaced shell-based crate version check with a new Rust tool

The \release-tools/check-crate-version\ script has been rewritten in Rust, introducing a new binary that downloads crates from crates.io and compares their source code and \Cargo.toml\ manifests against the local repository. The tool now reports specific statuses for each crate—such as whether the version needs bumping, if code or manifests have changed without a version update, or if the crate is not yet published—and panics if any crate requires a version bump, providing a more robust and detailed validation process for release preparation.

release-tools/check-crate-version · high confidence

Rust client library restructured with new admin and error handling APIs

The Fluvio Rust client library has been reorganized into a cleaner module structure, introducing a dedicated \FluvioAdmin\ interface for cluster management (creating, deleting, and watching topics and other objects) and a comprehensive \FluvioError\ enum that provides specific, descriptive error types for issues like version mismatches, missing topics/partitions, and producer/consumer configuration errors. The main \Fluvio\ client now exposes these capabilities through \connect()\, \connect\_with\_config()\, and \connect\_with\_profile()\, while re-exporting key types like \Offset\, \ProducerCallback\, and \DeliverySemantic\ for easier access. This change simplifies the API surface and improves error reporting for users interacting with Fluvio clusters.

crates/fluvio/src · high confidence

SMDK now defaults to the WASI target for builds and tests

The SmartModule Development Kit (SMDK) has shifted its default compilation target from \wasm32-unknown-unknown\ to \wasm32-wasip1\. This change is applied in the \smdk build\, \smdk test\, and \smdk load\ commands, meaning generated and tested SmartModules will now use the WASI standard by default. To maintain compatibility with older tooling or specific requirements, users can explicitly opt out of this behavior by setting the \SMDK\_NOWASI\ environment variable or passing the \--nowasi\ flag.

crates/smartmodule-development-kit · high confidence

SPU configuration and CLI restructured with TLS and memory limit support

The Streaming Processing Unit (SPU) now uses a new CLI interface built on Clap, allowing users to configure core settings such as SPU ID, public/private server bindings, and Smart Engine memory limits via command-line arguments or environment variables. This change introduces native TLS support for the SPU's public endpoint, enabling secure communication with server certificates and optional client authentication. Additionally, the configuration system now explicitly manages log storage parameters (base directory, size, index limits) and replication settings, providing a more robust and configurable foundation for SPU deployment.

crates/fluvio-spu/src/config · high confidence

SPU server initialization and monitoring endpoint

The SPU now initializes its monitoring subsystem before creating the proxy, exposing metrics via a Unix socket (configurable via FLUVIO\_METRIC\_SPU or defaulting to the standard path). The startup sequence logs platform, git, and system details, and the public server is now created with a RootAuthorization context. Additionally, the SPU binary uses mimalloc for memory allocation and clap for CLI argument parsing.

crates/fluvio-spu/src · high confidence

Standardized version output with table format and JSON support

The version command now displays information in a standardized table format, including the CLI name, version, architecture, SHA256 hash, and optional details like release channel, git commit, and OS info. Additionally, users can now request JSON output using the \--json\ flag, which serializes the same version data for easier parsing by scripts or other tools.

_crates/fluvio-cli-common/src/version\cmd · high confidence

Streaming Controller CLI and startup logic restructured with read-only mode and TLS proxy support

The Streaming Controller (SC) now uses a new CLI interface (clap) to configure run modes (local, Kubernetes, or read-only), bind addresses, and TLS settings. A new read-only mode allows the SC to start with a static metadata snapshot for debugging or testing. When TLS is enabled, the SC automatically starts an internal TLS proxy on a separate port to handle external traffic, supporting both basic RBAC and X.509 authorization scopes. The startup sequence has been refactored to initialize metadata dispatchers for all supported specs (SPUs, topics, partitions, etc.) and controllers before launching the public and internal servers.

crates/fluvio-sc/src · high confidence

Switch to mimalloc allocator for improved performance

The Storage Controller binary now uses the mimalloc allocator instead of the default system allocator, which is expected to improve memory management performance. This change is implemented by adding a global allocator declaration in the main entry point.

crates/fluvio-sc/src/bin · high confidence

Unified Admin API protocol with backward compatibility

The SC schema crate now uses a single, unified API version (19) for all admin operations (Create, Delete, List, Watch, Update, Mirroring), replacing the previous per-object versioning. To ensure compatibility with older clients, the new dynamic object protocol (version 11+) includes fallback logic to decode and encode requests using the legacy 'classic' protocol format when communicating with older versions.

crates/fluvio-sc-schema · high confidence

Updated Cargo connector template with Rust 2024 edition and kebab-case group names

The connector project template now uses the Rust 2024 edition and applies kebab-case formatting to connector group names in the generated metadata. It also includes a .gitignore file that excludes build artifacts and IDE files, and ensures the README is part of the generated manifest.

_connector/cargo\template · high confidence

Updated dev-tools log scripts to use new binary names and log categories

The shell scripts in dev-tools/log have been updated to reflect internal naming changes. The Storage Controller (SC) and Partition Unit (SPU) debug scripts now invoke the \fluvio-sc-k8\ and \fluvio-spu\ binaries instead of the previous \sc-server\ and \spu-server\. Additionally, the logging configuration has shifted from module-specific prefixes like \sc\_server\ and \spu\_server\ to broader categories such as \kf\, \fluvio\, and \sc\_core\, ensuring that debug output aligns with the current codebase structure.

dev-tools/log · high confidence

Updated sample CRDs for v2 API and local configuration

The sample Custom Resource Definitions in k8-util/samples/crd have been updated to use the fluvio.infinyon.com/v2 API version. Topic samples (topic1, topic2, topic3) now structure partition and replication settings under a replicas.computed block, and topic3 includes new cleanup and storage policies. SPU group samples were renamed and adjusted for single-replica debug configurations with debug-level logging. SPU endpoint samples (spu\_5001-5003) now use localhost for both public ingress and private hosts. The initialization script also applies a new crd\_mirror.yaml file.

k8-util/samples · high confidence

Updated smart module examples to use the latest interface and OnceLock

The \filter\_init\ and \filter\_with\_param\ example smart modules have been updated to adopt the latest smart module interface. This includes migrating static state initialization to use \std::sync::OnceLock\ instead of previous mechanisms, and updating imports to align with the current \fluvio\_smartmodule\ API (e.g., \SmartModuleExtraParams\). Users writing custom smart modules should note these structural changes in how initialization parameters are handled and stored.

_smartmodule/examples/filter\_init, smartmodule/examples/filter\_with\param · high confidence

Versioned connector configuration with structured topic settings

Connector configuration files now support explicit versioning (0.0.0, 0.1.0, and 0.2.0) to manage schema evolution. In version 0.2.0, the topic configuration has been upgraded from a simple string to a structured \TopicConfig\ object, allowing for more granular topic settings. The system also introduces custom serialization for byte-size values and exposes necessary types for constructing connector configurations.

crates/fluvio-connector-package/src/config · high confidence

Test coverage

Added UI tests for connector macro validation; Added UI tests for derive macro compatibility; Added UI tests for fluvio\_test macro validation; Added UI tests for the test derive macro; Added batching producer test suite; Added benchmark suite for ByteBuf and Vec\<u8\> encoding/decoding; Added concurrent producer/consumer integration tests; Added consumer offset management test suite; Added data generator longevity test; Added election test for leader failover and recovery; Added integration test for replica fetch operations; Added integration test suite for producer and consumer operations; Added integration tests for SPU mirroring fixtures and scenarios; Added integration tests for SPU public services; Added longevity test suite for extended producer/consumer validation; Added manual consumer offset strategy tests; Added reconnection tests for producer and consumer streams; Added regression and integration tests for fluvio-channel CLI; Added replication test infrastructure for SPU; Added self-test suite for validating test infrastructure; Added shell script tests for FVM installation; Added shell-based test suite for Fluvio example binaries; Added smoke tests for Fluvio produce, consume, and table format operations; Added test fixtures for Fluvio version manager; Added test for multiple partition consumer behavior; Added test for producer failure handling during SPU termination; Added tests for auto consumer offset management and periodic flushing; Added tests for connector package metadata serialization; Added tests for enum conversion and socket I/O; Added tests for protocol API versioning and derive macro validation; Added tests for protocol derive macros and encoding/decoding logic; Cleanup of Kubernetes test fixtures in the Streaming Coordinator; Expanded CLI and CDK integration test coverage.

Housekeeping

Added Apache 2.0 license files to test crates

The \fluvio-test\ and \fluvio-test-derive\ crates now include their own \LICENSE-APACHE\ files, explicitly stating that these components are distributed under the Apache License, Version 2.0. This ensures that users and redistributors have clear legal terms for using, modifying, and distributing these specific test utilities.

crates/fluvio-test, crates/fluvio-test-derive · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 61 → 61 (-0.3)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 70 → 70 (+0.0)
  • Architecture 95 → 92 (-2.6)
  • Maturity 64 → 69 (+4.5)
  • Readiness 60 → 60 (+0.0)
  • Security 55 → 54 (-1.9)
  • Performance 87 → 87 (+0.0)

Resolved (16)

  • Ambiguous 'force' semantics. 'force_delete' implies a different behavior than 'delete', but the API does not expose what that difference is (e.g., bypassing checks, immediate removal). This creates a cognitive load to determine when to use which.
  • Ambiguous loading behavior. 'load_default_or_new' suggests a specific fallback logic, while 'load' takes an optional path. It is unclear if 'load' with no path behaves the same as 'load_default_or_new'.
  • Conflicting delete semantics. 'delete_cluster' returns the deleted config object, while 'delete_cluster_check' returns a Result. It is unclear if 'check' implies validation before deletion or if they are alternative implementations.
  • Critical CVE: [GHSA redacted] (Cargo.lock)
  • Documentation: no project overview (crates/fluvio/README.md)
  • Inconsistent connection entry points. FluvioAdmin and Fluvio both offer connect() and connect_with_config(), but Fluvio adds a third connect_with_profile() variant while FluvioAdmin does not. This forces users to remember which client supports profile-based connection directly.
  • Inconsistent return types for watch operations. 'watch()' returns a generic Result (likely blocking or returning a generic stream), while specific watchers return typed streams. This breaks the principle of least surprise for stream-based APIs.
  • Medium vulnerability: RUSTSEC-2026-0346 (Cargo.lock)
  • Medium vulnerability: RUSTSEC-2026-0367 (Cargo.lock)
  • Medium vulnerability: RUSTSEC-2026-0367 (Cargo.lock)
  • Off-boarding risk: anonymized user #1
  • Proliferation of listing methods. 'all()', 'list()', 'list_with_params()', and 'list_with_config()' offer overlapping functionality. 'all()' is likely a no-filter 'list()', and 'list_with_params' vs 'list_with_config' is confusingly similar.
  • Redundant creation methods with unclear distinction. 'create' takes individual parameters (name, dry_run, spec) while 'create_with_config' takes a request object. It is unclear if 'create' is a convenience wrapper or if they handle different validation/execution paths.
  • Split crates/fluvio-cli
  • Split crates/fluvio-sc
  • Split crates/fluvio-spu

New (29)

  • Classify ADR enforcement: Add partition (rfc/add-partition.md)
  • Classify ADR enforcement: Enhancing Fluvio Message Sizes (rfc/produce-message-size.md)
  • Classify ADR enforcement: materialize_view (rfc/materialize_view.md)
  • Classify ADR enforcement: offset-management (rfc/offset-management.md)
  • Critical CVE: [GHSA redacted] (Cargo.lock)
  • Critical CVE: [GHSA redacted] (Cargo.lock)
  • Critical CVE: [GHSA redacted] (Cargo.lock)
  • Medium CVE: [GHSA redacted] (Cargo.lock)
  • Medium advisory (unmaintained): RUSTSEC-2026-0441 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0386 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0395 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0396 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0397 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0399 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0400 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0401 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0402 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0404 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0405 (Cargo.lock)
  • Medium advisory (unsound): RUSTSEC-2026-0406 (Cargo.lock)
  • …and 9 more

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

fluvio-community/fluvio was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 9 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 52673942c1c7364f36f2e05da972436f761243cd — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-bb0cb800cd0f.