Skip to content
CAI
Software that uses CAICheck a score

milvus-io/milvus

48.1

Weak · 13 November 2026

625k

lines of production code

Go

with C++, C

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add C++ linting and Go linting configuration files

The repository now includes configuration files for code style and static analysis: .clang-format and .clang-tidy define C++ formatting and checking rules, .golangci.yml configures the Go linter (golangci-lint) with specific linters and exclusions, and .pre-commit-config.yaml sets up pre-commit hooks for Go linting, Python typo checking, and Python formatting. These changes standardize code quality checks across the project.

(repo-wide) · high confidence

Add NumPy file import support

Users can now import data into Milvus using NumPy (.npy) files. This change introduces a new reader in the import utility layer that parses .npy files, mapping them to collection schema fields by filename. It supports scalar types, various vector types (including Float16 and BFloat16), and handles nullable fields and dynamic fields. The implementation includes validation for UTF-8 strings and rejects unsupported data types like Arrays and SparseFloatVectors.

internal/util/importutilv2/numpy · high confidence

Add TiKV support for timestamp allocation

The tsoutil package now supports TiKV as a backend for timestamp allocation, in addition to the existing etcd support. This enables users to configure the system to use TiKV for generating timestamps, providing an alternative storage backend for this critical component.

internal/util/tsoutil · high confidence

Add binlog command-line tool for printing binlog files

A new command-line utility \binlog\ has been added to the \cmd/tools\ directory. Users can now run this tool to inspect and print the contents of binlog files by passing the file paths as arguments (e.g., \binlog file1 file2 ...\). The tool utilizes the internal storage package to process the files and provides feedback on completion or errors.

cmd/tools/binlog · high confidence

Add config-docs-generator tool to automate Milvus configuration documentation

A new Go-based tool at cmd/tools/config-docs-generator has been added to automatically generate Markdown documentation for Milvus system configurations. The tool reads the milvus.yaml file, parses configuration sections and fields (including descriptions and default values), and outputs structured documentation files, streamlining the maintenance of configuration reference materials.

cmd/tools/config-docs-generator · high confidence

Add context utilities for streaming service metadata and state

The \contextutil\ package now provides helper functions to attach and retrieve streaming service metadata within gRPC contexts. Users can now pass cluster identifiers, CreateConsumer requests, and CreateProducer requests through context using base64-encoded protobuf metadata, enabling consistent state propagation across streaming service components. Additionally, a new utility allows storing and retrieving picked server IDs in context for internal routing decisions.

internal/util/streamingutil/service/contextutil · high confidence

Add datacoord segment metadata inspection tool

A new command-line utility at cmd/tools/datameta has been added to inspect segment metadata stored in etcd. Users can connect to an etcd instance and list segment information, filtering by collection, partition, segment ID, or channel. The tool supports a --detail flag to display comprehensive binlog, statslog, and deltalog path contents for each segment, aiding in data coordination debugging and verification.

cmd/tools/datameta · high confidence

Add embedded Milvus entry point for CGO integration

A new Go source file (cmd/embedded/embedded.go) introduces a CGO-exported function, startEmbedded, which initializes the Milvus server in embedded mode by setting the configuration path to /tmp/milvus/configs/ and invoking the existing run embedded command. This enables external applications to embed and start Milvus directly via CGO calls.

cmd/embedded · high confidence

Add empty default PGO profile

A new empty default Profile Guided Optimization (PGO) configuration file has been added to the project. This file serves as a baseline configuration for PGO, allowing users to opt into or customize optimization profiles without inheriting pre-defined settings.

configs/pgo · high confidence

Add gRPC-based broadcast service implementation for streaming coordination

Introduces a new gRPC client implementation for the broadcast service in the streaming coordination layer. This change adds \GRPCBroadcastServiceImpl\, which enables broadcasting mutable messages and acknowledging immutable messages via the \StreamingCoordBroadcastService\ RPC interface, replacing or supplementing previous local implementations. The implementation handles message serialization using \IntoMessageProto\ and \IntoImmutableMessageProto\, and includes corresponding unit tests to verify broadcast and acknowledgment flows.

internal/streamingcoord/client/broadcast · high confidence

Add internal reduce utilities for hybrid search group-by support

This change introduces new internal utility files in \internal/util/reduce\ to support hybrid search \group\_by\ functionality. The additions include \group\_key.go\ for hashing and comparing composite group-by keys, \field\_data.go\ for managing group-by field data in search results (including legacy singular fallback), and \reduce\_info.go\ for carrying reduce-stage parameters like group-by field IDs and search aggregation flags. Corresponding test files are added to verify the correctness of these new utilities.

internal/util/reduce · high confidence

Add metadata migration tool for upgrading from Milvus 2.1 to 2.2

A new migration tool located in \cmd/tools/migration/meta\ has been added to handle the metadata upgrade from version 2.1 to 2.2. This tool defines the data structures and conversion logic for migrating collections, aliases, and index information, transforming legacy protobuf formats into the new internal model structures required by version 2.2.

cmd/tools/migration/meta · high confidence

Add mmap migration tool for migrating metadata to memory-mapped storage

A new command-line tool at cmd/tools/migration/mmap/tool enables users to migrate existing metadata to support memory-mapped (mmap) storage. The tool reads a configuration file, initializes a TSO allocator and metadata catalogs for both RootCoord and DataCoord (supporting both Etcd and TiKV backends), and executes the migration via the mmap package. Users can run this tool to transition their metadata storage to leverage mmap capabilities, provided the MmapDirPath is configured in their environment.

cmd/tools/migration/mmap/tool · high confidence

Add pre-push git hook to run verifiers

A new pre-push git hook has been added to the repository. When pushing code, the system will automatically execute the 'make verifiers' command to ensure code quality checks pass before the push is completed. Documentation and ownership details for this hook are also included.

githooks · high confidence

Add vector field clustering key support

Users can now enable clustering on vector fields via configuration. The new clustering utility determines the clustering key by prioritizing an explicit clustering key field, then a partition key, and finally a single vector field if the 'EnableVectorClusteringKey' and 'UseVectorAsClusteringKey' settings are active. It also includes logic to calculate distances for FloatVector types, with placeholders for other vector data types.

internal/util/clustering · high confidence

Added C++ code style and license enforcement tooling

The build-support directory now includes scripts and configuration files to enforce Apache 2.0 license headers on C++ and CMake files, and to run parallel cpplint and clang-format/clang-tidy checks. This ensures consistent code style and licensing across the C++ source base during development and CI.

internal/core/build-support · high confidence

Added FM-index scalar index for exact LIKE prefix/infix/suffix on VARCHAR

This change introduces a new FM-index implementation (vendored from fm-index-lite and libsais) to support exact LIKE pattern matching (prefix, infix, and suffix) on VARCHAR columns. The index enables document-scoped substring searches, prefix/suffix anchoring, and case-insensitive matching, providing a high-performance alternative for scalar LIKE filters.

internal/core/thirdparty/fmindex · high confidence

Added Pulsar standalone Docker image with environment-based configuration

A new Dockerfile and configuration script have been added to build a Pulsar standalone container. The image is based on AdoptOpenJDK 11 and installs Apache Pulsar version 2.8.2. It includes a Python utility that allows users to override Pulsar configuration settings via environment variables (prefixed with PULSAR\PREFIX\) at runtime, enabling flexible deployment without rebuilding the image. The container exposes ports 6650 and 8080 and runs Pulsar in standalone mode without functions worker or stream storage.

build/docker/pulsar · high confidence

Added QueryNode server-to-client wrapper for standalone mode

A new \qnServerWrapper\ in \internal/util/wrappers\ allows a QueryNode server implementation to be used as a \QueryNodeClient\. This enables the standalone mode to avoid direct gRPC calls by wrapping the local QueryNode interface, with corresponding unit tests verifying the wrapper's behavior.

internal/util/wrappers · high confidence

Added VChannelTempStorage for vchannel-to-pchannel mapping

A new temporary storage component, VChannelTempStorage, has been introduced in the streaming node's WAL layer to maintain compatibility between old and new message stream formats. This component caches the mapping between physical channels and virtual channels for collections by querying the MixCoord service, allowing the system to resolve vchannel identifiers during the transition period. The implementation includes unit tests verifying the caching behavior, context error handling, and collection lookup logic.

internal/streamingnode/server/wal/vchantempstore · high confidence

Added Windows deployment scripts for etcd, Milvus, and MinIO

New batch files have been added to the Windows deployment directory to facilitate running and cleaning up core services on Windows. Users can now launch etcd, Milvus (in standalone mode), and MinIO (pointing to the s3data directory) using dedicated scripts (run\_etcd.bat, run\_milvus.bat, run\_minio.bat), and clean up associated data and logs using cleanup\_data.bat.

deployments/windows · high confidence

Added analyzer interface definitions and mocks

The \internal/util/analyzer/interfaces\ package now defines the \Analyzer\ and \TokenStream\ interfaces, establishing the contract for text tokenization and analysis. A corresponding \MockAnalyzer\ has been added to support unit testing of components that depend on these interfaces.

internal/util/analyzer/interfaces · high confidence

Added count utility functions for Milvus result types

The internal/util/funcutil package now includes count\_util.go, which provides helper functions to extract and wrap entity counts from various Milvus result structures (internalpb.RetrieveResults, segcorepb.RetrieveResults, and milvuspb.QueryResults). These utilities validate that results contain exactly one scalar int64 column representing the count and return appropriate errors for malformed data, alongside corresponding wrappers to construct these result formats.

internal/util/funcutil · high confidence

Added internal TLS support and gRPC graceful stop timeout

The new \internal/distributed/utils\ package introduces utilities for enabling internal TLS on gRPC servers via \EnableInternalTLS\ and \CreateCertPoolforClient\, allowing secure internal communication when configured. It also adds \GracefulStopGRPCServer\, which ensures the gRPC server shuts down cleanly by waiting for in-flight RPCs to finish, with a configurable timeout that forces a hard stop if the graceful shutdown takes too long.

internal/distributed/utils · high confidence

Added lint rules for the client package

Introduced a new \rules.go\ file in the \client/ruleguard\ directory containing a collection of static analysis rules for the Go ruleguard linter. These rules help enforce code quality by detecting unnecessary type conversions, improper comparisons of \time.Time\ values, variables named \err\ that are not of the \error\ type, identical if/else bodies, self-assignments, and other common coding anti-patterns within the client package.

client/ruleguard · high confidence

Added migration tool to upgrade mmap configuration from 2.3.x to 2.4.x

A new migration utility (mmap\_230\_240.go) has been added to automatically update metadata when upgrading from Milvus 2.3.x to 2.4.x. In 2.3.x, mmap was enabled via the global \querynode.MmapDirPath\ setting, whereas 2.4.x requires explicit collection and index properties. This tool iterates through all available collections and indexes, detecting those compatible with mmap (via \vecindexmgr\) and updating their metadata to set the \common.MmapEnabledKey\ to \true\, ensuring mmap functionality is preserved during the upgrade.

cmd/tools/migration/mmap · high confidence

Added phrase match slop computation utility

The \internal/util/textmatch\ package now includes a new \phrase\_match.go\ file that exposes the \ComputePhraseMatchSlop\ function. This function calculates the minimum slop required for a phrase match between query and data texts by invoking the underlying C++ \compute\_phrase\_match\_slop\_c\ routine via CGO. It handles CGO status codes and error mapping, returning the computed slop value or an error if terms are missing or the C runtime fails.

internal/util/textmatch · high confidence

Added rolling update script for Helm-installed Milvus

A new shell script (\rollingUpdate.sh\) and documentation have been added to the \deployments/upgrade\ directory to enable zero-downtime rolling updates for Milvus instances installed via Helm. The script automates the process of updating deployments in a specific order (coordinators, nodes, proxy) using \kubectl patch\ and verifies rollout status. It requires the Milvus version to be 2.2.0 or later and explicitly excludes support for standalone Milvus using RocksMQ. Users can specify the namespace, instance name, target version, and image tag to perform the update.

deployments/upgrade · high confidence

Blocked Bloom Filter implementation with performance optimizations

The bloom filter utility now supports a Blocked Bloom Filter implementation (using the blobloom library and xxh3 hashing) alongside the existing basic implementation. This change introduces a new filter type that can be selected via the BFType enum, allowing users to benefit from potentially better performance characteristics for specific workloads. The code also includes batch testing capabilities (BatchTestLocations) to reduce function call overhead when testing multiple locations at once, and utilizes an internal JSON library for serialization. Additionally, logging has been updated to use a context-aware mlog package.

internal/util/bloomfilter · high confidence

Centralized quota configuration and metrics for database-level rate limiting

The quota subsystem now includes a dedicated constant file that maps rate scopes (cluster, database, collection, partition) and rate types (DDL, DML, DQL) to their specific configuration parameters, introducing support for database-level limits such as MaxDBRate. This change also adds runtime configuration watching for insert rates across all scopes, automatically updating the MaxInsertRate metric with appropriate labels (cluster, db, collection, partition) whenever the underlying configuration values change.

internal/util/quota · high confidence

Centralized vector index type management and support detection

The \internal/util/vecindexmgr\ package now provides a unified manager for vector index types, replacing scattered type dependencies with a centralized registry. This manager dynamically queries the underlying vector index engine to determine supported features (such as Binary, Float32, Float16, BFloat16, SparseFloat32, and Int8 vectors) and index characteristics (like NoTrain, DiskANN, AISAQ, GPU, and MMap support) for each index type. Users benefit from consistent and accurate validation of vector data types against specific index implementations, ensuring that unsupported combinations are detected early. The implementation includes a new \VecIndexMgr\ interface and its singleton instance, along with comprehensive unit tests verifying data type support for indices like FLAT, BIN\_FLAT, IVF\_FLAT, IVF\_PQ, HNSW, and DISKANN.

internal/util/vecindexmgr · high confidence

Configurable column group splitting policies for Storage V2

The storage common layer now supports configurable policies for splitting columns into groups, allowing users to control how data is organized for storage. This change introduces a new \ColumnGroupSplitPolicy\ interface and several default policies, including splitting by local format, average size, and selected data types (such as vectors and text). These policies are applied when creating column groups, enabling more efficient storage and retrieval based on field characteristics and configuration settings.

internal/storagecommon · high confidence

DataCoord metastore catalog refactored for composite atomic updates and new entity support

The DataCoord KV metastore catalog has been rewritten to support composite, atomic updates via a new \Update\ method that batches multiple entity changes (segments, channels, data views, compaction targets, and segment change groups) into a single transaction, ensuring consistency during recovery and collection drops. This change introduces persistence support for new entities including DataViews, compaction targets, and segment change groups, while also refining segment handling by skipping per-FieldBinlog KV writes for V3 manifest-backed segments to reduce write amplification.

internal/metastore/kv/datacoord · high confidence

Enable OpenSSL FIPS mode for Milvus

Milvus now supports OpenSSL FIPS mode, allowing users to configure the application to use FIPS-compliant cryptographic algorithms. This is achieved by adding a new configuration file (configs/ssl/openssl-fips.cnf) that activates the FIPS provider and sets the default algorithm properties to require FIPS compliance, ensuring adherence to Federal Information Processing Standards for security-sensitive deployments.

configs/ssl · high confidence

Etcd client creation now supports authentication and TLS configuration

The \internal/util/dependency/kv\ package now passes authentication credentials (username/password) and TLS settings (certificates, keys, CA, min version) to the Etcd client constructor. This enables secure connections to Etcd clusters that require authentication and encrypted communication, previously only supporting unauthenticated or basic connections.

internal/util/dependency/kv · high confidence

Expose Go expression parser to C++ via new C wrapper

A new C wrapper layer has been added in the \internal/parser/planparserv2/cwrapper\ directory, enabling C++ applications to invoke the Go-based plan parser. This addition introduces \milvus\_plan\_parser.h\ and \milvus\_plan\_parser.cpp\ which provide a \PlanParser\ class wrapping the Go functions exported from \wrapper.go\. Users can now register schemas, parse filter expressions into serialized \PlanNode\ protobufs, and parse search expressions (including vector field and query info) directly from C++ code, with thread-safe schema management handled via atomic reference counting.

internal/parser/planparserv2/cwrapper · high confidence

Extracted shard client logic into a dedicated package with channel-based node blacklist

The shard client management and load balancing logic has been extracted from the proxy into a new \internal/proxy/shardclient\ package. This new package centralizes QueryNode connection management, shard leader caching (keyed by collection ID to prevent alias repoint races), and request routing. It introduces a channel-based node blacklist that automatically expires blacklisted nodes after a configurable duration, improving fault tolerance during replica failures. The package also includes a new \LookAsideBalancer\ that uses cost metrics (response time, service time, NQ) to prefer lightly loaded nodes, falling back to round-robin when workloads are similar, alongside a standard \RoundRobinBalancer\.

internal/proxy/shardclient · high confidence

Go SDK client module refactoring and new membership filter support

The Go SDK client module has been reorganized into internal sub-packages (conc, crypto, merr, retry, rowutil, typeutil) to decouple the client from the server-side milvus/pkg module, ensuring the SDK only depends on the public API protobuf module. This change introduces a new exact membership filter implementation using the MRB1 Roaring bitmap format in the roaringfilter package, alongside the existing MBF1 Split-Block Bloom Filter in the sbbf package, enabling efficient client-side construction of membership\_match expressions. The merr package provides a unified client-side RPC error handling layer with modern code mapping and retryability checks, while the retry package offers a configurable retry mechanism for schema cache refreshes. Additionally, the typeutil package adds vector conversion helpers for Float32, Float16, BFloat16, and Int8 vectors, and the rowutil package enhances struct tag parsing for row-based API conversions.

client/milvusclient · high confidence

Go SDK column handling refactored to generic base with array and dynamic JSON support

The Go SDK client's column handling has been rewritten to use a generic base implementation, introducing dedicated column types for Array fields (Bool, Int8, Int16, Int32, Int64, Float, Double) and a new ColumnDynamic type for extracting typed values from dynamic JSON fields. This change also standardizes nullable column behavior across scalar and vector types, ensuring consistent handling of valid/invalid data states during insert and query operations.

client/column · high confidence

Go SDK entity models for collection attributes, external tables, and functions

The Go client now exposes Go structs for managing collection attributes (TTL and auto-compaction), external table refresh states, and search-time functions (including FunctionScore). Users can define these entities to configure collection behavior, track external data refresh jobs, and apply reranking or boosting functions during queries. Also in this area: Go SDK client module refactoring and new membership filter support; Go SDK v3 index definitions and parameter builders.

client/entity, github.com/milvus-io/milvus/client/v3 · medium confidence

Go SDK v3 index definitions and parameter builders

The \client/index\ package now provides typed constructors and parameter builders for a comprehensive set of vector and scalar index types, including Flat, BinFlat, IVF variants (Flat, PQ, SQ8, RaBitQ), HNSW variants (plain, SQ, PQ, PRQ with optional refine), DiskANN, AISAQ, AutoIndex, GPU indexes (BruteForce, IVF-Flat, IVF-PQ, Cagra), MinHash LSH, and JSON Path indexes. This change introduces a unified \AnnParam\ interface for search-time parameters, a \WithExtraIndexParams\ helper to inject raw build parameters without abandoning typed constructors, and fixes to ensure index types and build parameters are correctly reported to the server (e.g., correcting RaBitQ and GPU index type reporting and omitting unset build params to avoid server validation failures).

client/index · high confidence

Idempotent write support in the streaming WAL

The streaming node's Write-Ahead Log now includes an idempotency interceptor that deduplicates insert and transaction commit messages using idempotency keys. This prevents duplicate rows when clients retry writes, while correctly bypassing deduplication for replicated messages to avoid data loss. The interceptor also handles transaction body buffering and result merging, and restores idempotency state from snapshots during recovery to ensure consistency after outages. Also in this area: 3 commits (1 fix) modifying internal/streamingnode.

internal/streamingnode/server/wal/interceptors/idempotency · medium confidence · unverified

Initial RPM packaging support for Milvus standalone server

This change introduces the initial RPM build infrastructure for Milvus, adding the \milvus.spec\ file and associated systemd service units for the main server, etcd, and MinIO components. It defines the package structure, including installation of binaries, shared libraries, configuration files, and ldconfig settings, along with pre/post uninstall scripts to manage service states and clean up residual files. A setup script is also included to configure the build environment with required dependencies like GCC 11, OpenBLAS, TBB, Boost, and Go 1.26.6.

build/rpm · high confidence

Initial configuration and governance files for Milvus

This change introduces the foundational configuration and governance files for the project. It adds the \milvus.yaml\ file, which serves as the central configuration for core components including etcd, TiKV, MinIO/S3 storage, and message queues (Pulsar, Kafka, Rocksmq, Woodpecker). Additionally, it includes \OWNERS\ to define code review responsibilities, \glog.conf\ for logging settings, and an empty \hook.yaml\ placeholder.

configs · high confidence

Introduce Arrow-based function chain pipeline for search reranking

Added a new internal function chain execution engine in \internal/util/function/chain\ that replaces the legacy rerank implementation with an Apache Arrow-based pipeline. This change introduces a \FuncChain\ structure that executes a sequence of operators (such as Map, Filter, and Merge) on search result data represented as immutable Arrow DataFrames. The new system includes a \DataFrameInputPlan\ compiler to resolve schema-backed inputs (including dynamic fields and JSON path projections) and a converter layer to handle type mapping between Milvus protobuf types and Arrow data types, enabling more efficient and flexible search aggregation and reranking operations.

internal/util/function/chain · high confidence

Introduce C++ FFI binding header for Tantivy search engine

A new C++ header file (tantivy-binding.h) is added to expose the Tantivy search engine via a stable C ABI. This binding defines structured error codes (TantivyBindingErrorCode) for reliable error classification, data types for indexing (Text, Keyword, I64, F64, Bool, JSON), and specific array structures (RustStringArray) to safely handle index terms that may contain interior NUL bytes. It provides functions for creating JSON key and ngram index writers, loading and reloading indexes, and executing boolean, I64, and F64 terms queries, enabling the C++ side to interact with the Rust-based search backend without parsing human-readable error messages.

internal/core/thirdparty/tantivy/tantivy-binding/include · high confidence

Introduce CDC service server implementation

The distributed CDC component now includes a concrete server implementation in \internal/distributed/cdc/service.go\. This file defines the \Server\ struct and its lifecycle methods (\NewServer\, \Prepare\, \Run\, \Stop\, \Health\), handling initialization of the etcd client, the core CDC server, and the replication manager, while managing component state and graceful shutdown via context cancellation.

internal/distributed/cdc · high confidence

Introduce CGO bytes converter with zero-copy optimization

Added a new \cgoconverter\ package that provides a \BytesConverter\ to efficiently convert C byte pointers to Go slices without copying data, using a lease-based system to manage memory lifecycles. The implementation includes methods to extract C pointers without releasing them and handles concurrent access safely, accompanied by tests verifying correctness and concurrency.

internal/util/cgoconverter · high confidence

Introduce CGO wrapper for vector clustering analysis

Added a new \analyzecgowrapper\ package that bridges Go and the Milvus C++ core for vector clustering tasks. This wrapper exposes an \Analyze\ function to initiate clustering jobs, handling the marshaling of \AnalyzeInfo\ and passing encryption context (zone ID, collection ID, and key) to the C++ layer. It provides a \CodecAnalyze\ interface with \Delete\ and \GetResult\ methods to manage the lifecycle of the analysis task and retrieve centroids and offset mapping files. The implementation includes specific fixes for CGO memory safety, such as avoiding dangling pointers and ensuring proper cleanup of C-allocated strings, along with context-aware logging via \mlog\.

internal/util/analyzecgowrapper · high confidence

Introduce Go SDK row-based API with schema parsing and caching

The Go SDK now supports a row-based data model, allowing users to insert and query data using Go structs instead of manual column arrays. This change adds core files in \client/row\ that parse struct tags (e.g., \milvus:"primary\_key"\, \milvus:"dim:16"\) to automatically generate Milvus schemas, convert row data into columns for insertion, and cache receiver candidates to optimize reflection-based unmarshalling. It also introduces support for nullable pointer types, dynamic fields, and various vector types (including Int8, FP16, and BF16) within the row-based API.

client/row · high confidence

Introduce Go wrapper for C++ index building and management

Added new Go files (helper.go, index.go) in internal/util/indexcgowrapper that provide a Go interface to the C++ index builder. This includes functions to create, serialize, upload, and delete indexes via CGO, along with helper utilities for handling C status errors and retrieving local disk usage statistics.

internal/util/indexcgowrapper · high confidence

Introduce Go-based analyzer wrapper for C tokenizer integration

The analyzer package now provides a Go wrapper layer that delegates text analysis operations to the underlying C-based tokenizer implementation. This change introduces functions to initialize analyzer options, create and validate analyzers, and manage global resource information, effectively bridging the Go application layer with the C segcore tokenizer. Users benefit from a unified interface for configuring and utilizing tokenizers (such as standard, jieba, grpc, and lindera) while maintaining the performance characteristics of the C backend.

internal/util/analyzer/canalyzer · high confidence

Introduce Go-layer ORDER BY pipeline types

Added the \internal/util/reduce/orderby\ package, which defines the core types and parsing logic for the new Go-layer ORDER BY pipeline. This includes the \OrderByField\ struct and utilities to parse sort specifications (direction and null handling) from string inputs or plan protobufs, establishing the foundation for query sorting in the Go implementation.

internal/util/reduce/orderby · high confidence

Introduce Go-layer aggregation reducer and dynamic CGO pool in Proxy

The Proxy now handles aggregation result reduction directly in Go via the new \MilvusAggReducer\, which processes group-by fields and computes averages from sum/count metrics before returning query results. Additionally, a dynamic CGO thread pool (\GetDynamicPool\) has been added to manage C-level index validation calls (e.g., \CheckVecIndexWithDataTypeExist\) more efficiently, ensuring thread safety for CGO operations.

internal/proxy · high confidence

Introduce Go-side metastore model types for core entities

This change adds new Go structs and serialization logic in the metastore model package for Alias, Collection, Credential, Database, Field, Function, Index, Partition, and Row-Level Security (RLS) policies. These models provide a structured, Go-native representation of metadata that is marshaled to and unmarshaled from protobuf definitions, replacing or supplementing previous ad-hoc handling. Users benefit from consistent schema handling, proper deep-copying of complex fields (such as type schemas and RLS policies), and reliable equality checks across these core entities. Also in this area: 1 commit (0 fixes) modifying github.com/milvus-io/milvus; Introduce Go-layer aggregation reducer and dynamic CGO pool in Proxy; Extract proxy DQL tasks into a dedicated package; QueryCoord autoscale adds pre-check for segment load resource estimation; Introduce Arrow-based function chain pipeline for search reranking; and 1 more.

github.com/milvus-io/milvus · medium confidence · unverified

Introduce L0 write buffer and dedicated delta buffer for growing segments

The DataNode now uses a new L0 write buffer implementation for growing segments, which creates dedicated L0 segments to buffer insert and delete data before flushing. This includes a new DeltaBuffer component specifically for handling delete operations (primary keys and timestamps) and an InsertBuffer for insert data, both managed by a central BufferManager. The L0 write buffer integrates with the sync manager to trigger flushes based on memory pressure and buffer policies, replacing the previous monolithic buffer approach for growing segments.

internal/flushcommon/writebuffer · high confidence

Introduce PK Oracle with Bloom Filter and External Segment Candidates

The \internal/querynodev2/pkoracle\ package now provides a unified Primary Key (PK) to segment mapping system. It introduces a \Candidate\ interface with two implementations: \BloomFilterSet\, which uses bloom filters to efficiently check for PK existence in standard segments (supporting both current and historical statistics), and \ExternalSegmentCandidate\, which uses deterministic virtual PK mapping for external collections. The \PkOracle\ orchestrates these candidates, allowing registration, filtering (by segment ID, type, partition, or worker), and batch lookups, while managing memory resource charging and refunding for bloom filter sets.

internal/querynodev2/pkoracle · high confidence

Introduce Storage V2 packed reader and writer with FFI integration

This change introduces the Storage V2 packed reader and writer components, enabling the system to read and write data in a packed columnar format via the milvus-storage FFI interface. It adds C ABI headers for Arrow data exchange, Go bindings for column group entries, and FFI wrappers for filesystem operations, index metadata, and statistics. Users benefit from improved storage efficiency and performance through the new packed format, along with better integration with external data sources via the FFI layer.

internal/storagev2/packed · high confidence

Introduce Tantivy C++ binding and build infrastructure

This change adds the CMake build system (CMakeLists.txt) and C++ wrapper headers (tantivy-wrapper.h, tokenizer.h, phrase\_match.h, etc.) that integrate the Rust-based Tantivy search engine into the Milvus core. It provides the TantivyIndexWrapper API for creating and managing inverted indexes, supports text analysis via pluggable tokenizers (including Jieba), and exposes query capabilities such as phrase matching, regex, and range queries through a stable FFI boundary with typed error handling.

internal/core/thirdparty/tantivy · high confidence

Introduce TiKV as a transactional key-value storage backend

This change adds a new TiKV-based implementation for the internal key-value store, providing transactional support for metadata operations. It introduces a new \txnTiKV\ client that wraps the TiKV Go client, enabling atomic multi-key saves and removes, configurable request timeouts, and hardened behavior such as respecting context cancellation during prefix scans and preventing mutation of caller-provided key slices. The update includes comprehensive test coverage for these behaviors, ensuring reliability for users migrating to or utilizing TiKV for storage.

internal/kv/tikv · high confidence

Introduce WAL Manager for streaming node lifecycle and state management

The streaming node now includes a new \walmanager\ package that centralizes the lifecycle management of Write-Ahead Log (WAL) instances. This manager handles opening, closing, and removing WALs per channel while enforcing term-based consistency to prevent stale operations. It integrates a chain of interceptors—including idempotency, redo, locking, replication, time-tick, shard validation, and partial updates—to ensure data integrity and correct ordering during writes. The implementation also provides metrics reporting and ensures that open operations are properly canceled during shutdown.

internal/streamingnode/server/walmanager · high confidence

Introduce WAL adaptor layer with dynamic backend selection and legacy message compatibility

The streaming node now includes a new WAL adaptor package that acts as a bridge between the internal WAL interface and pluggable message queue backends. This adaptor enables dynamic WAL backend selection at runtime by resolving the target implementation from the channel's checkpoint or a default selector, and it supports opening read-write or read-only WAL instances with appropriate interceptors. To ensure smooth upgrades, the adaptor includes logic to convert legacy v0 message formats (such as create/drop collection, insert, delete, and partition operations) into the current v1 message structure, allowing the streaming service to consume data from older clusters. Additionally, the adaptor introduces adaptive rate limiting for WAL append operations, with configurable high/low watermarks for recovery and slowdown states to manage memory pressure and throughput.

internal/streamingnode/server/wal/adaptor · high confidence

Introduce WAL shard stats manager for segment lifecycle and memory pressure control

The streaming node now includes a new stats management subsystem in the WAL shard interceptors to track insert and delete metrics for growing segments. This manager monitors segment sizes against configurable high and low watermarks, triggering automatic sealing based on capacity, time, or L0 blocking pressure. It also integrates with system memory monitoring to enforce flush decisions when node memory usage exceeds defined thresholds, and exposes these metrics via Prometheus for observability.

internal/streamingnode/server/wal/interceptors/shard/stats · high confidence

Introduce WAL-based DDL framework for collection and database operations

RootCoord now supports a new Write-Ahead Log (WAL) based DDL framework for creating, altering, and dropping collections and databases. This change introduces new internal components (such as \alter\_collection\_task.go\ and \alter\_collection\_analyzer\_resources.go\) to handle schema mutations and file resource reservations, and updates the \Broker\ interface to broadcast these changes via the new streaming broadcaster. This enables more robust and consistent metadata management for collection lifecycle operations.

internal/rootcoord · high confidence

Introduce Write-Ahead Buffer for streaming service

Added a Write-Ahead Buffer (WAB) component in the WAL interceptors to buffer and order messages by time tick before they are fully persisted. This buffer supports reading messages from a specific offset or exclusive time tick, handles eviction of old messages based on capacity and keep-alive duration, and ensures the last persisted message is retained to facilitate catch-up scanning. The implementation includes a pending queue for message storage, a reader for sequential access, and comprehensive tests to verify message ordering, eviction logic, and concurrent read/write operations.

internal/streamingnode/server/wal/interceptors/wab · high confidence

Introduce async CGO futures with thread-pool management and error classification

The \internal/util/cgo\ package now provides an asynchronous execution model for C++ Segcore operations via a new \Future\ abstraction. Users benefit from non-blocking search and load operations that integrate with Go contexts for cancellation and timeout handling. The implementation includes dedicated thread pools for search and load tasks, with sizes dynamically adjustable via configuration parameters (\MaxReadConcurrency\, \CGOPoolSizeRatio\, \MiddlePriorityThreadCoreCoefficient\). It also introduces standardized error classification, mapping C++ Segcore error codes to specific Milvus errors (e.g., \ErrSegcore\, \ErrSegcoreFollyOtherException\) and registering observers to log and metricize unmapped or unexpected error codes for better observability.

internal/util/cgo · high confidence

Introduce automated config file generation tool

Added a new CLI tool in \cmd/tools/config\ that generates \milvus.yaml\ and \configs.csv\ directly from the \paramtable\ definitions. This ensures the configuration file remains consistent with the codebase by reflecting current defaults, versions, and documentation comments, and includes a test to verify that the generated YAML matches the checked-in file.

cmd/tools/config · high confidence

Introduce centralized path utility for node-local cache directories

A new \pathutil\ package has been added to standardize the generation of paths for various node-local cache types, including growing MMap, local chunks, BM25, file resources, and expression caches. This utility constructs specific subdirectories under the shared \LocalCacheRootPath\ based on the node ID and cache type, replacing ad-hoc path construction and ensuring consistent directory structures for local storage operations.

internal/util/pathutil · high confidence

Introduce containerized build environment with Docker Compose and dev containers

The build directory now provides a standardized, containerized workflow for building and testing Milvus. New scripts (builder.sh, build\_image.sh, build\_image\_gpu.sh) orchestrate builds using Docker Compose, supporting multiple OS targets (ubuntu20.04, ubuntu22.04, amazonlinux2023) and architectures (amd64, arm64). The setup includes a dev container configuration for local development with VS Code integration, automated caching for compiler and Conan packages, and support for AddressSanitizer (ASAN) builds. Additionally, the build process now reports the final Docker image size and includes scripts for provisioning KinD clusters for end-to-end testing.

build · high confidence

Introduce custom gRPC load-balancing base with unready-subconnection awareness

The streaming service now uses a custom gRPC load-balancing base implementation in the \balancer\ package. This change introduces a \PickerBuildInfo\ structure that exposes both ready and unready sub-connections to the picker, allowing the load balancer to make more informed routing decisions by considering the state of all available connections rather than only those that are fully ready. The implementation includes a base builder, a base balancer that tracks connectivity states and resolver errors, and a test suite to validate attribute preservation and state handling.

internal/util/streamingutil/service/balancer · high confidence

Introduce dependency factory for unified MQ and storage initialization

The \internal/util/dependency\ package now provides a \DefaultFactory\ that centralizes the creation of message queue streams and storage chunk managers. This factory supports multiple message queue backends—RocksMQ, Pulsar, Kafka, and Woodpecker—and automatically selects the active backend based on configuration and deployment mode (standalone vs. cluster). It also includes a \HealthCheck\ function that resolves the 'default' MQ type to the actual enabled backend before reporting cluster status, ensuring accurate health monitoring.

internal/util/dependency · high confidence

Introduce dist package for QueryCoord distribution management

The \internal/querycoordv2/dist\ package has been added to centralize the logic for managing data distribution across QueryNodes. This includes a \Controller\ to start, stop, and sync distribution instances per node, and a \distHandler\ that pulls distribution data from nodes and dispatches tasks. The implementation supports dynamic interval updates for pull and dispatch loops, uses rated logging to manage high-frequency log volume, and includes unit tests and mock generation to ensure reliability.

internal/querycoordv2/dist · high confidence

Introduce embedded etcd support for standalone deployments

The \internal/kv/etcd\ package now supports running an embedded etcd server in addition to connecting to an external one. When the \etcd.use.embed\ configuration is enabled, the system starts an in-process etcd instance (using \go.etcd.io/etcd/server/v3/embed\) and uses it as the KV store. This includes a new \EmbedEtcdKV\ implementation, factory methods (\NewMetaKvFactory\, \NewWatchKVFactory\) that select the backend based on configuration, and a retry mechanism for server startup. The change also adds a security fix in \getContextWithTimeout\ to strip RBAC authentication metadata from the context before passing it to etcd requests, preventing credential leakage. Comprehensive tests are included to verify save/load operations, prefix walks, and data persistence across restarts.

internal/kv/etcd · high confidence

Introduce file resource manager with sync and reference modes

A new file resource management system has been added to the internal utilities, providing a centralized manager for handling file resources in either 'sync' mode (downloading files locally) or 'ref' mode (installing before use and cleaning up when no longer needed). This change introduces the Manager interface, listener registration for sync events, and the underlying logic for downloading, releasing, and tracking file resources, laying the groundwork for consistent file resource synchronization across components like query nodes and data nodes.

internal/util/fileresource · high confidence

Introduce function runner lifecycle management and concurrency controls for BM25 and MinHash

This change introduces a new \FunctionRunnerManager\ in \internal/util/function\ that manages the lifecycle of function runners (BM25, MinHash) using isolated lifecycle keys (e.g., WAL, delegator) and schema version signatures, ensuring runners are correctly initialized, updated, and released. It adds a configurable concurrency pool for analyzers to handle parallel text processing, and implements a local store fallback to handle legacy insert messages that lack embedded function data. Additionally, it adds validation for MinHash parameters (rejecting \num\_hashes\ that overflow dimension checks) and ensures UTF-8 compliance for BM25 inputs.

internal/util/function · high confidence

Introduce hierarchical rate limiter tree for multi-level quota enforcement

The \internal/util/ratelimitutil\ package now includes a \RateLimiterTree\ and \RateLimiterNode\ implementation that organizes rate limiting in a hierarchy: cluster → database → collection → partition. This structure allows the system to apply and traverse rate limits and quota states (such as deny-to-write, deny-to-read, and deny-to-DDL) at specific granularities, enabling more precise control over resource usage across different levels of the data organization.

internal/util/ratelimitutil · high confidence

Introduce in-memory key-value store implementation

The \internal/kv/mem\ package now provides a new \MemoryKV\ implementation of the \kv.TxnKV\ interface, backed by a concurrent-safe B-Tree. This allows components to use an ephemeral, in-memory storage backend for key-value operations such as \Load\, \Save\, \Remove\, and range-based queries, which is particularly useful for testing or scenarios where persistent storage is not required.

internal/kv/mem · high confidence

Introduce internal CDC replication controller and client infrastructure

This change adds the internal control plane for the new CDC replication feature. The \controller\ component now watches etcd for replicate pchannel metadata events (PUT/DELETE) to dynamically create and remove replicators. A new \MilvusClient\ implementation handles outbound connections to target clusters, supporting per-cluster TLS configuration and gRPC authority settings. The \ReplicateManagerClient\ and \ChannelReplicator\ manage the lifecycle of replication streams, including initialization, message consumption from the source WAL, and forwarding to the target. Mocks for these components are generated to support unit testing. Also in this area: Fixes replicate stream backoff, RLS model copying, and search aggregation metric direction.

internal/cdc/replication/replicatestream · high confidence

Introduce job-based load, release, and update workflows in QueryCoord

The \internal/querycoordv2/job\ package now defines a structured job execution model for collection and partition lifecycle operations. It introduces a \Job\ interface with \PreExecute\, \Execute\, and \PostExecute\ phases, along with a \BaseJob\ implementation for common context and error handling. Specific job types include \LoadCollectionJob\ for handling load requests (including incremental resource group expansion and local replica config overrides), \ReleaseCollectionJob\ for releasing collections and cleaning up metadata and caches, \UpdateLoadConfigJob\ for modifying replica numbers and resource group assignments with rollback support, and \SyncNewCreatedPartitionJob\ for synchronizing new partitions into the query coordinator's metadata. A \Scheduler\ manages these jobs, ensuring sequential execution per collection. This refactors the previous monolithic load/release logic into discrete, testable units, improving traceability and error handling for load and release operations. Also in this area: 1 commit (0 fixes) modifying internal/flushcommon, internal/querycoordv2.

internal/querycoordv2/job · medium confidence · unverified

Introduce lazy gRPC connection and service wrappers

Added a new \lazygrpc\ package that provides lazy-initialized gRPC connections and services. The \Conn\ interface wraps a gRPC client connection, initializing it asynchronously in the background to avoid blocking startup or creating dependency cycles, while \Service\ allows for lazy instantiation of gRPC services based on the connection. This includes retry logic with exponential backoff for dial failures and proper context-aware cancellation and closing.

internal/util/streamingutil/service/lazygrpc · high confidence

Introduce meta migration tool with backup, run, and rollback commands

A new meta migration tool is added to the CLI, providing a structured way to manage data migrations via a YAML configuration file. The tool introduces three distinct operations: 'run' to execute the migration (with an optional pre-migration backup), 'backup' to create a snapshot of the current state, and 'rollback' to revert changes. The implementation includes session management to ensure safe execution and prevents concurrent migrations.

cmd/tools/migration/command · high confidence

Introduce metadata migration tool for upgrading from Milvus 2.1 to 2.2

A new migration tool has been added to the \cmd/tools/migration\ directory to facilitate upgrading metadata from version 2.1 to 2.2. The tool supports etcd as the metadata store, including authentication and TLS configuration. It provides a \Backend\ interface with implementations for loading, saving, cleaning, and backing up metadata, specifically handling the structural differences between 2.1 and 2.2 versions (e.g., collection, alias, and index metadata). The tool includes a backup/restore mechanism using a custom binary format with a protobuf-defined header, and an example configuration file is provided to guide users on setting up the migration parameters.

cmd/tools/migration/backend · high confidence

Introduce milvus-storage as a third-party dependency

Added the milvus-storage library as a new third-party component, fetched via Git submodule and built using CMake FetchContent. The integration configures the build to support Azure filesystems (controlled by ENABLE\_AZURE\_FS), explicitly links the AWS S3 SDK to ensure symbol resolution, and links the Apple Network framework on macOS. A pkg-config template is also included to expose the library's include and library paths to consumers.

internal/core/thirdparty/milvus-storage · high confidence

Introduce new Go MilvusClient SDK (v3) with dedicated build and lint configuration

The client directory now contains the new Go MilvusClient SDK (v3), including a README with usage instructions, an OWNERS file, a Makefile for linting and mock generation, and a .golangci.yml configuration file. The SDK requires Go 1.24.12 or higher and provides a new client package at github.com/milvus-io/milvus/client/v3/milvusclient. The lint configuration enforces specific linters, restricts certain error handling patterns and deprecated packages, and excludes generated files and specific test patterns.

client · high confidence

Introduce new ID allocator components in internal/allocator

This change adds the core ID allocation logic to the \internal/allocator\ package, introducing a \CachedAllocator\ for batching requests, a \GlobalIDAllocator\ that delegates to the TSO service for global unique IDs, and an \IDAllocator\ that wraps the cache to batch-allocate from RootCoord. It also includes a \localAllocator\ for in-process range-based allocation, along with the \Interface\ definition, mock implementations for testing, and an OWNERS file to establish code ownership.

internal/allocator · high confidence

Introduce new Parquet import reader with comprehensive data type support

The internal Parquet import utility has been replaced with a new reader implementation that supports a wide range of data types, including nullable fields, default values, geometry, timestamptz, and various vector types (FloatVector, BinaryVector, Int8Vector, Float16Vector, BFloat16Vector, and SparseFloatVector). The new reader also handles complex nested structures like StructArrays and Arrays, with specific validation for UTF-8 strings, array capacities, and vector dimensions. It leverages the Apache Arrow Go v17 library for efficient columnar data processing and includes optimizations such as leaf pruning for struct sub-fields and memory management improvements to prevent OOM issues during large imports.

internal/util/importutilv2/parquet · high confidence

Introduce new WAL interface and scanner framework for streaming node

The streaming node now uses a new Write-Ahead Log (WAL) abstraction defined in the \internal/streamingnode/server/wal\ package. This change introduces core interfaces (\WAL\, \ROWAL\, \Opener\, \OpenerBuilder\, and \Scanner\) that decouple the streaming node from specific message queue implementations. Users benefit from a more modular architecture where WAL backends (like RMQ or Pulsar) are registered via a static registry and accessed through a unified API, supporting features such as rate-limiting controls for scanners, message filtering, and salvage checkpoints for force failover scenarios.

internal/streamingnode/server/wal · high confidence

Introduce new idalloc package with local-remote ID and timestamp allocation

The internal/util/idalloc package has been added to provide a unified mechanism for allocating IDs and timestamps. It implements a hybrid strategy that first attempts to allocate from a local cache to reduce latency, falling back to batch requests against the remote MixCoord (via AllocID and AllocTimestamp RPCs) when the local range is exhausted. The package includes a barrier mechanism (BarrierUntil) to ensure allocated IDs respect ordering constraints, and exposes a wrapper (mAllocator) to adapt this new allocator to the existing allocator.Interface contract.

internal/util/idalloc · high confidence

Introduce new metacache subsystem for segment management and BM25 stats

This change introduces a new \metacache\ package within \internal/flushcommon\ that provides a dedicated in-memory cache for segment metadata. It adds a \MetaCache\ interface and implementation to manage segment lifecycle (add, update, remove, query) using composable filters (by ID, state, partition) and actions (updating binlogs, state, checkpoints, row counts). A key addition is support for BM25 sparse vector statistics, including a \SegmentBM25Stats\ struct for merging and serializing stats, and corresponding \bm25logs\ fields in \SegmentInfo\. The cache also tracks a \FlushSourceMode\ (WriteBuffer vs. Growing) to determine the payload source for flush tasks and includes a \pkoracle\ sub-package with bloom filter-based primary key existence checks (\BloomFilterSet\) and lazy loading (\LazyPkStats\). Also in this area: 12 commits (3 fixes) modifying internal.

internal/flushcommon/metacache, internal/flushcommon/metacache/pkoracle · medium confidence

Introduce primary key index core storage library

Added the internal pkindex storage engine, which manages per-vchannel primary key indexing using a generation-based lifecycle (active, frozen, committed) backed by Pebble. The change includes order-preserving key encoding for int64 and varchar primary keys, hand-encoded value layouts for PK entries and tombstones, and the engine's read/write interfaces along with autogenerated mocks for testing.

internal/mocks/pkindex · high confidence

Introduce proxy-side search aggregation engine

Added a new Go-based aggregation layer in \internal/proxy/search\_agg\ that performs hierarchical aggregation (grouping, metric accumulation, top\_hits, and sub-aggregations) on search results after cross-shard reduction. This implementation handles context building, metric plan compilation, ordering, and size constraints, enabling complex aggregation queries to be processed efficiently within the proxy rather than relying solely on downstream components.

_internal/proxy/search\agg · high confidence

Introduce query node metrics collectors

The query node now includes a new internal collector package that provides thread-safe metrics tracking capabilities. This adds support for average-value tracking (with registration, addition, and reset operations), integer counters (supporting add, increment, decrement, set, and remove operations), and rate limiting via the existing rate collector utilities. These components are initialized at startup to monitor insert and delete consume throughput, enabling more granular observability and rate-limiting logic within the query node.

internal/querynodev2/collector · high confidence

Introduce resumable streaming consumer with automatic recovery and metrics

The streaming consumer now automatically recovers from stream breaks and log node rebalances by implementing a resumable consumer that tracks the last confirmed message ID and time tick to resume consumption from the correct position. This change includes a new consumer implementation that handles reconnection logic with exponential backoff, a message handler that ensures ordered processing and tracks delivery state, and Prometheus metrics to monitor consumer availability, bytes consumed, and reconnection attempts.

internal/distributed/streaming/internal/consumer · high confidence

Introduce semantic highlight function with Zilliz provider support

Added a new semantic highlight capability in the function utilities that leverages the Zilliz model service to identify and score relevant text segments. The implementation supports dynamic fields, allows configuration of the maximum client batch size (defaulting to 64), and includes comprehensive unit tests for the new highlight provider and semantic highlight logic.

internal/util/function/highlight · high confidence

Introduce server ID-based gRPC load balancing

Added a new 'server\_id\_picker' gRPC balancer that routes RPCs based on specific server identifiers. The picker supports two modes: round-robin distribution across available servers when no target is specified, and direct routing to a specific server when a server ID is provided in the request context. It also handles connection states by waiting for unready connections or returning an error if the target server does not exist.

internal/util/streamingutil/service/balancer/picker · high confidence

Introduce server-side command routing and persistent client configuration storage

The rootcoord telemetry package now includes a command router and store that allow the server to push persistent configurations (such as telemetry enablement, heartbeat intervals, and sampling rates) and one-time commands (like error retrieval or collection metrics control) to connected clients. Persistent configurations are stored in etcd and cached in memory, with a hash mechanism to detect changes and notify clients, while one-time commands are handled in-memory with optional TTLs. This enables dynamic, server-driven control over client-side telemetry behavior without requiring client restarts.

internal/rootcoord/telemetry · high confidence

Introduce standalone streaming coordination server

A new streaming coordination server has been added to manage streaming-specific responsibilities, including channel assignment and message broadcasting. This server initializes and recovers its balancer and broadcaster components, registers gRPC services for assignment and broadcast operations, and relies on a builder pattern for configuration via etcd, metadata KV, and session utilities.

internal/streamingcoord/server · high confidence

Introduce standalone text analyzer utility

Added a new \internal/util/analyzer\ package that provides a standalone \Run\ function for tokenizing text inputs. This utility allows users to analyze text strings using configurable tokenizer parameters (e.g., \standard\) and retrieve results including tokens, optional detailed metadata, and hash values, while properly managing native resource lifecycle and context cancellation.

internal/util/analyzer · high confidence

Introduce streaming assignment service client for node discovery and replication configuration

The streaming client now includes a new assignment service implementation that manages the discovery of streaming nodes and their channel assignments. Users benefit from a dedicated client that watches for assignment changes, reports assignment errors to the server, and retrieves the latest streaming service version. Additionally, this client supports updating and retrieving cluster replication configurations, including a 'fresh read' option to ensure strong consistency immediately after configuration updates.

internal/streamingcoord/client/assignment · high confidence

Introduce streaming channel balancer and manager

The streaming coordination service now includes a new channel balancer package that manages physical channel (pchannel) lifecycle, assignment, and replication availability. This change adds a ChannelManager that recovers channel state from the catalog, tracks vchannel statistics to inform balancing decisions, and exposes a global singleton for retrieving the current cluster channel topology. It also introduces metrics for pchannel status, vchannel counts, and assignment versions to support observability of the streaming service's internal state.

internal/streamingcoord/server/balancer/channel · high confidence

Introduce streaming consumer server with metrics and transaction support

The consumer handler now implements a dedicated gRPC server that manages the lifecycle of log message consumption, including creating consumers, scanning WAL data, and sending responses. This change adds Prometheus metrics to track consumer activity, such as total scanners, in-flight messages, and bytes consumed. It also introduces support for sending transactional messages atomically (begin, body, commit) and standardizes error handling and logging within the consumer stream.

internal/streamingnode/server/service/handler/consumer · high confidence

Introduce streaming coordination balancer for WAL channel assignment

A new load balancer component has been added to the streaming coordination server to manage the assignment of virtual channels to streaming nodes. This component implements a configurable rebalancing policy (defaulting to a vchannel-fair distribution) to ensure even load distribution across the cluster. It supports dynamic node management, including the ability to freeze or unfreeze specific nodes to control assignment, and monitors the primary resource group to ensure compliance during configuration changes. The balancer also handles recovery of channel assignments from the catalog on startup and provides interfaces for watching assignment changes and triggering rebalances.

internal/streamingcoord/server/balancer · high confidence

Introduce streaming coordination client for assignment and broadcast services

Users benefit from a new internal client component that manages communication with the streaming coordination service. This client provides access to assignment services for discovering streaming node information and managing WAL balance policies, as well as broadcast services for sending and acknowledging messages. It handles gRPC connections with lazy initialization, supports internal TLS for secure communication, and integrates with the existing session discovery mechanism to locate the coordination service.

internal/streamingcoord/client · high confidence

Introduce streaming coordination resource singleton

The streaming coordination server now uses a centralized, thread-safe singleton resource instance to manage core dependencies, including the etcd client, MixCoord client, ID allocator, streaming catalog, and streaming node manager client. This change ensures consistent access to these services across the streaming coordination component and includes initialization logic that validates all required dependencies are present before activation.

internal/streamingcoord/server/resource · high confidence

Introduce streaming error handling and gRPC status conversion utilities

Added a new \status\ package in \internal/util/streamingutil\ that provides a structured error type (\StreamingError\) with specific codes for streaming operations (such as channel fencing, transaction expiration, and rate limiting). This package includes utilities to convert between streaming errors and gRPC statuses, allowing clients to correctly identify canceled or deadline-exceeded errors and enabling servers to map internal streaming codes to appropriate gRPC response codes.

internal/util/streamingutil/status · high confidence

Introduce streaming node client handler with assignment watcher

This change introduces the \internal/streamingnode/client/handler\ package, which provides a new client-side interface for interacting with the streaming node service. It includes a \HandlerClient\ that manages gRPC connections with lazy initialization, internal TLS support, and retry policies. A key component is the \assignment.Watcher\, which monitors pchannel-to-node assignments via a resolver and allows producers and consumers to wait for and react to assignment changes. The handler also exposes methods for local WAL operations (such as retrieving MVCC timestamps, replicate checkpoints, and salvage checkpoints) and falls back to remote calls when the WAL is not local. Also in this area: Resumable TransformLog subscriptions with automatic reconnection and checkpointing; Internal refactoring of index layer and compaction policies; Introduce streaming node producer client with rate limiting and trace propagation; Introduces new streaming consumer client implementation.

internal/streamingnode/client/handler · medium confidence

Introduce streaming node manager client for WAL instance assignment

A new ManagerClient has been added to the streaming node client package to manage WAL (Write-Ahead Log) instances across streaming nodes. This client enables the system to discover streaming nodes via service discovery, collect their status (including resource group information), and assign or remove WAL instances for specific channels. It supports watching for node changes, handles internal TLS configuration, and includes logic to filter and fallback resource groups when collecting status, ensuring robust WAL balancing and assignment operations.

internal/streamingnode/client/manager · high confidence

Introduce streaming service gRPC resolver for dynamic service discovery

Added a new gRPC resolver implementation in the streaming utility that enables dynamic service discovery for streaming components. The resolver supports two schemes, \milvus-session\ and \channel-assignment\, and integrates with the internal discoverer to watch for state changes. It allows gRPC clients to automatically update their connection targets as the underlying service instances change, ensuring reliable connectivity for the streaming node client and proxy.

internal/util/streamingutil/service/resolver · high confidence

Introduce timetick interceptor for WAL message synchronization

Added a new timetick interceptor in the streaming node's Write-Ahead Log (WAL) pipeline. This component automatically assigns timestamps and last confirmed message IDs to messages before they are appended to the WAL, ensuring consistent ordering and recovery points. It also manages transaction sessions (begin, commit, rollback) and periodically syncs acknowledged timestamps to the WAL via a dedicated sync operator, improving data durability and crash recovery reliability.

internal/streamingnode/server/wal/interceptors/timetick · high confidence

Introduce unified MixCoord client for merged coordinator services

The system now uses a single MixCoord client to communicate with the merged Root, Data, and Query coordinators. This client handles gRPC connections to the unified service, including TLS configuration and session discovery, and provides a unified interface for collection, partition, credential, and metadata operations, replacing the previous separate coordinator clients.

internal/distributed/mixcoord/client · high confidence

Introduce v2 QueryNode pipeline with dedicated insert, delete, and filter nodes

The \internal/querynodev2/pipeline\ package now implements a new streaming pipeline architecture for the QueryNode, replacing the previous flow-graph model. This change introduces distinct pipeline nodes: \filterNode\ validates and routes incoming insert and delete messages (including schema change and manual flush messages) while enforcing policies like alignment and target checks; \insertNode\ processes insert payloads, handles BM25 statistics, and manages embedding function execution before passing data to the delegator; and \deleteNode\ aggregates delete operations by timestamp and forwards them to the delegator. A \Manager\ component orchestrates the lifecycle of these per-channel pipelines, and the \message.go\ definitions standardize the internal message types (\insertNodeMsg\, \deleteNodeMsg\) used for inter-node communication.

internal/querynodev2/pipeline · high confidence

Introduces TimeTickSyncInspector for WAL recovery checkpoint management

Adds a new TimeTickSyncInspector component within the Write-Ahead Log (WAL) interceptors to manage time tick synchronization for pchannels. This inspector ensures that recovery checkpoints are persisted on a bounded cadence by forcing persisted syncs when idle, preventing recovery storage from skipping non-persisted heartbeats. It coordinates sync operations via a background ticker and a signal notifier, allowing for both periodic forced persistence and manual triggers while preventing duplicate in-flight syncs per channel.

internal/streamingnode/server/wal/interceptors/timetick/inspector · high confidence

Introduces WAL Replicate Manager for cluster replication state and message handling

Adds the \ReplicatesManager\ implementation and its supporting components (secondary state, transaction helper) to the WAL interceptor layer. This new module manages the lifecycle of replication roles (primary vs. secondary), handles the recovery of replication checkpoints from snapshots, and processes incoming replicated messages on the secondary node. It includes logic to capture salvage checkpoints during force promotions, deduplicate transaction bodies, and ignore stale or self-controlled messages, ensuring consistent state synchronization between clusters.

internal/streamingnode/server/wal/interceptors/replicate/replicates · high confidence

Introduces a pluggable interceptor chain for WAL append operations

The Write-Ahead Log (WAL) now supports a chain of interceptors that wrap the core append logic, allowing cross-cutting concerns like metrics collection, time tick management, transaction handling, and replication to be applied uniformly. This change introduces the \Interceptor\ interface and a \ChainedInterceptor\ implementation that coordinates multiple interceptors, ensuring they are ready before accepting messages and collecting metrics for each step in the chain. Users benefit from a more modular and observable WAL system where additional processing steps can be added or modified without altering the core append path.

internal/streamingnode/server/wal/interceptors · high confidence

Introduces centralized resource management for the streaming node server

The streaming node now uses a new singleton resource manager to initialize and provide access to core dependencies, including the etcd client, chunk manager, MixCoord client, and allocators. This centralization ensures that critical components like the TimeTick inspector and write buffer manager are properly initialized and released, improving startup reliability and resource lifecycle management for the streaming service.

internal/streamingnode/server/resource · high confidence

Introduces comprehensive Write-Ahead Log (WAL) metrics and tracing

The streaming node now exposes detailed Prometheus metrics and structured logging for Write-Ahead Log operations. This change adds instrumentation for append operations (including interceptor durations and slow-log thresholds), segment assignment (allocations, flushes, insert/delete counts), time tick management (allocation, acknowledgment, and sync), transaction lifecycles, scanner activity (catchup vs. tailing modes, message filtering, and deduplication drops), and the new Write-Ahead Buffer. Additionally, WAL log entries now propagate OpenTelemetry trace contexts, enabling end-to-end request tracing for streaming operations.

internal/streamingnode/server/wal/metricsutil · high confidence

Introduces configurable assignment policies for segments and channels

The QueryCoord balance system now supports pluggable assignment policies for distributing segments and channels across nodes. Users can select from three strategies via configuration: round-robin (even distribution by count), row-count-based (prioritizes nodes with the least data volume), and score-based (uses a comprehensive scoring model including collection row counts, delegator overhead, and benefit evaluation to minimize rebalancing impact). A global factory manages these policies, allowing dynamic switching without restarts, and includes a fix to ensure benefit checks correctly handle negative node priorities.

internal/querycoordv2/assign · high confidence

Introduces singleton wrapper for the broadcaster with WAL-based DDL readiness checks

A new singleton wrapper in the broadcast package now manages the broadcaster instance and enforces that WAL-based DDL is ready before starting broadcasts. This wrapper provides entry points for resource-keyed broadcasts (including unreplicable and secondary-cluster-specific operations) and exposes pending schema file resources, ensuring that broadcast operations are gated by cluster state readiness.

internal/streamingcoord/server/broadcaster/broadcast · high confidence

Introduces streaming node producer metrics and gRPC response helpers

The streaming node now exposes Prometheus metrics for producer activity, specifically tracking the total number of produce operations and the count of currently inflight messages per pchannel. To support this, a new \producerMetrics\ component was added to the producer handler, along with a \produceGrpcServerHelper\ that standardizes how produce results, rate-limit states, and lifecycle events (created/closed) are sent back to clients over gRPC. Comprehensive unit tests were added to verify the metrics lifecycle and the correct serialization of gRPC responses.

internal/streamingnode/server/service/handler/producer · high confidence

Introduces streaming service gRPC interceptors and attribute helpers

Adds new gRPC interceptors for the streaming service that standardize error handling by converting Milvus-specific errors (such as node mismatch or cross-cluster routing issues) into streaming-aware errors, ensuring consistent client-side behavior. Additionally, introduces a utility package for managing gRPC attributes, allowing server IDs, session data, and channel assignment information to be attached to and retrieved from RPC contexts.

internal/util/streamingutil/service/interceptor · high confidence

Introduction of QueryCoord v2 Checkers

The \internal/querycoordv2/checkers\ package has been introduced, providing the core scheduling and monitoring logic for the new QueryCoord v2 architecture. This location contributes the \CheckerController\ which orchestrates five distinct checkers—Channel, Segment, Balance, Index, and Leader—each responsible for monitoring specific aspects of cluster distribution and generating balancing tasks. The implementation includes a priority-based \BalanceChecker\ that supports configurable sorting (by row count or collection ID) and batch limits, a \ChannelChecker\ with version caching to optimize performance, and activation/deactivation interfaces for all checkers. Comprehensive unit tests are included to validate the checker lifecycle, configuration loading, and specific balancing behaviors.

internal/querycoordv2/checkers · high confidence

Introduction of StreamingNode gRPC service and server lifecycle

This change introduces the \internal/distributed/streamingnode\ package, establishing the gRPC server interface and lifecycle management for the StreamingNode component. The new \service.go\ file defines the \Server\ struct, which handles initialization, running, and graceful shutdown of the StreamingNode's gRPC endpoint, including session registration/unregistration with etcd and integration with MixCoord and TiKV clients. An \OWNERS\ file is also added to define code review responsibilities for this new area.

internal/distributed/streamingnode · high confidence

Introduction of internal service interfaces and OWNERS file

The internal/types package now defines the core interfaces for Milvus components (DataNode, DataCoord, RootCoord, Proxy) and their clients, establishing the contract for inter-service communication and state management. This includes specific methods for rate limiting, state code updates, and component initialization. Additionally, an OWNERS file has been added to specify code review responsibilities for this package.

internal/types · high confidence

Introduction of new cmd entry point and ASAN leak-check utilities

This change introduces the \cmd/main.go\ entry point which delegates to the \milvus\ sub-package, alongside new \cmd/asan/\ files that conditionally enable AddressSanitizer leak checking via CGO when the \use\_asan\ build tag is present. It also adds a \cmd/OWNERS\ file to define reviewers and approvers for this directory.

cmd · high confidence

Merge RootCoord, DataCoord, and QueryCoord into MixCoord

The distributed service layer in internal/distributed/mixcoord has been consolidated to host the merged MixCoord component, replacing the previous separate RootCoord, DataCoord, and QueryCoord gRPC servers. This change introduces a unified gRPC server implementation (service.go) that initializes and manages the combined coordinator, along with new extension seams (extension\_seam.go) to support external coordinator engine lifecycles and test-only accessors (api\_testonly.go) for internal checker control.

internal/distributed/mixcoord · high confidence

New /livez endpoint and structured /healthz responses

The health check system now includes a dedicated /livez endpoint for lightweight liveness probes that confirms the process is active without checking component states, alongside a refactored /healthz handler that supports JSON responses via the Content-Type header and provides detailed per-component health status in its output.

internal/http/healthz · high confidence

New ANTLR-based expression parser with expanded filter syntax

The plan parser has been replaced with a new ANTLR-based implementation (Plan.g4) that supports a significantly broader range of filter expressions. Users can now use raw string literals (r"..."), regex operators (=\~, !\~), bitwise operators (&, \|, ^, \~, \<\<, \>\>), and the POW (\\) operator. The parser also supports new text matching functions (text\_match, text\_match\_fuzzy, phrase\_match, random\_sample), geospatial functions (st\_equals, st\_touches, etc.), and advanced membership filters (membership\_match with bloom filters, match\_all/any/exact/least/most). Additionally, it handles JSON and array field access with nested paths, template variables, and improved error handling for invalid expressions.

internal/parser/planparserv2 · high confidence

New Arrow-based function chain expressions for reranking

The \internal/util/function/chain/expr\ package introduces a new set of expression types that power the function chain pipeline using Apache Arrow arrays. This includes \BaseExpr\ for common expression logic, \RerankModelExpr\ for model-based reranking, \BoostScoreExpr\ for boosting scores, \DecayExpr\ for applying Gaussian, exponential, or linear decay functions, and \NumCombineExpr\ for combining numeric columns via sum, multiply, max, min, avg, or weighted modes. These expressions replace the legacy rerank implementation and enable more flexible, composable search reranking chains.

internal/util/function/chain/expr · high confidence

New BinlogIO abstraction and scalable IO pool for DataNode

The DataNode now uses a new \BinlogIO\ interface in \internal/flushcommon/io\ to handle binlog uploads and downloads, implementing async operations with retry logic and context-aware logging. Concurrent IO tasks are managed by a new pool (\GetOrCreateIOPool\) whose capacity automatically scales with the node's CPU count (defaulting to max(16, CPU\*2)) or respects an explicit \dataNode.dataSync.ioConcurrency\ configuration, replacing previous hard-coded limits. This change improves performance and resource utilization for object-storage IO operations.

internal/flushcommon/io · high confidence

New C-registry, thread watcher, and Holmes integration in metrics package

The internal/util/metrics package now includes a CRegistry that gathers Prometheus metrics from C/C++ components (Knowhere, storage, and jemalloc stats) and a MilvusRegistry that combines these with standard Go process metrics. A new ThreadWatcher monitors and exposes actual OS thread counts and CPU-active thread numbers by pool (e.g., knowhere, rocksdb, milvus\_search), addressing the inaccuracy of Go-only thread metrics. Additionally, Holmes is integrated to provide pprof-based memory/CPU profiling dumps on non-Darwin platforms, with a no-op stub for Darwin.

internal/util/metrics · high confidence

New CSV import reader with comprehensive type and schema validation

The CSV import utility now uses a new reader and row parser implementation that supports nullable fields, struct arrays, function output fields, and various vector types (including Int8Vector, Float16Vector, BFloat16Vector, and sparse vectors). The parser validates CSV headers against the collection schema, handles auto-generated primary keys, enforces max length and capacity constraints, and provides detailed error messages for missing or invalid fields. It also supports custom separators, null value keys, and retryable reads from object storage.

internal/util/importutilv2/csv · high confidence

New DataNode broker abstraction for DataCoord interactions

Introduced a new \Broker\ interface and its \coordBroker\ implementation in \internal/flushcommon/broker\ to standardize how the DataNode communicates with DataCoord. This new component wraps gRPC calls for segment ID assignment, time-tick reporting, segment info retrieval (with batched requests to respect message limits), channel checkpoint updates, binlog path saving, virtual channel dropping, and V2 imports. The implementation uses the new \MixCoordClient\ interface and includes context-aware logging and timeout handling, supported by unit tests and an autogenerated mock.

internal/flushcommon/broker · high confidence

New Debian packaging infrastructure for Milvus

This change introduces the complete build infrastructure for creating Debian (.deb) packages for Milvus. It includes a Docker-based build script (build\_deb.sh) that automates the packaging process, along with necessary Debian control files (control, rules, changelog, etc.) and systemd service definitions. The package is configured to run in standalone mode by default, embedding etcd and using local storage, and explicitly depends on the tzdata package to ensure correct IANA time zone ID recognition.

build/deb · high confidence

New Go SDK wrappers for Milvus bulk import REST API

The client/bulkwriter package now provides Go SDK functions to interact with the Milvus bulk import REST API. Users can initiate imports via BulkImport, track job status with ListImportJobs and GetImportProgress, and manage two-phase commit workflows using CommitImport and AbortImport. The SDK supports partitioning, API key authentication, and row-level security options (rls\_principal, skip\_rls) for cloud deployments.

client/bulkwriter · high confidence

New Go-based segcore utilities for Arrow data conversion, boost scoring, and CGO lifecycle management

This change introduces a suite of new Go packages in internal/util/segcore to support the self-managed segment loading architecture. It adds arrow\_to\_fielddata.go for converting Arrow RecordBatches into Milvus proto FieldData, boost\_score.go for computing and asynchronously applying boost scores to search results via CGO, and cancellation.go to manage C-layer cancellation sources safely from Go contexts. Additionally, it provides cgo\_util.go for CGO error handling and proto marshalling, collection.go for CCollection lifecycle management, jemalloc\_stats.go for memory statistics, and plan.go for search plan and request wrappers, all accompanied by comprehensive unit tests.

internal/util/segcore · high confidence

New JSON bulk import reader with advanced type support

This change introduces a new JSON import reader in the \internal/util/importutilv2/json\ package, enabling users to bulk insert data from JSON and JSONL (NDJSON) files. The reader supports multiple JSON formats (standard array, object-wrapped with 'rows' key, and line-delimited) and handles complex data types including nullable vectors, struct arrays, and function outputs (e.g., BM25). It also includes validation for numeric precision (using \UseNumber\ to preserve int64 precision), length constraints, and special float values (NaN/Inf), along with retry logic for storage reads.

internal/util/importutilv2/json · high confidence

New Jenkins pipelines for chaos, deployment, and scale testing

Added Jenkinsfile definitions for ChaosTest, ChaosTestKafkaMQ, DeployTest, DeployTestKafkaMQ, and Scale tests. These pipelines enable automated chaos engineering (pod-kill, network-latency, etc.), deployment verification (reinstall, upgrade), and scaling tests (data, index, proxy, query nodes) on Kubernetes, with configurable parameters for image tags, etcd versions, and node counts.

build/ci/jenkins · high confidence

New Jenkins pipelines for nightly, PR, and ARM/GPU image publishing

The CI system now includes new Jenkinsfile definitions for nightly testing (Nightly2), pull request validation (PR, PR-Arm, PRGPU, PR-for-go-sdk), and image publishing (MetaMigrationBuilder, PublishArmBasedGPUImages, PublishArmBasedImages). Nightly2 and PR pipelines have migrated to a Tekton-based execution model using the '4am' Kubernetes cloud, supporting deployment options such as standalone, distributed-pulsar, distributed-kafka, and woodpecker message queues. New pipelines enable ARM64 and GPU image builds and E2E testing, with specific configurations for Go SDK and PyMilvus clients. The MetaMigrationBuilder pipeline automates the building and publishing of migration tool images to the Milvus Harbor registry.

ci/jenkins · high confidence

New L0/L1 rerank function chain and heap-based reduce pipeline in QueryNode tasks

This change introduces the Go-layer implementation for L0 and L1 reranking function chains and a new heap-merge reduce pipeline within the QueryNode task layer. It adds \arrow\_import.go\ to convert Arrow RecordBatches into the internal DataFrame format, \boost\_score.go\ to compute and combine boost scores, and \group\_by\_columns.go\ to manage group-by metadata. The core reduction logic is handled by \heap\_merge\_reduce.go\, which performs k-way heap merging with support for group-by and element-level deduplication, replacing or augmenting previous reduce mechanisms. \l0\_function\_chain.go\ and \l1\_function\_chain.go\ implement the execution, validation, and column-pruning contracts for L0 (segment-level) and L1 (global) rerank stages respectively, including metrics for latency. Comprehensive tests are added for all new components to ensure correctness of metadata preservation, chunk splitting, memory safety, and rerank logic.

internal/querynodev2/tasks · high confidence

New QueryCoord metadata management components

The \internal/querycoordv2/meta\ package introduces a new metadata management layer for QueryCoord, including a \CollectionManager\ for handling collection and partition load states, a \ChannelDistManager\ for tracking DM channel distributions, and a \CoordinatorBroker\ to unify access to RootCoord and DataCoord APIs. It also adds a \FailedLoadCache\ to cache and retry load failures, and a \DistributionManager\ to expose segment and channel distribution data as JSON for observability.

internal/querycoordv2/meta · high confidence

New QueryCoord session management package

The \internal/querycoordv2/session\ package has been introduced to centralize the management of QueryNode connections and state within QueryCoord. This new module provides the \QueryCluster\ for handling RPC communication with QueryNodes (including operations like loading segments, watching channels, and running analyzers) and the \NodeManager\ for tracking node lifecycle, health, and resource exhaustion states. The package also includes generated mocks for testing and an OWNERS file to define review responsibilities for this component.

internal/querycoordv2/session · high confidence

New QueryNode v2 cluster worker management and pooling

The \internal/querynodev2/cluster\ package now introduces a new worker management layer for the QueryNode v2 architecture. This includes a \Manager\ interface and \grpcWorkerManager\ implementation that handles worker lifecycle, caching, and singleflight deduplication to prevent redundant connection creation. A new \Worker\ interface and \remoteWorker\ implementation wrap the gRPC client, adding support for client pooling (configurable via \WorkerPoolingSize\) to improve resource efficiency. The implementation also adds backward compatibility for legacy QueryNodes by gracefully handling \Unimplemented\ errors on \Delete\ and \DeleteBatch\ calls, falling back to split requests when necessary. The change is accompanied by generated mocks and comprehensive unit tests for the manager and worker logic.

internal/querynodev2/cluster · high confidence

New Spark-Milvus backfill E2E test infrastructure and mgit development tool

Added a comprehensive Python test suite for Spark-Milvus backfill end-to-end scenarios, including helpers for data preparation, Kubernetes job orchestration, contract validation, and remote entrypoints. Also introduced the mgit intelligent Git workflow tool to automate commit, PR, and cherry-pick workflows for Milvus development.

python · high confidence

New atomic ID allocator for migration tools

The migration tooling now includes a new \allocator\ package that provides an atomic, thread-safe ID generator. This component allows the migration process to allocate unique identifiers using configurable initial values and deltas, supporting both ascending and descending generation orders to handle existing data sequences correctly.

cmd/tools/migration/allocator · high confidence

New balancer factory and channel-level score balancing in QueryCoord

The \internal/querycoordv2/balance\ package now includes a \BalancerFactory\ that dynamically selects and caches balancer implementations (RoundRobin, RowCountBased, ScoreBased, MultiTarget, and ChannelLevelScore) based on the \QueryCoordCfg.Balancer\ configuration, defaulting to the ChannelLevelScore balancer. The new \ChannelLevelScoreBalancer\ introduces channel-level balance awareness, ensuring segments belonging to the same channel are balanced among the nodes assigned to that channel (exclusive mode), with fallback to score-based balancing if exclusive mode is not achievable. This change also adds a \BalanceReplicaHelper\ to handle streaming service compatibility for node selection during channel balancing.

internal/querycoordv2/balance · high confidence

New binlog import reader with L0 segment and time-range filtering support

The binlog import utility now includes a new reader implementation that supports importing L0 (delta) segments alongside standard insert binlogs. This reader introduces filter-based processing, allowing users to restrict imports by time range (tsStart/tsEnd) and automatically exclude rows deleted by L0 delta logs. It also handles Storage V2/V3 schema validation, including optional nullable vector fields, and ensures robust retry logic for listing and reading binlog files.

internal/util/importutilv2/binlog · high confidence

New binlogv2 tool for analyzing and exporting Parquet files

A new Python-based command-line tool has been added to \cmd/tools/binlogv2\ to analyze Parquet files and export their data. The tool provides a \parquet\_analyzer\_cli\ interface with commands to inspect metadata, analyze vector columns (including deserialization of Float, Int8, Binary, and Float16 vectors), and export data to JSON. It also includes a \MinioParquetAnalyzer\ class for connecting to MinIO storage to list, filter, and download Parquet files for analysis, as well as a dedicated \export\_to\_json\ utility for converting Parquet files to JSON format with options for row selection and vector formatting.

cmd/tools/binlogv2 · high confidence

New broadcast acknowledgment module for WAL recovery

The WAL recovery subsystem now includes a dedicated broadcast acknowledgment module that manages the lifecycle of broadcast messages. This module handles resource ordering, ensures exclusive ownership is maintained until acknowledgment succeeds, and implements retry logic with configurable delays for failed acknowledgments. It also supports poisoning mechanisms to block conflicting tasks when local work fails, preventing premature acknowledgment to the coordinator. The module integrates with the scheduler for background processing and includes comprehensive tests covering legacy import retries, exclusive owner handling, and poison-based conflict resolution.

internal/streamingnode/server/wal/recovery · high confidence

New channel assignment and session discoverers for streaming service

The streaming service now includes dedicated discoverers for managing service endpoints: a new ChannelAssignmentDiscoverer tracks node-to-channel assignments via a watcher interface, while the SessionDiscoverer watches etcd for peer sessions with support for version filtering, exclusive mode, and forced ports. The session discoverer also clears stale sessions on retry to prevent state drift after etcd watch compaction, and both components integrate with the gRPC resolver to expose server IDs and assignment metadata for load balancing.

internal/util/streamingutil/service/discoverer · high confidence

New component state management and health-check utilities

The \internal/util/componentutil\ package now provides a \ComponentStateService\ helper that manages node lifecycle states (Initializing, Healthy, Abnormal, Stopping) and implements the \GetComponentStates\ RPC response, along with generic \WaitForComponent\*\ functions that poll for specific component states and \CheckHealthRespWithErr\ helpers to construct health-check responses. Tests verify the state transitions and the polling logic against mock clients.

internal/util/componentutil · high confidence

New composite KV transaction builder with atomic and chunked commit paths

The metastore KV layer now includes a new \txn\ package that allows callers to accumulate multiple Save, Remove, and RemovePrefix operations into a single composite transaction. The \Commit\ function applies these operations atomically when the total size fits within the backend's transaction limit (e.g., etcd's small limit), or falls back to a chunked, order-preserving flush for larger sets, ensuring that non-commit operations are applied in recorded order before a final guarded commit marker is applied. This supports idempotent writes and conditional commits (check-and-set) to prevent race conditions, such as advancing a checkpoint past a newer takeover, while handling storage-agnostic limits between etcd and TiKV.

internal/metastore/kv/txn · high confidence

New concurrent-safe search scheduler with configurable policies

The search task scheduling logic in the query node has been replaced with a new, concurrent-safe scheduler located in \internal/util/searchutil/scheduler\. This component introduces a pluggable policy system, supporting both FIFO and user-task-polling scheduling strategies. It manages task execution via concurrency pools, enforces per-user pending task limits, and supports task merging to optimize throughput. The scheduler also provides mechanisms to clear queued tasks based on filters and handles lifecycle states to ensure graceful shutdown and resource cleanup.

internal/util/searchutil/scheduler · high confidence

New distributed streaming service layer for WAL access and replication

This change introduces the \internal/distributed/streaming\ package, which provides the core distributed interface for interacting with the Write-Ahead Log (WAL). It includes a balancer implementation for managing streaming node assignments and rebalancing, a forward service to route legacy proxy requests (DML/DQL) to the new streaming infrastructure, and a replicate service to handle cross-cluster message replication and configuration. Additionally, it adds a msgstream adaptor to bridge the delegator with the new streaming backend, along with comprehensive tests for these components.

internal/distributed/streaming · high confidence

New embedding and reranking providers: Ali, Cohere, Gemini, Hugging Face, and Siliconflow

This change adds client implementations for five new external model providers in the function models layer, enabling users to connect Milvus to Ali DashScope, Cohere, Google Gemini, Hugging Face, and Siliconflow for text embedding and reranking tasks. The diff introduces dedicated client packages (ali, cohere, gemini, huggingface, siliconflow) along with a shared common module for credential parsing and timeout configuration. Each provider client handles API authentication, request construction, and response parsing for embedding and reranking operations, with corresponding unit tests verifying successful calls and error handling.

internal/util/function/models · high confidence

New embedding providers for Ali, AWS Bedrock, and Cohere

This change adds new embedding model providers for Ali (AliDashScope), AWS Bedrock, and Cohere to the embedding function system. Users can now configure these providers to generate vector embeddings for text data, supporting features like configurable dimensions, batch sizes, and specific model parameters (e.g., truncation for Cohere, region/credentials for Bedrock). The implementation includes the provider logic, client creation, and corresponding unit tests to ensure correct embedding generation and error handling.

internal/util/function/embedding · high confidence

New entrypoint script for Docker builder container

A new entrypoint script (entrypoint.sh) has been added to the Docker builder image to handle container initialization. It ensures the HOME directory exists, configures a consistent bash prompt by copying skeleton files, and dynamically adds the current user to /etc/passwd and /etc/group if not already present. Additionally, it sources environment setup scripts for devtoolset-11, gcc-toolset-11, and llvm-toolset-11 if available, ensuring the correct compiler toolchains are active before executing the container's main command.

build/docker/builder · high confidence

New expression utility for partition key isolation and primary key predicate optimization

Added \internal/util/exprutil/expr\_checker.go\ to provide utilities for analyzing query expressions. This includes logic to parse and extract partition key values from binary, unary, and term expressions to enable segment pruning based on partition key predicates. It also introduces \HasOptimizablePkPredicate\ to identify primary key predicates that can be optimized using bloom filters or min/max pruning, following specific rules for AND/OR/NOT logical operators. The change includes corresponding unit tests for these new capabilities.

internal/util/exprutil · high confidence

New external rerank model providers and unified score mapping

The rerank function now supports additional external model providers, including Ali, Cohere, Hugging Face, Siliconflow, VoyageAI, and Zilliz, in addition to the existing vLLM and TEI integrations. A new \rerankScoresByIndex\ utility ensures that relevance scores are correctly mapped back to their original documents based on the service-reported index, preventing score mismatches when responses are reordered or partial. Each provider supports configurable batch sizes and timeouts, and the system now validates that the number of returned scores matches the number of input documents, returning an error if there is a mismatch.

internal/util/function/rerank · high confidence

New generic gRPC client for Proxy communication

The internal/distributed/proxy/client package now provides a new generic gRPC client implementation for Proxy-to-Proxy communication. This client supports internal TLS with SNI, automatic retry logic via a generic ReCall wrapper, and context-aware error handling. It exposes methods for component state checks, statistics channels, cache invalidation, metrics retrieval, rate limiting, and client info listing, ensuring consistent message base population and target ID routing for all proxy interactions.

internal/distributed/proxy/client · high confidence

New import execution pool, memory allocator, and scheduler in importv2

The importv2 package now includes a dedicated execution pool that dynamically scales based on CPU cores and the \ImportConcurrencyPerCPUCore\ configuration, a global memory allocator that enforces a configurable memory limit percentage to prevent OOM by blocking tasks when the limit is reached, and a scheduler that processes pending import tasks in batches. These components provide the underlying infrastructure for managing concurrency, resource limits, and task execution flow for bulk imports.

internal/datanode/importv2 · high confidence

New in-tree extension mechanism for hooks and ciphers

The \internal/util/hookutil\ package now provides a unified, in-tree extension mechanism that allows deployments to install custom hooks and ciphers either compiled directly into the binary or loaded as dynamic plugins. This introduces a singleton hook system that supports dynamic reconfiguration without restarts, a cipher system with backup/reload capabilities and KMS key state tracking, and a mutex-protected plugin loader to prevent concurrency issues. It also includes utilities for managing encryption zones (EZ) and ensures that configuration changes are safely applied to both compiled-in and plugin-based extensions.

internal/util/hookutil · high confidence

New index parameter checkers and unified manager for Bitmap, FM-index, and AutoIndex

The index parameter validation system in \internal/util/indexparamcheck\ has been expanded with new checkers for Bitmap, FM-index, and AutoIndex types. The Bitmap checker now enforces that JSON fields use only BOOL or VARCHAR cast types and validates specific scalar field constraints. The FM-index checker adds validation for VARCHAR fields, ensuring sample rate and block byte parameters fall within allowed ranges. Additionally, a new AutoIndex checker has been introduced, and a central \IndexCheckerMgr\ now unifies the registration and retrieval of all index checkers, including the new types, ensuring consistent validation across the system.

internal/util/indexparamcheck · high confidence

New initcore package centralizes C++ segcore and storage initialization

A new \internal/util/initcore\ package has been introduced to consolidate the initialization of the C++ segcore library and storage components. This change introduces dedicated Go wrappers for configuring the Arrow IO thread pool (with hot-reload support), JSON stats build executors, and query node parameters such as chunk rows, SIMD type, and thread pool sizes. It also adds a new \InitQueryNode\ entry point that enforces row visibility filtering and initializes the local and remote Arrow file systems, providing a single, structured location for these core startup configurations.

internal/util/initcore · high confidence

New krte Docker image for Kubernetes-in-Docker testing

Added a new Dockerfile and wrapper script in build/docker/krte to support local development and CI testing using KinD (Kubernetes in Docker). The image bundles Go, kubectl, Helm, Docker, Docker Compose, and KinD v0.11.1, while the wrapper script manages Docker-in-Docker setup, IPv6/iptables-nft configuration, GCP authentication, and test execution with proper cleanup.

build/docker/krte · high confidence

New log export script for Milvus and dependencies

A new shell script (\export-milvus-log.sh\) and documentation have been added to the \deployments/export-log\ directory to simplify log collection for Milvus deployments on Kubernetes. The tool exports logs for the Milvus component by default, and supports optional collection of logs from etcd, MinIO, Pulsar, and Kafka via specific flags. It handles both standard Helm installations and those managed by the Milvus operator, captures logs from restarted pods (including previous container logs), and allows filtering by time duration.

deployments/export-log · high confidence

New meta migration script for upgrading Milvus from v2.1.x to v2.2.0

A new migration tool has been added to the \deployments/migrate-meta\ directory to handle the structural changes in the segment index metadata introduced in Milvus 2.2. This bash script automates the upgrade process for clusters installed on Kubernetes, guiding users through stopping components, backing up metadata, performing the migration, and restarting with the new image. It supports key configurations such as specifying the source and target versions, custom image tags, external etcd services, and storage classes, and includes options to automatically clean up migration pods after completion.

deployments/migrate-meta · high confidence

New observer components for QueryCoord v2

The \internal/querycoordv2/observers\ package now contains the core observer implementations for the QueryCoord v2 subsystem, including \CollectionObserver\ (managing load status and timeouts), \TargetObserver\ (handling next/current target updates), \ReplicaObserver\ (checking replica node health and streaming nodes), \ResourceObserver\ (recovering resource group configurations), and \LeaderCacheObserver\ (invalidating proxy shard leader caches). This location also includes the corresponding unit tests for these components and an OWNERS file defining the review and approval workflow for this package.

internal/querycoordv2/observers · high confidence

New offline installation guide and image-saving utility

Added a new README in the offline deployment directory that provides step-by-step instructions for manually downloading and loading Docker images for Milvus, supporting both Docker Compose and Kubernetes (Helm) deployments. Included a new Python utility, save\_image.py, which parses Docker or Kubernetes manifests to automatically pull and save the required container images as tar.gz files, simplifying the process of preparing Milvus for air-gapped or offline environments.

deployments/offline · high confidence

New per-resource-group load and readiness utilities in QueryCoord

The \internal/querycoordv2/utils\ package now includes new utilities for resource-group-aware load management: \LoadPercentageByResourceGroup\ reports the load progress of a specific resource group (rather than the collection-wide average), and \ShardLeaderReadinessByResourceGroup\ checks whether the shards in that group are ready to serve. These functions allow deployments that load collections across multiple resource groups to monitor and gate traffic per group independently. The change also adds replica assignment helpers (\AssignReplica\, \SpawnReplicasWithRG\) and test infrastructure (\checker.go\, \test.go\) to support this per-group behavior.

internal/querycoordv2/utils · high confidence

New redo interceptor for WAL append retries

A new redo interceptor has been added to the Write-Ahead Log (WAL) pipeline to handle append operations that require retrying, such as when waiting for a growing segment to become ready. This component intercepts insert messages, waits for the target partition's growing segment to be ready via the shard manager, and retries the append if a specific redo error occurs. It also ensures that appends fail immediately with an unrecoverable error if the collection or partition has been dropped, preventing unnecessary retries on invalid targets.

internal/streamingnode/server/wal/interceptors/redo · high confidence

New replicate stream server for WAL message replication

The proxy now includes a new \ReplicateStreamServer\ in \internal/proxy/replicate\ that handles streaming replicate messages from clients. This server receives replicate requests, appends the corresponding messages to the Write-Ahead Log (WAL) via the streaming service, and sends back confirmation responses (including \ConfirmedTimeTick\) to the client. It also supports trace context propagation by extracting and overwriting trace information on the messages before WAL append, ensuring observability across the replication path.

internal/proxy/replicate · high confidence

New rule-based expression rewriter for query optimization

A new expression rewriter module has been added to the plan parser to perform rule-based logical rewrites on parsed expression trees before planning. This module optimizes queries by merging ARRAY contains predicates into single Any/All operations, simplifying range predicates (tightening AND bounds, weakening OR bounds, merging intervals), normalizing IN/NOT IN expressions, and merging TEXT\_MATCH predicates. It also normalizes empty array comparisons to length checks and respects the \common.enabledOptimizeExpr\ configuration to enable or disable these optimizations at runtime.

internal/parser/planparserv2/rewriter · high confidence

New schema-bump and clustering compaction tasks for StorageV3

The datanode compactor now includes dedicated tasks to handle schema version bumps and clustering compaction on StorageV3 segments. The new \bumpSchemaVersionCompactionTask\ manages schema evolution by performing full rewrites when fields are dropped, additive reconciliation when new fields are added, and lightweight version updates when only the schema version changes, while correctly preserving TEXT LOB references. The \clusteringCompactionTask\ is updated to support StorageV3, enabling clustering compaction on manifest-based segments and integrating LOB context handling for TEXT columns. Comprehensive test suites have been added to verify these new compaction paths.

internal/datanode/compactor · high confidence

New search parameter optimization hook and Knowhere defaults in the optimizers module

The \internal/util/searchutil/optimizers\ package now introduces a \QueryHook\ extension mechanism that allows external plugins or in-tree distributions to tune search parameters (such as TopK and search settings) at query time. This hook is integrated into the \OptimizeSearchParams\ function, which also applies configurable Knowhere search defaults when enabled. Additionally, the module enforces strict group-by iterator filtering via \applyStrictGroupSettings\ and provides utilities for two-stage search decisions and effective segment number calculation, giving users more control over search performance and accuracy through configurable optimization hooks.

internal/util/searchutil/optimizers · high confidence

New segment-level metrics for cache and search operations

The \metricsutil\ package now introduces a new observation system that tracks granular metrics for segment-level cache loads, cache evictions, query accesses, and search accesses. This change adds specific Prometheus counters and observers for these operations, allowing users to monitor cache hit/miss rates, load/eviction durations, and query/search latencies at the segment level, separated by database and resource group labels.

internal/querynodev2/segments/metricsutil · high confidence

New shallow-copy utilities for search and retrieve requests

Added \internal/util/shallowcopy\ with \ShallowCopySearchRequest\ and \ShallowCopyRetrieveRequest\ functions that create lightweight copies of internal request structures. These utilities share slice and byte fields with the original objects while allocating a new \Base\ message with a specified \TargetID\, enabling efficient request replication across shards without the overhead of deep-copying large payload fields.

internal/util/shallowcopy · high confidence

New shard delegator implementation with two-stage search and buffered delta forwarding

The \internal/querynodev2/delegator\ package has been replaced with a new implementation that introduces a \BufferForwarder\ to batch and forward delete operations to workers, improving efficiency for streaming and L0 segment deletions. It also adds a two-stage search mechanism (\delegator\_twostage.go\) that performs a filter-only stage to optimize search parameters before executing the vector search, enhancing performance for filtered queries. The new code includes comprehensive unit tests for the delegator data handling, two-stage search logic, and delta forwarding policies.

internal/querynodev2/delegator · high confidence

New snapshot serialization and version validation

The internal/snapshotio package now provides serialization for Milvus snapshot metadata and per-segment manifests using Avro, supporting format versions 1 through 5 (with version 5 as the current default). It introduces strict validation that rejects snapshot metadata with unsupported or future format versions, ensuring that external table refresh jobs fail safely when encountering incompatible snapshot structures.

internal/snapshotio · high confidence

New storage layer for external snapshot import and export

This change introduces a new \internal/snapshotio/storage\ package that provides the core logic for handling external snapshot operations. It adds endpoint normalization and identity inference for cloud providers (Azure, AWS, Aliyun, etc.), URI parsing for foreign storage locations, and a resolver that configures remote chunk managers for cross-account or cross-provider snapshot restores and exports. It also includes utilities for building and validating snapshot URIs, rebasing snapshot paths during relocation, and generating stable content fingerprints for snapshot integrity verification. Also in this area: 7 commits (3 fixes) modifying internal.

internal/snapshotio/storage · medium confidence · unverified

New streaming pipeline infrastructure with empty time tick filtering

The \internal/util/pipeline\ package now provides a new streaming pipeline implementation (\StreamPipeline\) that integrates with the message dispatcher to consume streams. This includes a core pipeline abstraction (\pipeline\, \Node\) and a specific \emptyTimeTickSlowdowner\ that filters out redundant empty time tick messages to reduce overhead, with the filter interval being configurable and watchable. The implementation also introduces a \DMLMsgPackBatcher\ that merges buffered delete message packs while keeping insert message packs separate, and defines new error types for pipeline operations.

internal/util/pipeline · high confidence

New streaming utility components for WAL selection and dynamic channel discovery

This change introduces new utility files in the streaming module to support configurable Write-Ahead Log (WAL) backends and dynamic DML channel management. The \wal\_selector.go\ component determines the active WAL type (Rocksmq, Pulsar, Kafka, or Woodpecker) at startup based on configuration and deployment mode, specifically preventing Woodpecker with local storage in cluster mode unless forced. It also ensures the selected WAL name is not written to the runtime overlay, preserving the ability to switch WAL types later via etcd. Additionally, \config\_channel\_provider.go\ and \topic.go\ provide a mechanism to discover DML channels from both static configuration and recovered pchannel statistics, notifying consumers of new channels as they appear.

internal/util/streamingutil/util · high confidence

New structured access log info package for gRPC and REST endpoints

The \internal/proxy/accesslog/info\ package introduces a unified, structured approach to proxy access logging by defining an \AccessInfo\ interface and concrete implementations for both gRPC (\GrpcAccessInfo\) and REST (\RestfulInfo\) protocols. This change enables the access log to capture a rich set of metrics—including method name, status, trace ID, user address, error details (code, message, and type), database/collection names, response size, and SDK version—via a configurable \MetricFuncMap\. The implementation ensures consistent logging across both transport layers, handles multi-collection requests, and formats error messages safely to prevent log corruption.

internal/proxy/accesslog/info · high confidence

New tokenizer and analyzer binding for advanced text processing

The Tantivy search engine binding now includes a comprehensive text analysis layer, introducing support for multiple tokenizers (Jieba, Lindera, language identifier, n-gram, and character group) and a wide range of language-specific analyzers for Chinese, English, Arabic, and Thai. This update enables advanced text matching capabilities, including phrase match, fuzzy match, and regex-based filtering, while also adding configurable stop words, synonyms, and user-defined dictionaries to improve search relevance for multilingual content.

internal/core/thirdparty/tantivy/tantivy-binding · high confidence

New v2 import utility with multi-format support and backup-restore capabilities

The \internal/util/importutilv2\ package introduces a new import reader framework that supports JSON, JSONL/NDJSON, NumPy, Parquet, and CSV file formats, along with a dedicated binlog reader for backup-restore operations. Users can now configure import behavior via options such as \backup\ (enabling time-range filtering with \start\_ts\/\end\_ts\ and encryption zone keys), \l0\_import\, \skip\_disk\_quota\_check\, and CSV-specific settings like \sep\ and \nullkey\. The implementation includes strict validation to prevent duplicate or case-insensitive option key conflicts and provides a mock reader for testing.

internal/util/importutilv2 · high confidence

New vchannelFair balancing policy for streaming nodes

A new balancing policy named 'vchannelFair' has been added to the streaming coordinator to distribute pchannels and vchannels more evenly across streaming nodes. This policy calculates an expected layout by minimizing a global unbalanced score that weighs pchannel count, vchannel count, and anti-affinity (ensuring vchannels from the same collection are spread across different nodes). It supports configuration via parameters for weights, rebalance tolerance, and max steps, and includes logic to preserve current assignments for stability while rebalancing new or unassigned channels when allowed.

internal/streamingcoord/server/balancer/policy/vchannelfair · high confidence

Optional root authentication for the management plane

Added a new \CachedRootVerifier\ and \RootCredentialVerifier\ in the \internal/util/adminauth\ package to gate management-plane endpoints (such as \/management/stop\ and \/debug/pprof\) with root credential verification. The implementation caches the root password hash fetched from the MixCoord, supports password rotation via a \Forget\ mechanism, and includes concurrency controls (semaphores and singleflight) to prevent resource exhaustion during credential lookups and bcrypt comparisons. Comprehensive tests cover cache invalidation, timeout handling for uncooperative credential lookups, and saturation behavior.

internal/util/adminauth · high confidence

Proxy access logs now support YAML-based formatting and RESTful API capture

The proxy's access logging system has been refactored to support dynamic, YAML-configurable log formats. Users can now define custom formatters via configuration keys (e.g., \proxy.accessLog.\<Name\>.format\) and map them to specific gRPC methods or RESTful URL paths. This change introduces a \FormatterManager\ that selects the appropriate format based on the request method, ensuring that both gRPC and REST API traffic are logged with the desired structure. Additionally, the logging infrastructure now includes a cache writer with auto-flush capabilities and MinIO retention policies, improving performance and storage management for access logs.

internal/proxy/accesslog · high confidence

QueryCoord V2 adopts WAL-based DDL framework for load, release, and resource group operations

QueryCoord V2 now handles collection and partition load/release, as well as resource group modifications, through a new WAL-based DDL framework. This change introduces DDL callbacks that broadcast configuration changes via a control channel, ensuring consistent state across active-standby clusters and streaming replicas. It also adds an autoscale precheck to validate resource availability before load operations, providing actionable suggestions for cluster expansion when resources are insufficient.

internal/querycoordv2 · high confidence

QueryCoord autoscale adds pre-check for segment load resource estimation

The QueryCoord autoscale component now includes a pre-check capability to estimate the memory and disk resources required before loading segments. This is implemented via new \EstimateSegmentsLoadResource\ and \EstimateIndexLoadResource\ functions in the \autoscale\ and \segcore/loadresource\ packages, which calculate resource usage based on collection schemas, segment binlogs, and index information. The estimation logic respects configuration options such as Mmap settings, tiered eviction ratios, and field-specific load filters, allowing the autoscaler to make more informed decisions about scaling based on accurate resource projections.

internal/querycoordv2/autoscale, internal/util/segcore/loadresource · high confidence

QueryCoord metadata catalog now supports atomic composite updates

The QueryCoord KV catalog introduces a new \Update\ method that allows multiple metadata operations—specifically replica upserts and replica removals—to be executed as a single atomic transaction. This change enables mixed save-and-release actions to be committed together, reducing the risk of inconsistent state during replica management and improving the efficiency of metadata writes compared to issuing separate individual calls.

internal/metastore/kv/querycoord · high confidence

The QueryNodeV2 component now supports delta-based data distribution reports, allowing clients to request only the changes since the last reported timestamp rather than a full snapshot, which reduces network payload and improves efficiency. Additionally, the local in-process search path has been optimized to use a fast, wire-equivalent decoder for search results, bypassing standard gRPC marshaling to significantly reduce latency and memory overhead for local calls. These changes are accompanied by comprehensive unit tests for the new delta tracking logic, the local worker wrapper, and the query hook initialization behavior.

internal/querynodev2 · high confidence

RESTful API v2 introduces file resource management and enhanced visibility checks

The RESTful API v2 handler now supports managing file resources (add, remove, list) via dedicated endpoints, allowing users to upload and organize external data files for collection use. Additionally, the DescribeCollection endpoint now explicitly propagates the verified HTTP user identity into the request context, ensuring that collection visibility and access control checks align with the authenticated user rather than relying solely on internal proxy state.

internal/distributed/proxy/httpserver · high confidence

Redesigned CLI entry point with new commands and profiling support

The Milvus command-line interface has been refactored to support a more modular startup process. A new \milvus mck\ command allows users to run data consistency checks and clean inconsistent trash data. The \milvus run\ command now supports a \--native-profile\ flag to enable jemalloc heap profiling for debugging performance issues. Additionally, the server startup now includes a banner displaying the effective version, build time, and FIPS status, and prints total and used memory information to help users verify system resources at launch.

cmd/milvus · high confidence

Server labels and session versioning support

The session utility now supports injecting server labels into sessions via environment variables prefixed with \MILVUS\_SERVER\LABEL\\, allowing role-specific or global configuration (e.g., \STANDALONE\, \RESOURCE\_GROUP\) to be discovered by sessions. Additionally, the \SessionRaw\ struct includes \IndexEngineVersion\ and \ScalarIndexEngineVersion\ fields, and the \SessionInterface\ exposes methods like \GetSessionsWithVersionRange\ and \WatchServicesWithVersionRange\ to enable version-aware service discovery and filtering.

internal/util/sessionutil · high confidence

Streaming node assignment discovery via gRPC stream

The streaming coordination service now exposes a new gRPC stream for assignment discovery, allowing streaming nodes to receive full channel-to-node assignments and updates. This change introduces the \AssignmentDiscoverServer\ and its helper, which manages a bidirectional stream: the server pushes full assignment snapshots and incremental updates (including CChannel assignments and version pairs) to the client, while the client can report errors or close the stream. The implementation relies on the balancer to watch for assignment changes and the streaming node manager to retrieve the current set of nodes, ensuring that nodes without assigned channels are also included in the response with empty channel lists.

internal/streamingcoord/server/service/discover · high confidence

Streaming node manager introduces resource group awareness and graceful shutdown caching

The new \StreamingNodeManager\ in the streaming coordinator now tracks streaming query nodes grouped by resource group, exposing them via \GetStreamingQueryNodeIDsByResourceGroup\ to support finer-grained load balancing and scaling. It also adds a \StreamingReadyNotifier\ to signal when the streaming service is fully initialized, and implements a fallback mechanism in \fetchStreamingNodes\ that caches node information to prevent panics during coordinator shutdown when the balancer becomes unavailable.

internal/coordinator/snmanager · high confidence

Streaming producer adds rate limiting, metrics, and resuming capabilities

The streaming producer in this location now includes an adaptive rate limiter that responds to WAL rate-limit states (normal, slowdown, reject) to control write throughput, along with Prometheus metrics tracking producer availability, message counts, bytes, and latency. It also features a resumable producer that automatically recovers from connection breaks or node rebalancing, and supports distributed tracing spans for autocommit and transactional writes.

internal/distributed/streaming/internal/producer · high confidence

Support for function output fields on external collections

DataNode now computes function-output columns for external segments during refresh. The new \function\_executor.go\ implements a streaming pipeline that reads external fragments, applies declared functions (such as embeddings), and writes the results to a new packed file, all while keeping memory usage bounded to a single Arrow batch. The \ExternalCollectionManager\ and \RefreshExternalCollectionTask\ have been updated to track and execute these function-refresh tasks, ensuring that external collections can now include and maintain function-generated data alongside their source files.

internal/datanode/external · high confidence

Unified MixCoord introduces file resource synchronization and management-plane REST APIs

The internal/coordinator package now consolidates RootCoord, DataCoord, and QueryCoord into a single MixCoord server, which initializes a FileResourceObserver to track and sync file resources to QueryNodes, DataNodes, and Proxies. This change adds a management-plane REST API layer (restful\_mgr\_routes.go) exposing endpoints for batch/streaming node distribution, garbage collection control, WAL alteration, and configuration updates, alongside a new compliance endpoint for replica load configuration. The implementation includes a recovery barrier to coordinate startup phases and ensures that sensitive configuration values are properly masked or hidden when read back through the new management APIs.

internal/coordinator · high confidence

WAL utility package introduces safe logging, rate tracking, and checkpoint management

The \internal/streamingnode/server/wal/utility\ package adds several new components to support the streaming node's Write-Ahead Log. \AlterWALStateLogFields\ ensures that sensitive configuration data (such as SASL passwords and SSL keys) is never logged by only recording the count of configuration entries. \AverageRateCounter\ provides a thread-safe, sliding-window mechanism to track the average rate of bytes written to the WAL. \WALCheckpoint\ and \ReplicateCheckpoint\ structures manage consume checkpoints, including control state for replication and WAL alterations, with methods for serialization, cloning, and applying control metadata. Additional utilities include \DecodeInsertBody\ and \DecodeDeleteBody\ for safely decoding message payloads with specific error classifications, \primarykey\ for validating and extracting primary keys from insert and delete messages, and context helpers for tracking append results and metrics.

internal/streamingnode/server/wal/utility · high confidence

Security

DataNode gRPC service initialization and validation

The DataNode gRPC server now enforces server ID and cluster validation interceptors during startup, ensuring that incoming RPCs are authenticated against the correct cluster and node identity before processing. This change improves security and stability by preventing misconfigured or unauthorized nodes from interacting with the DataNode.

internal/distributed/datanode · high confidence

Optional root authentication for the management plane

The HTTP metrics port now supports optional root authentication for management endpoints (such as /management/\, /debug/pprof/\, /webui/, and /eventlog) via the \common.security.adminAuthEnabled\ configuration flag. When enabled, these endpoints require HTTP Basic Auth with the \root\ user credentials and enforce strict cross-site request protections (using \Sec-Fetch-Site\ and the \X-Milvus-Admin-Request\ header) to prevent CSRF attacks. This change introduces a new authentication middleware and RBAC integration in the \internal/http\ package, ensuring that administrative operations are secured by default while keeping health probes and metrics scraping open.

internal/http · high confidence

Architecture

Centralized proxy client management and etcd session watching

The proxy client manager and etcd session watcher have been moved into the \internal/util/proxyutil\ package, providing a shared, reusable implementation for managing proxy connections and watching proxy session changes in etcd. This refactoring consolidates proxy client lifecycle logic (adding, removing, and invalidating caches on proxies) and etcd watch handling (including re-watching on recoverable errors) into a common utility, replacing previous scattered implementations and ensuring consistent behavior across components that interact with proxy services.

internal/util/proxyutil · high confidence

DataNode component restructured with new entrypoint and service interfaces

The \internal/datanode\ package has been reorganized into a new modular structure. A new \data\_node.go\ entrypoint defines the \DataNode\ struct, initializing core services like the sync manager, import task manager, and index task scheduler. New \services.go\ and \index\_services.go\ files expose the DataNode's RPC interfaces for handling data persistence, import tasks, and index building. A \chunk\_mgr\_factory.go\ provides a centralized factory for creating storage chunk managers. Additionally, \pool\_stats.go\ introduces Prometheus metrics collection for the component's long-lived goroutine pools, and a \.mockery.yaml\ configuration is added to automate mock generation for internal interfaces.

internal/datanode · high confidence

Extract shared proxy task model into dedicated package

The shared task model layer for the Milvus proxy has been extracted into a new \internal/proxy/taskmodel\ package. This change defines the core \Task\ and \DMLTask\ interfaces, the \BaseTask\ struct for common metadata and timing bookkeeping, and the \Condition\ primitive for task lifecycle notifications. It also introduces the \TaskNode\ and \QueryRunner\ interfaces, which allow concrete tasks to interact with the proxy node and execute queries without creating import cycles. This refactoring decouples the task scheduler from concrete task implementations, establishing a clear dependency direction where the scheduler consumes these interfaces while the concrete task logic remains in the proxy root package.

internal/proxy/taskmodel · high confidence

Introduce builder pattern and structured initialization for the streaming node server

The streaming node server now uses a \ServerBuilder\ to configure and assemble its dependencies (such as etcd, gRPC, and metadata stores) before construction. The server's initialization logic is explicitly separated into basic component setup (including WAL manager and file resource manager) and service registration, ensuring that the gRPC handler and manager services are registered only after underlying resources are ready.

internal/streamingnode/server · high confidence

Introduce composite Update API and split metastore catalog interfaces

The metastore layer now uses a unified composite Update API to apply multiple metadata changes (such as segment updates, binlog increments, and view modifications) as a single atomic write, improving consistency and performance. To support this, the monolithic catalog interface has been split into specialized interfaces for RootCoord, QueryCoord, DataCoord, and StreamingCoord, each exposing only the operations relevant to their component. Additionally, a new QueryViewCatalog has been added to persist query view metadata to ETCD, and the DataCoord catalog now supports granular binlog updates and external snapshot operations.

internal/metastore · high confidence

Privilege management extracted into a dedicated cache package

The proxy's privilege management logic has been reorganized into a new \internal/proxy/privilege\ package. This change introduces a dedicated \PrivilegeCache\ to store credential, privilege, and user-role information, ensuring that credential lookups are properly cached to avoid redundant RPC calls. It also adds a \MetaCacheCasbinAdapter\ to bridge the cache with the Casbin authorization engine, and a \result\_cache\ to cache enforcement outcomes with version control for consistency.

internal/proxy/privilege · high confidence

QueryNode segment management refactored into new Go package

The segment management logic for QueryNode v2 has been reorganized into the new \internal/querynodev2/segments\ package. This change introduces a dedicated \CollectionManager\ for handling collection schemas and versions, a \diskUsageFetcher\ for monitoring local disk consumption, and an optimized \IgnoreNonPkOps\ pipeline that leverages Apache Arrow for efficient field data retrieval. Additionally, an \IndexAttrCache\ has been added to optimize index resource estimation, and a new \cgo\_util.go\ provides standardized error handling for CGO calls.

internal/querynodev2/segments · high confidence

Behavioural changes

1 commit (0 fixes) modifying internal/distributed/streaming/internal/replicadiscovery, internal/streamingnode/server/service/handler/replicadiscovery

A change to existing behaviour in internal/distributed/streaming/internal/replicadiscovery, internal/streamingnode/server/service/handler/replicadiscovery — 1 commit, 5 files: “feat: extract QueryView query execution components (\#54550)” (refs \#54550).

internal/distributed/streaming/internal/replicadiscovery, internal/streamingnode/server/service/handler/replicadiscovery · medium confidence · unverified

12 commits (2 fixes) modifying internal/querynodev2/qnview

A change to existing behaviour in internal/querynodev2/qnview — 12 commits (2 fixs), 35 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: add deadline retries and per-task backoff to node scheduler (\#54956)”; “enhance: Reduce native FieldMeta copy overhead (\#55187)”; and 9 more (refs \#52977, \#54956, \#55187, \#55110, \#51887, …).

internal/querynodev2/qnview · medium confidence · unverified

12 commits (2 fixes) modifying internal/util/schemautil

A change to existing behaviour in internal/util/schemautil — 12 commits (2 fixs), 10 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “feat: \[ElementNull 1\] add Go-side element nullable array payload support (\#51164)”; “feat: \[ElemNull 1\] store element-nullable and nested Array fields as native Arrow lists (\#54513)”; and 9 more (refs \#52977, \#51164, \#54513, \#51794, \#51360, …).

internal/util/schemautil · medium confidence · unverified

12 commits (4 fixes) modifying internal/streamingnode/server/wal/vchannel/segment

A change to existing behaviour in internal/streamingnode/server/wal/vchannel/segment — 12 commits (4 fixs), 32 files: “enhance: refactor asynchronous WAL recovery and summary persistence (\#53595)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: add deadline retries and per-task backoff to node scheduler (\#54956)”; and 9 more (refs \#53595, \#52977, \#54956, \#53145, \#53819, …).

internal/streamingnode/server/wal/vchannel/segment · medium confidence · unverified

1232 commits (370 fixes) modifying internal/datacoord

A change to existing behaviour in internal/datacoord — 1232 commits (370 fixs), 477 files: “Add log for flowgraph (\#14441)”; “fix: retry worker cleanup during import task GC (\#54867)”; “fix: datanode L0 segment num wrong (\#29050)”; and 1229 more (refs \#14441, \#54867, \#29050, \#25524, \#23710, …).

internal/datacoord · medium confidence · unverified

13 commits (1 fix) modifying internal/streamingnode/server/wal/vchannel

A change to existing behaviour in internal/streamingnode/server/wal/vchannel — 13 commits (1 fix), 31 files: “enhance: refactor asynchronous WAL recovery and summary persistence (\#53595)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: add deadline retries and per-task backoff to node scheduler (\#54956)”; and 10 more (refs \#53595, \#52977, \#54956, \#54341, \#53819, …).

internal/streamingnode/server/wal/vchannel · medium confidence · unverified

15 commits (1 fix) modifying internal/views

A change to existing behaviour in internal/views — 15 commits (1 fix), 221 files: “enhance: coalesce QueryView release reconciliation (\#55462)”; “feat: extract QueryView coordinator components (\#54547)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; and 12 more (refs \#55462, \#54547, \#52977, \#54956, \#55213, …).

internal/views · medium confidence · unverified

18 commits (3 fixes) modifying github.com/milvus-io/milvus/pkg/v3

A change to existing behaviour in github.com/milvus-io/milvus/pkg/v3 — 18 commits (3 fixs), 5 files: “enhance: use recovery+shardmanager, remove segment assignment interceptor (\#41824)”; “enhance: timetick interceptor implementation (\#34238)”; “enhance: add message and msgstream msgpack adaptor (\#34874)”; and 15 more (refs \#41824, \#34238, \#34874, \#35289, \#44124, …).

github.com/milvus-io/milvus/pkg/v3 · medium confidence · unverified

18 commits (5 fixes) modifying internal/flushcommon/util

A change to existing behaviour in internal/flushcommon/util — 18 commits (5 fixs), 6 files: “feat: support to replicate import msg (\#39171)”; “enhance: Fix progress always zero due to integer divide (\#35374)”; “fix: deterministic autoID primary keys for CDC-replicated imports (\#51825)”; and 15 more (refs \#39171, \#35374, \#51825, \#35589, \#45266, …).

internal/flushcommon/util · medium confidence · unverified

18 commits (8 fixes) modifying internal/util/queryutil

A change to existing behaviour in internal/util/queryutil — 18 commits (8 fixs), 64 files: “fix: keep proxy metric status label stable, split cause into its own label (\#51495)”; “fix: \[ElemNull 0a\] write row valid\_data to both the legacy and the field-specific location (\#54738)”; “fix: preserve grouped query aggregates across distributed limits (\#54313)”; and 15 more (refs \#51495, \#54738, \#54313, \#49990, \#41675, …).

internal/util/queryutil · medium confidence · unverified

19 commits (6 fixes) modifying internal/cdc/replication/replicatemanager

A change to existing behaviour in internal/cdc/replication/replicatemanager — 19 commits (6 fixs), 7 files: “fix: ignore topology changes that predate a replication task (\#52728)”; “fix: Fix replicator cannot stop and enhance replicate config validator (\#44531)”; “fix: prevent CDC replicator panic after source scanner termination (\#54663)”; and 16 more (refs \#52728, \#44531, \#54663, \#51049, \#45029, …).

internal/cdc/replication/replicatemanager · medium confidence · unverified

2 commits (0 fixes) modifying internal/core/thirdparty/grep, internal/util/function/grep

A change to existing behaviour in internal/core/thirdparty/grep, internal/util/function/grep — 2 commits, 24 files: “feat: add TEXT NGRAM indexes, Grep index routing, and max-count (\#55136)”; “feat: Support Grep (\#54282)” (refs \#55136, \#54282).

internal/core/thirdparty/grep, internal/util/function/grep · medium confidence · unverified

2 commits (0 fixes) modifying internal/pkindex

A change to existing behaviour in internal/pkindex — 2 commits, 26 files: “enhance: add the primary key index core storage library (\#53910)”; “enhance: add the primary key index manifest service (\#54170)” (refs \#53910, \#54170).

internal/pkindex · medium confidence · unverified

2 commits (0 fixes) modifying internal/streamingnode/server/service/handler/transformlog

A change to existing behaviour in internal/streamingnode/server/service/handler/transformlog — 2 commits, 4 files: “feat: add remote TransformLog subscriptions and MVCC barriers (\#54344)”; “feat: extract QueryNode QueryView query and resource preparation (\#54553)” (refs \#54344, \#54553).

internal/streamingnode/server/service/handler/transformlog · medium confidence · unverified

20 commits (8 fixes) modifying internal/proxy/dml

A change to existing behaviour in internal/proxy/dml — 20 commits (8 fixs), 45 files: “fix: encrypt upsert delete messages in WAL (\#54477)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; “feat: \[RLS6\] enforce row-level security for bulk import (\#53911)”; and 17 more (refs \#54477, \#52977, \#53911, \#54738, \#53311, …).

internal/proxy/dml · medium confidence · unverified

2054 commits (575 fixes) modifying pkg

A change to existing behaviour in pkg — 2054 commits (575 fixs), 867 files: “feat: supporing load priority for loading (\#42413)”; “enhance: Unify data type check APIs for go (\#31887)”; “enhance: Introduce batch subscription in msgdispatcher (\#39863)”; and 2051 more (refs \#42413, \#31887, \#39863, \#24176, \#27013, …).

pkg · medium confidence · unverified

2067 commits (751 fixes) modifying internal/core

A change to existing behaviour in internal/core — 2067 commits (751 fixs), 1197 files: “enhance: remove redundant bitset count operations in filter execution (\#47534)”; “feat: supporing load priority for loading (\#42413)”; “Optimize performance of insert & query & search (\#22829)”; and 2062 more (refs \#47534, \#42413, \#22829, \#5727, \#49222, …).

internal/core · medium confidence · unverified

24 commits (4 fixes) modifying internal/streamingnode/server/service

A change to existing behaviour in internal/streamingnode/server/service — 24 commits (4 fixs), 10 files: “enhance: wal balance consider the wal status on streamingnode (\#43265)”; “feat: expose streaming collection schema access (\#54341)”; “enhance: add streaming client metrics (\#36523)”; and 21 more (refs \#43265, \#54341, \#36523, \#54966, \#44124, …).

internal/streamingnode/server/service · medium confidence · unverified

3 commits (0 fixes) modifying internal/distributed/streaming/internal/transformlog

A change to existing behaviour in internal/distributed/streaming/internal/transformlog — 3 commits, 12 files: “feat: add remote TransformLog subscriptions and MVCC barriers (\#54344)”; “enhance: preserve query view RPC errors and target context (\#54987)”; “feat: extract QueryNode QueryView query and resource preparation (\#54553)” (refs \#54344, \#54987, \#54553).

internal/distributed/streaming/internal/transformlog · medium confidence · unverified

3 commits (0 fixes) modifying internal/metastore/kv/queryview

A change to existing behaviour in internal/metastore/kv/queryview — 3 commits, 6 files: “feat: extract QueryNode QueryView query and resource preparation (\#54553)”; “feat: add query view management and reliable syncer (\#52432)”; “enhance: version QueryView load configurations and retain View references (\#55059)” (refs \#54553, \#52432, \#55059).

internal/metastore/kv/queryview · medium confidence · unverified

3 commits (0 fixes) modifying internal/querynodev2/client

A change to existing behaviour in internal/querynodev2/client — 3 commits, 11 files: “feat: extract QueryView coordinator components (\#54547)”; “enhance: preserve query view RPC errors and target context (\#54987)”; “feat: extract QueryNode QueryView query and resource preparation (\#54553)” (refs \#54547, \#54987, \#54553).

internal/querynodev2/client · medium confidence · unverified

3 commits (0 fixes) modifying internal/streamingnode/server/queryplan

A change to existing behaviour in internal/streamingnode/server/queryplan — 3 commits, 3 files: “feat: add streaming node query runtime and resource preparation (\#53819)”; “feat: extract QueryView query execution components (\#54550)”; “enhance: preserve query view RPC errors and target context (\#54987)” (refs \#53819, \#54550, \#54987).

internal/streamingnode/server/queryplan · medium confidence · unverified

3 commits (0 fixes) modifying internal/streamingnode/server/viewquery

A change to existing behaviour in internal/streamingnode/server/viewquery — 3 commits, 9 files: “feat: add streaming node query runtime and resource preparation (\#53819)”; “feat: Support Grep (\#54282)”; “feat: extract QueryView query execution components (\#54550)” (refs \#53819, \#54282, \#54550).

internal/streamingnode/server/viewquery · medium confidence · unverified

3 commits (1 fix) modifying internal/util/importutilv2/reshardmem

A change to existing behaviour in internal/util/importutilv2/reshardmem — 3 commits (1 fix), 2 files: “feat: implement Import V3 reshard (\#53334)”; “enhance: consolidate InsertData and FieldData construction with builders (\#55100)”; “fix: bound raw TEXT bytes retained by the import V3 writer (\#54915)” (refs \#53334, \#55100, \#54915).

internal/util/importutilv2/reshardmem · medium confidence · unverified

30 commits (4 fixes) modifying internal/proxy/ddl

A change to existing behaviour in internal/proxy/ddl — 30 commits (4 fixs), 40 files: “enhance: add the primary key index core storage library (\#53910)”; “fix: return invalid-parameter error for malformed max\_length (\#54631)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; and 27 more (refs \#53910, \#54631, \#52977, \#53311, \#54986, …).

internal/proxy/ddl · medium confidence · unverified

35 commits (12 fixes) modifying internal/proxy/dql

A change to existing behaviour in internal/proxy/dql — 35 commits (12 fixs), 72 files: “feat: \[RLS4\] enforce row-level security (\#52075)”; “enhance: add the primary key index core storage library (\#53910)”; “fix: forward the resolved consistency level from Query to QueryView (\#55474)”; and 32 more (refs \#52075, \#53910, \#55474, \#52977, \#53911, …).

internal/proxy/dql · medium confidence · unverified

4 commits (0 fixes) modifying internal/querynodev2/qvresource

A change to existing behaviour in internal/querynodev2/qvresource — 4 commits, 13 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: centralize versioned schema access for proxy (\#54977)”; “enhance: load PK candidate alongside segment data in query view loader (\#55115)”; and 1 more (refs \#52977, \#54977, \#55115, \#54553).

internal/querynodev2/qvresource · medium confidence · unverified

4 commits (0 fixes) modifying internal/streamingnode/server/wal/vchannel/growingruntime, internal/streamingnode/server/wal/walview

A change to existing behaviour in internal/streamingnode/server/wal/vchannel/growingruntime, internal/streamingnode/server/wal/walview — 4 commits, 26 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “feat: add streaming node query runtime and resource preparation (\#53819)”; “feat: extract QueryView query execution components (\#54550)”; and 1 more (refs \#52977, \#53819, \#54550, \#54344).

internal/streamingnode/server/wal/vchannel/growingruntime, internal/streamingnode/server/wal/walview · medium confidence · unverified

5 commits (0 fixes) modifying internal/dataview

A change to existing behaviour in internal/dataview — 5 commits, 15 files: “enhance: refactor asynchronous WAL recovery and summary persistence (\#53595)”; “feat: extract QueryView coordinator components (\#54547)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; and 2 more (refs \#53595, \#54547, \#52977, \#53892, \#52537).

internal/dataview · medium confidence · unverified

5 commits (0 fixes) modifying internal/streamingnode/server/wal/vchannel/idf

A change to existing behaviour in internal/streamingnode/server/wal/vchannel/idf — 5 commits, 13 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: prewarm manifests that record their size without a stat (\#55110)”; “feat: add streaming node query runtime and resource preparation (\#53819)”; and 2 more (refs \#52977, \#55110, \#53819, \#54550, \#54553).

internal/streamingnode/server/wal/vchannel/idf · medium confidence · unverified

6 commits (0 fixes) modifying internal/streamingnode/server/wal/moduleapi

A change to existing behaviour in internal/streamingnode/server/wal/moduleapi — 6 commits, 3 files: “enhance: refactor asynchronous WAL recovery and summary persistence (\#53595)”; “enhance: add deadline retries and per-task backoff to node scheduler (\#54956)”; “feat: expose streaming collection schema access (\#54341)”; and 3 more (refs \#53595, \#54956, \#54341, \#55398, \#55120, …).

internal/streamingnode/server/wal/moduleapi · medium confidence · unverified

6 commits (3 fixes) modifying internal/cdc/util

A change to existing behaviour in internal/cdc/util — 6 commits (3 fixs), 2 files: “fix: ignore topology changes that predate a replication task (\#52728)”; “fix: revert dropping replicate configuration via empty config (\#49080)”; “enhance: support dropping replicate configuration via empty config (\#48994)”; and 3 more (refs \#52728, \#49080, \#48994, \#49485, \#44898, …).

internal/cdc/util · medium confidence · unverified

7 commits (0 fixes) modifying internal/streamingnode/server/wal/vchannel/queryresource

A change to existing behaviour in internal/streamingnode/server/wal/vchannel/queryresource — 7 commits, 28 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: add deadline retries and per-task backoff to node scheduler (\#54956)”; “enhance: preempt QueryView leases on collection release (\#55213)”; and 4 more (refs \#52977, \#54956, \#55213, \#53819, \#54550, …).

internal/streamingnode/server/wal/vchannel/queryresource · medium confidence · unverified

8 commits (0 fixes) modifying internal/streamingnode/server/wal/snview

A change to existing behaviour in internal/streamingnode/server/wal/snview — 8 commits, 33 files: “feat: integrate QueryView lifecycle and query execution (\#52977)”; “enhance: preempt QueryView leases on collection release (\#55213)”; “feat: add streaming node query runtime and resource preparation (\#53819)”; and 5 more (refs \#52977, \#55213, \#53819, \#51887, \#54550, …).

internal/streamingnode/server/wal/snview · medium confidence · unverified

8 commits (2 fixes) modifying internal/cdc/controller

A change to existing behaviour in internal/cdc/controller — 8 commits (2 fixs), 4 files: “fix: Recover from all transient etcd errors, not only leader changes (\#54733)”; “feat: integrate QueryView lifecycle and query execution (\#52977)”; “feat: Add CDC support (\#44124)”; and 5 more (refs \#54733, \#52977, \#44124, \#44564, \#49485, …).

internal/cdc/controller · medium confidence · unverified

8 commits (4 fixes) modifying internal/datanode/importv3

A change to existing behaviour in internal/datanode/importv3 — 8 commits (4 fixs), 34 files: “enhance: remove RLS implementation while retaining proto APIs (\#54352)”; “fix: Validate import schemas when a numpy or parquet file counts to zero (\#55137)”; “fix: fail Import V3 tasks on malformed sources and unrecoverable errors (\#54497)”; and 5 more (refs \#54352, \#55137, \#54497, \#53334, \#55202, …).

internal/datanode/importv3 · medium confidence · unverified

81 commits (14 fixes) modifying internal/metastore/model

A change to existing behaviour in internal/metastore/model — 81 commits (14 fixs), 25 files: “Support dynamic schema for create collection (\#24176)”; “Change default shard number to 1 (\#23586)”; “Add logic of autoindex (\#19570)”; and 78 more (refs \#24176, \#23586, \#19570, \#18692, \#38799, …).

internal/metastore/model · medium confidence · unverified

9 commits (1 fix) modifying internal/cdc/cluster

A change to existing behaviour in internal/cdc/cluster — 9 commits (1 fix), 4 files: “feat: Add CDC support (\#44124)”; “enhance: add per-cluster TLS config for CDC outbound mTLS connections (\#47968)”; “enhance: bump Go dependencies to v3 modules (\#49485)”; and 6 more (refs \#44124, \#47968, \#49485, \#48397, \#44898, …).

internal/cdc/cluster · medium confidence · unverified

9 commits (4 fixes) modifying internal/proxy/fieldvalidator

A change to existing behaviour in internal/proxy/fieldvalidator — 9 commits (4 fixs), 4 files: “feat: \[ElementNull 1\] add Go-side element nullable array payload support (\#51164)”; “fix: \[ElemNull 0a\] write row valid\_data to both the legacy and the field-specific location (\#54738)”; “feat: support positional partial updates for array values (\#53163)”; and 6 more (refs \#51164, \#54738, \#53163, \#52840, \#54513, …).

internal/proxy/fieldvalidator · medium confidence · unverified

Added TLS certificate assets for proxy support

The \configs/cert\ directory now includes a complete set of TLS certificates and keys (CA, server, and client) to enable secure proxy communication. This change provides the necessary cryptographic material for the proxy to establish TLS connections, supporting the new proxy TLS capability.

configs/cert · high confidence

Async WAL summary persistence with chunk caching and integrity validation

The WAL summary manager now persists and reads summary data asynchronously using a dedicated scheduler, decoupling storage I/O from source-message acknowledgements. This change introduces a new on-disk chunk format (codec version 1) with header/footer magic, checksums, and version checks to detect corruption, alongside a chunk cache that shares object reads among concurrent waiters, limits concurrent loads, and evicts entries based on capacity. Summary backlog is flushed only under pressure, and the system supports idempotent writes and barrier handling while ensuring that restored frontiers and idempotency keys are correctly recovered from durable state. Also in this area: 3 commits (1 fix) modifying internal/streamingnode.

internal/streamingnode/server/wal/walsummary · medium confidence · unverified

Automated Go code formatting in pre-commit hook

The pre-commit hook now automatically formats Go source files before each commit. A new script identifies modified .go files and runs the project's make fmt target, ensuring consistent code style without manual intervention.

githooks/pre-commit · high confidence

Automatic migration of legacy local storage directories on startup

When running in standalone mode with local storage, Milvus now automatically detects and migrates legacy index and log directories (from versions 2.6 and 3.0) into the configured local storage root before any component opens data. This migration runs silently during startup, merging directories without overwriting files, and will refuse to start if it encounters conflicting files, ensuring data integrity for users upgrading to the new local storage layout.

cmd/roles · high confidence

C++ glog output is now routed through the Go zap logging system

Logs generated by the C++ core via glog are now forwarded to the application's unified zap-based logger (mlog) instead of writing directly to files. This change ensures that C++ logs are subject to the same configuration, formatting, and destination settings as Go logs, and prevents stray log files from appearing in temporary directories (such as /tmp) when file logging is disabled or misconfigured.

internal/util/cgo/logging · high confidence

Correct segment row counts and merge request costs

The segment utility now includes logic to re-calculate and correct the row count of segments based on binlog data, ensuring metadata consistency for non-V3 storage versions. It also provides functions to calculate deleted rows from delta logs and to merge request costs by selecting the highest response time among aggregated costs.

internal/util/segmentutil · high confidence

Credential lookups now support mixed-case names

The credentials utility in internal/util/credentials now normalizes credential names to lowercase before lookup, allowing users to reference credentials using mixed-case names (e.g., 'aIstaffhub\_credential') without errors. This change ensures that credential retrieval functions like GetAPIKeyCredential, GetAKSKCredential, and GetGcpCredential are case-insensitive regarding the credential name, improving usability and reducing configuration friction.

internal/util/credentials · high confidence

Extract proxy channel manager and optimize insert message packing

The \internal/proxy/channelmgr\ package has been introduced to centralize collection channel resolution and message packing. The new \ChannelsMgr\ component decouples channel lookups from the metadata cache by accepting an injected resolver, ensuring no internal caching and preventing stale metadata while validating virtual/physical channel alignment. Additionally, the \GenInsertMsgsByPartition\ function now optimizes insert payload packing by using range views to avoid copying contiguous batches, while enforcing WAL-specific single-row size limits to prevent oversized messages.

internal/proxy/channelmgr · high confidence

Graceful stop timeout with pprof diagnostics for Milvus components

The component entry points in cmd/components (CDC, DataNode, MixCoord, Proxy, QueryNode, and StreamingNode) now enforce a configurable graceful stop timeout. If a component does not stop within the configured duration, the system dumps detailed performance profiles (goroutine, heap, block, and mutex) to the configured pprof directory before forcing an exit, helping users diagnose hangs during shutdown.

cmd/components · high confidence

Improved compaction reliability for import/CDC segments and TEXT LOB fields

The compaction logic in internal/compaction now correctly handles import and CDC segments by using a commit timestamp to prevent premature entity expiration and to ignore deletes that occurred before the segment was committed. Additionally, compaction for TEXT LOB fields now uses a manifest-based hole ratio calculation to decide whether to reuse or rewrite LOB files, and enforces Storage V3 for any collection containing TEXT fields to ensure data integrity.

internal/compaction · high confidence

Improved import reliability and data validation

The import process now handles transient storage errors more robustly by retrying reads and directory walks on rate limits or timeouts, ensuring temporary outages do not abort the import. It also enforces stricter data validation, rejecting varchar and array values that exceed their defined limits and preventing imports of invalid UTF-8 strings. Additionally, the system now correctly classifies persistent storage failures as system errors rather than bad input, allowing the retry mechanism to function as intended, and supports importing nullable or default-valued function output fields.

internal/util/importutilv2/common · high confidence

Introduce KV-based streaming coordination catalog with legacy key compatibility

The streaming coordination metadata is now persisted using a new KV-based catalog (streamingcoord) that stores control channels, producer channels, broadcast tasks, replication configuration, and versioning under the 'streamingcoord-meta/' prefix. To ensure smooth upgrades from older versions, the catalog automatically detects and repairs legacy keys that incorrectly include a trailing slash, migrating them to the canonical key format on read.

internal/metastore/kv/streamingcoord · high confidence

Introduce QueryCoord v2 parameter management and test utilities

The \internal/querycoordv2/params\ package now provides a centralized \Params\ singleton for QueryCoord v2 configuration, replacing previous ad-hoc parameter handling. This change introduces test-only utilities to support isolated testing, including functions to generate random etcd root paths, random increment ID allocators, and a deterministic error ID allocator, ensuring test cases do not interfere with each other's state.

internal/querycoordv2/params · high confidence

Introduce concurrent-safe proxy connection manager with automatic client cleanup

The proxy now uses a new, thread-safe connection manager to track client sessions. This manager automatically purges client information that has exceeded the configured TTL or when the total number of connections exceeds the system limit, ensuring that stale or excessive client records do not accumulate. It also provides a gRPC interceptor to keep client sessions active during RPC calls, improving the accuracy of connection lifecycle management.

internal/proxy/connection · high confidence

Introduce migration runner with etcd auth, TLS, and session safety

The migration tool now supports etcd authentication and TLS configuration, allowing secure connections to protected metadata stores. It also enforces stricter session management by watching for component sessions and exiting the migration process if any are detected or if the migration session itself expires, preventing concurrent or stale migrations.

cmd/tools/migration/migration · high confidence

Introduce non-blocking tombstone sweeper for safe cleanup

The tombstone sweeper in the root coordinator has been replaced with a new, concurrency-safe implementation that operates in the background. This change ensures that adding tombstones (such as those from dropped collections or partitions) is non-blocking, even if the background garbage collection process is performing slow external calls. The sweeper now uses a generation token mechanism to prevent race conditions where a concurrently re-registered entity with the same ID could be incorrectly deleted by an older sweep cycle.

internal/rootcoord/tombstone · high confidence

Introduce structured configuration parsing for the migration tool

The migration tool now uses a dedicated \Config\ struct in \cmd/tools/migration/configs\ to parse and validate migration settings from a YAML file. This change centralizes the handling of run parameters (such as source/target versions and backup options) and infrastructure details (MetaStore and Etcd configurations), ensuring that these values are correctly initialized and displayed before migration execution begins.

cmd/tools/migration/configs · high confidence

Introduces MVCC-based timetick tracking for WAL persistence and query plans

The streaming node now uses a Multi-Version Concurrency Control (MVCC) manager to track the state of Write-Ahead Log (WAL) entries. This change introduces separate tracking for durable persistence (VChannel MVCC) and query-plan visibility (Query MVCC), ensuring that queries see consistent data states based on confirmed timeticks. The implementation handles transaction visibility, recovery barriers, and collection lifecycle events (Create/Drop) to maintain accurate watermarks for both storage and query subsystems.

internal/streamingnode/server/wal/interceptors/timetick/mvcc · high confidence

Introduces WAL timetick acknowledgment tracking

The streaming node now tracks which timestamps have been acknowledged via a new \AckManager\ and related components in the Write-Ahead Log (WAL) interceptor. This change introduces a heap-based mechanism to manage unacknowledged timestamps, ensuring that the \LastConfirmedMessageID\ is correctly updated only after all preceding timestamps are confirmed, which addresses potential issues with high-concurrency writes where the last confirmed ID could previously be incorrect.

internal/streamingnode/server/wal/interceptors/timetick/ack · high confidence

Introduces centralized error definitions for the streaming service

The streaming service now uses a dedicated error package to define specific error types, including ErrClosed, ErrCanceledOrDeadlineExceed, ErrUnrecoverable, ErrFenced, and ErrIgnoredOperation. This centralization ensures consistent error handling and marking for all errors within the streaming service package.

internal/distributed/streaming/internal/errs · high confidence

Introduces lock interceptor to serialize exclusive WAL messages and prevent transaction ordering violations

A new lock interceptor has been added to the Write-Ahead Log (WAL) pipeline to manage concurrency and ensure message ordering. It acquires a global lock for PChannel-level messages and exclusive control-channel messages (such as manual flushes), while using per-vchannel locks for other exclusive operations. Crucially, when an exclusive message is appended, the interceptor fails all pending transactions on the affected vchannel to prevent transaction commit messages from appearing after the exclusive operation, which would violate WAL ordering. It also handles partial update CAS commits with specific locking to avoid unnecessary transaction failures.

internal/streamingnode/server/wal/interceptors/lock · high confidence

Introduces stateful lock for segment load/release lifecycle

The segment loading and releasing logic in the query node now uses a new \LoadStateLock\ mechanism that enforces strict state transitions (meta → loading → loaded → releasing → released) and prevents concurrent load/release operations on the same segment. This change ensures that release operations wait for all active references to complete before proceeding, avoiding race conditions and potential panics during segment lifecycle management.

internal/querynodev2/segments/state · high confidence

Introduces structured migration exit codes and colored console output

The migration tool now uses a dedicated console package to manage user-facing output and process termination. Users will see colored messages (green for success, red for errors, yellow for warnings) instead of plain text. Additionally, the tool distinguishes between normal completion and specific failure modes: it exits with code 1 if a backup is unfinished, code 2 if a backup finished but the migration failed, and code 100 for unexpected errors, allowing users or scripts to programmatically determine the migration's status.

cmd/tools/migration/console · high confidence

Introduces thread-safe singleton registry for the streaming balancer

A new singleton pattern has been added to the streaming coordination balancer to manage the global balancer instance and file resource checker. This change introduces a mutex-protected registry (singleton.go) that ensures thread-safe access to the balancer lifecycle (registration, retrieval, and release), addressing data races in concurrent access. A test utility (test\_utility.go) is also provided to reset this singleton state during testing.

internal/streamingcoord/server/balancer/balance · high confidence

JSON library switched to Sonic for improved performance

The internal JSON handling in the application now uses the bytedance/sonic library instead of the standard Go encoding/json package. This change replaces the underlying serialization and deserialization engine, which should result in faster JSON processing for REST API requests and internal data handling without altering the public API surface.

internal/json · high confidence

Milvus Go SDK version bumped to 3.0.0

The client/common package now defines the SDK version as 3.0.0, reflecting the migration of the Go client module path to v3. This change updates the version constant and includes a test to verify the version string is a valid semantic version.

client/common · high confidence

New DataNode index task scheduler and pool management

The DataNode index subsystem has been refactored to use a new task scheduler and concurrency pool. This introduces a \TaskQueue\ and \TaskScheduler\ to manage index, analyze, and stats tasks, enforcing a configurable concurrency limit via the \MaxVecIndexBuildConcurrency\ parameter. The scheduler now distinguishes between transient and terminal errors, ensuring that malformed user data (e.g., invalid JSON) fails immediately instead of retrying indefinitely, while transient issues still trigger retries. Additionally, a dedicated pool for vector index builds is introduced, which can be resized dynamically at runtime without restarting the DataNode.

internal/datanode/index · high confidence

New GPU Dockerfiles for Ubuntu 20.04 with Security and Configuration Updates

This change introduces new Dockerfiles for the Milvus GPU runtime on Ubuntu 20.04. The base image now uses nvidia/cuda:11.8.0-runtime-ubuntu20.04 and includes tzdata with UTC timezone configuration. The runtime image upgrades openssl to address [CVE redacted], switches the default user to 'milvus' for better security and OpenShift compatibility, and enables jemalloc memory profiling via environment variables (LD\_PRELOAD and MALLOC\_CONF).

build/docker/milvus/gpu/ubuntu20.04 · high confidence

New Go-layer aggregation engine for search queries

The internal aggregation logic for search queries has been replaced with a new Go-layer implementation located in \internal/agg\. This change introduces a dedicated aggregation framework that supports \SUM\, \COUNT\, \AVG\, \MIN\, and \MAX\ operators, including \GROUP BY\ capabilities via a \GroupAggReducer\. The new engine handles field type validation, null-value accumulation, and result serialization to protocol buffers, providing a more robust and testable foundation for aggregation queries compared to the previous approach.

internal/agg · high confidence

New RootCoord KV catalog with legacy snapshot and tombstone cleanup

The RootCoord metastore now uses a new KV-based catalog implementation (kv\_catalog.go) that manages collection, partition, field, function, and alias metadata. To ensure safe migration from the previous SuffixSnapshot mechanism, the system includes background garbage collection for legacy snapshot keys (legacy\_snapshot\_gc.go) and legacy tombstone-valued keys (legacy\_tombstone\_gc.go), along with tests verifying that these legacy markers are correctly skipped or removed during metadata loads.

internal/metastore/kv/rootcoord · high confidence

New assignment and broadcast services for streaming coordination

The streaming coordination server now exposes dedicated gRPC services for managing cluster assignments and message broadcasting. The new AssignmentService handles replicate configuration updates, including a force-promote path for primary-secondary failover, and watches channel assignments to keep cluster state consistent. The BroadcastService manages message distribution to streaming nodes and includes backward-compatibility logic to forward legacy import requests from older proxies to DataCoord.ImportV2, ensuring smooth rolling upgrades. These services replace previous ad-hoc handling with a structured, WAL-based DDL/DCL pattern.

internal/streamingcoord/server/service · high confidence

New binlog path compression and decompression utilities

The metastore now includes a new \binlog\ package that handles the compression and decompression of binlog paths for insert, delta, stats, and BM25 logs. This change introduces logic to replace full file paths with compact log IDs during storage (compression) and reconstruct the correct paths during retrieval (decompression). A key behavioral improvement is the handling of V3 segment delta logs: the code now explicitly skips V1-style path reconstruction for segments with a manifest path, preventing the generation of incorrect delta log paths. This ensures accurate binlog resolution for compaction and stats tasks, particularly when dealing with mixed compaction scenarios involving BM25 sparse vectors.

internal/metastore/kv/binlog · high confidence

New callback registration system for broadcast message acknowledgments

The broadcaster registry now uses a typed, future-based callback registration mechanism to handle message acknowledgments. This change introduces \MessageAckCallback\ for batched acknowledgments (where a callback fires once all replicas have acked) and \MessageAckOnceCallback\ for individual channel acknowledgments. The registry now explicitly registers handlers for a wide range of message types—including Import 2PC, Snapshot, RBAC, and Collection operations—ensuring that downstream components are notified precisely when broadcast messages are confirmed, replacing the previous validation-and-broadcast pattern with a structured callback lifecycle.

internal/streamingcoord/server/broadcaster/registry · high confidence

New delegator delete buffer implementation with multiple backends

The delegator's delete buffer has been replaced with a new modular implementation in \internal/querynodev2/delegator/deletebuffer\ that supports three distinct storage backends: a SkipList for remote segment L0 deletes, a Double Cache for fixed-size bounded buffering, and a List-based buffer for unbounded growth with metrics tracking. This new structure introduces L0 segment registration and retention management, timestamp pinning to prevent premature cleanup, and integrated metrics reporting for delete buffer size and row count, addressing previous data loss issues in the deprecated DoubleBuffer.

internal/querynodev2/delegator/deletebuffer · high confidence

New distributed DataNode client implementation

The distributed DataNode client has been replaced with a new implementation in \internal/distributed/datanode/client\. This new client wraps both \datapb.DataNodeClient\ and \workerpb.IndexNodeClient\ (for IndexNode operations) into a single \DataNodeClient\ struct, enabling unified gRPC communication with the DataNode. It introduces a generic \grpcclient.GrpcClient\ wrapper for retry logic and connection management, supports internal TLS configuration (including SNI and cert pools), and implements methods like \GetComponentStates\, \FlushSegments\, \GetMetrics\, and IndexNode-specific tasks (\CreateJob\, \QueryTask\, etc.). The change also includes comprehensive unit tests for client initialization, error handling, and RPC call mocking.

internal/distributed/datanode/client · high confidence

New gRPC client for QueryNode communication

The internal QueryNode client has been refactored to use a new generic gRPC client implementation, replacing the previous logic. This change introduces a cleaner initialization process that supports internal TLS configuration (including SNI and certificate pools) and integrates with the updated \grpcclient\ package for connection management and retry policies. The client now implements the full \QueryNodeClient\ interface, covering operations such as Search, Query, Load/Release collections and partitions, and component state retrieval, ensuring consistent error handling and context propagation across all RPC calls.

internal/distributed/querynode/client · high confidence

New gRPC service layer for QueryNode

The QueryNode now exposes its functionality through a new gRPC server implementation located in internal/distributed/querynode. This change introduces a dedicated service layer (service.go) that handles gRPC communication, including initialization, etcd client connection, and MixCoord client management, alongside a test-only API helper (api\_testonly.go) for internal testing access to the server ID. The existing service logic has been replaced by this new gRPC-based architecture, which is supported by comprehensive unit tests (service\_test.go) covering server lifecycle and component state interactions.

internal/distributed/querynode · high confidence

Optimistic CAS enforcement for partial updates in the WAL

The StreamingNode WAL now enforces optimistic Compare-And-Swap (CAS) for partial update transactions. The new \partialupdate\ interceptor tracks primary key versions and validates that reads remain consistent against concurrent writes before committing, rejecting conflicts with a retryable error. It also enforces strict transaction ordering (Begin before Body, Body before Commit) and uses row locks to serialize conflicting commits, ensuring data integrity during concurrent partial updates.

internal/streamingnode/server/wal/interceptors/partialupdate · high confidence

Optimized query stream result merging and in-memory streaming support

The internal streamrpc package now includes an in-memory streamer for local gRPC stream simulation and refactors the result cache to accumulate message sizes incrementally instead of recomputing them on every merge. This change eliminates an O(N²) performance bottleneck in the query stream path, ensuring that large delete-by-expression batches are merged efficiently and sent in properly sized chunks without triggering gRPC response size limits.

internal/util/streamrpc · high confidence

Per-file ID range reservation for import jobs

Import jobs now reserve a specific ID range for each file during the pre-import phase, ensuring that primary keys and row IDs remain identical across clusters. This two-phase mechanism allocates ranges based on exact row counts from pre-import, preventing ID collisions and ensuring consistency for both auto-ID and explicit primary key collections.

internal/util/importutilv2/importid · high confidence

Proxy HTTP listener refactoring and REST v1 deprecation

The proxy's HTTP listener implementation has been refactored into a dedicated \listener\_manager\ to support both port-sharing (via cmux) and dedicated HTTP ports with TLS. Additionally, the RESTful v1 API is now deprecated and disabled by default; when disabled, v1 routes return 404, while v2 routes remain available, and the metrics port is protected by an admin authentication middleware.

internal/distributed/proxy · high confidence

Redesigned broadcast task management with idempotency and trace propagation

The streaming coordinator's broadcaster has been refactored to support idempotent broadcasts and preserve distributed tracing context across the WAL. A new \ackCallbackScheduler\ now manages the lifecycle of broadcast tasks, ensuring that acknowledgment callbacks are executed in the correct order and that resource keys are locked until a message is acknowledged. The system now injects trace context into broadcast messages before they are persisted, allowing DDL operations to be traced end-to-end even after the original caller span has ended. Additionally, the broadcaster now supports force-promote operations for primary-secondary failover, fixing incomplete broadcasts during recovery, and includes comprehensive tests to verify idempotency, trace propagation, and resource key lock ownership.

internal/streamingcoord/server/broadcaster · high confidence

Refactor build scripts and introduce static analysis guards

The scripts directory has been reorganized with a new \3rdparty\_build.sh\ that manages Conan 2.x dependency resolution and handles environment isolation (e.g., unsetting \LD\_PRELOAD\ to prevent Python segfaults). Several new static analysis scripts have been added to enforce code quality and safety: \check\_segcore\_error\_boundaries.sh\ ensures C++ exceptions are properly classified before crossing the CGo boundary, \check\_segment\_timestamp\_usage.sh\ prevents incorrect timestamp access in Go code, and \check\_cpp\_fmt.sh\ validates C++ formatting. Additionally, \build\_plan\_parser.sh\ now builds the plan parser as a shared library with correct macOS RPATH handling, and \build\_cmek\_fixtures.sh\ automates the creation of CMEK plugin test fixtures.

scripts · high confidence

Refactored C++ build system with new CMake modules and compilation optimizations

The C++ build configuration has been refactored by introducing a set of new CMake modules in the \internal/core/cmake\ directory to standardize and optimize the build process. \DefineOptions.cmake\ introduces build-time configuration options, including toggles for GPU support, ccache, precompiled headers (PCH), and Unity builds, along with a dependency source selector (AUTO, BUNDLED, SYSTEM). \BuildUtils.cmake\ and \ThirdPartyPackages.cmake\ provide infrastructure for managing third-party dependencies via ExternalProjects, including support for cached package files and standardized toolchain settings. \FindClangTools.cmake\ updates the detection logic for clang-tidy and clang-format, searching for versions 10, 12, and 15. \Utils.cmake\ adds helper macros for versioning and source file management, explicitly filtering out test files from production builds. Finally, \milvus\_pch.hxx\ defines a precompiled header containing common STL and third-party includes to accelerate compilation across the codebase.

internal/core/cmake · high confidence

Refactored QueryCoord task scheduler and executor

The task management layer in \internal/querycoordv2/task\ has been refactored to improve scheduling precision and resource utilization. The new scheduler introduces a priority-based task queue (Low, Normal, High) and splits execution into separate channel and non-channel task pools, allowing independent capacity management. The executor now calculates task execution capacity based on the QueryNode's CPU count and configurable parallelism factors, preventing resource exhaustion. Additionally, task deadlines are now armed only upon actual dispatch to a node, ensuring that RPC timeout budgets are not consumed while tasks are waiting in the scheduler queue.

internal/querycoordv2/task · high confidence

Refactored TSO allocator with retry logic and max-logic limit

The internal TSO (Timestamp Oracle) allocator has been refactored to improve reliability and control timestamp generation. The \GlobalTSOAllocator\ now includes a \LimitMaxLogic\ flag (enabled by default) that prevents logical timestamp overflow by forcing a physical time increment when the limit is reached, rather than allowing wraparound. Additionally, the \InitTimestamp\ process now implements bounded retries for loading the persisted timestamp window from etcd, ensuring that transient backend errors (like leader changes) do not cause initialization to fail or regress the timestamp to the current time, which could break monotonicity.

internal/tso · high confidence

Refactored gRPC client to be generic and added configurable compression codecs

The gRPC client in internal/util/grpcclient has been refactored to use Go generics, allowing a single ClientBase implementation to serve different component types. This change also introduces support for multiple gRPC compression algorithms (zstd, snappy, and s2) with configurable levels and concurrency, improving network efficiency for internal service communication.

internal/util/grpcclient · high confidence

Refactored segment lifecycle management with configurable seal policies and retry resilience

The streaming node's Write-Ahead Log (WAL) shard interceptors now use a structured policy system to determine when growing segments are sealed, supporting triggers such as capacity limits, idle time, node memory pressure, and blocking L0 state. Segment allocation and flushing are handled by dedicated asynchronous workers that implement exponential backoff retries; these workers build fresh messages on each attempt to prevent panics caused by reusing contaminated state after WAL append failures. Additionally, a jitter-based limitation policy is applied to segment sizes to scatter sealing times and reduce contention.

internal/streamingnode/server/wal/interceptors/shard/shards · high confidence

Refactored sync task dispatcher and growing-source flush handling

The sync manager's task execution model has been refactored to use a new key-based lock dispatcher that enforces per-segment serial execution while allowing concurrent processing across different segments, with semaphore-based backpressure to prevent queue overflow. Additionally, the growing-source flush path now supports StorageV3 manifests, where the manifest serves as the authoritative source for binlog paths and statistics are carried in the segment summary rather than per-field binlog arrays, and L0 segments now correctly provide their own start positions during sync.

internal/flushcommon/syncmgr · high confidence

Regenerated KV interface mocks with context support

The mock implementations for MetaKv, TxnKV, and WatchKV in internal/kv/mocks have been regenerated using mockery v2.53.3. This update aligns the mocks with the updated KV operation interfaces, specifically adding context.Context parameters to methods such as Has, Load, and CompareVersionAndSwap, and importing the predicates package from pkg/v3/kv/predicates to support transactional key-value operations.

internal/kv/mocks · high confidence

Replicate interceptor now rolls back in-flight transactions on force promote

The new replicate interceptor in the streaming node's write-ahead log (WAL) ensures transactional consistency during failover by rolling back all in-flight transactions when a replicate configuration change with the 'force promote' flag is applied. This change introduces a \replicate\ interceptor that coordinates with the \txn\ manager; specifically, when a \ForcePromote\ header is detected, the interceptor calls \RollbackAllInFlightTransactions\ on the transaction manager after persisting the config change to the WAL. This guarantees that no partial or conflicting transactions remain active after a primary-secondary failover event.

internal/streamingnode/server/wal/interceptors/txn · high confidence

RocksDB package configuration and installation rules added

The build system now explicitly installs RocksDB header files to the include directory and generates a pkg-config file (rocksdb.pc) to help downstream components locate the library and its dependencies (zlib, bz2).

internal/core/thirdparty/rocksdb · high confidence

Streaming node recovery snapshot persistence and fencing

The streaming node metastore now persists recovery snapshots (vchannels, segment assignments, schemas, and checkpoints) using guarded compare-and-swap transactions. This ensures that only the current owner can advance the consume checkpoint, fencing out stale or takeover survivors, and guarantees that component updates are applied atomically or in ordered batches with post-commit verification to handle transient KV errors.

internal/metastore/kv/streamingnode · high confidence

Streaming service delegation logic and environment configuration

The streaming utility now exposes functions to control and query the streaming service state via the MILVUS\_STREAMING\_SERVICE\_ENABLED environment variable. Specifically, UseStreamingQueryNodeAsDelegator determines whether query coordinators place shard delegators on streaming query nodes; this behavior is enabled only when the streaming service is active and no installed form is present, ensuring that installed forms continue to use regular query nodes for delegators. The package also includes helper functions to set, check, and enforce the streaming service enabled state, along with tests verifying the delegation logic across different form and streaming configurations.

internal/util/streamingutil · high confidence

Support for default values and new data types in field validation

The internal utility for handling null and default values now supports extracting default values for a broader range of data types, including the new Timestamptz type and Geometry (converted from WKT to WKB). It also validates that the length of valid data arrays matches the expected row count for nullable fields, ensuring data integrity during import operations.

internal/util/nullutil · high confidence

Unified MRB1 limit generation to prevent Go/C++ divergence

The \genmrb1limits\ tool now generates the C++ header \internal/core/src/common/MRB1Limits.generated.h\ directly from the Go constants in \pkg/util/roaringfilter\, eliminating the previous practice of maintaining a separate, manually synchronized copy in the C++ codebase. This ensures that MRB1 envelope parameters (such as magic bytes, version, and size limits) are enforced consistently by both the Go proxy validator and the C++ segcore, preventing production issues where one side would admit a request the other would reject. A new test, \TestGeneratedHeaderIsCurrent\, verifies that the generated header matches the current Go constants, catching any drift at build time.

cmd/tools/genmrb1limits · high confidence

Unified filesystem root path and metrics collection for storage v2

The storage v2 layer now ensures local storage keys are complete paths by rooting the underlying loon filesystem at "/" for local storage (while keeping the configured root path for remote/object storage), preventing double-prefixing of keys. Additionally, it exposes cached filesystem metrics (read/write counts, bytes, failures, and multipart upload stats) for all active filesystems, allowing users to monitor storage performance and health directly from the Go storage layer.

internal/storagev2 · high confidence

WAL shard interceptor now supports version-gated write-before function materialization

The shard interceptor in the Write-Ahead Log (WAL) layer has been refactored to manage function runner lifecycle and enforce schema version checks during insert operations. It now allocates and updates function runners based on collection schemas and gates the execution of write-before function materialization behind a cluster version flag, ensuring legacy behavior is preserved until the upgrade is complete. Additionally, the interceptor now handles a broader range of DDL/DCL messages (including TruncateCollection and AlterCollection) and validates schema versions to prevent mismatches during data ingestion.

internal/streamingnode/server/wal/interceptors/shard · high confidence

WebUI assets updated with new styles and React 18.3.1 runtime

The internal HTTP webui now serves updated frontend assets, including a new CSS file defining styles for buttons, icons, tables, and form controls, and a JavaScript bundle that embeds the React 18.3.1 production runtime (react, react-dom, and scheduler). These changes refresh the visual styling and underlying UI framework version used by the management WebUI.

internal/http/webui · high confidence

Fixes

Azure cross-account snapshot copy and Arrow null-sentinel safety

The storage layer now supports cross-account Azure snapshot copies by accepting a source SAS token and endpoint, enabling secure data movement between Azure accounts without exposing credentials. Additionally, a regression in Arrow C Data Interface handling of all-null string arrays (where a 0x1 sentinel value could be misinterpreted as a Go pointer) is fixed, preventing potential crashes or memory corruption during import and serialization.

internal/storage · high confidence

Fix external collection query failures and add partition key routing support

This change introduces two key capabilities in the typeutil package. First, it fixes a bug where external collection queries failed with a 'fieldID(1) not found' error when retrieving empty results; the new \FillRetrieveResultIfEmpty\ logic now correctly generates empty field data for system fields (RowID, Timestamp) that are not present in the user schema. Second, it adds support for partition keys by introducing \HashKey2Partitions\, which allows routing queries to specific partitions based on Int64 or VarChar partition key values using a hash-based routing table.

internal/util/typeutil · high confidence

Limit zstd compression concurrency to reduce memory usage

The internal zstd compression codec now limits the number of concurrent compression threads based on a configurable setting (defaulting to 1 if not explicitly set), rather than using all available CPU cores. This change prevents excessive memory consumption during binlog generation, addressing potential out-of-memory issues when CPU is throttled or under high load.

internal/storage/compress · high confidence

Preserve segment insert log paths for text index

The datanode now preserves explicit insert log paths when building segment insert file information for text indexes. If a binlog already has a LogPath set, that path is used directly; otherwise, the system falls back to constructing the path from the root path, collection, partition, segment, and column group IDs. This ensures that pre-existing path metadata is not overwritten during index operations.

internal/datanode/util · high confidence

Regenerated gRPC mocks for Go 1.22 compatibility

The internal mock implementations for \google.golang.org/grpc\ (specifically \mock\_ClientConn\ and \mock\_ClientStream\) have been regenerated using mockery v2.53.3 to resolve panics and unavailability issues encountered with Go 1.22. This update ensures that tests relying on these mocks for gRPC resolver and client stream interactions run correctly on the newer Go version.

internal/mocks/google.golang.org · high confidence

Regenerated streaming service mocks for Go 1.22 compatibility

The mock implementations for the streaming service utilities (Discoverer, Service, Builder, and Resolver) have been regenerated using mockery v2.53.3 to resolve panics and unavailability issues encountered after upgrading to Go 1.22. This update ensures that the test doubles for the streaming node client and gRPC utilities function correctly with the new Go version, maintaining the integrity of unit tests that rely on these mocks.

internal/mocks/util/streamingutil · high confidence

Restrict function parameter alterations and enforce MinHash validation

The system now prevents users from altering semantic parameters (such as model name, dimension, or endpoint) on existing functions, allowing only connection and runtime configuration changes (like URL, integration ID, and model deployment ID) for TextEmbedding functions to avoid mixing incompatible vectors in output fields. Additionally, schema validation for MinHash functions is now enforced on all DDL paths, including those where runtime checks are disabled, and invalid TEXT inputs are rejected.

internal/util/function/validator · high confidence

Test coverage

2 commits adding/updating tests in internal/core/thirdparty/tantivy/tantivy-binding/tests; 2929 commits adding/updating tests in tests; Added Lindera dictionary sharing regression test and allocation example; Added autogenerated mocks and test utilities for the CSegment interface; Added benchmarks for FastMemcpy and sorted scalar lookups; Added end-to-end tests for the Arrow retrieve transport; Added mock QueryNode for testing; Added mock for LatestMVCCTimeTickGetter interface; Added mock for RecoveryStorage interface; Added mock for SealOperator interface; Added mock for TombstoneSweeper interface; Added mock implementation for QueryHook interface; Added mock implementation for WAL Manager; Added mock implementation for the streaming coordinator balancer interface; Added mocks for WAL interceptor components; Added mocks for WAL replicate interceptors; Added mocks for WAL streaming node interfaces; Added mocks for streaming coordination and node catalog interfaces; Added mocks for streaming coordination client interfaces; Added mocks for streaming coordinator broadcaster interfaces; Added streaming service mocks for testing; Added tests for local storage migration logic; Added tests for mgit design doc reference validation; Added unit tests for JSON stats index components; New C++ unit test infrastructure and expanded test coverage; New C++ unit test utilities for segcore and storage; New mock gRPC clients for Milvus internal services; New test utilities for StructArray and environment reset; Regenerated MockChunkManager to support mmap and updated method signatures; Regenerated ShardManager mock to match updated interface; Regenerated TSO Allocator mock for test stability; Regenerated WAL interceptor mocks for updated interfaces; Regenerated internal mocks with mockery v2.53.3; Regenerated metastore catalog mocks for DataCoord, QueryCoord, and RootCoord; Regenerated mock for HandlerClient interface; Regenerated mocks for MsgHandler and BinlogIO interfaces; Regenerated rootcoord mock for updated IMetaTable interface; Regenerated streaming node client mocks for updated interfaces; Updated ManagerClient mock to support resource group-aware streaming node operations.

Dependencies

Knowhere submodule updated to commit 16e873b1 with pkg-config support

The Knowhere vector search library submodule has been updated to commit 16e873b1. This change introduces a pkg-config file (knowhere.pc.in) to facilitate library discovery and adds CMake logic to correctly install runtime shared libraries for optional dependencies like SVS and cuVS (specifically libsvs\_runtime.so, librmm.so, and librapids\_logger.so) alongside the main knowhere library.

internal/core/thirdparty/knowhere · high confidence

Major dependency and toolchain upgrades across Go, Rust, and Python stacks

This release upgrades the Go toolchain to 1.26.6 and the Go SDK client to v3, alongside significant bumps to core libraries including gRPC (v1.83.2), etcd (v3.5.33), Pulsar (v0.19.0), and Tantivy. The Rust-based search engine dependencies are also updated, and Python test requirements are refreshed to support newer runtimes and tooling.

(dependencies) · high confidence

Housekeeping

Added OWNERS file for deployments directory; Added OWNERS files for internal/distributed and internal/kv; Added OWNERS files for streaming components.

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 52 → 48 (-4.1)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.3) — scores are not directly comparable.

Lenses

  • Code Health 91 → 77 (-14.1)
  • Architecture 100 → 84 (-15.6)
  • Maturity 84 → 87 (+2.4)
  • Readiness 55 → 55 (-0.7)
  • Security 41 → 35 (-5.9)
  • Accessibility 49 → 47 (-1.7)
  • Performance 88 → 97 (+8.7)

Resolved (228)

  • ADR has no Consequences section (docs/design-docs/design_docs/20260413-search_embedded_agg.md)
  • ADR has no Consequences section (docs/design-docs/design_docs/20260720-segment-reopen-request-read-lease-drain-design.md)
  • ADR has no Context and Decision section (docs/design-docs/design_docs/20261005-arrow-retrieve-no-fielddata.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20220725-dynamic-config.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260129-search-orderby.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260131-client_side_telemetry.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260203-query-orderby.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260204-channel_exclusive_mode.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260209-pinyin_filter.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260226-manifest-format.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260227-yc-text-embedding-provider.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260305-local_format.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260313-scalar_index_version_management.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260324-segment_filter_pk_predicate_pruning.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260407-text_lob_storage.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260409-regex_filter.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260410-json_path_index_multi_type.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260521-external-table-function-output.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260601-rbac-user-description.md)
  • ADR has no Decision and Consequences section (docs/design-docs/design_docs/20260602-expression-result-cache.md)
  • …and 208 more

New (5894)

  • ADR declares no status (docs/design-docs/design_docs/20210521-datanode_recovery_design.md)
  • ADR declares no status (docs/design-docs/design_docs/20210604-datanode_flowgraph_recovery_design.md)
  • ADR declares no status (docs/design-docs/design_docs/20210731-index_design.md)
  • ADR declares no status (docs/design-docs/design_docs/20211109-milvus_flush_collections.md)
  • ADR declares no status (docs/design-docs/design_docs/20211115-milvus_drop_collection.md)
  • ADR declares no status (docs/design-docs/design_docs/20211214-milvus_hybrid_ts.md)
  • ADR declares no status (docs/design-docs/design_docs/20211215-milvus_timesync.md)
  • ADR declares no status (docs/design-docs/design_docs/20211217-milvus_create_collection.md)
  • ADR declares no status (docs/design-docs/design_docs/20211221-retrieve_entity.md)
  • ADR declares no status (docs/design-docs/design_docs/20211223-knowhere_design.md)
  • ADR declares no status (docs/design-docs/design_docs/20211224-drop_collection_release_resources.md)
  • ADR declares no status (docs/design-docs/design_docs/20211227-milvus_create_index.md)
  • ADR declares no status (docs/design-docs/design_docs/20220105-proxy.md)
  • ADR declares no status (docs/design-docs/design_docs/20220105-query_boolean_expr.md)
  • ADR declares no status (docs/design-docs/design_docs/20220105-root_coordinator_recovery_on_power_failure.md)
  • ADR declares no status (docs/design-docs/design_docs/20220725-dynamic-config.md)
  • ADR declares no status (docs/design-docs/design_docs/20230403-search_by_pk.md)
  • ADR declares no status (docs/design-docs/design_docs/20230405-default_value.md)
  • ADR declares no status (docs/design-docs/design_docs/20230418-querynode_v2.md)
  • ADR declares no status (docs/design-docs/design_docs/20230511-collection_level_autocompaction_switch.md)
  • …and 5874 more

Changes since last survey

  • 257 commits — 156 feature/other, 101 fixes

By area

  • internal/core — 55 commits
  • internal/datacoord — 27 commits
  • tests/python_client — 25 commits
  • internal/proxy — 20 commits
  • internal/util — 17 commits
  • internal/querynodev2 — 12 commits
  • internal/streamingnode — 11 commits
  • (root) — 7 commits
  • internal/distributed — 6 commits
  • build/docker — 5 commits
  • internal/querycoordv2 — 5 commits
  • tests/integration — 5 commits
  • internal/rootcoord — 4 commits
  • internal/views — 4 commits
  • pkg/util — 4 commits
  • client/milvusclient — 3 commits
  • configs/milvus.yaml — 3 commits
  • deployments/docker — 3 commits
  • internal/datanode — 3 commits
  • internal/parser — 3 commits

Notable commits

  • fix: [automated] fix: bump klauspost/compress to v1.20.2 ([CVE redacted]) (#55260)
  • fix: [automated] fix: bump protobuf to 6.33.5 in pyudf runtime for [CVE redacted] (#54945)
  • fix: build(deps): upgrade Go to 1.26.10 to fix [CVE redacted] (#54597)
  • fix: enhance: [ElemNull 4] row-level and element-level filters on element-nullable struct sub-fields, with search tests and struct sub-field fixes (#54792)
  • fix: fix(ci): stop macOS Conan recipe auto-updates (#54936)
  • fix: fix: Avoid ambiguous optional FieldId comparisons (#54963)
  • fix: fix: Bound Python E2E collection cleanup and report failures (#54349)
  • fix: fix: Correct embedded UniDic loader name (#54854)
  • fix: fix: Keep the QuotaCenter write cool-off after the first round (#54831)
  • fix: fix: Keep unassignNode's etcd retry from being cut short by the caller (#54801)
  • fix: fix: Let a stopping querynode leave when no node can take its data (#54812)
  • fix: fix: Limit QueryNode read retries for replica failover (#54889)
  • fix: fix: Preserve derived artifacts during deltalog refresh (#52892)
  • fix: fix: Preserve segment index types across snapshot restore (#54989)
  • fix: fix: Recover etcd watches after leader changes (#54707)
  • fix: fix: Recover from all transient etcd errors, not only leader changes (#54733)
  • fix: fix: Replay deltalogs when reopening external milvus-table segments (#54736)
  • fix: fix: Restore V3 column splits during recovery (#54651)
  • fix: fix: Reuse StorageV2 footers across batches (#54482)
  • fix: fix: Stabilize the Flush rate-limit integration test (#55311)
  • …and 237 more

Architecture

  • Containers 0 added · 0 removed · contexts 3 added · 0 removed · edges 1 added · 0 removed

Added bounded contexts (3)

  • github.com/milvus-io/milvus
  • github.com/milvus-io/milvus/client/v3
  • github.com/milvus-io/milvus/pkg/v3

Added dependency edges (1)

  • github.com/milvus-io/milvus → github.com/milvus-io/milvus/pkg/v3 (coupling)

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

milvus-io/milvus was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 13 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 03d63d65d0d6df37c7c6f2df37300addb791dffd — the exact code this score is about.
  • Scored under rubric-2026.11.3 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.61+preprod-f3449e6eb431.