milvus-io/milvus

This system is a distributed vector database platform that manages the full lifecycle of vector and scalar data, including ingestion, indexing, and search. It provides a Go SDK client for interacting with the system and supports various index types and data formats. The codebase includes extensive tooling for building, testing, and deploying the system across different environments, including Docker, Kubernetes, and Windows. It also features a comprehensive set of migration and maintenance utilities for data consistency, metadata upgrades, and configuration management.

92 changes.

This period focused on modernizing the codebase through a comprehensive C++ build system overhaul, the introduction of rigorous code quality and development tooling, and the implementation of a new generic gRPC client architecture. Concurrently, the project established robust metadata management with composite update support, added a metadata migration tool for version upgrades, and expanded test coverage across the distributed components.

150 changes.

This period focused on implementing the new streaming node architecture, introducing a Write-Ahead Log (WAL) system with interceptors, time-tick synchronization, and adaptive rate limiting. Concurrently, the project developed a comprehensive Go SDK client (v3.0.0) with support for row-based inserts and complex data types, while also adding data import utilities for various file formats.

71 changes.

The codebase underwent a major architectural consolidation, merging RootCoord, DataCoord, and QueryCoord into a single MixCoord service with a unified client. This period also introduced a comprehensive streaming node infrastructure, including a Write-Ahead Log (WAL) system, a streaming node manager, and a dedicated broadcaster for message coordination. Additionally, the system expanded its capabilities with CDC replication support, external function execution, and advanced text search indexing.

A new /livez HTTP endpoint has been added to the health check service, providing a lightweight liveness probe suitable for Kubernetes. This endpoint performs a minimal, non-blocking check to confirm the Go scheduler is active and the HTTP server can respond immediately, without making external calls or checking other component states. The implementation includes a dedicated handler and supporting constants for content types, ensuring compatibility with standard liveness probe expectations.

internal/http/healthz · high confidence

The Tantivy third-party module now includes a CMake build configuration (CMakeLists.txt) that compiles the Rust-based Tantivy binding into a static library and installs the resulting headers. This change introduces the C++ wrapper headers (tantivy-wrapper.h, tokenizer.h, etc.) and demo/test files (bench.cpp, test.cpp, etc.) that enable the C++ side of the inverted index and text search functionality.

internal/core/thirdparty/tantivy · high confidence

A new CDC server implementation is introduced in the distributed layer, providing lifecycle management (Prepare, Run, Stop, Health) for the Change Data Capture subsystem. This change exposes the CDC functionality as a managed component within the distributed runtime, allowing the system to start, monitor, and gracefully stop the CDC service.

internal/distributed/cdc · high confidence

Introduces the core components for Change Data Capture (CDC) replication, including a controller that watches etcd for replicate pchannel metadata changes and manages the lifecycle of channel replicators. The diff adds the \internal/cdc/controller\ package for scheduling and watching replication tasks, the \internal/cdc/cluster\ package for managing Milvus client connections with per-cluster TLS and gRPC authority support, and the \internal/cdc/replication\ package for managing replicator instances. Mocks and tests are also added for these new internal interfaces.

internal/cdc · high confidence

A new 'cgoconverter' package is introduced to manage the conversion of C-allocated byte arrays into Go byte slices. The implementation uses a global singleton 'BytesConverter' that tracks C pointers via a concurrent map, allowing safe, zero-copy access to C memory from Go. The API provides 'UnsafeGoBytes' to obtain a Go slice backed by C memory, 'Release' to free specific pointers, and 'Extract' to retrieve raw pointers without immediate deallocation. Test coverage includes concurrent access validation.

internal/util/cgoconverter · high confidence

The build/deb directory now includes the complete set of files required to build a .deb package for Milvus on Debian/Ubuntu systems. This includes the build script (build\_deb.sh), packaging metadata (debian/control, rules, compat, source/format), and system integration files (milvus.service, milvus.conf). Users can now build and install Milvus as a native Debian package, which manages the binary, libraries, configuration, and systemd service automatically.

build/deb · high confidence

Introduces a new FM-index based scalar index that supports exact substring, prefix, and suffix matching on VARCHAR columns. The change vendors the \fmindex\ and \libsais\ libraries (Apache 2.0) and adds C++ header implementations for the FM-index, wavelet matrices, and suffix arrays. This enables high-performance, case-insensitive, and document-scoped text search capabilities for scalar filtering.

internal/core/thirdparty/fmindex, internal/core/thirdparty/libsais · high confidence

Users can now perform bulk data imports, list import jobs, track import progress, and commit or abort import jobs using new Go SDK functions (BulkImport, ListImportJobs, GetImportProgress, CommitImport, AbortImport) that wrap the Milvus RESTful API endpoints. The implementation includes request builders, response structs, and helper methods for configuring API keys and partition names.

client/bulkwriter · high confidence

The internal/agg package now includes the Go implementation for query aggregation, introducing support for sum, count, avg, min, and max operators. This includes the core aggregate logic in aggregate.go, the GroupAggReducer for handling grouped results, and utility functions for field access and type checking. Tests confirm the correct behavior of these new components.

internal/agg · high confidence

Users can now import data from JSON and JSONL/NDJSON files using the v2 import utility. The new \internal/util/importutilv2/json\ package provides a reader that supports both standard JSON arrays and JSON Lines formats, handling various data types including vectors, arrays, and struct arrays. This enables direct bulk insertion of JSON-formatted data into the system.

internal/util/importutilv2/json · high confidence

A new MVCC (Multi-Version Concurrency Control) manager has been introduced to track the state of the Write-Ahead Log (WAL). This component maintains the last confirmed timetick for the entire partition channel and tracks the maximum timetick persisted for each individual virtual channel. It handles updates from incoming messages, ensuring that transactional messages do not advance the timetick until they are committed, and syncs the global state when a timetick message is received.

internal/streamingnode/server/wal/interceptors/timetick/mvcc · high confidence

A new shell script, export-milvus-log.sh, has been added to the deployments/export-log directory to facilitate debugging by exporting logs from running and previously restarted pods. The script supports exporting logs for Milvus, etcd, Minio, Pulsar, and Kafka components, with options to filter by time range, specify namespace, and handle installations via Helm or Milvus operator.

deployments/export-log · high confidence

A new \qn\_wrapper.go\ file introduces a \QueryNode\ client wrapper that delegates calls to the underlying \QueryNode\ implementation. This wrapper enables the system to use the \QueryNode\ interface as a client, which is specifically useful for standalone mode to avoid gRPC overhead. The change includes a corresponding test suite (\qn\_wrapper\_test.go\) that validates the wrapper's behavior for all supported methods, including search, query, and statistics operations.

internal/util/wrappers · high confidence

The RESTful API now supports managing file resources, including adding, removing, and listing them. This change introduces new endpoints for file resource operations, allowing users to interact with file-based resources through the HTTP server.

internal/distributed/proxy/httpserver · high confidence

Introduced a new RPM build specification (milvus.spec) that packages Milvus binaries, shared libraries, configuration files, and systemd service units (milvus, milvus-etcd, milvus-minio) into a distributable RPM. The package includes dependencies on libstdc++, libgomp, tbb-devel, and tzdata, and installs components to /usr/bin, /lib64/milvus, /etc/milvus, and /etc/systemd/system. Additionally, a setup-env.sh script was added to configure the build environment, installing devtoolset-11-gcc, OpenBLAS, TBB, Boost, and Go 1.26.5 to support the RPM build process.

build/rpm · high confidence

The system now supports TiKV as a backend for metadata storage. This change introduces the \internal/kv/tikv\ package, which provides a transactional key-value store implementation using the TiKV client-go library. The implementation includes configuration options for request timeouts, support for both local and remote TiKV instances for testing, and specific handling for transaction retries and error states. This enables users to deploy Milvus with TiKV for scalable metadata management.

internal/kv/tikv · high confidence

A new Zstd compression codec has been added to the storage layer, implementing the Codec interface to handle binlog generation. This implementation introduces a configurable concurrency limit for Zstd compression (defaulting to 1 to prevent out-of-memory issues during high-concurrency binlog generation) and registers the codec for use with the Zstd algorithm.

internal/storage/compress · high confidence

The migration tooling now includes a new \allocator\ package that provides a thread-safe, atomic ID generator. This new component exposes an \Allocator\ interface and an \AtomicAllocator\ implementation that supports configurable initial values and step deltas, enabling safe concurrent ID generation during migration processes.

cmd/tools/migration/allocator · high confidence

A new migration tool is introduced in the \deployments/migrate-meta\ directory, providing an automated script (\migrate.sh\) and documentation (\README.md\) to upgrade a Milvus cluster from version 2.1.x to 2.2.0. The script handles the four-step migration process: stopping components, backing up metadata, performing the migration, and restarting with the new image. It supports customizing the target image tag, specifying storage classes, configuring external etcd services, and optionally cleaning up the migration pod after completion.

deployments/migrate-meta · high confidence

A new command-line utility at cmd/tools/binlog has been added to the project. This tool allows users to print the contents of binlog files directly from the command line, providing a convenient way to inspect binary log data.

cmd/tools/binlog · high confidence

A new entrypoint script is added to the Docker builder environment to configure the build environment. It ensures the home directory exists, copies default shell configurations, and dynamically sources toolchain scripts for GCC 11 (via devtoolset and rhel-scl), LLVM 11, and other development tools. This script is intended for use in local development or CI within Kubernetes or Docker environments, ensuring consistent build conditions.

build/docker/builder · high confidence

The \internal/util/componentutil\ package now includes a \ComponentStateService\ to track and manage the lifecycle states (Initializing, Healthy, Abnormal, Stopping) of Milvus components, along with corresponding unit tests. Additionally, the package provides generic helper functions (\WaitForComponentStates\, \WaitForComponentInitOrHealthy\, \WaitForComponentInit\, \WaitForComponentHealthy\) to wait for specific component states, and utility functions (\CheckHealthRespWithErr\, \CheckHealthRespWithErrMsg\) to construct health check responses.

internal/util/componentutil · high confidence

The QueryCoord KV catalog now implements a composite Update method that applies multiple metadata actions (such as saving or removing replicas) in a single transaction. This change introduces a new \update.go\ file and corresponding tests that validate the atomic execution of mixed save and delete operations, ensuring that replica updates are handled consistently within the key-value store.

internal/metastore/kv/querycoord · high confidence

The Go SDK client now provides typed constructors and parameter builders for a wide range of vector and scalar index types, including Flat, HNSW (with quantization and refine options), DiskANN, AISAQ, IVF (including RaBitQ), MinHash LSH, GPU indexes (CAGRA, IVF-Flat/PQ), and AutoIndex. Each index type has a dedicated struct and constructor (e.g., NewHNSWIndex, NewDiskANNIndex, NewIvfRabitQIndex) that correctly sets the index type and metric type in the parameters. The diff also introduces a generic index wrapper (WithExtraIndexParams) to allow arbitrary build parameters without losing type safety, and ensures that unset optional parameters are omitted from the wire format to avoid server-side validation failures. Tests verify that index types are correctly reported and that extra parameters are properly merged or ignored when they conflict with reserved keys.

client/index · high confidence

A new Go-based tool has been added at cmd/tools/config-docs-generator. This tool parses the Milvus YAML configuration file and automatically generates structured Markdown documentation for system configuration parameters, including section headers, field descriptions, and default values.

cmd/tools/config-docs-generator · high confidence

A new console package is introduced to the migration tool, providing colored console output (green for success, red for errors, yellow for warnings) and a structured exit code system. The exit system supports abnormal and normal exit paths, allowing the migration tool to report specific failure states such as 'FailWithBackupUnfinished' or 'FailButBackupFinished', ensuring the tool exits with appropriate codes and messages for session management.

cmd/tools/migration/console · high confidence

Added new helper functions in the streaming service's context utility package to manage metadata for cluster IDs, consumer creation, producer creation, and server ID picking. These utilities allow passing structured request data and identifiers through gRPC metadata and standard Go contexts, enabling cleaner state management across streaming service handlers.

internal/util/streamingutil/service/contextutil · high confidence

A new \count\util.go\ file introduces helper functions to extract and format count results from various internal and segment core retrieve results. The implementation includes \CntOfInternalResult\, \CntOfSegCoreResult\, and \CntOfQueryResults\ to parse count data, alongside corresponding \WrapCntTo\\ functions to construct these result structures. Tests are added to verify the correct extraction and wrapping of count values.

internal/util/funcutil · high confidence

The codebase now includes a new internal utility for service load balancing, introducing a base balancer implementation that tracks both ready and unready SubConns. This allows the streaming service to make more informed routing decisions by distinguishing between healthy and unhealthy connections, with corresponding tests verifying attribute preservation in the picker build info.

internal/util/streamingutil/service/balancer · high confidence

A new command-line tool located at cmd/tools/datameta has been added to help users inspect segment metadata stored in etcd. The tool connects to an etcd instance and iterates through segment information, allowing users to filter results by collection, partition, segment ID, or channel name. It supports a --detail flag to display comprehensive information about binlogs, statslogs, and deltalog entries for each segment, facilitating debugging and verification of data coordination state.

cmd/tools/datameta · high confidence

The \configs\ directory now contains default configuration files for Milvus, including \milvus.yaml\ which defines settings for etcd, TiKV, MinIO/S3, message queues, and other components. This provides a standardized starting point for deployment and configuration.

configs · high confidence

The repository now includes configuration files for C++ code formatting (.clang-format), static analysis (.clang-tidy, .clang-tidy-ignore), and Go linting (.golangci.yml). A pre-commit hook is configured to run golangci-lint, typos, and ruff (Python linting/formatting). Additionally, a VS Code devcontainer (.devcontainer.json) and .env file for Docker Compose are added to standardize the local development environment.

(repo-wide) · high confidence

The internal/kv/etcd package now includes an implementation of the KV store backed by an embedded etcd instance, allowing the system to run without an external etcd dependency. This change introduces new files (embed\_etcd\_kv.go, metakv\_factory.go, options.go, util.go) and their corresponding tests, enabling the application to start and operate with an in-process etcd server. The factory pattern is updated to select between external and embedded etcd based on configuration, and utility functions are added to handle context timeouts and predicate parsing for the embedded instance.

internal/kv/etcd · high confidence

The client/entity package now includes new entity models and types to support advanced collection and data management features. Users can now define collection attributes such as TTL and auto-compaction settings, manage external table refresh states and job information, and define functions (including rerank and minhash) with their parameters. The schema model has been expanded to support struct arrays, external data sources, and function definitions, while common types like metric types, load states, and RBAC entities (users, roles, privilege groups) are also introduced to support the broader Go SDK API surface.

client/entity · high confidence

The \internal/util/function/chain/expr\ package introduces new expression types to support a function chain pipeline for search reranking. This includes \BaseExpr\ as a base struct for expression implementations, \BoostScoreExpr\ to apply boost scores, \DecayExpr\ to calculate decay factors (supporting Gaussian, exponential, and linear decay functions), and \NumCombineExpr\ to combine multiple numeric columns using modes like multiply, sum, max, min, average, and weighted combinations. These components enable more flexible and composable scoring and reranking logic within the function chain.

internal/util/function/chain/expr · high confidence

A new gRPC-based implementation of the broadcast service has been added to the streaming coordination client. This introduces a concrete GRPCBroadcastServiceImpl that handles Broadcast and Acknowledgement operations via the streamingpb service, enabling message broadcasting and acknowledgment through the streaming protocol. A corresponding test file has also been added to verify the new broadcast functionality.

internal/streamingcoord/client/broadcast · high confidence

A new \idalloc\ package has been introduced to manage timestamp and ID allocation. It provides a unified \Allocator\ interface that coordinates between a local in-memory allocator and a remote coordinator (RootCoord) via a \MixCoordClient\. The implementation features a fast-path allocation strategy that attempts to satisfy requests locally before falling back to remote calls, with synchronization mechanisms (\Sync\, \SyncIfExpired\) to keep the local cache consistent with the remote source. This change replaces previous ad-hoc allocation logic with a structured, testable component that handles both timestamp (TSO) and general ID generation.

internal/util/idalloc · high confidence

A new \lazygrpc\ package is introduced to provide a lazy gRPC connection and service wrapper. The \Conn\ interface and \NewConn\ function manage an asynchronous, retrying gRPC dial operation, allowing callers to wait for the connection to become ready without blocking the initial setup. The \Service\ interface wraps a gRPC client with a service creator, enabling lazy initialization of gRPC services. This change adds new internal utilities for managing gRPC connections and services, supporting asynchronous initialization and retry logic.

internal/util/streamingutil/service/lazygrpc · high confidence

A new Go file, client/ruleguard/rules.go, has been added to the codebase. This file introduces a suite of linting rules for the go-ruleguard static analysis tool, covering checks for unnecessary type conversions, improper time.Time comparisons, variable naming conventions, and various code smells such as identical if/else bodies, self-assignments, and odd bitwise or arithmetic expressions. These rules will help developers identify and fix common coding errors and style issues within the client package.

client/ruleguard · high confidence

A new migration tool is introduced in \cmd/tools/migration\ to facilitate the upgrade of metadata from version 2.1 to 2.2. The tool supports etcd as the backend, handling both backup/restore operations and the actual migration of collection, alias, and index metadata. It includes a configuration file (\example.yaml\) for setting source/target versions and etcd connection details, along with backend implementations for different Milvus versions (2.10, 2.20) and legacy data structures.

cmd/tools/migration/backend · high confidence

The QueryNode v2 component now includes a new \collector\ package that provides metrics collection utilities, specifically an \averageCollector\ for tracking average values and a \counter\ for tracking integer counts. These collectors are initialized at startup and expose metrics such as insert and delete consume throughput, enabling better observability into the QueryNode's performance.

internal/querynodev2/collector · high confidence

A new migration tool is introduced to automatically upgrade existing collections and indexes to use memory-mapped file (mmap) storage, which is the default in Milvus 2.4.x. The tool iterates through all collections and indexes, updating their metadata to enable mmap support, ensuring a smooth transition from the previous version's configuration.

cmd/tools/migration/mmap · high confidence

A new utility package, internal/util/nullutil, has been added to the codebase. This package provides two key functions for users: CheckValidData, which validates that the length of a boolean 'valid\_data' slice matches the expected number of rows for a field, and GetDefaultValue, which extracts the default value for a field by its data type (including Bool, Int8/16/32/64, Float, Double, String, VarChar, Geometry, and Timestamptz). This enables consistent handling of nullable fields and default values during data import and processing.

internal/util/nullutil · high confidence

A new \pathutil\ package has been introduced to centralize the generation of local storage paths for various cache and resource types, including growing mmap, local chunks, BM25, file resources, and expression caches. This utility constructs paths based on node ID and configuration, supporting the internal storage and caching mechanisms for these components.

internal/util/pathutil · high confidence

A new \ComputePhraseMatchSlop\ function has been added to the text matching utilities, enabling the calculation of the minimum 'slop' (distance) required for a phrase match between query and data texts. This Go wrapper interfaces with the underlying C core library to perform the computation and includes error handling that maps C runtime exceptions to structured errors via the \merr\ package.

internal/util/textmatch · high confidence

Introduced a new replicate interceptor and transaction manager within the streaming node's write-ahead log (WAL) interceptors. The replicate interceptor handles message replication logic, including support for force-promoting a replica and rolling back all in-flight transactions during failover scenarios. The transaction manager tracks and manages transaction sessions, supporting commit, rollback, and cleanup operations. Tests verify the behavior of the replicate interceptor and transaction session management.

internal/streamingnode/server/wal/interceptors/txn · high confidence

Added a new shell script (rollingUpdate.sh) and its documentation (README.md) in the deployments/upgrade directory. The script enables zero-downtime rolling updates for Milvus instances installed via Helm, updating deployments one by one using kubectl patch and rollout status checks. It supports specifying the target Milvus version and image tag, and enforces that the instance is not running on RocksMQ in standalone mode.

deployments/upgrade · high confidence

The internal/proxy/search\_agg package now includes a new SearchAggregationComputer that performs hierarchical aggregation over search results, handling grouping, metric accumulation, top\_hits, and sub-aggregation. The context builder validates and constructs the aggregation context, enforcing size limits and rejecting unsupported inputs. Tests cover the computer's behavior, context building, and ordering logic.

_internal/proxy/search\agg · high confidence

A new utility file \internal/util/segmentutil/utils.go\ is introduced, providing functions to recalculate segment row counts based on bin logs, calculate deleted row counts from delta logs, and merge request costs by selecting the highest response time. This supports internal data coordination logic by ensuring segment metadata (row counts) is consistent with actual bin log data and by aggregating performance metrics.

internal/util/segmentutil · medium confidence

A new server ID picker has been introduced to the streaming service's gRPC balancer, enabling requests to be routed to specific server instances by ID. The implementation supports both round-robin distribution across ready connections and direct routing to a specific server ID when provided in the request context. This change modifies the internal load balancing strategy for streaming services, allowing for more granular control over request distribution.

internal/util/streamingutil/service/balancer/picker · high confidence

A new \shallowcopy\ package is introduced, providing \ShallowCopySearchRequest\ and \ShallowCopyRetrieveRequest\ functions. These create lightweight copies of the respective request objects where all slice and byte fields are shared with the original, while a new \Base\ header is allocated with the specified \TargetID\. This optimization avoids deep-copying large fields per shard, reducing memory overhead and improving performance for search and retrieve operations.

internal/util/shallowcopy · high confidence

A new static HTML page titled 'Milvus Expr Executor' has been added to the internal HTTP server, providing a web-based interface for executing expressions. The page includes input fields for authentication and code, a submission button, and a result display area, along with usage instructions for injected objects like param, proxy, and various coordinators.

internal/http/static · high confidence

A new client implementation for the streaming coordination service has been introduced, providing interfaces for assignment and broadcast services. This client enables internal components to interact with the streaming coordination layer, supporting features such as fetching streaming node information, managing broadcast messages, and handling replication configurations. The implementation includes gRPC-based communication with TLS support, session-based service discovery, and lazy connection management.

internal/streamingcoord/client · high confidence

A new \status\ package is introduced in \internal/util/streamingutil/status\ to standardize error handling for streaming operations. It provides a \StreamingError\ type that maps internal streaming codes to gRPC status codes, allowing clients to distinguish between recoverable and unrecoverable errors (such as transaction expiration, rate limiting, or schema mismatches). The package also includes a \clientStreamWrapper\ that automatically converts gRPC errors into the standardized streaming error format, and utility functions to check for cancellation or specific error conditions.

internal/util/streamingutil/status · high confidence

The streaming node's producer handler now exposes Prometheus metrics for tracking total and in-flight produce operations, and includes helper functions to send produce, rate-limit, create, and close responses over the gRPC stream. These changes provide observability into producer activity and improve the reliability of streaming node responses.

internal/streamingnode/server/service/handler/producer · high confidence

New interceptors are introduced for both client and server sides of the streaming service. The client interceptors handle unary and streaming errors by converting them into streaming-specific error statuses. The server interceptors wrap error handling to convert Milvus global errors (such as node mismatch or cross-cluster routing errors) into streaming errors, ensuring consistent error reporting for streaming methods.

internal/util/streamingutil/service/interceptor · high confidence

The proxy access log now supports structured logging for both gRPC and RESTful API requests. This change introduces new files in the \internal/proxy/accesslog/info\ directory that extract and format key request details—such as method name, status, trace ID, user, error codes, and timing—for both transport protocols. This enables more granular and consistent access log entries for monitoring and debugging purposes.

internal/proxy/accesslog/info · high confidence

Users can now import data using NumPy binary files (.npy). This change introduces a new import reader that parses .npy files to extract field data, supporting various data types including vectors (float, binary, int8, float16, bfloat16) and scalar types. The implementation includes validation for UTF-8 compliance in string fields and handles nullable fields by generating validity masks. This allows users to leverage NumPy's efficient binary format for bulk data ingestion.

internal/util/importutilv2/numpy · high confidence

The system now supports a variety of external rerank model providers, including Ali, Cohere, Hugging Face, Silicon Flow, TEI, vLLM, Voyage AI, and Zilliz. Each provider is implemented as a dedicated module that handles API communication, parameter parsing, and score extraction. The implementation includes a central factory (\NewModelProvider\) that routes requests to the appropriate provider based on configuration, enabling users to leverage different third-party or self-hosted reranking services for their search and retrieval tasks.

internal/util/function/models, internal/util/function/rerank · high confidence

Users can now use text embedding functions backed by Ali (Aliyun), AWS Bedrock, Cohere, and Google Gemini. This change introduces new embedding provider implementations and their corresponding unit tests, expanding the available options for generating vector embeddings from text data.

internal/util/function/embedding · high confidence

The segcore package introduces new capabilities for search result processing and lifecycle management. A new \boost\_score.go\ file adds support for computing and applying score boosts to search results, enabling weighted scoring in search queries. Additionally, a \cancellation.go\ file introduces a \CancellationGuard\ mechanism that monitors Go contexts for cancellation and propagates it to the C layer, ensuring proper cleanup and preventing use-after-free issues. These changes enhance search functionality and improve reliability during asynchronous operations.

internal/util/segcore · high confidence

Added new examples in the telemetry\_demo directory: a basic end-to-end telemetry demo, a multi-database demo with database-targeted command push support, and a WebUI demo with custom command handlers. Also added a telemetry E2E test that validates the full client-side metrics push, heartbeat cycles, and server-side command reception. These examples demonstrate the new client-side telemetry capabilities, including heartbeat intervals, sampling rates, and command handling.

examples · high confidence

The tsoutil package now includes new functions, NewTSOTiKVBase and NewTSOKVBase, which initialize transactional key-value stores for timestamp allocation. This adds support for TiKV as a backend alongside the existing etcd implementation, enabling timestamp service operations on TiKV storage.

internal/util/tsoutil · high confidence

A new \tombstone\ package is introduced in the root coordinator to manage the lifecycle of deleted resources. The \TombstoneSweeper\ runs a background goroutine that periodically checks registered tombstones and automatically removes them once they are confirmed safe to delete, reducing manual cleanup overhead.

internal/rootcoord/tombstone · high confidence

Added build-support scripts and configuration files to enforce C++ code style and linting. This includes a cpplint.py checker, a run\_cpplint.py runner, a run\_clang\_format.py script for automatic formatting, and a run\_clang\_tidy.py script for deeper static analysis. The change also introduces helper utilities (lintutils.py), exclusion lists (ignore\_checks.txt, lint\_exclusions.txt), and license header scripts (add\_cmake\_license.sh, add\_cpp\_license.sh) to standardize code quality checks across the C++ codebase.

internal/core/build-support · high confidence

Added the \internal/util/analyzecgowrapper\ package, which wraps the C++ \Analyze\ and \DeleteAnalyze\ C API calls. This new utility provides a Go interface (\CodecAnalyze\) to manage the lifecycle of analysis tasks, including marshaling \AnalyzeInfo\ into C structures, handling encryption context, and retrieving analysis results (centroids and offset mappings) from the C++ core. The addition includes a helper function \HandleCStatus\ to map C++ error codes to Go errors, and corresponding unit tests to verify the wrapper's behavior with invalid inputs and plugin contexts.

internal/util/analyzecgowrapper · high confidence

A new Dockerfile and a Python helper script (apply-config-from-env.py) were added to the build/docker/pulsar directory. The Dockerfile builds an image based on adoptopenjdk:11-jdk-hotspot, installs Apache Pulsar version 2.8.2, and configures the container to run Pulsar in standalone mode. The Python script enables runtime configuration of Pulsar by applying environment variables prefixed with PULSAR\PREFIX\ to the configuration files, allowing users to override settings via environment variables at container startup.

build/docker/pulsar · high confidence

The build/docker/krte directory now contains a Dockerfile and a wrapper.sh script, establishing the foundation for the Kubernetes Runtime Environment (krte) container. The Dockerfile installs Go, kubectl, Helm, Docker, Docker Compose, and KinD (v0.11.1), while the wrapper script manages the test execution environment, including Docker-in-Docker setup, IPv6 configuration, and cleanup procedures.

build/docker/krte · high confidence

A new VChannelTempStorage component has been introduced in the streaming node's write-ahead log (WAL) layer to temporarily store vchannel-to-pchannel mappings. This storage acts as a compatibility layer, allowing the system to resolve vchannel information via the new MixCoord interface while maintaining support for older message stream formats. The implementation includes a retry mechanism for collection description calls and handles edge cases such as missing collections or context timeouts.

internal/streamingnode/server/wal/vchantempstore · high confidence

The \internal/streamingnode/server/wal/interceptors/replicate/replicates\ package introduces a new \ReplicatesManager\ that manages the state and behavior of the Write-Ahead Log (WAL) in replicating mode. This implementation handles the transition between primary and secondary roles, processes replicate messages, and manages checkpoints for data salvage during force failover. The addition includes the core manager logic, secondary state management, transaction handling, and comprehensive unit tests to verify the new replication behavior.

internal/streamingnode/server/wal/interceptors/replicate/replicates · high confidence

The Tantivy binding now includes a Rust benchmark suite (analyzer\_bench.rs) to measure the performance of various tokenizers, including Jieba, Lindera, and language identifier tokenizers. Additionally, a Cargo build script (build.rs) and configuration (cbindgen.toml) have been added to automatically generate C++ header files and compile protocol buffers, while .gitignore rules for Rust build artifacts are established.

internal/core/thirdparty/tantivy/tantivy-binding · high confidence

Added new utility files in internal/util/hookutil to support cipher operations and hook management. The changes introduce a cipher implementation (cipher.go) that handles encryption key management, including functions to create, remove, and backup encryption zones (EZ), as well as parsing and tidying database properties related to encryption. Additionally, the hook.go file provides a singleton pattern for loading and managing hook plugins, while plugin.go adds a mutex-protected LoadPlugin function to safely handle concurrent plugin loading. The default.go file introduces default hook and extension implementations, and constant.go defines operation types and keys. These changes enable the system to support dynamic hook loading and encryption key management.

internal/util/hookutil · high confidence

A new file cmd/embedded/embedded.go was added, defining a Go main package that sets the MILVUSCONF environment variable and invokes the embedded run command. This change introduces a new entry point for running Milvus in embedded mode, likely to support Cgo-based integration or testing scenarios where the application is embedded within a larger system.

cmd/embedded · high confidence

A new \expr\_checker.go\ module was introduced in \internal/util/exprutil\ to analyze expression trees for partition key and primary key predicates. This includes logic to parse and identify prunable expressions for partition keys (supporting AND/OR/NOT logical operators and term/unary range expressions) and to detect optimizable primary key predicates (checking for TermExpr, UnaryRangeExpr, and BinaryRangeExpr on primary keys). The accompanying test file \expr\_checker\_test.go\ validates these parsing and optimization detection capabilities.

internal/util/exprutil · high confidence

Generated mock implementations for gRPC's ClientConn and ClientStream interfaces have been added to the internal mocks directory. These auto-generated files, created by mockery v2.53.3, provide test doubles for the resolver and streaming components, enabling unit tests to simulate gRPC client behavior without a live connection.

internal/mocks/google.golang.org · high confidence

The streamrpc package now includes an in-memory streamer implementation (InMemoryStreamer) and a concurrent query stream server (ConcurrentQueryStreamServer) that caches and merges RetrieveResults. This allows the system to buffer and combine query results locally, which can help avoid gRPC response size limits and reduce network overhead during standalone or local query operations. A corresponding mock for the gRPC client stream is also added to support testing of these new components.

internal/util/streamrpc · high confidence

The distributed utilities now include helper functions to enable internal TLS for gRPC servers, allowing secure internal communication when configured. Additionally, the new \GracefulStopGRPCServer\ function ensures the gRPC server shuts down gracefully, with a configurable timeout before forcing a stop. A corresponding test was added to verify the graceful shutdown behavior.

internal/distributed/utils · high confidence

A new migration tool has been introduced in the \cmd/tools/migration/meta\ directory to handle the upgrade of metadata from version 2.1 to 2.2. This includes the implementation of conversion logic for collections, aliases, and index information, ensuring that data structures are correctly transformed and saved during the upgrade process.

cmd/tools/migration/meta · high confidence

A new command-line tool has been introduced at cmd/tools/migration/mmap/tool to facilitate the migration of data using memory-mapped files. The tool initializes configuration and TSO allocators, supports both Etcd and TiKV as metadata storage backends, and executes the migration process.

cmd/tools/migration/mmap/tool · high confidence

New mock implementations for gRPC clients (GrpcDataNodeClient, GrpcQueryCoordClient, GrpcQueryNodeClient, GrpcRootCoordClient) and a generic GRPCClientBase template are added to the internal/util/mock package. These mocks provide test-friendly, in-memory or error-throwing implementations of the core Milvus coordination and data node interfaces, enabling unit tests to interact with these components without requiring live gRPC connections.

internal/util/mock · high confidence

New auto-generated mock files have been added for the streaming node client handler interfaces, including MockWatcher, MockConsumer, and MockProducer. These mocks, generated by mockery v2.53.3, provide test doubles for the Watcher, Consumer, and Producer types, enabling unit testing of components that depend on these interfaces.

internal/mocks/streamingnode/client/handler · high confidence

New auto-generated mock files have been added for the streaming service's Discoverer, LazyGRPC Service, and Resolver interfaces. These mocks, generated by mockery v2.53.3, provide test doubles for the discoverer, gRPC service, and resolver components within the streaming utility package, enabling easier unit testing of dependent code.

internal/mocks/util/streamingutil · high confidence

A new redo interceptor has been introduced in the Write-Ahead Log (WAL) pipeline to handle append operations. This component waits for growing segments to become ready before proceeding, and will retry the append if a 'redo' condition is detected. The implementation includes a builder pattern for instantiation and a corresponding test suite that validates behavior for unrecoverable errors when collections or partitions are not found.

internal/streamingnode/server/wal/interceptors/redo · high confidence

A new ReplicateStreamServer implementation has been added to handle incoming replicate messages via a gRPC stream. The server receives replicate messages, appends them to the Write-Ahead Log (WAL), and sends back confirmation responses. The implementation includes trace context propagation for observability and handles ignored operations gracefully by still sending confirmation responses.

internal/proxy/replicate · high confidence

The streaming service client now includes a new \ResumableProducer\ that automatically resumes from stream breaks and node rebalances. This change introduces adaptive rate limiting for Write-Ahead Log (WAL) append operations, which can trigger backoff and retry behavior when the service is rate-limited. Additionally, Prometheus metrics have been added to track producer availability, produce latency, and rate-limit states, providing visibility into the streaming client's health and performance.

internal/distributed/streaming/internal/producer · high confidence

Introduced a new metrics utility in the query node to track segment-level access patterns. The change adds new Prometheus metrics for cache loads, cache evictions, query segment accesses, and search segment accesses, enabling better observability into segment performance and cache efficiency.

internal/querynodev2/segments/metricsutil · high confidence

The streaming node server service now includes new handler implementations for the streaming node's RPC interface, specifically handling produce and consume streams, as well as retrieving replicate and salvage checkpoints. Additionally, a new release manual flush preparer has been introduced to manage the handoff process for growing sources, ensuring that manual flushes are correctly prepared and committed based on buffer manager status and WAL availability.

internal/streamingnode/server/service · high confidence

The internal/util/streamingutil package was introduced to manage the MILVUS\_STREAMING\_SERVICE\_ENABLED environment variable. This includes functions to check if the streaming service is enabled, set it, and enforce it is enabled via a panic. A test helper was also added to unset the variable for testing purposes.

internal/util/streamingutil · high confidence

The configuration file (milvus.yaml) is now generated directly from the paramtable code, ensuring the file always matches the current default values and structure. This change introduces a new CLI tool (cmd/tools/config) that can generate the YAML configuration file and a CSV of all configuration keys, removing the need for manual edits to the YAML file.

cmd/tools/config · high confidence

The rootcoord now supports client-side telemetry, enabling servers to receive heartbeats and push commands to clients. This change introduces a new telemetry package in internal/rootcoord/telemetry that includes a CommandRouter to dispatch commands to specific handlers, a CommandStore to persist configurations in etcd and manage in-memory commands, and a TelemetryManager to track client metrics and handle heartbeats. The system supports three command types: show\_errors, collection\_metrics, and push\_config, each with specific payload structures and validation rules. The implementation includes comprehensive test coverage for all components.

internal/rootcoord/telemetry · high confidence

The DataNode's gRPC server is now initialized and managed through a dedicated \Server\ struct in \internal/distributed/datanode/service.go\. This change introduces explicit \Run()\ and \Stop()\ lifecycle methods that handle gRPC server startup, listener creation, and graceful shutdown. The server configuration includes gRPC keepalive policies, max message size limits, and interceptors for logging, cluster validation, and server ID validation. A corresponding test file (\service\_test.go\) verifies the server's initialization, state management, and error handling.

internal/distributed/datanode · high confidence

The DataNode component is now fully implemented in the internal/datanode package, providing the core logic for data persistence, index building, and import tasks. This includes the main DataNode struct, its initialization and lifecycle management (start/stop), and the implementation of gRPC services for index building (CreateJob, QueryJobs, DropJobs) and system metrics reporting. The change also introduces factory patterns for storage and task scheduling, along with corresponding unit tests and configuration files (.mockery.yaml, README.md).

internal/datanode · high confidence

The internal/types package now provides the foundational interface definitions for the system's core components. This includes the Component lifecycle interface, specific server interfaces for DataNode, DataCoord, RootCoord, and Proxy, as well as their corresponding client interfaces. The file also introduces a Limiter interface for request rate limiting and type aliases for metrics topology, establishing the contract layer for inter-service communication and internal service management.

internal/types · high confidence

Users can now load collections and partitions with a specified load priority, and the system supports dynamic schema updates for collection creation. Index building is optimized with a concurrency pool for the analyzer and a configurable thread pool size for the index node. The default value for the \segment.maxSize\ parameter is set to 1024M, and the default shard number is changed to 1. Additionally, the system now supports a force merge operation and allows users to set up the root user's password.

pkg · high confidence

Added a new C++ wrapper (milvus\_plan\_parser.cpp/h) and Go bindings (wrapper.go) in the internal/parser/planparserv2/cwrapper directory. This exposes the Go-based plan parser to C++ consumers, allowing C++ code to register schemas, parse filter expressions, and parse search plans via a thread-safe, lock-free interface that returns serialized PlanNode protobufs.

internal/parser/planparserv2/cwrapper · high confidence

The Tantivy search engine is now exposed to the C++ layer via a new header file, enabling direct integration with the search engine. The bindings support multiple data types including Text, Keyword, I64, F64, Bool, and JSON, allowing the system to perform term queries, range queries, and manage index writers and readers. This change provides the underlying infrastructure for features like JSON flat indexing, n-gram tokenization, and optimized inverted index operations.

internal/core/thirdparty/tantivy/tantivy-binding/include · high confidence

The client directory now contains the Go SDK package (milvusclient) along with supporting files: a golangci-lint configuration enforcing Go 1.25 and specific linter rules, a Makefile for linting and generating mock servers, an OWNERS file, and a README detailing installation and usage of the Go MilvusClient.

client · high confidence

The Go SDK client code has been refactored into a standalone v3 module, moving internal utilities (error handling, retry logic, type helpers, and singleflight) into the client package and decoupling the client from the server-side milvus/pkg module. This structural change improves the SDK's dependency graph and enables new capabilities such as bloom filter expressions, alias management, and admin/RBAC backup and restore operations.

client/milvusclient · high confidence

The Go SDK now supports row-based insert operations. This change introduces new files in the \client/row\ package: \cache.go\ adds a receiver parse result cache to improve performance; \data.go\ implements the \AnyToColumns\ function to convert row data into column-based data for insertion; \schema.go\ provides \ParseSchema\ to automatically infer the Milvus schema from Go structs using struct tags; and \type.go\ defines the \ReceiverCandidate\ struct used for unmarshalling results. These changes enable users to insert data using Go structs, with automatic schema detection and optimized parsing.

client/row · high confidence

The internal HTTP package has been restructured to support a more modular and secure HTTP server. New files introduce a centralized routing and constants definition (router.go, constant.go) that define paths for health checks, log level updates, and various management APIs. A new RBAC (Role-Based Access Control) module (rbac.go) enforces authentication and authorization for the /expr endpoint, supporting both root-only and RBAC-based access modes. The HTTP server implementation (server.go) is updated to use these new components, including embedding static files for the WebUI and registering handlers for component stop/ready checks. Tests (rbac\_test.go, server\_test.go) are added to verify the new authentication and server behavior.

internal/http · high confidence

The compaction module now supports TEXT columns with LOB storage and enforces Storage V3 for collections containing TEXT fields to prevent data loss. It also introduces an EntityFilter that uses a commit timestamp for import/CDC segments, ensuring that row timestamps are not prematurely expired and that pre-commit deletes are correctly ignored. Additionally, the system now calculates a hole ratio for TEXT LOB files to decide whether to reuse or rewrite them, and loads BM25 and bloom filter statistics from resolved file paths.

internal/compaction · high confidence

The \internal/util/bloomfilter\ package now includes a new \blockedBloomFilter\ implementation backed by the \blobloom\ and \xxh3\ libraries, alongside the existing \basicBloomFilter\. This adds support for a blocked (or segmented) bloom filter variant, providing an alternative to the standard bloom filter for membership testing. The change introduces the \BlockedBF\ type and associated logic, enabling users to utilize this new filter type for potentially improved performance or memory characteristics.

internal/util/bloomfilter · high confidence

The \internal/util/analyzer/canalyzer\ package now provides the core Go wrappers for the C/C++ tokenizer engine. This adds support for multiple tokenization strategies, including a standard tokenizer, a gRPC-based remote tokenizer, and a Lindera-based Japanese/Chinese tokenizer. Users can now configure and validate analyzers via JSON parameters, with runtime options and global resource information (such as dictionary paths and remote endpoints) being synchronized to the underlying C library.

internal/util/analyzer/canalyzer · high confidence

The streaming coordination module now uses a new KV-based catalog implementation that persists control channels, broadcast tasks, and versioning information in the metastore. This change introduces new metadata keys for streaming coordination, including prefixes for channels, broadcast tasks, and replication configuration, enabling more reliable and consistent state management for streaming services.

internal/metastore/kv/streamingcoord · high confidence

Added a new Parquet import reader implementation in the \internal/util/importutilv2/parquet\ package. This change enables the system to read and import data from Parquet files, supporting various data types including vectors, scalars, and nested structures. The implementation includes field readers, list-like array handling, and struct field readers to parse Parquet columns into the internal data structures. Tests are included to verify the correct parsing of different data types and error handling for invalid data.

internal/util/importutilv2/parquet · high confidence

Added new files in the \internal/querycoordv2/session\ directory to manage QueryNode sessions and cluster state. This includes \cluster.go\ which defines the \QueryCluster\ interface and implementation for communicating with QueryNodes, \node\_manager.go\ which handles node lifecycle and resource exhaustion tracking, \stats.go\ for node statistics, and corresponding test files. These changes provide the foundational session management layer for QueryCoord v2.

internal/querycoordv2/session · high confidence

The MixCoord now exposes a new RESTful endpoint to check if all loaded collections meet cluster-level replica configuration requirements, including replica count, resource group distribution, and serviceability. This endpoint returns a compliance state (Ready or NotReady) to help operators verify cluster health before proceeding with operations. Additionally, a new FileResourceObserver component has been added to the MixCoord to manage file resource synchronization between the coordinator and the various node types (QueryNode, DataNode, StreamingNode). This observer tracks the state of file resources and ensures they are properly synced. The implementation includes the core observer logic, the RESTful route for replica compliance, and associated unit tests.

internal/coordinator · high confidence

Added the \internal/distributed/streamingnode\ package, which implements the gRPC server for the StreamingNode component. This includes the server lifecycle management (Prepare, Run, Stop) and health check logic, establishing the distributed interface for streaming node operations.

internal/distributed/streamingnode · high confidence

A new TimeTickSyncInspector component has been added to the WAL interceptor layer to manage the synchronization of time ticks across pchannels. This includes a background goroutine that periodically triggers sync operations and handles immediate notifications, allowing the system to more effectively manage write-ahead buffers and ensure consistent state during recovery and normal operation.

internal/streamingnode/server/wal/interceptors/timetick/inspector · high confidence

Added a new WAL (Write-Ahead Log) manager in the streaming node to handle the lifecycle of WAL instances per channel. The manager coordinates opening, removing, and retrieving WAL instances, managing their state transitions (available/unavailable) and terms to ensure consistency. It includes a background task to serialize operations and handle state changes, with comprehensive unit tests covering state transitions, lifecycle management, and error handling.

internal/streamingnode/server/walmanager · high confidence

Added a new Write-Ahead Buffer (WAB) implementation for the streaming service, including the core buffer logic, a pending queue for managing message ordering and eviction, and a reader interface. This change enables the streaming service to buffer and manage messages with time ticks, supporting catch-up and tailing scan operations while ensuring the last persisted message is retained to prevent data loss during catch-up scenarios.

internal/streamingnode/server/wal/interceptors/wab · high confidence

The search utility module now includes a new concurrent-safe scheduler implementation located in \internal/util/searchutil/scheduler\. This change introduces a modular scheduler architecture that supports multiple scheduling policies, including First-In-First-Out (FIFO) and user-based task polling. The new scheduler manages task queues, handles task merging, and provides interfaces for adding, clearing, and executing search tasks, along with comprehensive unit tests for the new components.

internal/util/searchutil/scheduler · high confidence

Added a new \internal/util/cgo\ package that provides an asynchronous execution model for CGO calls via a \Future\ interface. This includes \futures.go\ for managing async operations and their states, \manager\_active.go\ for tracking and canceling active futures across shards, \executor.go\ for initializing and dynamically resizing thread pools for search and load operations, and \pool.go\ to limit concurrent CGO calls. The \errors.go\ file adds a helper to convert CGO status into Go errors, and \state.go\ implements a thread-safe state machine for future lifecycle management.

internal/util/cgo · high confidence

The plan parser now supports a new \bloom\_match\ filter expression that accepts a pre-built client-side SBBF (Sparse Bloom Filter) blob. This allows users to perform approximate membership tests on scalar fields (INT64, VARCHAR, and JSON paths) using a compact binary filter, with the proxy validating the MBF1 envelope and embedding the blob directly into the query plan without rebuilding. The implementation includes validation of field types, nested JSON paths, and size limits, along with helper functions to check predicate and plan node identity for caching and optimization.

internal/parser/planparserv2 · high confidence

A new \params\ package has been added to \internal/querycoordv2/params\, establishing a centralized location for component parameters and configuration helpers. This includes a \Params\ singleton that wraps \paramtable.ComponentParam\, along with test-only utilities for generating random etcd configurations, meta root paths, and ID allocators. An OWNERS file is also added to define code review responsibilities for this new directory.

internal/querycoordv2/params · high confidence

The streaming coordination layer now includes a new \ChannelManager\ that tracks physical channels (PChannels) and their assignments to streaming nodes. This manager handles recovery of channel metadata, maintains a global singleton for cluster topology queries, and exposes metrics for channel status and assignment versions. It also manages the state of each PChannel, including tracking assignment history and availability for replication, allowing the system to balance and monitor streaming resources more effectively.

internal/streamingcoord/server/balancer/channel · high confidence

Added new utility components in the streaming util package to manage channel discovery and write-ahead log (WAL) selection. The ConfigChannelProvider now watches Milvus configuration for new DML channel names and can also discover pchannels recovered from collection metadata, ensuring the system detects channel changes dynamically. Additionally, the WAL selector logic was implemented to automatically select the appropriate WAL implementation (Rocksmq, Pulsar, Kafka, or Woodpecker) based on runtime configuration and environment constraints, such as preventing Woodpecker with local storage in cluster mode. These utilities provide the foundational logic for channel and WAL management in the streaming subsystem.

internal/util/streamingutil/util · high confidence

The recovery module now exposes three new configuration options—\WALRecoveryPersistInterval\, \WALRecoveryMaxDirtyMessage\, and \WALRecoveryGracefulCloseTimeout\—allowing users to tune how frequently the recovery state is persisted, the threshold for dirty messages before a save is triggered, and the timeout for graceful shutdown. Additionally, Prometheus metrics are now exposed to monitor the recovery storage's state, time ticks, and inconsistency events, providing better observability into the recovery process.

internal/streamingnode/server/wal/recovery · high confidence

A new \DefaultFactory\ in \internal/util/dependency\ centralizes the creation of message queue streams and chunk managers. It supports selecting between Rocksmq, Pulsar, Kafka, and Woodpecker message queues, with a fallback logic that auto-selects the first available MQ type in standalone mode (preferring Rocksmq, then Pulsar, then Kafka, then Woodpecker) or cluster mode (preferring Pulsar, then Kafka). The factory also provides a \HealthCheck\ function to verify the status of the active MQ. Tests verify the MQ selection logic and factory instantiation.

internal/util/dependency · high confidence

The QueryCoord's distribution management logic has been refactored into a new \dist\ package. This introduces a \Controller\ that manages per-node \distHandler\ instances, each responsible for pulling and handling data distribution updates from QueryNodes. The \distHandler\ implements background loops for pulling distribution data and dispatching tasks, while the \Controller\ coordinates these handlers and provides a \SyncAll\ method to aggregate distribution states. This change also includes the corresponding unit tests for the controller and handler, as well as a generated mock for the \Controller\ interface.

internal/querycoordv2/dist · high confidence

Added a new file resource manager in internal/util/fileresource that supports syncing and downloading remote files to local storage. The manager exposes a global singleton, a Sync method to update local state, and a Download method to fetch files. It supports multiple modes (Sync, Ref, Close) and provides a listener mechanism to notify registered components about file resource changes.

internal/util/fileresource · high confidence

Added a new function runner framework in internal/util/function that manages the lifecycle of function runners (BM25, MinHash, Text Embedding) across collection schema versions. The implementation includes a FunctionRunnerManager that tracks schema versions and initializes runners asynchronously, a local store for handling old insert messages, and specific runners for BM25 (including multi-analyzer support) and MinHash. The BM25 runner supports configurable concurrency via a pool, validates UTF-8 input, and handles both single and multi-analyzer configurations. Tests verify signature stability, concurrency limits, and error handling.

internal/util/function · high confidence

The Go SDK client now uses a generic \genericColumnBase\ to simplify column implementations and support nullable fields across scalar types. New column types have been added for array fields (bool, int8/16/32/64, float32/64), dynamic JSON fields, geometry (WKT), and JSON data. These changes provide a more consistent API for handling structured data, null values, and complex types in the Go client.

client/column · high confidence

The datacoord now includes a new global task scheduler (global\_scheduler.go) that manages task execution using a priority queue (priority\_queue.go) to ensure tasks are processed in a consistent order. The scheduler implements an exponential backoff mechanism for failed tasks to prevent dispatch storms, and selects the least-loaded worker node for task assignment. This change introduces new interfaces and mock implementations (mock\_global\_scheduler.go, mock\_task.go) to support testing of the scheduler's behavior.

internal/datacoord/task · high confidence

The \internal/util/importutilv2\ package has been added to handle data import logic, including support for JSON, JSONL/NDJSON, Numpy, Parquet, and CSV file formats. The new codebase provides a unified \Reader\ interface and factory (\NewReader\) to abstract the underlying file parsing, while \option.go\ introduces configuration options such as \timeout\, \skip\_disk\_quota\_check\, \backup\ mode, and CSV-specific settings like \sep\ and \nullkey\. A corresponding mock (\mock\_reader.go\) and unit tests (\option\_test.go\, \reader\_test.go\) are included to support testing of import-related functionality.

internal/util/importutilv2 · high confidence

Added a new in-memory key-value store implementation (MemoryKV) in the internal/kv/mem package. This provides a lightweight, thread-safe, B-tree backed storage solution for testing and scenarios where persistent storage is not required. The implementation supports standard KV operations including Save, Load, Remove, and batch operations like MultiSave and MultiLoad, along with range and prefix-based queries. This change introduces a new internal component that can be used as a drop-in replacement for persistent KV stores in specific contexts.

internal/kv/mem · high confidence

The \internal/util/initcore\ package is introduced to consolidate the initialization of the C++ segcore, trace configuration, and storage systems. This change introduces Go wrappers for C++ initialization functions, including \InitTraceConfig\ and \ResetTraceConfig\ which now support OpenTelemetry (OTLP) headers and secure connections, and \InitQueryNode\ which configures thread pools, SIMD types, and various segcore parameters. The package also adds hot-reload support for Arrow IO thread pool capacity and Arrow reader configuration, allowing runtime updates to these settings without restarting the service.

internal/util/initcore · high confidence

Added a new registry in the streaming node client handler to manage local Write-Ahead Log (WAL) instances. This includes a \WALManager\ interface and registration functions (\RegisterLocalWALManager\, \GetLocalAvailableWAL\) that allow the streaming node to register and retrieve local WAL components. The implementation wraps local WALs with a \localWAL\ type to distinguish them from remote ones, and provides a test utility to reset the registry state. This change supports the broader effort to reduce downtime during upgrades and fix local WAL behavior differences.

internal/streamingnode/client/handler/registry · medium confidence

A new meta migration tool has been added to the CLI, providing commands for backup, run, and rollback operations. The tool supports a configuration file and handles session management, including exiting after session expiration. The implementation includes specific logic for each operation: 'run' validates and migrates data, 'backup' creates a backup, and 'rollback' reverts changes. This adds a new capability for managing migration states.

cmd/tools/migration/command · high confidence

Developers can now use the new \mgit.py\ script to automate the commit, PR, and cherry-pick workflows. The tool generates AI-powered commit messages, automatically handles DCO signing, manages branch creation, and enforces design document validation for feature PRs. A test suite has been added to verify design document reference validation.

tools · high confidence

Added new compaction types—bumpSchemaVersion and clustering—to the datanode compactor, enabling schema version updates and clustering operations on segments. The implementation includes the core task structures (bumpSchemaVersionCompactionTask, clusteringCompactionTask) and their corresponding test suites, supporting both StorageV2 and StorageV3 backends. This change allows the system to handle schema evolution and data clustering as distinct, configurable compaction workflows.

internal/datanode/compactor · high confidence

The Milvus CLI now includes a new 'mck' (data consistency check) command for validating and cleaning up inconsistent data in etcd and MinIO, as well as 'run' and 'stop' commands to manage the Milvus server lifecycle, including support for mixed-role deployments and embedded modes. The change also adds build-tag-gated OpenSSL FIPS support and prints hardware memory information at startup.

cmd/milvus · high confidence

Added a new CSV import utility in \internal/util/importutilv2/csv\ that provides a \reader\ and \row\_parser\ to parse CSV files into insert data structures. This implementation supports various data types including scalars, vectors, and struct arrays, handling nullable fields, auto-generated primary keys, and function outputs. The change includes comprehensive unit tests for the new CSV parsing logic.

internal/util/importutilv2/csv · high confidence

Added a new \Manager\ interface and \grpcWorkerManager\ implementation in \internal/querynodev2/cluster\ to manage and pool gRPC clients for QueryNodes. This introduces a worker management layer that caches and reuses worker connections per node ID, supporting connection pooling and singleflight deduplication. The change includes the \Manager\ and \Worker\ interfaces, the \remoteWorker\ implementation, and corresponding unit tests and mocks, enabling more robust and efficient communication with QueryNodes in the v2 architecture.

internal/querynodev2/cluster · high confidence

The \scripts/\ directory now includes a comprehensive set of new shell scripts to streamline the build, test, and development workflows for the C++ core. \core\_build.sh\ and \3rdparty\_build.sh\ provide configurable build options for the core library and its dependencies (via Conan), while \build\_plan\_parser.sh\ handles the Go-to-C++ shared library bridge. Additional scripts (\check\_cpp\_fmt.sh\, \check\_proto\_product.sh\, \generate\_proto.sh\, \gofmt.sh\, \download\_milvus\_proto.sh\, \collect\_arrow\_dep.sh\) automate code formatting, proto generation, and dependency collection. The \devcontainer.sh\ script simplifies setting up the Docker-based development environment, and \docker\_image\_find\_tag.sh\ assists in finding the correct Docker image tags. These changes centralize and standardize the build and development processes.

scripts · high confidence

A new migration tooling structure has been added to the codebase, introducing a \migrator\ interface and a \Runner\ that orchestrates the migration process. This includes a specific \migrator210To220\ implementation that handles the transition from version 2.1.0 to 2.2.0. The \Runner\ manages the migration lifecycle, including session validation, backup, and rollback capabilities, while also monitoring for active sessions to ensure safe execution.

cmd/tools/migration/migration · high confidence

The query node's data processing pipeline now includes dedicated nodes for handling insert and delete messages. The new \filterNode\ validates incoming messages, checking alignment, emptiness, and target collection, while the \insertNode\ processes insert payloads and the \deleteNode\ processes delete payloads. A \Manager\ is introduced to oversee the lifecycle of these pipelines per channel. These changes provide a more structured and testable approach to processing DML messages within the query node.

internal/querynodev2/pipeline · high confidence

Added new internal storage modules: \arrow\_util.go\ and \arrow\_util\_test.go\ provide utility functions for handling Apache Arrow arrays and default values; \binlog\_reader.go\ and \binlog\_record\_writer.go\ implement new binlog reading and writing logic, including support for encrypted binlogs and packed manifest records; \azure\_object\_storage.go\ and \azure\_object\_storage\_test.go\ add a new Azure Blob Storage client implementation for the chunk manager. These changes expand the storage layer's capabilities for handling diverse data types and cloud storage providers.

internal/storage · high confidence

Added a new producer client implementation for the streaming node client, including the gRPC client wrapper, the producer interface, and the concrete implementation with rate limiting and trace context propagation. The change also includes comprehensive unit tests for the producer's behavior, including rate limit state updates and trace context handling.

internal/streamingnode/client/handler/producer · high confidence

Added new internal components in the sync manager to handle growing segment flushes and task dispatching. This includes a new \GrowingSource\ interface and implementation to manage the lifecycle and data of growing segments, a \keyLockDispatcher\ that enforces per-key serial execution with cross-key concurrency and backpressure, and a \MetaWriter\ to persist segment sync metadata and binlog paths. The addition is accompanied by comprehensive unit tests for the dispatcher's concurrency and backpressure logic, as well as tests for the meta writer's behavior.

internal/flushcommon/syncmgr · high confidence

The internal/querynodev2/pkoracle package has been introduced to manage primary key to segment mapping. This includes a new BloomFilterSet implementation that uses bloom filters to check for primary key existence in segment statistics, as well as an ExternalSegmentCandidate for handling external collections using virtual primary keys. The package also provides a PkOracle interface and implementation for registering, querying, and removing segment candidates, along with candidate filtering capabilities. Tests have been added to verify the functionality of these components.

internal/querynodev2/pkoracle · high confidence

The streaming consumer implementation has been refactored to support automatic resumption from stream breaks and node rebalances. The new \ResumableConsumer\ interface and \resumableConsumerImpl\ implementation manage a background loop that detects failures, retries with exponential backoff, and resumes consumption from the last confirmed message or time tick. This ensures that consumers do not get stuck or lose messages when the underlying stream is interrupted. The change includes metrics tracking for consumer availability and message bytes, as well as a message handler that tracks confirmed message IDs and time ticks to facilitate accurate resume points.

internal/distributed/streaming/internal/consumer · high confidence

Added a new expression rewriter module at \internal/parser/planparserv2/rewriter\ that performs rule-based logical rewrites on parsed \planpb.Expr\ trees. This includes normalizing \IN\/\NOT IN\ expressions, merging \TEXT\_MATCH\ conditions, simplifying range predicates, and optimizing boolean and array comparisons. The rewriter respects the \common.enabledOptimizeExpr\ configuration to enable or disable these optimizations, ensuring that template values are handled correctly and that nullable fields preserve unknown semantics where appropriate.

internal/parser/planparserv2/rewriter · high confidence

The DataNode now uses a new \BinlogIO\ interface and implementation that performs downloads and uploads via a shared, auto-scaling goroutine pool. The pool size is now determined by the \dataNode.dataSync.ioConcurrency\ configuration, scaling with the node's CPU count (defaulting to max(16, CPU\*2)) when unset, and can be explicitly configured. This replaces the previous hard-coded concurrency limits, allowing the system to better utilize available resources for object storage IO operations.

internal/flushcommon/io · high confidence

Added a new semantic highlight implementation in the highlight utility package. This includes a \SemanticHighlight\ struct that manages query and input field parameters, resolves schema and dynamic fields, and delegates the actual highlighting and scoring to a \zillizHighlightProvider\ which calls the Zilliz client's Highlight API. The change also adds corresponding unit tests for both the semantic highlight logic and the Zilliz provider.

internal/util/function/highlight · high confidence

The internal/util/flowgraph package has been refactored to replace the previous graph-based processing model with a simplified pipeline architecture. This change introduces a new \TimeTickedFlowGraph\ that enforces a linear, single-output node structure, which reduces code complexity and improves recovery speed. The update also includes a new \InputNode\ implementation with configurable close behavior (graceful vs. immediate) and enhanced metrics tracking for data nodes.

internal/util/flowgraph · high confidence

A new singleton pattern for the broadcaster is introduced in the broadcast package, exposing \StartBroadcastWithResourceKeys\ and \StartBroadcastWithSecondaryClusterResourceKey\ as the primary entry points for initiating broadcasts. These functions coordinate with the balancer to ensure the WAL-based DDL framework is ready before proceeding, and enforce primary/secondary cluster constraints. The change also includes a test utility to reset the singleton state, facilitating isolated testing of the broadcaster lifecycle.

internal/streamingcoord/server/broadcaster/broadcast · high confidence

The streaming coordination server now includes a dedicated balancer component that manages load distribution across streaming nodes. This new module defines a \Balancer\ interface and its implementation, handling the assignment of virtual channels to nodes, tracking node availability, and managing resource groups. It supports dynamic policy updates, node freezing/unfreezing, and ensures consistency during primary resource group changes. The implementation includes a policy registry for balancing strategies and integrates with the WAL (Write-Ahead Log) system to track channel states and node metrics.

internal/streamingcoord/server/balancer · high confidence

The streaming coordination client now includes an assignment service implementation that manages streaming node discovery, WAL balance policy updates, and replicate configuration management. Users can retrieve the latest streaming version, discover node assignments, update WAL balance policies, and manage replication configurations with options for fresh reads to ensure strong consistency. The implementation includes a watcher for tracking assignment changes and a discoverer client for communicating with the streaming coordination service.

internal/streamingcoord/client/assignment · high confidence

Added a new singleton resource manager for the streaming coordination server, encapsulating dependencies such as the etcd client, mix-coordinator client, ID allocator, and streaming node manager. This change establishes the foundational resource layer for streaming node interactions, with corresponding tests verifying initialization and release behavior.

internal/streamingcoord/server/resource · high confidence

Added a new streaming node client handler implementation that manages pchannel-level producer and consumer creation. The handler includes a channel assignment watcher that monitors and tracks channel-to-node assignments, enabling the client to route requests to the correct streaming node. This new handler client provides methods for creating producers and consumers, retrieving WAL checkpoints, and managing local/remote WAL operations.

internal/streamingnode/client/handler · high confidence

Added a new \consumer\ package in \internal/streamingnode/client/handler/consumer\ that implements the \Consumer\ interface for managing gRPC streams. This includes the \consumerImpl\ struct and \CreateConsumer\ factory, which handle receiving messages from the streaming node, processing them via a message handler, and managing stream lifecycle (Close, Error, Done). The implementation supports transactional messages, trace context propagation, and configurable delivery policies and filters.

internal/streamingnode/client/handler/consumer · high confidence

Added new files in the streaming node handler consumer package: a gRPC server helper (consume\_grpc\_server\_helper.go) that wraps the streaming server to send consume, create, and close responses; the main consume\_server.go implementing the ConsumeServer which manages the receive/send loops for consuming messages from the WAL and handling transaction messages; and a metrics.go file that tracks consumer-level Prometheus metrics including total scanners, in-flight messages, and bytes consumed.

internal/streamingnode/server/service/handler/consumer · high confidence

A new \ManagerClient\ interface and its implementation (\manager\_client.go\, \manager\_client\_impl.go\) have been added to \internal/streamingnode/client/manager\. This client manages WAL (Write-Ahead Log) instances across all streaming nodes, providing methods to watch for node changes, retrieve all streaming nodes with their resource groups, collect status, and assign or remove WAL instances for channels. The implementation uses gRPC with internal TLS, context-aware logging, and service discovery to interact with the \StreamingNodeManagerService\. A corresponding test file (\manager\_test.go\) validates the client's behavior.

internal/streamingnode/client/manager · high confidence

Added a new \StreamingNodeManager\ in the \snmanager\ package to manage streaming query nodes, including tracking node availability, virtual channel allocation, and resource group-based node discovery. The manager provides methods to retrieve streaming node IDs, check service readiness, and handle fallback caching during shutdown. A corresponding test suite validates node retrieval, resource group grouping, and error fallback behavior.

internal/coordinator/snmanager · high confidence

Added new files in the \internal/flushcommon/pipeline\ directory to support the streaming pipeline architecture. This includes the \DataSyncService\ for managing flowgraphs per collection, a \FlowgraphManager\ to track active flowgraphs, and specific flowgraph nodes (\ddNode\ for DDL messages, \dmInputNode\ for data messages, and \ttNode\ for time ticks). These components handle message filtering, time tick propagation, and lifecycle management within the DataNode's data processing pipeline.

internal/flushcommon/pipeline · high confidence

Added a new \internal/distributed/streaming\ package that provides the distributed access layer for the streaming service. This includes a WAL accesser singleton for initialization and lifecycle management, a balancer implementation for managing streaming node assignments and rebalancing, a forward service to route legacy proxy requests, a replicate service for cross-cluster replication, and a msgstream adaptor for delegator integration. Tests are included for the balancer, forward, msgstream adaptor, and replicate service.

internal/distributed/streaming · high confidence

Added new gRPC resolver implementations for the streaming service, including a session resolver and a channel assignment resolver. These components enable dynamic service discovery and state updates for streaming coordination, with corresponding unit tests verifying their behavior.

internal/util/streamingutil/service/resolver · high confidence

The streamingcoord server is introduced to manage the state of streaming nodes, including channel assignment and broadcast services. This change adds the server implementation, builder, and associated tests, enabling the system to handle streaming-related coordination tasks such as balancing and broadcasting messages.

internal/streamingcoord/server · high confidence

The streaming node's Write-Ahead Log now includes a new timetick interceptor that manages timestamp synchronization and transaction handling for messages. This component ensures that time ticks are correctly assigned to messages, tracks acknowledgments, and handles transaction states (begin, commit, rollback) before appending to the WAL. It also integrates with a write-ahead buffer to optimize message persistence and ensures graceful shutdown of transaction managers.

internal/streamingnode/server/wal/interceptors/timetick · high confidence

Added a new \MixCoordClient\ implementation in \internal/distributed/mixcoord/client\ that consolidates gRPC communication with Root, Data, and Query coordination services into a single client. This change supports the architectural merge of RootCoord, DataCoord, and QueryCoord into a unified MixCoord service, enabling the system to route requests to a single, combined coordination endpoint rather than separate services.

internal/distributed/mixcoord/client · high confidence

The metrics package now provides a unified registry (MilvusRegistry) that aggregates both Go and C (C++/Cgo) metrics, including jemalloc statistics and thread activity counts. A new thread watcher monitors and exposes active thread counts by group (e.g., RocksDB, Knowhere, file write), and a Holmes-based memory profiler is integrated for non-Darwin platforms. These changes enhance internal observability by providing more accurate thread and memory metrics.

internal/util/metrics · high confidence

Added new utility functions in the typeutil package to support partition key hashing and empty retrieve result handling. The hash.go file introduces HashKey2Partitions, which maps int64 and varchar partition keys to partition names using a routing table, alongside helper functions HashMix and NextPowerOfTwo. The json\_util.go file adds ParseAndVerifyNestedPath for validating and escaping JSON nested paths. The result\_helper.go file introduces FillRetrieveResultIfEmpty, which populates empty retrieve results with appropriate empty field data, including handling for system fields like RowID and Timestamp. The retrieve\_result.go file defines the RetrieveResults interface and implementations for segcore, internal, and milvus result types.

internal/util/typeutil · high confidence

A new \vecindexmgr\ package has been added to \internal/util/vecindexmgr\, providing a centralized manager for querying vector index capabilities. This manager exposes methods to check support for specific vector data types (Binary, Float32, Float16, BFloat16, SparseFloat32, and Int8) and index features (NoTrain, DiskANN, GPU, Mmap, MV, Disk) for each index type. The implementation dynamically retrieves index features from the underlying C++ vector index engine and exposes them via a Go interface, enabling other components to verify index compatibility and capabilities programmatically.

internal/util/vecindexmgr · high confidence

The internal/storagev2/packed package now provides a new packed record reader and writer backed by the third-party milvus-storage library via a C++ FFI interface. This adds support for reading and writing data in packed format, enabling external table functionality and improved storage operations. The implementation includes C header definitions for Arrow C Data Interface and helper functions, Go bindings for filesystem metrics, and comprehensive test coverage for the new storage capabilities.

internal/storagev2 · high confidence

The codebase now supports chaining multiple interceptors for WAL operations, allowing for modular and composable behavior such as time tick handling, transaction management, and replication. This change introduces a \ChainedInterceptor\ that sequentially executes multiple \Interceptor\ implementations, each potentially contributing specific functionality like metrics collection or state recovery. The \Interceptor\ interface defines the contract for append operations, while \InterceptorWithReady\ allows interceptors to signal when they are prepared to handle requests. This structure enables more flexible and maintainable WAL processing logic.

internal/streamingnode/server/wal/interceptors · high confidence

A new \RateLimiterTree\ and \RateLimiterNode\ implementation has been added to \internal/util/ratelimitutil\. This change introduces a tree-based structure that organizes rate limiters across multiple levels (cluster, database, collection, and partition). This allows the system to apply and manage rate limits and quota states (such as deny-to-read/write) in a hierarchical manner, enabling more granular control over service quotas and rate limiting across different scopes.

internal/util/ratelimitutil · high confidence

The internal/streamingnode/server/wal package now defines the core WAL interfaces (WAL, ROWAL, Opener, Scanner) and a builder pattern for instantiating them. This introduces a structured way to open, read, and manage WAL instances, including support for rate limiting, message filtering, and salvage checkpoints, laying the groundwork for pluggable WAL backends.

internal/streamingnode/server/wal · medium confidence

Adds a new pipeline abstraction in internal/util/pipeline, including the core pipeline and stream pipeline implementations, node interfaces, and a consuming slowdown mechanism to filter empty time tick messages. This provides a structured way to process streaming data through a chain of nodes, with built-in support for batching and status monitoring.

internal/util/pipeline · high confidence

The streaming node's Write-Ahead Log (WAL) now includes a dedicated stats manager that tracks insert and delete metrics for growing segments across L0 and L1 levels. This manager monitors memory thresholds, flush pressure, and segment sizes to trigger sealing operations when segments exceed high-watermark (HWM) or low-watermark (LWM) limits. The implementation includes configuration validation, Prometheus metrics exposure for growing segment bytes and rows, and a background worker that handles time-based, memory-pressure, and blocking L0 policies to ensure segments are sealed appropriately.

internal/streamingnode/server/wal/interceptors/shard/stats · high confidence

A new \query\_hook.go\ file adds a \QueryHook\ interface and helper functions (\OptimizeSearchParams\, \CalculateEffectiveSegmentNum\, \ShouldUseTwoStageSearch\) that allow external or internal plugins to modify search parameters such as TopK, search parameters, and two-stage search decisions. The implementation reads configuration from \AutoIndexConfig\ (e.g., \Enable\, \EnableOptimize\, \GlobalRefineEnable\) and applies optimizations like top-K reduction, global refine ratios, and two-stage search selection based on segment counts and search types. A corresponding test file \query\_hook\_test.go\ is also added to verify the optimizer behavior.

internal/util/searchutil/optimizers · high confidence

The balancer package now provides a thread-safe singleton registry for the streaming coordinator's balancer. A new \singleton.go\ file implements a \Register\ function to set the balancer instance and a \GetWithContext\ function to retrieve it, both protected by a read-write mutex to prevent data races. A corresponding \test\_utility.go\ file exposes a \ResetBalancer\ function for testing purposes, allowing the singleton state to be cleared between tests.

internal/streamingcoord/server/balancer/balance · high confidence

The streaming node's WAL adaptor now implements adaptive rate limiting to control write throughput across different operational states, such as recovery, flushing, and append operations. This change introduces a new \rate\ subpackage containing an adaptive rate limit controller that dynamically adjusts limits based on system metrics and configuration parameters. The implementation includes a builder and opener that integrate these rate limiters into the WAL lifecycle, ensuring that the streaming service can throttle writes to prevent resource exhaustion during high-load scenarios like catch-up or recovery. Tests have been added to verify the configuration fetching and rate limit behavior.

internal/streamingnode/server/wal/adaptor · high confidence

The streaming node now exposes detailed Prometheus metrics and OpenTelemetry trace propagation for all WAL operations. This includes append metrics (durations, retries, and interceptor timings), segment assignment metrics (allocations, flushes, inserts, and deletes), time tick metrics (allocation, acknowledgment, and sync), transaction metrics, scanner metrics (catchup and tailing modes), and write-ahead buffer metrics. Additionally, append operations now carry trace context into logs, enabling distributed tracing for WAL writes.

internal/streamingnode/server/wal/metricsutil · high confidence

The internal/storagecommon package now includes a new column group splitting mechanism. New files column\_group\_splitter.go and split\_policy.go define a ColumnGroupSplitPolicy interface and several default split policies (e.g., by data type, local format, and average size). This allows the system to dynamically partition columns into groups based on configurable rules, which can improve storage efficiency and loading behavior for wide tables.

internal/storagecommon · high confidence

The \internal/querynodev2/segments\ package now includes several new files that support segment lifecycle and query processing. A \CollectionManager\ is introduced to track and update collection schemas with versioning and barrier-based freshness checks. A \cgo\_util.go\ module provides a centralized helper for handling CGO status errors. Query execution is supported by \ignore\_non\_pk\_ops.go\, which implements operators for merging results by primary key with offset tracking and fetching field data. An \index\_attr\_cache.go\ module caches index attribute calculations, and a \disk\_usage\_fetcher.go\ periodically updates disk usage metrics. These changes are accompanied by corresponding test files (\collection\_test.go\, \ignore\_non\_pk\_ops\_test.go\, \index\_attr\_cache\_test.go\) and an \OWNERS\ file.

internal/querynodev2/segments · high confidence

Added new files in the internal/util/reduce package to support search aggregation, hybrid search group-by, and Go-layer ORDER BY pipelines. This includes field\_data.go for group-by field resolution and emission, group\_key.go for composite key hashing and equality checks, orderby/types.go for ORDER BY field parsing and conversion, and reduce\_info.go for carrying reduce-stage parameters like group-by field IDs and search aggregation flags. Tests are added for all new functionality.

internal/util/reduce · high confidence

A new \ack\ package has been added to \internal/streamingnode/server/wal/interceptors/timetick/ack\, introducing the \AckManager\ and related structures (\Acker\, \AckDetail\, \lastConfirmedManager\) to manage the acknowledgment of time ticks in the Write-Ahead Log (WAL) interceptor. This implementation provides the core logic for tracking, sorting, and confirming message timestamps, including handling transaction sessions and updating the last confirmed message ID. The change includes comprehensive unit tests for the new components.

internal/streamingnode/server/wal/interceptors/timetick/ack · high confidence

Added new utility files in internal/flushcommon/util, including ChannelCheckpointUpdater, TimeTickSender, RateCollector, and MsgHandler interface, to support checkpoint updates, time tick reporting, rate collection, and message handling for the DataNode flush pipeline.

internal/flushcommon/util · high confidence

The metastore layer now supports composite, ordered writes via a new \Update\ method on all catalog interfaces (Root, Query, Data, StreamingCoord, and StreamingNode). This is enabled by a new \UpdateAction\ model and a shared \txn\ package that provides a \Builder\ to accumulate KV operations and a \Commit\ function that applies them atomically when possible, or falls back to an ordered chunked flush when the operation set exceeds the storage backend's transaction size limit. This change introduces the infrastructure for atomic multi-key updates across the metastore, ensuring that complex metadata changes (like segment updates, channel checkpoints, and collection targets) are persisted with strong consistency guarantees.

internal/metastore · high confidence

The internal allocator package now includes a new \CachedAllocator\ that batches and caches ID requests, a \GlobalIDAllocator\ backed by a TSO (timestamp + sequence) mechanism, and an \IDAllocator\ that pre-allocates IDs from the RootCoord. These components provide a unified \Interface\ for allocating unique IDs, supported by corresponding unit tests and mock implementations.

internal/allocator · high confidence

Added new Jenkinsfile-based CI pipelines to support building and testing on ARM64 architectures. This includes \PR-Arm.groovy\ for pull request validation on ARM, \PublishArmBasedImages.groovy\ and \PublishArmBasedImages.groovy\ for publishing ARM64 Docker images to the registry, and \MetaMigrationBuilder.groovy\ for building migration tools. Additionally, \Nightly2.groovy\ and \PR-for-go-sdk.groovy\ introduce new nightly and PR testing workflows using Tekton, expanding coverage for Go SDK and general e2e tests.

ci/jenkins · high confidence

Added new Jenkinsfile-based CI pipelines for running chaos tests (ChaosTest, ChaosTestKafkaMQ), deployment tests (DeployTest, DeployTestKafkaMQ), and scale tests (Scale). These pipelines define the build, configuration, and execution steps for these specific test suites, enabling automated validation of Milvus stability under failure conditions, upgrade/reinstall scenarios, and scaling operations.

build/ci/jenkins · high confidence

A new KV-based catalog implementation for the streaming node has been added to persist recovery information, including WAL checkpoints, vchannel metadata, segment assignments, and salvage checkpoints. The catalog provides methods to save and list these components, ensuring that the consume checkpoint is persisted last to maintain consistency. Tests verify the atomicity and correctness of the recovery snapshot saving process.

internal/metastore/kv/streamingnode · high confidence

The streaming coordination server now exposes dedicated gRPC services for managing channel assignments and handling message broadcasting. The new \AssignmentService\ allows clients to discover and update replication configurations, including support for force-promoting a secondary cluster to primary. The \BroadcastService\ handles message broadcasting to all channels and includes backward compatibility logic to forward legacy import messages to DataCoord. These services are accompanied by comprehensive unit tests covering success, error, and idempotency scenarios.

internal/streamingcoord/server/service · high confidence

The binlog import utility now includes new readers for handling L0 (delta) logs and insert binlogs, enabling more efficient and filtered imports. The L0 reader supports time-range filtering on delete logs, while the main reader applies time-range and delete-filters to both insert and delete data. This change introduces a filter-based approach to skip deleted or out-of-range records during import, improving performance and correctness for backup/restore and bulk insert workflows.

internal/util/importutilv2/binlog · high confidence

The \cmd/tools/binlogv2\ directory now includes a suite of Python tools for analyzing Parquet files and interacting with MinIO storage. This includes \parquet\_analyzer\_cli.py\ for command-line metadata and vector analysis, \export\_to\_json.py\ for exporting Parquet data to JSON, and \minio\_client.py\/\minio\_parquet\_analyzer.py\ for downloading and analyzing Parquet files from MinIO buckets. The \parquet\_analyzer\ package provides the underlying logic for parsing metadata and deserializing vector data.

cmd/tools/binlogv2 · high confidence

Added a new \Broker\ interface and its \dataCoordBroker\ implementation in \internal/flushcommon/broker\. This introduces a dedicated abstraction for DataNode to communicate with DataCoord, wrapping gRPC calls for segment ID assignment, time-tick reporting, segment info retrieval, checkpoint updates, and binlog path management. The change also includes unit tests and a generated mock for the broker interface.

internal/flushcommon/broker · high confidence

The build directory now includes a comprehensive set of shell scripts (builder.sh, build\_image.sh, kind\_provisioner.sh, etc.) and a detailed README that guide users through building Milvus using Docker containers. The scripts support building for multiple OS versions (Ubuntu, Amazon Linux), enable host network mode, and integrate with VS Code for remote development. The documentation provides step-by-step instructions for setting up the build environment, running unit tests, and managing the development container.

build · high confidence

Added new \channelAssignmentDiscoverer\ and \sessionDiscoverer\ implementations in the streaming service discovery layer. The channel assignment discoverer manages node assignments via a watcher, while the session discoverer watches etcd for session changes, including a fix to clear stale sessions during retry to prevent stale state accumulation. Tests verify correct state transitions and stale session cleanup.

internal/util/streamingutil/service/discoverer · high confidence

The cmd directory now contains a new main.go entry point that manages the Milvus process lifecycle. It supports running commands as subprocesses, forwarding signals to them, and cleaning up session files upon exit. It also integrates AddressSanitizer (ASan) leak checking when the 'use\_asan' build tag is active, providing memory leak detection capabilities for developers. Additionally, an OWNERS file was added to define reviewers and approvers for the cmd directory.

cmd · high confidence

The delegator's delete buffer has been refactored into a new \deletebuffer\ package, introducing multiple storage backends (skip-list, double-cache, and list-based buffers) to manage L0 segment and delete data. This change replaces the deprecated QueryNode DoubleBuffer, ensuring that delete records are not lost during slow segment loading or buffer full conditions, and provides metrics for monitoring delete buffer size and row counts.

internal/querynodev2/delegator/deletebuffer · high confidence

A new \internal/util/function/chain\ package introduces an Arrow-based function chain pipeline, providing a structured way to compose operators (such as Select, Filter, and Merge) for search reranking. This replaces the legacy rerank implementation with a more flexible and typed execution model, including a \FuncChain\ executor, a \DataFrame\ container for search results, and type converters for Arrow data.

internal/util/function/chain · high confidence

The importv2 package now includes a scheduler that periodically executes pending import tasks, a dynamic execution pool that scales with CPU cores and configuration, a global memory allocator that blocks and releases memory for import tasks, and utilities for copying segment files and indexing metadata. These additions support the new import execution model by managing concurrency, memory, and file operations for import tasks.

internal/datanode/importv2 · high confidence

The DataNode's index package is refactored to introduce a new task scheduler and a concurrency pool for vector index building. The scheduler manages the lifecycle of index, analyze, and stats tasks, while the pool dynamically resizes based on the \DataNodeCfg.MaxVecIndexBuildConcurrency\ configuration. This change improves resource management and allows hot-reloading of the concurrency limit without restarting the DataNode.

internal/datanode/index · high confidence

The internal/metastore/model package now includes new model structs and their serialization/deserialization logic for Alias, Collection, Credential, Database, Field, Function, Index, Partition, Role, Segment, and SegmentIndex. These changes introduce the data structures and helper functions (such as Marshal/Unmarshal methods) that map internal Go types to protocol buffer messages, enabling the system to store and retrieve metadata for these core entities.

internal/metastore/model · high confidence

A new \metacache\ package has been introduced to manage segment metadata and statistics. This includes a \MetaCache\ interface and implementation for tracking segment states, primary key statistics (via \BloomFilterSet\ and \LazyPkStats\), and BM25 sparse vector stats. The module provides utilities for filtering segments by ID, state, or partition, and applying batched actions to update segment information. This refactoring consolidates metadata handling for the flush source and supports the new BM25 embedding feature.

internal/flushcommon/metacache · high confidence

The \internal/querycoordv2/assign\ package introduces a new, extensible framework for assigning segments and channels to nodes. It defines an \AssignPolicy\ interface and provides three concrete implementations: \RoundRobinAssignPolicy\ for simple distribution, \RowCountBasedAssignPolicy\ for load-aware assignment based on row counts, and \ScoreBasedAssignPolicy\ for advanced scoring that considers collection row counts, global row counts, and benefit evaluation. The framework includes a factory pattern (\AssignPolicyFactory\) to manage policy instances and shared components like node filters and benefit evaluators, allowing the system to dynamically switch assignment strategies.

internal/querycoordv2/assign · high confidence

The internal/querycoordv2/balance package has been restructured to support multiple load balancing strategies. A new BalancerFactory has been introduced to manage and cache different balancer implementations, including RoundRobinBalancer, ChannelLevelScoreBalancer, and MultiTargetBalancer. This change enables dynamic switching between balancing policies based on configuration, allowing for more flexible and efficient distribution of segments and channels across query nodes.

internal/querycoordv2/balance · high confidence

Added new observer components in the QueryCoord v2 subsystem to monitor and manage various system states. This includes the CollectionObserver for tracking collection load status and timeouts, the LeaderCacheObserver for invalidating shard leader caches, the ReplicaObserver for managing read-only and streaming query nodes in replicas, the ResourceObserver for automatic resource group recovery, and the TargetObserver for managing collection targets. Each component includes corresponding unit tests to verify their behavior.

internal/querycoordv2/observers · high confidence

A new offline installation guide (README.md) and a Python helper script (save\_image.py) have been added to the deployments/offline directory. The guide provides step-by-step instructions for manually downloading and loading Docker images for both Docker Compose and Kubernetes (Helm) deployments, addressing issues where installation might fail due to image availability. The accompanying save\_image.py script automates the process of pulling specified Docker images and saving them as tarballs, streamlining the offline setup process for users without direct internet access on the target host.

deployments/offline · high confidence

Added new proxy client and watcher utilities in the proxyutil package, including the ProxyClientManager and ProxyWatcher implementations, along with their corresponding unit tests and auto-generated mocks. The ProxyClientManager manages a pool of proxy clients, handling addition, removal, and invalidation of cache entries, while the ProxyWatcher monitors etcd for proxy session changes and handles reconnection logic for recoverable errors.

internal/util/proxyutil · high confidence

The \internal/querynodev2/tasks\ directory now contains the Go implementation for the new search reduce pipeline, including \heap\_merge\_reduce.go\ for k-way heap merging of segment results, \l0\_function\_chain.go\ for executing L0 rerank function chains, \boost\_score.go\ for applying boost scores, and \arrow\import.go\ for converting Arrow record batches into DataFrames. These files introduce the core logic for merging, reducing, and formatting search results, supported by corresponding test files (\\\_test.go\) and an \OWNERS\ file.

internal/querynodev2/tasks · high confidence

Added new query pipeline operators in the queryutil package to support ORDER BY and GROUP BY query patterns. This includes a DeduplicatePKOperator that merges and deduplicates retrieve results by primary key using timestamp-based replacement, a ConcatAndCheckPKOperator for validating cross-shard primary key uniqueness, and an OrderByLimitOperator that performs heap-based partial sorting to efficiently retrieve the top-K results. These operators are wired into new pipeline builders that construct query reduction pipelines for plain, ORDER BY, GROUP BY, and GROUP BY+ORDER BY scenarios, enabling the system to sort and limit query results directly in the Go layer.

internal/util/queryutil · high confidence

The \internal/datacoord/session\ package has been introduced to centralize the management of worker node sessions. This new package provides a \NodeManager\ to track and manage DataNode clients, and a \Cluster\ interface to dispatch tasks (such as compaction, import, index building, and statistics collection) to specific nodes. It also includes a \Session\ struct to handle individual node connections and lifecycle, along with corresponding unit tests and mock implementations.

internal/datacoord/session · high confidence

Added new utility components to the streaming node's Write-Ahead Log (WAL) subsystem to support transactional and ordering guarantees. This includes an AverageRateCounter for tracking byte rates, a WALCheckpoint for managing consume checkpoints and replication state, a ReOrderByTimeTickBuffer for sorting messages by time tick, a TxnBuffer for managing transaction lifecycle (begin, commit, rollback, and expiration), and a PendingQueue for tracking pending message sizes. These additions provide the foundational logic for handling message ordering, transactional consistency, and metrics within the streaming node.

internal/streamingnode/server/wal/utility · high confidence

A new 'vchannel fair' policy has been introduced to improve load balancing across streaming nodes. This policy distributes vchannels more evenly among nodes while considering pchannel affinity to minimize hotspots. The implementation includes configuration validation, scoring mechanisms for balance optimization, and associated unit tests to verify the new behavior.

internal/streamingcoord/server/balancer/policy/vchannelfair · high confidence

The write buffer subsystem has been refactored to introduce a new \BufferManager\ that manages per-channel write buffers. This change introduces distinct buffer types for different data flows: \InsertBuffer\ for handling insert messages, \DeltaBuffer\ for delete operations, and \L0WriteBuffer\ for L0 segment management. The \BufferManager\ coordinates these buffers, handling registration, checkpointing, and memory-based eviction. This structural change supports the new flush mechanisms, including growing-source flushes and L0 segment tracking, providing a more modular and testable foundation for data node flushing.

internal/flushcommon/writebuffer · high confidence

The proxy's access logging system has been refactored to support configurable log formats and dynamic configuration updates. Users can now define custom log formats using placeholders for database, collection, partition, user, method, and timing information. The access logger can be enabled or disabled at runtime via configuration changes, and the system supports writing logs to local files with rotation, caching, and optional upload to MinIO with retention policies. The implementation includes a new formatter manager that maps RPC methods to specific log formats, and adds tests for the formatter, global logger, and MinIO handler.

internal/proxy/accesslog · high confidence

The broadcaster now uses a dedicated \ackCallbackScheduler\ to manage the execution order of broadcast task acknowledgments, ensuring that tasks are processed in the correct sequence (by broadcast ID) to maintain consistency. This new architecture introduces a \broadcastScheduler\ to handle the initial broadcasting of messages and a \tombstoneScheduler\ for cleanup. The \broadcastTaskManager\ now coordinates these components, supporting features like force promote failover and WAL-based DDL/DCL patterns. Additionally, trace context is injected into broadcast messages to preserve distributed tracing across the broadcast lifecycle.

internal/streamingcoord/server/broadcaster · high confidence

The session utility now supports injecting server labels into the session object via environment variables prefixed with MILVUS\_SERVER\LABEL\. These labels can be role-specific (e.g., MILVUS\_SERVER\_LABEL\_querynode\_label) or global, allowing components to expose metadata such as embedded query node status or resource group assignments to other services.

internal/util/sessionutil · high confidence

The datanode now executes functions on external collection segments, computing function-output columns and writing them to a new packed file while preserving references to the original external data. This change introduces the \function\_executor.go\ and \task\_update.go\ files, which handle the streaming pipeline for function execution, segment organization, and task lifecycle management. The \manager.go\ file provides a thread-safe task manager for supervising external collection tasks, while \milvus\_table\_deltalog.go\ and \milvus\_table\_refresh.go\ handle deltalog processing and refresh logic for Milvus-table format external sources. Tests are added for the manager, deltalog processing, and task update logic.

internal/datanode/external · high confidence

The QueryCoord now uses a WAL-based DDL framework to handle load, release, and transfer operations for collections and partitions. This change introduces new callback mechanisms that broadcast load configuration changes to all nodes, ensuring consistent state across the cluster. The implementation includes specific handlers for altering load configs, dropping load configs, and managing resource groups, all coordinated through the new DDL callbacks. This allows for more robust and scalable management of collection and partition states, particularly in active-standby and distributed environments.

internal/querycoordv2 · high confidence

Added Windows batch scripts to support running and cleaning up core services on Windows. Specifically, the repository now includes run\_etcd.bat, run\_milvus.bat, and run\_minio.bat to launch each service, along with a cleanup\_data.bat script to remove generated data and logs. This enables users to deploy and manage these components directly on Windows environments.

deployments/windows · high confidence

The allocator implementation has been moved into a new \internal/datacoord/allocator\ package, separating the \rootCoordAllocator\ and its associated interface from the broader \datacoord\ codebase. This change includes the core \allocator.go\ implementation and corresponding unit tests (\allocator\_test.go\) and mock generation (\mock\_allocator.go\), providing a cleaner structure for ID and timestamp allocation logic.

internal/datacoord/allocator · high confidence

The \internal/querycoordv2/utils\ directory now contains extracted utility functions for the QueryCoord component, including checker types, metadata helpers, and test fixtures. This refactoring consolidates shared logic such as \CheckDelegatorDataReady\, \PackSegmentLoadInfo\, and replica assignment helpers into a dedicated package, improving code organization and testability for QueryCoord's internal operations.

internal/querycoordv2/utils · high confidence

The streaming node server initialization is refactored to use a builder pattern, encapsulating the setup of dependencies like etcd, gRPC, and metadata stores within a \ServerBuilder\. This change simplifies the construction of the streaming node server by providing a clear, step-by-step API for configuring the server's components before the final \Build()\ call.

internal/streamingnode/server · high confidence

The distributed layer for the mixed coordinator has been consolidated into a single gRPC service. The \service.go\ file introduces a new \Server\ struct that wraps the \MixCoord\ component, handling initialization of etcd and TiKV clients, and starting the gRPC server. A new \api\_testonly.go\ file provides test-only hooks to start and stop the checker for the underlying \querycoordv2\ server. Corresponding tests in \service\_test.go\ validate the server lifecycle and health checks. This change reflects the architectural shift of merging multiple coordination services into one.

internal/distributed/mixcoord · high confidence

The RootCoord's internal communication with other components (such as QueryCoord and DataCoord) has been refactored to use a new \Broker\ interface. This change introduces a \ServerBroker\ implementation that encapsulates RPC calls for operations like releasing collections, syncing partitions, and watching channels. The refactoring also introduces helper functions for managing collection properties, consistency levels, and field attributes, alongside a new \checkGeneralCapacity\ function to enforce limits on the total number of collections, partitions, and shards. Additionally, a new \OWNERS\ file is added to the \internal/rootcoord\ directory to define code review responsibilities.

internal/rootcoord · high confidence

The QueryCoord metadata layer has been restructured into distinct managers for collections, channels, and distribution state. The \CollectionManager\ now explicitly tracks collection and partition load states, including schema and refresh notifications, while the \ChannelDistManager\ and \SegmentDistManager\ (aggregated in \DistributionManager\) handle node-channel and node-segment mappings. A new \CoordinatorBroker\ unifies access to RootCoord and DataCoord APIs, and a \FailedLoadCache\ prevents repeated failed load attempts for the same collection. These changes improve code modularity and recovery logic for the QueryCoord metadata store.

internal/querycoordv2/meta · high confidence

The internal/querycoordv2/job package was restructured to introduce a new job-based execution model for managing collection and partition lifecycle operations. This includes new job types for loading (LoadCollectionJob), releasing (ReleaseCollectionJob), updating configuration (UpdateLoadConfigJob), and syncing partitions (SyncNewLoadConfigJob), all orchestrated by a new Scheduler that ensures sequential execution per collection. The refactoring also adds a base Job interface and a Scheduler to handle concurrency, while undo mechanisms and utility functions support reliable state management and error handling during these operations.

internal/querycoordv2/job · high confidence

The QueryNode v2 delegator logic has been refactored into separate, focused files to improve code organization and maintainability. The main \delegator.go\ file now defines the \ShardDelegator\ interface and the \shardDelegator\ struct, while specific functionalities have been moved to dedicated files: \buffered\_forwarder.go\ handles buffering and forwarding of delta data, \delegator\_data.go\ manages insert and delete data processing, \delegator\_twostage.go\ implements the two-stage search flow, and \delta\_forward.go\ manages L0 and streaming delta forwarding policies. This separation clarifies the responsibilities of each component within the delegator.

internal/querynodev2/delegator · high confidence

The streaming node's flusher implementation has been refactored to use a new \dataSyncServiceWrapper\ that manages data sync services per vchannel, handling message dispatch, checkpointing, and graceful closure. A new \flusherComponents\ structure encapsulates the dependencies (WAL, broker, checkpoint updater, chunk manager) and manages the lifecycle of data sync services, including creation on collection creation and cleanup on drop. The \msg\_handler\_impl\ now handles various message types (flush, manual flush, create segment, schema change, truncate collection, WAL alteration) by delegating to the write buffer manager. Metrics are tracked via a dedicated \flusherMetrics\ struct. Tests have been added for the message handler and the main flusher logic.

internal/streamingnode/server/flusher · high confidence

The index parameter validation logic in \internal/util/indexparamcheck\ has been refactored from a monolithic structure into a modular checker system. This change introduces a \baseChecker\ and a \conf\_adapter\_mgr\ that manages a registry of specific index checkers (e.g., \AUTOINDEXChecker\, \BITMAPChecker\, \FMIndexChecker\). This structure allows each index type to validate its own parameters and data types independently, improving maintainability and testability of the index configuration logic.

internal/util/indexparamcheck · high confidence

New OWNERS files have been added to the internal/streamingcoord and internal/streamingnode directories. These files establish code review and approval workflows, designating 'chyezh' as a reviewer and 'maintainers' as approvers for these specific internal packages.

internal/streamingcoord, internal/streamingnode · low confidence

A new clustering utility has been introduced to handle vector distance calculations and clustering key resolution. The implementation includes functions to calculate distances for float vectors, serialize/deserialize float data, and determine the active clustering key field from a collection schema. The logic prioritizes an explicit clustering key, then a partition key, and finally a single vector field, with the ability to enable or disable vector field clustering keys via configuration.

internal/util/clustering · high confidence

The internal/util/cgo/logging package now bridges C++ glog output to the Go Zap logger. New files (glog\_logging.go, logging.go) implement a C-to-Go callback (goZapLogExt) that maps glog severity levels to Zap levels and writes logs via mlog, including an optimized path for async logging that avoids extra memory copies. A benchmark test (logging\_benchmark\_test.go) validates performance, and a unit test (logging\_test.go) confirms severity mapping.

internal/util/cgo/logging · high confidence

The githooks directory now includes a pre-push hook that runs the 'make verifiers' command, ensuring code checks are performed before pushing. The directory also contains an OWNERS file defining reviewers and approvers for this area, and a README with instructions for installing the git hooks.

githooks · high confidence

Added new certificate and key files in the configs/cert directory to support TLS for the proxy. The diff introduces a new Certificate Authority (CA) with its private key (ca.key, ca.pem, ca.srl) and separate server and client certificate/key pairs (server.key, server.pem, server.csr, client.key, client.pem, client.csr). These files provide the necessary cryptographic material for enabling TLS encryption in the proxy configuration.

configs/cert · high confidence

The quota utility now supports rate limiting at the database level, introducing a new \RateType\_DDLDB\ and corresponding configuration parameters (e.g., \MaxDBRate\, \DDLCollectionRatePerDB\). The \quota\_constant.go\ file establishes a mapping of rate types to configuration items across cluster, database, collection, and partition scopes, and registers dynamic metric updates for insert rates at each level. Tests verify the correct retrieval of quota values for each scope.

internal/util/quota · high confidence

The pre-commit hook now automatically formats Go source files using the 'make fmt' command. This ensures that all Go code is consistently formatted before being committed, reducing manual formatting overhead for developers.

githooks/pre-commit · high confidence

Credential lookups are now case-insensitive, allowing users to reference credentials using mixed-case names (e.g., 'aIstaffhub\_credential') while the system normalizes keys to lowercase for storage and retrieval. This ensures that credential names are matched regardless of capitalization, preventing lookup failures due to case mismatches.

internal/util/credentials · high confidence

The internal etcd client is now managed by a shared singleton in the kv utility package, ensuring that all components reuse the same client instance. This change introduces a centralized client creator that handles client lifecycle, including initialization and forced closure for testing, which improves resource management and supports future authentication and SSL configurations.

internal/util/dependency/kv · high confidence

The streaming service now uses a dedicated error package to define all error types, including closed, canceled, deadline exceeded, unrecoverable, fenced, and ignored operation errors. This centralizes error handling for the streaming service, making it easier to identify and handle specific error conditions such as fencing or closed connections.

internal/distributed/streaming/internal/errs · high confidence

The streaming coordinator's broadcaster registry now uses a centralized callback mechanism to handle message acknowledgments. This change introduces a typed, generic registry (\ack\_message\_callback.go\, \ack\_once\_message\_callback.go\) that maps specific message types (such as \Import\, \Snapshot\, \RBAC\, \Collection\ operations, and \TruncateCollection\) to their respective acknowledgment handlers. This allows the system to wait for and process acks from streaming nodes in a standardized way, improving reliability and reducing data race risks in the broadcast path.

internal/streamingcoord/server/broadcaster/registry · high confidence

Each component (CDC, DataNode, MixCoord, Proxy, QueryNode, StreamingNode) now has a dedicated wrapper in cmd/components that manages lifecycle (Prepare, Run, Stop, Health) and enforces a configurable graceful stop timeout. If a component fails to stop within the timeout, the system dumps pprof profiles (goroutine, heap, block, mutex) to aid debugging, then forcefully exits. This change prevents nodes from hanging during shutdown and provides diagnostic data when they do.

cmd/components · high confidence

The \cmd/roles\ package has been refactored to unify the management of Milvus components (Proxy, MixCoord, QueryNode, etc.) into a single \MilvusRoles\ struct. This change introduces a concurrent startup mechanism where components are prepared and run in parallel using a \conc.Future\-based runner, replacing the previous sequential or disjointed initialization. The entry point now orchestrates the lifecycle (Prepare, Run, Stop) of each component, ensuring that metrics, health checks, and logging are registered correctly during the unified startup sequence.

cmd/roles · high confidence

A new OpenSSL configuration file (configs/ssl/openssl-fips.cnf) has been added to enable FIPS (Federal Information Processing Standards) mode. This change configures OpenSSL to activate the FIPS provider and enforce FIPS-compliant algorithms, ensuring that cryptographic operations meet specific security standards.

configs/ssl · high confidence

Added validation logic in the schema utility to enforce that schema evolutions preserve field identities and structural invariants, and that function additions are bound to their new output fields. The new \ValidateSchemaEvolution\ function prevents unsafe changes to existing fields, struct containers, and dynamic fields, while \ValidateAlterSchemaAddFunctionPlan\ and \CheckNoFunctionCascade\ ensure that functions are added correctly and prevent unsupported cascading or standalone function additions.

internal/util/schemautil · high confidence

Added new OWNERS files for the internal/distributed and internal/kv directories, defining specific reviewers and approvers for code changes in these areas.

internal/distributed, internal/kv · high confidence

A new OWNERS file has been added to the deployments directory, defining the reviewers (LoveEachDay, zwd1208) and approvers (maintainers) responsible for this area.

deployments · high confidence

The shard client logic has been extracted into a new \shardclient\ package, providing a dedicated layer for managing QueryNode connections, caching shard leader information, and implementing load balancing policies. A key behavioral change is the introduction of a channel-based node blacklist, which temporarily excludes failed nodes from the load balancer's selection pool to prevent immediate retry failures. This refactoring also includes a fix for the shard leader cache to key by collection ID rather than name, resolving a potential alias repoint race condition.

internal/proxy/shardclient · high confidence

The Go SDK version constant has been updated to 3.0.0, reflecting the migration of the Go client module path to v3. This change introduces new common property keys for collection and field configurations, including support for JSON cast types, JSON paths, collection TTL, and Mmap settings, alongside a test to verify the new version string format.

client/common · high confidence

The import process now features a new retryable reader that automatically retries transient storage errors and recovers from premature EOFs, ensuring more robust data ingestion. Additionally, the system now enforces stricter validation rules: it checks varchar and array field lengths, validates UTF-8 string integrity, and handles timestamp import errors. These changes reduce import failures and improve error reporting for invalid data.

internal/util/importutilv2/common · high confidence

The streaming node now uses a centralized resource singleton to manage shared dependencies such as the etcd client, chunk manager, and coordinator clients. This change consolidates the initialization and lifecycle of these components, ensuring they are properly initialized and released, which improves stability and reduces memory usage during streaming node startup and shutdown.

internal/streamingnode/server/resource · high confidence

The proxy's client connection tracking has been refactored into a new, concurrent-safe \connectionManager\ located in \internal/proxy/connection\. This manager now handles client registration, active status updates, and periodic cleanup of inactive or excess clients using a priority queue for efficient eviction. The change includes a new \clientInfo\ struct, a singleton manager instance, and utility functions for extracting client identifiers from gRPC metadata. Tests verify the manager's ability to register clients, maintain activity status, and purge old entries based on configurable TTL and maximum connection limits.

internal/proxy/connection · high confidence

A new lock interceptor has been added to the Write-Ahead Log (WAL) pipeline to manage concurrency during append operations. The implementation acquires a global write lock for exclusive messages (such as flush or DDL control messages) to ensure correct ordering and prevent transaction conflicts, while using per-vchannel locks for regular exclusive messages. This change reduces contention and ensures that exclusive messages are processed in the correct sequence relative to transactional writes.

internal/streamingnode/server/wal/interceptors/lock · medium confidence

The C++ build process is restructured with new CMake modules (BuildUtils, DefineOptions, FindClangTools, ThirdPartyPackages, Utils) that standardize how dependencies are sourced, how build options are defined, and how third-party projects are managed. A precompiled header (milvus\_pch.hxx) is introduced to include common STL and third-party headers, which is expected to speed up compilation for all 350+ .cpp files.

internal/core/cmake · high confidence

The migration tool now uses a dedicated config package to parse and validate migration settings. This includes a RunConfig for tracking run, backup, and rollback commands along with source/target versions, and a MilvusConfig that loads metadata store and Etcd settings. Users benefit from clearer, structured configuration handling during migration operations.

cmd/tools/migration/configs · high confidence

The DataNode client has been refactored into a new package (internal/distributed/datanode/client) with a dedicated gRPC client implementation. This change introduces a new client structure that manages connections to DataNodes, including support for internal TLS configuration and server ID validation. The new client provides methods for component states, statistics channels, segment flushing, configuration retrieval, and metrics, all wrapped with context-aware logging and error handling.

internal/distributed/datanode/client · high confidence

A new Dockerfile for building the Milvus GPU image on Ubuntu 20.04 has been introduced. The image upgrades OpenSSL to address [CVE redacted], sets the timezone to UTC, and installs tzdata for IANA time zone support. It also configures jemalloc for memory profiling via environment variables and changes the default user to 'milvus' for OpenShift compatibility.

build/docker/milvus/gpu/ubuntu20.04 · medium confidence

Added new utility functions in internal/datanode/util to handle segment insert file paths. The GetSegmentInsertFiles function now preserves existing log paths when available, falling back to constructing paths from log IDs if necessary. This ensures that segment insert log paths are correctly maintained for text index operations.

internal/datanode/util · high confidence

The privilege management logic has been extracted into a new \internal/proxy/privilege\ package, introducing a \privilegeCache\ that caches credential and policy information to reduce redundant RPC calls. This change includes a new \MetaCacheCasbinAdapter\ that bridges the in-memory cache with the Casbin policy enforcer, and a \resultCache\ for caching enforcement results. These changes improve performance by caching credential lookups and policy evaluations, while the new package structure organizes privilege-related components separately from the main proxy code.

internal/proxy/privilege · high confidence

The proxy component received a large number of fixes and enhancements, including resolving data races, preventing panics from nil states or invalid parameters, and correcting search/reduce logic. Key fixes include handling empty search results, fixing hybrid search issues, and improving error messages and logging. The changes also address specific bugs like the proxy crashing when QueryCoord is offline, fixing the shard leader cache concurrency, and ensuring proper handling of nullable fields and array types.

internal/proxy · high confidence

The QueryCoord task scheduler and executor have been refactored to improve performance and reliability. The task execution pool is now split into separate capacities for channel and non-channel tasks, with the total capacity dynamically calculated based on the QueryNode's CPU core count. The scheduler now enforces strict concurrency limits to prevent overloading QueryNodes, and tasks are executed in separate goroutines with proper context cancellation. The refactoring includes new task types (Grow, Reduce, Move, Update, StatsUpdate) and actions (Grow, Reduce, Update, StatsUpdate, Reopen) to better represent the state of segment and channel operations. Additionally, the executor tracks executing tasks to prevent duplicate execution, and the scheduler manages task queues with priority-based scheduling.

internal/querycoordv2/task · high confidence

The shard interceptor in the Write-Ahead Log (WAL) subsystem has been refactored to use a builder pattern, introducing a new \interceptorBuilder\ that constructs the \shardInterceptor\. This change centralizes the initialization of the interceptor and the allocation of function runners for each collection's vchannel. The \shardInterceptor\ now explicitly manages function runner lifecycle through \allocFunctionRunners\ and \updateFunctionRunners\ methods, ensuring that function fields are materialized before WAL append operations. Additionally, the interceptor now handles schema version checks for insert messages and supports all partition operations within the shard manager for L0 segments.

internal/streamingnode/server/wal/interceptors/shard · high confidence

The DataCoord KV catalog has been refactored to use a new composite Update API that groups multiple metadata operations (such as segment, channel, and task updates) into a single transaction. This change introduces a new \Update\ method on the catalog that accepts a list of \UpdateAction\s, allowing atomic writes for related metadata changes. The refactoring also includes adding constants for various metadata prefixes (e.g., \SegmentPrefix\, \ImportJobPrefix\) and utility functions for building and validating segment metadata. This ensures that related metadata updates are persisted together, improving consistency and reducing the risk of partial updates.

internal/metastore/kv/datacoord · high confidence

The Proxy client implementation has been refactored to use a new generic gRPC client base (\grpcclient.GrpcClient\[proxypb.ProxyClient\]\), which standardizes connection management, retry policies, and internal TLS configuration. This change simplifies the client code by removing duplicated logic for creating connections, handling context cancellation, and managing session state, making the Proxy client more consistent with other distributed components.

internal/distributed/proxy/client · high confidence

The checkers in internal/querycoordv2/checkers have been refactored to use a new CheckerController that manages the lifecycle of individual checkers (Channel, Segment, Balance, Index, Leader). A new Checker interface and checkerActivation struct provide a standardized way to activate and deactivate checkers, allowing for better control over when checks run. The controller coordinates the execution of all checkers, each with its own interval and manual trigger channel. This change improves the modularity and testability of the checker system.

internal/querycoordv2/checkers · high confidence

The QueryNode gRPC client has been refactored into a dedicated internal package (internal/distributed/querynode/client). This change consolidates client initialization, connection management, and all RPC method wrappers (such as Search, Query, LoadSegments, and ReleaseCollection) into a single, unified client implementation. The new client supports internal TLS configuration and includes comprehensive unit tests to verify client creation, error handling, and context cancellation scenarios.

internal/distributed/querynode/client · high confidence

The QueryNode's gRPC service implementation has been refactored into a new \grpcquerynode\ package, introducing a cleaner separation between the server logic and the underlying \QueryNode\ component. This change includes adding a test-only helper to expose the server ID for testing purposes, and updating the service initialization to properly handle etcd client creation, MixCoord client initialization, and graceful shutdown. The refactoring also includes comprehensive unit tests for the new service structure.

internal/distributed/querynode · medium confidence

The binlog module was refactored to handle compression and decompression of insert, delta, stats, and BM25 logs. The new implementation uses a root path for path reconstruction and explicitly skips V1 delta log path reconstruction for V3 manifest segments, preventing incorrect path generation. This change ensures that binlog paths are correctly managed during compaction and stats tasks, improving reliability in segment metadata handling.

internal/metastore/kv/binlog · high confidence

The internal gRPC client utility has been refactored into a generic \ClientBase\ that wraps gRPC connections and provides a unified interface for calling remote services. This change introduces a generic \GrpcClient\ interface and a \clientConnWrapper\ to manage connection state and concurrency safely. The refactoring includes adding Zstandard (zstd) compression support for gRPC, implementing a \LocalGRPCClient\ for direct in-process calls (useful for testing or embedded scenarios), and defining specific error types like \ErrConnect\ for better error handling. These changes improve the reliability and testability of gRPC interactions within the system.

internal/util/grpcclient · high confidence

The proxy's network listener management has been restructured into a dedicated \listenerManager\ that handles both internal and external gRPC listeners, as well as HTTP/2 and HTTP/1.1 multiplexing via \cmux\. Additionally, a new \UnaryRequestStatsInterceptor\ has been introduced to capture unified gRPC request metrics, including latency, status codes, and error causes, with corresponding unit tests added to verify the metrics collection.

internal/distributed/proxy · high confidence

The streaming node's Write-Ahead Log (WAL) interceptors for shard management have been refactored to improve segment lifecycle handling. A new \partitionManager\ now explicitly manages segment allocation and flushing, replacing previous ad-hoc logic. This includes the introduction of \segmentAllocWorker\ and \segmentFlushWorker\ goroutines that handle asynchronous segment creation and flushing with exponential backoff retry logic. Additionally, a \seal\_policy\ package was added to define various conditions (e.g., capacity, memory, idle time) that trigger segment sealing. The \segmentAllocManager\ was updated to track state transitions (growing, flushed) and manage statistics, ensuring that segment operations like \AllocRows\ and \AsyncFlushSegment\ are handled through this new structured manager. These changes enhance the reliability of segment management, particularly regarding retry logic for WAL appends and consistent state tracking.

internal/streamingnode/server/wal/interceptors/shard/shards · high confidence

The streaming coordination service now uses a dedicated discover service to manage assignment discovery, separating concerns from the main streaming coordination logic. This change introduces a new \AssignmentDiscoverServer\ that handles the gRPC stream for node assignments, utilizing a balancer to watch for channel assignments and sending full assignment updates to clients. The implementation includes a helper struct to wrap the gRPC server interface and a test suite to verify the assignment discovery flow, including error reporting and close handling.

internal/streamingcoord/server/service/discover · high confidence

The build system for RocksDB has been updated to improve integration with the CMake-based build process. A new CMakeLists.txt file was added to configure the package and install the RocksDB headers to the correct destination. Additionally, a pkg-config template (rocksdb.pc.in) was introduced to define the library's dependencies (such as zlib and bzip2) and include paths, ensuring that downstream components can correctly link against RocksDB.

internal/core/thirdparty/rocksdb · medium confidence

The RootCoord catalog has been refactored to use a composite Update method for atomic collection operations, ensuring child metadata (partitions, fields, etc.) is persisted before the collection key, preserving crash-safety. Additionally, background garbage collection processes have been added to clean up legacy snapshot keys and tombstone markers from the metadata store, preventing unmarshal errors on restart. The implementation also includes comprehensive unit tests for the new catalog and GC logic.

internal/metastore/kv/rootcoord · medium confidence

The internal core library received a large number of fixes and enhancements, including performance optimizations for insert, query, and search operations, as well as memory usage improvements. Key changes include support for loading priority, text match operators, and various bug fixes related to segment loading, index handling, and error handling. The commit log also mentions the removal of redundant code and the addition of new features like support for the TEXT data type and geospatial data types.

internal/core · medium confidence

The internal JSON package now uses the bytedance/sonic library for serialization and deserialization, replacing the previous implementation. This change affects how the application handles JSON data, potentially improving performance and compatibility with standard Go JSON types.

internal/json · high confidence

The auto-generated mock files for MetaKv, TxnKV, and WatchKV have been updated to include context.Context as the first parameter for methods such as CompareVersionAndSwap, Has, and HasPrefix. This change aligns the mocks with the updated KV operation interfaces that now require a context, ensuring that tests using these mocks can properly support cancellation and timeouts.

internal/kv/mocks · high confidence

The internal milvus-storage submodule is updated to commit 63c29c6. The build configuration now explicitly links the AWS S3 SDK (AWS::aws-sdk-cpp-s3) to ensure S3 symbols are always resolved, and adds the Apple Network framework on macOS.

internal/core/thirdparty/milvus-storage · high confidence

The mock implementation for ChunkManager has been regenerated to include new methods such as Mmap, which utilizes the golang.org/x/exp/mmap package, alongside existing methods like Copy, Exist, and MultiRead. This ensures the test suite can properly simulate file storage operations, including memory-mapped file access, aligning with the underlying interface changes.

_internal/mocks/mock\storage · high confidence

The mock for the streaming node client has been regenerated to include new methods for resource group management, specifically AddResourceGroup, RemoveResourceGroup, and GetResourceGroup, allowing tests to verify interactions with resource group APIs.

_internal/mocks/streamingnode/client/mock\manager · high confidence

Added validation logic to ensure that function input and output fields are properly bound and that only specific connection/runtime parameters (such as integration\_id, model\_deployment\_id, and url) can be altered after creation. This prevents silent mixing of incompatible vectors by restricting changes to semantic parameters like dim, model\_name, or endpoint, and rejects attempts to change function type, name, or input/output fields.

internal/util/function/validator · medium confidence

A data race condition in the DataCoord component has been resolved. This fix ensures thread-safety when accessing shared metadata and segment information, preventing potential crashes or inconsistent states during concurrent operations such as compaction, flushing, and garbage collection.

internal/datacoord · high confidence

Add CGO wrapper tests for vector index building; Add test utilities for nullable fields and environment reset; Add unit tests for QueryNode v2 core components; Added mock for LastestMVCCTimeTickGetter; Added mock for QueryHook interface; Added mock for RecoveryStorage interface; Added mock for SealOperator interface; Added mock for TSO Allocator interface; Added mock for TombstoneSweeper; Added mock for WAL manager interface; Added mock implementation for streaming node client handler; Added mock implementation for the streaming coordinator balancer interface; Added mock implementations for Analyzer and TokenStream interfaces; Added mock implementations for CSegment and data utilities; Added mock implementations for WAL interfaces; Added mock implementations for WAL replicate interceptors; Added mock implementations for message and binlog I/O handlers; Added mock implementations for streaming broadcaster components; Added mock implementations for streaming interfaces; Added unit tests for JSON stats components; Consolidated C++ unit test build system and test suite; Expanded test coverage for new and existing features; Generated mock implementations for DataCoord, QueryCoord, and RootCoord catalogs; Introduce C++ test utilities for segcore; Refactored TSO allocator and added unit tests; Regenerate mocks with updated interfaces; Regenerated QueryNode mocks for the v3 Milvus proto API; Updated mock for ShardManager interface; Updated mock implementations for WAL interceptors; Updated mock interceptors for WAL to match current interface signatures; Updated rootcoord mock for IMetaTable interface; Updated streaming catalog mocks to support new metadata operations; Updated streaming coordination mock clients.

A new default PGO (Profile-Guided Optimization) configuration file has been added to support Go 1.21, aligning the build environment with the latest Go release.

configs/pgo · high confidence

The Go SDK client is now distributed as a separate module (github.com/milvus-io/milvus/client/v3) with Go 1.24.9, including updated dependencies like gRPC v1.80.0 and protobuf v1.36.11. Additionally, new requirements files are added for the binlog v2 tool and offline deployment, specifying versions for Python libraries such as streamlit, duckdb, and minio.

(dependencies) · high confidence

The internal/core/thirdparty/knowhere submodule has been updated to commit 16a71a3d. This update brings in the latest changes from the Knowhere library, which may include bug fixes, performance improvements, and new features. The CMake build configuration has been updated to fetch this specific commit, and the knowhere.pc.in file has been added to support pkg-config integration.

internal/core/thirdparty/knowhere · high confidence

The WebUI assets bundle (index-Cxslai7T.css and index-nmS7rbPW.js) has been regenerated, incorporating updated styles for buttons, icons, and form controls. The JavaScript bundle now includes React 18.3.1 and the corresponding react-dom production code, indicating a framework version upgrade that may affect component behavior and rendering.

internal/http/webui · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 49.

• Code Health 65
• Architecture 100
• Maturity 85
• Readiness 49
• Security 43
• Domain Modelling 61
• Accessibility 47

• 300 commits — 142 feature/other, 158 fixes

• internal/core — 106 commits
• internal/datacoord — 25 commits
• internal/querynodev2 — 20 commits
• tests/python_client — 20 commits
• internal/proxy — 13 commits
• internal/util — 11 commits
• internal/distributed — 9 commits
• internal/datanode — 8 commits
• internal/querycoordv2 — 8 commits
• internal/streamingnode — 8 commits
• docs/design-docs — 7 commits
• internal/parser — 7 commits
• pkg/util — 7 commits
• internal/rootcoord — 5 commits
• internal/storage — 5 commits
• (root) — 4 commits
• client/milvusclient — 4 commits
• internal/flushcommon — 4 commits
• internal/metastore — 3 commits
• internal/streamingcoord — 3 commits

• fix: enhance: bump Arrow recipe revision for Parquet fixes (#51829)
• fix: enhance: fix client telemetry and tracing gaps found auditing the Go SDK against the server (#51967)
• fix: fix: Avoid blocking compactor executor slot queries (#50789)
• fix: fix: Avoid external collection DML catchup on cold load (#51089)
• fix: fix: Avoid structured binding capture in OpenMP build (#51232)
• fix: fix: BulkIsValid non-nullable fall-through in ChunkedColumn/ProxyChunkColumn (#51598)
• fix: fix: CAS the schema-bump in-place manifest commit to stop dropping a concurrent index (#51376) (#51377)
• fix: fix: Check external fields against loaded manifest (#50773)
• fix: fix: Load StorageV3 child deletes during compaction fallback (#51208)
• fix: fix: NULL group-by on an index-only nullable scalar field (#51597)
• fix: fix: Raise bcrypt password hashing cost (#52144)
• fix: fix: Replace stale log call in expression bitwise test (#50972)
• fix: fix: Return actual db name and db id in cached DescribeCollection queried by collection id (#51254)
• fix: fix: Return early after parameter validation fails in RESTful v2 function DDL handlers (#51511)
• fix: fix: Validate REST quick-create enum fields (#51088)
• fix: fix: Validate quick-create binary metrics by auto index (#51168)
• fix: fix: [Storage] retry transient TiKV write transaction failures (#51426)
• fix: fix: add PK stats to growing-source flush (#51532)
• fix: fix: align load config compliance with replica override (#50825)
• fix: fix: allow aborting a failed CDC 2PC import to release the peer (#51083)
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

milvus-io/milvus was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.