lance-format/lance
64.0
Adequate · 14 November 2026
992.2k
lines of production code
Rust
primary language
5
measurements over time
What this system is
Features
Add BigANN benchmark dataset preparation scripts
Added a new benchmark suite for the BigANN dataset, including a Python script (\dataset.py\) and documentation (\README.md\) to prepare text-to-image datasets in Lance format. The script supports creating \text2image-10m\ and \yfcc-10m\ datasets, converting raw binary data into Lance files with ground truth queries, and includes a \.gitignore\ to exclude generated artifacts.
benchmarks/bigann · high confidence
Add Cohere Wikipedia embedding benchmark for index build performance
A new benchmark located in benchmarks/wiki allows users to generate a 35M vector dataset from the Cohere Wikipedia embeddings and build an IVF\_PQ index using a disk-based shuffler. Users can run datagen.py to create the Lance dataset and index.py to construct the index with configurable metrics (L2, cosine, dot), partition counts, and sub-vector counts, specifically designed to measure indexing speed rather than recall or latency.
benchmarks/wiki · high confidence
Add Dbpedia-entities benchmark for vector search performance evaluation
Users can now run a benchmark using the Dbpedia-entities-openai dataset (1M embeddings) to evaluate vector search performance. This includes a data generation script (\datagen.py\) to convert the Hugging Face dataset into Lance format, and a benchmark runner (\benchmarks.py\) that tests top-k recall across various IVF\_PQ index configurations (IVF 256/512/1024, PQ 32/96/192) and refine factors.
benchmarks/dbpedia-openai · high confidence
Add FSST compression library and benchmark example
Introduces the FSST (Fast Static Symbol Table) compression implementation in \rust/compression/fsst\, including the core logic in \src/fsst.rs\ and library entry point in \src/lib.rs\. Adds a benchmark example (\examples/benchmark.rs\) that demonstrates compressing and decompressing string data using the FSST algorithm, highlighting the output buffer size contract (8x expansion) and performance metrics. Also in this area: 1 commit (0 fixes) modifying rust/compression/fsst, rust/lance-encoding/src/array\_encoding.
rust/compression/fsst · medium confidence · unverified
Add OpenTelemetry metrics bridge for Lance
The Java library now includes a new \org.lance.otel\ package that bridges Lance's internal Rust metrics to OpenTelemetry. Users can call \LanceMetrics.instrument()\ to register observable instruments (counters, gauges, and histograms) on the global OpenTelemetry meter provider, allowing Lance's native metrics to be exported via standard OTel pipelines. The bridge supports querying the metric catalog and taking point-in-time snapshots of recorded data.
java/src/main/java/org/lance/otel · high confidence
Add owned, SIMD-accelerated bitpacking codecs
The \lance-bitpacking\ crate now includes its own internal bitpacking implementation (\bitpacker\_internal\) featuring SIMD-optimized kernels for x86\_64 (SSE3, AVX2) and aarch64 (NEON). This change introduces new owned codec types (\BitPacker4x\, \BitPacker8x\) that provide high-performance compression and decompression for \u32\ blocks, replacing or augmenting previous external dependencies with a Lance-owned, byte-compatible implementation.
rust/compression/bitpacking · high confidence
Add standalone CLI benchmark for PK-based point lookups across LSM levels
A new standalone CLI benchmark (\mem\_wal\_point\_lookup\_bench\) has been added to measure point lookup latency against three tiers of the LSM tree: the base table (on-disk, compacted data), SSTables (on-disk L0), and the active MemTable (in-memory write buffer). The tool supports two phases—\prepare\ to create the base dataset and initialize MemWAL, and \lookup\ to ingest rows via ShardWriter and time point lookups—allowing users to configure parameters such as base rows, maximum MemTable rows, number of SSTables, and query count, with results optionally output as JSON.
_rust/lance/benches/mem\_wal/point\lookup · high confidence
Added ExpLinkedList for memory-efficient storage
A new \ExpLinkedList\ container has been added to \lance-core\. This data structure stores elements in a linked list of vectors, where each vector's capacity doubles when full, providing a memory-efficient way to handle large numbers of elements. It supports standard operations like push, pop, and iteration, and implements \DeepSizeOf\ for accurate memory accounting.
rust/lance-core/src/container · high confidence
Added TPCH benchmark script to compare Lance and Parquet performance
Users can now run a Python-based benchmark in the \benchmarks/tpch\ directory to compare query performance between Lance and Parquet formats. The new \benchmark.py\ script supports TPCH queries Q1 and Q6, allowing users to generate sample datasets locally using DuckDB and measure latency differences between the two storage engines.
benchmarks/tpch · high confidence
Added flat vector index benchmark script
A new benchmark script has been added to the \benchmarks/flat\ directory to measure the latency of flat vector search. Running the script generates synthetic datasets with varying dimensions and lengths, executes nearest-neighbor searches using L2, cosine, and dot-product metrics, and outputs the results to \benchmark.csv\ along with a latency plot in \benchmark.html\.
benchmarks/flat · high confidence
Builder-style API for scalar index configuration and zone statistics
Java users can now configure scalar indices (B-Tree, Bitmap, Inverted, LabelList, NGram, ZoneMap) using dedicated builder classes (e.g., BTreeIndexParams.builder()) instead of raw JSON or maps. The InvertedIndexParams builder exposes full-text tuning options including analyzer presets (text/code), base tokenizers (simple, whitespace, raw, ngram, code, ICU, Lindera, Jieba), stemming, stop words, n-gram ranges, posting block size, and format version gating. B-Tree and ZoneMap builders allow setting zone/row sizing, while Bitmap supports explicit shard IDs for distributed builds. Additionally, a new ZoneStats class exposes per-zone min/max/null counts from zonemap indices via JNI, enabling users to inspect index statistics directly in Java.
java/src/main/java/org/lance/index/scalar · high confidence
Establish automated release and dependency governance tooling
The repository now includes configuration for \bumpversion\ to manage synchronized version bumps across Rust, Python, and Java artifacts, alongside \cargo-deny\ to enforce dependency license compliance, security advisory checks, and workspace dependency consistency. Pre-commit hooks are added to enforce code formatting, spell-checking, and lockfile synchronization, while a formal release process document and CI infrastructure (including LocalStack for S3 testing) standardize how beta, RC, and stable releases are built and published.
(repo-wide) · high confidence
Expose Lance datasets as DataFusion TableProviders
Users can now query Lance datasets directly using DataFusion's SQL and DataFrame APIs. This change introduces a \TableProvider\ implementation for \Dataset\ (in \logical\_plan.rs\) and a configurable \LanceTableProvider\ (in \dataframe.rs\) that supports filter, projection, and limit pushdown. The provider allows optional inclusion of system columns (\\_row\_id\, \\_row\_addr\), custom blob handling policies, and batch size configuration, enabling seamless integration of Lance data into DataFusion-based query engines.
rust/lance/src/datafusion · high confidence
Initial Java SDK release with Maven wrapper and documentation
This change introduces the initial Java bindings and SDK for Lance, providing a new Maven-based project structure under the \java/\ directory. It includes a Maven wrapper (version 3.3.2) for consistent builds, a \README.md\ with quick-start examples for dataset creation, random access, and schema evolution, and configuration files for code formatting (Spotless, Scalafmt). The release also bundles generated third-party license documents for both Java and Rust dependencies to ensure compliance.
java · high confidence
Internal tokenizer abstractions and implementations added to lance-tokenizer
The lance-tokenizer crate now provides the internal tokenizer abstractions and implementations used by Lance, including a TextAnalyzer builder, various tokenizers (ICU, Jieba, Lindera, CodeLex, N-gram, Simple, Raw, Whitespace), and token filters (AsciiFolding, LowerCaser, StopWord, Stemmer, RemoveLong, AlphaNumOnly, WordDelimiter).
rust/lance-tokenizer · high confidence
Introduce Binary Quantization (BQ) and RaBitQ support for vector indexes
Added a new Binary Quantization (BQ) module and RaBitQ (Rabit Quantization) support to the vector index pipeline. This includes a new \BinaryQuantization\ transformer that converts float vectors to binary codes based on sign bits, and a \RabitQuantizer\ that supports configurable bit-depths (1-9 bits) and rotation types (Fast/Matrix) for IVF\_RQ indexes. The change also integrates these quantizers into the IVF transformer pipeline, allowing users to leverage binary and reduced-bit quantization for more compact storage and faster search performance.
rust/lance-index/src/vector · high confidence
Introduce Bitmap and Bloom Filter scalar indices
Added Bitmap and Bloom Filter scalar index implementations to the Lance index library. The Bitmap index is designed for low-cardinality columns, storing a bitmap for each unique value to quickly identify matching rows. The Bloom Filter index provides a probabilistic data structure for efficient existence checks, helping to prune non-matching data during scans. Both indices integrate with the existing scalar index plugin framework, supporting training, querying, and caching.
rust/lance-index/src/scalar · high confidence
Introduce CacheCodec for FTS index entries and cross-column scoring
The inverted index module now supports persistent caching of posting lists and positions via a new \CacheCodec\ implementation, enabling efficient serialization and zero-copy deserialization of FTS index data for backends. Additionally, a new \combined\_fields\ scoring mode allows cross-column BM25F queries by treating target columns as a single virtual field, and a \cross\_column\ module enables compound FTS queries across multiple indexed columns by mapping document identities to a shared row-address domain.
rust/lance-index/src/scalar/inverted · high confidence
Introduce DataFilePart and external blob base resolution for staged writes
The dataset write path now supports caller-managed data file parts, allowing large writes to be staged and assembled later. A new DataFilePart struct serializes the identity and metadata of completed staging parts, which can be checkpointed and later assembled into a complete data file via DataFilePart::open\_all. This also introduces ExternalBaseResolver to map external blob URIs to specific storage bases, enabling multi-base blob storage and proper resolution of external blob paths during writes.
rust/lance/src/dataset · high confidence
Introduce FFI bindings for RecordBatchStream and new SpillStore for scratch storage
The lance-io crate now exposes a new \ffi\ module that wraps a \RecordBatchStream\ into an \FFI\_ArrowArrayStream\, enabling cross-language data exchange via the Arrow FFI interface. Additionally, a new \SpillStore\ trait and its \LocalSpillStore\ implementation have been added to provide reclaimable scratch storage on the local disk, allowing temporary data (such as index shuffle runs) to be written to disk and read back within the same process, with automatic cleanup and configurable disk capacity limits. Also in this area: 1 commit (0 fixes) modifying rust/lance-io, rust/lance/src/dataset/blob; 4 commits (2 fixes) modifying rust/lance-io; Simplified conditional copy logic and removed object\_store dependency; Fix I/O scheduler deadlock and spill quota leaks; Validate checkpoint file sizes before reading; and 5 more.
rust/lance-io · low confidence · unverified
Introduce IVF\_FLAT vector index implementation
Added the \IVF\_FLAT\ index type to the \lance-index\ crate, enabling exact vector search within IVF partitions. This change introduces the \FlatIndex\ as a sub-index implementation, along with \FlatFloatStorage\ for in-memory vector storage, \FlatPairScorer\ for exact distance calculations (supporting Float16, Float32, Float64, and binary vectors with L2, Cosine, Dot, and Hamming metrics), and \FlatTransformer\ for column normalization. The implementation includes optimized search paths using a binary heap for top-k results and supports distance range queries.
rust/lance-index/src/vector/flat · high confidence
Introduce Lance file format versions 2.1 and 2.2
This change adds the implementation for Lance file format versions 2.1 and 2.2, providing new writers and readers for these specific file grammars. Version 2.1 introduces a stable encoding strategy with dense u16 miniblock layouts and stricter validation for compressive encodings (e.g., requiring flat offsets for variable-width data). Version 2.2 expands on this by supporting u32 chunk grammars for miniblocks, allowing compressed constant-page levels, and enabling structural blob and map data types. Both versions include dedicated compression strategies and writer components that handle schema inference, batch writing, and footer generation according to their respective wire contracts.
_rust/lance-file/src/versions/v2\_1, rust/lance-file/src/versions/v2\2 · high confidence
Introduce Lance v2.0 file format support
Added the v2.0 file format implementation, including the reader, writer, and composition logic in the \rust/lance-file/src/versions/v2\_0\ module. This enables users to read and write data files using the v2.0 grammar, featuring specific handling for field encoding strategies, column metadata decoding, and page metadata spilling to bound memory usage during writes.
_rust/lance-file/src/versions/v2\0 · high confidence
Introduce Lance v2.3 file format with advanced compression and encoding
The Lance v2.3 file format is now available, introducing a new composition layer that supports modern compression strategies including ALP miniblock compression, per-tensor axis delta encoding, and dictionary-encode repeated fixed-size vectors. This version also implements planar page execution for structural arrays and allows explicit compression tuning via new writer options. Because this format is marked as unstable, a warning is emitted when writing files, and users are advised that v2.3 files may not be readable in future versions.
_rust/lance-file/src/versions/v2\3 · high confidence
Introduce Lance-native HNSW index for MemWAL in-memory tables
Added a new HNSW graph implementation within the MemWAL subsystem, enabling efficient approximate nearest neighbor search on vectors held in memory before they are flushed to disk. This change introduces configurable build parameters (such as graph levels, edge count, and construction beam width) and supports multiple distance metrics (L2, Dot, Cosine) via zero-copy access to Arrow-backed vector storage, allowing shard writers to maintain a high-performance in-memory index that mirrors the structure of flushed HNSW sub-indexes.
_rust/lance/src/dataset/mem\wal/hnsw · high confidence
Introduce MemWAL MemTable scanner with vector and full-text search support
This change introduces the new MemTable scanner implementation for the MemWAL (Write-Ahead Log) system, enabling in-memory search capabilities directly on the active memtable. Users can now perform vector searches (including HNSW-based nearest neighbor queries with configurable distance metrics and bounds) and full-text searches (supporting term, phrase, fuzzy, and boolean queries with cross-column support) on data that has not yet been flushed to disk. The scanner also handles nested column projections and ensures read-your-writes consistency by including the mutable tail in search results by default. Also in this area: Vector index compaction and dataset concurrency improvements; Fix memtable flush race conditions and unify distance computation; Improved vector search accuracy and performance in the active memtable.
_rust/lance/src/dataset/mem\wal/memtable/scanner · high confidence
Introduce MemWAL-based MemTable with lock-free batch storage and flush pipeline
The MemTable implementation now uses a new MemWAL architecture featuring a lock-free \BatchStore\ for concurrent reads and serialized appends, a dedicated flusher that writes data to persistent SSTables with optional cache warming, and a DataFusion-integrated scanner supporting full scans and index queries (BTree, HNSW, FTS) with MVCC visibility. This change replaces the previous in-memory storage mechanism with a persistent, WAL-backed design that improves concurrency and durability for dataset writes. Also in this area: 22 commits (12 fixes) fixing rust/lance/src/dataset; Vector index compaction and dataset concurrency improvements; Improved vector search accuracy and performance in the active memtable; Managed blob storage now requires explicit opt-in and MemWAL generations preserve base table field IDs.
_rust/lance/src/dataset/mem\wal/memtable · medium confidence · unverified
Introduce MemWAL: an in-memory write-ahead log with LSM-style tiering for Lance datasets
This change adds the MemWAL subsystem to the Lance dataset API, enabling high-throughput, low-latency writes by buffering data in an in-memory MemTable before flushing to persistent storage. Users can initialize MemWAL on a dataset using a builder API that supports sharding strategies (manual, unsharded, bucket, or identity) and configurable maintained indexes (BTree, HNSW, and FTS). The system introduces an LSM-like architecture with a fresh tier (active MemTable) and a frozen tier (SSTables), providing features such as read-your-writes consistency, schema evolution reconciliation, and configurable backpressure. This allows applications to perform rapid batch inserts and updates with immediate visibility, while maintaining durability and supporting vector and full-text search indexes on the buffered data.
_rust/lance/src/dataset/mem\wal · high confidence
Introduce data overlay files and stable row ID format support
The \lance-table\ crate now supports data overlay files, which allow updating specific cell values within a fragment without rewriting the underlying data files, and introduces the v2 data storage format (stable row IDs). This includes new \DataOverlayFile\ and \OverlayCoverage\ structures for managing these overlays, along with logic to handle index staleness when overlays modify indexed fields. The manifest format is updated to track the data storage format version, and \IndexMetadata\ now includes file sizes and creation timestamps.
rust/lance-table · high confidence
Introduce distributed execution support for vector search and filtered reads
Added protobuf serialization for the \ANNIvfPartitionExec\ and \FilteredReadExec\ execution nodes, enabling these plans to be serialized and sent to remote workers for distributed execution. Additionally, introduced a \LanceFilterExec\ wrapper around DataFusion's \FilterExec\ to preserve the original logical expression, allowing filter predicates to be serialized to Substrait for remote planning. These changes provide the necessary infrastructure for distributing vector search and filtered read operations across a cluster.
rust/lance/src/io/exec · high confidence
Introduce exact v2.x file format readers and writers
The \lance-file\ crate now supports reading and writing the exact v2.x file formats (v2.0 through v2.3) with version-specific encodings and metadata handling. This change replaces the legacy v1 reader with a modular, version-dispatched architecture that validates file structure, handles blob v2 descriptors, and supports structural projections. Users can now explicitly target stable (v2.2) or next (v2.3) file versions for new data, while existing v1 files remain readable via a separate legacy path. The new format includes improved encoding mechanisms, exact version identity, and better compatibility testing across versions.
rust/lance-file/src · high confidence
Introduce flat B-tree index page implementation
Added a new \FlatIndex\ structure in the scalar B-tree module to represent a single index page as a sorted batch of value/row-id pairs. This implementation enables efficient on-demand filtering of row IDs using Arrow's vectorized operations, supporting optimized handling of \IsIn\ predicates and null tracking without materializing intermediate structures per page.
rust/lance-index/src/scalar/btree · high confidence
Introduce hierarchical, type-safe session caches for metadata and indices
The session layer now uses a new hierarchical caching system to organize dataset metadata and index data, preventing collisions between different datasets and indices. This change introduces \GlobalMetadataCache\ and \GlobalIndexCache\ as top-level namespaces, with sub-caches for specific datasets and indices. It also implements type-safe cache keys for manifest data, transactions, deletion files, row address masks, and index metadata, ensuring that cache entries are correctly scoped and versioned. This improves cache reliability and performance by avoiding stale or incorrect data reuse across different datasets and index types.
rust/lance/src/session · high confidence
Introduce lance-arrow-scalar crate for efficient scalar comparisons
The rust/arrow-scalar crate (now named lance-arrow-scalar) provides an ArrowScalar type that wraps single-element Arrow arrays to enable O(1) comparison and hashing. By delegating to arrow\_row::OwnedRow, it ensures correct total ordering, proper NaN handling, and consistent null ordering, while also supporting serde serialization and conversion from primitive types.
rust/arrow-scalar · high confidence
Introduce lance-linalg crate with SIMD-accelerated distance metrics
The lance-linalg crate is introduced as an internal sub-crate containing native linear algebra algorithms for Lance. It provides distance metrics (L2, cosine, dot, hamming) for float types (f16, bf16, f32, f64) and integer types (u8, i8), featuring runtime-dispatched SIMD backends for x86\_64 (AVX2, AVX-512, AMX-FP16), aarch64 (NEON), and loongarch64 (LSX/LASX). The crate includes a build system that compiles C-based SIMD kernels and a \DeepSizeOf\ derive macro for Arrow-aware memory accounting.
rust/lance-linalg · high confidence
Introduce lance-tools CLI for inspecting Lance file metadata
A new \lance-tools\ command-line utility is added to the Rust workspace, providing a \meta\ subcommand that displays key metadata (version, row count, byte sizes, and schema) for Lance files. The tool supports both local filesystem paths and URI-based sources by leveraging the existing \ObjectStore\ registry and \ScanScheduler\ infrastructure.
rust/lance-tools · high confidence
Introduce memory allocation tracking for Python tests
Added a new \memtest\ utility that allows Python test suites to track memory allocations made by the Python interpreter and its libraries. The tool works by preloading a shared library (built in Rust) that intercepts memory calls, exposing statistics like peak usage and allocation counts via a Python API and CLI. It supports both Linux and macOS, enabling developers to detect memory leaks or regressions in their Python code.
memtest · high confidence
Introduce non-blocking async scanner and blob file APIs in Java SDK
The Java SDK now exposes an \AsyncScanner\ API that executes scans on a dedicated Tokio runtime and returns results via \CompletableFuture\, allowing non-blocking I/O for large queries. Additionally, new JNI bindings for \BlobFile\ provide direct read capabilities (\read\, \readUpTo\, \readRange\), and the \Dataset\ API now includes methods to retrieve blob data by row IDs or indices. These changes are supported by a new dispatcher mechanism that safely bridges asynchronous Rust tasks back to the Java thread. Also in this area: 1 commit (0 fixes) modifying (repo-wide); Add search effort parameter for IVF vector search.
java/lance-jni/src · low confidence · unverified
Introduce optional geo module with spatial UDFs and bounding box utilities
The \rust/lance-geo\ crate now provides a new optional \geo\ feature that exposes spatial capabilities. When enabled, it registers DataFusion UDFs for geometric measurements (Area, Distance, Length) and spatial relationships (Contains, Intersects, etc.) via the \geodatafusion\ integration. It also introduces a new \bbox\ module containing a \BoundingBox\ struct and helper functions for computing and managing spatial bounds, which are conditionally compiled and exported only when the \geo\ feature is active. Also in this area: 7 commits (3 fixes) modifying rust/lance-index; Extract lance-datafusion-exec crate to decouple execution logic from umbrella crate.
rust/lance-geo · medium confidence · unverified
Introduce pluggable cache backend architecture with URI configuration
The cache system now supports pluggable backends via a new \CacheBackend\ trait, allowing users to swap the underlying storage implementation. The module exposes a URI-based configuration system (\build\_from\_uri\) that parses strings like \moka://?capacity=1073741824\ into backend configurations, enabling consistent setup across Python, Java, and Rust bindings. Two built-in implementations are provided: \MokaCacheBackend\ for general use and \QuickCacheBackend\ for high-contention scenarios like session metadata caching. The change also introduces a versioned serialization envelope (\LCE1\) for cache entries, allowing persistent backends to store and retrieve data across restarts with backward-compatible decoding that treats invalid formats as cache misses.
rust/lance-core/src/cache · high confidence
Introduce protobuf schema guidelines and new wire-format definitions
The repository now includes a Protobuf Guidelines document (AGENTS.md) that establishes the change process, compatibility requirements, and schema design rules for protobuf schemas, including the requirement for PMC votes on persisted format changes. Additionally, new protobuf files have been added to define wire contracts and encoding specifications: ann.proto defines vector query parameters and execution plan serialization (including IVF sub-index and partition execution nodes); filtered\_read.proto defines serialization for filtered read options and plans; index.proto and index\_old.proto define vector index metadata and details; file.proto and file2.proto define file descriptors, schemas, and the v2 file format layout; encodings\_v2\_0.proto and encodings\_v2\_1.proto specify array and structural encodings for Lance file formats 2.0 and 2.1; and fragment\_metadata.proto defines data fragment, file, and overlay file metadata structures. Also in this area: 1 commit (0 fixes) modifying (repo-wide); 2 commits (0 fixes) modifying protos; Add search effort parameter and stabilize spilled row lineage feature flag.
protos · low confidence · unverified
Introduce public datagen API and benchmarks for array generation
The \lance-datagen\ crate now exposes a public API for generating test data arrays, including support for step, fill, random, and null patterns across various primitive types (Int8/16/32/64, Float32/64) and binary/string variants. This change adds a new \generator\ module with traits and implementations for array generation, along with a comprehensive benchmark suite (\benches/array\_gen.rs\) to measure performance of these generators. Users can now programmatically generate synthetic datasets with controlled characteristics for testing and development purposes.
rust/lance-datagen · high confidence
Introduce system indices for fragment reuse and MemWAL
The index library now supports two new system index types: FragmentReuse and MemWAL. FragmentReuse tracks stable partition mappings to enable deferred compaction and efficient row-remapping during data rewrites, while MemWAL provides a write-ahead log mechanism for index state. These are exposed as first-class index types with dedicated adapters, plugin registry entries, and progress monitoring, allowing the system to manage internal index metadata and write-ahead state transparently.
rust/lance-index/src · high confidence
Introduces versioned protobuf schema for index cache serialization
Adds a new \cache.proto\ file that defines the serialization format for index cache entries, including headers for FTS posting lists (compressed and plain), positions, posting groups, B-tree indices, and IVF vector partitions. This schema establishes the structure for cache keys and data, supporting features like configurable posting block sizes, impact data, and various quantizer types (PQ, Flat, SQ) with specific distance metrics and storage codecs.
rust/lance-index/protos-cache · high confidence
Introduction of the internal lance-index-core crate
The \lance-index-core\ crate has been introduced as an internal library containing the core traits and types used to implement index plugins for Lance. This new module defines the foundational \Index\ trait, the \IndexType\ enumeration (covering scalar types like BTree, Bitmap, and MinHashLsh, as well as vector types like IVF-PQ), and the \BuiltinIndexType\ enum for scalar index configuration. It also provides the \MetricsCollector\ trait for coarse-grained ANN stage timing and I/O statistics, and introduces asynchronous batch row-ID remapping utilities to handle row translation during index loading. This crate is explicitly marked as internal and not intended for external usage.
rust/lance-file, rust/lance-index · high confidence
Introduction of the lance-arrow internal sub-crate
The \lance-arrow\ crate has been introduced as an internal sub-crate containing Apache Arrow extensions for Lance. It provides support for bfloat16 vectors (including zero-copy array construction and round-tripping), JSON/JSONB read/write capabilities, and blob v2 schema handling. The crate also includes utilities for deep-copying Arrow arrays and RecordBatches, zero-copy IPC stream serialization, and memory accounting for Arrow buffers.
rust/lance-arrow · high confidence
Introduction of the lance-namespace Rust crate with core namespace APIs and compatibility shims
The \rust/lance-namespace\ crate has been added to provide the core Rust client and trait definitions for Lance namespaces, including the \LanceNamespace\ interface for managing namespaces and tables, schema conversion utilities between Arrow and JSON representations, and a comprehensive set of fine-grained error codes. To ensure backward compatibility with older SDK builds (Java and Python) that send scalar values for merge-insert keys, a deserialization shim in \compat.rs\ automatically promotes single-column string keys to the expected list format, while the error module standardizes error handling across all Lance implementations.
rust/lance-namespace · high confidence
Java SDK API overhaul with new core classes and session support
The Java SDK has been significantly refactored to align with the Rust engine, introducing a comprehensive set of new core classes including Dataset, CommitBuilder, Fragment, Branch, and BlobFile. This update adds support for session management, cache backends, and branch/tag metadata, while deprecating legacy fragment operations in favor of the new Transaction-based CommitBuilder. Users will now interact with a more modern, consistent API that exposes detailed dataset metadata, index build progress, and native blob handling capabilities.
java/src/main/java/org/lance · high confidence
Java bindings for MemWAL read and write operations
This change introduces Java bindings for the MemWAL subsystem, enabling Java applications to initialize MemWAL with sharding and index configuration, write and delete data via ShardWriter, and perform primary-key lookups, full scans, and IVF-PQ vector searches through LSM-aware planners and scanners.
java/src/main/java/org/lance/memwal · high confidence
Java bindings for Merge Insert operations
Added Java classes (MergeInsertParams, MergeInsertResult, MergeInsertStats) to support merge insert operations, including configuration for matched/unmatched row handling, write modes, and statistics reporting.
java/src/main/java/org/lance/merge · high confidence
Java cleanup API: new policy, explanation, and stats models
The Java cleanup module now exposes a structured API for configuring and inspecting dataset cleanup operations. Users can build cleanup policies via a builder that supports filtering by timestamp or version, targeting specific dataset versions, cleaning referenced branches, deleting unverified files, enforcing a delete rate limit, and erroring on tagged old versions. The API also returns a CleanupExplanation containing RemovalStats (bytes removed, counts of data/transaction/index/deletion files removed, and failed deletes) and a list of candidate files with their kind, size, and verification status, along with warnings and branch references.
java/src/main/java/org/lance/cleanup · high confidence
Java namespace client now supports dynamic per-request context and mTLS certificate reloading
The Java namespace client (DirectoryNamespace and RestNamespace) now accepts a DynamicContextProvider, allowing applications to inject per-request context such as rotating authentication headers before each operation. Additionally, the RestNamespace client automatically reloads rotated mTLS certificates based on the new tls.reload\_interval\_seconds configuration property, ensuring secure connections persist through certificate rotations without requiring a restart.
java/src/main/java/org/lance/namespace · high confidence
New CLI tools for FM index management and benchmarking
Added three new command-line utilities in the Rust binary directory: \fm\_index\_tool\ for listing, creating, and dropping FM (Fuzzy Matching) scalar indexes on datasets; \fm\_contains\_bench\ for benchmarking FM index query performance with configurable thread counts, caching, and output to JSON/CSV; and \lq\ for basic dataset inspection, querying, and vector index creation. These tools provide dedicated interfaces for managing and evaluating FM indexes, which are distinct from the existing vector index workflows.
rust/lance/src/bin · high confidence
New DataFusion integration for querying Lance namespaces via SQL
The \lance-namespace-datafusion\ crate introduces a bridge that allows Lance namespaces to be queried as native DataFusion catalogs, schemas, and tables. It exposes a \SessionBuilder\ to construct a DataFusion \SessionContext\ with dynamic \CatalogProvider\ and \SchemaProvider\ implementations backed by a \LanceNamespace\. This enables read-only SQL access to Lance datasets, mapping top-level namespaces to catalogs, child namespaces to schemas, and loading tables on-demand with caching.
rust/lance-namespace-datafusion · high confidence
New DatasetDelta API to inspect row-level changes between dataset versions
Added DatasetDelta and DatasetDeltaBuilder classes in the Java API, enabling users to compute and stream differences between two versions of a dataset. Users can specify a version range or compare against a specific version to retrieve inserted and updated rows as Arrow streams, list transactions, and obtain deleted row IDs. The implementation holds a read lock on the underlying dataset during native calls to ensure thread safety.
java/src/main/java/org/lance/delta · high confidence
New FragmentSession API for efficient repeated fragment reads
Users can now use the new FragmentSession API to perform repeated reads on a fragment without the overhead of opening a new reader each time. This session-based approach maintains internal state, such as sorted deletion vectors, allowing for more efficient row lookups and automatic handling of deleted rows across multiple operations. The implementation also ensures that JSON columns are correctly converted from the internal Lance format to standard Arrow JSON for user-facing outputs.
rust/lance/src/dataset/fragment · high confidence
New HD-Vila dataset benchmark with data generation script
Added a new benchmark entry for the HD-Vila-100M dataset, including a README with setup instructions and a \datagen.py\ script. The script enables users to generate the dataset in Lance format by downloading videos from YouTube, cutting specific clips, and processing them using Ray and PyArrow.
benchmarks/hd-vila · high confidence
New HNSW index implementation with online builder support
The HNSW vector index implementation has been replaced with a new version in the \lance-index\ crate. This update introduces a new \HNSWBuilder\ for offline graph construction and an \OnlineHnswBuilder\ that supports concurrent search during index building, which is particularly useful for in-memory MemTable indexes. The new implementation includes deterministic graph construction via a fixed random seed, configurable build parameters (such as \m\, \ef\_construction\, and \max\_level\), and support for both float and binary (Hamming distance) vectors. It also integrates with the existing IVF framework, allowing HNSW to be used as a sub-index within IVF partitions.
rust/lance-index/src/vector/hnsw · high confidence
New IVF index builder, shuffler, and transformer components
The IVF index module now includes dedicated builder, shuffler, storage, and transformer components. The builder introduces \IvfBuildParams\ with a new \target\_partition\_size\ option (replacing the deprecated \num\_partitions\), support for streaming k-means training via \streaming\_sample\_rate\ and \streaming\_coreset\_rate\, and the ability to load precomputed partition mappings or shuffle buffers. The shuffler handles disk-based partitioning of data streams, while the transformer computes partition IDs and distances for vectors. Storage is managed via the \IvfModel\ struct, which persists centroids, partition offsets, and lengths to Lance files.
rust/lance-index/src/vector/ivf · high confidence
New JSON serialization support for Arrow Schema types
Added a new \json.rs\ module that enables serializing and deserializing Apache Arrow \DataType\ and \Field\ structures to and from JSON. This feature supports a wide range of data types including primitives, lists, structs, and fixed-size binary/list types, allowing users to easily convert schema definitions into a portable JSON format for storage or interchange.
rust/lance/src/arrow · high confidence
New Java API for vector index configuration and training
The Java API now exposes a comprehensive set of builder classes for configuring vector index parameters, including HnswBuildParams, IvfBuildParams, PQBuildParams, RQBuildParams, and SQBuildParams. These are unified in VectorIndexParams, which provides factory methods to create IVF-based index configurations (IVF Flat, IVF PQ, IVF RQ, and IVF HNSW with PQ/SQ) and enforces validation rules such as mutual exclusivity of quantizers. Additionally, the new VectorTrainer utility allows users to pre-train IVF centroids and PQ codebooks with explicit distance type support, ensuring the training geometry matches the index build geometry to prevent silent recall degradation.
java/src/main/java/org/lance/index/vector · high confidence
New Java IPC layer with async scanning, full-text search, and vector search controls
The \org.lance.ipc\ package introduces a new Java interface to the native (Rust) engine, providing both synchronous (\LanceScanner\) and non-blocking asynchronous (\AsyncScanner\) data scanning via \CompletableFuture\. This release adds support for full-text search through the \FullTextQuery\ API, vector search tuning with the \ApproxMode\ enum (FAST, NORMAL, ACCURATE) for RQ-quantized indexes, and query optimization via \MaterializationStyle\ (early/late column fetching) and \ColumnOrdering\. Additionally, \ScanStats\ exposes detailed performance metrics, including per-query index cache hit/miss counts, to help users monitor and tune scan efficiency. Also in this area: 3 commits (1 fix) modifying java; Add search effort parameter for IVF vector search.
java/src/main/java/org/lance/ipc · medium confidence · unverified
New Java file I/O API with configurable blob reading and write options
The \org.lance.file\ package now provides dedicated \LanceFileReader\ and \LanceFileWriter\ classes for direct file-level operations. Users can control how blob-encoded columns are returned during reads via the new \BlobReadMode\ enum (materialized content or position/size descriptors) through \FileReadOptions\. The writer supports configuring buffering and page sizes via \FileWriteOptions\, allows specifying a data storage version, and enables attaching custom schema metadata that is flushed to the file footer on close.
java/src/main/java/org/lance/file · high confidence
New Java fragment metadata and result classes
Added new Java classes in the \org.lance.fragment\ package to support fragment-level metadata and update operations. \DataFile\ and \DeletionFile\ (with \DeletionFileType\) expose file paths, versions, and deletion details. \FragmentUpdateResult\ and \FragmentMergeResult\ provide structured outputs for column updates and merges, including row offset handling via JNI. \RowIdMeta\ and \VersionMeta\ wrap serialized Rust metadata for row IDs and version sequences, enabling stable row identification and version tracking in Java.
java/src/main/java/org/lance/fragment · high confidence
New Java operation model classes for dataset transactions
This change introduces the core Java classes that represent dataset transaction operations, aligning the Java layer with the underlying Rust implementation. Key additions include the Operation interface for base operation handling, SchemaOperation for managing schema changes and field ID allocation (including support for non-reusable field IDs on migrated datasets), KeyExistenceFilter for enforcing key uniqueness during updates via exact or Bloom filter strategies, RewrittenIndex to track index UUID mappings during rewrites, and UpdateConfig for modifying dataset configuration and metadata using a new UpdateMap pattern. These classes provide the structural foundation for transactional operations in the Java API.
java/src/main/java/org/lance/operation · high confidence
New LSM scanner execution nodes for MemTable scans, deduplication, and indexed searches
The MemTable scanner now includes dedicated DataFusion execution nodes to handle complex read patterns within the active MemTable. A new deduplication scan ensures that primary-key updates are resolved correctly by emitting only the newest version of each key, preventing stale rows from leaking through filters. Indexed searches are now supported directly on the MemTable via BTree, Full-Text Search (FTS), and Vector (HNSW) execution nodes, allowing scalar, text, and vector queries to leverage in-memory indexes with MVCC visibility. For vector searches where no HNSW index is available, a brute-force KNN executor provides exact distance calculations. All these nodes enforce visibility constraints, ensuring that only readable batches are scanned.
_rust/lance/src/dataset/mem\wal/memtable/scanner/exec · high confidence
New LSM scanner execution nodes for point lookups and cross-generation deduplication
The mem-wal scanner now includes a suite of new DataFusion execution nodes to optimize point lookups and handle multi-generation data. BloomFilterGuardExec skips generations that definitely do not contain a requested primary key, while CoalesceFirstExec short-circuits evaluation to return the newest matching row immediately. PkBlockFilterExec removes superseded rows by checking primary-key membership against newer generations, and FirstByPkExec deduplicates results (such as those from cross-column full-text search) by keeping only the first occurrence per primary key. MemtableGenTagExec annotates rows with their generation number for ordering, SchemaRelabelExec aligns storage and logical schemas, and ReconcileExec applies schema plans to batches.
_rust/lance/src/dataset/mem\wal/scanner/exec · high confidence
New Product Quantization implementation with 4-bit support and SIMD optimizations
The Product Quantization (PQ) vector index module has been rewritten to support 4-bit quantization alongside the existing 8-bit mode, enabling significantly higher compression ratios for vector data. The new implementation includes a dedicated builder for training PQ codebooks, a storage backend that persists PQ codes and row IDs, and a transformer for converting vector columns into PQ codes. Performance is improved through SIMD-optimized distance calculations, including AVX512-VBMI byte-plane lookups for 4-bit codes and pre-transposed distance tables to avoid runtime transposition overhead. The module also introduces a pairwise scorer for symmetric PQ code-to-code scoring, supporting L2, cosine, and dot distance metrics.
rust/lance-index/src/vector/pq · high confidence
New Python SDK modules for blobs, MemWAL, and benchmarking
The pylance package now includes several new modules that expose previously internal or Rust-only capabilities to Python users. The \lance.blob\ module introduces the \Blob\ dataclass and \BlobType\ PyArrow extension type, enabling users to write and read Lance blob columns with support for inline data, external URIs, and range selections. The \lance.mem\_wal\ module provides Python bindings for the LSM-tree-based MemWAL write path, exposing \ShardWriter\, \ShardSnapshot\, and \LsmScanner\ for high-throughput, low-latency writes and scans. Additionally, \lance.progress\ adds \IndexProgress\ and \FragmentWriteProgress\ classes for tracking index and fragment write operations, while \ci\_benchmarks/benchmark.py\ introduces an \io\_memory\_benchmark\ fixture for tracking peak memory and I/O statistics during test runs. Also in this area: 1 commit (1 fix) fixing pylance; 4 commits (1 fix) modifying pylance; 1 commit (0 fixes) modifying pylance, rust/lance/src/pb; Fix commit lock error propagation and update input type annotations; Introduce bfloat16 extension type and commit conflict error handling.
pylance · medium confidence · unverified
New Rust examples for vector search, full-text search, and dataset I/O
Added several new Rust example programs in the \rust/examples\ directory to demonstrate core Lance capabilities. The \full\_text\_search\ example shows how to create and query an inverted index for text data. The \hnsw\ and \ivf\_hnsw\ examples provide benchmarks and usage patterns for HNSW and IVF-HNSW vector indices, including building indexes and performing nearest-neighbor searches. The \llm\_dataset\_creation\ example demonstrates downloading and tokenizing text data from Hugging Face for LLM training. Finally, \write\_read\_ds\ illustrates basic dataset creation, reading, and cleanup operations.
rust/examples · high confidence
New SIFT/GIST-1M benchmark suite with LanceDB integration
The benchmarks/sift directory now provides a complete, reproducible benchmarking workflow for the SIFT-1M and GIST-1M datasets using the LanceDB API. This includes Python scripts for data generation (datagen.py), ground truth creation (gt.py), index building (index.py), and performance metric calculation (metrics.py), alongside a Jupyter notebook for result analysis. The suite supports multiple vector data types (f32, f16, bf16) and index configurations (IVF-PQ, DiskANN), with pre-computed statistics files (lance\_sift1m\_stats.csv, lance\_gist1m\_stats.csv) demonstrating recall and latency trade-offs across various partition and probe settings.
benchmarks/sift · high confidence
New Scalar Quantization (SQ) index implementation
This change introduces the core components for a new Scalar Quantization vector index within the \lance-index\ crate. It adds an SQ builder (\builder.rs\) to configure quantization parameters, a transformer (\transform.rs\) to convert input vectors into 8-bit codes, a storage layer (\storage.rs\) to manage the quantized data chunks, and a pairwise scorer (\pairwise.rs\) that performs exact integer-based distance calculations for L2, Cosine, and Dot metrics. This provides a new, efficient indexing option for users requiring lower memory footprint and faster search speeds through scalar quantization.
rust/lance-index/src/vector/sq · high confidence
New \`lance-test-macros\` crate for test-scoped tracing integration
A new \lance-test-macros\ crate has been added, providing a \\#\[test\]\ attribute macro that wraps existing test functions to automatically configure the \tracing\ library. When the \LANCE\_TRACING\ environment variable is set (e.g., to \debug\), tests wrapped with this macro initialize a tracing subscriber with a Chrome layer, generating \.json\ trace files compatible with Chrome DevTools or Perfetto. This allows developers to easily capture detailed performance and execution traces for individual tests without manual setup.
rust/lance-test-macros · high confidence
New arrow-stats crate for computing column statistics
Added the \lance-arrow-stats\ crate, which provides a \StatisticsAccumulator\ to compute min, max, null count, NaN count, and buffer memory usage for Apache Arrow arrays. This new capability supports numeric, temporal, boolean, string, binary, and list types, enabling more efficient predicate pushdown and query planning in Lance's columnar storage layer by tracking page-level statistics.
rust/arrow-stats · high confidence
New benchmarking examples for ACORN-1 HNSW traversal and proportional hierarchical k-means
Added three new example programs to the \lance-index\ crate to help users evaluate specific index features. \acorn\_bench.rs\ and \acorn\_bench\_sift.rs\ provide benchmarks comparing the new ACORN-1 mask-aware HNSW traversal against the standard \search\_basic\ and flat scans, measuring latency and recall under various filter selectivities and mask shapes (using synthetic data and the SIFT1M dataset respectively). \kmeans\_quality.rs\ allows users to measure the training time and clustering quality (WCSS, partition balance) of the new proportional hierarchical k-means algorithm for IVF training on large \.fbin\ datasets.
rust/lance-index/examples · high confidence
New benchmarking tool and credential vending infrastructure for Directory Namespace
This release introduces a copy-on-write \\_\_manifest\ commit benchmark (\manifest\_bench\) to measure Directory Namespace throughput under continuous and concurrent workloads, alongside a new \ConnectBuilder\ API for configuring namespace connections. It also adds a comprehensive credential vending system that automatically generates scoped, temporary credentials for AWS (STS AssumeRole), Azure (SAS tokens), and GCP, with built-in caching to reduce vendor call frequency and dynamic context providers for per-request header injection in the REST namespace.
rust/lance-namespace-impls · high confidence
New core utility modules for row addressing, rate limiting, and retry logic
The \rust/lance-core/src/utils\ directory now includes several new foundational modules that support internal operations. \address.rs\ introduces the \RowAddress\ type, which encodes fragment IDs and row offsets into a single 64-bit value to replace the previous \row\_id\/\row\_addr\ distinction. \aimd.rs\ provides an Additive Increase / Multiplicative Decrease (AIMD) rate controller for dynamically adjusting request rates, including validation to reject non-finite configuration values. \backoff.rs\ adds configurable exponential backoff and \SlotBackoff\ strategies to help spread out concurrent retries. Additional utilities include \assume.rs\ for release-checked invariants, \bit.rs\ for bit manipulation and alignment, \blob.rs\ for generating obfuscated blob sidecar paths, and \futures.rs\ for sharing streams between consumers.
rust/lance-core/src/utils · high confidence
New distributed index merging infrastructure
The distributed vector indexing subsystem now includes a new \index\_merger\ module and shared \partition\_merger\ helpers. This introduces the core logic for merging distributed index segments, including support for detecting various IVF index types (Flat, PQ, SQ, RQ, and HNSW variants) and writing unified metadata. It also adds strict and tolerant equality checks for vector data to ensure consistency during the merge process.
rust/lance-index/src/vector/distributed · high confidence
New lance-datafusion crate with DataFusion integration
A new \lance-datafusion\ crate has been introduced to provide a bridge between Lance and Apache DataFusion. This includes a build script for generating protobuf bindings, an \Aggregate\ struct for handling group-by and aggregate expressions, a \BatchReaderChunker\ for streaming record batches, and a \DataFrameExt\ trait that adds a \group\_by\_stream\ method to DataFusion DataFrames. The crate also implements core execution utilities like \OneShotExec\ for wrapping streams into execution plans, expression coercion logic in \expr.rs\ and \logical\_expr.rs\ to handle type conversions and filter simplification, and a \ProjectionBuilder\ to manage column selection and system column handling during query planning. Also in this area: Fixes float filter correctness and IVF build validation; Extract lance-datafusion-exec crate to decouple execution logic from umbrella crate; Improved column name resolution for backtick-quoted and literal names.
rust/lance-datafusion/src · high confidence
New lance-testing crate for internal test utilities
A new \lance-testing\ crate has been added to the repository, providing internal-only utilities for unit tests and benchmarks. It includes a \datagen\ module with generators for creating test data (such as incrementing integers and random vectors), a \progress\ module offering a macro to define in-memory progress recorders for testing distributed operations, and a \pprof\ module (Linux-only) that integrates with Criterion to generate flamegraph profiles during benchmarks.
rust/lance-testing · high confidence
New modular write API with staged transactions and retry support
The write operations (insert, delete, update, merge insert) now use a builder pattern that separates data preparation from the final commit. Users can stage changes using methods like \execute\_uncommitted\ to obtain a \Transaction\, which is then committed via the new \CommitBuilder\. This architecture introduces configurable retry logic for concurrent write conflicts, allowing users to set \conflict\_retries\ and \retry\_timeout\ on delete and update operations, and exposes a \write\_progress\ callback on inserts to track throughput. The \CommitBuilder\ also centralizes configuration for storage formats, object stores, and session caching.
rust/lance/src/dataset/write · high confidence
New quickstart and YouTube Q&A bot notebooks added
Added a new quickstart notebook demonstrating how to create and write Lance datasets using PyArrow and Pandas, including conversion from Parquet. Also added a new notebook for building a question-and-answer bot that searches YouTube transcripts using natural language, leveraging OpenAI embeddings and the Hugging Face datasets library.
notebooks · high confidence
New utility classes for JSON handling and range definitions
Added three new utility classes to the org.lance.util package: JsonFields provides helpers to create Arrow fields annotated with the 'arrow.json' extension for seamless JSON text handling; JsonUtils offers static methods to serialize and deserialize JSON using Jackson; and Range defines a simple immutable value object for representing integer ranges with inclusive start and exclusive end boundaries.
java/src/main/java/org/lance/util · high confidence
New utility modules for async primitives, time mocking, and test infrastructure
The \rust/lance/src/utils\ directory now includes three new modules: \future.rs\ introduces a \SharedPrerequisite\ struct that allows spawning an asynchronous background task and sharing its result (or error) across multiple threads, with synchronous access once the task completes; \temporal.rs\ provides a \utc\_now()\ function that abstracts the current system time, enabling unit tests to mock time via \mock\_instant\ while production code uses the real system clock; and \test.rs\ adds a \TestDatasetGenerator\ capable of creating datasets with randomized, 'hostile' column layouts (splitting fields across files, shuffling field IDs, and varying field orders) to improve test coverage for dataset operations under diverse storage configurations.
rust/lance/src/utils · high confidence
New v3 vector index architecture with shuffler and sub-index traits
The v3 vector index implementation introduces a new internal architecture for building and searching IVF-based indexes. This location adds the core shuffler component (shuffle\_bench.rs, shuffler.rs) which handles partitioning data streams into IVF partitions, and the sub-index trait definitions (subindex.rs) that standardize how sub-indexes like Flat and HNSW are built, searched, and remapped within the v3 format. Users benefit from the underlying performance and structural improvements of the v3 index format, including configurable CPU thread usage and optimized shuffle operations, as this code forms the foundational I/O and partitioning layer for the new index version.
rust/lance-index/src/vector/v3 · high confidence
New vector index details serialization and bounded partition stream for IVF builds
The vector index module now includes a new \details\ module that serializes and deserializes \VectorIndexDetails\ (including compression types like PQ, SQ, and RQ, and runtime hints) for index metadata, and a new \bounded\_partition\_stream\ module that manages memory-bounded, ordered execution of IVF partition builds. These changes support richer index introspection and more controlled resource usage during index construction. Also in this area: 11 commits (6 fixes) fixing rust/lance; 4 commits (2 fixes) modifying rust/lance; 5 commits (1 fix) modifying rust/lance/src/index; 1 commit (1 fix) fixing rust/lance/src/index/optimize; Fix Float16 IVF centroid type preservation and compaction fallback for metadata-only schema changes; and 11 more.
rust/lance/src/index · medium confidence · unverified
New vector index performance benchmark report
Added a new benchmark suite in the \benchmarks/full\_report\ directory to evaluate vector index performance. This includes a Python library (\\_lib.py\) for generating and processing vector data (specifically using the NYT dataset with TF-IDF and random projection), and a Jupyter notebook (\report.ipnb\) that runs recall tests against Lance IVF\_PQ indexes. The benchmark measures recall rates across different \nprobes\ and \refine\_factor\ settings for both in-sample and out-of-sample queries, visualizing the results as heatmaps.
_benchmarks/full\report · high confidence
Optimized execution paths for delete-only and in-place merge insert operations
The merge insert engine now uses specialized execution nodes to improve performance for specific operation types. A new delete-only path (\DeleteOnlyMergeInsertExec\) handles \WhenMatched::Delete\ scenarios by reading only row addresses and action columns, skipping the write step entirely to efficiently mark existing rows as deleted. Additionally, an in-place update path (\InPlaceMergeInsertExec\) allows narrow updates to existing fragments by patching only the source columns into existing data files and tombstoning old versions, avoiding the overhead of rewriting entire rows. These changes optimize bulk deletes and partial updates without creating new fragments.
_rust/lance/src/dataset/write/merge\insert/exec · high confidence
Restructure lance crate into modular sub-modules
The \rust/lance/src\ directory has been refactored from a flat structure into distinct sub-modules (\arrow\, \blob\, \datafusion\, \dataset\, \index\, \io\, \session\, \table\, \utils\). This change introduces new public APIs for blob v2 storage (including \BlobFieldOptions\ and dedicated writers), exposes DataFusion integration via \LanceTableProvider\, and adds object store metrics support (documented in \metrics.md\). The \lib.rs\ entry point now re-exports these modules and their specific capabilities, such as the new blob field helpers and the DataFusion table provider, while maintaining backward compatibility for core dataset operations.
rust/lance/src · high confidence
Vendored tokenizer stack with Lindera and Jieba support
The tokenizer implementation for full-text search has been moved into the lance-index crate, introducing dedicated tokenizers for Japanese and Korean text via Lindera and Chinese text via Jieba. This change adds new configuration handling for these language models and implements specific tokenization logic for JSON and text document types within the inverted index.
rust/lance-index/src/scalar/inverted/tokenizer · high confidence
Architecture
Extracted row-selection primitives into a new lance-select crate
The row-selection logic previously embedded in lance-core and lance-index has been extracted into a new, standalone lance-select crate. This crate provides the core mask types (RowAddrMask, NullableRowAddrMask) and index expression result wrappers (IndexExprResult, NullableIndexExprResult) that define which rows survive a filter, along with benchmarks to track their performance. This change decouples these primitives from larger dependencies, allowing downstream filtering code and benchmarks to depend on the mask substrate without pulling in the full lance-core or lance-index crates. Also in this area: 3 commits (1 fix) modifying rust/lance-index; 1 commit (1 fix) fixing rust/lance-select.
rust/lance-select · medium confidence · unverified
Introduce lance-core crate with core data types, error handling, and blob v2 schema definitions
This change introduces the \lance-core\ crate, consolidating foundational components previously scattered across the codebase. It defines the core data types and schema structures, including the new Blob v2 logical and descriptor field layouts, and provides a comprehensive error handling system with Levenshtein-based suggestions for missing fields. Additionally, it includes a custom \DeepSizeOf\ trait for accurate Arrow-aware memory accounting, replacing the previous \deepsize\ crate, and exposes system column metadata and dataset traits to decouple row-taking logic from the main dataset implementation.
rust/lance-core/src · high confidence
Introduction of the internal lance-index-core crate
The \lance-index-core\ crate has been introduced as an internal sub-crate containing the core traits and types used to implement index plugins for Lance. This new module defines the foundational \Index\ trait, the \IndexType\ enumeration (covering scalar types like BTree, Bitmap, and MinHashLsh, as well as vector types like IVF-PQ), and the \BuiltinIndexType\ enum. It also provides the \MetricsCollector\ trait for coarse-grained ANN stage timing and I/O statistics, the \BatchRowIdRemapper\ trait for asynchronous row-ID translation during index updates, and the \IndexWriter\/\IndexReader\ traits for serializing and loading index data. This crate is explicitly marked as internal and not intended for external usage.
rust/lance-index-core · high confidence
Migrate schema and data types to lance-core
The core schema and data type definitions (Field and Schema structs) have been moved into the lance-core crate. This refactoring centralizes the schema logic, introducing support for unenforced primary and clustering keys with explicit ordering, and adds validation for FixedSizeList dimensions to prevent runtime panics on zero-dimension lists.
rust/lance-core/src/datatypes · high confidence
Refactor encoding mechanisms to be version-free
The encoding implementation has been refactored to decouple the core encoding logic from specific file format versions. The \array\_encoding\ module now exposes a reusable \ArrayFieldEncodingStrategy\ that implements the persisted \ArrayEncoding\ grammar, while the \encoder/structural\ module provides version-free structural field encoder builders. This change allows the same encoding mechanisms to be composed and accepted across different file versions, with version-specific decisions handled separately. Also in this area: Fix memory duplication in dictionary-encoded array concatenation; Fixes list-item scheduling crashes and restores scan performance via early indirect reads; Fix dictionary overflow and empty chunk instructions in encoding; Version-free structural field encoding for blobs and primitives; Introduction of the internal lance-index-core crate; and 5 more.
rust/lance-encoding · high confidence
Vendor JSONB library into lance-jsonb crate
The \lance-jsonb\ crate now contains a vendored copy of the \databendlabs/jsonb\ library (adapted from commit fba895c5), replacing the previous external dependency. This change brings the JSONB encoding, parsing, and JSONPath evaluation logic directly into the Lance codebase, allowing Lance to evolve the encoding and read paths together with the file format. The vendored implementation is trimmed to include only the Databend binary layout and the public API required by Lance (parsing, \RawJsonb\ accessors, serde deserialization, and JSONPath selection), removing upstream features like the \sqlite\ backend, \jaq\ integration, and JSON manipulation functions. Numbers are parsed without the \arbitrary\_precision\ feature, matching the previous configuration where integers outside the \i64\/\u64\ range and decimals become \f64\. The crate includes a compatibility test (\tests/it/compat.rs\) that pins the bytes produced by jsonb 0.5.6 to ensure that any changes to the encoding remain readable by released Lance versions.
rust/lance-jsonb · high confidence
Behavioural changes
1 commit (0 fixes) modifying benchmarks/auto-ivf-effort
A change to existing behaviour in benchmarks/auto-ivf-effort — 1 commit, 21 files: “feat(search): calibrate recall targets by metric and k (\#10695)” (refs \#10695).
benchmarks/auto-ivf-effort · medium confidence · unverified
1 commit (0 fixes) modifying java/com
A change to existing behaviour in java/com — 1 commit, 1 file: “feat: support dynamic storage options provider with AWS credentials vending (\#4905)” (refs \#4905).
java/com · low confidence · unverified
1 commit (0 fixes) modifying protos/transaction
A change to existing behaviour in protos/transaction — 1 commit, 3 files: “feat(transaction): action-based transaction wire format (Transaction V2) (\#7954)” (refs \#7954).
protos/transaction · medium confidence · unverified
1 commit (0 fixes) modifying rust/lance-index
A change to existing behaviour in rust/lance-index — 1 commit, 24 files: “refactor: split scalar/inverted/wand.rs into submodules (\#10795)” (refs \#10795).
rust/lance-index/src/scalar/inverted/wand · medium confidence · unverified
11 commits (2 fixes) modifying rust/lance-datafusion-exec
A change to existing behaviour in rust/lance-datafusion-exec — 11 commits (2 fixs), 11 files: “chore: upgrade DataFusion to 55 and Arrow to 59 (\#10397)”; “refactor!: add lance-datafusion-exec so lance-index skips datafusion (\#10251)”; “perf(exec): report decode time in LanceRead metrics (\#10532)”; and 8 more (refs \#10397, \#10251, \#10532, \#10760, \#11037, …).
rust/lance-datafusion-exec · medium confidence · unverified
12 commits (3 fixes) modifying rust/lance/src/io/exec/fts
A change to existing behaviour in rust/lance/src/io/exec/fts — 12 commits (3 fixs), 16 files: “feat(fts): expose flat filter document binding (\#10649)”; “feat(fts): expose document expansion plan reconstruction (\#10622)”; “fix: do not push a limit below the FTS match filter (\#10862)”; and 9 more (refs \#10649, \#10622, \#10862, \#10643, \#10638, …).
rust/lance/src/io/exec/fts · medium confidence · unverified
13 commits (0 fixes) modifying rust/lance-index/src/scalar/minhash\_lsh
A change to existing behaviour in rust/lance-index/src/scalar/minhash\_lsh — 13 commits, 4 files: “chore: upgrade Rust toolchain to 1.98.1 (\#9577)”; “feat(index): remap indexes through fully covered fragment-reuse transitions”; “refactor!: add lance-datafusion-exec so lance-index skips datafusion (\#10251)”; and 10 more (refs \#9577, \#10251, \#9733, \#9518, \#10425, …).
_rust/lance-index/src/scalar/minhash\lsh · medium confidence · unverified
13 commits (4 fixes) modifying java/src/main/java/org/lance/compaction
A change to existing behaviour in java/src/main/java/org/lance/compaction — 13 commits (4 fixs), 8 files: “feat: record row-address moves for deferred compactions on stable-row-ID tables (\#9611)”; “feat(compaction)!: support max\_source\_rows and max\_source\_bytes limits (\#8235)”; “refactor!: use org.lance namespace for java package (\#5339)”; and 10 more (refs \#9611, \#8235, \#5339, \#6232, \#8575, …).
java/src/main/java/org/lance/compaction · medium confidence · unverified
14 commits (3 fixes) modifying java/lance-jni/src/blocking\_dataset
A change to existing behaviour in java/lance-jni/src/blocking\_dataset — 14 commits (3 fixs), 9 files: “feat(java)!: support bounded vector index consolidation (\#10755)”; “feat(index)!: add IVF\_HNSW\_RQ vector index (\#10522)”; “fix(java): release row address lookup lock before I/O (\#10931)”; and 11 more (refs \#10755, \#10522, \#10931, \#9438, \#11007, …).
_java/lance-jni/src/blocking\dataset · medium confidence · unverified
14 commits (4 fixes) modifying rust/lance/src/dataset/rowids
A change to existing behaviour in rust/lance/src/dataset/rowids — 14 commits (4 fixs), 2 files: “feat(dataset): write compaction's spilled lineage into the fragment's data file (\#9347)”; “perf(scan): skip element payloads for list length expressions (\#10759)”; “revert: compaction row-order suffix relabels (\#8400, \#9769) (\#10440)”; and 11 more (refs \#9347, \#10759, \#8400, \#9769, \#10440, …).
rust/lance/src/dataset/rowids · medium confidence · unverified
19 commits (1 fix) modifying java/src/main/java/org/lance/index
A change to existing behaviour in java/src/main/java/org/lance/index — 19 commits (1 fix), 14 files: “feat(java)!: support bounded vector index consolidation (\#10755)”; “feat(index)!: add IVF\_HNSW\_RQ vector index (\#10522)”; “fix(index): describe system indices without scalar plugins (\#10882)”; and 16 more (refs \#10755, \#10522, \#10882, \#5648, \#5339, …).
java/src/main/java/org/lance/index · medium confidence · unverified
19 commits (5 fixes) modifying rust/lance/src/dataset/scanner
A change to existing behaviour in rust/lance/src/dataset/scanner — 19 commits (5 fixs), 9 files: “perf(scan): skip element payloads for list length expressions (\#10759)”; “feat: support cross-column vector refinement”; “fix(fts): keep the scan limit out of FTS that filters or re-ranks a vector search (\#10836)”; and 16 more (refs \#10759, \#10836, \#10695, \#10812, \#10897, …).
rust/lance/src/dataset/scanner · medium confidence · unverified
2 commits (0 fixes) modifying benchmarks/auto-ivf-dot
A change to existing behaviour in benchmarks/auto-ivf-dot — 2 commits, 7 files: “refactor(python): organize dataset bindings by capability (\#10602)”; “feat(search): calibrate auto ivf probing for dot products (\#9467)” (refs \#10602, \#9467).
benchmarks/auto-ivf-dot · medium confidence · unverified
2 commits (0 fixes) modifying rust/lance
A change to existing behaviour in rust/lance — 2 commits, 2 files: “\[Rust\] Refactor lance rust crate into sub-modules (\#1250)”; “docs: update readme with latest lakehouse format info (\#5216)” (refs \#1250, \#5216).
rust/lance · medium confidence · unverified
2 commits (0 fixes) modifying rust/lance-index/src/pb
A change to existing behaviour in rust/lance-index/src/pb — 2 commits, 3 files: “build(protos): remove protoc from default Rust builds (\#10216)”; “feat(format): add the listelement scalar index specification (experimental) (\#10949)” (refs \#10216, \#10949).
rust/lance-index/src/pb · medium confidence · unverified
2 commits (0 fixes) modifying rust/lance-index/src/scalar/tuple
A change to existing behaviour in rust/lance-index/src/scalar/tuple — 2 commits, 1 file: “feat(index): plan tuple predicates over ordered key columns (\#10347)”; “feat(filter): keep the scalar index for comparisons of a column cast to a float (\#10820)” (refs \#10347, \#10820).
rust/lance-index/src/scalar/tuple · medium confidence · unverified
2 commits (0 fixes) modifying rust/lance-index/src/vector/storage
A change to existing behaviour in rust/lance-index/src/vector/storage — 2 commits, 1 file: “perf(index): overlap native RaBitQ plane prewarm (\#10843)”; “perf(index): overlap shared RaBitQ prewarm planes (\#11058)” (refs \#10843, \#11058).
rust/lance-index/src/vector/storage · medium confidence · unverified
2 commits (0 fixes) modifying skills
A change to existing behaviour in skills — 2 commits, 16 files: “docs: add lance skills as user guide (\#5877)”; “feat(skills): add a table maintenance skill (\#9610)” (refs \#5877, \#9610).
skills · medium confidence · unverified
20 commits (8 fixes) modifying rust/lance/src/dataset/blob
A change to existing behaviour in rust/lance/src/dataset/blob — 20 commits (8 fixs), 16 files: “refactor(dataset): separate blob rewrite and compaction responsibilities (\#10597)”; “fix(blob): deduplicate managed reference scans during cleanup (\#9435)”; “feat: expose a public plan for rewriting batches with Blob v2 columns (\#10724)”; and 17 more (refs \#10597, \#9435, \#10724, \#9869, \#8693, …).
rust/lance/src/dataset/blob · medium confidence · unverified
3 commits (0 fixes) modifying rust/lance-core/src/memory
A change to existing behaviour in rust/lance-core/src/memory — 3 commits, 8 files: “feat: opt-in process memory metrics with index build peaks (\#10451)”; “feat: log index build memory as trace events (\#10754)”; “feat: report memory held by I/O schedulers, DataFusion pools and MemWAL (\#10760)” (refs \#10451, \#10754, \#10760).
rust/lance-core/src/memory · medium confidence · unverified
3 commits (0 fixes) modifying rust/lance/src/pb
A change to existing behaviour in rust/lance/src/pb — 3 commits, 2 files: “build(protos): remove protoc from default Rust builds (\#10216)”; “feat: expose structured merge diagnostics and spill accounting (\#10752)”; “docs: say approx\_mode also affects HNSW searches that exclude rows (\#10751)” (refs \#10216, \#10752, \#10751).
rust/lance/src/pb · medium confidence · unverified
32 commits (13 fixes) modifying rust/lance/src/dataset/versions
A change to existing behaviour in rust/lance/src/dataset/versions — 32 commits (13 fixs), 1 file: “fix(encoding): account for compacted slices in page buffers (\#10344)”; “feat: validate mixed V2 data file versions (\#8581)”; “refactor(dataset): separate blob rewrite and compaction responsibilities (\#10597)”; and 29 more (refs \#10344, \#8581, \#10597, \#9347, \#9898, …).
rust/lance/src/dataset/versions · medium confidence · unverified
34 commits (9 fixes) modifying rust/lance/src/dataset/index
A change to existing behaviour in rust/lance/src/dataset/index — 34 commits (9 fixs), 2 files: “feat: introduce MemWAL index and system index (\#4069)”; “feat: record row-address moves for deferred compactions on stable-row-ID tables (\#9611)”; “feat!: auto-remap indexes before scan (\#3971)”; and 31 more (refs \#4069, \#9611, \#3971, \#8481, \#9620, …).
rust/lance/src/dataset/index · medium confidence · unverified
37 commits (3 fixes) modifying ci
A change to existing behaviour in ci — 37 commits (3 fixs), 27 files: “docs: shorten core major release vote window (\#6154)”; “refactor(dataset): separate blob rewrite and compaction responsibilities (\#10597)”; “refactor(namespace): split lance-namespace-impls dir.rs into submodules (\#10073)”; and 34 more (refs \#6154, \#10597, \#10073, \#10795, \#5339, …).
ci · medium confidence · unverified
4 commits (0 fixes) modifying rust/lance-index/src/vector/bq/storage
A change to existing behaviour in rust/lance-index/src/vector/bq/storage — 4 commits, 1 file: “feat(index)!: add IVF\_HNSW\_RQ vector index (\#10522)”; “feat(index): preserve native RaBitQ partition results (\#10683)”; “feat(index): support native lazy RaBitQ through fragment reuse (\#10609)”; and 1 more (refs \#10522, \#10683, \#10609, \#9834).
rust/lance-index/src/vector/bq/storage · medium confidence · unverified
4 commits (0 fixes) modifying rust/lance/src/io/commit/conflict\_matrix
A change to existing behaviour in rust/lance/src/io/commit/conflict\_matrix — 4 commits, 6 files: “refactor(dataset): separate blob rewrite and compaction responsibilities (\#10597)”; “test: characterize legacy conflict resolution end to end (\#9220)”; “refactor: split io/commit/conflict\_resolver.rs into submodules (\#10146)”; and 1 more (refs \#10597, \#9220, \#10146, \#10471).
_rust/lance/src/io/commit/conflict\matrix · medium confidence · unverified
4 commits (1 fix) modifying protos/codegen
A change to existing behaviour in protos/codegen — 4 commits (1 fix), 1 file: “build(protos): remove protoc from default Rust builds (\#10216)”; “feat: expose structured merge diagnostics and spill accounting (\#10752)”; “feat(transaction): action-based transaction wire format (Transaction V2) (\#7954)”; and 1 more (refs \#10216, \#10752, \#7954).
protos/codegen · medium confidence · unverified
4 commits (2 fixes) modifying rust/lance
A change to existing behaviour in rust/lance — 4 commits (2 fixs), 19 files: “fix: return an error for unknown columns in take projections (\#11032)”; “fix(fts): score explicit empty segment scopes without indexed statistics (\#11063)”; “perf(cleanup): delete cleanup candidates in bulk (\#10787)”; and 1 more (refs \#11032, \#11063, \#10787, \#10619). Also in this area: 4 commits (1 fix) modifying rust/lance/src/io; New search effort control and expanded auto-probing for vector search; Introduce distributed execution support for vector search and filtered reads.
rust/lance/src/io/exec/knn · medium confidence · unverified
4 commits (2 fixes) modifying rust/lance/src/io/commit/conflict\_resolver
A change to existing behaviour in rust/lance/src/io/commit/conflict\_resolver — 4 commits (2 fixs), 3 files: “fix(index): use read snapshot for cleanup coverage checks (\#10916)”; “feat(optimize)!: withdraw coverage an index cannot defer-remap in the compaction commit (\#10833)”; “refactor: split io/commit/conflict\_resolver.rs into submodules (\#10146)”; and 1 more (refs \#10916, \#10833, \#10146).
_rust/lance/src/io/commit/conflict\resolver · medium confidence · unverified
40 commits (12 fixes) modifying rust/lance/src/dataset/write/merge\_insert
A change to existing behaviour in rust/lance/src/dataset/write/merge\_insert — 40 commits (12 fixs), 10 files: “chore: upgrade datafusion to 48 (\#4106)”; “refactor: clean up lance-index/lance-table boundary (\#6988)”; “fix(merge-insert): track indexed primary key inserts (\#10606)”; and 37 more (refs \#4106, \#6988, \#10606, \#6934, \#8423, …).
_rust/lance/src/dataset/write/merge\insert · medium confidence · unverified
5 commits (1 fix) modifying rust/lance-datafusion/src/udf
A change to existing behaviour in rust/lance-datafusion/src/udf — 5 commits (1 fix), 3 files: “feat: add any\_element filter for per-element predicates on list\<struct\> columns (\#10901)”; “refactor!: add lance-datafusion-exec so lance-index skips datafusion (\#10251)”; “feat: add the experimental listelement scalar index over list\<struct\> elements (\#10950)”; and 2 more (refs \#10901, \#10251, \#10950, \#10947).
rust/lance-datafusion/src/udf · medium confidence · unverified
56 commits (16 fixes) modifying rust/lance/src/dataset/mem\_wal/index
A change to existing behaviour in rust/lance/src/dataset/mem\_wal/index — 56 commits (16 fixs), 11 files: “fix(mem\_wal): rank every vector search arm on one distance scale (\#9849)”; “fix(fts): require all ngrams in AND queries (\#10476)”; “fix: align mem-wal HNSW construction (\#8238)”; and 53 more (refs \#9849, \#10476, \#8238, \#8834, \#8836, …).
_rust/lance/src/dataset/mem\wal/index · medium confidence · unverified
59 commits (9 fixes) modifying rust/lance-index/src/vector/bq
A change to existing behaviour in rust/lance-index/src/vector/bq — 59 commits (9 fixs), 18 files: “fix(index): make RaBitQ dot scoring query-scale invariant (\#9452)”; “perf: speed up RaBitQ 4-bit LUT distance on ARM by 16x (\#6537)”; “perf(index): reuse RaBitQ LUTs and share IVF query state (\#9190)”; and 56 more (refs \#9452, \#6537, \#9190, \#10522, \#9577, …).
rust/lance-index/src/vector/bq · medium confidence · unverified
6 commits (0 fixes) modifying rust/lance-core/src/utils/cpu
A change to existing behaviour in rust/lance-core/src/utils/cpu — 6 commits, 1 file: “perf(linalg): select f64 l2/dot and norm\_l2 kernels once (\#10101)”; “refactor(core): generate Kernel::call impls with a macro (\#10214)”; “perf(linalg)!: select hamming kernels once and speed up batch and pairwise scans (\#10112)”; and 3 more (refs \#10101, \#10214, \#10112, \#10088, \#10113, …).
rust/lance-core/src/utils/cpu · medium confidence · unverified
66 commits (20 fixes) modifying rust/lance/src/dataset/optimize
A change to existing behaviour in rust/lance/src/dataset/optimize — 66 commits (20 fixs), 6 files: “feat: add file list with sizes to IndexMetadata (\#5497)”; “refactor(dataset): separate blob rewrite and compaction responsibilities (\#10597)”; “feat!: auto-remap indexes before scan (\#3971)”; and 63 more (refs \#5497, \#10597, \#3971, \#10635, \#9728, …).
rust/lance/src/dataset/optimize · medium confidence · unverified
82 commits (28 fixes) modifying rust/lance/src/dataset/mem\_wal/scanner
A change to existing behaviour in rust/lance/src/dataset/mem\_wal/scanner — 82 commits (28 fixs), 12 files: “fix(mem\_wal): rank every vector search arm on one distance scale (\#9849)”; “refactor: clean up lance-index/lance-table boundary (\#6988)”; “fix: honor nested projections in the mem-wal LSM scanner (\#8970)”; and 79 more (refs \#9849, \#6988, \#8970, \#7917, \#8836, …).
_rust/lance/src/dataset/mem\wal/scanner · medium confidence · unverified
Cached row-map reads for stable-partition rewrites
The \frag\_reuse\ module now supports caching of row-map label chunks to improve performance during stable-partition rewrites. A new \RowMapReader\ and associated caching infrastructure (\RowMapBlockCache\, \RowMapChunk\) allow the system to store decoded label blocks in the shared index cache, reducing repeated I/O when translating row IDs across rewritten fragments. This change introduces the \stable\_partition.rs\ mapping reader which integrates with this caching layer, enabling faster lookups for reordered data without changing the external API.
_rust/lance-index/src/frag\reuse · high confidence
Input validation and structural writer refactoring in file writing
The file writer now validates that input arrays match the file schema's Arrow types and extensions before encoding, rejecting mismatches with clear error messages to prevent data corruption. Additionally, the underlying writing logic has been refactored to use a new \StructuralFileSink\ for managing page metadata and spilling, which supports the exact current-format writers and improves memory management during large writes.
rust/lance-file/src/writer · high confidence
Introduces memory-bounded manifest scanning and typed file schemas
The dataset file handling now uses a shared, memory-bounded manifest walker that lists manifest locations and reads manifests with a 1 GB in-flight memory budget, preventing unbounded memory usage during scans. This infrastructure supports new, strongly-typed Arrow schemas for file metadata: tracked files are now represented with a dictionary-encoded type field (mapping discriminants for manifest, data, deletion, transaction, and index files) to ensure consistent labeling, while all files include size and last-modified timestamp fields. These changes provide a more robust and performant foundation for dataset introspection and file tracking.
rust/lance/src/dataset/files · high confidence
Inverted index rewritten with new format, caching, and search logic
The scalar inverted index implementation has been refactored into a new modular structure (cache, doc\_set, flat\_search, format, inverted\_index, partition, position\_blocks, posting\_batch\_builder, posting\_builder) to support FTS format version 3. This introduces a shared posting-list position stream, configurable posting blocks, and impact-aware caching for posting lists and groups. The change also adds a Unicode-scalar Levenshtein automaton for fuzzy matching, slice-aware cache-size accounting to reduce memory overhead, and a new flat full-text search path for unindexed data.
rust/lance-index/src/scalar/inverted/index · high confidence
Java SDK adopts org.lance namespace and introduces new transaction and concurrency APIs
The Java SDK has migrated its package structure to the \org.lance\ namespace, replacing the previous layout. This change introduces a new \SourcedTransaction\ class that simplifies commit workflows by pairing a transaction with its source dataset, alongside a new \LockManager\ that enforces read/write locking on dataset operations to prevent concurrency issues. Additionally, the SDK now exposes detailed MemWAL (Memory Write-Ahead Log) statistics and configuration via new classes like \MemTableStats\ and \ShardingSpec\, and updates the \Fragment\ API to acquire read locks during row deletion and column merging operations.
lance-core · high confidence
New commit retry timeout configuration and refactored I/O module structure
Users can now configure the commit conflict-retry timeout via the LANCE\_COMMIT\_RETRY\_TIMEOUT\_SECS environment variable, allowing long-running maintenance operations to succeed under high contention without hitting the default 30-second limit. Additionally, the I/O module has been restructured: commit handling logic is now centralized in a new commit.rs file with configurable backoff, deletion file reading is extracted into its own module with caching, and execution nodes are reorganized under io/exec.rs to expose updated scan, filter, and KNN execution interfaces.
rust/lance/src/io · high confidence
New conflict resolution logic for tagged fragment reuse indices
The commit handler now includes a comprehensive conflict resolver that correctly handles concurrent operations on tables with tagged fragment reuse indices. This ensures that mutations like deletes, updates, and merge inserts do not break the index's provenance tracking, and that the index coverage is correctly derived from live fragments rather than stale stored provenance.
rust/lance/src/io/commit · high confidence
New internal graph builder and I/O components for HNSW index construction
The \rust/lance-index/src/vector/graph\ module now includes a new \builder.rs\ file defining the \GraphBuilderNode\ struct, which manages neighbor relationships and ranking during the construction of HNSW graphs, and an \io.rs\ file stub for graph input/output operations. These changes provide the internal infrastructure required to build and persist HNSW graph structures, supporting the broader HNSW implementation updates in the vector index.
rust/lance-index/src/vector/graph · medium confidence
Python SDK development environment and documentation updates
The Python SDK development workflow has been updated to use \uv\ for environment management, with new guidelines in \AGENTS.md\ and \CONTRIBUTING.md\ specifying \uv sync\ and \uv run\ for setup and execution. The build configuration in \python/.cargo/config.toml\ now sets \haswell\ (AVX2) as the default target CPU for Linux wheels to improve performance. A new benchmark experiment configuration file was added at \benchmarks/full\_report/vbench/experiment.py\.
python · high confidence
Refactored Lance v1 file I/O into a modular, self-contained implementation
The v1 file format implementation has been reorganized into a dedicated module (\rust/lance-file/src/versions/v1\) with a clear separation of concerns. The new structure includes specific encoding modules for binary, dictionary, and plain data types, a format layer for metadata and page tables, and distinct reader and writer components. This refactoring establishes v1 as the canonical legacy file owner and composes exact current-format readers, ensuring that the v1 wire layout and schema dictionary payloads are handled by these dedicated codecs without leaking into the version-free I/O layer.
rust/lance-file/src/versions/v1 · high confidence
Fixes
1 commit (1 fix) fixing test\_image\_dataset
A fix in test\_image\_dataset — 1 commit (1 fix), 1 file: “fix: add versioning and bypass broken row counts (\#1534)” (refs \#1534).
_test\_image\dataset · low confidence · unverified
4 commits (3 fixes) fixing rust/lance-datafusion/src/pb
A fix in rust/lance-datafusion/src/pb — 4 commits (3 fixs), 1 file: “fix!: carry an explicit empty fragment selection through the FilteredReadExec proto (\#10863)”; “build(protos): remove protoc from default Rust builds (\#10216)”; “fix(scan)!: preserve list structure reads across plan transport (\#10766)”; and 1 more (refs \#10863, \#10216, \#10766).
rust/lance-datafusion/src/pb · medium confidence · unverified
4 commits (3 fixes) fixing rust/lance/src/dataset/overlay
A fix in rust/lance/src/dataset/overlay — 4 commits (3 fixs), 1 file: “fix: resolve struct validity from data overlays (\#9092)”; “feat(overlay): add a writer for data overlay files (\#8761)”; “fix: mask required struct children when splicing overlay values (\#11029)”; and 1 more (refs \#9092, \#8761, \#11029, \#10586).
rust/lance/src/dataset/overlay · medium confidence · unverified
9 commits (6 fixes) fixing rust/lance/src/dataset/cleanup
A fix in rust/lance/src/dataset/cleanup — 9 commits (6 fixs), 7 files: “fix(dataset): preserve complete branch source snapshots during cleanup (\#10488)”; “refactor(cleanup): move branch lineage tests into their own module (\#10620)”; “fix(cleanup): keep the files of detached versions (\#10611)”; and 6 more (refs \#10488, \#10620, \#10611, \#10327, \#10736, …).
rust/lance/src/dataset/cleanup · medium confidence · unverified
Bloom filter intersection logic corrected to prevent false positives
The bloom filter implementation in lance-core now reports filters as intersecting only when a block could actually hold a shared key. This fix ensures that bloom filter checks are more accurate, reducing false positive results during query execution by strictly validating block-level key overlap before declaring an intersection.
rust/lance-core/src/utils/bloomfilter · high confidence
Test coverage
11 commits adding/updating tests in rust/lance/src/io/commit/conflict\_resolver/tests/tagged\_rewrite\_conflicts; 2 commits adding/updating tests in java/src/test/java/org/lance/index/optimize; 2 commits adding/updating tests in rust/lance-table/tests; 2 commits adding/updating tests in rust/lance/src/dataset/optimize/tests/frag\_reuse\_contract; 27 commits adding/updating tests in java/src/test/java/org/lance/index; 27 commits adding/updating tests in rust/lance/tests; 3 commits adding/updating tests in java/src/test/java/org/lance/otel; 3 commits adding/updating tests in rust/lance/src/io/commit/conflict\_resolver/tests; 3 commits adding/updating tests in rust/lance/tests/topk\_late\_materialization; 335 commits adding/updating tests in rust/lance/src/dataset/tests; 4 commits adding/updating tests in rust/lance/src/dataset/write/merge\_insert/tests; 41 commits adding/updating tests in rust/lance/src/dataset/optimize/tests; 5 commits adding/updating tests in rust/lance/tests/count\_pushdown; 6 commits adding/updating tests in rust/lance/src/dataset/write/merge\_insert/job; 7 commits adding/updating tests in java/src/test/java/org/lance/ipc; Add benchmarks for v2 file reader performance and schema reconstruction; Added JNI test helper for FFI validation; Added Java SDK test coverage for async scanning, cleanup, compaction, and commit operations; Added Java SDK tests for dataset operations; Added MemTable vs RocksDB KV point-lookup benchmark; Added benchmarks and tests for cache key performance; Added benchmarks for I/O scheduler performance; Added benchmarks for manifest interning, row ID indexing, and system column streaming; Added common benchmarking utilities for encoding strategies; Added comprehensive test coverage for the inverted full-text index; Added historical test fixtures for backward compatibility; Added integration tests for GCS, GooseFS, and TOS object stores; Added integration tests for Java MemWAL bindings; Added integration tests for query execution across data types and index strategies; Added memory usage tests for vector indexing and data writing; Added test resource for Jieba language model; Added test to verify panic location in distance functions; Added test utilities for covering indexes, storage failure injection, and cache serialization; Added tests for InvertedIndexParams serialization and validation; Added tests for Java namespace implementations and dynamic context support; Added tests for MemWAL future depth and cross-generation fencing; Added tests for fragment update results and metadata classes; Added tests for memtable schema evolution and replay behavior; New FineWeb FTS benchmark suite for MemWAL; New LSM vector search benchmarks for MemWAL; New MemTable read performance benchmark; New MemWAL HNSW parity and recall benchmarks; New MemWAL write and replay benchmarks; New benchmark suite for lance-linalg distance and utility kernels; New benchmarking suite for Lance encoding performance; New benchmarks for concurrent appends, COUNT pushdown, and distributed vector builds; New benchmarks for scalar and vector index performance.
Dependencies
Add new benchmarking and testing utility packages
Introduces several new packages to support development and testing workflows: a \lance-memtest\ Rust/Python crate for memory allocation testing utilities, and multiple new benchmark projects (including \dbpedia-openai\, \bigann\, \hd-vila\, \sift\, \wiki\, and \full\_report\) with their respective Python dependencies and configurations. Additionally, adds new Rust crates for Arrow scalar support (\lance-arrow-scalar\), Arrow statistics (\lance-arrow-stats\), and various compression algorithms (\lance-bitpacking\, \fsst\), alongside updated example and core library manifests.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 62 → 64 (+2.0)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 56 → 56 (-0.6)
- Architecture 95 → 94 (-0.7)
- Maturity 79 → 82 (+2.7)
- Readiness 75 → 72 (-3.0)
- Security 61 → 63 (+2.8)
- Accessibility 64 → 76 (+11.9)
- Performance 82 → 94 (+12.2)
Resolved (736)
- BinaryMiniBlockDecompressor::decompress (cognitive 16) (rust/lance-encoding/src/encodings/physical/binary.rs)
- BlobV2FieldRewritePlan::try_new (cognitive 16) (rust/lance/src/dataset/optimize.rs)
- BooleanQueryExec::execute (cognitive 22) (rust/lance/src/io/exec/fts.rs)
- ByteStreamSplitEncoder::compress (cognitive 16) (rust/lance-encoding/src/encodings/physical/byte_stream_split.rs)
- Change coupling: blocking_dataset.rs ↔ WriteDatasetBuilder.java (java/lance-jni/src/blocking_dataset.rs)
- Change coupling: quantizer.rs ↔ subindex.rs (rust/lance-index/src/vector/quantizer.rs)
- Change-coupling hub: binary.rs → encoder.rs, fixed_size_list.rs, list.rs (rust/lance-encoding/src/array_encoding/logical/binary.rs)
- ClassTooLong: CompoundQueryExec (rust/lance/src/io/exec/fts.rs)
- ClassTooLong: MergeInsertJob (rust/lance/src/dataset/write/merge_insert.rs)
- ClassTooLong: Scanner (rust/lance/src/dataset/scanner.rs)
- ClassTooLong: TransactionRebase (rust/lance/src/io/commit/conflict_resolver.rs)
- CompactionOptions::apply_dataset_config (cyclomatic 24) (rust/lance/src/dataset/optimize.rs)
- CompoundQueryExec::execute (cognitive 37) (rust/lance/src/io/exec/fts.rs)
- CompoundQueryExec::execute (cyclomatic 24) (rust/lance/src/io/exec/fts.rs)
- Dataset::commit_existing_index_segments (cognitive 26) (rust/lance/src/index.rs)
- Dataset::commit_existing_index_segments (cyclomatic 22) (rust/lance/src/index.rs)
- Dataset::load_scalar_index (cognitive 21) (rust/lance/src/index.rs)
- Dataset::merge_existing_index_segments (cognitive 32) (rust/lance/src/index.rs)
- Dataset::merge_existing_index_segments (cyclomatic 26) (rust/lance/src/index.rs)
- Dataset::open_vector_index_from_metadata_with_plan (cognitive 20) (rust/lance/src/index.rs)
- …and 716 more
New (1899)
- ANNIvfSubIndexExec::late_search (cognitive 17) (rust/lance/src/io/exec/knn.rs)
- AccumulationQueue::insert_with_pages (cognitive 20) (rust/lance-encoding/src/utils/accumulation.rs)
- AccumulationQueue::insert_with_pages (cyclomatic 16) (rust/lance-encoding/src/utils/accumulation.rs)
- AutoProbePolicy::apply (cognitive 18) (rust/lance/src/io/exec/knn/adaptive_probe.rs)
- BatchRefineExec::refine (cognitive 76) (rust/lance/src/io/exec/knn/refine.rs)
- BatchRefineExec::refine (cyclomatic 22) (rust/lance/src/io/exec/knn/refine.rs)
- BinaryMiniBlockDecompressor::decompress_range (cognitive 26) (rust/lance-encoding/src/encodings/physical/binary.rs)
- BinaryMiniBlockDecompressor::decompress_range (cyclomatic 18) (rust/lance-encoding/src/encodings/physical/binary.rs)
- BlobPreprocessor::preprocess_field (cyclomatic 17) (rust/lance/src/dataset/blob/write.rs)
- BlobPreprocessor::rewrite_sidecars (cognitive 20) (rust/lance/src/dataset/blob/write.rs)
- BlobV2FieldRewritePlan::transform (cognitive 16) (rust/lance/src/dataset/blob/rewrite.rs)
- BlobV2FieldRewritePlan::try_new (cognitive 16) (rust/lance/src/dataset/blob/rewrite.rs)
- BooleanQueryExec::execute (cognitive 22) (rust/lance/src/io/exec/fts/boolean.rs)
- Branches::cleanup_branch_directories (cognitive 75) (rust/lance/src/dataset/refs.rs)
- Branches::cleanup_branch_directories (cyclomatic 35) (rust/lance/src/dataset/refs.rs)
- Branches::collect_local_owned_directories (cognitive 23) (rust/lance/src/dataset/refs.rs)
- BucketScan::pop_candidate (cognitive 25) (rust/lance-index/src/scalar/minhash_lsh/index.rs)
- Change coupling: binary.rs ↔ encoder.rs (rust/lance-encoding/src/array_encoding/logical/binary.rs)
- Change coupling: binary.rs ↔ list.rs (rust/lance-encoding/src/array_encoding/logical/binary.rs)
- Change coupling: index.rs ↔ index_caches.rs (rust/lance/src/index.rs)
- …and 1879 more
Changes since last survey
- 300 commits — 209 feature/other, 91 fixes
By area
- rust/lance — 125 commits
- rust/lance-index — 26 commits
- python/python — 20 commits
- rust/lance-encoding — 18 commits
- (root) — 17 commits
- docs/src — 14 commits
- .github/workflows — 10 commits
- java/src — 9 commits
- rust/lance-io — 8 commits
- rust/lance-core — 7 commits
- rust/lance-datafusion — 6 commits
- rust/lance-table — 5 commits
- rust/compression — 3 commits
- rust/lance-linalg — 3 commits
- test_data/v14.0.2 — 3 commits
- docs/snippets — 2 commits
- java/lance-jni — 2 commits
- protos/filtered_read.proto — 2 commits
- rust/lance-geo — 2 commits
- rust/lance-namespace-impls — 2 commits
Notable commits
- fix: fix!: carry an explicit empty fragment selection through the FilteredReadExec proto (#10863)
- fix: fix(arrow): deduplicate shared buffer allocations (#10848)
- fix: fix(bench): give the scalar index benches the schema their trainers read (#10776)
- fix: fix(blob): read legacy payloads from shallow clone bases (#10918)
- fix: fix(blob): reject Blob v2 nested in Map or FixedSizeList at write (#10612)
- fix: fix(build): merge macOS arm64 startup code into text (#10594)
- fix: fix(ci): use a larger runner for java release builds (#10835)
- fix: fix(cleanup): keep the files of detached versions (#10611)
- fix: fix(cleanup): remove orphaned local staging files (#10591)
- fix: fix(clustering): restore brings back the target's definitions, keeping only the allocation mark (#10629)
- fix: fix(compaction): add shared byte-aware consolidation policy (#10635)
- fix: fix(compaction): avoid materializing blob payload batches (#10696)
- fix: fix(compaction): skip byte-full files by default with a configurable threshold (#10681)
- fix: fix(datafusion): balance deep filters before the lambda rewrite
- fix: fix(dataset): avoid overflow when classifying take addresses (#11016)
- fix: fix(dataset): honor target stores in shallow clones (#10942)
- fix: fix(dataset): preserve row-address schema for empty takes (#11009)
- fix: fix(dataset): reject branch names that exceed storage limits (#10734)
- fix: fix(dataset): reject clone commits into existing targets (#11067)
- fix: fix(dataset): reject deleted branch writes and checkouts (#10736)
- …and 280 more
Architecture
- Containers 0 added · 0 removed · contexts 1 added · 0 removed · edges 0 added · 0 removed
Added bounded contexts (1)
- lance-docs-snippets
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
lance-format/lance was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit e1055501ec36786c26e8db71a8c1118715795eb2 — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.