quickwit-oss/quickwit

Quickwit is a distributed search and analytics engine designed for high-performance indexing and querying of large-scale datasets. It provides a comprehensive ingestion pipeline that supports diverse data sources including Kafka, Pulsar, and S3, alongside native compatibility with Elasticsearch and OpenTelemetry protocols. The system manages data lifecycle through dedicated services for indexing, search, compaction, and garbage collection, all coordinated by a control plane that handles shard management and cluster state. It offers robust deployment options via AWS ECS and Terraform, with a flexible storage layer supporting S3, Azure, and Google Cloud Storage.

65 changes.

This period focused on establishing the foundational architecture for Quickwit v0.9, introducing a new actor-based indexing framework, a dedicated janitor service, and a comprehensive REST API. It also expanded storage capabilities with new cloud backends and caching, while launching the initial React-based UI and OpenTelemetry trace support.

45 changes.

This period focused on restructuring Quickwit into a distributed system by introducing a dedicated Control Plane actor, a new Ingest V2 protocol with capacity-aware routing, and a comprehensive gRPC service layer. The work also established robust internal scaffolding through code generation, Tower middleware for resilience, and a new unified query engine with Elasticsearch compatibility.

35 changes.

This period focused on expanding Quickwit's ecosystem by introducing an Elasticsearch-compatible API layer and OpenTelemetry ingestion support to enhance interoperability. It also significantly broadened cloud infrastructure options with new AWS ECS deployment modules, Google Cloud Storage integration, and a PostgreSQL metastore backend. Concurrently, the codebase underwent substantial internal refactoring, including a modular document mapper, a standardized metrics system, and the introduction of a standalone compactor service.

Quickwit now exposes a Jaeger-compatible API (v1 and v2) to allow users to query OpenTelemetry trace data using standard Jaeger clients and tools. This new \quickwit-jaeger\ crate implements the Jaeger \SpanReaderPlugin\ and \TraceReader\ interfaces, enabling operations such as listing services, retrieving operations, finding trace IDs, and streaming trace data. The service maps OpenTelemetry trace indexes to Jaeger protocol responses, including support for filtering by service name, operation, and tags, and exposes metrics for request duration and error counts to monitor usage.

quickwit/quickwit-jaeger · high confidence

Quickwit now exposes HTTP endpoints for ingesting OpenTelemetry data, allowing users to send logs and traces via standard OTLP HTTP/protobuf requests. The new \otlp\_api\ module registers REST handlers for \/otlp/v1/logs\ and \/otlp/v1/traces\, as well as index-specific routes like \/{index}/otlp/v1/logs\. These endpoints accept \application/x-protobuf\ payloads and support specifying the target index either via the URL path or through optional headers, integrating with the existing gRPC services to handle the actual ingestion.

_quickwit/quickwit-serve/src/otlp\api · high confidence

Users can now follow a new tutorial that demonstrates ingesting Stack Overflow data via Apache Pulsar. This change introduces the necessary configuration files, including an index definition for the Stack Overflow dataset and a Pulsar source configuration, along with a Python script to send messages to the Pulsar topic.

config/tutorials/stackoverflow · high confidence

The search module now includes dedicated SpanId and TraceId types to support distributed tracing within search operations. These types provide serialization and deserialization logic for hex-encoded identifiers, enabling proper integration with OpenTelemetry-style tracing systems. The implementation includes validation for correct byte lengths (8 bytes for span IDs, 16 bytes for trace IDs) and error handling for invalid inputs.

quickwit/quickwit-proto/src/search · high confidence

Developers can now open the project directly in GitHub Codespaces. A new devcontainer configuration provides a pre-configured Rust environment with Node.js 24, Docker-in-Docker support, and the rust-analyzer extension. A post-create script automatically installs the CMake build tool and the Rust nightly toolchain with rustfmt, ensuring the local development environment is ready immediately upon container creation.

.devcontainer · high confidence

The metastore module now includes a README explaining how to start a local PostgreSQL instance for testing the PostgresqlMetastore implementation, along with instructions for running tests with the \postgres\ feature and using sqlx-cli for migrations. A new \build.rs\ script ensures that changes to the PostgreSQL migration directories (\migrations/postgresql\ and \migrations/postgresql\_deferred\) trigger a rebuild, supporting the deferred migration strategy described in the documentation.

quickwit/quickwit-metastore · high confidence

The \quickwit-codegen/example/src\ directory now contains a complete example of a code-generated gRPC service, including the \hello.proto\ definition, a handwritten \HelloError\ enum implementing \ServiceError\ and \GrpcServiceError\ traits for proper gRPC status mapping, and a \HelloImpl\ demonstrating both unary and server-side streaming RPCs. This provides users with a concrete reference for structuring error types and implementing generated service traits within the Quickwit codegen ecosystem.

quickwit/quickwit-codegen/example/src · high confidence

The \quickwit-actors\ crate now includes a \ping\_actor.rs\ example that demonstrates how to implement and interact with actors. This example defines a \PingReceiver\ actor that tracks the number of pings received and a \PingSender\ actor that periodically sends pings to a list of peer receivers via mailboxes. The example illustrates key actor framework concepts such as message handling, self-scheduling, peer management, and graceful shutdown within a \Universe\ context.

quickwit/quickwit-actors/examples · high confidence

The \quickwit-proto\ crate now includes local copies of third-party \.proto\ files, specifically the GoGo Protobuf extensions (\gogoproto\) and Google's well-known types (such as \Any\, \Timestamp\, \Duration\, \Empty\, \FieldMask\, \Struct\, \Type\, \Api\, \Descriptor\, \SourceContext\, and \Value\). These additions provide the necessary schema definitions for the project's internal protocol buffer compilation and API contracts.

quickwit/quickwit-proto/protos/third-party · high confidence

The quickwit-serve crate now embeds build-time information directly into the binary. A new build script captures the current UTC build date, build profile, target platform, and Git commit details (hash, date, and tags) at compile time, making this metadata available to the application for identification and debugging purposes.

quickwit/quickwit-serve · high confidence

The Elasticsearch API model layer now includes dedicated data structures and serialization logic for several endpoints, enabling richer compatibility. This adds support for the \\_bulk\ API with \create\ and \index\ actions and configurable refresh behavior (\false\, \true\, \wait\_for\), the \\_cat/indices\ API with JSON formatting and column filtering, the \\_field\_caps\ API with timestamp-based split pruning and index filtering, and the \\_mapping\ API with timestamp pushdown and dynamic field merging. Additionally, the \\_msearch\ endpoint now supports per-request and global \extra\_filters\, \\_source\_excludes\, and \\_source\_includes\, while the error model standardizes exception types (e.g., \index\_not\_found\_exception\) and maps internal service errors to Elasticsearch-compatible responses.

_quickwit/quickwit-serve/src/elasticsearch\api/model · high confidence

The indexing service now supports an experimental document clustering feature that reorders documents within a split before finalization to group similar log messages together, significantly improving compression ratios for recurring schemas and templates. This is achieved by computing per-document fingerprints based on configurable policies (structure, raw values, or tokenized signatures) and using a \DocIdClusterer\ to sort documents by group size at split finalization. The feature is controlled via \NodeConfig::docs\_clustering\_config\ and can be disabled with the \QW\_DISABLE\_DOCS\_CLUSTERING\ environment variable. While it enhances storage efficiency, it introduces trade-offs including increased CPU usage during ingestion, higher local disk traffic due to temporary uncompressed doc store writes, and additional memory consumption for fingerprint and ID tracking.

quickwit/quickwit-indexing/src · high confidence

The file-backed metastore implementation has been updated to support index templates and wildcard index ID patterns. This change introduces an \IndexIdMatcher\ that allows index IDs to be matched using wildcard patterns (e.g., \index-\*\) and supports negative patterns (e.g., \-index-exclude\) for exclusion. It also adds an \IndexTemplateMatcher\ that manages a list of index templates, matching them to index IDs based on priority and pattern rules. The metastore factory now parses a \polling\_interval\ from the URI fragment to configure periodic metadata polling. Additionally, the manifest format has been updated to version 0.9, which includes support for storing index templates and their associated metadata, while maintaining backward compatibility with the legacy manifest format (0.7/0.8).

_quickwit/quickwit-metastore/src/metastore/file\backed · high confidence

The codegen layer now produces Rust service traits and gRPC client implementations for the cluster, compaction, control plane, developer, indexing, and ingest (router and ingester) domains. These generated clients expose methods such as fetch\_cluster\_state, report\_compaction\_status, control plane index/shard management, get\_debug\_info, apply\_indexing\_plan, and ingest v2, along with support for Gzip/Zstd compression, configurable max message sizes, and test mocks. This provides the internal service contracts and transport wiring that other components depend on.

quickwit/quickwit-proto/src/codegen/quickwit · high confidence

Quickwit now supports automatic reloading of TLS certificates without requiring a service restart. The \quickwit-transport\ crate introduces a \ReloadableCertResolver\ that swaps certificate material at runtime when files change on disk, triggered by a configurable poll interval (\cert\_poll\_interval\) or an immediate reload via \SIGHUP\. This capability applies to both server-side TLS (REST and gRPC) and client-side mTLS identities, ensuring that certificate rotations are picked up by new connections while in-flight sessions remain unaffected.

quickwit/quickwit-transport · high confidence

The ingest v2 pipeline now broadcasts ingester capacity scores and local shard states (including short- and long-term ingestion throughput) to the cluster via Chitchat. Routers use this real-time data to make smarter routing decisions, directing documents to ingesters with available capacity and balancing load based on actual shard throughput rather than static heuristics.

_quickwit/quickwit-ingest/src/ingest\v2 · high confidence

The Quickwit UI is now available as a standalone React application built with Vite, replacing the previous setup. This release introduces a new development environment (running on port 3000 with API proxying) and production build configuration. It integrates Material UI for the interface, with specific build adjustments to suppress 'use client' directive warnings from Material UI components. The package includes standard tooling configurations for TypeScript, Biome formatting/linting, Jest unit testing, and Playwright end-to-end testing.

quickwit/quickwit-ui · high confidence

The Quickwit UI application entry point is established, bootstrapping the React app with a BrowserRouter and the App component. This change introduces core state management providers: EditorProvider for managing Monaco Editor references and LocalStorageProvider for persisting and restoring the last search request across sessions. Additionally, a test suite is added to verify that the main application view correctly renders the Discover, Query editor, and Admin sidebar links.

quickwit/quickwit-ui/src · high confidence

The Quickwit UI introduces a foundational set of utility modules to support the search interface. The \models.ts\ file defines the data structures for document mappings, search requests, and aggregation configurations (including term and histogram aggregations), along with logic to parse aggregation results. The \urls.ts\ module handles the serialization and parsing of search parameters to and from the browser URL, enabling state persistence for queries, timestamps, sorting, and aggregations. The \SearchComponentProps.ts\ file establishes the interface for the search component's inputs, and \theme.ts\ configures the visual styling, including custom fonts and color palettes, for the application.

quickwit/quickwit-ui/src/utils · high confidence

The \elastic\_query\_dsl\ module now supports a comprehensive set of Elasticsearch-compatible query types, including \bool\, \match\, \match\_phrase\, \match\_bool\_prefix\, \match\_phrase\_prefix\, \multi\_match\, \query\_string\, \range\, \exists\, \term\, \terms\, \prefix\, \wildcard\, and \regexp\ queries. This enables users to send standard Elasticsearch-style JSON queries to Quickwit, with specific behaviors such as \minimum\_should\_match\ resolution in boolean queries, field presence checks via \exists\, and phrase matching with slop.

_quickwit/quickwit-query/src/elastic\_query\dsl · high confidence

The Quickwit UI is now available, providing a web-based interface for interacting with Quickwit clusters. Users can perform full-text searches with support for aggregations via the Search view, manage and inspect index configurations (including doc mapping, indexing/search/retention settings, and splits) through the Indexes and Index views, and monitor cluster health and node details in the Cluster and Node Info views. Additionally, an API Playground is included to explore the OpenAPI specification directly in the browser.

quickwit/quickwit-ui/src/views · high confidence

This change establishes the foundational structure for the Quickwit project, introducing essential configuration and documentation files. It adds a \.dockerignore\ to optimize Docker builds, a \.env.example\ for local service configuration (including pinned versions for LocalStack and PostgreSQL), and a \Dockerfile\ that builds the UI with Node 24 and the Rust binary. It also includes \AGENTS.md\ for AI assistant context, \AI\_POLICY.md\ governing AI usage, \CHANGELOG.md\ documenting the 0.9.0 and 0.9.1 releases, \CODE\_OF\_CONDUCT.md\, \CODE\_STYLE.md\ for Rust coding standards, and \CONTRIBUTING.md\ with detailed setup instructions for developers.

(repo-wide) · high confidence

Quickwit now supports offloading leaf search operations to AWS Lambda to enable horizontal scaling. This change adds the \quickwit-lambda-client\ crate, which embeds a pre-built Lambda binary and automatically deploys it to AWS at startup if a matching version is not found. The client handles versioning based on the Quickwit release and binary hash, manages Lambda function lifecycle (including garbage collection of old versions), and implements retry logic with rate-limit backoff for invocations. It also exposes metrics for search duration and payload sizes to monitor Lambda performance.

quickwit/quickwit-lambda-client · high confidence

Added a new \quickwit-lambda-server\ crate that provides the entry point and handler logic for executing Quickwit leaf search tasks on AWS Lambda. This includes a binary entry point (\leaf\_search.rs\) to run the Lambda function, a context module (\context.rs\) that initializes the searcher environment based on Lambda memory constraints, and a handler module (\handler.rs\) that decodes incoming search requests, resolves storage, and executes split-level searches concurrently. This change enables offloading the execution of leaf search work to AWS Lambda functions.

quickwit/quickwit-lambda-server · high confidence

The control plane is now implemented as a dedicated actor that manages two independent sub-controllers: an \IndexingScheduler\ for assigning indexing tasks to indexer nodes and an \IngestController\ for managing shard lifecycle on ingesters. This change introduces a new \ControlPlaneModel\ for state management, a \PhysicalIndexingPlan\ for defining task distribution, and a \CooldownMap\ to prevent excessive shard pruning. It also adds a \Debouncer\ to handle event rate-limiting and exposes comprehensive metrics (e.g., \indexes\_total\, \shards\, \locality\_repairs\_total\) for monitoring the control plane's internal state and operations.

quickwit/quickwit-control-plane/src · high confidence

Users can now store and retrieve data in Google Cloud Storage (GCS) using the \gs://\ URI scheme. This change adds a new \GoogleCloudStorageFactory\ that resolves GCS URIs, parses bucket and prefix information, and configures authentication via a credential chain (supporting default credentials, file-based service accounts, and token exchange). The underlying storage layer is built on OpenDAL, featuring configurable retry policies, concurrency limits, and specific timeouts for control and I/O operations to ensure reliability. A custom memory-efficient copy loop is also included to prevent unbounded memory growth during large uploads.

_quickwit/quickwit-storage/src/opendal\storage · high confidence

The ingest module now includes the proto-generated code and helper implementations for the new Ingest V2 API. This adds the \IngestV2Error\ type with specific variants for shard not found, timeouts, and rate limiting (including \RateLimitingCause\), along with gRPC status code mappings. It also provides utility methods on \FetchMessage\, \FetchPayload\, \IngesterStatus\, \OpenFetchStreamRequest\, \PersistSuccess\, and \TruncateShardsSubrequest\ to extract queue IDs, and adds iterator helpers to \DocBatchV2\ for accessing documents. The router sub-module includes byte-counting helpers for \IngestRequestV2\ and \IngestSubrequest\.

quickwit/quickwit-proto/src/ingest · high confidence

Added a new \IngestSource\ implementation in the indexing pipeline that streams documents from shards via a \MultiFetchStream\ to the ingester pool. This source handles shard assignment, manages fetch retries with exponential backoff, and tracks indexing status (Active, Complete, Error, NotFound, ReachedEof) to ensure documents are correctly fetched, processed, and published with appropriate publish tokens.

quickwit/quickwit-indexing/src/source/ingest · high confidence

The \quickwit-opentelemetry\ crate is initialized with a new \lib.rs\ file that exposes the \otlp\ module, laying the groundwork for OpenTelemetry protocol integration within Quickwit.

quickwit/quickwit-opentelemetry/src · high confidence

Added a new \quickwit-rest-client\ crate that provides an asynchronous Rust client for interacting with the Quickwit REST API. The library includes a builder for configuring connection parameters such as timeouts (connect, general, search, ingest, commit), retry policies, and optional mTLS support (custom CA certificates and client identities). It exposes models for API responses, including a dedicated \SearchResponseRestClient\ for search results and detailed ingest response handling, along with utilities for batching document ingestion from files or stdin.

quickwit/quickwit-rest-client · high confidence

The \quickwit-codegen\ crate is now available to automatically generate Rust service traits, client adapters, and server implementations from \.proto\ files. Users can configure the code generation via a builder API, specifying the input proto files, output directory, and custom result/error types. The generated code supports Tower middlewares, distributed tracing (with configurable field recording), and includes utilities for mocking and testing. This simplifies the creation of gRPC services by handling the boilerplate for serialization, transport, and service definition.

quickwit/quickwit-codegen · high confidence

This change adds a new Terraform module that provisions a Quickwit service on AWS ECS using Fargate. It configures the container with specific environment variables for cluster identity and data storage, exposes REST (7280), gRPC (7281), and gossip ports, and supports mounting volumes for data and secrets. The module allows for configurable IAM policies, including external Postgres metastore URIs via AWS Secrets Manager, and integrates with service discovery.

distribution/ecs/quickwit/service · high confidence

Added the \quickwit-macros\ crate containing the \serde\_multikey\ attribute, which allows structs to define custom serialization and deserialization logic by generating a proxy struct. This enables users to map complex internal field structures to different external representations during serde operations, with support for both serialization and deserialization paths and integration with utoipa schema generation.

quickwit/quickwit-macros · high confidence

The control plane now includes a new \IngestController\ that manages the lifecycle of ingest shards, including creation, closing, and rebalancing across ingesters. This controller implements automatic shard scaling based on ingestion throughput, using both short-term and long-term metrics to decide when to add or remove shards, ensuring efficient resource usage. It also supports zonal-aware shard placement to optimize data locality when ingesters advertise availability zones, and handles shard rebalancing when ingester status changes or nodes are decommissioned.

quickwit/quickwit-control-plane/src/ingest · high confidence

The indexing service now supports cooperative indexing, a mechanism that staggers indexing pipelines so that only a limited number run concurrently. This reduces RAM usage and context switching by controlling the number of active IndexWriters. The feature is enabled via the \enable\_cooperative\_indexing\ configuration option and uses a semaphore to restrict concurrency, with pipelines taking turns based on a hashed target phase to distribute work uniformly over time.

quickwit/quickwit-indexing/src/actors · high confidence

Quickwit now includes a new Janitor service that centralizes index maintenance tasks. This service manages garbage collection of deleted splits, enforces retention policies by marking expired splits for deletion, and processes delete queries. It exposes health checks and metrics (such as GC runs and deleted bytes) to monitor these background operations, providing a unified component for keeping index data clean and compliant with configured policies.

quickwit/quickwit-janitor/src · high confidence

The \quickwit-ingest\ crate now provides a new \IngestApiService\ that manages document ingestion via persistent queues backed by \mrecordlog\. Users can now ingest documents into named queues, with the service enforcing configurable memory and disk usage limits to prevent resource exhaustion. The service exposes gRPC endpoints for ingesting, fetching, and tailing records, and supports explicit commit control (auto, wait-for, or force) to manage when documents become searchable. A notification system ensures that clients waiting for commits are notified when records are committed, and the service tracks ingestion metrics such as total documents and bytes ingested.

quickwit/quickwit-ingest/src · high confidence

The indexing pipeline now supports an in-memory indexing mode, activated via the \QW\_ENABLE\_IN\_MEMORY\_INDEXING\ environment variable. When enabled, \IndexedSplitBuilder\ uses a \RamDirectory\ instead of the default \MmapDirectory\ for the Tantivy index, allowing splits to be built entirely in RAM before being persisted to the scratch directory during finalization. This change is implemented in the \IndexedSplit\ model and its builder within the \quickwit-indexing/src/models\ directory.

quickwit/quickwit-indexing/src/models · high confidence

The \quickwit-cli\ crate now provides a restructured command-line interface organized into five main groups: \run\, \index\, \source\, \split\, and \tool\. This release adds a pre-execution connectivity checklist that validates the environment and reports errors with colored symbols (✔/✖) before running commands. The CLI now supports a \--no-color\ flag to disable ANSI terminal codes and includes a \--yes\ flag for non-interactive execution. New index management capabilities include \index clear\ to delete all splits and reset checkpoints, \index update\ to apply configuration changes, and enhanced \index ingest\ with \--wait\ and \--force\ options to control commit behavior. The interface also exposes jemalloc memory metrics and process I/O metrics on Linux.

quickwit/quickwit-cli · high confidence

The object storage layer now includes dedicated implementations for S3-compatible services and Azure Blob Storage. The S3 backend (\S3CompatibleObjectStorage\) leverages the AWS SDK, supports custom endpoints, respects region and credentials from the storage config, and includes a semaphore to limit concurrent requests. The Azure backend (\AzureBlobStorage\) provides a factory-based resolution using the Azure SDK, supporting account names, access keys, and token credentials. Both backends are registered via their respective factories (\S3CompatibleObjectStorageFactory\ and \AzureBlobStorageFactory\) and share a common multipart policy configuration.

_quickwit/quickwit-storage/src/object\storage · high confidence

Quickwit now caches search splits on disk to reduce remote storage latency and costs. The new \SearchSplitCache\ component manages a local directory of split files, tracking their status (candidate, downloading, or on-disk) via an in-memory table. It automatically downloads splits reported by the search service, enforces configurable size and file-descriptor limits, and evicts older files when capacity is reached. A background task handles concurrent downloads, while the cache wraps the underlying storage to serve split reads from the local disk when available.

_quickwit/quickwit-storage/src/split\cache · high confidence

Quickwit now includes a new, opt-in compactor service that handles merge operations independently of the indexing pipeline. This change introduces a dedicated \CompactorService\ to execute merge tasks and a \CompactionPlanner\ to schedule them, decoupling compaction from indexers. The service supports graceful decommissioning, allowing in-flight merges to complete before shutdown, and exposes detailed metrics for monitoring compaction progress, failures, and available slots.

quickwit/quickwit-compaction · high confidence

The \quickwit-doc-mapper\ crate now includes a new \routing\_expression\ module that defines a Domain Specific Language (DSL) for specifying how documents are routed to partitions. This feature allows users to define routing logic using field names, composite expressions, and modulo operations on hash values, enabling deterministic partitioning based on document content. The implementation uses a consistent SipHash algorithm to ensure stable partition IDs regardless of minor formatting changes in the expression string.

_quickwit/quickwit-doc-mapper/src/routing\expression · high confidence

Quickwit now supports an in-memory cache for storage reads, allowing users to improve performance for frequently accessed data (such as fast fields) by configuring cache policies (LRU, S3-FIFO, or TinyLFU) and capacity limits. The \StorageCache\ trait and \StorageWithCache\ wrapper enable this behavior, while the \QuickwitCache\ router directs requests to specific caches based on file suffixes (e.g., \.fast\). The implementation includes a \ByteRangeCache\ for efficient byte-range retrieval and merging, and exposes detailed metrics for monitoring cache hits, misses, and memory usage.

quickwit/quickwit-storage/src/cache · high confidence

This change introduces the \quickwit-actors\ crate, a new actor framework designed for Quickwit's indexing pipeline. It provides core components including the \Actor\ trait, \Universe\ for managing actor lifecycles, \Mailbox\ for message passing with priority support, and a \Supervisor\ for handling actor health and restarts. The framework includes a scheduler that supports time simulation for testing, a progress tracking system to detect stuck actors, and observability features like \ObservableState\ and health checks. The crate is licensed under the MIT license.

quickwit/quickwit-actors · high confidence

The \quickwit-proto\ crate now includes generated Rust code for the Jaeger v2 storage API (\jaeger.storage.v2.rs\) and the Jaeger v2 API definitions (\jaeger.api\v2.rs\), alongside OpenTelemetry protocol definitions (\opentelemetry.proto.\.rs\). This adds the necessary data structures and client stubs (such as \TraceReaderClient\) to support the newer Jaeger v2 interface, enabling future integration with Jaeger's v2 storage backend.

quickwit/quickwit-proto/src/codegen/jaeger · high confidence

The UI now includes a dedicated HTTP client service (src/services/client.ts) that handles communication with the Quickwit backend, replacing previous ad-hoc fetch calls. This client introduces support for search aggregations, allowing users to request metric and histogram aggregations via the new 'aggs' parameter in the search body. It also standardizes search request construction, including proper handling of sort fields, timestamp ranges, and hit limits (defaulting to 0 for aggregation queries). A corresponding unit test suite validates the URL construction and request body serialization.

quickwit/quickwit-ui/src/services · high confidence

The merge policy module now includes a new \ConstWriteAmplificationMergePolicy\ designed for partitioned indexes (e.g., by tenant) to limit write amplification by targeting a specific number of merge operations rather than a target document count. Additionally, the \NoMergePolicy\ has been renamed to \NopMergePolicy\ to follow Rust naming conventions, and the module exports this no-op policy which immediately marks all splits as mature since no merges occur.

_quickwit/quickwit-indexing/src/merge\policy · high confidence

Added a new example Terraform configuration in distribution/ecs/example that provisions a complete AWS environment for Quickwit on ECS. This includes a VPC with public and private subnets, an optional bastion host for secure access, and an ECR pull-through cache to mitigate Docker Hub rate limits. The module also provides examples for configuring sidecar containers, such as injecting Kafka SSL keys via AWS Secrets Manager, and supports external Postgres metastores and configurable IAM policies for indexers.

distribution/ecs/example · high confidence

This change introduces a new \elasticsearch\_api\ module that provides a compatibility layer for Elasticsearch endpoints. It includes handlers for bulk ingestion (supporting both legacy v1 and new v2 ingest protocols), search, multi-search, scroll, index management (create, delete, mapping), cluster health, and metadata operations (stats, field capabilities, resolve). The implementation uses the Warp framework for routing and includes filters for request validation and decompression, enabling clients to interact with Quickwit using standard Elasticsearch API calls.

_quickwit/quickwit-serve/src/elasticsearch\api · high confidence

Quickwit now exposes a new set of REST endpoints under the \/jaeger/api\ path to support Grafana tracing integration. This change introduces handlers for listing services, retrieving service operations, searching for traces with filters (such as service name, operation, time range, tags, and duration), and fetching specific trace details by ID. The implementation includes model definitions for Jaeger-compatible JSON responses, duration parsing logic, and Swagger/OpenAPI documentation via utoipa.

_quickwit/quickwit-serve/src/jaeger\api · high confidence

Quickwit now supports ingesting data from Amazon Kinesis Data Streams. This change introduces a new \KinesisSource\ that manages shard consumers to poll records, handles checkpointing by sequence number, and supports backfill mode. It includes a connectivity check and uses the AWS SDK for Rust with configurable retry parameters.

quickwit/quickwit-indexing/src/source/kinesis · high confidence

The QueryEditor component has been replaced with a new implementation built on the Monaco editor, providing syntax highlighting, custom language configuration, and field auto-completion (excluding the timestamp field). Additionally, a new AggregationEditor component allows users to configure metric types (count, average, sum, max, min) and aggregation kinds (histogram, term) directly within the UI, enabling visual construction of aggregation queries.

quickwit/quickwit-ui/src/components/QueryEditor · high confidence

The REST API now exposes several new endpoints to help users monitor and debug the Quickwit node. The \/version\ endpoint returns build metadata (commit hash, date, version) and runtime information (CPU count, thread configuration). The \/config\ endpoint exposes the current node configuration with sensitive credentials redacted. Health monitoring is now available via \/health/livez\ (liveness, checking indexer, janitor, compactor, and ingester status) and \/health/startupz\ (startup readiness, checking cluster node readiness). Additionally, the existing \/\ endpoint serves Prometheus metrics, and the \/health\ path has been standardized for these health checks.

quickwit/quickwit-serve/src · high confidence

The \index\_api\ module now exposes a comprehensive set of REST endpoints for managing Quickwit resources. Users can create, update, clear, and delete indexes, as well as manage sources (create, update, toggle, delete, reset checkpoints) and splits (list with filtering, mark for deletion) via dedicated handlers. The API also includes utility endpoints for analyzing text with tokenizers and parsing query strings into ASTs. These changes consolidate index management operations into a structured REST interface, replacing or supplementing previous mechanisms.

_quickwit/quickwit-serve/src/index\api · high confidence

This change introduces a new REST API endpoint for managing delete tasks, allowing users to create and list delete operations via HTTP. The API exposes \GET /{index\_id}/delete-tasks\ to retrieve existing delete tasks and \POST /{index\_id}/delete-tasks\ to submit new delete queries. The POST endpoint accepts a JSON body containing a query string, optional search fields, and optional timestamp ranges (\start\_timestamp\, \end\_timestamp\). Upon submission, the system validates the query against the index's document mapping and search settings before queuing the task for asynchronous execution.

_quickwit/quickwit-serve/src/delete\_task\api · high confidence

This change introduces a new set of REST endpoints for managing index templates, allowing users to create, retrieve, update, and delete templates via the \/templates\ route. The implementation in \quickwit-serve/src/template\_api\ exposes handlers that parse configuration formats, validate template structures, and communicate with the metastore service to persist template definitions.

_quickwit/quickwit-serve/src/template\api · high confidence

A new REST API endpoint has been added at the \/cluster\ path, allowing users to fetch the current cluster state via a GET request. The response returns a \ClusterSnapshot\ containing details about the nodes in the cluster, formatted according to the \format\ query parameter. This change introduces the \cluster\_api\ module within \quickwit-serve\, exposing the \ClusterApi\ for OpenAPI documentation and wiring the handler to return the cluster snapshot.

_quickwit/quickwit-serve/src/cluster\api · high confidence

This change introduces a new Terraform module in \distribution/ecs/quickwit\ that enables users to deploy the full Quickwit architecture (control plane, indexer, searcher, janitor, and metastore) on Amazon ECS. The module provisions an ECS cluster, configures AWS Service Discovery for internal service communication, and manages IAM policies for S3 and ECR access. It supports an external Postgres metastore via SSM Parameter Store or Secrets Manager, or automatically provisions an internal RDS PostgreSQL 16 instance. Users can customize service sizing, networking (VPC, subnets, security groups), and logging, with the indexer service configured with a longer termination grace period to ensure data persistence.

distribution/ecs/quickwit · high confidence

The \quickwit-common\ crate now provides a comprehensive suite of Tower middleware layers to enhance service reliability and monitoring. This includes a \CircuitBreakerLayer\ to reject requests when error thresholds are exceeded, a \LoadShedLayer\ to cap in-flight requests, and an \EventListenerLayer\ to publish successful request events. For observability, a \GrpcMetricsLayer\ tracks request counts, durations, and in-flight status, while an \EstimateRateLayer\ allows services to expose their processing capacity. Additional utilities like \DelayLayer\, \BufferLayer\, and \OneTaskPerCallLayer\ are also introduced to manage request timing, concurrency, and task lifecycle.

quickwit/quickwit-common/src/tower · high confidence

The \quickwit-common\ library now includes several new modules that provide foundational utilities for the rest of the system. This includes an allocation tracker and Jemalloc profiler for heap memory analysis, a \KillSwitch\ mechanism for coordinated process shutdown, and helpers for environment variable parsing with caching. It also adds utilities for CPU detection (supporting Kubernetes limits), random ID generation, and IO throughput limiting with metrics integration.

quickwit/quickwit-common/src · high confidence

This change introduces a new \ConfigValue\ type in \quickwit-config\ that standardizes how configuration fields are resolved, giving environment variables (prefixed with \QW\_\) strict precedence over config file values and defaults. It adds a new \DocsClusteringConfig\ module, allowing users to define fingerprint-based policies to group documents by structure and field values during indexing. Additionally, the \merge\_policy\_config\ module is updated to expose \ConstWriteAmplificationMergePolicyConfig\ and \StableLogMergePolicyConfig\ with explicit maturation periods and finalize options, while \qw\_env\_vars\ centralizes the definition of all Quickwit environment variable keys used across the system.

quickwit/quickwit-config/src · high confidence

The developer API now exposes several new endpoints under the \/api/developer\ path to aid in troubleshooting and performance analysis. Users can fetch cluster-wide debug information via the \/debug\ endpoint, which supports filtering by node ID and role, and automatically redacts sensitive configuration details like database credentials. Dynamic log level changes are now possible through the \/log-level\ endpoint. Additionally, CPU profiling is available via \/pprof/start\ and \/pprof/flamegraph\ (returning an SVG flamegraph), and heap profiling is supported via \/heap-prof/start\ and \/heap-prof/stop\ when the respective features are enabled.

_quickwit/quickwit-serve/src/developer\api · high confidence

Added several new scripts to the \quickwit/scripts\ directory to support development workflows. The \inspect\_split.py\ script allows users to inspect Quickwit \.split\ bundle files, printing per-file sizes and metadata. The \check\_license\_headers.sh\ script validates that source files contain the correct license headers. The \check\_log\_format.sh\ script enforces logging style guidelines for Rust source files. Additionally, \about.hbs\ and \about.toml\ provide the template and configuration for generating third-party license reports via cargo-about, while \dep-tree.py\ generates a visual dependency graph for core Quickwit crates, and \fix\_if\_needed.py\ automates the application of clippy lint fixes.

quickwit/scripts · high confidence

The \quickwit-directories\ crate now provides several new directory implementations to enhance storage handling and performance. \BundleDirectory\ enables reading split bundles as a unified Tantivy directory, while \HotDirectory\ and \CachingDirectory\ introduce static and dynamic caching layers to reduce redundant I/O. \UnionDirectory\ allows stacking multiple directories with shadowing logic, and \DebugProxyDirectory\ instruments read operations for performance analysis. \StorageDirectory\ continues to bridge the \Storage\ trait with the Tantivy \Directory\ API.

quickwit/quickwit-directories · high confidence

A new \quickwit-index-management\ crate has been introduced, consolidating core index lifecycle logic. It provides the \IndexService\ for creating, updating, and deleting indexes, and implements the garbage collection (\run\_garbage\_collect\) logic to detect and remove dangling splits from storage and the metastore based on configurable grace periods. This module exposes metrics for deleted splits and bytes, and handles the coordination between the metastore and storage backends for split deletion.

quickwit/quickwit-index-management · high confidence

A new REST endpoint is available at \/indexing\ that allows users to observe the current status of indexing pipelines. This feature exposes \IndexingServiceCounters\ via a GET request, enabling monitoring of indexing activity through the Quickwit API.

_quickwit/quickwit-serve/src/indexing\api · high confidence

The janitor service now includes dedicated actors to manage data lifecycle: \DeleteTaskService\ orchestrates pipelines that plan and execute document deletions on stale splits; \GarbageCollector\ periodically removes orphaned split files from storage; and \RetentionPolicyExecutor\ schedules and runs index retention policies. These components replace previous ad-hoc cleanup logic with a structured, supervised actor model.

quickwit/quickwit-janitor/src/actors · high confidence

Added two new command-line utilities in the metastore-utils crate: a proxy that intercepts and records gRPC calls to the Quickwit metastore into a JSON-lines file, and a replay tool that reads such a recorded file and re-sends the requests sequentially to a target metastore service. These tools enable users to capture metastore traffic for debugging or testing and to reproduce that traffic against different metastore configurations or versions.

quickwit/quickwit-metastore-utils/src/bin · high confidence

Quickwit now supports indexing files via queue notifications (specifically AWS SQS) using a new \QueueCoordinator\ implementation. This feature enables exactly-once processing by tracking each file as a shard in the metastore, managing message visibility timeouts to prevent duplicates, and handling shard ownership and garbage collection. The module includes an abstract \Queue\ trait, an SQS-specific implementation, and helper components for local state and shared metastore interactions, allowing users to configure file sources that react to S3 object creation events.

_quickwit/quickwit-indexing/src/source/queue\sources · high confidence

A new \quickwit-datetime\ crate has been introduced to centralize date and time handling, providing configurable input and output formats for both parsing and serialization. Users can now parse datetime strings using ISO8601, RFC2822, RFC3339, Unix timestamps (including fractional seconds), or custom Strptime patterns, with support for leading/trailing whitespace. The crate also supports Java-style date format tokens and enforces strict range checks for Unix timestamps to prevent silent wrapping errors. This module is designed to be consumed by \quickwit-query\ and \quickwit-docmapper\ without introducing a dependency on \tantivy\ in \quickwit-common\.

quickwit/quickwit-datetime · high confidence

The search module now includes a new \SearchServiceClient\ abstraction that manages connections to search nodes, supporting both local in-process service calls and remote gRPC communication with timeout and tracing interceptors. A \ClusterClient\ has been added to handle distributed search execution, providing retry logic for leaf searches and document fetches across nodes, as well as key-value replication for scroll contexts. The \LeafSearchCache\ has been introduced to memoize leaf search results per split, using a hashed cache key to reduce memory overhead and improve performance for repeated queries.

quickwit/quickwit-search/src · high confidence

Quickwit introduces several new data ingestion sources. The File source now supports reading from local paths and S3 (via SQS notifications), including handling of gzip-compressed files. New connectors are available for Kafka, Pulsar, and GCP PubSub, enabling ingestion from these streaming platforms. Additionally, the Ingest API source allows the indexing pipeline to consume documents directly from the local ingest queue, supporting both v1 and v2 ingest formats.

quickwit/quickwit-indexing/src/source · high confidence

The storage module introduces several new capabilities: BundleStorage enables reading split bundles by locating and parsing a footer trailer directly from object storage; CountingStorage provides a proxy wrapper to track per-request download bytes and request counts for observability; AsyncDebouncer and DebouncedStorage deduplicate concurrent read requests to the same data source to reduce redundant I/O; FileDescriptorCache manages local file handles with a configurable limit and semaphore-based concurrency control; and the error module defines structured error types including StorageErrorKind and BulkDeleteError to improve error handling and reporting.

quickwit/quickwit-storage/src · high confidence

The search service now exposes a structured API layer in \quickwit-serve/src/search\_api\, introducing a gRPC adapter (\GrpcSearchAdapter\) that implements the \SearchService\ trait for distributed search operations (root/leaf search, scroll, list terms, list fields, etc.) and a REST handler module that defines the HTTP endpoints and request models (including query parsing, sorting DSL, and pagination). This change establishes the concrete wiring for search requests to flow through the gRPC stack and REST interface, supporting features like multi-index patterns, timestamp filtering, and sort-by fields as defined in the request structures.

_quickwit/quickwit-serve/src/search\api · high confidence

Quickwit now supports a PostgreSQL metastore backend, allowing users to store index metadata in a managed relational database instead of object storage. This implementation includes a factory for connection pooling, a migration system with deferred migration handling for multi-node deployments, and a read-only mode for read replicas. It also introduces specific error handling for PostgreSQL constraints (such as foreign key and unique violations) and exposes database connection pool metrics for monitoring.

quickwit/quickwit-metastore/src/metastore/postgres · high confidence

The query parsing and execution engine has been redesigned, introducing a new \QueryAst\ enum that supports a wider variety of query types including \Bool\, \PhrasePrefix\, \FieldPresence\, \CalcField\ (calculated fields), and \Cache\ (predicate caching). This change adds support for boolean logic with \must\, \must\_not\, \should\, and \filter\ clauses, phrase prefix searches, and the ability to query for field existence. It also introduces a predicate cache mechanism to optimize repeated queries by caching results, and adds support for calculated field expressions using Tantivy's JIT expressions.

_quickwit/quickwit-query/src/query\ast · high confidence

The Quickwit UI now supports visualizing search aggregations alongside standard document results. When a search query includes aggregations, the interface renders them as interactive charts (line charts for histograms, bar charts for term aggregations) instead of a document table. For standard document results, the new table layout correctly extracts and displays the index's configured timestamp field, properly formatting Unix timestamps across seconds, milliseconds, microseconds, and nanoseconds. This change introduces new components (AggregationResult, ResultTable, Row) to handle these distinct display modes and ensures accurate time representation in search results.

quickwit/quickwit-ui/src/components/SearchResult · high confidence

This change introduces a new set of Protocol Buffers in the \quickwit-proto\ crate that define the internal gRPC contracts for Quickwit's distributed architecture. The new files establish the \ClusterService\ for fetching cluster state via Chitchat, the \ControlPlaneService\ for managing indexes, sources, and shard lifecycle, and the \IngesterService\ for document persistence and shard management. Additionally, it defines the \IngestRouterService\ for document ingestion, the \IndexingService\ for applying indexing plans, and the \MetastoreService\ for index and split metadata. Common types such as \IndexUid\, \ShardId\, and \Position\ are also standardized across these services to ensure consistent communication between the control plane, ingesters, and indexers.

quickwit/quickwit-proto/protos/quickwit · high confidence

A change to existing behaviour in quickwit/quickwit-query/src/tokenizers — 19 commits, 5 files.

quickwit/quickwit-query/src/tokenizers · medium confidence · unverified

A new startup script has been added to automatically configure the LocalStack environment before tests run. This ensures that the required S3 buckets (quickwit-dev and quickwit-integration-tests) and the Kinesis stream (quickwit-dev-stream) are present, reducing manual setup and improving the reliability of the test environment.

.localstack · high confidence

The cluster module now defines a \ClusterError\ enum that maps internal, timeout, too-many-requests, and unavailable states to specific service error codes. This enables consistent gRPC status code mapping and integrates with the load-shedding mechanism, where load-shed events are represented as \TooManyRequests\ errors. Internal errors are also rate-limited to prevent log flooding.

quickwit/quickwit-proto/src/cluster · high confidence

The cluster module has been refactored to replace pure UDP gossip with a hybrid approach that uses gRPC to fetch and synchronize cluster state. This change introduces a new \ClusterService\ gRPC endpoint and a \grpc\_gossip\ mechanism that periodically pulls the full cluster state from peer nodes, ensuring faster and more reliable catch-up for new or lagging nodes. The \ClusterChangeStream\ now emits structured \Add\, \Update\, and \Remove\ events based on these synchronized states, and metrics for gossip traffic (bytes and messages sent/received) are now exposed.

quickwit/quickwit-cluster · high confidence

The \quickwit-proto\ crate now defines explicit error types (\ControlPlaneError\ and \DeveloperError\) for the control plane and developer services. These types implement the \ServiceError\ and \GrpcServiceError\ traits, ensuring that internal errors, timeouts, rate-limiting (too many requests), and unavailability conditions are consistently mapped to gRPC status codes and error codes. This change provides a uniform error reporting mechanism for these specific service interfaces, replacing ad-hoc error handling with a structured approach that supports load shedding and standardized client-side error interpretation.

_quickwit/quickwit-proto/src/control\plane · high confidence

The metastore now includes a \ControlPlaneMetastore\ wrapper that proxies index and source lifecycle operations (create, update, delete, toggle) to the control plane service. This ensures the control plane maintains an accurate, real-time view of metastore state for these specific operations, while read-only and split-related operations continue to interact directly with the underlying metastore implementation.

quickwit/quickwit-metastore/src/metastore · high confidence

The document mapping logic in \quickwit-doc-mapper\ has been completely rewritten to replace the previous \DocMapper\ trait with a concrete \DocMapper\ struct and a \DocMapperBuilder\. This change introduces a new internal architecture where field mappings are represented by a \MappingNode\ tree and specific type options (such as \QuickwitDateTimeOptions\, \QuickwitNumericOptions\, and \QuickwitTextOptions\) are defined in dedicated modules. The refactoring also adds explicit field presence tracking via \field\_presence.rs\ to support existence queries and introduces a \WarmupInfo\ structure to optimize index warmup by identifying required terms and fast fields.

_quickwit/quickwit-doc-mapper/src/doc\mapper · high confidence

The example project now includes a build script that utilizes the new Codegen builder pattern to generate Rust code from a protobuf definition. This change illustrates how users can configure the code generator by specifying input proto files, output directories, and custom type paths for results and errors, while also enabling the generation of extra service methods and RPC name implementations.

quickwit/quickwit-codegen/example · high confidence

The file-backed metastore now manages shards per source, including opening, acquiring, and pruning them with publish-token-based conflict resolution to ensure correct ingestion state. This change introduces a new serialization format (version 0.9) for the file-backed index, maintaining backward compatibility with versions 0.7 and 0.8 by automatically migrating shard data and populating default document mapping UIDs during deserialization.

_quickwit/quickwit-metastore/src/metastore/file\_backed/file\_backed\index · high confidence

The code-generated service client and server adapters now explicitly configure gRPC transport settings, including support for Gzip and Zstd compression and configurable maximum message sizes. This allows users to optimize network performance and handle larger payloads when using the generated clients and servers, with compression applied via the \accept\_compressed\ and \send\_compressed\ methods on the tonic service builder.

quickwit/quickwit-codegen/example/src/codegen · high confidence

The AWS client library now uses a custom in-memory DNS resolver to cache address lookups, reducing latency and blocking calls when connecting to S3. Additionally, retry behavior has been refined: S3 dispatch failures are now retried when they involve transient or throttling errors, and the system respects the \x-amz-retry-after\ header to honor server-suggested backoff delays.

quickwit/quickwit-aws · high confidence

The REST ingest API response structure has been updated to provide granular feedback on document ingestion. Users can now see the exact number of documents successfully ingested versus those rejected due to parsing errors. Additionally, when the \detailed\_response\ query parameter is enabled, the API returns a list of specific parse failures, including the problematic document content and the reason for rejection, which was previously unavailable in the ingest v1 path.

_quickwit/quickwit-serve/src/ingest\api · high confidence

The compaction module now defines a \CompactionError\ enum and implements standard error traits (including \ServiceError\ and \GrpcServiceError\). This establishes the error contract for the new opt-in compactor service, allowing it to properly report timeouts, internal errors, and load-shedding conditions via gRPC status codes.

quickwit/quickwit-proto/src/compaction · high confidence

The \quickwit-proto\ types module now defines dedicated, strongly-typed identifier structs for key domain entities: \IndexUid\ (combining an index ID with a ULID incarnation ID to distinguish deleted/recreated indexes), \DocUid\ and \DocMappingUid\ (ULID-based unique identifiers for documents and their mappings), \PipelineUid\ (ULID-based identifier for indexing pipelines/tasks), \ShardId\ (globally unique shard identifier), \SplitId\ (opaque split identifier with optional random S3 prefixing), and \Position\ (offset/EOF tracking for sources). These types replace previous string-based or ambiguous ID representations, providing consistent serialization (JSON, Protobuf, PostgreSQL) and ensuring type safety across the indexing and search pipelines.

quickwit/quickwit-proto/src/types · high confidence

Users can now define reusable index templates that automatically generate unique document mapping UIDs when applied to new indices, ensuring isolation between instances. This change introduces a versioned configuration format (V0.8) for index templates, supporting glob-based index ID patterns, priority-based selection, and optional retention policies, while maintaining backward compatibility with previous template versions through serialization aliases.

_quickwit/quickwit-config/src/index\template · high confidence

The metastore now persists index metadata using a new versioned serialization format (version 0.9), which internally maps to the existing in-memory \IndexMetadata\ structure. This change ensures backward compatibility by allowing the system to read metadata stored in previous versions (0.7 and 0.8) seamlessly. For users, this means that existing index data remains accessible and valid during upgrades, while new metadata is written in the updated format to support future schema evolutions without breaking existing indices.

_quickwit/quickwit-metastore/src/metastore/index\metadata · high confidence

Quickwit now caches freshly indexed splits locally on the indexer node before they are merged. This \IndexingSplitCache\ stores splits in a local directory and serves them to the merger pipeline, preventing the need to re-download splits from remote storage for every merge operation. The cache is bounded by configurable quotas for the maximum number of splits and total disk size, and it automatically evicts older splits (older than 48 hours) to manage disk usage. This change significantly reduces disk I/O and network bandwidth consumption during the indexing and merging phases.

_quickwit/quickwit-indexing/src/split\store · high confidence

A new \ClusterConfig\ struct has been added to the \quickwit-config\ crate to define core cluster-level settings. This configuration includes the cluster identifier, a flag to enable automatic index creation, the default URI for index storage, a configurable shard throughput limit, and a shard scale-up factor. These parameters are now available for use in controlling indexing behavior and resource allocation within the cluster.

_quickwit/quickwit-config/src/cluster\config · high confidence

The quickwit-ingest crate now uses a build script to automatically generate Rust code from the ingest service protocol buffer definition (src/ingest\_service.proto). This change replaces manual implementation with generated code, configuring output to the src/codegen/ directory and integrating with the project's existing error handling and result types.

quickwit/quickwit-ingest · high confidence

The metastore-utils crate now provides a unified \GrpcRequest\ enum and \GrpcCall\ wrapper to encapsulate metastore API calls. This change introduces a tagged union that aggregates all metastore request types (such as \CreateIndexRequest\, \StageSplitsRequest\, and \DeleteIndexRequest\) into a single serializable structure, facilitating consistent handling and serialization of gRPC interactions within the metastore proxy layer.

quickwit/quickwit-metastore-utils/src · high confidence

The ingest service now exposes a formally defined API contract via the generated \ingest\_service.rs\ file, which establishes the structure for queue management (create, drop, list), document ingestion with configurable commit types (Auto, WaitFor, Force), and record retrieval (fetch, tail). This change introduces the \IngestService\ trait and associated request/response types, providing a clear interface for clients to interact with the ingest pipeline while clarifying delivery semantics (at-least-once vs. at-most-once) and payload limits.

quickwit/quickwit-ingest/src/codegen · high confidence

The control plane now maintains an in-memory model (\ControlPlaneModel\) that stays synchronized with the metastore, replacing the previous stateless approach. This model tracks indexes, sources, and shards, loading its full state from the metastore on startup. A new \ShardTable\ component manages shard lifecycle and locations, enforcing specific rate limits for scaling operations (5 scale-ups and 1 scale-down per minute per source) and tracking ingestion rates to support auto-scaling decisions.

quickwit/quickwit-control-plane/src/model · high confidence

The metastore module has been restructured to introduce a new checkpointing system for tracking indexing progress per source and partition, and to version split metadata (v0.8) with backward compatibility for older formats. This includes new types like \PartitionId\, \SourceCheckpoint\, and \IndexCheckpoint\ to manage indexing state, and a \MetastoreResolver\ to handle backend configuration and resolution. The split metadata now includes additional fields like \doc\_mapping\_uid\ and \num\_merge\_ops\ for better merge control and tracking.

quickwit/quickwit-metastore/src · high confidence

The metastore protocol definitions in \quickwit-proto/src/metastore\ have been restructured to use code-generated service definitions (via \include!\ for the generated Rust code and descriptor set). This change introduces a new \events.rs\ module defining specific event structs (\DeleteIndexEvent\, \AddSourceEvent\, \ToggleSourceEvent\, \DeleteSourceEvent\) and implements the \Event\ trait for request types. It also adds \SplitRecoveryMetadata\ serialization/deserialization logic with a custom container header for disaster recovery, and defines an \EntityKind\ enum to list metastore-managed object types. The \MetastoreError\ enum is also present in the new module structure.

quickwit/quickwit-proto/src/metastore · high confidence

The OpenTelemetry Collector protocol definitions (logs, metrics, and traces) have been moved into the \quickwit-proto\ crate. This change introduces generated Rust structs and gRPC client/server implementations for the OTLP collector endpoints, enabling Quickwit to natively receive telemetry data via the standard OpenTelemetry Collector protocol.

quickwit/quickwit-proto/src/codegen/opentelemetry · high confidence

The quickwit-metrics library has been rewritten to use the \metrics-rs\ ecosystem, introducing a new set of macros (\counter!\, \gauge!\, \histogram!\) and types (\LazyCounter\, \LazyGauge\, \LazyHistogram\) for declaring metrics. Users can now configure a custom system prefix and separator for metric names (e.g., changing \quickwit\_http\_requests\_total\ to \myapp.http.requests\_total\) and apply global labels via an environment variable, which are automatically appended to all metrics. The implementation includes a global deduplication cache using \DashMap\ and thread-local caching for performance, along with an example demonstrating Prometheus integration.

quickwit/quickwit-metrics · high confidence

Quickwit has replaced its previous metrics implementation with the \metrics-rs\ ecosystem. This change introduces a new \quickwit-metrics-inventory\ crate that uses a build script and linker flags to automatically discover and register all metric definitions across the workspace, and updates the Prometheus telemetry exporter to configure histogram buckets and manage the metrics recorder using the new standard library. Users benefit from a more robust and standardized metrics collection system that ensures all metrics are properly exposed via the Prometheus endpoint.

quickwit/quickwit-metrics-inventory, quickwit/quickwit-telemetry-exporters/src/prometheus · high confidence

The telemetry exporters module has been rewritten to use the OpenTelemetry SDK for traces, logs, and metrics, replacing the previous implementation. This change introduces a new \TelemetryHandle\ for managing provider lifecycle and initializes a global metrics recorder via the \metrics-rs\ crate, which fans out data to both Prometheus and OTLP exporters when configured. Additionally, the logging layer now supports a new Datadog-specific JSON format (\ddg\) that promotes event fields to top-level attributes, and metrics initialization now respects a global labels environment variable.

quickwit/quickwit-telemetry-exporters/src · high confidence

The OpenTelemetry module now defines the default index configurations for the \otel-logs-v0\_9\ and \otel-traces-v0\_9\ indices. The logs index maps fields such as \timestamp\_nanos\, \service\_name\, \severity\_number\, and \body\ (as JSON), while the traces index maps \trace\_id\, \span\_id\, \span\_name\, \span\_start\_timestamp\_nanos\, and \span\_status\. Both configurations use version 0.8 of the index format and set a 5-second commit timeout.

quickwit/quickwit-opentelemetry/src/otlp · high confidence

The \quickwit-proto\ crate now includes a \build.rs\ script that automatically generates Rust code from \.proto\ definitions for the cluster, compaction, control plane, developer, indexing, metastore, and ingest services. This change introduces specific serialization behaviors, such as zstd compression for index metadata responses and custom serde aliases for delete queries, ensuring the generated code aligns with the current API contract.

quickwit/quickwit-proto · high confidence

The \quickwit-doc-mapper\ crate has been restructured to introduce a new \DocMapping\ configuration model that explicitly defines how unmapped fields are handled via \ModeType\ (Lenient, Strict, or Dynamic) and includes new options to record field presence for \exists\ queries and document size for fast-field lookups. This change replaces the previous implicit mapping logic with a structured schema that supports dynamic field capture, custom tokenizers, and partitioning keys, while the query builder now utilizes a visitor pattern to accurately collect required fast fields and term dictionaries for split warmup, improving query execution efficiency and error handling for document parsing.

quickwit/quickwit-doc-mapper/src · high confidence

The \quickwit-proto\ crate now includes a new error handling module that serializes service errors into a dedicated gRPC header (\qw-error-bin\) for transparent propagation between clients and servers, replacing the previous limitation to standard status codes and messages. Additionally, a new \getters\ module provides standardized, macro-generated accessor methods for key fields (such as \index\_uid\, \doc\_mapping\_uid\, and \pipeline\_uid\) across control plane, indexing, ingest, and metastore API messages, ensuring consistent access to required fields.

quickwit/quickwit-proto/src · high confidence

The \quickwit-config\ crate now includes a dedicated \index\_config\ module that centralizes index configuration structures and serialization logic. This introduces a new configuration format version (v0.9) while maintaining backward compatibility with v0.8 and v0.7. For users, this brings explicit configuration options for ingestion, including a new \min\_shards\ parameter to control the minimum number of shards and a \validate\_docs\ flag (defaulting to true) to toggle document validation against the doc mapping during ingestion. The module also handles the serialization of \IndexConfig\ objects, ensuring that updates validate that immutable fields like \index\_id\ and \index\_uri\ remain unchanged.

_quickwit/quickwit-config/src/index\config · high confidence

The indexing module now defines a dedicated \IndexingError\ enum with specific variants for internal errors, metastore issues, timeouts, and rate limiting, implementing standard gRPC service error interfaces to ensure consistent error reporting. Additionally, new \IndexingPipelineId\ and \MergePipelineId\ structs are introduced to uniquely identify indexing and merge pipelines by node, index, source, and pipeline UID, providing a structured way to track pipeline lifecycle and metrics.

quickwit/quickwit-proto/src/indexing · high confidence

The indexing scheduler has been replaced with a new implementation that supports locality-aware scheduling (placing shards in the same availability zone as their data when enabled via QW\_ENABLE\_LOCALITY\_AWARE\_SCHEDULING) and variable shard load (controlled by QW\_ENABLE\_VARIABLE\_SHARD\_LOAD). The new scheduler also includes a change tracker for rebuild notifications and integrates with the indexer pool to manage shard assignments more efficiently.

_quickwit/quickwit-control-plane/src/indexing\scheduler · high confidence

The indexing scheduler now uses a new heuristic-based scheduling algorithm to assign shards to indexers, documented in the new \scheduling/README.md\. The logic prioritizes stability by avoiding unnecessary task movement, balancing load across nodes, and respecting availability zone locality to minimize cross-AZ traffic. It includes a four-phase assignment process (removing extraneous shards, enforcing capacity limits, greedy assignment with locality/affinity preferences, and optimization) and a separate optimization pass that improves pipeline density and locality over time, gated by plan stability and a cooldown period to prevent disruptive pipeline resets.

_quickwit/quickwit-control-plane/src/indexing\scheduler/scheduling · high confidence

The list fields search capability has been replaced with a new, optimized implementation located in \quickwit-search/src/list\fields\. This change introduces a dedicated caching layer (\ListFieldsCache\) to store serialized field metadata, reducing redundant storage reads. It adds support for filtering fields using wildcard patterns (e.g., \prefix\\ or \\*suffix\) via the \FieldPatterns\ module. The new logic enforces a default limit of 10,000 fields to prevent excessive response sizes, retaining the most frequently occurring fields when the limit is exceeded. The implementation splits the operation into root and leaf stages, handling distributed request routing and local field metadata merging with bounded concurrency.

_quickwit/quickwit-search/src/list\fields · high confidence

The \quickwit-query\ crate has been introduced to handle query parsing and aggregation result serialization, providing a layer partially compatible with the Elasticsearch/OpenSearch Query DSL. This change adds support for composite aggregations alongside existing bucket and metric types (such as range, histogram, terms, and multi-terms) in the aggregation results. It also introduces stricter validation for floating-point values by rejecting NaN in boost fields via the \NotNaNf32\ type, and enhances user input interpretation to support base16 (hex) decoding for byte fields in addition to base64.

quickwit/quickwit-query/src · high confidence

The source configuration system has been refactored to use a new \SourceConfig\ struct with explicit versioned serialization (supporting config versions 0.7 and 0.9, with 0.8 as an alias). This change introduces support for multiple input formats, including JSON, OTLP logs (JSON and Protobuf), and OTLP traces, allowing users to ingest structured telemetry data directly. The configuration now enforces stricter validation, such as restricting multiple indexing pipelines to only Kafka and GCP PubSub sources, and disabling VRL transforms for OTLP input formats. Additionally, reserved source IDs are defined for CLI ingestion, the ingest API, and native ingest v2, ensuring internal consistency.

_quickwit/quickwit-config/src/source\config · high confidence

The node configuration schema has been reorganized to improve clarity and modularity. Configuration options are now grouped into specific structs: \RestConfig\ for HTTP/REST settings (including CORS, extra headers, and TLS), \GrpcConfig\ for gRPC settings (including message size limits, TLS, keep-alive, and connection age), \HealthConfig\ for the dedicated health-check server, and \TlsConfig\ for certificate paths and hot-reload intervals. This change introduces new configuration keys under these sections (e.g., \rest.\\, \grpc.\\, \health.\*\) and deprecates or moves previously flat node-level settings into their respective domains.

_quickwit/quickwit-config/src/node\config · high confidence

The OTLP telemetry exporters in the \quickwit-telemetry-exporters\ crate have been restructured to include a common retry policy (5 retries, exponential backoff up to 30 seconds) for logs, metrics, and traces, improving resilience against transient network failures. Additionally, the exporters now support configurable OTLP protocols (gRPC, HTTP/Protobuf, HTTP/JSON) via environment variables, allowing users to select the transport protocol for each signal type independently.

quickwit/quickwit-telemetry-exporters/src/otlp · high confidence

This change introduces a new deferred PostgreSQL migration (migration 29) that creates a partial index on the \splits\ table to support the compaction planner. The index, \splits\_maturity\_timestamp\_idx\, is optimized for the planner's specific query pattern—scanning for 'Published' splits that are not yet mature—by using a partial predicate and a composite key on \(maturity\_timestamp, split\_id)\. This allows the database to satisfy the filter and ordering requirements without an extra sort step, improving the efficiency of the opt-in compactor service's merge operations. The migration uses \CREATE INDEX CONCURRENTLY\ to avoid blocking writes during index creation and is executed in a background task after the service is ready, as per the deferred migration strategy.

_quickwit/quickwit-metastore/migrations/postgresql\deferred · high confidence

The PostgreSQL metastore database schema has been updated with migrations 1 through 28 to support new indexing capabilities and metrics-specific data storage. Key changes include expanding \index\_uid\ and \index\_id\ column lengths to 282 and 255 characters respectively, adding a \shards\ table with a \SHARD\_STATE\ enum, and introducing \index\_templates\ for auto-creating indices. The schema now supports split maturity filtering via a \maturity\_timestamp\ column and tracks split size with a generated \split\_size\_bytes\ column. New tables \metrics\_splits\ and \sketch\_splits\ have been added to store low-cardinality tag data and DDSketch metrics with optimized GIN indexes for efficient pruning. Additionally, compaction metadata columns (e.g., \window\_start\, \sort\_fields\) were added to \metrics\_splits\, and various triggers for automatic timestamp updates were removed in favor of helper functions or explicit application logic.

quickwit/quickwit-metastore/migrations/postgresql · high confidence

The Quickwit UI has been completely rewritten to provide a modern interface for managing indexes and running queries. This update introduces a new sidebar for navigation (Discover, Admin, API Playground), an index selection autocomplete, and a detailed index summary view that now displays uncompressed document sizes. The query editor now features a dedicated action bar with tabs to switch between Search and Aggregation modes, alongside a new TimeRangeSelect component for filtering by time. Additional UI improvements include an API URL footer for easy copying, a JSON viewer for documents, and better error handling for missing search nodes.

quickwit/quickwit-ui/src/components · high confidence

The query processing pipeline has been redesigned to use a schema-agnostic \QueryAST\ (Query Abstract Syntax Tree) that normalizes inputs from the Elasticsearch-compatible API, Quickwit API, and search stream API before applying them to splits. This change introduces a \UserInputQuery\ node that is resolved against the current DocMapper at the root search level to validate field usage (such as range queries on fast fields) before being dispatched to leaf searches, ensuring better error reporting and handling of schema changes across legacy splits. Additionally, a new \CodeTokenizer\ has been added to improve tokenization of code-like text, accompanied by benchmarks comparing its performance against the standard regex tokenizer.

quickwit/quickwit-query · high confidence

The thread pool implementation in \quickwit-common\ has been updated to support task prioritization, allowing the search system to ensure that latency-sensitive operations like final result merging are executed before less urgent background work such as split searches. This change introduces a \ThreadPoolWithPriority\ executor that queues tasks and schedules high-priority items ahead of normal-priority ones, improving search responsiveness under load.

_quickwit/quickwit-common/src/thread\pool · high confidence

The Quickwit project now enforces standardized Rust formatting and linting rules via new \rustfmt.toml\ and \clippy.toml\ configurations, including disallowing specific \Option\ methods and \Path::exists\. It introduces automated license compliance checks using \cargo-deny\ (configured in \deny.toml\) and \cargo-about\ (generating \dependency-licenses.html\), alongside a \NOTICE\ file and license header template. Build infrastructure is updated with a \Cross.toml\ for cross-compilation support, a \Makefile\ for common tasks, and the Rust toolchain is set to version 1.96.

quickwit · high confidence

The build system now uses updated Docker images for cross-compiling to AArch64 and x86\_64 architectures. For GNU targets, the base images have been upgraded to Ubuntu 20.04 (focal) to provide libclang-10, which satisfies the requirements of bindgen \>= 0.72 used by zstd-sys. The toolchains now explicitly use GCC 10 to avoid compiler bugs in GCC 9.4 that affect aws-lc. Additionally, the AArch64 musl image has been updated to build static OpenSSL and zlib, ensuring consistent static linking for musl-based releases.

build · high confidence

Added REST API test suite with YAML-based scenario definitions; Added actor message throughput benchmarks; Added benchmark data for JSON parsing and routing expression evaluation; Added benchmark datasets for indexing performance testing; Added benchmarks for MRecord v1 protobuf encoding; Added end-to-end tests for Quickwit UI navigation; Added failpoint instrumentation for indexing pipeline resilience testing; Added integration tests for index update scenarios; Added mock implementations for testing external UI libraries; Added test assets for TLS certificate hot-reload; Added test infrastructure for GCS TLS certificate regeneration; Expanded metastore test coverage for core APIs; Integration tests for Azure, GCS, and S3 storage backends; New backward compatibility test framework for metastore resources; New cluster sandbox test utilities for integration testing; New integration test crate for Quickwit; New integration test suite for Quickwit cluster services.

The Quickwit Rust workspace has been restructured to use a centralized dependency management strategy, with all crate versions defined in the root \Cargo.toml\ and resolved via \Cargo.lock\. This update includes a major version bump of the Rust edition to 2024, sets the workspace version to 0.9.1, and introduces significant dependency upgrades such as \clap\ to v4.6, \hyper\ to v1.9, \tokio\ to v1.40, and \chitchat\ to v0.13.0. Additionally, CI workflows now utilize a pinned \requirements.txt\ for repeatable Python builds, and the \quickwit-cli\ tests have been updated to use a \Pipfile\ for managing AWS CLI local dependencies.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 59 → 60 (+1.6)
• Rubric changed (rubric-2026.09.8 → rubric-2026.09.17) — scores are not directly comparable.

• Code Health 87 → 88 (+0.6)
• Architecture 99 → 93 (-6.0)
• Maturity 91 → 78 (-13.3)
• Readiness 63 → 67 (+3.9)
• Security 42 → 52 (+10.3)
• Domain Modelling 89 → 86 (-2.4)
• Event Sourcing 100 → 100 (+0.0)
• Accessibility 71 → 71 (-0.2)
• Performance 60 (new)

• ArrowMetricsBatchBuilder::finish (cognitive 21) (quickwit/quickwit-opentelemetry/src/otlp/arrow_metrics.rs)
• ArrowSketchBatchBuilder::finish (cognitive 18) (quickwit/quickwit-parquet-engine/src/ingest/arrow_sketches.rs)
• Change coupling: mod.rs ↔ scheduling_logic_model.rs (quickwit/quickwit-control-plane/src/indexing_scheduler/mod.rs)
• CompactionModel::actions (cognitive 47) (quickwit/quickwit-dst/src/models/time_windowed_compaction.rs)
• CompactionModel::actions (cyclomatic 21) (quickwit/quickwit-dst/src/models/time_windowed_compaction.rs)
• DataModelModel::actions (cognitive 21) (quickwit/quickwit-dst/src/models/parquet_data_model.rs)
• Documentation: no installation or build instructions (README.md)
• Documentation: no project overview (README.md)
• Documentation: written for insiders (docs/internals/adr/gaps/README.md)
• Duplicated block (10 lines × 2) (quickwit/quickwit-dst/src/models/time_windowed_compaction.rs)
• Duplicated block (10 lines × 2) (quickwit/quickwit-indexing/src/actors/indexing_pipeline.rs)
• Duplicated block (10 lines × 2) (quickwit/quickwit-indexing/src/actors/merge_planner.rs)
• Duplicated block (10 lines × 2) (quickwit/quickwit-opentelemetry/src/otlp/logs.rs)
• Duplicated block (10 lines × 2) (quickwit/quickwit-parquet-engine/src/sort_fields/parser.rs)
• Duplicated block (11 lines × 2) (quickwit/quickwit-doc-mapper/src/query_builder.rs)
• Duplicated block (11 lines × 2) (quickwit/quickwit-indexing/src/actors/doc_processor.rs)
• Duplicated block (11 lines × 2) (quickwit/quickwit-indexing/src/actors/indexing_pipeline.rs)
• Duplicated block (11 lines × 2) (quickwit/quickwit-indexing/src/actors/merge_planner.rs)
• Duplicated block (11 lines × 2) (quickwit/quickwit-indexing/src/merge_policy/const_write_amplification.rs)
• Duplicated block (11 lines × 2) (quickwit/quickwit-janitor/src/actors/garbage_collector.rs)
• …and 186 more

• Coverage not measured — JavaScript/TypeScript suite
• Documentation: contradicts the code (quickwit/quickwit-metastore/migrations/postgresql_deferred/README.md)
• Duplicate intent: Same as above, Mailbox exposes both ask and ask_for_res with identical signatures. This inconsistency propagates from ActorContext.
• Duplicate intent: ask and ask_for_res have identical signatures. The suffix _for_res suggests a specific return type or behavior (perhaps returning a Result directly vs wrapping it), but the signature Result is generic enough to cover both. This creates ambiguity for the caller.
• Duplicate intent: kill and quit appear to perform the same action (terminating the actor) with identical signatures and return types. In actor models, 'kill' usually implies forceful termination while 'quit' might be graceful, but without distinct behavioral documentation or signature differences (e.g., returning a specific error for kill vs success for quit), they are confusingly redundant.
• Duplicate intent: search_index and search_index_cli have identical signatures and likely perform the same CLI execution logic. The _cli suffix is redundant if both are in the CLI module.
• Duplicated block (19 lines × 2) (quickwit/quickwit-indexing/src/actors/indexing_pipeline.rs)
• Duplicated block (6 lines × 2) (quickwit/quickwit-indexing/src/source/kafka_source.rs)
• Duplicated block (8 lines × 2) (quickwit/quickwit-query/src/query_ast/visitor.rs)
• FileTooLong: indexing_scheduler/mod.rs (quickwit/quickwit-control-plane/src/indexing_scheduler/mod.rs)
• FileTooLong: scheduling/mod.rs (quickwit/quickwit-control-plane/src/indexing_scheduler/scheduling/mod.rs)
• High CVE: [GHSA redacted] (quickwit/quickwit-ui/yarn.lock)
• High CVE: [GHSA redacted] (quickwit/quickwit-ui/yarn.lock)
• High CVE: [GHSA redacted] (quickwit/quickwit-ui/yarn.lock)
• Hotspot: quickwit/quickwit-datetime/src/java_date_time_format.rs (quickwit/quickwit-datetime/src/java_date_time_format.rs)
• Inconsistent naming and return types for observation: ActorContext.observe returns ObservableState, while ActorHandle.observe returns Observation. Universe.observe returns ActorObservation. These types likely contain overlapping data (state, type, etc.), but the naming (observe vs observe vs observe) and return types vary by context, making it unclear which method to use for a given level of detail.
• Low CVE: [GHSA redacted] (quickwit/quickwit-ui/yarn.lock)
• Medium vulnerability: RUSTSEC-2026-0285 (quickwit/Cargo.lock)
• Medium vulnerability: RUSTSEC-2026-0310 (quickwit/Cargo.lock)
• Medium: security finding (details withheld)
• …and 30 more

• 19 commits — 15 feature/other, 4 fixes

• quickwit/quickwit-control-plane — 3 commits
• quickwit/quickwit-indexing — 3 commits
• quickwit/quickwit-datetime — 2 commits
• quickwit/quickwit-ingest — 2 commits
• quickwit/quickwit-query — 2 commits
• .github/workflows — 1 commit
• quickwit/Cargo.lock — 1 commit
• quickwit/quickwit-common — 1 commit
• quickwit/quickwit-compaction — 1 commit
• quickwit/quickwit-jaeger — 1 commit
• quickwit/quickwit-parquet-engine — 1 commit
• quickwit/quickwit-storage — 1 commit

• fix: fix(Security): Apply patch update 0.9.1 to main (#6821)
• fix: fix(datetime): accept an eight-digit subsecond java date format token (#6804)
• fix: fix(datetime): reject a signed fractional part in a timestamp string (#6767)
• fix: fix(indexing): align sequencer capacity with upload concurrency (#6790)
• change: (Calculated fields) Add calculated-field predicates to QueryAst (#6777)
• change: AZ and decommissioning-aware indexing planning, plus optimizations (#6694)
• change: Add indexers AZ ready gate to indexer candidate pool (#6812)
• change: In memory indexing (#6807)
• change: Remove Parquet code (#6809)
• change: Remove ingest observation stream (#6822)
• change: Rework readiness/liveness; exit on server failures (#6673)
• change: Use generation ID for routing table entries, and react to the ingester pool (#6791)
• change: Use predicate cache for all non-timestamp predicates (#6760)
• change: Write split on source EOF (#6808)
• change: Zonally aware ingest controller (#6795)
• change: chore(CI): Run full tests in the merge queue (#6824)
• change: chore: update Tantivy to latest main (#6815)
• change: improvements to our gcs storage (#6802)
• change: use search priority for cpu scheduling (#6798)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

quickwit-oss/quickwit was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.