toshi-search/Toshi

Toshi is a distributed search engine built on the Tantivy library, providing a REST API for managing indexes and performing complex text searches. It supports a cluster architecture using the Raft consensus protocol to replicate data and maintain consistency across nodes. The system offers both synchronous and asynchronous HTTP clients for interacting with the service, handling operations such as document indexing, deletion, and various query types including boolean, fuzzy, and range searches.

9 changes.

The project established its foundational structure through comprehensive documentation, CI configuration, and core type definitions for search operations. It simultaneously transitioned from a monolithic HTTP server to a distributed architecture by implementing Raft consensus protocols and dedicated client libraries for asynchronous and synchronous interactions.

4 changes.

This period focused on establishing the core infrastructure for the Toshi search engine by introducing a Raft-based distributed consensus module and an initial HTTP-based search server. The work included implementing REST API handlers for index management and various query types, alongside integration tests to verify client-server interactions.

The \toshi-proto\ crate now includes the Protocol Buffers definitions required for distributed clustering. This adds \cluster.proto\, which defines the \IndexService\ RPC interface for operations like indexing, searching, and document management, along with specific RPCs for cluster membership (\join\) and Raft consensus (\raft\_request\). It also includes \eraftpb.proto\, providing the message structures for the Raft protocol (such as \Message\, \Entry\, and \ConfChangeV2\) and a \build.rs\ script to generate the corresponding Rust code using \tonic\_build\.

toshi-proto/proto · high confidence

New example files have been added to the toshi-client package to demonstrate how to perform search queries. These include examples for exact term matching, fuzzy queries, range queries, and boolean queries, showcasing both asynchronous (using HyperToshi) and synchronous (using ToshiClient) client usage patterns.

toshi-client/examples · high confidence

The repository is initialized with core project files including an MIT license, a security policy, and a codecov configuration. Comprehensive documentation is added to the README, detailing build requirements (Rust 1.39.0+), configuration options (host, port, data path, merge policy), and usage examples. Additionally, example HTTP request files and JSON schemas are provided to demonstrate index creation and search queries.

(repo-wide) · high confidence

This change introduces the foundational type system for the Toshi search engine within the \toshi-types\ crate. It defines the core data structures for client interactions, including \ScoredDoc\ and \SearchResults\ for handling search responses with relevancy scores and facets, as well as \SummaryResponse\ for index metadata. It establishes a comprehensive error handling framework via the \Error\ enum and \ErrorResponse\ struct, covering IO, query, and Tantivy-specific failures. Additionally, it provides the request body models (\AddDocument\, \DeleteDoc\, \SchemaBody\) for server operations and defines the critical \IndexHandle\, \Catalog\, and \Serve\ traits that dictate how indexes are managed, accessed, and served via HTTP.

toshi-types/src · high confidence

This change introduces the Toshi server, a new HTTP-based search service built on Tantivy. It provides a REST API for managing indexes (create, list, flush) and performing document operations (add, search, delete). The server includes an automatic commit watcher to periodically flush in-memory changes to disk, configurable via the \auto\_commit\_duration\ setting. It also supports bulk insertion with a configurable \max\_line\_length\ limit to prevent errors on excessively long lines, and optional CANG\_JIE tokenizer support for Chinese text processing when compiled with the \extra\_tokenizers\ feature.

toshi-server/src · high confidence

Added a new \toshi-raft\ module that integrates the Raft consensus algorithm into the Toshi search engine. This change introduces a \RaftHandle\ that implements the \Storage\ trait to persist and retrieve log entries from the Tantivy index, enabling the system to replicate data across a cluster. The module includes an RPC server (\rpc\_server.rs\) and utilities (\rpc\_utils.rs\) to handle cluster communication, proposal management (\proposal.rs\), and the main Raft event loop (\lib.rs\), allowing nodes to coordinate state changes and maintain consistency through leader election and log replication.

toshi-raft · high confidence

The toshi-client library now provides concrete HTTP client implementations for interacting with the Toshi search service. It introduces a unified error type (ToshiClientError) and two distinct client backends: HyperToshi, which uses the Hyper library, and ToshiClient, which uses Isahc. Both backends implement the AsyncClient trait for asynchronous operations and the SyncClient trait (specifically the Isahc backend) for synchronous operations, exposing methods to manage indices, add documents, and perform searches.

toshi-client/src · high confidence

The \toshi-types/src/query\ module now introduces a comprehensive set of query types and builders to support complex search operations. Users can construct boolean queries with \must\, \must\_not\, and \should\ clauses, perform fuzzy matching with configurable distance and transposition, search for exact terms, match phrases with optional term offsets, filter by value ranges (gte, lte, lt, gt), and execute regular expression searches. Additionally, a new \FacetQuery\ type is available for collecting facet data. All these types implement the \CreateQuery\ trait to generate Tantivy queries and are exposed via a fluent builder API for easier composition.

toshi-types/src/query · high confidence

The \toshi-server/src/handlers\ module has been introduced to manage HTTP request handling for the search engine. This includes handlers for bulk document insertion with configurable line length limits, index management (creation, deletion, and document addition), a new list endpoint to enumerate indexes, and search capabilities supporting term, phrase, fuzzy, range, regex, and boolean queries. The module also provides utilities for index summarization and flushing, along with comprehensive tests for these functionalities.

toshi-server/src/handlers · high confidence

The application's HTTP server implementation, including the main entry point, router, and request handlers, has been removed. This change eliminates the previous integration with the Gotham web framework, the Hyper HTTP library, and the associated routing logic for search and indexing endpoints.

src · high confidence

The protocol buffer definitions for the cluster RPC service have been moved into a new, dedicated \toshi-proto\ crate. This change restructures the codebase by extracting the \clusterrpc\ proto module, exposing the generated client and server implementations (\index\_service\_client\ and \index\_service\_server\) through a public \cluster\_rpc\ module, which simplifies how other parts of the application import and use these RPC components.

toshi-proto/src · high confidence

The CI system has been migrated to Azure Pipelines, introducing new YAML templates to install Rust toolchains and system dependencies on Linux, macOS, and Windows. The pipeline now includes steps to run code coverage using KCov and upload results to Codecov, alongside a local coverage script that generates HTML reports or uploads LCOV data.

ci · high confidence

Added a new integration test in \toshi-server/tests/lib.rs\ that verifies the client-server interaction by starting the server router and using the \HyperToshi\ client to fetch an index.

toshi-server/tests · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 64 → 62 (-1.6)
• Rubric changed (rubric-2026.09.11 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 99 → 99 (+0.0)
• Architecture 100 → 99 (-1.5)
• Maturity 46 → 46 (+0.0)
• Readiness 87 → 76 (-11.2)
• Security 64 → 64 (+0.0)
• Performance 100 (new)

• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)

• Dependency hygiene PARTLY measured — Cargo dependencies read, dependency currency not (crates.io unreachable)
• Duplicate function signatures in the same module. Two functions named setup_logging_from_file exist with identical signatures, one using a named parameter and one using an ignored parameter. This is a clear duplication error.
• Duplicate method signatures with different parameter names (s vs signal). While technically the same operation, having two overloads or methods with identical signatures but different names in the same module is confusing and suggests copy-paste errors or lack of cleanup.
• Inconsistent document ingestion interface. The client APIs (AsyncClient/SyncClient) accept a strongly-typed document: D and IndexOptions. The HTTP handler (index.add_document) and the Serve trait accept a generic body: Body (likely JSON bytes/string) and an index name string, with no explicit options parameter visible in the signature (options might be inside the body or default). This forces users to construct different payloads for clients vs HTTP.
• Inconsistent parameter naming and structure for retrieving index metadata. Async/Sync clients use a boolean flag include_sizes, while the HTTP handler uses a QueryOptions object. Furthermore, the HTTP handler's QueryOptions constructor takes pretty and include_sizes, but the client methods do not expose the pretty option, creating a fragmented API surface for the same logical operation.
• Inconsistent return types for listing indexes. list_indexes in the handler returns a ResponseFuture (HTTP response), Catalog returns a String (likely a serialized list or comma-separated names), and Serve returns a generic Result. The lack of a consistent structured return type (e.g., a list of index names) makes it difficult to parse or use the result uniformly.
• Inconsistent search input interface. The client API accepts a structured Search object. The HTTP handler and Serve trait accept a raw body: Body and an index string. This implies the Search object must be serialized into the body, but the signature doesn't reflect this transformation, hiding the complexity from the user and creating a disconnect between the typed client API and the HTTP API.
• Off the main sequence: toshi-types

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

toshi-search/Toshi was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.