valeriansaliou/sonic

This system is a high-performance, in-memory search backend written in Rust, designed for fast text indexing and retrieval. It provides a structured API for ingesting documents, executing full-text and suggestion queries, and managing index state through a dedicated channel protocol. The core engine supports advanced text processing features such as Unicode normalization, stemming, and configurable diacritic handling, backed by persistent storage options like RocksDB and FSTs.

New utility modules have been added to the server benchmarks to support loading and processing Hugging Face datasets. The \download\ module handles fetching and caching dataset shards (including offline support via local cache checks), while the \load\ module provides efficient iteration over Parquet files using Arrow record batches. A specific \WikipediaArticle\ model is included to parse Wikipedia dataset fields (id, title, url, text) for benchmarking purposes.

server/benches/common/huggingface · high confidence

This change introduces the initial set of Debian packaging files for the Sonic search backend, enabling installation via standard Debian package managers. The package includes a systemd service unit to run Sonic as a background service, a dedicated 'sonic' system user for security isolation, and configuration scripts that install the binary and config file to standard system locations (/usr/bin, /etc, /var/lib). It also provides a README with instructions for releasing the library and server components, and a Taskfile for automating maintenance tasks.

packaging · high confidence

The \client/src\ directory now contains a complete, new Rust client library for the Sonic search backend. This library provides a structured API for interacting with Sonic via its Channel protocol, featuring dedicated modules for ingest operations (push, pop, count), search operations (query, suggest, list), and server control (trigger, info). It includes a robust connection multiplexer built on \mio\ for handling asynchronous I/O, a transport layer for TCP communication, and support for experimental features like the \NEW\ option in push commands and raw API access.

client/src · high confidence

Added a new benchmarking suite under \server/benches/common\ that provides shared utilities for running and measuring server performance. This includes helpers to launch the Sonic binary with configurable profiling modes (via \PROFILING\_MODE\), manage temporary data paths, and handle logging levels. The infrastructure supports integration tests for all commands, parallelism benchmarks to detect race conditions, and optional store size reporting at the end of each run.

server/benches/common · high confidence

Added a suite of Rust examples in the client directory that demonstrate how to use the Sonic client library. These include blocking and asynchronous implementations for ingestion, search, and control channels, featuring both single-threaded and parallel execution patterns. The examples also introduce a common module with a logging transport wrapper to trace network traffic and shared test data for consistent demonstration.

client/examples · high confidence

The core executor now exposes new commands to inspect and manage search data: \COUNTC\, \COUNTB\, and \COUNTO\ allow users to count buckets in a collection, objects in a bucket, and terms in a specific object respectively; \LIST\ retrieves words from a bucket with pagination; and \FLUSHC\, \FLUSHB\, and \FLUSHO\ enable erasing entire collections, specific buckets, or individual objects. These additions provide finer-grained control over the search index state and visibility into its contents.

core/src/executor · high confidence

The core lexer has been restructured into dedicated modules (ranges, stemming, stopwords, token) to support advanced text processing. Users can now enable stemming for supported languages via the \stemming\ feature, which reduces words to their root forms. Unicode normalization (NFKD) is applied to ensure consistent matching, and diacritic folding is now available as an optional, disabled-by-default feature to support diacritics-insensitive search. The lexer also introduces configurable stopwords (allow/deny lists) and optional special pattern detection (e.g., email addresses) to improve tokenization flexibility.

core/src/lexer · high confidence

This change introduces a suite of new shell scripts to automate the Sonic release process. The \make\_release.sh\ script handles version bumping (major, minor, patch, or no-bump) for the server, core, and client components, updating their respective Cargo.toml files and creating git tags. The \build\_release\_archive.sh\ script compiles the server for specific architectures and platforms (e.g., Linux glibc/musl, macOS) and packages them into tarballs. Additionally, \build\_packages.sh\ automates the creation of Debian packages using the packpack tool, while \changelog\_prepare.sh\ assists in generating changelog entries by filtering commit history. Supporting utilities in \scripts/util/\ provide standardized logging and error handling, and \validate-dep-build-script-change.sh\ helps manage Cargo dependency build script allowances.

scripts · high confidence

The store module has been reorganized into dedicated files, introducing a new \generic.rs\ that defines the \StoreGeneric\ and \StoreGenericPool\ traits along with extension methods for pool management (acquire, janitor, erase). A new \types.rs\ file centralizes core type definitions including \StoreObjectIid\, \StoreObjectOid\, \StoreTermHash\, and \StoreItemPart\, replacing previous ad-hoc definitions. This change simplifies the internal architecture by separating generic pool logic from specific store implementations (fst, kv) and standardizing how store identifiers and item parts are handled.

core/src/store · high confidence

The core library has been reorganized into a dedicated crate, introducing a comprehensive configuration system (\Config\) that manages normalization, tokenization, stopwords, search limits, and store settings. Users can now enable Unicode normalization (NFC/NFKC), toggle diacritic folding (disabled by default), and define custom allow/deny stopwords. Tokenization behavior includes optional special pattern detection and compatibility splitting. The search configuration adds minimum term IDF thresholds and query limits, while the store configuration exposes detailed RocksDB tuning parameters for both KV and FST stores, including compression, parallelism, and buffer sizes.

core/src · high confidence

The \core/src/util\ module has been restructured to include new internal utilities: \fmt.rs\ provides \AsPrettyRwLock\ and \AsPrettyMutex\ wrappers for more readable debug output of synchronization primitives; \hash.rs\ introduces a \NoopU32Hasher\ to avoid redundant hashing of already-hashed terms; and \serde.rs\ adds deserialization helpers, including support for resolving environment variables via the \${env.\*}\ syntax in configuration files. These changes support the broader separation of core and server logic.

core/src/util · high confidence

The server's channel communication layer has been restructured into dedicated modules (command, format, handle, message, listen, mode, statistics) to separate core and server logic. This refactor introduces new ingest commands COUNTC, COUNTB, and COUNTO, and adds experimental CONFIG and TRIGGER commands (including flush and compact actions) behind the experimental-api feature flag. The protocol handling now correctly preserves unknown escape sequences in text parts and auto-decrements the CLIENTS\_CONNECTED statistic on connection closure, even during panics, to prevent connection count leaks.

server/src/channel · high confidence

The server now initializes logging before configuration is loaded, ensuring that startup diagnostics are captured even if the configuration file is missing or malformed. Configuration loading has been expanded to search multiple standard paths (such as /etc/sonic/) in addition to the previous defaults, and the application now supports a \--config\ command-line flag to specify a custom configuration file path. The server also validates that stopwords are not both allowed and denied at startup, exiting with an error if this conflict is detected.

server/src · high confidence

The stopwords data has been restructured from a single monolithic source into individual, language-specific Rust modules (e.g., \afr.rs\, \ara.rs\, \eng.rs\) within the \core/src/stopwords\ directory. This change introduces Unicode Normal Form KC storage, removes duplicate entries and mojibake, and explicitly excludes numeric tokens from the stopword lists. For languages without established stopword data, empty static arrays are provided to maintain the module structure.

core/src/stopwords · high confidence

The \core/src/store/kv\ and \core/src/store/fst\ modules have been split into smaller, focused files (backup, keys, pool, util, mod) and renamed to follow \\*Store\ and \\*Config\ conventions. This change introduces explicit backup and restore functionality for both KV (RocksDB) and FST stores, allowing users to dump and restore data to and from specified paths. It also includes internal fixes for document counting after flush operations, clearing the IIDIncr cache during flushes, and correcting object counter behavior on big-endian platforms.

core/src/store/kv · high confidence

Added compatibility tests for Sonic packaged configurations; Added integration tests for Sonic client buffering; Added integration tests for Sonic server commands and configuration; Added parallelism benchmarking infrastructure; Added test infrastructure for server integration tests; Added test infrastructure for the search core; Expanded test coverage for search features and bug fixes; Introduce server benchmarks for parallelism, Wikipedia ingestion, and BEIR evaluation.

The project has upgraded its Rust edition to 2024 and set the minimum supported Rust version (MSRV) to 1.92.0. This change requires users to update their local Rust toolchains to at least version 1.92.0 to build the software, as the codebase now relies on features and syntax introduced in the 2024 edition.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 48 → 66 (+17.8)
• Rubric changed (rubric-2026.08.15 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 100 → 98 (-2.1)
• Architecture 99 (new)
• Maturity 53 → 50 (-3.0)
• Readiness 27 → 73 (+46.1)
• Security 78 → 76 (-2.2)

• Dimension evaluation failed
• No automated tests
• No exposed public API
• No tests found
• Test reliability not included
• The banner links to a GitHub workflow badge for lint/test and a dependency status badge but does not link to the project's crates.io page or release channels (crates.io, GitHub Actions), so readers cannot verify that Sonic is published there as claimed. (README.md)

• Dependency advisory scan runs only on code events
• Dependency hygiene PARTLY measured — Cargo dependencies read, dependency currency not (crates.io unreachable)
• Duplicated block (10 lines × 2) (core/src/executor/search.rs)
• Duplicated block (11 lines × 2) (client/src/control.rs)
• Duplicated block (11 lines × 2) (core/src/lexer/token.rs)
• Duplicated block (12 lines × 2) (server/src/channel/command.rs)
• Duplicated block (12–13 lines × 2) (server/src/channel/command.rs)
• Duplicated block (13 lines × 2) (server/src/channel/command.rs)
• Duplicated block (14 lines × 3) (server/src/channel/command.rs)
• Duplicated block (15 lines × 2) (client/src/channel.rs)
• Duplicated block (16 lines × 2) (client/src/channel.rs)
• Duplicated block (16 lines × 4) (server/src/channel/command.rs)
• Duplicated block (18 lines × 2) (core/src/util/fmt.rs)
• Duplicated block (21 lines × 2) (server/src/channel/command.rs)
• Duplicated block (22 lines × 2) (client/src/control.rs)
• Duplicated block (6 lines × 2) (client/src/connection.rs)
• Duplicated block (8 lines × 2) (server/src/channel/command.rs)
• Duplicated block (8 lines × 3) (server/src/channel/command.rs)
• Duplicated block (9 lines × 2) (server/src/channel/command.rs)
• FixmeComment (client/src/transport.rs)
• …and 50 more

• 217 commits — 197 feature/other, 20 fixes

• core/src — 90 commits
• (root) — 40 commits
• server/benches — 22 commits
• (repo) — 11 commits
• server/src — 11 commits
• core/CHANGELOG.md — 9 commits
• server/CHANGELOG.md — 7 commits
• server/tests — 7 commits
• client/src — 5 commits
• server/Taskfile.dist.yaml — 4 commits
• client/CHANGELOG.md — 3 commits
• scripts/validate-dep-build-script-change.sh — 2 commits
• .zed/settings.json — 1 commit
• client/Cargo.toml — 1 commit
• client/TODO.md — 1 commit
• client/examples — 1 commit
• core/tests — 1 commit
• scripts/make_release.sh — 1 commit

• fix: Merge pull request #403 from dualfroz/fix-no-panic-on-read-error
• fix: Merge pull request #410 from fix-issue-405
• fix: Merge pull request #424 from Theryston/fix/core-cjk-byte-offsets
• fix: bench(server): Fix compiler warning
• fix: chore(core): Fix clippy error
• fix: fix(core): Clear IIDIncr from cache during a FLUSHO
• fix: fix(core): Fix get_new_iid not initializing from database
• fix: fix(core): Fix max_flushes
• fix: fix(core): Fix u32_max merge operator really being u32_min…
• fix: fix(core): Fix document count after a FLUSHO
• fix: fix(core): Fix object counter on big-endian platforms and after batch ingest
• fix: fix(core): use byte offsets in CJK tokenizer
• fix: fix(server): Auto-decrement CLIENTS_CONNECTED to support panics
• fix: fix(server): Do not panic when a channel read fails
• fix: fix(server): Fix panic when client closes connection before ENDED
• fix: fix(server): Make experimental flag NEW compatible with content > buffer_size
• fix: fix(server): Preserve escaped character after unrecognized escapes (#398)
• fix: fix(tools): Fix name of packaged configs in compatibility tests
• fix: fixup! fix(server): Preserve escaped character after unrecognized escapes (#398)
• fix: fixup! fix(server): Preserve escaped character after unrecognized escapes (#398)
• …and 197 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

valeriansaliou/sonic was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.