valeriansaliou/sonic
60.5
Adequate · 14 November 2026
34.8k
lines of production code
Rust
primary language
7
measurements over time
What this system is
Features
Add Hugging Face dataset benchmark utilities
New utility modules have been added to the server benchmarks to support loading and processing Hugging Face datasets. The \download\ module handles fetching and caching dataset shards (including offline support via local cache checks), while the \load\ module provides efficient iteration over Parquet files using Arrow record batches. A specific \WikipediaArticle\ model is included to parse Wikipedia dataset fields (id, title, url, text) for benchmarking purposes.
server/benches/common/huggingface · high confidence
Initial Debian packaging for Sonic
This change introduces the initial set of Debian packaging files for the Sonic search backend, enabling installation via standard Debian package managers. The package includes a systemd service unit to run Sonic as a background service, a dedicated 'sonic' system user for security isolation, and configuration scripts that install the binary and config file to standard system locations (/usr/bin, /etc, /var/lib). It also provides a README with instructions for releasing the library and server components, and a Taskfile for automating maintenance tasks.
packaging · high confidence
Introduce new Rust client library for Sonic
The \client/src\ directory now contains a complete, new Rust client library for the Sonic search backend. This library provides a structured API for interacting with Sonic via its Channel protocol, featuring dedicated modules for ingest operations (push, pop, count), search operations (query, suggest, list), and server control (trigger, info). It includes a robust connection multiplexer built on \mio\ for handling asynchronous I/O, a transport layer for TCP communication, and support for experimental features like the \NEW\ option in push commands and raw API access.
client/src · high confidence
Introduce server benchmarking infrastructure
Added a new benchmarking suite under \server/benches/common\ that provides shared utilities for running and measuring server performance. This includes helpers to launch the Sonic binary with configurable profiling modes (via \PROFILING\_MODE\), manage temporary data paths, and handle logging levels. The infrastructure supports integration tests for all commands, parallelism benchmarks to detect race conditions, and optional store size reporting at the end of each run.
server/benches/common · high confidence
New client examples demonstrating blocking and async search capabilities
Added a suite of Rust examples in the client directory that demonstrate how to use the Sonic client library. These include blocking and asynchronous implementations for ingestion, search, and control channels, featuring both single-threaded and parallel execution patterns. The examples also introduce a common module with a logging transport wrapper to trace network traffic and shared test data for consistent demonstration.
client/examples · high confidence
New counting, listing, and flushing commands for search collections
The core executor now exposes new commands to inspect and manage search data: \COUNTC\, \COUNTB\, and \COUNTO\ allow users to count buckets in a collection, objects in a bucket, and terms in a specific object respectively; \LIST\ retrieves words from a bucket with pagination; and \FLUSHC\, \FLUSHB\, and \FLUSHO\ enable erasing entire collections, specific buckets, or individual objects. These additions provide finer-grained control over the search index state and visibility into its contents.
core/src/executor · high confidence
New modular lexer with stemming, Unicode normalization, and configurable diacritics
The core lexer has been restructured into dedicated modules (ranges, stemming, stopwords, token) to support advanced text processing. Users can now enable stemming for supported languages via the \stemming\ feature, which reduces words to their root forms. Unicode normalization (NFKD) is applied to ensure consistent matching, and diacritic folding is now available as an optional, disabled-by-default feature to support diacritics-insensitive search. The lexer also introduces configurable stopwords (allow/deny lists) and optional special pattern detection (e.g., email addresses) to improve tokenization flexibility.
core/src/lexer · high confidence
New release tooling and build scripts for Sonic
This change introduces a suite of new shell scripts to automate the Sonic release process. The \make\_release.sh\ script handles version bumping (major, minor, patch, or no-bump) for the server, core, and client components, updating their respective Cargo.toml files and creating git tags. The \build\_release\_archive.sh\ script compiles the server for specific architectures and platforms (e.g., Linux glibc/musl, macOS) and packages them into tarballs. Additionally, \build\_packages.sh\ automates the creation of Debian packages using the packpack tool, while \changelog\_prepare.sh\ assists in generating changelog entries by filtering commit history. Supporting utilities in \scripts/util/\ provide standardized logging and error handling, and \validate-dep-build-script-change.sh\ helps manage Cargo dependency build script allowances.
scripts · high confidence
Architecture
Refactored store module structure and types
The store module has been reorganized into dedicated files, introducing a new \generic.rs\ that defines the \StoreGeneric\ and \StoreGenericPool\ traits along with extension methods for pool management (acquire, janitor, erase). A new \types.rs\ file centralizes core type definitions including \StoreObjectIid\, \StoreObjectOid\, \StoreTermHash\, and \StoreItemPart\, replacing previous ad-hoc definitions. This change simplifies the internal architecture by separating generic pool logic from specific store implementations (fst, kv) and standardizing how store identifiers and item parts are handled.
core/src/store · high confidence
Behavioural changes
Core library restructured with new configuration and normalization features
The core library has been reorganized into a dedicated crate, introducing a comprehensive configuration system (\Config\) that manages normalization, tokenization, stopwords, search limits, and store settings. Users can now enable Unicode normalization (NFC/NFKC), toggle diacritic folding (disabled by default), and define custom allow/deny stopwords. Tokenization behavior includes optional special pattern detection and compatibility splitting. The search configuration adds minimum term IDF thresholds and query limits, while the store configuration exposes detailed RocksDB tuning parameters for both KV and FST stores, including compression, parallelism, and buffer sizes.
core/src · high confidence
FST store backup and restore now works while stores are active
The FST store backup and restore operations have been fixed to function correctly while the stores are in use. The implementation now properly acquires a write lock on the graph access during backup to prevent concurrent modifications, ensuring data consistency. This change also includes a structural refactor that splits the previously monolithic store module into separate files for backup, pool management, and utilities, improving maintainability without altering the external API.
core/src/store/fst · high confidence
New utility modules for formatting, hashing, and configuration deserialization
The \core/src/util\ module has been restructured to include new internal utilities: \fmt.rs\ provides \AsPrettyRwLock\ and \AsPrettyMutex\ wrappers for more readable debug output of synchronization primitives; \hash.rs\ introduces a \NoopU32Hasher\ to avoid redundant hashing of already-hashed terms; and \serde.rs\ adds deserialization helpers, including support for resolving environment variables via the \${env.\*}\ syntax in configuration files. These changes support the broader separation of core and server logic.
core/src/util · high confidence
Refactored server channel protocol handling and added new ingest commands
The server's channel communication layer has been restructured into dedicated modules (command, format, handle, message, listen, mode, statistics) to separate core and server logic. This refactor introduces new ingest commands COUNTC, COUNTB, and COUNTO, and adds experimental CONFIG and TRIGGER commands (including flush and compact actions) behind the experimental-api feature flag. The protocol handling now correctly preserves unknown escape sequences in text parts and auto-decrements the CLIENTS\_CONNECTED statistic on connection closure, even during panics, to prevent connection count leaks.
server/src/channel · high confidence
Server startup and configuration overhaul
The server now initializes logging before configuration is loaded, ensuring that startup diagnostics are captured even if the configuration file is missing or malformed. Configuration loading has been expanded to search multiple standard paths (such as /etc/sonic/) in addition to the previous defaults, and the application now supports a \--config\ command-line flag to specify a custom configuration file path. The server also validates that stopwords are not both allowed and denied at startup, exiting with an error if this conflict is detected.
server/src · high confidence
Stopwords lists refactored into per-language Rust modules
The stopwords data has been restructured from a single monolithic source into individual, language-specific Rust modules (e.g., \afr.rs\, \ara.rs\, \eng.rs\) within the \core/src/stopwords\ directory. This change introduces Unicode Normal Form KC storage, removes duplicate entries and mojibake, and explicitly excludes numeric tokens from the stopword lists. For languages without established stopword data, empty static arrays are provided to maintain the module structure.
core/src/stopwords · high confidence
Store modules refactored and backup capabilities added
The \core/src/store/kv\ and \core/src/store/fst\ modules have been split into smaller, focused files (backup, keys, pool, util, mod) and renamed to follow \\*Store\ and \\*Config\ conventions. This change introduces explicit backup and restore functionality for both KV (RocksDB) and FST stores, allowing users to dump and restore data to and from specified paths. It also includes internal fixes for document counting after flush operations, clearing the IIDIncr cache during flushes, and correcting object counter behavior on big-endian platforms.
core/src/store/kv · high confidence
Test coverage
Added compatibility tests for Sonic packaged configurations; Added integration tests for Sonic client buffering; Added integration tests for Sonic server commands and configuration; Added parallelism benchmarking infrastructure; Added test infrastructure for server integration tests; Added test infrastructure for the search core; Expanded test coverage for search features and bug fixes; Introduce server benchmarks for parallelism, Wikipedia ingestion, and BEIR evaluation.
Dependencies
Update Rust toolchain to edition 2024 and Rust 1.92.0
The project has upgraded its Rust edition to 2024 and set the minimum supported Rust version (MSRV) to 1.92.0. This change requires users to update their local Rust toolchains to at least version 1.92.0 to build the software, as the codebase now relies on features and syntax introduced in the 2024 edition.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 61 → 61 (-0.7)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 67 → 66 (-0.5)
- Architecture 97 → 97 (+0.0)
- Maturity 48 → 48 (+0.3)
- Readiness 71 → 71 (-0.9)
- Security 78 → 71 (-7.2)
- Performance 85 → 85 (+0.0)
Resolved (4)
- Hotspot: core/src/lexer/token.rs (core/src/lexer/token.rs)
- Medium advisory (unsound): RUSTSEC-2026-0386 (Cargo.lock)
- No ADRs found
- Off-boarding risk: anonymized user #1
New (14)
- Duplicated block (10 lines × 2) (server/src/channel/command.rs)
- Edited copies of one member (3 members, 1+ corresponding lines) (client/src/control.rs)
- Edited copies of one member (5 members, 23+ corresponding lines) (core/src/store/kv/mod.rs)
- Edited copies of one member (6 members, 18+ corresponding lines) (server/src/channel/command.rs)
- Edited copy of a member (21 corresponding lines) (core/src/store/fst/pool.rs)
- Edited copy of a member (23 corresponding lines) (core/src/store/fst/mod.rs)
- Edited copy of a member (49 corresponding lines) (client/src/channel.rs)
- Edited copy of a member (7 corresponding lines) (server/src/channel/listen.rs)
- Edited copy of a member (75 corresponding lines) (server/src/channel/command.rs)
- Edited copy of a member (8 corresponding lines) (client/src/connection.rs)
- No ADRs found (server/Cargo.toml)
- Off-boarding risk: anonymized user #1
- Outdated: log
- Outdated: mio
Changes since last survey
- 4 commits — 2 feature/other, 2 fixes
By area
- (repo) — 2 commits
- (root) — 2 commits
Notable commits
- fix: Merge pull request #440 from MAHIR-BABBAR/fix/issue-438
- fix: fix(security): Bump dependencies
- change: Merge pull request #434 from dualfroz/feat/configurable-typo-factor
- change: docs: Mention work on Sonic v2 in the README
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
valeriansaliou/sonic was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit f16560b6ea67fd223354c065f7157646931db1b2 — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.