pat/thinking-sphinx

Thinking Sphinx is a Ruby gem that integrates the Sphinx search engine with ActiveRecord applications, enabling full-text search, faceting, and geospatial queries. It supports both traditional SQL-based indexing and real-time index updates that synchronize with database transactions. The system provides a modular architecture for managing index configurations, handling complex association scoping, and executing search commands via a middleware pipeline.

23 changes.

This period focused on a major architectural rewrite for Thinking Sphinx v6.0.0, introducing a modular, middleware-based design and native support for Rails 7 and Zeitwerk. The work involved completely restructuring the ActiveRecord integration, database adapters, and delta indexing into dedicated background jobs to improve reliability and performance. Comprehensive test suites were added to validate the new modular components and ensure compatibility with modern Ruby and Rails versions.

17 changes.

This period focused on introducing real-time Sphinx indexing support, allowing immediate index updates without scheduled tasks, and refactoring the search execution flow into a modular middleware pipeline. The work also restructured result processing using masks and panes for better modularity, while significantly expanding test coverage across these new subsystems.

19 changes.

This period focused on restructuring the codebase by replacing monolithic rake tasks with a modular command pattern and refactoring SQL generation and connection handling for better maintainability. It also introduced support for Rails 5.2 polymorphic associations, enhanced real-time indexing consistency, and added features like distributed facet searching and configurable indexing guards.

A new file lib/thinking/sphinx.rb has been added to the library. This file serves as a simple wrapper that requires the 'thinking\_sphinx' gem, enabling the application to load Sphinx search functionality through the thinking namespace. The file also includes the frozen\_string\_literal pragma for performance and safety.

lib/thinking · high confidence

New executable scripts have been added to the bin directory to improve the developer experience. The \bin/console\ script provides an interactive IRB environment with the gem loaded for local experimentation. The \bin/loadsphinx\ script automates the installation of specific versions of Sphinx and Manticore search engines for testing purposes. Additionally, \bin/testmatrix\ generates a JSON array of valid test combinations across Ruby, Rails, database, and search engine versions, facilitating dynamic CI configuration.

bin · high confidence

The indexing process now supports a configurable guard mechanism to prevent concurrent or conflicting index operations. A new file-based guard implementation uses temporary lock files (e.g., \ts-\<index\_name\>.tmp\) to track active indexing tasks; if a lock file exists, the index is skipped and a log message is generated. The system ensures these lock files are cleaned up even if an exception occurs during indexing. Additionally, a 'none' guard option is available to bypass this locking behavior entirely, allowing users to choose the approach that best fits their deployment environment.

_lib/thinking\sphinx/guard · high confidence

A new Template class has been added to define the internal Sphinx fields and attributes (such as class name, ID, and deletion status) used by the SQL source. This template handles Single Table Inheritance (STI) by correctly quoting and converting the inheritance column, ensuring that internal class references are properly escaped and formatted for the search index.

_lib/thinking\_sphinx/active\_record/sql\source · high confidence

Thinking Sphinx now supports real-time indices, allowing search indexes to be updated immediately as data changes without requiring scheduled re-indexing tasks. This feature introduces a new DSL within index definitions to declare fields and attributes, including support for JSON attributes, multi-value attributes, and scoping. The system handles data synchronization via a batched populator that translates ActiveRecord instances into Sphinx documents, manages deletions and updates, and ensures data integrity by persisting instances before indexing and handling UTF-8 encoding.

_lib/thinking\_sphinx/real\time · high confidence

Distributed indexes now support facet searching. The new \ThinkingSphinx::Distributed::Index\ class aggregates facets from its local index objects, allowing users to perform facet queries across distributed search configurations.

_lib/thinking\sphinx/distributed · high confidence

The Capistrano v3 deployment tasks for Thinking Sphinx now default the Rails environment to the \rails\_env\ variable (falling back to \stage\ if not set) and use the \thinking\_sphinx\_roles\ variable to determine which servers execute the tasks. This allows users to customize the environment and target roles for Sphinx operations like indexing and restarting the daemon, aligning the v3 implementation with modern Capistrano conventions while maintaining compatibility with existing configurations.

_lib/thinking\sphinx/capistrano · high confidence

The \lib/thinking\_sphinx/active\_record\ directory has been entirely rewritten to modernize the integration with ActiveRecord. The previous monolithic classes (such as \SQLBuilder\ and \SQLSource\) have been replaced by a modular architecture featuring dedicated classes for associations (\Association\, \JoinAssociation\), column handling (\Column\, \ColumnSQLPresenter\), and property queries (\PropertyQuery\, \SimpleManyQuery\). This change introduces support for complex association-scoped searching, polymorphic models, and multi-value attributes (MVA) via the new \Joiner\ integration. It also adds a \LogSubscriber\ for structured logging and updates the \FilterReflection\ to support Rails 5.2+ reflection APIs, ensuring compatibility with modern Rails versions while maintaining backward compatibility for older releases.

_lib/thinking\_sphinx/active\record · high confidence

Users can now control whether real-time index callbacks are active via a configuration setting, allowing them to disable automatic indexing if desired. The callback mechanism has been refactored to support both method path symbols and custom blocks, enabling more flexible logic for determining which objects to index after a commit or save. This change consolidates the callback logic into a new RealTimeCallbacks class that handles persistence to Sphinx indices.

_lib/thinking\_sphinx/real\time/callbacks · high confidence

Thinking Sphinx v6.0.0 now requires Ruby 3.0+ and Rails 6.1+, removing support for older versions. The test suite has been expanded to include Rails 7.2, 8.0, and 8.1, and CI now tests against Ruby 2.4–2.7 and 3.x with both Sphinx and Manticore engines.

(repo-wide) · high confidence

Thinking Sphinx now correctly handles attribute resolution for \has\_many :through\ associations by introducing dedicated \AttributeFinder\ and \AttributeMatcher\ classes. These new components ensure that foreign keys are accurately matched to index attributes, even when traversing intermediate models, thereby fixing scoping issues for real-time indices in complex association chains.

_lib/thinking\_sphinx/active\_record/association\proxy · high confidence

Thinking Sphinx now includes dedicated database adapters for MySQL and PostgreSQL, replacing generic handling with dialect-specific SQL generation. The MySQL adapter uses functions like \CONCAT\_WS\ and \GROUP\_CONCAT\ for string operations, while the PostgreSQL adapter uses \\|\|\ for concatenation and \array\_to\_string\ for grouping, ensuring correct syntax for each database. Both adapters handle type casting (e.g., bigints, timestamps) and null/blank conversion according to their respective database standards, and both set the database time zone to UTC to ensure consistent timestamp behavior.

_lib/thinking\_sphinx/active\_record/database\adapters · high confidence

Thinking Sphinx now supports configurable indexing strategies, introducing 'AllAtOnce' and 'OneAtATime' approaches to control how indices are processed. Additionally, the system now warns users via STDERR if indexing guard files are detected, helping to prevent concurrent indexing conflicts. This change also includes a reorganization of the library's core structure, adding support for various frameworks and integrating new modules for facets, batched searches, and connection pooling.

lib · high confidence

Thinking Sphinx introduces a new core module structure to handle index and field definitions. The new \ThinkingSphinx::Core::Index\ class manages index rendering, including path configuration, directory creation, and the assignment of infix and prefix search fields. It also ensures that indices are only created for models with existing database tables, logging a warning otherwise. The \ThinkingSphinx::Core::Field\ module adds support for checking if fields should use infix or prefix matching, while \ThinkingSphinx::Core::Property\ provides base property definitions. Additionally, the \ThinkingSphinx::Core::Interpreter\ class has been refactored to use \BasicObject\ for safer method missing handling during index definition interpretation.

_lib/thinking\sphinx/core · high confidence

The library introduces a new internal architecture using 'masks' to handle search result operations, replacing previous mixin-based approaches. This change adds specific mask classes to manage pagination (supporting standard methods like \next\_page\, \previous\_page\, and \total\_pages\), group and weight enumeration (allowing iteration with \each\_with\_group\, \each\_with\_count\, and \each\_with\_weight\), and scoped searches (enabling facet searches and \search\_for\_ids\ on scopes). This restructuring ensures that inherited methods are not passed to masks and provides a more modular way to handle SphinxQL data access.

_lib/thinking\sphinx/masks · high confidence

This change introduces four new classes in the \ThinkingSphinx::Panes\ namespace—\AttributesPane\, \DistancePane\, \ExcerptsPane\, and \WeightPane\—to provide structured access to specific search result components. \AttributesPane\ exposes raw Sphinx attributes, \DistancePane\ handles geospatial distance calculations, \WeightPane\ retrieves relevance weights, and \ExcerptsPane\ generates text excerpts based on the search query and conditions. These classes replace previous internal attribute and variable approaches, offering a more modular and consistent way to access search metadata and formatted content.

_lib/thinking\sphinx/panes · high confidence

A new \PopulatorSubscriber\ class has been introduced to handle events related to real-time index generation. This component subscribes to namespace-specific events (such as \start\_populating\, \populated\, and \finish\_populating\) to provide user-facing feedback, including printing progress dots and index names. It also includes an \error\ handler that logs specific transcription errors for instances without halting the process, improving visibility into real-time indexing issues.

_lib/thinking\sphinx/subscribers · high confidence

The real-time index template now automatically configures internal Sphinx attributes for every index, including a string facet on the class name, a bigint primary key ID, a deleted flag, and an optional updated-at timestamp when real-time tidying is enabled. This ensures consistent metadata handling and supports bigint primary keys out of the box.

_lib/thinking\_sphinx/real\time/index · high confidence

The Thinking Sphinx real-time indexing mechanism has been updated to register its callbacks using \after\_commit\ instead of the previous hook. This ensures that search index updates occur only after the associated database transaction has been successfully committed, improving data consistency for real-time search features. The change is implemented via a new \Appender\ class that explicitly manages core, delta, real-time, and update callbacks based on model behaviors.

_lib/thinking\sphinx/callbacks · high confidence

The MySQL connection logic has been restructured into separate client implementations for MRI and JRuby environments. The new MRI client uses the Mysql2 gem with multi-statement support, while the JRuby client uses JDBC. A key behavioral change is that socket connections are now disabled for JRuby, and for MRI, the host 'localhost' is automatically replaced with '127.0.0.1' to force TCP connections instead of socket connections, ensuring consistent behavior across environments.

_lib/thinking\sphinx/connection · high confidence

The SQL query construction logic in the ActiveRecord integration has been reorganized to improve maintainability and clarity. A new \Statement\ class now handles the preparation of SQL relations and filters, managing scopes for selection, where clauses, grouping, and joins. A new \ClauseBuilder\ class assists in composing these SQL clauses. Additionally, a \Query\ class manages pre-query scopes, including handling time zone settings (with a new \skip\_time\_zone\ option to bypass this), delta processor resets, and session settings. This refactoring centralizes the logic for building SQL statements, making the codebase easier to understand and modify.

_lib/thinking\_sphinx/active\_record/sql\builder · high confidence

The callback logic for ActiveRecord models has been reorganized into four distinct classes: AssociationDeltaCallbacks, DeleteCallbacks, DeltaCallbacks, and UpdateCallbacks. This change separates concerns for handling association changes, record deletions (including rollback support), delta index processing, and attribute updates. Users benefit from more robust handling of deletion events during rollbacks and improved management of delta indices, while the underlying implementation now uses dedicated classes for each callback type rather than a monolithic structure.

_lib/thinking\_sphinx/active\record/callbacks · high confidence

The Sphinx interface classes (Base, Daemon, RealTime, SQL) have been refactored to delegate command execution to a central Commander class. This change standardizes how commands are invoked across different index types and daemon operations, ensuring consistent handling of configuration and options. Users will benefit from more reliable command routing and clearer separation of concerns within the indexing and daemon management logic.

_lib/thinking\sphinx/interfaces · high confidence

The Sphinx management interface has been restructured from monolithic rake tasks into distinct, reusable command classes (such as IndexSQL, Merge, and StartDetached) that inherit from a new base command handler. This change introduces consistent error handling, standardized output formatting, and explicit support for configuration options like skipping directory creation and verbose logging. Users benefit from more reliable command execution, clearer failure diagnostics, and the ability to override Sphinx's running state checks or specify indices for operations like merging and indexing.

_lib/thinking\sphinx/commands · high confidence

Delta indexing and deletion operations are now handled by distinct background jobs (\IndexJob\ and \DeleteJob\) rather than being executed inline. The \IndexJob\ utilizes the \IndexSQL\ command and respects a new \quiet\_deltas\ configuration setting to control logging verbosity, while the \DeleteJob\ safely handles connection errors without raising exceptions. This change decouples the delta logic from the core delta class, providing a more robust and configurable mechanism for keeping search indexes in sync with database changes.

_lib/thinking\sphinx/deltas · high confidence

The configuration system now uses dedicated reconciler classes to handle specific index setup tasks. Distributed indices are automatically grouped by reference and assembled into distributed index objects. Internal primary keys (\sphinx\_internal\_id\) are explicitly enforced as big integers for consistency. The \sphinx\_internal\_class\_name\ field is removed from real-time and plain indices that do not use table inheritance. Duplicate field and attribute names are detected and raise errors, except for distributed indices which are skipped during this check. Default connection settings (address 127.0.0.1, port 9306) are centralized in a new Defaults module.

_lib/thinking\sphinx/configuration · high confidence

The search subsystem has been restructured to improve modularity and reliability. A new \BatchInquirer\ class now handles executing multiple queries in a single connection pool transaction, while a \Merger\ class centralizes query modification logic and prevents changes to already-populated searches. Search results are now wrapped in a \Glaze\ object that delegates method calls to specific panes, enabling richer result metadata without altering the underlying objects. Additionally, a \Context\ class manages search state and memory, and a dedicated \StaleIdsException\ provides clearer error reporting when Sphinx returns IDs that no longer exist in ActiveRecord.

_lib/thinking\sphinx/search · high confidence

The search execution flow has been restructured from a monolithic implementation into a chain of distinct middleware components (ActiveRecordTranslator, Geographer, Glazier, IdsOnly, Inquirer, SphinxQL, StaleIdChecker, StaleIdFilter, and ValidOptions). This change separates concerns such as SQL query construction, result translation to ActiveRecord objects, geospatial distance calculation, and stale ID handling, making the search behavior more predictable and easier to extend.

_lib/thinking\sphinx/middlewares · high confidence

The attribute handling logic now explicitly supports the JSON data type, mapping it to Sphinx's native JSON type, and ensures timestamps are consistently represented as unsigned integers. The system also provides more precise type detection for database columns, including specific handling for big integers and improved error messaging when a referenced column cannot be found, ensuring that index declarations accurately reflect the underlying schema.

_lib/thinking\_sphinx/active\record/attribute · high confidence

Thinking Sphinx now supports Rails 5.2 for polymorphic properties by introducing a new \OverriddenReflection\ strategy. This change adds specific logic to filter join constraints and scopes by the base class name, ensuring correct behavior for polymorphic associations in newer Rails versions while maintaining compatibility with earlier releases through existing strategies.

_lib/thinking\_sphinx/active\record/depolymorph · high confidence

Thinking Sphinx has been rewritten from the ground up, introducing a new middleware-based search pipeline and a command-driven architecture for index management. This update adds native support for Rails 7 and Zeitwerk, while significantly improving real-time index handling with dedicated processors and callbacks. The gem now features a modular design with separate components for connection pooling, deletion strategies, and delta processing, alongside new capabilities like batched searches, facet filtering, and custom rake interfaces.

_lib/thinking\sphinx · high confidence

Added acceptance tests for search and indexing features; Added comprehensive test coverage for Thinking Sphinx core components; Added comprehensive test suite for ThinkingSphinx ActiveRecord components; Added placeholder file for internal tmp directory; Added test coverage for DefaultDelta behavior; Added test coverage for Search::Glaze and Search::Query; Added test coverage for database adapter specifications; Added test coverage for pagination and scopes masks; Added test database schema for internal specs; Added test fixture for namespaced Admin::Person model; Added test fixture models for acceptance testing; Added test fixtures for search index definitions; Added tests for ActiveRecord callback separation; Added tests for ThinkingSphinx ActiveRecord attribute type detection; Added tests for ThinkingSphinx.count and .search methods; Added tests for minimum fields configuration logic; Added tests for real-time callback index updates; Added tests for the guard presence hook; Added unit tests for Sphinx management commands; Added unit tests for ThinkingSphinx MRI connection logic; Added unit tests for ThinkingSphinx interface commands; Added unit tests for ThinkingSphinx pane components; Added unit tests for real-time index components; Added unit tests for the new middleware-based search architecture; New acceptance test support infrastructure for Sphinx integration; Refactored test support infrastructure for multi-database and multi-schema testing; Removed obsolete Sphinx index and model acceptance tests.

Thinking Sphinx has been updated to version 6.0.0, raising the minimum supported Ruby version to 3.0 and the minimum ActiveRecord version to 6.1.0. The gem now depends on Riddle \~\> 2.4 and introduces new runtime dependencies on Joiner, Middleware, and Innertube. Development dependencies have also been updated, including RSpec to \~\> 3.12.0, DatabaseCleaner to \~\> 2.0.2, and Combustion to \~\> 1.1, while the gemspec metadata has been modernized with new URIs and MFA requirements.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 50.

• Code Health 100
• Architecture 97
• Maturity 21
• Readiness 62
• Security 86

• 300 commits — 266 feature/other, 34 fixes

• lib/thinking_sphinx — 98 commits
• (root) — 87 commits
• (repo) — 71 commits
• spec/thinking_sphinx — 14 commits
• spec/acceptance — 11 commits
• .github/workflows — 7 commits
• .circleci/config.yml — 5 commits
• spec/internal — 4 commits
• bin/loadsphinx — 2 commits
• spec/support — 1 commit

• fix: Fix FilterReflection with Rails 7.1
• fix: Fix a LogSubscriber deprecation in Rails 7.1
• fix: Fix attribute access with alternative primary keys.
• fix: Fix building sphinx 2.2.11 on Ubuntu 18
• fix: Fix deletion callbacks with alternative primary keys.
• fix: Fix for ActiveRecord 6.0.0.beta2 join generation.
• fix: Fix issue link in CHANGELOG.
• fix: Fix kwarg expectations with new rspec
• fix: Fix merge reference for ‘none’ scope
• fix: Fix mysql2 dependencies in Appraisals.
• fix: Fix real-time indices with non-integer primary keys.
• fix: Fix respond_to with scopes
• fix: Fix spec for version check.
• fix: Fix update callbacks with alternative primary keys.
• fix: Merge pull request #1090 from reamaze/octopus-fix
• fix: Merge pull request #1170 from pat/bundler-ci-fix
• fix: Merge pull request #1239 from pat/fix/default-cutoff
• fix: Revert "Remove rubygems update from build process."
• fix: Update CHANGELOG with recent fixes.
• fix: fix: ActiveRecord loading, eager_load_paths
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

pat/thinking-sphinx was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.