zouzias/spark-lucenerdd
52.2
Adequate · 7 November 2026
4.2k
lines of production code
Scala
primary language
4
measurements over time
What this system is
This system is a Scala library that integrates Apache Lucene with Apache Spark to enable distributed full-text search, faceted search, and geospatial indexing across RDDs and DataFrames. It provides core capabilities for text analysis, entity linkage, and deduplication, alongside specialized support for spatial queries using shape-based indexing. The library allows users to configure analyzers, similarity metrics, and storage modes to perform complex search operations in a distributed environment.
Features
Add TermDocMatrix for Lucene field term-document analysis
A new TermDocMatrix class has been introduced in the matrices package to represent a term-document matrix derived from a Lucene field. This component takes an RDD of TermVectorEntry objects and constructs a Spark CoordinateMatrix where rows correspond to unique terms and columns correspond to unique documents (identified by document ID and shard partition). It provides methods to retrieve the mapping between matrix indices and actual terms or documents, as well as statistics on the matrix dimensions and non-zero entry count, enabling downstream machine learning or statistical operations on term frequencies.
src/main/scala/org/zouzias/spark/lucenerdd/matrices · high confidence
Add reference configuration for LuceneRDD settings
A new reference.conf file has been added to provide default configuration values for the LuceneRDD library. This file defines settings for the text analyzer (defaulting to 'en'), similarity scoring (defaulting to 'bm25'), and linkage methods. It also configures index storage modes (defaulting to 'disk'), string field analysis options, and query parameters such as the maximum and default top-K result counts. Additionally, it includes spatial configuration defaults for the prefix tree data structure (defaulting to 'quad') and shape I/O format (defaulting to 'WKT'). These settings can be overridden via environment variables or application configuration.
src/main/resources · high confidence
Added SparkFacetResultMonoid for aggregating facet results
A new SparkFacetResultMonoid object has been introduced in the aggregate package to handle the aggregation of faceted search results. This component implements a monoid structure that combines SparkFacetResult instances from executors to the driver, ensuring that facets with matching names are summed correctly using an underlying MapMonoid for the facet counts.
src/main/scala/org/zouzias/spark/lucenerdd/aggregate · high confidence
Added example scripts for Spark LuceneRDD usage
New Scala scripts have been added to the scripts directory to demonstrate how to use the Spark LuceneRDD library. These examples include loading text data (Alice in Wonderland, city names, and word lists) into a LuceneRDD and performing operations such as counting, term queries, and more-like-this searches, as well as loading structured CSV data (H1B Visa records) to index specific columns.
scripts · high confidence
Added fuzzy and prefix linkage example scripts
New example scripts have been added to the record linkage module to demonstrate how to perform fuzzy and prefix-based record matching using the LuceneRDD library. The \linkageFuzzyExample.scala\ script shows how to link records using a fuzzy matching function with a specified fuzziness level, while \linkagePrefixExample.scala\ demonstrates linking based on string prefixes. These examples provide users with ready-to-run code for common linkage patterns.
scripts/recordLinkage · high confidence
Added new geographic dataset files
The data directory now includes new Parquet dataset files for capitals, countries (bounding box and polygon), and world cities points, indicated by the addition of their respective \_SUCCESS markers.
data · high confidence
Added spatial data loading scripts for Solr linkage and Swiss cities
New scripts have been added to the spatial module to facilitate spatial data processing. The \loadSolrSpatialData.scala\ script demonstrates how to load country and capital city shapes from Parquet files, create a ShapeLuceneRDD, and link them by radius to find nearby capitals. Additionally, \loadSwissCities.scala\ provides a script for loading Swiss city coordinates from a TSV file into a ShapeLuceneRDD for spatial indexing.
scripts/spatial · high confidence
Configurable Lucene analyzers via configuration file
Users can now specify the text analyzers used for indexing and querying through configuration keys (lucenerdd.index.analyzer.name and lucenerdd.query.analyzer.name). The system supports short language codes (e.g., 'en', 'de', 'fr') to select built-in Lucene analyzers, defaults to English if unspecified, and allows loading custom analyzer classes by their full class name if they are present on the classpath.
src/main/scala/org/zouzias/spark/lucenerdd/analyzers · high confidence
Initial project scaffolding and documentation
This change introduces the foundational structure for the spark-lucenerdd project. It adds a comprehensive README detailing the library's capabilities (LuceneRDD, FacetedLuceneRDD, ShapeLuceneRDD), supported query types, and compatibility matrix for Spark versions up to 3.5.0. It also includes configuration for the build system (version.sbt, scalastyle-config.xml), CI/CD pipelines (.travis.yml, codecov integration), and developer tooling such as a spark-shell helper script, a Docker Compose setup for Zeppelin, and deployment instructions for Sonatype.
(repo-wide) · high confidence
Initial release of LuceneRDD with entity linkage and serialization support
This change introduces the core LuceneRDD component, enabling users to perform distributed search, deduplication, and entity linkage across Spark RDDs and DataFrames. The new API includes methods for querying (query, exists), deduplicating data (dedup), and linking external datasets (linkDataFrame, linkByQuery). It also provides index statistics (IndexStatistics, FieldStatistics) for monitoring and a Kryo registrator to ensure efficient serialization of Lucene objects and Spark data structures.
src/main/scala/org/zouzias/spark/lucenerdd · high confidence
Introduce FacetedLuceneRDD for faceted search capabilities
This change adds the FacetedLuceneRDD component, enabling users to perform faceted search queries on LuceneRDD data. The new class provides facetQuery and facetQueries methods to aggregate results by specific fields, while the accompanying package.scala defines implicit conversions that automatically index primitive types, tuples, maps, and Spark Rows with facet-specific fields (suffixed with \_facet or \_numFacet) to support these aggregations. Additionally, a Versionable trait is introduced to expose project version information to Spark.
src/main/scala/org/zouzias/spark/lucenerdd/facets · high confidence
Introduce ShapeLuceneRDD for geospatial and full-text search
This change adds a new ShapeLuceneRDD component that enables combined geospatial and full-text search capabilities. Users can now link RDDs using K-nearest neighbors (linkByKnn) or distance-based radius searches (linkByRadius), with support for implicit conversions from common Scala types (tuples, strings, lists, arrays) to spatial shapes via the package object. The implementation includes Kryo serialization support for efficient distributed processing and integrates with Spark SQL DataFrames for structured data linkage.
src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape · high confidence
Introduction of SpatialStrategy trait for Lucene spatial indexing
A new \SpatialStrategy\ trait has been added to the \org.zouzias.spark.lucenerdd.spatial.shape.strategies\ package. This trait extends \PrefixTreeLoader\ and initializes a \RecursivePrefixTreeStrategy\ from Apache Lucene, providing a unified API for indexing and searching spatial shapes with distance calculations. This change establishes the foundational strategy component for spatial data handling within the library.
src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape/strategies · high confidence
New LuceneRDDResponse type with DataFrame conversion and top-k retrieval
The response handling in the LuceneRDD module has been refactored to introduce a new \LuceneRDDResponse\ class (alongside \FieldType\ and \LuceneRDDResponsePartition\). This change adds the ability to convert search results directly into a Spark DataFrame via the \toDF()\ method, simplifying downstream data analysis. Additionally, the response now supports efficient top-k result retrieval using a \TopKMonoid\ for scoring-based ordering, and overrides \collect()\ to handle result aggregation more robustly.
src/main/scala/org/zouzias/spark/lucenerdd/response · high confidence
New model classes for facets, search results, and term vectors
Added three new model classes to the library: SparkFacetResult, which wraps Lucene facet results into a Spark-friendly structure with sorted facet counts; SparkScoreDoc, which represents a scored search result document and provides conversion to Spark SQL Rows with automatic type inference for numeric and text fields; and TermVectorEntry, which models term vector data including document ID, term text, and count.
src/main/scala/org/zouzias/spark/lucenerdd/models · high confidence
Partition-level Lucene indexing and query API introduced
The \src/main/scala/org/zouzias/spark/lucenerdd/partition\ area now contains the core partition implementation (\LuceneRDDPartition\) and its abstract base (\AbstractLuceneRDDPartition\). This change introduces the internal mechanism for constructing per-partition Lucene indexes, applying analyzers and similarity metrics, and executing various search operations (term, prefix, fuzzy, phrase, multi-term, and More-Like-This queries) via the new \LuceneRDDResponsePartition\ return type. Users relying on this library benefit from the underlying infrastructure that enables partitioned indexing and diverse query capabilities within the Spark RDD context.
src/main/scala/org/zouzias/spark/lucenerdd/partition · high confidence
Behavioural changes
Configurable disk-based Lucene index storage with taxonomy support
The library now supports storing Lucene indexes and taxonomy data on disk rather than exclusively in memory. A new \IndexStorable\ trait manages index and taxonomy directories in the system's temporary folder, using \MMapDirectory\ for disk storage (selected via the \lucenerdd.index.store.mode\ config parameter set to 'disk') and falling back to \RAMDirectory\ for in-memory storage. The \IndexWithTaxonomyWriter\ trait integrates these storage mechanisms with index and taxonomy writers, enabling facet queries to persist data on disk, which improves performance and resource usage for large indexes by leveraging the OS file system cache instead of the Java heap.
src/main/scala/org/zouzias/spark/lucenerdd/store · high confidence
Configurable search similarity and new query helper utilities
Users can now configure the Lucene search similarity algorithm via the \lucenerdd.similarity.name\ setting, choosing between BM25 and the default Classic similarity. Additionally, a new \LuceneQueryHelpers\ object provides utility methods for parsing queries with per-field analyzers, executing faceted text searches using taxonomy readers, and retrieving field names or total document counts from the index.
src/main/scala/org/zouzias/spark/lucenerdd/query · high confidence
Introduce PrefixTreeLoader for spatial grid configuration
A new PrefixTreeLoader trait has been added to the spatial grids module to centralize the configuration of spatial prefix trees. This component initializes the underlying Lucene SpatialPrefixTree based on configurable parameters such as the tree type (e.g., geohash or quad), maximum levels for precision, and maximum distance error, providing a standardized way to load spatial grid structures for shape-based operations.
src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape/grids · high confidence
Introduces configurable traits for LuceneRDD settings
The library now exposes a set of Scala traits (Configurable, LuceneRDDConfigurable, ShapeLuceneRDDConfigurable) that allow users to externalize and customize core indexing and querying parameters via Typesafe Config. This includes defaults for top-K query results, facet counts, string field analysis options (such as term vectors, positions, and index options), and spatial search settings like prefix tree configuration and shape formats. Users can now adjust these behaviors through configuration files rather than hard-coded values.
src/main/scala/org/zouzias/spark/lucenerdd/config · high confidence
Refactored spatial partitioning with explicit search interfaces
The spatial partitioning logic in the shape module has been restructured to introduce an abstract base class (\AbstractShapeLuceneRDDPartition\) and a concrete implementation (\ShapeLuceneRDDPartition\). This change exposes explicit methods for nearest-neighbor search (\knnSearch\), circle search (\circleSearch\), arbitrary spatial search (\spatialSearch\), and bounding-box search (\bboxSearch\), allowing users to perform targeted spatial queries. The implementation also adds logging for indexing start, completion, and duration, providing visibility into indexing performance.
src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape/partition · high confidence
Test coverage
Added LuceneRDDTestUtils for spatial and scoring test assertions; Added comprehensive test suite for LuceneRDD core and linkage features; Added test coverage for ShapeLuceneRDD spatial and linkage operations; Added test data models for Kryo serialization tests; Added test data resources for text and geospatial analysis; Added tests for FacetedLuceneRDD facet queries and case class integration; Added tests for LuceneQueryHelpers; Added tests for ShapeLuceneRDD implicits; Added tests for analyzer configuration; Added unit tests for LuceneRDDResponse.
Dependencies
Initial build configuration for Spark Lucene RDD
The project now includes a \build.sbt\ file that defines the build structure, specifying Scala 2.12.19 and Apache Spark 3.5.9 as provided dependencies. It integrates Apache Lucene 8.11.4 (including facet, analyzer, query parser, expressions, and spatial extras modules) alongside supporting libraries such as JTS Core 1.19.0, Spatial4J 0.8, Algebird 0.13.10, and Joda Time 2.12.7. The configuration also sets up test dependencies using ScalaTest 3.2.20 and spark-testing-base 3.5.1\_1.5.3, enables cross-building for Scala 2.12.19, and configures Maven-style publishing to Sonatype.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 53 → 52 (-0.7)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 62 → 62 (+0.0)
- Architecture 98 → 98 (+0.0)
- Maturity 44 → 44 (+0.0)
- Readiness 47 → 46 (-1.1)
- Security 87 → 82 (-5.2)
Resolved (8)
- Outdated: com.typesafe:config
- Outdated: joda-time:joda-time
- Outdated: org.apache.lucene:lucene-expressions
- Outdated: org.apache.lucene:lucene-facet
- Outdated: org.apache.lucene:lucene-queryparser
- Outdated: org.apache.lucene:lucene-spatial-extras
- Outdated: org.joda:joda-convert
- Outdated: org.locationtech.jts:jts-core
New (1)
- Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
zouzias/spark-lucenerdd was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 2d4eb039d684c5bb596254316cb85651b979940e — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-110e4fee044f.