Skip to content
CAI
Software that uses CAICheck a score

zouzias/spark-lucenerdd

52.2

Adequate · 7 November 2026

4.2k

lines of production code

Scala

primary language

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

This system is a Scala library that integrates Apache Lucene with Apache Spark to enable distributed full-text search, faceted search, and geospatial indexing across RDDs and DataFrames. It provides core capabilities for text analysis, entity linkage, and deduplication, alongside specialized support for spatial queries using shape-based indexing. The library allows users to configure analyzers, similarity metrics, and storage modes to perform complex search operations in a distributed environment.

Features

Add TermDocMatrix for Lucene field term-document analysis

A new TermDocMatrix class has been introduced in the matrices package to represent a term-document matrix derived from a Lucene field. This component takes an RDD of TermVectorEntry objects and constructs a Spark CoordinateMatrix where rows correspond to unique terms and columns correspond to unique documents (identified by document ID and shard partition). It provides methods to retrieve the mapping between matrix indices and actual terms or documents, as well as statistics on the matrix dimensions and non-zero entry count, enabling downstream machine learning or statistical operations on term frequencies.

src/main/scala/org/zouzias/spark/lucenerdd/matrices · high confidence

Add reference configuration for LuceneRDD settings

A new reference.conf file has been added to provide default configuration values for the LuceneRDD library. This file defines settings for the text analyzer (defaulting to 'en'), similarity scoring (defaulting to 'bm25'), and linkage methods. It also configures index storage modes (defaulting to 'disk'), string field analysis options, and query parameters such as the maximum and default top-K result counts. Additionally, it includes spatial configuration defaults for the prefix tree data structure (defaulting to 'quad') and shape I/O format (defaulting to 'WKT'). These settings can be overridden via environment variables or application configuration.

src/main/resources · high confidence

Added SparkFacetResultMonoid for aggregating facet results

A new SparkFacetResultMonoid object has been introduced in the aggregate package to handle the aggregation of faceted search results. This component implements a monoid structure that combines SparkFacetResult instances from executors to the driver, ensuring that facets with matching names are summed correctly using an underlying MapMonoid for the facet counts.

src/main/scala/org/zouzias/spark/lucenerdd/aggregate · high confidence

Added example scripts for Spark LuceneRDD usage

New Scala scripts have been added to the scripts directory to demonstrate how to use the Spark LuceneRDD library. These examples include loading text data (Alice in Wonderland, city names, and word lists) into a LuceneRDD and performing operations such as counting, term queries, and more-like-this searches, as well as loading structured CSV data (H1B Visa records) to index specific columns.

scripts · high confidence

Added fuzzy and prefix linkage example scripts

New example scripts have been added to the record linkage module to demonstrate how to perform fuzzy and prefix-based record matching using the LuceneRDD library. The \linkageFuzzyExample.scala\ script shows how to link records using a fuzzy matching function with a specified fuzziness level, while \linkagePrefixExample.scala\ demonstrates linking based on string prefixes. These examples provide users with ready-to-run code for common linkage patterns.

scripts/recordLinkage · high confidence

Added new geographic dataset files

The data directory now includes new Parquet dataset files for capitals, countries (bounding box and polygon), and world cities points, indicated by the addition of their respective \_SUCCESS markers.

data · high confidence

Added spatial data loading scripts for Solr linkage and Swiss cities

New scripts have been added to the spatial module to facilitate spatial data processing. The \loadSolrSpatialData.scala\ script demonstrates how to load country and capital city shapes from Parquet files, create a ShapeLuceneRDD, and link them by radius to find nearby capitals. Additionally, \loadSwissCities.scala\ provides a script for loading Swiss city coordinates from a TSV file into a ShapeLuceneRDD for spatial indexing.

scripts/spatial · high confidence

Configurable Lucene analyzers via configuration file

Users can now specify the text analyzers used for indexing and querying through configuration keys (lucenerdd.index.analyzer.name and lucenerdd.query.analyzer.name). The system supports short language codes (e.g., 'en', 'de', 'fr') to select built-in Lucene analyzers, defaults to English if unspecified, and allows loading custom analyzer classes by their full class name if they are present on the classpath.

src/main/scala/org/zouzias/spark/lucenerdd/analyzers · high confidence

Initial project scaffolding and documentation

This change introduces the foundational structure for the spark-lucenerdd project. It adds a comprehensive README detailing the library's capabilities (LuceneRDD, FacetedLuceneRDD, ShapeLuceneRDD), supported query types, and compatibility matrix for Spark versions up to 3.5.0. It also includes configuration for the build system (version.sbt, scalastyle-config.xml), CI/CD pipelines (.travis.yml, codecov integration), and developer tooling such as a spark-shell helper script, a Docker Compose setup for Zeppelin, and deployment instructions for Sonatype.

(repo-wide) · high confidence

Initial release of LuceneRDD with entity linkage and serialization support

This change introduces the core LuceneRDD component, enabling users to perform distributed search, deduplication, and entity linkage across Spark RDDs and DataFrames. The new API includes methods for querying (query, exists), deduplicating data (dedup), and linking external datasets (linkDataFrame, linkByQuery). It also provides index statistics (IndexStatistics, FieldStatistics) for monitoring and a Kryo registrator to ensure efficient serialization of Lucene objects and Spark data structures.

src/main/scala/org/zouzias/spark/lucenerdd · high confidence

Introduce FacetedLuceneRDD for faceted search capabilities

This change adds the FacetedLuceneRDD component, enabling users to perform faceted search queries on LuceneRDD data. The new class provides facetQuery and facetQueries methods to aggregate results by specific fields, while the accompanying package.scala defines implicit conversions that automatically index primitive types, tuples, maps, and Spark Rows with facet-specific fields (suffixed with \_facet or \_numFacet) to support these aggregations. Additionally, a Versionable trait is introduced to expose project version information to Spark.

src/main/scala/org/zouzias/spark/lucenerdd/facets · high confidence

This change adds a new ShapeLuceneRDD component that enables combined geospatial and full-text search capabilities. Users can now link RDDs using K-nearest neighbors (linkByKnn) or distance-based radius searches (linkByRadius), with support for implicit conversions from common Scala types (tuples, strings, lists, arrays) to spatial shapes via the package object. The implementation includes Kryo serialization support for efficient distributed processing and integrates with Spark SQL DataFrames for structured data linkage.

src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape · high confidence

Introduction of SpatialStrategy trait for Lucene spatial indexing

A new \SpatialStrategy\ trait has been added to the \org.zouzias.spark.lucenerdd.spatial.shape.strategies\ package. This trait extends \PrefixTreeLoader\ and initializes a \RecursivePrefixTreeStrategy\ from Apache Lucene, providing a unified API for indexing and searching spatial shapes with distance calculations. This change establishes the foundational strategy component for spatial data handling within the library.

src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape/strategies · high confidence

New LuceneRDDResponse type with DataFrame conversion and top-k retrieval

The response handling in the LuceneRDD module has been refactored to introduce a new \LuceneRDDResponse\ class (alongside \FieldType\ and \LuceneRDDResponsePartition\). This change adds the ability to convert search results directly into a Spark DataFrame via the \toDF()\ method, simplifying downstream data analysis. Additionally, the response now supports efficient top-k result retrieval using a \TopKMonoid\ for scoring-based ordering, and overrides \collect()\ to handle result aggregation more robustly.

src/main/scala/org/zouzias/spark/lucenerdd/response · high confidence

New model classes for facets, search results, and term vectors

Added three new model classes to the library: SparkFacetResult, which wraps Lucene facet results into a Spark-friendly structure with sorted facet counts; SparkScoreDoc, which represents a scored search result document and provides conversion to Spark SQL Rows with automatic type inference for numeric and text fields; and TermVectorEntry, which models term vector data including document ID, term text, and count.

src/main/scala/org/zouzias/spark/lucenerdd/models · high confidence

Partition-level Lucene indexing and query API introduced

The \src/main/scala/org/zouzias/spark/lucenerdd/partition\ area now contains the core partition implementation (\LuceneRDDPartition\) and its abstract base (\AbstractLuceneRDDPartition\). This change introduces the internal mechanism for constructing per-partition Lucene indexes, applying analyzers and similarity metrics, and executing various search operations (term, prefix, fuzzy, phrase, multi-term, and More-Like-This queries) via the new \LuceneRDDResponsePartition\ return type. Users relying on this library benefit from the underlying infrastructure that enables partitioned indexing and diverse query capabilities within the Spark RDD context.

src/main/scala/org/zouzias/spark/lucenerdd/partition · high confidence

Behavioural changes

Configurable disk-based Lucene index storage with taxonomy support

The library now supports storing Lucene indexes and taxonomy data on disk rather than exclusively in memory. A new \IndexStorable\ trait manages index and taxonomy directories in the system's temporary folder, using \MMapDirectory\ for disk storage (selected via the \lucenerdd.index.store.mode\ config parameter set to 'disk') and falling back to \RAMDirectory\ for in-memory storage. The \IndexWithTaxonomyWriter\ trait integrates these storage mechanisms with index and taxonomy writers, enabling facet queries to persist data on disk, which improves performance and resource usage for large indexes by leveraging the OS file system cache instead of the Java heap.

src/main/scala/org/zouzias/spark/lucenerdd/store · high confidence

Configurable search similarity and new query helper utilities

Users can now configure the Lucene search similarity algorithm via the \lucenerdd.similarity.name\ setting, choosing between BM25 and the default Classic similarity. Additionally, a new \LuceneQueryHelpers\ object provides utility methods for parsing queries with per-field analyzers, executing faceted text searches using taxonomy readers, and retrieving field names or total document counts from the index.

src/main/scala/org/zouzias/spark/lucenerdd/query · high confidence

Introduce PrefixTreeLoader for spatial grid configuration

A new PrefixTreeLoader trait has been added to the spatial grids module to centralize the configuration of spatial prefix trees. This component initializes the underlying Lucene SpatialPrefixTree based on configurable parameters such as the tree type (e.g., geohash or quad), maximum levels for precision, and maximum distance error, providing a standardized way to load spatial grid structures for shape-based operations.

src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape/grids · high confidence

Introduces configurable traits for LuceneRDD settings

The library now exposes a set of Scala traits (Configurable, LuceneRDDConfigurable, ShapeLuceneRDDConfigurable) that allow users to externalize and customize core indexing and querying parameters via Typesafe Config. This includes defaults for top-K query results, facet counts, string field analysis options (such as term vectors, positions, and index options), and spatial search settings like prefix tree configuration and shape formats. Users can now adjust these behaviors through configuration files rather than hard-coded values.

src/main/scala/org/zouzias/spark/lucenerdd/config · high confidence

Refactored spatial partitioning with explicit search interfaces

The spatial partitioning logic in the shape module has been restructured to introduce an abstract base class (\AbstractShapeLuceneRDDPartition\) and a concrete implementation (\ShapeLuceneRDDPartition\). This change exposes explicit methods for nearest-neighbor search (\knnSearch\), circle search (\circleSearch\), arbitrary spatial search (\spatialSearch\), and bounding-box search (\bboxSearch\), allowing users to perform targeted spatial queries. The implementation also adds logging for indexing start, completion, and duration, providing visibility into indexing performance.

src/main/scala/org/zouzias/spark/lucenerdd/spatial/shape/partition · high confidence

Test coverage

Added LuceneRDDTestUtils for spatial and scoring test assertions; Added comprehensive test suite for LuceneRDD core and linkage features; Added test coverage for ShapeLuceneRDD spatial and linkage operations; Added test data models for Kryo serialization tests; Added test data resources for text and geospatial analysis; Added tests for FacetedLuceneRDD facet queries and case class integration; Added tests for LuceneQueryHelpers; Added tests for ShapeLuceneRDD implicits; Added tests for analyzer configuration; Added unit tests for LuceneRDDResponse.

Dependencies

Initial build configuration for Spark Lucene RDD

The project now includes a \build.sbt\ file that defines the build structure, specifying Scala 2.12.19 and Apache Spark 3.5.9 as provided dependencies. It integrates Apache Lucene 8.11.4 (including facet, analyzer, query parser, expressions, and spatial extras modules) alongside supporting libraries such as JTS Core 1.19.0, Spatial4J 0.8, Algebird 0.13.10, and Joda Time 2.12.7. The configuration also sets up test dependencies using ScalaTest 3.2.20 and spark-testing-base 3.5.1\_1.5.3, enables cross-building for Scala 2.12.19, and configures Maven-style publishing to Sonatype.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 53 → 52 (-0.7)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 62 → 62 (+0.0)
  • Architecture 98 → 98 (+0.0)
  • Maturity 44 → 44 (+0.0)
  • Readiness 47 → 46 (-1.1)
  • Security 87 → 82 (-5.2)

Resolved (8)

  • Outdated: com.typesafe:config
  • Outdated: joda-time:joda-time
  • Outdated: org.apache.lucene:lucene-expressions
  • Outdated: org.apache.lucene:lucene-facet
  • Outdated: org.apache.lucene:lucene-queryparser
  • Outdated: org.apache.lucene:lucene-spatial-extras
  • Outdated: org.joda:joda-convert
  • Outdated: org.locationtech.jts:jts-core

New (1)

  • Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

zouzias/spark-lucenerdd was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 2d4eb039d684c5bb596254316cb85651b979940e — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-110e4fee044f.