RedisLabs/spark-redis

This system is a connector library that integrates Apache Spark with Redis, enabling data processing via RDDs, DataFrames, and Structured Streaming. It supports reading and writing data to Redis using various storage models, including hashes and binary formats, while handling key patterns and schema inference. The library also provides robust streaming capabilities by consuming from Redis Streams with consumer group management and offset tracking. It is designed to operate in both standalone and cluster environments, with support for secure TLS connections and ACL-based authentication.

6 changes.

The project upgraded its dependencies to Spark 3.2.1 and Scala 2.12 while removing the legacy Redis RDD provider. It introduced initial support for reading and writing DataFrames and streaming data via Redis Streams, integrating these capabilities with both the RDD and Structured Streaming APIs.

12 changes.

This period focused on significantly expanding the test suite for the Redis Spark provider, covering DataFrame, RDD, and Stream operations across standalone and cluster environments. The work included adding specific test suites for ACL authentication, SSL/TLS security, and performance benchmarks, alongside utility improvements for connection handling and configuration validation.

The build system now includes local shell scripts (build/sbt and build/sbt-launch-lib.bash) to manage the SBT build tool. This allows the project to download and run a specific version of SBT automatically, ensuring consistent builds without requiring users to have SBT pre-installed globally.

build · high confidence

This change introduces the core infrastructure for integrating Redis with Spark SQL and Structured Streaming. It adds a DataSourceRegister service file to expose the \org.apache.spark.sql.redis\ format, enabling users to read and write DataFrames using options like \table\, \keys.pattern\, and \model\ (hash or binary). It also implements a Structured Streaming \Source\ (\RedisSource\) that reads from Redis Streams, managing consumer groups and offsets automatically. Additionally, it provides a legacy \RedisContext\ API for RDD-based operations and a \RedisStreamingContext\ for DStream-based streaming from Redis Lists and Streams.

src/main · high confidence

This release introduces the ability to consume data from Redis Streams via both the RDD API and Structured Streaming. Users can now read stream entries using \RedisStreamReceiver\ for Spark Streaming or the new \RedisSourceRdd\ and \RedisSourceConfig\ classes for Structured Streaming, with support for consumer groups, offset management (earliest, latest, or specific IDs), and configurable batch sizes and blocking timeouts.

repository · high confidence

The legacy Redis provider implementation located in \src/com/redislab/provider\ has been removed. This deletion eliminates the \RedisConfig\, \RedisPartition\, \RedisPartitioner\, and the core RDD classes (\RedisKVRDD\, \RedisListRDD\) along with the \RedisFunctions\ trait. Users relying on this specific provider package for reading from or writing to Redis via Spark RDDs will no longer have access to these classes.

src/com · high confidence

Added Spark DataFrame benchmark suites for Redis persistence models; Added TLS test resources for SSL connection testing; Added cluster-specific test suites for DataFrames and Spark SQL; Added standalone test suites for Dataframe operations; Added test environment traits for Redis ACL and SSL configurations; Added test resources for streaming and data validation; Added test suites for connection pool library info, configuration, and ACL support; Added test utilities and unit tests for Redis provider components; Added tests for Redis RDD operations and ACL authentication; Added tests for Redis Stream source offset handling and configuration; Added tests for Redis Streams integration; Added tests for Redis table key extraction patterns; Expanded test coverage for DataFrame operations and authentication.

The spark-redis library has been upgraded to support Apache Spark 3.2.1 and Scala 2.12, replacing the previous Spark 3.0 and Scala 2.12 configuration. This change updates the Jedis client dependency to version 3.9.0 and restructures the Maven build to use the modern scala-maven-plugin, ensuring compatibility with the latest Spark ecosystem versions.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 54 → 60 (+6.2)
• Rubric changed (rubric-2026.09.8 → rubric-2026.09.16) — scores are not directly comparable.

• Code Health 99 → 99 (+0.0)
• Architecture 100 → 100 (+0.0)
• Maturity 38 → 51 (+13.1)
• Readiness 70 → 70 (+0.0)
• Security 54 → 54 (+0.0)

• Documentation: no contributor guidance (README.md)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Off-boarding risk: anonymized user #1

• Off-boarding risk: anonymized user #1
• Scanner failed to run — not a clean result

• Unchanged — 0 containers · 1 contexts · 0 edges

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

RedisLabs/spark-redis was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.