Skip to content
CAI
Software that uses CAICheck a score

ray-project/raydp

57.2

Adequate · 6 November 2026

10.4k

lines of production code

Scala

with Python, Java

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add Docker support for RayDP on Kubernetes

Users can now build and deploy RayDP on Kubernetes using the provided Dockerfile, build script, and Kubernetes configuration. The Dockerfile installs Java 8 and RayDP on top of the Ray base image, while the included \legacy.yaml\ file provides a Kubernetes cluster configuration for running Ray head and worker nodes with appropriate resource limits and shared memory settings.

docker · high confidence

Add MPI job execution support on Ray

Introduces a new \raydp.mpi\ module that enables running MPI jobs on Ray clusters. This includes a \create\_mpi\_job\ API to launch jobs with support for OpenMPI, Intel MPI, and MPICH, along with the underlying worker and driver services for process coordination and function execution.

python/raydp/mpi · high confidence

Add NYC Taxi Fare Prediction examples with data processing and training scripts

The examples directory now includes a complete NYC Taxi Fare Prediction workflow, featuring a random dataset generator (\random\_nyctaxi.py\) and a preprocessing script (\data\_process.py\) that handles feature engineering and JDK 17 compatibility options. Users can now run distributed training using PyTorch via \TorchEstimator\ (\pytorch\_nyctaxi.py\), Horovod (\horovod\_nyctaxi.py\), or the Ray Train API (\raytrain\_nyctaxi.py\), alongside a DLRM implementation notebook and a submission helper script.

examples · high confidence

Add Spark 3.4.0–3.4.4 shim layer

This change introduces a new shim implementation for Spark versions 3.4.0 through 3.4.4, enabling RayDP to run on these specific Spark releases. The shim provides version-specific adapters for core operations, including DataFrame creation from Arrow batches, Arrow schema conversion, command-line argument handling via SparkSubmitUtils, and the instantiation of custom executor backends (RayCoarseGrainedExecutorBackend) through a dedicated factory. This allows the platform to correctly bridge RayDP's internal APIs with the Spark 3.4.x runtime environment.

core/shims/spark340 · high confidence

Add Spark 4.2 shim support

This change adds a new shim module for Apache Spark 4.2 (version 4.2.0) to enable RayDP compatibility with this specific Spark minor line. The implementation includes the necessary service provider configuration, a shim provider descriptor, and concrete shim classes that bridge RayDP internals to Spark 4.2 APIs. Specifically, it provides utilities for converting DataFrames to and from Arrow formats, creates a custom executor backend factory for Spark 4.2, and handles command-line submission arguments, allowing users to run RayDP workloads on Spark 4.2 clusters.

core/shims/spark330, core/shims/spark350, core/shims/spark420 · high confidence

Add support for Spark 4.0 and 4.1 shims

New shim implementations for Spark 4.0 and 4.1 have been added to the core/shims/spark400 and core/shims/spark410 directories. These changes introduce version-specific providers, SQL utilities, executor backend factories, and task context utilities, enabling RayDP to run on Spark 4.x patch releases.

core/shims/spark400, core/shims/spark410 · high confidence

Added MPI network communication layer with gRPC services

The \python/raydp/mpi/network\ module now includes the protocol definition and generated Python code for a gRPC-based communication layer between RayDP driver and worker processes. This adds the \DriverService\ (handling worker registration and function result reporting) and \WorkerService\ (handling function execution and shutdown) definitions, enabling the underlying MPI job execution infrastructure to coordinate tasks across distributed nodes.

python/raydp/mpi/network · high confidence

Initial project scaffolding and documentation

The repository is initialized with core project files including an Apache 2.0 LICENSE, a SECURITY.md policy pointing to the Intel Security Center, and a CONTRIBUTING.md guide for setting up the mixed Scala/Python development environment (requiring JDK 8/17, Python 3.10+, and Maven 3.6+). A build.sh script is added to compile the core module and Python wheel, alongside a README.md detailing the RayDP architecture, installation, and usage for running Spark on Ray and integrating with AI libraries.

(repo-wide) · high confidence

Initial release of RayDP 1.7.0.dev0 with Spark-on-Ray integration

This change introduces the RayDP library (version 1.7.0.dev0), providing a Python API to run Apache Spark on Ray. Users can now initialize a Spark cluster within a Ray environment using \raydp.init\_spark()\, which supports configuration for executors, memory, and placement groups. The release includes a context manager for session lifecycle management, an abstract \EstimatorInterface\ for machine learning workflows, and utilities for parsing memory sizes and handling Spark DataFrames. It also introduces version-aware configuration selection for Spark's logging system (log4j vs log4j2) based on the installed PySpark version.

python/raydp · high confidence

Introduce TFEstimator for distributed TensorFlow Keras training

Users can now train TensorFlow Keras models in a distributed manner using a scikit-learn-like API via the new TFEstimator class. This component leverages Ray Train's TensorflowTrainer to handle distributed execution, allowing users to specify model architectures, optimizers, losses, and metrics while automatically managing data sharding and worker resources.

python/raydp/tf · high confidence

New XGBoostEstimator API for distributed training

Users can now train XGBoost models using a new \XGBoostEstimator\ class in the \raydp.xgboost\ module. This estimator wraps Ray's \XGBoostTrainer\ to provide a unified interface for distributed training, supporting both direct Ray Dataset inputs and Spark DataFrames via \fit\_on\_spark\. It allows configuration of worker counts, per-worker resources, data shuffling, and checkpointing, and exposes the trained model via \get\_model\.

python/raydp/xgboost · high confidence

New raydp-submit script for launching Spark on Ray

A new \bin/raydp-submit\ shell script has been added to simplify submitting Spark applications to a Ray cluster. This script automatically detects \SPARK\_HOME\, \RAY\_HOME\, and the driver node IP, and injects necessary JVM arguments (including JDK 17+ module access flags) and Spark configurations (such as the Ray driver host and log4j settings) before invoking \SparkSubmit\.

bin · high confidence

New tutorials for PyTorch and Ray Train integration

Added two new Colab-compatible Jupyter notebooks demonstrating how to use RayDP with PyTorch and Ray Train. The PyTorch example (\pytorch\_example.ipynb\) shows an end-to-end pipeline using Spark for data preprocessing and a distributed estimator for training, including a fix to add the \torchmetrics\ dependency. The Ray Train example (\raytrain\_example.ipynb\) illustrates using the new Ray Train API for distributed training and evaluation. Both tutorials use the healthcare stroke prediction dataset (\healthcare-dataset-stroke-data.csv\) to predict stroke risk based on patient data.

tutorials · high confidence

Behavioural changes

10 commits (0 fixes) modifying core/raydp-main/src/main/java

A change to existing behaviour in core/raydp-main/src/main/java — 10 commits, 6 files: “\[Fault tolerance\] Make executor actor restartable and fetch data from blockManager in actor calls (\#249)”; “Changes for executor actor recovery using ray's fault tolerance. (\#391)”; “remove usage of ray.put(data, \_owner) and private ray object ownership manipulation API (\#454)”; and 7 more (refs \#249, \#391, \#454, \#318, \#349, …).

core/raydp-main/src/main/java · medium confidence · unverified

12 commits (3 fixes) modifying repository

A change to existing behaviour in repository — 12 commits (3 fixs), 6 files: “fix(RayAppMaster): refactor executor state in RayAppMaster. (\#487)”; “Support --py-files to distribute files to executors (\#441)”; “\[Fault tolerance\] Make executor actor restartable and fetch data from blockManager in actor calls (\#249)”; and 9 more (refs \#487, \#441, \#249, \#450, \#391, …).

repository · medium confidence · unverified

21 commits (6 fixes) modifying core/raydp-main/src/main/scala

A change to existing behaviour in core/raydp-main/src/main/scala — 21 commits (6 fixs), 14 files: “fix(RayAppMaster): refactor executor state in RayAppMaster. (\#487)”; “dynamic allocation executor pending for addition flood fix (\#396)”; “fix executor custom resource request+support pyarrow \>=15 & spark 3.5.2/3/4 (\#419)”; and 18 more (refs \#487, \#396, \#419, \#249, \#450, …).

core/raydp-main/src/main/scala · medium confidence · unverified

3 commits (0 fixes) modifying core/shims/common

A change to existing behaviour in core/shims/common — 3 commits, 4 files: “Support Spark 4.x (\#450)”; “Support every Spark 4.x patch release and add Spark 4.2 (\#490)”; “update to use Spark 3.3 (\#253)” (refs \#450, \#490, \#253).

core/shims/common · medium confidence · unverified

Introduce TorchEstimator with Ray Train integration and Intel oneCCL support

The \python/raydp/torch\ module now provides a \TorchEstimator\ that leverages the Ray Train API for distributed PyTorch training, replacing previous internal implementations. This change introduces a new \TorchConfig\ that enables the Intel oneCCL backend for distributed process group initialization, requiring \intel\_extension\_for\_pytorch\ and \torch-ccl\. Users can now specify evaluation metrics via \torchmetrics\, configure placement strategies for Ray actors, and utilize the \TorchMLDataset\ wrapper for efficient data loading from Ray's MLDataset.

python/raydp/torch · high confidence

Introduce recoverable Spark-to-Ray Dataset conversion and SparkCluster configuration

The \python/raydp/spark\ module now exposes a recoverable pipeline for converting Spark DataFrames to Ray Datasets via \from\_spark\_recoverable\ and \spark\_dataframe\_to\_ray\_dataset\, which caches Arrow bytes in Spark and builds Ray Dataset blocks using Ray task lineage to survive executor loss. The \SparkCluster\ class has been updated to automatically set the Spark driver host to the Ray node IP address and to support custom Spark locations by picking up the \$SPARK\_HOME\ environment variable.

python/raydp/spark · high confidence

Introduces Java agent to redirect SLF4J logs and prepend job IDs to worker logs

Adds a new Java agent (Agent.java) and SLF4J binding (StaticLoggerBinder.java) to the RayDP core. The agent intercepts system output/error streams during startup to redirect SLF4J binding warnings and logs into a dedicated file (slf4j-\<pid\>.log) instead of cluttering the Spark shell console, then restores the original streams. Additionally, for Ray versions 2.3.0–2.3.1, it prepends the job ID in the format ':job\_id:\<jobid\>' to the beginning of the java-worker log file to comply with Ray's log parsing requirements, using explicit UTF-8 encoding for file paths and content.

core/agent · high confidence

Fixes

3 commits (2 fixes) fixing core/shims/spark322

A fix in core/shims/spark322 — 3 commits (2 fixs), 7 files.

core/shims/spark322 · medium confidence · unverified

Test coverage

1 commit adding/updating tests in core/raydp-main/src/main/test; Added test suite for RayDP core, estimators, and utilities; Added tests for Spark version parsing and application actor-slot state management.

Dependencies

Python package initialization and dependency configuration

The Python package is initialized with a new MANIFEST.in and pylintrc configuration. The setup.py defines the package metadata, specifying a base version of 1.7.0 and enforcing Python 3.10+ compatibility. It establishes strict dependency constraints, requiring Ray \>= 2.37.0, PySpark \>= 4.0.0, PyArrow \>= 4.0.1, and Protobuf \> 3.19.5, while also including optional TensorFlow support.

python · high confidence

RayDP 1.7.0 dependency and shim updates

This change updates the RayDP build to version 1.7.0-SNAPSHOT, raising the default Spark target to 4.0.0 and adding dedicated shim modules for Spark 4.0.0, 4.1.0, and 4.2. It upgrades the Ray dependency to 2.1.0, bumps Jackson to 2.18.2, Rhino to 1.7.14.1, and commons-lang3 to 3.18.0, while introducing a new core/agent module with log4j 2.25.3 and slf4j 1.7.32.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

This is the PUBLIC form of this artifact. Findings are listed in full, but the details of SECURITY findings — which rule fired, in which file, on which line, and how to fix it — are deliberately withheld, and any secret-scanner results are excluded entirely. Where detail is absent here it was REMOVED FOR PUBLICATION; it is not missing from the analysis. The complete artifact is available from the repository owner.

Score

  • CAI 58 → 57 (-0.6)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 55 → 55 (+0.0)
  • Architecture 100 → 100 (+0.0)
  • Maturity 61 → 59 (-1.4)
  • Readiness 50 → 50 (+0.0)
  • Security 72 → 66 (-5.3)
  • Performance 100 → 100 (+0.0)

Resolved (4)

  • High: security finding (details withheld)
  • No AppArmor/SELinux confinement (docker/legacy.yaml)
  • No seccomp profile (docker/legacy.yaml)
  • Off-boarding risk: anonymized user #1

New (2)

  • High: security finding (details withheld)
  • Off-boarding risk: anonymized user #1

Changes since last survey

  • 1 commits — 0 feature/other, 1 fixes

By area

  • .github/workflows — 1 commit

Notable commits

  • fix: fix: cap pyarrow below 26 for the TensorFlow extras and release workflow (#493)

Architecture

  • Unchanged — 0 containers · 2 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

ray-project/raydp was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 6 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 3e0e59aae38bde7e9aaa3825651913ceb83a5d09 — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-60d1b390edf0.