ray-project/raydp
57.2
Adequate · 6 November 2026
10.4k
lines of production code
Scala
with Python, Java
4
measurements over time
What this system is
Features
Add Docker support for RayDP on Kubernetes
Users can now build and deploy RayDP on Kubernetes using the provided Dockerfile, build script, and Kubernetes configuration. The Dockerfile installs Java 8 and RayDP on top of the Ray base image, while the included \legacy.yaml\ file provides a Kubernetes cluster configuration for running Ray head and worker nodes with appropriate resource limits and shared memory settings.
docker · high confidence
Add MPI job execution support on Ray
Introduces a new \raydp.mpi\ module that enables running MPI jobs on Ray clusters. This includes a \create\_mpi\_job\ API to launch jobs with support for OpenMPI, Intel MPI, and MPICH, along with the underlying worker and driver services for process coordination and function execution.
python/raydp/mpi · high confidence
Add NYC Taxi Fare Prediction examples with data processing and training scripts
The examples directory now includes a complete NYC Taxi Fare Prediction workflow, featuring a random dataset generator (\random\_nyctaxi.py\) and a preprocessing script (\data\_process.py\) that handles feature engineering and JDK 17 compatibility options. Users can now run distributed training using PyTorch via \TorchEstimator\ (\pytorch\_nyctaxi.py\), Horovod (\horovod\_nyctaxi.py\), or the Ray Train API (\raytrain\_nyctaxi.py\), alongside a DLRM implementation notebook and a submission helper script.
examples · high confidence
Add Spark 3.4.0–3.4.4 shim layer
This change introduces a new shim implementation for Spark versions 3.4.0 through 3.4.4, enabling RayDP to run on these specific Spark releases. The shim provides version-specific adapters for core operations, including DataFrame creation from Arrow batches, Arrow schema conversion, command-line argument handling via SparkSubmitUtils, and the instantiation of custom executor backends (RayCoarseGrainedExecutorBackend) through a dedicated factory. This allows the platform to correctly bridge RayDP's internal APIs with the Spark 3.4.x runtime environment.
core/shims/spark340 · high confidence
Add Spark 4.2 shim support
This change adds a new shim module for Apache Spark 4.2 (version 4.2.0) to enable RayDP compatibility with this specific Spark minor line. The implementation includes the necessary service provider configuration, a shim provider descriptor, and concrete shim classes that bridge RayDP internals to Spark 4.2 APIs. Specifically, it provides utilities for converting DataFrames to and from Arrow formats, creates a custom executor backend factory for Spark 4.2, and handles command-line submission arguments, allowing users to run RayDP workloads on Spark 4.2 clusters.
core/shims/spark330, core/shims/spark350, core/shims/spark420 · high confidence
Add support for Spark 4.0 and 4.1 shims
New shim implementations for Spark 4.0 and 4.1 have been added to the core/shims/spark400 and core/shims/spark410 directories. These changes introduce version-specific providers, SQL utilities, executor backend factories, and task context utilities, enabling RayDP to run on Spark 4.x patch releases.
core/shims/spark400, core/shims/spark410 · high confidence
Added MPI network communication layer with gRPC services
The \python/raydp/mpi/network\ module now includes the protocol definition and generated Python code for a gRPC-based communication layer between RayDP driver and worker processes. This adds the \DriverService\ (handling worker registration and function result reporting) and \WorkerService\ (handling function execution and shutdown) definitions, enabling the underlying MPI job execution infrastructure to coordinate tasks across distributed nodes.
python/raydp/mpi/network · high confidence
Initial project scaffolding and documentation
The repository is initialized with core project files including an Apache 2.0 LICENSE, a SECURITY.md policy pointing to the Intel Security Center, and a CONTRIBUTING.md guide for setting up the mixed Scala/Python development environment (requiring JDK 8/17, Python 3.10+, and Maven 3.6+). A build.sh script is added to compile the core module and Python wheel, alongside a README.md detailing the RayDP architecture, installation, and usage for running Spark on Ray and integrating with AI libraries.
(repo-wide) · high confidence
Initial release of RayDP 1.7.0.dev0 with Spark-on-Ray integration
This change introduces the RayDP library (version 1.7.0.dev0), providing a Python API to run Apache Spark on Ray. Users can now initialize a Spark cluster within a Ray environment using \raydp.init\_spark()\, which supports configuration for executors, memory, and placement groups. The release includes a context manager for session lifecycle management, an abstract \EstimatorInterface\ for machine learning workflows, and utilities for parsing memory sizes and handling Spark DataFrames. It also introduces version-aware configuration selection for Spark's logging system (log4j vs log4j2) based on the installed PySpark version.
python/raydp · high confidence
Introduce TFEstimator for distributed TensorFlow Keras training
Users can now train TensorFlow Keras models in a distributed manner using a scikit-learn-like API via the new TFEstimator class. This component leverages Ray Train's TensorflowTrainer to handle distributed execution, allowing users to specify model architectures, optimizers, losses, and metrics while automatically managing data sharding and worker resources.
python/raydp/tf · high confidence
New XGBoostEstimator API for distributed training
Users can now train XGBoost models using a new \XGBoostEstimator\ class in the \raydp.xgboost\ module. This estimator wraps Ray's \XGBoostTrainer\ to provide a unified interface for distributed training, supporting both direct Ray Dataset inputs and Spark DataFrames via \fit\_on\_spark\. It allows configuration of worker counts, per-worker resources, data shuffling, and checkpointing, and exposes the trained model via \get\_model\.
python/raydp/xgboost · high confidence
New raydp-submit script for launching Spark on Ray
A new \bin/raydp-submit\ shell script has been added to simplify submitting Spark applications to a Ray cluster. This script automatically detects \SPARK\_HOME\, \RAY\_HOME\, and the driver node IP, and injects necessary JVM arguments (including JDK 17+ module access flags) and Spark configurations (such as the Ray driver host and log4j settings) before invoking \SparkSubmit\.
bin · high confidence
New tutorials for PyTorch and Ray Train integration
Added two new Colab-compatible Jupyter notebooks demonstrating how to use RayDP with PyTorch and Ray Train. The PyTorch example (\pytorch\_example.ipynb\) shows an end-to-end pipeline using Spark for data preprocessing and a distributed estimator for training, including a fix to add the \torchmetrics\ dependency. The Ray Train example (\raytrain\_example.ipynb\) illustrates using the new Ray Train API for distributed training and evaluation. Both tutorials use the healthcare stroke prediction dataset (\healthcare-dataset-stroke-data.csv\) to predict stroke risk based on patient data.
tutorials · high confidence
Behavioural changes
10 commits (0 fixes) modifying core/raydp-main/src/main/java
A change to existing behaviour in core/raydp-main/src/main/java — 10 commits, 6 files: “\[Fault tolerance\] Make executor actor restartable and fetch data from blockManager in actor calls (\#249)”; “Changes for executor actor recovery using ray's fault tolerance. (\#391)”; “remove usage of ray.put(data, \_owner) and private ray object ownership manipulation API (\#454)”; and 7 more (refs \#249, \#391, \#454, \#318, \#349, …).
core/raydp-main/src/main/java · medium confidence · unverified
12 commits (3 fixes) modifying repository
A change to existing behaviour in repository — 12 commits (3 fixs), 6 files: “fix(RayAppMaster): refactor executor state in RayAppMaster. (\#487)”; “Support --py-files to distribute files to executors (\#441)”; “\[Fault tolerance\] Make executor actor restartable and fetch data from blockManager in actor calls (\#249)”; and 9 more (refs \#487, \#441, \#249, \#450, \#391, …).
repository · medium confidence · unverified
21 commits (6 fixes) modifying core/raydp-main/src/main/scala
A change to existing behaviour in core/raydp-main/src/main/scala — 21 commits (6 fixs), 14 files: “fix(RayAppMaster): refactor executor state in RayAppMaster. (\#487)”; “dynamic allocation executor pending for addition flood fix (\#396)”; “fix executor custom resource request+support pyarrow \>=15 & spark 3.5.2/3/4 (\#419)”; and 18 more (refs \#487, \#396, \#419, \#249, \#450, …).
core/raydp-main/src/main/scala · medium confidence · unverified
3 commits (0 fixes) modifying core/shims/common
A change to existing behaviour in core/shims/common — 3 commits, 4 files: “Support Spark 4.x (\#450)”; “Support every Spark 4.x patch release and add Spark 4.2 (\#490)”; “update to use Spark 3.3 (\#253)” (refs \#450, \#490, \#253).
core/shims/common · medium confidence · unverified
Introduce TorchEstimator with Ray Train integration and Intel oneCCL support
The \python/raydp/torch\ module now provides a \TorchEstimator\ that leverages the Ray Train API for distributed PyTorch training, replacing previous internal implementations. This change introduces a new \TorchConfig\ that enables the Intel oneCCL backend for distributed process group initialization, requiring \intel\_extension\_for\_pytorch\ and \torch-ccl\. Users can now specify evaluation metrics via \torchmetrics\, configure placement strategies for Ray actors, and utilize the \TorchMLDataset\ wrapper for efficient data loading from Ray's MLDataset.
python/raydp/torch · high confidence
Introduce recoverable Spark-to-Ray Dataset conversion and SparkCluster configuration
The \python/raydp/spark\ module now exposes a recoverable pipeline for converting Spark DataFrames to Ray Datasets via \from\_spark\_recoverable\ and \spark\_dataframe\_to\_ray\_dataset\, which caches Arrow bytes in Spark and builds Ray Dataset blocks using Ray task lineage to survive executor loss. The \SparkCluster\ class has been updated to automatically set the Spark driver host to the Ray node IP address and to support custom Spark locations by picking up the \$SPARK\_HOME\ environment variable.
python/raydp/spark · high confidence
Introduces Java agent to redirect SLF4J logs and prepend job IDs to worker logs
Adds a new Java agent (Agent.java) and SLF4J binding (StaticLoggerBinder.java) to the RayDP core. The agent intercepts system output/error streams during startup to redirect SLF4J binding warnings and logs into a dedicated file (slf4j-\<pid\>.log) instead of cluttering the Spark shell console, then restores the original streams. Additionally, for Ray versions 2.3.0–2.3.1, it prepends the job ID in the format ':job\_id:\<jobid\>' to the beginning of the java-worker log file to comply with Ray's log parsing requirements, using explicit UTF-8 encoding for file paths and content.
core/agent · high confidence
Fixes
3 commits (2 fixes) fixing core/shims/spark322
A fix in core/shims/spark322 — 3 commits (2 fixs), 7 files.
core/shims/spark322 · medium confidence · unverified
Test coverage
1 commit adding/updating tests in core/raydp-main/src/main/test; Added test suite for RayDP core, estimators, and utilities; Added tests for Spark version parsing and application actor-slot state management.
Dependencies
Python package initialization and dependency configuration
The Python package is initialized with a new MANIFEST.in and pylintrc configuration. The setup.py defines the package metadata, specifying a base version of 1.7.0 and enforcing Python 3.10+ compatibility. It establishes strict dependency constraints, requiring Ray \>= 2.37.0, PySpark \>= 4.0.0, PyArrow \>= 4.0.1, and Protobuf \> 3.19.5, while also including optional TensorFlow support.
python · high confidence
RayDP 1.7.0 dependency and shim updates
This change updates the RayDP build to version 1.7.0-SNAPSHOT, raising the default Spark target to 4.0.0 and adding dedicated shim modules for Spark 4.0.0, 4.1.0, and 4.2. It upgrades the Ray dependency to 2.1.0, bumps Jackson to 2.18.2, Rhino to 1.7.14.1, and commons-lang3 to 3.18.0, while introducing a new core/agent module with log4j 2.25.3 and slf4j 1.7.32.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
This is the PUBLIC form of this artifact. Findings are listed in full, but the details of SECURITY findings — which rule fired, in which file, on which line, and how to fix it — are deliberately withheld, and any secret-scanner results are excluded entirely. Where detail is absent here it was REMOVED FOR PUBLICATION; it is not missing from the analysis. The complete artifact is available from the repository owner.
Score
- CAI 58 → 57 (-0.6)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 55 → 55 (+0.0)
- Architecture 100 → 100 (+0.0)
- Maturity 61 → 59 (-1.4)
- Readiness 50 → 50 (+0.0)
- Security 72 → 66 (-5.3)
- Performance 100 → 100 (+0.0)
Resolved (4)
- High: security finding (details withheld)
- No AppArmor/SELinux confinement (docker/legacy.yaml)
- No seccomp profile (docker/legacy.yaml)
- Off-boarding risk: anonymized user #1
New (2)
- High: security finding (details withheld)
- Off-boarding risk: anonymized user #1
Changes since last survey
- 1 commits — 0 feature/other, 1 fixes
By area
- .github/workflows — 1 commit
Notable commits
- fix: fix: cap pyarrow below 26 for the TensorFlow extras and release workflow (#493)
Architecture
- Unchanged — 0 containers · 2 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
ray-project/raydp was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 6 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 3e0e59aae38bde7e9aaa3825651913ceb83a5d09 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-60d1b390edf0.