leobenkel/ZparkIO

ZparkIO is a Scala library that integrates the ZIO effect system with Apache Spark, enabling structured, effect-based data processing workflows. It provides abstractions for managing Spark sessions, command-line arguments, and logging within a ZIO runtime, while offering utilities to safely transform Spark datasets. The system supports cross-compilation across multiple Spark and Scala versions and includes test helpers for validating Spark integration.

7 changes.

The ZparkIO project was initialized with core configuration, build automation, and development standards, establishing a baseline structure for version 1.1.0. The library underwent a significant migration to ZIO 2.x, introducing a new application lifecycle trait, environment types, and internal service abstractions for CLI, logging, and Spark session management.

6 changes.

This period focused on integrating the Zparkio framework with ZIO by introducing ZIO-based service implementations for logging, command-line parsing, and dataset operations. The work also established a robust testing infrastructure through new helper utilities, test coverage for core components, and example applications demonstrating the framework's capabilities. Additionally, cross-version build scripts were added to support development across multiple Spark versions.

Added two new example applications, Example1\_mini and Example2\_small, demonstrating the Zparkio framework's integration with ZIO. Example1\_mini provides a minimal entry point with basic Spark context creation and CLI argument parsing via Scallop. Example2\_small illustrates a more complex workflow involving parallel database queries for users and posts, followed by data transformations to identify authors, supported by dedicated services for FileIO and Database access.

_examples/Example1\_mini, examples/Example2\small · high confidence

New shell scripts have been added to the scripts directory to automate SBT operations across multiple Spark versions. These scripts (spark-cross-clean, spark-cross-compile, spark-cross-fmt, spark-cross-publish, spark-cross-test, and spark-cross-version) iterate through versions listed in the sparkVersions file, allowing users to clean, compile, format, test, publish, or check versions for each Spark target. The publish script additionally supports specifying a project and snapshot status.

scripts · high confidence

The repository has been initialized with the core configuration files required for the ZparkIO project, including a detailed \UPGRADE\_NOTES.md\ documenting dependency updates (such as Spark 3.5.6 support and ZIO 2.1.9), a \soteria.json\ security policy, \scalafmt.conf\ and \scalastyle-config.xml\ for code formatting and linting, and a \Makefile\ for build automation. This entry establishes the project's baseline structure, versioning (\v1.1.0\), and development standards.

(repo-wide) · high confidence

This change adds a new implementation for command-line argument parsing using the Scallop library, replacing the previous approach. It introduces a ZIO-integrated service layer that handles option validation, help message formatting, and error reporting (including specific handling for help requests and unknown options). The implementation includes a custom converter for environment arguments and is accompanied by unit tests covering valid inputs, missing required options, and unknown options.

configLibs/Scallop · high confidence

A new Environment type has been added to the library, providing a sealed trait with case objects for Local, Development, Staging, and Production environments. This change introduces a parseEnv function that accepts a string input and returns the corresponding environment object, enabling more robust and type-safe environment configuration within the application.

Library/src/main/scala/com/leobenkel/zparkio/Env · high confidence

The library introduces three new internal service modules: CommandLineArguments, Logger, and SparkModule. CommandLineArguments provides a typed service layer for managing command-line argument validation and debugging output. Logger defines a generic logging interface with a default implementation that writes to the console, supporting info, error, and debug levels. SparkModule offers a ZIO-managed service for accessing the SparkSession and its configuration, allowing for more structured dependency injection and session management within the application.

Library/src/main/scala/com/leobenkel/zparkio/Services · high confidence

The TestHelper module now includes three new Scala components to support testing: a \LoggerService\ that implements the \Logger.Service\ interface using standard output; a \SparkExecution\ case class that constructs and runs \spark-submit\ commands with configurable arguments, configurations, and files; and a \TestWithSpark\ trait that extends \DataFrameSuiteBase\ to provide a reusable Spark testing environment with optional UI debugging support.

testModules/TestHelper · high confidence

The library has been upgraded to support ZIO 2.x, replacing the previous runtime model with a new \ZparkioApp\ trait that manages the application lifecycle, environment construction, and error handling using ZIO layers. This change introduces a new \ZFuture\ utility for converting Scala \Future\ instances to ZIO effects during the transition period, and adds implicit helper methods in \implicits.scala\ (such as \zMap\ for datasets and broadcast support) to facilitate working with Spark data structures within the ZIO effect system. Users will need to extend \ZparkioApp\ to define their application logic, which now runs via a ZIO runtime rather than the previous execution model.

Library/src/main/scala/com/leobenkel/zparkio · high confidence

Added LoggerTest service implementation for testing; Added empty test directory placeholder; Added tests for DatasetZ operations; Added tests for ZIO service module failure handling.

The build configuration has been updated to support Spark versions 3.1.3 through 3.5.6, with corresponding Scala 2.12 and 2.13 cross-compilation. Key dependencies have been upgraded to ZIO 2.1.9, ScalaTest 3.2.19, Netty 4.1.115.Final, and Scallop 5.2.0. The project now maps specific Spark versions to compatible spark-testing-base releases and configures crossScalaVersions to ensure compatibility between Spark and Scala versions.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 62.

• Code Health 99
• Architecture 69
• Maturity 56
• Readiness 59
• Security 75

• 300 commits — 285 feature/other, 15 fixes

• (root) — 148 commits
• (repo) — 101 commits
• project/plugins.sbt — 12 commits
• .github/workflows — 11 commits
• Library/src — 9 commits
• project/build.properties — 7 commits
• examples/Example2_small — 4 commits
• configLibs/Scallop — 3 commits
• .github/FUNDING.yml — 1 commit
• Tests/src — 1 commit
• assets/README.md — 1 commit
• project/soteria.sbt — 1 commit
• scripts/spark-cross-clean.sh — 1 commit

• fix: Fix FMT
• fix: Fix Soteria
• fix: Fix Soteria
• fix: Fix backtick encoding
• fix: Fix build, tests, and update dependencies (#349)
• fix: Fix coveralls
• fix: Fix for unaccounted for combination between Spark, Scala and spark-testing-base versions
• fix: Fix master / main branch rename
• fix: Fix spark testing for spark 311
• fix: Fixes
• fix: Fixes more links
• fix: Merge pull request #98 from z4f1r0v/fix-backtick
• fix: Try to fix the release process
• fix: fix
• fix: revert soteria change
• change: Abstract Spark versions to accommodate for different dependency configurations
• change: Add 'Reformat with scalafmt 3.6.0' to .git-blame-ignore-revs
• change: Add alternative
• change: Add build badge
• change: Add converalls repo token
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

leobenkel/ZparkIO was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.