SETL-Framework/setl

This system is the SETL framework, a Scala-based library designed to simplify the construction and management of Apache Spark sessions and data connectors. It provides a builder pattern for initializing Spark contexts with support for various storage formats like Parquet, Delta, and Hudi, alongside utilities for configuration loading and data archiving. The codebase includes comprehensive test infrastructure and automation scripts for local development, deployment, and version management.

This change introduces the foundational components of the SETL framework, including the core \Setl\ context for managing Spark repositories and connectors, a \ConfigLoader\ for environment-based configuration, and a \Builder\ pattern for constructing Spark sessions and connectors. It adds support for multiple storage formats (CSV, Parquet, Delta, Hudi, JDBC, etc.) via the \Storage\ enum and connector configurations, along with new capabilities for data compression (XZ, GZIP, Snappy), benchmarking factory execution times, and handling external deliveries. The release also establishes a structured exception hierarchy and defines interface stability annotations to guide API evolution.

src/main · high confidence

The dev directory now includes a suite of shell scripts to streamline project maintenance and testing. The change-scala-version.sh script allows developers to switch the project's Scala version between 2.11 and 2.12 by automatically updating relevant pom.xml files. Deployment is handled by deploy-release.sh and deploy-snapshot.sh, which configure Maven settings and GPG keys before executing the build. For local testing, test.sh sets up fake AWS credentials and runs the test suite with coverage reporting, while docker-compose.yml provides containerized instances of PostgreSQL, Cassandra, and DynamoDB to support unit tests that previously relied on embedded dependencies.

dev · high confidence

Added comprehensive test suite for Setl framework components; Added test fixtures for recursive file listing; Added test resource files for connector and configuration testing; Added test resources for StructuredStreamingConnector; Added test resources for ZipArchiver validation.

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 69.

• Code Health 99
• Architecture 87
• Maturity 66
• Readiness 70
• Security 65

• 300 commits — 254 feature/other, 46 fixes

• (root) — 149 commits
• src/main — 65 commits
• src/test — 34 commits
• (repo) — 12 commits
• .github/workflows — 12 commits
• docs/data_access_layer — 7 commits
• .github/ISSUE_TEMPLATE — 4 commits
• dev/deploy-release.sh — 4 commits
• docs/Connector.md — 4 commits
• .github/dependabot.yml — 2 commits
• docs/img — 2 commits
• .github/stale.yml — 1 commit
• dev/test.sh — 1 commit
• docs/Setl.md — 1 commit
• docs/_config.yml — 1 commit
• docs/index.md — 1 commit

• fix: WIP Fix Delta issue (#234)
• fix: [SETL-120] fix: Setl.hasExternalInput always return false (#121)
• fix: [SETL-57] Fix Mermaid URL generation (#58)
• fix: [SETL-74] fix: Spark master URL can't be overwritten in local environment (#75)
• fix: [SETL-97] fix: FileConnector doesn't list path recursively when there are subdirectories inside the given path (#98)
• fix: fix broken link
• fix: fix: add JDBCConnector constructor for ConnectorBuilder
• fix: fix: add back deploy plugin
• fix: fix: add maven cli compatible version options
• fix: fix: add pom.xml for of dc-spark-sdk_2.12
• fix: fix: add profile to mvn cmd
• fix: fix: add test fixture to avoid RpcEnv error
• fix: fix: artifact path
• fix: fix: broken link on documentation (#178)
• fix: fix: change DynamoDB connector DF reader format class name
• fix: fix: change badges' url
• fix: fix: change dependency scope for spark connectors
• fix: fix: change sonatype repo url
• fix: fix: ci
• fix: fix: ci
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

SETL-Framework/setl was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.