SETL-Framework/setl
67.4
Adequate · 7 November 2026
7.8k
lines of production code
Scala
with Java
4
measurements over time
What this system is
This system is the SETL framework, a Scala-based library designed to simplify the construction and management of Apache Spark sessions and data connectors. It provides a builder pattern for initializing Spark contexts with support for various storage formats like Parquet, Delta, and Hudi, alongside utilities for configuration loading and data archiving. The codebase includes comprehensive test infrastructure and automation scripts for local development, deployment, and version management.
Features
Initial release of the SETL framework core
This change introduces the foundational components of the SETL framework, including the core \Setl\ context for managing Spark repositories and connectors, a \ConfigLoader\ for environment-based configuration, and a \Builder\ pattern for constructing Spark sessions and connectors. It adds support for multiple storage formats (CSV, Parquet, Delta, Hudi, JDBC, etc.) via the \Storage\ enum and connector configurations, along with new capabilities for data compression (XZ, GZIP, Snappy), benchmarking factory execution times, and handling external deliveries. The release also establishes a structured exception hierarchy and defines interface stability annotations to guide API evolution.
src/main · high confidence
New development and deployment automation scripts
The dev directory now includes a suite of shell scripts to streamline project maintenance and testing. The change-scala-version.sh script allows developers to switch the project's Scala version between 2.11 and 2.12 by automatically updating relevant pom.xml files. Deployment is handled by deploy-release.sh and deploy-snapshot.sh, which configure Maven settings and GPG keys before executing the build. For local testing, test.sh sets up fake AWS credentials and runs the test suite with coverage reporting, while docker-compose.yml provides containerized instances of PostgreSQL, Cassandra, and DynamoDB to support unit tests that previously relied on embedded dependencies.
dev · high confidence
Test coverage
Added comprehensive test suite for Setl framework components; Added test fixtures for recursive file listing; Added test resource files for connector and configuration testing; Added test resources for StructuredStreamingConnector; Added test resources for ZipArchiver validation.
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 55 → 67 (+12.1)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 75 → 75 (+0.0)
- Architecture 81 → 81 (+0.0)
- Maturity 42 → 66 (+24.4)
- Readiness 63 → 64 (+0.8)
- Security 66 → 72 (+6.3)
Resolved (270)
- Documentation: no contributor guidance (README.md)
- Documentation: no project overview (README.md)
- Medium CVE: [GHSA redacted] (pom.xml)
- Outdated (compile): com.amazonaws:aws-java-sdk-core
- Outdated (compile): com.amazonaws:aws-java-sdk-dynamodb
- Outdated (compile): com.amazonaws:aws-java-sdk-kms
- Outdated (compile): com.amazonaws:aws-java-sdk-s3
- Outdated (compile): com.amazonaws:aws-java-sdk-sts
- Outdated (compile): com.amazonaws:jmespath-java
- Outdated (compile): com.crealytics:spark-excel_2.12
- Outdated (compile): com.datastax.oss:java-driver-core-shaded
- Outdated (compile): com.datastax.oss:java-driver-mapper-runtime
- Outdated (compile): com.datastax.oss:java-driver-query-builder
- Outdated (compile): com.datastax.oss:native-protocol
- Outdated (compile): com.datastax.spark:spark-cassandra-connector-driver_2.12
- Outdated (compile): com.datastax.spark:spark-cassandra-connector_2.12
- Outdated (compile): com.fasterxml.jackson.core:jackson-annotations
- Outdated (compile): com.fasterxml.jackson.core:jackson-core
- Outdated (compile): com.fasterxml.jackson.core:jackson-databind
- Outdated (compile): com.fasterxml.jackson.dataformat:jackson-dataformat-cbor
- …and 250 more
New (4)
- Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- Medium CVE: [GHSA redacted] (pom.xml)
Architecture
- Unchanged — 0 containers · 1 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
SETL-Framework/setl was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 68f4e0213f5b6793acb96b2d9e08c102439565c4 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-5f8314471808.