snowflakedb/spark-snowflake
54.2
Adequate · 7 November 2026
11.9k
lines of production code
Scala
primary language
4
measurements over time
What this system is
Features
Initial release of the Snowflake Spark Connector
This change introduces the initial version of the Snowflake Spark Connector, providing the core infrastructure for reading from and writing to Snowflake tables via Spark. It establishes the foundational components including parameter management (Parameters.scala), JDBC interaction (SnowflakeJDBCWrapper.scala), and the Spark Data Source API implementation (SnowflakeRelation.scala). The connector supports data transfer through both direct SELECT queries and cloud storage stages (S3/Azure), handling encryption, compression, and various data formats (CSV, JSON, Parquet). It also includes basic authentication support (OAuth, key pairs) and telemetry logging.
repository · high confidence
Initial repository setup with CI/CD, build, and documentation
The repository is initialized with essential infrastructure files, including a GitHub Actions workflow for integration and cluster testing, a deployment script for publishing to Maven Central, and an SBT-based build system supporting cross-compilation for Spark 3.5 and 4.0. The project adds a Codecov configuration for coverage reporting, a pre-commit hook for secret scanning, and a Scalastyle configuration to enforce code standards. Documentation is provided via a main README, developer notes, and test instructions, alongside example configuration files for Snowflake connections and a standard Apache 2.0 license.
(repo-wide) · high confidence
Spark 3.5 and 4.x compatibility with native Variant support and credential redaction
The connector now includes dedicated source implementations for Spark 3.5 and Spark 4.x. In Spark 4.x, the connector enables native support for the Spark VariantType, allowing VARIANT columns to be read and written directly via JSON staging. For both versions, the connector hardens security by registering sensitive option keys (such as private keys and OAuth secrets) with Spark's plan redaction, ensuring these credentials are masked in logs and the History Server UI.
src/main · high confidence
Behavioural changes
Integration test logging configuration updated
Added new log4j and log4j2 configuration files for the integration test suite to control log verbosity and output. The default test logging is now set to WARN to reduce log volume, while specific configurations are provided for console output, file logging, and compatibility with Spark 3.3, ensuring cleaner test runs and better separation of master/worker logs for security reviews.
src/it/resources · high confidence
Legacy spark-redshift codebase archived and migrated to Snowflake package structure
The legacy \spark-redshift\ codebase has been moved into the \legacy\ directory and restructured under the \net.snowflake.spark.snowflake\ package namespace. This change includes the addition of \Utils.SNOWFLAKE\_SOURCE\_NAME\ to enable compile-time validation of connection strings, and updates to configuration files and documentation (including a new \OLD-README.md\ and tutorial files) to reflect the transition from the Databricks \com.databricks.spark.redshift\ format to the Snowflake-specific implementation.
legacy · high confidence
Test coverage
Add Spark 3.5 integration test infrastructure; Added Mockito compatibility helper for test builds; Added Python test for Spark UDF integration; Added Snowflake-specific integration test infrastructure; Added Spark 4.x unit tests for Variant type handling; Added integration test suites for Snowflake Spark Connector; Added integration tests for SnowflakeSparkUtils; Added integration tests for internal stage file operations; Added placeholder directories for Spark 3.5, 4.0, and 4.1 test suites; Added tests for IO module security, GCP support, and upload abort handling; Added tests for SnowflakeFallbackCatalog error handling and V1Table fallback logic; Added unit tests for Snowflake Spark Connector; Integration tests for Spark 4.x compatibility and native VariantType support.
Dependencies
Upgrade to Spark 4.x support and Snowflake JDBC 4.x
The connector now supports Apache Spark 4.x (including 4.0 and 4.1) and Scala 2.13, requiring Java 17 for compilation and runtime. The Snowflake JDBC driver has been upgraded to version 4.0.2, and the connector version is set to 3.2.2. This update includes specific source directory handling for Spark 4.x, adjusted JVM options for Java 17 module opens, and updated dependency versions for Parquet Avro (1.15.2 for Spark 4.x, 1.13.1 for earlier versions) and Jackson (2.15.2).
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 55 → 54 (-1.1)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 54 → 54 (+0.0)
- Architecture 86 → 84 (-1.9)
- Maturity 51 → 57 (+6.0)
- Readiness 59 → 55 (-3.3)
- Security 65 → 51 (-13.9)
Resolved (6)
- Off-boarding risk: anonymized user #1
- Outdated: com.amazonaws:aws-java-sdk-s3
- Outdated: com.fasterxml.jackson.core:jackson-annotations
- Outdated: com.fasterxml.jackson.core:jackson-core
- Outdated: com.fasterxml.jackson.core:jackson-databind
- Outdated: net.snowflake:snowflake-jdbc
New (2)
- Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
- Off-boarding risk: anonymized user #1
Changes since last survey
- 2 commits — 2 feature/other, 0 fixes
By area
- (root) — 2 commits
Notable commits
- change: SNOW-4355138: Generate Utils.VERSION from sparkConnectorVersion (#680)
- change: bump version to 3.2.3 (#678)
Architecture
- Unchanged — 0 containers · 1 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
snowflakedb/spark-snowflake was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 775f5928be41d28c536b0457a89cb0132e631ed7 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-5f8314471808.