swoop-inc/spark-alchemy
46.4
Weak · 7 November 2026
1.2k
lines of production code
Scala
with JavaScript
4
measurements over time
What this system is
Spark-Alchemy is a Scala library for Apache Spark that provides HyperLogLog++ functions for approximate distinct counting and cardinality estimation. It enables users to create, merge, and intersect composable sketches with configurable error rates, supporting pluggable backends like StreamLib and Aggregate Knowledge. The system includes comprehensive testing infrastructure to validate accuracy and interoperability with PostgreSQL.
Features
Initial project scaffolding and documentation
This change introduces the foundational structure for the spark-alchemy project, including the README, development guidelines, and versioning strategy. It establishes the supported SDK versions via .sdkmanrc (Java 8, SBT 1.6.2, Scala 2.12.17) and sets up the local development environment with a docker-compose configuration for Postgres interoperability testing. The entry also includes standard project files like .gitignore, NOTICE, and code style settings, marking the initial public release of the library's core documentation and build infrastructure.
(repo-wide) · high confidence
New HyperLogLog++ functions for cardinality estimation
Alchemy now provides a suite of HyperLogLog++ functions for approximate distinct counting, including \hll\_init\, \hll\_init\_collection\, \hll\_init\_agg\, \hll\_init\_collection\_agg\, \hll\_merge\, \hll\_row\_merge\, \hll\_cardinality\, \hll\_intersect\_cardinality\, and \hll\_convert\. These functions allow users to create composable sketches, merge them across partitions, and estimate cardinalities with configurable error rates. The implementation supports pluggable backends (StreamLib and Aggregate Knowledge) and uses a specialized hash function to better differentiate data types and handle nulls compared to Spark's built-in approximations.
alchemy/src/main · high confidence
Test coverage
Added test infrastructure and unit tests for HyperLogLog functions
Added comprehensive test coverage for the new HyperLogLog (HLL) functions, including \hll\_init\, \hll\_merge\, \hll\_cardinality\, \hll\_row\_merge\, and \hll\_intersect\_cardinality\. The changes introduce \CardinalityHashFunctionTest\ and \HLLFunctionsTest\ to verify cardinality estimation accuracy for simple types and collections, validate null handling, and ensure correct behavior for both the Aggregate Knowledge (AGKN) and StreamLib (STRM) implementations. A new \PostgresInteropTest\ validates that HLL aggregates produced in Spark match results from PostgreSQL when using the AGKN implementation. Additionally, the test suite includes new test utilities (\SparkSessionSpec\, \DebugFilesystem\, \SparkFunSuite\, \PlanTest\, \SQLHelper\, \SQLTestData\, \SQLTestUtils\, \SharedSparkSessionBase\, \TestSparkSession\) to support isolated Spark session management, resource leak detection, and SQL plan comparison.
alchemy/src/test · high confidence
Dependencies
Initial project setup with Spark 3.5.2 and Scala 2.12.15
The build configuration establishes the project's core dependencies, targeting Apache Spark 3.5.2 and Scala 2.12.15. It includes the HyperLogLog++ library (net.agkn:hll:1.6.0) for cardinality estimation, PostgreSQL and ScalaTest for testing, and configures the microsite documentation generation using GitHub4s.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 48 → 46 (-1.9)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 74 → 74 (+0.0)
- Architecture 69 → 69 (+0.0)
- Maturity 54 → 54 (+0.0)
- Readiness 29 → 28 (-1.4)
- Security 86 → 80 (-5.7)
New (1)
- Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
Architecture
- Unchanged — 0 containers · 1 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
swoop-inc/spark-alchemy was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 58293e4cceabdd6a1c963733eddf46100d80b6a9 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-5f8314471808.