Skip to content
CAI
Software that uses CAICheck a score

ClickHouse/spark-clickhouse-connector

66.2

Adequate · 7 November 2026

19.2k

lines of production code

Scala

with Java

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add default Docker configuration files for Kyuubi, Spark, Hive, and CloudBeaver

This change introduces a set of default configuration files for the Docker environment, establishing the baseline setup for the platform's services. It includes \kyuubi-defaults.conf\ and \spark-defaults.conf\ which configure Spark to use S3 storage, register various catalogs (TPCDS, TPCH, Postgres, Iceberg, and multiple ClickHouse instances), and enable async inserts for ClickHouse. Additionally, it adds \core-site.xml\ and \hive-site.xml\ to define S3 and Hive Metastore connections, and a \cloudbeaver-conf\ directory containing \cloudbeaver.conf\, \initial-data-sources.conf\, and \product.conf\ to set up the CloudBeaver web interface with a pre-configured Kyuubi connection and specific security and UI settings.

docker/conf · high confidence

Added Spark 4.0 example applications for debugging and testing

New Scala example applications have been added to the \examples/scala/spark-4.0\ directory to facilitate debugging and testing of the ClickHouse connector. The \SimpleBatchExample\ demonstrates batch processing by creating sample employee data, writing it to a ClickHouse table via the catalog-aware API, and performing read-back and aggregation operations. The \StreamingRateExample\ illustrates streaming capabilities by generating synthetic data using a rate source, enriching it, and writing it to ClickHouse in micro-batches using \foreachBatch\. Both examples are configured to connect to a local ClickHouse instance and include instructions for setting breakpoints in the connector code.

examples/scala/spark-4.0 · high confidence

Added standalone Spark 3.5 example application

A new standalone Scala example has been added for Spark 3.5 that demonstrates how to configure a SparkSession to connect to a ClickHouse catalog. The example sets up connection parameters such as host, port, and SSL options, and executes a simple query to list tables, providing a reference implementation for users integrating Spark with ClickHouse.

examples/scala/spark-3.5 · high confidence

Initial Spark 3.4 connector support with catalog and table provider APIs

The connector now supports Spark 3.4, introducing a new \ClickHouseCatalog\ for namespace and table management, a \ClickHouseTableProvider\ for format-based access (e.g., \spark.read.format("clickhouse")\), and a \ClickHouseCommandRunner\ for external SQL execution. This release also adds built-in ClickHouse hash functions (CityHash64, MurmurHash2/3) via a function registry, configurable timezone handling, and detailed write metrics (batch sizes, flush counts, serialization time) to help users monitor and tune data ingestion performance.

spark-3.4/clickhouse-spark/src/main · high confidence

Initial Spark 3.5 connector support with catalog and format APIs

The connector now supports Spark 3.5, introducing a new \ClickHouseCatalog\ for catalog-based table access and a \ClickHouseTableProvider\ that enables the \format("clickhouse")\ read/write pattern. This location provides the core wiring for these capabilities, including the \DataSourceRegister\ service file, helper traits for SQL and connection management, a command runner for external SQL execution, and a function registry with built-in hash functions (e.g., cityHash64, murmurHash) and shard-number mapping to support distributed table writes.

spark-3.5/clickhouse-spark/src/main · high confidence

Initial Spark 4.0 connector implementation with catalog and format support

This change introduces the initial implementation of the ClickHouse connector for Spark 4.0. It adds a \ClickHouseCatalog\ to support the Spark catalog API (enabling table discovery and function resolution) and a \ClickHouseTableProvider\ to support the \format("clickhouse")\ API for ad-hoc queries and table creation. The implementation includes a \DataSourceRegister\ service file to register the provider, helper traits for SQL compilation and cluster/node specification, and a function registry for ClickHouse-specific hash functions.

spark-4.0/clickhouse-spark · high confidence

Initial repository structure and documentation for Spark ClickHouse Connector

The repository is initialized with the foundational structure for the Spark ClickHouse Connector, including the Apache 2.0 license, a comprehensive CHANGELOG documenting versions from 0.5.0 through 0.10.1, and a README detailing usage, requirements (Java 8/17, Scala 2.12/2.13, Spark 3.3–4.0), and build instructions. The project uses Gradle for building and testing, with a wrapper script and configuration files (\.scalafmt.conf\, \.gitattributes\) added to enforce code style and version control settings. Documentation guidelines for contributors and AI agents are also included.

(repo-wide) · high confidence

Introduce Docker-based one-click playground with centralized version management

Users can now spin up a complete local development and testing environment using Docker Compose, which orchestrates Kyuubi, Spark, ClickHouse, MinIO, PostgreSQL, and CloudBeaver. The setup is configured via new \.env\ and \.env-dev\ files that centralize version definitions for the stack, including Spark 3.5.2, Kyuubi 1.9.2, Iceberg 1.6.0, and ClickHouse 25.3 (stable) or 23.8 (dev). A new \build-image.sh\ script allows developers to build custom images with cross-platform support via buildx, while the \compose.yml\ and \compose-dev.yml\ files provide pre-configured services for immediate use, including a web-based SQL interface via CloudBeaver.

docker · high confidence

Introduce Spark 3.3 connector with catalog-based and format-based access

The Spark 3.3 connector is now available, providing both catalog-based access via the new ClickHouseCatalog (supporting namespaces, functions, and table management) and format-based access via the ClickHouseTableProvider (enabling .format("clickhouse") reads and writes). The connector registers itself through the DataSourceRegister service file and includes helper utilities for SQL compilation, schema inference, and metadata columns, along with custom metrics for monitoring read/write performance.

spark-3.3/clickhouse-spark/src/main · high confidence

Introduce modular Docker image build system for Spark components

The project now provides a set of layered Dockerfiles (scc-base, scc-hadoop, scc-spark, scc-kyuubi, scc-metastore) that construct the runtime environment for the Spark Connector. This structure allows users to build images with specific versions of Spark, Hadoop, Kyuubi, and Hive, while automatically including necessary JDBC drivers (PostgreSQL, ClickHouse), cloud storage libraries (AWS SDK, Hadoop Cloud Storage), and TPC-DS/TPC-H connectors. The base image uses Eclipse Temurin 8 on Ubuntu Focal, and the Spark image is pre-configured with the ClickHouse Spark runtime jar and Iceberg support.

docker/image · high confidence

New developer utility scripts for backporting and code formatting

Added two new executable scripts to the dev directory to streamline maintenance tasks. The \dev/backport\ script allows developers to automatically backport a specific commit from one supported Spark version (3.4 or 3.5) to another (3.3 or 3.4) by generating and applying the necessary patch. The \dev/reformat\ script simplifies code style enforcement by running the Spotless formatter across all supported Spark binary versions (3.3, 3.4, and 3.5) in a single step.

dev · high confidence

Behavioural changes

17 commits (5 fixes) modifying clickhouse-core/src/main

A change to existing behaviour in clickhouse-core/src/main — 17 commits (5 fixs), 30 files: “feat: implement SupportsPushDownTopN (top-N pushdown) across spark-3.3/3.4/3.5/4.0 (\#543)”; “feat(write): add flush count, min/max batch size, and connection write metrics (\#558)”; “fix: remove false-positive Databricks signatures from RuntimeDetector (\#536)”; and 14 more (refs \#543, \#558, \#536, \#471, \#428, …).

clickhouse-core/src/main · medium confidence · unverified

Test coverage

4 commits adding/updating tests in spark-3.4/clickhouse-spark/src/test/scala/com, spark-3.5/clickhouse-spark/src/test/scala/com; Added Cloud test tag for ClickHouse Cloud test filtering; Added integration tests for ClickHouse cluster operations; Added integration tests for Spark 3.5 single-node connector; Added integration tests for hash functions and utility parsing; Added log4j configuration for test logging; Added test fixtures for capturing log warnings; Added test infrastructure and utilities for Spark 3.3 integration tests; Added test suites for Spark 3.4 connector configuration, expression handling, and schema mapping; Added tests for ClickHouse SQL engine clause parsing; Added tests for cluster node sorting, shard calculation, and table engine resolution; Added tests for write metrics and read plan stability; Added unit tests for ClickHouse Spark connector configuration, expression, and schema utilities; Added unit tests for ClickHouse Spark connector internals; Added unit tests for NodeClient configuration warnings and WriteMetricsProjection logic; Added unit tests for column parsing and utility functions; Added unit tests for metrics, sharding, scan identity, and write sorting; Integration test suite for Spark 4.0 ClickHouse connector; Integration tests for ClickHouse cluster operations; Refactored test infrastructure with new ClickHouse provider traits and retry utilities.

Dependencies

Build system upgraded to support Spark 4.0 and Java 21

The build configuration has been updated to introduce support for Apache Spark 4.0 (version 4.0.1) and align the project with Java 21. This includes bumping the Gradle Shadow plugin to 8.1.1 and adding ASM 9.6 to the buildscript classpath to enable shading of Java 21 compiled classes. Additionally, the build now supports Scala 2.13.16 for Spark 4.0 and updates the default ClickHouse JDBC client to version 0.9.5.

(dependencies) · high confidence

Upgrade Gradle wrapper to version 8.9

The project's Gradle wrapper has been updated to use Gradle 8.9. This change ensures that builds are executed with the specific 8.9 distribution, providing users with the latest features, performance improvements, and bug fixes included in this version of the Gradle build tool.

gradle · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 66 → 66 (+0.6)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 58 → 58 (-0.0)
  • Architecture 100 → 100 (+0.0)
  • Maturity 69 → 72 (+3.5)
  • Readiness 72 → 72 (+0.0)
  • Security 70 → 70 (+0.0)

Resolved (284)

  • Dependency hygiene PARTLY measured — Gradle build resolved part of the dependency graph, currency reported but not scored
  • Documentation: no installation or build instructions (README.md)
  • Documentation: no project overview (README.md)
  • Duplicated block (14 lines × 3) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Outdated (compile): ch.qos.reload4j:reload4j
  • Outdated (compile): com.clickhouse:clickhouse-jdbc
  • Outdated (compile): com.clickhouse:client-v2
  • Outdated (compile): com.dimafeng:testcontainers-scala-clickhouse_2.13
  • Outdated (compile): com.dimafeng:testcontainers-scala-core_2.13
  • Outdated (compile): com.dimafeng:testcontainers-scala-jdbc_2.13
  • Outdated (compile): com.dimafeng:testcontainers-scala-scalatest_2.13
  • Outdated (compile): com.eed3si9n:shaded-jawn-parser_2.13
  • Outdated (compile): com.eed3si9n:sjson-new-core_2.13
  • Outdated (compile): com.eed3si9n:sjson-new-scalajson_2.13
  • Outdated (compile): com.esotericsoftware:minlog
  • Outdated (compile): com.fasterxml.jackson.core:jackson-annotations
  • Outdated (compile): com.fasterxml.jackson.core:jackson-core
  • …and 264 more

New (9)

  • Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
  • Duplicated block (12 lines × 4) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (spark-3.3/clickhouse-spark/src/main/scala/com/clickhouse/spark/ClickHouseTableProvider.scala)
  • Orphaned file (spark-3.3/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SchemaUtils.scala)
  • Orphaned file (spark-3.3/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SparkOptions.scala)
  • Orphaned file (spark-3.4/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SparkOptions.scala)
  • Orphaned file (spark-3.5/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SparkOptions.scala)

Changes since last survey

  • 4 commits — 3 feature/other, 1 fixes

By area

  • (root) — 3 commits
  • .github/workflows — 1 commit

Notable commits

  • fix: fix: stop leaking ClickHouse credentials into logs and CI artifacts (#590)
  • change: Bump version to 0.10.3-SNAPSHOT (#592)
  • change: Release 0.10.2 (#591)
  • change: ci: switch Cloud CI to the CONNECTORS_TEAM_TESTS_CLOUD_* secrets (#593)

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

ClickHouse/spark-clickhouse-connector was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit affab517acf40cd226b92f1cc249b2c781f055b5 — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-5f8314471808.