Skip to content
CAI
Software that uses CAICheck a score

streamnative/pulsar-spark

56.0

Adequate · 7 November 2026

4.6k

lines of production code

Scala

primary language

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

This system is a unified data processing connector for Apache Pulsar and Apache Spark, enabling structured streaming and batch operations. It provides capabilities to read from and write to Pulsar topics using Spark SQL, including support for JSON and Avro schemas. The implementation handles offset management, schema mapping, and configuration for authentication and error handling.

Features

Added Maven Wrapper configuration

The project now includes a Maven Wrapper (maven-wrapper.properties) that pins the build environment to Maven 3.9.0 and the wrapper JAR version 3.2.0. This ensures that all developers and CI systems use a consistent, reproducible Maven version without requiring a pre-installed Maven binary.

.mvn · high confidence

Initial release of the Spark-Pulsar connector

This change introduces the initial implementation of the Apache Spark connector for Apache Pulsar, enabling users to read from and write to Pulsar topics using Spark SQL and Structured Streaming. The release includes the core provider logic, batch and streaming source/sink implementations, and utilities for serializing and deserializing data (including JSON and Avro schemas). It also adds configuration options for authentication, topic management, and error handling, along with a service registration file to make the connector discoverable by Spark.

src/main · high confidence

Introduce Pulsar source connector for Spark Structured Streaming

This change adds the core implementation for the Apache Pulsar source connector, enabling Spark Structured Streaming to read data from Pulsar topics. The new codebase includes the \PulsarSource\ and \PulsarMicroBatchReader\ classes to manage streaming offsets and batch processing, \PulsarHelper\ for interacting with the Pulsar client and admin APIs, and \PulsarSourceRDD\ for executing the actual data reading on executors. It also introduces configuration management via \PulsarConfigUpdater\, offset serialization logic in \PulsarSourceUtils\, and schema mapping utilities in \SchemaUtils\ to convert Pulsar schema types into Spark SQL data types.

repository · high confidence

Behavioural changes

Added Scalastyle configuration and license template

The project now includes a Scalastyle configuration file (\scalastyle-config.xml\) that enforces code style rules, such as header matching, line length limits, and naming conventions, along with a standard Apache 2.0 license template (\license.template\) to ensure consistent licensing across source files.

src/resources · high confidence

Project renamed to pulsar-spark with updated compatibility matrix and build tooling

The project has been renamed from SegmentX to pulsar-spark, reflecting its role as a unified data processing connector for Apache Pulsar and Apache Spark. The README now documents a version compatibility matrix supporting Spark 4.1.1, Pulsar 3.0+, and Scala 2.13, alongside updated usage examples for reading and writing data. Additionally, the build environment has been standardized with the addition of Maven Wrapper scripts (mvnw/mvnw.cmd) and a .scalafmt configuration, while the .gitignore has been expanded to cover IDE and build artifacts.

(repo-wide) · high confidence

Test coverage

Added test suite for Pulsar Spark connector

Added comprehensive tests for the Pulsar Spark connector, including initialization checks, JSON utility serialization, Pulsar admission control, configurable client factories, configuration updates, micro-batch source behavior (including TTL and data loss handling), available-now triggers, sink operations, and offset management.

src/test · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 57 → 56 (-0.6)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 73 → 73 (+0.0)
  • Architecture 98 → 98 (+0.0)
  • Maturity 46 → 43 (-3.2)
  • Readiness 61 → 62 (+0.4)
  • Security 61 → 67 (+5.7)

Resolved (215)

  • Critical CVE: [GHSA redacted] (pom.xml)
  • Critical CVE: [GHSA redacted] (pom.xml)
  • Critical CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • High CVE: [GHSA redacted] (pom.xml)
  • Low CVE: [GHSA redacted] (pom.xml)
  • Medium CVE: [GHSA redacted] (pom.xml)
  • Medium CVE: [GHSA redacted] (pom.xml)
  • Medium CVE: [GHSA redacted] (pom.xml)
  • …and 195 more

New (2)

  • Critical CVE: [GHSA redacted] (pom.xml)
  • Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

streamnative/pulsar-spark was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit f9a497f257ef5fcd3fc3811f1e4cfece1c5a59d5 — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-110e4fee044f.