streamnative/pulsar-spark

This system is a unified data processing connector for Apache Pulsar and Apache Spark, enabling structured streaming and batch operations. It provides capabilities to read from and write to Pulsar topics using Spark SQL, including support for JSON and Avro schemas. The implementation handles offset management, schema mapping, and configuration for authentication and error handling.

The project now includes a Maven Wrapper (maven-wrapper.properties) that pins the build environment to Maven 3.9.0 and the wrapper JAR version 3.2.0. This ensures that all developers and CI systems use a consistent, reproducible Maven version without requiring a pre-installed Maven binary.

.mvn · high confidence

This change introduces the initial implementation of the Apache Spark connector for Apache Pulsar, enabling users to read from and write to Pulsar topics using Spark SQL and Structured Streaming. The release includes the core provider logic, batch and streaming source/sink implementations, and utilities for serializing and deserializing data (including JSON and Avro schemas). It also adds configuration options for authentication, topic management, and error handling, along with a service registration file to make the connector discoverable by Spark.

src/main · high confidence

This change adds the core implementation for the Apache Pulsar source connector, enabling Spark Structured Streaming to read data from Pulsar topics. The new codebase includes the \PulsarSource\ and \PulsarMicroBatchReader\ classes to manage streaming offsets and batch processing, \PulsarHelper\ for interacting with the Pulsar client and admin APIs, and \PulsarSourceRDD\ for executing the actual data reading on executors. It also introduces configuration management via \PulsarConfigUpdater\, offset serialization logic in \PulsarSourceUtils\, and schema mapping utilities in \SchemaUtils\ to convert Pulsar schema types into Spark SQL data types.

repository · high confidence

The project now includes a Scalastyle configuration file (\scalastyle-config.xml\) that enforces code style rules, such as header matching, line length limits, and naming conventions, along with a standard Apache 2.0 license template (\license.template\) to ensure consistent licensing across source files.

src/resources · high confidence

The project has been renamed from SegmentX to pulsar-spark, reflecting its role as a unified data processing connector for Apache Pulsar and Apache Spark. The README now documents a version compatibility matrix supporting Spark 4.1.1, Pulsar 3.0+, and Scala 2.13, alongside updated usage examples for reading and writing data. Additionally, the build environment has been standardized with the addition of Maven Wrapper scripts (mvnw/mvnw.cmd) and a .scalafmt configuration, while the .gitignore has been expanded to cover IDE and build artifacts.

(repo-wide) · high confidence

Added comprehensive tests for the Pulsar Spark connector, including initialization checks, JSON utility serialization, Pulsar admission control, configurable client factories, configuration updates, micro-batch source behavior (including TTL and data loss handling), available-now triggers, sink operations, and offset management.

src/test · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 55.

• Code Health 92
• Architecture 96
• Maturity 45
• Readiness 56
• Security 56

• 203 commits — 172 feature/other, 31 fixes

• (root) — 84 commits
• src/main — 51 commits
• segment/src — 22 commits
• connectors/spark — 11 commits
• src/test — 7 commits
• test/common — 6 commits
• .github/CODEOWNERS — 4 commits
• .github/workflows — 4 commits
• .circleci/config.yml — 3 commits
• (repo) — 2 commits
• .github/ISSUE_TEMPLATE — 2 commits
• .github/PULL_REQUEST_TEMPLATE.md — 1 commit
• connectors/hive — 1 commit
• connectors/hive-all — 1 commit
• gradle/wrapper — 1 commit
• src/resources — 1 commit
• test/pulsar — 1 commit
• test/pulsar-common — 1 commit

• fix: Revert "Remove Jenkinsfile (#15)" (#16)
• fix: Add a bug report template
• fix: Fix Build Errors (#6)
• fix: Fix Jenkins build
• fix: Fix PulsarEntryConverterTest
• fix: Fix checkstyle error
• fix: Fix checkstyle issue
• fix: Fix checkstyle issue in PulsarServiceTestCase
• fix: Fix data loss of initial batch and add available now trigger test (#156)
• fix: Fix dependencies
• fix: Fix issue where a message is skipped when messages are cleaned up (#185)
• fix: Fix jackson (#43)
• fix: Fix package location (#10)
• fix: Fix release badge and add bintray & jenkins badges (#12)
• fix: Fix the build failure caused by spark upgrade (#155)
• fix: Fix the command
• fix: Fix the issue of getting initial offset with user provided start (#149)
• fix: Revert "Expose some monitor metrics (#106)" (#119)
• fix: Revert "Remove maven_profiles" (#145)
• fix: Revert "[maven-release-plugin] prepare branch branch-2.4.0
• …and 183 more

• 0 containers · 1 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

streamnative/pulsar-spark was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.