neo4j/neo4j-spark-connector
61.1
Adequate · 6 November 2026
8.2k
lines of production code
Scala
with Python
4
measurements over time
What this system is
Features
Add Neo4j and Spark example notebooks
Added new example notebooks (\neo4j\_data\_engineering.ipynb\ and \neo4j\_data\_science.ipynb\) demonstrating data engineering and data science workflows using the Neo4j Connector for Apache Spark. The engineering notebook includes a setup section that configures the Spark environment (using Spark 3.3.4 and Java 17) and connects to a Neo4j instance, providing users with a starting point for testing the connector in Google Colab.
examples · high confidence
Initial TeamCity build pipeline configuration
The project now includes a TeamCity build pipeline defined in Kotlin DSL under the .teamcity directory. This configuration establishes three main build streams: a main branch build running on Java 17 and 21 with Scala 2.13 and PySpark 4.0/4.1; a pull-request build triggered on pull/\* branches, also on Java 17/21 and Scala 2.13, with a specific failure condition to prevent unintended main-branch runs when PRs are closed; and a scheduled compatibility suite that runs daily at 6 AM across various Neo4j versions. The setup also includes an .editorconfig file to enforce Kotlin formatting rules consistent with ktfmt for developers working on the pipeline scripts.
.teamcity · high confidence
Introduce TeamCity Kotlin DSL build pipeline
The project now uses a TeamCity-based CI pipeline defined in Kotlin DSL under .teamcity/builds. This pipeline configures builds for the neo4j-spark-connector (defaulting to the 6.0 branch) with support for Java 17/21, Scala 2.13, and PySpark 4.0/4.1 across Python 3.10–3.14, and includes Semgrep static analysis, Maven packaging and unit tests, Java and PySpark integration tests, and a release step via JReleaser. Pull requests trigger whitelist and PR checks, docs-only changes are short-circuited, and Slack notifications are sent on build outcomes.
.teamcity/builds · high confidence
New utility infrastructure and strict query enforcement
This change introduces a suite of new utility components in the connector's core. It adds a thread-safe DriverCache to manage driver lifecycle and prevent race conditions, and a StrictDriver that enforces a new 'strict query mode' by checking for Cypher warning notifications on every result consumption. Additionally, it provides new validation logic (Validations, ValidationUtil) to enforce schema and connection constraints, and utility classes like Neo4jImplicits for property path handling and Neo4jOptions for configuration parsing.
src/main/scala/org/neo4j/spark/util · high confidence
Register Neo4j Spark connector as a data source and expose version
The connector is now automatically discoverable by Spark SQL as a data source by registering the \org.neo4j.spark.DataSource\ class in the \META-INF/services/org.apache.spark.sql.sources.DataSourceRegister\ file. Additionally, a new \neo4j-spark-connector.properties\ file exposes the connector's version via the Maven placeholder \${project.version}\, allowing users to verify the installed version at runtime.
src/main/resources · high confidence
Repository initialization and build infrastructure setup
The repository has been initialized with the Neo4j Connector for Apache Spark, including a new README, license files, and a .gitignore. Build and release processes are now supported via a Maven Wrapper (version 3.3.4), a custom maven-release script, and JReleaser configuration for Maven Central deployment. Development workflows are enforced through Conventional Commits linting and Danger.js PR checks.
(repo-wide) · high confidence
Streaming write support with detailed metrics and offset tracking
The Scala module now supports streaming writes to Neo4j. This change introduces \Neo4jOffset\ to track streaming offsets, \BaseDataWriter\ to handle batched record writing with retry logic and failure handling, and \DataWriterMetrics\ to expose granular metrics (records written, nodes/relationships created/deleted, properties set, labels added/removed) via Spark's custom metric interface.
scala · high confidence
Behavioural changes
Enforce commit message format and code style on TeamCity and project files
The pre-commit hook now automatically formats code using Maven wrappers (mvnw) for both the main project and the .teamcity directory, ensuring consistent style via sortpom and spotless. Additionally, a new commit-msg hook enforces commit message standards by running commitlint via pnpm before allowing commits.
.husky · high confidence
Enhanced Cypher query handling with version control, tuning, and subquery embedding
The Cypher execution layer now supports overriding the Cypher version via configuration, automatically selecting the appropriate dialect based on the Neo4j server version (including support for the 2025 dialect), and applying user-defined Cypher tuning parameters. Additionally, read queries are now embedded as CALL subqueries to preserve original return order and ensure valid aliasing, while write queries are similarly wrapped to handle event batching.
src/main/scala/org/neo4j/spark/cypher · high confidence
Migrate from RDD-based API to Spark DataSource V2
The connector now uses the Spark DataSource V2 API (DataSource.scala, Neo4jTable.scala) for reading and writing data, replacing the legacy RDD-based classes (CypherRDD, CypherRowRDD, CypherResultRdd, CypherTupleRDD) which have been removed. This change modernizes the integration with Spark SQL, enabling standard DataFrame/Dataset operations and improved write capabilities via SupportsRead/SupportsWrite, while removing the older low-level RDD interfaces.
src/main/scala/org/neo4j/spark · high confidence
Migration from legacy Cypher-Bolt-RDDs to Spark Connector API
The legacy Cypher-Bolt-RDD integration has been removed in favor of the modern Spark Connector API. The \Neo4jSparkContext\ class, which previously provided RDD-based query methods (such as \query\, \queryTuple\, \queryRow\, and \queryDF\), has been deleted. To support the new connector's requirements, a new \ReflectionUtils\ utility has been added to handle compatibility with Spark's \Aggregation\ interface, specifically managing the transition between \groupByColumns\ and \groupByExpressions\.
src/main/java · high confidence
New Spark Connector V2 writer implementation
The writer component in \src/main/scala/org/neo4j/spark/writer\ has been replaced with a new implementation based on the Spark Connector V2 API. This change introduces \Neo4jWriterBuilder\ to handle batch and streaming write construction, \Neo4jBatchWriter\ to manage batch execution and index waiting, and dedicated \Neo4jDataWriter\ and \Neo4jDataWriterFactory\ classes to handle row-level data writing. This refactoring aligns the write path with modern Spark standards, supporting both batch and streaming modes through the new builder pattern.
src/main/scala/org/neo4j/spark/writer · high confidence
New data conversion architecture with native type support and legacy mode
The converter package has been restructured into dedicated DataConverter and TypeConverter components, introducing native support for Spark types such as byte, short, decimal, timestamp without timezone (TimestampNTZ), and day-time/year-month intervals (mapped to Neo4j durations). By default, timestamps are now converted as zoned datetimes and local datetimes as timestamp-without-timezone; users can opt back into the previous behavior by enabling the legacy type conversion option.
src/main/scala/org/neo4j/spark/converter · high confidence
New structured streaming read and write implementations
The streaming module now includes new core components for structured streaming: \Neo4jMicroBatchReader\ manages micro-batch offsets and partition planning, \BaseStreamingPartitionReader\ and \Neo4jStreamingPartitionReader\ handle data retrieval with explicit range filtering via \$stream.from\ and \$stream.to\ parameters (replacing the deprecated \$stream.offset\), and \Neo4jStreamingWriter\ along with \Neo4jStreamingDataWriterFactory\ enable data writing. This change introduces the underlying infrastructure for streaming operations, including the enforcement of new streaming parameter syntax and the removal of legacy offset usage in queries.
src/main/scala/org/neo4j/spark/streaming · high confidence
Refactored Spark DataSource V2 reader implementation
The partition reader logic has been restructured to use a new DataSource V2 architecture. A new \BasePartitionReader\ abstract class now centralizes connection management, retry logic, and query execution, while \Neo4jPartitionReader\ implements the specific partition reading interface. A \Neo4jPartitionReaderFactory\ is introduced to instantiate these readers, and \Neo4jScanBuilder\ now implements V2 pushdown interfaces (filters, aggregates, columns, limits, and TopN) to allow the Spark optimizer to push down operations to Neo4j.
src/main/scala/org/neo4j/spark/reader · high confidence
Rewrite of write mapping and query generation using Cypher DSL
The write path in the Spark connector has been refactored to use the Neo4j Cypher DSL for generating queries, replacing the previous string-based construction. This change introduces a new MappingService that handles the conversion of Spark DataFrames into Neo4j node and relationship structures, supporting both KEYS and PROPERTIES strategies for key and property mapping. The Neo4jQueryService now builds write statements for nodes, relationships, and custom queries using the DSL, ensuring consistent query generation and better handling of relationship writes with source and target node metadata. This refactor aims to improve reliability and maintainability of write operations.
src/main/scala/org/neo4j/spark/service · high confidence
Updated release archive documentation to clarify thin JAR usage
The README template for the release ZIP archive has been updated to explicitly document the two JARs included in the distribution. It clarifies that the standard JAR bundles all dependencies for easy use with spark-submit, while the new 'thin' JAR (identified by the \thin\ classifier) contains only the connector code, allowing users who manage their own classpath to avoid dependency clashes with libraries like Netty.
src/jreleaser · high confidence
Test coverage
25 commits adding/updating tests in src/test/scala/org/neo4j/spark; Added PySpark integration tests and declarative pipeline example; Added test configuration and resources for parallel execution and Neo4j SSO integration; Added tests for Cypher query generation, embedding, and schema handling; Added tests for Neo4j Spark connector utility components; Added tests for Spark data source type reading and refactored test suite structure; New test infrastructure and support utilities for integration tests; New test support utilities for assertions and parallel execution.
Dependencies
Project restructured for Neo4j Connector for Spark 6.1.1
The project has been reorganized to align with the Neo4j team connectors setup, updating the Maven coordinates to \org.neo4j.connectors:spark\ and setting the version to 6.1.1-SNAPSHOT. This change upgrades the build to use Java 17, Scala 2.13.18, and Apache Spark 4.1.2, while also updating the Neo4j Java driver to version 6.3.0 and the Cypher DSL to 2025.3.4. Additionally, documentation examples for Java, Python, and Scala have been added to guide users on integrating the connector, and the build system now utilizes pnpm for managing Node.js dependencies in the documentation and CI tooling.
(dependencies) · high confidence
Updated Maven wrapper to version 3.3.4 with Maven 3.10.0 distribution
The Maven wrapper configuration has been updated to use version 3.3.4, which now downloads and uses Apache Maven 3.10.0 instead of the previous 3.9.x versions. This ensures that builds run with a consistent, newer version of Maven across all environments.
.mvn · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 60 → 61 (+0.9)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 56 → 56 (-0.0)
- Architecture 96 → 96 (-0.7)
- Maturity 58 → 62 (+3.5)
- Readiness 73 → 74 (+0.9)
- Security 61 → 61 (+0.1)
Resolved (179)
- Critical CVE: [GHSA redacted] (pom.xml)
- Documentation: no installation or build instructions (docs/modules/ROOT/pages/installation.adoc)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pom.xml)
- High CVE: [GHSA redacted] (pnpm-lock.yaml)
- Medium CVE: [GHSA redacted] (pom.xml)
- Medium CVE: [GHSA redacted] (pnpm-lock.yaml)
- Medium CVE: [GHSA redacted] (pom.xml)
- Medium CVE: [GHSA redacted] (pom.xml)
- Medium CVE: [GHSA redacted] (pom.xml)
- Off-boarding risk: anonymized user #1
- Outdated (compile -- module annotations (auto)): org.jetbrains:annotations
- Outdated (compile -- module io.netty.buffer): io.netty:netty-buffer
- Outdated (compile -- module io.netty.codec): io.netty:netty-codec-base
- …and 159 more
New (7)
- Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
- Off-boarding risk: anonymized user #1
- Orphaned file (src/main/scala/org/neo4j/spark/reader/Neo4jScan.scala)
- Orphaned file (src/main/scala/org/neo4j/spark/writer/DataWriterMetrics.scala)
- Orphaned files with no living knowledge
- Repeated repair: src/main/scala/org/neo4j/spark/service/SchemaService.scala (src/main/scala/org/neo4j/spark/service/SchemaService.scala)
- TodoComment (src/test/scala/org/neo4j/spark/WriteIT.scala)
Changes since last survey
- 18 commits — 17 feature/other, 1 fixes
By area
- (root) — 11 commits
- docs/modules — 4 commits
- .teamcity/builds — 1 commit
- src/main — 1 commit
- src/test — 1 commit
Notable commits
- fix: fix: array schema inference stability (#1238)
- change: build(deps): bump com.diffplug.spotless:spotless-maven-plugin from 3.10.4 to 3.11.0 (#1248)
- change: build(deps): bump fast-uri from 3.1.7 to 3.1.9 (#1251)
- change: build(deps): bump org.apache.maven.plugins:maven-enforcer-plugin from 3.6.3 to 3.7.0 (#1242)
- change: build(deps): bump org.apache.maven.plugins:maven-failsafe-plugin from 3.6.0 to 3.6.1 (#1233)
- change: build(deps): bump org.apache.maven.plugins:maven-surefire-plugin from 3.6.0 to 3.6.1 (#1234)
- change: build(deps): bump qs from 6.16.0 to 6.17.1 (#1250)
- change: build(repo)(deps-dev): bump danger from 14.0.7 to 14.1.0 (#1239)
- change: build(repo)(deps-dev): bump the commitlint group with 4 updates (#1228)
- change: build: release version 6.1.2
- change: build: update version to 6.1.3-SNAPSHOT
- change: ci: pin Testcontainers host for IPv6-enabled TeamCity agents (#1245)
- change: docs: clarify relationship save mode semantics (#1224)
- change: docs: consistency check for save strategy and save.mode defaults (#1225)
- change: docs: minor fixes scattered around the docs (#1226)
- change: docs: thin jar available (and new release) (#1227)
- change: feat: support mTLS options (#1236)
- change: refactor: move remaining write tests into WriteIT (#1230)
Architecture
- Unchanged — 0 containers · 1 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
neo4j/neo4j-spark-connector was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 6 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 03e66118b9579d58b412fdbc880368ac042f3893 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-60d1b390edf0.