AbsaOSS/spline

This system is a data lineage tracking platform that ingests execution plans and events from Spark applications via REST or Kafka, then persists the resulting graph data in ArangoDB. It provides a consumer API to query and visualize historical lineage, forward impact, and attribute-level dependencies, alongside an administrative CLI for managing the database schema and data retention. The architecture has migrated from MongoDB to ArangoDB, utilizing Foxx services for transactional integrity and supporting multiple producer API versions for backward compatibility.

30 changes.

The project migrated its persistence layer from MongoDB to ArangoDB, introducing a new consumer REST API for querying lineage and execution metadata. This period involved stripping out legacy core harvesting logic and Angular UI components while establishing a modern producer API and administrative CLI for database management.

16 changes.

This period focused on migrating the persistence layer to ArangoDB, introducing a schema versioning framework, and updating the data model to support Spline Producer API v1.1. It also involved enhancing the Kafka Gateway with versioned ingestion handlers and adding operational dashboards for both the REST and Kafka gateways.

14 changes.

This period focused on implementing a comprehensive ArangoDB-based persistence layer, including Foxx services for transactional lineage storage, data retention, and index management. The work involved defining the v1.2 producer model to support execution labels and refining expression handling, while simultaneously introducing a test data generator and extensive test coverage for the new database interactions.

The producer-model-mapper module now includes a complete v1.0 model mapper (ModelMapperV10) that converts Spline v1 execution plans and events into the internal v1.2 model. This includes Spark-specific converters for attributes, expressions, and operation outputs, as well as a factory for creating component converters based on the agent version. The mapper handles attribute dependencies, schema resolution, and expression conversion for Spark operations like Project, Aggregate, and SubqueryAlias.

producer-model-mapper · high confidence

The persistence layer now includes a migration framework for managing ArangoDB schema changes. This introduces a system that loads JavaScript migration scripts from the classpath, resolves the required migration chain between semantic versions using a graph-based path finder, and executes them sequentially. The Migrator component handles database version tracking, supports dry-run mode for safe testing, and ensures atomicity by updating the database version status only after successful script execution.

persistence/src/main/scala/za/co/absa/spline/persistence/migration · high confidence

The test-data-generator now includes a Dockerfile and a dedicated shell script (send-plans-and-events-docker.sh) to run the generator as a container. This setup allows users to configure the generated lineage graph parameters—such as graph type, number of reads, operations, and attributes—via environment variables (SPLINE\_URL, GRAPH\_TYPE, READS, OPERATIONS, ATTRIBUTES) and automatically sends the resulting execution plans and events to a Spline producer endpoint.

test-data-generator · high confidence

Adds a new ArangoDB Foxx service that registers a PRE\_COMMIT transaction hook to update the \lastWriteDetails\ of target data sources. The service listens for execution events and, if the incoming event is newer than the current data source timestamp, atomically updates the data source document to reflect the latest write progress, ensuring consistent index state across transactions.

arangodb-foxx-services · high confidence

The Spline Admin CLI is now available as a standalone command-line tool for managing the ArangoDB backend. It supports initializing the database schema (db-init), upgrading it (db-upgrade), and executing maintenance actions like data pruning. The tool provides interactive prompts for credentials and backup confirmation, while also supporting non-interactive and dry-run modes for automation. Users can configure SSL validation, logging levels, and database shard/replication settings directly via command-line flags.

admin/src/main/scala/za/co/absa/spline/admin · high confidence

A new test-data-generator module has been added to the Spline project, providing a command-line tool for generating synthetic lineage data. The generator supports three graph topologies—chain, diamond, and triangle—and allows users to configure the number of reads, operations, and attributes via CLI arguments, including variable ranges (e.g., start-end/step) for automated expansion of test scenarios. Generated data is written to JSON files and includes execution plans and events structured according to the Spline producer model v1.2, facilitating testing of lineage processing pipelines.

test-data-generator/src · high confidence

The persistence layer now defines the data model for ArangoDB, including node and edge collections (such as ExecutionPlan, Operation, and DataSource) along with their specific index configurations. This change introduces support for ArangoDB-specific features like ArangoSearch views and custom analyzers, replacing or augmenting previous storage abstractions to enable more efficient querying and search capabilities within the graph database.

persistence/src/main/scala/za/co/absa/spline/persistence/model · high confidence

The producer model now includes a new v1.1 schema definition, introducing case classes for \ExecutionPlan\, \Attribute\, \Expression\, and \Operation\ to structure lineage data. This update adds a dedicated \AttrOrExprRef\ type to distinguish between attribute and expression references, enforces unique ID constraints on attributes, operations, and expressions, and supports null values in literal expressions via a custom Jackson deserializer.

_producer-model/src/main/scala/za/co/absa/spline/producer/model/v1\1 · high confidence

The Spline Producer REST API is now available with support for version 1.2, which is designated as the current and Long-Term Support (LTS) version. This update introduces new HTTP headers for API versioning and request encoding (e.g., ABSA-Spline-API-Version), and configures the REST server to handle JSON deserialization using Jackson with specific naming strategies and Scala module support. The API also includes configuration for handling asynchronous responses with configurable timeouts.

producer-rest-core/src/main/scala/za/co/absa/spline/producer/rest · high confidence

The Kafka Gateway now includes a default webapp structure to aid deployment and operational visibility. A new META-INF/context.xml file defines environment variables for ArangoDB connection details, Kafka consumer settings (including group ID, bootstrap servers, concurrency, timeouts, backoff parameters, and dead-letter queue configuration), and database options like active failover and SSL validation. Additionally, an index.html page serves as a landing dashboard that displays the gateway's version, build timestamp, and revision, while actively polling the /about/liveness endpoint to show a real-time status badge (Running, Disconnected, Not ready, or Error) and providing links to the diagnostics API endpoints for version, readiness, and liveness checks.

kafka-gateway/src/main/webapp · high confidence

The Kafka Gateway now supports processing incoming execution plans and events using specific protocol versions (v1.1 and v1.2). New \HandlerV11\ and \HandlerV12\ traits define how \ExecutionPlan\ and \ExecutionEvent\ DTOs are mapped and persisted via the \IngestionListener\. This allows the gateway to handle data from producers using either v1.1 or v1.2 serialization formats, ensuring compatibility with different producer versions.

kafka-gateway/src/main/scala/za/co/absa/spline/gateway/kafka/listener · high confidence

The admin module now includes a new ArangoDB management layer (ArangoManager) that handles database initialization, schema upgrades, and data retention pruning. A key addition is the \--dry-run\ capability, allowing administrators to preview database changes without applying them. The module also introduces auxiliary actions for managing Foxx services, indices, and search views, along with an auto-closing proxy to manage database connections safely.

admin/src/main/scala/za/co/absa/spline/arango · high confidence

The persistence module now introduces a complete ArangoDB backend implementation. This includes an ArangoDatabaseFacade that supports SSL connections (with an option to disable validation), active failover mode, and enforces a supported ArangoDB server version range (3.9.0 to \<3.12.0). A new ArangoConnectionURL parser handles connection strings, while ArangoRepoConfig wires these components via Spring. The system also adds DatabaseVersionManager and DatabaseVersionChecker to track and enforce database schema versions, and introduces retry logic for transient errors (like deadlocks or timeouts) via RetryableExceptionUtils.

persistence/src/main/scala/za/co/absa/spline/persistence · high confidence

The arangodb-foxx-api module now includes a comprehensive set of Scala case classes and traits (e.g., ExecutionPlan, Progress, DataSource, Operation, Schema, Attribute) that define the data model for storing and querying execution plan metadata in ArangoDB. This change introduces the underlying graph structure (vertices and edges) required to persist lineage information, supporting the backend's ability to store and retrieve detailed execution statistics and schema details.

arangodb-foxx-api · high confidence

The consumer service now includes a new repository layer in \consumer-services/src/main/scala/za/co/absa/spline/consumer/service/repo\ that implements data access for execution events, data sources, execution plans, operations, labels, and lineage. These repositories use ArangoDB AQL queries to support filtering by labels, timestamps, application IDs, and data source URIs, as well as pagination, sorting, and full-text search. This provides the backend data retrieval capabilities for the consumer API endpoints.

consumer-services/src/main/scala/za/co/absa/spline/consumer/service/repo · high confidence

The Consumer REST API now exposes a comprehensive set of endpoints for exploring data lineage and execution metadata. Users can retrieve execution events and plans via /execution-events and /execution-plans, with support for pagination, sorting, and filtering by labels, search terms, and write modes. A new /data-sources endpoint lists data sources with similar filtering capabilities. Lineage is accessible through /lineage-overview (historical flow), /impact-overview (forward impact), and /lineage-detailed (full DAG). Attribute-level lineage and impact are available via /attribute-lineage-and-impact. Execution plan details include /execution-plans/{planId}/write-op to redirect to operation details, /execution-plans/{planId}/data-sources to list associated data sources, and /operations/{operationId} to retrieve operation details with expression graphs. Labels can be queried via /labels/names and /labels/names/{labelName}/values. Error handling has been standardized to return appropriate HTTP status codes (400, 404, 503, 500) for various exception types.

consumer-rest-core/src/main/scala/za/co/absa/spline/consumer/rest/controller · high confidence

A new Dockerfile and entrypoint script have been added to the admin module, enabling users to run the Spline Admin Tool via \docker run --rm -it absaoss/spline-admin:latest\. The image is based on AdoptOpenJDK 11 JRE, runs as an unprivileged user (UID 1001), and uses Tini for proper signal forwarding. The entrypoint script launches the \admin.jar\ executable, passing through command-line arguments to the CLI.

admin · high confidence

The consumer service now exposes a comprehensive set of model classes to support detailed lineage and execution tracking. This includes models for execution plans and events (ExecutionPlanInfo, ExecutionEventInfo), detailed lineage graphs with operations and transitions (LineageDetailed, LineageDetailedGraph, Operation, Transition), and attribute-level dependency graphs (AttributeGraph, AttributeNode, AttributeEdge). Additionally, expression-level lineage is supported via ExpressionGraph and ExpressionNode, while data source interactions are modeled through DataSourceInfo and DataSourceActionType. Pagination and sorting capabilities are provided by Pageable, PageRequest, and SortRequest, and data types are handled via a polymorphic DataType hierarchy (Simple, Array, Struct). These models enable users to query granular execution details, trace attribute dependencies, and visualize complex data lineage structures through the consumer API.

consumer-services/src/main/scala/za/co/absa/spline/consumer/service/model · high confidence

This change introduces a new \commons\ module providing shared Scala utilities for the application. It includes a configuration stack supporting environment variables and JNDI, a REST client with SSL support, and Spring MVC components such as a permissive CORS filter, Swagger configuration, and handlers for Scala Futures. It also adds diagnostic endpoints for liveness and readiness probes, along with various helper classes for JSON path manipulation, graph traversal, and validation.

commons/src/main · high confidence

This change introduces a new transaction management layer in the persistence module, adding \AbstractTxBuilder\, \JSTxBuilder\, and \FoxxPostTxBuilder\ to construct and execute ArangoDB transactions. It defines query models (\NativeQuery\, \InsertQuery\, \UpdateQuery\) and an \ArangoTx\ trait to handle transaction execution, replacing the previous \SparkJobRunner\ utility which has been removed from this location. This provides the foundation for atomic database operations within the Spline persistence layer.

persistence/src/main/scala/za/co/absa/spline/persistence/tx · high confidence

A new index page has been added to the Spline REST Gateway that provides a dashboard for monitoring the service. It displays the gateway version, build timestamp, and revision, along with the current status of the Producer API (including API version and request encoding) and Consumer API base URLs. The page also lists endpoints for the Diagnostics API, specifically the liveness and readiness probes, and updates this information automatically every five seconds.

rest-gateway/src/main/webapp · high confidence

The ArangoDB Foxx service now supports configurable data retention via a new admin API endpoint (DELETE /admin/data/before/:timestamp) that prunes execution events, plans, and data sources older than a specified timestamp. Additionally, the service implements a custom application-level distributed transaction protocol to ensure atomicity and read-committed isolation for lineage data, and exposes new API endpoints to record execution events and plans while providing backward and forward (impact) lineage overviews.

arangodb-foxx-services/src · high confidence

The \ErrorCode\ class and its companion object, previously located in \web/src/main/scala/za/co/absa/spline/web/logging\, have been removed from the codebase. This component was responsible for wrapping unexpected exceptions with unique UUID identifiers to facilitate log tracing and client support; its removal indicates that this specific error-handling pattern is no longer used or has been replaced by a different mechanism.

web/src/main · high confidence

The entire Lineage UI module has been removed from the web application. This includes the main lineage view, the DAG graph visualization, operation node details, attribute and data-type views, expression inline views, and the associated lineage service and routing module. Users will no longer have access to the web-based lineage tracking interface provided by this module.

web/ui/src/app/lineage · high confidence

The \core/src/main\ module has removed the original data lineage harvesting and persistence implementation, including the \DataLineageHarvester\, \DataLineageListener\, and the entire \storage\ package (containing \DataLineageStorage\, \ExecutionStorage\, and their MongoDB-specific implementations). This change eliminates the previous mechanism for intercepting Spark query executions, building lineage graphs from logical plans, and persisting them to MongoDB, effectively stripping out the core lineage collection and storage logic from this location.

core/src/main · high confidence

The mock REST API located in the \web/ui/mock-rest-api\ directory has been removed. This change deletes the \db.js\ file, which previously provided hardcoded lineage data for UI development and testing, and the \routes.json\ configuration file that mapped API endpoints. Users relying on this local mock service for offline UI development or testing will no longer have access to these static data fixtures.

web/ui/mock-rest-api · high confidence

The sample directory no longer includes the \SampleJob1\ and \SampleJob2\ Scala examples, nor the \domain.csv\ and \wikidata.csv\ input files they relied on. Users can no longer run these specific lineage-tracking demonstrations using the provided sample data.

sample · high confidence

The admin module now includes a dedicated logback configuration for console logging with timestamped, structured output, and a properties file defining the executable path for the Spline CLI tool, replacing previous test and UI resource files.

admin/src/main/resources · high confidence

The ArangoDB persistence layer has been updated with a series of database migration scripts covering versions 0.4.0 through 1.0.0. These migrations transition the data model by introducing new document and edge collections (such as 'schema', 'attribute', 'counter', and 'txInfo'), extracting nested attributes into dedicated collections, and adding support for execution labels and plan keys. Performance and search capabilities are improved by replacing legacy AQL User-Defined Functions with ArangoSearch views and adding persistent indexes on fields like 'dataSource.name' and 'progress.execPlanDetails.dataSourceName'.

persistence/src/main/resources · high confidence

The persistence module introduces three new internal Scala files to support database operations: InternalArangoDatabaseOps provides a REST client wrapper for executing admin scripts against ArangoDB, handling SSL context detection and credential extraction from the underlying connection; VstImplicits adds implicit extension methods to access user credentials and connect to the Velocystream communication layer; and Implicits provides an extension method to determine the index type from IndexOptions classes. These changes appear to be internal infrastructure updates to support the ArangoDB driver integration.

persistence/src/main/scala/com · medium confidence

The producer model in the v1.2 package has been updated to support execution labels, allowing users to group and filter data on the Consumer API and UI. This change introduces new model classes including \AttrOrExprRef\ for unified attribute and expression references, \ExecutionEvent\ with a labels field, and \ExecutionPlan\ which now includes labels and a \discriminator\. Additionally, the model refines expression handling by introducing \ExpressionLike\, \FunctionalExpression\, and \Literal\ types, and updates \Attribute\ to use \AttrOrExprRef\ for child references.

_producer-model/src/main/scala/za/co/absa/spline/producer/model/v1\2 · high confidence

The Kafka Gateway's application initialization and configuration have been consolidated into the kafka-gateway module. The previous web-specific initializer is replaced by a new AppInitializer that registers the KafkaGatewayConfig, ProducerServicesConfig, and ArangoRepoConfig as root context beans, with bean definition overriding explicitly disabled. The KafkaGatewayConfig now centrally defines the Kafka listener container factory, consumer/producer factories, Jackson-based message conversion with explicit type mappings for ExecutionPlan and ExecutionEvent models (v1.1 and v1.2), and an error handler that supports a configurable dead-letter queue and excludes ConversionException and InconsistentEntityException from retries. Configuration is driven by the spline prefix with dedicated sections for consumer, producer, and back-off settings, including timeouts for plan and event insertion.

kafka-gateway/src/main/scala/za/co/absa/spline/gateway/kafka · high confidence

The REST consumer model now includes a new LabelQuery object that parses label strings in the format 'name:value1,value2' into structured Label objects, enabling multi-value label filtering. Additionally, the previous Angular attribute-list component has been replaced by a Scala WriteModeQuery object that parses comma-separated boolean strings into an array of optional booleans, shifting write-mode query handling from the UI to the backend REST model layer.

consumer-rest-core/src/main/scala/za/co/absa/spline/consumer/rest/model · high confidence

The producer-services module has been restructured to improve execution plan handling and persistence. A new ExecutionPlanPersistentModelBuilder constructs the internal persistence model, including logic to infer schemas for operations and detect ambiguous or cyclic graph structures, throwing InconsistentEntityException on validation failure. An ExecutionPlanDeserFixAspect intercepts repository calls to fix deserialized ExecutionPlan models, ensuring attribute and expression references are correctly typed. The ExecutionProducerRepositoryImpl now uses an async ArangoDB driver with retry logic, checks for discriminator collisions to prevent UUID conflicts, and persists data sources and execution plans via Foxx services. Additionally, legacy classes like Execution and DataStorageFactory have been replaced or renamed to UUIDCollisionDetectedException and ExecutionEventKeyCreator, and configuration is centralized in ProducerServicesConfig.

producer-services · high confidence

The repository has been restructured to improve developer experience and community standards. A new .editorconfig file enforces consistent coding styles for Scala and TypeScript/JavaScript files. The project now requires Java 21, as specified in the new .sdkmanrc file. Documentation has been significantly condensed and moved to GitHub Pages, with the README.md updated to include build badges (Maven, Docker, TeamCity, Codacy, SonarCloud) and a link to a demo video. Additionally, a CODE\_OF\_CONDUCT.md file has been added to establish community guidelines, and the .gitignore has been updated to exclude node\_modules and macOS artifacts.

(repo-wide) · high confidence

The REST producer now supports gzip-compressed requests via a new GzipFilter, automatically decompressing payloads before processing. Additionally, incoming message sizes are captured and appended to the execution plan's extraInfo under the key \_\_spline\_msg\_size, enabling users to monitor payload sizes. Error handling has been refined to provide more specific HTTP responses: validation, type mismatch, and inconsistent entity errors now return 400 Bad Request with the error message, while 404 responses include the message, and 500 errors return a structured ErrorRef instead of a generic JSON object.

producer-rest-core/src/main/scala/za/co/absa/spline/producer/rest/controller · high confidence

The REST Gateway now initializes distinct API contexts for the Producer and Consumer services, allowing them to be managed and exposed separately. This change introduces a new AppInitializer that registers specific servlet filters (including a permissive CORS filter, Gzip compression, and message length capturing) and sets up dedicated REST dispatchers for consumer, producer, and diagnostics endpoints, replacing the previous unified initialization approach.

rest-gateway/src/main/scala · high confidence

The producer model has been refactored to introduce new case classes for execution events and plans, where the ExecutionPlan ID is now optional (defaulting to a random UUID) and both ExecutionEvent and Operation types include an 'extra' field to prevent property name collisions. Additionally, DataOperation now defaults childIds to an empty sequence, and the schema semantics for operations have been clarified so that a None value indicates an unchanged or unknown schema rather than a missing one.

producer-model/src/main/scala/za/co/absa/spline/producer/model · high confidence

The Consumer REST API now applies Jackson type hints exclusively to \LineageOverviewNode\ and \DataType\ classes, replacing the previous broader typing strategy. This change, implemented via the new \ConsumerTypeResolver\ and \ConsumerRESTConfig\, ensures that only these specific model types include the \\_type\ property in serialized JSON, reducing payload size and improving compatibility for other response types.

consumer-rest-core/src/main/scala/za/co/absa/spline/consumer/rest · high confidence

The web UI build system has removed the legacy Webpack 3 configuration, TypeScript configuration, and associated tooling files. Specifically, the old webpack.config.js, tsconfig.json, typings.json, and the browser-check script have been deleted, along with the json-server mock configuration and the .gitignore file. This cleanup aligns with the upgrade to Angular 6 and Webpack 4, removing obsolete build artifacts and dependencies that are no longer part of the new build pipeline.

web/ui · high confidence

The \app.module.ts\ file, which previously defined the root Angular module, bootstrapped the \AppComponent\, and configured the primary routing structure (redirecting to the lineage view), has been removed. This change eliminates the legacy module-based application entry point and its associated route definitions, indicating a shift in the application's architecture away from this specific module configuration.

web/ui/src/app · high confidence

The static HTML entry point for the web application, which previously relied on Thymeleaf templating to inject base hrefs, favicon paths, and bundled JavaScript assets, has been removed. This change indicates a shift away from server-side rendered HTML templates for the main application shell, likely as part of the broader Angular 6 and Webpack 4 upgrade, where the build process now handles the generation of the entry HTML file directly.

web/ui/src · high confidence

Added comprehensive test coverage for the Spline Admin CLI; Added tests for ArangoDB index type resolution; Added tests for migration script repository and loader; Added tests for topological graph sorting; Added tests for unique constraint validation; Added unit tests for DataSource name extraction logic; Added unit tests for Execution Plans and Operation Details controllers; Added unit tests for JSTxBuilder JavaScript generation; Added unit tests for Spark Spline expression and object converters; Added unit tests for common utility classes; Added unit tests for configuration implicits and environment variable handling; Added unit tests for persistence components; Added unit tests for transaction management, AQL generation, and utility functions; Added unit tests for web MVC components; Integration tests: new ArangoDB 3.11 fixtures and Admin CLI dry-run coverage; Removed obsolete test suites for DataLineageHarvester and Mongo storage; Removed test for Salat-based JSON serialization.

This change introduces the foundational Maven module structure and dependency definitions for the Spline project, establishing the build environment for version 1.1.0-SNAPSHOT. It adds POM files for key components including the Admin CLI, ArangoDB Foxx services, consumer and producer services, persistence layer, and various gateway modules, while also configuring the parent POM with Scala 2.13.18, Logback 1.5.35, and Jackson 2.17.1. The update includes the initial setup of the ArangoDB Foxx services' Node.js dependencies (such as Webpack 5.105.0 and TypeScript 4.8.4) and configures the integration tests module with Testcontainers 1.21.4, effectively setting the stage for the application's build and runtime architecture.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 56 → 55 (-0.1)
• Rubric changed (rubric-2026.09.8 → rubric-2026.09.16) — scores are not directly comparable.

• Code Health 89 → 90 (+1.6)
• Architecture 90 → 85 (-4.4)
• Maturity 49 → 49 (+0.0)
• Readiness 46 → 46 (+0.2)
• Security 92 → 94 (+2.1)
• Accessibility 63 → 63 (+0.0)

• Low CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)

• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• Medium CVE: [GHSA redacted] (arangodb-foxx-services/package-lock.json)
• No ADRs found
• Off the main sequence: arangodb-foxx-api
• Off the main sequence: commons
• Off the main sequence: producer-model

• 1 commits — 1 feature/other, 0 fixes

• arangodb-foxx-services/package-lock.json — 1 commit

• change: build(deps-dev): bump @babel/core in /arangodb-foxx-services

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

AbsaOSS/spline was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.