Skip to content
CAI
Software that uses CAICheck a score

starlake-ai/starflow

38.0

Weak · 7 November 2026

70.4k

lines of production code

Scala

primary language

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add BigQuery and DuckDB schema conversion utilities

New utility objects have been added to the conversion package to support schema handling for BigQuery and DuckDB. BigQueryUtils provides functions to convert Spark schemas to BigQuery schemas, normalizes integer types to long types for compatibility, and handles field mapping for BigQuery data types. DuckDbUtils enables conversion of Spark schemas to DuckDB SQL column definitions, supporting nested structures, decimal precision, and JSON type detection. Conversions.scala adds an implicit conversion to treat Hadoop RemoteIterators as Scala Iterators.

src/main/scala/ai/starlake/utils/conversion · high confidence

Add BigQuery sample with deployment scripts and documentation

The BigQuery sample area now includes a complete set of shell scripts (init, import, load, and transform) and environment configuration files to guide users through the data loading workflow. It also provides documentation and scripts for deploying the solution on Google Cloud Dataproc and Cloud Run, along with sample data files for HR and sales domains.

samples/bigquery · high confidence

Add CLI command to generate standalone documentation site

Users can now run the \starlake site\ command to generate a standalone, responsive documentation website from project metadata (schemas, tasks, lineage). The command supports outputting either HTML or JSON, allows specifying a custom output directory, and uses Jinja2 templates (defaulting to a 'standalone' theme) to render the site locally without requiring external tools like Docusaurus.

src/main/scala/ai/starlake/job/site · high confidence

Add DuckDB data source integration via StarLake provider

Users can now read from and write to DuckDB databases using the 'starlake-duckdb' data source format in Spark SQL. This new \DuckDBRelationProvider\ extends the existing JDBC integration, allowing standard Spark DataFrame operations to interact with DuckDB tables, including support for save modes like Overwrite, Append, ErrorIfExists, and Ignore.

src/main/scala/org/apache/spark/sql/execution/datasources/duckdb · high confidence

Add DuckDB sample with metadata, data, and Airflow DAG templates

This change introduces a complete sample configuration for DuckDB within the \samples/duckdb\ directory. It includes metadata definitions for loading HR and sales data (JSON/CSV), transforming KPIs, and validating data quality via expectations. It also provides sample datasets, test fixtures, and Jinja2 templates for generating Apache Airflow DAGs to orchestrate these loads and transforms, enabling users to run and test the pipeline against a local DuckDB instance.

samples/duckdb · high confidence

Add Elasticsearch data loading capability

Users can now load datasets into Elasticsearch indices using the new \esload\ (and \index\) CLI commands. This feature supports input in Parquet, JSON, or JSON-array formats, allows custom Elasticsearch mappings, and enables timestamp-based index suffixes. The implementation handles index template registration and data ingestion via Spark.

src/main/scala/ai/starlake/job/sink/es · high confidence

Add Iceberg sample with Starbake dataset and multi-engine DAG templates

This change introduces a new Iceberg sample in the \samples/iceberg\ directory, providing a complete end-to-end example for the Starbake dataset. It includes sample data files (CSV, JSON, XML) and comprehensive metadata definitions for loading tables like \order\_lines\, \orders\, and \products\. The sample also adds configuration files for various execution engines, including Airflow and Dagster shell templates, as well as Snowflake SQL templates, enabling users to run the pipeline with different orchestrators. Additionally, it provides a suite of Jinja2 macros for data quality expectations (completeness, numeric, schema, uniqueness, validity, volume) and environment configurations for local Spark, DuckDB, BigQuery, and other connections.

samples/iceberg · high confidence

Add KPI transformation sample for Postgres

A new sample configuration and SQL script have been added to the Postgres metadata transform location. The \\_config.sl.yml\ file sets the default write behavior to OVERWRITE for the KPI transform, and the \byseller.sql\ script defines a transformation that aggregates sales amounts by seller email, joining the \hr.sellers\ and \sales.orders\ tables.

samples/postgres/metadata/transform · high confidence

Add OpenAPI schema support for extraction

Users can now extract schemas from OpenAPI specifications. This change introduces the OpenAPI schema extractor, which parses OpenAPI files and transforms them into Starflow-compatible domain and schema structures. The extractor supports configuration via domains, routes, and schema patterns, allowing for flexible mapping of API endpoints to tables. It handles various OpenAPI schema types and provides options for field exclusion, renaming, and explosion strategies.

src/main/scala/ai/starlake/extract/impl/openapi · high confidence

Add Snowflake sample with Starlake 1.0.0 integration

The Snowflake sample has been upgraded to version 1.0.0, introducing a new environment configuration file (.snowflake-env) and a dedicated env.sh script that enforces the presence of SNOWFLAKE\_SCHEMA alongside other required credentials. The sample now includes a complete set of initialization and execution scripts (0.init.sh through 3.transform.sh) to manage data import, watching, and transformation, alongside metadata definitions for a KPI transform that overwrites existing data.

samples/snowflake · high confidence

Add Starflow interactive console command

Users can now launch an interactive web-based console via the new \starlake console\ command to explore metadata, browse domains, tables, and task definitions, and run commands locally. This change introduces the \ConsoleCmd\ entry point and its associated \ConsoleConfig\ to support this new interface.

src/main/scala/ai/starlake/console · high confidence

Add Summarize command for table statistical summaries

Users can now run the 'summarize' command to generate a statistical summary for a specific domain and table, providing key data profiling insights such as column names, types, and descriptions. This new capability is implemented via the SummarizeCmd and SummarizeConfig classes, which accept domain and table arguments and output the results in a specified report format.

src/main/scala/ai/starlake/job/tools · high confidence

Add cloud-specific configuration profiles for Azure, GCP, HDFS, and local filesystems

New reference configuration files have been added for Azure, Google Cloud Platform (GCP), HDFS, and local filesystem deployments. These profiles pre-configure storage backends (e.g., Azure Blob Storage, GCS, HDFS), default dataset and metadata paths, and environment-specific settings such as JDBC connections for audit logs and Spark master URLs, allowing users to select a cloud provider profile to quickly initialize the system with appropriate defaults.

src/main/resources-other · high confidence

Add in-JVM Quack server with shutdown handling

Users can now run a Quack server directly within the JVM process. This change introduces a new QuackServer component that installs and loads the Quack extension, starts the server on a specified bind address and port with an optional authentication token, and registers a shutdown hook to cleanly stop the server and close the database connection when the process terminates. Configuration for the server (including connection name, bind address, port, token, and report format) is handled via the new QuackConfig model.

src/main/scala/ai/starlake/job/quack · high confidence

Add parquet2csv command to convert Parquet files to CSV

Users can now convert Parquet files to CSV format using the new \starlake parquet2csv\ command. This feature allows specifying input and output directories, filtering by domain or schema, controlling the number of output partitions, choosing a write mode (e.g., overwrite, append), and passing custom Spark options. An optional \--delete\_source\ flag is available to remove the original Parquet files after conversion.

src/main/scala/ai/starlake/job/convert · high confidence

Add sample data for HR and sales domains

New sample data files have been added to the \samples/sample-data\ directory to support testing and demonstration of HR and sales functionalities. The HR section includes JSON files for locations (\flat\_locations-2018-01-01.json\) and sellers (\sellers-2018-01-01.json\). The sales section provides CSV and PSV files for categories, customers (spanning two dates), orders, and product stock levels, providing realistic datasets for validating data ingestion and processing logic.

samples/sample-data · high confidence

Added BigQuery and Snowflake sample KPI transform definitions

New sample files have been added to the BigQuery and Snowflake Spark metadata transform directories to demonstrate a Key Performance Indicator (KPI) calculation. This includes a configuration file for BigQuery that sets the write mode to OVERWRITE, and a SQL script (byseller.sql) available in both BigQuery and Snowflake samples that calculates the total sales amount grouped by seller email by joining the sellers and orders tables.

samples/bigquery/metadata/transform, samples/snowflake-spark/metadata/transform · high confidence

Added DDL generation and data extraction templates for multiple database platforms

New Jinja2 templates have been added to support DDL generation (create, alter, drop) and data extraction scripts for BigQuery, PostgreSQL, Snowflake, Synapse, and MySQL. The BigQuery, PostgreSQL, and Synapse templates now include logic for adding, dropping, and modifying columns with specific options and data types. Additionally, new extraction scripts for MySQL and PostgreSQL enable full and delta table exports to CSV, including audit status tracking.

(repo-wide) · high confidence

Added Jinja2 macro library for Snowflake data expectations

A new Jinja2 template file (default.j2) has been added to the Snowflake metadata samples, providing a set of reusable SQL macros for defining data quality expectations. These macros, including is\_col\_value\_not\_unique, is\_row\_count\_to\_be\_between, col\_value\_count\_greater\_than, count\_by\_value, and column\_occurs, allow users to parameterize and standardize common validation logic such as uniqueness checks, row count ranges, and value frequency counts within Snowflake tables.

samples/snowflake/metadata/expectations · high confidence

Added KPI transform samples for order and product analysis

New SQL transform definitions and configuration have been added to the starbake metadata samples, enabling users to generate Key Performance Indicator views. The new files include a configuration setting the write strategy to OVERWRITE, and SQL scripts that define intermediate summaries for revenue, product performance (including profit and units sold), and order items, culminating in a final order summary view that joins these metrics together.

samples/starbake/metadata/transform · high confidence

Added Markdown CLI documentation template

A new Jinja2 template (md-cli.j2) has been added to generate Markdown documentation for CLI tools. This template structures output with a YAML front matter section for metadata (title, description, keywords, sidebar position), followed by sections for Synopsis, Description, and a table of Parameters including their names, values, required status, and descriptions.

src/main/resources/templates/cli · high confidence

Added MySQL write-strategy templates

New Jinja2 templates have been added to the MySQL write-strategies directory, providing SQL generation for append, create, delete-then-insert, overwrite, overwrite-by-partition, SCD2, upsert-by-key, upsert-by-key-and-timestamp, and view strategies.

src/main/resources/templates/write-strategies/mysql · high confidence

Added Postgres sample with Starlake integration scripts

A new sample directory for Postgres has been added, providing a complete set of shell scripts (init, import, load, transform) and environment configuration files to demonstrate how to use the Starlake CLI with a Postgres database. The sample includes an initialization script to copy sample data, an import script using the Starlake CLI, a watch script for loading, and a transform script for a specific KPI, along with environment setup that validates the Starlake installation and requires Postgres connection details.

samples/postgres · high confidence

Added Snowflake-Spark sample with bootstrap scripts and environment configuration

A new sample integration for connecting Spark with Snowflake has been added to the samples directory. This includes a set of executable shell scripts (0.init.sh through 3.transform.sh) that automate the workflow from copying sample data to importing, watching, and transforming it using the Starlake CLI. The sample also provides an environment configuration file (env.sh) that validates the Starlake installation and requires specific Snowflake connection details (account, user, password, warehouse, database, and schema) to be set, along with a default snowflake-env file containing placeholder credentials for immediate testing.

samples/snowflake-spark · high confidence

Added StarBake lineage sample with documentation, data, and configuration

The \samples/lineage\ area now includes a complete demonstrative project called StarBake, designed to showcase Starflow's data transformation and analytics capabilities using Google BigQuery. This addition provides a README explaining the e-commerce bakery domain (Customers, Orders, Products, Ingredients) and its analytical tables, along with sample datasets in CSV, JSON, and XML formats. It also includes lineage diagrams defining table schemas and column-level dependencies for analytics tasks like customer purchase history and KPIs, plus Starflow metadata configurations for Airflow DAGs and connections to local filesystems, DuckDB, and BigQuery.

samples/lineage · high confidence

Added default SQL expectation macros for data validation

The sample project now includes a new Jinja2 template file (default.j2) in the metadata expectations directory. This file provides a set of reusable SQL macros for common data quality checks, including verifying column uniqueness, validating row counts within a range, counting occurrences of specific values, and checking if a column value appears a specific number of times. These macros enable users to easily define and execute standard data validation rules within their Starbake audit workflows.

samples/starbake/metadata/expectations · high confidence

Added sample analytics and KPI transform definitions for the Starbake project

The sample project template now includes metadata and SQL definitions for two new analytics domains: \starbake\_analytics\ and \starbake\_kpis\. The \starbake\_analytics\ domain provides transforms for \customer\_purchase\_history\ (aggregating order counts, spend, and categories per customer) and \order\_items\_analysis\ (aggregating items and value per order). The \starbake\_kpis\ domain adds an \overall\_kpis\ transform that computes high-level metrics such as total revenue, average order value, and daily rates by joining the analytics tables. These files define the schema, write strategies (APPEND or OVERWRITE), and the underlying SQL logic for these sample data transformations.

src/main/resources/templates/bootstrap/samples/sample-project/metadata/transform · high confidence

Added sample data and validation templates for BigQuery, Postgres, and Snowflake samples

This change adds sample data files (JSON, CSV, PSV) and Jinja2 metadata expectation templates to the BigQuery, Postgres, and Snowflake sample directories. The sample data includes HR locations and sellers, as well as sales customers and orders, providing realistic datasets for testing. The metadata templates define reusable SQL macros for data quality checks, such as verifying column uniqueness, row count ranges, and value occurrences, which can be used to validate the sample data in each respective database environment.

(repo-wide) · high confidence

Added sample data files for Snowflake tutorial

New sample data files have been added to the \samples/snowflake/sample-data\ directory to support the Snowflake integration tutorial. This includes HR data (\locations-2018-01-01.json\, \sellers-2018-01-01.json\) and sales data (\customers-2018-01-01.psv\, \orders-2018-01-01.csv\), along with a shell script (\gcloud beta run jobs execute starlake-am\) demonstrating how to load these schemas into BigQuery using Starlake.

samples/snowflake/sample-data · high confidence

Added sample data for the Starbake demo

The Starbake sample-data directory now includes structured datasets for testing and demonstration purposes. This adds a CSV file containing 100 order lines with product IDs, quantities, and sale prices; a JSON file listing 32 orders with customer IDs, timestamps, and statuses (Pending, Delivered, Cancelled); and an XML file defining 12 products across categories like Cakes, Cookies, Breads, and Pies, including their prices and costs.

samples/starbake/sample-data · high confidence

Added sample dataset files for the Starbake project

The sample-project template now includes initial dataset files for the 'Starbake' scenario, providing ready-to-use sample data for testing and demonstration. Specifically, it adds a customers.csv file containing 10 customer records, a products.json file listing 20 bakery products with details like price and category, and an orders.json file with 20 sample orders linking customers to products. These files are located in the incoming/starbake directory and serve as the foundational data for the sample project's data pipeline.

src/main/resources/templates/bootstrap/samples/sample-project/datasets · high confidence

Added utility to split Starlake JSON schema into smaller files

A new Python utility script (utils/split\_schema.py) has been added to decompose the monolithic starlake.json schema into individual JSON files for specific definitions (e.g., TypeV1, DagGenerationConfigV1). This tool extracts targeted schema properties and their referenced dependencies, saving them as separate files in the resources directory to reduce token consumption when loading schema data.

utils · high confidence

Catalog browsing and preview capabilities with credential scrubbing

Users can now browse database schemas, tables, and columns, and preview data rows via the new catalog commands (catalog-connections, catalog-schemas, catalog-tables, catalog-columns, catalog-preview). The catalog service supports both JDBC and BigQuery backends, with BigQuery providing full browsing and preview functionality (including view support) and JDBC providing browsing plus row preview with identifier quoting and row limits. All catalog responses are wrapped in a JSON envelope, and large integers beyond 2^53 are sent as strings to avoid JSON precision loss. Additionally, credentials and sensitive URL parameters are automatically masked in connection listings and error messages to prevent accidental exposure.

src/main/scala/ai/starlake/serve/catalog · high confidence

Initial DuckDB write-strategy templates

Added a complete set of Jinja2 templates for DuckDB write strategies, including append, create, delete-then-insert, overwrite, overwrite-by-partition, SCD2, upsert-by-key, upsert-by-key-and-timestamp, and view creation. These templates define the SQL logic for inserting, updating, and managing data in DuckDB tables and materialized views.

src/main/resources/templates/write-strategies/duckdb · high confidence

Introduce DataFrame transformation pipeline for sink jobs

A new \DataFrameTransform\ trait and utility object have been added to the sink module, enabling data transformation logic to be applied to DataFrames before they are written to a sink. This introduces a pluggable mechanism where custom transformers can be injected to modify the data structure or content (such as adding headers or serializing values) prior to the final write operation, with an \IdentityDataFrameTransformer\ provided as the default no-op implementation.

src/main/scala/ai/starlake/job/sink · high confidence

Introduce KafkaClient utility for topic management and offset handling

Added a new KafkaClient utility class that provides centralized management for Kafka topics and consumer/producer offsets. This component supports creating and deleting topics, retrieving end offsets, and saving offsets either to a Kafka topic or a file-based storage path depending on the configured mode. It also handles partition extraction and admin client operations, serving as a foundational utility for Kafka integration within the application.

src/main/scala/ai/starlake/utils/kafka · high confidence

Introduce Starflow installation and runtime scripts with Windows and Unix parity

The distrib area now includes the primary installation and runtime scripts for Starflow (renamed from Starlake), providing unified setup and execution experiences across platforms. setup.sh and setup.ps1 handle interactive installation, version selection from GitHub Releases, Java version validation, and PATH configuration, while starlake.sh and starlake.cmd manage the runtime environment, including embedded JDK fallback, proxy support, and connector enablement flags. A new log4j2.properties file configures logging to suppress verbose third-party output, and .gitattributes ensures correct line endings for Windows batch files.

distrib · high confidence

Introduce dedicated CLI command for schema inference

Users can now run the \starlake infer-schema\ command to automatically detect file formats (CSV, JSON, XML, Parquet) and generate YAML schema definitions. This new command-line interface supports options to force specific formats, handle XML row tags, specify file encoding, and output results to a designated directory, providing a streamlined way to discover and define table schemas from sample data files.

src/main/scala/ai/starlake/job/infer · high confidence

Introduce file listing strategies for load jobs

Added a new \LoadStrategy\ trait and two concrete implementations (\IngestionNameStrategy\ and \IngestionTimeStrategy\) in the \src/main/scala/ai/starlake/job/load\ package. These components define how load jobs discover input files by delegating to the storage handler, with the strategies differing in whether the resulting file list is sorted by name or not.

src/main/scala/ai/starlake/job/load · high confidence

Introduce generic REST API data and schema extraction commands

Users can now extract data and infer schemas from generic REST API endpoints using the new \starlake extract-rest-data\ and \starlake extract-rest-schema\ CLI commands. The data extraction command supports multiple pagination strategies (offset, cursor, link header, page number), various authentication methods (Bearer, API Key, Basic, OAuth2), and features such as incremental extraction, resume capability, and configurable output formats (CSV or JSONL). The schema extraction command fetches sample data to infer table structures and detects schema evolution by comparing inferred fields against existing definitions.

src/main/scala/ai/starlake/extract/impl/restapi · high confidence

Introduce interactive console, BigQuery table info extraction, and Ducklake attachment parsing

This change adds an interactive REPL console (Console.scala) for local single-user execution, enabling users to run commands, query Spark SQL via dot-commands, and manage environment settings directly. It also introduces BigQueryTableInfo.scala to extract and sink dataset and table metadata (creation time, size, row counts) into the audit schema, and DucklakeAttachment.scala to parse Ducklake ATTACH statements for connection configuration. Additionally, it adds ExtractJdbcDataConfig.scala for detailed JDBC extraction settings, FileFormat.scala for CSV export configuration, and SanitizeStrategy.scala to control when data sanitization occurs (on load vs. on extract).

repository · high confidence

Introduce kafkaload command for bidirectional Kafka and BigQuery data pipelines

The \kafkaload\ CLI command is now available to load data from and offload data to Apache Kafka topics, supporting both batch and streaming modes. This change introduces a new \KafkaJob\ implementation that enables BigQuery as a source and sink format, allowing users to stream data from Kafka to BigQuery or ingest BigQuery tables/queries into Kafka. The command accepts configuration via \--config\ (topic name) and \--connectionRef\ (sink connection), with options to specify formats, paths, and streaming triggers.

src/main/scala/ai/starlake/job/sink/kafka · high confidence

Introduce native BigQuery job execution and branching support

This change introduces a new native BigQuery execution path alongside the existing Spark-based approach. A new \BigQueryNativeJob\ class allows direct execution of SQL queries and native data loads (including local file uploads) via the BigQuery API, bypassing Spark for simpler workloads. Additionally, a \BigQueryBranchHandler\ enables data branching by creating isolated datasets, cloning source and target tables, and rewriting SQL references to support safe experimentation or parallel processing. The refactoring also consolidates BigQuery job logic into a shared \BigQueryJobBase\ trait and introduces \BigQueryJobResult\ for standardized result handling.

src/main/scala/ai/starlake/job/sink/bigquery · high confidence

Introduce native SQL transformation tasks for BigQuery and JDBC

The transform module now supports executing SQL transformations natively on BigQuery and JDBC databases (including Snowflake, Redshift, PostgreSQL, and DuckDB) without requiring a Spark cluster, offering improved performance for MERGE and SCD2 operations. This change introduces a new \AutoTask\ abstraction with engine-specific implementations (\BigQueryAutoTask\, \JdbcAutoTask\) alongside the existing \SparkAutoTask\, enabling direct execution, transaction support, and native expectation handling. The \TransformCmd\ CLI has been updated to support these new execution paths, and the \TransformContext\ factory now automatically selects the appropriate task type based on the target engine and sink configuration.

src/main/scala/ai/starlake/job/transform · high confidence

Introduce native loaders for BigQuery, DuckDB, and Snowflake

Added new native ingestion loaders for BigQuery, DuckDB, and Snowflake that bypass Spark to load data directly into the target data warehouse. These loaders support two-step ingestion (using temporary tables) for complex schemas, POSITION format handling, and robust reject capture with replay file generation. BigQuery native loading includes support for data branching via SL\_DATA\_BRANCH, while DuckDB and Snowflake loaders handle fixed-width and CSV formats with detailed error reporting to the audit rejected table. Also in this area: 5 commits (4 fixes) fixing src/main.

src/main/scala/ai/starlake/job/ingest/loaders · medium confidence · unverified

Introduce new row-level validation framework

The validator module now includes a new row-level validation architecture with a \GenericRowValidator\ trait and specific implementations (\AcceptAllValidator\, \FlatRowValidator\, \TreeRowValidator\, \NativeValidator\) that process dataframes against schema attributes and privacy options. This replaces previous validation logic with a structured \RowValidator\ class that handles data preparation, type fitting, and error collection, returning a \CheckValidityResult\ containing accepted, rejected, and error datasets.

src/main/scala/ai/starlake/job/validator · high confidence

Introduce project validation command and dedicated storage handler abstractions

Users can now run the \starlake validate\ command to reload all YAML configuration files from disk and check for errors or warnings, including testing database connections for all defined connections. This change also introduces a new \StorageHandler\ trait and its implementations (\HdfsStorageHandler\, \LocalStorageHandler\) to abstract filesystem operations, alongside new helper objects for access control (\AccessControlQueries\), DAG configuration (\DagHandler\), and scheduling (\SchedulingQueries\) to centralize metadata handling.

src/main/scala/ai/starlake/schema/handlers · high confidence

Native audit logging and ingestion job factory

The ingestion module now uses a dedicated \AuditLog\ model and \AuditTaskBuilder\ to write load and transform outcomes directly to the audit table, ensuring string literals are safely escaped against SQL injection and Jinja template injection. A new \IngestionJobFactory\ centralizes the creation of format-specific ingestion jobs (such as \DsvIngestionJob\, \JsonIngestionJob\, and native loaders for BigQuery, Snowflake, and DuckDB) via an \IngestionContext\, replacing the previous scattered instantiation logic. Additionally, new CLI commands \autoload\ and \ingest\ are introduced to support automatic schema inference and direct file ingestion.

src/main/scala/ai/starlake/job/ingest · high confidence

New 'any-engine' sample demonstrating multi-environment data pipelines

This change introduces a comprehensive sample project in \samples/any-engine\ that demonstrates how to configure and run Starlake pipelines across multiple environments (LOCAL, BQ, SNOWFLAKE, DUCKDB, REDSHIFT, POSTGRESQL). It includes environment-specific shell scripts for initialization, data import, lineage, loading, and transformation, alongside metadata definitions for various data sources (JSON, CSV, XML, DSV) and write strategies (OVERWRITE, UPSERT, SCD2). The sample also provides DAG templates for Airflow and Dagster orchestration, connection configurations for BigQuery and Snowflake, and example SQL/expectation macros, serving as a reference implementation for setting up a complete data ingestion and transformation workflow.

samples/any-engine · high confidence

New BigQuery schema converter utility added

Added a repackaged BigQuery schema converter utility (BigQuerySchemaConverters.java) and supporting data type definitions (SupportedCustomDataType.java) to handle schema translation between BigQuery and Spark formats. This includes logic for converting BigQuery schemas to Spark StructTypes, handling pseudo-columns like \_PARTITIONTIME, and managing custom data types such as Spark ML vectors and matrices, addressing compatibility issues with the Spark BigQuery connector.

src/main/java/ai/starlake/utils · high confidence

New BigQuery-specific schema and data extraction commands

Added new CLI commands \bq-info\ and \extract-bq-schema\ to retrieve metadata and extract schemas from BigQuery datasets, including support for filtering by table type (TABLE, VIEW, MATERIALIZED\_VIEW, EXTERNAL, SNAPSHOT) and handling view schemas via dry-run queries. Introduced \extract-data\ and \extract-schema\ commands to parallelize data and schema extraction from JDBC databases and BigQuery, allowing users to configure parallelism, partitioning, incremental exports, and output formats (CSV/Parquet). The \extract-script\ command remains available for generating custom extraction scripts from Mustache templates.

src/main/scala/ai/starlake/extract · high confidence

New BigQuery/Spark loader sample with multi-database metadata and schema evolution

This sample now includes a complete metadata configuration for loading data into BigQuery via Spark, featuring application connection settings, environment variables, and type definitions mapped to BigQuery, Snowflake, Postgres, and Synapse. It provides table schemas for HR, sales, and KPI datasets with support for primary keys, foreign keys, and data quality expectations. Additionally, it introduces Jinja2 templates for generating DDL statements, enabling automatic schema evolution (adding, dropping, or altering columns) when source data changes.

samples/bigquery-spark-loader/metadata · high confidence

New CLI command to load files directly into JDBC tables

Users can now use the new \starlake cnxload\ command to load Parquet, CSV, or JSON files directly into a JDBC-connected database table. This feature introduces a dedicated CLI entry point (\JdbcConnectionLoadCmd\) and backend logic (\SparkJdbcWriter\) that supports configurable connection options and write strategies (APPEND or OVERWRITE). The implementation automatically handles schema evolution by adding or dropping columns as needed, ensures column names are uppercase for JDBC compatibility, and uses a temporary Parquet intermediate step for DuckDB targets to avoid driver issues.

src/main/scala/ai/starlake/job/sink/jdbc · high confidence

New CLI commands for ACL export, ACL dependency visualization, task lineage, column lineage, and table dependencies

This change introduces five new command-line tools in the \ai.starlake.lineage\ package to improve data governance and lineage visibility. The \starlake acl --export\ command allows users to export Access Control List (ACL) and Row Level Security (RLS) entries to a YAML file. The \starlake acl-dependencies\ command generates GraphViz diagrams (DOT, SVG, PNG, or JSON) visualizing ACL relationships across domains and tasks. The \starlake lineage\ command creates task dependency graphs to visualize data lineage across the project. The \starlake col-lineage\ command builds and exports column-level lineage for specific tasks as JSON. Finally, the \starlake table-dependencies\ command generates dependency diagrams showing foreign key and lineage relationships between tables, with support for SVG, PNG, and JSON outputs.

src/main/scala/ai/starlake/lineage · high confidence

New CLI commands for DAG generation, deployment, and Excel-to-YAML conversion

This change introduces several new command-line capabilities in the schema generator module. Users can now generate workflow DAG files for tasks and domains using the new \dag-generate\ command, which supports tag filtering and role definitions. These generated DAGs can be deployed to orchestration tools like Airflow via the new \dag-deploy\ command, which also handles library installation and \.airflowignore\ configuration. Additionally, the \xls2yml\ and \xls2ymljob\ commands allow users to convert Excel spreadsheets describing domains, schemas, and transform jobs into Starflow YAML configuration files, including support for IAM policy tags and access control policies.

src/main/scala/ai/starlake/schema/generator · high confidence

New DuckLake sample for TPC-H data pipeline

Added a new sample project in samples/ducklake that demonstrates an end-to-end data pipeline using Starflow and DuckLake. The sample includes configuration for loading TPC-H benchmark data (CUSTOMER, ORDERS, LINEITEM, PART, PARTSUPP, SUPPLIER, NATION, REGION) into DuckLake, along with SQL transformations for analytics and KPIs. It provides orchestration templates for Airflow, Dagster, and Snowflake, data quality expectations, and a data contract definition.

samples/ducklake · high confidence

New JSON ingestion utility for schema validation and type coercion

Added JsonIngestionUtil.scala, a new utility object that provides schema validation and type coercion logic for JSON data sources. It includes a compareTypes method to check compatibility between a defined schema and incoming dataset types (handling structs, arrays, and numeric promotions) and a findTightestCommonTypeOfTwo function to determine the common type between two data types, mirroring Spark's TypeCoercion behavior.

src/main/scala/org/apache/spark/sql/execution/datasources/json · high confidence

New Java-based Setup utility for dependency management

A new \Setup.java\ class has been introduced to handle the installation, configuration, and management of project dependencies. This utility replaces previous setup mechanisms by providing a comprehensive Java-based installer that supports interactive CLI menus for selecting features (such as Spark, DuckDB, Kafka, Snowflake, and various database connectors) and non-interactive configuration via environment variables. It manages the downloading and extraction of JARs and binaries from Maven Central, GitHub Releases, and other sources, while also handling platform-specific requirements like Windows \winutils\ runtime dependencies and proxy configuration (HTTP/HTTPS/SOCKS).

src/main/java · high confidence

New Redshift write strategy templates

Added new Jinja2 SQL templates for Amazon Redshift write strategies, including append, create, delete\_then\_insert, overwrite, overwrite\_by\_partition, scd2, upsert\_by\_key, upsert\_by\_key\_and\_timestamp, and view. These templates provide the underlying SQL logic for various data loading and transformation operations on Redshift tables.

src/main/resources/templates/write-strategies/redshift · high confidence

New SPI for generic REST and OpenAPI schema extraction

The schema extraction subsystem now supports extracting schemas from generic REST APIs and OpenAPI definitions, in addition to existing JDBC sources. This is enabled by new SPI components in the extract package: a SchemaExtractor trait for connector implementations, a SchemaExtractorFactory that routes configuration to OpenAPI or REST API extractors, and a SchemaExtractorWorkflow that orchestrates the extraction, merges results with existing domain/table configurations, and serializes the output as YAML files.

src/main/scala/ai/starlake/extract/spi · high confidence

New SQL expectation macros for data quality validation

The empty-project template now includes a comprehensive set of Jinja2 macros for generating SQL-based data quality checks. These new files—Completeness, Numeric, Schema, UnexpectedRowsExpectation, Uniqueness, Validity, and Volume—provide ready-to-use queries for validating nulls, numeric ranges, schema structure, row counts, uniqueness, and value sets, enabling users to immediately apply these standards to their data.

src/main/resources/templates/bootstrap/samples/empty-project/metadata/expectations · high confidence

New Starlake test command for load and transform validation

Users can now run unit tests for load and transform tasks via the new \starlake test\ command. This feature allows filtering tests by domain, table, or specific test name, and supports generating HTML reports or JUnit XML output. The implementation includes configuration handling, result aggregation, and template loading to facilitate data validation and reporting.

src/main/scala/ai/starlake/tests · high confidence

New \`starlake migrate\` command for automatic project configuration upgrades

Users can now run the \starlake migrate\ command to automatically upgrade their Starflow project configuration files (YAML) to the latest schema version. This new feature, implemented in the \src/main/scala/ai/starlake/migration\ package, introduces a \MigrateCmd\ entry point and a \ProjectMigrator\ that processes metadata folders (extract, load, refs, types, dags, env, transform). It includes specific file migrators to handle breaking changes, such as automatically splitting domain descriptions into separate table files and renaming folders where necessary, ensuring a smoother transition to version 1.2.0 and beyond.

src/main/scala/ai/starlake/migration · high confidence

New bootstrap command to scaffold Starflow projects

Users can now initialize a new Starflow project using the \starlake bootstrap\ command. This new CLI command scaffolds a project directory structure, copies configuration files and templates (such as quickstart or userguide), sets up IDE support (VS Code settings, JSON schema), and configures agent folders. It supports options to specify a custom template, define the absolute project location, and control exit behavior, streamlining the initial setup process for new Starflow applications.

src/main/scala/ai/starlake/job/bootstrap · high confidence

New configuration and connection management components

The project introduces a new set of configuration and connection management components in the \ai.starlake.config\ package. This includes \ConnectionInfo\ for defining and managing database connections (including encryption/decryption of passwords and access tokens), \Settings\ for centralizing application configuration using PureConfig, \DatasetArea\ for handling dataset path construction, \SparkEnv\ and \SparkSessionBuilder\ for managing Spark session lifecycle and configuration (including Spark Connect support), \KryoSerialization\ for registering classes for Spark serialization, and \SettingsCmd\ for providing CLI commands to display settings and test connections. These changes refactor and centralize how configuration, connections, and Spark environments are handled.

src/main/scala/ai/starlake/config · high confidence

New configuration merging and schema extraction utilities

This change introduces new utility classes in the core module to support schema extraction and configuration merging. LoadConfigMelder adds configurable strategies for merging extracted metadata with existing domain and table schemas, allowing users to define precedence rules for attributes like comments, tags, and renames. CaseClassToPojoConverter enables the conversion of Scala case classes to Java maps, facilitating easier usage in Jinja-style templating. Additionally, StringUtils and NamingUtils provide helpers for normalizing table and attribute names, including support for removing OpenAPI parameters and handling accents.

src/main/scala/ai/starlake/core · high confidence

New data quality expectation templates for sample projects

The sample project template now includes a comprehensive set of Jinja2 macros for generating SQL-based data quality checks. New files in the metadata expectations directory provide templates for validating Completeness (null checks), Numeric properties (KL divergence, min/max/mean/median ranges, most common values), Schema integrity (column existence, type checks, column count and ordering), Uniqueness (distinct value sets, unique constraints), Validity (value sets, patterns, lengths, pair equality), Volume (row counts), and Unexpected Rows (set differences between tables).

src/main/resources/templates/bootstrap/samples/sample-project/metadata/expectations · high confidence

New extract sample demonstrating REST API data ingestion

The samples/extract directory now includes a new example for ingesting data from REST APIs. This addition features a configuration file (rest-api-extract.sl.yml) that defines endpoints with support for bearer authentication, offset, cursor, and page-number pagination strategies, as well as child endpoints and POST requests with request bodies. The sample is accompanied by a shell script (1.extract-schema.sh) and a JDBC connection template to set up a local H2 database and execute the extraction, providing a complete workflow for SaaS/API data ingestion.

samples/extract · high confidence

New metrics command for data quality profiling

Users can now run the \starlake metrics\ command to compute and publish data quality metrics (continuous, discrete, and frequency) for a specified domain and schema. This new capability introduces a dedicated metrics job that aggregates statistics into parquet files and supports optional Google Cloud authentication for report generation.

src/main/scala/ai/starlake/job/metrics · high confidence

New project comparison and adaptive write capabilities

This update introduces two new capabilities in the schema module. First, it adds a \compare\ command that allows users to compare two versions of a Starflow project (via file paths, git commits, or tags) and generate a diff report using a customizable HTML template. Second, it implements an adaptive write strategy that dynamically selects the appropriate write mode (such as APPEND, OVERWRITE, UPSERT\_BY\_KEY, or DELETE\_THEN\_INSERT) for a schema based on conditions evaluated against incoming file metadata, grouping files by the resulting strategy for efficient ingestion.

src/main/scala/ai/starlake/schema · high confidence

New release, Docker, and utility scripts for Starflow

This change introduces a suite of new shell scripts in the \scripts/\ directory to support the Starflow release process, Docker image management, and development utilities. The \local-release.sh\ script provides an idempotent, resumable workflow for building and publishing \starlake-core\ and \starlake-api\ to GitHub Releases, including steps for version management, UI injection, and Discord announcements via \announce-release-discord.sh\. Docker operations are handled by \docker-build.sh\, \docker-prepare.sh\, and \docker-multi-arch-build.sh\, which manage environment-specific builds (cloud, local, dev) and artifact publishing. Supporting infrastructure includes \release-lib.sh\ for shared release helpers, \versions.sh\ for version extraction, and \pyspark-smoke-test.sh\ for validating PySpark execution within Docker images.

scripts · high confidence

New semantic-export command to convert models to LookML, Power BI TMDL, and Apache Ossie

Users can now export semantic models from Starflow's metadata store to external BI and semantic formats using the new \starlake semantic-export\ command. The command supports three target formats: Apache Ossie (the default), LookML (generating a project with view and model files, requiring a \--connection\ flag for the Looker connection), and Power BI TMDL (generating a folder structure with TMDL files, also using \--connection\ for Power Query sources). This allows teams to migrate or sync their Starflow semantic definitions to Looker, Power BI, or Ossie-compatible tools.

src/main/scala/ai/starlake/semantic · high confidence

Register DuckDB as a Spark SQL data source

The application now registers the DuckDB relation provider with Spark SQL via the standard Java SPI mechanism. This enables users to query DuckDB data directly using Spark SQL syntax (e.g., CREATE TABLE ... USING duckdb) without needing to manually configure the data source class.

src/main/resources/META-INF · high confidence

Support for expectations

The system now supports defining and validating data expectations, allowing users to specify quality rules for their datasets. This feature enables automated checks to ensure data integrity and consistency during ingestion and processing workflows.

(repo-wide) · high confidence

Architecture

Decompose utils into focused utility modules

The monolithic \Utils\ object has been decomposed into a set of focused, single-responsibility utility modules within the \ai.starlake.utils\ package. This refactoring introduces dedicated classes for specific concerns: \AESEncryption\ for data encryption, \AnyTemplateLoader\ for DAG template management, \CliConfig\ and \CliEnvConfig\ for command-line interface handling, \CometJacksonModule\ for custom JSON serialization, \CompilerUtils\ for expectation code compilation, \DataTypeEx\ for Spark type checks, \FileLock\ for distributed locking, \Formatter\ for string templating, \GcpCredentials\ and \GcpUtils\ for Google Cloud integration, \GraphVizUtils\ for diagram generation, \HadoopModule\ for Hadoop path serialization, and \JarUtil\ for resource loading. This change improves code maintainability and separation of concerns without altering external product behavior.

src/main/scala/ai/starlake/utils · high confidence

Workflow logic restructured into focused trait mixins

The monolithic workflow implementation has been split into distinct, composable traits (InferWorkflow, IngestionWorkflow, MetricsSecurityWorkflow, SinkWorkflow, TestWorkflow, and TransformWorkflow) that are mixed into the main IngestionWorkflow class. This change organizes specific capabilities—such as schema inference, data ingestion staging, security policy application, sink operations, testing, and transformations—into separate modules, improving code modularity and maintainability without altering the external user-facing behavior.

src/main/scala/ai/starlake/workflow · high confidence

Behavioural changes

BigQuery write strategies now use native MERGE and partition-aware operations

The BigQuery write templates have been replaced with new implementations that leverage BigQuery's native MERGE statement for upserts, SCD2, and delete-then-insert strategies, replacing previous approaches. These templates now support partition pruning via DECLARE variables to optimize performance on large partitioned tables, handle materialized views explicitly, and ensure partition filter requirements are managed correctly during overwrites.

src/main/resources/templates/write-strategies/bigquery · high confidence

CLI command execution now returns proper exit codes

The Starflow CLI now correctly propagates command results to the operating system via exit codes. Previously, the main entry point did not distinguish between successful and failed executions; now, the \Main\ object inspects the result of each command and calls \System.exit\ with 0 for success and 1 for any failure (including exceptions or soft failures like empty preload results). This allows scripts and orchestration tools to reliably detect when a Starflow command has failed.

src/main/scala/ai/starlake/job · high confidence

Data Catalog site replaces Docusaurus with a standalone Jinja/React implementation

The documentation and catalog interface has been migrated from Docusaurus to a lightweight, standalone site rendered with Jinja templates and styled with custom CSS. The new layout features a responsive sidebar for navigating domains and jobs, and table/task detail pages now use tabs to organize General, Attributes, SQL/Python, and Expectations content. Interactive data lineage and access control diagrams are rendered client-side using React and the @xyflow/react library, replacing the previous static or framework-dependent visualizations.

src/main/resources/templates/site · high confidence

Fallback for Dataset.showString in Spark 4 compatibility layer

A new \DatasetLogging\ trait provides an implicit helper to safely call \showString\ on Spark \Dataset\ objects. Since Spark 4 moved the \Dataset\ implementation to \org.apache.spark.sql.classic\, this helper detects classic datasets and delegates to their \showString\ method; for other dataset types (such as Connect datasets), it falls back to returning the schema tree string instead of failing.

src/main/scala/org/apache/spark/sql · high confidence

Installer now downloads only changed dependencies

The setup process has been updated to perform incremental installs and upgrades. Instead of re-downloading all dependencies every time, the installer now compares the desired artifacts against files already present on disk, downloading only those that are missing, outdated, or have changed size. It also automatically removes obsolete or disabled dependency files to keep the installation directory clean.

src/main/java/ai/starlake/setup · high confidence

Introduce modular reference configuration files and sample data

The monolithic reference configuration has been split into dedicated files (reference-audit.conf, reference-connections.conf, reference-dags.conf, reference-engines.conf, reference-expectations.conf, reference-extra.conf, reference-general.conf, reference-internal.conf, reference-kafka.conf, reference-metrics.conf, reference-refs.conf, and reference-service.conf) to organize settings by domain. Additionally, sample data files (SCHEMA-VALID.dsv, SCHEMA-VALID-NOHEADER.dsv) and an \_\init\\_.py file have been added to the resources directory.

src/main/resources · high confidence

Introduces Caffeine-based caching for settings and Snowflake tokens

The server now uses Caffeine to cache settings, schema handlers, storage handlers, and Snowflake user tokens, with entries expiring after 10 minutes of inactivity and a maximum size of 10,000. This change improves load performance by reducing repeated configuration parsing and token lookups, and the settings cache can be explicitly refreshed or reset via the API.

src/main/scala/ai/starlake/serve · high confidence

Migration tooling now supports versioned metadata and updated type mappings

The migration module has been updated to support a new versioned metadata format (version 1) for configuration files, introducing a structured schema for application connections, DAG templates, environment settings, and extract/load/transform definitions. This change includes a comprehensive update to the default type definitions, which now provide explicit DDL mappings for BigQuery, Snowflake, Postgres, and Synapse across various data types (e.g., string, integer, timestamp). Additionally, the migration logic has been adjusted to handle legacy 'unversioned' metadata structures, ensuring compatibility during the transition to the new versioned format.

migration · high confidence

New Project Comparison Report template with Jinja2 syntax fixes

The compare report view now uses a new Jinja2 template (index.html.j2) that renders a structured HTML report for project diffs, including summary cards and collapsible sections for load, domain, table, and attribute changes. The template also displays metadata field names in the report to clarify what changed. Additionally, Jinja2 syntax was corrected by replacing the non-standard is-not-empty filter with the standard length filter to ensure proper evaluation of empty collections.

src/main/resources/templates/compare · high confidence

New Spark SQL templates for write strategies

The Spark write-strategy templates have been replaced with new Jinja2 files that generate specific SQL operations. The system now supports append, delete-then-insert, overwrite (with Delta Lake and materialized view handling), overwrite by partition, SCD2 (Slowly Changing Dimension Type 2), upsert by key, upsert by key and timestamp, and view creation. These templates define the exact SQL logic used for data loading and transformation in Spark environments.

src/main/resources/templates/write-strategies/spark · high confidence

New default write-strategy template with column quoting fixes

A new default Jinja2 template for write strategies has been introduced, providing macros to generate SQL for strategy keys, joins, and column lists. This template includes a specific fix to unquote column names before re-quoting them, ensuring that identifiers are handled correctly without double-quoting or syntax errors when generating SQL statements.

src/main/resources/templates/write-strategies · high confidence

New exception types for ingestion and validation errors

The system now introduces specific exception classes to handle distinct failure scenarios during data processing. Users will see \DisallowRejectRecordException\ when a rejected record count limit is exceeded, \DataExtractionException\ when data extraction fails for a specific domain and table, and \SchemaValidationException\ when JSON schema validation fails. These changes improve error granularity and clarity for ingestion and validation issues.

src/main/scala/ai/starlake/exceptions · high confidence

Package namespace renamed from com.ebiznext to ai.starlake

The project's Java/Scala package namespace has been renamed from com.ebiznext to ai.starlake. This change affects all source files and imports within the project, requiring users to update any external code or configurations that reference the previous package names.

project · high confidence

PostgreSQL write strategies now use MERGE and standard SQL constructs

The PostgreSQL write-strategy templates have been rewritten to use modern SQL syntax, replacing previous implementations with MERGE statements for upserts (upsert\_by\_key, upsert\_by\_key\_and\_timestamp) and SCD2, and standard INSERT/DELETE patterns for append, overwrite, and delete\_then\_insert. This change improves compatibility with PostgreSQL 15+ and ensures consistent behavior across strategies like SCD2, which now correctly handles temporal updates using MERGE and temporary views.

src/main/resources/templates/write-strategies/postgresql · high confidence

Refactor SQL utilities and add type mappings for BigQuery, Snowflake, and DuckDB

The SQL utility module has been split into dedicated components: SqlParser and SqlFormatter handle parsing and formatting logic respectively, while SQLUtils now delegates to them. A new SQLTypeMappings object provides explicit type conversion maps for BigQuery, Snowflake, DuckDB, PostgreSQL, and Redshift to Spark types. Additionally, a SqlCommentStripper component was added to aggressively remove SQL comments while preserving string literals, and a new SQLCompletion class was introduced.

src/main/scala/ai/starlake/sql · high confidence

Restored Oracle database export script template and expected outputs

The \samples/database\ area now includes the \EXTRACT\_TABLE.sql.j2\ Jinja2 template for generating Oracle SQL\*Plus scripts, along with \expected\_script\_payload.txt\ and \expected\_script\_payload2.txt\ as reference outputs. This restores the capability to generate incremental and full data exports from Oracle tables, handling configuration such as custom delimiters, date formats, and export status tracking via the \COMET\_EXPORT\_STATUS\ table.

samples/database · high confidence

Schema model refactoring and new comparison capabilities

The schema model layer has been restructured to support project comparison and more granular schema synchronization. A new \Project\ object enables comparing two project directories to identify differences in domains and jobs, returning a structured \ProjectDiff\. Attribute merging logic has been extracted into a dedicated \AttributeMerger\ object, allowing for configurable strategies when merging schema attributes (e.g., handling container mismatches or deciding which source takes precedence). Additionally, new model classes have been introduced to support these features, including \AttributeStatus\ for tracking changes (Added, Modified, Removed, Unchanged), \TableSync\ for defining synchronization strategies (None, Add, All), and \ExpectationLibrary\ for managing reusable validation macros. Several other model types like \Materialization\, \MergeOn\, and \WriteStrategyType\ have also been added or refined to support hybrid table support and advanced write strategies.

src/main/scala/ai/starlake/schema/model · high confidence

Snowflake write strategies now support hybrid tables, materialized views, and clustered columns

The Snowflake write-strategy templates have been updated to support new Snowflake capabilities. Users can now create and overwrite HYBRID TABLES and MATERIALIZED VIEWs in addition to standard tables. The templates also support defining clustered columns via the sinkTableClusteringClause and utilize the QUALIFY clause for deduplication logic in source-and-target strategies. Existing strategies like SCD2, upsert, and delete-then-insert have been refined to handle these new object types and clustering configurations correctly.

src/main/resources/templates/write-strategies/snowflake · high confidence

Standardized load-strategy templates and added Snowflake context support

The load-strategy templates for BigQuery, DuckDB, JDBC, PostgreSQL, Redshift, and Spark have been replaced with empty placeholders, while the Snowflake template now explicitly exposes the Python JSON context, statements, expectation items, audit data, and expectations to the rendering engine. A new shared default template provides reusable macros for handling pseudo-columns, join conditions, and column quoting, ensuring consistent SQL generation logic across supported databases.

src/main/resources/templates/load-strategies · high confidence

Unify JDBC write strategy templates

The SQL templates for JDBC write strategies (append, create, delete\_then\_insert, overwrite, overwrite\_by\_partition, scd2, upsert\_by\_key, upsert\_by\_key\_and\_timestamp, and view) have been consolidated into a single unified location. This change standardizes how data is written to target tables, ensuring consistent behavior across different write modes such as upserts, SCD2, and overwrites.

src/main/resources/templates/write-strategies/jdbc · high confidence

Updated bootstrap samples with new documentation and configuration files

The bootstrap samples now include a README explaining how to run extract, load, and transform workflows for various data warehouses (BigQuery, Snowflake) and local environments. Additionally, an extensions.json file recommends specific VS Code extensions (IBM.output-colorizer, samuelcolvin.jinjahtml, Starlake.starlake), and a gitignore file is provided to exclude most files except metadata and datasets directories.

src/main/resources/templates/bootstrap/samples · high confidence

Test coverage

1 commit adding/updating tests in src/test/resources/sample/new-types, src/test/resources/sample/privacy; 2 commits adding/updating tests in src/test/resources/integration; 6 commits adding/updating tests in src/test/scala; Added BigQuery integration test suite; Added DDL template tests for BigQuery and Synapse; Added Jinja2 macro templates for test expectations; Added Kafka integration test suites; Added SQL test cases for view rewriting logic; Added comprehensive test coverage for data extraction and migration features; Added comprehensive test coverage for schema model components; Added comprehensive tests for RowValidator; Added configuration and connection-info unit tests; Added disabled Snowflake freshness integration test; Added expected DOT graph outputs for ACL and domain model tests; Added integration test infrastructure for BigQuery and JDBC; Added integration tests for BigQuery, Postgres, Redshift, and Snowflake transforms; Added integration tests for Starbake workflows; Added integration tests for Starlake CLI commands and utilities; Added integration tests for data and schema extraction; Added integration tests for data loading across multiple storage backends; Added integration tests for lineage and dependency commands; Added native test resources for sales data loading; Added sample Python Pi job and its SL configuration; Added sample data files for schema inference tests; Added sample metadata and Jinja templates for testing transform configurations; Added sample test data for sales quickstart; Added sample test resources for schema validation and ingestion; Added test coverage for JSON, position, and load strategy ingestion; Added test coverage for Starlake utility functions; Added test coverage for metrics jobs and expectation report escaping; Added test coverage for schema generator components; Added test coverage for transform job handling and DuckDB write strategies; Added test fixtures for CSV schema inference; Added test fixtures for DSV max-errors validation; Added test fixtures for DSV rename and position-based header replay; Added test fixtures for HR quickstart validation; Added test fixtures for JSON DuckDB native loader; Added test fixtures for Jinja delimiter escaping in DSV loading; Added test fixtures for POSITION format validation and DSV options; Added test fixtures for SQL export script generation; Added test fixtures for adaptive write, merge, and schema extraction scenarios; Added test fixtures for merge operations; Added test fixtures for merge scenarios; Added test fixtures for position-based parsing and custom types; Added test for ESLoad command usage output; Added test for JDBC-to-JDBC connection jobs; Added test resource configuration and data fixtures; Added test resource marker file; Added test resources for DSV DuckDB replay failure scenarios; Added test resources for DSV pre-existing table loading; Added test resources for Dream sample data and schema; Added test resources for DuckDB native DSV reject handling; Added test resources for DuckDB native loader POSITION format; Added test resources for XSD-based XML validation and type definitions; Added test resources for adaptive write strategies; Added test resources for position-based encoding with ISO-8859-1 data; Added test resources for position-based ingestion with regex filtering and ignore logic; Added test resources for the test framework; Added test template for DAG generation with raw domain support; Added tests for BigQuery and DuckDB schema conversion utilities; Added tests for BootstrapConfig usage output; Added tests for CaseClassToPojoConverter template rendering; Added tests for DuckDB native load second-step failure and retry scenarios; Added tests for JDBC connection load job command usage; Added tests for JSON Schema inference and CLI usage verification; Added tests for JSON ingestion schema validation; Added tests for LoadConfigMelder domain merging logic; Added tests for Parquet to CSV conversion job; Added tests for Quack CLI parsing, state management, and server integration; Added tests for SettingsManager configuration updates; Added tests for lineage dependency graph cycle handling and table metadata emission; Added tests for native loader reject capture, replay file naming, and Snowflake POSITION support; Added tests for schema inference, ingestion, and domain directory handling; Added tests for semantic model export to LookML, TMDL, and Ossie formats; Added tests for the generic REST API extractor; Added tests for the new catalog service and credential scrubbing; Added tests for the privacy engine's data transformation and hashing capabilities; Added unit tests for SQL reference substitution and table extraction; Added unit tests for the Starlake test framework; Added unit tests for transform strategy SQL generation across multiple engines; Expanded test coverage for DuckDB native ingestion and audit behaviors; New HTML test report templates for Starflow; New test infrastructure for JDBC validation and adaptive write strategies.

Dependencies

Upgrade to Spark 4.1.3 and Java 17 with updated Google Cloud dependencies

The build configuration has been updated to target Apache Spark 4.1.3 and Java 17 (via \--release 17\), replacing previous Spark 3 and Java 8/11 support. Scala 2.13.18 is now the default and only supported version, with Scala 2.12 and 2.11 dropped. Google Cloud dependencies have been aligned to BigQuery 2.68.0, DataCatalog 1.101.0, Logging 3.36.0, and Protobuf 4.35.1 (with a runtime override to 4.36.1 to satisfy gencode requirements). The assembly process now excludes Conscrypt to prevent JVM crashes on macOS arm64 and Linux, and uses specific merge strategies for Spark service files.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

This is the PUBLIC form of this artifact. Findings are listed in full, but the details of SECURITY findings — which rule fired, in which file, on which line, and how to fix it — are deliberately withheld, and any secret-scanner results are excluded entirely. Where detail is absent here it was REMOVED FOR PUBLICATION; it is not missing from the analysis. The complete artifact is available from the repository owner.

Score

  • CAI 43 → 38 (-5.4)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 61 → 62 (+0.2)
  • Architecture 85 → 84 (-1.3)
  • Maturity 71 → 75 (+3.1)
  • Readiness 50 → 39 (-11.1)
  • Security 35 → 27 (-8.1)
  • Accessibility 41 → 41 (+0.0)

Resolved (27)

  • Change coupling: ExtractDataCmd.scala ↔ JdbcDbUtils.scala (src/main/scala/ai/starlake/extract/ExtractDataCmd.scala)
  • Change-coupling hub: BootstrapCmd.scala → InferSchemaCmd.scala, LoadCmd.scala, TransformCmd.scala (src/main/scala/ai/starlake/job/bootstrap/BootstrapCmd.scala)
  • Change-coupling hub: TableInfo.scala → BigQueryLoadConfig.scala, XlsAutoJobReader.scala, AutoTaskInfo.scala (src/main/scala/ai/starlake/schema/model/TableInfo.scala)
  • ClassTooLong: BigQueryNativeJob (src/main/scala/ai/starlake/job/sink/bigquery/BigQueryNativeJob.scala)
  • Duplicated block (12 lines × 2) (src/main/scala/ai/starlake/job/metrics/ExpectationAssertionHandler.scala)
  • Duplicated block (12–13 lines × 3) (src/main/scala/ai/starlake/job/ingest/loaders/DuckDbNativeLoader.scala)
  • Duplicated block (13 lines × 2) (src/main/scala/ai/starlake/job/transform/AutoTask.scala)
  • Duplicated block (19–21 lines × 2) (src/main/scala/ai/starlake/schema/handlers/InferSchemaHandler.scala)
  • Edited copy of a member (20 corresponding lines) (src/main/scala/ai/starlake/schema/model/SchemaInfo.scala)
  • High: security finding (details withheld)
  • Members sharing a duplicated core (4 members, 50+ identical tokens) (src/main/scala/ai/starlake/job/transform/TransformContext.scala)
  • Off-boarding risk: anonymized user #1
  • Outdated: ai.starlake:starlake-airflow
  • Outdated: com.fasterxml.jackson.dataformat:jackson-dataformat-csv
  • Outdated: com.github.ben-manes.caffeine:caffeine
  • Outdated: com.github.pureconfig:pureconfig_2.13
  • Outdated: com.github.scopt:scopt_2.13
  • Outdated: com.google.cloud.bigdataoss:gcs-connector
  • Outdated: com.google.cloud:google-cloud-bigquery
  • Outdated: com.google.cloud:google-cloud-logging
  • …and 7 more

New (51)

  • BigQueryNativeLoader.findRejectedRows (cognitive 21) (src/main/scala/ai/starlake/job/ingest/loaders/BigQueryNativeLoader.scala)
  • BigQueryNativeLoader.run (cognitive 40) (src/main/scala/ai/starlake/job/ingest/loaders/BigQueryNativeLoader.scala)
  • BigQueryNativeLoader.run (cyclomatic 17) (src/main/scala/ai/starlake/job/ingest/loaders/BigQueryNativeLoader.scala)
  • BigQueryNativeSchema.decimalType (cognitive 16) (src/main/scala/ai/starlake/utils/conversion/BigQueryNativeSchema.scala)
  • BigQueryNativeSchema.patchFields (cognitive 25) (src/main/scala/ai/starlake/utils/conversion/BigQueryNativeSchema.scala)
  • Change coupling: AutoTaskInfo.scala ↔ TableInfo.scala (src/main/scala/ai/starlake/schema/model/AutoTaskInfo.scala)
  • Change coupling: BootstrapCmd.scala ↔ InferSchemaCmd.scala (src/main/scala/ai/starlake/job/bootstrap/BootstrapCmd.scala)
  • Change coupling: BootstrapCmd.scala ↔ LoadCmd.scala (src/main/scala/ai/starlake/job/bootstrap/BootstrapCmd.scala)
  • Change coupling: XlsAutoJobReader.scala ↔ TableInfo.scala (src/main/scala/ai/starlake/schema/generator/XlsAutoJobReader.scala)
  • Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
  • Duplicated block (10–11 lines × 2) (src/main/scala/ai/starlake/job/transform/TransformContext.scala)
  • Duplicated block (12 lines × 2) (src/main/scala/ai/starlake/job/metrics/ExpectationAssertionHandler.scala)
  • Duplicated block (12–13 lines × 3) (src/main/scala/ai/starlake/job/ingest/loaders/DuckDbNativeLoader.scala)
  • Duplicated block (22–24 lines × 2) (src/main/scala/ai/starlake/schema/handlers/InferSchemaHandler.scala)
  • Duplicated block (6 lines × 2) (src/main/scala/ai/starlake/job/transform/JdbcAutoTask.scala)
  • Edited copy of a member (20 corresponding lines) (src/main/scala/ai/starlake/schema/model/SchemaInfo.scala)
  • ExtractBigQuerySchema.attributeOf (cognitive 17) (src/main/scala/ai/starlake/extract/ExtractBigQuerySchema.scala)
  • ExtractBigQuerySchema.attributeOf (cyclomatic 28) (src/main/scala/ai/starlake/extract/ExtractBigQuerySchema.scala)
  • FileTooLong: bigquery/BigQueryNativeJob.scala (src/main/scala/ai/starlake/job/sink/bigquery/BigQueryNativeJob.scala)
  • FileTooLong: loaders/BigQueryNativeLoader.scala (src/main/scala/ai/starlake/job/ingest/loaders/BigQueryNativeLoader.scala)
  • …and 31 more

Changes since last survey

  • 47 commits — 34 feature/other, 13 fixes

By area

  • (root) — 17 commits
  • src/main — 10 commits
  • distrib/setup.jar — 8 commits
  • src/test — 6 commits
  • .github/workflows — 3 commits
  • (repo) — 1 commit
  • project/Versions.scala — 1 commit
  • scripts/local-release.sh — 1 commit

Notable commits

  • fix: Port branch-1.7 to master: v1.7.8 fixes (#1877, #1878, #1879, #1880) (#1882)
  • fix: Port branch-1.7 to master: v1.7.9 fixes (#1883, #1884, #1885) (#1914)
  • fix: fix(bq-info): leave out what is deleted while it scans the project (#1923)
  • fix: fix(launcher): a proxy in the environment stops starlake.cmd on Windows (#1918)
  • fix: fix(launcher): keep the proxy password out of starlake.sh and setup.sh (#1919)
  • fix: fix(launcher): use the AppCDS archive on JDK 21 to 24 (#1905)
  • fix: fix(release): rebuild setup.jar before tagging (#1922)
  • fix: fix(security): keep access tokens and other credentials out of the logs (#1911)
  • fix: fix(setup): look up the latest release through the proxy (#1904)
  • fix: fix(transform): make --parallelism reliable and faster on DuckDB (#1925)
  • fix: fix(transform): make the transform paths safe to run concurrently (#1785) (#1907)
  • fix: fix: a task no longer reads a stale SL_THIS, and a lost audit row is reported (#1916)
  • fix: test(launcher): fix two launcher tests on macOS (#1903)
  • change: Feature/appcds startup (#1901)
  • change: Merge pull request #1860 from starlake-ai/build/duckdb-2.0
  • change: Port branch-1.7 to master: everything up to v1.7.7 (#1854)
  • change: Setting version to 1.8.14
  • change: Setting version to 1.8.15
  • change: Setting version to 1.8.15-SNAPSHOT
  • change: Setting version to 1.8.16
  • …and 27 more

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

starlake-ai/starflow was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 8ee45270700e3f3e0ddfa7a5a68f854f3274cf12 — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-5f8314471808.