Skip to content
CAI
Software that uses CAICheck a score

DataSQRL/sqrl

55.4

Adequate · 22 October 2026

62.8k

lines of production code

Java

primary language

7

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

This system is a data engineering framework that compiles high-level SQL-like scripts into executable pipelines for Apache Flink and various database engines. It provides a CLI for project scaffolding, compilation, and testing, while managing the generation of deployment artifacts for streaming, storage, and API servers. The platform supports multiple execution targets including PostgreSQL, DuckDB, Snowflake, and Kafka, and includes a GraphQL server for data access and API generation.

How it got here

54 changes.

The project removed extensive legacy components, including the custom SQL parser, GraphQL server, and various database-specific modules, to streamline the codebase. This cleanup paved the way for a major refactoring of the planner to support Flink as the primary streaming engine, alongside the introduction of new capabilities for DuckDB, Kafka, and Iceberg.

2025 — planner architecture and dialect expansion

96 changes.

The project overhauled the core planner infrastructure by introducing a DAG-based execution model, cost-based optimization, and a modular SQL dialect system supporting PostgreSQL, Snowflake, Trino, and Spark. This architectural shift was accompanied by extensive CLI refactoring using Spring Boot, enhanced schema discovery, and a comprehensive expansion of integration tests to validate the new capabilities.

2026 — GraphQL server and planner expansion

52 changes.

This period focused on introducing a new GraphQL server module with schema validation, pagination, and observability, while significantly expanding the planner's support for PostgreSQL extensions, DuckDB translations, and dialect-specific DDL generation. The work was heavily supported by comprehensive integration and unit tests covering complex mutations, Kafka retention, and various database dialects to ensure reliability.

Features

Add Iceberg data type mapping for JSON and Vector types

Introduces a new IcebergDataTypeMapper that handles specific Flink data types when writing to Iceberg. It explicitly maps Flink's JSON type to a string-only conversion and Vector types to a double-only conversion, while falling back to a byte conversion for other unsupported types.

sqrl-planner/src/main/java/com/datasqrl/datatype/flink/iceberg · high confidence

Add Print export engine for debugging output

A new Print export engine has been introduced, allowing users to configure an export target that prints data to standard output. This feature is implemented via the new PrintEngine class and its corresponding PrintEngineFactory, which registers the 'print' engine type within the export pipeline. The engine integrates with the existing connector configuration system to allow basic customization of the output format.

sqrl-planner/src/main/java/com/datasqrl/engine/export · high confidence

Add initial Postgres log engine implementation

This change introduces the \postgres\_log\ execution engine, providing the core classes (\PostgresLogEngine\, \PostgresLogEngineFactory\, \PostgresLogPhysicalPlan\) and a data type mapper for PostgreSQL as a log destination. The engine is registered via \AutoService\ and includes a \ListenNotifyAssets\ structure for handling database notifications, though the physical planning and table creation methods are currently marked as unsupported, indicating this is a foundational addition rather than a fully functional feature.

sqrl-planner/src/main/java/com/datasqrl/engine/log/postgres · high confidence

The documentation site now includes the Apache Flink logo as a static SVG image in the blog assets folder, and a \.nojekyll\ file has been added to the static root to ensure the hosting platform does not process the documentation with Jekyll, preserving the intended static file structure.

documentation/static · high confidence

Added Java UDF templates for scalar and aggregate functions

The CLI now includes new Java template files for user-defined functions, enabling users to create custom scalar and aggregate functions via the \add-func\ command. The scalar template provides a basic \ScalarFunction\ implementation that repeats a string, while the aggregate template offers a time-weighted average \AggregateFunction\ using Flink's API, allowing users to generate these specific function types directly from the command line.

sqrl-cli/src/main/resources/templates/functions · high confidence

Added PostgreSQL JSON function translations

The planner now supports translating a set of internal JSON functions to their PostgreSQL equivalents. This includes aggregations like \jsonb\_agg\ (via \JsonArrayAggSqlTranslation\), construction functions like \jsonb\_build\_array\ and \jsonb\_build\_object\ (via \JsonArraySqlTranslation\ and \JsonObjectSqlTranslation\), and extraction/querying functions like \jsonb\_path\_query\_first\ (via \JsonExtractTranslation\). Additionally, it handles JSON concatenation (\\|\|\), existence checks (\jsonb\_path\_exists\), string conversion (\\#\>\>\), and the casting of typed ROW values to JSONB objects, ensuring these operations work correctly when targeting a PostgreSQL dialect.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/json · high confidence

Added Postgres vector function translations and metadata

The planner now supports translation and metadata for several Postgres vector operations. New SQL translation classes handle the \center\, \cosine\_distance\, \cosine\_similarity\, and \euclidean\_distance\ functions, ensuring they are correctly unparsed for the Postgres backend. Additionally, metadata classes define the indexability and operand constraints for \text\_search\, \cosine\_similarity\, and \euclidean\_distance\, enabling the planner to optimize queries involving these vector and text search functions.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/vector · high confidence

Added schema compatibility checking and Calcite type utilities

The planner now includes a new TypeCompatibility utility that validates whether a new data schema can safely read data produced by an older schema, ensuring backwards compatibility by checking nullability, structural recursion for rows/arrays/maps, and precision constraints. Additionally, a new TypeFactory component extends Flink's type system to provide standardized methods for creating specific Calcite types (such as timestamps, UUIDs, and integers) and mapping them to Java classes, supporting the underlying type resolution logic.

sqrl-planner/src/main/java/com/datasqrl/calcite/type · high confidence

Automated documentation generation and versioning scripts

Added build scripts to automate the documentation pipeline: \prepare-versioned-docs.sh\ snapshots release branches into versioned docs (labeling pre-1.0 versions by minor version), \checkout-stdlib-docs.sh\ checks out the Flink SQL Runner's stdlib-docs submodule at the matching runner version, \generate-function-docs.js\ creates system and library function reference pages from YAML sources, and \generate-llms-txt.mjs\ produces agent-friendly \llms.txt\ and \llms-full.txt\ bundles for each docs version.

documentation/scripts · high confidence

Automatic index optimization and pagination performance improvements

The planner now automatically analyzes query patterns to generate optimal database indexes for JDBC tables, supporting various types such as BTREE, HASH, and partitioned indexes. This includes a specific optimization for paginated queries that automatically adds BTREE indexes on rowtime columns to speed up offset-based pagination. Additionally, the DAG visualization now includes row count estimations for tables to provide better insight into data volume.

sqrl-planner/src/main/java/com/datasqrl/plan/global · high confidence

CI now validates documentation commands and image references

New scripts in the documentation test suite ensure that the product documentation remains accurate and usable. The \run\_doc\_commands.py\ script now executes shell commands marked with the \docs-test\ attribute verbatim in a simulated terminal, verifying that the steps work as written and allowing image references to be swapped for testing (e.g., using pull requests). Additionally, the \check-doc-images.sh\ script scans documentation files to verify that all referenced DataSQRL Docker images are publicly accessible, preventing broken links for new users.

documentation/scripts/docs-test, python · high confidence

Configurable DuckDB memory limits and enhanced JSON utilities

The server now supports setting a configurable memory limit for DuckDB via the \memory-limit\ configuration option, allowing users to control resource usage during initialization. Additionally, new utility classes \DuckDbInitializer\ and \JsonUtils\ have been added to the server base module; \DuckDbInitializer\ handles the construction of initialization SQL including extension loading and memory settings, while \JsonUtils\ provides a pre-configured Jackson \ObjectMapper\ with environment variable resolution support and a recursive merge function for JSON nodes.

sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/util · high confidence

Containerized Vert.x server with Prometheus metrics and structured logging

The Vert.x server is now packaged as a container image that launches via a new SqrlLauncher entry point, which automatically registers a Prometheus metrics registry for observability. The container includes a dedicated entrypoint script that supports debug mode via the SQRL\_DEBUG environment variable and writes application logs to the console. Logging is configured with two separate Log4j2 profiles: a standard log4j2.properties for general INFO-level output and a log4j2-debug.properties that enables DEBUG-level logging for the DetailedRequestTracer, routing those detailed request traces to a rotating file at /opt/sqrl/logs/request-trace.log.

sqrl-server/sqrl-server-vertx · high confidence

Expanded DuckDB function translations for arrays, strings, and dates

The planner now includes a comprehensive set of new SQL translation implementations for the DuckDB dialect, covering array operations (ARRAY\_JOIN, ARRAY\_PREPEND, ARRAY\_REMOVE, ELEMENT, ELT), string manipulation (DATE\_FORMAT, LOCATE, ENDS\_WITH, IS\_ALPHA, IS\_DECIMAL, IS\_DIGIT, JSON\_OBJECT, DECODE, ENCODE), and utility functions (CONVERT\_TZ, E). These additions ensure that Flink SQL expressions using these functions are correctly translated into valid DuckDB syntax, improving compatibility and query execution reliability.

sqrl-planner/src/main/java/com/datasqrl/function/translation/duckdb · high confidence

The planner now supports a broader set of Flink built-in functions when targeting PostgreSQL, translating operations such as array manipulation (ARRAY\_CONCAT, ARRAY\_CONTAINS, ARRAY\_JOIN, ARRAY\_POSITION, ARRAY\_PREPEND, ARRAY\_REMOVE), string and binary encoding (BIN, DECODE, ENCODE, FROM\_BASE64, HEX), date/time handling (CONVERT\_TZ, CURRENT\_TIME, CURRENT\_TIMESTAMP, DAY\_OF\_WEEK), and utility functions (COLLECT, ELT, ELEMENT, IFNULL, ENDS\_WITH, E). This ensures these functions execute correctly in PostgreSQL-compatible environments.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/builtinflink · high confidence

Initial Snowflake JSON function translation mappings added

Added new SQL translation classes for Snowflake to map internal JSON functions (such as JSON\_ARRAYAGG, JSON\_OBJECT, JSON\_EXTRACT, and TO\_JSON) to their Snowflake equivalents (like ARRAY\_AGG, OBJECT\_CONSTRUCT, and PARSE\_JSON). These translation files are currently disabled (not registered via AutoService) and some functions explicitly throw runtime errors, indicating this is a preparatory step for Snowflake JSON support.

sqrl-planner/src/main/java/com/datasqrl/function/translation/snowflake · high confidence

Introduce DuckDB as a standalone query engine with Iceberg scan optimization

Users can now configure DuckDB as a dedicated query engine to execute SQL queries against Iceberg tables. This change adds a new \DuckDBEngine\ and its factory, along with a \DuckDbStatementFactory\ that implements cost-based optimization for Iceberg scans. The planner automatically materializes repeated scans of large Iceberg tables as Common Table Expressions (CTEs) to improve query performance, while handling Iceberg-specific path resolution and catalog configurations.

sqrl-planner/src/main/java/com/datasqrl/engine/database/relational · high confidence

Introduce FlexibleTableConverter for schema traversal

Added the FlexibleTableConverter class in the flexible schema package, which implements a visitor pattern to traverse and process flexible table schemas. This component handles the conversion of field types, including support for nested relations, array depths, and nullability constraints, enabling the planner to interpret complex schema structures during query planning.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible · high confidence

The streaming engine location now provides a concrete Flink implementation, replacing the previous abstract or placeholder structure. This includes the FlinkStreamEngine for runtime configuration and checkpointing, a FlinkEngineFactory registering the engine as type PROCESS, and a suite of SQL planning utilities (FlinkCalciteParser, FlinkSqlNodePlanner, FlinkDialect) to handle SQL parsing, validation, and relational-to-SQL conversion. The package also introduces Calcite optimization rules (ShapeBushyCorrelateJoinRule, ToStubAggRule) to improve query planning and execution for Flink sinks.

sqrl-planner/src/main/java/com/datasqrl/engine/stream · high confidence

Introduce Kafka as a configurable log engine

The planner now supports Kafka as a dedicated log engine, allowing users to configure topic creation parameters such as the number of partitions, replication factor, and retention TTL. This implementation handles both stream and mutation tables, automatically applying upsert formats for state tables with primary keys and supporting configurable watermarks for source and transactional processing.

sqrl-planner/src/main/java/com/datasqrl/engine/log/kafka · high confidence

Introduce extensible DataSQRL coding agent with Pi integration and user-facing Docker commands

The agent area now provides a baseline interactive coding agent that combines the DataSQRL CLI with the Pi coding agent framework. The agent image installs Pi (bumped to 1.4.0) and bundles DataSQRL-specific instructions (AGENTS.md), CLI reference documentation (CLI\_REFERENCE.md), and on-demand skills (skills/) to guide the model through project scaffolding, connector configuration, pipeline implementation, and API design. To improve usability, the agent now instructs users to run equivalent \docker run\ commands using the \datasqrl/cmd\ image rather than the internal \/opt/agent/cmd.sh\ wrapper, and the entrypoint automatically selects the provider and default model based on available API keys or explicit environment variables (PI\_PROVIDER, PI\_MODEL). The build process also generates a bundled \llms-full.txt\ documentation file for the agent's reference.

agent · high confidence

Introduce new planner utility classes and interfaces

The \sqrl-planner\ module adds a suite of new utility classes and interfaces to support configuration loading, file handling, and service discovery. This includes \MainScript\ and \MainScriptImpl\ for resolving and reading the primary SQRL script and associated mutation database definitions, \ConfigLoaderUtils\ for loading and validating \package.json\ configurations against JSON schemas, and \ServiceLoaderDiscovery\ for caching and retrieving service implementations. Additional utilities such as \AbstractDAG\ for directed acyclic graph operations, \FileUtil\ and \FileCompression\ for file system and compression handling, and \CalciteUtil\ for SQL planner type manipulations are also introduced to centralize common logic.

sqrl-planner/src/main/java/com/datasqrl/util · high confidence

Introduce new statistics and schema generation components in sqrl-discovery

The sqrl-discovery module now includes a new set of classes for collecting and merging data statistics and generating schemas. This includes the Accumulator interface and its implementations (SourceTableStatistics, RelationStats, FieldStats, FieldTypeStats) for tracking data metrics, a LogarithmicHistogram for cardinality estimation, and the SchemaGenerator interface with its DefaultSchemaGenerator implementation for merging table definitions with observed statistics. Additionally, the Metric interface has been moved from sqml-core to this package to support these new statistical operations.

sqrl-discovery/src/main/java/com/datasqrl/discovery/stats · high confidence

Introduces cost-based query planning for engine selection

The planner now includes a cost analysis module that evaluates the relative expense of executing operations on different engine types (Database, Process, Server, Log). This new capability allows the system to automatically select the most efficient execution strategy by weighing factors such as join complexity, table types (Stream, State, Versioned, Static, Lookup, Relation), and engine-specific penalties, thereby optimizing query performance without manual configuration.

sqrl-planner/src/main/java/com/datasqrl/planner/analyzer/cost · high confidence

A new FlexibleJsonFlinkFormatTypeMapper is added to handle data type conversions within the Flink JSON format. This mapper defines specific rules for mapping SQL types to JSON-compatible formats, including handling of arrays, rows, and vector types, ensuring that complex data structures are correctly serialized or converted to bytes when necessary.

sqrl-planner/src/main/java/com/datasqrl/datatype/flink/json · high confidence

Introduction of TableType enum for table classification

A new TableType enum has been added to define and classify table behaviors within the planner, including types such as STREAM, VERSIONED\_STATE, STATE, LOOKUP, RELATION, and STATIC. This enum provides methods to determine specific table properties like whether they have a timestamp, primary key, or support temporal joins, and defines logic for combining different table types.

sqrl-planner/src/main/java/com/datasqrl/io/tables · high confidence

New CLI resources for project initialization and visualization

The CLI now includes support for bootstrapping new projects and visualizing data structures. An \init-project.properties\ file defines default engine configurations (Flink, Kafka, Iceberg, etc.) for stream, dataset, and API components, enabling the \init\ command to scaffold projects with sensible defaults. Additionally, two new HTML resources, \visualize\_dag.html\ and \data\_model\_visual.html\, provide browser-based visualization for deployment graphs and data models respectively, leveraging the \@datasqrl/dag-visualization\ library to render these views in the CLI's Voyager interface.

sqrl-cli/src/main/resources · high confidence

New CLI utility classes and Spring-based dependency injection

The CLI now includes a suite of new utility classes in the \com.datasqrl.util\ package to support core operations: \DirectoryUtils\ for managing target directories, \FileHash\ for computing MD5 checksums, \FlinkOperatorStatusChecker\ for monitoring Flink job states via REST, \JBangRunner\ for building and repairing fat JARs for UDFs, \OsProcessManager\ for dynamically starting dependent services (PostgreSQL, Redpanda) and managing file ownership, \ResourceUtils\ for loading resources from both JARs and the file system, \ResultSetPrinter\ for formatting SQL results, and \SqrlInjector\ which configures Spring-based dependency injection for these components and the broader compilation pipeline.

sqrl-cli/src/main/java/com/datasqrl/util · high confidence

New ExecutionGoal enum for validation phases

A new ExecutionGoal enum has been added to the validation package, defining three distinct phases: COMPILE, RUN, and TEST. This change introduces a structured way to categorize execution goals within the planner's validation logic, likely supporting more granular control over how plans are processed and verified.

sqrl-planner/src/main/java/com/datasqrl/plan/validate · high confidence

The planner now includes a new \DataTypeMapping\ interface and \DataTypeMappings\ registry to handle conversions between Flink types and database engine types. This change introduces specific mappers for JSON (using \jsonb\_to\_string\ and \to\_jsonb\), vector data (using \vector\_to\_double\ and \double\_to\_vector\), and byte serialization (\serialize\_to\_bytes\), enabling more precise type handling for these data structures during query execution.

sqrl-planner/src/main/java/com/datasqrl/datatype · high confidence

New GraphQL schema-to-API converter for function backends

The server now includes a new \GraphQLSchemaConverter\ component that transforms GraphQL operation definitions into executable API functions. This converter extracts queries and mutations from a GraphQL schema and generates corresponding \ApiOperation\ definitions, allowing the system to automatically expose GraphQL operations as backend functions. The conversion process is configurable via \GraphQLSchemaConverterConfig\, which allows users to control operation filtering (e.g., hiding operations by prefix), set maximum conversion depth, manage naming prefixes, and define supported API protocols (REST, MCP).

sqrl-planner/src/main/java/com/datasqrl/server/converter · high confidence

New GraphQL server module with schema validation and paginated query support

This change introduces the core server module for handling GraphQL APIs, including schema generation, validation, and model management. Users can now define custom GraphQL schemas that are validated against the underlying data model, with specific checks ensuring mutation inputs match their target tables. The module also adds opt-in support for paginated queries using an \OffsetPageInfo\ metadata wrapper, allowing clients to retrieve pagination details like total records and page counts alongside results.

sqrl-planner/src/main/java/com/datasqrl/server · high confidence

New Jackson serialization module with automatic plugin discovery

The planner now includes a new serialization infrastructure in the \com.datasqrl.serializer\ package. A central \Deserializer\ class provides unified JSON and YAML mapping capabilities using Jackson, configured with specific modules for Java 8, Java Time, and Vert.x. The system introduces a \SqrlSerializerModule\ that automatically discovers and registers custom serializers and deserializers via Java's \ServiceLoader\. This enables extensible serialization, demonstrated by the inclusion of \NewTopicSerializer\ and \NewTopicDeserializer\ for Kafka topic configurations, and a generic \JacksonDeserializer\ for polymorphic type resolution based on a type key.

sqrl-planner/src/main/java/com/datasqrl/serializer · high confidence

New and improved planner hints for caching, indexing, and documentation

The planner now supports several new hints to fine-tune query execution and data management. Users can define caching behavior with the \cache\ hint (specifying a TTL duration) and manage data retention with the \ttl\ hint. Indexing is more flexible, allowing explicit \index\ hints with specified types and sort orders (ASC/DESC), as well as \vector\_dim\ hints for vector column indexing. Table access patterns are controlled via \query\_by\_all\, \query\_by\_any\, and \no\_query\ hints. Additional hints include \engine\ (replacing the legacy \exec\ hint) for DAG stage assignment, \row\_count\ for cardinality estimation, \mutation\_insert\ for specifying mutation types, \maintenance\ for regular maintenance flags, and \workload\ to mark tables as query-only sources. The system also introduces \DocStringParser\ to extract column and argument documentation from Rust-style markdown comments, and \PartitionKeyHint\ to assign partition keys. Unknown hints now generate warnings instead of fatal errors, improving robustness during planning.

sqrl-planner/src/main/java/com/datasqrl/planner/hint · high confidence

Added seven new documentation/blog posts detailing the DataSQRL 0.6 and 0.7 releases, Flink SQL extensions for defining data interfaces, the Flink SQL Runner toolkit, and the conceptual framework for data platform automation. These posts explain how DataSQRL compiles SQRL scripts into data pipelines, supports temporal joins, and provides a world model for AI coding agents.

documentation/blog · high confidence

New documentation site components for versioning and homepage layout

This change introduces new React components for the documentation site: a full-width version banner that appears below the navbar for non-latest releases, a version picker moved to the docs sidebar, and new homepage components (header and features) to structure the landing page content.

documentation/src/components · high confidence

New flexible schema type converters for Calcite integration

Added two new converter classes, FlexibleTable2RelDataTypeConverter and SqrlTypeRelDataTypeConverter, within the flexible schema converters package. These components enable the mapping between the system's internal flexible schema types (such as Boolean, BigInt, String, Timestamp, Interval, and Array) and Apache Calcite's RelDataType structures, allowing the planner to correctly interpret and process flexible schema definitions during query execution.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/converters · high confidence

New flexible-JSON record reader with ISO-8601 timestamp default

The file discovery module now includes a new JsonRecordReader implementation that supports the 'flexible-json' format for .jsonl files. This reader automatically configures ISO-8601 as the default timestamp format, ensuring consistent date parsing for users ingesting JSON Line data without needing to manually specify timestamp settings.

sqrl-discovery/src/main/java/com/datasqrl/discovery/file · high confidence

New internal function metadata and PostgreSQL operator support

This change introduces new internal infrastructure for handling function metadata and adds support for specific PostgreSQL operators. It adds a new \FunctionMetadata\ interface (replacing a previously renamed exception file) and several new interfaces (\IndexableFunction\, \InputPreservingFunction\) to classify functions for optimization and primary-key/timestamp determination. It also introduces \PgSpecificOperatorTable\, which defines SQL operators for PostgreSQL full-text search (e.g., \to\_tsvector\, \@@\) and distance metrics (cosine, Euclidean), along with utility classes (\CalciteFunctionUtil\, \FlinkUdfNsObject\) to support these function definitions within the planner.

sqrl-planner/src/main/java/com/datasqrl/function · high confidence

New observability, tracing, and pagination capabilities in the server base

This update introduces several new capabilities to the server base module. For observability, it adds GraphQL query metrics via \GraphQLQueryMetricsInstrumentation\ (recording latency to Micrometer) and tail-sampling tracing via \GraphQLTailSampleTracingInstrumentation\ (logging structured warnings for slow or failed queries). HTTP-level visibility is improved with \DetailedRequestTracer\ (logging full request/response bodies and headers) and \ForwardedClientAddressLogFormatter\ (logging the resolved client IP in access logs). For data access, the server now supports an opt-in \OFFSET\_PAGE\_INFO\ pagination type, allowing GraphQL queries to return page metadata (such as \hasNextPage\, \totalRecords\, and event times) alongside results. Additionally, the module adds an \McpBridgeVerticle\ to expose GraphQL operations as MCP tools and resources, and introduces \FlinkFunctionExecutor\ to execute Flink-based functions.

sqrl-server-vertx-base · high confidence

New planner utility classes for index mapping and primary key management

The planner now includes new utility classes in the \com.datasqrl.plan.util\ package to handle column index mapping and primary key tracking. \IndexMap\ and its implementation \SelectIndexMap\ provide a functional interface for remapping column indices, including a \RexIndexMapShuttle\ to update Calcite \RexNode\ references during plan transformations. \PrimaryKeyMap\ manages primary key column sets, supporting operations to check coverage, simplify keys, and remap indices. Additionally, \RelWriterWithHints\ extends Calcite's \RelWriterImpl\ to include query hints in the relational node explanation output, improving debuggability.

sqrl-planner/src/main/java/com/datasqrl/plan/util · high confidence

New schema factories for CSV and JSONL discovery

The preprocessor now includes dedicated schema factories for CSV and JSONL files. These new components implement the TableSchemaFactory interface to automatically detect file extensions, read records, compute statistics, and infer schemas for these specific formats, enabling the system to process CSV and JSONL data sources during discovery.

sqrl-discovery/src/main/java/com/datasqrl/discovery/preprocessor · high confidence

Postgres text\_search function translation to GIN-compatible expressions

Added a new translation rule for the Postgres dialect that maps the \text\_search\ function to PostgreSQL's native \@@\ match operator and \ts\_rank\_cd\ ranking. This implementation constructs \to\_tsvector\ and \to\_tsquery\ expressions that are explicitly designed to match the format of GIN indexes, ensuring that Postgres can effectively utilize existing full-text search indexes for these queries.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/text · high confidence

Repository restructured with new documentation and agent support files

The repository root has been restructured to support the new agentic data engineering workflow and improved documentation. New files have been added to guide contributors and AI agents: AGENTS.md provides specific instructions for coding agents like Claude Code and Codex; CODE\_OF\_CONDUCT.md and CONTRIBUTING.md establish community guidelines and contribution workflows; SECURITY.md defines the vulnerability reporting process; and a .claude directory is now ignored. The project's legal and operational documentation has been updated with a new LICENSE (Apache 2.0), README.md, and RELEASE.md. Additionally, the sqml-examples submodule has been removed and replaced with a documentation submodule pointing to the flink-sql-runner repository, and .gitignore has been updated to exclude agent helper scripts and generated documentation artifacts.

(repo-wide) · high confidence

Support bare relation aliases in SELECT lists

The planner now allows selecting a table alias directly in the SELECT clause (e.g., \SELECT p FROM Projects p\). Instead of requiring explicit column listing, the system automatically expands the bare alias into a nested ROW value containing all columns of the referenced relation, enabling users to treat an entire row as a single field in the result set.

sqrl-planner/src/main/java/com/datasqrl/calcite/expand · high confidence

Support for pg\_partman partitioning and pgvector extension

The planner now automatically generates SQL to configure the pg\_partman extension for tables using RANGE partitioning with a Time-To-Live (TTL), including configurable premake counts and historical start partitions. Additionally, it now supports the pgvector extension, enabling vector data types and associated operators for similarity search capabilities.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/extensions · high confidence

Removals

Removal of Postgres-specific migration and optimization components

The Postgres-specific implementation details have been removed from the sqml-postgres module. This includes the deletion of the PostgresResult class and its associated MigrationObject structure, the PostgresFunctions registry, the PostgresSqmlMigration executor, the ShreddingSqlOptimizer, the PostgresStatisticsProvider, and the commented-out Rewriter logic. Users relying on this module for Postgres schema migration or Postgres-specific SQL optimization will no longer have access to these capabilities in this location.

sqml-postgres · high confidence

Removal of incomplete GraphQL schema builder components

The \GraphqlSchemaBuilder\ and \SqlGraphqlSchema\ classes in the \sqml-graphql\ module have been removed. These files contained stub implementations for converting SQL logical plans into GraphQL schemas, including a visitor pattern for traversal and basic type mapping logic that was not fully functional. This cleanup eliminates unused code from the codebase.

sqml-graphql/src/main/java · high confidence

Removal of legacy HTTP source ingress

The \HttpIngress\ class, which previously allowed users to ingest data streams from HTTP URLs via a buffered reader and asynchronous timeout handling, has been removed from the \sqml-ingress-http\ module. This change eliminates the legacy URL-based source capability from the HTTP ingress component.

sqml-ingress-http · high confidence

Removal of legacy Main.java entry point

The legacy \Main.java\ entry point in \sqml-main/src/main/java\ has been removed. This file previously contained the hardcoded application startup logic, including the definition of a \meetup\ data source, schema configuration, Postgres migration setup, and the initialization of the GraphQL servlet on port 8080. Its deletion indicates a shift away from this specific monolithic startup pattern, likely as part of the broader refactoring and integration of Flink into the pipeline.

sqml-main/src/main/java · high confidence

Removal of legacy SQL grammar definition

The legacy SQL grammar file (SqlBase.g4) has been deleted from the parser module. This change removes the previously defined syntax rules for statements, queries, joins, and expressions, indicating that the parser implementation has been replaced or significantly restructured in other parts of the codebase.

sqml-parser/src/main/antlr4 · high confidence

Removal of legacy SQL query analysis and model components

The \sqml-core\ module has removed a significant set of internal classes responsible for SQL query analysis and model representation, including \MetadataManager\, \Model\, \ModelRelation\, \QueryUtil\, \Session\, \StubModel\, \StubModels\, and the entire \analyzer\ package (containing \Analyzer\, \Analysis\, \AggregationAnalyzer\, \ExpressionAnalyzer\, and related utilities). This change eliminates the legacy code paths for parsing, analyzing, and managing SQL query plans and metadata within this core library.

sqml-core · high confidence

Removal of legacy SQML API components

The \sqml-api\ module has removed a large set of legacy classes, including the analyzer, optimizer, execution strategies, schema definitions, and vertex infrastructure. This cleanup eliminates unused code and simplifies the API surface by discarding the previous implementation of script analysis, query optimization, and local execution logic.

sqml-api · high confidence

Removal of local script registry implementation

The LocalScriptRegistry class, which previously allowed users to register and retrieve scripts from a local in-memory map, has been removed from the codebase. This change eliminates the ability to use this specific local-only script storage mechanism.

sqml-script-registry-local · high confidence

Removal of presto-matching library and sqml-server components

This change removes the \presto-matching\ library entirely, deleting core classes such as \Pattern\, \Matcher\, \Match\, and \Captures\ along with their associated pattern implementations (e.g., \TypeOfPattern\, \WithPattern\). It also removes the \sqml-server\ module, deleting the GraphQL endpoint implementation (\GraphqlEndpoint\), server orchestration (\SqmlServer\), and various management and handler classes (e.g., \ScriptManager\, \SchemaManager\). Additionally, the \sqml-metadata\ module is stripped of its \ColumnType\, \Identifier\, and \MetadataStore\ definitions.

presto-matching, sqml-metadata, sqml-server · high confidence

Removal of unused PostgresViewVertexFactory class

The unused PostgresViewVertexFactory class has been removed from the sqml-postgres-view module. This cleanup eliminates dead code that previously provided a factory method for creating SqlVertexFactory instances, simplifying the codebase without affecting active functionality.

sqml-postgres-view · high confidence

Removed GraphqlServlet stub

The GraphqlServlet class, which served as a stub for a GraphQL endpoint in the sqml-graphql-jetty module, has been removed. This eliminates the unused servlet infrastructure that previously exposed a builder pattern for configuring the GraphQL runtime and schema.

sqml-graphql-jetty · high confidence

Removed meetup.sqml resource file

The sample resource file meetup.sqml has been removed from the application resources. This file previously defined a data source import for 'meetup' and contained specific field calculations for Rsvp records, including a derived id\_mult field and a sum\_id aggregation.

sqml-main/src/main/resources · high confidence

SQML parser implementation removed

The entire \sqml-parser\ implementation has been deleted, including the \SqlParser\, \SqmlParser\, \AstBuilder\, and all associated support classes (e.g., \ErrorHandler\, \ParsingOptions\). This removes the capability to parse and analyze SQML scripts and SQL statements within this module.

sqml-parser/src/main/java/ai/dataeng/sqml/parser · high confidence

SQML parser tree model removed

The entire AST node hierarchy and visitor infrastructure in the \sqml-parser\ tree package has been deleted. This includes all node classes (such as \AliasedRelation\, \ArithmeticBinaryExpression\, \Cast\, and \Query\), the \AstVisitor\ interface, and the default traversal implementations. This change removes the parser's internal representation of SQL queries and expressions, likely as part of a migration to a different parsing or analysis engine.

sqml-parser/src/main/java/ai/dataeng/sqml/tree · high confidence

Architecture

Extract deployment file model classes to sqrl-deployment-model

The deployment file model classes (FlinkPlanModel, JdbcPlanModel, KafkaPlanModel, etc.) have been extracted into the sqrl-deployment-model module. This refactoring centralizes the data structures representing deployment plans (such as Flink SQL plans, JDBC statements, and Kafka topics) into a dedicated module, improving modularity and separation of concerns for deployment configuration.

sqrl-deployment-model · high confidence

New GraphQL AST visitor interfaces for schema traversal

The server module now includes a comprehensive set of visitor interfaces (such as GraphqlDefinitionVisitor, GraphqlSchemaVisitor, and GraphqlSelectionVisitor) that enable structured traversal of GraphQL language nodes. This refactoring replaces previous internal optimizer and plan-replacement utilities with a standardized visitor pattern, allowing the planner to inspect and process GraphQL schema definitions, arguments, directives, and selections in a more flexible and decoupled manner.

sqrl-planner/src/main/java/com/datasqrl/server/visitor · high confidence

Refactor SQL statement interfaces and introduce database extension points

The SQL planning layer has been restructured to support database-specific extensions and cleaner statement definitions. Two new service-loader interfaces, DatabaseTableExtension and DatabaseTypeExtension, have been added to allow plugins to inject custom DDL and manage database-specific type operators. Additionally, the SqlFunction interface has been renamed and relocated to SqlDDLStatement within the new package, simplifying its contract to a single getSql method to better reflect its role in generating DDL statements.

sqrl-planner/src/main/java/com/datasqrl/sql · high confidence

Refactor planner analyzer into dedicated analysis classes

The planner's analysis logic in the \analyzer\ package has been restructured into a set of focused classes: \AbstractAnalysis\ provides common relational node utilities, \CapabilityAnalysis\ tracks required engine features, \RelNodeAnalysis\ holds intermediate state, and \TableAnalysis\ aggregates table metadata. This refactoring consolidates the analysis components previously scattered or managed by \TypeManager\, improving the clarity and maintainability of the planning pipeline.

sqrl-planner/src/main/java/com/datasqrl/planner/analyzer · high confidence

Refactor schema import infrastructure with new factory and conversion abstractions

The schema import mechanism has been refactored to support a more modular and extensible architecture. A new \TableSchemaFactory\ interface introduces a ServiceLoader-based discovery system, allowing schema conversion logic to be registered and retrieved by file extension. This is supported by a new \SqrlTypeConverter\ interface that extends the type visitor pattern, and a \SchemaConversionResult\ record that bundles the converted type with connector options. This change restructures how external schemas are parsed and mapped to the internal SQRL type system, laying the groundwork for more flexible schema handling.

sqrl-planner/src/main/java/com/datasqrl/io/schema · high confidence

Refactor server module and key interfaces for flexibility

The server module and its core interfaces have been refactored to be more flexible and decoupled. This introduces a new thread-safe \GlobalEnvironmentStore\ for centralized environment variable access, standardizes environment variable naming via \EnvVariableNames\, and restructures the GraphQL execution layer with new interfaces like \FunctionExecutor\, \MetadataReader\, and \ServerContext\. Additionally, the \GraphQLEngineBuilder\ and \RootGraphQLModel\ have been updated to support these new abstractions, and custom scalars (such as \DOUBLE\ and \FLEXIBLE\_DATETIME\) are now explicitly registered in the GraphQL wiring.

sqrl-server/sqrl-server-core · high confidence

Behavioural changes

Avro schema parsing now respects legacy timestamp mapping configuration

The Avro schema factory and converter now explicitly handle the Avro legacy timestamp mapping setting. When processing Avro schemas, the system checks for a specific configuration property (including prefixed variants) and automatically enables legacy timestamp mapping if the 'avro-confluent' format is detected. This ensures that timestamp fields are converted correctly according to the expected Avro format, preventing potential data type mismatches during schema inference.

sqrl-planner/src/main/java/com/datasqrl/io/schema/avro · high confidence

CLI restructured with Spring DI and new project scaffolding commands

The CLI command architecture has been refactored to use Spring Boot for dependency injection, replacing the previous Guice-based setup. This change introduces a new command hierarchy (BaseCmd, BaseOsProcessManagerCmd, AbstractCompileCmd) and adds two new user-facing commands: 'init' to scaffold new SQRL projects from templates, and 'add-func' to generate Java UDF stubs. The refactoring also standardizes error handling via a unified ErrorCollector and improves output formatting for compile and test operations.

sqrl-cli/src/main/java/com/datasqrl/cli · high confidence

Configured IOExceptionHandler for error processing

The application now registers com.datasqrl.error.IOExceptionHandler as the implementation for the com.datasqrl.error.ErrorHandler service interface via the standard Java SPI mechanism. This ensures that I/O-related errors are handled by this specific component.

sqrl-planner/src/main/resources/META-INF · high confidence

Custom PlannerModule for SQRL Docker image classloading

The SQRL \cmd\ Docker image now uses a custom \PlannerModule\ to initialize the Flink SQL planner correctly within a single-JVM process. This change overrides the standard Flink classloading behavior to prevent \ClassNotFoundException\ errors when \flink-table-planner.jar\ is not present on the classpath, ensuring the planner initializes properly in the containerized environment without affecting real Flink SQL runner deployments.

sqrl-planner/src/main/java/org/apache/flink · high confidence

The planner now uses custom Calcite SQL converters to generate Flink-compatible SQL, addressing specific execution requirements. This includes rewriting correlated joins with snapshots into temporal joins, handling UNNEST operations as lateral joins, and preserving join types during conversion. The new converters also inject execution hints into SELECT statements, simplify table references by stripping catalog/database prefixes, and ensure SELECT lists are correctly populated to avoid parser quirks.

sqrl-planner/src/main/java/org/apache/calcite · high confidence

Custom documentation theme with dark mode support

The documentation site now applies a custom visual theme defined in \documentation/src/css/custom.css\. This introduces a specific color palette using blue and purple accents for both light and dark modes, updates the styling for code blocks, and adds specific visual treatments for warning, alert, and success background indicators as well as image alignment.

documentation/src/css · high confidence

Docs site versioning, layout, and agent-friendly content generation

The documentation site now supports versioning by major release (with pre-1.0 minor versions), with a full-width version banner and a version picker in the sidebar. The docs layout uses a fixed sidebar that shrinks above the footer to avoid overlap. Agent-friendly documentation is generated at build time (llms.txt, llms-full.txt, and per-page Markdown) from the sidebar order. Additionally, documented commands can be tested verbatim in CI via docs-test attributes, and retired pages (/flink, /ai) redirect to the homepage.

documentation · high confidence

Expanded SQL dialect support and refactored function translation

The planner now supports generating SQL for Trino, Redshift, and Spark SQL in addition to existing dialects, as indicated by the new entries in the Dialect enum. To facilitate this, the codebase has been refactored to separate function translation into two mechanisms: structural transformations applied at the RelNode level via the new OperatorRuleTransformer and OperatorRuleTransform interface, and simpler name/argument adjustments handled via SqlTranslation during unparsing. Supporting infrastructure includes the DynamicParamSqlPrettyWriter for tracking dynamic parameters and SqrlRexUtil for improved join condition decomposition.

sqrl-planner/src/main/java/com/datasqrl/calcite · high confidence

The planner now automatically resolves Flink INSERT conflict clauses (ON CONFLICT) for upsert sinks by analyzing the optimized plan, reducing the need for manual configuration and preventing non-deterministic results. Additionally, users can now tune the Flink optimizer's predicate pushdown behavior via the new \LIMITED\_RULES\ and \LIMITED\_RULES\_NO\_SOURCE\ configuration options, which strip specific filter and table-source rules to improve subgraph reuse and query performance.

sqrl-planner/src/main/java/com/datasqrl/planner · high confidence

Improved query optimization with accurate row count and join cost estimation

The planner now uses custom metadata handlers to provide more accurate statistics for query optimization. Row count estimates for table scans are derived from table analysis (primary keys and row count hints), and join cost models now include nested loop joins alongside hash joins, ensuring better plan selection for non-equi-join conditions. Selectivity and column uniqueness estimates are also enhanced to leverage primary key information, leading to more efficient query execution plans.

sqrl-planner/src/main/java/com/datasqrl/plan/rules · high confidence

Introduce flexible schema input model with constraint system

The flexible input schema package now defines a new type-safe model for schema definitions, including \FlexibleFieldSchema\, \FlexibleTableSchema\, and \RelationType\, which replace the previous \VarcharType\ and \AbstractType\ classes. This change introduces a new constraint system (\Constraint\, \NotNull\, \Cardinality\, \Unique\) to validate field properties, and adds \FlexibleTypeMatcher\ and \TypeSignatureUtil\ to handle automatic type detection and matching against flexible schema definitions.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/input · high confidence

Introduce generic Java server engine and Vert.x implementation

The server engine module now uses a new \GenericJavaServerEngine\ base class and a \VertxEngineFactory\ to manage server-side execution. This change introduces a Jackson-based configuration system (replacing the previous Vert.x \JsonObject\ approach) that merges default templates with package-specific engine configs to produce a \server-config.json\ deployment artifact. The \ServerPhysicalPlan\ now explicitly tracks server functions, mutations, and the merged \ServerConfig\, while a custom \PrettyPrinter\ ensures the generated configuration is human-readable.

sqrl-planner/src/main/java/com/datasqrl/engine/server · high confidence

Introduces engine capability model and refactored execution interfaces

The engine package now defines a structured capability model via the \EngineFeature\ enum, allowing the planner to verify support for specific operations such as mutations, temporal joins, and partitioning before generating physical plans. This is supported by new interfaces including \EngineConfiguration\ for engine initialization, \ExecutionEngine\ for capability checks, and \EnginePhysicalPlan\ for serializable deployment artifacts. Additionally, legacy interfaces like \FunctionHandle\ and \ParametricType\ have been renamed and repurposed to \ExecutableQuery\ and \IExecutionEngine\ respectively, aligning the internal API with the new execution stage and engine type abstractions.

sqrl-planner/src/main/java/com/datasqrl/engine · high confidence

Introduction of LogEngine and MutationEngine interfaces for mutation support

The planner now exposes new interfaces in the \com.datasqrl.engine.log\ package to handle mutation operations. \LogEngine\ serves as a composite interface extending \ExportEngine\, \MutationEngine\, and \DatabaseEngine\, replacing the previous \TimeWithTimeZoneType\ class which has been removed. \MutationEngine\ defines the contract for creating mutation tables, including support for specifying an insertion type and a Time-To-Live (TTL) duration, enabling the engine to manage data mutations with configurable expiration policies.

sqrl-planner/src/main/java/com/datasqrl/engine/log · high confidence

JVM configuration added for code formatting plugins

A new .mvn/jvm.config file has been introduced to configure the JVM with specific --add-opens and --add-exports flags for the jdk.compiler module. This change enables internal access required by code formatting plugins to operate correctly during the build process.

.mvn · high confidence

New DAG-based planning architecture for SQRL pipelines

The planner now constructs a Directed Acyclic Graph (DAG) of table and function definitions to optimize and assemble physical execution plans. This change introduces a new \DAGBuilder\ to track node dependencies, a \PipelineDAG\ structure to manage execution stages and eliminate inviable paths, and a \DAGPlanner\ that assigns nodes to the most cost-effective stages (such as PROCESS or data store engines) before generating the final physical plan. This replaces the previous linear planning approach with a graph-based optimization layer.

sqrl-planner/src/main/java/com/datasqrl/planner/dag · high confidence

A new FlinkSqrlPostgresDataTypeMapper has been introduced to handle PostgreSQL data types within the Flink JDBC connector. This mapper defines specific conversion rules: standard scalar types (such as integers, decimals, dates, and strings) are passed through without modification, while complex types like MAP, ROW, and ARRAY are converted to JSON. Additionally, any other types that do not match the specific Flink JSON or Vector types are cast to bytes, ensuring consistent handling of diverse PostgreSQL data structures in the planner.

sqrl-planner/src/main/java/com/datasqrl/datatype/flink/jdbc · high confidence

New compilation pipeline with test planning and API artifact generation

The compile module now uses a new \CompilationProcess\ orchestrator that generates OpenAPI JSON deployment artifacts for each API version and writes visual data-model HTML files. It also introduces a test-planning phase that builds a \TestPlan\ containing GraphQL queries, mutations, and subscriptions, allowing the test runner to use a custom GraphQL schema and respect \test(no\_rows)\ hints to skip snapshot generation for empty test queries.

sqrl-cli/src/main/java/com/datasqrl/compile · high confidence

New structured error handling and reporting infrastructure

The \sqrl-planner\ now uses a dedicated error-handling package (\com.datasqrl.error\) to provide structured, user-friendly error messages. This change introduces a system where errors are collected via \ErrorCollector\ and \ErrorCatcher\, which route exceptions through specific \ErrorHandler\ implementations (such as the new \IOExceptionHandler\) to produce consistent \ErrorMessage\ objects with severity levels (FATAL, WARN, NOTICE) and precise file locations. The \ErrorPrinter\ generates pretty-printed output that includes code snippets and context descriptions, while \CollectedException\ ensures stack traces are trimmed to highlight the root cause rather than internal framework noise.

sqrl-planner/src/main/java/com/datasqrl/error · high confidence

New table planning infrastructure and visibility controls

The planner now introduces a dedicated \AccessVisibility\ record to control whether table functions are exposed as queryable endpoints, subscriptions, or kept internal, alongside a \FlinkConnectorConfigWrapper\ that maps Flink connector types (such as \kafka\, \iceberg\, and \jdbc\) to internal table types like \STREAM\ or \VERSIONED\_STATE\. A new \FlinkTableBuilder\ simplifies the construction of Flink table definitions by handling column lists, watermark settings, and partition keys, while \SqrlTableFunction\ and \SqrlFunctionParameter\ provide the core representation for user-defined functions with support for parameters, documentation, and caching durations.

sqrl-planner/src/main/java/com/datasqrl/planner/tables · high confidence

Rationalized SQL dialect conversion and added new shallow query engine support

The SQL conversion logic in the planner has been refactored to simplify the addition of new dialects, introducing a unified \AbstractSqlConverters\ base class and a \SqlConvertersFactory\ that discovers implementations via \@AutoService\. This structural change enables the addition of several new shallow query engines: Trino, Redshift, and Spark SQL. The diff also includes new Calcite transformation rules (\SimpleCallTransform\ and \SimplePredicateTransform\) to assist with SQL rewriting, and updates the DuckDB converter to handle nested \TIMESTAMPDIFF\ rewrites.

sqrl-planner/src/main/java/com/datasqrl/calcite/convert · high confidence

Redesigned documentation site with new landing pages and reusable components

The documentation website has been revamped to feature dedicated landing pages for the main product, AI capabilities, and Flink integration, alongside a new Community page. This update introduces reusable React components for the homepage header and feature lists, allowing for consistent styling and easier content management across the new pages. The main index page now highlights the 'Agentic Data Engineering Harness' positioning with updated messaging, code examples, and architectural diagrams, while the AI and Flink pages showcase specific use cases and technical benefits. The community page directs users to GitHub discussions and other support channels.

documentation/src/pages · high confidence

Redesigned sqrl-cli container with updated dependencies and improved workspace handling

The sqrl-cli Docker image has been rebuilt from a new base structure, upgrading JBang to version 0.141.0 and PostgreSQL to version 18. The container now includes Redpanda and pre-installed DuckDB extensions, and supports S3A IRSA credentials via a new core-site.xml configuration. For users, the entrypoint script now enforces the /workspace mount point (deprecating /build) and ensures file ownership matches the host directory, while PostgreSQL is configured to listen on all addresses to allow external connections.

sqrl-cli · high confidence

Refactor configuration loading to use Spring-managed interfaces

The configuration loading logic in the planner has been refactored to replace the previous implementation with a new set of Spring-managed components and interfaces. This change introduces specific implementation classes such as \CompilerApiConfigImpl\, \ConnectorConfImpl\, and \EngineConfigImpl\ to handle compiler, connector, and engine settings respectively. It also adds a \CompilerApiConfigConverter\ to map API configurations and a \GraphqlSourceLoader\ to manage GraphQL schema loading and inference. For users, this represents a structural update to how the system reads and validates configuration files, ensuring that configuration objects are properly instantiated and managed by the Spring context.

sqrl-planner/src/main/java/com/datasqrl/config · high confidence

Refactor server execution context and metadata handling

The server module introduces a new StandardExecutionContext class to handle GraphQL data fetching and parameter resolution (arguments, metadata, computed values) more flexibly. Additionally, the UserDefinedType class from the sqml-core module has been renamed to ResolvedMetadata and moved to the server package, simplifying its structure to a record focused on metadata type, name, and requirement status.

sqrl-server-core · medium confidence

Refactored DDL generation with dialect-specific factories and new index support

The DDL generation logic in the planner has been restructured to use a new set of factory classes (e.g., \GenericCreateTableDdlFactory\, \PostgresCreateTableDdlFactory\, \TrinoCreateTableDdlFactory\) that handle schema creation, partitioning, and identifier quoting per database dialect. This change introduces support for creating indexes with specific types (GIN for text search, HNSW for vector similarity) and sort directions, as well as late-binding views for Redshift. It also adds infrastructure for PostgreSQL \LISTEN/NOTIFY\ triggers to support real-time change notifications.

sqrl-planner/src/main/java/com/datasqrl/engine/database/relational/ddl · high confidence

Refactored SQL dialect architecture with new base classes and extended dialects

The SQL dialect implementation has been restructured to simplify the addition of new database targets. A new \BasePostgresSqlDialect\ abstract class centralizes unparse logic and translation dispatching, which is now handled by a \SqlTranslationDispatcher\ that leverages service-loaded \SqlTranslation\ plugins. This foundation supports new or updated dialect implementations: \ExtendedPostgresSqlDialect\ now explicitly maps types (e.g., avoiding JSONB casts for arrays, mapping Flink types to PostgreSQL equivalents) and enforces PostgreSQL-specific SQL conformance rules; \DuckDbSqlDialect\ is introduced as a standalone dialect extending the base; \ExtendedSparkSqlDialect\ and \ExtendedTrinoSqlDialect\ provide specialized type casting and SQL rendering (e.g., handling \SUBSTRING\ vs \SUBSTR\, \APPROX\_COUNT\_DISTINCT\); and \ExtendedSnowflakeSqlDialect\ supports Snowflake-specific operations including \CREATE ICEBERG TABLE\ from object storage. This change improves modularity and ensures accurate SQL generation across PostgreSQL, DuckDB, Spark, Trino, and Snowflake targets.

sqrl-planner/src/main/java/com/datasqrl/calcite/dialect · high confidence

Refactored SQL parser infrastructure with improved error location tracking and comment handling

The parser module has been restructured to improve how SQRL scripts are split, parsed, and how errors are reported. A new \SqlScriptStatementSplitter\ now handles script splitting, correctly ignoring statement delimiters inside string literals and preserving block comments while filtering line comments. Error reporting is enhanced via \ParsePosUtil\ and \SqlParserExceptionHandler\, which map Flink/Calcite parser exceptions to precise file locations and clean up error messages. New core types like \ParsedObject\ and \SqrlComments\ track source positions and separate documentation from hints, ensuring that error locations remain accurate even when definitions are transformed or stacked.

sqrl-planner/src/main/java/com/datasqrl/planner/parser · high confidence

Refactored canonicalizer to use a new Name abstraction

The canonicalizer module has been restructured to introduce a new \Name\ interface that distinguishes between a canonical representation (used for internal comparison and equality) and a display representation (preserving the user's original input). This change replaces the previous implementation with new classes such as \AbstractPath\, \NamePath\, \StandardName\, and specific canonicalizers like \LowercaseEnglishCanonicalizer\ and \IdentityCanonicalizer\, allowing for more flexible handling of field names during script parsing and data ingestion.

sqrl-planner/src/main/java/com/datasqrl/canonicalizer · high confidence

Refactored database engine planning model with new interfaces and combined plan support

The database engine planning model has been restructured to introduce distinct interfaces for database and query engines, allowing for clearer separation of concerns between data persistence and query execution. New interfaces such as DatabaseEngine, QueryEngine, and AnalyticDatabaseEngine define the capabilities and relationships between these components, while CombinedEnginePlan enables the aggregation of multiple engine plans into a single deployment artifact. This change also includes the introduction of DatabaseEngineFactory to standardize engine creation and DatabasePhysicalPlan to represent the output of database-specific planning stages.

sqrl-planner/src/main/java/com/datasqrl/engine/database · high confidence

Refactored execution pipeline to support multiple database engines and shallow query validation

The execution pipeline has been refactored to support multiple database engines and introduce a new shallow query engine type. The \SimplePipeline\ now explicitly manages stages for LOG, PROCESS (formerly STREAMS), DATABASE, SERVER, and EXPORT engines, allowing for more flexible engine configurations. A key behavioral change is the addition of validation logic: when a SERVER engine is enabled with a JDBC table format database, the system now enforces that a standard query engine is present, rejecting configurations that rely solely on shallow query engines which are not integrated at the database level. This ensures that server-side queries are processed by engines capable of handling the required database interactions.

sqrl-planner/src/main/java/com/datasqrl/engine/pipeline · high confidence

Refactored flexible type system with visitor pattern and new basic types

The flexible type system in the planner has been restructured to support a visitor-based architecture and a new set of basic data types. A new \SqrlTypeVisitor\ interface and \Type\ base interface have been introduced, allowing types to be processed via the visitor pattern. The system now includes dedicated classes for basic types such as \BigIntType\, \BooleanType\, \DoubleType\, \IntervalType\, \ObjectType\, \StringType\, and \TimestampType\, each implementing specific conversion logic and type distance calculations. A \BasicTypeManager\ centralizes type registration, name lookup, and combination logic. Legacy classes like \DecimalParseResult\ and \LikePatternType\ have been replaced or renamed to \ArrayType\ and \Type\ respectively, aligning with the new package structure \com.datasqrl.io.schema.flexible.type\.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/type · high confidence

Refactored module and function loading infrastructure

The \sqrl-planner\ module's loader subsystem has been restructured to support individual table imports and exports. This introduces a new \ClasspathFunctionLoader\ to discover standard library functions via Java's \ServiceLoader\, a \ModuleLoaderImpl\ that caches and resolves modules from both the file system and classpath, and a \ScriptSqrlModule\ that lazily initializes \CREATE TABLE\ statements for granular access. The refactoring also adds a \UdfJarClassLoaders\ component to manage user-defined function JARs and ensures these classloaders are properly closed after compilation to prevent resource leaks.

sqrl-planner/src/main/java/com/datasqrl/loaders · high confidence

Refactored packaging pipeline with file preprocessing and improved error diagnostics

The packager module has been restructured to introduce a dedicated FilePreprocessingPipeline component that handles copying relevant source files (SQL, GraphQL, config, schema) into the build directory and executing registered preprocessors. The main Packager component now orchestrates this preprocessing, including support for included sub-projects, and explicitly cleans the default Iceberg warehouse directory before building. Additionally, if Flink compilation fails, the system now writes out the Directed Acyclic Graph (DAG) and the generated SQL to log files in the build directory to aid in debugging.

sqrl-cli/src/main/java/com/datasqrl/packager · high confidence

Refactored preprocessing pipeline with new static data and JBang UDF handlers

The preprocessing logic in the CLI packager has been restructured into a modular pipeline of Spring-managed components. A new CopyStaticDataPreprocessor now automatically handles .jsonl, .avro, and .csv files, specifically stripping header rows from CSVs to ensure Flink compatibility. JBang-based Java UDFs are now processed by a dedicated JBangPreprocessor that exports fat JARs with class-path support and caches builds for performance. Existing JAR UDFs are handled by a JarPreprocessor, while SqrlPreprocessor manages script templating via Mustache. All UDF manifest generation is centralized in a new UdfManifestPreprocessor base class, and the Preprocessor interface standardizes the processing contract.

sqrl-cli/src/main/java/com/datasqrl/packager/preprocess · high confidence

Schema loading refactored to support explicit CSV configuration and Avro timestamp mapping

The schema loading mechanism has been refactored to enforce explicit schema declarations for CSV files, preventing automatic inference which can lead to data loss if header rows are present. Users must now configure the 'filesystem' connector with 'format' = 'csv' and declare columns manually for CSV tables. Additionally, the system now respects the Avro timestamp mapping configuration option during schema inference, ensuring accurate timestamp handling for Avro data sources.

sqrl-planner/src/main/java/com/datasqrl/loaders/schema · high confidence

Server logging now includes forwarded client addresses

The server now resolves and logs the actual client IP address from the X-Forwarded-For header in access logs and detailed request traces, rather than only logging the immediate connection address. This ensures accurate client identification when the server is behind a proxy or load balancer.

sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server · high confidence

Standardized default configuration and logging resources

The planner now ships with a standardized set of default configuration files to simplify setup and ensure consistent behavior. A new \default-package.json\ defines baseline settings for enabled engines (Vert.x, Postgres, Kafka, Flink), compiler options (including the new \LIMITED\_RULES\_NO\_SOURCE\ predicate pushdown rules and API pagination defaults), and connector properties (such as Kafka compression and Iceberg warehouse paths). Run-specific Flink execution details (like checkpointing and local target settings) have been moved to \default-run-package.json\. Additionally, a \log4j2.properties\ file standardizes logging to INFO level on the console, and \saffron.properties\ sets Calcite defaults, while a sample GraphQL schema (\c360-small.graphqls\) is provided for reference.

sqrl-planner/src/main/resources · high confidence

Fixes

Fix init-project templates to ensure production and test configurations compile and run correctly

The \init\ command now generates project templates that are ready to compile and execute out of the box. Production configurations now include a valid Iceberg connector definition for reading from a lakehouse, while test configurations explicitly disable source-idleness (\table.exec.source.idle-timeout: 0 s\) to prevent test hangs. Additionally, the generated package files correctly reference the main script name, and test data snapshots are included to validate the initial setup.

sqrl-cli/src/main/resources/templates/init-project · high confidence

Test coverage

Added GraphQL schema test fixtures for the planner converter; Added JUnit 4 compatibility stubs for testcontainers; Added JWT authorization test suite for DataSQRL; Added Snowflake-disabled integration test case; Added analytics-only integration test case with Snowflake and Iceberg support; Added banking-batch integration test resources; Added clickstream integration test suite with paginated GraphQL support; Added integration test for Kafka retention configuration; Added integration test for PostgreSQL index selection edge cases; Added integration test for Postgres text search GIN index usage; Added integration test for connector compilation; Added integration test for multi-batch compilation with table imports; Added integration test for sensors-teaser use case; Added integration test for stateful temporal joins with TTL; Added integration tests for Avro schema handling and Java UDFs; Added integration tests for DuckDB function translation; Added integration tests for DuckDB use cases; Added integration tests for Flink SQL to PostgreSQL function mappings; Added integration tests for Flink string and utility functions; Added integration tests for Flink-only compilation and Iceberg export workflows; Added integration tests for JWT-authorized GraphQL mutations, subscriptions, and queries; Added integration tests for OpenAI stdlib functions; Added integration tests for PostgreSQL array handling and GraphQL mapping; Added integration tests for PostgreSQL function translation and row serialization; Added integration tests for Redshift, Spark SQL, and Trino dialects; Added integration tests for TO\_CHANGELOG functionality; Added integration tests for complex mutation scenarios with Iceberg and Kafka; Added integration tests for filtered distinct and ARRAY\_AGG operations; Added integration tests for math and vector standard library functions; Added integration tests for passthrough employee hierarchy queries; Added integration tests for seedshop tutorial connectors and load functions; Added integration tests for sensor mutation and documentation use cases; Added integration tests for server-side function execution; Added integration tests for the conference use case; Added repository use-case tests for package and topic queries; Added shared banking test fixtures; Added snapshot for ApplicationInfoTest in analytics-only use case; Added snapshot tests for JWT-authorized GraphQL mutations, subscriptions, and queries; Added snapshot tests for sensors-teaser use case; Added test case for Kafka topic prefix compilation; Added test case for OAuth authentication with MCP endpoints; Added test case for combined JWT-OAuth metadata access; Added test case for handling null timestamps in event streams; Added test case for nested input structures in MCP tools; Added test coverage for server-side client address resolution, pagination metadata, and GraphQL integration; Added test coverage for table identity preservation in deferred INSERT statements; Added test data fixtures for book club scenarios; Added test data fixtures for multiple use-case examples; Added test for Avro legacy timestamp mapping in schema inference; Added test for DuckDB CTE materialization with Iceberg scan reuse; Added test for Flink HTTP connector lookup; Added test for LIKE source table validation in mutation database; Added test for ORDER BY alias shadowing in DuckDB; Added test for distinct materialization with Postgres export; Added test for unauthorized JWT access; Added test infrastructure for integration testing; Added test resources for Avro schema compilation; Added test resources for logging and Vert.x REST configuration; Added test resources for user activity use case; Added test utilities and assertions in sqrl-planner; Added tests for Flink SQL node generation and identity preservation; Added tests for Flink compile error output handling; Added tests for Flink planner conflict resolution, Iceberg configuration, and deployment model serialization; Added tests for GraphQL query metrics, tail sample tracing, and operation-only preparsing; Added tests for GraphQL schema conversion; Added tests for JBangRunner, OsProcessManager, and SqrlInjector utilities; Added tests for JWT configuration merging and serialization; Added tests for Kafka health tracking and SASL retry logic; Added tests for OAuth failure handling and protected resource URL generation; Added tests for PgPartmanExtension DDL generation; Added tests for SQL script statement splitting edge cases; Added tests for SQRL error code descriptions; Added tests for TestOutputManager console log redirection; Added tests for environment variable resolution and SQL table name extraction; Added tests for hint parsing and validation logic; Added tests for index selection logic and sort order handling; Added tests for module loading and UDF classloader lifecycle; Added tests for nullable GraphQL API arguments; Added tests for pagination metadata and JDBC type mapping; Added tests for protocol-specific API exposure and configuration; Added tests for relational planner components; Added tests for relationship field preservation in complex queries; Added tests for schema type handling and combination logic; Added tests for server configuration template serialization; Added tests for type compatibility and ORDER BY alias handling; Added unit tests for Avro schema factory and type conversion; Added unit tests for PostgreSQL DDL generation and notification triggers; Added unit tests for SQRL CLI commands and utilities; Added unit tests for SQRL configuration and compiler components; Added unit tests for preprocessors; Added unit tests for server configuration and GraphQL parser settings; Added unit tests for server utility classes; Expanded container integration test coverage for server features; New JUnit 5-based integration test infrastructure for compiler and use-case validation; Removed TPC-H and scratch SQL test resources; Removed unused example parsing test.

Dependencies

Documentation site dependencies updated to React 19 and Node 22

The documentation site now requires Node.js 22 or higher and has upgraded its React and React-DOM dependencies to version 19.2.8. Additionally, the Docusaurus tooling has been updated to version 3.10.2, and the local search plugin has been bumped to 0.55.3.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

This is the PUBLIC form of this artifact. Findings are listed in full, but the details of SECURITY findings — which rule fired, in which file, on which line, and how to fix it — are deliberately withheld, and any secret-scanner results are excluded entirely. Where detail is absent here it was REMOVED FOR PUBLICATION; it is not missing from the analysis. The complete artifact is available from the repository owner.

Score

  • CAI 59 → 55 (-3.3)
  • Rubric changed (rubric-2026.10.1 → rubric-2026.10.5) — scores are not directly comparable.

Lenses

  • Code Health 63 → 63 (-0.2)
  • Architecture 83 → 83 (+0.0)
  • Maturity 74 → 50 (-23.6)
  • Readiness 49 → 49 (+0.0)
  • Security 67 → 69 (+2.3)
  • Performance 100 (new)

Resolved (13)

  • FunctionTooLong: flink.Home (documentation/src/pages/flink.tsx)
  • High CVE: [GHSA redacted] (sqrl-testing/sqrl-testing-container/pom.xml)
  • High CVE: [GHSA redacted] (sqrl-testing/sqrl-testing-container/pom.xml)
  • High CVE: [GHSA redacted] (documentation/package-lock.json)
  • High CVE: [GHSA redacted] (documentation/package-lock.json)
  • Medium CVE: [GHSA redacted] (documentation/package-lock.json)
  • Medium CVE: [GHSA redacted] (documentation/package-lock.json)
  • Medium CVE: [GHSA redacted] (documentation/package-lock.json)
  • Medium CVE: [GHSA redacted] (documentation/package-lock.json)
  • Off-boarding risk: anonymized user #1
  • Off-boarding risk: anonymized user #2
  • PR-triggered workflow without a permissions block
  • Workflow holding a long-lived secret is unscoped

New (29)

  • High CVE: [GHSA redacted] (documentation/package-lock.json)
  • High CVE: [GHSA redacted] (documentation/package-lock.json)
  • High CVE: [GHSA redacted] (documentation/package-lock.json)
  • High secret: WD-SECRET-0005 (profiles/default/.env.ftl)
  • High vulnerability: [GHSA redacted] (documentation/package-lock.json)
  • High: security finding (details withheld)
  • Medium CVE: [GHSA redacted] (documentation/package-lock.json)
  • Medium CVE: [GHSA redacted] (documentation/package-lock.json)
  • No assertions: givenInitializedProject_whenCompileProdAndTestPackages_thenSucceeds (sqrl-testing/sqrl-testing-integration/src/test/java/com/datasqrl/InitProjectCompileTest.java)
  • Off-boarding risk: anonymized user #1
  • Off-boarding risk: anonymized user #2
  • Orphaned file (sqrl-planner/src/main/java/com/datasqrl/calcite/dialect/postgres/PostgresConformance.java)
  • Orphaned file (sqrl-planner/src/main/java/com/datasqrl/calcite/type/TypeFactory.java)
  • Orphaned file (sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/FlexibleTableConverter.java)
  • Orphaned file (sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/converters/SqrlTypeRelDataTypeConverter.java)
  • Orphaned file (sqrl-planner/src/main/java/com/datasqrl/util/FunctionUtil.java)
  • Orphaned file (sqrl-planner/src/main/java/org/apache/calcite/rel/rel2sql/FlinkRelToSqlConverter.java)
  • Outdated (compile): com.fasterxml.jackson.core:jackson-core
  • Outdated (compile): com.fasterxml.jackson.core:jackson-databind
  • Outdated (compile): com.fasterxml.jackson.dataformat:jackson-dataformat-yaml
  • …and 9 more

Changes since last survey

  • 53 commits — 36 feature/other, 17 fixes

By area

  • (root) — 15 commits
  • agent/skills — 11 commits
  • sqrl-testing/sqrl-testing-integration — 7 commits
  • agent/Dockerfile — 3 commits
  • documentation/src — 3 commits
  • sqrl-planner/src — 3 commits
  • .github/workflows — 2 commits
  • .agents/skills — 1 commit
  • .circleci/config.yml — 1 commit
  • .github/pull_request_template.md — 1 commit
  • agent/AGENTS.md — 1 commit
  • documentation/docs — 1 commit
  • documentation/scripts — 1 commit
  • documentation/static — 1 commit
  • sqrl-cli/src — 1 commit
  • sqrl-server/sqrl-server-vertx-base — 1 commit

Notable commits

  • fix: chore: Remove outdated files, fix mailto link in code of conduct
  • fix: fix!: Drop schema discovery for CSV files and copy data files unchanged (#2476)
  • fix: fix(agent): Give users docker run commands instead of the container-only cmd.sh wrapper (#2503)
  • fix: fix(skill): Timestamp reasoning in implement-sqrl (#2490)
  • fix: fix(skills): Move index hint guidance from implement-sqrl into a reactive guide (#2495)
  • fix: fix: Accept ID and JSON mutation inputs for string columns and allow omitting nullable columns (#2489)
  • fix: fix: Default the agent to Pi's default model of the selected provider (#2480)
  • fix: fix: Default the docker images to UTC to avoid TIMESTAMP_LTZ shifts (#2463)
  • fix: fix: Do not misparse table names starting with Import/Export (#2449)
  • fix: fix: Exit vertx server on OutOfMemoryError (#2467)
  • fix: fix: Keep REST and MCP operations for queries without arguments (#2494)
  • fix: fix: Make Postgres text_search queries match the generated GIN index (#2508)
  • fix: fix: Make init prod configs and add-func UDFs compile out of the box (#2462)
  • fix: fix: Resolve custom GraphQL relationship fields on the function's base table (#2500)
  • fix: fix: Teach the CSV header-warning false positive and LIKE failure in agent skills (#2498)
  • fix: fix: Use configured OAuth resource origin for WWW-Authenticate resource_metadata (#2497)
  • fix: fix: use data files for streaming tests instead of datagen (short-term) (#2492)
  • change: chore(deps): Bump iceberg.version from 1.11.0 to 1.12.0 (#2429)
  • change: chore(deps): Bump io.netty:netty-codec from 4.2.18.Final to 4.2.19.Final (#2457)
  • change: chore(deps): Bump io.reactiverse:vertx-maven-plugin from 2.1.0 to 2.2.0 (#2487)
  • …and 33 more

Architecture

  • Containers 0 added · 0 removed · contexts 1 added · 0 removed · edges 0 added · 0 removed

Added bounded contexts (1)

  • python

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

DataSQRL/sqrl was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 22 October 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit da0367eaa9b49edf28264595160b1cd7b8c972b1 — the exact code this score is about.
  • Scored under rubric-2026.10.5 — the same rubric and the same method as every other entry in this index.
  • Measured by watchdog.canine.dev using codehealth-analyzer measure-2026.10.3+preprod-fddbc92cb8df.