DataSQRL/sqrl

This system is a data engineering framework that compiles high-level SQRL scripts into executable data pipelines using Apache Flink. It manages the full lifecycle of data processing, including schema inference, cost-based query optimization, and deployment to various engines like PostgreSQL, Kafka, and Iceberg. The platform provides a CLI for project scaffolding and compilation, alongside a Vert.x-based server that exposes the processed data via GraphQL and REST APIs.

54 changes.

This period focused on stripping out legacy components, including the custom SQL parser, GraphQL server, and various outdated API modules, to clean the codebase. Concurrently, the project shifted its execution engine to Apache Flink, introducing new interfaces for streaming, logging, and database engines while adding support for Iceberg, Kafka, and DuckDB.

82 changes.

The planner underwent a major architectural overhaul, introducing a DAG-based planning system with cost-based optimization and a flexible, visitor-based type system. This period focused on expanding database support by refactoring SQL dialects to enable targets like PostgreSQL, Snowflake, Trino, and Spark, while adding extensive integration tests and CLI scaffolding tools.

40 changes.

This period focused on expanding DuckDB support through comprehensive function translations, memory configuration, and CTE materialization, alongside significant server-side refactoring for GraphQL pagination and client address logging. The team also introduced new deployment model structures and enhanced PostgreSQL capabilities with pg\_partman and pgvector extensions. Extensive test coverage was added across integration, planner, and server modules to validate these new features and ensure stability.

Introduces a new IcebergDataTypeMapper that handles specific Flink data types when writing to Iceberg. It explicitly maps Flink's JSON type to a string-only conversion and Vector types to a double-only conversion, while falling back to a byte conversion for other unsupported types.

sqrl-planner/src/main/java/com/datasqrl/datatype/flink/iceberg · high confidence

A new Print export engine has been introduced, allowing users to configure an export target that prints data to standard output. This feature is implemented via the new PrintEngine class and its corresponding PrintEngineFactory, which registers the 'print' engine type within the export pipeline. The engine integrates with the existing connector configuration system to allow basic customization of the output format.

sqrl-planner/src/main/java/com/datasqrl/engine/export · high confidence

This change introduces the \postgres\_log\ execution engine, providing the core classes (\PostgresLogEngine\, \PostgresLogEngineFactory\, \PostgresLogPhysicalPlan\) and a data type mapper for PostgreSQL as a log destination. The engine is registered via \AutoService\ and includes a \ListenNotifyAssets\ structure for handling database notifications, though the physical planning and table creation methods are currently marked as unsupported, indicating this is a foundational addition rather than a fully functional feature.

sqrl-planner/src/main/java/com/datasqrl/engine/log/postgres · high confidence

The documentation site now includes the Apache Flink logo as a static SVG image in the blog assets folder, and a \.nojekyll\ file has been added to the static root to ensure the hosting platform does not process the documentation with Jekyll, preserving the intended static file structure.

documentation/static · high confidence

The CLI now includes new Java template files for user-defined functions, enabling users to create custom scalar and aggregate functions via the \add-func\ command. The scalar template provides a basic \ScalarFunction\ implementation that repeats a string, while the aggregate template offers a time-weighted average \AggregateFunction\ using Flink's API, allowing users to generate these specific function types directly from the command line.

sqrl-cli/src/main/resources/templates/functions · high confidence

The planner now supports translating a set of internal JSON functions to their PostgreSQL equivalents. This includes aggregations like \jsonb\_agg\ (via \JsonArrayAggSqlTranslation\), construction functions like \jsonb\_build\_array\ and \jsonb\_build\_object\ (via \JsonArraySqlTranslation\ and \JsonObjectSqlTranslation\), and extraction/querying functions like \jsonb\_path\_query\_first\ (via \JsonExtractTranslation\). Additionally, it handles JSON concatenation (\\|\|\), existence checks (\jsonb\_path\_exists\), string conversion (\\#\>\>\), and the casting of typed ROW values to JSONB objects, ensuring these operations work correctly when targeting a PostgreSQL dialect.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/json · high confidence

The planner now supports translation and metadata for several Postgres vector operations. New SQL translation classes handle the \center\, \cosine\_distance\, \cosine\_similarity\, and \euclidean\_distance\ functions, ensuring they are correctly unparsed for the Postgres backend. Additionally, metadata classes define the indexability and operand constraints for \text\_search\, \cosine\_similarity\, and \euclidean\_distance\, enabling the planner to optimize queries involving these vector and text search functions.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/vector · high confidence

The planner now includes a new TypeCompatibility utility that validates whether a new data schema can safely read data produced by an older schema, ensuring backwards compatibility by checking nullability, structural recursion for rows/arrays/maps, and precision constraints. Additionally, a new TypeFactory component extends Flink's type system to provide standardized methods for creating specific Calcite types (such as timestamps, UUIDs, and integers) and mapping them to Java classes, supporting the underlying type resolution logic.

sqrl-planner/src/main/java/com/datasqrl/calcite/type · high confidence

A new script has been added to automatically generate markdown documentation for system and library functions. This tool reads function definitions from YAML source files and produces structured reference pages, ensuring that the documentation for built-in operations and importable library functions remains consistent with the underlying codebase.

documentation/scripts · high confidence

The planner now automatically analyzes query patterns to generate optimal database indexes for JDBC tables, supporting various types such as BTREE, HASH, and partitioned indexes. This includes a specific optimization for paginated queries that automatically adds BTREE indexes on rowtime columns to speed up offset-based pagination. Additionally, the DAG visualization now includes row count estimations for tables to provide better insight into data volume.

sqrl-planner/src/main/java/com/datasqrl/plan/global · high confidence

The server now supports setting a configurable memory limit for DuckDB via the \memory-limit\ configuration option, allowing users to control resource usage during initialization. Additionally, new utility classes \DuckDbInitializer\ and \JsonUtils\ have been added to the server base module; \DuckDbInitializer\ handles the construction of initialization SQL including extension loading and memory settings, while \JsonUtils\ provides a pre-configured Jackson \ObjectMapper\ with environment variable resolution support and a recursive merge function for JSON nodes.

sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/util · high confidence

The Vert.x server is now packaged as a container image that launches via a new SqrlLauncher entry point, which automatically registers a Prometheus metrics registry for observability. The container includes a dedicated entrypoint script that supports debug mode via the SQRL\_DEBUG environment variable and writes application logs to the console. Logging is configured with two separate Log4j2 profiles: a standard log4j2.properties for general INFO-level output and a log4j2-debug.properties that enables DEBUG-level logging for the DetailedRequestTracer, routing those detailed request traces to a rotating file at /opt/sqrl/logs/request-trace.log.

sqrl-server/sqrl-server-vertx · high confidence

The planner now includes a comprehensive set of new SQL translation implementations for the DuckDB dialect, covering array operations (ARRAY\_JOIN, ARRAY\_PREPEND, ARRAY\_REMOVE, ELEMENT, ELT), string manipulation (DATE\_FORMAT, LOCATE, ENDS\_WITH, IS\_ALPHA, IS\_DECIMAL, IS\_DIGIT, JSON\_OBJECT, DECODE, ENCODE), and utility functions (CONVERT\_TZ, E). These additions ensure that Flink SQL expressions using these functions are correctly translated into valid DuckDB syntax, improving compatibility and query execution reliability.

sqrl-planner/src/main/java/com/datasqrl/function/translation/duckdb · high confidence

The planner now supports a broader set of Flink built-in functions when targeting PostgreSQL, translating operations such as array manipulation (ARRAY\_CONCAT, ARRAY\_CONTAINS, ARRAY\_JOIN, ARRAY\_POSITION, ARRAY\_PREPEND, ARRAY\_REMOVE), string and binary encoding (BIN, DECODE, ENCODE, FROM\_BASE64, HEX), date/time handling (CONVERT\_TZ, CURRENT\_TIME, CURRENT\_TIMESTAMP, DAY\_OF\_WEEK), and utility functions (COLLECT, ELT, ELEMENT, IFNULL, ENDS\_WITH, E). This ensures these functions execute correctly in PostgreSQL-compatible environments.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/builtinflink · high confidence

This change introduces the first stage of database-specific function mapping. For PostgreSQL, it adds a new \TextSearchTranslation\ implementation that converts the platform's text search operator into native Postgres \to\_tsvector\, \to\_tsquery\, and \ts\_rank\_cd\ calls. For Snowflake, it adds a suite of stub translation classes for JSON functions (such as \JsonArrayAgg\, \JsonObject\, and \JsonExtract\). Most of these Snowflake translations are currently disabled (not registered via \@AutoService\) and will throw runtime errors if invoked, serving as placeholders for future implementation.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/text, sqrl-planner/src/main/java/com/datasqrl/function/translation/snowflake · high confidence

Users can now configure DuckDB as a dedicated query engine to execute SQL queries against Iceberg tables. This change adds a new \DuckDBEngine\ and its factory, along with a \DuckDbStatementFactory\ that implements cost-based optimization for Iceberg scans. The planner automatically materializes repeated scans of large Iceberg tables as Common Table Expressions (CTEs) to improve query performance, while handling Iceberg-specific path resolution and catalog configurations.

sqrl-planner/src/main/java/com/datasqrl/engine/database/relational · high confidence

Added the FlexibleTableConverter class in the flexible schema package, which implements a visitor pattern to traverse and process flexible table schemas. This component handles the conversion of field types, including support for nested relations, array depths, and nullability constraints, enabling the planner to interpret complex schema structures during query planning.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible · high confidence

The streaming engine location now provides a concrete Flink implementation, replacing the previous abstract or placeholder structure. This includes the FlinkStreamEngine for runtime configuration and checkpointing, a FlinkEngineFactory registering the engine as type PROCESS, and a suite of SQL planning utilities (FlinkCalciteParser, FlinkSqlNodePlanner, FlinkDialect) to handle SQL parsing, validation, and relational-to-SQL conversion. The package also introduces Calcite optimization rules (ShapeBushyCorrelateJoinRule, ToStubAggRule) to improve query planning and execution for Flink sinks.

sqrl-planner/src/main/java/com/datasqrl/engine/stream · high confidence

The planner now supports Kafka as a dedicated log engine, allowing users to configure topic creation parameters such as the number of partitions, replication factor, and retention TTL. This implementation handles both stream and mutation tables, automatically applying upsert formats for state tables with primary keys and supporting configurable watermarks for source and transactional processing.

sqrl-planner/src/main/java/com/datasqrl/engine/log/kafka · high confidence

The \sqrl-planner\ module adds a suite of new utility classes and interfaces to support configuration loading, file handling, and service discovery. This includes \MainScript\ and \MainScriptImpl\ for resolving and reading the primary SQRL script and associated mutation database definitions, \ConfigLoaderUtils\ for loading and validating \package.json\ configurations against JSON schemas, and \ServiceLoaderDiscovery\ for caching and retrieving service implementations. Additional utilities such as \AbstractDAG\ for directed acyclic graph operations, \FileUtil\ and \FileCompression\ for file system and compression handling, and \CalciteUtil\ for SQL planner type manipulations are also introduced to centralize common logic.

sqrl-planner/src/main/java/com/datasqrl/util · high confidence

The planner now includes a cost analysis module that evaluates the relative expense of executing operations on different engine types (Database, Process, Server, Log). This new capability allows the system to automatically select the most efficient execution strategy by weighing factors such as join complexity, table types (Stream, State, Versioned, Static, Lookup, Relation), and engine-specific penalties, thereby optimizing query performance without manual configuration.

sqrl-planner/src/main/java/com/datasqrl/planner/analyzer/cost · high confidence

A new FlexibleJsonFlinkFormatTypeMapper is added to handle data type conversions within the Flink JSON format. This mapper defines specific rules for mapping SQL types to JSON-compatible formats, including handling of arrays, rows, and vector types, ensuring that complex data structures are correctly serialized or converted to bytes when necessary.

sqrl-planner/src/main/java/com/datasqrl/datatype/flink/json · high confidence

A new TableType enum has been added to define and classify table behaviors within the planner, including types such as STREAM, VERSIONED\_STATE, STATE, LOOKUP, RELATION, and STATIC. This enum provides methods to determine specific table properties like whether they have a timestamp, primary key, or support temporal joins, and defines logic for combining different table types.

sqrl-planner/src/main/java/com/datasqrl/io/tables · high confidence

The CLI now includes support for bootstrapping new projects and visualizing data structures. An \init-project.properties\ file defines default engine configurations (Flink, Kafka, Iceberg, etc.) for stream, dataset, and API components, enabling the \init\ command to scaffold projects with sensible defaults. Additionally, two new HTML resources, \visualize\_dag.html\ and \data\_model\_visual.html\, provide browser-based visualization for deployment graphs and data models respectively, leveraging the \@datasqrl/dag-visualization\ library to render these views in the CLI's Voyager interface.

sqrl-cli/src/main/resources · high confidence

The CLI now includes a suite of new utility classes in the \com.datasqrl.util\ package to support core operations: \DirectoryUtils\ for managing target directories, \FileHash\ for computing MD5 checksums, \FlinkOperatorStatusChecker\ for monitoring Flink job states via REST, \JBangRunner\ for building and repairing fat JARs for UDFs, \OsProcessManager\ for dynamically starting dependent services (PostgreSQL, Redpanda) and managing file ownership, \ResourceUtils\ for loading resources from both JARs and the file system, \ResultSetPrinter\ for formatting SQL results, and \SqrlInjector\ which configures Spring-based dependency injection for these components and the broader compilation pipeline.

sqrl-cli/src/main/java/com/datasqrl/util · high confidence

A new ExecutionGoal enum has been added to the validation package, defining three distinct phases: COMPILE, RUN, and TEST. This change introduces a structured way to categorize execution goals within the planner's validation logic, likely supporting more granular control over how plans are processed and verified.

sqrl-planner/src/main/java/com/datasqrl/plan/validate · high confidence

The planner now includes a new \DataTypeMapping\ interface and \DataTypeMappings\ registry to handle conversions between Flink types and database engine types. This change introduces specific mappers for JSON (using \jsonb\_to\_string\ and \to\_jsonb\), vector data (using \vector\_to\_double\ and \double\_to\_vector\), and byte serialization (\serialize\_to\_bytes\), enabling more precise type handling for these data structures during query execution.

sqrl-planner/src/main/java/com/datasqrl/datatype · high confidence

The planner now includes a new serialization infrastructure in the \com.datasqrl.serializer\ package. A central \Deserializer\ class provides unified JSON and YAML mapping capabilities using Jackson, configured with specific modules for Java 8, Java Time, and Vert.x. The system introduces a \SqrlSerializerModule\ that automatically discovers and registers custom serializers and deserializers via Java's \ServiceLoader\. This enables extensible serialization, demonstrated by the inclusion of \NewTopicSerializer\ and \NewTopicDeserializer\ for Kafka topic configurations, and a generic \JacksonDeserializer\ for polymorphic type resolution based on a type key.

sqrl-planner/src/main/java/com/datasqrl/serializer · high confidence

The \sqrl-cli\ now includes an \init\ command that generates a complete starter project structure, including a \.gitignore\, production and test package configuration files (\\_\projectname\\-prod-package.json.mustache\, \\\projectname\\-test-package.json.mustache\), a main SQRL script (\\\projectname\\_.sqrl\) with a basic transformation, and connector definitions for both production and test environments. The generated project also includes sample test data (\connectors/test-data/messages.jsonl\) and a pre-computed test snapshot (\snapshots/HelloWorldTest.snapshot\) to facilitate immediate testing of the generated pipeline.

sqrl-cli/src/main/resources/templates/init-project · high confidence

The planner now supports several new hints to fine-tune query execution and data management. Users can define caching behavior with the \cache\ hint (specifying a TTL duration) and manage data retention with the \ttl\ hint. Indexing is more flexible, allowing explicit \index\ hints with specified types and sort orders (ASC/DESC), as well as \vector\_dim\ hints for vector column indexing. Table access patterns are controlled via \query\_by\_all\, \query\_by\_any\, and \no\_query\ hints. Additional hints include \engine\ (replacing the legacy \exec\ hint) for DAG stage assignment, \row\_count\ for cardinality estimation, \mutation\_insert\ for specifying mutation types, \maintenance\ for regular maintenance flags, and \workload\ to mark tables as query-only sources. The system also introduces \DocStringParser\ to extract column and argument documentation from Rust-style markdown comments, and \PartitionKeyHint\ to assign partition keys. Unknown hints now generate warnings instead of fatal errors, improving robustness during planning.

sqrl-planner/src/main/java/com/datasqrl/planner/hint · high confidence

Added seven new documentation/blog posts detailing the DataSQRL 0.6 and 0.7 releases, Flink SQL extensions for defining data interfaces, the Flink SQL Runner toolkit, and the conceptual framework for data platform automation. These posts explain how DataSQRL compiles SQRL scripts into data pipelines, supports temporal joins, and provides a world model for AI coding agents.

documentation/blog · high confidence

The discovery module now supports automatic schema inference for CSV and JSONL data files. New \CSVRecordReader\ and \JsonlRecordReader\ implementations (registered via \@AutoService\) parse these formats, while \AbstractDiscoveryTableSchemaFactory\ orchestrates the process by reading file records, computing field-level statistics (counts, nulls, types, array depths), and merging them into a \FlexibleTableSchema\ via \DefaultSchemaGenerator\. This replaces previous static schema approaches with a dynamic, statistics-based discovery process that adapts to the actual data content.

sqrl-discovery/src · high confidence

Added two new converter classes, FlexibleTable2RelDataTypeConverter and SqrlTypeRelDataTypeConverter, within the flexible schema converters package. These components enable the mapping between the system's internal flexible schema types (such as Boolean, BigInt, String, Timestamp, Interval, and Array) and Apache Calcite's RelDataType structures, allowing the planner to correctly interpret and process flexible schema definitions during query execution.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/converters · high confidence

This change introduces new internal infrastructure for handling function metadata and adds support for specific PostgreSQL operators. It adds a new \FunctionMetadata\ interface (replacing a previously renamed exception file) and several new interfaces (\IndexableFunction\, \InputPreservingFunction\) to classify functions for optimization and primary-key/timestamp determination. It also introduces \PgSpecificOperatorTable\, which defines SQL operators for PostgreSQL full-text search (e.g., \to\_tsvector\, \@@\) and distance metrics (cosine, Euclidean), along with utility classes (\CalciteFunctionUtil\, \FlinkUdfNsObject\) to support these function definitions within the planner.

sqrl-planner/src/main/java/com/datasqrl/function · high confidence

The planner now includes new utility classes in the \com.datasqrl.plan.util\ package to handle column index mapping and primary key tracking. \IndexMap\ and its implementation \SelectIndexMap\ provide a functional interface for remapping column indices, including a \RexIndexMapShuttle\ to update Calcite \RexNode\ references during plan transformations. \PrimaryKeyMap\ manages primary key column sets, supporting operations to check coverage, simplify keys, and remap indices. Additionally, \RelWriterWithHints\ extends Calcite's \RelWriterImpl\ to include query hints in the relational node explanation output, improving debuggability.

sqrl-planner/src/main/java/com/datasqrl/plan/util · high confidence

This change establishes the project's foundational structure by adding essential documentation files (CLAUDE.md, CONTRIBUTING.md, RELEASE.md, README.md), the Apache 2.0 license, and configuration files (codecov.yml, .gitignore). It also introduces Dockerfiles for DuckDB extensions and the MCP Inspector, updates the .gitmodules to point the stdlib-docs submodule to the flink-sql-runner repository, and removes the old sqml-examples submodule.

(repo-wide) · high confidence

The planner now allows selecting a table alias directly in the SELECT clause (e.g., \SELECT p FROM Projects p\). Instead of requiring explicit column listing, the system automatically expands the bare alias into a nested ROW value containing all columns of the referenced relation, enabling users to treat an entire row as a single field in the result set.

sqrl-planner/src/main/java/com/datasqrl/calcite/expand · high confidence

The planner now automatically generates SQL to configure the pg\_partman extension for tables using RANGE partitioning with a Time-To-Live (TTL), including configurable premake counts and historical start partitions. Additionally, it now supports the pgvector extension, enabling vector data types and associated operators for similarity search capabilities.

sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/extensions · high confidence

The Postgres-specific implementation details have been removed from the sqml-postgres module. This includes the deletion of the PostgresResult class and its associated MigrationObject structure, the PostgresFunctions registry, the PostgresSqmlMigration executor, the ShreddingSqlOptimizer, the PostgresStatisticsProvider, and the commented-out Rewriter logic. Users relying on this module for Postgres schema migration or Postgres-specific SQL optimization will no longer have access to these capabilities in this location.

sqml-postgres · high confidence

The \GraphqlSchemaBuilder\ and \SqlGraphqlSchema\ classes in the \sqml-graphql\ module have been removed. These files contained stub implementations for converting SQL logical plans into GraphQL schemas, including a visitor pattern for traversal and basic type mapping logic that was not fully functional. This cleanup eliminates unused code from the codebase.

sqml-graphql/src/main/java · high confidence

The \HttpIngress\ class, which previously allowed users to ingest data streams from HTTP URLs via a buffered reader and asynchronous timeout handling, has been removed from the \sqml-ingress-http\ module. This change eliminates the legacy URL-based source capability from the HTTP ingress component.

sqml-ingress-http · high confidence

The legacy \Main.java\ entry point in \sqml-main/src/main/java\ has been removed. This file previously contained the hardcoded application startup logic, including the definition of a \meetup\ data source, schema configuration, Postgres migration setup, and the initialization of the GraphQL servlet on port 8080. Its deletion indicates a shift away from this specific monolithic startup pattern, likely as part of the broader refactoring and integration of Flink into the pipeline.

sqml-main/src/main/java · high confidence

The legacy SQL grammar file (SqlBase.g4) has been deleted from the parser module. This change removes the previously defined syntax rules for statements, queries, joins, and expressions, indicating that the parser implementation has been replaced or significantly restructured in other parts of the codebase.

sqml-parser/src/main/antlr4 · high confidence

The \sqml-core\ module has removed a significant set of internal classes responsible for SQL query analysis and model representation, including \MetadataManager\, \Model\, \ModelRelation\, \QueryUtil\, \Session\, \StubModel\, \StubModels\, and the entire \analyzer\ package (containing \Analyzer\, \Analysis\, \AggregationAnalyzer\, \ExpressionAnalyzer\, and related utilities). This change eliminates the legacy code paths for parsing, analyzing, and managing SQL query plans and metadata within this core library.

sqml-core · high confidence

The \sqml-api\ module has removed a large set of legacy classes, including the analyzer, optimizer, execution strategies, schema definitions, and vertex infrastructure. This cleanup eliminates unused code and simplifies the API surface by discarding the previous implementation of script analysis, query optimization, and local execution logic.

sqml-api · high confidence

The LocalScriptRegistry class, which previously allowed users to register and retrieve scripts from a local in-memory map, has been removed from the codebase. This change eliminates the ability to use this specific local-only script storage mechanism.

sqml-script-registry-local · high confidence

This change removes the \presto-matching\ library entirely, deleting core classes such as \Pattern\, \Matcher\, \Match\, and \Captures\ along with their associated pattern implementations (e.g., \TypeOfPattern\, \WithPattern\). It also removes the \sqml-server\ module, deleting the GraphQL endpoint implementation (\GraphqlEndpoint\), server orchestration (\SqmlServer\), and various management and handler classes (e.g., \ScriptManager\, \SchemaManager\). Additionally, the \sqml-metadata\ module is stripped of its \ColumnType\, \Identifier\, and \MetadataStore\ definitions.

presto-matching, sqml-metadata, sqml-server · high confidence

The unused PostgresViewVertexFactory class has been removed from the sqml-postgres-view module. This cleanup eliminates dead code that previously provided a factory method for creating SqlVertexFactory instances, simplifying the codebase without affecting active functionality.

sqml-postgres-view · high confidence

The GraphqlServlet class, which served as a stub for a GraphQL endpoint in the sqml-graphql-jetty module, has been removed. This eliminates the unused servlet infrastructure that previously exposed a builder pattern for configuring the GraphQL runtime and schema.

sqml-graphql-jetty · high confidence

The sample resource file meetup.sqml has been removed from the application resources. This file previously defined a data source import for 'meetup' and contained specific field calculations for Rsvp records, including a derived id\_mult field and a sum\_id aggregation.

sqml-main/src/main/resources · high confidence

The entire \sqml-parser\ implementation has been deleted, including the \SqlParser\, \SqmlParser\, \AstBuilder\, and all associated support classes (e.g., \ErrorHandler\, \ParsingOptions\). This removes the capability to parse and analyze SQML scripts and SQL statements within this module.

sqml-parser/src/main/java/ai/dataeng/sqml/parser · high confidence

The entire AST node hierarchy and visitor infrastructure in the \sqml-parser\ tree package has been deleted. This includes all node classes (such as \AliasedRelation\, \ArithmeticBinaryExpression\, \Cast\, and \Query\), the \AstVisitor\ interface, and the default traversal implementations. This change removes the parser's internal representation of SQL queries and expressions, likely as part of a migration to a different parsing or analysis engine.

sqml-parser/src/main/java/ai/dataeng/sqml/tree · high confidence

The SQL planning layer has been restructured to support database-specific extensions and cleaner statement definitions. Two new service-loader interfaces, DatabaseTableExtension and DatabaseTypeExtension, have been added to allow plugins to inject custom DDL and manage database-specific type operators. Additionally, the SqlFunction interface has been renamed and relocated to SqlDDLStatement within the new package, simplifying its contract to a single getSql method to better reflect its role in generating DDL statements.

sqrl-planner/src/main/java/com/datasqrl/sql · high confidence

The planner's analysis logic in the \analyzer\ package has been restructured into a set of focused classes: \AbstractAnalysis\ provides common relational node utilities, \CapabilityAnalysis\ tracks required engine features, \RelNodeAnalysis\ holds intermediate state, and \TableAnalysis\ aggregates table metadata. This refactoring consolidates the analysis components previously scattered or managed by \TypeManager\, improving the clarity and maintainability of the planning pipeline.

sqrl-planner/src/main/java/com/datasqrl/planner/analyzer · high confidence

The server module and its core interfaces have been refactored to be more flexible and decoupled. This introduces a new thread-safe \GlobalEnvironmentStore\ for centralized environment variable access, standardizes environment variable naming via \EnvVariableNames\, and restructures the GraphQL execution layer with new interfaces like \FunctionExecutor\, \MetadataReader\, and \ServerContext\. Additionally, the \GraphQLEngineBuilder\ and \RootGraphQLModel\ have been updated to support these new abstractions, and custom scalars (such as \DOUBLE\ and \FLEXIBLE\_DATETIME\) are now explicitly registered in the GraphQL wiring.

sqrl-server/sqrl-server-core · high confidence

The Avro schema factory and converter now explicitly handle the Avro legacy timestamp mapping setting. When processing Avro schemas, the system checks for a specific configuration property (including prefixed variants) and automatically enables legacy timestamp mapping if the 'avro-confluent' format is detected. This ensures that timestamp fields are converted correctly according to the expected Avro format, preventing potential data type mismatches during schema inference.

sqrl-planner/src/main/java/com/datasqrl/io/schema/avro · high confidence

The CLI command architecture has been refactored to use Spring Boot for dependency injection, replacing the previous Guice-based setup. This change introduces a new command hierarchy (BaseCmd, BaseOsProcessManagerCmd, AbstractCompileCmd) and adds two new user-facing commands: 'init' to scaffold new SQRL projects from templates, and 'add-func' to generate Java UDF stubs. The refactoring also standardizes error handling via a unified ErrorCollector and improves output formatting for compile and test operations.

sqrl-cli/src/main/java/com/datasqrl/cli · high confidence

The application now registers com.datasqrl.error.IOExceptionHandler as the implementation for the com.datasqrl.error.ErrorHandler service interface via the standard Java SPI mechanism. This ensures that I/O-related errors are handled by this specific component.

sqrl-planner/src/main/resources/META-INF · high confidence

The SQRL \cmd\ Docker image now uses a custom \PlannerModule\ to initialize the Flink SQL planner correctly within a single-JVM process. This change overrides the standard Flink classloading behavior to prevent \ClassNotFoundException\ errors when \flink-table-planner.jar\ is not present on the classpath, ensuring the planner initializes properly in the containerized environment without affecting real Flink SQL runner deployments.

sqrl-planner/src/main/java/org/apache/flink · high confidence

The planner now uses custom Calcite SQL converters to generate Flink-compatible SQL, addressing specific execution requirements. This includes rewriting correlated joins with snapshots into temporal joins, handling UNNEST operations as lateral joins, and preserving join types during conversion. The new converters also inject execution hints into SELECT statements, simplify table references by stripping catalog/database prefixes, and ensure SELECT lists are correctly populated to avoid parser quirks.

sqrl-planner/src/main/java/org/apache/calcite · high confidence

The documentation website has been reorganized and configured using Docusaurus 3.7.0, deployed at https://docs.datasqrl.com. The new structure includes a comprehensive sidebar with sections for Core Concepts, Configuration (covering Flink, Kafka, Vert.x, PostgreSQL, and Iceberg engines), Functions, and How-To guides. The site now features a local search plugin, Mermaid diagram support, and a custom dark/light theme with specific color palettes. Deployment is handled via CI/CD to GitHub Pages, and the site requires Node.js 22+ and a git submodule for standard library documentation.

documentation · high confidence

The planner now supports generating SQL for Trino, Redshift, and Spark SQL in addition to existing dialects, as indicated by the new entries in the Dialect enum. To facilitate this, the codebase has been refactored to separate function translation into two mechanisms: structural transformations applied at the RelNode level via the new OperatorRuleTransformer and OperatorRuleTransform interface, and simpler name/argument adjustments handled via SqlTranslation during unparsing. Supporting infrastructure includes the DynamicParamSqlPrettyWriter for tracking dynamic parameters and SqrlRexUtil for improved join condition decomposition.

sqrl-planner/src/main/java/com/datasqrl/calcite · high confidence

The planner now automatically resolves Flink INSERT conflict clauses (ON CONFLICT) for upsert sinks by analyzing the optimized plan, reducing the need for manual configuration and preventing non-deterministic results. Additionally, users can now tune the Flink optimizer's predicate pushdown behavior via the new \LIMITED\_RULES\ and \LIMITED\_RULES\_NO\_SOURCE\ configuration options, which strip specific filter and table-source rules to improve subgraph reuse and query performance.

sqrl-planner/src/main/java/com/datasqrl/planner · high confidence

The planner now uses custom metadata handlers to provide more accurate statistics for query optimization. Row count estimates for table scans are derived from table analysis (primary keys and row count hints), and join cost models now include nested loop joins alongside hash joins, ensuring better plan selection for non-equi-join conditions. Selectivity and column uniqueness estimates are also enhanced to leverage primary key information, leading to more efficient query execution plans.

sqrl-planner/src/main/java/com/datasqrl/plan/rules · high confidence

The flexible input schema package now defines a new type-safe model for schema definitions, including \FlexibleFieldSchema\, \FlexibleTableSchema\, and \RelationType\, which replace the previous \VarcharType\ and \AbstractType\ classes. This change introduces a new constraint system (\Constraint\, \NotNull\, \Cardinality\, \Unique\) to validate field properties, and adds \FlexibleTypeMatcher\ and \TypeSignatureUtil\ to handle automatic type detection and matching against flexible schema definitions.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/input · high confidence

The server engine module now uses a new \GenericJavaServerEngine\ base class and a \VertxEngineFactory\ to manage server-side execution. This change introduces a Jackson-based configuration system (replacing the previous Vert.x \JsonObject\ approach) that merges default templates with package-specific engine configs to produce a \server-config.json\ deployment artifact. The \ServerPhysicalPlan\ now explicitly tracks server functions, mutations, and the merged \ServerConfig\, while a custom \PrettyPrinter\ ensures the generated configuration is human-readable.

sqrl-planner/src/main/java/com/datasqrl/engine/server · high confidence

The engine package now defines a structured capability model via the \EngineFeature\ enum, allowing the planner to verify support for specific operations such as mutations, temporal joins, and partitioning before generating physical plans. This is supported by new interfaces including \EngineConfiguration\ for engine initialization, \ExecutionEngine\ for capability checks, and \EnginePhysicalPlan\ for serializable deployment artifacts. Additionally, legacy interfaces like \FunctionHandle\ and \ParametricType\ have been renamed and repurposed to \ExecutableQuery\ and \IExecutionEngine\ respectively, aligning the internal API with the new execution stage and engine type abstractions.

sqrl-planner/src/main/java/com/datasqrl/engine · high confidence

The planner now exposes new interfaces in the \com.datasqrl.engine.log\ package to handle mutation operations. \LogEngine\ serves as a composite interface extending \ExportEngine\, \MutationEngine\, and \DatabaseEngine\, replacing the previous \TimeWithTimeZoneType\ class which has been removed. \MutationEngine\ defines the contract for creating mutation tables, including support for specifying an insertion type and a Time-To-Live (TTL) duration, enabling the engine to manage data mutations with configurable expiration policies.

sqrl-planner/src/main/java/com/datasqrl/engine/log · high confidence

A new .mvn/jvm.config file has been introduced to configure the JVM with specific --add-opens and --add-exports flags for the jdk.compiler module. This change enables internal access required by code formatting plugins to operate correctly during the build process.

.mvn · high confidence

The planner now constructs a Directed Acyclic Graph (DAG) of table and function definitions to optimize and assemble physical execution plans. This change introduces a new \DAGBuilder\ to track node dependencies, a \PipelineDAG\ structure to manage execution stages and eliminate inviable paths, and a \DAGPlanner\ that assigns nodes to the most cost-effective stages (such as PROCESS or data store engines) before generating the final physical plan. This replaces the previous linear planning approach with a graph-based optimization layer.

sqrl-planner/src/main/java/com/datasqrl/planner/dag · high confidence

A new FlinkSqrlPostgresDataTypeMapper has been introduced to handle PostgreSQL data types within the Flink JDBC connector. This mapper defines specific conversion rules: standard scalar types (such as integers, decimals, dates, and strings) are passed through without modification, while complex types like MAP, ROW, and ARRAY are converted to JSON. Additionally, any other types that do not match the specific Flink JSON or Vector types are cast to bytes, ensuring consistent handling of diverse PostgreSQL data structures in the planner.

sqrl-planner/src/main/java/com/datasqrl/datatype/flink/jdbc · high confidence

The compile module now uses a new \CompilationProcess\ orchestrator that generates OpenAPI JSON deployment artifacts for each API version and writes visual data-model HTML files. It also introduces a test-planning phase that builds a \TestPlan\ containing GraphQL queries, mutations, and subscriptions, allowing the test runner to use a custom GraphQL schema and respect \test(no\_rows)\ hints to skip snapshot generation for empty test queries.

sqrl-cli/src/main/java/com/datasqrl/compile · high confidence

The sqrl-deployment-model module now includes new Java record classes that define the structure of deployment configuration files. These models represent the contents of Flink plans (FlinkPlanModel), JDBC database plans (JdbcPlanModel, JdbcStatementModel), Kafka topic definitions (KafkaPlanModel, KafkaNewTopicModel), and pipeline mutation database configurations (MutationDatabaseModel). This refactoring centralizes the data structures used to parse and validate deployment file formats.

sqrl-deployment-model · high confidence

The \sqrl-planner\ now uses a dedicated error-handling package (\com.datasqrl.error\) to provide structured, user-friendly error messages. This change introduces a system where errors are collected via \ErrorCollector\ and \ErrorCatcher\, which route exceptions through specific \ErrorHandler\ implementations (such as the new \IOExceptionHandler\) to produce consistent \ErrorMessage\ objects with severity levels (FATAL, WARN, NOTICE) and precise file locations. The \ErrorPrinter\ generates pretty-printed output that includes code snippets and context descriptions, while \CollectedException\ ensures stack traces are trimmed to highlight the root cause rather than internal framework noise.

sqrl-planner/src/main/java/com/datasqrl/error · high confidence

The planner now introduces a dedicated \AccessVisibility\ record to control whether table functions are exposed as queryable endpoints, subscriptions, or kept internal, alongside a \FlinkConnectorConfigWrapper\ that maps Flink connector types (such as \kafka\, \iceberg\, and \jdbc\) to internal table types like \STREAM\ or \VERSIONED\_STATE\. A new \FlinkTableBuilder\ simplifies the construction of Flink table definitions by handling column lists, watermark settings, and partition keys, while \SqrlTableFunction\ and \SqrlFunctionParameter\ provide the core representation for user-defined functions with support for parameters, documentation, and caching durations.

sqrl-planner/src/main/java/com/datasqrl/planner/tables · high confidence

The SQL conversion logic in the planner has been refactored to simplify the addition of new dialects, introducing a unified \AbstractSqlConverters\ base class and a \SqlConvertersFactory\ that discovers implementations via \@AutoService\. This structural change enables the addition of several new shallow query engines: Trino, Redshift, and Spark SQL. The diff also includes new Calcite transformation rules (\SimpleCallTransform\ and \SimplePredicateTransform\) to assist with SQL rewriting, and updates the DuckDB converter to handle nested \TIMESTAMPDIFF\ rewrites.

sqrl-planner/src/main/java/com/datasqrl/calcite/convert · high confidence

The documentation website has been revamped to feature dedicated landing pages for the main product, AI capabilities, and Flink integration, alongside a new Community page. This update introduces reusable React components for the homepage header and feature lists, allowing for consistent styling and easier content management across the new pages. The main index page now highlights the 'Agentic Data Engineering Harness' positioning with updated messaging, code examples, and architectural diagrams, while the AI and Flink pages showcase specific use cases and technical benefits. The community page directs users to GitHub discussions and other support channels.

documentation/src/pages · high confidence

The sqrl-cli Docker image has been rebuilt from a new base structure, upgrading JBang to version 0.141.0 and PostgreSQL to version 18. The container now includes Redpanda and pre-installed DuckDB extensions, and supports S3A IRSA credentials via a new core-site.xml configuration. For users, the entrypoint script now enforces the /workspace mount point (deprecating /build) and ensures file ownership matches the host directory, while PostgreSQL is configured to listen on all addresses to allow external connections.

sqrl-cli · high confidence

The configuration loading logic in the planner has been refactored to replace the previous implementation with a new set of Spring-managed components and interfaces. This change introduces specific implementation classes such as \CompilerApiConfigImpl\, \ConnectorConfImpl\, and \EngineConfigImpl\ to handle compiler, connector, and engine settings respectively. It also adds a \CompilerApiConfigConverter\ to map API configurations and a \GraphqlSourceLoader\ to manage GraphQL schema loading and inference. For users, this represents a structural update to how the system reads and validates configuration files, ensuring that configuration objects are properly instantiated and managed by the Spring context.

sqrl-planner/src/main/java/com/datasqrl/config · high confidence

The SQL dialect implementation has been restructured to simplify the addition of new database targets. A new \BasePostgresSqlDialect\ abstract class centralizes unparse logic and translation dispatching, which is now handled by a \SqlTranslationDispatcher\ that leverages service-loaded \SqlTranslation\ plugins. This foundation supports new or updated dialect implementations: \ExtendedPostgresSqlDialect\ now explicitly maps types (e.g., avoiding JSONB casts for arrays, mapping Flink types to PostgreSQL equivalents) and enforces PostgreSQL-specific SQL conformance rules; \DuckDbSqlDialect\ is introduced as a standalone dialect extending the base; \ExtendedSparkSqlDialect\ and \ExtendedTrinoSqlDialect\ provide specialized type casting and SQL rendering (e.g., handling \SUBSTRING\ vs \SUBSTR\, \APPROX\_COUNT\_DISTINCT\); and \ExtendedSnowflakeSqlDialect\ supports Snowflake-specific operations including \CREATE ICEBERG TABLE\ from object storage. This change improves modularity and ensures accurate SQL generation across PostgreSQL, DuckDB, Spark, Trino, and Snowflake targets.

sqrl-planner/src/main/java/com/datasqrl/calcite/dialect · high confidence

The parser module has been restructured to improve how SQRL scripts are split, parsed, and how errors are reported. A new \SqlScriptStatementSplitter\ now handles script splitting, correctly ignoring statement delimiters inside string literals and preserving block comments while filtering line comments. Error reporting is enhanced via \ParsePosUtil\ and \SqlParserExceptionHandler\, which map Flink/Calcite parser exceptions to precise file locations and clean up error messages. New core types like \ParsedObject\ and \SqrlComments\ track source positions and separate documentation from hints, ensuring that error locations remain accurate even when definitions are transformed or stacked.

sqrl-planner/src/main/java/com/datasqrl/planner/parser · high confidence

The canonicalizer module has been restructured to introduce a new \Name\ interface that distinguishes between a canonical representation (used for internal comparison and equality) and a display representation (preserving the user's original input). This change replaces the previous implementation with new classes such as \AbstractPath\, \NamePath\, \StandardName\, and specific canonicalizers like \LowercaseEnglishCanonicalizer\ and \IdentityCanonicalizer\, allowing for more flexible handling of field names during script parsing and data ingestion.

sqrl-planner/src/main/java/com/datasqrl/canonicalizer · high confidence

The database engine planning model has been restructured to introduce distinct interfaces for database and query engines, allowing for clearer separation of concerns between data persistence and query execution. New interfaces such as DatabaseEngine, QueryEngine, and AnalyticDatabaseEngine define the capabilities and relationships between these components, while CombinedEnginePlan enables the aggregation of multiple engine plans into a single deployment artifact. This change also includes the introduction of DatabaseEngineFactory to standardize engine creation and DatabasePhysicalPlan to represent the output of database-specific planning stages.

sqrl-planner/src/main/java/com/datasqrl/engine/database · high confidence

The execution pipeline has been refactored to support multiple database engines and introduce a new shallow query engine type. The \SimplePipeline\ now explicitly manages stages for LOG, PROCESS (formerly STREAMS), DATABASE, SERVER, and EXPORT engines, allowing for more flexible engine configurations. A key behavioral change is the addition of validation logic: when a SERVER engine is enabled with a JDBC table format database, the system now enforces that a standard query engine is present, rejecting configurations that rely solely on shallow query engines which are not integrated at the database level. This ensures that server-side queries are processed by engines capable of handling the required database interactions.

sqrl-planner/src/main/java/com/datasqrl/engine/pipeline · high confidence

The flexible type system in the planner has been restructured to support a visitor-based architecture and a new set of basic data types. A new \SqrlTypeVisitor\ interface and \Type\ base interface have been introduced, allowing types to be processed via the visitor pattern. The system now includes dedicated classes for basic types such as \BigIntType\, \BooleanType\, \DoubleType\, \IntervalType\, \ObjectType\, \StringType\, and \TimestampType\, each implementing specific conversion logic and type distance calculations. A \BasicTypeManager\ centralizes type registration, name lookup, and combination logic. Legacy classes like \DecimalParseResult\ and \LikePatternType\ have been replaced or renamed to \ArrayType\ and \Type\ respectively, aligning with the new package structure \com.datasqrl.io.schema.flexible.type\.

sqrl-planner/src/main/java/com/datasqrl/io/schema/flexible/type · high confidence

The \sqrl-planner\ module's loader subsystem has been restructured to support individual table imports and exports. This introduces a new \ClasspathFunctionLoader\ to discover standard library functions via Java's \ServiceLoader\, a \ModuleLoaderImpl\ that caches and resolves modules from both the file system and classpath, and a \ScriptSqrlModule\ that lazily initializes \CREATE TABLE\ statements for granular access. The refactoring also adds a \UdfJarClassLoaders\ component to manage user-defined function JARs and ensures these classloaders are properly closed after compilation to prevent resource leaks.

sqrl-planner/src/main/java/com/datasqrl/loaders · high confidence

The packager module has been restructured to introduce a dedicated FilePreprocessingPipeline component that handles copying relevant source files (SQL, GraphQL, config, schema) into the build directory and executing registered preprocessors. The main Packager component now orchestrates this preprocessing, including support for included sub-projects, and explicitly cleans the default Iceberg warehouse directory before building. Additionally, if Flink compilation fails, the system now writes out the Directed Acyclic Graph (DAG) and the generated SQL to log files in the build directory to aid in debugging.

sqrl-cli/src/main/java/com/datasqrl/packager · high confidence

The preprocessing logic in the CLI packager has been restructured into a modular pipeline of Spring-managed components. A new CopyStaticDataPreprocessor now automatically handles .jsonl, .avro, and .csv files, specifically stripping header rows from CSVs to ensure Flink compatibility. JBang-based Java UDFs are now processed by a dedicated JBangPreprocessor that exports fat JARs with class-path support and caches builds for performance. Existing JAR UDFs are handled by a JarPreprocessor, while SqrlPreprocessor manages script templating via Mustache. All UDF manifest generation is centralized in a new UdfManifestPreprocessor base class, and the Preprocessor interface standardizes the processing contract.

sqrl-cli/src/main/java/com/datasqrl/packager/preprocess · high confidence

The schema loading and conversion logic in the planner has been refactored to use a new, extensible architecture. A new \SchemaLoader\ interface and its \SchemaLoaderImpl\ implementation now handle locating schema files via a \ResourceResolver\ and delegating conversion to pluggable \TableSchemaFactory\ instances discovered via ServiceLoader. This is supported by new \SchemaConversionResult\ and \SqrlTypeConverter\ types to standardize the output and conversion process, replacing the previous monolithic schema handling with a modular, factory-based approach.

sqrl-planner/src/main/java/com/datasqrl/io/schema, sqrl-planner/src/main/java/com/datasqrl/loaders/schema · medium confidence

The server module has been restructured to introduce new core classes for managing API sources and generating GraphQL schemas, including \ApiSource\, \ApiSources\, \GraphqlSchemaFactory\, and \GraphqlSchemaHandler\. This refactoring enables opt-in offset-based pagination for GraphQL queries via the \OffsetPageInfo\ metadata type, allowing paginated results to include pagination details like page size and total records. Additionally, the module now supports generating and serving OpenAPI deployment artifacts, with validation against existing specifications to ensure backward compatibility.

sqrl-planner/src/main/java/com/datasqrl/server · high confidence

The server now resolves and logs the actual client IP address from the X-Forwarded-For header in access logs and detailed request traces, rather than only logging the immediate connection address. This ensures accurate client identification when the server is behind a proxy or load balancer.

sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server · high confidence

The planner now ships with a standardized set of default configuration files to simplify setup and ensure consistent behavior. A new \default-package.json\ defines baseline settings for enabled engines (Vert.x, Postgres, Kafka, Flink), compiler options (including the new \LIMITED\_RULES\_NO\_SOURCE\ predicate pushdown rules and API pagination defaults), and connector properties (such as Kafka compression and Iceberg warehouse paths). Run-specific Flink execution details (like checkpointing and local target settings) have been moved to \default-run-package.json\. Additionally, a \log4j2.properties\ file standardizes logging to INFO level on the console, and \saffron.properties\ sets Calcite defaults, while a sample GraphQL schema (\c360-small.graphqls\) is provided for reference.

sqrl-planner/src/main/resources · high confidence

Added GraphQL schema test fixtures for the planner converter; Added JUnit 4 compatibility stubs for testcontainers; Added JWT authorization test suite for DataSQRL; Added analytics-only integration test case with Snowflake and Iceberg support; Added banking-batch integration test resources; Added clickstream integration test suite with paginated GraphQL support; Added integration test for Kafka retention configuration; Added integration test for PostgreSQL index selection edge cases; Added integration test for multi-batch compilation with table imports; Added integration test for stateful temporal joins with TTL; Added integration test resources for sensor and user activity use cases; Added integration tests for Avro schema handling and Java UDFs; Added integration tests for DuckDB function translation; Added integration tests for Flink SQL to PostgreSQL function mappings; Added integration tests for Flink string and utility functions; Added integration tests for Flink-only compilation and Iceberg export workflows; Added integration tests for JWT-authorized GraphQL mutations, subscriptions, and queries; Added integration tests for OpenAI stdlib use cases; Added integration tests for PostgreSQL array handling and GraphQL mapping; Added integration tests for PostgreSQL function translation and row serialization; Added integration tests for Redshift, Spark SQL, and Trino dialects; Added integration tests for TO\_CHANGELOG functionality; Added integration tests for complex mutation scenarios with Iceberg and Kafka; Added integration tests for filtered distinct and ARRAY\_AGG operations; Added integration tests for math and vector standard library functions; Added integration tests for passthrough employee hierarchy queries; Added integration tests for seedshop tutorial connectors and load functions; Added integration tests for server-side function execution; Added integration tests for the sensors-full-compile use case; Added shared banking test fixtures; Added snapshot for ApplicationInfoTest in analytics-only use case; Added snapshot tests for JWT-authorized GraphQL mutations, subscriptions, and queries; Added test case for OAuth authentication with MCP endpoints; Added test case for combined JWT-OAuth metadata access; Added test case for handling null timestamps in event streams; Added test coverage for server-side client address resolution, pagination metadata, and GraphQL integration; Added test coverage for table identity preservation in deferred INSERT statements; Added test data fixtures for book club scenarios; Added test data fixtures for multiple use-case examples; Added test for Avro legacy timestamp mapping in schema inference; Added test for DuckDB CTE materialization with Iceberg scan reuse; Added test for Flink HTTP connector lookup; Added test for LIKE source table validation in mutation database; Added test for distinct materialization with Postgres export; Added test for unauthorized JWT access; Added test infrastructure for integration testing; Added test resources for Avro schema compilation; Added test resources for logging and Vert.x REST configuration; Added test resources for sensors-shared use case; Added test utilities and assertions in sqrl-planner; Added tests for Flink SQL node generation and identity preservation; Added tests for Flink compile error output handling; Added tests for Flink planner conflict resolution, Iceberg configuration, and deployment model serialization; Added tests for GraphQL query metrics, tail sample tracing, and operation-only preparsing; Added tests for GraphQL schema conversion; Added tests for JBangRunner, OsProcessManager, and SqrlInjector utilities; Added tests for JWT configuration merging and serialization; Added tests for Kafka health tracking and case-insensitive JSON fetching; Added tests for PgPartmanExtension DDL generation; Added tests for SQL script statement splitting edge cases; Added tests for SQRL error code descriptions; Added tests for TestOutputManager console log redirection; Added tests for environment variable resolution and SQL table name extraction; Added tests for hint parsing and validation logic; Added tests for index selection logic and sort order handling; Added tests for module loading and UDF classloader lifecycle; Added tests for nullable GraphQL API arguments; Added tests for protocol-specific API exposure and configuration; Added tests for relational planner components; Added tests for relationship field preservation in complex queries; Added tests for schema evolution type compatibility; Added tests for schema type handling and combination logic; Added tests for server configuration template serialization; Added unit tests for Avro schema factory and type conversion; Added unit tests for PostgreSQL DDL generation and notification triggers; Added unit tests for SQRL CLI commands and utilities; Added unit tests for SQRL configuration and compiler components; Added unit tests for preprocessors; Added unit tests for server configuration and GraphQL parser settings; Expanded container integration test coverage for server features; New JUnit 5-based integration test infrastructure for compiler and use-case validation; Removed TPC-H and scratch SQL test resources; Removed unused example parsing test.

The documentation site now requires Node.js 22 or higher and has upgraded its React and React-DOM dependencies to version 19.2.8. Additionally, the Docusaurus tooling has been updated to version 3.10.2, and the local search plugin has been bumped to 0.55.3.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 62 → 66 (+3.8)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 81 → 88 (+6.8)
• Architecture 100 → 98 (-2.3)
• Maturity 75 → 75 (-0.2)
• Readiness 59 → 58 (-0.8)
• Security 54 → 64 (+10.0)

• Boundary-crossing change coupling: CompilationProcess.java ↔ GraphqlSchemaHandler.java (sqrl-cli/src/main/java/com/datasqrl/compile/CompilationProcess.java)
• Change coupling: MutationConfigurationImpl.java ↔ SubscriptionConfigurationImpl.java (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/MutationConfigurationImpl.java)
• Change coupling: VertxJdbcClient.java ↔ VertxQueryExecutionContext.java (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/jdbc/VertxJdbcClient.java)
• Change coupling: docusaurus.config.ts ↔ index.tsx (documentation/docusaurus.config.ts)
• Coverage not included — suite not readable by the collector
• Critical CVE: [GHSA redacted] (documentation/package-lock.json)
• Critical CVE: [GHSA redacted] (documentation/package-lock.json)
• DatasqrlTest.run (cognitive 19) (sqrl-cli/src/main/java/com/datasqrl/cli/DatasqrlTest.java)
• Dependency hygiene not measured — dependency manifest found but not parsed for hygiene
• Duplicated block (10 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/planner/util/CompiledPlanCondenser.java)
• Duplicated block (11 lines × 2) (sqrl-cli/src/main/java/com/datasqrl/cli/ExecCmd.java)
• Duplicated block (12 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/auth/OAuth2AuthFactory.java)
• Duplicated block (13 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/modules/ApiDeploymentModule.java)
• Duplicated block (14 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/auth/JwtFailureHandler.java)
• Duplicated block (16 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/swagger/SwaggerService.java)
• Duplicated block (6 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/builtinflink/LogSqlTranslation.java)
• Duplicated block (7 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/function/translation/PostgresLikeTranslations.java)
• Duplicated block (7 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/function/translation/duckdb/builtin/ElementSqlTranslation.java)
• Duplicated block (7 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/graphql/GraphQLQueryMetricsInstrumentation.java)
• Duplicated block (7 lines × 4) (sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/builtinflink/FromBase64SqlTranslation.java)
• …and 64 more

• CI installs an unverified third-party binary (.circleci/config.yml)
• ClassTooLong: CalciteUtil (sqrl-planner/src/main/java/com/datasqrl/util/CalciteUtil.java)
• ClassTooLong: DAGPlanner (sqrl-planner/src/main/java/com/datasqrl/planner/dag/DAGPlanner.java)
• ClassTooLong: FlinkSqlNodes (sqrl-planner/src/main/java/com/datasqrl/engine/stream/flink/FlinkSqlNodes.java)
• ClassTooLong: GraphQLSchemaConverter (sqrl-planner/src/main/java/com/datasqrl/server/converter/GraphQLSchemaConverter.java)
• ClassTooLong: McpBridgeVerticle (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/McpBridgeVerticle.java)
• ClassTooLong: SQRLLogicalPlanAnalyzer (sqrl-planner/src/main/java/com/datasqrl/planner/analyzer/SQRLLogicalPlanAnalyzer.java)
• ClassTooLong: SqlScriptPlanner (sqrl-planner/src/main/java/com/datasqrl/planner/SqlScriptPlanner.java)
• ClassTooLong: Sqrl2FlinkSQLTranslator (sqrl-planner/src/main/java/com/datasqrl/planner/Sqrl2FlinkSQLTranslator.java)
• Critical CVE: [CVE redacted] (documentation/package-lock.json)
• Critical CVE: [CVE redacted] (documentation/package-lock.json)
• Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
• Duplicated block (10 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/json/JsonArrayAggSqlTranslation.java)
• Duplicated block (10 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/json/JsonArraySqlTranslation.java)
• Duplicated block (10 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/function/translation/postgres/json/JsonObjectSqlTranslation.java)
• Duplicated block (10 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/util/BaseFileUtil.java)
• Duplicated block (11 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/auth/OAuth2AuthFactory.java)
• Duplicated block (11 lines × 2) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/kafka/JsonDeserializer.java)
• Duplicated block (11 lines × 3) (sqrl-server/sqrl-server-vertx-base/src/main/java/com/datasqrl/server/GraphQLServerVerticle.java)
• Duplicated block (12 lines × 2) (sqrl-planner/src/main/java/com/datasqrl/engine/stream/flink/FlinkSqlNodes.java)
• …and 187 more

• 118 commits — 98 feature/other, 20 fixes

• (root) — 48 commits
• sqrl-testing/sqrl-testing-integration — 24 commits
• sqrl-planner/src — 18 commits
• documentation/docs — 10 commits
• sqrl-cli/src — 6 commits
• sqrl-server/sqrl-server-vertx-base — 4 commits
• documentation/package-lock.json — 2 commits
• .agents/skills — 1 commit
• .circleci/config.yml — 1 commit
• .github/workflows — 1 commit
• documentation/blog — 1 commit
• documentation/static — 1 commit
• sqrl-testing/sqrl-testing-container — 1 commit

• fix: ci: Fix JaCoCo reports for containered tests (#1794)
• fix: ci: Fix minio image tag (#2377)
• fix: fix!: Enforce a single ROWTIME for SqrlDefinitions (#2268)
• fix: fix: Add INSERT statements to DAG planning (#2278)
• fix: fix: Allow NOW() predicates on Flink INSERT INTO stages (#2315)
• fix: fix: Analyze process table function inputs (#2293)
• fix: fix: Flaky banking-batch use-case test (#2286)
• fix: fix: INSERT analysis should not mutate query (#2364)
• fix: fix: Ignore internal aliases during RelNode normalization (#2332)
• fix: fix: Include partition key in hashed primary keys (#2382)
• fix: fix: Make index selection independent of the number of queries on a table (#2318)
• fix: fix: PostgreSQL relation alias serialization (#2383)
• fix: fix: Preserve distinct state for materialized stream consumers (#2297)
• fix: fix: Preserve hinted view identity during collapse (#2334)
• fix: fix: Preserve table identity in deferred INSERT statements (#2366)
• fix: fix: Resolve Flink insert conflicts after changelog inference (#2319)
• fix: fix: Resolve LIKE source table across databases in mutation database validation (#2292)
• fix: fix: Resolve conflicts for mismatched upsert keys properly (#2329)
• fix: fix: Restrict custom build folders to build/ subdirectories (#2314)
• fix: fix: Update unit test snapshots to latest source state
• …and 98 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

DataSQRL/sqrl was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.