lensesio/stream-reactor
64.2
Adequate · 14 November 2026
73.6k
lines of production code
Scala
with Java
5
measurements over time
What this system is
Features
Add Azure Event Hubs source connector implementation
Introduces the core source connector component for Azure Event Hubs, specifically the BlockingQueueProducerProvider which manages the creation and configuration of Kafka consumers to ingest data from Event Hubs into Kafka topics. This enables users to stream events from Azure Event Hubs as a data source within the Kafka Connect ecosystem.
kafka-connect-azure-eventhubs · high confidence
Added Field class to support KCQL field parsing and hierarchy
The Kafka Connect Query Language (KCQL) module now includes a new Field class that models individual fields within a query. This class supports field names, aliases, data types, and parent field hierarchies, enabling more complex field resolution. It includes logic to parse special syntax prefixes (like underscore for topic, offset, partition, timestamp, and key fields) and provides methods for equality checking, hashing, and string representation of nested field paths.
kafka-connect-query-language · high confidence
Azure CosmosDB sink connector introduced with bulk mode, proxy support, and configurable throughput
The Azure CosmosDB sink connector is now available, allowing Kafka Connect to stream data into Azure CosmosDB. This implementation supports bulk submission to reduce API chatter, configurable consistency levels, and proxy settings for secure network environments. Users can enable automatic database and collection creation, with the option to provision new collections at a specific manual throughput (defaulting to 400 RU/s). The connector also includes robust error handling policies, retry mechanisms, and configurable flush settings based on size, interval, or count.
kafka-connect-azure-cosmosdb/src/main · high confidence
Azure Data Lake Storage Gen2 sink connector introduced
A new sink connector for Azure Data Lake Storage Gen2 is now available, allowing users to stream Kafka data into ADLS. The connector supports three authentication modes—Default Azure Credential, Shared Key (account name and key), and Connection String—and provides configuration for HTTP timeouts, connection pooling, and error handling policies. It also includes tunable upload performance settings (parallelism, block size, and single-upload size thresholds) and features such as schema change rollover, null record skipping, and configurable local staging areas.
kafka-connect-azure-datalake/src/main · high confidence
Centralized configuration key definitions for cloud connectors
A new \PropsKeyEnum\ has been introduced in the \kafka-connect-cloud-common\ module to serve as the single source of truth for KCQL configuration property keys. This enum defines constants for text reading options (such as regex, start/end tags, and line ranges), envelope storage settings (including key, headers, value, and metadata storage), file naming parameters, flush controls, and post-processing actions (like copy, move, delete, and late arrival handling). By centralizing these keys, the connector ensures consistent configuration handling across source and sink components.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/config/kcqlprops · high confidence
Configurable HTTP retries and timeouts for GCP connectors
GCP connectors now allow users to configure HTTP retry behavior and socket/connection timeouts. The new configuration properties include the number of retries, retry interval, and backoff multiplier, along with optional socket and connection timeout settings in milliseconds. These settings are applied to the underlying GCP service clients to improve reliability and control over network interactions.
java-connectors/kafka-connect-gcp-common/src/main · high confidence
Configurable padding for sink file names
The cloud sink connectors now support configurable padding for the offset and partition fields in output file names. Users can choose between left-padding, right-padding, or no padding via the \padding.strategy\ configuration property, and specify the padding length and character. This ensures consistent file ordering and naming conventions when writing data to cloud storage.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/config/padding · high confidence
Customizable cloud sink object key naming with template validation
Users can now define the file names and paths for cloud storage sinks using a configurable template via the \object.key.template\ property. This new \TemplateFileNamer\ supports placeholders for topic, partition, offsets, timestamps, record count, and file extension, allowing for dynamic, user-defined naming schemes. To prevent data loss from silent overwrites, the system enforces that templates must include at least one flush-varying placeholder (such as \{start-offset}\ or \{record-count}\) and rejects any unknown placeholders at configuration time. This logic is implemented in the \kafka-connect-cloud-common\ module, which also introduces the \CloudKeyNamer\ and \KeyNamer\ interfaces to support this flexible naming strategy across S3, GCS, and Azure sinks.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/naming · high confidence
Initial GCP Pub/Sub Source Connector implementation
Adds a new GCP Pub/Sub Source Connector for Kafka Connect, enabling users to ingest data from Google Cloud Pub/Sub topics into Kafka. This change introduces the core configuration definition (\PubSubConfigSettings\) to manage connector settings and a \LooselyBoundedQueue\ utility to handle message buffering with flexible size limits, ensuring the connector remains resilient by logging warnings rather than failing when queue capacity is temporarily exceeded.
kafka-connect-gcp-pubsub · high confidence
Initial implementation of GCP Storage Source and Sink connectors
This change introduces the GCP Storage connector, enabling users to ingest data from Google Cloud Storage buckets into Kafka (Source) and write Kafka data to GCS buckets (Sink). The implementation includes the connector and task classes for both directions, along with configuration definitions for error policies, retries, compression, and schema handling. It also provides GCP-specific authentication client creation, bucket name validation, and a directory lister for source polling.
kafka-connect-gcp-storage/src/main · high confidence
Initial implementation of the MQTT Sink connector
This change introduces the core components for the MQTT Sink connector, enabling users to publish Kafka records to an MQTT broker. The new MqttSinkConnector and MqttSinkTask classes handle configuration and task lifecycle, while the MqttWriter class manages the actual message publishing. Key capabilities include dynamic topic targeting (using record keys, topics, or nested struct/map fields), automatic conversion of various data types (Structs, Maps, Collections) to JSON payloads, and support for configurable QoS and retained messages.
kafka-connect-mqtt/src/main/scala/io/lenses/streamreactor/connect/mqtt/sink · high confidence
Initial release of Azure Service Bus Source and Sink connectors
This change introduces the new Azure Service Bus connectors for Kafka Connect, enabling data ingestion from and export to Azure Service Bus. The sink connector (ServiceBusSenderFacade) supports both batched and sequential message sending, with explicit offset commit logic to ensure reliable delivery tracking. The source connector (TaskToReceiverBridge, RetryMessageAck) provides a bridge between Service Bus receivers and the connector task, featuring automatic retry logic for message acknowledgment to handle transient failures during consumption.
kafka-connect-azure-servicebus · high confidence
Initial release of the Kafka Connect Query Language (KCQL) parser and model
This change introduces the core components for the Kafka Connect Query Language, including the ANTLR4 grammar files (ConnectorLexer.g4, ConnectorParser.g4) and the Java model classes (Kcql, Field, FieldType, etc.) that represent parsed queries. It enables users to define data movement and transformation rules using a SQL-like syntax, supporting features such as INSERT/UPSERT modes, field selection and aliasing, partitioning strategies (including NOPARTITION), and target type parsing (key, value, header). The entry also includes example CQL files demonstrating basic usage patterns like ignoring fields and renaming columns.
java-connectors/kafka-connect-query-language/src/main · high confidence
Integrate Graft context graph into Claude and Cursor workflows
This change adds configuration files to integrate the 'Graft' tool into the development environment. For Claude Code, it installs helper scripts and settings that trigger Graft hooks on specific actions (like edits or bash commands) and update the status line, while also granting permissions for Graft-related bash commands. For Cursor, it configures the Graft MCP server and adds a rule file that instructs the AI to use the Graft context graph for understanding code and finding file spans before performing tasks.
.claude, .cursor · high confidence
Introduce Azure Event Hubs source connector
Adds a new Kafka Connect source connector for Microsoft Azure Event Hubs. The connector reads events from Event Hubs and writes them to Kafka topics, supporting configurable KCQL mappings for field selection and data routing. It includes configuration for consumer offset behavior (seeking to earliest or latest), a 30-second close timeout, and validates input/output topic names against a specific pattern. The implementation supports exactly-once processing semantics and handles consumer rebalancing to ensure correct offset management.
java-connectors/kafka-connect-azure-eventhubs · high confidence
Introduce Azure Service Bus Kafka Connect connector
Adds a new Azure Service Bus connector for Kafka Connect, providing both source and sink capabilities. The sink component maps Kafka records to Service Bus messages, supporting configurable retry policies, batch sending, and KCQL-based routing, while the source component ingests Service Bus messages into Kafka topics with configurable prefetch, backoff, and polling behavior.
java-connectors/kafka-connect-azure-servicebus/src/main · high confidence
Introduce BigQuery Sink Connector with batch loading and upsert capabilities
The BigQuery sink connector is now available in the java-connectors/kcbq-connector module, enabling users to stream Kafka data into Google BigQuery. This connector supports standard row-based writes as well as high-throughput batch loading via Google Cloud Storage (GCS), configurable through the \enableBatchLoad\ and \gcsBucketName\ settings. It also introduces support for upserts and deletes, allowing existing records to be updated or removed based on a configured Kafka key field. The connector handles schema evolution automatically, manages GCP authentication via JSON keys or application defaults, and routes failed records to a dead-letter queue for inspection.
java-connectors/kcbq-connector · high confidence
Introduce Elastic 6 sink connector with strict error handling and HTTP basic auth
The Elastic 6 sink connector is now available, providing a dedicated implementation for Elasticsearch 6 clusters. This connector supports HTTP basic authentication and SSL/TLS via key/trust stores, and enforces document types in index, update, and delete operations (defaulting to the index name when not explicitly configured). Per-item bulk errors are surfaced by default (strict mode), allowing the connector to distinguish between 429 rate-limit errors and mapper errors; a tolerant mode is available to log and drop these errors instead. The connector also uses a 5-minute write timeout for bulk operations and index creation.
kafka-connect-elastic6/src/main/scala/io/lenses/streamreactor/connect/elastic6 · high confidence
Introduce FTP source connector with configurable file monitoring and record conversion
The FTP source connector now supports two file-monitoring modes—Tail (tracking file position) and Update (tracking whole-file changes)—configured via \connect.ftp.monitor.tail\ and \connect.ftp.monitor.update\. Output keys can be set to either a simple string (filename) or a structured key (filename plus offset) using \connect.ftp.keystyle\. The connector includes built-in converters: SimpleFileConverter (writes the entire file into a single record) and MaxLinesFileConverter (writes up to the last line separator in each slice), with a default NopSourceRecordConverter that passes records through unchanged.
kafka-connect-ftp/src/main/scala · high confidence
Introduce GCP Pub/Sub Source Connector
Adds a new GCP Pub/Sub Source Connector that ingests messages from Google Cloud Pub/Sub subscriptions into Kafka topics. The connector supports configurable output modes (DEFAULT and COMPATIBILITY) to control how message keys, values, and headers are mapped to Kafka Connect records, and includes validation to prevent configuration conflicts when multiple KCQL statements reference the same subscription.
java-connectors/kafka-connect-gcp-pubsub/src/main · high confidence
Introduce HTTP Sink connector for Kafka Connect
Adds a new HTTP Sink connector that sends Kafka records to an HTTP endpoint. The connector supports configurable batching (by count, size, or time interval), multiple HTTP methods (POST, PUT, PATCH), and authentication modes including Basic and OAuth2. It includes built-in retry logic with fixed or exponential backoff, backpressure via a semaphore-based queue, and metrics reporting for both successes and failures.
kafka-connect-http/src/main · high confidence
Introduce OpenSearch Sink Connector with secure authentication and strict error handling
Adds a new Kafka Connect sink connector for OpenSearch 2.x, providing a dedicated implementation (KOpenSearchClient) that validates cluster compatibility at startup to prevent misconfiguration against Elasticsearch or unsupported 1.x clusters. The connector supports AWS SigV4 signing and JWT Bearer token authentication (with file-backed rotation and path-traversal guards), while enforcing a cleartext-auth guard by default to prevent credential interception. It also introduces a strict bulk item error mode (enabled by default) that surfaces per-item failures to the error policy, replacing the previous tolerant behavior that silently dropped failed documents.
kafka-connect-opensearch/src/main · high confidence
Introduce SQL-based record transformation in Kafka Connect
Added a new SQL transformation capability to the Kafka Connect SQL common module, allowing users to filter, project, and flatten data in message keys and values using SQL queries. This change introduces the \Transformation\ class, which reads SQL configuration per topic, and a suite of supporting components including \Sql\ for parsing MySQL-compatible queries via Apache Calcite, \StructSql\ for applying transformations to Kafka Connect Structs, \Transform\ for handling various payload types (JSON, Struct, Bytes), and \FieldValueGetter\ for navigating nested schema paths. Users can now configure SQL statements to reshape incoming and outgoing records directly within the connector configuration.
kafka-connect-sql-common/src/main/scala/io/lenses/connect · high confidence
Introduce configurable Kafka-based sink reporting
The sink-reporting module now supports sending connector success and error reports to a configurable Kafka topic. This change introduces a new reporting pipeline (ReportHolder, ReportSender, ReportingController) that uses a dedicated Kafka producer, allowing users to enable reporting and specify the target topic, partition, and security settings (SASL/SSL) via connector configuration prefixes (connect.reporting.error.config. and connect.reporting.success.config.). The implementation ensures reliable lifecycle management by properly shutting down the reporting executor and producer on connector close to prevent thread leaks.
java-connectors/kafka-connect-sink-reporting/src/main/java/io/lenses/streamreactor/connect/reporting · high confidence
Introduce new Cassandra sink connector and cloud sink common infrastructure
This change adds a new Cassandra sink connector module, including its settings, connection handling, and SSL/authentication configuration, alongside a significant refactoring of the cloud sink architecture. The cloud sink common module now introduces a new \CloudSinkTask\ base class, a \ConsumerGroupsWriter\ for persisting Kafka consumer offsets to cloud storage, and new model classes for compression codecs, cloud locations, and uploadable files. These updates provide the foundational components for the new Cassandra sink and enhance the existing cloud sinks with better offset management and configuration structures.
repository · high confidence
Introduce shared elastic-common module for Elastic and OpenSearch connectors
A new \kafka-connect-elastic-common\ library has been added to centralize shared logic for the Elastic and OpenSearch sink connectors. This module provides core utilities for transforming Kafka Connect data into JSON (\JsonPayloadExtractor\, \Transform\), extracting primary keys from values, keys, and headers (\PrimaryKeyExtractor\, \TransformAndExtractPK\), and managing bulk operations (\KBulkClient\, \BulkItemErrorClassifier\). It also standardizes configuration handling (\ElasticCommonConfigDef\, \ElasticCommonSettings\) and defines common constants, enabling consistent behavior and easier maintenance across both connector implementations. Also in this area: 4 commits (3 fixes) fixing kafka-connect-elastic-common; 8 commits (0 fixes) modifying kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/writer.
kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/config, kafka-connect-elastic-common/src/test/scala/io/lenses/streamreactor/connect/elastic/common, kafka-connect-elastic-common/src/test/scala/io/lenses/streamreactor/connect/elastic/common/config · medium confidence · unverified
Introduces IndexManager trait and NoIndexManager for cloud sink indexing
The cloud sink connectors now use a new \IndexManager\ trait in the common module to manage indexing operations, including opening topic partitions, updating offsets, and retrieving seeked offsets. This abstraction supports granular locking for exactly-once semantics with PARTITIONBY and batch deduplication floors. A \NoIndexManager\ implementation is provided to disable indexing when not required, allowing connectors to operate without the overhead of index management.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/seek · high confidence
Introduces new SQL-focused configuration and conversion utilities
This change adds a new set of Scala classes and traits to the \kafka-connect-sql-common\ module to support SQL-based data projection and conversion. Specifically, it introduces \KcqlWithFieldsSettings\ and \Projections\ to manage KCQL settings such as field mappings, primary keys, and write modes. It also adds several converter utilities, including \MsgKey\ for handling message keys, \AvroConverter\ and \BytesConverter\ for sink record transformations, and helper classes like \SinkRecordConverterHelper\ and \StructHelper\ for extracting and reducing schema fields. These components provide the foundational logic for processing and converting data within the SQL connector context.
kafka-connect-sql-common/src/main/scala/io/lenses/streamreactor/common · high confidence
Introduction of generic ClientCreator trait for authentication
A new \ClientCreator\ trait has been added to the common authentication module, providing a generic interface for creating client instances from configuration. This abstraction allows for standardized client creation logic across different cloud sources, facilitating the reuse of authentication components and enabling the integration of additional data sources.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/auth · high confidence
MongoDB sink connector implementation
The MongoDB sink connector is now available, enabling Kafka Connect to write records to MongoDB. This change introduces the core connector components, including MongoSinkConnector and MongoSinkTask for lifecycle management, MongoWriter for handling bulk insert and upsert operations, and utilities like KeysExtractor and SinkRecordToDocument for converting Kafka Connect schema and data structures into MongoDB-compatible documents.
kafka-connect-mongodb/src/main/scala/io/lenses/streamreactor/connect/mongodb/sink · high confidence
New MQTT Source Connector implementation
This change introduces the core Scala implementation for the MQTT source connector, adding the MqttSourceConnector, MqttSourceTask, MqttManager, and MqttSSLSocketFactory classes. The MqttSourceTask handles configuration validation (including SSL certificate file existence checks) and instantiates the MqttManager to manage the MQTT client connection. The MqttManager is responsible for subscribing to MQTT topics, matching incoming messages to configured KCQL statements (supporting wildcards and regex), converting payloads via registered converters, and queuing SourceRecords for Kafka Connect. The MqttSSLSocketFactory provides SSL/TLS support using BouncyCastle for loading CA, client certificates, and private keys. The MqttSourceConnector manages task configuration splitting, including logic to replicate shared subscription topics across all tasks if configured.
kafka-connect-mqtt/src/main/scala/io/lenses/streamreactor/connect/mqtt/source · high confidence
New SQL-based JSON transformation capabilities in Kafka Connect SQL Common
The \kafka-connect-sql-common\ module now includes a new \io.lenses.json.sql\ package containing \JacksonJson\ and \JsonSql\ utilities. \JacksonJson\ provides a configured Jackson \JsonMapper\ that uses \NON\_EMPTY\ inclusion to exclude empty values during serialization. \JsonSql\ introduces extension methods on Jackson \JsonNode\ objects, allowing users to execute SQL \SELECT\ queries (parsed via Apache Calcite) against JSON data. This includes support for flattening JSON structures into a single-level object and handling specific field selections, enabling SQL-style transformations directly on JSON payloads within the connector framework.
kafka-connect-sql-common/src/main/scala/io/lenses/json · high confidence
New cloud source task infrastructure with late-arrival handling and partition discovery
The \kafka-connect-cloud-common\ module now includes the core \CloudSourceTask\ implementation, introducing a background \LateArrivalTouchTask\ that periodically updates file timestamps to ensure late-arriving data is processed, alongside a new \CloudPartitionSearcher\ for discovering new partitions in cloud storage. This change also adds support for writing watermark information (partition and offset) to Kafka headers via a new configuration flag, and introduces a unified \SourceWatermark\ system for managing source offsets and context.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/source · high confidence
New common cloud sink format writers and schema change detection
The \kafka-connect-cloud-common\ module now includes a new set of format writers (Avro, Parquet, JSON, CSV, Text, and Bytes) that handle writing Kafka records to cloud storage. These writers support configurable compression codecs (such as Snappy, GZIP, ZSTD, and Brotli) and introduce schema change detection strategies—allowing users to choose between version-based or compatibility-based detection to determine when to roll over output files.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/formats/writer · high confidence
New common configuration, security, and utility infrastructure for connectors
This change introduces a foundational set of classes in the Kafka Connect common library to standardize how connectors handle configuration, security, and operational utilities. It adds a new \ConfigSource\ abstraction and \BaseConfig\ base class to decouple configuration parsing from specific connector implementations, alongside \KcqlSettings\ for unified KCQL parsing and \RetryConfig\ for standardized retry behavior. Security is enhanced with new \StoresInfo\, \KeyStoreInfo\, and \TrustStoreInfo\ classes that manage SSL/TLS context initialization and keystore loading. Additionally, utility classes like \TopicPartitionOffsetAndMetadataStorage\ for offset management and \EitherUtils\ for functional error handling are added to support these new patterns.
java-connectors/kafka-connect-common/src/main · high confidence
New common utility libraries for Kafka Connect Cassandra
This change introduces a set of new shared Scala utilities within the \kafka-connect-cassandra\ connector to standardize common operations. Users benefit from improved concurrency handling via \ExecutorExtension\ and \FutureAwaitWithFailFastFn\, which provide safer ways to execute tasks and wait for futures with fail-fast semantics. Configuration is streamlined with \ThreadPoolSettings\ for automatic thread pool sizing and \SSLConfigContext\ (now deprecated in favor of \StoresInfo\) for managing SSL contexts. Additionally, \OffsetHandler\ simplifies offset recovery for source tasks, and \QueueHelpers\ offers utilities for draining queues in batches.
kafka-connect-cassandra/src/main/scala/io/lenses/streamreactor/common · high confidence
New common validation and file utility components for cloud storage locations
The \kafka-connect-cloud-common\ module now includes a \CloudLocationValidator\ trait for validating cloud storage locations and a \FileUtils\ object to handle file system operations. \FileUtils\ provides a configurable staging write buffer size (defaulting to 64 KB) and utilities for creating files and their parent directories, supporting consistent file handling across cloud connectors.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/model · high confidence
New configuration schema for cloud source connectors
The cloud source connectors now support a structured configuration schema for post-processing actions (Delete or Move), text reading modes (Regex, Start/End Tag, or Start/End Line), and late-arrival watermarking. This change introduces the \CloudSourcePropsSchema\ and associated enumerations in the \kafka-connect-cloud-common\ module, enabling users to define these behaviors via Kcql properties for more granular control over file processing and data ingestion.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/source/config/kcqlprops · high confidence
New converters for byte, text, and envelope data formats
Added five new converter implementations in the cloud-common module to handle diverse input formats for cloud source connectors. BytesOutputRowConverter and TextConverter now support raw byte and string payloads respectively, while SchemaAndValueConverter handles structured schema-and-value pairs. Additionally, SchemaAndValueEnvelopeConverter and SchemalessEnvelopeConverter enable ingestion of complex envelope structures (containing key, value, headers, and metadata) for both schema-aware and schemaless (JSON) data, with support for extracting metadata fields like timestamp and partition from the payload.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/formats/reader/converters · high confidence
New file-source batching and queueing components in cloud-common
The \kafka-connect-cloud-common\ module now includes a new set of classes and traits for managing file ingestion batches: \BatchLister\ (with \DefaultOrderingBatchLister\ for standard UTF-8 binary ordering and \DateOrderingBatchLister\ for sorting by file modification date), \CloudSourceFileQueue\ (which handles the queueing logic and tracks the last seen file for resumption), and supporting types like \SourceFileQueue\. These components provide the underlying mechanism for listing and ordering files from cloud storage before they are processed by connectors.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/source/files · high confidence
New local repro harness for GCS sink time-bucket partitioning and local smoke test for OpenSearch sink
Added a new \dev-scripts/gcs-local\ harness that spins up a local Kafka and Connect stack to reproduce and demonstrate how fine-grained \HHmm\ partition keys cause directory proliferation in the GCS sink, and how applying a 15-minute rolling window via Lenses SMTs collapses these into manageable buckets. The harness includes three connector configurations (\gcs-bad\, \gcs-good\, \gcs-rolling\) and scripts to build, deploy, produce data, and verify the resulting GCS directory structure. Additionally, added a local smoke test environment for the OpenSearch sink connector (\dev-scripts/opensearch-local\) that sets up a local Kafka, Connect, and OpenSearch stack with TLS and basic auth to exercise the connector end-to-end.
dev-scripts · high confidence
New schema-aware and schemaless envelope transformers for cloud sinks
The cloud sink connector now includes a new set of message transformers in the common module to handle envelope creation for different data formats. For JSON sinks, a \SchemalessEnvelopeTransformer\ wraps messages into a map-based structure, while for Avro and Parquet sinks, an \AddConnectSchemaTransformer\ and \EnvelopeWithSchemaTransformer\ pair ensures that payloads without native Connect schemas (such as arrays of maps) are enriched with proper schema definitions before being wrapped in an envelope containing key, value, headers, and metadata. This change ensures consistent envelope structures across all supported sink formats.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/transformers · high confidence
New source record converters for Avro and JSON payloads
This change introduces a new set of source record converters in the \kafka-connect-sql-common\ module, providing dedicated implementations for Avro (\AvroConverter\) and various JSON formats (\JsonSimpleConverter\, \JsonOptNullConverter\, \JsonPassThroughConverter\, \JsonResilientConverter\, and \JsonConverterWithSchemaEvolution\). These converters implement a common \Converter\ trait to transform raw byte payloads into Kafka Connect \SourceRecord\ objects, handling schema resolution, key extraction, and data type mapping. The addition of \JsonResilientConverter\ specifically improves robustness by ignoring malformed JSON messages rather than failing, while \JsonConverterWithSchemaEvolution\ supports dynamic schema updates. A \KeyExtractor\ utility is also included to support extracting specific fields from JSON nodes and Connect structs for use as record keys.
kafka-connect-sql-common/src/main/scala/io/lenses/streamreactor/connect · high confidence
New test infrastructure for OpenSearch security and various service containers
The test-common module now includes comprehensive support for testing OpenSearch with security enabled, featuring PKI certificate generation, TLS configuration, and security plugin setup (basic auth, client-cert, and JWT). Additionally, new test container wrappers have been added for Azurite, Cassandra, Cosmos DB, GCP Storage, MongoDB, and Redis, along with model classes for Order and blockchain transactions to support integration testing across these services.
test-common · high confidence
New validation, batching, and configuration infrastructure in kafka-connect-common
This change introduces a new validation framework for KCQL configurations, including a \ConnectorConfigKcqlValidator\ and pattern-matching validators for source and target names, allowing connectors to enforce specific naming rules. It also adds a new batching engine (\RecordsQueueBatcher\) that optimizes batch evaluation by reusing a mutable context to avoid allocations on the hot path, alongside utility classes for offset merging and batch record handling. Additionally, the module provides a standardized set of configuration traits (e.g., \BaseConfig\, \KcqlSettings\, \ErrorPolicySettings\, \SSLSettings\) to simplify connector configuration, and introduces new error handling types (\FatalConnectException\, \RetriableIntegrityException\) to better manage data integrity and retry behaviors.
kafka-connect-common · high confidence
Project initialization and repository configuration
The repository has been initialized with essential configuration files and documentation. This includes a \.gitignore\ to manage build artifacts and IDE files, a \.jvmopts\ file to set JVM memory and stack size limits, and build tool configurations for Scala formatting (\.scalafmt.conf\) and import organization (\.scalafix.conf\). The project also introduces a \.mcp.json\ file to configure the 'graft' MCP server, a \suppression.xml\ to handle false-positive security warnings from the OWASP Dependency Check tool, and a \WORKFLOW.md\ guide explaining the GitHub Actions module-based CI/CD pipeline. Additionally, the \README.md\ has been updated to reflect the Lenses Connectors branding, support policies, and build instructions.
(repo-wide) · high confidence
Streaming Parquet input support for cloud storage
The \kafka-connect-cloud-common\ module now includes a new streaming Parquet reader implementation (\ParquetSeekableInputStream\ and \ParquetStreamingInputFile\) that allows Parquet files to be read from cloud storage without loading the entire file into memory. This change enables more efficient processing of large Parquet datasets by supporting chunked reads and stream recreation, addressing previous limitations in handling large files via S3.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/formats/reader/parquet · high confidence
Architecture
Establishes Java connectors build infrastructure and code style standards
The java-connectors module is initialized with a Gradle build system (version 8.14) and comprehensive code quality configurations. This includes standardized code formatting rules for IntelliJ IDEA and Eclipse, Checkstyle linting rules based on Google Java Style, and license header templates for source files. These changes provide the necessary tooling and style guidelines for developing and maintaining Java-based connectors within the project.
java-connectors · high confidence
Behavioural changes
15 commits (4 fixes) modifying kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/bulk
A change to existing behaviour in kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/bulk — 15 commits (4 fixs), 2 files: “fix(elastic): exclude batch-wide HTTP 400s from item-error diversion”; “Apply the linter”; “Enhance error handling in JsonBulkWriter and BulkItemErrorClassifier”; and 11 more.
kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/bulk · medium confidence · unverified
16 commits (3 fixes) modifying kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7
A change to existing behaviour in kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7 — 16 commits (3 fixs), 4 files: “Apply the 2025 header (\#188)”; “Enabling dynamic index names based on \_key, \_value and \_header (\#83)”; “scalafmt changes”; and 13 more (refs \#188, \#83, \#1019, \#172, \#1029, …).
kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7 · medium confidence · unverified
3 commits (1 fix) modifying kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common
A change to existing behaviour in kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common — 3 commits (1 fix), 6 files: “fix(elastic): case-insensitive null-value behavior, nested IGNORE path traversal, Long retry default”; “Ensure a message for each the fields cannot be extracted, or the values type don't match is considered a poision message and gets redirected rather than failing the task”; “feat(elastic, opensearch): add OpenSearch sink connector and extract shared elastic-common module”.
kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common · medium confidence · unverified
8 commits (0 fixes) modifying kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/sink
A change to existing behaviour in kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/sink — 8 commits, 2 files: “Ensure a message for each the fields cannot be extracted, or the values type don't match is considered a poision message and gets redirected rather than failing the task”; “Apply the linter”; “Refactor error handling and documentation in BulkItemErrorClassifier and related components”; and 4 more. Also in this area: 4 commits (3 fixes) fixing kafka-connect-elastic-common; Introduce shared elastic-common module for Elastic and OpenSearch connectors.
kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/sink, kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/writer · medium confidence · unverified
9 commits (4 fixes) modifying kafka-connect-elastic7/src/it
A change to existing behaviour in kafka-connect-elastic7/src/it — 9 commits (4 fixs), 5 files: “Apply the linter”; “fix(elastic,opensearch): close adversarial review findings in item-error diversion”; “fix(elastic): surface per-item bulk errors instead of dropping them”; and 6 more (refs \#1029, \#1004).
kafka-connect-elastic7/src/it · medium confidence · unverified
AWS S3 Connector configuration and client initialization refactored
The AWS S3 connector's configuration and client initialization have been restructured to improve clarity and maintainability. A new \AwsS3ClientCreator\ now handles the construction of the AWS SDK S3 client, centralizing logic for authentication (supporting both explicit credentials and default providers), region selection, custom endpoints, and HTTP connection pooling. Configuration properties have been standardized under the \connect.s3\ prefix, with a new \DeprecationConfigDefProcessor\ that enforces the migration from legacy property names (e.g., \aws.access.key\ to \connect.s3.aws.access.key\) by failing startup if old keys are detected. Additionally, a new \DeleteModeSettings\ configuration allows users to choose between \BatchDelete\ and \SeparateDelete\ modes for cleaning up index files, and the connector now includes dedicated ASCII art headers for both the sink and source connectors to aid in log identification.
kafka-connect-aws-s3/src/main · high confidence
BigQuery connector code moved to java-connectors with new batch and thread-pool implementations
The BigQuery connector source files have been relocated to the java-connectors directory and are now built via Gradle. This change introduces new internal components for the connector: a thread pool executor (KCBQThreadPoolExecutor) to manage concurrent write tasks and track errors, a batch table writer (GCSBatchTableWriter) that uploads records to Google Cloud Storage before loading them into BigQuery, and a runnable (GCSToBQLoadRunnable) that handles the GCS-to-BigQuery load jobs while respecting Google Cloud quotas. Existing row-writing logic is preserved in SimpleBigQueryWriter.
kcbq-connector · high confidence
Cassandra connector configuration and runtime logic refactored
The Cassandra connector's configuration and runtime behavior have been restructured to use a new, centralized configuration model. The \CassandraConfig\ class now explicitly defines all connection, SSL, and operational parameters via \ConfigDef\, replacing the previous implicit or scattered configuration approach. Configuration parsing is now handled by dedicated \CassandraConfigSource\ and \CassandraConfigSink\ classes, which feed into \CassandraSettings\ to produce strongly-typed \CassandraSourceSetting\ and \CassandraSinkSetting\ objects. This change simplifies the connector's internal logic by removing complexity from \KcqlSettings\ and ensuring that configuration validation and defaults are consistently applied across both source and sink tasks. Users will see no change in available configuration options, but the underlying handling of settings is now more robust and aligned with common configuration traits.
kafka-connect-cassandra/src/main/scala/io/lenses/streamreactor/connect · high confidence
Cloud sink configuration refactored with new validation and retry settings
The cloud sink connector's configuration handling has been restructured into a modular set of settings traits (e.g., FlushSettings, CommitRetrySettings, SkipNullSettings) that are mixed into the main config builder. This change introduces configurable commit-retry parameters (max attempts, base delay, multiplier, max delay) to handle transient network errors during file copy and delete operations, and adds a validator that rejects non-deterministic wallclock-based PARTITIONBY keys when exactly-once semantics are enabled in granular commit mode to prevent silent data loss. Additionally, the legacy KCQL keywords WITH\_FLUSH\_COUNT/SIZE/INTERVAL and WITHPARTITIONER are now rejected in favor of PROPERTIES-based configuration, and a new validation ensures that key suffixes do not start with digits to avoid filename collisions.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/config · high confidence
Cloud storage connectors gain unified partition search, file filtering, and resilient commit-chain retries
Cloud source connectors now use a unified partition search mechanism that walks directory trees to a configured depth, ensuring consistent behavior across S3, GCS, and ADLS, and support filtering files by extension. Cloud sink connectors are more resilient to transient network issues, as the commit-chain Copy and Delete operations now use bounded exponential-backoff retries for errors like TCP resets, while correctly aborting retries on thread interruption to allow prompt shutdown. Additionally, the storage layer introduces a new exact-key check for file existence, a mechanism to 'touch' late-arrival files to update their timestamps, and improved error handling for empty or missing lock files to aid in crash recovery.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/storage · high confidence
Elastic 6 connector configuration model restructured
The configuration handling for the Elastic 6 sink connector has been reorganized into a new set of files (ElasticConfig, ElasticConfigConstants, and ElasticSettings). This change introduces explicit configuration definitions for connection details (protocol, hosts, port, cluster name, prefix), error handling policies (noop, throw, retry with configurable retries and intervals), and performance settings (batch size, write timeout). It also adds support for HTTP Basic Authentication credentials, a configurable primary key joiner separator, and a strict mode for bulk item errors, ensuring that per-item bulk errors are surfaced rather than dropped.
kafka-connect-elastic6/src/main/scala/io/lenses/streamreactor/connect/elastic6/config · high confidence
Elastic 7 connector configuration and settings initialization
The Elastic 7 sink connector now initializes its configuration via a dedicated \ElasticConfig\ definition and \ElasticSettings\ factory. This change introduces explicit support for HTTP Basic Authentication (username and password), configurable write timeouts, batch sizes, and retry intervals, while also enabling strict bulk item error reporting to surface per-item failures rather than dropping them silently.
kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7/config · high confidence
Enhanced cloud-sink observability with precise latency tracking and error recording
The cloud sink metrics subsystem now provides deeper visibility into storage operations. A new \OpTimer\ class and \CloudSinkMetricsRegistrar\ replace previous timing mechanisms, enabling thread-safe, lock-free accumulation of operation counts, sums, and maximums, while the registrar ensures robust MBean registration with retry logic. Crucially, the \StorageInterfaceWithMetrics\ decorator now records both latency and error counts for upload, copy, and get operations even when the underlying storage SDK throws an exception, ensuring that transient failures are captured in JMX metrics. Additionally, the metrics exposed via the MBean have been refined, with redundant attributes removed and new ones added for better operational insight.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/metrics · high confidence
Introduce common cloud data format readers
The \kafka-connect-cloud-common\ module now includes a suite of new stream readers for Avro, JSON, CSV, Parquet, and raw bytes, providing a shared foundation for cloud source connectors. This addition includes specific behavioral fixes: Avro and Parquet readers now disable Avro's fast-read mode to prevent DATE logical-type fields from being incorrectly cast to \java.time.LocalDate\ instead of the expected raw integer, and the JSON reader adds support for GZIP-compressed streams.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/formats/reader · high confidence
Introduces ReportHolder for non-blocking report queuing
The sink reporting mechanism now uses a new ReportHolder component to manage report delivery. This component queues reports in a blocking queue and provides non-blocking offer and poll operations with configurable timeouts, ensuring that reporting failures do not block the connector's main processing loop.
kafka-connect-sink-reporting · high confidence
Introduces WriterManagerCreator for centralized sink component initialization
Adds a new WriterManagerCreator class that centralizes the construction of the WriterManager and IndexManager instances for cloud sinks. This component resolves topic-specific bucket options, key naming strategies, and format writers from the configuration, and explicitly wires in the skipNullValues setting and the commit mode (Granular when indexing is disabled) to control exactly-once semantics behavior.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink · high confidence
Introduces new Writer and WriterCommitManager for cloud sink commits
The cloud sink connectors now use a new Writer and WriterCommitManager implementation in the common module. The Writer manages per-topic-partition state transitions (NoWriter, Writing, Uploading, Staged) and escalates IOExceptions to fatal errors to prevent corrupt files. The WriterCommitManager handles commit logic with support for both Granular and Batch commit modes, enabling selective flushing and batch staging to improve exactly-once semantics and reduce latency.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/writer · high confidence
Introduces new model classes for structured Kafka reporting records
The reporting module now includes a new set of model classes to structure data sent to the reporting topic. A new \ReportingRecord\ class encapsulates standard metadata (topic, partition, offset, timestamp, endpoint, payload) alongside connector-specific data via a generic \ConnectorSpecificRecordData\ interface. A \RecordConverter\ handles the transformation of these records into Kafka \ProducerRecord\s, explicitly mapping fields to headers defined in \ReportHeadersConstants\ (such as \input\_offset\, \input\_timestamp\, etc.), replacing previous ad-hoc header handling.
java-connectors/kafka-connect-sink-reporting/src/main/java/io/lenses/streamreactor/connect/reporting/model · high confidence
Introduces structured schema for cloud sink KCQL properties
The cloud sink configuration now uses a formalized schema (SinkPropsSchema) to define and validate KCQL properties. This change explicitly maps configuration keys such as ObjectKeyTemplate, FlushCount, and various StoreEnvelope flags to their specific data types, ensuring that properties like padding, partitioning, and envelope storage are parsed and validated consistently when reading from KCQL statements.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/config/kcqlprops · high confidence
JMS connector refactored to use Jakarta JMS API and new configuration system
The JMS connector implementation has been rewritten to migrate from the legacy Java EE JMS API to the Jakarta JMS API (imported as \jakarta.jms\), ensuring compatibility with modern JMS providers. This change introduces a new configuration and settings layer (\JMSConfig\, \JMSSettings\, \JMSConfigConstants\) that centralizes connector properties and introduces a flexible converter system (\ConverterConfigurator\, \ConverterClassLoader\) for handling source and sink message transformations. The core session management is now handled by a new \JMSSessionProvider\ which manages JMS connections, sessions, and destination lookups (supporting both JNDI and CDI selectors) for both queues and topics.
kafka-connect-jms/src/main/scala · high confidence
MQTT connector configuration and connection logic refactored
The MQTT connector's configuration and connection handling have been restructured to use a new, centralized settings model. The \MqttConfig\ object now explicitly defines all connector parameters (such as hosts, credentials, QoS, SSL paths, and KCQL) using Kafka Connect's \ConfigDef\, while dedicated traits and case classes (\HostSettings\, \MqttSourceSettings\, \MqttSinkSettings\) manage the parsing and validation of these settings. This change introduces stricter validation for SSL certificate files (requiring all three or none) and QoS levels, and updates the \MqttClientConnectionFn\ to build connections based on these new settings objects, ensuring that connection options like timeouts, keep-alive intervals, and automatic reconnection are applied consistently for both source and sink operations.
kafka-connect-mqtt/src/main/scala/io/lenses/streamreactor/connect/mqtt/config · high confidence
MongoDB connector configuration and conversion logic restructured
The MongoDB connector's configuration and data handling have been refactored to improve type safety and maintainability. The configuration definition is now centralized in a new \MongoConfig\ object using Kafka Connect's \ConfigDef\, with constants moved to \MongoConfigConstants\. A new \MongoSettings\ case class replaces the previous configuration model, providing a strongly-typed representation of connection details, KCQL mappings, error policies, and SSL settings. Additionally, the \SinkRecordConverter\ has been rewritten to handle Kafka Connect schema types (such as Date, Time, Timestamp, Decimal, and complex Maps/Structs) more robustly, ensuring accurate conversion to MongoDB BSON documents.
kafka-connect-mongodb/src/main/scala/io/lenses/streamreactor/connect/mongodb/config · high confidence
New Avro and JSON sink data converters with enhanced type handling
The cloud sink connectors now use new \ToAvroDataConverter\ and \ToJsonDataConverter\ implementations in the \kafka-connect-cloud-common\ module. The Avro converter explicitly handles logical types (Date, Time, Timestamp, Decimal) and Connect Union structs, ensuring correct schema mapping and default value retrieval. The JSON converter improves serialization by using Jackson for POJO arrays and properly delegating Structs to the Kafka Connect JsonConverter, resolving issues with mixed-type maps and SMT-returned POJOs.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/conversion · high confidence
New ReaderManagerBuilder for robust state management and backoff
A new ReaderManagerBuilder has been added to the cloud common source state module to centralize the creation of ReaderManager instances. This builder introduces configurable exponential backoff for empty source queues, supports resuming file processing from a specific previous location (state management), and includes a flag to populate Kafka headers with watermark partition and offset information. It also handles post-processing actions and validates cloud locations during initialization.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/source/state · high confidence
New cloud connector configuration traits and conversion interface
The connector now uses a new set of Scala traits to define configuration structures for cloud sources and sinks. CloudSinkConfig exposes settings for bucket options, compression, retry policies, error handling, schema change detection, and null-value skipping. CloudSourceConfig adds configuration for partition searching, file extension filtering, backoff settings, watermark header writing, and late-arrival intervals. A new PropsToConfigConverter trait provides a standardized interface for parsing connector properties into these configuration objects, enabling more modular and extensible configuration handling across different cloud storage implementations.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/config/traits · high confidence
New local output stream implementation with 64-bit pointer tracking
The cloud sink now uses a new \BuildLocalOutputStream\ class in the common stream module to handle local file buffering. This implementation tracks the written byte count using a 64-bit \Long\ pointer, preventing overflow issues for files larger than 2 GiB that could occur with 32-bit integer tracking. The class implements the \CloudOutputStream\ trait, providing a standardized interface for completing writes and reporting the current position.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/stream · high confidence
Refactored cloud connector configuration and storage settings into a shared common module
The configuration logic for cloud connectors has been reorganized into the \kafka-connect-cloud-common\ module to enable reuse across multiple connectors. This change introduces a new \CloudConfigDef\ class that standardizes configuration parsing and applies a lowercase-key processor to ensure case-insensitive property handling. It also centralizes \DataStorageSettings\ to manage envelope and field storage options, \IndexSettings\ to configure exactly-once semantics (including the new \Batch\ commit mode alongside the existing \Granular\ mode) and garbage collection parameters, and \FormatSelection\ to handle format-specific reader construction. Additionally, new utilities like \ConnectorTaskId\ and \TaskDistributor\ are provided to manage task indexing and distribution, while \ConsumerGroupsWriter\ is added to support storing consumer offsets in cloud storage.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/config · high confidence
Refactored cloud source connector file reading into a common reusable module
The file reading logic for Data Lake source connectors has been extracted from connector-specific implementations into a shared \kafka-connect-cloud-common\ module. This change introduces \PartitionDiscovery\ to handle dynamic directory scanning and \ReaderManager\ to orchestrate the lifecycle of file readers, allowing connectors to reuse the same robust state management, polling loops, and post-processing hooks. Users benefit from consistent behavior across different cloud providers (such as AWS S3 and GCP) and improved reliability in handling file discovery and record retrieval.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/source/reader · high confidence
Unified cloud source configuration and enhanced file processing controls
The cloud source connectors now use a common configuration module that introduces a unified \source.partition.search.depth\ setting to replace the cloud-specific \recurse.levels\ parameter, ensuring consistent directory traversal across S3, GCS, and Azure. Users can now filter source files by extension using \source.extension.includes\ and \source.extension.excludes\, and control file ordering via a new \ordering.type\ setting (AlphaNumeric or LastModified). Additionally, the post-processing actions (Move and Delete) now support a \retain\ flag to preserve directory structures after processing, and a \processLateArrival\ flag for the Move action to handle late-arriving data. The connectors also support writing watermark information to Kafka record headers via \source.write.watermark.headers\ and allow configuring late arrival check intervals.
kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/source/config · high confidence
Fixes
14 commits (8 fixes) fixing kafka-connect-opensearch/src/it
A fix in kafka-connect-opensearch/src/it — 14 commits (8 fixs), 11 files: “Apply the linter”; “fix(elastic,opensearch): close adversarial review findings in item-error diversion”; “feat(elastic,opensearch): per-record DLQ for permanent bulk item errors”; and 10 more.
kafka-connect-opensearch/src/it · medium confidence · unverified
Test coverage
10 commits adding/updating tests in kafka-connect-elastic-common/src/test/scala/io/lenses/streamreactor/connect/elastic/common/dlq; 11 commits adding/updating tests in kafka-connect-elastic-common/src/test/scala/io/lenses/streamreactor/connect/elastic/common/sink; 16 commits adding/updating tests in kafka-connect-elastic-common/src/test/scala/io/lenses/streamreactor/connect/elastic/common/writer; 6 commits adding/updating tests in kafka-connect-elastic-common/src/test/scala/io/lenses/streamreactor/connect/elastic/common/bulk; Add functional tests for S3 connector compression codecs; Added HTTP vs GCS sink throughput benchmarks with mocked egress; Added byte-array test fixtures for integration testing; Added configuration validation tests for MongoDB and MQTT connectors; Added configuration validation tests for the OpenSearch sink connector; Added end-to-end test for the MQTT sink connector; Added functional test for Elastic 7 sink connector; Added functional tests for Cassandra source and sink connectors; Added functional tests for the OpenSearch sink connector; Added integration test infrastructure for GCP Storage sink; Added integration test suite for GCP Storage Sink; Added integration test utilities for GCP Storage connector; Added integration tests for Azure Data Lake sink task; Added integration tests for Elastic 6 bulk error handling and writer selection; Added integration tests for GCP Storage Sink schema evolution handling; Added integration tests for GCP storage directory listing and path existence checks; Added integration tests for JMS source and sink connectors; Added test coverage for source data converters; Added test coverage for the Elastic 7 sink connector; Added test resources for JMS sink connector validation; Added test resources for embedded Cassandra and transaction validation; Added test utilities for Cyclops Either and Option types; Added tests for AWS S3 directory listing and storage interface operations; Added tests for Azure Datalake location validation; Added tests for Cassandra connector configuration, settings, and sink behavior; Added tests for HTTP sink authentication and metrics; Added tests for HTTP sink metrics and reset logic; Added tests for HTTP sink record batching logic; Added tests for KcqlWithFieldsSettings upsert key extraction; Added tests for MongoDB SinkRecordConverter date handling; Added tests for S3 configuration processors; Added tests for S3 source configuration and delete mode settings; Added tests for S3 source configuration parsing; Added tests for SQL transformation and struct handling in Kafka Connect SQL common; Added tests for schema conversion and projection utilities; Added tests for wallclock SMT partition key validation; Added unit tests for Azure CosmosDB connector configuration and client initialization; Added unit tests for Azure Data Lake storage interface and page iteration; Added unit tests for Azure Service Bus KCQL configuration mapping; Added unit tests for Azure Service Bus sink connector components; Added unit tests for Azure Service Bus source and sink record mappers; Added unit tests for Elastic 6 sink connector components; Added unit tests for FTP file listing and filtering logic; Added unit tests for GCP Storage connector configuration and storage interface; Added unit tests for GCP connector authentication and configuration; Added unit tests for HTTP sink template rendering; Added unit tests for HTTP sink template substitution logic; Added unit tests for JMS connector configuration, converters, and message handling; Added unit tests for KCQL parsing logic; Added unit tests for Kafka Connect common utilities and configuration; Added unit tests for MQTT source connector topic mapping and shared subscription replication; Added unit tests for MongoDB sink key extraction and connector task configuration; Added unit tests for MqttSinkConnector and MqttWriter; Added unit tests for OpenSearch authentication components; Added unit tests for ReaderManagerBuilder; Added unit tests for RecordConverter; Added unit tests for S3 connector configuration and connector logic; Added unit tests for S3 model validation and partition extraction; Added unit tests for S3 sink configuration and validation logic; Added unit tests for S3 source reader components; Added unit tests for cloud-common configuration and data handling components; Added unit tests for concurrent execution, SSL configuration, and offset handling; Added unit tests for the GCP Pub/Sub Source Connector; Added unit tests for the HTTP Sink connector; Added unit tests for the KCQL target type parser; Added unit tests for the new OpenSearch sink connector; Added unit tests for the sink-reporting module; Cassandra integration tests migrated to Scala; FTP source integration tests migrated to Scala; Functional tests for Elastic6 and MongoDB connectors migrated to Scala; Integration tests for HTTP Sink and request sender; Integration tests for MongoDB sink migrated to Scala with new test fixtures; Migrate MQTT connector integration tests from Java to Scala; New integration tests for Cassandra sink connector features.
Dependencies
Build system migration to SBT 1.11.2 with consolidated dependency management
The project has migrated its build infrastructure to SBT version 1.11.2, introducing a centralized \Dependencies.scala\ file that explicitly defines versions for all libraries and build plugins. This change consolidates dependency management, setting Scala 2.13.16, Kafka 4.1.0, and AWS SDK 2.29.52, while updating key plugins such as \sbt-assembly\ to 2.3.1, \sbt-scoverage\ to 2.3.1, and \sbt-pack\ to 0.20. The build now enforces Java 11 compatibility and includes new packaging plugins for license headers and assembly configuration.
project · high confidence
Introduce Gradle build system for Java connectors
The Java connectors module now uses Gradle for building, replacing the previous build configuration. This introduces the Shadow plugin (v8.3.6) for creating fat JARs and Spotless (v7.0.4) for code formatting. The build targets Java 17 and includes dependencies for Azure Service Bus (azure-core 1.55.4, azure-messaging-servicebus 7.17.12), GCP (google-cloud libraries-bom 26.38.0), and Kafka 4.1.0.
(dependencies) · high confidence
Housekeeping
Documentation of Exactly Once Offset and Locking Logic
Added a UML sequence diagram to the \kafka-connect-cloud-common\ module that visualizes the task lifecycle for S3 and GCP storage sinks. The diagram details the initialization, lockfile management, record processing, and offset validation steps required to ensure exactly-once delivery semantics, specifically illustrating how pending operations are tracked and how offset consistency is verified before finalizing uploads.
kafka-connect-cloud-common/src/uml · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 60 → 64 (+4.3)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 67 → 65 (-2.3)
- Architecture 89 → 89 (+0.1)
- Maturity 70 → 77 (+6.7)
- Readiness 50 → 53 (+2.8)
- Security 62 → 75 (+12.8)
- Domain Modelling 100 → 100 (+0.0)
- Performance 79 → 100 (+20.6)
Resolved (24)
- Concentrated knowledge decay
- Duplicated block (17 lines × 2) (kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/seek/IndexManagerV2.scala)
- Duplicated block (23 lines × 2) (kafka-connect-elastic6/src/main/scala/io/lenses/streamreactor/connect/elastic6/config/ElasticSettings.scala)
- Duplicated block (37 lines × 2) (kafka-connect-elastic6/src/main/scala/io/lenses/streamreactor/connect/elastic6/KElastic6BulkClient.scala)
- Duplicated block (6 lines × 2) (kafka-connect-sql-common/src/main/scala/io/lenses/connect/sql/Transform.scala)
- Duplicated block (6–7 lines × 2) (kafka-connect-sql-common/src/main/scala/io/lenses/connect/sql/Transform.scala)
- Duplicated block (80 lines × 2) (kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/PrimaryKeyExtractor.scala)
- High CVE: [GHSA redacted] (project/Dependencies.scala)
- High CVE: [GHSA redacted] (project/Dependencies.scala)
- Hotspot: kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/seek/IndexManagerV2.scala (kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/seek/IndexManagerV2.scala)
- Leaked secret: hardcoded-credential (dev-scripts/opensearch-local/connector-config.json)
- Leaked secret: hardcoded-credential (dev-scripts/opensearch-local/docker-compose.yml)
- Leaked secret: hardcoded-credential (dev-scripts/opensearch-local/scripts/06-verify.sh)
- Most significant orphaned file (java-connectors/kafka-connect-query-language/src/main/java/io/lenses/kcql/Kcql.java)
- Most significant orphaned file (kafka-connect-cassandra/src/main/scala/io/lenses/streamreactor/connect/cassandra/source/CassandraTableReader.scala)
- No ADRs found
- Off-boarding risk: anonymized user #1
- Off-boarding risk: anonymized user #2
- Orphaned file (kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/indexname/CreateIndex.scala)
- Rotate the exposed credentials — git history can't be un-committed
- …and 4 more
New (76)
- Change coupling: ErrantRecordHandler.scala ↔ AbstractElasticSinkTask.scala (kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/dlq/ErrantRecordHandler.scala)
- Duplicated block (16 lines × 2) (kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7/KElasticBulkClient.scala)
- Duplicated block (29 lines × 2) (kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7/KElasticBulkClient.scala)
- Duplicated block (45 lines × 2) (kafka-connect-elastic-common/src/main/scala/io/lenses/streamreactor/connect/elastic/common/PrimaryKeyExtractor.scala)
- Duplicated block (5 lines × 2) (kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7/KElasticBulkClient.scala)
- Duplicated block (6–7 lines × 2) (kafka-connect-sql-common/src/main/scala/io/lenses/connect/sql/Transform.scala)
- Duplicated block (8 lines × 2) (kafka-connect-sql-common/src/main/scala/io/lenses/connect/sql/Transform.scala)
- Duplicated block (9 lines × 2) (kafka-connect-elastic6/src/main/scala/io/lenses/streamreactor/connect/elastic6/config/ElasticSettings.scala)
- Duplicated block (9 lines × 2) (kafka-connect-elastic7/src/main/scala/io/lenses/streamreactor/connect/elastic7/KElasticBulkClient.scala)
- Edited copies of one member (3 members, 11+ corresponding lines) (test-common/src/main/scala/io/lenses/streamreactor/connect/testcontainers/connect/KafkaConnectClient.scala)
- Edited copies of one member (3 members, 15+ corresponding lines) (kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/formats/reader/converters/BytesOutputRowConverter.scala)
- Edited copies of one member (3 members, 22+ corresponding lines) (kafka-connect-sql-common/src/main/scala/io/lenses/connect/sql/Sql.scala)
- Edited copies of one member (4 members, 7+ corresponding lines) (java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapper.java)
- Edited copy of a member (10 corresponding lines) (java-connectors/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/BigQuerySinkTask.java)
- Edited copy of a member (10 corresponding lines) (java-connectors/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/SchemaManager.java)
- Edited copy of a member (10 corresponding lines) (kafka-connect-cassandra-sink/src/main/scala/io/lenses/streamreactor/connect/cassandra/CassandraSinkConnector.scala)
- Edited copy of a member (11 corresponding lines) (java-connectors/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/write/batch/GCSBatchTableWriter.java)
- Edited copy of a member (11 corresponding lines) (kafka-connect-cloud-common/src/main/scala/io/lenses/streamreactor/connect/cloud/common/sink/transformers/EnvelopeWithSchemaTransformer.scala)
- Edited copy of a member (11 corresponding lines) (kafka-connect-sql-common/src/main/scala/io/lenses/connect/sql/StructSql.scala)
- Edited copy of a member (11 corresponding lines) (test-common/src/main/scala/io/lenses/streamreactor/connect/testcontainers/connect/KafkaConnectClient.scala)
- …and 56 more
Changes since last survey
- 15 commits — 4 feature/other, 11 fixes
By area
- (repo) — 9 commits
- kafka-connect-elastic-common/src — 3 commits
- docs/elastic-opensearch-item-errors.md — 1 commit
- kafka-connect-aws-s3/src — 1 commit
- kafka-connect-cloud-common/src — 1 commit
Notable commits
- fix: Merge branch 'master' into fix/LC-346-date-ordering-ties-and-relist
- fix: Merge branch 'master' into fix/OPS-2652-java-connector-raw-packaging
- fix: Merge branch 'master' into fix/OPS-2652-java-connector-raw-packaging
- fix: Merge pull request #422 from lensesio-dev/fix/OPS-2652-java-connector-raw-packaging
- fix: Merge pull request #427 from lensesio-dev/fix/LC-344-s3-list-filtered-page-cursor
- fix: Merge pull request #428 from lensesio-dev/fix/LC-346-date-ordering-ties-and-relist
- fix: fix(aws-s3): advance source cursor past fully filtered list pages
- fix: fix(cloud-source): stop LastModified ordering skipping same-second files
- fix: fix(elastic): address PR review findings (locale, DLQ ack wait, validateOp position)
- fix: fix(elastic): do not throw on unrecognised bulk.item.errors for ES6; parse strict flag like ConfigDef
- fix: test(elastic): fix stale drop-mode writer-stage WARN assertion
- change: Merge branch 'master' into feat/elastic-opensearch-item-error-dlq
- change: Merge origin/master into feat/elastic-opensearch-item-error-dlq
- change: Merge pull request #416 from lensesio-dev/feat/elastic-opensearch-item-error-dlq
- change: docs(elastic): correct NOOP upgrade note - permanent item errors already failed the task
Architecture
- Unchanged — 0 containers · 8 contexts · 8 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
lensesio/stream-reactor was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit e5d3e3739eaf7c19c8581ccc6d32c19defe36ce2 — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.