Skip to content
CAI
Software that uses CAICheck a score

apache/carbondata

53.7

Adequate · 14 November 2026

242.2k

lines of production code

Java

with Scala

6

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

CarbonData is a columnar storage engine designed for high-performance analytical queries on large datasets, supporting integration with Apache Spark, Flink, Hive, and Presto. It provides a robust data loading pipeline that handles CSV and JSON ingestion, complex data types, and various sorting and bucketing strategies, while storing data in an optimized V3 columnar format. The system features advanced query acceleration through multiple indexing mechanisms, including Bloom filters, Lucene-based fine-grain indexes, and materialized views with time-series support. Additionally, it offers real-time streaming ingestion, spatial indexing capabilities, and a pluggable architecture for diverse storage backends like HDFS, S3, and Alluxio.

How it got here

2016–2017 — V3 format and engine overhaul

160 changes.

This period focused on a comprehensive architectural overhaul to introduce the CarbonData V3 file format, featuring a new columnar storage layout, adaptive encoding, and unsafe off-heap memory management. The team simultaneously rebuilt the query execution engine to support vectorized processing, complex data types, and integration with Spark 2.3/2.4 and Presto, while establishing a pluggable file system abstraction for diverse storage backends.

2018–2026 — AI integration and index expansion

79 changes.

This period focused on integrating CarbonData with AI workflows through the introduction of the PyCarbon SDK and the AI\_carbon agent archive format. It also significantly expanded query capabilities by implementing new indexing strategies, including Bloom filters, Lucene-based fine-grain indexes, and GeoHash spatial indexing, alongside a modular infrastructure for materialized view query rewriting.

Features

Adaptive encoding for primitive data types

The core module now includes an adaptive encoding framework for primitive data types (byte, short, int, long, float, double, and decimal). This change introduces a new \AdaptiveCodec\ base class and specific implementations (such as \AdaptiveIntegralCodec\, \AdaptiveFloatingCodec\, and their delta-encoding variants) that dynamically select the most storage-efficient target data type based on column statistics. By casting page values to the smallest necessary integer or floating-point type, the system reduces memory footprint and stored size for columns with limited value ranges, while maintaining backward compatibility for existing data formats.

core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/adaptive · high confidence

Add DictionaryThresholdReachedException for local dictionary limits

A new exception class, DictionaryThresholdReachedException, has been added to the core local dictionary module. This exception is thrown when the capacity limit of a local dictionary is reached during data loading, allowing the system to signal this specific boundary condition to callers.

core/src/main/java/org/apache/carbondata/core/localdictionary/exception · high confidence

A new Scala example file (FlinkExample.scala) has been added to the examples/flink directory. This example demonstrates how to write a CarbonData file using Spark and then read specific columns from that file using Apache Flink via the CarbonTableInputFormat.

examples/flink · high confidence

The integration/flink-proxy module introduces a new 'proxy' file system (ProxyFileSystem) and associated writer components (ProxyRecoverableWriter, ProxyFileWriter) that act as an adapter between Flink's StreamingFileSink and the CarbonData SDK. This change allows Flink streaming jobs to write data directly to Carbon tables by intercepting file output streams, handling the serialization of recoverable state for checkpointing, and delegating the actual write operations to CarbonWriter, thereby enabling Flink-to-Carbon ingestion.

carbondata-flink-proxy, integration/flink-proxy · high confidence

This change introduces the concrete writer implementations that enable Flink to stream data into CarbonData. It adds \CarbonLocalWriter\ and \CarbonS3Writer\ to handle data persistence to local temporary paths and Amazon S3 respectively, along with their corresponding factories (\CarbonLocalWriterFactory\, \CarbonS3WriterFactory\) and builders. The update also includes \StageManager\ to handle stage input metadata and success files, and registers the new factory builders via the \META-INF/services\ SPI mechanism, allowing the Flink integration to dynamically select the appropriate storage backend based on configuration.

integration/flink/src/main · high confidence

Added Boolean data type conversion utilities

A new BooleanConvert class has been introduced to handle conversions for the Boolean data type. This utility provides methods to convert boolean values to bytes (where true is 1 and false is 0), convert bytes back to booleans, and parse string inputs ('true' or 'false') into Boolean objects, enabling proper storage and retrieval of boolean data within the CarbonData page encoding system.

core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool · high confidence

Added DictionaryByteArrayWrapper for dictionary value comparison

A new DictionaryByteArrayWrapper class has been introduced in the core cache dictionary package to handle byte array-based dictionary values. This wrapper provides custom equals and hashCode implementations, allowing dictionary entries to be compared and hashed correctly based on their underlying byte data rather than object identity, which supports the local dictionary data loading functionality.

core/src/main/java/org/apache/carbondata/core/cache/dictionary · high confidence

Added English localization strings for Carbon Data Writer dialog

Added a new resource bundle (messages\_en\_US.properties) for the processing store module, providing English translations for user-facing labels in the Carbon Data Writer dialog, including titles, store location, blocklet size, table name, and max node settings.

processing/src/main/java/org/apache/carbondata/processing/store/messages · high confidence

Added EscapeSequences enum for standard escape characters

A new \EscapeSequences\ enum has been introduced in the core module to centralize definitions for standard escape sequences, including new line, backspace, tab, and carriage return. This provides a consistent way for the application to reference these character representations.

core/src/main/java/org/apache/carbondata/core/enums · high confidence

Added Lucene-based Fine-Grain Index implementation and MinMax Index example

This change introduces a new Lucene-based Fine-Grain Index implementation in the \index/lucene\ module, providing classes for index creation (\LuceneIndexBuilder\), writing (\LuceneIndexWriter\), and searching (\LuceneFineGrainIndex\) with support for text matching and configurable blocklet-wise storage. Additionally, it adds a MinMax Index example in \index/examples/src/minmaxindex\ to demonstrate coarse-grain index development, including data writing, blocklet pruning logic, and a corresponding test suite.

index/lucene · high confidence

Added QueryExecutionException class

A new \QueryExecutionException\ class has been introduced in the core scan executor package to handle query execution errors. This exception extends the standard \Exception\ class and provides a constructor that accepts a \Throwable\ cause, along with an overridden \getLocalizedMessage\ method, allowing users and developers to properly catch and inspect specific query execution failures.

core/src/main/java/org/apache/carbondata/core/scan/executor/exception · high confidence

Added RLE encoding for integral column pages

The RLECodec implementation for integral column pages has been added, introducing a new encoding strategy that tracks repeated and non-repeated runs to optimize storage for boolean, byte, short, int, and long data types. This change provides a new compression method for these specific data types within the CarbonData core datastore.

core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/rle · high confidence

Added V3 measure chunk readers for CarbonData file format

New \MeasureChunkReaderV3\ and \MeasureChunkPageReaderV3\ classes have been added to the core module to support reading and decoding measure column data stored in the V3 file format. These components handle the specific V3 data structure, including reading compressed data chunks, managing column offsets, and decoding pages using the configured compressor, enabling the system to process data files written in this newer format.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/measure/v3 · high confidence

Added common utility classes for Java 7 compatibility

The carbondata-common module now includes new utility classes: CarbonIterator, which provides a base iterator with default remove behavior and lifecycle methods for Java 7 compatibility; Maps, which adds a getOrDefault helper to avoid JDK 8 dependencies; and Strings, which provides mkString (similar to Scala's mkString) and formatSize for human-readable byte size formatting. These additions support running CarbonData on older Java versions by replacing JDK 8+ standard library features with custom implementations.

common/src/main/java/org/apache/carbondata/common · high confidence

Added configuration templates for CarbonData system and data loading settings

The repository now includes \conf/carbon.properties.template\ and \conf/dataload.properties.template\ files, providing users with documented default configurations for system performance (such as data loading cores, sort buffer sizes, and compaction thresholds) and CSV parsing options (including delimiters, quoting, and null value handling). These templates serve as the reference for tuning CarbonData behavior during installation and operation.

conf · high confidence

Added custom audit and statistic logging levels with dedicated file appenders

The logging module now includes new classes to support specialized logging: \AuditLevel\ and \StatisticLevel\ extend log4j's \Level\ to introduce distinct log categories for audit trails and statistical data. Additionally, \AuditExtendedRollingFileAppender\ and \ExtendedRollingFileAppender\ provide custom rolling file appenders that allow for specific file naming, size-based rotation, and backup index management, enabling users to separate and manage audit and statistic logs independently from standard application logs.

common/src/main/java/org/apache/carbondata/common/logging/impl · high confidence

Added documentation and scripts for building CarbonData notebook Docker images

New build artifacts have been added to help users create CarbonData-enabled Jupyter notebook environments. This includes a Dockerfile that bases the image on \jupyter/all-spark-notebook:spark-3.1.1\ and automatically downloads the CarbonData 2.3.0 binary JAR, along with two documentation guides (\How-to-build-carbondata-notebook-docker-image-by-dockerfile.md\ and \How-to-build-carbondata-notebook-docker-image-by-manual.md\) detailing both automated and manual build processes. Additionally, \build/README.md\ provides updated Maven build instructions for Spark 2.4.5, and new shell (\carbondata-build-info.sh\) and batch (\carbondata-build-info.bat\) scripts are introduced to generate version, revision, branch, and date metadata into a properties file during the build process.

build · high confidence

Added interface audience and stability annotations

The project now includes \InterfaceAudience\ and \InterfaceStability\ annotations in the common module, ported from Apache Hadoop. These annotations allow developers to mark classes and methods as intended for users, developers, or internal use, and to define stability guarantees (Stable, Evolving, Unstable) to clarify compatibility expectations across releases.

common/src/main/java/org/apache/carbondata/common/annotations · high confidence

Added localized logging resource files for core and processing modules

New resource property files (CARBON\_CORELogResource.properties and CARBON\_PROCESSINGLogResource.properties) have been added to the core and processing modules. These files define localized log message keys (carbon.core and carbon.processing) and include standard Apache Software Foundation license headers, enabling consistent and localized logging output for these specific components.

core, processing · high confidence

Added sample data files for Spark examples

The examples/spark module now includes a set of CSV data files (Test\_Data1.csv, complexdata.csv, data.csv, data1.csv, dataSample.csv, dimSample.csv, factSample.csv, sample.csv, streamSample.csv) and a log4j.properties configuration file in src/main/resources. These files provide sample datasets for testing various data types and scenarios, as well as logging configuration for the Spark examples.

examples/spark/src/main/resources · high confidence

Added table operation enum to support column rename

A new TableOperation enum has been introduced in the core features package to explicitly define supported table modification actions. This enum includes ALTER\_COLUMN\_RENAME alongside other operations such as ALTER\_DROP, ALTER\_ADD\_COLUMN, and ALTER\_CHANGE\_DATATYPE, providing a structured way to identify and handle these specific table alteration capabilities within the system.

core/src/main/java/org/apache/carbondata/core/features · high confidence

Atomic file operations for S3 and Alluxio

The core file operations module now supports atomic writes for S3 and Alluxio storage backends. A new factory and interface (\AtomicFileOperations\) dispatch to an S3-specific implementation that overwrites files directly, avoiding the brief window where a file might not exist during delete-recreate cycles. The default implementation continues to use a temporary file that is renamed on success, ensuring data integrity during data loads, index updates, or compactions.

core/src/main/java/org/apache/carbondata/core/fileoperations · high confidence

CarbonData V3 file format schema definition

The format module now defines the V3 file format schema via new Thrift definitions. This introduces V3-specific structures including \BlockletIndex\ (supporting min-max and B-tree indexes), \DataChunk3\ (grouping \DataChunk2\ pages), and \BlockletInfo3\ (tracking page offsets and row counts for table alterations). The schema also adds \is\_sort\ to the index footer to support no-sort compaction and includes \PresenceMeta\ to handle null values in spatial data.

format · high confidence

EXPLAIN command now displays query rewrite and block pruning details

When the \enable.query.statistics\ configuration is enabled, the EXPLAIN command output now includes detailed information about SQL rewrites and block pruning. This new output shows which materialized views were used for rewriting, along with specific metrics on how many blocks and blocklets were skipped by the default index, CG Index, and FG Index during query execution.

core/src/main/java/org/apache/carbondata/core/profiler · high confidence

Initial Hive integration support for CarbonData

This change introduces the core Java components required to read and write CarbonData tables directly from Apache Hive. It adds a Hive Storage Handler, a custom SerDe for schema inference and data serialization, input/output formats for MapReduce execution, and expression pushdown logic to convert Hive query filters into CarbonData scan expressions, enabling Hive users to query CarbonData files natively.

integration/hive/src/main/java/org/apache/carbondata/hive · high confidence

Introduce AI\_carbon module for Agent file and context archiving

The new Agent\_module provides an \AI\_carbon\ archive format that stores Agent-generated files alongside their generation context, conversation history, and revision metadata in a single ZIP file. Users can create and inspect archives via a Python API or a local web management interface running on port 8765, which allows browsing files, viewing context, and reviewing conversation history. The module also supports an optimization workflow where an Agent callback receives the current file content and full historical context to produce improved revisions, preserving previous versions. Command-line tools are included for creating, showing, and serving archives, along with a Codex synchronization skill to automate context capture during development sessions.

_Agent\module · high confidence

Introduce AI\_carbon project archive format for agent data storage

The Agent module now includes a new \AI\_carbon\ class that implements a self-contained, atomic \.AI\_carbon\ project archive format. This format stores project metadata, generated files, optimization contexts, and conversation histories within a ZIP container, enabling the agent to function as a persistent data store rather than just a format converter. Users can create, open, and manage these archives to track artifacts with full revision history, context, and conversation logs.

apache-carbondata-agent · high confidence

Introduce CarbonCli standalone tool for data inspection and benchmarking

A new standalone CLI tool, CarbonCli, is added to the tools/cli module, enabling users to inspect CarbonData files and segments directly from the command line. The tool supports a 'summary' command to display basic statistics (blocks, shards, rows, sizes), schema details, segment status, table properties, and per-column or all-column statistics. It also includes a 'benchmark' command to measure performance of reading headers, footers, and column chunks, as well as a 'sort\_columns' command to list sort columns for a segment folder. This provides a lightweight way to debug and analyze CarbonData files without requiring a full Spark or Hive environment.

tools/cli · high confidence

Introduce CarbonData V3 file format writer and page-based storage

The processing store layer now supports the V3 columnar file format, replacing the previous V1/V2 writers which are no longer supported. This change introduces a new \CarbonFactDataHandlerColumnar\ that organizes data into \TablePage\ structures, enabling configurable page sizes and a producer-consumer execution model for data loading. A new \CarbonDataWriterFactory\ routes write operations to the V3 implementation (\CarbonFactDataWriterImplV3\), and new model classes like \CarbonFactDataHandlerModel\ and \CarbonDataFileAttributes\ manage the metadata and attributes required for this new storage layout.

processing/src/main/java/org/apache/carbondata/processing/store · high confidence

Introduce DirectCompressCodec for direct data compression

Added a new DirectCompressCodec implementation that applies compression directly to input data during encoding and handles decompression during decoding. This codec supports both standard direct compression and specific handling for VARCHAR types, enabling more efficient storage and retrieval of column page data by bypassing intermediate encoding steps.

core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/compress · high confidence

Introduce Hadoop stream input format and record reader

Added new Hadoop MapReduce components (CarbonStreamInputFormat, StreamRecordReader, StreamBlockletReader, and CarbonStreamUtils) to the stream module, enabling the system to read and process stream segment data via the standard Hadoop InputFormat/RecordReader interfaces.

hadoop/src/main/java/org/apache/carbondata/hadoop/stream · high confidence

Introduce PyCarbon Python SDK for AI framework integration

Adds the PyCarbon Python SDK, providing a unified API for reading CarbonData datasets directly into AI training workflows. The new package includes a \make\_carbon\_reader\ function that supports reading from local, HDFS, and S3 paths, with features like shuffling, predicate filtering, and configurable worker pools. It bridges CarbonData with popular deep learning frameworks by offering specific adapters for TensorFlow (\make\_dataset\, \make\_tensor\) and PyTorch (\make\_data\_loader\), allowing developers to leverage CarbonData's indexing and caching for faster data ingestion during model training.

python · high confidence

Introduce Thrift-based schema serialization interface and implementation

Added the \SchemaConverter\ interface and its \ThriftWrapperSchemaConverterImpl\ class to handle bidirectional conversion between CarbonData's internal wrapper schema objects (TableInfo, TableSchema, ColumnSchema, etc.) and the external Thrift format. This enables the core module to serialize and deserialize schema metadata using the Thrift protocol, supporting the persistence and transmission of table structures, column definitions, data types, and encoding configurations.

core/src/main/java/org/apache/carbondata/core/metadata/converter · high confidence

Introduce V3 columnar format reader interfaces and factory

The core data store chunk reader module now supports the V3 columnar format through new interfaces (DimensionColumnChunkReader, MeasureColumnChunkReader) and a factory (CarbonDataReaderFactory). Users querying V3-format files will have their requests routed to V3-specific implementations (DimensionChunkPageReaderV3, DimensionChunkReaderV3, etc.), enabling page-level reading and optimized scan performance for this format version.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader · high confidence

Introduce V3 dimension chunk reader for improved memory efficiency

Added new V3 dimension chunk readers (AbstractDimensionChunkReader, DimensionChunkReaderV3, and DimensionChunkPageReaderV3) that support the V3 data format. This change enables page-level reading and decompression of dimension columns, allowing the system to process data in smaller chunks rather than loading entire blocklets into memory, which is particularly beneficial for memory-constrained operations like compaction.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/dimension · high confidence

Introduce V3 file format infrastructure and column unique ID generation

This change adds the core utilities and services required for the new V3 data file format, including the AbstractDataFileFooterConverter and CarbonMetadataUtil for handling V3-specific metadata and footers, and the ColumnUniqueIdService with its implementation to generate unique identifiers for columns. It also introduces the BitSetGroup utility to manage filter result sets during query execution and provides a new CarbonLoadStatisticsDummy class to support optional load statistics recording.

core/src/main/java/org/apache/carbondata/core/util · high confidence

Introduce V3 format chunk abstractions and dimension column page interface

The core data store chunk layer now includes new abstractions to support the V3 data format. AbstractRawColumnChunk provides a base class for handling uncompressed blocklets, introducing fields for min/max values, page counts, and row counts, along with a reference to the V3 DataChunk3 structure. Additionally, the new DimensionColumnPage interface defines the contract for dimension column chunks, specifying methods for filling raw data, surrogate keys, and vector information, as well as handling adaptive encoding and null bit sets. These changes lay the groundwork for optimized scanning and reading capabilities in the V3 format.

core/src/main/java/org/apache/carbondata/core/datastore/chunk · high confidence

Introduce V3 format writer interface and abstract implementation

The data loading process now supports the V3 file format through a new \CarbonFactDataWriter\ interface and its \AbstractFactDataWriter\ base implementation. This change introduces the core infrastructure for writing V3 format data files, including handling block metadata, managing file channels for direct HDFS/S3 writes, and tracking data load metrics, laying the groundwork for subsequent V3-specific writer implementations.

processing/src/main/java/org/apache/carbondata/processing/store/writer · high confidence

Introduce blocklet index metadata classes for min/max and B-Tree storage

Added three new classes in the core metadata blocklet index package to support storing and serializing blocklet index information. BlockletBTreeIndex holds the bit-packed start and end keys for a blocklet, while BlockletMinMaxIndex stores the bit-packed min and max values for all columns within a blocklet, including a flag array indicating whether min/max values are set. BlockletIndex serves as a container that aggregates both the B-Tree index and the min/max index for a file, enabling the system to persist and retrieve these specific index structures.

core/src/main/java/org/apache/carbondata/core/metadata/blocklet/index · high confidence

Introduce blocklet-level index implementation for finer-grained data pruning

The core module now includes a new blocklet-level index implementation (BlockIndex, BlockletIndex, and supporting classes) that enables more granular data pruning during query execution. This change adds the ability to cache and access index metadata at the blocklet level, improving query performance by reducing the amount of data scanned. The implementation includes new factory classes, input split handling, and metadata storage mechanisms to support this finer granularity.

core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex · high confidence

Introduce core Materialized View schema and catalog management

This change adds the foundational in-memory catalog and schema management layer for Materialized Views (MV) within the CarbonData core module. It introduces the \MVCatalog\ and \MVCatalogFactory\ interfaces to support MV query rewriting, alongside the \MVManager\ abstract class which coordinates MV schema lifecycle operations (create, read, update, delete) and sync status verification. The update also includes the \MVProvider\ for persisting MV schemas to disk as JSON, the \MVSchema\ class to define MV metadata (including support for incremental refresh modes and modified queries), and supporting types like \MVProperty\, \MVStatus\, and \MVStatusDetail\ to handle configuration and state tracking.

core/src/main/java/org/apache/carbondata/core/view · high confidence

Introduce core vectorized scan result interfaces and processors

This change introduces the foundational classes for CarbonData's vectorized query execution engine within the core scan module. It adds the \CarbonColumnVector\ interface, which defines methods for populating and accessing columnar data (including support for lazy loading and dictionary vectors), and the \CarbonColumnarBatch\ class to manage batches of these vectors. Additionally, it provides the \MeasureDataVectorProcessor\ with specific fillers for integral and boolean data types, enabling efficient, vectorized population of result sets from column pages. These components form the internal mechanism for high-performance, column-oriented data retrieval.

core/src/main/java/org/apache/carbondata/core/scan/result/vector · high confidence

Introduce dedicated processing utility classes and audit logging

This change introduces a new set of utility classes in the processing module to support data loading, query handling, and operational auditing. The new Auditor class provides structured JSON-based audit logging for operations, configurable via log4j. CarbonBadRecordUtil centralizes bad record handling logic, including path resolution and file renaming. CarbonDataProcessorUtil and CarbonLoaderUtil consolidate data loading workflows, including temporary folder management, block assignment strategies, and segment metadata recording. CarbonQueryUtil adds file path parsing support, and TableOptionConstant defines DDL option keys.

processing/src/main/java/org/apache/carbondata/processing/util · high confidence

Introduce dedicated scanned result classes for filter and non-filter queries

Added FilterQueryScannedResult and NonFilterQueryScannedResult to the scan result implementation package. FilterQueryScannedResult handles data retrieval based on filtered row indices, while NonFilterQueryScannedResult returns complete data for queries without filters. These classes extend BlockletScannedResult and provide specific implementations for filling columnar dictionaries, no-dictionary columns, and measure columns, as well as retrieving dictionary keys, complex type keys, and current row IDs.

core/src/main/java/org/apache/carbondata/core/scan/result/impl · high confidence

Introduce distributed input split and block metadata structures for Hadoop integration

Added new Hadoop integration classes to support distributed reading of CarbonData files. CarbonInputSplit now implements the new Block interface and Distributable, carrying detailed metadata such as segment info, blocklet IDs, delete delta files, and index-specific fields to enable efficient data pruning and serialization. CarbonInputSplitWrapper provides a mechanism to batch and serialize multiple splits for transmission. Supporting internal classes ObjectArrayWritable and the Block interface define the contract for block-level metadata (path, matched blocklets, full scan flag) used by the distributed index server and executor components.

core/src/main/java/org/apache/carbondata/hadoop · high confidence

Introduce extensible column compression with Gzip and Zstd support

The core module now provides a new, extensible compression framework for column data, introducing a \Compressor\ interface and an \AbstractCompressor\ base class to standardize compression logic. This change adds built-in support for Gzip and Zstd compression algorithms via new \GzipCompressor\ and \ZstdCompressor\ implementations, registered through a \CompressorFactory\ that also allows for custom compressor registration. Users can now configure column compression using these algorithms, benefiting from standardized handling of primitive types and memory management.

core/src/main/java/org/apache/carbondata/core/datastore/compression · high confidence

Introduce lazy loading for blocklet and page data during scan

Added BlockletScanner interface, LazyBlockletLoader, and LazyPageLoader classes to the scan scanner package. These components enable lazy loading of blocklet column chunks and page vectors, meaning data is only read from disk and decompressed when the execution engine actually accesses it. This optimization is particularly beneficial for filter queries involving high cardinality columns, as it reduces unnecessary I/O and memory usage by deferring data loading until required.

core/src/main/java/org/apache/carbondata/core/scan/scanner · high confidence

Introduce modular query plan infrastructure for materialized views

This change adds a new modular plan framework within the materialized view (MV) planning subsystem, introducing core components such as \ModularPlan\, \Modularizer\, and \Harmonizer\ to represent and optimize MV query structures. It includes a DSL (\mv/dsl/package.scala\) for constructing these plans, a \BirdcageOptimizer\ for logical plan optimization, and specific logic for aggregate push-down (\AggregatePushDown\) and plan harmonization. This infrastructure provides the foundational representation and optimization rules required for the MV engine to handle complex queries, partial query matching, and incremental updates.

mv/plan · high confidence

Introduce new complex data type processing classes

Added new Java classes in the processing datatypes package to support complex data types during data loading. Specifically, \GenericDataType\ defines the interface for complex types, while \ArrayDataType\, \StructDataType\, and \PrimitiveDataType\ provide the concrete implementations for handling array, struct, and primitive column data respectively.

processing/src/main/java/org/apache/carbondata/processing/datatypes · high confidence

Introduce new scan processor components for blocklet iteration and data handling

The scan processor module now includes three new classes to support data scanning: \BlockletIterator\ for iterating over data block references, \RawBlockletColumnChunks\ to hold raw dimension and measure column data along with file reader context, and \DataBlockIterator\ which orchestrates the scanning process by leveraging the blocklet iterator and scanner implementations to collect results in batches. These components form the core infrastructure for reading and processing CarbonData blocklets during query execution.

core/src/main/java/org/apache/carbondata/core/scan/processor · high confidence

Introduce new schema model for index metadata storage

Added \CarbonRowSchema\ and \SchemaGenerator\ to define and generate the binary schema for index metadata (block and blocklet). This new structure explicitly models fixed-length, variable-length, and struct types, and calculates byte positions to support efficient storage and retrieval of min/max statistics, row counts, and blocklet information.

core/src/main/java/org/apache/carbondata/core/indexstore/schema · high confidence

Introduce pluggable file system abstraction for CarbonData

The core module now supports a pluggable file operations architecture, allowing CarbonData to read from and write to various file systems beyond the local disk. This change introduces a \FileTypeInterface\ and a \DefaultFileTypeProvider\ that dispatches file operations to specific implementations (such as \HDFSCarbonFile\, \S3CarbonFile\, \AlluxioCarbonFile\, and \ViewFSCarbonFile\) based on the path prefix. A new \DFSFileReaderImpl\ handles remote file reading with stream caching, while \FileFactory\ serves as the central entry point to instantiate the correct file handlers. This enables users to store CarbonData tables on HDFS, S3, Alluxio, and ViewFS seamlessly.

core/src/main/java/org/apache/carbondata/core/datastore/impl · high confidence

Introduce result collector factory and interface for scan aggregation

Added a new \ResultCollectorFactory\ and \ScannedResultCollector\ interface in the core scan collector package to centralize the creation and aggregation of scan results. The factory selects specific collector implementations (such as dictionary-based, raw-based, vector-based, and row-id-based variants) based on block execution metadata, enabling the system to efficiently aggregate query results in either row or columnar formats.

core/src/main/java/org/apache/carbondata/core/scan/collector · high confidence

Introduce unsafe memory-based sorting for data loading

Added UnsafeCarbonRowPage and UnsafeSortDataRows to the processing module, enabling data rows to be sorted in off-heap or on-heap memory using direct unsafe memory allocation. This new component manages memory pages for intermediate sort data, handles spilling to disk when memory is insufficient, and performs the actual sorting via TimSort, providing a more efficient alternative to previous sorting mechanisms for data load operations.

processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe · high confidence

Introduces CarbonTablePath utility for centralized table path construction

A new CarbonTablePath utility class has been added to centralize the construction and retrieval of paths for table metadata, segments, dictionaries, and stage data. This change provides a single source of truth for path logic, supporting features such as stage data management, dictionary file identification, and schema file location resolution, which improves consistency across the codebase.

core/src/main/java/org/apache/carbondata/core/util/path · high confidence

Introduces GeoHash-based spatial indexing and polygon filtering capabilities

This change adds a new GeoHash spatial index implementation and associated query processing logic to the geo module. Users can now define a geohash-type spatial index on longitude and latitude columns, which generates GeoHash IDs to enable efficient block and blocklet pruning during scans. The update includes new expression processors for polygon-based filtering (IN\_POLYGON, IN\_POLYGON\_LIST, IN\_POLYLINE\_LIST) and range-list queries, along with the underlying utility classes for coordinate conversion, quadtree processing, and geometry operations using the JTS library.

geo · high confidence

Introduces a new event listener bus for CarbonData operations

This change adds the core infrastructure for an event-driven architecture within CarbonData, introducing the \OperationListenerBus\ singleton, the \Event\ base class, the \OperationEventListener\ interface, and the \OperationContext\ holder. This allows the system to notify registered listeners when specific operations occur, enabling extensibility for monitoring, logging, or reacting to data loading and other internal events without modifying core logic.

core/src/main/java/org/apache/carbondata/events · high confidence

Introduces core filter expression and row interfaces

Adds new interfaces and enums to the scan filter package to support the query execution engine. This includes the \ExpressionType\ enum defining logical, comparison, and arithmetic operators (such as AND, OR, EQUALS, ADD), the \FilterExecutorType\ enum specifying execution modes like INCLUDE, EXCLUDE, and RANGE, and the \FilterOptimizer\ interface for optimizing filter expressions. Additionally, it introduces the \RowIntf\ interface and its \RowImpl\ implementation to provide a standardized way for the filter layer to access and manipulate row data during scans.

core/src/main/java/org/apache/carbondata/core/scan/filter/intf · high confidence

Introduces core vector and dictionary implementation classes

Adds CarbonColumnVectorImpl and CarbonDictionaryImpl to the core scan result vector package. CarbonColumnVectorImpl provides the underlying storage and access methods for vectorized data (supporting primitive types, decimals, strings, and complex array/struct structures) and integrates with lazy page loading and local dictionaries. CarbonDictionaryImpl implements the CarbonDictionary interface to manage dictionary values and metadata.

core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl · high confidence

Introduces new V3 column chunk implementation classes

The core data store chunk implementation now includes new classes to support the V3 file format: \AbstractDimensionColumnPage\ provides a base implementation for dimension column pages, while \FixedLengthDimensionColumnPage\ and \VariableLengthDimensionColumnPage\ handle fixed-size and variable-size dimension data respectively. \DimensionRawColumnChunk\ and \MeasureRawColumnChunk\ manage the raw data and decoding of dimension and measure column pages. These changes enable page-level reading and decoding, optimizing scan performance for the new V3 format.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl · high confidence

Introduces new table metadata schema classes and builders

The table metadata schema package now includes new model classes (CarbonTable, TableInfo, TableSchema, IndexSchema, RelationIdentifier) and builders (CarbonTableBuilder, TableSchemaBuilder) to define and construct table schemas. It also adds a PartitionType enum and Writable utilities to support serialization and partition type definitions.

core/src/main/java/org/apache/carbondata/core/metadata/schema/table · high confidence

Introduces unsafe on-heap and off-heap memory management for data loading and sorting

The core memory module now includes a new set of classes (CarbonUnsafe, HeapMemoryAllocator, UnsafeMemoryAllocator, UnsafeMemoryManager, UnsafeSortMemoryManager) ported from Apache Spark to manage direct memory access. This enables configurable off-heap memory usage for working and sort operations, providing a fallback to on-heap allocation when off-heap is unavailable or insufficient, and introduces memory pooling for heap allocations to improve performance during data loading and sorting tasks.

core/src/main/java/org/apache/carbondata/core/memory · high confidence

Introduces vectorized columnar reading for Presto queries

The Presto integration now supports vectorized (columnar) reads, allowing queries to fetch data in batches rather than row-by-row. This change adds new internal components—including \CarbonVectorBatch\, \PrestoCarbonVectorizedRecordReader\, and direct-fill vector wrappers—to handle columnar data transfer, alongside configuration options for batch sizes and unsafe memory usage. This optimization is intended to significantly improve query performance by reducing overhead and leveraging columnar processing capabilities within the Presto engine.

integration/presto/src/main · high confidence

Introduction of AbstractMeasureChunkReader for grouped measure chunk reading

A new abstract base class, AbstractMeasureChunkReader, has been added to the measure chunk reader package to standardize how measure data is read from blocklets. This class implements the core logic for reading raw measure chunks in groups based on column index ranges, allowing multiple columns to be read together to reduce I/O overhead, while leaving the specific decoding logic to subclasses via the abstract readRawMeasureChunksInGroup method.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/measure · high confidence

Introduction of BiDictionary interface for dictionary operations

A new BiDictionary interface has been added to the core development API, defining methods for bidirectional key-value lookups (getOrGenerateKey, getKey, getValue) and size reporting. This interface establishes the contract for dictionary-based data encoding and decoding during the loading process, enabling the system to manage mappings between dictionary keys and their corresponding values.

core/src/main/java/org/apache/carbondata/core/devapi · high confidence

Introduction of CarbonReadSupport interface for Hadoop data reading

A new CarbonReadSupport interface has been added to the Hadoop module to define how column data from a RecordReader is converted into row representations. This interface provides default methods for initialization based on projected columns, reading rows from column data, and cleanup, establishing the contract for data reading support within the CarbonData Hadoop integration.

hadoop/src/main/java/org/apache/carbondata/hadoop/readsupport · high confidence

Introduction of KeyGeneratorFactory for MDK key generation

A new KeyGeneratorFactory class has been added to the core key generator package. This factory provides a static method to retrieve a MultiDimKeyVarLengthGenerator instance based on dimension cardinality, handling both fully filled and non-fully filled bit configurations via CarbonUtil.

core/src/main/java/org/apache/carbondata/core/keygenerator/factory · high confidence

Introduction of Local Dictionary Generator for Column Data

This change introduces the \LocalDictionaryGenerator\ interface and its \ColumnLocalDictionaryGenerator\ implementation within the core module. These components provide the foundational logic for generating and managing local dictionaries for column data, including handling dictionary thresholds, storing dictionary values via a map-based store, and managing length variations for different data types. This enables the system to efficiently map column data to dictionary keys during query and load operations.

core/src/main/java/org/apache/carbondata/core/localdictionary/generator · high confidence

Introduction of LoggerAction enum for bad record handling

A new LoggerAction enum has been added to define the specific behaviors for handling bad records during data loading. This enum provides four distinct modes: FORCE, which converts bad data to null; REDIRECT, which moves bad records to a raw CSV file without conversion; IGNORE, which discards bad records without writing them; and FAIL, which causes the data load to abort if any bad records are detected. This change establishes the configuration options available for controlling bad record processing within the system.

common/src/main/java/org/apache/carbondata/common/constants · high confidence

Introduction of MDKey key generation infrastructure

The core module now includes the foundational classes for multi-dimensional key generation in the \org.apache.carbondata.core.keygenerator.mdkey\ package. This change introduces the \KeyGenerator\ interface and \KeyGenException\, along with the \AbstractKeyGenerator\ base class and the \MultiDimKeyVarLengthGenerator\ implementation, which handles the packing of multiple dimension keys into a single byte array using the new \Bits\ utility class. This provides the specific mechanism for generating and extracting variable-length multi-dimensional keys within the core library.

core/src/main/java/org/apache/carbondata/core/keygenerator/mdkey · high confidence

Introduction of QueryExecutor interface and factory for row-based reading

A new QueryExecutor interface and a corresponding QueryExecutorFactory have been added to the core scan executor package. The factory selects between VectorDetailQueryExecutor and DetailQueryExecutor based on the query model, enabling the execution of queries via a CarbonIterator for row-based CarbonRecordReader scenarios.

core/src/main/java/org/apache/carbondata/core/scan/executor · high confidence

Introduction of V3 Data Format Writer

Added the V3 format writer implementation (CarbonFactDataWriterImplV3) and its supporting data structures (BlockletDataHolder) in the processing module. This new writer supports configurable blocklet sizes via table properties, writes metadata including sort scope and application name to the V3 file footer, and manages page encoding and blocklet persistence according to the V3 data format specification.

processing/src/main/java/org/apache/carbondata/processing/store/writer/v3 · high confidence

Introduction of direct dictionary key generation for date and timestamp types

Added the DirectDictionaryGenerator interface and its factory (DirectDictionaryKeyGeneratorFactory) to the core key generator package. This new component enables direct dictionary key generation specifically for DATE and TIMESTAMP data types, instantiating DateDirectDictionaryGenerator and TimeStampDirectDictionaryGenerator respectively based on the column's data type.

core/src/main/java/org/apache/carbondata/core/keygenerator/directdictionary · high confidence

New BlockletFilterScanner and BlockletFullScanner implementations

The scan scanner implementation now includes two new classes: BlockletFilterScanner, which handles filter query processing by applying min-max pruning and filter executors before reading data, and BlockletFullScanner, which performs full blocklet scans for non-filter queries. These classes introduce configurable min-max filtering and BitSet pipeline optimization for filter evaluation, while also improving query statistics tracking for blocklet and page scanning metrics.

core/src/main/java/org/apache/carbondata/core/scan/scanner/impl · high confidence

New Bloom Filter Index implementation for blocklet-level pruning

The \index/bloom\ module now provides a complete Bloom Filter index implementation, introducing new classes for writing (\AbstractBloomIndexWriter\, \BloomIndexWriter\, \BloomIndexBuilder\), querying (\BloomCoarseGrainIndex\), and managing index files (\BloomIndexFileStore\). This change adds support for blocklet-level pruning using Bloom filters, allowing queries to skip irrelevant blocklets based on indexed column values. The implementation includes a dedicated cache (\BloomIndexCache\) to accelerate query performance and supports configurable bloom filter size, false positive probability, and compression. Additionally, it introduces file merging capabilities for bloom index shards to optimize storage and retrieval.

index/bloom · high confidence

New C++ SDK bindings for CarbonData read, write, and schema operations

The C++ SDK (CSDK) now provides a complete set of JNI-wrapped classes to interact with CarbonData from C++ applications. This includes CarbonReader for building and executing data reads with projection and batch configuration, CarbonWriter for configuring and executing data writes with sorting and load options, CarbonRow for accessing column data by ordinal, CarbonSchemaReader for inspecting file schemas, and supporting classes like Configuration and CarbonProperties for managing settings. Users can now perform full CRUD-style data operations directly from C++ code.

sdk/sdk · high confidence

New CLI and Distributed Index Server management scripts

Users can now start the CarbonData SQL command-line interface using the new \bin/carbon-spark-sql\ script, which wraps Spark SQL with CarbonData classes. Additionally, the \bin/start-indexserver.sh\ and \bin/stop-indexserver.sh\ scripts have been added to manage the Distributed Index Server process, allowing users to start and stop the server daemon via Spark's daemon utilities.

bin · high confidence

New CSV input processing components for CarbonData

This change introduces a new set of classes in the \processing/src/main/java/org/apache/carbondata/processing/loading/csvinput\ package to handle CSV data loading. The \CSVInputFormat\ and \CSVRecordReader\ provide a Hadoop MapReduce-compatible interface for reading CSV files, supporting configuration for delimiters, comments, quotes, escape characters, and header presence. It includes \BlockDetails\ to manage file splits and \BoundedInputStream\ to handle reading limits and line boundaries efficiently. Additionally, \StringArrayWritable\ enables the serialization of parsed CSV rows, and \CSVRecordReaderIterator\ wraps the record reader for easier iteration. These components collectively enable the system to parse and ingest CSV data into CarbonData tables.

processing/src/main/java/org/apache/carbondata/processing/loading/csvinput · high confidence

New CarbonRowReadSupport for date and timestamp conversion

A new CarbonRowReadSupport implementation has been added to the Hadoop read support layer. This component handles the conversion of raw data into CarbonRow objects, specifically applying logic to transform integer-based date values into java.sql.Date objects and long-based timestamp values into java.sql.Timestamp objects before returning the row.

hadoop/src/main/java/org/apache/carbondata/hadoop/readsupport/impl · high confidence

New Hadoop API for reading and writing CarbonData files

This change introduces a new set of Hadoop InputFormat and OutputFormat classes in the \org.apache.carbondata.hadoop.api\ package, enabling direct MapReduce integration with CarbonData. The new \CarbonTableInputFormat\ and \CarbonFileInputFormat\ allow users to query CarbonData tables by inferring schema from metadata or data file footers, supporting segment-level pruning and block filtering. The \CarbonTableOutputFormat\ and \CarbonOutputCommitter\ provide the corresponding write path, managing segment creation, table status updates, and transactional integrity during data loads.

hadoop/src/main/java/org/apache/carbondata/hadoop/api · high confidence

New Hadoop integration classes for CarbonData record reading and metrics

This change introduces the core Hadoop integration components for reading CarbonData files, including \AbstractRecordReader\, \CarbonRecordReader\, \CarbonMultiBlockSplit\, \CarbonProjection\, and the \InputMetricsStats\ interface. These classes enable the system to read data via Hadoop's \RecordReader\ API, support concurrent queries by wrapping multiple blocks into a single split, allow column projection, and track read statistics such as record counts and input bytes.

hadoop/src/main/java/org/apache/carbondata/hadoop · high confidence

New Hadoop utility classes for input formatting, split handling, and vectorized reading

This change introduces three new utility classes in the Hadoop module to support CarbonData integration with Hadoop MapReduce. CarbonInputFormatUtil provides helper methods to configure CarbonTableInputFormat with database/table identifiers and manage index job configurations. CarbonInputSplitTaskInfo implements the Distributable interface to manage task-specific input splits and determine optimal node placement based on data locality. CarbonVectorizedRecordReader enables vectorized query execution by reading data directly into CarbonColumnarBatches, improving query performance through columnar processing.

hadoop/src/main/java/org/apache/carbondata/hadoop/util · high confidence

New Hive integration utility classes for type conversion and table loading

Added two new utility classes, DataTypeUtil and HiveCarbonUtil, to the Hive integration module. DataTypeUtil provides a comprehensive mapping from Hive SQL data types (including complex types like arrays, maps, and structs) to CarbonData internal data types. HiveCarbonUtil introduces methods to construct CarbonLoadModel and CarbonTable instances from Hive metastore properties, handling schema inference, transactional table flags, and complex delimiter settings, thereby supporting the underlying read/write operations for Hive tables.

integration/hive/src/main/java/org/apache/carbondata/hive/util · high confidence

New Java SDK and SQL example programs for CarbonData

Added five new Java example programs in the examples/spark directory to demonstrate CarbonData usage. The SDK examples (CarbonReaderExample, SDKS3Example, SDKS3ReadExample, SDKS3SchemaReadExample) show how to write and read CarbonData files using the SDK API, including reading schemas and filtering data on S3 storage. The SQL example (JavaCarbonSessionExample) demonstrates creating a CarbonSession, loading data, and running various SQL queries including joins and aggregations.

examples/spark/src/main/java · high confidence

New PageLevelDictionary class for blocklet-level dictionary generation

A new PageLevelDictionary class has been added to the local dictionary module to manage unique dictionary values at the page level. This component generates dictionary surrogate keys for column data, merges values across pages, and produces encoded LocalDictionaryChunks required for writing blocklet-level dictionaries in CarbonData files. It supports various data types including VARCHAR and complex type primitives, utilizing configured compressors for efficient storage.

core/src/main/java/org/apache/carbondata/core/localdictionary · high confidence

New SQL exception types for schema, load options, and materialized views

The common module now includes a set of new SQL exception classes to provide more specific error reporting for schema evolution, data loading, and materialized view operations. Users will see distinct exceptions for invalid load options (InvalidLoadOptionException), malformed SQL commands (MalformedCarbonCommandException), and specific failures related to indexes (MalformedIndexCommandException, NoSuchIndexException) and materialized views (MalformedMVCommandException, NoSuchMVException). These changes improve the clarity of error messages when schema validation fails or when referenced indexes and materialized views are not found.

common/src/main/java/org/apache/carbondata/common/exceptions/sql · high confidence

New SQL parser and execution engine for Spark 2.3/2.4/3.1

This change introduces a new SQL parser and execution engine layer for CarbonData on Spark versions 2.3, 2.4, and 3.1. It adds a custom ANTLR grammar (CarbonSqlBase.g4) to support Carbon-specific SQL commands like MERGE INTO, DELETE, and UPDATE, alongside standard SQL. The implementation includes new Scala and Java classes for parsing (CarbonExtensionSqlParser, CarbonSparkSqlParser), AST building (CarbonSqlAstBuilder), and execution (CarbonDataSourceScanHelper, CarbonAnalyzer). This enables CarbonData to integrate with Spark's Catalyst optimizer and Tungsten execution engine, allowing users to query and modify CarbonData tables using standard Spark SQL interfaces.

integration/spark/src/main · high confidence

New Snappy compression safety and implicit column filtering support

The core module introduces a new SnappyCompressor that validates uncompressed data lengths to prevent integer overflow and excessive memory allocation attacks during decompression. It also adds implicit column filtering capabilities through new filter executors (CDCBlockImplicitExecutorImpl and ImplicitIncludeFilterExecutorImpl) that prune blocks and blocklets based on implicit column metadata, alongside new result iterators (DetailQueryResultIterator and VectorDetailQueryResultIterator) to handle detail query execution efficiently.

carbondata-core · high confidence

New Spark example applications for CarbonData operations

Added a suite of new Scala example applications in the Spark examples directory to demonstrate CarbonData capabilities, including performance benchmarking (SimpleQueryBenchmark), Alluxio integration, table alterations, DataFrame read/write operations, CarbonSession usage, sort column configurations, custom compaction, data management (segment deletion, major compaction, file cleanup), MERGE INTO SQL command support, UPSERT/DELETE/INSERT/UPDATE via both SQL and DataFrame APIs, and direct SQL execution on Carbon files without table creation.

examples/spark/src/main/scala · high confidence

New V3 data format and sort scope configuration support

This change introduces the constants and configuration options required for the new V3 data format, including blocklet group sizing and column-read batching, and adds support for a 'no\_sort' load scope alongside the existing local and global sort options. Users can now configure the new V3-specific parameters and choose the no\_sort strategy to optimize data loading performance.

core/src/main/java/org/apache/carbondata/core/constants · high confidence

New benchmarking and example applications for CarbonData

Added new Scala examples and benchmarks to the repository, including a ConcurrentQueryBenchmark for testing query performance against Parquet, ORC, and CarbonData formats, an SCDType2Benchmark demonstrating Slowly Changing Dimension Type 2 updates, and several usage examples for Change Data Capture (CDC), data deduplication, complex types, and streaming ingestion via both Spark Streaming DStreams and Structured Streaming.

repository · high confidence

New block metadata and caching infrastructure classes

This change introduces a new set of core classes in the \core/src/main/java/org/apache/carbondata/core/datastore/block\ package to manage block metadata, segment properties, and task distribution. \AbstractIndex\ provides a base implementation for index structures with access counting and memory size tracking. \SegmentProperties\ and \SegmentPropertiesAndSchemaHolder\ handle the definition of block columns (dimensions, measures, complex types) and provide a singleton holder for caching segment properties to avoid redundant construction. \TableBlockInfo\ and \TaskBlockInfo\ define the structure for passing block details (file paths, offsets, lengths, locations) from the driver to executors, with \TableBlockInfo\ implementing the new \Distributable\ interface to support locality-aware task distribution.

core/src/main/java/org/apache/carbondata/core/datastore/block · high confidence

Added BlockletInfo and DataFileFooter classes to store detailed metadata about blocklets and data files, including row counts, chunk offsets, page information, sorting status, and schema versioning, enabling more granular control over data layout and compatibility.

core/src/main/java/org/apache/carbondata/core/metadata/blocklet · high confidence

New columnar block indexer storage classes for optimized encoding

The columnar datastore module introduces a new set of classes to handle block indexing with optimized encoding strategies. A new abstract base class, BlockIndexerStorage, provides the foundation for storing row ID and data pages, including logic for run-length encoding (RLE) on data and sequential number compression on row IDs. Specific implementations are added for different data types: ByteArrayBlockIndexerStorage and ByteArrayBlockIndexerStorageWithoutRowId for byte arrays, and ObjectArrayBlockIndexerStorage for generic object arrays, each handling sorting and encoding based on configuration. Helper classes like ByteArrayColumnWithRowId and ObjectColumnWithRowId support the sorting and comparison logic required for these encodings. Additionally, a DummyBlockIndexerStorage is provided for cases where no indexing is needed, and an UnBlockIndexer utility class is added to decompress the encoded index and data structures during query execution.

core/src/main/java/org/apache/carbondata/core/datastore/columnar · high confidence

New complex-type and JSON row parsers for data loading

The data loading pipeline now includes dedicated parsers for complex data structures and JSON input. ArrayParserImpl, MapParserImpl, and StructParserImpl handle string-based parsing for arrays, maps, and structs respectively, supporting custom delimiters and null formats. JsonRowParser enables direct ingestion of JSON rows, converting JSON fields into CarbonData-compatible objects including nested arrays and structs. RowParserImpl and RangeColumnParserImpl coordinate these parsers during row processing, with RowParserImpl handling column mapping and spatial index column exclusion. PrimitiveParserImpl provides a pass-through parser for simple types.

processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl · high confidence

New conditional filter expression implementations added

The scan engine now includes a comprehensive set of new conditional expression classes in the core module to support richer filtering capabilities. This adds implementations for equality (EqualTo, NotEquals), comparison (GreaterThan, GreaterThanEqualTo, LessThan, LessThanEqualTo), set membership (In, NotIn), and pattern matching (StartsWith). It also introduces specialized expressions for implicit filtering (ImplicitExpression, CDCBlockImplicitExpression) to support block-level pruning and Change Data Capture scenarios, along with a base ConditionalExpression interface and a ListExpression to handle collections of filter predicates.

core/src/main/java/org/apache/carbondata/core/scan/expression/conditional · high confidence

New constants for data loading configuration and optimized insert flow

A new constants class, DataLoadProcessorConstants, has been added to the processing module to centralize configuration keys for data loading operations. This includes flags for handling bad records (logging enablement, action, and empty data detection), serialization formats, and complex delimiters. Notably, it introduces the NO\_REARRANGE\_OF\_ROWS constant, which signals an optimized insert flow that avoids rearranging data rows, addressing performance issues in global sort scenarios with large datasets.

processing/src/main/java/org/apache/carbondata/processing/loading/constants · high confidence

New core datastore interfaces and metadata classes for data access and table specification

This change introduces several new foundational classes in the core datastore package that define how data is read, referenced, and structured. The \FileType\ enum (inferred from context, though \ColumnType\ is explicitly shown) and \ColumnType\ enum categorize column encoding strategies (e.g., Global Dictionary, Direct Dictionary, Plain Value, Complex). The \DataRefNode\ interface defines the contract for accessing data blocks, including methods to read dimension and measure chunks, retrieve min/max values for filtering, and access indexed data. The \FileReader\ interface abstracts low-level file I/O operations, supporting both byte array and ByteBuffer reads, and introduces a \setReadPageByPage\ flag to enable page-by-page reading for memory-constrained scenarios. Additionally, \ReusableDataBuffer\ provides a mechanism to resize and reuse byte buffers to reduce allocation overhead, while \TableSegmentUniqueIdentifier\ and \TableSpec\ handle the unique identification of table segments and the specification of dimension/measure layouts, including logic to rearrange partition columns to the end of the sort order.

core/src/main/java/org/apache/carbondata/core/datastore · high confidence

New core expression evaluation framework for query filters

The scan expression module has been refactored to introduce a new expression evaluation architecture. This change adds foundational classes including \Expression\, \BinaryExpression\, \LeafExpression\, \ColumnExpression\, and \LiteralExpression\ to represent query filter components. It also introduces \ExpressionResult\ for handling typed evaluation outputs and \RangeExpressionEvaluator\ to optimize filter logic by transforming separate greater-than/less-than conditions into efficient range filters. Additionally, \MatchExpression\ is added to support text matching capabilities.

core/src/main/java/org/apache/carbondata/core/scan/expression · high confidence

New core file readers for delete deltas, dictionaries, and metadata

The core reader module now includes new components to support delete delta processing and structured metadata access. CarbonDeleteDeltaFileReader and its implementation allow queries to read and exclude deleted records, while CarbonDeleteFilesDataReader handles concurrent reading of multiple delta files to aggregate deleted row data. Additionally, new readers for dictionary metadata (CarbonDictionaryReader, CarbonDictionaryColumnMetaChunk), file headers (CarbonHeaderReader), and file footers (CarbonFooterReader, CarbonFooterReaderV3) have been added, along with a generic ThriftReader to facilitate the parsing of these binary metadata structures.

core/src/main/java/org/apache/carbondata/core/reader · high confidence

New core mutation utility classes for update and delete delta handling

The \core/src/main/java/org/apache/carbondata/core/mutate\ package now includes a set of new utility and value-object classes to support internal update and delete delta operations. \CarbonUpdateUtil\ provides static methods for parsing tuple IDs, constructing block paths, and managing segment update status files. Supporting value objects include \CdcVO\, \UpdateVO\, \DeleteDeltaVo\, \FilePathMinMaxVO\, \SegmentUpdateDetails\, and \DeleteDeltaBlockDetails\/\DeleteDeltaBlockletDetails\ to track block-level deletion details and update metadata. \TupleIdEnum\ defines the structure for tuple IDs. These changes introduce new internal data structures and logic for handling mutations, likely improving performance and correctness for update/delete operations.

core/src/main/java/org/apache/carbondata/core/mutate · high confidence

New developer tooling and configuration files added

The \dev\ directory now includes a Python script (\carbon\_pr.py\) and documentation (\carbon-pr-readme.md\) to automate the process of reviewing and merging GitHub pull requests, including squashing commits and updating JIRA tickets. Additionally, new configuration files have been added to enforce code quality and style standards: a Checkstyle configuration (\javastyle-config.xml\) with suppressions (\javastyle-suppressions.xml\), an IntelliJ IDEA code format template (\java-code-format-template.xml\), a FindBugs exclusion filter (\findbugs-exclude.xml\), and a standard Java license header (\java.header\).

dev · high confidence

New direct dictionary generators for Date and Timestamp types

This change introduces a new package \org.apache.carbondata.core.keygenerator.directdictionary.timestamp\ containing \AbstractDirectDictionaryGenerator\, \DateDirectDictionaryGenerator\, and \TimeStampDirectDictionaryGenerator\. These classes provide the logic for converting Date and Timestamp string values into direct dictionary surrogate keys. The Date generator enforces valid date ranges (0001-01-01 to 9999-12-31) and stores keys as days offset from a cutoff. The Timestamp generator supports configurable time granularity (seconds, minutes, hours, days) and a customizable cutoff timestamp via CarbonProperties, allowing for more efficient storage and retrieval of timestamp data by mapping time values to integer keys.

core/src/main/java/org/apache/carbondata/core/keygenerator/directdictionary/timestamp · high confidence

New distributed locking framework with configurable backends

CarbonData now uses a new locking subsystem in the core module to manage concurrent access to tables and metadata. The system supports multiple lock backends—local file locks, HDFS/ViewFS, S3, Alluxio, ZooKeeper, and custom implementations—selected automatically based on the file system type or explicitly configured via the \carbon.lock.type\ property. Lock files are stored in a configurable path (\carbon.lock.path\), and the framework includes retry logic with configurable attempts and timeouts to handle transient lock acquisition failures. This change introduces new classes such as \CarbonLockFactory\, \AbstractCarbonLock\, and specific implementations like \HdfsFileLock\ and \ZooKeeperLocking\ to replace previous ad-hoc locking mechanisms, improving reliability in distributed environments.

core/src/main/java/org/apache/carbondata/core/locks · high confidence

New exception classes for data loading errors

The data loading module now includes three new exception classes: CarbonDataLoadingException, BadRecordFoundException, and NoRetryException. These classes provide specific error handling for loading operations, allowing users to catch and handle data loading failures more precisely.

processing/src/main/java/org/apache/carbondata/processing/loading/exception · high confidence

New exception classes for data loading, matching, and slice merging

The processing module now includes three new exception classes: DataLoadingException, MultipleMatchingException, and SliceMergerException. These allow the system to throw specific, typed errors for data loading failures, ambiguous matches, and slice merger issues, enabling more precise error handling and debugging for users encountering these processing stages.

processing/src/main/java/org/apache/carbondata/processing/exception · high confidence

New exception classes for data writing and index building

Added CarbonDataWriterException and IndexBuilderException classes to the core datastore exception package. CarbonDataWriterException extends RuntimeException to handle errors during data writing operations, while IndexBuilderException extends IOException for block builder failures. Both classes provide constructors for error messages and cause exceptions, along with methods for retrieving localized and standard error messages.

core/src/main/java/org/apache/carbondata/core/datastore/exception · high confidence

New filter executor architecture with include/exclude and boolean logic support

The scan filter execution layer has been restructured to introduce dedicated executors for include and exclude filters, enabling more granular control over row selection during scans. This change adds \IncludeFilterExecutorImpl\ and \ExcludeFilterExecutorImpl\ to handle positive and negative matching respectively, supported by a \BitSetUpdaterFactory\ that manages BitSet operations for inclusion (setting bits) and exclusion (flipping bits). Additionally, new executors for boolean logic (\AndFilterExecutorImpl\, \OrFilterExecutorImpl\) and a \FalseFilterExecutor\ allow for complex filter composition and early termination of scans. The \RangeValueFilterExecutorImpl\ and \RestructureEvaluatorImpl\ are also introduced to handle range-based filtering and restructure scenarios, while new interfaces like \ImplicitColumnFilterExecutor\ and \FilterBitSetUpdater\ define the contract for these operations. This refactoring aims to improve filter performance and flexibility by decoupling filter logic into specialized components.

core/src/main/java/org/apache/carbondata/core/scan/filter/executer · high confidence

New filter resolver implementations for conditional and row-level filtering

The core scan filter resolver package introduces new classes to handle filter expression resolution: \ConditionalFilterResolverImpl\ resolves binary and conditional expressions into dictionary-based surrogate info for dimensions and measures; \LogicalFilterResolverImpl\ manages logical (AND/OR) filter trees by delegating to left and right evaluators; \RowLevelFilterResolverImpl\ resolves filters for row-level execution, collecting dimension and measure filter info lists and supporting a limit pushdown; and \RowLevelRangeFilterResolverImpl\ extends conditional resolution to support range-based filtering, extracting no-dictionary range values and measure range values while handling data type conversions and sorting. These changes enable more granular and efficient filter evaluation during query scans.

core/src/main/java/org/apache/carbondata/core/scan/filter/resolver · high confidence

New index and delete-delta file writers with atomic operation support

The core writer module introduces new classes to handle specific file-writing tasks: CarbonDeleteDeltaWriter and its implementation now write delete delta records as JSON, while CarbonIndexFileWriter and ThriftWriter provide a dedicated path for serializing Thrift objects. ThriftWriter is enhanced to support atomic file operations, ensuring that index file writes are performed safely. Additionally, CarbonIndexFileMergeWriter is added to merge segment index files, including support for partitioned tables.

core/src/main/java/org/apache/carbondata/core/writer · high confidence

New index metadata classes and index type enumeration

Added BlockIndexInfo to store block-level index metadata including row count, file name, offset, blocklet index, blocklet info, and file size. Added IndexType enum to define supported index types (Lucene, BloomFilter, Secondary Index) with their corresponding factory class names and short names, enabling lookup of index providers by name.

core/src/main/java/org/apache/carbondata/core/metadata/index · high confidence

New index metadata schema classes for secondary index support

The core metadata schema now includes IndexMetadata and IndexTableInfo classes to manage secondary index properties, parent table references, and index status. This enables the system to store and retrieve detailed index configuration, supporting features like index renaming and status tracking for secondary indexes used in Presto queries.

core/src/main/java/org/apache/carbondata/core/metadata/schema/indextable · high confidence

New index store data structures and serialization support

This change introduces a set of new classes in the index store module to support detailed blocklet metadata, distributed index server communication, and memory-efficient data map storage. Specifically, it adds \Blocklet\, \BlockMetaInfo\, and \BlockletDetailInfo\ to hold block and blocklet metadata, including file paths, sizes, and schema information. It also introduces \ExtendedBlocklet\ and its wrappers (\ExtendedBlockletWrapper\, \ExtendedBlockletWrapperContainer\) to handle the serialization and deserialization of blocklet splits between executors and drivers, supporting features like count-star optimization and CDC. Additionally, \AbstractMemoryDMStore\, \SafeMemoryDMStore\, and \UnsafeMemoryDMStore\ provide the foundation for in-memory data map storage, while \BlockletIndexStore\ manages the caching and loading of blocklet indexes using an LRU cache. \PartitionSpec\ is added to handle partition location metadata, and \BlockletDetailsFetcher\ defines the interface for fetching detailed blocklet information.

core/src/main/java/org/apache/carbondata/core/indexstore · high confidence

New index writing infrastructure for coarse-grain and secondary indexes

The processing module now includes a new IndexWriterListener and IndexWriterException to support the creation and management of coarse-grain (CG) and secondary indexes (SI). The listener registers index writers for non-lazy, non-secondary indexes during data load, delegates block and page events to these writers, and handles index finalization, enabling the system to build these specific index types during the insert stage.

processing/src/main/java/org/apache/carbondata/processing/index · high confidence

New legacy dimension index codec implementations

The core datastore module now includes a new set of legacy dimension index codecs (Complex, DirectDict, Plain, and their supporting base classes) in the \legacy\ package. These classes provide specific encoding strategies for dimension data, handling operations such as flattening byte arrays, applying column compression, and managing row ID and RLE pages, thereby expanding the available encoding options for dimension columns.

core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/dimension · high confidence

New load lifecycle event classes for pre/post execution and status updates

The \LoadEvents.java\ file introduces four new event classes (\LoadTablePreExecutionEvent\, \LoadTablePostExecutionEvent\, \LoadTablePreStatusUpdateEvent\, and \LoadTablePostStatusUpdateEvent\) within the processing module. These classes provide structured hooks for listeners to intercept and react to specific stages of the data loading process, such as validation before execution, loading materialized views after data load, and status updates before final commit. This enables external components to perform custom logic at key points in the load lifecycle without modifying the core loading logic.

processing/src/main/java/org/apache/carbondata/processing/loading/events · high confidence

New logging service with audit and statistic capabilities

The common module now provides a dedicated logging API via LogService and LogServiceFactory. LogService extends Log4j Logger and adds specific methods for audit logging (which includes hostname, username, and thread ID) and statistic logging, allowing users to track processing metrics and security events alongside standard log levels.

common/src/main/java/org/apache/carbondata/common/logging · high confidence

New logical filter expression types and exception classes

Added new logical expression classes (AndExpression, OrExpression, RangeExpression, TrueExpression, FalseExpression) and their base class (BinaryLogicalExpression) to the core scan expression package, along with FilterIllegalMemberException and FilterUnsupportedException. These changes enable the query engine to evaluate AND, OR, and range-based filter conditions, as well as handle specific filter evaluation errors gracefully without interrupting execution.

core/src/main/java/org/apache/carbondata/core/scan/expression/logical · high confidence

New metadata schema classes for partitioning, bucketing, and schema evolution

The core metadata schema package now includes new model classes to support advanced table features. PartitionInfo defines the structure for partitioned tables, including support for range and list partitioning types. BucketingInfo and SortColumnRangeInfo provide the necessary metadata for column bucketing and user-specified sort column bounds, respectively. Additionally, SchemaEvolution and SchemaEvolutionEntry classes have been added to persist and track schema restructuring history (added and removed columns), while SchemaReader provides utilities to read and infer table schemas from storage.

core/src/main/java/org/apache/carbondata/core/metadata/schema · high confidence

New parallel merge sort implementations for data loading

Added new sort implementations (ParallelReadMergeSorterImpl, UnsafeParallelReadMergeSorterImpl, and their column-range variants) that perform merge sorting in parallel using a thread pool, replacing the previous single-threaded approach. This change improves data loading performance by concurrently processing row batches and managing temporary sort files, with specific support for sorting data within specified column value ranges (e.g., for bucketing or sort\_column\_bounds).

processing/src/main/java/org/apache/carbondata/processing/loading/sort/impl · high confidence

New parser interfaces and factory for complex data types

The loading parser module now introduces a new architecture for handling complex data types (Array, Map, Struct) via the \GenericParser\ and \ComplexParser\ interfaces, along with a \RowParser\ for row-level processing. A new \CarbonParserFactory\ has been added to instantiate specific parsers (\ArrayParserImpl\, \MapParserImpl\, \StructParserImpl\, \PrimitiveParserImpl\) based on the column's data type, enabling recursive parsing of nested structures using configurable delimiters.

processing/src/main/java/org/apache/carbondata/processing/loading/parser · high confidence

New partitioning implementations for data loading

The data loading process now includes new partitioner implementations in the processing module. A generic Partitioner interface is introduced alongside specific implementations: HashPartitionerImpl for standard hash-based distribution, RangePartitionerImpl for range-based distribution using binary search, RawRowComparator for comparing raw CarbonRows during sorting, and SparkHashExpressionPartitionerImpl which uses Murmur3\_x86\_32 hashing to ensure compatibility with Spark's hash expressions for bucket tables.

processing/src/main/java/org/apache/carbondata/processing/loading/partition · high confidence

New query and restructure utility classes for executor support

Added two new utility classes, QueryUtil and RestructureUtil, to the scan executor package. QueryUtil provides helper methods for determining dimension and measure block indexes during query execution, handling sort dimension mapping, and managing masked keys. RestructureUtil supports table restructuring operations by creating dimension information, matching query dimensions against existing table blocks (including complex types), and filling new datatypes for complex children, ensuring correct data projection when table schemas change.

core/src/main/java/org/apache/carbondata/core/scan/executor/util · high confidence

New query statistics recording and logging framework

This change introduces a new core statistics infrastructure in the \org.apache.carbondata.core.stats\ package to capture and report query performance data. It adds a \QueryStatisticsRecorder\ interface with a dummy implementation (for environments where stats are disabled) and a concrete \QueryStatisticsRecorderImpl\ that collects timing and count metrics for query phases (such as SQL parsing, block loading, and scanning) and logs them. The system defines \QueryStatisticsConstants\ for metric names, a \QueryStatistic\ model to hold individual phase data, and a \TaskStatistics\ class that aggregates these into a structured table output including scan counts, I/O times, and result preparation metrics. A \QueryStatisticsModel\ manages the recorder instance and enables/disables collection via configuration.

core/src/main/java/org/apache/carbondata/core/stats · high confidence

New query type handlers for complex data structures

The core scan engine now includes dedicated query type classes for handling complex data structures, specifically arrays, maps, structs, and primitives. These new components (ArrayQueryType, MapQueryType, StructQueryType, PrimitiveQueryType) provide the logic to decode, parse, and retrieve data from complex column pages during query execution, enabling the system to correctly process nested data types like Array\<Struct\<key,Value\>\> for maps and multi-level complex columns.

core/src/main/java/org/apache/carbondata/core/scan/complextypes · high confidence

New result scanning classes for blocklet and row batch processing

The scan result module introduces two new classes: BlockletScannedResult, which manages the storage and retrieval of scanned blocklet data including dimension and measure column pages, reusable buffers, and delete deltas; and RowBatch, which provides an iterator interface for accessing query result rows. These classes form the foundation for the query result handling layer in the scan process.

core/src/main/java/org/apache/carbondata/core/scan/result · high confidence

New row batch and intermediate sort temp row structures for data loading

The data loading module now introduces two new classes to handle row processing during sort operations. CarbonRowBatch provides a batched container for CarbonRow objects, supporting iteration, addition, removal, and rewinding to facilitate row reuse. IntermediateSortTempRow structures the data written to sort temp files during the sort procedure, separating dictionary-sorted dimensions, non-dictionary-sorted dimensions, and measures/no-sort dimensions to support both intermediate and final merger phases.

processing/src/main/java/org/apache/carbondata/processing/loading/row · high confidence

New row data structures and write-step utilities for complex and primitive data types

This change introduces three new classes in the core datastore row package to support the handling of complex data types (such as STRUCT, ARRAY, MAP) and primitive types during the write path. CarbonRow serves as a container for row data, separating schema columns, raw data, and range identifiers. ComplexColumnInfo provides a wrapper to hold metadata for complex columns, including their type, data types, names, and dictionary status. WriteStepRowUtil provides specific logic to construct and parse CarbonRow instances during the write step, organizing data into dictionary dimensions, no-dictionary/complex columns, and measures, which facilitates the new adaptive encoding and compaction support for complex types.

core/src/main/java/org/apache/carbondata/core/datastore/row · high confidence

New safe dimension data chunk stores and vector fillers for direct reading

Added a new set of classes in the \core.datastore.chunk.store.impl.safe\ package to support direct vector reading of dimension data. This includes \SafeAbstractDimensionDataChunkStore\ and its implementations (\SafeFixedLengthDimensionDataChunkStore\, \SafeVariableLengthDimensionDataChunkStore\, \SafeVariableIntLengthDimensionDataChunkStore\, \SafeVariableShortLengthDimensionDataChunkStore\) which handle storing and retrieving dimension data chunks, including support for inverted indexes. Additionally, \AbstractNonDictionaryVectorFiller\ and its factory \NonDictionaryVectorFillerFactory\ provide specialized fillers for various data types (String, Timestamp, Boolean, Int, Long, etc.) to efficiently populate \CarbonColumnVector\ objects, optimizing performance for direct scan queries.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe · high confidence

New scan result iterator components for row-based processing and column drift handling

Added several new iterator classes in the core scan result iterator package to support row-based query execution and data restructuring. RawResultIterator introduces asynchronous prefetching of row batches to improve read performance during compaction and queries. ChunkRowIterator and PartitionSplitterRawResultIterator provide mechanisms to iterate over row batches and split partitions into individual rows. ColumnDriftRawResultIterator handles schema evolution by mapping and converting columns when source and destination segment properties differ, ensuring correct data types and column ordering for no-dictionary columns.

core/src/main/java/org/apache/carbondata/core/scan/result/iterator · high confidence

New streaming module with row-based ingestion and segment management

This change introduces the core streaming ingestion framework in the \streaming\ module, enabling real-time data loading into CarbonData. It adds the \CarbonStreamRecordWriter\ and \StreamBlockletWriter\ to handle row-by-row data processing, compression, and blocklet creation, alongside \StreamSegment\ to manage the lifecycle (open, close, finish) of streaming segments. The module also includes a pluggable parser interface (\CarbonStreamParser\) with default implementations for CSV and Spark SQL Rows, and \SparkStreamingUtil\ adapters to support both Spark 2.x and 3.1 runtimes.

streaming · high confidence

New unsafe in-memory and file-based sort chunk holders for data loading

The data loading sort process now introduces a new set of holder classes in the \processing\ module to manage temporary sort data more efficiently. \SortTempChunkHolder\ defines the interface for merging temporary sort files and in-memory data, implemented by \UnsafeFinalMergePageHolder\ for merging in-memory pages, \UnsafeInmemoryHolder\ for single in-memory row pages, \UnsafeInmemoryMergeHolder\ for merging unsafe in-memory intermediate data, and \UnsafeSortTempFileChunkHolder\ for reading and prefetching from sorted temporary files. These components replace previous mechanisms for handling sort chunks, enabling optimized reading, writing, and merging of sort temp rows during data load operations.

processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/holder · high confidence

New unsafe row comparators for data loading sort

Added UnsafeRowComparator and UnsafeRowComparatorForNormalDims to the processing module's unsafe sort comparator package. These classes implement Comparator\<UnsafeCarbonRow\> to enable direct comparison of in-memory rows during data loading, supporting both dictionary-encoded and non-dictionary-encoded columns via unsafe memory access.

processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/comparator · high confidence

New value objects for block and segment mapping details

Added BlockMappingVO and RowCountDetailsVO classes to the core mutate data package. BlockMappingVO stores mappings for block row counts, segment block counts, and block-to-segment relationships, while RowCountDetailsVO holds total and deleted row counts for a block. These value objects provide structured data containers for tracking segment and block metadata during mutation operations.

core/src/main/java/org/apache/carbondata/core/mutate/data · high confidence

New wrapper classes for query scan data handling

Added ByteArrayWrapper and IntArrayWrapper classes in the core scan wrappers package. ByteArrayWrapper stores dimension column data (dictionary keys, no-dictionary keys, and complex type keys) as byte arrays and implements Comparable, Serializable, equals, hashCode, and compareTo for use as aggregation keys. IntArrayWrapper provides a simple wrapper for int\[\] data with proper equals and hashCode implementations.

core/src/main/java/org/apache/carbondata/core/scan/wrappers · high confidence

Schema synchronization between Spark and Hive via MetastoreListener

A new CarbonHiveMetastoreListener has been added to automatically sync table schemas between Spark and Hive. When a table created via Spark (identified by the CarbonSource provider) is created or altered in the Hive Metastore, this listener intercepts the event to reconstruct the Hive schema from Spark's JSON-encoded schema parts, ensuring Hive sees the correct column definitions, input/output formats, and storage handler settings.

integration/hive/src/main/scala · high confidence

Segment-level min/max metadata for query pruning

The core module now maintains segment-level min and max values for columns, enabling more effective data pruning and reduced driver memory usage during queries. This is implemented via new metadata classes in the segmentmeta package: BlockColumnMetaDataInfo stores per-block column schemas and min/max byte arrays; SegmentColumnMetaDataInfo holds segment-level min/max values along with sort column and column drift flags; SegmentMetaDataInfo aggregates these per column; and SegmentMetaDataInfoStats acts as a singleton cache that aggregates min/max values across all blocks within a segment, allowing the query engine to skip irrelevant segments based on these bounds.

core/src/main/java/org/apache/carbondata/core/segmentmeta · high confidence

Streaming segment pruning with min/max index support

The streaming module now supports pruning stream segments based on min/max index data, allowing the system to skip segments that do not match query filters. This is enabled by new classes in the core stream package: StreamFile, which holds segment metadata and min/max index information; StreamPruner, which evaluates filter conditions against these indices to determine which segments require scanning; and ExtendedByteArray/InputStream/OutputStream classes that provide low-level stream manipulation capabilities used during index reading and processing.

core/src/main/java/org/apache/carbondata/core/stream · high confidence

Support for direct vector filling with inverted index and delete delta

The direct-read vector implementation now supports queries involving inverted indexes and delete deltas. New classes in the \directread\ package—\ColumnarVectorWrapperDirectWithInvertedIndex\, \ColumnarVectorWrapperDirectWithDeleteDelta\, and \ColumnarVectorWrapperDirectWithDeleteDeltaAndInvertedIndex\—allow column vectors to be filled directly from column pages while filtering out deleted rows and mapping data via inverted indices. A factory (\ColumnarVectorWrapperDirectFactory\) automatically selects the appropriate wrapper based on the presence of these structures, and a new \AbstractCarbonColumnarVector\ base class provides a common foundation for these specialized vector wrappers.

core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread · high confidence

Support for local dictionary encoded dimension data

Added LocalDictDimensionDataChunkStore to handle dimension data chunks that use local dictionary encoding. This new component wraps an existing dimension chunk store and a CarbonDictionary instance, enabling the system to correctly fill column vectors and retrieve rows by resolving surrogate keys to their actual dictionary values. This change supports features such as local dictionary queries and improves compatibility with engines like Presto that rely on dictionary-encoded data for complex types and null handling.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl · high confidence

Support for time-series pre-aggregation granularity and functions

The pre-aggregation module now includes core components to support time-series data handling, specifically enabling the creation of time-series aggregations alongside main tables. This change introduces a \TimeSeriesUDF\ class that applies various time-granularity functions (such as second, minute, hour, day, week, month, and year) to timestamp data, along with supporting enums for \DaysOfWeekEnum\ and \TimeSeriesFunctionEnum\. These additions allow users to define and query pre-aggregated data at specific time intervals, with configurable first-day-of-week settings.

core/src/main/java/org/apache/carbondata/core/preagg · high confidence

Unsafe intermediate and final data merging for data loading

The data loading process now uses a new set of unsafe (direct memory) merger components to handle intermediate and final sort steps. This includes \UnsafeIntermediateMerger\ to coordinate merging of in-memory pages and sort temp files, \UnsafeInMemoryIntermediateDataMerger\ for merging in-memory row pages (with optional spilling to disk based on memory configuration), \UnsafeIntermediateFileMerger\ for merging intermediate sort temp files, and \UnsafeSingleThreadFinalSortFilesMerger\ to perform the final merge of all sorted data sources. These changes replace or supplement previous merging logic with implementations optimized for direct memory access, aiming to improve data loading performance and memory management during the sort phase.

processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger · high confidence

Unsafe off-heap storage for dimension data chunks

Added a new set of classes in the unsafe datastore package to store dimension data in off-heap memory using direct memory access. This includes an abstract base class and specific implementations for fixed-length, variable-length (short and int), and variable-int-length dimension data. These stores handle the allocation, population, and retrieval of dimension data and inverted indices directly in memory, aiming to improve query performance and reduce garbage collection pressure by avoiding heap allocation for large data sets.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/unsafe · high confidence

Architecture

Refactored column metadata schema into dedicated model classes

The column metadata schema in the core module has been restructured to improve clarity and maintainability. The previous monolithic approach is replaced by a hierarchy of specific classes: \CarbonColumn\ serves as the base, with \CarbonDimension\ and \CarbonMeasure\ handling dimension and measure specifics respectively, and \CarbonImplicitDimension\ managing internal columns like \tupleId\. \ColumnSchema\ now centralizes metadata storage, including support for complex types, local dictionaries, and time-series functions, while \ParentColumnTableRelation\ explicitly tracks relationships between child and parent table columns.

core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column · high confidence

Refactored column page encoding and decoding into a new codec-based architecture

The core data store encoding logic has been restructured to use a new interface-driven design. New interfaces \ColumnPageCodec\, \ColumnPageEncoder\, and \ColumnPageDecoder\ define the contract for encoding and decoding column pages, replacing the previous monolithic implementation. A new \DefaultEncodingFactory\ now selects specific codecs (such as \AdaptiveIntegralCodec\, \AdaptiveFloatingCodec\, \DirectCompressCodec\, and \RLECodec\) based on column data types and statistics. This change introduces \ColumnPageEncoderMeta\ to handle metadata serialization for various data types (including decimals and timestamps) and \EncodedColumnPage\ to wrap the resulting binary data and metadata, improving modularity and maintainability of the compression and encoding strategies.

core/src/main/java/org/apache/carbondata/core/datastore/page/encoding · high confidence

Refactored query execution into a new executor hierarchy

The scan executor implementation has been restructured to introduce an abstract base class, AbstractQueryExecutor, which centralizes query property initialization and block management. Two concrete executors have been added: DetailQueryExecutor for standard row-based queries and VectorDetailQueryExecutor for columnar vector-based queries. This change decouples the execution logic from the property setup, providing a cleaner foundation for query processing.

core/src/main/java/org/apache/carbondata/core/scan/executor/impl · high confidence

Behavioural changes

Added encoding type validation for data file compatibility

The new Encoding enum in the metadata encoder package now includes a validateEncodingTypes method that checks whether the encodings stored in a data file are supported by the current version. If a file contains an unsupported or mismatched encoding type, the system throws an UnsupportedOperationException with a clear message, preventing silent failures or data corruption during read operations.

core/src/main/java/org/apache/carbondata/core/metadata/encoder · high confidence

Data type system refactored from enum to class hierarchy

The internal data type representation has been migrated from a simple enum to a class-based hierarchy (DataType and its subclasses like ArrayType, DecimalType, etc.). This change introduces support for complex types (arrays, structs, maps) and adds precision and scale attributes to DecimalType, while also implementing custom serialization adapters to ensure backward compatibility when deserializing table metadata from older versions.

core/src/main/java/org/apache/carbondata/core/metadata/datatype · high confidence

Decoder-based fallback mechanism for local dictionary encoding

The blocklet encoding logic now supports a decoder-based fallback mechanism to reduce memory footprint when handling columns that are not fully encoded with local dictionaries. If a blocklet contains a mix of pages encoded with and without local dictionaries, the system can asynchronously re-encode the dictionary-encoded pages using a decoder instead of retaining the dictionary metadata, thereby freeing memory. This behavior is controlled by the \isDecoderBasedFallBackEnabled\ flag passed to the new \BlockletEncodedColumnPage\ and \EncodedBlocklet\ classes, which manage the encoding and fallback execution for dimension and measure columns.

core/src/main/java/org/apache/carbondata/core/datastore/blocklet · high confidence

Introduce CarbonCache as the unified caching layer

The core module now uses CarbonCache instead of Guava Cache for managing data maps and blocklet indexes. This change introduces a new \Cache\ interface and a \CacheProvider\ that instantiates \CarbonLRUCache\ instances backed by ExpiringMap, allowing for configurable LRU eviction based on memory size and access counts. Users benefit from more predictable memory usage on executors, where the LRU cache size can be set via a percentage of total executor memory (defaulting to 70% if not explicitly configured), and from the ability to invalidate specific cache entries or clear access counts to manage resource cleanup more effectively.

core/src/main/java/org/apache/carbondata/core/cache · high confidence

Introduce lightweight query projection model classes

The core scan model now uses new, lightweight classes—ProjectionColumn, ProjectionDimension, ProjectionMeasure, QueryProjection, QueryModel, and QueryModelBuilder—to represent query metadata. These classes hold only the necessary column names, ordinals, and references to schema objects (CarbonDimension, CarbonMeasure) to avoid serializing heavy objects when sending query plans from the driver to executors. This change refactors how projection and filter information is structured internally, supporting optimizations like complex column merging and page-by-page reading, without altering the external SQL interface.

core/src/main/java/org/apache/carbondata/core/scan/model · high confidence

Introduces unsafe column pages and decoder-based local dictionary fallback

The data store page layer now supports an unsafe, direct-memory-backed column page implementation (e.g., UnsafeFixLengthColumnPage, LVByteBufferColumnPage) for primitive and variable-length types, selected via the enable\_unsafe\_column\_page property to improve performance. Additionally, a decoder-based fallback mechanism for local dictionary encoding has been added (controlled by local\_dictionary\_decoder\_based\_fallback), allowing pages that could not be fully dictionary-encoded to fall back to actual data encoding, which reduces memory footprint and prevents data corruption issues during fallback scenarios.

core/src/main/java/org/apache/carbondata/core/datastore/page · high confidence

Local dictionary size limit and configurable threshold

The local dictionary mechanism now enforces a size-based fallback, limiting the dictionary to a configurable maximum size (defaulting to 16MB via \carbon.local.dictionary.size.threshold.in.mb\) and a count-based threshold. If either limit is exceeded during dictionary construction, a \DictionaryThresholdReachedException\ is thrown, preventing unbounded memory growth and ensuring the system falls back to non-dictionary encoding when the dictionary becomes too large.

core/src/main/java/org/apache/carbondata/core/localdictionary/dictionaryholder · high confidence

New ReadCommittedScope interface for managed and non-transactional table reads

The readcommitter module introduces a new ReadCommittedScope interface with two implementations: TableStatusReadCommittedScope for managed Carbon tables and LatestFilesReadCommittedScope for non-transactional tables. This change refactors how committed index files and segment metadata are retrieved during reads, enabling distinct handling paths for managed versus non-transactional table scenarios while maintaining a unified interface for read commitment scope.

core/src/main/java/org/apache/carbondata/core/readcommitter · high confidence

New column page statistics collection framework

The core statistics collection logic has been refactored to use a new interface-based architecture. A new \ColumnPageStatsCollector\ interface and \SimpleStatsResult\ contract replace the previous implementation, introducing specific collectors for different data types: \PrimitivePageStatsCollector\ for numeric and boolean types, \StringStatsCollector\ which adds configurable byte-limit logic to skip min/max storage for large strings, \KeyPageStatsCollector\ for byte-array keys, and \DummyStatsCollector\ for complex types. \TablePageStatistics\ now aggregates these per-column stats to compute dimension and measure min/max values for the page metadata.

core/src/main/java/org/apache/carbondata/core/datastore/page/statistics · high confidence

New exception classes for deprecated features and streaming metadata

This change introduces four new exception classes in the common module to improve error handling for specific scenarios. DeprecatedFeatureException is added to explicitly signal when deprecated features, such as global dictionary and custom partition, are used, indicating they are no longer supported in CarbonData 2.0. MetadataProcessException handles failures during metadata processing for Carbon commands. NoSuchStreamException is introduced to manage cases where a specified stream is not found, supporting the new StreamSQL capabilities. Finally, TableStatusLockException addresses issues related to acquiring locks for table status metadata.

common/src/main/java/org/apache/carbondata/common/exceptions · high confidence

New exception classes for file, concurrency, and configuration errors

The core module now includes three new exception classes: CarbonFileException for file operation failures, ConcurrentOperationException for table-level concurrency conflicts, and InvalidConfigurationException for invalid settings. These exceptions provide more specific error handling for file system interactions, concurrent command execution, and configuration validation.

core/src/main/java/org/apache/carbondata/core/exception · high confidence

New metadata model and table identification classes

The core metadata package now includes new classes to manage table identity and metadata caching. AbsoluteTableIdentifier and CarbonTableIdentifier provide a unified way to identify tables using both the physical store path and the logical database/table name, ensuring consistent identification across the system. CarbonMetadata introduces an expiring map to cache CarbonTable objects, improving performance by reducing redundant metadata loads. Additionally, ColumnIdentifier standardizes column identification with properties and data types, while ColumnarFormatVersion explicitly defines supported format versions (V1, V2, V3). DatabaseLocationProvider allows for configurable database name resolution, and SegmentFileStore handles the reading and writing of segment metadata files, including support for partitioned tables and merge index flows. These changes collectively enhance the robustness and clarity of metadata handling in CarbonData.

core/src/main/java/org/apache/carbondata/core/metadata · high confidence

New serializable comparator framework for sort operations

The sort subsystem now uses a new set of serializable comparator implementations (Boolean, Byte, Decimal, Double, Float, Int, Long, Short, and String) managed by a central Comparator factory. This change ensures that comparator instances are reused and can be serialized, which optimizes sort performance and supports distributed execution contexts.

core/src/main/java/org/apache/carbondata/core/util/comparator · high confidence

New visitor-based filter resolution for dimension and measure columns

The filter resolution layer in the core scan module now uses a visitor pattern to handle different column types and encoding strategies. A new \ResolvedFilterInfoVisitorIntf\ interface and a \FilterInfoTypeVisitorFactory\ route filter resolution to specific visitors: \CustomTypeDictionaryVisitor\ and \RangeDirectDictionaryVisitor\ for direct dictionary columns (including date types), \NoDictionaryTypeVisitor\ and \RangeNoDictionaryTypeVisitor\ for high-cardinality columns without dictionaries, \ImplicitColumnVisitor\ for implicit encoding, and \MeasureColumnVisitor\ for measure columns. This change ensures that filter values are resolved directly based on the column's encoding and data type, improving compatibility and performance for various filter scenarios.

core/src/main/java/org/apache/carbondata/core/scan/filter/resolver/resolverinfo/visitor · high confidence

Prevents data loading hangs on queue poll timeouts

The new CarbonOutputIteratorWrapper class introduces a timeout mechanism when polling for data batches from the internal queue. Previously, if the consumer thread was unresponsive or dead, the loading task could block indefinitely waiting for data. This change ensures that if a batch is not available within 5 minutes, the iterator logs a warning and retries, preventing the loading process from hanging forever and allowing for better recovery or failure detection.

processing/src/main/java/org/apache/carbondata/processing/loading/iterator · high confidence

Refactor query execution metadata into dedicated info classes

The query execution engine now uses dedicated metadata classes—BlockExecutionInfo, MeasureInfo, DimensionInfo, and DeleteDeltaInfo—to manage block-level scan context. This change decouples execution properties (such as projection lists, filter trees, and block indices) from the core scan logic, while explicitly tracking dimension and measure existence, default values, and data types to ensure correct aggregation and filtering when querying blocks with differing or restructured schemas.

core/src/main/java/org/apache/carbondata/core/scan/executor/infos · high confidence

Refactored compaction engine with new merger and executor classes

The compaction logic in the processing module has been restructured into a new set of classes to improve modularity and performance. This includes the introduction of \AbstractResultProcessor\ as a base for result handling, \CarbonCompactionExecutor\ to manage query execution on segments to be merged, and \CarbonDataMergerUtil\ to coordinate the merging workflow. New processors like \CompactionResultSortProcessor\ and \RowResultMergerProcessor\ handle the conversion and merging of data into the final format, while \CompactionType\ defines the specific compaction modes (Minor, Major, IUD, etc.). This refactoring aims to streamline the compaction flow, making it more robust for various scenarios including restructure and streaming compaction.

processing/src/main/java/org/apache/carbondata/processing/merger · high confidence

Refactored core filter execution and resolution infrastructure

The filter processing logic in the core module has been restructured to improve performance and maintainability. A new \FilterExpressionProcessor\ now handles the resolution of filter expression trees into resolver trees, while \FilterExecutorUtil\ provides optimized, type-specific execution for measure column filters using primitive collections to reduce garbage collection overhead. New data structures like \ColumnFilterInfo\ and \MeasureColumnResolvedFilterInfo\ centralize filter metadata, and a \RangeFilterOptimizer\ has been added to enhance range query performance.

core/src/main/java/org/apache/carbondata/core/scan/filter · high confidence

Refactored data loading converter architecture with new interfaces and binary decoding support

The data loading conversion logic in the processing module has been restructured to use a new \FieldConverter\ and \RowConverter\ interface hierarchy, replacing the previous implementation. This change introduces a \BadRecordLogHolder\ to centralize bad record error reporting and adds support for configurable binary data decoding (Base64 and Hex) via new \BinaryDecoder\ implementations. Additionally, a \SpatialIndexFieldConverterImpl\ is now included to handle spatial index column conversions during data load.

processing/src/main/java/org/apache/carbondata/processing/loading/converter · high confidence

Refactored data loading model and option handling

The data loading model has been restructured to improve serialization and configuration management. A new CarbonDataLoadSchema wrapper was introduced to manage CarbonTable information during load operations, while the CarbonLoadModel class was expanded to support additional loading features such as configurable line separators, complex delimiters, and date/timestamp formats. The CarbonLoadModelBuilder and LoadOption utilities were updated to populate these new options with defaults and validate them, ensuring that load configurations are correctly applied from user input and table properties.

processing/src/main/java/org/apache/carbondata/processing/loading/model · high confidence

Refactored data loading pipeline with new processing steps and bad record handling

The data loading mechanism in the processing module has been restructured to use a new step-based pipeline architecture. A new \AbstractDataLoadProcessorStep\ base class and \DataLoadExecutor\ now orchestrate the load process, while \DataLoadProcessBuilder\ constructs specific processing chains (input, conversion, sorting, and writing) based on load configurations such as bucketing, JSON input, or sort scope. The refactoring introduces dedicated classes for handling bad records (\BadRecordsLogger\, \BadRecordsLoggerProvider\) and complex data types (\ArrayObject\, \StructObject\, \ComplexDelimitersEnum\). Additionally, support for JSON data loading has been added via new \JsonInputFormat\ and \JsonStreamReader\ components, and local temporary folder cleanup is now handled asynchronously by \TableProcessingOperations\.

processing/src/main/java/org/apache/carbondata/processing/loading · high confidence

Refactored data loading pipeline with new processor steps and direct dictionary support

The data loading pipeline in the processing module has been refactored to introduce a new set of processor steps, including InputProcessorStepImpl, DataConverterProcessorStepImpl, SortProcessorStepImpl, DataWriterProcessorStepImpl, and CarbonRowDataWriterProcessorStepImpl, which handle input parsing, row conversion, sorting, and data writing in a modularized flow. A new DirectDictionary class has been added to support generating dictionary keys directly from values (e.g., for timestamps), and a JsonInputProcessorStepImpl has been introduced to enable direct JSON data loading without AVRO conversion. These changes modify how data is ingested, converted, and written to CarbonData files, impacting the underlying data load behavior.

processing/src/main/java/org/apache/carbondata/processing/loading/steps · high confidence

Refactored data loading sort infrastructure with new sorter abstraction and unsafe sort support

The data loading sort process in the processing module has been restructured to improve performance and maintainability. A new \Sorter\ interface and \SorterFactory\ have been introduced to abstract the sorting logic, allowing the system to dynamically select between standard and 'unsafe' (off-heap) sorting implementations based on configuration. This change includes the addition of \AbstractMergeSorter\ for common sorter behavior, \CarbonPriorityQueue\ for optimized heap operations during row retrieval, and \SortStepRowHandler\ to manage the conversion and serialization of rows during the sort step. These components collectively replace previous ad-hoc sorting mechanisms, enabling more efficient handling of complex data types and large strings during data loading.

processing/src/main/java/org/apache/carbondata/processing/loading/sort · high confidence

Refactored dimension chunk storage with new wrapper and factory

The dimension chunk storage layer has been refactored to introduce a new \ColumnPageWrapper\ that handles data encoding (including adaptive primitive encoding for floats, doubles, and integers) and local dictionary lookups, alongside a \DimensionChunkStoreFactory\ that selects between safe and unsafe store implementations based on configuration. This change modifies how dimension data is stored in memory and retrieved for query execution, supporting features like local dictionary queries and optimized vector filling.

core/src/main/java/org/apache/carbondata/core/datastore/chunk/store · high confidence

Refactored file system abstraction with pluggable CarbonFile implementations

The core file system handling has been restructured around a new \CarbonFile\ interface and an \AbstractDFSCarbonFile\ base class, replacing the previous monolithic or ad-hoc file operations. This change introduces specific, pluggable implementations for different storage backends: \LocalCarbonFile\ for local file systems, \HDFSCarbonFile\ for HDFS (including support for \FilterFileSystem\ wrappers), \S3CarbonFile\ for Amazon S3, \AlluxioCarbonFile\ for Alluxio, and \ViewFSCarbonFile\ for Hadoop ViewFS. Each implementation correctly instantiates the appropriate subclass when listing files and handles backend-specific behaviors, such as S3's lack of native rename-overwrite or Alluxio's path formatting requirements. This refactoring enables CarbonData to seamlessly operate across diverse storage systems by delegating file operations to the correct backend-specific logic.

core/src/main/java/org/apache/carbondata/core/datastore/filesystem · high confidence

Refactored index query execution with distributed pruning and new core abstractions

The index query execution path has been refactored to support distributed index pruning and a cleaner index abstraction. A new \IndexInputFormat\ now handles the distribution of index pruning tasks across the cluster, while \IndexChooser\ manages the selection of Coarse Grain (CG) and Fine Grain (FG) indexes based on filter expressions. The \IndexFilter\ class now handles filter resolution and serialization, and the \IndexRegistry\ allows index implementations to be registered by short names. These changes underpin the ability to leverage secondary indexes at the segment level and improve query performance through distributed processing.

core/src/main/java/org/apache/carbondata/core/index · high confidence

Refactored scan result collectors to support schema restructuring and vectorized execution

The scan result collector implementation in the core module has been refactored to introduce an abstract base class and specialized collectors for dictionary-based, raw, and vectorized data retrieval. This change adds support for handling restructured tables by introducing dedicated collectors (e.g., RestructureBasedDictionaryResultCollector, RestructureBasedVectorResultCollector) that correctly fill default values for newly added or missing dimension and measure columns during query execution. Additionally, the vectorized collectors now support direct page-wise vector fill, improving performance for batch processing, while the raw collectors have been updated to properly handle dictionary, no-dictionary, and complex type keys for newly added columns in restructured schemas.

core/src/main/java/org/apache/carbondata/core/scan/collector/impl · high confidence

Refactored status management with new core classes and stage file support

The status manager module has been refactored to introduce several new core classes: FileFormat defines supported data formats (COLUMNAR\_V3, ROW\_V1); LoadMetadataDetails now stores timestamps as long values (milliseconds) instead of timezone-dependent strings to ensure consistency across environments; SegmentStatus is now an enum with backward-compatible serialization; SegmentRefreshInfo tracks segment update metadata; and StageInput/StageInputCollector enable querying of staged data files by creating input splits from committed stage files. SegmentStatusManager and SegmentUpdateStatusManager have been restructured to use these new types, improving timestamp handling, status tracking, and stage file integration.

core/src/main/java/org/apache/carbondata/core/statusmanager · high confidence

Register CarbonData and test query executor services for Spark

The integration module now explicitly registers the CarbonData data source and the test query executor via Java SPI service files. Users can now rely on the standard Spark service discovery mechanism to locate \CarbonSource\ and \SparkTestQueryExecutor\, simplifying format registration and ensuring test infrastructure is correctly wired without manual configuration.

integration/spark/src/resources · high confidence

Fixes

Refactored sort pipeline to support legacy schema ordering and partitioned inserts

The sort processing module has been refactored to introduce a \SortTempRowUpdater\ interface with \SchemaBasedRowUpdater\ and \DummyRowUpdater\ implementations, enabling the system to correctly reorder data columns to match the current table schema during final merge. This change specifically fixes data loading failures when inserting into partitioned tables with local sort and resolves issues where sort columns added in older versions (e.g., 1.1) caused load failures in the latest version. Additionally, the \FileMergeSortComparator\ and \IntermediateSortTempRowComparator\ have been updated to use schema-order mappings, ensuring that intermediate sort files are merged correctly even when the column order in the sort temp data differs from the current schema definition.

processing/src/main/java/org/apache/carbondata/processing/sort · high confidence

Test coverage

Added C++ SDK test harness and build configuration; Added CI test suite for Spark examples; Added Hive integration test coverage for expression pushdown, complex types, and data integrity; Added PrestoSQL integration test infrastructure; Added SDV cluster tests for CarbonData table operations; Added SDV test suite registration for CarbonData cluster tests; Added Spark SQL common test utilities and base classes; Added comprehensive test suite for CarbonData bucketing operations; Added embedded Hive server utility for integration tests; Added integration tests for Bloom coarse-grain index functionality; Added integration tests for CarbonData streaming, bad record handling, and table status backup; Added integration tests for Flink streaming writes and stage management; Added integration tests for Materialized Views with TimeSeries data; Added integration tests for Presto-Carbon data types and operations; Added integration tests for Spark Carbon Data Source and File Format; Added integration tests for Spark SQL commands and data types; Added integration tests for complex data types and adaptive encoding; Added integration tests for integer, big integer, and decimal data types; Added integration tests for primitive data types and adaptive encoding; Added sample data files and logging configuration for Hive integration tests; Added test configuration for CarbonData audit logging; Added test configuration properties file; Added test coverage for Carbon Data Spark data loading scenarios; Added test coverage for CarbonData table alteration operations; Added test coverage for SET command, numeric bad records, DELETE subqueries, and query profiler; Added test coverage for materialized view plan rewriting and query matching; Added test coverage for vector reader with schema evolution operations; Added test data files for CSV processing tests; Added test data for OLAP integration tests; Added test resources for Presto integration tests; Added test suites for Lucene coarse-grain and fine-grain indexes; Added test suites for query execution features; Added test utilities and unit tests for stream processing; Added test utility for creating CarbonData stores in Presto integration; Added tests for CSVInputFormat and RawRowComparator; Added tests for Carbon SQL extension integration and data size metrics; Added tests for Carbon command execution and Spark utility version checks; Added tests for Carbon table registration and DDL operations; Added tests for Carbon table registration and maintenance operations; Added tests for CarbonData SQL commands and cache management; Added tests for CarbonData table schema validation and truncate operations; Added tests for CarbonLoaderUtil block assignment strategies; Added tests for CarbonSegmentUtil APIs; Added tests for IndexServer distributed RDD utilities and temp folder management; Added tests for ObjectSerializationUtil and sample test data; Added tests for Spark Carbon Data Source integration; Added tests for Spark command syntax and table deletion; Added tests for Strings utility methods; Added tests for binary data handling and table status recovery; Added tests for empty row and CSV space character handling; Added tests for local file and ZooKeeper locking mechanisms; Added tests for materialized view creation and configuration; Added tests for secondary index data file merge operations; Added tests for secondary index file merge behavior; Added tests for spatial index validation and geo query UDFs; Added tests for vector reader execution plans; Added unit tests for CarbonData logging components; Added unit tests for TimeStampDirectDictionaryGenerator; Added unit tests for core configuration, caching, and file system components; Added unit tests for sort comparators; Expanded test coverage for secondary index operations; New StoreCreator test utility for Hadoop module.

Dependencies

Initial dependency manifests for CarbonData modules

The project now includes explicit dependency and build configuration files (pom.xml and pyproject.toml) for its core modules, including the assembly packaging, common utilities, core storage, format definitions, geospatial indexing, Hadoop integration, and various index types (Bloom, Lucene, secondary). This establishes the build structure and dependency graph for the CarbonData platform.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 53 → 54 (+0.5)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 55 → 52 (-2.2)
  • Architecture 95 → 95 (-0.0)
  • Maturity 48 → 48 (-0.0)
  • Readiness 50 → 53 (+2.4)
  • Security 57 → 59 (+2.6)
  • Performance 100 → 100 (+0.0)

Resolved (51)

  • Blocklet.computePercentage (cognitive 16) (tools/cli/src/main/java/org/apache/carbondata/tool/DataFile.java)
  • Critical CVE: [GHSA redacted] (common/pom.xml)
  • Critical CVE: [GHSA redacted] (common/pom.xml)
  • Critical CVE: [GHSA redacted] (common/pom.xml)
  • Critical CVE: [GHSA redacted] (common/pom.xml)
  • Critical CVE: [GHSA redacted] (common/pom.xml)
  • Critical CVE: [GHSA redacted] (common/pom.xml)
  • DataSummary.printColumnStats (cognitive 32) (tools/cli/src/main/java/org/apache/carbondata/tool/DataSummary.java)
  • DataSummary.run (cognitive 22) (tools/cli/src/main/java/org/apache/carbondata/tool/DataSummary.java)
  • DataSummary.run (cyclomatic 20) (tools/cli/src/main/java/org/apache/carbondata/tool/DataSummary.java)
  • Duplicated block (10 lines × 2) (core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/adaptive/AdaptiveIntegralCodec.java)
  • Duplicated block (10 lines × 2) (index/bloom/src/main/java/org/apache/carbondata/index/bloom/BloomIndexFileStore.java)
  • Duplicated block (11 lines × 2) (core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGreaterThanEqualFilterExecutorImpl.java)
  • Duplicated block (11–30 lines × 2) (examples/spark/src/main/scala/org/apache/carbondata/examples/AlterTableExample.scala)
  • Duplicated block (12 lines × 2) (integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala)
  • Duplicated block (12 lines × 2) (processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java)
  • Duplicated block (17–19 lines × 2) (integration/spark/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala)
  • Duplicated block (21 lines × 2) (core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/adaptive/AdaptiveDeltaIntegralCodec.java)
  • Duplicated block (7 lines × 2) (core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/adaptive/AdaptiveIntegralCodec.java)
  • Duplicated block (7 lines × 2) (mv/plan/src/main/scala/org/apache/carbondata/mv/plans/util/SQLBuildDSL.scala)
  • …and 31 more

New (345)

  • Critical CVE: [GHSA redacted] (integration/flink-build/pom.xml)
  • Duplicated block (7 lines × 2) (core/src/main/java/org/apache/carbondata/core/scan/expression/logical/BinaryLogicalExpression.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/scan/filter/resolver/resolverinfo/DimColumnResolvedFilterInfo.java)
  • Edited copies of one member (3 members, 12+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/indexstore/schema/SchemaGenerator.java)
  • Edited copies of one member (3 members, 12+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/metadata/schema/SchemaReader.java)
  • Edited copies of one member (3 members, 12+ corresponding lines) (integration/spark/src/main/scala/org/apache/spark/sql/profiler/ProfilerLogger.scala)
  • Edited copies of one member (3 members, 12+ corresponding lines) (processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepImpl.java)
  • Edited copies of one member (3 members, 13+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryByteArrayWrapper.java)
  • Edited copies of one member (3 members, 14+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java)
  • Edited copies of one member (3 members, 14+ corresponding lines) (integration/spark/src/main/scala/org/apache/spark/sql/execution/CastExpressionOptimization.scala)
  • Edited copies of one member (3 members, 15+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/DimensionRawColumnChunk.java)
  • Edited copies of one member (3 members, 15+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java)
  • Edited copies of one member (3 members, 16+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java)
  • Edited copies of one member (3 members, 16+ corresponding lines) (processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java)
  • Edited copies of one member (3 members, 20+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java)
  • Edited copies of one member (3 members, 23+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java)
  • Edited copies of one member (3 members, 24+ corresponding lines) (core/src/main/java/org/apache/carbondata/core/reader/CarbonDeleteDeltaFileReaderImpl.java)
  • Edited copies of one member (3 members, 24+ corresponding lines) (integration/spark/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessorStepOnSpark.scala)
  • Edited copies of one member (3 members, 24+ corresponding lines) (processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeInMemoryIntermediateDataMerger.java)
  • Edited copies of one member (3 members, 38+ corresponding lines) (processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/FileMergeSortComparator.java)
  • …and 325 more

Architecture

  • Unchanged — 0 containers · 4 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

apache/carbondata was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit b653d1336bb25b973d7b7a650d1c7c4bb3538480 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.