Skip to content
CAI
Software that uses CAICheck a score

pola-rs/polars

66.3

Adequate · 14 November 2026

625.7k

lines of production code

Rust

with Python

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add Avro file reading support

Introduces a new single-threaded, blocking Avro reader in \crates/polars-arrow/src/io/avro/read\ that converts Avro files into Arrow \RecordBatch\ objects. The implementation includes schema inference from Avro records to Arrow types (handling primitives, lists, structs, unions, enums, and logical types like timestamps and decimals), a streaming block iterator for efficient memory usage, and a deserializer that maps Avro data to mutable Arrow arrays, including support for dictionary-encoded enums and nested structures.

crates/polars-arrow/src/io/avro/read · high confidence

Add Avro read and write support

Users can now read and write Apache Avro files using the new \AvroReader\ and \AvroWriter\ in \polars-io\. The reader supports column projection, row limiting, and rechunking, while the writer allows setting a schema name and choosing a compression level.

crates/polars-io/src/avro · high confidence

Add Avro write support in polars-arrow

The \crates/polars-arrow/src/io/avro/write\ module has been added, introducing the core logic for serializing Arrow data into the Avro format. This includes schema conversion from Arrow types to Avro records (handling primitives, lists, structs, and logical types like dates and timestamps) and a streaming serialization engine that transposes columnar data into row-based Avro blocks. This change provides the underlying write capability for Avro files within the Polars Arrow crate.

crates/polars-arrow/src/io/avro/write · high confidence

Add BooleanArray and MutableBooleanArray implementations to polars-arrow

This change introduces the core BooleanArray and MutableBooleanArray types to the polars-arrow crate, providing the underlying Arrow representation for boolean data. It includes the immutable BooleanArray with support for validity masks, iteration, and slicing, as well as the MutableBooleanArray for efficient construction. The implementation also provides a BooleanArrayBuilder for building arrays, FFI serialization/deserialization support, formatting/debug output, and property-based tests (proptest) to ensure correctness.

crates/polars-arrow/src/array/boolean · high confidence

Add FixedSizeBinaryArray and MutableFixedSizeBinaryArray implementations

The \polars-arrow\ crate now supports the Arrow \FixedSizeBinary\ data type. This change introduces the \FixedSizeBinaryArray\ (immutable) and \MutableFixedSizeBinaryArray\ (mutable) types, along with their builders, iterators, FFI serialization/deserialization, and debug formatting. Users can now read, write, and manipulate columns containing fixed-size binary data (such as fixed-length byte arrays) within Polars.

_crates/polars-arrow/src/array/fixed\_size\binary · high confidence

Add LazyFrame visualization methods

Users can now generate DOT language representations of their lazy query plans via the new \to\_dot\ method on \LazyFrame\, supporting both optimized and unoptimized logical plans. Additionally, when the \streaming\ feature is enabled, the \to\_dot\_streaming\_phys\ method allows users to visualize the physical plan generated by the streaming engine, aiding in debugging and understanding query execution.

crates/polars-lazy/src · high confidence

Add PyO3-Polars derive expression example library

This change introduces a new example library (\expression\_lib\) that demonstrates how to create custom Polars expressions using the \pyo3-polars\ derive macros. It provides concrete implementations for string manipulation (Pig Latin), distance metrics (Hamming, Jaccard, Haversine), and utility functions (leap year check, kwargs handling), showcasing parallel execution capabilities and custom argument parsing within the Polars expression engine.

_pyo3-polars/example/derive\_expression/expression\lib · high confidence

Add UnionArray implementation with FFI, iteration, and formatting support

This change introduces the \UnionArray\ type in \crates/polars-arrow/src/array/union\, enabling the handling of Arrow Union data structures. It includes FFI bindings (\ffi.rs\) for importing and exporting Union arrays, an iterator (\iterator.rs\) for efficient traversal of union values, and debug formatting (\fmt.rs\). The core implementation (\mod.rs\) defines the \UnionArray\ struct, supporting both sparse and dense union modes, and provides constructors for creating new, null, or existing union arrays.

crates/polars-arrow/src/array/union · high confidence

Add estimated\_bytes\_size for array memory estimation

The \polars-arrow\ aggregate module now exposes an \estimated\_bytes\_size\ function that calculates the total heap-allocated size of an array, including buffers, validity bits, and nested structures. This implementation correctly handles various data types, including new \BinaryView\ and \Utf8View\ types, and accounts for shared buffers to avoid overestimating memory usage, which helps prevent unnecessary out-of-core operations.

crates/polars-arrow/src/compute/aggregate · high confidence

Add example PyO3-Polars IO plugin with random data sampling

An example IO plugin for PyO3-Polars has been added to the repository, demonstrating how to implement custom data sources in Rust. The plugin exposes a \RandomSource\ class that generates random data using Bernoulli and Uniform samplers, supporting projection and predicate pushdown optimizations to limit data sampling. This serves as a reference implementation for users building custom IO plugins with PyO3-Polars.

_pyo3-polars/example/io\_plugin/io\plugin · high confidence

Add example Rust extension for parallel Jaccard similarity computation

This change introduces a new example Rust extension module (\extend\_polars\) that demonstrates how to extend Polars with custom Python-callable functions. It exposes \parallel\_jaccard\ and \lazy\_parallel\_jaccard\ functions to Python, which compute Jaccard similarity between two list columns by splitting the DataFrame into chunks and processing them in parallel using Rayon. The module also includes helper functions \debug\ and \empty\_df\ for testing and demonstration purposes.

_pyo3-polars/example/extend\_polars\_python\_dispatch/extend\polars · high confidence

Add exponential weighted mean, sum, variance, and standard deviation computations

The \polars-compute\ crate now includes the core algorithms for exponential weighted moving (EWM) statistics. Users can now compute \ewm\_mean\, \ewm\_sum\, \ewm\_var\, and \ewm\_std\ using the \EWMOptions\ configuration (alpha, adjust, bias, min\_periods, ignore\_nulls). This implementation provides the underlying state-update logic for these operations, supporting both biased and unbiased variance calculations and handling null values according to the specified options.

crates/polars-compute/src/ewm · high confidence

Add memory-mapped IPC file reading for out-of-core processing

Users can now read Apache Arrow IPC files directly from disk using memory mapping, enabling out-of-core processing of datasets that exceed available RAM. This new capability in the \polars-arrow\ crate allows record batches and dictionaries to be mapped into memory without full file loading, supporting variable-length types like strings and binary data via the new \binview\ implementation.

crates/polars-arrow/src/mmap · high confidence

Add null-handling implementations for rolling aggregations

New source files in \crates/polars-compute/src/rolling/nulls\ implement rolling aggregations (sum, mean, min, max, quantile, rank, and moments) that correctly handle null values in the input data. This change ensures that rolling operations no longer panic or produce incorrect results when encountering nulls, instead respecting \min\_periods\ and validity masks to return appropriate null results when the window does not contain enough valid data.

crates/polars-compute/src/rolling/nulls · high confidence

Add polars-utils and polars-compute internal crates

The repository now includes the \polars-utils\ and \polars-compute\ internal sub-crates. These crates provide private utility functions and compute kernels for the main Polars library and are not intended for external usage.

crates/polars-utils · high confidence

Add sample food datasets to examples

The examples directory now includes a new \datasets\ folder containing five sample CSV files (\foods1.csv\ through \foods5.csv\) and one dataset with null values (\null\_nutriscore.csv\), along with a \.gitignore\ file to exclude generated data artifacts. These files provide ready-to-use data for users testing or demonstrating functionality.

examples · high confidence

Added DSL schema validation tool

A new CLI tool (\dsl-schema\) is available to generate, update, and check hashes of the DSL schema. This allows developers to detect unintended changes to the schema structure by comparing current schema hashes against stored expected values, helping to prevent merge conflicts and ensure consistency across the codebase.

crates/polars-plan/src/bin · high confidence

Added IEJoin algorithm for non-equi joins

Users can now perform inequality joins (e.g., \\>\, \\<\, \\>=\, \\<=\) using the IEJoin algorithm, which is implemented in the \crates/polars-ops/src/frame/join/iejoin\ module. This new capability includes a \FilteredBitArray\ for efficient sparse bit manipulation and L1/L2 array structures to optimize the search for matching rows based on inequality predicates, supporting both strict and non-strict inequality operators.

crates/polars-ops/src/frame/join/iejoin · high confidence

Added legacy compatibility layer for Arrow arrays and utilities

The \crates/polars-arrow/src/legacy\ module has been introduced to provide backward-compatible utilities and type aliases for the Arrow implementation. This includes bit manipulation helpers (\bit\_util\), index conversion traits (\index\), validity checking (\is\_valid\), and conversion functions (\conversion\). It also defines legacy type aliases such as \LargeStringArray\ and \LargeBinaryArray\ in the prelude, and re-exports error handling from \polars\_error\. This layer supports existing code paths that rely on these specific Arrow abstractions without requiring immediate refactoring to the newer \RecordBatch\ or \ArrowDataType\ naming conventions.

crates/polars-arrow/src/legacy · high confidence

Added primitive type matching macros for 128-bit and Float16 support

The \crates/polars-arrow/src/util\ module now includes new utility macros (\with\_match\_primitive\_type\, \with\_match\_primitive\_type\_full\, and \match\_integer\_type\) that enable runtime dispatch for a comprehensive set of primitive types. These macros explicitly handle \Int128\, \UInt128\, and \Float16\ (mapped to \pf16\ from \polars\_utils\), alongside standard integer and float types, providing the foundational infrastructure required for the new 128-bit integer and Float16 data type support throughout the Arrow implementation.

crates/polars-arrow/src/util · high confidence

Bitmap module restructured with new aligned slicing and bitwise operations

The \polars-arrow\ bitmap module has been reorganized into distinct source files (immutable, mutable, builder, iterator, bitmask, aligned, ops) to improve code organization. This change introduces \AlignedBitmapSlice\ for efficient processing of bitmaps split into prefix, bulk, and suffix sections, and adds comprehensive bitwise assignment operators (\&=\, \\|=\) and unary/binary operations for \MutableBitmap\ and \Bitmap\. It also includes optimized chunk iterators and \nth\_set\_bit\ utilities with BMI2 support.

crates/polars-arrow/src/bitmap · high confidence

Expanded array namespace with statistical, aggregation, and transformation operations

The \arr\ namespace in \polars-ops\ now supports a comprehensive set of new operations for fixed-size arrays. Users can compute measures of dispersion (mean, median, standard deviation, and variance) across array elements, including support for Float16 and Duration types. Aggregation capabilities have been extended with \arr.sum\ (handling nulls and various integer/float types), \arr.min\/\arr.max\, and a new \arr.dot\ for native fixed-array dot products. Additional features include \arr.count\_matches\ to count occurrences of a value, \arr.join\ to concatenate string arrays with an optional \ignore\_nulls\ parameter, and \arr.to\_struct\ to convert arrays into struct columns. Indexing is also enhanced with \arr.get\ supporting \null\_on\_oob\ and overflow handling, as well as \arr.shift\, \arr.slice\, \arr.head\, and \arr.tail\.

_crates/polars-ops/src/chunked\array/array · high confidence

Expanded if-then-else kernel support for new array types

The if-then-else computation kernel in polars-compute now supports FixedSizeListArray, ListArray, and BooleanArray in addition to the previously supported primitive and view types. This allows users to perform conditional selection operations on nested list structures and boolean columns directly, with optimized implementations for broadcasting scalar values into these new array types.

_crates/polars-compute/src/if\_then\else · high confidence

Experimental Unity Catalog client added

Added an experimental Unity Catalog client in \crates/polars-io/src/catalog\ that enables listing and managing catalogs, namespaces (schemas), and tables, as well as retrieving table metadata and temporary credentials. The client handles paginated API responses, maps Unity data types to Polars schemas, and uses the \USER\_AGENT\ header for requests.

crates/polars-io/src/catalog · high confidence

IPC read module restructured with append support and flight streaming

The IPC read implementation in \crates/polars-arrow/src/io/ipc/read\ has been reorganized into a modular structure with new files for common logic, deserialization, error handling, file/stream/flight reading, and schema parsing. A key new capability is the ability to append to existing IPC files via \FileWriter::try\_from\_file\, which reads the existing file's dictionaries and seeks to the end to write new batches. Additionally, async Flight stream support has been added via \into\_flight\_stream\ and \FlightStreamProducer\, enabling reading IPC data over async readers for Arrow Flight integration. The module also introduces a comprehensive \OutOfSpecKind\ error enum for detailed IPC format validation errors.

crates/polars-arrow/src/io/ipc/read · high confidence

IPC writer supports custom schema metadata and extension types

The IPC writers (FileWriter and StreamWriter) now support writing custom schema-level metadata via a new \set\_custom\_schema\_metadata\ method, allowing users to attach arbitrary key-value pairs to the IPC schema message. Additionally, the serialization logic has been updated to correctly handle Arrow Extension types, ensuring that extension names and metadata are properly encoded in the field definitions during IPC writes.

crates/polars-arrow/src/io/ipc/write · high confidence

Initial implementation of Arrow FFI C Data and Stream interfaces

The \polars-arrow\ crate now exposes the Arrow C Data Interface (FFI) for importing and exporting arrays and schemas, enabling zero-copy data exchange with other Arrow-compatible systems. This change adds the \ffi\ module containing \export\_array\_to\_c\ and \import\_array\_from\_c\ for single arrays, as well as \ArrowArrayStreamReader\ and \export\_iterator\ for streaming data via the C Stream Interface. The implementation supports all standard Arrow physical types, including the new \BinaryView\ and \Utf8View\ types, and handles nested structures like Lists, Structs, and Dictionaries correctly.

crates/polars-arrow/src/ffi · high confidence

Initial release of the polars-arrow crate

The \polars-arrow\ crate is introduced as a new module providing the foundational Arrow data structures and utilities for Polars. It includes core array implementations, offset management, record batch handling, temporal conversions, and iterator traits, establishing the low-level data representation layer for the library.

crates/polars-arrow/src · high confidence

Introduce BinaryView and Utf8View array types in polars-arrow

This change introduces the new \BinaryView\ and \Utf8View\ array types in the \polars-arrow\ crate, replacing the traditional offset-based binary/string arrays with a view-based representation that inlines small values and references larger ones directly. The implementation includes the core \BinaryViewArrayGeneric\ structure, builders (\BinaryViewArrayBuilder\, \MutableBinaryViewArray\), iterators, formatting, and FFI support, enabling more efficient memory usage and faster access for string and binary data in Polars.

crates/polars-arrow/src/array/binview · high confidence

Introduce DataTypeExpr and DataTypeFunction for schema-level expression operations

Users can now perform operations on data types themselves within the expression DSL, rather than just on data values. This change introduces the \DataTypeExpr\ type and a new \DataTypeFunction\ enum, enabling expressions that inspect or manipulate type information. Specific capabilities added in this module include checking if a type matches a selector (\Matches\), retrieving a string representation of a type (\Display\), comparing two type expressions for equality (\Eq\), and generating default values based on type characteristics (\DefaultValue\). Additionally, namespace-specific type introspection is now available via \ArrayDataTypeFunction\ (providing width and shape of array types) and \StructDataTypeFunction\ (providing field names for struct types). These features allow for more dynamic schema-aware logic in queries.

crates/polars-plan/src/dsl/expr · high confidence

Introduce FixedSizeListArray and its builders in polars-arrow

This change adds the core implementation for FixedSizeListArray to the polars-arrow crate, including the immutable array structure, a mutable builder, and a new StaticArrayBuilder (FixedSizeListArrayBuilder) that supports optimized operations like gather, extend\_nulls, and subslice\_extend. It also provides FFI serialization/deserialization, debug formatting, and iteration capabilities, enabling users to construct and manipulate fixed-size list data structures efficiently within the Arrow memory model.

_crates/polars-arrow/src/array/fixed\_size\list · high confidence

Introduce MapArray for Arrow Map data type support

Users can now work with Arrow Map data types via the new MapArray implementation in polars-arrow. This change adds the core array structure along with FFI serialization/deserialization (ffi.rs), debug formatting (fmt.rs), and iteration capabilities (iterator.rs), enabling the storage and processing of key-value pair collections within the Arrow memory model.

crates/polars-arrow/src/array/map · high confidence

Introduce Object Extension Types for Custom Rust Types

Users can now store custom Rust types (implementing \PolarsObject\) in Polars Series using a new extension type mechanism. This feature allows arbitrary Rust structs to be serialized into Arrow-compatible binary arrays while preserving ownership and ensuring proper cleanup via custom drop functions. The implementation includes support for nested lists of objects and requires the \POLARS\_ALLOW\_EXTENSION\ environment variable to be set or the \set\_polars\_allow\_extension\ function to be called to enable this capability, ensuring safety against unintended extension type creation.

_crates/polars-core/src/chunked\array/object/extension · high confidence

Introduce Polars FFI crate for cross-language data exchange

Added a new \polars-ffi\ crate that provides a stable interface for exporting and importing Polars Series and DataFrames across language boundaries. This includes versioned FFI structures (\SeriesExport\) and utilities to convert between Polars internal types and Arrow C Data Interface formats, enabling plugins and external tools to safely share memory with Polars without unsafe transmutation.

crates/polars-ffi · high confidence

Introduce Python DSL module for UDFs and custom IO sources

The \crates/polars-plan/src/dsl/python\_dsl\ module has been added to centralize the representation of Python-specific operations within the Polars query plan. This includes \PythonUdfExpression\ for handling user-defined functions (UDFs) applied to columns and dataframes, supporting serialization, type inference, and schema validation, as well as \PythonOptionsDsl\ and \PythonScanSource\ to define custom Python-based data sources (such as PyArrow, CUDA, or IO plugins) with lazy schema resolution and configurable explain output.

_crates/polars-plan/src/dsl/python\dsl · high confidence

Introduce Scalar API for individual array elements

The \polars-arrow\ crate now exposes a new \Scalar\ API in \crates/polars-arrow/src/scalar\ that provides trait-object representations for individual items within Arrow arrays. This API includes concrete implementations for all major physical types—such as \BooleanScalar\, \PrimitiveScalar\, \Utf8Scalar\, \BinaryScalar\, \ListScalar\, \StructScalar\, and the new \BinaryViewScalar\—allowing users to extract and manipulate single values with nullability and type information. A central \new\_scalar\ function enables the creation of these scalar objects from any array at a given index, and equality checks are implemented to support comparison across different scalar types.

crates/polars-arrow/src/scalar · high confidence

Introduce execution state and window expression caching

The \crates/polars-expr/src/state\ module now defines the \ExecutionState\ struct and associated caching mechanisms (\WindowCache\, \df\_cache\) that track execution context, schema information, and intermediate results during physical plan execution. This change enables caching of window expression group tuples and join tuples, improving performance for window functions, and introduces a mechanism to interrupt queries via a cancel token.

crates/polars-expr/src/state · high confidence

Introduce external reader abstraction for Python-backed file formats

Added a new \external\_reader\ module in \polars-io\ that provides a unified \ExternalReaderBuilder\ enum to support reading from external sources via Python. This change introduces the \PythonFileReaderBuilder\ struct, which wraps a Python object to enable lazy scanning of external file formats (such as Lance) by delegating property explanation and multi-scan context initialization to the Python side, while maintaining a Rust placeholder for future native implementations.

_crates/polars-io/src/external\reader · high confidence

Introduce in-memory engine executor implementations

The in-memory execution engine now includes concrete executor implementations for core DataFrame operations, including filtering, projection, joining, grouping (standard, dynamic, and rolling), sorting, slicing, and horizontal concatenation. This change provides the physical execution logic required to run these operations in memory, replacing previous placeholder or missing implementations.

crates/polars-mem-engine/src/executors · high confidence

Introduce internal polars-python crate and Python bindings

The \polars-python\ crate has been introduced as an internal sub-crate to enable running Polars workloads in Python. This change adds the foundational Python bindings, including the C API allocator, Unity catalog client integration, cloud plan preparation, and GPU engine execution support. It also implements core data conversion utilities for handling Python objects, categorical types, and datetime conversions, effectively establishing the bridge between the Rust Polars engine and the Python environment.

crates/polars-python · high confidence

Introduce new Arrow data types and schema abstraction

The \polars-arrow\ datatypes module now supports several new logical types, including 128-bit integers (\Int128\, \UInt128\), 16-bit floats (\Float16\), \Map\ types, and \Extension\ types for custom data. It also introduces \BinaryView\ and \Utf8View\ for optimized string/binary handling. The schema representation has been refactored to use \polars\_schema::Schema\<Field, Metadata\>\, and the \Field\ struct now explicitly tracks nullability and metadata, with specific support for legacy and new Enum/Categorical encodings.

crates/polars-arrow/src/datatypes · high confidence

Introduce new StructArray builder and FFI handling in polars-arrow

The \polars-arrow\ crate now includes a dedicated \StructArrayBuilder\ that implements the \ArrayBuilder\ interface, providing methods to construct struct arrays with support for extending nulls, gathering, and subslicing. Additionally, the FFI module for \StructArray\ has been updated to correctly handle view buffer sizes and offset propagation for nested extension types, ensuring that sliced struct arrays round-trip correctly through the Arrow FFI interface regardless of whether the data comes from old or new exporters.

crates/polars-arrow/src/array/struct\ · high confidence_

Introduce new streaming JSON serialization implementation

The \crates/polars-json/src/json/write\ module has been replaced with a new implementation that provides streaming JSON serialization capabilities. This includes a \Serializer\ for serializing individual arrays to JSON bytes and a \RecordSerializer\ for writing record-oriented JSON (compatible with pandas) from \RecordBatchT\ structures. The new code leverages the \polars\_arrow\ crate for array handling and utilizes the \zmij\ library for high-performance float serialization, replacing previous methods. It also introduces dedicated serializers for various data types including booleans, integers, floats, floats16, decimals, and dictionary-encoded strings, along with a UTF-8 string escaping utility.

crates/polars-json/src/json/write · high confidence

Introduce polars-arrow compute module with array operation kernels

The \crates/polars-arrow/src/compute\ directory now contains a new module providing core compute kernels for Arrow arrays. This includes unary and binary arithmetic operations (\arity\), bitwise operators (\bitwise\), and array concatenation (\concatenate\). It also adds temporal extraction kernels for dates and times (\temporal\) and utilities for managing validity bitmaps (\utils\).

crates/polars-arrow/src/compute · high confidence

Introduce polars-buffer crate with zero-copy Buffer and SharedStorage

The new \polars-buffer\ crate provides a \Buffer\<T\>\ type that behaves like an \Arc\<Vec\<T\>\>\ but allows slicing and cloning in O(1) time by sharing the underlying memory region. It also introduces \SharedStorage\, a reference-counted container that supports backing data from \Vec\, static slices, or external owners, enabling efficient, zero-copy memory management across thread boundaries.

crates/polars-buffer/src · high confidence

Introduce polars-config crate for runtime configuration

The new \polars-config\ internal crate centralizes runtime configuration for Polars, exposing environment variables and settings that control execution behavior. It defines configuration enums for the execution engine (auto, in-memory, streaming, gpu), file access advice (normal, sequential, random, willneed), metadata resolution modes (none, row\_counts, full, sampled), and spill formats (ipc). The crate also establishes defaults and parsing logic for out-of-core (OOC) memory and disk budgets, HTTP caching, NUMA awareness, and various performance tuning knobs, providing a unified entry point for these runtime options.

crates/polars-config · high confidence

Introduce polars-stream crate and polars-descriptions for new streaming engine

This change introduces the \polars-stream\ crate, an internal sub-crate containing the new async-based streaming execution engine for Polars. It adds a new \polars-descriptions\ crate that defines serializable schema for IR nodes, physical nodes, and per-node metrics, enabling external visualization and observability of the query plan. The \polars-stream\ crate exposes a \build\_streaming\_query\_executor\ that dispatches queries to this new engine, handling sink wrapping and rechunking, and provides a \StreamingQuery\ struct for execution. The engine includes core components like \Morsel\ for data flow, \SourceToken\ for backpressure, and a metrics system tracking poll times, IO bytes, and custom metrics.

crates/polars-stream · high confidence

Introduce pyo3-polars crate for Python-Rust bindings

This change introduces the new \pyo3-polars\ crate, which provides the core Rust-side bindings for the Python Polars library. It defines wrapper types (\PySeries\, \PyDataFrame\, \PyLazyFrame\, \PyExpr\) that implement conversion logic to and from Python objects, enabling seamless data exchange. The crate also includes a custom \PolarsAllocator\ to relay memory allocations to the Python-side Polars allocator, improving performance and memory management consistency between the Python and Rust components.

pyo3-polars/pyo3-polars · high confidence

Introduce pyo3-polars for shared library plugins and PyO3 extensions

The new pyo3-polars crate provides two mechanisms for extending Polars: shared library plugins that circumvent the GIL for better parallelism, and PyO3 extension types for convenient DataFrame/Series wrapping. This release includes example implementations for custom expressions (such as pig latin, distance metrics, and date utilities) and custom IO sources, along with the necessary Rust derive macros and Python registration code to make these plugins accessible from the Polars API.

pyo3-polars · high confidence

Introduce pyo3-polars-derive proc-macro for plugin function generation

Added the \pyo3-polars-derive\ crate, which provides a procedural macro to automatically generate the FFI-bound C functions required for Polars plugins. This macro parses function signatures to determine argument types (such as \kwargs\ or \context\) and emits the necessary boilerplate to import series inputs, parse keyword arguments, call the user-defined Rust function, and export the resulting series back to the Polars engine.

pyo3-polars/pyo3-polars-derive · high confidence

Introduces LazyFrame resolver API for Python

Adds a new \DslResolver\ system in the Rust core that enables Python-based resolvers to handle DSL resolution, schema extraction, and filter expression conversion. This change introduces the \LazyFrame\ resolver API, allowing Python code to define custom resolution logic for lazy frames, including support for lowering arithmetic operations into PyArrow predicates.

_crates/polars-plan/src/dsl/dsl\resolver · high confidence

Introduces chunked-array gather operations via TakeChunked trait

This change adds a new \TakeChunked\ trait and its implementations for \DataFrame\, \Column\, and \Series\ within \polars-ops\, enabling efficient gathering of elements by specifying chunk indices and offsets (\ChunkId\). The implementation handles various data types including primitives, booleans, strings, binaries, lists, arrays, and structs, with specific optimizations for binary/string views and scalar columns. This functionality is gated behind the \chunked\_ids\ feature flag.

_crates/polars-ops/src/chunked\array/gather · high confidence

Introduces generic tree traversal infrastructure for query plan optimization

The \crates/polars-plan/src/traversal\ module now provides a generic, reusable framework for traversing the query plan graph. This includes a \NodeEdgesProvider\ trait for abstracting node connections, a \NodeVisitor\ trait with pre/post-visit hooks for custom traversal logic, and a \tree\_traversal\ implementation that supports edge value propagation and subtree skipping. This infrastructure enables more efficient and flexible optimizations, such as projection pushdown and cache handling, by allowing the query planner to traverse and modify the plan tree with consistent edge management.

crates/polars-plan/src/traversal · high confidence

Introduces legacy kernel implementations for masking, scattering, and time-zone handling

This change adds a new \legacy/kernels\ module to \polars-arrow\ containing foundational array operations. It introduces \MaskedSlicesIterator\ and \BinaryMaskedSliceIterator\ to efficiently identify and process contiguous regions of boolean masks, enabling optimized functions like \set\_with\_mask\ and \set\_at\_nulls\ for primitive arrays. It also adds \scatter\_single\_non\_null\ for indexed value updates, string length calculation utilities (\utf8view\_len\_bytes\, \string\_len\_chars\), and time-zone conversion logic (\convert\_to\_naive\_local\) that supports \Ambiguous\ and \NonExistent\ resolution strategies for datetime localization.

crates/polars-arrow/src/legacy/kernels · high confidence

Introduces new low-level type abstractions for aligned bytes, bit chunks, and indexing

The \polars-arrow\ types module has been expanded with new foundational traits and structs to improve type safety and performance. A new \AlignedBytes\ trait and associated types (e.g., \Bytes8Alignment8\) allow physical types to be represented as raw bytes with correct alignment, reducing monomorphization overhead. A \BitChunk\ trait and \BitChunkIter\ iterator enable efficient, SIMD-friendly iteration over bitmaps. Additionally, an \Index\ trait and \IndexRange\ iterator provide a unified way to handle array indexing types (\i32\, \i64\, etc.), while the \Offset\ trait explicitly distinguishes between \i32\ and \i64\ offsets for variable-length arrays.

crates/polars-arrow/src/types · high confidence

Introduces out-of-core execution with automatic memory management and disk spilling

Polars now supports out-of-core (OOC) execution, allowing DataFrames to be spilled to disk when memory limits are exceeded. This change adds a global memory allocator that tracks usage and can abort the process if it exceeds the POLARS\_OOMKILL\_THRESHOLD\_MB limit. A background MemoryManager monitors memory pressure and automatically triggers spilling or prefetching based on the POLARS\_OOC\_DISK\_BUDGET\_MB and other configuration thresholds. Spilled data is stored as compressed IPC files in a dedicated directory, managed by a background cleaner thread that removes stale files from dead processes. The system uses a multi-armed bandit algorithm to optimize spill and prefetch decisions based on historical performance statistics. Also in this area: Out-of-core memory management and spill tracking improvements; Improved out-of-core memory management and allocator behavior.

_crates/polars-ooc/src, crates/polars-ooc/src/spill\context · high confidence

Introduction of PlanCallback and IR-based plan utilities

The polars-plan crate now includes a new PlanCallback mechanism that allows logical plan operations to execute custom Rust or Python callbacks, enabling extensibility for plan transformations. Additionally, the crate exposes new utility functions for deep-cloning IR subtrees, formatting column lists, and traversing expression trees to identify leaf columns, literals, and window functions, providing a more robust foundation for plan manipulation and optimization.

crates/polars-plan/src · high confidence

Introduction of dedicated runtime packages for Polars

The Polars library now includes dedicated runtime packages (polars-runtime-32, polars-runtime-64, and polars-runtime-compat) that expose the C API. This change allows users to access the underlying C interface directly through these new runtime crates, facilitating integration with other systems or custom builds that require specific runtime configurations.

(repo-wide) · high confidence

Introduction of internal polars-schema crate

The \polars-schema\ crate has been added as an internal sub-crate to provide private schema utility functions. It introduces a generic \Schema\<Field, Metadata\>\ struct that manages field names and types using an index-preserving map, supporting operations such as insertion, renaming, and duplicate detection. This crate is not intended for external usage; users should continue to rely on the main Polars crate for schema-related functionality.

crates/polars-schema · high confidence

Introduction of new Array module design and builder infrastructure

The \crates/polars-arrow/src/array\ module has been restructured to introduce a formal design contract for Arrow arrays, including new traits like \ArrayBuilder\ and \StaticArrayBuilder\ that standardize how arrays are constructed, extended, and frozen. This change adds comprehensive equality comparison implementations (\PartialEq\) for all array types, ensuring consistent behavior across primitive, binary, list, struct, and dictionary arrays. Additionally, it introduces new FFI (C Data Interface) support traits (\ToFfi\, \FromFfi\) and display formatting utilities, while establishing a new \Indexable\ trait for efficient value extraction from mutable arrays.

crates/polars-arrow/src/array · high confidence

Introduction of polars-arrow crate with Avro support and nightly detection

The polars-arrow crate is introduced, consolidating code from arrow2 and establishing a new folder structure. This change adds Apache Avro read/write capabilities via the \io/avro\ module, including a zigzag decoding macro for handling corrupt files. Additionally, a build script is added to automatically enable a 'nightly' feature when the Rust compiler channel is detected as nightly.

crates/polars-arrow · high confidence

Introduction of polars-expr internal crate with license and build configuration

The polars-expr crate has been introduced as an internal sub-crate for physical expression implementations. This change adds the MIT license file, a README clarifying that the crate is internal and not for external usage, and a build script that conditionally enables a nightly feature based on the Rust toolchain version.

crates/polars-expr · high confidence

Introduction of the polars-row internal crate for row encoding

The \polars-row\ crate has been added as an internal sub-crate of Polars to provide row encodings for the DataFrame library. This new module implements the conversion of \ArrayRef\ columns into a row-oriented binary format that preserves sort order, supporting fixed-width types (integers, floats, booleans, decimals), variable-length types (strings, binary), and nested structures (lists, structs). It includes specific encodings for categorical/enum data and handles both ordered and unordered (no-order) row encoding modes to optimize performance for different use cases.

crates/polars-row · high confidence

Introduction of the polars-sql crate

Polars now includes a dedicated \polars-sql\ crate that provides a SQL transpiler, allowing SQL queries to be converted into Polars logical plans. This new module introduces the \SQLContext\ for registering tables and executing queries, a \FunctionRegistry\ for managing SQL functions and user-defined functions (UDFs), and comprehensive support for SQL syntax including complex aggregations, window functions, and grouping sets.

crates/polars-sql · high confidence

Native Iceberg and Delta deletion vector support in file scans

The file scan DSL now natively handles Iceberg V3 deletion vectors and Delta Lake deletion vectors, allowing scans to correctly apply row deletions without relying on external workarounds. Additionally, the scan logic now supports Iceberg table features such as filling missing fields with partition values and respecting the \initial-default\ metadata, ensuring schema consistency and accurate data retrieval for tables with evolving schemas or partition-based defaults.

_crates/polars-plan/src/dsl/file\scan · high confidence

Native serde support for DataFrame and Series via IPC serialization

Rust users can now directly serialize and deserialize \DataFrame\ and \Series\ objects using standard \serde\ traits. The implementation uses Apache Arrow IPC format internally to ensure fidelity, preserving column schemas, data types, and internal statistics flags (such as sortedness) during round-trips. This enables efficient, type-safe persistence and interchange of Polars data structures in Rust applications without relying on intermediate formats like JSON for the binary payload, while still exposing JSON-compatible serialization interfaces for compatibility.

crates/polars-core/src/serde · high confidence

New BinView-based Parquet string and binary decoding engine

The \polars-parquet\ crate now includes a new deserialization path for string and binary columns that targets \BinView\ (view-based) arrays instead of the previous \BinaryArray\/\LargeBinary\ approach. This new engine, located in \src/arrow/read/deserialize/binview/\, supports Plain, Dictionary, DeltaLengthByteArray, and DeltaByteArray encodings, and introduces specialized predicate evaluation kernels for \is\_in\ and equality checks that can operate directly on the view buffers. This change provides a more efficient internal representation for reading Parquet strings and binaries, reducing memory overhead and enabling faster predicate pushdown for these column types.

crates/polars-parquet · high confidence

New BinaryArray builder and FFI support in polars-arrow

The \crates/polars-arrow/src/array/binary\ module now includes a \BinaryArrayBuilder\ for constructing binary arrays incrementally, along with FFI serialization/deserialization (\ToFfi\/\FromFfi\) to enable interoperability with external Arrow implementations. This change also introduces comprehensive debug formatting and iterator support for \BinaryArray\, providing users with easier inspection and traversal of binary data structures.

crates/polars-arrow/src/array/binary · high confidence

New DSL options for Iceberg sinks and file path providers

This change introduces new internal DSL option structures to support Iceberg sink operations and flexible file path generation. It adds \IcebergSinkState\ to manage Iceberg-specific sink configuration (such as commit mode, schema mode, and catalog details) and \FileProviderType\ to abstract how output file paths are generated for Hive and Iceberg formats, including support for object-storage partitioning layouts. These components form the planning-layer foundation for the new Iceberg sink capabilities.

crates/polars-plan/src/dsl/options · high confidence

New DataFrame operations: gather, is\_sorted, and drop\_nulls in to\_dummies

This change introduces new capabilities to the DataFrame API in polars-ops. Users can now check if a DataFrame is sorted by one or more columns with configurable order and null handling via the new \is\_sorted\ method. Row selection is enhanced with a new \gather\ operation that supports selecting rows by index columns or series, including options for out-of-bounds handling. Additionally, the \to\_dummies\ method now accepts a \drop\_nulls\ parameter, allowing users to exclude null values when generating dummy variables.

crates/polars-ops/src/frame · high confidence

New DictionaryArray implementation in polars-arrow

The \crates/polars-arrow/src/array/dictionary\ module has been introduced, providing a complete implementation of the Arrow DictionaryArray type. This includes the core \DictionaryArray\ struct and its mutable counterpart, \MutableDictionaryArray\, which manages value deduplication via a \ValueMap\. The change adds support for iterating over dictionary values (both generic and typed iterators), formatting for debugging, and FFI serialization/deserialization. It also defines the \DictionaryKey\ trait, enabling dictionary keys for integer types ranging from 8-bit to 128-bit (signed and unsigned).

crates/polars-arrow/src/array/dictionary · high confidence

New IPC array readers for binary, binview, boolean, dictionary, fixed-size types, lists, maps, structs, unions, and UTF-8

The IPC read module now includes dedicated readers and skip functions for all major Arrow array types (binary, binview, boolean, dictionary, fixed-size binary, fixed-size list, list, map, struct, null, primitive, union, and UTF-8). This enables reading and skipping these array types from Arrow IPC streams, including support for compression, optional checked/unchecked modes, and variadic buffers for binview types.

crates/polars-arrow/src/io/ipc/read/array · high confidence

New IPC module with compression and schema support

The \crates/polars-arrow/src/io/ipc\ location now contains the core IPC implementation, introducing \compression.rs\ to handle LZ4 and Zstd encoding/decoding (with configurable Zstd levels) and \endianness.rs\ for native endianness checks. The \mod.rs\ file exposes the public API for reading and writing Arrow IPC formats via \FileReader\/\FileWriter\ and \StreamReader\/\StreamWriter\, along with schema definitions like \IpcField\ and \IpcSchema\ to manage dictionary IDs and field structures.

crates/polars-arrow/src/io/ipc · high confidence

New IR graph visualization and canonicalization infrastructure

The \plans/ir\ module now includes a new set of files (\canonical.rs\, \dot.rs\, \equality.rs\, \format.rs\, \hash.rs\, \inputs.rs\, \mod.rs\, \schema.rs\, \tree\_format.rs\) that provide the underlying infrastructure for improved logical plan debugging and optimization. This adds support for generating GraphViz DOT diagrams (\IRDotDisplay\) and structured text/tree representations (\IRDisplay\, \TreeFmtNode\) for the IR, allowing users to inspect plan details like predicates, projections, and cache IDs in \explain\ output. It also introduces \CanonicalIRMap\ for structural equality checking and common subplan elimination (CSE), enabling the optimizer to identify and deduplicate identical plan subgraphs.

crates/polars-plan/src/plans/ir · high confidence

New Iceberg schema handling and schema utility methods

The schema module now includes dedicated support for Iceberg physical IDs via a new \IcebergSchema\ type that maps Iceberg field IDs to columns, enabling correct handling of Iceberg table schemas including nested types and list element IDs. Additionally, the \Schema\ type now exposes a \contains\_dtype()\ method that allows users to check whether a specific data type exists within the schema, optionally searching recursively through nested structures.

crates/polars-core/src/schema · high confidence

New ListArray builder and FFI support in Polars Arrow

The \crates/polars-arrow/src/array/list\ module now includes a new \ListArrayBuilder\ and \MutableListArray\ implementation, enabling efficient construction of list-type arrays. The builder supports advanced operations such as \gather\, \extend\_nulls\, and \freeze\_reset\, while the module also introduces FFI (Foreign Function Interface) support via \To\ and \From\ traits, allowing ListArrays to be exchanged with external systems. Additionally, debug formatting and iterators for list values have been added to improve usability and inspection.

crates/polars-arrow/src/array/list · high confidence

New NDJSON read and write implementation

The \polars-json\ crate now includes a dedicated module for handling Newline Delimited JSON (NDJSON). This change introduces new \FileReader\ and \FileWriter\ components that stream data line-by-line, along with \Serializer\ and \deserialize\_iter\ functions to convert between Polars arrays and NDJSON text. It also adds schema inference capabilities (\iter\_unique\_dtypes\, \infer\_iter\) that analyze NDJSON content to determine the resulting Arrow data types, enabling users to read and write NDJSON files with proper type handling.

crates/polars-json/src/ndjson · high confidence

New Series implementations for Array, Binary, Boolean, Categorical, Date, Datetime, Decimal, and Duration types

This change introduces dedicated \SeriesWrap\ implementations for several core data types in \polars-core\, moving their logic into separate files to reduce compilation times and improve code organization. Users benefit from more robust handling of these types, including specific arithmetic operations (e.g., Date/Duration subtraction, Datetime arithmetic), optimized group-by aggregations (min, max, sum, list), and consistent hashing and sorting behaviors. The implementation ensures that logical types (like Date, Datetime, Decimal) correctly dispatch to their physical representations for computation while preserving their logical metadata, and adds support for operations like \zip\_with\, \filter\, \take\, and \append\ with proper type validation.

crates/polars-core/src/series/implementations · high confidence

New Series operations for Map canonicalization, JSON decoding, and physical type reconstruction

The \crates/polars-core/src/series/ops\ module now includes new operations to support the \Map\ data type and improve type safety. Users can now canonicalize nested \Map\ structures using \Series::canonicalize\_maps\, which normalizes keys bottom-up. JSON reading for \Map\ types is handled via \Series::from\_json\_decoded\ and \DataType::json\_decode\_dtype\, ensuring keys are valid strings, categoricals, or enums. Additionally, \Series::try\_from\_physical\ provides a safe way to reconstruct logical types (like \Map\, \Categorical\, \Decimal\) from their physical Arrow representations, validating constraints such as category codes and decimal precision. These changes also introduce \Series::full\_null\ support for \Map\ and \Extension\ types, and \Series::extend\_constant\ for efficient constant appending.

crates/polars-core/src/series/ops · high confidence

New Utf8Array implementation in polars-arrow

The \crates/polars-arrow/src/array/utf8\ module now provides a complete implementation of the \Utf8Array\ type, including immutable and mutable variants (\MutableUtf8Array\, \MutableUtf8ValuesArray\), FFI serialization/deserialization, debug formatting, and iteration support. This adds the foundational string array type to the polars-arrow crate, enabling users to construct, manipulate, and exchange UTF-8 string data via the Arrow format.

crates/polars-arrow/src/array/utf8 · high confidence

New \`polars-testing\` crate for DataFrame and Series equality assertions

The \polars-testing\ sub-crate is now available, providing Rust macros \assert\_dataframe\_equal!\ and \assert\_series\_equal!\ for unit testing Polars data structures. These assertions allow users to verify equality with configurable options, including checks for data types, column/element order, and names, as well as tolerance settings for approximate floating-point comparisons. The crate also exposes \assert\_schema\_equal\ for validating structure without comparing values.

crates/polars-testing · high confidence

New abort mechanism and warning system in polars-error

The polars-error crate now includes a new abort mechanism (abort.rs) that allows long-running computations to be interrupted via SIGINT (KeyboardInterrupt) or when out-of-core operations run out of disk space, enabling graceful returns to Python. Additionally, a warning system (warning.rs) has been added with a configurable warning function and specific warning types (Deprecation, UserWarning, CategoricalRemappingWarning, MapWithoutReturnDtypeWarning) that can be triggered via the polars\_warn! macro. The error types have also been expanded to include dedicated AssertionError, SQLInterface, and SQLSyntax variants, and Python-specific error handling has been improved with a PyErrWrap type for better integration with Python exceptions.

crates/polars-error/src · high confidence

New aggregate operations module for ChunkedArray

The \crates/polars-core/src/chunked\_array/ops/aggregate\ module has been introduced to centralize aggregation logic for \ChunkedArray\. This change adds support for computing sums (including specific handling for float16, float32, and float64 precision), minimums, maximums, and min-max pairs, with optimizations for sorted data. It also introduces dedicated modules for quantile calculations (supporting multiple methods like linear and nearest) and variance/standard deviation computations, providing the foundational aggregation traits and implementations used by the rest of the library.

_crates/polars-core/src/chunked\array/ops/aggregate · high confidence

New aggregation kernels for boolean arg\_min/arg\_max and numerical variance

The \crates/polars-arrow/src/legacy/kernels/take\_agg\ module now includes new implementations for \arg\_min\ and \arg\_max\ operations on boolean arrays, allowing users to find the position of the first false (min) or true (max) value in boolean columns. Additionally, a numerically stable online variance calculation (using Welford's algorithm) has been added, exposing kernels to compute variance for primitive arrays with or without nulls. These changes enhance the underlying arrow kernel capabilities for specific aggregation scenarios.

_crates/polars-arrow/src/legacy/kernels/take\agg · high confidence

New approximate quantile sketching and bitwise operations in the compute layer

The \polars-compute\ crate now includes the core implementation for approximate quantiles, introducing the \ApproxQuantileMethod\ enum (supporting \Auto\, \KLL\, \ReqSketch\, and \DoubleReqSketch\) and the \FinalizedSketch\ structure for estimating quantile values from data streams. Additionally, the crate adds a new \BitwiseKernel\ trait and its implementations for primitive arrays, enabling bitwise operations such as \count\_ones\, \leading\_zeros\, and reductions like \bit\_and\ and \bit\_or\ directly within the compute layer.

crates/polars-compute/src · high confidence

New binary namespace methods for slicing, indexing, and inspection

Users can now access a dedicated \bin\ namespace for binary data types, providing methods to retrieve individual bytes (\bin.get\), extract portions of binary data (\bin.slice\, \bin.head\, \bin.tail\), and inspect content (\bin.size\, \bin.contains\, \bin.starts\_with\, \bin.ends\_with\). These operations support broadcasting and negative indexing, allowing for more flexible manipulation of binary columns similar to existing string operations.

_crates/polars-ops/src/chunked\array/binary · high confidence

New bitmap utility iterators and formatting

The \polars-arrow\ bitmap utilities now include new modules for iterating and formatting bitmaps. \BitChunksExactMut\ allows iterating over mutable byte chunks of exact size, while \SlicesIterator\ efficiently extracts contiguous regions of set bits. \BitmapIter\ has been updated with optimized methods like \take\_leading\_ones\ and \take\_leading\_zeros\ for bulk consumption, and \ZipValidity\ provides a specialized iterator that skips validity checks when no nulls are present. Additionally, a new \fmt\ module enables human-readable debugging output for bitmaps with offsets and lengths.

crates/polars-arrow/src/bitmap/utils · high confidence

New categorical mapping and categories registry implementation

The \polars-dtype\ crate now introduces a new internal structure for handling categorical data types. This includes a \CategoricalMapping\ struct that manages the bidirectional mapping between string values and categorical IDs, supporting configurable maximum category limits and custom hashers. A \Categories\ registry system is added to maintain unique, shared instances of categorical mappings based on name, namespace, and physical type (u8, u16, or u32), ensuring consistency across the system. Additionally, a \CatNative\ trait and \CatSize\ type alias are defined to standardize how different integer types interact with categorical IDs, facilitating efficient storage and retrieval of categorical data.

crates/polars-dtype/src · high confidence

New configuration and formatting infrastructure in polars-core

This change introduces the core configuration and formatting infrastructure for polars-core, establishing the foundation for user-facing display and environment-based settings. It adds a new config module defining environment variables (such as POLARS\_FMT\_MAX\_ROWS and POLARS\_FMT\_TABLE\_FORMATTING) and a fmt module that implements table display logic, including configurable row/column limits, string truncation, and ASCII-safe ellipsis handling. Additionally, it provides a runtime module for managing the Rayon thread pool and a random module for global seed management, while also introducing a QueryResult enum to handle single or multiple DataFrame outputs from queries.

crates/polars-core/src · high confidence

New configuration, metrics, and hive partitioning infrastructure in polars-io

This change introduces new internal modules in \crates/polars-io\ to support advanced I/O capabilities. The \configs\ module adds environment-variable-based tuning for cloud upload concurrency and chunk sizes (e.g., \POLARS\_UPLOAD\_CONCURRENCY\), as well as cloud writer coalescing and footer read sizes. The \metrics\ module introduces an \IOMetrics\ struct to track bytes requested/received/sent and I/O session timing. The \hive\ module provides logic to materialize hive partition columns into DataFrames and merge them into schema order, while \options.rs\ defines \HiveOptions\ and \ExternalCompression\ types. Additionally, \pl\_async.rs\ implements a semaphore-based concurrency budget with an auto-tuner for cloud downloads, and \predicates.rs\ adds specialized Parquet column predicate evaluation. These components collectively enable more performant, tunable, and correct handling of hive-partitioned files and cloud I/O operations.

crates/polars-io/src · high confidence

New expression property analysis utilities for the query optimizer

Added \crates/polars-plan/src/plans/aexpr/properties\ containing helper functions to analyze Abstract Expression (AExpr) nodes. This includes methods to determine if an expression is elementwise, row-separable, or does not modify state, as well as traversal utilities to check these properties recursively across expression trees. These utilities enable the query optimizer to make more informed decisions about expression evaluation and optimization strategies.

crates/polars-plan/src/plans/aexpr/properties · high confidence

New internal async runtime and scheduling primitives

Polars introduces the \polars-async\ internal sub-crate, which provides a custom async executor and runtime utilities. This includes a NUMA-aware task scheduler (behind a feature flag), precise task metrics tracking, and specialized concurrency primitives such as single-producer single-consumer connectors, multi-consumer distributor channels, and ordered linearizers. The crate also exposes a global \RuntimeManager\ for managing Tokio-based async execution and blocking-in-place operations, replacing ad-hoc async handling with a dedicated, optimized infrastructure.

crates/polars-async · high confidence

New iterator API for ChunkedArray and List/Array types

This change introduces a new iterator implementation for \ChunkedArray\ and its specialized variants (\ListChunked\, \ArrayChunked\). Users can now iterate over chunked arrays using \iter()\ (which handles nulls) and \no\_null\_iter()\ (which skips nulls), returning physical types. For list and array types, new methods \series\_iter()\ and \no\_null\_series\_iter()\ allow iterating over the contained series. These iterators implement the \PolarsIterator\ trait, ensuring they are \ExactSizeIterator\, \DoubleEndedIterator\, \Send\, \Sync\, and \TrustedLen\, which may improve performance and safety when iterating over polars data structures.

_crates/polars-core/src/chunked\array/iterator · high confidence

New key-row representation for group-by and join operations

A new \key\_rows\ module has been added to \polars-expr\ to store multi-column group-by and join keys as fixed-width rows of \u64\ words. This implementation introduces \KeyRowLayout\ to define the row schema, \KeyRowKeys\ to hold the key data, and \KeyRowIndexMap\ to perform batched lookups and insertions with hash-based collision handling. The module also includes \HotKeyRows\ for efficient access to frequently used keys and \KeyRowCollector\ for aggregating keys. This change provides the underlying data structures for the streaming out-of-core group-by and join optimizations.

_crates/polars-expr/src/key\rows · high confidence

New legacy array builder traits and implementations for List, FixedSizeList, and Utf8/Binary arrays

The \crates/polars-arrow/src/legacy/array\ module now includes new builder utilities and traits to simplify the construction of complex array types. \AnonymousBuilder\ structs for \ListArray\ and \FixedSizeListArray\ handle the aggregation of inner arrays, automatically managing offsets, validity bitmaps, and the conversion of nested null types during concatenation. Additionally, \ListFromIter\ provides safe, zero-copy iteration paths for constructing List arrays from iterators of primitives, booleans, and binary views, while \Utf8FromIter\ and \BinaryFromIter\ offer optimized, trusted-length iteration for creating \Utf8Array\ and \BinaryArray\ instances directly from value iterators.

crates/polars-arrow/src/legacy/array · high confidence

New list operations: count\_matches, dispersion, and optimized min/max/sum/mean

This change introduces several new list namespace operations and optimizes existing ones. Users can now count occurrences of a specific value in a list using \list.count\_matches\, and calculate dispersion metrics (median, standard deviation, variance) via \list.median\, \list.std\, and \list.var\. Additionally, \list.min\ and \list.max\ now use a fast-path implementation for numerical types, and \list.sum\ and \list.mean\ have optimized fast-paths for numerical data, improving performance for these common aggregations.

_crates/polars-ops/src/chunked\array/list · high confidence

New low-level I/O utility modules in polars-io

The \crates/polars-io/src/utils\ directory has been restructured into a set of new, dedicated modules that provide foundational I/O capabilities for the library. \byte\_source.rs\ introduces a \ByteSource\ trait and implementations like \FileByteSource\ and \BufferByteSource\ to abstract asynchronous byte fetching (prefix, suffix, range) and manage global read concurrency via semaphores. \compression.rs\ adds a \CompressedReader\ enum for streaming decompression of GZIP, ZLIB, and ZSTD data, replacing older in-memory decompression methods. \file.rs\ defines a unified \Writable\ enum to handle local and cloud file writing with configurable sync behaviors, while \mkdir.rs\ provides utilities for recursively creating parent directories for output paths. Additional modules include \bytes\_bufferer.rs\ for efficient byte chunking, \stream\_buf\_reader.rs\ for async streaming buffers, and \slice.rs\ for calculating file-specific slice offsets in multi-file scans.

crates/polars-io/src/utils · high confidence

New optimized rolling aggregation kernels for null-free data

The \polars-compute\ crate now includes a dedicated \no\_nulls\ module with new, optimized implementations for rolling sum, mean, min/max, variance, skew, kurtosis, quantile, and rank operations. These kernels are designed to handle data series that contain no null values, offering improved performance and numerical stability for these common aggregation tasks.

_crates/polars-compute/src/rolling/no\nulls · high confidence

New per-column and skip-batch predicate logic for optimized scanning

The \polars-plan\ crate now includes new modules (\column\_expr.rs\ and \skip\_batches.rs\) that split row predicates into per-column constraints and generate statistics-based skip-batch predicates. This enables the query engine to prune Parquet row groups and skip record batches more effectively by leveraging min/max/null-count statistics, including support for struct field paths and specialized handling for string operations (regex, startswith, endswith) and \is\_in\ checks.

crates/polars-plan/src/plans/aexpr/predicates · high confidence

New plan-time statistics and cost estimation subsystem

The query planner now includes a new statistics module (\plans/stats\) that provides cardinality estimates and cost models for query execution. This introduces a \Card\ type to represent row counts as exact, approximate, or unknown, and a \ScanStats\ structure to track per-column statistics (such as null counts and integer ranges) derived from source metadata. The \node\_stats\ function estimates the row output of individual plan nodes, while \subplan\_cost\ calculates the total evaluation cost of a query subtree. These estimates allow the optimizer to make more informed decisions regarding join ordering, filter pushdown, and cache usage, leading to improved query performance. Also in this area: 9 commits (5 fixes) fixing crates/polars-plan; 4 commits (1 fix) modifying crates/polars-plan; Improved Parquet scan statistics and Iceberg sink options.

crates/polars-plan/src/plans/stats · medium confidence · unverified

New polars-dylib crate for dynamic library builds

A new \polars-dylib\ crate has been added to allow building Polars as a dynamic library. Users can now configure their Cargo workspace to use \polars-dylib\ as the package and set the \prefer-dynamic\ rustflag to produce a shared library instead of a static one. The crate re-exports the main \polars\ public API and conditionally exposes internal components like \polars\_arrow\, \polars\_core\, and \polars\_python\ when specific features are enabled.

crates/polars-dylib · high confidence

New row-based DataFrame construction and transposition logic

The \crates/polars-core/src/frame/row\ module now provides the core implementation for building DataFrames from row-wise data and for transposing DataFrames. It introduces \AnyValueBuffer\ to efficiently accumulate heterogeneous values into typed column buffers, enabling \DataFrame::from\_rows\ and \from\_rows\_iter\_and\_schema\ to construct DataFrames from iterators of \Row\ structs while validating row widths and inferring schemas. Additionally, the new \transpose\ and \transpose\_from\_dtype\ implementations handle the expensive operation of swapping rows and columns, supporting numeric types directly and using \AnyValueBuffer\ for complex types, while ensuring contiguous memory layouts and handling column name preservation.

crates/polars-core/src/frame/row · high confidence

New series operations module in polars-ops

The \polars-ops\ crate now exposes a dedicated \series\ module containing a comprehensive set of element-wise and aggregation operations for Series. This includes implementations for absolute value (\abs\), approximate quantiles (\approx\_quantile\), minimum/maximum index finding (\arg\_min\/\arg\_max\), binning (\binning\), bitwise operations (\bitwise\), business day calculations (\business\), clipping (\clip\), array concatenation (\concat\_arr\), cumulative aggregations (\cum\_agg\), and data cutting (\cut\). These functions provide the underlying logic for various Series methods, centralizing their implementation within the operations crate.

crates/polars-ops/src/series · high confidence

New sorted join kernels for inner and left joins

The \crates/polars-arrow/src/legacy/kernels/sorted\_join\ module now includes implementations for inner and left joins on sorted arrays. These new kernels (\inner.rs\ and \left.rs\) provide optimized join logic that handles duplicate keys and array offsets, supporting more efficient sorted data operations within the Polars Arrow library.

_crates/polars-arrow/src/legacy/kernels/sorted\join · high confidence

New streaming IPC writer implementation

A new streaming IPC writer implementation has been added in \crates/polars-arrow/src/io/ipc/write2\, introducing a new module structure (\array\, \footer\, \message\, \schema\) to handle Arrow IPC serialization. This implementation supports writing record batches and dictionary batches with optional LZ4 and ZSTD compression, utilizes \polars\_arrow\_format\ for flatbuffer serialization, and targets Arrow IPC Metadata Version V5. The new writer handles various array types (primitive, binary, utf8, list, struct, etc.) and includes logic for padding buffers to 64-byte alignment and managing continuation markers, providing a foundation for optimized streaming IPC output.

crates/polars-arrow/src/io/ipc/write2 · high confidence

New streaming engine introduces optimized groupby implementations

The \crates/polars-expr/src/groups\ module now implements the grouping logic for the new streaming engine, replacing the previous approach with specialized hash groupers. This includes \SingleKeyHashGrouper\ for primitive, temporal, and categorical types with block-based prefetching; \BinviewHashGrouper\ for efficient string and binary view handling; \KeyRowHashGrouper\ for multi-column keys using a row-based layout; and \RowEncodedHashGrouper\ as a fallback for complex types. These components enable partitioned, parallel groupby operations and support streaming reductions.

crates/polars-expr/src/groups · high confidence

New streaming engine reduction implementations

The streaming engine now implements native reduction kernels for a wide range of aggregations, including \any\/\all\ (with \ignore\_nulls\ support), \approx\_n\_unique\, \approx\_quantile\, bitwise operations (\and\/\or\/\xor\), \count\, \cov\/\corr\, \first\/\last\ (including \first\_non\_null\/\last\_non\_null\), \has\_nulls\, and \implode\. These new \GroupedReduction\ implementations in \crates/polars-expr/src/reduce\ allow the streaming engine to execute these operations natively, improving performance and consistency with the in-memory engine.

crates/polars-expr/src/reduce · high confidence

New string operations and performance optimizations

This update introduces several new string manipulation capabilities and improves performance for existing ones. Users can now escape regex special characters with \str.escape\_regex\, normalize Unicode strings with \str.normalize\ (supporting NFC, NFD, NFKC, and NFKD forms), and reverse strings with \str.reverse\ (handling grapheme clusters correctly). The \str.join\ function (formerly \str.concat\) now supports an \ignore\_nulls\ option to control null propagation. Performance is enhanced through a new \LiteralChain\ optimizer for simple \LIKE\-style patterns, faster case conversion in \to\_lowercase\ and \to\_uppercase\, and optimized padding in \zfill\, \pad\_start\, and \pad\_end\ that correctly handles Unicode character counts. Additionally, \str.extract\_groups\ now properly returns a struct series with correct field names and types, and \str.json\_decode\ supports inferring schema or decoding to explicit types including maps and enums.

_crates/polars-ops/src/chunked\array/strings · high confidence

New sublist gather kernels for List and FixedSizeList arrays

The \polars-compute\ crate now includes new gather kernels for extracting specific elements from nested list structures. This change introduces \sublist\_get\ functions for both \ListArray\ and \FixedSizeListArray\, allowing users to retrieve values at a specific index from within each list element. The implementation handles negative indexing, out-of-bounds checks, and null values correctly, exposing these capabilities through the \polars\_compute::gather::sublist\ module.

crates/polars-compute/src/gather/sublist · high confidence

New trusted-length iteration traits for BooleanArray and Vec

The \polars-arrow\ crate now exposes new traits in the \legacy/trusted\_len\ module to optimize iteration over known-length sequences. \TrustedLenPush\ provides unchecked push and extend operations for \Vec\<T\>\, allowing callers to bypass capacity checks when the iterator length is guaranteed. \FromIteratorReversed\ enables efficient reverse iteration, and \BooleanArray\ now implements \FromTrustedLenIterator\ and \FromIteratorReversed\ to build arrays directly from trusted iterators using \BitmapBuilder\ and \MutableBitmap\, improving performance for boolean data construction.

_crates/polars-arrow/src/legacy/trusted\len · high confidence

New typed builders for ChunkedArray construction

The \crates/polars-core/src/chunked\_array/builder\ module now provides dedicated builder implementations for Boolean, Categorical, Fixed-Size List, Null, Primitive, String, and Binary data types. These builders expose a unified \ChunkedBuilder\ trait with \append\_value\, \append\_null\, and \finish\ methods, allowing users and internal code to efficiently construct typed \ChunkedArray\ instances from iterators or slices while handling nulls and memory pre-allocation correctly.

_crates/polars-core/src/chunked\array/builder · high confidence

New unified arithmetic kernel implementation for primitive types

The \crates/polars-compute/src/arithmetic\ module has been restructured to provide a comprehensive, unified arithmetic kernel for all primitive numeric types (signed integers, unsigned integers, floats, and Float16). This change introduces a new \ArithmeticKernel\ trait and \PrimitiveArithmeticKernelImpl\ that standardizes operations such as addition, subtraction, multiplication, floor/truncation division, modulo, absolute value, and negation across all numeric data types. It specifically adds support for Float16 (\pf16\) arithmetic, implements optimized scalar operations (e.g., power-of-two multiplication shifts), and ensures consistent handling of edge cases like division by zero (returning nulls or zero as appropriate) and validity propagation. This provides a single, robust foundation for all numeric computations in Polars.

crates/polars-compute/src/arithmetic · high confidence

Reorganized IPC I/O implementation with new async reader and metadata support

The IPC I/O module has been restructured into distinct files for file-based reading (ipc\_file.rs), streaming (ipc\_stream.rs), memory-mapped access (mmap.rs), and a new asynchronous cloud reader (ipc\_reader\_async.rs). This change introduces an async API for reading IPC files from cloud storage, adds support for reading and writing custom schema-level metadata, and exposes record batch statistics as configurable options for both scanning and writing.

crates/polars-io/src/ipc · high confidence

Repository initialization with comprehensive development tooling and policies

The repository is initialized with a complete set of configuration files and policies to support development. This includes a Makefile for building and linting, Nix flake for reproducible development environments, and configuration for formatting (rustfmt, dprint, ruff), linting (clippy, typos), and dependency auditing (cargo-deny). A new AI Usage Policy is established, strictly forbidding automated agents from interacting with the repository and requiring disclosure of AI assistance. Additionally, a security policy is defined, and documentation is structured using Zensical.

(repo-wide) · high confidence

Scalar values are now serializable and support new data types

The \Scalar\ struct in \polars-core\ now implements \Serialize\ and \Deserialize\ via a new \SerializableScalar\ enum, enabling users to persist and restore scalar values (including Categorical, Enum, and Struct types) to and from formats like JSON. This change also introduces native support for constructing and handling scalars for newly added data types, specifically 128-bit integers (\Int128\, \UInt128\) and 16-bit floats (\Float16\), as well as \Map\ and \Decimal\ types, ensuring these values can be correctly created and serialized.

crates/polars-core/src/scalar · high confidence

StructChunked implementation and DataFrame conversion

The \StructChunked\ type is now implemented in \crates/polars-core/src/chunked\array/struct\\, providing core functionality for creating struct arrays from series or columns, handling field broadcasting, and converting between logical and physical representations. Additionally, a new \DataFrame::into\_struct\ method allows users to convert a DataFrame directly into a \StructChunked\.

_crates/polars-core/src/chunked\array/struct\ · high confidence_

Support for Hugging Face paths and secure temporary directories

Users can now read data directly from Hugging Face datasets and spaces using the \hf://\ URI scheme, with the system automatically handling API pagination, revision resolution, and percent-encoding. Additionally, temporary directories created by Polars are now restricted to owner-only permissions (0o700) on Unix systems to enhance security, a behavior that can be overridden by setting the \POLARS\_ALLOW\_UNSECURED\_TEMP\_DIR\ environment variable.

_crates/polars-io/src/path\utils · high confidence

Support for custom file-level key-value metadata in Parquet writers

The Parquet write implementation in \crates/polars-io/src/parquet/write\ now supports attaching custom key-value metadata to the output file. This is enabled via the \key\_value\_metadata\ field in \ParquetWriteOptions\ and the \with\_key\_value\_metadata\ builder method on \ParquetWriter\. The new \KeyValueMetadata\ type allows users to provide static key-value pairs or dynamic functions (in Rust and Python) to generate metadata at write time, which is then collected and written to the Parquet footer via the \BatchedWriter\ and \ParquetWriter\ finish methods.

crates/polars-io/src/parquet/write · high confidence

Support for named and anonymous aggregations in the query plan

The query plan now supports anonymous aggregations, allowing users to define custom aggregation logic via opaque functions that can be serialized and deserialized. This change introduces new traits and types (\AnonymousAgg\, \OpaqueStreamingAgg\) to handle these custom aggregations, enabling more flexible and extensible data processing capabilities within Polars.

crates/polars-plan/src/dsl/expr/anonymous · high confidence

Removals

Removal of legacy Column and error modules

The internal \Column\ struct and the \PolarsError\ error type have been removed from the codebase. This eliminates the previous dynamic, chunk-based column implementation and its associated error handling, reflecting a shift toward the newer \ChunkedArray\ and \Series\ abstractions that now handle data storage and arithmetic operations.

src · high confidence

Architecture

Cast kernels moved to polars-compute with new binary-view support

The casting logic previously located in polars-arrow has been relocated to the polars-compute crate, introducing dedicated source files for binary, binary-view, boolean, decimal, dictionary, primitive, and UTF-8 conversions. This refactor adds support for casting to and from BinaryView and Utf8View arrays, enabling more efficient handling of variable-length data, and includes updated implementations for decimal-to-integer/float conversions and dictionary packing.

crates/polars-compute/src/cast · high confidence

Hash join implementation moved to polars-ops

The hash join logic has been relocated from polars-core to the polars-ops crate, introducing a new modular structure under \crates/polars-ops/src/frame/join/hash\_join\. This change organizes the join implementation into dedicated modules for inner, left, outer, semi/anti, and sort-merge joins, while exposing the \SeriesJoin\ trait for dispatching join operations across different data types.

_crates/polars-ops/src/frame/join/hash\join · high confidence

In-memory engine scan executors moved to dedicated module

The scan execution logic for the in-memory engine has been reorganized into a new \crates/polars-mem-engine/src/executors/scan\ module. This change introduces \DataFrameExec\ for producing DataFrames from in-memory data with projection support, and \AnonymousScanExec\ for handling generic scans with unified arguments, predicate pushdown, and window function flagging. Additionally, \PythonScanExec\ is now located here, managing Python-based scans (including IO plugins and PyArrow) with batched processing, predicate serialization, and schema validation. This refactoring consolidates scan-related executor code into a single, dedicated location within the in-memory engine crate.

crates/polars-mem-engine/src/executors/scan · high confidence

Introduce in-memory engine physical plan creation and multiple-plan execution

The in-memory engine's physical plan creation logic has been moved into the new \polars-mem-engine\ crate, exposing \create\_physical\_plan\ and \create\_multiple\_physical\_plans\ in the planner module. This change introduces a \CachePrefiller\ executor to handle cache nodes and enables the concurrent execution of multiple physical plans via \MultiplePhysicalPlans::execute\, which parallelizes plan execution across Rayon threads while managing branch indices and state splitting.

crates/polars-mem-engine/src/planner · high confidence

Introduction of polars-ops crate for shared operations

A new \polars-ops\ crate has been added to the repository, serving as a central location for shared DataFrame and Series operations. This module exposes functionality such as join operations, sorting checks (\is\_sorted\), and conditional pivoting (\unpivot\), making these capabilities available to other crates in the ecosystem through a dedicated prelude.

crates/polars-ops/src · high confidence

Introduction of the polars-mem-engine crate

The in-memory execution engine has been moved into its own dedicated crate, \polars-mem-engine\. This structural change reorganizes the codebase by exposing core execution components such as the \Executor\ trait, physical plan creation, and streaming executor building, while establishing a local prelude to manage internal dependencies on core, expression, operations, and planning modules.

crates/polars-mem-engine/src · high confidence

Introduction of the polars-time internal crate

The \polars-time\ crate has been introduced as an internal sub-crate of the Polars library to centralize time-related utilities. This change reorganizes the codebase by moving temporal logic—including date and datetime extraction methods, duration handling, rolling window aggregations, and dynamic group-by operations—into this dedicated location. The crate is not intended for external usage; users should continue to rely on the main Polars crate for these capabilities.

crates/polars-time · high confidence

Logical types restructured into dedicated module files

The internal implementation of logical data types (Date, Datetime, Time, Duration, Decimal, Categorical, Extension, and Map) has been reorganized from a monolithic structure into individual, dedicated source files within the \crates/polars-core/src/chunked\_array/logical\ directory. This change improves code maintainability and modularity by separating the logic for each type into its own module, while preserving the existing \Logical\<T, Physical\>\ wrapper pattern and \LogicalType\ trait interface that users interact with.

_crates/polars-core/src/chunked\array/logical · high confidence

Migrate physical expression evaluation to the polars-expr crate

The physical expression evaluation logic has been moved from the core crate into the new \polars-expr\ crate. This change introduces dedicated expression types for aggregations, binary operations, filtering, gathering, and list/array evaluation, establishing a new execution layer that handles group-aware contexts and scalar propagation.

crates/polars-expr/src/expressions · high confidence

New IR-based query planner and optimizer infrastructure

The query planning layer has been refactored to use a new Intermediate Representation (IR) with dedicated arena-based storage for logical plans and expressions. This introduces the \IRBuilder\ for constructing plans, \ExprIR\ for managing expression metadata and output names, and a new \prune\ module for optimizing plans by removing unreachable nodes. The change also adds support for anonymous scans via the \AnonymousScan\ trait, refactors hive partitioning logic into a dedicated module, and implements new optimizer rules for pruning and schema derivation.

crates/polars-plan/src/plans · high confidence

New chunked-array operations moved to polars-ops

This change relocates the implementations of several chunked-array operations—including covariance/correlation, histogram binning, mode calculation, peak detection, repeat-by, scatter, and top\_k/bottom\_k—into the polars-ops crate. For users, this consolidates these operations into a single, dedicated module, ensuring consistent availability and feature-gating across the library while keeping the core polars-core crate leaner.

_crates/polars-ops/src/chunked\array · high confidence

New expression and logical-plan visitor infrastructure

The \polars-plan\ crate now includes a dedicated visitor module (\plans/visitor\) that provides a \TreeWalker\ trait and concrete implementations for traversing and rewriting both expression trees (\Expr\, \AexprNode\) and logical-plan nodes (\IRNode\). This infrastructure introduces specific recursion controls (\VisitRecursion\, \RewriteRecursion\) and arena-aware mapping, enabling the query engine to systematically walk, inspect, and modify internal plan structures without mutating the arena by default unless explicitly requested.

crates/polars-plan/src/plans/visitor · high confidence

New expression dispatch layer for array, binary, bitwise, boolean, and other namespaces

The \polars-expr\ crate now includes a dedicated dispatch module that routes expression functions for array, binary, bitwise, boolean, business, categorical, cumulative, datetime, extension, group-by, horizontal, list, map, and membership operations to their respective implementations. This new structure centralizes the mapping of internal IR function variants (such as \IRArrayFunction\, \IRBinaryFunction\, \IRBooleanFunction\, etc.) to their execution logic, providing a unified and organized entry point for evaluating these diverse expression types within the Polars engine.

crates/polars-expr/src/dispatch · high confidence

New expression evaluation engine in polars-expr crate

The \crates/polars-expr/src\ directory has been restructured into a new, dedicated crate for expression evaluation. This change introduces a new \planner\ module that converts abstract expression IRs into physical expressions, a new \hash\_keys\ module providing optimized hashing strategies (including single-key, binview, and row-encoded variants) for grouping and joining operations, and a new \EvictIdx\ type for managing eviction bits in streaming contexts. This refactoring separates the expression evaluation logic from the plan construction, laying the groundwork for the new streaming engine.

crates/polars-expr/src · high confidence

New internal expression infrastructure for canonicalization, determinism, and constraint solving

The \crates/polars-plan/src/plans/aexpr\ module has been restructured into a set of specialized sub-modules that provide the foundation for advanced query optimization. This includes \canonical.rs\ for structural expression equality and common-subexpression elimination, \determinism.rs\ to identify inherently non-deterministic expressions (like UDFs) to prevent unsafe optimizer rewrites, and \filter\_constraint.rs\ for analyzing and tightening filter predicates. Additionally, \builder.rs\ introduces a fluent API for constructing \AExpr\ nodes, while \equality.rs\ and \hash.rs\ provide robust comparison and hashing logic for the expression arena.

crates/polars-plan/src/plans/aexpr · high confidence

New internal polars-defs crate for shared definitions

The new internal \polars-defs\ crate consolidates shared option, argument, and enum definitions used by both the Polars query planner and compute crates. This includes definitions for expressions (e.g., \SetOperation\, \RankMethod\), join arguments (e.g., \JoinArgs\, \JoinType\), and temporal operations (e.g., \Duration\, \DynamicGroupOptions\, \RollingOptionsDynamicWindow\). This crate is not intended for external usage. Also in this area: 1 commit (1 fix) fixing crates/polars-defs; Support for full joins in join\_where and expanded type support for distinctness checks.

crates/polars-defs · medium confidence · unverified

Refactor DSL-to-IR conversion into modular sub-components

The \dsl\_to\_ir\ conversion logic has been reorganized from a single large module into distinct, focused sub-modules (\concat\, \datatype\_fn\_to\_ir\, \expr\_expansion\, \expr\_to\_ir\, \functions\, \join\, \scans\, \sql\, \utils\). This structural change improves code maintainability and separation of concerns for specific conversion tasks (such as handling joins, scans, and expression expansion) without altering the external behavior of the query planner.

_crates/polars-plan/src/plans/conversion/dsl\_to\ir · high confidence

Refactor FunctionExpr module into namespaced submodules

The \crates/polars-plan/src/dsl/function\expr\ module has been reorganized into distinct submodules (e.g., \array\, \binary\, \boolean\, \datetime\, \list\, \strings\, \struct\\) to better organize the internal representation of expression functions. This change introduces specific enums like \ArrayFunction\, \BinaryFunction\, and \BooleanFunction\ to handle namespace-specific operations, while the main \FunctionExpr\ enum now dispatches to these specialized types. This structural refactor improves code maintainability and clarity for the expression DSL without altering the external API.

_crates/polars-plan/src/dsl/function\expr · high confidence

Refactor expression functions into IRFunctionExpr variants

The expression function dispatch logic has been restructured to use specific IRFunctionExpr enum variants (such as IRArrayFunction, IRListFunction, IRBooleanFunction, and IRStringFunction) instead of the previous generic FunctionExpr. This change moves the schema resolution (get\_field) and execution options (function\_options) into these dedicated modules, providing stricter type safety and clearer separation of concerns within the query planner.

_crates/polars-plan/src/plans/aexpr/function\expr · high confidence

Refactored ChunkedArray operations into modular source files

The \crates/polars-core/src/chunked\_array/ops\ module has been reorganized from a single monolithic file into distinct, modular source files (e.g., \any\_value.rs\, \append.rs\, \apply.rs\, \arity.rs\, \binning.rs\, \bit\_repr.rs\, \bits.rs\, \bitwise\_reduce.rs\, \chunkops.rs\, \compare\_inner.rs\, \decimal.rs\). This change improves code maintainability and separation of concerns by grouping related operations—such as element-wise application, bitwise reductions, chunk management, and type-specific conversions—into their own dedicated modules without altering the public API or user-facing behavior.

_crates/polars-core/src/chunked\array/ops · high confidence

Refactored core utility modules for type coercion and data flattening

The \crates/polars-core/src/utils\ module has been reorganized into dedicated sub-modules (\any\_value\, \cut\, \flatten\, \schema\, \series\, \supertype\) to improve code structure and maintainability. This change introduces explicit logic for determining supertypes between data types (including lossless numeric upcasts and struct field validation), provides utilities for flattening DataFrames and Series into their constituent chunks, and adds helper functions for generating cut labels and schemas. These internal refactors support more robust type coercion, casting error reporting, and chunk management throughout the Polars core.

crates/polars-core/src/utils · high confidence

Refactored lazy scan API to use UnifiedScanArgs and PlRefPath

The lazy scan implementation in \crates/polars-lazy/src/scan\ has been restructured to use a unified \UnifiedScanArgs\ argument structure and the new \PlRefPath\ type for path handling. This change standardizes how scan options (such as schema, row index, cloud options, and hive settings) are passed to the underlying DSL builders for CSV, Parquet, IPC, and NDJSON formats. Users interacting with the Rust API will see these internal types consolidated, ensuring consistent behavior across different file scan methods and preparing the codebase for further unified scan optimizations.

crates/polars-lazy/src/scan · high confidence

Refactored predicate pushdown optimizer into modular sub-components

The predicate pushdown optimizer logic has been reorganized from a single monolithic module into distinct, specialized files (\dynamic.rs\, \group\_by.rs\, \hive.rs\, \join.rs\, \keys.rs\, \rename.rs\, \utils.rs\). This change introduces \DynamicPred\ for runtime-evaluated predicates, implements hive-partition-aware rewriting for \GroupBy\ and \Join\ operations, and adds predicate deduplication and column-reference mapping utilities. Users benefit from improved optimizer maintainability and specific performance enhancements in hive-partitioned scans and group-by operations, while the core predicate pushdown behavior remains consistent.

_crates/polars-plan/src/plans/optimizer/predicate\pushdown · high confidence

Reorganized CSV I/O module structure

The CSV reading and writing functionality has been reorganized into a new module structure under \crates/polars-io/src/csv\. The \mod.rs\ file now serves as the entry point, exposing \read\ and \write\ submodules, which simplifies the internal organization of CSV-related code without changing the external API.

crates/polars-io/src/csv · high confidence

Reorganized DSL code into modular namespace files

The expression DSL implementation in \crates/polars-plan/src/dsl\ has been restructured from a monolithic layout into distinct, modular files organized by feature area. This change introduces dedicated modules for arithmetic operations (\arithmetic.rs\), conditional logic (\arity.rs\), and specialized namespaces for data types including arrays (\array.rs\), binary data (\binary.rs\), bitwise operations (\bitwise.rs\), categoricals (\cat.rs\), dates (\dt.rs\), and lists (\list.rs\). By separating these concerns, the codebase now provides a clearer, more maintainable structure for expression builders while preserving the existing public API surface.

crates/polars-plan/src/dsl · high confidence

Reorganized DSL functions into dedicated modules

The expression functions in \crates/polars-plan/src/dsl/functions\ have been reorganized into separate, feature-gated modules (such as \business\, \concat\, \correlation\, \horizontal\, \range\, \selectors\, and \temporal\). This change improves code structure and maintainability by grouping related functions, while preserving the existing public API surface through the \mod.rs\ re-exports.

crates/polars-plan/src/dsl/functions · high confidence

Reorganized Parquet module structure in polars-io

The Parquet functionality within the polars-io crate has been reorganized to improve modularity. A new metadata module was introduced to expose file metadata and statistics types by re-exporting them from the underlying polars-parquet crate, while the main parquet module now explicitly declares its sub-modules for reading, writing, and metadata handling.

crates/polars-io/src/parquet · high confidence

Reorganized join implementation into polars-ops

The join logic has been moved from polars-core to polars-ops, restructuring the codebase into modular files (args, cross\_join, dispatch\_left\_right, general, hash\_join, merge\_join, merge\_sorted, validation). This change improves code organization and maintainability without altering the user-facing join API or behavior.

crates/polars-ops/src/frame/join · high confidence

Behavioural changes

10 commits (3 fixes) modifying crates/polars-io/src/cloud/concurrency

A change to existing behaviour in crates/polars-io/src/cloud/concurrency — 10 commits (3 fixs), 5 files: “chore(rust): Disallow usage of \tokio::spawn\ in clippy (\#28123)”; “fix: Clamp inflight budget from HTTP rate-limiter (\#29497)”; “fix: Avoid segfault on environment changes after a cloud scan (\#30665)”; and 7 more (refs \#28123, \#29497, \#30665, \#27924, \#29131, …).

crates/polars-io/src/cloud/concurrency · medium confidence · unverified

28 commits (3 fixes) modifying crates/polars/src/docs

A change to existing behaviour in crates/polars/src/docs — 28 commits (3 fixs), 3 files: “refactor(rust!): Rename \Utf8\ data type to \String\ (\#13224)”; “chore(rust): Re-organize folder structure for Rust crates (\#10141)”; “docs: Fix example of \as\_struct\ (\#19116)”; and 25 more (refs \#13224, \#10141, \#19116, \#11583, \#10251, …).

crates/polars/src/docs · medium confidence · unverified

4 commits (0 fixes) modifying crates/polars-cli

A change to existing behaviour in crates/polars-cli — 4 commits, 10 files: “feat(rust): Script and stdin input for polars-cli (\#30752)”; “test(rust): Replace the golden files of polars-cli with plain tests (\#30776)”; “feat(rust): Output formats for polars-cli (\#30754)”; and 1 more (refs \#30752, \#30776, \#30754, \#30750).

crates/polars-cli · medium confidence · unverified

4 commits (0 fixes) modifying crates/polars-core/src/chunked\_array/temporal/string

A change to existing behaviour in crates/polars-core/src/chunked\_array/temporal/string — 4 commits, 4 files: “refactor(rust): Remove polars-time dep from polars-io (\#29342)”; “refactor(rust): Rename arrow(-format) to polars-arrow(-format) (\#29356)”; “chore: Remove typos over all of Polars (\#30553)”; and 1 more (refs \#29342, \#29356, \#30553, \#29483).

_crates/polars-core/src/chunked\array/temporal/string · medium confidence · unverified

5200 commits (2511 fixes) modifying py-polars

A change to existing behaviour in py-polars — 5200 commits (2511 fixs), 806 files: “fix: Reverse on chunked \struct\ (\#25281)”; “fix: Fix \to\_datetime()\ fallible identification (\#23735)”; “fix: Handle column indexing in \read\_parquet\/\read\_csv\ with pyarrow reader (\#27397)”; and 5197 more (refs \#25281, \#23735, \#27397, \#25284, \#16198, …).

py-polars · medium confidence · unverified

AVX-512 SIMD acceleration for filter operations

The filter implementation in \polars-compute\ now includes dedicated AVX-512 kernels for primitive types (u8, u16, u32, u64) and optimized boolean filtering. On x86\_64 systems with AVX-512 support, these new routines use hardware instructions like \maskz\_compress\ to significantly speed up array filtering, particularly for sparse masks. The change also introduces a specialized sparse-path algorithm for boolean filters to handle low-density masks more efficiently.

crates/polars-compute/src/filter · high confidence

Automated Cargo configuration and warning-aware build wrapper

The build system now uses two new Python tools to manage the Cargo environment and build behavior. The \update-cargo-env.py\ script automatically generates \.cargo/config.generated.toml\ to set Python-specific environment variables (such as \PYO3\_ENVIRONMENT\_SIGNATURE\ and \PYO3\_PYTHON\) and propagate \RUSTFLAGS\ from the shell. It also configures \split-debuginfo = "unpacked"\ for the \dev\ and \debug-release\ profiles on Linux to prevent link-time failures caused by debug symbol size limits. Additionally, \cargo-fail-warning.py\ is introduced as a wrapper for Cargo commands; it parses JSON diagnostic output to detect warnings, prints them to stderr, and ensures the build process continues (rather than failing immediately on the first warning) while still exiting with a non-zero status if any warnings were encountered.

tools · high confidence

CSV reading restructured into a new builder-based API with streaming and decompression support

The CSV reading implementation in \polars-io\ has been reorganized into a new modular structure under \csv/read\, introducing \CsvReadOptions\ and \CsvReader\ as the primary configuration and execution entry points. This change replaces the previous incremental batched reader with a new streaming CSV source that supports parallel reading, streaming decompression (gzip, zlib, zstd), and cloud storage access. Users benefit from improved performance through width-aware chunking, batched UTF-8 validation, and parallel row counting, while gaining new capabilities like \skip\_lines\ for skipping raw newline-delimited lines and \infer\_schema\_files\ for multi-file schema inference. The refactoring also includes fixes for edge cases such as header duplication with leading empty lines, incorrect row counts with comment prefixes, and proper handling of quoted newlines during line skipping.

crates/polars-io/src/csv/read · high confidence

CSV writer options and serialization logic restructured

The CSV write module has been reorganized to use a dedicated \CsvWriterOptions\ and \SerializeOptions\ configuration structure, replacing the previous inline builder pattern. This change introduces new serialization capabilities, including support for scientific notation on floats (\float\_scientific\), configurable decimal separators (\decimal\_comma\), and explicit control over quoting styles (\QuoteStyle\). The \CsvWriter\ now delegates serialization logic to a new \CsvSerializer\ component, which handles type-specific formatting for dates, times, and floats, while also enforcing that nested data types (lists, structs) are rejected as they are unsupported in CSV format.

crates/polars-io/src/csv/write · high confidence

Client-side validation for Polars Cloud execution eligibility

A new client-side check ensures that logical plans are eligible for execution on Polars Cloud before serialization. The \prepare\_cloud\_plan\ function now validates that the plan contains a sink, rejects scans of opened files or in-memory buffers, and disallows memory sinks. This validation can be bypassed by setting the \POLARS\_SKIP\_CLIENT\_CHECK\ environment variable.

crates/polars-plan/src/client · high confidence

DataFrame operations restructured into modular components

The \polars-core\ DataFrame implementation has been reorganized into distinct modules (\arithmetic\, \builder\, \chunks\, \explode\, \filter\, \from\, \horizontal\, \projection\, \top\_k\, \upstream\_traits\, \validation\) to improve code maintainability and performance. This change introduces a new \DataFrameBuilder\ for efficient column-wise construction, implements parallel arithmetic operations for both DataFrames and Series, and adds optimized chunking and filtering logic. It also refactors horizontal concatenation to handle scalar broadcasting and null-padding more robustly, and introduces a new \AmortizedColumnSelector\ for faster column lookups in wide DataFrames.

crates/polars-core/src/frame · high confidence

Enforce boolean type for filter predicates

A new TypeCheckRule optimization pass has been added to the query plan conversion layer. This rule validates that filter predicates (in both IR::Scan with predicates and IR::Filter nodes) are of type Boolean, ensuring that non-boolean filter conditions are rejected with a clear error message during query optimization rather than failing later.

_crates/polars-plan/src/plans/conversion/type\check · high confidence

Expression optimizer refactored into modular sub-modules with new simplification rules

The expression simplification optimizer has been restructured into dedicated modules (arity, functions, sort lowering) to improve maintainability and enable more targeted optimizations. This change introduces new logical simplifications, including the elimination of contradictory predicates in filters (e.g., \x \> N AND x \<= N\ becomes \false\), self-negation detection (\x AND NOT(x)\ becomes \false\), and De Morgan's law application for \NOT\ expressions. It also adds performance optimizations for sorting patterns, such as rewriting \sort().first()\ to \min\_by\ and limiting sort reads when only the head is needed. Additionally, the optimizer now simplifies \is\_null().any()\ to \null\_count() \> 0\ and handles \Float16\ literal arithmetic.

_crates/polars-plan/src/plans/optimizer/simplify\expr · high confidence

Internal crate reorganization and schema hash generation

The \polars-plan\ crate now includes a \build.rs\ script that generates a deterministic SHA-256 hash of the \dsl-schema-hashes.json\ file at compile time. This hash is embedded into the binary to ensure DSL compatibility checks remain invariant to Git EOL conversion differences (CRLF vs LF) on Windows. Additionally, the \clippy.toml\ file enforces the use of deterministic \PlIndexMap\ and \PlIndexSet\ over standard \HashMap\ and \HashSet\ to prevent non-deterministic behavior in the logical planning layer.

crates/polars-plan · high confidence

Introduce Column abstraction with lazy ScalarColumn representation

The core data structure for DataFrame columns has been refactored from a direct Series wrapper to a new \Column\ enum that can represent either a standard \Series\ or a \ScalarColumn\. A \ScalarColumn\ stores a single repeated value and its length, deferring the creation of the full materialized \Series\ until it is actually needed. This change enables significant performance improvements by avoiding unnecessary memory allocation and copying for columns that consist of a single repeated value, while also providing new arithmetic and comparison operators that natively support broadcasting between these column types.

crates/polars-core/src/frame/column · high confidence

Introduce generic fallback for unregistered extension types

Polars now handles Arrow extension types that are not explicitly registered by falling back to a generic extension type representation instead of failing or silently dropping the type. This change introduces a new extension type registry system in \polars-core\ that allows users to register custom extension type factories, while providing a \GenericExtensionType\ that preserves the type name and metadata for unknown types. The default behavior when encountering an unregistered extension type is to load it as this generic type, ensuring data integrity and preventing errors during operations like Parquet I/O with Iceberg V2 schemas that may contain custom extension types.

crates/polars-core/src/datatypes/extension · high confidence

Introduces amortized iterator for ListChunked to improve performance

The \ListChunked\ type now exposes an \amortized\_iter\ method that provides an iterator over list elements using a reusable series container. This optimization reduces memory allocations by reusing the underlying series structure during iteration, which benefits operations that process list elements individually, such as \apply\_amortized\_generic\ and \for\_each\_amortized\.

_crates/polars-core/src/chunked\array/list · high confidence

LazyFrame concatenation functions moved to DSL module

The implementation for concatenating LazyFrames (vertical, horizontal, and diagonal) has been relocated into the \crates/polars-lazy/src/dsl\ module. This change consolidates the DSL-specific logic for \concat\, \concat\_lf\_diagonal\, and \concat\_lf\_horizontal\ alongside other DSL functions, ensuring these multi-LazyFrame operations are properly exposed through the lazy API's domain-specific language layer.

crates/polars-lazy/src/dsl · high confidence

LazyFrame schema resolution is now cached for performance

The \LazyFrame\ implementation in \crates/polars-lazy/src/frame\ now caches the logical plan arena and expression arena after the first schema resolution. This optimization ensures that subsequent calls to \collect\_schema()\ or schema-dependent operations are significantly faster, as the plan does not need to be re-converted from the DSL to the IR for every access. The change introduces a \CachedArena\ structure and updates the \collect\_schema\ and \schema\_with\_arenas\ methods to store and retrieve these arenas, reducing overhead in iterative or schema-heavy workflows.

crates/polars-lazy/src/frame · high confidence

New IR-based optimizer pipeline with expanded optimization rules

The query optimizer has been refactored to operate on the new Intermediate Representation (IR) instead of the legacy logical plan. This change introduces a suite of new optimization rules including \collapse\_sort\ (to merge or prune redundant sorts), \collapse\_and\_project\ (to simplify projections and caches), \cluster\_with\_columns\ (to optimize \with\_columns\ chains), \extract\_window\ (to move window functions into dedicated IR nodes), \flatten\_merge\_sorted\ and \flatten\_union\ (to simplify nested structures), and \filter\_constraint\ (to simplify predicates). It also adds \expand\_datasets\ to handle dataset scan expansion and \call\_dsl\_resolvers\ for resolving DSL plans, alongside \collect\_members\ to track plan features and \deep\_copy\ for arena manipulation. These changes improve query performance and enable more advanced optimizations like join reordering and common subplan elimination.

crates/polars-plan/src/plans/optimizer · high confidence

New JSON parsing and schema inference implementation

The \polars-json\ crate now provides a dedicated implementation for JSON and NDJSON handling, replacing the previous logic. This new module introduces a custom schema inference engine (\infer\_schema.rs\) that determines Arrow data types from JSON values, including specific coercion rules for mixed types (e.g., Int64/Float64 to Float64, scalars/lists to lists). It also implements a new deserializer (\deserialize.rs\) that supports a wider range of data types, including Float16, Decimal, Int128, and Map types, while ensuring deterministic output for list-based structures. Additionally, an ordered tape converter (\ordered.rs\) is included to preserve key order in Map objects during parsing.

crates/polars-json/src/json · high confidence

New JSON schema inference and serialization logic in polars-io

The \crates/polars-io/src/json\ module has been restructured to include a new \infer.rs\ file that implements \json\_values\_to\_supertype\ and \dtypes\_to\_supertype\ for determining the common data type from JSON values or existing types. The \mod.rs\ file now explicitly defines the \JsonFormat\ enum (supporting \Json\ and \JsonLines\), the \JsonWriter\ struct for serializing DataFrames, and the \ensure\_json\_writable\ helper to validate that data types can be serialized to JSON. This change consolidates the core logic for reading schema inference and writing JSON output within the \polars-io\ crate.

crates/polars-io/src/json · high confidence

New Parquet read module with async cloud support and prefiltered parallelism

The parquet reading logic has been reorganized into a new \crates/polars-io/src/parquet/read\ module, introducing a \ParquetObjectStore\ for async, zero-copy footer reads from cloud object stores and a \ColumnStore\ abstraction for memory-mapped or prefetched column data. This change enables the \ParallelStrategy::Prefiltered\ mode by default for new streaming scans, which evaluates predicates in parallel to determine a row mask before reading, and adds support for reading Parquet files with hive columns stored inside the file itself.

crates/polars-io/src/parquet/read · high confidence

New adaptive HTTP rate-limiter and concurrency control for cloud scans

Cloud I/O now uses an adaptive AIMD rate-limiter and a bytes-based concurrency controller to manage HTTP requests to cloud storage. The rate-limiter dynamically adjusts request throughput based on observed success rates, with configurable init, floor, and ceiling rates via \CloudRateLimitConfig\. Concurrency is controlled by a new \ConcurrencyStrategy\ (BytesBased, Legacy, or Unbounded) and chunk sizes (\POLARS\_DOWNLOAD\_CHUNK\_SIZE\, \POLARS\_DOWNLOAD\_CHUNK\_SIZE\_RANDOM\_ACCESS\, \POLARS\_DOWNLOAD\_CHUNK\_SIZE\_STREAMING\), replacing the previous fixed concurrency model. This improves stability and performance when scanning large datasets from S3, Azure, GCS, or HTTP sources.

crates/polars-io/src/cloud · high confidence

New decimal arithmetic implementation with scale-aware operations

The decimal arithmetic logic in \crates/polars-core/src/chunked\_array/arithmetic\ has been restructured into a dedicated \decimal.rs\ module. This change introduces scale-aware operations for \DecimalChunked\, ensuring that addition, subtraction, multiplication, division, and remainder calculations correctly handle and preserve decimal scales. The implementation includes optimized paths for i64 values when they fit, rescaling single scalar operands once rather than per-row, and computing results without per-row validity checks where possible to improve performance. It also handles edge cases like division by zero and overflow with explicit error messages.

_crates/polars-core/src/chunked\array/arithmetic · high confidence

New dynamic MinMax kernel implementation with SIMD support

The min\_max module in polars-compute has been refactored to introduce a new \MinMaxKernel\ trait and a dynamic dispatch layer (\dyn\_array.rs\) that enables min/max operations on any array type at runtime. This change adds support for \Float16\ data types, which were previously missing from the dynamic dispatch, and implements SIMD-optimized kernels for primitive arrays when the \simd\ feature is enabled. Users benefit from consistent min/max behavior across all supported types, including proper handling of NaN values (both ignoring and propagating them) and correct results for boolean arrays with nulls, while gaining performance improvements on supported hardware through the new SIMD implementations.

_crates/polars-compute/src/min\max · high confidence

New file cache with per-file TTL and cloud support

The file cache in \polars-io\ has been rewritten to support caching both local and cloud files, including HTTP and cloud storage URIs. It introduces a configurable per-file Time-To-Live (TTL) mechanism, allowing cache entries to expire based on individual file settings rather than a global default. The new implementation includes robust concurrency controls to prevent deadlocks and race conditions, uses ETags for HTTP cache invalidation, and handles allocation errors gracefully when \fallocate()\ is not permitted.

_crates/polars-io/src/file\cache · high confidence

New internal IO module structure and utilities in polars-arrow

The \crates/polars-arrow/src/io\ module has been restructured to provide a cleaner separation of concerns for input/output operations. A new \iterator.rs\ file introduces a \BufStreamingIterator\ to efficiently stream items as byte slices using an internal buffer, reducing memory allocations. A \write\_owned.rs\ module adds a \WriteBytesOwned\ trait and an \IoWriteWrap\ adapter, allowing owned buffer writes to be used seamlessly with standard \std::io::Write\ interfaces. The module now explicitly exposes \ipc\ and \avro\ sub-modules (feature-gated) and includes a \README.md\ documenting design rules, such as separating data/metadata reading and IO/CPU-bound operations. This change refactors how Arrow data is streamed and written at a low level, improving performance and API clarity for consumers of the \polars-arrow\ crate.

crates/polars-arrow/src/io · high confidence

New internal representation for DataFrame functions and plan hints

The \polars-plan\ crate now introduces a dedicated module for internal function representations (\FunctionIR\) and DSL definitions (\DslFunction\), separating the logical plan's function nodes from their execution details. This change adds support for serializing and deserializing these nodes (including \OpaquePython\ functions and \Hint\ nodes like sortedness) via bincode and serde, ensuring that common subplan elimination (CSE) correctly identifies identical operations by hashing function pointers and schema metadata. Users benefit from more robust plan caching and serialization, particularly when using Python UDFs or when plan hints (such as sortedness) are preserved across serialization boundaries.

crates/polars-plan/src/plans/functions · high confidence

New join reordering optimizer for improved query performance

The query optimizer now automatically reorders chains of inner equi-joins to minimize intermediate result sizes. It identifies contiguous runs of inner equi-joins, estimates the row counts for different join orders using column statistics and key domains, and rebuilds the plan to process smaller relations first. This change applies specifically to the join-order optimization pass, handling cluster extraction, greedy left-deep ordering, and plan reconstruction while preserving the original output schema.

_crates/polars-plan/src/plans/optimizer/join\order · high confidence

New modular cloud writer implementation with configurable buffering

The cloud upload logic in \polars-io\ has been refactored into a new, modular \cloud\_writer\ module. This change introduces a \BytesBufferer\ that coalesces incoming data into chunks of a configurable target size before uploading, and an \InternalCloudWriter\ that manages multipart uploads with controlled concurrency and IO metrics. A \CloudWriterIoTraitWrap\ now provides \std::io::Write\ and \tokio::io::AsyncWrite\ compatibility for this new backend, while a \PlMultipartUpload\ wrapper standardizes error context for object store operations.

_crates/polars-io/src/cloud/cloud\writer · high confidence

New modular group-by aggregation implementation

The group-by aggregation logic in \polars-core\ has been restructured into a new modular directory (\aggregations/\) containing specialized implementations for different data types. This change introduces dedicated modules for list aggregation (\agg\_list.rs\), boolean operations (\boolean.rs\), categorical data (\categorical.rs\), string/binary data (\string.rs\), and a central dispatch layer (\dispatch.rs\). Users benefit from this refactoring through improved code organization and the inclusion of optimized paths for specific types, such as fast-explode optimizations for list aggregations and specialized min/max/arg\_min/arg\_max kernels for booleans, categoricals, and strings.

_crates/polars-core/src/frame/group\by/aggregations · high confidence

New optimized gather kernels for binary, boolean, and primitive arrays

The \polars-compute\ crate now includes specialized, unsafe \take\ (gather) implementations for Boolean, Binary, BinaryView, Primitive, FixedSizeList, List, and Struct arrays. These new kernels in \crates/polars-compute/src/gather\ replace generic fallbacks with optimized paths that handle validity (nulls) in values and indices separately, reducing allocation overhead and improving performance for data selection operations.

crates/polars-compute/src/gather · high confidence

New optimizer module for tracking and simplifying data ordering

A new \simplify\_ordering\ module has been added to the query plan optimizer. It introduces a graph-based analysis that tracks whether the output order of operations is observed by downstream consumers. If an operation's order is not required, the optimizer now sets \maintain\_order\ to false on scans, allowing them to read data out of order for better performance. The module also includes logic to simplify expression ordering during projections and handles caching nodes correctly within the traversal graph.

_crates/polars-plan/src/plans/optimizer/simplify\ordering · high confidence

New primitive array implementation in polars-arrow

The \crates/polars-arrow/src/array/primitive\ module has been replaced with a new implementation of \PrimitiveArray\ and its associated components. This change introduces a new \PrimitiveArrayBuilder\ for efficient construction, updates the FFI bindings to use \polars\_buffer::Buffer\ for value storage, and expands the debug formatting (\fmt.rs\) to support additional data types including Int128, UInt128, Float16, and various Decimal types. The module also includes new iterator implementations and property-based tests (\proptest.rs\) for the primitive array types.

crates/polars-arrow/src/array/primitive · high confidence

New scan predicate engine in polars-mem-engine

The scan predicate logic has been moved into the \polars-mem-engine\ crate, introducing a new \ScanPredicate\ structure that manages row filtering, batch skipping via statistics, and hive partition pruning. This change adds support for skipping record batches based on column min/max/null-count statistics, normalizes enum statistics to respect declaration order rather than lexical order, and introduces a \SkipFilesMask\ to efficiently track excluded or included files during multi-file scans.

_crates/polars-mem-engine/src/scan\predicate · high confidence

New specialized unique kernels for Boolean and Dictionary arrays

The \unique\ module in \polars-compute\ now includes dedicated \RangedUniqueKernel\ implementations for \BooleanArray\ and \DictionaryArray\. The new \BooleanUniqueKernelState\ tracks seen values (true, false, null) using a compact 3-bit integer, enabling highly efficient distinct counting and unique extraction for boolean data. Similarly, \DictionaryRangedUniqueState\ leverages the underlying primitive key range to compute unique dictionary values without full hashing. These changes provide optimized, low-overhead paths for boolean and categorical/dictionary data types within the unique computation pipeline.

crates/polars-compute/src/unique · high confidence

New streaming hot-group-by implementation with fixed-size hash tables

The streaming engine's group-by operation now uses a new 'hot groups' subsystem that maintains a fixed-size in-memory hash table for frequently seen keys, evicting less-accessed keys to bound memory usage. This change introduces a \HotGrouper\ trait and specific implementations for single-column types (including UInt128, Int128, Float16, and Categorical/Enum), multi-column keys (via row encoding and key rows), and binary/string views. Users benefit from improved memory efficiency and performance during streaming group-by operations, as the system can now handle datasets with many distinct keys without unbounded memory growth, while still correctly handling nulls and evictions.

_crates/polars-expr/src/hot\groups · high confidence

New streaming join index table implementations

The \idx\_table\ module in \crates/polars-expr\ has been replaced with a new set of specialized index table implementations (\SingleKeyIdxTable\, \KeyRowIdxTable\, \BinviewKeyIdxTable\, and \RowEncodedIdxTable\) designed to optimize the new-streaming join engine. These changes introduce single-key optimizations with prefetching, binary hash table views, and key-row layouts to improve join performance. The \new\_idx\_table\ factory function now selects the appropriate table type based on the key schema, supporting a wide range of data types including unsigned 128-bit integers (\UInt128\), signed 128-bit integers (\Int128\), and \Float16\, as well as Categorical/Enum types.

_crates/polars-expr/src/idx\table · high confidence

New total-order comparison kernels for all array types

The comparison logic in \crates/polars-compute/src/comparisons\ has been restructured into a new modular system that implements \TotalEqKernel\ and \TotalOrdKernel\ traits for every supported array type, including primitives, \BinaryView\/\Utf8View\, \FixedSizeList\, \List\, \Struct\, \Dictionary\, and \Boolean\ arrays. This change introduces consistent total-order semantics where nulls are treated as distinct values (equal only to themselves) and adds optimized SIMD-accelerated paths for primitive comparisons. Users benefit from more predictable equality and ordering behavior across complex nested types and improved performance for string and numeric comparisons.

crates/polars-compute/src/comparisons · high confidence

New vectorized hashing implementation for numeric and string types

The hashing subsystem in polars-core has been restructured to introduce a new vectorized hashing engine. This change adds a dedicated \identity.rs\ module for efficient integer key hashing and a comprehensive \vector\_hasher.rs\ module that implements the \VecHash\ trait for all numeric types (including Float16, UInt128, and Int128 when enabled) and String types. This new implementation replaces previous hashing logic with a more optimized, vector-aware approach that handles nulls and combines hashes more efficiently, directly impacting the performance of group-by, join, and unique operations.

crates/polars-core/src/hashing · high confidence

Optimized unique operations for sorted numeric arrays

The \unique\, \arg\_unique\, and \n\_unique\ operations for numeric chunked arrays now detect sorted data and use a linear scan to identify unique values, avoiding the overhead of sorting or hashing when the data is already ordered. This change improves performance for sorted inputs while maintaining correct handling of nulls and unsorted data.

_crates/polars-core/src/chunked\array/ops/unique · high confidence

Python API cleanup and new Iceberg/Delta scanning infrastructure

This release removes deprecated parameters from \DataFrame\, \LazyFrame\, \Series\, and \Expr\ methods, enforcing stricter API contracts for the upcoming 2.0 version. It introduces a new \py-polars/src/polars/\_utils/expired.py\ module to manage removed and renamed parameters with clear error messages. Additionally, it adds a comprehensive native scanning infrastructure for Iceberg and Delta tables, including new modules for Iceberg caching (\\_cache.py\), dataset handling (\\_dataset.py\), and sinks (\\_sink.py\), as well as Delta dataset support (\\_dataset.py\). A new external reader interface (\\_external\_reader.py\) is also introduced to support pluggable file readers.

polars · high confidence

Query optimizer flags migrated to a bitflags-based OptFlags struct

The query optimizer's configuration in \crates/polars-plan/src/frame\ now uses a \bitflags\-based \OptFlags\ struct instead of the previous implementation. This change consolidates optimization controls (such as projection pushdown, predicate pushdown, CSE, streaming, GPU execution, and join reordering) into a single, efficient bitmask, providing helper methods like \cluster\_with\_columns()\ and \join\_order()\ to check specific flags. Users benefit from a more robust and performant internal representation of optimizer settings, which supports finer-grained control and better integration with new optimization features like join reordering and GPU execution slots.

crates/polars-plan/src/frame · high confidence

Range functions now support broadcasting for array outputs

The \polars-expr\ crate now implements broadcasting logic for range functions (\int\_ranges\, \datetime\_ranges\, \date\_ranges\, \time\_ranges\, and \linear\_spaces\) when generating list or array columns. Previously, these functions likely required inputs to match in length; the new implementation in \crates/polars-expr/src/dispatch/range/utils.rs\ allows scalar values to be broadcast against columns (e.g., a single start date with a column of end dates), enabling more flexible generation of range arrays without requiring manual expansion of inputs.

crates/polars-expr/src/dispatch/range · high confidence

Re-implemented Object dtype storage and iteration

The internal storage for Polars Object types has been restructured to use a \Buffer\<T\>\ for values and a \Bitmap\ for validity, replacing the previous implementation. This change introduces new builder and iterator components that handle nulls and slicing more explicitly, ensuring that validity masks are correctly maintained during operations like slicing and appending. Users relying on Object series will benefit from more robust handling of null values and improved consistency in how object data is stored and accessed internally.

_crates/polars-core/src/chunked\array/object · high confidence

Refactor IPC serialization into modular type-specific handlers

The IPC write serialization logic in \crates/polars-arrow/src/io/ipc/write/serialize\ has been restructured from a monolithic implementation into separate modules for each data type (binary, binview, boolean, fixed\_size\_binary, fixed\_sized\list, list, map, primitive, struct, union). This change introduces dedicated \write\\*\ functions for each array type, improving code organization and maintainability while preserving the existing serialization behavior for Arrow IPC format output.

crates/polars-arrow/src/io/ipc/write/serialize · high confidence

Refactor Python IO plugin predicate handling to use PyO3 and FFI

The Python IO plugin infrastructure has been refactored to improve reliability and performance. Predicate filtering for Python-based data sources now uses PyO3 to construct PyArrow predicate strings directly, replacing the previous unsafe \eval\-based approach. Additionally, DataFrame conversion between Python and Rust now utilizes the FFI interface instead of \transmute\, ensuring safer memory handling. The module structure has also been reorganized to separate predicate logic, source options, and utility functions into dedicated files.

crates/polars-plan/src/plans/python · high confidence

Refactored Arrow export logic into dedicated module with improved categorical handling

The Series-to-Arrow export implementation has been reorganized from \into.rs\ into a new \arrow\_export\ module, introducing a \CategoricalToArrowConverter\ that manages categorical and enum mappings more robustly. This change ensures consistent index remapping across multiple exports and correctly handles the conversion of Polars Categorical and Enum types to Arrow Dictionary arrays, including support for Utf8View and LargeUtf8 value types. The refactoring also integrates support for nested types (List, Array, Struct, Map) within the conversion pipeline, ensuring that complex data structures are exported with correct metadata and field alignment.

_crates/polars-core/src/series/arrow\export · high confidence

Refactored AsOf join implementation in polars-ops

The AsOf join logic has been moved from polars-core to polars-ops and rewritten to use a generic, state-based approach. This new implementation in crates/polars-ops/src/frame/join/asof supports forward, backward, and nearest strategies for both numeric and binary (string) keys, including grouped joins. It introduces a \TotalOrd\ trait for consistent comparison handling (e.g., treating -0.0 == 0.0) and adds support for the \allow\_exact\_matches\ parameter to control equality behavior in nearest joins.

crates/polars-ops/src/frame/join/asof · high confidence

Refactored CSV writing engine with new serializer and formatting options

The CSV write implementation has been restructured to use a new \CsvSerializer\ module, introducing a generic, compile-time optimized serialization pipeline that handles quoting styles (Always, Never, Necessary, NonNumeric) and null values more efficiently. This change includes the addition of support for Float16 data types in CSV output, a new \float\_scientific\ option to control scientific notation formatting for floats, and the replacement of the \ryu\ library with \zmij\ for faster float-to-string conversion. Additionally, decimal comma support is now correctly applied to Float16 values, and the codebase has been updated to use \polars-arrow\ instead of \arrow-format\.

_crates/polars-io/src/csv/write/write\impl · high confidence

Refactored ChunkedArray implementation with new module structure and optimized operations

The ChunkedArray implementation in polars-core has been reorganized into a modular structure with dedicated files for operations like arg\_min\_max, bitwise logic, casting, and iteration. This refactoring introduces optimized algorithms for finding minimum/maximum indices (arg\_min/arg\_max) that leverage sorted flags and specialized kernels for better performance. Bitwise operations (AND, OR, XOR) on BooleanChunked now include safeguards against stack overflows with unit-length arrays and use dedicated kernels. The casting system has been restructured with explicit CastOptions (Strict, NonStrict, Overflowing) for more controlled type conversions. New parallel iterator implementations enable efficient concurrent processing of list chunks. The flags system now uses interior mutability with RelaxedCell for thread-safe statistics tracking. These changes improve performance, safety, and code maintainability while preserving the existing API surface.

_crates/polars-core/src/chunked\array · high confidence

Refactored NDJSON reader with new buffer and core modules

The NDJSON reading implementation in polars-io has been restructured into dedicated buffer and core modules. The new buffer module handles type-specific deserialization for numeric, boolean, string, and temporal types, while the core module manages multi-threaded parsing, schema inference, and projection/predicate pushdown. This refactoring improves code organization and maintainability of the NDJSON parsing logic.

crates/polars-io/src/ndjson · high confidence

Refactored Series arithmetic into a modular, fallible dispatch system

The arithmetic logic in \crates/polars-core/src/series/arithmetic\ has been reorganized into distinct modules (\bitops\, \borrowed\, \owned\, \list\, \fixed\_size\_list\) to support a new dispatch architecture. This change introduces fallible arithmetic operations (returning \PolarsResult\) for bitwise operators and numeric types, ensuring that invalid type combinations or shape mismatches raise informative errors instead of panicking. It also adds optimized owned arithmetic paths for primitive numeric types (including Int128 and Float16) that mutate data in-place when possible, and implements specific arithmetic dispatch for List and FixedSizeList (Array) types, enabling element-wise operations on nested structures with proper broadcasting and type coercion.

crates/polars-core/src/series/arithmetic · high confidence

Refactored and hardened Common Subexpression/Plan Elimination (CSE)

The CSE optimizer in polars-plan has been restructured into dedicated modules (cache\_states, csee, cspe) to improve correctness and performance. This change introduces stricter handling of nondeterministic expressions to prevent incorrect caching, ensures predicates and projections are correctly pushed past cache nodes without causing column-not-found errors, and adds cost-based evaluation to decide whether caching is beneficial. It also includes fixes for streaming engine compatibility, prevents the removal of nested caches during predicate pushdown, and optimizes hash-based equality checks for better performance.

crates/polars-plan/src/plans/optimizer/cse · high confidence

Refactored in-memory group-by internals for performance and stability

The in-memory group-by implementation in \polars-core\ has been restructured to improve performance and fix several stability issues. The hashing logic is now centralized in a new \hashing.rs\ module, utilizing \DirtyHash\ and partitioned parallel execution via Rayon to speed up key grouping. The \IntoGroupsType\ trait in \into\_groups.rs\ now supports a fast-path for sorted keys and explicitly handles Float16, Float32, and Float64 types, while falling back to bit-representation encoding for other numeric types to ensure correct hashing. Additionally, the \GroupsIdx\ structure in \position.rs\ has been optimized for cheap cloning and sliceability, with a parallelized flattening process for group indices and a background-thread drop strategy for large datasets to prevent UI/query blocking.

_crates/polars-core/src/frame/group\by · high confidence

Refactored internal data type definitions and serialization

The internal representation of Polars data types has been reorganized into a new \datatypes\ module, introducing a redesigned \DataType\ enum and a dedicated \SerializableDataType\ for serialization. This change updates how core types like Categorical, Enum, and Extension are stored and serialized, ensuring that metadata and category information are preserved correctly during round-trips. Users benefit from more robust type handling and improved consistency in how data types are defined and converted internally.

crates/polars-core/src/datatypes · high confidence

Refactored join predicate pushdown optimizer with new predicate pruning logic

The join predicate pushdown optimizer has been restructured into a new module (\crates/polars-plan/src/plans/optimizer/predicate\_pushdown/join\) containing a dedicated \predicate\_pruning.rs\ file. This change introduces refined logic for handling join conditions, specifically enabling the pushdown of filters for Left/Right outer joins when they commute with filtering, and adding support for joining Decimal columns with different scales by automatically casting keys to a common precision. The refactoring also includes tighter invariants for join options and improved handling of Hive partitioned data rewrites within the join optimization path.

_crates/polars-plan/src/plans/optimizer/predicate\pushdown/join · high confidence

Refactored temporal data types into dedicated modules

The temporal data types have been reorganized into a new \temporal\ module within \polars-core\, introducing dedicated \TimeUnit\ and \TimeZone\ structs. \TimeUnit\ now explicitly manages conversions between nanoseconds, microseconds, and milliseconds, while \TimeZone\ provides a wrapper for canonical timezone representation, including validation and Levenshtein-based suggestions for invalid timezone strings when the \timezones\ feature is enabled. This change centralizes temporal logic and improves type safety for time-based operations.

crates/polars-core/src/datatypes/temporal · high confidence

Reimplemented list builders with explicit fast\_explode tracking

The list builder implementations in \crates/polars-core/src/chunked\_array/builder/list\ have been rewritten to use new concrete builder types (\AnonymousOwnedListBuilder\, \ListStringChunkedBuilder\, \ListBinaryChunkedBuilder\, \ListBooleanChunkedBuilder\, \ListNullChunkedBuilder\, and \ListPrimitiveChunkedBuilder\) that explicitly track a \fast\_explode\ flag. This flag is automatically set to false when appending nulls or empty series, ensuring the resulting \ListChunked\ correctly reflects whether it can be safely exploded without checking for empty lists. The \get\_list\_builder\ factory function now dispatches to these specific builders based on the inner data type, replacing the previous implementation to improve correctness and performance for list operations.

_crates/polars-core/src/chunked\array/builder/list · high confidence

Reorganized temporal logic into a dedicated module with improved string formatting and parsing

The temporal handling code in polars-core has been restructured into a new \crates/polars-core/src/chunked\_array/temporal\ module, splitting logic into separate files for conversion, date, datetime, duration, time, and string parsing. This change introduces a \get\_strftime\_format\ helper that validates the 'polars' format specifier (raising an error for non-Duration types) and supports 'iso' and 'iso:strict' options for Datetime, Date, and Time types. It also adds a custom, faster \strptime\ parser in \string/strptime.rs\ to replace the standard library's chrono parsing for better performance, and refactors time-zone replacement logic in \replace\_time\_zone.rs\ to handle ambiguous and non-existent times more robustly.

_crates/polars-core/src/chunked\array/temporal · high confidence

Restructured DSL-to-IR conversion with predicate splitting and optimization

The DSL-to-IR conversion logic in \polars-plan\ has been reorganized into dedicated modules (\convert\_utils\, \dsl\_to\_ir\, \ir\_to\_dsl\, \stack\_opt\). This introduces a \ConversionOptimizer\ that applies simplification, type coercion, and type-checking rules during the conversion phase. A key behavioral change is the new \SplitPredicates\ utility, which breaks down complex ANDed filter conditions into separate \IR::Filter\ nodes to enable more effective predicate pushdown. Additionally, OR-factoring is now applied to lift shared conjuncts to the top level, improving optimizer performance for complex boolean expressions.

crates/polars-plan/src/plans/conversion · high confidence

Restructured comparison logic with optimized sorted scalar comparisons

The comparison implementation in \polars-core\ has been reorganized into a dedicated module (\chunked\_array/comparison\) with separate files for categorical, scalar, and general logic. A key behavioral improvement is that scalar comparisons (e.g., \column \> 5\) on sorted numeric columns now use a binary search algorithm to find the matching range, significantly speeding up these operations when the data is known to be sorted and contains no nulls.

_crates/polars-core/src/chunked\array/comparison · high confidence

Restructured type coercion into modular, domain-specific modules

The type coercion logic in the query planner has been refactored from a monolithic implementation into distinct, modular files (\binary.rs\, \datetime.rs\, \functions.rs\, \membership.rs\). This change organizes coercion rules by domain: binary operations (including struct and list arithmetic), temporal types (handling timezone conversions and date/datetime coercion), function argument validation, and membership operations (\is\_in\ and map lookups). For users, this ensures more precise type checking and error messages for specific operations like comparing categoricals to strings or checking membership in nested lists, while maintaining the same overall behavior for existing expressions.

_crates/polars-plan/src/plans/conversion/type\coercion · high confidence

Rewritten projection pushdown optimizer with new edge-based traversal

The projection pushdown optimizer in \crates/polars-plan\ has been completely refactored to use a new graph-traversal model. This change introduces an \Edge\ structure that explicitly tracks projection state and parent-child relationships during optimization, replacing the previous logic. For users, this results in more robust handling of complex query plans, specifically fixing several regressions where projection pushdown was ignored or caused panics (e.g., with \select(len())\, caches, or row indices). The new implementation also improves performance by ensuring projections are pushed down more effectively, reducing memory usage and execution time for wide frames and specific scan operations.

_crates/polars-plan/src/plans/optimizer/projection\pushdown · high confidence

Rolling aggregations now support nulls and use optimized linear-time algorithms

The rolling computation module in \polars-compute\ has been restructured to introduce dedicated kernels for rolling min/max, argmin/argmax, mean, sum, moments, and quantiles. A key behavioral change is that rolling operations (including \rolling\_mean\_by\, \rolling\_min\_by\, \rolling\_max\_by\, and \rolling\_rank\_by\) now correctly handle null values, whereas previous versions may have ignored or mishandled them. The implementation uses a monotonic deque for \arg\_min\/\arg\_max\ and \min\/\max\ operations to achieve O(n) linear-time complexity, and employs Kahan summation for rolling sums to improve numerical accuracy. Additionally, rolling rank operations now respect the 'current location' of items for correct tie-breaking, and quantile calculations respect window trimming and interpolation methods (like \midpoint\) more accurately.

crates/polars-compute/src/rolling · high confidence

Rust crate reorganization and new public API surface

The \polars\ Rust crate has been restructured to serve as a unified facade, exposing sub-crates like \polars-core\, \polars-lazy\, \polars-ops\, and \polars-sql\ through a consolidated \prelude\ and dedicated modules. This change introduces new public types such as \PlRefPath\ and \CloudScheme\ for path abstraction, adds \pf16\ support for Float16 data types, and exposes SQL context utilities directly from the main crate, simplifying imports and aligning the public API with the new modular architecture.

crates/polars/src · high confidence

Series construction and iteration logic refactored

The Series module in polars-core has been restructured to improve type safety and performance. A new \SeriesBuilder\ provides a unified, type-erased interface for constructing Series from various sources, while \AmortSeries\ introduces an amortized iteration pattern to reduce allocation overhead during repeated operations. The \from\_any\_values\ logic has been rewritten to handle strict and non-strict type inference more robustly, and comparison operations now explicitly validate categorical and enum category alignment before delegating to physical types.

crates/polars-core/src/series · high confidence

Unified sort implementation using row-encoding and new options structs

The sorting logic in \polars-core\ has been refactored to use a unified row-encoding approach for multi-column and complex sorts, introducing \SortOptions\ and \SortMultipleOptions\ structs to control descending order, null placement, multithreading, and order maintenance. This change adds efficient \arg\_bottom\_k\ and \arg\_sort\ implementations that leverage row encoding for better performance on nested types and categorical data, while also fixing bugs related to descending sorts with nulls and maintaining stable order.

_crates/polars-core/src/chunked\array/ops/sort · high confidence

Unsafe amortized iterators for ArrayChunked

The ArrayChunked type now exposes unsafe amortized iteration methods (amortized\_iter, apply\_amortized\_same\_type, etc.) that allow efficient, allocation-free traversal of fixed-size list arrays by reusing a pinned Series container. Users must handle these iterators carefully: they are unsafe because cloning the returned AmortSeries or holding it beyond the iterator's lifetime causes undefined behavior, and closures applied via these methods must preserve the input's dtype and element count.

_crates/polars-core/src/chunked\array/array · high confidence

Fixes

Fix direct I/O alignment detection for older glibc and musl builds

The direct I/O implementation now correctly discovers file alignment requirements on Linux systems with glibc versions older than 2.28 and on musl-based builds. Previously, the code relied on glibc wrappers that were unavailable or insufficient, causing build failures or incorrect alignment assumptions. The new code uses raw syscalls for \statx\ to get accurate alignment data on glibc \< 2.28 and falls back to reading \/sys/dev/block/.../queue/logical\_block\_size\ for musl and older kernels, ensuring direct I/O works reliably or safely falls back to buffered I/O when alignment cannot be determined.

_crates/polars-io/src/utils/byte\source · high confidence

Test coverage

Added FFI round-trip tests for Arrow arrays and streams; Added IPC I/O tests for Arrow record batches; Added comprehensive test suite for polars-lazy; Added core integration tests for Polars data operations; Added integration tests for CSV, JSON, IPC, and partitioned I/O operations; Added integration tests for Parquet Arrow read/write operations; Added integration tests for Parquet I/O roundtrips and statistics; Added integration tests for Polars LazyFrame operations; Added integration tests for joins, schema, and core functionality; Added integration tests for lazy expression evaluation; Added integration tests for temporal operations and date ranges; Added test for left join behavior after Parquet round-trip; Added tests for aggregate memory estimation; Added tests for bitmap utility functions; Added tests for list gather operations with nulls and empty values; Added tests for polars-arrow array implementations; Added tests for polars-arrow compute kernels; Added tests for the polars-buffer crate; Added unit tests for Parquet page and dictionary deserialization; Added unit tests for Parquet write operations and sidecar metadata; Added unit tests for polars-arrow bitmap operations.

Dependencies

1455 commits updating dependencies (60 manifests)

A dependency / build maintenance change in (dependencies) — 1455 commits (160 fixs), 60 files: “chore: Python dependency failure (\#24576)”; “fix: use larger recursive stack in debug mode (\#15593)”; “chore(python): Switch over some of the custom Python date/time conversions to native PyO3 conversions (\#16203)”; and 1451 more (refs \#24576, \#15593, \#16203, \#25284, \#16198, …).

(dependencies) · high confidence · unverified

Housekeeping

Internal crate documentation and build configuration updates

The \polars-core\, \polars-lazy\, and \polars-ops\ crates now include explicit README files clarifying that they are internal sub-crates not intended for external usage, alongside new LICENSE files and build scripts that conditionally enable the \nightly\ feature based on the Rust toolchain version.

crates/polars-core, crates/polars-lazy, crates/polars-ops · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 66 → 66 (+0.1)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 76 → 81 (+4.5)
  • Architecture 93 → 93 (+0.2)
  • Maturity 68 → 68 (-0.1)
  • Readiness 71 → 72 (+1.2)
  • Security 77 → 70 (-7.0)
  • Event Sourcing 100 (new)
  • Performance 60 → 60 (+0.0)

Resolved (390)

  • AExpr::to_field_impl (cognitive 56) (crates/polars-plan/src/plans/aexpr/schema.rs)
  • AExpr::to_field_impl (cyclomatic 65) (crates/polars-plan/src/plans/aexpr/schema.rs)
  • AdaptiveRateController::settle_lazy_tick (cyclomatic 18) (crates/polars-io/src/cloud/http_rate_limit.rs)
  • BinViewDecoder::extend_filtered_with_state (cognitive 16) (crates/polars-parquet/src/arrow/read/deserialize/binview/mod.rs)
  • BuildState::finalize_ordered (cognitive 18) (crates/polars-stream/src/nodes/joins/equi_join.rs)
  • BuildState::finalize_unordered (cognitive 20) (crates/polars-stream/src/nodes/joins/equi_join.rs)
  • Change coupling: apply_extra_ops.rs ↔ mod.rs (crates/polars-stream/src/nodes/io_sources/multi_scan/components/apply_extra_ops.rs)
  • Change coupling: asof_join.rs ↔ merge_join.rs (crates/polars-stream/src/nodes/joins/asof_join.rs)
  • Change coupling: context.rs ↔ keywords.rs (crates/polars-sql/src/context.rs)
  • Change coupling: cross_join.rs ↔ predicate_pruning.rs (crates/polars-ops/src/frame/join/cross_join.rs)
  • Change coupling: duration.rs ↔ month_start.rs (crates/polars-defs/src/time/duration.rs)
  • Change coupling: merge_join.rs ↔ mod.rs (crates/polars-stream/src/nodes/joins/merge_join.rs)
  • Change coupling: replace_time_zone.rs ↔ month_start.rs (crates/polars-core/src/chunked_array/temporal/replace_time_zone.rs)
  • Change coupling: single_keys_inner.rs ↔ single_keys_outer.rs (crates/polars-ops/src/frame/join/hash_join/single_keys_inner.rs)
  • Change-coupling hub: lower_ir.rs → sink.rs, row_deletions.rs, merge_join.rs, to_graph.rs (crates/polars-stream/src/physical_plan/lower_ir.rs)
  • Change-coupling hub: strings.rs → mod.rs, namespace.rs, strings.rs, string.rs (crates/polars-plan/src/dsl/function_expr/strings.rs)
  • ChunkedArray::agg_std (cognitive 17) (crates/polars-core/src/frame/group_by/aggregations/mod.rs)
  • ChunkedArray::agg_var (cognitive 17) (crates/polars-core/src/frame/group_by/aggregations/mod.rs)
  • ClassTooLong: ListNameSpaceImpl (crates/polars-ops/src/chunked_array/list/namespace.rs)
  • Duplicated block (10 lines × 2) (crates/polars-compute/src/rolling/min_by_max_by.rs)
  • …and 370 more

New (1216)

  • AExpr::to_field_impl_inner (cognitive 56) (crates/polars-plan/src/plans/aexpr/schema.rs)
  • AExpr::to_field_impl_inner (cyclomatic 65) (crates/polars-plan/src/plans/aexpr/schema.rs)
  • Boundary-crossing change coupling: csv.rs ↔ _options.py (crates/polars-lazy/src/scan/csv.rs)
  • Boundary-crossing change coupling: expand_datasets.rs ↔ _options.py (crates/polars-plan/src/plans/optimizer/expand_datasets.rs)
  • Boundary-crossing change coupling: functions.rs ↔ _options.py (crates/polars-mem-engine/src/scan_predicate/functions.rs)
  • Boundary-crossing change coupling: ndjson.rs ↔ _options.py (crates/polars-lazy/src/scan/ndjson.rs)
  • Boundary-crossing change coupling: parquet.rs ↔ _options.py (crates/polars-lazy/src/scan/parquet.rs)
  • Change coupling: _dataset.py ↔ _options.py (py-polars/src/polars/io/iceberg/_dataset.py)
  • Change coupling: lib.rs ↔ lib.rs (crates/polars-config/src/lib.rs)
  • Change coupling: lib.rs ↔ sql.rs (crates/polars-sql/src/lib.rs)
  • Change coupling: log.rs ↔ schema.rs (crates/polars-ops/src/series/ops/log.rs)
  • Change coupling: mod.rs ↔ init.rs (crates/polars-stream/src/nodes/io_sources/ipc/mod.rs)
  • Change-coupling hub: lower_ir.rs → sink.rs, row_deletions.rs, to_graph.rs (crates/polars-stream/src/physical_plan/lower_ir.rs)
  • Change-coupling hub: strings.rs → mod.rs, namespace.rs, string.rs (crates/polars-plan/src/dsl/function_expr/strings.rs)
  • ClassTooLong: AggregationContext (crates/polars-expr/src/expressions/mod.rs)
  • ClassTooLong: ApplyExpr (crates/polars-expr/src/expressions/apply.rs)
  • ClassTooLong: Builder (crates/polars-io/src/csv/read/builder.rs)
  • ClassTooLong: IpcFileReader (crates/polars-stream/src/nodes/io_sources/ipc/mod.rs)
  • ClassTooLong: SelectionWalker (crates/polars-plan/src/plans/optimizer/projection_pushdown/field_pruning.rs)
  • DataFrame.write_delta (cyclomatic 17) (py-polars/src/polars/dataframe/frame.py)
  • …and 1196 more

Changes since last survey

  • 272 commits — 163 feature/other, 109 fixes

By area

  • crates/polars-plan — 46 commits
  • crates/polars-sql — 43 commits
  • crates/polars-stream — 26 commits
  • crates/polars-core — 23 commits
  • py-polars/tests — 20 commits
  • py-polars/src — 19 commits
  • crates/polars-expr — 17 commits
  • crates/polars-io — 12 commits
  • crates/polars-ops — 11 commits
  • crates/polars-arrow — 9 commits
  • crates/polars-compute — 9 commits
  • crates/polars-parquet — 5 commits
  • crates/polars-python — 5 commits
  • (root) — 4 commits
  • crates/polars-cli — 4 commits
  • crates/polars-config — 3 commits
  • docs/source — 3 commits
  • crates/polars — 2 commits
  • crates/polars-time — 2 commits
  • crates/polars-utils — 2 commits

Notable commits

  • fix: chore(rust): Fix cargo deny advisories (ctutils, tokio, crypto-bigint) (#30285)
  • fix: chore(rust): Fix clippy-stable for Rust 1.100 (#30762)
  • fix: chore: Disallow trivial first-time contributor fixes (#30549)
  • fix: chore: Fix OOC sort decay test (#30345)
  • fix: chore: Fix flaky encryption test (#30437)
  • fix: fix(python): Accept column name for list.gather indices (#30375)
  • fix: fix(python): Correct scalar // Series and DataFrame floordiv (#30452)
  • fix: fix(python): Prevent to_pandas() deadlock with map_elements on another thread (#30361)
  • fix: fix(python): Support numpy 0.28 and builds without the cloud feature (#30722)
  • fix: fix(python): Sync prepare_cloud_plan return type (#30733)
  • fix: fix(rust): Don't silently drop extra struct fields under a strict Expr::cast (#29231)
  • fix: fix: Add UInt128 to Product type mapper (#30555)
  • fix: fix: Allow TOP as a SQL table alias or column name (#30476)
  • fix: fix: Allow expressions on the right input in outer join_where predicates (#30368)
  • fix: fix: Allow repeated indices in a group in debug aggregation checks (#30771)
  • fix: fix: Apply Delta deletion vectors to files in escaped partition paths (#30454)
  • fix: fix: Apply ORDER BY to a SQL VALUES list (#30468)
  • fix: fix: Apply calendar parts of offset_by in one step (#30744)
  • fix: fix: Arrow C stream Decimal32/64/256 import (#30435)
  • fix: fix: Asof join with Categorical key column (#30554)
  • …and 252 more

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

pola-rs/polars was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 5c36cdd75a30a8ab12a9f70e251c0f267235b2a7 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.