elixir-explorer/explorer

This system is a high-performance data manipulation library for Elixir that leverages the Polars Rust backend to handle DataFrames and Series. It supports declarative, composable queries with lazy evaluation, enabling deferred execution for efficient processing of large datasets. The library provides robust I/O capabilities for various formats including CSV, Parquet, and IPC, with native support for cloud storage like AWS S3. It also includes comprehensive data type handling, mathematical transformations, and integration with Jupyter notebooks via Kino.

12 changes.

This period focused on a major architectural shift to support lazy evaluation by integrating the Polars LazyFrame API into the native Rust backend and Elixir layer. The work introduced a new declarative query syntax, modernized I/O and backend APIs, and stabilized the project with comprehensive testing and dependency updates for version 0.12.0.

6 changes.

This period focused on expanding the Polars backend by adding native LazyFrame support for Parquet and IPC I/O, including cloud storage capabilities. The work also involved consolidating DataFrame IO operations into a dedicated module and significantly increasing test coverage for data frame operations, backend conversions, and inspection outputs.

4 changes.

This period focused on expanding the Explorer library's Series capabilities by adding support for complex data types such as decimals, structs, and categorical data, alongside implementing logarithmic and exponential mathematical functions. Significant effort was also dedicated to strengthening test coverage through property-based testing infrastructure and local S3 simulation to ensure robust handling of these new features.

Users can now apply logarithmic and exponential transformations to Series data. This change introduces three new NIFs: \s\_log\ for natural logarithm, \s\_log\_natural\ (exposed as \log/1\), \s\_log\ with a base argument (exposed as \log/2\), and \s\_exp\ for the exponential function. The implementation leverages Polars' underlying \log\ and \exp\ methods, handling base arguments including special float values like NaN and Infinity.

native/explorer/src/series · high confidence

The Explorer library now supports a broader range of data types, including null values, decimal numbers with configurable precision and scale, and struct types with nested fields. This update also introduces support for categorical data, time zones for datetime types, and various integer and float sizes, enabling more precise data handling and complex nested structures in datasets.

native/explorer/src/datatypes · high confidence

Explorer now includes a new \Explorer.Query\ module that allows users to write declarative, composable data manipulation expressions. This feature enables the use of Elixir's for-comprehensions and the \across/1\ helper to apply operations across multiple columns concisely, and supports conditional logic via \if/2\, \unless/2\, and \cond/1\ within queries. This new query syntax is integrated into core DataFrame verbs like \filter/2\, \mutate/2\, \summarise/2\, and \sort\_by/2\, providing a more powerful and readable way to transform data compared to previous callback-based approaches.

lib/explorer · high confidence

This change adds the \Explorer.Backend.LazySeries\ and \Explorer.Backend.QueryFrame\ modules, enabling deferred execution for Series and DataFrame operations. The \LazySeries\ struct represents operations and their arguments without immediate computation, supporting a wide range of element-wise, window, aggregation, and string functions. The \QueryFrame\ acts as a limited backend for use within query macros like \filter\_with\ and \mutate\_with\. Additionally, the \Explorer.Backend.Series\ and \Explorer.Backend.DataFrame\ behaviors are updated to support lazy types, new IO options (such as \nil\_values\, \eol\delimiter\, and streaming), and renamed callbacks (e.g., \filter\ to \mask\, \take\ to \slice\, \cum\\\ to \cumulative\\*\).

lib/explorer/backend · high confidence

The LazyFrame module now includes native functions for reading and writing Parquet and IPC files, enabling lazy evaluation for these formats. Users can scan Parquet and IPC files into a LazyFrame (\lf\_from\_parquet\, \lf\_from\_ipc\) and write LazyFrames to disk in Parquet or IPC format (\lf\_to\_parquet\, \lf\_to\_ipc\) with options for compression and streaming execution. Additionally, when compiled with the AWS feature, cloud-based Parquet I/O is supported via \lf\_from\_parquet\_cloud\ and \lf\_to\_parquet\_cloud\, allowing direct interaction with S3 storage.

native/explorer/src/lazyframe · high confidence

Users can now access the classic Iris and Wine datasets directly from the public \datasets\ directory. The \iris.csv\ file contains sepal and petal measurements for three Iris species, while \wine.csv\ provides chemical analysis data for three wine cultivars. These additions expand the available sample data for exploration and testing without requiring private or internal paths.

datasets · high confidence

This release updates the \Decimal\ dependency to version \~\> 3.1 to mitigate [CVE redacted] (a Denial of Service vulnerability) and updates the \Rustler\ dependency to version \~\> 0.38 to address multiple vulnerabilities (EEF-[CVE redacted], EEF-[CVE redacted], EEF-[CVE redacted], EEF-[CVE redacted], EEF-[CVE redacted], EEF-[CVE redacted]) related to \Req\ and \Mint\. These changes are documented in the new \CHANGELOG.md\ file.

(repo-wide) · high confidence

The native Rust backend has been significantly restructured to leverage the Polars LazyFrame API and a new expression system. This introduces new resource types (ExExpr, ExLazyFrame) and dedicated modules for expressions, lazy frames, and encoding, while removing the previous eager-only DataFrame implementation. The change also adds a CloudWriter to support synchronous writing to cloud object stores and updates the error handling and data type definitions to align with the new architecture.

native/explorer/src · high confidence

The IO functions for DataFrames (CSV, Parquet, NDJSON, Arrow) have been moved from scattered locations into a single, dedicated \io.rs\ module. This change introduces a consistent pattern of \dump\, \load\, \from\, and \to\ functions for reading and writing data, making the codebase easier to navigate and maintain. Users benefit from a more organized internal structure without any change to the external API or behavior.

native/explorer/src/dataframe · high confidence

The configuration file now enables the \check\_polars\_frames\ setting for the Explorer application and registers the SQLite driver for ADBC support. This means Polars frame validation is active by default, and the system is pre-configured to use ADBC with SQLite out of the box.

config · high confidence

The Polars backend has been significantly restructured to introduce a dedicated LazyFrame implementation, enabling lazy evaluation for DataFrame operations and allowing users to defer computation until explicitly collected. This change includes a comprehensive API modernization: CSV and other IO functions now accept normalized keyword-list options (such as \nil\_values\, \skip\_rows\_after\_header\, and \quote\_delimiter\) instead of positional arguments, and the backend now supports reading from and writing to AWS S3 and HTTP sources via the FSS abstraction. Additionally, the \Native\ module has been updated to use \RustlerPrecompiled\ for precompiled NIFs with CPU feature detection, and many series operations have been refactored to use a unified \Shared.apply\_series\ pattern for better consistency and error handling.

_lib/explorer/polars\backend · high confidence

The Explorer module's documentation is now dynamically generated from the README file instead of being hardcoded in the module. Additionally, the local API for setting and getting the default backend via process dictionary (Explorer.default\_backend/1 and /0) has been removed from this file, indicating that backend configuration logic has been moved or refactored elsewhere.

lib · high confidence

The 'Ten Minutes to Explorer' introductory notebook has been updated to target Explorer version 0.12.0 and Kino 0.19.0. The content now reflects the current API, including the use of \Series.size/1\ instead of the deprecated \Series.length/1\, the \{:f, size}\ notation for floating-point data types, and the \nil\_values\ option for CSV reading. It also demonstrates loading the built-in \fossil\_fuels\ dataset and showcases the updated dataframe printing format.

notebooks · high confidence

The Cargo configuration for the native Explorer component has been updated to improve build stability on specific Linux platforms. The previous configuration targeting Apple Darwin (macOS) has been removed, while new settings have been added for x86\_64 and aarch64 Linux musl targets to disable static CRT linking. Additionally, libatomic is now explicitly linked for 32-bit ARM Linux builds to resolve dependency issues, and thin Link-Time Optimization (LTO) is enabled for release builds to reduce binary size.

native/explorer/.cargo · high confidence

Added test coverage for Series datetime, duration, list, struct, and inferred dtype behaviors; Added test support infrastructure for property-based testing and S3 integration; Added tests for Polars backend expression conversion; Added tests for lazy series and query frame inspection; Expanded test coverage for DataFrame I/O, grouping, and lazy operations; Expanded test coverage for Explorer core components; Test suite configuration updates.

Explorer has been updated to version 0.12.0-dev, raising the minimum Elixir requirement to 1.15 and introducing a comprehensive dependency overhaul. The core Rust backend now uses Polars 0.52, replacing the previous git-based dependency with a stable hex crate, and upgrades Rustler to 0.38 (with precompiled binaries via rustler\_precompiled 0.7). Key Elixir dependencies have been updated, including Nx (now supporting 1.0), TableRex (4.1), and ExDoc (0.40). The package also adds optional support for ADBC (0.12) and introduces new build aliases for Rust linting and formatting.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 60 → 70 (+10.3)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 96 → 95 (-0.9)
• Architecture 100 → 100 (+0.0)
• Maturity 58 → 58 (+0.0)
• Readiness 62 → 73 (+10.9)
• Security 48 → 76 (+27.5)
• Domain Modelling 100 → 100 (+0.0)

• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — no supported dependency manifest was read
• Duplicated block (10 lines × 2) (lib/explorer/polars_backend/lazy_frame.ex)
• Duplicated block (14 lines × 2) (lib/explorer/data_frame.ex)
• Duplicated block (16 lines × 2) (lib/explorer/data_frame.ex)
• Duplicated block (21 lines × 2) (lib/explorer/polars_backend/data_frame.ex)
• Duplicated block (6 lines × 2) (lib/explorer/fss.ex)
• Duplicated block (7 lines × 2) (lib/explorer/data_frame.ex)
• FileTooLong: explorer/query.ex (lib/explorer/query.ex)
• FileTooLong: explorer/shared.ex (lib/explorer/shared.ex)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (native/explorer/Cargo.lock)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (mix.lock)
• High vulnerability: [GHSA redacted] (native/explorer/Cargo.lock)
• High: security finding (details withheld)
• High: security finding (details withheld)
• High: security finding (details withheld)
• …and 32 more

• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (11 lines × 2) (lib/explorer/polars_backend/lazy_frame.ex)
• Duplicated block (16 lines × 2) (lib/explorer/data_frame.ex)
• Duplicated block (16 lines × 2) (lib/explorer/data_frame.ex)
• Duplicated block (21 lines × 2) (lib/explorer/polars_backend/data_frame.ex)
• Duplicated block (5 lines × 2) (lib/explorer/data_frame.ex)
• Duplicated block (5 lines × 2) (lib/explorer/fss.ex)
• Duplicated block (8–13 lines × 2) (lib/explorer/data_frame.ex)
• FileTooLong: dataframe/io.rs (native/explorer/src/dataframe/io.rs)
• FileTooLong: src/datatypes.rs (native/explorer/src/datatypes.rs)
• FileTooLong: src/encoding.rs (native/explorer/src/encoding.rs)
• FileTooLong: src/expressions.rs (native/explorer/src/expressions.rs)
• FileTooLong: src/series.rs (native/explorer/src/series.rs)
• FixmeComment (native/explorer/src/lazyframe.rs)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (native/explorer/Cargo.lock)
• High CVE: [GHSA redacted] (mix.lock)
• High CVE: [GHSA redacted] (native/explorer/Cargo.lock)
• …and 57 more

• 2 commits — 2 feature/other, 0 fixes

• (root) — 1 commit
• lib/explorer — 1 commit

• change: Added support rle_id (#1160)
• change: Allow Nx 1.0 (#1161)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

elixir-explorer/explorer was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.