medialab/xan

Xan is a high-performance, SIMD-accelerated command-line tool for CSV data manipulation and analysis, written in Rust. It provides a comprehensive suite of data processing commands alongside a powerful expression engine, Moonblade, which supports complex aggregations, windowing functions, and HTML scraping. The system also includes specialized infrastructure for handling graph data structures and Parquet files, enabling efficient pipeline processing and statistical computation.

5 changes.

This period marks the initial release of the xan CLI tool, featuring a major rebrand from xcsv and a comprehensive dependency overhaul to support Rust 2024. The work established foundational documentation and citation infrastructure while introducing core capabilities for graph analysis, Parquet data support, and pipeline processing. Extensive test coverage was added for core CSV manipulation commands to ensure stability alongside these new features.

8 changes.

This period focused on a comprehensive rewrite of the Moonblade expression engine, introducing a Pest-based parser and a modular aggregation architecture with advanced statistical capabilities. The work also involved refactoring the type system for better temporal support and optimizing core collection primitives, alongside adding automation scripts to streamline development workflows.

This commit introduces the initial version of the \xan\ CLI tool (renamed from \xcsv\/\xsv\), providing a high-performance, SIMD-accelerated CSV data manipulation command line interface written in Rust. It establishes the project's foundational documentation, including a comprehensive README with installation instructions for various package managers (Cargo, Homebrew, Nix, etc.), a CHANGELOG detailing features and breaking changes from version 0.62.0 onwards, and an AUTHORS file. Additionally, it adds \.zenodo.json\ to enable formal citation of the software via Zenodo, listing the project's creators and affiliations.

(repo-wide) · high confidence

The moonblade aggregation engine now supports a wider range of statistical computations, including geometric mean, harmonic mean, and root-mean-square (RMS) via Welford's algorithm, as well as approximate cardinality and quantiles using HyperLogLogPlus and TDigest. Users can also access new aggregators for all/any boolean checks, first/last values, and numeric extents with argmin/argmax tracking. The system introduces sparkline generation for number distributions and temporal granularity inference, alongside improved precision for sums using the Kahan-Babuska algorithm.

src/moonblade/agg/aggregators · high confidence

Three new shell scripts have been added to the repository to streamline development workflows. The \docs.sh\ script automates the generation of command-line help documentation and Moonblade reference materials by invoking the built binary. The \release.sh\ script simplifies the publishing process by handling version retrieval, cargo publishing, and git tagging in a single step. Additionally, \test\_segmentation.sh\ provides a functional test suite for file segmentation logic, iterating through various chunk configurations to verify data integrity.

scripts · high confidence

The application now includes core infrastructure for graph analysis, Parquet data support, and pipeline processing. A new \graph.rs\ module implements graph data structures, degree mapping, and GEXF serialization (supporting both 1.2 and 1.3 namespaces). Parquet file reading is enabled via a new \parquet.rs\ module that handles schema projection and type mapping. Additionally, a \processing.rs\ module introduces a \Children\ abstraction for managing child processes in pipelines, including a \check\ method for error handling, and a \ratatui.rs\ module provides utilities for rendering TUI frames to stdout.

src · high confidence

The \src/collections\ module now provides several new data structures to improve performance and functionality: \ClusteredInsertHashmap\ for faster lookups on clustered keys, \ContextBuffer\ for managing before/after context windows, \TopKHeap\ for efficient top-k selection, and \IncrementalId\ for generating unique IDs. Additionally, the module's default \HashMap\ and \HashSet\ types have been switched to use the \ahash\ crate (via \AHashMap\/\AHashSet\) for improved hashing performance.

src/collections · high confidence

The moonblade function library has been reorganized from a single monolithic file into distinct submodules (fmt, fuzzy, io, maps, ops, sequences, strings, temporal, utils, web). This structural change improves code maintainability and allows for clearer separation of concerns across formatting, file I/O, data manipulation, and temporal operations, while preserving the existing public API surface for users.

src/moonblade/functions · high confidence

The \src/moonblade/agg\ module has been restructured into a new, modular architecture that separates aggregation logic into distinct components: \program.rs\ for defining aggregation programs (including group-by, pivot, and along-columns variants), \stats.rs\ for descriptive statistics (mean, variance, quantiles, frequencies), and \window.rs\ for window functions (rolling sums/averages, cumulative min/max, ranking, lead/lag). This change introduces a unified \Aggregator\ enum and \AggregationProgram\ interface, enabling more maintainable and extensible aggregation pipelines while supporting parallel execution and memory-optimized layouts for existing and new functions like covariance, correlation, and geometric/harmonic means.

src/moonblade/agg · high confidence

A change to existing behaviour in src/cmd — 1674 commits (418 fixs), 67 files.

src/cmd · medium confidence · unverified

The Moonblade expression engine has been completely rewritten, replacing the previous implementation with a new Pest-based parser (grammar.pest) and a fresh interpreter architecture. This change introduces new program types including \ChooseProgram\ (for handling current/new record comparisons), \SelectionProgram\ (for named and plural column selections), and \ScrapingProgram\ (for HTML scraping with CSS selectors and text extraction). The parser now supports a richer syntax including list and map literals, lambda expressions, named arguments, pipeline operators, and improved string/regex handling. Error reporting has been restructured with dedicated \ConcretizationError\ and \EvaluationError\ types for clearer diagnostics.

src/moonblade · high confidence

The moonblade types module has been restructured to introduce a new \BoundArgument\ system for more efficient argument passing and a \FunctionArguments\ API that supports named arguments, variadic parameters, and arity validation. \DynamicValue\ now includes dedicated temporal variants (\Zoned\, \DateTime\, \Date\, \Time\, \Span\) powered by the \jiff\ library, alongside a new \DynamicNumber\ type for numeric operations. The \HeadersIndex\ has been updated to support negative indexing and duplicate header names, while \DynamicValue\ serialization and deserialization have been modernized to use \TryFrom\ and improved visitor patterns.

src/moonblade/types · high confidence

Added test resources for text scraping and HTML parsing; Expanded test coverage for core CSV manipulation commands.

The project has been renamed from xcsv to xan, updated to Rust edition 2024 with a minimum supported Rust version of 1.88.0, and its dependency tree has been significantly expanded and modernized. Key updates include upgrading simd-csv to 0.14.0, paltoquet to 0.12.0, and calamine to 0.36.1, while adding support for new formats and features via crates like bgzip, npyz, and parquet. The Cargo.lock file has been regenerated to reflect these changes, introducing a comprehensive list of new dependencies such as ahash, bstr, and fast-float2, and removing older git-based dependencies in favor of versioned crates.io packages.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 69 → 59 (-10.0)
• Rubric changed (rubric-2026.09.11 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 73 → 73 (-0.1)
• Architecture 100 → 89 (-11.1)
• Maturity 75 → 75 (+0.0)
• Readiness 74 → 45 (-29.8)
• Security 61 → 69 (+8.6)

• Args::convert_parquet (cyclomatic 24) (src/cmd/from.rs)
• Documentation: no contributor guidance (README.md)
• Documentation: no installation or build instructions
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (14 lines × 2) (src/select.rs)
• Duplicated block (9 lines × 2) (src/moonblade/functions/temporal.rs)
• Hotspot: src/cmd/cat.rs (src/cmd/cat.rs)
• Hotspot: src/cmd/from.rs (src/cmd/from.rs)
• Hotspot: src/cmd/spark.rs (src/cmd/spark.rs)
• Medium CVE: [GHSA redacted] (Cargo.lock)
• Medium advisory (unmaintained): RUSTSEC-2025-0057 (Cargo.lock)
• Medium advisory (unsound): RUSTSEC-2026-0002 (Cargo.lock)
• Medium advisory (unsound): RUSTSEC-2026-0097 (Cargo.lock)
• Medium advisory (unsound): RUSTSEC-2026-0253 (Cargo.lock)
• Medium vulnerability: RUSTSEC-2026-0194 (Cargo.lock)
• Medium vulnerability: RUSTSEC-2026-0194 (Cargo.lock)
• Members sharing a duplicated core (4 members, 50+ identical tokens) (src/cmd/parallel.rs)
• Near-duplicate member pair (42 shared lines) (src/moonblade/functions/temporal.rs)

• Args::bivar (cognitive 17) (src/cmd/matrix.rs)
• Args::convert_ndjson (cognitive 28) (src/cmd/from.rs)
• Documentation: no project overview (docs/cmd/complete.md)
• Documentation: no project overview (docs/cmd/count.md)
• Duplicated block (13 lines × 2) (src/select.rs)
• Duplicated block (13 lines × 2) (src/select.rs)
• Duplicated block (15 lines × 2) (src/cmd/matrix.rs)
• Duplicated block (8 lines × 2) (src/moonblade/functions/temporal.rs)
• End-of-life runtime: Rust 1.88
• FileTooLong: src/json.rs (src/json.rs)
• Flaky test: xan::tests.test_select::select_all::headers
• Flaky test: xan::tests.test_select::select_not::headers
• Flaky test: xan::tests.test_select::select_range::headers
• Flaky test: xan::tests.test_select::select_range_multi_idx::headers
• Flaky test: xan::tests.test_select::select_range_no_start::headers
• Flaky test: xan::tests.test_select::select_range_no_start::no_headers
• Flaky test: xan::tests.test_sort::sort_check_reverse
• Flaky test: xan::tests.test_sort::sort_count_empty
• Flaky test: xan::tests.test_sort::sort_count_one_group
• Hotspot: src/cmd/headers.rs (src/cmd/headers.rs)
• …and 12 more

• 26 commits — 21 feature/other, 5 fixes

• (root) — 12 commits
• src/cmd — 8 commits
• docs/cmd — 2 commits
• src/moonblade — 2 commits
• (repo) — 1 commit
• src/config.rs — 1 commit

• fix: Adding chars & graphemes moonblade functions Fix #1170
• fix: Adding the ord & chr moonblade functions Fix #1171
• fix: Clarifying filter example in quick tour Fix #1157
• fix: numfmt-related utils to avoid allocating needlessly Fix #1047
• fix: xan from --model now renames columns Fix #1167
• change: Adding easy way to pass null-byte to -d
• change: Adding xan from -f=parquet -s/--select
• change: Adding xan matrix bivar
• change: Better xan headers parquet integration wtf parquet.
• change: Clarifying xan from help wrt compressed inputs
• change: Clippy pass
• change: Config::is_parquet
• change: Documenting parquet integration in README
• change: Drafting parquet utils
• change: Dropping parquet feature Parquet integration is now the default
• change: Merge branch 'master' into parallel-ndjson
• change: Modernizing code as per clippy recommendations
• change: Optimizing xan from -p
• change: Parallel xan from -f ndjson Related to #1156
• change: Refactoring ParquetReader for later
• …and 6 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

medialab/xan was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.