apache/arrow-erlang

Apache Arrow Erlang is an Erlang library that provides native bindings for serializing and deserializing Apache Arrow data structures via the IPC format. It leverages a Rust NIF to handle low-level binary encoding and decoding of schemas, record batches, and file footers using Flatbuffers. The system supports core Arrow types, including nested lists and validity bitmaps, and includes comprehensive tooling for building, testing, and documenting the integration.

A new 'Quick run-through' guide has been added to the documentation, providing a step-by-step tutorial on serializing Erlang data into the Arrow IPC format. The guide walks users through creating schemas, defining Arrow arrays for various data types (including nested lists), constructing record batches, and finally combining these into IPC streams or files.

guides · high confidence

The native Rust NIF module now includes the initial codebase for serializing Arrow IPC messages and file footers into Flatbuffers. This change introduces the core data models for Messages, Schemas, RecordBatches, and File Footers, along with the serialization logic required to convert these structures into the binary format used by the Arrow IPC specification.

_native/arrow\_format\nif · high confidence

This change introduces the initial version (0.1.0) of the \arrow\ OTP library, providing core Erlang implementations for Apache Arrow data structures. It adds modules for handling Arrow Arrays (including fixed-primitive and fixed-list layouts), Buffers, and Validity Bitmaps, alongside records and serialization logic for the Arrow IPC format (Schemas, RecordBatches, Messages, and File Footers). The library also includes a NIF stub (\arrow\_format\_nif\) for Flatbuffers integration and sets up the project with Erlang formatting via \erlfmt\ and Apache 2.0 licensing.

src · high confidence

The project has been renamed from 'serde\_arrow' to 'arrow-erlang' and adopted Apache Software Foundation governance, including the addition of .asf.yaml for GitHub integration, a Code of Conduct, CONTRIBUTING guidelines, and updated license/notice files. The build system was upgraded to use rebar3 with the rebar3\_cargo plugin to compile the Rust NIF, erlfmt for code formatting, and ex\_doc for documentation generation, while the README now reflects the new name and current serialization capabilities.

(repo-wide) · high confidence

Added new Common Test suites (\\*\_SU.erl\) and supporting utilities (\arrow\_test\_utils.erl\, \arrow\_ipc\_marks\_data.hrl\) to verify the correctness of Arrow array creation, metadata access, and serialization. The tests cover fixed and variable primitive arrays, buffers, bitmaps, offsets, and nested list structures, ensuring proper handling of null values (\undefined\/\nil\) and validity bitmaps. Additionally, new tests validate the Arrow IPC module, including schema and record batch encoding/decoding, message serialization, and file footer generation using Flatbuffers.

test · high confidence

The native Arrow format NIF module now depends on \rustler\ version 0.38.0 and \arrow-format\ version 0.8 (with IPC features enabled). This establishes the foundational Rust dependencies required for the new Flatbuffers NIF code generation, replacing earlier versions (such as rustler 0.30.0/0.35.0) with the latest stable release to ensure compatibility with the new native interface.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 65.

• Code Health 98
• Architecture 100
• Maturity 51
• Readiness 66
• Security 80

• 92 commits — 82 feature/other, 10 fixes

• .github/workflows — 31 commits
• (root) — 23 commits
• native/arrow_format_nif — 12 commits
• src/serde_arrow_array.erl — 8 commits
• src/serde_arrow_ipc_message.erl — 3 commits
• .github/pull_request_template.md — 2 commits
• guides/quick-run-through.livemd — 2 commits
• src/serde_arrow_bitmap.erl — 2 commits
• src/serde_arrow_fixed_list_array.erl — 2 commits
• src/serde_arrow_ipc_field.erl — 2 commits
• (repo) — 1 commit
• .github/ISSUE_TEMPLATE — 1 commit
• .github/dependabot.yml — 1 commit
• src/serde_arrow_buffer.erl — 1 commit
• src/serde_arrow_ipc_file.erl — 1 commit

• fix: Add support for the Fixed-Size Primitive Layout (#7)
• fix: Fix Dialyzer Errors (#43)
• fix: Fix documentation deployment via GitHub Pages (#45)
• fix: Fix failing tests (#34)
• fix: Fix link in Fixed-Size List Layout docs
• fix: Revert "Generate and deploy docs to GitHub Pages (#2)"
• fix: Revert "Generate and deploy docs to GitHub Pages"
• fix: Support Fixed-Sized Lists (#13)
• fix: fix: arrow_buffer:size/1 use 64 byte padding not 8 (#114)
• fix: fix: use stable tag for dtolnay/rust-toolchain (#94)
• change: Add .asf.yaml (#48)
• change: Add Representation for Arrow's Primitive Datatypes (#6)
• change: Add Roadmap for v0.1.0 (#5)
• change: Add Rust CI (#27)
• change: Add Variable-Size List (#15)
• change: Add a run-through guide (#39)
• change: Add shorthand for creating message body (#40)
• change: Add support for the Variable-Sized Binary Layout (#9)
• change: Allow length to be passed into buffer:from_erlang (#19)
• change: Bump array-init-cursor from 0.2.0 to 0.2.1 in /native/arrow_format_nif (#47)
• …and 72 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

apache/arrow-erlang was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.