huggingface/datasets
67.6
Adequate · 17 November 2026
44.1k
lines of production code
Python
primary language
5
measurements over time
What this system is
Features
Add AudioFolder packaged loader
Introduces a new AudioFolder builder that allows users to load audio datasets directly from folder structures. The loader supports a wide range of audio and video container formats (including .wav, .mp3, .flac, .ogg, .mp4, .mkv, and others) and provides configuration options to drop labels or metadata during loading.
_src/datasets/packaged\modules/audiofolder · high confidence
Add CoNLL dataset format loader
Users can now load CoNLL-style datasets (including CoNLL-2000, CoNLL-2003, and CoNLL-U formats) using the \conll\ builder. This new module parses whitespace-separated token and tag columns, handles sentence boundaries via empty lines, and supports custom column schemas, encoding options, and skipping of document-start markers or comment lines.
_src/datasets/packaged\modules/conll · high confidence
Add GenBank file format support for biological sequence data
Users can now load GenBank files (nucleotide or protein sequences with annotations) using the new GenBank dataset builder. This feature introduces a pure Python state machine parser that handles multi-line feature locations, wrapped qualifier values, and header continuations, outputting structured data via PyArrow without requiring external dependencies.
_src/datasets/packaged\modules/genbank · high confidence
Add HDF5 dataset support
Users can now load HDF5 files (\.h5\, \.hdf5\) as datasets. This new module automatically infers schema from the HDF5 structure, supporting nested arrays, complex numbers (converted to real/imaginary components), compound dtypes, and variable-length strings, converting them into Arrow tables for efficient access.
_src/datasets/packaged\modules/hdf5 · high confidence
Add Harbor dataset builder for agent evaluation tasks
Users can now load Harbor task repositories as a dataset, where each row represents a single agent-evaluation task directory. The builder parses \task.toml\ and \instruction.md\ files to expose task metadata, instructions, keywords, and file listings, while optionally stripping benchmark canary markers from instructions. This enables exploring Harbor benchmarks using standard dataset APIs without requiring a Harbor installation.
_src/datasets/packaged\modules/harbor · high confidence
Add Parquet dataset loader with streaming, filtering, and bad-file handling
Users can now load Parquet files directly into a Dataset using the new packaged Parquet builder. This change introduces support for streaming large Parquet files, applying predicate pushdown filters to skip unnecessary data, and configuring how the loader handles invalid files via the new \on\_bad\_files\ argument (error, warn, or skip). It also allows specifying a subset of columns to load and supports complex feature types for data casting.
_src/datasets/packaged\modules/parquet · high confidence
Add Spark dataset builder and IterableDataset integration
Users can now load Apache Spark DataFrames directly into the datasets library using \Dataset.from\_spark\ and \IterableDataset.from\_spark\. This new packaged module (\src/datasets/packaged\_modules/spark\) provides a \Spark\ builder that handles caching, partitioning, and sharding of Spark data, enabling efficient iteration over large distributed datasets without full materialization.
_src/datasets/packaged\modules/spark · high confidence
Add TsFile (Apache IoTDB) packaged builder with per-device wide format
Users can now load TsFile data (from Apache IoTDB) using the new \tsfile\ dataset builder. This builder reads time-series data in a per-device wide format, where each output row represents a single device identified by its TAG values, with the \time\ column and every FIELD column stored as Arrow list columns. Data from multiple TsFiles is merged and sorted by time, and peak memory usage is bounded by a single device's payload rather than the entire split.
_src/datasets/packaged\modules/tsfile · high confidence
Add XML dataset builder
Users can now load datasets from XML files using the \load\_dataset\ function with the \xml\ format. This new builder reads the entire content of each XML file as a string and stores it in a column named \xml\ (or a custom feature column if specified), supporting configurable encoding and error handling.
_src/datasets/packaged\modules/xml · high confidence
Add support for loading FASTA biological sequence files
Users can now load FASTA files (e.g., .fa, .fasta, .fna) directly into datasets. This new module parses nucleotide and peptide sequences, exposing columns for sequence ID, description, and the sequence data itself. It supports configurable batching to handle large genomic files and optionally includes raw record bytes or Biopython-compatible sequence objects when the biopython library is installed.
_src/datasets/packaged\modules/fasta · high confidence
Add support for loading FASTQ sequencing data files
Users can now load FASTQ files (with .fq or .fastq extensions) directly into datasets. This new module parses nucleotide sequences and their associated quality scores, exposing columns for sequence ID, description, sequence, and quality. It includes a configurable batch size and a default 256MB byte limit to prevent Parquet page size errors, and optionally provides a raw 'record' column for the original bytes.
_src/datasets/packaged\modules/fastq · high confidence
Add support for loading mmCIF protein structure files
Users can now load macromolecular structures in mmCIF format (.cif, .mmcif) using the new MmcifFolder builder. This feature follows the existing folder-based pattern, allowing datasets to be constructed from directories where folder names serve as labels or via an optional metadata CSV file. Each row in the resulting dataset contains one complete structure file, stored in a BioStructure column that returns a Bio.PDB.Structure.Structure object when decoded, requiring the biopython library.
_src/datasets/packaged\modules/mmcif · high confidence
Add support for reading Apache Iceberg tables
Users can now load datasets directly from Apache Iceberg tables by specifying a \pyiceberg\ catalog and table identifier. This new module supports reading specific columns, applying row filters with predicate pushdown, selecting specific snapshots for time-travel queries, and parallelizing reads across multiple processes.
_src/datasets/packaged\modules/iceberg · high confidence
Add support for reading Lance datasets
Users can now load datasets stored in the Lance format by specifying \data\_files\ with \.lance\ directories or files. This new builder automatically infers schema and feature types (including Audio, Image, and Video) from binary magic bytes, handles authentication for Hugging Face Hub-hosted Lance datasets, and supports both fragment-based and single-file reading modes.
_src/datasets/packaged\modules/lance · high confidence
Added DVC plot configuration templates
The .dvc directory now includes configuration files for visualizing metrics, specifically adding templates for confusion, default line, scatter, and smooth plots. These Vega-Lite JSON files define how DVC renders metric data, enabling users to generate standardized visualizations for model performance tracking directly within the repository.
.dvc · high confidence
Added automated release version management utility
A new \utils/release.py\ script has been added to automate the process of updating version numbers across the project. This tool handles pre-release and post-release workflows by updating the \\_\version\\\ in \src/datasets/\\init\\_.py\ and the \version\ argument in \setup.py\, ensuring consistent versioning for new releases and development cycles.
utils · high confidence
Added iteration and performance benchmarks
The benchmarks directory now includes a suite of scripts to measure dataset performance, including \benchmark\_iterating.py\ for read speeds, \benchmark\_map\_filter.py\ for map/filter operations, \benchmark\_indices\_mapping.py\ for select/sort/shuffle, \benchmark\_array\_xd.py\ for Array2D handling, and \benchmark\_getitem\_100B.py\ for large-scale indexing. These scripts generate test datasets and record execution times for various operations, outputting results as JSON files for comparison.
benchmarks · high confidence
Initial release of the HuggingFace Datasets library (v5.0.2.dev0)
This change introduces the \datasets\ library as a new package, establishing the core API for loading, manipulating, and saving datasets. The \src/datasets/\_\init\\_.py\ file exposes key classes such as \Dataset\, \IterableDataset\, \DatasetDict\, and \IterableDatasetDict\, along with loading functions like \load\_dataset\ and \load\_from\_disk\. It also includes utilities for dataset inspection, splitting, and fingerprinting, providing the foundational interface for users to interact with structured data. Also in this area: 1 commit (1 fix) fixing src/datasets.
_src/datasets/packaged\modules/eval · medium confidence · unverified
Introduce Cache dataset builder for reloading cached datasets
A new \Cache\ dataset builder has been added to \src/datasets/packaged\_modules/cache\, enabling users to reload datasets directly from the local cache directory. This builder supports automatic cache lookup via \hash='auto'\ and \version='auto'\, allowing seamless resumption or reuse of previously processed datasets without re-downloading or re-processing data. It handles configuration matching, cache path resolution (including snake-case normalization), and streaming from cached Arrow files.
_src/datasets/packaged\modules/cache · high confidence
Introduce FolderBasedBuilder and PDB dataset support
This change introduces a new \FolderBasedBuilder\ base class that standardizes how folder-based datasets (like ImageFolder and AudioFolder) are loaded, handling label inference, metadata file discovery (CSV, JSONL, Parquet), and streaming extraction. It also adds a new \PdbFolder\ packaged module that leverages this builder to load Protein Data Bank (PDB) structure files, storing them as \BioStructure\ objects. Users can now load PDB datasets directly using the \pdb\ identifier, with support for folder-based labels and custom metadata files.
_src/datasets/packaged\_modules/folder\_based\builder · high confidence
Introduce Generator packaged module for creating datasets from callables
A new Generator packaged module has been added to the library, enabling users to create datasets directly from a Python callable (generator function). This module includes a GeneratorConfig that validates the input, ensuring the provided generator is callable and specifying the dataset split, features, and keyword arguments. The Generator builder class implements the logic to split generation tasks across shards and yield examples, providing a structured way to build datasets from custom generation logic.
_src/datasets/packaged\modules/generator · high confidence
Introduce ImageFolder packaged dataset builder
Adds a new ImageFolder packaged module that allows users to load image datasets directly from local directory structures. The implementation extends the generic FolderBasedBuilder, defining a specific configuration for dropping labels or metadata and registering a comprehensive list of supported image file extensions (such as .jpg, .png, .gif, and .webp) while explicitly excluding ambiguous formats like .h5.
_src/datasets/packaged\modules/imagefolder · high confidence
Introduce WebDataset packaged module with multi-modal support
Adds a new \WebDataset\ builder to the \packaged\modules\ directory, enabling users to load datasets stored in WebDataset TAR archives. The module automatically infers schema from the first few examples and supports decoding for images, audio, video, and 3D meshes based on file extensions. It also handles compressed files and ensures special columns like \\\key\\\ and \\\url\\_\ are positioned last in the output examples.
_src/datasets/packaged\modules/webdataset · high confidence
Introduces parallel processing backend and refactors utility modules
This change introduces a new \parallel\ module that allows users to configure a parallel backend (such as joblib or Spark) for dataset operations via the \parallel\_backend\ context manager and \parallel\_map\ function. It also restructures the \utils\ package by introducing dedicated modules for file locking (\\_filelock\), extraction (\extract\), and deprecation handling (\deprecation\_utils\), while updating the pickling logic in \\_dill\ to support additional types like \torch.Generator\ and \tiktoken.Encoding\ and ensuring consistent hashing for Arrow tables.
src/datasets/utils · high confidence
Introduction of dataset metadata resource files
The \src/datasets/utils/resources\ directory now contains a set of JSON and YAML configuration files that define the valid metadata categories for datasets. These include \languages.json\ for language codes, \creators.json\ for annotation and language source types, \multilingualities.json\ for language distribution types, \size\_categories.json\ for dataset size buckets, and \readme\_structure.yaml\ for validating dataset card sections. These resources provide the schema definitions used to validate and standardize dataset metadata.
src/datasets/utils/resources · high confidence
New JSON packaged module with agent trace support and streaming
Introduces a new \Json\ packaged builder module (\src/datasets/packaged\_modules/json\) that enables loading JSON and JSON Lines datasets. This module supports streaming via on-the-fly extraction, allows specifying encoding and error handling, and includes specific logic to parse and convert agent traces (such as Hermes traces) into training data when the \parse\_agent\_traces\ config is enabled. It also handles schema inference, nested type casting, and preserves column order.
_src/datasets/packaged\modules/json · high confidence
New decodable features for bio-sequences, bio-structures, 3D meshes, NIfTI, and PDFs
The \src/datasets/features\ module now exports five new decodable feature types: \BioSequence\ and \BioStructure\ for reading biological sequence and macromolecular structure files (using Biopython), \Mesh\ for 3D mesh data (using trimesh), \Nifti\ for neuroimaging files (using nibabel), and \Pdf\ for PDF documents. These features allow users to load complex file formats directly into their datasets, with automatic encoding/decoding and support for both local paths and remote Hub storage.
src/datasets/features · high confidence
New modular formatting system with JAX, Polars, and PyTorch uint16/uint32 support
The \src/datasets/formatting\ module has been restructured into a new, modular formatting system that introduces dedicated formatters for JAX (\JaxFormatter\), Polars (\PolarsFormatter\), and PyTorch (\TorchFormatter\), alongside existing support for NumPy, Pandas, TensorFlow, and Arrow. This change enables users to set the dataset format to 'jax', 'polars', 'pt', 'pytorch', 'np', 'pd', 'tensorflow', 'arrow', 'pyarrow', or 'python' via \Dataset.set\_format\. The JAX formatter adds device selection capabilities and handles serialization constraints for JAX devices. The Polars formatter provides native DataFrame and Series extraction. The PyTorch formatter now automatically converts \uint16\ and \uint32\ NumPy dtypes to compatible \int64\ tensors to prevent errors, and supports decoding from Torchvision/Torchcodec video/audio readers. The core \formatting.py\ module also includes optimized table querying and extraction logic.
src/datasets/formatting · high confidence
New packaged dataset modules and extension mappings
The \src/datasets/packaged\modules/\\init\\_.py\ module has been introduced to centralize the registration of built-in dataset loaders. This change adds support for a wide range of new file formats, including Lance, CoNLL, Parquet, NDJSON, GenBank, NIfTI, PDB, mmCIF, FASTA, FASTQ, Apache Iceberg, Vortex, TsFile, Harbor, and PDF, alongside existing formats like CSV, JSON, and HDF5. It also registers folder-based builders for images, audio, video, meshes, and NIfTI data. The file defines the \\_EXTENSION\_TO\_MODULE\ mapping, which automatically infers the correct loader based on file extensions (including case-insensitive variants for media folders), and maintains hashes for caching module integrity.
_src/datasets/packaged\modules · high confidence
Repository initialization with modern tooling and documentation
The repository is initialized with a comprehensive set of configuration and documentation files. A pre-commit configuration is added to enforce code quality using Ruff (v0.11.8) for linting and formatting. Documentation standards are established through a Code of Conduct, Security Policy, and updated Contributing guidelines that reflect the current project structure. Citation metadata is provided via CITATION.cff and .zenodo.json to ensure proper academic attribution. Additionally, a Makefile is introduced to streamline common development tasks such as quality checks, styling, and testing, while .gitignore and .dvcignore are configured to manage version control and data versioning artifacts.
(repo-wide) · high confidence
Support for loading and streaming Arrow datasets
Users can now load and stream datasets stored in the Apache Arrow IPC format (both stream and file binary formats) using the new Arrow packaged module. The builder automatically infers dataset features from the Arrow schema if not explicitly provided, validates record batches for integrity, and handles file reading errors gracefully.
_src/datasets/packaged\modules/arrow · high confidence
Support for streaming compressed datasets via new filesystem protocols
The library now allows users to stream datasets directly from compressed files (BZ2, GZIP, LZ4, XZ, and Zstandard) by registering dedicated filesystem protocols (e.g., \gzip://\, \zstd://\). This change introduces a new \compression\ module and updates \src/datasets/filesystems/\_\init\\_.py\ to register these protocols with fsspec, enabling transparent decompression during data loading without requiring local extraction. It also includes a fix for local file renaming to use efficient OS-level moves and disables URL re-quoting for aiohttp to prevent redirect issues.
src/datasets/filesystems · high confidence
Text dataset builder now supports loading by paragraph or document
The text dataset builder has been extended to allow users to load data by line (the default), paragraph, or entire document. This is controlled via the new \sample\_by\ configuration parameter, which splits input text into rows based on double newlines for paragraphs or reads the whole file for documents. Additionally, a new \keep\_linebreaks\ boolean parameter allows users to preserve newline characters in the output, and the builder now supports complex feature types for data casting.
_src/datasets/packaged\modules/text · high confidence
Removals
Removal of legacy NLP CLI commands
The \src/nlp/commands\ module has been removed, eliminating the legacy command-line interface tools for dataset conversion, model downloading, environment inspection, and user authentication (login/logout/whoami/S3 management). Users can no longer use these specific CLI subcommands to interact with TensorFlow datasets or manage Hugging Face credentials via the \nlp\ package.
src/nlp/commands · high confidence
Removal of legacy NLP dataset builders
The \datasets\ module has removed a large set of legacy NLP dataset builders that relied on the deprecated \nlp\ library. Specifically, the following dataset scripts have been deleted: \glue\, \aeslc\, \amazon\_us\_reviews\, \big\_patent\, \billsum\, \blimp\, \c4\ (including \c4\_utils\), \cfq\, and \civil\_comments\. Users relying on these datasets will need to migrate to the current \datasets\ library implementation.
datasets · high confidence
Removal of legacy NLP utility modules
The \src/nlp/utils\ package has been removed, deleting the \\_\init\\_.py\, \file\_utils.py\, \py\_utils.py\, \tqdm\_utils.py\, and \version.py\ files. This eliminates the legacy utilities for dataset caching, file downloading, Python helpers, progress bars, and version management that were previously exposed via wildcard imports from this location.
src/nlp/utils · high confidence
Removal of the legacy \`nlp\` package
The \src/nlp\ directory, containing the legacy NLP library (including core modules like \arrow\_dataset\, \builder\, \features\, and \arrow\_writer\), has been completely removed. This deletion eliminates the old \nlp\ namespace and its associated dataset loading, feature definition, and Arrow I/O implementations from the codebase.
src/nlp · high confidence
Removed legacy NLP download module
The \src/nlp/download\ package, which previously provided the \DownloadManager\ API for downloading and extracting dataset files, has been removed. This change eliminates the legacy download and extraction logic, including checksum verification, archive handling, and Kaggle integration, from the library.
src/nlp/download · high confidence
Behavioural changes
3 commits (1 fix) modifying src/datasets/packaged\_modules/meshfolder
A change to existing behaviour in src/datasets/packaged\_modules/meshfolder — 3 commits (1 fix), 2 files: “Fast streaming decode for image/audio/video/etc. folders datasets (\#8791)”; “fix(mesh): accept .gltf in the Mesh feature and the meshfolder builder (\#8958)”; “feat: add 3D mesh support and MeshFolder builder (\#8055)” (refs \#8791, \#8958, \#8055).
_src/datasets/packaged\modules/meshfolder · medium confidence · unverified
CSV builder now supports pandas 2.0+ and 2.2+ parameters
The CSV dataset builder has been updated to handle version-specific parameters in pandas' \read\_csv\. It now supports the \date\_format\ parameter introduced in pandas 2.0 and correctly handles the deprecation of the \verbose\ parameter in pandas 2.2. The builder dynamically filters these arguments based on the installed pandas version to ensure compatibility and prevent errors when loading CSV files.
_src/datasets/packaged\modules/csv · high confidence
Deprecation of the Pandas builder
The Pandas builder is now deprecated and will be removed in the next major version of the datasets library. Users relying on this builder will see a FutureWarning when loading data, indicating that they should migrate to alternative data loading methods.
_src/datasets/packaged\modules/pandas · high confidence
Introduce structured datasets-cli with env, test, and delete commands
The datasets CLI is now structured around a base command class and three specific subcommands: \env\ (prints system and library versions for debugging), \test\ (validates dataset loading with options for caching, verification, and info saving), and \delete\_from\_hub\ (removes a dataset config from the Hub). This refactoring replaces the previous monolithic CLI entry point with a modular architecture, improving maintainability and allowing users to run targeted dataset validation or environment checks directly from the command line.
src/datasets/commands · high confidence
New modular I/O subsystem for datasets
The \src/datasets/io\ package introduces a new modular architecture for reading and writing datasets, replacing the previous monolithic implementation. This change adds dedicated reader and writer classes for CSV, JSON, Parquet, SQL, Spark, and text formats, as well as a generator stream. Users benefit from a consistent interface across all I/O operations, including support for multi-process parallelism in CSV, JSON, and SQL writers, fsspec integration for remote storage access, and configurable batch sizes and compression options. The new structure also standardizes streaming and non-streaming dataset creation across all supported formats.
src/datasets/io · high confidence
Refactored download module with new configuration and streaming support
The download subsystem has been reorganized into a dedicated \src/datasets/download\ package, introducing a structured \DownloadConfig\ dataclass that centralizes settings such as cache directories, parallel processing limits (\num\_proc\), authentication tokens, and storage options. This change separates the logic into \DownloadManager\ for standard cached downloads and \StreamingDownloadManager\ for lazy, on-the-fly data access, providing a clearer API for users managing data retrieval and streaming workflows.
src/datasets/download · high confidence
Remove legacy notebooks and add documentation index
The notebooks directory has been cleaned up by removing the outdated 'HuggingFace Datasets library demo' notebook (and its checkpoint) and the 'playground.py' script, which relied on the deprecated \nlp\ library. A new README.md has been added to serve as an index for official and community notebooks, including a link to the Quickstart guide.
notebooks · high confidence
Test coverage
Add test suite for distributed dataset splitting; Added comprehensive test coverage for new and existing feature types; Added test coverage for CSV, JSON, Parquet, SQL, and Text I/O readers and writers; Added test fixtures for Protein Data Bank (PDB) and CIF formats; Added tests for packaged module loaders and caching; Added tests for the CLI test command and dataset info YAML generation; New test fixtures for file formats, fsspec, and Hub integration; Removal of download subsystem test suite; Removed obsolete test suite files.
Dependencies
Migrate configuration to pyproject.toml and configure ruff linter
Project configuration has moved from setup.cfg to pyproject.toml, introducing the ruff linter with specific rule selections (C, E, F, I, W) and ignores for line length, undefined names in type annotations, and complex functions. The file also configures import sorting via isort and sets up pytest to treat FutureWarnings from huggingface\_hub as errors during CI runs.
(dependencies) · high confidence
Housekeeping
Add benchmark results for array, iteration, and mapping operations; Release 5.0.2.dev0.
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 65.0 → 67.6 (+2.6)
- Rubric changed (rubric-2026.11.2 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 70.5 → 67.9 (-2.6)
- Architecture 90.4 → 99.7 (+9.3)
- Maturity 57.1 → 63.6 (+6.5)
- Readiness 72.1 → 69.4 (-2.7)
- Security 67.9 → 67.9 (0.0)
- Performance 88.6 → 88.6 (0.0)
Resolved (6)
- Duplicated block (7 lines × 2) (src/datasets/packaged_modules/parquet/parquet.py)
- Members sharing a duplicated core (4 members, 50+ identical tokens) (src/datasets/arrow_dataset.py)
- No ADRs found
- Off-boarding risk: anonymized user #1
- Off-boarding risk: anonymized user #2
- Projects may be oversized for their cohesion
New (132)
- Duplicated block (8 lines × 2) (src/datasets/io/csv.py)
- Edited copies of one member (3 members, 10+ corresponding lines) (src/datasets/packaged_modules/fasta/fasta.py)
- Edited copies of one member (3 members, 11+ corresponding lines) (src/datasets/table.py)
- Edited copies of one member (3 members, 14+ corresponding lines) (src/datasets/formatting/jax_formatter.py)
- Edited copies of one member (3 members, 15+ corresponding lines) (src/datasets/iterable_dataset.py)
- Edited copies of one member (3 members, 24+ corresponding lines) (src/datasets/utils/file_utils.py)
- Edited copies of one member (3 members, 28+ corresponding lines) (src/datasets/io/csv.py)
- Edited copies of one member (3 members, 34+ corresponding lines) (src/datasets/arrow_dataset.py)
- Edited copies of one member (3 members, 34+ corresponding lines) (src/datasets/arrow_dataset.py)
- Edited copies of one member (3 members, 41+ corresponding lines) (src/datasets/dataset_dict.py)
- Edited copies of one member (4 members, 17+ corresponding lines) (src/datasets/io/csv.py)
- Edited copies of one member (4 members, 9+ corresponding lines) (src/datasets/iterable_dataset.py)
- Edited copies of one member (6 members, 61+ corresponding lines) (src/datasets/features/audio.py)
- Edited copy of a member (10 corresponding lines) (src/datasets/iterable_dataset.py)
- Edited copy of a member (12 corresponding lines) (src/datasets/arrow_reader.py)
- Edited copy of a member (12 corresponding lines) (src/datasets/formatting/formatting.py)
- Edited copy of a member (12 corresponding lines) (src/datasets/utils/file_utils.py)
- Edited copy of a member (13 corresponding lines) (src/datasets/formatting/formatting.py)
- Edited copy of a member (14 corresponding lines) (src/datasets/iterable_dataset.py)
- Edited copy of a member (14 corresponding lines) (src/datasets/packaged_modules/arrow/arrow.py)
- …and 112 more
Changes since last survey
- 15 commits — 7 feature/other, 8 fixes
By area
- src/datasets — 10 commits
- (root) — 3 commits
- docs/source — 2 commits
Notable commits
- fix: Fix Dataset.take raising IndexError when taking more elements than available (#8312)
- fix: Fix IterableDataset state_dict freezing when a sibling dataset is created (#8992)
- fix: Fix IterableDataset.rename_columns with arrow, pandas and polars formats (#8977)
- fix: Fix batch(by_column=) merging groups that are separated by null values (#8966)
- fix: Fix mixed-type JSON fields with slashes in their names (#8976)
- fix: Fix pickle RCE in pandas dataset builder ([CVE redacted]) (#8879)
- fix: Fix save_to_disk failing when a batch holds more than 2GB of image or audio files (#8828)
- fix: Fix the eval loader for string sample ids (#8972)
- change: Cast large_string and string_view storage in ClassLabel (#8688)
- change: Preserve headers and document structure when exporting empty datasets (#8811)
- change: docs: clarify download modes and transform caching (#8804)
- change: more distributed docs (hsdp) (#8957)
- change: release: 5.2.0 (#8984)
- change: remove fsspec upper bound (#9057)
- change: set dev version (#8985)
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
huggingface/datasets was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 17 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit dbaac5a033dd128bbbcf764bb3a0d2995e495219 — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.84+preprod-d645ca9bbedd.