huggingface/tokenizers
68.7
Adequate · 14 November 2026
38.8k
lines of production code
Rust
primary language
5
measurements over time
What this system is
Features
Introduction of the new \`tk-encode\` inference crate
The \tokenizers\ library now includes a dedicated \tk-encode\ crate for the inference runtime, separating it from the legacy engine. This new crate provides a read-only \PipelineTokenizer\ that processes text through Normalizers, PreTokenizers, Models, and PostProcessors to produce Encodings. It is designed for performance and small binary size by excluding \serde\ dependencies (using \hifijson\ for parsing instead) and excluding training logic. Users can enable specific tokenizer models (BPE, Unigram, WordPiece, WordLevel) and features like parallelism or HTTP-based model downloading via feature flags.
tokenizers/tk-encode/src · high confidence
New Python binding examples for encoding, batching, and training
Added six new example scripts in the Python bindings directory to demonstrate core tokenizer capabilities. \encode\_decode.py\ and \from\_pretrained.py\ show basic tokenization and loading from the Hugging Face Hub. \batch\_padding.py\ and \truncation.py\ illustrate how to configure and apply padding and truncation strategies both globally and per-call. \multiprocessing\_workers.py\ demonstrates efficient parallel encoding using Python's multiprocessing pool. Finally, \train\_parity\_bpe.py\ provides a reference implementation for training a parity-aware BPE tokenizer across multiple languages, though it is currently commented out as the Python bindings do not yet expose the training functionality. Also in this area: 3 commits (0 fixes) modifying bindings, bindings/python; Support encoding pairs of texts in Python bindings.
bindings/python/examples · medium confidence · unverified
New Unicode Scripts pre-tokenizer for script-aware tokenization
Users can now apply a pre-tokenizer that splits text boundaries based on Unicode script changes (e.g., separating Latin from CJK). This new \UnicodeScripts\ pre-tokenizer, available in \tk-encode\, groups characters by their script (such as Han, Hiragana, Katakana, Latin, etc.) and emits spans where the script changes, while treating spaces and common punctuation as neutral characters that do not trigger splits. It includes specific handling to map Hiragana, Katakana, and the prolonged sound mark (ー) to the Han script for CJK text processing.
_tokenizers/tk-encode/src/pre\_tokenizers/unicode\scripts · high confidence
New Unigram and WordLevel tokenizer models in tk-encode
The \tk-encode\ crate now includes implementations for the Unigram and WordLevel tokenization models. The Unigram model introduces a lattice-based structure to perform Viterbi decoding for optimal tokenization paths and supports sampling from possible encodings, along with features like byte-fallback and configurable unknown token handling. The WordLevel model provides a straightforward lookup-based tokenization strategy. Both models are integrated into the new encoding pipeline, replacing legacy engine components and enabling more flexible and performant text tokenization for users.
tokenizers/tk-encode/src/models/unigram · high confidence
New WordPiece tokenizer model implementation
The \tk-encode\ crate now includes a new \WordPiece\ model implementation in \tokenizers/tk-encode/src/models/wordpiece\. This change introduces a builder-based API for configuring vocabulary, unknown tokens, and subword prefixes, replacing the legacy encode engine and its serialization layer. Users can now construct WordPiece tokenizers programmatically, with the model handling tokenization logic including character limits and subword continuation prefixes.
tokenizers/tk-encode/src/models/wordpiece · high confidence
New dev-time tool to generate bitcannon atom classification tables
A new \bitmap\_gen\ crate has been added to bake Unicode character classification data into static tables for the \bitcannon\ tokenizer. This dev-time tool generates \atom\_tables.rs\ by mapping codepoints to specific atom tags (handling ASCII, 2-byte, 3-byte, BMP RLE, and astral ranges) based on Unicode general categories and script properties (such as Han). This ensures the runtime tokenizer uses pre-computed, consistent classification data rather than performing per-byte range tests.
_tokenizers/bitmap\gen · high confidence
New tokenization decoders and encoding utilities in tk-encode
The \tk-encode\ crate now includes a comprehensive set of decoder implementations (BPE, ByteFallback, ByteLevel, CTC, Fuse, Metaspace, Replace, Strip, and WordPiece) and supporting utilities for padding, truncation, caching, and parallel execution. This adds the capability to decode tokenized sequences back into readable text using various standard algorithms and to manage encoding parameters like padding direction and strategy, significantly expanding the encoding/decoding pipeline's functionality.
tokenizers/tk-encode/src/utils · high confidence
New tokenization decoders and encoding utilities in tk-encode
The \tk-encode\ crate now includes a comprehensive set of decoder implementations (BPE, ByteFallback, ByteLevel, CTC, Fuse, Metaspace, Replace, Strip, and WordPiece) and supporting utilities for padding, truncation, caching, and parallel execution. This adds the capability to decode tokenized sequences back into readable text using various standard algorithms and to manage encoding parameters like padding direction and strategy, significantly expanding the encoding/decoding pipeline's functionality. Also in this area: Improved Unicode normalization and WordPiece vocabulary handling.
tokenizers/tk-encode/src/decoders · high confidence
Node bindings scaffolded with build and tooling configuration
The node bindings directory has been initialized with essential build and development infrastructure. This includes a Cargo configuration for cross-compiling the native Rust backend to aarch64 Linux musl targets, a Yarn 3.5.1 release binary for package management, and standard editor tooling (EditorConfig, Prettier, Taplo, Git attributes, and Gitignore) to ensure consistent code formatting and repository hygiene.
bindings/node · high confidence
Behavioural changes
2 commits (1 fix) modifying tokenizers/bitcannon
A change to existing behaviour in tokenizers/bitcannon — 2 commits (1 fix), 10 files: “perf(bitcannon): add RVV literal matching (\#2650)”; “Fix Kimi pretokenization for overlapping Han categories (\#2567)” (refs \#2650, \#2567). Also in this area: Introduce bitcannon bitstream pre-tokenizer with unified class-run handling.
tokenizers/bitcannon · medium confidence · unverified
3 commits (0 fixes) modifying bindings, bindings/python
A change to existing behaviour in bindings, bindings/python — 3 commits, 28 files: “feat(composition): post-processors (\#2573)”; “feat(composition): models and Tokenizer(model) (\#2569)”; “feat(composition): generate Python bindings for components (\#2629)” (refs \#2573, \#2569, \#2629). Also in this area: Support encoding pairs of texts in Python bindings; Python array access now returns copies instead of views, and WebAssembly bindings are added; Python bindings rework: new Tokenizer API with Encoding views and padding/truncation options.
bindings/python/python · medium confidence · unverified
6 commits (0 fixes) modifying bindings/python/src/components
A change to existing behaviour in bindings/python/src/components — 6 commits, 13 files: “feat(composition): ComponentSequence (\#2577)”; “feat(composition): post-processors (\#2573)”; “feat(composition): models and Tokenizer(model) (\#2569)”; and 3 more (refs \#2577, \#2573, \#2569, \#2579, \#2551, …).
bindings/python/src/components · medium confidence · unverified
7 commits (1 fix) modifying bindings/python/tools/stub-gen
A change to existing behaviour in bindings/python/tools/stub-gen — 7 commits (1 fix), 1 file: “chore: bump pyo3 to version 0.29 (\#2115)”; “Add type hint, update to pyo3 0.27, add automatic type hint generator (\#1928)”; “feat(composition): component handles (\#2579)”; and 4 more (refs \#2115, \#1928, \#2579, \#1978, \#2220, …).
bindings/python/tools/stub-gen · medium confidence · unverified
Introduce canonical tokenizer.json 2.0 reader and writer
The \tk-serialize\ crate now implements the canonical \tokenizer.json\ version 2.0 format, replacing the legacy 1.0 handling. It provides a strict reader that only accepts version 2.0 files (refusing legacy 1.0 files and requiring offline conversion via \tk-convert\ for older formats) and a writer that produces version 2.0 output. This change standardizes the serialization schema, enforcing exact key ordering and explicit component types (e.g., \TemplateProcessing\ instead of legacy spellings like \BertProcessing\), and ensures that legacy constructs like \Metaspace\ pre-tokenizers or \ByteLevel\ pre-tokenizers are refused unless converted, thereby guaranteeing that the serialized configuration matches the canonical specification exactly. Also in this area: 1 commit (1 fix) fixing tokenizers/bitmap\_gen, tokenizers/tk-serialize; 2 commits (1 fix) modifying tokenizers, tokenizers/tk-serialize; Fix Split pre-tokenizer to respect the active regex backend; Fix StripAccents normalizer instantiation in tk-serialize; Improved error handling and duplicate detection in BPE vocabulary building.
tokenizers/tk-serialize · medium confidence · unverified
Legacy tokenizer.json upgrade pass
The tk-convert crate now provides a JSON-to-JSON upgrade pass that rewrites legacy tokenizer configurations into a canonical format (version 2.0). This ensures that the serialization reader no longer needs to maintain backwards-compatibility branches. The pass handles specific legacy patterns, such as inferring missing model types (e.g., distinguishing BPE from WordPiece based on merge presence), rewriting merge entries from space-separated strings to arrays, and normalizing Metaspace and ByteLevel pre-tokenizers into their canonical decoder and normalizer forms. It also validates ambiguous configurations and refuses them with descriptive errors rather than guessing.
tokenizers/tk-convert · high confidence
New SIMD-accelerated pre-tokenizers for BERT, ByteLevel, and delimiter-based splitting
The \tk-encode\ pre-tokenization pipeline now includes native implementations for \BertPreTokenizer\, \ByteLevel\, \CharDelimiterSplit\, \Digits\, \FixedLength\, \Punctuation\, \Split\, \Whitespace\, and \WhitespaceSplit\. These new components replace legacy scalar logic with \bitcannon\-powered SIMD finite-state machines and optimized byte-scanning, delivering byte-exact behavior while significantly improving performance. Users benefit from faster tokenization for common patterns (such as BERT-style punctuation isolation, GPT-style regex splitting, and fixed-length chunking) without changing their configuration, as the new implementations are designed to be drop-in replacements for existing pre-tokenizer definitions.
_tokenizers/tk-encode/src/pre\tokenizers · high confidence
New encode pipeline with configurable options and parallel processing
The tokenizer's encoding pipeline has been restructured to use a new \EncodeOptions\ struct that exposes \add\_special\_tokens\ and \encode\_special\_tokens\ flags, allowing users to control how special tokens are handled during encoding. The pipeline now supports parallel processing for large inputs (above 8KB) via a thread pool with sharded scratch buffers to reduce contention, and includes a new \PipelinePostProcessor\ that manages template-based wrapping of sequences with type IDs.
tokenizers/tk-encode/src/tokenizer/pipeline · high confidence
New high-performance BPE merge engines and pipeline integration
The BPE model in \tk-encode\ has been replaced with a new pipeline implementation that uses two specialized merge engines: a multipass engine for short words and a hot/cold queue engine for longer words, significantly improving tokenization speed. This change introduces a new \PipelineBPE\ model, a word cache for repeated tokens, and a new serialization format (\BpeConfig\) for loading and saving models, while removing the legacy encode engine.
tokenizers/tk-encode/src/models/bpe · high confidence
New optimized vocabulary storage and added-token handling in tk-encode
The \tk-encode\ tokenizer engine now uses a new vocabulary backend located in \tokenizers/tk-encode/src/vocab\. This introduces \BucketVocabStore\, which replaces standard hash maps with a minimal perfect hash function (MPHF) and packed keys to reduce memory usage and improve lookup speed for token IDs. It also adds \AddedVocabulary\ and \AddedToken\ structures, allowing users to append new tokens to an existing trained model with configurable behaviors such as \single\_word\ matching, whitespace stripping (\lstrip\/\rstrip\), and normalization. The implementation leverages SIMD instructions (via \pshufb\) and multi-bucket prefix matching to accelerate byte-level token matching.
tokenizers/tk-encode/src/vocab · high confidence
New rc0 release with read-only pipeline tokenizer and legacy config upgrade
The tokenizers library has been restructured into separate crates (\tk\_encode\, \tk\_serialize\, \tk\_convert\) with this release providing a read-only \PipelineTokenizer\ that encodes and decodes based on a \tokenizer.json\ configuration. This version does not yet support building tokenizers programmatically, saving, or training; it focuses on inference and includes a legacy config upgrade pass (\tk\_convert\) to rewrite older tokenizer configurations into the canonical form required by the new reader.
tokenizers/src · high confidence
New tokenization engine with redesigned Encoding and Pattern APIs
The \tk-encode\ crate introduces a new tokenization engine, replacing the legacy encode engine and its serialization layer. This change brings a redesigned \Encoding\ struct that explicitly manages token IDs, type IDs, tokens, word IDs, offsets, special token masks, attention masks, and sequence ranges, ensuring compatibility with existing Python pickle formats. It also introduces a new \Pattern\ trait for pre-tokenization, supporting splitting via literals (offloaded to the \atomsplit\ module for performance), system regexes, and character predicates, along with a \SplitDelimiterBehavior\ enum to control how delimiters are handled during splitting.
tokenizers/tk-encode/src/tokenizer · high confidence
New tokenizer normalizer implementations in tk-encode
The \tokenizers/tk-encode/src/normalizers\ module now contains fresh, standalone implementations for the core text normalizers used in the encoding pipeline. This includes \BertNormalizer\ (handling text cleaning, Chinese character spacing, accent stripping, and lowercasing), \ByteLevel\ (converting bytes to their character representations), \MetaspaceNormalizer\ (inserting the SentencePiece delimiter and supporting prepend behaviors), \PrecompiledNormalizer\ (applying precompiled character maps), \Prepend\ (adding a prefix string), \Replace\ (pattern-based literal or regex replacement), \Strip\ (trimming whitespace), \StripAccents\ (removing combining marks), \Lowercase\, and Unicode normalization forms (NFC, NFD, NFKC, NFKD) plus the NMT normalizer. These modules replace the legacy normalizer logic, providing the specific text transformation capabilities required by models like BERT, T5, and Albert within the new encode path.
tokenizers/tk-encode/src/normalizers · high confidence
Optimized BPE encoding with character folding shortcuts
The BPE tokenizer now uses a precomputed 'fold' table to accelerate encoding. For characters that are guaranteed to merge into a single token without interference from neighbors, the encoder skips the standard merge loop and emits the token directly. This optimization significantly reduces processing time for common characters while maintaining exact encoding correctness.
tokenizers/tk-encode/src/models/bpe/fold · high confidence
Python bindings rework: new Tokenizer API with Encoding views and padding/truncation options
The Python bindings have been reworked to expose a modern Tokenizer API. Users can now load tokenizers from local files or directly from the Hugging Face Hub via \Tokenizer.from\_pretrained\. The \Encoding\ object provides \ids\, \type\_ids\, and \attention\mask\ as both Python lists and read-only NumPy array views (\\\_array\), allowing zero-copy access for downstream libraries like PyTorch. Padding and truncation are now controlled via explicit \Padding\ and \Truncation\ objects, supporting options like \pad\_to\_multiple\_of\, \direction\, and \strategy\, which can be set globally on the tokenizer or passed per-call. The bindings also support pickling for multiprocessing and include auto-generated \.pyi\ stubs for type checking. Also in this area: 3 commits (0 fixes) modifying bindings, bindings/python; Unified tokenizer crate dependency for Node and Python bindings; Support encoding pairs of texts in Python bindings; Python bindings now return copies instead of views for token arrays.
bindings/python, bindings/python/src · medium confidence · unverified
Simplified processor module structure
The processor module in the encode path has been streamlined to re-export only the byte-level pre-tokenizer, while complex post-processing logic (including the PostProcessorWrapper and Sequence processor) has been moved to the tk-convert crate. Users relying on the encode path will now interact with a cleaner API where post-processing is handled via the pipeline configuration rather than direct processor instantiation.
tokenizers/tk-encode/src/processors · medium confidence
Tokenizer training logic moved to a dedicated tk-train crate
The training components for tokenizers (BPE, Unigram, WordLevel, and WordPiece) have been extracted from the inference-focused tk-encode crate into a new tk-train crate. This separation ensures that the inference library no longer carries training-related dependencies or coupling. Users now drive training directly by feeding data to a Trainer instance and calling train, rather than using legacy tokenizer-level entry points. The crate also includes a new TrainerWrapper enum for dispatching across model types and a dedicated serde module for serializing AddedTokens during training. Also in this area: Feature-gated model training and parity trainer API changes; Improved error handling and duplicate detection in BPE vocabulary building.
tokenizers/tk-train · high confidence
Tokenizers library enters v1.0.0 release candidate with major architectural changes
The library has shifted from version 0.23 to a v1.0.0 release candidate, introducing a new pipeline-native architecture that strips the legacy configuration layer. This change removes the \tk-train\ crate from the default build, meaning training is currently unavailable, and replaces the old \Tokenizer\ object with a read-only \PipelineTokenizer\. Consequently, Python bindings can no longer modify tokenizer components (such as setting normalizers) or save modified tokenizers, and several legacy APIs like \Tokenizer::save\ and \from\_pretrained\ are deprecated in favor of new serialization and loading methods. The repository also adds a \CITATION.cff\ file for academic referencing and updates the \README\ to reflect the new installation and usage patterns.
(repo-wide) · high confidence
Tokenizers library restructured into modular crates with rc0 release
The tokenizers library has been restructured into distinct crates: \tk\_encode\ for inference engines, \tk\_serialize\ for reading tokenizer configurations, and \tk\_convert\ for upgrading legacy JSON files. This release (rc0) introduces a read-only \PipelineTokenizer\ that encodes and decodes text based on a \tokenizer.json\ file, but explicitly excludes the full \Tokenizer\ object model (such as \add\_tokens\, \save\, or training capabilities) which are deferred for v1. The change also includes a new Makefile for managing test data, benchmarks, and size profiling, along with a script to verify Python binding integrity.
tokenizers · high confidence
WASM binding package renamed to tokenizers-wasm and updated to use umbrella crate
The WebAssembly binding package has been renamed from \tokenizers-web\ to \tokenizers-wasm\. Users must update their npm installation command to \npm install tokenizers-wasm@latest\ and change their import statements from \tokenizers-web\ to \tokenizers-wasm\. Internally, the Rust implementation now relies on the \tokenizers\ umbrella crate instead of the previous \tk\\\ sub-crates, which may affect internal module paths but maintains the same public API surface for the tokenizer.
bindings/wasm · high confidence
Test coverage
17 commits adding/updating tests in bindings/python/tests; Added benchmark suite for tokenizer encode/decode performance; Added regression tests for tokenizer encoding parity.
Dependencies
Node.js bindings restructured with new Rust engine and platform packages
The Node.js bindings have been restructured to use a new Rust-based engine (\tk-encode\, \tk-serialize\, \tk-convert\) instead of the previous umbrella crate, exposing the pipeline encode path via N-API 3. The main \tokenizers\ package is now version 1.0.0-dev.0 and includes platform-specific sub-packages (e.g., \tokenizers-linux-x64-gnu\) for prebuilt binaries, all targeting Node.js \>= 10. The Rust dependencies have been updated to use \napi\ 3 and \pyo3\ 0.29 for the Python bindings, and the Node.js build now uses \napi-build\ 2.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 71 → 69 (-2.4)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 91 → 89 (-1.7)
- Architecture 97 → 98 (+0.1)
- Maturity 69 → 70 (+0.9)
- Readiness 66 → 66 (+0.1)
- Security 71 → 63 (-7.8)
- Performance 91 → 91 (+0.0)
Resolved (81)
- Change coupling: wordlevel.rs ↔ wordpiece.rs (tokenizers/tk-train/src/trainers/wordlevel.rs)
- Coverage not measured — JavaScript/TypeScript suite
- Duplicated block (10 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10 lines × 5) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (10–11 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–11 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–11 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–11 lines × 3) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–15 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11 lines × 3) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11–12 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11–12 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11–12 lines × 3) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (12–13 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (12–17 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (13 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (15 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (6 lines × 2) (tokenizers/bitcannon/src/models/cl100k.rs)
- Duplicated block (6 lines × 2) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- …and 61 more
New (50)
- Duplicated block (12 lines × 5) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (14 lines × 2) (tokenizers/tk-train/src/trainers/unigram.rs)
- Duplicated block (15 lines × 3) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (6 lines × 2) (tokenizers/bitcannon/src/models/cl100k.rs)
- Duplicated block (6 lines × 2) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (7 lines × 2) (tokenizers/tk-encode/src/models/unigram/lattice.rs)
- Edited copies of one member (3 members, 12+ corresponding lines) (tokenizers/tk-encode/src/pre_tokenizers/bert.rs)
- Edited copies of one member (3 members, 13+ corresponding lines) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Edited copies of one member (3 members, 19+ corresponding lines) (tokenizers/tk-encode/src/tokenizer/pattern.rs)
- Edited copies of one member (4 members, 11+ corresponding lines) (tokenizers/tk-encode/src/tokenizer/pipeline/mod.rs)
- Edited copy of a member (1 corresponding lines) (bindings/python/src/components/decoders.rs)
- Edited copy of a member (10 corresponding lines) (tokenizers/tk-encode/src/vocab/bucket_vocab_store.rs)
- Edited copy of a member (11 corresponding lines) (tokenizers/bitcannon/src/simd/x86.rs)
- Edited copy of a member (12 corresponding lines) (tokenizers/bitcannon/src/classify/avx.rs)
- Edited copy of a member (12 corresponding lines) (tokenizers/tk-encode/src/utils/unicode.rs)
- Edited copy of a member (12 corresponding lines) (tokenizers/tk-train/src/trainers/mod.rs)
- Edited copy of a member (14 corresponding lines) (tokenizers/tk-encode/src/models/wordlevel/mod.rs)
- Edited copy of a member (15 corresponding lines) (bindings/node/src/pipeline.rs)
- Edited copy of a member (19 corresponding lines) (tokenizers/bitcannon/src/classify/wasm.rs)
- Edited copy of a member (19 corresponding lines) (tokenizers/bitcannon/src/literal.rs)
- …and 30 more
Changes since last survey
- 33 commits — 22 feature/other, 11 fixes
By area
- tokenizers/tk-encode — 12 commits
- bindings/python — 10 commits
- .github/workflows — 5 commits
- (root) — 2 commits
- tokenizers/bitcannon — 2 commits
- bindings/node — 1 commit
- tokenizers/tk-convert — 1 commit
Notable commits
- fix: Fix Kimi pretokenization for overlapping Han categories (#2567)
- fix: fix(added-vocab): fix panic in single_word check (#2666)
- fix: fix(added-vocab): use splitmix64 as hash function for Mphf (#2667)
- fix: fix(bpe): fuse unk tokens on models with affixes (#2676)
- fix: fix(byte-level): drop unknown bytes + warn (#2550)
- fix: fix(convert): canonicalize singleton Metaspace sequences (#2591)
- fix: fix(parallelism): don't build a pool when the lock is contended (#2633)
- fix: fix(parallelism): register the fork handler with Once (#2632)
- fix: fix(tk-encode): update ptr_hash to 2.1.3, fix forked Tokenizer build (#2581)
- fix: fix(unigram): drop long lattice paths iteratively (#2658)
- fix: fix(wordpiece): encode vocab tokens that contain null byte (#2661)
- change: Load byte-fallback BPE vocabs that lack some <0xNN> codes (#2461)
- change: Mark generated tokenizers.pyi as linguist-generated (#2606)
- change: chore(deps): bump the actions group across 1 directory with 6 updates (#2659)
- change: chore(deps): bump the actions group with 5 updates (#2587)
- change: chore(python): drop support for python 3.10 (#2668)
- change: chroe: bump deps, centralize audit allowlist (#2609)
- change: ci(stale): never auto-close PRs (#2621)
- change: feat(composition): ComponentSequence (#2577)
- change: feat(composition): TokenizerSpec: JSON config as source of truth (#2562)
- …and 13 more
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
huggingface/tokenizers was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 39e28180be35af322d0abf08007d937641a5493b — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.