Skip to content
CAI
Software that uses CAICheck a score

huggingface/swift-transformers

68.5

Adequate · 15 November 2026

8.5k

lines of production code

Swift

primary language

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Added tokenizer performance benchmarks

A new benchmark suite has been added to measure tokenizer loading, encoding, and decoding performance. The benchmarks evaluate WordPiece, BPE, and Unigram tokenizers against short, medium, long, and code text inputs, reporting metrics such as wall-clock time, throughput, peak memory, and allocation counts.

Benchmarks · high confidence

Initial implementation of the Swift tokenizer pipeline

This change introduces the core tokenization engine for the library, adding the foundational components required to process text into tokens. It includes the \Tokenizer\ protocol and factory for loading models from Hugging Face Hub configurations, as well as specific implementations for BPE, Unigram, and WordPiece tokenization algorithms. The update also brings in the full preprocessing and postprocessing stack, including normalizers (e.g., Bert, NFD/NFC), pre-tokenizers (e.g., Whitespace, Bert, Metaspace), decoders (e.g., ByteLevel, WordPiece), and post-processors (e.g., BertProcessing, RobertaProcessing). Additionally, it provides the \TokenLattice\ structure for Viterbi decoding used by Unigram models and utility extensions for string splitting.

Sources/Tokenizers · high confidence

Introduce CoreML-based language model inference with Safetensors weight loading

This change introduces a new CoreML-backed inference path for language models, available on macOS 15.0+, iOS 18.0+, tvOS 18.0+, visionOS 2.0+, and watchOS 11.0+. It adds a \LanguageModel\ class and \LanguageModelProtocol\ that wrap CoreML models, handling context length management, token prediction, and optional prebuilt tokenizer injection. The \LanguageModel.loadCompiled\ factory method automatically detects stateful KV-cache support and returns the appropriate implementation. Additionally, a \Weights\ struct is added to load model parameters from Safetensors files (with GGUF and MLX formats recognized but currently unsupported), enabling the use of pre-trained models converted to CoreML format.

Sources/Models · high confidence

Introduce CoreML-based text generation with configurable sampling strategies

The Sources/Generation module now provides a complete text generation pipeline for CoreML models, supporting both greedy decoding and multinomial sampling. Users can control generation behavior via GenerationConfig, which applies logit processors for temperature scaling, top-k, top-p, min-p, and repetition penalty. The implementation uses a CPU-based inverse CDF search for sampling to ensure correctness with vocabularies larger than 65,536 tokens, and supports streaming output via callbacks.

Sources/Generation · high confidence

Introduce Hub module with high-performance JSON parsing and configuration loading

The Hub module now provides a new entry point for interacting with the Hugging Face Hub, featuring a \Config\ structure for handling model and tokenizer settings and a \YYJSONParser\ that uses the yyjson library for significantly faster JSON parsing compared to Foundation's \JSONSerialization\. This change includes a \BinaryDistinctString\ type to preserve exact byte sequences without Unicode normalization, which is critical for tokenizer vocabularies, and adds built-in fallback configurations for GPT-2 and T5 tokenizers to ensure robust loading when explicit configuration files are missing.

Sources/Hub · high confidence

New Mistral 7B example and CLI for Core ML text generation

Users can now export the Mistral 7B Instruct v0.3 model to Core ML and generate text using a new command-line interface. The \Examples/Mistral7B\ directory provides a Python script to export the model with a stateful key/value cache for efficient inference, while the \transformers-cli\ tool allows running generation with configurable parameters such as temperature, top-k, top-p, min-p, and repetition penalty, supporting both greedy and sampling decoding modes.

Examples · high confidence

New logits warpers for controlled text generation

Added a new \LogitsWarper\ module in \Sources/Generation/LogitsWarper\ that introduces several \LogitsProcessor\ implementations for CoreML-based generation: \TemperatureLogitsWarper\ for scaling randomness, \TopKLogitsWarper\ and \TopPLogitsWarper\ for nucleus/top-k filtering, \MinPLogitsWarper\ for min-p probability filtering, and \RepetitionPenaltyLogitsProcessor\ to discourage repeated tokens. These processors can be chained via \LogitsProcessorList\ to modify the token probability distribution during generation, giving users finer control over output style and diversity.

Sources/Generation/LogitsWarper · high confidence

Behavioural changes

Swift 6.1 support with strict concurrency and Xet download trait

The package now requires Swift 6.1 and enables strict concurrency checking to improve safety. It introduces an opt-in 'Xet' package trait that allows users to enable fast, parallel downloads from the Hugging Face Hub via swift-xet; when this trait is not enabled, the Hub falls back to the default URLSession-based transport. The project also adopts swift-format for code linting and formatting, and updates the Jinja dependency to version 2.4.2.

(repo-wide) · high confidence

Test coverage

Added benchmark test suite for tokenization performance; Added test coverage for Hub configuration parsing and API interactions; Added test coverage for tokenizer components; Added tests for Weights safetensors loading; Added tests for text generation and logits processing; Removed placeholder test file in tokenizers-tests.

Dependencies

Swift 5.9 upgrade and new library dependencies

The package now requires Swift 5.9 and introduces several new dependencies to support expanded functionality: swift-jinja (2.4.2) for template processing, swift-huggingface (0.8.1) for hub interactions, swift-collections (1.0.0) for data structures, swift-crypto (3.0.0–4.x) for security, and yyjson (0.12.0) for faster JSON parsing. Additionally, the minimum platform versions are set to iOS 16 and macOS 13.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 69 → 68 (-0.1)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 72 → 71 (-1.0)
  • Architecture 96 → 96 (-0.4)
  • Maturity 62 → 62 (-0.3)
  • Readiness 67 → 67 (-0.3)
  • Security 93 → 85 (-7.3)
  • Performance 82 → 94 (+12.2)

Resolved (4)

  • Duplicated block (10–11 lines × 2) (Sources/Generation/LogitsWarper/MinPLogitsWarper.swift)
  • Duplicated block (6 lines × 2) (Sources/Generation/LogitsWarper/RepetitionPenaltyLogitsProcessor.swift)
  • Duplicated block (8 lines × 2) (Sources/Models/LanguageModel.swift)
  • Hotspot: Sources/Tokenizers/Normalizer.swift (Sources/Tokenizers/Normalizer.swift)

New (20)

  • BinaryDistinctString.decodeUTF16 (cognitive 16) (Sources/Hub/BinaryDistinct.swift)
  • Edited copies of one member (3 members, 11+ corresponding lines) (Sources/Generation/LogitsWarper/MinPLogitsWarper.swift)
  • Edited copies of one member (3 members, 3+ corresponding lines) (Sources/Hub/HubApi.swift)
  • Edited copies of one member (4 members, 4+ corresponding lines) (Sources/Hub/HubApi.swift)
  • Edited copy of a member (10 corresponding lines) (Sources/Tokenizers/BertTokenizer.swift)
  • Edited copy of a member (10 corresponding lines) (Sources/Tokenizers/PreTokenizer.swift)
  • Edited copy of a member (11 corresponding lines) (Sources/Tokenizers/Tokenizer.swift)
  • Edited copy of a member (13 corresponding lines) (Sources/Tokenizers/Normalizer.swift)
  • Edited copy of a member (13 corresponding lines) (Sources/Tokenizers/Tokenizer.swift)
  • Edited copy of a member (14 corresponding lines) (Sources/Tokenizers/BertTokenizer.swift)
  • Edited copy of a member (16 corresponding lines) (Sources/Tokenizers/Decoder.swift)
  • Edited copy of a member (3 corresponding lines) (Sources/Hub/HubApi.swift)
  • Edited copy of a member (4 corresponding lines) (Sources/Hub/BinaryDistinct.swift)
  • Edited copy of a member (4 corresponding lines) (Sources/Hub/HubApi.swift)
  • Edited copy of a member (6 corresponding lines) (Sources/Generation/LogitsWarper/RepetitionPenaltyLogitsProcessor.swift)
  • Further sole-owners (lower concentration)
  • No assertions: missingVocabThrowsFirst (Tests/TokenizersTests/BPEMergesTests.swift)
  • Off-boarding risk: anonymized user #1
  • HubRepoIDCacheActor spells ID where 46 names in the project spell it Id (Sources/Hub/HubApi.swift)
  • resolveHubClientRepoID spells ID where 46 names in the project spell it Id (Sources/Hub/HubApi.swift)

Changes since last survey

  • 7 commits — 5 feature/other, 2 fixes

By area

  • Sources/Tokenizers — 4 commits
  • (root) — 2 commits
  • Tests/TokenizersTests — 1 commit

Notable commits

  • fix: Fix WordPieceDecoder crash on empty input and ByteFallbackDecoder trailing byte drop (#358)
  • fix: fix(Tokenizers): avoid quadratic regex matching for added tokens without whitespace padding (#386)
  • change: Add Murmur to projects that use swift-transformers (#422)
  • change: BPETokenizer: faster UTF-16 decoding and merge-rank construction at load (#419)
  • change: Fuse unknown tokens in linear time (#420)
  • change: Update README (#421)
  • change: Use BinaryDistinctString for consistent hashing in Linux and macOS (#418)

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

huggingface/swift-transformers was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 15 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit bd29696aa427a913812c7ef5cb03874633b04bfa — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-3c33c1882dc1.