Skip to content
CAI
Software that uses CAICheck a score

explosion/spaCy

59.4

Adequate · 15 November 2026

90.1k

lines of production code

Python

with JavaScript

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add 'xx' language ID for multi-language support

Introduced a new 'xx' language identifier and a corresponding MultiLanguage class in spacy.lang.xx, enabling models to explicitly support multiple languages. This change includes a new examples module with test sentences from various languages (German, English, Spanish, French, Italian, Dutch, Polish, Portuguese, Russian) to facilitate testing of these multi-lingual models.

spacy/lang/xx · high confidence

Add Amharic (am) language support

Introduces initial support for the Amharic language (ISO code 'am') in spaCy. This includes a new language class with configuration for left-to-right writing, tokenizer suffixes for Amharic punctuation, a stop-word list, and specific tokenizer exceptions for abbreviations like 'ት/ቤት' and 'ዓ.ም.'. It also provides a custom \like\_num\ attribute to recognize Amharic number words and ordinals, along with example sentences for testing.

spacy/lang/am · high confidence

Add Ancient Greek (grc) language support

Adds a new language module for Ancient Greek (ISO code 'grc'), enabling tokenization and basic linguistic processing for this language. The implementation includes a dedicated \Language\ class, tokenizer rules (prefixes, suffixes, infixes) that handle Ancient Greek-specific punctuation such as angle brackets and diaeresis, a list of stop words, lexical attributes for number recognition, and specific tokenizer exceptions to correctly normalize contracted forms and enclitics common in Ancient Greek texts.

spacy/lang/grc · high confidence

Add Arabic language support

Introduces a new Arabic language class (spacy.lang.ar) with tokenizer suffixes, stop words, and tokenizer exceptions for common abbreviations (e.g., 'د.', 'ق.م'). Includes lexical attributes to recognize Arabic number words and ordinals, and defines the writing system as right-to-left.

spacy/lang/ar · high confidence

Add Azerbaijani language support

Introduces a new Azerbaijani language module in spaCy, enabling tokenization and basic linguistic processing for the 'az' locale. This includes configuration for stop words, lexical attributes (such as number recognition), and example sentences for testing.

spacy/lang/az · high confidence

Add Gujarati language support

Introduces a new Gujarati language class (spacy.lang.gu) that enables spaCy to process Gujarati text. This includes a stop-words list for tokenization and a set of example sentences for testing the language model.

spacy/lang/gu · high confidence

Add Irish (ga) language support with lemmatizer and tokenizer rules

Users can now process text in Irish (ga) using spaCy. This change introduces the \Irish\ language class, which includes a custom \IrishLemmatizer\ based on the BuNaMo dataset to handle Irish-specific morphology (such as lenition and eclipsis) via POS-based lookups. It also provides a set of Irish stop words and specific tokenizer exceptions to correctly split common contractions and abbreviations.

spacy/lang/ga · high confidence

Add Kurdish Kurmanji language support

Added support for the Kurdish Kurmanji language (code 'kmr') to spaCy. This includes the language class definition, a set of stop words, lexical attribute getters for recognizing Kurmanji number words and ordinals, and example sentences for testing.

spacy/lang/kmr · high confidence

Add Kyrgyz (ky) language support

Introduces a new Kyrgyz language subclass in spaCy, providing essential tokenizer configuration including stop words, punctuation infixes, and specific tokenizer exceptions for abbreviations (such as days, months, and common suffixes). It also includes lexical attributes for recognizing numeric words and example sentences for validation.

spacy/lang/ky · high confidence

Add Latin language support

Added support for Latin (lang code 'la') in spaCy, including a new language class, tokenizer exceptions for common abbreviations and enclitics (e.g., 'mecum'), stop words, lexical attributes for recognizing Roman numerals and number words, and a custom noun chunking syntax iterator.

spacy/lang/la · high confidence

Add Ligurian (lij) language support

Added support for the Ligurian language (code 'lij') to spaCy. This includes a new language class with configuration for tokenizer exceptions, infixes, and stop words, along with example sentences for testing.

spacy/lang/lij · high confidence

Add Lithuanian language support

Adds a new Lithuanian language class (spacy.lang.lt) to spaCy, enabling tokenization and basic language processing for Lithuanian text. This includes configuration for tokenizer infixes, suffixes, and exceptions, as well as a set of stop words and lexical attribute getters specific to the language.

spacy/lang/lt · high confidence

Add Lower Sorbian language support

Users can now process text in Lower Sorbian (dsb) using spaCy. This change introduces the \LowerSorbian\ language class and its associated defaults, including a list of stop words, lexical attributes for number recognition (including ordinals), and example sentences for testing.

spacy/lang/dsb · high confidence

Add Luganda (lg) language support

Added a new language extension for Luganda (ISO code 'lg'), including tokenizer infixes, stop words, and lexical attributes for number recognition, along with example sentences for testing.

spacy/lang/lg · high confidence

Add Macedonian language support

Added the Macedonian language (mk) to spaCy, including a custom lemmatizer, tokenizer exceptions for abbreviations, stop words, and lexical attributes for number detection.

spacy/lang/mk · high confidence

Add Malayalam language support

Introduces a new language class for Malayalam (code 'ml'), enabling users to initialize a spaCy pipeline for this language. The implementation includes a stop-word list, example sentences for testing, and custom lexical attribute getters that allow the tokenizer to recognize Malayalam number words as numeric tokens.

spacy/lang/ml · high confidence

Add Marathi language support

Users can now initialize a spaCy pipeline for Marathi (language code 'mr') using the new Marathi class. This addition includes a default stop word list sourced from community repositories, enabling basic tokenization and stop-word filtering for Marathi text.

spacy/lang/mr · high confidence

Add Persian (Farsi) language support

Added support for the Persian language (code 'fa') to spaCy, enabling tokenization, stop-word filtering, and noun chunk detection. The implementation includes a rule-based lemmatizer with a comprehensive set of verb exceptions, tokenizer suffixes and exceptions tailored to Persian orthography (including Right-To-Left direction and Zero-Width Non-Joiner handling), and a custom \like\_num\ function for recognizing Persian numerals.

spacy/lang/fa · high confidence

Add Polish language support with lemmatizer and tokenizer

Introduces a new Polish language model (\spacy.lang.pl\) including a \Polish\ class, a \PolishLemmatizer\ that uses POS-based lookup tables from the Morfeusz dictionary, and tokenizer rules (prefixes, infixes, suffixes) and stop words. This enables users to initialize and process Polish text with spaCy, including lemmatization and tokenization capabilities.

spacy/lang/pl · high confidence

Add Slovak language support

Added initial support for the Slovak language (code 'sk') in spaCy, including language defaults, stop words, and lexical attributes for number recognition.

spacy/lang/sk · high confidence

Add Slovenian language support

Added the Slovenian language module (code 'sl') to spaCy, providing the base language class and essential linguistic data. This includes a stop words list, tokenizer rules (prefixes, suffixes, infixes), and exceptions for common abbreviations and company types. The module also defines lexical attributes for recognizing numbers, ordinals, and currencies, along with example sentences for testing.

spacy/lang/sl · high confidence

Add Tamil (ta) language support

Introduces initial support for the Tamil language in spaCy. This includes the core language class configuration, a set of stop words, lexical attribute definitions for recognizing Tamil numerals, and example sentences for testing.

spacy/lang/ta · high confidence

Add Tatar language support

Introduces initial support for the Tatar language (code 'tt') in spaCy. This includes a new language class with configuration for tokenization (infixes, exceptions, and stop words), a custom \like\_num\ function to recognize Tatar number words and prefixed numerals, and a set of example sentences for testing.

spacy/lang/tt · high confidence

Add Telugu language support

Added initial support for the Telugu language (code 'te') in spaCy. This includes the core language class, a set of Telugu stop words, lexical attributes for recognizing numbers, and example sentences for testing.

spacy/lang/te · high confidence

Add Tibetan (bo) language support

Added support for the Tibetan language (code 'bo') to spaCy. This includes the core language class configuration, a list of stop words sourced from Zenodo, lexical attributes for recognizing Tibetan numerals, and example sentences for testing.

spacy/lang/bo · high confidence

Add Tigrinya language support

Added support for the Tigrinya language (code 'ti') to spaCy, enabling tokenization and basic language-specific rules. This includes configuration for tokenizer suffixes and exceptions, a list of stop words, lexical attributes for recognizing numbers and ordinals, and example sentences for testing.

spacy/lang/ti · high confidence

Add Ukrainian language support

Introduced a new Ukrainian language model (\spacy.lang.uk\) that includes a lemmatizer (defaulting to \pymorphy3\ mode), tokenizer exceptions for common abbreviations, stop words, and lexical attributes for number recognition.

spacy/lang/uk · high confidence

Add Upper Sorbian language support

New files have been added to the spacy/lang/hsb module to enable support for Upper Sorbian (hsb). This includes the language class definition, lex attributes for number recognition, a stop word list, tokenizer exceptions, and example sentences for testing.

spacy/lang/hsb · high confidence

Add Urdu language support

Added initial support for the Urdu language (ur) to spaCy, including tokenizer suffixes, stop words, and lexical attributes such as number recognition (like\_num) for Urdu numerals and ordinals. This enables basic tokenization and text processing capabilities for Urdu text.

spacy/lang/ur · high confidence

Add Vietnamese language support

Introduces a new Vietnamese language model (\spacy.lang.vi\) that includes a custom tokenizer leveraging the external \pyvi\ library for word segmentation, along with default stop words, lexeme attributes (such as number recognition), and example sentences for testing.

spacy/lang/vi · high confidence

Add base language support for Afrikaans, Estonian, Icelandic, and Latvian

Users can now initialize spaCy pipelines for Afrikaans (af), Estonian (et), Icelandic (is), and Latvian (lv). This change introduces the base language classes and default stop-word lists for these four languages, enabling basic tokenization and stop-word filtering capabilities.

(repo-wide) · high confidence

Add initial Albanian language support

Introduces the Albanian (sq) language module, providing the base language class, a set of Albanian stop words, and example sentences for testing.

spacy/lang/sq · high confidence

Add initial Armenian (hy) language support

Introduces the Armenian language module to spaCy, enabling users to initialize an Armenian pipeline via \spacy.load('hy\_core\_web\_sm')\ or similar. This change adds the core language data structure (\Armenian\ class), a set of Armenian stop words for tokenization filtering, and lexical attributes that allow the tokenizer to recognize Armenian number words (e.g., 'մեկ', 'երկու') as numeric entities.

spacy/lang/hy · high confidence

Add initial Czech (cs) language support

Introduces support for the Czech language in spaCy by adding the \spacy.lang.cs\ module. This includes the \Czech\ language class, a list of Czech stop words, lexical attribute definitions (such as number recognition), and example sentences for testing.

spacy/lang/cs · high confidence

Add initial Kannada language support

Introduces basic support for Kannada (language code 'kn') in spaCy. This includes a new language class with default configurations, a set of Kannada stop words for tokenization and filtering, and a collection of example sentences for testing purposes.

spacy/lang/kn · high confidence

Add initial Malay (ms) language support

Introduces a new language module for Malay, enabling tokenization, stop-word filtering, and syntax iteration. The implementation includes custom tokenizer rules for prefixes, suffixes, and infixes (handling currency, units, and HTML tags), a list of stop words, and specific tokenizer exceptions for common abbreviations and reduplicated words. It also provides a noun-chunk iterator and example sentences for testing.

spacy/lang/ms · high confidence

Add initial Setswana (tn) language support

Added the Setswana language module to spaCy, enabling tokenization and basic language-specific defaults for the 'tn' locale. This includes configuration for tokenizer infixes, a list of stop words, lexical attribute getters for recognizing numbers and ordinals, and example sentences for testing.

spacy/lang/tn · high confidence

Add initial Sinhala language support

Users can now initialize a spaCy pipeline for Sinhala (code 'si') using the new \Sinhala\ class. This addition includes basic language data such as a stop-word list, lexical attribute getters for number recognition, and a set of example sentences for testing.

spacy/lang/si · high confidence

Add initial Yoruba language support

Introduces the Yoruba language (\yo\) to spaCy by adding the \spacy/lang/yo\ module. This includes the \Yoruba\ language class, a set of stop words, lexical attributes for number detection, and example sentences for testing.

spacy/lang/yo · high confidence

Add initial support for the Nepali language

Introduces the Nepali language module (code 'ne') to spaCy, enabling users to initialize a Nepali pipeline via \spacy.load('ne\_core\news\\*')\. This addition includes core language data such as a comprehensive list of stop words, lexical attribute getters for normalization and number detection, and a set of example sentences for testing.

spacy/lang/ne · high confidence

Add support for Sanskrit language

Users can now process Sanskrit text using the 'sa' language code. This change introduces the core language data for Sanskrit, including a custom tokenizer configuration for recognizing Sanskrit numerals, a comprehensive list of stop words, and example sentences for testing.

spacy/lang/sa · high confidence

Added Haitian Creole (ht) language support

Users can now process Haitian Creole text using spaCy by instantiating the new \HaitianCreole\ class. This addition includes a dedicated tokenizer with rules for Creole-specific prefixes, suffixes, and infixes (such as apostrophe contractions), a lemmatizer with rule-based and lookup capabilities, noun chunk detection, stop words, and normalization maps for handling common contractions and number formats.

spacy/lang/ht · high confidence

Added NER and text classification example datasets

New example data files have been added to support Named Entity Recognition (NER) and text classification (textcat) training workflows. The NER dataset includes IOB and JSON formats for both sentence-per-line and token-per-line annotations, along with documentation on converting these files using the \spacy convert\ CLI for spaCy v2 and v3. The textcat dataset provides JSON training examples for mutually exclusive and multi-label classification tasks, sourced from Cooking Stack Exchange and the Jigsaw Toxic Comments Challenge, with associated license files (CC0, CC BY-SA 3.0, CC BY-SA 4.0) included for compliance.

_extra/example\data · high confidence

Added Norwegian Nynorsk (nn) and Faroese (fo) language support

Introduced new language extensions for Norwegian Nynorsk and Faroese, enabling tokenization and normalization for these languages. The Norwegian Nynorsk module includes specific tokenizer rules for prefixes, infixes, and suffixes, along with exceptions for common abbreviations and month names, and leverages syntax iterators from the Norwegian Bokmål (nb) module. The Faroese module provides tokenizer exceptions for a comprehensive list of abbreviations. Example sentences are also included for the Norwegian Nynorsk module to facilitate testing.

spacy/lang/nn · high confidence

Added Scottish Gaelic (gd) language support

SpaCy now supports the Scottish Gaelic language (code 'gd'). This change introduces a new language module including a \Scottish\ class, a list of stop words, and specific tokenizer exceptions for handling Gaelic orthographic conventions (such as lenition and initial mutations) based on the Gaelic Orthographic Conventions of 2009.

spacy/lang/gd · high confidence

Added quickstart widget generation script

A new Python script (jinja\_to\_js.py) and shell wrapper (setup.sh) have been added to the website setup directory to generate the JavaScript code for the training quickstart widget from Jinja2 templates and YAML configuration files.

website/setup · high confidence

Added release automation scripts

New shell scripts have been added to the bin directory to streamline the release process. get-package.sh and get-version.sh extract the package name and version from spacy/about.py, while push-tag.sh and release.sh automate the creation and publishing of git tags for releases, ensuring the repository is clean before proceeding.

bin · high confidence

Basque language support added to spaCy

Added the Basque (eu) language model to spaCy, including tokenizer suffixes, a stop-word list, and lexical attributes for recognizing numbers and ordinals. This enables users to initialize and process text in Basque using the standard spaCy API.

spacy/lang/eu · high confidence

Initial Bengali (bn) language support

Adds the Bengali language model to spaCy, enabling tokenization, stop-word filtering, and rule-based lemmatization for the 'bn' locale. The implementation includes specific tokenizer rules for Bengali punctuation and digits, a set of Bengali stop words, and tokenizer exceptions for common abbreviations (e.g., 'ডঃ' to 'ডক্টর').

spacy/lang/bn · high confidence

Initial Bulgarian language support added

This change introduces the \Bulgarian\ language class to spaCy, enabling tokenization and processing for Bulgarian text. It includes configuration for stop words, tokenizer exceptions for common abbreviations (such as titles, measurements, and academic fields), and lexicographical attributes to recognize number words. Example sentences are also provided for testing.

spacy/lang/bg · high confidence

Initial Croatian language support added

Added the Croatian (hr) language module to spaCy, providing the core language class, a set of stop words, and example sentences for testing. This enables users to initialize a Croatian pipeline and perform basic tokenization and processing tasks.

spacy/lang/hr · high confidence

Initial Danish language support added to spaCy

This change introduces the Danish (da) language model to spaCy, enabling tokenization, stop-word filtering, and syntax iteration for Danish text. The implementation includes specific tokenizer exceptions for Danish abbreviations (such as months, weekdays, and common titles), a custom \like\_num\ function to recognize Danish numeric words and ordinals, and a noun-chunk iterator tailored to Danish syntax rules. Example sentences are also provided for testing.

spacy/lang/da · high confidence

Initial Finnish language support with tokenizer and noun chunker

Adds the Finnish language module (spacy.lang.fi), providing the foundational language data required for processing Finnish text. This includes a tokenizer with specific infixes, suffixes, and exception rules for common abbreviations and conjunction contractions (e.g., 'ett' + 'että'), a stop-word list, and lexical attributes for recognizing numbers. It also introduces a noun-chunking syntax iterator tailored to Finnish dependency structures, enabling users to extract base noun phrases from dependency parses.

spacy/lang/fi · high confidence

Initial Hebrew language support added

Added the Hebrew language module to spaCy, enabling users to initialize and process Hebrew text. This includes configuration for right-to-left writing direction, a list of Hebrew stop words, lexical attributes for recognizing numbers and ordinals, and a set of example sentences for testing.

spacy/lang/he · high confidence

Initial Indonesian language support added to spaCy

This change introduces the Indonesian (id) language model to spaCy, enabling tokenization, stop-word filtering, and noun chunk detection for Indonesian text. The implementation includes specific tokenizer rules for Indonesian punctuation and currency, a list of common Indonesian stop words, and logic to recognize Indonesian number words and abbreviations (such as titles and month names) during tokenization.

spacy/lang/id · high confidence

Initial Korean language support

Added the Korean language module (spacy.lang.ko), enabling tokenization, part-of-speech tagging, and lemmatization for Korean text. The implementation uses the mecab-ko library via natto-py to parse text into tokens, mapping MeCab's specific POS tags to Universal Dependencies tags via a provided tag map. The module includes default configurations for stop words, punctuation infixes, and number detection, along with example sentences for testing.

spacy/lang/ko · high confidence

Initial Romanian language support

Added the Romanian language module to spaCy, enabling tokenization, stop-word filtering, and basic lexical attribute detection (such as identifying numbers and ordinals) for text in Romanian. This includes configuration for tokenizer prefixes, suffixes, and infixes tailored to Romanian orthography, as well as a set of example sentences for testing.

spacy/lang/ro · high confidence

Initial Serbian language support added

This change introduces the Serbian language module (code 'sr') to spaCy, enabling tokenization, stop-word filtering, and lemmatization for Serbian text. The implementation includes specific tokenizer exceptions for common abbreviations (such as days, months, and titles like 'др' for doctor), a list of stop words, and rules for recognizing number words. It also provides example sentences for testing and includes the necessary license attribution for the underlying lemmatization data.

spacy/lang/sr · high confidence

Initial Swedish language support with rule-based lemmatizer and noun chunks

This change introduces the Swedish (sv) language module to spaCy, providing the foundational data and components required for processing Swedish text. It includes a rule-based lemmatizer configured as the default, which allows for lemmatization without requiring a trained model. The module also adds support for noun chunk detection via syntax iterators, enabling the identification of base noun phrases. Additionally, it provides Swedish-specific tokenizer exceptions for common abbreviations (such as months and weekdays), stop words, punctuation rules, and number recognition logic, along with example sentences for testing.

spacy/lang/sv · high confidence

Initial Thai language support with PyThaiNLP tokenizer

Added the Thai language model to spaCy, introducing a custom \ThaiTokenizer\ that relies on the external PyThaiNLP library for word segmentation. This release includes the core language data components: a comprehensive list of Thai stop words, lexeme attributes for recognizing Thai number words, and specific tokenizer exceptions for Thai government agencies, academic institutions, and military ranks to ensure accurate tokenization of these entities.

spacy/lang/th · high confidence

Initial Turkish language support added to spaCy

This change introduces the Turkish (tr) language model to spaCy, enabling tokenization, lemmatization, and syntactic analysis for Turkish text. The implementation includes a custom tokenizer with rules for handling Turkish-specific abbreviations (such as military ranks and academic titles) and number formats, a stop-word list, and a noun-chunk iterator tailored to Turkish dependency structures. It also provides example sentences for testing and configuration defaults for the Turkish pipeline.

spacy/lang/tr · high confidence

Initial support for Catalan language processing

This change introduces the first official support for the Catalan language (code 'ca') in spaCy. It adds a complete language module including a rule-based lemmatizer, tokenizer configuration (prefixes, suffixes, infixes, and exceptions for contractions like 'del' and 'al'), stop words, and syntax iterators for noun chunks. Users can now initialize a Catalan pipeline using \spacy.blank('ca')\ or load pre-trained models for this language.

spacy/lang/ca · high confidence

Initial support for Greek language processing

Added the Greek language module (spacy/lang/el), providing a tokenizer with language-specific prefixes, suffixes, and infixes, a rule-based lemmatizer, stop words, noun chunk detection, and utilities for number recognition.

spacy/lang/el · high confidence

Initial support for Hindi language processing

Added the Hindi language module to spaCy, enabling tokenization and basic linguistic analysis for Hindi text. This includes a language class with stop words, lexical attribute getters for normalizing stems and recognizing numbers (including prefixed and ordinal forms), and example sentences for testing.

spacy/lang/hi · high confidence

Initial support for Luxembourgish (lb)

Adds a new language module for Luxembourgish, enabling tokenization, stop-word filtering, and basic lexical attribute detection (such as number recognition) for the 'lb' locale. This includes configuration for tokenizer infixes and exceptions to handle common abbreviations and punctuation patterns specific to the language.

spacy/lang/lb · high confidence

Initial support for Norwegian Bokmål (nb)

Adds the Norwegian Bokmål language module, enabling tokenization, stop words, syntax iterators (including noun chunks), and a rule-based lemmatizer for the 'nb' language code. This includes example sentences for testing and specific tokenizer rules for abbreviations and punctuation.

spacy/lang/nb · high confidence

Initial support for Tagalog language

Added the Tagalog (tl) language model to spaCy, enabling tokenization, stop-word filtering, and number recognition for Tagalog text. This includes configuration for tokenizer exceptions to handle common contractions (such as 'y' and 'ng' clitics) and a custom rule to identify Tagalog number words as numeric entities.

spacy/lang/tl · high confidence

Introduce DependencyMatcher and fuzzy text matching in Matcher

The matcher module now includes a new DependencyMatcher class that allows matching patterns against dependency parse trees using relational operators (e.g., parent, child, sibling, precede/follow relationships). Additionally, the standard Matcher now supports fuzzy text matching via a configurable \fuzzy\_compare\ function (defaulting to a Levenshtein distance implementation), enabling approximate string matches in token patterns. These changes expand the rule-based matching capabilities to cover syntactic structure and tolerant string matching.

spacy/matcher · high confidence

Introduce Dutch language support with rule-based lemmatization and noun chunking

Adds the Dutch (nl) language module to spaCy, providing a complete set of language-specific resources including tokenizer rules, stop words, and syntax iterators. The module introduces a rule-based lemmatizer that uses lookup tables and exception lists to determine base forms, and implements noun chunk detection to identify base noun phrases in dependency parses. It also includes example sentences for testing and a \like\_num\ implementation to recognize numeric words and ordinals.

spacy/lang/nl · high confidence

Introduce KnowledgeBase and InMemoryLookupKB for entity linking

This change introduces the core components for Spacy's entity linking feature: the \KnowledgeBase\ abstract class and the \InMemoryLookupKB\ implementation, along with the \Candidate\ class and helper functions (\get\_candidates\, \get\_candidates\_batch\). Users can now link named entities to real-world concepts by adding entities and aliases to an in-memory knowledge base, which stores entity identifiers, textual aliases, prior probabilities, and vectors for disambiguation.

spacy/kb · high confidence

Introduce centralized website metadata and navigation configuration

The website now uses a dedicated \website/meta\ directory to manage site-wide configuration and navigation data. This includes \site.json\ for core site identity and settings, \sidebars.json\ defining the documentation structure, \languages.json\ for supported language models, and \universe.json\ for community projects. New JavaScript/TypeScript modules (\dynamicMeta.mjs\, \languageSorted.tsx\, \recordLanguages.tsx\, \recordSections.tsx\, \recordUniverse.tsx\, \sidebarFlat.tsx\) provide programmatic access to this data, enabling dynamic generation of navigation menus, language lists, and universe entries based on the current build context (e.g., nightly vs. stable).

website/meta · high confidence

Introduce displaCy span visualizer and dependency parser enhancements

The displaCy visualization module now includes a new 'span' style, allowing users to render named entity spans directly in text with support for overlapping spans and custom templates. The dependency parser ('dep' style) gains a 'fine\_grained' option to display specific POS tags instead of coarse-grained ones, an 'add\_lemma' option to show lemmas, and a 'collapse\_phrases' option to merge noun chunks. Additionally, the visualizer now supports custom entity colors via entry points, generates unique IDs for SVGs to prevent clashes, and ensures proper rendering in Jupyter notebooks by disabling MathJax processing for the output.

spacy/displacy · high confidence

Introduce new pipeline components and refactor component registration

This change introduces several new pipeline components to the library: AttributeRuler for setting token attributes via Matcher patterns, SpanRuler for span-based rule matching, EditTreeLemmatizer for lemmatization using predicted edit trees, and SpanFinder for identifying candidate spans. It also adds utility functions and components for retokenization (merge\_entities, merge\_noun\_chunks, merge\_subtokens), token splitting (TokenSplitter), and document cleaning (DocCleaner). Under the hood, the component factory registration system has been refactored to centralize registrations in factories.py, and a legacy EntityLinker\_v1 is provided for backwards compatibility.

spacy/pipeline · high confidence

Italian language support with POS-aware lemmatization and improved tokenization

Added the Italian language module (\spacy.lang.it\), introducing a new POS-aware lemmatizer that uses morphological lookup tables for accurate lemmatization, improved tokenizer rules for handling Italian-specific punctuation and elisions, noun chunk detection, and a comprehensive stop word list.

spacy/lang/it · high confidence

New machine learning layer registry in spacy.ml

The new spacy.ml package introduces a registry of machine learning layers and utilities, including CharacterEmbed, StaticVectors, FeatureExtractor, extract\_ngrams, extract\_spans, and PrecomputableAffine, alongside a C-based parser model implementation (parser\_model) and NVTX profiling callbacks for GPU performance tracing.

spacy/ml · high confidence

New reusable UI component library for the documentation site

The website now includes a comprehensive set of new React components in the \website/src/components\ directory, providing a unified design system for the documentation. This includes layout primitives like \Grid\, \Main\, and \Landing\ blocks, interactive elements such as \Accordion\, \Dropdown\, \Button\, and \Alert\, and specialized content renderers like \CodeBlock\ (with dynamic loading), \Embed\ (supporting YouTube, SoundCloud, and Google Sheets), and \Juniper\ (for executing Python code in Jupyter kernels). These components replace previous ad-hoc implementations, ensuring consistent styling, accessibility (e.g., ARIA attributes in \Accordion\ and \Icon\), and behavior across the site.

website/src/components · high confidence

New website widgets for changelog, features, languages, and quickstart guides

The website now includes a set of new React components in the widgets directory to enhance the user documentation experience. The Changelog widget dynamically fetches and displays stable and pre-release versions from the GitHub API. The Features widget presents key spaCy capabilities, such as multi-task learning and speed, by calculating counts from language metadata. The Languages widget provides a detailed table of supported languages, their codes, and associated model packages, along with a section for external tokenizer dependencies. Additionally, new Quickstart widgets (Install, Models, Training) offer interactive, configurable guides for installing spaCy, downloading models, and generating training configurations, while the Integration and Project widgets display partner logos and provide copyable commands for project templates.

website/src/widgets · high confidence

Portuguese language module initialization and configuration

The Portuguese language module (spacy/lang/pt) has been initialized with core language data, including tokenizer exceptions for common abbreviations (e.g., 'Dr.', 'etc.'), punctuation rules, stop words, and syntax iterators for noun chunk detection. It also provides lexical attributes for identifying numeric and ordinal words, along with example sentences for testing.

spacy/lang/pt · high confidence

Russian language support with pymorphy3 lemmatizer

This change introduces the Russian language module for spaCy, providing core NLP capabilities including tokenization, stop words, and lemmatization. The default lemmatizer mode is now pymorphy3, which replaces the previous pymorphy2 implementation, offering improved morphological analysis for Russian text. The module also includes expanded lists of number words, abbreviations, and tokenizer exceptions to enhance accuracy for common Russian linguistic patterns.

spacy/lang/ru · high confidence

Architecture

Introduce modular machine learning model builders in spacy.ml.models

The \spacy/ml/models\ package now provides a dedicated, modular structure for constructing neural network architectures, replacing the previous flat or monolithic organization. This change introduces specific builder functions for key pipeline components, including \build\_nel\_encoder\ for Entity Linking, \build\_tb\_parser\_model\ for dependency and NER parsing, \build\_spancat\_model\ for Span Categorization, \build\_tagger\_model\ for Part-of-Speech tagging, and various text classification models (CNN, BOW, and ensemble) in \textcat.py\. It also includes \build\_hash\_embed\_cnn\_tok2vec\ and \MultiHashEmbed\ for token-to-vector embeddings, as well as \multi\_task.py\ for pretraining objectives like masked language modeling. This refactoring isolates model construction logic, making it easier to customize, extend, or swap specific architectural components (such as the parser's lower/upper layers or the text classifier's output heads) without affecting the broader pipeline code.

spacy/ml/models · high confidence

Refactor spacy.tokens into a structured subpackage

The monolithic tokens module has been split into a dedicated \spacy/tokens\ subpackage, organizing core classes like \Doc\, \Token\, and \Span\ into their own files (\doc.pyx\, \token.pyx\, \span.pyx\). This change introduces new components for advanced functionality, including \MorphAnalysis\ for morphological features, \SpanGroup\ and \SpanGroups\ for managing arbitrary span annotations, and a \Retokenizer\ context manager for merging and splitting tokens. Serialization logic is now handled by a new \DocBin\ class in \\_serialize.py\, which uses a gzipped msgpack format for efficient storage of document annotations.

spacy/tokens · high confidence

Behavioural changes

Centralized global language data and tokenizer rules

The \spacy/lang\ module now provides a unified set of global configuration files for tokenization and lexical attributes. This includes \char\_classes.py\ for defining Unicode character ranges (e.g., Latin, CJK, Cyrillic), \lex\_attrs.py\ for global attribute functions like \like\_url\ and \like\_num\, \norm\_exceptions.py\ for normalizing punctuation and currency symbols, \punctuation.py\ for default tokenizer prefixes/suffixes/infices, and \tokenizer\_exceptions.py\ for base exceptions and URL patterns. This change consolidates language-specific logic into shared defaults that individual language models can extend or override.

spacy/lang · high confidence

Chinese language module restructured with configurable tokenizers and scoring

The Chinese language module has been reorganized to support multiple tokenization strategies (character-based, jieba, and pkuseg) via a new \ChineseTokenizer\ class and a standardized configuration system. Users can now select their preferred segmenter and, when using pkuseg, configure custom user dictionaries and models. The module also introduces tokenization scoring capabilities for evaluation and includes updated lexical attributes for number recognition and a comprehensive stop-word list.

spacy/lang/zh · high confidence

Converters now yield documents via generators

The training data converters (CoNLL NER, CoNLL-U, IOB, and JSON) have been refactored to return generators instead of lists. This means that when converting training files into spaCy Doc objects, the system now yields documents one by one rather than loading the entire dataset into memory at once, which improves memory efficiency for large training sets.

spacy/training/converters · high confidence

Expanded quickstart templates with new pipeline components and language-specific transformer recommendations

The quickstart configuration generator now supports additional pipeline components, including spancat, spancat\_singlelabel, trainable\_lemmatizer, entity\_linker, and span\_finder, allowing users to generate configs for these tasks directly. The template logic has been updated to omit tok2vec/transformer layers for TextCatBOW when efficiency is optimized, and to use core language models as vectors by default. Additionally, the recommendations file has been updated with specific transformer model names for a wider range of languages (e.g., Arabic, Bulgarian, Bangla, Catalan, Danish, etc.) to improve out-of-the-box performance for those locales.

spacy/cli/templates · high confidence

German language module restructured with improved tokenization and noun chunking

The German language module has been reorganized to provide more accurate tokenization and syntactic analysis. Tokenization now includes specific rules for German contractions (such as 'auf'm' expanding to 'auf dem') and a comprehensive list of abbreviation norms (e.g., 'z.B.' to 'zum Beispiel'). A new, language-specific noun chunk iterator has been added to better detect base noun phrases, addressing previous issues with overlapping chunks. Additionally, the module now includes a curated set of German stop words and example sentences for testing.

spacy/lang/de · high confidence

Hungarian tokenizer rules and data restructured

The Hungarian language module has been reorganized to improve tokenizer accuracy and maintainability. Tokenization rules (prefixes, suffixes, and infixes) are now defined in a dedicated \punctuation.py\ file, with specific adjustments such as excluding the degree symbol (°) from icon concatenation to preserve tokens like '99°' and removing the percent sign (%) from unit suffixes to fix percentage tokenization. A comprehensive list of tokenizer exceptions for abbreviations and special forms is provided in \tokenizer\_exceptions.py\, and stop words are managed in \stop\_words.py\. Additionally, example sentences for testing have been added to \examples.py\.

spacy/lang/hu · high confidence

Introduce Sudachi-based Japanese tokenizer with morphological analysis

The Japanese language module now uses a new tokenizer backed by SudachiPy, replacing previous implementations. This change introduces detailed morphological analysis on tokens, exposing Inflection and Reading data via the Token.morph extension, and adds support for configurable split modes (A, B, C) to control token granularity. The update also includes a comprehensive POS tag map, bigram-based part-of-speech resolution, noun chunk detection, and a curated set of Japanese stop words.

spacy/lang/ja · high confidence

Introduce rule-based lemmatizer and comprehensive language data for French

The French language module now includes a dedicated rule-based lemmatizer (\FrenchLemmatizer\) that handles lemmatization for nouns, verbs, adjectives, and other parts of speech using lookup tables and suffix-stripping rules, falling back to a lookup table for out-of-vocabulary words. This change also adds essential language data including a comprehensive list of tokenizer exceptions for hyphenated words and abbreviations, updated stop words, punctuation rules for French-specific elisions and hyphens, and a \like\_num\ implementation to recognize numeric words and ordinals. Additionally, example sentences and syntax iterators for noun chunks are provided to support testing and parsing.

spacy/lang/fr · high confidence

Migrate documentation website from Gatsby to Next.js

The website templates have been rewritten to use Next.js, replacing the previous Gatsby-based structure. This migration introduces a new Layout template that manages navigation, SEO, and alerts for nightly or legacy versions, alongside a dedicated Docs template that handles sidebar generation and content rendering. The Models template now dynamically loads components to reduce initial file size and supports label schemes in the model directory, while the Universe template has been updated to use Next.js image optimization and dynamic markdown rendering.

website/src/templates · high confidence

Migrate to Pydantic v2 and modernize build configuration

The project has fully migrated its data validation layer from Pydantic v1 to v2, requiring \pydantic\>=2.0.0\ and updating the \setup.cfg\ configuration to reflect this dependency. This change is accompanied by a shift in the build system: \setup.py\ now uses \setuptools\ and \sysconfig\ instead of \distutils\, and the project introduces a \.pre-commit-config.yaml\ that replaces the previous \black\/\flake8\ linting setup with \ruff\ for faster formatting and checking. Additionally, the \Makefile\ has been updated to use \pex\ for building isolated environments and testing, and \build-constraints.txt\ now pins \numpy\>=2.0.0\ to support the new build requirements.

(repo-wide) · high confidence

New Cython-based parser and NER internals

The parser and NER components have been rewritten in Cython to improve performance and maintainability. This change introduces a new internal module \spacy.pipeline.\_parser\_internals\ containing core state management (\\_state\, \stateclass\), transition systems (\transition\_system\, \arc\_eager\, \ner\), beam search utilities (\\_beam\_utils\), and projectivization logic (\nonproj\). The new implementation uses C-level data structures for faster state transitions and memory management, and includes specific optimizations such as cycle detection during projectivization and constant-time head lookups.

_spacy/pipeline/\_parser\internals · high confidence

New modular SASS design system for the documentation website

The website's styling has been refactored from a monolithic approach into a modular component-based system using SASS modules. This change introduces a comprehensive set of new stylesheets (e.g., \accordion.module.sass\, \alert.module.sass\, \code.module.sass\, \layout.sass\) that define the visual appearance of UI elements like navigation, sidebars, code blocks, and landing page cards. It also establishes a unified design token system in \layout.sass\ for colors, fonts, and spacing, and adds specific styles for features like Algolia DocSearch integration and responsive table scrolling.

website/src/styles · high confidence

Project CLI commands now delegate to Weasel

The \spacy project\ subcommands (assets, clone, document, dvc, pull, push, remote\_storage, and run) no longer contain their own implementation logic. Instead, they now import and re-export all functionality from the \weasel.cli\ module. This means that the behavior, arguments, and output of these project management commands are now determined by the Weasel library rather than spaCy's internal code.

spacy/cli/project · high confidence

Refactored English language data and lemmatizer into modular components

The English language module has been restructured to improve maintainability and performance. Language-specific data such as tokenizer exceptions, stop words, syntax iterators, and punctuation rules are now defined in separate, dedicated modules (e.g., \tokenizer\_exceptions.py\, \stop\_words.py\) rather than being embedded in the main language class. A new \EnglishLemmatizer\ component has been introduced, which uses a dedicated \is\_base\_form\ logic to optimize lemmatization by skipping tokens that are already in their base form. Additionally, the \lex\_attrs.py\ module now includes enhanced logic for detecting numeric words and ordinals, improving the accuracy of the \LIKE\_NUM\ attribute for English text.

spacy/lang/en · high confidence

Refactored training alignment and data augmentation infrastructure

The training module has been restructured to improve performance and flexibility. Alignment logic has been moved to optimized Cython implementations (align.pyx, alignment\_array.pyx) using a simplified ragged array type for faster indexing, and is now exposed via a new Python Alignment class. Data augmentation capabilities have been expanded with new combined, whitespace, and orth-variant augmenters in augment.py, allowing for more sophisticated training data generation. Additionally, the training loop and corpus readers have been updated to support these new augmentation strategies and improved alignment caching.

spacy/training · high confidence

Spanish language module restructured with rule-based lemmatizer and improved tokenization

The Spanish language module has been reorganized into a modular structure, introducing a new rule-based lemmatizer that uses morphological features to determine lemmas, improving accuracy for Spanish text. Tokenization has been enhanced with updated infixes, suffixes, and exception rules to better handle Spanish-specific patterns like abbreviations and time formats. Additionally, the module now includes example sentences for testing, a refined stop word list, and improved noun chunk detection logic to prevent overlapping phrases.

spacy/lang/es · high confidence

Updated system architecture diagram

The public-facing architecture diagram in the documentation has been updated to reflect the current system design, ensuring that visual representations of components and data flows match the actual implementation.

website/public · high confidence

Website migrated from Gatsby to Next.js

The spacy.io website has been rebuilt using Next.js, replacing the previous Gatsby-based architecture. This migration introduces a new project structure with Next-specific configuration files (next.config.mjs, tsconfig.json), a custom MDX plugin pipeline for processing documentation content, and support for server-side rendering and static generation. The site now uses Node 18, includes Prettier and ESLint configurations for code quality, and supports Docker-based local development. The migration also brings improved caching strategies via Netlify headers and a PWA service worker for better offline performance and bandwidth usage.

website · high confidence

Website pages migrated to Next.js

The website's page structure has been rewritten using Next.js, replacing the previous Gatsby-based implementation. This change introduces Next.js-specific routing patterns, including dynamic catch-all routes for documentation pages (\[...listPathPage\].tsx) and static generation for models and universe categories. The migration also includes a new custom \_app.tsx for global layout and analytics (Plausible), a \_document.tsx for HTML structure, and dedicated pages for the 404 error, landing, and specific content sections, all leveraging Next.js's getStaticPaths and getStaticProps for pre-rendering.

website/pages · high confidence

spaCy CLI rewritten with Typer and new subcommand structure

The command-line interface has been completely rewritten using the Typer framework, replacing the previous implementation. This introduces a new hierarchical command structure with dedicated subcommand groups for debugging (debug config, debug data, debug diff, debug model), benchmarking (benchmark speed), and initialization (init config, init pipeline). The rewrite also adds new capabilities such as the \spacy apply\ command for running inference on documents, the \spacy assemble\ command for building pipelines from config files, and the \spacy find-threshold\ command for tuning multi-label classifier thresholds. Additionally, the CLI now supports config overrides via environment variables and the \--opt=value\ syntax, and model symlinks are deprecated in favor of loading packages by full name or directory path.

spacy/cli · high confidence

Test coverage

Added German language test suite; Added Hungarian tokenizer tests; Added Italian language test suite; Added Serbian (sr) tokenizer and exception tests; Added comprehensive English tokenizer and language-specific test suite; Added comprehensive serialization test suite; Added comprehensive test suite for Danish language processing; Added comprehensive test suite for parser and NER internals; Added language-specific test suite for initialization, lemmatizers, and attributes; Added test coverage for Arabic tokenizer; Added test coverage for Catalan tokenizer; Added test coverage for Dutch language processing; Added test coverage for Finnish language processing; Added test coverage for Greek language tokenization; Added test coverage for Indonesian language tokenizer and lexer; Added test coverage for Korean language processing; Added test coverage for Latin language support; Added test coverage for Luxembourgish (lb) tokenizer; Added test coverage for Russian tokenizer, lemmatizer, and text attributes; Added test coverage for Slovenian language tokenization; Added test coverage for Spanish and Norwegian tokenization and noun chunking; Added test coverage for Swedish (sv) language processing; Added test coverage for Thai language tokenizer serialization and tokenization; Added test coverage for Tigrinya tokenizer; Added test coverage for Vietnamese tokenizer serialization and tokenization; Added test suite for Japanese language processing; Added test suite for Turkish language processing; Added test suite for Ukrainian language processing; Added test suite for the Matcher module; Added test suite for training components; Added test suite for vocab, vectors, and lookups components; Added test to verify consistency of package dependencies; Added tests for Ancient Greek (grc) tokenizer and text processing; Added tests for Armenian language tokenizer and lex attributes; Added tests for Basque language tokenizer; Added tests for Bulgarian tokenizer and text attributes; Added tests for Chinese tokenizer serialization, tokenization, and configuration; Added tests for Czech number-like token detection; Added tests for Gujarati tokenizer; Added tests for Haitian Creole (ht) language support; Added tests for Hebrew tokenizer and lexical attributes; Added tests for Irish (ga) tokenizer exception handling; Added tests for Kurdish Kurmanji language support; Added tests for Lithuanian tokenizer behavior; Added tests for Malayalam tokenizer; Added tests for Morphology feature conversion, memory usage, and serialization; Added tests for Nepali language tokenizer; Added tests for Persian noun chunking behavior; Added tests for Polish tokenizer and text attributes; Added tests for Sanskrit tokenizer behavior; Added tests for Tatar language tokenizer; Added tests for Urdu tokenizer behavior; Added tokenizer tests for Afrikaans, Croatian, Icelandic, Latvian, and Slovak; Added tokenizer tests for Albanian (sq) and multilingual (xx) languages; Added tokenizer tests for Bengali (bn); Added tokenizer tests for Faroese and Norwegian Nynorsk; Added tokenizer tests for Kyrgyz (ky); Added unit tests for French tokenizer and noun chunking; Comprehensive test suite for Doc, Span, and Token behavior; Comprehensive test suite for pipeline components; Expanded tokenizer test coverage for edge cases and debugging; Initial test suite structure and language-specific coverage.

Dependencies

485 commits updating dependencies (6 manifests)

A dependency / build maintenance change in (dependencies) — 485 commits (50 fixs), 6 files.

(dependencies) · high confidence · unverified

Migrate from Pydantic v1 to v2

The library has upgraded its data validation and serialization engine from Pydantic v1 to v2. This change requires users to have Pydantic version 2.0.0 or higher installed. As part of this migration, the internal configuration schemas and model metadata validation now use the Pydantic v2 API, which may affect custom components or scripts that rely on the previous Pydantic v1 internals.

spacy · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 58 → 59 (+1.4)
  • Rubric changed (rubric-2026.11.1 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 62 → 62 (-0.4)
  • Architecture 92 → 93 (+1.2)
  • Maturity 75 → 65 (-10.4)
  • Readiness 50 → 52 (+2.1)
  • Security 83 → 80 (-3.2)
  • Domain Modelling 100 → 100 (+0.0)
  • Accessibility 66 → 100 (+34.3)
  • Performance 60 → 60 (+0.0)

Resolved (11)

  • Duplicated block (13 lines × 2) (spacy/cli/debug_data.py)
  • Duplicated block (5 lines × 2) (spacy/pipeline/entity_linker.py)
  • Duplicated block (5 lines × 2) (spacy/pipeline/entity_linker.py)
  • Duplicated block (9–10 lines × 2) (spacy/lang/ga/lemmatizer.py)
  • Leaked secret: aws-access-key (website/meta/universe.json)
  • Leaked secret: hardcoded-credential (website/package-lock.json)
  • Leaked secret: hardcoded-credential (website/package-lock.json)
  • No ADRs found
  • Off-boarding risk: anonymized user #1
  • Projects may be oversized for their cohesion
  • Secret: aws-access-token (website/meta/universe.json)

New (35)

  • AWS access key ID with no secret key beside it (website/meta/universe.json)
  • AWS access key ID with no secret key beside it (website/meta/universe.json)
  • Edited copy of a member (10 corresponding lines) (spacy/pipeline/span_ruler.py)
  • Edited copy of a member (11 corresponding lines) (spacy/cli/init_pipeline.py)
  • Edited copy of a member (11 corresponding lines) (spacy/pipeline/attributeruler.py)
  • Edited copy of a member (13 corresponding lines) (spacy/lang/hi/lex_attrs.py)
  • Edited copy of a member (14 corresponding lines) (spacy/pipeline/lemmatizer.py)
  • Edited copy of a member (15 corresponding lines) (spacy/pipeline/_edit_tree_internals/schemas.py)
  • Edited copy of a member (15 corresponding lines) (spacy/pipeline/entityruler.py)
  • Edited copy of a member (17 corresponding lines) (spacy/displacy/render.py)
  • Edited copy of a member (17 corresponding lines) (spacy/ml/models/tok2vec.py)
  • Edited copy of a member (17 corresponding lines) (spacy/pipeline/attributeruler.py)
  • Edited copy of a member (18 corresponding lines) (spacy/pipeline/span_ruler.py)
  • Edited copy of a member (21 corresponding lines) (spacy/pipeline/span_ruler.py)
  • Edited copy of a member (22 corresponding lines) (spacy/displacy/render.py)
  • Edited copy of a member (24 corresponding lines) (spacy/util.py)
  • Edited copy of a member (25 corresponding lines) (spacy/lang/ru/lemmatizer.py)
  • Edited copy of a member (28 corresponding lines) (spacy/pipeline/spancat.py)
  • Edited copy of a member (3 corresponding lines) (spacy/lang/ja/init.py)
  • Edited copy of a member (3 corresponding lines) (spacy/lang/ja/init.py)
  • …and 15 more

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

explosion/spaCy was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 15 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 7ff14d4da84817f84c32b980882178ddacc066a1 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.73+preprod-52de90b74216.