Skip to content
CAI
Software that uses CAICheck a score

hankcs/HanLP

53.1

Adequate · 14 November 2026

50.5k

lines of production code

Python

primary language

6

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add Japanese multi-task learning demo

A new demo module for Japanese language processing has been added to the HanLP plugin. It utilizes a multi-task learning model (NPCMJ/UD/Kyoto) trained on Japanese corpora, providing capabilities for tokenization, part-of-speech tagging, constituency parsing, and other NLP tasks. Users can now run this demo to process Japanese text using the pre-trained BERT-based model.

_plugins/hanlp\_demo/hanlp\demo/ja · high confidence

Add TensorFlow-based Biaffine Dependency Parser

The parser component now includes a new TensorFlow implementation of the Biaffine Dependency Parser, providing an alternative backend to the existing PyTorch version. This addition introduces the \BiaffineDependencyParserTF\ class and associated model layers, algorithms (including Eisner and Chu-Liu/Edmonds decoding), and utility functions specifically for TensorFlow, enabling users to run dependency parsing tasks using the TensorFlow framework.

hanlp/components/parsers · high confidence

Add core evaluation metrics for classification and regression

The hanlp/metrics module now provides a suite of evaluation tools including CategoricalAccuracy (with Top-K and tie-breaking support), BooleanAccuracy, F1 score, Spearman correlation, and a MetricDict for multi-task learning. These components allow users to track model performance on classification tasks (accuracy, F1) and regression tasks (Spearman correlation) directly within the HanLP framework.

hanlp/metrics · high confidence

Add custom sparse categorical crossentropy loss functions

The \hanlp/losses\ module now includes three new loss functions: \SparseCategoricalCrossentropyOverNonzeroWeights\, \SparseCategoricalCrossentropyOverBatchFirstDim\, and \MaskedSparseCategoricalCrossentropyOverBatchFirstDim\. These provide specialized ways to compute sparse categorical crossentropy, including handling sample weights by summing over nonzero weights, averaging over the batch dimension, and masking specific values (default 0) in the true labels before computing the loss.

hanlp/losses · high confidence

Add language-specific utility modules for English, Japanese, and Chinese

This change introduces new utility packages under \hanlp/utils/lang\ to support specific languages. It adds an English tokenizer (\en/english\_tokenizer.py\) that provides regex-based word tokenization, including handling of contractions and possessives. It includes a Japanese tokenizer wrapper (\ja/bert\_tok.py\) that extends Hugging Face's \BertJapaneseTokenizer\ to fix offset mapping issues. Additionally, it adds Chinese utilities (\zh/char\_table.py\, \zh/localization.py\) for character normalization and mapping NLP task labels (POS, NER, dependency, etc.) to Chinese descriptions.

hanlp/utils/lang · high confidence

Add transformer-based Semantic Textual Similarity component

Introduces a new STS component that fine-tunes a transformer model (e.g., Electra) for sentence pair similarity. The implementation includes a \TransformerSemanticTextualSimilarity\ class with support for flat or list-of-pairs input data, training with Spearman correlation metrics, and prediction capabilities.

hanlp/components/sts · high confidence

Added CRF layers and file-reading utility

The library now includes Conditional Random Field (CRF) sequence-labeling layers for both PyTorch and TensorFlow backends, enabling users to perform sequence tagging tasks with Viterbi decoding and log-likelihood computation. Additionally, a new utility module for reading files backwards line-by-line has been added to support efficient log or large-file processing.

_hanlp/layers/crf, hanlp/utils/file\_read\backwards · high confidence

Added Chinese NLP training demos and Windows compatibility warnings

This update introduces new demonstration scripts for training Chinese NLP models within the HanLP plugin. It includes \finetune\_ner.py\ for fine-tuning Named Entity Recognition models, and \open\_base.py\ and \open\_small.py\ which demonstrate multi-task learning (tokenization, POS tagging, NER, SRL, dependency parsing, semantic dependency parsing, and constituency parsing) using Electra base and small backends. Additionally, a new \block\_windows.py\ module was added to explicitly prevent execution on Windows, issuing a warning that these training scripts are not supported on that platform.

_plugins/hanlp\_demo/hanlp\demo/zh/train · high confidence

Added demo scripts for Ancient Chinese (LZH) models

New demo scripts have been added to the HanLP plugin to demonstrate usage of the Ancient Chinese (LZH) models. Users can now run \demo\_tok.py\ to test tokenization using the \KYOTO\_EVAHAN\_TOK\_LZH\ model, and \demo\_mtl.py\ to test multi-task learning (tokenization, lemmatization, POS tagging, UDEP) using the \KYOTO\_EVAHAN\_TOK\_LEM\_POS\_UDEP\_LZH\ model, including examples of skipping specific tasks.

_plugins/hanlp\_demo/hanlp\demo/lzh · high confidence

Added multi-task learning wrappers for BIO and Ranking Semantic Role Labeling

New task classes, SpanBIOSemanticRoleLabeling and SpanRankingSemanticRoleLabeling, have been added to the multi-task learning (MTL) framework. These classes integrate the existing BIO-based and Span-Ranking-based SRL implementations into the MTL pipeline, providing standardized interfaces for data loading, loss computation, metric updates, and model building. This allows users to combine SRL tasks with other NLP tasks within a unified multi-task training setup.

hanlp/components/mtl/tasks/srl · high confidence

Initial release of TensorFlow-based data transformation components

This change introduces a new \hanlp/transform\ module providing a suite of data transformation classes built on TensorFlow. It includes \TransformerTokenizer\ for handling subword tokenization with support for sliding windows and special token configurations, alongside dataset-specific transforms for CoNLL dependency parsing, GLUE benchmark tasks (SST-2, MRPC), TACRED relation extraction, and generic text/tagging formats (TSV, TXT). A \PrependSpace\ utility is also added to handle whitespace normalization for token sequences.

hanlp/transform · high confidence

Initial release of hanlp\_common package

The hanlp\_common package is introduced as a new component providing common utilities and structures for HanLP. This release establishes the package foundation with version 0.0.23, including a README for installation and usage, and defines dependencies such as phrasetree and optional AMR tools (networkx, penman) for extended functionality.

_plugins/hanlp\common · high confidence

Initial release of the hanlp\_common plugin with core NLP utilities

This change introduces the new \hanlp\_common\ plugin, providing foundational classes and utilities for the HanLP ecosystem. It adds a \Document\ class for managing linguistic annotations (tokens, POS, NER, dependencies, and constituency trees) with support for CoNLL-U serialization and pretty-printing. The plugin includes robust visualization tools for dependency and constituency trees, AMR (Abstract Meaning Representation) graph handling via \penman\, and configuration management via \Configurable\ and \AutoConfigurable\ classes. It also provides I/O utilities for JSON and pickle, reflection helpers for classpath resolution, and constants for common NLP tokens.

_plugins/hanlp\_common/hanlp\common · high confidence

Initial repository structure and Python packaging configuration

The repository is initialized with standard project files including a comprehensive .gitignore, an Apache 2.0 LICENSE, a CITATION.cff file for academic referencing, and a README.md documenting the HanLP 2.1 NLP toolkit. The Python package is configured via setup.py, which defines dependencies such as PyTorch, Transformers, and SentencePiece, and specifies support for Python versions 3.6 through 3.10. Additionally, legacy Java-specific configuration files (HanLP.iml, config.yaml) are removed.

(repo-wide) · high confidence

Introduce Biaffine dependency and semantic dependency parsers

This change adds the Biaffine neural dependency parser component to HanLP, including the core biaffine scoring layer, MLP helpers, and a variational LSTM encoder. It provides two parser implementations: BiaffineDependencyParser for standard syntactic dependency parsing and BiaffineSemanticDependencyParser for semantic dependency parsing, both supporting transformer-based or traditional embeddings. The semantic parser adds constraints such as single-root enforcement and cycle avoidance, while the dependency parser exposes a conll output format option in predict.

hanlp/components/parsers/biaffine · high confidence

Introduce CRF Constituency Parser with TreeCRF decoding

Adds a new constituency parser component based on the FANCY (Fast and Accurate Neural CRF ConstituencY) model. This implementation uses a two-stage approach with Biaffine span and label prediction layers, integrated with a TreeCRF decoder for structured prediction and Minimum Bayes-Risk (MBR) decoding. The change introduces the \CRFConstituencyParser\ class, the \CRFConstituencyDecoder\ model, and the underlying \CRFConstituency\ TreeCRF module, enabling users to perform constituency parsing with this specific neural architecture.

hanlp/components/parsers/constituency · high confidence

Introduce HanLP RESTful API client with new NLP capabilities and configuration options

The \hanlp\_restful\ package is introduced, providing a Python client (\HanLPClient\) to interact with HanLP's RESTful APIs. This client supports a wide range of NLP tasks including text style transfer, abstract meaning representation, semantic textual similarity, language identification (176 languages), extractive and abstractive summarization, coreference resolution, grammatical error correction, and keyphrase extraction. Users can now specify a default language or per-request language (e.g., Japanese with coarse tokenization) and control SSL certificate verification for environments behind private TLS gateways. The package also allows anonymous usage of the RESTful API.

_plugins/hanlp\restful · high confidence

Introduce HanLP Trie plugin with dictionary-based tokenization

Adds the \hanlp\_trie\ plugin, providing a Trie-based data structure and dictionary classes (\TrieDict\, \TupleTrieDict\) for rule-based NLP tasks. This enables users to perform longest-prefix matching and batch tokenization using custom dictionaries, bridging the gap between statistical models and rule-based gazetteers. The implementation includes support for tuple keys, configurable serialization, and standard Python dict-like operations.

_plugins/hanlp\trie · high confidence

Introduce Masked Language Model component with special token filtering

A new MaskedLanguageModel component is added to the hanlp.components.lm package, enabling users to fill masked tokens in text using a pre-trained transformer model. The implementation includes a predict method that returns the top-k probable tokens for each mask. To ensure output quality, the model explicitly blocks special tokens (such as \[PAD\], \[CLS\], \[SEP\]) from being predicted as replacements for masked positions by setting their logits to negative infinity.

hanlp/components/lm · high confidence

Introduce SpanRanking Semantic Role Labeler component

A new Semantic Role Labeling (SRL) component based on the SpanRanking architecture has been added to the hanlp.components.srl.span\_rank module. This implementation jointly predicts predicates and arguments by generating candidate triples and ranking them, featuring custom PyTorch layers such as HighwayBiLSTM and StackedHighwayBiLSTM for context encoding, a SpanRankingSRLDecoder with biaffine scoring, and greedy/dynamic programming inference utilities for decoding non-overlapping spans.

_hanlp/components/srl/span\rank · high confidence

Introduce Transformer-based taggers with dual backend support

Added new Transformer-based tagging components for both PyTorch and TensorFlow backends, enabling users to perform sequence labeling tasks (such as NER and POS tagging) using transformer architectures. The PyTorch implementation includes support for CRF layers, extra embeddings, and knowledge distillation, while the TensorFlow implementation provides a Keras-based model with AdamW optimization. A custom metric class was added to handle padding mask values correctly during evaluation.

hanlp/components/taggers/transformers · high confidence

Introduce unified component, dataset, and transform abstractions for PyTorch and TensorFlow backends

The \hanlp/common\ module now provides a consistent API for building and running NLP models across both PyTorch and TensorFlow. Users can interact with models through the \Component\ base class, which exposes a standard \predict\ method and callable interface. The new \TorchComponent\ and \KerasComponent\ classes handle backend-specific workflows for loading, saving, and training models, including weight management and configuration persistence. Data processing is standardized via \TransformableDataset\ and \Transform\ classes, allowing users to apply configurable transformations to datasets and manage vocabularies through \Vocab\ and \VocabTF\ classes. This refactoring unifies the internal architecture, making it easier to switch between backends or extend existing components with new capabilities.

hanlp/common · high confidence

Introduction of AdamW optimizer with learning rate warmup

The \hanlp/optimizers\ module now includes a new \adamw\ subpackage providing an \AdamWeightDecay\ optimizer and a \WarmUp\ learning rate schedule. This addition allows users to apply L2 weight decay correctly with the Adam optimizer and supports a polynomial decay schedule with an initial warmup period, addressing compatibility and optimization strategy needs for TensorFlow 2.x models.

hanlp/optimizers · high confidence

Introduction of Multi-Task Learning (MTL) framework

The \hanlp/components/mtl\ module introduces a new Multi-Task Learning framework that allows sharing a single encoder across multiple task-specific decoders. This component supports defining dependencies between tasks, handling topological sorting for execution order, and managing data loading with configurable sampling weights (controlled by the \tau\ parameter) for training. It integrates with existing HanLP infrastructure via the \TorchComponent\ base class and provides utilities for building task-specific data loaders, caching, and prefetching.

hanlp/components/mtl · high confidence

Introduction of Multi-Task Learning (MTL) task components

This change introduces the foundational structure for HanLP's multi-task learning framework by adding the \hanlp/components/mtl/tasks\ module. It defines a base \Task\ class that standardizes training loops, dataloader construction, and metric updates across different NLP tasks. Specific task implementations are provided for dependency parsing (\dep\, \dep\_2nd\), semantic dependency parsing (\sdp\), constituency parsing (\constituency\), part-of-speech tagging (\pos\), lemmatization (\lem\), and Abstract Meaning Representation parsing (\amr\). These components enable users to configure and run multiple linguistic tasks within a unified multi-task learning environment.

hanlp/components/mtl/tasks · high confidence

Introduction of NER components with dictionary-based entity merging and gazetteer support

The NER module now includes dedicated components for RNN and Transformer backends (both TensorFlow and PyTorch). A key behavioral change is the introduction of whitelist/blacklist gazetteer support in the Transformer NER, which uses longest-prefix matching to override statistical predictions. Additionally, the system now supports merging consecutive entities of the same type (configurable via \merge\_types\) and handles various tagging schemes (IOBES, BIO, BIOUL) with automatic span conversion.

hanlp/components/ner · high confidence

Introduction of Transformer Encoder layer with model mirroring and logging suppression

The \hanlp/layers/transformers\ package now provides a dedicated Transformer encoder layer (\TransformerEncoder\) that wraps Hugging Face models, supporting features like subword averaging, scalar mix layer attention, and word dropout. To improve reliability and performance, the layer automatically mirrors model and tokenizer downloads via \resource.py\ for specific models (e.g., \xlm-roberta-base\, \ernie-gram\, \mMiniLMv2\) and suppresses noisy logs from the \transformers\ library by default. Additionally, the \pt\_imports\ module handles compatibility quirks, such as disabling tokenizer parallelism and fixing regex issues for XLM-R and mMiniLMv2 tokenizers.

hanlp/layers/transformers · high confidence

Introduction of the pretrained model registry

The \hanlp/pretrained\ package has been introduced to centralize the discovery and loading of pre-trained models. This change adds a structured registry (\\_\init\\_.py\) that exposes modules for various NLP tasks, including tokenization, POS tagging, NER, dependency parsing, semantic dependency parsing, constituency parsing, SRL, AMR, and word embeddings. It provides direct access to specific model weights (e.g., \MSRA\_NER\_BERT\_BASE\_ZH\, \CTB9\_DEP\_ELECTRA\_SMALL\) and embedding sources (e.g., GloVe, FastText, Word2Vec), allowing users to easily retrieve and use these resources without hardcoding URLs.

hanlp/pretrained · high confidence

New BART-based AMR parsing and generation components

Added a new AMR component under \hanlp/components/amr/amrbart\ that implements graph pre-training for Abstract Meaning Representation (AMR) parsing and generation. This introduces two new TorchComponents—\BART\_AMR\_Parser\ for converting text to AMR graphs and \BART\_AMR\_Generation\ for converting AMR graphs back to text—along with the necessary supporting infrastructure including a custom \AMRBartTokenizer\, BART model interface, dataset handling, and post-processing logic.

hanlp/components/amr · high confidence

New Biaffine NER component with PyTorch backend

This change introduces a new Named Entity Recognition component (\BiaffineNamedEntityRecognizer\) located in \hanlp/components/ner/biaffine\_ner\. The component implements the 'Named Entity Recognition as Dependency Parsing' approach, supporting both flat and nested NER tasks. It is built on PyTorch, utilizing a \BiaffineNamedEntityRecognitionModel\ and a decoder that employs sparse tensor operations (via \sparse\_coo\_tensor\) for efficient label handling. The implementation includes full training and evaluation loops, optimizer/scheduler construction (with support for transformer-based optimizers), and metric tracking (F1). This represents a new architectural addition rather than a modification of existing behavior.

_hanlp/components/ner/biaffine\ner · high confidence

New Chinese-language demo notebooks and scripts for HanLP tasks

Added a collection of Jupyter notebooks and Python scripts in the \plugins/hanlp\_demo/hanlp\_demo/zh\ directory demonstrating various HanLP capabilities for Chinese text processing. These include examples for abstractive summarization, Abstract Meaning Representation (AMR), text classification, constituency parsing, and coreference resolution, covering both RESTful API usage (\hanlp\_restful\) and local model loading (\hanlp\). The demos also illustrate advanced features such as multi-task learning (MTL) pipelines, custom dictionary integration for tokenization, and task removal from MTL models.

_plugins/hanlp\_demo/hanlp\demo/zh · high confidence

New English language demo scripts and pipeline examples

This update introduces a new set of demonstration scripts for the English language module within the HanLP plugin. Users can now explore various natural language processing capabilities through dedicated examples, including dependency parsing, part-of-speech tagging, named entity recognition, semantic dependency parsing, sentiment analysis, and language modeling. Additionally, a comprehensive pipeline example demonstrates how to chain tokenization, tagging, parsing, and semantic analysis steps, along with instructions for saving and loading the pipeline configuration for deployment.

_plugins/hanlp\_demo/hanlp\demo/en · high confidence

New Java RESTful client for HanLP APIs

This change introduces a new Java client library (\HanLPClient\) for interacting with HanLP's RESTful APIs. The client provides methods to parse text, sentences, and pre-tokenized inputs, as well as dedicated convenience methods for tokenization, semantic textual similarity, coreference resolution, keyphrase extraction, summarization (extractive and abstractive), grammatical error correction, language identification, sentiment analysis, and abstract meaning representation. It supports configuration via URL, authentication keys (including environment variables), and language settings, and includes data models for handling responses like coreference clusters and graph-based meaning representations.

_plugins/hanlp\_restful\java · high confidence

New MTL-integrated NER tasks with dictionary support and entity merging

The NER task module now includes two new implementations for multi-task learning: \BiaffineNamedEntityRecognition\, which uses a biaffine decoder to support both flat and nested NER, and \TaggingNamedEntityRecognition\, a linear/CRF-based tagger that supports conditional matching via custom dictionaries (\dict\_tags\), whitelists, and blacklists. Additionally, the tagging model introduces a \merge\_types\ parameter to allow consecutive entities of the same type to be merged during decoding.

hanlp/components/mtl/tasks/ner · high confidence

New NLP components for knowledge distillation, sentence boundary detection, and semantic role labeling

This release introduces several new model components to the HanLP library. The \distillation\ module adds support for knowledge distillation, including \DistillableComponent\ for training student models from teachers, along with various loss functions (MSE, cross-entropy, attention, hidden state, cosine, PKD, FSP) and temperature schedulers (constant, FLSP, CWSP). A new \NgramSentenceBoundaryDetector\ component is added for sentence splitting using n-gram features and RNNs. Additionally, new components for Semantic Role Labeling (SRL) are introduced, including an RNN-based tagging model with CRF support and metrics for evaluating against the official CoNLL-05 benchmark.

(repo-wide) · high confidence

New Universal Dependencies parser with lemmatization and dual backends

Adds a new Universal Dependencies parser component that performs lemmatization, part-of-speech tagging, morphological feature prediction, and dependency parsing. The implementation introduces a new \TagDecoder\ for sequence tagging tasks and a \UniversalDependenciesModel\ that shares a contextual word embedding encoder across all tasks. Lemmatization is handled via a rule-based system (adapted from UDPipe Future) that generates and applies edit scripts to transform word forms into lemmas, rather than predicting lemma tokens directly. The parser supports both PyTorch and TensorFlow backends and is designed to work with multiple languages.

hanlp/components/parsers/ud · high confidence

New character-level and contextual embedding layers for PyTorch and TensorFlow

The embeddings module now includes new implementations for character-level embeddings (CharCNN and CharRNN) and contextual string embeddings, available for both PyTorch and TensorFlow backends. Users can now leverage character-aware representations via convolutional or recurrent networks to capture subword information, as well as load pre-trained RNN language models for contextual string embeddings. Additionally, a new ContextualWordEmbedding layer provides access to transformer-based word embeddings, and a ConcatEmbedding layer allows combining multiple embedding sources into a single feature vector.

hanlp/layers/embeddings · high confidence

New chunking evaluation metrics with dual-backend support

The chunking metrics module now includes new evaluation classes: BinaryChunkingF1 for PyTorch-based binary tagging, and IOBES\_F1\_TF and BMES\_F1\_TF for TensorFlow-based sequence labeling. These metrics enable users to compute F1 scores for chunking tasks using either PyTorch or TensorFlow backends, with support for IOBES and BMES tagging schemes. The implementation includes utilities for entity extraction from tag sequences and span-based evaluation.

hanlp/metrics/chunking · high confidence

New classifier components for Hugging Face models, FastText, and multi-label support

The classifier component now includes new implementations for loading Hugging Face models directly (TransformerClassifierHF and TransformerRegressionHF), a FastText-based classifier (FastTextClassifier), and updated transformer classifiers with multi-label support in the TensorFlow backend. Users can now leverage pre-trained Hugging Face models for classification and regression tasks without manual conversion, use FastText for lightweight text classification, and handle multi-label classification scenarios in the TensorFlow transformer classifier.

hanlp/components/classifiers · high confidence

New dataset loaders and dataset definitions for multiple NLP tasks

This change introduces a comprehensive set of new dataset loaders and dataset definitions within the \hanlp/datasets\ package, covering tasks such as Named Entity Recognition (NER), Coreference Resolution, Sentence Boundary Detection (EOS), Language Modeling, and Parsing. Specific additions include loaders for CoNLL-12 coreference, JSON-lines and TSV NER formats, and utilities for converting Chinese Treebank (CTB) data into various parsing formats. It also adds dataset constants and definitions for standard corpora including CoNLL03, MSRA, Resume, Weibo, Stanford Sentiment Treebank (SST2), Microsoft Research Paraphrase Corpus (MRPC), and various language model datasets like PTB and SETimes, enabling users to easily access and train on these resources.

hanlp/datasets · high confidence

New multilingual NLP demo scripts and training utilities

This change introduces a new \plugins/hanlp\_demo/hanlp\_demo/mul\ directory containing demonstration scripts and training code for multilingual capabilities. Specifically, it adds \demo\_lid.py\ and \demo\_lid\_restful.py\ to showcase language identification for 176 languages using the LID\_176\_FASTTEXT\_BASE model, both via local loading and the RESTful API. It also includes \demo\_mtl.py\ to demonstrate multi-task learning (tokenization, POS, lemmatization, NER, SRL, dependency parsing) using the UD\_ONTONOTES\_TOK\_POS\_LEM\_FEA\_NER\_SRL\_DEP\_SDP\_CON\_XLMR\_BASE model, and \train/mul\_base.py\ which provides a script for training a custom multi-task model using the mMiniLMv2 transformer on Universal Dependencies data.

_plugins/hanlp\_demo/hanlp\demo/mul · high confidence

New neural network layer components for PyTorch and TensorFlow

The \hanlp/layers\ package now includes a suite of new building blocks for constructing neural networks. For PyTorch, this adds a \CnnEncoder\ for sentence classification using convolutional and max-pooling layers, several dropout variants (\WordDropout\, \SharedDropout\, \IndependentDropout\, \LockedDropout\) for regularization, a \FeedForward\ module for multi-layer perceptrons, a \ScalarMixWithDropout\ layer for weighted combinations of model layers (such as BERT outputs), and a \TimeDistributed\ wrapper to apply layers across time steps. For TensorFlow/Keras, a \WeightNormalization\ wrapper is introduced to reparameterize layer weights, decoupling magnitude and direction to accelerate training convergence.

hanlp/layers · high confidence

New parsing metrics and evaluation utilities

Added a new set of metrics and evaluation tools for syntactic parsing in the \hanlp/metrics/parsing\ module. This includes \AttachmentScore\ and \LabeledF1\ (with a TensorFlow variant \LabeledF1TF\) for computing Unlabeled/Labelled Attachment Scores and F1 metrics, \SpanMetric\ for span-based evaluation, and standalone evaluation scripts for CoNLL-X and semantic dependency parsing formats.

hanlp/metrics/parsing · high confidence

New pipeline, lemmatizer, and tagging components

This change introduces several new components to the HanLP library. The \Pipeline\ class allows users to chain multiple processing steps together, supporting custom functions via \LambdaComponent\ and providing methods for saving, loading, and copying the pipeline state. A new \TransformerLemmatizer\ is added, which uses a transformer-based tagger to generate lemma rules and applies heuristics to handle digits correctly. Additionally, an \NgramConvTagger\ is introduced for sequence tagging tasks, utilizing n-gram features and convolutional layers within a TensorFlow backend. These components expand the library's capabilities for building custom NLP pipelines and performing lemmatization and tagging with modern architectures.

hanlp/components · high confidence

New regression and tagging tokenization tasks for multi-task learning

The tokenization task module in the multi-task learning framework has been expanded with two new task implementations: \RegressionTokenization\ and \TaggingTokenization\. The regression task treats tokenization as a span-chunking problem, using a linear decoder and Binary Chunking F1 metric to predict token boundaries from transformer hidden states. The tagging task casts tokenization into a sequence labeling problem (BMES/BI schemes), supporting optional CRF decoding and custom dictionary integration via \dict\_force\ and \dict\_combine\ parameters. Both tasks integrate with the existing transformer-based tokenizer infrastructure and dataset loaders, enabling tokenization as a trainable component within multi-task pipelines.

hanlp/components/mtl/tasks/tok · high confidence

New span-based Semantic Role Labeling component with biaffine decoder

A new Semantic Role Labeling (SRL) component has been added to HanLP, implementing a span-based approach using the BIO tagging scheme. This component utilizes a biaffine attention mechanism to predict relations between predicates and tokens, supporting both standard cross-entropy loss and Conditional Random Fields (CRF) for sequence decoding. It includes a complete training loop with gradient accumulation, learning rate scheduling for transformer embeddings, and early stopping, allowing users to train and deploy SRL models directly.

_hanlp/components/srl/span\bio · high confidence

New tokenizer components and transformer-based tokenization support

This change introduces new tokenizer implementations for HanLP, including \RNNTokenizer\ and \MultiCriteriaTransformerTaggingTokenizer\ for PyTorch, and \NgramConvTokenizerTF\, \TransformerTokenizerTF\, and \RNNTokenizerTF\ for TensorFlow. It also adds \TransformerTaggingTokenizer\, which integrates high-priority (\dict\_force\) and low-priority (\dict\_combine\) dictionaries for longest-prefix-matching to handle edge cases, and supports configurable \output\_spans\ to return token positions. These components provide users with multiple backends (RNN, N-gram Conv, Transformer) and frameworks (PyTorch, TensorFlow) for text tokenization, with enhanced accuracy through dictionary integration and span output capabilities.

hanlp/components/tokenizers · high confidence

New utility modules and component loading infrastructure

This change introduces a new \hanlp.utils\ package and associated callback utilities that enhance how users load models and interact with the library. The \component\_util\ module adds \load\_from\_meta\_file\, enabling direct loading of components via \meta.json\ or \config.json\ files, including automatic migration of legacy TensorFlow classpaths to their new locations for HanLP v2.1+. It also provides improved error handling that detects version conflicts and guides users to upgrade or install missing dependencies. The \io\_util\ module adds robust file and download utilities, including \check\_version\_conflicts\ and \check\_outdated\ to warn users about incompatible package versions or outdated HanLP installations. Additionally, new callback utilities like \FineCSVLogger\ allow for detailed CSV logging of training metrics, and utility modules for string manipulation (\rules\, \string\_util\), span tagging (\span\_util\), and logging (\log\_util\) are added to support these features.

hanlp/utils · high confidence

Taggers now support conditional custom dictionary tags

The tagger components (both PyTorch and TensorFlow backends) now support applying custom dictionary-based tags via a \dict\_tags\ parameter. During prediction, if a \dict\_tags\ object is provided, the system tokenizes the input sentence using this dictionary and overwrites the model's predicted tags for any spans that match the dictionary entries. This allows users to enforce specific tag assignments based on custom rules or lexicons, overriding the model's default output for matched text.

hanlp/components/taggers · high confidence

TensorFlow-based Chinese NLP demos and training scripts

This location now provides a comprehensive set of TensorFlow-based examples and training scripts for Chinese NLP tasks. The demo scripts allow users to quickly load and run pre-trained models for word segmentation (CWS), part-of-speech tagging (POS), named entity recognition (NER), dependency parsing (DEP), semantic dependency parsing (SDP), and sentiment classification using models like BERT, ALBERT, and Electra. Additionally, the \train/\ directory contains detailed scripts for training these models on standard datasets (CTB, MSRA, CONLL03, PTB) with various architectures (RNN, ConvSeg, Transformer). The collection also includes specialized demos for serving models via TensorFlow Serving, using custom dictionaries with Trie structures, and handling multiprocessing with TensorFlow.

_plugins/hanlp\_demo/hanlp\demo/zh/tf · high confidence

Removals

Removal of legacy internal algorithm and data structure implementations

This change removes a significant number of internal utility classes and data structures from the \main/java\ source tree, including the \SmartBinTrie\ (BinTrie), \DoubleArrayTrie\, \DynamicArray\ variants, and various algorithmic helpers like \Viterbi\, \EditDistance\, and \VectorDistance\. It also deletes the empty \Main.java\ entry point and several dictionary maker utilities. These components appear to be legacy or experimental implementations that were deemed to have little practical advantage or were superseded by other parts of the codebase, resulting in a cleaner and more maintainable core library.

main/java · high confidence

Behavioural changes

HanLP version 2.1.5 release

This update releases HanLP version 2.1.5. The diff confirms the version bump in hanlp/version.py and introduces the core package initialization in hanlp/\_\init\\_.py, which sets up the legacy Keras environment variable, exposes the main load and pipeline functions, and imports the common, components, pretrained, and utils modules. Additionally, new AMR evaluation metrics are added in hanlp/metrics/amr/, including smatch\_eval support for AMR 1.0, 2.0, and 3.0.

hanlp · high confidence

Test coverage

Added unit tests for HanLP core components; Removal of legacy test and utility classes.

Dependencies

Introduce HanLP RESTful Java client and remove legacy root POM

This change adds a new Maven module for the HanLP RESTful Java client (version 0.0.15) which depends on Jackson Databind 2.14.1 and JUnit Jupiter, and simultaneously removes the legacy root project POM that previously managed the main HanLP build with JDK 1.7 and older dependencies like Jackson 1.9.13 and JUnit 3.8.1.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 53 → 53 (+0.2)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 46 → 40 (-5.4)
  • Architecture 95 → 95 (-0.6)
  • Maturity 56 → 65 (+9.6)
  • Readiness 47 → 57 (+9.5)
  • Security 72 → 61 (-10.7)
  • Domain Modelling 100 → 100 (+0.0)

Resolved (14)

  • Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
  • Duplicated block (17 lines × 3) (hanlp/components/parsers/alg_tf.py)
  • Duplicated block (24–25 lines × 2) (hanlp/components/amr/amrbart/common/postprocessing.py)
  • Duplicated block (2–8 lines × 6) (hanlp/components/amr/amrbart/model_interface/tokenization_bart.py)
  • Duplicated block (5 lines × 2) (hanlp/components/amr/amrbart/common/postprocessing.py)
  • Duplicated block (5 lines × 2) (hanlp/components/parsers/alg_tf.py)
  • Duplicated block (8 lines × 2) (hanlp/components/amr/amrbart/common/postprocessing.py)
  • Duplicated block (82–84 lines × 2) (hanlp/components/amr/amrbart/common/postprocessing.py)
  • Duplicated block (9–14 lines × 2) (hanlp/components/amr/amrbart/preprocess/amr_io.py)
  • Near-duplicate member pair (112 shared lines) (hanlp/components/amr/amrbart/common/postprocessing.py)
  • Near-duplicate member pair (36 shared lines) (hanlp/components/parsers/alg_tf.py)
  • Near-duplicate member pair (64 shared lines) (hanlp/components/parsers/alg_tf.py)
  • Rotate the exposed credentials — git history can't be un-committed
  • Secret: generic-api-key (plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb)

New (94)

  • Edited copies of one member (3 members, 15+ corresponding lines) (hanlp/datasets/parsing/ud/ud210.py)
  • Edited copies of one member (3 members, 17+ corresponding lines) (hanlp/components/parsers/constituency/crf_constituency_parser.py)
  • Edited copies of one member (3 members, 18+ corresponding lines) (hanlp/components/eos/ngram.py)
  • Edited copies of one member (3 members, 21+ corresponding lines) (hanlp/components/parsers/constituency/crf_constituency_parser.py)
  • Edited copies of one member (3 members, 29+ corresponding lines) (hanlp/components/amr/amrbart/preprocess/read_and_process.py)
  • Edited copies of one member (3 members, 3+ corresponding lines) (hanlp/components/mtl/tasks/ner/tag_ner.py)
  • Edited copies of one member (3 members, 4+ corresponding lines) (hanlp/components/ner/biaffine_ner/biaffine_ner_model.py)
  • Edited copies of one member (3 members, 42+ corresponding lines) (hanlp/components/amr/amrbart/model_interface/tokenization_bart.py)
  • Edited copies of one member (3 members, 8+ corresponding lines) (hanlp/common/vocab.py)
  • Edited copies of one member (4 members, 15+ corresponding lines) (hanlp/components/classifiers/transformer_classifier.py)
  • Edited copies of one member (6 members, 5+ corresponding lines) (hanlp/layers/crf/crf_layer_tf.py)
  • Edited copies of one member (6 members, 7+ corresponding lines) (plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/HanLPClient.java)
  • Edited copy of a member (10 corresponding lines) (hanlp/components/parsers/biaffine_tf/layers.py)
  • Edited copy of a member (10 corresponding lines) (hanlp/layers/embeddings/char_cnn_tf.py)
  • Edited copy of a member (11 corresponding lines) (hanlp/common/vocab.py)
  • Edited copy of a member (11 corresponding lines) (hanlp/components/amr/amrbart/bart_amr_generation.py)
  • Edited copy of a member (11 corresponding lines) (hanlp/components/parsers/biaffine/biaffine.py)
  • Edited copy of a member (11 corresponding lines) (hanlp/components/rnn_language_model_tf.py)
  • Edited copy of a member (118 corresponding lines) (hanlp/layers/transformers/utils_tf.py)
  • Edited copy of a member (12 corresponding lines) (hanlp/components/parsers/biaffine/structual_attention.py)
  • …and 74 more

Changes since last survey

  • 2 commits — 2 feature/other, 0 fixes

By area

  • .github/workflows — 1 commit
  • plugins/hanlp_demo — 1 commit

Notable commits

  • change: Test on ubuntu-latest, macos-latest, windows-latest
  • change: Translate documents to Chinese

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

hankcs/HanLP was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 959a6966f6e7e26d2e89c4112ca9bafbe32eb743 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.