microsoft/unilm
51.2
Adequate · 4 October 2026
633.3k
lines of production code
Python
primary language
2
measurements over time
What this system is
This system is a comprehensive research repository for sequence modeling and multimodal AI, centered around the Fairseq toolkit. It provides implementations for a wide range of transformer architectures, including specialized variants for speech, vision, and document understanding, alongside efficient decoding and attention mechanisms. The codebase supports end-to-end workflows for pre-training, fine-tuning, and evaluating models across tasks such as translation, OCR, and mathematical reasoning.
How it got here
2020–2021 — multimodal and domain-specific model expansion
23 changes.
This period focused on expanding the library's capabilities into multimodal and specialized domains by releasing toolkits for document understanding (LayoutLM, TrOCR, MarkupLM), speech processing (WavLM), and image pre-training (BEiT). It also introduced advanced fine-tuning frameworks for sequence-to-sequence generation, domain adaptation, and cross-lingual transfer, alongside significant updates to multilingual and vision-language models.
2022 — multimodal and speech model expansion
22 changes.
This period focused on releasing a wide array of new model architectures and training infrastructures, including EdgeFormer, DiT variants for document analysis, LayoutLMv3, and several speech and audio models like SpeechLM and BEATs. The work also introduced significant performance optimizations to the fairseq library through CUDA-accelerated convolution layers and product quantization, alongside new capabilities for non-autoregressive decoding and vision-language pre-training with BEiT-3.
2023–2026 — multimodal and reasoning model releases
19 changes.
This period focused on releasing and expanding a suite of advanced multimodal and reasoning models, including TextDiffuser, Kosmos-2, and LatentLM, alongside specialized architectures like YOCO and Diff Transformer. The work involved providing complete training, inference, and evaluation infrastructure for these projects, with a strong emphasis on text-intensive image generation and mathematical reasoning benchmarks. Additionally, efficiency improvements were introduced through features like Rectified Sparse Attention and preference optimization techniques such as PFPO.
Features
Add BEATs audio pre-training and tokenization models
Introduces the BEATs (Audio Pre-Training with Acoustic Tokenizers) module, providing PyTorch implementations for acoustic tokenizers and pre-trained/fine-tuned models. The \Tokenizers\ class enables audio tokenization via \extract\_labels\, while the \BEATs\ class supports feature extraction and classification via \extract\_features\. The release includes configuration classes, a Transformer-based backbone, and a vector quantizer component, along with documentation and download links for multiple model iterations.
beats · high confidence
Add BEiT semantic segmentation support for ADE20K
Introduces a new semantic segmentation implementation using the BEiT (BERT Pre-Training of Image Transformers) backbone, integrated with the mmsegmentation framework. This change adds the BEiT backbone code, along with configuration files for UperNet models (base and large variants) on the ADE20K dataset at 512x512 and 640x640 resolutions. It includes support for multi-scale testing, layer-decay optimizers, and mixed-precision training via custom Apex runner hooks.
_beit/semantic\segmentation · high confidence
Add BEiT v2 semantic segmentation support for ADE20K
Introduces a new semantic segmentation pipeline for the ADE20K dataset using the BEiT v2 backbone integrated with the UperNet decoder. This change adds the BEiT transformer implementation, custom training utilities including a layer-decay optimizer constructor and Apex-based mixed-precision distributed training hooks, and configuration files for both base and large model variants trained with 160k iterations.
_beit2/semantic\segmentation · high confidence
Add BEiT-3 vision-language model implementation and fine-tuning scripts
Introduces the official PyTorch implementation of BEiT-3, a vision and vision-language pretraining model. This change adds the core model architecture (leveraging the torchscale library), dataset handling for image-text pairs, and specific fine-tuning engines for downstream tasks including image classification, visual question answering (VQAv2), visual reasoning (NLVR2), image captioning (COCO/NoCaps), and image-text retrieval (COCO/Flickr30k). It also includes necessary utilities for data augmentation, optimization, and text normalization.
beit3 · high confidence
Add CUDA-accelerated Dynamic and Light Convolution layers with Product Quantization support
This change introduces new \DynamicconvLayer\ and \LightconvLayer\ modules to the fairseq library, featuring custom CUDA kernels for optimized forward and backward passes during training, while falling back to efficient PyTorch operations for inference. Additionally, it adds a Product Quantization (PQ) subpackage containing an Expectation-Maximization (EM) algorithm for clustering weights, along with quantized counterparts for convolution (\PQConv2d\), embedding (\PQEmbedding\), and linear (\PQLinear\) layers to enable model compression.
_edgelm/fairseq/modules/dynamicconv\_layer, edgelm/fairseq/modules/lightconv\layer, edgelm/fairseq/modules/quantization · high confidence
Add DIT-OD text detection model components
This change introduces the core components for a new text detection system based on the DIT-OD (Document Image Transformer with Object Detection) architecture. It adds a Vision Transformer (ViT) backbone implementation supporting multiple pre-trained models (BEiT, DiT, DeiT, MAE) with FPN integration, along with necessary training infrastructure including a custom dataset mapper, checkpointer, and trainer. Additionally, it includes utility modules for configuration management, data conversion, and evaluation metrics (IoU, DetEval, ICDAR2013/2018 standards) required for assessing text detection performance.
_dit/text\detection/ditod · high confidence
Add DiT image classification fine-tuning support for RVL-CDIP
This change introduces a new \dit/classification\ module that enables fine-tuning and evaluation of the DiT model on the RVL-CDIP document image classification dataset. The addition includes the full training and evaluation pipeline (\run\_class\_finetuning.py\, \engine\_for\_finetuning.py\), model definitions adapted for classification (\modeling\_finetune.py\), dataset handling for RVL-CDIP (\datasets.py\, \dataset\_folder.py\), and configuration for DeepSpeed distributed training (\deepspeed\_configs/config.json\). Users can now leverage this directory to train, evaluate, and fine-tune DiT checkpoints specifically for document classification tasks using the provided scripts and arguments.
dit/classification · high confidence
Add DiT-based text detection training and evaluation scripts
This change introduces the \dit/text\_detection\ module, providing the necessary infrastructure to train and evaluate text detection models using the DiT (Document Image Transformer) backbone. It includes a \train\_net.py\ script that leverages Detectron2's Mask R-CNN implementation to support fine-tuning on the FUNSD dataset, along with a README documenting the usage, data preparation steps, and available pre-trained weights.
_dit/text\detection · high confidence
Add LAION-OCR example dataset and processing scripts
Added a new example dataset under \textdiffuser/data/mario-laion-example\ containing structured data (captions, OCR text with bounding boxes, and metadata) for several sample images, alongside a Python script (\mario-laion-unzip.py\) to multiprocess-unzip LAION-OCR archives and a Jupyter notebook for visualizing character segments.
textdiffuser/data · high confidence
Add LayoutLMv3 object detection training examples and infrastructure
The \layoutlmv3/examples\ directory now includes a complete set of scripts and configuration files for training LayoutLMv3 on object detection tasks. This adds a YAML configuration (\cascade\_layoutlmv3.yaml\) for fine-tuning on the PubLayNet dataset, a dataset mapper (\DetrDatasetMapper\) that handles LayoutLMv3-specific inputs (text, bounding boxes, images) and disables certain augmentations, and a custom backbone implementation (\build\_vit\_fpn\_backbone\) that integrates the LayoutLMv3 model into the Detectron2 framework. Additionally, utility scripts are provided for preprocessing ICDAR datasets (\convert\_to\_coco\_format.py\, \adaptive\_binarize.py\) and a custom evaluator (\ICDAREvaluator\) for table detection metrics.
layoutlmv3/examples · high confidence
Add MARIOEval evaluation and generation scripts
The \textdiffuser/eval\ directory now includes scripts to generate and evaluate text-to-image outputs using the MARIOEval benchmark. \MARIOEval\_generate.py\ supports generation via Stable Diffusion, ControlNet (Canny), and DeepFloyd IF, while \MARIOEval\_evaluate.py\ computes CLIPScore and FID metrics. The directory also provides \ocr\_eval.py\ for OCR-based accuracy, precision, and recall, alongside helper modules for CLIPScore and FID calculations and shell scripts to run the workflows.
textdiffuser/eval · high confidence
Add MWPBench evaluation suite and Fresh-GaokaoMath-2023 dataset
Users can now evaluate math instruction-tuned models using the new MWPBench benchmark located in mathscale/MWPBench. This addition includes a Dockerfile for setting up the evaluation environment, a README with usage instructions for both vLLM and OpenAI drivers, and new JSON data files containing the full test set (aggregating datasets like GSM8K, MATH, and others) and a specific 30-question 'Fresh-GaokaoMath-2023' subset derived from the 2023 Gaokao exam.
mathscale · high confidence
Add PFPO analysis and pseudo-test-case pipeline scripts
This change introduces a suite of new scripts under \PFPO/scripts/apps/\ to support the PFPO (ICLR 2025) workflow. The \analyze/\ directory adds tools for computing output frequencies (\get\_output\_frequency.py\) and generating visualization plots (\freq2image.py\) for self-consistency and program self-consistency metrics. The \prm/\ directory includes scripts for constructing process reward model samples (\construct\_process\_rm\_sample.py\, \construct\_process\_rm\_sample\_fix.py\) and sampling code steps (\sample\_steps.py\). Additionally, the \pseudo\_test\_cases/\ directory provides utilities for cleaning and processing synthetic test inputs from external models (Mistral, GPT-4o), along with shell pipelines for executing these analyses and constructing preference pairs for DPO training.
(repo-wide) · high confidence
Add SpeechLM speech pre-training and fine-tuning code
Added the SpeechLM module, including the main model implementation in SpeechLM.py, supporting modules in modules.py, and comprehensive documentation in README.md. This addition provides pre-trained SpeechLM models (Base and Large variants) along with scripts and instructions for feature extraction, fine-tuning on ASR (LibriSpeech) and speech translation (CoVoST-2) tasks, and pre-training. The codebase includes a fairseq submodule and cleaned checkpoints available via Google Drive and OneDrive links.
speechlm · high confidence
Add SpeechT5 unified speech and text processing implementation
Introduces the SpeechT5 module, providing an official PyTorch implementation of the SpeechT5 model for unified-modal encoder-decoder pre-training in spoken language processing. This addition includes the core model architecture, data handling via a MultitaskDataset, and specific loss functions (criterions) for pre-training, speech-to-text (ASR), and text-to-speech tasks. It also provides a script for generating speech outputs and integrates with the fairseq library (via a submodule) to support training and inference workflows.
speecht5 · high confidence
Add TrOCR optical character recognition module with image augmentation and fairseq integration
Adds the TrOCR (Transformer-based Optical Character Recognition) module, providing an end-to-end text recognition approach using pre-trained image and text Transformers. The package includes the core model definitions (TrOCRModel, ViTTRModel), a custom fairseq task for text recognition, and a specialized GPT2BPEEnhancedSpace tokenizer. It also provides a comprehensive suite of image augmentation techniques (blur, geometry, noise, weather effects) for training robustness, along with scripts for fine-tuning and evaluation on datasets like IAM, SROIE, and STR Benchmarks.
trocr · high confidence
Add UniLM v1 source code and documentation for sequence-to-sequence generation
This change introduces the complete source code and documentation for UniLM v1, a unified pre-training model for natural language understanding and generation. The addition includes the \README.md\ with setup instructions (Docker, dependencies) and fine-tuning examples for abstractive summarization tasks (Gigaword, CNN/Daily Mail). The codebase under \src/biunilm\ provides the core implementation for pre-training and fine-tuning, including data loading (\seq2seq\_loader.py\, \loader\_utils.py\), training scripts (\run\_seq2seq.py\), decoding utilities (\decode\_seq2seq.py\, \gen\_seq\_from\_trace.py\), and evaluation tools (\cnndm/bs\_pyrouge.py\). This enables users to reproduce the SOTA results reported in the NeurIPS 2019 paper for sequence-to-sequence generation tasks.
unilm-v1 · high confidence
Add VLMo vision-language pre-training implementation
Introduces the VLMo (Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts) model, including the core PyTorch Lightning training script, configuration for pre-training and fine-tuning tasks (VQAv2, NLVR2, COCO, F30K), and data modules for datasets like GCC, SBU, Visual Genome, and COCO. The release also includes documentation for dataset preparation and links to pre-trained model weights.
vlmo · high confidence
Add object detection inference and training scripts for DiT
This change introduces the \dit/object\_detection\ directory, providing scripts to run object detection using the DiT backbone with Detectron2. It includes \inference.py\ for running inference on images and visualizing results, \train\_net.py\ for fine-tuning Mask R-CNN and Cascade Mask R-CNN models on PubLayNet and ICDAR 2019 cTDaR datasets, and utility scripts (\convert\_to\_coco\_format.py\, \adaptive\_binarize.py\) for preparing the ICDAR dataset. The entry also adds a README with usage instructions, data preparation steps, and evaluation commands.
_dit/object\detection · high confidence
Add relation extraction decoder module
The library now includes a new relation extraction (RE) decoder component in the \layoutlmft.modules.decoders\ package. This module provides a \REDecoder\ class that implements binary relation classification using a biaffine attention operator, allowing users to extract relationships between entities from document layouts.
layoutlmft/layoutlmft/modules/decoders · high confidence
Add support for MiniLM and UniLM model variants in s2s-ft
The s2s-ft module now supports fine-tuning MiniLM and UniLM models in addition to BERT and RoBERTa. This includes new configuration classes (MinilmConfig, UnilmConfig), tokenizers (MinilmTokenizer, UnilmTokenizer), and state-dict conversion utilities to handle model-specific weight formats (e.g., UniLM's QKV linear layers). The modeling code has been updated to load these variants from Azure Blob storage URLs and handle their specific embedding structures.
_s2s-ft/s2s\ft · high confidence
Add xTune cross-lingual fine-tuning scripts and documentation
Adds the xTune repository for the ACL2021 paper 'Consistency Regularization for Cross-Lingual Fine-Tuning', including a README with setup instructions and a suite of shell scripts for downloading XTREME datasets (XNLI, PAWS-X, UD-POS, PANX, MLQA, TyDiQA, XQuAD) and pre-trained XLM-RoBERTa models. The change introduces training pipelines for both 'cross-lingual-transfer' and 'translate-train-all' settings, supporting a two-stage training process with consistency regularization and data augmentation (BPE sampling, code-switching, machine translation) for token-level and text classification tasks.
xtune · high confidence
Added CUDA-accelerated Dynamic and Light Convolution layers for faster sequence generation
New \DynamicconvLayer\ and \LightconvLayer\ modules have been added to the fairseq library, featuring custom CUDA kernels to accelerate sequence-to-sequence generation. These layers replace standard convolution operations with optimized forward and backward passes, providing a lossless speedup for decoding tasks. The implementation includes the necessary Python wrappers, PyTorch extensions, and CUDA kernels for both training and inference (including incremental state handling).
(repo-wide) · high confidence
Added English-to-German and LibriSpeech dataset configurations and assets
New configuration files, tokenizers, and sample data have been added for the English-to-German speech translation task using Common Voice v4, including SentencePiece models and a 100-sample development manifest. Additionally, vocabulary files and configuration for LibriLM hidden-unit and phone-unit models, along with LibriSpeech ASR dictionary references and a 100-sample text manifest, have been introduced to support speech language model training.
speechlm/dataset · high confidence
Added TextDiffuser demo assets and example inputs
The TextDiffuser demo now includes example input files for text-to-image, text-inpainting, and text-to-image-with-template scenarios, along with core model utility files (modeling\_utils, DDPMScheduler, UNet2DConditionModel) and a font placeholder. These assets provide the necessary data and code components to run the TextDiffuser demonstration locally.
textdiffuser/assets · high confidence
Added TextDiffuser layout prediction and text segmentation model components
The \textdiffuser/model\ directory now includes the core model architecture for the TextDiffuser system, which predicts keyword layouts and performs text segmentation. This adds a \LayoutTransformer\ (with a \TextConditioner\ using a frozen CLIP encoder) to determine bounding box coordinates for keywords in a prompt, and a UNet-based \text\_segmenter\ to generate segmentation masks. These components enable the system to 'paint' text by first calculating where keywords should appear and then segmenting the resulting image.
textdiffuser/model, textdiffuser/textdiffuser-ckpt · high confidence
Added angle and quadrilateral text rendering extensions
The TextDiffuser-2 extensions now support rendering text with specific geometric shapes, specifically angles and quadrilaterals. This change introduces new training scripts (\train\_textdiffuser2\_t2i\_full\_angle.py\, \train\_textdiffuser2\_t2i\_full\_quadrilateral.py\) and inference scripts (\inference\_textdiffuser2\_t2i\_full\_angle.py\, \inference\_textdiffuser2\_t2i\_full\_quadrilateral.py\) along with their corresponding shell launchers and template files. These additions allow users to train and generate images where text is positioned according to angle or quadrilateral coordinate layouts, expanding the model's capability beyond standard bounding boxes.
textdiffuser-2/extensions · high confidence
Added fairseq documentation and project metadata
Added the Sphinx-based documentation source files for the fairseq sequence modeling toolkit, including configuration, command-line tool references, library API documentation (models, tasks, optimizers, data), and tutorials for extending the framework. Also added standard project metadata files such as the MIT license, code of conduct, contributing guidelines, and a .gitignore to support the project's open-source workflow.
(repo-wide) · high confidence
Adds GAD (Global Attention Decoding) plugins for non-autoregressive sequence generation
This change introduces a new plugin suite under \decoding/GAD/block\_plugins\ that enables Global Attention Decoding (GAD) for non-autoregressive (NAT) sequence-to-sequence models. It registers a new \block\ model architecture (\BlockNAT\) and a \glat\_loss\ criterion that implements a dual imitation learning objective with label smoothing. The implementation includes a modified translation task (\translation\_lev\_modified\) supporting various noise injection strategies (random delete, random mask, block mask) for training, along with the necessary data assets (BPE vocabularies and dictionaries for WMT14 en-de and WMT16 en-ro).
decoding · high confidence
Initial release of AdaLM domain adaptation toolkit
This change introduces the AdaLM (Adapt-and-Distill) repository, providing code to fine-tune pre-trained language models for specific domains such as biomedicine and computer science. The release includes finetuning scripts for sequence classification (run\_classifier.py) and named entity recognition (run\_ner.py, run\_pico.py), along with utility modules for data processing and metrics. It also provides an incremental BPE vocabulary builder (incr\_bpe/) to extend tokenizers for domain-specific corpora, and a README documenting model checkpoints and usage examples.
adalm · high confidence
Initial release of BEiT: BERT Pre-Training of Image Transformers
This change introduces the official PyTorch implementation and pretrained models for BEiT, a method for pre-training image transformers using BERT-style masked token prediction. The repository includes the core model architecture, training engines for both pre-training and fine-tuning, and data augmentation pipelines that utilize a discrete VAE (specifically DALL-E) to convert images into visual tokens. It provides scripts and documentation for fine-tuning on ImageNet-1k and ADE20K, as well as pre-training on ImageNet-22k, supporting both BEiT-base and BEiT-large configurations.
beit · high confidence
Initial release of EdgeFormer documentation and project scaffolding
This change introduces the initial project structure and documentation for EdgeFormer, a parameter-efficient transformer for on-device sequence-to-sequence generation. The update adds the README with model details, benchmarks, and fine-tuning instructions, along with standard project files including the MIT License, Code of Conduct, and Contributing guidelines. It also includes a comprehensive Sphinx-based documentation set covering fairseq command-line tools, library references (models, tasks, data, optimizers), and tutorials for extending the framework.
edgelm · high confidence
Initial release of Kosmos-2 code, data, and demo
This change introduces the Kosmos-2 multimodal model repository, providing the complete infrastructure to use, evaluate, and train the model. It includes a Gradio-based interactive demo for visual grounding and text generation, along with Python scripts to prepare the GRIT dataset, visualize grounding results, and decode bounding boxes from model outputs. The release also provides evaluation code and benchmark results for phrase grounding, referring expression comprehension, image captioning, and visual question answering, alongside documentation for setting up the required environment.
kosmos-2 · high confidence
Initial release of Kosmos-2.5 multimodal literate model
This change introduces the Kosmos-2.5 repository, a multimodal model for machine reading of text-intensive images. The release includes the inference script (\inference.py\) and model architecture (\UniGPTmodel\) which support two primary tasks: generating spatially-aware text blocks (OCR) and producing structured Markdown output from images. The implementation relies on a shared decoder-only auto-regressive Transformer architecture, integrates with Fairseq and Hugging Face Transformers (specifically Pix2Struct for image processing), and requires Ampere, Ada, or Hopper GPUs. The entry also adds standard project documentation including the README, license, code of conduct, and security policy.
kosmos-2.5 · high confidence
Initial release of LayoutLMv3 document AI model
This change introduces the LayoutLMv3 model, a multimodal transformer for Document AI that uses unified text and image masking for pre-training. The release includes the core package setup (setup.py, setup.cfg), a .gitignore file, and a README detailing installation, pre-trained model availability (base, large, and base-chinese), and fine-tuning examples for tasks like form understanding and document layout analysis.
layoutlmv3 · high confidence
Initial release of LayoutXLM and XFUN support
This change introduces the LayoutXLM model and the XFUN dataset integration to the library. It adds the LayoutXLM configuration, model classes (for token classification and relation extraction), and tokenizers (both slow and fast) which extend the existing LayoutLMv2 architecture. Additionally, it provides the XFUN dataset builder, enabling users to load and process the XFUN dataset for cross-lingual document understanding tasks, alongside the necessary data collators and evaluation utilities for relation extraction.
layoutlmft/layoutlmft · high confidence
Initial release of MarkupLM pre-trained models and fine-tuning code
This change introduces the MarkupLM component, a multimodal pre-training method for text and markup language designed for visually-rich document understanding. The release includes the \setup.py\ package definition for \markuplmft\ (version 0.1) and a comprehensive \README.md\ detailing the installation, pre-trained models (Base and Large), and fine-tuning procedures for tasks such as WebSRC and SWDE.
markuplm · high confidence
Initial release of PFPO: Preference Optimization for Reasoning with Pseudo Feedback
This change introduces the PFPO (Preference Optimization for Reasoning with Pseudo Feedback) module, a method for improving reasoning in large language models by generating pseudo-feedback based on test cases. The release includes the core source code, a comprehensive README detailing experimental results on mathematical reasoning (MATH, GSM8K) and coding tasks (LiveCodeBench, APPs, HumanEval, MBPP), and a dataset split file (\apps\_train\_sub\_val\_ids.json\). It also provides a suite of Hydra configuration files for running inference and evaluation using vLLM against models such as Deepseek-Coder-7B-v1.5, Mathstral-7B, Llama-3.1-70B, and Mistral-Large, covering various prompt templates and sampling strategies for tasks like code completion and test-case input generation.
PFPO · high confidence
Initial release of TextDiffuser-2 assets and layout planning data
This change introduces the foundational components for the TextDiffuser-2 project. It adds a custom \attention\_processor.py\ module to handle attention mechanisms, likely supporting optimized inference via xFormers. It also includes a \reference\_requirements.txt\ file pinning the Python environment (including PyTorch 2.0.1, Diffusers 0.24.0, and Gradio 3.50.2) and provides a 5,000-sample dataset (\layout\_planner\_data\_5k.json\) for training or evaluating text layout planning, along with a utility script to visualize these layouts.
textdiffuser-2/assets, textdiffuser-2/data · high confidence
Initial release of TextDiffuser-2 codebase and demos
This change introduces the complete TextDiffuser-2 repository, including training scripts for the layout planner (using FastChat/Vicuna) and diffusion models (full-parameter and LoRA fine-tuning), as well as inference scripts for text-to-image generation. It also adds interactive Gradio demos for both standard text-to-image and text inpainting tasks, along with a Replicate prediction interface (cog.yaml) and comprehensive documentation (README.md).
textdiffuser-2 · high confidence
Initial release of s2s-ft sequence-to-sequence fine-tuning toolkit
Introduces the s2s-ft package, a PyTorch-based toolkit for fine-tuning pre-trained Transformer models (including BERT, RoBERTa, XLM-RoBERTa, MiniLM, and UniLM) for sequence-to-sequence generation tasks. The release includes core training and decoding scripts (\run\_seq2seq.py\, \decode\_seq2seq.py\), a utility for generating sequences from beam search traces (\gen\_seq\_from\_trace.py\), and evaluation modules for XSum, CNN/Daily Mail, and Gigaword datasets using ROUGE metrics. It also provides documentation and configuration for these specific model types.
s2s-ft · high confidence
Initial release of s2s\_ft layout reader components
This release adds the \s2s\_ft\ subpackage to LayoutReader, introducing the core configuration, modeling, and tokenization classes required for sequence-to-sequence fine-tuning. The new files provide configuration and tokenizer support for MiniLM and UniLM models, alongside a sequence-to-sequence model implementation (\modeling.py\) that supports pre-trained models from BERT, RoBERTa, XLM-RoBERTa, LayoutLM, UniLM, and MiniLM. The package also includes utilities for converting state dictionaries between these different model formats and a data loading pipeline (\s2s\_loader.py\) for preparing source and target sequences for training.
_layoutreader/s2s\ft · high confidence
Initial release of the layoutlmft toolkit
The layoutlmft package is introduced as a multimodal fine-tuning toolkit for document understanding, supporting LayoutLM, LayoutLMv2, and LayoutXLM models. This release includes example scripts for fine-tuning on FUNSD and XFUN datasets (covering both sequence labeling and relation extraction tasks), along with standard project configuration files (setup.py, setup.cfg, Makefile) and a .gitignore to facilitate local development and installation.
layoutlmft · high confidence
Introduce BEiT v2 implementation with vector-quantized visual tokenizers
Adds the official PyTorch code and pretrained models for BEiT v2, a masked image modeling approach that uses vector-quantized visual tokenizers (VQ-KD) instead of raw pixels. This location provides the core training engines for pretraining, fine-tuning, and VQ-KD tokenizer training, along with dataset handling and data augmentation pipelines that support the new semantic codebook-based pretraining objective. Documentation includes instructions for pretraining on ImageNet-1k, fine-tuning for image classification, semantic segmentation, and object detection, as well as links to pretrained checkpoints.
beit2 · high confidence
Introduce Diff Transformer attention implementations
Added a new Diff Transformer module providing multiple implementations of differential multi-head attention, including a naive PyTorch version, a FlashAttention-optimized version for libraries supporting different Q/K/V dimensions (e.g., xformers), and a variant for standard FlashAttention. The package includes a Triton-based rotary embedding kernel, a fallback RMSNorm implementation, and an example script to compare parameter counts between Diff Transformer and baseline attention mechanisms.
Diff-Transformer · high confidence
Introduce LatentLM multimodal latent language modeling implementation
Adds the LatentLM component, providing a PyTorch implementation of multimodal latent language modeling with next-token diffusion. This includes core model architectures (DiT, Transformer), supporting modules (RMSNorm, EMA, attention kernels), and evaluation utilities for computing FID and Inception Score metrics, along with scripts for inference speed benchmarking and fidelity evaluation.
LatentLM · high confidence
Introduce Rectified Sparse Attention (ReSA) for efficient long-context decoding
This change adds the Rectified Sparse Attention (ReSA) module, which improves sparse decoding by periodically refreshing the KV cache to prevent error accumulation and maintain generation quality. The implementation includes a new \KVManager\ in \context\_manager.py\ that tracks block-level key statistics to select relevant KV cache blocks, and custom Triton kernels (\flash\_sparse\_decoding.py\, \flash\_attention\_with\_kv\_cache.py\) to execute the sparse attention and rotary embeddings efficiently. The model architecture (\model.py\) is updated to integrate this sparse decoding path, allowing users to enable ReSA via configuration parameters such as \resa\_sparse\_ratio\, \resa\_block\_size\, and \resa\_rec\_freq\ to achieve significant speedups on long sequences.
ReSA · high confidence
Introduce SpeechLM pre-training, fine-tuning, and decoding configurations
This change adds the configuration files for the SpeechLM model, including Hydra configs for pre-training (base and large variants on LibriSpeech and LibriLight), fine-tuning (100h and 960h subsets), and inference decoding (using fairseq LM, KenLM, and Viterbi decoders). These files define the task, model architecture, optimization, and dataset settings required to run the SpeechLM pipeline.
speechlm/speechlm · high confidence
Introduce ViT-based object detection backbone and table-aware evaluation
This change adds a new Vision Transformer (ViT) backbone implementation for object detection, supporting models like BEiT, DiT, DeiT, and MAE with FPN integration and configurable positional embeddings (absolute, shared relative, relative). It also introduces a custom dataset mapper for DETR-style data augmentation, a specialized checkpointer that handles position embedding interpolation for different image sizes, and an ICDAR table detection evaluator that converts predictions to XML and computes table-level wF1 scores.
_dit/object\detection/ditod · high confidence
Introduce WavLM large-scale self-supervised speech pre-training models
Adds the WavLM model implementation and documentation, providing pretrained checkpoints (Base, Base+, and Large) for full-stack speech processing tasks such as speaker verification, speech separation, diarization, and recognition. The release includes the core PyTorch implementation in \wavlm/WavLM.py\ and supporting utility modules in \wavlm/modules.py\, along with a \README.md\ detailing usage, download links, and performance benchmarks on SUPERB and other benchmarks.
wavlm · high confidence
Introduce XDoc document understanding model and fine-tuning toolkit
Adds the XDoc unified pre-trained model for cross-format document understanding, including pre-trained weights and fine-tuning code for SQuAD, FUNSD, and WebSRC tasks. The release includes a \layoutlmft\ library with LayoutLMv2 and LayoutXLM model implementations, dataset builders for FUNSD and XFUN, and evaluation utilities for token classification and relation extraction.
xdoc · high confidence
Introduce YOCO decoder-architecture evaluation and training scripts
This change adds the YOCO (You Only Cache Once) project to the repository, providing a complete set of evaluation and training scripts for a decoder-decoder large language model architecture. Users can now run Harness evaluations (e.g., BoolQ, ARC) and long-context 'Needle in a Haystack' retrieval tests using the provided \eval\_task.sh\ and \eval\_needle.sh\ scripts, which leverage Fairseq criteria like \harness\_eval\ and \multi\_needle\. Additionally, \train.sh\ enables pretraining from scratch with distributed support, configuring the \yoco\_base\ or \yoco\_3b\_new\ architectures with specific optimizer, learning rate, and data-loading parameters.
YOCO · high confidence
Introduce attention head selection for speech-to-text models
This change adds a new \attention\_head\_selection\ example that enables dynamic selection of attention heads in Fairseq's speech-to-text (S2T) and text Transformer models. The implementation introduces a \HeadSelectionS2TTransformerModel\ and \HeadSelectionTransformerModel\ that replace standard encoder/decoder layers with \HeadSelectionTransformerEncoderLayer\ and \HeadSelectionTransformerDecoderLayer\. These layers utilize a custom \MultiheadAttentionSelection\ module and an \AttnHeadSelector\ to dynamically choose which attention heads to use based on task-specific IDs (such as language or domain). The system supports both 'subset' and 'group' selection strategies and integrates a KL-divergence loss (\HeadSelectionLoss\) into the training loop to regularize the head selection process.
(repo-wide) · high confidence
Introduces Differential Transformer V2 attention mechanism
Adds a new \MultiheadFlashDiffV2\ module that implements the Differential Transformer V2 architecture, enabling faster inference and improved training stability compared to the previous version. This implementation uses FlashAttention to subtract attention outputs from paired heads within the same GQA group, controlled by a token-specific, head-wise projected lambda parameter, and can be toggled on or off via the \use\_diff\_v2\ flag to act as a standard baseline Transformer when disabled.
Diff-Transformer/Diff-Transformer-V2 · high confidence
New fine-tuning examples for SWDE and WebSRC datasets
Added new example scripts for fine-tuning MarkupLM on the SWDE (Web Information Extraction) and WebSRC (Web-based Question Answering) datasets. The SWDE example includes utilities for data packing, feature preparation, and evaluation with site-level voting, alongside a training script for token classification. The WebSRC example provides dataset generation and splitting tools for converting raw CSV data into SQuAD-style JSON, along with a training script for question answering.
markuplm/examples · high confidence
Release E5 embedding models with evaluation code and multilingual support
This change introduces the E5 text embedding models (including English, multilingual, and LLM-based variants like e5-mistral-7b-instruct) along with the necessary evaluation infrastructure. It adds configuration files to define pooling strategies (e.g., average, last token, position-weighted mean) and instruction prefixes for each model variant. The release includes Python scripts and shell wrappers to evaluate these models on the BEIR and MTEB benchmarks, supporting both English and multilingual tasks, and provides updated documentation with model specifications and usage instructions.
e5 · high confidence
Release SimLM pre-training and fine-tuning code
The simlm directory now contains the complete codebase for the SimLM retrieval-oriented pre-training architecture, including a four-stage supervised fine-tuning pipeline for dense retrievers and cross-encoder re-rankers. This release provides scripts and configuration for reproducing results on the MS-MARCO passage ranking task, supporting training with BM25 hard negatives and knowledge distillation, alongside utilities for data preparation, metric computation, and model evaluation.
simlm · high confidence
Release of DeltaLM encoder-decoder model with Fairseq integration
This change introduces the DeltaLM model, an encoder-decoder architecture for language generation and translation that supports 100+ languages. The release includes the core model implementation (\deltalm/models/deltalm.py\) which extends Fairseq's TransformerModel to load and adapt pretrained multilingual encoder checkpoints. It provides CLI entry points (\train.py\, \generate.py\, etc.) and example scripts for the IWSLT14 German-English task, covering data preparation, tokenization with Sentencepiece, binary data creation, fine-tuning, and evaluation. The codebase is structured as a submodule of Fairseq.
deltalm · high confidence
Release of LayoutReader for reading order detection
This change introduces the LayoutReader component, a seq2seq model designed to predict the reading order of text lines in documents by capturing both text and layout information. The release includes the core training script (run\_seq2seq.py) and decoding script (decode\_seq2seq.py) which leverage the LayoutLM architecture, along with a README detailing installation, usage, and performance benchmarks on the ReadingBank dataset. It also adds the s2s-ft package setup (setup.py) required to install the necessary dependencies for fine-tuning sequence-to-sequence models.
layoutreader · high confidence
Release of MiniLM v2 and Multilingual MiniLM v1 models with fine-tuning examples
This update introduces the MiniLM v2 pre-trained models, which utilize self-attention relation distillation to compress teacher models (such as RoBERTa-Large and XLMR-Large) into smaller, faster variants (e.g., L6xH384, L12xH384) for both monolingual and multilingual tasks. It also includes the previously released Multilingual MiniLM v1 models. To support adoption, the repository now provides example scripts for fine-tuning these models on specific benchmarks, including \run\_xnli.py\ for cross-lingual natural language inference and documentation for fine-tuning on SQuAD 2.0 (NLU) and XSum (NLG).
minilm · high confidence
Release of TextDiffuser training, inference, and demo scripts
The textdiffuser directory now includes the complete source code for the TextDiffuser model, providing users with scripts to train the model on the MARIO-10M dataset (train.py, train.sh), run inference for text-to-image, text-to-image-with-template, and text-inpainting tasks (inference.py, evaluate.py, and corresponding shell scripts), and interact with the model via a Gradio web interface (gradio\_app.py).
textdiffuser · high confidence
Removals
Deprecated LayoutLM module with fine-tuning examples and data processors
The \layoutlm/deprecated\ directory now contains the legacy LayoutLM implementation, including the core model classes (\LayoutlmConfig\, \LayoutlmForSequenceClassification\, \LayoutlmForTokenClassification\) and data processors for the FUNSD and RVL-CDIP datasets. This location also provides example scripts for sequence labeling and document image classification, along with configuration files for code formatting (flake8, isort, pre-commit) and a README detailing the model's background and usage.
layoutlm/deprecated · high confidence
Behavioural changes
Introduce InfoXLM multi-task training with XLM-Align data loading fix
This change introduces the InfoXLM framework, enabling multi-task training that combines Masked Language Modeling (MLM), Token-Level Multilingual (TLM), and eXtreme Language Contrastive Optimization (XlCo). It adds new Fairseq tasks (\infoxlm\, \xlm\_align\, \tlm\, \mlm\) and models (\infoxlm\, \xlm\_align\, \reload\_roberta\) along with corresponding data loaders and criteria. Specifically, it fixes a bug in the XLM-Align data loading process (referenced as \#404) by implementing the \XlmAlignTask\ and its associated dataset utilities (\xlm\_align.py\ in data and criterions), ensuring correct handling of word alignment and offset data during training.
infoxlm/src-infoxlm · high confidence
Dependencies
Add dependency manifests for new model implementations
This change introduces requirements.txt and pyproject.toml files for several new model implementations, including PFPO, YOCO, BEiT v2/v3, Kosmos-2.5, InfoXLM, and MWPBench. These files define the specific Python package versions and build dependencies required to run these models, such as PyTorch, Transformers, and DeepSpeed, ensuring that users can install the correct environment for each new component.
(dependencies) · high confidence
Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 59 → 51 (-7.5)
- Rubric changed (rubric-2026.09.15 → rubric-2026.10.1) — scores are not directly comparable.
Lenses
- Code Health 78 → 68 (-10.2)
- Architecture 86 → 88 (+2.0)
- Maturity 70 → 70 (+0.0)
- Readiness 40 → 26 (-14.5)
- Security 74 → 74 (+0.0)
- Domain Modelling 100 → 100 (+0.0)
- Performance 100 (new)
Resolved (10)
- Documentation: no installation or build instructions (README.md)
- Documentation: no installation or build instructions (textdiffuser/README.md)
- Documentation: no project overview (README.md)
- Medium CVE: PYSEC-2026-3716 (layoutlmft/requirements.txt)
- Medium CVE: PYSEC-2026-3716 (layoutlmv3/requirements.txt)
- Medium CVE: PYSEC-2026-3716 (markuplm/requirements.txt)
- Medium CVE: PYSEC-2026-3716 (simlm/requirements.txt)
- Medium CVE: PYSEC-2026-3716 (textdiffuser-2/assets/reference_requirements.txt)
- Medium CVE: PYSEC-2026-3716 (xdoc/fine_tuning/funsd/requirements.txt)
- Off-boarding risk: anonymized user #1
New (13)
- Low cohesion: Attention (LCOM4 4) (textdiffuser-2/assets/attention_processor.py)
- Low cohesion: AutoencoderKL (LCOM4 4) (LatentLM/tokenizer_models/vae.py)
- Low cohesion: Logger (LCOM4 5) (dit/text_detection/ditod/concern/log.py)
- Low cohesion: T5TransformerModel (LCOM4 4) (speecht5/speecht5/models/speecht5.py)
- Medium CVE: [GHSA redacted] (layoutlmft/requirements.txt)
- Medium CVE: [GHSA redacted] (layoutlmv3/requirements.txt)
- Medium CVE: [GHSA redacted] (markuplm/requirements.txt)
- Medium CVE: [GHSA redacted] (simlm/requirements.txt)
- Medium CVE: [GHSA redacted] (textdiffuser-2/assets/reference_requirements.txt)
- Medium CVE: [GHSA redacted] (xdoc/fine_tuning/funsd/requirements.txt)
- Off-boarding risk: anonymized user #1
- Split PFPO
- Split kosmos-2/fairseq
Changes since last survey
- 1 commits — 1 feature/other, 0 fixes
By area
- layoutreader/README.md — 1 commit
Notable commits
- change: Restore LayoutReader checkpoint downloads and loading guidance
Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
microsoft/unilm was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 4 October 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 31c5b904ca1bf2afb4c234a6675c683a4e5fc7cd — the exact code this score is about.
- Scored under rubric-2026.10.1 — the same rubric and the same method as every other entry in this index.
- Measured by watchdog.canine.dev using codehealth-analyzer preprod-b94e107d0cec.