Skip to content
CAI
Software that uses CAICheck a score

hiyouga/LlamaFactory

52.3

Adequate · 14 November 2026

45.1k

lines of production code

Python

primary language

6

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add FSDP+QLoRA example for Llama 3

Added a new example configuration and launch script for fine-tuning Meta-Llama-3-8B-Instruct using Fully Sharded Data Parallel (FSDP) with 4-bit QLoRA. The YAML config sets up supervised fine-tuning (SFT) with LoRA (rank 8, targeting all modules) on the identity and alpaca\_en\_demo datasets, while the shell script provides the command to launch training via Accelerate, including a warning against using GPTQ/AWQ models in this setup.

_examples/extras/fsdp\qlora · high confidence

Add HyperParallel distributed training backend

Users can now train models using the HyperParallel backend, which replaces the standard Accelerate FSDP2 implementation with HyperParallel's fully\_shard for improved distributed efficiency. This change introduces support for HyperParallel-specific features including Context Parallel (CP), Expert Parallel (EP), and activation optimization, accessible via new configuration arguments (e.g., \hyper\_parallel\_cp\_size\, \hyper\_parallel\_ep\_size\). The \src/llamafactory/train/hyper\_parallel\ module provides the \HyperParallelTrainer\ and workflow entry points (\run\_pt\, \run\_sft\) that handle model preparation, data sharding, and gradient clipping specific to this backend.

_src/llamafactory/train/hyper\parallel · high confidence

Add KTO (Kahneman-Tversky Optimization) training support

Users can now fine-tune models using the KTO algorithm, a preference optimization method that learns from binary human feedback without requiring paired comparisons. This change introduces the KTO training workflow, including a custom trainer that integrates with the existing LlamaFactory infrastructure, supports distributed training (DeepSpeed/FSDP), and allows for configurable weights for chosen and rejected responses.

src/llamafactory/train/kto · high confidence

Add Muon optimizer and Triton-based chunked attention kernels

This update introduces the Muon optimizer, a new training algorithm that applies orthogonalization to parameter updates via Newton-Schulz iterations, and a suite of Triton-accelerated kernels for chunked gated delta-rule attention. The Muon optimizer is available in the third-party module for use in training workflows, while the new Triton implementations (including chunked forward/backward passes, triangular solvers, and cumulative sum utilities) provide optimized, hardware-aware computation paths for attention mechanisms, particularly targeting NPU and GPU acceleration.

_src/llamafactory/third\party · high confidence

Add Muon optimizer plugin for v1 training

The v1 trainer now includes a Muon optimizer plugin. This optimizer applies standard SGD-momentum with Newton-Schulz orthogonalization to 2D hidden weight matrices, while routing 1D biases, embeddings, output heads, and LoRA adapter factors to an internal AdamW optimizer. The implementation is DTensor-aware, ensuring correct behavior under FSDP2 and sequence parallelism by all-gathering full gradients before orthogonalization.

_src/llamafactory/v1/plugins/trainer\plugins/optimizers · high confidence

Added English and Chinese Alpaca-format demo datasets and C4 pre-training sample

The \data\ directory now includes \alpaca\_en\_demo.json\, \alpaca\_zh\_demo.json\, and \c4\_demo.jsonl\ as reference files for users configuring custom datasets. These files demonstrate the expected JSON structure for instruction-tuning (Alpaca format) and pre-training (C4 format), providing concrete examples for both English and Chinese data ingestion.

data · high confidence

Added Llama-PRO expansion and freeze-SFT example for Llama 3

The Llama-PRO example directory now includes a shell script to expand the Meta-Llama-3-8B-Instruct model and a YAML configuration for performing freeze-based Supervised Fine-Tuning (SFT) on the expanded model. This enables users to apply the Llama-PRO technique to Llama 3 architectures.

_examples/extras/llama\pro · high confidence

Added PiSSA initialization and LoRA SFT example for LLaMA-3

Users can now run a complete example for Parameter-preserving SVD-based Initialization (PiSSA) fine-tuning on the Meta-Llama-3-8B-Instruct model. This change introduces an initialization script to prepare the model weights and a YAML configuration file that sets up Supervised Fine-Tuning (SFT) with LoRA, specifically enabling PiSSA initialization and conversion while disabling validation by default.

examples/extras/pissa · high confidence

Initial model loading and adapter infrastructure

The \src/llamafactory/model\ package is introduced, providing the core infrastructure for loading models, tokenizers, and configurations, as well as managing fine-tuning adapters. This includes support for full, freeze, and LoRA/OFT tuning methods, integration with Unsloth and KTransformers backends, and specific patches for models like Qwen3.5 and Qwen3-Omni to handle NPU compatibility and attention mechanisms.

src/llamafactory/model · high confidence

Introduce DPO training workflow and custom trainer

Users can now perform Direct Preference Optimization (DPO) fine-tuning via the new \run\_dpo\ entry point in \src/llamafactory/train/dpo\. This change adds a \CustomDPOTrainer\ that extends TRL's \DPOTrainer\ to support specific preference loss types (ORPO, SimPO, BCO), integrates with custom optimizers/schedulers, and handles reference model preparation for DeepSpeed and FSDP. The workflow manages data loading, reference model creation, training, evaluation, and metrics plotting specifically for the DPO stage.

src/llamafactory/train/dpo · high confidence

Introduce LlamaFactory v1 core training and inference architecture

This change introduces the foundational v1 core components for model training and inference. It adds a new data engine for loading and indexing datasets, a model engine for initializing models and processors, and a renderer that converts messages into model inputs using Hugging Face chat templates with special-token escaping for security. Additionally, it provides base trainer classes for orchestration and specific implementations for Direct Preference Optimization (DPO) and Reward Modeling (RM), enabling users to train models with these new capabilities.

src/llamafactory/v1/core · high confidence

Introduce MCA training workflow with Megatron-Core adapter support

Added a new training module under src/llamafactory/train/mca that enables training via the mcore\_adapter. This includes a custom trainer to handle 3D position IDs, a workflow module exposing run\_pt, run\_sft, and run\_dpo entry points, and specific support for Qwen-VL series models (including qwen3.5) with logic to freeze vision/language components and manage multimodal data collation.

src/llamafactory/train/mca · high confidence

Introduce OpenAI-compatible REST API server

The \src/llamafactory/api\ package now provides a new FastAPI-based server that exposes an OpenAI-compatible interface for chat completions and score evaluations. Users can interact with the model via standard endpoints (\/v1/chat/completions\, \/v1/score/evaluation\, \/v1/models\) supporting streaming, tool calls, and multimodal inputs (images, audio, video). The API includes built-in security features such as optional API key authentication, local file access restrictions, and SSRF protection for remote media URLs.

src/llamafactory/api · high confidence

Introduce PPO-based reinforcement learning finetuning

Users can now perform Proximal Policy Optimization (PPO) finetuning to align models using reinforcement learning. This change adds a new PPO training module that integrates with the existing LlamaFactory workflow, supporting features such as DeepSpeed ZeRO-3, custom reward models (including remote API rewards), and compatibility with TRL versions 0.8.6 through 0.9.6.

src/llamafactory/train/ppo · high confidence

Introduce dedicated Reward Model training workflow

Adds a new \src/llamafactory/train/rm\ module that provides a complete pipeline for training reward models. This includes a \PairwiseTrainer\ that computes pairwise loss, supports disabling data shuffling, integrates BAdam optimization, and correctly saves safetensors checkpoints. The \run\_rm\ workflow handles loading the model with a value head, executing training/evaluation/prediction, and saving prediction results.

src/llamafactory/train/rm · high confidence

Introduce dedicated pretraining workflow with FP8 and BAdam support

A new pretraining module (src/llamafactory/train/pt) is added, providing a standalone workflow for causal language modeling that integrates custom optimizer and scheduler creation, FP8 training configuration (including Transformer Engine backend support), and BAdam optimizer integration. The workflow also supports disabling data shuffling via a new configuration flag and includes logic to handle processor saving and loss plotting during training and evaluation.

src/llamafactory/train/pt · high confidence

Introduce new Gradio-based Web UI

The LLaMA Factory Web UI has been completely rewritten using Gradio, replacing the previous implementation. This new interface introduces a modular architecture with dedicated components for managing Gradio elements, handling training and inference runners, and controlling model loading. Key user-facing features include native support for reasoning models with collapsible 'thinking' process displays, the ability to switch between model hubs (Hugging Face, ModelScope, OpenMind), and integration with SwanLab for experiment tracking. The UI also supports advanced configuration options such as extra inference arguments, quantization methods (BNB, HQQ, EETQ), and booster settings (FlashAttention-2, Unsloth, Liger Kernel).

src/llamafactory/webui · high confidence

Introduce v1 configuration and CLI sampling infrastructure

This change introduces the v1 configuration layer and a new CLI sampler for LlamaFactory. The \src/llamafactory/v1/config\ package now provides a structured argument system using dataclasses (\ModelArguments\, \DataArguments\, \TrainingArguments\, \SampleArguments\) and a new \arg\_parser\ that supports YAML, JSON, and CLI argument parsing with OmegaConf. It also defines core enums like \BatchingStrategy\ (normal, padding-free, dynamic) and \SampleBackend\ (HF, vLLM). Additionally, a new CLI sampler (\src/llamafactory/v1/samplers/cli\_sampler.py\) is added, enabling synchronous chat and batch inference via a command-line interface.

src/llamafactory/v1/config · high confidence

Introduce v1 launcher for distributed training and command routing

The new \src/llamafactory/v1/launcher.py\ module serves as the entry point for the v1 training pipeline, handling command routing for SFT, DPO, and reward modeling (rm) tasks. It automatically detects multi-GPU environments and relaunches the process via \torchrun\ to enable distributed training, supporting both single-node and multi-node configurations with elastic fault-tolerance options. The launcher also provides specific handlers for chat and model merge workflows, ensuring that training commands are correctly forwarded to their respective v1 trainers.

src/llamafactory/v1 · high confidence

Introduce v1 launcher with elastic and fault-tolerant distributed training

The LLaMA-Factory CLI now supports a new v1 backend for training, activated via the USE\_V1 environment variable. When enabled, the launcher provides elastic and fault-tolerant distributed training capabilities using PyTorch Elastic (torchrun with rdzv\_id), allowing for automatic restarts and multi-node coordination. This change also introduces environment variables to control distributed execution (FORCE\_TORCHRUN, OPTIM\_TORCH) and integrates support for KTransformers and Megatron-bridge backends, while maintaining backward compatibility with the legacy launcher when USE\_V1 is not set.

src/llamafactory · high confidence

Introduces v1 utility infrastructure for training and data handling

The \src/llamafactory/v1/utils\ package is added, providing core support for the v1 training engine. This includes a plugin system for extensible components, a dtype registry for managing precision (fp16, bf16, fp32) and availability checks, and deterministic seeding utilities for reproducible distributed training. It also defines strict type definitions for multimodal inputs (text, images, video, audio) and batch structures, alongside helpers for logging, environment configuration, and package version validation.

src/llamafactory/v1/utils · high confidence

Introduction of v1 plugin architecture for batching and learning rate scheduling

The v1 trainer plugin system is introduced, establishing a modular structure for core training components. This change adds a new \trainer\_plugins\ package containing a \BatchingPlugin\ family (including a \PaddingFreeBatcher\ for optimized sequence packing) and an \LRSchedulerPlugin\ base class, replacing or supplementing previous monolithic implementations with a plugin-based approach for managing batch generation and learning rate strategies.

_src/llamafactory/v1/plugins/trainer\plugins · high confidence

New LoRA training examples for Qwen3 and MOSS-VL models

Added configuration files and a shell script in the \examples/train\_lora\ directory to support LoRA fine-tuning for the Qwen3 (4B) and MOSS-VL model families. The new examples cover multiple training stages including Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO), reward modeling, and pretraining for Qwen3, as well as SFT for the multimodal MOSS-VL model. These files provide ready-to-use templates for users to adapt these specific models with LoRA techniques.

_examples/train\lora · high confidence

New NPU and SUPA fused RMSNorm and RoPE kernels for Qwen3 models

This change introduces new kernel plugins for NPU and SUPA accelerators to accelerate specific operations in Qwen3-family models. The NPU RMSNorm plugin adds fused implementations for standard, residual, and gated RMSNorm variants, specifically targeting Qwen3, Qwen3-MoE, Qwen3-Next, Qwen3-Omni-MoE, Qwen3-VL, and Qwen3.5 model types. The NPU RoPE plugin provides fused rotary embedding acceleration with automatic Partial RoPE support for the same Qwen3 model families. Additionally, a SUPA fused RMSNorm kernel is added to accelerate standard RMSNorm operations (e.g., Llama, Mistral, Qwen2) on SUPA hardware using the \sudnn\ operator.

_src/llamafactory/v1/plugins/model\_plugins/kernels/ops/rms\norm · high confidence

New Ulysses sequence parallelism and GDN attention support

The parallelization plugin now includes a new Ulysses sequence parallelism implementation and support for GDN (linear attention) layers. This adds new files for Ulysses attention, GDN attention, sequence communication primitives, and a language-boundary hook to enable efficient multi-GPU training for models using these architectures.

_src/llamafactory/v1/plugins/model\plugins/parallelization · high confidence

New distributed training plugins for DeepSpeed and FSDP2

The distributed training backend now includes dedicated plugins for DeepSpeed and FSDP2, replacing manual initialization with structured engine classes. The DeepSpeed plugin leverages the \accelerate\ library to handle model sharding, optimizer wrapping, and ZeRO-3 bucket size resolution, while the FSDP2 plugin implements native PyTorch Fully Sharded Data Parallel 2 support with custom checkpointing via \torch.distributed.checkpoint\. These changes enable users to configure distributed training strategies through the existing plugin system, supporting features like mixed precision, gradient accumulation, and efficient state saving/loading for large-scale model training.

_src/llamafactory/v1/plugins/trainer\plugins/distributed · high confidence

New evaluation module for benchmarking model performance

A new evaluation module has been added to the LlamaFactory package, introducing an Evaluator class that loads a model and tokenizer, processes benchmark datasets (such as MMLU or similar subjects defined in SUBJECTS), and runs batched inference to calculate accuracy scores. Users can now run standardized evaluations by specifying tasks and splits, with results saved to JSON and log files for easy tracking of model performance across different categories.

src/llamafactory/eval · high confidence

New fused MoE and SwiGLU kernels for NVIDIA GPUs and NPUs

This change introduces new optimized kernel implementations for Mixture-of-Experts (MoE) and SwiGLU activation layers. For NVIDIA GPUs, a pure-Triton fused MoE kernel replaces the standard per-expert Python loop with a fully fused pipeline (scatter, grouped GEMM, activation, gather) for forward and backward passes, supporting models like Mixtral and Qwen3-MoE. For Huawei NPUs, new fused MoE kernels (using \torch\_npu.npu\_grouped\_matmul\) and a fused SwiGLU kernel are added, patching specific Qwen3 and Qwen3.5 model variants to leverage hardware-accelerated grouped matrix multiplications and SwiGLU operations.

_src/llamafactory/v1/plugins/model\plugins/kernels/ops/mlp · high confidence

New kernel plugin system with Liger and NPU optimizations

The model plugin architecture now includes a dedicated kernel optimization subsystem. This introduces a registry-based plugin system for applying hardware-specific kernel patches, starting with support for Liger Kernel (covering Qwen3, Qwen3 MoE, and related variants) and a suite of NPU-optimized kernels (Fused MoE, RMSNorm, RoPE, and SwiGLU). Users can now enable these optimizations via configuration, with automatic device detection applying the appropriate NPU kernels or allowing manual selection of Liger operations.

_src/llamafactory/v1/plugins/model\plugins/kernels · high confidence

New model conversion and tiny-model generation scripts

Added scripts to convert Baichuan2 and Qwen model checkpoints into the LLaMA2 format (scripts/convert\_ckpt/llamafy\_baichuan2.py, scripts/convert\_ckpt/llamafy\_qwen.py), and introduced scripts to generate tiny versions of Llama4 and Qwen3 models for testing (scripts/convert\_ckpt/tiny\_llama4.py, scripts/convert\_ckpt/tiny\_qwen3.py).

_scripts/convert\ckpt · high confidence

New model conversion, fine-tuning, and inference scripts

Added a suite of new scripts to the repository: dcp2hf.py and hf2dcp.py for converting checkpoints between Distributed Checkpoint (DCP) and HuggingFace formats; llama\_pro.py for performing block expansion on LLaMA, Mistral, Qwen2, or Yi models; loftq\_init.py and pissa\_init.py for initializing LoRA weights using LoftQ and PiSSA techniques respectively; megatron\_merge.py for converting Megatron/Core-Adapter checkpoints to HuggingFace format; qwen\_omni\_merge.py for merging Qwen-Omni model components; vllm\_infer.py for batched inference using the vLLM engine; and eval\_bleu\_rouge.py for computing BLEU and ROUGE metrics on prediction results.

scripts · high confidence

New model profiling and dataset analysis utilities

Added four new scripts to the \scripts/stat\_utils\ directory to help users analyze model and dataset characteristics before training. \cal\_flops.py\ calculates FLOPs, MACs, and parameter counts using DeepSpeed's profiler. \cal\_lr.py\ suggests an optimal learning rate based on the LLaMA paper's scaling laws, accounting for valid token ratios and model type (e.g., Mistral/Gemma). \cal\_mfu.py\ measures Model FLOPs Utilization (MFU) by running a short training step and comparing actual throughput against theoretical device capabilities. \length\_cdf.py\ computes the cumulative distribution of input sequence lengths in a dataset to help users choose appropriate cutoff lengths.

_scripts/stat\utils · high confidence

New modular CUDA Docker setup with optional Megatron Bridge support

The \docker/docker-cuda\ directory now provides a structured, multi-file Docker configuration for running LLaMA Factory on NVIDIA GPUs. The new \Dockerfile\ uses PyTorch 2.6.0 (CUDA 12.4) as its base and introduces build arguments to optionally install Flash Attention (\INSTALL\_FLASHATTN\) and configure HTTP proxies during the build. A separate \Dockerfile.base\ pre-installs Flash Attention 2.7.4 and FlashInfer 0.2.2 to optimize image layers. Additionally, \Dockerfile.mbridge\ and \Dockerfile.megatron\ provide specialized runtimes for Megatron Bridge training, bundling PyTorch 2.12.1/2.8.0, TransformerEngine, and Megatron Core. The \docker-compose.yml\ has been updated to use the new \Dockerfile\ and explicitly reserves all available GPUs.

docker/docker-cuda · high confidence

New training backends and FP8 support

The training module now supports three new backend engines—Megatron Bridge, HyperParallel, and Megatron Core Adapter (MCA)—allowing users to run pretraining and SFT stages with these specialized distributed frameworks. Additionally, FP8 training is now available via HuggingFace Accelerate, supporting both the TorchAO and Transformer Engine backends. The module also introduces several new training callbacks, including PiSSA adapter conversion, processor saving, and integration with SwanLab, Ray, and the Torch profiler.

src/llamafactory/train · high confidence

New training callback system with persistent JSONL logging

The v1 training pipeline now includes a new callback infrastructure in \src/llamafactory/v1/utils/callbacks\. This introduces a \TrainerCallback\ base class with lifecycle hooks (e.g., \on\_step\_begin\, \on\_log\, \on\_save\) and a \CallbackHandler\ to manage them. A default \LoggingCallback\ is provided, which logs training metrics to stdout (with improved formatting for small learning rates) and persists every log entry as a JSON line in \\<output\_dir\>/trainer\_log.jsonl\, ensuring training history survives crashes.

src/llamafactory/v1/utils/callbacks · high confidence

New v1 accelerator module with multi-device support

The v1 accelerator package has been introduced, providing a unified interface for distributed training across CPU, CUDA, MPS, NPU, and XPU devices. This module includes helper utilities for environment detection, device synchronization, and collective communication operations, enabling the framework to automatically select the appropriate backend (such as NCCL for CUDA or HCCl for NPU) based on the available hardware.

src/llamafactory/v1/accelerator · high confidence

New v1 core batching, collation, and checkpoint utilities

The \src/llamafactory/v1/core/utils\ package introduces the foundational utilities for the v1 training and inference pipeline. This includes a stateful \BatchGenerator\ that supports dynamic and normal batching strategies, along with collation logic that correctly handles multimodal data (images, video, audio) by aligning and truncating media features. Checkpointing is now managed by dedicated helpers that save and restore model, optimizer, and per-rank RNG states to ensure reproducible training and reliable resume-from-checkpoint functionality. Additionally, an asynchronous inference engine (\HuggingFaceEngine\) is provided to support streaming token generation and batch inference.

src/llamafactory/v1/core/utils · high confidence

New v1 data plugin system for loading and converting datasets

The v1 data processing pipeline now uses a plugin-based architecture in \src/llamafactory/v1/plugins/data\_plugins\ to handle dataset ingestion and format conversion. The new \loader.py\ module supports loading local datasets from files (Arrow, CSV, JSON, Parquet, Text) with configurable splits and streaming, while \converter.py\ introduces specific converters for Alpaca and ShareGPT formats. These converters normalize raw samples into a unified SFT structure and support multimodal content (images, videos, audio) via inline placeholders that are parsed into distinct content blocks.

_src/llamafactory/v1/plugins/data\plugins · high confidence

New v1 model plugin infrastructure for initialization and DeepSpeed ZeRO-3 support

The v1 model plugin system now includes dedicated modules for handling model initialization strategies and DeepSpeed ZeRO-3 configuration. Users can select initialization modes such as 'init\_on\_meta' (loading weights on meta device) or 'init\_on\_rank0' (loading on rank 0 only) to optimize memory usage during distributed training. Additionally, the new deepspeed\_utils module provides automatic detection and configuration for DeepSpeed ZeRO-3 model loading, including mixed-precision inference (bf16) and proper handling of distributed config files, ensuring seamless integration with the accelerator for low-memory weight loading scenarios.

_src/llamafactory/v1/plugins/model\plugins · high confidence

Unified chat interface with multi-backend support

The chat module now provides a single \ChatModel\ entry point that automatically routes inference to the Hugging Face, vLLM, or SGLang backend based on configuration. This unified interface supports synchronous and asynchronous chat, streaming, and scoring, while handling multimodal inputs (images, videos, and audio) consistently across all three engines.

src/llamafactory/chat · high confidence

Removals

Removal of HhRlhfEn dataset builder

The dataset builder script for the Anthropic 'hh-rlhf' dataset (hh\_rlhf\_en.py) has been removed from the codebase. This change eliminates the local implementation that previously parsed and structured the harmless-base, helpful-base, helpful-online, and helpful-rejection-sampled splits into instruction, output, and history fields for ChatGLM-style multi-turn chat formatting.

_data/hh\_rlhf\en · high confidence

Removal of LLaMA-specific utility modules

The \src/utils\ package has been removed, eliminating the LLaMA-specific implementation details for data collation, configuration, and training loops. This includes the deletion of \common.py\ (model loading and adapter initialization), \config.py\ (argument definitions), \data\_collator.py\ (left-padding logic), \other.py\ (helper utilities like logits processing and device mapping), \pairwise.py\ (reward modeling trainer), \peft\_trainer.py\ (PEFT checkpoint handling), \ppo.py\ (PPO training loop), and \seq2seq.py\ (generation metrics and prediction saving).

src/utils · high confidence

Removal of UltraChat dataset loader

The custom dataset loading script for UltraChat has been removed from the codebase. This change eliminates the ability to load the UltraChat multi-turn dialogue data using the previous local implementation, which previously handled downloading and parsing the JSONL shards from the Hugging Face repository.

_data/ultra\chat · high confidence

Removal of example dataset builder

The example dataset builder script (example\_dataset.py) has been removed from the data/example\_dataset directory. This file previously defined a GeneratorBasedBuilder for loading an example LLaMA dataset containing instruction, input, output, and history fields from a JSON source.

_data/example\dataset · high confidence

Architecture

Refactored model utilities into modular components

The model utility logic has been reorganized into dedicated modules (attention, checkpointing, embedding, kv\_cache, liger\_kernel, longlora, misc, mod, moe, packing, quantization). This change improves code maintainability and separation of concerns without altering user-facing behavior.

_src/llamafactory/model/model\utils · high confidence

Web UI components are restructured into a modular package

The web UI interface is now organized into a dedicated \src/llamafactory/webui/components\ package, with individual modules for the chatbot, data preview, evaluation, export, footer, inference, top-level settings, and training tabs. This refactoring centralizes the Gradio UI logic, making the web interface easier to maintain and extend while preserving the existing user-facing features.

src/llamafactory/webui/components · high confidence

Behavioural changes

4 commits (1 fix) modifying data/mllm\_demo\_data

A change to existing behaviour in data/mllm\_demo\_data — 4 commits (1 fix), 11 files: “video datasets”; “\[data\] Fix bugs of \use\_audio\_in\_video\ in Qwen2.5 Omni (\#7638)”; “\[model\] support audio (\#6701)”; and 1 more (refs \#7638, \#6701, \#9224).

_data/mllm\_demo\data · medium confidence · unverified

Introduce LLaMA-Factory v0.9.6.dev0 with updated dependency constraints and runtime utilities

This release sets the development version to 0.9.6.dev0 and enforces stricter dependency ranges: transformers (\>=4.55.0, \<=5.8.0, excluding 4.57.0 and 5.6.0), datasets (\>=2.16.0, \<=4.0.0), accelerate (\>=1.3.0, \<=1.15.0), peft (\>=0.18.0, \<=0.20.0), and trl (\>=0.18.0, \<=0.24.0). The \extras\ module now provides a comprehensive environment reporting tool (\print\_env\) that displays versions for all core libraries, GPU/NPU details, and git commit hashes, alongside a robust logging system with rank-aware handlers for multi-GPU training. New utility functions include \calculate\_tps\ for measuring training throughput, \count\_parameters\ for model size analysis, and device detection logic that explicitly supports Intel XPU, Ascend NPU, and MPS in addition to CUDA. The package availability checks have been patched to return version information when available, and loss plotting capabilities have been added to visualize training metrics in LlamaBoard.

src/llamafactory/extras · high confidence

Introduce structured SFT training workflow with advanced loss and optimization support

The supervised fine-tuning (SFT) module has been reorganized into a dedicated package (\src/llamafactory/train/sft\) containing a new entry-point workflow, a custom trainer, and a public API. This change enables users to disable training data shuffling, apply specialized loss functions (DFT, EAFT, and ASFT) that utilize reference models, and leverage the BAdam optimizer. The trainer also now supports FP8 training via the Transformer Engine backend, integrates KTransformers for AMX MoE acceleration, and provides more accurate generation metrics by correctly stripping prompt tokens from generated outputs during evaluation.

src/llamafactory/train/sft · high confidence

New modular data processor architecture for training datasets

The data pipeline has been refactored to use a new modular processor structure in \src/llamafactory/data/processor\. This change introduces specific dataset processors for different training modes: \FeedbackDatasetProcessor\ (supporting KTO-style preference learning with KL mismatch pair creation), \PairwiseDatasetProcessor\ (for chosen/rejected pair formatting), \UnsupervisedDatasetProcessor\, \SupervisedDatasetProcessor\, \PackedSupervisedDatasetProcessor\, \PretrainDatasetProcessor\, and \DatasetProcessor\ utilities. Users benefit from a more structured and extensible data preparation layer that handles tokenization, multi-modal plugin integration, and sequence length inference consistently across all training types.

src/llamafactory/data/processor · high confidence

Refactored data pipeline with modular converters and cloud storage support

The data loading and preprocessing logic has been restructured into a modular pipeline. A new \converter.py\ module introduces a \DatasetConverter\ base class with specific implementations (e.g., \AlpacaDatasetConverter\, \SharegptDatasetConverter\) to standardize dataset formats, handling history, prompts, responses, and multi-modal media. The \loader.py\ module now explicitly supports loading datasets from Hugging Face, ModelScope, and OpenMind hubs, as well as local files and cloud storage (S3/GCS) via \fsspec\. Utility functions in \data\utils.py\ provide robust dataset merging (concat/interleave), splitting, and cloud JSON reading, while \\\init\\_.py\ exposes the new public API including collators and templates.

src/llamafactory/data · high confidence

Refactored data pipeline with modular processors and multimodal support

The data loading and preprocessing system has been restructured to use a modular processor architecture. New abstract and concrete classes (DatasetProcessor, PretrainDatasetProcessor, SupervisedDatasetProcessor, PackedSupervisedDatasetProcessor) now handle data preparation, replacing the previous monolithic approach. This change introduces dedicated data collators (MultiModalDataCollatorForSeq2Seq) and plugins (mm\_plugin.py) to properly handle images, videos, and audio inputs, including complex logic for multimodal rope position indexing. A new parser (parser.py) standardizes dataset attribute loading, and utility scripts for benchmarking (bench\_qwen.py) and perplexity calculation (cal\_ppl.py) have been added to support these new capabilities.

llamafactory · high confidence

Refactored entry points for API, training, and Web UI

The project has reorganized its command-line entry points to provide a cleaner, more modular interface. The previous standalone scripts for the CLI demo, web demo, model export, and specific training stages (SFT, PPO, Reward Model) have been removed. They are replaced by three new unified entry points: \api.py\ for launching the OpenAI-compatible API server, \train.py\ for executing training experiments, and \webui.py\ for the Gradio-based Web UI. This change centralizes the application logic, delegating specific tasks to internal modules (such as \llamafactory.api.app\, \llamafactory.train.tuner\, and \llamafactory.webui.interface\) rather than handling them directly in the top-level scripts.

src · high confidence

Test coverage

Added API example tests for image understanding and tool calling; Added comprehensive test suite for the data processing pipeline; Added end-to-end tests for chat, training, and SGLang inference; Added test for SFT trainer data shuffling behavior; Added tests for DataEngine and ModelEngine; Added tests for FSDP2 meta-loading consistency and FSDPTurbo plugin configuration; Added tests for Megatron Bridge training integration; Added tests for data converter plugins; Added tests for evaluation template formatting; Added tests for the v1 distributed accelerator interface; Added tests for the v1 rendering engine; Added tests for v1 DPO loss precision and FSDP2 trainer integration; Added tests for v1 batching strategies; Added tests for v1 configuration argument parsing; Added tests for v1 model plugins; Added unit tests for data processor stages; Added unit tests for model finetuning and inference modes; Added unit tests for model utilities; Added unit tests for the CLI synchronous sampler; New test infrastructure and license validation; New v1 test infrastructure with device-aware fixtures and markers.

Dependencies

Migration to pyproject.toml and Python 3.11+ requirement

The project has migrated its build system from setuptools to hatchling, defined in a new pyproject.toml, and raised the minimum supported Python version to 3.11. This change updates the dependency specifications for core libraries such as transformers (now supporting versions up to 5.8.0) and TRL (0.18–0.24), and introduces a dedicated docs/requirements.txt for Sphinx-based documentation. Users must upgrade their Python environment and install dependencies via the new pyproject.toml configuration.

(dependencies) · high confidence

Housekeeping

Added v1 plugin directory structure; Repository initialization with project scaffolding and documentation.

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 52 → 52 (+0.1)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 54 → 53 (-0.9)
  • Architecture 76 → 79 (+3.8)
  • Maturity 61 → 65 (+3.7)
  • Readiness 41 → 41 (+0.1)
  • Security 70 → 65 (-4.8)
  • Performance 100 → 100 (+0.0)

Resolved (32)

  • Duplicate FunctionCall definition. The API protocol includes an 'id' and 'type' field (OpenAI style), while the internal data tool utils uses a simpler structure with just name and arguments. This forces conversion logic between the API layer and the internal processing layer.
  • Duplicate Role definitions with divergent values. The API protocol defines 'TOOL' as a role, while the data utils defines 'OBSERVATION'. These represent the same semantic concept (tool output/observation) but are named differently in different parts of the public surface, causing confusion for users implementing custom formatters or parsers.
  • Duplicated block (15 lines × 2) (src/llamafactory/launcher.py)
  • Duplicated block (15 lines × 3) (src/llamafactory/model/model_utils/longlora.py)
  • Duplicated block (9 lines × 2) (scripts/dcp2hf.py)
  • Hotspot: src/llamafactory/model/adapter.py (src/llamafactory/model/adapter.py)
  • Hotspot: src/llamafactory/model/loader.py (src/llamafactory/model/loader.py)
  • Hotspot: src/llamafactory/train/hyper_parallel/workflow.py (src/llamafactory/train/hyper_parallel/workflow.py)
  • Hotspot: src/llamafactory/train/tuner.py (src/llamafactory/train/tuner.py)
  • Hotspot: src/llamafactory/v1/core/model_engine.py (src/llamafactory/v1/core/model_engine.py)
  • Hotspot: src/llamafactory/v1/core/rendering/rendering.py (src/llamafactory/v1/core/rendering/rendering.py)
  • Hotspot: src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdp2.py (src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdp2.py)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Leaked secret: hardcoded-credential (data/glaive_toolcall_en_demo.json)
  • Near-duplicate member family (3 members, 40 shared lines) (src/llamafactory/model/model_utils/longlora.py)
  • …and 12 more

New (56)

  • Duplicated block (9 lines × 2) (src/llamafactory/model/patcher.py)
  • Edited copies of one member (3 members, 13+ corresponding lines) (src/llamafactory/v1/trainers/dpo_trainer.py)
  • Edited copies of one member (3 members, 16+ corresponding lines) (src/llamafactory/data/tool_utils.py)
  • Edited copies of one member (3 members, 25+ corresponding lines) (src/llamafactory/v1/plugins/model_plugins/kernels/ops/mlp/npu_fused_moe.py)
  • Edited copies of one member (3 members, 4+ corresponding lines) (src/llamafactory/data/processor/feedback.py)
  • Edited copies of one member (5 members, 3+ corresponding lines) (src/llamafactory/train/dpo/trainer.py)
  • Edited copies of one member (5 members, 4+ corresponding lines) (src/llamafactory/train/dpo/trainer.py)
  • Edited copy of a member (10 corresponding lines) (src/llamafactory/data/mm_plugin.py)
  • Edited copy of a member (11 corresponding lines) (src/llamafactory/v1/plugins/trainer_plugins/batching.py)
  • Edited copy of a member (12 corresponding lines) (scripts/convert_ckpt/llamafy_baichuan2.py)
  • Edited copy of a member (14 corresponding lines) (src/llamafactory/data/tool_utils.py)
  • Edited copy of a member (14 corresponding lines) (src/llamafactory/train/trainer_utils.py)
  • Edited copy of a member (16 corresponding lines) (src/llamafactory/v1/trainers/dpo_trainer.py)
  • Edited copy of a member (17 corresponding lines) (src/llamafactory/data/mm_plugin.py)
  • Edited copy of a member (17 corresponding lines) (src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdp2.py)
  • Edited copy of a member (18 corresponding lines) (src/llamafactory/v1/plugins/model_plugins/kernels/ops/rms_norm/npu_rms_norm.py)
  • Edited copy of a member (19 corresponding lines) (src/llamafactory/chat/hf_engine.py)
  • Edited copy of a member (19 corresponding lines) (src/llamafactory/v1/plugins/model_plugins/kernels/ops/mlp/cuda_fused_moe.py)
  • Edited copy of a member (19 corresponding lines) (src/llamafactory/v1/plugins/model_plugins/kernels/ops/mlp/npu_fused_moe.py)
  • Edited copy of a member (20 corresponding lines) (src/llamafactory/data/template.py)
  • …and 36 more

Changes since last survey

  • 15 commits — 7 feature/other, 8 fixes

By area

  • src/llamafactory — 8 commits
  • .github/workflows — 2 commits
  • (root) — 1 commit
  • .agents/skills — 1 commit
  • docker/docker-npu — 1 commit
  • requirements/vllm.txt — 1 commit
  • tests_v1/plugins — 1 commit

Notable commits

  • fix: Revert "[ci] Trigger docs workflow for README PRs (#10959)" (#10961)
  • fix: [data] fix: parse nested JSON literals in LFM2 tools (#10940)
  • fix: [docs] fix stale paths in the README (#10934)
  • fix: [npu] Fix fused MoE backward, GDN patching and EP freezing (#10951)
  • fix: [v1] Fix custom model export with FSDP2 (#10916)
  • fix: [v1] fix: avoid HashStore in FSDP2 unit test on Windows (#10943)
  • fix: fix model type cast (#10971)
  • fix: fix(longlora): extend device type check to include Ascend NPU (#10829)
  • change: [ci] Trigger docs workflow for README PRs (#10959)
  • change: [deps] upgrade vllm to 0.28.0 (#10857)
  • change: [docker] update CANN to 9.2.0 and torch-npu to 2.10.0.post7 (#10925)
  • change: [model] add Xing4 model support (#10754)
  • change: [npu] Skip unnecessary gradients in NPU fused MoE backward (#10918)
  • change: [train] migrate HyperParallel model plans to replace schema (#10962)
  • change: docs: add Qwen3.8 LlamaFactory SFT skill (#10901)

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

hiyouga/LlamaFactory was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 5f1f6ad537e69216e0604671223cc1eeecbb7ba6 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.