Skip to content
CAI
Software that uses CAICheck a score

kvcache-ai/ktransformers

36.6

Weak · 5 November 2026

132.2k

lines of production code

Python

with C++

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Archive kt-sft module and introduce balance-serve distributed inference

The kt-sft package is archived, moving its components into the main ktransformers server directory. This change introduces a new 'balance-serve' module that enables distributed, multi-GPU inference. It includes a pure-Python CUDART wrapper for CUDA operations, a PyNccl wrapper for NCCL communication, and a parallel state manager to handle model parallel groups. Additionally, it adds a batched penalty orchestrator for sampling constraints (like repetition and frequency penalties) and updates the server's argument definitions and chat completion schemas to support these new distributed capabilities.

ktransformers · high confidence

Initial release of kt-kernel with KVCache operator and performance demos

This change introduces the kt-kernel module, adding a new Key-Value (KV) Cache operator for attention mechanisms and a suite of C++ and Python benchmarking tools. The KVCache implementation (kt-kernel/operators/kvcache) provides a configurable cache structure supporting fp16 and Q4\_0/Q8\_0 quantization, with features for block-based storage, anchor-based retrieval strategies (FIXED\_ANCHOR, DYNAMIC, QUEST, etc.), and serialization (load/dump) capabilities. Alongside the operator, the kt-kernel/demo directory adds several performance demonstration programs: simple\_test.cpp and simple\_test\_aocl.cpp for int8 GEMM throughput, fp16-test.cpp and bf16-test.cpp for floating-point matrix multiplication benchmarks, and bandwidth benchmarks for data reordering. These demos allow users to measure TFLOPS and bandwidth on supported hardware configurations.

kt-kernel/demo, kt-kernel/operators/kvcache · high confidence

Introduce CPU backend infrastructure with Ascend NPU callback support

This change adds the core CPU backend implementation for kt-kernel, introducing a task queue, a NUMA-aware worker pool, and shared memory buffers to handle asynchronous inference tasks. It specifically adds support for Huawei Ascend NPUs by implementing a dedicated callback worker that manages ACL stream reports, ensuring that host callbacks are correctly dispatched and preventing hangs during stream synchronization.

_kt-kernel/cpu\backend · high confidence

Introduce KVC2 disk-backed KV cache with async I/O and NPU support

The KVC2 module in the balance serve layer now provides a disk-backed key-value cache implementation. It introduces an asynchronous I/O subsystem (async\_store) that offloads read/write operations to background threads using either io\_uring or libaio, and adds a CudaStreamManager that supports both CUDA and Ascend NPU devices for managing GPU/NPU memory streams. The cache manages data via page-aligned memory pools and GPU page caches, persisting KV blocks to disk and exposing metrics (prefix nodes, block counts, operation latencies) via Prometheus.

_archive/csrc/balance\serve/kvc2, archive/ktransformers/website · high confidence

Introduce kt-cli command-line interface

This change adds the \kt-cli\ command-line interface, providing a suite of new commands for managing and interacting with the system. Users can now run model inference servers (\kt run\), interact with models via an interactive chat interface (\kt chat\), and manage model lifecycles including downloading, listing, and configuring storage paths (\kt model\). Additional capabilities include quantizing model weights for CPU inference (\kt quant\), running performance benchmarks (\kt bench\), and diagnosing environment issues (\kt doctor\). The CLI also includes commands for managing configuration settings (\kt config\), viewing version information (\kt version\), and shell completion scripts for Bash, Zsh, and Fish to improve usability.

kt-kernel/python/cli/commands · high confidence

Introduce kt-kernel as a standalone high-performance CPU/GPU MoE inference backend

kt-kernel is now available as a separate package providing optimized MoE expert kernels for KTransformers, supporting CPU instruction sets (AVX2, AVX512, AMX) and multiple GPU backends (CUDA, SYCL, ROCm, MUSA, Ascend NPU, MACA). The package includes a C++ backend with Python bindings, a PyPI installation option, and an automated build system that detects CPU capabilities to select the optimal kernel variant at runtime.

kt-kernel · high confidence

Introduce kt-kernel for CPU-optimized MoE inference and SFT

Adds the kt-kernel Python package, providing high-performance kernel operations for KTransformers. The package automatically detects CPU capabilities (AMX, AVX512, or AVX2) and loads the optimal variant at runtime, with environment variable overrides. It exposes the KTMoEWrapper factory for both inference (supporting methods like AMXINT4, RAWINT4, FP8, LLAMAFILE) and supervised fine-tuning (SFT) with LoRA support (e.g., AMXBF16\_SFT, MXFP4\_SFT). The implementation includes base infrastructure for handling GPU/CPU expert offloading, NUMA node mapping, and specific backend integrations such as Ascend NPU callback workers and SYCL support.

kt-kernel/python · high confidence

Introduces kt-kernel SFT submodule for MoE fine-tuning

The new \kt-kernel/python/sft\ package provides a complete supervised fine-tuning (SFT) capability for Mixture-of-Experts (MoE) models, extending the inference-only base with full forward/backward pass support, LoRA adaptation, and distributed training utilities. It introduces a modular backend system supporting BF16, INT8, FP8, MXFP4, and RAWINT4 quantization, alongside architecture detection for models like DeepSeek, Kimi K2, Qwen MoE, and Mixtral. Users can now perform full-parameter and LoRA fine-tuning using hardware-specific AMX backends, manage expert weight artifacts and checkpoints, and leverage activation checkpointing and reuse policies for optimized training memory usage.

kt-kernel/python/sft · high confidence

Introduction of the KTransformers CLI (kt-cli)

A new unified command-line interface, \kt-cli\, is introduced to manage KTransformers workflows. This tool provides a user-friendly interface for model inference, fine-tuning, benchmarking, and environment diagnostics. Key capabilities include an interactive first-run setup wizard for language selection (English/Chinese) and model discovery, commands for downloading and quantizing models, running inference servers via SGLang, and diagnosing system health (GPU, CPU, memory). The CLI supports internationalization and integrates with existing components like \kt-kernel\ and \LlamaFactory\.

kt-kernel/python/cli · high confidence

New AMX-based BF16 GEMM backend with AVX512 fallback

The \kt-kernel/operators/amx/la\ directory now contains a complete implementation of an Intel AMX-accelerated matrix multiplication backend for BF16 data types. This includes optimized GEMM kernels, buffer management for BF16 and quantized formats (Q4/Q8), and a unified activation function entry point that supports both standard SiLU and the MiniMax M3 'swigluoai' variant with configurable clamping. To ensure compatibility on hardware lacking native AMX instructions, the implementation provides a fallback path using AVX512F+BW intrinsics for BF16 operations, allowing the system to utilize available vector extensions while maintaining functional correctness.

kt-kernel/operators/amx/la · high confidence

New AMX-based MoE operator implementations for multiple quantization formats

The \kt-kernel/operators/amx\ directory now includes a suite of new C++ header files implementing Mixture-of-Experts (MoE) operators optimized for Intel AMX and AVX512 instruction sets. These additions introduce native support for several quantization schemes: AWQ Int4 with KGroup quantization and zero-point support (\awq-moe.hpp\), native BF16 inference without quantization (\bf16-moe.hpp\), MXFP4 (FP4 E2M1 weights with BF16 activations) using AVX512 look-up tables (\fp4-moe.hpp\), FP8 with 128x128 block-wise scales (\fp8-moe.hpp\), FP8 with per-channel quantization (\fp8-perchannel-moe.hpp\), and Kimi-K2 Int4 with KGroup scales (\k2-moe.hpp\). A staging helper (\fp8\_tp\_staging.hpp\) is also added to manage tensor-parallel FP8 weight loading. These operators provide the underlying compute kernels for high-performance MoE inference on compatible hardware.

kt-kernel/operators/amx · high confidence

New AVX2 CPU backends for BF16, FP8, and INT4 MoE inference

Added a suite of AVX2-optimized kernels for Mixture-of-Experts (MoE) models, enabling high-performance inference on CPUs that support AVX2. The new backends include native BF16 GEMM and MoE operators, FP8 E4M3 MoE with LUT-based dequantization, and GPTQ INT4 MoE support (both standard and AVX-VNNI-256 variants). The AVX-VNNI-256 INT4 backend introduces a packed weight mode that keeps GPTQ int4 nibbles resident in memory, significantly reducing the weight footprint compared to pre-unpacked int8 formats. These additions provide users with efficient, quantization-aware execution paths for BF16, FP8, and INT4 models on compatible hardware.

kt-kernel/operators/avx2 · high confidence

New Ascend NPU deployment tools for DeepSeek-V4-Flash

Added a new toolset in \kt-kernel/tools/ascend\_dsv4\ to deploy DeepSeek-V4-Flash on a single Ascend NPU. This includes \setup.sh\ to build and install the required dependencies (kt-kernel, sgl-kernel, CANN ops) and \serve.sh\ to launch the SGLang server with Ascend-specific configurations. The suite also provides \verify.sh\ for acceptance testing and an interactive chat client, alongside \dsv4\_env.sh\ for centralized environment configuration. Additionally, new scripts in \kt-kernel/tools/mxfp4\_gguf\ allow converting native MXFP4 model weights to the per-layer GGUF format required by the kernel, and \kt-kernel/tools/ascendc\_mxfp4\ contains the fused AscendC kernel implementation for efficient MXFP4-to-W8A8 conversion.

kt-kernel/tools · high confidence

New CPU kernel operator library for MoE and MLA inference

The \kt-kernel/operators\ directory now contains a new set of C++ header files implementing core neural network operators for CPU inference. This includes Tensor Parallel (TP) wrappers for Mixture-of-Experts (MoE) layers (\moe-tp.hpp\, \moe-sft-tp.hpp\) and Multi-Head Latent Attention (MLA) (\mla-tp.hpp\), along with foundational primitives such as RMS normalization, RoPE (Rotary Positional Embeddings), Softmax, and reduction operations. The implementation supports specific hardware optimizations, including AMX for Intel CPUs and ARM SVE for aarch64, and introduces a detailed profiling system (\sft\_profile.hpp\) to track forward and backward pass stages for SFT (Supervised Fine-Tuning) workflows.

kt-kernel/operators · high confidence

New CUDA kernels for GGUF dequantization, GPTQ-Marlin GEMM, and MoE topk-softmax

This change introduces the \kt-kernel/cuda\ module, providing Python bindings (\KTransformersOps\) for new CUDA-accelerated operations. Users can now dequantize GGUF weights (q8\_0, q6\_k, q5\_k, q4\_k, q3\_k, q2\_k, and iq4\_xs) directly on the GPU, perform GPTQ-Marlin matrix multiplications for 4-bit quantized models, and execute optimized topk-softmax routing for Mixture-of-Experts (MoE) architectures. These kernels are built using PyTorch C++ extensions and support both FP16 and BF16 data types.

kt-kernel/cuda · high confidence

New Docker build and distribution tooling for KTransformers

Added a new Dockerfile and supporting shell scripts (build-docker-tar.sh, push-to-dockerhub.sh, docker-utils.sh) that enable building KTransformers images with automatic version detection, multi-CPU variant support (AMX, AVX512, AVX2), and standardized naming conventions. The tooling supports exporting images as local tar files for offline distribution and pushing to DockerHub with both full versioned tags and simplified tags, while allowing configuration of CUDA versions, Ubuntu mirrors, and proxy settings.

docker · high confidence

New Python utility package for MoE weight loading and CPU backends

The \kt-kernel/python/utils\ module has been introduced to centralize weight loading and CPU-based Mixture-of-Experts (MoE) execution. It provides \SafeTensorLoader\ and \GGUFLoader\ for reading model weights, alongside specialized wrappers: \AMXMoEWrapper\ for Intel AMX/AVX2/AVX-VNNI/SYCL backends, \LlamafileMoEWrapper\ for GGUF-based inference, and \GeneralMoEWrapper\ for INT4/INT8 quantization. This structure allows users to load and run MoE models on CPU hardware with support for various quantization formats and hardware-specific optimizations.

kt-kernel/python/utils · high confidence

New documentation for AMX optimization, DeepSeek-V4-Flash, and Intel XPU support

Added comprehensive English documentation for new capabilities: an AMX optimization guide detailing Intel Advanced Matrix Extensions for Qwen 3 MoE models; tutorials for running DeepSeek-V4-Flash with SGLang and KT-Kernel on both NVIDIA GPUs and Ascend NPUs; and a Docker guide for Intel XPU devices. These files provide users with hardware-specific setup instructions, configuration parameters, and performance benchmarks for heterogeneous CPU-GPU and CPU-NPU inference scenarios.

doc/en · high confidence

New kt-cli utility modules for model analysis, discovery, and environment detection

The kt-cli now includes a new set of utility modules in \kt-kernel/python/cli/utils\ to enhance model management and system interaction. \analyze\_moe\_model.py\ adds the ability to detect and analyze Mixture-of-Experts (MoE) architectures by leveraging sglang's model registry and caching results. \model\_discovery.py\ and \model\_scanner.py\ provide functions to scan local directories for model weights (safetensors/gguf) and automatically register them in the user model registry. \model\_registry.py\ expands the built-in supported model list with entries for DeepSeek-V4-Flash, MiniMax-M2/M2.1/M3, and Kimi-K2-Thinking, including specific inference parameters. \environment.py\ introduces comprehensive system detection for Python, CUDA, GPUs, and virtual environment managers. \console.py\ standardizes CLI output with Rich-based formatting, while \download\_helper.py\ supports listing and verifying files from HuggingFace and ModelScope repositories. \kv\_cache\_calculator.py\ allows users to estimate KV cache sizes for specific models and configurations.

kt-kernel/python/cli/utils · high confidence

New kt-kernel benchmark suite for CPU inference kernels

The kt-kernel/bench directory now includes a comprehensive set of Python-based performance benchmarks for the kt-kernel CPU inference library. These scripts allow users to measure throughput and latency for various kernel implementations, including native BF16 MoE, MXFP4 MoE (V4-Flash shape), FP8 MoE (block-wise and per-channel), AMX int4 MoE (Kimi K2/K2.6), and linear/attention kernels with diverse quantization modes (FP32, FP16, BF16, Q8\_0, Q6\_K, Q4\_K\_M, etc.). The benchmarks cover both custom kt-kernel backends and PyTorch baselines, providing detailed metrics like tokens per second and bandwidth.

kt-kernel/bench · high confidence

New kt-kernel examples for DeepSeek-V3, AMX MoE, and LLAMAFILE integration

The kt-kernel/examples directory now includes a comprehensive set of new scripts and configuration files. This adds a DeepSeek-V3 model implementation (configuration and modeling) alongside a corresponding validation test script. It also introduces benchmarking and testing utilities for AMX-accelerated MoE kernels, including support for INT8, BF16, and AWQ quantized weights. Additionally, a repro harness for LLAMAFILE-based inference is provided to validate CPU-GPU expert scheduling and CUDA stream integration.

kt-kernel/examples · high confidence

New llamafile-based CPU inference operators for linear, MLP, and MoE layers

This change introduces a new set of CPU inference operators in the \kt-kernel/operators/llamafile\ directory, implementing linear, multi-layer perceptron (MLP), and mixture-of-experts (MoE) layers using the llamafile sgemm backend. The implementation includes quantization-aware conversion utilities (\conversion.h\) and handles various \ggml\_type\ quantizations (e.g., Q8\_K, BF16) by managing intermediate float buffers and leveraging work-stealing job pools for parallel execution. It also includes a specific workaround for ARM NEON cores without SVE to ensure BF16 weights are correctly processed via F32 paths, and supports weight aliasing to avoid unnecessary memory copies when loading from GGUF files.

kt-kernel/operators/llamafile · high confidence

New weight conversion and CPU feature detection scripts for CPU-GPU hybrid inference

Added a suite of Python scripts in \kt-kernel/scripts\ to support heterogeneous expert placement in CPU-GPU hybrid inference. \convert\_cpu\_weights.py\ and \convert\_cpu\_weights\_ds4.py\ quantize model weights to INT4/INT8 (and FP4 for DeepSeek V4) optimized for AMX on CPU, while \convert\_gpu\_weights.py\ applies GPTQ/RTN quantization for GPU-resident experts. \convert\_kt\_to\_sglang\_adapter.py\ and \export\_dsv4\_sglang\_adapter.py\ convert KT SFT fused expert LoRA checkpoints into SGLang-compatible adapter formats. Additional utilities include \check\_cpu\_features.py\ for verifying CPU instruction set support (AMX/AVX512), \compare\_weights.py\ for validating quantization accuracy, and \convert\_moe\_to\_bf16.py\ for dequantizing MoE experts. The \README.md\ documents these tools and their usage.

kt-kernel/scripts · high confidence

Architecture

Repository restructured around kt-kernel with new top-level package and install flow

The project has been reorganized to focus on the kt-kernel source tree, exposing two main user capabilities: high-performance inference and fine-tuning (SFT) via LlamaFactory. A new top-level Python package (ktransformers.py) provides version detection and an SFT availability check, while a new install.sh script offers a unified, subcommand-driven installer for system dependencies, the sglang integration, and kt-kernel. Documentation has been consolidated into a new README structure with separate entry points for Inference and SFT, and a MAINTAINERS.md file has been added to define contributor responsibilities.

(repo-wide) · high confidence

Behavioural changes

Archive legacy KTransformers framework code

The original integrated KTransformers framework has been moved to the \archive/\ directory for reference, as the project now focuses on the modular \kt-kernel\ and \kt-sft\ components. This change includes the preservation of legacy documentation, build configurations (Dockerfiles, CMakeLists), and source code to maintain historical context while streamlining the active repository structure.

archive · high confidence

Enforce conventional commit messages and auto-format code on commit

The kt-kernel area now installs git hooks that enforce stricter development workflows. The commit-msg hook requires all commit subjects to follow the Conventional Commits format (e.g., \[feat\]: description), blocking non-conforming messages. The pre-commit hook automatically formats staged C/C++ files using clang-format and Python files using Black; if formatting changes are applied, the commit is aborted so the user can review the updates before re-committing.

kt-kernel/.githooks · high confidence

Improved Windows compilation support and expanded quantization handling in llamafile

The llamafile third-party submodule now compiles correctly on Windows (MSVC) by updating preprocessor checks to recognize the \_M\_X64 target across multiple source files (sgemm.cpp, tinyblas\cpu\.cpp, iqk\_mul\mat\.cpp). Additionally, the matrix multiplication logic in tinyblas\_cpu\_sgemm.inc has been extended to accept a wider range of B-type quantizations (including Q8\_0 and Q8\_1) and natively supports MXFP4 GGUF weights via a GGML vec-dot fallback, while ARM builds now utilize a dedicated iqk\_mul\_mat\_arm.inc implementation.

_third\party/llamafile · high confidence

Test coverage

Added AMX kernel unit and latency tests; Added per-commit test suite for KT-Kernel; New CI test infrastructure and validation suites for kt-kernel.

Dependencies

Archives legacy packages and refactors build configuration

The \kt-sft\ package and the \ktransformers\ website assets have been moved to the \archive/\ directory, effectively removing them from the active distribution. The main \pyproject.toml\ has been refactored to use dynamic dependency resolution and a minimal top-level module structure, while the \kt-kernel\ package now pins \torch\ to version 2.9.1 and restricts \compressed-tensors\ to versions below 0.16.0 to ensure compatibility with the chosen PyTorch baseline.

(dependencies) · high confidence

Updated sglang and added custom\_flashinfer submodules

The sglang submodule has been updated to commit 541ddc37, and a new custom\_flashinfer submodule has been added at commit fd94393. These changes integrate the latest upstream updates and custom kernel support into the third-party dependencies.

_third\party · high confidence

Housekeeping

Publish KTransformers-FineTune author preprint with versioned changelog

The \papers/kt-ft\ directory now contains the author preprint for KTransformers-FineTune, including the manuscript PDF, README files (English and Chinese), a BibTeX citation file, and a SHA-256 checksum file. A CHANGELOG.md has been added to track paper versions, starting with the initial v1 release on October 8, 2026, and a subsequent v1.1 update on October 10, 2026, which restored Figure 2 to a single-column layout.

papers · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 36 → 37 (+0.2)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.

Lenses

  • Code Health 52 → 52 (+0.0)
  • Architecture 94 → 94 (+0.0)
  • Maturity 48 → 75 (+27.1)
  • Readiness 50 → 50 (-0.1)
  • Security 41 → 37 (-3.6)
  • Accessibility 22 → 21 (-0.7)
  • Performance 100 → 100 (+0.0)

Resolved (4)

  • Hotspot: kt-kernel/python/sft/lora.py (kt-kernel/python/sft/lora.py)
  • Medium CVE: PYSEC-2026-139 (kt-kernel/requirements.txt)
  • Off-boarding risk: anonymized user #1
  • TodoComment repeated across 13 files (archive/kt-sft/ktransformers/models/modeling_qwen2_moe.py)

New (4)

  • High CVE: [GHSA redacted] (kt-kernel/requirements.txt)
  • Off-boarding risk: anonymized user #1
  • Outdated: python-dateutil
  • TodoComment repeated across 13 files (kt-kernel/examples/modeling_deepseek_v3.py)

Changes since last survey

  • 2 commits — 2 feature/other, 0 fixes

By area

  • kt-kernel/operators — 1 commit
  • third_party/sglang — 1 commit

Notable commits

  • change: [build]: sync sglang submodule to 18c0de72e59359baba793f7a5675f983aa656fc9 (#2217)
  • change: perf: AVX2 MXFP4 decodes each weight group once for the last 2-3 rows of an expert (#2253)

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

kvcache-ai/ktransformers was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 5 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit c9873312643b615a717832946fba0346e69ab035 — the exact code this score is about.
  • Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.2+preprod-fee7626c131f.