Skip to content
CAI
Software that uses CAICheck a score

Dao-AILab/flash-attention

59.2

Adequate · 16 November 2026

82.1k

lines of production code

Python

with C, C++

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add AMD ROCm FlashAttention backend using ComposableKernel

Introduces a new C++ implementation of the FlashAttention kernels (forward, backward, variable-length, and KV-cache variants) for AMD ROCm devices, leveraging the ComposableKernel (CK) library. This adds a new execution path for ROCm users, exposing standard PyTorch bindings for multi-head attention operations including support for ALiBi, causal masking, dropout, and rotary embeddings.

_csrc/flash\_attn\ck · high confidence

Add Flash Attention CrossEntropyLoss with advanced options

The \flash\_attn.losses\ module now exposes a \CrossEntropyLoss\ class that leverages Triton for efficient CUDA-based computation. This implementation supports several advanced features including label smoothing, logit scaling, and z-loss (via \lse\_square\_scale\) for improved training stability. It also allows returning the z-loss component separately for logging purposes, supports tensor parallelism via \process\_group\, and accepts precomputed log-sum-exp (LSE) values to enable integration with fused attention kernels.

_flash\attn/losses · high confidence

Add Hugging Face model converters for Baichuan, BERT, BigCode, BTLM, Falcon, GPT-NeoX, GPT-J, LLaMA, and OPT

The \flash\_attn/models\ package now includes dedicated converter modules for a wide range of Hugging Face models. These modules provide bidirectional state-dict remapping (Hugging Face ↔ flash\_attn internal format) and config translation for Baichuan, BERT, BigCode (StarCoder), BTLM, Falcon, GPT-NeoX, GPT-J, LLaMA (including Meta and Hugging Face formats), and OPT. This enables users to load pre-trained weights from these specific architectures directly into the flash\_attn implementation without manual weight manipulation.

_flash\attn/models · high confidence

Add sass\_diff tool and two\_pass\_tests.sh script

Added a new Python utility, sass\_diff.py, to compare SASS files by normalizing register assignments so that instructions with different register allocations are treated as identical, aiding in the analysis of compiled output. Additionally, introduced a bash script, two\_pass\_tests.sh, to facilitate fast two-pass testing of kernels: the first pass compiles kernels in parallel using FakeTensorMode (without GPU work), and the second pass executes them with a persistent compile cache to detect any compile-key drift or skipped variants.

tools · high confidence

Flash Attention 3 (Hopper) version 3.0.0 release

This change introduces the Flash Attention 3 implementation for NVIDIA Hopper (SM90) architecture, marking the module as version 3.0.0. It provides a new set of CUDA kernels and Python interfaces (including \flash\_attn\_interface\) that leverage Hopper-specific hardware features like TMA (Tensor Memory Accelerator) and bulk reduce operations for improved performance. The release includes comprehensive benchmarking scripts for MLA decode, split-KV, and FP8 attention variants, alongside the core kernel implementations for forward and backward passes.

hopper · high confidence

Flash Attention 3 module initialization and versioning

The \hopper/flash\_attn\3\ package now includes its core initialization files, establishing the module structure for Flash Attention 3. The \\\init\\_.py\ file implements dynamic version detection via \importlib.metadata\, falling back to the hardcoded version \3.0.1\ if the package is not installed, while \flash\_attn\_config.py\ and \flash\_attn\_interface.py\ expose the underlying configuration and C-extension interface functions (\\_flash\_attn\_forward\, \\_flash\_attn\_backward\) to the Python layer.

_hopper/flash\_attn\3 · high confidence

FlashAttention-2 CUDA kernel implementation for Ampere+ architectures

The \csrc/flash\_attn\ directory now contains the core CUDA kernel implementation for FlashAttention-2. This includes the Python API entry point (\flash\_api.cpp\), shared parameter structures (\flash.h\), and specialized backward pass kernels for head dimensions 32, 64, 96, 128, 192, and 256 in both FP16 and BF16 formats. The implementation introduces support for local attention windows, softcapping, and Alibi biases, while refactoring dropout and masking logic into separate header files to improve modularity and compilation speed.

_csrc/flash\attn · high confidence

FlashAttention-4 (CuTeDSL) release with Hopper and Blackwell support

The repository now includes FlashAttention-4, a new generation of attention kernels written in CuTeDSL (NVIDIA CUTLASS DSL) targeting Hopper (SM90) and Blackwell (SM100/SM110) GPUs. Users can install this via \pip install flash-attn-4\. The package provides \flash\_attn\_func\ and \flash\_attn\_varlen\_func\ entry points, supporting features like SplitKV, paged KV cache, and 2CTA instructions on Blackwell. The build system has been updated to support CUDA 11.8+ and ROCm, with submodules for CUTLASS and Composable Kernel. Older streaming attention implementations have been removed.

(repo-wide) · high confidence

FlashAttention-4 (CuTeDSL) standalone package release

FlashAttention-4 is now available as a standalone installable package (\flash-attn-4\ or \fa4\) built on the CuTeDSL framework for Hopper and Blackwell GPUs. This release introduces the \flash\_attn.cute\ module, exposing \flash\_attn\_func\ and \flash\_attn\_varlen\_func\ for attention computation, and includes comprehensive benchmarking utilities, block-sparse runtime support, and SM100/SM90 specific helpers.

_flash\attn/cute · high confidence

Initial release of optimized training infrastructure and scripts

This change introduces the training environment, providing a Dockerfile that sets up a PyTorch-based container with FlashAttention 2.6.3, DeepSpeed 0.7.7, and other necessary dependencies, alongside a README detailing usage for GPT-2 and GPT-3 training on OpenWebText and The Pile datasets. It also includes the main entry point \run.py\, which configures Hydra for experiment management and enables TensorFloat-32 acceleration for improved training performance.

training · high confidence

Introduce fused activation, dense, and normalization operations

This change adds new high-performance PyTorch modules in the \flash\_attn/ops\ package, including \activations.py\ with JIT-compiled GeLU and SwiGLU functions, \fused\_dense.py\ with tensor-parallel linear layers, and \layer\_norm.py\/\rms\_norm.py\ with fused dropout-add normalization and RMSNorm support. These components provide optimized building blocks for transformer architectures, enabling features like sequence parallelism and efficient residual handling.

_flash\attn/ops · high confidence

Introduce fused dense CUDA extension with bfloat16 support

Added a new CUDA extension in csrc/fused\_dense\_lib that provides fused matrix multiplication with bias and activation (GELU or ReLU) operations for both forward and backward passes. The implementation supports bfloat16 and float16 data types, leveraging cuBLASLt for optimized performance on modern GPUs (including Hopper architecture with increased workspace allocation) and requires CUDA 11.8+ for best results. The extension is adapted from NVIDIA Apex and includes configurable compilation threads via the NVCC\_THREADS environment variable.

_csrc/fused\_dense\lib · high confidence

Introduces modular Transformer building blocks for FlashAttention

Adds new modules in \flash\_attn/modules\ that provide reusable Transformer components, including \Block\ (with prenorm/postnorm support and fused dropout/add/layer-norm), \MHA\ (Multi-Head Attention with FlashAttention kernels, ALiBi, and windowing), \Mlp\ (standard and gated variants like SwiGLU), and \Embedding\ layers (including parallel embeddings for tensor parallelism). These modules allow users to construct custom Transformer architectures using FlashAttention's optimized attention mechanisms while supporting features like sequence parallelism and tensor parallelism.

_flash\attn/modules · high confidence

New CI infrastructure for Flash Attention 4 (FA4) with Apptainer and runtime dependency enforcement

This change introduces a dedicated CI workflow for Flash Attention 4 (FA4) that runs on self-hosted GPU runners using Apptainer (SIF) containers. The system implements a two-pass test strategy (fake tensor compilation followed by real GPU execution) and supports registry-free mode via runner-local SIF images. A key behavioral improvement is the decoupling of fast-moving dependencies (cutlass-dsl, quack) from the baked container image by installing them at runtime based on version floors defined in \flash\_attn/cute/pyproject.toml\. The CI now includes a pre-flight check (\assert\_dsl\_floor.py\) to ensure installed dependencies meet these floors, preventing cryptic runtime errors from stale images, and verifies the provisioned overlay from a fresh session to avoid race conditions.

tools/ci · high confidence

New CUDA LayerNorm extension with dropout, residual, and RMSNorm support

This location introduces a new CUDA extension for fused LayerNorm that supports dropout and residual addition, works with both pre-norm and post-norm architectures, and includes an optional RMSNorm mode. It supports hidden dimensions up to 8192 (divisible by 8) and is tested on A100 GPUs. Note: this extension is no longer used in the FlashAttention repo as of 2024-01-05; the project has switched to a Triton-based implementation.

_csrc/layer\norm · high confidence

New FA4 CI container tooling for registry-free GPU jobs

Added a new Dockerfile and build scripts in tools/ci/docker to create a container image based on Ubuntu 24.04 with CUDA 13.0, PyTorch nightly (cu130), and Flash Attention 4 dependencies. The tooling includes build.sh for standard Docker builds, build\_local\_sif.sh for creating Apptainer SIF images directly on the runner without a registry, and tag\_and\_push.sh for pushing images to a remote registry, enabling CI workflows to run on GPU runners using local container artifacts.

tools/ci/docker · high confidence

New Triton-based implementations for core attention operations

The \flash\_attn/ops/triton\ module introduces a suite of new high-performance kernels implemented in Triton, replacing or supplementing previous CUDA/C++ implementations. This includes a new \CrossEntropyLoss\ operation supporting label smoothing, logit scaling, z-loss, and tensor-parallel splits; a \LayerNorm\ kernel with support for zero-centered weights, dropout, and compatibility with \torch.compile\; a \Rotary\ embedding kernel supporting variable-length sequences, interleaved layouts, and GQA block layouts; and fused \FusedDense\ MLP blocks that combine linear layers with squared ReLU activations. These changes provide users with optimized, flexible building blocks for training large language models.

_flash\attn/ops/triton · high confidence

New suite of benchmarking scripts for FlashAttention backends and configurations

Added a collection of new benchmarking scripts in the \benchmarks/\ directory to evaluate FlashAttention performance across various backends (FA2, FA3, FA4, cuDNN, PyTorch, Triton, xFormers) and hardware configurations. \benchmark\_attn.py\ provides a unified interface to compare forward and backward pass throughput for multiple attention implementations, including support for variable-length sequences and specific SM90/SM100 optimizations. \bench\_sm90.py\ offers a targeted benchmark for SM90 architectures with command-line arguments to sweep tile sizes, backward optimizations, and causal/non-causal modes. Additional scripts include \benchmark\_alibi.py\ for ALiBi bias performance, \benchmark\_causal.py\ and \benchmark\_flash\_attention.py\ for causal attention profiling, \benchmark\_gemm.py\ for comparing cuBLAS/hipBLAS matmul performance, \benchmark\_mla\_paged\_kv.py\ for Multi-Head Latent Attention with paged KV caches, \benchmark\_sparse\_mla\_fwd.py\ for sparse MLA forward kernels, and \benchmark\_varlen\_sched.py\ for comparing variable-length tile schedulers.

benchmarks · high confidence

New training infrastructure: callbacks, fault-tolerant sampling, and dataset modules

This change introduces a suite of new PyTorch Lightning callbacks and data modules to the training pipeline. Users can now monitor training health with new tools including CausalityMonitor (tracking gradient leakage), EMA (Exponential Moving Average for model weights), FLOP counting, GPU affinity management, loss scale monitoring, norm monitoring, and speed monitoring. Fault tolerance is improved via new RandomFaultTolerantSampler and FaultTolerantDistributedSampler classes that allow resuming from interruptions without data duplication or skipping. Additionally, new data modules support ImageNet classification and Hugging Face-based language modeling with shared-memory optimizations, detokenization support, and mixup augmentation.

training/src · high confidence

New utility modules for benchmarking, distributed training, and model generation

This release introduces a suite of new utility modules in \flash\_attn/utils\ to support advanced usage patterns. The \benchmark\ module provides functions to measure forward, backward, and combined pass performance using PyTorch's benchmarking tools, including AMP support. The \distributed\ module adds utilities for tensor and sequence parallelism, implementing \all\_gather\, \reduce\_scatter\, and \all\_reduce\ operations with autograd support, along with helpers for synchronizing shared parameters and reducing sequence-parallel gradients. The \generation\ module offers a comprehensive decoding loop supporting greedy search, top-k, and top-p sampling, with optional CUDA graph caching for accelerated inference. Additionally, \pretrained\ enables loading Hugging Face checkpoints (including sharded and safetensors formats) with safe loading practices, while \library\ provides a \triton\_op\ decorator for registering Triton kernels as PyTorch custom ops, and \torch\ ensures compatibility with deprecated AMP decorators.

_flash\attn/utils · high confidence

Removals

Removal of legacy Fused Multi-Head Attention (FMHA) implementation

The \csrc/stream\_attn\ directory has been cleared of the previous Fused Multi-Head Attention (FMHA) codebase. This change removes the C++ API entry point (\fmha\_api.cpp\), the core CUDA kernel headers (\fmha.h\, \gemm.h\, \gmem\_tile.h\, \smem\_tile.h\, \softmax.h\, \mask.h\, \kernel\_traits.h\), and the associated documentation (\README.md\). Users relying on this specific implementation within this module will no longer have access to these components.

_csrc/stream\attn · high confidence

Behavioural changes

FlashAttention-2 package restructure and version bump to 2.8.4

The flash\attn package has been reorganized into a dedicated subdirectory and updated to version 2.8.4. The public API is now explicitly exported via the package's \\init\\_.py, exposing core functions such as flash\_attn\_func, flash\_attn\_with\_kvcache, and their variable-length and packed variants. The module also includes bert\_padding utilities for handling unpadding and indexing, and the block-sparse attention implementation has been renamed from 'stream' to 'flash' to align with the new package structure.

_flash\attn · high confidence

FlashAttention-3 kernel instantiations split into separate compilation units

The auto-generated kernel instantiation files in the \hopper/instantiations\ directory have been reorganized into individual \.cu\ files (e.g., \flash\_bwd\_hdim128\_bf16\_sm80.cu\, \flash\_fwd\_hdim128\_bf16\_sm100.cu\) to speed up compilation. This change introduces support for the new SM100 architecture alongside existing SM80, SM86, and SM90 targets, and expands backward pass support to include head dimensions of 64, 96, 128, 192, and 256. The refactoring also ensures that features like PagedKV and PackGQA are consistently enabled for relevant architectures to reduce binary size and compilation overhead.

hopper/instantiations · high confidence

New PatchEmbed layer and refactored Rotary Embedding implementation

The \flash\attn/layers\ module now includes a new \PatchEmbed\ class for converting 2D images to patch embeddings, offering an API compatible with timm but using \nn.Linear\ for approximately 8x faster performance. Additionally, the rotary embedding logic has been significantly refactored: the \ApplyRotaryEmbQKV\\ operation is now wrapped as a custom Triton op, support for interleaved (GPT-J style) rotary embeddings has been added, and the implementation now handles variable sequence lengths and sequence length offsets for inference with KV caches. The \pos\_idx\_in\_fp32\ option has been removed, and frequency calculations are ensured to be in fp32.

_flash\attn/layers · high confidence

Test coverage

Add comprehensive test suites for FlashAttention kernels and rotary embeddings; Added comprehensive test suite and benchmarking infrastructure for CuTe DSL kernels; Added comprehensive tests for Triton LayerNorm operations; Added model verification tests for Baichuan, BERT, BigCode, BTLM, Falcon, GPT, GPT-NeoX, and GPT-J; Added parallel module tests; Added tests for CrossEntropyLoss features and parallel execution; Added tests for NeoX and GPT-J style rotary embeddings; Added tests for fused linear, MLP, and layer normalization operations; Added tests for the Hugging Face language modeling data module.

Dependencies

Flash Attention 4 CUTE becomes a standalone installable package

The Flash Attention CUTE (CUDA Template Engine) implementation is now available as a standalone Python package named 'flash-attn-4', installable independently from the main library. This new package requires Python 3.10 or later and includes specific dependencies such as nvidia-cutlass-dsl (\>=4.6.2), quack-kernels (\>=0.5.3), and apache-tvm-ffi (\>=0.1.11). It also introduces optional dependencies for CUDA 13 support and development tools, along with configuration for build systems and code linting via Ruff.

(dependencies) · high confidence

Updated AITER submodule to commit f0966b0

The AITER third-party submodule has been updated to commit f0966b0aed0b38198e2e1ca1cff8cca343bbcb7e, incorporating upstream changes from the AITER library.

_third\party · high confidence

Updated Cutlass and ComposableKernel submodules

The csrc directory now points to new commits for the Cutlass (7127592) and ComposableKernel (c56c675) submodules, incorporating upstream changes from these libraries into the build.

csrc · high confidence

Housekeeping

Added AI debugging documentation and methodology guides

Added a suite of new documentation files in the AI directory to support kernel development and debugging. These include \CLC\_TRACE\_DEBUG.md\ for tracing the CLC work scheduler, \DEBUG\_2CTA.md\ for diagnosing GPU kernel hangs and deadlocks in 2CTA kernels, \DEBUG\_METHODOLOGY.md\ establishing a root-cause investigation protocol, \RACECHECK\_TMA\_HAZARD.md\ documenting false-positive race hazards with \cp.async.bulk\, \SASS\_MMA\_ANALYSIS.md\ for analyzing HGMMA instructions, \SM90\_BLOCK\_SIZE\_TUNING.md\ for tuning tile sizes on Hopper, \SM90\_R2P\_MASKING\_SASS.md\ detailing R2P masking optimizations, and \SPARSE\_MLA\_1CTA.md\ and \SPARSE\_MLA\_64H.md\ describing the SM100 1CTA forward kernel and 64-row tile backward specializations.

AI · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 60.8 → 59.2 (-1.6)
  • Rubric changed (rubric-2026.11.2 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 66.3 → 59.2 (-7.1)
  • Architecture 92.3 → 93.4 (+1.1)
  • Maturity 59.4 → 60.4 (+1.0)
  • Readiness 55.4 → 53.6 (-1.8)
  • Security 68.0 → 68.0 (0.0)
  • Performance 80.0 → 80.0 (0.0)

Resolved (170)

  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.call (cognitive 27) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.call (cyclomatic 23) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.compute (cognitive 75) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.compute (cyclomatic 31) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.dkdv_bwd (cognitive 150) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.dkdv_bwd (cyclomatic 55) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.epilogue (cognitive 41) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.epilogue (cyclomatic 17) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.load (cognitive 25) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.mma_2cta (cognitive 33) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDKDVKernel.mma_2cta (cyclomatic 18) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDQKernel.call (cognitive 25) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dqkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDQKernel.call (cyclomatic 26) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dqkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDQKernel.dQ_epilogue (cognitive 35) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dqkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDQKernel.kernel (cognitive 689) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dqkernel.py)
  • BlackwellFusedMultiHeadAttentionBackwardDQKernel.kernel (cyclomatic 139) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dqkernel.py)
  • Duplicated block (10 lines × 2) (flash_attn/cute/block_sparse_utils.py)
  • Duplicated block (10 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (10 lines × 2) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • Duplicated block (10 lines × 2) (flash_attn/cute/sm100_hd256_2cta_fmha_backward_dkdvkernel.py)
  • …and 150 more

New (131)

  • Duplicated block (10 lines × 3) (flash_attn/cute/interface.py)
  • Duplicated block (11 lines × 4) (flash_attn/cute/block_sparse_utils.py)
  • Duplicated block (12 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (12 lines × 3) (flash_attn/cute/block_sparse_utils.py)
  • Duplicated block (12 lines × 4) (flash_attn/cute/block_sparse_utils.py)
  • Duplicated block (12–13 lines × 2) (flash_attn/cute/block_sparse_utils.py)
  • Duplicated block (14 lines × 2) (flash_attn/cute/block_sparse_utils.py)
  • Duplicated block (14 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (14–16 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (15 lines × 4) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (16 lines × 2) (flash_attn/cute/flash_fwd_sm100.py)
  • Duplicated block (16 lines × 2) (flash_attn/cute/interface.py)
  • Duplicated block (20 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (20 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (21–25 lines × 5) (flash_attn/cute/flash_bwd_mla_sm100.py)
  • Duplicated block (21–35 lines × 2) (flash_attn/cute/interface.py)
  • Duplicated block (22 lines × 6) (flash_attn/cute/flash_bwd_mla_sm100.py)
  • Duplicated block (23 lines × 4) (flash_attn/cute/flash_bwd_mla_sm100.py)
  • Duplicated block (24 lines × 5) (flash_attn/cute/flash_bwd_sm100.py)
  • Duplicated block (24–26 lines × 2) (flash_attn/cute/flash_bwd_sm100.py)
  • …and 111 more

Changes since last survey

  • 27 commits — 26 feature/other, 1 fixes

By area

  • flash_attn/cute — 16 commits
  • csrc/flash_attn — 5 commits
  • tests/cute — 4 commits
  • hopper/flash_api.cpp — 2 commits

Notable commits

  • fix: Fix FA3 out-of-bounds reads with empty KV-cache page tables (#3161)
  • change: [CuTe, Bwd, SM100] Fold the hdim64 LSE / dPsum subtractions into the S / dP MMA (#3098)
  • change: [CuTe, SM100] Add MXFP8 QK forward with BF16 PV (#3019)
  • change: [CuTe, SM100] Add MXFP8 quantization and scale-layout helpers (#3018)
  • change: [CuTe, SM100] Backward: explicit work-tile and head coordinates (#3141)
  • change: [CuTe, SM100] Backward: hdim 256 on the general kernel by default; delete the dedicated kernels (#2968)
  • change: [CuTe, SM100] Compile autotune candidates for sm_107a (#3121)
  • change: [CuTe, SM100] Give BF16 2CTA hd128 softmax 192 registers (#3016)
  • change: [CuTe, SM100] Group FP8 descales in a typed scales argument (#3027)
  • change: [CuTe, SM100] Make the dynamic varlen forward scheduler bind tiles late and self-reset (#3086)
  • change: [CuTe, SM100] Preserve the 15-bit sm_107 SMEM descriptor address (#3029)
  • change: [CuTe, SM100] Release softmax statistics per stage by default (#3017)
  • change: [CuTe, SM100] Route padded varlen forward calls to the dynamic persistent scheduler (#3087)
  • change: [CuTe, SM100] Unify hd256 backward (6/7): pack 2 query heads per tile and loop the KV head's remaining heads inside the CTA (#3059)
  • change: [CuTe, SM100] hd256 backward follow-ups from #2968 (#3155)
  • change: [CuTe, SM100] hd256 keeps PackGQA when 2CTA is not possible (#3148)
  • change: [CuTe] Clamp aux_tensors indices instead of wrapping them modulo seqlen (#3150)
  • change: [CuTe] Clamp huge local window sides before they overflow int32 (#3173)
  • change: [CuTe] Reject a paged KV cache whose key count reaches 2**31 (#3168)
  • change: [CuTe] Return early for a paged KV cache whose page table has no column (#3167)
  • …and 7 more

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

Dao-AILab/flash-attention was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 16 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 328dd7b5a6905c75500a9bce334ac012763409a9 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.83+preprod-a489942cf94b.