lucidrains/vit-pytorch
51.5
Adequate · 16 November 2026
15.6k
lines of production code
Python
primary language
5
measurements over time
What this system is
Features
Added training script for ViT with decorrelation auxiliary loss
Users can now train Vision Transformers using a decorrelation auxiliary loss to regularize multi-head attention and feedforward subspaces. This change introduces a new \ViT\ variant (\vit\_pytorch.vit\_with\_decorr\) and a dedicated training script (\train\_vit\_decorr.py\) that demonstrates how to apply this regularization on CIFAR-100 using Accelerate and Weights & Biases.
(repo-wide) · high confidence
New Vision Transformer architectures and self-supervised learning wrappers
This release adds several new Vision Transformer variants and self-supervised learning frameworks to the library. New model implementations include CaiT (with LayerScale and cross-attention), CrossViT (multi-scale encoder with cross-attention), CrossFormer (dynamic positional bias), CvT (convolutional token mixing), DeepViT (re-attention), CCT (compact conv-based transformer), and ATS-ViT (adaptive token sampling). Additionally, wrappers and implementations for self-supervised techniques are introduced, including DINO (with EMA teacher and distillation loss), MAE (Masked Autoencoder), and Masked Position Prediction (MPP). A new AcceptVideoWrapper allows wrapping existing image networks for video processing with optional time positional embeddings and MOSS integration.
_vit\pytorch · high confidence
New example notebook for training a Visual Transformer on Cats vs Dogs data
Added a new Jupyter notebook example (\examples/cats\_and\_dogs.ipynb\) that demonstrates how to train a Visual Transformer using the Linformer efficient attention implementation on the Kaggle Dogs vs Cats dataset. The notebook includes setup for dependencies (\vit\_pytorch\, \linformer\), data loading, and model training configuration.
examples · high confidence
Test coverage
Added tests for DINO loss stability and ViT model functionality
Added new test coverage for the DINO distillation loss function, specifically verifying that loss values and gradients remain finite and numerically stable when using low-precision data types (float16 and bfloat16), and confirming gradient correctness against a reference implementation. Additionally, added a basic functional test for the ViT model to ensure it produces correctly shaped output logits.
tests · high confidence
Dependencies
vit-pytorch version 1.26.7 release
The package has been updated to version 1.26.7. The build system now requires setuptools\>=61 and wheel, and the project mandates Python 3.8 or higher. Runtime dependencies are specified as einops\>=0.8.2, torch\>=2.5, and torchvision. Test dependencies include pytest and specific pinned versions of torch (2.5.0) and torchvision (0.20.0).
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 52 → 52 (-0.4)
- Rubric changed (rubric-2026.11.2 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 67 → 57 (-10.5)
- Architecture 86 → 100 (+14.2)
- Maturity 50 → 50 (+0.0)
- Readiness 38 → 38 (+0.0)
- Security 100 → 100 (+0.0)
Resolved (4)
- Duplicated block (15 lines × 2) (vit_pytorch/vit_nd_pope.py)
- Duplicated block (19 lines × 2) (vit_pytorch/vaat.py)
- Duplicated block (7 lines × 3) (vit_pytorch/vaat.py)
- Projects may be oversized for their cohesion
New (73)
- Edited copies of one member (3 members, 10+ corresponding lines) (vit_pytorch/local_vit.py)
- Edited copies of one member (3 members, 10+ corresponding lines) (vit_pytorch/simple_uvit.py)
- Edited copies of one member (3 members, 13+ corresponding lines) (vit_pytorch/levjepa_nd.py)
- Edited copies of one member (3 members, 15+ corresponding lines) (vit_pytorch/cvt.py)
- Edited copies of one member (3 members, 15+ corresponding lines) (vit_pytorch/deepvit.py)
- Edited copies of one member (3 members, 17+ corresponding lines) (vit_pytorch/mp3.py)
- Edited copies of one member (3 members, 18+ corresponding lines) (vit_pytorch/vit.py)
- Edited copies of one member (3 members, 23+ corresponding lines) (vit_pytorch/cait.py)
- Edited copies of one member (3 members, 3+ corresponding lines) (vit_pytorch/dino.py)
- Edited copies of one member (3 members, 31+ corresponding lines) (vit_pytorch/jet_vit.py)
- Edited copies of one member (3 members, 4+ corresponding lines) (vit_pytorch/dino.py)
- Edited copies of one member (3 members, 4+ corresponding lines) (vit_pytorch/levjepa.py)
- Edited copies of one member (3 members, 58+ corresponding lines) (vit_pytorch/vit_nd.py)
- Edited copies of one member (4 members, 2+ corresponding lines) (vit_pytorch/regionvit.py)
- Edited copies of one member (4 members, 4+ corresponding lines) (vit_pytorch/levjepa.py)
- Edited copies of one member (4 members, 4+ corresponding lines) (vit_pytorch/looped_vit.py)
- Edited copies of one member (4 members, 8+ corresponding lines) (vit_pytorch/cait.py)
- Edited copies of one member (5 members, 25+ corresponding lines) (vit_pytorch/deepvit.py)
- Edited copies of one member (6 members, 25+ corresponding lines) (vit_pytorch/ats_vit.py)
- Edited copies of one member (7 members, 14+ corresponding lines) (vit_pytorch/cross_vit.py)
- …and 53 more
Changes since last survey
- 1 commits — 1 feature/other, 0 fixes
By area
- (root) — 1 commit
Notable commits
- change: turns out a group out there thought of the multihead attention residual idea concurrently
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
lucidrains/vit-pytorch was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 16 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 42298e16ad7915ea6f531049e7909e7ee07bc5fe — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.73+preprod-52de90b74216.