Skip to content
CAI
Software that uses CAICheck a score

labmlai/annotated_deep_learning_paper_implementations

52.4

Adequate · 3 November 2026

31.6k

lines of production code

Python

with JavaScript

3

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add ALiBi (Attention with Linear Biases) transformer implementation

This change introduces a new implementation of the Attention with Linear Biases (ALiBi) mechanism, replacing standard positional encodings with linear biases added to attention scores to enable input length extrapolation. The \labml\_nn/transformers/alibi\ module provides the \AlibiMultiHeadAttention\ class and helper functions for calculating head-specific slopes and bias matrices. Additionally, an experiment script is included to train this ALiBi-based model on the Tiny Shakespeare dataset, demonstrating its use in an autoregressive language modeling context.

_labml\nn/transformers/alibi · high confidence

Add Attention Free Transformer (AFT) implementation and experiment

Introduces a PyTorch implementation of the Attention Free Transformer (AFT), specifically the AFT-local variant which uses local pairwise position biases to reduce memory complexity to O(Td). The module provides the AFTLocal layer and an accompanying experiment script that demonstrates training this model on the Tiny Shakespeare dataset using a character-level tokenizer.

_labml\nn/transformers/aft · high confidence

Add Batch Normalization implementation and MNIST/CIFAR-10 experiments

Introduces a new PyTorch Batch Normalization layer (\labml\_nn.normalization.batch\_norm.BatchNorm\) that supports image, embedding, and sequence inputs, with configurable affine parameters and running statistics tracking. Includes an MNIST experiment demonstrating the layer in a CNN classifier and a CIFAR-10 experiment using the same normalization technique in a VGG-style model.

_labml\_nn/normalization/batch\norm · high confidence

Add CFR implementation and tutorial for Kuhn Poker

Introduces an annotated implementation and interactive notebook for applying Counterfactual Regret Minimization (CFR) to Kuhn Poker. The change adds the core game logic in \labml\nn/cfr/kuhn/\\init\\_.py\, defining the \InfoSet\ and \History\ classes specific to the two-player, three-card betting game, and provides \experiment.ipynb\ to demonstrate training the CFR algorithm with configurable epochs.

_labml\nn/cfr/kuhn · high confidence

Add Capsule Networks implementation and MNIST training example

Introduces a PyTorch implementation of Capsule Networks based on the 'Dynamic Routing Between Capsules' paper, including core modules (Squash, Router, MarginLoss) and an annotated MNIST classification example with training configuration.

_labml\_nn/capsule\networks · high confidence

Add ConvMixer implementation and CIFAR-10 training experiment

Added a PyTorch implementation of the ConvMixer architecture (from the paper 'Patches Are All You Need?') and a corresponding experiment script to train it on CIFAR-10. The model uses depth-wise convolutions for spatial mixing and point-wise convolutions for channel mixing, with batch normalization and residual connections applied only to the spatial mixing step. The included experiment demonstrates training this model on the CIFAR-10 dataset using standard configurations.

_labml\_nn/conv\mixer · high confidence

Add Cycle GAN implementation and experiment notebook

Added a PyTorch implementation of Cycle GAN (Unpaired Image-to-Image Translation) including the model architecture (GeneratorResNet, Discriminator) and training configuration in labml\_nn/gan/cycle\gan/\\init\\_.py, along with an accompanying experiment notebook (experiment.ipynb) and documentation (readme.md).

_labml\_nn/gan/cycle\gan · high confidence

Add DCGAN implementation for MNIST

Added a new Deep Convolutional Generative Adversarial Network (DCGAN) implementation for the MNIST dataset. This includes the \Generator\ and \Discriminator\ neural network modules, a configuration setup to integrate with the existing experiment framework, and a Jupyter notebook tutorial for running the training experiment.

_labml\nn/gan/dcgan · high confidence

Add DDPM and DDIM sampling algorithms for stable diffusion

This change introduces the \DiffusionSampler\ base class along with concrete implementations for Denoising Diffusion Probabilistic Models (DDPM) and Denoising Diffusion Implicit Models (DDIM) within the stable diffusion module. Users can now generate images using either the standard DDPM sampling loop or the faster, deterministic DDIM sampler (with configurable discretization and eta parameters), supporting features like unconditional guidance scaling, noise temperature control, and step skipping.

_labml\_nn/diffusion/stable\diffusion/sampler · high confidence

Add DeepNorm transformer implementation and experiment

Introduces a PyTorch implementation of DeepNorm, a normalization technique designed to stabilize extremely deep transformers (up to 1,000 layers) by replacing standard LayerNorm with a residual-connected normalization function and specific weight initialization schemes. The \labml\_nn/normalization/deep\_norm\ module provides the \DeepNorm\ and \DeepNormTransformerLayer\ classes, which apply alpha-scaled residuals and beta-scaled weight initialization to feed-forward and attention layers. An accompanying experiment demonstrates training a 50-layer autoregressive transformer on the Tiny Shakespeare dataset using these components.

_labml\_nn/normalization/deep\norm · high confidence

Add Evidential Deep Learning for classification uncertainty

Added a new tutorial and implementation for Evidential Deep Learning (EDL) to quantify classification uncertainty. This includes PyTorch modules for computing evidence-based loss functions (Maximum Likelihood, Cross-Entropy Bayes Risk, Squared Error Bayes Risk) and a complete training experiment on the MNIST dataset using a LeNet-based architecture.

_labml\nn/uncertainty · high confidence

Add FNet token mixing implementation and AG News experiment

Introduces a PyTorch implementation of FNet, which replaces standard self-attention with two Fourier transforms to mix tokens, offering approximately 7x efficiency over self-attention with about 92% of BERT's accuracy on the GLUE benchmark. The \FNetMix\ module provides a drop-in replacement for attention layers, and an accompanying experiment demonstrates training this model on the AG News classification dataset.

_labml\nn/transformers/fnet · high confidence

Add Feedback Transformer implementation and training experiment

Introduces a PyTorch implementation of the Feedback Transformer (based on the 'Accessing Higher-level Representations in Sequential Transformers with Feedback Memory' paper) within the \labml\_nn\ library. This includes the core \FeedbackAttention\ module, which supports both the original recurrent attention mechanism and an optimized variant with precomputed, shared keys and values to reduce memory usage during prediction. The change also adds the training infrastructure (\experiment.py\) and a Colab notebook (\experiment.ipynb\) for training the model on the Tiny Shakespeare dataset, allowing users to choose between the standard and KV-shared configurations.

(repo-wide) · high confidence

Add Fuzzy Tiling Activations (FTA) implementation and experiment

Introduces a new PyTorch implementation of Fuzzy Tiling Activations (FTA), a sparse activation function that uses soft boundaries to provide non-zero gradients and preserve precision compared to hard binning. The change adds the \labml\_nn/activations/fta\ module containing the \FTA\ nn.Module and a companion experiment (\experiment.py\ and \experiment.ipynb\) that demonstrates training a transformer with FTA in the feed-forward network on the Tiny Shakespeare dataset.

_labml\nn/activations/fta · high confidence

Add GATv2 implementation and training experiment

Introduces a PyTorch implementation of the Graph Attention Networks v2 (GATv2) operator, which addresses the static attention limitations of standard GAT by allowing dynamic attention ranks. The update includes the core \GraphAttentionV2Layer\ module supporting configurable head concatenation/averaging and weight sharing, a two-layer \GATv2\ model class, and a complete training experiment script for the Cora dataset.

_labml\nn/graphs/gatv2 · high confidence

Add GPT model implementation and training tutorial

Introduces a new GPT model implementation in \labml\_nn/transformers/gpt\, providing a PyTorch-based autoregressive transformer with specific tweaks such as GELU activation, custom weight initialization (normal distribution for weights, zero for biases), and a custom optimizer configuration that applies weight decay only to linear layer weights. This includes the \GPT\ module class, configuration options via \Configs\, and an accompanying Colab notebook (\experiment.ipynb\) demonstrating how to train the model on the Tiny Shakespeare dataset.

_labml\nn/transformers/gpt · high confidence

Add GPT-NeoX model implementation with checkpoint loading and evaluation

This change introduces a new GPT-NeoX implementation in the \labml\nn/neox\ module, providing the core model definition (\model.py\), checkpoint downloading and loading utilities (\checkpoint.py\), and tokenizer support (\tokenizer.py\). It includes an evaluation harness adapter (\evaluation/\\init\\_.py\) to test the model on NLP tasks, along with specific evaluation scripts for half-precision (\evaluation/half\_precision.py\) and LLM.int8() quantization (\evaluation/llm\_int8.py\). The model supports Flash Attention and includes sample scripts for text generation and fine-tuning.

_labml\nn/neox · high confidence

Add Gated Linear Unit (GLU) variants for transformer feed-forward networks

Users can now train auto-regressive transformers using different Gated Linear Unit activation functions in the position-wise feed-forward network. This change introduces a new module with two implementations: a configurable experiment using \labml.configs\ and a simpler, standalone script. The available GLU variants include GLU, Bilinear, ReGLU, GEGLU, and SwiGLU, allowing users to experiment with different gating mechanisms for improved model performance.

_labml\_nn/transformers/glu\variants · high confidence

Add Graph Attention Network (GAT) implementation and Cora training experiment

Added a PyTorch implementation of Graph Attention Networks, including the \GraphAttentionLayer\ module with masked self-attention and multi-head mechanisms, and a complete training experiment for node classification on the Cora dataset. The new code introduces the \labml\_nn.graphs.gat\ package, providing both the core model architecture and the configuration-driven training script for users to reproduce the GAT results.

_labml\nn/graphs/gat · high confidence

Add Hourglass hierarchical transformer model and training experiment

Introduces an implementation of the Hourglass hierarchical transformer architecture, which handles long sequences efficiently by recursively down-sampling tokens in the first half of the layers and up-sampling with skip connections in the second half. The module includes specific components for sequence shifting, average-pooling down-sampling, and naive up-sampling. Additionally, a training experiment is provided to train this model on the Tiny Shakespeare dataset using character-level tokenization.

_labml\_nn/transformers/hour\glass · high confidence

Add HyperLSTM implementation and experiment

Added a new PyTorch implementation of HyperLSTM (from the HyperNetworks paper) in \labml\_nn/hypernetworks\, including the core \HyperLSTMCell\ and \HyperLSTM\ modules, an autoregressive model configuration, and a Jupyter notebook experiment for training on the Shakespeare dataset.

_labml\nn/hypernetworks · high confidence

Add Instance Normalization layer and CIFAR-10 experiment

Introduces a new PyTorch implementation of the Instance Normalization layer (\labml\_nn.normalization.instance\_norm.InstanceNorm\), which normalizes inputs across spatial dimensions to support style transfer and other use cases. The module includes an optional affine transformation (learnable scale and shift parameters) and is designed to handle inputs of arbitrary spatial dimensions. Additionally, an experiment script is provided that demonstrates using this layer within a VGG-style convolutional neural network for CIFAR-10 classification.

_labml\_nn/normalization/instance\norm · high confidence

Add Layer Normalization implementation

Added a new PyTorch implementation of Layer Normalization in the labml\_nn library. This component provides a custom LayerNorm class that normalizes inputs across specified dimensions, supporting variable input shapes (NLP sequences, images, etc.) and optional element-wise affine parameters (gain and bias). The implementation includes documentation explaining its advantages over Batch Normalization for small batch sizes and RNNs, and is intended for use in transformer and other neural network architectures.

_labml\_nn/normalization/layer\norm · high confidence

Add MLP-Mixer transformer component and experiment

Added a new MLP-Mixer module that serves as a drop-in replacement for the self-attention layer in the existing transformer implementation, applying an MLP across the sequence dimension instead of the feature dimension. Included an experiment script demonstrating how to configure and train this model on a masked language modeling task using the Tiny Shakespeare dataset.

_labml\_nn/transformers/mlp\mixer · high confidence

Add PonderNet adaptive computation tutorial and parity task dataset

This change introduces a new tutorial section for adaptive computation, specifically implementing PonderNet: Learning to Ponder. It includes the core model architecture (ParityPonderGRU) which dynamically adjusts the number of recurrent steps based on input complexity, along with the associated reconstruction and regularization loss functions. Additionally, a ParityDataset is provided to generate synthetic data for the parity task, and a training experiment script is included to demonstrate training the model on this task.

_labml\_nn/adaptive\computation · high confidence

Add Primer EZ transformer implementation with Squared ReLU and Multi-DConv-Head Attention

This change introduces a new PyTorch implementation of the Primer EZ architecture, which offers faster training compared to vanilla transformers. It adds a Squared ReLU activation function for the position-wise feedforward module and a Multi-DConv-Head Attention (MDHA) mechanism that applies spatial depth-wise convolutions to query, key, and value projections. The module also includes configurable options for these components in the transformer configs, along with variations like shared and per-head convolutions, and an experiment script for training on the Tiny Shakespeare dataset.

_labml\_nn/transformers/primer\ez · high confidence

Add RETRO (Retrieval-Enhanced Transformer) implementation

Added a new PyTorch implementation of the RETRO model, which improves language models by retrieving relevant text chunks from a large key-value database during prediction. This addition includes the core model architecture with nearest-neighbor encoding, a BERT-based embedding module for indexing text chunks, a FAISS-powered database for efficient nearest-neighbor retrieval, a dataset builder that pre-calculates neighbors to speed up training, and training/sampling scripts using the Tiny Shakespeare dataset.

_labml\nn/transformers/retro · high confidence

Add RWKV language model implementation and training experiment

Introduces a new Receptance Weighted Key Value (RWKV) model implementation in PyTorch, including core components like ChannelMixing and TimeMixing layers, along with a configurable experiment setup for training on autoregressive tasks (e.g., Tiny Shakespeare). This adds a new model architecture option to the labml\_nn library, allowing users to train and evaluate RWKV-based language models with specific hyperparameters for layers, heads, and embedding dimensions.

_labml\nn/rwkv · high confidence

Add Recurrent Highway Network implementation

Added a new PyTorch implementation of Recurrent Highway Networks (RHN), including the \RHNCell\ and \RHN\ classes in \labml\_nn/recurrent\_highway\networks/\\init\\_.py\. This provides a multilayer RHN module that processes sequential data, implementing the specific gating mechanisms and weight-tying described in the original paper, along with accompanying documentation.

_labml\_nn/recurrent\_highway\networks · high confidence

Add ResNet implementation and CIFAR-10 training experiment

Introduces a PyTorch implementation of the ResNet architecture, including \ResidualBlock\ and \BottleneckResidualBlock\ classes that handle residual connections and linear projections for stride adjustments. Adds a training experiment script (\experiment.py\) that configures and runs ResNet on the CIFAR-10 dataset using the LabML experiment framework, and includes documentation (\readme.md\, \\_\init\\_.py\) explaining the residual learning approach.

_labml\nn/resnet · high confidence

Add Stable Diffusion example scripts for text-to-image, image-to-image, and in-painting

New CLI scripts have been added to demonstrate Stable Diffusion usage. The text-to-image script supports switching between DDIM and DDPM samplers and includes a flag to enable Flash Attention for performance. The image-to-image script allows transforming a source image based on a text prompt with configurable strength, while the in-painting script enables modifying specific regions of an image using a mask and prompt.

_labml\_nn/diffusion/stable\diffusion/scripts · high confidence

Add Stable Diffusion model components with Flash Attention support

This change introduces the core model architecture for Stable Diffusion, including the U-Net, Autoencoder, and CLIP text embedder modules. The U-Net implementation integrates a Spatial Transformer for attention mechanisms and includes conditional logic to utilize Flash Attention for improved performance when the library is available and conditions are met.

_labml\_nn/diffusion/stable\diffusion/model · high confidence

Add StyleGAN 2 implementation and training code

Added a minimalistic, single-GPU PyTorch implementation of StyleGAN 2, including the core model components (MappingNetwork, Generator, Discriminator) and a training experiment script. The implementation features weight modulation and demodulation, path length regularization, and lazy regularization for gradient and path penalties, trained on the CelebA-HQ dataset.

_labml\nn/gan/stylegan · high confidence

Add Swish activation function

A new Swish activation function is now available in the neural network library. It is implemented as a PyTorch \nn.Module\ that computes the element-wise product of the input and its sigmoid, and is exported from the \labml\_nn.activations\ package for direct use.

_labml\nn/activations · high confidence

Add Switch Transformer implementation and experiment

Introduces a miniature PyTorch implementation of the Switch Transformer, featuring a SwitchFeedForward module that routes tokens to multiple expert FFNs based on a learned router, with support for load balancing, token dropping, and probability scaling. The change includes the core model components (SwitchTransformerLayer, SwitchFeedForward), a training experiment script for autoregressive language modeling on the Tiny Shakespeare dataset, and associated documentation.

_labml\nn/transformers/switch · high confidence

Add Triton-based Flash Attention 2 implementation

Introduces a new PyTorch/Triton implementation of Flash Attention 2 in the \labml\_nn.transformers.flash\ module. This adds a custom \AttentionFunc\ that optimizes the transformer attention mechanism by reducing memory reads/writes between GPU HBM and SRAM, supporting group query attention, causal masking, and both forward and backward passes. A corresponding test suite (\test.py\) is included to verify numerical correctness against naive PyTorch attention and to benchmark performance against the official \flash\_attn\ library.

_labml\nn/transformers/flash · high confidence

Add U-Net model implementation and Carvana dataset training experiment

Introduces a PyTorch implementation of the U-Net architecture for biomedical image segmentation, including the core model components (contracting/expansive paths, skip connections) and a complete training experiment on the Carvana dataset for vehicle mask generation.

_labml\nn/unet · high confidence

Add Vision Transformer (ViT) implementation and CIFAR-10 training experiment

This change introduces a new PyTorch implementation of the Vision Transformer (ViT) model within the \labml\_nn\ library. The module includes core components such as \PatchEmbeddings\ (using a convolutional layer to split images into patches), \LearnedPositionalEmbeddings\, a \ClassificationHead\ (MLP), and the main \VisionTransformer\ class that combines these with a standard transformer encoder. Additionally, a training experiment (\experiment.py\) is provided to demonstrate how to train this ViT model on the CIFAR-10 dataset, including configuration for the optimizer, learning rate, and data augmentation.

_labml\nn/transformers/vit · high confidence

Add Wasserstein GAN and WGAN-GP implementations

Added PyTorch implementations for Wasserstein GAN (WGAN) loss functions and the Gradient Penalty technique (WGAN-GP) to improve training stability. The \labml\_nn/gan/wasserstein\ module now provides \DiscriminatorLoss\ and \GeneratorLoss\ classes that enforce the 1-Lipschitz constraint via ReLU clipping, while the \gradient\_penalty\ submodule implements the penalty term from the Improved Training of Wasserstein GANs paper to avoid weight clipping issues. Experiment scripts are included to demonstrate training these models on MNIST using convolutional networks.

_labml\nn/gan/wasserstein · high confidence

Add Weight Standardization implementation and CIFAR-10 experiment

Introduces a PyTorch implementation of Weight Standardization, a technique that normalizes convolutional weights to improve training stability and avoid elimination singularities, particularly useful for small batch sizes. This includes a custom \Conv2d\ layer that applies standardization before the convolution operation, an annotated tutorial explaining the mathematical background, and a complete experiment demonstrating how to train a VGG network on CIFAR-10 using Weight Standardization combined with Batch-Channel Normalization.

_labml\_nn/normalization/group\_norm, labml\_nn/normalization/weight\standardization · high confidence

Add Zero-DP memory optimization for large-scale model training

Introduces a new Zero-DP (ZeRO Data Parallel) memory optimization implementation in PyTorch, which shards optimizer states, gradients, and parameters across multiple devices to significantly reduce per-node memory consumption. This includes the core \Zero3Layer\ module for wrapping model layers and a demonstration script for fine-tuning GPT-NeoX using this optimizer.

_labml\nn/scaling · high confidence

Add annotated Deep Q-Network (DQN) implementation with dueling architecture and prioritized replay

This change introduces a new PyTorch implementation of Deep Q-Networks (DQN) in the \labml\_nn/rl/dqn\ module. It includes a dueling network model that separates state value and action advantage streams, a prioritized experience replay buffer using a binary segment tree, and a trainer supporting double Q-learning with Huber loss. An experiment module and Jupyter notebook are provided to train agents on Atari Breakout using multi-process workers and dynamic hyperparameters.

_labml\nn/rl/dqn · high confidence

Add annotated LoRA implementation for GPT-2 fine-tuning

This change introduces a new annotated PyTorch implementation of Low-Rank Adaptation (LoRA) within the \labml\_nn\ library, enabling efficient fine-tuning of large language models by injecting trainable rank decomposition matrices into pre-trained weights. The update includes core LoRA modules (\Linear\ and \Embedding\) that freeze original weights while adding low-rank updates, a complete GPT-2 model architecture (\GPTModel\) utilizing these LoRA layers for token embeddings, attention projections, and feed-forward networks, and a training experiment script that demonstrates fine-tuning a pre-trained GPT-2 model on the Tiny Shakespeare dataset using the Hugging Face tokenizer and model weights.

_labml\nn/lora · high confidence

Add annotated Sketch RNN implementation

Introduces an annotated PyTorch implementation of the Sketch RNN model (A Neural Representation of Sketch Drawings). This includes a dataset loader for processing Quick! Draw stroke data, a Bivariate Gaussian Mixture class for predicting stroke distributions, and the core sequence-to-sequence variational auto-encoder architecture for generating sketches.

_labml\_nn/sketch\rnn · high confidence

Add gMLP (Pay Attention to MLPs) implementation and experiment

Added a PyTorch implementation of the gMLP architecture, featuring a GMLPBlock with a Spatial Gating Unit and a training experiment on the Tiny Shakespeare dataset.

_labml\nn/transformers/gmlp · high confidence

Add k-Nearest Neighbor Language Model (kNN-LM) implementation

Introduces a PyTorch implementation of k-Nearest Neighbor Language Models to improve the perplexity of autoregressive transformer models. This addition includes training code for the base transformer, a script to build a FAISS index of transformer embeddings, and an evaluation module that merges k-NN predictions with transformer outputs to achieve better results.

_labml\nn/transformers/knn · high confidence

Add knowledge distillation tutorial for CIFAR-10

Added a new tutorial implementing the 'Distilling the Knowledge in a Neural Network' paper. This includes a large teacher model and a small student model trained on CIFAR-10, along with a distillation training step that combines soft target KL divergence loss with standard cross-entropy loss to improve the small model's accuracy.

_labml\nn/distillation · high confidence

Add original GAN implementation with label smoothing

Added a new implementation of the original Generative Adversarial Network (GAN) architecture, including the core loss functions in \\_\init\\_.py\ and a complete training experiment in \experiment.py\. The loss classes (\DiscriminatorLogitsLoss\ and \GeneratorLogitsLoss\) incorporate label smoothing (default 0.2) to improve training stability, and the experiment demonstrates training on MNIST using multi-layer perceptrons for both the generator and discriminator.

_labml\nn/gan/original · high confidence

Add reusable experiment trainers for MNIST, CIFAR-10, and NLP tasks

This change introduces a new \labml\_nn/experiments\ module containing reusable trainer configurations and datasets for common machine learning tasks. It adds \MNISTConfigs\ and \CIFAR10Configs\ for image classification, including a VGG model implementation for CIFAR-10, and \NLPAutoRegressionConfigs\ and \NLPClassificationConfigs\ for natural language processing tasks, featuring gradient clipping, sampling, and configurable tokenizers. An \ArithmeticDataset\ is also included to support arithmetic reasoning experiments.

_labml\nn/experiments · high confidence

Add transformer auto-regression experiment and Sophia-G optimizer variant

This change introduces a new auto-regressive transformer experiment in the \labml\_nn/transformers/basic\ module, providing both a standard implementation using the Noam optimizer and a variant using the Sophia-G optimizer. The standard experiment trains a transformer on the Tiny Shakespeare dataset using character-level tokenization, while the Sophia-G variant integrates a Gauss-Newton-Bartlett Hessian diagonal estimator for second-order optimization updates.

_labml\nn/transformers/basic · high confidence

Added Batch-Channel Normalization implementation

This change introduces a new \BatchChannelNorm\ module in the \labml\_nn\ library. It implements the Batch-Channel Normalization technique from the paper 'Micro-Batch Training with Batch-Channel Normalization and Weight Standardization'. The module combines a batch normalization step (supporting both standard and estimated running mean/variance modes for small batches) with a subsequent channel normalization step, providing a normalization layer suitable for scenarios where batch sizes are small.

_labml\_nn/normalization/batch\_channel\norm · high confidence

Added Masked Language Model (MLM) implementation and training experiment

Added a new Masked Language Model (MLM) module and a corresponding training experiment. The \MLM\ class implements the token masking strategy used in BERT pre-training, where 15% of tokens are masked, with 10% replaced by the original token and 10% by random tokens to improve representation learning. The accompanying experiment demonstrates training this model on the Tiny Shakespeare dataset, including configuration for masking probabilities, loss calculation (ignoring padding tokens), and accuracy tracking.

_labml\nn/transformers/mlm · high confidence

Added PyTorch LSTM implementation with layer normalization support

The labml\_nn/lstm module now provides a custom PyTorch implementation of Long Short-Term Memory (LSTM) networks, including an LSTMCell and a multilayer LSTM wrapper. This addition introduces support for optional layer normalization on the cell gates and cell state, which is noted to improve training stability. The implementation replaces the previous dependency on labml\_helpers by inheriting directly from torch.nn.Module and uses standard PyTorch operations for the forward pass, including chunking linear transformations for input, forget, gate, and output gates.

_labml\nn/lstm · high confidence

Added RoPER: Rotary Positional Embeddings with Relative Distance

This change introduces RoPER, a new positional embedding method that explicitly adds relative distance information to value embeddings in multi-head attention. The implementation includes a \ReverseRotaryPositionalEmbeddings\ module and a \RotaryValuePEMultiHeadAttention\ layer that rotate value embeddings relative to the current position before the weighted sum, then reverse-rotate the result. Two experiment scripts are provided to demonstrate the approach on arithmetic addition and text generation tasks.

_labml\_nn/transformers/rope/value\pe · high confidence

Added Rotary Positional Embeddings (RoPE) implementation and training experiment

Added a new PyTorch implementation of Rotary Positional Embeddings (RoPE) in \labml\nn/transformers/rope/\\init\\_.py\, which encodes token positions using rotation matrices to incorporate relative position dependency in attention mechanisms. The module includes the \RotaryPositionalEmbeddings\ class and integrates with the existing multi-head attention framework via \RotaryPEMultiHeadAttention\. Additionally, an experiment script (\experiment.py\) was added to demonstrate training a transformer model with RoPE on the Tiny Shakespeare dataset, configuring the model to use rotary embeddings instead of fixed positional embeddings.

_labml\nn/transformers/rope · high confidence

Added annotated Proximal Policy Optimization (PPO) implementation with Generalized Advantage Estimation

This change introduces a new, fully annotated PyTorch implementation of the Proximal Policy Optimization (PPO) algorithm in the \labml\_nn/rl/ppo\ module. It includes the core PPO loss logic with clipped policy and value function updates, a Generalized Advantage Estimation (GAE) component for variance reduction, and a complete experiment setup for training an agent on Atari Breakout using multiple worker processes. The addition also provides a Jupyter notebook and documentation to guide users through running the experiment and adjusting dynamic hyperparameters.

_labml\nn/rl/ppo · high confidence

Added annotated Stable Diffusion implementation and utilities

This change introduces a new annotated PyTorch implementation of Stable Diffusion within the \labml\_nn\ library. It adds the core \LatentDiffusion\ model, which wraps a U-Net with attention, an autoencoder, and a CLIP text embedder to match the structure of the official CompVis repository for direct weight compatibility. Additionally, it provides utility functions for loading the model from checkpoints, preprocessing images, and saving generated outputs, serving as the foundation for the library's image generation scripts.

_labml\_nn/diffusion/stable\diffusion · high confidence

Added utility scripts for generating documentation assets

New utility scripts have been added to the \utils\ directory to automate the generation of documentation assets. \papers\_list.py\ scans the project's Python source files to extract arXiv paper IDs and generates a \papers.json\ index mapping papers to their documentation URLs. \sitemap.py\ iterates through the source files using Git history to determine modification dates and produces a \sitemap.xml\ file for the documentation site. \diagrams.py\ provides helper functions to process and style SVG diagrams by converting inline color attributes to CSS classes.

utils · high confidence

Initial release of CFR tutorial and Kuhn poker implementation

Adds a new annotated tutorial and Python implementation for Counterfactual Regret Minimization (CFR) and Monte Carlo CFR (MCCFR), including a playable example in Kuhn poker. The package includes core algorithm definitions, utilities for visualizing information set strategies via Altair charts, and a checkpoint saver to persist and restore game state.

_labml\nn/cfr · high confidence

Initial release of annotated Transformer implementations

This change introduces the core \labml\_nn/transformers\ module, providing PyTorch implementations and educational explanations for the original Transformer architecture and its variants. It includes foundational components such as Multi-Head Attention (MHA), Position-wise Feed-Forward Networks (FFN) with support for GLU variants, and fixed/learned positional encodings. The module also provides configurable Encoder, Decoder, and Encoder-Decoder models, along with utilities like label smoothing loss and subsequent masking, serving as the base for more advanced architectures like Transformer-XL, RoPE, and ALiBi documented in the module's index.

_labml\nn/transformers · high confidence

Initial release of labml-nn v0.5.1 with comprehensive deep learning implementations

The repository is packaged as the 'labml-nn' Python library (version 0.5.1), providing PyTorch implementations and side-by-side documentation for a wide range of deep learning architectures and algorithms. The release includes implementations for Transformers (including Flash Attention, Transformer XL, LoRA, and GPT-NeoX), Diffusion models (DDPM, Stable Diffusion), GANs (StyleGAN2, CycleGAN), Reinforcement Learning (PPO, DQN), and various Optimizers and Normalization layers. The package depends on torch, einops, numpy, and fairscale, and includes build infrastructure (Makefile, setup.py) to generate HTML documentation and distribute the library via PyPI.

(repo-wide) · high confidence

Introduce DDPM implementation and training infrastructure

Adds a PyTorch implementation of Denoising Diffusion Probabilistic Models (DDPM), including a U-Net architecture with time-step embeddings and attention, a \DenoiseDiffusion\ class for the forward/reverse diffusion processes, and a \Sampler\ for image generation and interpolation. The change also provides training and evaluation scripts (\experiment.py\, \evaluate.py\) and an interactive notebook (\experiment.ipynb\) to train the model on datasets like CelebA HQ or MNIST.

_labml\nn/diffusion/ddpm · high confidence

A new \\_\init\\_.py\ file has been added to the \labml\_nn/gan\ directory, establishing the module's public interface. This file serves as the entry point for the Generative Adversarial Networks tutorial series, documenting and linking to specific implementations including Original GAN, DCGAN, Cycle GAN, Wasserstein GAN, Wasserstein GAN with Gradient Penalty, and StyleGAN 2.

_labml\nn/gan · high confidence

Introduction of JAX-based Transformer implementation

Added a new module at labml\_nn/transformers/jax\_transformer providing an autoregressive transformer decoder implemented in JAX. This includes a custom Module base class for managing parameters and sub-modules, along with core layers such as embeddings, multi-head attention, feed-forward networks, and layer normalization, enabling users to build and train transformer models using JAX.

_labml\_nn/transformers/jax\transformer · high confidence

Introduction of diffusion model documentation index

The \labml\nn/diffusion\ package now includes an \\\init\\_.py\ file that serves as the main entry point and documentation index for diffusion model implementations. This file lists available tutorials and models, specifically highlighting Denoising Diffusion Probabilistic Models (DDPM), Stable Diffusion, Latent Diffusion Models, and Denoising Diffusion Implicit Models (DDIM) sampling, providing users with a structured overview of the available resources in this module.

_labml\nn/diffusion · high confidence

Introduction of labml\_nn package with annotated implementations

The labml\_nn package is introduced, providing a collection of simple PyTorch implementations of neural networks and related algorithms, documented with explanations and rendered as side-by-side formatted notes. This initial release includes implementations for Transformers (including Flash Attention, RoPE, ALiBi, GPT, etc.), Diffusion models (DDPM, DDIM, Stable Diffusion), GANs (DCGAN, CycleGAN, StyleGAN 2), Graph Neural Networks (GAT, GATv2), Reinforcement Learning (PPO, DQN), and various other architectures like ResNet, U-Net, and optimizers.

_labml\nn · high confidence

Introduction of multi-process Atari game wrapper

A new \labml\_nn.rl\ module has been added, providing a \Game\ class that wraps the OpenAI Gym environment for Atari Breakout. This wrapper implements standard preprocessing steps including frame grayscaling, resizing to 84x84 pixels, stacking the last four frames, and limiting episodes to a single life. It also includes a \Worker\ class that runs the game environment in a separate process using Python's \multiprocessing\ module to enable parallel execution.

_labml\nn/rl · high confidence

New NeoX sample scripts for inference, fine-tuning, and int8 quantization

Added sample scripts in the NeoX samples module to demonstrate text generation, fine-tuning, and quantized inference. The new generate.py script enables single-GPU text generation using a KV-cache for efficiency. The finetune.py script demonstrates fine-tuning GPT-NeoX biases using Fairscale pipeline parallelism. The llm\_int8.py script shows how to run inference with LLM.int8() quantization, reducing memory requirements to fit on a 24GB GPU by converting layers to int8 on the CPU before moving them to the device.

_labml\nn/neox/samples · high confidence

New configurable training infrastructure and optimizer implementations

This change introduces a new set of helper modules for training configuration and a comprehensive library of PyTorch optimizer implementations. The \labml\_nn/helpers\ package now provides configurable components for datasets (MNIST, CIFAR-10), device selection (CPU/GPU), metrics (Accuracy), optimizer configuration (SGD, Adam, Noam), learning rate schedules (Flat, Dynamic, Piecewise), and the training loop itself. Additionally, the \labml\_nn/optimizers\ package adds specific optimizer classes including Adam, AMSGrad, RAdam, AdaBelief, Sophia-G, and variants with warmup and cosine decay, all built upon a shared \GenericAdaptiveOptimizer\ base class.

_labml\nn/optimizers · high confidence

New sampling techniques for language models

Added PyTorch implementations for Greedy, Temperature, Top-k, and Nucleus sampling techniques, along with a base \Sampler\ interface and demo experiments using HuggingFace's GPT-2 and a custom autoregressive transformer. Users can now apply these specific sampling strategies to control text generation behavior in language models.

_labml\nn/sampling · high confidence

New utility functions and tokenizer configurations

Added a new \labml\_nn.utils\ module providing core utilities for neural network experiments. This includes helper functions such as \clone\_module\_list\ for duplicating PyTorch modules and \cycle\_dataloader\ for infinite data iteration, as well as a \MapStyleDataset\ wrapper to convert iterable datasets into shufflable map-style datasets. Additionally, a \TokenizerConfigs\ configuration class is introduced, offering options to switch between character-level and basic English tokenizers via the \torchtext\ library.

_labml\nn/utils · high confidence

New utility modules for GPT-NeoX training, inference, and quantization

Added a new \labml\_nn/neox/utils\ package providing core infrastructure for GPT-NeoX models. This includes \llm\_int8.py\ for LLM.int8() 8-bit quantization to reduce memory footprint, \cache.py\ for caching intermediate activations to speed up inference, \finetune.py\ with a \FineTuner\ base class and \FineTuneBiases\ implementation for parameter-efficient fine-tuning, \trainer.py\ with \TrainerConf\ and \PipelineParallelTrainerConf\ for managing training loops and mixed-precision optimization, \text\dataset.py\ for loading and tokenizing text datasets, and \\\init\\_.py\ with helper functions for tokenization and output printing.

_labml\nn/neox/utils · high confidence

Test coverage

1 commit adding/updating tests in test

Test coverage in test — 1 commit, 1 file: “Handle Gym observation and episode API variants in Atari wrapper”.

test · high confidence · unverified

Dependencies

Initial dependency specification for PyTorch ecosystem and utilities

The project now includes a requirements.txt file that explicitly defines the Python package dependencies. This includes the core PyTorch stack (torch, torchvision, torchtext) with minimum version constraints, along with supporting libraries such as labml, labml-helpers, numpy, matplotlib, einops, gym, opencv-python, Pillow, and faiss.

(dependencies) · high confidence

Housekeeping

Add Normalization Layers documentation index

Added a new documentation index page for the normalization layers module, providing links to tutorials and implementations for Batch Normalization, Layer Normalization, Instance Normalization, Group Normalization, Weight Standardization, Batch-Channel Normalization, and DeepNorm.

_labml\nn/normalization · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 50 → 52 (+2.9)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.1) — scores are not directly comparable.

Lenses

  • Code Health 98 → 98 (+0.0)
  • Architecture 88 → 86 (-2.1)
  • Maturity 32 → 39 (+7.0)
  • Readiness 38 → 38 (+0.0)
  • Security 99 → 99 (+0.0)

Resolved (169)

  • Documentation: no project overview (docs/adaptive_computation/ponder_net/readme.html)
  • Dormant codebase
  • Inconsistent handling of time embeddings in residual blocks. The DDPM UNet ResidualBlock and Stable Diffusion UNet ResBlock accept a time embedding tensor, but the Stable Diffusion Autoencoder ResnetBlock does not accept any time embedding argument, despite being part of the same diffusion pipeline architecture. This suggests an inconsistency in how time conditioning is applied across different components of the diffusion models.
  • Inconsistent method signatures for spatial resizing operations. The DDPM UNet implementation includes a t: Tensor (time step) argument in Upsample and Downsample forward methods, whereas the Stable Diffusion implementations (both autoencoder and UNet modules) do not include this argument. Since these operations are typically spatial and time-independent, the presence of t in one and absence in the other is confusing and inconsistent.
  • Inconsistent parameter naming and typing for conditional inputs in attention mechanisms. The DDPM AttentionBlock uses t: Optional[Tensor] for time conditioning, while the Stable Diffusion SpatialTransformer uses cond: Tensor for text/context conditioning. While the semantic intent differs (time vs text), the naming convention t vs cond is inconsistent across the library's attention modules, and the optionality of the parameter is not standardized.
  • Inconsistent parameter naming in constructors for upsampling layers. The DDPM implementation uses n_channels, while both Stable Diffusion implementations use channels. This minor inconsistency makes it harder to switch between implementations or understand the API surface uniformly.
  • Most significant orphaned file (labml_nn/gan/stylegan/init.py)
  • Most significant orphaned file (labml_nn/transformers/flash/init.py)
  • Most significant orphaned file (labml_nn/transformers/jax_transformer/init.py)
  • Orphaned file (labml_nn/init.py)
  • Orphaned file (labml_nn/activations/fta/init.py)
  • Orphaned file (labml_nn/activations/fta/experiment.py)
  • Orphaned file (labml_nn/adaptive_computation/ponder_net/init.py)
  • Orphaned file (labml_nn/adaptive_computation/ponder_net/experiment.py)
  • Orphaned file (labml_nn/capsule_networks/init.py)
  • Orphaned file (labml_nn/capsule_networks/mnist.py)
  • Orphaned file (labml_nn/cfr/init.py)
  • Orphaned file (labml_nn/cfr/kuhn/init.py)
  • Orphaned file (labml_nn/conv_mixer/init.py)
  • Orphaned file (labml_nn/conv_mixer/experiment.py)
  • …and 149 more

New (1)

  • Orphaned files with no living knowledge

Changes since last survey

  • 5 commits — 4 feature/other, 1 fixes

By area

  • (root) — 2 commits
  • (repo) — 1 commit
  • docs/index.html — 1 commit
  • labml_nn/rl — 1 commit

Notable commits

  • fix: Merge pull request #326 from mvanhorn/fix/245-ppo-gym-observation-api
  • change: Add note about using Annotated Implementation Skill
  • change: Add note on Annotated Implementation Skill usage
  • change: Clarify note about using annotated implementation skill
  • change: Handle Gym observation and episode API variants in Atari wrapper

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

labmlai/annotated_deep_learning_paper_implementations was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 3 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit db59a04a8a2c6af922f1656f8c0fd113dd9a51b8 — the exact code this score is about.
  • Scored under rubric-2026.11.1 — the same rubric and the same method as every other entry in this index.
  • Measured by watchdog.canine.dev using codehealth-analyzer measure-2026.11.1+preprod-7b4ed5af9e84.