Skip to content
CAI
Software that uses CAICheck a score

huggingface/open-r1

62.0

Adequate · 16 November 2026

4.6k

lines of production code

Python

primary language

4

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

This system is an infrastructure framework designed to reproduce the DeepSeek-R1 training pipeline, focusing on supervised fine-tuning and Group Relative Policy Optimization (GRPO). It provides the core training scripts, reward function registries, and dataset management utilities necessary for model development. The system also integrates external code execution environments, such as E2B and MorphCloud, to support automated evaluation and scoring of code submissions against competitive programming benchmarks.

Features

Add dataset pass-rate filtering script

A new script and supporting files have been added to the \scripts/pass\_rate\_filtering\ directory to enable filtering datasets based on model pass rates. The \compute\_pass\_rate.py\ script loads a dataset, generates completions using a specified LLM, computes rewards, and filters examples based on configurable minimum and maximum pass-rate thresholds. A shell script (\launch\_filtering.sh\) is also provided to orchestrate this process in chunks, and a README documents the usage for merging and pushing filtered datasets to the Hugging Face Hub.

_scripts/pass\_rate\filtering · high confidence

Initial project scaffolding with Open R1 training and evaluation infrastructure

This change introduces the foundational structure for the Open R1 project, including the core Python package setup via setup.py and setup.cfg, a Makefile for managing development tasks (style, quality, tests) and running evaluations, and a comprehensive README.md detailing the project's goal to reproduce the DeepSeek-R1 pipeline. The setup defines dependencies for training (TRL, vLLM, DeepSpeed, Transformers) and evaluation (LightEval, math-verify), while the Makefile provides commands to install the environment and execute SFT and GRPO training scripts as well as evaluation tasks.

(repo-wide) · high confidence

Introduce open\_r1 training scripts and reward functions

Adds the initial \open\_r1\ package containing GRPO and SFT training scripts, configuration dataclasses, and a registry of reward functions (including accuracy, format, length, and reasoning steps rewards) for training language models.

_src/open\r1 · high confidence

New unified Slurm infrastructure for serving, training, and evaluation

This change introduces a comprehensive set of new Slurm scripts and documentation to streamline the deployment and execution of the OpenR1 workflow. Users can now launch DeepSeek-R1 inference servers using either SGLang (via \serve\_r1.slurm\ and \serve\_router.slurm\) or vLLM (via \experimental/serve\_r1\_vllm.slurm\) on multi-node GPU clusters. The suite includes \train.slurm\ to orchestrate SFT and GRPO training jobs with configurable data/tensor parallelism and vLLM integration, \evaluate.slurm\ to run LightEval benchmarks and automatically upload results to the Hugging Face Hub, and \generate.slurm\ for scalable data generation. Additionally, new scripts support code execution via Piston workers (\piston/launch\_piston\_workers.sh\) and an E2B router (\e2b\_router.slurm\), providing a complete operational environment for model development and testing.

slurm · high confidence

New utility module for training infrastructure and code execution

This change introduces the \src/open\_r1/utils\ package, providing core infrastructure for the training workflow. It adds a dataset mixer to combine multiple datasets with configurable weights and splits, and implements code execution providers for E2B and MorphCloud to support reward-based evaluation. The package also includes utilities for asynchronous model pushing to Hub revisions, automated benchmark evaluation via LightEval, and Weights & Biases logging configuration.

_src/open\r1/utils · high confidence

New utility scripts for dataset decontamination, code execution routing, and benchmarking

This update introduces several new scripts to the \scripts/\ directory to support data preparation and evaluation workflows. \decontaminate.py\ allows users to check datasets for n-gram overlap against benchmark datasets (AIME, MATH-500, GPQA, LiveCodeBench) and optionally remove contaminated rows. \e2b\_router.py\ and \morph\_router.py\ provide FastAPI-based batch execution servers for running code in isolated sandboxes using E2B and MorphCloud respectively. \benchmark\_e2b.py\ benchmarks the \code\_reward\ function's performance with varying parallelization levels, while \run\_benchmarks.py\ serves as a CLI entry point to trigger benchmark jobs after training. Additional utilities include \generate\_reasoning.py\ for asynchronous LLM response generation, \get\_tensor\_parallel\_size.py\ for calculating optimal tensor parallelism, and \upload\_details.py\ for pushing LightEval results to the Hub.

scripts · high confidence

Support for Codeforces and IOI competitive programming problem scoring

Added a new utility module for evaluating code submissions against competitive programming problems from Codeforces and the International Olympiad in Informatics (IOI). The module includes scoring logic for Codeforces (supporting pass/fail, partial, and weighted sum modes) and IOI (supporting subtask-based scoring with detailed status reporting). It also provides code patching utilities to fix common Python 3 and C++ compilation issues (such as deprecated imports and missing headers) and integrates with Piston and MorphCloud execution environments to run and score submissions asynchronously.

_src/open\_r1/utils/competitive\programming · high confidence

Test coverage

Added slow integration tests for code reward execution via E2B and Morph Cloud routers; Added test coverage for reward functions; Added tests for dataset loading and mixture configuration.

Housekeeping

Added logs directory placeholder

A new .gitkeep file has been added to the logs directory to ensure the folder is tracked by version control.

logs · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 61 → 62 (+0.6)
  • Rubric changed (rubric-2026.11.2 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 63 → 64 (+0.9)
  • Architecture 100 → 100 (+0.1)
  • Maturity 54 → 50 (-3.9)
  • Readiness 57 → 67 (+10.8)
  • Security 84 → 84 (+0.0)
  • Performance 100 → 100 (+0.0)

Resolved (1)

  • generate_reasoning.main (cognitive 18) (scripts/generate_reasoning.py)

New (1)

  • Edited copy of a member (47 corresponding lines) (src/open_r1/rewards.py)

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

huggingface/open-r1 was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 16 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 5b6ff22b3fb7aa069c54866e517f39dfc3160e09 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.73+preprod-52de90b74216.