stanfordnlp/dspy

This system is DSPy, a framework for programming with large language models that provides a structured way to define, optimize, and execute AI programs. It features a modernized adapter layer for interacting with various LLM providers, supports multimodal inputs like images and audio, and includes tools for automated prompt adaptation and evaluation. The codebase has been refactored to remove legacy components, introducing a vendored provider layer and comprehensive testing infrastructure to ensure reliability across different models and execution environments.

10 changes.

This period focused on the DSPy 3.4.0 release, introducing a vendorized lm15 provider layer, new adapters, and updated core dependencies. It also involved a comprehensive cleanup of legacy DSP modules, including the removal of old caching mechanisms, primitives, templates, and evaluation utilities to align with v2 standards.

12 changes.

This period focused on establishing a robust and extensive test suite for the DSPy framework, covering core primitives, prediction modules, evaluation engines, and client infrastructure. Significant effort was dedicated to reliability testing across multiple LLM providers, ensuring deterministic behavior through mock interpreters and fixtures, and validating new features like file/image types and callback systems.

9 changes.

This period focused on significantly expanding test coverage across core infrastructure, including adapters, streaming, retrievers, and the new Flex module, while verifying the migration to lm15 types. It also introduced automated scripts to vendor the lm15 dependency and sync LiteLLM metadata, ensuring reliable runtime fallbacks and dependency management.

Added two new scripts to automate the maintenance of bundled dependencies. \scripts/update\_vendored\_lm15.py\ manages the \lm15\ package as a Git subtree within \dspy/\_vendor\, handling version pinning, contract verification, and license storage. \scripts/update\_model\_metadata.py\ fetches the latest LiteLLM wheel to extract and compress its model pricing and context window data into a local snapshot, ensuring the runtime loader has an up-to-date fallback map without requiring a direct import of the upstream library.

scripts · high confidence

The repository now includes a \.pre-commit-config.yaml\ file that enforces code quality using \ruff\ for linting and formatting on Python files within the \dspy\ and \tests\ directories. A \CONTRIBUTING.md\ guide has been added to instruct developers on the fork-and-PR workflow, environment setup via \uv\ or \conda\, and policies for AI-assisted contributions. Additionally, a \LICENSE\ file (MIT) and a \SECURITY.md\ file defining vulnerability reporting procedures are now present, establishing the legal and security framework for the project.

(repo-wide) · high confidence

The \dsp.primitives\ package has been removed, deleting the \\_\init\\_.py\ and all its submodules (\demonstrate\, \predict\, \primitives\, \search\). This eliminates the legacy implementation of demonstration handling, prediction generation, and retrieval primitives that were previously exposed through this namespace.

dsp/primitives · high confidence

The \dsp/modules\ package has removed the legacy \GPT3\ class, the \ColBERTv2\ retrieval module, and the \cache\_utils\ infrastructure. Users relying on these specific components for OpenAI text completion, ColBERT-based retrieval, or the joblib-based caching mechanism will no longer have access to them through this module path, as the files and their exports have been deleted.

dsp/modules · high confidence

The \dsp/templates\ package has been completely removed, deleting the \\_\init\\_.py\ entry point and the underlying implementation files (\template\_v2.py\, \template\_v3.py\, and \utils.py\). This eliminates the legacy \TemplateV2\ and \Template\ classes, along with helper functions like \passages2text\ and \format\_answers\, meaning users can no longer import or utilize these specific template structures for prompt formatting.

dsp/templates · high confidence

The dsp package no longer exposes its internal modules, primitives, templates, or settings via the main \_\init\\_.py file. This change removes the automatic re-export of these components, meaning users can no longer import them directly from the top-level dsp namespace and must instead import them from their specific submodules.

dsp · high confidence

The \dsp/utils\ package has been removed, deleting the \\_\init\\_.py\, \metrics.py\, \settings.py\, and \utils.py\ files. This eliminates legacy utilities including the singleton \Settings\ context manager, text normalization and evaluation metrics (such as EM, F1, and HotPotF1), and general helper functions like \dotdict\ and \print\_message\. Users relying on these specific legacy components will need to migrate to the updated DSPy equivalents or external libraries.

dsp/utils · high confidence

The \dsp/evaluation/utils.py\ module has been removed. This deletion eliminates the legacy \evaluate\ function that previously executed a model against a dataset, printed summary statistics, and displayed results in an IPython environment. Users relying on this specific utility for evaluation workflows will need to adopt alternative methods.

dsp/evaluation · high confidence

The legacy v1 joblib cache wrapper files for ColbertV2 and GPT3 modules have been removed. This cleanup eliminates the deprecated caching mechanism that previously wrapped request functions with \@NotebookCacheMemory.cache\ and \@functools.lru\_cache\, aligning the codebase with the v2 release standards.

cache/joblib/dsp/modules/colbertv2 · high confidence

This entry covers the DSPy 3.4.0 release, which introduces a new vendorized \lm15\ package (version 1.0.0rc3) to handle low-level, provider-neutral LLM API interactions. The \dspy/\_\init\\_.py\ module is restructured to expose this new layer alongside updated adapters (\ChatAdapter\, \JSONAdapter\, \XMLAdapter\, \TwoStepAdapter\), new data types (\Image\, \Audio\, \File\, \Code\, \Reasoning\), and utility functions (\asyncify\, \syncify\, \streamify\, \track\_usage\). The release also includes a comprehensive set of new error classes for better LM error handling and updates to the settings configuration API.

dspy · high confidence

Added GEPA test fixtures for dummy LM and custom component selection; Added comprehensive test suite for DSPy primitives; Added comprehensive test suite for DSPy utility modules; Added comprehensive test suite for dspy.Flex module binding, GEPA optimization, and interpreter sandboxing; Added test coverage for decision types, aggregation, and interpreter-based prediction modules; Added test coverage for evaluation metrics and engine behavior; Added test infrastructure for DSPy reliability, interpreters, and vendored lm15; Added test infrastructure for LiteLLM server integration testing; Added tests for ColBERTv2 error handling and Embeddings save/load functionality; Added tests for Dataset seed reset and CSV input handling; Added tests for File, Image, and custom type handling in signatures; Added tests for GroundedProposer instruction generation; Added tests for package metadata; Added tests for streaming ReActV2 tool calls and status message providers; Added tests for the ReAnchor calibration optimizer; Added tests for the callback system; Expanded test coverage for DSPy client infrastructure and safety features; Expanded test coverage for adapter infrastructure and type handling; Initial DSPy reliability test suite and generation tooling; Tests verify retirement of legacy LM types and migration to lm15.

The main package is updated to version 3.4.0 with a Python requirement of 3.10–3.14 and new core dependencies including pydantic\>=2.11.0, openai\>=1.66.2, litellm\>=1.65.8, and gepa\[dspy\]==0.1.4. A compatibility alias package (dspy-ai) is added to redirect users to the main dspy package. Documentation tooling is updated with mkdocstrings 1.0.6, mkdocstrings-python 2.0.7, and mistune 3.3.4, alongside a new docs/requirements.txt and Pipfile for building the site.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 45 → 62 (+16.7)
• Rubric changed (rubric-2026.08.15 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 91 → 79 (-12.0)
• Architecture 96 → 94 (-1.5)
• Maturity 62 → 62 (-0.4)
• Readiness 30 → 52 (+22.8)
• Security 43 → 77 (+33.7)

• Coverage not measured — test suite did not build
• Dimension evaluation failed
• Duplicated block (11 lines × 3) (dspy/teleprompt/grpo.py)
• Duplicated block (14 lines × 2) (dspy/teleprompt/grpo.py)
• Duplicated block (5 lines × 2) (dspy/core/types.py)
• Duplicated block (5 lines × 2) (dspy/teleprompt/copro_optimizer.py)
• Duplicated block (6 lines × 2) (dspy/teleprompt/mipro_optimizer_v2.py)
• Duplicated block (7 lines × 2) (dspy/teleprompt/copro_optimizer.py)
• Duplicated block (8 lines × 2) (dspy/primitives/base_module.py)
• Duplicated block (8 lines × 2) (tests/predict/test_rlm.py)
• Duplicated block (9 lines × 2) (dspy/datasets/gsm8k.py)
• LLM evaluation failed
• Low: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• …and 12 more

• Adapter._call_postprocess (cognitive 33) (dspy/adapters/base.py)
• Adapter._call_postprocess (cyclomatic 19) (dspy/adapters/base.py)
• Adapter._call_preprocess (cognitive 24) (dspy/adapters/base.py)
• Adapter.format_conversation_history (cognitive 53) (dspy/adapters/base.py)
• Adapter.format_conversation_history (cyclomatic 26) (dspy/adapters/base.py)
• AnthropicLM._part (cognitive 25) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM._part (cyclomatic 18) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM._payload (cognitive 119) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM._payload (cyclomatic 82) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM.normalize_error (cognitive 24) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM.normalize_error (cyclomatic 25) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM.parse_response (cognitive 38) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM.parse_response (cyclomatic 24) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM.parse_stream_events (cognitive 70) (dspy/_vendor/lm15/providers/anthropic.py)
• AnthropicLM.parse_stream_events (cyclomatic 54) (dspy/_vendor/lm15/providers/anthropic.py)
• AsyncVideoJob.wait (cognitive 18) (dspy/_vendor/lm15/video_jobs.py)
• BaseModule.named_parameters (cognitive 17) (dspy/primitives/base_module.py)
• BaseModule.named_sub_modules (cognitive 19) (dspy/primitives/base_module.py)
• BaseProviderLM._bind_access (cognitive 20) (dspy/_vendor/lm15/providers/base.py)
• BaseProviderLM._bind_access (cyclomatic 17) (dspy/_vendor/lm15/providers/base.py)
• …and 537 more

• 109 commits — 82 feature/other, 27 fixes

• .github/workflows — 19 commits
• docs/docs — 17 commits
• (root) — 8 commits
• dspy/primitives — 8 commits
• dspy/adapters — 7 commits
• dspy/clients — 6 commits
• dspy/predict — 6 commits
• docs/scripts — 5 commits
• dspy/_vendor — 5 commits
• dspy/teleprompt — 5 commits
• docs/versioning — 3 commits
• dspy/utils — 3 commits
• docs/Pipfile.lock — 2 commits
• docs/requirements.txt — 2 commits
• dspy/streaming — 2 commits
• tests/predict — 2 commits
• tests/primitives — 2 commits
• .agents/skills — 1 commit
• docs/Pipfile — 1 commit
• docs/hooks — 1 commit

• fix: Fix Deno versioning and isolate PythonInterpreter dependencies (#10186)
• fix: Fix Document.format() using text source type for PDFs (#9977)
• fix: Fix max_iters default value in docstring of ReAct (#10013)
• fix: Proposed fix of Issue #10064, rolling up nested cost trackers to the parent on closing (#10065)
• fix: fix(adapters): add a default timeout to Image.from_url / Audio.from_url (#10149)
• fix: fix(adapters): coerce non-string Literal members in parse_value (#10010)
• fix: fix(adapters): drop tool_choice when tools aren't sent to the LM (#10399)
• fix: fix(adapters): only quote string members of Literal in BAMLAdapter schema (#10074)
• fix: fix(callbacks): report base exceptions to end handlers (#10194)
• fix: fix(ci): install pytest-mock in release test environments (#10384)
• fix: fix(deps): require Pydantic 2.11 (#10271)
• fix: fix(dspy): propagate LocalInterpreter host-tool cancellation (#10379)
• fix: fix(dspy): reject sync async-tool calls inside running loops (#10146)
• fix: fix(dspy): stabilize local interpreter timeout test (#10404)
• fix: fix(evaluate): raise descriptive ValueError on empty devset instead of ZeroDivisionError (#9978)
• fix: fix(gepa): keep trace-capture outputs aligned to the batch when the program raises (#10305)
• fix: fix(interpreter): desync interpreter on unhandled sandbox rejections (#10190)
• fix: fix(predict): ProgramOfThought._parse_code drops output-capturing echo for single-line generated code (#10050)
• fix: fix(predict): raise when ReActV2 cannot submit final outputs (#10355)
• fix: fix(rlm)!: accept interpreter factories at call time (#10493)
• …and 89 more

• Containers 0 added · 0 removed · contexts 1 added · 0 removed · edges 0 added · 0 removed

• dspy

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

stanfordnlp/dspy was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.