Skip to content
CAI
Software that uses CAICheck a score

datalab-to/marker

56.4

Adequate · 15 November 2026

16.8k

lines of production code

Python

primary language

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add example for deploying Marker via Modal

Added a new example demonstrating how to deploy the Marker PDF conversion service using Modal. This includes a self-contained Python script (\marker\_modal\_deployment.py\) that sets up a FastAPI endpoint on a GPU-enabled container, handles model caching via a persistent volume, and exposes a \/convert\ endpoint for converting documents to Markdown, HTML, or JSON. A corresponding \README.md\ provides instructions for prerequisites, deployment, and testing the service locally or via API.

examples · high confidence

Added LaTeX to Markdown conversion utility

A new shell script (data/latex\_to\_md.sh) has been added to convert LaTeX files into Markdown format using Pandoc. The script processes all .tex files in the 'latex' directory, outputs the converted Markdown to the 'references' directory, and performs post-processing to replace non-breaking spaces and remove citation commands and other formatting artifacts. A .gitignore file has also been added to exclude generated 'latex', 'pdfs', and 'references' directories from version control.

data · high confidence

Dynamic CLI configuration via automatic config crawling

The CLI now automatically discovers and exposes configuration options for all internal components (Builders, Processors, Converters, Providers, Renderers, and Services) by crawling their Python type annotations. This allows users to pass arbitrary settings via command-line flags (e.g., \--processor\_name\_attr\) or JSON config files without hard-coded CLI definitions, while maintaining backward compatibility for common options like output format and debug mode.

marker/config · high confidence

Initial project scaffolding with Apache 2.0 licensing and pre-commit tooling

The repository is initialized with the core project structure, including entry-point scripts for the CLI, Streamlit app, and server. The project adopts the Apache 2.0 license for the codebase and the OpenRAIL-M license for the models, accompanied by a Contributor License Agreement. Development workflow is established via a pre-commit configuration for Ruff linting and formatting, alongside a \uv.lock\ file pinning dependencies for Python 3.10–3.14 and a \pytest.ini\ defining test markers and warning filters.

(repo-wide) · high confidence

Introduce structured document schema and block registry

The \marker/schema\ package now provides a formalized data model for document processing. It defines a \BlockTypes\ enumeration covering 29 distinct content types (such as \Table\, \Equation\, \ComplexRegion\, and \ChemicalBlock\) and implements a registry system that maps these types to their corresponding Pydantic model classes. This foundation supports the \Document\ and \PageGroup\ structures, enabling the system to track page navigation, manage table of contents, and handle layout labels from the Surya OCR engine.

marker/schema · high confidence

Introduces LLM-based processors for correcting complex document elements

Adds a new suite of LLM-powered processors in the \marker/processors/llm\ module to improve the accuracy of extracted content. These processors allow the system to use large language models to correct and refine specific block types, including equations (converting to LaTeX), complex regions, forms, handwriting, images, tables, and section headers. The implementation includes base classes for handling LLM inference, concurrency, and prompt management, along with specific processors for each block type that send images and extracted text to the LLM service to generate corrected HTML or Markdown output.

marker/processors/llm · high confidence

Introduces structured document group types for layout processing

The schema now defines specific group classes—FigureGroup, TableGroup, ListGroup, PictureGroup, and PageGroup—to organize document elements. These groups provide structured containers for child blocks (such as figures with captions or list items) and include logic for assembling HTML output, managing page-level image rendering (including deferred high-resolution loading), and handling block intersections and replacements. This change establishes the foundational data structures for grouping and rendering complex document layouts.

marker/schema/groups · high confidence

New CLI, API, and Streamlit entrypoints for PDF conversion

The marker package now includes dedicated scripts for running conversions via command line, REST API, and interactive web app. The \convert\ and \convert\_single\ scripts provide CLI entry points for batch and single-file processing, featuring multi-node sharding, worker recycling, and automatic device-based mode selection (fast on CPU/MPS, balanced on GPU). A new FastAPI-based server (\server.py\) exposes \/marker\ and \/marker/upload\ endpoints for programmatic access, supporting markdown, JSON, HTML, and chunks output formats. Additionally, a Streamlit application (\streamlit\_app.py\) offers a user-friendly interface for uploading PDFs or images, selecting output formats and modes, and viewing results with optional debug visualizations. Also in this area: 1 commit (0 fixes) modifying marker; 11 commits (3 fixes) modifying marker; 2 commits (2 fixes) fixing marker.

marker/scripts · medium confidence · unverified

New document structure processors for layout, content, and debugging

The \marker/processors\ module now includes a comprehensive suite of new processors that handle specific document structure tasks. These include \BlankPageProcessor\ to filter out blank pages, \BlockRelabelProcessor\ to heuristically relabel blocks based on confidence thresholds, \BlockquoteProcessor\ for tagging blockquotes, \CodeProcessor\ for formatting code blocks, \DebugProcessor\ for dumping layout and PDF debug data, \DocumentTOCProcessor\ for generating a table of contents, \EquationProcessor\ for recognizing equations and inline math via OCR, \FootnoteProcessor\ for pushing footnotes to the bottom and assigning superscripts, \IgnoreTextProcessor\ for filtering common elements like headers and footers, \LineMergeProcessor\ for merging inline math lines, \LineNumbersProcessor\ for ignoring line numbers, \ListProcessor\ for merging lists across pages and columns, \MarginaliaProcessor\ for relabeling running headers/footers, \PageHeaderProcessor\ for moving page headers to the top, \ReferenceProcessor\ for adding references to the document, and \SectionHeaderProcessor\ for recognizing section headers. These processors collectively enhance the accuracy and structure of the extracted document content.

marker/processors · high confidence

New modular LLM service architecture with multi-provider support

The marker package introduces a new, unified service layer in marker/services that replaces the previous ad-hoc LLM integration with a structured, extensible architecture. This change adds a BaseService abstract class and concrete implementations for OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, Google Vertex AI, OpenRouter, and local Ollama models. Users benefit from consistent configuration patterns (timeout, retries, image formats) across all providers, structured output parsing via Pydantic schemas, and improved reliability through standardized retry logic for rate limits and timeouts. The new system also supports dynamic schema passing for better compatibility with different LLM processing units and includes proper token usage tracking for block metadata.

marker/services · high confidence

New provider architecture supporting multiple document formats

The system now introduces a modular provider architecture that extends beyond PDFs to natively support DOCX, EPUB, HTML, images, PowerPoint, and spreadsheets. These new formats are processed by converting them into PDFs via WeasyPrint (using libraries like mammoth, ebooklib, and openpyxl) before being handled by the existing PDF extraction pipeline, while images are processed directly. A file-type registry automatically detects the input format and routes it to the appropriate provider.

marker/providers · high confidence

New renderer architecture with configurable output options

The marker/renderers module has been restructured to introduce a BaseRenderer and specialized renderers (HTML, JSON, Chunk, Markdown, OCRJSON) that support new configuration options. Users can now control image extraction via \extract\_images\ (defaulting to true) and \image\_extraction\_mode\ (lowres/highres), choose to retain page headers and footers with \keep\_pageheader\_in\_output\ and \keep\_pagefooter\_in\_output\, and inject block identifiers into HTML output using \add\_block\_ids\. The HTML renderer specifically supports paginated output via \paginate\_output\ and the Markdown renderer allows for configurable math delimiters, HTML table preservation, and pagination.

marker/renderers · high confidence

New text schema with rich formatting and hyphenation handling

The text schema now includes dedicated classes for characters, lines, and spans, enabling more granular text representation. Spans support a wide range of formats including bold, italic, math, superscript, subscript, code, and underline, which are correctly rendered as HTML tags. Lines now handle hyphenation logic to remove trailing hyphens when appropriate and strip extra whitespace, improving text continuity. The schema also supports URL links within spans and integrates with a math unwrapping utility for cleaner inline math rendering.

marker/schema/text · high confidence

Architecture

Architectural shift to shared inference servers and removal of legacy local models

Marker now offloads heavy model inference (layout, recognition, OCR error) to shared external servers, allowing multiple worker processes to share a single GPU without each loading a model copy. The legacy local models (Nougat, LayoutLMv3) and their associated processing modules (bbox, code, equations, extract\_text, headers, markdown, schema, segmentation) have been removed. Configuration has been updated to support this architecture, including new settings for output encoding, image format, and font handling, while the \TORCH\_DEVICE\ setting now primarily applies to lightweight local models like the OCR error predictor.

marker · high confidence

Behavioural changes

3 commits (1 fix) modifying benchmarks

A change to existing behaviour in benchmarks — 3 commits (1 fix), 3 files: “Add docling and liteparse OmniParseBench scores, charts, and positioning”; “Add indra-ocr integration, 4 conversion modes, updated viewer/CLI, and accuracy fixes”; “Update benchmarks and charts for OmniParseBench and representative public scores”. Also in this area: 1 commit (0 fixes) modifying benchmarks, examples; New reproducible benchmark harness for quality and throughput.

benchmarks · medium confidence · unverified

Introduce modular converter architecture with specialized PDF, OCR, and table extraction modes

The \marker/converters\ package has been restructured into a modular system centered on a \BaseConverter\ that handles dependency resolution and processor initialization, including a new meta-processor for batching LLM requests. This introduces three distinct conversion paths: the standard \PdfConverter\ (which now defaults to 'balanced' mode on GPU and 'fast' mode on CPU/MPS), an \OCRConverter\ for forced OCR workflows, and a new \TableConverter\ that isolates table/form extraction by disabling general OCR and filtering document structure. Users can now choose the appropriate converter for their specific data type, with the table converter offering a dedicated pipeline for complex table and form structures.

marker/converters · high confidence

New block schema and HTML rendering logic for document elements

The \marker/schema/blocks\ module now defines a comprehensive set of block types (including Text, Equation, Table, ListItem, Footnote, and ComplexRegion) with dedicated HTML assembly logic. This change introduces structured HTML output for document components, supporting features such as nested blockquotes, list indentation, table cell spanning, and configurable inclusion of page headers/footers via \block\_config\ options like \keep\_pageheader\_in\_output\ and \add\_block\_ids\.

marker/schema/blocks · high confidence

Optimized batch concurrency and blank-content detection

The marker utility layer now includes a new batch processing module that calculates worker counts based on physical CPU cores and server parallelism settings, allowing for more efficient resource utilization during conversion tasks. Additionally, the image processing logic has been refactored to use a faster, bit-for-bit equivalent method for detecting blank images by leveraging adaptive thresholding directly, eliminating expensive connected-components operations. This change also introduces a specific fix to treat polygons with identical coordinates as blank, ensuring consistent filtering of empty content.

marker/utils · high confidence

Refactored document processing pipeline with new builder architecture

The marker/builders module has been completely rewritten to introduce a modular builder architecture (BaseBuilder, DocumentBuilder, LayoutBuilder, LineBuilder, OcrBuilder, StructureBuilder). This refactoring changes how documents are processed: layout detection now supports a 'fast' mode using a lightweight detector for CPU/MPS devices, while 'balanced' mode uses a VLM model on GPU. Reading order is now determined by the PDF's character stream on text-rich pages to improve accuracy, and high-resolution images are rendered lazily only for pages that require them (e.g., those with tables or equations). OCR is now performed as a full-page request for pages with bad embedded text, with a fallback to block-level OCR for specific garbled regions, and structure grouping (lists, captions) is handled by a dedicated StructureBuilder.

marker/builders · high confidence

Fixes

2 commits (2 fixes) fixing marker

A fix in marker — 2 commits (2 fixs), 69 files: “Add indra-ocr integration, 4 conversion modes, updated viewer/CLI, and accuracy fixes”; “Fix layout/table/caption detection, existing OCR handling, and HF model fallback”.

marker/ocr · medium confidence · unverified

Test coverage

2 commits adding/updating tests in tests/schema/blocks; 7 commits adding/updating tests in tests; Added comprehensive test coverage for PDF converters and modes; Added comprehensive test suite for document builders; Added test coverage for document processors; Added test coverage for renderer components; Added test for list grouping structure validation; Added tests for LLM service initialization; Added tests for document, image, and PDF providers; Added unit tests for CLI configuration parsing; Initial test infrastructure for PDF conversion pipeline.

Dependencies

Migration from Poetry to Hatch and major dependency overhaul

The project has switched its build system from Poetry to Hatch (pyproject.toml), removing the legacy poetry.lock and requirements.txt files. This update bumps the package version to 2.0.0 and significantly updates dependencies: it replaces PyMuPDF with pdftext, swaps nougat-ocr for surya-ocr, and updates core libraries like Pillow, Pydantic, and transformers to newer major versions. It also adds support for new document formats via optional dependencies (mammoth, openpyxl, python-pptx, ebooklib, weasyprint) and introduces new CLI entry points for server and single-file conversion.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 56 → 56 (+0.9)
  • Rubric changed (rubric-2026.11.1 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 79 → 71 (-7.5)
  • Architecture 85 → 96 (+10.2)
  • Maturity 64 → 64 (-0.2)
  • Readiness 46 → 46 (+0.0)
  • Security 57 → 60 (+3.0)
  • Performance 70 → 74 (+4.3)

Resolved (18)

  • Critical CVE: [GHSA redacted] (uv.lock)
  • Duplicated block (10 lines × 2) (marker/processors/llm/llm_table.py)
  • Duplicated block (12 lines × 3) (marker/schema/blocks/inlinemath.py)
  • Duplicated block (12 lines × 3) (marker/services/claude.py)
  • Duplicated block (16–17 lines × 2) (marker/processors/llm/llm_complex.py)
  • Duplicated block (16–17 lines × 2) (marker/services/azure_openai.py)
  • Duplicated block (16–19 lines × 2) (marker/config/parser.py)
  • Duplicated block (19 lines × 2) (marker/processors/llm/init.py)
  • Duplicated block (9 lines × 2) (marker/schema/blocks/pagefooter.py)
  • LLMPageCorrectionProcessor.handle_reorder (cognitive 16) (marker/processors/llm/llm_page_correction.py)
  • Low vulnerability: [GHSA redacted] (uv.lock)
  • No ADRs found
  • Off-boarding risk: anonymized user #1
  • Orphaned file (marker/processors/blank_page.py)
  • Orphaned file (marker/providers/epub.py)
  • Orphaned file (marker/providers/spreadsheet.py)
  • Projects may be oversized for their cohesion
  • TodoComment (marker/schema/text/span.py)

New (69)

  • AzureOpenAIService.call (cognitive 16) (marker/services/azure_openai.py)
  • Block.contained_blocks (cognitive 20) (marker/schema/blocks/base.py)
  • Block.contained_blocks (cyclomatic 16) (marker/schema/blocks/base.py)
  • ConfigParser.generate_config_dict (cyclomatic 21) (marker/config/parser.py)
  • Duplicated block (11 lines × 2) (marker/schema/blocks/pagefooter.py)
  • Duplicated block (11 lines × 2) (marker/scripts/convert.py)
  • Duplicated block (13 lines × 2) (marker/processors/equation.py)
  • Duplicated block (18–21 lines × 2) (marker/processors/llm/llm_table.py)
  • Duplicated block (19 lines × 2) (marker/processors/llm/init.py)
  • Duplicated block (21–22 lines × 2) (marker/services/azure_openai.py)
  • Duplicated block (5 lines × 5) (marker/providers/document.py)
  • Duplicated block (6 lines × 2) (marker/ocr/indra.py)
  • Duplicated block (7 lines × 2) (marker/builders/ocr.py)
  • Edited copies of one member (3 members, 13+ corresponding lines) (marker/processors/llm/llm_complex.py)
  • Edited copies of one member (3 members, 20+ corresponding lines) (marker/processors/llm/llm_complex.py)
  • Edited copies of one member (5 members, 12+ corresponding lines) (marker/providers/document.py)
  • Edited copy of a member (14 corresponding lines) (marker/processors/llm/llm_handwriting.py)
  • Edited copy of a member (15 corresponding lines) (marker/converters/ocr.py)
  • Edited copy of a member (24 corresponding lines) (marker/processors/llm/init.py)
  • Edited copy of a member (28 corresponding lines) (marker/processors/debug.py)
  • …and 49 more

Changes since last survey

  • 32 commits — 25 feature/other, 7 fixes

By area

  • (repo) — 6 commits
  • marker/scripts — 6 commits
  • (root) — 3 commits
  • marker/services — 3 commits
  • signatures/version1 — 3 commits
  • data/images — 2 commits
  • marker/processors — 2 commits
  • .github/workflows — 1 commit
  • benchmarks/competitors — 1 commit
  • marker/builders — 1 commit
  • marker/converters — 1 commit
  • marker/renderers — 1 commit
  • marker/settings.py — 1 commit
  • tests/schema — 1 commit

Notable commits

  • fix: Add indra-ocr integration, 4 conversion modes, updated viewer/CLI, and accuracy fixes
  • fix: Fix BlockId dict key serialization in JSONRenderer and viewer
  • fix: Fix apt package name libgdk-pixbuf-2.0-0 in CI workflow
  • fix: Fix layout/table/caption detection, existing OCR handling, and HF model fallback
  • fix: fix : fix the order of default processors to fix heading_level in meta json (#1025)
  • fix: fix: only strip the leading bullet from a ListItem first line (#1056)
  • fix: fix: respect Gemini API RetryInfo delay on 429 responses (#1041)
  • change: @gkriegspeedbay has signed the CLA in datalab-to/marker#1141
  • change: @gkriegspeedbay has signed the CLA in datalab-to/marker#1141
  • change: @logan-markewich has signed the CLA in datalab-to/marker#1144
  • change: Add docling and liteparse OmniParseBench scores, charts, and positioning
  • change: Add prominent sticky progress banner, live timer, and in-flight page card to viewer
  • change: Benchmark liteparse with its markdown output
  • change: Bump surya-ocr to >=0.24.0,<0.25.0, drop torch/streamlit deps, and bump version to 2.2.1
  • change: Bump version to 2.2.0
  • change: Clarify batch conversion worker settings (#1116)
  • change: Decouple viewer upload lock from page inference and abort stale page requests
  • change: Derive FONT_PATH from FONT_DIR at runtime and cache the font outside the package (#1108)
  • change: Eliminate viewer spinner and progress banner DOM re-render flicker
  • change: Gracefully handle missing FastAPI in server_cli and silence non-runtime Streamlit warnings
  • …and 12 more

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

datalab-to/marker was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 15 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit d580b63874d9c8510fa54494975b9df1e8b527d0 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.73+preprod-52de90b74216.