Skip to content
CAI
Software that uses CAICheck a score

D4Vinci/Scrapling

61.8

Adequate · 14 November 2026

14.8k

lines of production code

Python

primary language

7

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Add Scrapy integration for Scrapling responses

Users can now integrate Scrapling with Scrapy projects by decorating spider callbacks with \scrapling\_response\ or using \convert\_response\ directly. This allows existing Scrapy spiders to receive Scrapling Response objects instead of standard Scrapy responses, enabling the use of Scrapling's parsing API without altering the crawl structure. The integration is optional and requires Scrapy to be installed separately.

scrapling/integrations · high confidence

Add sponsor advertisement section to the website

The website now includes a dedicated section for displaying sponsor advertisements. This change introduces a new HTML template (\includes/sponsors.html\) that renders a grid of banner images for various proxy and web scraping service providers, linking to their respective sites. Also in this area: 1 commit (0 fixes) modifying includes.

includes · low confidence · unverified

Added Scrapling agent skill examples

The agent-skill/Scrapling-Skill/examples directory now includes a complete set of usage examples for the Scrapling library, accompanied by a README. These examples demonstrate four distinct scraping approaches: FetcherSession for fast, persistent HTTP requests; DynamicSession for JavaScript-heavy pages using Playwright; StealthySession for bypassing anti-bot protections like Cloudflare; and Spider for automated multi-page crawling with structured data export. The examples target quotes.toscrape.com and serve as a practical guide for users to integrate Scrapling into their workflows.

agent-skill/Scrapling-Skill/examples · high confidence

Introduce Scrapling Shell and TextHandler for interactive scraping

This change introduces the Scrapling Shell, an interactive CLI environment that allows users to inspect and manipulate scraped data directly. It includes a new TextHandler class that extends Python's standard string to provide convenient methods for cleaning, regex extraction, and JSON parsing while preserving the handler type through operations. The shell also supports parsing curl commands for quick testing and includes utilities for generating CSS and XPath selectors from HTML elements.

scrapling/core · high confidence

Introduce new Spider framework with pause/resume, auto-throttling, and export capabilities

The \scrapling/spiders\ package introduces a new crawling system that supports pausing and resuming crawls via disk checkpoints, automatically throttling request rates based on server response times, and exporting collected items to JSON, CSV, or XML formats. The engine now respects \robots.txt\ directives (with configurable compliance), caches HTTP responses for development replay, and includes specialized spider templates for sitemaps, Shopify, and feed parsing.

scrapling/spiders · high confidence

Introduction of the Scrapling engines module with static HTTP fetching capabilities

This change introduces the new \scrapling/engines\ package, establishing the foundational structure for the library's fetching logic. It adds a \static.py\ module implementing a \\_SyncSessionLogic\ class that leverages \curl\cffi\ for HTTP requests, supporting features like browser impersonation, stealth headers, proxy rotation, and HTTP/3. The module includes \constants.py\ with optimized Chromium launch arguments for stealth and performance, and an empty \\\init\\_.py\ to define the package namespace. This represents the initial implementation of the static engine component within the broader engine architecture.

scrapling/engines · high confidence

New AI-driven browser automation tools and session management

The \scrapling/core/ai\ module now exposes a comprehensive set of tools for AI agents to control browsers via the Model Context Protocol (MCP). This includes a new \BrowserAction\ type that supports a wide range of interactions such as mouse movements, clicks, scrolling, keyboard input, and form filling (text, checkboxes, radios, and comboboxes). A new \dialog\ action allows agents to handle browser alerts and prompts. The module also introduces session management tools to create, inspect, and close browser sessions (stealthy or static), along with tools to set and load browser state. Additionally, snapshot searching capabilities have been added to help agents locate specific content within browser snapshots, and network request details are now formatted for easier AI consumption. Also in this area: 6 commits (3 fixes) modifying scrapling.

scrapling/core/ai · medium confidence · unverified

New browser-based fetcher engine with stealth and session management

The browser fetching logic has been replaced with a new engine built on Patchright (a fork of Playwright) that introduces persistent browser sessions, page pooling, and advanced stealth capabilities. Users can now enable Cloudflare Turnstile and interstitial solving, block specific domains or ads, and record XHR/network requests for inspection. The engine supports persistent contexts to reuse browser state across requests, allows custom init scripts and extra browser flags, and provides tools to capture AI-ready ARIA snapshots of pages.

_scrapling/engines/\browsers · high confidence

New spider templates for feeds, sitemaps, and content extraction

The \scrapling/spiders/templates\ package now provides several new spider classes to simplify common scraping tasks. \XMLFeedSpider\ and \CSVFeedSpider\ allow iterating over XML and CSV data feeds, automatically handling gzip decompression and BOM removal. \SitemapSpider\ seeds crawls from sitemap files or robots.txt, supporting recursive sitemap index traversal and configurable link extraction rules. \CrawlSpider\ offers a generic link-following engine with BFS/DFS strategies and customizable \CrawlRule\s. \ShopifySpider\ extracts product data from Shopify stores via their JSON API, and \ContentSpider\ (aliased as \SiteToMarkdownSpider\) crawls sites to convert pages into clean Markdown for LLM/RAG pipelines, with optional file output.

scrapling/spiders/templates · high confidence

New toolbelt utilities for proxy rotation, ad blocking, and response handling

The \scrapling/engines/toolbelt\ package now provides core utilities for the library's fetching engine. Users can leverage the new \ProxyRotator\ class for thread-safe, cyclic proxy rotation across fetchers. The library introduces built-in ad and tracker blocking capabilities via the \ad\_domains.py\ list and navigation interceptors, allowing requests to specific domains or resource types to be dropped. Response handling is standardized through a new \Response\ class and \ResponseFactory\, which unify data from Playwright and other sources, support XHR capture, and expose a \follow()\ helper for spider workflows.

scrapling/engines/toolbelt · high confidence

Scrapling 0.4.15: Lazy imports, CLI shell, and MCP server with auth

Scrapling 0.4.15 introduces lazy loading for top-level imports (Fetcher, Selector, etc.) to reduce startup memory, adds a \scrapling install\ CLI command to manage Playwright dependencies, and includes a new \scrapling mcp\ command that runs an MCP server with optional HTTP transport, bearer-token authentication, and DNS-rebinding protection.

scrapling · high confidence

Behavioural changes

Fetcher API restructured with new configuration merging and lazy imports

The fetcher module has been restructured to improve configuration handling and import performance. The \Fetcher\ and \AsyncFetcher\ classes now automatically merge class-level parser arguments (set via \configure\) with per-request \selector\config\ arguments, ensuring that base settings are applied to all requests unless explicitly overridden. Additionally, the \\\init\\_.py\ file now uses lazy imports for all fetcher classes and sessions, which reduces initial import overhead. The \ProxyRotator\ class is also explicitly exposed in the public API.

scrapling/fetchers · high confidence

Introduce flexible, context-aware logging and new utility modules

The \scrapling/core/utils\ package has been restructured to support more flexible logging and provide new helper utilities. Logging is now context-aware via \ContextVar\, allowing spider classes to temporarily override the logger using \set\_logger\ and \reset\_logger\ without affecting other concurrent contexts, while retaining a default INFO-level console output. Additionally, new modules \\_shell.py\ and \\_utils.py\ introduce utilities for parsing HTTP headers and cookies, flattening iterables, cleaning whitespace, and converting HTML elements to dictionaries, supporting the core scraping logic with improved memory efficiency and modularity.

scrapling/core/utils · high confidence

Test coverage

24 commits adding/updating tests in tests/cli; Added async fetcher test suite; Added comprehensive test coverage for the AI MCP server and browser automation tools; Added comprehensive test suite for sync fetchers and sessions; Added integration tests for Scrapy compatibility; Added test coverage for core shell and storage modules; Added test package initialization; Comprehensive test suite for the spiders subsystem; Expanded test coverage for fetcher components; Expanded test coverage for parser and selector features.

Dependencies

Project migration to pyproject.toml and dependency restructuring

The project has migrated its build configuration from legacy setup files to a modern pyproject.toml format, establishing version 0.4.15 and raising the minimum Python requirement to 3.10. Core dependencies have been updated, notably replacing tldextract with the tld library. The dependency structure has been reorganized into optional extras to reduce bloat: fetchers (including playwright, patchright, and curl\_cffi), rag, ai (adding MCP v2 support), and shell. Additionally, dedicated requirements files have been introduced for documentation (docs/requirements.txt) and testing (tests/requirements.txt) to isolate tooling dependencies.

(dependencies) · high confidence

Housekeeping

Initial project scaffolding and development tooling

The repository is initialized with essential configuration files for the Scrapling project. This includes a Dockerfile for containerized deployment, a .dockerignore file, and a MANIFEST.in for package distribution. Development workflows are established via a .pre-commit-config.yaml integrating Ruff and Bandit, alongside ruff.toml and .bandit.yml for linting and security checks. Documentation is configured for ReadTheDocs using Zensical, and the project structure includes a CHANGELOG, AI\_POLICY, CODE\_OF\_CONDUCT, and CONTRIBUTING guides to standardize contributions.

(repo-wide) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

This is the PUBLIC form of this artifact. Findings are listed in full, but the details of SECURITY findings — which rule fired, in which file, on which line, and how to fix it — are deliberately withheld, and any secret-scanner results are excluded entirely. Where detail is absent here it was REMOVED FOR PUBLICATION; it is not missing from the analysis. The complete artifact is available from the repository owner.

Score

  • CAI 64 → 62 (-2.4)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 63 → 62 (-1.3)
  • Architecture 86 → 87 (+1.8)
  • Maturity 75 → 75 (+0.1)
  • Readiness 57 → 57 (+0.0)
  • Security 71 → 71 (+0.0)
  • Accessibility 61 (new)
  • Performance 100 → 100 (+0.0)

Resolved (13)

  • BaseSessionMixin.__generate_options__ (cognitive 32) (scrapling/engines/_browsers/_base.py)
  • BaseSessionMixin.__generate_options__ (cyclomatic 20) (scrapling/engines/_browsers/_base.py)
  • Documentation: contradicts the code (docs/requirements.txt)
  • Documentation: no project overview (README.md)
  • Duplicate configuration types. PlaywrightConfig and _fetch_params contain nearly identical fields (e.g., timeout, wait, proxy, headers), as do StealthConfig and StealthFetchParams. This duplication increases maintenance burden and risk of drift.
  • Duplicated block (39–40 lines × 2) (scrapling/engines/_browsers/_stealth.py)
  • Hotspot: scrapling/engines/_browsers/_base.py (scrapling/engines/_browsers/_base.py)
  • Hotspot: scrapling/parser.py (scrapling/parser.py)
  • Inconsistent naming for state persistence. The session objects use save_state/load_state, while the MCP server uses browser_save_state/browser_load_state. This inconsistency extends to other methods like close vs close_session.
  • Near-duplicate member pair (69 shared lines) (scrapling/engines/_browsers/_stealth.py)
  • No ADRs found
  • Overlapping functionality between browser fetch methods. browser_fetch and browser_fetch_once appear to perform similar HTTP-like extraction tasks within a browser context, while session_make_request handles HTTP requests within a session. The distinction between 'fetch' and 'make_request' in the context of a browser session is unclear and likely redundant.
  • Redundant HTTP verb methods vs generic fetch. The CLI exposes specific methods for GET, POST, PUT, and DELETE, but also exposes a generic fetch method. This creates confusion about which method to use for standard HTTP requests versus complex browser-based fetching.

New (24)

  • Duplicated block (41–42 lines × 2) (scrapling/engines/_browsers/_stealth.py)
  • Edited copy of a member (115 corresponding lines) (scrapling/engines/_browsers/_stealth.py)
  • Edited copy of a member (14 corresponding lines) (scrapling/engines/_browsers/_stealth.py)
  • Edited copy of a member (14 corresponding lines) (scrapling/spiders/spider.py)
  • Edited copy of a member (16 corresponding lines) (scrapling/engines/static.py)
  • Edited copy of a member (22 corresponding lines) (scrapling/core/ai/server.py)
  • Edited copy of a member (22 corresponding lines) (scrapling/engines/_browsers/_base.py)
  • Edited copy of a member (24 corresponding lines) (scrapling/engines/toolbelt/navigation.py)
  • Edited copy of a member (30 corresponding lines) (scrapling/engines/_browsers/_base.py)
  • Edited copy of a member (38 corresponding lines) (scrapling/engines/toolbelt/convertor.py)
  • Edited copy of a member (49 corresponding lines) (scrapling/fetchers/stealth_chrome.py)
  • Edited copy of a member (7 corresponding lines) (scrapling/parser.py)
  • Edited copy of a member (8 corresponding lines) (scrapling/engines/_browsers/_remote.py)
  • Edited copy of a member (8 corresponding lines) (scrapling/spiders/templates/crawler.py)
  • Edited copy of a member (89 corresponding lines) (scrapling/engines/toolbelt/convertor.py)
  • Edited copy of a member (9 corresponding lines) (scrapling/engines/_browsers/_base.py)
  • High: security finding (details withheld)
  • Hotspot: scrapling/core/ai/_browser_actions.py (scrapling/core/ai/_browser_actions.py)
  • Near-duplicate member pair (71 shared lines) (scrapling/engines/_browsers/_stealth.py)
  • No ADRs found (pyproject.toml)
  • …and 4 more

Changes since last survey

  • 44 commits — 31 feature/other, 13 fixes

By area

  • docs/locales — 8 commits
  • (root) — 7 commits
  • scrapling/core — 5 commits
  • (repo) — 4 commits
  • scrapling/parser.py — 3 commits
  • scrapling/spiders — 3 commits
  • tests/ai — 3 commits
  • agent-skill/Scrapling-Skill — 2 commits
  • scrapling/engines — 2 commits
  • tests/fetchers — 2 commits
  • tests/spiders — 2 commits
  • .github/workflows — 1 commit
  • images/olostep.png — 1 commit
  • scrapling/cli.py — 1 commit

Notable commits

  • fix: fix(cli): make --ai-targeted enable ad blocking on fetch (#527)
  • fix: fix(fetchers): decode percent-encoded credentials in proxy URLs (#545)
  • fix: fix(parser): handle empty HTML roots (#536)
  • fix: fix(parser): honor zero similarity threshold without reference attributes (#539)
  • fix: fix(parser): return the value of XPath expressions that do not select nodes (#552)
  • fix: fix(shell): accept curl's -L flag (#554)
  • fix: fix(shell): stop dropping cookies whose values SimpleCookie rejects (#541)
  • fix: fix(spiders): fetch robots.txt once per domain under concurrent requests (#538)
  • fix: fix(spiders): handle CR record separators in CSV feeds (#551)
  • fix: fix(spiders): isolate temporary files for concurrent cache writes (#546)
  • fix: fix(spiders): skip malformed links during extraction (#548)
  • fix: test: fix flaky Cloudflare readiness test (#515)
  • fix: test: fix repeated issue on GitHub Actions
  • change: Merge branch 'main' into dev
  • change: Merge branch 'main' into dev
  • change: Merge branch 'main' into dev
  • change: agent: update skill with the current changes
  • change: ci: add a new workflow to handle stale issues/PRs
  • change: docs: Update changelog with the latest changes
  • change: docs: add a new sponsor
  • …and 24 more

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

D4Vinci/Scrapling was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit bb223f2f5fc8f743772751a5ce283ef7851ca6be — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.