Skip to content
CAI
Software that uses CAICheck a score

unclecode/crawl4ai

36.6

Weak · 14 November 2026

46.6k

lines of production code

Python

primary language

7

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

Added scripts for SBOM generation and stats dashboard updates

Added a new shell script (scripts/gen-sbom.sh) to generate a CycloneDX Software Bill of Materials (SBOM) using Syft, and a Python script (scripts/update\_stats.py) to fetch live data from GitHub, PyPI, and Docker Hub to generate a stats dashboard page (docs/md\_v2/stats.md) with embedded Chart.js charts.

scripts · high confidence

Crawl4AI v0.9.4 release with security hardening and new crawling capabilities

This release introduces Crawl4AI version 0.9.4, bringing significant security improvements alongside new features for adaptive and deep crawling. The library now includes a robust anti-bot detection system with multi-tier structural integrity checks to identify and handle blocked pages from providers like Cloudflare and Akamai. Security is further hardened through strict deserialization allowlists that prevent arbitrary code execution from untrusted inputs, and by removing dangerous eval() usage in computed fields. Users gain access to an Adaptive Crawler that intelligently determines when sufficient information has been gathered for a query, as well as new deep crawling strategies including Best-First and Depth-First Search (DFS). The release also adds an AsyncUrlSeeder for intelligent URL discovery, support for local file and raw HTML processing, and a new modular dispatcher system with memory-adaptive concurrency control.

Crawl4AI · high confidence

New Crawl4AI Playground UI for Docker deployments

A new interactive playground interface is now available at the /playground route within the Docker deployment. This single-page application provides a user-friendly environment for testing API requests, featuring a token-based authentication bar, tabbed navigation between the Playground and Stress Test modes, and a JSON configuration editor powered by CodeMirror. The UI includes syntax highlighting for responses and integrates with the application's monitoring dashboard.

deploy/docker/static/playground · high confidence

Removals

Removal of Python example script

The example script \examples/test.py\, which demonstrated how to initialize the WebCrawler and fetch pages using OpenAI providers, has been removed from the repository.

examples · high confidence

Removal of crawler module and associated components

The entire crawler package has been removed from the codebase. This deletion eliminates the web crawling functionality, including the \WebCrawler\ class that utilized Selenium for page fetching, the SQLite-based caching and database logic, the configuration for AI provider models, the Pydantic data models, the prompt templates for content extraction, and the utility functions for HTML processing and JSON parsing.

crawler · high confidence

Removal of legacy web-based crawling interfaces

The \index.html\ and \index\_pooling.html\ landing pages have been removed from the \pages\ directory. These files previously provided browser-based UIs for crawling websites, including input fields for URLs, API tokens, and model selection, as well as tabs for viewing results in JSON, Cleaned HTML, or Markdown. Their removal indicates that the web-based interactive crawling experience is no longer part of this component.

pages · high confidence

Security

Introduce egress policy to enforce network egress controls

The library now supports a process-wide egress policy that restricts outbound network requests to a configured proxy. This mechanism is used by the Docker server to enforce egress controls for internal HTTP clients, ensuring that requests made during crawling (such as fetching robots.txt or link previews) are routed through a pinned proxy that validates destinations. This change closes SSRF vulnerabilities by preventing untrusted requests from reaching internal or cloud-metadata endpoints.

(repo-wide) · high confidence

Behavioural changes

Secure-by-default Docker deployment with mandatory API authentication and hardened artifact storage

The Docker server now enforces a secure-by-default posture: it requires a \CRAWL4AI\_API\_TOKEN\ to bind to external interfaces (defaulting to loopback-only without one), implements JWT authentication for API access, and replaces the insecure \output\_path\ parameter for screenshots and PDFs with a server-managed, opaque-ID artifact store that prevents arbitrary file writes. Additionally, the deployment introduces a smart 3-tier browser pool for memory efficiency, declarative-only hook actions (replacing arbitrary code execution), and comprehensive security hardening including SSRF protection, Redis password requirements, and read-only root filesystem support.

deploy/docker · high confidence

Test coverage

Add Wikipedia sample HTML for async tests; Added Docker server behavioral security and performance test suites; Added adversarial and integration tests for DomainMapper; Added browser module test suite; Added cache validation test suite; Added cache validation test suite; Added demo script for browser pooling and page pre-warming; Added high-volume stress testing and benchmarking framework; Added integration tests for MCP server protocols; Added regression tests for release 0.6.4 and 0.7.0; Added test coverage for deep crawling stability, cancellation, and crash recovery; Added test coverage for proxy configuration, anti-bot detection, and sticky sessions; Added test coverage for webhook, CDP, logging, and anti-bot fixes; Added test suite for adaptive crawler performance and embedding strategies; Added tests for BrowserProfiler profile creation and keyboard input handling; Added tests for ContentTypeFilter PHP MIME type handling; Added tests for CrawlerHub integration examples; Added tests for Docker-based browser automation strategy; Added tests for the AI Assistant extract pipeline; Added tests for the new file logger implementation; Added unit tests for browser, security, and PDF processing changes; Comprehensive regression test suite for Crawl4AI; Expanded Docker integration test coverage; Expanded test coverage for crawler strategies, deep crawling, and markdown generation.

Dependencies

Modernize build system and pin dependencies for security and compatibility

The project has migrated from legacy setup files to a modern pyproject.toml configuration, establishing a strict Python 3.10+ requirement and defining core dependencies with specific version constraints. Key updates include pinning litellm to the secure unclecode-litellm fork (v1.81.13) to address a PyPI supply chain compromise, upgrading pyOpenSSL to \>=25.3.0 to fix a security vulnerability, and replacing chromedriver\_autoinstaller with playwright-stealth and patchright for browser management. The dependency list has also been expanded to include chardet, brotli, fake-useragent, and pdf2image, while optional dependencies for PDF, torch, and transformers are now explicitly categorized.

(dependencies) · high confidence

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 34 → 37 (+3.0)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

Lenses

  • Code Health 22 → 22 (-0.2)
  • Architecture 94 → 90 (-4.1)
  • Maturity 70 → 73 (+2.2)
  • Readiness 29 → 39 (+10.0)
  • Security 68 → 72 (+4.1)
  • Accessibility 42 → 45 (+3.0)
  • Performance 60 → 60 (+0.0)

Resolved (257)

  • 0.print_summary (cognitive 20) (docs/releases_review/demo_v0.8.0.py)
  • 5.print_summary (cognitive 20) (docs/releases_review/demo_v0.8.5.py)
  • 8.print_summary (cognitive 20) (docs/releases_review/demo_v0.7.8.py)
  • 8.test_adaptive_crawler_embedding (cognitive 22) (docs/releases_review/demo_v0.7.8.py)
  • 8.test_import_formatting (cognitive 19) (docs/releases_review/demo_v0.7.8.py)
  • AdminDashboard.getAppForm (cognitive 19) (docs/md_v2/marketplace/admin/admin.js)
  • AdminDashboard.getAppForm (cyclomatic 20) (docs/md_v2/marketplace/admin/admin.js)
  • ApiHandler.chatStream (cognitive 16) (docs/apps/linkdin/templates/ai.js)
  • AppDetailPage.renderAppDetails (cognitive 22) (docs/md_v2/marketplace/app-detail.js)
  • AppDetailPage.renderAppDetails (cyclomatic 20) (docs/md_v2/marketplace/app-detail.js)
  • BlocklyManager.parseC4AToBlocks (cognitive 19) (docs/md_v2/apps/c4a-script/assets/blockly-manager.js)
  • BlocklyManager.parseLineToBlock (cognitive 50) (docs/md_v2/apps/c4a-script/assets/blockly-manager.js)
  • BlocklyManager.parseLineToBlock (cyclomatic 33) (docs/md_v2/apps/c4a-script/assets/blockly-manager.js)
  • ClassTooLong: MarkdownConverter (docs/md_v2/apps/crawl4ai-assistant/content/markdownConverter.js)
  • ClassTooLong: ScriptBuilder (docs/md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js)
  • ContentAnalyzer.analyzeRelationships (cognitive 29) (docs/md_v2/apps/crawl4ai-assistant/content/contentAnalyzer.js)
  • ContentAnalyzer.extractMetadata (cognitive 27) (docs/md_v2/apps/crawl4ai-assistant/content/contentAnalyzer.js)
  • ContentAnalyzer.extractMetadata (cyclomatic 19) (docs/md_v2/apps/crawl4ai-assistant/content/contentAnalyzer.js)
  • ContentAnalyzer.identifyContentType (cognitive 19) (docs/md_v2/apps/crawl4ai-assistant/content/contentAnalyzer.js)
  • ContentAnalyzer.identifySemanticRegions (cognitive 24) (docs/md_v2/apps/crawl4ai-assistant/content/contentAnalyzer.js)
  • …and 237 more

New (46)

  • Critical CVE: [GHSA redacted] (sbom/sbom.cdx.json)
  • Critical CVE: [GHSA redacted] (sbom/sbom.cdx.json)
  • Critical CVE: [GHSA redacted] (sbom/sbom.cdx.json)
  • Critical CVE: [GHSA redacted] (sbom/sbom.cdx.json)
  • Critical CVE: [GHSA redacted] (sbom/sbom.cdx.json)
  • Critical CVE: [GHSA redacted] (sbom/sbom.cdx.json)
  • Edited copies of one member (3 members, 11+ corresponding lines) (crawl4ai/legacy/database.py)
  • Edited copy of a member (10 corresponding lines) (crawl4ai/async_configs.py)
  • Edited copy of a member (11 corresponding lines) (crawl4ai/async_database.py)
  • Edited copy of a member (11 corresponding lines) (deploy/docker/monitor_routes.py)
  • Edited copy of a member (127 corresponding lines) (crawl4ai/extraction_strategy.py)
  • Edited copy of a member (13 corresponding lines) (crawl4ai/migrations.py)
  • Edited copy of a member (15 corresponding lines) (crawl4ai/content_filter_strategy.py)
  • Edited copy of a member (19 corresponding lines) (deploy/docker/job.py)
  • Edited copy of a member (27 corresponding lines) (crawl4ai/utils.py)
  • Edited copy of a member (29 corresponding lines) (deploy/docker/server.py)
  • Edited copy of a member (32 corresponding lines) (crawl4ai/browser_adapter.py)
  • Edited copy of a member (35 corresponding lines) (crawl4ai/content_filter_strategy.py)
  • Edited copy of a member (4 corresponding lines) (crawl4ai/domain_mapper.py)
  • Edited copy of a member (41 corresponding lines) (crawl4ai/browser_profiler.py)
  • …and 26 more

Changes since last survey

  • 9 commits — 9 feature/other, 0 fixes

By area

  • deploy/docker — 3 commits
  • .github/workflows — 2 commits
  • docs/md_v2 — 2 commits
  • (root) — 1 commit
  • docs/assets — 1 commit

Notable commits

  • change: Docker Hub call-to-action image: space above the box
  • change: Docker Hub page: current README, synced from main (#2499)
  • change: Docker Hub page: lead with Crawl4AI Cloud, self-hosted server second (#2500)
  • change: Docker Hub page: the Cloud call to action as a shaded box
  • change: Docker Hub page: the Cloud call to action in a bordered box
  • change: Docker Hub page: the Cloud call to action in a bordered box
  • change: README: lead with Crawl4AI Cloud, keep self-hosting below (#2498)
  • change: docs: fix 33 broken links across the docs site
  • change: docs: fix the remaining broken links in release notes and table docs

Architecture

  • Unchanged — 0 containers · 1 contexts · 0 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

unclecode/crawl4ai was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit cae74b203a33994f7e13b9052c3b0d5c93daa0e6 — the exact code this score is about.
  • Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-36d4cd83012f.