ScrapeGraphAI/Scrapegraph-ai
57.7
Adequate · 15 November 2026
16.4k
lines of production code
Python
primary language
5
measurements over time
What this system is
ScrapeGraphAI is a Python library that constructs web scraping pipelines as directed graphs, allowing users to define complex data extraction workflows using natural language prompts. It integrates with a wide range of LLM providers to parse, reason over, and transform web content into structured formats like JSON, CSV, or code. The system supports diverse input sources, including static HTML, dynamic JavaScript-rendered pages, and images via OCR, while offering utilities for cost tracking, error handling, and workflow orchestration.
Features
Added model-specific token counting utilities for Mistral, Ollama, and OpenAI
New tokenization modules have been added to the utils/tokenizers directory to provide accurate token counts for specific LLM providers. The Mistral tokenizer now uses the \mistral\_common\ library to encode chat completions, the Ollama tokenizer delegates to the model's built-in \get\_num\_tokens\ method, and the OpenAI tokenizer utilizes \tiktoken\ with the \gpt-4o\ encoding. These changes ensure that token usage estimates are correctly calculated for each supported model family.
scrapegraphai/utils/tokenizers · high confidence
Added telemetry module with environment-variable-based opt-out
A new telemetry module has been introduced to scrapegraphai, enabling the collection of usage data such as graph execution details, prompts, and model information sent to a remote tracing endpoint. Users can now explicitly disable this data collection by setting the SCRAPEGRAPHAI\_TELEMETRY\_ENABLED environment variable, which takes precedence over the local configuration file, ensuring immediate control over privacy settings without requiring a config file change.
scrapegraphai/telemetry · high confidence
Initialize scrapegraphai package with logging setup
The scrapegraphai package now includes an \_\init\\_.py file that initializes the module by importing and configuring logging utilities. Specifically, it exposes the get\_logger and set\_verbosity\_info functions from the utils.logging module and sets the initial logging verbosity to info, ensuring consistent logging behavior for users of the package.
scrapegraphai · high confidence
Introduction of new graph nodes and batch processing capabilities
This release introduces a comprehensive set of new nodes for the ScrapeGraphAI workflow, including FetchNodeLevelK for recursive link traversal, FetchScreenNode for screenshot capture, and BatchGenerateAnswerNode which leverages the OpenAI Batch API to process multiple documents in a single request for significant cost savings. The update also adds specialized nodes for handling CSV data (GenerateAnswerCSVNode), image analysis (GenerateAnswerFromImageNode), and deep content refinement (DescriptionNode, PromptRefinerNode, ReasoningNode). Additionally, the ConditionalNode now supports custom boolean expressions for dynamic graph branching, and the base node infrastructure has been standardized with centralized logging and improved input key validation.
scrapegraphai/nodes · high confidence
New GraphBuilder for natural language scraping graph generation
A new GraphBuilder class has been added to the builders module, allowing users to generate web scraping graph configurations from natural language prompts. The builder uses an LLM (supporting OpenAI, Gemini, and Ernie models) to parse the user's intent and output a structured graph definition, which can then be visualized using Graphviz.
scrapegraphai/builders · high confidence
New document loaders and integration modules added
The docloaders package now includes new modules for integrating with BrowserBase, Scrape.do, and Plasmate, alongside an enhanced Chromium loader. Users can now fetch content via the BrowserBase API (with synchronous or asynchronous modes), use the Scrape.do service for proxy-based fetching, and leverage the lightweight PlasmateLoader for static pages (with an optional fallback to Chromium for JavaScript-heavy sites). The ChromiumLoader has been updated to support multiple backends (Playwright and Selenium/undetected-chromedriver), configurable browser names (Chromium/Firefox), proxy settings, retry limits, timeouts, and storage state handling, while also surfacing HTTP errors more clearly.
scrapegraphai/docloaders · high confidence
New graph-based scraping examples and cookbook
Added a comprehensive set of new examples demonstrating the library's graph-based scraping capabilities, including SmartScraper, CodeGenerator, CSVScraper, CustomGraph, DepthSearch, and DocumentScraper. Each example provides both OpenAI and Ollama configurations, along with setup instructions and environment variable templates. A new Jupyter notebook cookbook was also added to guide users through using the SmartScraperGraph with OpenAI models.
examples · high confidence
New helper module for model tokens, node metadata, and graph schemas
The \scrapegraphai/helpers\ package has been introduced to centralize configuration data and utilities. It provides a comprehensive \models\_tokens\ dictionary mapping providers (OpenAI, Azure, Google, Ollama, etc.) to their specific model names and context window limits, ensuring accurate token handling. Additionally, it exposes \nodes\_metadata\ describing the arguments and return values for graph nodes, a \robots\_dictionary\ for AI agent user-agents, a \default\_filters\ configuration for link filtering, and a \graph\_schema\ JSON schema for validating graph structures.
scrapegraphai/helpers · high confidence
New integrations for Burr workflow orchestration and scrapegraph-py SDK compatibility
This update introduces new integration modules within the \scrapegraphai/integrations\ package. It adds a \BurrBridge\ class that allows ScrapeGraphAI graphs to be converted into Burr applications, enabling workflow tracking, forking, and spawning capabilities via Burr's class-based API. Additionally, a compatibility layer (\scrapegraph\_py\_compat.py\) is added to support both the v2 \Client\ API and the newer v3 \ScrapeGraphAI\ API from the \scrapegraph-py\ SDK, ensuring seamless interaction with the external scraping service regardless of the installed SDK version. An \IndexifyNode\ is also included for content indexing within the graph state.
scrapegraphai/integrations · high confidence
New model integrations and multimodal capabilities
The models module now supports several new LLM providers and multimodal features. Users can connect to Atlas Cloud, Cheaper Inference, CLōD, DeepSeek, MiniMax, OneAPI, and xAI (Grok) via OpenAI-compatible wrappers. NVIDIA LLM support is added with lazy loading of the langchain\_nvidia\_ai\_endpoints dependency. Additionally, new classes enable image-to-text conversion (OpenAIImageToText) and text-to-speech generation (OpenAITextToSpeech).
scrapegraphai/models · high confidence
New screenshot scraping and OCR utilities
Added a new \screenshot\_scraping\ module providing functions to capture webpage screenshots (\take\_screenshot\), manually select image regions via OpenCV or ipywidgets (\select\_area\_with\_opencv\, \select\_area\_with\_ipywidget\), and extract text from images using Surya OCR (\detect\_text\). These utilities are optional dependencies, requiring \pip install scrapegraphai\[ocr\]\ to function, and handle their own import errors gracefully.
_scrapegraphai/utils/screenshot\scraping · high confidence
New utility modules for code error handling, HTML cleanup, and LLM cost tracking
The \scrapegraphai/utils\ package now includes new modules for analyzing and correcting code errors (syntax, execution, validation, and semantic), cleaning and minifying HTML content, and tracking LLM token usage and costs. These utilities support the graph's ability to self-heal generated code and optimize resource usage.
scrapegraphai/utils · high confidence
Repository initialization and development infrastructure setup
The repository has been initialized with a complete development environment, including a Python 3.11 Dockerfile, a Makefile for automation (linting, type-checking, testing), and a semantic-release configuration (\.releaserc.yml\) to manage versioning. The project now enforces code quality via pre-commit hooks (Black, Ruff, isort) and utilizes \uv\ for dependency management. Additionally, a comprehensive testing infrastructure has been added, featuring a mock HTTP server, performance benchmarking tools, and a GitHub Actions workflow for CI/CD.
(repo-wide) · high confidence
Behavioural changes
Centralized prompt template management
The \scrapegraphai/prompts\ package has been restructured to centralize all LLM prompt templates. A new \\_\init\\_.py\ file now exports a comprehensive set of template constants (e.g., \TEMPLATE\_CHUNKS\, \TEMPLATE\_REASONING\, \TEMPLATE\_ROBOT\) from dedicated modules for specific graph nodes such as answer generation, code generation, search, and reasoning. This change organizes the prompt definitions into a single, importable namespace, making it easier to manage and access the instructions used by the scraping pipeline.
scrapegraphai/prompts · high confidence
ScrapeGraphAI v2 API surface and graph architecture overhaul
The \scrapegraphai/graphs\ module has been completely rewritten to align with the scrapegraph-py v2 API. This change introduces a new \AbstractGraph\ and \BaseGraph\ foundation that standardizes how scraping pipelines are constructed and executed. Users will now interact with a unified set of graph classes (such as \SmartScraperGraph\, \CodeGeneratorGraph\, and \DocumentScraperGraph\) that support Pydantic output schemas, configurable LLM providers via a standardized config dict, and new capabilities like multi-URL iteration (\GraphIteratorNode\) and Burr integration. The previous graph implementations have been replaced by this new modular structure, which centralizes common parameters (like \verbose\, \headless\, and \loader\_kwargs\) and enforces a consistent execution flow across all scraper types.
scrapegraphai/graphs · high confidence
Test coverage
Added comprehensive test suite for graph components and LLM integrations; Added integration tests and test fixtures for ScrapeGraphAI; Added sample input files for graph tests; Added sample input files for node tests; Added sample test input files; Added test coverage for utility modules; Added unit tests for Fetch, Robots, SearchInternet, and SearchLink nodes; Comprehensive test infrastructure and coverage expansion.
Dependencies
Initial release of ScrapegraphAI v2.3.0 with uv build system
The project is now packaged using the uv build system (hatchling) and targets Python 3.12+. This release introduces a comprehensive dependency set including LangChain v1.2.0+, scrapegraph-py v2.0.0, and playwright v1.57.0, while making pandas optional and adding optional integrations for Burr, NVIDIA, and OCR capabilities.
(dependencies) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 57 → 58 (+1.1)
- Rubric changed (rubric-2026.11.1 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 55 → 53 (-1.5)
- Architecture 87 → 99 (+11.5)
- Maturity 55 → 59 (+3.8)
- Readiness 54 → 53 (-1.3)
- Security 56 → 73 (+16.9)
- Performance 85 → 85 (+0.0)
Resolved (9)
- Duplicated block (10 lines × 2) (scrapegraphai/utils/screenshot_scraping/screenshot_preparation.py)
- Duplicated block (21 lines × 2) (examples/extras/chromium_selenium.py)
- Medium CVE: PYSEC-2026-139 (uv.lock)
- Off-boarding risk: anonymized user #1
- Projects may be oversized for their cohesion
- Rotate the exposed credentials — git history can't be un-committed
- Secret: generic-api-key (examples/benchmarks/GenerateScraper/inputs/example_2.txt)
- Secret: generic-api-key (examples/benchmarks/SmartScraper/inputs/example_2.txt)
- Secret: generic-api-key (examples/benchmarks/inputs/example_2.txt)
New (19)
- Edited copies of one member (4 members, 27+ corresponding lines) (scrapegraphai/nodes/generate_code_node.py)
- Edited copies of one member (4 members, 28+ corresponding lines) (scrapegraphai/graphs/csv_scraper_graph.py)
- Edited copies of one member (4 members, 44+ corresponding lines) (scrapegraphai/graphs/document_scraper_graph.py)
- Edited copies of one member (4 members, 7+ corresponding lines) (scrapegraphai/nodes/generate_answer_csv_node.py)
- Edited copies of one member (6 members, 50+ corresponding lines) (scrapegraphai/utils/code_error_analysis.py)
- Edited copies of one member (8 members, 3+ corresponding lines) (scrapegraphai/graphs/csv_scraper_multi_graph.py)
- Edited copy of a member (13 corresponding lines) (scrapegraphai/integrations/scrapegraph_py_compat.py)
- Edited copy of a member (15 corresponding lines) (scrapegraphai/nodes/generate_answer_node.py)
- Edited copy of a member (19 corresponding lines) (scrapegraphai/docloaders/chromium.py)
- Edited copy of a member (19 corresponding lines) (scrapegraphai/docloaders/plasmate.py)
- Edited copy of a member (19 corresponding lines) (scrapegraphai/utils/tokenizers/tokenizer_ollama.py)
- Edited copy of a member (33 corresponding lines) (scrapegraphai/graphs/csv_scraper_multi_graph.py)
- Edited copy of a member (42 corresponding lines) (scrapegraphai/nodes/get_probable_tags_node.py)
- Edited copy of a member (46 corresponding lines) (scrapegraphai/nodes/prompt_refiner_node.py)
- Edited copy of a member (5 corresponding lines) (scrapegraphai/graphs/omni_scraper_graph.py)
- Edited copy of a member (5 corresponding lines) (scrapegraphai/nodes/description_node.py)
- Edited copy of a member (54 corresponding lines) (scrapegraphai/docloaders/chromium.py)
- Edited copy of a member (7 corresponding lines) (scrapegraphai/graphs/omni_search_graph.py)
- High CVE: [GHSA redacted] (uv.lock)
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
ScrapeGraphAI/Scrapegraph-ai was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 15 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit 194055e203afce41ed4e70365dbc416bad756115 — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.73+preprod-52de90b74216.