blogdaren/PHPCreeper

This system is a PHP-based web crawling framework that handles the full lifecycle of data extraction, including headless browser rendering for dynamic content. It provides a modular architecture for managing tasks, scheduling, and distributed locking, while offering middleware for HTTP requests, database access, and message queues. The system is designed to scale with features like Bloom filter deduplication and Redis-backed task queues.

The framework now supports scheduled tasks via a new Crontab library, enabling time-based execution of crawlers. It also introduces a headless browser middleware (Chrome) to handle dynamic, JavaScript-rendered pages. For duplicate detection, the system now supports configurable Bloom filters (Redis and Predis) that automatically compute optimal bit size and hash counts based on expected insertions and false positive rates. Additionally, the Uuid generation has been refactored to use an internal polyfill, removing the external Ramsey/UUID dependency.

src/Kernel · high confidence

A new database middleware component has been introduced in the kernel, providing a Medoo-style database wrapper. This adds a singleton-based database access layer that supports configuration via global config and allows for database logging and debugging.

src/Kernel/Middleware/Database · high confidence

Users can now execute JavaScript on dynamic pages via the new HeadlessBrowserServiceProvider, which registers a factory for headless browser instances. Additionally, the framework introduces a Medoo-style database middleware, exposing getDbo and newDbo service bindings for database access, alongside createTask and createMultiTask for task creation. The language service provider was also updated to store language configuration in a static backup property.

src/Kernel/Service/Provider · high confidence

A new \Examples/start.php\ script is added, providing a standalone, humanized example for bootstrapping the PHPCreeper framework. It includes comprehensive configuration options for Redis, task contexts, and headless browser settings, allowing users to run the crawler without the full application framework. This serves as a simplified entry point for testing and development.

Examples · high confidence

The PHPCreeper framework now includes a new Server module (src/Server.php) that provides a complete event-driven architecture with user callbacks for server lifecycle events (start, stop, reload, connect, close, message, buffer full/drain, error). The framework also introduces a shorter namespace (PHPCreeper) for core classes like Cookie, Crontab, Timer, and Tool, while removing the legacy Autoloader.php. In the Downloader, the onWorkerStop method now explicitly destroys the headless browser to prevent Chrome processes from lingering, and adds new callbacks onDownloaderConnectToParser and onTaskEmpty. The Parser class now triggers onParserConnect and onParserFindUrl callbacks, and the task depth check logic is inverted (returning true when depth is exceeded). The Producer class refactors task initialization to use a new task\_init configuration key and replaces usleep with a 1-second sleep to prevent URL duplication.

src · high confidence

The HTTP client middleware now supports setting cookies via a dedicated setCookies method, handles multipart/form-data requests by converting file contents to streams, and provides explicit methods to disable SSL verification, enable it, or set a custom CA certificate file. Additionally, it includes logic to track request arguments for logging and normalizes header inputs to support both string and array formats.

src/Kernel/Middleware/HttpClient · high confidence

The Tool helper's URL validation has been expanded to support domains, IPv4, and IPv6 addresses, each with optional port numbers, replacing the previous basic filter\_var check. Additionally, several input validation methods (checkEmail, checkMobile, checkTelphone, checkIsInt, checkIsIntOrZero, checkIsFloat, checkIsIntFloat, checkIsIntFloatZero) now explicitly return false for empty or null inputs, and directory creation is wrapped in a try-catch block to handle errors gracefully.

src/Kernel/Library/Helper · high confidence

The English and Chinese language packs for the Kernel/Language module have been updated to support new logging and tracing capabilities. Log prefixes for the Producer, Downloader, and Parser now include the process ID (PID) for better process identification. New log entries have been added for headless browser status (enabled, disabled, binary not found, proxy server, and credentials), request argument tracking, task package tracking, and Redis server errors. Additionally, the system now supports a new \msgpack\ assemble package method, with appropriate warnings if the extension is missing. The \queue\_full\ message has been updated to reflect the total backlog of tasks, and the \downloader\_cache\_disabled\/\enabled\ messages now include the task ID.

src/Kernel/Language · high confidence

Users can now use headless browsers (e.g., Chrome) to crawl dynamic pages via the new HeadlessBrowserFactoryService. The ExtractorService has been updated to support 'preg' and 'pregs' regular expression selectors, independent CSS actions, and a new 4th argument for rule callbacks. Additionally, the default implementations for the Redis client, lock manager, and message queue have been switched from the 'redis' extension to 'predis' (and 'predis' for locks), and the extractor's setRange() method has been fixed to prevent unexpected document object changes.

src/Kernel/Service/Wrapper · high confidence

The bundled phpQuery library has been updated for PHP 8+ compatibility by replacing the deprecated \create\_function()\ with anonymous functions. The default document charset has been changed from ISO-8859-1 to UTF-8. Additionally, the library now supports the \:nth-of-type\ pseudo-class selector and the \:parent\ CSS action for extraction, while also including various bug fixes and optimizations.

src/Kernel/Library/phpQuery · medium confidence

The default Redis client for the message queue middleware has been changed from the native PHP extension to the Predis library, introducing a new PredisClient implementation. This change includes refactoring the Redis and AMQP extensions to support multiple server connections, partitioning, and heartbeat mechanisms. Additionally, the PhpQueue class was modified to use an internal queue object instead of extending the native &\#x2019;&\#x2019;SplQueue&\#x2019;&\#x2019; class, and all affected classes now use the \#\[&\#x2019;AllowDynamicProperties\] attribute to resolve PHP 8.2 deprecation warnings regarding dynamic properties.

src/Kernel/Middleware/MessageQueue · medium confidence

The LockManager middleware now supports a more secure distributed RedLock mechanism. For both Redis and Predis clients, the system can automatically detect when to use RedLock based on configuration, falling back to the previous locking behavior if RedLock is not applicable. This upgrade improves security and reliability for distributed locking scenarios.

src/Kernel/Middleware/LockManager · high confidence

The project's Composer dependencies have been updated to support PHP 8+ and add headless browser capabilities. The Workerman package is now constrained to versions 3.5.0 through 4.x (less than 5.0.0). Guzzle is updated to support version 7.x alongside 6.4. A new dependency on chrome-php/chrome ^1.11 is added, enabling the execution of JavaScript code for crawling dynamic pages. The Predis client is pinned to version 2.0.\*. The ramsey/uuid dependency has been removed.

(dependencies) · medium confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 45 → 43 (-1.8)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 64 → 77 (+13.2)
• Architecture 96 → 93 (-2.9)
• Maturity 48 → 32 (-16.1)
• Readiness 24 → 26 (+1.4)
• Security 97 → 98 (+0.8)

• Change coupling: Downloader.php ↔ Producer.php (src/Downloader.php)
• Change coupling: Task.php ↔ Producer.php (src/Kernel/Task.php)
• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — dependency manifest found but not parsed for hygiene
• Duplicated block (10 lines × 2) (src/Kernel/Library/Helper/Tool.php)
• Duplicated block (10 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Validate/File/Count.php)
• Duplicated block (10 lines × 2) (src/Kernel/Middleware/HeadlessBrowser/Chrome.php)
• Duplicated block (10 lines × 2) (src/Kernel/Middleware/MessageQueue/PredisClient.php)
• Duplicated block (10 lines × 4) (src/Kernel/Middleware/LockManager/DistributedRedLock.php)
• Duplicated block (11 lines × 2) (src/Downloader.php)
• Duplicated block (11 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Validate/Date.php)
• Duplicated block (11 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/phpQueryObject.php)
• Duplicated block (11 lines × 2) (src/Kernel/Middleware/LockManager/DistributedRedLock.php)
• Duplicated block (11 lines × 2) (src/Kernel/Middleware/MessageQueue/PredisClient.php)
• Duplicated block (12 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Http/Client/Adapter/Proxy.php)
• Duplicated block (12 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/phpQueryObject.php)
• Duplicated block (12 lines × 2) (src/Kernel/PHPCreeper.php)
• Duplicated block (13 lines × 2) (src/Kernel/Library/Polyfill/Uuid.php)
• Duplicated block (13 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Uri/Http.php)
• Duplicated block (13 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Validate/Barcode/Ean13.php)
• …and 23 more

• ClassTooLong: DOMDocumentWrapper (src/Kernel/Library/phpQuery/phpQuery/DOMDocumentWrapper.php)
• ClassTooLong: Downloader (src/Downloader.php)
• ClassTooLong: Medoo (src/Kernel/Middleware/Database/Database.php)
• ClassTooLong: PHPCreeper (src/Kernel/PHPCreeper.php)
• ClassTooLong: Tool (src/Kernel/Library/Helper/Tool.php)
• Dependency hygiene PARTLY measured — Composer dependencies read, no committed lock to grade for currency
• Documentation: no installation or build instructions (README.md)
• Duplicate entry point. 'phpQuery.pq' and 'src.Kernel.Library.phpQuery.phpQuery.pq' appear to be the same function exposed under different namespaces/paths, likely due to legacy code structure or autoloading issues.
• Duplicate intent with inconsistent naming. 'checkMobile' and 'checkTelphone' (note the typo in 'Telphone') appear to serve the same validation purpose for phone numbers. The parameter name 'mobile' is also reused for the telephone check, suggesting they are interchangeable or redundant.
• Duplicated block (10 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Validate/File/Count.php)
• Duplicated block (10 lines × 2) (src/Kernel/Middleware/HeadlessBrowser/Chrome.php)
• Duplicated block (11 lines × 3) (src/Downloader.php)
• Duplicated block (11–13 lines × 4) (src/Kernel/Middleware/LockManager/DistributedRedLock.php)
• Duplicated block (12 lines × 2) (src/Kernel/Library/Helper/Tool.php)
• Duplicated block (12 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/phpQueryObject.php)
• Duplicated block (12 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/phpQueryObject.php)
• Duplicated block (12 lines × 2) (src/Kernel/Middleware/LockManager/PredisLock.php)
• Duplicated block (12–13 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Http/Client/Adapter/Proxy.php)
• Duplicated block (13 lines × 2) (src/Kernel/Library/phpQuery/phpQuery/Zend/Uri/Http.php)
• Duplicated block (13 lines × 2) (src/Kernel/Middleware/DropDuplicate/BloomFilterPredis.php)
• …and 110 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

blogdaren/PHPCreeper was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.