gocolly/colly

Colly is a high-performance web scraping framework for Go that provides a comprehensive toolkit for building robust crawlers. It supports advanced features such as parallel and asynchronous scraping, request queuing, and rate limiting to manage concurrency and avoid overwhelming target servers. The system includes built-in extensions for user-agent rotation, URL filtering, and proxy switching, alongside a CLI for rapid project scaffolding. Additionally, it offers debugging tools and in-memory storage to facilitate development and maintenance.

This period focused on expanding the Colly framework with new features such as HTTP tracing, proxy rotation, and a CLI code generator, alongside a significant addition of diverse web scraping examples. The release of version 2.1.0 introduced functional options for configuration and enhanced debugging capabilities, while removing outdated examples to streamline the codebase.

19 changes

This period focused on enhancing Colly's capabilities for concurrent crawling and request management by introducing a request queue and in-memory storage backend. The team also expanded the library's utility with an extensions package for common scraping tasks and provided new examples demonstrating these features.

8 changes

gocolly/colly was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.