jhao104/proxy_pool

This system is a proxy pool management service that aggregates, validates, and stores HTTP and HTTPS proxy addresses from multiple sources. It provides a REST API for querying and manipulating the pool, backed by Redis or SSDB for data persistence. The architecture includes a scheduler for periodic fetching, a plugin-based fetcher system for diverse data sources, and a CLI for operational control.

9 changes.

The project underwent a comprehensive architectural overhaul, introducing a unified service management script, standardized project scaffolding, and a modular codebase with dedicated API, database, and fetcher layers. Legacy components were removed while new abstractions for proxy handling, dependency management, and cross-platform compatibility were established to support scalable deployment and testing.

4 changes.

This period focused on establishing a comprehensive pytest-based test infrastructure, including shared fixtures and extensive unit and integration tests for the proxy pool components and API endpoints. Concurrently, the project expanded its proxy data sources by integrating three new providers: daili66, Proxifly, and RoundProxies.

The util package now includes several new modules: a six.py compatibility layer for Python 2 and 3 (handling iteritems, urlparse, reload, and Queue), a Singleton metaclass, a LazyProperty descriptor, and a WebRequest class that wraps the requests library with automatic User-Agent rotation, retry logic, and response parsing via lxml.

util · high confidence

The database module now provides a unified factory interface (DbClient) that routes operations to specific backend implementations for Redis and SSDB. This change introduces new client classes (RedisClient, SsdbClient) that handle proxy storage in hash structures, supporting both HTTP and HTTPS proxy differentiation. The implementation includes methods for getting, putting, popping, updating, and deleting proxies, along with connection pooling and error handling for Redis connections.

db · high confidence

Users now have a single entry point to manage the proxy pool service via the new \proxy\_pool.sh\ script, which handles starting, stopping, restarting, and checking the status of both the API server and scheduler in foreground (for Docker) and background modes. This change is accompanied by the addition of a \CLAUDE.md\ file providing technical guidance for AI coding assistants, a \tox.ini\ for automated multi-version testing, and a \mkdocs.yml\ for documentation generation. The project scaffolding is further standardized with a new \LICENSE\, updated \Dockerfile\ using Python 3.10-alpine, and a \docker-compose.yml\ for easy local deployment with Redis.

(repo-wide) · high confidence

The proxy fetcher module now includes three new data sources: daili66 (fetching from its API), Proxifly (filtering Chinese HTTP proxies from a JSON list), and RoundProxies (fetching free proxies via API). These additions expand the pool of available proxy addresses for users.

fetcher/sources · high confidence

The \getFreeProxy.py\ module, which previously contained logic for scraping free proxy IPs from sources like Kuaidaili and 66ip.cn, has been removed from the codebase. This deletion eliminates the associated proxy fetching functionality and its underlying parsing logic.

ProxyGetter · high confidence

The API logic is now organized in a dedicated \api/\ package, with the main application moved to \api/proxyApi.py\. The \/count/\ endpoint has been updated to return a more detailed breakdown of the proxy pool, including counts by HTTP type (HTTP/HTTPS) and by source, rather than just a total count.

api · high confidence

The proxy fetching module has been restructured to introduce a \BaseFetcher\ base class in \fetcher/baseFetcher.py\. This class centralizes common functionality, including a regex-based method for extracting IP:port pairs from text (\parseProxiesFromText\) and a utility for yielding unique proxies (\yieldUniqueProxies\). Subclasses are now expected to implement the \fetch\ method while inheriting these shared utilities, simplifying the implementation of new proxy sources and removing previously duplicated parsing logic.

fetcher · high confidence

The helper module has been reorganized into distinct components: fetch.py now uses a dynamic plugin system to auto-discover and run proxy fetchers from the fetcher/sources directory, check.py implements multi-threaded validation that distinguishes between HTTP and HTTPS support and retrieves geographic region data via ip.sb, validator.py adds support for authenticated proxy formats (username:password@ip:port), and scheduler.py introduces timezone configuration for scheduled tasks. The Proxy object now tracks region, source, and HTTPS status, enabling more granular proxy management and filtering.

helper · high confidence

The proxy handling logic has been reorganized into specific modules: \proxyHandler.py\ now manages CRUD operations for HTTP and HTTPS proxies, \configHandler.py\ centralizes configuration retrieval from environment variables and settings, and \logHandler.py\ implements logging with a fix for Windows thread-safety issues by avoiding \TimedRotatingFileHandler\. This change improves code modularity and ensures reliable logging across platforms.

handler · high confidence

Added proxy API test suite; Added pytest test suite with shared fixtures; Comprehensive test suite added for proxy pool components.

The project introduces its first dependency management files, establishing a baseline for the application and its testing environment. The main requirements.txt pins specific versions for core libraries including Flask 2.1.1, Werkzeug (constrained to \<2.2), Requests 2.31.0, Gunicorn 19.9.0, lxml 4.9.2, and APScheduler (versioned by Python version), alongside Redis and Click. A new requirements-test.txt file defines the testing stack, specifying pytest, pytest-cov, fakeredis, and async\_timeout. Additionally, pyproject.toml is added to configure pytest test paths and coverage reporting settings.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 64.

• Code Health 98
• Architecture 97
• Maturity 62
• Readiness 56
• Security 67

• 300 commits — 286 feature/other, 14 fixes

• (repo) — 104 commits
• (root) — 101 commits
• .github/workflows — 13 commits
• docs/changelog.rst — 11 commits
• tests/unit — 11 commits
• fetcher/proxyFetcher.py — 9 commits
• fetcher/sources — 7 commits
• docs/user — 6 commits
• api/proxyApi.py — 5 commits
• docs/dev — 3 commits
• helper/check.py — 3 commits
• helper/fetch.py — 3 commits
• helper/scheduler.py — 3 commits
• docs/changelog.md — 2 commits
• docs/conf.py — 2 commits
• handler/configHandler.py — 2 commits
• handler/logHandler.py — 2 commits
• DB/SsdbClient.py — 1 commit
• Util/init.py — 1 commit
• config/init.py — 1 commit

• fix: Merge pull request #472 from StarkShang/stark/fix
• fix: [bugfix] fix freeProxy09 with request params, and add freeProxy05 page param
• fix: [fix] :heavy_check_mark: fix python2 & python3 metaclass
• fix: [fix] :heavy_check_mark: fix python2 & python3 metaclass
• fix: [fix] 代理验证时数量小于minSize时运行抓取和验证
• fix: [update] :heavy_check_mark: Windows下TimedRotatingFileHandler线程不安全, 不再使用 fix #469
• fix: fix some doubts
• fix: fix: mock apscheduler import for tox/uv compatibility (py39+)
• fix: fix: pin werkzeug<2.2 to fix url_quote ImportError with Flask 2.1.1
• fix: fix: 修复No module named 'config'报错
• fix: fix: 修复采集器若干问题并优化模块加载性能
• fix: fix: 升级Docker基础镜像至python:3.10-alpine修复构建失败
• fix: fix: 避免CI在PR时重复触发测试
• fix: refactor: update ihuan/geonode fetchers and fix ihuan tests
• change: :lollipop:
• change: Add files via upload
• change: Add timezone config for the scheduler
• change: Bump lxml from 4.3.1 to 4.6.2
• change: Bump lxml from 4.6.2 to 4.6.3
• change: Bump requests from 2.20.0 to 2.31.0
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

jhao104/proxy_pool was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.