felipecsl/wombat

Wombat is a Ruby gem that provides a DSL-based framework for web scraping and crawling. It enables users to define extraction logic through a flexible block-based API, supporting various HTTP methods, CSS/XPath selectors, and data formats like lists and XML. The system handles metadata inheritance, link following, and configuration options such as proxies and user agents.

6 changes.

This period marks the initial release of the Wombat gem, a Ruby DSL for web scraping, replacing the previous event-crawler structure with a block-based API. The work involved establishing the core crawling logic, configuration options, and a comprehensive test suite using RSpec and VCR. It also included setting up the build system and dependency management for the new library.

8 changes.

The project refactored the DSL property model into a modular class hierarchy to support iterators, link following, and HTTP headers. It expanded core capabilities to include multiple HTTP methods and flexible node selection, accompanied by comprehensive integration and unit tests.

Added five new example scripts in the examples directory demonstrating Wombat's crawling capabilities: hashes.rb and no\_class.rb show block-based and string-based selector syntax for extracting data; iterator.rb and list.rb demonstrate how to crawl repeated elements like repository lists and gem categories; and xml.rb shows how to parse XML documents with namespace support.

examples · high confidence

The library introduces the Wombat gem, replacing the previous event-crawler structure. Users can now perform web crawling using a new \Wombat.crawl\ method that accepts a block, providing a more flexible API. The gem also supports configuration via \Wombat.configure\, allowing users to set proxy arguments, custom user agents, and user agent aliases. The \crawl\ method is aliased as \scrape\ for convenience.

lib · high confidence

The web crawler now supports GET, POST, PUT, PATCH, DELETE, and HEAD requests, allowing users to interact with APIs and forms beyond simple page fetching. Additionally, the new NodeSelector module enables more flexible data extraction by allowing selectors to be specified as strings (with 'xpath=' or 'css=' prefixes), symbols, or hashes, providing greater control over how content is parsed from HTML and XML responses.

lib/wombat/processing · high confidence

The \Wombat::Crawler\ module now supports a \crawl\ method that accepts a block, allowing users to define dynamic scraping logic at runtime rather than relying solely on class-level definitions. This block-based approach clones the crawler's metadata to ensure isolation during execution. Additionally, the crawler now automatically clones metadata into subclasses upon inheritance, enabling child crawlers to inherit and extend the configuration of their parents.

lib/wombat · high confidence

The Wombat DSL has been restructured to introduce distinct property types for handling different data extraction patterns. Users can now define properties that extract multiple values using iterators, follow hyperlinks, and retrieve HTTP response headers, in addition to standard text extraction. This change also updates the underlying property model to support these new formats and allows configuration of base URLs, HTTP methods, and document formats within the metadata.

lib/wombat/dsl · high confidence

The property extraction logic in lib/wombat/property has been restructured from a monolithic implementation into a modular class hierarchy under the Locators module. This change introduces a Base locator class and specific subclasses (Text, Html, List, Iterator, PropertyGroup, Follow, Headers) that handle distinct data formats, along with a Factory to instantiate the correct locator based on the property format. This refactoring improves code organization and maintainability for users defining page object properties.

lib/wombat/property · high confidence

This release updates the Wombat web scraper to version 3.3.0, raising the minimum supported Ruby version to 3.4.7 and bumping all dependencies to ensure compatibility with Ruby v3.1+. The changelog documents several prior behavioral changes, including the introduction of hash-based property selectors (e.g., \css: 'header'\), support for HTTP methods like POST and PUT, and the addition of a \:follow\ property type for crawling links. The release also includes a DSL syntax simplification where nested blocks no longer take arguments, and the \Metadata\#format\ method was renamed to \Metadata\#document\_format\ to avoid conflicts with Ruby's Kernel. Additionally, the project migrated its build system from Jeweler to Juwelier and updated its test runner to RSpec.

(repo-wide) · high confidence

Added integration tests for crawler inheritance, relative link following, and core crawling behaviors; Added test coverage for the Wombat processing parser; Added tests for Wombat::DSL::Property; Added tests for property locator implementations; Initial test suite for Wombat crawler and configuration; Removal of initial test scaffolding.

This change introduces the initial version (3.3.0) of the Wombat gem, a Ruby DSL for scraping web pages. The package defines runtime dependencies on activesupport, mechanize (\~\> 2.14.0), and rest-client, while development dependencies include bundler, coveralls, juwelier, rake, rspec, vcr, webmock, and yard. The accompanying Gemfile.lock and gemspec establish the complete dependency tree and file structure for the project.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 68.

• Code Health 100
• Architecture 69
• Maturity 59
• Readiness 74
• Security 84

• 300 commits — 278 feature/other, 22 fixes

• (root) — 195 commits
• lib/wombat — 52 commits
• (repo) — 26 commits
• fixtures/vcr_cassettes — 7 commits
• lib/event-crawler — 5 commits
• spec/crawler_spec.rb — 4 commits
• spec/integration — 3 commits
• spec/property — 3 commits
• .github/dependabot.yml — 1 commit
• .github/workflows — 1 commit
• spec/dsl — 1 commit
• spec/helpers — 1 commit
• spec/processing — 1 commit

• fix: Another readme fix
• fix: Bug fix in Property Iterator
• fix: Code styling fixes
• fix: Fix README code example
• fix: Fix bug on malformed selectors
• fix: Fix reame
• fix: Fix specs.
• fix: Fixes #53 missing missing mechanize_page accessor
• fix: Formatting fix
• fix: Merge pull request #59 from santib/fix-readme
• fix: README formatting fix
• fix: README formatting fix
• fix: Update dependencies and fix tests
• fix: Updates dependencies and fixes rspec new expectation format
• fix: Updates dependencies. Adds Coveralls. Fixes rspec deprecation warning
• fix: chore: Add CI workflow, bump dependencies and fix tests (#97)
• fix: fixed bug in for_each
• fix: fixed bug in parser where callback was not being invoked for properties inside an iterator
• fix: fixed bug in property callback
• fix: fixed dependency on guard
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

felipecsl/wombat was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.