jaimeiniesta/metainspector

MetaInspector is a Ruby library designed for extracting metadata and structural information from web pages. It handles HTTP requests with robust error handling and security features, such as SSRF prevention, while parsing HTML into specialized components for images, links, and text. The system provides utilities for link auditing, image optimization, and metadata normalization, supported by comprehensive test coverage for edge cases.

A new executable script at bin/console has been added, allowing users to launch an interactive Pry session with the Metainspector library and Bundler environment pre-loaded for debugging or exploration.

bin · high confidence

The examples directory now includes several new scripts demonstrating advanced MetaInspector usage: a basic scraping script (basic\_scraping.rb), a spider that recovers from unparseable links (spider.rb), a broken link checker (link\_checker.rb), and a demonstration of Faraday's redirect options to prevent Server-Side Request Forgery (SSRF) attacks (faraday\_redirect\_options.rb, supported by a local redirect server). These additions provide users with concrete patterns for handling HTTP redirects securely, recovering from parsing errors, and auditing site links.

examples · high confidence

The library has been reorganized into a modular structure, replacing the previous monolithic implementation with specialized components for handling requests, URLs, and document parsing. This change introduces dedicated parsers for specific content types, including images, links, head links, meta tags, and texts, allowing for more focused and maintainable scraping logic. The main entry point now delegates to a Document class, which coordinates these specialized parsers to extract metadata, while also providing a convenience method for quick scraping.

lib · high confidence

This release updates the library to version 5.17.2 and introduces a significant architectural shift in how HTTP requests and document parsing are handled. The \Request\ class now utilizes Faraday with specific middleware for gzip compression, retries, and cookie persistence, while enforcing strict HTTP URL validation and adding a custom fatal timeout to handle streaming responses that might otherwise hang. Parsing logic has been extracted into a dedicated \Parser\ class that delegates to specialized sub-parsers for meta tags, links, images, and text, allowing for early failure if the document cannot be parsed. Additionally, the \URL\ class now supports tracking parameter detection and removal, and the \Document\ class exposes response status and headers via the \to\_hash\ method and \response\ accessor.

_lib/meta\inspector · high confidence

The library's parsing logic has been restructured into specialized components (HeadLinks, Images, Links, MetaTags, Texts) inheriting from a new Base class. This change introduces new methods for users: \images.with\_size\ returns images sorted by area, \images.largest\ finds the most square large image, and \images.best\ prioritizes Open Graph/Twitter images. Link handling now includes \links.internal\ and \links.external\ categorization, while the \HeadLinksParser\ adds \feeds\ and \canonicals\ methods. Text extraction now supports \best\_title\, \best\_author\, and \best\_description\ with fallback logic, and heading tags (h1-h6) are now accessible.

_lib/meta\inspector/parsers · high confidence

Added comprehensive test coverage for MetaInspector parsing features; Added test coverage for Document, Parser, Request, and URL components; Expanded test fixtures for HTML parsing edge cases.

The project now defines its dependencies explicitly in the gemspec file, establishing the core runtime requirements for web scraping: Nokogiri (\~\> 1.19.0), Faraday (\~\> 2.5) with several specific plugins (cookie\_jar, encoding, follow\_redirects, gzip, http-cache, retry), Addressable (\~\> 2.9.0), and FastImage (\~\> 2.2). Development dependencies are also configured, including RSpec (\~\> 3.11), WebMock (\~\> 3.17), and Puma (\~\> 8.0.0). The Gemfile delegates all dependency management to this gemspec.

(dependencies) · high confidence

Establishes the project structure with a new .gitignore file (excluding build artifacts and lockfiles), example configuration files for RSpec and RuboCop, and comprehensive documentation including a detailed README, a full changelog, and the MIT license.

(repo-wide) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 61.

• Code Health 100
• Architecture 69
• Maturity 52
• Readiness 66
• Security 68

• 300 commits — 291 feature/other, 9 fixes

• (root) — 227 commits
• lib/meta_inspector — 42 commits
• (repo) — 10 commits
• .circleci/config.yml — 8 commits
• spec/fixtures — 5 commits
• spec/meta_inspector — 2 commits
• .github/FUNDING.yml — 1 commit
• .github/dependabot.yml — 1 commit
• examples/link_checker.rb — 1 commit
• examples/spider.rb — 1 commit
• spec/document_spec.rb — 1 commit
• spec/request_spec.rb — 1 commit

• fix: Fix for empty base_url (#267)
• fix: Fix issue #215 by not normalizing image URLs (#241)
• fix: Fix parser best title and author priority (#514)
• fix: Fixes documentation about MetaInspector::TimeoutError. Closes #182
• fix: Revert "remove Gemfile.lock from version control"
• fix: Update addressable to ~> 2.9.0 and fix Faraday 2.x test compatibility (#512)
• fix: fix feed parsing
• fix: update to Addressable 2.8.1 which fixes invalid_byte_sequence exception
• fix: use the Gzip middleware. Fixes #191
• change: #240 Absolutified relative base href
• change: - description will only return the meta description if present. - best_description will try several candidates, return nil if no match.
• change: 5.4.0
• change: Add .circleci/config.yml (#285)
• change: Add ArgumentError Exception to URL.absolutify
• change: Add method best_description
• change: Add presence validation for empty string on meta tags image values (#227)
• change: Add spec to verify 'invalid byte sequence' handling in links
• change: Adds a timeout on top of the ones handled by Faraday, as Faraday won't raise a timeout on streaming responses.
• change: Bump addressable from 2.8.4 to 2.8.5 (#396)
• change: Bump addressable from 2.8.5 to 2.8.7 (#444)
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

jaimeiniesta/metainspector was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.