tribbloid/spookystuff

SpookyStuff is a distributed agent swarm built on Apache Spark and Scala, designed for complex data extraction and web scraping workflows. It provides a type-safe, LINQ-style API for manipulating records and integrates Selenium for Sizzle-based web interaction. The system includes specialized infrastructure for graph-based parsing, incremental RDD caching, and resource lifecycle management to optimize performance and reliability.

This change introduces a new graph-based finite state machine (FSM) parser combinator in the \parsing\ module, allowing users to define parsers as directed graphs with typed nodes and edges, and compile them into deterministic decision processes. It also adds an \IncrementallyCachedRDD\ mechanism in the \interleaved\ module, which provides an RDD wrapper that caches computed partitions incrementally to optimize repeated or partial data access patterns. Additionally, a \PendingEffect\ type system is added to the \misc\ module to support effect tracking for optional (\Maybe\) and throwable (\MayThrow\) contexts, alongside various test suites and benchmarks validating these new components.

\~unused/interleaved, \~unused/misc, \~unused/parsing · high confidence

This change introduces a new, type-safe record abstraction in the \module/linq\ module (replacing legacy shapeless-based structures) that provides fluent APIs for merging, updating, and appending rows, along with support for sorting and conflict resolution. It also adds a new \BySizzleSelector\ in the \module/web\ module, enabling Selenium-based web scraping to use Sizzle CSS selectors via JavaScript execution.

dev/profiles, module/linq, web · high confidence

This change introduces several new internal modules within the commons and benchmark packages to support core functionality. It adds a \BenchmarkHelper\ and \Benchmark\ class for running and reporting performance tests, a \Cleanable\ trait with \Lifespan\ types (JVM, Task, Hadoop, Spark) for managing resource cleanup and shutdown hooks, and a comprehensive \EAVLike\ system with \EAVSchema\ and \EAVOps\ for structured entity-attribute-value data management. Additionally, it includes utility classes such as \AwaitWithHeartbeat\ for monitored asynchronous waits, \Stopwatch\ for timing, \RangeMagnet\ for range handling, and \ClasspathResolver\ for resource scanning via ClassGraph.

module/core, repository · high confidence

This change adds a comprehensive set of new utility classes and traits to the \module/commons\ package to support a new, type-safe extraction model. Key additions include \TypeMagnet\ and associated reflection utilities (\FromClassMixin\, \ReflectionUtils\) to unify Scala type representations, \AttrLike\ for structured attribute handling, and \UnreifiedObjectType\ for preserving Scala types within Spark schemas. The update also introduces robust cross-platform file handling via \CrossPlatformFileUtils\ and \WindowsFileCompatibility\, a new \URIResolver\ for resilient I/O operations, and lifecycle management tools like \FutureInterruptable\ and \Lifespan\ types. These components provide the foundational building blocks for the refactored extractor logic.

module, module/commons, module/web · high confidence

The dev/CI directory now includes a new set of shell scripts (download.sh, main.sh, pipeline.sh) that automate the setup and execution of the CI environment. download.sh fetches Spark version 4.0.1 (default) and sets the Scala version to 2.13, configuring the environment for the build. main.sh acts as the entry point, sourcing specific profiles and triggering the Gradle build, submodule updates, and the pipeline. pipeline.sh orchestrates the compilation and test phases, ensuring the exit status is correctly propagated after generating test reports. This replaces previous ad-hoc CI logic with a structured, reusable workflow.

dev/CI · high confidence

The dev directory now includes a suite of shell scripts to streamline local development and CI workflows, replacing previous tooling. These scripts manage the Gradle wrapper (upgraded to version 8.14.3), handle code formatting via scalafmt and scalafix, run tests with specific flags like --rerun and --continue, and update git submodules. A new .shared.sh script sets environment variables required for Spark compatibility (SPARK\_LOCAL\_HOSTNAME) and logging. Backward-compatibility wrappers for CI environments (apache-latest, apache-local, apache-stable) are also provided to delegate to the main CI script.

dev · high confidence

The project has been rebranded from 'spookystuff' to 'SpookyStuff' (Distributed Agent Swarm) and migrated to a Gradle build system with Kotlin DSL, replacing the previous setup. This change introduces a multi-module architecture (core, web, linq, assembly, etc.), adds AI agent configuration files (AGENTS.md, CLAUDE.md), and updates the README to reflect the new branding, supported technologies (Spark 3.5, Scala 2.13), and tooling.

(repo-wide) · high confidence

Added a comprehensive set of integration tests for the \module/integration\ module, including a base specification (\ITBaseSpec\) and specific test cases for web exploration, page navigation, and data extraction. Also added benchmark tests for parsing performance and hash calculations in \module/benchmark\, and a jar conflict detection test in \module/assembly\.

module/assembly, module/benchmark, module/integration · high confidence

The project has switched its build system to Gradle (Kotlin DSL), introducing a new multi-module structure with specific submodules for core, web, assembly, and repackaged libraries like Selenium and Tika. This change includes upgrading Apache Spark to version 4.0.1 and Selenium to 4.39.0, while also configuring the build to use Scala 2.13.18 and enabling parallel builds and caching for improved performance.

(dependencies) · high confidence

The project's Gradle wrapper has been updated to version 8.14.3. This ensures that builds use a specific, modern version of the Gradle build tool, providing access to the latest features, performance improvements, and bug fixes available in that release.

gradle · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 47.

• Code Health 78
• Architecture 95
• Maturity 39
• Readiness 50
• Security 43

• 300 commits — 250 feature/other, 50 fixes

• core/src — 104 commits
• mldsl/src — 63 commits
• (root) — 40 commits
• parent/core — 24 commits
• parent/mldsl — 13 commits
• integration/src — 6 commits
• parent/commons — 6 commits
• .github/workflows — 5 commits
• build-tools/CI — 3 commits
• dev/CI — 3 commits
• module/core — 3 commits
• test-resources/src — 3 commits
• benchmark/src — 2 commits
• build-tools/test-install.sh — 2 commits
• buildSrc/src — 2 commits
• dev/profiles — 2 commits
• dev/tree.sh — 2 commits
• repackaged/selenium-bundle — 2 commits
• web/src — 2 commits
• .circleci/config.yml — 1 commit

• fix: Cleanable now inherits Closeable Retry now ignores ControlThrowable fix all tests for ExternalAppendOnlyArray and IncrementallyCachedRDD improve IncrementallyCachedRDD, now the unpersist() API has been disabled
• fix: SPARK_V in download.sh is inconsistent, fixed now
• fix: SerBox renamed to SerDeOverride fix a compatibility problem between TypeTag serialisation in scala 2.11 and Kryo
• fix: SerDeOverride.obj is now private SerDeOverride.override can only be initialised from SparkConf ScalaType.FromTypeTag initialization is now improved fix an unstable test in RDDDisperseSuite
• fix: TestSparkSession now validates if called on executor net.jpountz.lz4 is now evicted from dependency as a similar package is already in Spark SparkJobGroupMixin renamed to SparkUISupport and is now inherited by SpookyEnvFixture revise ExternalAppendOnlyArray and tests fix bugs in IncrementallyCachedRDD & tests to benefit from the new ExternalAppendOnlyArray
• fix: circleCI save the wrong directory to the cache, fixed
• fix: fix "java.lang.IllegalArgumentException: bound must be positive" error on CI
• fix: fix 2 volatile tests in AbstractURIResolverSuite
• fix: fix 3 unit test errors fix a stackoverflow error add a test to ensure that it won't happen
• fix: fix 6 test errors regarding catalystType => ScalaType conversion
• fix: fix Selenium driver download through WebDriverDeployment (this only affects driver, not browser)
• fix: fix a broken unit test
• fix: fix a bug caused by reusing an iterator
• fix: fix a bug caused by simplification
• fix: fix a bug in sparkConf
• fix: fix a bug where apache-stable CI profile still uses scala 2.12
• fix: fix a compatibility issue with scala-2.12
• fix: fix a compilation error
• fix: fix a compilation error
• fix: fix a few warnings
• …and 280 more

• 0 containers · 2 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

tribbloid/spookystuff was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.