scalacenter/scaladex

Scaladex is a Scala ecosystem index and discovery platform that ingests, normalizes, and stores metadata for projects and artifacts from GitHub and Maven Central. It provides a REST API and web interface for searching, filtering, and browsing this data, while also offering administrative tools to manage background synchronization jobs and view adoption insights. The system relies on PostgreSQL for persistent storage and Elasticsearch for full-text search, with a backend built on Apache Pekko and a frontend serving static pages and API documentation.

37 changes.

The project underwent a major infrastructure overhaul, migrating the runtime from Akka to Apache Pekko and upgrading the build system to SBT 1.12 and Scala 3. This period established a new modular architecture with a dedicated core service layer, database abstractions, and a new REST API v1, while introducing comprehensive test coverage and administrative tooling for ecosystem data management.

8 changes.

This period focused on launching the Scaladex REST API v1, providing comprehensive endpoints for project and artifact search while maintaining backward compatibility with the legacy v0 API. The work was supported by the introduction of Swagger UI for documentation and the implementation of a Gatling-based load testing harness to validate performance under realistic traffic conditions. Additionally, test coverage was expanded to include integration tests for GitHub client interactions, search relevance, and detailed parsing logic for SCM URLs and Maven POM files.

A new ScmInfoParser component has been introduced in the data cleanup module to parse SCM information strings. It uses fastparse to extract GitHub repository references from various URL formats (including git@, https, git://, and ssh), normalizing them by stripping the .git suffix to produce a clean Project.Reference for downstream use.

modules/data/src/main/scala/scaladex/data/cleanup · high confidence

A new JavaScript file (swagger-initializer.js) has been added to the server resources to configure the Swagger UI. This setup loads the UI with definitions for both the current Scaladex API v1 and the legacy v0, enabling users to view and interact with the API documentation directly through the standalone layout.

modules/server/src/main/resources/lib · high confidence

This change introduces three new resource files for the data module: a Logback configuration that sets up console logging with specific log levels for Akka, Elasticsearch, and Flyway (silencing RestClient warnings); a reference configuration file setting the environment to 'local'; and a subindex file containing a curated list of Scala libraries and tools.

modules/data/src/main/resources · high confidence

New documentation files have been added to the \doc/dev\ directory to assist with development workflows. These include guides for authenticating and using the Bintray, GitHub, and StackOverflow APIs, as well as sample Maven POM files (\noscm\_2.11-1.0.0.pom\ and \test\_2.11-1.1.5.pom\) demonstrating project structures and dependency configurations.

doc/dev · high confidence

The server now exposes a new AdminService that manages background jobs (such as syncing search indexes, updating project dependencies, and refreshing GitHub metadata) and on-demand administrative tasks (like adding empty projects, updating GitHub info, and republishing artifacts). This service relies on new service classes including JobScheduler for periodic execution, TaskRunner for user-triggered operations, and dedicated updaters for artifacts, dependencies, and search synchronization. Additionally, a ScaladocService is introduced to lazily download and cache ScalaDoc JARs from Maven Central with bounded disk usage and protection against zip bombs, while MavenCentralService handles finding and indexing missing artifacts.

modules/server/src/main/scala/scaladex/server/service · high confidence

A pre-commit hook has been added to automatically check code formatting using scalafmt before changes are committed. This ensures that all code adheres to the project's style guidelines without requiring manual intervention from developers.

bin · high confidence

The repository is initialized with essential configuration files to support the Scaladex project's development workflow. This includes setting up code formatting and linting via \.scalafmt.conf\ (version 3.11.5, Scala 3 dialect) and \.scalafix.conf\ (rules for explicit result types, import organization, and unused removal). Git submodules for \small-index\ and \contrib\ data repositories are added via \.gitmodules\, and a \.git-blame-ignore-revs\ file is created to ignore specific formatting and migration commits. Development environment settings are defined in \.jvmopts\ (increasing heap to 2G and stack to 2M) and \.scala-steward.conf\ for automated dependency updates. Documentation is provided through \README.md\ and \CONTRIBUTING.md\, and a \LICENSE\ file is added.

(repo-wide) · high confidence

The Scaladex API now exposes a new v1 endpoint structure that provides comprehensive access to project and artifact information. Users can retrieve detailed project metadata (including stars, forks, licenses, and topics), list project versions and artifacts with filtering options (by language, platform, binary version, and stability), and perform full-text search across projects with pagination and sorting. The API also supports artifact-specific queries, such as fetching the latest artifact for a group and ID, and includes an autocomplete endpoint for quick discovery. This replaces the previous API versioning scheme with a dedicated v1 path, ensuring backward compatibility for existing v0 consumers while offering a richer, more structured data model for new integrations.

modules/core/shared/src/main/scala/scaladex/core/api · high confidence

The Scaladex server now exposes a new REST API v1 alongside the existing v0 endpoints. This new version adds capabilities for searching projects, retrieving project dependencies and dependents, and managing authenticated user states and project settings (including patching settings). The implementation includes dedicated route definitions, OpenAPI documentation for both v0 and v1, and authentication logic using GitHub tokens via basic auth.

modules/server/src/main/scala/scaladex/server/route/api · high confidence

The \scaladex.data\ module now provides the core application entry point (\Main\) to manage indexed POMs, supporting three distinct operations: \init\ to populate the database via Flyway migrations, \subIndex\ to export a curated subset of projects defined in \subindex.txt\ to the filesystem, and \generateFeeders\ to produce CSV files (orgs\_repos, artifacts, search\_terms) for the Gatling load-test harness. This change introduces the \GenerateFeeders\ utility to create test data from the local index, the \SubIndex\ logic to filter and save specific projects, and a \PidLock\ utility to prevent concurrent execution in development or production environments.

modules/data/src/main/scala/scaladex/data · high confidence

This change introduces a new set of Twirl templates and supporting Scala view helpers in the \modules/template\ module, establishing the frontend structure for several key areas of the site. It adds the 'Awesome Scala' curated list experience (including category browsing, filtering, and result lists), a dedicated 'Insights' page for visualizing Scala version adoption and migration status, and a comprehensive 'Admin' interface for managing background jobs and tasks. Additionally, it provides the foundational HTML structure for the main frontpage, project artifact details, and standard error pages (404/403), along with utility functions for pagination, pluralization, and URI handling.

modules/template · high confidence

This change introduces four new utility files in the core shared module: JsoupUtils for extracting directory listings and Maven metadata versions from HTML/XML; Parsers providing fastparse-based combinators for alpha, digit, and number parsing; ScalaExtensions adding extension methods for Future, Try, Option, Iterator, and FiniteDuration (including pretty-printing); and TimeUtils for measuring task duration and calculating percentages. These utilities support the broader system's parsing and metadata-fetching capabilities.

modules/core/shared/src/main/scala/scaladex/core/util · high confidence

The scaffolded demo application that used the autowire library for client-server communication has been removed. This eliminates the example HTML page, the Scala.js client code, and the Akka HTTP server implementation that previously handled the '/api' endpoint and served static assets.

webapp/js, webapp/jvm · high confidence

The \Api\ trait, which previously defined a \hello\ method, has been removed from the shared webapp module. This cleanup eliminates dead code that was not contributing to the application's functionality.

webapp/shared · high confidence

Adds a new set of case classes in the \scaladex.core.api\ package to define the structure of the new API v1 responses and search inputs. This includes \ArtifactResponse\ for artifact details, \ProjectResponse\ for project metadata, \SearchResult\ for search matches, and various parameter classes (\ProjectSearchParams\, \ProjectsParams\, \ProjectArtifactsParams\, etc.) to handle filtering and sorting. These models replace or supplement the previous data structures, enabling the new API endpoints to return standardized, typed JSON responses for artifacts, projects, and search queries.

repository · high confidence

A change to existing behaviour in doc/assets — 2 commits, 3 files.

doc/assets · medium confidence · unverified

The webclient now fixes relative links and images in server-rendered READMEs by rewriting them to point to GitHub's raw and blob URLs, avoiding client-side re-fetches that could hit rate limits. It also introduces an ActiveNavObserver that uses IntersectionObserver to highlight the current section in the navigation bar as the user scrolls, and adds a Sparkline chart to display commit activity on project pages.

modules/webclient · high confidence

This change introduces a suite of Flyway database migrations to correct and enrich artifact data. It adds new fields \is\_semantic\ and \is\_prerelease\ to the artifacts table (V13\_2), fixes platform and language version assignments for SBT artifacts (V9, V25), corrects artifact naming conventions for SBT 2.0.0-M2 (V26), and handles Mill platform-specific artifacts by updating valid plugins and deleting invalid ones (V17). These migrations ensure that the stored platform, language, and version metadata accurately reflects the build tool and versioning scheme used by the libraries.

modules/infra/src/main/scala/scaladex/infra/migrations · high confidence

This update applies a series of database migrations to the \artifacts\, \project\_settings\, \github\_info\, and \user\_sessions\ tables. Key changes include enforcing non-null constraints on \release\_date\, adding boolean flags for semantic and pre-release status, and introducing a \full\_scala\_version\ field. Project settings have been refactored by renaming columns (e.g., \default\_stable\_version\ to \prefer\_stable\_version\), removing obsolete fields, and adding a \chatroom\ column. The schema also introduces new tables for release dependencies, adds license and commit activity tracking to GitHub info, and implements various indexes to improve query performance for artifact lookups and version filtering.

modules/infra/src/main/resources/migrations · high confidence

This change bundles the deployed logback configuration and establishes a new reference configuration for the application's infrastructure components. Users can now rely on sensible defaults for caching, Elasticsearch, database connection pooling, and HTTP client behavior (including rate limiting, retries, and circuit breakers for GitHub and Maven Central). Additionally, a new production logging profile is introduced that separates HTTP access logs from general application logs, applying specific log levels to key libraries like Akka, Play, and Elasticsearch to improve observability without excessive noise.

modules/infra/src/main/resources · high confidence

The \scaladex/core/service\ module now provides a set of new traits and classes that define the core service layer and database abstractions for the application. This includes \GithubAuth\ and \GithubClient\ for handling GitHub authentication and project/user data retrieval, \MavenCentralClient\ and \PomResolver\ for fetching artifact metadata and POM files, and \ProjectService\ for managing project versions, dependencies, and settings. Additionally, \WebDatabase\ and \SchedulerDatabase\ define the data access interfaces for artifacts, projects, and user sessions, while \SearchEngine\ and \Storage\ handle search indexing and persistent storage operations.

modules/core/shared/src/main/scala/scaladex/core/service · high confidence

The project's build system has been significantly upgraded and restructured. The SBT version has been bumped from 0.13.11 to 1.12.15, and the Scala.js plugin has been upgraded from 0.6.8 to 1.22.0, enabling modern cross-compilation features. New SBT plugins have been added, including sbt-twirl (2.0.9) for template rendering, sbt-sassify (1.5.2) for CSS processing, sbt-native-packager (1.11.7) for packaging, and gatling-sbt (4.19.1) for load testing. The build now integrates Testcontainers for local development and testing, with new SBT tasks (\startElasticsearch\, \startPostgres\) that automatically manage Docker containers for Elasticsearch (8.19.15) and PostgreSQL (16.15), caching them across reloads. Deployment logic has been consolidated into new \Deployment.scala\ and \Docker.scala\ modules, supporting both direct and jump-host deployments to production and development servers. The legacy \Helper.scala\ has been removed and replaced by \ScalaJSHelper.scala\ and \CurrentThread.scala\ to handle Scala.js packaging and classloader context management for Testcontainers.

project · high confidence

The infrastructure layer has been migrated from Akka to Apache Pekko, replacing Akka HTTP and Akka Streams with their Pekko equivalents. This change introduces a new \CommonAkkaHttpClient\ (now backed by Pekko) that provides configurable connection pooling, request throttling, and a circuit breaker for resilience. The migration also includes updated codecs for serialization, a new \CoursierResolver\ for POM resolution, and refactored clients for GitHub and Maven Central that utilize the new Pekko-based HTTP stack.

modules/infra/src/main/scala/scaladex/infra · high confidence

The server implementation has been migrated from Akka HTTP to Pekko HTTP, updating all internal imports and runtime dependencies to use the Apache Pekko actor system and HTTP server components. This change affects the core server startup, authentication flow, and HTTP request logging, ensuring the application runs on the Pekko platform while maintaining existing functionality.

modules/server/src/main/scala/scaladex/server · high confidence

The infrastructure layer now defines a dedicated Elasticsearch mapping for project data, introducing fields such as \scalaPercentage\, \commitsPerYear\, \formerReferences\, and specific handling for \artifactNames\ and \deprecatedArtifactNames\. This mapping also configures custom analyzers for README content to strip code blocks and URLs, ensuring more accurate text search capabilities for project descriptions.

modules/infra/src/main/scala/scaladex/infra/elasticsearch · high confidence

The SQL infrastructure module has been rewritten to use a new set of Doobie-based table definitions, introducing dedicated tables for artifacts, projects, dependencies, and user sessions. This change adds support for tracking semantic versioning and pre-release status on artifacts, deduplicates reverse dependency counts by source group and artifact, and enables configurable database connection pool sizes and statement timeouts. It also introduces an on-demand insights system that computes Scala version adoption statistics directly from the artifacts table.

modules/infra/src/main/scala/scaladex/infra/sql · high confidence

The system now uses a dedicated \Init\ component to handle database setup and data population. This process first resets the database schema using Flyway (cleaning and migrating tables) before importing all projects, artifacts, and dependencies from local storage. Upon completion, it logs the total counts of inserted projects, settings, artifacts, and dependencies to verify the operation.

modules/data/src/main/scala/scaladex/data/init · high confidence

The core model layer now includes helper utilities for handling avatar URLs with size parameters and SHA-1 hashing, alongside a new search sorting framework. This framework defines five distinct sorting options—Stars, Commit Activity, Contributors, Dependent, and Created—allowing users to order search results by these specific metrics.

modules/core/shared/src/main/scala/scaladex/core/model · high confidence

The Maven POM conversion logic in \PomConvert\ has been updated to read the \info.apiURL\ and \info.versionScheme\ properties from the POM model and include them in the resulting \ArtifactModel\. This allows the system to capture and store these specific metadata fields directly from the Maven artifact definition.

modules/data/src/main/scala/scaladex/data/maven · high confidence

The Scaladex web interface has been restructured with new dedicated pages for browsing the ecosystem (AwesomePages), viewing project details and artifacts (ProjectPages), and searching (SearchPages). A new Insights page is now available exclusively for logged-in users to view Scala version adoption data. The platform also introduces a new Admin page allowing administrators to manage background jobs and run maintenance tasks such as syncing GitHub info or republishing artifacts. Additionally, the Scaladoc viewing experience has been improved by hosting it behind a branded shell with an iframe, and the authentication flow has been updated to use GitHub OAuth with session management.

modules/server/src/main/scala/scaladex/server/route · high confidence

The server now includes explicit configuration files for logging, runtime settings, and web crawler access. A new logback.xml configures console logging with specific levels for Akka, Elasticsearch, and Flyway, while keeping HTTP access logging disabled by default but available via a commented-out debug logger. A reference.conf file defines default server endpoints, session secrets, OAuth2 credentials, and Pekko HTTP settings including idle timeouts. Additionally, a robots.txt file has been added to block major AI training bots (such as GPTBot, ChatGPT-User, and CCBot) while allowing standard search engine crawlers.

modules/server/src/main/resources · high confidence

The site now uses a dark navy background across the entire page, applied at the HTML level via the new base styles. This change is part of a broader restructuring of the frontend assets, which consolidates the previous scattered stylesheets into a single, modular SCSS build (\main-8.scss\). This new architecture imports shared utilities (variables, mixins) and vendor extensions (Bootstrap, Font Awesome, Select2) alongside distinct partials for the header, footer, frontpage, project pages, admin panel, and the new Insights page, ensuring consistent styling for components like dropdowns, toggles, and search results.

modules/server/src/main/assets · high confidence

Added Gatling load-test harness for performance testing; Added in-memory test infrastructure for core services; Added integration tests for GitHub client and search relevance; Added route tests for assets, badges, and project pages; Added test coverage for API routes and publishing endpoints; Added test for server configuration loading; Added test infrastructure and coverage for infrastructure components; Added tests for IndexConfig loading; Added tests for POM reading with distributionManagement handling; Added tests for PageParams validation; Added tests for SCM URL parsing; Added tests for core model components.

The build configuration has been updated to use Scala 3.9.0 and upgraded several key libraries, including logback-classic to 1.6.3, nscala-time, pekko-http, elastic4s, and flyway-core to 12.11.0. The project structure was reorganized into multiple modules (webclient, data, core, infra, server, template, loadtest), and the build now uses version variables (V.\*) for dependency management instead of hardcoded versions.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 43.

• Code Health 94
• Architecture 74
• Maturity 59
• Readiness 69
• Security 32
• Accessibility 38

• 300 commits — 222 feature/other, 78 fixes

• (repo) — 94 commits
• modules/server — 62 commits
• (root) — 49 commits
• modules/infra — 41 commits
• modules/template — 23 commits
• modules/core — 11 commits
• project/Elasticsearch.scala — 6 commits
• project/plugins.sbt — 5 commits
• modules/data — 2 commits
• modules/webclient — 2 commits
• scripts/PublishMissingArtifacts.scala — 2 commits
• modules/infra-it — 1 commit
• project/Deployment.scala — 1 commit
• project/build.properties — 1 commit

• fix: Admin page: card styling + fix uneven Form column alignment
• fix: Awesome Scala: fix lopsided mountain background at short header height
• fix: Awesome Scala: fix result count pluralization and hide it when empty
• fix: Buttons: restyle base + fix growing-on-load transition bug
• fix: Dependencies/dependents: fix long-name overflow + restyle badge
• fix: Dependents tab: fix gap between tabs and content box
• fix: Dropdowns: fix serif font leaking in from wrapping <h2>
• fix: Fix Formats.plural/wordPlural: "No X" now actually pluralizes X
• fix: Fix commit-activity sparkline never rendering
• fix: Fix community profile decoding when files are null
• fix: Fix pluralization on ecosystem badges and the search placeholder
• fix: Fix sbt-scalajs-crossproject version to 1.4.0
• fix: Fix scalafmt violation in ProjectPages
• fix: Footer: fix uneven GitHub/Discord icons
• fix: Header: fix sharp/kanciaste autocomplete dropdown corners
• fix: Header: revert nav buttons to ghost/outline style
• fix: Merge branch 'main' into fix/github-shared-http-client
• fix: Merge pull request #1763 from tbekas/fix/badges-latest-by-scala-version
• fix: Merge pull request #1764 from tbekas/fix/search-query-parse-crash
• fix: Merge pull request #1768 from tbekas/fix/search-aggregation-parse-crash
• …and 280 more

• 0 containers · 1 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

scalacenter/scaladex was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.