broadinstitute/cromwell
49.7
Weak · 14 November 2026
96.2k
lines of production code
Scala
primary language
5
measurements over time
What this system is
Features
Add Centaur configuration templates for JES and local Cromwell backends
New configuration templates have been added to the Centaur test suite to support running Cromwell against the JES (Google Life Sciences) and local backends. The \jes-compose.yml.ctmpl\ and \local-compose.yml.ctmpl\ files define Docker Compose services for MySQL and Cromwell, including specific JVM options and backend settings (e.g., \backend.defaultBackend=JES\ vs \local\). Additionally, \cromwell-account.pem.ctmpl\ provides a template for injecting the service account private key via environment variables, enabling authenticated access to Google Cloud services during testing.
centaur/src/main/config · high confidence
Add Docker registry flows for AWS ECR, Docker Hub, Google, and Quay
New registry flow implementations have been added to handle Docker image resolution for AWS ECR (including public ECR), Docker Hub, Google Container/Artifact Registry, and Quay. These changes enable Cromwell to authenticate and fetch manifests from these specific registries using their respective protocols and SDKs.
dockerHashing/src/main/scala/cromwell/docker/registryv2/flows · high confidence
Add FTP filesystem support to Cloud NIO
Introduces a new FTP implementation for the Cloud NIO abstraction, enabling users to read, write, list, and manage files on FTP servers. This change adds the \FtpCloudNioFileSystemProvider\ and supporting classes to handle connection pooling, authentication (including anonymous and account-based login), and file operations like copying and directory creation. It also includes configuration options for connection modes (active/passive) and caching strategies to manage resources efficiently.
cloud-nio/cloud-nio-impl-ftp, filesystems/ftp · high confidence
Add Google Cloud PAPIv2 resource monitoring and file transfer utilities
This change introduces a new Docker-based Stackdriver task monitor for Google PAPIv2 workflows, allowing users to track real-time CPU, memory, disk utilization, and IOPS metrics via the \monitoring\_image\ workflow option. It also adds a \gcs\_transfer.sh\ script to handle file localization and delocalization with support for requester-pays buckets and parallel composite uploads.
supportedBackends/google/pipelines/common · high confidence
Add Simpletons conversion utility for database entries
A new \Simpletons\ object has been added to the \cromwell\ package to handle the conversion of database simpleton entries (\CallCachingSimpletonEntry\ and \JobStoreSimpletonEntry\) into \WomValueSimpleton\ instances. This utility explicitly maps WDL/WOM types (such as String, Int, Float, Long, Boolean, File, and Directory) to their corresponding WomValue implementations, ensuring that File and Directory types are correctly instantiated as \WomSingleFile\ and \WomUnlistedDirectory\ respectively during the conversion process.
engine/src/main/scala/cromwell · high confidence
Add WDL 1.0 language support via Draft 3 factory
Introduces the WdlDraft3LanguageFactory, enabling the Cromwell engine to parse, validate, and execute workflows written in WDL version 1.0. This new factory handles the full lifecycle of WDL 1.0 files, including import resolution, AST parsing, and conversion to the WomExecutable, while leveraging the existing parser cache for performance.
languageFactories/wdl-draft3/src/main · high confidence
Add WDL 1.1 language factory
The WDL 1.1 language factory (WdlBiscayneLanguageFactory) is now available, enabling the system to parse and execute workflows declaring 'version 1.1'. This factory handles validation, bundle creation, and executable generation for this specific WDL version, distinct from other supported versions like 'Cascades' or 'development'.
languageFactories/wdl-biscayne · high confidence
Add WDL transformation utilities for database migration
A new \WdlTransformation\ object has been added to the database migration module to handle WDL-specific data transformations. This includes an \inflate\ method that decompresses and decodes Base64-encoded GZIP strings (falling back to the raw value if decompression fails) and a \coerceStringToWdl\ method that converts string representations into appropriate WDL/WOM types based on the target type.
database/migration/src/main/scala/cromwell/database/migration · high confidence
Added Cromwell server management script
A new shell script, \scripts/server/cromwell.server\, has been added to provide a convenient interface for managing the Cromwell server lifecycle. Users can now easily start, stop, check the status of, view logs for, or edit the configuration of the Cromwell server using simple command-line arguments (e.g., \cromwell.server start\).
scripts/server · high confidence
Added JSON serialization support for WomValue types
A new WomValueJsonFormatter has been introduced to enable the serialization and deserialization of WDL workflow values (such as strings, integers, files, arrays, maps, and objects) to and from JSON. This allows WDL values to be easily converted for storage or transmission, with specific handling for optional values and nested structures.
core/src/main/scala/cromwell/util/JsonFormatting · high confidence
Added WDL workflow definitions for process execution
New WDL (Workflow Description Language) files, specifically 3step.wdl and sample.wdl, have been added to the engine resources. These files define reusable workflow components: 3step.wdl outlines a three-step process involving process listing, pattern matching, and word counting, while sample.wdl provides a scatter-gather pattern for grepping and counting lines across multiple input files. These additions enable the engine to execute these specific workflow definitions.
engine/src/main/resources · high confidence
Added script to generate reference disk manifests
A new shell script, create\_images.sh, has been added to the scripts/reference\_disks directory. This tool automates the creation of HOCON manifest files for reference images by reading an input TSV, verifying existing resources (images, disks, mounts), and computing CRC32C hashes for cloud objects to ensure data integrity in the generated manifests.
_scripts/reference\disks · high confidence
Automated Java client generation and publishing workflow
This change introduces a new set of scripts to automate the build and distribution of the Cromwell Java client. The \gen\_java\_client.sh\ script uses the OpenAPI Generator to create the client code from the Swagger definition, specifically patching the API spec to include OAuth2 security definitions and replacing 'file' types with 'string' for better usability. It also configures the generated client to include a custom User-Agent header containing the current Cromwell commit hash. Additionally, \publish-client.sh\ handles publishing the generated library to Artifactory, distinguishing between snapshot and release builds based on the CI branch, while \make\_pr\_clone\_branch.sh\ provides a utility for creating isolated branches to test pull request changes in Travis CI.
scripts · high confidence
Centaur test runner configuration and AWS credential support
The Centaur test runner now includes explicit configuration files for managing test behavior and cloud credentials. A new \application.conf\ sets Akka HTTP connection pool limits (max 1024 open requests, 20 max connections, 10 min connections) to optimize network performance during testing. A new \reference.conf\ defines core Centaur settings, including Cromwell interaction modes (URL or JAR), timeouts for workflow progress and metadata consistency, and Google Cloud configuration defaults such as authentication schemes and endpoint URLs. Additionally, a new \centaur\_aws\_credentials.conf\ file allows AWS access keys, secret keys, and region to be configured via environment variables, enabling the test suite to run against AWS backends.
centaur/src/main/resources · high confidence
Enable local Docker image hashing via CLI
Cromwell now supports looking up and pulling Docker images using the local Docker CLI instead of relying solely on the Docker REST API. This change introduces a new \DockerCliFlow\ and \DockerCliClient\ in the \dockerHashing\ module that execute \docker images\ and \docker pull\ commands to resolve image digests. The implementation includes a 5-second timeout for the initial lookup to mitigate known Docker CLI hanging issues and automatically pulls an image if it is not found locally before attempting to resolve its hash. This provides a fallback mechanism for environments where the Docker API might be unavailable or unreliable.
dockerHashing/src/main/scala/cromwell/docker/local · high confidence
Engine-level instrumentation for HTTP, I/O, jobs, and workflows
The engine now includes dedicated instrumentation modules (HttpInstrumentation, IoInstrumentation, JobInstrumentation, WorkflowInstrumentation) that expose detailed metrics to StatsD. HTTP requests are tracked by normalized path, method, and status code; I/O operations are measured by filesystem type (local vs. GCS) and outcome (success, failure, retry); job execution times are recorded per state (succeeded, aborted, failed); and workflow lifecycle metrics (queued, running, on-hold, aborting) are exposed as gauges and counters, enabling users to monitor engine performance and resource utilization.
engine/src/main/scala/cromwell/engine/instrumentation · high confidence
GCS filesystem introduces batched I/O command execution
The GCS filesystem implementation now uses a new batched I/O architecture for operations such as size, delete, copy, hash, touch, exists, and isDirectory. This change introduces \GcsBatchCommandBuilder\ and a suite of \GcsBatchIoCommand\ classes (e.g., \GcsBatchCopyCommand\, \GcsBatchSizeCommand\) that wrap Google Cloud Storage API calls into batchable units. This enables more efficient handling of GCS requests by allowing them to be grouped and executed in batches, improving performance for workflows involving many file operations.
filesystems/gcs/src/main/scala/cromwell/filesystems/gcs/batch · high confidence
Initial Docker entrypoint and build scripts
Added \docker/install.sh\ to build the Cromwell JAR using SBT and \docker/run.sh\ to start the server. The run script now executes the JAR using the \JAVA\_OPTS\ environment variable, allowing users to configure JVM options externally rather than relying on hard-coded settings.
docker · high confidence
Initial Draft 3 WDL parser and transformation pipeline
This change introduces the core infrastructure for parsing and transforming WDL Draft 3 workflows. It adds a generated Java parser (WdlParser) and Scala wrappers that convert raw WDL source code into an Abstract Syntax Tree (AST). The diff also implements the transformation chain from this AST to the WDL Object Model (WDLOM), and subsequently links and evaluates expressions (handling types, file dependencies, and values) before converting the WDLOM into the Workflow Object Model (WOM) used for execution. This establishes the foundational parsing and linking logic for Draft 3 workflows.
wdl/transforms/draft3 · high confidence
Initial S3 storage client implementation
Added S3Storage.scala, which provides factory methods to create AWS S3 clients. The new code supports configuring S3 acceleration, dual-stack networking, and path-style access via configuration, and allows specifying an optional AWS region and credentials provider for client instantiation.
cloudSupport/src/main/scala/cromwell/cloudsupport/aws · high confidence
Initial build configuration for Java code generator
The Java code generator module now includes its own build infrastructure, introducing sbt 1.5.8 and configuration files for versioning, publishing, and Artifactory integration. This enables the module to be built independently and published to the Broad Institute's Artifactory repositories, with version strings automatically appended with the git commit hash.
_codegen\java · high confidence
Initial implementation of the AWS Batch backend
Cromwell now supports executing workflows on AWS Batch. This change introduces the \AwsBatchAsyncBackendJobExecutionActor\ and its supporting lifecycle actors, configuration, and job definition logic, enabling users to submit and manage jobs via the AWS Batch service. The implementation includes support for S3 and local filesystems, EFS/FSx mounting, Docker mirroring, and private Docker token storage via AWS Secrets Manager.
supportedBackends/aws · high confidence
Initial release of the Cromwell API client
Introduces the CromwellClient, a new Scala library for interacting with the Cromwell workflow engine. This client provides methods to submit workflows (including batch submissions), query workflow status, retrieve metadata, outputs, logs, and costs, manage workflow labels, and check the engine version. It supports HTTP authentication via optional credentials and handles JSON marshalling/unmarshalling for Cromwell API responses.
cromwellApiClient/src/main/scala/cromwell/api · high confidence
Introduce Biscayne WDL parser and engine functions
This change introduces the Biscayne WDL parser (generated via Hermes) and its associated AST-to-model transformation pipeline, enabling support for WDL 2.0/development features. Users can now use new engine functions including \keys\, \as\_map\, \as\_pairs\, \collect\_by\_key\, \min\, \max\, \sep\, \sub\ (POSIX-flavored), \suffix\, \quote\, \squote\, and \unzip\. The parser also supports struct literals, task input passthrough syntax (e.g., \foo\ instead of \foo=foo\), and the \None\ literal. Additionally, the legacy \read\_object\, \read\_objects\, \write\_object\, and \write\_objects\ functions are removed in favor of \read\_json\ and \write\_json\.
wdl/transforms/biscayne · high confidence
Introduce CromIAM health monitoring subsystem
Added a new health monitoring system for CromIAM that periodically checks the status of dependent services (Cromwell and Sam). The implementation includes a \StatusService\ which configures and schedules a \HealthMonitor\ actor to poll these subsystems at a configurable interval, and a \StatusCheckedSubsystem\ trait that provides a reusable method for performing HTTP status checks against specific endpoints. This allows users to query the overall health status of the CromIAM infrastructure via a dedicated status endpoint.
CromIAM/src/main/scala/cromiam/server/status · high confidence
Introduce configurable language factory registry and import resolution infrastructure
This change introduces the core infrastructure for registering and resolving WDL language versions via configuration, replacing hardcoded logic with a dynamic registry in CromwellLanguages. It adds a robust import resolution system (ImportResolver) supporting local filesystem, HTTP, and zipped imports, including host allowlisting and authentication providers. Additionally, it implements a parser cache (ParserCache) to optimize repeated workflow parsing by hashing source, URL, root, and import resolvers, and provides utility functions for validating WDL namespaces and resolving workflow sources.
languageFactories/language-factory-core/src/main/scala/cromwell/languages/util · high confidence
Introduces batched S3 I/O command implementation
Adds new Scala classes in the S3 filesystem module to support batched I/O operations. The \S3BatchCommandBuilder\ constructs specific command objects for actions like size checks, deletions, copies, hashing, and existence checks, while \S3BatchIoCommand\ defines the command structures that map AWS SDK responses (such as \HeadObjectResponse\ for size/hash/exists and \CopyObjectResponse\ for copies) to Cromwell's internal I/O model. This enables more efficient, batched communication with S3 for these specific operations.
filesystems/s3/src/main/scala/cromwell/filesystems/s3/batch · high confidence
Introduction of GCS storage configuration and client builder utilities
A new GcsStorage object has been added to the cloud support library to centralize Google Cloud Storage client initialization. This component provides factory methods for constructing Storage instances with configurable retry settings and credentials, and defines a default CloudStorageConfiguration that allows empty path components, strips prefix slashes, uses pseudo-directories, and respects a configurable upload buffer size.
cloudSupport/src/main/scala/cromwell/cloudsupport/gcp · high confidence
Introduction of WDL Draft 2 Language Factory with HTTP Import Resolution
A new WDL Draft 2 language factory has been added, enabling the execution of workflows using the Draft 2 specification. This implementation introduces support for HTTP-based import resolution, allowing workflows to import external dependencies via HTTP URLs. It also includes validation logic that enforces a maximum length of 100 characters for workflow names and integrates with the existing parser caching mechanism for improved performance.
languageFactories/wdl-draft2/src/main · high confidence
Introduction of engine core infrastructure and failure mode handling
The engine module now includes foundational components for workflow execution, specifically introducing the \WorkflowFailureMode\ trait with \ContinueWhilePossible\ and \NoNewCalls\ options to control how workflows proceed after call failures. This change adds the \EngineWorkflowDescriptor\ to manage workflow state and path builders, \EngineFilesystems\ for configuring local and pluggable filesystems, and \EngineIoFunctions\ for I/O operations. Additionally, a \CromwellTerminator\ trait is introduced to handle coordinated shutdowns, and utility extensions are added to the package object for handling fully qualified names and job output maps.
engine/src/main/scala/cromwell/engine · high confidence
Introduction of standardized instrumentation keys and prefixes
Cromwell now provides a centralized set of constants for metric instrumentation, defining standard keys for workflow outcomes (success, aborted, failure, retry) and prefixes for different system components (API, backend, job, I/O, services, workflow, and CromIAM). This allows services to communicate metrics consistently and ensures that instrumentation data is properly categorized by source.
core/src/main/scala/cromwell/core/instrumentation · high confidence
New API client model types and JSON serialization support
The Cromwell API client now includes a comprehensive set of model classes and JSON formatters for interacting with the Cromwell server. Users can now parse and construct data for workflow submissions (including single and batch modes with labels and zipped imports), workflow status tracking, query results, call cache diffs, workflow descriptions, and cost information. The client also standardizes time formatting to UTC with millisecond precision and provides utilities for handling HTTP responses via Cats Effect IO.
cromwellApiClient/src/main/scala/cromwell/api/model · high confidence
New AWS integration test cases for Mutect2 and CNV-Pair workflows
Added new integration test configurations for the Mutect2 and CNV-Pair somatic workflows to support execution on AWS infrastructure. This includes new WDL workflow definitions, input JSON files pointing to S3 data sources (e.g., \s3://cromwell-integration-tests/...\), and runtime options for Google Cloud Platform (for CNV-Pair). The Mutect2 test specifically configures the workflow to run with Oncotator annotation and scatter over 50 intervals, while the CNV-Pair test covers matched-pair analysis with optional Oncotator support.
centaur/src/main/resources/integrationTestCases/Somatic/Mutect2 · high confidence
New Centaur integration test reporting infrastructure
The Centaur integration test framework now includes a new reporting subsystem located in \centaur/src/it/scala/centaur/reporting\. This change introduces a pluggable error reporting architecture (\ErrorReporter\, \ErrorReporters\) that supports multiple destinations, including Google Cloud Storage (\GcsReporter\) for test metadata and SLF4J (\Slf4jReporter\) for logging. It also adds support for querying the Cromwell database directly from tests (\ErrorReporterCromwellDatabase\) to retrieve job key-value and metadata entries, and provides utilities for aggregating exceptions (\AggregatedIo\) and capturing CI environment details (\CiEnvironment\).
centaur/src/it/scala/centaur/reporting · high confidence
New Cromwell reference disk manifest creator tool
Added a new Java application (CromwellRefdiskManifestCreatorApp) that scans a directory of reference files and generates a JSON manifest. The tool calculates CRC32C checksums for each file, supports parallel processing via configurable thread counts, and includes the image identifier and disk size (diskSizeGb) in the output manifest. A unit test verifies the manifest creation logic against a reference JSON file.
CromwellRefdiskManifestCreator · high confidence
New JobStore actor system for job completion tracking
The engine now includes a new JobStore subsystem (in engine/src/main/scala/cromwell/jobstore) that manages job completion data. This introduces an EmptyJobStoreActor (a no-op implementation), a JobStore trait with SQL-backed SqlJobStore for persisting results, and a set of Akka actors (JobStoreActor, JobStoreReaderActor, JobStoreWriterActor) that handle batching, throttling, and graceful shutdown of job completion writes and reads. A new JSON formatter (JobResultJsonFormatter) serializes job results, and core data types (JobStoreKey, JobResultSuccess, JobResultFailure) are defined to represent completed jobs and their outputs or failures.
engine/src/main/scala/cromwell/jobstore · high confidence
New backpressure\_report script to analyze Cromwell runner I/O backpressure
A new Python tool has been added at scripts/backpressure\_report to measure the time Cromwell runner instances spend in a high I/O state that triggers internal backpressure, which otherwise slows job starts and restarts. The script parses JSON-formatted Google Logs Explorer logs (specifically looking for 'IoActor backpressure' start and end messages), calculates the duration of these backpressure events per pod, and outputs a CSV report showing total and per-pod backpressure durations in hourly intervals. This allows users to identify which pods are contributing most to backpressure and understand its impact on job execution throughput.
_scripts/backpressure\_report, scripts/backpressure\_report/backpressure\report · high confidence
New example backend configuration files for Cromwell
The \cromwell.example.backends\ directory now includes example configuration files for a wide range of backend providers, allowing users to easily copy and customize settings for their environment. Supported backends include cloud providers (AWS Batch, Google Cloud Batch, PAPIv2, TES, TESK), local and container runtimes (Docker, Singularity, Singularity+Slurm, udocker, udocker+Slurm), and HPC workload managers (HtCondor, LSF, SGE, SLURM, TORQUE, Volcano). A comprehensive README and a central \cromwell.examples.conf\ file are also provided to guide users on how to integrate these providers into their Cromwell setup.
cromwell.example.backends · high confidence
New metadata alert types for workflow monitoring
A new sealed trait \MetadataAlert\ and its implementations \HeavyMetadataAlert\ and \MaxMetadataAlert\ have been added to the core events module. These case classes provide a structured way to report metadata-related issues for specific workflows, including the workflow ID, the current count, and (in the case of \MaxMetadataAlert\) the configured limit, enabling downstream systems to detect and react to metadata thresholds.
core/src/main/scala/cromwell/core/events · high confidence
New metadata comparison scripts for PAPI performance analysis
Added a new \metadata\_comparison\ toolset to extract, digest, and compare performance metadata from Cromwell workflows running on PAPI. The \extractor\ script fetches workflow and operation metadata from Cromwell and Google Cloud Storage, including local Cromwell checkouts and configurations. The \digester\ processes this raw metadata into structured JSON digests, calculating timing metrics (startup, docker pull, localization, user command, delocalization) and cost estimates based on machine types and disk usage. The \comparer\ script then takes two digests and produces a CSV report comparing performance and costs between two workflow runs (e.g., PAPI v1 vs v2), with options to remove call prefixes for cleaner reporting.
_scripts/metadata\_comparison/metadata\comparison · high confidence
New retry and backoff utilities for handling transient failures
The core retry module now includes new \GoogleBackoff\ and \Retry\ components. \GoogleBackoff\ provides \InitialGapBackoff\ and \SimpleExponentialBackoff\ strategies that wrap Google's \ExponentialBackOff\ for configurable retry intervals. \Retry\ introduces \withRetry\, a generic function to retry asynchronous operations with customizable backoff, transient/fatal error handling, and retry limits, and \withRetryForTransactionRollback\, which specifically handles SQL deadlock exceptions (\SQLTransactionRollbackException\) by retrying up to a specified number of times.
core/src/main/scala/cromwell/core/retry · high confidence
New runtime attribute validation framework for Cromwell backends
This change introduces a comprehensive validation system for workflow runtime attributes (such as memory, CPU, GPU requirements, container images, and retry logic) within the Cromwell backend. It replaces ad-hoc validation with a structured, type-safe approach using traits like \RuntimeAttributesValidation\ and \ValidatedRuntimeAttributesBuilder\, ensuring that attributes like \memory\, \cpu\, \gpuRequired\, \continueOnReturnCode\, and \docker\/\container\ are correctly coerced, validated against WDL specifications (including WDL 1.1 features), and consistently handled for call caching and metadata reporting.
backend/src/main/scala/cromwell/backend/validation · high confidence
New shared transform utilities for WDL 1.0 and 1.1
A new \common/transforms\ package has been introduced to provide shared transformation logic for both WDL 1.0 and 1.1. This change adds a \CheckedAtoB\ type alias based on \cats.data.Kleisli\ and a companion object containing factory methods (\fromCheck\, \fromErrorOr\) that wrap validation logic with optional context for error reporting. It also includes a \firstSuccess\ helper to attempt multiple transformation options and return the first successful result, consolidating common transform patterns previously duplicated across WDL versions.
common/src/main/scala/common/transforms · high confidence
New utility libraries for actor management and resource handling
Added several new helper classes to the core utility package to improve actor lifecycle management and code safety. GracefulShutdownHelper enables actors to cleanly shut down dependent actors before stopping themselves. PromiseActor provides a timeout-free alternative to Akka's ask pattern for synchronous-style actor communication. DatabaseUtil introduces retry logic with exponential backoff for transient database errors. StopAndLogSupervisor simplifies actor supervision by automatically stopping failed children and logging the error. TryWithResource implements a Scala version of Java's try-with-resources for safe handling of AutoCloseable resources.
core/src/main/scala/cromwell/util · high confidence
New utility libraries for retry logic, URI masking, and time formatting
This change introduces a suite of new utility classes in the common package to support safer and more consistent operations across the application. IORetry provides a stateful retry mechanism for Cats Effect IO operations, allowing callers to accumulate state and handle failures with configurable backoff strategies. UriUtil and StringUtil now include methods to mask sensitive parts of URIs (such as credentials and signatures) before logging, addressing security concerns around sensitive data exposure. TimeUtil standardizes timestamp formatting to UTC with millisecond precision, ensuring consistent time representation in logs and outputs. Additional utilities include Backoff for defining retry intervals, IntrospectableLazy for managing lazy initialization with existence checks, TryUtil for aggregating multiple Scala Try results, TerminalUtil for formatted console output, and VersionUtil for reading project version metadata from SBT-generated config files.
common/src/main/scala/common/util · high confidence
New validation library with ErrorOr, Checked, and IOChecked types
The common module now includes a new validation library providing \ErrorOr\ (based on cats \Validated\), \Checked\ (based on \Either\), and \IOChecked\ (based on \EitherT\[IO\]\) types. These types offer consistent, composable error handling with support for accumulating multiple errors, parallel evaluation of IO operations, and convenient conversion between \Try\, \Future\, and \IO\ contexts, simplifying validation logic across the application.
common/src/main/scala/common/validation · high confidence
New workflow timing diagram visualization
A new HTML-based timing diagram view is now available for workflows, rendering execution events as a Google Timeline chart. This view visualizes call start and end times, including nested subworkflows, retry attempts, and specific execution phases such as queued, starting, and overhead periods. Users can interact with the chart to expand or collapse subworkflow details, providing a clearer view of workflow execution history and performance bottlenecks.
engine/src/main/resources/workflowTimings · high confidence
Pluggable filesystem configuration system
Cromwell now supports a pluggable filesystem architecture, allowing users to configure and register custom path builder factories via the global configuration. The new CromwellFileSystems component validates filesystem entries, instantiates the specified classes, and manages singleton configurations, enabling extensibility beyond the default local filesystem without code changes.
core/src/main/scala/cromwell/core/filesystem · high confidence
Support for runtime attribute overrides and new engine functions in WDL shared transforms
This change introduces new capabilities in the WDL shared transforms layer. It adds a \transpose\ function to the engine functions, allowing users to transpose two-dimensional arrays in their workflows. Additionally, it implements runtime attribute overrides, enabling users to specify task runtime requirements (such as CPU, memory, and disk) directly in the input JSON using dot-notation keys (e.g., \task.runtime.cpu\), which are now correctly parsed and grouped into runtime objects. The update also includes logic to automatically add default outputs for scatter and conditional blocks in the WOM graph, ensuring that outputs from these control structures are exposed by default.
wdl/transforms/shared · high confidence
WDL Draft 2 workflow and task transformation to WOM
This change introduces the core transformation logic for converting WDL Draft 2 source code into the Workflow Execution Model (WOM). The new files in \wdlom2wom\ implement the builders for workflows, tasks, call nodes, scatter nodes, and conditional nodes, enabling Cromwell to parse and execute Draft 2 compliant workflows. It also includes specific handling for Draft 2 features such as aliased calls, input validation, and file read limits.
wdl/transforms/draft2 · high confidence
Architecture
Database access layer refactored into domain-specific SQL traits
The database access layer in the SQL module has been restructured to separate concerns by domain. The previous monolithic database interface has been split into distinct traits—\MetadataSqlDatabase\, \EngineSqlDatabase\, \CallCachingSqlDatabase\, \WorkflowStoreSqlDatabase\, \JobStoreSqlDatabase\, \SubWorkflowStoreSqlDatabase\, \DockerHashStoreSqlDatabase\, \JobKeyValueSqlDatabase\, and \GroupMetricsSqlDatabase\—each exposing methods specific to its area of responsibility. \EngineSqlDatabase\ now aggregates the engine-related traits, while \MetadataSqlDatabase\ handles workflow metadata, summarization, and archival status. This change also introduces new join models (\CallCachingJoin\, \JobStoreJoin\, \CallCachingDiffJoin\) and a \MetadataJobQueryValue\ filter mechanism to support more granular metadata queries.
database/sql/src/main/scala/cromwell/database/sql · high confidence
Introduce pluggable filesystem path abstraction
The core path handling logic has been refactored to support pluggable filesystems. A new \Path\ trait and \PathBuilder\ interface allow Cromwell to resolve file paths for different storage backends (such as local, GCS, or DRS) via a prioritized list of builders. This change replaces the previous direct dependency on \better.files\ and \java.nio.file\ with a unified abstraction, enabling the system to handle diverse URI schemes and backend-specific behaviors while maintaining a consistent API for file operations.
core/src/main/scala/cromwell/core/path · high confidence
New services sub-project for modular service infrastructure
The codebase has introduced a new \services\ sub-project to centralize and modularize service-related components. This change includes the creation of a \ServiceRegistryActor\ for dynamic service discovery and configuration, along with dedicated stores for engine and metadata database interactions (\EngineServicesStore\, \MetadataServicesStore\). It also adds foundational infrastructure such as \EnhancedBatchActor\ and \EnhancedThrottlerActor\ for standardized instrumentation and load control, and introduces new capabilities like a \GithubAuthVending\ service for GitHub token resolution and a \GcpCostCatalogService\ for GCP cost lookups. Also in this area: 1 commit (0 fixes) modifying backend, repository; Refactored GCP Batch backend to use a new request manager architecture; Database access layer restructured into separated Engine and Metadata components; Database access logic is split into separate Engine and Metadata services; Add GA4GH Workflow Execution Service (WES) API endpoints; and 54 more.
repository · medium confidence · unverified
Behavioural changes
1 commit (0 fixes) modifying services/src/main/scala/cromwell/services/metadata/impl/aws, services/src/main/scala/cromwell/services/metadata/impl/hybridaws
A change to existing behaviour in services/src/main/scala/cromwell/services/metadata/impl/aws, services/src/main/scala/cromwell/services/metadata/impl/hybridaws — 1 commit, 4 files: “AN-699 Merge changes from AWS fork (\#7788)” (refs \#7788).
services/src/main/scala/cromwell/services/metadata/impl/aws, services/src/main/scala/cromwell/services/metadata/impl/hybridaws · medium confidence · unverified
10 commits (0 fixes) modifying core/src/main/scala/cromwell/core/labels
A change to existing behaviour in core/src/main/scala/cromwell/core/labels — 10 commits, 1 file: “Changed Label Formatting (\#2389)”; “move cromwell”; “\[WX-1346\] Scalafmt (\#7257)”; and 5 more (refs \#2389, \#7257, \#3539).
core/src/main/scala/cromwell/core/labels · medium confidence · unverified
10 commits (0 fixes) modifying engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/callcaching
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/callcaching — 10 commits, 2 files: “CTM-489 Type annotations for implicits (\#7887)”; “Strawman 2: Revenge of the straw”; “move cromwell”; and 5 more (refs \#7887, \#1232, \#7257, \#5150).
engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/callcaching · medium confidence · unverified
10 commits (0 fixes) modifying engine/src/main/scala/cromwell/subworkflowstore
A change to existing behaviour in engine/src/main/scala/cromwell/subworkflowstore — 10 commits, 3 files: “Sub Workflows closes \#1532 \#1473 (\#1682)”; “Remove unneeded SubWorkflowStoreCompleteSuccess message”; “patvars”; and 5 more (refs \#1532, \#1473, \#1682, \#7257, \#2499, …).
engine/src/main/scala/cromwell/subworkflowstore · medium confidence · unverified
10 commits (1 fix) modifying engine/src/main/scala/cromwell/engine/io
A change to existing behaviour in engine/src/main/scala/cromwell/engine/io — 10 commits (1 fix), 3 files: “GATK SV inspired I/O backpressuring \[BT-343\] (\#6463)”; “AN-699 Merge changes from AWS fork (\#7788)”; “\[WX-1346\] Scalafmt (\#7257)”; and 7 more (refs \#6463, \#7788, \#7257, \#6683, \#4423, …).
engine/src/main/scala/cromwell/engine/io · medium confidence · unverified
10 commits (2 fixes) modifying cloud-nio/cloud-nio-spi
A change to existing behaviour in cloud-nio/cloud-nio-spi — 10 commits (2 fixs), 9 files: “Reduce likelihood of DRS hogging IoActor NIO threads \[BT-341\] (\#6439)”; “WX-1409 Java 17 (\#7342)”; “Revert "WX-1409 Java 17 (\#7342)" (\#7343)”; and 7 more (refs \#6439, \#7342, \#7343, \#7344, \#7346, …).
cloud-nio/cloud-nio-spi · medium confidence · unverified
100 commits (11 fixes) modifying womtool
A change to existing behaviour in womtool — 100 commits (11 fixs), 311 files: “Add womtool outputs to display output format”; “1.0: default workflow outputs (\#3659)”; “Regression test for comments before 'version' (\#4384)”; and 96 more (refs \#3659, \#4384, \#4154, \#3925, \#3775, …).
womtool · medium confidence · unverified
107 commits (9 fixes) modifying services/src/main/scala/cromwell/services/metadata
A change to existing behaviour in services/src/main/scala/cromwell/services/metadata — 107 commits (9 fixs), 8 files: “CTM-489 Type annotations for implicits (\#7887)”; “Query using Labels (\#2253)”; “Metadata decider logic (\#5205)”; and 102 more (refs \#7887, \#2253, \#5205, \#1243, \#2380, …).
services/src/main/scala/cromwell/services/metadata · medium confidence · unverified
11 commits (2 fixes) modifying filesystems/drs/src/main
A change to existing behaviour in filesystems/drs/src/main — 11 commits (2 fixs), 1 file: “Revert "BT-614 Remove OSS references in a few last files"”; “Add support for non-W3C/IETF CIB DRS URIs BT-4 BT-5 CROM-6642 (\#6020)”; “BT-614 Remove Alibaba/Aliyun backend (\#6785)”; and 8 more (refs \#6020, \#6785, \#6031, \#4927, \#7257, …).
filesystems/drs/src/main · medium confidence · unverified
115 commits (16 fixes) modifying database/sql/src/main/scala/cromwell/database/slick/tables
A change to existing behaviour in database/sql/src/main/scala/cromwell/database/slick/tables — 115 commits (16 fixs), 20 files: “Pick up workflows to run from hog group with lowest actively running workflows \[BW-568\] (\#6680)”; “CTM-489 Type annotations for implicits (\#7887)”; “Query using Labels (\#2253)”; and 110 more (refs \#6680, \#7887, \#2253, \#5584, \#2243, …).
database/sql/src/main/scala/cromwell/database/slick/tables · medium confidence · unverified
12 commits (0 fixes) modifying services/src/main/scala/cromwell/services/metadata/impl/deleter
A change to existing behaviour in services/src/main/scala/cromwell/services/metadata/impl/deleter — 12 commits, 2 files: “undid renaming, will consider after major changes approved”; “PR feedback”; “Emit 'rows deleted' metric on success as well as failure (\#6358)”; and 8 more (refs \#6358, \#6232, \#7257, \#6724).
services/src/main/scala/cromwell/services/metadata/impl/deleter · medium confidence · unverified
121 commits (11 fixes) modifying services/src/main/scala/cromwell/services/metadata/impl
A change to existing behaviour in services/src/main/scala/cromwell/services/metadata/impl — 121 commits (11 fixs), 6 files: “Carbonite Freeze Actor \[BA-5865\] (\#5210)”; “CTM-489 Type annotations for implicits (\#7887)”; “Graceful Server Shutdown (\#2490)”; and 115 more (refs \#5210, \#7887, \#2490, \#2253, \#5584, …).
services/src/main/scala/cromwell/services/metadata/impl · medium confidence · unverified
125 commits (12 fixes) modifying supportedBackends/sfs
A change to existing behaviour in supportedBackends/sfs — 125 commits (12 fixs), 29 files: “implement jes-style globbing for sfs traitifying common glob functions between backends separate rc of command from rc of globManipulation script toRealString to convert to GCS path”; “SingleWorkflowMode support for WDL import directories closes \#1525 (\#1652)”; “fix for issue 1737 on develop (\#1755)”; and 120 more (refs \#1525, \#1652, \#1755, \#7852, \#3697, …).
supportedBackends/sfs · medium confidence · unverified
13 commits (1 fix) modifying engine/src/main/scala/cromwell/webservice/routes/wes
A change to existing behaviour in engine/src/main/scala/cromwell/webservice/routes/wes — 13 commits (1 fix), 2 files: “Provides WES status and abort endpoints directly to the Cromwell server. (\#4425)”; “BW-1354 - Porting CBAS preliminary step (\#6837)”; “Implement the WES service-info endpoint (\#4455)”; and 10 more (refs \#4425, \#6837, \#4455, \#7257, \#4617, …).
engine/src/main/scala/cromwell/webservice/routes/wes · medium confidence · unverified
13 commits (2 fixes) modifying engine/src/main/scala/cromwell/engine/workflow/lifecycle
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/lifecycle — 13 commits (2 fixs), 3 files: “CTM-362 Fix OOMs with large workflows & final outputs copying (\#7860)”; “EJEA transition timings to statsD (\#4083)”; “WX-1716 Extract outputs path logic to shared location (\#7479)”; and 8 more (refs \#7860, \#4083, \#7479, \#7509, \#1282, …).
engine/src/main/scala/cromwell/engine/workflow/lifecycle · medium confidence · unverified
139 commits (12 fixes) modifying wom
A change to existing behaviour in wom — 139 commits (12 fixs), 185 files: “CTM-489 Type annotations for implicits (\#7887)”; “Handle globs in cwl output expressions”; “PAPI Directories and Literals (\#3697)”; and 134 more (refs \#7887, \#3697, \#4938, \#5378, \#3281, …).
wom · medium confidence · unverified
14 commits (1 fix) modifying services/src/main/scala/cromwell/services/instrumentation/impl/statsd
A change to existing behaviour in services/src/main/scala/cromwell/services/instrumentation/impl/statsd — 14 commits (1 fix), 3 files: “Increase gauge processing throughput (\#3404)”; “Add StatsD support (\#2587)”; “Load Control (\#3366)”; and 9 more (refs \#3404, \#2587, \#3366, \#3294, \#5010, …).
services/src/main/scala/cromwell/services/instrumentation/impl/statsd · medium confidence · unverified
14 commits (1 fix) modifying services/src/main/scala/cromwell/services/womtool
A change to existing behaviour in services/src/main/scala/cromwell/services/womtool — 14 commits (1 fix), 3 files: “\womtool/describe\ endpoint should populate \submittedDescriptorType\ for invalid workflows \[BA-6084\]”; “Revert \[CTM-336\] Add zip imports to Describe endpoint”; “\[CTM-336\] Add zip imports to Describe endpoint (\#7845)”; and 11 more (refs \#7845, \#4940, \#4571, \#7365, \#5098, …).
services/src/main/scala/cromwell/services/womtool · medium confidence · unverified
16 commits (1 fix) modifying core/src/main/scala/cromwell/core/simpleton
A change to existing behaviour in core/src/main/scala/cromwell/core/simpleton — 16 commits (1 fix), 1 file: “WomTypes part II”; “InputRecordSchema and Command line sorting (\#3112)”; “Simpleton and Metadata for CWL Files / Directories (\#3139)”; and 11 more (refs \#3112, \#3139, \#7257, \#6724, \#3505, …).
core/src/main/scala/cromwell/core/simpleton · medium confidence · unverified
17 commits (1 fix) modifying centaur/src/main/scala/centaur
A change to existing behaviour in centaur/src/main/scala/centaur — 17 commits (1 fix), 5 files: “Give Centaur-managed Cromwell more time to restart and a custom exit code”; “Retry call caching centaur tests BT-271 (\#6344)”; “CI updates for Travis/Jenkins/Sentry. - As sentry may drop some metadata, print the metadata to slf4j. - For centaur-restarting-cromwell, remember if cromwell was alive, and log more of the connection status. - Pass more jenkins variables through docker. - Increase papi v2 cwl conformance test timeout due to problematic test 55. - Use pr branch name during pr builds”; and 14 more (refs \#6344, \#7257, \#5738, \#6724, \#3583, …).
centaur/src/main/scala/centaur · medium confidence · unverified
17 commits (1 fix) modifying engine/src/main/scala/cromwell/engine/io/gcs
A change to existing behaviour in engine/src/main/scala/cromwell/engine/io/gcs — 17 commits (1 fix), 2 files: “Add StatsD support (\#2587)”; “GATK SV inspired I/O backpressuring \[BT-343\] (\#6463)”; “Fix warning on startup \[no JIRA\] (\#5570)”; and 12 more (refs \#2587, \#6463, \#5570, \#6007, \#3492, …).
engine/src/main/scala/cromwell/engine/io/gcs · medium confidence · unverified
17 commits (1 fix) modifying services/src/main/scala/cromwell/services/keyvalue
A change to existing behaviour in services/src/main/scala/cromwell/services/keyvalue — 17 commits (1 fix), 4 files: “Graceful Server Shutdown (\#2490)”; “Load Control (\#3366)”; “Throttle kv store an job store reads (\#3327)”; and 12 more (refs \#2490, \#3366, \#3327, \#3495, \#1255, …).
services/src/main/scala/cromwell/services/keyvalue · medium confidence · unverified
17 commits (2 fixes) modifying engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/keys
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/keys — 17 commits (2 fixs), 6 files: “Evaluate task call input expressions using the backend IoFunctionSet”; “Scatters Seagull (\#2751)”; “WomTypes part II”; and 12 more (refs \#2751, \#3628, \#4483, \#3255, \#3045, …).
engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/keys · medium confidence · unverified
18 commits (2 fixes) modifying services/src/main/scala/cromwell/services/keyvalue/impl
A change to existing behaviour in services/src/main/scala/cromwell/services/keyvalue/impl — 18 commits (2 fixs), 4 files: “CTM-489 Type annotations for implicits (\#7887)”; “Backend Store performing recovery closes \#751 (\#1241)”; “Load Control (\#3366)”; and 13 more (refs \#7887, \#751, \#1241, \#3366, \#1255, …).
services/src/main/scala/cromwell/services/keyvalue/impl · medium confidence · unverified
18 commits (2 fixes) modifying services/src/main/scala/cromwell/services/metadata/impl/builder
A change to existing behaviour in services/src/main/scala/cromwell/services/metadata/impl/builder — 18 commits (2 fixs), 2 files: “Testing correctness of post-archival metadata retrieval \[BA-6244\] (\#5436)”; “Return metadataSource in metadata response \[BA-6242\] (\#5419)”; “\[AN-596\] Use last VmStart/EndTime in cost calculations (\#7767)”; and 15 more (refs \#5436, \#5419, \#7767, \#6234, \#5430, …).
services/src/main/scala/cromwell/services/metadata/impl/builder · medium confidence · unverified
2 commits (0 fixes) modifying common/src/main/scala/common/exception
A change to existing behaviour in common/src/main/scala/common/exception — 2 commits, 1 file: “\[WX-1346\] Scalafmt (\#7257)”; “DRS filesystem: Read contents support (\#4531)” (refs \#7257, \#4531).
common/src/main/scala/common/exception · medium confidence · unverified
2 commits (0 fixes) modifying filesystems/http
A change to existing behaviour in filesystems/http — 2 commits, 1 file: “Support HTTP URL workflow inputs for SFS and PAPI v2 backends. (\#3933)”; “\[WX-1346\] Scalafmt (\#7257)” (refs \#3933, \#7257).
filesystems/http · medium confidence · unverified
20 commits (0 fixes) modifying server/src/main/resources
A change to existing behaviour in server/src/main/resources — 20 commits, 3 files: “\[WX-1568\] Bump Akka max-response-reason-length limit (\#7406)”; “Downgrade WorkflowFailedResponse Evennt log level to INFO”; “Updates for TESK demo (\#4168)”; and 17 more (refs \#7406, \#4168, \#7886, \#3620, \#7157, …).
server/src/main/resources · medium confidence · unverified
21 commits (3 fixes) modifying engine/src/main/scala/cromwell/engine/workflow/lifecycle/finalization
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/lifecycle/finalization — 21 commits (3 fixs), 2 files: “copy all workflow output files (\#3298)”; “CTM-489 Type annotations for implicits (\#7887)”; “plug workflow log filehandle leaks”; and 18 more (refs \#3298, \#7887, \#7479, \#7483, \#7565, …).
engine/src/main/scala/cromwell/engine/workflow/lifecycle/finalization · medium confidence · unverified
24 commits (1 fix) modifying services/src/main/scala/cromwell/services/metadata/impl/archiver
A change to existing behaviour in services/src/main/scala/cromwell/services/metadata/impl/archiver — 24 commits (1 fix), 2 files: “get root workflow id, make it first part of path”; “cleanup”; “Archiving scale fixes: Pin fetchSize for DB stream, better metric reporting \[BW-610\] (\#6314)”; and 20 more (refs \#6314, \#6318, \#6232, \#6303, \#6234, …).
services/src/main/scala/cromwell/services/metadata/impl/archiver · medium confidence · unverified
24 commits (3 fixes) modifying services/src/main/scala/cromwell/services/instrumentation
A change to existing behaviour in services/src/main/scala/cromwell/services/instrumentation — 24 commits (3 fixs), 5 files: “CTM-489 Type annotations for implicits (\#7887)”; “Add StatsD support (\#2587)”; “Load Control (\#3366)”; and 19 more (refs \#7887, \#2587, \#3366, \#5521, \#3327, …).
services/src/main/scala/cromwell/services/instrumentation · medium confidence · unverified
25 commits (1 fix) modifying filesystems/gcs/src/main/scala/cromwell/filesystems/gcs
A change to existing behaviour in filesystems/gcs/src/main/scala/cromwell/filesystems/gcs — 25 commits (1 fix), 4 files: “extract status code from baseservice exceptions (\#3506)”; “Add StatsD support (\#2587)”; “Google Auth Updates - Replaced to-be-deprecated Credential (no 's') with Adapter around Credentials - Removed dupe credentials adapting from PipelinesApiFactoryInterface - Move service specific scopes (KMS, Genomics) out of GoogleAuthMode - Changed credential creation methods to take scala collections”; and 20 more (refs \#3506, \#2587, \#5341, \#2277, \#1519, …).
filesystems/gcs/src/main/scala/cromwell/filesystems/gcs · medium confidence · unverified
25 commits (2 fixes) modifying CromIAM/src/main/scala/cromiam/webservice
A change to existing behaviour in CromIAM/src/main/scala/cromiam/webservice — 25 commits (2 fixs), 7 files: “Pass womtool endpoint through CromIAM (\#4753)”; “Support for a Cromwell abort server. (\#4263)”; “Fix Swagger auth (\#5383) \[BA-6185\]”; and 22 more (refs \#4753, \#4263, \#5383, \#6684, \#3485, …).
CromIAM/src/main/scala/cromiam/webservice · medium confidence · unverified
26 commits (0 fixes) modifying wdl/model/draft3
A change to existing behaviour in wdl/model/draft3 — 26 commits, 16 files: “Adding line numbers to workflow definitions (\#4938)”; “Call element WDLOM without inputs or outputs to WOM (\#3452)”; “WX-1461 Remove \womtool upgrade\ command (\#7382)”; and 23 more (refs \#4938, \#3452, \#7382, \#7402, \#3388, …).
wdl/model/draft3 · medium confidence · unverified
3 commits (0 fixes) modifying filesystems/sra
A change to existing behaviour in filesystems/sra — 3 commits, 2 files: “Add NCBI Sequence Read Archive support via fusera (\#3950)”; “\[WX-1346\] Scalafmt (\#7257)”; “Scala Steward PRs 5680 - 5692 \[BA-6559\] (\#5751)” (refs \#3950, \#7257, \#5751).
filesystems/sra · medium confidence · unverified
3 commits (0 fixes) modifying services/src/main/scala/cromwell/services/healthmonitor
A change to existing behaviour in services/src/main/scala/cromwell/services/healthmonitor — 3 commits, 1 file: “\[WX-1346\] Scalafmt (\#7257)”; “Scala 2.13 \[CROM-6036\] (\#6724)”; “Allow finer-grained customization of the status endpoint (\#4807)” (refs \#7257, \#6724, \#4807).
services/src/main/scala/cromwell/services/healthmonitor · medium confidence · unverified
3 commits (0 fixes) modifying services/src/main/scala/cromwell/services/instrumentation/impl/selectivetsv
A change to existing behaviour in services/src/main/scala/cromwell/services/instrumentation/impl/selectivetsv — 3 commits, 1 file: “WX-772 Update Scala to 2.13.9 (\#6928)”; “\[WX-1346\] Scalafmt (\#7257)”; “Remove waiting for queue space status (and add testing via Dummy backend) \[BW-387\] (\#6047)” (refs \#6928, \#7257, \#6047).
services/src/main/scala/cromwell/services/instrumentation/impl/selectivetsv · medium confidence · unverified
3 commits (1 fix) modifying engine/src/main/scala/cromwell/engine/io/nio
A change to existing behaviour in engine/src/main/scala/cromwell/engine/io/nio — 3 commits (1 fix), 1 file: “AN-568 Remove unused Azure-related code (\#7761)”; “WX-1566 Fix Morgan's call cache file hash CPU thrash Cromwell crash (\#7419)”; “AN-380 Make call cache hashing strategy configurable per filesystem and backend (\#7683)” (refs \#7761, \#7419, \#7683).
engine/src/main/scala/cromwell/engine/io/nio · medium confidence · unverified
3 commits (1 fix) modifying services/src/main/scala/cromwell/services/cost
A change to existing behaviour in services/src/main/scala/cromwell/services/cost — 3 commits (1 fix), 2 files: “AN-224 Fix vmCostPerHour being 10x what it should be (\#7576)”; “\[WX-1782\] Cost Catalog Fetching (\#7507)”; “\[WX-1792\] Emit Cost Per Hour (\#7550)” (refs \#7576, \#7507, \#7550).
services/src/main/scala/cromwell/services/cost · medium confidence · unverified
30 commits (7 fixes) modifying server/src/main/scala
A change to existing behaviour in server/src/main/scala — 30 commits (7 fixs), 2 files: “WX-1122 Enable Azure ApplicationInsights (\#7143)”; “Fixed bug with actor system being created and not stopped if exception occurs during validation of command line arguments \[BA-4061\]”; “Silence error messages on shutdown (\#3620)”; and 27 more (refs \#7143, \#3620, \#7157, \#5259, \#3890, …).
server/src/main/scala · medium confidence · unverified
31 commits (0 fixes) modifying core/src/main/scala/cromwell/core/io
A change to existing behaviour in core/src/main/scala/cromwell/core/io — 31 commits, 10 files: “When an exception/timeout occurs during asyncHashing, report it as a failure”; “Call-by-name to avoid instantiating default commands if unused (\#5991)”; “apply exponential backoff when retrying from backpressure IO (\#4043)”; and 26 more (refs \#5991, \#4043, \#2587, \#2363, \#2347, …).
core/src/main/scala/cromwell/core/io · medium confidence · unverified
31 commits (1 fix) modifying engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution — 31 commits (1 fix), 1 file: “Limit number of queued jobs per workflow (\#3356)”; “WDL projects refactor part I: Separate WDL model from WOM transforms (\#3247)”; “Sub Workflows closes \#1532 \#1473 (\#1682)”; and 26 more (refs \#3356, \#3247, \#1532, \#1473, \#1682, …).
engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution · medium confidence · unverified
31 commits (3 fixes) modifying engine/src/main/scala/cromwell/webservice/routes
A change to existing behaviour in engine/src/main/scala/cromwell/webservice/routes — 31 commits (3 fixs), 2 files: “CTM-489 Upgrade logback et al. (\#7886)”; “Testing correctness of post-archival metadata retrieval \[BA-6244\] (\#5436)”; “Return ArchivedAndDeleted status for metadata requests for workflow whose metadata is deleted \[BW-565\] (\#6291)”; and 28 more (refs \#7886, \#5436, \#6291, \#7845, \#6365, …).
engine/src/main/scala/cromwell/webservice/routes · medium confidence · unverified
34 commits (2 fixes) modifying services/src/main/scala/cromwell/services
A change to existing behaviour in services/src/main/scala/cromwell/services — 34 commits (2 fixs), 9 files: “Now setting config directly in the \database\ stanza. Deprecated \database.config\. Database migration now just uses the key \database.migration\ for configuration settings. Moved database migration defaults into the reference”; “Graceful Server Shutdown (\#2490)”; “ServiceRegistryActor tests. Closes \#1047”; and 29 more (refs \#2490, \#1047, \#7253, \#2587, \#3366, …).
services/src/main/scala/cromwell/services · medium confidence · unverified
35 commits (3 fixes) modifying wdl/model/draft2
A change to existing behaviour in wdl/model/draft2 — 35 commits (3 fixs), 71 files: “1.0: default workflow outputs (\#3659)”; “Wom Coproduct Type (\#3281)”; “WDL projects refactor part I: Separate WDL model from WOM transforms (\#3247)”; and 32 more (refs \#3659, \#3281, \#3247, \#3580, \#3669, …).
wdl/model/draft2 · medium confidence · unverified
37 commits (1 fix) modifying engine/src/main/scala/cromwell/engine/workflow/workflowstore
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/workflowstore — 37 commits (1 fix), 5 files: “SingleWorkflowMode support for WDL import directories closes \#1525 (\#1652)”; “Add StatsD support (\#2587)”; “Synchronize access to workflow store”; and 32 more (refs \#1525, \#1652, \#2587, \#4071, \#4401, …).
engine/src/main/scala/cromwell/engine/workflow/workflowstore · medium confidence · unverified
39 commits (3 fixes) modifying cloud-nio/cloud-nio-impl-drs/src/main
A change to existing behaviour in cloud-nio/cloud-nio-impl-drs/src/main — 39 commits (3 fixs), 5 files: “Adding tests”; “Reduce likelihood of DRS hogging IoActor NIO threads \[BT-341\] (\#6439)”; “Retry retrieving DOS/DRS metadata on HTTP 429/5xx responses BA-6647 (\#5964)”; and 36 more (refs \#6439, \#5964, \#5819, \#6020, \#6351, …).
cloud-nio/cloud-nio-impl-drs/src/main · medium confidence · unverified
4 commits (0 fixes) modifying backend/src/main/scala/cromwell/backend/dummy
A change to existing behaviour in backend/src/main/scala/cromwell/backend/dummy — 4 commits, 3 files: “ID-1276 Introduce Bard Service for sending metrics (\#7434)”; “\[WX-1346\] Scalafmt (\#7257)”; “Scala 2.13 \[CROM-6036\] (\#6724)”; and 1 more (refs \#7434, \#7257, \#6724, \#6047).
backend/src/main/scala/cromwell/backend/dummy · medium confidence · unverified
4 commits (0 fixes) modifying services/src/main/scala/cromwell/services/auth
A change to existing behaviour in services/src/main/scala/cromwell/services/auth — 4 commits, 5 files: “AN-568 Remove unused Azure-related code (\#7761)”; “WM-2461\] Add support for running private workflows on Azure (\#7373)”; “WM-2454: Private GitHub support on describe api (\#7365)”; and 1 more (refs \#7761, \#7373, \#7365, \#7392).
services/src/main/scala/cromwell/services/auth · medium confidence · unverified
41 commits (2 fixes) modifying backend/src/main/scala/cromwell/backend/standard
A change to existing behaviour in backend/src/main/scala/cromwell/backend/standard — 41 commits (2 fixs), 7 files: “PAPI Directories and Literals (\#3697)”; “Graceful Server Shutdown (\#2490)”; “Moved globbing behavior to standard. Standard Backend has even more default implementations, and a few breaking changes. Moved some code for testing JES out of main. Increased the timeout for the ask inside SingleWorkflowRunnerActorSpec. Bumped apache http client to the latest version”; and 36 more (refs \#3697, \#2490, \#6928, \#2277, \#3596, …).
backend/src/main/scala/cromwell/backend/standard · medium confidence · unverified
5 commits (0 fixes) modifying filesystems/s3/src/main/scala/cromwell/filesystems/s3
A change to existing behaviour in filesystems/s3/src/main/scala/cromwell/filesystems/s3 — 5 commits, 1 file: “Initial implementation of AWS Backend”; “Bump libraries and run centaur bcs”; “\[WX-1346\] Scalafmt (\#7257)”; and 2 more (refs \#7257, \#4228).
filesystems/s3/src/main/scala/cromwell/filesystems/s3 · medium confidence · unverified
5 commits (1 fix) modifying cromwell-refdisk-manifest-creator
A change to existing behaviour in cromwell-refdisk-manifest-creator — 5 commits (1 fix), 2 files: “Bump libraries and run centaur bcs”; “AN-699 Merge changes from AWS fork (\#7788)”; “AWS backend reworked (Issue \#5004) (\#5468)”; and 2 more (refs \#7788, \#5004, \#5468, \#4858).
cromwell-refdisk-manifest-creator · medium confidence · unverified
60 commits (0 fixes) modifying core/src/main/scala/cromwell/core
A change to existing behaviour in core/src/main/scala/cromwell/core — 60 commits, 12 files: “Detect stalled workflows (\#4486)”; “SingleWorkflowMode support for WDL import directories closes \#1525 (\#1652)”; “Limit number of queued jobs per workflow (\#3356)”; and 55 more (refs \#4486, \#1525, \#1652, \#3356, \#7887, …).
core/src/main/scala/cromwell/core · medium confidence · unverified
61 commits (2 fixes) modifying engine/src/main/scala/cromwell/webservice
A change to existing behaviour in engine/src/main/scala/cromwell/webservice — 61 commits (2 fixs), 5 files: “CTM-489 Type annotations for implicits (\#7887)”; “CTM-152: Copy fix over from PR \#7822 (\#7831)”; “Evaluate runtime attributes outside the backend”; and 56 more (refs \#7887, \#7822, \#7831, \#1243, \#2380, …).
engine/src/main/scala/cromwell/webservice · medium confidence · unverified
67 commits (7 fixes) modifying wdl/transforms/new-base
A change to existing behaviour in wdl/transforms/new-base — 67 commits (7 fixs), 73 files: “Add optional output support for WDL draft 2 on PAPI v2 \[BA-6618\] (\#5890)”; “CTM-489 Type annotations for implicits (\#7887)”; “Adding line numbers to workflow definitions (\#4938)”; and 63 more (refs \#5890, \#7887, \#4938, \#3925, \#4036, …).
wdl/transforms/new-base · medium confidence · unverified
68 commits (11 fixes) modifying database/sql/src/main/scala/cromwell/database/slick
A change to existing behaviour in database/sql/src/main/scala/cromwell/database/slick — 68 commits (11 fixs), 10 files: “Pick up workflows to run from hog group with lowest actively running workflows \[BW-568\] (\#6680)”; “Tracking summary status in the database instead of in memory”; “Updated to Slick 3.2. (\#2104)”; and 63 more (refs \#6680, \#2104, \#2587, \#1532, \#1473, …).
database/sql/src/main/scala/cromwell/database/slick · medium confidence · unverified
7 commits (0 fixes) modifying core/src/main/scala/cromwell/core/actor
A change to existing behaviour in core/src/main/scala/cromwell/core/actor — 7 commits, 1 file: “Partial Unification Compiler Flag (\#3580)”; “Load Control (\#3366)”; “Throttle kv store an job store reads (\#3327)”; and 4 more (refs \#3580, \#3366, \#3327, \#7257).
core/src/main/scala/cromwell/core/actor · medium confidence · unverified
7 commits (0 fixes) modifying engine/src/main/scala/cromwell/engine/workflow/tokens
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow/tokens — 7 commits, 1 file: “Hog limits for job tokens (\#4108)”; “Job restart tokens \[CROM-6839\] (\#6590)”; “Refactored token dispenser to accommodate for rate limiting (\#3332)”; and 4 more (refs \#4108, \#6590, \#3332, \#4567, \#7257, …).
engine/src/main/scala/cromwell/engine/workflow/tokens · medium confidence · unverified
7 commits (0 fixes) modifying services/src/main/scala/cromwell/services/healthmonitor/impl
A change to existing behaviour in services/src/main/scala/cromwell/services/healthmonitor/impl — 7 commits, 1 file: “WX-1839 Remove GCS health check (\#7533)”; “Remove carboniting code and references \[BW-595\] (\#6234)”; “\[WX-1346\] Scalafmt (\#7257)”; and 4 more (refs \#7533, \#6234, \#7257, \#5459, \#7467, …).
services/src/main/scala/cromwell/services/healthmonitor/impl · medium confidence · unverified
7 commits (1 fix) modifying services/src/main/scala/cromwell/services/loadcontroller
A change to existing behaviour in services/src/main/scala/cromwell/services/loadcontroller — 7 commits (1 fix), 3 files: “Load Control (\#3366)”; “keep monitoring (\#3389)”; “Lots of simplifications into just using "ActorSystem"”; and 4 more (refs \#3366, \#3389, \#7257, \#6724, \#3540, …).
services/src/main/scala/cromwell/services/loadcontroller · medium confidence · unverified
8 commits (0 fixes) modifying backend/src/main/scala/cromwell/backend/standard/callcaching
A change to existing behaviour in backend/src/main/scala/cromwell/backend/standard/callcaching — 8 commits, 2 files: “Call caching grouping and hit support \[BA-6427\] \[BA-6428\] (\#5518)”; “I/O Actor (\#2056)”; “move cromwell”; and 3 more (refs \#5518, \#2056, \#7257).
backend/src/main/scala/cromwell/backend/standard/callcaching · medium confidence · unverified
8 commits (0 fixes) modifying dockerHashing/src/main/scala/cromwell/docker
A change to existing behaviour in dockerHashing/src/main/scala/cromwell/docker — 8 commits, 4 files: “Boot disk auto-size in Papi v2 (\#4472)”; “move cromwell”; “Enable local docker lookup (\#2139)”; and 3 more (refs \#4472, \#2139, \#7257, \#6589).
dockerHashing/src/main/scala/cromwell/docker · medium confidence · unverified
8 commits (0 fixes) modifying engine/src/main/scala/cromwell/engine/workflow
A change to existing behaviour in engine/src/main/scala/cromwell/engine/workflow — 8 commits, 2 files: “Always shift times to UTC, and render w/ 3 digits of millis”; “Log instead of store CC fails by default”; “\[WX-1346\] Scalafmt (\#7257)”; and 5 more (refs \#7257, \#4961, \#6196, \#2808, \#3239).
engine/src/main/scala/cromwell/engine/workflow · medium confidence · unverified
8 commits (1 fix) modifying services/src/main/scala/cromwell/services/instrumentation/impl/stackdriver
A change to existing behaviour in services/src/main/scala/cromwell/services/instrumentation/impl/stackdriver — 8 commits (1 fix), 2 files: “CTM-489 Type annotations for implicits (\#7887)”; “If skipping localization don't skip DOS/DRS resolution, plus... WA-351 WA-380 (\#5912)”; “Stackdriver Instrumentation BA-4788 (\#5010)”; and 5 more (refs \#7887, \#5912, \#5010, \#5036, \#7257, …).
services/src/main/scala/cromwell/services/instrumentation/impl/stackdriver · medium confidence · unverified
84 commits (6 fixes) modifying supportedBackends/tes
A change to existing behaviour in supportedBackends/tes — 84 commits (6 fixs), 19 files: “PAPI Directories and Literals (\#3697)”; “Graceful Server Shutdown (\#2490)”; “The concurrent-job-limit now available to all backends. Removed unused \TestBackendLifecycleActorFactory\”; and 78 more (refs \#3697, \#2490, \#7122, \#4168, \#7886, …).
supportedBackends/tes · medium confidence · unverified
9 commits (0 fixes) modifying engine/src/main/scala/cromwell/logging
A change to existing behaviour in engine/src/main/scala/cromwell/logging — 9 commits, 1 file: “Moving towards a multi-multi-project cromwell repo, many rough edges”; “Patches to offset date time usage. More use (and naming) of \OffsetDateTime\, less of \Timestamp\. Replaced use of \java.util.Date\ with \OffsetDateTime\. Removed use of \OffsetDateTime.toInstant\ & \.toEpochSecond\. Using \OffsetDateTime.now\ instead of the \KnowsWhatTimeItIs\ trait. Using \OffsetDateTime.toString\ instead of \DateTimeFormatter.ISO\_OFFSET\_DATE\_TIME\. Removed unused \joda-convert\ dependency, plus relying on transitive project dependencies. \TestableJesJobExecutionActor\ no longer tries to interact with the services in the engine. Cleaned up a few more places where \MetadataValue.apply\ was called with an extra \.toString\”; “Error reporting (\#1769)”; and 4 more (refs \#1769, \#7257).
engine/src/main/scala/cromwell/logging · medium confidence · unverified
9 commits (0 fixes) modifying services/src/main/scala/cromwell/services/healthmonitor/impl/common
A change to existing behaviour in services/src/main/scala/cromwell/services/healthmonitor/impl/common — 9 commits, 1 file: “Initial health monitor support (\#2654)”; “move cromwell”; “Make the cat noises go away”; and 4 more (refs \#2654, \#7257, \#4807).
services/src/main/scala/cromwell/services/healthmonitor/impl/common · medium confidence · unverified
9 commits (1 fix) modifying services/src/main/scala/cromwell/services/womtool/models
A change to existing behaviour in services/src/main/scala/cromwell/services/womtool/models — 9 commits (1 fix), 6 files: “Meta section \[BA-5797\] (\#5053)”; “\womtool/describe\ endpoint should populate \submittedDescriptorType\ for invalid workflows \[BA-6084\]”; “Womtool endpoint: indicate when a workflow itself is valid, not considering inputs (\#4940)”; and 6 more (refs \#5053, \#4940, \#4764, \#4571, \#7257, …).
services/src/main/scala/cromwell/services/womtool/models · medium confidence · unverified
91 commits (6 fixes) modifying backend/src/main/scala/cromwell/backend
A change to existing behaviour in backend/src/main/scala/cromwell/backend — 91 commits (6 fixs), 9 files: “Allow for a 'name-for-call-caching-purposes' to override the backend name. (\#4490)”; “PAPI Directories and Literals (\#3697)”; “Graceful Server Shutdown (\#2490)”; and 86 more (refs \#4490, \#3697, \#2490, \#4108, \#7740, …).
backend/src/main/scala/cromwell/backend · medium confidence · unverified
AWS integration test workflow for HaplotypeCaller added
Added AWS-specific integration test files (inputs, options, and WDL) for the HaplotypeCaller workflow, enabling users to run the GATK4 HaplotypeCaller in GVCF mode on AWS using S3 data sources and Broad's reference bucket. The new test configuration specifies the GATK 4.0.0.0 Docker image, sets Java memory options to 8 GB for HaplotypeCaller and 30 GB for MergeGVCFs, allocates 100 GB of disk space per task, and configures preemptible instance retries to 3.
centaur/src/main/resources/integrationTestCases/germline/haplotype-caller-workflow · high confidence
CI infrastructure restructured into modular, shared test scripts
The CI test execution logic has been refactored from monolithic scripts into a modular system. A new \src/ci/bin/test.inc.sh\ provides shared build variables and environment setup, while backend-specific tests (AWS, GCP Batch, TES, Slurm, etc.) are now driven by dedicated entry scripts (e.g., \testCentaurAws.sh\) that source common libraries like \test\_gcpbatch.inc.sh\ and \test\_singularity.inc.sh\. This change introduces a standardized \cromwell::build\ namespace for test functions, improves environment isolation, and adds a new \get\_cromwell\_hosts.py\ utility to dynamically discover Cromwell container hosts for load-balanced testing.
src/ci · high confidence
Centaur integration test suite restructured with new restart and upgrade capabilities
The Centaur testing framework has been reorganized into a dedicated subdirectory with a new README and a \centaur.sh\ wrapper for Docker-based execution. The test runner script (\test\_cromwell.sh\) now supports running against a specific Cromwell branch or JAR, allows parallel test execution, and enables tag-based inclusion/exclusion. A new \engineUpgradeTestCases\ directory introduces an \engine\_upgrade\ tag and a \cromwell\_restart.test\ case, which validates Cromwell's ability to recover and continue workflow execution after a server restart.
centaur · high confidence
Centaur integration tests now use a dedicated Cromwell client with robust retry logic
The Centaur test suite now uses a new \CentaurCromwellClient\ that wraps Cromwell API calls in \cats-effect IO\ and implements a configurable retry mechanism for transient network or server errors. This change ensures that test failures caused by temporary Cromwell unavailability (such as connection timeouts or 404s during startup) are automatically retried up to five times with a fixed delay, rather than failing immediately. It also introduces a dedicated daemon thread pool for blocking HTTP operations to prevent thread starvation and adds a lightweight \isAlive\ health check that queries the Cromwell database directly to verify service readiness before submitting workflows.
centaur/src/main/scala/centaur/api · high confidence
Centaur test infrastructure restructured with new upgrade and parallel execution suites
The Centaur integration test framework has been reorganized to support distinct test execution modes and engine upgrade validation. A new base class, AbstractCentaurTestCaseSpec, centralizes test discovery, duplicate name detection, and retry logic using Cats Effect IO. Two new abstract suites, AbstractCromwellEngineOrBackendUpgradeTestCaseSpec and its concrete implementations (EngineUpgradeTestCaseSpec, PapiUpgradeTestCaseSpec), enforce empty database constraints before and after engine upgrades. ParallelTestCaseSpec and SequentialTestCaseSpec now explicitly separate test execution based on the test format, allowing parallel runs by default while ensuring restart-heavy tests run sequentially. An ExternalTestCaseSpec allows running single test files via environment variable, and a new CromwellDatabaseCallCaching utility supports clearing call cache entries during retries.
centaur/src/it/scala/centaur · high confidence
Configurable read limits and new value evaluation utilities
The WDL model now supports configurable limits for reading various data types (lines, bool, int, float, string, json, tsv, map, object) via a new \FileSizeLimitationConfig\ that loads settings from the \system.input-read-limits\ configuration block, defaulting to \Int.MaxValue\. Additionally, new utility objects \FileEvaluatorUtil\ and \ValueEvaluation\ have been added to support value serialization to JSON and TSV, and to identify files for delocalization within WDL expressions.
wdl/model/shared · high confidence
Configured logging for integration tests
Added application.conf and logback-test.xml to the integration test resources to control log output. Akka logs are now routed through SLF4J with dead-letter logging disabled, and specific libraries (MockServer, Liquibase, HikariCP, and Google Cloud Storage) are set to WARN or ERROR levels to reduce noise in test logs.
centaur/src/it/resources · high confidence
CromIAM configuration and logging infrastructure
CromIAM now includes its own standalone configuration and logging setup. The new application.conf defines service endpoints for CromIAM (port 8001), SAM (port 443), and Cromwell (port 8000), and sets HTTP request and connection timeouts to 40 seconds. A dedicated logback.xml configures logging with enhanced thread/date formatting, supports Standard, Pretty, and FileRoller modes, and routes WARN-level events to Sentry. A sentry.properties file is also added to initialize the Sentry DSN.
CromIAM/src/main/resources · high confidence
CromIAM server configuration model and validation
The CromIAM server configuration is now defined by a new \CromIamServerConfig\ case class that aggregates settings for CromIAM, Cromwell, the SAM client, and Swagger OAuth. This change introduces a robust validation layer using \ErrorOr\ to ensure configuration paths are present and correctly typed, specifically adding a \check-submit-whitelist\ boolean flag to the \SamClientConfig\ to control access restrictions.
CromIAM/src/main/scala/cromiam/server/config · high confidence
CromIAM server initialization and shutdown behavior
The CromIAM server entry point (CromIamServer.scala) has been introduced to handle server startup and shutdown. Upon starting, the server logs its version number using VersionUtil. It initializes an Akka ActorSystem, materializer, and execution context, and configures routes including Swagger UI. A custom shutdown hook is registered to log 'Shutting down the server' and signal completion when the process receives a shutdown signal (e.g., Ctrl-C), replacing the default behavior.
CromIAM/src/main/scala/cromiam/server · high confidence
Database migration for workflow options encryption and key renaming
This change introduces database migration logic to handle workflow options stored in the metadata and workflow store. It adds a migration to rename legacy workflow option keys (such as 'defaultRuntimeOptions' and 'workflowFailureMode') to their new snake\_case equivalents (e.g., 'default\_runtime\_attributes') in the METADATA\_ENTRY table. Additionally, it provides infrastructure to encrypt workflow options in the WORKFLOW\_STORE\_ENTRY table and clear encrypted values from the METADATA\_ENTRY table, ensuring that sensitive data is properly secured and formatted according to the new configuration.
database/migration/src/main/scala/cromwell/database/migration/workflowoptions · high confidence
Database migration refactoring and job store simpleton handling
The database migration logic has been restructured to improve reliability and maintainability. A new abstract base class, AbstractRestartMigration, now standardizes how migrations interact with the database connection, ensuring transactions are properly committed. The JobStoreSimpletonMigration has been updated to correctly handle nullable job store simpletons, preventing failures that occurred when Cromwell recovered after a migration due to incorrect quoting of file values. Additionally, the RenameWorkflowOptionKeysMigration now explicitly handles the renaming of workflow option keys in the WORKFLOW\_STORE table, ensuring consistency in stored workflow configurations.
database/migration/src/main/scala/cromwell/database/migration/restart · high confidence
Database migration to normalize failure metadata structure
A new database migration has been added to the failure metadata module to restructure how workflow failure information is stored in the METADATA\_ENTRY table. The migration introduces two batched tasks: ExpandSingleFailureStrings, which splits flat failure entries (e.g., 'failures\[0\]') into distinct message and causedBy components (e.g., 'failures\[0\]:message' and 'failures\[0\]:causedBy\[\]'), and DeduplicateFailureMessageIds, which resolves duplicate failure message records by assigning them unique identifiers. This ensures that failure metadata is consistently structured for downstream processing.
database/migration/src/main/scala/cromwell/database/migration/failuremetadata · high confidence
Database schema updates for call caching, execution tracking, and workflow metadata
This release applies a series of Liquibase migration changesets to the database schema. It introduces a new GROUP\_METRICS\_ENTRY table to track Cloud Quota exhaustion events and adds a HOG\_GROUP column to the WORKFLOW\_STORE\_ENTRY table. Call caching is enhanced with a new CALL\_CACHING\_AGGREGATION\_ENTRY table for optimized hash lookups, a CREATED\_AT timestamp on CALL\_CACHING\_ENTRY, and a new CALL\_CACHING\_JOB\_DETRITUS table to store job output files. The EXECUTION table is expanded with an ATTEMPT column (making it part of the unique constraint), an IDX column, and START\_DT/END\_DT timestamps, while also adding columns for result reuse tracking (ALLOWS\_RESULT\_REUSE, DOCKER\_IMAGE\_HASH, EXECUTION\_HASH). Additionally, the schema adds a WORKFLOW\_NAME column to WORKFLOW\_EXECUTION, a WORKFLOW\_URL column to WORKFLOW\_STORE\_ENTRY, and a new BACKEND\_KV\_STORE table for backend-specific key-value pairs.
database/migration/src/main/resources/changesets · high confidence
Enhanced metadata flattening to include attempt numbers in shard keys
The \JsonUtils\ module in the Centaur testing framework now supports flattening Cromwell metadata responses in a way that preserves both shard indices and attempt numbers. Previously, flattening shard arrays only included the shard index in the resulting key structure; the new logic detects arrays containing both \shardIndex\ and \attempt\ fields and generates flattened keys that incorporate both (e.g., \shardIndex.attemptNumber\). This ensures that metadata for multiple attempts of the same shard is distinct and not lost during conversion, while maintaining backward compatibility with older structures that only used shard indices.
centaur/src/main/scala/centaur/json · high confidence
Improved database schema synchronization and migration reliability
The database migration system now includes a new DiffResultFilter to ignore harmless schema differences (such as VARCHAR length variations and column reordering) during synchronization checks, ensuring that minor cosmetic differences do not trigger false alarms. Additionally, Liquibase API access is now synchronized via a mutex to prevent concurrent access issues, and logging output from Liquibase is properly routed to the application logger instead of standard output.
database/migration/src/main/scala/cromwell/database/migration/liquibase · high confidence
Improved logging accuracy and reliability with Akka-aware converters and file handle management
This change introduces a new logging infrastructure in the core module to provide more accurate timestamps and thread information in logs, while also fixing file handle leaks. It adds custom Logback converters (EnhancedDateConverter, EnhancedThreadConverter) and an Akka logger (EnhancedSlf4jLogger) that prioritize Akka-specific MDC properties (akkaTimestamp, sourceThread) for more precise log entries. Additionally, it implements a WorkflowLogger that synchronizes file appender creation and teardown to prevent ConcurrentModificationExceptions and ensure file handles are properly closed, and includes a Java Logging Bridge to unify Java Util Logging with SLF4J.
core/src/main/scala/cromwell/core/logging · high confidence
Improved server shutdown and logging behavior
Cromwell now implements a coordinated, graceful shutdown process that stops new workflow submissions, unbinds the HTTP server, and orderly terminates internal actors (such as the I/O and job store actors) to prevent data loss. Additionally, log noise is reduced during shutdown by silencing dead-letter messages and specific Akka stream termination errors that occur while the system is winding down.
engine/src/main/scala/cromwell/server · high confidence
Introduction of structured reference configuration files
Cromwell now provides a new, structured set of default configuration files (reference.conf, reference\_database.inc.conf, and reference\_local\_provider\_config.inc.conf) that define baseline settings for the web service, Akka dispatchers, system limits, database migration batching, and the local backend. This replaces the previous single-file approach, allowing users to include these references in their own cromwell.conf for clearer, modular configuration management.
core/src/main/resources · high confidence
New abort response types for workflow abort operations
Added a new \AbortResponse\ sealed trait and associated case classes (\WorkflowAbortFailureResponse\, \WorkflowAbortedResponse\, \WorkflowAbortRequestedResponse\) to model the outcomes of workflow abort requests. This provides a structured way to distinguish between successful aborts (whether immediately aborted or requested for later execution) and failures, improving error handling and status reporting for abort operations.
core/src/main/scala/cromwell/core/abort · high confidence
New metadata migration implementation for Cromwell database
The database migration module now includes a new set of Scala classes to migrate legacy execution, symbol, and workflow execution tables into the new METADATA\_JOURNAL schema. This change introduces specific migration handlers for execution events, failure events, call inputs/outputs, and workflow-level metadata, ensuring that historical workflow data is correctly transformed and stored in the updated metadata structure.
database/migration/src/main/scala/cromwell/database/migration/metadata · high confidence
Refactor backend I/O into modular path and glob function traits
The backend I/O logic in \cromwell.backend.io\ has been restructured into distinct, reusable traits to separate concerns. \WorkflowPaths\ and \JobPaths\ (including Docker-specific variants) now handle the construction and resolution of execution, input, and output directories, supporting configurable roots and sub-workflow nesting. \GlobFunctions\ and \DirectoryFunctions\ manage file discovery, with globs now resolved via list files to decouple pattern matching from execution, while \FileEvaluationIoFunctionSet\ prevents globbing during file evaluation to avoid side effects. This modularization standardizes how paths are generated and accessed across different backend implementations.
backend/src/main/scala/cromwell/backend/io · high confidence
Refactored GCP Batch backend to use a new request manager architecture
The GCP Batch backend has been restructured to replace the previous execution model with a new \BatchApiRequestManager\ and associated worker actors. This change introduces a centralized request handling system that manages job submission, status polling, and abortion through dedicated actors (\BatchApiRunCreationClient\, \BatchApiStatusRequestClient\, \BatchApiAbortClient\), improving reliability and separation of concerns in how Cromwell interacts with the GCP Batch API.
supportedBackends/google/batch · high confidence
Refactored database migration to use batched processing
The database migration logic has been refactored to process large tables in batches rather than loading all rows into memory at once. This change introduces a new \BatchedTaskChange\ trait and supporting pagination utilities (\QueryPaginator\, \ResultSetIterator\) that allow migrations to read and write data in configurable chunks (controlled by \database.migration.read-batch-size\ and \database.migration.write-batch-size\). This improves stability and performance for workflows with large metadata tables by preventing memory exhaustion during migration. An example migration, \PreemptedIsABackendStatus\, now uses this batched approach to update execution statuses.
database/migration/src/main/scala/cromwell/database/migration/custom · high confidence
S3 filesystem provider upgraded to AWS SDK v2
The S3 filesystem implementation in \filesystems/s3/src/main/java\ has been migrated from the AWS SDK v1 to the AWS SDK v2. This change introduces new core classes (such as \AmazonS3ClientFactory\, \S3FileSystem\, and \S3FileChannel\) that utilize the v2 client libraries, enabling support for modern S3 features like cross-region access and updated authentication mechanisms while maintaining the JSR-203 NIO2 interface.
filesystems/s3/src/main/java · high confidence
Sanitized LanguageFactory interface and introduced ValidatedWomNamespace
The core language factory interface has been refactored to provide a cleaner, more explicit contract for workflow processing. The new LanguageFactory trait defines specific methods for generating WomBundles, creating executables, and validating namespaces, alongside configuration-driven controls for enabling factories and strict validation modes. Additionally, a new ValidatedWomNamespace case class now encapsulates the result of workflow validation, explicitly bundling the executable, input mappings, and imported file contents for downstream use.
languageFactories/language-factory-core/src/main/scala/cromwell/languages · high confidence
Standardized local database setup scripts for PostgreSQL and MySQL
The local development environment now uses dedicated shell scripts to launch PostgreSQL and MySQL containers with configurations aligned to Cromwell's CI requirements. The PostgreSQL setup includes a new initialization script that sets \max\_connections\ to 300 and enables the \lo\ (large object) extension, while the MySQL script launches a container compatible with the expected CI parameters. These changes replace previous ad-hoc or undocumented methods for spinning up local databases.
scripts/local-docker-database · high confidence
Support for OCI manifest formats in Docker image hashing
The Docker registry v2 client now accepts OCI manifest media types (application/vnd.oci.image.manifest.v1+json and application/vnd.oci.image.index.v1+json) in addition to standard Docker manifest types. If a request for a Docker manifest fails, the system automatically falls back to requesting an OCI manifest, allowing compatibility with registries and images that use the OCI specification.
dockerHashing/src/main/scala/cromwell/docker/registryv2 · high confidence
Updated GATK joint-discovery test inputs to use new reference bucket
The GATK joint-discovery integration test inputs have been updated to point to the new \gcp-public-data--broad-references\ bucket for all reference files (such as the hg38 FASTA, DBSNP, and 1000 Genomes resources). This change ensures the test workflows can successfully locate the required reference data in the new storage location.
centaur/src/main/resources/integrationTestCases/germline/joint-discovery-gatk · high confidence
Updated integration test cases for arrays and single-sample workflows
The integration test cases for the arrays and single-sample workflows have been updated to use the new public reference bucket \gcp-public-data--broad-references\ instead of the previous \broad-references\ path. Additionally, the arrays test configuration now sets preemptible tries to zero and uses a development Docker image, while the single-sample test options have been adjusted to include additional availability zones.
centaur/src/main/resources/integrationTestCases/green · high confidence
Upgrade test script for Cromwell 25 to 26 migration
The \scripts/test\_upgrade\ directory now contains a complete test harness (\test\_upgrade.sh\) and associated WDL workflow (\scatter\_files.wdl\) and input files to validate Cromwell upgrades. The script automates a rolling upgrade test by submitting jobs via the Cromwell API using the previous version (cromwell-25-31ae549-SNAP.jar), shutting down the service, restarting it with the new version (cromwell-26-88630db-SNAP.jar), and resubmitting jobs to ensure continuity and correctness across the version boundary.
_scripts/test\upgrade · high confidence
Test coverage
14 commits adding/updating tests in server/src/test/scala/cromwell/subworkflowstore; 16 commits adding/updating tests in server/src/test/scala/cromwell/engine/workflow/lifecycle; 195 commits adding/updating tests in services/src/test; 26 commits adding/updating tests in server/src/test/scala/cromwell/engine/workflow; 30 commits adding/updating tests in server/src/test/scala/cromwell/engine; 41 commits adding/updating tests in server/src/test/scala/cromwell/engine/workflow/lifecycle/execution/ejea; 47 commits adding/updating tests in server/src/test/scala/cromwell; 8 commits adding/updating tests in server/src/test/scala/cromwell/jobstore; Add SmartSeq2 single-sample integration test case; Added 'hello' external test case for workflow execution validation; Added CNV somatic test cases for GATK-launch; Added Centaur test cases for WDL 'after' keyword and workflow abort scenarios; Added CromIAM webservice test suite and mock clients; Added backend test infrastructure and validation specs; Added integration test for high-scale scattered workflows; Added integration tests for SmartSeq2, HaplotypeCaller, Mutect2, and CNV workflows; Added metadata parsing and comparison utilities for Centaur tests; Added test case for Cromwell restart with GCP Batch backend; Added test coverage for common collection utilities and exception handling; Added test coverage for common utility modules; Added test for graceful handling of unparseable Docker registry responses; Added test infrastructure and specs for core components; Added test infrastructure for backpressure report; Added test infrastructure for engine components; Added test logging configuration for WDL Draft 2; Added test resources for CromIAM; Added test resources for PAPI v1/v2 metadata comparison and forkjoin workflow validation; Added test resources for workflow metadata and configuration; Added test utilities and specifications for core Cromwell components; Added tests for AWS configuration parsing and validation; Added tests for Cromwell API client request construction and error handling; Added tests for Cromwell API model JSON formatters; Added tests for CromwellFileSystems configuration and validation; Added tests for DefaultPathBuilder and PathBuilderFactory; Added tests for Docker image hashing and mirroring logic; Added tests for GCS path building and requester-pays error handling; Added tests for GCS storage configuration and options; Added tests for GoogleConfiguration parsing and validation; Added tests for GroupMetricsActor and StandardValidatedRuntimeAttributesBuilder; Added tests for JobPaths and WorkflowPaths path resolution; Added tests for S3 storage configuration and client creation; Added tests for WDL draft-3 language factory caching behavior; Added tests for WDL draft2 language factory features; Added tests for WDL import resolution and version detection; Added tests for WomValueBuilder simpleton conversion logic; Added tests for call caching blacklist and file hashing actors; Added tests for call caching hash strategies and core I/O actor behavior; Added tests for common validation utilities and mock helpers; Added tests for core actor batching, robust client, and stream helpers; Added tests for label key and value validation; Added tests for retry and backoff logic; Added tests for standard Centaur test case execution and format definitions; Added unit tests for AWS ECR and ECR Public registry flows; Added unit tests for Centaur test infrastructure components; Added unit tests for CromwellClient workflow ID and collection resolution; Added unit tests for DRS Localizer command-line parsing and main execution logic; Added unit tests for DRS filesystem components; Added unit tests for GCP authentication modes in cloudSupport; Added unit tests for GCS batch IO commands; Added unit tests for S3 file system operations; Added unit tests for S3 path building and validation; Added unit tests for SamClient authorization and submission logic; Added unit tests for core logging components; Added unit tests for metadata comparison scripts; Added unit tests for the DRS Cloud NIO implementation; Added validation tests for runtime attributes and return codes; New test formula library for workflow execution and validation; New test utilities for assertions, timeouts, and repetition; Refactored Centaur test infrastructure with new exception handling and cloud-agnostic file checking; Refactored Centaur test workflow configuration and execution model.
Dependencies
Dependency updates for CromwellRefdiskManifestCreator and Java client codegen
The CromwellRefdiskManifestCreator Maven module now uses Jackson Databind 2.13.4.1, Log4j 2.17.1, and JUnit 4.13.1. The Java client codegen build (codegen\_java) targets Scala 2.13.9 and includes OkHttp 4.9.3, Gson 2.9.0, Commons Lang3 3.12.0, Swagger Annotations 1.6.5, and JUnit 5.8.2. Additionally, the Cromwell Monitor requirements.txt for the Google Pipelines backend now lists google-cloud-monitoring, psutil, and requests.
(dependencies) · high confidence
New sbt build infrastructure for CI, dependencies, and Docker publishing
The build system introduces dedicated Scala files to manage project configuration: ContinuousIntegration.scala adds a \renderCiResources\ task that uses the \broadinstitute/dsde-toolbox\ Docker image to render templates via Hashicorp Vault and validates that all sub-projects are correctly aggregated; Dependencies.scala centralizes library versions (including Akka 2.5.32, AWS SDK 2.29.20, Jackson 2.13.2, and Netty 4.2.16.Final); Merging.scala defines custom assembly merge strategies to handle conflicts in AWS/Google jars and metadata files; and Publishing.scala configures Docker image tagging (including \latest\ for releases) and entrypoints that support \JAVA\_OPTS\ and \--add-opens\ flags for Java 17 compatibility.
project · high confidence
Housekeeping
Repository initialization and configuration standardization
The repository has been initialized with standard configuration files to streamline development and deployment. This includes a \.dockerignore\ symlink to \.gitignore\, \.gitattributes\ for consistent text file handling, and a comprehensive \.gitignore\ covering build artifacts, IDE files, and local execution logs. Build and formatting are standardized via \.sbtopts\ (setting JVM memory limits), \.scalafmt.conf\ (Scala 2.13 formatting rules), and \.sdkmanrc\ (pinning Java 17). Documentation is configured for Read the Docs via \.readthedocs.yaml\, and the project now includes \AUTHORS\, \CITATION.md\, \CONTRIBUTING.md\, and \LICENSE-ASL-2.0\ files.
(repo-wide) · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 47 → 50 (+2.6)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.
Lenses
- Code Health 57 → 57 (-0.0)
- Architecture 94 → 94 (-0.0)
- Maturity 60 → 69 (+8.6)
- Readiness 60 → 66 (+6.1)
- Security 31 → 36 (+4.3)
- Performance 63 → 60 (-3.4)
Resolved (177)
- Critical CVE: [GHSA redacted] (project/Dependencies.scala)
- Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
- Duplicated block (11 lines × 2) (scripts/metadata_comparison/metadata_comparison/lib/operations_digesters.py)
- Duplicated block (12 lines × 2) (wdl/transforms/new-base/src/main/scala/wdl/transforms/base/wdlom2wom/graph/IfElementToGraphNode.scala)
- Duplicated block (29–31 lines × 3) (supportedBackends/google/batch/src/main/scala/cromwell/backend/google/batch/api/request/AbortRequestHandler.scala)
- Duplicated block (6 lines × 2) (wdl/model/draft2/src/main/scala/wdl/draft2/model/expression/TypeEvaluator.scala)
- High CVE: [GHSA redacted] (project/Dependencies.scala)
- Most significant orphaned file (wdl/transforms/new-base/src/main/scala/wdl/transforms/base/linking/expression/values/EngineFunctionEvaluators.scala)
- No ADRs found
- Off-boarding risk: anonymized user #1
- Orphaned file (CromIAM/src/main/scala/cromiam/webservice/SubmissionSupport.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/OutputEvaluator.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/async/KnownJobFailureException.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/dummy/DummyAsyncExecutionActor.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/io/JobPaths.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/io/JobPathsWithDocker.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/standard/StandardLifecycleActorFactory.scala)
- Orphaned file (backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala)
- Orphaned file (centaur/src/main/scala/centaur/api/CentaurCromwellClient.scala)
- Orphaned file (cloud-nio/cloud-nio-impl-ftp/src/main/scala/cloud/nio/impl/ftp/operations/FtpOperation.scala)
- …and 157 more
New (87)
- Change coupling: JobPaths.scala ↔ StandardAsyncExecutionActor.scala (backend/src/main/scala/cromwell/backend/io/JobPaths.scala)
- Edited copies of one member (3 members, 11+ corresponding lines) (services/src/main/scala/cromwell/services/metadata/WorkflowQueryKey.scala)
- Edited copies of one member (3 members, 17+ corresponding lines) (languageFactories/wdl-biscayne/src/main/scala/languages/wdl/biscayne/WdlBiscayneLanguageFactory.scala)
- Edited copies of one member (3 members, 20+ corresponding lines) (backend/src/main/scala/cromwell/backend/validation/RuntimeAttributesValidation.scala)
- Edited copies of one member (3 members, 23+ corresponding lines) (engine/src/main/scala/cromwell/engine/workflow/WorkflowActor.scala)
- Edited copies of one member (3 members, 3+ corresponding lines) (wdl/transforms/biscayne/src/main/scala/wdl/transforms/biscayne/ast2wdlom/BiscayneGenericAstNode.scala)
- Edited copies of one member (3 members, 4+ corresponding lines) (supportedBackends/google/batch/src/main/scala/cromwell/backend/google/batch/actors/GcpBatchAsyncBackendJobExecutionActor.scala)
- Edited copies of one member (3 members, 5+ corresponding lines) (wdl/transforms/new-base/src/main/scala/wdl/transforms/base/linking/expression/consumed/BinaryOperatorEvaluators.scala)
- Edited copies of one member (3 members, 7+ corresponding lines) (wdl/transforms/biscayne/src/main/scala/wdl/transforms/biscayne/parsing/BiscayneParser.scala)
- Edited copies of one member (3 members, 7+ corresponding lines) (wom/src/main/scala/wom/values/WomFloat.scala)
- Edited copies of one member (4 members, 5+ corresponding lines) (wom/src/main/scala/wom/values/WomBoolean.scala)
- Edited copies of one member (4 members, 5+ corresponding lines) (wom/src/main/scala/wom/values/WomBoolean.scala)
- Edited copies of one member (4 members, 9+ corresponding lines) (wdl/transforms/new-base/src/main/scala/wdl/transforms/base/wdlom2wom/expression/renaming/BinaryOperatorEvaluators.scala)
- Edited copies of one member (5 members, 11+ corresponding lines) (wdl/transforms/new-base/src/main/scala/wdl/transforms/base/linking/expression/files/BinaryOperatorEvaluators.scala)
- Edited copies of one member (5 members, 6+ corresponding lines) (wom/src/main/scala/wom/values/WomFile.scala)
- Edited copy of a member (10 corresponding lines) (CromIAM/src/main/scala/cromiam/instrumentation/CromIamInstrumentation.scala)
- Edited copy of a member (10 corresponding lines) (backend/src/main/scala/cromwell/backend/io/GlobFunctions.scala)
- Edited copy of a member (10 corresponding lines) (backend/src/main/scala/cromwell/backend/validation/MaxRetriesValidation.scala)
- Edited copy of a member (10 corresponding lines) (wdl/transforms/new-base/src/main/scala/wdl/transforms/base/linking/expression/values/EngineFunctionEvaluators.scala)
- Edited copy of a member (10 corresponding lines) (wom/src/main/scala/wom/graph/GraphOutputNode.scala)
- …and 67 more
Changes since last survey
- 4 commits — 3 feature/other, 1 fixes
By area
- CromIAM/src — 1 commit
- backend/src — 1 commit
- centaur/src — 1 commit
- supportedBackends/google — 1 commit
Notable commits
- fix: [CTM-747] CI Fixes (#7910)
- change: CTM-649 Clean up CWL complexity in standard streams (#7912)
- change: [CTM-684][CTM-683][CTM-682][CTM-696][CTM-697] Security dependency updates (#7907)
- change: [CTM-747] Dont call python3 inside cloud-sdk image (#7911)
Architecture
- Unchanged — 0 containers · 2 contexts · 0 edges
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
broadinstitute/cromwell was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 14 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit e938a289f99776123e02eae9e4e79295350c0d02 — the exact code this score is about.
- Scored under rubric-2026.11.4 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.65+preprod-e1a3b69fc8ee.