marciok/gust

Gust is a distributed workflow orchestration engine that executes Directed Acyclic Graphs (DAGs) defined in Elixir or Python. It manages the full lifecycle of data pipelines, including scheduling, distributed run claiming, task execution with support for mapping and waiting, and error reporting. The system provides a web dashboard, a REST API, and an MCP server for monitoring and controlling these workflows.

30 changes.

This period focused on a major architectural overhaul of the DAG execution engine, introducing a pluggable adapter system for multi-language support and a gateway-based runner for distributed task management. Concurrently, the project launched a comprehensive web dashboard with LiveView, a REST API, and an MCP server, while modernizing configuration and expanding test coverage to support these new capabilities.

43 changes.

The project refactored the core run execution engine to support distributed claiming and leasing, while introducing extensible adapter protocols for DAG parsing and task execution. This period saw the addition of native Elixir support alongside a new GustPy application for Python workflows, enabling the orchestration of diverse pipeline definitions. Concurrently, the system enhanced operational capabilities with database-backed task waiting, external error reporting, and improved Docker release configurations.

Introduces a new Elixir adapter (\Gust.DAG.Parser.Adapters.Elixir\) that enables the Gust DAG engine to parse and validate Elixir files using the \Gust.DSL\. This adapter compiles Elixir source files, extracts task definitions and options (including \store\_result\, \skip\_if\, \map\_over\, and \wait\_for\), and constructs the corresponding DAG graph and stages, allowing users to define data pipelines directly in Elixir.

apps/gust/lib/gust/dag/parser/adapters · high confidence

Introduces a new parser adapter for Python-based DAGs, enabling the system to parse \.py\ files by invoking the \uv\ executable to run a parse command. The adapter handles various error conditions, including when \uv\ is not found on the PATH, when the file is not a valid GustPy file, or when the parse command fails, and constructs the DAG definition from the resulting JSON output.

_apps/gust\_py/lib/gust\py/parser · high confidence

The system now supports executing Python tasks via the \uv\ package manager. A new \GustPy.Executor.UV\ module has been introduced to handle task invocation, which locates the \uv\ executable and runs the specified DAG task using \uv run\. This executor manages the working directory context and handles cases where \uv\ is not found on the system path, providing a dedicated execution backend for Python-based workflows.

_apps/gust\_py/lib/gust\py/executor · high confidence

A new implementation for database-backed advisory locking has been added for PostgreSQL. This module, Gust.DBLocker.Postgres, implements the Gust.DBLocker behavior by using pg\_try\_advisory\_lock within a repository checkout to safely attempt acquiring a lock and passing the result to a callback function.

_apps/gust/lib/gust/db\locker · high confidence

The release now includes shell scripts to support distributed operation and tooling. A new \env.sh.eex\ configures the release node name based on the host IP to enable node connections. The \start-core\ script provides a dedicated entry point to start the core service, while \gust-cli\ allows users to execute CLI commands in a console context. Additionally, a \migrate\ script is provided to run database migrations.

apps/gust/rel · high confidence

The application now respects the user's system color scheme preference, automatically applying dark or light themes and updating Mermaid charts in real-time when the system setting changes. Additionally, JSON and Python syntax highlighting is now available for viewing task parameters and results via new Prism.js components.

_apps/gust\web/assets/js · high confidence

The Elixir adapter for the DAG task worker has been implemented, enabling Gust to execute tasks written in Elixir. This adapter supports conditional task skipping via a \skip\_if\ option, validates that task results are JSON-serializable when required, and includes robust error handling that captures stack traces for both runtime errors and non-recoverable errors. It also handles linked process exits to ensure proper error reporting when dependent processes fail.

_apps/gust/lib/gust/dag/task\worker · high confidence

Introduces the GustPy application, enabling users to define and execute Python-based Directed Acyclic Graphs (DAGs) within the Gust orchestration engine. This allows teams to reuse existing Python workflows (such as those from Airflow) without rewriting them in Elixir. The feature includes a Python adapter configuration, a dedicated Python runtime that spawns processes for task execution via JSON messages, and supporting project scaffolding (formatter, gitignore, and documentation).

_apps/gust\py · high confidence

Added a new communication layer for interacting with Python-based tasks, consisting of a frame codec for binary framing and a JSON-based messenger implementation. The messenger decodes incoming JSON messages to handle logging, retrieve secrets, and fetch task details by name and run ID, while also processing results and errors from the Python runtime. A specific behavioral change is included: attempting to retrieve a task that does not exist for a given run ID now raises an exception rather than returning a null result.

_apps/gust\_py/lib/gust\_py/task\messenger · high confidence

The Gust DAG engine now supports tasks that explicitly wait for external signals before proceeding. A new \TaskWaiter.Repo\ implementation provides the persistence layer for this capability, allowing tasks to enter a 'waiting' state and be resumed via a \resume/2\ function that operates within a database transaction. When a wait condition is satisfied, the system updates the task status, broadcasts the change via PubSub, and triggers the next steps in the run. The implementation also includes logic to handle task failures and clearing wait states, ensuring that the workflow engine can manage long-running or event-driven tasks reliably.

_apps/gust/lib/gust/dag/task\waiter · high confidence

The DAG runtime now supports pluggable adapters via a new \Gust.DAG.Runtime.Adapter\ protocol, enabling the execution engine to be extended for different languages or environments. This change introduces a standardized interface defining \setup\, \teardown\, \on\_finished\_callback\, and \kill\ operations, allowing the runtime to manage lifecycle events and task termination consistently. An initial Elixir adapter implementation is provided, which handles compiling DAG definitions into modules, executing finish callbacks, and forcefully terminating tasks via process exit signals.

apps/gust/lib/gust/dag/runtime · high confidence

Added example DAGs to the \examples/dags\ directory demonstrating key Gust capabilities. The \fusion\_prompt.ex\ DAG implements an OpenRouter model-fusion pattern, dispatching prompts to a panel of models (Claude, GPT, Gemini) in parallel, using a judge model to analyze responses, and synthesizing a final answer. The \daily\_stock\_decider.ex\ DAG showcases a scheduled workflow that aggregates market, company, news, and macro data to generate a buy/hold/sell recommendation via multiple LLMs. Additionally, \parent\_waits\_for\_child.ex\ and \child\_resumes\_parent.ex\ demonstrate inter-DAG coordination, where a parent DAG triggers a child DAG and waits for its result using \TaskWaiter\. A basic \hello\_world.ex\ example is also included.

examples/dags · high confidence

A new Mix task, \gust.cli\, has been added to the application. Running this task sets the \GUST\_ROLE\ environment variable (defaulting to \console\ if not already set) and executes the \Gust.CLI.exec\ function, displaying the result to the user. This provides a convenient command-line interface entry point for the Gust application.

apps/gust/lib/mix · high confidence

A new repository implementation for run claiming has been added, enabling distributed nodes to safely acquire and renew execution leases for DAG runs. The system now supports a configurable lease duration via the \claim\_lease\_seconds\ environment variable (defaulting to 15 seconds), allowing nodes to renew their claim on a run or safely skip to the next available run if a previous claim has expired. This change introduces a robust, database-backed locking strategy (\FOR UPDATE SKIP LOCKED\) to prevent race conditions when multiple nodes compete for the same run.

apps/gust/lib/gust/run/claim · high confidence

The project now provides a \setup\_gust\_app.sh\ script that bootstraps new applications using Igniter and a minimal Phoenix setup, replacing the previous \mix gust.start\ command with \mix phx.server\. A \.env.example\ file has been added to define required environment variables for database and secret configuration, and the \.gitignore\ has been updated to exclude these environment files and new static asset paths.

(repo-wide) · high confidence

This release introduces the core web interface and integration endpoints for Gust. Users can now access a LiveView-based dashboard to visualize DAG definitions, monitor run history, and manage tasks (including restarting, cancelling, and viewing logs) via the new \GustWeb.DashboardRouter\. A new authenticated REST API (\GustWeb.API\) allows external programs to trigger DAG runs and resume waiting tasks. Additionally, a Model Context Protocol (MCP) server (\GustWeb.MCPRouter\) is provided, exposing tools for querying DAGs, managing runs, and reading resource files, along with a system status page for runtime diagnostics.

_apps/gust\web/lib · high confidence

The \Gust.DAG.RunRestarter.Worker\ module, which previously handled DAG restarts and task retries via a GenServer, has been removed from the codebase. This change eliminates the specific logic for restarting runs with retrying status and preventing DAGs with errors from being restarted or run through this worker component.

_apps/gust/lib/gust/dag/run\restarter · high confidence

The gust\_web application now includes specific release environment and overlay scripts to support running in a Docker container. The \env.sh.eex\ script configures the release to distribute by name using the container's hostname, enabling node connections. Additionally, new \server\ and \migrate\ scripts (for both Linux and Windows) are provided to explicitly start the Phoenix server and run database migrations, facilitating single-mode execution within Docker.

_apps/gust\web/rel · high confidence

This change updates the application's configuration to improve security and operational flexibility. Database credentials, secret keys, and the secrets cloak key now read from environment variables (PGHOST, PGUSER, PGPASSWORD, SECRET\_KEY\_BASE, B64\_SECRETS\_CLOAK\_KEY) instead of being hardcoded, with production enforcing the presence of B64\_SECRETS\_CLOAK\_KEY. The mailer functionality has been removed entirely across all environments. A new run dispatcher is introduced, defaulting to Gust.PGNotifier.Worker in production and Gust.Run.Pooler in test, replacing the previous stage-based runner architecture. Additionally, the MCP feature is enabled by default in dev and test, and the DAG cron scheduler is enabled in dev.

config · high confidence

The DAG execution engine has been restructured to support pluggable adapters for different languages, introducing a new \Gust.DAG.Adapter\ module that manages parser, runtime, and task worker implementations. The previous \Scheduler\, \RunRestarter\, and \StageRunnerSupervisor\ modules have been removed in favor of a new gateway-based architecture, while the \TaskWorker\ has been renamed and updated to handle task execution via the adapter system. New capabilities include a \NonRecError\ for immediate task failure without retry, a \TaskWaiter\ behavior for resuming tasks parked on external events, and a \TaskExpander\ with a \MapOver\ implementation to dynamically expand tasks based on upstream results. Additionally, the \StageCoordinator\ now supports skipping tasks, waiting for external conditions, and restarting tasks, while the \Terminator\ interface has been simplified to focus on canceling tasks and stopping runs.

apps/gust/lib/gust/dag · high confidence

The DAG parser has been refactored to support multiple file formats through a new adapter system. Instead of hardcoding support for only \.ex\ files, the parser now iterates over registered adapter modules, allowing new languages to be plugged into Gust. This change introduces a new \Gust.DAG.Parser.Adapter\ behavior for defining parsers and extensions, and removes the previous \Gust.DAG.Compiler.Code\ module which handled Elixir-specific compilation and AST patching. Users can now extend the parser to handle other file types by implementing the adapter protocol.

apps/gust/lib/gust/dag/parser · high confidence

The database schema has been updated to support new task and run capabilities. The \gust\_runs\ table now includes claim fields (\claimed\_by\, \claim\_expires\_at\, \claim\_token\) and a \params\ map field. The \gust\_tasks\ table has been extended with \map\_index\ for dynamic mapping, \params\ for task parameters, \waiting\_for\ and \wait\_satisfied\_at\ for wait states, and \retry\at\ for retry logic. Additionally, existing tables (\dags\, \runs\, \tasks\, \logs\, \secrets\) have been renamed with the \gust\\ prefix to avoid naming collisions.

apps/gust/priv/repo · high confidence

The Gust flows engine now supports more granular run and task states, introducing 'enqueued', 'starting', 'waiting', and 'skipped' statuses to better reflect execution progress. Runs can now carry arbitrary parameters and support claim-based distributed execution via new fields like \claimed\_by\ and \claim\_token\. Tasks are enhanced with support for map-based parallelism (\map\_index\), explicit waiting conditions (\waiting\_for\, \wait\_satisfied\_at\), and retry scheduling (\retry\at\). Additionally, all database tables have been prefixed with 'gust\' to prevent naming collisions, and unique constraints have been added to ensure task identity within a run.

apps/gust/lib/gust/flows · high confidence

The file monitoring worker now delegates file parsing to a pluggable adapter system rather than using a hardcoded parser. When a file event is detected, the worker determines the appropriate adapter based on the file extension and passes the path to that adapter for parsing. This change enables support for new file types and languages by allowing new adapters to be plugged into the system, while maintaining the existing behavior of broadcasting parsed DAG data to the loader.

_apps/gust/lib/gust/file\monitor · high confidence

The system now uses a dedicated \GustPy.TaskWorker.Adapter\ to manage Python task execution, replacing the previous implementation. This adapter handles the lifecycle of the Python process (including starting, stopping, and handling kills), decodes task results and errors from the Python side, and logs standard error output. A new \GustPy.TaskWorker.Error\ module provides structured error reporting for task failures.

_apps/gust\_py/lib/gust\_py/task\worker · high confidence

The GustPy application now exposes extensible interfaces for executing Python tasks and communicating with the Python runtime. A new Executor module provides a dispatch layer for running commands and starting DAG tasks, defaulting to a UV-based implementation, while a TaskMessenger module handles decoding and replying to messages from the Python side, defaulting to JSON. These changes replace the previous direct port-based communication with a structured, configurable messaging system that allows the underlying implementation to be swapped via application configuration.

_apps/gust\_py/lib/gust\py · high confidence

A new \Requeue\ trigger module has been introduced to handle the resetting and re-queuing of DAG runs and tasks. This component implements the \Gust.DAG.Run.Trigger\ behavior, providing specific logic to reset entire runs or downstream task branches, update run statuses to \:enqueued\, and broadcast PubSub events. It integrates with the \RunGateway\ to manage restarts and utilizes the \Dispatcher\ to enqueue runs, ensuring that enabled DAGs are properly dispatched after a reset operation.

apps/gust/lib/gust/dag/run/trigger · high confidence

The DAG runner now supports configurable error reporting and run triggering via new behavior modules. ErrorReporter defines a contract for reporting terminal task errors to external services (e.g., Sentry or custom HTTP endpoints), with asynchronous worker handling that captures exceptions and optionally stores external references in the task's error map. Trigger introduces a behavior for resetting, requeuing, and dispatching DAG runs and tasks, delegating to a configured implementation (defaulting to Gust.DAG.Run.Trigger.Requeue). These changes enable users to customize error tracking and run control logic without modifying core runner code.

apps/gust/lib/gust/dag/run · high confidence

The run execution subsystem has been replaced with a new architecture consisting of a Claimer, Dispatcher, and Pooler. The Claimer (Gust.Run.Claimer) is a GenServer that periodically claims runnable DAG runs from a durable queue and starts their execution via the RunnerSupervisor. The Dispatcher (Gust.Run.Dispatcher) provides a configurable abstraction for enqueuing runs, with the default implementation being the Pooler (Gust.Run.Pooler). The Pooler manages the polling loop, updates run statuses to 'enqueued', broadcasts status changes via PubSub, and wakes the Claimer to process new runs. This structure introduces a clear separation of concerns between dispatching, claiming, and execution, replacing the previous monolithic run handling logic.

apps/gust/lib/gust/run · high confidence

The Python runtime adapter now supports full task lifecycle management, including the ability to kill running tasks via the runtime protocol, execute a callback upon task completion to report status, and handle setup and teardown by copying DAG files to a temporary directory and cleaning them up afterward.

_apps/gust\_py/lib/gust\py/runtime · high confidence

The DAG terminator implementation has been refactored to replace the previous direct process-based termination logic with a new Gateway module. The old \process.ex\ module, which handled task cancellation and timer cancellation by directly killing processes and sending messages via the Registry, has been removed. It is replaced by \gateway.ex\, which implements the \Gust.DAG.Terminator\ behavior by delegating to \RunGateway\ for both task cancellation and run stopping. This change centralizes termination logic through a gateway interface, improving how run statuses are handled during graceful stops.

apps/gust/lib/gust/dag/terminator · high confidence

The DAG runner has been restructured to replace the monolithic StageWorker with a modular architecture. A new RunGateway (with a default implementation) now handles command routing and lease protection for distributed runs, while a StageBuilder manages task expansion and reconciliation. Task execution logic has been extracted into a dedicated TaskExecution module, and the DAGWorker has been updated to use a StageCoordinator for state management, supporting features like task cancellation, retry countdowns, and dynamic mapped task results.

apps/gust/lib/gust/dag/runner · high confidence

The cron scheduling logic has been restructured to support dynamic job lifecycle management. A new JobLoader component now handles the creation, reloading, and deletion of scheduled jobs in response to DAG file changes, including automatic removal of jobs when their corresponding DAG files are deleted. The underlying scheduler module has been renamed and relocated to Gust.DAG.Run.Cron.Scheduler. Additionally, the system now supports hot-reloading of cron schedules during development via a configurable reload option, improving developer experience when modifying DAG definitions.

apps/gust/lib/gust/dag/run/cron · high confidence

The application's visual theme has been updated to use a more consistent, high-contrast color palette across both light and dark modes, shifting from generic gray/blue tones to specific semantic colors (e.g., sky, rose, emerald). This change also introduces a visible keyboard focus indicator for better accessibility and adds specific styling for new task states, including 'skipped' and 'waiting', ensuring they are visually distinct in the interface.

_apps/gust\web/assets/css · high confidence

Gust now supports distinct runtime roles (console, web, core, single) to control which background processes are active, such as DAG dispatching and file watching, based on the GUST\_ROLE environment variable. A new CLI entry point (gust-cli) allows triggering DAG runs with optional JSON parameters and inspecting DAG definitions. The DAG execution engine has been refactored to use a PostgreSQL-based notification system for run dispatching and introduces a leader election mechanism for distributed locking. Additionally, the DSL now supports new task options including skip\_if, map\_over, and wait\_for, while the mailer has been removed.

apps/gust/lib/gust · high confidence

The error reporting system for DAG runs has been enhanced to handle external error references. A new GenServer worker processes error reports and, upon receiving a valid external reference from the configured reporter, attaches it to the corresponding task via the Flows service and broadcasts the update. This includes validation of the reference (ensuring it is a valid HTTP/HTTPS URL) and handles edge cases such as stale tasks or invalid reference formats by logging appropriate messages.

_apps/gust/lib/gust/dag/run/error\reporter · high confidence

The stage coordinator has been refactored to support skipping tasks when upstream dependencies are skipped, and to handle wait conditions more explicitly. Previously, only upstream failures would stop downstream tasks; now, if any upstream task is skipped, dependent tasks are also skipped. Additionally, the coordinator now tracks waiting tasks separately and respects wait satisfaction timers, allowing tasks to pause execution until specific conditions are met. This change improves the reliability of DAG execution by ensuring that skipped or waiting states are correctly propagated through the dependency graph.

_apps/gust/lib/gust/dag/stage\coordinator · high confidence

Added test coverage for assets, MCP, and well-known controllers; removed page controller test; Added test helper with Mox mocks for GustPy; Added test support helpers for environment manipulation and gateway peer simulation; Added tests for API authentication plug; Added tests for API, dashboard, MCP, and Mermaid chart modules; Added tests for DAG adapter, folder, task status, waiter, and terminator components; Added tests for DAG run requeue and mapped task reset logic; Added tests for DBLocker Postgres module; Added tests for Elixir task worker adapter; Added tests for MCP tools server and tool definitions; Added tests for Task Messenger JSON handling; Added tests for claim repository run management; Added tests for core layout and DAG run UI components; Added tests for external reference validation; Added tests for the DAG Run Gateway default implementation; Added tests for the Gust CLI mix task; Added tests for the Gust Web installation task; Added tests for the MCP server resource and tool modules; Added tests for the Python DAG parser adapter; Added tests for the Python executor's UV integration; Added tests for the Python runtime adapter; Added tests for the Task Worker adapter; Added tests for the cron job loader; Added tests for the refactored DAG runner architecture; Added tests for the renamed RetryingTask component; Added tests for the run claimer, pooler, and dispatcher supervisor; Expanded test coverage for Gust runtime and CLI features; Expanded test coverage for live dashboard, system, and breadcrumb components; Reformat DAG validation tests; Removed tests for DAG Scheduler Worker; Removed tests for the DAG RunRestarter worker; Test helpers now import DashboardPath sigil; Updated test mocks and configuration for new DAG and MCP components; Updated tests for DAG parser adapter delegation; Updated tests for the DAG task runner supervisor.

This change introduces a new \gust\_py\ application (version 0.1.12) to support Python DAGs, alongside a comprehensive update of the project's Elixir dependencies. Key version bumps include \bandit\ (1.8.0 to 1.12.0), \ecto\ and \ecto\_sql\ (3.13.x to 3.14.0), \phoenix\ (1.8.0 to 1.8.8), \phoenix\_live\_view\ (1.1.3 to 1.2.1), \plug\ (1.18.1 to 1.19.2), and \postgrex\ (0.21.1 to 0.22.2). The \gust\_web\ app now conditionally includes \gust\_py\ via the \GUST\_WITH\_PYTHON\ environment variable and updates its static asset file inclusion. The \gust\ app removes the \swoosh\ dependency and updates its GitHub link. The umbrella release configuration now conditionally includes \gust\_py\ in the release applications.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 52.

• Code Health 98
• Architecture 93
• Maturity 67
• Readiness 38
• Security 69
• Accessibility 53

• 300 commits — 260 feature/other, 40 fixes

• apps/gust_web — 103 commits
• (repo) — 74 commits
• apps/gust — 69 commits
• (root) — 36 commits
• apps/gust_py — 5 commits
• .github/workflows — 4 commits
• skills/elixir-dag-creator — 3 commits
• examples/dags — 2 commits
• examples/docker — 2 commits
• config/prod.exs — 1 commit
• config/runtime.exs — 1 commit

• fix: Merge pull request #72 from marciok/fix/70-disabled-dag-api-runs
• fix: Merge pull request #84 from marciok/fix-83/handle-tasks-error-exists
• fix: Merge pull request #85 from marciok/fix-asset-download
• fix: Merge pull request #88 from marciok/fix-double-js-importing
• fix: Revert "Pagination queries should have a deterministic order."
• fix: chore: fix docker files
• fix: fix #70
• fix: fix trigger run per copilot review
• fix: fix: #104
• fix: fix: #22 update task after cancelling it
• fix: fix: #83 handle process exits
• fix: fix: #86 load only the necessary fields
• fix: fix: #87 remove double js importing
• fix: fix: asset test
• fix: fix: async task failure leaked
• fix: fix: bump gust_py version
• fix: fix: bump version
• fix: fix: duplicated css
• fix: fix: ensure data is parsed safely
• fix: fix: job name atom does not exists
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

marciok/gust was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.