ghostwright/ghost-os

Ghost OS is a macOS automation system that enables AI agents to interact with desktop and web applications through a combination of accessibility trees, visual grounding, and recorded workflows. It provides tools for capturing screenshots, annotating UI elements, and executing precise interactions like clicks and drags, while supporting self-learning mode to generate automation recipes from user actions. The system integrates a local vision sidecar for visual element detection and includes diagnostic utilities to manage permissions and configuration.

Adds the Vision Sidecar component, providing a local HTTP server (port 9876) that uses the MLX-powered ShowUI-2B model for precise visual grounding of UI elements. The sidecar includes a launcher script (\ghost-vision\) that automatically detects the Python environment and a server implementation that handles health checks and grounding requests, replacing previous PyTorch dependencies with MLX-compatible libraries (mlx-vlm, transformers\<4.49) to ensure compatibility with Apple Silicon hardware.

vision-sidecar · high confidence

Ghost OS now includes a \ghost setup\ command that guides users through first-time configuration, including detecting the host terminal, requesting Accessibility and Screen Recording permissions, configuring the MCP connection to Claude Code, installing bundled recipes, and setting up the vision sidecar. A new \ghost doctor\ command provides a non-interactive health check that diagnoses issues with permissions, processes, and configuration, offering specific fixes for common problems like stale MCP processes or missing recipes. The existing \ghost status\ command has been simplified to provide a quick overview and direct users to \ghost setup\ if permissions are missing.

Sources/ghost · high confidence

Added a new self-learning subsystem in the Learning module that allows users to record their interactions to generate automation recipes. The system captures clicks, keystrokes (with coalescing), scroll events, and app switches via a background CGEvent tap, while enriching data with Accessibility element context. It includes secure field detection to prevent recording sensitive input, handles app switching detection via NSWorkspace polling, and exposes MCP tools (ghost\_learn\_start, ghost\_learn\_stop, ghost\_learn\_status) to control the recording lifecycle and retrieve the serialized action history.

Sources/GhostOS/Learning · high confidence

Ghost OS v2 introduces a new vision subsystem that allows the agent to locate and interact with UI elements when standard accessibility trees are insufficient. This includes a new \ghost\_ground\ tool that uses a local Python sidecar running the ShowUI-2B model to find precise screen coordinates for described elements via screenshot analysis, and a \ghost\_parse\_screen\ tool for detecting interactive elements (currently returning sidecar status as YOLO detection is pending). For web applications, a new \CDPBridge\ provides a faster, DOM-based alternative to accessibility trees by connecting directly to Chrome's debugging port, enabling instant element finding via CSS selectors and JavaScript evaluation.

Sources/GhostOS/Vision · high confidence

The Perception module now includes a new \ghost\_annotate\ tool that captures screenshots and overlays numbered labels on interactive UI elements to assist visual agents. The existing \ghost\_find\ element search has been hardened with per-element accessibility timeouts to prevent hangs in complex apps like Chrome, and now features a multi-stage fallback strategy: if standard accessibility tree searches fail, it attempts a Chrome DevTools Protocol (CDP) lookup, and finally falls back to vision-based grounding for web applications.

Sources/GhostOS/Perception · high confidence

The MCP server now exposes 29 tools, adding vision-based detection (ghost\_parse\_screen, ghost\_ground), screenshot annotation (ghost\_annotate), and self-learning workflow recording (ghost\_learn\_start, ghost\_learn\_stop, ghost\_learn\_status). Existing interaction tools have been expanded with hover, long-press, and drag actions, while wait conditions now support URL and title change detection. To prevent the server from hanging on frozen applications, the AX messaging timeout is set to 5 seconds, and the transport layer now correctly matches input framing (NDJSON vs. Content-Length) for output. Screenshot and annotate tools return inline image content for better display, and the server now loads instructions from Homebrew paths in addition to development directories.

Sources/GhostOS/MCP · high confidence

The RecipeEngine now implements full step-by-step execution with parameter substitution, smart precondition checks (verifying app status with diagnostics), focus management, and detailed failure reporting including screenshots and context. The RecipeStore has been updated to log decode errors for broken recipe files instead of silently skipping them, and provides specific error details for invalid JSON when saving recipes.

Sources/GhostOS/Recipes · high confidence

The ScreenshotResult type has been updated to include a mimeType field (defaulting to image/png) and four new fields (windowX, windowY, windowWidth, windowHeight) that expose the window frame in logical screen coordinates. This allows consumers of screenshot data to identify the image format and map visual elements back to their position on the screen, supporting improved coordinate mapping for vision-based interactions.

Sources/GhostOS/Common · high confidence

The screenshot capture mechanism in GhostOS now uses the synchronous \CGWindowListCreateImage\ API instead of the asynchronous ScreenCaptureKit. This change resolves reliability issues on macOS 26 where the previous async approach failed to dispatch task continuations correctly. The new implementation provides a dedicated synchronous capture path (\captureWindowSync\) for the MCP server, includes specific error reasons for debugging capture failures (such as missing permissions or unavailable windows), and retains the existing behavior of downscaling wide images to a maximum width of 1280px while supporting full Retina resolution when requested.

Sources/GhostOS/Screenshot · high confidence

A new \scripts/build-release.sh\ script automates the creation of the Ghost OS v2 release tarball. It builds the Swift binary, stages all necessary files (including the vision sidecar and recipes), and generates the final archive. A key fix in this script ensures the macOS binary is re-signed after being copied to the staging directory, preventing linker-signed adhoc signatures from breaking during the copy process.

scripts · high confidence

The Actions module has been rewritten to use AXorcist's command system (runCommand with Locators) instead of direct element action methods, introducing a structured action loop with pre-flight checks, execution, and post-verification. The click action now supports coordinate-based clicks, AX-native clicks via PerformActionCommand, and fallbacks to CDP (Chrome DevTools Protocol) or Vision-based grounding for web apps. FocusManager now retries app activation twice and polls for up to 1 second to verify focus, improving reliability for app switching.

Sources/GhostOS/Actions · high confidence

The vision sidecar now explicitly pins its Python dependencies in requirements.txt, specifically constraining transformers to versions below 4.49 to avoid an unwanted PyTorch dependency and setting mlx-lm below 0.30.0 to prevent resolver conflicts. Additionally, the AXorcist dependency has been switched from a local path reference to a remote Git URL (steipete/AXorcist) with a pinned version of 0.1.0.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 56 → 55 (-0.7)
• Rubric changed (rubric-2026.09.11 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 78 → 78 (+0.2)
• Architecture 100 → 97 (-3.3)
• Maturity 61 → 58 (-2.1)
• Readiness 33 → 36 (+2.4)
• Security 100 → 100 (+0.0)

• Dependency hygiene PARTLY measured — SwiftPM pinning read, dependency currency NOT established
• Documentation: contradicts the code (docs/progress.md)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Hotspot: Sources/GhostOS/Screenshot/ScreenCapture.swift (Sources/GhostOS/Screenshot/ScreenCapture.swift)

• Coverage not measured — Swift suite
• Duplicate Screenshot Functionality with Different Signatures: ScreenCapture provides low-level capture by PID, while Perception provides high-level capture by App Name. Both return screenshot data but in different formats (ScreenshotResult vs ToolResult). This creates confusion for users on which API to use for capturing the screen.
• Inconsistent Error Handling: LearningRecorder.start returns an optional LearningError? directly, while stop returns a Result enum. This forces callers to handle errors differently depending on the method.
• Inconsistent Locator/Query Parameterization: The Actions class uses a flat set of optional parameters (query, role, domId) for targeting elements, while Perception methods also use these but with varying subsets and additional parameters (domClass, identifier, depth). There is no unified 'Locator' object passed to these methods, leading to signature drift and ambiguity about which parameters are valid together.
• Off the main sequence: GhostOS
• Orphaned files with no living knowledge
• Outdated: axorcist

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

ghostwright/ghost-os was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.