abi/screenshot-to-code

This release delivers a comprehensive backend refactoring that introduces modular components for asset extraction, cost tracking, and evaluation runs, alongside a new agent engine with output validation and budget controls. The frontend has been significantly upgraded with a unified input pane, screen recording capabilities, and a new Zustand-based state management system. Additionally, the update includes a new Evals dashboard for structured testing, improved select-and-edit UX, and a suite of new UI components and hooks to enhance the overall development and user experience.

A new ThinkingIndicator component has been added to the frontend, providing a visual indicator for the AI's reasoning process. The component displays different states: a pulsing 'AI is thinking...' message while the AI is generating, a 'reasoning' status during active thought, and a completed state showing the duration of the thought process. The component is designed to only display for the latest commit, hiding the indicator when viewing historical commits, and includes an expandable view to show the full reasoning text or a truncated preview.

frontend/src/components/thinking · high confidence

Users can now record their screen to generate code from a video input. The new ScreenRecorder component captures the display stream, converts the resulting WebM video to a base64 data URL, and displays a preview of the recording. Users can then choose to re-record or proceed to generate code from the captured video.

frontend/src/components/recording · high confidence

The backend now supports uploading and persisting image assets. A new \uploaded\_assets\ module provides a \save\_assets\ tool that accepts temporary image IDs, promotes them to permanent public URLs, and returns the asset details. The system handles image storage, metadata management, and serves assets locally. This enables models to reference uploaded images in generated code.

_backend/uploaded\assets · high confidence

The backend now includes new utility functions for handling video data. Specifically, extract\_tag\_content allows for extracting content from XML-style tags, and get\_video\_bytes\_and\_mime\_type decodes base64-encoded video data into bytes and identifies the MIME type. These utilities support the broader video-to-code feature by providing the necessary parsing and decoding capabilities.

backend/video · high confidence

A new shell script, scripts/cursor-cloud-install.sh, has been introduced to automate the installation of dependencies for the Cursor Cloud environment. This script installs Python's Poetry, sets up the backend with Playwright, and installs frontend and admin dependencies using pnpm, streamlining the setup process for this specific configuration.

scripts · high confidence

The backend codebase has been reorganized into distinct modules for better maintainability and feature expansion. Key changes include: a new \asset\_extraction.py\ module for handling screenshot asset detection using the Gemini API; a \costs\ package (\pricing.py\, \token\_usage.py\) to track and cap LLM generation costs; a \debug\ module (\DebugFileWriter.py\) for writing request logs to files; and a \utils.py\ module with prompt formatting and preview utilities. Additionally, the \llm.py\ file now explicitly maps model names to their respective providers (OpenAI, Anthropic, Gemini) and includes reasoning effort configurations. The \main.py\ entry point has been updated to include new routes for capabilities, screenshots, code generation, evals, and more. A new \start.py\ script allows the backend to find an available port. These changes support new features like image generation, video uploads, and comprehensive evaluation runs.

backend · high confidence

The backend/evals directory now contains a new evaluation framework that supports running code-generation benchmarks against multiple models and stacks. The system supports both image-based and text-based eval sets, allowing users to run structured evaluations that track session state, model metadata, and per-run timing. The runner handles retries, budget limits, and local asset URL normalization, while sessions and sets provide metadata for organizing and reviewing results.

backend/evals · high confidence

The frontend now includes a new design systems client that provides a standardized HTTP interface for creating, fetching, updating, and deleting design systems, exposing a clean API for UI components to consume. Additionally, the models module has been updated to include new model variants (such as GPT 5.6 Sol, GPT 5.6 Terra, and various GPT 5.5 and Claude Opus 5/4.8 options) and now supports labeling model variants with badges like 'Fast' or 'Max' in the UI, depending on the input mode and generation type. The changes also include a utility for normalizing Babel CDN URLs to ensure consistent React rendering, and a helper for taking screenshots of preview iframes.

frontend/src/lib · high confidence

Users can now create, edit, and delete custom design systems (collections of color, typography, and layout rules) via a new DesignSystemsManager component, which is exposed through a DesignSystemsModal. The settings interface has been restructured into a full-page SettingsTab that groups theme, API key, and generation settings. The GenerationSettings component now accepts and manages design system selection, and the OutputSettingsSection integrates the new DesignSystemSelector alongside the existing stack selector.

frontend/src/components/settings · high confidence

Users can now generate code from a text prompt alone, without needing to upload a screenshot, via a new 'Generate from text' interface. Additionally, the application now includes a suite of evaluation and testing pages, including views for run results, best-of-N comparisons, prompt reports, and agent activity logs, accessible via the /evals routes.

frontend/src · high confidence

A new unified input pane has been added to the frontend, consolidating image, URL, text, and code import tabs into a single interface for code generation. The pane includes controls for selecting the target stack and design system, an optional asset extraction toggle, and an expandable section for additional text instructions.

frontend/src/components/unified-input · high confidence

The start pane now renders a new UnifiedInputPane component, consolidating the interface for code generation. This change introduces support for optional text prompts when uploading images, allows for asset extraction preferences in code generation, and manages persistent design systems for prompts, providing a more integrated experience for users initiating code generation tasks.

frontend/src/components/start-pane · medium confidence

The unified input interface now provides four distinct tabs for generating code: Import (for pasting or dragging HTML files), Text (for writing prompts with example suggestions), Upload (for multiple images/videos), and URL (for capturing screenshots of public webpages). Each tab includes its own stack selector and design system options, allowing users to choose their preferred input method for code generation.

frontend/src/components/unified-input/tabs · high confidence

The frontend now includes a new Evals Dashboard (AllEvalsPage) that provides a central hub for navigating to various evaluation tools, including a new Agent Runs Page (AgentRunsPage) for viewing full agent-run logs with a timeline view, alongside existing and new pages for Best-of-N comparisons, session matrices, prompt reports, and input comparison. The navigation bar (EvalNavigation) has been updated to include links to all these new and existing eval tools.

frontend/src/components/evals · high confidence

A new set of reusable UI components has been added to the application, including an accordion, alert dialog, badge, button, checkbox, collapsible, dialog, hover card, input, label, popover, progress, scroll area, select, separator, switch, tabs, and textarea. These components, built on top of Radix UI and Tailwind CSS, provide a consistent and accessible foundation for the user interface.

frontend/src/components/ui · high confidence

The backend now exposes several new API endpoints to support evaluation workflows and system configuration. The \/eval-sets\ and \/eval-sessions\ routes allow users to manage and run evaluation sets, while \/agent-runs\ provides a viewer for recorded agent runs. Additionally, \/api/design-systems\ enables CRUD operations for persistent design systems, and \/api/capabilities\ exposes feature availability to the frontend.

backend/routes · high confidence

The preview pane now includes a dedicated 'Code' tab that displays the generated code in a syntax-highlighted editor (CodeMirror) with copy and CodePen integration. A version navigation bar has been added to the top of the preview pane, allowing users to browse previous versions of their project. Additionally, the pane now supports downloading the generated code as a ZIP file, and handles image scanning previews for the initial upload state.

frontend/src/components/preview · high confidence

Added the ImportCodeSection component, which provides a dialog for pasting HTML code and selecting a target stack, enabling users to import existing code. Added the UpdateImageUpload component, which allows users to upload and preview multiple reference images (up to 5) for updates. Added the TermsOfServiceDialog component, which collects user emails and displays partner logos on the hosted version.

frontend/src/components · high confidence

The frontend now includes several new core components: a KeyboardShortcutBadge for displaying keyboard shortcuts with special handling for the Enter key, a Spinner for loading states, a StackLabel that renders framework logos and names for different tech stacks, and a WorkingPulse indicator. These components provide consistent visual feedback and branding for framework selection and status updates.

frontend/src/components/core · high confidence

Added four new React hooks in the frontend: useBrowserTabDynamicTitle and favicon updates for coding states, a generic usePersistedState hook for localStorage-backed state, a useThrottle hook for rate-limiting updates, and a useDesignSystems hook for fetching and managing design system data. These hooks provide reusable abstractions for UI state management, local storage persistence, and performance optimization.

frontend/src/hooks · high confidence

The backend now captures comprehensive, on-disk logs for every agent run, including full LLM request payloads, token usage, and costs, as well as tool call details and streamed deltas. Additionally, the system records individual LLM prompt reports (gated by the PROMPT\_REPORTS\_ENABLED flag) and provides utilities to compare OpenAI input payloads. These logs are stored in the filesystem and indexed in a local SQLite database, making the data available for review via the new /evals/agent-runs and /evals/prompt-reports viewers.

_backend/fs\logging · high confidence

The agent's tooling has been restructured into a new modular package under backend/agent/tools. This introduces a typed runtime (AgentToolRuntime) that dispatches tool calls, along with dedicated modules for tool definitions, parsing, summaries, and specific tools like extract\_assets and screenshot\_preview. The refactoring enforces a strict invariant on ToolMultimodalPart: local (localhost) image URLs are converted to base64 data URLs or bytes before being sent to external models, preventing failures with providers like Anthropic and OpenAI that cannot fetch localhost addresses. The change also adds support for a Replicate API key in settings and improves error handling for asset extraction.

backend/agent/tools · high confidence

A new Terms of Service page has been added to the public legal files, establishing the legal framework for using the AI screenshot-to-code generation services. The page outlines user responsibilities, intellectual property rights, payment and credit policies, and liability limitations, specifically noting that uploaded materials may be used to train and improve AI models.

frontend/public/legal · medium confidence

The Variants component now supports switching between generated code variants using Alt + number keys (⌥1–⌥9), allowing faster navigation without a mouse. Each variant tile also displays a color-coded badge (e.g., Flash, Max) indicating the underlying model, and shows a status indicator (green for complete, red for errors) alongside a live preview iframe. This change improves usability by making variant switching more efficient and providing clearer visual feedback about each option's status and model.

frontend/src/components/variants · medium confidence

The Anthropic provider now automatically resizes base64-encoded images when a request contains more than 20 images, enforcing a 2000x2000 pixel dimension limit. This change ensures compliance with Anthropic's stricter API constraints for multi-image requests, preventing potential rejections or errors by processing images through the new \image.py\ module before sending them to the API.

backend/agent/providers/anthropic · high confidence

The frontend project has been restructured with a new Vite-based build system, introducing a \vite.config.ts\ that configures a dev-server proxy for backend endpoints (\/generate-code\, \/api\, \/local-assets\) to support same-origin deployments and tunnelled previews. A \tsconfig.json\ and \tsconfig.node.json\ are added to enforce strict TypeScript checking, while \tailwind.config.js\ and \components.json\ configure Tailwind CSS and shadcn/ui. The \index.html\ template is updated to include Open Graph and Twitter meta tags and inject Plausible analytics scripts when the \VITE\_IS\_DEPLOYED\ environment variable is set. Additionally, a \.env.example\ file is added to document the \VITE\_WS\_BACKEND\_URL\ variable, and a \jest.config.js\ is introduced for test coverage.

frontend · high confidence

The agent execution logic has been refactored into a new \AgentEngine\ class in \backend/agent/engine.py\. This update introduces validation to fail runs that finish without producing any HTML output, preventing silent failures in evaluations. It also enforces a spend ceiling, aborting generations that exceed a defined resource limit. Additionally, the engine now supports extracting and reusing assets from screenshots, while explicitly excluding video-only prompts from asset extraction to avoid tool failures.

backend/agent · high confidence

The select-and-edit feature now provides clearer visual feedback and easier exit. A new toolbar button lets users toggle select-and-edit mode, with the button text changing to "Exit select mode" while active. The preview area shows two distinct overlay states: a hover state (light ring, soft fill) and a selection state (solid offset ring, stronger fill, and a checkmark label) so users can easily distinguish the locked target from the current hover. Pressing Escape exits select-and-edit mode. Additionally, the system now generates precise element context (DOM path and duplicate-count warnings) to help the AI target the correct element when multiple identical elements exist, and truncates large HTML snippets to keep prompts manageable.

frontend/src/components/select-and-edit · high confidence

The agent activity view now displays a chronological timeline of the agent's steps, with assistant responses and tool calls grouped to preserve the original order while collapsing consecutive thinking and tool events into compact step groups. Completed variants show a stable, concise duration (e.g., '1m 55s') that does not change as the page loads, and the activity display is updated to show plain text for assistant messages and collapsible steps when they are done.

frontend/src/components/agent · medium confidence

The backend now uses dedicated modules to construct update prompts from either a file snapshot or conversation history. New functions, build\_update\_prompt\_from\_file\_snapshot and build\_update\_prompt\_from\_history, generate system and user messages by combining selected stack policies, image generation policies, and optional design system blocks. This refactors how update prompts are assembled, ensuring consistent inclusion of stack context, image policies, and design system instructions in both snapshot and history-based update flows.

backend/prompts/update · high confidence

Added new TypeScript types for Commit and Variant states, including support for agent events, thinking models, and backend error messages. This refactors the frontend's data layer to explicitly track variant status, agent events, and model labels, enabling non-blocking updates and better error handling in the UI.

frontend/src/components/commits · medium confidence

The backend image generation logic has been refactored to use the Replicate API, replacing the previous implementation. This change introduces support for the 'z\_image\_turbo' and 'flux\_2\_klein' models, with 'z\_image\_turbo' set as the default. The new implementation includes a background removal tool and an image editing tool, both backed by Replicate. Concurrent API requests are limited to 20 per batch to manage load.

_backend/image\generation · high confidence

The frontend state management has been refactored to use a new Zustand-based store (app-store.ts and project-store.ts). This introduces new capabilities for handling multiple reference images, select-and-edit mode, and variant generation states. The store now tracks commit history, variant statuses, and UI states like the update instruction and selected elements. Tests have been added to verify the new store's behavior, particularly for head navigation and variant completion timestamps.

frontend/src/store · high confidence

The backend now uses a new \backend/prompts/create\ package to generate AI prompts for three distinct input modes: image, text, and video. Each mode has its own builder (\build\_image\_prompt\_messages\, \build\_text\_prompt\_messages\, \build\_video\_prompt\_messages\) that constructs specific system and user messages. This refactoring separates the prompt generation logic by input type, allowing for more targeted instructions—for example, image prompts now include detailed guidance on asset extraction and upscaling, while video prompts focus on replicating interactions and functionality shown in the video.

backend/prompts/create · high confidence

The history component has been refactored to display a chronological list of versions, each showing a thumbnail preview of images or videos, a version number badge, and a styled pill badge indicating the type of edit (Create, Edit, or Imported from code). The new layout includes an expanded view for media and supports auto-playing videos when a thumbnail is clicked. The underlying utility functions have been updated to pre-process history data for display, enabling better test coverage and cleaner separation of concerns.

frontend/src/components/history · high confidence

The sidebar has been redesigned to feature a vertical icon strip on the left, providing quick access to the Editor, Versions (History), New Project, and Settings. This new layout includes a dedicated history panel for navigating between code versions, a prominent prompt box pinned to the bottom, and improved visual hierarchy for variant generation and error messages.

frontend/src/components/sidebar · medium confidence

The screenshot preview feature has been refactored to support pluggable rendering backends. The default implementation now uses a local, headless Chromium browser via Playwright, which allows the tool to render pages that load assets from localhost. The architecture introduces a \ScreenshotBackend\ interface, enabling deployments to swap in alternative renderers (such as an external API) without changing the core application code. The preview tool now probes for backend availability at startup and gates the feature if the backend is unavailable.

_backend/preview\screenshot · medium confidence

The backend's prompt generation has been reorganized into a structured, type-safe pipeline. A new \pipeline.py\ module routes requests to specific builders based on a derived construction plan, separating logic for creating new prompts versus updating existing ones. This includes a new \message\_builder.py\ that correctly formats user history with mixed media (images, videos, text) for the LLM, and a \plan.py\ module that determines whether to use history or file snapshots for updates. The system prompt (\system\_prompt.py\) has been expanded to include detailed, stack-specific instructions for React, Vue, Ionic, and Bootstrap, along with updated guidance for image manipulation and element selection. This change ensures type safety, reduces code duplication, and clarifies how prompts are assembled for different generation types.

backend/prompts · high confidence

The agent provider layer has been restructured into a modular, provider-specific architecture. Each supported LLM (OpenAI, Anthropic, and Gemini) now has its own dedicated implementation file, all unified under a new \providers\ package that exposes a common \ProviderSession\ interface. This change introduces unified token usage tracking across all providers, meaning users will see consistent cost and token metrics regardless of which model is selected. Additionally, the factory function now dynamically enables or disables specific agent tools (such as image editing and asset extraction) based on the availability of required API keys (e.g., Replicate, Gemini) and configuration flags, ensuring that features like screenshot previews and image generation are only offered when the necessary credentials are present.

backend/agent/providers · high confidence

Added deterministic end-to-end QA test suite; Adds comprehensive backend tests for the agent engine, tool runtime, and asset extraction.

The project's Python and JavaScript dependencies have been updated. The backend now uses Anthropic SDK 0.84.0, OpenAI SDK 2.16.0, FastAPI 0.115.6, and Playwright 1.61.0. The frontend has been migrated to use pnpm as its package manager, and dependencies such as React 18.3.1, Vite 6.4.1, and Radix UI components have been updated to their latest compatible versions.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 41 → 41 (-0.3)
• Rubric changed (rubric-2026.08.15 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 76 → 70 (-5.9)
• Architecture 94 → 85 (-8.9)
• Maturity 55 → 61 (+6.2)
• Readiness 12 → 23 (+11.1)
• Security 78 → 82 (+3.5)
• Accessibility 42 (new)

• Change coupling: App.tsx ↔ IconStrip.tsx (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ ImportCodeSection.tsx (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ SettingsTab.tsx (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ StartPane.tsx (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ TermsOfServiceDialog.tsx (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ app-store.ts (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ generateCode.ts (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ select.tsx (frontend/src/App.tsx)
• Change coupling: App.tsx ↔ utils.ts (frontend/src/App.tsx)
• Change coupling: constants.ts ↔ types.ts (frontend/src/constants.ts)
• Coverage not measured — test suite did not build
• Critical CVE: [GHSA redacted] (frontend/pnpm-lock.yaml)
• Critical CVE: [GHSA redacted] (backend/poetry.lock)
• Dimension evaluation failed
• Duplicated block (12 lines × 2) (backend/agent/tools/runtime.py)
• Duplicated block (12 lines × 2) (backend/tests/test_prompts.py)
• Duplicated block (5 lines × 2) (backend/tests/test_prompts.py)
• High CVE: [GHSA redacted] (frontend/pnpm-lock.yaml)
• High CVE: [GHSA redacted] (backend/poetry.lock)
• High CVE: [GHSA redacted] (backend/poetry.lock)
• …and 49 more

• AgentActivity.AgentActivity (cognitive 22) (frontend/src/components/agent/AgentActivity.tsx)
• AgentActivity.AgentActivity (cyclomatic 20) (frontend/src/components/agent/AgentActivity.tsx)
• AgentActivity.getEventTitle (cognitive 122) (frontend/src/components/agent/AgentActivity.tsx)
• AgentActivity.getEventTitle (cyclomatic 55) (frontend/src/components/agent/AgentActivity.tsx)
• AgentActivity.renderToolDetails (cognitive 57) (frontend/src/components/agent/AgentActivity.tsx)
• AgentActivity.renderToolDetails (cyclomatic 111) (frontend/src/components/agent/AgentActivity.tsx)
• AgentEngine._extract_input_images (cognitive 16) (backend/agent/engine.py)
• AgentEngine._run_with_session (cognitive 32) (backend/agent/engine.py)
• AgentRunRecorder._snapshot_output (cognitive 22) (backend/fs_logging/agent_runs.py)
• AgentRunRecorder._snapshot_output (cyclomatic 16) (backend/fs_logging/agent_runs.py)
• AgentRunRecorder.record_llm_response (cognitive 19) (backend/fs_logging/agent_runs.py)
• AgentRunRecorder.record_llm_response (cyclomatic 16) (backend/fs_logging/agent_runs.py)
• AgentRunsPage.AgentRunsPage (cognitive 30) (frontend/src/components/evals/AgentRunsPage.tsx)
• AgentRunsPage.AgentRunsPage (cyclomatic 39) (frontend/src/components/evals/AgentRunsPage.tsx)
• AgentToolRuntime._edit_images (cognitive 33) (backend/agent/tools/runtime.py)
• AgentToolRuntime._edit_images (cyclomatic 22) (backend/agent/tools/runtime.py)
• App.App (cognitive 91) (frontend/src/App.tsx)
• App.App (cyclomatic 91) (frontend/src/App.tsx)
• BestOfNEvalsPage.BestOfNEvalsPage (cognitive 68) (frontend/src/components/evals/BestOfNEvalsPage.tsx)
• BestOfNEvalsPage.BestOfNEvalsPage (cyclomatic 69) (frontend/src/components/evals/BestOfNEvalsPage.tsx)
• …and 206 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

abi/screenshot-to-code was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.