apache/texera
39.8
Weak · 7 November 2026
170.1k
lines of production code
Scala
with TypeScript, Python
4
measurements over time
What this system is
Features
Add Arrow Flight end-to-end micro-benchmark
A new end-to-end micro-benchmark has been added to measure the performance of the Arrow Flight data path through a live PythonWorkflowWorker actor. This benchmark times the round-trip of DataFrames sent to and echoed by a real Python subprocess via Arrow Flight, sweeping across various batch sizes, schema widths, and string lengths. Results are exported to CSV and JSON files for integration with CI dashboards, allowing users to track throughput and latency trends over time.
amber/src/bench · high confidence
Add Iris dataset example and loader script
The single-node example setup now includes a pre-built Iris dataset (Iris.csv) with a description, along with a corresponding machine learning workflow JSON that reads the data, visualizes it, and prepares features for training. A new load-examples.sh script automates the process of registering/logging in, creating the dataset in the file service, and importing the workflow, ensuring the example is ready to run out-of-the-box.
bin/single-node/examples · high confidence
Add Sign in with Apple and ORCID authentication providers
Users can now log in to Texera using their Apple or ORCID accounts. The new Apple integration validates identity tokens against Apple's public keys and provisions accounts, handling cases where Apple does not provide an email address by creating identity-only accounts that prompt for an email later. The ORCID integration implements the full OAuth authorization code flow to authenticate users, also provisioning identity-only accounts since the ORCID scope used does not return an email address. Both providers are integrated into the existing external authentication provisioning system.
amber/src/main/scala/org/apache/texera/web/resource/auth · high confidence
Add user quota and storage usage API endpoints
Introduces the UserQuotaResource to expose user storage quotas and usage statistics via REST endpoints. Users can now query the size of their created datasets and workflows, including detailed breakdowns of execution storage (result, runtime stats, and log bytes) for each workflow execution. This provides visibility into disk usage per user, supporting quota management and storage monitoring. Also in this area: 3 commits (0 fixes) modifying amber; 3 commits (2 fixes) fixing amber.
amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/quota · medium confidence · unverified
Added Flarum forum installation and configuration scripts
New shell scripts have been added to the \bin/forum\ directory to automate the setup of a Flarum forum on macOS and Ubuntu. The \macos-install.sh\ and \ubuntu-install.sh\ scripts handle environment preparation (Homebrew/Apt, PHP, Apache, Composer), Flarum project creation, and necessary plugin installations (discussion-views, byobu). They also configure Apache virtual hosts and apply a provided SQL dump (\flarum.sql\) to initialize the database schema and seed data. A \start-flarum.sh\ script is included to launch the Apache server.
bin/forum · high confidence
Added Python and R UDF example operators
The \amber/src/main/python/pytexera/udf\ location now includes a set of example user-defined function (UDF) operators to demonstrate usage patterns. This adds Python implementations for tuple, table, batch, and source operations (including \EchoOperator\, \JoinOperator\, and \CountBatchOperator\) alongside R-based examples for tuple and table processing, providing reference code for integrating custom logic into the pipeline.
amber/src/main/python/pytexera/udf · high confidence
Added cluster node failure handling and status reporting
The Amber clustering module now includes a ClusterListener that monitors Apache Pekko cluster membership changes. When a node is removed, the system checks if fault tolerance is enabled: if so, it triggers a recovery process for active workflows; otherwise, it forcefully stops the execution. In both cases, it updates the workflow state and notifies connected clients via ClusterStatusUpdateEvent with the current number of worker nodes. A SingleNodeListener is also provided for single-node deployments to return the local address when queried.
amber/src/main/scala/org/apache/texera/amber/clustering · high confidence
Added license collection utility for third-party dependencies
A new script, collect-licenses.ts, has been added to the agent-service/bin directory to scan node\_modules and generate a JSON manifest of third-party package names, versions, and licenses. This tool is designed to be run after installation to produce output compatible with frontend license generation plugins, facilitating the creation of NOTICE and LICENSE files for binary distributions.
agent-service/bin · high confidence
Admin dashboard gains user management and execution monitoring endpoints
The admin dashboard now exposes new REST endpoints to manage users and monitor workflow executions. Administrators can view a list of users including their affiliation, joining reason, and last login time, and can update user roles or create new inactive user accounts. The system also enforces that accounts without an email address cannot be activated and prevents email removal once set. Additionally, admins can now view the latest execution status for each workflow, with support for filtering by status, sorting, and pagination, as well as viewing created datasets and workflows for specific users to support quota management.
amber/src/main/scala/org/apache/texera/web/resource/dashboard/admin/user · high confidence
Admin dashboard gains user management and execution monitoring endpoints
The admin dashboard now exposes new REST endpoints to manage users and monitor workflow executions. Administrators can view a list of users including their affiliation, joining reason, and last login time, and can update user roles or create new inactive user accounts. The system also enforces that accounts without an email address cannot be activated and prevents email removal once set. Additionally, admins can now view the latest execution status for each workflow, with support for filtering by status, sorting, and pagination, as well as viewing created datasets and workflows for specific users to support quota management. Also in this area: 3 commits (0 fixes) modifying amber.
amber/src/main/scala/org/apache/texera/web/resource/dashboard/admin/execution · medium confidence · unverified
Centralized authentication and authorization infrastructure
The \common/auth\ module now provides a unified, shared authentication stack for all Texera microservices. This includes a centralized \AuthFeatures\ registration for Dropwizard/Jersey, a \JwtAuthFilter\ that enforces strict RFC 6750-compliant 401 responses (distinguishing between missing tokens and invalid ones) while supporting \@PermitAll\ for public endpoints, and a \RoleAnnotationEnforcer\ that fails service startup if any HTTP endpoint lacks explicit \@RolesAllowed\, \@PermitAll\, or \@DenyAll\ annotations. The module also introduces a \RequestLoggingFilter\ to replace Dropwizard's built-in access logs with SLF4J-controlled logging, and a \UserActivityTracker\ that asynchronously records user last-active timestamps to the database with throttling to prevent write storms.
common/auth/src/main/scala/org/apache/texera/auth · high confidence
Centralized configuration files for Texera services
This change introduces a new set of default configuration files in the common config module, providing a unified location for service settings. The new files include application.conf for core engine parameters (such as worker counts, flow control, and WebSocket payload limits), cluster.conf for Pekko actor system and cluster settings, and gui.conf for user-facing features like login providers (Google, ORCID, Apple), execution modes, and UI toggles. Additional files cover auth.conf for JWT settings, computing-unit.conf for local and shared unit options, kubernetes.conf for cluster deployment and curated image management, storage.conf for Iceberg, LakeFS, and S3 storage backends, user-system.conf for user authentication and email verification, udf.conf for Python and R UDF paths, llm.conf for AI assistant integration, and default.conf for site-wide GUI and dataset defaults. These configurations are designed to be overridden via environment variables, allowing deployments to customize behavior without modifying source code.
common/config/src/main/resources · high confidence
Centralized error handling and console message utilities in Amber
The Amber module now includes a new \ErrorUtils\ object that centralizes the creation of console messages and control errors. This change introduces helper methods to format stack traces into user-friendly console output (including specific handling for WARNING prefixes) and to reconstruct throwable objects from serialized control errors, improving how runtime errors and warnings are reported to users during workflow execution.
amber/src/main/scala/org/apache/texera/amber/error · high confidence
Centralized health check and resource access control library
The \common/resource\ module now provides shared infrastructure for service health monitoring and unified access control. It introduces a \HealthCheckResource\ endpoint at \/healthcheck\ that returns a JSON \status: ok\ response, available to all services without authentication. Additionally, it consolidates ownership and privilege logic into \ResourceAccess\ and \ResourceTables\, enabling consistent read/write permissions and access-list management for \dataset\ and \model\ resources across the application.
common/resource · high confidence
Enhanced Python template string encoding and lexical safety
The PyBuilder now introduces a new \@EncodableStringAnnotation\ and an \EncodableInspector\ macro helper to better distinguish between literal strings and values that should be safely encoded when injected into Python code templates. This change ensures that arguments marked with the annotation are wrapped as \EncodableStringRenderer\ rather than raw literals, reducing injection risks. Additionally, new \PythonLexerUtils\ provide lightweight lexical checks to detect unclosed quotes and comments outside of string contexts, improving the safety of string splicing within generated Python code.
common/pybuilder/src/main · high confidence
Export workflows as standalone Python scripts
The workflow operator now supports exporting a workflow as a standalone Python script. This change introduces the \StandaloneCodeGenerator\ trait and \StandaloneHelpers\ (including transcriptions of \AttributeTypeUtils\ and \java.util.Random\) to ensure that exported scripts reproduce the engine's data types and sampling behavior exactly. It also adds \PythonOperatorDescriptor\ to handle Python code generation and physical op creation, and \DummyProperties\ to demonstrate the new schema annotation support.
common/workflow-operator · high confidence
File service introduces model management and replaces MinIO with RustFS
The file-service now includes a complete model backend, exposing REST endpoints for model creation, versioning, access control, and file uploads, alongside the existing dataset capabilities. Additionally, the default object store has been switched from MinIO to RustFS, with updated Docker Compose and configuration files to support the new storage backend.
file-service · high confidence
Frontend project scaffolding and configuration files added
The frontend workspace now includes standard configuration files for code formatting, linting, and environment management. This adds an EditorConfig for consistent whitespace and indentation, ESLint rules tailored for Angular and RxJS (including strict guidelines for testing patterns), Prettier formatting rules, and a .gitignore to exclude build artifacts and generated files. It also sets the Node.js version via .nvmrc and initializes the Yarn 4 package manager.
frontend · high confidence
Initial Apache Incubator governance and project scaffolding
The repository is initialized with the standard Apache Software Foundation (ASF) Incubator structure, including the \.asf.yaml\ configuration for GitHub rulesets, branch protection, and merge queues, alongside essential project files such as \LICENSE\, \NOTICE\, \DISCLAIMER\, \SECURITY.md\, \CONTRIBUTING.md\, and \AGENTS.md\. This change establishes the legal, security, and contribution frameworks required for the project's entry into the Apache Incubator.
(repo-wide) · high confidence
Initial release of the agent-service component
Introduces the new \agent-service\ module, which manages LLM agents. This addition includes the foundational project configuration files: a \bun.lock\ dependency manifest pinning packages such as \elysia\, \@ai-sdk/openai\, and \pino\; a \tsconfig.json\ for TypeScript compilation; a \.prettierrc\ for code formatting; a \.dockerignore\ for build context; a \.env.example\ defining environment variables like \TEXERA\_SERVICE\_LOG\_LEVEL\ and service endpoints; and an \LICENSE-binary\ file for licensing compliance.
agent-service · high confidence
Introduce Hub models and unified search resource-agnostic layer
The Hub dashboard now supports a unified search and resource-agnostic backend layer. This change introduces new Scala models (ActionType, EntityType, EntityTables) that standardize how workflows, datasets, and models are tracked for actions like viewing, liking, cloning, and accessing. It also refactors the HubResource to be entity-type agnostic, allowing the same API endpoints to handle interactions across different resource types without duplicating logic for each specific type.
amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub · high confidence
Introduce Iceberg-based result storage for workflow outputs
The result storage layer now uses Apache Iceberg tables to persist workflow execution results, including runtime statistics and console messages. This change introduces new schema definitions for these outputs and implements Iceberg-backed read and write operations, allowing for efficient, append-only storage of result data with support for range queries and file-level iteration.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/result · high confidence
Introduce LakeFS storage client and storage utilities
Added a new LakeFS storage client implementation that handles repository initialization, file uploads (including presigned multipart uploads), and file retrieval, along with a health check that retries on startup using exponential backoff. Also added storage utility helpers for managing read/write locks.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/util · high confidence
Introduce LargeBinary type for S3-backed large objects
The workflow core now supports a new LargeBinary attribute type, allowing tuples to reference large binary objects stored in S3 via URIs instead of holding the data in memory. This change introduces the LargeBinary class and updates the AttributeType enum and parsing utilities to handle this new type, enabling workflows to process large files without memory constraints.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple · high confidence
Introduce LinkedBlockingMultiQueue with keyed priority support
The \amber\ module now includes a new \customized\_queue\ package providing \LinkedBlockingMultiQueue\, a thread-safe, keyed queue that supports priority groups and per-key enable/disable controls. This implementation replaces previous queue utilities in this location, introducing inner-class-based node and sub-queue structures for efficient memory management and locking, along with base protocols (\IQueue\, \IKeyedQueue\) to standardize queue interactions across the system.
_amber/src/main/python/core/util/customized\queue · high confidence
Introduce Python LSP server container with automatic restart logic
Added a new Dockerfile and entrypoint script for the Python Language Server Protocol (pylsp) service. The container is based on Python 3.12-slim and installs python-lsp-server version 1.14.0 along with pylint 4.0.6. The entrypoint script starts the LSP server on port 3000 via WebSocket and implements a watchdog loop that automatically restarts the process every 5 minutes to ensure stability.
bin/pylsp · high confidence
Introduce Python Virtual Environment management and real-time installation streaming
Users can now create, persist, and manage isolated Python virtual environments (PVEs) for their computing units, with environment details and installed packages stored in the database. The system supports installing user-defined packages and streaming pip installation logs back to the frontend in real time via a WebSocket endpoint (/wsapi/pve), allowing users to monitor installation progress. Additionally, a REST API (/pve) is provided to list, save, update, and delete PVE configurations, as well as fetch system packages.
amber/src/main/scala/org/apache/texera/web/resource/pythonvirtualenvironment · high confidence
Introduce Python code template builder with safety checks
Added a new Python code template builder (pyb) in the pybuilder module that allows constructing Python source code from Scala. The builder distinguishes between plain Python literals and 'Encodable' strings (typically UI-provided text), which are base64-encoded at compile time and rendered as runtime decode expressions to prevent syntax errors. It includes compile-time boundary validation to ensure Encodable splices are not placed inside quotes, comments, or adjacent to invalid characters, and adds runtime guards for nested builders to maintain safety when content is unknown at compile time.
pybuilder · high confidence
Introduce Python control-plane handlers for the Amber engine
This change adds the Python-side control message handlers for the Amber engine, located in \amber/src/main/python/core/architecture/handlers/control\. The new files implement the RPC service methods that manage the worker lifecycle and data flow, including starting and stopping workers, initializing and updating executors, managing input/output ports and partitioning, handling channel start/end markers, and processing debug commands. It also includes handlers for pausing/resuming workers, retrying tuples, evaluating Python expressions, and querying worker statistics, ensuring the Python worker can correctly coordinate with the Amber control plane.
amber/src/main/python/core/architecture/handlers/control · high confidence
Introduce Python core data models and operator framework
This change establishes the foundational Python models for the Amber data processing engine. It adds a \Batch\ class wrapping pandas DataFrames, a \Table\ class for handling tabular data conversions, and a \Tuple\ class for lazy field access with support for large binary data via PyArrow. The \State\ model is redefined as a JSON-serializable dictionary that now includes loop-control bookkeeping columns (loop counter and start ID) for runtime loop management. Additionally, the \Operator\ abstract base class is introduced, featuring a \PythonTemplateDecoder\ for base64 decoding and a corrected \is\_source\ property that properly handles name mangling to identify source operators.
amber/src/main/python/core/models · high confidence
Introduce Python worker architecture managers
The \amber/src/main/python/core/architecture/managers\ directory now contains the core Python-side worker management components, including \Context\, \StateManager\, \PauseManager\, \ExecutorManager\, and \StatisticsManager\. This adds the Python runtime's internal state machine, pause/resume handling, executor loading, and statistics tracking to the Amber architecture, mirroring the existing Scala implementation to support Python-based operators.
amber/src/main/python/core/architecture/managers · high confidence
Introduce Python-based ProxyClient and ProxyServer for Arrow Flight communication
Added new Python modules in the core proxy package that implement a ProxyClient and ProxyServer using Apache Arrow Flight. The ProxyClient provides methods to call remote actions and send data batches, handling timeouts and optional handshakes. The ProxyServer acts as a Flight server base, registering default actions for heartbeat, shutdown, and control messages, while exposing hooks for custom data, control, and actor message processing. This establishes the Python-side infrastructure for proxying requests between components.
amber/src/main/python/core/proxy · high confidence
Introduce PythonWorker and Stoppable utilities for thread management
This change adds the PythonWorker class, which manages background threads for network communication, heartbeats, and the main processing loop, and introduces the Stoppable protocol along with the StoppableQueueBlockingRunnable utility to safely interrupt threads blocked on queue operations.
amber/src/main/python/core, amber/src/main/python/core/util/stoppable · high confidence
Introduce Texera Agent for workflow editing and execution
The agent-service now includes a new Texera Agent that manages LLM-driven interactions for building and modifying dataflows. This component introduces in-memory state management for workflows (operators, links, and positions) and execution results, allowing the agent to track conversation history via a ReAct step tree and maintain versioned operator results. It exposes tools for adding, modifying, and deleting operators, as well as executing them, while generating system prompts that guide the LLM based on available operator metadata and UDF (Python/R) usage guidelines.
agent-service/src/agent · high confidence
Introduce agent-service for managing LLM agents with WebSocket-based ReAct loop control
This change introduces the \agent-service\ component, which manages the lifecycle of LLM agents and exposes them via REST and WebSocket endpoints. Users can create, list, and delete agents through the \/agents\ API, with authentication enforced via Bearer tokens that delegate to the LLM gateway as the calling user. The service supports a WebSocket protocol at \/agents/:id/react\ for real-time interaction, allowing clients to send prompts and stop commands while receiving live step events, status updates, and initial snapshots. Configuration is handled through environment variables (e.g., \TEXERA\_SERVICE\_LOG\_LEVEL\, \LLM\_ENDPOINT\), and the service includes structured logging with optional pretty-printing for development.
agent-service/src · high confidence
Introduce asynchronous RPC client and server for Amber
The Amber engine now uses an asynchronous RPC architecture, adding \AsyncRPCClient\ and \AsyncRPCServer\ to handle control invocations via internal queues rather than direct network calls. The client manages promise-based communication with the coordinator, while the server routes incoming control commands to specific handlers (such as worker start/stop, channel management, and debugging) and returns results through the same internal queue mechanism.
amber/src/main/python/core/architecture/rpc · high confidence
Introduce dedicated computing-unit-managing-service
A new standalone service has been added to manage computing units, providing REST endpoints for listing all units (admin-only), managing user access grants and revocations, handling unit creation and termination, and administering curated container images. The service includes an idle-cleanup job to automatically terminate unused units, enforces role-based access control via annotations, and is configured to run on port 8888 with configurable log levels.
computing-unit-managing-service · high confidence
Introduce materialized execution mode and workflow context
The workflow engine now supports a materialized execution mode, defined by the new ExecutionMode enum (PIPELINED, MATERIALIZED), allowing users to choose between streaming and materialized processing. Additionally, the core workflow context has been expanded to include a warehouse configuration option, enabling per-warehouse Iceberg catalog and warehouse-scoped URI routing for data storage. Supporting infrastructure includes new traits for execution-time bindings and location preferences to manage operator placement and late-bound values during execution.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/workflow · high confidence
Introduce materialized state and port storage runnables
Added new runnable components in the storage layer to handle materialized state and port data persistence. The \InputPortMaterializationReaderRunnable\ reads tuples and state from materialized storage, applying partitioning logic (such as round-robin, hash-based, or broadcast) to distribute data to downstream workers, while ensuring state is broadcast to all workers before tuple processing begins. The \PortStorageWriter\ component manages the writing of tuples to buffered storage via a queue-based interface, supporting the new materialization workflow across regions.
amber/src/main/python/core/storage/runnables · high confidence
Introduce notebook-migration-service microservice for read-only Jupyter notebook viewing
A new notebook-migration-service microservice has been added to the backend to provide a read-only view of generated workflows as embedded Jupyter notebooks. The service exposes REST endpoints to retrieve Jupyter iframe URLs and manage stored notebook mappings, while enforcing JWT authentication and role-based access control. To ensure a safe viewing experience, the embedded Jupyter interface is customized via CSS and JavaScript to hide the classic chrome, disable editing and execution, and prevent markdown cells from becoming editable. The service supports both single-node deployments with a shared Jupyter instance and Kubernetes deployments with per-user JupyterLab pods, dynamically resolving the correct Jupyter URL and token for each authenticated user.
notebook-migration-service · high confidence
Introduce per-user warehouse management REST API
Adds a new REST endpoint at /warehouse to manage user-specific data warehouses. The API allows users to check feature status, list their existing warehouses (including owner details), and create new warehouses with validated names. It integrates with the Lakekeeper client for storage operations and enforces a feature flag to gate mutating actions, ensuring the frontend can conditionally display the feature.
amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/warehouse · high confidence
Introduce pyright-language-service module
A new \pyright-language-service\ module has been added to provide a Python language server integration. This component includes the necessary source files (\main.ts\, \language-server-runner.ts\, \server-commons.ts\) to launch the Pyright language server via WebSocket, along with configuration (\config.json\) and build settings (\tsconfig.json\). It relies on the \vscode-languageserver\ and \vscode-ws-jsonrpc\ libraries to handle the language server protocol communication.
pyright-language-service · high confidence
Introduce storage model abstractions for file and buffered I/O operations
The storage model layer now includes new abstractions for handling file-based resources and buffered writes. \VirtualDocument\ and \ReadonlyVirtualDocument\ define the interface for read/write and read-only operations on single resources, while \LakeFSFileDocument\ implements reading versioned files from LakeFS via presigned URLs or direct fetch. \BufferedItemWriter\ provides a trait for buffering items before flushing to storage, and \OnVersionedFileResource\ exposes repository, version, and path details for versioned files. \ReadonlyLocalFileDocument\ adds support for reading local files as read-only documents.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model · high confidence
Introduce structured execution state tracking for workflow regions and operators
The Amber engine now tracks execution progress and metrics through a new set of coordinator-side state objects: WorkflowExecution, RegionExecution, OperatorExecution, LinkExecution, ChannelExecution, and WorkerPortExecution. These classes manage the lifecycle of workers and operators within regions, aggregate runtime statistics (such as tuple counts, processing times, and worker counts), and determine the overall workflow state (e.g., RUNNING, COMPLETED, PAUSED) by aggregating individual operator and region states. This change provides a more granular and structured view of execution status, enabling better monitoring and state management for running workflows.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/coordinator/execution · high confidence
Introduces centralized database access and site settings management
This change adds new data-access components to the common DAO layer, introducing a \SqlServer\ class that manages a PostgreSQL connection via a HikariCP connection pool to reduce connection overhead. It also adds a \SiteSettings\ accessor object that centralizes reading and writing key-value configuration from the \site\_settings\ table, replacing scattered inline logic in other services. Additionally, a \jooq-conf.xml\ configuration file is added to define code generation for jOOQ DAOs, and a \SqlStates\ object provides reusable SQLSTATE constants for error handling.
common/dao/src/main · high confidence
Introduces shared TypeScript type definitions for the agent-service
This change adds the foundational TypeScript interfaces and enums for the new agent-service module, located in \agent-service/src/types\. It defines the data structures for agent lifecycle and configuration (\AgentState\, \AgentSettings\, \AgentDelegateConfig\), execution results (\SyncExecutionResult\, \OperatorResultSummary\), and workflow content (\WorkflowContent\, \LogicalPlan\). These types establish the contract for how the agent service interacts with workflow data and manages LLM agent states, supporting features like workflow editing and execution delegation.
agent-service/src/types · high confidence
Introduction of timed buffer for console message output
The buffer utility module now includes a TimedBuffer implementation that manages ConsoleMessage output by flushing data either when a configurable message count threshold is reached or after a specified time interval, ensuring efficient and timely delivery of console logs to users.
amber/src/main/python/core/util/buffer · high confidence
New API client library for agent-service backend interactions
The agent-service now includes a dedicated API client layer in \agent-service/src/api\ to communicate with backend services. This adds support for authenticating requests via JWT bearer tokens (with user extraction and validation), persisting and retrieving workflows, compiling logical plans into physical plans, and fetching operator metadata. The implementation handles endpoint configuration from environment variables, serializes workflow content as nested JSON strings, and provides corresponding unit tests for all client functions.
agent-service/src/api · high confidence
New Amber engine RPC protocol definitions
The Amber engine's architecture now exposes a comprehensive set of Protocol Buffer definitions for its internal RPC communication. This includes the \CoordinatorService\ and \WorkerService\ interfaces, which define the control commands (such as workflow reconfiguration, checkpointing, and jump-to-operator) and responses exchanged between the engine's coordinator and worker nodes. The diff also introduces specific message types for partitioning semantics, worker state and metrics tracking, and a dedicated \RPCTester\ service for testing, establishing the foundational contract for the engine's distributed execution model.
amber/src/main/protobuf/org/apache/texera/amber/engine/architecture · high confidence
New Kubernetes Helm chart and out-of-pod mounter service
This change introduces a new Helm chart (bin/k8s) for deploying Texera on Kubernetes, replacing previous deployment methods. The chart configures the application stack using RustFS as the default in-cluster object store (with optional external S3 support via values-aws.yaml), Groonga-based PostgreSQL, LakeFS, and Lakekeeper for Iceberg cataloging. It also adds a new privileged 'mounter' service (bin/mounter/mounter.py) that performs GeeseFS FUSE mounts on behalf of computing-unit pods, enabling secure, out-of-pod access to data repositories without requiring privileged containers in the user workloads. The deployment includes comprehensive tests for the mounter's mount management and authentication logic.
bin/k8s · high confidence
New Python-based input and output managers for data handling
The Amber Python core now includes dedicated InputManager and OutputManager classes in the packaging module to handle data flow. The InputManager manages input ports, channels, and background threads for reading materialized data from storage URIs, while the OutputManager handles output ports, partitioning strategies (such as round-robin, hash-based shuffle, and broadcast), and background threads for writing tuples to storage. This introduces a new mechanism for managing data serialization, partitioning, and storage I/O within the Python execution environment.
amber/src/main/python/core/architecture/packaging · high confidence
New WebSocket event models for execution metrics and pagination
The Amber backend now exposes additional real-time updates via WebSocket, including execution duration, operator statistics (with a new \reusedFromCache\ flag to indicate cache hits), paginated query results, region states, and worker assignments. These new event types allow the frontend to display more granular performance insights and handle large result sets efficiently.
amber/src/main/scala/org/apache/texera/web/model/websocket/event · high confidence
New WebSocket request models for workflow control and pagination
The Amber module introduces a set of new Scala case classes in the \org.apache.texera.web.model.websocket.request\ package to handle specific user actions over the WebSocket interface. These include \ResultPaginationRequest\ to support paginated and searchable wide-column table views, \WorkflowCheckpointRequest\ for saving workflow states, and \ModifyLogicRequest\ for updating logical operators. Additionally, new requests allow users to retry or skip tuples (\RetryRequest\, \SkipTupleRequest\), manage execution lifecycles (\WorkflowKillRequest\, \WorkflowPauseRequest\, \WorkflowResumeRequest\), and interact with Python operators (\PythonExpressionEvaluateRequest\, \DebugCommandRequest\). A \HeartBeatRequest\ is also added for connection monitoring. All these types are registered in \TexeraWebSocketRequest\ for polymorphic deserialization.
amber/src/main/scala/org/apache/texera/web/model/websocket/request · high confidence
New access control and repository mounting endpoints for computing units
The access-control-service now exposes two new JAX-RS resource endpoints: AccessControlResource, which centralizes authorization logic for WebSocket API, execution stats, and Python Virtual Environment (PVE) routes by validating tokens and checking computing unit privileges, and ComputingUnitMountResource, which allows users with WRITE access to a computing unit to mount specific versions of datasets or models from a repository, ensuring both unit and repository read permissions are verified before forwarding the request to the mounter.
access-control-service/src/main/scala/org/apache/texera/service/resource · high confidence
New agent tools for workflow CRUD and execution with improved result formatting
The agent-service now exposes tools that allow LLM agents to create, modify, and delete operators within a workflow, as well as execute them. When an operator is added or modified, the tool response now explicitly lists any created or deleted links (e.g., "created links: \[op1 --\> op2\]"). Execution results are formatted into a readable table shape (e.g., "Output table shape: (7, 2)") and include metadata about upstream input operators. Internal tracking keys like \\_\_is\visualization\\_\ are automatically filtered from the visible column headers to keep the output clean.
agent-service/src/agent/tools · high confidence
New agent utility modules for workflow layout, context assembly, and metadata handling
The agent-service now includes a set of utility modules in \src/agent/util\ that support the Texera Agent's ability to edit and execute workflows. \auto-layout.ts\ introduces an \autoLayoutWorkflow\ function that uses the dagre library to automatically position workflow operators, mirroring the frontend's layout configuration (rank separation, node dimensions) to ensure visual consistency. \context-utils.ts\ adds an \assembleContext\ function that constructs the prompt context for the LLM agent, formatting completed and ongoing tasks, serializing the current dataflow (operators and links) in topological order, and including input/output schemas and execution results. \workflow-system-metadata.ts\ provides a \WorkflowSystemMetadata\ singleton that fetches and caches operator metadata (JSON schemas, descriptions) from the backend, supporting validation and schema compaction. \workflow-utils.ts\ contains helper functions like \extractOperatorInputPortSchemaMap\ to resolve input port schemas from upstream operators and a \WorkflowUtilService\ to instantiate new operators with default properties derived from their JSON schemas. Comprehensive unit tests have been added for all these new utilities.
agent-service/src/agent/util · high confidence
New bin/ directory structure and tooling for Texera
The repository introduces a centralized \bin/\ directory to manage development, build, and deployment workflows. This includes a new local development stack manager (\local-dev.sh\), a single-node Docker deployment entry point (\single-node.sh\), and a unified script for running benchmarks (\run-benchmarks.sh\). Image build and tagging are now handled by \build-images.sh\ and \merge-image-tags.sh\, which support multi-architecture builds via Docker Buildx. Code generation for frontend and Python protobufs is consolidated into \frontend-proto-gen.sh\ and \python-proto-gen.sh\, while \fix-format.sh\ provides a single interface for running Scala, frontend, and Python formatters (including Ruff). The directory also contains configuration for a LiteLLM proxy (\litellm-config.yaml\) and a Flarum-based community forum (\config.php\, \.htaccess\), alongside shared utility scripts for logging and path resolution.
bin · high confidence
New core utility modules for Amber
The \amber/src/main/python/core/util\ package now includes several new modules: \atomic.py\ provides a thread-safe \AtomicInteger\ class; \virtual\_identity.py\ adds helpers for parsing worker IDs and serializing/deserializing \GlobalPortIdentity\ objects; \expression\_evaluator.py\ introduces an \ExpressionEvaluator\ for evaluating Python expressions with context; \base\protocols.py\ defines protocols like \Putable\, \Getable\, and \EmtpyCheckable\; \runnable.py\ defines a \Runnable\ protocol; and \\\init\\_.py\ exposes these utilities. These changes provide foundational utilities for concurrency, identity management, expression evaluation, and protocol definitions within the Amber core.
amber/src/main/python/core/util · high confidence
New licensing audit and compliance tooling in bin/licensing
This change introduces a suite of Python scripts in the bin/licensing directory to automate and enforce binary dependency licensing. The new tools include audit\_jar\_licenses.py to scan bundled JARs for license and notice files and flag those requiring per-dependency license files; check\_binary\_deps.py to verify that the actual bundled dependencies match the declared LICENSE-binary, supporting ecosystems like JAR, npm, and Python with options to ignore transitive version drift; concat\_license\_binary.py to merge per-module LICENSE-binary files into a single combined file while preserving license groupings; and generate\_notice\_binary.py to create NOTICE-binary files by extracting and deduplicating META-INF/NOTICE content from JARs. Comprehensive unit tests are added for these scripts to ensure correct behavior in indexing, version handling, and file generation.
bin/licensing · high confidence
New local development stack manager with Linux support and TUI
The \bin/local-dev.sh\ script and its supporting files in \bin/local-dev/\ provide a unified entry point for managing the local development environment. This tooling now supports Linux (in addition to macOS) and includes a Textual-based interactive dashboard (\-i\) for monitoring service states. It automatically handles PostgreSQL schema migrations before building, allows deploying from sibling git worktrees, and offers to install missing toolchains (JDK, Node, Python). The setup also configures the local stack to bypass email verification by default and correctly routes storage endpoints for host-native JVMs.
bin/local-dev · high confidence
New partitioner implementations for data distribution semantics
The \amber/src/main/python/core/architecture/sendsemantics\ module now includes concrete implementations for distributing data across workers: \BroadcastPartitioner\ (sends every tuple to all receivers), \OneToOnePartitioner\ (routes to a single specific worker), \RoundRobinPartitioner\ (cycles through receivers), \HashBasedShufflePartitioner\ (routes based on tuple hash), and \RangeBasedShufflePartitioner\ (routes based on value ranges). These classes implement the \Partitioner\ base class to handle batching, flushing, and state synchronization according to their specific send semantics.
amber/src/main/python/core/architecture/sendsemantics · high confidence
New serialization helpers for workflow port identities
Added three new Scala files in the \common/workflow-core\ serde utility package to handle serialization of workflow identity objects. \GlobalPortIdentitySerde\ provides a custom string format for \GlobalPortIdentity\ that avoids characters like underscores and slashes to prevent conflicts with VFS URI parsing, while \PortIdentityKeySerializer\ and \PortIdentityKeyDeserializer\ implement Jackson-based serialization for \PortIdentity\ keys using an underscore-separated string format.
common/workflow-core/src/main/scala/org/apache/texera/amber/util/serde · high confidence
New single-node deployment via Docker Compose
Users can now run the full Texera platform locally on a single machine using a new Docker Compose stack managed by the \bin/single-node.sh\ wrapper script. This deployment includes the core application services, RustFS as the S3-compatible object store, Lakekeeper as the Iceberg REST catalog, and PostgreSQL for metadata. It also introduces built-in support for the Texera Agent (powered by LiteLLM and Claude Haiku) and a Python notebook migration tool (JupyterLab), both of which can be enabled via environment variables. The setup includes example workflows and datasets, configurable ports, and a startup banner for easy verification.
bin/single-node · high confidence
New utility library for Iceberg, Arrow, and workflow identity management
This change introduces a new set of utility classes in the workflow-core module to support Iceberg table operations, Arrow data conversion, and worker identity handling. IcebergUtil now provides factory methods for creating Hadoop, REST, and PostgreSQL catalogs, including a Windows-specific workaround (WinutilsFreeLocalFileSystem) that allows local file system operations without requiring the winutils binary. ArrowUtils handles the conversion between Arrow schemas and Texera types, with specific logic to correctly interpret timestamp wall-clock values based on their field metadata. JSONUtils includes an iterative JSON-to-Map converter to prevent stack overflows on deeply nested structures and configures Jackson to properly serialize scalapb sealed-oneof types. VirtualIdentityUtils centralizes the parsing and creation of worker actor identities, enforcing constraints like rejecting hyphenated layer names to ensure consistent parsing.
common/workflow-core/src/main/scala/org/apache/texera/amber/util · high confidence
New utility modules for avatar sanitization, email validation, and retry logic
The common/util module now includes three new Scala utilities: AvatarUtil sanitizes profile-picture URLs by storing the full provider-supplied URL and validating that the host is on an allowlist (currently googleusercontent.com); EmailUtil provides methods to validate and normalize email addresses; and RetryUtil offers a blocking exponential-backoff retry mechanism with configurable attempts, delays, and failure predicates. These utilities are accompanied by comprehensive test coverage.
common/util · high confidence
New web application entry points and supporting infrastructure
This change introduces the primary server entry points for the platform: \TexeraWebApplication\ serves the frontend static assets and registers the main REST API resources (including authentication, workflow, and AI assistant endpoints), while \ComputingUnitMaster\ handles workflow execution coordination and WebSocket connections. Supporting this architecture are new components for managing WebSocket session state (\SessionState\, \SubscriptionManager\), handling user identity during WebSocket handshakes (\ServletAwareConfigurator\), caching static frontend assets (\StaticAssetCacheFilter\), and managing workflow lifecycle cleanup (\WorkflowLifecycleManager\).
amber/src/main/scala/org/apache/texera/web · high confidence
New web resource endpoints for collaboration, feedback, Hugging Face integration, and synchronous execution
This change introduces several new REST and WebSocket endpoints in the web resource layer. It adds a collaboration WebSocket endpoint (/wsapi/collab) for real-time workflow editing and locking, a feedback API (/feedback) for users to submit and view feedback and for admins to review it, and a Hugging Face model resource (/huggingface) to browse models, proxy media, and handle audio uploads. It also introduces a synchronous execution endpoint (/execution/{wid}/{cuid}/run) that allows clients to run workflows and receive immediate results with configurable limits, alongside a WebSocket payload size tuner to prevent dropping large result frames.
amber/src/main/scala/org/apache/texera/web/resource · high confidence
New web service layer for execution management and notifications
The \amber/src/main/scala/org/apache/texera/web/service\ directory now contains a new set of services that handle the web-facing aspects of workflow execution. \ExecutionRuntimeService\ manages the execution lifecycle (pause, resume, kill) and integrates with \EmailNotificationService\ to send status updates. \ExecutionConsoleService\ and \ExecutionStatsService\ stream real-time console output and runtime statistics to the frontend. \ExecutionReconfigurationService\ handles dynamic logic modifications during paused executions, while \ExecutionResultService\ and \ResultExportService\ manage result retrieval and export (CSV, Arrow, ZIP). \LakekeeperClient\ provides the interface for managing per-user warehouses, and \ExecutionsMetadataPersistService\ handles database persistence for execution metadata.
amber/src/main/scala/org/apache/texera/web/service · high confidence
New websocket response models for cluster, region, and logic updates
The Amber backend now exposes specific WebSocket response types to support frontend updates for cluster status, region layers, and logic modifications. Users will see improved real-time feedback in the workflow editor, including region layer visualization and validation status for logic changes, as the system now sends structured events like ClusterStatusUpdateEvent, RegionUpdateEvent, ModifyLogicResponse, and ModifyLogicCompletedEvent alongside existing heartbeat and Python evaluation responses.
amber/src/main/scala/org/apache/texera/web/model/websocket/response · high confidence
New workflow management REST API endpoints in Amber
The Amber module now exposes a new set of REST resources for managing workflows, including WorkflowResource for core operations (create, read, update, delete, clone, cover image, default view), WorkflowAccessResource for sharing and permission management (grant, list, owner), WorkflowExecutionsResource for execution history and export, and WorkflowVersionResource for versioning and diffing. These endpoints handle workflow lifecycle, access control, execution data, and version history, replacing or supplementing previous mechanisms for these functions within the user dashboard workflow area.
amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow · high confidence
New workflow-operator components for JSON schema generation, aggregation, and stable sorting
The workflow-operator module introduces several new Scala components: a JsonSchemaGenerator (derived from mbknor-jackson-jsonschema) with HTML5 and nullable configurations to produce operator metadata schemas; an AggregationOperation implementation supporting distributed sum, count, average, min, max, and concat functions, including COUNT(\*) via empty-attribute detection; a DistributedAggregation trait defining init/iterate/merge/finalAgg stages for distributed computation; a TaskCodegen trait and CodegenContext for Hugging Face pipeline code generation; an OperatorMetadataGenerator that builds operator schemas and metadata for the frontend; and a StableMergeSortOpExec using a binary-carry bucket strategy to provide stable in-memory sorting with nulls ordered last.
workflow-operator · high confidence
Python Iceberg storage implementation with REST catalog and Windows support
The Python storage layer in the Amber worker now supports Iceberg tables via a new \IcebergDocument\ and \IcebergTableWriter\ implementation. This adds support for REST catalogs, allowing multiple warehouses to be managed within a single process, and fixes local storage on Windows by normalizing drive-letter paths to \file://\ URIs. It also introduces handling for large binary data by encoding those fields with a specific suffix to ensure compatibility with the Java/Scala side.
amber/src/main/python/core/storage/iceberg · high confidence
Python UDF library exposes loop operators and large binary I/O
The pytexera Python library now exports LoopStartOperator and LoopEndOperator, enabling Python UDFs to define and manage loop structures within workflows. Additionally, it exposes large binary input and output streams (LargeBinaryInputStream, LargeBinaryOutputStream) and the largebinary type, allowing Python UDFs to handle large binary data efficiently. The library also continues to provide standard UDF operator classes (UDFOperatorV2, UDFTableOperator, UDFBatchOperator, UDFSourceOperator) and utility exports like overrides and logger.
amber/src/main/python/pytexera · high confidence
Python UDF support for UI parameters and resource selection
The Python UDF framework now supports UI parameters, allowing users to configure operator properties directly from the UI. This includes the ability to select mounted resources like models and datasets by name, which are passed to the UDF as directory paths. The implementation adds the \UiParameter\ API and \Resource\ enum to \pytexera/udf/udf\_operator.py\, enabling end-to-end configuration of Python operators through the interface.
python · high confidence
Python storage layer for datasets and large binaries
The Python storage module now provides classes to manage dataset files and large binary objects. DatasetFileDocument parses logical paths (supporting both new resource-type prefixes and legacy unprefixed paths for backward compatibility) and retrieves files via a presigned URL endpoint with configurable timeouts and retries. Large binary data is handled through LargeBinaryManager (which manages a singleton S3 client and generates execution-scoped object keys) and stream wrappers (LargeBinaryInputStream and LargeBinaryOutputStream) that lazily download or upload data in the background using multipart uploads.
amber/src/main/python/pytexera/storage · high confidence
Python storage layer now uses Iceberg and supports per-warehouse URIs
The Python storage module in \amber/src/main/python/core/storage\ has been implemented to manage data persistence via Iceberg tables. This change introduces a \DocumentFactory\ that creates, opens, and checks for the existence of documents (results and state) by mapping them to specific Iceberg namespaces. It also adds a \VFSURIFactory\ that constructs and parses Virtual File System URIs, including support for optional warehouse-scoped paths (e.g., \/wh/\<name\>\) to route storage operations to the correct catalog. Configuration for these storage backends, including S3 and Iceberg catalog details, is centralized in a new \StorageConfig\ class.
amber/src/main/python/core/storage · high confidence
Python support for large binary data types via S3 references
The Python core models now include a \largebinary\ type that represents references to large objects stored externally in S3. This new class allows users to create, read, and write large binary data by managing S3 URIs, with automatic cleanup of storage based on execution ID. The implementation provides methods to extract bucket names and object keys from the URI, enabling seamless integration of large binary attributes into Python-based workflows.
amber/src/main/python/core/models/type · high confidence
Secure node-specific mount authorization for LakeFS repositories
The access-control-service now includes utilities to safely authorize and perform repository mounts on specific Kubernetes nodes. A new ComputingUnitNodeLocator resolves the correct node IP for a computing unit by querying the Kubernetes API, ensuring mount requests are directed to the right host without allowing callers to specify arbitrary nodes. The MounterClient handles the actual mount operation by sending authenticated HTTP requests to the per-node texera-mounter, using a projected service-account token for secure authorization. Input validation ensures that computing unit IDs, repository names, and commit hashes are safe path segments, preventing injection or path traversal issues when the mounter constructs the final directory path.
access-control-service/src/main/scala/org/apache/texera/service/util · high confidence
Support for loop variable references in Java UDF operators
The workflow engine now allows Java-based operators to reference loop variables (e.g., $i) within their configuration descriptors. This is enabled by new executor infrastructure in the Amber core that dynamically compiles Java code at runtime and binds loop variable values from the execution state into the operator's settings. The system ensures type safety by coercing values to match the descriptor's JSON schema and validates that loop variables remain consistent within a single iteration, failing the workflow if conflicts or missing references are detected.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/executor · high confidence
Support for loop variable references in operator settings
The workflow engine now allows operator settings to reference loop variables (e.g., \$i\) when used inside loop blocks. This is implemented by introducing a \StateReferencing\ trait and a \StateReferenceModule\ that intercepts JSON deserialization to capture these references in a sidecar map, while a \NoLoopVariable\ annotation prevents such references in properties used to build the static workflow plan. This enables dynamic configuration of operators based on loop iteration state.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/state · high confidence
Support for storing and streaming large binary data via S3
The workflow core now introduces a dedicated mechanism for handling data larger than 2GB, which cannot fit in standard tuples. This change adds a new \LargeBinary\ type and a suite of utilities in the \service/util\ package: \LargeBinaryManager\ handles the lifecycle (creation and deletion) of these objects, scoping them by execution ID to ensure proper cleanup; \LargeBinaryOutputStream\ enables streaming data to S3 using multipart uploads in the background; and \LargeBinaryInputStream\ provides lazy, on-demand reading from S3. The underlying \S3StorageClient\ is updated to support these operations, including paginated deletion of directories to handle large numbers of objects and multipart upload logic for efficient data transfer.
common/workflow-core/src/main/scala/org/apache/texera/service · high confidence
Unified dashboard search across workflows, datasets, and models
The dashboard now provides a single search interface that queries workflows, datasets, and models simultaneously. Users can filter results by resource type, owner email, creation or modification dates, and specific IDs, and sort the combined results by name, creation time, edit time, or execution time. The search supports full-text matching on names and descriptions, and the results include owner details and resource-specific metadata like cover images and execution times.
amber/src/main/scala/org/apache/texera/web/resource/dashboard · high confidence
Unified file resolution for datasets, models, and LakeFS mounts
The storage layer now provides a single entry point for resolving file paths, supporting local files, versioned datasets and models, and LakeFS repository mounts. Logical paths for datasets and models are resolved via a new \FileResolver\ that dispatches to database lookups based on a resource-type prefix (e.g., \/dataset/...\ or \/model/...\), while still accepting legacy unprefixed dataset paths for backward compatibility. LakeFS repositories are mounted on-demand via \RepositoryMountManager\, which requests mounts from the access control service and waits for propagation. Iceberg catalog access is managed by \IcebergCatalogInstance\, which caches catalogs per warehouse (for REST catalogs) with a bounded size and idle-time expiration to prevent resource leaks.
common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage · high confidence
Workflow Compiling Service introduces standalone Python script export
The workflow-compiling-service now exposes a new /api/workflow-to-python endpoint that translates a workflow's logical plan into a standalone, executable Python script. This feature allows users to export their workflows as self-contained Python code (using pandas) that can be run independently of the Texera engine, facilitating offline analysis, debugging, or integration with other Python-based data pipelines. The service also includes the standard compilation endpoint and is configured to run on port 9090 with role-based access control.
workflow-compiling-service · high confidence
Architecture
Amber worker promise handlers are restructured into dedicated traits
The worker's RPC promise handlers in the \promisehandlers\ package have been refactored from a single monolithic class into individual Scala traits (e.g., \AddInputChannelHandler\, \StartHandler\, \PrepareCheckpointHandler\). This change improves code organization and maintainability by isolating the logic for specific control messages—such as checkpointing, port assignment, and worker lifecycle management—into separate, focused components without altering the external behavior of the Amber execution engine.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers · high confidence
Refactor Amber coordinator promise handlers into traits
The coordinator's promise handlers in the \promisehandlers\ package have been refactored from concrete classes into traits (e.g., \PortCompletedHandler\, \PauseHandler\, \StartWorkflowHandler\). These traits are designed to be mixed into the \CoordinatorAsyncRPCHandlerInitializer\, centralizing the implementation of workflow control logic—such as starting, pausing, resuming, and checkpointing—within the coordinator's initialization structure.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/coordinator/promisehandlers · high confidence
Behavioural changes
1 commit (0 fixes) modifying amber/src/main/scala/org/apache/texera/web/observability, observability
A change to existing behaviour in amber/src/main/scala/org/apache/texera/web/observability, observability — 1 commit, 2 files: “feat(observability): backend metrics + tracing primitives (\#5376)” (refs \#5376).
amber/src/main/scala/org/apache/texera/web/observability, observability · medium confidence · unverified
3 commits (0 fixes) modifying common/observability
A change to existing behaviour in common/observability — 3 commits, 11 files: “feat(observability): logging foundations (bootstrap, bridge, sanitizer) (\#5375)”; “refactor(observability): drop TraceparentValidator, SpanAttrs (\#9233)”; “feat(observability): backend metrics + tracing primitives (\#5376)” (refs \#5375, \#9233, \#5376).
common/observability · medium confidence · unverified
3 commits (1 fix) modifying amber/src/main/scala/org/apache/texera/web/resource/dashboard/user
A change to existing behaviour in amber/src/main/scala/org/apache/texera/web/resource/dashboard/user — 3 commits (1 fix), 1 file: “feat: add user registration form (\#4185)”; “feat: add affiliation attribute to user (\#4113)”; “fix(backend): use dynamic DSLContext lookup to fix stale DB connections in tests (\#4179)” (refs \#4185, \#4113, \#4179).
amber/src/main/scala/org/apache/texera/web/resource/dashboard/user · medium confidence · unverified
4 commits (1 fix) modifying amber
A change to existing behaviour in amber — 4 commits (1 fix), 20 files: “fix(licensing): restore the Apache License text in the LICENSE-binary files (\#9299)”; “refactor(amber): consolidate websocket and collab message models (\#9265)”; “feat(workflow-operator): add UI parameters for R UDFs (\#9279)”; and 1 more (refs \#9299, \#9265, \#9279, \#8144). Also in this area: Relocate Amber module to new directory structure.
amber/src/main/scala/org/apache/texera/web/model/collab · medium confidence · unverified
Access control service relocated and expanded with LakeFS mount support
The access-control-service has been moved to the org.apache.texera.service package and now includes new capabilities for LakeFS repository mounts. This change introduces the ComputingUnitMountResource and supporting utilities (ComputingUnitNodeLocator, MounterClient) that allow authorized users to mount repositories by locating the correct Kubernetes node and authenticating with the privileged node mounter. Additionally, the service now enforces role annotations at startup, records user activity via a dedicated event listener, and configures request logging through the TEXERA\_SERVICE\_LOG\_LEVEL environment variable.
access-control-service/src/main/scala/org/apache/texera/service · high confidence
Aligns Amber's JWT authentication with microservices claim parsing
Amber now uses a shared JWT claim parser to construct the SessionUser object, ensuring that authentication and authorization logic (including role checks) behaves identically to the microservices. This change standardizes how user identity and roles are extracted from tokens across the platform.
amber/src/main/scala/org/apache/texera/web/auth · high confidence
Amber engine client interface migrated to Apache Texera package
The Amber engine's client-side components, specifically the AmberClient and ClientActor, have been moved from the org.apache.amber namespace to org.apache.texera.amber. This change updates the package declarations and internal imports to reflect the project's rebranding, ensuring the client actor correctly initializes the Coordinator and handles control requests within the new organizational structure.
amber/src/main/scala/org/apache/texera/amber/engine/common/client · high confidence
Amber engine common utilities and runtime initialization
The Amber engine's common module now includes foundational utilities and runtime initialization logic. This adds a centralized configuration object for the master node address, a Kryo initializer that registers specific serializers for lambda and closure handling, and a logging trait that formats log messages with actor identity. The runtime module provides the core mechanism to start and manage the Pekko actor system for both master and worker nodes, including cluster seed node configuration. Additionally, new checkpoint support classes enable the serialization and deserialization of operator states, while a Fries reconfiguration algorithm is introduced to handle workflow updates by computing minimal cut sets. Utility functions for exponential-backoff retries, path resolution, and workflow state mapping are also included.
amber/src/main/scala/org/apache/texera/amber/engine/common · high confidence
Amber engine migrates actor infrastructure from Akka to Apache Pekko
The Amber engine's core architecture components—including the actor service, message transfer, and reference mapping services—have been migrated from Akka to Apache Pekko. This change updates the underlying actor framework used for workflow execution, requiring the engine to use Pekko-specific imports and APIs for actor lifecycle, scheduling, and remote communication.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/common · high confidence
Build system relocated to project/ with updated tooling and licensing automation
The root build configuration has been moved into a dedicated project/ directory, introducing new SBT plugins (scalafmt, scalafix, jacoco, license-report, native-packager, fs2-grpc) and updating core build dependencies: sbt to 1.12.9, jOOQ codegen to 3.19.36, Typesafe Config to 1.4.9, and the PostgreSQL driver to 42.7.13. This relocation includes automated license handling that generates per-module META-INF/LICENSE files (including specific MIT attribution for the workflow-operator module) and ships LICENSE-binary, NOTICE-binary, DISCLAIMER, and the licenses/ directory in distribution zips. Additionally, a new test filtering mechanism (TestFilters) enables CI jobs to split tests into unit and integration subsets using environment variables and ScalaTest tags.
project · high confidence
Centralized configuration management for platform services
The platform now uses a unified set of Scala configuration objects in the common config module to manage settings for authentication, Kubernetes computing units, storage, and the GUI. This change introduces support for new authentication providers (ORCID, Apple) and features like email verification, per-user warehouses, and out-of-pod dataset mounting. It also adds configuration for curated computing-unit images, adaptive network buffering, and AI assistant integration, while replacing the previous hand-rolled secret lock with a lazy initialization pattern for improved thread safety.
common/config/src/main/scala/org/apache/texera/common · high confidence
Centralized console message reporting for errors and print statements
The Amber engine now uses a unified console message system for operator-facing output. Uncaught exceptions are captured and reported as structured ERROR messages including source location and full tracebacks, while Python print statements are intercepted via a context manager to be logged as structured PRINT messages with caller metadata. All console messages are timestamped using the system's local timezone to ensure consistent chronological ordering in the UI.
_amber/src/main/python/core/util/console\message · high confidence
Config service relocated and restructured with new API endpoints
The config-service has been moved to a new location and restructured as a standalone Dropwizard application. It now exposes configuration via specific API endpoints: /config/pre-login for unauthenticated access (including login provider flags and invite-only status), /config/gui for authenticated GUI settings, /config/amber for engine configs, and /config/settings for admin management of site settings. The service initializes by preloading default settings into the database and enforces role-based access control on all endpoints.
config-service · high confidence
Configurable service logging and new resource files for Amber
The Amber service resources now include configuration files for the computing-unit-master, compiling-service-web, and web components, alongside a new cache configuration and logback setup. Log levels for these services are now controllable via the TEXERA\_SERVICE\_LOG\_LEVEL environment variable, defaulting to INFO, and all log output is consistently directed to the logs/ directory.
amber/src/main/resources · high confidence
Database schema migration and versioning infrastructure
The system now uses Liquibase for database schema management, introducing a changelog that applies 51 incremental updates. This migration introduces new tables for user feedback, dataset contributors, model metadata, and per-user warehouse registrations, while also adding support for ORCID and Apple authentication providers. Existing structures are updated to widen execution size columns to BIGINT to prevent truncation, split authentication credentials into a dedicated provider table, and remove the deprecated project feature.
sql · high confidence
Dockerfiles relocated to bin/dockerfiles/
The Dockerfiles for all Texera services (including access-control-service, agent-service, computing-unit-master, computing-unit-worker, config-service, file-service, jupyter, mounter, notebook-migration-service, texera-web-application, workflow-compiling-service, and workflow-computing-unit-managing-service) have been moved from the bin/ root directory into a new bin/dockerfiles/ subdirectory. This change reorganizes the build context structure without altering the service definitions or build logic. Also in this area: 4 commits (2 fixes) modifying bin; Support for VM-based runtime classes in microservices and computing units.
bin/dockerfiles · medium confidence · unverified
Helm templates reorganized into base, aws, and on-prem directories with external S3 support
The Kubernetes Helm templates have been restructured into a modular layout: \base/\ contains resources for all deployments (micro-services, Envoy Gateway, Postgres, LakeFS, computing-unit pool, RBAC), while \aws/\ and \on-prem/\ hold environment-specific resources gated by values. This change introduces configurable external S3 storage via \storage.s3\ values, allowing the system to use an external S3-compatible store (with provided credentials) instead of the default in-cluster RustFS. It also adds an AWS-specific S3 credentials Secret template and organizes templates by component for easier maintenance.
bin/k8s/templates · high confidence
Introduce Python-based Amber worker architecture with loop-aware state handling
The Python worker runtime in \amber/src/main/python/core/runnables\ has been restructured into a new set of components: \MainLoop\ (the central coordinator), \DataProcessor\ (handles tuple, state, and internal marker execution), \NetworkReceiver\ and \NetworkSender\ (manage serialization/deserialization of data and control messages via a proxy server), and \Heartbeat\ (monitors JVM connectivity and parent process health). This change introduces specific loop-bookkeeping logic in \MainLoop\ to read loop input tables from materialized storage rather than shipping them in state, and to correctly handle unstamped boundary states through LoopEnd, ensuring stable iteration behavior.
amber/src/main/python/core/runnables · high confidence
Introduce execution-scoped large binary storage and Python Virtual Environment support
The scheduling configuration now supports executing Python UDFs inside Python Virtual Environments (PVEs) and scopes large binary storage to specific executions. Worker configurations include a \pveName\ field to identify the virtual environment and an \largeBinaryBaseUri\ that is generated per execution ID, ensuring that large binary artifacts are isolated and cleaned up by execution rather than globally.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/config · high confidence
Introduce new Amber engine coordinator architecture
The Amber engine's coordinator component has been restructured into a new set of classes within the \coordinator\ package. This includes a new \Coordinator\ actor that manages workflow lifecycle and message processing, a \CoordinatorProcessor\ to handle execution logic, and a \CoordinatorTimerService\ for periodic status and statistics updates. The change also introduces a \GlobalReplayManager\ to coordinate workflow recovery states and a \WorkflowScheduler\ that uses a cost-based generator to create execution schedules from physical plans. Additionally, a \ClientEvent\ trait and several case classes (such as \ExecutionStateUpdate\ and \OperatorPortResultUriAvailable\) define the messages exchanged between the coordinator and clients.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/coordinator · high confidence
Migrate workflow core protobuf definitions to Apache Texera Amber namespace
The workflow core protocol buffer definitions (executor, virtual identity, workflow, and runtime state) have been moved and renamed from the \apache.amber\ package to \org.apache.texera.amber.core\. This change updates the protobuf package declarations and file paths to reflect the project's rebranding from Amber to Texera Amber, ensuring that generated code and internal messaging contracts align with the new namespace.
common/workflow-core/src/main/protobuf · high confidence
New HTTP request and response models for authentication and result export
This change introduces new Scala case classes that define the structure of HTTP payloads for the Amber web service. For authentication, it adds \UserLoginRequest\ for username/password login, \UserRegistrationRequest\ (which includes an optional \code\ field for email verification), \RegistrationResponse\, and \TokenIssueResponse\. For data handling, it adds \ResultExportRequest\ (supporting various export types like CSV and Google Sheets) and \ResultExportResponse\. These models serve as the contract for the API endpoints handling user signup, login, and workflow result exports.
amber/src/main/scala/org/apache/texera/web/model/http · high confidence
New asynchronous RPC client and server for control messaging
The engine now uses a new asynchronous RPC layer (AsyncRPCClient, AsyncRPCServer, and AsyncRPCHandlerInitializer) to handle control messages. This replaces the previous manual request-response handling with a dynamic proxy model that multiplexes requests over the message-passing infrastructure, allowing callers to use promise-based APIs for coordinator and worker interactions while ensuring errors are surfaced as control responses rather than causing hangs.
amber/src/main/scala/org/apache/texera/amber/engine/common/rpc · high confidence
New cost-based scheduling and region execution engine
The scheduling subsystem has been replaced with a cost-based approach that partitions the physical plan into regions and schedules them using a \DefaultCostEstimator\. This estimator prioritizes historical execution statistics (operator wall-clock times) to minimize total workflow runtime, falling back to a port-count heuristic for new workflows. The new engine introduces a two-phase region execution model (dependee ports followed by non-dependee ports) to handle input-port dependencies and materialized state, and enforces causal state ordering on workers to prevent stale reports from overwriting newer states.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling · high confidence
New materialization and storage writer threads for checkpoint state round-trips
The worker managers now include dedicated threads to handle state materialization and storage writing, ensuring that checkpoint state round-trips correctly. InputPortMaterializationReaderThread reads persisted state and tuples from storage, rebuilding loop envelopes (loop counter and start ID) so that JVM operators inside loops carry the correct context through to LoopEnd. OutputPortStorageWriterThread writes tuples to storage and captures failures to prevent silent worker completion when results are missing. StatisticsManager uses plain maps without withDefaultValue to ensure its statistics survive Kryo serialization during checkpointing.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/managers · high confidence
New messaging layer with credit-based flow control and congestion management
The Amber engine's messaging layer has been replaced with a new architecture that introduces credit-based flow control and congestion control to prevent network saturation. This change adds a \DeadLetterMonitorActor\ to surface dropped messages, \AmberFIFOChannel\ to guarantee exactly-once delivery semantics, and a suite of partitioners (RoundRobin, Hash-based, Range-based, Broadcast, One-to-One) to manage how data tuples are distributed to downstream workers. Additionally, an \InputManager\ now handles materialized input ports via dedicated reader threads, and a \WorkerTimerService\ enables adaptive network buffer flushing to optimize throughput.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/messaginglayer · high confidence
New protobuf definitions for Amber engine common messages and metrics
The Amber engine's common message definitions have been reorganized under the \org.apache.texera.amber.engine.common\ package. New protocol buffer files define core actor commands (including backpressure and credit updates), Python-specific control and data headers, and execution runtime state. Notably, the \OperatorMetrics\ message now includes a \reused\_from\_cache\ boolean flag to indicate when operator results are served from a cache rather than computed, providing visibility into cache-reuse status for users monitoring execution performance.
amber/src/main/protobuf/org/apache/texera/amber/engine/common · high confidence
New utility objects for computing unit access checks and HTTP header constants
This change introduces two new utility files in the auth module. ComputingUnitAccess.scala provides a centralized method to determine a user's privilege level (owner/write/none) for a specific computing unit by performing a single SQL join between the workflow and access tables, replacing potential N+1 query patterns. HeaderField.scala defines constants for standard HTTP header names used to pass user identity and access information (such as user ID, name, email, and computing unit access) in requests. Also in this area: 2 commits (0 fixes) modifying common/workflow-core.
common/auth/src/main/scala/org/apache/texera/auth/util · medium confidence · unverified
Python schema model supports LARGE\_BINARY and fixes numeric coercion
The Python schema model now includes a LARGE\_BINARY type, enabling the system to handle large binary data via PyArrow metadata preservation and serialization as URI strings. Additionally, the schema utilities now correctly coerce integral floats and NumPy scalar types to integers for INT and LONG fields, preventing precision loss or type errors when processing numeric data.
amber/src/main/python/core/models/schema · high confidence
Python worker communication migrated to Apache Arrow Flight
The Python worker proxy layer has been rewritten to use Apache Arrow Flight for data and control communication between the JVM and Python processes. This change introduces new proxy components (PythonProxyClient and PythonProxyServer) that handle data serialization via Arrow schemas, support state frames with loop-envelopes, and transmit embedded control messages. Additionally, the Python worker startup configuration is now Base64-encoded to ensure safe transmission as a command-line argument across platforms, particularly resolving Windows argv quoting issues.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/pythonworker · high confidence
Python worker startup config now uses Base64-encoded JSON with strict key validation
The Python worker entry point (texera\_run\_python\_worker.py) now receives its startup configuration as a Base64-encoded JSON string rather than raw positional arguments. This change ensures the configuration survives Windows command-line argv quoting issues and enforces strict validation: the worker now checks for an exact set of expected keys (including new fields for Iceberg REST catalog, S3 large binary storage, and R path) and rejects any missing, unexpected, or non-string values. This improves reliability and configurability for users running Python workers across different platforms and storage backends.
amber/src/main/python · high confidence
Rebranding to Apache Texera Amber
The project has been rebranded from 'Apache Amber' to 'Apache Texera Amber', reflected in the new package namespace \org.apache.texera.amber\ across the engine, configuration, and workflow core modules. This change includes the introduction of new storage implementations (\HDFSRecordStorage\, \VFSRecordStorage\, \EmptyRecordStorage\) and a \SequentialRecordStorage\ abstraction for handling record I/O, alongside utility classes like \ConfigParserUtil\ and \WorkflowRuntimeException\ to support the updated architecture.
amber/src/main/scala/org/apache/texera/amber/engine/common/storage, common/config/src/main/scala/org/apache/texera/amber, common/workflow-core/src/main/scala/org/apache/texera/amber/core · high confidence
Refactor Amber message types and introduce StateFrame loop envelope
The Amber engine's message-passing layer has been restructured to support loop execution across mixed JVM and Python operators. A new \StateFrame\ payload now carries loop metadata (\loopCounter\, \loopStartId\) alongside the actual state, ensuring that JVM operators within a loop body can transparently pass through the loop envelope without interfering with user state. The message hierarchy has been cleaned up with new traits like \WorkflowFIFOMessagePayload\ and \RecoveryPayload\, and the \WorkflowMessage\ companion object now includes memory-size calculation logic. Additionally, common virtual identity constants (Coordinator, Self, Client) have been centralized in a utility object.
amber/src/main/scala/org/apache/texera/amber/engine/common/ambermessage · high confidence
Refactored log replay architecture with new logging and ordering components
The log replay subsystem in the Amber engine has been restructured to improve fault tolerance and replay reliability. A new \AsyncReplayLogWriter\ now handles log persistence in a separate thread, decoupling I/O from the main processing flow. The \ReplayLogger\ and \ReplayLogManager\ classes manage the recording of processing steps and workflow messages, while the \ReplayOrderEnforcer\ and its \ReplayOrderEnforcer\ implementation ensure that replayed operations occur in the correct sequence relative to the original execution. Additionally, a \ReplayLogGenerator\ was introduced to parse existing logs and extract processing steps and messages for replay.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/logreplay · high confidence
Reintroduce worker architecture with execution-scoped large binary storage
The Amber engine's worker layer has been re-implemented to support execution-scoped large binary storage and re-enabled operator reconfiguration. The new \DPThread\ seeds a \largeBinaryBaseUri\ at startup, ensuring large binaries are scoped to the specific execution and cleaned up by execution ID. The \DataProcessor\ now binds loop-variable references before processing tuples and carries the loop StateFrame envelope through JVM operators to maintain correct loop bookkeeping. Additionally, the worker now uses a \PauseManager\ to handle global and channel-specific pauses and an \EmbeddedControlMessageManager\ to manage alignment, providing a more robust foundation for fault tolerance and checkpointing.
amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker · high confidence
Relocate Amber module to new directory structure
The Amber module has been relocated to a new directory structure, moving core Python architecture and storage model components (such as BufferedItemWriter and VirtualDocument abstractions) and Scala collaborative WebSocket event/request models into their new locations under the amber source tree.
(repo-wide) · high confidence
Unified workflow compiler with robust loop-variable binding and error handling
The workflow compiler has been refactored into a common module that consolidates duplicate logic and introduces stricter, more informative error handling. Users now benefit from a lenient mode that collects all per-operator compilation errors (useful for editing-time validation) and a strict mode that fails fast before execution. A key behavioral improvement is the correct binding of loop-variable references (e.g., $i) inside operators, including support for Python code-generation operators that previously could not resolve these variables. The compiler also now validates that operators referencing loop variables are correctly positioned within loop blocks and that their inputs are fed from within the same loop scope, preventing runtime state errors. Additionally, the LogicalLink model now robustly handles JSON serialization round-trips for operator identities, ensuring compatibility with frontend data shapes.
common/workflow-compiler · high confidence
Worker state transitions now use a logical version clock instead of timestamps
The StateManager in the Amber engine now tracks state changes using a monotonically increasing logical version rather than wall-clock timestamps. This change ensures that state reports from workers are ordered causally, allowing the controller to correctly reject stale or out-of-order updates without relying on synchronized clocks across distributed processes. The WorkerStateManager now exposes a state version alongside the current state, improving reliability in distributed execution scenarios.
amber/src/main/scala/org/apache/texera/amber/engine/common/statetransition · high confidence
Workflow compiler refactored into common module with loop variable support
The workflow compiler has been unified into the common/workflow-compiler module, introducing a new LogicalPlan model and strict error handling modes (Lenient for editing, Strict for execution). This change enables the compiler to correctly bind and validate loop variable references ($i) within operators, preventing errors when loop variables are used in properties that are resolved at compile time, such as output column names or partitioning keys.
workflow-compiler · high confidence
Workflow core refactored to use simplified State, Iceberg storage, and new partition types
The workflow core has been updated to simplify the internal State representation to a JSON dictionary, while adding loop-control bookkeeping columns (loop counter and start ID) to support loop operators. Storage is now backed by Iceberg tables via a new DocumentFactory and VFS URI scheme, enabling document existence checks and create-or-reuse logic for output storage. Additionally, a new OneToOnePartition type has been added to the PartitionInfo hierarchy to support one-to-one data distribution strategies.
workflow-core · high confidence
Fixes
3 commits (2 fixes) fixing amber
A fix in amber — 3 commits (2 fixs), 7 files: “fix: Clean up annotations and remove redundant request IDs (\#9275)”; “fix(amber): return 403 instead of 401 when a user lacks access to an execution (\#9325)”; “feat(workflow-operator): export the JSON Lines and Arrow sources as Python (\#8704)” (refs \#9275, \#9325, \#8704). Also in this area: Added unit tests for UserQuotaResource.
amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/quota · medium confidence · unverified
Test coverage
1 commit adding/updating tests in common/config/src/test/scala/org/apache/texera/observability; 15 commits adding/updating tests in bin/local-dev/tests; 3 commits adding/updating tests in amber/src/test/python/pytexera/udf/examples; Added NonParallelTest annotation to serialize flaky shared-resource tests; Added Python test coverage for core Amber runnables; Added end-to-end tests for Amber engine batch size propagation, data processing, pause/resume, and reconfiguration; Added fault-tolerance tests for checkpointing and log replay; Added integration test suite for loop, multi-region, and reconfiguration workflows; Added integration tests for AccessControlService bootstrap and initialization; Added integration tests for Iceberg storage components; Added smoke tests for single-node deployment and example loading scripts; Added test coverage for Amber web services; Added test coverage for Amber worker promise handlers; Added test coverage for GlobalPortIdentity and PortIdentity serialization; Added test coverage for SiteSettings, SqlServer transactions, and UserWarehouse schema; Added test coverage for dashboard search and resource logic; Added test coverage for dataset and workflow resource endpoints; Added test coverage for storage utilities and LakeFS client; Added test coverage for the authentication resource module; Added test coverage for the web application's core components; Added test coverage for workflow resource endpoints; Added test data resources for workflow-core; Added test suite for VirtualDocument storage model; Added tests for AI assistant initialization logic; Added tests for ActorCommandHandler base class behavior; Added tests for ComputingUnitNodeLocator and MounterClient; Added tests for EvaluatedValue serialization in ControlReturn; Added tests for FileResolver and RepositoryMountManager; Added tests for Git version control storage and readonly local file documents; Added tests for LargeBinary I/O and S3 storage integration; Added tests for LargeBinary model validation and behavior; Added tests for Python proxy client and server components; Added tests for Python storage components; Added tests for State JSON serialization and loop-bookkeeping envelope handling; Added tests for UserActivityEventListener; Added tests for WebSocket protocol serialization and deserialization; Added tests for access control resource routing and privilege checks; Added tests for access-control-service resource and utility components; Added tests for computing unit access resolution and auth header fields; Added tests for console message error reporting and print replacement utilities; Added tests for partitioner send semantics; Added tests for the Amber control-plane architecture; Added tests for the Python worker startup configuration parsing; Added tests for the Warehouse REST API resource; Added tests for user joining reason endpoints; Added unit and integration tests for Iceberg storage components; Added unit and integration tests for InputManager, OutputManager, and state materialization; Added unit tests for AddressInfo and WorkerExecution components; Added unit tests for Amber engine common components; Added unit tests for Amber scheduling configuration classes; Added unit tests for AmberClient and ClientActor; Added unit tests for AmberMessage envelopes and DataFrame payload logic; Added unit tests for AsyncRPCClient and AsyncRPCServer; Added unit tests for ConfigParserUtil; Added unit tests for ErrorUtils; Added unit tests for JWT authentication and role-based authorization; Added unit tests for NetworkOutputBuffer and non-range partitioners; Added unit tests for PyBuilder validation and rendering logic; Added unit tests for Python AsyncRPC components; Added unit tests for Python UDF example operators and UI parameter support; Added unit tests for Python Virtual Environment resource and websocket endpoints; Added unit tests for Python core utilities and worker wiring; Added unit tests for StateManager and WorkerStateManager; Added unit tests for StoppableQueueBlockingRunnable lifecycle and error handling; Added unit tests for TimedBuffer; Added unit tests for admin execution and user resources; Added unit tests for config object defaults; Added unit tests for control-plane worker handlers; Added unit tests for coordinator promise handlers; Added unit tests for core data models and operators; Added unit tests for core executor traits and state reference binding; Added unit tests for customized queue utilities and LinkedBlockingMultiQueue; Added unit tests for dataset statistics utilities; Added unit tests for resource scheduling policies; Added unit tests for schema model and Arrow conversion utilities; Added unit tests for storage model and runnables; Added unit tests for storage model classes; Added unit tests for storage result models and schemas; Added unit tests for the Amber Coordinator architecture; Added unit tests for the Amber messaging layer components; Added unit tests for the Amber record-storage subsystem; Added unit tests for the Hub dashboard resource and entity models; Added unit tests for the Projection operator; Added unit tests for the Python worker bridge and internal queue; Added unit tests for the cluster membership listener; Added unit tests for the common auth module; Added unit tests for the log-replay subsystem; Added unit tests for the tuple and schema core library; Added unit tests for web resource endpoints; Added unit tests for worker manager components; Added unit tests for workflow-core core types and partitioning logic; Added unit tests for workflow-core utility modules; Added validation to ensure Helm template values are defined in values.yaml.
Dependencies
Amber operator environment and licensing structure established
The amber module now includes a structured dependency and licensing setup. The operator-requirements.txt file defines the runtime Python dependencies for operators, including specific versions for pillow (12.3.0), torch (2.13.0), scikit-learn (1.7.2), and transformers (5.10.1). A separate dev-requirements.txt file isolates test and development tools like pytest (9.0.3) and ruff (0.14.7). Additionally, the module introduces LICENSE-binary-java, LICENSE-binary-python, and NOTICE-binary files to track third-party license compliance for Java and Python components, alongside R package metadata in DESCRIPTION and code formatting rules in .scalafmt.conf and .scalafix.conf.
amber · high confidence
Relocate core services and standardize build configurations
This change relocates several backend services to new directories (access-control-service, agent-service, amber, config-service, computing-unit-managing-service, file-service, workflow-compiling-service, workflow-operator) and introduces standardized build.sbt files for each. The build configuration updates include upgrading Apache Pekko to 1.7.0, Jackson to 2.18.8, and Hadoop to 3.5.0, while excluding conflicting Jersey and log4j 1.x dependencies to ensure compatibility with Dropwizard 4.0.7 and Arrow 19. The agent-service is now defined with a package.json specifying dependencies like Elysia 1.4.29 and AI SDK 4.0.36. Amber's build.sbt includes Pekko, Dropwizard, Lucene, Hadoop, and Google service dependencies, with specific exclusions for Hadoop's web stack. Python dependencies in amber are updated via requirements.txt (e.g., numpy 2.1.0, pandas 2.2.3, pyarrow 23.0.1) and configured with pyproject.toml for Ruff and Pytest.
(dependencies) · high confidence
Housekeeping
Added per-module license and notice binary files
The access-control-service now includes generated LICENSE-binary and NOTICE-binary files. These files consolidate the Apache License 2.0 terms and the third-party attribution notices for dependencies such as Eclipse Jetty and Jersey, ensuring compliance with open-source licensing requirements for the distributed binary.
access-control-service · high confidence
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
How this codebase got here
Score
- CAI 40 → 40 (-0.4)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.2) — scores are not directly comparable.
Lenses
- Code Health 51 → 51 (+0.1)
- Architecture 92 → 92 (-0.3)
- Maturity 40 → 64 (+23.7)
- Readiness 33 → 25 (-8.0)
- Security 60 → 62 (+1.7)
- Accessibility 43 → 42 (-0.2)
- Performance 78 → 100 (+22.3)
Resolved (163)
- Change coupling: joint-ui.service.ts ↔ sync-texera-model.ts (frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts)
- Critical CVE: [GHSA redacted] (amber/operator-requirements.txt)
- Documentation: contradicts the code (docs/contribution-guidelines/guide-for-developers.md)
- Duplicated block (11 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/difference/DifferenceOpDesc.scala)
- Duplicated block (12 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/regex/RegexOpDesc.scala)
- Duplicated block (17 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/ParallelCSVScanSourceOpDesc.scala)
- Duplicated block (21 lines × 2) (common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/util/dataset/JGitVersionControl.java)
- Duplicated block (65 lines × 3) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/carpetPlot/CarpetPlotOpDesc.scala)
- ExecutionResultService.convertTuplesToJson (cognitive 18) (amber/src/main/scala/org/apache/texera/web/service/ExecutionResultService.scala)
- FunctionTooLong: hugging-face-field-bindings.applyHuggingFaceFieldBindings (frontend/src/app/workspace/util/hugging-face-field-bindings.ts)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- High CVE: [GHSA redacted] (frontend/yarn.lock)
- …and 143 more
New (161)
- AggregateOpDesc.aggExprGroupby (cognitive 20) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/aggregate/AggregateOpDesc.scala)
- AggregateOpDesc.aggExprGroupby (cyclomatic 20) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/aggregate/AggregateOpDesc.scala)
- AggregateOpDesc.aggExprScalar (cognitive 20) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/aggregate/AggregateOpDesc.scala)
- AggregateOpDesc.aggExprScalar (cyclomatic 20) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/aggregate/AggregateOpDesc.scala)
- Change coupling: WorkflowAccessResource.scala ↔ WorkflowResource.scala (amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowAccessResource.scala)
- Critical CVE: [GHSA redacted] (amber/operator-requirements.txt)
- Critical CVE: [GHSA redacted] (frontend/yarn.lock)
- Dependency hygiene PARTLY measured — sbt declarations read, dependency currency not (Maven Central unreachable)
- Documentation: no build, run or usage example (pyright-language-service/README.md)
- Duplicated block (11 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/set/difference/DifferenceOpDesc.scala)
- Duplicated block (18–20 lines × 3) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala)
- Duplicated block (37–45 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala)
- Duplicated block (5 lines × 2) (amber/src/main/scala/org/apache/texera/web/resource/dashboard/WorkflowSearchQueryBuilder.scala)
- Duplicated block (6 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanOpDesc.scala)
- Duplicated block (6 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/bulletChart/BulletChartOpDesc.scala)
- Duplicated block (7 lines × 2) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/regex/RegexOpDesc.scala)
- FileScanOpDesc.generateStandaloneCode (cognitive 22) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanOpDesc.scala)
- FileScanSourceOpDesc.generateStandaloneCode (cognitive 31) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDesc.scala)
- FileScanSourceOpDesc.generateStandaloneCode (cyclomatic 18) (common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDesc.scala)
- High CVE: [CVE redacted] (agent-service/bun.lock)
- …and 141 more
Changes since last survey
- 131 commits — 97 feature/other, 34 fixes
By area
- frontend/src — 51 commits
- common/workflow-operator — 22 commits
- amber/src — 19 commits
- .github/workflows — 5 commits
- agent-service/src — 4 commits
- common/workflow-core — 4 commits
- (root) — 3 commits
- bin/k8s — 3 commits
- common/config — 3 commits
- common/observability — 3 commits
- computing-unit-managing-service/src — 3 commits
- bin/dockerfiles — 2 commits
- bin/single-node — 2 commits
- access-control-service/LICENSE-binary — 1 commit
- bin/forum — 1 commit
- bin/licensing — 1 commit
- bin/local-dev — 1 commit
- file-service/src — 1 commit
- pyright-language-service/package.json — 1 commit
- pyright-language-service/src — 1 commit
Notable commits
- fix: fix(agent-service): check operator edits before changing the workflow (#8981)
- fix: fix(agent-service): send the delegate user's token on compile requests (#9079)
- fix: fix(amber): compare dashboard workflow uids by value (#8647)
- fix: fix(amber): guard AmberRuntime scheduling against null actor system (#9153)
- fix: fix(amber): return 403 instead of 401 when a user lacks access to an execution (#9325)
- fix: fix(amber): return 404 for unavailable public workflow (#9061)
- fix: fix(ci): raise merge queue timeout (#9096)
- fix: fix(deploy): route /api/access/computing-unit in nginx (#9283)
- fix: fix(deps, ci): bump sbt/setup-sbt to v1.5.14 to restore CI (#9346)
- fix: fix(deps, ci): bump setup-sbt and playwright (#9100)
- fix: fix(deps, ci): bump the Docker actions to restore the image build (#9301)
- fix: fix(file-service): return 404 for another dataset's version and 400 for a missing upload type (#9099)
- fix: fix(file-service, workflow-core): delete the temp files a version zip download leaves behind (#9145)
- fix: fix(frontend): block clicks on disabled context menu items (#8968)
- fix: fix(frontend): clear the picked version when switching datasets in the dataset picker (#8928)
- fix: fix(frontend): clear the search bar’s suggestions when the user changes (#9090)
- fix: fix(frontend): fall back to a running unit on workflow load (#8935)
- fix: fix(frontend): handle model hits in search-bar autocomplete (#8701)
- fix: fix(frontend): hold the Python UDF parameter table's Name and Type cells locked when the panel enables its form group (#8848)
- fix: fix(frontend): keep the computing-unit refresh and poll alive after a failed request (#8854)
- …and 111 more
Architecture
- Containers 0 added · 0 removed · contexts 1 added · 0 removed · edges 1 added · 5 removed
Added bounded contexts (1)
- observability
Added dependency edges (1)
- amber → observability (coupling)
Removed dependency edges (was a dependency, now gone) (5)
- amber → computing-unit-managing-service
- computing-unit-managing-service → amber
- computing-unit-managing-service → pybuilder
- workflow-core → pybuilder (coupling)
- workflow-operator → amber (coupling)
Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.
Survey your own repository
apache/texera was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.
About this page
- The score is its most recent published measurement, taken on 7 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
- Measured at commit d6c9050c5d78d685cf246d97fce004d5675cdc12 — the exact code this score is about.
- Scored under rubric-2026.11.2 — the same rubric and the same method as every other entry in this index.
- Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.3+preprod-5f8314471808.