Hydrospheredata/mist

Hydrosphere Mist is a serverless proxy for Apache Spark clusters that manages the lifecycle of batch and streaming jobs across Scala, Java, and Python runtimes. It provides a centralized master service to deploy, execute, and monitor functions, while workers handle the actual Spark context initialization and job execution. The system exposes an HTTP API and WebSocket streaming for real-time status updates, supporting job submission via HTTP or MQTT and persisting execution history in a relational database.

36 changes.

The project was rebranded from Lymph to Hydrosphere Mist and rebuilt from the ground up as a serverless proxy for Spark clusters. This period established the core architecture, including the master-worker model, HTTP API, and support for Scala, Java, and Python jobs, alongside comprehensive testing and CI infrastructure.

18 changes.

This period focused on introducing Python job execution capabilities through the new mistpy library and Py4J integration, alongside significant refactoring of the worker and execution layers. Key changes included implementing per-job connection isolation, centralizing status reporting, and establishing robust Docker and local worker starters. The work also involved enhancing the API with improved JSON encoding, remote logging, and comprehensive test coverage for the new features.

The project now includes two new configuration files: \configs/default.conf\ and \configs/docker.conf\. The default configuration sets the Spark master to local mode and configures the worker runner to use the local runner. The Docker configuration similarly sets the Spark master to local mode but configures the worker runner to use Docker, specifying the image name and network type as 'auto-master'. These files provide baseline settings for running the application in local and Docker environments respectively.

configs · high confidence

New Python scripts have been added to the examples/misc directory to demonstrate how to interact with the MIST platform. The manual\_worker\_start.py script shows how to construct and execute a spark-submit command using environment variables for configuration. The run\_job\_http.py script provides an example of triggering a job and polling for results via the HTTP API. Additionally, run\_job\_mqtt.py demonstrates job execution and result retrieval using the MQTT protocol, including handling job status events and logs.

examples · high confidence

A new executable script \sbt/sbt\ has been added to the project to manage the sbt build tool. This script automatically downloads and caches the sbt launch JAR (version 1.1.5) from Maven Central if it is not already present locally, allowing users to run sbt without a global installation. It supports both \wget\ and \curl\ for downloads and configures specific JVM memory settings for the build process.

sbt · high confidence

A new SparkClassLoader utility has been introduced in the worker module to manage class loading with specific URLs. This component wraps the parent class loader, adding provided URLs if the parent is a MutableURLClassLoader, or creating a new MutableURLClassLoader otherwise, ensuring that custom class paths are correctly integrated into the Spark execution environment.

mist/worker/src/main/scala/org · high confidence

Added a new Java Domain-Specific Language (DSL) in the \mist.api.jdsl\ package, enabling Java users to define jobs using \JavaSparkContext\, \JavaStreamingContext\, and \SparkSession\. This release introduces \JArgsDef\ for declaring typed arguments (int, double, string, boolean, lists, and options) with validation support, and \JEncoders\ to handle serialization of Java types to JSON. The entry point \Jdsl\ combines these components to provide a cohesive API for Java-based job definitions.

mist-lib/src/main/scala/mist/api/jdsl · high confidence

The mist-lib module now includes the necessary configuration files (setup.py and setup.cfg) to build and distribute the 'mistpy' Python library. This enables users to install the library via standard Python packaging tools. The package is configured to support Python 2.7, 3.4, and 3.5, and includes pytest 4.6.4 as a test requirement.

mist-lib/src/main/python · high confidence

The master service now includes a default configuration file (master.conf) that defines networking, cluster, HTTP, database, worker, and security settings. This file establishes baseline values for connection pooling, worker modes, and Akka actor system behavior, providing a structured starting point for deployment without requiring manual configuration overrides.

mist/master/src/main/resources · high confidence

Introduces the \mistpy\ Python library, providing the runtime components necessary to execute Python-based Spark jobs within the Mist environment. The library includes a \ContextWrapper\ to manage Spark contexts (SparkContext, SQLContext, HiveContext, SparkSession, StreamingContext), decorators for defining function arguments and system dependencies, and entry-point handlers to load and invoke user code from both single-file scripts and JAR/EGG archives. It also features a metadata extractor to communicate function signatures and type hints to the Scala backend via Py4J, enabling proper argument mapping and execution.

mist-lib/src/main/python/mistpy · high confidence

The worker starter module now includes new implementations for launching workers via Docker (DockerStarter) and locally (LocalSparkSubmit), replacing or supplementing previous mechanisms. The DockerStarter handles container creation, networking configuration (bridge, host, or named networks), and lifecycle management using the Docker Java client. The LocalSparkSubmit executes workers directly on the host, logging output to a specified directory. Both starters rely on a new SparkSubmitBuilder and PsUtil, which introduce robust argument parsing for complex run options, ensuring that user-provided commands are correctly split and passed to spark-submit. This change enables more flexible worker deployment strategies and better handling of complex command-line arguments.

mist/master/src/main/scala/io/hydrosphere/mist/master/execution/workers/starter · high confidence

Added the core Scala components for executing Python jobs within the MIST worker. PythonExecuter.scala implements the logic to launch a Python process via py4j, handle argument metadata extraction, and manage the gateway server lifecycle for communication between the JVM and Python scripts. PythonUtils.scala provides helper methods for converting Java collections to Scala sequences and defining argument type mappings, supporting the integration of Python functions into the Mist runtime.

mist/worker/src/main/scala/io/hydrosphere/mist/python · high confidence

Added new Scala source files defining the \AsyncInput\ trait and its implementations for Kafka (\TopicConsumer\/\TopicProducer\) and MQTT (\MqttAsyncClient\). This introduces a structured interface for consuming asynchronous messages from these brokers, with the MQTT implementation explicitly enabling automatic reconnection via \setAutomaticReconnect(true)\ to improve connection resilience.

mist/master/src/main/scala/io/hydrosphere/mist/master/interfaces/async · high confidence

A new StatusReporter component has been added to handle job status updates. This component coordinates with the job repository, event streamer, and logging service to ensure that status changes are persisted, broadcast to connected clients, and recorded in logs, providing a unified way to track job execution state.

mist/master/src/main/scala/io/hydrosphere/mist/master/execution/status · high confidence

The execution module now includes a new ExecutionService class that centralizes job lifecycle management, including starting and stopping jobs, querying job status, and managing worker connections. This service integrates with the job repository, status reporter, and worker hub to handle job execution requests and cancellations. Additionally, a FutureSubscribe trait has been added to provide a mechanism for subscribing to future completions within actors, enabling cleaner asynchronous event handling in the execution flow.

mist/master/src/main/scala/io/hydrosphere/mist/master/execution · high confidence

A new PythonRunner component has been added to the worker module, enabling the execution of Python-based jobs. This implementation bridges the Mist worker infrastructure with the PythonFunctionExecutor, handling job requests by mapping artifact paths and invoking the Python execution context within the Spark environment.

mist/worker/src/main/scala/io/hydrosphere/mist/worker/runners/python · high confidence

Added two new utility files to the Akka utils package: ActorF.scala provides a factory pattern for creating child actors with a consistent API, including syntax extensions and static/props helpers for easier instantiation and testing; WhenTerminated.scala introduces a helper to monitor actor termination and return a Future that completes when the watched actor stops, simplifying lifecycle management.

mist/core/src/main/scala/io/hydrosphere/mist/utils/akka · high confidence

The Mist master now exposes a new HTTP interface layer that includes CORS support, a static UI server for single-page applications, a hidden development endpoint for running jobs, and WebSocket routes for real-time event streaming. Users can now access the UI via the /ui path, trigger development runs via the hidden /v2/hidden/devrun endpoint, and subscribe to job or system events over WebSockets at /v2/api/ws/all or /v2/api/ws/jobs/{id}, with an option to include logs in the stream.

mist/master/src/main/scala/io/hydrosphere/mist/master/interfaces/http · high confidence

The examples module now includes new Java and Python implementations for Spark-based functions, complementing the existing Scala examples. Java users can now reference JavaPiExample, JavaSparkContextExample, JavaStreamingContextExample, and JavaTestingExample, which demonstrate usage of the new Java DSL (Jdsl) for defining arguments, handling Spark contexts, and managing streaming data. Python users gain access to session\_example, sparkctx\_example, sqlctx\_example, and streamingctx\_example, providing clear patterns for interacting with SparkSession, SparkContext, SQLContext, and StreamingContext using the mistpy decorators. These additions expand the available reference implementations for users adopting Java or Python with the Mist framework.

examples/examples · high confidence

This change introduces a new Mist API library (mist-lib) featuring a redesigned function declaration API with explicit argument type extraction (ArgType) and a custom JSON-like data structure (JsData) to decouple worker-master communication from third-party JSON libraries. It also adds a new example demonstrating the less verbose function syntax (LessVerboseExample) and a comprehensive HTTP integration test suite (InsideDockerSpec, MistHttpInterface) that validates function deployment and execution within Docker containers.

repository · high confidence

The encoding subsystem now supports automatic extraction of case classes from JSON data, including the ability to fill in missing fields with their default values. This is achieved through new \ObjectExtractor\ and \DefaultsPatcher\ components that work together to parse JSON into typed objects, while \ObjectEncoder\ handles the reverse serialization. Users can now rely on the library to handle complex object structures and default value patching automatically during data ingestion.

mist-lib/src/main/scala/mist/api/encoding · high confidence

The system now uses a new \ArtifactRepository\ implementation to manage artifacts, introducing a file-system based storage layer (\FsArtifactRepository\) that stores files in a specified root directory. This repository supports listing, retrieving, storing, and deleting artifacts, with a fallback mechanism (\SimpleArtifactRepository\) that also checks default endpoints defined in \FunctionConfig\. The creation process ensures the storage directory exists at startup.

mist/master/src/main/scala/io/hydrosphere/mist/master/artifact · high confidence

Added executable shell scripts in the bin directory to start and stop the Mist Master and the Function Info Provider. The master script now supports configuration via MIST\_OPTS, accepts command-line flags for config files and debug mode, and manages a PID file for process control. The new function info provider script handles Spark environment setup and launches the provider class with appropriate logging and memory configurations.

bin · high confidence

The Mist core module now includes a new set of utility objects in the \io.hydrosphere.mist.utils\ package to support internal operations. \ConfigUtils\ provides extension methods for Typesafe Config to safely retrieve finite durations and optional values. \Collections\ offers recursive conversion helpers between Scala and Java collections. \EitherOps\ and \FutureOps\ supply functional syntax for handling \Either\ and \Future\ types, reducing external dependencies. \NetUtils\ adds a helper to find the local IP address, and \Logger\ provides a standard SLF4J logging trait.

mist/core/src/main/scala/io/hydrosphere/mist/utils · high confidence

The project has been renamed from 'Lymph' to 'Hydrosphere Mist', as reflected in the updated README.md which now describes it as a serverless proxy for Spark clusters. This commit establishes the initial repository structure, including the sbt build configuration (mist.sbt) defining the core, master, worker, and library modules, along with CI pipelines for Travis CI and Jenkins to test against multiple Spark (2.1.0–2.4.0) and Scala (2.11.12, 2.12.7) versions. It also introduces Docker support via a new entrypoint script, JVM optimization options, and standardizes the license under Apache 2.0.

(repo-wide) · high confidence

A new \Json4sConversion.scala\ file has been added to the \spark-2.4.0\ module to provide JSON parsing and formatting capabilities using the json4s library. This implementation supports running Mist jobs directly via spark-submit by converting between JSON strings and internal data structures (JsData). The file mirrors the existing implementation in the default \spark\ module but is specifically placed in the Spark 2.4.0 source directory to ensure compatibility with that version.

mist-lib/src/main/spark · high confidence

The application now persists job execution data using a new \job\_details\ table, available for both H2 and PostgreSQL databases. The schema includes fields for job identification (\job\_id\, \external\_id\), execution context (\path\, \class\_name\, \namespace\, \action\, \source\), parameters, timing (\start\_time\, \end\_time\, \create\_time\), results, status, and worker assignment. A specific behavioral change is applied to the column naming: the \endpoint\ column is renamed to \function\ in the H2 migration (V2), while the PostgreSQL migration (V2) creates the table directly with the \function\ column name, ensuring consistent schema evolution across supported databases.

mist/master/src/main/resources/db · high confidence

A new \Level\ sealed trait and its companion object have been added to the core logging package, defining four specific log levels (Debug, Info, Warn, Error) with integer values and string names. This change introduces a local abstraction for log severity, replacing or supplementing previous logging mechanisms, and allows for programmatic conversion from integer codes to these named levels.

mist/core/src/main/scala/io/hydrosphere/mist/core · high confidence

The job persistence layer in the master store has been rewritten to use the Doobie functional SQL library and HikariCP connection pooling, replacing the previous Slick-based implementation. This change introduces a new \HikariDataSourceTransactor\ for managing database connections and a \HikariJobRepository\ that executes queries via Doobie fragments. To support different database engines, the update logic now uses database-specific SQL dialects: H2 uses a \MERGE\ statement, while PostgreSQL uses \INSERT ... ON CONFLICT DO UPDATE\. The repository also integrates Flyway for optional schema migrations and supports filtering jobs by status, worker ID, and function ID with pagination.

mist/master/src/main/scala/io/hydrosphere/mist/master/store · high confidence

The Mist master service has been restructured to improve reliability and separation of concerns. A new event streaming system (EventsStreamer) now broadcasts system events to subscribers, while a JobEventPublisher enables external notification of job events via Kafka or MQTT. Job information extraction has been decoupled into a separate process (FunctionInfoProviderRunner) that runs as a distinct service, communicating with the master via an actor-based registration mechanism. Context management is now handled through a dedicated CRUD mixin (ContextsCRUDMixin) that ensures storage updates are synchronized with the execution service. These changes provide a more robust foundation for job execution, monitoring, and configuration management.

mist/master/src/main/scala/io/hydrosphere/mist/master · high confidence

The master service now includes a dedicated JSON support layer (SprayJsonSupport and jsonCodecs) to handle serialization and deserialization of API payloads. This introduces specific codecs for job details, worker information, and Mist's internal data structures (JsData), enabling the HTTP API to correctly parse and return JSON responses for job status, configuration, and execution results.

mist/master/src/main/scala/io/hydrosphere/mist/master/interfaces · high confidence

The worker now includes a new Python entry point (\_\main\\_.py) that routes execution to either the metadata extractor or the Python script executor based on command-line arguments. Additionally, a new job-extractor.conf file has been added to configure the separate job info extraction process, setting Akka remote transport parameters (such as maximum-frame-size) to match the master configuration. The existing worker.conf has also been updated to include HTTP server settings (transparent-head-requests and idle-timeout) and cluster host/port defaults.

mist/worker/src/main/resources · high confidence

The worker now uses a new \JobRunner\ trait and \RunnerSelector\ to dispatch jobs based on artifact file extension. Python artifacts (\.py\, \.egg\) are handled by \PythonRunner\, while Scala artifacts (\.jar\) are handled by \ScalaRunner\. The \ScalaRunner\ implementation loads the job class using a dedicated classloader and executes it via the \FunctionInstanceLoader\, returning results as \JsData\ wrapped in an \Either\ to handle failures explicitly.

mist/worker/src/main/scala/io/hydrosphere/mist/worker/runners · high confidence

The application now provides three distinct Log4j configuration files to control logging verbosity and output targets. The 'default' profile logs to a file at INFO level, the 'debug' profile adds console output for detailed tracing, and the 'infoprovider' profile directs logs to a separate file while suppressing verbose output from Spark, Jetty, Parquet, and Hive components to reduce noise.

configs/logging · high confidence

A new RemoteAppender class has been added to the worker logging module, enabling job logs to be written to a remote destination via a LogsWriter. This appender extends AppenderSkeleton and translates Log4j LoggingEvents into internal LogEvent objects (INFO, DEBUG, ERROR, WARN) before sending them, addressing previous issues with job logging where the previous implementation's finalize behavior caused problems.

mist/worker/src/main/scala/io/hydrosphere/mist/worker/logging · high confidence

The logging subsystem has been replaced with a new implementation that captures structured log events from jobs via a TCP server and Akka Streams. This change introduces a \LogService\ that accepts log messages, batches them (up to 1,000 events or every second), and persists them using a dedicated writer while streaming updates to an event streamer. The system uses Kryo serialization for efficient transmission and includes a resuming supervision strategy to ensure log collection continues even if batch writes fail, providing more reliable and detailed job execution logs for users.

mist/master/src/main/scala/io/hydrosphere/mist/master/logging · high confidence

New wrapper classes (ConfigurationWrapper, DataWrapper, ErrorWrapper, SparkStreamingWrapper) have been added to the Python runner location to structure how job configuration, input data, errors, and streaming durations are passed to and from the Python subprocess.

mist/worker/src/main/scala/io/hydrosphere/mist/worker/runners/python/wrappers · medium confidence

The Mist Scala API has been refactored to replace the previous \Map\[String, Any\]\ approach with a strongly-typed, composable argument definition system. Users now define jobs by extending \MistFn\ and using \withArgs\ to declare specific input arguments (e.g., \`arg\[Int\](

mist-lib/src/main/scala/mist/api · high confidence

The data layer for MIST has been restructured to persist Context and Function configurations as individual .conf files on the filesystem rather than relying on in-memory or monolithic configuration structures. This change introduces a new \FsStorage\ abstraction with read-write locking to safely manage concurrent access to these configuration files, alongside dedicated storage classes (\ContextsStorage\ and \FunctionConfigStorage\) that merge these persisted files with default settings. Users will benefit from more robust handling of configuration updates and deletions via the HTTP API, as the system now explicitly supports creating, reading, updating, and deleting individual context and function definitions stored on disk.

mist/master/src/main/scala/io/hydrosphere/mist/master/data · high confidence

The worker execution layer has been restructured to improve connection lifecycle handling and isolation. A new \ConnectionsMirror\ now centrally tracks active worker connections, while \WorkerHub\ orchestrates the startup and shutdown of workers and connectors. The introduction of \PerJobConnection\ and \ExclusiveConnector\ enables per-job connection isolation, allowing connections to be released back to the pool after a job completes rather than remaining permanently bound. This change also refactors the \WorkerRunner\ to better handle process termination and registration, ensuring that connections are properly cleaned up when workers fail or are stopped.

mist/master/src/main/scala/io/hydrosphere/mist/master/execution/workers · high confidence

The Mist worker has been refactored to handle only one job request at a time, ensuring sequential execution. It now introduces a CancellableFuture mechanism that allows jobs to be interrupted via thread interruption, supporting proper cancellation of both batch and streaming jobs. Additionally, the worker integrates remote log appending, which directs Spark logs for specific jobs to a remote appender, and provides full stack traces for job invocation errors to improve debugging visibility.

mist/worker/src/main/scala/io/hydrosphere/mist/worker · high confidence

The Mist worker now automatically determines whether a JVM-based function is written in Scala or Java. This is achieved by inspecting the bytecode of the loaded class via the new JvmLangDetector, which checks the source file name embedded in the class file. The detected language is exposed through the BaseFunctionInstance trait, allowing the system to distinguish between Scala and Java implementations at runtime, while Python functions remain explicitly identified as "python".

mist/worker/src/main/scala/io/hydrosphere/mist/job · high confidence

The library now correctly enables Hive support when requested by explicitly setting the catalog implementation configuration before calling \enableHiveSupport()\. This change ensures that Spark sessions are initialized with Hive capabilities in Spark versions 2.3.0 and later, preventing failures or missing functionality for users relying on Hive integration.

mist-lib/src/main/scala/org · high confidence

The internal argument processing system has been refactored to resolve issue \#478. This change introduces new internal components (ArgCombiner, ArgDefJoiner, HLister, JoinToTuple) that handle the combination and validation of function arguments more robustly, ensuring that argument extraction and validation errors are correctly aggregated when multiple arguments fail.

mist-lib/src/main/scala/mist/api/internal · high confidence

Added comprehensive test coverage for the Mist worker module; Added core test infrastructure and specifications; Added integration tests for local process execution and PostgreSQL job repository; Added master service test suite and infrastructure; Added test resources for Hive job data and logging configuration; Added test resources for Hive jobs and new Python API context; Added tests for Java API argument definitions and validation; Added tests for JsData conversion and serialization; Added tests for argument decorators and executable entry metadata; Added tests for generic case class encoding/decoding and schemed row encoding; Added unit tests for the Mist API argument extraction and job definition logic.

This change introduces the foundational SBT build configuration for the project, establishing the build tool version (sbt 1.2.1) and defining core library dependencies such as Akka 2.5.9, Doobie 0.6.0, and Spark components. It adds build plugins for Docker, Sonatype publishing, and code coverage, alongside custom sbt tasks to manage the Mist UI download, Python project distribution, and artifact staging. Additionally, it includes a boilerplate generator for creating functional interfaces and syntax helpers, and configures Maven repository proxies for dependency resolution.

project · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 53.

• Code Health 98
• Architecture 87
• Maturity 50
• Readiness 41
• Security 67

• 300 commits — 186 feature/other, 114 fixes

• mist/master — 72 commits
• (repo) — 68 commits
• (root) — 51 commits
• mist/worker — 32 commits
• mist-lib/src — 30 commits
• docs/src — 22 commits
• examples/examples — 4 commits
• mist-tests/scala — 4 commits
• project/Ui.scala — 4 commits
• mist-lib-py/mistpy — 3 commits
• mist/core — 2 commits
• project/PyProject.scala — 2 commits
• bin/mist-worker — 1 commit
• configs/default.conf — 1 commit
• examples/examples-python — 1 commit
• examples/misc — 1 commit
• mist-lib-py/tests — 1 commit
• project/Library.scala — 1 commit

• fix: - Http api - print error stack traces - Job repository - fix sql building for query with statuses - Job actor - handle artifact downloading failure
• fix: Fix #501 - provide full stack traces from job invocation errors
• fix: Fix #503 - handle DELETE /v2/api/jobs/{id}
• fix: Fix broken Kafka client
• fix: Fix comments issued by dos65
• fix: Fix job cancellation - respond with correct job details Fix connection leak on context frontend
• fix: Fix microsite name
• fix: Fix/broken context handling (#484)
• fix: Fix/info provider startup fix (#517)
• fix: Fix/spark context init (#516)
• fix: Fix/worker directory (#446)
• fix: Merge branch 'fix/rewrite_cluster' of github.com:Hydrospheredata/mist into fix/job_evt_logging
• fix: Merge branch 'fix/rewrite_cluster' of github.com:Hydrospheredata/mist into fix/rewrite_cluster_complete
• fix: Merge branch 'master' of github.com:Hydrospheredata/mist into fix/manual_mode_fixes
• fix: Merge branch 'master' of github.com:Hydrospheredata/mist into fix/restart_infoprovider
• fix: Merge pull request #416 from Hydrospheredata/fix/rewrite_cluster
• fix: Merge pull request #426 from Hydrospheredata/fix/rewrite_cluster_complete
• fix: Merge pull request #428 from Hydrospheredata/fix/job_evt_logging
• fix: Merge pull request #429 from Hydrospheredata/fix/job_cancel_fix
• fix: Merge pull request #434 from Hydrospheredata/fix/shared_connector_shutdown
• …and 280 more

• 0 containers · 1 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Hydrospheredata/mist was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.