apache/livy

Apache Livy is a server-side REST service that enables remote interaction with Apache Spark clusters through interactive sessions, batch job submission, and JDBC/ODBC connectivity via a Thrift server. It provides client libraries for Java, Scala, and Python to programmatically manage Spark contexts, submit jobs, and monitor execution states. The system supports multiple execution modes including REPL interpreters for Scala, Python, and R, while ensuring reliability through session recovery, access control, and configurable timeouts.

16 changes.

This period focused on Apache Livy's graduation from the Hue incubator, involving the removal of legacy Hue-integrated code and the restructuring of the repository for standalone operation. The work established the Maven module hierarchy for version 1.0.0, modernized build scripts for Java 17, and introduced a new Python client API alongside comprehensive test infrastructure to support the release.

33 changes.

This period focused on establishing a robust client-side infrastructure by introducing dedicated Java and Scala API libraries, alongside a new HTTP client and Remote Spark Context (RSC) support. Concurrently, the server underwent significant architectural changes, including the implementation of batch and interactive session management, session recovery mechanisms, and role-based access control. The work also encompassed migrating the codebase to the Apache Livy namespace, upgrading the REPL to Python 3, and adding comprehensive test coverage for these new components.

13 changes.

This period focused on expanding Livy's connectivity and security by introducing a dedicated ThriftServer module with comprehensive UI monitoring and LDAP authentication support. It also extended interpreter capabilities to include Scala 2.12 and Spark 4.x/Scala 2.13 REPLs, while establishing a Docker-based local development environment to streamline testing and debugging workflows.

Users can now authenticate against an LDAP directory by configuring the server with LDAP provider URLs, base DNs, and optional domain suffixes. The new LdapAuthenticationHandlerImpl supports both standard LDAP binds and StartTLS for encrypted connections, allowing administrators to secure access using existing LDAP credentials.

server/src/main/scala/org/apache/livy/server/auth · high confidence

The Livy server UI now includes a dedicated view for ThriftServer sessions. This update adds a new HTML table template and JavaScript client that fetches and displays active JDBC/ODBC sessions, showing their Session ID, associated Livy Interactive Session ID (with a link to the session summary), owner, and creation time. This provides visibility into ThriftServer activity directly within the existing Livy dashboard.

thriftserver/server/src/main · high confidence

A new SparkInterpreter class has been added for the Scala 2.12 build, enabling the Read-Eval-Print Loop (REPL) functionality to operate with this specific Scala version. This implementation configures the Scala compiler settings, manages the Spark interpreter loop, and handles classpath management for user jars, allowing users to execute interactive Scala code within Livy when running on Scala 2.12.

repl/scala-2.12 · high confidence

This change introduces a new client-side Python API for interacting with Livy, allowing users to submit and manage Spark jobs directly from Python scripts. The \livy.client\ module provides an \HttpClient\ to connect to a Livy server, with methods to \submit\ jobs for asynchronous execution or \run\ them immediately, as well as \add\_file\ and \add\_jar\ to manage dependencies in the remote context. Jobs are tracked via \JobHandle\, which extends \concurrent.futures.Future\ to support status monitoring, cancellation, and callback registration. The API also defines a \JobContext\ interface that exposes shared Spark contexts (SparkContext, SQLContext, HiveContext, SparkSession, etc.) to the submitted job functions.

python-api/src/main · high confidence

This change introduces the \livy-python-api\ package, providing a Python client for interacting with the Livy REST API. The package is configured via \setup.py\ and \setup.cfg\ to support Python versions 3.8 through 3.14, with dependencies on \requests\, \cloudpickle\, \responses\, and \requests-kerberos\. It adopts PEP 440 compliant versioning (using \.dev0\ instead of Maven-style \-SNAPSHOT\) and sets up the project structure for testing and distribution.

python-api · high confidence

Users can now configure interactive sessions to automatically expire and be cleaned up using two new parameters: \ttl\ (time-to-live) and \idleTimeout\. The \CreateInteractiveRequest\ model accepts these optional string fields, which are validated and passed to the \InteractiveSession\ creation logic. The server enforces these limits through a new \SessionHeartbeat\ trait and a background \SessionHeartbeatWatchdog\ thread that periodically checks for expired sessions and deletes them. Additionally, the \SessionInfo\ returned by the REST API now includes the configured \ttl\ and \idleTimeout\ values, allowing users to inspect the expiration settings of their sessions.

server/src/main/scala/org/apache/livy/server/interactive · high confidence

Users can now connect to Livy via HTTP using the new \HttpClient\ and \HttpClientFactory\ classes located in \client-http/src/main\. This implementation allows applications to submit jobs, upload JARs and files, and manage interactive sessions over HTTP(S), including support for Kerberos/SPNEGO authentication when configured. The client is automatically discovered via the \META-INF/services\ registration, enabling seamless integration with the standard \LivyClientFactory\ SPI.

client-http/src/main · high confidence

Added a new programmatic Java API in the org.apache.livy package, providing interfaces and classes to submit and monitor Spark jobs remotely. The API includes LivyClient for job submission (submit/run), jar/file management, and context control, along with Job and JobContext interfaces for defining job logic and accessing shared Spark contexts. A LivyClientBuilder facilitates client instantiation via URI and configuration, leveraging ServiceLoader for factory discovery, while JobHandle enables job state monitoring and listener registration.

api/src/main · high confidence

The RSC driver now supports shared variables across jobs, allowing users to persist and reuse state between different statements within a session via the new \JobContextImpl\ implementation. Additionally, the driver's Spark integration layer (\SparkEntries\) has been updated to detect and support Spark 4, ensuring compatibility with the latest Spark runtime while maintaining support for Spark 3.

rsc/src/main/java/org/apache/livy/rsc/driver · high confidence

This change adds the core server components for submitting and managing Spark batch jobs. It introduces the \BatchSession\ class to handle session lifecycle, recovery metadata, and interaction with the Spark application, along with the \BatchSessionServlet\ to expose REST endpoints for creating sessions and viewing their status. The \CreateBatchRequest\ model defines the input parameters for job submission, including file paths, arguments, resource configurations (memory, cores, executors), and proxy user support.

server/src/main/scala/org/apache/livy/server/batch · high confidence

This change introduces the new \client-common\ module, which centralizes shared client-side components for the Livy service. It provides a type-safe configuration system (\ClientConf\) with support for typed entries, deprecation warnings, and time-string parsing, alongside a Kryo-based \Serializer\ for efficient job data serialization. The module also includes \AbstractJobHandle\ to manage job lifecycle states and listener notifications, \HttpMessages\ to define the JSON structures for the client protocol (including session details and job status), and utility classes like \BufferUtils\ for byte buffer handling. Additionally, it adds \TestUtils\ to standardize test configurations, such as pinning Spark drivers to loopback addresses for macOS compatibility.

client-common · high confidence

The Livy server now includes a built-in Web UI accessible via the /ui path, allowing users to view all active sessions (interactive, batch, and optionally ThriftServer), inspect individual session details and statements, and view session logs. The UI is built using Scalatra, Bootstrap 3.4.1, and jQuery 3.4.1, and supports reverse proxy configurations through a configurable base path. This change introduces the frontend serving infrastructure for Livy's management interface.

server/src/main/scala/org/apache/livy/server/ui · high confidence

The Livy server now includes a new ThriftServer module, allowing users to connect via standard Thrift protocols in addition to the existing REST API. This change introduces a comprehensive set of configuration properties in LivyConf.scala to manage the new server, including enabling/disabling the service, setting the port, configuring connection limits, and tuning async execution threads. Additionally, the update adds configuration entries for LDAP and Kerberos authentication, security headers, session metrics gating, and Hadoop credential providers, giving administrators finer control over security, performance, and integration with existing Hadoop ecosystems.

server/src/main/scala/org/apache/livy · high confidence

This change introduces the \rsc\ module, a new client-side library for Livy's Interactive Sessions. It adds the core \RSCClient\ implementation, which manages the lifecycle of remote Spark contexts, including launching driver processes via \ContextLauncher\, handling RPC connections, and tracking job states through \JobHandleImpl\. The update also defines the \BaseProtocol\ for message serialization, \RSCConf\ for configuration management, and supporting utilities, establishing the foundation for the new remote context architecture.

rsc/src/main/java/org/apache/livy/rsc · high confidence

Developers can now spin up a local, standalone Spark cluster integrated with Livy using Docker Compose for easier debugging and integration testing. This change introduces a new \dev/docker\ directory containing Dockerfiles, a \build-images.sh\ script to assemble images for the base, Spark, and Livy server components, and a \docker-compose\ setup to launch the cluster. The environment includes pre-configured UIs for Livy (port 8998) and Spark Master/Workers, supports remote debugging via JDWP on port 9010, and allows customization of configurations and logs within the \livy-dev-cluster\ container.

dev/docker, dev/docker/livy-dev-cluster · high confidence

The examples directory now includes new reference implementations for the Livy Job API in Java (PiApp), Python (pi\_app.py), and Scala (WordCountApp). These examples demonstrate how to initialize a Livy client, upload necessary JARs, and submit jobs—ranging from simple parallel computations to Spark Streaming and SQL operations—providing users with concrete templates for integrating with the Livy server.

examples · high confidence

This change introduces a new Scala client library (\org.apache.livy.scalaapi\) that wraps the existing Java Livy client. It provides Scala-native types for job submission and monitoring, including \LivyScalaClient\ for submitting and running jobs, \ScalaJobHandle\ which implements Scala's \Future\ interface for asynchronous result handling and callbacks, and \ScalaJobContext\ for accessing shared Spark contexts and variables within jobs. Users can now interact with Livy using idiomatic Scala constructs like \Future\ and implicit conversions.

scala-api/src/main · high confidence

The dev directory now includes several new scripts to support development workflows and release processes. A Python-based livy-shell script provides an interactive command-line interface for creating and managing Livy sessions. A beeline script allows users to run the Hive Beeline client against a Livy Thrift server. A merge\_livy\_pr.py utility automates the merging of GitHub pull requests into the Apache Git repository, handling conflicts and commit metadata. Additionally, a release-build.sh script streamlines the creation of signed source and binary packages (for Scala 2.11 and 2.12) and their publication to Apache Nexus, while livy-build-info.sh generates version metadata properties. License management is supported by a third-party-file.ftl template and a third-party-missing-license.properties file to track dependency licenses.

dev · high confidence

Livy now includes a new recovery subsystem that persists session state to enable session restoration after server restarts. This location introduces the core storage abstractions and implementations: a \StateStore\ interface with \FileSystemStateStore\ (writing to HDFS with safe-mode checks and atomic writes) and \ZooKeeperStateStore\ (using Curator with SASL and SSL/TLS support), plus a \BlackholeStateStore\ for when recovery is disabled. A \SessionStore\ manages the high-level session metadata, and a \RecoveryServlet\ exposes \/sessions/refresh\, \/batches/refresh\, and \/refresh\ operator endpoints (guarded by superuser access) to import sessions from the state store into a running server.

server/src/main/scala/org/apache/livy/server/recovery · high confidence

The initial code import from Hue has been removed, deleting the legacy \livy-repl\ module and core session management classes from \livy-core\. This includes the removal of the standalone REPL server (\Main.scala\, \WebApp.scala\), interpreter implementations for Python, Scala, and R, and the foundational session infrastructure (\Session\, \SessionManager\, \SessionFactory\, \SessionState\, \Kind\). Consequently, the application no longer supports the original Hue-integrated REPL mode or the specific session lifecycle management logic defined in these deleted files.

(repo-wide) · high confidence

The \livy-yarn\ module has removed its original YARN application submission and management code, including the \Client\ and \Job\ Scala classes, the \ApplicationState\ trait, and associated build artifacts (\dist.xml\, \run-class.sh\, \logback.xml\). This change eliminates the initial code imported from Hue, effectively stripping out the previous mechanism for submitting and tracking YARN applications within this module.

livy-yarn · high confidence

The session management code for the Livy Thrift Server has been extracted into a new \thriftserver/session\ module. This change introduces a new set of internal classes—including \ThriftSessionState\, \StatementState\, \ColumnBuffer\, and \ResultSet\—to handle session lifecycle, statement execution, and result serialization, alongside new job implementations for catalog operations such as retrieving tables, columns, schemas, and functions.

thriftserver/session/src/main · high confidence

A change to existing behaviour in dev/helmchart — 1 commit, 23 files.

dev/helmchart · medium confidence · unverified

The REPL now includes a new \SparkInterpreter\ class specifically for Spark 4.x and Scala 2.13. This implementation adapts the interpreter to Spark 4's API changes, such as using \ReplCompletion\ for code completion instead of reflection-based \PresentationCompilerCompleter\, and adjusting how the Scala compiler classpath is constructed to correctly handle user JARs while excluding conflicting Livy and Scala-reflect artifacts. It also updates internal bindings to use \intp.reporter.withoutPrintingResults\ and retrieves term values via \$result\ to align with Scala 2.13's \IMain\ behavior.

repl/scala-2.13 · high confidence

A new service provider configuration file has been added to register the RSCClientFactory implementation, enabling automatic discovery of the client factory via the Java Service Provider Interface (SPI).

rsc/src/main/resources · high confidence

This change reflects Apache Livy's graduation from the Hue incubator to a standalone Apache project. The repository has been restructured with a new README.md replacing the old README.rst, updated build instructions targeting Spark 3.x/4.x and Scala 2.12/2.13, and the addition of ASF governance files (.asf.yaml, LICENSE, NOTICE). The Travis CI configuration has been updated to run tests against Spark 3.x with Scala 2.12, and various build artifacts and configuration files have been added or removed to support the new standalone build process.

(repo-wide) · high confidence

The core Scala sources have been migrated from the \com.cloudera.hue.livy\ namespace to \org.apache.livy\, including the addition of a new \EOLUtils\ helper for handling line-endings and updates to \Utils\ (such as adding Base64 support and daemon thread utilities). Additionally, the messaging protocol has been extended to support code completion by adding \CompletionRequest\ and \CompletionResponse\ message types, and the \ExecuteRequest\ message now accepts an optional \kind\ parameter to specify the language.

core/src/main/scala/org/apache/livy · high confidence

The Livy server now enforces role-based access control (view, modify, super) via a new AccessFilter and AccessManager, configurable through ACL settings. It also adds security-related HTTP headers (e.g., X-Content-Type-Options, X-Frame-Options) via a SecurityHeadersFilter, and supports API versioning through the Accept header to allow clients to target specific API versions. Additionally, the server now rejects session creation requests when the number of concurrent creations exceeds the configured limit, and session metrics are exposed as Codahale gauges for monitoring.

server/src/main/scala/org/apache/livy/server · high confidence

The bin/livy-server script has been completely rewritten to support start, stop, and status commands, replacing the previous direct-execution model. This change introduces configurable environment variables (such as LIVY\_PID\_DIR and LIVY\_LOG\_DIR), log rotation, and proper process management via PID files. Additionally, the script now includes specific JVM flags (--add-opens and --add-exports) required for Java 17 compatibility. Several legacy helper scripts (livy-repl, livy-yarn-client, livy-yarn-server, setup-classpath) have been removed as their functionality is either deprecated or integrated into the new server structure.

bin · high confidence

This change refactors the core utility classes in the Livy server, moving them from the legacy com.cloudera.hue.livy package to org.apache.livy.utils and updating the license headers to the Apache Software Foundation. It introduces a new Clock abstraction to mock time for testing, adds a LivyUncaughtException class, and updates LivySparkUtils to support Spark versions 3.0 through 4.2 (with Scala 2.12/2.13 mappings). The SparkApp, SparkYarnApp, and SparkKubernetesApp utilities are enhanced with improved state tracking, namespace resolution for Kubernetes, and better handling of leaked applications. Additionally, LineBufferedProcess and LineBufferedStream are modified to use a circular queue for log buffering with configurable size, and SparkProcessBuilder is simplified to remove user-configurable path restrictions and support verbose output.

server/src/main/scala/org/apache/livy/utils · high confidence

The session management core now uses sealed abstract classes for \Kind\ (session types like Spark, PySpark, SQL, etc.) and \SessionState\ (lifecycle states like Idle, Running, Killed, etc.) instead of previous implementations. This change introduces specific case classes for terminal states (Killed, Error, Dead, Success) that record the termination time, and integrates custom Jackson serializers/deserializers to ensure these enums and states are correctly serialized to and from JSON strings in API payloads.

core/src/main/scala/org/apache/livy/sessions · high confidence

The REPL module has been reorganized under the \org.apache.livy\ package, replacing the previous \com.cloudera.hue.livy\ namespace. This change includes a migration of the Python interpreter to Python 3, with the default executable now set to \python3\ (respecting \spark.pyspark.python\ and environment variables). The update also introduces a new SQL interpreter for executing queries and returning JSON results, adds autocompletion support for Scala, and implements fixes for handling \java.math.BigDecimal\ in JSON serialization and \java.sql.Date\ types.

repl/src/main/scala · high confidence

The session management layer in Livy has been restructured to support more granular session lifecycle controls. The core Session class now accepts TTL (time-to-live) and idleTimeout parameters, allowing sessions to expire based on either total age or inactivity. The SessionManager has been refactored into a generic base class with specific implementations for Batch and Interactive sessions, introducing a new mechanism to look up sessions by their assigned name in addition to their ID. This change also includes the introduction of a garbage collector thread to handle expired sessions and ensures that duplicate session names are rejected during registration to prevent conflicts.

server/src/main/scala/org/apache/livy/sessions · high confidence

The configuration directory has been reorganized to use a consistent template-based approach. New template files have been introduced for the client configuration (livy-client.conf.template), environment variables (livy-env.sh.template), server settings (livy.conf.template), logging (log4j2.properties.template), and Spark property blacklisting (spark-blacklist.conf.template). To support this structure, the previous livy-defaults.conf.template and spark-user-configurable-options.template files have been removed, consolidating configuration definitions into the new templates.

conf · high confidence

The PySpark interpreter resources in the REPL module have been updated to support Python 3 (including versions \>= 3.8) by replacing Python 2-specific constructs like \cStringIO\ and \unicode\ with their Python 3 equivalents (\io\ and \str\). Additionally, the \fake\_shell.py\ script now includes logic to handle shared variables across jobs via a new \JobContextImpl\ class and supports executing bypass jobs through \PySparkJobProcessorImpl\, enabling better integration with Livy's job API and shared state management.

repl/src/main/resources · high confidence

The server's static web resources now include the minified CSS for Bootstrap v3.4.1. This update refreshes the styling of the Livy web interface, applying the latest visual and layout improvements from the Bootstrap framework to the UI components served from this location.

server/src/main/resources · high confidence

A fix in core/src/main/resources — 1 commit (1 fix), 1 file.

core/src/main/resources · low confidence · unverified

Added Log4j2 test configuration files for client-http, rsc, and thriftserver modules; Added REPL test suite and migrated test package to Apache namespace; Added Scala API client and job handle tests; Added fake spark-submit script for test environments; Added integration test resources for PySpark, SparkR, and secure cluster validation; Added integration tests for batch, interactive, and job APIs; Added test for RSCDriver cancel job failure during shutdown; Added test resource configurations for logging and Spark settings; Added test suite for interactive session servlets and heartbeat logic; Added test-lib with reusable test jobs and apps; Added tests for LivyClientBuilder and test infrastructure; Added tests for Thrift Server session and column buffer operations; Added tests for ThriftServer LDAP authentication; Added tests for ThriftServer connection limits and data type handling; Added tests for the Python Livy client API; Added unit tests for EOL handling and a base test suite trait; Added unit tests for HTTP client and connection handling; Added unit tests for LDAP authentication handler; Added unit tests for Livy Spark utilities and app integrations; Added unit tests for Livy server components; Added unit tests for RSC driver address resolution and job handle state management; Added unit tests for batch session servlet and request handling; Added unit tests for session management and configuration validation; New integration test framework for external clusters and ThriftServer JDBC.

The project now includes a complete set of Maven POM files defining the module hierarchy for version 1.0.0-SNAPSHOT. This change introduces the \livy-api\ module, which exposes Spark dependencies (core, hive, sql, streaming) as provided, and the \livy-rsc\ module, which depends on the API. The \livy-client-http\ module is defined with an HTTP client implementation using Apache HttpClient and Jackson, configured to shade internal dependencies like Kryo and Jackson to avoid conflicts. The \livy-core\ module is split into Scala 2.12 and 2.13 variants, both depending on the parent core module which includes Log4j2 and Jackson. The \livy-repl\ module is also split by Scala version, depending on the RSC and core modules. Additionally, the \livy-assembly\ module is introduced to package the server components, and the \livy-integration-test\ module is defined with dependencies on Hadoop, Spark, and Jackson for testing. The \docs\ directory now includes a Gemfile for Jekyll-based documentation generation.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 57 → 46 (-10.5)
• Rubric changed (rubric-2026.09.8 → rubric-2026.09.16) — scores are not directly comparable.

• Code Health 67 → 68 (+1.4)
• Architecture 98 → 98 (-0.4)
• Maturity 53 → 54 (+0.8)
• Readiness 59 → 58 (-0.5)
• Security 63 → 68 (+4.6)
• Accessibility 55 → 30 (-24.5)

• Change coupling: all-sessions.js ↔ LivyServer.scala (server/src/main/resources/org/apache/livy/server/ui/static/js/all-sessions.js)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Off-boarding risk: anonymized user #1

• Duplicated block (6 lines × 2) (rsc/src/main/java/org/apache/livy/rsc/driver/SparkEntries.java)
• Duplicated block (7–8 lines × 2) (thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/GetColumnsJob.java)
• FileTooLong: js/bootstrap.js (docs/assets/themes/apache/bootstrap/js/bootstrap.js)
• Hotspot: server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala (server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala)
• No ADRs found
• Off-boarding risk: anonymized user #1
• SparkKubernetesApp.monitorSparkKubernetesApp (cognitive 17) (server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala)
• bootstrap.Affix.checkPosition (cognitive 16) (docs/assets/themes/apache/bootstrap/js/bootstrap.js)
• bootstrap.Affix.checkPosition (cyclomatic 16) (docs/assets/themes/apache/bootstrap/js/bootstrap.js)
• bootstrap.Carousel.slide (cognitive 18) (docs/assets/themes/apache/bootstrap/js/bootstrap.js)
• bootstrap.Modal.backdrop (cognitive 20) (docs/assets/themes/apache/bootstrap/js/bootstrap.js)
• bootstrap.Tooltip.show (cognitive 29) (docs/assets/themes/apache/bootstrap/js/bootstrap.js)
• bootstrap.Tooltip.show (cyclomatic 22) (docs/assets/themes/apache/bootstrap/js/bootstrap.js)

• 10 commits — 9 feature/other, 1 fixes

• server/src — 6 commits
• repl/src — 3 commits
• conf/livy.conf.template — 1 commit

• fix: [LIVY-1072][FOLLOWUP] Fix Scala 2.13 compile error in SparkKubernetesApp
• change: [LIVY-1041] Add Spark 4 support and make it the default build
• change: [LIVY-1065][TEST] Pin Spark driver to loopback in remaining macOS tests
• change: [LIVY-1070] Gate session metrics behind livy.server.session-metrics.enabled
• change: [LIVY-1073] Add SessionManager refresh and /recovery operator endpoint
• change: [LIVY-1079] Support multi-tenant (multi-namespace) Kubernetes clusters
• change: [LIVY-1080] Add missing livy.server.launch.kerberos.* properties to default livy.conf
• change: [LIVY-1082] Add config to disable Kubernetes driver pod log polling
• change: [LIVY-1084] Prefer spark container exit status over pod phase when sidecars are enabled
• change: [LIVY-1085][TEST] Skip remaining R test paths when -DskipRTests is specified

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

apache/livy was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.