Skip to content
CAI
Software that uses CAICheck a score

apache/hadoop

43.2

Weak · 13 November 2026

1496k

lines of production code

Java

primary language

5

measurements over time

CAI band scale
CAI trend line
CAI lens gauges

What this system is

Features

ABFS adds Azure Workload Identity OAuth2 support and configuration validation

The ABFS driver now supports authenticating to Azure Storage using Azure AD Workload Identity via the new WorkloadIdentityTokenProvider, which reads JWT assertions from a file or a custom provider to obtain OAuth2 tokens. To support this and improve reliability, the filesystem initialization now excludes incompatible credential providers, and new diagnostic validators (IntegerConfigurationBasicValidator, LongConfigurationBasicValidator) enforce bounds on configuration values.

hadoop-azure · high confidence

Add Avro serialization support for Hadoop IO

This change introduces a new Avro serialization package to Hadoop Common, providing two concrete serializers: AvroSpecificSerialization for classes generated by Avro's specific compiler and AvroReflectSerialization for general Java classes (which can be accepted via the avro.reflect.pkgs configuration or by implementing the new AvroReflectSerializable interface). The implementation includes the base AvroSerialization class and associated serializer/deserializer components, replacing the previous package documentation which described a retry mechanism.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/avro · high confidence

Add Baidu BOS (BOS) FileSystem implementation

Introduces a new Hadoop FileSystem implementation for Baidu Object Storage (BOS), enabling users to access BOS buckets via the 'bos' URI scheme. The change adds core components including \BaiduBosFileSystem\ for standard file operations, \BOS\ and \BaiduBosFileSystemAdapter\ for AbstractFileSystem integration, and \BosClientProxy\ for SDK interaction. It also includes configuration constants in \BaiduBosConstants\ (e.g., endpoints, credentials, block sizes) and build-time suppressions for FindBugs and Checkstyle to accommodate the new code's design choices.

hadoop-cloud-storage-project/hadoop-bos · high confidence

Add CSI driver adaptor module to NodeManager

The NodeManager now supports a new CSI (Container Storage Interface) adaptor module, allowing it to manage CSI drivers as an auxiliary service. This change introduces a Java-based client library and a set of protocol translators that bridge YARN's internal volume management requests (such as publish, unpublish, and validate capabilities) with the standard CSI gRPC protocol. Administrators can now configure specific CSI drivers in the NodeManager settings, and the system will automatically launch and manage the corresponding adaptor services to handle volume operations on the node.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi · high confidence

Add Cosmos DB as a DocumentStore backend for ATSv2

Users can now use Azure Cosmos DB as a storage backend for the Application Timeline Service v2. This change introduces the \CosmosDBDocumentStoreReader\ implementation, which handles reading timeline documents from Cosmos DB collections using the Azure Cosmos DB SDK. It supports querying documents by collection, fetching entity types, and applying predicates such as cluster ID filtering, enabling users to persist and retrieve timeline data in Cosmos DB instead of the default HBase backend.

hadoop-yarn-server-timelineservice-documentstore · high confidence

Add DocumentStore backend for ATSv2 timeline service

Introduces a new DocumentStore implementation for the YARN Timeline Service v2 (ATSv2), enabling storage of application timeline information in a document-based backend (defaulting to Azure Cosmos DB). This change adds the core components for this backend: \DocumentStoreCollectionCreator\ to initialize the database and collections (Application, Entity, FlowActivity, FlowRun); \DocumentStoreTimelineReaderImpl\ and \DocumentStoreTimelineWriterImpl\ to handle reading and writing timeline entities; and supporting document models (\TimelineEntityDocument\, \FlowActivityDocument\, etc.) and utilities (\DocumentStoreUtils\) for configuration and data transformation.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore · high confidence

Add Dynamometer user documentation and site styling

The Dynamometer tool now includes a comprehensive user guide (Dynamometer.md) detailing how to set up and run performance tests against the HDFS NameNode, including steps for preparing fsimage files, configuring Hadoop binaries, and replaying audit logs. Additionally, custom CSS styling (site.css) has been added to ensure the documentation renders correctly on the Hadoop site.

hadoop-tools/hadoop-dynamometer · high confidence

Add HBase 2.0-compatible Timeline Service v2 storage components

This change introduces the server-side implementation for the Timeline Service v2 (ATSv2) backend when using HBase 2.0. It adds the \FlowRunCoprocessor\ and \FlowScanner\ classes, which handle HBase region-level operations for flow run tables, including tag-based cell modification during writes and metric aggregation during scans. Supporting utilities in \HBaseTimelineServerUtils\ and a Maven assembly descriptor for the coprocessor JAR are also included to enable this optional compilation profile.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 · high confidence

Add HBase storage schema definitions for the Application and AppToFlow tables

The timelineservice-hbase-common module now includes the core schema definitions for the new Application and AppToFlow tables. This adds the \ApplicationTable\ and \AppToFlowTable\ classes along with their associated column families (info, configs, metrics for Application; mapping for AppToFlow), column enumerations, row key structures, and key converters. These components define how YARN application data and application-to-flow mappings are structured and stored in HBase, supporting the refactored timeline service storage architecture.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common · high confidence

Add Hadoop Auth examples application

The hadoop-auth-examples module now includes a complete sample web application demonstrating authentication integration. This includes a RequestLoggerFilter for debugging HTTP headers, a WhoServlet to display user identity, and a WhoClient for testing authenticated connections. The application is configured via web.xml to support anonymous, simple, and Kerberos authentication modes, with corresponding HTML landing pages for each.

hadoop-common-project/hadoop-auth-examples · high confidence

Add Hadoop version info and application classloader configuration

This change introduces two new resource files to the Hadoop Common module. The \common-version-info.properties\ file is added to expose build-time metadata (such as version, SCM commit, branch, and build time) for runtime inspection. Additionally, \org.apache.hadoop.application-classloader.properties\ is added to define the default classloader isolation rules for applications, specifying which system and Hadoop classes should be loaded by the parent or child classloader.

hadoop-common-project/hadoop-common/src/main/resources · high confidence

Add HadoopIllegalArgumentException and configuration documentation stylesheet

This change introduces a new HadoopIllegalArgumentException class to provide a distinct exception type for invalid arguments within Hadoop, differentiating it from the standard JDK IllegalArgumentException. Additionally, a new XSL stylesheet (configuration.xsl) is added to transform Hadoop configuration XML files into an HTML table format, displaying property names, values, and descriptions for documentation purposes.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop, hadoop-common-project/hadoop-common/src/main/xsl · high confidence

Add HuaweiCloud OBS as a Hadoop File System backend

Introduces a new \obs://\ URI scheme and \OBSFileSystem\ implementation, allowing Hadoop to read from and write to HuaweiCloud Object Storage. This includes the core file system logic, block-based output streams for efficient uploads, credential handling, and configuration constants for connection and proxy settings.

hadoop-cloud-storage-project/hadoop-huaweicloud/src/main · high confidence

Add JSON parser wrapper for Rumen trace data

The Rumen tool now includes a new JsonObjectMapperParser class that uses Jackson's ObjectMapper to parse JSON-encoded Hadoop trace data. This change enables the tool to read and process JSON-formatted job and task traces, expanding the types of input data it can analyze beyond its previous formats.

hadoop-rumen · high confidence

Add LZ4 compression and decompression support

This change introduces new Java classes (Lz4Compressor and Lz4Decompressor) in the org.apache.hadoop.io.compress.lz4 package, enabling users to compress and decompress data using the LZ4 algorithm. The implementation relies on the lz4-java library (net.jpountz.lz4) and provides both fast and high-compression modes for the compressor, along with a safe decompressor, allowing for faster data processing in Hadoop workloads that support this compression codec.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 · high confidence

Add NfsTime utility class for NFS time serialization

Added a new NfsTime class in the hadoop-nfs package to encapsulate time values (seconds and nanoseconds) and handle their serialization and deserialization via ONC RPC XDR streams. This class provides constructors for creating time instances from seconds/nanoseconds or milliseconds, and includes methods to retrieve time in different units, as well as standard object methods like equals, hashCode, and toString.

hadoop-nfs · high confidence

Add Protocol Buffer implementations for security and tools protocols

Introduces Protocol Buffer (PB) client and server translators for the RefreshAuthorizationPolicyProtocol, RefreshUserMappingsProtocol, and GetUserMappingsProtocol. This change enables these security and tools protocols to communicate via the Protocol Buffer RPC mechanism, providing a standardized serialization format for operations such as refreshing authorization policies, updating user-to-group mappings, and retrieving user group information.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/protocolPB, hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/protocolPB · high confidence

Add Range utility class for index bounds in Timeline Service HBase common

The timelineservice-hbase-common module now includes a new Range class that encapsulates a start and end index. This utility validates that the start index is non-negative and less than or equal to the end index, providing methods to retrieve the start position, end position, and the length of the range. This class is marked as private and unstable, indicating it is an internal implementation detail for the HBase-based timeline storage component.

hadoop-yarn-server-timelineservice-hbase-common · high confidence

Add SFTP FileSystem implementation

Users can now access SFTP servers as a Hadoop-compatible file system. This change introduces the SFTPFileSystem class, along with supporting components like SFTPConnectionPool for managing concurrent connections and SFTPInputStream for data transfer, enabling standard Hadoop file operations over SFTP.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp · high confidence

Add SQL scripts for YARN Federation StateStore on MySQL and SQL Server

This change introduces the initial database schema and stored procedures for the YARN Federation StateStore, supporting both MySQL and SQL Server backends. The new files in \hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/\ include scripts to create the database, tables (such as \membership\, \applicationsHomeSubCluster\, \reservationsHomeSubCluster\, and \policies\), and stored procedures for managing sub-cluster registration, heartbeats, application home sub-clusters, and policy configurations. It also provides corresponding scripts for user creation and database teardown, enabling administrators to set up and maintain the SQL-based state store required for YARN Federation.

hadoop-yarn-project · high confidence

Add TFile I/O utility classes and TFileDumper

This change introduces a new set of classes in the \org.apache.hadoop.io.file.tfile\ package to support TFile operations. It adds \BoundedRangeFileInputStream\ for reading contiguous regions of an \FSDataInputStream\, \Chunk\ with \ChunkDecoder\ for handling chunk-encoded sub-streams, and \SimpleBufferedOutputStream\ for buffered output with borrowed buffers. It also includes \Compression\ for managing LZO and GZ compression algorithms, \CompareUtils\ for byte-array comparison logic, and the \RawComparable\ interface for raw byte comparison. Additionally, \TFileDumper\ is added to dump TFile metadata and statistics, and new exceptions \MetaBlockAlreadyExists\ and \MetaBlockDoesNotExist\ are introduced for meta-block handling errors.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file · high confidence

Add Tencent Cloud COS File System implementation

Introduces the \CosNOutputStream\ class within the \hadoop-cos\ module, enabling users to write data to Tencent Cloud COS. This implementation supports streaming uploads via multipart uploads (up to 40TB per file), utilizes a fixed thread pool for performance, and handles block buffering and MD5 checksums.

hadoop-cos · high confidence

Add VolcanoEngine TOS File System implementation

This change introduces the initial source files for the VolcanoEngine TOS (Tencent Object Storage) File System implementation within the hadoop-tos module. The diff adds core data structures and utilities required for the new file system, including \RawFileStatus\ for file metadata with checksums, \BucketInfo\ to distinguish between general-purpose and directory buckets, \ObjectInfo\ and \TosObjectInfo\ for representing object storage entities, \MultipartUpload\ for handling large file uploads, and \Tasks\ for managing concurrent operations with retry and failure handling logic.

hadoop-cloud-storage-project/hadoop-tos, hadoop-tos · high confidence

Add ZStandard compression support via zstd-jni

This change introduces the ZStandard codec implementation for Hadoop, adding the \ZStandardCompressor\ and \ZStandardDecompressor\ classes along with package documentation. The implementation replaces native C bindings with the \zstd-jni\ library, enabling streaming compression and decompression with configurable levels and multi-threaded worker support.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd · high confidence

Add block generator tool for Dynamometer scale testing

The hadoop-dynamometer-blockgen module introduces a new MapReduce-based tool that parses HDFS fsImage XML files to generate block lists for Dynamometer DataNodes. It includes a driver, an XML parser that ignores non-INodeSection entries, and a reducer that outputs per-DataNode block files, along with a helper shell script to locate the Hadoop command and set the classpath. Tests verify the parser's state machine and the end-to-end block generation process.

hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen · high confidence

Add default text-format audit logger for KMS

The Key Management Service (KMS) now includes a default SimpleKMSAuditLogger that writes audit events in a specific text format. This logger handles both standard operations and aggregate operations (logging access counts and intervals for successful requests), ensuring that audit logs remain backwards-compatible for existing parsing tools. This change provides the baseline implementation for the pluggable audit logger architecture introduced in this release.

hadoop-kms · high confidence

Add device isolation module to container-executor

The container-executor now includes a new native device isolation module that allows administrators to restrict which host devices (such as GPUs or block devices) are accessible to YARN containers. This is achieved by configuring a list of denied device numbers in the container-executor configuration file and/or passing excluded device lists via command-line arguments; the executor then updates the container's cgroups device rules to enforce these restrictions, ensuring containers cannot access unauthorized hardware resources.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/devices · high confidence

Add documentation and site assets for HuaweiCloud OBS support

This change adds the user-facing documentation and site styling for the new HuaweiCloud Object Storage (OBS) integration. The \index.md\ file provides a comprehensive guide on configuring the \obs\ filesystem scheme, setting up authentication via access keys or custom providers, and using the \hadoop-huaweicloud\ JAR in Maven builds. It also lists supported features (such as multipart uploads) and known limitations (such as lack of support for symbolic links and POSIX ACLs). Additionally, a custom \site.css\ is included to style the project documentation page.

hadoop-cloud-storage-project/hadoop-huaweicloud/src/site, hadoop-huaweicloud · high confidence

Add new MapReduce example programs and update ExampleDriver

The hadoop-mapreduce-examples module now includes several new example programs: AggregateWordCount and AggregateWordHistogram for word aggregation and histogramming, BaileyBorweinPlouffe for computing digits of Pi, and DBCountPageView for database pageview counting. The ExampleDriver has been updated to register these new examples alongside existing ones like WordCount, Grep, and TeraGen, making them available for execution via the standard examples jar.

hadoop-mapreduce-project/hadoop-mapreduce-examples · high confidence

Add static CSS styles for Hadoop web UI

A new static stylesheet (hadoop.css) has been added to the webapps directory, defining visual styles for the Hadoop web interface. This includes layout rules for data node and file system tables, specific styling for warning messages, and the implementation of a fixed-position quick links menu in the top-right corner of the page.

hadoop-common-project/hadoop-common/src/main/webapps · high confidence

Add support for Baidu BOS non-hierarchical buckets

Users can now store data in Baidu BOS buckets that do not support native directory hierarchies. This change introduces the NonHierarchyBosNativeFileSystemStore, which simulates directory structures using key prefixes and delimiters, allowing standard Hadoop file system operations to work correctly on flat BOS buckets.

hadoop-bos · high confidence

Add the \`hadoop fs -find\` command with expression-based filtering

Introduces a new \hadoop fs -find\ command that allows users to recursively search the filesystem and filter results using a tree of expressions. This location provides the core command implementation (\Find\), the expression framework (\Expression\, \BaseExpression\, \ExpressionFactory\), and specific filtering capabilities such as matching by name (\-name\, \-iname\), modification/access time (\-mtime\, \-atime\), size, ownership, and logical operators (\-a\).

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/find · high confidence

Added Docker and Docker Compose configuration for local Hadoop development

The hadoop-dist module now includes a Dockerfile and a docker-compose setup (located in src/main/docker and src/main/compose/hadoop) to facilitate running a local Hadoop cluster. The Dockerfile builds an image based on the hadoop-runner base image, while the docker-compose configuration defines services for the NameNode, DataNode, ResourceManager, and NodeManager, along with pre-configured Hadoop XML settings (core-site, hdfs-site, mapred-site, yarn-site, capacity-scheduler) and logging properties to enable immediate local testing.

hadoop-dist · high confidence

Added FindBugs exclusions and test infrastructure for hadoop-aliyun, hadoop-aws, and hadoop-azure

This change introduces FindBugs exclusion files for the hadoop-aliyun, hadoop-aws, and hadoop-azure modules to suppress specific static analysis warnings (such as synchronization and return-value checks) that are intentional in these connectors. It also adds a .gitignore for hadoop-azure and a suite of shell scripts in hadoop-azure for running combination tests against Azure Blob and DFS endpoints. Additionally, a new hadoop-compat-bench tool is added with shell test cases covering standard Hadoop FileSystem operations like copy, move, read, write, and snapshot management to verify cross-implementation compatibility.

hadoop-tools · high confidence

Added HdfsCompatShellScope for shell-based compatibility benchmarking

The Hadoop Compatibility Benchmark now includes a new HdfsCompatShellScope component that enables running shell-script-based test cases against HDFS implementations. This class manages the execution environment by copying test scripts and Hadoop configuration, preparing snapshot paths for testing snapshot-related features, and executing the shell scripts to generate compatibility reports.

hadoop-compat-bench · high confidence

Added JDiff API documentation for Hadoop Common 2.10.0, 2.10.2, 2.6.0, 2.7.2, and 2.8.0

New JDiff XML files have been added to the dev-support/jdiff directory for Hadoop Common versions 2.6.0, 2.7.2, 2.8.0, 2.10.0, and 2.10.2. These files contain the generated API difference reports for their respective releases, enabling users to track public API changes across these specific Hadoop Common versions.

hadoop-common-project/hadoop-common/dev-support/jdiff · high confidence

Added Java-based CSI client library for YARN

This change introduces a new Java-based CSI (Container Storage Interface) client library within the hadoop-yarn-csi module. It provides the CsiClientImpl and CsiGrpcClient classes, which enable YARN to communicate with CSI drivers over Unix domain sockets using gRPC. This client implementation supports synchronous calls for identity, controller, and node operations, allowing YARN to interact with storage plugins for tasks such as validating volume capabilities and publishing/unpublishing volumes.

hadoop-yarn-csi · high confidence

Added Maven Wrapper with Maven 3.9.15 and JVM memory settings

Developers can now use the Maven Wrapper to build the project without manually installing Maven, ensuring a consistent build environment. The wrapper is configured to download and use Maven version 3.9.15 from the Apache repository, with a SHA-256 checksum for verification. Additionally, a JVM configuration file has been added to set the initial heap size to 256MB and the maximum heap size to 3072MB for Maven processes.

.mvn · high confidence

Added MySQL TokenStore scripts for HDFS Federation Router

Added SQL scripts to initialize a MySQL database for the Router Federation TokenStore, enabling the use of SQLDelegationTokenSecretManagerImpl for delegation token management in federated HDFS deployments.

hadoop-hdfs-project · high confidence

Added Tencent COS integration documentation and site assets

The Hadoop-COS module now includes official documentation and site resources for integrating Tencent Cloud Object Storage (COS) with Hadoop. The new \index.md\ file details the features, system requirements, and configuration properties (such as \fs.defaultFS\, credentials, and implementation classes) needed to use COS as a storage backend for Hadoop MapReduce, Spark, and other systems. Additionally, a \site.css\ file has been added to style the documentation site.

hadoop-cloud-storage-project/hadoop-cos/src/site · high confidence

Added bash tab completion for the Hadoop script

Users can now enable tab completion for the main Hadoop command-line script. The new \hadoop.sh\ script in the \bash-tab-completion\ contrib directory provides completion for top-level commands (such as \fs\, \dfsadmin\, \job\, \namenode\, and \pipe\), subcommand options, and file paths for DFS and local operations. On Debian-like systems, the script can be placed in \/etc/bash\_completion.d/\ for automatic loading, or sourced manually on other distributions.

hadoop-common-project/hadoop-common/src/contrib · high confidence

Added example Byteman script for Hadoop RPC debugging

A new example Byteman script (hadooprpc.btm) has been added to the dev-support/byteman directory to help debug Hadoop RPC traffic. This script instruments the Hadoop RPC layer to print request and response messages to standard output, which can be activated by setting the BYTEMAN\_SCRIPT\_URL environment variable when running the hadoop-runner Docker image.

dev-support/byteman · high confidence

Added example shell profile demonstrating pluggable profile structure

An example shell profile file (example.sh) has been added to the shellprofile.d configuration directory. This file serves as a template for users to understand the structure and API of the pluggable shell profile system, illustrating how to register a profile and implement optional capabilities such as init, classpath, nativelib, and finalize hooks.

hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d · high confidence

Added vectored read benchmark

A new JMH-based benchmark has been added to the Hadoop tools to measure the performance of vectored read operations. The \VectoredReadBenchmark\ class allows users to evaluate read throughput using both synchronous and asynchronous I/O paths, with configurable options for file system type (local or raw) and buffer allocation (direct or array). This benchmark is included in the distribution via a new assembly descriptor and has associated FindBugs exclusions for generated code.

hadoop-tools/hadoop-benchmark · high confidence

AliyunOSS: New fast-upload buffer strategies and file-listing acceptors

The AliyunOSS filesystem implementation now supports configurable buffer strategies for uploads via the new \OSSDataBlocks\ component, allowing users to select between array, disk, direct byte buffer, or hybrid memory/disk buffers to optimize write performance. Additionally, the new \FileStatusAcceptor\ interface and its implementations (\AcceptFilesOnly\, \AcceptAllButSelf\) provide precise control over how directory listings and file statuses are filtered during \listFiles\ and \listLocatedStatus\ operations, while \OutputStreamStatistics\ tracks block and byte allocation metrics for these upload streams.

hadoop-aliyun · high confidence

Application History Server merges timeline storage with generic history service

The Application History Server now uses the Timeline Service as its primary storage backend, replacing the previous FileSystem-based store. This change introduces a new \ApplicationHistoryManagerOnTimelineStore\ implementation that reads application, attempt, and container history from the Timeline Store, enabling unified access to cluster history via the generic history service and REST APIs.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice · high confidence

Automatic SSL keystore and truststore reloading

Hadoop services can now automatically detect and reload SSL keystores and truststores when the underlying certificate files change on disk, eliminating the need for service restarts during certificate rotation. This capability is implemented via the new FileBasedKeyStoresFactory, which schedules file monitoring tasks to refresh ReloadingX509KeystoreManager and ReloadingX509TrustManager instances, and is controlled by the ssl.{0}.stores.reload.interval configuration property.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl · high confidence

Azure Data Lake Store (ADLS) connector initial implementation

The hadoop-azure-datalake module introduces a new FileSystem implementation for Azure Data Lake Store, exposing the \adl://\ scheme. It supports multiple OAuth2 authentication methods including Refresh Token, Client Credentials, Managed Service Identity (MSI), and Device Code flow, with Client Credential set as the default provider type. The connector allows per-account configuration, respects ACL bits in file permissions, and provides configurable HTTP timeouts and SSL channel modes.

hadoop-tools/hadoop-azure-datalake · high confidence

Bloom filter classes added to hadoop-common

The \org.apache.hadoop.util.bloom\ package is introduced, providing a suite of probabilistic data structures for set membership testing. This includes the standard \BloomFilter\, a \CountingBloomFilter\ that supports dynamic deletions via 4-bit counting buckets, a \DynamicBloomFilter\ that expands its internal matrix of filters as data grows, and a \RetouchedBloomFilter\ that allows selective removal of false positives. The package also includes supporting classes such as \Filter\ (the abstract base), \HashFunction\, \Key\, and the \RemoveScheme\ interface defining strategies for false positive removal.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom · high confidence

DataJoin utility library added to Hadoop

The hadoop-datajoin module now provides a set of classes to perform map/reduce joins across multiple data sources. This includes the core DataJoinJob driver, abstract base classes DataJoinMapperBase and DataJoinReducerBase for defining custom join logic, and supporting utilities like TaggedMapOutput and ArrayListBackedIterator. A sample implementation (SampleDataJoinMapper/Reducer) and corresponding unit tests (TestDataJoin) are included to demonstrate an inner join on tab-separated text files.

hadoop-tools/hadoop-datajoin · high confidence

DistCp V2 driver class implementation

The DistCp tool now uses a new V2 driver class (DistCp.java) that supports parallel block copying, snapshot diff-based copy lists, and preservation of HDFS ACLs and extended attributes (XAttrs). The implementation includes a new execute() method for programmatic use, improved error handling for invalid arguments and unsupported features, and proper job cleanup after completion.

hadoop-distcp · high confidence

DistCp introduces file filtering and snapshot-based diff copying

DistCp now supports excluding files from copy operations via a configurable filter mechanism (CopyFilter), allowing users to define custom exclusion logic or use regex-based filters from a file. Additionally, DistCp can now utilize HDFS snapshot diff reports to identify differences between source and target, enabling efficient incremental copies and restores to earlier snapshots without scanning the entire file tree.

hadoop-tools/hadoop-distcp · high confidence

DistributedShell application rewritten for YARN and modern Java compatibility

The DistributedShell application has been completely rewritten to support modern YARN features and improve stability. The new Client and ApplicationMaster implementations allow users to specify custom resource types (e.g., GPUs, memory-mb) via the \--container\_resources\ flag, define complex node placement constraints using \--placement\_spec\, and configure log aggregation via \--log\_aggregation\_context\. The ApplicationMaster now supports opportunistic containers with automatic promotion, retains containers across application restarts, and uses asynchronous callbacks for better responsiveness. The client now supports allocation tags, resource profiles, and improved error handling for container launch failures. The codebase has been migrated from JUnit 4 to JUnit 5 and from Log4j to SLF4J, ensuring compatibility with JDK 17+ and modern logging standards.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell · high confidence

FsShell now supports Access Control Lists (ACLs) and file concatenation

Users can now manage file system permissions via the new \getfacl\ and \setfacl\ shell commands, which display and modify Access Control Lists respectively, including support for recursive operations and effective permission display. Additionally, the \concat\ command has been added, allowing users to merge multiple existing source files into a single target file within the same directory.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell · high confidence

Google Test framework added to native build

The Google C++ Testing Framework (Google Test) source code, including the fused gtest-all.cc implementation and public headers, has been added to the hadoop-common native project. This change enables native C++ components within Hadoop to utilize Google Test for unit testing, consolidating the testing infrastructure previously located in the mapreduce module into the common native library.

hadoop-common-project/hadoop-common/src/main/native/gtest · high confidence

Gridmix3 benchmark source code restored to hadoop-gridmix

The Gridmix3 benchmark implementation has been re-added to the hadoop-gridmix module. This restores the core simulation engine, including the main driver (Gridmix), data generation utilities (GenerateData, GenerateDistCacheData), compression and distributed-cache emulation, and execution summarization. Users can now run the Gridmix3 workload simulation again using the standard hadoop-gridmix tool.

hadoop-tools/hadoop-gridmix · high confidence

Hadoop Streaming adds deprecated record serialization and auto-detecting input utilities

Hadoop Streaming now includes a new \org.apache.hadoop.record\ package providing binary and CSV serialization for Hadoop Records (marked as deprecated in favor of Avro), alongside new streaming utilities: \AutoInputFormat\ for automatically detecting text or sequence file inputs, \DumpTypedBytes\ and \LoadTypedBytes\ for converting between files and typed bytes, and an \Environment\ class to capture host environment variables for streaming jobs.

hadoop-tools/hadoop-streaming · high confidence

HttpFS server and client implementation added

The HttpFS module now includes the core server and client components required to access HDFS over HTTP/HTTPS. This adds the HttpFSServer (a JAX-RS based HTTP server), the HttpFSFileSystem and HttpsFSFileSystem client implementations, and supporting configuration files (httpfs-env.sh, httpfs-site.xml, httpfs-log4j.properties). The server includes authentication filters, content-type validation for uploads, and audit logging, while the clients expose the webhdfs and swebhdfs schemes for file system operations.

hadoop-hdfs-project/hadoop-hdfs-httpfs · high confidence

HttpFS server implementation and operation handling

The HttpFS module now includes the core server infrastructure and operation logic. The HttpFSServerWebServer initializes the HTTP server using HttpServer2, manages configuration properties (such as port, SSL, and authentication filters), and handles deprecated environment variables. The FSOperations class implements the backend logic for file system operations, including serializing FileStatus objects to JSON with support for HDFS-specific fields (like storage policy, erasure coding, ACL bits, and snapshot status) and handling various file types and permissions.

hadoop-hdfs-httpfs · high confidence

In-memory storage for resource skyline history and estimations

The Resource Estimator tool now includes an in-memory implementation of the SkylineStore interface, allowing it to persist and retrieve historical resource usage data (ResourceSkyline) and predicted resource allocations for application pipelines. This component manages the lifecycle of skyline data by supporting the addition, updating, and deletion of history records for recurring pipeline executions, while also storing estimated resource requirements, enabling the estimator to base future predictions on prior execution data.

hadoop-resourceestimator · high confidence

Initial Docker image for the YARN Application Catalog service

This change introduces the Docker container image for the YARN Application Catalog, providing the runtime environment for the catalog's Solr-based storage. The image includes sample application definitions (such as Httpd, Jenkins, and Docker Registry) pre-loaded into the catalog, and supports configuration for HDFS-backed storage and Kerberos/SPNEGO authentication via environment variables.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker · high confidence

Initial SharedCacheManager server implementation

This change introduces the core server components for the YARN Shared Cache Manager, providing a centralized service to manage shared application resources. It adds the main \SharedCacheManager\ entry point which orchestrates several sub-services: a pluggable metadata store (\SCMStore\), a background \CleanerService\ that periodically removes stale cache entries, and RPC services for NodeManagers (\SharedCacheUploaderService\), clients (\ClientProtocolService\), and administrators (\SCMAdminProtocolService\). The implementation includes an \AppChecker\ abstraction (with a \RemoteAppChecker\ implementation) to verify application liveness before cleanup, and exposes metrics for cleaner activity, client cache hits/misses, and uploader acceptance rates.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager · high confidence

Initial Windows native build support for libhadoop

This change introduces the Visual Studio project files (native.sln and native.vcxproj) and associated metadata (AUTHORS, COPYING, README) required to build the Hadoop native library on Windows. The build configuration compiles the native code into a dynamic library (hadoop.dll) and includes source files for core components such as NativeIO, CRC32, and Windows-specific security mappings. It also adds conditional compilation support for the Intel ISA-L library to enable hardware-accelerated erasure coding when the library is available.

hadoop-common-project/hadoop-common/src/main/native · high confidence

Initial implementation of YARN-based MapReduce ApplicationMaster components

This change introduces the core ApplicationMaster implementation for MapReduce on YARN, including the TaskAttemptImpl state machine, MapTaskAttemptImpl and ReduceTaskAttemptImpl wrappers, and the LegacyTaskRuntimeEstimator for speculative execution. It establishes the foundation for launching and managing map and reduce tasks within YARN containers, handling container allocation, task lifecycle events, and runtime estimation.

hadoop-mapreduce-client-app · high confidence

Initial implementation of the Aliyun OSS FileSystem connector

This change introduces the core implementation for accessing Alibaba Cloud OSS as a Hadoop FileSystem. It adds the main \AliyunOSSFileSystem\ class along with supporting components for authentication (\AliyunCredentialsProvider\), high-performance I/O (\AliyunOSSBlockOutputStream\ for uploads and \AliyunOSSInputStream\ with multi-part read-ahead for downloads), and parallel operations (copy tasks). The implementation also includes configuration constants, utility classes, and the underlying store interface to bridge Hadoop's filesystem API with the Aliyun OSS SDK.

hadoop-tools/hadoop-aliyun · high confidence

Initial implementation of the Hadoop NFS Gateway

This change introduces the Hadoop NFS Gateway, a Java library that enables HDFS to be mounted as a standard NFSv3 file system. The implementation includes core protocol components such as ONCRPC and XDR serialization, a Mount daemon (mountd) for handling mount requests and export lists, and an NFSv3 server (nfsd) supporting standard file operations like read, write, and directory listing. It also provides configuration for export access control based on hostnames or IP ranges, allowing clients to mount HDFS paths via standard NFS clients.

hadoop-common-project/hadoop-nfs · high confidence

Initial import of the MapReduce Native Task module

This change introduces the initial codebase for the MapReduce Native Task module, providing a new CMake-based build system (src/CMakeLists.txt) and a set of Java classes that define the native runtime interface. The Java code establishes the core architecture for offloading MapReduce operations to native code, including the NativeRuntime for JNI communication, Platform abstractions for key serialization and comparison, and handlers for data collection and command dispatching.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask · high confidence

Initial introduction of the MaWo application core module

This change adds the core implementation for the new MaWo (MapReduce on YARN) application within the YARN project. It introduces the foundational server components, including configuration management via \MawoConfiguration\ and \mawo-default.properties\, task modeling classes (\AbstractTask\, \SimpleTask\, \CompositeTask\, \NullTask\, \TeardownTask\), and identity structures (\TaskId\, \JobId\, \WorkerId\). It also defines the RPC protocol interface \WorkAssignmentProtocol\ for worker-master communication and provides the assembly descriptor for binary packaging. Additionally, it includes initial unit tests for the configuration module.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core · high confidence

Initial release of the Key Management Server (KMS)

This change introduces the Hadoop Key Management Server, a new component that implements the KeyProvider API to centralize key management for HDFS encryption zones. The KMS provides a REST API for creating, deleting, and rolling keys, as well as generating and decrypting encrypted encryption keys (EEKs). It features configurable access control lists (ACLs) for operations like CREATE, GET, and DELETE, supports hot-reloading of ACL configurations, and includes a pluggable audit logging framework. The server is built on Jetty and includes an eager key generation cache to improve performance for decryption operations.

hadoop-common-project/hadoop-kms · high confidence

Initial release of the YARN Capacity Scheduler UI

This change introduces the YARN Capacity Scheduler UI, a new client-side single-page application for managing Hadoop YARN Capacity Scheduler configurations. The UI provides visual tools for queue management, placement rules, capacity planning, and staged configuration changes with validation before applying them to a live cluster. It includes a React-based frontend with Zustand state management, a design document, development guides, ESLint and Prettier configurations, and mock API fixtures to support local development without a running YARN cluster.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui · high confidence

Initial scaffolding of the new YARN UI2 Ember.js application

The new YARN UI2 web interface is introduced, built on the Ember.js framework. This change adds the foundational application structure, including configuration files for Ember CLI, Bower, and JS linting, alongside the core data layer. The UI now utilizes Ember Data adapters to communicate with the ResourceManager, Application Timeline Server, and JobHistory Server, enabling the display of cluster metrics, application details, and container logs.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui · high confidence

Introduce AMRMProxy and Federation infrastructure in yarn-server-common

The yarn-server-common module now includes the core components for the AMRMProxy and Federation features. This adds the AMRMClientRelayer and AMHeartbeatRequestHandler to manage asynchronous heartbeats and proxy communication with the ResourceManager, alongside a ZKClient library for service registration. It also introduces the CollectorNodemanagerProtocol and DistributedSchedulingAMProtocol interfaces to support timeline aggregation and distributed scheduling, and adds the ContainerAllocationHistory class to track allocation latency metrics.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common · high confidence

Introduce AbstractFileSystem implementations for the local file system

The local file system now provides AbstractFileSystem implementations (LocalFs and RawLocalFs) that delegate to the existing RawLocalFileSystem, enabling consistent integration with the AbstractFileSystem API. This change introduces LocalConfigKeys to define server-side default settings (such as block size, replication, and checksum type) and ensures that the local file system returns these defaults via getServerDefaults, aligning its behavior with other Hadoop file systems.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local · high confidence

Introduce Application Catalog REST API backend

The Application Catalog webapp now exposes a new REST API (under the /service path) to manage YARN applications. This includes endpoints to browse and search the application store, view details of deployed services, and perform lifecycle operations such as deploying, deleting, restarting, stopping, and upgrading applications. The backend implementation uses Solr to index and query application metadata and templates, and communicates with the YARN Resource Manager via the ApiServiceClient to execute service actions.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp · high confidence

Introduce ApplicationClassLoader for application isolation

Added the ApplicationClassLoader utility, a URLClassLoader that enables application isolation by loading classes from application JARs in preference to the parent loader. It supports configurable system class patterns to prevent certain classes from being loaded by the application classloader, and handles classpath expansion including wildcard support for directories.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util · high confidence

Introduce Dynamometer infrastructure components and helper scripts

Adds the core infrastructure for the Hadoop Dynamometer scale-testing tool, including the YARN ApplicationMaster and Client for launching simulated DataNodes and NameNodes, utility classes for configuration and metrics parsing, and bash scripts to create slim Hadoop tarballs, upload fsimages, and start the cluster. This enables users to run HDFS NameNode scale tests using real metadata and workloads within a YARN-managed environment.

hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra · high confidence

Introduce FedBalance tool for cross-cluster data balancing

Adds the FedBalance tool to the hadoop-federation-balance module, enabling users to balance data between federation clusters using a distcp-based workflow. The tool implements a state-machine procedure chain (DistCpProcedure followed by TrashProcedure) that handles pre-checks, incremental copying with snapshot diffs, optional open-file forcing, and source cleanup (trash, delete, or skip). It exposes CLI options for map count, bandwidth limits, retry delays, and a diff threshold to control when the final sync round occurs, and persists job state via an HDFS-based journal for recovery.

hadoop-tools/hadoop-federation-balance · high confidence

Introduce Global Policy Generator (GPG) for YARN Federation

This change introduces the Global Policy Generator (GPG), a new YARN Federation component that automatically tunes and balances cluster policies. The GPG operates out-of-band to enforce global invariants, manage load balancing, and handle sub-cluster maintenance. Key capabilities include a load-based policy generator that adjusts sub-cluster weights based on pending queue metrics, an application cleaner that removes stale application entries from the Federation State Store and Registry, and a sub-cluster cleaner. The implementation provides a pluggable policy framework, secure mode support with HTTPS, and a web application for monitoring and querying policies.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator · high confidence

Introduce HBase-backed Timeline Service v2 client components

This change adds the core client-side implementation for the HBase backend of the YARN Timeline Service v2 (ATSv2). It introduces the HBase-specific reader, writer, and schema creator classes, along with a storage monitor to check HBase availability and a utility class for constructing HBase filters. These components enable the timeline service to store and retrieve application, flow, and entity data in HBase, including support for configuring table names, metrics TTLs, and handling HBase connection states.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client · high confidence

Introduce HDFS Dynamometer workload generator for NameNode scale testing

Adds the Hadoop Dynamometer workload generator module, providing tools to simulate real-world NameNode load for scale testing. This includes a \WorkloadDriver\ to launch map-only jobs, a \CreateFileMapper\ to generate file-object pressure, and an \AuditReplayMapper\ to replay historical audit logs against the NameNode. The module also provides shell scripts (\start-workload.sh\, \parse-start-timestamp.sh\) to facilitate execution and a pluggable audit log parsing framework (\AuditCommandParser\) with implementations for direct log files and Hive table exports.

hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload · high confidence

Introduce HDFS encryption zone support with AES-CTR and SM4 codecs

This change adds the core Java components for HDFS encryption zones, enabling transparent encryption of data at rest. It introduces a new \CipherSuite\ enum supporting both \AES/CTR/NoPadding\ and \SM4/CTR/NoPadding\ algorithms, along with a \CryptoCodec\ framework that allows pluggable implementations. The diff provides two codec implementations: \JceAesCtrCryptoCodec\ and \JceSm4CtrCryptoCodec\ using the Java Cryptography Extension (JCE), and \OpensslAesCtrCryptoCodec\ using JNI to OpenSSL for high-performance AES-CTR encryption. Supporting classes include \CryptoInputStream\ and \CryptoOutputStream\ for buffer-based encryption/decryption, \CryptoProtocolVersion\ for client-server negotiation, and utility classes like \CipherOption\ and \CryptoStreamUtils\ to manage keys, IVs, and buffer sizes.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto · high confidence

Introduce HDFS-based NFSv3 and Mountd gateway implementation

This change adds the core implementation for the HDFS NFSv3 gateway and the associated mountd daemon within the hadoop-hdfs-nfs module. It introduces the main server entry point (Nfs3), the mount protocol handler (Mountd/RpcProgramMountd), and the NFSv3 RPC program (RpcProgramNfs3). The implementation includes configuration keys for ports, bind addresses, and export points, as well as supporting components like a DFS client cache, asynchronous write-back service, and metrics collection. This provides the foundational capability to mount HDFS file systems via the NFSv3 protocol.

hadoop-hdfs-project/hadoop-hdfs-nfs · high confidence

Introduce HTTP delegation token support in Hadoop Common

This change adds a new set of classes to the \org.apache.hadoop.security.token.delegation.web\ package to enable HTTP-based delegation token management. It introduces \DelegationTokenAuthenticationFilter\ to handle authentication and token lifecycle operations via servlet filters, \DelegationTokenAuthenticatedURL\ for client-side token acquisition and usage, and specific handlers like \KerberosDelegationTokenAuthenticationHandler\ and \PseudoDelegationTokenAuthenticationHandler\ to support Kerberos SPNEGO and simple authentication schemes respectively. The implementation includes \DelegationTokenManager\ for token creation and renewal, and \DelegationTokenAuthenticator\ to wrap existing authenticators with delegation token logic, allowing services to issue and validate delegation tokens over HTTP.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web · high confidence

Introduce Hadoop Credential Provider API and CLI

This change introduces a new, thread-safe credential provider API and a corresponding command-line interface (CredentialShell) for managing secrets. The API, located in the \org.apache.hadoop.security.alias\ package, provides an abstraction to separate credential storage from applications, supporting multiple backends including Java KeyStore (JCEKS) and Bouncy Castle FIPS KeyStore (BCFKS) via the \CredentialProviderFactory\. The new \CredentialShell\ utility allows users to create, list, check, and delete credentials using commands like \hadoop credential create\ and \hadoop credential list\, with support for specifying provider paths and handling password requirements securely.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias · high confidence

Introduce Hadoop-native MCP HTTP server framework

Added a new Hadoop-native Model Context Protocol (MCP) HTTP server framework in hadoop-common. This includes a Jetty-based HTTP/HTTPS server (McpHttpServer) that hosts a servlet (McpHttpServlet) for handling streamable JSON-RPC 2.0 requests. The framework provides core components for MCP protocol support, including JSON serialization via Jackson (JacksonMcpJsonMapper), request validation (McpJsonRpcValidator), and a transport-neutral request handler (McpRequestHandler) that manages tool registration, initialization, and execution. It also defines the necessary schema types (McpSchema) for tools, text content, and call results, along with context and response objects to facilitate integration with Hadoop services.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/mcp · high confidence

Introduce KMS client-side KeyProvider implementation with load balancing and delegation token support

Adds the KMS client-side KeyProvider implementation (KMSClientProvider) and supporting classes (KMSDelegationToken, KMSRESTConstants, LoadBalancingKMSClientProvider, ValueQueue) to the hadoop-common module. This enables Hadoop clients to interact with a Key Management Server (KMS) for encryption operations, including support for delegation tokens, automatic retry on network failures, and load balancing across multiple KMS instances for high availability.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms · high confidence

Introduce KeyProvider API and CLI for encryption key management

This change introduces the core KeyProvider API and the \hadoop key\ command-line interface (KeyShell) to Hadoop Common, enabling applications to manage encryption keys independently of the storage layer. It adds a new \KeyProvider\ interface along with extension points for caching (\CachingKeyProvider\), cryptographic operations (\KeyProviderCryptoExtension\), and delegation tokens (\KeyProviderDelegationTokenExtension\). A concrete \JavaKeyStoreProvider\ implementation is included to store keys in JKS files on any Hadoop FileSystem, supporting password configuration via environment variables or config files. The new CLI allows users to create, list, roll, and delete keys, providing a standardized way to handle key lifecycle operations for features like HDFS Encryption Zones.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key · high confidence

Introduce MapReduce framework uploader tool

Adds the \FrameworkUploader\ tool, which packages MapReduce framework JARs into a tarball and uploads it to HDFS. The implementation includes default whitelist and blacklist patterns for YARN server JARs, supports configurable replication and timeout settings, and handles symlinks and file permissions during the upload process. A corresponding unit test suite validates argument parsing and upload behavior.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-uploader · high confidence

Introduce NativeIO package with POSIX errno mapping and shared file descriptor support

This change introduces the \org.apache.hadoop.io.nativeio\ package, providing a structured layer for JNI-based native I/O operations. It adds an \Errno\ enum to map POSIX error codes and a \NativeIOException\ class that exposes these codes (or Windows error codes) for better diagnostic clarity. The package also includes \SharedFileDescriptorFactory\ to enable sharing file descriptors across processes via unlinked temporary files in specified directories (e.g., \/dev/shm\), and extends \NativeIO.POSIX\ with PMDK support state tracking and \fadvise\ capability flags. This infrastructure supports cross-platform native I/O features like short-circuit reads and persistent memory caching.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio · high confidence

Introduce Protobuf-based RPC for HA and ZKFC protocols

This change adds the client-side translators, server-side translators, and protocol interfaces for the HAServiceProtocol and ZKFCProtocol using Apache Protobuf. For HAServiceProtocol, the new implementation supports monitoring health, retrieving service status, and transitioning services between Active, Standby, and Observer states. For ZKFCProtocol, it enables graceful failover and ceding active status. These components replace or supplement previous RPC mechanisms, allowing HA services and the ZK Failover Controller to communicate via a standardized, versioned Protobuf interface.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB · high confidence

Introduce Protobuf-based RPC for the TaskUmbilicalProtocol

The MapReduce Application Master's communication with child JVMs (YarnChild) has been migrated from the legacy Writable-based RPC to a Protobuf-based implementation. This change introduces the \TaskUmbilicalProtocolPB\ interface and its client implementation, allowing tasks to report status, fetch tasks, and handle commit events via a more robust protocol. This migration is a prerequisite for supporting newer Java versions and improving serialization performance.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app · high confidence

Introduce Resource Estimator tool for pipeline resource prediction

Adds the Resource Estimator tool, a new capability that estimates resource requirements for application pipelines based on prior executions. This change introduces the \ResourceEstimatorServer\ daemon, launch scripts (\estimator.sh\, \start-estimator.sh\, \stop-estimator.sh\), and a REST API (\ResourceEstimatorService\) for interacting with the estimator. It includes configuration options for the linear programming solver (alpha, beta, minJobRuns), time intervals, and pluggable components for log parsing, skyline storage, and solving, allowing users to tune prediction granularity and accuracy.

hadoop-tools/hadoop-resourceestimator · high confidence

Introduce Tencent Cloud COS File System implementation

Adds a new Hadoop-compatible file system implementation for Tencent Cloud Object Storage (COS), enabling users to access COS buckets using the standard Hadoop FileSystem API via the 'cosn' scheme. This change introduces the core components required for this integration, including the \CosNFileSystem\ and \CosN\ abstract file system classes, a native storage bridge (\CosNativeFileSystemStore\) for direct COS API interaction, and optimized I/O utilities such as a thread-safe \BufferPool\, \ByteBufferInputStream\/\OutputStream\, and asynchronous read-ahead tasks (\CosNFileReadTask\) to improve performance for large file operations.

hadoop-cloud-storage-project/hadoop-cos/src/main · high confidence

Introduce UNIX Domain Socket support for short-circuit data access

Added the DomainSocket and DomainSocketWatcher classes to enable HDFS DataNodes to use UNIX domain sockets for short-circuit read operations. This new capability allows clients to bypass the network stack and access local data files directly via shared memory and domain sockets, improving performance for local reads. The implementation includes native code integration for socket binding, path security validation, and event watching, and is limited to non-Windows environments where native libraries are available.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix · high confidence

Introduce Unmanaged AM Launcher for external Application Master processes

The UnmanagedAMLauncher client is now available, allowing users to register an application with the YARN ResourceManager and launch an Application Master process externally (outside the ResourceManager's direct control). The launcher handles obtaining the necessary AMRMToken, passing the container ID via environment variables, and monitoring the external process and application state, enabling integration of custom or non-Java AMs with YARN.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher · high confidence

Introduce YARN Registry client API and CLI

This change adds the core client API and command-line interface for the Hadoop Registry, enabling users to manage service records and DNS entries via a unified tool. The new \RegistryCli\ tool supports commands such as \ls\, \resolve\, \bind\, \mknode\, and \rm\ to interact with the registry. The API exposes \RegistryOperations\ for standard CRUD operations on service records and \DNSOperations\ for registering and deleting DNS entries, with factory classes (\RegistryOperationsFactory\, \DNSOperationsFactory\) handling initialization and authentication (including Kerberos and digest auth). Utility classes like \JsonSerDeser\ and \RegistryTypeUtils\ provide JSON serialization and endpoint creation helpers, while \RegistryConstants\ defines configuration keys for DNS and ZooKeeper settings.

hadoop-common-project/hadoop-registry · high confidence

Introduce YARN Service Application Master lifecycle and upgrade management

The YARN Services core module now includes the Application Master (AM) implementation, enabling users to deploy, manage, and upgrade multi-container services on YARN. This change adds the \ServiceMaster\ entry point and the \ServiceScheduler\ which handles container lifecycle, placement, and recovery. It introduces a state machine in \ServiceManager\ to manage service states (STABLE, UPGRADING) and supports both standard and express upgrades, as well as component scaling (flexing) and decommissioning. Security is enforced via the new \ClientAMService\ and \ClientAMProtocol\ for client-to-AM communication, including Kerberos support. Additionally, a \ContainerFailureTracker\ is added to automatically blacklist nodes experiencing excessive container failures.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core · high confidence

Introduce YARN Services REST API server and client

Adds a new REST API layer for managing YARN services, including the \ApiServer\ and \ApiServerWebApp\ components that expose endpoints for creating, updating, and destroying services, as well as the \ApiServiceClient\ and \SystemServiceManagerImpl\ classes that handle client-side interactions and system service lifecycle management. The change also includes the Swagger 2.0 API definition (\YARN-Simplified-V1-API-Layer-For-Services.yaml\), example service definitions, and associated configuration files like \web.xml\ and \log4j-server.properties\ to support this new service management capability.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api · high confidence

Introduce core Erasure Coding infrastructure in Hadoop Common

This change adds the foundational classes for the Erasure Coding feature to the hadoop-common module. It introduces ECSchema to define coding parameters (data/parity units, codec), CodecRegistry to dynamically discover and register coder implementations via ServiceLoader, and CodecUtil to create encoders and decoders with support for native (ISA-L) and Java fallbacks. It also adds supporting types like ECBlock, ECChunk, and BlockGrouper to manage block groups and chunks during encoding and decoding operations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode · high confidence

Introduce hadoop-auth client authentication library

Adds the \hadoop-auth\ client library to enable Java applications to connect to HTTP endpoints protected by the Hadoop Authentication Filter. This new component provides the \AuthenticatedURL\ class to manage authentication sessions and tokens, along with built-in support for Kerberos SPNEGO and Hadoop Simple (pseudo) authentication, allowing clients to automatically handle cookie-based session management and fallback between authentication schemes.

hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client · high confidence

Introduce internal prefetching block management infrastructure

Added a new set of internal classes in the \org.apache.hadoop.fs.impl.prefetch\ package to support high-performance, block-based prefetching and caching for file system input streams. This includes \BlockData\ for tracking file block states, \BlockManager\ and \CachingBlockManager\ for abstracting read and caching operations, \BufferPool\ for managing reusable byte buffers, and \BlockCache\ interfaces for local disk caching. These components provide the foundational logic for asynchronous prefetching, local block caching, and resource management, which are utilized by S3A and other file system implementations to improve read performance.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch · high confidence

Introduce native DomainSocket and DomainSocketWatcher implementations

Added new native C source files (DomainSocket.c and DomainSocketWatcher.c) to the hadoop-common native library. These files implement the low-level UNIX domain socket communication and file descriptor watching mechanisms required for Hadoop's short-circuit read feature, replacing or supplementing previous Java-only or less robust implementations with optimized native code for socket setup, error handling, and I/O multiplexing.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net · high confidence

Introduce platform-aware package resolver for Hadoop build environments

The dev-support/docker/pkg-resolver directory now includes a new set of scripts and configuration files that enable platform-specific dependency resolution for building Apache Hadoop. This change introduces a Python-based resolver (resolve.py) and a platform checker (check\platform.py) that read from platforms.json and packages.json to determine the correct packages to install for supported platforms like ubuntu:noble, rockylinux:8, debian:12, and debian:13. Additionally, new shell scripts (install-\.sh) and PowerShell scripts (install-\*.ps1) are provided to install specific dependencies such as Boost (1.86.0), Maven (3.9.15), Protobuf (3.25.5), and others, ensuring consistent and correct package installation across different operating systems and architectures.

dev-support/docker/pkg-resolver · high confidence

Introduce pluggable DNS-to-switch mapping and domain name resolution infrastructure

The network package now provides a base class (AbstractDNSToSwitchMapping) and a caching implementation (CachedDNSToSwitchMapping) for DNS-to-switch (rack) mappings, allowing administrators to plug in custom topology resolvers. It also introduces a DomainNameResolver interface and factory (DomainNameResolverFactory) to abstract DNS lookups for service discovery, with a default implementation (DNSDomainNameResolver) that handles forward/reverse lookups and FQDN resolution.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net · high confidence

Introduce structured RPC retry and failover framework

The \org.apache.hadoop.io.retry\ package has been refactored to support robust client-side retry and failover for RPC calls. This change introduces a new \FailoverProxyProvider\ interface and \DefaultFailoverProxyProvider\ implementation to manage proxy lifecycle and failover events, replacing the previous direct invocation handler pattern. It adds \Idempotent\ and \AtMostOnce\ annotations to mark methods that can be safely retried, and a new \RetryInvocationHandler\ that coordinates these policies. Additionally, an \AsyncCallHandler\ and \CallReturn\ state machine are added to support asynchronous call retry logic, while \RetryPolicies\ and \RetryUtils\ provide configurable policies like \MultipleLinearRandomRetry\ and \WrapperRetryPolicy\ for handling specific exception types.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry · high confidence

Introduction of HDFS Access Control List (ACL) support

This change introduces the core data structures and parsing logic for HDFS Access Control Lists (ACLs) within the hadoop-common permission package. It adds new public, stable classes and enums—AclEntry, AclEntryScope, AclEntryType, AclStatus, and ScopedAclEntries—to represent and manage ACL entries, scopes (ACCESS vs. DEFAULT), and status. It also includes utility classes (AclUtil) for manipulating ACLs and parsers (ChmodParser, UmaskParser, PermissionParser, RawParser) to handle permission mode strings, enabling the system to support fine-grained access control beyond standard POSIX permissions.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission · high confidence

Introduction of Hadoop-specific interface classification and testing annotations

This change introduces three new annotations in the \org.apache.hadoop.classification\ package to help manage API stability and visibility. \InterfaceAudience\ (with sub-annotations \Public\, \LimitedPrivate\, and \Private\) and \InterfaceStability\ (with sub-annotations \Stable\, \Evolving\, and \Unstable\) allow developers to explicitly mark the intended audience and stability guarantees of classes and methods, with all sub-annotations retaining at runtime. Additionally, a new \VisibleForTesting\ annotation is provided as an alternative to Guava's version, marking elements specifically for use in test code within Hadoop modules and implicitly treating them as \Private\ and \Unstable\.

hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification · high confidence

Introduction of Manifest Committer for object stores

The MapReduce core module now includes an intermediate manifest committer designed to improve resilience and performance when writing output to object storage systems like Azure Blob Storage and Google Cloud Storage. This change introduces a new package structure under \org.apache.hadoop.mapreduce.lib.output.committer.manifest\ containing core components such as \ManifestStoreOperations\ for abstracting filesystem actions, \CleanupJobStage\ for handling parallel directory cleanup to mitigate timeouts on large directories, and \ManifestPrinter\ as a diagnostic tool. These classes provide the implementation logic for managing task manifests and success files, supporting the new commit protocol required for these cloud storage backends.

hadoop-mapreduce-client-core · high confidence

Introduction of MapReduce v2 client and history server protocols

This change introduces the core protocol interfaces and client implementations for the MapReduce v2 (YARN-based) architecture. It adds the \MRClientProtocol\ and \HSClientProtocol\ interfaces, along with their Protobuf-based RPC implementations (\MRClientProtocolPB\, \HSClientProtocolPB\), enabling clients to interact with the new ApplicationMaster and JobHistoryServer. The diff also includes the \LocalJobRunner\ and \LocalDistributedCacheManager\ classes, which provide the local execution mode for debugging and testing, and the \TypeConverter\ utility to bridge between the legacy \mapred\ and new \mapreduce\ data types.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common · high confidence

Introduction of advanced RPC scheduling, asynchronous calls, and caller context tracking

The IPC layer now supports a new \DecayRpcScheduler\ that prioritizes requests based on a decaying cost model, configurable via \ipc.scheduler.decay-scheduler.\*\ properties, and a \FairCallQueue\ for multi-level priority scheduling. Clients can now initiate asynchronous RPC calls using \CompletableFuture\ via \Client.getAsyncRpcResponse()\, with limits enforced by \AsyncCallLimitExceededException\. Additionally, the \CallerContext\ class allows passing structured metadata (e.g., real user, proxy user) for auditing, and the \AlignmentContext\ interface enables state synchronization between client and server during RPC exchanges.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc · high confidence

Introduction of annotation-based metrics instrumentation

This change introduces a new annotation-driven approach for defining metrics in Hadoop. It adds \@Metric\ and \@Metrics\ annotation interfaces in the \org.apache.hadoop.metrics2.annotation\ package, allowing developers to mark fields and methods as counters, gauges, tags, or default metrics. The \org.apache.hadoop.metrics2.lib\ package now includes supporting infrastructure such as \MetricsSourceBuilder\ to parse these annotations and generate \MetricsSource\ objects, \MutableMetricsFactory\ for creating metric instances, and \DefaultMetricsFactory\/\DefaultMetricsSystem\ for managing the metrics system lifecycle. Existing mutable metric classes like \MutableCounterInt\, \MutableGaugeLong\, and \MutableInverseQuantiles\ are also present, providing the underlying implementation for the annotated metrics.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib · high confidence

Introduction of native task serialization infrastructure

The native task module now includes a new \NativeSerialization\ class that manages the mapping between Writable types and their corresponding native serializers. This component allows users and internal components to register custom serializers for specific Writable classes and retrieve them efficiently, falling back to a default serializer when no specific registration is found, thereby enabling the serialization of data for the native output collector feature.

hadoop-mapreduce-client-nativetask · high confidence

Introduction of new HTTP server components and utilities

The HttpServer2 module introduces several new capabilities and structural improvements. A new AdminAuthorizedServlet provides admin-only access to specific endpoints, while IsActiveServlet offers a standard endpoint for load balancers to detect active cluster nodes. Performance and observability are enhanced with ProfileServlet and ProfileOutputServlet for async-profiler integration, and PrometheusServlet for publishing metrics in Prometheus format. Security and usability are improved via HtmlQuoting for safe HTML character handling, NoCacheFilter to prevent caching of dynamic content, and WebServlet to support Kerberos SPNEGO token redirects. Additionally, the HttpConfig class introduces a policy-based configuration framework for HTTP/HTTPS, and HttpRequestLog switches request logging to SLF4J for better logging integration.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http · high confidence

Introduction of pluggable service-level authorization and proxy user impersonation framework

This change introduces a new authorization framework in the \org.apache.hadoop.security.authorize\ package, replacing the previous monolithic implementation with a modular, pluggable architecture. It adds \AccessControlList\ for managing user and group permissions, \ServiceAuthorizationManager\ for enforcing service-level ACLs and host restrictions, and a new \ImpersonationProvider\ interface with a \DefaultImpersonationProvider\ implementation to handle proxy user (doAs) authorization. This allows ecosystem projects to configure and extend how service access and user impersonation are validated, while maintaining backward compatibility through existing configuration keys.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize · high confidence

Introduction of the Metrics 2.0 framework core interfaces

The metrics2 package now provides the core interfaces and abstract classes for the new Metrics 2.0 framework, including MetricsSystem for managing sources and sinks, MetricsSource and MetricsSink for defining data producers and consumers, and supporting types like AbstractMetric, MetricsRecord, and MetricsFilter. This change establishes the foundational API for instrumenting and publishing metrics, replacing or supplementing the previous metrics implementation.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2 · high confidence

Introduction of the hadoop-hdfs-client module with core client classes

The hadoop-hdfs-client module is introduced, providing a dedicated library for HDFS client-side functionality. This includes the core DFSClient for file operations, HdfsAdmin for administrative tasks, and supporting classes like KeyProviderCache for encryption key management, LocatedBlocksRefresher for asynchronous block location updates, and ReaderStrategy for optimized read operations. The module also adds XAttr for extended attribute support and DFSOpsCountStatistics for tracking DFS operation counts.

hadoop-hdfs-client · high confidence

Job History Server introduces in-memory caching and structured audit logging

The Job History Server now features a new \CachedHistoryStorage\ component that manages an in-memory cache of parsed job history files, allowing for configurable cache sizes and task-based weight limits to optimize memory usage and performance. Additionally, the server now includes \HSAuditLogger\ to record structured audit trails for user operations, and \HSProxies\ to standardize RPC connections for administrative tasks like refreshing user mappings.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs · high confidence

Migrated Hadoop Pipes native build to CMake and added example applications

The Hadoop Pipes native build system has been migrated from the previous build method to a new CMake framework (minimum version 3.5), which resolves build failures on modern platforms like CentOS 8 and ensures compatibility with OpenSSL 1.1.0. Alongside this build infrastructure change, the release includes the Hadoop Pipes example applications (wordcount-simple, wordcount-part, wordcount-nopipe, and pipes-sort) with their corresponding configuration files and documentation, allowing users to compile and run these native MapReduce examples directly.

hadoop-tools/hadoop-pipes · high confidence

Native CGroups module for device isolation support

The NodeManager's native container-executor now includes a new CGroups module (cgroups-operations.c/h) that provides the underlying mechanism for device isolation. This implementation adds functions to construct CGroups paths based on configuration (root and yarn-hierarchy) and write parameter values to CGroups files, enabling the NodeManager to enforce device limits (such as GPU access) on containers via the Linux cgroups interface.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/cgroups · high confidence

Native CRC32/C checksumming with hardware acceleration

Hadoop now uses native C code for CRC32 and CRC32C checksum calculations, significantly improving performance on supported platforms. The new implementation includes pipelined software algorithms and hardware-accelerated paths for x86 (SSE4.2), ARM64 (AArch64), and RISC-V (Zbc extension). It also adds JNI bindings in NativeCodeLoader to detect build-time support for Snappy, OpenSSL, and ISA-L, and provides a getLibraryName method to report the loaded native library path on both Unix and Windows.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util · high confidence

Native FPGA isolation support via cgroups in container-executor

The NodeManager's native container-executor now includes a new FPGA module that enforces device isolation using cgroups. This allows administrators to restrict which FPGA devices a container can access by configuring allowed minor device numbers and excluding specific devices, ensuring that containers only interact with permitted hardware resources.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga · high confidence

Native GPU isolation via CGroups in container-executor

The NodeManager's native container-executor now supports isolating GPU devices using CGroups. This change introduces a new GPU module that allows administrators to configure allowed GPU minor device numbers and blacklist specific GPUs for containers, ensuring that containers only access the resources they are permitted to use. The implementation handles device major/minor number configuration and integrates with the CGroups device controller to enforce these restrictions at the OS level.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu · high confidence

Native I/O library adds PMDK support and cross-platform file descriptor handling

The native I/O library now includes support for Persistent Memory Development Kit (PMDK) libraries, enabling Hadoop to check for and utilize PMDK via the \hadoop checknative\ command and supporting HDFS cache on Storage Class Memory (SCM). Additionally, the native code now provides robust, cross-platform file descriptor management for both UNIX and Windows environments, and introduces a \SharedFileDescriptorFactory\ to create and manage temporary file descriptors for shared memory segments, improving reliability and performance for in-memory replica persistence.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio · high confidence

Native Intel ISA-L erasure coding implementation added

This change introduces the native C implementation for Hadoop's erasure coding, leveraging the Intel ISA-L library for optimized Reed-Solomon and XOR encoding/decoding. The new files in the native source directory provide the JNI bindings (jni\_rs\_encoder.c, jni\_rs\_decoder.c, jni\_xor\_encoder.c, jni\_xor\_decoder.c) that expose the native functionality to Java, along with the core coder logic (erasure\_coder.c), GF utility functions (gf\_util.c), and the dynamic library loader (isal\_load.c). This enables Hadoop to use hardware-accelerated native libraries for erasure coding operations, improving performance for data durability and storage efficiency.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode · high confidence

Native JNI group mapping implementation and cross-platform support

The native security library now includes a complete C implementation for the JniBasedUnixGroupsMapping, providing robust user and group lookups on Unix systems with proper handling of buffer resizing, signal interruptions (EINTR), and platform-specific errors. A Windows-specific native module is added to support group lookups on Windows Server environments. Additionally, a new native module implements JniBasedUnixGroupsNetgroupMapping to resolve users from netgroups on Linux and macOS, ensuring correct exception handling when netgroup lookups fail.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security · high confidence

Native OpenSSL crypto codec adds SM4 support and fixes OpenSSL 3.x compatibility

The native OpenSSL-based crypto codec now supports the SM4 encryption algorithm (SM4/CTR/NoPadding) in addition to the existing AES/CTR modes, allowing users to encrypt data with SM4 when using the native library. The implementation also resolves compatibility issues with OpenSSL 3.x by dynamically loading symbols that changed names in the 3.0 API (such as EVP\_CIPHER\_CTX\_get\_block\_size and EVP\_CIPHER\_CTX\_is\_encrypting), ensuring the native library loads and functions correctly on systems with newer OpenSSL versions.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto · high confidence

Native OpenSSL-based secure random number generator

Added a new native implementation (OpensslSecureRandom.c) that provides a high-performance secure random number source by dynamically loading and using the OpenSSL library (specifically ENGINE and RAND APIs) on UNIX and Windows platforms. This change introduces the underlying C/JNI code required for Hadoop to utilize OpenSSL's cryptographic random number generation capabilities, supporting both pre-1.1.0 and newer OpenSSL versions through conditional compilation.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random · high confidence

Native bzip2 compression and decompression implementation

This change introduces the native C implementation for the Hadoop bzip2 codec, adding Bzip2Compressor.c and Bzip2Decompressor.c along with their JNI header. These files provide the low-level bindings to the libbz2 library, enabling direct memory access for compression and decompression operations to improve performance over the pure-Java alternative.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2 · high confidence

Native-code implementation of BZip2 codec

The BZip2 compression codec in hadoop-common now supports a native-code implementation for improved performance. The \Bzip2Factory\ detects whether the native library is available and instantiates \Bzip2Compressor\ and \Bzip2Decompressor\ (which wrap the native calls) instead of the previous pure-Java \BZip2DummyCompressor\ and \BZip2DummyDecompressor\. This change also introduces thread-safety to the compressor and decompressor classes by synchronizing their methods, and updates the internal \CBZip2InputStream\ to better track stream position for Hadoop splitting.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2 · high confidence

New CLI shell framework and user-group mapping tools in hadoop.tools

This change introduces a new \CommandShell\ base class in the \org.apache.hadoop.tools\ package to standardize CLI command structure, providing a consistent interface for subcommands, usage printing, and exception handling. Alongside this framework, it adds \GetGroupsBase\ and the \GetUserMappingsProtocol\ interface to enable tools to fetch and display user group mappings via RPC, and includes a \TableListing\ utility for formatted tabular output. These additions provide the foundational components for building and displaying structured command-line tools within Hadoop.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools · high confidence

New Capacity Scheduler UI: Placement Rules Management and Migration

The Capacity Scheduler UI now includes a dedicated Placement Rules feature, allowing administrators to view, add, edit, and reorder rules via a split-pane interface. This update introduces a migration tool that detects legacy placement rules and offers a one-click conversion to the new JSON format, and adds a help dialog explaining custom placement variables and usage examples for dynamic queue hierarchy creation.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/placement-rules · high confidence

New Erasure Codec Framework and Implementations

This change introduces a new erasure codec framework in the \org.apache.hadoop.io.erasurecode.codec\ package, providing a standardized interface for erasure coding operations. It adds an abstract \ErasureCodec\ base class that manages schema and coder options, along with concrete implementations for XOR, Reed-Solomon (RS), and Hitchhiker XOR (HHXOR) coding schemes. A \DummyErasureCodec\ is also included for testing and performance comparison purposes. This framework allows users to leverage these specific erasure coding algorithms for data redundancy and recovery.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec · high confidence

New FileSystem stream capability and batch listing interfaces

The FileSystem API introduces new interfaces to expose advanced stream capabilities and bulk operations. New interfaces such as Abortable, CanSetDropBehind, CanSetReadahead, and CanUnbuffer allow output and input streams to support specific optimizations like aborting writes, controlling cache behavior, and adjusting readahead buffers. Additionally, the BatchListingOperations interface and BulkDeleteSource provide standardized ways for filesystems to offer batched directory listing and bulk file deletion, with support indicated via path capabilities.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs · high confidence

New Ganglia 3.0 and 3.1 Metrics2 sinks

This change introduces new Ganglia sink implementations (GangliaSink30 and GangliaSink31) for the Metrics2 framework, replacing the legacy context-based approach. The new sinks support sending metrics to multiple Ganglia servers, enable multicast configuration, and allow including metrics system tags in metric names. They also handle metric type detection and slope calculation automatically, and support both dense and sparse metric publishing modes.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia · high confidence

New Git/Jira FixVersion validation utility for release candidates

Added a new Python script, \git\_jira\_fix\_version\_check.py\, and its documentation to help release managers verify consistency between Git commit messages and Jira fixVersions during RC preparation. The tool identifies discrepancies such as commits lacking Jira IDs, commits referencing Jiras without the expected fixVersion, reverted commits, and resolved Jiras that lack corresponding commits.

dev-support/git-jira-validation · high confidence

New HA client target and base event class in YARN common

The hadoop-yarn-common module introduces two new classes to support high-availability and event handling. RMHAServiceTarget provides a client-side target for ResourceManager HA operations, exposing the admin service address and auto-failover status while currently throwing exceptions for ZKFC and fencing features. AbstractEvent serves as the foundational class for all YARN events, allowing subclasses to specify an event type and an optional timestamp.

hadoop-yarn-common · high confidence

New HDFS Compatibility Benchmark tool for FileSystem implementations

A new compatibility benchmark tool has been added to the hadoop-compat-bench module to assess how well different FileSystem implementations adhere to the HDFS definition. The tool, invoked via \HdfsCompatTool\, accepts a target URI and an optional suite filter (e.g., 'shell') to run a suite of compatibility checks covering core operations (create, read, delete), directory management, ACLs, snapshots, storage policies, and symlinks. It generates a report detailing the percentage of passed and failed compatibility cases, helping users identify functional gaps or deviations in their custom or third-party Hadoop-compatible storage systems.

hadoop-tools/hadoop-compat-bench · high confidence

New HDFS client API classes and exceptions introduced

The hdfs-client module now exposes new public and private API classes to support advanced client operations. This includes the CacheFlag enum for controlling cache directive semantics, the HdfsBlockLocation wrapper for detailed block queries, and the BlockMissingException for handling read failures on missing blocks. Additionally, new internal context classes (ClientGSIContext) and protocol definitions (SWebHdfs, WebHdfs) are added to support enhanced client-server interactions and observer read capabilities.

hadoop-hdfs-project/hadoop-hdfs-client · high confidence

New HTTP security filters for CORS and clickjacking protection

Hadoop Common now includes two new servlet filters in the \org.apache.hadoop.security.http\ package to enhance web application security. The \CrossOriginFilter\ enables Cross-Origin Resource Sharing (CORS) support by allowing administrators to configure allowed origins, methods, and headers via initialization parameters, with support for regex-based origin patterns. The \XFrameOptionsFilter\ protects against clickjacking attacks by setting the \X-Frame-Options\ header (defaulting to \DENY\) and preventing other filters from overriding this value. These filters are publicly exposed as evolving interfaces for use in Hadoop web UIs and REST APIs.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http · high confidence

New HTTP/HTTPS FileSystem implementations for direct remote file access

This change introduces new \HttpFileSystem\ and \HttpsFileSystem\ classes in the \org.apache.hadoop.fs.http\ package, enabling Hadoop to read data directly from HTTP and HTTPS endpoints. The implementation provides a read-only filesystem interface, allowing users to open and stream files from remote web servers using standard Hadoop FileSystem APIs, while explicitly throwing \UnsupportedOperationException\ for write, rename, delete, and directory listing operations to enforce the read-only nature of these connectors.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/http · high confidence

New High Availability (HA) framework for service failover and fencing

This change introduces a new, comprehensive High Availability framework in the \org.apache.hadoop.ha\ package. It provides the core \ActiveStandbyElector\ for ZooKeeper-based leader election, a \FailoverController\ to manage state transitions and pre-failover checks, and a \HealthMonitor\ daemon to track service health. The framework includes a pluggable fencing system via the \FenceMethod\ interface and \NodeFencer\, with a new \PowerShellFencer\ implementation for Windows environments. Administrators can manage HA services using the new \HAAdmin\ CLI tool, which supports commands like \-transitionToActive\, \-transitionToStandby\, and \-checkHealth\. The \HAServiceProtocol\ interface defines the RPC contract for state changes and health checks, supporting states such as ACTIVE, STANDBY, and OBSERVER.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha · high confidence

New Hitchhiker erasure coding implementation and coder framework

This change introduces the Hitchhiker erasure coding algorithm (HADOOP-11828) alongside a refactored coder framework. The \coder\ package now contains new classes including \HHXORErasureEncoder\, \HHXORErasureDecoder\, and their associated steps (\HHXORErasureEncodingStep\, \HHXORErasureDecodingStep\) which utilize \HHUtil\ for piggyback calculations to reduce network traffic and disk I/O during data reconstruction. The diff also adds a new \ErasureCoder\ interface and \ErasureCodingStep\ abstraction, replacing previous direct raw-coder usage with a step-based model for encoding and decoding block groups. Existing coders like \RSErasureEncoder/Decoder\ and \XORErasureEncoder/Decoder\ have been updated to implement this new interface, and dummy encoders/decoders are added for performance isolation testing.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder · high confidence

New I/O utility classes and Writable types introduced

The \org.apache.hadoop.io\ package now includes several new classes to support advanced serialization and I/O operations. \ArrayPrimitiveWritable\ provides a wrapper for primitive arrays (e.g., int\[\], long\[\]) with an optimized wire format, enabling efficient serialization of primitive collections without per-element object creation. \BoundedByteArrayOutputStream\ offers a resettable byte array output stream with a configurable limit, useful for bounded buffer reuse. \ByteBufferPool\ and \DataInputByteBuffer\ introduce a buffer pooling mechanism and a multi-buffer input stream wrapper, respectively, to reduce allocation overhead during data transfer. Additionally, \SecureIOUtils\ now supports opening \FSDataInputStream\ on local files with security checks, and \Sizes\ provides named constants for common binary sizes. These additions enhance performance and security for internal Hadoop I/O operations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io · high confidence

New JVM metrics source with memory, GC, and thread details

The JvmMetrics source in hadoop-common now exposes detailed JVM telemetry, including heap and non-heap memory usage (used, committed, max), garbage collection counts and times (excluding ZGC concurrent cycles), and thread state counts (new, runnable, blocked, waiting, timed-waiting, terminated). It also supports GC pause monitoring via GcTimeMonitor and JvmPauseMonitor, and allows users to opt into using ThreadMXBean for thread info via the Hadoop configuration key hadoop.metrics.jvm.useThreadMXBean.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source · high confidence

New Jenkins CI pipelines for Linux and Windows platforms

The dev-support directory now includes Jenkinsfile definitions for running pre-commit CI on Rocky Linux 8, Debian 13, and Windows 10. These pipelines use Apache Yetus to execute builds and tests, with the Linux jobs running in Docker containers and the Windows job using a dedicated agent. The configuration ensures that C/C++ changes trigger cross-platform validation on all supported OSes, while JUnit results are published only when surefire reports are present.

dev-support · high confidence

New Kafka metrics sink for Hadoop metrics2

A new KafkaSink component has been added to the hadoop-kafka module, enabling Hadoop metrics2 records to be published directly to an Apache Kafka topic. Users can configure the sink by specifying a broker list and topic in the metrics2 configuration file; the sink formats metrics as JSON lines and sends them to the specified Kafka cluster.

hadoop-tools/hadoop-kafka · high confidence

New MapReduce examples: SecondarySort and DistSum

The hadoop-mapreduce-examples module now includes two new example applications. SecondarySort demonstrates how to sort data by a secondary key using custom WritableComparable types, partitioners, and grouping comparators. DistSum provides an example of computing sums using MapReduce jobs, supporting map-side, reduce-side, and mixed execution types, and utilizes SLF4J for logging and monotonic time for duration measurements.

hadoop-mapreduce-examples · high confidence

New Maven plugins for native builds, protocol buffer generation, and resource compression

The hadoop-maven-plugins module introduces several new Maven goals to streamline build processes. The \cmake-compile\ and \cmake-test\ goals allow building and testing native code via CMake, while the \protoc\ and \test-protoc\ goals generate Java classes from \.proto\ files with incremental build support via checksums. Additionally, the \parallel-tests-createdir\ goal prepares directories for parallel test execution, and the \resource-gz\ goal compresses static assets like JavaScript and CSS files.

hadoop-maven-plugins · high confidence

New ProxyUserAuthenticationFilter for web protocol impersonation

Added a new \ProxyUserAuthenticationFilter\ and its initializer to the Hadoop authentication server framework. This filter enables web protocol impersonation by allowing operations to be performed on behalf of an end user specified via the \doAs\ query parameter, rather than the proxy user. It validates the request against proxy authorization rules and updates the request context to reflect the impersonated user, supporting secure delegation in web-facing Hadoop services.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authentication · high confidence

New RM fatal event handling and reservation planning algorithms

The Resource Manager now includes a new RMFatalEvent class to better capture and report non-recoverable errors with their underlying causes. Additionally, the reservation system introduces new planning algorithms: StageAllocatorGreedyRLE for efficient stage allocation using run-length encoding, and TryManyReservationAgents to attempt multiple reservation agents in sequence until one succeeds.

hadoop-yarn-server-resourcemanager · high confidence

New RPC metrics classes for detailed server statistics

The RPC metrics subsystem now includes new classes to expose granular server-side statistics. RpcMetrics aggregates overall RPC activity, including queue times, processing times, authentication/authorization outcomes, and connection counts, with support for configurable quantiles. RpcDetailedMetrics provides per-method breakdowns of RPC processing times, distinguishing between deferred and overall processing. DecayRpcSchedulerDetailedMetrics adds priority-level queue and processing time metrics for the FairCallQueue, while RetryCacheMetrics tracks cache hits, updates, and clears. These changes allow users to monitor RPC performance and health with greater specificity via JMX and metrics systems.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics · high confidence

New StaticUserWebFilter for web UI access without authentication

A new StaticUserWebFilter has been added to the hadoop.http.lib package, allowing administrators to configure a static username (via the hadoop.http.static.user property) so that the Hadoop web UI remains accessible in secure clusters without requiring individual user authentication. The filter intercepts requests and pretends the user is the configured static identity (defaulting to 'Dr.Who') unless the user is already authenticated, and it supports the deprecated dfs.web.ugi configuration key by extracting the username from it while issuing a warning.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/lib · high confidence

New WrappedIO and WrappedStatistics APIs for reflection-friendly filesystem access

Added the \org.apache.hadoop.io.wrappedio\ package, introducing \WrappedIO\ and \WrappedStatistics\ classes that provide static, reflection-friendly access to newer Hadoop filesystem APIs (such as bulk delete, openFile, and IO statistics snapshots). These APIs are designed to allow libraries to call modern Hadoop features without complex reflection logic or version-specific code paths. The \impl\ subpackage provides \DynamicWrappedIO\ and \DynamicWrappedStatistics\ to dynamically load and bind these methods at runtime, ensuring compatibility across different Hadoop versions.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio · high confidence

New YARN CLI and client APIs for Shared Cache and Interactive Shells

The YARN client module introduces new command-line and API capabilities for managing the Shared Cache and interacting with containers. Administrators can now use the \yarn scmadmin\ CLI to manually trigger shared cache cleanup tasks via the new \SCMAdmin\ tool. Additionally, the \SharedCacheClient\ API allows applications to claim and release resources in the shared cache, while the \ContainerShellWebSocket\ class enables interactive shell connections to containers via WebSocket. The \NMTokenCache\ class is also exposed to allow Application Masters to manage NodeManager tokens, supporting scenarios with multiple AMs in the same JVM.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client · high confidence

New YARN persistence-based node selector in Hadoop Registry

The Hadoop Registry now includes a new \SelectByYarnPersistence\ class that implements the \RegistryAdminService.NodeSelector\ interface. This component allows the registry to select service entries based on YARN-specific persistence policies and IDs, enabling more targeted management of YARN-registered services within the Hadoop ecosystem.

hadoop-registry · high confidence

New ZKCuratorManager utility for ZooKeeper interactions

A new ZKCuratorManager class has been added to the org.apache.hadoop.util.curator package to provide utility methods for interacting with ZooKeeper via Apache Curator. This helper class manages ZooKeeper connections, supporting configuration of connection parameters, authentication information, and ACLs, and exposes the underlying CuratorFramework instance for use by other Hadoop components.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator · high confidence

New archive-logs tool to combine YARN aggregated logs into HAR archives

This change introduces the \mapred archive-logs\ command, which aggregates YARN application logs into Hadoop Archives (HAR) to reduce NameNode pressure from small files. The tool identifies eligible applications based on criteria such as log count and size, then uses the Distributed Shell to run parallel containers that create the archives and replace the original log files. It supports options to limit the number of processed applications, control memory usage, and handle user impersonation (with a \-noProxy\ flag to run as the invoking user). The implementation includes the main \HadoopArchiveLogs\ tool, a \HadoopArchiveLogsRunner\ for distributed execution, and a shell profile to register the subcommand.

hadoop-tools/hadoop-archive-logs · high confidence

New authentication handlers for alternative Kerberos and JWT WebSSO

The authentication server module introduces two new authentication mechanisms: AltKerberosAuthenticationHandler, which allows administrators to configure a custom browser authentication flow while retaining standard Kerberos SPNEGO for Java clients, and JWTRedirectAuthenticationHandler, which enables Web Single Sign-On (WebSSO) by validating RS256-signed JWT tokens from an external provider. These handlers are registered within the AuthenticationFilter and AuthenticationHandlerUtil, expanding the supported authentication types to include 'alt-kerberos' and JWT-based flows.

hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server · high confidence

New authentication utility classes for token management, signing, and Kerberos handling

This change introduces a suite of new utility classes in the hadoop-auth module to support authentication workflows. AuthToken provides a mechanism for creating and managing authentication tokens with expiration and inactivity controls. Signer, SignerSecretProvider, and its implementations (FileSignerSecretProvider, RandomSignerSecretProvider, RolloverSignerSecretProvider, ZKSignerSecretProvider) enable secure string signing with support for secret rotation, file-based secrets, and ZooKeeper-synchronized secrets. KerberosName and KerberosUtil handle Kerberos principal name parsing, realm resolution, and OID management, including compatibility for IBM JDK. CertificateUtil assists in parsing RSA public keys from PEM format. JaasConfiguration allows programmatic setup of JAAS login configurations, particularly for ZooKeeper connections. SubjectUtil manages JAAS Subject propagation across threads, with specific handling for JDK 22+ changes where native Subject inheritance was removed.

hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util · high confidence

New builder-based openFile API and vector IO infrastructure

This change introduces a new builder-based API for opening files (openFile) and performing vector I/O operations, providing a fluent interface for configuring buffer sizes, file ranges, and other options. It includes new implementation classes such as AbstractFSBuilderImpl, FutureDataInputStreamBuilderImpl, and FileRangeImpl to support these capabilities, along with helper classes like CombinedFileRange for merging ranges and FSBuilderSupport for option parsing. Additionally, it adds a default implementation for bulk file deletion (DefaultBulkDeleteOperation) and a multipart uploader builder (FileSystemMultipartUploaderBuilder) to facilitate large file uploads via the new builder pattern.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl · high confidence

New built-in metrics sinks for File, Graphite, Prometheus, StatsD, and RollingFileSystem

This release adds five new metrics sink implementations to the Hadoop metrics2 system, allowing users to export metrics to a wider variety of backends. The new FileSink writes metrics to a local file or standard output, while GraphiteSink sends data to a Graphite server over TCP. PrometheusMetricsSink exposes metrics in Prometheus exposition format for scraping, and StatsDSink sends metrics via UDP to a StatsD daemon. Additionally, RollingFileSystemSink provides time-based log rotation for metrics written to any Hadoop-compatible file system (such as HDFS, S3, or local disk), supporting configurable roll intervals, append modes, and secure (Kerberos) authentication.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink · high confidence

New centralized checkstyle configuration and report stylesheets in hadoop-build-tools

The hadoop-build-tools module now ships a new, centralized checkstyle configuration (checkstyle.xml) and a no-frames sorted HTML report stylesheet (checkstyle-noframes-sorted.xsl), along with a default suppressions file. The configuration enforces a 100-character line length, allows nested blocks in switch cases, ignores overridden methods for parameter count limits, and suppresses Javadoc package checks for test and generated source directories. This provides a consistent, out-of-the-box coding standard and reporting format for Hadoop projects without requiring external configuration files.

hadoop-build-tools · high confidence

New concurrent utility classes for executor management and async operations

Added a new \org.apache.hadoop.util.concurrent\ package containing \HadoopThreadPoolExecutor\ and \HadoopScheduledThreadPoolExecutor\, which extend standard Java executors to automatically log uncaught exceptions via \ExecutorHelper.logThrowableFromAfterExecute\. Introduced \HadoopExecutors\ as a factory for creating these executors, including a robust \shutdown\ helper that handles graceful and forced termination with timeouts. Also added \AsyncGet\ and \AsyncGetFuture\ to support asynchronous operations with timeout capabilities, wrapping \AsyncGet\ implementations in a \Future\-compatible interface.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent · high confidence

New configurable hashing API in hadoop-common

This change introduces a new public API for hashing in the hadoop-common module, located in the org.apache.hadoop.util.hash package. It adds an abstract Hash base class with two concrete implementations: JenkinsHash (based on Bob Jenkins' lookup3) and MurmurHash (MurmurHash 2.0). The API allows users to select the hash algorithm via configuration (using the key hadoop.util.hash.type, defaulting to jenkins) or programmatically, and supports hashing byte arrays with optional offsets and seed values. This provides a standardized, pluggable way for Hadoop components to perform non-cryptographic hashing for tasks like hash table lookups.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/hash · high confidence

New configuration logging and filesystem statistics infrastructure

Added ConfigurationWithLogging to automatically log all configuration access with sensitive data redaction, and introduced FileSystemStorageStatistics to expose per-filesystem metrics (bytes read/written, operation counts, read distances) via the StorageStatistics API. Also added new filesystem options (Options.CreateOpts) for builder-based file creation, PathHandle implementations (LocalFileSystemPathHandle, BBPartHandle, BBUploadHandle) for efficient file state tracking, and utility classes (PartialListing, FileEncryptionInfo, AvroFSInput, UserProvider) to support advanced listing, encryption, and Avro integration.

hadoop-common · high confidence

New configuration management and redaction utilities

This change introduces several new components in the configuration package to enhance runtime management and security. A new \ConfServlet\ allows administrators to view the running configuration via a REST API, supporting both JSON and XML output formats based on the request's Accept header. To prevent sensitive data exposure, a \ConfigRedactor\ class has been added to automatically redact sensitive configuration values (such as cloud storage keys) when displaying configuration data. Additionally, new classes \StorageSize\ and \StorageUnit\ provide a structured way to parse and handle human-readable storage size values (e.g., '100MB') within configurations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf · high confidence

New crypto-aware FSDataInputStream and FSDataOutputStream wrappers

Added CryptoFSDataInputStream and CryptoFSDataOutputStream classes that wrap existing Hadoop FS data streams with encryption capabilities. These new stream implementations extend FSDataInputStream and FSDataOutputStream respectively, delegating to CryptoInputStream and CryptoOutputStream to provide transparent encryption/decryption for file operations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/crypto · high confidence

New default configuration files for Hadoop Common

This change introduces a set of new default configuration files into the Hadoop Common distribution, providing out-of-the-box settings for core-site, environment variables (shell and Windows), logging (log4j), metrics, security policies, and SSL. Users will now have ready-to-use templates for hadoop-env.sh and hadoop-env.cmd, including default heap sizes and security logger configurations, alongside updated log4j.properties that route security and audit logs to NullAppender by default to reduce noise. The inclusion of ssl-server.xml.example and ssl-client.xml.example provides explicit cipher inclusion and exclusion lists to enhance SSL/TLS security, while hadoop-policy.xml defines default ACLs for various Hadoop protocols.

hadoop-common-project/hadoop-common/src/main/conf · high confidence

New delegation token management utilities and interfaces

This change introduces a new set of classes and interfaces in the \org.apache.hadoop.security.token\ package to support delegation token operations. It adds the \DelegationTokenIssuer\ interface for issuing tokens, \DtFetcher\ for abstracting token fetching, and \DtFileOperations\ for managing token files (reading, writing, and printing in protobuf or Java serialization formats). A new command-line utility, \DtUtilShell\, is provided to manage token files via subcommands like print, get, edit, append, cancel, remove, and renew. Additionally, \SecretManager\ is introduced to handle server-side secret management for tokens, and existing classes like \Token\, \TokenIdentifier\, \TokenRenewer\, and \TokenSelector\ are updated or added to support these new capabilities.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token · high confidence

New delegation token secret manager implementations for SQL and ZooKeeper

This change introduces new classes in the \org.apache.hadoop.security.token.delegation\ package to support alternative storage backends for delegation tokens. \SQLDelegationTokenSecretManager\ allows tokens to be persisted in an SQL database, utilizing a loading cache to manage memory usage for high token volumes. \ZKDelegationTokenSecretManager\ provides a ZooKeeper-based backend for highly available services, ensuring token identifiers and keys are replicated across nodes. These implementations extend the existing \AbstractDelegationTokenSecretManager\, enabling ecosystem projects to choose the storage mechanism that best fits their infrastructure.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation · high confidence

New direct-buffer-based Snappy compression codecs

The Snappy compression and decompression implementations in hadoop-common now use Java NIO direct buffers (via the snappy-java library) instead of the previous native-code approach. This change introduces SnappyCompressor and SnappyDecompressor classes that manage memory using direct ByteBuffers, which can improve performance and reduce garbage collection pressure for large data streams. The package is marked as Private and Unstable, indicating it is an internal implementation detail subject to change.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy · high confidence

New distributed metadata change tool (DistCh) and user group lookup utility

This release introduces the \distch\ command-line tool, which allows users to recursively change file ownership, group, and permissions across Hadoop Distributed File System (HDFS) paths using a MapReduce job. The tool is registered as a subcommand in the Hadoop shell profile and includes a corresponding test suite (\TestDistCh\) to verify its behavior. Additionally, a new \GetGroups\ tool is added to the MapReduce tools package, enabling users to retrieve the groups a specific user belongs to.

hadoop-tools/hadoop-extras · high confidence

New documentation for cluster compatibility guidelines and setup

The site documentation now includes a new AdminCompatibilityGuide.md that distills compatibility policies for system administrators, covering release categories, native dependencies, network protocols, REST APIs, CLI stability, and state data handling. Additionally, new markdown files have been added for ClusterSetup.md, Benchmarking.md (NNThroughputBenchmark), AsyncProfilerServlet.md, and CLIMiniCluster.md, providing updated instructions for cluster configuration, performance benchmarking, profiling via the async-profiler servlet, and running a CLI-based mini-cluster.

hadoop-common-project/hadoop-common/src/site · high confidence

New filesystem audit logging constants and statistic names

The S3A connector now includes structured audit telemetry by introducing the \org.apache.hadoop.fs.audit\ package. This adds \AuditConstants\ defining specific HTTP header parameters for audit logs (such as operation type, path, range, and principal) and \AuditStatisticNames\ providing standardized statistic keys for tracking audit events like request execution, span creation, and access check failures. Users will see these new headers and metrics in their audit logs, enabling more granular visibility into S3A operations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/audit · high confidence

New fine-grained entity caching for Timeline Server v1.5

The Timeline Server v1.5 plugin storage now supports fine-grained entity caching to improve read performance. This change introduces a new \EntityGroupFSTimelineStore\ that manages entity groups and utilizes a configurable cache layer, with a \LevelDBCacheTimelineStore\ implementation provided for persistent caching. The system includes \EntityCacheItem\ for managing cache lifecycle and refresh logic, \TimelineEntityGroupPlugin\ for mapping queries to entity groups, and comprehensive metrics tracking for cache operations, log scanning, and cleaning.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage · high confidence

New fs2img tool to generate HDFS FSImage from external filesystems

The hadoop-fs2img module now includes a new command-line tool that creates an HDFS FSImage from an external namespace. Users can traverse an external filesystem, mapping files to HDFS blocks and metadata, with support for configurable block sizes, replication factors, and user/group identity resolution. The tool also handles ACLs from the external store when enabled, allowing seamless integration of external data into the HDFS namespace.

hadoop-tools/hadoop-fs2img · high confidence

New fs2img tool to generate HDFS FSImages from external stores

Adds the hadoop-fs2img tool, which crawls an existing hierarchical file system and emits a valid HDFS FSImage. The implementation includes ImageWriter for serializing the image, TreeWalk for traversing the source hierarchy, and UGIResolver for mapping external ownership and permissions (including ACLs) to the HDFS namespace.

hadoop-fs2img · high confidence

New functional and dynamic reflection utilities in hadoop-common

Added a new set of utility classes to support functional programming patterns with IO exceptions and dynamic class loading. The \org.apache.hadoop.util.functional\ package introduces functional interfaces (such as \FunctionRaisingIOE\, \ConsumerRaisingIOE\, and \CallableRaisingIOE\) that allow standard Java lambdas to throw \IOException\, along with helper classes like \FutureIO\ for awaiting asynchronous results and \LazyAtomicReference\ for deferred initialization. Additionally, the \org.apache.hadoop.util.dynamic\ package provides \BindingUtils\, \DynMethods\, and \DynConstructors\ to assist in binding to Hadoop APIs through reflection, sourced from the Parquet project to improve portability.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional · high confidence

New high-performance secure random number sources

Hadoop now includes two new secure random implementations in the crypto.random package to improve performance and resource management. OpensslSecureRandom provides a JNI-based implementation that leverages OpenSSL and Intel RDRAND hardware instructions when available, falling back to Java's SecureRandom otherwise. OsSecureRandom introduces a lazy-fill reservoir strategy for reading from the OS random device (e.g., /dev/urandom), which reduces file descriptor usage and improves efficiency by batching reads into an 8192-byte buffer.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random · high confidence

New internal IOStatistics implementation classes

This location introduces the internal implementation classes for the Hadoop IOStatistics framework, including the \IOStatisticsStore\ interface and its builder, dynamic statistics builders, empty/no-op stubs for testing or disabled states, and context integration for thread-level statistics collection.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl · high confidence

New internal shared classes for object store uploads and auditing

Added a new \org.apache.hadoop.fs.store\ package containing internal utilities to be shared across Hadoop object store implementations (such as S3A and ABFS). This includes \DataBlocks\ to support huge file uploads by buffering data to disk, direct byte buffers, or arrays to manage memory usage, \ByteBufferInputStream\ for efficient stream handling, \EtagChecksum\ for object integrity verification, and \LogExactlyOnce\ for safe logging. Additionally, a new \audit\ sub-package introduces the internal auditing framework (\AuditSpan\, \AuditSpanSource\, \AuditingFunctions\) to track and log filesystem operations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/store · high confidence

New internal utilities for raw erasure coding debugging and Galois field arithmetic

Added a new internal utility package (\org.apache.hadoop.io.erasurecode.rawcoder.util\) to support raw erasure coders. This includes \DumpUtil\ for verbose debugging output of erasure coding data, and core Galois field arithmetic classes (\GF256\, \GaloisField\, \RSUtil\) that provide the mathematical foundation for Reed-Solomon encoding and decoding operations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util · high confidence

New metrics and CSI volume support classes in YARN server common

This update introduces new internal components to the YARN server common module. It adds AMRMClientRelayerMetrics to expose performance and usage metrics for the AMRMClientRelayer within the FederationInterceptor context, allowing operators to monitor pending requests, fulfillment latency, and heartbeat statistics. Additionally, it introduces VolumeId and VolumeCapabilityRange classes to support the new CSI volume manager, enabling the definition and validation of volume capacity ranges (min/max) and unique volume identification for CSI lifecycle management.

hadoop-yarn-server-common · high confidence

New metrics2 pattern filters (Glob and Regex)

The metrics2 filter package now includes AbstractPatternFilter, GlobFilter, and RegexFilter classes, allowing users to configure metrics filtering using either glob patterns (via GlobFilter) or regular expressions (via RegexFilter) in their metrics configuration files. GlobFilter delegates to Hadoop's existing GlobPattern utility, while RegexFilter uses the RE2J library for pattern matching, providing more flexible and performant filtering options for metrics names and tags.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/filter · high confidence

New modular Docker build environments for Debian, Ubuntu, Rocky Linux, and Windows

The \dev-support/docker\ directory has been restructured to provide dedicated Dockerfiles for building Hadoop on specific platforms: Debian 12 and 13, Ubuntu 24.04 (including ARM64), Rocky Linux 8, and Windows 10. These images default to JDK 17 (using OpenJDK on Debian/Ubuntu/Rocky and Azul Zulu 8 on Windows) and Maven 3.9.15. The build process now relies on a new \pkg-resolver\ system to manage platform-specific dependencies, replacing previous monolithic or obsolete images. This change ensures that developers can build Hadoop on modern operating systems and architectures that were previously unsupported or broken.

dev-support/docker · high confidence

New native exception handling and cross-platform utility headers

The native source tree now includes \exception.c\, \exception.h\, and \org\_apache\_hadoop.h\ to standardize error handling and platform compatibility. \exception.c\ and \exception.h\ introduce a new JNI-based exception creation API (\newException\, \newRuntimeException\, \newIOException\) that allows native code to throw Java exceptions with formatted messages, along with a \terror()\ function that safely retrieves error strings by preferring \strerror()\ over the deprecated \sys\_errlist\ to support modern libc implementations like Alpine musl. \org\_apache\_hadoop.h\ provides shared macros and helper functions for both Unix and Windows, including exception throwing (\THROW\), dynamic symbol loading (\LOAD\_DYNAMIC\_SYMBOL\), and platform-specific definitions (such as Unicode support and \snprintf\ shims on Windows), ensuring consistent native behavior across operating systems.

hadoop-common-project/hadoop-common/src/main/native/src · high confidence

New protobuf definitions for Hadoop RPC, HA, and security protocols

This change introduces a comprehensive set of new Protocol Buffer (protobuf) schema files in the hadoop-common project, establishing the wire-format definitions for core Hadoop services. These files define the RPC engine headers and connection context (ProtobufRpcEngine, IpcConnectionContext), the High Availability service protocol including observer state transitions (HAServiceProtocol), and security-related protocols for delegation tokens and user mappings (Security, RefreshUserMappingsProtocol). Additionally, it adds schemas for administrative refresh operations (GenericRefreshProtocol, RefreshCallQueueProtocol, RefreshAuthorizationPolicyProtocol), file system status encoding (FSProtos), and tracing administration (TraceAdmin).

hadoop-common-project/hadoop-common/src/main/proto · high confidence

New public IOStatistics API and supporting classes

This change introduces a new public API for collecting and reporting filesystem I/O statistics. It adds the core \IOStatistics\ interface and \IOStatisticsSource\ marker, along with \IOStatisticsSnapshot\ for capturing and serializing statistics (including Jackson annotations for frameworks like Spark and Flink). Supporting infrastructure includes \IOStatisticsContext\ for thread-level statistics, \DurationTracker\ and \DurationTrackerFactory\ for timing operations, and \BufferedIOStatisticsInputStream\ to wrap streams with statistics collection. The package also defines common statistic name constants in \StoreStatisticNames\, \StreamStatisticNames\, and \FileSystemStatisticNames\, and provides utility classes like \IOStatisticsLogging\ and \DurationStatisticSummary\ for reporting and parsing.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics · high confidence

New pure-Java Gzip compression and decompression implementation

Added BuiltInGzipCompressor and BuiltInGzipDecompressor classes that provide a pure-Java implementation of the gzip format, allowing gzip compression and decompression without relying on native libraries. The ZlibFactory now supports configuration of compression levels (0-9) and strategies via the 'zlib.compress.level' and 'zlib.compress.strategy' configuration properties, and uses these settings when instantiating compressors. The existing ZlibCompressor and ZlibDecompressor classes were refactored to support reinitialization with new configuration settings, and logging was migrated from Commons Logging to SLF4J across the zlib package.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib · high confidence

New queue comparison and validation features in the Capacity Scheduler UI

The Capacity Scheduler UI now includes a queue comparison tool and a comprehensive validation engine. Users can select two or more queues to compare their configuration properties side-by-side in a dialog, with differences highlighted and the option to export the comparison as a CSV file. Additionally, the UI introduces a multi-layered validation system that checks configuration changes against schema rules and cross-queue dependencies (such as parent-child capacity constraints) before they are applied, preventing invalid configurations from being submitted to the cluster.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-comparison, hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/validation · high confidence

New runtime log management and throttling utilities

The \org.apache.hadoop.log\ package now includes \LogLevel.java\, a command-line tool and servlet that allows administrators to get and set log levels for specific classes on running Hadoop daemons via HTTP or HTTPS (including Kerberos/SPNEGO support). Additionally, \LogThrottlingHelper.java\ provides a thread-safe utility for application developers to limit the frequency of log statements, preventing log flooding by aggregating redundant messages and emitting them at a configurable minimum interval.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log · high confidence

New security infrastructure components and APIs

This change introduces several new classes and interfaces to the Hadoop security package to support advanced authentication, authorization, and group mapping capabilities. It adds the \CompositeGroupsMapping\ provider to allow combining multiple user-to-group mapping services, and introduces the \IdMappingServiceProvider\ interface for user/group ID resolution. For authentication, it adds \AnnotatedSecurityInfo\ to derive Kerberos and token info from protocol annotations, \CustomizedCallbackHandler\ for extensible SASL callback handling, and \AuthorizationContext\ for managing thread-local RPC authorization headers. Performance and configuration improvements include \FastSaslClientFactory\ and \FastSaslServerFactory\ for caching SASL factories, \IngressPortBasedResolver\ for port-specific SASL property negotiation, and \AuthenticationFilterInitializer\ to configure HTTP SPNEGO authentication filters.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security · high confidence

New service launcher framework for YARN services

The \org.apache.hadoop.service.launcher\ package introduces a new framework to launch and manage YARN services from the command line. It provides the \ServiceLauncher\ class to instantiate, initialize, and start services, along with the \LaunchableService\ interface that allows services to receive command-line arguments and return specific exit codes. The package also includes \InterruptEscalator\ for handling shutdown signals with timeouts, \HadoopUncaughtExceptionHandler\ for robust error handling, and \LauncherExitCodes\ to standardize process exit codes.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher · high confidence

The Windows winutils utility now supports chmod, chown, groups, and hardlink commands, allowing users to manage file permissions, ownership, group membership, and hard links directly on Windows. The implementation includes a new RPC-based service interface (hadoopwinutilsvc) for privileged operations, a configuration module for reading XML settings, and a static library (libwinutils) providing core file system utilities like long path handling and permission mapping.

hadoop-common-project/hadoop-common/src/main/winutils · high confidence

Node Labels management interface added to Capacity Scheduler UI

The Capacity Scheduler UI now includes a dedicated Node Labels feature, allowing administrators to view, create, and manage node labels for resource partitioning. This update adds a split-panel view: the left panel lists available labels (with support for creating new exclusive or non-exclusive labels via a dialog and removing non-default ones), while the right panel displays cluster nodes filtered by the selected label, showing their state, resource utilization, and allowing label assignment. The implementation uses Zustand with shallow selectors to optimize re-renders and includes validation for label names against YARN specifications.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/node-labels, hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/staged-changes · high confidence

NodeManager introduces extensible resource handling and secure GPU device discovery

The NodeManager now supports a pluggable resource handler architecture via the new ResourceHandlerChain, allowing modular management of container lifecycle operations (bootstrap, pre-start, teardown) and easing the addition of new resource types. Additionally, the NodeManager adds support for discovering FPGA devices through configurable strategies (script-based or settings-based) and introduces a secure GpuDeviceInformationParser that disables external DTDs and entities to prevent XML injection vulnerabilities when parsing GPU script output.

hadoop-yarn-server-nodemanager · high confidence

Promote YARN service lifecycle library to Hadoop Common

The service lifecycle library (including the Service interface, AbstractService base class, CompositeService, and state management utilities) is now available in Hadoop Common. This allows all Hadoop projects to use a standardized, thread-safe service lifecycle model with explicit state transitions (NOTINITED, INITED, STARTED, STOPPED), listener support, and quiet stop operations, rather than relying on project-specific implementations.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service · high confidence

Repository initialization and developer environment configuration

The repository has been initialized with essential configuration files to support development workflows. This includes a new .asf.yaml file to configure GitHub and Jira integration, a .gitattributes file to enforce consistent line endings across platforms, and a comprehensive .gitignore file to exclude build artifacts and IDE-specific files. Additionally, a SECURITY.md file has been added to define the project's threat model and vulnerability reporting guidelines, and an AGENTS.md file provides guidance for AI-assisted development tools. The BUILDING.txt documentation has been updated to reflect current build requirements, including JDK 17, Maven 3.9.15, and specific native library dependencies.

(repo-wide) · high confidence

Router RPC protocol supports asynchronous execution

The HDFS Router-based Federation (RBF) RPC protocol now supports asynchronous execution. This change introduces an asynchronous RPC client and server implementation, allowing the router to handle RPC calls asynchronously. This is achieved through new utility classes and protocol translators that wrap synchronous calls with asynchronous logic, improving performance and scalability for router operations.

hadoop-hdfs-project/hadoop-hdfs-rbf · high confidence

Router now supports delegation token storage in SQL and Memory modes

The YARN Router Federation component now supports storing delegation tokens in SQL and Memory stores, enabling token management without relying solely on ZooKeeper. This change introduces the \RouterDelegationTokenSupport\ class for serializing token information and integrates token secret manager lifecycle management into the Router service, allowing administrators to configure token storage backends via the FederationStateStoreFacade.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router · high confidence

Rumen tooling refactored to use Jackson JSON and new parsing infrastructure

The Rumen module has been significantly refactored to replace its previous serialization and parsing mechanisms with a Jackson-based JSON approach. This introduces new core classes for handling cluster topology and job traces, including \ClusterStory\ and \AbstractClusterStory\ for topology management, \ClusterTopologyReader\ for reading JSON-encoded topology files, and \JobTraceReader\ for job history. The \Folder\ tool now utilizes \DeskewedJobTraceReader\ to handle out-of-order job submissions and outputs results via \DefaultOutputter\. Additionally, a new \Anonymizer\ tool has been added to allow users to strip sensitive information from trace and topology files before sharing, and legacy parsers like \Hadoop20JHParser\ and \CurrentJHParser\ have been updated to fit the new event-driven architecture.

hadoop-tools/hadoop-rumen · high confidence

Template configuration and placement rules validation in Capacity Scheduler UI

The Capacity Scheduler UI now includes a template configuration feature that lets users define default properties for auto-created queues. This adds a dialog to manage template scopes (legacy leaf, flexible shared/parent/leaf) and a form to edit the properties applied to those scopes, including support for wildcard paths. Additionally, the UI introduces type definitions and Zod-based validators for placement rules, ensuring that user-defined rules for routing applications to queues are correctly structured and validated.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/template-config, hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/types/features/placement-rules · high confidence

Timeline service store implementations moved to timeline package

The Application History Service now includes in-memory and LevelDB-backed implementations of the Timeline Store (MemoryTimelineStore, LeveldbTimelineStore, RollingLevelDBTimelineStore) within the org.apache.hadoop.yarn.server.timeline package. These classes provide the underlying storage mechanisms for the Timeline Service v1.5, handling entity and event persistence, domain-based ACLs, and time-to-live (TTL) expiration via rolling LevelDB files.

hadoop-yarn-server-applicationhistoryservice · high confidence

ViewFs adds regex-based mount points and chrooted filesystem support

ViewFs now supports regex-based mount points, allowing administrators to define mount links using regular expressions with captured groups for dynamic target resolution, and introduces ChRootedFileSystem and ChRootedFs to allow mounting a specific sub-path of a base filesystem as the root. These changes are implemented via new classes (RegexMountPoint, ChRootedFileSystem, ChRootedFs) and configuration constants (linkRegex, linkFallback, linkMergeSlash) within the viewfs package, expanding the flexibility of the mount table configuration.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs · high confidence

YARN Router Federation UI displays sub-cluster metrics

The YARN Router web interface now includes a Federation block that renders a table of sub-cluster details, including state, last start time, last heartbeat, resources, and nodes. This change adds the \FederationBlock.java\ component to parse and display cluster metrics for both federated and local cluster configurations, enabling users to view operational status of individual sub-clusters directly from the Router UI.

hadoop-yarn-server-router · high confidence

YARN Web App Proxy refactored to support Application History Server and Federation

The YARN Web App Proxy has been restructured to introduce an \AppReportFetcher\ abstraction with \DefaultAppReportFetcher\ and \FedAppReportFetcher\ implementations. This change enables the proxy to fetch application reports from the Application History Server (AHS) as a fallback when the ResourceManager does not track an application, and adds support for YARN Federation by routing requests to the correct sub-cluster ResourceManagers. The proxy server now initializes these fetchers based on the configuration, allowing for more robust handling of application tracking URLs in HA and federated environments.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy · high confidence

Removals

Removal of core Hadoop common components and configuration

The \core-default.xml\ configuration file and several core classes in the \src/java\ directory—including \Configuration\, \DistributedCache\, \BlockLocation\, \ChecksumFileSystem\, \ContentSummary\, \DF\, \DU\, \FSDataOutputStream\, and \FileStatus\—have been deleted. This removes the default configuration settings and the implementation of fundamental file system utilities, caching, and data handling classes from the common library.

src/java · high confidence

Security

Router federation rename now requires permission checks

The HDFS Router now enforces permission validation before executing federation rename operations across sub-clusters. This change prevents unauthorized users from performing cross-namespace renames, addressing a security gap where such operations previously proceeded without adequate authentication checks.

hadoop-hdfs-rbf · high confidence

Architecture

Fuse-DFS implementation moved to hdfs-native-client

The fuse-dfs source code, build configuration (CMakeLists.txt), and wrapper scripts have been relocated from the previous module into the hadoop-hdfs-native-client module. This change consolidates the native HDFS client components, making the fuse-dfs filesystem mount tool part of the unified native client build process.

hadoop-hdfs-project/hadoop-hdfs-native-client · high confidence

Hadoop GCP connector relocated to hadoop-cloud-storage-project

The Google Cloud Storage connector module has been moved from the hadoop-tools directory to the new hadoop-cloud-storage-project. This relocation includes the addition of project-specific build and linting configurations, such as FindBugs and Checkstyle suppression files, alongside the core Java implementation files for the connector.

hadoop-cloud-storage-project/hadoop-gcp, hadoop-gcp · high confidence

Metrics system implementation refactored into new internal classes

The internal implementation of the Metrics System has been restructured into a new set of classes within the \org.apache.hadoop.metrics2.impl\ package. This includes new core components such as \AbstractMetricsRecord\, \MetricsRecordImpl\, and \MetricsRecordBuilderImpl\ for handling metric data structures, \MetricsCollectorImpl\ for collecting records, \MetricsConfig\ for loading and managing configuration via Commons Configuration 2, \MetricsSinkAdapter\ and \MetricsSourceAdapter\ for managing sink and source interactions, and supporting utilities like \MBeanInfoBuilder\, \MetricsBufferBuilder\, and \SinkQueue\. This refactoring centralizes the metrics infrastructure logic, replacing previous inline or scattered implementations with these dedicated classes to improve maintainability and separation of concerns.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl · high confidence

Moved ONC/RPC and portmap packages to hadoop-common

The ONC/RPC implementation classes (including RPC message handling, XDR serialization, and simple TCP/UDP clients and servers) and the portmap mapping utilities have been moved into the hadoop-common module. This change centralizes these networking primitives, making them available as a shared library for other Hadoop components like HDFS NFS Gateway rather than being embedded within them.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/oncrpc · high confidence

Behavioural changes

1175 commits (126 fixes) modifying hadoop-hdfs

A change to existing behaviour in hadoop-hdfs — 1175 commits (126 fixs), 54 files: “HDFS-5754. Split LayoutVerion into NameNodeLayoutVersion and DataNodeLayoutVersion. Contributed by Brandon Li”; “HDFS-6538. Comment format error in ShortCircuitRegistry javadoc. Contributed by David Luo. (harsh)”; “HDFS-13609. \[SBN read\] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen”; and 1158 more (refs \#4194, \#3031, \#6715, \#4419, \#5622, …).

hadoop-hdfs · medium confidence · unverified

1381 commits (190 fixes) modifying hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

A change to existing behaviour in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager — 1381 commits (190 fixs), 677 files: “YARN-4916. TestNMProxy.tesNMProxyRPCRetry fails. Contributed by Tibor Kiss”; “YARN-7664. Several javadoc errors. Contributed by Sean Mackrory”; “YARN-426. Failure to download a public resource prevents further downloads (Jason Lowe via bobby)”; and 1368 more (refs \#7748, \#5503, \#3789, \#7019, \#4687, …).

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager · medium confidence · unverified

1476 commits (185 fixes) modifying hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common

A change to existing behaviour in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common — 1476 commits (185 fixs), 710 files: “YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB”; “YARN-2107. Refactored timeline classes into o.a.h.y.s.timeline package. Contributed by Vinod Kumar Vavilapalli”; “YARN-7708. BackPort \[GPG\] Load based policy generator. (\#5902) Contributed by Young Chen”; and 1456 more (refs \#5902, \#4769, \#3793, \#5503, \#5243, …).

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common · medium confidence · unverified

2846 commits (389 fixes) modifying hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

A change to existing behaviour in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager — 2846 commits (389 fixs), 1262 files: “YARN-7664. Several javadoc errors. Contributed by Sean Mackrory”; “YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko”; “YARN-10167. FS-CS Converter: Need to validate c-s.xml after converting. Contributed by Peter Bacsko”; and 2825 more (refs \#3793, \#5896, \#5503, \#3789, \#3203, …).

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager · medium confidence · unverified

7056 commits (751 fixes) modifying hadoop-hdfs-project/hadoop-hdfs

A change to existing behaviour in hadoop-hdfs-project/hadoop-hdfs — 7056 commits (751 fixs), 1764 files: “HDFS-5754. Split LayoutVerion into NameNodeLayoutVersion and DataNodeLayoutVersion. Contributed by Brandon Li”; “HDFS-10883. 's behavior is not consistent in DFS after enabling EZ. Contributed by Yuanbo Liu”; “HDFS-9435. TestBlockRecovery\#testRBWReplicas is failing intermittently. Contributed by Rakesh R”; and 6994 more (refs \#2332, \#7120, \#3661, \#7005, \#2528, …).

hadoop-hdfs-project/hadoop-hdfs · medium confidence · unverified

991 commits (80 fixes) modifying hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api

A change to existing behaviour in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api — 991 commits (80 fixs), 403 files: “YARN-7664. Several javadoc errors. Contributed by Sean Mackrory”; “YARN-7708. BackPort \[GPG\] Load based policy generator. (\#5902) Contributed by Young Chen”; “YARN-976. Document the meaning of a virtual core. (Sandy Ryza)”; and 979 more (refs \#5902, \#4769, \#3793, \#7019, \#4687, …).

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api · medium confidence · unverified

Azure Data Lake Store input and output streams now support configurable buffering and positional reads

The Azure Data Lake Store FileSystem implementation now includes dedicated input and output stream classes that wrap the underlying Microsoft SDK streams. Users can now control the read-ahead buffer size via the \fs.adl.readahead.buffer.size\ configuration key, and the write buffer size via \fs.adl.write.buffer.size\. Additionally, the input stream supports experimental positional reads (random access reads) when the \fs.adl.experimental.positional.read\ configuration is enabled, allowing applications to read from specific file offsets without seeking the stream pointer. The output stream also implements the \Syncable\ interface, ensuring that \hflush()\ and \hsync()\ calls properly flush data to the service.

hadoop-azure-datalake · high confidence

Build tooling updates for static analysis and API documentation

The build process now includes a new FindBugs exclusion file to suppress known, intentional warnings in generated protobuf code, MapReduce components, and core IPC classes, ensuring cleaner static analysis reports. Additionally, a patch is applied to the MetricsSystem API to correct a documentation error where the register method was mislabeled as registering a sink instead of a source, and the corresponding implementation and test code are adjusted to reflect this clarification.

hadoop-common-project/hadoop-common/dev-support · high confidence

Compression codec framework refactored with new interfaces and constants

The compression module has been restructured to support modern Java features and improved resource management. A new \CodecConstants\ class centralizes default file extensions for all codecs (e.g., \.bz2\, \.lz4\, \.zst\). The \CompressionCodecFactory\ now uses \ServiceLoader\ to automatically discover available codecs, making it easier to extend the system with custom implementations. To prevent resource leaks, a new \AlreadyClosedException\ is thrown when operations are attempted on closed streams, and the \CompressionInputStream\ now automatically returns tracked decompressors to the \CodecPool\ upon closing. Additionally, new \DirectDecompressionCodec\ and \DirectDecompressor\ interfaces have been introduced to enable efficient decompression directly into Java NIO \ByteBuffer\s, bypassing intermediate byte array copies.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress · high confidence

Custom styling for the Hadoop Auth documentation site banner

The Hadoop Authentication documentation site now applies custom CSS to the banner area, specifically setting a fixed height of 93px and adjusting the margins for the left and right banner images to improve visual layout.

hadoop-common-project/hadoop-auth/src/site/resources · high confidence

FTP FileSystem configuration and behavior updates

The FTP file system now supports configurable data connection modes and transfer modes via new configuration keys (fs.ftp.data.connection.mode, fs.ftp.transfer.mode), allowing users to tailor FTP behavior to their server requirements. Additionally, the control keep-alive timeout is now configurable through fs.ftp.timeout, and the default port retrieval is correctly overridden to return the standard FTP port. The implementation also fixes a bug where the output stream was closed even when not initialized, and replaces the legacy commons-logging logger with SLF4J for consistent logging.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp · high confidence

Gridmix synthetic jobs now use the MapReduce v2 (YARN) API

Gridmix has been migrated from the legacy MapReduce v1 API to the MapReduce v2 (YARN) API. This change updates the synthetic job generation logic to use the modern Job API (e.g., Job.getInstance) and the new InputFormat/OutputFormat interfaces, replacing deprecated constructors and classes. Users running Gridmix simulations will benefit from improved compatibility with current Hadoop clusters and the YARN resource manager, ensuring that generated workloads are structured according to the supported execution framework.

hadoop-gridmix · high confidence

Hadoop 3.5.0 release landing page and site navigation

The hadoop-project site now features a new landing page (index.md.vm) for version 3.5.0, highlighting that Java 17 is required on the server side (with Java 21 supported on the client), introducing a Google Cloud Storage (GCS) FileSystem implementation, removing the deprecated WASB FileSystem in favor of ABFS, adding a modern UI for the YARN Capacity Scheduler, upgrading tests to JUnit 5, and addressing transitive CVEs through dependency upgrades. The site.xml navigation has been updated to reflect these changes, including new links for the GCS integration, the Capacity Scheduler UI, and the runC container runtime documentation.

hadoop-project · high confidence

Hadoop Archive tool (HAR) now acquires delegation tokens from source and destination file systems

The Hadoop Archives utility has been updated to automatically acquire delegation tokens from both the source and destination file systems when creating archives. This change ensures that the underlying MapReduce job can access secure or authenticated storage locations without requiring manual token handling by the user, improving compatibility with secured Hadoop clusters.

hadoop-tools/hadoop-archives · high confidence

Improved detection of IBM Java environments for authentication compatibility

A new PlatformName utility class has been added to the Hadoop authentication module to provide more robust identification of the underlying Java Virtual Machine. This change introduces logic to distinguish between IBM Java Technology Edition (which includes specific security modules like Krb5LoginModule) and IBM Semeru runtimes. By checking for the presence of specific IBM security classes rather than relying solely on the vendor string, the system can now correctly support Kerberos SPNEGO authentication on IBM JDKs and remain compatible with newer IBM Semeru JVM versions where vendor naming conventions have changed.

hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/util · high confidence

Introduce MR client-side caching and delegation for History Server connections

The MapReduce client now uses a \ClientCache\ to cache \ClientServiceDelegate\ instances and a shared \MRClientProtocol\ proxy to the JobHistoryServer, reducing the overhead of establishing new RPC connections for every job status or history query. This caching layer is wired into the \YARNRunner\ via the \YarnClientProtocolProvider\ service, ensuring that clients interacting with the History Server reuse existing connections where possible.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient · high confidence

Javadoc generation now filters by Hadoop stability annotations

The \hadoop-annotations\ module now includes new doclets (\ExcludePrivateAnnotationsJDiffDoclet\, \IncludePublicAnnotationsJDiffDoclet\, and \ExcludePrivateAnnotationsStandardDoclet\) that filter Javadoc output based on Hadoop's \InterfaceAudience\ and \InterfaceStability\ annotations. This ensures that generated documentation only includes public APIs and respects stability levels (Stable, Evolving, Unstable), excluding private, limited-private, and unstable elements as configured. The tooling supports JDK 17 and provides command-line options (\-stable\, \-evolving\, \-unstable\) to control which stability levels are included in the final documentation.

hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools · high confidence

Javadoc generation now filters by public annotations and supports JDK 17

The Hadoop annotations module now uses a custom Javadoc doclet that excludes classes and elements not explicitly marked as public (excluding private and limited-private annotations) and treats unannotated classes as private by default. This change also updates the tooling to support JDK 17, enabling the generation of aggregated Javadoc that strictly reflects the public API surface for users.

hadoop-annotations · high confidence

MapReduce history server tracking URI now supports HTTPS

The MapReduceTrackingUriPlugin in the history server plugins module has been updated to correctly generate tracking URIs when the history server is configured for HTTPS. Previously, the plugin did not respect the HTTPS configuration, potentially returning incorrect HTTP links. This change ensures that users accessing job history via the history server receive the correct protocol-specific URI (http or https) based on the server's configuration, improving connectivity for secure deployments.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins · high confidence

MiniKDC migrated to Apache Kerby with configurable realm settings

The MiniKDC component has been rewritten to use the Apache Kerby KDC backend instead of the previous Apache Directory Server implementation. This change introduces a new base test class, KerberosSecurityTestcase, for easier integration of MiniKDC into other test suites, and updates the test suite to use JUnit 5. Additionally, the KDC configuration now supports customizing the realm's organization name and domain (org.name and org.domain), allowing tests to use realms other than the default EXAMPLE.COM.

hadoop-common-project/hadoop-minikdc · high confidence

Native zlib bindings now load zlib1.dll from the Hadoop native directory on Windows

The native C implementation for ZlibCompressor and ZlibDecompressor has been updated to support Windows by dynamically loading zlib1.dll. Instead of relying on system-wide library paths, the code now attempts to locate zlib1.dll in the same directory as hadoop.dll, falling back to standard system paths if necessary. This ensures that the native compression and decompression functions work correctly in Windows Server and Azure environments without requiring manual installation of zlib.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib · high confidence

New JMX JSON servlet with NaN filtering and TRACE protection

The JMX web interface now uses a new \JMXJsonServlet\ that disables the TRACE HTTP method to prevent potential security vulnerabilities and introduces a \JMXJsonServletNaNFiltered\ variant that replaces NaN metric values with 0.0 to ensure valid JSON output. These changes improve the security posture and data reliability of the read-only JMX metrics endpoint exposed at /jmx.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx · high confidence

Protocol buffer serialization for FileStatus and snapshot status support

The new PBHelper utility class in the protocolPB package enables serialization of FileStatus objects via protocol buffers, allowing file metadata to be transmitted over the network. This change introduces support for the 'snapshot enabled' flag in ListStatus query results, ensuring that clients can now detect whether a directory supports snapshots. Additionally, owner and group names are interned during deserialization to reduce memory usage.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/protocolPB · high confidence

Refactored metrics2 utility classes and improved statistical accuracy

The metrics2 utility package has been restructured and enhanced: MBeans registration now supports additional JMX properties and logs InstanceAlreadyExistsException at trace level to reduce log noise; SampleStat now uses a weighted incremental algorithm (Welford's) for more accurate mean and variance calculations, and its MinMax helper uses Float.MAX\_VALUE instead of Double.MAX\_VALUE to prevent buffer overflows in Ganglia gmetad; new utility classes Contracts, Metrics2Util (TopN priority queue), MetricsCache, Quantile, and QuantileEstimator have been added to support programming by contract, top-N metric tracking, caching for sparse-update sinks, and streaming quantile estimation; Servers was renamed from Util and refactored to use Hadoop's NetUtils and Lists; Guava dependencies were replaced with Hadoop's shaded third-party equivalents.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util · high confidence

Refactored raw erasure coders with new state classes and native wrappers

The raw erasure coder implementation has been restructured to improve performance and code clarity. New abstract base classes, AbstractNativeRawEncoder and AbstractNativeRawDecoder, now handle native library interactions using ReentrantReadWriteLocks to protect against concurrent access and enforce the use of direct ByteBuffers for efficiency. Encoding and decoding logic is now encapsulated in dedicated state classes (ByteBufferEncodingState, ByteArrayEncodingState, ByteBufferDecodingState, ByteArrayDecodingState) which manage buffer validation and conversion. A new CoderUtil class provides shared utilities for buffer management, and a DecodingValidator has been added to verify decoding correctness. Additionally, DummyRawEncoder and DummyRawDecoder factories were introduced to allow performance isolation during testing.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder · high confidence

Regenerated proto2 source code for ProtobufRpcEngine

The tracked proto2-generated source file ProtobufRpcEngineProtos.java has been regenerated, updating the Java classes that define the RPC request header and protocol metadata used by the Protobuf RPC engine. This change ensures the generated code remains synchronized with the underlying ProtobufRpcEngine.proto definition, which is necessary for correct client-server communication in Hadoop's IPC layer.

hadoop-common-project/hadoop-common/src/main/proto2-generated · high confidence

Replace legacy dev-support scripts with Apache Yetus wrappers and new release tooling

The \dev-support/bin\ directory has been refactored to replace the previous custom test and release scripts with wrappers around Apache Yetus (defaulting to version 0.14.1) and new dedicated release automation scripts. Scripts such as \test-patch\, \smart-apply-patch\, \qbt\, \releasedocmaker\, and \shelldocs\ now delegate to \yetus-wrapper\, which handles downloading, verifying, and executing the corresponding Yetus components. Additionally, new scripts have been introduced to manage the release process: \create-release\ orchestrates the build and packaging, \dist-layout-stitching\ assembles the final distribution directory structure, \dist-tar-stitching\ creates the release archive, \dist-copynativelibs\ bundles native libraries, \dist-tools-hooks-maker\ generates tool hooks, and \verify-license-files\ validates license and notice files in the distribution artifacts. A new \checkcompatibility.py\ script has also been added to check Java API compatibility between revisions using the Java API Compliance Checker.

dev-support/bin · high confidence

Restored and modernized the mapred shell script and configuration

The \bin/mapred\ command-line interface has been resurrected and rewritten to support dynamic subcommands (such as \classpath\, \envvars\, \job\, \queue\, and \historyserver\) and improved usage reporting. This change introduces dedicated configuration scripts (\mapred-config.sh\, \mapred-config.cmd\) and environment files (\mapred-env.sh\, \mapred-env.cmd\) to manage MapReduce-specific settings, while deprecating the legacy \mr-jobhistory-daemon.sh\ script in favor of the new \mapred\ interface.

hadoop-mapreduce-project · high confidence

Rewrite of Hadoop shell scripts to a new modular framework

The Hadoop bin scripts (hadoop, hdfs, yarn, mapred, and related utilities) have been rewritten to use a new, modular shell framework. This change introduces a standardized configuration and execution flow via hadoop-config.sh and hadoop-functions.sh, replacing the previous ad-hoc script logic. Users will see improved consistency in argument parsing, environment variable handling, and error reporting across all Hadoop commands. Deprecated commands (like direct invocation of HDFS or MapReduce tools via the hadoop script) now emit clear warnings and delegate to the appropriate sub-commands (hdfs, mapred). The new framework also enhances Windows support with updated .cmd scripts and improves cross-platform compatibility by standardizing path resolution and library loading.

hadoop-common-project/hadoop-common/src/main/bin · high confidence

S3A connector introduces AWS SDK v2 exception types and retry infrastructure

The S3A filesystem connector now wraps AWS SDK v2 errors in a new hierarchy of checked exceptions (such as AWSBadRequestException, AWSNoResponseException, and AWSServiceThrottledException) and adds a dedicated Invoker class to manage AWS API calls with configurable retry policies. This change improves error handling and reliability for S3A operations by providing more granular exception types and centralized retry logic.

hadoop-tools/hadoop-aws · high confidence

S3A connector upgraded to AWS SDK v2 with new authentication and error handling

The S3A filesystem connector has been migrated from the AWS SDK v1 to v2, introducing new credential providers such as ProfileAWSCredentialsProvider for standard AWS profile support and AssumedRoleCredentialProvider for IAM assumed roles. The migration includes a new AWSClientIOException to wrap SDK errors and custom signer implementations (CustomHttpSigner, CustomSdkSigner) to support flexible authentication strategies. Core I/O classes like S3ABlockOutputStream and Listing have been rewritten to use the v2 SDK APIs, affecting how uploads, listings, and metrics are handled.

hadoop-aws · high confidence

SLS distribution and shell scripts restructured for new layout

The SLS distribution package layout has been updated to include a new \rumen2sls.sh\ script for converting Rumen traces, alongside the existing \slsrun.sh\ runner. The Maven assembly descriptor now packages the \bin\, \sample-data\, \html\, and \sample-conf\ directories into the final distribution, and the HTML assets now bundle Bootstrap v2.1.1 for the web interface.

hadoop-tools/hadoop-sls · high confidence

Serialization framework stability, visibility, and null-safety improvements

The serialization framework in hadoop-common has been updated to improve stability and API clarity. SerializationFactory now returns null instead of throwing NullPointerExceptions when a requested serialization class is missing or the configuration list is empty, preventing crashes during deserialization lookups. The default configuration has been expanded to include AvroSpecificSerialization and AvroReflectSerialization alongside WritableSerialization. Additionally, the core interfaces (Serializer, Deserializer, Serialization) and WritableSerialization have been annotated with @InterfaceAudience and @InterfaceStability to define their public/private scope and evolution guarantees, and logging has been migrated from Commons Logging to SLF4J.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/serializer · high confidence

ServiceLoader discovery for Hadoop core components

Hadoop now uses the JDK ServiceLoader mechanism to automatically discover and register implementations for key subsystems, including FileSystems, CompressionCodecs, KeyProviders, CredentialProviders, ErasureCoders, SecurityInfo, and KMS Delegation Tokens. This change replaces manual configuration with standard Java service-provider interfaces, ensuring that components like HttpFileSystem, ZStandardCodec, BCFKS keystores, and KMS token renewers are available without explicit wiring.

hadoop-common-project/hadoop-common/src/main/resources/META-INF · high confidence

ShuffleHandler refactored to use Netty 4 with optimized I/O and OS cache management

The MapReduce ShuffleHandler has been migrated to use Netty 4 for handling shuffle connections, introducing new classes like ShuffleChannelHandler and ShuffleChannelInitializer to manage the Netty pipeline. This change includes the addition of FadvisedChunkedFile and FadvisedFileRegion, which utilize POSIX fadvise calls (specifically POSIX\_FADV\_DONTNEED) to manage the OS page cache during file transfers, improving performance by reducing memory pressure. The refactoring also addresses race conditions in closing file regions and supports configurable shuffle buffer sizes and connection keep-alive settings, while maintaining backward compatibility with existing shuffle protocols.

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle · high confidence

Timeline Service v2 collector architecture refactored to per-node auxiliary service

The Timeline Service v2 collector implementation has been restructured to run as a per-node auxiliary service on the NodeManager. This introduces a new \PerNodeTimelineCollectorsAuxService\ that manages \NodeTimelineCollectorManager\ instances, which in turn handle \AppLevelTimelineCollector\ services for individual applications. The diff shows the addition of context classes (\TimelineContext\, \TimelineCollectorContext\) to encapsulate cluster, user, flow, and application metadata, and a base \TimelineCollector\ class that handles async writes and aggregation logic. This change shifts the collector lifecycle from a centralized model to a distributed, per-node model integrated into the NodeManager's auxiliary service framework.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice · high confidence

Tracing subsystem replaced with no-op implementation

The tracing infrastructure in Hadoop Common has been decoupled from the HTrace library and replaced with a no-op implementation. New classes such as Tracer, Span, TraceScope, NullTraceScope, SpanContext, and TraceConfiguration have been introduced to maintain API compatibility while effectively disabling distributed tracing. Existing code relying on these tracing interfaces will continue to compile, but tracing data will no longer be collected or propagated.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing · high confidence

WASB filesystem driver deprecated and disabled

The \NativeAzureFileSystem\ (WASB) implementation has been deprecated and now throws an \UnsupportedOperationException\ on initialization, preventing users from using the \wasb://\ or \wasbs://\ schemes. Users must migrate to the ABFS driver (\abfs://\ or \abfs://\) for accessing Azure Blob Storage. The \org.apache.hadoop.fs.azure\ package is marked as private and evolving, signaling the end of support for the legacy WASB implementation.

hadoop-tools/hadoop-azure · high confidence

Windows Secure Container Executor now supports non-privileged NodeManager execution

The Windows Secure Container Executor native implementation has been updated to allow the YARN NodeManager to run without requiring a privileged (Administrator) account. This change introduces the \windows\_secure\_container\_executor.c\ and \.h\ native source files, which implement the JNI bindings for creating and managing container processes as specific users via \RpcCall\_TaskCreateAsUser\. By handling the elevation and process creation logic within this native layer, the NodeManager no longer needs to be launched with elevated privileges, improving security posture and operational ease on Windows.

hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/yarn · high confidence

Test coverage

1 commit adding/updating tests in hadoop-common-project/hadoop-common/src/test/resources/lz4; 1 commit adding/updating tests in hadoop-common-project/hadoop-common/src/test/scripts/kms-http-benchmark; Add filesystem test base classes and helpers; Added CLI test infrastructure and helper classes; Added CLI test utility framework for command-line validation; Added HAR test resource files; Added IPC test suite and benchmarks; Added JUnit 5 test tag annotations for test classification; Added JUnit 5 tests for Hadoop tools; Added JUnit 5-based tests for ViewFileSystem components; Added KerberosTestUtils helper for authentication tests; Added SFTP contract tests for seek functionality; Added SSL security tests and test utilities; Added TFile test suite and test utilities; Added YARN server test infrastructure and test cases; Added automated jar content validation script; Added automated validation for Hadoop client JAR contents; Added comprehensive unit tests for Hadoop compression codecs and utilities; Added contract tests for Huawei Cloud OBS file system; Added contract tests for the local filesystem; Added contract tests for the raw local filesystem; Added delegation token tests using JUnit 5; Added integration tests for SFTPFileSystem; Added integration tests for shaded Hadoop client codecs and MiniCluster; Added legacy proto2 generated test sources for non-x86\_64 platforms; Added load generator test utilities; Added native tests for erasure coding and bulk CRC32; Added protobuf definitions for RPC test services and benchmarks; Added security module test suite and utilities; Added test coverage for Bloom filters and hash utilities; Added test coverage for FsShell commands; Added test coverage for Hadoop common utilities; Added test coverage for Hadoop security authorization components; Added test coverage for erasure coder implementations; Added test coverage validation script and allowlist; Added test infrastructure and coverage for erasure code components; Added test infrastructure for HA Failover Controller; Added test infrastructure for Hadoop service lifecycle; Added test resource configuration files for Hadoop unit tests; Added test resource file for ZStandard compression tests; Added test resources for Avro serialization and webapp content-type testing; Added test service provider configurations for key providers and token identifiers; Added test services for service launcher lifecycle scenarios; Added test suite for Tencent Cloud COS File System; Added test suite for the Service Launcher component; Added test utilities and comprehensive tests for Hadoop Metrics2 implementation; Added test utilities and unit tests for network topology and DNS resolution; Added tests for Avro serialization edge cases; Added tests for BZip2 block splitting and compressor/decompressor behavior; Added tests for HBase storage flow run compaction; Added tests for HTTP FileSystem path resolution; Added tests for Hadoop Key Provider components; Added tests for JMX JSON servlet behavior and configuration; Added tests for LogLevel CLI and LogThrottlingHelper; Added tests for Metrics2 utility components; Added tests for NativeIO initialization, POSIX operations, and shared file descriptors; Added tests for ProxyUserAuthenticationFilter; Added tests for RPC and DecayRpcScheduler metrics lifecycle; Added tests for Snappy compressor and decompressor edge cases; Added tests for StaticUserWebFilter configuration and behavior; Added tests for Timeline Service HBase reader web services; Added tests for WrappedIO and WrappedStatistics reflection implementations; Added tests for ZKCuratorManager SSL/TLS and transaction support; Added tests for ZStandard compressor and decompressor; Added tests for raw erasure coders; Added tests for serialization factory and Writable serialization; Added tests for the Hadoop-native MCP HTTP server framework; Added tests for the HitchHiker XOR erasure codec; Added tests for the common audit context; Added tests for the new IOStatistics API; Added unit tests for ACL and FsPermission functionality; Added unit tests for Capacity Scheduler UI store slices; Added unit tests for CredentialShell and CredentialProvider components; Added unit tests for DataBlocks, EtagChecksum, and FSBuilderSupport; Added unit tests for FPGA cgroups isolation; Added unit tests for FTPFileSystem configuration and behavior; Added unit tests for GPU isolation and cgroups configuration in the native container executor; Added unit tests for Ganglia sink socket and multi-server configuration; Added unit tests for Hadoop Configuration subsystem; Added unit tests for Hadoop HttpServer2 functionality; Added unit tests for Hadoop authentication client components; Added unit tests for Hadoop authentication handlers; Added unit tests for Hadoop metrics2 library components; Added unit tests for Hadoop retry policies and failover proxy logic; Added unit tests for Hadoop shell script functions; Added unit tests for JvmMetrics, JvmPauseMonitor, and GcTimeMonitor; Added unit tests for KMS client provider delegation token selection and load balancing configuration; Added unit tests for Lz4 compressor and decompressor; Added unit tests for Metrics2 pattern filter behavior; Added unit tests for OpenSSL and OS secure random implementations; Added unit tests for S3A prefetching components; Added unit tests for UNIX domain socket functionality; Added unit tests for filesystem implementation utilities; Added unit tests for hadoop-auth authentication utilities; Added unit tests for hadoop-common functional and dynamic utility classes; Added unit tests for security HTTP filters; Added unit tests for the Hadoop \hadoop fs -find\ command expressions; Added unit tests for the ONCRPC and portmap packages in Hadoop Common; Added unit tests for the native container executor's device isolation module; Added unit tests for token utilities and secret manager configuration; Added unit tests for web-based delegation token authentication; Expanded filesystem contract test coverage; Expanded test coverage for Hadoop crypto streams and codecs; Expanded test coverage for Hadoop metrics2 sinks; Expanded unit test coverage for Hadoop IO classes; New JUnit 5 test infrastructure and utilities in hadoop-common.

Dependencies

2022 commits updating dependencies (128 manifests)

A dependency / build maintenance change in (dependencies) — 2022 commits (241 fixs), 128 files: “YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB”; “HADOOP-13812. Upgrade Tomcat to 6.0.48. Contributed by John Zhuge”; “HADOOP-18949. upgrade maven dependency plugin due to [CVE redacted]. (\#6219)”; and 2001 more (refs \#6219, \#4700, \#3963, \#3031, \#3936, …).

(dependencies) · high confidence · unverified

Housekeeping

Add package-info.java for filecache module; New and updated YARN site documentation for schedulers, Docker, and device plugins.

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

How this codebase got here

Score

  • CAI 40 → 43 (+3.4)
  • Rubric changed (rubric-2026.10.5 → rubric-2026.11.3) — scores are not directly comparable.

Lenses

  • Code Health 61 → 61 (-0.4)
  • Architecture 43 → 55 (+12.0)
  • Maturity 53 → 58 (+5.1)
  • Readiness 34 → 40 (+6.4)
  • Security 64 → 70 (+6.3)
  • Domain Modelling 98 → 98 (+0.0)
  • Accessibility 36 → 34 (-2.1)
  • Performance 80 → 91 (+11.4)

Resolved (275)

  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Duplicated block (10 lines × 2) (hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java)
  • Duplicated block (10 lines × 2) (hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java)
  • Duplicated block (10 lines × 2) (hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Job20LineHistoryEventEmitter.java)
  • Duplicated block (10 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java)
  • Duplicated block (10–11 lines × 2) (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java)
  • Duplicated block (10–11 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java)
  • Duplicated block (10–11 lines × 4) (hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java)
  • Duplicated block (10–12 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet2/HamletGen.java)
  • Duplicated block (10–13 lines × 2) (hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Job20LineHistoryEventEmitter.java)
  • Duplicated block (11 lines × 2) (hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java)
  • Duplicated block (11 lines × 2) (hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java)
  • Duplicated block (11 lines × 2) (hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java)
  • Duplicated block (11 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java)
  • Duplicated block (11 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java)
  • …and 255 more

New (2046)

  • AppsBlock.fetchData (cyclomatic 16) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java)
  • ClassTooLong: QueueConfigResolver (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/resolver/QueueConfigResolver.java)
  • ClassTooLong: QueueProperties (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/resolver/QueueProperties.java)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Critical CVE: [GHSA redacted] (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock)
  • Duplicated block (10 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java)
  • Duplicated block (13 lines × 2) (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java)
  • Duplicated block (7 lines × 3) (hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java)
  • Duplicated block (9 lines × 2) (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/resolver/ResolvedQueueConfig.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/utils/BosCRC32CCheckSum.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SnapshotCommands.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MD5Hash.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ExtendedBlockId.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java)
  • Edited copies of one member (3 members, 10+ corresponding lines) (hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java)
  • …and 2026 more

Changes since last survey

  • 38 commits — 33 feature/other, 5 fixes

By area

  • hadoop-yarn-project/hadoop-yarn — 13 commits
  • hadoop-common-project/hadoop-common — 7 commits
  • hadoop-hdfs-project/hadoop-hdfs — 7 commits
  • (root) — 4 commits
  • hadoop-mapreduce-project/hadoop-mapreduce-client — 3 commits
  • .github/workflows — 2 commits
  • hadoop-hdfs-project/hadoop-hdfs-native-client — 1 commit
  • hadoop-tools/hadoop-distcp — 1 commit

Notable commits

  • fix: HADOOP-19782. Snappy security fixes CVE 2026 108106 (and [CVE redacted], [CVE redacted]) (#8900). Contributed by Jose Angel Riarola
  • fix: HADOOP-20031. Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI (#8844)
  • fix: HADOOP-20035. Upgrade JLine to 3.30.14 to fix (unreachable) [CVE redacted] and [CVE redacted] (#8734)
  • fix: YARN-11991. Fix counting virtual memory in cgroup v2 (#8764)
  • fix: YARN-12039. Fix container report lookup for retained containers (#8854)
  • change: HADOOP-17611. Preserve DistCp file times after chunk concatenation. (#8858). Contributed by Xinyu Tan
  • change: HADOOP-18896. Large file.bytes-per-checksum cause NegativeArraySizeException in FSOutputSummer (#8622)
  • change: HADOOP-20000. Remove the JAX-RS 1.1 API from hadoop-yarn-server-timelineservice-hbase-tests. (#8940)
  • change: HADOOP-20014. Run the Jersey tests on the JDK HTTP server instead of Jetty. (#8892)
  • change: HADOOP-20015. Stop using Jetty 9.4 APIs that Jetty 12 removes (#8893)
  • change: HADOOP-20032. Bump Jackson to 2.18.12 due to CVEs (#8848). Contributed by PJ Fanning
  • change: HADOOP-20033. Upgrade lz4-java 1.12.0 due to CVEs (#8849). Contributed by PJ Fanning
  • change: HADOOP-20037. Fix typos in Common documentation and configuration descriptions (#8866). Contributed by Yuan Luo
  • change: HADOOP-20040. SecurityUtil host resolver cache shares one mutable InetAddress per IP literal, leaking reverse lookups to other callers. (#8880)
  • change: HADOOP-20045. Avoid global write-lock contention during ZooKeeper token creation and add a KMS HTTP benchmark (#8888)
  • change: HADOOP-20047. RetryInvocationHandler leaves the call id set when an attempt issues no RPC (#8902)
  • change: HADOOP-20050. Bump expired GitHub Actions pins to ASF-allowlisted versions and add Dependabot (#8906)
  • change: HADOOP-20056. Replace the pull_request_target Build check bridge with ASF fork-ci (#8912)
  • change: HADOOP-20057. Remove leftover Log* JVM metrics and EventCounter references after HADOOP-17524 (#8921)
  • change: HADOOP-20067. RawLocalFileSystem: broken double-checked locking in LocalFSFileInputStream#getAsyncChannel() can leak AsynchronousFileChannels (#8931) Contributed by Jie Yang
  • …and 18 more

Architecture

  • Unchanged — 0 containers · 39 contexts · 71 edges

Written by cca.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Survey your own repository

apache/hadoop was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.

About this page

  • The score is its most recent published measurement, taken on 13 November 2026 at a pinned commit. It is not a live figure and does not change until the project is measured again.
  • Measured at commit 904cde0c848781e67dde0b0aece8acad2e320204 — the exact code this score is about.
  • Scored under rubric-2026.11.3 — the same rubric and the same method as every other entry in this index.
  • Measured by Canine Code Analyzer (cca.canine.dev) using cca measure-2026.11.61+preprod-f3449e6eb431.