TianLangStudio/DataXServer

Data Hamal is a distributed data processing system that leverages the Alibaba DataX engine to execute and manage data synchronization tasks. It provides multiple interfaces for task submission and monitoring, including HTTP REST endpoints, a Thrift RPC service, and native YARN integration for cluster resource management. The platform standardizes task execution through a refactored core engine that supports JSON-based configurations and robust error handling, replacing legacy XML and Thrift-only architectures.

The common module now includes a new set of shared classes for the Hamal system, including constants for task status and configuration, a TaskCost utility for calculating task duration, and a TaskResult class for standardizing operation outcomes. Additionally, a renamed DataHamalException class (moved from the DataX exception package) and an ExceptionUtil helper are provided to streamline error handling and logging across the application.

common · high confidence

Introduces a new HTTP server built on Akka HTTP that exposes REST endpoints for managing data processing tasks. Users can submit tasks via POST requests to the /task endpoint, retrieve task results, check task status, and view execution costs via GET requests. The server is configurable via application.conf (defaulting to port 9808) and integrates with the existing task handler infrastructure to process jobs.

httpserver · high confidence

The system now exposes a Thrift RPC interface for submitting and managing tasks. This change adds the Thrift service definition, generated Java client/server code, and a handler that delegates task operations (submit, status, result, cost, cancel) to the existing internal task handler. The application entry point has been refactored to initialize and run this Thrift server, replacing the previous main class and removing external metrics dependencies in favor of standard logging.

thriftserver · high confidence

Users can now run the system in a standalone Yarn mode that supports local multi-process execution via a new LocalApplicationMaster. This mode introduces explicit Akka configuration (akka\_config.conf) to prevent runtime config-key errors caused by classpath conflicts, and updates the Yarn client to submit applications as 'datax-server' with improved logging and dynamic command construction. The change also migrates internal utilities to use HamalConf and modern Scala collection converters, and adds a new task ID generation mechanism.

hamal-yarn/src/main · high confidence

The default configuration files for the DataX engine, logging, and plugins have been removed from the repository. Specifically, \conf/engine.xml\ (which defined storage and scheduling settings), \conf/log4j.properties\ (which configured logging appenders), and \conf/plugins.xml\ (which listed available readers and writers such as MySQL, Oracle, HDFS, and HBase) are no longer present. Users relying on these default configurations will need to provide their own configuration files or update their deployment to accommodate the absence of these defaults.

conf · high confidence

This change removes the legacy Thrift-based server implementation and its associated components from the codebase. Specifically, it deletes the \ThriftServerHandler\ and \AkkaThriftServerHandler\ classes, the \ApplicationMaster\ implementations for both local and YARN execution modes, and the \ThriftServer.thrift\ interface definition. It also removes the \HelloClient\ and \HelloServiceImpl\ demo services, the \log4j.properties\ configuration files, and the entire Maven site documentation structure (including APT, FML, and XDoc files). This cleanup eliminates the old job submission and execution infrastructure in favor of the current architecture.

src/main · high confidence

The core execution engine has been refactored to accept task configurations directly as strings or file paths, replacing the previous requirement for XML-based job definitions. The new Engine implementation parses JSON-like configuration content, dynamically resolves plugin dependencies (readers, writers, handlers), and includes a retry mechanism for plugin loading failures. Task submission now validates configuration content immediately, and the Worker and Task components have been updated to track metrics and manage lifecycle using the new string-based configuration model instead of the legacy XML parsing approach.

core · high confidence

The \JobScheduler\ has been renamed to \TaskScheduler\ and the \Executor\ component has been moved to the \hamal-yarn\ module, reflecting a shift from a generic job-processing model to a task-oriented architecture integrated with YARN. This change updates internal state management (e.g., \jobId\ to \taskId\), renames message types (e.g., \SubmitJob\ to \SubmitTask\), and modifies the \Engine\ initialization to no longer rely on XML configuration parsing, instead using a default constructor. The \ApplicationMaster\ is introduced to handle YARN-specific resource allocation and container management, while the executor now connects to the new \TaskScheduler\ via Akka actors.

repository · high confidence

The JobCounterMetricSet class has been renamed to TaskCounterMetricSet and moved from the esun.fbi.datax.ext.monitor package to org.tianlangstudio.data.hamal.common.monitor. This change updates the class name and package structure to better reflect its role in monitoring tasks within the Hamal common module, while preserving its existing functionality for managing metric counters.

hamal-common · high confidence

This change introduces a comprehensive set of tests for the hamal-yarn module, including a new ClientTest that validates the Yarn client against a MiniYARNCluster, and updated ThriftServerTest cases that verify task submission and result retrieval via the Thrift interface. It also includes ConfigTest to ensure configuration fallback logic works correctly, ConfigUtilTest for placeholder replacement, and moves various test resources (application.conf, master.conf, logback-test.xml) into the test scope to support these validations.

hamal-yarn/src/test · high confidence

The project has been rebranded as Data Hamal (version 2.0.0-SNAPSHOT) and restructured into five Maven modules: common, core, hamal-yarn, httpserver, and thriftserver. This update upgrades the Scala runtime from 2.11 to 2.13.9, Akka from 2.3.11 to 2.5.25, and Thrift from 0.10.0 to 0.14.0. It also migrates the DataX engine from the legacy com.taobao.datax artifacts to the modern com.alibaba.datax suite (datax-common, datax-core, datax-transformer) and upgrades Hadoop from 2.6.1 to 3.4.0. Additionally, Logback is updated to 1.5.25, JUnit to 4.13.1, and dom4j to 2.1.3, while the build configuration now imports Spring Boot 2.1.7.RELEASE for dependency management.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 51.

• Code Health 97
• Architecture 66
• Maturity 55
• Readiness 33
• Security 89

• 63 commits — 59 feature/other, 4 fixes

• (root) — 24 commits
• (repo) — 13 commits
• src/main — 12 commits
• hamal-yarn/src — 5 commits
• common/pom.xml — 2 commits
• core/src — 2 commits
• server/src — 2 commits
• common/src — 1 commit
• conf/engine.xml — 1 commit
• thriftserver/src — 1 commit

• fix: add scalatest dependency httpserver fixed compile error
• fix: fixed akka no config key error
• fix: fixed httpserver submit with job file name but not config content
• fix: upgrade version of dom4j,scala and thrift to fixed security alerts
• change: Bump ch.qos.logback:logback-classic from 1.2.3 to 1.2.13
• change: Bump ch.qos.logback:logback-core from 1.2.3 to 1.3.15
• change: Bump ch.qos.logback:logback-core from 1.3.15 to 1.5.25
• change: Bump dom4j from 2.1.1 to 2.1.3
• change: Bump junit from 4.12 to 4.13.1
• change: Bump libthrift from 0.12.0 to 0.14.0
• change: Bump org.apache.commons:commons-lang3 from 3.9 to 3.18.0 in /common
• change: Bump org.apache.hadoop:hadoop-common from 2.6.1 to 3.4.0
• change: Bump scala-library from 2.13.0 to 2.13.9
• change: Delete READEME.md for typo
• change: Initial commit
• change: Merge branch 'HttpServer'
• change: Merge branch 'master' of https://github.com/TianLangStudio/DataXServer
• change: Merge branch 'master' of https://github.com/TianLangStudio/DataXServer
• change: Merge pull request #11 from TianLangStudio/dependabot/maven/org.dom4j-dom4j-2.1.3
• change: Merge pull request #13 from TianLangStudio/dependabot/maven/junit-junit-4.13.1
• …and 43 more

• 0 containers · 2 bounded contexts · 0 dependency edges (baseline)

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

TianLangStudio/DataXServer was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.