discoproject/disco

Disco is a distributed computing framework that manages cluster operations, job scheduling, and data processing. It provides a Python-based CLI and web interface for controlling the master and nodes, while supporting MapReduce and pipeline-based job execution across various storage backends like HDFS, S3, and Redis. The system includes a distributed filesystem (DDFS) for data management and offers deployment automation for cloud environments.

10 changes.

This period established the foundational architecture of the Disco project, replacing legacy Python and shell-based build systems with a standardized Erlang/Rebar build process and C bindings. The team implemented core utility modules, configuration constants, and a new Sphinx-based documentation structure to support the framework's development.

15 changes.

This period focused on introducing the DDFS distributed file system and a comprehensive set of command-line interfaces for cluster and data management. The team also reorganized the Python library structure to support multiple input schemes and improved Python 3 compatibility across the codebase.

13 changes.

This period focused on stabilizing the codebase through comprehensive test coverage and introducing robust packaging for Debian and RPM distributions. It also expanded deployment options with Docker, systemd, and cloud automation scripts, while providing practical examples for complex workflows like job chaining and MapReduce algorithms.

The ext directory now includes disco.c and disco.h, providing the C-language bindings for the external interface. This adds the core logic for reading and writing key-value pairs and parameters via stdin/stdout, enabling the external job processing to function correctly.

ext · high confidence

Users can now build and run Disco inside a Docker container. This includes a Dockerfile that sets up an Ubuntu-based image with SSH and supervisor, a supervisor configuration to manage the SSH and Disco services, and a README with instructions for building and running the image.

contrib/docker · high confidence

A new example has been added to the IPython examples directory, demonstrating how to perform a map-reduce job to count letter frequencies from a Wikipedia page and visualize the results using matplotlib and scipy.

examples/ipython · high confidence

Added a new example in the \examples/tsp\ directory that demonstrates a brute-force Map/Reduce solution to the Travelling Salesman Problem. The change includes the Python implementation (\travelling\_salesman.py\) which uses the \mrjob\/\disco\ framework to parallelize the search for the shortest tour, along with corresponding JSON test cases (\example\_9\_nodes.json\ through \example\_13\_nodes.json\) that provide graph data and starting nodes for the algorithm.

examples/tsp · high confidence

A new example file, examples/worker/simplejob.py, introduces a SimpleJob class that demonstrates a basic map/reduce pattern. The map function outputs 'hello world!' and the reduce function outputs 'goodbye world!', providing a concrete reference for implementing simple job logic.

examples/worker · high confidence

Users can now deploy Disco clusters on Google Cloud and Amazon Web Services using new automation scripts. For Google Cloud, the setup process creates a master and specified number of slave nodes using f1-micro instances, installs Disco, and configures SSH access. For AWS, the scripts create a Disco AMI and launch nodes, with a README providing setup instructions. The Google Cloud scripts also include a delete\_all script to clean up resources.

cloud · high confidence

The clx library has been added to the project, providing a decorator-based command-line interface framework with automatic help generation from docstrings, a built-in settings mechanism that supports file and environment variable overrides, and utilities for managing PID-based server processes (start, stop, restart, status).

lib/clx · high confidence

The master/src directory now includes config.hrl, which defines global constants for time intervals (SECOND, MINUTE, HOUR, DAY), memory limits (MB, MAX\_JSON\_POST, MAX\_JOB\_PACKET), HTTP connection limits, and task failure pause durations. Additionally, new modules are introduced: disco.erl provides core utility functions for job paths, URLs, and host management; disco\_profile.erl integrates Folsom for application profiling; disco\_proxy.erl manages the reverse proxy (Lighttpd/Varnish) for the cluster; disco\_util.erl offers common utilities like timestamp formatting and random selection; and disco\_web.erl handles the HTTP API endpoints for job control and status. A custom zip module (discozip.erl) is also added to handle jobpack extraction.

master/src · high confidence

Added three new data mining examples to the \examples/datamining\ directory: a k-clustering implementation using both classic workers and pipelines, a naive Bayes classifier using MapReduce, and an iterative PageRank algorithm. These scripts demonstrate distributed data processing patterns for clustering, classification, and graph analysis.

examples/datamining · high confidence

A new systemd service file (disco.service) has been added to the contrib/systemd directory. This allows users to manage the Disco application as a background service, with an environment variable (systemd) now being set to enable it.

contrib/systemd · high confidence

Added several new example scripts in the \examples/util\ directory to demonstrate common distributed computing patterns and integrations. These include \chunk.py\ for implementing chunking with Job classes, \wordcount.py\ and \wordcount\_ddb.py\ for basic and Discodb-backed word counting, \grep.py\ and \dgrep\ for text filtering, \estimate\_pi.py\ for Monte Carlo estimation, \simple\_innerjoin.py\ for CSV joins, and readers for XML (\xml\_reader.py\), Wikipedia (\wiki\_reader.py\), Discodb (\query\_ddb.py\), HDFS (\wordcount\_hdfs.py\), and Redis (\wordcount\_redis.py\, \wordcount\_redis\_pipe.py\).

examples/util · high confidence

The contrib directory now includes several new submodules: 'odisco', 'discodb', 'discodex', and 'java-ext'. These are linked to specific commits of external repositories, making them available as part of the project's contrib ecosystem.

contrib · high confidence

The project introduces a complete Debian packaging system for Disco, including build rules, control files, and maintainer scripts. This enables the creation of .deb packages for the Disco master, Disco node, and Python client libraries (both Python 2 and Python 3). The packaging ensures that the Disco service runs under a dedicated 'disco' user, manages the user and group creation/removal, and handles service lifecycle (start/stop/restart) via init scripts.

pkg/debian · high confidence

The web interface now includes dedicated CSS files (reset-fonts-grids.css and style.css) that define the visual layout and styling for the UI components. This includes styling for the header, node boxes (with specific styles for connected/disconnected states and disk usage meters), and the job list (with color-coded states for active, dead, and ready jobs).

master/www/css · high confidence

The documentation system has been migrated to use Sphinx, introducing a new build infrastructure via a Makefile and configuration file (conf.py). This change enables automated generation of HTML, PDF, and man pages from reStructuredText sources. The initial set of documentation pages includes the main index, an introduction, a technical overview, a glossary, a FAQ, and release notes, all structured within the doc directory.

doc · high confidence

The DDFS (Distributed Data File System) module is introduced, providing a complete distributed storage and retrieval system. This includes the core Erlang interface (ddfs.erl) for tag and blob operations, a comprehensive configuration header (config.hrl) defining system parameters like timeouts and limits, and the full garbage collection and replication logic (ddfs\_gc, ddfs\_gc\_main, ddfs\_gc\_node) to manage data lifecycle and node health. Additionally, HTTP handling (ddfs\_http, ddfs\_get) is added to serve files and proxy requests.

master/src/ddfs · high confidence

The repository was bootstrapped with a new Makefile that manages building, testing, and installing Disco components (master, node, and examples) using Rebar for Erlang and standard Python tools. This includes setting up submodules for related projects (Java extensions, Odisco, Discodb, Discodex), configuring Travis CI for Erlang 17.4–18.2.1, and establishing a standard GNU-style installation layout (prefix, sysconfdir, etc.). The previous lighttpd configuration was removed.

(repo-wide) · high confidence

Added new Python-based command-line utilities for interacting with the system. The \ddfs\ command provides a CLI for managing data in the distributed filesystem, supporting operations such as chunking, blob listing, attribute management, and file concatenation. The \disco\ command offers a CLI for cluster management, including starting, stopping, and checking the status of the master, as well as submitting jobs and viewing job history and events.

bin · high confidence

Added a new pipeline worker implementation that allows users to define a sequence of processing stages, each with optional grouping, custom input/output chains, and the ability to specify concurrent execution. The worker parses job dictionaries to construct a pipeline of stages, supporting features like sorting and combining tasks before each stage, and enables users to control the order and concurrency of stage execution.

lib/disco/worker/pipeline · medium confidence

The web interface JavaScript has been reorganized into dedicated modules (config, disco, job, joblist, status) and moved to the master/www/js directory. This includes adding a DDFS (GC) blacklist to the config page, displaying GC status and stats on the status page, and exposing a 'Start GC' action in the UI. The jQuery library has been updated to version 1.9.1, and the typewatch library has been updated to version 2.1. Additionally, the jquery.uitableedit plugin was moved to the new directory structure.

master/www/js · high confidence

Added two new example scripts to the FAQ section: chain.py demonstrates how to implement job chaining by importing job classes from a separate module to avoid \_\main\\_ limitations, while profile.py provides a template for enabling profiling in Disco jobs.

examples/faq · high confidence

Disco now supports reading from HDFS, Amazon S3, Redis, and local directories in addition to the existing file, HTTP, HTTPS, raw, and disco schemes. The \scheme\_hdfs.py\ module enables reading from Hadoop Distributed File System via HTTP endpoints, \scheme\_s3.py\ adds support for Amazon S3 buckets using the \boto\ library, \scheme\_redis.py\ allows using Redis as an input source, and \scheme\_dir.py\ enables reading from local directories. These new schemes are automatically discovered and used based on the URL scheme provided in job inputs.

lib/disco/schemes · high confidence

The web interface now includes dedicated HTML templates for system configuration, cluster status, and individual job details. The configuration page allows administrators to manage available nodes, maintain blacklists for Disco and DDFS, and adjust miscellaneous settings like max failure rates and job acceptance. The status page displays node information and DDFS garbage collection (GC) statistics and status. The job page provides detailed information about a specific job, including its status, pipeline stages, current nodes, inputs, results, and event logs, along with actions to copy the job name, kill the job, or purge its data.

master/www · high confidence

The build system now supports platform-specific configuration for installing Disco on Linux, Darwin (macOS), Debian, FreeBSD, NetBSD, and OpenBSD. A shared settings file (gen.settings.sh) defines common defaults, while OS-specific scripts (gen.settings.sys-\*) source it and add or override settings (e.g., Debian user). This enables 'make install' to correctly set paths and variables for each platform.

conf · high confidence

The legacy Python modules \py/disco.py\, \py/disco\_test.py\, and \py/disco\_worker.py\ have been removed from the codebase. This change eliminates the previous implementation of the Disco framework's core logic, worker processes, and associated tests, likely as part of a larger restructuring or migration to a new architecture.

py · high confidence

The \lib/disco\ package has been reorganized into a new modular structure, introducing a \compat\ module to handle Python 2/3 differences (such as \bytes\/\str\ splits and import path changes). This refactor includes the creation of new files for the CLI (\cli.py\), core logic (\core.py\), and communication layers (\comm.py\, \comm\_pycurl.py\), alongside updates to the distributed filesystem client (\ddfs.py\) and error handling (\error.py\).

lib/disco · high confidence

The \lib/disco/worker\ package has been reorganized into a modular structure, introducing a new \disco.worker\ package containing \\_\init\\_.py\, \modutil.py\, \simple.py\, and \task\_io.py\. This refactoring extracts I/O utilities into \task\_io.py\, moves module dependency resolution logic into \modutil.py\, and introduces a \SimpleWorker\ in \simple.py\ that provides a straightforward map/reduce interface. These changes support the new pipeline task model and improve the separation of concerns within the worker execution environment.

lib/disco/worker · high confidence

The classic Disco worker has been restructured into a self-contained module under \lib/disco/worker/classic\, separating the core worker logic, function definitions, and external interface into distinct files (\worker.py\, \func.py\, \external.py\). This refactoring includes significant Python 3 compatibility updates, such as handling bytes/string splits, updating import paths, and fixing exception syntax. The \netstring\ utility has also been moved to this location and updated to use modern string formatting and exception handling.

lib/disco/worker/classic · high confidence

A new layout template for the documentation has been introduced, which repositions the sidebar to its correct location and adds a link to the Disco Project homepage in the root breadcrumb navigation.

doc/.templates · medium confidence

The codebase now includes several new header files in the master/include directory that define core types and records for the system. common\_types.hrl defines generic types such as host, path, url, and various collection types (disco\_dict, disco\_gbtree, etc.). ddfs.hrl introduces types for disk info, volumes, and object types. disco.hrl defines node information and port mapping types. fair\_scheduler.hrl adds types for scheduler messages and load statistics. gs\_util.hrl provides utilities for gen\_server callbacks. job\_coordinator.hrl introduces records for stage and task information, including fields for tracking running tasks, failures, and outputs. pipeline.hrl defines types for pipeline stages, task specifications, and job information, including fields for save\_results and save\_info. These headers support the job coordination and pipeline execution logic.

master/include · medium confidence

The project has switched from a custom shell-based build process to Rebar for dependency management and compilation. This change introduces a new rebar.config file that explicitly defines dependencies on lager (v2.0.3), folsomite, plists, and mochiweb (v2.9.2), alongside test dependencies like triq. The legacy compile.sh and start\_disco.sh scripts have been removed, indicating a shift in how the Disco master is built and started, with the new build system handling the compilation and dependency resolution automatically.

master · high confidence

Added a setup.py file for the lib directory, defining the package structure and metadata for the 'disco' project, including its version, description, and URL.

lib · high confidence

The legacy web interface for the control panel has been removed. This includes the deletion of the main HTML file (index.html) which previously provided a table-based configuration interface, along with the associated jQuery 1.2.1 library and the Yahoo! CSS reset/grid styles. Users will no longer have access to this specific web-based configuration method.

ctrl · high confidence

The documentation now uses a custom stylesheet (sphinxdoc.css) that aligns the visual appearance of the docs with the main Discoproject.org site, including specific styling for code blocks, navigation headers, and typography.

doc/.static · high confidence

Added RPM packaging specification for Disco; Added test suites for DDFS tags, JSON validation, and worker protocol; Added unit tests for fileutils, pickle, and transport layers; Comprehensive test suite migration and expansion.

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 37 → 39 (+2.0)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 65 → 64 (-0.8)
• Architecture 99 → 100 (+0.7)
• Maturity 43 → 60 (+17.1)
• Readiness 27 → 34 (+6.7)
• Security 66 → 45 (-20.4)
• Event-Driven 39 → 39 (+0.0)
• Event Sourcing 100 → 100 (+0.0)
• Accessibility 47 → 38 (-8.8)

• Change coupling: ddfs_get.erl ↔ ddfs_put.erl (master/src/ddfs/ddfs_get.erl)
• Change coupling: fair_scheduler_fair_policy.erl ↔ fair_scheduler_fifo_policy.erl (master/src/fair_scheduler_fair_policy.erl)
• Change coupling: fair_scheduler_fifo_policy.erl ↔ fair_scheduler_job.erl (master/src/fair_scheduler_fifo_policy.erl)
• Coverage not included — suite not readable by the collector
• Dependency hygiene not measured — dependency manifest found but not parsed for hygiene
• High: security finding (details withheld)
• High: security finding (details withheld)
• Medium IaC: CKV_DOCKER_1 (contrib/docker/Dockerfile)
• Medium IaC: CKV_DOCKER_4 (contrib/docker/Dockerfile)
• Medium IaC: CKV_DOCKER_6 (contrib/docker/Dockerfile)
• Medium IaC: CKV_DOCKER_7 (contrib/docker/Dockerfile)
• Medium IaC: DS-0013 (contrib/docker/Dockerfile)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• Medium: security finding (details withheld)
• No exposed public API
• Release notes are excellent but only cover the 0.5.x series; there is no dedicated architecture or usage document for the newer Disco versions. (doc/releases.rst)
• Test reliability not included
• TooManyMethods: ddfs_gc_main (master/src/ddfs/ddfs_gc_main.erl)
• TooManyMethods: ddfs_tag (master/src/ddfs/ddfs_tag.erl)
• …and 4 more

• Coverage not measured — no coverage collector is wired up
• Dependency hygiene PARTLY measured — rebar3 pinning read, dependency currency not (no rebar.lock-pinned Hex declaration to grade)
• Documentation: no contributor guidance (README.md)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (10–16 lines × 2) (master/src/discozip.erl)
• Duplicated block (15 lines × 2) (master/src/discozip.erl)
• Duplicated block (5 lines × 2) (master/src/ddfs/ddfs_gc_main.erl)
• Duplicated block (5 lines × 3) (master/src/ddfs/ddfs_master.erl)
• Duplicated block (7 lines × 2) (master/src/ddfs/ddfs_web.erl)
• Duplicated block (7 lines × 2) (master/src/jobpack.erl)
• Duplicated block (8 lines × 2) (master/src/event_server.erl)
• Duplicated block (9 lines × 2) (master/src/discozip.erl)
• FileTooLong: js/jquery-1.9.1.js (master/www/js/jquery-1.9.1.js)
• FixmeComment (master/src/json_validator.erl)
• Floating source dependency: folsomite
• Floating source dependency: plists
• HackComment (examples/datamining/kclustering.py)
• HackComment (examples/datamining/kclustering_pipeline.py)
• High IaC: WD-DOCKER-0002 (contrib/docker/Dockerfile)
• …and 92 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

discoproject/disco was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.