nsqio/nsq

This system is NSQ, a real-time distributed messaging platform designed for high-throughput message passing. It consists of core daemons for message routing and topology management, alongside a suite of command-line utilities for monitoring, data ingestion, and message forwarding. The system also includes a web-based administrative interface for cluster visualization and configuration, supported by internal libraries for protocol handling, metrics, and testing.

12 changes.

This period established the foundational architecture of NSQ by implementing the core nsqd broker, the nsqlookupd topology daemon, and the nsqadmin interface. It also introduced a suite of utility applications for monitoring, message forwarding, and performance benchmarking to support the new distributed messaging system.

25 changes.

This period focused on modernizing the project's architecture by introducing dedicated service entry points with TOML configuration and go-svc lifecycle management for all core daemons. Simultaneously, the nsqadmin interface was refactored into a Backbone.js single-page application with Bootstrap 3 styling, while internal infrastructure was expanded with new protocol, authentication, and utility packages to support these changes.

4 changes.

This period focused on enhancing internal tooling and build processes, introducing a new leveled logging package and buffered writers for controlled output. It also involved updating test certificates for TLS verification and checking in pre-compiled static assets for nsqadmin to simplify UI rendering.

The contrib directory now includes an RPM spec file (nsq.spec) for building NSQ packages and example configuration files (nsqd.cfg.example, nsqlookupd.cfg.example, nsqadmin.cfg.example) in TOML format. These files provide a reference for packaging the distribution and configuring the core services with options such as TLS settings, HTTP timeouts, and statsd integration.

contrib · high confidence

Users can now pass multiple values for float and string parameters via comma-separated command-line flags. The new FloatArray type parses comma-delimited strings into sorted slices of floats, while the StringArray type collects comma-delimited strings into a slice, enabling easier configuration of list-based settings without requiring multiple flag invocations.

internal/app · high confidence

The internal/stringy package now includes new helper functions: Add and Union for managing string slices, an optimized Uniq function to remove duplicates, and NanoSecondToHuman for converting nanosecond values into human-readable time units (ns, us, ms, s).

internal/stringy · high confidence

A new Python 3 script (bench/bench.py) has been added to automate the setup and execution of distributed NSQ benchmarks on AWS EC2. The script provisions EC2 instances, bootstraps the NSQ codebase (cloned from nsqio/nsq), builds the nsqd and benchmark binaries, and runs producer/consumer workloads across the cluster. It relies on boto3 for AWS interaction and paramiko for remote command execution, with configuration managed via tornado.options.

bench · high confidence

A new benchmark utility has been added at bench/bench\_channels to stress-test NSQ by creating and subscribing to a configurable number of channels (default 10,000). The tool connects to an nsqd instance via TCP, performs the NSQ v2 protocol handshake (including IDENTIFY and SUBSCRIBE commands), and processes messages to measure throughput and stability under load.

_bench/bench\channels · high confidence

The internal/dirlock package now implements file locking using syscall.Flock on Unix-like platforms to prevent multiple nsqd instances from using the same data directory simultaneously. On Windows and illumos, the locking mechanism is currently a no-op, allowing the application to start but without enforcing exclusive access at the directory level.

internal/dirlock · high confidence

A new internal priority queue package has been added to support min-heap operations. This provides a reusable component for managing items by priority, including features like peeking and shifting based on a maximum priority threshold, which will be utilized by other internal components for ordered task processing.

internal/pqueue · high confidence

This entry establishes the foundational project files for NSQ, including the Makefile for building binaries, shell scripts for testing and benchmarking, a Dockerfile for containerization, and standard documentation files such as the README, ChangeLog, Code of Conduct, and Contributing guidelines.

(repo-wide) · high confidence

Added new internal packages for calculating and aggregating quantile statistics, specifically targeting end-to-end processing latency. The \quantile\ package provides a sliding-window based stream processor using the \perks/quantile\ library to track percentile values over time, while \aggregate.go\ introduces logic to merge multiple latency aggregates by averaging percentiles and handling JSON serialization for API responses.

internal/quantile · high confidence

A new command-line tool, nsq\_stat, is added to the apps directory to poll and display aggregate statistics for NSQ producers. It supports monitoring ingress/egress rates, depth (total, memory, disk), and metadata counts (in-flight, deferred, requeued, timed-out, messages, clients) for a specified topic and channel. Users can configure polling intervals, request timeouts, and the number of reports via flags like --interval, --count, --nsqd-http-address, and --lookupd-http-address.

_apps/nsq\stat · high confidence

The \nsqlookupd\ service is now available as a separate daemon responsible for managing topology metadata and serving client requests to discover the location of topics at runtime. It exposes a V1 TCP protocol for \nsqd\ nodes to register and unregister topics and channels, and provides an HTTP API with endpoints such as \/lookup\, \/topics\, \/channels\, \/nodes\, \/ping\, and \/info\ for querying the current state. The daemon includes a registration database that supports producer tombstoning, ephemeral channel cleanup, and configurable inactive producer timeouts, listening on default TCP port 4160 and HTTP port 4161.

nsqlookupd · high confidence

The internal/statsd package now provides a basic StatsD client implementation and a host-key utility. The client (client.go) allows sending counter, gauge, and timing metrics via an io.Writer with a configurable prefix, formatting them according to the StatsD protocol. A new utility function (host.go) normalizes hostnames by replacing dots and colons with underscores, likely for use as metric tags or keys.

internal/statsd · high confidence

A new benchmark reader utility has been added to the bench suite to measure NSQ message consumption performance. The tool connects to an nsqd instance, subscribes to a specified topic and channel, and processes messages concurrently using multiple workers. It now supports a configurable run duration and includes a new deadline flag that allows users to synchronize the start of the benchmark run to a specific timestamp. The reader reports throughput metrics including bandwidth, operations per second, and average operation time upon completion.

_bench/bench\reader · high confidence

A new benchmark writer utility has been added to the bench/bench\_writer directory to measure nsqd publishing performance. The tool connects to an nsqd instance via TCP, sends multi-publish batches of configurable size, and reports throughput metrics (MB/s, ops/s, and microseconds per operation). Users can control the benchmark duration using the -runfor flag or synchronize the start time across multiple instances using the -deadline flag. The tool also allows configuration of the target topic, message size, and batch size via command-line flags.

_bench/bench\writer · high confidence

The internal writers package now includes two new components: BoundaryBufferedWriter, which wraps an io.Writer with a bufio.Writer to flush only when a write exceeds the buffer's available space, and SpreadWriter, which buffers writes and distributes them over a specified interval using a ticker, supporting graceful shutdown via an exit channel. These additions enable more controlled and efficient output handling for internal components like nsqd.

internal/writers · high confidence

The internal/clusterinfo package has been introduced to centralize the logic for querying nsqlookupd and nsqd nodes to aggregate cluster state. This new component provides methods to retrieve unioned lists of topics and channels across all lookupd instances, fetch producer lists with version and topology data, and aggregate topic and channel statistics (such as depth, message counts, and latency) from multiple nodes. It includes robust error handling via a PartialErr interface to report partial failures when some nodes are unreachable, and ensures data consistency by sorting and deduplicating results. Tests verify correct address formatting for IPv4, IPv6, and hostnames.

internal/clusterinfo · high confidence

The internal/lg package introduces a new leveled logging system supporting DEBUG, INFO, WARN, ERROR, and FATAL levels. It provides a LogLevel type that implements the flag.Value interface, allowing log levels to be configured via command-line flags, and includes a Logf function that filters messages based on the configured minimum level.

internal/lg · high confidence

The internal/protocol package has been introduced, providing core infrastructure for the NSQ system. This includes a TCP server implementation (TCPServer) that manages listener acceptance and goroutine lifecycle, utility functions for sending length-prefixed and framed responses over network connections, and validation logic for topic and channel names. Additionally, it defines error types (ClientErr, FatalClientErr) for structured error handling and a helper function for converting byte slices to base-10 integers, along with corresponding benchmarks.

internal/protocol · high confidence

Introduces the nsq\_to\_http tool, which reads from an NSQ topic/channel and forwards messages to specified HTTP GET or POST endpoints. The application supports multiple upstream addressing modes (hostpool, round-robin, all), allows configuration of custom headers and content types, and includes configurable HTTP client timeouts. It also provides per-host status reporting and supports sampling of messages.

_apps/nsq\_to\http · high confidence

A new command-line utility, to\_nsq, has been added to publish data from standard input to an NSQ topic. Users can specify the target topic, NSQ daemon addresses, and a custom delimiter for splitting input. The tool also supports throttling the message rate via a --rate flag and allows passing configuration options to the NSQ producer through --producer-opt.

_apps/to\nsq · high confidence

The internal utility package now includes a new \UniqRands\ function that generates a slice of unique random integers, along with \TypeOfAddr\ to distinguish between TCP and Unix socket addresses. Additionally, a \WaitGroupWrapper\ helper is provided to simplify goroutine lifecycle management by automatically handling WaitGroup add/done calls when wrapping callbacks.

internal/util · high confidence

The new nsq\_tail application allows users to tail messages from multiple NSQ topics simultaneously by accepting multiple --topic flags. It introduces a --n flag to limit the total number of messages displayed before exiting, and a --print-topic flag to include the source topic name in the output. The tool also supports passing custom consumer options via --consumer-opt and identifies itself with a specific UserAgent string.

_apps/nsq\tail · high confidence

The nsq\_to\_nsq application now supports filtering messages based on JSON fields (via --require-json-field and --whitelist-json-field) and offers multiple upstream host selection modes (round-robin, hostpool, and epsilon-greedy) to improve reliability and control over message routing.

_apps/nsq\_to\nsq · high confidence

This change introduces the foundational implementation for the nsqd message broker, including the core channel and topic logic, the V2 TCP protocol client handling, and the HTTP API server. It adds support for message persistence via a disk-based backend queue, configurable output buffering for client connections, and integration with nsqlookupd for service discovery. The update also includes a new GUID generation algorithm for message IDs and comprehensive test coverage for the new components.

nsqd · high confidence

The nsqlookupd application now includes a dedicated main entry point that supports running as a service and reading configuration from TOML files. Users can now specify the log level via the --log-level flag or the log\_level key in the config file, replacing the deprecated --verbose flag. A README has also been added to describe the daemon's role in managing topology metadata.

apps/nsqlookupd · high confidence

The authentication client now supports configurable HTTP methods (GET or POST) for querying authd endpoints, allowing users to tailor request styles to their backend requirements. Additionally, the client passes the client TLS certificate's common name to the authd service during authentication, enabling identity-based authorization decisions. The implementation also introduces configurable connection and request timeouts for the HTTP client used in auth queries, improving control over latency and failure handling.

internal/auth · high confidence

Added a new internal package file that defines the current binary version as 1.3.0 and provides a helper function to format the application name, version, and Go runtime version into a single string.

internal/version · high confidence

The internal HTTP API package has been restructured into dedicated files for request handling, response formatting, compression, and server serving. A new HTTP client with configurable deadline timeouts and TLS support is introduced, featuring automatic fallback from HTTP to HTTPS on 403 errors. Response handling now uses a decorator pattern with V1 JSON formatting and logging, while request parameter parsing and topic/channel validation are extracted into reusable helpers.

_internal/http\api · high confidence

The nsq\_to\_file application has been refactored to support listening to multiple topics simultaneously via a new TopicDiscoverer component, which dynamically manages FileLogger instances for each topic. Users can now filter topics using the --topic-pattern flag and configure file rotation by size (--rotate-size) or interval (--rotate-interval). The tool also introduces new command-line options for logging verbosity (--log-level, --log-prefix), configurable HTTP client timeouts, and a --consumer-opt flag to pass options to the underlying nsq.Consumer, while requiring either --topic or --topic-pattern to be specified.

_apps/nsq\_to\file · high confidence

The nsqadmin interface has been updated to use Bootstrap v3.3.4, replacing the previous version to modernize the underlying UI framework. This change introduces updated component styles and layout behaviors across the application. Additionally, a new base stylesheet (base.scss) has been added to define custom visual elements, including specific color classes for zones and regions, a loading animation, and styling for the navbar brand image.

nsqadmin/static/css · high confidence

The nsqadmin interface has been restructured into a Single Page Application, replacing the previous multi-page or server-rendered approach with a single entry point (index.html) that loads the core JavaScript bundles (vendor.js, main.js). This change introduces client-side routing and state management, allowing for a more responsive user experience. The template now injects configuration variables such as version, graphite URL, statsd formatting options, and base path directly into the client-side JavaScript, enabling the SPA to initialize correctly with the server's settings.

nsqadmin/static/html · high confidence

The nsqadmin frontend has been refactored into a single-page application (SPA) using Backbone.js and Handlebars. This introduces client-side routing for topics, channels, nodes, and counters, managed by a new router and a centralized application state model. The change includes a new AJAX setup that enforces a 20-second timeout and specific content-type headers for all API requests, as well as utility helpers for formatting statsd metrics and sanitizing graphite keys. Users will now experience a more responsive interface with URL-based navigation and improved error handling for AJAX calls.

nsqadmin/static/js, nsqadmin/static/js/views · high confidence

The nsqadmin component has been restructured to introduce a modern JavaScript build pipeline using Gulp 4, Browserify, and ESLint (configured for ES2020), replacing the previous static asset handling. The Go backend now features a new HTTP server implementation (\http.go\) that registers a comprehensive set of RESTful API endpoints for managing topics, channels, and nodes, alongside a new notification system for admin actions. Configuration is centralized in \options.go\, supporting TLS client verification, base paths, and Statsd integration, while static assets are embedded into the binary using Go's \embed\ package.

nsqadmin · high confidence

The nsqadmin application has been refactored to use the go-svc library for proper service lifecycle management (Start/Stop) and now supports a broader set of configuration flags. Users can now configure HTTP client TLS settings (insecure skip verify, root CA, cert, key), set a URL base path, and define CIDR-based access control for the /config endpoint. Additionally, statsd counter and gauge formatting options are now configurable, and the verbose flag is deprecated in favor of --log-level. The application also supports loading configuration from a TOML file.

apps/nsqadmin · high confidence

The nsqadmin static build directory now includes pre-compiled CSS and SVG assets (base.css, bootstrap.min.css, glyphicons-halflings-regular.svg), ensuring the UI renders correctly without requiring a local build step.

nsqadmin/static/build · high confidence

The nsqd application is now structured with a dedicated entry point in apps/nsqd that initializes the daemon via go-svc, handles signal management, and loads configuration from TOML files. This change introduces a new options layer that supports typed flag parsing for TLS versions (including TLS 1.3), log levels, and backward compatibility for max-defer-timeout, while also exposing broadcast ports and supporting unix sockets for listeners.

apps/nsqd · high confidence

The nodes collection now correctly handles IPv6 addresses by wrapping them in square brackets when constructing the HTTP broadcast address. This ensures that nodes with IPv6 addresses are linked and displayed correctly in the UI, preventing broken links or display errors caused by the presence of colons in the raw IPv6 address.

nsqadmin/static/js/collections · high confidence

Added internal test utilities for assertions, fakes, and logging; Updated test TLS certificates and generation scripts; Updated test certificates for nsqadmin TLS verification.

The project has switched from the legacy \dep\ tool to Go modules, introducing \go.mod\ and \go.sum\ files to manage Go dependencies (including \go-nsq\, \httprouter\, and \toml\). The \bench\ directory now includes a \requirements.txt\ for Python dependencies like \boto\ and \tornado\. Additionally, the \nsqadmin\ frontend build system has been updated with a new \package.json\ and \package-lock.json\, incorporating modern versions of Gulp, ESLint, and Handlebars to support reproducible builds.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 50 → 45 (-5.0)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 52 → 58 (+6.2)
• Architecture 100 → 92 (-7.7)
• Maturity 43 → 42 (-0.3)
• Readiness 49 → 74 (+25.2)
• Security 70 → 69 (-0.4)
• Accessibility 34 (new)

• Coverage not included — suite not readable by the collector
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical vulnerability: [GHSA redacted] (nsqadmin/package-lock.json)
• Dependency hygiene not measured — dependency manifest found but not parsed for hygiene
• Duplicated block (10 lines × 2) (bench/bench_channels/bench_channels.go)
• Duplicated block (10 lines × 2) (nsqd/http.go)
• Duplicated block (10 lines × 2) (nsqd/http.go)
• Duplicated block (10 lines × 2) (nsqd/protocol_v2.go)
• Duplicated block (10 lines × 2) (nsqlookupd/http.go)
• Duplicated block (10 lines × 4) (nsqlookupd/http.go)
• Duplicated block (11 lines × 2) (apps/nsq_to_http/nsq_to_http.go)
• Duplicated block (11 lines × 2) (internal/clusterinfo/data.go)
• Duplicated block (11 lines × 2) (internal/clusterinfo/data.go)
• Duplicated block (11 lines × 2) (nsqd/nsqd.go)
• Duplicated block (12 lines × 2) (bench/bench_channels/bench_channels.go)
• …and 65 more

• ClassTooLong: ClusterInfo (internal/clusterinfo/data.go)
• ClassTooLong: httpServer (nsqadmin/http.go)
• ClassTooLong: protocolV2 (nsqd/protocol_v2.go)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical CVE: [GHSA redacted] (nsqadmin/package-lock.json)
• Critical vulnerability: [GHSA redacted] (nsqadmin/package-lock.json)
• Dependency pinned to a stale untagged commit: github.com/bmizerany/perks
• Duplicated block (10 lines × 2) (apps/nsq_to_http/nsq_to_http.go)
• Duplicated block (10 lines × 2) (internal/clusterinfo/data.go)
• Duplicated block (10 lines × 3) (bench/bench_channels/bench_channels.go)
• Duplicated block (10 lines × 3) (nsqd/http.go)
• Duplicated block (10–11 lines × 2) (bench/bench_channels/bench_channels.go)
• Duplicated block (11 lines × 2) (internal/clusterinfo/data.go)
• Duplicated block (11 lines × 2) (internal/clusterinfo/data.go)
• Duplicated block (11 lines × 2) (internal/clusterinfo/data.go)
• Duplicated block (11 lines × 2) (nsqd/channel.go)
• …and 107 more

• 1 commits — 1 feature/other, 0 fixes

• (repo) — 1 commit

• change: Merge pull request #1534 from jehiah/tcp_connection_blocking_shutdown_1534

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

nsqio/nsq was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.