ZeroMQ: Messaging for Distributed Systems

• ZeroMQ is a messaging middleware that augments TCP/IP sockets to enable robust, asynchronous communication in distributed architectures.
• It supports communication patterns such as publish‐subscribe, request‐reply, and push‐pull, often working with tools like Protocol Buffers and JSON.
• Widely used in scientific data acquisition, robotics, and edge computing, ZeroMQ offers flexible deployment in brokerless or hybrid middleware designs.

Searching arXiv for ZeroMQ-related papers to ground the article in cited research. ZeroMQ is a messaging middleware and transport substrate used to connect distributed processes, services, and analysis stages across scientific instrumentation, robotics, edge computing, and data-processing systems. Across the literature, it is described as a middleware that augments the capabilities of TCP/IP sockets, a high-performance asynchronous messaging library, and a low-level communication middleware with a unified socket API supporting patterns such as publish-subscribe, request-reply, and push-pull (Lyard et al., 2015, Das et al., 16 Feb 2026, Li et al., 2018, Corte et al., 11 Aug 2025). Its role is typically architectural rather than application-specific: it carries messages between components, while serialization, orchestration, scheduling, persistence, and higher-level control are supplied by adjacent software layers such as Protocol Buffers, JSON, RDMA services, shared-memory transports, or robotics runtimes (Lyard et al., 2015, Li et al., 2018, Lawrence, 2021, Zhang et al., 7 Mar 2025).

1. Definitions and conceptual position

ZeroMQ occupies a middle layer between raw sockets and heavyweight distributed-object or brokered middleware. In the CTA data-acquisition literature, it is characterized as middleware that augments the capabilities of TCP/IP sockets, making sockets easier, more robust and almost as effective as raw TCP, while not implementing very high-level features like those found in CORBA (Lyard et al., 2015). In the JUNO DAQ online software, it is the low-level communication middleware for a service-oriented architecture in which services communicate directly without a traditional broker in the middle (Li et al., 2018). In brokerless messaging benchmarks, it is described as a C++ library started in 2007 with extensive documentation, a large community footprint, and a unified socket API with transport-agnostic operations such as bind, connect, send, and receive (Corte et al., 11 Aug 2025).

The communication abstractions associated with ZeroMQ recur with notable consistency. Multiple papers identify request-reply, publish-subscribe, and push-pull as standard patterns enabled by ZeroMQ (Li et al., 2018, Lyard et al., 2015). Later systems build higher-level semantics on top of these patterns: REST-style resource access and observations in Zest (Moore et al., 2019), file-level Unlimited Sliding-Window flow control in ENGAS (Shi et al., 2021), and a robotics-oriented abstraction layer with dynamic discovery, type-safe messages, and native NumPy and PyTorch support in Cortex (Das et al., 16 Feb 2026).

A common misconception is that ZeroMQ is itself a brokered messaging system. The literature shows a more differentiated picture. Some systems use it explicitly to avoid a centralized broker, emphasizing direct service-to-service exchange or many-to-many communication (Li et al., 2018, Collaboration et al., 2017). Other systems build a broker or broker-like component with ZeroMQ, such as the Meta-ROS Message Broker and the NCEM streaming Aggregator (Ranjan et al., 28 Jan 2026, Welborn et al., 2024). This suggests that ZeroMQ is better understood as a messaging substrate whose topology is chosen by the application rather than imposed by the library.

2. Communication models and protocol layering

ZeroMQ’s practical significance lies in the communication models layered over it. The most frequently cited patterns are publish-subscribe, request-reply, and push-pull (Li et al., 2018, Lyard et al., 2015). These patterns are used either directly or as primitives for more specialized workflows.

In monitoring and observability systems, publish-subscribe often provides the control or telemetry plane rather than the bulk data path. HOSS uses a ZeroMQ pub-sub server over TCP/IP to check whether files or directories already exist on a host, report file sizes and checksums, and publish statistics for monitoring, while RDMA and local filesystem moves handle the actual file movement (Lawrence, 2021). ALICE Overwatch receives time-stamped QA histograms and other ROOT-based objects from HLT mergers via ZeroMQ, then stores, enriches, and visualizes them in a web application (Ehlers et al., 2018).

In data-transfer systems, push-pull and request-reply are often combined with buffering or application-level routing. HAWC uses ZeroMQ between Readout Processes, Data Queues, Reconstruction Clients, and Event Sorters, with request-reply between Data Queues and Reconstruction Clients and one-way push-style transfer from Readout Process to Data Queue (Collaboration et al., 2017). In rta-dp, Producers send data via ZeroMQ PUSH/PUB to a Supervisor, while commands are carried on PUB/SUB channels and results may be forwarded to new Supervisors to build multi-stage distributed pipelines (Bulgarelli et al., 5 Nov 2025).

Application protocols layered on top of ZeroMQ vary substantially. Glyph’s glyph-remote uses a client-server architecture in which the GP loop is separated from the experimental loop using ZeroMQ, with communication encoded in JSON as { "action": "value", "payload": "value" } and actions CONFIG, EXPERIMENT, and SHUTDOWN (Quade et al., 2018). Zest defines a REST-over-ZeroMQ protocol with a 32-bit header, option fields, request codes such as GET, POST, and DELETE, and a router/dealer topology for observations and notifications (Moore et al., 2019). ENGAS uses P-messages and S-messages over ZeroMQ Pub/Sub, with file metadata including file identification, file version, file size, format, checksum, checksum method, compression flag, and compression method (Shi et al., 2021).

This diversity indicates that ZeroMQ is typically transport and framing infrastructure rather than a fixed wire protocol. A plausible implication is that ZeroMQ’s main technical value lies in decoupling message movement from application semantics, allowing systems to choose their own envelopes, schemas, and reliability policies.

3. Architecture across scientific data acquisition and high-throughput systems

ZeroMQ appears extensively in scientific DAQ, online monitoring, and real-time analysis infrastructures. These deployments show its use both for high-throughput horizontal data flow and for lighter-weight coordination.

The CTA DAQ presents a clear separation: ACS handles vertical control flow, while ZeroMQ handles horizontal high-throughput data flow between logically separated pipeline components (Lyard et al., 2015). The system is designed for up to $550\ \mathrm{Gbps}$ input and uses ZeroMQ together with Protocol Buffers to rapidly develop a robust prototype including data persistence to compressed FITS files (Lyard et al., 2015). The HAWC observatory similarly uses ZeroMQ to connect SBC readout, farm-side reconstruction, event sorting, archival writing, and live analysis, enabling real-time software triggering on a raw stream of about $500\ \mathrm{MB/s}$ with latency of a few seconds (Collaboration et al., 2017).

In ALICE Overwatch, ZeroMQ is the interface between the HLT QA subsystem and the Overwatch backend. Messages contain metadata and collections of ROOT-based objects, primarily TH1-derived histograms but also more complex objects such as TObjArray, with payloads ranging from ~10 kB/request to >1 MB/request (Ehlers et al., 2018). The cumulative, time-stamped character of these messages enables time-slice reprocessing and later replay, including investigation of transient effects such as a hot trigger channel (Ehlers et al., 2018).

Other systems use ZeroMQ in more selective roles. HOSS explicitly does not use ZeroMQ as the bulk data transport mechanism; that role is handled by RDMA and local filesystem moves. Instead, ZeroMQ is part of the control/observability plane, exposing file and directory state queries and statistics reporting from RDMA servers to the Python orchestration layer and GUI (Lawrence, 2021). This distinction is important because it shows ZeroMQ being deployed where coordination, safety checks, and live status are more critical than raw transfer bandwidth.

Newer streaming workflows continue this pattern of composability. The NCEM-to-Perlmutter detector pipeline uses ZeroMQ-based Producers, an Aggregator, and Consumers/NodeGroups to stream detector data directly from server RAM to compute-node RAM, coordinated through the Clone pattern as a distributed key-value store (Welborn et al., 2024). For a 695 GB dataset, the paper reports $442.6 \pm 53.5$ s for file transfer versus $97.2 \pm 4.1$ s for streaming, with an enhancement of 4.6× and a streaming pipeline performance of 7.2 GB/s (Welborn et al., 2024). This is an instance where ZeroMQ is not merely carrying control messages but is central to bypassing storage I/O bottlenecks.

4. ZeroMQ in robotics, edge systems, and middleware interoperability

Robotics and edge-computing systems use ZeroMQ both as a direct transport and as a backend hidden behind higher-level APIs. These papers emphasize hybrid transport selection, cross-device communication, and middleware interoperability.

HyperGraph ROS uses Intel TBB Flow Graph with C++ pointer passing for intra-process messaging and switches to ZeroMQ for inter-process and cross-device communication (Zhang et al., 7 Mar 2025). The computational hypergraph is formed by TBB pointer-based connections for local messaging plus ZeroMQ links for distributed messaging, with dynamic communication mechanism selection via YAML configuration (Zhang et al., 7 Mar 2025). The paper reports that for cross-device communication at 4 MB message size, ROS 2 latency is about 970 ms, whereas HyperGraph ROS latency is less than 80 ms (Zhang et al., 7 Mar 2025). At the same time, it acknowledges that inter-process ZeroMQ communication on the same machine is slightly less efficient than ROS 1 and ROS 2 because of serialization isolation before transmitting via shared memory (Zhang et al., 7 Mar 2025).

Wrapyfi treats ZeroMQ as one supported backend among YARP, ROS, and ROS 2 (Abawi et al., 2023). It abstracts middleware through the MiddlewareCommunicator class and register decorator, allowing the same method to be registered for ZeroMQ with carrier='tcp' and used in forwarding and channeling schemes (Abawi et al., 2023). Because ZeroMQ does not provide specialized message structures for some data types, Wrapyfi uses multipart messages for images and transmits audio as a string encoding the auditory signal along with its properties as a single multipart message (Abawi et al., 2023).

SMOTEC leverages ZeroMQ as the connector of its components and explicitly calls it an asynchronous messaging library for edge computing applications (Nezami et al., 2023). In this testbed, edge agents, mobile agents, SUMO, EPOS, and K3s are coordinated through message-driven runtime interactions including handover requests, service request forwarding, and decentralized placement decisions (Nezami et al., 2023). The proof-of-concept reports workload balanced by more than 92%, with utilization variance reduced from 0.078 to 0.004 in the homogeneous case and from 0.088 to 0.007 in the heterogeneous case (Nezami et al., 2023).

RIMAOS2C provides a more specialized hybrid-network example. It retains DDS/UDP for cross-chip low-latency small messages and uses TCP-based ZeroMQ for cross-chip large-message transmission between Orin units (Sang et al., 31 Jul 2025). With two cross-chip subscribers, it eliminates message redundancy and improves large-data transmission efficiency by 36 to 40 percent while reducing callback latency variation by 42 to 906 percent (Sang et al., 31 Jul 2025). The Message Bridge subscribes via SHM, forwards data once across chips, and republishes via SHM on the target chip, with ZeroMQ used as the inter-chip bulk-transport layer (Sang et al., 31 Jul 2025).

These cases show that in robotics, ZeroMQ is frequently chosen not as an exclusive middleware but as the distributed part of a hybrid architecture. This suggests that its comparative advantage is strongest at the process boundary, the device boundary, or the network boundary, especially when local shared-memory or pointer-based delivery can handle the purely local path more efficiently.

5. Reliability, performance, and system-level tradeoffs

The literature presents ZeroMQ as high-performance, but performance claims are strongly workload- and architecture-dependent. Some systems report near-line-rate transfer or high throughput; others report that ZeroMQ performs similarly to simpler socket-based designs.

The CTA prototype achieved about $9\ \mathrm{Gbps}$ between two nodes, roughly 10% slower than raw C-structs plus TCP sockets in low-latency environments (Lyard et al., 2015). HAWC measured about 940 Mbit/s for a point-to-point connection between two ZeroMQ sockets over a 1 Gbps network using 1 MB blocks, very close to the network’s theoretical limit (Collaboration et al., 2017). In brokerless messaging benchmarks using PUB/SUB, ZeroMQ reached about 4.8 GB/s in In-Process throughput and about 2.9 GB/s in IPC and TCP peak throughput, while often dominating throughput for large payloads and large-subscriber scaling, though not always latency or CPU efficiency (Corte et al., 11 Aug 2025).

Cortex, the ZeroMQ-based communication layer used with Neurosim, reports 100+ kHz publish-subscribe rates for payloads below 39 KB, about 250 Hz for 1080p RGB images (~23.7 MB), throughput up to 7 GB/s, and same-machine IPC communication latency below 0.7 ms (Das et al., 16 Feb 2026). These results are tied to additional design features such as zero-copy IPC when possible, type-safe message definitions, and native NumPy and PyTorch support rather than to raw ZeroMQ alone (Das et al., 16 Feb 2026).

However, some papers show that ZeroMQ’s benefits are not automatic. In federated learning benchmarking, ZMTP is treated as an optimized application-layer protocol, but the paper states that it performs similarly to TCP in most network conditions, whereas AMQP and MQTT achieve much better communication times (Cleland et al., 2022). The explanation given is architectural: ZMTP in that implementation relies on sockets and one-way communication rather than broker-based distribution (Cleland et al., 2022). This is an important corrective to the assumption that adopting ZeroMQ or ZMTP necessarily yields better performance.

Reliability and ordering semantics also vary. ENGAS uses ZeroMQ’s Pub/Sub pattern to support asynchronous file delivery, but the Unlimited Sliding-Window cannot guarantee that files arrive in order; it can guarantee that the files sent by the Provider eventually arrive at the Subscriber (Shi et al., 2021). In the NCEM streaming workflow, push sockets block instead of dropping messages at the high water mark, which the paper treats as important for microscope data, but upstream UDP loss still means that about 0.1% of sectors are lost before entering the ZeroMQ pipeline for very large scans (Welborn et al., 2024). In Cortex, scalability degrades beyond about 8 subscribers on the same machine because of CPU contention from busy-waiting subscribers (Das et al., 16 Feb 2026).

A plausible implication is that ZeroMQ’s performance profile is best described as enabling rather than guaranteeing high throughput and low latency. The strongest results arise when socket pattern, batching, serialization, local-memory strategy, and topology are co-designed with the application.

6. Security, control, and broader design implications

Beyond transport efficiency, ZeroMQ is repeatedly used to enforce architectural decoupling, support control planes, and integrate heterogeneous software environments. In some systems, this includes explicit security and access-control layers.

Zest is the clearest security-oriented example. It combines a RESTful architecture with a variant of a publisher/subscriber topology that has enhanced routing support, with encryption as standard, access control through Macaroons, auditability of communication across nodes, and asynchronous communication between nodes (Moore et al., 2019). ZeroMQ is chosen there because it offers flexible communication topologies, a simple abstraction over TCP sockets, and secure connections based on elliptic-curve cryptography (Moore et al., 2019). Observations on data, audit, and notify paths create asynchronous channels for logging, audit trails, and callback-style notifications (Moore et al., 2019).

In control systems, ZeroMQ often replaces more rigid or domain-specific coordination mechanisms. In the pulsed ion accelerator control infrastructure, ZeroMQ replaced LabVIEW shared variables and was implemented as a message broker between different LabVIEW and Python programs, enabling Python scripting, parameter scans, timestamp coordination, and shot notifications (Persaud et al., 2015). In JUNO, by contrast, ZeroMQ is explicitly part of a brokerless service-to-service design, while ZooKeeper and a custom HA library handle availability and service mapping (Li et al., 2018). These contrasting deployments underscore that ZeroMQ does not prescribe a control philosophy.

Recent middleware papers use ZeroMQ as one component in broader interoperability strategies. Meta-ROS builds its Message Broker with ZeroMQ to provide topic-based, event-driven communication that decouples processes, while also using Zenoh for distributed data handling and emphasizing that Zenoh provides the best performance and the least latency in that framework (Ranjan et al., 28 Jan 2026). This suggests a broader trend toward hybrid middleware stacks in which ZeroMQ provides a lightweight broker or messaging substrate within a larger ecosystem.

Taken together, these studies define ZeroMQ less by a single application domain than by a recurring systems role: it is a practical messaging substrate for decoupled architectures, adaptable enough to support direct brokerless exchange, broker-mediated routing, REST-style protocols, typed tensor streaming, monitoring planes, and hybrid robotics transport stacks. The literature does not present it as a universal replacement for all communication technologies. Instead, it repeatedly appears in combinations such as ZeroMQ plus Protocol Buffers (Lyard et al., 2015, Li et al., 2018), ZeroMQ plus shared memory (Sang et al., 31 Jul 2025), ZeroMQ plus RDMA (Lawrence, 2021), ZeroMQ plus JSON (Quade et al., 2018, Bulgarelli et al., 5 Nov 2025), and ZeroMQ plus dynamic discovery and zero-copy tensor handling (Das et al., 16 Feb 2026). This suggests that ZeroMQ’s enduring significance lies in being a composable foundation for distributed systems rather than a complete distributed-systems framework in itself.