Over-the-Top Broker in Networked Systems

• Over-the-top brokers are intermediary layers that add policy, orchestration, and validation capabilities beyond basic message forwarding.
• They extend functionalities in MQTT, IoT messaging, network slicing, and financial markets by enforcing client-requested security and dynamic resource allocation.
• Empirical evaluations demonstrate that advanced broker policies can maintain near-normal forwarding performance while introducing minimal latency overhead for enhanced security.

Searching arXiv for relevant papers on “Over-the-Top Broker” across messaging, IoT, networking, and broker-mediated market settings. An over-the-top broker is a broker or intermediary layer that sits above a baseline relay, allocation, or matching mechanism and adds control logic that the underlying broker does not provide. In MQTT publish/subscribe, the term denotes an enhancement to a normal broker that enforces client-requested security levels across the broker boundary rather than merely securing a single hop (Spielvogel et al., 2021). In adjacent literatures, comparable broker layers are used to move message brokers closer to devices, replicate and validate publications across trust boundaries, annotate astronomical alert streams, dynamically lease network and cloud resources, or internalize and externalize client flow in broker-mediated markets (Abdelwahab et al., 2016, Ramachandran et al., 2018, Möller et al., 2020, Samdanis et al., 2016, Friese et al., 11 Aug 2025, Aqsha et al., 2024). This suggests a recurring architectural pattern rather than a single standardized definition: the broker remains in the path, but acquires policy, orchestration, validation, or inference responsibilities that exceed ordinary forwarding.

1. Conceptual scope and recurring architectural role

In the provided literature, the over-the-top broker is consistently broker-centric rather than broker-less. FogMQ is explicitly described as an “over-the-top, broker-centric messaging system” that preserves the broker abstraction while making brokers “mobile, distributed, and edge-adaptive” (Abdelwahab et al., 2016). Fink is designed to “go beyond traditional broker features” by not only ingesting and redistributing alerts but also enriching, classifying, ranking, and versioning them (Möller et al., 2020). The 5G Network Slice Broker and the split-computing resource broker similarly operate as allocation and abstraction layers between tenant-side requests and heterogeneous infrastructure-side resources (Samdanis et al., 2016, Friese et al., 11 Aug 2025).

A useful common denominator is that the broker ceases to be a passive conduit. In IoT pub/sub, it must inspect metadata and decide whether forwarding is permissible; in distributed messaging, it may self-deploy or replicate data to an immutable ledger; in astronomy, it becomes a real-time annotation and filtering system; in network slicing and split computing, it exposes a uniform service interface over heterogeneous providers; and in OTC finance, it internalizes and externalizes client flow while learning from order flow and managing inventory (Spielvogel et al., 2021, Ramachandran et al., 2018, Friese et al., 11 Aug 2025, Aqsha et al., 2024). This suggests that “over-the-top” denotes an added control plane layered on top of an existing brokerage primitive.

The term does not imply removal of trust assumptions. On the contrary, several of the cited systems remain explicitly broker-mediated. The MQTT security extension depends on a “trusted and non-compromised broker” (Spielvogel et al., 2021). Trinity distributes trust across multiple organizations, but still routes publications through brokers integrated with a permissioned blockchain (Ramachandran et al., 2018). The financial models are broker-mediated by construction: clients trade through the broker, and the broker strategically manages the resulting exposure (Bergault et al., 2024, Aqsha et al., 2024).

2. MQTT security enforcement beyond ordinary forwarding

The clearest systems interpretation of an over-the-top broker appears in the MQTT security literature. A standard mixed-mode broker may accept both TLS and plain TCP connections, yet forward messages solely on the basis of topic subscription. The consequence is that a TLS subscriber “has no information about the security level of the connection of publishers,” and a TLS publisher “can not enforce the security level of the connection of subscribers.” The message may therefore be protected only from publisher to broker or only from broker to subscriber, “but it is never enforced beyond” the broker (Spielvogel et al., 2021).

The proposed remedy is broker-mediated client-to-client security, defined as “a continuous protection of authentication, confidentiality and integrity of the message on all ways throughout the network mediated by the broker from the publisher to the subscriber or vice versa.” This is stronger than ordinary client-to-broker TLS, but it is not end-to-end security in the usual cryptographic sense, because the broker remains an intermediary and must be trusted (Spielvogel et al., 2021).

Mechanistically, the broker inspects MQTT 5 User Properties carried in CONNECT, PUBLISH, or SUBSCRIBE. A modified client can explicitly signal “its wish for enforcement of its own security level beyond the broker,” or the broker can infer intent implicitly from whether the client connected via TLS or plain TCP. Enforcement is operational and case-based. If a TLS publisher requests client-to-client security, the broker forwards only to subscribers satisfying the required security level. If a TCP publisher sends to a subscriber that requested client-to-client security, delivery is blocked. Otherwise, forwarding proceeds normally. The broker must therefore “decide whether the message may be forwarded” and “detect a conflict between the interests of a publisher and the ones of the current and future subscribers” (Spielvogel et al., 2021).

The prototype is MQTT-specific and uses MQTT v5 features. It was written in Python 3.8.5, designed to be comprehensible and adaptable to brokers such as Mosquitto or HiveMQ, and evaluated with Eclipse Mosquitto (version 1.4.10) on a Raspberry Pi 3B+ (Spielvogel et al., 2021).

Aspect	Reported result
Initial CONNECT→CONNACK over insecure TCP	average 9.54 ms
Initial CONNECT→CONNACK over TLS	average 3207.53 ms
TLS setup variability	standard deviation 153.57 ms
Insecure setup variability	standard deviation 3.69 ms
Broker forwarding median, normal	109 ms
Broker forwarding median, enhanced enforcement	111 ms
Extra message overhead with implicit inference	no extra message overhead
Extra message overhead with MQTT 5 User Properties	minimum 7 additional bytes
Estimated overhead with dedicated MQTT field	2 bytes total

The measurements show that TLS setup was roughly 335× slower on average and took “more than 3 seconds,” whereas the broker-side forwarding path showed no significant performance difference between normal and extended forwarding across 500 simulation runs. This makes the proposal notable for shifting policy enforcement into broker forwarding rather than into heavier endpoint cryptographic machinery on constrained devices (Spielvogel et al., 2021).

3. Distributed broker overlays for latency and immutability

A second major usage of the over-the-top broker idea appears in distributed IoT messaging. FogMQ addresses the tail-latency problem of centralized brokers by moving broker functionality toward the edge. Instead of forcing all traffic through a distant centralized endpoint, FogMQ creates a clone for each device in the closest available cloud or cloudlet from a heterogeneous set $A$, and each clone acts as a high-capacity message broker and computation-offloading node (Abdelwahab et al., 2016).

The empirical motivation is explicit. Hosting Redis one hop away in an edge cloudlet produced a tail end-to-end latency of 15.6 ms, compared with 24.2 ms for EC2-Redis, about a 1.5× tail improvement and about 3× average improvement. Under constrained devices capped at 10% CPU, D2D ZeroMQ had about 7× longer average latency and 4× longer tail latency than an edge broker. FogMQ therefore retains the broker abstraction while making it mobile, distributed, and edge-adaptive (Abdelwahab et al., 2016).

FogMQ’s control logic is autonomous discovery, self-deployment, and online migration. Clones discover hosting options, monitor demand, and migrate according to the Flock algorithm, modeled as a singleton symmetric weighted congestion game minimizing

$C(\sigma) = \sum_{x \in A} w_x f(w_x).$

Migration from cloud $x$ to $y$ occurs when

$$u_i(y)\, f(w_y + u_i(y)) \le \eta\, u_i(x)\, f(w_x - u_i(x)).$$

The paper states that Flock converges to a Nash equilibrium, and for $f(w)=\exp\!\left(-\frac{1}{w+a}\right)$ the Price of Anarchy is tightened to $1+\epsilon$ for sufficiently large $a$. In simulation, observed average convergence was better than $O(n)$, despite a worst-case bound of $O(n \log(n f_{\max}))$ (Abdelwahab et al., 2016).

Trinity represents a different extension: the over-the-top broker as a trust-distributing and validation layer. It combines a distributed publish/subscribe broker network with a permissioned blockchain so that data published to one broker is replicated to all brokers, validated by smart contracts, and stored in an immutable ledger (Ramachandran et al., 2018). The architecture is aimed at multi-stakeholder settings such as supply-chain monitoring, where centralized brokers create a single point of failure and a single point of trust.

Its workflow is broker-integrated rather than broker-replacing. Contractual policies are published under the Contract topic; verified outputs are returned under a topic formed by appending _verified; the blockchain interface exposes DeliverTransaction, GetCurrentBlockHeight(), and GetBlock(BlockHeight); and Tendermint commits transactions when roughly $C(\sigma) = \sum_{x \in A} w_x f(w_x).$0 of validators approve them (Ramachandran et al., 2018).

The trade-off is resource and latency overhead. On a 20-node Raspberry Pi 3 testbed, loopback forwarding was about 90 ms, but blockchain-validated delivery rose to about 3.5 seconds for local subscription and about 3.7 seconds for verified subscription from a different broker. Network overhead reached about 300 MB per 1000 seconds for a 20-device network at 1 TPS, and worst-case CPU usage reached about 85% on Raspberry Pi 3. The gain is persistence, ordering, and immutability across trust boundaries (Ramachandran et al., 2018).

4. Alert brokers as scientific enrichment layers

In time-domain astronomy, the over-the-top broker becomes an alert-processing system that adds scientific intelligence to a raw alert stream. Fink is designed for LSST-scale alert rates, where the problem is not merely transport but filtering and understanding a firehose of candidate events quickly enough for follow-up. LSST is expected to produce of order 10 million alerts per night, each arriving within about 60 seconds of observation, and Fink is explicitly designed to ingest, enrich, classify, prioritize, redistribute, and archive alerts rather than simply pass them through (Möller et al., 2020).

Its architecture is cloud-deployable and distributed: Apache Spark for processing, Apache Kafka for communication, Apache HBase for the science portal, and HDFS for persistent distributed storage. The broker loop is ingestion, quality filtering, cross-matching, classification, substream generation, redistribution, and archival. For ZTF prototyping, Spark processes alerts when about 400 alerts have accumulated or 15 seconds have elapsed, whichever comes first, and the system was shown to handle up to 100,000 alerts per minute. A ZTF quality filter removed about 70% of the raw stream, leaving roughly 30% for science processing (Möller et al., 2020).

Fink’s added-value layer includes cross-matching with CDS xMatch, internal tools for small catalogues, and stream cross-matching with external alert systems through Comet and VOEvent. It also includes machine-learning modules such as SuperNNova for supernova classification and active-learning pipelines that incorporate new labels. In tests on simulated data, SuperNNova SN1 reached about 75% accuracy on complete light curves and about 59% before maximum light; SN2 reached about 87% on complete light curves. On the ZTF public stream, Fink reduced the alert sample to about 8% while retaining more than 75% of spectroscopically identified SNe Ia and SNe, with a median classification delay for SNe Ia of about 6 days before observed peak brightness (Möller et al., 2020).

ALeRCE shows a closely related broker pattern. It ingests the public ZTF stream using Kafka, stores alert packets in AWS S3, crossmatches with external catalogs, runs a stamp classifier, preprocesses photometry, extracts features, performs light-curve classification, detects outliers, and reports the annotated stream through its output system (Sánchez-Sáez et al., 2020). The light-curve classifier uses 152 features, including 124 detection features, 18 non-detection features, and 10 contextual features, and applies a two-level Balanced Random Forest over 15 subclasses.

The operational emphasis is scale and rapid updating. Using data up to 2020/06/09, ALeRCE classified 868,371 ZTF sources with sufficient detections, and when a new alert arrives it recomputes relevant variability features and updates the classification in about 1 second. The top-level classifier achieved macro precision 0.96, macro recall 0.99, and macro F1 0.97; the bottom level achieved macro precision 0.57, macro recall 0.76, and macro F1 0.59 (Sánchez-Sáez et al., 2020). In this domain, the over-the-top broker is not primarily a transport broker but a real-time scientific annotation service.

5. Resource and network-slice brokers in 5G and 6G systems

In mobile networking and distributed computing, the over-the-top broker is a resource-allocation and abstraction layer. The 5G Network Slice Broker is introduced as a capacity broker generalized for full multi-tenancy: a logically centralized entity inside the infrastructure provider that receives slice requests from tenants, performs admission control, maps the request to actual RAN and core resources, configures the network, and monitors SLA satisfaction (Samdanis et al., 2016).

The broker is positioned at the Master Operator-Network Manager and interacts with the Master Operator Shared RAN Domain Manager and underlying network elements via Type 5, Type 2/Itf-N, and Type 1/Itf-B interfaces. It is distinct from the SCEF: SCEF exposes capabilities, while the Network Slice Broker actively allocates and orchestrates resources. The request metadata discussed in the paper include amount of resources, timing, resource type and QoS, traffic volume or content size, service-related attributes such as mobility level and service disruption tolerance, and cell set or geographic scope (Samdanis et al., 2016).

The 6G split-computing resource broker extends this logic to heterogeneous cloud and network infrastructures. It is described as a service-layer intermediary that hides provider-specific differences behind a uniform resource-access and provisioning interface. Its abstraction model is built from attributes, templates, and offers: attributes are the basic building blocks; the broker compiles a chain of templates from a standardized customer or service template to a provider-specific representation; and incoming generic events such as create are transformed into provider-specific events such as create_Y (Friese et al., 11 Aug 2025).

The two scenarios examined are Volumetric Video Chat and Smart Drones. In the first, the broker creates a nearby rendering instance when the client device lacks CPU, GPU, or memory; in the second, it brokers both compute and reliable, low-latency connectivity near the flight path. The proof-of-concept uses Docker containers, Kubernetes, Kafka as event bus, MongoDB as database, Java + Spring Boot, and JSON for data representation. The main entry point is a broker service exposing a REST interface, complemented by webhook-based asynchronous callbacks, a broker worker, specialized resource brokers, the Kafka event bus, and a shared MongoDB store for templates, instances, and configuration or state data (Friese et al., 11 Aug 2025).

A plausible implication is that the over-the-top broker in networking has evolved from a pure capacity allocator toward a unified orchestration layer over compute, connectivity, policy, and provider heterogeneity.

6. Broker-mediated OTC markets and strategic intermediation

In financial-market models, the over-the-top broker is a strategic intermediary that internalizes client flow, externalizes risk to the lit market, and may learn from the flow itself. A mean-field formulation considers a broker facing a large population of informed traders, each trading through the broker while the broker provides liquidity and hedges in the lit market. In the mean-field equilibrium, the broker’s optimal strategy is linear in his own inventory, the average inventory among informed traders, and the common signal or average trading speed of the informed traders (Bergault et al., 2024). The equilibrium is characterized by a coupled FBSDE system and solved through a matrix Riccati equation, yielding a closed-form policy of the form

$C(\sigma) = \sum_{x \in A} w_x f(w_x).$1

A related stochastic-control model studies a broker providing liquidity to an informed trader and noise traders while filtering information from prices and client flow. The key result is that brokers hold a strategic advantage over traders who rely solely on prices to filter information, and that information leakage in the client’s trading flow yields an economic value to the broker comparable to transaction costs (Aqsha et al., 2024). The broker can speculate profitably and mitigate risk effectively, whereas price-based filtering leaves broker performance indistinguishable from a naive strategy that internalizes noise flow, externalizes informed flow, and offloads inventory at a constant rate.

A further infinite-horizon model derives a closed-form broker dealing strategy for toxic flow. The broker’s optimal trading rate is linear:

$C(\sigma) = \sum_{x \in A} w_x f(w_x).$2

or equivalently

$C(\sigma) = \sum_{x \in A} w_x f(w_x).$3

The interpretation is explicit: mean-revert broker inventory, hedge toxic exposure from informed traders, exploit signal information, and partially hedge uninformed flow. The paper’s baseline calibration indicates that broker profitability is maximized when the broker charges roughly 60% of the lit-market liquidity cost to both client types (Cartea et al., 23 Mar 2025).

Agent-based OTC simulation produces a complementary structural view. In that model, all trades are intermediated solely by market makers and agent visibility is constrained by a network topology. Market makers quote bid and offer prices, update quotes based on the last observed trade and inventory, and may trade with one another if they breach a soft position limit. As the link probability $C(\sigma) = \sum_{x \in A} w_x f(w_x).$4 decreases, the system enters a critical region around $C(\sigma) = \sum_{x \in A} w_x f(w_x).$5, beyond which the market fragments into distinct clusters and arbitrage becomes rapidly possible between the prices of different market makers (Wilkinson et al., 2024). This suggests that over-the-top intermediation in OTC markets is not only a strategic-control problem but also a network-topology problem.

7. Welfare, capacity, and heterogeneity in brokered systems

Broker layers also raise welfare and allocation questions. In bilateral trade with a broker, buyer and seller interact exclusively through the broker, who strategically maximizes profit by arbitraging between buying and selling prices. Under symmetric agents with monotone-hazard-rate distributions, the gains from trade achieve a $C(\sigma) = \sum_{x \in A} w_x f(w_x).$6-approximation to first-best when the broker uses an optimal posted-pricing mechanism (Hajiaghayi et al., 2024). For arbitrary profit-maximizing BNIC and IR mechanisms with uniform buyer and seller distributions of arbitrary support, the induced gains from trade satisfy a tight $C(\sigma) = \sum_{x \in A} w_x f(w_x).$7-approximation. But in general distributions the approximation can become arbitrarily bad, so brokerage does not guarantee any constant-factor efficiency without additional structure (Hajiaghayi et al., 2024).

Capacity constraints appear in platform-mediated broker assignment as well. In online real-estate matching, standard top-$C(\sigma) = \sum_{x \in A} w_x f(w_x).$8 recommendation overloads top brokers because capacity is ignored. The reported measurements show that in City A, when brokers serve more than 40 requests per day, average sign-up rates drop sharply from about 14.3%–27.5% down to 2.5%–17.8%, with statistically significant negative correlation between workload and sign-up rate ($C(\sigma) = \sum_{x \in A} w_x f(w_x).$9) (Wei et al., 2023). The top-1 broker serves 38.26 requests per day on average, compared with 3.18 for the average broker, or about 12.03× the average workload. Learned Assignment with Contextual Bandits (LACB) reframes the problem from recommendation to capacity-aware assignment, and on City A, 80.8% of brokers improve utility compared with Top-K; across real datasets, 72.0%–82.2% of brokers improve, while the optimized LACB-Opt is up to 284.9× faster than other KM-based methods (Wei et al., 2023).

Broker heterogeneity is a related theme in market microstructure. When broker identity is observed in order-book data, the response of prices can be decomposed into contributions from the same broker and from different brokers:

$x$0

The data show that brokers are very heterogeneous in liquidity provision, with the cross-broker distribution of the fraction of price-changing market orders ranging from near 0 to about 0.8 (Toth et al., 2011). The central empirical result is that the total impact of market orders is a near-cancellation between same-broker persistence and opposite-signed reactions from other brokers. This supports an ecological view of markets in which broker interaction, rather than a representative-agent abstraction, shapes observed impact (Toth et al., 2011).

Prediction work in OTC corporate bonds extends this heterogeneity perspective. Using FINRA TRACE data for 2015, dealer behavior is modeled as a sequence prediction problem, and the proposed Pointwise-Product ReZero Transformer achieves the best average F1 among the tested models, with reported cluster scores of 93.0, 88.3, 76.1, and 60.1 and an average of 82.3 (Lin et al., 2021). The paper also reports that individual history provides the best predictive model for the most active dealers, while cluster-level models work better for less active dealers. A plausible implication is that over-the-top brokerage in OTC settings is not only structural and strategic but also behaviorally learnable from historical flow.

Across these literatures, the over-the-top broker is best understood as an intermediary whose added value lies in decisions that ordinary brokerage leaves unspecified: enforcement of security levels beyond the first hop, clone placement and migration, blockchain-backed validation, scientific annotation and classification, slice admission and orchestration, internalization and externalization of flow, or capacity-aware assignment. The unifying feature is not the application domain but the elevation of the broker from a relay point to a policy-bearing control surface.