U-HPNF: Underwater High-Performance Network Framework

• U-HPNF is a hierarchical framework that integrates DRL, federated learning, and a two-level digital twin design to ensure self-management and self-healing in underwater networks.
• It employs a three-tier architecture to separate local autonomous decision-making, subnet-level adaptation with lightweight DTs, and global policy management at the data center.
• The framework addresses challenges like propagation delays, limited bandwidth, and energy constraints, demonstrating enhanced performance compared to traditional methods.

U-HPNF, short for Underwater High-Performance Maintenance Network Framework, is a hierarchical framework for underwater communication networks within space-air-ground-aqua integrated networks (SAGAIN). It is designed to build a high-performance maintained underwater network with self-management, self-configuration, self-optimization, and, in the paper’s conclusion, self-healing capabilities by combining deep reinforcement learning (DRL), federated learning (FL), and a two-level digital twin (DT) design (Gou et al., 18 Aug 2025). The framework addresses the fact that underwater communication networks are constrained by long propagation delay, limited network capacity, time-varying and unstable link quality, and energy limitations, all of which make the underwater segment a bottleneck for end-to-end service quality in SAGAIN (Gou et al., 18 Aug 2025).

1. Concept and problem setting

U-HPNF is proposed for underwater communication networks in which conventional static protocol design is insufficient for dynamic, resource-constrained, and failure-prone environments. The motivating claim is that underwater networks must adapt not only to changing channel and topology conditions but also to changing QoS requirements, while avoiding excessive communication overhead and limiting privacy leakage from exchanging raw observations (Gou et al., 18 Aug 2025).

The framework is centered on three resource classes explicitly named in the paper: communication bandwidth, computational resources, and energy supplies. Its purpose is not merely local link optimization; it is intended as a network-maintenance framework that continuously preserves performance under node failures, subnet changes, and shifts in high-level optimization goals such as network capacity versus communication fairness (Gou et al., 18 Aug 2025).

A central premise of U-HPNF is that the underwater segment of SAGAIN requires an AI-native management architecture. This means that decision-making, model updating, and scenario emulation are built into the network stack itself rather than treated as external offline tools. The paper repeatedly frames the target system as an underwater high-performance maintained network (U-HPNet), with U-HPNF as the framework that realizes it (Gou et al., 18 Aug 2025).

2. Hierarchical architecture and two-level digital twins

U-HPNF uses a three-tier network design together with two levels of digital twins (Gou et al., 18 Aug 2025). The three layers differ in capability and responsibility.

Layer	Main entities	Main role
Underwater perception layer	sensors, submarines, AUVs	local sensing and DRL-based decisions
Intelligent sink layer	buoys, surface gliders, USVs, ship-based platforms	subnet scheduling, parameter updates, lightweight DT
Intelligent aggregation layer	data center	network-wide coordination, versatile DT, online optimization

At the underwater perception layer, each node contains four internal components: a P-Module, a DM-Module, a Net-Module, and a replay buffer. The P-Module perceives node state and channel state through built-in energy, channel, request, and traffic models. The DM-Module applies a DRL-based policy network to these observations and outputs transmission configurations. The Net-Module executes the selected communication behavior, and the replay buffer stores transition tuples for local training when resources allow (Gou et al., 18 Aug 2025).

At the intelligent sink layer, more capable platforms act as cross-domain gateways and subnet coordinators. Their responsibilities include sub-network level resource scheduling, data fusion, dynamic network topology reconfiguration, and node-model updating when subnet composition changes or nodes fail. This layer hosts a lightweight network digital twin, which is used to mimic local scenarios and update model parameters without requiring raw node data (Gou et al., 18 Aug 2025).

At the intelligent aggregation layer, implemented in a data center, U-HPNF performs network-level orchestration. This layer maintains a more versatile DT system, runs online network optimization, performs what-if analysis, reacts to changing QoS requirements, and aggregates subnet-level behaviors into a network-wide policy. The paper distinguishes this top-level DT from the sink-level DT by scope and fidelity: the sink-level DT is for fast local adaptation, whereas the aggregation-layer DT supports broader scenario emulation and objective-driven policy redesign (Gou et al., 18 Aug 2025).

This layered organization suggests a division between decentralized execution at underwater nodes, subnet adaptation at sinks, and global policy management at the data center. A plausible implication is that U-HPNF is designed to preserve responsiveness without requiring every decision to traverse the entire hierarchy.

3. DRL, FL, and DT as the core mechanisms

The local decision mechanism in U-HPNF is DRL. The paper states that DRL is used to manage limited underwater-network resources, including bandwidth, computation, and energy, and in the evaluation the concrete decision variable is primarily transmission power, with the action also implicitly deciding whether a node transmits in a slot (Gou et al., 18 Aug 2025). Because the action set includes $0\text{ W}$, the agent jointly controls activity and power level.

The federated component is FL, introduced to iteratively optimize the decision-making model while reducing communication overhead and protecting privacy. U-HPNF does not upload local observations or replay data; it transfers model parameters only. The paper states explicitly that “only model parameters are transferred” and that local node observations and model structure are not shared (Gou et al., 18 Aug 2025). This gives FL a dual role: reducing acoustic communication burden and limiting exposure of local state information.

The third mechanism is the digital twin. U-HPNF deploys DTs at both the sink and aggregation layers. The sink-layer DT supports local adaptation, especially under node failures and topology changes. The aggregation-layer DT supports broader scenario simulation, offline preparation of optimization models, and online what-if analysis when QoS requirements change (Gou et al., 18 Aug 2025). The paper also motivates DT use by acknowledging the underwater sim-to-real gap, attributed to factors such as temperature, turbulence, ambient noise, and other time-varying effects.

The interaction among these three components is structurally important. DRL provides the policy model, FL provides the update pathway, and DT provides scenario generation and adaptation support. The paper further states that the sink layer extends node-level behavior from an independent Q-learning (IQL) style toward centralized training with decentralized execution (CTDE) (Gou et al., 18 Aug 2025). This places U-HPNF within a multi-agent learning regime in which local policies are coordinated rather than purely selfish.

4. Decision model and implementation details

The paper is mathematically lightweight, but it specifies the core control setting used in evaluation. Each node chooses transmit power from the discrete set

$[0, 2, 4, 8, 16, 64]\ \text{W},$

where $0\text{ W}$ means the node does not transmit in the current slot (Gou et al., 18 Aug 2025). In the reported experiments, transmission slots have duration $10\,\text{s}$ (Gou et al., 18 Aug 2025).

At the node level, the DM-Module uses a recurrent DRL architecture with the following layers: a first fully connected layer with 64 hidden units, a GRU with 64 hidden units and ReLU, and a second fully connected layer with 7 hidden units generating Q-values for available actions (Gou et al., 18 Aug 2025). The paper also compares U-HPNF against IQL, uses a target network, a replay buffer, and an $\varepsilon$-greedy policy, so the implemented learner is best understood as a Q-learning-based recurrent deep RL model, although the paper does not give a formal algorithm name (Gou et al., 18 Aug 2025).

The reported training hyperparameters are:

• replay buffer size $B_y = 10{,}000$
• mini-batch size $b = 32$
• total training episodes $= 300{,}000$
• target network update interval $= 200$ episodes
• discount factor $\gamma = 0.99$
• $[0, 2, 4, 8, 16, 64]\ \text{W},$0 decays linearly from $[0, 2, 4, 8, 16, 64]\ \text{W},$1 to $[0, 2, 4, 8, 16, 64]\ \text{W},$2 over the first $[0, 2, 4, 8, 16, 64]\ \text{W},$3 episodes, then remains at $[0, 2, 4, 8, 16, 64]\ \text{W},$4 (Gou et al., 18 Aug 2025)

The local state observed by the P-Module is described in terms of energy, channel, request, and traffic models rather than an explicit Markov state vector (Gou et al., 18 Aug 2025). This suggests that the practical learning problem is at least partially observable, which is consistent with the use of a GRU.

At the network-objective level, the paper evaluates two optimization goals: Model I, which maximizes network capacity, and Model II, which optimizes communication fairness (Gou et al., 18 Aug 2025). Network capacity is stated to be calculated using the Shannon formula as described in the cited reference, while fairness is defined using Jain’s fairness index (Gou et al., 18 Aug 2025). The exact formulas are not typeset in the paper, but the objective distinction is explicit and central to the framework’s adaptive behavior.

5. Learning workflow and adaptive control loop

Operationally, U-HPNF follows a layered training-and-control cycle. At the underwater node, the P-Module observes local conditions, the DM-Module selects an action, the Net-Module executes it, and the resulting transition is stored in the replay buffer. If the node has sufficient energy and processing capability, it performs local training using replayed experience (Gou et al., 18 Aug 2025).

Instead of sending raw data upward, the node uploads model parameters. These are aggregated first at the subnet level and then at the network level. At the intelligent sink layer, the lightweight DT simulates local scenarios such as node failures and topology changes, enabling rapid parameter revisions that preserve subnet performance. At the intelligent aggregation layer, the data-center DT updates virtual nodes, combines subnet-level policies into a network-wide joint behavior policy, and performs online optimization and what-if analysis (Gou et al., 18 Aug 2025).

This results in a multi-stage control loop:

1. local perception and DRL decision-making at underwater nodes
2. local execution and replay-buffer storage
3. local training when resources allow
4. federated parameter upload
5. sink-level aggregation and DT-assisted subnet adaptation
6. data-center aggregation, DT-based scenario analysis, and policy revision
7. redistribution of updated parameters downward (Gou et al., 18 Aug 2025)

The paper makes an important distinction between operational disturbances and objective changes. Sink-level adaptation is presented as sufficient for many topology and failure events, but when the high-level scheduling objective changes, the corresponding optimization model must be retrained (Gou et al., 18 Aug 2025). This means U-HPNF does not claim complete objective invariance; instead, it separates local resilience from global goal reconfiguration.

6. Evaluation, empirical behavior, and limitations

The evaluation is simulation-based. The underwater network is modeled in a cylindrical region with radius $[0, 2, 4, 8, 16, 64]\ \text{W},$5 and height $[0, 2, 4, 8, 16, 64]\ \text{W},$6, with transmitters placed at the bottom and receivers at the top (Gou et al., 18 Aug 2025). Two classes of experiments are reported: subnet-level robustness under node failures and network-level adaptation under changing optimization goals.

In the subnet-level case, a five-node subnet is trained to maximize the number of concurrent communications, reported as network reuse. Each underwater node may fail with probability $[0, 2, 4, 8, 16, 64]\ \text{W},$7, modeled as a Bernoulli distribution, after which it transmits at random power regardless of global consequences (Gou et al., 18 Aug 2025). The tested failure probabilities are $[0, 2, 4, 8, 16, 64]\ \text{W},$8, $[0, 2, 4, 8, 16, 64]\ \text{W},$9, $0\text{ W}$0, and $0\text{ W}$1 (Gou et al., 18 Aug 2025).

The framework is compared against Greedy, TDMA, Random, and IQL baselines. The main reported findings are that Greedy performs poorly in dense interference settings, TDMA is brittle under failures, Random is more robust than TDMA but still inferior, and IQL improves performance but remains weaker than U-HPNF because it optimizes individual rather than coordinated network behavior (Gou et al., 18 Aug 2025). Among U-HPNF variants, the best performance under increasing failure rates is obtained by the w/i responsive version, which is periodically updated through the sink layer, supporting the value of the sink-level DT-assisted adaptation mechanism (Gou et al., 18 Aug 2025).

In the network-level case, the paper examines node counts $0\text{ W}$2, $0\text{ W}$3, and $0\text{ W}$4, under the two high-level models. At $0\text{ W}$5, capacity-oriented and fairness-oriented models behave similarly, but the tradeoff sharpens as $0\text{ W}$6 increases. At $0\text{ W}$7, the fairness-oriented model yields 17,779.38 kb, which the paper states is 40% less than the capacity-oriented model (Gou et al., 18 Aug 2025). This quantifies the cost of fairness in a denser network and illustrates why model switching at the aggregation layer matters.

Several limitations are also explicit or strongly implied. The failure model is simplified to Bernoulli-driven random behavior. The paper does not provide a formal MDP specification, an explicit FL aggregation equation, or a convergence analysis for the combined DRL-FL-DT system (Gou et al., 18 Aug 2025). It also does not numerically quantify communication-overhead reduction or privacy gains, even though both are major motivations. The evaluation scope is relatively small, centered on slot-based power control and modest network sizes.

More fundamentally, U-HPNF does not claim universal adaptation to arbitrary policy changes without retraining. The paper explicitly states that when high-level scheduling objectives change, the optimization model must be retrained (Gou et al., 18 Aug 2025). This suggests that the framework is adaptive within a structured set of operating regimes rather than fully objective-agnostic.

7. Position within underwater network management

U-HPNF is best understood as a hierarchical AI-native management framework for underwater networking rather than a single scheduling algorithm. Its distinctive feature is the coordinated use of DRL for local decisions, FL for parameter exchange, and two-level DTs for local and global adaptation (Gou et al., 18 Aug 2025). The framework’s significance lies in how it decomposes underwater-network intelligence across node, sink, and data-center layers.

The paper’s strongest architectural claim is that underwater networks require more than local autonomy: they need subnet-level repair and reconfiguration plus network-level objective management. This is why U-HPNF separates underwater agents, intelligent sinks, and the aggregation layer instead of collapsing all learning into a single controller (Gou et al., 18 Aug 2025).

In that sense, U-HPNF occupies a specific position in the SAGAIN literature. It treats underwater communication not merely as a difficult physical channel, but as a managed cyber-physical subsystem requiring continual model updating, coordinated decision-making, and virtualized scenario testing. A plausible implication is that its long-term relevance depends less on the particular DRL instantiation than on the architectural principle of combining distributed decision-making, federated updating, and multi-level digital twins in a single underwater-network control stack.