---
title: Segment Hysteresis RL for ISAC Systems
url: https://www.emergentmind.com/topics/segment-hysteresis-based-reinforcement-learning-shrl-algorithm
type: topic
---

# Segment Hysteresis RL for ISAC Systems

Segment Hysteresis Based Reinforcement Learning (SHRL) is a reinforcement learning algorithm specifically designed for integrated sensing and communication (ISAC) optimization within segmented waveguide-enabled pinching-antenna array (SWAN) systems. SHRL addresses the challenge of dynamic segment selection—critical for both communication throughput and sensing performance—by introducing a hysteresis mechanism that governs when segment allocations are updated, mitigating instability and improving overall system reward. The algorithm is formalized within a Markov decision process (MDP) framework, jointly optimizing transmit beamforming, segment selection, and antenna positioning [2601.20658].

## 1. Markov Decision Process Formulation

SHRL frames the SWAN-ISAC control problem as an MDP comprising the following components:

- **State Space**: At timestep $t$, the state is $s_t = [h_{k_c,t},\, h_{k_s,t},\, \psi_{t-1},\, \phi_{t-1}]$, where $h_{k_c,t}$ and $h_{k_s,t}$ denote instantaneous channel state information (CSI) for all $K_c$ communication users and $K_s$ sensing targets, $\psi_{t-1}$ is the previous pinching-antenna pose vector, and $\phi_{t-1}$ encodes the last segment selection assignments.
- **Action Space**: The agent outputs a tuple $a_t = [\psi_t,\, \phi_t,\, W_t]$, with $\psi_t \in \mathbb{R}^{M\times N\times 2}$ (antenna positions), $\phi_t \in \mathbb{R}^M$ (raw segment-selection logits), and $W_t \in \mathbb{C}^{K_c\times M}$ (beamforming weights). Segment assignments are executed as $\tilde\phi_t$ after processing through the hysteresis gate.
- **Transition Dynamics**: The system updates via $(\psi_t,\, \tilde\phi_t,\, W_t)$, yielding new CSI, illumination, and data-rate metrics, then transitioning to $s_{t+1}$. Channel evolution follows deterministic (e.g., line-of-sight) or stochastic models.
- **Reward Function**:

  $$
  r_t = \sum_{k_c=1}^{K_c} R_{k_c,t} - \sum_{k_s=1}^{K_s} \mathbb{I}\{\Gamma_{k_s,t} < \tilde\Gamma\}
  $$

  where $R_{k_c,t} = B\log_2\left(1 + \frac{\sum_m |h_{k_c,m}^T s_m|^2}{\sum_{m\ne m'} |h_{k_c,m}^T s_{m'}|^2 + \sigma^2}\right)$ is the communication rate for user $k_c$, and $\Gamma_{k_s,t}$ represents the expected illumination power for sensing target $k_s$; $\tilde\Gamma$ is a threshold, and violations incur a unit penalty per target.

## 2. Segment Hysteresis Mechanism

The defining feature of SHRL is the segment hysteresis mechanism. Rather than immediately reassigning segments upon every change in the policy network’s output logits $\phi_t$, SHRL applies a probabilistic or threshold-based gate. This mechanism stabilizes assignments and filters out minor or spurious fluctuations, thus suppressing rapid remapping:

- **Probabilistic Gate**: For each segment $m$:
  $$
  \tilde\phi_{m,t} = \begin{cases}
    \phi_{m,t}, & \text{with probability } p_{\text{update}} \\
    \tilde\phi_{m,t-1}, & \text{otherwise}
  \end{cases}
  $$
  where $p_{\text{update}}\in(0,1)$ regulates update aggressiveness. Smaller values yield more persistent segment allocations.
- **Threshold (Dead-band) Variant**:
  $$
  \tilde\phi_{m,t} = \begin{cases}
    \phi_{m,t}, & |\phi_{m,t} - \tilde\phi_{m,t-1}| \ge \delta_{\text{hys}} \\
    \tilde\phi_{m,t-1}, & \text{otherwise}
  \end{cases}
  $$
  This enforces assignment updates only for changes exceeding a hysteresis gap $\delta_{\text{hys}}$.

This mechanism avoids structural oscillation in segment-to-antenna mapping, which alleviates non-stationarity in the environment as perceived by the RL agent.

## 3. Learning Algorithm and Update Rules

SHRL employs the Advantage Actor–Critic (A2C) framework:

- **Actor** ($\pi_\theta(a|s)$): Parameterized by $\theta$, maps states to actions $(\psi_t,\, \phi_t,\, W_t)$.
- **Critic** ($V_\psi(s_t)$): Parameterized by $\psi$, estimates the value function.

Update rules are as follows:

- **Critic Loss**:

  $$
  \mathcal{L}_V = \big(V_\psi(s_t) - \hat{R}_t\big)^2, \qquad \hat{R}_t = r_t + \gamma V_\psi(s_{t+1})
  $$

- **Actor Loss**:

  $$
  \mathcal{L}_\pi = -\log \pi_\theta(a_t|s_t) \cdot \hat{A}_t, \qquad \hat{A}_t = \hat{R}_t - V_\psi(s_t)
  $$

Gradients with respect to $\theta$ and $\psi$ update the respective networks toward higher-reward policies.

**Pseudocode for main steps (abridged):**

```python
Initialize θ, ψ; set memory tildeφ₀ (e.g. uniform)
for episode=1…N_episodes:
  reset environment, observe s₀
  for t=0…T_max:
    aₜ = π_θ(sₜ)
    extract raw logits φₜ from aₜ
    for each segment m:
      with prob. p_update:
        tildeφ_{m,t} = φ_{m,t}
      else:
        tildeφ_{m,t} = tildeφ_{m,t-1}
    execute (ψₜ, tildeφₜ, Wₜ) in env
    observe rₜ, s_{t+1}
    compute hatRₜ = rₜ + γ V_ψ(s_{t+1})
    compute Aₜ = hatRₜ - V_ψ(sₜ)
    update critic via ∇_ψ (V_ψ(sₜ) - hatRₜ)²
    update actor via ∇_θ [ -log π_θ(aₜ|sₜ)·Aₜ ]
  end for
end for
return θ*, ψ*
```

## 4. Hyperparameters and Training Protocol

SHRL is trained using the following key hyperparameters and procedural setup:

| Parameter               | Value/Range                      | Description                                    |
|-------------------------|----------------------------------|------------------------------------------------|
| Actor LR ($\alpha_{actor}$)   | $1\!\times\!10^{-4}$              | Step size for actor network                    |
| Critic LR ($\alpha_{critic}$) | $1\!\times\!10^{-3}$              | Step size for critic network                   |
| Discount ($\gamma$)            | 0.99                             | Return discount factor                         |
| Hysteresis $p_{update}$        | {0.05, 0.1, 0.2}                 | Grid searched                                  |
| Exploration             | Additive Gaussian noise          | Applied to outputs during training              |
| Episode length          | $T_{max}=100$                    | Per episode or until fixed geometry eval        |
| Convergence detection   | $<$ $10^{-3}$ reward improv/200 episodes | Via moving-average (window=50)                 |

Episodes are initialized with randomized user and target positions. Convergence is declared when the moving-average reward plateaus under a specified threshold for a sustained interval.

## 5. Comparative Performance and Ablation

The effectiveness of SHRL is evaluated against several baselines: standard A2C (without hysteresis), SPRL (periodic segment-update every 5 steps), PPO (proximal policy optimization), and a random action agent.

Key evaluation highlights:

- SHRL achieved the highest final reward (≈13.2), outperforming A2C (≈9.9), SPRL (≈7.7), PPO (≈6.4), and random policies (≈3.0).
- SHRL converged within approximately 4000 episodes, exhibiting the smallest reward variance.
- The hysteresis mechanism prevented the reward collapses and non-stationarity seen in standard A2C/PPO. 
- For waveguide configuration $M=3$, $L_{wg}=40\,\mathrm{m}$, SHRL yielded 13.15 bps/Hz for rate and $1.09\!\times\!10^{-5}$ for illumination, outperforming all tested alternatives. For increased $L_{wg}$, SHRL continued to improve the rate-illumination trade-off.
- Compared to A2C, SHRL achieved approximately 30% higher final rate and required about 50% fewer episodes to converge.
- Compared to SPRL with fixed update intervals, SHRL’s adaptive hysteresis delivered a ~70% higher reward ceiling and smoother training.

## 6. Significance and Implications

Segment Hysteresis Based Reinforcement Learning introduces a principled module for stabilizing assignment dynamics in RL-based resource control for ISAC, particularly when assignment policies are sensitive to minor environmental or agent perturbations. The probabilistic hysteresis gate, governed by $p_{update}$, effectively reduces environmental non-stationarity as perceived by the RL agent. This yields efficient convergence and robust performance, particularly relevant in complex multi-objective wireless systems where aggressive segment remapping is inefficient or detrimental [2601.20658].

A plausible implication is that similar hysteresis primitives may generalize to other RL-controlled resource allocation problems plagued by instability or oscillatory structural decisions.

## 7. Related and Future Directions

SHRL represents a step beyond baseline A2C and PPO approaches by incorporating memory into the segment assignment policy via hysteresis. This approach is specifically tailored for SWAN-ISAC system design but may be extensible to other domains requiring persistent control actions or delayed adaptation. Future research may investigate the generalization of hysteresis-based RL modules to broader classes of resource allocation, the exploration of adaptive or learned hysteresis parameters, and integration with more advanced actor-critic architectures.

Source: https://www.emergentmind.com/topics/segment-hysteresis-based-reinforcement-learning-shrl-algorithm