Graph Transformers and Stabilized Reinforcement Learning for Large-Scale Dynamic Routing Modulation and Spectrum Allocation in Elastic Optical Networks

Abstract: Reinforcement learning (RL) has been widely applied to dynamic routing, modulation and spectrum assignment (RMSA) in optical networks, yet no prior work has trained a transformer model for this task. We attribute this to the high data and compute requirements of transformers and potential training instabilities with RL. We address this gap by combining recent advances from the machine learning literature (rotary positional encodings for graph-structured data, off-policy invalid action masking, and valid mass regularization) with GPU-accelerated simulation to achieve, for the first time, stable RL training of a transformer for dynamic RMSA. We demonstrate, through systematic benchmarking against previous RL methods and heuristic algorithms, that ours is the first RL method to exceed all benchmarks, increasing the supportable traffic load by up to 13\%. To demonstrate the scalability of our approach, we train on real network topologies from the TopologyBench database up to 143 nodes and 362 links, with 320 x 12.5\,GHz frequency slot units per link, and 100\,Gbps traffic requests. To our knowledge, these are the largest dynamic RMSA problems to which RL has been applied. We find up to 4\% increased traffic load can be supported at low blocking probability (<0.1\%) with our method compared to the best available benchmark algorithm. We present an ablation study of the components of our training algorithm, the dynamics of the loss function during training, and analyze the allocation decisions of the trained models. We make all code used to produce this paper openly available for reproduction and future benchmarking: https://github.com/micdoh/XLRON.

• The paper introduces a graph transformer with stabilized RL to overcome limitations of previous methods, achieving up to 13% higher traffic support compared to heuristics.
• The methodology uses GPU-accelerated simulations, tokenized network state representations, and wavelet-induced rotary encodings to capture spatiotemporal dynamics.
• Ablation studies confirm that off-policy invalid action masking and valid mass regularization are essential for stable convergence and optimal performance.

Graph Transformers and Stabilized RL for Large-Scale Dynamic RMSA in Elastic Optical Networks

Introduction and Problem Context

The paper addresses the application of transformer-based reinforcement learning to dynamic Routing, Modulation and Spectrum Assignment (RMSA) in Elastic Optical Networks (EONs). Prior RL methodologies for RMSA have largely failed to outperform optimized heuristic algorithms, either due to insufficient architectural expressivity or instability inherent in training with policy gradients and invalid action masking. Despite the transformative impact of transformers in other domains, their successful deployment for dynamic RMSA had not been realized due to their data/computational demands and the unique instabilities posed by the RL setting.

This work delivers the first evidence that a transformer—enhanced with graph-specific inductive biases and stabilized RL training—can consistently outperform both established RL methods and optimized heuristics on RMSA, delivering up to 13% higher supportable traffic load on standard benchmarks and demonstrating state-of-the-art scalability on network topologies up to 143 nodes and 362 links.

Figure 1: The XLRON training pipeline: GPU-based, multi-environment experience generation, a graph-structured transformer agent, and a suite of RL stabilization strategies.

Architecture and Methodology

Simulation Framework and Data Flow

A foundational contribution is XLRON, a GPU-accelerated simulation and training architecture enabling scaling to millions of transitions per second with parallelized environments. The simulation exposes, at each time step, a tokenized representation of the entire network: each token corresponds to a link, with occupancy described as normalized remaining holding times (rather than binary occupancy). This vectorized encoding provides the agent with rich spatiotemporal state.

Graph Transformer Model

Figure 2: Actor-critic Graph Transformer: Attention over link tokens, with pooling tailored for actor and critic pathways.

The agent is structured as a graph transformer employing full self-attention over link tokens. Two distinct readouts are employed:

• Actor: Pools embeddings along candidate lightpaths using min/mean/max statistics, facilitating action selection over all valid route-spectrum combinations.
• Critic: Aggregates link information through attention pooling with a learned query, optimizing value estimation sensitivity to bottleneck links.

Critical architectural advances enabling effective transformer application to RMSA include:

• Pre-LayerNorm transformers for gradient stability.
• Wavelet-Induced Rotary Encodings (WiRE) for graph-structured positional information.

Figure 3: Wavelet-Induced Rotary Encoding (WiRE): Spectral graph features enable self-attention to encode spatial relationships in arbitrary topologies.

Training Stabilization Techniques

Instabilities from invalid action masking in policy-gradient RL—where the ratio between masked and unmasked policy probabilities can collapse—are confronted by:

• Off-policy invalid action masking: Differentiates through masked actions (as per Hou et al.), improving feature learning and accelerating convergence.
• Valid mass regularization: Penalizes log-barrier loss for insufficient valid-action probability, anneals influence over time, and employs per-step loss damping based on valid mass.
• Hard gating: Excludes training transitions with an insufficient number of valid actions, limiting the influence of degenerate states.

Collectively, these enable large-scale, stable transformer-RL training, as established by systematic ablations.

Comparative Evaluation: Standard Topology Benchmarking

Benchmarking is conducted against the canonical four topologies (NSFNET, COST239, USNET, JPN48) and five highly-cited RL methods, with simulation parameters strictly matched for fidelity. Heuristic baselines are optimized using extensive candidate paths and sort criteria.

Figure 4: Service blocking probability vs. traffic load: Graph Transformer RL surpasses all previous RL methods and matches or exceeds the optimized heuristic references and upper bound estimates on all standard topologies.

Key findings include:

• All prior RL methods are outperformed by strong heuristics; the Graph Transformer is the first RL-based agent to exceed heuristic capacity in every benchmark.
• On USNET with PtrNet-RSA-80 settings, the method supports 13% higher load at sub-0.1% blocking probability—a significant margin relative to the tight cut-set and defragmentation bounds.
• On COST239, the method achieves lower blocking than even the defragmentation upper bound, suggesting the discovery of globally superior routing/spectrum policies.
• Across topologies, the gap to upper bounds (mainly due to spectrum continuity constraints) is minimal, with Graph Transformer RL narrowing the achievable-vs-ideal gap beyond prior art.

Scalability: Large Topology Experiments

Application is extended to TopologyBench networks (TataInd: 143 nodes, USA100: 100 nodes), representing the largest RMSA-RL deployments in literature.

Figure 5: Large-scale topologies: TataInd (143 nodes) and USA100 (100 nodes).

Key experimental insights:

• FF-KSP with large K is established as the strongest heuristic, yet is surpassed by the trained agent.
• Stable, efficient RL training is achieved on both networks (training time ${\sim}$4–5 hours on a single H100), made possible by GPU-based batched simulation and the full suite of stabilization techniques.

Ablation and Training Dynamics

Figure 6: Ablations: Removing off-policy IAM, valid mass loss, or per-step loss damping breaks convergence or degrades performance.

• Off-policy invalid action masking and valid mass regularization are indispensable for avoiding policy collapse and ensuring convergence.
• Reliable and monotonic loss evolution persists throughout training; entropy regularization (cosine annealing) aids exploration and deterministic policy refinement.

Figure 7: Loss decomposition: Actor loss leads, valid mass and value loss reflect appropriate regularization and environmental stochasticity.

Performance, Path, and Spectral Allocation Analysis

Blocking Probability

Figure 8: The Transformer agent achieves lower blocking than the FF-KSP heuristic. At 0.1% blocking, the agent supports 4% higher load (USA100) and 3% (TataInd).

Transient and Steady-State Behavior

Figure 9: Bitrate-centric blocking over a single episode: RL agent advantage emerges and persists in steady-state operation.

Path Length and Routing Diversity

Figure 10: Mean path length (km and hops): Transformer selects shorter paths on USA100, comparable lengths on TataInd.

Figure 11: Per-request path difference: USA100 sees consistent RL-driven path shortening; TataInd shows symmetric path selection shifts.

Figure 12: Distribution of assigned path lengths: The RL-trained agent avoids excessive path stretching, favoring spectrally efficient, shorter routes.

Spectrum Resource Utilization

Figure 13: FSU occupancy delta: Transformer achieves more distributed spectrum allocation compared to the low-index bias of FF-KSP.

Figure 14: Per-link usage shift: RL agent reduces reliance on most links (USA100), reflecting more efficient traffic engineering; TataInd shows a modest, targeted redistribution.

Theoretical and Practical Implications

This work establishes that:

• RL-based transformers, with appropriate graph inductive bias (via WiRE), invalid action handling, and stabilized training, can discover RMSA policies that fully exploit spatial and spectral diversity in EONs.
• Transformers’ expressiveness enables them to encode and utilize comprehensive topological and resource state—outperforming heuristically designed path/spectrum selection paradigms, especially as network size and action-space expand.
• Systematic advancements in RL instability mitigation (notably valid mass anchoring and action masking) are crucial for tractable RL in extremely large combinatorial spaces typical of RMSA/EON contexts.

Potential directions for further research include:

• Extension to SNR-aware and multi-objective RMSA: The architecture is compatible with incorporating real-time or analytically derived QoT constraints, thereby enabling direct optimization over throughput, reach, and energy objectives.
• Advance in environment-model realism and complexity: Multi-band, multi-core, or dynamically reconfigurable networks, with full physical layer modeling, could fully realize the architectural generality.
• Transfer learning and fine-tuning for real deployments: Given the tokenized and parallelized setup, foundation model approaches could be targeted at RMSA for rapid cross-topology adaptation.

Conclusion

This work conclusively demonstrates that a carefully constructed transformer—equipped with structural graph biases via spectral rotary encodings and advanced RL training procedurals—delivers domain-dominant RMSA performance across standard and large-scale EONs. The open-sourced ecosystem further positions this paradigm as a new baseline for dynamic optical network resource allocation research and deployment.