Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration

Abstract: Accurate network traffic prediction is a critical element for efficient resource allocation in dynamic urban cellular networks. However, prediction remains challenging because network demand is influenced by complex mobility patterns, congestion dynamics, and heterogeneous user behavior. This paper introduces the Parameter-Efficient Hybrid Transformer (PEHT), a network traffic prediction framework that integrates urban mobility and congestion information into a Transformer-based architecture. PEHT separates primary network communication features from secondary urban mobility features and incorporates Low-Rank Adaptation (LoRA) into the Transformer encoder to reduce the number of trainable parameters while maintaining high predictive accuracy. A multimodal fusion strategy then injects external mobility and congestion features into the decoder to improve traffic forecasting. Experiments on the Telecom Italia Milan dataset and multiple synthetic congestion scenarios show that PEHT outperforms state-of-the-art baselines in terms of RMSE, MAE, and $R^2$. The implementation is available in the GitHub repository.

• The paper introduces PEHT, a novel hybrid Transformer that integrates LoRA for parameter efficiency to reduce model size without sacrificing accuracy.
• It fuses cellular metrics with urban congestion data through a dual-stage encoder-decoder architecture, enhancing predictive performance even on edge devices.
• Experimental results reveal significant RMSE reductions and high R² scores, validating low-rank constraints and late fusion strategies for robust forecasting.

Parameter-Efficient Hybrid Transformer for Network Traffic Forecasting via Urban Congestion Integration

Overview

The Parameter-Efficient Hybrid Transformer (PEHT) introduces a novel paradigm for cellular network traffic prediction that unites Transformer-based neural models with real-time urban mobility and congestion features. PEHT is architected to address the dual challenges of high-dimensional input and parameter explosion commonly faced by Transformer methods and the insufficient integration of external spatiotemporal signals in conventional approaches. The framework operates by partitioning features into primary network communication indicators and secondary urban mobility/congestion vectors, leveraging LoRA-based parameter efficiency within the encoder, and employing a late fusion strategy for external features downstream in the decoder.

Figure 1: The PEHT architecture, incorporating LoRA in the encoder and fusing vehicle mobility features for decoder input.

Motivation and Related Work

Contemporary network traffic forecasting within urban cellular environments necessitates accurate modeling of highly dynamic, heterogeneous spatiotemporal phenomena. Prior art predominantly falls into two families: GNN-based models and Transformer-based models. GNNs [zhang2024cellular, guo2023capturing] capture spatial graph dependencies but are limited by scalability and often fail to integrate exogenous urban context without violating temporal causality. Transformer-based architectures [liu2021st, zhang2023st2t] are capable of learning long-range dependencies but impose significant computational and parameter overhead, especially when the dimensionality of multimodal input grows.

Neither approach sufficiently bridges the gap between efficient parameterization and the robust, causal integration of real-world congestion signals. PEHT is specifically designed to encapsulate both efficiency and external data integration, leveraging LoRA for parameter reduction and a principled fusion mechanism for auxiliary signals.

Methodological Framework

Preprocessing and Feature Construction

PEHT initiates by remapping raw Milan cellular grid data to a virtual base station (VBS) space via a spatial-temporal clustering routine. This clustering maximizes within-cluster correlation of temporal patterns while minimizing spatial and load variance, thus reducing input sparsity while maintaining meaningful mobility granularity. The result is a multivariate time series for each VBS, synthesizing traffic volumes across communication categories (SMS, Call, Internet) and embedding both grid topology and load characteristics.

Multimodal Embedding and Input Fusion

Feature vectors are constructed per VBS to ingest multiple modalities: historical traffic ($X_\text{hist}$), time covariates, static network meta-data, and fully dynamic external congestion features. These are converted to numerical tensors using both continuous and categorical embedding layers and joined with positional encodings for sequential modeling. Fusion occurs at two levels: (1) initial feature concatenation forms the encoder's input; (2) encoder outputs are subsequently merged with urban mobility/congestion features before entering the decoder, establishing explicit context for predictive inference.

Encoder-Decoder Architecture and LoRA

At the model core is a Transformer encoder-decoder stack. The encoder operates on a context window of historical fused features, producing contextualized representations using self-attention. The decoder predicts future series autoregressively, conditioning each step solely on past context and known covariates, thereby preserving temporal causality. Crucially, LoRA decompositions are embedded directly in the encoder's multi-head attention and feedforward modules at initialization—not mere fine-tuning—substantially reducing the parameter footprint (>90%) without degrading predictive efficacy. Weight updates in high-dimensional layers are restricted to low-rank manifolds parameterized by $A$ and $B$ for each $W$, with rank hyperparameters carefully tuned for balance.

External Feature Fusion and Causality

External signals (cellular congestion, vehicle/pedestrian flow) are injected after encoder processing via vector concatenation, timed to only reflect historical or autoregressive/estimated future values—strictly adhering to causal masking constraints. This integration injects critical urban context without the leakage of ground-truth future data to the decoder, a frequent pitfall of naive feature fusion in time series models.

Experimental Results

Numerical Evaluation

Experiments are conducted on the Telecom Italia Milan dataset augmented with synthetically generated congestion/mobility scenarios for ablation. The PEHT model, implemented with a sequence length of 48, hidden size 128, three encoder layers, and LoRA rank $r=8$, is compared to state-of-the-art baselines including HGCRN, ST2T, and MVSTGN.

Empirically, PEHT demonstrates substantial improvements:

• RMSE reduction in SMS prediction by 14.6% (18.42 vs 21.58)
• 11.9% RMSE reduction for Internet traffic compared to ST2T
• $R^2$ coefficients exceeding 0.96 in Internet traffic prediction

This superior performance validates that low-rank constraint via LoRA not only regularizes against noise but maintains expressivity, while decoder fusion robustly incorporates exogenous dynamics.

Hardware Deployment

PEHT's efficient parameterization enables deployment on edge devices with stringent memory/compute constraints.

Figure 2: NVIDIA Jetson AGX Orin hardware, supporting real-world, model-efficient deployment.

Ablation Analysis

The ablation study isolates the contribution of Encoder Fusion, LoRA, and their combination (the Full Model) on five synthetic datasets representing distinct congestion and mobility regimes. LoRA alone outperforms Encoder Fusion on most metrics; however, the full fusion achieves best aggregate performance and consistency, reaffirming the complementarity of efficient encoding and robust context integration.

Figure 3: Loss trajectories for Full Model, Encoder Fusion-only, and LoRA-only variants, highlighting the impact of LoRA.

Parameter count analysis shows a drastic reduction: for input/output dimension 4,372 and $r=8$, LoRA reduces parameters from over 19 million to below 70,000—a >99% savings—without notable performance loss.

Theoretical and Practical Implications

The PEHT framework demonstrates that advanced parameter-efficient tuning (i.e., LoRA) is potent when applied in the initial model training, not merely fine-tuning, within urban network forecasting domains characterized by high-dimensional, noisy inputs. This observation supports broader theoretical claims about the generalization properties of low-rank constraints in spatiotemporal transformers.

The late fusion strategy exemplifies best-practice for causality preservation in sequential prediction, informing future designs of real-time forecasting architectures in both cellular and urban computing tasks. The architecture facilitates practical, scalable deployment in edge computing settings, enabling cellular operators to deploy high-capacity forecasting without prohibitive compute or memory budgets.

Conclusion

The work establishes PEHT as a robust and highly parameter-efficient baseline for cellular network traffic prediction, setting a benchmark for methods that must accommodate high-dimensional, multimodal spatiotemporal input. Its results demonstrate the viability and effectiveness of LoRA when incorporated directly into core model training, and highlight the critical role of causally-aware external data fusion in advancing both predictive accuracy and real-world deployability. The architecture provides a reproducible template for future advances in integrating network analytics with urban sensing data streams.