Benchmarking Transformer and xLSTM for Time-Series Forecasting of Heat Consumption

Abstract: Obtaining an accurate short-term forecasting for heat demand is an essential part of operating district heating networks cost-efficient and reliable. Heat consumption time series at the building level are highly dependent on exogenous variables such as outdoor temperature and individual usage patterns, making forecasting in this context a challenging task. Thus, this paper benchmarks novel Transformer-based and xLSTM architectures for short-term heat-demand forecasting. Using hourly data from 25 German buildings (2017-2025), we compare three-hour and 24-hour forecasting horizons relevant for intraday control and day-ahead scheduling. We establish a multi-building benchmark that tests whether models trained on pooled, heterogeneous building data are able to generalize across diverse building stock. The results show that the xLSTM achieves the lowest RMSE (19.88 kWh for three-hour, 21.47 kWh for 24-hour forecasts), while the Temporal Fusion Transformer attains the best MAE (9.16 kWh for three-hour forecasts). As xLSTMs and Transformers require long training times and have a huge number of trainable parameters, their sustainability remains questionable. Therefore, this paper further investigates the trade-off between predictive accuracy and computational resource demand of the evaluated forecasting models. The findings indicate that also low-parameter models like a traditional fully-connected network achieve good predictive results, highlighting that marginal accuracy gains of the novel prediction models come at substantial resource expense for this use case.

• The paper demonstrates that xLSTM achieves the lowest RMSE in both short-term and day-ahead heat demand forecasting, outperforming traditional models.
• Advanced benchmarks reveal that accuracy gains (8.8%-11.4% RMSE improvement) come with increased computational demands and higher CO₂ emissions.
• The study highlights a trade-off between predictive performance, as seen with TFT's superior MAE, and computational sustainability in real-world deployments.

Benchmarking Transformer and xLSTM Architectures for Short-Term Heat Consumption Forecasting

Introduction

Accurate short-term heat demand forecasting is critical for sustainable, cost-efficient operation of district heating networks. Building-level heat consumption time series are characterized by dependency on exogenous variables (e.g., outdoor temperature) and highly individual usage patterns, posing significant challenges for predictive modeling. This study systematically benchmarks advanced neural architectures—including xLSTM and Transformer variants—against strong traditional baselines (FCN, LSTM) for both intraday and day-ahead heat demand forecasting. The benchmark leverages a pooled dataset from 25 diverse German buildings spanning 2017–2025 and scrutinizes models not only for forecasting performance but also for computational sustainability.

Model Architectures and Dataset

The benchmark comprises four model families: FCN (fully connected network), LSTM (traditional recurrent), xLSTM (extended memory recurrent), Transformer Encoder (TE), and Temporal Fusion Transformer (TFT). All models were evaluated using standardized inputs—including historical consumption, weather, calendar, and building features—across three-hour and 24-hour prediction horizons. The dataset consists of over 136k hourly data points after preprocessing, with rigorous outlier removal and feature engineering.

The xLSTM architecture, implemented as described by Beck et al., utilizes both mLSTM and sLSTM blocks following an up-projection of standard input features, supporting enhanced long-range pattern capture and partial parallelism.

Figure 1: Architecture of the adapted xLSTM, integrating memory blocks and feature up-/down-projection.

Forecasting Performance

Three-Hour Horizon

For three-hour forecasting, xLSTM achieved the lowest RMSE at 19.88 kWh, outperforming all other models by a margin of 8.8% compared to TFT (second-best). TFT attained the best MAE (9.16 kWh) and nRMSE. Both xLSTM and TFT markedly surpassed traditional FCN and LSTM baselines.

Figure 2: Average RMSE across forecasting steps for all models on the 24-hour horizon.

24-Hour Horizon

In the day-ahead (24-hour) task, xLSTM again delivered the lowest RMSE (21.47 kWh), outperforming TFT by 11.4%. However, TFT continued to yield the lowest MAE. No architecture outperformed the naive baseline in terms of nRMSE, indicating continued difficulty with certain range-specific forecasting scenarios. xLSTM exhibited robust medium-range prediction, maintaining superior performance up to the eleventh forecast step. The TE underperformed, highlighting inherent limitations of vanilla Transformers for this domain.

Figure 3: Example of xLSTM delivering a 24-hour forecast on test data; forecast closely tracks actual heat consumption.

Model Stability

Variability analysis across multiple training seeds revealed that FCN provided the most stable outcomes, while xLSTM—though most accurate—displayed greater training sensitivity. TFT's variability was moderate for short-term tasks but increased substantially for longer horizons.

Figure 4: Comparison of average RMSE and MAE for 5-seeds of models with 95% confidence intervals; TFT reported with fewer seeds.

Per-Building Performance

Evaluation of nRSE distributions per building showed heterogeneous error rates, with no observable correlation between energy consumption (building size) and forecasting difficulty.

Figure 5: Per-building evaluation of nRSE distributions for 24-hour prediction, indicating heterogeneity in model error by building.

Computational Sustainability Assessment

Memory and training time analyses expose the pronounced computational footprint of advanced neural architectures. FCN stands out with minimal parameter count and fastest training, making it optimal for edge deployment and hardware-constrained environments. xLSTM, despite improved accuracy, requires 2.1M parameters and produces ~190x more CO₂ emissions during training compared to FCN. The TFT is even larger (5.7M parameters) and incurred the highest emissions due to CPU-only training requirements. Transformer Encoder (TE) and LSTM occupy middle ground in terms of parameter and compute requirements. Differences in hardware access (CPU vs. GPU) affected comparative runtimes and should be considered when interpreting resource efficiency.

Figure 6: Training time for 24-hour prediction per epoch by model architecture and hardware (log scale), highlighting trade-offs in computational sustainability.

Implications and Future Work

The results demonstrate that, for pooled, heterogeneous building datasets, modern gated-memory architectures (xLSTM) generalize effectively and achieve the lowest RMSE in both short-term and day-ahead forecasting. TFT’s superior MAE and nRMSE are attributable to its multi-horizon specialized design and rigorous feature processing, but require substantial computational trade-offs. Traditional architectures like FCN remain attractive alternatives for deployments constrained by compute or carbon budgets.

Theoretical impacts include reaffirming the limitations of conventional Transformer encoders for time-series forecasting without architectural specialization. Practically, the study validates the feasibility of deploying “single-model-to-many-buildings” forecasting systems—critical for scalable operational optimization in urban heating.

Future research should focus on hardware-fair benchmarking across architectures, porting TFT to GPU-compatible frameworks, conducting cross-building (leave-out) experiments, and further adapting xLSTM and TFT for integration of future covariates. The ability of these models to generalize to unseen buildings and cold-start scenarios, as well as their operational carbon footprint, are pivotal dimensions for real-world adoption in sustainable energy management.

Conclusion

This benchmark establishes that xLSTM and TFT are technically superior for short-term and day-ahead heat demand forecasting in building-level heterogeneous datasets, provided computational resources are sufficient. FCN remains a resource-efficient baseline. The results underscore the necessity of balancing predictive gains against sustainability cost, and recommend further investment in model adaptability and hardware compatibility for large-scale, low-carbon heat forecasting solutions.