An iterative energy-based multimodal transformer for joint retrieval of wheat soil moisture, leaf area index, and plant height from Sentinel-1 and Sentinel-2 time series

Abstract: Field-scale retrieval of surface soil moisture (SM), leaf area index (LAI), and plant height (PH) is essential for precision agriculture, yet it remains an ill-posed inverse problem. Concurrent variations in soil moisture and canopy density generate substantial ambiguities in radar backscatter and spectral responses, which reduces the effectiveness of traditional feedforward regression models in heterogeneous smallholder cropping systems. This study presents the Iterative Energy-Based Transformer (iEBT) for the joint retrieval of coupled soil-canopy states from Sentinel-1 C-band SAR and Sentinel-2 multispectral time series. Instead of direct regression, iEBT embeds multi-modal predictors within a shared sequence, produces an initial state estimate, and iteratively updates the target [SM, LAI, PH] vector through normalized gradient descent to minimize a learned scalar compatibility energy function. Using 700 quality-controlled field measurements from Varanasi, India, iEBT achieved the highest learned-model performance on the random test split, with a four-seed mean R^2 of 0.854 \pm 0.012 (R_SM^2 = 0.841, R_LAI^2 = 0.905, R_PH^2 = 0.821). WCM and PROSAIL were retained as physically interpretable SAR and optical reference models for comparison. Modality ablations confirmed that Sentinel-1 drives SM retrieval, while Sentinel-2 dominates LAI, whereas PH relies on combined structural-phenological signatures. Crucially, the model's terminal energy functions as an uncalibrated post-retrieval quality diagnostic; screening the 10% highest-energy samples markedly reduced target level root-mean-square errors. While leave-one-campaign-out validation highlights persistent cross-season domain shift challenges due to localized management variations, compatibility-guided multimodal fusion offers a structured self-diagnostic path toward reliable biophysical parameter estimation

• The paper presents an iterative energy-based transformer that refines joint retrieval estimates of soil moisture, LAI, and plant height through gradient descent on a learned compatibility energy.
• The model outperforms standard baselines with high R² scores (average 0.854), demonstrating robust multimodal integration of Sentinel-1 SAR and Sentinel-2 multispectral data.
• The approach incorporates self-diagnostic energy metrics that flag low-confidence predictions, offering a practical quality assessment tool for heterogeneous wheat fields.

Iterative Energy-Based Multimodal Transformer for Joint Retrieval of Soil Moisture, LAI, and Plant Height in Wheat Systems

Introduction

The estimation of surface soil moisture (SM), leaf area index (LAI), and plant height (PH) at field scale is a multi-variable inverse problem central to precision agriculture and crop monitoring. Within heterogeneous, smallholder wheat systems, concurrent variation in both soil and canopy state introduces considerable ambiguity into SAR and multi-spectral satellite responses. Historically, operational retrieval approaches rely on direct regression through radiative transfer (e.g., PROSAIL) or semi-empirical (e.g., Water Cloud Model, WCM) models, or on pure data-driven regressors such as random forests or neural networks. However, these approaches do not fully exploit the mutual compatibility of multimodal evidence from Sentinel-1 SAR and Sentinel-2 multispectral imagery, nor do they provide internal diagnostics concerning retrieval quality or consistency.

This work proposes an Iterative Energy-Based Transformer (iEBT) architecture for multi-output retrieval of SM, LAI, and PH from temporally matched Sentinel-1/-2 time series combined with field measurements. The iEBT handles the ill-posed nature of the retrieval problem by embedding both predictors and response variables in a learned, multimodal context, performing state refinement through minimization of a data-driven compatibility energy.

Model Formulation and Multimodal Integration

The iEBT architecture is founded on three elements: modality-specific encoders for SAR, optical, and temporal features; a transformer-based fusion strategy for joint representation; and an iterative, energy-based inference loop. Rather than directly mapping from input features to target variables, the model first generates a state proposal and then refines this estimate by minimizing a learned scalar energy, representing the compatibility between the candidate state and observed context.

The refinement is performed via normalized gradient descent on the energy surface, with learnable step size and clipping to enforce plausible physical ranges. Terminal energy serves as an internal, sample-wise compatibility diagnostic, flagging low-confidence retrievals without recourse to calibrated uncertainty intervals.

Compared to direct regression, this formulation embeds the physical intuition of forward-inverse modeling within the transformer. The model is supervised using a combination of proposal loss, energy-margin ranking loss (drawing negative targets from campaign-matched shuffles or local perturbations), and uncertainty-weighted final regression. All model parameters are optimized jointly with AdamW.

Experimental Design and Baselines

Experiments were conducted using 700 strict-QC field-measurements from three seasonal campaigns (Varanasi, India), paired with temporally matched Sentinel-1 (VV, VH, VV/VH) and Sentinel-2 (B2-B4, B8, vegetation indices) within ±5 days of ground sampling. Predictors further include temporal gaps and cyclic season-progress encodings.

Baselines include:

1. Random Forest (RF): as a standard ensemble regressor.
2. Transformer-based direct regression (TT): encoding-and-fusion without iteration or energy minimization.
3. Non-iterative Energy-Based Transformer (tEBT).
4. Physical models: WCM for SAR-only inversion and PROSAIL-based neural network inversion for LAI from optical data.

Model performance is quantified by $R^2$ and RMSE on a 70/15/15 random train/validation/test split (with four-seed mean scores), ablation experiments isolating SAR vs. optical features, and cross-campaign (leave-one-campaign-out, LOCO) validation for domain adaptation.

Results

Numerical Performance

iEBT achieves the highest average test-set $R^2$ of $0.854 \pm 0.012$ across [SM, LAI, PH]. For individual targets: $R^2 = 0.836$ (SM), $0.905$ (LAI), $0.821$ (PH). This outperforms all baselines, including tEBT ($0.775$), TT ($0.749$), and RF ($0.728$). LAI is consistently the best retrieved variable, while PH, though not directly measured by either Sentinel-1 or -2, exhibits considerable correlation, reflecting indirect encoding via canopy structure and phenology. WCM and PROSAIL reference models achieve lower $R^2$ (e.g., WCM-VH for LAI: $R^2$0; WCM-VV for SM: $R^2$1; PROSAIL for LAI: $R^2$2).

Modality ablation shows Sentinel-1 governs SM retrieval, Sentinel-2 dominates LAI, and PH depends on joint multimodal structure (SAR-only: $R^2$3; Optical-only: $R^2$4; Full: $R^2$5 with iEBT).

Iterative Refinement and Diagnostics

The iterative energy-based inference step yields significant refinement, especially for SM and LAI (e.g., SM: $R^2$6 increases from $R^2$7 to $R^2$8 across steps). The compatibility energy is positively correlated with prediction error (Spearman 0.35–0.46 for targets). Screening the top 10% highest-energy samples (incompatible predictions) reduces RMSE for all variables, indicating the practical utility of terminal energy as an uncalibrated retrieval-quality diagnostic.

Transferability and Generalization

LOCO validation reveals substantial performance drop for inter-campaign generalization: iEBT average $R^2$9 ranges from $0.854 \pm 0.012$0 (2019–2020 held-out), $0.854 \pm 0.012$1 (2023), to $0.854 \pm 0.012$2 (2024). This highlights persistent cross-season distribution shift and the need for more robust domain adaptation techniques and extended field datasets.

Theoretical and Practical Implications

This work demonstrates that compatibility-guided, energy-based multimodal fusion provides a structured solution to the coupled, ill-posed retrieval of soil and canopy state variables from satellite time series. The iterative energy minimization process implicitly enforces multimodal consistency, enables post hoc reliability assessment, and integrates physical and data-driven paradigms.

The physical references (WCM, PROSAIL) offer interpretability but suffer from sensor/spectral specificity, calibration requirements, and breakdown under complex field conditions. Their lower performance in heterogeneous wheat systems reaffirms the need for flexible, self-diagnostic learned models. Notably, iEBT’s architecture, through refinement and energy diagnostics, addresses the ambiguity between soil and plant state in SAR/optical signals—critical for operational deployment.

From an applied perspective, the integration of uncalibrated energy diagnostics into retrieval workflows provides a mechanism for interpretability and automated sample screening. This is particularly relevant for precision agriculture, where management decisions are contingent on the reliability—not just the point accuracy—of satellite-derived biophysical indicators.

Future Directions

Key limitations remain. The sample size is moderate by deep learning standards, challenging cross-season generalization. The predictor set, while comprehensive, excludes promising additional variables such as multi-temporal SAR texture, spatial context, and external agro-meteorological drivers. Potential improvements include:

• Integration of explicit domain-invariant or domain-adaptive training,
• Incorporation of physics-informed priors into the energy function,
• Calibration of energy values for quantitative uncertainty estimation (e.g., via conformal prediction or model ensembles),
• Expansion to larger, multi-region datasets.

Furthermore, adding height-sensitive remote sensing sources (LiDAR, or high-resolution SAR missions) may improve PH estimation. Exploring model behavior under extreme events (e.g., irrigation, phenological shift, drought) will refine the detection of outlier states and further validate energy-based diagnostics.

Conclusion

The iterative energy-based multimodal transformer paradigm offers a robust, interpretable, and diagnostically transparent approach for simultaneous retrieval of SM, LAI, and PH from SAR-optical satellite data. The methodological advancements in compatibility-guided fusion and gradient-based refinement constitute an important contribution for biophysical state estimation in heterogeneous cropping systems. The results indicate that self-diagnostic, multimodal energy-based models are poised to enhance the operational value of satellite remote sensing in precision agriculture, conditional on continued advances in generalization, uncertainty quantification, and multi-year data integration.

Reference: "An iterative energy-based multimodal transformer for joint retrieval of wheat soil moisture, leaf area index, and plant height from Sentinel-1 and Sentinel-2 time series" (2606.25174).