Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence

Abstract: Edge-based multimodal medical monitoring requires models that balance diagnostic accuracy with severe energy constraints. Continuous acquisition of ECG, PPG, EMG, and IMU streams rapidly drains wearable batteries, often limiting operation to under 10 hours, while existing systems overlook the high temporal redundancy present in physiological signals. We introduce Adaptive Multimodal Intelligence (AMI), an end-to-end framework that jointly learns when to sense and how to infer. AMI integrates three components: (1) a lightweight Agentic Modality Controller that uses differentiable Gumbel-Sigmoid gating to dynamically select active sensors based on model confidence and task relevance; (2) a Learned Sigma-Delta Sensing module that applies patch-wise Delta-Sigma operations with learnable thresholds to skip temporally redundant samples; and (3) a Foundation-backed Multimodal Prediction Model built on unimodal foundation encoders and a cross-modal transformer with temporal context, enabling robust fusion even under gated or missing inputs. These components are trained jointly via a multi-objective loss combining classification accuracy, sparsity regularization, cross-modal alignment, and predictive coding. AMI is hardware-aware, supporting dynamic computation graphs and masked operations, leading to real energy and latency savings. Across MHEALTH, HMC Sleep, and WESAD datasets, it reduces sensor usage by 48.8% while improving state-of-the-art accuracy by 1.9% on average.

• The paper introduces AMI, a unified agentic sensing–inference framework that jointly optimizes sensor activation and prediction while reducing energy consumption.
• It employs adaptive patch-wise Sigma–Delta sensing and modality gating via Gumbel–Sigmoid, achieving a 48.8% sensor reduction with state-of-the-art accuracy.
• Extensive evaluations across biomedical tasks demonstrate robust performance with up to 31.9% lower latency and 24.8% lower energy usage on edge hardware.

Agentic Multimodal Transformers for Energy-Efficient Edge Medical Intelligence

Introduction

Edge medical monitoring systems demand joint maximization of diagnostic accuracy and optimization of energy consumption. The integration of multimodal sensor streams—such as ECG, PPG, EMG, and IMU—on battery-constrained wearables substantially accelerates battery depletion, principally due to high temporal redundancy and the compute burden imposed by contemporary large-scale multimodal neural architectures. The paper "Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence" (2604.10404) presents Adaptive Multimodal Intelligence (AMI)—a unified, agentic, and hardware-aware transformer-based framework capable of simultaneously optimizing sensing policy and inference on edge medical devices. The framework leverages three principal mechanisms: an Agentic Modality Controller (AMC) for dynamic sensor selection, a learnable Sigma–Delta Sensing module for adaptive sampling, and a foundation-model based multimodal prediction backbone with robust, context-aware fusion and temporal modeling.

Unified Agentic Multimodal Sensing-Inference Framework

The AMI pipeline introduces an end-to-end differentiable framework in which decisions about when and how much to sense are intertwined with model predictions. The key architectural principle is to incorporate the decision-theoretic concept of value of information at both modality and temporal resolution, jointly optimized with predictive loss and hardware constraints. The framework is depicted schematically below.

Figure 2: The unified agentic multimodal sensing and inference pipeline, balancing accuracy and energy efficiency for edge biomedical intelligence.

The AMC leverages Gumbel–Sigmoid differentiable gating to enable per-modality selection, using the fused model state and operating on a rolling temporal window policy. Simultaneously, the Sigma–Delta module modulates sensing rate intra-modality via learned patch-wise thresholds, adapting sampling to local signal variability. The prediction backbone employs lightweight, foundation-pretrained unimodal encoders that feed a cross-modal transformer with contextual history, facilitating robust inference under modality dropout.

Figure 1: AMI architecture: Sigma–Delta module for adaptive patch-wise skipping; Gumbel-MLP agent for next-window per-modality mask generation; early cross-attention fusion; Tiny-Transformer backbone with temporal integration.

Adaptive Patch-wise and Modality-wise Sensing

Sigma–Delta Sensing provides patch-level adaptive skipping of redundant input segments per modality by learning thresholds for meaningful change, enabling dynamic reduction of computation and sensor activation. Patch-wise differences are passed to the tokenizer only when signal activity surpasses a learned threshold. This modulation is not static; thresholds are end-to-end differentiable and updated as a function of downstream prediction utility, directly supporting low-power edge use cases.

Figure 3: Visualization of patch-level thresholding and adaptive skipping in Sigma–Delta Sensing.

On the modality level, the AMC's gating logits are perturbed with Gumbel noise and subjected to a straight-through estimator, allowing sampling-based exploration of sensor subsets during training and precise deterministic gating during inference. This agentic mechanism enables coarse-to-fine adjustment of system energy consumption in accordance with prediction confidence and learned context, offering a tight feedback loop between perception and action.

Multi-objective Temporal and Contrastive Alignment

AMI is trained via a composite loss enforcing four orthogonal objectives: classification accuracy, gating sparsity, multimodal latent alignment (contrastive InfoNCE), and future-state predictive consistency. This design maintains cross-modal representation coherence and ensures robustness against both scheduled and random sensor dropouts.

Figure 4: Each fused state produces losses for prediction, gating, contrastive alignment, and predictive coding; controller actions influence future observations and losses are propagated temporally.

The model incorporates a memory bank for contrastive alignment, using positive pairs from synchronized modalities and negative pairs from the batch to encourage discriminative, cross-modally consistent embeddings. The predictive coding regularizer forecasts future latent states, mitigating information loss due to missed samples.

Empirical Evaluation and Quantitative Results

Evaluation on three reference biomedical datasets—MHEALTH (activity recognition), HMC Sleep (sleep staging), and WESAD (stress detection)—demonstrates that AMI reduces total sensor activation by 48.8% on average, achieving state-of-the-art or stronger accuracy (+1.9%) compared to specialized and foundation-model baselines, while further lowering energy and latency. For instance, on MHEALTH, accuracy of 99.12% and F1 of 99.12 are attained with only 38.19% sensor usage, showing negligible degradation compared to full-sensing models. Strong performance under highly sparse sensing policies is achieved even as the regularization coefficient for gating is increased.

Heatmaps of sensing-rate over time windows highlight the emergence of highly selective, temporally-adaptive activation patterns, consistently aligning with modal signal informativeness.

Figure 5: Patch-wise sensing-rate heatmaps on MHEALTH and HMC datasets, illustrating learned adaptive activation policies across time and sensors.

Robustness studies under random modality dropout indicate that accuracy remains nearly unchanged for up to 50% missing sensors, underscoring resilience under real-world sensor failure or occlusions. Similar insensitivity is observed under reduced input sampling rates down to 25 Hz (from 50 Hz), due to the model’s reliance on temporal context and predictive coding.

Hardware-aware Efficiency Analysis

The AMI framework is explicitly co-designed for edge deployment, supporting dynamic computational graphs and masked computation via TensorRT optimizations on platforms ranging from ARM CPUs to Jetson Orin and RTX A6000. Per-inference iteration, AMI achieves 31.9% lower latency and 24.8% lower energy consumption on average versus dense multimodal transformer baselines, extending operational horizons for wearables and implantables.

Figure 6: Per-iteration latency and energy as a function of sensing rate across hardware platforms; FMPM dominates latency, while AMC overhead is minimal.

Notably, hardware speedups are maximized with TensorRT execution and aggressive sensing reduction, revealing substantial benefits of architecture-algorithm co-design in hardware-constrained settings.

Theoretical Analysis

Theoretical guarantees on sample complexity are asserted: for $M$ modalities where only $k^* \ll M$ are informative for a given $\epsilon$ error, the joint optimization in AMI requires $\mathcal{O}(k^*/\epsilon^2)$ samples and converges in $\mathcal{O}(\log(1/\epsilon))$ rounds, a multiplicative improvement over decoupled approaches that require $\mathcal{O}(M/\epsilon^2)$ samples. This formalizes the advantage of agentic, context-dependent sensor selection under task-specific objectives.

Implications and Future Prospects

Practically, the proposed paradigm enables markedly longer-term, reliable physiological monitoring on resource-constrained devices by drastically cutting both sensor and compute duty cycles without sacrificing inference integrity. The model’s resilience to signal loss and temporal downsampling suggests direct applicability to next-generation neurostimulation systems and continuous patient monitoring—especially in ambulatory or home environments where battery longevity and robustness are paramount.

Theoretically, the demonstration that joint, differentiable optimization of agentic sensing and inference yields provable sample complexity reductions may inform design of broader active perception-action frameworks, particularly in safety-critical domains. Further, the integration of foundation model representations with online, task-conditional gating mechanisms may stimulate advances in multimodal self-supervised learning and hardware-algorithm co-adaptation.

Conclusion

The AMI framework constitutes a methodical advance in agentic, hardware-aware multimodal sensing and inference for medical edge intelligence. By unifying data-driven sensor selection, temporal redundancy exploitation, and robust transformer-based prediction within a single, theory-grounded system, AMI achieves substantial reductions in sensing and computation—directly translating to longer battery runtime—while delivering benchmark and occasionally superlative predictive performance. This approach represents a substantial step toward persistent, real-world deployment of AI-powered biomedical monitoring, and delineates clear avenues for generalization to arbitrary multimodal sensor settings in future edge AI systems.