Foundation Models for Sparse Sensing

• Foundation model-based sparse sensing is a technique that leverages large, pretrained neural models to dynamically select and process only the most informative sensor signals.
• It reduces computational and sensing requirements by employing dynamic token selection, geometry-aware warping, and efficient multi-scale feature extraction.
• This approach achieves state-of-the-art performance in remote sensing and mobile AR by maintaining high accuracy using significantly fewer data and computational resources.

Foundation model-based sparse sensing refers to strategies that leverage large, pretrained neural models (foundation models) to address the core challenges of accurately capturing and interpreting information from input data while acquiring only a subset of the available sensor signals, both spatially and temporally. Emerging evidence demonstrates that careful model design and integration can substantially reduce both computational and sensing requirements without sacrificing task accuracy, particularly in remote sensing and mobile AR domains. Foundation models, such as DynamicVis for remote sensing and Metric3DV2 for mobile AR, enable scalable, cross-task generalization and efficient feature modeling in scenarios where the data exhibits sparse, highly localized patterns of interest.

1. Sparse Sensing Principles in Foundation Models

Foundation model-based sparse sensing exploits the underlying observation that, in many practical settings, objects or regions of interest occupy only a small fraction of the total data—typically ~1% in remote sensing imagery and variable subsets in mobile AR sequences. Historically, uniform processing of high-dimensional data required quadratic computational and memory resources (as with ViT’s attention over large token sets), rendering large-scale and real-time applications infeasible. Foundation models address this by selectively attending to or augmenting only the most informative portions of the input.

In remote sensing, DynamicVis utilizes SSM-based token reduction, dynamically routing only the top $rL$ tokens through state-space mixing while the remainder undergo parameter-free residual updates. In mobile AR, Metric3DV2 generates depth maps that enable warping and interpolation of unsensed frames, allowing aggressive frame skipping in time or pose without loss of reconstruction quality.

2. Architectural Innovations: Dynamic Region Perception and Geometry-Aware Warping

DynamicVis (Chen et al., 20 Mar 2025) applies a Selective State-Space Model (SSSM) backbone, which down-samples inputs with small strides (4, not 16), maintains token resolution for fine details, then processes only a fraction of tokens at each stage:

Mathematical Formulations

The SSM is defined as:

$h'(t) = A h(t) + B x(t),\quad y(t) = C h(t)$

Discretized:

$$h_k = \bar{A} h_{k-1} + \bar{B} x_k,\quad y_k = C h_k$$

For token sequences $s \in \mathbb{R}^{L \times d}$, SSM mixing is performed via a 1D convolution $y = s * \bar{K}$.

Dynamic token selection uses importance scores, computed via

$$p = \text{MLP}(s),\quad w' = \text{softmax}(p + \epsilon),\quad \epsilon \sim \text{Gumbel}(0, \cdot)$$

The top-$k$ tokens $x_r$ are extracted and processed alongside global tokens $$x_g = \text{AdaptivePool}_{1D}(s) \in \mathbb{R}^{\sqrt{L} \times d}$$ with dual-path Mamba scanning.

In mobile AR sparse sensing (Zhao et al., 4 Nov 2025), geometric image warping is defined as:

$$p_{t'} \sim K \left[ R_{t \rightarrow t'} \cdot ( D_t(p_t) K^{-1} p_t ) + t_{t \rightarrow t'} \right]$$

Attributes (e.g., RGB, depth) are sampled and fused via bilinear interpolation for mesh construction.

3. Multi-Instance Learning and Meta-Embedding Strategies

DynamicVis implements a multi-instance learning (MIL) paradigm using region-level pooled embeddings and meta-embeddings per semantic class initialized from CLIP. Generic RoI Extractor (GRoIE) pooling is carried out across all feature scales, yielding consistent region vectors $v \in \mathbb{R}^{1\times d}$. The MIL objective is a batch-wise NCE loss:

$$\mathcal{L}_{\rm MIL} = -\log\,\frac{\sum_{(v,t)\in P} \exp(\langle v, t\rangle / \tau)}{\sum_{(v,t)\in P} \exp(\langle v, t\rangle / \tau) + \sum_{(v',t')\in N} \exp(\langle v', t'\rangle / \tau)}$$

where positive pairs are region–meta-embedding tuples and negatives are mismatched region/class pairs. This enforces semantic clustering in feature space, supporting generalization across classification, retrieval, and detection tasks.

4. Computational Efficiency and Scaling Properties

The adoption of foundation model principles enables near-linear scaling of compute and memory. In remote sensing, DynamicVis processes 2048×2048 images with 97 ms latency and 0.83 GB GPU usage—6% and 3%, respectively, of the ViT-B baseline. Sparse mixer blocks prune up to 90% of token computations at early stages, concentrating modeling power where object density is highest.

In mobile AR, the warping step is $\mathcal{O}(HW)$ per frame, and mesh generation via Poisson reconstruction and ICP scales with the number of vertices and points, kept tractable by frame skipping. Empirical evidence shows that only 27% of frames are required to maintain $\geq$80% overlap, substantially reducing sensing and post-processing overhead.

Comparative Table: Latency and Memory (Remote Sensing)

Model	Resolution (px)	Latency (ms)	GPU Mem (GB)
ViT-B	2048 × 2048	1581	25.3
DynamicVis-B	2048 × 2048	97	0.83

5. Cross-Task Generalization Across Modalities

DynamicVis demonstrates state-of-the-art accuracy across nine standard remote sensing tasks spanning region-level classification, image retrieval, instance detection, and dense pixel segmentation. The backbone design permits simultaneous, multi-level feature modeling, with FPN merging context from multiple stages to support pixel masks, region classification, and instance localization.

In mobile AR, the use of Metric3DV2 enables improved performance on geometry-aware warping and 3D scene reconstruction. For example, RGB SSIM increases by 25.5% and depth SSIM by 30.7% (FM vs. LiDAR), with Poisson+ICP mesh reconstruction yielding a 48% reduction in Hausdorff distance even at 1/4 the frame rate. Warped frames using FM depth preserve detail, enabling content rendering and mesh consistency in aggressive sparse sensing conditions.

6. Open Challenges, Limitations, and Future Research Directions

Current limitations of foundation model-based sparse sensing include large model sizes and power demands that restrict real-time on-device inference, particularly on mobile hardware. Static frame-or-motion-based sampling policies cannot guarantee optimal view overlap due to unpredictable user motion. Geometric warping, while accurate for scene-consistent re-use, fails under severe occlusions or non-Lambertian surfaces.

Research directions proposed include development of hybrid sparse sensing controllers that combine temporal, spatial, and semantic triggers, lightweight and quantized foundation models for mobile deployment, end-to-end trainable warping modules with self-supervised losses, and learned priors for volumetric multi-view fusion rather than heuristic mesh merging.

Overview of Key Challenges and Promising Directions

Challenge	Approach/Directions
Model scale, device power	Quantization, dynamic scaling, NPU offloading
Overlap guarantee (mobile AR)	Semantic/hybrid scheduling, FM confidence-based triggers
Warping under occlusion	End-to-end trainable warping, robust priors
Volumetric fusion	Neural SDFs, FM-driven multi-view integration

Collectively, foundation model-based sparse sensing represents a significant advancement toward tractable, scalable perception under adverse data and computational constraints. Deploying these systems requires addressing both architectural and systemic challenges, particularly for mobile applications demanding real-time performance and energy efficiency. The cited works provide empirical validation and identify key algorithms and metrics for continued research.