Adaptive View Sampling Methods

• Adaptive view sampling is a method that dynamically selects the most informative views from a limited sampling budget for tasks like reconstruction and rendering.
• It leverages geometric, physical, and learned criteria—such as visibility, variance reduction, and differentiable modules—to optimize view selection.
• Empirical studies show that coupling adaptive sampling with iterative refinement and task-driven loss functions improves performance and reduces computational costs.

Adaptive view sampling denotes a family of methods that allocate a limited sampling budget to the most informative “views” for a downstream objective such as reconstruction, rendering, recognition, or error detection. In the literature, the term spans projection angles and sinograms in sparse-view CT, camera poses in aerial 3D reconstruction, slices or frames in medical volumes and videos, measurement indices in linear sensing systems, LiDAR rays, depth planes in multiplane images, per-ray samples in neural radiance fields, screen-space pixels in volume visualization, and sparse token subsets in cross-view video analysis (Yang et al., 2024, Peng et al., 2018, Shankaranarayana et al., 14 Oct 2025, Wang et al., 2023, Shomer et al., 2023, Navarro et al., 2021, Kurz et al., 2022, Weiss et al., 2020, Li et al., 13 Mar 2026). Across these settings, the central problem is to reduce acquisition, compute, memory, or trajectory cost while preserving task-relevant quality.

1. Formal problem classes

A recurring formulation is constrained optimization over a subset of candidate views. In aerial 3D reconstruction, the trajectory $\mathcal{J}=\{(s_i,\phi_i)\}$ is chosen to minimize path length while enforcing visibility, reconstruction quality, and a view budget: $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$ Here $\kappa(f,\mathcal{J})$ is the set of views that see face $f$, and $t \ge 2$ enforces robust triangulation (Peng et al., 2018).

In adaptive linear sensing, the acquisition is written as

$y = M A x + n,$

with $M$ a binary mask over measurement indices, $A$ the sensing operator, and $n \sim \mathcal{N}(0,\sigma^2 I)$. The practical adaptive rule is greedy variance reduction in the measurement domain: $$U(i) = \mathrm{Var}_s\big([A x^{(s)}]_i\big), \qquad i^* = \arg\max_{i \in \mathcal{I}_{\mathrm{unacq}}} U(i),$$ where $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$0 are posterior samples from SGLD (Wang et al., 2023).

Task-driven formulations replace explicit geometric or Bayesian utilities by a downstream loss. In adaptive LiDAR, the binary mask $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$1 is optimized under a budget $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$2 to minimize depth completion error: $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$3 In attention-guided frame or slice subsampling, the objective is

$$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$4

or, for stochastic selection, its expectation under $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$5; in the reported experiments, $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$6, so $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$7 is not needed (Shomer et al., 2023, Shankaranarayana et al., 14 Oct 2025).

These formulations show that adaptive view sampling is not tied to a single modality or solver. The common structure is a budgeted selection problem whose constraints may be geometric, probabilistic, or task-driven.

2. Geometric, physical, and representational criteria

Geometry-aware methods define view quality through visibility, incidence angles, distance consistency, and coverage. The aerial reconstruction framework uses line-of-sight visibility to mesh faces, a nominal viewing distance $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$8 with tolerance $$\min \ \mathrm{len}(\mathcal{J}) \quad \text{s.t.}\quad |\kappa(f,\mathcal{J})| \ge t,\ \forall f\in\mathcal{F}, \quad Q(f,\mathcal{J}) \ge Q^*,\ \forall f\in\mathcal{F}, \quad |\mathcal{J}| \le B.$$9, a field-of-view of $\kappa(f,\mathcal{J})$0, and incidence/triangulation angle bounds $\kappa(f,\mathcal{J})$1. Candidate views are constrained to satisfy $\kappa(f,\mathcal{J})$2, with $\kappa(f,\mathcal{J})$3, so that resolutions remain comparable across views (Peng et al., 2018).

In sparse-view CT, physical sampling is represented directly in the sinogram domain. With full-view sinogram $\kappa(f,\mathcal{J})$4 and binary sampling mask $\kappa(f,\mathcal{J})$5, sparse measurements are

$\kappa(f,\mathcal{J})$6

CT-SDM defines a degradation operator

$\kappa(f,\mathcal{J})$7

so that forward “diffusion” corresponds to deterministic removal of projection views rather than Gaussian corruption. The grouped-random allocation strategy divides the $\kappa(f,\mathcal{J})$8 angles into evenly spaced groups, selects whole groups in order, and randomly samples the remainder from the current group when necessary, combining angular uniformity with randomness for data augmentation (Yang et al., 2024).

Adaptive sampling can also occur in representational depth space rather than physical sensor space. In compact and adaptive MPIs, inference begins from regular inverse-depth sampling, computes a provisional MPI, removes planes whose opacity never exceeds $\kappa(f,\mathcal{J})$9, weights the retained depth intervals by mean $f$0, and redistributes the discarded planes proportionally to those interval weights. This concentrates depth samples where scene geometry and occlusions are actually present (Navarro et al., 2021).

Continuous camera optimization further generalizes the notion of a view. Camera Splatting models each camera as a 3D Gaussian-like “camera splat” $f$1, places omnidirectional point cameras near proxy geometry, and optimizes camera distributions so that each point camera observes a target intensity pattern scaled by a View-Dependency Score. Visibility is gated by a FoV test $f$2, and self-occlusion is handled by an occlusion mask rendered from the proxy geometry (Lee et al., 19 Sep 2025).

Taken together, these methods define informativeness through physically meaningful structure: surface normals, line of sight, projection geometry, opacity concentration, or view-dependent appearance.

3. Learned and differentiable selection mechanisms

A major development in adaptive view sampling is the replacement of heuristic or combinatorial selection by differentiable modules that can be trained end-to-end. In DAS, per-view importance is produced by a lightweight feature extractor, a multi-head attention layer, and Gumbel-Softmax sampling with an input-conditioned temperature

$f$3

The final sampling matrix uses a straight-through estimator: $f$4 A key distinction made in that work is between task-adaptive but static samplers and input-adaptive samplers that continue to change at inference time (Shankaranarayana et al., 14 Oct 2025).

SAVA-X applies discrete Top-$f$5 selection to asynchronous ego and exo video streams. Exo features are scored by self-attention, ego features by cross-attention conditioned on sampled exo features, and the hard selection is accompanied by a residual gating path

$f$6

so that downstream modules see sparse tokens while gradients flow through the soft probabilities $f$7 (Li et al., 13 Mar 2026).

SampleDepth uses a U-Net-based CNN to predict a per-pixel sampling probability volume $f$8 and converts it into a differentiable soft mask with SoftArgmax. For pixel $f$9,

$t \ge 2$0

allowing the depth completion loss to supervise the mask predictor directly (Shomer et al., 2023).

In volume visualization, the sampler is defined in screen space. A normalized importance map $t \ge 2$1 is compared against a deterministic low-discrepancy pattern $t \ge 2$2, and training replaces the hard decision by

$t \ge 2$3

with $t \ge 2$4 reported as the best train-test match. The selected sparse samples are then passed through differentiable pull-push inpainting and residual reconstruction (Weiss et al., 2020).

AdaNeRF uses another differentiable mechanism: the sampling network predicts per-cell weights $t \ge 2$5 on a fixed ray grid, and the shading density is multiplied by $t \ge 2$6 during compositing. This makes the sample-allocation decision trainable without a straight-through estimator and supports hard thresholding and top-$t \ge 2$7 capping at inference (Kurz et al., 2022).

4. Coupling sampling with reconstruction, rendering, and iterative refinement

Adaptive view sampling is frequently inseparable from the inverse problem or rendering model that follows it. CT-SDM is explicit on this point. Its reverse process predicts a full-view sinogram $t \ge 2$8 and updates the current sparse sinogram by TACoS: $t \ge 2$9 The final image is obtained by filtered backprojection and image-domain refinement,

$y = M A x + n,$0

so the adaptive mask schedule and the reconstruction network are jointly coupled (Yang et al., 2024).

Aerial view planning uses an iterative proxy-update loop. A first-pass image set produces a coarse mesh; low-quality regions are then clustered; Adaptive Viewing Rectangles are fit by least squares to elevated patch clouds; rectangular grids are imposed; and new images are collected by following a tour built from per-rectangle sweeps and a doubled MST connector. Reconstruction quality is then re-evaluated, and the process repeats until every face satisfies the quality and visibility constraints or the gain becomes negligible (Peng et al., 2018).

The linear sensing framework is similarly iterative, but with a Bayesian posterior in place of a mesh proxy. At each step, SGLD draws posterior samples using

$y = M A x + n,$1

then forward projections $y = M A x + n,$2 are used to score unacquired measurements by posterior predictive variance before the next measurement is acquired (Wang et al., 2023).

Inference-time refinement appears again in adaptive MPIs. Plane reallocation is not learned via gradients; instead, it is applied after one provisional refinement iteration, followed by recomputation of plane-sweep volumes and a rerun of the full iterative MPI estimation from scratch on the adapted depth set (Navarro et al., 2021).

These examples illustrate a broad pattern: adaptive view sampling rarely acts as an isolated preprocessing step. It is usually embedded in a closed loop with a proxy geometry, a posterior sampler, or a rendering/reconstruction operator.

5. Reported empirical behavior

The empirical literature reports gains in reconstruction quality, rendering quality, or task performance under fixed budgets, as well as reductions in compute or trajectory cost.

Setting	Reported result	Source
Sparse-view CT on LDCT, averaged across sampling rates	CT-SDM: PSNR $y = M A x + n,$3, SSIM $y = M A x + n,$4, LPIPS $y = M A x + n,$5; FreeSeed: PSNR $y = M A x + n,$6, SSIM $y = M A x + n,$7, LPIPS $y = M A x + n,$8	(Yang et al., 2024)
Aerial 3D reconstruction, third visit	Ours (3rd visit): $y = M A x + n,$9 mm avg error, $M$0 mm std, $M$1 completeness; ZigZag: $M$2 mm, $M$3 mm, $M$4 completeness	(Peng et al., 2018)
Adaptive linear sensing for MRI	Compared to non-adaptive sampling, image quality improved by $M$5-$M$6 dB in PSNR, with better restoration of subtle details	(Wang et al., 2023)
Adaptive LiDAR on SHIFT, 19K points	Agnostic: $M$7 m RMSE; Adaptive PredNet: $M$8 m; mask prediction time $M$9 ms on $A$0 NVIDIA GTX 1080 Ti	(Shomer et al., 2023)
Input-adaptive frame/slice subsampling	With $A$1, compute and memory reduce to $A$2 of the full-view baseline; on in-house ultrasound, DAS: AUC $A$3, Acc $A$4; Full: AUC $A$5, Acc $A$6	(Shankaranarayana et al., 14 Oct 2025)
Real-time NeRF rendering	AdaNeRF on DONeRF: $A$7 samples, $A$8 ms, PSNR $A$9; DONeRF with $n \sim \mathcal{N}(0,\sigma^2 I)$0 samples: $n \sim \mathcal{N}(0,\sigma^2 I)$1 ms, PSNR $n \sim \mathcal{N}(0,\sigma^2 I)$2	(Kurz et al., 2022)
Ego-to-exo imitation error detection	Validation Mean AUPRC: SAVA-X $n \sim \mathcal{N}(0,\sigma^2 I)$3; strongest baseline Exo2EgoDVC $n \sim \mathcal{N}(0,\sigma^2 I)$4; tIoU $n \sim \mathcal{N}(0,\sigma^2 I)$5 for SAVA-X	(Li et al., 13 Mar 2026)

This evidence suggests a consistent empirical pattern: adaptive allocation tends to outperform fixed-rate or agnostic sampling when redundancy is high, when the acquisition budget is tight, or when the test sampling pattern departs from the training pattern. Several studies also report that adaptation is especially helpful on residual low-quality regions, padded or empty frames, and view-dependent or occluded content.

6. Distinctions, limitations, and recurrent failure modes

A common misconception is that adaptive view sampling is synonymous with online next-best-view camera planning. The literature is broader. Some methods optimize physical viewpoints or trajectories, but others adapt measurement masks, slices, frames, LiDAR rays, depth planes, per-ray rendering samples, or sparse token subsets. CT-SDM is explicit that it performs adaptive reconstruction across sampling rates from a given mask and “does not itself choose the next best angles,” whereas active view selection would decide which angles to acquire next (Yang et al., 2024). DAS likewise distinguishes input-adaptive inference from earlier Gumbel-max approaches that remain static after training (Shankaranarayana et al., 14 Oct 2025).

The reported limitations are also heterogeneous but structurally similar. CT-SDM assumes small measurement noise, fixed fan-beam geometry, and accurate masks $n \sim \mathcal{N}(0,\sigma^2 I)$6; poor calibration of the exponential schedule or mismatch between $n \sim \mathcal{N}(0,\sigma^2 I)$7 and clinical sampling distributions can reduce optimality (Yang et al., 2024). The SGLD-based linear sensing method notes that real-time feasibility is difficult because repeated $n \sim \mathcal{N}(0,\sigma^2 I)$8 and $n \sim \mathcal{N}(0,\sigma^2 I)$9 evaluations dominate compute, so practical use may require pilot-frame design and mask reuse across frames (Wang et al., 2023). SampleDepth is sensitive to ego-motion, occlusions, calibration, and hardware control granularity, and its explicit temporal predictor degrades with large inter-frame motion (Shomer et al., 2023). Camera Splatting depends on the quality of the proxy geometry; in the reported robustness study, optimization fails when the proxy is severely degraded at $$U(i) = \mathrm{Var}_s\big([A x^{(s)}]_i\big), \qquad i^* = \arg\max_{i \in \mathcal{I}_{\mathrm{unacq}}} U(i),$$0 initial views and becomes stable from $$U(i) = \mathrm{Var}_s\big([A x^{(s)}]_i\big), \qquad i^* = \arg\max_{i \in \mathcal{I}_{\mathrm{unacq}}} U(i),$$1 views onward (Lee et al., 19 Sep 2025).

Further recurrent issues involve selection collapse, duplicate selections, and domain shift. DAS notes that duplicate selections across the $$U(i) = \mathrm{Var}_s\big([A x^{(s)}]_i\big), \qquad i^* = \arg\max_{i \in \mathcal{I}_{\mathrm{unacq}}} U(i),$$2 rows can occur if distributions are similar and recommends argmax-without-replacement or TopK when uniqueness is desired (Shankaranarayana et al., 14 Oct 2025). SAVA-X introduces entropy, variance, covariance, and dictionary-diversity regularizers precisely because sparse discrete sampling can otherwise collapse or overfit to redundant tokens (Li et al., 13 Mar 2026).

These limitations indicate that adaptive view sampling is most reliable when the adaptation signal is well calibrated to the downstream objective: a faithful proxy mesh, an accurate mask, a stable temporal prior, a representative posterior sampler, or a geometry-aware scene prior. Where those ingredients are weak, the adaptation mechanism can amplify rather than correct the underlying model error.