Active-Vision Foundation Models

• Active-Vision Foundation Models are defined as systems that integrate sequential sensing, persistent memory, and active control to guide efficient scene exploration.
• They demonstrate significant gains in label-efficient learning, zero-shot scene understanding, and embodied occlusion-aware perception across various tasks.
• Empirical studies reveal challenges in stable spatial belief revision and low-latency updates, highlighting key open research directions in active vision.

Searching arXiv for the specified papers to ground the article and verify bibliographic details. Active-Vision Foundation Models (AVFMs) denote a class of systems in which perception is not restricted to a single passive feedforward encoding of a full scene, but is instead coupled to sequential information acquisition, memory, and view selection, while retaining the general-purpose transfer properties associated with foundation models. Across recent work, the term covers at least three closely related formulations: active learning systems that leverage frozen or lightly adapted vision foundation representations to improve label efficiency (Gupte et al., 2024); active vision encoders that accumulate scene understanding from localized glimpses into a persistent spatial workspace and transfer across tasks and viewing policies (Berreby et al., 23 Mar 2026); and embodied perception-planning systems that use vision-language and segmentation/detection foundation models to drive viewpoint control, spatial reasoning, and manipulation under occlusion (Shi et al., 16 Mar 2025). A complementary line of evaluation frames the central challenge as constructing, revising, and exploiting a spatial belief through self-directed exploration, and shows that current foundation-model agents still exhibit substantial deficits in active spatial intelligence (Zhang et al., 4 Feb 2026).

1. Terminological scope and conceptual definition

The recent literature does not use a single uniform definition of AVFMs. In work on active learning with vision foundation models, the term is not explicit; it is introduced as a useful label for active learning systems that leverage frozen or lightly adapted representations from models such as DINOv2 and OpenCLIP to maximize labeling efficiency (Gupte et al., 2024). In work on active computer vision, an AVFM is defined more structurally as an active vision encoder whose scene representation and processing are general-purpose, task-agnostic, and policy-agnostic, making it transferable across downstream tasks and viewing policies and scalable to large scenes and long rollouts (Berreby et al., 23 Mar 2026). In robotic grasping, AVFMs are operationalized as perception-planning systems that leverage off-the-shelf vision-language and segmentation/detection foundation models to drive object-centric spatial reasoning and active viewpoint control (Shi et al., 16 Mar 2025).

Despite this heterogeneity, the formulations share a common substrate. Passive vision encoders process a scene in a single feedforward pass and do not reuse computation over time, whereas active systems select what to observe next, integrate information across timesteps, and maintain a state that persists across partial observations (Berreby et al., 23 Mar 2026). This suggests a broad unifying characterization: AVFMs combine foundation-model priors with sequential sensing, explicit or implicit memory, and some mechanism for choosing informative observations.

A recurrent misconception is that AVFMs are simply passive vision foundation models with an added control loop. The surveyed work points to a stronger requirement. In active learning, the representation geometry of frozen foundation embeddings changes the behavior of query strategies themselves (Gupte et al., 2024). In active vision, the encoder architecture must support persistent scene-wide state and low-latency recurrent updates (Berreby et al., 23 Mar 2026). In embodied exploration, the decisive issue is not merely perception quality but the ability to maintain a revisable spatial belief during autonomous information gathering (Zhang et al., 4 Feb 2026).

2. Representation, memory, and the active-passive distinction

A central distinction in AVFM research is between retinotopic processing of the current observation and spatiotopic or allocentric memory over the scene. "CanViT: Toward Active-Vision Foundation Models" formalizes this by decoupling a retinotopic Vision Transformer backbone from a spatiotopic scene-wide latent workspace called the canvas (Berreby et al., 23 Mar 2026). The backbone processes each glimpse’s patch tokens, a recurrent CLS token, ephemeral register tokens, and a viewpoint encoding token, while the canvas is a persistent grid of tokens anchored to scene coordinates and updated only through cross-attention. The paper explicitly removes canvas-side self-attention, MLPs, and canvas-side QKVO, enforcing a “no compute on canvas” constraint that preserves scalability to large spatial workspaces (Berreby et al., 23 Mar 2026).

The role of spatial binding is equally central. CanViT uses scene-relative 2D RoPE to express both glimpse patch centers and canvas patch centers in shared scene coordinates, so that active views at varying scales are written into and read from a common spatial frame (Berreby et al., 23 Mar 2026). A dedicated viewpoint encoding token supplements this mechanism through a scale-invariant map $u(x, y, s) = (x/s, y/s, \log s)$ lifted with Random Fourier Features and LayerNorm (Berreby et al., 23 Mar 2026). The architectural intent is explicit: compact retinotopic computation performs the local “thinking,” while the large spatiotopic canvas performs the “memory” (Berreby et al., 23 Mar 2026).

The same distinction appears, in different form, in the "Theory of Space" benchmark. There, the relevant internal object is the spatial belief $B_t$, a probabilistic representation over spatial structure $S$ conditioned on observation-action history $h_t = (o_{0:t}, a_{0:t})$ (Zhang et al., 4 Feb 2026). The benchmark treats a cognitive map as the canonical externalization of this belief and probes it at every step using a JSON map of object positions and headings together with a local snapshot (Zhang et al., 4 Feb 2026). The resulting diagnosis is that perception is only one bottleneck: even when local perception is adequate, global beliefs suffer from instability and later observations can overwrite previously correct facts, producing belief drift (Zhang et al., 4 Feb 2026).

This distinction clarifies why active and passive competence diverge. In the same benchmark, the Active-Passive Gap is defined as the performance drop when models must autonomously gather information rather than receive standardized exploration logs (Zhang et al., 4 Feb 2026). In the vision world, overall averages include GPT-5.2 dropping from 57.1 to 46.0 and Gemini-3 Pro dropping from 60.5 to 57.3 (Zhang et al., 4 Feb 2026). The implication is not merely that exploration is hard; it is that active spatial intelligence requires an integrated stack of perceptual fidelity, explicit or implicit memory, stable update rules, and information-seeking control.

3. AVFMs for label-efficient visual learning

One major AVFM instantiation emerges in active learning with frozen foundation features. "Revisiting Active Learning in the Era of Vision Foundation Models" studies how DINOv2 and OpenCLIP representations alter three core components of active learning: initial labeled pool selection, diverse sampling, and the trade-off between representative and uncertainty sampling (Gupte et al., 2024). The framing is that semantically structured, robust embeddings make cold start tractable and permit cheap diversity enforcement in embedding space, especially in the low-budget regime (Gupte et al., 2024).

The paper’s empirical picture differs from older active learning intuitions derived from scratch-trained CNNs. First, initial pool selection matters strongly at ultra-low budget: centroid-based initialization in feature space yields very large early gains over random selection, with examples including CIFAR100 $t=1$ showing approximately $+24$ percentage points, Food101 approximately $+22$, ImageNet-100 approximately $+26$, and DomainNet-Real approximately $+23$ (Gupte et al., 2024). Second, diversity remains crucial: top-$B$ uncertainty acquisition tends to correlate selections, whereas clustering uncertain items improves performance substantially (Gupte et al., 2024). Third, the classical phase transition from representative sampling early to uncertainty sampling later largely disappears with foundation features; uncertainty-based queries become competitive or superior within one to two rounds (Gupte et al., 2024).

The proposed strategy, LimeGreen, combines uncertainty estimation via dropout-induced prediction inconsistency with diversity enforced by clustering (Gupte et al., 2024). Let $B_t$0 be a frozen foundation embedding and $B_t$1 a classifier. Under input dropout applied $B_t$2 times to $B_t$3, with dropout rate $B_t$4 and $B_t$5, the method compares stochastic predictions $B_t$6 with the base prediction $B_t$7 and defines

$B_t$8

where $B_t$9 (Gupte et al., 2024). Candidates are filtered by a threshold $S$0, then clustered in foundation-embedding space with $S$1, and the items nearest cluster centroids are selected (Gupte et al., 2024). The same paper also situates this strategy relative to entropy, BALD, variation ratio, margin, CoreSet, BADGE, Typiclust, ProbCover, ALFA-Mix, and PowerBALD (Gupte et al., 2024).

The quantitative pattern is consistent across datasets. Diversity-enforced uncertainty outperforms diversity-only methods on CIFAR100, Food101, ImageNet-100, and DomainNet-Real (Gupte et al., 2024). In the paper’s “Dropout Query” table, CIFAR100 at $S$2 improves from Random 64.5 to DQ 81.0 and DQ_c 83.5; ImageNet-100 improves from 76.5 to 88.8 and 89.4; Food101 from 64.4 to 72.7 and 76.1; DomainNet-Real from 61.8 to 73.0 and 75.0 (Gupte et al., 2024). Gains persist into mid rounds, as in CIFAR100 at $S$3, where 86.1 improves to 89.8–90.1 (Gupte et al., 2024). The study further reports strong results on fine-grained tasks, on Places365, and on out-of-domain biomedical datasets including Blood Smear, Diabetic Retinopathy, HeLa, and HAM10000, with significance assessed by paired $S$4-tests at $S$5 (Gupte et al., 2024).

This line of work broadens the meaning of AVFMs beyond embodied or glimpse-based models. Here, the “active” component is sample acquisition rather than camera control, but the same theme recurs: strong foundation priors change the statistical regime of active perception, making uncertainty usable earlier and making diversity enforcement in representation space both cheap and semantically meaningful (Gupte et al., 2024).

4. Scene-wide active vision encoders and latent workspaces

The most direct architectural realization of an AVFM appears in CanViT (Berreby et al., 23 Mar 2026). The model processes a large scene through sequential, localized glimpses defined by viewpoints $S$6, where $S$7 and $S$8, each resized to a fixed glimpse resolution (Berreby et al., 23 Mar 2026). At each timestep, a 128×128 crop is patchified into 16×16 patches, producing 64 patch tokens; with 5 backbone registers and CLS+VPE, the backbone token count is $S$9 for ViT-B (Berreby et al., 23 Mar 2026).

The canvas is a persistent $h_t = (o_{0:t}, a_{0:t})$0 grid of $h_t = (o_{0:t}, a_{0:t})$1-dimensional tokens plus non-spatial canvas registers. In CanViT-B, $h_t = (o_{0:t}, a_{0:t})$2, there are 16 canvas registers, and the pretraining canvas is $h_t = (o_{0:t}, a_{0:t})$3, giving $h_t = (o_{0:t}, a_{0:t})$4 tokens (Berreby et al., 23 Mar 2026). Interaction between backbone and canvas is implemented by Canvas Attention, an asymmetric cross-attention layer alternating Read and Write operations with stride 2 across the 12 ViT-B blocks, yielding 3 reads and 3 writes per glimpse (Berreby et al., 23 Mar 2026). The asymmetry is computationally motivated: learned QKVO projections are applied only on the backbone side, while the canvas side applies LayerNorm, scene-relative RoPE to Q/K, and residual addition (Berreby et al., 23 Mar 2026).

The efficiency argument is explicit. Adding a single $h_t = (o_{0:t}, a_{0:t})$5 linear projection on $h_t = (o_{0:t}, a_{0:t})$6 canvas tokens would incur a FLOP ratio relative to the accompanying SDPA of

$h_t = (o_{0:t}, a_{0:t})$7

which, with $h_t = (o_{0:t}, a_{0:t})$8 and $h_t = (o_{0:t}, a_{0:t})$9, is approximately $t=1$0 (Berreby et al., 23 Mar 2026). For a 32×32 canvas, the paper reports that adding canvas-side QKVO would raise each Read/Write pair from 1.1 to 9.8 GFLOPS (Berreby et al., 23 Mar 2026). The model therefore treats the canvas as a high-capacity working memory that can be large in token count but cheap in per-step compute.

Pretraining is label-free and policy-agnostic. A frozen DINOv3 ViT-B/16 teacher produces dense patch tokens and a CLS token at 512×512 scene resolution; CanViT reconstructs these scene-wide teacher embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths (Berreby et al., 23 Mar 2026). The dense latent distillation loss is

$t=1$1

where $t=1$2 decodes canvas tokens and $t=1$3 decodes the recurrent CLS token (Berreby et al., 23 Mar 2026). Two rollout branches are averaged during pretraining: F-IID, which starts with a full-scene glimpse and then samples i.i.d. random viewpoints, and R-IID, which uses i.i.d. random viewpoints throughout (Berreby et al., 23 Mar 2026). Training uses truncated BPTT with chunk size $t=1$4 and stop probability $t=1$5, giving average sequence length $t=1$6 while occasionally exposing longer sequences (Berreby et al., 23 Mar 2026).

The scale is notable. CanViT-B is pretrained from random initialization on 13.2 million ImageNet-21k scenes and approximately 1 billion random glimpses in 166 hours on a single H100 (Berreby et al., 23 Mar 2026). The reported per-glimpse cost at 8×8 glimpse grid and 32×32 canvas grid is approximately 15.9 GFLOPS, decomposed into 12.3 GFLOPS for PatchEmbed+ViT blocks, approximately 3.2 GFLOPS for 3 Reads + 3 Writes, and 0.3 GFLOPS for the segmentation head (Berreby et al., 23 Mar 2026). The passive DINOv3 ViT-B/16 teacher at 512×512, by contrast, uses approximately 215.2 GFLOPS per pass (Berreby et al., 23 Mar 2026). This is the paper’s core systems claim: active recurrent processing with a persistent spatial workspace can narrow the efficiency gap between dense scene understanding and low-latency inference.

5. Embodied AVFMs for occlusion-aware perception and manipulation

A different AVFM lineage appears in "VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility" (Shi et al., 16 Mar 2025). Here the active component is physical viewpoint control with an eye-in-hand RGB-D camera, and the foundation-model substrate consists of Qwen2.5-VL-72B-Instruct-AWQ, Grounding DINO, and SAM2 (Shi et al., 16 Mar 2025). The goal is target-oriented 6-DoF grasping when the target may be heavily occluded or entirely invisible (Shi et al., 16 Mar 2025).

The system couples three capabilities. First, FM-driven spatial reasoning produces structured instance-centric descriptions and relational cues, grounds them to 2D boxes and masks, lifts them to 3D, and maintains a historical object list with attributes, oriented bounding boxes, and relations such as Proximity, Below, High, and Low (Shi et al., 16 Mar 2025). The 3D lifting step is

$t=1$7

followed by PCA-based fitting of 3D oriented bounding boxes (Shi et al., 16 Mar 2025). Second, a target-guided Next-Best-View planner constructs a continuous velocity field over camera poses on a constrained hemisphere around the scene (Shi et al., 16 Mar 2025). For a target center $t=1$8, occluder center $t=1$9, and camera position $+24$0, the tangential direction is obtained by rejecting the occluder-to-target direction onto the sphere tangent, modulating its magnitude by an angle-dependent factor $+24$1, and integrating the resulting field until stagnation $+24$2 (Shi et al., 16 Mar 2025). Third, a probabilistic grasp module, vMF-Contact with a PointNeXt-B backbone, produces contact grasps with uncertainty distributions; Bayesian fusion updates contact points, vMF baseline direction parameters, and approach-bin scores across multiple views (Shi et al., 16 Mar 2025).

The system architecture is explicitly modular.

Module	Input	Output
AMOV3D	RGB-D stream and target language prompt	Multi-view instance set with labels, masks, point clouds, and oriented BBs
TGV-Planner	Historical object list and occluder relations	Velocity field and commanded camera motion
RT-UMGF	Depth frames	Fused multi-view grasp distributions and execution triggers

The experimental setup uses a UR10e arm, a Robotiq 2F85 gripper, and an Orbbec Femto Mega RGB-D wrist camera (Shi et al., 16 Mar 2025). Compared methods include static viewpoints, a version without grasp fusion, and Breyer’s closed-loop NBV with manually annotated target bounding boxes (Shi et al., 16 Mar 2025). Aggregate results report AFSR 87.50%, #AGA 3.10, and AGSR 83.86% for the full system, compared with 70.00%, 3.43, and 73.19% without grasp fusion, and substantially lower figures for static and reconstruction-centric baselines (Shi et al., 16 Mar 2025). The paper concludes that active NBV and uncertainty-guided multi-view fusion are jointly critical under occlusion (Shi et al., 16 Mar 2025).

This robotic line expands AVFMs beyond scene understanding into active intervention. A plausible implication is that the defining AVFM ingredients are not tied to a particular embodiment: language-conditioned object semantics, spatially grounded memory, and active view planning can operate at the level of dataset querying, camera-glimpse sequencing, or robotic sensorimotor control.

6. Empirical performance, evaluation regimes, and failure analysis

The four cited works collectively suggest that AVFMs should be evaluated along at least three axes: label efficiency under selective querying, scene understanding under sequential glimpses, and belief quality under autonomous exploration.

For dense active vision, CanViT reports that a frozen CanViT-B achieves 38.5% ADE20K mIoU from a single low-resolution glimpse at 15.86 GFLOPS, surpassing the best active model’s 27.6% with 19.5× fewer inference FLOPs and also outperforming FLOP- or input-matched teacher settings (Berreby et al., 23 Mar 2026). With additional glimpses, it reaches 45.9% ADE20K mIoU on 1024×1024 scenes with a 64×64 canvas under the unseen Coarse-to-Fine policy, and it attains 81.2% ImageNet-1k top-1 accuracy at $+24$3 with frozen teacher probes (Berreby et al., 23 Mar 2026). Generalization is reported across longer rollouts, larger scenes and canvases, and unseen structured or dynamic policies such as C2F, F2C, and entropy-guided C2F (Berreby et al., 23 Mar 2026).

The same paper’s ablations identify several architectural sensitivities. Reducing canvas width from 1024 to 256 degrades patch reconstruction by 12.0% and CLS by 2.1%; removing canvas reads reduces patch by 6.5% and CLS by 8.0%; reducing read/write frequency degrades patch by 4.1% and CLS by 6.0%; and removing patch-level loss collapses spatial reconstruction by 98.8% (Berreby et al., 23 Mar 2026). These results are consistent with the claim that dense supervision, frequent bidirectional interaction, and large-capacity memory are necessary for AVFMs aimed at dense prediction.

For embodied spatial belief, the "Theory of Space" benchmark provides a contrasting diagnostic. In the vision setting, active models remain substantially below human performance and below their passive counterparts (Zhang et al., 4 Feb 2026). Exploration is inefficient relative to program proxies such as Scout: Scout reaches target coverage in approximately 9 steps, while active models require at least 14 steps and do not exceed proxy performance (Zhang et al., 4 Feb 2026). On cognitive map probing, GPT-5.2 in vision has overall correctness 32.2%, with orientation 20.2 and position 42.0; Gemini-3 Pro achieves 52.1%, with orientation 32.2 and position 62.5 (Zhang et al., 4 Feb 2026). Stability scores indicate that previously correct beliefs are later degraded, and Uncertainty F1 is limited, especially for GPT-5.2 at 53.7 (Zhang et al., 4 Feb 2026).

The false belief paradigm sharpens this diagnosis. After changes to object locations or orientations, vision agents exhibit poor change identification and severe Belief Inertia, especially for orientation (Zhang et al., 4 Feb 2026). For changed objects, vision GPT-5.2 attains orientation/position identification F1 of 14.3/68.0 and updated belief correctness of 16.7/42.9, with Belief Inertia 68.9/34.7; Gemini-3 Pro improves these figures but still exhibits substantial orientation inertia at 51.1 (Zhang et al., 4 Feb 2026). The paper’s conclusion is that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration (Zhang et al., 4 Feb 2026).

A common misconception is therefore that strong passive perception automatically yields strong active vision. The active learning study shows that robust foundation features can indeed make uncertainty useful earlier (Gupte et al., 2024), and CanViT shows that active encoders can become highly competitive on dense prediction (Berreby et al., 23 Mar 2026). Yet the exploration benchmark demonstrates that autonomous information gathering, map maintenance, and belief revision remain unsolved at the level of general-purpose agents (Zhang et al., 4 Feb 2026).

7. Broader significance, limitations, and research directions

Across these works, AVFMs emerge less as a single model class than as a research axis linking foundation-model priors with active information acquisition. In active learning, robust embeddings flatten the classical representative-to-uncertainty phase transition and make simple cluster-based diversity mechanisms effective in the low-budget regime (Gupte et al., 2024). In active vision architecture, decoupling retinotopic computation from spatiotopic memory with scene-relative binding enables scalable recurrent inference and strong zero-shot dense transfer (Berreby et al., 23 Mar 2026). In robotics, object-centric language grounding, continuous NBV control, and uncertainty-aware grasp fusion permit target-oriented manipulation under severe occlusion and invisibility (Shi et al., 16 Mar 2025). In evaluation, explicit probing of cognitive maps and false-belief revision exposes the unresolved challenges of belief instability, inefficient exploration, and inertia under changing evidence (Zhang et al., 4 Feb 2026).

Several limitations recur. Dependence on embedding quality is explicit in active learning: poor domain coverage can reduce uncertainty reliability and misaligned features can over-represent outlier clusters (Gupte et al., 2024). CanViT relies on a frozen passive teacher for dense latent distillation, may suffer from extreme zoom aliasing, and retains capacity bottlenecks through fixed canvas width and resolution (Berreby et al., 23 Mar 2026). VISO-Grasp notes non-negligible VLM latency, vulnerability to FM semantic errors, and difficulty under extreme clutter or challenging materials (Shi et al., 16 Mar 2025). The Theory of Space study shows that even when perception is acceptable, active models can fail to construct stable and revisable spatial beliefs, particularly for orientations in vision (Zhang et al., 4 Feb 2026).

The research directions suggested in the source material are correspondingly diverse. For active learning, future work includes semi-supervised active learning with better calibration, adaptive scheduling between uncertainty and diversity, and extension to localization and segmentation (Gupte et al., 2024). For active vision encoders, natural extensions include multimodal inputs, video with motor control, larger models, and task-specific fine-tuning (Berreby et al., 23 Mar 2026). For embodied manipulation, the next steps include better real-time scheduling, stronger handling of semantic errors, and broader robustness under domain shift (Shi et al., 16 Mar 2025). For general active spatial intelligence, the proposed path is to ground AVFMs in explicit uncertainty-aware cognitive maps, SLAM-inspired belief updates, information-gain-driven exploration, and evaluation protocols that directly measure Active-Passive Gap, Stability, Local↔Global consistency, and Belief Inertia (Zhang et al., 4 Feb 2026).

Taken together, these works suggest that AVFMs are best understood as foundation-model systems in which representation learning, memory, uncertainty, and control are jointly optimized around the question of what to observe next. The field has already shown that strong foundation priors can materially improve active querying, dense glimpse-based scene understanding, and occlusion-aware robotic perception. At the same time, current evidence indicates that general-purpose active spatial cognition—especially stable belief revision under autonomous exploration—remains a central open problem rather than a solved capability.