---
title: Active-Vision Foundation Models
url: https://www.emergentmind.com/topics/active-vision-foundation-models-avfms
type: topic
---

# Active-Vision Foundation Models

Searching arXiv for the specified papers to ground the article and verify bibliographic details.
Active-Vision Foundation Models (AVFMs) denote a class of systems in which perception is not restricted to a single passive feedforward encoding of a full scene, but is instead coupled to sequential information acquisition, memory, and view selection, while retaining the general-purpose transfer properties associated with foundation models. Across recent work, the term covers at least three closely related formulations: active learning systems that leverage frozen or lightly adapted vision foundation representations to improve label efficiency [2401.14555]; active vision encoders that accumulate scene understanding from localized glimpses into a persistent spatial workspace and transfer across tasks and viewing policies [2603.22570]; and embodied perception-planning systems that use vision-language and segmentation/detection foundation models to drive viewpoint control, spatial reasoning, and manipulation under occlusion [2503.12609]. A complementary line of evaluation frames the central challenge as constructing, revising, and exploiting a spatial belief through self-directed exploration, and shows that current foundation-model agents still exhibit substantial deficits in active spatial intelligence [2602.07055].

## 1. Terminological scope and conceptual definition

The recent literature does not use a single uniform definition of AVFMs. In work on active learning with vision foundation models, the term is not explicit; it is introduced as a useful label for active learning systems that leverage frozen or lightly adapted representations from models such as DINOv2 and OpenCLIP to maximize labeling efficiency [2401.14555]. In work on active computer vision, an AVFM is defined more structurally as an active vision encoder whose scene representation and processing are general-purpose, task-agnostic, and policy-agnostic, making it transferable across downstream tasks and viewing policies and scalable to large scenes and long rollouts [2603.22570]. In robotic grasping, AVFMs are operationalized as perception-planning systems that leverage off-the-shelf vision-language and segmentation/detection foundation models to drive object-centric spatial reasoning and active viewpoint control [2503.12609].

Despite this heterogeneity, the formulations share a common substrate. Passive vision encoders process a scene in a single feedforward pass and do not reuse computation over time, whereas active systems select what to observe next, integrate information across timesteps, and maintain a state that persists across partial observations [2603.22570]. This suggests a broad unifying characterization: AVFMs combine foundation-model priors with sequential sensing, explicit or implicit memory, and some mechanism for choosing informative observations.

A recurrent misconception is that AVFMs are simply passive vision foundation models with an added control loop. The surveyed work points to a stronger requirement. In active learning, the representation geometry of frozen foundation embeddings changes the behavior of query strategies themselves [2401.14555]. In active vision, the encoder architecture must support persistent scene-wide state and low-latency recurrent updates [2603.22570]. In embodied exploration, the decisive issue is not merely perception quality but the ability to maintain a revisable spatial belief during autonomous information gathering [2602.07055].

## 2. Representation, memory, and the active-passive distinction

A central distinction in AVFM research is between retinotopic processing of the current observation and spatiotopic or allocentric memory over the scene. "CanViT: Toward Active-Vision Foundation Models" formalizes this by decoupling a retinotopic Vision Transformer backbone from a spatiotopic scene-wide latent workspace called the canvas [2603.22570]. The backbone processes each glimpse’s patch tokens, a recurrent CLS token, ephemeral register tokens, and a viewpoint encoding token, while the canvas is a persistent grid of tokens anchored to scene coordinates and updated only through cross-attention. The paper explicitly removes canvas-side self-attention, MLPs, and canvas-side QKVO, enforcing a “no compute on canvas” constraint that preserves scalability to large spatial workspaces [2603.22570].

The role of spatial binding is equally central. CanViT uses scene-relative 2D RoPE to express both glimpse patch centers and canvas patch centers in shared scene coordinates, so that active views at varying scales are written into and read from a common spatial frame [2603.22570]. A dedicated viewpoint encoding token supplements this mechanism through a scale-invariant map $u(x, y, s) = (x/s, y/s, \log s)$ lifted with Random Fourier Features and LayerNorm [2603.22570]. The architectural intent is explicit: compact retinotopic computation performs the local “thinking,” while the large spatiotopic canvas performs the “memory” [2603.22570].

The same distinction appears, in different form, in the "Theory of Space" benchmark. There, the relevant internal object is the spatial belief $B_t$, a probabilistic representation over spatial structure $S$ conditioned on observation-action history $h_t = (o_{0:t}, a_{0:t})$ [2602.07055]. The benchmark treats a cognitive map as the canonical externalization of this belief and probes it at every step using a JSON map of object positions and headings together with a local snapshot [2602.07055]. The resulting diagnosis is that perception is only one bottleneck: even when local perception is adequate, global beliefs suffer from instability and later observations can overwrite previously correct facts, producing belief drift [2602.07055].

This distinction clarifies why active and passive competence diverge. In the same benchmark, the Active-Passive Gap is defined as the performance drop when models must autonomously gather information rather than receive standardized exploration logs [2602.07055]. In the vision world, overall averages include GPT-5.2 dropping from 57.1 to 46.0 and Gemini-3 Pro dropping from 60.5 to 57.3 [2602.07055]. The implication is not merely that exploration is hard; it is that active spatial intelligence requires an integrated stack of perceptual fidelity, explicit or implicit memory, stable update rules, and information-seeking control.

## 3. AVFMs for label-efficient visual learning

One major AVFM instantiation emerges in active learning with frozen foundation features. "Revisiting Active Learning in the Era of Vision Foundation Models" studies how DINOv2 and OpenCLIP representations alter three core components of active learning: initial labeled pool selection, diverse sampling, and the trade-off between representative and uncertainty sampling [2401.14555]. The framing is that semantically structured, robust embeddings make cold start tractable and permit cheap diversity enforcement in embedding space, especially in the low-budget regime [2401.14555].

The paper’s empirical picture differs from older active learning intuitions derived from scratch-trained CNNs. First, initial pool selection matters strongly at ultra-low budget: centroid-based initialization in feature space yields very large early gains over random selection, with examples including CIFAR100 $t=1$ showing approximately $+24$ percentage points, Food101 approximately $+22$, ImageNet-100 approximately $+26$, and DomainNet-Real approximately $+23$ [2401.14555]. Second, diversity remains crucial: top-$B$ uncertainty acquisition tends to correlate selections, whereas clustering uncertain items improves performance substantially [2401.14555]. Third, the classical phase transition from representative sampling early to uncertainty sampling later largely disappears with foundation features; uncertainty-based queries become competitive or superior within one to two rounds [2401.14555].

The proposed strategy, LimeGreen, combines uncertainty estimation via dropout-induced prediction inconsistency with diversity enforced by clustering [2401.14555]. Let $z = f(x)$ be a frozen foundation embedding and $g(\cdot;\theta)$ a classifier. Under input dropout applied $M$ times to $z$, with dropout rate $\rho = 0.75$ and $M=3$, the method compares stochastic predictions $\hat{y}^m$ with the base prediction $\hat{y}$ and defines
$$
U(x) = 1 - \frac{n}{M},
$$
where $n = \sum_{m=1}^{M} \mathbf{1}[\hat{y}^m = \hat{y}]$ [2401.14555]. Candidates are filtered by a threshold $\tau \approx 0.5$, then clustered in foundation-embedding space with $k=B$, and the items nearest cluster centroids are selected [2401.14555]. The same paper also situates this strategy relative to entropy, BALD, variation ratio, margin, CoreSet, BADGE, Typiclust, ProbCover, ALFA-Mix, and PowerBALD [2401.14555].

The quantitative pattern is consistent across datasets. Diversity-enforced uncertainty outperforms diversity-only methods on CIFAR100, Food101, ImageNet-100, and DomainNet-Real [2401.14555]. In the paper’s “Dropout Query” table, CIFAR100 at $t=2$ improves from Random 64.5 to DQ 81.0 and DQ\_c 83.5; ImageNet-100 improves from 76.5 to 88.8 and 89.4; Food101 from 64.4 to 72.7 and 76.1; DomainNet-Real from 61.8 to 73.0 and 75.0 [2401.14555]. Gains persist into mid rounds, as in CIFAR100 at $t=8$, where 86.1 improves to 89.8–90.1 [2401.14555]. The study further reports strong results on fine-grained tasks, on Places365, and on out-of-domain biomedical datasets including Blood Smear, Diabetic Retinopathy, HeLa, and HAM10000, with significance assessed by paired $t$-tests at $p<0.05$ [2401.14555].

This line of work broadens the meaning of AVFMs beyond embodied or glimpse-based models. Here, the “active” component is sample acquisition rather than camera control, but the same theme recurs: strong foundation priors change the statistical regime of active perception, making uncertainty usable earlier and making diversity enforcement in representation space both cheap and semantically meaningful [2401.14555].

## 4. Scene-wide active vision encoders and latent workspaces

The most direct architectural realization of an AVFM appears in CanViT [2603.22570]. The model processes a large scene through sequential, localized glimpses defined by viewpoints $v=(x,y,s)$, where $(x,y)\in[-1,+1]^2$ and $s\in(0,1]$, each resized to a fixed glimpse resolution [2603.22570]. At each timestep, a 128×128 crop is patchified into 16×16 patches, producing 64 patch tokens; with 5 backbone registers and CLS+VPE, the backbone token count is $N_{bb}=71$ for ViT-B [2603.22570].

The canvas is a persistent $H_c \times W_c$ grid of $D_{can}$-dimensional tokens plus non-spatial canvas registers. In CanViT-B, $D_{can}=1024$, there are 16 canvas registers, and the pretraining canvas is $32 \times 32$, giving $N_{can}=32^2+16=1040$ tokens [2603.22570]. Interaction between backbone and canvas is implemented by Canvas Attention, an asymmetric cross-attention layer alternating Read and Write operations with stride 2 across the 12 ViT-B blocks, yielding 3 reads and 3 writes per glimpse [2603.22570]. The asymmetry is computationally motivated: learned QKVO projections are applied only on the backbone side, while the canvas side applies LayerNorm, scene-relative RoPE to Q/K, and residual addition [2603.22570].

The efficiency argument is explicit. Adding a single $D_{can}\rightarrow D_{can}$ linear projection on $N_{can}$ canvas tokens would incur a FLOP ratio relative to the accompanying SDPA of
$$
\text{ratio} = \frac{2N_{can}D_{can}^2}{4N_{bb}N_{can}D_{can}} = \frac{D_{can}}{2N_{bb}},
$$
which, with $D_{can}=1024$ and $N_{bb}=71$, is approximately $7.2\times$ [2603.22570]. For a 32×32 canvas, the paper reports that adding canvas-side QKVO would raise each Read/Write pair from 1.1 to 9.8 GFLOPS [2603.22570]. The model therefore treats the canvas as a high-capacity working memory that can be large in token count but cheap in per-step compute.

Pretraining is label-free and policy-agnostic. A frozen DINOv3 ViT-B/16 teacher produces dense patch tokens and a CLS token at 512×512 scene resolution; CanViT reconstructs these scene-wide teacher embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths [2603.22570]. The dense latent distillation loss is
$$
L = \frac{1}{T} \sum_{t=0}^{T-1} \left[ \frac{1}{HW} \|Z_t-\mathcal{Z}\|_2^2 + \|z_t-\zeta\|_2^2 \right],
$$
where $Z_t = W_{\text{spatial}}\cdot \text{LayerNorm}(C_t)$ decodes canvas tokens and $z_t = W_{\text{global}}\cdot \text{LayerNorm}(h_t)$ decodes the recurrent CLS token [2603.22570]. Two rollout branches are averaged during pretraining: F-IID, which starts with a full-scene glimpse and then samples i.i.d. random viewpoints, and R-IID, which uses i.i.d. random viewpoints throughout [2603.22570]. Training uses truncated BPTT with chunk size $K=2$ and stop probability $p_{\text{stop}}=0.5$, giving average sequence length $T=4$ while occasionally exposing longer sequences [2603.22570].

The scale is notable. CanViT-B is pretrained from random initialization on 13.2 million ImageNet-21k scenes and approximately 1 billion random glimpses in 166 hours on a single H100 [2603.22570]. The reported per-glimpse cost at 8×8 glimpse grid and 32×32 canvas grid is approximately 15.9 GFLOPS, decomposed into 12.3 GFLOPS for PatchEmbed+ViT blocks, approximately 3.2 GFLOPS for 3 Reads + 3 Writes, and 0.3 GFLOPS for the segmentation head [2603.22570]. The passive DINOv3 ViT-B/16 teacher at 512×512, by contrast, uses approximately 215.2 GFLOPS per pass [2603.22570]. This is the paper’s core systems claim: active recurrent processing with a persistent spatial workspace can narrow the efficiency gap between dense scene understanding and low-latency inference.

## 5. Embodied AVFMs for occlusion-aware perception and manipulation

A different AVFM lineage appears in "VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility" [2503.12609]. Here the active component is physical viewpoint control with an eye-in-hand RGB-D camera, and the foundation-model substrate consists of Qwen2.5-VL-72B-Instruct-AWQ, Grounding DINO, and SAM2 [2503.12609]. The goal is target-oriented 6-DoF grasping when the target may be heavily occluded or entirely invisible [2503.12609].

The system couples three capabilities. First, FM-driven spatial reasoning produces structured instance-centric descriptions and relational cues, grounds them to 2D boxes and masks, lifts them to 3D, and maintains a historical object list with attributes, oriented bounding boxes, and relations such as Proximity, Below, High, and Low [2503.12609]. The 3D lifting step is
$$
(X,Y,Z)^{\top} = \mathbf{D}(u,v)\cdot K^{-1}\cdot (u,v,1)^{\top},
$$
followed by PCA-based fitting of 3D oriented bounding boxes [2503.12609]. Second, a target-guided Next-Best-View planner constructs a continuous velocity field over camera poses on a constrained hemisphere around the scene [2503.12609]. For a target center $\mathbf{p}_*$, occluder center $\mathbf{p}_{oc}$, and camera position $\mathbf{x}$, the tangential direction is obtained by rejecting the occluder-to-target direction onto the sphere tangent, modulating its magnitude by an angle-dependent factor $\beta$, and integrating the resulting field until stagnation $\|\dot{\mathbf{x}}\|\approx 0$ [2503.12609]. Third, a probabilistic grasp module, vMF-Contact with a PointNeXt-B backbone, produces contact grasps with uncertainty distributions; Bayesian fusion updates contact points, vMF baseline direction parameters, and approach-bin scores across multiple views [2503.12609].

The system architecture is explicitly modular.

| Module | Input | Output |
|---|---|---|
| AMOV3D | RGB-D stream and target language prompt | Multi-view instance set with labels, masks, point clouds, and oriented BBs |
| TGV-Planner | Historical object list and occluder relations | Velocity field and commanded camera motion |
| RT-UMGF | Depth frames | Fused multi-view grasp distributions and execution triggers |

The experimental setup uses a UR10e arm, a Robotiq 2F85 gripper, and an Orbbec Femto Mega RGB-D wrist camera [2503.12609]. Compared methods include static viewpoints, a version without grasp fusion, and Breyer’s closed-loop NBV with manually annotated target bounding boxes [2503.12609]. Aggregate results report AFSR 87.50%, #AGA 3.10, and AGSR 83.86% for the full system, compared with 70.00%, 3.43, and 73.19% without grasp fusion, and substantially lower figures for static and reconstruction-centric baselines [2503.12609]. The paper concludes that active NBV and uncertainty-guided multi-view fusion are jointly critical under occlusion [2503.12609].

This robotic line expands AVFMs beyond scene understanding into active intervention. A plausible implication is that the defining AVFM ingredients are not tied to a particular embodiment: language-conditioned object semantics, spatially grounded memory, and active view planning can operate at the level of dataset querying, camera-glimpse sequencing, or robotic sensorimotor control.

## 6. Empirical performance, evaluation regimes, and failure analysis

The four cited works collectively suggest that AVFMs should be evaluated along at least three axes: label efficiency under selective querying, scene understanding under sequential glimpses, and belief quality under autonomous exploration.

For dense active vision, CanViT reports that a frozen CanViT-B achieves 38.5% ADE20K mIoU from a single low-resolution glimpse at 15.86 GFLOPS, surpassing the best active model’s 27.6% with 19.5× fewer inference FLOPs and also outperforming FLOP- or input-matched teacher settings [2603.22570]. With additional glimpses, it reaches 45.9% ADE20K mIoU on 1024×1024 scenes with a 64×64 canvas under the unseen Coarse-to-Fine policy, and it attains 81.2% ImageNet-1k top-1 accuracy at $T=21$ with frozen teacher probes [2603.22570]. Generalization is reported across longer rollouts, larger scenes and canvases, and unseen structured or dynamic policies such as C2F, F2C, and entropy-guided C2F [2603.22570].

The same paper’s ablations identify several architectural sensitivities. Reducing canvas width from 1024 to 256 degrades patch reconstruction by 12.0% and CLS by 2.1%; removing canvas reads reduces patch by 6.5% and CLS by 8.0%; reducing read/write frequency degrades patch by 4.1% and CLS by 6.0%; and removing patch-level loss collapses spatial reconstruction by 98.8% [2603.22570]. These results are consistent with the claim that dense supervision, frequent bidirectional interaction, and large-capacity memory are necessary for AVFMs aimed at dense prediction.

For embodied spatial belief, the "Theory of Space" benchmark provides a contrasting diagnostic. In the vision setting, active models remain substantially below human performance and below their passive counterparts [2602.07055]. Exploration is inefficient relative to program proxies such as Scout: Scout reaches target coverage in approximately 9 steps, while active models require at least 14 steps and do not exceed proxy performance [2602.07055]. On cognitive map probing, GPT-5.2 in vision has overall correctness 32.2%, with orientation 20.2 and position 42.0; Gemini-3 Pro achieves 52.1%, with orientation 32.2 and position 62.5 [2602.07055]. Stability scores indicate that previously correct beliefs are later degraded, and Uncertainty F1 is limited, especially for GPT-5.2 at 53.7 [2602.07055].

The false belief paradigm sharpens this diagnosis. After changes to object locations or orientations, vision agents exhibit poor change identification and severe Belief Inertia, especially for orientation [2602.07055]. For changed objects, vision GPT-5.2 attains orientation/position identification F1 of 14.3/68.0 and updated belief correctness of 16.7/42.9, with Belief Inertia 68.9/34.7; Gemini-3 Pro improves these figures but still exhibits substantial orientation inertia at 51.1 [2602.07055]. The paper’s conclusion is that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration [2602.07055].

A common misconception is therefore that strong passive perception automatically yields strong active vision. The active learning study shows that robust foundation features can indeed make uncertainty useful earlier [2401.14555], and CanViT shows that active encoders can become highly competitive on dense prediction [2603.22570]. Yet the exploration benchmark demonstrates that autonomous information gathering, map maintenance, and belief revision remain unsolved at the level of general-purpose agents [2602.07055].

## 7. Broader significance, limitations, and research directions

Across these works, AVFMs emerge less as a single model class than as a research axis linking foundation-model priors with active information acquisition. In active learning, robust embeddings flatten the classical representative-to-uncertainty phase transition and make simple cluster-based diversity mechanisms effective in the low-budget regime [2401.14555]. In active vision architecture, decoupling retinotopic computation from spatiotopic memory with scene-relative binding enables scalable recurrent inference and strong zero-shot dense transfer [2603.22570]. In robotics, object-centric language grounding, continuous NBV control, and uncertainty-aware grasp fusion permit target-oriented manipulation under severe occlusion and invisibility [2503.12609]. In evaluation, explicit probing of cognitive maps and false-belief revision exposes the unresolved challenges of belief instability, inefficient exploration, and inertia under changing evidence [2602.07055].

Several limitations recur. Dependence on embedding quality is explicit in active learning: poor domain coverage can reduce uncertainty reliability and misaligned features can over-represent outlier clusters [2401.14555]. CanViT relies on a frozen passive teacher for dense latent distillation, may suffer from extreme zoom aliasing, and retains capacity bottlenecks through fixed canvas width and resolution [2603.22570]. VISO-Grasp notes non-negligible VLM latency, vulnerability to FM semantic errors, and difficulty under extreme clutter or challenging materials [2503.12609]. The Theory of Space study shows that even when perception is acceptable, active models can fail to construct stable and revisable spatial beliefs, particularly for orientations in vision [2602.07055].

The research directions suggested in the source material are correspondingly diverse. For active learning, future work includes semi-supervised active learning with better calibration, adaptive scheduling between uncertainty and diversity, and extension to localization and segmentation [2401.14555]. For active vision encoders, natural extensions include multimodal inputs, video with motor control, larger models, and task-specific fine-tuning [2603.22570]. For embodied manipulation, the next steps include better real-time scheduling, stronger handling of semantic errors, and broader robustness under domain shift [2503.12609]. For general active spatial intelligence, the proposed path is to ground AVFMs in explicit uncertainty-aware cognitive maps, SLAM-inspired belief updates, information-gain-driven exploration, and evaluation protocols that directly measure Active-Passive Gap, Stability, Local↔Global consistency, and Belief Inertia [2602.07055].

Taken together, these works suggest that AVFMs are best understood as foundation-model systems in which representation learning, memory, uncertainty, and control are jointly optimized around the question of what to observe next. The field has already shown that strong foundation priors can materially improve active querying, dense glimpse-based scene understanding, and occlusion-aware robotic perception. At the same time, current evidence indicates that general-purpose active spatial cognition—especially stable belief revision under autonomous exploration—remains a central open problem rather than a solved capability.

Source: https://www.emergentmind.com/topics/active-vision-foundation-models-avfms