CanViT: Dual-Stream Active-Vision Model

• The paper introduces CanViT, which builds persistent scene-level representations through a dual-stream recurrent architecture combining a retinotopic backbone and a spatiotopic canvas.
• CanViT employs scene-relative Rotary Position Embeddings and an asymmetric Canvas Attention mechanism to efficiently fuse localized glimpse features with global memory.
• Pretrained on large-scale datasets, CanViT achieves strong results in semantic segmentation and classification by decoupling perception from policy learning.

Searching arXiv for CanViT and related papers mentioned in the provided data. CanViT is an Active-Vision Foundation Model (AVFM) introduced as a task- and policy-agnostic architecture for constructing persistent scene-level representations from sequences of localized glimpses rather than from a single global image. It is formulated as a dual-stream recurrent Vision Transformer in which a retinotopic backbone processes the current glimpse and a spatiotopic latent workspace, the canvas, maintains scene-wide state across timesteps. The model combines scene-relative Rotary Position Embeddings, an asymmetric cross-attention mechanism called Canvas Attention, and a label-free pretraining objective based on reconstructing scene-wide DINOv3 embeddings from randomized glimpse sequences. In the paper’s framing, this establishes the first foundation-model-style approach to active vision and targets the core perception problem independently of the policy used to select future viewpoints (Berreby et al., 23 Mar 2026).

1. Problem setting and conceptual scope

The paper organizes active vision around three separable axes: instantaneous vision, memory, and action selection or policy. Instantaneous vision concerns what is perceived in a single glimpse; memory concerns accumulation of a persistent scene representation over time; and policy concerns where and at what zoom to look next. CanViT is designed to isolate the first two axes from the third by learning a reusable perceptual and recurrent model without coupling it to a learned control policy (Berreby et al., 23 Mar 2026).

This separation is central to the model’s claim of generality. Prior active-vision work is characterized as policy-centric, whereas passive foundation models such as DINOv3 are described as strong at instantaneous vision but lacking explicit mechanisms for sequential glimpses and working memory. CanViT therefore targets what the paper presents as the missing middle: a general-purpose model that can ingest variable localized crops, maintain a persistent scene-wide latent state, and support both dense and non-spatial decoding. A plausible implication is that the architecture is intended not merely as a task-specific active model, but as a reusable representation backbone for downstream active-perception settings.

The paper’s strongest empirical framing is semantic segmentation. It argues that the principal bottleneck in active vision, especially for dense prediction, is the quality of the perceptual and recurrent model rather than policy learning alone. This suggests a shift in emphasis from end-to-end policy optimization toward scalable pretrained active representations.

2. Dual-stream recurrent architecture

CanViT is a dual-stream recurrent ViT composed of a retinotopic backbone and a spatiotopic scene-wide latent workspace called the canvas (Berreby et al., 23 Mar 2026). At timestep $t$, the input is a glimpse defined by

$v_t = (x_t, y_t, s_t),$

where $(x_t, y_t) \in [-1,1]^2$ denotes the crop center in scene coordinates and $s_t \in (0,1]$ denotes crop scale or half-side-length. The crop is always resized to a fixed input resolution, so scale controls the tradeoff between spatial coverage and local detail.

The glimpse is patchified and processed by a ViT-B/16 backbone containing patch tokens, a recurrent CLS token, a viewpoint encoding token, and ephemeral register tokens. The recurrent CLS token carries non-spatial state across time within the backbone stream, while the remaining components provide the tokenized representation of the current view.

The canvas is the model’s persistent memory. It consists of a small set of non-spatial canvas registers together with a dense $H \times W$ grid of spatial canvas patches. Each canvas patch corresponds to a fixed region of scene coordinates rather than to the current retina-centered view, making the canvas directly decodable into dense outputs such as segmentation masks. The canvas is initialized from a single learned patch and then broadcast to any desired grid size at inference time, allowing canvas resolution to change at test time. This test-time resizability is one of the paper’s defining design choices because it ties memory capacity to scene scale rather than to a fixed training lattice.

The architecture is explicitly described as separating “thinking” from “memory.” The backbone is the thinking stream, where most computation occurs on the current glimpse, whereas the canvas is the memory stream, where scene state persists across timesteps.

3. Scene-relative geometry and Canvas Attention

A key mechanism in CanViT is Scene-Relative Rotary Position Embeddings (SR-RoPE), computed in the shared scene coordinate system $[-1,1]^2$ for both backbone patch tokens and canvas tokens (Berreby et al., 23 Mar 2026). This binds retinotopic and spatiotopic representations to the same reference frame, so localized glimpse features and scene-wide memory tokens are positionally compatible.

The viewpoint encoding token represents $(x,y,s)$ as

$(x, y, \log s),$

which the paper argues gives scale-invariance, translation invariance, and planar isotropy. This triplet is lifted into embedding space using Random Fourier Features and LayerNorm. In ablations, however, removing the viewpoint encoding token has only a modest effect, which the paper interprets as evidence that SR-RoPE already captures much of the viewpoint structure.

The paper’s most distinctive systems-level decision is the elimination of all canvas-side self-attention and fully connected layers. Canvas tokens do not pass through MLPs, self-attention, or QKV/output projections on the canvas side. They evolve only through asymmetric cross-attention interactions with the backbone. The stated motivation is scalability: the canvas contains many more tokens than the backbone, so quadratic or dense computation on the canvas would be prohibitively expensive.

These interactions are implemented through Canvas Attention, which alternates Read and Write operations across depth with stride 2 in the backbone blocks. In Read, backbone tokens query canvas tokens and the output is residually added to the backbone stream. In Write, canvas tokens query backbone tokens and the result is residually added to the canvas. The asymmetry is deliberate: heavy QKVO projections are placed on the backbone side, while the canvas side uses only LayerNorm, RoPE, and residual addition. The appendix summary describes Read as using backbone-side query and output projections with identity key and value on the canvas side, and Write as using backbone-side key and value projections with identity query and output on the canvas side. The paper gives the effective cost advantage through the ratio

$$\frac{\text{projection FLOPs}}{\text{SDPA FLOPs}} = \frac{4N_{bb}N_{can} + 2N_{can}D_{can}}{D_{can}N_{bb}},$$

and reports that for the CanViT-B setting this is about $7.2\times$, motivating the removal of canvas-side projections.

4. Pretraining objective and rollout construction

CanViT is pretrained with a label-free active vision objective termed policy-agnostic passive-to-active dense latent distillation (Berreby et al., 23 Mar 2026). The target is not pixels but scene-wide latent structure: the model reconstructs dense DINOv3 ViT-B teacher features for the whole scene from a sequence of low-resolution glimpses. The stored teacher targets include dense patch tokens and the teacher CLS token, with teacher inference excluded from the training loop by precomputing and storing the features.

Teacher features are standardized per spatial position and embedding dimension using z-score normalization, with statistics computed from 4096 images. After each timestep $v_t = (x_t, y_t, s_t),$0, CanViT produces canvas patch tokens $v_t = (x_t, y_t, s_t),$1 and a recurrent CLS token $v_t = (x_t, y_t, s_t),$2, which are decoded into DINOv3 space by linear projections: $v_t = (x_t, y_t, s_t),$3 The loss averages dense patch-level and global CLS-level reconstruction over time: $v_t = (x_t, y_t, s_t),$4 with the exact standardized targets defined by teacher feature statistics.

Policy agnosticism is enforced through dual rollouts. Each training step runs two independent branches: R-IID, in which all timesteps including $v_t = (x_t, y_t, s_t),$5 are random; and F-IID, in which the first timestep is the full-scene view $v_t = (x_t, y_t, s_t),$6 followed by random crops. Their losses are averaged. The paper reports that this improves convergence and robustness relative to using two purely random rollouts, suggesting that a guaranteed global-context observation at the start of one branch regularizes the latent memory formation process.

For non-initial timesteps, and for the initial R-IID step, viewpoint sampling proceeds by drawing $v_t = (x_t, y_t, s_t),$7, setting $v_t = (x_t, y_t, s_t),$8, and then sampling $v_t = (x_t, y_t, s_t),$9 uniformly over the valid crop-center region. The induced marginal scale density is

$(x_t, y_t) \in [-1,1]^2$0

which biases the distribution toward smaller and more localized glimpses. The minimum glimpse size is

$(x_t, y_t) \in [-1,1]^2$1

corresponding to $(x_t, y_t) \in [-1,1]^2$2 of scene area.

Temporal optimization uses truncated BPTT with chunk size

$(x_t, y_t) \in [-1,1]^2$3

At each chunk boundary, the rollout stops with probability

$(x_t, y_t) \in [-1,1]^2$4

yielding an average training rollout length

$(x_t, y_t) \in [-1,1]^2$5

The paper notes that this still occasionally exposes the model to longer sequences.

5. Pretraining scale, compute, and benchmark results

The flagship instantiation is CanViT-B, pretrained from random initialization on 13.2 million ImageNet-21k scenes using about 1 billion random $(x_t, y_t) \in [-1,1]^2$6 glimpses, with a $(x_t, y_t) \in [-1,1]^2$7 canvas during pretraining, in 166 hours on a single H100 (Berreby et al., 23 Mar 2026). The optimization setup uses AdamW, peak learning rate $(x_t, y_t) \in [-1,1]^2$8, 100k warmup steps, batch size 64 scenes, weight decay $(x_t, y_t) \in [-1,1]^2$9, gradient clipping at 1.0, and mixed-precision bfloat16 for most operations while keeping canvas and coordinate computations in float32. The paper highlights the scene count as an order of magnitude larger than previous active models.

The main experimental outcomes are concentrated in ADE20K semantic segmentation and ImageNet-1k classification.

Setting	Result	Context
ADE20K, single $s_t \in (0,1]$0 glimpse	38.5% mIoU	15.86 GFLOPs
ADE20K, stronger policy and larger evaluation canvas	45.9% mIoU	more glimpses
ImageNet-1k, C2F, $s_t \in (0,1]$1	81.2% top-1	frozen teacher probes

On ADE20K, a frozen CanViT-B achieves 38.5% mIoU from a single low-resolution glimpse at 15.86 GFLOPs, exceeding the previous active state of the art, AME (SETR), at 27.6% mIoU, which requires up to 309 GFLOPs. The paper also states that this single-glimpse result exceeds its FLOP-matched or input-matched DINOv3 teacher comparison. With a larger canvas and additional glimpses, CanViT-B reaches 45.9% mIoU. The paper emphasizes that these segmentation results are obtained from a model trained from scratch, without segmentation labels and without fine-tuning.

On ImageNet-1k classification, CanViT-B reaches 81.2% top-1 accuracy with frozen teacher probes under the C2F policy and $s_t \in (0,1]$2. The paper reports that accuracy increases from 76.8% to 81.2% under C2F as the number of glimpses grows. It further states that this exceeds AdaGlimpse’s 77.5% and is second only to AdaptiveNN’s 82.2% among active models, despite those baselines using end-to-end reinforcement learning and task-specific training.

The efficiency argument is a separate contribution. The paper reports 19.5× fewer inference FLOPs than the previous active segmentation state of the art on ADE20K and emphasizes linear scaling in the number of glimpses $s_t \in (0,1]$3, in contrast to MAE-style active methods whose encoder cost grows roughly quadratically with glimpse count because they re-encode accumulated patches at each step. The appendix gives analytic FLOP values of 15.9 GFLOPs for CanViT at $s_t \in (0,1]$4, 215.2 GFLOPs for DINOv3 ViT-B at $s_t \in (0,1]$5, 309.0 GFLOPs for AME at $s_t \in (0,1]$6, and 911.3 GFLOPs for AdaGlimpse at $s_t \in (0,1]$7.

6. Generalization, ablations, and interpretation

A major empirical claim is that CanViT generalizes beyond its training regime (Berreby et al., 23 Mar 2026). Although pretraining uses an average rollout length of about four glimpses, the model continues improving out to $s_t \in (0,1]$8 at test time on both ADE20K and ImageNet-1k. This indicates that the recurrent memory mechanism is not narrowly specialized to the short-horizon distribution induced by truncated BPTT.

The model also generalizes to larger scenes and larger canvas resolutions. Training is performed on $s_t \in (0,1]$9 scenes with a $H \times W$0 canvas, whereas evaluation successfully uses $H \times W$1 scenes with a $H \times W$2 canvas. Performance improves with the larger canvas, particularly at longer horizons; on ADE20K under C2F at $H \times W$3, the paper reports an increase from 44.2% with a $H \times W$4 canvas to 45.9% with a $H \times W$5 canvas. This suggests that scene-wide latent memory capacity remains a limiting factor even when the backbone and pretraining objective are fixed.

Policy transfer is another explicit evaluation axis. The paper tests policies not seen during training: C2F coarse-to-fine quadtree traversal, F2C as the reverse order, entropy-guided C2F using segmentation entropy to prioritize uncertain regions, and Repeated Full-Scene using the same zoomed-out view at every step. C2F outperforms the training-time policies at later steps, F2C is substantially worse, entropy-guided C2F gives an early-step boost, and Repeated Full-Scene shows that recurrence alone helps somewhat but that diverse views are markedly better. The contrast between C2F and F2C is particularly informative because it indicates that active vision in this framework is not permutation-invariant with respect to glimpse order.

The ablation results provide the paper’s main explanatory evidence. Reducing canvas dimension to 256 hurts strongly, and adding canvas-side QKVO projections does not recover the loss when compute is matched, supporting the asymmetric Canvas Attention design. Removing canvas reads significantly harms both spatial and CLS reconstruction, and increasing the read/write stride from 2 to 6 also degrades performance, indicating that frequent bidirectional interaction between backbone and canvas is important. Removing the dense patch-level loss causes a dramatic collapse in spatial reconstruction and also lowers CLS quality, implying that dense supervision improves not only segmentation-oriented structure but the overall internal representation. Training with one full-scene rollout plus one random rollout is better than using two random rollouts, and using $H \times W$6 instead of $H \times W$7 in truncated BPTT degrades performance, showing that even minimal temporal credit assignment contributes. Finally, shrinking the backbone from ViT-B to ViT-S causes the largest loss in CLS quality, confirming that backbone capacity remains a major determinant of non-spatial representation quality.

In aggregate, the paper’s interpretation is that active vision can be brought into a foundation-model regime by combining a compact but expressive thinking stream with a large persistent memory canvas, tied together by scene-relative geometry and efficient asymmetric interaction. A plausible implication is that CanViT reframes active vision from a narrowly policy-learning problem into a representation-learning problem in which policy can be modular, downstream, and potentially external to pretraining.