Stateful Visual Encoders for Vision-Language Models

Abstract: Vision-LLMs (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes. However, in existing open-weight VLMs, visual comparisons happen only inside the LLM, while the visual encoder itself remains stateless: each image is encoded independently, without access to the prior visual context. As a result, small but task-critical changes may be attenuated before the LLM has a chance to compare them, especially when those changes do not affect the high-level semantics of the scene. We introduce a Stateful Visual Encoder, which conditions each visual representation on prior visual features. Under supervised finetuning, VLMs equipped with stateful encoders achieve consistent improvements on controlled tasks involving cross-image spatial aggregation, multi-object visual differencing, and visual trajectory behavior cloning. These improvements are consistent across input resolutions, LLM sizes, and VLM backbones. Finally, we validate our model on real-world tasks, including longitudinal radiology, fine-grained image comparison, and remote sensing, where stateful encoders consistently improve generalist VLM baselines and can match or surpass specialized models in selected domains. Project page: https://statefulvisualencoders.github.io/

• The paper presents a novel Stateful Visual Encoder (SVE) that integrates cross-image context using per-layer cross-attention combined with an FFN, enabling effective temporal conditioning.
• It demonstrates significant performance improvements across spatial estimation, multi-object differencing, and behavioral cloning tasks, reducing errors and enhancing accuracy metrics.
• The plug-and-play SVE easily augments pretrained ViT backbones without retraining, proving robust across diverse resolutions, architectures, and real-world applications.

Stateful Visual Encoders for Multi-Image Vision-LLMs

Introduction and Motivation

Traditional vision-LLMs (VLMs) operate with stateless visual encoders: each image passes independently through a visual backbone, producing per-image visual tokens that are later serialized and compared only at the level of the LLM. This late-fusion paradigm limits the ability to capture subtle, task-critical cross-image changes, particularly when visual differences are small and do not significantly alter global scene semantics. The stateless assumption is particularly restrictive for agentic settings, longitudinal analysis, and structured multi-image reasoning, where decisions critically depend on changes over time or across views.

The paper "Stateful Visual Encoders for Vision-LLMs" (2606.04433) addresses this limitation by introducing the Stateful Visual Encoder (SVE), a minimally invasive architectural extension that enables cross-image conditioning inside the visual encoder prior to serialization to the LLM. SVEs condition the current visual representation on features extracted from previous images, aligning the encoder with real agent-environment interaction patterns in which only the past is available for context (Figure 1).

Figure 1: The left-to-right direction ensures that the current image can attend only to past visual features, matching interactions where future observations may not yet be available.

SVE Architecture and Design

The authors systematically explore several methods for integrating context from previous images into the visual encoder. The approaches include:

• Self-Ext: Extending the self-attention layer's key-value set with features from the previous image.
• AdaLN-Zero: Aggregating features from the past via pooling to modulate the encoder using adaptive layer normalization.
• Cross: Inserting token-level cross-attention layers at each block, allowing the current visual features to query those from the previous timestep.
• Cross+FFN: Enhancing the Cross design by supplementing with an additional feedforward network after cross-attention.

Crucially, the Cross+FFN design — involving a per-layer cross-attention followed by an FFN, with stop-gradient and careful initialization from the pretrained self-attention weights (Figure 2) — was found to yield the most consistent and robust performance improvements.

Figure 2: Several techniques to condition current visual tokens $Z_t$ on past tokens $Z_{t-1}$, highlighting the recipe for Cross+FFN including stop-grad, cloned initialization, and zero-init of output projections.

This approach maintains compatibility with existing pretrained VLMs: modifications can be applied to any family of ViT-based visual backbones without the need for retraining from scratch, preserving the original model's learned representations and only augmenting them with minimal, zero-initialized layers.

Benchmarking on Controlled Multi-Image Tasks

Evaluation of SVEs spans both synthetic and real-world tasks, systematically designed to probe spatial, categorical, and behavioral reasoning across images.

Task Families

• Cross-image Spatial Aggregation: Requires the model to integrate fine-grained spatial cues across multiple frames to estimate geometric configurations such as Euclidean distances and convex-hull areas.
• Multi-object Visual Differencing: Based on synthetic CLEVR scenes with 30–40 objects, with the model asked to localize and describe subtle changes (additions, deletions, replacements).
• Visual Trajectory Behavioral Cloning: Behavioral imitation tasks where the model must track agent state across interleaved interaction steps (e.g., VisGym tasks involving patch reassembly and 3D rotations).

  Figure 3: Illustration of the three task categories: cross-image spatial aggregation, multi-object differencing, and visual trajectory behavioral cloning.

Main Numerical Findings

The Cross+FFN SVE design delivers substantial improvements over stateless baselines:

• Spatial Aggregation: Reduces MAE and RMSE by more than 40% across geometric estimation tasks, with the strongest gains on fine spatial localization.
• Visual Differencing: On CLEVR-Multi-Change, increases change detection accuracy from 91.1% to 92.7%, and CIDEr from 529.5 to 543.9.
• Behavioral Cloning: Reduces perplexity across all VisGym sub-tasks, outperforming baselines, particularly in tasks requiring cross-image memory and state tracking.

No variant that does not access prior visual context (including capacity-matched controls) matches the performance of Cross+FFN. Simple pooling or expanding attention masks fails to provide comparable gains or even degrades performance, substantiating the need for explicit, token-level cross-image interaction.

Ablation and Recipe Analysis

The study conducts ablations to identify crucial initialization and architectural aspects:

• Removing output zero-initialization causes catastrophic degradation, confirming its importance for stable feature propagation.
• Omitting stop-gradient destabilizes the update of the cross-image retrieval path and harms generalization.
• Using pooled (overly compressed) historical features rather than token-level retrieval blunts the model's ability to surface fine-grained changes.
• The presence and preservation of positional embeddings in the cross-attention pathway is critical for localizing spatially structured changes.

These findings highlight the sensitivity of stateful encoding to both initialization policy and the architecture of cross-image connections.

Generalization Across Scale and Backbones

SVEs demonstrate robustness across image resolutions ($256^2$ to $768^2$), LLM sizes (0.8B–9B), and diverse VLM architectures (Qwen3-VL, Qwen3.5, GLM-4.6V-Flash, InternVL3.5, Gemma-3).

Figure 4: SVE improves multi-object visual differencing across input resolutions (top, varying image scale) and model sizes (bottom), especially when baselines are weaker.

Notably, smaller SVE-equipped models can rival or outperform much larger stateless models, and gains persist across connector designs, backbone families, and attention variants. This suggests that stateful visual encoding orthogonally complements backbone scaling.

Analysis of Learned Visual Representations

Feature analysis reveals several properties unique to SVEs:

• Context Dependence: SVE feature maps are sensitive to the identity of the previous image; swapping predecessors induces substantial representational changes (Figure 5a).
• Task-Relevant Change Localization: In cases where SVE and baseline predictions disagree, SVE delivers superior change localization and captioning (Figure 5b).
• Spatial Sparsity: Cross-image updates induced by SVE are spatially sparse, concentrated at the sites of actual visual differences, preserving compactness of the representation (Figure 5c).

  Figure 5: (a) SVE features are contextually adaptive while stateless baselines are invariant. (b) SVE decisively outperforms on cases where the models disagree. (c) Cross-image feature changes via SVE are spatially targeted.

Validation on Real-World Applications

Longitudinal Radiology

On Medical-Diff-VQA, a benchmark for temporal change annotation in chest X-rays, SVE enhances all captioning metrics and clinically meaningful finding-level F1 scores over a stateless VLM:

• Micro F1: from 31.55 → 32.20
• Change Accuracy: from 86.83% → 89.21%
• CIDEr: from 145.1 → 178.9

Fine-Grained Image Comparison

On the ImgEdit benchmark, SVE wins pairwise preferences against both stateless baselines and reference instructions in model-judge evaluations. This demonstrates improved capability for edit verification and semantic difference reasoning in open-world web images.

Remote Sensing Change Captioning

On LEVIR-CC, SVE outperforms all evaluated specialist architectures and substantially strengthens generalist VLM ability for describing dataset-localized changes:

• $S_m^*$ (multi-metric average): 80.46, exceeding all reported specialist and baseline models

  Figure 6: Qualitative samples from longitudinal radiology, fine-grained image edits, and remote sensing; green text for correct change descriptions and red for errors.

Practical and Theoretical Implications

SVEs reimagine the VLM visual encoder as a locally stateful component, directly confronting the bottleneck posed by stateless encoders for multi-image reasoning. The demonstrated improvements across agentic, medical, and geospatial tasks indicate that fine-grained temporal context — even with lightweight, local cross-attention — can materially expand the operational frontier of out-of-the-box VLM deployment.

Theoretically, the results underscore the representational gap imposed by conventional late-fusion and encourage further exploration of interface points between vision and language modalities. Future work may examine scalable sparse memory mechanisms, attention beyond only the immediate predecessor, efficient state caching for very long visual histories, and applications in broader multi-modal or embodied domains (e.g., robotics, persistent agents).

Conclusion

Stateful Visual Encoders provide an effective, plug-and-play mechanism for enabling token-level cross-image context inside any ViT-based visual encoder attached to a VLM. With carefully crafted architectural and initialization strategies, SVEs yield significant improvements on synthetic and real-world visual comparison tasks, generalize across model scales and backbones, and introduce minimal disturbance to established multimodal pipelines. These results motivate ongoing reassessment of visual encoding paradigms in agentic VLM systems and open new directions for research into stateful, context-aware perception in foundation models.