The Hidden Evolution of Disguised Visual Context inside the VLM

Abstract: Visual tokens enter LLMs as raw, foreign signals. How they are transformed into meaningful representations and interact with the language space depends entirely on the integration architecture. Whether by treating visual tokens as in-context prompts within the input sequence or injecting them directly into the LLM's intermediate layers. A controlled comparison and understanding of how these architectural choices affect visual information and its internal transformation to integrate with the LLM remains underexplored. We provide a fair comparison by evaluating in-context and layer-wise injection VLM integration paradigms under identical training conditions across single image, multi-image, and video benchmarks. In doing so, we uncover a hidden evolution where visual tokens enter the LLM as disguised visual context, raw representations lacking linguistic structure, but are progressively reshaped depending on the integration paradigm, each capturing fundamentally different frequency characteristics of the visual signal. We show that this evolution inside the LLM determines what visual features the VLM can utilize effectively, how visual representations align with the language space, and ultimately how each paradigm performs across different tasks. We further demonstrate that attention allocation alone is insufficient, and that performance is driven by the quality of visual representations at each layer.

• The paper demonstrates that the IN-CT paradigm outperforms layerwise methods by ensuring continuous, progressive evolution of visual tokens across LLM layers.
• The study employs techniques like CKA, Fourier log amplitude analysis, and PCA to reveal key differences in frequency capture and modality alignment between integration schemes.
• Hybrid models combining IN-CT and LW-AT achieve superior performance by leveraging complementary high- and low-frequency representations, highlighting design tradeoffs in VLM architectures.

Layerwise vs In-Context Visual Integration in VLMs: Representational Dynamics and Task Implications

Integration Paradigms and Architectural Comparison

The architectural design of vision-LLMs (VLMs) fundamentally impacts how visual context is processed and utilized throughout an LLM backbone. This work systematically evaluates three primary paradigms under tightly controlled training conditions: in-context injection (IN-CT), layerwise gated cross-attention (LW-GC), and layerwise attention-only injection (LW-AT). IN-CT concatenates visual tokens with text tokens at the input, allowing both modalities to participate in standard self-attention across all layers. LW-GC and LW-AT, in contrast, inject visual features at intermediate layers—either via gated cross-attention blocks or direct attention-only mechanisms into keys and values, respectively.

Figure 1: Overview of the three VLM integration paradigms: (a) IN-CT concatenation, (b) LW-GC gated cross-attention, and (c) LW-AT attention-wise injection.

All approaches employ identical vision encoders and connectors to ensure that observed differences arise exclusively due to integration mechanisms. Extensive training across image, multi-image, and video domains is used to evaluate architectural impact in both general visual and specialized tasks.

Benchmark Outcomes Across Image, Multi-Image, and Video Tasks

In single-image understanding tasks, IN-CT dominates across general, knowledge, vision-centric, and OCR benchmarks, followed by LW-AT; LW-GC is decisively outperformed—especially on OCR and chart-based tasks demanding local spatial composition. These trends persist and even accentuate when training datasets emphasize text-heavy visual inputs. In multi-image and video benchmarks, the superiority of IN-CT remains consistent, especially in contexts requiring temporal or spatial integration across distributed tokens.

Crucially, performance gaps cannot be attributed to token budget or model scale confounds, as recipe, vision encoder, and optimization schedules are standardized. The results demonstrate that integration is not a superficial issue of insertion point but determines the representational accessibility of visual evidence throughout the LLM pipeline.

Representational Dynamics: Semantics, Frequency, and Modality Alignment

Continuous Transformation vs. Discontinuity

Token-level Centered Kernel Alignment (CKA) analysis reveals paradigm-dependent representational evolution. IN-CT supports progressive, continuous transformation of visual tokens through successive LLM layers—mirroring language tokens—whereas LW-GC and LW-AT display severe layerwise discontinuities; visual representations are static and independently projected at each layer with no residual semantic refinement.

Figure 2: CKA heatmaps reveal that only IN-CT enables smooth visual token evolution across layers; LW-GC and LW-AT produce discontinuous representations.

Frequency Spectrum Capture

Fourier-based log amplitude analysis illustrates that IN-CT progressively shifts representations toward high-frequency features in intermediate layers (capturing fine-grained local spatial detail), then consolidates to low-frequency abstractions in final layers. LW-GC and LW-AT lack coherent frequency progression, remaining biased toward low-frequency information and fluctuating erratically.

Figure 3: Relative log amplitude of Fourier-transformed visual token representations: only IN-CT captures structured high-frequency dynamics.

Modality Gap and Representational Alignment

3D PCA projections demonstrate that only IN-CT enables visual tokens to converge toward the language representation subspace in final layers, closing the modality gap. LW-AT and LW-GC maintain strict orthogonality between image and text tokens, never achieving cross-modal representational harmonization.

Figure 4: 3D PCA shows that IN-CT visual tokens merge with text token space; LW-AT keeps visual and linguistic representations orthogonal.

Attention Utilization: Allocation vs. Quality

Attention mass allocation analysis, across both general and OCR/Chart tasks, shows that visual token attention generally reduces in deeper layers, except in tasks requiring persistent visual evidence (e.g., ChartQA). However, attention allocation patterns are similar for IN-CT and LW-AT, even as IN-CT outperforms LW-AT. This underscores that performance differentials are not explained by attention mass alone, but rather by the semantic quality of visual representations at each layer—specifically, their frequency characteristics and alignment with the language space.

Figure 5: Attention mass allocated to visual tokens exhibits pronounced task-dependence; IN-CT and LW-AT mirror each other, but only IN-CT achieves superior downstream quality.

LW-GC relies on fixed gate values, limiting dynamic visual contribution. The underlying mechanism suggests that dynamic guided injection (e.g., using conditionally adaptive gating) could improve LW-GC but would not overcome its frequency spectrum limitations.

Hybrid Integration and Ablation Studies

Hybrid models co-integrating IN-CT and LW-AT (i.e., concatenated input visual tokens and layerwise attention-wise injection) achieve superior performance on both single and multi-image/video tasks, outperforming either paradigm in isolation. This supports the claim that IN-CT and layerwise injection capture complementary frequency information; providing both high- and low-frequency access boosts task performance. However, token count and computational cost constraints limit practical deployment, motivating future architectural exploration for efficient frequency-compositional fusion.

Figure 6: Hybrid IN-CT + LW-AT architecture integrates complementary visual token access across input and intermediate layers.

Architectural Implications and Design Recommendations

The evidence is robust: integration paradigm fundamentally shapes representational continuity, frequency access, and cross-modal alignment in VLMs. IN-CT provides the only mechanism capable of progressive refinement, high-frequency capture, and linguistic-space convergence—requirements for tasks necessitating distributed spatial or temporal composition (e.g., OCR, video). Layerwise injection variants offer efficiency but are intrinsically limited in frequency coverage and modality alignment.

Designing future VLMs should not be reduced to benchmark-driven trial and error. Instead, architectural choice should be guided by mechanistic representational needs of target tasks: for evidence-heavy composition and cross-modal reasoning, IN-CT or hybrid schemes are essential; for tasks where efficiency dominates and low-frequency representations suffice, layerwise injection remains viable. Efficient hybridization may provide optimal tradeoffs. The framework and methodology are extensible to other domains (audio, sensor streams), where analogous dynamics may underlie cross-modal integration.

Conclusion

This analysis systematically isolates visual integration paradigms as a key variable in VLM design, demonstrating through controlled benchmarks, CKA, frequency, PCA, and attention analyses that IN-CT enables a 'hidden evolution' of disguised visual context critical for effective multimodal reasoning. Layerwise injection approaches, despite their efficiency, fail to replicate this progression and face modality gap constraints. Future VLM and MLLM designs should prioritize integration strategies that support representational refinement and frequency versatility, especially as visual-text reasoning scenarios expand in complexity and scale.

Figures Supporting Analysis

Figure 7: LW-AT positional ID scheme and architecture establishes layerwise injection without FFN interaction.

Figure 8: Token sequence and position ID assignment for IN-CT, LW-AT, and hybrid IN-CT + LW-AT.

Figure 9: CKA heatmaps for ChartQA confirm IN-CT's smooth evolution; LW-GC and LW-AT remain discontinuous.

Figure 10: Attention mass allocation on additional datasets confirms task-dependent visual utilization patterns.

Figure 11: IN-CT: visual tokens progressively merge with text-space.

Figure 12: LW-AT: visual and text spaces stay strictly orthogonal.

Figure 13: LW-GC: lack of transformation, visual and text spaces are distinct throughout.

See "[The Hidden Evolution of Disguised Visual Context inside the VLM]" (2606.20077) for full methodology, ablations, and supplementary analyses.