2000 character limit reached
Mechanisms of Object Localization in Vision-Language Models
Published 19 May 2026 in cs.CV | (2605.19792v1)
Abstract: Visually-grounded LLMs (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.
Summary
- The paper demonstrates that object localization in VLMs operates through token containerization, effectively predicting bounding boxes even when token order is shuffled.
- It reveals that integrating global spatial cues with local high-resolution views significantly improves classification, particularly for small objects.
- The study uses causal mediation analysis to identify narrow computational pathways in attention heads, providing key insights for optimizing future VLM architectures.
"Mechanisms of Object Localization in Vision-LLMs" Overview
Introduction to Visually-Grounded LLMs
Visually-grounded LLMs (VLMs) integrate vision and language processing capabilities through architectures that combine vision encoders with LLMs. Recent advancements have enabled VLMs to effectively tackle tasks such as visual question answering, image captioning and open-ended image reasoning. However, they still experience significant performance challenges in basic tasks such as classification and localization. The paper investigates these limitations focusing specifically on object localization mechanisms using a variety of interpretability tools such as token ablation, attention knockout, and causal mediation analysis.
Mechanism and Methodology
The research dissected two main architectures: LLaVA-1.5 and InternVL-3.5, each with distinct architectural complexities and configurations. LLaVA-1.5 uses a more straightforward setup with a CLIP ViT-L/14 visual backbone mapped into the language domain via a simple MLP adapter. InternVL-3.5, on the other hand, adopts a comprehensive architecture with token compression and dynamic high-resolution processing, allowing for fine-grained visual token representation.
An intricate suite of experiments was devised to isolate and study the visual evidence processed within these models, focusing on positional encoding, containerization of object tokens, and task-specific attention heads within the models. The experiments leveraged a specialized dataset derived from the COCO validation split with meticulously filtered annotations to ensure accuracy throughout evaluation.
Key Findings
Containerization Mechanism: Localization in VLMs appears to be governed by a mechanism where objects are containerized via token assignment, establishing spatial boundaries independent of internal arrangement. The experiments showed that VLMs predict bounding boxes effectively even when object tokens are shuffled, signifying the importance of token presence rather than semantic organization.
Spatial and Semantic Cues Integration: InternVL's global view carries significant spatial information crucial for localization, whereas the local high-resolution crops refine classification processes, particularly for small objects. This highlights the complementary nature of the two views within the model for various tasks.
Sparse, Task-Critical Attention Heads: Through causal mediation analysis, it was determined that only a few attention heads are crucially involved in task processes within the models. These specialized heads predominantly arise in early-mid layers for LLaVA and mid-late layers for InternVL, revealing narrow computational pathways for effective model operation.
Figure 1: Alignment between predicted and scaled ground-truth bounding boxes under object padding.
Implications for Model Design and Future Directions
The insights garnered from this study have profound implications on future VLM architectures. By identifying the narrow pathways and specialized attention heads responsible for task-critical operations, researchers can pursue optimized architectures, potentially involving dynamic head fine-tuning or grounding-aware attention mechanisms. Furthermore, extending the interpretability framework and methodology applied in this study to complex vision tasks, segmentation, or video processing can bolster architectural understanding and task performance.
Future research may also explore refining dataset selections and task definitions to incorporate more complicated visual scenarios or assess model responses under diverse settings. Such endeavors will undoubtedly advance both understanding and capabilities within the domain of VLMs, paving the way for models with superior task performance and interpretability.
Conclusion
This investigation conclusively revealed mechanisms by which VLMs decode, encode, and spatially ground visual data, highlighting distinctive processes pertinent to object localization. Despite notable limitations in current architectures, the reported methodologies and insights afford substantial avenues for refining VLM designs and contributing to enhanced grounding and spatial reasoning capabilities in AI models.
Figure 2: Positional decoding results showcasing higher accuracy at the image corners.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, framed to guide follow‑up research:
- Limited model coverage: conclusions are drawn from LLaVA‑1.5 (7B/13B) and InternVL‑3.5 (8B) only; it is unclear whether the “containerization” mechanism, layer localization of critical heads, and global/local view roles generalize to other VLM families (e.g., BLIP‑2/Q‑Former, Perceiver‑style encoders, cross‑attention VLMs, SAM/DINOv2 backbones, different patch sizes/encoders, MoE LLMs).
- Training origin of mechanisms: the emergence timing and stability of localization‑critical heads during pretraining vs. instruction tuning remain unknown; no longitudinal analysis of when “containerization” and head specialization appear.
- Adapter and positional encoding effects: the role of the multimodal adapter architecture (2‑layer MLP) and positional encoding choice/order (e.g., RoPE variants, absolute encodings, token interleaving order) in enabling positional reconstruction and containerization is not dissected.
- Scope of tasks: analysis is limited to single‑category presence and single‑box localization; it remains open how mechanisms extend to multi‑object detection, multiple instances of the same class, referring expressions, spatial relations, or segmentation/instance masks.
- Dataset filtering bias: the curated COCO subset excludes very small/dominant objects and images with multiple valid targets; it is unclear whether findings hold for the filtered‑out regimes (tiny/huge objects, dense scenes, heavy occlusions, clutter, truncation).
- Inpainting control artifacts: the object‑removed “base” images rely on LaMa inpainting; potential artifacts (texture/edge inconsistencies) could confound CMA and ablation results; no human verification or artifact sensitivity analysis is provided.
- Model‑dependent subset construction: selecting only image pairs where all models succeed before and fail after inpainting may bias the CMA sample toward “easy” instances; generality of mediation patterns across broader samples is unknown.
- Small CMA sample size: causal mediation is reported on 50 images, which limits statistical power; robustness across categories, sizes, and scene types is not established.
- Perplexity‑based mediation proxy: MF is computed from token‑level perplexity under teacher forcing (with template tokens masked); the degree to which MF correlates with actual localization quality (IoU) at the sample level is not validated.
- Component coverage in CMA: mediation is measured only for attention heads; the causal roles of MLP blocks, residual streams, layer norms, and adapter layers are not analyzed.
- Nature of negative mediation heads: numerous negative MF heads (especially in InternVL) are reported but not characterized; whether they encode counter‑evidence, context priors, or regularizers remains open.
- Mechanistic circuitry of containerization: the concrete computations that expand from object‑aligned tokens to spatial extents (e.g., query/key patterns, boundary/anchor heads, row/column aggregation paths) are not identified.
- Positional reconstruction mechanism: while corner anchors and “line breaks” are implicated, the exact tokens/heads that reconstruct 2D layout from the 1D sequence and how this depends on input ordering and padding are not mapped.
- Global vs. local view interplay: the division of labor is shown but not parametrically tested across tile sizes, counts (beyond a cap of 6), cropping policies, or compression strength (pixel shuffle); sensitivity curves and failure modes are missing.
- Generality across object scales and occlusions: view‑ablation analyses by object size are partial (appendix); systematic evaluation across occlusion, aspect ratio, visibility, and cross‑tile objects is not provided.
- Robustness of ablation baselines: replacing visual tokens with a global average embedding (from ImageNet) is one choice; whether conclusions hold with alternative nulls (zero vectors, Gaussian noise matched to moments, shuffled in‑batch tokens) is untested.
- Register tokens: register‑token identification via high‑norm thresholds may be brittle; alternate definitions and their contribution to spatial grounding are not explored.
- Token shuffling limits: object‑internal shuffling minimally harms localization in LLaVA but strongly affects InternVL; the causes (compression, multi‑view fusion, grid differences) and category‑specific effects are not analyzed.
- Quantization and mask‑to‑token mapping: mapping ground‑truth masks to token grids via ≥1‑pixel overlap induces boundary quantization; the sensitivity of results to the mapping rule and padding beyond ±2 tokens is unknown.
- Prompt dependence: results use specific prompts (list‑based for evaluation and binary for CMA); robustness to prompt phrasing, ordering, and instruction templates (including numeric output formats) is not tested.
- Label space constraints: classification is restricted to COCO’s fixed categories and exact name matching; behavior under open‑vocabulary synonyms or long‑tail categories remains unexamined.
- Parsing and output errors: localization depends on parsing model‑generated box strings; the impact of parsing failures vs. true reasoning errors is not quantified.
- Generalization beyond COCO: transfer of the identified mechanisms to other datasets/domains (e.g., LVIS, Objects365, indoor/medical/satellite imagery) is not evaluated.
- Stability across random seeds and checkpoints: whether the same high‑MF heads recur across seeds, training runs, or minor finetuning is untested; repeatability of head identities is unknown.
- Causality under intervention: while head ablation supports necessity, proactive interventions (e.g., targeted finetuning/pruning/regularization of high‑MF heads) are not used to establish sufficiency or to deliberately enhance localization.
- Downstream implications: whether leveraging identified heads (e.g., supervising attention, auxiliary losses) improves grounding without harming other abilities is left unexplored.
- Temporal and 3D extensions: applicability of the mechanisms to video localization, tracking, motion cues, or 3D spatial grounding is not addressed.
Practical Applications
Immediate Applications
Below are concrete, deployable uses that leverage the paper’s findings on containerization, multi-view integration, implicit spatial layout learning, and sparse task-critical heads in VLMs.
- Task-specific head pruning to cut inference cost and latency
- Sectors: software infrastructure, energy/cloud, mobile/edge, robotics
- What: Use causal mediation analysis (CMA) to identify non-causally-contributing attention heads for localization or classification and prune them for a given deployment.
- Tools/products/workflows:
- A pruning toolkit that runs MF-based head ranking and ablates near-zero-MF heads.
- CI pipeline step that re-validates accuracy post-prune on target workload.
- Assumptions/dependencies:
- Access to model weights and inference stack allowing head-level modification.
- Head-importance patterns remain consistent on the target domain and prompt style.
- Sufficient evaluation data to guard against rare-case regressions.
- Targeted fine-tuning of localization-critical heads
- Sectors: robotics, AR/VR, retail/visual search, autonomous systems
- What: Fine-tune only the small set of localization-critical heads (identified by MF) on domain-specific imagery to improve grounding while keeping overall model stable.
- Tools/products/workflows:
- Lightweight adapter or LoRA layers grafted on top of identified heads.
- “Head-aware” hyperparameter recipes focused on stability and data efficiency.
- Assumptions/dependencies:
- CMA reliably isolates localization heads for the chosen architecture.
- Access to domain-labeled images with bounding boxes or synthetic supervision.
- Sequential prompt/workflow design: “identify → then localize”
- Sectors: software, VQA products, AR UX
- What: Operationalize the paper’s finding that localization depends on classification-critical heads by structuring pipelines into two steps—first confirm object presence/class, then request box coordinates.
- Tools/products/workflows:
- Prompt templates for two-stage queries; guardrails that only attempt localization if classification is confident.
- Assumptions/dependencies:
- Multi-turn prompting supported by the VLM.
- Slight latency increase tolerated; improves reliability where mis-localization is costly.
- Dynamic tiling policies to reduce cost without hurting accuracy
- Sectors: mobile/edge, UAVs/drones, robotics, mapping/inspection
- What: In InternVL-like systems, allocate high-resolution local crops primarily when small objects are expected; rely more on the global view for spatial grounding otherwise, per the observed global/local complementarity.
- Tools/products/workflows:
- Scheduler that predicts object size and toggles tile count; budget-aware inference controller.
- Assumptions/dependencies:
- Architecture exposes global and local views or can emulate them (e.g., multi-scale processing).
- Validation on target image distributions to avoid missing small objects.
- Model debugging and QA via head-level audit
- Sectors: ML engineering, regulated industries (e.g., automotive), internal governance
- What: Use attention knockout and CMA to diagnose where classification/localization fails; surface task-critical heads and layer bands as part of QA.
- Tools/products/workflows:
- Dashboard that visualizes MF heatmaps and shows performance deltas under knockouts.
- Assumptions/dependencies:
- Compute budget to run patching/knockout experiments on representative samples.
- Internal access to intermediate activations.
- Hallucination control and evaluation with object-removed control sets
- Sectors: academia, data-centric ML, safety/audit teams
- What: Adopt the paper’s object-inpainting control to ensure grounding relies on true object evidence rather than context; integrate into validation suites.
- Tools/products/workflows:
- Data curation pipeline using inpainting (e.g., LaMa) to generate object-removed pairs.
- Automated checks that models fail appropriately on inpainted controls.
- Assumptions/dependencies:
- Inpainting does not leak obvious artifacts for the given domain.
- Legal/ethical review for manipulating content in sensitive datasets.
- Assisted annotation tools that snap to “containerized” objects
- Sectors: data labeling platforms, CV dataset providers
- What: Exploit the containerization behavior by using coarse user cues (e.g., point/click or rough lasso) to collect object-aligned tokens and propose boxes that match the token “container.”
- Tools/products/workflows:
- Interactive bounding-box tool that maps image regions to token grids and returns boxes consistent with token extent.
- Assumptions/dependencies:
- Access to the model’s token-grid mapping and positional decoder.
- Smooth UX latency and acceptable proposal quality for human-in-the-loop correction.
- Robustness checks using containerization-aware perturbations
- Sectors: security auditing, QA
- What: Add tests that shuffle tokens inside object masks or manipulate padding around masks to probe over-reliance on container signals.
- Tools/products/workflows:
- Automated perturbation suite to evaluate sensitivity of localization vs classification.
- Assumptions/dependencies:
- Ability to intercept and modify visual tokens at the LLM input.
- Clear acceptance thresholds for robustness metrics.
- Cost- and energy-aware token compression strategy
- Sectors: energy/cloud, mobile/edge
- What: Emphasize global tokens for localization and apply pixel-shuffle or similar compression; reserve high-res tokens to cases that require classification detail.
- Tools/products/workflows:
- Token budget allocator; profile-based policies tuned per application SLA.
- Assumptions/dependencies:
- Architecture supports compression/aggregation without large accuracy loss.
- Target tasks tolerate occasional detail loss on large-object scenes.
- Improved VLM benchmarks for spatial grounding
- Sectors: academia, standards bodies, platform benchmarking
- What: Integrate IoU-threshold success rates, object-removed controls, and positional decoding probes as standard metrics for “grounding” evaluations.
- Tools/products/workflows:
- Benchmark suite with scripts for attention/positional probes and curated test splits.
- Assumptions/dependencies:
- Community alignment on test protocols and licensing for datasets.
- Educational and training modules on mechanistic VLM interpretability
- Sectors: education, internal upskilling
- What: Use the paper’s analysis (token ablations, CMA, attention knockout) as lab exercises to teach interpretability and multimodal grounding.
- Tools/products/workflows:
- Course notebooks linked to the authors’ code repository; reusable didactic datasets.
- Assumptions/dependencies:
- Students/teams have access to GPUs for small batches of CMA experiments.
Long-Term Applications
These opportunities require further research, scaling, or engineering investment before deployment.
- Architectures with dedicated “container heads” and grounding-aware objectives
- Sectors: AI model development, robotics, AR/VR
- What: Design modules or losses that explicitly encourage container-like spatial grouping and strengthen corner/row-structure anchoring; supervise attention to object boundaries.
- Tools/products/workflows:
- Grounding-aware attention supervision; container-token objectives; enhanced positional anchors.
- Assumptions/dependencies:
- Demonstrated generalization across datasets and tasks (detection, segmentation).
- Training data with reliable spatial labels; stability of specialized heads during scale-up.
- General-purpose grounded VLMs that replace classical detectors in pipelines
- Sectors: retail (shelf analytics), logistics, media search, smart cameras
- What: Build VLMs that robustly localize and classify with detector-level precision using few specialized heads for efficiency.
- Tools/products/workflows:
- Unified vision-language detectors; APIs that return boxes and natural-language rationales.
- Assumptions/dependencies:
- Matching or exceeding accuracy/latency of dedicated detectors in production scenarios.
- Robustness to scale, occlusion, and domain shift.
- Runtime head gating and early-exit controllers
- Sectors: edge inference, energy-efficient AI, specialized hardware
- What: Gate or skip non-critical heads/layers adaptively at inference time based on a confidence controller, leveraging head sparsity and sequential processing (identify → localize).
- Tools/products/workflows:
- Confidence estimators tied to MF-ranked head groups; hardware schedulers for dynamic execution.
- Assumptions/dependencies:
- Reliable uncertainty estimates; low overhead for gating logic.
- Hardware/runtime support for dynamic graph execution.
- Weakly supervised detection/segmentation from image-level labels
- Sectors: data-scarce domains, long-tail categories
- What: Use containerization to convert coarse supervision (class labels, points) into pseudo boxes/masks for training detectors with limited annotation.
- Tools/products/workflows:
- Pseudo-label generation that aggregates object-aligned tokens into boxes/masks; iterative self-training.
- Assumptions/dependencies:
- Containerization holds under weak supervision; noise tolerant training procedures.
- Validation on challenging categories and cluttered scenes.
- Cross-domain grounded VLMs (e.g., remote sensing, industrial inspection, medical imaging)
- Sectors: aerospace, manufacturing, healthcare
- What: Adapt mechanisms (containerization, sparse head specialization) to modalities with different textures/scales; validate “global-for-localization, local-for-classification” in new domains.
- Tools/products/workflows:
- Domain-specific adapters; MF-guided fine-tuning; multi-scale backbones.
- Assumptions/dependencies:
- Availability of curated domain data; regulatory approval in sensitive areas (e.g., healthcare).
- Potential need for different positional anchoring strategies.
- Video grounding and tracking with sequential containerization
- Sectors: surveillance, sports analytics, autonomous navigation
- What: Extend container heads to spatio-temporal tokens; maintain object “containers” over time for tracking and action understanding.
- Tools/products/workflows:
- Temporal CMA; cross-frame attention supervision; memory modules that preserve containers.
- Assumptions/dependencies:
- Stable temporal anchoring and robustness to motion/occlusion.
- Scalable training with long sequences.
- Certifiable interpretability and audit standards for spatial grounding
- Sectors: policy/regulation, public-sector procurement, safety-critical systems
- What: Develop standards that require head-level mediation audits and object-removed controls to demonstrate that localization is visually grounded.
- Tools/products/workflows:
- Compliance test harnesses; reports summarizing MF distributions and knockout sensitivity.
- Assumptions/dependencies:
- Regulator and industry consensus; reproducibility across implementations.
- Hardware co-design targeting sparse, specialized heads
- Sectors: semiconductor, embedded AI
- What: Architect accelerators optimized for a small set of high-utility attention heads and dynamic tiling/token budgets.
- Tools/products/workflows:
- Compiler support for head-level scheduling; memory layouts aligned with grid tokens.
- Assumptions/dependencies:
- Stable head sparsity patterns at scale; economic viability of specialized silicon.
- Tooling ecosystems for MF-driven model surgery
- Sectors: MLOps, model hosting platforms
- What: Provide managed services that run CMA, recommend head pruning/fine-tuning, and auto-generate optimized variants for different tasks and budgets.
- Tools/products/workflows:
- “One-click” analysis and export; continuous monitoring of MF drift in production.
- Assumptions/dependencies:
- Privacy/security controls for model introspection; scalable patching infrastructure.
- Human-centered AR/UX that leverages container snapping and dynamic resolution
- Sectors: consumer devices, creative tools
- What: Build UIs that let users loosely select objects and have the system snap to containerized boundaries; dynamically increase resolution only where needed.
- Tools/products/workflows:
- On-device VLM with token-grid visualization; latency-aware tiling scheduler.
- Assumptions/dependencies:
- Efficient on-device models; robust mapping from tokens to pixel coordinates; acceptable battery impact.
These applications translate the paper’s mechanistic insights into concrete steps for building faster, more reliable, and more transparent VLM-based systems across sectors. Each item’s feasibility depends on access to internal model components, the stability of identified head roles across domains, and careful validation on target use cases.
Glossary
- Activation patching: An interpretability intervention that replaces activations from one run into another to test causal contributions of components. "We apply CMA using activation patching to identify which attention heads causally contribute to solving the visual task."
- Attention head: An individual attention mechanism within a transformer layer that processes a subset of the attention computation. "Only a very small set of attention heads mediates the causal effect for both classification and localization,"
- Attention knockout: A method that zeroes or blocks specific attention pathways to test their necessity for a task. "We apply the attention knockout technique"
- Causal mediation analysis: A technique to estimate the causal contribution of internal components by comparing counterfactuals under patched activations. "including token ablations, attention knockout, and causal mediation analysis."
- Containerization: A proposed mechanism where object-aligned tokens collectively define an object’s spatial extent regardless of internal token arrangement. "localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object,"
- Contrastively pre-trained: Trained with a contrastive objective to bring matched pairs closer and mismatched pairs farther in embedding space. "uses a custom, contrastively pre-trained InternViT-300M backbone"
- Corner anchors: Salient positional cues near image corners that help reconstruct grid layout in sequence models. "strong corner anchors are sufficient for the model to reconstruct approximate row boundaries and a grid-like layout."
- Counterfactual output: The model output obtained after patching activations from a different condition, used to test causal effects. "yielding the counterfactual output ."
- Dynamic High-Resolution Processing: An architectural strategy that adaptively tiles inputs at high resolution to capture fine details. "Dynamic High-Resolution Processing: Input images are split into a variable number of px tiles that are processed independently by the visual backbone."
- Global view: A downsampled, whole-image representation providing coarse context in multi-view architectures. "We refer to the high-resolution tiles as local views and the thumbnail as the global view."
- Head ablation: Removing or zeroing the outputs of selected attention heads to assess their importance. "Localization accuracy under cumulative head ablation."
- Integrated Gradients: An attribution method that integrates gradients along a path from a baseline to the input to measure feature importance. "by computing Integrated Gradients \cite{sundararajan2017axiomatic} with respect to the correct class logits (for classification) or bounding box coordinates (for localization)."
- Intersection-over-Union (IoU): A metric measuring overlap between predicted and ground-truth regions, defined as the area of intersection divided by the area of union. "using the intersection-over-union (IoU) metric."
- Inpainting: Filling in missing image regions to remove objects while maintaining realistic backgrounds. "the missing region is inpainted using LaMa"
- Local view: High-resolution crops that capture fine-grained details around regions of interest in multi-view models. "We refer to the high-resolution tiles as local views and the thumbnail as the global view."
- Mechanistic interpretability: A set of techniques aimed at explaining model computations by analyzing internal components and circuits. "using a suite of mechanistic interpretability tools"
- Mediation Fraction (MF): A scalar measuring how much of the performance gap between conditions is closed by patching a component. "We quantify the causal contribution of a component using the Mediation Fraction (MF):"
- Multimodal adapter: A module that maps visual features into the language embedding space for joint processing. "a multimodal adapter maps them into the language embedding space"
- Multimodal projection: The transformation that projects visual token embeddings into the text/LLM space before further processing. "including the multimodal projection, to predict the position of every image token in the input grid."
- Object mask: A binary map indicating which image regions belong to a target object. "We project the object mask onto the image token grid"
- Perplexity: A token-level uncertainty measure of a model’s predictions, often used to evaluate language-model outputs. "All outputs are evaluated under teacher-forcing using token-level perplexity."
- Pixel Shuffle: A compression step merging neighboring token patches into a single token while preserving local structure. "Pixel Shuffle: Each block of visual tokens from the backbone is merged into a single token before projection into the text space using a learned compression."
- Positional identifiability: The degree to which a token’s original spatial position can be decoded from internal representations. "positional identifiability is initially low."
- Register tokens: Special tokens hypothesized to store global image features within transformer-based vision models. "Global image features are hypothesized to be encoded in register tokens"
- Residual positional signals: Remaining positional cues embedded in representations after projection, used by the LLM to infer spatial layout. "residual positional signals at the multimodal projection"
- Teacher-forcing: Evaluating or training sequence models by providing ground-truth previous tokens to condition predictions. "All outputs are evaluated under teacher-forcing using token-level perplexity."
- Token ablation: Removing or replacing selected input tokens to test their contribution to performance. "including token ablations, attention knockout, and causal mediation analysis."
- Token compression: Reducing the number of tokens (e.g., by merging patches) to lower computational cost while retaining structure. "a state-of-the-art variant incorporating token compression and multi-view processing."
- Token shuffling perturbations: Randomly permuting token positions to test sensitivity of spatial or semantic processing. "we shuffle the image tokens within the object mask directly at the LLM input."
- Vision Transformer (ViT): A transformer-based architecture operating on image patches treated as token sequences. "We study vision–LLMs that follow the ViT â MLP â LLM paradigm"
- Vision–language instruction tuning: Fine-tuning with multimodal instructions to align vision encoders and LLMs for downstream tasks. "typically refined through vision-language instruction tuning."
- Visually-grounded LLMs (VLMs): Models that process and align visual and textual inputs jointly for multimodal tasks. "Visually-grounded LLMs (VLMs) are highly effective in linking visual and textual information,"
Collections
Sign up for free to add this paper to one or more collections.