From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks

Abstract: Visual in-context learning models are designed to adapt to new tasks by leveraging a set of example input-output pairs, enabling rapid generalization without task-specific fine-tuning. However, these models operate in a fundamentally static paradigm: while they can adapt to new tasks, they lack any mechanism to incorporate user-provided guidance signals such as scribbles, clicks, or bounding boxes to steer or refine the prediction process. This limitation is particularly restrictive in real-world applications, where users want to actively guide model predictions, e.g., by highlighting the target object for segmentation, indicating a region which should be visually altered, or isolating a specific person in a complex scene to run targeted pose estimation. In this work, we propose a simple method to transform static visual in-context learners, particularly the DeLVM approach, into highly controllable, user-driven systems, i.e., Interactive DeLVM, enabling seamless interaction through natural visual cues such as scribbles, clicks, or drawing boxes. Specifically, by encoding interactions directly into the example input-output pairs, we keep the philosophy of visual in-context learning intact: enabling users to prompt models with unseen interactions without fine-tuning and empowering them to dynamically steer model predictions with personalized interactions. Our experiments demonstrate that SOTA visual in-context learning models fail to effectively leverage interaction cues, often ignoring user guidance entirely. In contrast, our method excels in controllable, user-guided scenarios, achieving improvements of $+7.95%$ IoU for interactive segmentation, $+2.46$ PSNR for directed super-resolution, and $-3.14%$ LPIPS for interactive object removal. With this, our work bridges the gap between rigid static task adaptation and fluid interactivity for user-centric visual in-context learning.

• The paper’s main contribution is i-DeLVM, a method that converts static visual in-context models into interactive systems by integrating diverse user cues.
• It employs a unified interaction encoding strategy and full masking in autoregressive prediction to enhance performance across tasks like segmentation, super-resolution, and pose estimation.
• Quantitative results, including substantial gains in IoU, SSIM, and PSNR, demonstrate the model’s effective adaptation to unseen interaction types.

Expert Review: Adapting Visual In-Context Learners for User-Driven Interaction

Motivation and Context

The proliferation of visual in-context learning (ICL) models in computer vision has enabled rapid adaptation to new tasks via few-shot prompting without task-specific fine-tuning. However, most state-of-the-art approaches remain fundamentally static, unable to incorporate user-provided interaction cues such as scribbles, clicks, or bounding boxes to dynamically steer predictions. This incapacity restricts their practical deployment in tasks ranging from interactive segmentation to object-directed editing and poses estimation, where user guidance is critical. "From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks" (2604.06748) systematically addresses this gap, proposing an efficient, generic method to convert static sequence-predictive ICL models—specifically DeLVM—into user-adaptive, interactive systems through an innovative interaction encoding and training protocol.

Methodology

The proposed Interactive DeLVM (i-DeLVM) augments visual in-context learners with seamless user guidance capabilities by blending interaction signals directly into the context and query images. The encoding strategy homogenizes diverse interaction types (boxes, ellipses, scribbles, clicks, positive/negative clicks) as colored overlays in the image input, circumventing the need for interaction-specific prompt embeddings. By blending these cues with image content (typically fully overwriting interaction regions), the sequence tokenization and model context remain consistent, enabling generalization to unseen interaction types at test-time.

The training procedure leverages autoregressive token prediction conditioned exclusively on the context set (fully masking previously generated output tokens), maximizing the model's reliance on user-provided cues. Fine-tuning is performed via LoRA on a pre-trained DeLVM transformer (300M parameters), with specific context windows and batch sizes optimized for computational feasibility. To evaluate generalization, a hold-one-interaction-out scheme is employed: i-DeLVM Unseen models are trained on all interaction types except the one tested, directly probing adaptation to previously unseen user cues.

Quantitative Evaluation

Extensive benchmarks on four diverse tasks—interactive segmentation (PASCAL VOC), object removal (RORD), pose estimation (MPII), and super-resolution (ADE20K)—demonstrate strong numerical results:

• Interactive Segmentation: i-DeLVM Unseen achieves up to +14.64% IoU improvement over static DeLVM and LVM baselines for unseen interaction cues, indicating genuine context-sensitive adaptation. Notably, state-of-the-art ICL models exhibit negligible responsiveness to interaction cues, often performing only slightly above naive copying baselines, signifying a hard failure to capture interactivity.
• Directed Super-Resolution: While perceptual metrics (LPIPS) remain challenging, structural metrics are significantly bolstered, with SSIM and PSNR improvements of +2.46 and up to +3.86 for i-DeLVM Unseen over generalist models, confirming enhanced modeling of user-guided regions.
• Interactive Object Removal: Consistently lower LPIPS and higher SSIM/PSNR than static model baselines, with -3.14% LPIPS improvement, indicating more natural, context-aware inpainting when following user cues.
• Human Pose Estimation: Qualitative analysis reveals that only i-DeLVM variants reliably isolate and reconstruct the pose of the user-identified target, whereas static ICL variants either ignore cues or hallucinate irrelevant poses.

The ablation studies further validate that full masking during prediction (conditioning exclusively on context) leads to higher token accuracy (40.3% vs. 37.86% deficit for partial masking), and that recoloring augmentation of segmentation masks significantly increases IoU and reduces token perplexity, augmenting context-sensitive learning.

Qualitative Analysis and Interaction Encoding

Blending interaction overlays into image inputs introduces a subtle trade-off: while sparse cues (e.g., clicks) minimally degrade image reconstruction quality, spatially expansive cues (scribbles, boxes, ellipses) incur modest reductions in SSIM and PSNR. Empirical reconstruction demonstrates that interaction signals are effectively preserved in token space for cues above a threshold size/thickness, but ultra-sparse signals (single-pixel clicks) may be lost. The unified image-based representation enables flexible adaptation to new cue types (e.g., heatmaps, gaze traces) without prompt embedding retraining.

Practical and Theoretical Implications

The proposed generic interaction encoding and context-driven training strategy substantially upgrades visual ICL models' utility in user-centric workflows. By facilitating adaptation to arbitrary, unseen guidance modalities, i-DeLVM allows domain experts—image editors, clinicians, annotators—to express task intent via natural visual cues, unlocking personalized and iterative model control. The model's composable architecture and token-space consistency suggest scalability to even more complex interactions and multimodal prompts (e.g., mixed visual and textual guidance).

On the theoretical plane, this research interrogates the limits of sequence-predictive vision transformers' contextual generalization and reveals inherent shortcomings in current ICL pretraining approaches. The inability of static models to utilize interaction cues, even when present, underscores a barren regime in their objective formulation, necessitating architectures and training objectives explicitly sensitive to user-driven context.

Future Directions

Integrating the generic interaction encoding protocol into generative diffusion models may yield substantial gains in high-fidelity output and iterative refinement. Addressing tokenization bottlenecks—especially for sparse and fine-grained cues—will be essential for improved quality in generative tasks like super-resolution and detailed segmentation. Further, exploration of multi-modal interaction cues (touch, voice) and adaptive schemes for real-time, human-in-the-loop editing appear promising.

Conclusion

"From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks" introduces a principled, efficient framework for transforming visual in-context learners into interactive, user-guided systems. The i-DeLVM architecture demonstrates robust generalization to unseen interaction cues, consistently outperforming static ICL models across several tasks. The findings both challenge the static paradigm of visual prompting and chart a pathway toward fluid, user-centric model interfaces, with implications for interactive AI systems across vision and multi-modal domains (2604.06748).