Instruction-ViT: Vision Transformer with Prompts

• The paper introduces a framework that integrates instruction tuning and multi-modal prompt learning into Vision Transformers for scalable adaptation.
• It employs a dual loss strategy combining cross-entropy and prompt–CLS alignment to enhance class-wise discrimination in limited-data regimes.
• Experimental results demonstrate improved accuracy and faster convergence across diverse datasets while reducing computational overhead.

Instruction-ViT is a framework that integrates instruction tuning and prompt learning paradigms—originally pioneered in natural language processing—into Vision Transformer (ViT) architectures, with an emphasis on multi-modal prompts for image classification and, by extension, general visual instruction tuning. It enables parameter-efficient, scalable adaptation of vision models to new tasks and domains by injecting text and image prompts as additional input tokens, leveraging prototype-style supervision, and employing multi-objective training to enhance class-wise discrimination under limited or shifting data regimes (Xiao et al., 2023, Huang et al., 2023).

1. Motivation and Foundations

Instruction-ViT is motivated by the paradigm shift in LLMs where instruction tuning and soft prompt learning enable frozen backbones to generalize across tasks by steering them with small sets of learned vectors. The vision analog aims to achieve the following:

• Parameter-efficient adaptation to new classification tasks via prompt tokens without full fine-tuning.
• Seamless multi-task scalability: swapping prompts addresses new classes, tasks, or domains.
• Improved data efficiency, convergence speed, and generalization under limited supervision.

Significant distinctions from NLP prompt learning include the need to adapt to high-dimensional, continuous, and inherently non-linguistic visual patch embeddings. Unlike textual prompts—which align naturally with discrete vocabularies—visual prompt tokens must be reconciled with patchified image representations, and computational overhead depends linearly on the prompt count unless managed via selection or gating (Xiao et al., 2023, Huang et al., 2023).

2. Model Architecture and Multi-Modal Prompt Integration

The canonical Instruction-ViT maintains a ViT backbone as the main encoder. The image input, after patchification, is combined with a learnable [CLS] token, and further augmented by concatenating prompt embeddings ahead of the image patch sequence. These prompts are multi-modal—textual, visual, or mixed—and encode class-level priors.

• Prompt Construction:
  • Text prompts are generated by populating 30 natural language templates per class (e.g., “a photo of a C”), encoded with a frozen CLIP text encoder, and averaged.
  • Image prompts use a representative sample image per class, encoded with a frozen CLIP vision encoder.
  • Mixed prompts entail arithmetic mean fusion of textual and image prototypes.

The input sequence thus takes the form:

$$X' = [x_{cls};\,P_{text};\,P_{image};\,X_{patches}]$$

where $X_{patches}$ denotes image patches, $P_{text}$ and $P_{image}$ denote prompt embeddings per class. This sequence is fed into the transformer, which learns to attend to both the multi-modal prompts and patch tokens without bespoke cross-modal modules.

The token dimension $d$ aligns with the ViT configuration (e.g., $d=768$ for ViT-B). During training, the number of prompts per modality is set to the class count, but can be pruned at inference. The model's architecture is compact, and the extra computational burden is minor relative to patch tokens (Xiao et al., 2023).

3. Training Objectives, Losses, and Prompt–CLS Alignment

Instruction-ViT utilizes a composite loss. The two principal terms are:

• Cross-Entropy Classification Loss:

$$L_{pred} = - \sum_{c=1}^{C} y_c \log \operatorname{softmax}(logits_c)$$

where $y_c$ is the one-hot ground truth for class $c$.

• Prompt–CLS Softmax-NCA Alignment Loss:

Prompt outputs serve as class “prototypes.” The model calculates the cosine similarity between the [CLS] token output and each prompt embedding, enforcing the correct class’s prompt to be maximally similar:

$$L_{score} = -\log \left(\frac{\exp(z_{+})}{\sum_{i=1}^{C}\exp(z_i)}\right)$$

with $z_i$ the CLS-to-prompt similarity for class $i$ and $z_{+}$ matching the ground-truth class.

The overall objective is additive:

$L = L_{pred} + L_{score}$

This dual supervision encourages the transformer to use prompt context in its decision making, blending semantic priors (from class-descriptive text and prototypes) with learned visual features (Xiao et al., 2023). In prompt-tuning variants, only prompts and head layers train, while the ViT backbone remains frozen for efficiency.

4. Experimental Results, Comparisons, and Ablation

The Instruction-ViT framework was validated on multiple image classification datasets: Caltech-101, Oxford-III Pets, Stanford Cars, Oxford-102 Flowers.

Key findings include:

• Top-1 accuracy (mean across four datasets):
  • Full fine-tune: Instruction-ViT 95.59%, ViT-B 95.57%.
  • Prompt-tune: Instruction-ViT+text 73.38%, Visual Prompt Tuning (VPT) 65.30%.
• Prompt modality ablation:
  • Caltech-101 favors image prompts (80.85%).
  • Oxford Pets and Flowers favor text prompts (84.74%, 63.20%).
  • Stanford Cars favors mixed prompts (66.11%).
• Computational advantages: Overhead is minimal since the number of prompt tokens per batch is minor compared to patch tokens. At inference, computational cost can be curtailed by filtering to the top $K$ class prompts using CLIP similarity, reducing complexity from $N$ prompts to $K+1$ (Xiao et al., 2023).

The framework demonstrates faster convergence (fewer epochs required for peak accuracy) and enhanced adaptability under domain shifts, due to explicit class-level prior injection by multi-modal prompts.

5. Broader Context and Extensions: Visual Instruction Tuning

Instruction-ViT exemplifies the proliferation of Visual Instruction Tuning (VIT) frameworks, which generalize the approach of merging natural-language prompts and vision models for diverse vision-language tasks. The essential design criteria across the domain include:

• General Recipe: Multimodal model $f_\theta$ receives image $I$ and task instruction $\tau$, producing target output $y$; trained on datasets $D = \{(I_n, \tau_n, y_n)\}$ (Huang et al., 2023).
• Architectural Strategies: Multi-modal prompt tokens (Instruction-ViT), region-level conditioning (e.g., PVIT), adapters for patch-token/LLM-token alignment, and mixture-of-modality approaches.
• Objective Functions: Cross-entropy for discriminative tasks, sequence likelihoods for generative outputs, and various forms of contrastive or regularization terms.
• Benchmarks and Datasets: ImageNet variants, domain-shift sets, fine-grained and OOD benchmarks for classification; VQA, captioning, and grounding data for broader instruction-following tasks.
• Comparative Insights: Prompt tuning yields competitive accuracy with much lower parameter cost than full fine-tuning ($\sim$2% gap), and prompt filtering strategies can optimize compute-performance trade-offs.

Instruction-ViT, by directly fusing textual and visual prompts into the ViT backbone and utilizing joint loss objectives, assumes a position at the intersection of classical classification, multi-modal representation learning, and modern instruction-following paradigms (Huang et al., 2023).

6. Limitations, Open Problems, and Future Directions

Current limitations and future research priorities for Instruction-ViT include:

• Prompt construction: Use of 30 templates and single example images per class is heuristic; more systematic, possibly learned, prompt generation or expansion may yield greater generalization.
• Prompt inference selection: Reliance on zero-shot CLIP similarity for prompt filtering may bias results towards pretraining domain priors.
• Expansion to detection/segmentation: Current application is limited to classification; robust extension to detection, segmentation, and vision–language generation remains open.
• Data and task generalization: Investigation into prompt scaling laws (prompt count and length vs. performance), multimodal data efficiency, and robust adaptation to new or few-shot tasks.
• Instruction data quality and alignment: Broader VIT challenges include the need for richer and more diverse instruction sets, evaluation of instruction fidelity, and mitigation of instruction-induced biases or hallucinations (Xiao et al., 2023, Huang et al., 2023).

Future work is likely to center on dynamically learned prompt generators, generalized region- and temporal-prompt tokens for structured vision tasks, unified encoder-decoder architectures accommodating arbitrary task specifications, and rigorous benchmarking for instruction alignment, safety, and generalization.

7. Summary Table: Prompt Modalities and Ablation Results (from (Xiao et al., 2023))

Dataset	Best Prompt Modality	Top-1 Accuracy (%)
Caltech-101	Image prompt	80.85
Oxford Pets	Text prompt	84.74
Oxford Flowers	Text prompt	63.20
Stanford Cars	Mixed prompt	66.11

This table demonstrates that optimal prompt modality is dataset-dependent, highlighting the importance of prompt engineering and modality selection for maximal performance.

---

Instruction-ViT substantiates the viability of bringing lightweight, instruction-driven adaptation to vision transformers, capturing the benefits of prompt learning for both efficient fine-tuning and robust generalization, and thus marks a significant step towards truly unified, prompt-driven vision systems (Xiao et al., 2023, Huang et al., 2023).