SteerViT: Steerable Visual Representations

• SteerViT is a class of visual encoders that support explicit control over feature representations through transformation and semantic conditioning.
• It integrates equivariant latent mappings, SE(d)-equivariant architectures, and cross-attention for language driving, enabling precise manipulation.
• Empirical results demonstrate improved robustness, accuracy, and retrieval performance compared to traditional invariant or late fusion models.

Steerable Visual Representations (SteerViT) refer to a class of visual encoders whose representations can be explicitly directed or conditioned along specific axes—such as image transformations or semantic concepts—while maintaining compatibility with generic visual tasks. The term encompasses advances in (1) embedding-level steerability with respect to image-space augmentations via equivariant mappings, (2) group-theoretic steerability for volumetric and geometric data, and (3) language-driven steering of visual feature extractors via early multimodal fusion. Three principal lines of research illustrate SteerViT methodologies: equivariant representation learning for image augmentations (Bhardwaj et al., 2023), SE(d)-equivariant Transformers for geometric data (Kundu et al., 2024), and language-steerable vision transformers for semantic control (Ruthardt et al., 2 Apr 2026).

1. Definition and Theoretical Foundations

Steerability in visual representation learning is operationalized via explicit mappings that allow representations to respond predictably under transformations or conditioning. Given a group (or family) of image transformations $G$, a representation $h: X \to \mathbb{R}^d$ is $G$-equivariant if

$$\forall g \in G,\ \exists M_g: \mathbb{R}^d \to \mathbb{R}^d\quad \text{such that}\quad M_g h(x) = h(g \cdot x).$$

The existence of known, learnable maps $M_g$ enables direct manipulation in latent space—a property denoted as steerability. Invariance emerges as the degenerate case with $M_g = I_d$ so that $h(g \cdot x) = h(x)$ (Bhardwaj et al., 2023).

In the geometric setting, steerable convolutions are constrained to ensure equivariance to the special Euclidean group SE$(d)$, encapsulating translations and rotations. Here, a feature map $f_\ell: \mathbb{R}^d \times SO(d) \to \mathbb{R}^C$ satisfies

$$f'_\ell(x, R') = [(g \cdot f_\ell)](x, R') := f_\ell(Rx + t, RR'),$$

with corresponding convolutional kernel constraints (Kundu et al., 2024).

In language-driven contexts, steerability denotes the capacity of a vision encoder to route semantic control signals—typically provided as text—deep into its feature extraction layers, enabling dynamic rerouting of visual attention and embedding focus at inference time (Ruthardt et al., 2 Apr 2026).

2. Methodologies for Steerable Representation Construction

2.1 Equivariant Latent Mappings with Augmentation Families

For per-augmentation steerability, the backbone encoder $h: X \to \mathbb{R}^d$0 (e.g., a ResNet or ViT) is augmented with learned MLP-based mappings $h: X \to \mathbb{R}^d$1, one per augmentation type $h: X \to \mathbb{R}^d$2. Each $h: X \to \mathbb{R}^d$3 takes as input the latent feature $h: X \to \mathbb{R}^d$4 and augmentation parameters $h: X \to \mathbb{R}^d$5, predicting an embedding to match $h: X \to \mathbb{R}^d$6. Training optimizes both classification and equivariance losses: \begin{align*} L_{CE}(x) & = -\sum_c y_c(x) \log p_c(x; w),\ L_E^a(x) & = | M_a(h(x; w), \theta_a) - h(g_a(x; \theta_a); w) |^2_2, \end{align*} with optional regularization to enforce representation diversity (Bhardwaj et al., 2023).

2.2 SE(d)-Equivariant Architectures

For volumetric and geometric data, steerability is enforced via group convolutional layers respecting SE$h: X \to \mathbb{R}^d$7 equivariance. Steerable self-attention is achieved by defining queries, keys, and values as functions of group elements and operating in the Fourier domain over irreducible representations (irreps) of $h: X \to \mathbb{R}^d$8. Nonlinearities are implemented via harmonic-norm or Clebsch–Gordan mechanisms to preserve group-equivariant structure. Hybrid architectures interleave steerable convolution blocks with multihead SE$h: X \to \mathbb{R}^d$9-equivariant Transformer layers for enhanced global feature interaction (Kundu et al., 2024).

2.3 Language-Steering of Vision Transformers

SteerViT (in the language-guided context) augments a frozen pretrained ViT by injecting lightweight cross-attention (CA) modules directly into the residual stream of the vision encoder. A text encoder (RoBERTa-Large) produces prompt embeddings, projected to the visual dimension and fused into the ViT every other block (total six points in a 12-block backbone). Fusing the CA output with a learned scalar gate $G$0 via

$G$1

allows continuous control between pure visual and fully language-conditioned representations. All original backbone and text encoder weights remain frozen; only the CA modules and projection MLP are trained (Ruthardt et al., 2 Apr 2026).

3. Training Paradigms and Data

Equivariant latent models are trained with standard classification objectives plus augmentation-indexed equivariance losses. Uniformity regularization promotes non-collapsed, discriminative embeddings. Training data comprises standard image/task pairs with labels, as well as augmentation parameter samples (Bhardwaj et al., 2023).

SE$G$2-equivariant Transformers are trained on volumetric or 2D datasets (e.g., RotMNIST, ModelNet10) with random transformations sampled from the relevant group, ensuring exposure to full transformation variability (Kundu et al., 2024).

Language-steerable SteerViT leverages masked referential segmentation as the supervision signal. Image-text pairs are curated from datasets including RefCOCO/g, Visual Genome, LVIS, and Mapillary Vistas, amassing 2.28M pairs. No contrastive or reconstruction losses are employed; only a cross-entropy over predicted patch masks supports training. Backbones and language encoders remain frozen, and the only trainable parameters are in the CA and associated projection modules (~21M parameters) (Ruthardt et al., 2 Apr 2026).

4. Empirical Results and Performance Characteristics

4.1 Latent Equivariance and Representation Robustness

On transfer learning linear probes (Flowers-102, DTD, Pets, Caltech-101), steerable equivariant embeddings yield 1–3% absolute improvements over invariant baselines. On ImageNet-C corruption robustness, mean classification error improves from $G$30.845 to $G$40.810 (+3.4% relative accuracy) (Bhardwaj et al., 2023).

Latent steerability enables batch-efficient test-time application of large augmentation sets, accelerating OOD detection by $G$5 (e.g., $G$6 s vs $G$7 s per batch for $G$8), and increasing OOD AUC by 5–15% over invariant representations.

4.2 Group Equivariance in Geometric Data

On Rotated MNIST, SteerViT (Steerable Transformer) achieves best-known error (0.97–1.18% depending on cutoff) against other $G$9-equivariant Transformers, and on ModelNet10 attains accuracy competitive with volumetric SOTA at matched parameter counts (Kundu et al., 2024). Higher Fourier cutoffs and grid resolutions demonstrate monotonic improvement, limited by quadratic attention scaling.

4.3 Language-Steered Vision Transformers

In the semantic steering regime, SteerViT offers state-of-the-art conditional retrieval and discrimination:

Benchmark	SteerViT	Next-Best Model(s)
CORE (cond. retrieval R@1)	96.0%	FLAIR 81.3%, CLIP/SigLIP $$\forall g \in G,\ \exists M_g: \mathbb{R}^d \to \mathbb{R}^d\quad \text{such that}\quad M_g h(x) = h(g \cdot x).$$040–60%, InternVL3-2B 89.4%
GeneCIS zero-shot retrieval R@1	25.4%	DINOv2 9.6%
PODS PR-AUC (personal objects)	58.1%	DINOv2: 29.6, specialist DINOv2: 48.0

Zero-shot anomaly segmentation (MVTec AD): SteerViT achieves PRO=82.1, surpassing CLIPseg=34.6, MaskCLIP=40.5, SAM3=54.5, approaching specialist FADE at 84.5.

Performance is robust across ViT, SigLIP, and MAE backbones; early fusion confers 15–35 point advantages on weakly supervised backbones compared to late fusion (Ruthardt et al., 2 Apr 2026).

5. Architectural Ablations and Trade-offs

Ablations indicate:

• Early fusion of language via cross-attention outperforms late fusion (post-encoder) in both steerability and base representation quality.
• Gated fusion (with tanh/α) maintains performance during initial training and allows continuous tradeoff at inference—scaling $$\forall g \in G,\ \exists M_g: \mathbb{R}^d \to \mathbb{R}^d\quad \text{such that}\quad M_g h(x) = h(g \cdot x).$$1 by $$\forall g \in G,\ \exists M_g: \mathbb{R}^d \to \mathbb{R}^d\quad \text{such that}\quad M_g h(x) = h(g \cdot x).$$2 dials between steerability and backbone fidelity, with optimal performance for $$\forall g \in G,\ \exists M_g: \mathbb{R}^d \to \mathbb{R}^d\quad \text{such that}\quad M_g h(x) = h(g \cdot x).$$3.
• Finer prompt specificity enhances instance discrimination (e.g., PODS PR-AUC climbs from 27.9 with a category prompt (“mug”) to 58.1 with a fully descriptive prompt).
• Too few CA layers significantly diminish steerability, while over-insertion yields diminishing returns.
• For augmentation-equivariant models, uniformity losses are essential to avoid representation collapse, and separate $$\forall g \in G,\ \exists M_g: \mathbb{R}^d \to \mathbb{R}^d\quad \text{such that}\quad M_g h(x) = h(g \cdot x).$$4 are required per augmentation family (Bhardwaj et al., 2023).

6. Practical Implications and Applications

Steerable Visual Representations unlock fine-grained, “what-if” style manipulation of visual descriptors without costly recomputation. Efficient test-time “steering” enables OOD detection by exposing classifier sensitivity to non-nuisance transformations. In group-equivariant settings, rigorous treatment of spatial symmetries via steerable convolutions and attention allows tractable, accurate processing of 3D/volumetric data within globally aware transformer frameworks (Kundu et al., 2024).

Language-driven SteerViT augments standard ViT feature spaces with on-demand semantic focus, outperforming both unimodal and contemporary multimodal models on conditional retrieval, attention re-routing, personalized object discrimination, and anomaly localization—all while preserving the core structure and quality of the original vision encoder (Ruthardt et al., 2 Apr 2026).

These results collectively shift the Pareto frontier of visual representation learning: SteerViT methods achieve previously incompatible properties—strong steerability and high generic vision performance—within unified, modular frameworks.