Steerable Visual Representations

• Steerable visual representations are learned features that can be actively controlled to align with specific transformations while preserving mathematically defined equivariance properties.
• They leverage methods such as steerable CNNs, dynamic steerable blocks, and transformer-based architectures to enable efficient, parameter-light adaptations for robust and interpretable visual inference.
• Applications of these representations include improved robustness, enhanced interpretability, and controllability in tasks ranging from aerial imaging to 3D object segmentation and multimodal reasoning.

Steerable visual representations are a class of learned features in computer vision and multimodal systems that can be continuously or discretely controlled—steered—to align with specific transformations, concepts, or user intentions, while preserving or enabling equivariance properties. Steerability, originally grounded in harmonic analysis and equivariant neural networks, now includes a spectrum of techniques spanning convolutional, transformer-based, multimodal, and user-interactive architectures. The core principle is that such representations maintain predictable, mathematically defined responses under transformations—either explicit group actions (such as rotations) or semantic modifications (such as text prompts)—allowing robust controllability, interpretability, and task-specific adaptation of visual features (Cohen et al., 2016, Papapetros et al., 20 Jul 2025, Bhardwaj et al., 2023, Ruthardt et al., 2 Apr 2026).

1. Mathematical Foundations: Equivariance, Steerability, and Induced Representations

Steerability in visual representations formalizes the notion that, for a given group of transformations $G$ (e.g., spatial rotation, photometric shift, or semantic prompt), image features or embeddings transform under group actions via explicit, often linear, operators. For a learned encoder $f: \mathcal{X} \to \mathbb{R}^d$ and transformation $g \in G$, equivariance is expressed as

$f(g \cdot x) = L_g f(x)$

where $L_g$ is a known or learnable linear map. Steerability requires that $L_g$ be not only mathematically well-defined, but computable by a parameter-efficient procedure, often via basis functions or representation theory (Cohen et al., 2016, Papapetros et al., 20 Jul 2025, Bhardwaj et al., 2023).

In group convolutional networks, equivariant filter banks $\Psi$ satisfy the "intertwiner" constraint:

$$\rho_{\mathrm{out}}(h) \Psi = \Psi \pi_{\mathrm{in}}(h), \quad \forall h \in H$$

where $\rho_{\mathrm{out}}, \pi_{\mathrm{in}}$ are representations of $H \leq G$ on the output and input fibers, respectively, and $f: \mathcal{X} \to \mathbb{R}^d$0 is the symmetry group (Cohen et al., 2016). In steerable filter parameterizations, rotated versions of a filter are constructed as linear combinations of a small filter basis with predetermined functional coefficients, e.g., circular harmonics for $f: \mathcal{X} \to \mathbb{R}^d$1 or spherical harmonics for $f: \mathcal{X} \to \mathbb{R}^d$2 (Papapetros et al., 20 Jul 2025, Shen et al., 2022, Melnyk et al., 2021). This allows rotation of a filter or feature map without explicit spatial rotation.

For 3D domains, equivariant convolution and steerable neurons are designed via representation theory of $f: \mathcal{X} \to \mathbb{R}^d$3, conformal embeddings, or partial differential operator (PDO) bases (Shen et al., 2022, Melnyk et al., 2021). These methods ensure that representations are equivariant (and thus steerable) to arbitrary rotations and subgroups, with steerability realized via explicit interpolation across basis responses.

2. Steerable Architectures: CNNs, Transformers, and 3D Extensions

Steerable representations are realized across multiple neural architectures:

• Steerable CNNs: Each convolutional filter is constrained or parameterized to be equivariant to a group $f: \mathcal{X} \to \mathbb{R}^d$4 (e.g., $f: \mathcal{X} \to \mathbb{R}^d$5 for rotations/reflections in 2D, $f: \mathcal{X} \to \mathbb{R}^d$6 subgroups for 3D). This is achieved by constructing filter banks in intertwiner subspaces, or as steerable linear combinations of a small set of filter bases (e.g., via complex exponentials or PDOs). Steerable convolutional layers replace standard convolutions in network backbones such as ResNet, resulting in architectures (e.g., sresnet50c8) with true built-in equivariance (Cohen et al., 2016, Papapetros et al., 20 Jul 2025, Shen et al., 2022).
• Dynamic Steerable Blocks: Early generalizations include parameterizing filters over steerable frames (e.g., Gaussian derivatives, wavelets) and using dynamic, input-conditioned steering matrices $f: \mathcal{X} \to \mathbb{R}^d$7 to adapt filter coefficients spatially. Such blocks provide localized, input-adaptive equivariance, further enhancing expressivity and sample efficiency (Jacobsen et al., 2017).
• Steerable Transformers: Vision Transformers are extended with equivariant attention mechanisms and steerable convolutions. Attention is defined over Fourier-parameterized irrep fibers, with queries, keys, and positional embeddings constructed to preserve $f: \mathcal{X} \to \mathbb{R}^d$8-equivariance pointwise in irrep space. Steerable transformers, when added to steerable-convolutional networks, consistently improve performance on both 2D and 3D tasks by enhancing global context aggregation while retaining strict symmetry constraints (Kundu et al., 2024).
• 3D Steerable Neurons and PDO-s3DCNNs: For 3D data, steerable networks employ conformal embeddings, minimal tetrahedral bases, or PDO-based filtering to ensure rotation-equivariance (or full steerability) under $f: \mathcal{X} \to \mathbb{R}^d$9. These constructions yield efficient, minimal parameterizations and strong empirical results on point sets, volumetric data, and segmentation tasks (Melnyk et al., 2021, Shen et al., 2022).

3. Training Strategies and Implementation Protocols

Steerable models are typically either equivariant "by design" or by explicit regularization.

• Group-convolutional and frame-based architectures: Equivariance is ensured structurally by filter constraints or parameterization (see above). Training follows standard supervised or metric-learning regimes, e.g., cross-entropy or multi-similarity loss, but often with greatly reduced need for data augmentation (rotations, flips) since equivariance is already baked in (Papapetros et al., 20 Jul 2025).
• Latent Steerability via Regularized Mapping: Architectures that reparameterize only the embedding space (e.g., steerable equivariant representation learning) add auxiliary losses to encourage that embeddings respond to transformation parameters via explicit, learned maps $g \in G$0. The equivariance loss

$g \in G$1

promotes steerability with respect to augmentation $g \in G$2, with uniformity losses added to preserve embedding diversity (Bhardwaj et al., 2023).

• Steerable MLLMs and Vision-Language Interventions: Steering methods in VLMs include activation-space interventions using concept vectors derived from sparse autoencoders (SAEs), mean-shifted activations, linear probing, or explicitly aligned joint sparse factorizations. These are injected at targeted network layers, requiring no weight updates, and support both directed enhancement and suppression of semantic factors (Gan et al., 20 May 2025, Shu et al., 24 Jun 2026). MoReS applies per-layer linear “down–up” projections to visual features, achieving steerability (and low parameter count) by directly amplifying the visual subspace before multi-head attention (Bi et al., 2024).

4. Applications: Robustness, Interpretability, and Controllability

Steerable visual representations deliver concrete benefits across a spectrum of vision and multimodal tasks:

• Equivariance for Robustness and Data Efficiency: In aerial visual place recognition, steerable CNNs achieve up to 12% higher recall relative to the strongest non-steerable networks, particularly under severe rotational ambiguity inherent in UAV imagery (Papapetros et al., 20 Jul 2025). Similar gains are reported across 3D object retrieval and segmentation with O, V, and $g \in G$3 subgroup-equivariant architectures (Shen et al., 2022, Melnyk et al., 2021).
• Transfer Learning and Downstream Tasks: Steerable representations improve transfer accuracy (1–3% gains for linear probes) and robustness against synthetic corruptions (up to +3.4% ImageNet-C accuracy), and enable dramatic inference-time speed-ups via latent-space augmentation (~50x, as only embeddings need transformation) (Bhardwaj et al., 2023).
• Interpretability and Fine-grained Control: Techniques such as Diffusion Steering Lens render the direct visual effect of targeted submodules (e.g., single ViT heads) by modulating internal states and neutralizing all other activations, providing interpretable, steerable visualizations of internal network features—with surgical precision and no retraining (Takatsuki et al., 18 Apr 2025).
• Controllable Multimodal Reasoning: In MLLMs and VLMs, steering vectors derived from text-only LLMs, joint sparse autoencoders, or visual subspace projections can improve spatial reasoning, counting, and alignment with human semantic intent. For example, mean-shift steering boosts spatial relation accuracy by +7.3pp and supports robust OOD generalization, outperforming prompt-only interventions (Gan et al., 20 May 2025, Shu et al., 24 Jun 2026).
• User-driven Representations: In interactive DR scenarios, users can dynamically steer visual projections via natural language prompts, specifying semantic relationships or exploration axes not present in original data dimensions. This approach fuses raw representations and semantic prototypes in low-dimensional projections, leading to improved cluster separation and interpretability in data visualization (Oliveira et al., 18 Jun 2025).
• Language-Controllable Vision: Early fusion of text into frozen ViT layers (e.g., SteerViT) creates representations that can focus on any prompted object, concept, or attribute, outperforming late-fusion methods on anomaly detection, personalized discrimination, and zero-shot conditional retrieval, all while maintaining standard transfer performance (Ruthardt et al., 2 Apr 2026).

5. Empirical Results and Benchmarking

Steerable models consistently show strong empirical gains over conventional baselines across diverse benchmarks and modalities:

Task/Benchmark	Steerable Method	Best Non-steerable	Steerable Result	Δ (pp or %)
Aerial vPR R@1 (LASED)	sresnet50c8	CosPlace	64.5%	+6.6pp
SHREC'17 3D retrieval	PDO-s3DCNN (O-reg)	SE3–CNN	58.6	+3.1
CV-Bench (Relation/Count)	MLLM MeanShift	Baseline	+7.3pp / +3.3pp	Up to +34.2pp OOD
Core Retrieval Accuracy	SteerViT	DINOv2	96.0%	+52pp
Zero-shot Anomaly (PRO)	SteerViT	MaskCLIP	82.1%	+41.6pp
DR Cluster Silhouette	Steerable projections	Unsteered	0.70 (MNIST)	+0.22
PEFT param. reduction	MoReS (LLaVA Steering)	LoRA	~500–1150× fewer	Comparable accuracy

All quantitative improvements are reported as specified in the cited papers (Papapetros et al., 20 Jul 2025, Shen et al., 2022, Gan et al., 20 May 2025, Ruthardt et al., 2 Apr 2026, Bi et al., 2024, Oliveira et al., 18 Jun 2025).

6. Limitations and Open Challenges

Several limitations and outstanding questions emerge in current steerable representation research:

• Group-specific Design: Most existing steerable architectures focus on specific transformation groups (e.g., $g \in G$4, $g \in G$5, $g \in G$6), requiring custom basis constructions and sometimes combinatorial explosion in high-dimensional or composite-group cases (Cohen et al., 2016, Shen et al., 2022).
• Sparse and Localized Steering: Generic frameworks for learning steerable interventions at spatial or token granularity remain underdeveloped for many architectures (contrast JSAE's broadcast codes, localized steerability in SteerViT) (Shu et al., 24 Jun 2026, Ruthardt et al., 2 Apr 2026).
• Prompt and Data Dependence: Semantic or user-driven steering relies on high-quality zero-shot MLLM outputs and the existence of well-specified prompts or prototypes. Misclassifications or ambiguous prompts can degrade steerability and cluster coherence (Oliveira et al., 18 Jun 2025, Ruthardt et al., 2 Apr 2026).
• Transfer and Generalization: While universal input-based approaches (VISOR++) enable cross-model behavioral steering without model access, transfer effectiveness varies across architectures and steering directions, with uneven success for positive and negative alignment tasks and scalability to ensemble settings an open area (Balakrishnan et al., 29 Sep 2025).
• Computational and Data Efficiency: Although parameter efficiency can be dramatic (e.g., 500–1000× reduction in MoReS) (Bi et al., 2024), some methods incur significant optimization overhead (e.g., VISOR++ universal image synthesis), or require group-specific SVD/basis precomputation.
• Theory vs. Practice: Explicit equivariance by construction outperforms learned equivariance under weak data or group constraints. However, steerability for non-group semantic factors (e.g., "style", "scene context") remains partly heuristic and lacks exact mathematical guarantees.

7. Outlook and Future Research Directions

The landscape of steerable visual representations is rapidly expanding:

• Unified Steerability Frameworks: Integrating group-based steerability with semantic or instruction-driven control holds promise for both interpretable representations and flexible task adaptation (Ruthardt et al., 2 Apr 2026, Gan et al., 20 May 2025).
• Efficient, Generalizable Mechanisms: Exploring universal input-based steering, joint sparse autoencoders for cross-modal controllability, and scalable equivariant transformer layers across dimensions and groups enables broader deployment in vision and multimodal systems (Kundu et al., 2024, Shu et al., 24 Jun 2026, Balakrishnan et al., 29 Sep 2025).
• Human-centered Interaction: Embedding user feedback and semantic prompts directly in representation adaptation, as in steerable projection DR, supports interactive, explainable AI workflows (Oliveira et al., 18 Jun 2025).
• Localized and Hierarchical Steering: Future methods will likely support token-level, patch-level, and hierarchically compositional steering, combining local equivariance with global semantic control (Ruthardt et al., 2 Apr 2026).
• Theory-Driven Robustness: Analyzing the trade-offs between strict equivariance, steerability, and generalization, as well as the impact of grouped and ungrouped symmetries on robustness and transfer, remains a key direction (Bhardwaj et al., 2023).

Steerable visual representations have emerged as a foundational innovation in aligning vision systems with the structure of both data and tasks—enabling precise, efficient, and controllable visual inference across domains from robotics and UAV navigation to human-centered data exploration (Cohen et al., 2016, Papapetros et al., 20 Jul 2025, Ruthardt et al., 2 Apr 2026, Oliveira et al., 18 Jun 2025).