Semantic Feature Conditioning

• Semantic feature conditioning is a technique that injects structured semantic signals (textual, visual, geometric, or latent) to guide internal model computations.
• It employs mechanisms such as cross-attention, residual adapters, and spatial grounding to align outputs with object identity, region semantics, and navigational intent.
• Recent studies highlight design patterns like lightweight adapters and dynamic routing that balance semantic fidelity with generative diversity.

Semantic feature conditioning denotes the use of semantically structured side information to steer a model’s internal computation toward semantically plausible outputs. Across the works surveyed here, the conditioning signal may be textual, visual, geometric, compositional, or latent, but its function is consistent: it injects information about object identity, region semantics, action structure, navigational intent, or concept-level abstractions into the model so that generation, restoration, segmentation, correspondence, or prediction is semantically constrained rather than purely appearance-driven. In diffusion-based super-resolution, this takes the form of text- or feature-conditioned denoising UNets (Chen et al., 26 Oct 2025, D'Oronzio et al., 28 Apr 2026); in Diffusion Transformers, it appears as adaptive fusion of hidden states from multiple LLM layers (Li et al., 3 Feb 2026); in autoregressive image models, it appears as semantic prefilling and hidden-state alignment (Jin et al., 18 Nov 2025); and in sequence models, it appears as explicit semantic channels or latent steering features (Huang et al., 14 Sep 2025, Yang et al., 19 Jan 2025).

1. Core definitions and mathematical formulations

In diffusion-based image super-resolution, semantic feature conditioning refers to injecting text-derived semantic features into the denoising UNet so that restoration is guided by object- and region-level semantics. Given a low-resolution image $x$ and text tags $c$, the denoising model conditions its prediction on $c$ via cross-attention, with visual features as queries and text embeddings as keys and values. A standard formulation is

$$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$

with classifier-free guidance further amplifying the conditional component during sampling (Chen et al., 26 Oct 2025).

In DiT-based text-to-image systems with LLM text encoders, semantic feature conditioning is defined as synthesizing the text condition fed to a Diffusion Transformer by aggregating hidden-state sequences from multiple LLM layers. The fused condition at diffusion time $t$ and network depth $d$ is

$$H_{\mathrm{cond}}(t,d) = \sum_{\ell=1}^{L} \alpha_{t,d}^{(\ell)} \cdot \mathrm{LN}(H^{(\ell)}),$$

with $\sum_{\ell} \alpha_{t,d}^{(\ell)} = 1$ and $\alpha_{t,d} = \mathrm{Softmax}(z_{t,d})$. This normalized convex fusion formalizes semantic conditioning as a routing problem over the semantic hierarchy of the LLM (Li et al., 3 Feb 2026).

In autoregressive image generation and editing, semantic feature conditioning is defined as injecting high-level, dense visual semantics into the decoding process so the model can follow instructions or visual controls before and during token prediction. SCAR implements this with Compressed Semantic Prefilling and Semantic Alignment Guidance. The model consumes a prefix $P_s$ derived from compressed DINOv2 features, concatenates it with text and target VQ tokens, and trains with

$c$0

where $c$1 matches last-layer hidden states at semantic-prefix positions to the target image semantics (Jin et al., 18 Nov 2025).

In controllable language modeling, semantic feature conditioning is explicitly token-level and interpretable. Semantic Fusion augments a Transformer LM with a parallel fuzzy-membership feature channel $c$2 encoding POS-like cues, shallow roles, boundary flags, and sentiment polarity or strength. The fused representation is

$c$3

so the LM’s hidden state is directly modulated by a human-readable semantic scaffold (Huang et al., 14 Sep 2025).

A different but related formulation appears in feature-level activation steering for LLMs. LF-Steering maps hidden states into a sparse autoencoder feature space, edits only the selected semantic coordinates, and decodes back to model space. This treats semantic feature conditioning as targeted manipulation of disentangled latent factors rather than injection of external modalities (Yang et al., 19 Jan 2025).

2. Conditioning pathways inside modern architectures

The dominant implementation pattern is to inject semantic information into internal representations rather than only at the input. Cross-attention remains the most common mechanism. Stable Diffusion-based super-resolution systems use cross-attention at every UNet scale; SeeSR contains 16 UNet cross-attention layers, and SRSR modifies all of them at inference time (Chen et al., 26 Oct 2025). SCOPE injects frozen DINOv2 embeddings at the fifth down-sampling block and the second up-sampling block of an attention UNet, with UNet features supplying queries and DINOv2 tokens supplying keys and values (Hönig et al., 29 Sep 2025). ReNoV uses cross-view attention in a dual-U-Net architecture, concatenating keys and values from target and reference views so that projected multi-view features act as conditions throughout denoising (Kwak et al., 12 Feb 2026).

Other systems avoid direct cross-attention and instead use additive or residual conditioning. CLIP-Guided SAM injects CLIP-derived text, patch-level vision features, and patch-wise text–image similarity maps into SAM’s image encoder through lightweight semantic adapters placed in parallel to each MLP. At layer $c$4, the conditioned feature is

$c$5

so semantics influences SAM’s internal feature formation while preserving the original promptable interface (Jalilian et al., 24 May 2026). Control-DINO similarly uses a lightweight ControlNet-style branch that produces residuals added to the first 16 blocks of a frozen video diffusion backbone, with optional spatial masking $c$6 for localized control (Dominici et al., 2 Apr 2026).

Several works replace text entirely with dense visual features. GramSR feeds DINOv3 patch tokens from the low-resolution image through a two-layer MLP adapter and uses them in place of the usual text-conditioning tensor of Stable Diffusion v2.1, while keeping the SD backbone frozen and training only LoRA modules and the adapter (D'Oronzio et al., 28 Apr 2026). In local correspondence, semantic feature conditioning combines refined texture descriptors and DINOv2 semantic descriptors, with the final similarity defined as

$c$7

so semantic agreement multiplicatively gates texture similarity (Cadar et al., 2024).

Neural fields provide a compact comparative case. For 2D semantic segmentation, conditioning via simple concatenation, FiLM, and Cross-Attention was compared under identical encoder-decoder settings, and conditioning via Cross-Attention achieved the best results and was competitive with a CNN-based decoder (Gromniak et al., 2023). This suggests that when semantic conditioning must combine global context and localized detail, token-to-token interaction is typically more expressive than static feature concatenation.

3. Spatial grounding, semantic alignment, and locality

A central problem in semantic feature conditioning is that semantic guidance is often too coarse unless it is grounded in space. SRSR isolates this issue explicitly. Standard text conditioning in diffusion SR suffers from inaccurate or incomplete tags and from cross-attention drift, in which tokens attend to irrelevant pixels. SRSR addresses this by grounding each degradation-aware prompt extractor tag with Grounded SAM 2, constructing binary token-region gates $c$8, and re-normalizing the masked attention weights. It further introduces Spatially Targeted Classifier-Free Guidance so that ungrounded pixels follow unconditional sampling. On RealSR, this yields PSNR 26.40, SSIM 0.7632, LPIPS 0.2718, and DISTS 0.2092, each reported as best; on DRealSR it yields PSNR 29.50, SSIM 0.8128, LPIPS 0.2866, and DISTS 0.2176, again best (Chen et al., 26 Oct 2025).

ReNoV grounds semantics in geometry rather than text. Its Projected Representation Conditioning binds external visual features to 3D pointmaps, projects them into the target camera, and uses visibility masks and z-buffering to preserve correspondence. The resulting target conditioning tensor combines projected 3D coordinates, projected features, and the visibility mask:

$c$9

This produces geometry-consistent reconstructions of visible regions and semantically plausible inpainting of unseen regions (Kwak et al., 12 Feb 2026).

Spatially aligned conditioning also appears in CLIP-Guided SAM. Patch-wise CLIP similarities are reshaped into dense masks and sampled positive points, while the same CLIP signals are injected into the encoder. The ablations show that similarity features carry the dominant localization signal: on VOC 1/16, full fusion gives 78.3 mIoU, text-only gives 67.2, vision-only gives 68.8, and adapters without semantic inputs give 67.1 (Jalilian et al., 24 May 2026).

In controllable face diffusion, semantic masks and global attributes play complementary roles. A mask encoder without global average pooling preserves an $c$0 grid of 64 spatial tokens, which is concatenated with a single 512-dimensional attribute token. The multi-condition token sequence thus has length 65 and enables simultaneous control of spatial layout and global appearance. Mask-conditioned generation with spatial tokens achieves FID 8.31, mask accuracy 93.91%, and mIoU 79.06%, while multi-conditioning yields FID 8.39, attribute accuracy 90.19%, mask accuracy 94.06%, and mIoU 79.20 (Giambi et al., 2023).

4. Dynamic routing, inference-time steering, and semantic structure

Not all semantic conditioning is static. Some methods explicitly route semantic information as a function of model depth, timestep, or latent state. In Diffusion Transformers with LLM encoders, “Depth-wise Semantic Routing” assigns a learned convex combination of LLM layers to each DiT block. Among time-wise, depth-wise, and joint fusion, depth-wise fusion is reported as the best overall strategy, with aggregate scores of 67.07 on GenEval, 79.07 on GenAI-Bench, and 3.06 on UnifiedReward-Style, outperforming the penultimate-layer baseline at 64.54, 74.96, and 3.02. The largest capability gains appear in advanced compositional tasks, including +9.97 on Counting, +8.92 on Differentiation, and +5.59 on Comparison (Li et al., 3 Feb 2026).

The same paper also identifies a failure mode of time-only routing: under classifier-free guidance and FlowMatch Euler sampling, nominal timesteps do not track effective denoising progress, so purely time-wise fusion can degrade fidelity. This suggests that semantic feature conditioning is not only a question of what semantic features to inject, but also when and where in the network to inject them (Li et al., 3 Feb 2026).

Inference-time semantic steering is even more explicit in REPA-G. Here the conditioning signal is not a prompt but a target representation $c$1 from a differentiable feature extractor $c$2. The method defines a similarity potential $c$3, adds $c$4 to the reverse dynamics, and thereby samples from the tilted distribution

$c$5

Because $c$6 may encode global semantics, local patches, or multiple concepts, the method supports fine-grained texture matching, semantic guidance through global image tokens, and multi-concept composition entirely at inference time (Sereyjol-Garros et al., 3 Feb 2026).

LLMs exhibit a different form of routing: geometric entanglement among semantic axes. Projections of word features onto semantic axes such as beautiful–ugly or soft–hard correlate strongly with human ratings, with the strongest axes exceeding $c$7 and the weakest exceeding $c$8. Steering a word along one axis produces spillover along others in proportion to axis cosine similarity, approximately

$c$9

A plausible implication is that feature-level semantic conditioning in LLMs must account for the geometry of semantic subspaces rather than treat control dimensions as orthogonal (Kozlowski et al., 29 Apr 2026).

LF-Steering operationalizes that idea at inference time by editing sparse autoencoder features at a selected layer. On LLaMA2-7B-Chat, feature-level steering improves RobustBOOLQ accuracy to 66.40 ± 3.39 from a 46.40 ± 10.55 baseline, and improves PopQA_Capital mean pairwise cosine to 0.83 from 0.73, while preserving or slightly improving out-of-domain AG News accuracy at 70.60 versus 70.00 (Yang et al., 19 Jan 2025).

5. Domain-specific instantiations

Semantic feature conditioning has been instantiated across a wide range of tasks, with the semantic source and injection mechanism tailored to the prediction target.

Domain	Conditioning signal	Representative paper
Real-world super-resolution	Text tags, dense DINOv3 features, or grounded token-region masks	(Chen et al., 26 Oct 2025, D'Oronzio et al., 28 Apr 2026)
Text-to-image and video diffusion	Multi-layer LLM states, DINOv3 features, projected visual representations	(Li et al., 3 Feb 2026, Dominici et al., 2 Apr 2026, Kwak et al., 12 Feb 2026)
Autoregressive image generation/editing	Compressed DINOv2 semantic prefixes and alignment losses	(Jin et al., 18 Nov 2025)
Pose estimation and robotics	Frozen DINOv2 patch tokens plus RGB-D geometry	(Hönig et al., 29 Sep 2025)
Promptable segmentation	CLIP text, patch vision features, and similarity maps	(Jalilian et al., 24 May 2026)
Action segmentation	Structured verb–object–target–tool descriptions	(Zheng et al., 29 May 2026)
Local correspondence	DINOv2 semantic descriptors fused with local texture descriptors	(Cadar et al., 2024)
Trajectory prediction	Next Key Point as a semantic navigational intent variable	(Gan et al., 26 Jan 2026)

In robotics, SCOPE replaces discrete category labels with continuous DINOv2 semantics inside a diffusion-based NOCS predictor. Semantics are injected through cross-attention at the fifth down-sampling and second up-sampling blocks, and the recovered NOCS image is registered with TEASER++ to estimate $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$0, $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$1, and $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$2. On REAL275, SCOPE achieves 5°5cm = 49.2%, 10°5cm = 73.4%, and IoU75 = 73.7%, a relative improvement of 31.9% over DPDN on 5°5cm; beyond category-level evaluation, cross-attention with DINOv2 improves YCB-V ADD-S from 59.5% to 76.0% and TYOL ADD-S from 60.6% to 82.5% compared with concatenation plus labels (Hönig et al., 29 Sep 2025).

In dual-hand action segmentation, Polyphony encodes each action class with a structured sentence of the form “Action verb is $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$3; manipulated object is $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$4; target object is $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$5; tool is $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$6,” embeds that sentence with MiniLM-L6, and trains a TCN to align visual features to the semantic embedding. The resulting Motion-Action-Semantic feature $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$7 is then passed to a diffusion-based segmentation head. On HA-ViD, adding semantic features improves left-hand accuracy from 55.8 to 57.1 and right-hand accuracy from 58.3 to 60.6 relative to motion-plus-action features alone, with especially strong gains on visually similar action pairs (Zheng et al., 29 May 2026).

In long-horizon vessel prediction, SKETCH conditions an autoregressive decoder on a Next Key Point that represents high-level navigational intent. The model factorizes

$$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$8

with $$A = \mathrm{softmax}(QK^\top/\sqrt d), \qquad O = AV,$$9 estimated by retrieval-augmented contrastive verification. On the private test set, SKETCH achieves MSEP 0.41, MSEC $t$0, and MFD 7.80, outperforming TrAISformer at 0.71, $t$1, and 19.78. This suggests that semantic feature conditioning is not limited to visual semantics; intent variables can act as semantic constraints on future trajectory support (Gan et al., 26 Jan 2026).

6. Empirical patterns, limitations, and design principles

Several recurring empirical patterns emerge. First, spatial alignment matters. GramSR reports that replacing text conditioning with dense DINOv3 features improves RealSR from 25.48 PSNR, 26.72 LPIPS, and 19.39 DISTS for the text-conditioned PiSA-SR baseline to 26.81 PSNR, 23.05 LPIPS, and 17.91 DISTS, while also improving FID to 100.71. Its conditioning ablation further shows that a fixed conditioning tensor gives PSNR 26.37 and LPIPS 25.19, a learnable conditioning tensor gives 26.26 and 25.09, and visual conditioning gives 26.81 and 23.05, indicating that dense, spatially aligned visual features outperform both fixed and learnable abstract tensors (D'Oronzio et al., 28 Apr 2026).

Second, lightweight trainable adapters on top of frozen semantic backbones are common. SCAR keeps DINOv2-B frozen and learns only the compressor and AR model updates; SCOPE keeps DINOv2 frozen; GramSR freezes SD v2.1, VAE, and DINOv3 while training LoRAs and a small MLP adapter; CLIP-Guided SAM fine-tunes CLIP vision attention blocks and SAM adapters but freezes CLIP text; Control-DINO freezes CogVideoX-5B-I2V and DINOv3 while training only the temporal adapter and residual branch (Jin et al., 18 Nov 2025, Hönig et al., 29 Sep 2025, D'Oronzio et al., 28 Apr 2026, Jalilian et al., 24 May 2026, Dominici et al., 2 Apr 2026). This suggests a general design pattern in which semantic feature conditioning is implemented as a thin trainable interface between a frozen semantic prior and a frozen or lightly adapted generative backbone.

Third, stronger semantic control often trades off against diversity or some fidelity metrics. In SCAR, 4× compression nearly matches 1× conditioning quality while reducing GPU memory by 23.9% and accelerating training by 1.42×, but 16× compression harms quality and consistency (Jin et al., 18 Nov 2025). In GramSR, increasing $t$2 improves LPIPS, DISTS, and FID but can slightly reduce PSNR and SSIM (D'Oronzio et al., 28 Apr 2026). In face diffusion, multi-conditioning improves constraint fidelity but lowers LPIPS diversity from 0.446 in mask-only conditioning to 0.432 in multi-conditioning (Giambi et al., 2023). In Control-DINO, high-dimensional DINO conditioning improves structure preservation but restricts generative freedom, as reflected in lower latent and CLIP variance (Dominici et al., 2 Apr 2026).

Fourth, conditioning quality is only as good as the semantic source. Poor grounding in SRSR can over-prune relevant tags or fail to isolate objects (Chen et al., 26 Oct 2025). CLIP-Guided SAM depends on the localization quality of CLIP similarity maps and reports that pre-finetuning CLIP alone harms final joint performance, while train–test prompt consistency is critical: training on ground-truth points but evaluating on CLIP-sampled points drops VOC 1/16 mIoU from 78.3 to 69.8 (Jalilian et al., 24 May 2026). REPA-G similarly depends on representation alignment between the feature extractor and the diffusion model; poor alignment yields noisy gradients and artifacts (Sereyjol-Garros et al., 3 Feb 2026).

A final, broader pattern is that semantic conditioning often improves the metrics most closely tied to semantic fidelity, while no-reference or purely local measures can be misleading. SRSR explicitly cautions that no-reference metrics often prefer hallucinated outputs (Chen et al., 26 Oct 2025). Polyphony shows that structured descriptors outperform naive class labels even when the latter are shorter and simpler, because semantic decomposition reduces ambiguity rather than merely increasing label information (Zheng et al., 29 May 2026). BCTR shows that aligning features to a CLIP semantic space improves zero-shot recall from 3.6/5.1 to 4.4/6.2 when combined with bidirectional conditioning, indicating that semantic regularization affects compositional generalization rather than only in-distribution fitting (Hao et al., 2024).

Taken together, these results position semantic feature conditioning not as a single mechanism but as a design principle: construct a semantically meaningful side representation, inject it at the locus where the backbone can use it, preserve or recover spatial and structural correspondence when the task requires locality, and regulate the strength and timing of conditioning so that semantic fidelity improves without collapsing diversity or destabilizing the underlying model.