Training-Free Label-Space Alignment

• Training-free label-space alignment is a set of methods that enhance model predictions by manipulating label tokens and related structures without updating model parameters.
• These techniques include prompt-based label distribution enhancement, domain adaptation with refined label vocabularies, and geometric adapter merging via null-space compression.
• Applications span in-context visual classification and LLM preference alignment, yielding measurable improvements on datasets like ImageNet and Office31.

Training-free label-space alignment denotes a class of methods that improve prediction, transfer, or behavioral control by manipulating, refining, or conditioning on label-related structures without updating the core model parameters. In the cited literature, this idea appears in prompt-level reshaping of the label space for in-context visual classification, in filtering and refinement of discovered class names for universal domain adaptation, and, in adjacent training-free alignment work, in output-agnostic adapter merging and retrieval-conditioned preference alignment that avoid conventional supervised post-training (Chen et al., 2023, Lee et al., 22 Sep 2025, Lee et al., 27 Mar 2026, Halloran, 11 May 2026).

1. Conceptual scope and problem setting

In "Manipulating the Label Space for In-Context Classification" (Chen et al., 2023), label-space alignment is defined operationally as shaping the textual context, specifically the set of label tokens presented in the in-context examples, so that the induced probability mass is better aligned toward the true class. In "Training-Free Label Space Alignment for Universal Domain Adaptation" (Lee et al., 22 Sep 2025), the same phrase refers to aligning label spaces instead of visual spaces by filtering and refining noisy labels between domains and then constructing a universal classifier from the refined set. These formulations share a common constraint: the alignment step itself avoids parameter updates in the primary model.

A useful distinction is between explicit label-space alignment and adjacent training-free alignment. The former directly edits or refines class labels or class-name prompts. The latter does not necessarily manipulate class labels, but it uses training-free signals to alter the effective decision structure of a model. "Label-Free Cross-Task LoRA Merging with Null-Space Compression" (Lee et al., 27 Mar 2026) is label-free and output-agnostic rather than label-space based; "Leveraging RAG for Training-Free Alignment of LLMs" (Halloran, 11 May 2026) is preference alignment rather than class-label alignment. This suggests a broader research pattern in which alignment is increasingly performed by prompt design, retrieval, or geometric surrogates rather than by additional gradient-based training.

Setting	Core mechanism	Training status
In-context classification	Manipulate ICE label tokens and descriptions	Prompt-only
Universal domain adaptation	Filter and refine discovered target labels	Training-free up through label discovery
Cross-task LoRA merging	Minimize average null-space ratio	No joint multi-task training
Preference alignment	Retrieve preferred and dispreferred responses	Online, training-free

A recurrent misconception is to equate training-free with optimization-free or auxiliary-model-free. The surveyed methods instead replace conventional fine-tuning with prompt assembly, retrieval, label discovery, or coefficient optimization over frozen components.

2. Prompt-level manipulation of the label space in in-context classification

The in-context classification setting of (Chen et al., 2023) casts prediction in a vision-LLM as conditional text generation. Given in-context examples $\{(I_1,L_1),\dots,(I_n,L_n)\}$ and a test image $\hat I$, the model forms the multimodal sequence $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$ and autoregressively produces the token sequence of the predicted class name. A candidate class $c$ is scored by length-normalized log-probability,

$$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$

The paper’s central intervention is to replace each single ICE label with a richer label-space representation, thereby increasing the knowledge density of each example.

The first strategy, Label Distribution Enhancement (LDE), uses CLIP similarities between an ICE image $I$ and the full candidate label set $\mathcal{C}=\{L_1,\dots,L_M\}$ to construct a soft label distribution over the top-$k$ labels. The distribution is

$$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$

Three prompt-only variants are defined: Equidistributed Label (EL), Distributed Label (DL), and Descriptive Distribution (DD). EL assigns equal mass to the ground-truth label and its top $(k-1)$ neighbors; DL uses the raw softmax weights; DD injects numerical weights into natural language, for example, “This image is 70% ‘sparrow’, 20% ‘finch’, 10% ‘warbler’.”

The second strategy, Visual Descriptions Enhancement (VDE), augments the ICE label token with a short natural-language description of discriminative visual features. For each ICE $\hat I$0, the frozen VLM is prompted with “What are the most useful visual features to distinguish a $\hat I$1 in this image?” and the generated description is appended to the label. No parameters are updated; only the prompt text is enriched.

Empirically, these prompt-only manipulations improve performance on ImageNet, CUB-200, Stanford Dogs, and Stanford Cars under 1-shot, 2-shot, and 4-shot settings. On ImageNet, accuracy increases from $\hat I$2 in a 4-shot setting to $\hat I$3 with just 2 shots, surpassing CLIP by $\hat I$4. On CUB-200, 1-shot accuracy rises from $\hat I$5 to $\hat I$6, which is $\hat I$7 higher than CLIP. The paper also reports that Retrieval In-Context Example Selection consistently outperforms Random Sampling, that DL and DD yield the best 1-shot gains, and that DD saturates beyond 2-shot, likely due to prompt-length and model-capacity limits (Chen et al., 2023).

These results position label-space manipulation as a substitute for simply increasing the number of in-context examples. Rather than extending context length, the method densifies the semantic content of each example.

3. Label-space alignment for universal domain adaptation

In Universal Domain Adaptation, the source domain is labeled, the target domain is unlabeled, and the label spaces may differ. The formulation in (Lee et al., 22 Sep 2025) distinguishes shared classes $\hat I$8, source-private classes $\hat I$9, and target-private classes $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$0. The goal is to build a universal classifier that recognizes shared classes, rejects source-private classes, and discovers target-private classes.

The method begins with CLIP’s frozen image and text encoders. For a label name $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$1, a prompt such as “a photo of a backpack.” is encoded as

$\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$2

while an image $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$3 is encoded as

$\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$4

The zero-shot logit is then $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$5. The difficulty is that the target label space is not known in advance.

To discover target-private labels, the method queries BLIP-VQA for each unlabeled target image using $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$6 paraphrased prompts, discards any answer that is not a single noun, and takes a majority vote to produce a raw discovered label

$\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$7

The resulting open vocabulary contains synonyms of source classes, hypernyms or hyponyms, and pure noise. TLSA then applies a three-stage training-free filtering pipeline.

The first stage is Synonym Label Alignment, which removes any discovered label whose WordNet path similarity to a source label exceeds $\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$8. The second stage is Semantic Label Alignment, which embeds source labels and filtered discovered labels jointly, computes cosine similarities to target-image embeddings, forms a top-$\mathcal{S}=\{(I_1,L_1);\dots;(I_n,L_n);\hat I\}$9 prediction set using adaptive thresholds $c$0 and $c$1, and updates a frequency bank depending on whether the sample appears known or unknown. The third stage is Frequency-Based Noise Filtering, which sets $c$2 with $c$3 and retains only labels whose support exceeds that threshold.

After filtering, the universal classifier is constructed with weight matrix

$c$4

and prediction is obtained by $c$5. If the top-1 class lies in $c$6, the sample is assigned to the unknown bucket. This is described as a threshold-free decision rule separating shared and private categories.

Evaluation uses the H-score

$c$7

where $c$8 is shared-class accuracy and $c$9 is target-private-to-unknown accuracy, and the H$$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$0-score

$$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$1

Across Office31, Office-Home, VisDA, and DomainNet, and across open-partial, open, closed, and partial splits, the method reports an average improvement of $$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$2 in H-score and $$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$3 in H$$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$4-score over prior methods; adding self-training yields an additional $$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$5 in both metrics. The paper also reports that semantic alignment outperforms naive top-1 or fixed-margin baselines by $$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$6–$$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$7, and that removing frequency filtering causes a collapse in private-class precision (Lee et al., 22 Sep 2025).

The significance of this line of work is its inversion of the usual UniDA emphasis. Previous methods primarily focused on visual space alignment; TLSA instead treats label discovery and label refinement as the principal adaptation mechanism.

4. Output-agnostic alignment through null-space compression in LoRA merging

A related but distinct development appears in (Lee et al., 27 Mar 2026), which addresses model merging rather than label refinement. In LoRA, each layer’s update is $$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$8, where $$s(c,\hat I)=\frac1{l(c)}\sum_{t=1}^{l(c)}\log P(w_t\mid \mathcal{S},w_{<t}) .$$9 is the down-projection and $I$0 is the up-projection. Because $I$1, $I$2 discards components lying in its null space. For an incoming activation $I$3, the sample-level null-space ratio is

$I$4

equivalently

$I$5

The paper’s key empirical claim is that the average null-space ratio

$I$6

steadily decreases over finetuning on both classification and regression tasks, and that this decrease closely tracks reduced validation loss and increased task accuracy. Lower $I$7 is associated with higher accuracy both before and after merging. This observation motivates Null-Space Compression (NSC) Merging, which learns per-task, per-layer merge coefficients $I$8 in

$I$9

by minimizing the average null-space ratio:

$\mathcal{C}=\{L_1,\dots,L_M\}$0

NSC is explicitly label-free and output-agnostic. It relies only on adapter geometry $\mathcal{C}=\{L_1,\dots,L_M\}$1 and unlabeled inputs, and it does not require logits or probability distributions. As a result, it extends naturally to regression, and its cost is independent of sequence length because the computation occurs at target layers on the input side rather than at every output token. To make the method practical, it caches the small $\mathcal{C}=\{L_1,\dots,L_M\}$2 matrices $\mathcal{C}=\{L_1,\dots,L_M\}$3 and reduces the extra cost to $\mathcal{C}=\{L_1,\dots,L_M\}$4 per layer per batch, with $\mathcal{C}=\{L_1,\dots,L_M\}$5 typically $\mathcal{C}=\{L_1,\dots,L_M\}$6–$\mathcal{C}=\{L_1,\dots,L_M\}$7.

The empirical results are broad. Across twenty heterogeneous vision tasks on ViT-B, NSC achieves an average normalized score of $\mathcal{C}=\{L_1,\dots,L_M\}$8, compared with $\mathcal{C}=\{L_1,\dots,L_M\}$9 for RobustMerge. On NYUD Segmentation it reaches $k$0, versus $k$1–$k$2 for Task Arithmetic and TIES; on PASCAL Parts it reaches $k$3 versus $k$4–$k$5; and on Taskonomy all 11 tasks are reported at approximately $k$6 of finetuned performance. On six NLI benchmarks with LLaMA-3-8B + LoRA-16, the average normalized accuracy is $k$7. On six vision-language benchmarks with LLaVA-7B + LoRA-16, the average is $k$8, matching or exceeding full-token AdaMerging at $k$9 while costing one-seventh the time. On VLM compute, NSC preparation is reported as approximately $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$0 min, optimization as $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$1 min, validation as approximately $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$2 min, and peak memory as approximately $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$3 GB on an A6000; full-token entropy-based merging requires $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$4 min (Lee et al., 27 Mar 2026).

Although this work is not framed as label-space alignment, it demonstrates that training-free alignment can be generalized beyond label spaces to geometric properties of task-specific adapters.

5. Retrieval-conditioned preference alignment as an adjacent training-free paradigm

"Leveraging RAG for Training-Free Alignment of LLMs" (Halloran, 11 May 2026) addresses a different setting: preference alignment and refusal guardrails for LLMs. Its proposed method, RAG-Pref, is online and training-free. The pipeline uses an embedding model $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$5, a preferred-response index $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$6, a dispreferred-response index $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$7, and an off-the-shelf autoregressive LLM. At test time, for a query $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$8, the method computes $$d_I^{L_j} = \begin{cases} \displaystyle\frac{\exp(\mathrm{sim}(I,L_j)/\tau)} {\sum_{m\in\mathrm{Top}_k(I)}\exp(\mathrm{sim}(I,L_m)/\tau)} & j\in\mathrm{Top}_k(I) \ 0 & \text{otherwise.} \end{cases}$$9, retrieves top-$(k-1)$0 preferred and dispreferred examples by cosine similarity,

$(k-1)$1

constructs a prompt instructing the model to follow the preferred examples and avoid the dispreferred examples, and then generates a response from the frozen LLM.

The method’s contrastive structure is central. Conditioning on both preferred and dispreferred examples yields a distribution $(k-1)$2, and the paper states that adding the contrastive set $(k-1)$3 can only reduce the model’s entropy further. An optional refusal rule defines

$(k-1)$4

and overrides the generation with a fixed refusal response if $(k-1)$5.

The computational comparison is explicit. Offline DPO or SafeDPO requires back-propagation over preference pairs, whereas RAG-Pref only computes embeddings once and then performs two vector lookups and one long-prompt LLM call. On an Nvidia L40S with DeepSeek-R1-14B, DPO training time is reported as approximately $(k-1)$6 hrs, while RAG-Pref preparation is approximately $(k-1)$7 hrs. For inference over 115 samples, a DPO-aligned model without RAG takes $(k-1)$8 hrs, OPAD takes $(k-1)$9 hrs and $\hat I$00 GB, and RAG-Pref takes $\hat I$01 hrs and $\hat I$02 GB. The paper summarizes this as approximately $\hat I$03 faster preparation than DPO and $\hat I$04 faster inference than OPAD while using $\hat I$05 less memory.

On agentic falsely benign attacks, base models refuse $\hat I$06–$\hat I$07 of attacks, DPO or SafeDPO alone raises refusal to approximately $\hat I$08–$\hat I$09, RAG-Pref on base models reaches approximately $\hat I$10–$\hat I$11, and RAG-Pref combined with DPO or SafeDPO reaches approximately $\hat I$12–$\hat I$13, corresponding to an average $\hat I$14 factor improvement across five widely used LLMs. On AlpacaEval 2 and MT-Bench, it improves over standard RAG and OPAD, and Table 4 reports an additional $\hat I$15 mutual information over standard RAG (Halloran, 11 May 2026).

This is not label-space alignment in the narrow sense of class-name refinement. A plausible implication is that training-free alignment methods increasingly rely on conditioning over curated symbolic spaces—class labels in vision tasks, discovered label vocabularies in adaptation, or preferred/dispreferred response sets in LLM alignment.

6. Limitations, open questions, and boundary conditions

The prompt-based label-space methods of (Chen et al., 2023) rely exclusively on prompt manipulation, with no parameter fine-tuning or embedding updates. The experiments were conducted on a mid-sized 3B-parameter VLM, and the paper notes that scaling to much larger models or more in-context examples may affect prompt-length generalization. It also identifies future directions including adaptation to soft prompts or embedding spaces, dynamic prompt-length calibration for many-shot settings, and joint optimization of retrieval, label-distribution construction, and visual-description selection.

The UniDA framework of (Lee et al., 22 Sep 2025) is training-free up through label discovery, but it can be extended with a teacher-student self-training phase using cross-entropy over pseudo-labels. This clarifies an important boundary condition: training-free may apply to the core alignment stage even when a later refinement stage introduces learning. The method also depends on the quality of generative label discovery and on the effectiveness of WordNet synonym filtering, semantic selection, and frequency-based pruning in separating true target-private classes from noise and semantic ambiguity.

The NSC merging framework of (Lee et al., 27 Mar 2026) depends on the assumption that all task-specialized adaptations uniformly compress null spaces. The paper states that tasks with near-orthogonal latent demands might break this trend. It does not explicitly address multi-task preference or Pareto optimality, and it identifies the optimal choice of target layers $\hat I$16 and rank selection as open problems. It also states that the theoretical reason null-space compression generalizes across very different modalities remains open.

The adjacent preference-alignment setting of (Halloran, 11 May 2026) shows that training-free alignment can preserve or improve general preference performance while strengthening refusal behavior, but it also makes clear that the method depends on an embedding model, vector indexes, retrieval quality, and longer prompts. This further undermines the common misconception that training-free implies a trivial or zero-overhead pipeline.

Taken together, these works suggest that training-free label-space alignment is best understood not as a single algorithmic recipe but as a family of non-parametric control strategies. In the strict sense, it includes prompt-level reshaping of class labels and discovery-and-refinement of cross-domain label vocabularies. In a broader and adjacent sense, it includes geometric and retrieval-based mechanisms that alter model behavior without conventional fine-tuning. The common research question is how much alignment can be achieved by operating on labels, prompts, retrieved exemplars, or frozen adapter geometry rather than on the model weights themselves.