---
title: Semantic Orthogonal Calibration (SoC)
url: https://www.emergentmind.com/topics/semantic-orthogonal-calibration-soc
type: topic
---

# Semantic Orthogonal Calibration (SoC)

Semantic Orthogonal Calibration (SoC) is a Huber-based regularizer designed for test-time prompt tuning (TPT) in vision-language models (VLMs), with the principal aim of improving the calibration of uncertainty estimates while maintaining discriminative performance. SoC enforces smooth prototype separation that respects semantic proximity, addressing the shortcomings of previous orthogonality-based regularization techniques that induce overconfidence by artificially separating semantically related classes. The method has demonstrated empirical state-of-the-art calibration across diverse classification benchmarks [2601.08617].

## 1. Theoretical Motivation and Problem Statement

In the context of TPT, uncertainty calibration is critical for robust deployment in sensitive domains such as healthcare and autonomous driving. Standard prompt tuning minimizes the softmax entropy of class predictions, frequently causing the model to become overconfident. Recent approaches, notably O-TPT (Sharifdeen et al., CVPR'25), introduce a quadratic full-orthogonality regularizer:

\[
\mathcal L_{\rm ortho} = \big\|S - I_K\big\|_F^2 = \sum_{i\neq j} s_{ij}^2
\]

where \(S_{ij} = t_i^\top t_j\) denotes the cosine similarity between text prompt prototypes \(t_i, t_j\). This penalty enhances class separability but unduly pushes apart even those classes that are semantically similar (e.g., “dog” vs. “puppy”). For collinear prototypes (\(s_{ij} \approx 1\)), the gradient step induced by this loss is proportional to \(-4\eta s_{ij}\), amplifying the semantic drift and inflating prediction confidence, particularly for ambiguous samples.

## 2. SoC Regularizer: Mathematical Formulation

SoC introduces a Huber-based regularization that caps prototype repulsion based on semantic margins derived from class name similarities. The Huber function with threshold \(\delta > 0\) is defined as:

\[
H_\delta(x) =
\begin{cases}
\tfrac12\,x^2, & |x| \le \delta \\
\delta \Big(|x| - \tfrac{\delta}{2} \Big), & |x| > \delta
\end{cases}
\]

Class margin \(m_{ij}\) is set via pre-computed semantic similarities \(s_{ij}\) (such as cosine similarity between frozen CLIP name embeddings):

\[
m_{ij} = 1 - s_{ij}
\]

For prompt prototypes \(p_i \in \mathbb{R}^d\), SoC applies the pairwise regularizer:

\[
\mathcal L_{\rm SoC} = \sum_{i<j} H_\delta\big(\|p_i-p_j\|_2 - m_{ij}\big)
\]

If two classes are highly similar (\(s_{ij}\) is large), their semantic margin \(m_{ij}\) is small, so their embeddings are allowed to be close—minor deviations incur mild penalty, and only excessive collapse/trivial separation triggers linear repulsion.

## 3. Combined Prompt-Tuning Objective

At test time, the SoC regularizer is incorporated alongside the cross-entropy loss on pseudo-labels:

\[
\mathcal L = \mathcal L_{\rm CE} + \lambda \mathcal L_{\rm SoC}
\]

where \(\lambda > 0\) governs the trade-off between discriminative adaptation and semantic calibration.

## 4. Implementation and Algorithmic Workflow

Test-time SoC tuning proceeds as follows (single AdamW step):

```python
# Inputs:
# {xₙ}: Unlabeled test batch
# {pᵢ}: Initial prompts
# [mᵢⱼ]: Margin matrix
# δ: Huber threshold
# λ: Regularizer weight

1. For each xₙ, compute vₙ = f_θ(xₙ) // image embedding
2. Compute logits zₙᵏ = (1/T)·vₙᵀ pₖ; softmax pₙᵏ
3. Form pseudo-labels ŷₙ from highest-confidence pₙᵏ (e.g., top-ρ% entropies)
4. Compute L_CE = −∑ₙ log pₙ^{ŷₙ}
5. Compute pairwise Dᵢⱼ = ‖pᵢ − pⱼ‖₂ and L_SoC = ∑_{i<j} H_δ(Dᵢⱼ − mᵢⱼ)
6. Compute L = L_CE + λ L_SoC
7. Update prompts: pᵢ ← pᵢ − η ∂L/∂pᵢ  (AdamW step)
# Output: Refined {pᵢ}
```

Notable implementation aspects include the use of ViT-L/14 or ViT-B/16 encoder backbones; learning rate 0.005; batch size 64 (with heavy augmentation); temperature \(T\) fixed by CLIP; \(\delta\) set to the 20th percentile of zero-shot cosine distances; and typical \(\lambda\) values around 30 (or 14 under distribution shift).

## 5. Empirical Evaluation

SoC was validated across eleven classification benchmarks—including ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, SUN397, FGVC-Aircraft, DTD, UCF101, EuroSAT—and four distribution-shift variants (ImageNet-A, v2, R, Sketch). The following Table summarizes accuracy and Expected Calibration Error (ECE) for ViT-L/14:

| Method       | Avg. Acc. (%) | Avg. ECE (%) |
|--------------|---------------|--------------|
| Zero-Shot    | 71.1          | 5.1          |
| TPT          | 72.0          | 14.9         |
| C-TPT        | 72.1          | 10.0         |
| O-TPT        | 71.4          | 7.7          |
| SoC          | **72.3 (+0.9)** | **5.4 (–2.3)** |

Qualitative reliability diagrams indicate SoC undershoots the confidence–accuracy gap relative to O-TPT, particularly on high-similarity class pairs (e.g., EuroSAT). Under natural distribution shifts, SoC matches O-TPT’s accuracy but lowers ECE by approximately 1.5%, and is less sensitive to multi-step prompt updates. The approach is robust to CLIP prompt template variations and supervised CoOp prompt initializations. Post-hoc SaLS calibration further reduces ECE for all methods, with SoC maintaining the lowest error.

## 6. Core Insights and Practical Recommendations

SoC’s Huber-based, margin-aware repulsion preserves semantic alignment learned by the VLM backbone, mitigating the confidence inflation and semantic drift induced by quadratic orthogonality penalties. Theoretical results demonstrate that SoC’s margin-respecting separation yields smoother reduction in worst-case cosine similarity, which translates to improved calibration empirically.

### Hyperparameter Guidance

- **δ (Huber threshold):** Set to a low percentile (10–30%) of zero-shot cosine distances; decrease for fine-grained tasks.
- **λ (regularizer weight):** Typically 20–50; increasing λ enhances calibration at the expense of minor accuracy drops. Cross-validate using held-out, unlabeled batches (monitoring ECE vs. accuracy).

### Extensions

Potential extensions include learning adaptive semantic margins from external knowledge graphs (e.g., WordNet), leveraging a small labeled seed, multi-step TPT with dynamic λ or δ scheduling, and integration with complementary post-hoc calibration schemes.

## 7. Relation to Prior Work and Future Perspectives

SoC builds upon O-TPT and related orthogonality- and cross-modal prompt-tuning literature. It specifically challenges the assumption that full prototype separation always benefits calibration, offering a theoretically and empirically grounded alternative that respects semantic structure. Future work may explore adaptive margin learning, integration with external knowledge sources, or joint optimization with supervised prompt initializations.

In sum, Semantic Orthogonal Calibration (SoC) provides a margin-aware, semantically-conscious regularization strategy for test-time prompt tuning. It achieves state-of-the-art calibration—measured by reduced ECE and improved reliability—across standard VLM benchmarks, without compromising discriminative accuracy [2601.08617].

Source: https://www.emergentmind.com/topics/semantic-orthogonal-calibration-soc