---
title: 'CAPNET: Adaptive Correlation for Visual Recognition'
url: https://www.emergentmind.com/topics/correlation-adaptation-prompt-network-capnet
type: topic
---

# CAPNET: Adaptive Correlation for Visual Recognition

The Correlation Adaptation Prompt Network (CAPNET) is an end-to-end framework for long-tailed multi-label visual recognition, leveraging pre-trained vision-language models, specifically CLIP, to explicitly model label correlations and address head-tail data imbalance. CAPNET utilizes a graph convolutional network (GCN) for label propagation, learnable soft prompts for refined class embeddings, a distribution-balanced focal loss with class-aware re-weighting, test-time ensembling for robust inference, and parameter-efficient fine-tuning (PEFT) to prevent overfitting on scarce classes. The framework demonstrates substantial improvements over state-of-the-art baselines on benchmarks including VOC-LT, COCO-LT, and NUS-WIDE [2511.20641].

## 1. Architecture Overview

CAPNET is built atop the dual-encoder structure of CLIP, integrating both a vision encoder \(E_I(\cdot)\) (e.g., ResNet-50, ViT-Base/16) and a Transformer-based text encoder \(E_T(\cdot)\). The system processes an input image \(\boldsymbol{x}\) as follows:

- Learnable soft prompts \(\{\boldsymbol{t}_c\}_{c=1}^C\) for each class are fed into \(E_T\), producing initial per-class textual features \(\boldsymbol{F}_t = [\boldsymbol{f}_1,\dots,\boldsymbol{f}_C]^\top\).
- These features are refined via a three-layer GCN over a label correlation graph \(\mathcal{G}\), generating residuals \(\boldsymbol{H}_L\) for each class:
  $$
  \boldsymbol{F}_t^* = \boldsymbol{F}_t + \boldsymbol{H}_L = [\boldsymbol{f}_1^*,\dots,\boldsymbol{f}_C^*]^\top
  $$
- The image encoder outputs a visual latent \(\boldsymbol{v} = E_I(\boldsymbol{x})\). Per-class probabilities are computed via cosine similarity and a sigmoid activation:
  $$
  p(y_c\,|\,\boldsymbol{x}) = \sigma \left( \frac{\cos(\boldsymbol{v}, \boldsymbol{f}_c^*)}{\tau}\right)
  $$
  where \(\tau\) is a trainable temperature parameter.

Training employs a binary cross-entropy-style loss, modified to address label imbalance and correlation.

## 2. Semantic Label Correlation Modeling

CAPNET departs from conventional reliance on co-occurrence statistics—which are unreliable for tail classes—by explicitly constructing a semantic affinity matrix using CLIP's textual encoder. For each class \(c\), a prompt-encoded representation \(\boldsymbol{z}_c\) is generated from a template, such as “a photo of a [CLS]”.

The raw correlation matrix is obtained via cosine similarity:
$$
\mathcal{A}_{ij} = \cos(\boldsymbol{z}_i, \boldsymbol{z}_j)
$$
Self-loop and neighbor weights are balanced:
$$
a'_{ij} = 
\begin{cases}
\frac{s}{ \sum_{k\neq i} \mathcal{A}_{ik} } \mathcal{A}_{ij}, & i \neq j \\
1-s, & i=j
\end{cases}
$$
where \(s \in [0,1]\) controls self-loop strength. The final row-normalized correlation matrix, using softmax with temperature \(\tau'\):
$$
\tilde{\mathcal{A}}_{ij} = \frac{\exp(a'_{ij}/\tau')}{\sum_{k} \exp(a'_{ik}/\tau')}
$$

## 3. Graph Convolutional Network and Prompt Learning

Each class serves as a node in a three-layer GCN, with features propagated according to the learned label correlation graph:
$$
H^{(l+1)} = \sigma \left( \tilde{D}^{-1/2} \tilde{A} \tilde{D}^{-1/2} H^{(l)} W^{(l)} \right)
$$
where \(\tilde{A} = \tilde{\mathcal{A}} + I\), \(\tilde{D}_{ii} = \sum_j \tilde{A}_{ij}\), \(H^{(0)} = F_t\), and \(\sigma\) is the ReLU activation. Only prompt tokens, GCN, and adapter weights are updated during training; the remainder of the text encoder is frozen.

Class prompt vectors \(\boldsymbol{t}_c\) of length \(M\) (empirically, \(M=4\)) are initialized as
$$
\boldsymbol{t}_c = [V, \ldots, V]_{M\text{ tokens},\; \mathrm{[CLS]=class\ name}}
$$
with token vectors \(V \in \mathbb{R}^d\) being learnable.

## 4. Loss Function and Imbalance Mitigation

CAPNET employs a distribution-balanced focal loss tailored for long-tailed distributions. The class-specific re-weighting factor:
$$
r_c = \alpha + \sigma \left( \beta \left( \frac{1}{n_c / N} - \theta \right) \right)
$$
with \(n_c\) the number of positives for class \(c\) and \(N\) the dataset size. A margin term:
$$
v_c = \kappa \log \left( \frac{1}{n_c / N} - 1 \right)
$$
is incorporated into the logits.

The per-class loss for an image \(i\) and class \(c\):
$$
\ell_{cls}(y_c^i, z_c^i) = 
\begin{cases}
- r_c (1 - q_c^i)^\gamma \ln(q_c^i), & y_c^i = 1 \\
- \frac{r_c}{\zeta} (q_c^i)^\gamma \ln(1 - q_c^i), & y_c^i = 0
\end{cases}
$$
where \(q_c^i\) is the margin-adjusted, sigmoid-modulated logit, and \(\zeta\), \(\gamma\) are hyperparameters. The full loss is averaged across all images and classes.

Rare tail classes (\(n_c \ll N\)) receive higher \(r_c\), amplifying their impact during training, while head class penalization is moderated by the margin and sigmoid mapping.

## 5. Inference Robustness and Parameter-Efficient Fine-Tuning

For robust test-time prediction, CAPNET employs five-crop ensembling: the input is resized and five spatial crops are generated, with class probabilities averaged:
$$
\boldsymbol{p}_* = \frac{1}{5} \sum_{cr=1}^5 p\left(y\,|\,\boldsymbol{x}_*^{(cr)}\right)
$$

Parameter-efficient fine-tuning is realized via the AdaptFormer scheme in each ViT MLP block:
$$
h_\ell = h'_\ell + s\,\mathrm{ReLU}\left(\mathrm{LN}(h'_\ell) W_{\downarrow}\right) W_{\uparrow} + \mathrm{MLP}\left(\mathrm{LN}(h'_\ell)\right)
$$
where \(W_{\downarrow} \in \mathbb{R}^{d' \times \hat{d}}\), \(W_{\uparrow} \in \mathbb{R}^{\hat{d} \times d'}\) with \(\hat{d} \ll d'\). Only the adapter weights, prompts, and GCN parameters are trained. For ViT-Base/16, this requires approximately 6.6 million parameters versus 91.3 million for full fine-tuning.

## 6. Empirical Evaluation and Ablation

CAPNET delivers substantial improvements on long-tailed multi-label recognition. Key results:

| Benchmark   | Backbone      | Prior Best (mAP) | CAPNET (mAP) | CAPNET + TTE + PEFT (mAP) |
|-------------|--------------|------------------|--------------|---------------------------|
| VOC-LT      | ResNet-50    | 84.37 % (MLC-NC) | 87.46 %      | 88.42 %                   |
| COCO-LT     | ResNet-50    | 60.52 %          | 64.44 %      | 65.75 %                   |
| VOC-LT      | ViT-Base/16  | 87.88 % (LMPT)   | —            | **93.03 %**               |
| COCO-LT     | ViT-Base/16  | 66.19 % (LMPT)   | —            | **76.36 %**               |
| NUS-WIDE    | ViT-Base/16  | 57.29 % (finetune CLIP) | 60.34 %   | —                         |

Ablation studies reveal:

- Distribution-balanced loss \(\mathcal{L}_{cls}\) improves mAP by up to 1.47%.
- GCN-based correlation propagation yields further gains.
- Test-time ensembling (TTE) and PEFT both contribute incrementally, cumulatively resulting in state-of-the-art mAP (e.g., 93.03% / 76.36% on VOC-LT / COCO-LT).
- Prompt-based cosine correlation matrices outperform data-derived co-occurrence matrices (+1.2% VOC-LT, +2.3% COCO-LT).
- Optimal GCN self-loop weight \(s=0.3\), softmax temperature \(\tau' = 0.3\), and prompt length 4 initialized with “a photo of a [CLS]”.

## 7. Context and Impact

CAPNET establishes an integrated method for exploiting vision-language priors in long-tailed, multi-label settings by explicitly modeling inter-class label relationships and tailoring the optimization to class imbalance. It unifies prompt tuning, graph-based semantic propagation, distribution-aware losses, robust inference, and resource-efficient adaptation, achieving empirically verified, superior performance on established benchmarks. Contemporary analyses underscore the effectiveness of prompt-based semantic correlations over traditional data-driven label affinity schemes and demonstrate that PEFT with adapters can match or exceed full fine-tuning using an order of magnitude fewer trainable parameters [2511.20641].

Source: https://www.emergentmind.com/topics/correlation-adaptation-prompt-network-capnet