---
title: Classifier-based Feature Reconstruction (ClaFR)
url: https://www.emergentmind.com/topics/classifier-based-feature-reconstruction-clafr
type: topic
---

# Classifier-based Feature Reconstruction (ClaFR)

Classifier-based Feature Reconstruction (ClaFR) denotes a classifier design in which class evidence is computed from reconstruction quality in representation space rather than only from logits, cosine similarity, or input-space likelihood. In the most explicit usage, ClaFR is a **post-hoc OOD detector** that performs an orthogonal decomposition of the classifier’s weights to extract a **class-known subspace**, projects penultimate-layer features into that subspace, and uses the feature reconstruction error as the OOD score [2509.06988]. Closely related formulations reformulate few-shot classification as reconstruction of a query feature map from class support features [2012.01506], and adapt frozen CLIP by projecting class prototype features into the query image feature space to reconstruct the query image feature map [2601.20224]. A further extension reconstructs surrogate class prototypes from queried samples and counterfactuals using Wasserstein barycenters, yielding a prototype-based reconstruction of a target classifier’s decision surface [2512.10878].

## 1. Reconstruction as a classifier principle

The central ClaFR principle is the replacement of a “similarity-as-score” paradigm with a “reconstruction-as-score” paradigm. In FRN, the decision for each class is: “how well can I reconstruct this query’s feature map from that class’s support feature maps?” In FPL, each class is represented by a pool of spatial visual features, and classification is done by how well these prototypes can reconstruct the query’s feature map. In the OOD formulation named ClaFR, the classifier’s weight matrix induces a low-dimensional class-known subspace, and feature reconstruction error within that subspace becomes the confidence signal [2012.01506; 2601.20224; 2509.06988].

This suggests a broad technical usage in which ClaFR covers methods that classify, detect distribution shift, or reconstruct a target model by measuring how well class-specific structures reconstruct a query representation. The structures differ across works—support feature pools, CLIP class prototype features, singular-vector subspaces, or Wasserstein prototype distributions—but the operational rule remains reconstruction-based [2512.10878].

| Work | Reconstruction object | Decision signal |
|---|---|---|
| FRN | Query feature map \(Q\) from class support pool \(S_c\) | Negative mean squared reconstruction error |
| FPL | Query CLIP feature map \(M\) from class prototype features \(F_d\) | Negative average squared reconstruction error |
| ClaFR | Penultimate-layer feature \(z\) from class-known subspace \(U_M\) | Projection norm or reconstruction error |
| Counterfactual-aware prototypes | Input \(x\) against barycentric prototypes \(Q_0,Q_1\) | Nearest-prototype Wasserstein rule |

A common misconception is to equate reconstruction-based classifiers with autoencoder-style anomaly detectors. The OOD ClaFR paper explicitly distinguishes the two: traditional autoencoder-based OOD detection reconstructs in input space with an extra encoder–decoder network, whereas ClaFR reconstructs in feature space and the “decoder” is simply the orthogonal projector \(U_MU_M^T\) derived from classifier weights [2509.06988].

## 2. Closed-form feature map reconstruction in few-shot classification

“Few-Shot Classification with Feature Map Reconstruction Networks” formulates few-shot classification as a reconstruction problem in latent space. An episode has a set of classes \(C\) with \(|C| = n\), a support set of \(k\) labeled images per class, and query images drawn from the same classes. A convolutional backbone maps each image to a feature map \(F(x) \in \mathbb{R}^{h \times w \times d}\), flattened as \(Q \in \mathbb{R}^{r \times d}\) with \(r = h \cdot w\). For each class \(c\), all support feature vectors are pooled into a matrix \(S_c \in \mathbb{R}^{(kr) \times d}\), and the class score is the negative mean squared error between the query map and its class-conditional reconstruction [2012.01506].

The reconstruction mechanism is a closed-form ridge regression. Given \(Q\) and \(S_c\), the method seeks \(W \in \mathbb{R}^{r \times kr}\) such that \(W S_c \approx Q\), solving
\[
\bar W = \arg\min_{W} \Bigl\|Q - W S_c\Bigr\|_F^2 + \lambda \, \|W\|_F^2.
\]
The standard closed form is
\[
\bar W = Q S_c^T \bigl(S_c S_c^T + \lambda I\bigr)^{-1},
\qquad
\bar Q_c = Q S_c^T \bigl(S_c S_c^T + \lambda I\bigr)^{-1} S_c.
\]
When \(kr > d\), the Woodbury identity yields an equivalent \(d \times d\) inversion:
\[
\bar Q_c = \rho \, Q \bigl(S_c^T S_c + \lambda I\bigr)^{-1} S_c^T S_c.
\]
The class posterior is then
\[
P(y_q=c\mid x_q)
=
\frac{\exp(-\gamma \langle Q,\bar Q_c\rangle)}
{\sum_{c' \in C} \exp(-\gamma \langle Q,\bar Q_{c'}\rangle)},
\qquad
\langle Q,\bar Q_c \rangle = \frac{1}{r}\|Q-\bar Q_c\|_F^2.
\]

The method stabilizes the regression by scaling the regularizer with \(\frac{kr}{d}\),
\[
\lambda = \frac{kr}{d} e^{\alpha},
\]
and introduces a learned recalibration
\[
\rho = e^{\beta},
\]
so that the only learned parameters in the reconstruction head are the three scalars \(\alpha\), \(\beta\), and \(\gamma\). There is no class-specific learnable weight and no additional neural reconstruction module. The paper further uses episodic meta-training, an auxiliary loss
\[
L_{\text{aux}} = \sum_{i\in C} \sum_{\substack{j\in C \\ j\neq i}} \bigl\| \hat S_i \hat S_j^T \bigr\|_F^2,
\]
scaled by \(0.03\), and a pre-training stage with learnable “dummy” feature maps \(M_c\) in order to produce feature maps that are suitable for reconstructive classification [2012.01506].

Empirically, FRN is both accurate and efficient. On fine-grained 5-way benchmarks with ResNet-12, it reports, for example, **CUB** 1-shot \(83.55\%\) and 5-shot \(92.92\%\), and **Aircraft** 1-shot \(70.17\%\) and 5-shot \(83.81\%\). On **mini-ImageNet**, ResNet-12, 5-way, it reports 1-shot \(66.45\%\) and 5-shot \(82.83\%\); on **tiered-ImageNet**, 1-shot \(71.16\%\) and 5-shot \(86.01\%\), with a larger-resolution variant at \(86.89\%\). The paper also emphasizes substantial speed advantages over DeepEMD, especially in 5-shot settings [2012.01506].

## 3. Feature projection learning on frozen CLIP

“Feature Projection Learning for Better Vision-Language Reasoning” instantiates the same classifier-based feature reconstruction logic on top of a frozen CLIP backbone. The baseline CLIP classifier predicts a class for a query image \(x_q\) by cosine similarity between the image embedding \(f_v = E_v(x_q)\) and text embeddings \(f_{t_i} = E_t(\Pi_i)\). FPL replaces this with a reconstruction-based classifier: it uses CLIP’s visual encoder without the final attention pooling, denoted \(E_{vm}\), extracts a spatial query feature map \(M = E_{vm}(x_q) \in \mathbb{R}^{H \times W \times C}\), pools all spatial features from the \(N\) support images of class \(d\) into \(F_d \in \mathbb{R}^{NHW \times C}\), and asks how well \(F_d\) can reconstruct \(M\) [2601.20224].

The projection model is analytic rather than neural. It seeks a matrix \(\theta \in \mathbb{R}^{HW \times NHW}\) such that \(\theta F_d\) approximates \(M\), solving
\[
\theta^* = \underset{\theta}{\text{arg min}} \ \|M-\theta F_d\|^2 + \delta \|\theta\|^2,
\]
with closed-form solution
\[
\theta^*  = MF_d^T(F_dF_d^T+\delta I)^{-1},
\qquad
M_d^* = MF_d^T(F_dF_d^T+\delta I)^{-1}F_d.
\]
The class score is the normalized squared reconstruction error
\[
\langle M, M_d^* \rangle = \frac{1}{HW}\|M- M_d^*\|^2,
\]
and the reconstruction-based probability is
\[
P_R(y_q=d|x_q) = \frac{\exp{(-\epsilon\langle M,\bar M_d \rangle)} }
{\sum_{d'\in D} \exp{(-\epsilon\langle M,\bar M_{d'} \rangle)} }.
\]
Here \(\delta = e^{\mu}\) with \(\mu\) a learnable scalar, and \(\epsilon\) is a learnable temperature inspired by Wertheimer et al.’s feature map reconstruction networks. The paper explicitly states that “only \(\mu\) and \(\epsilon\) are needed to be updated by gradient descent” [2601.20224].

FPL does not discard CLIP’s original classifier. Instead, it adds the reconstruction-based prediction to the frozen text-image similarity prediction:
\[
P_{total}(y_q=d|x_q) = P_{CLIP}(y_q=d|x_q) + \eta P_R(y_q=d|x_q).
\]
The total loss is
\[
\mathcal{L}=\mathcal{L}_{ce} + \gamma \mathcal{L}_{po},
\]
where \(\mathcal{L}_{po}\) is a projection orthogonality loss encouraging reconstructed feature maps for different classes to be dissimilar when applied to the same query, and \(\gamma = 0.1\) in all experiments. The CLIP encoders \(E_v\), \(E_{vm}\), and \(E_t\) remain frozen; the manual prompt is “a photo of {class}”; training uses AdamW with initial learning rate \(1\times 10^{-3}\), cosine annealing, and a single NVIDIA RTX 3090 [2601.20224].

The paper presents FPL as extremely parameter-efficient and training-efficient. On 16-shot ImageNet, Table 4 reports **FPL: 20 epochs, 1 minute training, 0.001 GFLOPs, \(\sim 0.001\)M parameters, 66.68% accuracy**; **Tip-Adapter-F: 5 minutes, 0.030 GFLOPs, 16.3M parameters, 65.51% accuracy**; and **CoOp: 15h, \(>10\) GFLOPs, 0.01M parameters, 62.95% accuracy**. On domain generalization from 16-shot ImageNet to ImageNet-V2, -Sketch, -A, and -R, FPL reports **46.46** average accuracy, compared with **43.89** for TPT and **41.59** for zero-shot CLIP. The few-shot study further reports relative improvements over Tip-Adapter-F of **+5.1%** on UCF101, **+2.2%** on EuroSAT, **+4.5%** on FGVC-Aircraft, and **+4.2%** on DTD [2601.20224].

## 4. Post-hoc ClaFR for out-of-distribution detection

The work titled “Frustratingly Easy Feature Reconstruction for Out-of-Distribution Detection” uses the name **Classifier-based Feature Reconstruction (ClaFR)** directly. Its setting is a pre-trained classifier with feature extractor \(f:\mathcal{X}\to\mathbb{R}^D\), penultimate-layer feature \(z = f(x)\), and final linear classifier weights \(W \in \mathbb{R}^{D \times C}\). ClaFR performs SVD,
\[
W = U \Sigma V^T,
\]
selects the top-\(m\) left singular vectors
\[
U_M = [u_1,\dots,u_m] \in \mathbb{R}^{D \times m},
\]
and interprets them as a class-known subspace. The hyperparameter \(m\) is chosen by a cumulative explained variance ratio \(\alpha\), typically \(0.9\), as the smallest \(m\) such that
\[
\sum_{i=1}^m \sigma_i \;>\; \alpha \sum_{i=1}^D \sigma_i.
\]
At test time, the feature is optionally normalized, projected as \(zU_M\), reconstructed as \(zU_MU_M^T\), and scored by
\[
e(x) = -\| z U_M U_M^T - z \|_2
=
-\sqrt{\|z\|_2^2 - \|z U_M\|_2^2},
\]
or equivalently
\[
S(x) = \|z U_M\|_2.
\]
Large \(S(x)\) indicates ID, and small \(S(x)\) indicates OOD [2509.06988].

The method is notable for not requiring access to training data. The paper states that it only needs a pre-trained classifier, its weights, and penultimate-layer features, and “never touches the original training data.” This is positioned as particularly attractive in scenarios where training data cannot be stored or re-used due to privacy or legal constraints. The method is also one-time and lightweight: SVD of \(W\) is performed once, and inference requires only a matrix-vector multiplication and an L2 norm [2509.06988].

The main experimental setup includes **ImageNet-1K** as ID with OOD benchmarks **SUN**, **iNaturalist**, **Places**, **Textures**, and **ImageNet-O**; **ResNet-50** trained from scratch with contrastive loss on ImageNet-1K with **78.12% top-1 accuracy** and feature dimension \(D=2048\); **MobileNet-V2** with \(D=1280\); and CIFAR experiments with a **ResNet-34** trained under supervised contrastive learning for 1000 epochs, reaching **95.28%** on CIFAR-10 and **73.23%** on CIFAR-100 with \(D=512\). Evaluation uses **AUROC (AUC)** and **FPR@95%TPR** [2509.06988].

On **ImageNet-1K, ResNet-50**, the average across the five OOD sets is **89.32 AUC, 41.82 FPR** for ClaFR, compared with **86.99 AUC, 50.13 FPR** for Neco and **86.46 AUC, 48.76 FPR** for ASH. On **ImageNet-1K, MobileNet**, ClaFR reports **83.63 AUC, 54.21 FPR**. On the CIFAR benchmark, for **ID = CIFAR-10**, ClaFR reports **82.82 AUC, 68.41 FPR**; for **ID = CIFAR-100**, **96.29 AUC, 23.49 FPR**. In the computational comparison to KNN at ImageNet scale, ClaFR reports **\(\mathcal{O}(1)\)** per-sample complexity, **\(\approx 0.012\) ms** inference, and **\(\approx 12\) MiB** storage, compared with **\(\approx 8.26\) ms** and **\(\approx 20\) GiB** for KNN [2509.06988].

Geometrically, the method treats the classifier weight spectrum as a task-specific low-dimensional manifold in feature space. This suggests a ClaFR interpretation in which the classifier’s own geometry supplies the reconstruction basis, without any episode-specific support set or learned decoder [2509.06988].

## 5. Counterfactual-aware prototype reconstruction

“Classifier Reconstruction Through Counterfactual-Aware Wasserstein Prototypes” does not use the name ClaFR directly, but it “strongly overlaps conceptually with classifier-based feature reconstruction.” Its setting is black-box reconstruction of a binary probabilistic target classifier
\[
m : \mathbb{R}^d \to [0,1],
\qquad
\hat m(x) = \mathbf{I}_{[0,1]}[m(x) \ge 0.5],
\]
under a low-query regime of **300–500 queries**, using queried points, labels returned by \(m\), and counterfactuals generated by a mechanism \(g_m\). The counterfactual generator is one-sided: it is activated only when \(m(x)<0.5\) and produces a counterfactual that flips the decision to class 1. The paper’s central performance measure is fidelity,
\[
\text{Fid}_{m, D_{\text{ref}}}(\hat n)
=
\frac{1}{|D_{\text{ref}}|} \sum_{x \in D_{\text{ref}}}
\mathbf{I}_{[0,1]}[\hat m(x) = \hat n(x)].
\]
Counterfactuals are treated as “soft samples for both classes” by assigning label \(0.5\) in the extended label space \(\mathcal{Y}=\{0,0.5,1\}\) [2512.10878].

For each class \(c \in \{0,1\}\), the paper defines a soft prototype distribution \(Q_c\) as a Wasserstein barycenter between the class distribution \(P_c\) and the counterfactual distribution \(P_{\text{cf}}\):
\[
Q_c = \arg\min_{Q \in \mathcal{P}(\mathcal{X})}
\left(
W_2(Q, P_c) + \lambda_c W_2(Q, P_{\text{cf}})
\right),
\]
with \(\lambda_c = 0.5\) in the experiments. To mitigate decision boundary shift from one-sided counterfactuals, it adds a symmetry regularization term
\[
R(Q_0, Q_1) = \left( W_2(Q_0, P_{\text{cf}}) - W_2(Q_1, P_{\text{cf}}) \right)^2,
\]
and optimizes
\[
\min_{Q_0, Q_1}
\sum_{c \in \{0,1\}}
\bigl(
W_2(Q_c, P_c) + \lambda W_2(Q_c, P_{\text{cf}})
\bigr)
+ \gamma R(Q_0,Q_1),
\]
with \(\gamma = 0.3\) in experiments. Classification is then a nearest-prototype rule in Wasserstein space using distances from the Dirac measure \(\delta_x\) to \(Q_0\) and \(Q_1\), with margin \(\tau\) [2512.10878].

The paper works on **Adult Income**, **HELOC**, **COMPAS**, and **DCCC**, with logistic regression target models trained on unknown training data. The proposed method is compared to a model-extraction baseline that treats counterfactuals as normal labeled instances, and to a neural surrogate with **Counterfactual Clamping** loss. On the **Adult** dataset, the fidelity results reported in Table 1 are: at **500 queries**, **Baseline 1: \(91 \pm 3.2\)**, **Baseline 2: \(94 \pm 3.2\)**, **Ours: \(96 \pm 2.5\)**; at **300 queries**, **Baseline 1: \(87 \pm 3.8\)**, **Baseline 2: \(90 \pm 3.8\)**, **Ours: \(93 \pm 3.2\)**. The paper states that similar patterns hold on COMPAS, DCCC, and HELOC, with gains most pronounced at 300 queries [2512.10878].

This line of work broadens the ClaFR perspective from reconstructing query features to reconstructing class-conditional prototype distributions and decision boundaries. The paper explicitly argues that high-quality, realistic, on-manifold counterfactuals lead to higher fidelity, and that low-capacity prototype-based surrogates remain more stable than more complex neural surrogates in low-data regimes [2512.10878].

## 6. Comparative properties, misconceptions, and limitations

Across these works, ClaFR is not a single algorithm but a recurring design pattern. FRN and FPL classify by reconstructing a query feature map from class-specific support features or class prototype features; the OOD ClaFR method scores a feature by its reconstruction within a class-known subspace induced by classifier weights; and the counterfactual-aware prototype method reconstructs class representations as Wasserstein barycenters and uses those prototypes to approximate a target classifier’s boundary [2012.01506; 2601.20224; 2509.06988; 2512.10878].

Several common properties follow directly from the cited formulations. First, reconstruction is usually **analytic rather than amortized**: FRN uses closed-form ridge regression; FPL computes \(\theta^*\) analytically per query; OOD ClaFR uses the orthogonal projector \(U_MU_M^T\); and the barycentric variant uses optimal transport prototype optimization rather than a learned decoder. Second, the learnable overhead can be very small: FRN uses only \(\alpha,\beta,\gamma\) in the reconstruction head; FPL updates only \(\mu\) and \(\epsilon\); and OOD ClaFR performs no retraining at all [2012.01506; 2601.20224; 2509.06988]. Third, the data requirements differ sharply: OOD ClaFR requires no training data, FRN and FPL require support features, and the counterfactual-aware method requires queried samples plus counterfactuals [2509.06988; 2012.01506; 2601.20224; 2512.10878].

The main misconceptions are likewise clarified by the papers. ClaFR is not necessarily an input-space reconstruction model; in the OOD setting it is explicitly a feature-space subspace projection method, and in FRN/FPL it reconstructs spatial feature maps rather than pixels [2509.06988; 2012.01506; 2601.20224]. ClaFR is also not tied to a single application domain: the literature spans few-shot classification, vision-language adaptation, post-hoc OOD detection, and black-box model reconstruction [2601.20224; 2509.06988; 2512.10878].

The limitations are equally consistent. FRN identifies **linearity**, **dependence on feature quality**, and the need for careful regularization when \(kr \ge d\), since support features may span or nearly span \(\mathbb{R}^d\) [2012.01506]. OOD ClaFR identifies **dependence on classifier quality**, **assumption of linear subspace structure**, **sensitivity to architecture**, and the absence of explicit OOD examples [2509.06988]. The counterfactual-aware prototype method identifies **sensitivity to counterfactual quality** and the computational cost of optimal transport [2512.10878]. FPL’s ablations show that learning the ridge penalty is crucial: on 16-shot ImageNet, **FPL full: 66.68%**, **w/o PO: 66.51%**, **fixed \(\delta\): 62.65%**, **w/o PO + fixed \(\delta\): 62.56%** [2601.20224].

The open directions listed in the papers point toward a more general ClaFR research program. FRN proposes **nonlinear reconstruction**, **multi-layer reconstruction**, **learned class-specific bases**, **hybrid metric + reconstruction**, and **task-dependent regularization** [2012.01506]. The OOD ClaFR paper proposes extensions to **Transformers**, **large foundation models**, **non-linear subspaces**, **combination with other scores**, and **better theoretical understanding** [2509.06988]. FPL suggests that frozen VLM backbones can support extremely parameter-efficient reconstruction-based classifiers, especially when combined with a text-based prior and orthogonality regularization [2601.20224]. The counterfactual-aware prototype work suggests that counterfactuals are best used as boundary information rather than as hard-labeled training points, and that prototype distributions can reconstruct both class structure and decision boundaries under strict query budgets [2512.10878].

Taken together, these results suggest a unifying view: classifier-based feature reconstruction treats class discrimination as a question of whether a query representation lies in, near, or can be reconstructed from a class-conditioned subspace, dictionary, or prototype distribution. The precise reconstruction object varies, but the score is consistently induced by reconstruction quality, projection geometry, or prototype proximity in a representation space shaped by the task.

Source: https://www.emergentmind.com/topics/classifier-based-feature-reconstruction-clafr