CapRecover: Semantic Feature Inversion

• CapRecover is a cross-modality feature inversion attack framework that recovers semantic information (image labels or captions) from intermediate visual features in split-DNN deployments.
• The framework employs a three-stage pipeline—including feature projection, feature-text alignment, and caption generation/classification—to directly map visual features to textual semantics.
• Empirical results show strong semantic recovery from deeper network layers, while a reversible Gaussian noise defense effectively mitigates the privacy leakage risk.

CapRecover is a cross-modality feature inversion attack framework on vision-LLMs that targets split-DNN deployments in which a visual encoder on a user device transmits intermediate features to the cloud. Rather than reconstructing pixels, it recovers semantic information—specifically image labels or full captions—directly from leaked intermediate visual features. The framework is motivated by the observation that intermediate representations in practical edge-cloud inference pipelines can encode high-level semantic content sufficient to expose private information, especially in deeper layers of the visual encoder (Xiu et al., 30 Jul 2025).

1. Definition and threat model

CapRecover is formulated for the setting in which a visual encoder such as ResNet, ViT, or MobileNet processes an input image locally and outputs an intermediate representation $\mathbf{F}$ that is transmitted to a server or consumed by downstream modules. The attacker does not need the raw image $I$, the ground-truth caption $T_{\text{cap}}$, or the true class label $y_{\text{cls}}$; access to the leaked intermediate representation is sufficient:

$\mathbf{F}=\mathcal{V}_{\text{Image}}(I).$

The central privacy claim is that these intermediate features are not merely low-level edge or texture descriptors. In deeper layers especially, they encode objects, scenes, and relations that correlate strongly with textual descriptions. CapRecover therefore attacks semantic leakage directly, bypassing the conventional strategy of feature-to-image reconstruction followed by semantic inference. The paper explicitly contrasts this with prior approaches that reconstruct images from intermediate features but often obtain blurry, semantically ambiguous outputs (Xiu et al., 30 Jul 2025).

This threat model is particularly relevant to split inference and edge-cloud vision-language systems, where the division of computation between device and server may create an attack surface at the feature-transmission boundary. A plausible implication is that privacy risk in such systems persists even when raw images never leave the client device.

2. Architecture and feature-to-text inversion pipeline

CapRecover is organized as a three-stage pipeline: feature projection, feature-text alignment, and caption generation or classification. Its defining idea is to learn a mapping from intermediate visual features to textual semantics through feature-to-text alignment rather than through image synthesis (Xiu et al., 30 Jul 2025).

The first stage is the Feature Projection Module, which converts victim-model features of heterogeneous dimensionality into a unified representation. If the victim feature is already a vector $\mathbf{F}_i\in\mathbb{R}^d$, the projection is linear:

$$\mathbf{F}_i^{\text{proj}}=\mathbf{W}_p\mathbf{F}_i+\mathbf{b}_p,\quad \mathbf{W}_p\in\mathbb{R}^{d'\times d},\; \mathbf{b}_p\in\mathbb{R}^{d'}.$$

If the feature is spatial, $\mathbf{F}_i\in\mathbb{R}^{C\times H\times W}$, CapRecover uses a ResNet-based projection function $g(\cdot)$:

$$\mathbf{F}_i^{\text{proj}}=\mathbf{W}_p\cdot g(\mathbf{F}_i)+\mathbf{b}_p,\quad g:\mathbb{R}^{C\times H\times W}\to\mathbb{R}^{d'}.$$

The paper states that $I$0 is used as the unified feature size in many settings (Xiu et al., 30 Jul 2025).

The second stage is the Feature-Text Alignment Module, implemented with a Q-Former and trainable query tokens $I$1. During training, the ground-truth caption is tokenized into embeddings $I$2, and the aligned latent is computed as

$I$3

with $I$4. This output is then mapped into the LLM embedding space:

$I$5

The third stage depends on the semantic target. For caption recovery, the framework uses a pretrained OPT model as a frozen LLM and autoregressively generates text:

$I$6

For label recovery, the same front-end is retained, but the LLM is replaced by a linear classifier. The paper states that a simple classifier is sufficient to obtain strong results (Xiu et al., 30 Jul 2025).

3. Training objective, model choices, and evaluation setup

CapRecover freezes the LLM parameters and trains only the feature projection module and alignment module. The Q-Former is pretrained, and the LLM is pretrained OPT (Xiu et al., 30 Jul 2025). Caption generation uses cross-entropy loss:

$I$7

while label recovery uses standard classification cross-entropy:

$I$8

The evaluation spans five datasets. Caption reconstruction is tested on COCO2017, Flickr8K, and ImageNet-1K. Label recovery is tested on CIFAR-10 and TinyImageNet. For ImageNet-1K, captions are generated using Qwen2.5 because the dataset lacks human-annotated captions; the paper reports that 12,000 training images and 1,000 test images are sampled (Xiu et al., 30 Jul 2025).

The victim models include CLIP ViT-B/16, CLIP ViT-B/32, ResNet50, ResNet101, MobileNetV2, and MobileNetV3. For captioning, the attack is evaluated on multiple internal layers: ViT base/no-proj, ResNet layer1 through layer4 and base, and MobileNet base. For label recovery, the experiments focus on final-output features from ResNet50 and CLIP ViT-B/32 (Xiu et al., 30 Jul 2025).

The captioning metrics are BLEU-1 to BLEU-4, METEOR, ROUGE-L, CIDEr, SPICE, and cosine similarity in an embedding space. The paper emphasizes ROUGE-L as the primary caption metric and states that scores above 0.3 indicate moderate success and scores above 0.5 indicate strong semantic recovery. For cosine similarity, values above 0.7 are described as successful. Label recovery is evaluated with Top-1 accuracy, Top-5 accuracy, and per-class precision, recall, and F1 for CIFAR-10 (Xiu et al., 30 Jul 2025).

4. Empirical performance on caption and label recovery

The principal empirical finding is that CapRecover can recover semantics effectively from intermediate features, especially for stronger visual encoders and deeper layers (Xiu et al., 30 Jul 2025).

On COCO2017, the reported caption-recovery results are strong across several victim models. CLIP ViT-B/16 achieves BLEU-1 0.72, ROUGE-L 0.53, and cosine similarity above 0.7 for 84.38% of outputs. CLIP ViT-B/32 obtains BLEU-1 0.70, ROUGE-L 0.53, and cosine similarity 80.38%. ResNet50 reaches BLEU-1 0.70, ROUGE-L 0.52, and cosine similarity 76.84%, while ResNet101 attains BLEU-1 0.70, ROUGE-L 0.53, and cosine similarity 79.98% (Xiu et al., 30 Jul 2025).

On Flickr8K, performance drops substantially. The paper attributes this to the dataset’s small size and shorter, more semantically sparse captions. For CLIP ViT-B/16, BLEU-1 is 0.30, ROUGE-L is 0.27, and cosine similarity is 22.40%; ResNet50 achieves BLEU-1 0.28 and ROUGE-L 0.25, with MobileNet models performing still worse (Xiu et al., 30 Jul 2025).

On ImageNet-1K, despite the absence of native captions, the framework still recovers meaningful text. CLIP ViT-B/16 reports BLEU-1 0.45, ROUGE-L 0.41, and cosine similarity 40.78%; ResNet50 yields BLEU-1 0.42 and ROUGE-L 0.38; CLIP ViT-B/32 obtains BLEU-1 0.44 and ROUGE-L 0.40 (Xiu et al., 30 Jul 2025). This suggests that classification-oriented encoders can preserve caption-relevant semantics in their intermediate representations.

The weakest caption results are associated with MobileNetV2/V3. On COCO2017, ROUGE-L is reported as only about 0.31 and cosine similarity is near zero to a few percent. The paper explains this by reference to MobileNet’s efficiency-oriented design and reduced representational richness (Xiu et al., 30 Jul 2025).

For label recovery, the results are similarly strong. On CIFAR-10, ResNet50 features yield Top-1 83.35% and Top-5 99.55%, while CLIP ViT-B/32 features reach Top-1 92.71% and Top-5 99.82%. On TinyImageNet, ResNet50 achieves Top-1 60.13% and Top-5 83.79%, whereas CLIP ViT-B/32 reaches Top-1 72.62% and Top-5 91.60% (Xiu et al., 30 Jul 2025).

The CIFAR-10 classwise analysis further reports that per-class F1 scores are mostly above 0.90. Especially strong classes are Automobile 0.97, Horse 0.96, Ship 0.96, and Truck 0.96, while Cat is the hardest class at F1 0.85. The confusion matrix is described as strongly diagonal, which the paper takes as evidence that labels are reconstructed accurately (Xiu et al., 30 Jul 2025).

5. Semantic depth and layerwise leakage

A major analytical result is that layer depth strongly affects semantic leakage. The deeper the layer, the more semantic information is encoded, and the easier it is to recover captions or labels from the leaked representation (Xiu et al., 30 Jul 2025).

For ResNet50 on COCO2017, the reported layerwise progression is explicit. At layer1, CapRecover obtains BLEU-1 0.24, CIDEr 0.19, and cosine similarity approximately 0.00. At layer2, BLEU-1 rises to 0.51, CIDEr to 0.31, and cosine similarity to 43.76%. At layer3, BLEU-1 is 0.58, CIDEr 0.55, and cosine similarity 31.42%. At layer4, BLEU-1 reaches 0.62, CIDEr 0.68, and cosine similarity 85.64%. At the base feature, BLEU-1 is 0.70, CIDEr 0.90, and cosine similarity 90.52% (Xiu et al., 30 Jul 2025).

For CLIP ViT-B/16, the paper reports no-proj BLEU-1 0.69, CIDEr 0.90, and cosine similarity 93.04%, compared with base BLEU-1 0.72, CIDEr 0.99, and cosine similarity 94.76% (Xiu et al., 30 Jul 2025).

The qualitative interpretation provided in the paper is that shallow layers attend mostly to edges and textures, whereas deeper layers focus on semantically meaningful regions such as objects and actions. The visualizations reportedly show that captions generated from deeper ResNet layers improve progressively from vague or nonsensical outputs to captions closely matching the ground truth (Xiu et al., 30 Jul 2025).

This depth dependence is central to the security significance of CapRecover. A plausible implication is that privacy risk in split-DNN systems is not uniform across split points: earlier partitioning may reduce semantic exposure relative to deeper-feature transmission, although the paper does not present that as a deployment prescription.

6. Defense through reversible Gaussian noise injection

To mitigate semantic leakage, the paper proposes a simple protection mechanism for split-DNN settings: adding random Gaussian noise to intermediate features locally and then removing the same noise before the next layer. For a feature $I$9, the client samples

$T_{\text{cap}}$0

forms the transmitted feature

$T_{\text{cap}}$1

and reconstructs the clean value before further local processing:

$T_{\text{cap}}$2

The paper stresses that the noise is generated and canceled entirely on the client side, so it is never transmitted or stored remotely. The stated properties of the defense are that it requires no retraining, no changes to final model predictions, negligible computation overhead, and no communication overhead. The extra per-inference time overhead is reported as less than 1% (Xiu et al., 30 Jul 2025).

On COCO2017 with ResNet50, the defense sharply reduces attack quality on deeper layers. Without noise, the layerwise BLEU-1 values are layer1 0.24, layer2 0.51, layer3 0.58, and layer4 0.62. With noise, the corresponding values are layer1 0.49, layer2 0.03, layer3 0.02, and layer4 0.05 (Xiu et al., 30 Jul 2025). The paper identifies the layer1 outcome as somewhat atypical, but characterizes the overall trend as a substantial disruption of CapRecover at the layers where semantic leakage is strongest.

The defense therefore targets the specific attack surface exploited by CapRecover: semantically informative transmitted tensors. This suggests a design strategy in which privacy protection is implemented at the representation interface rather than through post hoc filtering of textual outputs.

7. Significance, scope, and limitations

CapRecover’s main contribution is to reframe feature inversion as semantic inversion. The framework argues that privacy risk in split vision-language systems cannot be reduced to the possibility of reconstructing visually recognizable images. Intermediate features can expose object categories and semantically faithful descriptions even when no pixel-level reconstruction is attempted (Xiu et al., 30 Jul 2025).

The paper’s broader implication is that intermediate visual representations should not be treated as intrinsically safe merely because they are not human-interpretable images. The reported results show recovery of labels with very high accuracy, captions with strong lexical and semantic overlap, and semantically faithful descriptions from layers several stages before the final output (Xiu et al., 30 Jul 2025).

The stated limitations are also important. The experiments are confined to particular datasets and victim models, and the defense is evaluated only through the reported noise-injection mechanism. The paper does not claim universal attack success across all architectures or deployment settings. It also reports weaker performance on Flickr8K and substantially weaker leakage from MobileNetV2/V3, indicating that semantic recoverability depends materially on model family, representation depth, and dataset characteristics (Xiu et al., 30 Jul 2025).

Within the literature represented in the source material, CapRecover is best understood as a privacy analysis of split VLM inference that shifts the emphasis from image reconstruction fidelity to direct semantic extraction. Its central result is that feature transmission alone can constitute a high-capacity leakage channel, and that this leakage can be operationalized through a lightweight feature-to-text bridge without access to the original image (Xiu et al., 30 Jul 2025).