Variational Feature Compression for Model-Specific Representations

Abstract: As deep learning inference is increasingly deployed in shared and cloud-based settings, a growing concern is input repurposing, in which data submitted for one task is reused by unauthorized models for another. Existing privacy defenses largely focus on restricting data access, but provide limited control over what downstream uses a released representation can still support. We propose a feature extraction framework that suppresses cross-model transfer while preserving accuracy for a designated classifier. The framework employs a variational latent bottleneck, trained with a task-driven cross-entropy objective and KL regularization, but without any pixel-level reconstruction loss, to encode inputs into a compact latent space. A dynamic binary mask, computed from per-dimension KL divergence and gradient-based saliency with respect to the frozen target model, suppresses latent dimensions that are uninformative for the intended task. Because saliency computation requires gradient access, the encoder is trained in a white-box setting, whereas inference requires only a forward pass through the frozen target model. On CIFAR-100, the processed representations retain strong utility for the designated classifier while reducing the accuracy of all unintended classifiers to below 2%, yielding a suppression ratio exceeding 45 times relative to unintended models. Preliminary experiments on CIFAR-10, Tiny ImageNet, and Pascal VOC provide exploratory evidence that the approach extends across task settings, although further evaluation is needed to assess robustness against adaptive adversaries.

• The paper introduces a variational latent bottleneck framework that uses KL regularization and classification loss to selectively compress features for a designated classifier.
• It employs dynamic binary masking based on gradient saliency and per-dimension KL divergence to effectively suppress transfer to unintended models.
• Empirical evaluations on CIFAR-100, Tiny ImageNet, and Pascal VOC demonstrate significant cross-model transfer suppression while maintaining competitive target classifier accuracy.

Variational Feature Compression for Model-Specific Representations

Motivation and Problem Scope

Shared and cloud-based inference platforms (MLaaS) expose input data to potential misuse via input repurposing—where data intended for one task is re-utilized by unauthorized models for alternative tasks, e.g., facial images submitted for gender classification exploited for identity inference. Conventional privacy defenses (homomorphic encryption, secure multi-party computation, differential privacy) restrict data access but fail to directly control the residual utility of outputs for unintended downstream models. The principal objective of this work is to design a transformation pipeline that maximizes utility for a designated classifier while empirically suppressing cross-model transfer, thus enforcing selective utility in feature representations.

Figure 1: Overview of the proposed selective-utility inference framework showing variational bottleneck, dynamic masking, decoder, and frozen target classifier evaluation.

Framework and Methodology

Variational Latent Bottleneck

The central architectural element is a variational latent bottleneck, inspired by the Variational Information Bottleneck (VIB) paradigm [alemi2016vib]. Inputs are encoded into a latent representation parameterized by $(\mu, \log \sigma^2)$ and sampled via reparameterization. The encoder loss consists purely of the downstream task's cross-entropy evaluated by a frozen target classifier, coupled with a KL regularization term penalizing divergence from the isotropic Gaussian prior. Notably, the decoder is supervised exclusively by classification loss, with no pixel-level reconstruction objective, resulting in representations biased toward task-aligned cues, not visual fidelity. This enables decoupling latent compression from visual preservation, effectively restricting information to the minimum necessary for the designated classifier.

Figure 2: Information Bottleneck view of the proposed variational latent bottleneck, enforcing retention of task-relevant information via controllable compression--utility trade-off $\lambda$.

Saliency-Guided Dynamic Feature Masking

To further refine latent selectivity, the method introduces a dynamic binary masking stage. Each latent dimension is scored by (i) per-dimension KL divergence, and (ii) gradient-based saliency (average absolute gradient of task loss with respect to latent component). These scores are normalized and combined with a tunable parameter $\gamma$, forming a unified importance measure. Thresholding yields a binary mask, retaining only dimensions with high task-relevance and statistical deviation from the prior. The mask is periodically recomputed during training to adapt to evolving feature importance. This dual criterion addresses KL's insufficiency in capturing task relevance and more aggressively eliminates dimensions that promote transferability.

Figure 3: Feature filtering pipeline: encoding, KL and saliency scoring, normalization, and binary mask selection.

Figure 4: Latent masking mechanism combining KL divergence and task-loss saliency for dimension-wise importance estimation and suppression.

Training Procedure and System Architecture

Training proceeds in two phases: latent-space warmup without masking (to stabilize task relevance), followed by masking phase with periodic mask updates. Encoder and decoder are optimized jointly, while the target model remains frozen—requiring white-box access during training for gradient computation, but only a forward pass at inference. The mask, encoder, and decoder are tightly bound to a particular target architecture, highlighting the model-specific character of the defense.

Empirical Evaluation and Numerical Results

CIFAR-100: Cross-Model Transfer Suppression

The method is systematically evaluated on CIFAR-100 with ResNet152, DenseNet121, ConvNeXt-V2, and VGG16 as target and unintended models. Integrated dynamic masking (KL and saliency, $\gamma=0.5$) yields up to 72.23% top-1 accuracy for the designated classifier (ConvNeXt-V2), while unintended models are suppressed below 1.6%, close to the 1% random baseline (100 classes). Suppression ratio exceeds 45× relative to unintended models. Notably, masking is not simply destructive: ablation analysis shows that adding saliency improves target accuracy while maintaining suppression, i.e., target accuracy rises to 70.12% for ResNet152 with mean unintended accuracy at 1.01%.

Figure 5: Classification accuracy heatmap on CIFAR-100: integrated masking yields pronounced diagonal contrast, demonstrating selective utility and reduced cross-model transfer.

Cross-Task Transfer and Extensions

On Tiny ImageNet, processed inputs retain classification utility for ResNet-50 (60.81%) but severely degrade the output of a style transfer network, substantiating suppression of generative semantic information. Preliminary experiments reveal strong suppression of fine-grained classifier accuracy on CIFAR-10 when trained for binary label discrimination (multiclass accuracy falls to 17.64%, random baseline 10%). The pipeline also preserves multi-label utility on Pascal VOC (Top-1: 61.27%, Top-5: 90.18%), but does not yet establish quantitative suppression of unintended models.

Figure 6: Original input image for reference in cross-task transfer evaluation.

Implications and Theoretical Interpretation

The method achieves empirical selective utility—processed representations are optimized for one classifier's decision rule while rendered non-transferable to other models, even those trained for identical tasks. The drop in target accuracy (from 88.34% to 72.23% for ConvNeXt-V2) is interpreted as the necessary trade-off for enforcing non-reusability; overly natural representations would lead to higher cross-model transfer. The defense is inherently target-specific, demanding retraining upon change of protected model. It addresses off-the-shelf reuse, not formal privacy guarantees or adaptive adversarial retraining scenarios, which remain open for further research.

The explicit mapping of the privacy--utility frontier (target accuracy vs. transfer suppression) is underspecified and merits systematic exploration. Comparison against prior attribute suppression or statistical privacy schemes (e.g., DPFE [osia2020deep], adversarial feature extraction [ding2020privacy]) is non-trivial due to divergent objectives.

Limitations and Practical Deployment Considerations

• White-box Requirement: Gradient access to the target classifier is essential during training to compute saliency; the method is unsuitable for pure black-box MLaaS environments.
• Model Dependence: Suppression mechanism is bound to a specific classifier; generality across architectures is not achieved.
• Empirical Commitment: No formal privacy guarantees; results are valid for a closed set of unintended models, not for adaptive adversaries.
• Extension Potential: Preliminary evidence supports efficacy for cross-task suppression and multi-label settings, but robustness and generalization remain to be systematically validated.

Conclusion

Variational feature compression with selective utility masking provides a principled empirical framework for enforcing model-specific representations in cloud inference scenarios. By combining VIB-style latent space encoding with gradient-informed masking, it achieves strong classifier utility while suppressing input reuse across architectures. The method is practically relevant for scenarios where input repurposing is a privacy concern and white-box access is available. Future directions include adversarial robustness evaluation, systematic privacy--utility frontier characterization, and extension to broader modalities and task formats.

(2604.06644)