Targeted Overfitting in ML Optimization

• Targeted overfitting is a selective adaptation where optimization aggressively fits specific submodels, samples, or features, leading to overly optimistic in-sample results.
• It arises when data is reused for both model search and evaluation, causing discrepancies between training performance and true generalization metrics.
• Mitigation strategies include disjoint data splits, layer-specific regularization, and bilevel optimization to curb localized overfitting effects.

Targeted overfitting, as the term is used across several adjacent research areas, does not denote a single universally standardized object. Taken together, the literature supports a broader interpretation: overfitting can be concentrated on a specified target—such as a selected submodel, object class, client, feature component, or evaluation protocol—so that optimization yields unusually favorable in-sample behavior on that target while degrading generalization, calibration, robustness, or privacy elsewhere (Hong et al., 2017, Zhang et al., 2024, Khalil et al., 20 Nov 2025). Under this view, the defining feature is selective adaptation. The resulting pathology is not merely a global train–test gap, but a localized failure induced by reusing the same data for search and assessment, by fitting a narrow subset of samples or classes too aggressively, or by optimizing internal control variables that encode task-specific structure more strongly than the available data can justify.

1. Statistical foundations and the logic of selective fit

A foundational formulation of targeted overfitting appears in post-selection regression. In the Gaussian linear model $y = X\beta + \sigma \varepsilon$, model selection over subsets $S \subseteq \{1,\dots,p\}$ using an information criterion

$\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$

creates a targeted search for a submodel with exceptionally favorable residual fit on the same sample later used for uncertainty quantification. The selected model

$\hat S = \arg\min_S \gamma_n(S)$

is therefore not a fixed model in the classical sense. When $\hat S \supset S^\star$, the reduction in residual sum of squares is search-enhanced rather than typical of a prespecified larger model. The paper formalizes this by showing that if

$1-\exp(-a_n D_n)> D_n,$

then the post-selection residual standard deviation estimate is necessarily downward biased relative to the oracle model: $\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$ The mechanism is sample reuse plus targeted fit optimization, and the inferential consequence is anti-conservative intervals and p-values (Hong et al., 2017).

A closely related statistical perspective defines overfitting directly as a discrepancy between training empirical risk and population risk at the data-dependent fitted hypothesis $y_{\mathbf S}$. In that formulation, $y_{\mathbf S}$ $\epsilon$-overfits if

$S \subseteq \{1,\dots,p\}$0

Because the model is chosen using $S \subseteq \{1,\dots,p\}$1, the usual concentration logic that would justify evaluating a fixed hypothesis on training data no longer applies at the random point $S \subseteq \{1,\dots,p\}$2. With an independent holdout sample $S \subseteq \{1,\dots,p\}$3, the discrepancy

$S \subseteq \{1,\dots,p\}$4

admits the bound

$S \subseteq \{1,\dots,p\}$5

under the null that the fitted model $S \subseteq \{1,\dots,p\}$6-generalizes. This gives a formal test for training-sample overfitting and, more generally, clarifies why any evaluation sample that influences model choice can become a target of over-optimization rather than a valid measure of generalization (Schmidt, 2023).

These two analyses establish the core principle that recurs in later work: once optimization is allowed to adapt to a specific sample, split, or target variable, the resulting fit can become systematically too optimistic for that target. This suggests that targeted overfitting is often best understood as a dependence phenomenon rather than merely as large capacity in the abstract.

2. Sample-, class-, and direction-selective mechanisms

Several papers show that overfitting is frequently concentrated on particular subsets of examples rather than distributed uniformly across the data. In deep classifiers trained with softmax and cross-entropy, continued gradient updates amplify logits even after classification accuracy stabilizes. Because cross-entropy is scale-sensitive, that amplification lowers loss on already correctly classified examples while increasing loss on incorrectly classified ones. The paper explicitly reports that validation-loss increase is carried by the incorrectly classified subset, whereas the correctly classified subset continues to improve. In this sense, overfitting is sample-selective: it is concentrated on unstable, ambiguous, or persistently misclassified examples rather than on the entire validation distribution (Salman et al., 2019).

The same logic appears in adversarial training, where the problematic subset is the set of currently misclassified samples. Standard adversarial training generates perturbations by maximizing loss with respect to the true label for every example. For a misclassified sample, however, this moves the example farther into the wrong region rather than toward the decision boundary, creating boundary-far, high-loss training points that induce sharp local boundary deformations. Accuracy Aware Adversarial Training changes the inner maximization to use the predicted label $S \subseteq \{1,\dots,p\}$7 for misclassified points, while the outer loss still uses the true label. The resulting perturbation

$S \subseteq \{1,\dots,p\}$8

is designed to move misclassified examples toward the boundary instead of away from it, reducing a specific sample-level source of adversarial overfitting (Altinisik et al., 2022).

A privacy-oriented variant of this phenomenon appears in membership inference beyond global overfitting. Even well-generalized models can leak information about a small subset of training examples. The most vulnerable points are reported to be outliers within their classes, often located near class-cluster boundaries, hard to classify, noisy, or otherwise atypical. On CIFAR-10 with DenseNet-12, the paper shows that a RegDrop model with $S \subseteq \{1,\dots,p\}$9 train accuracy and $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$0 test accuracy still yields attack AUC $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$1 and attacker advantage $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$2, demonstrating that modest global train–test gaps do not preclude sample-specific leakage. A targeted logit-reweighting defense then reduces AUC from $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$3 to $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$4 and advantage from $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$5 to $\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$6 without changing accuracy, reinforcing the idea that overfitting and memorization can be localized to atypical points rather than reflected only in aggregate metrics (Khalil et al., 20 Nov 2025).

At a more abstract level, benign overfitting provides a direction-selective version of the same idea. In overparameterized linear regression, the standard risk

$\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$7

can vanish while the adversarial risk

$\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$8

remains large because adversarial risk depends not only on prediction error under the data distribution but also on the estimator norm. The paper shows that

$\gamma_n(S)= n\log\{\mathrm{SSE}(S)\} + c_n |S|$9

and that in the benign-overfitting regime the ridgeless interpolator can satisfy $\hat S = \arg\min_S \gamma_n(S)$0 while $\hat S = \arg\min_S \gamma_n(S)$1. This is not target-class-specific overfitting, but it is overfitting concentrated in low-variance directions that standard risk barely penalizes and an adversary can directly exploit (Hao et al., 2024).

3. Task- and object-targeted finetuning

A particularly explicit use of the term occurs in downstream adaptation of foundation models. In BLO-SAM, downstream semantic segmentation is posed as a collection of binary tasks, one per target object or class, such as segmenting lungs in radiographs or polyps in endoscopy. Standard SAM is promptable rather than autonomous, so it cannot determine which object is intended without user input. BLO-SAM replaces manual prompts with a learned prompt embedding $\hat S = \arg\min_S \gamma_n(S)$2 that encodes the target object or task itself. The paper’s main claim is that ordinary SAM adaptation overfits in the few-shot regime because both the mask-decoder parameters $\hat S = \arg\min_S \gamma_n(S)$3 and the prompt representation $\hat S = \arg\min_S \gamma_n(S)$4 are optimized on the same tiny sample, especially under domain shift from general imagery to medical imaging (Zhang et al., 2024).

The anti-overfitting mechanism is structural. The few-shot training set is split into two disjoint subsets $\hat S = \arg\min_S \gamma_n(S)$5 and $\hat S = \arg\min_S \gamma_n(S)$6. The lower level learns $\hat S = \arg\min_S \gamma_n(S)$7 on $\hat S = \arg\min_S \gamma_n(S)$8,

$\hat S = \arg\min_S \gamma_n(S)$9

while the upper level learns the task prompt on held-out data,

$\hat S \supset S^\star$0

The loss is a weighted combination of cross-entropy and Dice loss with tradeoff parameter $\hat S \supset S^\star$1. The paper emphasizes that no additional explicit regularizer is introduced; the regularization is induced by separating the optimization of weights and prompt across disjoint data splits. This makes the prompt function like a hyperparameter chosen for generalization rather than for in-sample co-adaptation.

The empirical evidence is correspondingly targeted. Figure 1 shows that SAMed and Med-SA exhibit large train–test Dice gaps from roughly epochs 60 to 100, whereas BLO-SAM shows a much smaller gap. In a direct few-shot comparison on human body segmentation with 4 labeled examples, full finetuning of all SAM parameters yields $\hat S \supset S^\star$2 Dice while BLO-SAM yields $\hat S \supset S^\star$3, which the paper attributes to severe overfitting in the unrestricted finetuning regime. The same structure appears in ablations: joint optimization of $\hat S \supset S^\star$4 and $\hat S \supset S^\star$5 on $\hat S \supset S^\star$6 is worse than bilevel optimization with separate data splits, and second-order bilevel optimization is best but more expensive. In this setting, targeted overfitting means over-specialization to a task-specific object prompt learned on the same examples used to tune the task model.

4. Objective- and representation-targeted manifestations

In self-supervised contrastive learning, overfitting can be localized not to a class or sample subset alone, but to one component of the training objective. In SimCLR experiments on a four-class CIFAR-10 subset using ResNet18, training loss continues to decrease for up to 5K epochs while validation loss begins increasing after about 500 epochs. Decomposing the InfoNCE loss reveals that the deterioration is driven by the positive-pair term,

$\hat S \supset S^\star$7

while the negative-similarity term continues to improve on validation data. The authors interpret this as training-instance-specific invariance learning: the model becomes increasingly good at recognizing positive pairs for training images but loses the ability to align positive pairs from unseen validation images (Rabin et al., 2024).

Targeted transfer attacks show a related overfitting dilemma in representation space. Simple iterative targeted adversarial attacks can achieve near-perfect white-box targeted success yet transfer poorly to unseen models, which the paper interprets as overfitting of the adversarial example to the source model. The proposed remedy is a second-stage feature-space fine-tuning that encourages target-class features and suppresses original-class features in a middle layer. Using

$\hat S \supset S^\star$8

with $\hat S \supset S^\star$9, the method optimizes

$1-\exp(-a_n D_n)> D_n,$0

Only a few fine-tuning iterations are reported to improve targeted transfer substantially, which suggests that the original failure lies in overly source-specific output-space optimization rather than insufficient perturbation magnitude (Zeng et al., 2024).

A more semantic form of targeted overfitting appears in multi-targeted attacks conditioned on class embeddings. Cross-domain multi-targeted attack work argues that prior methods overfit to the training label space because they rely on labels, logits, CLIP text embeddings, or other class-specific semantic coordinates. In the paper’s overfitting diagnostic, adversarial examples aimed at 800 unseen ImageNet classes are disproportionately redirected into the 200 ImageNet classes seen during training: CGNC and GAKer send about $1-\exp(-a_n D_n)> D_n,$1 of misclassified images into known classes, versus about $1-\exp(-a_n D_n)> D_n,$2 for random perturbations. CD-MTA reduces this effect by replacing class-level conditioning with target-image conditioning and class-agnostic feature losses,

$1-\exp(-a_n D_n)> D_n,$3

thereby reducing overreliance on known class semantics (Gonçalves et al., 27 May 2025).

5. Detection, auditing, and client-side verification

Because targeted overfitting is often localized, several papers propose diagnostics that look beyond a single final train–validation gap. One broad trajectory-level measure is the Overfitting Index,

$1-\exp(-a_n D_n)> D_n,$4

which aggregates the stronger of the loss gap or accuracy gap at each epoch and weights later discrepancies more heavily. In experiments on the Breast Ultrasound Images Dataset, OI differentiates architecture-specific and setup-specific overfitting: for example, MobileNet without augmentation has OI $1-\exp(-a_n D_n)> D_n,$5, whereas U-Net without augmentation has OI $1-\exp(-a_n D_n)> D_n,$6, and augmentation lowers OI for all reported models. The method is heuristic and unnormalized, but it formalizes the idea that overfitting can be targeted to specific architectures, datasets, and training regimens rather than treated as a single global property (Aburass, 2023).

In deep reinforcement learning, the most informative diagnostic in the paper is not return, not Q-gap, and not training TD error, but held-out TD error on a validation buffer of transitions. The critic loss is

$1-\exp(-a_n D_n)> D_n,$7

and the paper argues that poor sample efficiency in high-UTD off-policy RL is associated with driving down this loss on the training replay while allowing held-out TD error to remain high. AVTD therefore trains several regularized agents on a shared replay buffer and lets the agent with the lowest validation TD error act in the environment most of the time, while maintaining a held-out buffer by collecting one held-out episode every 10 episodes. Across 9 DMC and 4 Gym tasks, AVTD achieves the best average rank, $1-\exp(-a_n D_n)> D_n,$8, outperforming fixed regularizers and selection by Q-gap (Li et al., 2023).

Federated learning contains the most explicit detection setting for targeted overfitting as a named attack. A dishonest orchestrator can aggregate only a targeted subset of client models,

$1-\exp(-a_n D_n)> D_n,$9

and return this biased model only to those clients, thereby inducing client-specific overfitting. The paper proposes three client-side integrity tests. Label flipping measures a Poison Effectiveness Score,

$\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$0

backdoor trigger injection measures the persistence of a client-specific trigger under aggregation, and model fingerprinting injects a signature $\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$1 into the local update and then checks its survival via dot product or cosine similarity. All three methods detect single-client targeting reliably; fingerprinting is fastest in that regime, while label flipping is strongest when several clients are targeted simultaneously (Mestari et al., 15 Sep 2025).

6. Mitigation strategies and conceptual limits

Mitigation work reflects the same localization principle as the diagnoses. One representation-level approach treats overfitting as a layer-specific pathology of activation sparsity. For each receptive field activation vector $\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$2, the distribution

$\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$3

defines an entropy

$\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$4

The paper reports that mean entropy drops sharply in layers such as conv3, conv4, and fc1 near the onset of overfitting. It then regularizes only those layers via

$\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$5

and optionally decorrelates filters using

$\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$6

This targeted, layer-by-layer regularization outperforms global application in several settings and, on CIFAR-100 with an xxl network, sr3Net reaches $\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$7 versus $\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$8 for batch normalization and $\hat\sigma_{\hat S}<\hat\sigma_{S^\star}.$9 for dropout (Huesmann et al., 2020).

A post-training class-specific approach treats overfitting as inflated maximum margins. In ReLU classifiers, the mitigated network clips hidden activations by

$y_{\mathbf S}$0

forming a bounded classifier $y_{\mathbf S}$1. For backdoor mitigation, MMAC preserves clean logits while penalizing classwise estimated maximum margins; for ordinary overfitting from class imbalance or overtraining, MMOM replaces the logit-preservation term with cross-entropy on a small clean balanced set while still suppressing those maximum margins. On CIFAR-10 LT-100, MMOM improves accuracy from $y_{\mathbf S}$2 to $y_{\mathbf S}$3, and on the same task after 1000 training epochs it raises accuracy to $y_{\mathbf S}$4, indicating that excessive class-specific margins can be reduced post hoc without retraining (Wang et al., 2023).

Tree boosting offers a non-neural but structurally similar mitigation strategy. PaloBoost uses out-of-bag samples not merely for error estimation but as a second training batch for stage-local regularization. After fitting a tree to in-bag pseudo-residuals, it prunes sibling leaves whose updates increase OOB loss and assigns region-specific learning rates

$y_{\mathbf S}$5

then clips them to $y_{\mathbf S}$6. This is targeted overfitting control at the leaf and region level rather than at the whole-model level, and the paper reports substantially greater stability to learning-rate and depth choices on several noisy datasets (Park et al., 2018).

Taken together, these results indicate that targeted overfitting is best understood not as a single metric but as a family of localized generalization failures. Depending on the domain, the target may be a selected submodel, a prompt embedding, a margin to one class, a subset of atypical samples, a positive-pair objective component, a specific client, or a narrow semantic coordinate system. A plausible implication is that effective countermeasures must usually be equally localized: split the optimization problem, hold out data for the parameter group most liable to over-specialize, regularize only the layers or regions that exhibit the pathology, or explicitly audit the particular target to which the training process could become miscalibrated.