In-Target Minimization Strategies

• In-target minimization is a strategy that explicitly optimizes risk within a specific target region, ensuring models achieve optimal performance under adversarial, domain, or value-based constraints.
• The approach restructures loss functions and employs online adaptation techniques to bridge the gap between nominal and target-specific solutions, as evidenced by significant accuracy improvements in adversarial and imbalanced settings.
• The methodology offers theoretical guarantees, such as uniqueness of global optima and rapid adaptivity with minimal computational overhead, thereby strengthening robustness and generalization in various applications.

In-target minimization refers to a broad class of strategies in optimization, learning theory, and robust control where the objective is not only to optimize some risk or loss under typical scenarios, but specifically to drive relevant quantities to their optimum within a designated target region or with respect to a target measure, constraint, or adversarial scenario. This concept is pertinent across adversarial robustness, domain generalization, minimax risk learning, Bayesian optimization, and stochastic control, manifesting in distinct but formally similar minimization principles that close gaps between surrogate or nominal solutions and those evaluated at or wrt the target of interest.

1. Formalizations Across Domains

Several canonical settings exemplify in-target minimization:

• Adversarial Robustness via Target Training: For neural net classifiers, adversarial attacks are modeled as two-term minimizations over input perturbations $\delta$:

$$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$

with $L$ a classification loss and $\lambda\|\delta\|_p$ penalizing deviation size. In "Target Training," adversarial samples are replaced with original samples assigned alternative labels in an expanded output space. The model is specifically trained to make $L(f(x), y_{\mathrm{target}})$ minimal at $\delta=0$, rendering the attack's own minimization "in-target" and thereby thwarting the adversary's progress by construction (Lindqvist, 2021).

• Domain Generalization and Adaptivity Gap: In domain generalization, one seeks to minimize the excess risk (the adaptivity gap)

$\Delta(h; T) = R_T(h) - R_T(h_T^*)$

on an unseen target domain $T$, where $h_T^*$ is the optimal hypothesis for $T$. Two strategies are used: (a) hypothesis selection/ensembling targeting best match to the target, and (b) online adaptation—steering model parameters to locally minimize risk on each target batch/sample as data arrives, explicitly tightening the in-target risk (Zhang et al., 2022).

• Minimax Risk for Imbalanced Classification: For imbalanced data, minimax learners define a risk

$$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$0

with target prior $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$1 (not necessarily empirical), seeking $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$2 for a potentially adversarially chosen $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$3. The inner minimization, dubbed in-target minimization, is realized with specialized surrogate losses that guarantee Bayes-optimality for the current $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$4. The process alternates between this minimization and an ascent step updating $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$5 towards the worst-case (Choi et al., 24 Feb 2025).

• Bayesian Optimization with Target Values: When the goal is to find $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$6 such that $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$7 matches a pre-specified target $$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$8, the acquisition functions for "target-value" BO minimize the expected squared error

$$\min_{\delta} L(f(x+\delta), y_{\mathrm{target}}) + \lambda\|\delta\|_p$$9

and are computed in closed form even under known/estimated aleatoric noise, making the surrogate minimization strictly in-target (Hoffer et al., 2023).

• Stochastic Control with Target Constraints: In continuous-time control, one may seek to minimize control effort subject to reaching stochastic target constraints, e.g., $L$0 on an event $L$1. The value function for such problems is the minimal positive solution to a semilinear ODE determined by the structure of the target constraint, explicitly characterizing optimal strategies conditioned on hitting the target (Dolinsky et al., 2019).

2. Methodological Principles and Algorithmic Structures

In-target minimization modifies conventional optimization or training objectives in the following structural ways:

• Constraint Realignment or Loss Restructuring: In adversarial robustness and minimax risk, the loss is reparameterized or reweighted so that the optimum with respect to the target or adversarial measure occurs at a feasible or desirable point (e.g., $L$2 in adversarial setting, Bayes-optimal boundary under non-empirical $L$3).
• Expansion of Hypothesis or Output Space: Target Training extends classifier outputs to $L$4 classes and trains each original example with both its true and shifted label. This doubles the representational space allowing for strict in-target minimization by construction (Lindqvist, 2021).
• Online/Adaptive Procedures: Both domain generalization and disagreement minimization methods design algorithms where, at inference/test time, updates are performed to reduce error (risk, entropy, disagreement) specifically on incoming target samples, as opposed to static, source-trained hypotheses (Zhang et al., 2022, Zhang et al., 2022).
• Surrogate and Acquisition Design for Target-Value Problems: In Bayesian optimization, target-inclusion drives acquisition functions (e.g., Target EI, Target LCB) where probabilities and expected improvements are expressed in terms of achieving or improving upon the desired target error, not just extremal function values (Hoffer et al., 2023).

3. Theoretical Analysis and Guarantees

Rigorous analysis for in-target minimization techniques often yields strengthened or target-specific generalization, robustness, or optimality results:

• Uniqueness and Global Optimality: For adversarial minimization objectives including a norm penalty, the global minimum for targeted classes is uniquely achieved at $L$5 once the network is in-target at the shifted label, as all other perturbations strictly increase the cost (Lindqvist, 2021).
• Generalization Bounds: In minimax risk classifiers, new prior-dependent generalization bounds demonstrate that the targeted logit adjustment ("TLA") surrogate provides strictly smaller estimation error for rare/targeted classes compared to naive prior-reweighting, underlining the benefit of in-target minimization for skewed classes (Choi et al., 24 Feb 2025).
• PDE/ODE Characterizations: In stochastic control, the optimal value function under target endpoint constraints reduces to minimal positive solutions of specific ODEs—with existence, uniqueness, and feedback-optimal controls constructed explicitly in terms of the target survival probability (Dolinsky et al., 2019).
• Rapid Adaptivity with Minimal Latency: For online adaptation in domain generalization, in-target minimization algorithms incur negligible computational overhead (<2 ms/sample) but can achieve OOD improvements of 3–5% in accuracy over strong ERM and invariant risk minimization (IRM) baselines (Zhang et al., 2022, Zhang et al., 2022).

4. Empirical Evidence and Performance Implications

Direct evaluation across domains confirms the practical advantages of in-target minimization:

• Adversarial Training Without Adversarial Examples: Target Training achieves 84.8% accuracy on CIFAR-10 against CW-$L$6 ($L$7) attacks—substantially above both standard training (8.5%) and classic adversarial training (22.8%)—without using any adversarial examples for these attacks (Lindqvist, 2021).
• Outer Distribution (OOD) Generalization: Domain-Specific Risk Minimization and AdaODM both yield OOD accuracy gains (2–5%) over ERM/IRM/Fishr and other meta-learning-based methods on structured vision benchmarks, with ablations revealing that the adaptation rate and batch size have minimal influence on the OOD gain as long as basic tuning is applied (Zhang et al., 2022, Zhang et al., 2022).
• Robustness Under Imbalanced Sampling: The TLA minimax classifier yields near-optimal worst-class accuracy under imbalanced-label scenarios (CIFAR-100 step imbalance: 16.0% vs. 5.6% for standard CE) and preserves or enhances feature-space clustering for rare classes (Choi et al., 24 Feb 2025).
• Target-Value Optimization in the Presence of Noise: Robust target-aimed acquisition functions halve the number of iterations needed to attain optimal squared error in noisy or heteroskedastic real-world settings relative to classical GP-based BO methods (Hoffer et al., 2023).
• Efficiency and Complexity: In-target minimization typically results in trivial additional complexity—e.g., only extra domain heads in DRM, only a batch norm parameter update at test time in AdaODM, and no need for expensive adversarial sample generation in Target Training (Lindqvist, 2021, Zhang et al., 2022).

5. Interpretations, Implications, and Limitations

The in-target minimization paradigm suggests several conceptual and practical shifts:

• Attack and Defense Alignment: For robustness, rather than merely populating gaps with adversarial examples, robust training can directly align the minimization objective of the attacker with a model-induced minimum by making the in-target optimum trivial to achieve only at the unperturbed example (Lindqvist, 2021).
• Target-specific Optimality: In minimax schemes, minimizing loss under a chosen adversarial or rare-class prior ensures that learned decision boundaries track the hardest/most relevant class distributions, rather than mean-aggregated or prior-unweighted performance (Choi et al., 24 Feb 2025).
• Adaptive and Contextual Performance: Online adaptation, as in DRM or AdaODM, illustrates that continual in-target minimization at deployment is critical for generalization under distribution shift, fundamentally bridging the gap between source-trained and truly target-optimized models (Zhang et al., 2022, Zhang et al., 2022).

A potential limitation is that when the target definition or achievable minimum does not coincide with feasible or computationally efficient training objectives, performance gains may saturate or the approach may fail to generalize beyond cases where the target region is well-characterized or readily accessible.

6. Broader Context and Ongoing Developments

In-target minimization unifies and extends classic regularization, constraint-satisfaction, and adversarial/robust optimization frameworks. It enables architectures and learning procedures that are maximally attuned to specific targets—be those adversarial goals, rare classes, or domain-shifted data distributions. Methodologies continue to evolve, especially in adaptation to non-stationary data, maximizing probabilistic target satisfaction under uncertainty, and scaling approaches to large, real-world deployments.

References:

• "Target Training Does Adversarial Training Without Adversarial Samples" (Lindqvist, 2021)
• "Domain-Specific Risk Minimization for Out-of-Distribution Generalization" (Zhang et al., 2022)
• "ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation" (Vu et al., 2018)
• "Deep Minimax Classifiers for Imbalanced Datasets with a Small Number of Minority Samples" (Choi et al., 24 Feb 2025)
• "Robust Bayesian Target Value Optimization" (Hoffer et al., 2023)
• "Adaptive Domain Generalization via Online Disagreement Minimization" (Zhang et al., 2022)
• "A Note on Costs Minimization with Stochastic Target Constraints" (Dolinsky et al., 2019)