Perception-Aligned Training (PAT)

• PAT is a set of machine learning techniques that embed human perceptual measures into training objectives, aligning model outputs with human perception.
• It replaces conventional L_p norm constraints with perceptual metrics like LPIPS, leading to significant improvements in adversarial robustness and generalization.
• PAT incorporates gradient alignment, transfer learning, and tailored data augmentations, resulting in enhanced model performance across vision, audio, and multimodal tasks.

Perception-Aligned Training (PAT) refers to a family of machine learning techniques that explicitly incorporate measures of human perception into either the construction of the training objective or the underlying data representation. The goal is to align model behavior, robustness, or representation learning with human perceptual judgments, rather than rely solely on conventional mathematical norms or criteria such as pixel-level errors. This paradigm has recently yielded a range of promising approaches in adversarial robustness, perceptual diversity modeling, transfer learning, multimodal alignment, and domain-specific tasks spanning vision, audio, and cross-modal understanding.

1. Foundations: Neural Perceptual Threat Models and Perceptual Adversarial Training

The foundational approach for PAT in adversarial robustness is described via the Neural Perceptual Threat Model (NPTM), which leverages a neural network–based perceptual distance, such as the Learned Perceptual Image Patch Similarity (LPIPS) metric (Laidlaw et al., 2020). The NPTM is intended to model perturbations that are imperceptible to humans, operating in lieu of the more restricted $L_p$ ball constraints commonly used in adversarial machine learning. LPIPS computes the distance between images %%%%1%%%% and $x_2$ via:

$d(x_1, x_2) \equiv \| \phi(x_1) - \phi(x_2) \|_2$

where $\phi(\cdot)$ is the feature vector obtained from a pretrained CNN (e.g., AlexNet or ResNet). The authors validate NPTM by correlating LPIPS distances with human perceptibility in large-scale perceptual studies involving Mechanical Turk annotators.

Perceptual Adversarial Training (PAT) uses NPTM to form its constraint set within adversarial training:

$$\min_{\theta_f} \mathbb{E}_{(x,y)\sim D} [\max_{\delta\,:\,d(x+\delta, x) \leq \epsilon} \mathcal{L}_{ce}(f(x+\delta), y)]$$

where the inner maximization is performed over perturbations that are bounded by LPIPS distance $\epsilon$, rather than $L_2$ or $L_\infty$ norms. Empirically, models trained with PAT show dramatically improved "union accuracy" (robust accuracy under the union of diverse attacks) and generalization to unseen perturbation types across datasets such as CIFAR-10 and ImageNet-100. For example, PAT-AlexNet achieves union accuracy of 27.8% and unseen mean accuracy of 48.5% on CIFAR-10, whereas narrow $L_\infty$ adversarial training yields union accuracies near 1–5% (Laidlaw et al., 2020).

2. Gradient Alignment and Robustness

Recent work has established that models whose input gradients are perceptually aligned with human vision—termed Perceptually Aligned Gradients (PAG)—exhibit enhanced adversarial robustness (Ganz et al., 2022). PAG can be characterized by the property that small input perturbations in the direction of the model gradient induce semantic, "class-like" changes in the image, as observed qualitatively by human annotators.

To train for PAG, the following loss function is used:

$$L_{total}(x, y) = L_{CE}(f_\theta(x), y) + \frac{\lambda}{C} \sum_{y_t=1}^C L_{cos}(\nabla_x f_\theta(x)_{y}, g(x, y_t))$$

Here, $$L_{cos}(v, u) = 1 - \frac{v^T u}{\|v\|_2 \|u\|_2 + \epsilon}$$ penalizes the misalignment between the model's input gradient and a "ground-truth" perceptually aligned gradient, which may be estimated either heuristically (e.g., using nearest-neighbor images) or via Score-Based Gradients from a diffusion model. Unlike conventional adversarial training, this strategy can be performed solely on clean images with no adversarial examples.

Extensive experiments show that inducing PAG increases both clean and adversarial accuracy across different architectures (ResNet, VGG, ViT, MLP Mixer) and datasets (CIFAR-10, CIFAR-100, Tiny ImageNet), and that gradient-alignment regularization can further improve existing adversarial training methods such as AT and TRADES (Ganz et al., 2022).

3. Human Perceptual Regularization, Transfer, and Diversity

PAT principles have been applied beyond robustness to regularize transfer learning and diversity objectives with explicit human perceptual measurements:

• In Perceptual Transfer Learning (PSYPHY-TL) (Dulay et al., 2022), human reaction times and psychophysical labels are injected as regularization into the loss function, modulating the penalty based on whether stimuli are "easy" or "hard" for human observers:

$$L = -[\sum_j y_j \log(\hat{y}_j) + \lambda \sum_j R(w_j) \cdot \psi_j]$$

where $\psi_j$ amplifies penalties for easy stimuli (error on examples with fast human RT) and relaxes them for difficult ones.

• For diversity in people images, the Perception-Aligned Text-derived Human representation Space (PATHS) (Srinivasan et al., 25 Jan 2024) is constructed via text-guided subspace extraction from a pretrained image–text model (CoCa), followed by linear fine-tuning on human annotator triplet judgments. PATHS captures perceived diversity in images, reflecting aspects such as disability and cultural attire, which are not explicitly labeled. Used in Maximal Marginal Relevance (MMR) ranking, PATHS achieves a +57.4% diversity improvement over baseline methods on occupational queries, and is validated by side-by-side human ratings (Srinivasan et al., 25 Jan 2024).

4. Individuation and Multimodal Alignment

Next-generation PAT methods enable model alignment at the individual rather than population level. POV Learning (Werner et al., 7 May 2024) integrates eye tracking and fixation sequences from individual humans to tailor multimodal models to the personal context and expectation of users. Key architectural elements include:

• Sequence modeling (LSTM) of fixation data, combined with participant embeddings for individuation
• Content-based transformer encoding of image regions and textual features, fused with participant meta-information
• Hybrid models that inject transition matrices from fixation sequences into transformer attention weights:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(QK^T/\sqrt{d_k} + \lambda \cdot (M_{Transition} + M_{Transition}^T)) V$$

POV Learning reliably boosts individual predictive accuracy (up to 77% in 3-class entailment scenarios), exceeding zero-shot, one-shot, and two-shot GPT-4 baselines (Werner et al., 7 May 2024).

5. Customized Augmentations for Perceptual Robustness

PAT is also evident in the structured augmentation of training data for robust perception models under challenging operational design domains (ODDs) (Hammam et al., 30 Aug 2024). Here, domain-specific physics-based augmentations (e.g., synthetic rain, optical effects) are devised and optimized via:

• Hyperparameter tuning and latent space optimization, ensuring augmented data is statistically indistinguishable from real adverse examples in DNN feature space
• Integration strategies such as mini-batching original images with augmented versions
• Performance measured via mAP and mIoU, showing up to +43.09% mIoU improvement for semantic segmentation in adverse conditions (while balancing slight losses in clean accuracy)

Optimal strategies are highly model- and task-specific; domain and latent space knowledge are crucial.

6. Implications and Future Directions

The emergence of PAT highlights several broader themes:

• Replacement of narrow $L_p$ threat models and handcrafted diversity criteria with neural or perceptual proxies validated by human judgment
• Synergies with prior robustification strategies—gradient regularization can act as a powerful stand-alone or auxiliary technique, and new loss formulations (e.g., robust perception regularizers) can reposition the accuracy-robustness trade-off (Wang et al., 4 Aug 2025)
• Expansion beyond vision to audio, as recent work shows psychoacoustic conditioning and perceptually-motivated contrastive losses (e.g., FiLM-modulated sequential transformers) yield representations with Spearman correlations (ρ = 0.65) to human similarity judgments in music (Liu et al., 5 Sep 2025)
• Applications in multimodal generation (video motion), where perception-grounded metrics and prompt libraries (VMBench) drive model evaluation and training toward human-aligned dynamism, improving human–machine agreement by 35.3% in Spearman’s correlation (Ling et al., 13 Mar 2025)
• Architectural integration, with efficient mechanisms such as Partial visual ATtention (PAT) blocks infusing global context via selective attention in convolutional networks, enhancing accuracy and inference speed (Huang et al., 5 Mar 2025)

7. Controversies, Limitations, and Open Challenges

PAT presents new opportunities but also raises critical challenges:

• The selection of perceptual proxies—feature-based metrics such as LPIPS, diffusion scores, or psychoacoustic embeddings—requires validation and may not generalize across modalities or tasks.
• Over-sufficient learning of adversarial samples (excessive perception consistency) may induce brittle, overly-local decision boundaries; future objectives must balance smooth perception transitions (Wang et al., 4 Aug 2025).
• Most studies depend on the availability and scale of human judgments or individualized behavioral data, requiring substantial annotation effort and potential geographical/cultural biases.
• Transferability and deployment in real-world systems remain open questions, especially as perceptual measures evolve or as operational contexts diversify.

In sum, Perception-Aligned Training constitutes a rapidly evolving paradigm, integrating human perceptual constraints and judgments directly into machine learning objectives, data representations, and model architectures. While empirical results demonstrate marked improvements in robustness, generalization, diversity, and individualized alignment, ongoing research is focused on refining perceptual proxies, loss functions, and scale-up strategies to make PAT both efficient and universally deployable.