Plain Training Technique

• Plain training technique is defined by minimal architectural modifications, basic data augmentation, and straightforward optimization aimed at maintaining simplicity in training pipelines.
• In applications like Vision Transformers and unsupervised anomaly detection, it achieves competitive results, with metrics such as up to 80% Top-1 accuracy on ImageNet and 85.4 mAD on MVTec AD.
• By employing fixed positional embeddings and plain SGD variants, this approach streamlines model training without relying on complex regularization or ad hoc tricks.

The plain training technique refers to a set of model training methodologies that emphasize minimal architectural modifications, avoidance of elaborate regularization, and use of straightforward optimization and augmentation strategies. In leading research, including high-performing Vision Transformers (ViT) and anomaly detection models, "plain" training denotes a deliberate focus on simplicity of pipeline components, such as basic data augmentation, no architectural overengineering, and minimization of additional losses or ad hoc tricks, while achieving state-of-the-art results or competitive performance relative to more complex counterparts (Beyer et al., 2022, Zhang et al., 2023). The term also surfaces in optimization literature, where plain stochastic gradient descent (SGD) and its transition variants are explicitly contrasted with momentum-based and adaptive optimizers (Zeng et al., 2021).

1. Fundamental Characteristics of Plain Training

Plain training protocols are distinguished by their parsimonious use of components beyond those essential for end-to-end supervised or self-supervised learning. For example, the ImageNet-1K “plain ViT” recipe dispenses with heavyweight regularizers such as stochastic depth, dropout, CutMix, and knowledge distillation, relying instead on classical data preprocessing, elementary augmentation (RandAugment, Mixup with marginal probabilities), and a linear classification head (Beyer et al., 2022). In anomaly detection, plain ViT-based pipelines avoid the use of multi-resolution pyramidal CNNs, adversarial losses, and feature-fusion networks, instead favoring symmetric ViT encoder–decoder architectures with a single linear fuser and a basic pixelwise reconstruction loss (Zhang et al., 2023).

Key algorithmic features include:

• Fixed positional embedding schemes (e.g., 2D sin-cos), supplanting learned embeddings.
• Global average pooling for final feature aggregation instead of using designated special tokens (e.g., [CLS]).
• Reduction in batch sizes for improved convergence dynamics (e.g., 1024 images per step on ViT).
• Minimal optimizer customizations: AdamW with canonical hyperparameters and cosine or step-wise learning-rate schedules.

2. Plain Training in Vision Transformers

Plain training paradigms in Vision Transformers refute the prevailing assumption that extensive regularization and fine-tuning (e.g., dropout, stochastic-depth, high-resolution fine-tuning) are required for competitive accuracy on large-scale datasets. Modest data pipeline augmentations—decode, inception-crop, flip, RandAugment B(2,10), Mixup at $p=0.2$—prove sufficient for ImageNet-scale data. Batch sizes of 1024, fixed sin-cos positional embeddings, a global-average-pooling head, and a linear classifier achieve:

• 76.5% Top-1 accuracy on ImageNet-1k at 90 epochs (<7h on TPUv3-8), rivaling ResNet-50.
• 80.0% Top-1 with 300 epochs, all without resorting to high-complexity or computationally intensive tricks.

Ablation studies show the main contributors to accuracy improvements are RandAugment, Mixup, global average pooling, and lowered batch size. Other modifications, such as reverting to learned positional embeddings or [CLS]-based pooling, consistently degrade performance (Beyer et al., 2022).

3. Optimization Under Plain Training Regimes

Plain stochastic gradient descent is foundational for deep network training. The "momentum-to-plain SGD transition" method (TSGD) operationalizes a scalar schedule that interpolates between momentum SGD and its plain counterpart via a dynamically annealed scaling factor. The TSGD rule, for parameters $\theta_t$ at iteration $t$:

• Computes an exponential moving average $m_t$ of gradients with coefficient $\beta$.
• Blends $m_t$ and the instant gradient $g_t$ through a decay factor $\gamma^{-t}$, yielding the update direction $\hat{h}_t = (m_t - g_t)/\gamma^{t} + g_t$.
• Updates parameters with a linear time-decayed learning rate.

This protocol achieves smooth convergence by accelerating early optimization with momentum and refining late-stage convergence with a pure SGD regime. Empirical results show TSGD improves over both classical SGDM and adaptive optimizers (Adam, AdaBound) on MNIST, CIFAR-10, and CIFAR-100 benchmarks in both training speed and final accuracy (Zeng et al., 2021).

4. Plain Training for Unsupervised Anomaly Detection

The "plain" approach in unsupervised anomaly detection is exemplified by ViTAD, which leverages a symmetric 12-layer ViT encoder and decoder, no skip connections, and a single linear layer for feature fusion. Only the deepest feature stage output is fused, and only simple pixel-wise distance losses (e.g., flattened cosine distance) are applied to reconstructed features at intermediate encoder stages. Data augmentation is eschewed after direct comparison, as augmentations such as CenterCrop+ColorJitter and RandomResizedCrop consistently degrade mAD performance.

Compared to prior multi-resolution hand-crafted pipelines, which require pyramidal CNNs, heavy fusion/modules, and adversarial losses (leading to high GPU memory and compute time), plain ViTAD reaches:

• 85.4 mAD on MVTec AD (single V100, 1.1hr, 2.3GB), surpassing methods such as UniAD and DRAEM. Ablation confirms no significant gain from alternate losses or complex augmentations, and self-supervised DINO pre-training is critical for feature separability (Zhang et al., 2023).

5. Empirical Performance and Ablation Study Results

Benchmarking plain training methodologies yields the following empirical trends:

Model Variant	90 ep	150 ep	300 ep
Full plain ViT improvements	76.5	78.5	80.0
Without RandAug+Mixup	73.6	73.7	73.7
sincos2d → learned pos-emb	75.0	78.0	79.6
batch 1024 → 4096	74.7	77.3	78.6
GAP → [CLS] token	75.0	76.9	78.2
linear head → 2-layer MLP+Tanh	76.7	78.6	79.8

For anomaly detection:

• Switching ViTAD’s loss among $L_1$, MSE, pixelwise/cosine variants changes mAD by only ±0.4.
• Eliminating skip connections and limiting fusion to the last encoder stage is empirically justified.

These results underscore that minimal configurations in data pipeline, backbone, and loss suffice for strong accuracy and efficiency.

6. Implications, Limitations, and Best Practices

The plain training paradigm provides evidence that careful selection of minimum-necessary ingredients can deliver competitive or superior results in vision and optimization tasks. The combination of basic data augmentation, global pooling, non-learned positional embeddings, simple optimizers, and elimination of hand-crafted fusion or adversarial modules often enables rapid, scalable training with reduced memory and compute requirements.

A plausible implication is that a wide swath of tasks—where pipeline complexity has been historically justified by incremental gains—may admit equally or more effective plain recipes, provided architecture and optimizer choices exploit the global modeling capacity (e.g., in transformers) and fully leverage statistical power without shortcut mechanisms (e.g., skip connections in reconstruction). Limitations manifest when domain-specific inductive biases (e.g., spatial hierarchies in vision) cannot be expressed within the plain regime, or where complex data distributions require explicit multi-scale feature extraction.

In summary, plain training techniques—characterized by reduction to essential architectural and pipeline components—have demonstrated, via ablation and benchmark evidence, an ability to match or exceed conventional baselines while greatly simplifying reproduction and deployment (Beyer et al., 2022, Zhang et al., 2023, Zeng et al., 2021).