- The paper identifies that HAT’s success hinges on gradient expectation consistency, bounded gradient variance, and non-degenerate sensitivity.
- It categorizes hardware-induced perturbations into fully compensable, conditionally compensable, and non-compensable regimes based on experiments with ResNet-20 on CIFAR datasets.
- Findings advocate for hardware-software co-design strategies that prioritize trainability alongside physical accuracy in analog and mixed-signal AI systems.
Diagnostic Perspective on Hardware-Induced Distortions in Hardware-Aware Training
Motivation and Problem Statement
Analog and mixed-signal AI accelerators, especially analog in-memory computing (IMC) systems, offer substantial throughput and efficiency advantages. However, their deployment is constrained by a plethora of device- and circuit-level non-idealities, such as variability, drift, non-linearity, IR-drop, and quantization. Hardware-Aware Training (HAT)—injecting perturbations during the forward pass—has become the de facto strategy to endow DNNs with robustness against such distortions, leveraging adaptation during the training process itself. Despite its ubiquity, HAT's effectiveness is heterogeneous: while some perturbation classes can be compensated successfully, others induce persistent optimization failures.
This paper conducts a systematic, controlled diagnostic study quantifying when and why classic forward-perturbation HAT is effective, using a taxonomy of six representative hardware-induced perturbations. The key contribution is the empirical identification and measurement of diagnostic signals in gradient-based optimization that separate learnable perturbations amendable to HAT from those requiring hardware or algorithmic intervention. The framework translates directly to guidance for hardware-software co-design.
Abstraction of Hardware Non-Idealities
The study formalizes hardware-induced distortions as structured perturbations of the neural network’s effective forward operator. The abstraction is not rooted in device-level fidelity, but instead in the algebraic coupling between trainable weights, inputs, and injected non-idealities. The six considered perturbation classes are:
- Additive noise: e.g., read/sense amplifier noise, modeled as additive perturbation to weights.
- Multiplicative (scaling) noise: e.g., conductance variability and drift, as random or systematic scaling.
- Projection (stuck-at faults): e.g., write failures, as random freezing of weights to specific values.
- Input-dependent structured scaling: simplified IR-drop where scaling is an explicit function of input activity.
- Strongly-coupled nonlinear perturbations: high-fidelity IR-drop/parasitic coupling models, coupling weights and activations.
- Discretization: finite-precision ADC/quantization, manifesting as non-smooth, non-differentiable operators.
By expressing all these perturbations as effective operators acting on the forward computation, the framework allows for direct comparison of their impact on optimization and compensation.
Diagnostic Framework for Learnability
The central diagnostic insight is that HAT compensation is governed not by perturbation severity, but by how the perturbation structure interacts with the optimization dynamics. Three empirical diagnostics separate compensable from non-compensable regimes:
- Gradient Expectation Consistency: The expected gradient with respect to model parameters should align with the gradient of the expected perturbed forward operator. This enables stable trajectory of optimization targeting a well-defined surrogate objective.
- Bounded Gradient Variance: The perturbation-induced variance of gradients must remain controlled. Unbounded variance causes instability and disrupts effective learning, even if the gradient mean is well-aligned.
- Non-degenerate Sensitivity: The perturbed forward mapping must retain responsive dependence on trainable parameters. If gradients vanish, no compensation is possible regardless of regularity.
The authors show that learnability—i.e., successful compensation through vanilla HAT—requires all three criteria. These diagnostics are measured empirically via gradient norm and variance dynamics throughout training.
Empirical Regimes and Representative Results
Systematic experiments on ResNet-20/CIFAR-10 (with similar conclusions for CIFAR-100) under a baseline of realistic non-idealities reveal three distinct compensation regimes:
Regime I: Fully Compensable Perturbations
Additive, multiplicative, and input-dependent scaling classes are consistently compensable. Despite strong forward distortion, HAT recovers near-ideal accuracy, and gradient statistics remain stable and well-behaved. These structures preserve the necessary expectation consistency, bounded variance, and non-degenerate sensitivity.
Regime II: Conditionally Compensable Perturbations
Projection (stuck-at) perturbations degrade the effective trainable subspace rather than destabilize optimization. Compensation is feasible contingent on parameter redundancy: as long as enough degrees of freedom remain, expectation consistency and variance are preserved within the non-faulty subspace.
Regime III: Non-Compensable Under Vanilla HAT
Strongly coupled nonlinearities (high-fidelity IR-drop/parasitic) and direct discretization fundamentally break classic gradient-based optimization; the former through substantial gradient variance amplification and unstable dynamics, the latter by annihilating task-relevant gradients nearly everywhere. Surrogate gradients (e.g., STE) or external intervention become requisite for any progress in optimization.
For instance, under strongly nonlinear IR-drop, both compensated and uncompensated models fail, and gradient norms display large oscillations. In contrast, quantization effects with direct discretization collapse gradients, making weights unresponsive to loss. These observations are consistent across seeds, architectures, and perturbation strengths.
Implications for Hardware-Software Co-Design
The structured diagnostic perspective has the following implications:
- Perturbation structure supersedes severity: Large forward distortion can still be compatible with HAT if operator structure preserves optimization signals; high physical model fidelity is not always desirable if it induces unlearnable couplings.
- Redundancy as a buffer: Model and hardware redundancy mitigate conditionally compensable classes (stuck-at faults) by preserving parameter subspaces, motivating explicit redundancy and fault-tolerant mapping in architecture design.
- Mitigation beyond HAT: Strong nonlinear coupling and non-smooth distortions (quantization without surrogate gradient) fundamentally require mitigation outside software retraining (circuit-level partitioning, calibration, or STE).
- Trainability as a design constraint: Effective co-design must treat gradient-accessibility and stability as hard constraints alongside energy or area constraints. Hardware modeling should be evaluated not just by physical accuracy, but also by optimization compatibility.
For future accelerators, this suggests that modeling simplifications that enhance trainability may be more practical for algorithm-hardware mapping than fully-detailed high-order physical models, unless specialized compensation is available.
Theoretical and Practical Outlook
The diagnostic framework here is intentionally controlled—focused on a fixed architecture and realistic but not exhaustive non-ideality regimes. It does not propose a universal theory of compensability but instead offers a repeatable, empirically-grounded approach to assess when HAT can be a primary compensation mechanism, and when alternative strategies must be prioritized.
A promising avenue for future work is generalizing the diagnostic methodology to joint online training-and-calibration, hardware-in-the-loop fine-tuning, and dynamically adapting architecture-to-distortion mappings, potentially automating the software-hardware boundary in next-generation AI accelerators.
Conclusion
This work establishes that the compensability of hardware-induced perturbations via hardware-aware training is governed by specific properties of the interaction between perturbation structure and gradient-based optimization dynamics, not merely perturbation magnitude. The identification of gradient expectation consistency, bounded gradient variance, and non-degenerate sensitivity as practical empirical diagnostics provides a foundation for systematic co-design decisions. The framework presented clarifies which non-idealities can be reliably delegated to training and which require explicit hardware-side intervention and paves the way for new advances in learning under perturbations in analog and mixed-signal AI hardware.
For replication and further exploration, code for the experimental and simulation procedures is provided at: https://github.com/MiCrSYZ/HW-aware_training.git
