PEPPP: Optimal Low-Precision Configuration

• PEPPP is a meta-learning framework for selecting optimal low-precision configurations in deep neural network training based on Pareto error-memory trade-offs.
• It uses low-rank matrix factorization and a design-of-experiments approach to efficiently predict test error under memory constraints, minimizing the need for exhaustive retraining.
• The method offers theoretical guarantees and empirical success across various tasks, enabling resource-aware precision tuning for improved model efficiency.

Pareto Estimation to Pick the Perfect Precision (PEPPP) is a meta-learning-based methodology for identifying optimal low-precision configurations in deep learning model training, with a focus on balancing prediction error against memory consumption. It is motivated by the trade-offs inherent in low-precision arithmetic for model efficiency and the combinatorial nature of bit-width choices across neural network components. PEPPP frames the selection of precision as a hyperparameter tuning problem and applies low-rank matrix factorization and design-of-experiments theory to efficiently estimate the Pareto frontier of error vs. memory, minimizing the need for exhaustive network retraining (Yang et al., 2021).

1. Formulation of the Memory-Error Pareto Frontier

The PEPPP framework operates on a discrete set $C$ of low-precision "configurations," each specifying bit-widths for activations, weights, optimizer, and related components. For any configuration $c \in C$ and dataset (or "task") $i$, the following quantities are defined:

• $m(c) \in \mathbb{R}^+$: memory footprint (in MB) of configuration $c$.
• $E_i(c) \in [0,1]$: test error of configuration $c$ on task $i$.

Given a target memory budget $M_{\max}$, the problem is to identify $c$ for a new ("meta-test") dataset that minimizes $c \in C$0 subject to $c \in C$1. This is equivalent to selecting the lowest-error configuration on the non-dominated (Pareto-optimal) set under the memory constraint: $c \in C$2 This multi-objective formulation facilitates principled memory-error trade-offs by practitioners.

2. Low-Rank Matrix Factorization for Error Modeling

PEPPP exploits the empirical observation that the task-configuration error matrix is approximately low-rank. Let $c \in C$3 be the error matrix over $c \in C$4 "meta-training" datasets and $c \in C$5 configurations, and $c \in C$6 the memory matrix (with identical rows). The model posits a factorization: $c \in C$7 with each $c \in C$8. The unknowns $c \in C$9 are estimated via regularized least-squares matrix completion over observed entries $i$0: $i$1 The SoftImpute algorithm, based on nuclear-norm minimization and alternating SVD with soft-thresholding, is used for this purpose. The completed matrix is further factorized by truncated SVD to obtain $i$2.

3. PEPPP Workflow: Meta-Training and Meta-Test Procedures

PEPPP proceeds in two phases: meta-training (offline) and meta-test (query).

Meta-Training Phase:

1. Compute the full memory matrix $i$3.
2. Sample a fraction of entries $i$4 of $i$5 (uniform or $i$6).
3. Complete $i$7 via SoftImpute, yielding $i$8.
4. Extract low-rank factors $i$9 from $m(c) \in \mathbb{R}^+$0.
5. Optionally, determine true/estimated frontiers $m(c) \in \mathbb{R}^+$1 for each training task.

Meta-Test Phase: Given $m(c) \in \mathbb{R}^+$2, $m(c) \in \mathbb{R}^+$3, memory vector $m(c) \in \mathbb{R}^+$4, memory budget $m(c) \in \mathbb{R}^+$5, and query budget $m(c) \in \mathbb{R}^+$6:

1. $m(c) \in \mathbb{R}^+$7 specifies feasible configurations.
2. Select $m(c) \in \mathbb{R}^+$8, $m(c) \in \mathbb{R}^+$9, via greedy D-optimal experiment design on the configuration embeddings $c$0.
3. Train/test $c$1 on the target task to obtain errors $c$2.
4. Solve for the new task’s embedding $c$3 by

$c$4

1. Predict $c$5 for all $c$6 in $c$7.
2. Form the estimated Pareto frontier $c$8 from $c$9.
3. Recommend $E_i(c) \in [0,1]$0 (maximal memory under budget, i.e., smallest error for feasible $E_i(c) \in [0,1]$1).

Pseudocode (Meta-Test Stage)

$i$8

4. Theoretical Guarantees

Two central results underpin PEPPP:

• Matrix-Completion Accuracy: If $E_i(c) \in [0,1]$2 is exactly rank $E_i(c) \in [0,1]$3 and incoherent, uniform sampling of $E_i(c) \in [0,1]$4 suffices for exact recovery with high probability by nuclear-norm minimization. In practice, empirical singular value decay ensures a small $E_i(c) \in [0,1]$5 yields accurate completion.
• Design-of-Experiments Optimality: Under a linear model for the meta-test error vector ($E_i(c) \in [0,1]$6), greedy D-optimal selection (which approximately minimizes det$E_i(c) \in [0,1]$7) provides a $E_i(c) \in [0,1]$8-approximation to the optimal subset. Consequently, with $E_i(c) \in [0,1]$9 empirical queries, the new-task embedding and entire error vector can be predicted with small variance, enabling correct Pareto frontier reconstruction and reliable configuration selection.

5. Empirical Evaluation

Experiments were conducted on 87 meta-training tasks encompassing CIFAR-10, CIFAR-100 subsets, ImageNet sub-hierarchies, and various small-image datasets ($c$0, $c$1). The model configurations (total $c$2) comprise combinations of activation+weight bit-widths (5–9 bits, with exponent/mantissa splits) and optimizer precisions (14–20 bits), with architectures such as ResNet-18, and further ablation over ResNet-34 and VGG.

Key evaluation metrics:

• Convergence: average Euclidean distance from each estimated frontier point to the true frontier.
• Hypervolume Difference: normalized $c$3 under respective frontiers.
• Matrix-Completion Error: relative Frobenius norm error between estimated and true per-task error vectors, $c$4.

Results demonstrate that PEPPP, with $c$5, $c$6–$c$7 empirical queries, and $c$8 uniform sampling, reliably identifies Pareto-optimal configurations, sharply reducing the cost of a full combinatorial search.

6. Practical Implementation and Interpretation

Meta-Training Recommendations:

• Collect dataset$c$9configuration error matrix with about $i$0–$i$1 coverage.
• Use SoftImpute with $i$2, $i$3 for completion and truncated SVD.

Meta-Test Guidelines:

• Compute memory usages $i$4 for all configurations.
• Fix the target memory cap $i$5.
• Use the greedy D-optimal design to select $i$6 configurations for empirical evaluation.
• Fit the embedding, reconstruct errors, and extract the Pareto front to select the best configuration under the memory constraint.

Extensions:

• Incorporating additional hyperparameters (learning rate, epochs) as matrix dimensions.
• Multi-architecture meta-learning by stacking error matrices.

Pareto Frontier Visualization:

• Points in the $i$7 plane represent configuration memory and error.
• Pareto-optimal configurations are those for which no other offers both lower error and memory.
• The chosen configuration is the one on the estimated Pareto front with maximal memory fitting within the budget (i.e., error-minimizing at fixed memory).

7. Significance and Implications

PEPPP enables precise, resource-aware decision-making for low-precision deep learning, using only a modest number of network runs on the target task. Its matrix-factorization backbone, coupled with D-optimal sampling, facilitates generalization to unseen datasets (meta-test) and reduces the computational and experimental burden of full configuration sweeps. By capturing the memory–prediction error trade-off systematically, PEPPP provides a reproducible and theoretically justifiable methodology for practical precision selection in neural network training (Yang et al., 2021).