TESTNAV: Pareto-Guided Search for Compositional Robustness Testing

Abstract: Deep learning models remain vulnerable to real-world input perturbations, especially when multiple corruptions co-occur in the same input (e.g., brightness shifts and motion blur). Compositional testing reveals these interaction effects but introduces two challenges: combinatorial growth of the perturbation space as dimensions and severity levels increase, and uneven diagnostic value-many combinations yield unrealistically degraded inputs with limited practical relevance. We present TESTNAV, 1 a Pareto-guided robustness testing framework for efficiently exploring discrete, compositional perturbation spaces when only a limited number of perturbation configurations can be evaluated. TESTNAV prioritises severe yet realistic failures by formulating robustness testing as bi-objective optimisation: maximise performance degradation while preserving input fidelity measured by modality-specific metrics (e.g., SSIM and KID for vision; chrF and BERT-F1 for language and code). It uses NSGA-II to approximate the bi-objective Pareto front. Across four benchmarks spanning vision, natural language, and code generation, TESTNAV recovers Pareto fronts up to 2.15x faster than search-based baselines, using 35.8%-89.3% of the discrete perturbation space defined by four perturbation dimensions with six levels each.

• The paper introduces TestNav, an NSGA-II-based bi-objective search that identifies perturbation combinations causing maximum model degradation while preserving input fidelity.
• Across vision, language, and code benchmarks, TestNav recovered complete Pareto fronts up to 2.15× faster than competing methods while evaluating only 35.8%–89.3% of configurations.
• The results show that compositional failures span one to four perturbations, and that Pareto search is most valuable for broad fronts, while fidelity-metric choice strongly affects which failures are discovered.

TestNav addresses a concrete gap in deep learning robustness evaluation: perturbation combinations, not isolated corruptions, dominate real-world degradation, yet exhaustive exploration of compositional configuration spaces is computationally prohibitive and many configurations are diagnostically worthless. The paper formalises compositional robustness testing as budget-constrained bi-objective search over discrete perturbation configurations—maximising performance degradation while preserving input fidelity—and instantiates it with NSGA-II. Across four benchmarks spanning vision, natural language, and code generation, TestNav recovers ground-truth Pareto fronts up to $2.15\times$ faster than search-based baselines, evaluating only 35.8%–89.3% of the full discrete space.

Problem formulation

The framework defines a perturbation configuration space $\Theta = \{0,\ldots,\ell-1\}^n$, where each of $n$ perturbation types is applied at one of $\ell$ discrete severity levels (level 0 denoting no perturbation). With $n{=}4$ and $\ell{=}6$, this yields $|\Theta| = 1{,}296$ configurations per benchmark—a scale at which exhaustive evaluation requires inference over the entire test set for every configuration.

Each configuration $\bm{\theta}$ is scored on two objectives: performance degradation $\updelta(\bm{\theta}) = \max(\psi(\mathcal{D}) - \psi(\mathcal{T}_{\bm{\theta}}), 0)$, where $\psi$ is task accuracy or Robust Pass RP$\Theta = \{0,\ldots,\ell-1\}^n$0@1; and input fidelity $\Theta = \{0,\ldots,\ell-1\}^n$1, instantiated as SSIM or KID for images and chrF or BERT-F1 for text and code. Both objectives are normalised to $\Theta = \{0,\ldots,\ell-1\}^n$2. The target is the Pareto front $\Theta = \{0,\ldots,\ell-1\}^n$3 of non-dominated configurations—the set of failures that are simultaneously severe and faithful to the original inputs. This framing rests on an explicit assumption: that high-fidelity failures are more informative than heavily corrupted ones, since the latter may reflect expected sensitivity rather than genuine robustness weakness.

Search procedure

TestNav applies NSGA-II directly over the configuration lattice. An initial population of $\Theta = \{0,\ldots,\ell-1\}^n$4 configurations is sampled uniformly; each generation applies simulated binary crossover (SBX) and polynomial mutation, with fractional offspring rounded to valid integer severity levels via a repair step from pymoo. Candidates are ranked by Pareto dominance on $\Theta = \{0,\ldots,\ell-1\}^n$5, and population selection combines Pareto rank with crowding distance to maintain diversity across the trade-off surface. A deduplication cache ensures no configuration is evaluated twice; the proposal budget $\Theta = \{0,\ldots,\ell-1\}^n$6 is set to $\Theta = \{0,\ldots,\ell-1\}^n$7 only to observe convergence traces, with efficiency measured by unique configurations required for full front recovery.

Hyperparameters were selected by grid search over crossover index $\Theta = \{0,\ldots,\ell-1\}^n$8, mutation index $\Theta = \{0,\ldots,\ell-1\}^n$9, and population size $n$0—64 settings across all benchmarks, two fidelity metrics each, and 10 seeds (5,120 runs total). Selection maximised the number of benchmark–metric settings reaching Recall@$n$1, yielding $n$2, $n$3, $n$4.

Experimental design

Four benchmarks were exhaustively evaluated to construct ground-truth fronts:

Benchmark	Model	Perturbations	Fidelity metrics	Task metric
Tiny-ImageNet (10k)	CaiT-S36	speckle noise, glass blur, brightness, pixelate	SSIM, KID	Accuracy
QQP (1k)	RoBERTa-base	synonym, typo, contraction, punctuation	chrF, BERT-F1	Accuracy
HumanEval (164)	CodeGen-2B-mono	butterfingers, char case, whitespace, newline	chrF, BERT-F1	RP$n$5@1
MBPP (974)	CodeGen-2B-mono	butterfingers, char case, whitespace, newline	chrF, BERT-F1	RP$n$6@1

A notable structural finding is that Pareto fronts contain configurations of order one through four—Pareto-optimal failures are not limited to single perturbations, confirming that compositional effects matter even under fidelity constraints. Front sizes range from 12 to 30 configurations depending on benchmark and metric, and the choice of fidelity metric materially changes which configurations are Pareto-optimal: on Tiny-ImageNet, SSIM and KID produce same-sized fronts ($n$7) but different order distributions, with SSIM concentrating mass in low-order configurations and KID in third-order ones.

Results

Efficiency against baselines. Compared against random search, greedy hill-climbing, and a scalar-scored genetic algorithm using $n$8, TestNav fully recovers $n$9 up to $\ell$0 earlier than competing methods that also achieve Recall@$\ell$1. On Tiny-ImageNet with SSIM, TestNav attains AUC-Recall 0.704 versus 0.557 for greedy search and 0.510 for random search. However, the advantage is conditional on front geometry: when the front is compact—as under KID-derived fidelity for vision and BERT-F1 for language/code—greedy search achieves higher AUC on three of four benchmarks. Scalar search concentrates on one region of the trade-off surface, which suffices for small concentrated fronts but fails on broad ones; crowding-distance selection is what preserves spread in the latter case. This is an honest limitation of the headline claim: Pareto guidance pays off specifically when the front is broad or dispersed.

Single-objective ablations. Removing either objective substantially degrades recovery. Degradation-only search achieves AUC between 0.185 and 0.384—up to 0.545 below TestNav—because it sacrifices fidelity; fidelity-only search reaches 0.612–0.771 but lacks any failure signal. Both ablations exhibit wider variance across seeds than TestNav, indicating that bi-objective selection also improves stability.

Input-level metrics are insufficient proxies. Seven activation-coverage and prioritisation signals (NAC, SNAC, KMNC, TKNC, DeepGini, LSA, DSA), each used as the sole search objective, fail to match TestNav's final recall. Under SSIM, TestNav reaches Recall@$\ell$2 of 0.993 versus 0.900 for the best coverage metric (KMNC); TKNC (0.496) and LSA (0.581) perform worst. Under KID-derived fidelity, KMNC matches TestNav at 0.993, but no other metric comes close. The explanation is structural: these metrics characterise individual inputs, not where a configuration lies on the $\ell$3 surface, so maximising them can favour novel or activation-diverse configurations without identifying severe, high-fidelity failures.

Limitations and open questions

The framework operates exclusively over discrete, predefined perturbation families and severity levels; extension to continuous perturbation spaces remains open. The informativeness assumption—that extreme Pareto points are the right targets—is goal-dependent, and the paper leaves open how practitioners should select regions of $\ell$4 and match search algorithms to front geometry. Additionally, because the geometry of $\ell$5 depends on modality- and task-specific choices of $\ell$6 and $\ell$7, results do not transfer automatically across metric choices, as the SSIM-versus-KID discrepancy demonstrates.

Conclusion

TestNav contributes a principled reformulation of compositional robustness testing as bi-objective Pareto search, together with empirical evidence that diversity-preserving multi-objective selection recovers degradation–fidelity trade-offs that scalar and single-objective methods miss—provided the Pareto front is sufficiently broad. The exhaustive ground-truth construction over all 1,296 configurations per benchmark gives the efficiency claims unusual rigor, and the negative result on neural-coverage metrics clarifies that input-level adequacy signals cannot substitute for configuration-level bi-objective search.