NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems

Abstract: Industrial advertising recommender models are continuously improved through architecture evolution. Upgrades such as RankMixer, TokenMixer-Large, and MixFormer show that better structures remain a key source of quality and business gains. Yet developing such upgrades in production is expert-intensive and difficult to scale. Existing automation is insufficient: AutoML mainly tunes hyper-parameters, while effective gains often require cross-module changes under strict constraints; generic LLM coding agents optimize for runnable code, but runnable code does not imply a valid recommender architecture. Candidates may pass local tests while causing silent failures that degrade performance. We present NOVA, a level-aware agent harness for verification-aware architecture evolution. NOVA uses an architecture gradient, an SGD-inspired, non-differentiable update signal that aggregates prior modifications, verification diagnostics, metric feedback, and trajectory memory to guide the next modification. A verification cascade checks structure semantics, local executability, offline effectiveness, and online impact; invalid candidates are blocked early, with failure patterns recorded as forbidden directions. L1--L4 task-level control matches automation to task complexity and risk, routing high-risk tasks to Copilot for human oversight. Deployed in an industrial advertising system, NOVA achieves the highest effective pass rate on L2 ScaleUp and L3 Literature-to-Production tasks (54.5% and 60.0%), reduces silent failures compared with coding-agent baselines, and shortens one literature-to-production cycle by over 13x in human-attended time. In online A/B testing, the selected L3 candidate improves GMV on three pCVR objectives by +1.25%, +1.70%, and +2.02%, while reducing pCVR bias by 58.8%, 66.7%, and 37.3%.

• The paper presents a verification-aware, gradient-driven framework that decomposes architecture evolution into a closed-loop workflow, enabling effective candidate updates.
• It leverages a multi-stage verification cascade to filter structural and semantic errors pre-training, significantly reducing silent failures and boosting effective pass rates.
• Empirical evaluations demonstrate over 13× reduction in human engineering time and notable GMV increases, proving NOVA’s impact on production-scale recommender systems.

NOVA: Verification-Aware, Gradient-Driven Architecture Evolution for Industrial Recommender Systems

Introduction

NOVA introduces a rigorous, verification-aware agent harness for industrial recommender system architecture evolution, targeting production-scale advertising platforms. This method fundamentally departs from generic AutoML and code-generation agents by integrating domain-specific, architecture-level semantics and emphasizing iterative, feedback-driven modification cycles. NOVA operates through a structured, agentic workflow that leverages trajectory memory, business-metric-aligned gradient signals, and a multi-stage verification cascade, ultimately reducing silent failures and accelerating effective deployment of architectural innovations.

Framework Overview

NOVA decomposes architecture evolution into a closed-loop, level-aware workflow. The Main Agent determines task complexity (L1 to L4) and automation mode (AutoRun or Copilot). It then orchestrates seven specialized stages: Initialization, Solution Design, Code Generation, Quality Assessment, Local Testing, Offline/Online Training, and Evaluation. Critically, a non-differentiable "architecture gradient" guides feasible modification selection, aggregating prior edits, verification outcomes, metric feedback, and trajectory memory. This gradient updates candidate proposals in subsequent rounds, ensuring that both successes and structural failures inform the search direction.

Figure 1: NOVA orchestrates a closed-loop, level-aware architecture-gradient workflow, integrating progress signals and semantic verification across seven pipeline stages.

The architecture state is represented as $A_t = (G_t, \phi_t, F_t)$, encompassing topology, structural hyperparameters, and feature configuration. Modification proposals $e_t \in \mathcal{E}$ are constrained by production requirements (e.g., interface compatibility, latency, and parameter budgets).

Silent-Failure-Aware Verification Cascade

A central contribution is NOVA’s multi-stage verification cascade, which addresses both hard and silent failures before incurring expensive training or online validation. Each candidate modification passes through (a) structure-semantic checks (shape, dtype, masking, feature-to-token mapping, logit-fusion), (b) local executability (single-node tests), (c) offline evaluation (AUC screening), and finally (d) online business metric validation (e.g., GMV, pCVR bias). Structural or semantic errors caught early are logged as forbidden modification patterns, directly influencing the next architecture gradient and candidate proposal.

Figure 2: Silent-failure-aware cascade: invalid architectures are filtered pre-training, their diagnostics become forbidden directions shaping subsequent search.

Verification serves not only as a filter but, importantly, as gradient denoising: recurring structural errors incrementally bias the agent away from non-productive zones in the modification space, increasing the effective pass rate and reducing wasted resource expenditure.

Comparative Results and Numerical Findings

Empirical evaluation in a large-scale advertising recommender production setting demonstrates NOVA's superiority over human expert iteration, AutoML (Optuna-TPE), and coding-agent baselines (OpenHands, ReActAgent-only). Benchmarks span two industrial tasks:

• L2 ScaleUp: Coupled hyperparameter scaling under hard constraints (e.g., token count, token dimension, RankMixer layers).
• L3 Literature-to-Production: Transplantation and adaptation of research architectures (e.g., TokenMixer-Large) into production.

Key metrics include Local Pass Rate (LPR), Silent Failure Rate (SFR), and Effective Pass Rate (EPR):

Method	L2 EPR (%)	L3 EPR (%)
Human Expert	49.3	31.1
OpenHands	6.7	10.2
ReActAgent-only	12.5	7.1
Optuna-TPE	4.7	--
NOVA	54.5	60.0

NOVA achieves the highest EPR on both tasks, with LPR/EPR on L3 more than doubling that of the human-expert baseline. NOVA also demonstrates substantial reductions in silent failures, a dominant pain point for agentic code-generation approaches in nontrivial, production-constrained search spaces.

In measured production cycles, NOVA compresses human-attended engineering time by over $13\times$ compared to legacy workflows. In online A/B validation, NOVA-selected architectures yield statistically significant GMV lifts ($+1.25\%$, $+1.70\%$, $+2.02\%$ across pCVR objectives) while reducing pCVR bias ($-58.8\%$, $-66.7\%$, $-37.3\%$).

Ablation Study

Component ablation reveals that both architectural paper reproduction and structured solution design are critical for high EPR. Removing structured paper analysis increases runnable-but-ineffective candidates (SFR $63.6\%$), demonstrating that successful literature-to-production transfer requires more than code synthesis. Removing the design stage that translates research architectures into production-compatible plans severely degrades search effectiveness (EPR $e_t \in \mathcal{E}$0). Quality assessment, multi-candidate proposal generation, and especially architecture-gradient feedback (forbidden direction learning) are all shown to be indispensable.

Case Studies: Production-Code Modification

NOVA's operational effectiveness is further illustrated via production code adaptation case studies. In one instance, NOVA evolves a more parameter-efficient TokenMixer variant, matching offline gains with only $e_t \in \mathcal{E}$1 of dense parameter cost via block number and residual connection adjustments. In another, architecture-gradient feedback corrects a misapplied auxiliary loss, balancing target-specific performance and averting over-specialization, ultimately converging to business-positive designs.

Implications and Future Work

The integration of architectural semantics and verification-aware agentic search marks a significant advance in AutoML/agentic system design for recommender architectures. NOVA demonstrates that effective, scalable automation in production demands:

• Iterative, feedback-encoding memory (trajectories of diagnostics and forbidden directions),
• Multi-source, task-aware signal aggregation (offline and online metrics, semantics, verification),
• Separation of architecture-level search spaces from naive hyperparameter or operator-tuning regimes.

Practically, this approach shifts the limiting factor on experiment throughput from human-bound design, debugging, and verification to pure GPU/batch training resources, enabling radical acceleration of architectural innovation cycles in industrial systems.

Theoretically, the architecture gradient mechanism suggests a generalizable approach to non-differentiable, discrete search in highly constrained, domain-specific spaces, with explicit semantic denoising in the update signal.

Figure 3: Roadmap for NOVA evolution towards full-funnel architecture search, full-lifecycle R&D, self-evolving skills, and resource-aware scheduling.

Immediate extensions identified by the authors include expansion to full-funnel recommender pipelines (retrieval to re-ranking) and end-to-end R&D agentic automation. There is also a clear trajectory towards self-evolving skill libraries and resource-adaptive budget allocation conditioned on real-time diagnostic feedback, which could further optimize system-wide resource expenditure and iteration velocity.

Conclusion

NOVA establishes a rigorous, production-ready paradigm for architecture evolution in recommender systems that tightly couples agentic proposal generation with verification-aware, gradient-driven search. Empirical gains in pass rates, production velocity, and online business impact validate its relevance for both industrial practitioners and agentic AutoML research. The outlined future roadmap provides a compelling foundation for integrated, fully-automated recommender R&D pipelines capable of sustaining continuous architecture innovation cycles.

---

Reference: "NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems" (2606.27243)