Metric Aggregation Divergence: A Hidden Validity Threat in Agent-Based Policy Optimization and a Contractual Remedy

Abstract: Metric aggregation divergence (MAD) is the silent inconsistency that arises when distinct pipeline stages in an agent-based model coupled with a multi-objective evolutionary algorithm (ABM+MOEA) independently re-implement how an outcome metric is extracted from simulation trajectories. Unlike deliberate analytical choices, MAD operates at the level of pipeline architecture: each stage is internally coherent, and the inconsistency becomes visible only when cross-stage outputs are compared. Code inspection of EpidemiOptim, a JAIR-published epidemic policy toolbox, reveals three structurally independent aggregation paths in peer-reviewed code. A faithful replication of this structure produces champion disagreement in 64.2% of independent runs (n=500, 95% CI: [59.9%, 68.3%]). In a 300-seed policy-flip experiment, divergent aggregation causes the optimizer to recommend the wrong champion in 83% of replications, with a mean welfare gap of 2.19 units and a Gini inequality gap of 0.050 units. In a follow-up inference audit, 3 of 249 flipped seeds cross the significance boundary itself. A complementary enterprise follow-up produces the predicted null under near-commensurable rankings (rho = 0.991), while a public upstream rerun of the Lake Problem DPS workflow shows that the archived published-path recommendation reaches joint-threshold success 0.401 whereas a shared contract-path rule reaches 0.552. We introduce the metric contract - a single shared callable enforced at dispatch time across all pipeline stages - as the remedy. Framed as standard engineering discipline applied to the cross-stage metric interface, the contract eliminates divergence by construction with approximately 3% runtime overhead.

• The paper shows that independent aggregation in ABM pipelines leads to policy reversals in up to 83% of runs, compromising outcome validity.
• It employs controlled experiments and code audits to reveal that modular aggregation strategies cause cross-stage inconsistencies and significant champion disagreements.
• The study proposes a 'metric contract' remedy that standardizes aggregation across all stages, eliminating divergence with minimal runtime overhead.

Metric Aggregation Divergence in ABM+MOEA Policy Optimization: Threats and Formal Remedies

Introduction

The paper "Metric Aggregation Divergence: A Hidden Validity Threat in Agent-Based Policy Optimization and a Contractual Remedy" (2606.29038) rigorously investigates latent validity risks in agent-based model (ABM) pipelines coupled with multi-objective evolutionary algorithms (MOEAs) for policy search. It identifies metric aggregation divergence (MAD) as a structural, non-obvious threat that can lead to inconsistent policy evaluation and suboptimal or misleading champion selection. The work characterizes the mechanisms and empirical consequences of MAD and proposes a formal, practical remedy—the metric contract—substantiated by experimental results across synthetic and real-world ABM+MOEA workflows.

Metric Aggregation Divergence: Conceptual and Formal Scope

The authors define MAD as arising whenever distinct pipeline stages (optimization, tournament evaluation, statistical inference) independently implement their own aggregation of outcome metrics from simulation trajectories. MAD fundamentally differs from researcher degrees of freedom or standard data bugs—it is an architectural non-decision, not an analysis choice. Each pipeline stage is internally consistent, so the error emerges only at cross-stage comparisons, which are not typically instrumented or audited.

MAD is formally defined as a scenario where two aggregators, $f$ and $g$, used by different pipeline stages, induce conflicting rankings over the policy space:

$$\exists \, \pi_1, \pi_2 : f(\tau_{\pi_1}) > f(\tau_{\pi_2}) \;\wedge\; g(\tau_{\pi_1}) < g(\tau_{\pi_2})$$

MAD is only active when the aggregators are incommensurable (e.g., mean vs. entropy) and the underlying episodes are non-monotone or exhibit temporally complex dynamics; monotone trajectories render all standard aggregators commensurate.

Mechanistic Investigation and Experimental Validation

The empirical core leverages a spectrum of controlled and real-world experimental cases to dissect the origins and impact of MAD.

Aggregator Distance as the Principal Driver

Systematic experiments demonstrate that the rank-reversal rate (RRR) between pipeline champions is dictated primarily by the functional "distance" between aggregator pairs, operationalized as $1-\rho$ (where $\rho$ is Spearman correlation across candidate policies). Aggregator distance ($\delta$), not episode-level stochasticity ($\tau$), explains the majority of variance in RRR:

$$\text{RRR} = \alpha + \beta_{\delta} \cdot \delta + \beta_{\tau} \cdot \tau + \varepsilon$$

with $\beta_{\delta} = 4.1 \times \beta_{\tau}$ and $R^2 = 0.769$.

Figure 1: Rank-reversal rate (RRR) and Cohen's $g$0 as functions of stochastic noise $g$1 across aggregation conditions, showing ad-hoc entropy aggregation leads to consistently higher RRR irrespective of noise.

Attempts to reduce MAD impact via increased sampling or reduced simulation noise are thus empirically futile if aggregator inconsistency persists.

Cross-ABM Replication and Null Cases

Replication across diverse ABMs (Schelling, Wolf–Sheep, Boltzmann) consistently shows high champion flip rates when candidate rankings induced by aggregators diverge due to non-monotone dynamics. Models with monotone trajectories (e.g., Wolf–Sheep under specific parameters) do not exhibit MAD, confirming the predicted mechanistic boundary.

Demonstrated Policy Consequences in Real Pipelines

A comprehensive code audit of EpidemiOptim, a JAIR-published toolbox, reveals three stage-dependent, structurally independent metric aggregation paths. This multiplexing of metric definitions, previously undetected, is faithfully replicated in a controlled experimental pipeline.

Across 500 independent ABM+MOEA replications:

• Champion disagreement rate: 64.2% (95% CI: [59.9%, 68.3%]).
• In a synthetic ecological policy task, divergent aggregation causes the optimizer to recommend the wrong champion in 83% of replications, with a substantial mean welfare gap of 2.19 units and Gini inequality gap of 0.050 units—both materially consequential.

  Figure 2: Policy-flip consequences of MAD—a representative NSGA-II Pareto front and distribution of welfare gaps over all champion flips.

Even the downstream inference (e.g., statistical significance) can shift due to MAD: in 3/249 affected cases, contraction to the correct metric reverses the reported significance boundary.

An audit of the Lake Problem DPS workflow, emblematic in water-resources management, affirms MAD: the archived published-path recommendation yields only 0.401 joint-threshold success, whereas the contract-path lifts performance to 0.552, driven by systematic candidate ranking errors due to aggregator inconsistency.

The Metric Contract: Engineering the Solution

The metric contract is a formalization of the operationalization consistency principle from measurement theory. It requires that all pipeline stages bind to a single registered callable implementing the aggregation function, enforced at dispatch and easily checked for byte-level identity. This is an application of dependency injection and Design by Contract at the aggregation interface.

Major attributes:

• Eliminates divergence by construction—champion flip rate drops to 0% under contract enforcement.
• Runtime overhead is negligible (approx. 3%), and does not impede efficiency or modularity.
• Easily retrofitted—can be introduced incrementally and validated in under five minutes by a coincidence check.

Accompanying the contract, the authors specify a practitioner-oriented 6-item reporting checklist targeting declaration, audit, scope-condition assessment, aggregator distance quantification, coincidence checks, and archival deposit of the contract object.

Discussion and Implications

This work exposes pipeline architecture as an unregistered, under-theorized locus of researcher degrees of freedom, parallel to statistical flexibility documented in empirical sciences. The risk is shown to be highly active: in code-available ABM+MOEA pipelines, observed divergence rates reach 56-75%.

The proposed contract is positioned not as a reimagining of modeling, but as rigorous engineering discipline analogous to statistical preregistration—explicitly fixing the outcome metric operationalization at pipeline assembly.

The study’s results highlight key domains for future concern:

• MAD is not mitigated by increased sampling, cross-validation, or sensitivity analysis unless aggregator consistency is enforced.
• Ensemble and parallel optimization pipelines (with heterogeneous metric definitions across workers) are especially vulnerable, with cross-worker champion disagreements observed at 100% in some cases.
• Dimensionality (e.g., in three-objective problems) does not alone suppress the problem.

Pragmatic adoption is straightforward for existing policy-modeling software frameworks, with immediate benefits in correctness and reproducibility.

Limitations and Future Directions

The generalizability of results to other ABM domains (energy, transportation) and more complex time-series metrics (beyond scalar episode summaries) remains to be directly evidenced. The full empirical payoff of contract-optimized candidate set regeneration versus same-candidate-set replays has not yet been demonstrated in artifact-rich, real policy domains.

Promising avenues for further research include formal diagnosis of temporal or structural features in simulation trajectories that mediate MAD severity, extending contract enforcement to ensemble/parallel optimization, and granular documentation tooling that integrates the reporting checklist.

Conclusion

This work identifies and rigorously characterizes MAD as a structural validity threat in ABM+MOEA policy pipelines—showing it is both highly prevalent and consequential for downstream policy recommendations. The metric contract remedy, requiring cross-stage aggregator binding, is validated to entirely suppress MAD at minimal practical cost. This formalization closes a critical gap in pipeline engineering for computational policy research, with immediate implications for auditability, reliability, and scientific transparency. Future progress will involve expanding contract-enabled optimization throughout complex, modular policy modeling domains and formalizing the relation between trajectory structure and aggregator commensurability.