Multi-Version Experimental Evaluation (MVEE)

• Multi-Version Experimental Evaluation (MVEE) is an evaluation method that treats distinct compiled outputs as separate experimental objects to expose compiler-induced variability.
• It employs assembly-level analysis and truediff-based structural diffing to monitor and track build anomalies across successive compilations.
• MVEE enhances benchmarking fairness by requiring performance results to be interpreted against the full set of relevant compiled versions rather than a single arbitrary build.

Multi-Version Experimental Evaluation (MVEE) is an experimental methodology that rejects the assumption that a method or baseline has a single stable compiled form during an optimization workflow. In its most specific recent formulation, MVEE detects all distinct compiled versions that arise across builds, treats those versions as separate experimental objects, and attaches benchmark results to the version that actually produced them, thereby making compiler-induced variability explicit rather than latent (Jörz et al., 29 Mar 2026). Related literatures use analogous multi-version reasoning at other granularities—notably common-framework algorithm bake-offs, continuously evolving benchmark releases, multi-version model histories, and longitudinal software datasets—so the term is best understood as a family of evaluation practices centered on version awareness rather than as a single domain-bound procedure (Bagnall et al., 2016, Liu et al., 2 Jul 2026, Barkowsky et al., 2023, Liu et al., 2021).

1. Motivation and problem setting

The immediate motivation for MVEE arises in database and systems research, where a new method and its baselines are often integrated into one shared benchmarking codebase, compiled into a single binary, and compared repeatedly while the new method is optimized (Jörz et al., 29 Mar 2026). That workflow assumes that unchanged baseline source code yields a single compiled version across builds. MVEE is motivated by the observation that this assumption is delicate: even when the source code of a baseline is untouched, general purpose compilers can generate highly different compiled code across builds because of inlining decisions, link-time and interprocedural optimizations, profile-guided optimizations, and heuristic changes in register allocation, code layout, and loop optimization (Jörz et al., 29 Mar 2026).

A concrete case was reported in prior work on scan-accelerating index structures in main-memory databases, where a baseline method $B$ appeared in two distinct compiled versions, $B.\mathrm{V0}$ and $B.\mathrm{V1}$, although the source code of $B$ never changed (Jörz et al., 29 Mar 2026). The differences were found at the assembly level and were attributable to seemingly unrelated changes elsewhere in the codebase. The paper uses this case to identify two failure modes of traditional evaluation. The first is merging results from different builds: if different selectivities are benchmarked under different compiled versions of the baseline, a plot can mix incomparable measurements and support a false conclusion. The second is using a single arbitrary build: results are internally consistent, but the chosen compiled version may omit another observed version of the same baseline that is faster on part of the workload (Jörz et al., 29 Mar 2026).

The central implication is methodological rather than merely anecdotal. If performance claims depend on an untracked compiler state, then baseline comparison is not only noisy but structurally ambiguous. MVEE answers that ambiguity by materializing compiled-version multiplicity and making it part of the experiment definition.

2. Core definitions and formal model

MVEE models each monitored method $i$ as a set of compiled versions,

$V_i = \{ v_{i,0}, v_{i,1}, \dots, v_{i,k_i} \},$

where each $v_{i,j}$ corresponds to a distinct assembly sequence produced by some build (Jörz et al., 29 Mar 2026). For a workload or benchmark configuration $w$, the runtime of version $j$ of method $i$ is denoted

$B.\mathrm{V0}$0

Traditional single-version evaluation implicitly assumes $B.\mathrm{V0}$1. MVEE makes $B.\mathrm{V0}$2 explicit and requires claims about a method $B.\mathrm{V0}$3 versus a baseline $B.\mathrm{V0}$4 to be interpreted against the ensemble $B.\mathrm{V0}$5, not against one arbitrary build artifact (Jörz et al., 29 Mar 2026).

The mechanism that generates new versions is the build anomaly. The paper defines a build anomaly as the case in which the source code of a monitored method is unchanged between two builds, yet the compiler produces non-equivalent assembly code for that method (Jörz et al., 29 Mar 2026). Let $B.\mathrm{V0}$6 and $B.\mathrm{V0}$7 be two assembly instruction sequences for the same source section in two builds. They are equivalent only if all three of the following hold:

1. Instruction set equality: every instruction in $B.\mathrm{V0}$8 also exists in $B.\mathrm{V0}$9 and vice versa.
2. Same data: each instruction in $B.\mathrm{V1}$0 operates on the same data as the corresponding instruction in $B.\mathrm{V1}$1.
3. Same control flow: the control flow of $B.\mathrm{V1}$2 is the same as of $B.\mathrm{V1}$3. (Jörz et al., 29 Mar 2026)

If any condition fails, MVEE treats the new assembly as a distinct compiled version. The implementation relaxes some checks for efficiency: indirect jumps based on register contents are ignored, allowing static analysis, and register assignments are ignored, so anomalies are detected at the instruction and control-flow level rather than at the exact register-allocation level (Jörz et al., 29 Mar 2026).

Version evolution is then represented by a version graph. Source code modifications continue a path and replace older versions; anomalies fork paths and create additional versions that must be considered concurrently (Jörz et al., 29 Mar 2026). At interpretation time, MVEE identifies the currently relevant versions by traversing the graph and selecting the first version on each path that is reachable without passing through a source modification that invalidates older versions (Jörz et al., 29 Mar 2026). This version-graph semantics is the key distinction between “a history of builds” and “a set of versions that remain experimentally relevant.”

3. Architecture and implementation

The reference implementation described for MVEE consists of two components: a VSCode extension and an MVEE core (Jörz et al., 29 Mar 2026). The extension is the user-facing interface, orchestrating compilation, anomaly inspection, version graph visualization, and plotting. The core performs assembly-level analysis and equivalence checking, implements structural diffing using the truediff algorithm extended for x86 assembly, maintains intermediate representations, and performs anomaly detection (Jörz et al., 29 Mar 2026).

The workflow is a compile-analyze-benchmark loop. A user triggers “compile & run” from the IDE; the build system produces a new binary and an associated .s assembly file, which is stored in a directory indexed by timestamp (Jörz et al., 29 Mar 2026). The extension asks the core to analyze all methods whose source code did not change since the previous build. The core compares the relevant assembly regions, returns equivalence results and edit scripts, and the extension allows anomaly inspection directly in the IDE, with links from source lines to assembly differences (Jörz et al., 29 Mar 2026). The binary is then run to produce performance results. If an anomaly was detected for method $B.\mathrm{V1}$4, the new result is recorded as a new version $B.\mathrm{V1}$5; otherwise the prior version’s result is updated (Jörz et al., 29 Mar 2026).

Monitored code sections are explicitly marked in the source. The paper uses preprocessor marks around a run function:

size_t input_B1 = 42;
gen_begin_mark(B1, size_t, input_B1);
size_t res_B1 = run_B1(input_B1); // monitored for anomalies
gen_end_mark(B1, size_t, res_B1);

Here, B1 is the logical identifier for the method, and gen_begin_mark / gen_end_mark create a dependency between the method’s inputs and outputs and the instrumentation, enabling MVEE to locate the relevant assembly regions (Jörz et al., 29 Mar 2026).

Assembly analysis proceeds in three stages. First, the core parses x86 assembly into a generic intermediate representation, extracts only the portions corresponding to monitored code sections, and builds the control flow graph from each start mark to its corresponding end mark (Jörz et al., 29 Mar 2026). To preserve control flow, instructions are grouped into fallthrough-groups: sequences that can be executed one after another without an unconditional jump or call in between (Jörz et al., 29 Mar 2026). Second, the resulting fallthrough-group trees are given to truediff, which returns an edit script in the truechange format, including insertions, deletions, operand updates, and reorderings (Jörz et al., 29 Mar 2026). Third, MVEE checks the edit script for violations of the three equivalence conditions: significant structural edits, data inconsistency in immediates or memory references, and control-flow violations caused by impermissible reordering (Jörz et al., 29 Mar 2026). Any such violation is reported as a build anomaly.

4. Evaluation semantics and interpretation

MVEE changes not only instrumentation but also the semantics of reporting. Each benchmark result is attached to a specific version: $B.\mathrm{V1}$6 where $B.\mathrm{V1}$7 is the method, $B.\mathrm{V1}$8 the version index, and $B.\mathrm{V1}$9 the workload parameters (Jörz et al., 29 Mar 2026). Performance reporting therefore operates over sets of relevant versions rather than over a single line in a results table.

The paper proposes summarizing performance over the relevant version set $B$0 with version-aware statistics such as

$B$1

$B$2

and

$B$3

The point is not that one aggregation is universally correct, but that conclusions such as “$B$4 is faster than baseline $B$5” must be evaluated against all observed versions of $B$6, not just the version emitted by a convenient build (Jörz et al., 29 Mar 2026).

The principal case study is the storage views project, where the two compiled versions $B$7 and $B$8 of a baseline produced significant performance differences at different selectivities (Jörz et al., 29 Mar 2026). In some selectivities, $B$9 outperformed $i$0 but not $i$1. Traditional reporting could therefore support the claim that $i$2 “wins globally,” whereas MVEE yields a more constrained conclusion: $i$3 wins over some compiled versions and workloads, but loses against others (Jörz et al., 29 Mar 2026). The broader methodological consequence is a shift from point comparison to version-set comparison.

This logic resonates with earlier large-scale empirical evaluation traditions. In time series classification, “The Great Time Series Classification Bake Off” implemented 20 classifiers in a common Java/WEKA framework, evaluated them on 85 datasets, performed 100 resampling experiments per dataset, and used Friedman and Nemenyi tests to determine which classifiers were significantly more accurate than the benchmarks and each other (Bagnall et al., 2016). That study did not track compiler-induced build anomalies, but it embodied the same demand for common infrastructure, repeated evaluation, shared preprocessing, and reproducible statistical comparison that later becomes explicit in MVEE’s build-aware form.

5. Related meanings and adjacent research programs

The literature uses “multi-version” evaluation and closely related acronyms in several technically distinct ways. The common thread is that version multiplicity is treated as part of the object of study rather than as noise to be abstracted away.

Context	Versioned entity	Role in evaluation
Database benchmarking	Compiled method versions across builds (Jörz et al., 29 Mar 2026)	Detect build anomalies and evaluate all relevant compiled versions
Algorithm bake-offs	Alternative classifier implementations in a common framework (Bagnall et al., 2016)	Compare many methods under one protocol
Live benchmarks	Successive benchmark releases (Liu et al., 2 Jul 2026)	Preserve cross-version comparability while updating data
Model-driven engineering	Multi-version models and transformation histories (Barkowsky et al., 2023)	Jointly transform and analyze many versions
Defect prediction	Multi-version-project datasets (Liu et al., 2021)	Evaluate models across releases while auditing label quality
Browser and systems security	Concurrently executing software variants (Venkateshwaran et al., 2021, Volckaert et al., 2016)	Synchronize versions or variants under shared inputs

In multimodal evaluation, MMBench-Live operationalizes a benchmark-centric form of multi-version evaluation. It treats benchmark evolution as task-guided dataset construction, introduces a distribution-consistent update strategy, reports Pearson and Spearman correlations between old and live benchmark versions, and uses PaCoST-based contamination analysis to show weaker memorization-related signals in the updated benchmark (Liu et al., 2 Jul 2026). This is not the same as build-anomaly tracking, but it addresses the parallel problem that a static benchmark version can become stale or contaminated while remaining superficially “the same benchmark.”

In model-driven engineering, multi-version models provide a compact representation of version histories, and triple graph grammars are adapted so that one multi-version forward transformation is semantically equivalent to independent single-version transformations projected per version (Barkowsky et al., 2023). The reported speedups—a factor of 4 on a small repository and a factor of 74 on a larger one—suggest that version-aware evaluation can also be computationally advantageous when many versions share structure (Barkowsky et al., 2023).

In defect prediction, multi-version-project datasets reveal a different risk: label quality can degrade precisely because multiple versions are available. One extensive empirical study reported that more than 90% of investigated versions contained inconsistent labels and argued that such inconsistencies can considerably change prediction performance, the identification of true defective modules, and the importance ranking of independent variables (Liu et al., 2021). That result does not redefine MVEE, but it shows that multi-version evaluation is only as reliable as the versioned data to which it is applied.

A distinct terminological branch concerns execution rather than evaluation. SINATRA uses “Multi-Version eXecution (MVX)” to keep an old and an updated browser synchronized by replaying the same JavaScript non-deterministic events on both versions (Venkateshwaran et al., 2021). Security-oriented “Multi-Variant Execution Environments (MVEEs)” run diversified replicae in lock-step and monitor system calls to detect divergence indicative of memory-corruption attacks (Volckaert et al., 2016). These systems share the multi-version or multi-variant premise but address runtime fault tolerance, updating, or security, not experimental benchmarking in the narrow sense.

6. Methodological implications, limitations, and open problems

MVEE’s strongest methodological claim is that unchanged source code does not imply a unique baseline. The corresponding misconceptions are explicit in the literature. First, a clean optimization history does not guarantee stable baselines, because unrelated changes can alter code generation elsewhere in the codebase (Jörz et al., 29 Mar 2026). Second, a single arbitrary build guarantees consistency, but not fairness, because it may exclude another observed compiled version of the same method that is faster on part of the workload (Jörz et al., 29 Mar 2026). Third, static benchmark releases are not automatically adequate for long-term evaluation, because temporal staleness and contamination can distort results even when benchmark identity appears unchanged (Liu et al., 2 Jul 2026).

The reference MVEE implementation also has concrete limitations. Assembly parsing and structural diffing must scale to large codebases and many builds, so the system monitors only marked code sections and currently focuses on x86/gcc-style assembly (Jörz et al., 29 Mar 2026). MVEE adds analysis steps between compilation and execution, which can introduce noticeable overhead in rapid development cycles (Jörz et al., 29 Mar 2026). Long-running projects may accumulate many versions in the version graph, raising questions about pruning, summarization, and presentation (Jörz et al., 29 Mar 2026). These are not conceptual objections to MVEE, but they delimit its current engineering envelope.

Across adjacent areas, the limitations are different but structurally related. MMBench-Live notes residual memorization, bounded automation quality, and task-space expansion limits in continuously evolving benchmarks (Liu et al., 2 Jul 2026). Multi-version model transformation depends on efficient version-set operations and, in the reported correctness proof, on deterministic TGGs without critical pairs (Barkowsky et al., 2023). Multi-version-project defect prediction requires explicit detection and exclusion of inconsistent labels if evaluation conclusions are to remain interpretable (Liu et al., 2021). Taken together, these studies suggest that version awareness improves evaluation only when the semantics of “version” are formalized and the mechanism generating versions is itself audited.

A plausible implication is that MVEE will continue to broaden from a build-aware benchmarking technique into a more general principle of empirical design: when versions exist and materially affect outcomes, they should be modeled, tracked, and reported rather than hidden behind a single nominal artifact. In the narrow database-and-systems sense, however, MVEE retains a precise core: it is a methodology and toolchain for detecting build anomalies at the assembly level, materializing them as new versions, and including all observed versions of each method in experimental performance evaluation (Jörz et al., 29 Mar 2026).