Towards a Linear-Algebraic Hypervisor

Abstract: Many techniques in program synthesis, superoptimization, and array programming require parallel rollouts of general-purpose programs. GPUs, while capable targets for domain-specific parallelism, are traditionally underutilized by such workloads. Motivated by this opportunity, we introduce a pleasingly parallel virtual machine and benchmark its performance by evaluating millions of concurrent array programs, observing speedups up to $147\times$ relative to serial evaluation.

• The paper introduces a vectorized, algebraic virtual machine based on a modified RASP model for massive parallel program simulation.
• It employs a branchless, GPU-friendly execution model, achieving up to 147× speedup over serial CPU evaluation.
• The study demonstrates efficient compilation from a heapless array language, enabling extensive empirical analysis and program synthesis research.

Linear-Algebraic Hypervisor for Massive Parallel Program Simulation

Abstract Model and Motivation

The paper presents a vectorized virtual machine (VM) grounded in the RASP (Random-Access Stored Program) model, with an architecture specifically optimized for parallel execution of independent, general-purpose programs on massively parallel devices such as GPUs. While GPUs are well-exploited for domain-specific parallelism (e.g., graphics, dense linear algebra), they remain underutilized for workloads involving the concurrent rollout of diverse programs—common in areas like program synthesis, symbolic superoptimization, and array programming. The study introduces a formalization of a RASP variant and its 'word RASP' instantiation, establishing the theoretical and practical basis for an efficient, parallel, SIMD-compatible hypervisor.

RASP and Linear-Algebraic Formulation

The classic RASP is an abstract machine equipped with a vectorized configuration space, encoding program counter, accumulator, memory, input, and output. The transition function maps configurations stepwise, emulating atomic operations akin to realistic assembly, which is critical for program analysis and synthesis tasks.

The word RASP creates a finite, fixed-width, modular-memory setting: words reside in $\mathbb{F}_2^w$, with statically allocated memory regions for code, input, output, and scratch space, all under modulo arithmetic. The transition relation is encoded as a multilinear polynomial over $\mathbb{F}_2$—an algebraic representation ensuring that the entire compute step for $d$ VMs can be batched as coordinatewise polynomial maps, naturally vectorizable for GPU compute.

A notable technical element is the explicit design for branchless minimalism, essential for high-throughput, warp-efficient execution on GPUs, mitigating the inefficiencies that traditional branching induces on SIMD/SIMT hardware.

Compilation of Heapless Array Languages

The framework incorporates a novel lowering strategy from a heapless, statically analyzable array language to the word RASP's instruction set. The language supports first-order functions, sequencing, (bounded) loops, array indexing, and elementary arithmetic, with each construct equipped with formal operational semantics mapping to sequences of RASP instructions. The memory layout uses disjoint slices for inputs, outputs, and scratch, with all array accesses statically resolvable due to the heapless discipline and modulo-indexing.

This approach enables systematic compilation of high-level program synthesis benchmarks and combinatorial search spaces—where exhaustively evaluating millions of candidate straight-line or bounded programs is necessary—directly to the hypervisor with minimal interpretative overhead and predictable execution profiles.

Implementation and Empirical Results

Two VM backends are implemented: Kotlin/JVM for general-purpose CPU and CUDA for GPU execution (validated on Nvidia A10G/B200). The evaluation targets a challenging setup: 8 million random, syntactically valid array programs of length 100, each executed for up to $10^6$ steps or until reaching a halting configuration.

Key findings:

• The CUDA backend demonstrates up to 147× speedup over serial CPU evaluation for this workload.
• The system efficiently sustains high occupancy, running millions of independent VMs with bounded memory (constrained primarily by per-thread register budgets and VRAM capacity).
• The halting probability is empirically estimated, and the distribution of halting times for randomly generated heapless array programs is analyzed, revealing practical empirical behavior of program synthesis search spaces at substantial scale.

These results substantiate that the proposed algebraic VM formulation and hypervisor design not only permit leveraging modern GPUs for highly irregular, VM-intensive workloads—a key limitation in prior art—but also enable empirical studies that were previously computationally infeasible.

Theoretical and Practical Implications

The algebraic RASP and its hypervisor establish a foundation for the systematic exploitation of SIMD hardware for tasks in program synthesis, superoptimization, and enumerative search over instruction-level programs. By targeting a minimalist IR, the system maintains a low memory and implementation complexity footprint, in contrast to higher-level WASM or LLVM JIT-based approaches, and is directly suitable for algorithmic analysis and pedagogical settings.

Potential future applications include:

• Large-scale search for fast matrix multiplication algorithms via program enumeration and synthesis.
• Accelerated context-free grammar parsing and symbolic search over straight-line programs.
• Empirical analysis of halting distributions, busy beaver candidates, and other undecidable-property estimation in Turing-complete or sub-Turing models.
• Real-time, fine-grained evaluation of program sets for synthesis-guided program repair or optimization.

The construction of a coordinatewise polynomial, vectorized VM also hints at future integration with neural program synthesis—where differentiable approximations and joint search/optimization could benefit from such hypervising infrastructure.

Conclusion

This work provides a formal and empirical investigation into a linear-algebraic, GPU-resident hypervisor for exhaustive program simulation. By leveraging a concise algebraic abstract machine with a branchless execution model, the study demonstrates that it's feasible to parallelize the evaluation of millions of array programs with significant acceleration, enabling both theoretical and practical advances in empirical algorithmics, symbolic AI, and program synthesis research. The proposed methods offer a blueprint for future systems seeking to scale program analysis and synthesis tasks using modern parallel hardware.