Live Gaming Benchmark Overview

• Live gaming benchmarks are dynamic evaluation frameworks that use real-time game interactions to assess system performance, video quality, and AI agent capabilities.
• They measure runtime behavior, perceptual output, and interactive decision processes using metrics such as Instability Ratio, MOS, and win rates to capture complex system dynamics.
• Applications span online server stress tests, cloud rendering optimization, and AI-agent performance, offering actionable insights into both system stability and emergent gameplay behavior.

Live gaming benchmark denotes a class of evaluation frameworks in which games or game-derived workloads are used as dynamic, interactive testbeds for measuring system behavior under live conditions rather than through static datasets alone. Across recent work, the term spans at least three established uses: benchmarking the runtime behavior of online game servers and cloud-rendering stacks, benchmarking perceptual quality on gaming video streams and user-generated gaming videos, and benchmarking the reasoning, perception, planning, coordination, and adaptation capabilities of LLM or VLM agents through real-time gameplay (Eickhoff et al., 2021). Dynamic evaluation is a recurring design objective in this literature because static datasets are described as vulnerable to saturation or data contamination, while live game environments expose temporal dependence, control latency, multi-step planning, and variability that are suppressed by one-shot benchmarks (Hu et al., 2024).

1. Scope and taxonomy of live gaming benchmarks

The contemporary literature supports a broad taxonomy of live gaming benchmarks. Meterstick targets “Minecraft-like games” as operational systems with tightly looped server ticks, focusing on performance variability under player-based and environment-based workloads (Eickhoff et al., 2021). Pictor defines a benchmark suite for “interactive 3D applications in the cloud,” covering four desktop games and two VR titles, and instruments the end-to-end cloud graphics stack (Liu et al., 2020). LIVE-YT-Gaming, LIVE-Meta-MCG, and GameScope treat gaming video as the benchmark object, pairing subjective opinion scores with objective VQA evaluation (Yu et al., 2022). GameArena, VideoGameBench, lmgame-Bench, MindAgent, and OmniGameArena use live play as an evaluation substrate for AI models, but differ in whether they isolate reasoning, perception and control, multi-agent coordination, or improvement under reflection (Hu et al., 2024).

Benchmark	Primary object of evaluation	Representative paper
Meterstick	MLG server performance variability	(Eickhoff et al., 2021)
Pictor	Cloud interactive-3D application performance	(Liu et al., 2020)
LIVE-YT-Gaming / LIVE-Meta-MCG / GameScope	Gaming video quality assessment	(Yu et al., 2022)
GameArena / VideoGameBench / lmgame-Bench / MindAgent / OmniGameArena	LLM or VLM gameplay capability	(Zhang et al., 23 May 2025)

A central distinction in this taxonomy is whether the benchmark measures the game system, the delivered game media, or the agent acting in the game. Meterstick and Pictor instrument the runtime pipeline itself. LIVE-YT-Gaming, LIVE-Meta-MCG, GAMIVAL, and GameScope measure perceptual output quality under authentic or encoded distortions. GameArena and related agent benchmarks treat the game as a controlled interactive decision process in which outcomes, trajectories, and procedural traces can be scored (Chen et al., 2023).

This suggests that “live” is not tied to a single modality. In the cited work, it may refer to live server execution, live human-in-the-loop interaction, live emulator control, or live subjective viewing sessions. A plausible implication is that the common denominator is temporally extended interaction under controlled but non-static conditions.

2. Runtime and systems benchmarks for live game services

Meterstick formalizes the operational model of a Minecraft-like game as a tick-driven server nominally running at 20 Hz, with “Network Queues,” a “Game Loop,” and persistent “Game State.” The game loop includes a “Player Handler,” a “Terrain Simulator,” and an “Entity Simulator,” and the trace of $N$ consecutive tick durations $\{t_1,t_2,\dots,t_N\}$ is summarized by the sample mean and variance

$$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$

Meterstick distinguishes two orthogonal workload dimensions: player-based workloads, in which simulated bots connect as real clients and move in a bounded area, and environment-based workloads built from four curated world templates: “Control,” “TNT,” “Farm,” and “Lag Machine” (Eickhoff et al., 2021).

Its distinctive variability metric is the “Instability Ratio” (ISR), defined using nominal tick period $b=50\rm\,ms$, expected tick count $N_e$, and actual ticks $N_a$: $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$ ISR near $0$ indicates steady ticks, while ISR near $1$ indicates maximal alternation between nominal and arbitrarily large ticks (Eickhoff et al., 2021). The benchmark is implemented in a Controller/Worker pattern with SSH deployment to AWS, Azure, or DAS-5, 60 s iterations, and 50 repetitions per workload per environment per MLG.

The empirical findings are explicitly variability-centered. Under “Control,” the 95th-percentile response time can be $4.1\times$ the mean, with absolute spikes up to $\{t_1,t_2,\dots,t_N\}$0 the mean and $\{t_1,t_2,\dots,t_N\}$1 above the 118 ms “unplayable” threshold. Environment workloads dominate variability; on AWS, TNT and Farm worlds push ISR from approximately $\{t_1,t_2,\dots,t_N\}$2 to $\{t_1,t_2,\dots,t_N\}$3, with momentary tick durations up to $\{t_1,t_2,\dots,t_N\}$4 s. Entity simulation is reported as the dominant cost, accounting for more than $\{t_1,t_2,\dots,t_N\}$5 of non-idle tick time and more than $\{t_1,t_2,\dots,t_N\}$6 of state-update messages. The paper further reports that common “2 vCPU, 4 GB” recommendations are insufficient and that up-sizing to 8 vCPU is required to reduce mean tick below 50 ms and ISR below 0.05 for PaperMC or 0.15 for vanilla/Forge (Eickhoff et al., 2021).

Pictor addresses a different layer of the live gaming stack: cloud rendering for interactive 3D applications. It combines an “Intelligent Client Framework,” which uses real-time screen frames plus AI to simulate human gameplay, with a “Performance Analysis Framework” that tags every synthetic input and tracks it through the network stack, VNC/TurboVNC proxies, the CPU–GPU rendering pipeline, and back to the client display (Liu et al., 2020). Its six benchmarks are SuperTuxKart, 0 A.D., Red Eclipse, DOTA2, InMind, and IMHOTEP, covering racing, RTS, FPS, MOBA, and VR.

Pictor measures end-to-end frame latency, frame rate, latency distributions, and resource throughputs. Its basic formulas include

$\{t_1,t_2,\dots,t_N\}$7

and percentile latency $\{t_1,t_2,\dots,t_N\}$8 (Liu et al., 2020). The reported bottlenecks place server processing, rather than network transfer, at the center of cloud gaming latency: server processing dominates RTT at 61–106 ms, while network send is 14–35 ms and input send is below 10 ms. CPU workloads are described as memory bound, with L3 miss rates of 70–90%. Two optimizations—memoizing XGetWindowAttributes and a two-step asynchronous frame-copy—improved server FPS by 57.7% on average and reduced RTT by 8.5% average (Liu et al., 2020).

Together, these benchmarks establish a systems-oriented interpretation of live gaming benchmarking: the benchmark is not merely the game title, but the operational model, workload parameterization, per-stage instrumentation, and variability-aware metric design.

3. Subjective and objective benchmarks for gaming video quality

A second major branch of the literature uses live gaming content to benchmark perceptual video quality. LIVE-YT-Gaming was introduced as a benchmark for UGC gaming video quality assessment, with 600 distinct UGC gaming clips, each lasting 8–9 s, drawn from 59 different game titles and covering 360p, 480p, 720p, and 1080p at 30 fps or 60 fps (Yu et al., 2022). The distortions are authentic mixtures rather than synthetic single-factor degradations, including screen-recording compression artifacts, bitrate variability due to live broadcast, frame stalls, temporal freezes, chromatic and luminance noise, and YouTube re-encoding. Subjective labels were obtained in a controlled online study using 61 vetted but otherwise naïve viewers; each video was rated by approximately 30 distinct subjects, yielding 18,600 individual opinion scores transformed to MOS in accordance with ITU-P.910 guidelines (Yu et al., 2022).

The earlier database description emphasizes content-diversity checks using spatial information and temporal information,

$\{t_1,t_2,\dots,t_N\}$9

and reports final MOS in the range $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$0, inter-subject median SROCC of approximately $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$1, and intra-subject median SROCC of approximately $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$2 (Yu et al., 2022).

GAME-VQP is a blind VQA model designed for LIVE-YT-Gaming. It combines NSS-based features and gaming-specific CNN features. The NSS pipeline includes conversion from sRGB to CIELCh, MSCN normalization

$$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$3

GGD fitting over 42 processed coefficient maps, and multiscale doubling to 168 NSS features per video. The semantic branch uses a frozen DenseNet-201 backbone and averages 1920-dimensional final global average pooling features across frames. Two independent $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$4-SVR regressors produce $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$5 and $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$6, and the final score is

$$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$7

On LIVE-YT-Gaming, the reported median over 100 random 80/20 splits is SROCC $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$8, PLCC $$\mu \;=\;\frac1N\sum_{i=1}^N t_i,\qquad \sigma^2 \;=\;\frac1N\sum_{i=1}^N\bigl(t_i-\mu\bigr)^2.$$9, and RMSE $b=50\rm\,ms$0, with one-sided Wilcoxon rank-sum tests at 95% confidence indicating statistical superiority over all comparison models in both SROCC and PLCC (Yu et al., 2022).

LIVE-Meta-MCG extends the live gaming benchmark concept to mobile cloud gaming. The database contains 600 landscape and portrait gaming videos derived from 30 reference clips taken from 16 cloud-rendered games, encoded at four resolutions and five bitrates, and rated by 72 university volunteers for 14,400 subjective quality ratings (Saha et al., 2023). Distortions are generated through spatial resizing and H.264 compression under constant-bit-rate settings, with no time-varying network loss introduced. The final video scores are “MLE-MOS,” derived from a Li and Bampis maximum-likelihood model of observer bias, inconsistency, and content ambiguity (Saha et al., 2023).

GAMIVAL was proposed for the LIVE-Meta MCG benchmark. It combines spatial gaming distorted scene statistics, temporal NSS on Haar-filtered frame-difference subbands, additive “neural noise” regularization with $b=50\rm\,ms$1, and DenseNet-121 semantic features pretrained in NDNetGaming (Chen et al., 2023). Its input to an RBF-kernel $b=50\rm\,ms$2-SVR is a 2180-dimensional feature vector $b=50\rm\,ms$3. The reported median performance over 1000 content-wise train/test splits is SRCC $b=50\rm\,ms$4, PLCC $b=50\rm\,ms$5, KRCC $b=50\rm\,ms$6, and RMSE $b=50\rm\,ms$7, outperforming VSFA, RAPIQUE, GAME-VQP, and NDNet-Gaming on the benchmark (Chen et al., 2023).

GameScope scales this line of work to 4,048 encoded clips derived from 424 unique 10-second source sequences, with approximately even distribution between UGC and PGC content and support for H.264, H.265, and AV1 (Sureddi et al., 2 May 2026). Each clip is annotated by an average of 37 MOS ratings, and coarse-grained attributes are also collected for “Clarity,” “Pixelation & Blockiness,” and “Immersive Game Experience.” In its test split, representative metrics include PLCC/SROCC pairs of $b=50\rm\,ms$8 for PSNR, $b=50\rm\,ms$9 for VMAF, $N_e$0 for GAMIVAL, and $N_e$1 for Qwen3-VL-4B (Sureddi et al., 2 May 2026).

This branch of the literature treats live gaming benchmarking as the construction of diverse gaming-video corpora with authentic distortions, controlled subjective protocols, and standardized objective comparisons. A recurring conclusion is that gaming-specific statistics differ materially from natural video statistics and that hybrid NSS-plus-deep approaches are consistently competitive or dominant (Yu et al., 2022).

4. Live game benchmarks for LLM and VLM evaluation

GameArena defines a dynamic benchmark for evaluating LLM reasoning capabilities through live computer games with humans. Its web-based frontend serves three “games with a purpose”: Akinator, Taboo, and Bluffing. These are designed to isolate deductive and multi-hop reasoning, abductive and multi-hop reasoning, and inductive and multi-hop reasoning, respectively (Hu et al., 2024). The benchmark recruits participants through CloudResearch, pairs five LLM endpoints with optimized system prompts found via DSPy, and uses a retrospective analysis pipeline that re-prompts the same model at each turn to extract hidden chain-of-thought outputs such as object lists, word lists, or truthfulness judgments (Hu et al., 2024).

The games are formalized as interactive decision processes. At turn $N_e$2, the state comprises the system prompt, user messages, and model outputs; actions are ordinary questions or answers, or a final guess $N_e$3; the game ends when $N_e$4 or the round cap $N_e$5 is reached; and payoff is binary. Outcome metrics include

$N_e$6

Procedural metrics include RecallRate, Top-$N_e$7 Recall, DisparityRatio, AvgFirstAppear, AvgFinalRank, Spearman’s $N_e$8 over convergence in Bluffing, and HoppingPenalty (Hu et al., 2024). The reported dataset contains 2,240 total sessions over 10 weeks, and 86.9% of GameArena sessions were “useful” versus 4% of Chatbot Arena conversations with votes. The paper also reports that GameArena’s procedural rankings on deductive and abductive metrics correlate strongly with LiveBench-Reasoning and GPQA, with $N_e$9, RBO $N_a$0, and $N_a$1 (Hu et al., 2024).

VideoGameBench evaluates VLMs on ten popular video games from the 1990s through direct real-time interaction with raw pixel frames and only a high-level description of objectives and controls (Zhang et al., 23 May 2025). Three games are kept secret to encourage generalization to unseen environments. The benchmark intentionally avoids game-specific scaffolding, RAM inspection, handcrafted overlays, and intermediate rewards. Progress is measured via checkpoint matching based on representative frames from human playthrough videos: $N_a$2 The reported real-time results are extremely low: Gemini 2.5 Pro achieves 0.48% overall, GPT-4o 0.09%, and several models 0%. In the paused “Lite” setting, GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro each achieve 1.6% overall on the three Lite games (Zhang et al., 23 May 2025). The reported failure modes are the “knowing–doing gap,” perceptual errors, memory and planning breakdown, and latency-induced staleness.

lmgame-Bench takes the opposite design stance on several points. It argues that directly dropping LLMs into games cannot make an effective evaluation because of brittle vision perception, prompt sensitivity, and potential data contamination. It therefore provides a unified Gym-style API, lightweight perception and memory scaffolds, and prompt standardization via empirical formatting plus DSPy-based SIMBA optimization (Hu et al., 21 May 2025). The suite includes Super Mario Bros., Tetris, Sokoban, Candy Crush, 2048, and Ace Attorney. The benchmark reports that prompt optimization reduces variance by up to 63.5%, and contamination mitigation on Ace Attorney breaks the predictive link between text similarity and score after masking, paraphrasing, and enforced causal reasoning (Hu et al., 21 May 2025). With the harness, o3 and o1 are reported as top performers, and reinforcement learning on a single game transfers to unseen games and external planning tasks such as Blocksworld and WebShop (Hu et al., 21 May 2025).

OmniGameArena introduces a UE5 benchmark for VLM game agents that includes Solo, PvP, and Coop regimes under a common action API and adds the “Improvement Dynamics Curve” (IDC) (Lin et al., 8 Jun 2026). The twelve purpose-built games are authored from scratch to avoid pre-training leakage. For frozen policy $N_a$3 and skill prompt $N_a$4, each round evaluates

$N_a$5

Held-out generalization is computed as $N_a$6 over $N_a$7 task variants under the best skill prompt (Lin et al., 8 Jun 2026). On the cold-start leaderboard, GPT-5.5 is reported as best in Solo mean $N_a$8 std at $N_a$9, PvP average win rate $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$0, and Coop mean $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$1 std $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$2. Under IDC, all four top agents improve over $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$3, but LastStand gains often peak at rounds $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$4 and then drift down, motivating best-skill rollback (Lin et al., 8 Jun 2026).

MindAgent and CuisineWorld move the focus from single-agent gameplay to centralized scheduling and human–NPC collaboration. MindAgent is a centralized LLM coordinator that dispatches $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$5 agents through a textual DSL comprising goto, get, put, activate, and noop, with prompt modules for recipes, instructions, hints, one-shot demonstration, current state, and memory history (Gong et al., 2023). The associated benchmark defines the Collaboration Score: $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$6 where $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$7 and $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$8 are completed and failed tasks across different task-arrival intensities $$\mathrm{ISR} \;=\; \frac{\displaystyle\sum_{i=1}^{N_a}\bigl|\max(b,t_i)\;-\;\max(b,t_{i-1})\bigr|} {2\,b\,N_e} \quad\in[0,1].$$9 (Gong et al., 2023). Evaluation spans GPT-4, Claude-2, GPT-3.5-turbo, and LLaMA-2-70B-chat, includes human–AI teaming, and reports that GPT-4 $0$0 Claude-2 $0$1 GPT-3.5 $0$2 LLaMA-2 in cross-model comparison (Gong et al., 2023).

Across these AI-oriented benchmarks, live gameplay is used not merely as a source of overall scores but as a structured generator of trajectories, reflection signals, reasoning traces, and human-in-the-loop evidence.

5. Metrics, protocols, and recurrent methodological design choices

Despite targeting different objects, live gaming benchmarks exhibit recurring methodological structure. First, they define a low-level interaction model. Meterstick uses ticks and sub-phases within the game loop (Eickhoff et al., 2021). Pictor uses tagged inputs and hook-based timestamps across client, proxy, application, and GPU stages (Liu et al., 2020). GameArena formalizes each conversational game as an interactive decision process (Hu et al., 2024). VideoGameBench and OmniGameArena use real-time action loops with visual observations and action APIs (Zhang et al., 23 May 2025). lmgame-Bench uses a Gymnasium-compatible MDP interface with reset() and step(a_i) (Hu et al., 21 May 2025).

Second, they emphasize multidimensional metrics rather than single aggregate scores. Meterstick combines mean latency, variance, peak-to-mean ratio, tail-latency ratios, ISR, sub-phase tick distributions, and system-level CPU, memory, thread-count, disk I/O, and network I/O (Eickhoff et al., 2021). Pictor measures RTT, FPS, percentile latency, PMU counters, PCIe throughput, and CPU/GPU utilization (Liu et al., 2020). GameArena separates outcome metrics from procedural metrics such as RecallRate, AvgFinalRank, DisparityRatio, or HoppingPenalty (Hu et al., 2024). OmniGameArena explicitly argues that one should report “score trajectories, improvement rates, convergence indices, held-out generalization—rather than single numbers” (Lin et al., 8 Jun 2026).

Third, most benchmarks enforce repeated trials and standardized splits. Meterstick uses 50 repetitions per workload per environment per MLG (Eickhoff et al., 2021). GAME-VQP reports medians over 100 random 80/20 train/test splits (Yu et al., 2022). GAMIVAL and LIVE-Meta-MCG report medians over 1000 random content-wise 80/20 train/test splits (Chen et al., 2023). GameArena aggregates over 2,240 sessions (Hu et al., 2024). OmniGameArena’s cold-start leaderboard uses $0$3 episodes per cell and IDC uses $0$4 rounds with $0$5 episodes each (Lin et al., 8 Jun 2026).

Fourth, contamination resistance is a recurrent design principle, though implemented differently. GameArena motivates live games partly because static datasets are vulnerable to contamination (Hu et al., 2024). OmniGameArena’s twelve games are “authored from scratch to avoid pre-training leakage” (Lin et al., 8 Jun 2026). VideoGameBench withholds three games on an evaluation server (Zhang et al., 23 May 2025). lmgame-Bench explicitly measures contamination through frame-order and Sentence-BERT similarity analyses and then masks entities and paraphrases narrative content (Hu et al., 21 May 2025).

Fifth, benchmark authors repeatedly distinguish realism from synthetic simplification. LIVE-YT-Gaming rejects synthetic “one-distortion-only” processing and instead uses authentic mixtures of distortions (Yu et al., 2022). Meterstick argues that pure player-count workloads understate MLG server stress and therefore includes farms, TNT, redstone circuits, and community maps (Eickhoff et al., 2021). VideoGameBench emphasizes raw visual inputs without game-specific toolkits or overlays (Zhang et al., 23 May 2025). By contrast, lmgame-Bench deliberately introduces symbolic perception and memory scaffolds in order to stabilize evaluation and reduce confounds from brittle vision (Hu et al., 21 May 2025). This is a substantive design divergence rather than a contradiction: one line of work measures end-to-end embodied capability, while the other isolates reasoning under controlled perceptual assistance.

6. Findings, controversies, and directions implied by the literature

Several consistent empirical findings emerge across these benchmarks. In systems benchmarking, variability is often more consequential than average performance. Meterstick states that environment-based workloads and cloud deployment are significant sources of performance variability and recommends always reporting ISR and latency-tail metrics alongside means (Eickhoff et al., 2021). Pictor finds that server-side processing, memory-bound CPU behavior, and frame-copy stalls dominate cloud interactive-3D performance, indicating that network transport is not the sole or even primary bottleneck in these workloads (Liu et al., 2020).

In video-quality benchmarking, gaming content repeatedly violates the assumptions of natural-scene-statistics methods designed for photographic content. LIVE-YT-Gaming reports that synthetic gaming frames have more sharply peaked and heavy-tailed MSCN histograms than photographic videos, and both LIVE-YT-Gaming and LIVE-Meta-MCG show that hybrid NSS-plus-deep models such as GAME-VQP and GAMIVAL outperform standard blind quality predictors (Yu et al., 2022). GameScope extends the problem to cross-codec generality and attribute-level interpretation, with a zero-shot VLM outperforming established metrics on overall MOS prediction (Sureddi et al., 2 May 2026).

In AI-agent benchmarking, current models remain weak in live game interaction unless substantial scaffold design is added. VideoGameBench reports that frontier VLMs struggle to progress beyond the beginning of each game and identifies inference latency as a major limitation (Zhang et al., 23 May 2025). GameArena, however, shows that live gaming can still yield high-quality reasoning traces and useful session rates in a human-in-the-loop setting (Hu et al., 2024). lmgame-Bench argues that direct, unscaffolded play produces unreliable evaluations because brittle vision perception, prompt sensitivity, and contamination dominate outcomes, and its harness is intended to turn games into reliable evaluations (Hu et al., 21 May 2025). OmniGameArena then adds a further layer by showing that cold-start scores miss learning dynamics, mid-curve peaking, and variant transfer behavior (Lin et al., 8 Jun 2026).

A recurring controversy concerns what exactly a live gaming benchmark should test. One design philosophy minimizes scaffolding to preserve end-to-end realism, as in VideoGameBench (Zhang et al., 23 May 2025). Another introduces symbolic perception, memory support, prompt optimization, or retrospective reasoning extraction to isolate specific target capabilities, as in lmgame-Bench, GameArena, and MindAgent (Hu et al., 21 May 2025). This suggests that benchmark interpretation depends on whether the goal is ecological validity, diagnostic granularity, or controllable reproducibility.

The literature also points to future extensions. GAME-VQP explicitly recommends incorporating network-level metrics such as latency and jitter into end-to-end quality predictors for cloud-streamed interactive gaming (Yu et al., 2022). OmniGameArena recommends including Solo, PvP, and Coop regimes and releasing both cold-start leaderboards and an IDC-style self-reflection harness (Lin et al., 8 Jun 2026). MindAgent recommends defining a clear minimal DSL, providing immediate environment feedback, and parameterizing load to stress-test schedulers (Gong et al., 2023). GameScope proposes extending the codec ladder to VVC and AV2 and adding cloud gaming service captures under identical subjective conditions (Sureddi et al., 2 May 2026).

Taken together, the research literature characterizes the live gaming benchmark as a general experimental paradigm rather than a single benchmark family. Its defining properties are dynamic interaction, explicit temporal structure, controlled observability, and metrics that capture not only level of performance but also variability, procedural behavior, or longitudinal improvement.