HARNESS-LM (HLM): Modular LLM Scaffolding

Updated 5 July 2026

HARNESS-LM is a modular framework that integrates perception, memory, and reasoning around a primary LLM/VLM for multi-turn gaming and retrieval applications.
It establishes a plug-and-play scaffold that constructs context, mediates tool exposure, and standardizes prompts, enhancing decision-making and execution.
Empirical evaluations reveal substantial performance gains in gaming and sponsored-search tasks, emphasizing the critical impact of harness design on output reliability.

HARNESS-LM (HLM) is an overloaded designation in recent arXiv literature. In the sense most directly mapped by the paper "General Modular Harness for LLM Agents in Multi-Turn Gaming Environments," HLM denotes a general-purpose, plug-and-play scaffold that composes perception, memory, and reasoning modules on top of a single backbone LLM or VLM for multi-turn gaming. In adjacent literatures, however, the “harness” denotes the execution layer that constructs context, mediates tools, orchestrates control, and verifies outputs; one later paper also uses HARNESS-LM as the formal name of a three-phase training recipe for sponsored search retrieval. The term therefore refers both to a specific modular gaming-agent framework and to a broader harness-centric view of LLM systems, and it cannot be interpreted reliably without reference to the underlying paper (Zhang et al., 15 Jul 2025, Zhang et al., 7 May 2026, Gupta et al., 22 May 2026).

1. Definition, scope, and nomenclature

The gaming-agent usage is the most direct mapping from the label “HARNESS-LM (HLM)” in the supplied literature. The source paper itself does not introduce the formal name “HARNESS-LM (HLM),” but it consistently describes a “modular harness” for LLM/VLM agents, and that harness is explicitly mapped onto HLM: a domain-agnostic scaffold that composes perception, memory, and reasoning around a single backbone model for multi-turn gaming environments (Zhang et al., 15 Jul 2025).

A separate line of work uses “harness” in a more general systems sense. There, the harness is the infrastructure layer between the model and the task: it determines context construction, tool exposure, orchestration, scheduling, memory handling, and verification. In that literature, HLM is best understood not as a single named model but as a harness-conditioned or harness-centric LLM agent (Kim et al., 24 Jun 2026, Zhang et al., 7 May 2026).

A third usage is formally named: "HARNESS-LM: A Three-Phase Training Recipe for Harnessing SLMs in Sponsored Search Retrieval" defines HARNESS-LM as a retrieval training framework rather than an agent harness for interactive environments. That system addresses query-encoder compression and online serving constraints in sponsored search, not multi-turn control in games (Gupta et al., 22 May 2026).

Usage	Meaning	Primary source
Modular gaming harness	Plug-and-play scaffold with Perception, Memory, and Reasoning around one backbone LLM/VLM	(Zhang et al., 15 Jul 2025)
Harness-centric agent view	Harness as controller or execution layer governing context, tools, orchestration, and verification	(Kim et al., 24 Jun 2026, Zhang et al., 7 May 2026)
Named retrieval framework	Three-phase recipe: teacher training, query alignment, contrastive refinement	(Gupta et al., 22 May 2026)

This terminological multiplicity is a recurrent source of confusion. A common misconception is that HARNESS-LM denotes a single standardized model family. The literature instead supports a narrower claim: “harness” is the stable conceptual unit, whereas “HARNESS-LM” may name a modular gaming scaffold, a harness-conditioned methodological view, or a sponsored-search retrieval recipe depending on context.

2. Modular architecture in multi-turn gaming environments

In the gaming-agent formulation, HLM is a unified workflow for studying multi-turn decision making by composing three interoperable modules—Perception, Memory, and Reasoning—on top of a single backbone LLM or VLM. The environment interface is built over Gymnasium-compatible environments and Stable Retro ROMs. An adapter wraps screenshots and backend state into a “Customized Observation,” and it maps natural-language actions emitted by the reasoning module to environment action indices such as {"up": 0, "left": 1, "right": 2, "down": 3, …} (Zhang et al., 15 Jul 2025).

The Perception module accepts raw visual frames and/or backend symbolic state and produces structured textual descriptions, optionally accompanied by augmented images with overlaid grid lines and coordinate labels. The paper distinguishes three operating modes. In text-based mode, backend state is converted directly into grid-structured descriptions such as object positions, which reduces perception errors by avoiding reliance on raw image interpretation. In vision-based mode, VLM capabilities operate over rendered UI images enhanced with overlays to reduce recognition mistakes on large boards. In combined mode, the module supplies both backend-derived text and enhanced imagery as redundant cues. The reported benefit is strongest in visually noisy or spatially complex environments such as TETRIS and SOKOBAN, where structured inputs unlock planning behavior that raw tokens do not (Zhang et al., 15 Jul 2025).

The Memory module implements short-term trajectory storage at the prompt level. It maintains the past $N$ states and actions and generates Reflexion-style verbal self-evaluations that summarize how the state changed and whether the previous action was effective. The paper does not define memory with explicit read/write equations or attention-style retrieval; memory remains prompt-level storage plus reflective text. Within the authors’ framing, reflection operates under the same reward signature used for evaluation,

$R : S \times A \times S \rightarrow \mathbb{R},$

and acts as a self-evaluation signal that helps correct invalid or suboptimal decisions (Zhang et al., 15 Jul 2025).

The Reasoning module integrates Perception output, Memory reflection, and a standardized prompt, and emits an action string in a strict format. The episode loop is explicitly staged: observe $s_t$ , produce a textual state description and optional enhanced image, compose a brief reflection from the last $N$ steps, generate “thought” and “action,” map the action string to an environment action index, execute the step, and continue until termination. The paper does not supply an explicit policy equation such as $\pi(a \mid s_t)$ , and it does not introduce Q-values, policy gradients, or argmax decision rules; control remains prompt-driven and feedback-conditioned (Zhang et al., 15 Jul 2025).

3. Prompt standardization, optimization, and control semantics

The reasoning controller in HLM is standardized rather than learned through fine-tuning or reinforcement learning. The paper states that HLM uses zero-shot prompting plus prompt optimization, and no model fine-tuning or reinforcement learning training is performed. Prompts include trajectory history $J$ , reflection $R_{i-1}$ , and current state $S_i$ , and the controller can toggle Perception and Memory on or off for ablation studies (Zhang et al., 15 Jul 2025).

Prompt construction follows a two-stage pipeline. The first stage is empirical prompt engineering following agentic design practices. The second stage is DSPy-based refinement using SIMBA to optimize prompt templates across $k$ steps with cumulative reward as the objective. The algorithm initializes a CHAINOFTHOUGHT template from “state $\rightarrow$ action,” compiles it over training environments, evaluates it over target LMs on development environments, and retains the best template $R : S \times A \times S \rightarrow \mathbb{R},$ 0. The optimizer-LM set is explicitly listed as $R : S \times A \times S \rightarrow \mathbb{R},$ 1, and the DSPy experiments use $R : S \times A \times S \rightarrow \mathbb{R},$ 2 optimization steps for prompt search. The paper provides two versions of this optimization routine, Algorithm 1 and a formatting variant Algorithm 2, with equivalent content (Zhang et al., 15 Jul 2025).

The control semantics are therefore modular but not policy-theoretic in the conventional RL sense. The backbone model receives a harness-constructed prompt, produces a thought-action pair, and is evaluated through environment reward. A plausible implication is that HLM is designed less as a new decision-theoretic algorithm than as a controlled experimental scaffold for isolating where performance gains arise—state extraction, short-horizon trajectory reflection, or prompt-mediated action selection.

4. Evaluation methodology, empirical results, and stated limitations

The gaming-agent study evaluates HLM on four classic titles exposed through Gymnasium and Stable Retro: Sokoban, Tetris, 2048, and Candy Crush. These environments were selected to span long-horizon planning, spatial reasoning, partial observability, delayed rewards, and visually noisy interfaces. Reward types are partitioned into progression rewards, such as Tetris score or cumulative eliminated candies, and long-horizon rewards, such as Sokoban completion progress. The per-game metrics are task-specific: Sokoban is scored by total boxes pushed onto targets until first deadlock; Tetris by total pieces dropped plus total lines cleared until game over; 2048 by the transformed merged-tile score

$R : S \times A \times S \rightarrow \mathbb{R},$ 3

and Candy Crush by total candies eliminated over a fixed 50-move session (Zhang et al., 15 Jul 2025).

Because of API cost, the evaluation uses three runs per game for each model, except asterisked single-run models, and 30 random runs per game for the random baseline. Relative to that baseline, the paper reports positive Glass’s delta values in 29 of 30 harnessed conditions versus 17 of 30 unharnessed conditions, with harnessed performance exceeding unharnessed performance in 23 of 30 cases, or approximately $R : S \times A \times S \rightarrow \mathbb{R},$ 4. The reported mean effect sizes are $R : S \times A \times S \rightarrow \mathbb{R},$ 5, $R : S \times A \times S \rightarrow \mathbb{R},$ 6, and $R : S \times A \times S \rightarrow \mathbb{R},$ 7. Paired-sample $R : S \times A \times S \rightarrow \mathbb{R},$ 8-tests across 10 models yield $R : S \times A \times S \rightarrow \mathbb{R},$ 9 for all four games: Candy Crush improves by $s_t$ 0 points with $s_t$ 1, $s_t$ 2; Sokoban by $s_t$ 3 with $s_t$ 4, $s_t$ 5; 2048 by $s_t$ 6 with $s_t$ 7, $s_t$ 8; and Tetris by $s_t$ 9 with $N$ 0, $N$ 1 (Zhang et al., 15 Jul 2025).

Representative score gains illustrate the scale of change. On Candy Crush, o4-mini rises from $N$ 2 to $N$ 3, and o3 rises from $N$ 4 to $N$ 5. On Tetris, Gemini-2.5-Pro rises from $N$ 6 to $N$ 7. On Sokoban, Claude-3.7 rises from $N$ 8 to $N$ 9. Module ablations reveal the attribution pattern emphasized by the paper: Perception-only configurations strongly improve spatial games, while Memory-only configurations strongly improve long-horizon games. Examples include o4-mini on Sokoban, rising from $\pi(a \mid s_t)$ 0 to $\pi(a \mid s_t)$ 1 with Perception only; o4-mini on Tetris, rising from $\pi(a \mid s_t)$ 2 to $\pi(a \mid s_t)$ 3; llama-4 on 2048, rising from $\pi(a \mid s_t)$ 4 to $\pi(a \mid s_t)$ 5 with Memory only; and claude-3.5 on Candy Crush, rising from $\pi(a \mid s_t)$ 6 to $\pi(a \mid s_t)$ 7. Combined modules frequently produce the best results, including Gemini-2.5-Pro on Candy Crush at $\pi(a \mid s_t)$ 8 and Claude-3.7 at $\pi(a \mid s_t)$ 9 (Zhang et al., 15 Jul 2025).

Prompt standardization also reduces variance. With DSPy+SIMBA at $J$ 0, discrepancy between two candidate prompts drops by $J$ 1 to $J$ 2 for Gemini-2.5-Flash, Claude-3.5, and o4-mini. The paper further reports benchmark-correlation observations: Sokoban correlates strongly with math and coding, Tetris and 2048 align with pattern recognition benchmarks such as EnigmaEval and NYT Connections, and Candy Crush relates notably to coding. These are presented as Spearman correlations across 20 public benchmarks (Zhang et al., 15 Jul 2025).

The stated limitations are equally clear. The paper does not report runtime per decision, memory footprint, or compute throughput. It provides no explicit policy equation, no RL training, and no formal memory read/write mechanism. Evaluation remains small-scale because of API costs, with three runs per game for most models and some single-run entries. Future directions suggested in the paper include scaling the same harness to more environments, integrating learned memory or retrieval mechanisms and explicit policy optimization, and studying native agentic models trained via multi-turn RL under the same scaffold (Zhang et al., 15 Jul 2025).

5. Harness-centric theory beyond the gaming setting

Outside the gaming-agent formulation, the harness has become a general abstraction for the execution layer around an LLM agent. "Stop Comparing LLM Agents Without Disclosing the Harness" formalizes the harness as the controller of a closed-loop dynamical system, with the LLM acting as a stochastic policy over harness-constructed context. Its Binding Constraint Thesis states that, for long-horizon tasks and comparable frontier models, harness-induced variance is often comparable to or larger than model-induced variance, making undisclosed-harness leaderboard comparisons incomplete and potentially misleading. The same paper defines harness variance and model variance explicitly and reports a controlled factorial experiment with average harness variance $J$ 3, average model variance $J$ 4, and $J$ 5 (Zhang et al., 7 May 2026).

"The Interplay of Harness Design and Post-Training in LLM Agents" extends this view into harness-aware RL. There the harness is a controllable design dimension $J$ 6 that enters prompts, per-step history, tool schemas, and tool-environment validation. On ALFWorld, harness-aware post-training improves both in-distribution and out-of-distribution robustness, whereas post-training under a low-effort harness suffers severe degradation under stronger tool-environment shifts. One cited example is Qwen2.5-7B GRPO trained under $J$ 7-low on schema v1.0, whose all-category performance drops to $J$ 8 under schema v2.0, below its zero-shot baseline of $J$ 9 (Kim et al., 24 Jun 2026).

"Harnesses for Inference-Time Alignment over Execution Trajectories" reframes harness engineering as trajectory alignment. It decomposes the harness into task decomposition and guided execution, parameterized as $R_{i-1}$ 0, and derives failure modes including over-decomposition, over-pruning, and hallucinated execution. The central claim is that stronger scaffolds are not uniformly better: increasing decomposition or guidance can improve execution but can also reduce final task success. On Terminal-Bench v2, pass rate is reported as non-monotonic in workflow granularity, peaking at six steps; on a controlled chart-analysis task, misaligned guidance increases hallucination from approximately $R_{i-1}$ 1 to approximately $R_{i-1}$ 2 as guidance grows, while aligned guidance stays near approximately $R_{i-1}$ 3 (Wang et al., 15 May 2026).

A still more formal treatment appears in "Harness Engineering as Categorical Architecture," which identifies the harness with an ArchAgents Architecture triple $R_{i-1}$ 4. In that formulation, $R_{i-1}$ 5 is syntactic wiring, $R_{i-1}$ 6 contains replayable structural certificates, and $R_{i-1}$ 7 is the deployment map into concrete models and tools. The paper maps Memory, Skills, Protocols, and Harness Engineering onto this triple and reports compiler functors targeting Swarms, DeerFlow, Ralph, Scion, and LangGraph, with preservation of integrity-gate, escalation, and convergence certificates by identity or replay (Banu, 12 May 2026).

Two additional lines extend the harness concept in complementary directions. "It’s Not the Size: Harness Design Determines Operational Stability in Small LLMs" studies model-only, minimal-shell, and plan $R_{i-1}$ 8 execute $R_{i-1}$ 9 verify $S_i$ 0 recover pipelines for 2–3B models and reports that the pipeline condition reaches $S_i$ 1 and $S_i$ 2 on Gemma4 E2B over T1–T5 tasks, while a non-monotonic degradation appears in some minimal-shell conditions (Cho, 12 May 2026). "Meta-Harness: End-to-End Optimization of Model Harnesses" automates harness search in code space and improves over ACE on online text classification by $S_i$ 3 points while using $S_i$ 4 fewer context tokens; it also finds retrieval harnesses that improve average accuracy on 200 IMO-level problems by $S_i$ 5 points across five held-out models (Lee et al., 30 Mar 2026).

Taken together, these papers support a broad, technically specific picture: a harness is not merely prompt wrapping. It is the external control structure that determines state projection, tool admissibility, history formatting, verification, recovery, and often deployment robustness. The gaming HLM of (Zhang et al., 15 Jul 2025) fits naturally into this broader harness-centric lineage, but it is one instantiation among several.

The strongest source of terminological divergence is the sponsored-search paper that formally adopts the title "HARNESS-LM." In that work, HARNESS-LM is a three-phase training framework for transferring the capabilities of large SLM retrievers into compact student encoders. Its phases are: training a high-performance teacher retriever, aligning the student query encoder to the teacher via the $S_i$ 6 objective

$S_i$ 7

and then applying a final contrastive refinement stage against a frozen teacher document tower. On a Bing Ads evaluation benchmark, the paper reports that HARNESS-LM recovers over $S_i$ 8 of the reference retriever’s precision across multiple settings, reaches $S_i$ 9 $k$ 0 against an $k$ 1B teacher at $k$ 2, and can reduce online query-encoder latency from $k$ 3 ms to $k$ 4 ms while increasing throughput from $k$ 5 to $k$ 6 qps on NVIDIA A100 GPUs. Online A/B testing with a deployed 190M-parameter model reports $k$ 7 Revenue, $k$ 8 Clicks, and $k$ 9 to $\rightarrow$ 0 Impressions over the production ensemble (Gupta et al., 22 May 2026).

A separate but related system is "HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning." That paper does not use the spelling HARNESS-LM, but the name is close enough that confusion is common. HarnessLLM trains models to write executable test harness code rather than input-output test pairs. The harness generates inputs programmatically and validates outputs through exact checks, reference implementations, or invariant testing. Training uses a two-stage SFT $\rightarrow$ 1 RLVR pipeline with the verifiable reward

$\rightarrow$ 2

On seen bug-finding benchmarks with Qwen3-4B, the harness-based RL model exceeds the input-output RL baseline on True Bug Rate across MBPP+Fix (Hard), LiveCodeBench, and Codeforces, and it also improves best-of-8 code generation selection at inference time (Liu et al., 2 Nov 2025).

These systems share the harness motif—external executable structure around a model—but they solve different problems and should not be conflated. The gaming HLM of (Zhang et al., 15 Jul 2025) studies modular perception, memory, and reasoning in multi-turn environments; the sponsored-search HARNESS-LM of (Gupta et al., 22 May 2026) is a retrieval compression recipe; HarnessLLM of (Liu et al., 2 Nov 2025) is a testing-harness generator. The common denominator is architectural externalization: model behavior is materially shaped by the surrounding scaffold, whether that scaffold is an agent loop, a retrieval training pipeline, or executable test code.