Inference-Time Harnesses for LLM Agents

• Inference-time harnesses are external control systems that mediate a fixed LLM’s context, tool use, verification, state persistence, and execution, enabling agent behavior changes without modifying model weights.
• They support practical workflows such as proposer–verifier search, multi-agent orchestration, adaptive memory, OS-level policy enforcement, and executable harness evolution for coding, reasoning, forecasting, and computer-use tasks.
• Reliable harness design requires clear boundary separation, calibrated verifiers, matched token and latency budgets, provenance-preserving memory, layered security controls, and held-out evaluation to distinguish reusable improvements from task-specific search.

Inference-time harnesses are external control systems that mediate the interaction between a fixed LLM, tools, environments, persistent state, verifiers, and subsequent model calls. They determine what the model observes, how outputs become executable actions, how observations and failures return to the model, how candidate trajectories are generated or selected, and how execution state persists. Unlike model-weight adaptation, inference-time harnessing changes the effective agent policy without modifying the underlying model parameters. Depending on the system, a harness may be a proposer–verifier–selector pipeline, a multi-agent workflow, a learned bidirectional controller, an operating-system policy engine, an evolving executable program, a prompt-level agent loop, or a task-adaptive combination of deterministic code and model calls.

1. Conceptual foundations and system boundaries

An LLM by itself generates text, code, or structured action proposals. The harness supplies their operational semantics: it constructs context, exposes tools, validates outputs, dispatches accepted actions, forms observations, persists selected state, and determines whether and how execution continues. Coderlet expresses this organization through three boundaries—model, execution, and state—connected by a recurrent request lifecycle (Li, 10 Aug 2026).

At the model boundary, the harness assembles instructions, selected history, memory, tool descriptions, schemas, and observations, then invokes the model service and decodes its response. At the execution boundary, it distinguishes proposed calls from accepted calls and executes only validated actions against files, processes, services, or other resources. At the state boundary, it saves request records and memory updates and later selects a bounded subset for subsequent contexts. Environmental effects and persistent records are separate channels: an external action may succeed even if its request record is not saved.

A general agent execution can be represented as:

$$\text{state selection} \rightarrow \text{model context} \rightarrow \text{model proposal} \rightarrow \text{validation} \rightarrow \text{execution} \rightarrow \text{observation} \rightarrow \text{new context} \rightarrow \text{state persistence}.$$

This separation supports boundary-oriented diagnosis. Missing instructions implicate context construction or selection; valid proposals with no environmental effect implicate validation or dispatch; missing later history implicates persistence or retrieval; a saved record without a completion notification implicates the interface layer. The distinction between model proposal, executed action, environmental result, returned observation, durable record, and completion event is central to reliable harness design.

A related formalization decomposes the inference environment into:

$C_t=(P_t,R_t,A_t,V_t),$

where $P_t$ is the model-facing prompt and skill text, $R_t$ is retrieved context, memory, and experience, $A_t$ is the available tool and runtime interface, and $V_t$ is the verifier and feedback channel. This view treats prompt engineering, memory, tool mediation, and evaluation as distinct control surfaces around a frozen model (Wu et al., 6 Jun 2026).

The harness is therefore not synonymous with a prompt, a tool wrapper, or a static multi-agent topology. It may include orchestration, memory, decoding, output processing, verification, retry logic, context management, and execution policy. In autonomous multi-agent systems, a static infrastructure $\mathcal H$ may dynamically instantiate a task-specific harness $h_x$ containing roles, delegation, planning, context flow, tool order, verification, and termination behavior (Cui et al., 30 Jul 2026). Exact implementation recovery is not required for behavioral imitation: multiple harnesses may be behaviorally indistinguishable on a task distribution.

2. Core inference-time computation mechanisms

A large class of reasoning harnesses follows a proposer–verifier–selector architecture. The proposer generates candidate trajectories, the verifier assigns scalar or process-level rewards, and the selector returns or aggregates candidates. A stopping controller terminates sampling, search, or refinement under a budget or convergence condition (Liu et al., 11 Feb 2025).

For an input $x$, instruction $I$, inference model $C_t=(P_t,R_t,A_t,V_t),$0, and candidate count $C_t=(P_t,R_t,A_t,V_t),$1, candidates are sampled as:

$C_t=(P_t,R_t,A_t,V_t),$2

A verifier $C_t=(P_t,R_t,A_t,V_t),$3 produces a reward $C_t=(P_t,R_t,A_t,V_t),$4, and scalar-reward selection returns:

$C_t=(P_t,R_t,A_t,V_t),$5

The main mechanisms differ in where computation is allocated:

Method	Primary computation pattern	Characteristic trade-off
Best-of-$C_t=(P_t,R_t,A_t,V_t),$6	Independent complete trajectories followed by reward selection	Parallel and simple, but duplicates shared-prefix computation and depends on verifier quality
Step-level Best-of-$C_t=(P_t,R_t,A_t,V_t),$7	Branching and reward selection at each reasoning step	Uses process rewards but is more synchronized and vulnerable to early mistakes
Beam Search	Breadth-wise expansion retaining the best prefixes	Reuses prefixes but often collapses diversity and is token-inefficient
MCTS	Adaptive tree search with selection, expansion, rollout, evaluation, and backup	Allocates computation adaptively but has substantial sequential bookkeeping
Self-Consistency	Independent trajectories followed by majority answer voting	Avoids a separate reward model and is strong for canonical answers, but agreement is not correctness
Self-Refine	Sequential critique and revision of one trajectory	Can exploit useful feedback but is latency-bound and commonly weaker than Self-Consistency

Temperature and nucleus sampling are important harness parameters. Higher temperature increases diversity, while lower temperature concentrates decoding. The reported recommendation is approximately $C_t=(P_t,R_t,A_t,V_t),$8 and top-$C_t=(P_t,R_t,A_t,V_t),$9, although parts of the benchmark use $P_t$0 and top-$P_t$1. Across the reported experiments, temperature tuning improved reasoning performance by approximately 2.32–4.83 percentage points, with improvements of up to roughly 5%, while top-$P_t$2 was generally strongest (Liu et al., 11 Feb 2025).

Fixed-compute comparison requires equal token budgets rather than equal candidate counts:

$P_t$3

Equal $P_t$4 can favor methods with shorter trajectories and conceal verifier calls, intermediate expansions, and sequential overhead. A robust harness separately tracks proposer tokens, verifier tokens, model-call count, GPU memory, latency, and orchestration cost.

The empirical comparison across Llama, Qwen, and Mistral models and eight reasoning tasks found Self-Consistency and Best-of-$P_t$5 to be strong general baselines. Beam Search often exhibited weak scaling with compute; MCTS and step-level Best-of-$P_t$6 improved more slowly with budget; and Self-Refine generally underperformed Self-Consistency. Reward-model quality was task-dependent: RLHF rewards helped knowledge-oriented tasks, while process evaluation was more useful for difficult mathematics, code, and logical reasoning. Self-evaluation could fall below random selection, and increasing candidate count could produce performance inflation by increasing selected reward without increasing true correctness.

3. Workflow decomposition, guided execution, and adaptive structure

Inference-time harnesses also control long-horizon execution by separating workflow-level decomposition from local action selection. A formal treatment represents a harness as $P_t$7, where $P_t$8 controls task decomposition, $P_t$9 controls guidance strength, and $R_t$0 specifies the local guidance rule (Wang et al., 15 May 2026).

Task decomposition produces ordered sub-goals:

$R_t$1

It determines the scale and order of progress but not necessarily the exact primitive actions. Guided execution changes the probability distribution over local trajectories through a nonnegative weight:

$R_t$2

The final objective remains terminal task success. Under a recoverability factorization, success is decomposed into conditional probabilities of remaining recoverable after each stage:

$R_t$3

This makes process losses interpretable as an additive decomposition of terminal-success loss rather than an unrelated auxiliary objective.

The analysis identifies three principal failure modes. Over-decomposition creates milestones smaller than the agent’s controllable action scale or larger than its retry-supported progress. Over-pruning removes actions that are individually plausible but necessary for future recoverability. Hallucinated execution occurs when guidance rewards surface compliance, confident detail, or textual completion without sufficient evidence. The retention gap,

$R_t$4

measures whether guidance favors recoverable over non-recoverable trajectories. Positive guidance alignment improves stage recoverability; negative alignment amplifies failure. The empirical pattern is frequently inverted-U: more decomposition or guidance initially helps and later harms.

This framework motivates partial harnessing. A harness may specify initial orientation, inspections, invariants, and high-value checks, then release control to the agent. Adding another scaffolded stage is beneficial only while its reduction in autonomous tail risk exceeds its local reliability cost. Synthetic addition experiments and Terminal-Bench results support capability-dependent granularity: pass rate peaks at different workflow lengths for agents with different action scales, and Terminal-Bench v2 reported a peak at six workflow steps.

Several agent architectures instantiate these principles. The Explorer–Definer Pipeline separates pattern discovery from executable transformation synthesis, while the Reflective Orchestrator allows synthesis failure to trigger focused re-exploration. On the ARC-AGI-1 public evaluation set, the pipeline achieved 57.50% pass@2 at $R_t$50.62, compared with a 15.50% one-shot baseline (Moghe et al., 7 Jul 2026). The diagnostic concluded that the pipeline was generation-bound rather than selection-bound: training-pair selection captured approximately 95% of the candidate ceiling, so adaptive re-exploration was more valuable than increasingly sophisticated ranking.

The pipeline uses parallel PatternExplorers, compressed exploration summaries, executable Python Definers, training-pair execution, candidate ranking, and deduplication by predicted test grid. The orchestrator adds explore_new_patterns(guidance), enabling a downstream synthesis agent to revisit an upstream conceptual hypothesis. The combination of adaptive spawning and prediction deduplication allows a dissenting correct candidate to survive several convergent but incorrect candidates.

4. Harnesses for context, actions, memory, and policy enforcement

Long-horizon agents require control over both what the model sees and what the environment executes. HarnessBridge implements this as a learnable bidirectional controller with an observation projection and an action projection (Wang et al., 11 Jun 2026).

The observation projection maps raw history $R_t$6 to a generator-visible state $R_t$7. Each history unit is assigned Pass, Compress, or Drop, while a learned active-state block summarizes unresolved errors, open constraints, established facts, pending goals, important files or hypotheses, and failed approaches. The raw trajectory remains authoritative, reducing the risks of irreversible information loss and unsupported summaries.

The action projection receives the proposed action and returns Pass or Reject. A rejection must include a concern, trajectory-grounded evidence, and a suggestion. If adequate evidence is unavailable, the controller defaults to Pass. This permits semantic productivity filtering beyond syntactic tool validation, including rejection of duplicated, repetitive, contradicted, premature, or weakly grounded actions.

HarnessBridge is trained as a lightweight shared controller through unified instruction tuning. On Terminal-Bench 2.0 and SWE-bench Verified, it reduced generator input-token usage while matching or exceeding several specialized harnesses. Its strongest efficiency effects occurred for long-trajectory models. Ablations showed that both observation projection and action projection matter, while strict rejection could over-block useful exploration; tolerant rejection gave the best reported quality-efficiency balance.

Operating-system enforcement addresses a different layer of the harness. ActPlane places policy enforcement at Linux kernel hooks while leaving semantic policy instantiation to the task-local agent or harness (Zheng et al., 23 Jun 2026). Its DSL expresses source–target constraints with effects such as notify, block, and kill, together with Boolean label predicates, temporal gates, lineage conditions, and exit-status requirements. A representative rule is:

$A_t$1

ActPlane uses object-level information-flow control. Processes, files, and endpoints carry label bitmasks, and labels propagate over fork, exec, read, write, and connect edges. Hierarchical policy domains ensure that child domains inherit and cannot weaken higher-authority rules. BPF-LSM hooks support pre-operation enforcement, while tracepoints support observation and post-operation termination.

In the reported decision-compliance evaluation, ActPlane achieved 75.8% DCR compared with 45.3–48.9% for prompt filtering, tool-regex controls, FIDES, and opaque ActPlane. It detected indirect violations involving scripts and hidden subprocesses that tool-level systems could not observe. Semantic feedback was essential: kernel enforcement paired with policy-specific corrective messages produced better compliance than opaque denials. The prototype reported 1.9–8.4% overhead across agent traces and kernel-build settings.

These results motivate layered enforcement. Prompt and trajectory controls address semantic requirements and intent interpretation; application-level validators filter obvious calls; OS-level mechanisms cover subprocesses, shell-outs, process lineage, data flow, and event ordering. No layer is sufficient for all semantic, content, service-level, or side-channel effects.

5. Evolving, adaptive, and distillable harnesses

Harnesses can be optimized during or across executions rather than fixed before deployment. Several systems differ chiefly in what is changed, what evidence is used, and whether adaptation persists.

Bayesian-Agent treats reusable skills and SOPs as hypotheses about conditional success under a frozen model, prompt, context, tool, memory, and runtime configuration (Wu et al., 6 Jun 2026). Its environment is represented as $R_t$8, and verified trajectories update a feature-conditioned categorical posterior. Posterior state maps to inspectable actions: explore, patch, split, compress, or retire. Repeated failures can become executable guardrails, heterogeneous contexts can produce context-specific variants, stable instructions can be compressed, and consistently harmful skills can be retired. In incremental repair experiments, performance increased from 80% to 95% on SOP-Bench, 90% to 100% on Lifelong AgentBench, and 45% to 65% on RealFin-Bench. The approach relies on external verification rather than ungrounded self-critique.

MemoHarness decomposes a harness into six editable dimensions: context assembly, tool interaction, generation control, orchestration, memory management, and output processing (Huang et al., 14 Jul 2026). It maintains a dual-layer experience bank containing per-case execution entries and distilled cross-case patterns. A global harness is learned from labeled search cases, then adapted once for each new unlabeled case by retrieving similar successful and failed experiences. The system improves the reported base-harness results from 0.722 to 0.806 on Terminal-Bench, 0.900 to 0.967 on LiveCodeBench, and 0.600 to 0.767 on FinanceAgent. Its cost argument depends strongly on cache reuse, and complete component-level causal attribution is not established.

Test-Time Harness Evolution (TTHE) evolves an executable Python harness during evaluation using unlabeled traces, execution-derived proxies, proposer mutations, and a judge (Nie et al., 9 Jul 2026). The selected harness persists across subsequent task batches. The paper distinguishes shared harness evolution from parallel sampling, sequential refinement, and instance-specific harness scaling. On Terminal-Bench 2.1, harness evolution was weaker than parallel sampling and sequential refinement under matched rollout budgets, especially with unit-test feedback. Its held-out improvement was only 0.6 points on average, compared with larger same-task gains, highlighting the danger of transductive overfitting and proxy misoptimization.

HarnessCompass responds to these problems with constrained evolution, proactive first-person feedback, and component-wise optimization (Zhang et al., 3 Aug 2026). A generalization gate rejects task-specific identifiers and requires transferable applicability conditions. Structural components—middleware, tools, and sub-agents—are evolved separately from guidance components—prompts, memory, skills, and tool descriptions. R$R_t$9 integration performs revision, recombination, and refinement of the winning and losing tracks. On SWE-bench Verified with GPT-5.4, HarnessCompass improved Pass@1 from 54% to 66% in five evolution iterations, with held-out performance of 60.4%, compared with 54.7% for AHE. The ablation indicates that the generalization gate improves transfer, proactive feedback improves diagnosis but can increase overfitting, and R$A_t$0 restores held-out performance while reducing evolution turns.

Recursive Harness Self-Improvement (RHI) treats the harness as a prompt-level specification of a multi-agent loop. It uses pairwise evaluation between successive repositories and updates contracts, workflow hops, roles, instructions, acceptance gates, recall rules, and termination criteria (Lee et al., 17 Jul 2026). On 30 synthetic machine-learning research tasks, RHI improved the performance ceiling of low-reasoning-effort agents relative to stronger reasoning settings, with reported inference-cost reductions of up to 60% relative to Opus ultracode. The principal mechanistic interpretation is improved inter-agent information flow: contracts and hops become task-specific while avoiding redundant context propagation. RHI is a noisy local-ascent procedure rather than global harness optimization, and its pairwise evaluator introduces substantial dependence on judgment quality.

Harness-assisted distillation can transfer behavior into model parameters. A time-truncation harness restricts search and browsing to documents published before a forecasting cutoff, preventing historical forecasting from becoming hindsight retrieval (Cai et al., 28 Jul 2026). In teacher synthesis, the harness eliminated observed publication-date leakage in the reported audit, raising the valid fraction from 7.0% without the harness to 88.5% with it. Students trained on harness-generated trajectories improved ForecastBench Brier Scores relative to non-harness mixtures, from 0.3167 to 0.2550 for Qwen3-8B and from 0.2563 to 0.1998 for Qwen3-32B. Here the harness remains external during teacher inference, while temporal search and reasoning behavior are partially internalized through supervised fine-tuning.

Strong-to-weak capability transfer follows a similar pattern without updating the target model. A builder model constructs a harness containing routing, deterministic symbolic rules, structured extraction, fallback procedures, parsing, and format enforcement around a weaker target (Qian et al., 12 Aug 2026). On four Theory-of-Mind benchmarks, the best GPT-5.4-mini-target scaffold reached 0.912 macro accuracy from a 0.488 vanilla baseline. The main gains arose from deterministic offloading, benchmark-specific routing, and strict output-format control rather than broader target-model sampling.

6. Evaluation, security, and unresolved limitations

Evaluation must distinguish reusable harness improvement from additional task-level search. Parallel sampling and sequential refinement may improve performance simply by generating more candidates or revising more trajectories. Harness evolution may similarly benefit from repeated verifier calls, task-specific memory, and selection. Therefore, fair comparison requires matched rollout, token, verifier, meta-agent, and environment budgets, together with both single-shot and multi-sample metrics (Wang et al., 14 Jul 2026).

A reusable harness should be evaluated on disjoint search, validation, and held-out test tasks. Same-benchmark adaptation can encode task-specific paths, commands, bugs, tests, or verifier artifacts. The critique of automatic harness evolution on Terminal-Bench 2.1 found that shared harness evolution did not consistently outperform simple test-time scaling and showed limited held-out generalization. This does not prove that harness evolution is intrinsically ineffective; it demonstrates that claims of general improvement require leakage controls, matched baselines, and held-out evaluation.

Several recurrent limitations span the literature:

• Verifier and proxy unreliability: executable success, public-test success, reward-model score, round-trip consistency, or LLM judgment can diverge from true correctness.
• Task and model dependence: optimal decomposition, guidance, decoding, memory, and scaffolding vary across tasks and models.
• Over-intervention: compression can discard decisive evidence, rejection can block useful exploration, and scaffolding can interfere with already-competent agents.
• Correlated search: proposers or candidates sharing a backbone may reproduce identical errors rather than provide genuine diversity.
• Incomplete attribution: many systems evaluate bundles of components, so causal contributions of memory, tools, prompts, routing, and verification remain uncertain.
• Resource accounting: token, latency, controller, verifier, debugger, and environment costs are often reported incompletely.
• Distribution shift: a harness trained on coding traces may not transfer to web, multimodal, computer-use, research, or open-world environments.
• Semantic and content gaps: OS hooks and tool validators may miss service-side effects, file-content semantics, custom implementations, or policies expressed only at the level of intent.
• State and provenance risks: summaries, memories, and evolved rules may become stale, unsupported, or task-specific.
• Benchmark artifacts: structured benchmarks can make deterministic compilation disproportionately effective, while public-set adaptation can inflate apparent generalization.

Inference-time harnesses also create intellectual-property and security risks. Agent Harness Distillation formalizes black-box extraction of dynamic orchestration behavior from autonomous multi-agent systems (Cui et al., 30 Jul 2026). An attacker can use probes to infer roles, delegation, tool patterns, context management, verification, and memory behavior, then construct a functionally similar harness. The threat concerns behavioral replication rather than exact source-code recovery. A deception-based defense attempts to return plausible but materially false workflow descriptions while preserving task utility, although its effectiveness depends on probe detection and may reduce legitimate auditability.

The resulting research agenda includes stronger budget-normalized comparisons, prequential rather than transductive evaluation, causal component ablations, calibrated verifiers, provenance-preserving compression, robust cross-model and cross-domain transfer, service-level policy enforcement, privacy-preserving trace handling, and formal methods for distinguishing portable harness principles from benchmark- or ecosystem-specific adaptations. The central methodological principle is that an inference-time harness should be evaluated as an executable policy over context, actions, evidence, state, and computation—not merely as a prompt or a collection of extra model calls.