Test-Time Trace Selection

• Test-Time Trace Selection is a set of algorithms and heuristics for dynamically choosing, predicting, or pruning execution traces during inference to optimize efficiency and accuracy without extra retraining.
• It encompasses methods such as static prediction, latent-state analysis, temporal aggregation, and heuristic reward-guided pruning, applied across software testing, language reasoning, and vision tasks.
• Empirical studies demonstrate significant gains including up to 70% reduction in token usage and latency, as well as improvements in accuracy and fault localization across various domains.

Test-time trace selection refers to the suite of algorithms, criteria, and heuristics for dynamically choosing, predicting, or pruning execution traces, reasoning paths, or model activation subsets during inference (“test time”). This concept spans software testing, LLM reasoning, interactive agents, and image analysis, with the shared objective of improving efficiency, robustness, or predictive accuracy by identifying only the most relevant traces per-input at inference—often without additional retraining. Approaches leverage static analysis, learned predictors, temporal aggregation, latent state signals, or human-in-the-loop feedback to guide which traces or paths are selected or abandoned.

1. Formal Definitions and Problem Settings

A “trace” in test-time trace selection is domain-specific but generally denotes a sequence or set (possibly ordered) of intermediate entities generated or traversed by a model or program. In automated software testing, a test trace is formally defined as the set of functions $τ(t) \subseteq F$ invoked by an automated test $t\in T$ during its execution, where $F$ is the set of functions in the program being tested (Hadad et al., 2019). For LLM reasoning, a trace can refer to the sequence of intermediate hidden states or tokens generated during chain-of-thought inference (Vilas et al., 12 Oct 2025, Liang et al., 14 Jan 2026, Li et al., 19 Apr 2026). In human-in-the-loop vision, the “trace” may be a subset of feature map channels selected at inference to bias a decision (Bissoto et al., 2023).

Test-time trace selection entails: (a) selecting which candidate traces to execute, further expand, or aggregate; (b) dynamically pruning or curtailing unpromising traces; or (c) constructing predictive models that output trace subsets without running the full computation.

2. Methodological Taxonomy

Test-time trace selection methods are categorized by the underlying type of signal or criterion used for trace prioritization or pruning:

• Static Predictors: Methods predict traces from program structure and static features without execution (e.g., call-graph features, syntactic similarity, static log analysis) (Hadad et al., 2019, Yaraghi et al., 23 Jun 2025).
• Learned Latent-State Metrics: Methods leverage representations from intermediate model states (hidden activations, embeddings) and derive selection signals by measuring net change, cumulative change, or aligned progress across reasoning steps (Vilas et al., 12 Oct 2025, Liang et al., 14 Jan 2026).
• Temporal Aggregation: Aggregates multi-step answer consistency and confidence trajectory over sliding windows to identify convergence and allow early exit (Li et al., 19 Apr 2026).
• Heuristic Reward-Guided Pruning: Uses auxiliary process reward models, such as rubric-based trajectory evaluators, to rerank and filter partial action sequences in sequential decision pipelines (Han et al., 16 Apr 2026).
• Human-in-the-Loop Masking: Integrates user-provided positive/negative keypoints as constraints to select a channel-subset in CNNs for robust inference (Bissoto et al., 2023).
• Embedding-Based Ordering: Encodes test traces to latent spaces and performs test-time selection by similarity to past failures or diversity maximization (Jabbar et al., 2022).

A summary of representative approaches is given below:

Paper/Domain	Selection Signal	Application
(Hadad et al., 2019)	Static features + NN	SW test trace prediction
(Vilas et al., 12 Oct 2025)	Latent-state trajectory signals	LLM reasoning (CoT)
(Li et al., 19 Apr 2026)	Temporal answer, confidence agg.	LLM reasoning
(Han et al., 16 Apr 2026)	Rubric-based process reward model	SWE RL agent pruning
(Bissoto et al., 2023)	User keypoints → channel mask	Vision model debiasing
(Jabbar et al., 2022)	Trace embeddings: sim/diversity	Test prioritization
(Yaraghi et al., 23 Jun 2025)	Static log CFG, call refinement	Pruned test fault loc.
(Liang et al., 14 Jan 2026)	MLP over hidden state, memory-GPU	Stepwise LLM path pruning

3. Core Algorithms and Inference Pipelines

Static Trace Prediction in Software Testing

Given $T$ (tests), $F$ (functions), and a partially labeled set of traces, a binary classifier is trained to estimate $p_\theta(f \in τ(t) \mid x_{t,f})$, where $x_{t,f} \in \mathbb{R}^d$ includes call-graph and syntactic features (Hadad et al., 2019). At inference, for each new test $t$, the classifier outputs a ranked list or thresholded subset of $f$ to predict $τ̂(t)$. Predicted traces are then used in downstream utilities, such as test planning or fault localization, with empirical AUCs of 0.795 (Lang) and 0.602 (Math).

Reasoning Path Selection in LLMs

Latent-Trajectory Selection

For $t\in T$0 hidden states $t\in T$1 from an LLM, step-level metrics are computed:

• Net change: $t\in T$2
• Cumulative: $t\in T$3
• Aligned: $t\in T$4

Traces with favorable metrics are accepted early; others are pruned, and fallbacks aggregate remaining traces by majority vote. This reduces token usage by up to 70% and can increase accuracy by 2.6% relative to standard majority vote (Vilas et al., 12 Oct 2025).

Step-level Pruning with Memory-Awareness

Hidden state vectors at reasoning step boundaries are scored by an MLP, producing per-step correctness estimates $t\in T$5 (Liang et al., 14 Jan 2026). GPU memory utilization triggers pruning: when key-value (KV) cache approaches saturation, the trace with the lowest running score is pruned, and inference continues, producing up to 70% lower latency with improved accuracy over self-consistency baselines.

Temporal Aggregation for Early-Exit

TRACE’s training-free dynamic stopping mechanism computes answer consistency $t\in T$6 and confidence trajectory $t\in T$7 over a window $t\in T$8. The stability score $t\in T$9 is used to trigger early termination when $F$0 exceeds threshold $F$1 (Li et al., 19 Apr 2026).

Heuristic-Guided Selection in Software Agents

In SWE-TRACE, a rubric-based reward model $F$2 evaluates partial action prefixes; at each decision point $F$3, $F$4 candidates are scored and only the top $F$5 are executed (“early pruning” or “guide TTS”) (Han et al., 16 Apr 2026). This achieves 43% lower latency and 67% fewer environment calls than full-trajectory parallel sampling, improving solve rate by 1.3 percentage points on SWE-bench-verified.

Human-in-the-Loop Feature Selection

Test-Time Selection (TTS) for skin lesion classifiers constructs a sparse channel mask $F$6 by maximizing a linear score $F$7, based on user-provided positive and negative keypoints. The mask is constructed via top-$F$8 selection. A single positive + negative click yields improvements of up to +10 AUC with minimal annotation (Bissoto et al., 2023).

Black-Box Trace Estimation and Fault Localization

Using a single failing execution log, static log-to-code matching, gap filling, CFG analysis, and call-site refinement yield a pruned trace $F$9 without executing the program (Yaraghi et al., 23 Jun 2025). Pruned traces drive LLM-based fault localization with up to 34% search-space reduction and no loss in accuracy.

Trace Embedding Prioritization

Test2Vec leverages CodeBERT+BiLSTM to embed execution traces, enabling test-time prioritization by similarity to historical failures or by maximizing diversity in the latent space. A logistic regression switcher chooses between similarity and diversity ranking per suite. This reduces FFR by up to 66% and improves APFD by nearly 30% vs. coverage baselines (Jabbar et al., 2022).

4. Quantitative Performance and Empirical Trends

Empirical evaluations consistently report substantial efficiency or robustness gains:

• Software test trace prediction: AUC 0.795 (Lang), 0.602 (Math); utility in planning nearly matches using ground-truth traces (Hadad et al., 2019).
• SWE agent search pruning: Heuristic guide TTS achieves 71.2% solve vs. 69.9% for parallel sampling; 36.5 min/issue vs. 63.8 min, 128 vs. 392 env calls (Han et al., 16 Apr 2026).
• LLM reasoning: Latent-Trajectory selection provides up to 70% fewer tokens generated and 2.6% higher accuracy over majority-vote (Vilas et al., 12 Oct 2025); STEP reduces latency by up to 70% and increases accuracy by up to 7.5% (Liang et al., 14 Jan 2026).
• Temporal aggregation: TRACE achieves token savings of 25–30% with ≤2 point accuracy drop compared to full-length inference (Li et al., 19 Apr 2026).
• Pruned test code fault localization: 81% Hit@3 at block-level for LLM-driven fault localization, 34% inference-time reduction, no performance loss (Yaraghi et al., 23 Jun 2025).
• Human-in-the-loop image analysis: TTS achieves up to 75 AUC (ISIC2019 trap-set, artifact clicks), outperforming both entropy-based regularization and noise-masking (Bissoto et al., 2023).
• Test2Vec prioritization: Reduces rank of first failing test (FFR) by up to 66%, APFD by 29.5% (Jabbar et al., 2022).

5. Failure Modes, Limitations, and Best-Use Scenarios

• Static call-graph predictors can mispredict when dynamic dispatch is not captured or when syntactic similarity is non-informative (Java limitations, heavy class-imbalance) (Hadad et al., 2019).
• Latent-state early signals may misestimate trace promise if distributional shifts misalign calibration for scoring thresholds (Vilas et al., 12 Oct 2025).
• Heuristic-guided test-time scaling depends critically on the quality of the reward model; poor rubrics or evaluators risk discarding promising search branches (Han et al., 16 Apr 2026).
• Memory-aware pruning (STEP) is robust to wide ranges of KV-cache thresholds, but aggressive pruning of traces before sufficient signal accumulates can degrade accuracy (Liang et al., 14 Jan 2026).
• TRACE relies on text-only reasoning and is untested for multimodal or program synthesis tasks (Li et al., 19 Apr 2026).
• In human-in-the-loop TTS, insufficient keypoint coverage or poor annotation may underselect relevant features, though performance degrades gracefully (Bissoto et al., 2023).
• In black-box trace estimation, context granularity (function, block, line) trades off search space and interpretability; block-level provides a practical sweet spot (Yaraghi et al., 23 Jun 2025).
• Trace embedding methods’ effectiveness depends on the representational match of embedded space to actual behavioral diversity of test cases (Jabbar et al., 2022).

6. Broader Impact and Research Directions

Test-time trace selection has established itself as a critical paradigm in both software engineering and the scaling of reasoning models, enabling more efficient, robust, and actionable inference. By leveraging diverse sources of signal—learned representations, static structure, human guidance, and reward models—it mitigates the resource costs and pitfalls associated with brute-force or uniform inference.

Emerging research is beginning to integrate these test-time techniques into the training phase via diversity-promoting objectives, cross-project learning, or by co-designing training and inference-time heuristics (e.g., rubric-aligned RL pipelines) (Han et al., 16 Apr 2026, Wu et al., 22 Sep 2025). Extensions to multimodal reasoning, richer symbolic domains, and real-world interactive systems are current frontiers (Li et al., 19 Apr 2026). Comprehensive benchmarks across software, reasoning, and recognition tasks continue to drive refinements in both methodology and evaluation practices.

Test-time trace selection thus occupies a central role in the ongoing effort to optimize inference under tight computational and latency budgets, while maintaining or even surpassing standard accuracy and robustness metrics.