Trajectory-Reward Guided Adaptive Pre-training

• Trajectory-Reward Guided Adaptive Pre-training defines a design pattern where full sequential trajectories, not just final outputs, are evaluated via reward signals to guide model adaptation.
• It unifies methods across various domains—web agents, video masked modeling, diffusion—and uses adaptive filtering, repair, and selection to optimize efficiency and process quality.
• Empirical results show that TRAP reduces tool-call rounds and improves performance metrics in tasks like tool-integrated reasoning and action recognition.

Searching arXiv for the cited works and closely related terminology to ground the article in current papers. “Trajectory-Reward Guided Adaptive Pre-training (TRAP)” (Editor’s term) denotes a family of training and adaptation schemes in which whole trajectories—not only terminal outputs—serve as the basic objects for selection, scoring, repair, or policy improvement. Across recent work, the term can be used to unify methods that score complete reasoning-and-action traces, denoising paths, masking decisions, or decoded response sequences with reward signals, then use those signals to adapt supervised fine-tuning, reinforcement learning, masking policies, or inference-time search. In the supplied literature, TRAP is not introduced as a canonical method name; rather, it is a useful umbrella description for a recurring design pattern spanning web agents, tool-use reasoning, masked video modeling, diffusion test-time scaling, and reward-guided decoding (Xie et al., 5 Jun 2026).

1. Conceptual definition and scope

Within this umbrella usage, a “trajectory” is a full sequential object whose internal structure matters for optimization. In web agents, a trajectory includes the initial query, a sequence of reasoning “thoughts” and tool actions, and the final answer string (Xie et al., 5 Jun 2026). In Tool-Integrated Reasoning (TIR), a trajectory is defined as

$\tau = \{(r_k, t_k, e_k)\}_{k=1}^{K},$

where $r_k$ is reasoning, $t_k$ is a tool invocation, and $e_k$ is the tool’s execution result (Gong et al., 30 Jan 2026). In video masked modeling, the relevant trajectory is the motion path of a visual token across time, encoded by trajectory attention (Rai et al., 13 May 2025). In diffusion, the trajectory is the entire denoising path in latent space,

$\mathcal{Z} = \{\mathbf{z}_L,\mathbf{z}_{path}\},$

optimized under a terminal reward (Dai et al., 21 May 2026). In reward-guided decoding for frozen LLMs, the trajectory is the full generated response sequence $\mathbf{y}$, explored by a Markov chain in trajectory space (Wang et al., 27 May 2026).

The “reward-guided” component likewise varies by domain but retains the same structural role. In SlimSearcher, the final reward is correctness-gated and multiplied by tool and token efficiency terms,

$$R_{final}(\tau) = r_{correct}(\tau) \cdot r_{tool}(\tau) \cdot r_{len}(\tau),$$

so that only correct trajectories receive efficiency-based reinforcement (Xie et al., 5 Jun 2026). In TATS for masked video modeling, the reward is derived from the MAE reconstruction loss, so PPO trains a masking policy to sample motion-centric tokens that are hard to reconstruct (Rai et al., 13 May 2025). In RTS for diffusion, a reward model such as ImageReward, CLIP, or an LLM-based grader evaluates the final image, and the search procedure optimizes the full denoising trajectory toward higher reward (Dai et al., 21 May 2026). In EBD, the target decoding distribution is reward-tilted,

$$\pi_\beta^*(\mathbf{y}\mid\mathbf{x}) \propto p_\theta(\mathbf{y}\mid\mathbf{x}) \exp(\beta S_{\mathbf{x}}(\mathbf{y})),$$

making sequence-level reward the principal steering signal for frozen pre-trained models (Wang et al., 27 May 2026).

This suggests that TRAP is best understood not as a single algorithm but as a general recipe: define trajectories as structured sequential objects, attach a reward or preference signal to them, and use that signal to adapt training data, policy updates, masking strategies, latent search, or decoding behavior.

2. Core design pattern: trajectory objects, reward signals, and adaptive updates

A common TRAP structure can be decomposed into three elements: trajectory representation, trajectory evaluation, and adaptive optimization. The representation stage determines what constitutes the relevant sequential object. The evaluation stage attaches utility, preference, or efficiency scores to that object. The adaptive stage then uses those signals either to filter demonstrations, train reward models, update policies, or guide test-time search.

SlimSearcher provides a particularly explicit instantiation. It operates on full web-agent trajectories and defines three trajectory-level metrics: correctness, tool efficiency, and token efficiency (Xie et al., 5 Jun 2026). Correctness is modeled as

$$r_{correct}(\tau) = \mathbb{I}(\text{Answer}(\tau) \approx y_{gt}),$$

tool cost is

$$C(\tau_i) = \sum_{a \in \mathcal{A}(\tau_i)} w_{type}(a),$$

and token length is $r_k$0, the number of model-generated tokens (Xie et al., 5 Jun 2026). Efficiency is not assessed by an absolute penalty but by cohort-relative deviation from the best successful trajectory for the same query. This is a defining adaptive feature: the standard of efficiency depends on what is empirically necessary for that query rather than on a global fixed threshold (Xie et al., 5 Jun 2026).

AutoTraj uses a related pattern but with explicit preference construction. It first generates candidate tool-use trajectories, evaluates them with a multi-dimensional quality score

$r_k$1

then repairs low-quality trajectories and converts repaired/original pairs into a trajectory preference dataset for reward-model training (Gong et al., 30 Jan 2026). Here adaptation occurs in two places: the SFT set is adaptively improved by repair, and RL is later guided by a learned trajectory-level reward model.

In TATS, the adaptive mechanism is a policy over visible tokens for video masked autoencoding. The policy network outputs a categorical distribution over tokens,

$r_k$2

and PPO updates that policy using advantage estimates derived from reconstruction loss (Rai et al., 13 May 2025). The trajectory signal lies in motion trajectories encoded by trajectory attention, and the reward signal arises from self-supervised reconstruction difficulty.

RTS and EBD show that the same pattern extends beyond conventional training. RTS performs reward-guided trajectory optimization at test time over the entire denoising path of a diffusion model (Dai et al., 21 May 2026). EBD performs reward-guided trajectory editing at inference time for frozen LLMs, using block-wise Metropolis–Hastings to resample suffixes and accept or reject proposals based on sequence-level reward differences (Wang et al., 27 May 2026). A plausible implication is that TRAP encompasses both parameter-updating and parameter-free adaptive procedures, so long as whole trajectories and explicit reward signals jointly determine system behavior.

3. Supervised and self-supervised trajectory shaping

One major branch of TRAP-style methods uses trajectory rewards to shape the supervised data distribution before or alongside RL. SlimSearcher’s supervised fine-tuning stage is the clearest example of trajectory-level filtration for efficiency-aware learning. For each query, multiple candidate trajectories are sampled, correctness is used as a hard gate, and the retained demonstration is the one maximizing the joint efficiency score

$r_k$3

The resulting SFT dataset is therefore composed of correct and economical trajectories, described in the paper as selecting the “Minimal Necessary Path” (Xie et al., 5 Jun 2026). This procedure directly addresses what the paper calls “Acc-only rejection sampling,” under which any correct but bloated trajectory can become a positive supervised example (Xie et al., 5 Jun 2026).

AutoTraj uses a more interventionist version of the same principle. Rather than discarding low-quality traces, it employs an LLM-as-Repairer. High-quality trajectories are directly retained, while low-quality trajectories are repaired using Qwen2.5-32B-Instruct under strict constraints on reasoning tags, tool calls, answer format, and avoidance of uncertain language (Gong et al., 30 Jan 2026). Successfully repaired trajectories are added to the SFT dataset, and each repaired trajectory paired with its original low-quality counterpart becomes a preference example for trajectory reward modeling (Gong et al., 30 Jan 2026). The final SFT dataset contains 13,116 trajectories and the preference dataset contains 7,178 trajectory pairs (Gong et al., 30 Jan 2026).

TATS supplies a self-supervised analogue. Instead of selecting demonstrations, it learns which visible tokens to preserve for MAE pre-training. The token sampler is conditioned on trajectory-aware motion embeddings, and PPO is used to optimize the masking policy jointly with the MAE after an initial warm-up phase using random masking (Rai et al., 13 May 2025). The paper describes this as a unified training strategy enabling joint optimization of both MAE and TATS from scratch using PPO (Rai et al., 13 May 2025). Because reconstruction loss itself provides the reward, no human annotation or explicit verifier is required.

These approaches differ in modality and objective, but they share an important structural feature: pre-training or fine-tuning is no longer driven purely by raw likelihood over all available sequences. Instead, whole trajectories are filtered, repaired, or selected according to trajectory-level quality signals. This suggests that TRAP-style training redefines the effective training distribution by privileging traces that exhibit desirable process properties, not only desirable end states.

4. Reinforcement learning and reward shaping over trajectories

A second major branch of TRAP centers on RL or RL-like optimization with trajectory-level rewards. SlimSearcher’s RL stage uses GRPO and keeps the same multi-stage gating logic as in SFT. Correctness remains a strict binary gate, and among correct trajectories tool and token efficiency are computed relative to the best successful trajectory in the sampled cohort (Xie et al., 5 Jun 2026). The group-normalized advantage is

$r_k$4

and this trajectory-level advantage is applied to all tokens in the trajectory under the GRPO objective (Xie et al., 5 Jun 2026). The paper argues that this design avoids brevity bias and reward hacking because incorrect but short trajectories receive zero reward (Xie et al., 5 Jun 2026).

AutoTraj also performs trajectory-aware RL, though with a learned reward model rather than relative efficiency anchors. It combines three rewards: a binary format reward, token-level F1 outcome reward, and a trajectory reward $r_k$5 trained from repaired/original preference pairs (Gong et al., 30 Jan 2026). The final reward is piecewise: $r_k$6 with $r_k$7 and reward clipping to stabilize training (Gong et al., 30 Jan 2026). This directly addresses the paper’s critique of sparse outcome-only rewards in tool-use RL.

TATS uses PPO rather than GRPO, but the logic is again trajectory-reward adaptation. The advantage is computed as

$r_k$8

so the masking policy is rewarded for selecting informative token subsets that are challenging under the current MAE (Rai et al., 13 May 2025). Training alternates between freezing the sampler and updating the MAE, then freezing the MAE and updating the sampler via PPO (Rai et al., 13 May 2025).

RTS provides a gradient-free analogue in diffusion. It estimates a surrogate gradient from reward differences among nearby noise perturbations,

$r_k$9

and uses a coarse-to-fine search over initial and selected intermediate noises (Dai et al., 21 May 2026). Although this is not policy-gradient RL in the usual sense, it is still trajectory-level reward optimization over sequential latent states.

Across these methods, the central RL lesson is consistent: a trajectory-level reward is more informative than final-answer correctness alone when the desired behavior depends on process quality, efficiency, or information structure. The exact implementation may be GRPO, PPO, pairwise reward modeling, or zeroth-order search, but the optimization target remains a function of the whole trajectory.

5. Inference-time counterparts and the boundary of “pre-training”

TRAP, as an umbrella term, sits near a broader class of reward-guided adaptive procedures that do not necessarily update model parameters. EBD is explicit on this point: it is a training-free framework for frozen pre-trained LLMs that activates task-oriented behavior via reward-guided decoding (Wang et al., 27 May 2026). Its target distribution,

$t_k$0

arises from a KL-regularized objective over full responses (Wang et al., 27 May 2026). The actual decoding algorithm initializes from a small candidate pool, then repeatedly proposes suffix edits and accepts them with probability

$t_k$1

so sequence-level reward directly governs movement through trajectory space (Wang et al., 27 May 2026).

RTS does something analogous for diffusion models at test time. The denoising process is treated as a trajectory in latent space, and test-time scaling becomes optimization over $t_k$2 rather than merely sampling more outputs (Dai et al., 21 May 2026). The method adds two specific adaptive mechanisms: reward-guided noise optimization and sparse test-time scaling based on PCA-driven curvature analysis to prioritize key intermediate denoising steps (Dai et al., 21 May 2026).

These methods are not pre-training in the narrow sense. However, the supplied papers explicitly frame them as conceptually adjacent to trajectory-reward guided adaptive learning. EBD is described as “trajectory-reward guided adaptive behavior without parameter updates,” while RTS is described as a stepping stone toward a training-time TRAP formulation (Wang et al., 27 May 2026, Dai et al., 21 May 2026). This suggests that the boundary of TRAP is methodological rather than chronological: the defining feature is not whether adaptation occurs during pre-training, fine-tuning, or inference, but whether trajectories are the unit of control and rewards provide the adaptive signal.

A common misconception is that reward-guided methods necessarily require policy updates. The recent literature contradicts this. EBD achieves large changes in task-oriented behavior with frozen parameters (Wang et al., 27 May 2026), and RTS improves diffusion outputs through test-time trajectory optimization alone (Dai et al., 21 May 2026). Conversely, SlimSearcher, AutoTraj, and TATS show that when parameter updates are used, trajectory-level rewards can be embedded deeply into SFT and RL (Xie et al., 5 Jun 2026, Gong et al., 30 Jan 2026, Rai et al., 13 May 2025). The distinction is therefore best drawn between parameter-free and parameter-updating TRAP variants, not between “true” and “false” forms of the idea.

6. Empirical patterns, strengths, and limitations

The empirical record across the cited works indicates three recurring strengths. First, trajectory-level guidance tends to improve efficiency or data usefulness. SlimSearcher reports reductions in average tool-call rounds by 17%–58% while maintaining or improving accuracy on GAIA, BrowseComp, and XBenchDeepSearch (Xie et al., 5 Jun 2026). AutoTraj uses only about 13K SFT trajectories, compared with Tool-Star’s 54K, yet achieves better TIR performance and much shorter trajectories on average (Gong et al., 30 Jan 2026). TATS maintains or improves downstream action recognition performance under aggressive masking up to $t_k$3, with strong results on UCF101, HMDB51, Kinetics-400, and Something-Something v2 (Rai et al., 13 May 2025).

Second, adaptive, relative, or learned reward signals appear more robust than static penalties. SlimSearcher explicitly contrasts cohort-relative efficiency shaping with absolute penalties such as $t_k$4, arguing that relative anchoring avoids brevity bias across heterogeneous task complexities (Xie et al., 5 Jun 2026). AutoTraj’s learned trajectory reward stabilizes RL relative to format-and-outcome-only supervision (Gong et al., 30 Jan 2026). RTS similarly finds that sparse optimization at PCA-selected key steps outperforms dense scaling under the same NFE budget (Dai et al., 21 May 2026).

Third, inference-time trajectory guidance can expose latent capabilities without retraining. EBD improves Qwen3-8B-Base on AlpacaEval2.0 from 8.8 to 44.5, increases behavioral similarity to instruct-tuned counterparts, and reduces Mistral-7B Math500 latency by 18.9x relative to prior decoding work (Wang et al., 27 May 2026). This suggests that trajectory-level reward control is not merely a training trick but a more general interface for aligning model behavior with task objectives.

The limitations are equally recurrent. Reward dependence is central: if the reward is misaligned, the policy or search procedure may overfit to artifacts rather than desired behavior (Dai et al., 21 May 2026, Wang et al., 27 May 2026). SlimSearcher’s current experiments use uniform tool weights $t_k$5, even though real systems have heterogeneous latency and monetary costs (Xie et al., 5 Jun 2026). AutoTraj’s trajectory reward model is trained on self-generated preference pairs and may not capture all aspects of human-desired reasoning (Gong et al., 30 Jan 2026). TATS is evaluated only for video MAE and action recognition, and its RL-based pre-training increases complexity relative to pure SSL masking (Rai et al., 13 May 2025). These limitations suggest that TRAP-style systems are only as reliable as their trajectory representation, reward definition, and adaptation budget.

7. Relationship to adjacent research directions

TRAP overlaps with, but is not identical to, several adjacent paradigms. It differs from ordinary supervised fine-tuning because it does not treat all correct trajectories as equally desirable. SlimSearcher’s Pareto-efficient filtration and AutoTraj’s repair-and-retain mechanism both reject the premise that correctness alone should determine supervision (Xie et al., 5 Jun 2026, Gong et al., 30 Jan 2026).

It differs from conventional RLHF-style outcome optimization because the reward is often process-sensitive rather than outcome-only. AutoTraj explicitly criticizes sparse outcome-only rewards in tool-integrated reasoning and adds a learned trajectory reward to assess the quality of reasoning paths (Gong et al., 30 Jan 2026). SlimSearcher similarly argues that optimizing only task success produces “efficiency collapse,” with increasingly long and tool-heavy trajectories during RL (Xie et al., 5 Jun 2026).

It also differs from static efficiency penalties or prompt-only control. Prompt-based instructions to “be efficient” are reported as insufficient in SlimSearcher, whereas training-time trajectory shaping changes the underlying decision policy (Xie et al., 5 Jun 2026). Likewise, absolute length penalties are contrasted with adaptive relative shaping in both web-agent RL and reward-guided diffusion search (Xie et al., 5 Jun 2026, Dai et al., 21 May 2026).

Finally, TRAP intersects with energy-based inference and test-time compute scaling. EBD’s reward-tilted posterior over complete responses and RTS’s reward-guided optimization over denoising paths show that the same underlying mathematical view—a prior over trajectories reweighted by reward—can support training-free adaptation (Wang et al., 27 May 2026, Dai et al., 21 May 2026). A plausible implication is that future work may converge on a unified framework in which trajectory-level rewards drive both offline pre-training choices and online inference-time refinement.

Taken together, the current literature portrays TRAP not as a single named algorithm but as an emerging systems principle: use whole trajectories as the atomic units of evaluation, couple them to explicit reward or preference signals, and adapt learning or decoding accordingly. In web agents, this yields efficiency-aware SFT and RL; in tool-use reasoning, repair-based data construction and trajectory reward modeling; in masked video modeling, PPO-trained token selection from motion trajectories; in diffusion, sparse reward-guided denoising search; and in frozen LLMs, reward-tilted decoding without parameter updates (Xie et al., 5 Jun 2026, Gong et al., 30 Jan 2026, Rai et al., 13 May 2025, Dai et al., 21 May 2026, Wang et al., 27 May 2026).