Self-taught ActioN Deliberation (SAND)

• Self-taught ActioN Deliberation (SAND) is a framework that teaches LLM agents to explicitly deliberate over multiple candidate actions by generating self-critique and comparing alternatives.
• The method uses self-consistency action sampling and execution-guided rollouts to produce enriched trajectory data, which is used for iterative fine-tuning of the agent.
• Empirical results show that SAND significantly enhances decision-making, outperforming standard ReAct-style supervision by reducing over-commitment and promoting selective deliberation.

Searching arXiv for the SAND paper and a closely related deliberative acting paper to ground the article. Self-taught ActioN Deliberation (SAND) is a training framework for LLM agents that teaches explicit, step-wise deliberation over candidate actions before committing to one, using only the agent’s own base model as a teacher. In SAND, ordinary ReAct-style expert trajectories are augmented with self-generated “action deliberation thoughts” that enumerate plausible alternative actions, critique them using rollouts, and explain why the chosen action is preferable. These enriched trajectories are then used to finetune the agent, and the procedure is iterated. The framework targets interactive, text-based environments in which ambiguity at individual decision points can make agents trained only by imitation or preference optimization over-commit to superficially plausible but suboptimal actions (Xia et al., 10 Jul 2025).

1. Problem formulation and motivation

SAND is situated in the setting of interactive LLM agents that interleave a natural-language thought $z_t$ with a discrete action $a_t$, then receive an observation $o_t$ from the environment after executing that action. The paper considers ALFWorld, ScienceWorld, and WebShop as representative environments. The task begins with an instruction $u \in \mathcal{U}$, and the history up to step $t-1$ is

$$h_{t-1} = (u, z_1, a_1, o_1, \dots, z_{t-1}, a_{t-1}, o_{t-1}).$$

A trainable LLM agent $T_\theta$ samples a thought and then an action conditioned on this history; a full trajectory of length $L$ is denoted by $e = (u, z_1, a_1, o_1, \dots, z_L, a_L)$, and the environment returns a scalar task reward $r(u,e)\in[0,1]$ (Xia et al., 10 Jul 2025).

The immediate motivation is the limitation of standard ReAct-style supervised finetuning and trajectory-level preference optimization. In the formulation used by the paper, supervised finetuning trains the model to imitate expert trajectories step-wise, while preference-based methods such as ETO, DMPO, and IPR typically optimize for successful or preferred rollouts via pairwise comparisons. Both families primarily teach the model what to do by imitating the expert action and its rationale or by preferring one rollout over another. They do not explicitly teach the model to enumerate plausible alternative actions at a state, to evaluate them in depth, or to articulate why the chosen action is better than alternatives (Xia et al., 10 Jul 2025).

The paper identifies three consequences. First, an agent may over-commit to plausible but suboptimal actions when several actions appear linguistically reasonable. Second, action-space exploration is poor because the model mostly observes expert actions rather than near-miss alternatives. Third, there is no explicit subroutine for comparative reasoning over actions. SAND is designed to address precisely these deficiencies by making action comparison an object of supervision rather than an implicit by-product of rollout-level optimization.

2. Formal structure and learning objective

The probability assigned by the policy $a_t$0 to a trajectory $a_t$1 is written as

$a_t$2

The initial behavior model is obtained by supervised finetuning on expert trajectories $a_t$3:

$a_t$4

SAND then constructs a second dataset, $a_t$5, of deliberation trajectories. A deliberation trajectory $a_t$6 preserves the original action sequence but replaces some ordinary thoughts $a_t$7 with richer deliberation thoughts $a_t$8 that explicitly compare actions. Finetuning on these trajectories uses

$a_t$9

The resulting supervision is still standard next-token cross-entropy over all tokens in $o_t$0, but the token stream now contains explicit comparative reasoning over action alternatives (Xia et al., 10 Jul 2025).

A central design choice is that all deliberation data are self-generated by a frozen instruction-tuned base model $o_t$1. Human supervision enters only through the initial expert trajectories. This is the sense in which the method is “self-taught”: critiques and synthesized deliberation thoughts are produced by the model’s own base version rather than by additional human annotation.

3. Action deliberation mechanism

The core loop of SAND operates at individual decision steps along an expert trajectory. It first samples candidate actions from the current agent using self-consistency action sampling:

$o_t$2

Along with the expert action $o_t$3, these samples form a candidate set. The framework then decides whether full deliberation is necessary through an inconsistency indicator:

$o_t$4

If all samples and the expert action coincide, the step is treated as trivial and no deliberation is inserted. If at least two distinct actions exist, the state is treated as uncertain and deliberation is triggered (Xia et al., 10 Jul 2025).

For each unique candidate action, SAND performs an execution-guided rollout. The first action at step $o_t$5 is fixed to the candidate, and the agent then continues the episode according to $o_t$6, producing a rollout trajectory $o_t$7 and a final reward $o_t$8. This provides an environment-grounded evaluation of the consequences of each action rather than a purely linguistic comparison.

The frozen base model then generates a concise natural-language critique for each candidate action:

$o_t$9

The critique prompt provides the task instruction, the current interaction history, the sampled action, the simulated future, and asks for a 3-sentence paragraph titled Action Evaluation:. The prompt guides the model to comment on action validity, whether the action advances or hinders progress toward the goal, relevant affordances or commonsense, common failure patterns, and an overall practical judgment. The critiques are explicitly natural language rather than scalar scores (Xia et al., 10 Jul 2025).

Finally, the base model synthesizes a single deliberation thought from the candidate-action/critique pairs:

$u \in \mathcal{U}$0

where $u \in \mathcal{U}$1 is the number of unique candidates. The synthesis prompt supplies a private scratch-pad listing each candidate action with its critique and imposes a hard constraint that the final Action: must be the expert action, or, under the optional expert-switch mechanism, a better discovered action. The output structure is:

$t-1$3

This text becomes the new thought $u \in \mathcal{U}$2 at step $u \in \mathcal{U}$3. SAND thereby converts execution traces and verbal critiques into explicit training targets for comparative action selection (Xia et al., 10 Jul 2025).

4. Iterative training procedure and implementation details

The training loop begins with behavior initialization: the base instruction-tuned LLM $u \in \mathcal{U}$4 is finetuned on expert trajectories to produce an initial agent $u \in \mathcal{U}$5. SAND then constructs deliberation trajectories for the expert dataset and finetunes the agent on the new dataset. The procedure is iterative: after each round, $u \in \mathcal{U}$6 is replaced with $u \in \mathcal{U}$7, and the next iteration uses the updated policy to sample candidate actions and generate further deliberation data (Xia et al., 10 Jul 2025).

The paper uses Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct as base models. Expert trajectories are taken from Song et al. (2024b). The initial supervised finetuning uses batch size 64, learning rate $u \in \mathcal{U}$8, a cosine learning-rate scheduler, and 3 epochs, implemented in OpenRLHF on 8 × NVIDIA A100 80GB GPUs. SAND runs for $u \in \mathcal{U}$9 iterations. Self-consistency sampling uses temperature 1.0, with $t-1$0 candidate actions for ALFWorld and ScienceWorld and $t-1$1 for WebShop. Critiques are generated by the frozen base model at temperature 0. The deliberation finetuning stage uses the same learning rate and batch size as the initial supervised finetuning, with 3 epochs for iteration 1 and 1 epoch for iterations 2 and 3 to avoid overfitting (Xia et al., 10 Jul 2025).

The optional expert-switch mechanism allows replacement of the expert action if the agent discovers a rollout with higher reward than the expert path. The paper notes that this mechanism is disabled for ScienceWorld because of shortcut issues. At inference time, the final agent receives the same ReAct-style evaluation prompt and directly outputs Thought: ... Action: ... per step. It does not sample multiple actions or run rollouts during inference; the deliberation procedure has been distilled into the policy through supervised finetuning (Xia et al., 10 Jul 2025).

A plausible implication is that SAND shifts most of the cost of comparative reasoning from test time to training time. The method does not require a separate reward model or an explicit test-time search controller, but it relies on training-time access to an executable environment for candidate rollouts.

5. Empirical performance and ablation findings

The main evaluation uses ALFWorld and ScienceWorld, with WebShop reported in the appendix. ALFWorld uses binary success reward, ScienceWorld uses fine-grained partial rewards, and the metric is average reward over tasks. Test-time decoding temperature is 0 for all agents (Xia et al., 10 Jul 2025).

Model / setting	Average reward
Llama-3.1-8B + SFT	~72.9
Llama-3.1-8B + SAND (I=3)	88.9
Qwen2.5-7B + SFT	~69.4
Qwen2.5-7B + SAND (I=3)	84.6
ETO & MPO	82.1
WKM	78.5

Across the two representative interactive agent tasks, SAND achieves an average 20% improvement over initial supervised finetuning and outperforms strong agent-tuning baselines. The paper also reports that SAND outperforms or matches large proprietary LLM agents such as GPT-4o and Llama-3.1-70B + MPO on ALFWorld and ScienceWorld. Performance improves across iterations, and the later iterations show notably high generalization to unseen ALFWorld tasks, with unseen performance exceeding seen performance (Xia et al., 10 Jul 2025).

The ablation study isolates the role of self-consistency action sampling (SAS) and execution-guided action critique (EAC). Removing SAS and prompting the base model to imagine alternatives directly often performs worse than the supervised-finetuning baseline, with the paper attributing this to random or degenerate candidate actions. Removing EAC still improves over supervised finetuning, indicating that deliberation over multiple actions alone is useful, but full SAND performs best, showing that rollout-grounded critiques further improve thought quality (Xia et al., 10 Jul 2025).

The step-level analysis reports two derived quantities: per-step average reward, defined as final reward divided by the number of steps per task and then averaged, and per-step deliberation rate, defined as the fraction of steps where the agent emits a deliberation-style thought. Both quantities generally increase across SAND iterations. On ScienceWorld unseen tasks, difficulty bands are defined by tertiles of the baseline Llama-3.1-8B reward distribution; the median deliberation rate is approximately 0.75 for hard tasks and approximately 0.30 for easy tasks across iterations. This suggests that the finetuned policy learns a selective pattern of deliberation, thinking more on difficult tasks and less on easier ones (Xia et al., 10 Jul 2025).

Inference-time cost rises because learned deliberation produces longer thoughts. The paper reports approximately 2–3× tokens per task relative to supervised finetuning, while noting that test-time Best-of-$t-1$2 or Q-value search methods may cost 5× tokens. Later iterations show slight reductions in token usage, which the paper interprets as learned efficiency (Xia et al., 10 Jul 2025).

The appendix evaluation on WebShop indicates that the approach is not restricted to ALFWorld and ScienceWorld. With Llama-3.1-8B as the base, the reported values are 55.3 for the base model, 65.4 for supervised finetuning, 68.5 for SAND iteration 1, 72.4 for iteration 2, and 71.8 for iteration 3, corresponding to a consistent improvement of about 10% over supervised finetuning (Xia et al., 10 Jul 2025).

6. Relation to adjacent approaches, limitations, and significance

SAND is explicitly contrasted with ReAct and plain chain-of-thought prompting. ReAct interleaves thinking and acting but still selects a single action at each step; SAND adds a training-time layer in which the model learns from multi-candidate evaluation and environment-grounded critiques. The paper also positions SAND as orthogonal to tool-use frameworks, which emphasize tool invocation, and as distinct from RLHF or preference optimization, which typically rank trajectories or step-level outputs rather than synthesize natural-language comparisons explaining why one action is better than others (Xia et al., 10 Jul 2025).

The framework is also placed in a broader self-improvement lineage that includes STaR, Self-Refine, SELF, and Agent-R. In that comparison, the distinctive contribution of SAND is not generic self-critique at the text level but step-wise evaluative comparison among actions in an interactive environment. The paper further argues that SAND is complementary to test-time search methods based on process reward models or Q-value models such as QLASS and AgentRM, because SAND concentrates deliberation cost in training and supports single-pass inference, whereas search-based methods expend additional compute at test time (Xia et al., 10 Jul 2025).

A broader tradition of deliberative acting integrates acting and planning by interleaving execution with online look-ahead, as in systems based on the Reactive Acting Engine and the UCT-like Monte Carlo planner UPOM, which use the same operational models for planning and acting (Patra et al., 2020). SAND operates in a different technical regime—ReAct-style LLM agents rather than hierarchical operational models—but it shares the premise that deliberation should be an explicit component of action selection rather than an implicit by-product of imitation alone.

The paper notes several limitations. Learned deliberation increases inference-time token usage. Execution-guided critique depends on an executable environment that can simulate candidate rollouts and provide rewards. In some environments, discovered alternatives may exploit shortcuts that harm generalization, which is why the expert-switch mechanism is disabled for ScienceWorld. Because all deliberation data are self-generated by the base model, systematic bias or error in the critiques can be distilled into the finetuned policy (Xia et al., 10 Jul 2025).

The significance of SAND lies in its reformulation of agent tuning as supervision over comparative decision making. The paper’s core argument is that explicit step-wise action deliberation, learned via self-critique and rollout comparisons, yields better policies than imitation or global preference ranking alone because the agent learns “why this action, not others” at ambiguous states. This suggests a shift in LLM-agent training from reproducing successful behavior to internalizing discriminative reasoning over nearby alternatives.