Self-Reflective Generation at Test Time

• The paper introduces SRGen, a test-time decoding framework that uses dynamic entropy thresholding to detect uncertainty and apply a transient correction vector.
• It employs token-level monitoring and reflective mechanisms to proactively refine outputs, achieving notable improvements in benchmark performance across reasoning and code tasks.
• SRGen represents a versatile design pattern with variants such as self-refinement, retrieval, and human-guided adaptation, shifting part of error correction from training into inference.

Self-Reflective Generation at Test Time (SRGen) denotes a class of inference-time methods in which a model does not treat decoding as a purely forward-only process, but instead generates, critiques, retrieves, verifies, refines, or selectively restarts generation during the same test-time episode. In the narrow sense introduced by “Self-Reflective Generation at Test Time,” SRGen is a lightweight, plug-and-play decoding framework that reflects before generating at uncertain points by using dynamic entropy thresholding and a transient corrective vector (Mu et al., 3 Oct 2025). In a broader usage across subsequent work, the same label has been applied to parallel self-refinement, reflective retrieval, human-guided knowledge adaptation, compiler- or test-driven correction, and multimodal self-reflective refinement, all of which shift part of the error-detection and error-repair process from training time into inference (Wang et al., 27 Aug 2025, Asai et al., 2023, He et al., 23 Jul 2025).

1. Conceptual foundations

The central motivation for SRGen is the fragility of autoregressive generation on long reasoning traces. The token-level SRGen paper states that LLMs increasingly solve complex reasoning tasks via long chain-of-thought, but their forward-only autoregressive generation process is fragile because early token errors can cascade; it therefore distinguishes its method from post-hoc revision over full drafts and from self-correction learned through expensive training, characterizing those alternatives as fundamentally reactive and inefficient (Mu et al., 3 Oct 2025).

Across the literature, SRGen is not a single algorithm but a design pattern. Some variants intervene at the token level during one decoding pass; some compare parallel trajectories and synthesize a superior solution; some use retrieval or external memory; some invoke execution, compiler, or unit-test signals; and some internalize reflection via RL or supervised fine-tuning. A common misconception is that self-reflection necessarily means rewriting a complete answer after it has already been produced. The narrow SRGen formulation explicitly rejects that restriction by reflecting during generation, while other systems broaden the idea to iterative refinement, redrafting, or verifier-guided rollout selection.

A theoretical account is provided by “The Two-Stage Decision-Sampling Hypothesis,” which decomposes behavior into a sampling policy $\pi_{\mathrm{sample}}$ for generation and a decision policy $\pi_d$ for verification, and argues that RL’s superior generalization stems primarily from improved decision-making rather than sampling capabilities. That work formalizes Balanced Gradient Attribution for surrogate rewards, Unbalanced Gradient Attribution for SFT and KL penalties, and presents a first-principles mechanistic explanation for self-correction in thinking models (Zhao et al., 4 Jan 2026).

2. Token-level proactive SRGen

In the narrow formulation, SRGen monitors token-level uncertainty online. At decode step $t$, with logits $z_t$ and temperature $\tau$, the model distribution is

$p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$

and uncertainty is measured by Shannon entropy

$H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$

A sliding entropy window $\mathcal H_t=\{H_{t-N},\dots,H_{t-1}\}$ yields a dynamic threshold

$\tau_t=\mu(\mathcal H_t)+k\,\sigma(\mathcal H_t),$

and self-reflection is triggered iff $H_t>\tau_t$. When triggered, the model freezes pretrained weights and optimizes a transient correction vector $\pi_d$0, injected into the final hidden state so that corrected logits become

$\pi_d$1

The optimization objective is a hybrid loss

$\pi_d$2

where $\pi_d$3 is a retrospective context-fidelity term over the existing prefix and $\pi_d$4 is anticipatory entropy minimization on the current token. The paper states that minimizing this weighted sum is exactly equivalent to solving $\pi_d$5 subject to $\pi_d$6, so the hybrid loss is presented as a principled constrained optimization rather than an ad hoc blend (Mu et al., 3 Oct 2025).

This design is proactive rather than post-hoc. Reflection occurs only at unusually uncertain tokens, and the correction vector is discarded after the token is sampled. The resulting overhead is bounded: on Qwen2.5-Math-7B over the full AIME2024 test, SRGen triggers on average $\pi_d$7 times per problem, and increasing the inner-loop iteration count from 0 to 9 raises average runtime per problem by at most 50%, after which it plateaus. Empirically, the gains are substantial on mathematical reasoning. On AIME2024 with DeepSeek-R1-Distill-Qwen-7B, SRGen improves Avg@5 from 49.3% to 61.3% and Cons@5 from 50.0% to 63.3%; on Qwen2.5-Math-7B, Avg@5 rises from 14.6% to 22.0% and Cons@5 from 6.7% to 23.3%; on Qwen3-32B, Avg@5 rises from 76.7% to 82.7% and Cons@5 from 80.0% to 90.0% (Mu et al., 3 Oct 2025).

3. Reflection over candidate trajectories

Many SRGen systems operate not on single uncertain tokens but on sets of candidate solutions, drafts, or thought trajectories. In these formulations, the reflective operation is synthesis, redrafting, ranking, or recursive selection over alternative reasoning paths.

Framework	Reflective mechanism	Reported result
GSR / selfRef@4	Generate $\pi_d$8 candidates, then self-refine into a superior solution	73.6% average accuracy across 5 benchmarks
Self-Redraft	Critique suggests pass, refine, or redraft	0.615% average absolute gain over Self-Refine by iteration 16
MetaStone-S1	Sample $\pi_d$9 thought chains, score with SPRM, select best	85.2 on AIME24 and 89.7 on C-Eval for 32B-high
TRT	Strategy-conditioned rollouts, self-verification, knowledge-list updates	100% accuracy on AIME-25/24

“Learning to Refine” introduces Generative Self-Refinement (GSR), in which a unified model first samples $t$0 diverse solutions $t$1, then constructs a self-refinement prompt containing the problem and all candidates, explicitly instructing the model to diagnose flaws, borrow valid insights selectively, and derive a wholly new correct solution if needed. Because prompting alone did not reliably elicit refinement, the paper trains with a hybrid supervised objective,

$t$2

with $t$3. On AIME24, AIME25, AMC22–23, MATH500, and OlympiadBench, selfRef@4 reaches 73.6% average accuracy, compared with $t$4 for maj@4, $t$5 for the best BoN@4 using RM-R1, and $t$6 for fusion with Synthesizer-8B. The paper also reports that after SRGen fine-tuning on 14B and 32B checkpoints, selfRef@4 gains 20–25% relative accuracy over maj@4, and that on Knights-and-Knaves logic puzzles, base selfRef@4 is 8.2% while SRGen selfRef@4 is 48.7% (Wang et al., 27 Aug 2025).

“SELF-REDRAFT” extends the purely exploitative Self-Refine loop by adding a third action, redraft, alongside refine and pass. At each iteration the model emits XML-structured feedback with a <critique> and a <suggestion> tag; refine performs local edits, whereas redraft starts over from scratch. On LiveCodeBench, by iteration 16, Self-Redraft yields an average absolute accuracy gain of 0.615% over Self-Refine across six models, but the paper also emphasizes two bottlenecks: constrained capacity for generating instructive feedback and fragile discriminative judgment. Low Recall on Draft correlates with lower gains ($t$7), and although Self-Redraft increases the improvement rate $t$8, it also increases the regression rate $t$9 (Chen et al., 31 Oct 2025).

MetaStone-S1 unifies generation and process scoring inside one network. A shared backbone $z_t$0 supports a policy head and a self-supervised process reward model head, with only 53M extra parameters for trajectory scoring on the 32B model. Step scores are aggregated by geometric mean,

$z_t$1

and the best trajectory among $z_t$2 is selected for final answer generation. On the 32B model, MetaStone-S1-high reports 85.2 on AIME24, 73.6 on AIME25, 64.2 on LiveCodeBench, and 89.7 on C-Eval (Wang et al., 2 Jul 2025).

Test-time Recursive Thinking (TRT) replaces explicit scoring heads with iterative strategy-conditioned rollouts, a self-verification function $z_t$3, and a knowledge list $z_t$4 of accumulated negative constraints or insights. The loop repeatedly designs strategies, samples candidates, selects

$z_t$5

and updates the knowledge list with loser-versus-winner reflections. On AIME-25 and AIME-24, open-source models reach 100% accuracy, and on LiveCodeBench hard problems, closed-source models improve by 10.4–14.8 percentage points after 8 rounds (Zhuang et al., 3 Feb 2026).

4. Retrieval, memory, and human-guided SRGen

A distinct SRGen line treats reflection as control over retrieval, memory, or expert interaction rather than as direct rewriting of the current draft. In these systems, self-reflection determines whether outside evidence, prior experience, or human guidance should enter the prompt.

Self-RAG augments a generator LM with special reflection tokens for retrieval-on-demand, relevance, support, and utility. The model first emits a retrieval-gating token $z_t$6; if retrieval is triggered, it evaluates retrieved passages with relevance and support tokens and uses segment-level beam search with critique-token weights. The adaptive retrieval score is

$z_t$7

and retrieval occurs when this score exceeds a threshold $z_t$8. On representative tasks, Self-RAG$z_t$9 reports 54.9 on PopQA, 72.4 on PubHealth, and 66.9% ASQA citation precision, compared with 46.7, 40.2, and 5.5% for Alpaca$\tau$0+RAG; Self-RAG$\tau$1 reports 55.8, 74.5, and 70.3% (Asai et al., 2023).

ARIA implements SRGen as uncertainty-guided self-dialogue plus human-in-the-loop knowledge adaptation. For each instance $\tau$2, it produces $\tau$3, forms a self-dialogue $\tau$4 from reflective questions, maps that to a confidence level $\tau$5, and decides whether to query an expert subject to a budget $\tau$6. Human feedback updates a timestamped knowledge repository $\tau$7 whose items have timestamps and status labels $\tau$8, and retrieval is ranked by

$\tau$9

On TikTok Pay CDD name screening with GPT-4o base at $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$0, ARIA reports Sensitivity 0.8910 and Specificity 0.8026, compared with 0.7051 / 0.6539 for static GPT-4o, 0.6987 / 0.6791 for offline FT, 0.8333 / 0.7462 for RAG, and 0.8718 / 0.7853 for simple uncertainty. The system is deployed within TikTok Pay serving over 150 million monthly active users (He et al., 23 Jul 2025).

Experiential Reflective Learning (ERL) turns reflection into a transferable heuristic memory. From each task trajectory $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$1 and binary reward $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$2, a reflection function $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$3 extracts a structured heuristic; at test time the system retrieves the top-$p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$4 heuristics by an LLM-based scorer $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$5 and injects them into the prompt. On Gaia2, ERL reaches 56.1 success rate, compared with 48.3 for ReAct, 46.4 for few-shot raw trajectories, 50.9 for ExpeL, and 50.8 for AutoGuide. The ablation “no retrieval (inject all heuristics)” scores 53.8, and the paper states that selective retrieval is essential (Allard et al., 25 Mar 2026).

5. Domain-specific realizations

In code generation, SRGen often couples generation with explicit verification. OriGen uses two LoRA adapters on a shared base model: Gen LoRA produces an initial Verilog candidate $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$6, and Fix LoRA iteratively repairs $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$7 using compiler errors $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$8 from Icarus Verilog until compilation passes or an iteration budget $p_t(v)=\mathrm{softmax}(z_t/\tau)_v,$9 is exhausted. On VerilogEval–Human, Gen LoRA reaches 54.4% pass@1, exceeding the prior best open-source model by 12.8% and slightly exceeding GPT-4 Turbo’s 54.2%; on VerilogFixEval, Fix LoRA reaches 89.1% syntactic correctness and 33.5% functional correctness, with syntactic correctness 19.9 points above GPT-4 Turbo’s 69.2% (Cui et al., 2024).

RefleXGen frames SRGen as an iterative security loop consisting of initial code generation, automated security assessment, retrieval of relevant secure-coding exemplars, prompted self-reflection and refinement, and update of a local security knowledge base. Its formal update rule reweights the generation distribution as

$H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$0

where $H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$1 is a security score. On a CWE benchmark, RefleXGen raises GPT-3.5 Turbo from 75.5% to 89.1% Sec.Rate, GPT-4o from 92.3% to 99.0%, CodeQwen-1.5 from 83.7% to 88.2%, and Gemini-1.0Pro from 80.2% to 86.0% (Wang et al., 27 Oct 2025).

CoSPlay addresses GT-free code generation by co-evolving code candidates and self-generated unit tests. It constructs an execution matrix

$H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$2

derives code pass counts $H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$3 and unit-test pass counts $H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$4, then alternates code cleaning, breaking spurious code–UT coupling, code fixing, and replacing zero-discrimination UTs. Final selection uses output-consensus clustering among top pass-count codes. On Qwen2.5-7B-Instruct, CoSPlay improves average BoN from 22.1% to 33.2% and UT accuracy from 14.6% to 78.3%; on CURE-7B it improves BoN from 32.9% to 38.6% (Hu et al., 22 May 2026).

ReflexiCoder internalizes the full reflection–correction trajectory into model weights through RL-zero training. Its output format explicitly contains >, <answer>, <reflection>, and revised <answer> segments, and the reward combines format compliance, iterative quality improvement, and efficiency. At inference time, no external executor is used in pure SRGen mode. ReflexiCoder-8B reports 94.51% on HumanEval, 81.80% on MBPP, 35.00% on BigCodeBench, 52.21% on LiveCodeBench, and 37.34% on CodeForces in a single-attempt setting, while reducing inference-time compute overhead by approximately 40% (Jiang et al., 6 Mar 2026).

In medicine, MedReflect teaches a single-pass reflection chain $H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$5 that factorizes as

$H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$6

Its stages are initial hypothesis generation, self-questioning, self-answering, and decision refinement, with special tokens such as <Think> and <Modified>. Fine-tuned on only 2,000 examples, MedReflect-7B raises MedQA from 57.0 to 74.2, MedMCQA from 55.6 to 77.1, PubMedQA from 72.7 to 85.2, and GPQA Genetics from 36.2 to 61.1 (Huang et al., 4 Oct 2025).

In multimodal generation, AlphaGRPO combines GRPO with a Decompositional Verifiable Reward (DVReward), which decomposes a request into atomic semantic and quality questions, evaluates them with a verifier, and aggregates scores as

$H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$7

Its inference-time self-reflective refinement loop repeatedly diagnoses errors, generates a diagnostic reasoning string, samples a refined image conditioned on the diagnosis, and stops when $H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$8 or $H_t=-\sum_{v\in\mathcal V} p_t(v)\log p_t(v).$9. On TIIF-Bench and GenEval, Bagel baseline reports TIIF-short 75.2%, TIIF-long 78.6%, and GenEval 84.0, while AlphaGRPO + Inf. SRR reports 83.9%, 83.2%, and 88.2 (Huang et al., 12 May 2026).

6. Empirical profile, limitations, and open questions

The empirical record suggests that SRGen is best viewed as a family of test-time control mechanisms rather than a single universally dominant recipe. Some variants are fully training-free and plug-and-play, such as the token-level entropy-triggered method; some require hybrid supervised fine-tuning, as in GSR; some internalize reflection through RL-zero or GRPO; and some require external signals at inference, including retrieval, human experts, compiler feedback, or code execution. This suggests that “test-time self-reflection” does not imply the absence of training or tools, but rather a relocation of at least part of the correction loop into inference (Mu et al., 3 Oct 2025, Wang et al., 27 Aug 2025, Jiang et al., 6 Mar 2026).

Several limitations recur across papers. Self-Redraft shows that intrinsic exploration–exploitation balancing remains fragile: low Recall on Draft correlates with lower gains, and higher $\mathcal H_t=\{H_{t-N},\dots,H_{t-1}\}$0 is accompanied by higher $\mathcal H_t=\{H_{t-N},\dots,H_{t-1}\}$1, so redrafting can correct wrong solutions but also break correct ones (Chen et al., 31 Oct 2025). Self-RAG shows a controllability trade-off: increasing the weight on support tokens can push citation precision up to $\mathcal H_t=\{H_{t-N},\dots,H_{t-1}\}$2, but at the cost of fluency, with MAUVE decreasing (Asai et al., 2023). ERL reports prompt-bloat and retrieval-scaling issues, noting that even $\mathcal H_t=\{H_{t-N},\dots,H_{t-1}\}$3 heuristics can double token usage and increase latency and cost by $\mathcal H_t=\{H_{t-N},\dots,H_{t-1}\}$4, while conflict resolution remains unresolved (Allard et al., 25 Mar 2026).

A plausible implication is that SRGen performance is increasingly bottlenecked not by raw generation capacity alone, but by the quality of the reflective signal: uncertainty estimation, critique generation, verifier calibration, unit-test discriminativeness, retrieval selectivity, or expert-query formulation. This interpretation is directly consistent with works that emphasize dynamic entropy thresholding, process reward modeling, retrieval gating, knowledge-repository maintenance, or pass-count-based self-play as the real control surface of inference-time improvement. Under that reading, SRGen marks a shift from scaling generation only to scaling decision quality during generation.