R2FT: Rejecting Rule-based Fine-Tuning

• The paper introduces R2FT as a targeted post-hoc fine-tuning method that employs rule-based corruption to minimize repetitive and irrelevant outputs in diffusion language models.
• It constructs negative samples by algorithmically corrupting reference completions and applies a DPO-style preference loss to steer the model away from undesired generation behaviors.
• Empirical results demonstrate that R2FT improves open-ended generation by enhancing fluency and context alignment while reducing decoding steps by nearly threefold.

Searching arXiv for the primary paper and closely related work on rejection/rule-based fine-tuning. Rejecting Rule-based Fine-Tuning (R2FT) is a post-hoc training scheme for diffusion LLMs (DLMs) that uses rule-based corruption to construct undesired outputs and then trains the model to prefer reference completions over those corrupted alternatives. In the formulation introduced in "Fast and Fluent Diffusion LLMs via Convolutional Decoding and Rejective Fine-tuning" (Seo et al., 18 Sep 2025), R2FT addresses a specific failure mode of diffusion-based decoding in open-ended generation: tokens generated far from the input context often become irrelevant or repetitive. Rather than restricting decoding with hard segmentation, R2FT directly corrects the model’s tendency to over-prefer repetition and common tokens, with the stated goal of improving fluency, contextual alignment, and step efficiency in parallel text generation.

1. Definition and scope

R2FT is defined as a targeted fine-tuning method for DLMs, not as a general-purpose alignment framework for all LLMs. The method is introduced in a setting where autoregressive LLMs generate text one token at a time, while diffusion-based LLMs decode multiple tokens in parallel over several denoising steps (Seo et al., 18 Sep 2025). Within that setting, R2FT is designed to discourage undesirable generations that arise systematically in diffusion decoding, especially at positions far from recent context.

The method is described as inspired by preference optimization schemes such as DPO and SimPO, but with a different object of optimization: it targets undesired outputs rather than desired outputs, and it obtains negatives from rule-based corruption rather than from expensive human annotation or model-generated preference data (Seo et al., 18 Sep 2025). This positioning is important because it distinguishes R2FT from standard supervised fine-tuning, from inference-time penalties, and from RLHF-style preference learning.

A plausible implication is that the “rejecting” in R2FT refers to a training objective that suppresses specific pathological completions rather than to simple dataset filtering. That interpretation is consistent with the paper’s emphasis on unlearning likely errors rather than merely excluding low-quality samples.

2. Long decoding-window failure mode

The immediate motivation for R2FT is the long decoding-window (LDW) problem in diffusion LLMs. The paper states that in DLMs many tokens are sampled in positions far from the recent context, and that these positions are weakly aligned with the input. As a consequence, the model tends to generate repetition of the previous context, high-prior tokens such as common function words, uninformative output, or output that is irrelevant to the input (Seo et al., 18 Sep 2025).

Previous solutions such as semi-autoregressive decoding mitigate this failure mode by splitting windows into blocks, but this sacrifices speed and bidirectionality. The paper therefore frames R2FT as complementary to architectural or decoding-level interventions: instead of enforcing hard segmentation, it modifies the model so that distant tokens are better aligned with context during open-ended generation (Seo et al., 18 Sep 2025).

This problem setting also clarifies why the method is not presented as a generic refusal or safety fine-tuning recipe. In the provided literature, refusal-oriented dynamic adversarial fine-tuning studies analyze harmful-request robustness and refusal geometry in standard causal LLMs, whereas R2FT is introduced specifically for diffusion LMs and their open-ended generation pathologies (Lan et al., 29 Apr 2026). The shared theme is training-time control of undesirable outputs, but the mechanisms and evaluation targets differ.

3. Training objective and negative-sample construction

R2FT operates on supervised fine-tuning data consisting of instruction-reference pairs $(x, y_w)$. For each such pair, the method constructs a negative completion $y_l$ by applying rule-based corruption patterns to the good response. The summary specifies repetition-based corruption patterns such as repeating the last few tokens of the prompt or answer, copy-pasting context tokens, and excessive padding with EOS. The negative is generated algorithmically rather than via model generation or human annotation (Seo et al., 18 Sep 2025).

The procedural description is explicit. One starts with a pretrained and instruction-finetuned model, randomly corrupts reference completions to obtain negatives, continues training for several $200\text{--}300$ steps with the R2FT loss, and does not modify the model architecture or forward process (Seo et al., 18 Sep 2025). The method is therefore post-hoc in the narrow sense that it extends the data and objective rather than redesigning the decoder.

The loss given in the summary is

$$\mathcal{L}_{\text{R2FT}}(\pi_\theta) = -\frac{\gamma}{|y_w|} \log \pi_\theta(y_w | x) - \mathbb{E}_{(x, y)\sim \mathcal{D}}\left[ \log \sigma \left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) \right) \right].$$

The first term is standard fine-tuning loss on the reference completion, and the second term is a DPO-style preference loss encouraging the model to assign higher log-probability to $y_w$ than to $y_l$ (Seo et al., 18 Sep 2025). The summary also gives the penalty weighting

$$s_\theta = \sigma \left( \frac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \frac{\beta}{|y_w|} \log \pi_\theta(y_w|x) \right),$$

with the interpretation that implausible negatives receive little penalty, while negatives that the model would actually assign relatively high probability receive a stronger corrective gradient (Seo et al., 18 Sep 2025).

This suggests that R2FT is not merely rule-based in the sense of hand-authored filters; it is rule-based at the negative-construction stage and preference-based at the optimization stage.

4. Relationship to convolutional decoding and decoding-time control

R2FT is introduced alongside convolutional decoding (Conv), a normalization-based method that narrows the decoding window without hard segmentation (Seo et al., 18 Sep 2025). The two methods are separable but complementary. Conv modifies decoding behavior structurally, whereas R2FT modifies the learned scoring of candidate continuations through training. The paper’s summary states that combining R2FT with Conv further boosts performance, but also that R2FT alone is already a strong improvement over vanilla categorical or semi-autoregressive baselines.

The comparative framing provided in the source can be summarized as follows:

Approach	Needs Human Data?	Main Tradeoff/Issue
SFT (only)	No	Poor open-ended fluency (LDW)
Semi-AR	No	Slow, less flexible
Conv Decoding	No	Needs normalization
R2FT	No	Needs rule tuning

The same source states that SFT alone does not penalize unwanted easy patterns such as repetition and high-prior tokens, that direct training is more robust than applying penalties or filters only at inference time, and that R2FT aims for generalization rather than merely post-hoc filtering (Seo et al., 18 Sep 2025). In that sense, R2FT can be read as a training-time correction to a decoding-time pathology.

A plausible implication is that R2FT is especially valuable when one wants to preserve the main attraction of diffusion models—parallel generation—while reducing the degradation that appears when decoding windows are long.

5. Empirical behavior and claimed effects

The reported empirical effect of R2FT is twofold: it changes local token preferences and it improves system-level open-ended generation. Qualitatively, the summary states that after R2FT the model’s candidate tokens at distant positions are much better aligned with context, repetition and high-prior zones in candidate probability distributions are greatly reduced, and “meaning” tokens rise in rank (Seo et al., 18 Sep 2025). It also states that generations become more informative, fluent, and contextually grounded.

Quantitatively, the paper reports state-of-the-art results on open-ended generation benchmarks among diffusion LM baselines and emphasizes lower step size than previous works (Seo et al., 18 Sep 2025). More specifically, the summary states that R2FT with top-$k$ decoding obtains nearly autoregressive-level or even superior win rates on AlpacaEval, MT-Bench, and Wiki at about one-third the step size, and that with R2FT a step size of $128$ rather than the default $512/1024$ suffices for high-quality open-ended generation, yielding approximately $y_l$0 faster decoding than autoregressive generation (Seo et al., 18 Sep 2025).

The same source also notes several qualifications. R2FT works best when regularized with top-$y_l$1 decoding at inference and has limited benefit with categorical sampling alone. Benefits are reported at both $y_l$2M and $y_l$3B parameter scales. Structural coherence, described as the proportion of coherent, non-repetitive, on-topic generations, is also reported as improved (Seo et al., 18 Sep 2025).

These claims place R2FT in a specific empirical niche: it is not presented as a universal fine-tuning recipe, but as a practical method for improving open-ended diffusion-language generation without giving up speed and flexibility.

6. Position within the broader fine-tuning landscape

R2FT belongs to a broader family of methods that reject, recycle, mask, or reweight low-quality training signals, but it differs materially from each of the neighboring approaches represented in the supplied literature.

RuleR uses predefined rule-based edits to recycle supervised fine-tuning data and improve controllability. It appends rule-instructions to original instructions and may modify responses accordingly; its focus is multi-constraint controllability rather than diffusion-decoding pathologies (Li et al., 2024). SRFT, by contrast, operates on agent trajectories, uses a critic LLM to label each step as productive or harmful, and masks the loss for harmful steps while keeping them in context, thereby retaining signal from unresolved trajectories in SWE-bench-style settings (Slinko et al., 11 May 2026). RIFT repurposes negative trajectories by reweighting the loss with scalar rewards and uses a stabilized piecewise loss so that both positive and negative self-generated samples can contribute to optimization without instability (Liu et al., 14 Jan 2026).

Across these methods, a common pattern is the rejection of coarse all-or-nothing filtering. SRFT rejects step-level errors rather than entire trajectories, RIFT rejects hard-threshold sample discarding in favor of reward-informed reweighting, and R2FT rejects only specific rule-defined corruptions while training the model to prefer reference completions (Slinko et al., 11 May 2026). What changes is the object being filtered or downweighted: steps, full trajectories, scalar-rewarded samples, or synthetically corrupted diffusion completions.

Related work also shows that rule-based or reward-oriented fine-tuning can have undesirable side effects when the target behavior is incompletely specified. In "The Hallucination Tax of Reinforcement Finetuning," standard RFT is reported to reduce refusal rates on unanswerable questions by more than $y_l$4, thereby increasing hallucination, while adding $y_l$5 SUM substantially restores refusal behavior with minimal answerable-task accuracy trade-offs (Song et al., 20 May 2025). In visual reinforcement fine-tuning, explicit “thinking” is reported as unnecessary or even detrimental for simple image-classification tasks, with No-Thinking-CLS-RL matching or outperforming thinking-based variants (Li et al., 20 Mar 2025). These results do not evaluate R2FT directly, but they indicate that rule-based or rejection-oriented fine-tuning methods are highly task-dependent and that training objectives can move the model toward distinct behavioral regimes.

7. Interpretive issues, limitations, and future directions

The published framing of R2FT is careful about scope. It is a post-hoc training stage, it uses rule-based negatives focused on repetition and high-prior-token pathologies, and it is evaluated primarily in the context of open-ended diffusion-language generation (Seo et al., 18 Sep 2025). The summary explicitly notes that R2FT needs rule tuning and that its benefit is limited with categorical sampling alone. Those constraints matter because they delimit where the method should be expected to transfer directly.

A common misconception would be to treat R2FT as equivalent to ordinary rejection sampling fine-tuning. The supplied materials do not support that equation. In RIFT and SRFT, “rejection” refers to selection, masking, or weighting of samples or steps in training data (Liu et al., 14 Jan 2026). In R2FT, the central mechanism is preference learning against algorithmically corrupted negatives, not simple removal of bad examples. Another misconception would be to interpret R2FT as a safety-refusal method; the provided evidence instead situates it in open-ended text generation for diffusion models (Seo et al., 18 Sep 2025).

Future directions are suggested indirectly by adjacent work. Rule-based data recycling points toward richer automatically verifiable constraints (Li et al., 2024). Reward-informed use of mixed-quality data points toward alternatives to hard rejection (Liu et al., 14 Jan 2026). The hallucination-tax results suggest that fine-tuning schemes may need explicit modeling of abstention or uncertainty when deployment requires refusal behavior (Song et al., 20 May 2025). This suggests that a broader research question remains open: whether R2FT-style rule-based negative construction can be generalized beyond repetition and high-prior-token failures to other model-specific pathologies without sacrificing the speed and bidirectionality that motivate diffusion LLMs in the first place.