Long-Horizon LLM SFT Techniques

• Long-horizon LLM SFT is a method to adapt large language models for tasks that demand extended context and persistent multi-step reasoning.
• It employs techniques like trajectory splitting, negative-augmented training, and chain-of-thought enhancements to overcome standard context limitations.
• Empirical benchmarks demonstrate marked improvements in planning, constraint satisfaction, and overall performance on long, complex tasks.

Long-horizon LLM SFT (Supervised Fine-Tuning for Long-Horizon LLM tasks) encompasses a class of methodologies for adapting foundation models toward robust, constraint-satisfying, and contextually persistent performance on tasks where solution trajectories or contexts far exceed standard prompt lengths. Such settings, including real-world planning, multi-step reasoning, and agentic problem-solving, pose unique technical and methodological challenges—ranging from credit assignment under context fragmentation to maintaining global coherence over thousands of tokens. Recent advances demonstrate that specialized SFT regimes, often augmented by negative feedback or structural data manipulation, are critical to scaling LLM capability on these long-horizon challenges.

1. Defining Long-Horizon SFT

Long-horizon SFT refers to supervised learning strategies tailored to tasks where input or trajectory lengths approach or exceed model context limitations (e.g., 7,000–64,000 tokens), or where solving requires maintaining causal and semantic dependencies over extensive spans. Standard SFT (minimizing token-level cross-entropy on reference plans or problem solutions) often fails in these regimes due to (a) early context degradation, (b) diluted attention to relevant constraints, and (c) the inability to propagate credit robustly across lengthy or split contexts. The field addresses not just computational scaling but the statistical and structural design of training data, loss functions, and auxiliary signals—especially via curated negative samples, chain-of-thought scaffolds, or trajectory-splitting techniques (Chen et al., 2024, Liu et al., 19 Feb 2026).

2. Methodologies: Architectures and Data Schemes

2.1 Trajectory-Splitting SFT

For tasks where full trajectories exceed $L_{\max}$ (the model's context size), trajectory-splitting SFT decomposes each trajectory $\tau=(s_1, a_1, \dots, s_N, a_N)$ into overlapping sub-trajectories of length $L$ and overlap $O$. Each sub-trajectory includes a prefix $p$ (task spec, high-level context) to “pin” global information and ensure continuity.

Key formulae:

• $t_i = 1 + (i-1)(L-O)$ denotes chunk start indices.
• SFT loss computed per sub-trajectory:

$$\mathcal{L}_{\text{split‐SFT}} = - \sum_{i=1}^K \sum_{t=t_i}^{t_i+L-1} \log P_\theta(a_t \mid [p, s_{t_i}, a_{t_i}, ..., s_{t-1}, a_{t-1}])$$

Empirically, splitting SFT recovers global planning ability and long-dependency credit assignment that are lost with naïve truncation (Liu et al., 19 Feb 2026).

2.2 Feedback-Aware and Negative-Augmented SFT

Feedback-Aware Fine-Tuning (FAFT) augments the SFT objective with plan-level or step-level structured feedback. Using benchmarks such as TravelPlanner, each auto-evaluated output is labeled across $K$ constraints ($$f=(f_1,\dots,f_K), f_k\in\{\mathrm{success}, \mathrm{fail}\}$$) to construct sets of positive (“all-success”) and negative samples.

The objective becomes a mixture of standard log-likelihood on positive examples and a scaled penalty on negatives:

$$\mathcal{L}_{\mathrm{FAFT}}(\theta) = \mathbb{E}_{(x, f, y) \in \mathcal{D}_{\mathrm{pos}}} [-\log p_\theta(y \mid x, f)] + \lambda \mathbb{E}_{(x, f, \hat y) \in \mathcal{D}_{\mathrm{neg}}} [-\log p_\theta(\hat y \mid x, f)]$$

where $\lambda$ controls the positive/negative balance. This approach has demonstrated significant gains in final constraint satisfaction rates for long-horizon planning (Chen et al., 2024).

2.3 Chain-of-Thought (CoT) Enhanced SFT

On long-context symbolic reasoning (e.g., math), SFT with long and diverse chain-of-thought (CoT) traces (>2,400 tokens, 20–40 atomic steps) produces the largest gains, far exceeding mixed or short-form data influences. Zone-of-proximal-development (ZPD) filtering improves data utility by focusing training on “hard” examples (Ou, 3 Sep 2025).

3. Specialized Losses and Policy Alignment

Standard SFT, when optimized in isolation, can overfit to offline data distributions and misalign with downstream online RL policy distributions encountered in agentic deployments. The PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting) family of objectives introduces importance sampling reweighting—at token, block, or sequence level—such that:

• Tokens/trajectories with higher likelihood under the RL policy receive higher loss weighting.
• Sequence-level: $$w_{1:T}(x,y)=\frac{\pi_\theta(y|x)}{\pi_\beta(y|x)}$$ multiplies the per-trajectory loss.
• Token-level: per-timestep and discounted suffix-wise weights $G_t$.

In empirical studies, PEAR-SFT followed by RL (e.g., GRPO) yields up to +14.6 percentage-point post-RL gains on challenging mathematics and logic tasks, while also producing more RL-aligned parameter updates and reduced drift in parameter space (Zhang et al., 1 Feb 2026). Inclusion of verified negative trajectories further refines policy avoidance of failure modes.

4. Benchmarking and Empirical Outcomes

4.1 Planning and Real-World Tasks

The TravelPlanner benchmark diagnoses LLM agent performance under multi-day, constraint-rich travel itinerary generation. Extensive SFT and FAFT experiments benchmark agents on:

Planner	Delivery	Comm. (µ/M)	Hard (µ/M)	Final Pass
Vanilla Llama-3-8B	94.4%	49.5/1.1%	7.9/0.0%	0.0%
+SFT	97.8%	64.2/11.1%	12.4/6.1%	3.9%
+FAFT	98.9%	81.7/28.9%	36.9/15.0%	8.3%

FAFT, with explicit constraint-level feedback (both positive and negative), doubles the Final Pass Rate compared to SFT and significantly increases hard-constraint and commonsense satisfaction (Chen et al., 2024).

4.2 Extremely Long-Horizon Agent Tasks

KLong, a 106B-parameter LLM, is trained on trajectories exceeding practical model window sizes via trajectory-splitting SFT, cold-started with comprehensive SFT for agentic primitives and scaled with progressive RL. On PaperBench, KLong achieves a score of 62.59 (+11.28% versus 1T-parameter open-source baselines), with further generalization to SWE-bench Verified and MLE-bench tasks.

Ablations reveal that splitting SFT alone raises performance from 38.6 to 55.9 points and increases average assistant turns from 114.9 to 732.7, with progressive longer-RL stages yielding additional improvements (Liu et al., 19 Feb 2026).

4.3 Reasoning on Lightweight MLLMs

Long-CoT SFT on 40K text-only, difficulty-filtered examples raises average “boxed” answer accuracy by ∼3–10 points, outperforming shorter and multimodal data by a wide margin. For MLLMs, SFT on long-form CoT is required for further RL-stage gains (Ou, 3 Sep 2025).

5. Practical Guidelines and Best Practices

• Context Compression: Apply context scrubbing/pruning agents to reduce irrelevant input columns or rows, as long contexts (>10K tokens) degrade focus and increase hallucination risk (Chen et al., 2024).
• Feedback Integration: Annotate plans or reasoning traces with explicit constraint evaluations and integrate these labels as auxiliary training signals.
• Negative Sample Expansion: Systematically generate synthetic negative examples using model rollouts, oracle/constraint checkers, or automated judge/rubric pipelines; balance positive and negative loss terms.
• Chain-of-Thought & Difficulty Filtering: Emphasize long, diversified chain-of-thought and stratify training by ZPD for maximal reasoning gain (Ou, 3 Sep 2025).
• Trajectory Splitting for Context Overreach: For $N\gg L_{\max}$, split episodes with overlapping context and prefix pinning to preserve global conditioning (Liu et al., 19 Feb 2026).
• RL Preparation: Use SFT objectives (e.g., PEAR) that explicitly account for downstream RL policy distribution; avoid optimizing SFT in isolation from the intended interactive or RL policy regime (Zhang et al., 1 Feb 2026).
• Objective Structure: Prefer cross-entropy-based methods augmented with feedback and reweighting to unstable RL (e.g., PPO) when oracle evaluators or explicit constraints are available.

6. Limitations and Open Challenges

• Credit Assignment Boundaries: Trajectory splitting with fixed-length and overlap is an approximate decomposition, potentially missing dependencies longer than the chunk+overlap window.
• Compute Demands: Scaling progressive RL with ultra-long rollouts (e.g., 12-hour agentic environments) necessitates substantial computational infrastructure.
• Reward Bias and Evaluation: Automated judges, heuristic rubrics, and synthetic feedback risk introducing systematic biases; evaluation under human or multi-metric assessment remains underexplored.
• Policy Distribution Mismatch: Misalignment between SFT training data and downstream sampled policy can hinder online learning efficiency—a gap directly addressed by PEAR but requiring further analysis for more diverse tasks (Zhang et al., 1 Feb 2026).
• Sample Saturation: Increasing SFT/FAFT data size beyond certain thresholds shows early saturation, pointing to diminishing returns without further algorithmic innovations (Chen et al., 2024).

7. Extensions and Generalization

Best practices from long-horizon LLM SFT have been generalized to diverse long-context and agentic domains—multi-step robotics, supply-chain logistics, verifiable mathematical reasoning, and research reproduction. Whenever constraint checkers or oracle evaluators exist, negative-augmented SFT and structured feedback tokens demonstrably improve sample efficiency and convergence. “Splitting + feedback-aware SFT” is a broadly recommended recipe for tasks where solution length, compositional dependency, and plan validity are critical.

Current research converges on the principle that SFT is not a one-size-fits-all pretraining step; rather, its structure, objective, and data balance should be designed in function of the end-to-end long-horizon deployment—accounting for context length, policy shift, and feedback structure (Chen et al., 2024, Ou, 3 Sep 2025, Zhang et al., 1 Feb 2026, Liu et al., 19 Feb 2026).