Closed-Loop Supervised Fine-Tuning

• CL-SFT is a family of supervised fine-tuning methods that exposes models to their own induced state distributions to reduce covariate shift.
• Techniques like CAT-K and RoaD use on-policy rollouts to align training with deployment environments, improving metrics in traffic simulation and end-to-end driving.
• Extensions such as data rewriting and crowdsourced feedback further refine model alignment and accelerate convergence in complex tasks like LLM tuning.

Closed-Loop Supervised Fine-Tuning (CL-SFT) comprises a family of methodologies for mitigating covariate shift and distributional mismatch in supervised learning, particularly in imitation learning, LLM alignment, and complex multi-agent domains. Unlike classical open-loop behavior cloning (BC), CL-SFT exposes the model to its own induced state distribution during training, aligning the training and deployment distributions without requiring reinforcement learning, adversarial imitation, or ongoing expert data collection. This paradigm has proven effective across domains including traffic simulation, LLM fine-tuning, crowdsourced preference alignment, and data-centric optimization for LLMs.

1. Core Problems and Motivation

CL-SFT arises in response to the distributional drift that occurs when a policy trained on off-policy or i.i.d. expert data is deployed in a closed-loop environment. In settings such as next-token prediction in multi-agent traffic simulation or autonomous driving, standard BC minimizes the one-step prediction error using ground truth (GT) data but is prone to cascading error: small misalignments at each step accumulate as the policy visits states not present in the training set—a phenomenon known as covariate shift (Zhang et al., 5 Dec 2024, Garcia-Cobo et al., 1 Dec 2025). This leads to a rapid loss of realism, increased collision or off-road rates, and lower utility in real-world deployments.

Traditional alternatives such as DAgger require querying the expert in novel states, which is impractical or infeasible in many settings, while RL-based fine-tuning is sample-inefficient and often struggles to encode nuanced behavioral distribution (e.g., “realism metrics” for driving). CL-SFT seeks a supervised, data-driven solution that retrains the policy on either rollouts generated by itself (possibly with expert-biased guidance) or by actively rewriting the training data to match the evolving target policy (Zhang et al., 5 Dec 2024, Zhao et al., 18 Sep 2025, Garcia-Cobo et al., 1 Dec 2025).

2. Algorithmic Instantiations

CL-SFT manifests in several formalizations. Two representative algorithmic frameworks are CAT-K and RoaD.

CAT-K (Closest Among Top-K) Rollouts

In CAT-K, the policy is unrolled for T steps. At each step for each agent, the K most likely tokens are enumerated. The candidate whose resulting next state is closest (under some distance metric) to the GT is selected as the action. Simultaneously, the GT transition is quantized to the nearest token in the vocabulary to provide the supervised loss target. This process enforces a balance between sampling from the actual model (on-policy) and retaining valid, quantifiable GT supervision. The supervised objective is standard cross-entropy over these newly induced trajectories:

$$\mathcal{L}(\theta) = -\frac{1}{N\,T}\sum_{i=1}^N\sum_{t=0}^{T-1} \log \pi_\theta(\hat c_t^i \mid h_t, \mathcal{M})$$

No additional regularization is required beyond typical weight decay. CAT-K is integrated after a BC pretraining phase, and fine-tuning repeats until convergence, allowing the model to adapt to its own state visitation distribution (Zhang et al., 5 Dec 2024).

RoaD (Rollouts as Demonstrations)

RoaD generalizes CL-SFT to settings where the action space is continuous or high-dimensional (e.g., end-to-end driving). Here, K i.i.d. candidate actions are sampled at each step, and the action whose predicted trajectory most closely matches the expert’s via a generalized distance metric is selected. If the candidate diverges beyond a threshold, a recovery mode interpolates toward the expert trajectory. Generated rollouts, biased but not fully dictated by expert data, are aggregated with the expert dataset and used as new BC training data, thus adapting the policy under its actual closed-loop error profile (Garcia-Cobo et al., 1 Dec 2025). RoaD does not require hand-crafted reward functions or explicit inverse dynamics.

Method	Action Space	GT Use	Data Regeneration	Applicability
CAT-K	Discrete tokens	Required (quant.)	Every step	Tokenized traffic sims
RoaD	Continuous, trajectories	As reference	Amortized	E2E driving, traffic

A plausible implication is that the ability to handle high-dimensional or continuous control is a principal differentiator between recent CL-SFT variants (Garcia-Cobo et al., 1 Dec 2025).

3. Data Rewriting and Off-Policy Alignment

In LLM fine-tuning, CL-SFT also appears in the form of data rewriting (“closed-loop SFT/data rewriting” as an Editor's term). Supervised fine-tuning is recognized as an off-policy problem: expert data comes from an earlier policy πself while the goal is to optimize expected reward under the evolving target policy πθ. Importance sampling can in principle correct for this distributional gap, but large KL divergence between πself and πθ leads to high-variance estimates and unstable training.

The data rewriting framework actively aligns the training distribution with π_θ prior to optimization. Through an operator 𝒯(·), the supervised dataset is transformed into on-policy or nearly on-policy samples via:

• Self-alignment: Sample k completions from π_θ; if any are correct, add as new on-policy data.
• Guided alignment: If self-alignment fails, introduce a prompt “digest-and-retell” using the expert demonstration, and accept correct completions as rewritten data.
• Fallback: Default to the original expert demonstration if both steps fail.

Fine-tuning then proceeds on the revised dataset with importance weighting, stabilizing the updates by minimizing variance in the policy gap. In practice, rewriting yields a majority of on-policy or near-on-policy data, lowering KL(πmx∥πθ), and resulting in robust, accelerated convergence (Zhao et al., 18 Sep 2025).

4. Extensions: Crowdsourced CL-SFT and Data Optimization

CL-SFT principles extend to large-scale crowdsourced feedback and adaptive data-centric optimization.

Crowd-SFT

Crowd-SFT frameworks introduce user-in-the-loop closed-loop alignment. Feedback is partitioned among disjoint user groups; each group fine-tunes a model clone, and candidates are evaluated against a fixed expert target. Iterative selection, scoring, and regrouping proceed over multiple rounds. Contribution is measured via point-based rewards, with demonstrated high correlation to approximate Shapley values, providing fair and scalable attribution (Sotiropoulos et al., 4 Jun 2025). Multi-model selection and dynamic group assignment are empirically shown to accelerate convergence, reducing target distance by up to 55% in emotion alignment tasks.

Middo: Model-Informed Data Refinement

Model-driven dynamic data optimization incorporates a closed diagnostic loop, identifying suboptimal (complex, diverse, or low-aligned) samples via tri-axial signals: pre-/post-update loss (complexity), embedding cluster dynamics (diversity), and self-alignment with respect to clarity/completeness/factuality. Targeted transforms (simplification, augmentation, or refinement) are applied, evolving the dataset through successive iterations. Performance gains of 7–8 percentage points over open-loop baselines are reported across Alpaca, WizardLM, and LLaMA/Mistral platforms (Tang et al., 29 Aug 2025). As the model evolves, so do the thresholding criteria, maintaining focus on the “frontier” of challenging or suboptimal data.

5. Empirical Validation Across Domains

CL-SFT methodologies have demonstrated state-of-the-art performance across diverse settings:

• Traffic Simulation: CAT-K fine-tuning of a 7M-parameter SMART-tiny model surpassed a 102M-parameter baseline on the WOSAC Realism Meta-Metric (RMM=0.7635 vs. 0.7614), with improved interactive and map sub-metrics and minADE reductions (Zhang et al., 5 Dec 2024).
• E2E Driving: RoaD received a 41% boost in driving score and halved the collision rate (0.0239 vs. 0.0525) in the AlpaSim simulator, outperforming continued BC and expert re-rendered baselines (Garcia-Cobo et al., 1 Dec 2025).
• Mathematical Reasoning: Data rewriting combined with dynamic fine-tuning achieved 42.03% average accuracy on mathematical benchmarks, significantly outperforming vanilla SFT and DFT (Zhao et al., 18 Sep 2025).
• LLM General Tasks: Middo yielded mean improvements of 7.15 points over Alpaca baselines on MMLU, GSM8K, and other tasks (Tang et al., 29 Aug 2025).
• Crowdsourcing: Multi-model closed-loop SFT in Crowd-SFT reduced target distance up to 55% relative to single-model baselines, with robust Shapley-aligned reward distribution (Sotiropoulos et al., 4 Jun 2025).

6. Limitations, Trade-offs, and Open Challenges

Several limitations apply across CL-SFT frameworks:

• Dependence on the initial model quality; poorly initialized models may generate uninformative “on-policy” data and produce unreliable diagnostics (Tang et al., 29 Aug 2025, Garcia-Cobo et al., 1 Dec 2025).
• Simulator or environment requirement for closed-loop rollouts; sim-to-real transfer remains a challenge.
• Additional compute and implementation complexity, e.g., for rollout generation, data rewriting, or multi-group parallel training.
• Possible bias amplification through repeated self-improvement or reward assignment.
• Some methods (such as CAT-K) are restricted to discrete action spaces with small vocabulary and known dynamics, while others (such as RoaD) mitigate these constraints but require alternative proximal metrics.

Potential avenues for extension include integrating RLHF signals for more robust reward shaping, dynamic schedule adaptation for data refinement thresholds, and expanding CL-SFT to domain-specific metrics by replacing automated self-alignment with domain-expert or in-domain LLMs (Tang et al., 29 Aug 2025).

7. Summary Table of Principal CL-SFT Methods

Method	Core Mechanism	Domains	Key Results
CAT-K	Top-K candidate selection, GT quantization	Traffic simulation	7M model surpasses 102M model on WOSAC (Zhang et al., 5 Dec 2024)
RoaD	Sample-K, expert-biased rollouts + recovery	E2E driving, traffic sims	+41% driving score, -54% collisions (Garcia-Cobo et al., 1 Dec 2025)
Data Rewriting	On-policy/guided sample selection, import. weighting	Math reasoning/LLMs	SOTA accuracy, stabilized convergence (Zhao et al., 18 Sep 2025)
Crowd-SFT	Multi-clone closed-loop with group selection	LLM alignment, crowd RLHF	-55% target distance, Shapley-aligned rewards (Sotiropoulos et al., 4 Jun 2025)
Middo	Iterative data diagnostics+transforms	LLM supervised tuning	+7–8 pp on general/coding/math benchmarks (Tang et al., 29 Aug 2025)

A plausible implication is that closed-loop exposure, whether via self-generated trajectories, targeted data rewriting, or model-driven dataset refinement, is converging as a central strategy for addressing the distributional mismatch and brittleness of open-loop supervised fine-tuning in complex sequential, interactive, and multi-agent environments.