Base-Anchored Training Curriculum

• Base-anchored training curriculum is a strategy that starts with a pretrained, simpler model to initialize learning and progressively shifts to harder objectives.
• It bridges stable methods like Flow Matching with challenging targets such as MeanFlow and verifier-based RL to control optimization difficulty.
• Empirical results show notable improvements in metrics and training speed across image generation and program reasoning tasks.

Base-anchored training curriculum denotes a class of training strategies in which optimization begins from a base model already aligned with an easier, compatible regime, and then progresses through staged objectives that preserve that alignment while moving toward a harder target. In recent work, this pattern appears in two technically distinct settings: Discrete Meanflow (DMF), where a pretrained Flow Model anchors a staged transition from Flow Matching to MeanFlow in image generation, and ANCORA, where iterative self-distilled SFT projects a base model onto a valid-output manifold before verifier-driven self-play and RL in program reasoning (Hsu et al., 10 Apr 2026, Yang et al., 30 Apr 2026). In both cases, anchoring is introduced to control optimization difficulty rather than to replace the final objective.

1. Core definition and shared structure

In DMF, the curriculum is explicitly described as bridging Flow Matching and MeanFlow. It starts from a pretrained Flow Model, uses the Flow Matching objective at the first stage, and then progressively decreases a discretization step so that training moves toward the harder MeanFlow objective. The curriculum is therefore “base-anchored” because initialization, the first target, and the early optimization geometry all come from the pretrained flow model (Hsu et al., 10 Apr 2026).

In ANCORA, the curriculum is described as an anchored-curriculum framework in which a unified policy alternates between a Proposer that synthesizes specifications and a Solver that produces verified solutions. It is anchored because training does not begin from raw RL on a sparse verifier. Instead, the base model is first projected onto the valid-output manifold with iterative self-distilled SFT, and the subsequent RL curriculum is restricted by a UCB-guided DAG of verified seeds (Yang et al., 30 Apr 2026).

Taken together, the two instantiations share a common structure: initialization from a base model, an initial phase that remains compatible with that base model, and staged progression toward a more difficult objective under explicit constraints that prevent drift into uninformative or unstable regions.

Instantiation	Base anchor	Curriculum progression
DMF	Pretrained Flow Model	FM objective $\rightarrow$ discrete consistency targets $\rightarrow$ full MF objective
ANCORA	Base model plus iterative self-distilled SFT	manifold projection $\rightarrow$ verifier-gated self-play $\rightarrow$ UCB-guided DAG expansion

2. Optimization pathologies that motivate anchoring

The DMF construction is motivated by an asymmetry between training stability and sampling efficiency in flow-based image generation. Flow Matching is characterized as stable and easy to train, but typically multi-step at sampling time. MeanFlow provides better one-step and few-step behavior, but training is expensive and unstable because the continuous-time identity depends on a derivative of the model output and therefore requires a Jacobian-vector product (JVP). MeanFlow models with strong one-step performance had also required extremely large training budgets (Hsu et al., 10 Apr 2026).

ANCORA is motivated by a different pathology: sparse-verifier RL. Let $\mathcal{Y}=\mathcal{V}^T$ be the space of token sequences, let $\mathcal{M}\subset \mathcal{Y}$ be the valid manifold, and define

$$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$

If $p_{\mathcal{M}}\ll 1/N$, then for a batch of size $N$, most groups contain zero valid samples, so binary-reward GRPO gives zero gradient. The appendix states

$$\Pr\!\left(\forall i\in[N]: y_i\notin \mathcal{M}\right)=(1-p_{\mathcal{M}})^N \approx e^{-Np_{\mathcal{M}}}\to 1.$$

The resulting failure modes are gradient starvation, sparse-sampling distortion, and AdamW momentum drift (Yang et al., 30 Apr 2026).

These two motivations are structurally analogous. In DMF, the hard regime is a derivative-dependent continuous objective; in ANCORA, it is a sparse-verifier RL signal. In both cases, the anchor is introduced because naive training begins in a regime where either targets or rewards are too weak, too costly, or too unstable to support reliable optimization. This suggests that base anchoring functions as an optimization preconditioner at the level of training dynamics rather than merely as an initialization heuristic.

3. Anchors, compatible starting points, and manifold support

The DMF paper formulates the probability flow ODE

$\rightarrow$0

and defines MeanFlow as predicting an average velocity over an interval $\rightarrow$1,

$\rightarrow$2

The MeanFlow identity is

$\rightarrow$3

with total derivative

$\rightarrow$4

The curriculum is initialized directly from a pretrained flow model,

$\rightarrow$5

and the first stage target is exactly the Flow Matching target,

$\rightarrow$6

This ensures that the model begins on a compatible solution manifold rather than from random weights (Hsu et al., 10 Apr 2026).

ANCORA uses a different anchor but an analogous logic. The base Qwen2.5-Coder-3B-Instruct model has base pass@1 of $\rightarrow$7 on Dafny2Verus, and direct RL from that point collapses. The training sequence is: start from small seed demos from basic Verus tasks, generate early Q-Gen/S-Gen traces, keep only Verus-verified traces, distill those traces back into the model, and repeat for a few early iterations. The empirical progression reported is base model $\rightarrow$8, SFT iter 1 $\rightarrow$9, SFT iter 2 $\rightarrow$0, SFT iter 3 $\rightarrow$1, and then ANCORA RL $\rightarrow$2. The appendix states that $\rightarrow$3 is needed so that most groups contain at least one valid sample, and the SFT phase is presented as the mechanism that makes this plausible (Yang et al., 30 Apr 2026).

In both systems, the anchor is not merely a warm start. It defines a region of parameter space in which the first-stage target is already meaningful. A plausible implication is that base anchoring stabilizes training by increasing overlap between the model’s support and the support of the supervision signal, whether that signal is a regression target or a binary verifier reward.

4. Curriculum mechanics and stagewise update rules

DMF derives a discrete approximation to the MeanFlow identity:

$\rightarrow$4

This exposes a consistency property: the target at $\rightarrow$5 is expressed in terms of the target at a nearby earlier point $\rightarrow$6. In practice, DMF trains $\rightarrow$7 against

$\rightarrow$8

where $\rightarrow$9 denotes stop gradient. The curriculum uses $\rightarrow$0 stages. For $\rightarrow$1, the preferred schedule defines $\rightarrow$2 and

$\rightarrow$3

The stage-$\rightarrow$4 target becomes

$\rightarrow$5

At the final stage, DMF returns to the full MeanFlow objective,

$\rightarrow$6

The training loss is a regression loss,

$\rightarrow$7

with an adaptive loss on CIFAR-10 and a Cauchy loss with robust value $\rightarrow$8 on ImageNet $\rightarrow$9 (Hsu et al., 10 Apr 2026).

ANCORA uses a unified policy $\mathcal{Y}=\mathcal{V}^T$0 in an alternating self-play loop. Given a seed $\mathcal{Y}=\mathcal{V}^T$1 from the curriculum DAG, the Proposer generates a new task $\mathcal{Y}=\mathcal{V}^T$2, the Solver samples a solution $\mathcal{Y}=\mathcal{V}^T$3, and a deterministic verifier returns $\mathcal{Y}=\mathcal{V}^T$4. The RL update is two-level. For a specification $\mathcal{Y}=\mathcal{V}^T$5 with $\mathcal{Y}=\mathcal{V}^T$6 solver rollouts and verifier outcomes $\mathcal{Y}=\mathcal{V}^T$7, the solver advantage is

$\mathcal{Y}=\mathcal{V}^T$8

For the Proposer, with $\mathcal{Y}=\mathcal{V}^T$9 and $\mathcal{M}\subset \mathcal{Y}$0, the paper defines entropy-targeting rewards

$\mathcal{M}\subset \mathcal{Y}$1

and the main run uses

$\mathcal{M}\subset \mathcal{Y}$2

The Proposer advantage is then

$\mathcal{M}\subset \mathcal{Y}$3

Seed selection is governed by a UCB-style index

$\mathcal{M}\subset \mathcal{Y}$4

with discounted statistics

$\mathcal{M}\subset \mathcal{Y}$5

Admission to the curriculum requires format and structural checks, compiler-stub syntax validation, semantic heuristics rejecting vacuous postconditions, MinHash novelty filtering, and solved-only verification. Every non-original node has therefore been solved at least once before admission (Yang et al., 30 Apr 2026).

The two mechanisms differ in implementation but share a common logic. DMF creates local recursive consistency along a trajectory in continuous-time generative dynamics; ANCORA creates verifier-gated local expansion from solved seeds in a curriculum DAG. In both cases, the curriculum tightens constraints only after the model has entered a regime where those constraints are informative.

5. Empirical behavior, compute tradeoffs, and failure modes

The reported empirical results show that base anchoring is associated not only with better final metrics but also with altered training dynamics and compute profiles.

Setting	Anchored curriculum result	Comparator or note
CIFAR-10, DMF$\mathcal{M}\subset \mathcal{Y}$6	one-step FID 3.36 in 2000 + 2000 epochs	MF from scratch: 4000 epochs, FID 3.85; MF fine-tuned from FM: FID 3.93
CIFAR-10, DMF compute	66.6 GPU hours	MF baseline: 85.33 GPU hours
ImageNet $\mathcal{M}\subset \mathcal{Y}$7, DMF$\mathcal{M}\subset \mathcal{Y}$8	one-step FID 21.18, 18.03, 16.95, 14.53 at 6, 12, 24, 48 epochs	At 96 epochs, FID collapses to 294.13
Dafny2Verus, ANCORA	81.5% pass@1	SFT baseline: 26.6%; PSV: 65.63% in TTT
MBPP, ANCORA TTT	44.11% pass@1	PSV: 36.78%
HumanEval, ANCORA TTT	19.15% pass@1	PSV: 19.07%

For DMF, avoiding JVP in most stages yields a reported per-batch speedup: on CIFAR-10, MF requires 0.38 sec/batch versus 0.32 sec/batch for DMF; on ImageNet $\mathcal{M}\subset \mathcal{Y}$9, MF requires 3.08 sec/batch versus 1.71 sec/batch for DMF, described as roughly $$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$0 to $$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$1 faster per batch. End-to-end on CIFAR-10, the curriculum is reported as about $$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$2 faster while improving one-step FID (Hsu et al., 10 Apr 2026).

For ANCORA, the curriculum effect is visible in training dynamics and ablations. Q-Gen validity stays around $$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$3 during training, the seed pool grows monotonically, and in the test-time-training setting it expands from 436 roots to 1,258 nodes by step 1500. All novel nodes were solved at least once before admission. Branch-freeze ablations indicate that the curriculum is genuinely coupled: Freeze-Q tracks early and then plateaus far below full ANCORA; Freeze-S causes the Proposer to drift into unsolvable or trivial specifications and collapse; No-Descent with $$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$4 initially performs well and then collapses once curated roots are exhausted. A separate ablation reports that empirical mean UCB collapses whereas ratcheted UCB remains stable (Yang et al., 30 Apr 2026).

These results also delimit the scope of the method. In DMF, overly fine discretization in latent space can become unstable, as reflected in the ImageNet divergence at 96 epochs. In ANCORA, the anchoring mechanisms are presented as load-bearing; without manifold projection, strict gating, and DAG-based seed control, sparse-verifier RL collapses.

6. Relation to adjacent methods and common misconceptions

Base-anchored training curriculum is not equivalent to generic pretraining. In DMF, the paper explicitly characterizes the method as more than a different objective: it is a training strategy that controls optimization difficulty. Standard MeanFlow is typically trained directly with the continuous MeanFlow identity, requires JVPs throughout training, and often uses a hybrid mixture of FM and MF losses, whereas DMF starts from a pretrained FM, uses a stage-wise discrete approximation, treats early stages as essentially FM, and only later transitions to full MF/JVP (Hsu et al., 10 Apr 2026).

It is also not equivalent to unconstrained self-play or simply “generate more data.” ANCORA states that its curriculum is a manifold-anchored curriculum that expands only through verified, novel, solved specifications. Heuristic scores and MinHash novelty drive only gating and deduplication, while the policy gradient is driven exclusively by binary verifier outcomes. This design distinguishes the method from proposer-solver-verifier baselines such as PSV, in which the proposer is not trained with RL in the same way and the solver uses rejection fine-tuning (Yang et al., 30 Apr 2026).

A further misconception is that the key ingredient is reward shape alone. ANCORA compares Band-1-of-$$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$5, Bernoulli variance $$p_{\mathcal{M}} \triangleq \sum_{y\in\mathcal{M}} \pi_\theta(y\mid x).$$6, and exponential decay proposer rewards, and reports that collapse can still occur without the manifold stabilizers. Similarly, DMF’s improvement is not attributed merely to replacing one regression target with another; the important point is the continuation path from pure Flow Matching, through discrete consistency-based approximations, to the full MeanFlow objective.

The relation to neighboring paradigms is nonetheless explicit. DMF’s discrete recursion is described as very similar in spirit to consistency training and self-distillation because predictions at adjacent points must agree after a velocity correction. ANCORA’s solver update is explicitly motivated by Maximum Likelihood Reinforcement Learning (MLRL), and the mean-normalized binary advantage is presented as matching the likelihood-aligned weighted-pass estimator up to a baseline. The broader implication is that base-anchored curricula integrate initialization, target design, and sampling policy into a single stabilization scheme. Within the cited work, the defining property is not the domain—image generation or verifiable reasoning—but the use of a base-compatible starting regime to make later, harder supervision usable.