AP2O: Adaptive Progressive Preference Optimization

• The paper introduces AP2O as an offline preference learning framework that leverages error notebooks and adaptive replay to systematically address failure modes in LLM outputs.
• It employs a progressive curriculum that orders and revisits error types, allowing for targeted improvements in syntax, runtime, and logical errors.
• Empirical results demonstrate up to a 3% pass@k performance gain with improved data efficiency and reduced catastrophic forgetting across various model scales.

Adaptively Progressive Preference Optimization (AP2O) is an offline preference learning framework for LLMs that structures post-training around diagnostically meaningful failure modes rather than undifferentiated pass/fail labels. In the coding setting, AP2O analyzes failed generations with an external verifier, organizes them into an “error notebook” indexed by error type, and then trains progressively on those types while adaptively replaying failures that remain prevalent under the current model (Zhang et al., 1 Oct 2025). A closely related precursor appears in the plug-and-play training framework for mathematical reasoning, which does not use the name AP2O but introduces adaptive, difficulty-aware weighting of preference pairs using multi-sample statistics; with re-computation and annealing, that framework maps naturally onto an AP2O-style regime (Ma et al., 2024).

1. Definition and conceptual scope

AP2O is defined in the code-generation literature as a human-inspired preference optimization method that “correct[s] errors type by type like humans” through two coupled mechanisms: progressive preference optimization and adaptive error replay (Zhang et al., 1 Oct 2025). The progressive component orders failures by error type and exposes the model to them according to a schedule, while the replay component periodically re-estimates the model’s current weaknesses on a validation “quiz” and shifts training emphasis accordingly.

The framework is situated within offline preference optimization. In standard preference optimization for LLMs, a policy $\pi_\theta$ is trained to prefer a chosen response $y_w$ over a rejected response $y_l$ for prompt $x$, often via a Bradley–Terry-style likelihood and, in DPO, a pairwise logistic loss relative to a frozen reference model $\pi_{\mathrm{ref}}$ (Ma et al., 2024). AP2O retains that pairwise preference backbone but changes how the rejected examples are organized and revisited.

A useful distinction within the literature is between explicit AP2O and AP2O-like adaptive weighting. The former is the named framework introduced for code generation in “AP2O: Correcting LLM-Generated Code Errors Type by Type Like Humans via Adaptive Progressive Preference Optimization” (Zhang et al., 1 Oct 2025). The latter is the difficulty-aware plug-and-play method for mathematical reasoning in “Plug-and-Play Training Framework for Preference Optimization,” where adaptive weighting is present but progressive scheduling is only an extension rather than part of the reported training loop (Ma et al., 2024). This suggests that AP2O is best understood not as a single loss, but as a family of preference-optimization procedures that combine adaptive diagnostics with staged emphasis.

2. Preference-optimization foundation

The mathematical core of AP2O inherits the standard DPO objective. In the formulation summarized for the math-oriented framework, the standard DPO loss is

$$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$

Here $x$ is a prompt, $y_w$ the preferred output, $y_l$ the dispreferred output, $\pi_\theta$ the trainable policy, $y_w$0 a fixed reference policy, and $y_w$1 a scaling parameter (Ma et al., 2024). The coding AP2O paper uses the same DPO sample loss in ratio form,

$y_w$2

with dataset loss

$y_w$3

(Zhang et al., 1 Oct 2025)

What AP2O changes is not the basic pairwise preference formalism, but the semantics of the preference pair and the schedule with which such pairs are presented. In ordinary DPO-style code training, unit-test outcomes yield binary labels: passing generations become chosen and failing generations become rejected. AP2O identifies three deficiencies in this setup for code error reduction: the model is unaware of deep-level error types, cannot focus on specific errors in a concentrated fashion, and receives a stale signal as its weaknesses change during training (Zhang et al., 1 Oct 2025).

The 2024 math framework addresses a related but distinct issue: current preference optimization methods do not account for varying difficulty levels of training samples, which degrades performance on tasks with high accuracy requirements, especially mathematical reasoning (Ma et al., 2024). That work therefore introduces per-prompt weighting inside the pairwise logistic objective. The weighted Bradley–Terry objective is

$y_w$4

and the corresponding weighted DPO objective is

$y_w$5

This establishes the main bridge between AP2O and earlier weighted preference optimization: both modify the effective training emphasis placed on different preference pairs, but AP2O does so through explicit error-typed curricula and adaptive replay, whereas the math framework does so through static difficulty-aware weights (Ma et al., 2024).

3. Error notebook and progressive optimization

The distinctive object in AP2O-Coder is the error notebook, built from LLM-generated failed code samples and analyzer outputs (Zhang et al., 1 Oct 2025). In the code-generation setting, the model self-generates $y_w$6 candidate solutions per programming prompt under exploration at temperature $y_w$7, and each candidate is executed against unit tests. Passing outputs form the pool of chosen responses; failing outputs are sent through a programming-language-specific analyzer $y_w$8, such as a Python interpreter or sandbox coupled with unit tests, to obtain an ErrorType tag (Zhang et al., 1 Oct 2025).

The error taxonomy reported in the paper includes SyntaxError, TypeError, KeyError, IndexError, OSError, ValueError, WrongResult, and other runtime exceptions (Zhang et al., 1 Oct 2025). Compiler or interpreter tracebacks determine syntax and runtime categories, while unit tests determine WrongResult and some mismatch-based failures. The resulting error-annotated dataset is organized by frequency $y_w$9 over error types $y_l$0.

The notebook construction pipeline is explicitly staged as: exam, analysis, aggregation, and representation. The model first self-generates answers and partitions them into pass/fail sets; the analyzer then extracts error types and messages; error types are sorted into an ordered list $y_l$1 by frequency, either ascending for L2H or descending for H2L; and each notebook entry stores a counter $y_l$2, example tuples, and per-problem subsets $y_l$3 identifying failed answers of type $y_l$4 for problem $y_l$5 (Zhang et al., 1 Oct 2025).

Progressive optimization then traverses the ordered error types. For H2L, the ordering satisfies $y_l$6, and training advances through a sliding window over the active subset of error types. The progressive AP2O objective is

$y_l$7

where $y_l$8 is the number of failed answers of type $y_l$9 for problem $x$0 (Zhang et al., 1 Oct 2025). The paper describes H2L as starting from high-frequency errors and shifting toward lower-frequency ones, while L2H reverses the order. It further notes that H2L favors specialization first, whereas L2H favors broad exposure first.

A concrete example in the paper illustrates the notebook semantics. For a prompt asking for mean(nums), a model output that computes sum(nums) / len(nums) and is then applied to an empty list may trigger ZeroDivisionError: division by zero. Depending on the test specification, the analyzer may record WrongResult or ZeroDivisionError, increment the corresponding frequency, store the failed code and message, and later use that failure as a rejected sample paired with passing code for DPO-style optimization (Zhang et al., 1 Oct 2025).

4. Adaptive replay and the relation to weighted AP2O-style training

The second AP2O module is adaptive replay, which periodically evaluates the current model on a held-out validation set, generates one answer per problem, analyzes the failures, and computes the current error-type ratios

$x$1

Training then replays rejected examples sampled from the training pool according to $x$2, thereby emphasizing the error types on which the current model is presently weak (Zhang et al., 1 Oct 2025). The mixture of progressive and replayed pairs is balanced by count in the reported implementation, effectively corresponding to $x$3 in a general mixture objective.

The replay-augmented objective is written as

$x$4

with replay samples drawn according to the validation error ratios (Zhang et al., 1 Oct 2025). In the authors’ interpretation, this mechanism mitigates catastrophic forgetting because the training set is not used in a purely static way.

The 2024 math framework provides a complementary perspective on adaptive preference optimization. There, the model samples $x$5 outputs per prompt $x$6 at temperature $x$7, extracts final answers, counts correct responses $x$8 and incorrect responses $x$9, and builds a difficulty-aware weight

$\pi_{\mathrm{ref}}$0

The pair is formed by selecting $\pi_{\mathrm{ref}}$1 as a correct sampled response if one exists, otherwise the gold solution, and $\pi_{\mathrm{ref}}$2 as a frequent wrong response, typically from the wrong answer class with highest frequency among the $\pi_{\mathrm{ref}}$3 samples (Ma et al., 2024). This creates adaptive, pair-level weights that emphasize problems the model systematically gets wrong or handles uncertainly.

The paper is explicit that these weights are computed once in a pre-training sampling pass and then kept fixed during training. There is no explicit annealing schedule and no re-computation of $\pi_{\mathrm{ref}}$4 over optimization steps (Ma et al., 2024). However, the same source also states that AP2O can be realized explicitly by re-computing difficulty statistics every $\pi_{\mathrm{ref}}$5 steps or each epoch and annealing the influence of weights over time, for example through

$\pi_{\mathrm{ref}}$6

or via a progressively increasing $\pi_{\mathrm{ref}}$7 in the logistic margin (Ma et al., 2024). This suggests that AP2O, in a broader sense, can be instantiated either by discrete error-type curricula with replay, as in coding, or by dynamic difficulty-aware reweighting, as in mathematical reasoning.

5. Empirical performance

The coding AP2O paper reports experiments on EvalPlus (HumanEval and MBPP) and LiveCodeBench v6, using pass@$\pi_{\mathrm{ref}}$8 for $\pi_{\mathrm{ref}}$9 and sample efficiency as primary metrics (Zhang et al., 1 Oct 2025). The pass@$$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$0 estimator is given as

$$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$1

aggregated over tasks by averaging (Zhang et al., 1 Oct 2025). The training/validation sources are MBPP (384/90) and TACO (1678/420), and problems with fewer than two failed answers after self-generation are filtered out (Zhang et al., 1 Oct 2025).

Across code LLMs and general LLMs adapted to code, AP2O is reported to improve performance by up to $$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$2 in pass@$$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$3 while using less preference data, with models ranging from $$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$4B to $$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$5B parameters (Zhang et al., 1 Oct 2025). On HumanEval pass@1, selected results include CodeLlama-34B improving from 46.2 to 49.6 under AP2O-H2L, DeepSeek-Coder-33B improving from 78.4 to 80.1, and Qwen2.5-Coder-32B improving from 91.5 to 92.2, while DPO is reported at 91.0 for that model (Zhang et al., 1 Oct 2025). On MBPP pass@1 for Qwen2.5-Coder, AP2O-H2L improves 7B performance from DPO 83.5 to 85.4, AP2O-L2H improves 3B performance from 76.0 to 77.5, and the 0.5B model improves from DPO 51.9 to 56.7 under AP2O-L2H (Zhang et al., 1 Oct 2025). On LiveCodeBench v6 pass@1, AP2O-H2L improves Qwen2.5-Coder-7B from DPO 18.4 to 19.0 and Qwen2.5-Coder-3B from 14.8 to 15.2, while the 0.5B model improves from 2.9 to 3.3 under AP2O-L2H (Zhang et al., 1 Oct 2025).

The same work emphasizes data efficiency: AP2O requires only 4–60% of the preference pairs used by DPO to reach optimal results, with H2L generally more data-efficient than L2H because high-frequency errors have broader impact (Zhang et al., 1 Oct 2025). Larger-$$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$6 results on DeepSeek-Coder indicate that AP2O-H2L preserves or improves pass@5 and pass@10, whereas curriculum DPO variants may degrade there; the paper interprets this as evidence that AP2O mitigates catastrophic forgetting and maintains diversity (Zhang et al., 1 Oct 2025).

In mathematical reasoning, the plug-and-play weighted framework is evaluated on GSM8K and MATH500, with training data drawn from MetaMath and weights computed from $$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$7 samples at temperature $$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$8 and $$\mathcal{L}_{\text{DPO}(\theta)} \;=\; -\,\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\!\left( \beta \Big( \big[\log \pi_\theta(y_w\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_w\!\mid\!x)\big] - \big[\log \pi_\theta(y_l\!\mid\!x)-\log \pi_{\mathrm{ref}}(y_l\!\mid\!x)\big] \Big) \right) \right].$$9 (Ma et al., 2024). Models include Qwen2-1.5B-Instruct, Qwen2-7B-Instruct, GLM4-9B-Chat, and ChatGLM3-6B, trained with PEFT via LoRA in bf16 on A100 40G hardware (Ma et al., 2024). Headline results on MATH500 include Qwen2-7B-Instruct with DPO improving from 55.8 to 57.6 under weighting, SimPO from 52.6 to 53.4, IPO from 52.0 to 54.0, and DPOP remaining at 53.2; for Qwen2-1.5B-Instruct, DPO improves from 24.8 to 25.6 and DPOP from 24.4 to 26.0, while IPO decreases from 25.8 to 25.2 (Ma et al., 2024). GLM4-9B-Chat shows DPO improving from 46.2 to 47.2 and IPO from 46.6 to 47.2, while some SimPO and DPOP settings show small or no gains (Ma et al., 2024).

These results are not AP2O in the strict named sense, but they are directly relevant to AP2O’s adaptive principle. The paper further reports that weighted DPO on Qwen2-7B yields higher major@$x$0 on MATH500 across $x$1, upward movement in unique-answer-versus-correct-ratio distribution plots, and improved reward/probability separation between chosen and rejected responses during training (Ma et al., 2024). A plausible implication is that adaptive curricula and adaptive weighting act on related signal-quality bottlenecks: both attempt to concentrate optimization on failures that are informative rather than already solved.

6. Interpretations, practical constraints, and limitations

A central interpretive claim in the coding paper is that AP2O improves code generation because progressive optimization isolates specific failure modes—syntax, type, runtime, and WrongResult—while adaptive replay retargets learning toward whatever still fails on the current model (Zhang et al., 1 Oct 2025). The paper further attributes AP2O’s preservation of larger-$x$2 performance to reduced catastrophic forgetting and more efficient use of LLM-generated preference data (Zhang et al., 1 Oct 2025).

The error notebook also increases interpretability. Because each rejected sample is linked to an explicit analyzer-derived type and message, training focus is inspectable at the level of error categories rather than only aggregate rewards or binary test outcomes (Zhang et al., 1 Oct 2025). In the math framework, a parallel interpretability benefit arises from the transparent weight function based on $x$3, $x$4, and the concentration of wrong answers (Ma et al., 2024). There, the authors also analyze the number of unique answers $x$5 among $x$6 samples and the theoretical maximum accuracy bound

$x$7

as well as major@$x$8 and an unbiased pass@$x$9 estimator over sampled responses (Ma et al., 2024).

Both frameworks, however, carry substantial assumptions. In AP2O-Coder, reliable learning depends on analyzer quality, unit-test coverage, and the granularity of the error taxonomy. Weak tests can miss or mislabel failures, WrongResult may collapse heterogeneous logical bugs into a single category, and schedule choice matters: L2H and H2L behave differently depending on model maturity, with some transient increases in particular error types before replay corrects them (Zhang et al., 1 Oct 2025). In the math framework, the method presumes an automatic way to group final answers into correctness-equivalence classes, which is straightforward for math but harder for free-form tasks such as translation or reading comprehension (Ma et al., 2024). The paper explicitly notes that semantic clustering or semantic entropy may help, but those directions are outside its experiments.

Computational overhead is another shared concern. AP2O requires self-generation, testing, and error analysis before training, together with periodic validation “quizzes” for replay (Zhang et al., 1 Oct 2025). The math framework similarly incurs per-prompt multi-sampling cost during training-data construction, although the authors note that sampling can be cached offline and reused (Ma et al., 2024). Both approaches are therefore best suited to settings where verifiable signals are available and the preprocessing expense is acceptable.

The literature also cautions against a narrow reading of AP2O as universally superior to static PO. In the math experiments, some model-method combinations show little gain or even degradation, such as IPO on Qwen2-1.5B-Instruct and ChatGLM3-6B (Ma et al., 2024). In code, AP2O’s choice between H2L and L2H depends on model maturity: H2L is described as especially effective on mature models for high-frequency error types, whereas L2H can better support immature code capability by broader early exposure (Zhang et al., 1 Oct 2025). This indicates that “adaptive” and “progressive” do not eliminate hyperparameter sensitivity; they relocate it into schedule design, replay balance, and diagnostic taxonomy.

7. Broader significance and extensions

AP2O occupies a point of contact between preference optimization, curriculum learning, and dynamic weighting. In the coding formulation, progression occurs over an ordered set of explicit error types, and adaptivity is realized through replay driven by validation-time error ratios (Zhang et al., 1 Oct 2025). In the mathematical formulation, adaptivity is encoded by a per-prompt weight derived from multi-sample correctness statistics, while progression appears only as an extension through re-computation and annealing of $y_w$0 or $y_w$1 (Ma et al., 2024). This suggests that AP2O is less a single algorithmic template than a design principle for preference optimization under verifiable feedback.

The coding paper explicitly argues that AP2O is readily applicable to other verifiable domains, giving theorem proving with proof checkers and data transformations with validators as examples (Zhang et al., 1 Oct 2025). The math paper likewise suggests extensions using entropy over answer classes, variance of sampled scores, or softmax-normalized difficulty scores, although these are not used in the reported experiments (Ma et al., 2024). A plausible implication is that the key portability criterion is not the task domain itself, but the availability of an analyzer that can map failures into stable, training-relevant equivalence classes.

Within current arXiv literature, AP2O therefore denotes both a specific coding method and a broader trajectory in post-training research: moving from static pairwise preference datasets toward training loops that diagnose errors, order them, and refocus optimization as the model changes. The named AP2O framework in code generation provides the clearest explicit instantiation of that trajectory (Zhang et al., 1 Oct 2025), while the earlier plug-and-play weighting framework demonstrates that analogous adaptive mechanisms can already yield measurable gains in mathematical reasoning, even before explicit progression is introduced (Ma et al., 2024).