MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization

Abstract: Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.

• The paper introduces MASS-DPO, a greedy D-optimal design method that selects complementary negative responses by maximizing Fisher-information coverage rather than relying on random or heuristic sampling.
• The paper shows that using only three to five actively selected negatives can match or exceed larger negative pools, with gains across recommendation and question-answering benchmarks and up to 1.85× lower per-epoch training cost.
• The paper provides theoretical error bounds that decrease with the selected-negative budget and finds stable selections across training, while noting that its guarantees rely on log-linear policy and Fisher-compatibility assumptions.

Motivation and problem setting

Multi-negative preference optimization extends Direct Preference Optimization (DPO) from pairwise Bradley–Terry comparisons to a Plackett–Luce (PL) ranking objective, in which one preferred response is trained against multiple rejected candidates. Existing multi-negative methods such as Softmax-DPO (S-DPO) and Direct Multi-Preference Optimization (DMPO) select or weight negatives randomly or heuristically. In large candidate pools this wastes training signal: near-duplicate negatives produce gradients pointing in similar directions, so additional computation yields diminishing returns in policy updates. The paper's central observation is that the composition of the negative set, not merely its size, determines how informative each gradient step is.

MASS-DPO addresses this by formulating negative selection as a D-optimal experimental design problem. Rather than selecting which prompts or comparisons to acquire — as prior active-learning work for preference optimization does at the instance level — MASS-DPO applies optimal design within each prompt: given a fixed pool of candidate negatives for one preferred response, it selects a compact subset whose gradients span complementary directions in parameter space.

Method

The method rests on a curvature analysis of the multi-negative PL loss under a log-linear policy assumption $$\pi(y\mid x;\theta)\propto\exp(\phi(x,y)^\top\theta)$$. Writing the feature difference and reference offset for each negative relative to the preferred response as $\phi_i=\phi(x,y_i)-\phi(x,y^*)$ and $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$, the multi-negative loss takes the compact form $$L(\theta;S_n)=-\log\sigma(-\log\sum_{i\in S_n}\exp[\beta(\phi_i^\top\theta+b_i)])$$. Two lemmas characterize its gradient and Hessian:

• Gradient structure: the gradient is a softmax-weighted combination of feature differences scaled by the misranking probability $(1-\sigma(Z_n))$, so subset weights emphasize borderline, hard-to-rank negatives.
• Hessian lower bound: the Hessian dominates (in the Loewner order) a weighted covariance of centered feature differences. Subsets whose feature differences spread along orthogonal directions therefore maximize information volume.

This motivates maximizing $\log\det H(S)$ of the regularized Fisher information matrix $$H(S)=\gamma I+\alpha_0\sum_{j\in S}v_j^0(v_j^0)^\top$$, where $v_j^0=\sqrt{q_j^0}(\phi_j-\bar\phi_0)$ are full-pool-centered, softmax-weighted features evaluated once at a frozen preprocessing checkpoint $\theta_0$. The authors justify D-optimality over A- or E-optimality because maximizing log-determinant minimizes the volume of the joint confidence ellipsoid, promoting coverage across parameter directions rather than emphasizing a single mode.

The exact combinatorial problem is NP-hard, so the paper uses greedy incremental construction with rank-one updates: at each step it selects the negative maximizing the marginal log-determinant gain, which by the matrix determinant lemma reduces to maximizing the $H_{k-1}$-induced squared norm $\phi_i=\phi(x,y_i)-\phi(x,y^*)$0. Sherman–Morrison updates cost $\phi_i=\phi(x,y_i)-\phi(x,y^*)$1 per selection; scoring all remaining candidates costs $\phi_i=\phi(x,y_i)-\phi(x,y^*)$2 per step. Selection is performed once as preprocessing and the subsets remain fixed during training, so there is no per-step mining overhead. Empirically, this fixed-selection design is well justified: recomputing selections at initialization, mid-training, and the final checkpoint on MovieLens yields at least 98.8% exact overlap and 0.997 mean Jaccard similarity across checkpoints, indicating the Fisher geometry is stable over the training horizon.

Theoretical guarantees

Two results connect subset selection to estimation quality. The first bounds the relative logit error — the worst-case distortion of pairwise candidate-margin rankings when using a subset-trained estimator instead of the full-pool optimum:

$\phi_i=\phi(x,y_i)-\phi(x,y^*)$3

with probability at least $\phi_i=\phi(x,y_i)-\phi(x,y^*)$4, decaying as $\phi_i=\phi(x,y_i)-\phi(x,y^*)$5 in the selected-negative budget. The proof combines self-normalized concentration for generalized linear models with a centered leverage-score decay bound showing that greedy D-optimal selection drives leverage scores down at rate $\phi_i=\phi(x,y_i)-\phi(x,y^*)$6. The second result gives a batch-level estimation error bound in the $\phi_i=\phi(x,y_i)-\phi(x,y^*)$7-norm over $\phi_i=\phi(x,y_i)-\phi(x,y^*)$8 i.i.d.-sampled prompts, implying stable generalization across prompts.

These guarantees depend on several assumptions that should be weighed when interpreting them: bounded feature differences and offsets, bounded curvature scale on the relevant parameter region, a diversity condition on the candidate pool (a constant $\phi_i=\phi(x,y_i)-\phi(x,y^*)$9 bounding how much any single candidate exceeds the best remaining marginal gain), and a Fisher-compatibility condition requiring the regularized subset Hessian to dominate the selection objective up to a constant $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$0. The log-linear policy model used for the selection criterion is itself an approximation to the actual language-model policy; the theory applies to the log-linear surrogate, not directly to the fine-tuned transformer.

Experimental results

Experiments cover four benchmarks — LastFM and MovieLens (recommendation), MedMCQA and QASC (multiple-choice QA) — across three backbone families (Qwen3-4B, SmolLM3, Llama-3.2-3B-Instruct), with all multi-negative methods using $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$1 selected negatives during training and evaluation against up to 20 candidates.

Accuracy. MASS-DPO achieves the highest average accuracy on Qwen3 (57.18% vs. 55.80% for the next-best DPO-k) and SmolLM3 (56.78%), and remains competitive on Llama3 (62.99% vs. 63.23% for S-DPO). Baselines without principled negative selection (DPO, DMPO, DPO-k) generally underperform; DMPO in particular degrades substantially on some settings (e.g., 25.70% on MedMCQA with Llama3 versus 71.29% for MASS-DPO), confirming that negative composition materially affects outcomes.

Ranking quality. On Recall/NDCG at $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$2, MASS-DPO attains best or tied-best scores on most cells across both recommendation datasets. It also improves MRR over S-DPO on most configurations (e.g., average MRR of 70.97 vs. 69.26 on Qwen3).

Optimization dynamics. Compared against S-DPO, which differs only in using random softmax-weighted negatives, MASS-DPO exhibits larger margins, faster early accuracy gains, higher plateaus, and smoother chosen-reward trajectories throughout training. This isolates the contribution of the D-optimal selection criterion itself.

Compute efficiency. At matched wall-clock budget on MovieLens with Llama-3.2-3B (~11h vs. ~10.2h), MASS-DPO with $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$3 matches S-DPO trained on all 19 negatives in accuracy while outperforming it on R@3 (0.810 vs. 0.781), NDCG@5 (0.751 vs. 0.740), MRR (0.725 vs. 0.718), and margin (5.049 vs. 3.904). Full-pool training is 1.85× slower per epoch than MASS-DPO-5, while the entire selection procedure costs only 45.5 minutes one-time, of which 42 minutes is feature/log-probability extraction and only 3.5 minutes is the core rank-one selection algorithm. This supports the claim that actively selected negatives provide more useful signal per gradient step than the full pool.

Ablations. Sweeping $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$4 shows $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$5 consistently performs best, with performance degrading monotonically as $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$6 increases. Varying the negative budget $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$7 shows monotonic gains from 1 → 3 → 5 across models and datasets, consistent with the D-optimal prediction that added negatives improve parameter estimation until information coverage saturates.

Limitations and open questions

The paper is explicit about scope: evaluation is restricted to settings with finite per-prompt candidate pools where negatives are well-defined. Extending MASS-DPO to open-ended generation or instruction tuning would require an upstream pool-construction step (e.g., sampling negatives from the reference policy), which the authors do not evaluate. Several further caveats bear on the results. The theoretical bounds hold under the log-linear policy approximation and require the Fisher-compatibility and estimator-stability assumptions; whether these conditions hold for real transformer policies is not established. The selection criterion is evaluated at a frozen preprocessing checkpoint $$b_i=\log\pi_{\mathrm{ref}}(y^*\mid x)-\log\pi_{\mathrm{ref}}(y_i\mid x)$$8; the stability experiment suggests re-selection would be redundant within these training horizons, but behavior under much longer training or larger distribution shift is untested. Finally, the empirical advantage over S-DPO, while consistent, is modest on some configurations (e.g., Llama3 average accuracy trails S-DPO slightly), leaving open the question of when the Fisher-based criterion's benefit outweighs its preprocessing cost in low-pool-size regimes.

Conclusion

MASS-DPO contributes a theoretically grounded alternative to heuristic negative selection in multi-negative preference optimization: a PL-specific Fisher-information objective solved greedily via efficient rank-one log-determinant updates, applied within each prompt rather than across instances. Finite-sample bounds relate subset size to relative-logit error, and experiments across three model families and four recommendation/QA benchmarks show that compact actively selected subsets match or exceed full-pool training at reduced compute, with more stable optimization dynamics and stronger ranking metrics.