CoAct: Co-Active LLM Preference Learning with Human-AI Synergy

Abstract: Learning from preference-based feedback has become an effective approach for aligning LLMs across diverse tasks. However, high-quality human-annotated preference data remains expensive and scarce. Existing methods address this challenge through either self-rewarding, which scales by using purely AI-generated labels but risks unreliability, or active learning, which ensures quality through oracle annotation but cannot fully leverage unlabeled data. In this paper, we present CoAct, a novel framework that synergistically combines self-rewarding and active learning through strategic human-AI collaboration. CoAct leverages self-consistency to identify both reliable self-labeled data and samples that require oracle verification. Additionally, oracle feedback guides the model to generate new instructions within its solvable capability. Evaluated on three reasoning benchmarks across two model families, CoAct achieves average improvements of +13.25% on GSM8K, +8.19% on MATH, and +13.16% on WebInstruct, consistently outperforming all baselines.

• The paper presents CoAct's hybrid human-AI preference learning framework that integrates self-labeling and targeted oracle annotation for robust LLM alignment.
• It employs self-consistency and a k-NN-based OOD metric to optimize selection, yielding improvements up to +20% on key reasoning benchmarks.
• The framework demonstrates strong out-of-domain generalization and offers a scalable blueprint for effective mixed-supervision data curation.

CoAct: Co-Active LLM Preference Learning with Human-AI Synergy

Introduction

Preference-based alignment of LLMs is critically bottlenecked by the availability and quality of human-annotated preference data. Two major paradigms have been explored in response to this: self-rewarding pipelines, which exploit LLMs to self-label preference data at scale but risk error amplification due to self-bias, and active learning, which utilizes human (oracle) judgements for high-quality preference annotation but is constrained by annotation budgets and underutilizes the vast pool of unlabeled samples. The CoAct framework ("CoAct: Co-Active LLM Preference Learning with Human-AI Synergy" (2604.17501)) addresses this central dilemma by formalizing a co-active preference learning regime that roots its supervision in both systematic human-AI data curation and adaptive division of annotation labor.

Figure 1: Schematic contrasting (a) self-rewarding (AI-only) preference construction, (b) active preference learning with strict human annotation, and (c) CoAct’s human-AI collaborative pipeline.

Co-Active Preference Learning Framework

Rather than a simple hybrid, CoAct proposes a structured, iterative framework combining self-consistency-based self-labeling for AI-generated preference pairs and strategic selection for high-value oracle labeling. At each iteration, for a batch of unlabeled instructions, multiple diverse responses are sampled per instruction. Preference pairs are then constructed according to a self-consistency criterion: responses that dominate the sample distribution are marked as positive examples, while those with minimal support are marked negative. The consistency of the positive response stratifies each pair into high- or low-confidence subsets.

For highly consistent samples, a k-NN-based OOD metric is employed to detect potentially high-confidence erroneous regions—these are routed for oracle annotation. The low-consistency set, presumed less reliable, is also partially sampled for human review. Unique to CoAct, the oracle (human or frontier LLM) feedback is then exploited not only for direct correction, but as an anchor for controlling question augmentation: verified high-quality pairs are recycled as in-context exemplars to synthesize new, solvable instructions. This, in turn, refines the distribution of the training set to match model capabilities and furthers sample diversity.

Figure 2: High-level overview of CoAct’s core loop, highlighting self-consistency construction, dual-path annotation assignment, and oracle-guided instruction augmentation.

Supervised training occurs with a modified DPO objective that fuses supervised pairwise preference maximization with NLL-based regularization, balancing policy-shaping and generative likelihoods for improved robustness.

Empirical Evaluation

Comprehensive empirical validation is conducted across three challenging reasoning benchmarks (GSM8K, MATH, WebInstruct; covering arithmetic, mathematical competition, and physics reasoning) and two contemporary model architectures (Llama3-8B and Qwen3-4B). At each iteration, comparative baselines include random, entropy-based, and preference-certainty–driven active learning selection, as well as combinations thereof.

Key empirical findings:

• Superior Iterative Gains: CoAct consistently achieves the highest gains by the fourth iteration, yielding improvements up to +20.05% (GSM8K, Llama3-8B), +14.46% (MATH), and +8.28% (WebInstruct), outperforming the strongest baseline by 4–8 percentage points.
• Model Strength Effects: With stronger initial models (e.g., Qwen3-4B), the marginal benefit of sophisticated selection and self-labeling is diminished: all methods converge, but CoAct persists at the front.
• Criticality of Oracle/AI Data Fusion: Random-only pipelines suffer from performance degradation, confirming theoretical results that effective integration of (possibly noisy) AI-labeled data with trustworthy human supervision provides statistically superior alignment rates (see Theorem in main text).

Figure 3: (Left) Progression in average majority-vote share (self-consistency) across iterations; (Right) Oracle quality analysis (ground-truth vs. LLM oracle) showing robust annotation reliability.

Analysis and Interpretation

Mechanism Ablations

Ablations confirm that both question augmentation and self-consistency–driven self-labeling are vital: removing either substantially degrades learning, while question augmentation via oracle-guided bootstrapping enables broader domain generalization, especially on harder datasets. The k-NN OOD detection mechanism in high-consistency subsets is particularly effective at surfacing self-consistent but incorrect predictions, further increasing net annotation value per oracle interaction.

Consistency as Quality Signal

Self-consistency is empirically strongly correlated with true answer accuracy, as measured by Pearson $\geq$ 0.95 after four iterations across all datasets. Thus, selecting pairs by self-consistency is a data-driven proxy for identifying reliable self-labeled data.

Oracle Choice and Annotation

Results demonstrate that high-quality LLMs (GPT-5) are nearly indistinguishable from human annotators for oracle roles in this context. This facilitates efficient scaling of preference pipelines without significant quality compromise, assuming robust LLM evaluators and ground-truth answer availability.

Figure 4: Sensitivity analysis and ablations. (Left) Incorrect rate of different high-consistency selection strategies; (Middle) Impact of consistency threshold $\tau$; (Right) Performance deltas for ablated modules.

Out-of-Domain Generalization

The robustness of the CoAct pipeline is affirmed by strong out-of-domain generalization to benchmarks such as GPQA and MMLU-Pro, with CoAct models exhibiting consistently higher transfer accuracy relative to all baselines, indicating the beneficial effect of oracle-guided data augmentation and OOD-aware selection.

Figure 5: CoAct’s models yield superior zero-shot generalization on diverse domains (GPQA and MMLU-Pro).

Theoretical Underpinnings

CoAct’s design is supported by a formal generalization bound (Theorem in section 4), proving that when the noise rate of AI-generated preferences is below 50%, inclusion of such pairs—along with clean oracle pairs—increases effective Fisher information and strictly tightens policy sub-optimality bounds compared to oracle-only regimes. This formalizes the statistical wisdom of mixed supervision for preference alignment.

Practical and Theoretical Implications

Practically, CoAct offers a blueprint for scalable preference-alignment data curation in resource-constrained settings by leveraging both efficient AI-label scaling and high-leverage human (or strong LLM) interventions. Theoretically, it prescribes a mechanism for automatic error detection in self-labeled data and demonstrates the feasibility of learning curves surpassing oracle-only strategies under broad conditions.

Potential extensions include adapting the framework for other domains beyond reasoning, exploring additional OOD detection functions (e.g., feature geometric or probabilistic approaches), and further scaling active human-in-the-loop strategies as annotator cost drops with improved LLM oracles.

Conclusion

CoAct operationalizes a synergistic protocol for LLM preference alignment, integrating active human-AI co-supervision, self-consistency–rooted pseudo-labeling, and targeted oracle annotation augmented by question synthesis. Empirical and theoretical results demonstrate its superiority over prevalent baselines, with robust performance across iterations, datasets, and architectures, as well as in out-of-domain transfer. CoAct substantially advances sample-efficient LLM alignment methodology in mixed-supervision settings.