Autonomous Preference Alignment via Self-Injection

• APASI is a framework that autonomously curates and refines preference data using self-generated signal pairs, reducing reliance on static human annotations.
• It employs iterative, curriculum-guided learning by injecting controlled errors to mitigate hallucinations and enhance model alignment in LVLMs and LLMs.
• Empirical evaluations show APASI improves scalability, cost-effectiveness, and generalization across diverse benchmarks compared to traditional methods.

Autonomous Preference Alignment via Self-Injection (APASI) is a framework for aligning model outputs to human preferences by autonomously generating and iteratively refining preference signals within the model itself, rather than relying on external annotation or reward models. APASI is designed to overcome persistent challenges in model alignment, including distributional shifts, hallucination mitigation, adaptable supervision, and cost scalability. Recent work (Lu et al., 14 Sep 2025) formally introduces APASI for large vision-LLMs, but a broader research ecosystem exists, with adversarial, off-policy, trajectory-level, and sparse feature steering methods converging on foundational principles of autonomous self-injected preference alignment.

1. Motivation and Historical Context

The classic challenge in alignment methods—such as RLHF and DPO—is that preference supervision traditionally comes from static, human-annotated pairwise datasets. As LLMs and LVLMs evolve, the distribution of model outputs drifts away from the supervised data, causing reward model miscalibration and necessitating recurring annotation cycles (Cheng et al., 2023). This dependency substantially increases cost and impedes adaptation to new tasks or domains.

APASI emerges from the observation that models can construct their own preference labels or pairs by exploiting domain-specific phenomena (e.g., hallucination patterns in LVLMs (Lu et al., 14 Sep 2025)), internal logit-based preference judgments (Kim et al., 6 Jun 2024), or a self-improver mechanism (Lee et al., 27 Jul 2025), thus closing the alignment loop autonomously without human intervention or external reward models.

2. Core Mechanisms of Self-Injection

APASI operationalizes self-injection through the following mechanism (Lu et al., 14 Sep 2025):

• The model generates a preferred response to a given input (e.g., an image and prompt for LVLMs).
• It then deliberately constructs a dis-preferred response by injecting error—such as simulating hallucinations using empirical observations (object co-occurrence statistics, language priors, and positional factors).
• The paired responses (preferred/dis-preferred) serve as input for a direct preference optimization routine (e.g., DPO), typically expressed as:

$$\max_\theta \ \mathbb{E}_{v,x,y^+,y^-} \left[ \log \sigma \left( \beta \left( \log \frac{p_\theta(y^+|v,x)}{p_{\text{ref}}(y^+|v,x)} - \log \frac{p_\theta(y^-|v,x)}{p_{\text{ref}}(y^-|v,x)} \right) \right) \right]$$

• The process is iterated, using the updated (aligned) model in subsequent self-injection cycles and decreasing the injection rate (curriculum learning), thus maintaining stable learning and increasing challenge as the model’s hallucination propensity diminishes.

Several related frameworks adapt the self-injection mechanism for different modalities and tasks: adversarial preference optimization (Cheng et al., 2023), segment-level off-policy self-play (Yin et al., 31 May 2024), trajectory-wise reward synthesis in robotics (Zhang et al., 28 Nov 2024), and self-improver-based policy optimization (Lee et al., 27 Jul 2025). All operate on the principle of autonomously curating preference data from internal or simulation-based mechanisms.

3. Curriculum and Iterative Alignment

A defining design of APASI is iterative alignment coupled with curriculum adaptation (Lu et al., 14 Sep 2025). Rather than a single fine-tuning pass, each generation of the model spawns a new round of self-injection, wherein the difficulty of distinguishing preferred vs. dis-preferred outputs is gradually increased. For hallucination mitigation, the injection rate $\rho$—the proportion of the response replaced or corrupted—is decayed over iterations by a curriculum function $f_c(t)$. Early training features pronounced injected errors, and later stages induce more subtle hallucinations, forcing the model to fine-tune discrimination and robustness.

This curriculum design is significant for stable and continuous improvement, preventing the model from overfitting on superficial or easily-detectable errors and promoting generalization across tasks and manifestations of alignment drift.

4. Empirical Observations and Data Construction

APASI leverages empirical characteristics of misalignment to drive effective self-injection:

• Object Co-occurrence: In LVLMs, hallucinations tend to involve objects statistically likely to co-occur with those in the image. APASI constructs a co-occurrence graph and injects hallucinations based on these statistics, thus mimicking realistic failure modes (Lu et al., 14 Sep 2025).
• Language Priors: LVLMs over-extend from LLMs, producing plausible but visually unsupported content. APASI uses language-only guidance to simulate this effect in constructing dis-preferred responses.
• Positional Factor: Hallucinations are more prevalent near the response end. APASI preferentially corrupts later sentences through weighted sampling, aligning synthetic errors with observed patterns.

The artificial preference pairs thus generated form the backbone of the APASI alignment dataset. This strategy is adaptable to other domains; for instance, in LLMs, self-improving mechanisms produce pairs by refining current outputs with reference signals, ensuring the improved response is learnable and on-policy (Lee et al., 27 Jul 2025).

5. Comparative Evaluation and Generalization

Across extensive benchmarks—Object-Hal, AMBER, MMBench, MMVet, LLaVA-Bench—APASI reduces hallucination ratios and achieves performance comparable or superior to external alignment-based methods (including those that use reward models or human annotation) (Lu et al., 14 Sep 2025). The method is generalizable: it effectively adapts with no architecture-specific constraints and demonstrates success on multiple vision-language and language-only models (LLaVA-v1.6-7B, Qwen2-VL-7B).

Crucially, improvements are not strictly limited to hallucination mitigation but extend to comprehension and reasoning ability. This indicates that the quality and fidelity of self-injected preference data are sufficient to drive general improvements in alignment, a key advantage for sustainable model adaptation.

6. Broader Frameworks: Adversarial and Self-Play Extensions

Adversarial Preference Optimization (APO) (Cheng et al., 2023) recasts the alignment process as a min-max game where the reward model and policy alternate in adaptation, using self-generated comparison pairs to bridge distribution shifts without requiring external annotation. APO frameworks thus support an APASI-like loop, where both components dynamically adapt as the model evolves.

Similarly, off-policy self-play strategies (Yin et al., 31 May 2024) autonomously construct negative examples from the model’s own outputs, enabling segment-wise and list-wise preference learning in the absence of fixed ground truth. These methods are robust to data drift and scale efficiently, underpinning the core principles of autonomous self-injection.

7. Practical Implications and Future Directions

APASI and its related frameworks present compelling opportunities for cost-effective, scalable model alignment:

• Annotation Efficiency: Complete elimination of external annotation cycles substantially reduces resource requirements and enables sustainable deployment and improvement.
• Model Generalization and Robustness: Empirically validated gains across diverse benchmarks and architectures indicate the broad utility of self-injection mechanisms in both LVLMs and LLMs.
• Stable Continuous Adaptation: Iterative, curriculum-guided alignment fosters stable learning and adaptation to evolving data and objectives.

Open directions include extending APASI-style alignment to multi-objective settings (as in Pareto self-improvement frameworks (Li et al., 20 Feb 2025)), integrating trajectory-level reward synthesis in embodied and control systems (Zhang et al., 28 Nov 2024), and incorporating mechanistic interpretability diagnostics (e.g., sparse feature steering (Ferrao et al., 16 Sep 2025)) to better target explicit alignment concepts beyond stylistic proxies.

---

In summary, Autonomous Preference Alignment via Self-Injection (APASI) establishes a paradigm where models autonomously curate and refine preference information through self-generated data, using empirical insights into failure modes and iterative curriculum-based training. This enables sustainable, generalizable, and cost-efficient alignment improvements for large-scale language and vision-LLMs, representing a significant evolution in alignment methodology beyond traditional manual supervision. The approach is supported by open-source implementations (Lu et al., 14 Sep 2025), facilitating ongoing research and practical adoption.