GraphDPO: Graph-Based Preference Optimization

• GraphDPO is a family of methodologies blending graph theory and Direct Preference Optimization to enable effective model unlearning, language alignment, and term graph rewriting.
• It employs innovative techniques such as boundary-aware sampling and DAG-based ranking to provide robust negative signals and maintain semantic consistency.
• Empirical evaluations show significant performance gains, including notable improvements in MRR metrics and efficient scalability compared to traditional retraining methods.

GraphDPO refers to a family of methodologies in machine learning, graph theory, and symbolic rewriting that leverage the interplay between graph structures and Direct Preference Optimization (DPO). Depending on context, GraphDPO has appeared as: (1) a preference-optimization approach for knowledge graph unlearning, (2) a generalization of DPO for graph-structured preference supervision in language modeling, and (3) a double-pushout (DPO) framework for semantics-preserving term graph rewriting. The term has also been used as an acronym for Graph Diffusion Policy Optimization in graph generation. This article systematically reviews the technical foundations and principal instantiations of GraphDPO.

1. Direct Preference Optimization in Graph Unlearning

GraphDPO, as introduced in the context of knowledge graph embedding (KGE) unlearning, reframes the problem of eliminating specific “forgetting” triples from a trained KGE model as an offline preference optimization task (Liu et al., 28 Jul 2025). Given a knowledge graph $$\mathcal{G} = (\mathcal{E}, \mathcal{R}, \mathcal{T})$$ and a model $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$ where $f(h, r, t)$ scores triples, the set of training triples $\mathcal{D}_t \subseteq \mathcal{T}$ is partitioned into a forgetting set $\mathcal{D}_f$ and retained set $\mathcal{D}_r$. Traditional fine-tuning often fails to truly “forget” due to graph connectivity; inferrable facts persist via indirect paths and local updates distort embeddings near the forgetting boundary.

GraphDPO addresses this by:

• Reformulating forgetting as a preference task, penalizing dis-preferred (forbidden) triples relative to sampled alternatives.
• Employing the DPO loss:

$$\mathcal{L}_{dpo} = -\mathbb{E}_{(x, y_l, y_w)\sim\mathcal{D}_f^{po}} [\log P_\theta(y_w \succ y_l \mid x)]$$

where $P_\theta(y_w \succ y_l \mid x)$ is a sigmoid of the log-odds that the model prefers sampled $y_w$ over the original entity $y_l$, with normalization against a frozen reference model.

This approach is theoretically shown to be affinely equivalent to the original unlearning objective over $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$0, ensuring robust gradient signals targeted at forgetting (Liu et al., 28 Jul 2025).

2. Specialized Sampling and Boundary-Aware Knowledge Recall

Naive negative sampling for preference pairs is insufficient in highly connected knowledge graphs. GraphDPO introduces out-boundary sampling: for each forbidden $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$1, alternatives $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$2 are sampled outside its “forgetting boundary” $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$3. This reduces semantic and structural overlap, providing stronger negative signals and limiting the leakage of forbidden information via nearby entities.

Local distortion at forgetting boundaries is mitigated by boundary-aware knowledge recall. This comprises:

• Boundary Replay: Margin-ranking loss on all triples adjacent to any $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$4, ensuring continued high scores for retained knowledge near forgotten facts.
• Boundary Distillation: L2-Huber distillation penalizes large embedding drifts for boundary entities, regularizing them against the reference model.

The full model employs a joint objective:

$\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$5

with all $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$6 coefficients set to unity in empirical evaluation.

3. GraphDPO for Preference Graph Supervision in LLMs

A separate instantiation of GraphDPO appears as a generalization of DPO for LLM alignment (Liu et al., 8 May 2026). Here, the limitation of pairwise DPO—namely, the loss of transitive or partial ordinal information in rankings over multiple responses—is addressed. Given $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$7 model completions per input, GraphDPO encodes the overall supervision as a directed acyclic graph (DAG) of dominance (preference) relations:

• Equivalence classes $\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$8 represent total or partial orderings, with intra-layer ties inducing zero-loss constraints.
• The objective optimizes a local Plackett–Luce loss per node:

$\mathcal{M} = \{ \mathrm{Emb}, \mathrm{Rmb}, f \}$9

• Aggregating over all nodes with non-empty dominated neighborhoods:

$f(h, r, t)$0

• If an oracle solution is available, an anchoring loss is employed, with time-annealed weighting.

This structure enforces transitivity, reduces label noise from redundant pairwise decompositions, and supports efficient optimization, retaining linear complexity per prompt when the equivalence-class decomposition induces a layered DAG.

4. Experimental Evaluation and Empirical Properties

Empirical studies in knowledge graph unlearning (Liu et al., 28 Jul 2025) show that GraphDPO achieves up to +10.1% absolute improvement in $f(h, r, t)$1 and +14.0% in $f(h, r, t)$2 over approximate baselines, matching 80–99% of exact retraining performance at 4–5$f(h, r, t)$3 speedup. Ablation confirms that the DPO loss, boundary replay/distill, and out-boundary sampling are all critical to effectiveness.

In LLM alignment (Liu et al., 8 May 2026), GraphDPO outperforms both pairwise DPO and strict listwise objectives by 3–4 points on complex reasoning tasks such as GSM8K, MATH-500, and APPS, with optimal gains when supervision reflects partial orders or equivalence classes. The method handles ties naturally via masked zero-loss intra-layer edges.

5. Double-Pushout Approach to Term Graph Rewriting

In symbolic and categorical frameworks, GraphDPO refers to the double-pushout (DPO) method for rewriting term graphs (Kahl et al., 2019). A DPO production is specified as a span of graph morphisms $f(h, r, t)$4, where $f(h, r, t)$5 (left-hand side), $f(h, r, t)$6 (right-hand side), and $f(h, r, t)$7 (interface or gluing graph) are term graphs. The transformation applies by identifying a matching $f(h, r, t)$8 in the host graph $f(h, r, t)$9, computing a pushout complement $\mathcal{D}_t \subseteq \mathcal{T}$0, and then gluing in $\mathcal{D}_t \subseteq \mathcal{T}$1 via the categorical pushout, subject to dangling and identification conditions.

Kahl and Zhao show that, provided $\mathcal{D}_t \subseteq \mathcal{T}$2 and $\mathcal{D}_t \subseteq \mathcal{T}$3 have identical semantics under a chosen gs-monoidal functor $\mathcal{D}_t \subseteq \mathcal{T}$4 to a semantic domain $\mathcal{D}_t \subseteq \mathcal{T}$5, DPO rewriting preserves semantics:

$\mathcal{D}_t \subseteq \mathcal{T}$6

where $\mathcal{D}_t \subseteq \mathcal{T}$7 is the rewritten graph. This underpins the use of DPO for semantics-preserving and modular program transformations.

6. Related and Confusable Terminology

The term GraphDPO has also been used to denote Graph Diffusion Policy Optimization (Liu et al., 2024), an RL-based algorithm for optimizing diffusion models on graph-structured data. This technique is not based on preference-graph supervision or symbolic DPO rewriting, but rather modifies diffusion model policy gradients for graph generation and optimization tasks. It is not related to preference-optimization GraphDPO and should not be conflated with it.

7. Strengths, Limitations, and Research Directions

For knowledge graph unlearning, strengths of GraphDPO (Liu et al., 28 Jul 2025) include principled preference-based regularization, exploitation of graph connectivity for robust negative sampling, prevention of boundary entity damage, and scalability near fine-tuning cost. Noted limitations include fixed forgetting rates in evaluation and storage scaling with boundary size; future extensions are anticipated in adaptive forgetting, continual learning integration, and adaptation to contextualized KGE architectures.

In language modeling (Liu et al., 8 May 2026), GraphDPO’s compact encoding of arbitrary partial orders and equivalence classes enables support for nuanced preference data, robust gradient signals, and transitivity enforcement. Open questions concern handling continuous rewards, arbitrary partial orders, and the design of hybrid loss architectures.

The DPO-based term graph rewriting framework (Kahl et al., 2019) is foundational in semantics-preserving program transformation, with theoretical guarantees arising from the categorical compositionality of the graphs and rewriting rules.

---

Summary Table: Main Instantiations of GraphDPO

Context	Core Principle	Key Reference
KGE Unlearning (Preference DPO)	Pairwise preference optimization for forgetting	(Liu et al., 28 Jul 2025)
LLM Alignment (DAG DPO)	Plackett–Luce loss on preference DAGs	(Liu et al., 8 May 2026)
Term Graph Rewriting (DPO Matching)	Double-pushout categorical rewriting	(Kahl et al., 2019)
Diffusion Policy Opt. (RL on Graphs)	Eager policy gradient for diffusion models	(Liu et al., 2024)

Each instantiation maintains the unifying theme of leveraging graph structures to inform preference, optimization, or transformation, but differs fundamentally in objective, underlying mathematics, and application domain.