DZ-TDPO: Dual-Zone Temporal DPO

• DZ-TDPO is a non-destructive alignment framework for dialogue systems that overcomes state inertia by dynamically decaying stale context signals.
• It employs Conflict-Aware Dynamic KL constraints (TDPO-DKL) to adjust temporal loss weights based on semantic conflicts detected via SBERT embeddings.
• Dual-Zone Temporal Attention (DZ-TA) separates anchor tokens from mutable state tokens, ensuring robust recency focus without sacrificing foundational context.

Searching arXiv for DZ-TDPO and directly related alignment papers to ground the article in current literature. DZ-TDPO, short for Dual-Zone Temporal DPO, is a non-destructive alignment framework for long-context dialogue with mutable state tracking. It is designed to address State Inertia, a failure mode in which static alignment methods treat prior dialogue context as an effectively immutable prior and consequently over-trust stale historical context when user intents or world-state facts change. The framework combines Conflict-Aware Dynamic KL constraints (TDPO-DKL) with Dual-Zone Temporal Attention (DZ-TA), coupling a temporally adaptive preference-optimization objective with a learnable attention-logit bias that preserves anchor tokens while attenuating obsolete mutable-state tokens (Liao, 3 Dec 2025). In experiments on the Multi-Session Chat (MSC) dataset, the method is reported to achieve 86.2% win rate on Phi-3.5-mini-instruct and 99.4% on Qwen2.5-7B, while maintaining comparatively low perplexity overhead and robust zero-shot behavior (Liao, 3 Dec 2025).

1. Problem setting and motivation

DZ-TDPO is situated in the problem of long-context dialogue alignment, especially in multi-session interaction where user preferences, commitments, and factual states evolve over time. In this setting, earlier turns may become semantically incompatible with later turns. The central diagnosis in the source work is that standard preference optimization methods, exemplified by Direct Preference Optimization (DPO), can exhibit State Inertia because the optimization and attention patterns remain anchored to a large body of earlier tokens (Liao, 3 Dec 2025).

The failure mode is formalized through the distribution of attention mass across turns. Let $\alpha_t$ denote the total attention mass on turn $t$. In a long dialogue of $T$ turns, standard DPO is described as leading to

$$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$

meaning that distant history receives disproportionately large attention relative to the most recent $k$ turns, even when those recent turns contain the relevant state update (Liao, 3 Dec 2025).

The paper also frames a second pathology: when one attempts to override stale context through aggressive weight updates, general language competence may degrade. This is termed the Alignment Tax, described in terms of perplexity spikes and hallucinations under destructive reweighting of model behavior (Liao, 3 Dec 2025). A plausible implication is that the paper treats temporal alignment not merely as a preference-learning problem, but as a joint optimization problem over state mutability, stability of general capabilities, and contextual recency sensitivity.

2. Core architecture: TDPO-DKL and DZ-TA

DZ-TDPO consists of two modules: Conflict-Aware Dynamic KL constraints (TDPO-DKL) and Dual-Zone Temporal Attention (DZ-TA) (Liao, 3 Dec 2025). The first operates at the loss level, adjusting the effective KL constraint and temporal weighting as a function of conflict and recency; the second operates in the forward pass by reshaping attention logits.

The source paper contrasts its approach with standard DPO, whose objective is given as

$$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$

With a fixed $\beta$, historical tokens dominate the KL anchoring effect in long contexts, which the paper identifies as a mechanism behind inertia (Liao, 3 Dec 2025).

TDPO-DKL modifies this picture by making the KL coefficient and loss weighting time-varying. State conflict is detected by encoding each user turn $u_t$ into an SBERT embedding $e_t \in \mathbb{R}^d$ and computing

$$\max_{i<t}\cos(e_t,e_i),\quad \tau(u_t)=\tau_{\rm base}\bigl[1-\gamma\max_{i<t}\cos(e_t,e_i)\bigr].$$

A low $t$0 is interpreted as indicating high semantic conflict, and therefore as a signal to accelerate decay of historical influence (Liao, 3 Dec 2025).

For a preference pair at turn $t$1 within context length $t$2, the time-varying KL penalty coefficient is

$t$3

and the temporal loss weight is

$t$4

The margin term is

$t$5

yielding the TDPO-DKL objective

$t$6

The stated purpose is to “unshackle” the model at conflict points while preserving stability on static contexts (Liao, 3 Dec 2025).

DZ-TA complements this by partitioning the context into an Anchor Zone $t$7, containing tokens $t$8 such as the system prompt and safety instructions, and a State Zone $t$9, containing tokens $T$0 representing mutable user history (Liao, 3 Dec 2025). The attention bias is

$T$1

where $T$2, $T$3 is a preset scale, and $T$4 is a shared learnable scalar initialized at 0.5 (Liao, 3 Dec 2025). In each attention head,

$T$5

This is described as preserving anchor tokens while forcing heads to focus more strongly on recent state under conflict (Liao, 3 Dec 2025).

3. Training objective, optimization, and implementation details

The combined objective is

$T$6

subject to the modified forward pass produced by DZ-TA (Liao, 3 Dec 2025). The framework therefore combines a temporal preference-learning term with explicit regularization on the learnable bias scalar.

The paper reports the following typical hyperparameters: $T$7, $T$8, $T$9, $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$0, and $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$1; the anchor length $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$2 is approximately the system-prompt size; the learning rates are 1.5e-5 for the backbone and 1e-4 for $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$3 (Liao, 3 Dec 2025). These values define the concrete operating regime in which the paper’s empirical results are obtained.

An algorithm sketch is also given. For each preference tuple $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$4 in MSC, the method computes an embedding for the final user turn $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$5, derives $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$6, computes $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$7 and $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$8 for each preference pair at turn $$\frac{1}{T-k}\sum_{t=1}^{T-k}\alpha_t \;\gg\; \sum_{t=T-k+1}^T\alpha_t,$$9, injects the bias $k$0 into attention logits, evaluates the margin $k$1, and optimizes

$k$2

with the stated learning rates (Liao, 3 Dec 2025). The source text states that the formulas and hyperparameters are sufficient to re-implement the core framework.

This training design is explicitly characterized as non-destructive alignment. The claim is that by suppressing conflicting signals at the attention level before they propagate into gradient updates, the optimization proceeds on a higher-SNR landscape and avoids broader degradation of fluency and general knowledge (Liao, 3 Dec 2025). This suggests a division of labor between modules: DZ-TA filters contextual relevance in the forward pass, and TDPO-DKL governs how preference supervision acts on the filtered representation.

4. Theoretical analysis and the capacity–stability trade-off

The paper proposes a theoretical account of the trade-off between temporal selectivity and statistical stability. With temporal weighting $k$3 and per-turn distribution drift bounded by $k$4, the generalization error on the current turn $k$5 is modeled as

$k$6

The first term is interpreted as approximation bias, increasing with $k$7 because forcing too much history with drift onto the current state raises error. The second term is estimation variance, increasing as $k$8 when too little history is used (Liao, 3 Dec 2025).

Optimizing this bound gives

$k$9

The accompanying interpretation is that larger models have smaller effective $$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$0 because they adapt more smoothly, so their optimal temporal horizon $$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$1 is larger (Liao, 3 Dec 2025). The source presents this as the Capacity-Stability Trade-off, linking parametric scale to the cost of temporal realignment.

This argument is tied directly to the empirical notion of alignment tax. The paper reports that Phi-3.5 (3.8B) incurs +2.7 PPL to achieve 86.2% win rate, whereas Qwen2.5-7B incurs +1.95 PPL for 99.4% win rate (Liao, 3 Dec 2025). The source explicitly states that the alignment tax is proportional to the model’s parametric capacity in the sense that smaller models possess less buffer for introducing a temporal bias without distorting general knowledge.

The theoretical account should be read as a model-based interpretation rather than a formal impossibility theorem. A plausible implication is that DZ-TDPO is meant not only as a method for current systems but also as an argument that temporal alignment mechanisms should scale with model size, rather than relying on uniform preference-optimization protocols across architectures.

5. Experimental setting and quantitative results

The reported experiments use the Multi-Session Chat (MSC) dataset, which contains user–assistant dialogues across 5 sessions, with up to approximately 1.7k tokens per 4-session context (Liao, 3 Dec 2025). The study focuses on Session 4 as the current turn $$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$2, concatenating the prior 3 sessions with the current session to obtain a context of approximately 2k tokens (Liao, 3 Dec 2025). Preference pairs $$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$3 are filtered using two criteria: semantic similarity < 0.5 to avoid false negatives, and $$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$4 to avoid length bias (Liao, 3 Dec 2025).

The evaluation protocols are in-domain Win Rate (WR), defined as the fraction of cases in which the model selects the temporally correct response; OOD WR on UltraChat (zero-shot); and Perplexity (PPL) on MSC and MMLU as measures of general knowledge retention (Liao, 3 Dec 2025).

For Phi-3.5-mini-instruct (3.8B), the main results are as follows (Liao, 3 Dec 2025):

Method	MSC WR	OOD WR	MSC PPL
Base Model	20.2%	10.6%	22.1
Standard DPO (β=0.1)	52.2%	63.2%	124.1
SimPO	60.8%	30.8%	99.6
TDPO-DKL (w/o DZ-TA)	76.4%	68.1%	100.9
DZ-TDPO (ours)	86.2%	71.0%	24.8

The same table in the source also reports MMLU PPL values of 5.27 for the base model, 5.35 for standard DPO, 5.28 for SimPO, 5.67 for TDPO-DKL without DZ-TA, and 5.45 for DZ-TDPO (Liao, 3 Dec 2025). Within the paper’s own framing, this indicates that the full combination yields the strongest temporal alignment while avoiding the severe perplexity inflation observed for purely objective-level baselines.

The scaling analysis comparing Phi-3.5 3.8B and Qwen2.5-7B reports the following (Liao, 3 Dec 2025):

Metric	3.8B Phi-3.5	7B Qwen2.5
In-Domain WR (MSC)	86.2%	99.4%
Alignment Tax (ΔPPL)	+2.7	+1.95
OOD WR (4k)	71.0%	91.8%
OOD WR (8k extrapolation)	48.4%	78.0%
MMLU PPL Variation	+0.18	+0.46

The ablation study is also central to the paper’s interpretation. TDPO-DKL only reaches 76.4% MSC WR with PPL 100.9; DZ-TA only (static DPO + bias) reaches 65% WR with PPL ~24; the full DZ-TDPO combination yields the best WR together with low PPL (Liao, 3 Dec 2025). This is used to support the claim that neither dynamic temporal weighting nor attention bias alone fully resolves the inertia problem.

6. Relation to preference optimization and temporal alignment

DZ-TDPO belongs to the lineage of preference optimization methods derived from DPO-style objectives, but it departs from standard DPO by making the KL pressure explicitly context-position dependent and by modifying the forward attention mechanism itself (Liao, 3 Dec 2025). In the source framing, standard DPO uses a fixed $$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(c,y_w,y_l)}\Big[ \log\sigma\bigl(\beta\bigl[\ln\tfrac{\pi_\theta(y_w|c)}{\pi_{\rm ref}(y_w|c)} -\ln\tfrac{\pi_\theta(y_l|c)}{\pi_{\rm ref}(y_l|c)}\bigr]\bigr) \Bigr].$$5, which causes historical tokens to dominate because they vastly outnumber recent tokens in long contexts. DZ-TDPO addresses this by decaying both effective supervision weight and KL anchoring according to conflict-aware temporal distance.

The method is therefore not merely a variant loss, nor merely an attention patch. Its defining claim is the synergy between objective-level and inference-path interventions: TDPO-DKL reduces destructive optimization pressure near conflict points, while DZ-TA preserves anchor content and down-weights stale mutable-state content in the attention graph (Liao, 3 Dec 2025).

The paper situates this as an alternative to methods that rely on strong weight updates to overwrite old state. The terminology non-destructive alignment expresses the view that temporal adaptation should occur through precise attention regulation rather than destructive weight updates, thereby preserving general capabilities such as performance measured through MMLU perplexity (Liao, 3 Dec 2025). This suggests a broader methodological distinction between alignment mechanisms that primarily alter model parameters globally and those that reshape how context is consulted locally.

A further point of comparison appears in the ablation against SimPO and standard DPO. On the reported Phi-3.5 setup, both alternatives improve in-domain WR over the base model, but with substantially higher MSC perplexity than the full method (Liao, 3 Dec 2025). The source interprets this as evidence that naïve preference optimization can solve the temporal conflict only by paying a large alignment tax, whereas DZ-TDPO aims to separate temporal correction from broad degradation of language modeling behavior.

7. Limitations, failure modes, and prospective extensions

The source identifies several limitations. First is the semantic-logic gap: cosine similarity over SBERT embeddings may fail to capture subtle negations, so conflict detection may miss semantically incompatible but lexically similar turns (Liao, 3 Dec 2025). The paper explicitly suggests that future work could integrate NLI heads to address this issue.

Second is Ping-Pong instability under rapid oscillations, for which the paper proposes the possible introduction of a belief inertia term (Liao, 3 Dec 2025). This indicates that while the method is designed to overcome state inertia, very frequent reversals of user intent may create a different instability regime in which recency sensitivity itself becomes problematic.

Third, the paper notes a heuristic trailing-edge assumption that may ignore valid long-distance corrections (Liao, 3 Dec 2025). This is a substantial caveat for any method that privileges recency, since some dialogues contain delayed references or long-range corrective dependencies that should not be forgotten.

Future directions listed in the source include incorporating hybrid NLI conflict detectors, extending DZ-TA to efficient transformer variants such as Mamba/Jamba, and exploring combinations with noise-canceling SSM modules (Liao, 3 Dec 2025). These proposed extensions imply that the authors view DZ-TDPO not as a fixed architecture-specific patch, but as a more general strategy for temporal state management in long-context generative systems.

Taken together, DZ-TDPO defines a specific research program in long-context alignment: mutable state should be treated as a first-class optimization object, conflict detection should modulate both supervision and attention, and preservation of anchor tokens should be separated from forgetting of stale user-state tokens. Its principal empirical claim is that this combination can alleviate temporal alignment failures while reducing the alignment tax relative to standard preference-optimization baselines, especially as model capacity increases (Liao, 3 Dec 2025).