Factorized Conversational Dynamics Reward (FCDR)

• The paper presents FCDR, a reward formulation that decouples conversational timing from semantic response generation to optimize 'when to speak'.
• It factorizes the reward into interpretable components for turn initiation, backchannel timing, yielding after barge-in, and regularized participation.
• Empirical results on datasets like Fisher and Seamless show improved turn-taking initiation, lower onset MAE, and enhanced backchannel performance.

Factorized Conversational Dynamics Reward (FCDR) is a reward formulation for full-duplex spoken dialogue models that treats conversational timing as a standalone optimization target rather than as a by-product of semantic imitation. In its explicit form, FCDR is introduced to optimize “when to speak” while preserving “what to say,” converting human timing patterns into local reinforcement-learning signals over short dynamics-critical windows. The reward is factorized into interpretable components for turn initiation, backchannel timing, yielding after user barge-in, and regularized participation, and is used within the DuplexPO framework to train temporal interaction behavior without directly rewarding semantic content (Li et al., 8 Jul 2026). The term also admits a broader conceptual reading across adjacent reward-modeling literature: several works instantiate closely related ideas by factorizing conversational reward across causal vs non-causal latent factors, turn-level local rewards, geometric interaction features, or structured dialogue-action hierarchies (Yang et al., 29 Jan 2026).

1. Definition and conceptual scope

In the DuplexPO formulation, FCDR is a window-level scalar reward for full-duplex spoken interaction. The underlying problem is framed on a regular frame grid, where a model listens continuously to user audio $x_{1:T}$ and decides an interaction state $y_t$ at each frame via

$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$

The central claim is that conversational dynamics can be learned as a separate real-time decision policy from human dialogue data, thereby decoupling timing behavior from semantic response generation (Li et al., 8 Jul 2026).

This decoupling is motivated by an intelligence--dynamics trade-off observed in recent full-duplex spoken dialogue models: systems that are effective at low-latency response, backchanneling, and barge-in handling often exhibit weaker reasoning and instruction-following abilities. DuplexPO addresses this by preserving the semantic response capability of an instruction-tuned assistant and optimizing only temporal interaction behavior over selected high-impact windows from long human conversations. FCDR is the reward function that makes this separation operational (Li et al., 8 Jul 2026).

A broader interpretation is supported by neighboring work. One paper on robust reward modeling in RLHF states that it gives “exactly the kind of mechanism you would want for a ‘Factorized Conversational Dynamics Reward (FCDR)’ in RLHF,” because it decomposes conversational representations into reward-relevant and reward-irrelevant latent factors (Yang et al., 29 Jan 2026). Another paper on session-level feedback does not use the term “FCDR,” but conceptually implements “a factorized reward over conversational dynamics” by redistributing a single global conversational score into turn-level local implicit rewards (Lee et al., 21 May 2025). This suggests that FCDR can denote both a specific analytical reward for conversational timing and a more general design pattern in which conversational reward is decomposed into interpretable factors.

2. Formal specification

FCDR is defined over dynamics-critical windows $W_i$ extracted around human agent segments $(a_i,b_i)$ from Fisher and Seamless human conversations. On the frame grid, the reference speaking interval is

$$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$

with window bounds

$$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$

and

$$e_i=\min\!\left(\bar{e}_i,\, \left\lfloor (a_{i+1}-L)/\Delta \right\rfloor,\, T\right).$$

Each window is therefore

$W_i = (s_i, e_i, g_i, h_i, c_i),$

where $c_i$ records metadata such as whether the segment is a backchannel and whether the user barges in (Li et al., 8 Jul 2026).

Within a window, the model’s behavior is represented as the set of speaking frames

$y_t$0

If the model speaks at least once, its predicted onset frame is

$y_t$1

and the onset delay relative to the human timing is

$y_t$2

From the event metadata, three binary masks are derived: $y_t$3 for turn initiation relevance, $y_t$4 for backchannel relevance, and $y_t$5 for barge-in yielding relevance (Li et al., 8 Jul 2026).

The overall reward is

$y_t$6

where $y_t$7, $y_t$8, $y_t$9, and $\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$0, and the $\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$1 terms are scalar weights (Li et al., 8 Jul 2026).

The factorization is analytically explicit rather than learned from a black-box teacher. This is a central distinction of FCDR: it is designed from conversational analysis and frame-level timing geometry, not inferred solely from end-to-end transcript scoring. The paper reports low “cross-talk” between start and stop rewards, which is presented as empirical evidence that the factorization yields specialized signals (Li et al., 8 Jul 2026).

3. Component structure

FCDR decomposes conversational dynamics into four interpretable factors.

Component	Range	Function
$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$2	$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$3	Turn initiation reward
$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$4	$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$5	Backchannel timing reward
$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$6	$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$7	Yielding reward after user barge-in
$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$8	$\pi_\theta(y_t \mid y_{<t}, x_{\le t}).$9	Pattern regularization penalty

The turn initiation reward encourages the agent to start speaking once per window, at roughly the same relative time as the human agent, with stronger penalties for early starts than for moderate positive delays. Its form is

$W_i$0

The bandwidth $W_i$1 is asymmetric: it is smaller for $W_i$2, so interruptive early starts are punished more strongly (Li et al., 8 Jul 2026).

The backchannel reward applies in designated backchannel windows and favors brief agent speech whose timing falls within or very close to the human backchannel interval $W_i$3: $W_i$4 If any speaking frame falls inside the annotated backchannel interval, the reward can reach 1; otherwise it decays exponentially with distance from the interval (Li et al., 8 Jul 2026).

The yielding reward handles user interruption. Let $W_i$5 be the total agent speech duration after user barge-in and $W_i$6 the maximum tolerated overlap. Then

$W_i$7

If $W_i$8, there is no penalty; if overlap continues beyond tolerance, the penalty grows toward $W_i$9. The implementation further incorporates a “No-<EOS> penalty scale,” “Stop tolerance,” “Early/late stop penalties,” and an “Explicit <EOS> bonus” (Li et al., 8 Jul 2026).

The regularization term penalizes undesirable interaction patterns not fully captured by the primary timing objectives: $(a_i,b_i)$0 Concrete patterns include false alarms, missed events, backchannel overlong behavior, silence near a target start, and early or late stopping. The paper lists missed-event penalty $(a_i,b_i)$1, false-alarm penalty $(a_i,b_i)$2, backchannel overlong penalty $(a_i,b_i)$3, near-target PAD penalty $(a_i,b_i)$4, and early / late stop penalties $(a_i,b_i)$5 / $(a_i,b_i)$6 (Li et al., 8 Jul 2026).

4. Temporal credit assignment and optimization in DuplexPO

The policy operates at frame resolution $(a_i,b_i)$7 s, corresponding to a 12.5 Hz decision grid. Training windows use a 1.0 s lead $(a_i,b_i)$8 before the human onset and a 2.0 s buffer $(a_i,b_i)$9 after the human offset; validation uses a slightly larger window. Each sampled window yields a full trajectory of decisions, but the reward is assigned only at the window level. Credit assignment remains temporally local because each factor is computed from frame-level events such as the first speech frame, distance to the backchannel interval, and overlap duration after barge-in (Li et al., 8 Jul 2026).

DuplexPO uses a GRPO-style objective. For each window $$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$0, the current policy produces $$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$1 rollouts, with samples per window set to 4. For rollout $$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$2,

$$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$3

Within the window, the mean and standard deviation are

$$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$4

and the group-normalized advantage is

$$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$5

The training objective is

$$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$6

with $$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$7, and the policy gradient is applied only within the window $$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$8 (Li et al., 8 Jul 2026).

Two design choices enforce the separation of timing from semantics. First, RL updates are restricted to dynamics-critical windows. Second, the only RL reward is FCDR, which depends purely on timing and overlap geometry rather than semantic content. The KL penalty keeps the policy close to the SFT reference model, so the system learns to emit $$g_i = \left\lfloor a_i / \Delta \right\rfloor,\qquad h_i = \left\lfloor b_i / \Delta \right\rfloor,$$9, $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$0, and PAD with different timing while not receiving reinforcement for altering utterance content (Li et al., 8 Jul 2026).

This role separation distinguishes FCDR from reward formulations that operate over textual helpfulness or session-level satisfaction. A plausible implication is that FCDR is best understood not as a general conversation-quality reward, but as a specialized control objective for conversational timing behavior.

5. Empirical behavior and ablation evidence

The reported gains are concentrated in timing-sensitive interaction metrics. On Fisher and Seamless, DuplexPO with FCDR and GRPO improves turn-taking initiation and yielding, with turn-taking initiation rate up to 100%, yield rates for full turns and backchannels close to 100%, and reduced onset MAE. On Fisher, onset MAE is reported as 0.69 s for DuplexPO versus 0.98 s for the SFT baseline and 1.50–2.0 s for other models. On FDB-v3, DuplexPO achieves turn-take success 100%, latency 0.24 s, yield rate 100%, and voiced interrupt rate 5% (Li et al., 8 Jul 2026).

Human- or LLM-based evaluation shows that these metric improvements are reflected in perceived interaction quality. Gemini 3.0 Pro prefers DuplexPO over the SFT baseline in 76.9% of Fisher conversations and 69.3% of Seamless conversations, with strong preference on backchanneling and barge-in handling. Judge calibration with synthetic time shifts is reported to verify sensitivity to timing only, not content (Li et al., 8 Jul 2026).

The paper includes several ablations directly relevant to FCDR. A neural reward model based on a teacher that predicts temporal states improves over an SFT-dynamics baseline but often fails to surpass the original SFT baseline, and is weaker on backchannel yielding and ambiguous turn transitions. FCDR consistently outperforms that neural reward model and the SFT baselines on Fisher and Seamless initiation and yield metrics, and achieves the best barge-in yield on FDB-v3. The reported conclusion is that explicit factorized analytical rewards provide clearer, more stable signals for precise $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$1/$$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$2 decisions than a learned black-box reward model (Li et al., 8 Jul 2026).

Optimization and window-design ablations further sharpen the characterization. GRPO yields higher yield rates and better backchannel metrics than DPO when both use FCDR, suggesting that the full distribution of rewards in a group matters for subtle timing policies. Longer lead times significantly reduce RL reward, whereas increasing the buffer length has smaller effect, supporting the choice of windows that closely surround the event (Li et al., 8 Jul 2026).

6. Relation to adjacent factorized reward paradigms

Although the explicit name FCDR is tied to full-duplex spoken timing, related work supplies broader templates for factorized conversational reward.

A closely related RLHF reward-modeling paper proposes a factored representation learning framework that decomposes the contextual embedding $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$3 into causal factors $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$4 and non-causal factors $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$5, constraining the reward head to depend only on $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$6 and using an adversarial head with gradient reversal to discourage reward information in $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$7. The paper explicitly frames this as “exactly the kind of mechanism you would want” for FCDR, with reward-relevant conversational dynamics placed in $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$8 and reward-irrelevant artifacts such as length or sycophantic bias placed in $$s_i=\max\!\left(e_{i-1},\,\left\lfloor (a_i-L)/\Delta \right\rfloor\right), \qquad \bar{e}_i=\left\lfloor (b_i+B)/\Delta \right\rfloor$$9 (Yang et al., 29 Jan 2026). This suggests an “unsupervised FCDR” interpretation in which factorization is learned at the representation level rather than analytically specified from event geometry.

Another paper proposes using a frozen pretrained LLM to decompose a single session-level scalar feedback signal into turn-level local implicit rewards $$e_i=\min\!\left(\bar{e}_i,\, \left\lfloor (a_{i+1}-L)/\Delta \right\rfloor,\, T\right).$$0, with a prompt-level soft constraint that their sum approximates the global reward. The authors state that their paper does not use the term “FCDR,” but conceptually implements a factorized reward over conversational dynamics, especially in the multimodal variant where listener affect, gaze, and prosody are translated into natural-language descriptions aligned to each turn (Lee et al., 21 May 2025). This provides a temporal factorization of global conversational outcome into local rewards.

A third line of work defines TRACE, a trajectory-based reward over conversational geometry. TRACE computes a feature vector $$e_i=\min\!\left(\bar{e}_i,\, \left\lfloor (a_{i+1}-L)/\Delta \right\rfloor,\, T\right).$$1 from embedding-space distances, timestamps, goal adherence measures, and temporal volatility statistics, then predicts user satisfaction with a Random Forest Regressor. The paper’s description explicitly groups these features into interpretable families such as inefficiency and repetition, temporal dynamics, semantic cohesion and relevance, and goal orientation, and argues that this is “very close in spirit” to FCDR (Gooding et al., 11 Nov 2025). TRACE can therefore be read as a geometry-only conversational reward whose factors are summary statistics of interaction trajectory rather than event-local timing terms.

Older dialogue-management work offers yet another factorization pattern: a reward over domain, act, and slot, with sequential dependence $$e_i=\min\!\left(\bar{e}_i,\, \left\lfloor (a_{i+1}-L)/\Delta \right\rfloor,\, T\right).$$2. That work proposes multi-level and sequential reward modeling for task-oriented dialogue, factorizing the reward into domain-level, act-level, and slot-level components and combining them through a gated hierarchy (Hou et al., 2021). Although the setting differs from full-duplex timing, the structural principle is the same: replace a monolithic dialogue reward with multiple interpretable factors whose interactions reflect the underlying action schema.

Taken together, these neighboring formulations show that FCDR sits within a wider family of decomposed conversational reward designs. The explicit DuplexPO form is event-local and timing-geometric; adjacent work realizes similar factorization through latent-variable decomposition, turn-level reward redistribution, embedding-trajectory statistics, or structured dialogue-action hierarchies.

7. Limitations, assumptions, and prospective extensions

The DuplexPO paper identifies several limitations of FCDR as presently defined. Human timing annotations from Fisher and Seamless are treated as ground truth even though conversational timing varies across individuals, languages, and cultures. The reward is explicit and geometric, and therefore does not model subtle pragmatic factors such as user intent, discourse structure, or prosody. Because updates are restricted to short windows, longer-term consequences of timing decisions are not directly optimized. The reward also depends on multiple hyperparameters, including $$e_i=\min\!\left(\bar{e}_i,\, \left\lfloor (a_{i+1}-L)/\Delta \right\rfloor,\, T\right).$$3 weights, Gaussian bandwidths, decay constants, stop tolerances, and penalty scales, which likely require tuning per domain (Li et al., 8 Jul 2026).

A broader limitation, highlighted by the causal reward-modeling perspective, is that factorization is not provably identifiable: multiple decompositions may fit the data, and invariance may be only heuristically enforced rather than causally guaranteed (Yang et al., 29 Jan 2026). In the turn-level decomposition setting, decomposition quality depends on LLM reasoning, prompt design, and the fidelity of multimodal feature extraction and textualization (Lee et al., 21 May 2025). In conversational-geometry approaches, reward quality depends on the embedding space and remains content-insensitive, potentially missing hallucinations or subtle semantic failures that do not manifest in trajectory structure (Gooding et al., 11 Nov 2025).

The literature nevertheless converges on several extension paths. The DuplexPO paper explicitly suggests adding more factors such as prosody-sensitive rewards, user-specific timing preferences, adaptive weighting of the $$e_i=\min\!\left(\bar{e}_i,\, \left\lfloor (a_{i+1}-L)/\Delta \right\rfloor,\, T\right).$$4 terms conditioned on user profile or interaction history, multilingual and style-adaptive variants, and combinations with long-horizon objectives such as dialogue satisfaction or engagement (Li et al., 8 Jul 2026). The causal factorization work suggests multi-head reward decomposition on the causal latent and optional auxiliary heads on the non-causal latent for explicit style dynamics (Yang et al., 29 Jan 2026). The session-level decomposition work suggests multi-aspect and hierarchical factorization, while TRACE suggests combining dynamics-based and content-based rewards in hybrid models (Lee et al., 21 May 2025).

These developments indicate that FCDR is both a concrete analytical reward for real-time spoken interaction and a broader research direction in which conversational reward is decomposed into behaviorally meaningful factors. In the narrow sense, it is a mathematically specified, window-local timing reward used to optimize turn initiation, backchanneling, yielding, and regularized participation. In the wider sense, it names a family of approaches that seek interpretable, fine-grained credit assignment over conversational dynamics rather than reliance on a single undifferentiated scalar reward.