Style-Enhanced DPO (SeDPO) in WeStar

• SeDPO is a style-aware Direct Preference Optimization that uses controlled preference pair construction to isolate subtle stylistic differences while keeping context intact.
• It employs cluster-specific LoRA parameters on quality-filtered CQSA instances within the WeStar framework, boosting stylistic alignment in contextual question answering.
• Empirical results show that SeDPO improves stylistic strength and contextual coherence with faster inference compared to traditional DPO and SFT methods.

Style-Enhanced Direct Preference Optimization (SeDPO) is a style-aware instantiation of Direct Preference Optimization in which preference pairs are constructed to isolate fine-grained stylistic distinctions while keeping context and semantics as constant as possible. In the industrial framework WeStar, SeDPO is used to train style-cluster-specific LoRA parameters for stylized contextual question answering: chosen responses are taken from the top 10,000 high-quality Context, Question, Stylized Answer (CQSA) instances within a style cluster, while rejected responses answer the same question, remain stylistically similar overall, and differ in at least one style label, often through sibling nodes in a style tree (Fan et al., 22 Sep 2025). The broader survey literature on DPO does not list SeDPO as a standard taxonomy entry, so the term currently refers primarily to this pair-construction-centric, style-clustered formulation rather than to a universally standardized objective class (Liu et al., 12 Mar 2025).

1. Conceptual scope and defining idea

SeDPO arises from a specific limitation of ordinary stylized supervised fine-tuning. In stylized contextual question answering, the model must be both contextually grounded in retrieved articles and stylistically aligned with an author’s historical reply behavior. Ordinary supervised training can imitate stylized targets, but it does not explicitly teach the model why one response is better than another for the same question when the difference is primarily stylistic rather than semantic. WeStar introduces SeDPO to sharpen this boundary: for the same contextual QA situation, the model is trained to prefer the response that better matches the target style cluster and to reject a near-miss response that remains semantically similar but is stylistically different (Fan et al., 22 Sep 2025).

The defining property of SeDPO is therefore not a new analytic loss term, but a new preference construction regime. The paper explicitly states that the rejected sample is retrieved for the same question and is chosen to have high stylistic similarity while differing in a certain style label. This is framed as a controlled-variable setup: when the negative sample shares most contextual and semantic features with the positive sample, the model is encouraged to focus on fine-grained stylistic distinctions. In this sense, SeDPO is best understood as a style-sensitive data and optimization protocol layered onto standard DPO machinery rather than as a fundamentally new preference objective (Fan et al., 22 Sep 2025).

A common misconception is to treat SeDPO as merely “DPO with a style label.” The WeStar formulation is narrower and more structured. It depends on cluster-aware pair formation, quality-filtered stylized data, and cluster-specific low-rank adaptation parameters. Another misconception is to assume that SeDPO is already a canonical DPO variant recognized across the field. The survey literature explicitly does not mention SeDPO as a named category, which places it closer to an emerging application-specific formulation than to a settled family of objectives (Liu et al., 12 Mar 2025).

2. Architectural setting in WeStar

SeDPO is embedded in WeStar, a lite-adaptive framework for stylized contextual question answering designed for millions of official accounts. WeStar separates factual grounding from style control through a dual injection strategy. Question-specific knowledge is injected into the prompt through retrieval-augmented generation (RAG), while style-specific knowledge is injected into the model parameters through a Parametric RAG (PRAG) mechanism that dynamically activates LoRA modules associated with style clusters (Fan et al., 22 Sep 2025).

This architecture is motivated by the distinction between two knowledge sources. Published articles provide the factual and topical basis for answering user questions, whereas historical author replies capture conversational style more faithfully than articles do. WeStar therefore does not rely on long prompts containing both retrieved factual passages and style exemplars. Instead, context remains prompt-level, while style is stored parametrically in compact trainable adapters (Fan et al., 22 Sep 2025).

The style representation is explicitly hierarchical. Authors are labeled using 12 style labeling standards across four dimensions. At the semantic level, the standards are intention type and degree of authority. At the grammatical level, they include omitted features, use of inversion, and use of passive voice. At the syntactic level, they include sentence complexity, rhetorical features, and cohesion mechanisms. At the lexical level, they include lexical complexity, emotional polarity, emoji frequency, and degree of formality. Authors with similar profiles are grouped into a hierarchical style tree, and a leaf corresponds to a style cluster (Fan et al., 22 Sep 2025).

Style is thus represented in two coupled forms: a discrete cluster identity and a cluster-specific set of LoRA parameters. This cluster-based parameter sharing is the scalability mechanism. WeStar does not train one model per account; it trains one style adapter per cluster, reducing storage and training cost and making sparse-style accounts easier to support. SeDPO is the optimization stage that sharpens the behavior of these cluster-specific adapters (Fan et al., 22 Sep 2025).

3. Optimization formulation and style-aware preference pairs

The WeStar paper does not present a new closed-form loss beyond standard DPO. Its novelty lies in how the chosen and rejected responses are constructed. For a given style cluster, the chosen set consists of the top 10,000 CQSA instances, and each rejected sample is retrieved as an answer to the same question with high stylistic similarity but a difference in a certain style label. The paper states that this setup facilitates more effective learning of style-specific behaviors within each stylistic cluster (Fan et al., 22 Sep 2025).

The standard DPO objective used for pairwise preference optimization is

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)} \left[ \log \sigma \left( \beta \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right) \right) \right].$$

In WeStar, a faithful reading is that this same reference-relative structure is applied to style-aware preference pairs, with the trainable parameters being the LoRA parameters associated with a style cluster rather than a full independently trained model. The paper states that it adopts the same parameterization and injection paradigm as PRAG, employing LoRA as the fine-tuning and parameter storage strategy; this strongly indicates that SeDPO updates cluster-specific LoRA modules rather than the shared backbone (Fan et al., 22 Sep 2025).

What matters operationally is the conditioning structure of the pair. The input is a contextual QA instance associated with a target style cluster. The positive example is not just any preferred response, but a filtered high-quality stylized answer from that cluster. The negative example is not a generic poor response, but a style-confusable competitor answering the same question. This makes SeDPO closer to metric learning over stylistic margins than to coarse preference alignment over obviously good and bad outputs (Fan et al., 22 Sep 2025).

Several implementation details are explicit. The LoRA rank is 16. Training runs for one epoch. The top 10,000 CQSA instances per style cluster are used as chosen samples. The style-tree node-size threshold is $k=100$. Several usual DPO hyperparameters are not stated in the excerpt: the DPO temperature $\beta$, batch size, learning rate, and exact reference-model specification are all left unspecified there (Fan et al., 22 Sep 2025).

4. Upstream data construction and style representation pipeline

SeDPO depends on a multi-stage upstream data pipeline. It does not operate on raw account interactions directly. The first stage constructs Context, Question, Answer (CQA) triples by two large-language-model-based strategies: a forward-thinking procedure that generates QA pairs from article segments, and a bottom-up procedure that simulates realistic user roles and questions from the account domain, retrieves context, and generates answers. This yields context-grounded QA data (Fan et al., 22 Sep 2025).

The second stage performs style labeling and style-tree construction. Each author corpus is labeled over the 12 style standards, and majority labels define the author’s stylistic profile. The resulting tree is not only a clustering device; it also defines the neighborhood structure from which near-miss negatives are later retrieved for SeDPO (Fan et al., 22 Sep 2025).

The third stage rewrites CQA instances into CQSA instances. The rewrite prompt includes the original context and question, the original answer, the full set of 12 style standards with target labels, and $m$ in-context examples from the same cluster. These stylized answer candidates are then scored by an LLM along four dimensions:

Dimension	Meaning
Q-A	Question Relevance
C-A	Contextual Alignment
S-A	Stylistic Strength
Fluency	Naturalness and grammaticality

The scores are aggregated, and the top 10,000 CQSA instances are retained as high-quality positives for each style cluster (Fan et al., 22 Sep 2025).

This filtering stage is integral to SeDPO rather than incidental preprocessing. The chosen samples in SeDPO are therefore already constrained to be contextually aligned, question relevant, stylistically strong, and fluent before pairwise preference optimization begins. The negative samples then refine the supervision by holding question and much of the semantic content fixed while perturbing style. A plausible implication is that SeDPO uses pairwise optimization to discriminate within an already high-quality region instead of relying on DPO to separate good outputs from obviously defective ones (Fan et al., 22 Sep 2025).

5. Empirical performance and comparative interpretation

The experimental evidence for SeDPO comes from comparisons among WeStar variants. The main baselines are LoRA-SFT on 10,000 randomly selected CQSA examples, LoRA-SFT-S on the top 10,000 high-quality CQSA examples, and WeStar MDPO, a DPO-style baseline whose rejected samples are distilled from the base LLM using $(C,Q)$ prompts and selected by large metric deviation rather than by style-sensitive controlled contrasts (Fan et al., 22 Sep 2025).

Method	Q-A	C-A	S-A	Fluency
LoRA-SFT	4.35	4.43	3.92	4.73
LoRA-SFT-S	4.41	4.49	4.22	4.77
WeStar MDPO	4.44	4.52	4.20	4.76
WeStar (with SeDPO)	4.43	4.55	4.25	4.77

Relative to random CQSA fine-tuning, data curation alone improves all metrics. Relative to curated SFT, SeDPO still improves contextual alignment from 4.49 to 4.55 and stylistic strength from 4.22 to 4.25, while preserving fluency at 4.77. Relative to WeStar MDPO, SeDPO yields the highest S-A score and higher C-A, even though Q-A is nearly unchanged. The paper attributes this to the structure of the negative samples: in WeStar MDPO, rejected responses differ strongly across all four dimensions and can therefore be separated using coarse quality differences, whereas in SeDPO the rejected responses are closer in semantics and context and differ more specifically in style, forcing the model to learn finer stylistic preferences (Fan et al., 22 Sep 2025).

This comparison clarifies the empirical meaning of “style-enhanced.” SeDPO is not merely a stronger DPO baseline; it is a DPO instantiation whose training signal is concentrated on hard stylistic confounders. The strongest measurable effect is on Stylistic Strength rather than on broad utility metrics. That is consistent with the paper’s stated goal: style-specific parameter training for cluster-level LoRA modules (Fan et al., 22 Sep 2025).

The paper also reports a system-level efficiency result for WeStar as a whole: inference latency is 2.08 s/sample for WeStar versus 2.47 s/sample for SFT-Prompt, corresponding to a 1.19x speedup. This improvement is attributed to parameterized style injection through LoRA rather than to SeDPO alone, but it situates SeDPO within a deployment-oriented design in which style quality is improved without relying on long prompt-based style exemplars (Fan et al., 22 Sep 2025).

6. Relation to adjacent DPO variants and terminological boundaries

SeDPO should be distinguished from several adjacent but non-identical DPO lines. The DPO survey does not include SeDPO as a named variant, which suggests that the term remains application-specific rather than taxonomically standardized (Liu et al., 12 Mar 2025). The similarly named Diffusion-SDPO is unrelated in meaning: there, “SDPO” stands for Safeguarded Direct Preference Optimization, not Style-Enhanced Direct Preference Optimization (Fu et al., 5 Nov 2025).

Among adjacent style-oriented works, “Style Transfer with Multi-iteration Preference Optimization” is especially relevant methodologically. It frames style transfer as preference optimization over target style strength, meaning similarity, and fluency, and constructs preference pairs automatically from composite style-aware rewards, but its final optimizer is CPO rather than DPO (Liu et al., 2024). This suggests a broader interpretation of style-enhanced preference learning in which the critical design choice is often the construction of style-sensitive contrastive pairs rather than the invention of a new preference loss.

Other DPO research reinforces this interpretation. Work on preference data construction shows that naively choosing the highest-reward sample as positive and the lowest-reward sample as negative can become harmful as sample counts grow, and that moderately bad negatives near $\mu-2\sigma$ may be preferable to extreme outliers (Xiao et al., 24 Feb 2025). A plausible implication for SeDPO is that style negatives should be hard but not pathological: if they are too poor, the model may learn coarse quality differences instead of stylistic boundaries. Similarly, uncertainty-penalized DPO argues that ambiguous or weakly reliable pairs should receive attenuated updates (Houliston et al., 2024). That observation is directly pertinent to stylistic preference learning, since style judgments are often more subjective and context-dependent than factual or functional preferences.

Active-learning results for DPO also suggest a future direction for SeDPO. By linearizing DPO at the last layer and selecting the most informative preference pairs via D-optimal design, one can focus annotation or subset selection on the pairs that most reduce uncertainty about preference boundaries (Kveton et al., 3 Mar 2025). In a style-aware setting, this suggests selective collection of pairs that are maximally informative about fine stylistic distinctions rather than about generic response quality.

Taken together, these neighboring literatures suggest that SeDPO is best viewed as one point in a larger shift within preference optimization: from optimizing against coarse chosen/rejected differences to constructing comparison sets that isolate the attribute of interest. In SeDPO, that attribute is style, operationalized through style-cluster structure, CQSA filtering, and controlled stylistic negatives (Fan et al., 22 Sep 2025).