Context-Aware Post Rewriter

• Context-Aware Post Rewriter is a neural system that leverages session graphs, hierarchical pooling, and dual-encoder architectures to transform posts with enhanced clarity and style.
• It integrates dialogue history, user intent, and semantic tagging to disambiguate ambiguous inputs and enforce factual, stylistic, and discourse coherence.
• Empirical results demonstrate significant performance gains in metrics like MRR, BLEU, and ROUGE, underscoring its effectiveness across conversational search and content moderation tasks.

A context-aware post rewriter is a neural system designed to transform user-generated posts or utterances into contextually appropriate, clarified, or stylistically modified versions, leveraging discourse, conversational, or sequential context beyond isolated sentence-level rewriting. This class of systems has become central in conversational search, dialogue systems, information retrieval, and content moderation, where context-dependent ambiguities and stylistic requirements demand precise, contextually coherent outputs. Context-aware post rewriters systematically encode session or dialogue history, user intent, and content features, employing architectural innovations such as session graphs, group tags, hierarchical context pooling, or dual-encoder mechanisms to fuse and exploit context.

1. Motivation and Problem Scope

Conventional sentence-level rewriting—whether for query clarification, style transfer, detoxification, or summarization—frequently operates in a myopic setting, leading to generic, ambiguous, or incoherent rewrites that degrade end-task performance and user experience (Yerukola et al., 2023). In domains such as e-commerce search, conversational AI, and social media, individual inputs are short and under-specified, failing to reveal true user intent or to resolve coreference and omission phenomena. Context-aware post rewriters address these challenges by:

• Integrating history queries, prior dialogue turns, or document-level context to disambiguate intent and fill in omitted or elliptical content (Zuo et al., 2022, Jin et al., 2022).
• Enforcing stylistic, factual, or discourse coherence in multi-turn or multi-post threads (Yerukola et al., 2023, Bao et al., 2022, Bao et al., 2021).
• Enhancing downstream performance in retrieval, generation, or evaluation, as measured by task-appropriate metrics such as MRR, BLEU, or contextual fit.

These methods presuppose both the existence of rich session or context data and the necessity of moving beyond pointwise or turnwise modeling.

2. Model Architectures and Context Integration

2.1 Session Graphs and Bipartite Attention

In e-commerce search and query rewriting, session graphs formalize the contextual structure of a user’s query history. Each graph comprises:

• Query nodes $Q_i$ (each query in the session)
• Token nodes $T_j$ (each distinct token across all queries)
• Undirected edges $(Q_i, T_j)$ if $T_j$ occurs in $Q_i$

Node representations are initialized via a Transformer encoder (<boq> token output for queries, learned token embeddings for tokens). Bipartite multi-head graph attention networks (GATs) propagate information between queries and tokens using stacked, alternated $K$-rounds of GAT$_{\text{q}\to\text{t}}$ and GAT$_{\text{t}\to\text{q}}$ updates. This approach enables the model to capture and diffuse cross-query intent, supporting more effective context aggregation before rewriting (Zuo et al., 2022).

2.2 Hierarchical and Dual-Encoder Architectures

General contextualized rewriting for summarization and dialogue employs the following:

• Hierarchical context encoders: Each preceding post, utterance, or context block $c_j$ is mapped to an embedding $h_j = E_1(c_j)$, which is then pooled (by LSTM, mean, or attention) into an aggregate context $\bar{h}_C$ (Yerukola et al., 2023, Bao et al., 2022, Bao et al., 2021).
• Source/post encoders: The current source sentence or query is encoded into hidden states $H_I$.
• Decoder fusion: At each decode step, the aggregate context $\bar{h}_C$ is fused with source features—by concatenation, gating, or cross-attention—to inform the output token distribution (Yerukola et al., 2023).

In multi-source post-editing, a dual-encoder Transformer integrates both the source input and the system-generated candidate via separate encoders. The machine translation (mt) encoder self-attends and then cross-attends the source encoder output, producing context-fused representations before decoding the post-edited sequence (Lee et al., 2019).

2.3 Specialized Tagging and Structured Fusion

Hierarchical context tagging (HCT) for utterance rewriting predicts at each position:

• An edit action (KEEP or DELETE)
• A slotted rule $r_i$ (e.g., “besides [SL$_0$]”) with arbitrary function words and $k$ slots
• The spans from context to fill each slot (autoregressively predicted)

This formulation enables more expressive rewriting, including out-of-context token insertion and multi-span grouping, while bounding the search space through rule clustering (Jin et al., 2022).

Semantic role labeling (SRL)-guided rewriters augment input with predicate-argument triples extracted from the context, focusing attention via sparse masking to highlight core “who did what to whom” semantics, thereby suppressing distraint from irrelevant context and improving faithfulness in rewrite (Xu et al., 2020).

3. Training Objectives, Losses, and Optimization

Most context-aware post rewriters are trained with classical maximum-likelihood cross-entropy (token-level or sequence-level), optimizing $-\sum_t \log P(y_t|y_{<t}, \text{context})$ on parallel or pseudo-parallel rewriting data (Zuo et al., 2022, Yerukola et al., 2023, Bao et al., 2022, Bao et al., 2021, Lee et al., 2019). Domain-specific variants include:

• Multi-task losses: Adding style classification loss for contextual style transfer, balancing $\mathcal{L}_{\text{CE}}$ and $-\log P(\text{style}_1\,|\,X)$ with hyperparameter $\lambda_{\text{style}}$ (Yerukola et al., 2023).
• Reinforcement learning (RL): Policy-gradient (REINFORCE) objectives leverage sequence-level rewards such as BLEU or downstream retrieval/generation accuracy to optimize for fluency or end-task utility (Zhou et al., 2019, Jin et al., 2022). Typically the combined RL + MLE loss takes the form $$\mathcal{L} = (1-\lambda)\,\mathcal{L}_e + \lambda\,\mathcal{L}_r$$.
• Knowledge distillation: For LLM-based rewriting, a lightweight student (e.g., T5-base) distills from LLM-generated pseudo-labels via mixed cross-entropy and KL divergence loss, $$L = \alpha\,L_{\mathrm{CE}} + (1-\alpha)\,L_{\mathrm{KD}}$$ (Ye et al., 2023).

Auxiliary mechanisms such as copy modes, pointer networks, or group/tag embeddings are integrated where required by the architectural substrate.

4. Evaluation Metrics and Empirical Findings

Proper evaluation of context-aware post rewriters extends beyond n-gram overlap to metrics of semantic and contextual fit.

• Standard metrics: BLEU and ROUGE for summarization and rewriting fidelity (Bao et al., 2021, Bao et al., 2022, Jin et al., 2022, Xu et al., 2020).
• Context-sensitive metrics: CtxSimFit combines BERTScore and BERT Next Sentence Prediction probability to jointly reward semantic similarity and contextual cohesion. At $\alpha=0.5$, Spearman $\rho\approx0.85$ with human ratings, far above non-contextual metrics ($\rho=0-0.3$) (Yerukola et al., 2023).
• Retrieval-centric metrics: MRR, MAP, Recall@10, and NDCG@k for end-to-end search or QA performance, where context-aware LLM pipelines yield +9.6 MRR over human rewrites with BM25, and distilled students recover performance at $\sim6\times$ lower latency (Ye et al., 2023).
• Domain-generalization: HCT and graph-based architectures show +2 BLEU gains and increased ROUGE/EM, particularly in scenarios with out-of-vocabulary insertions or multi-span reconstructions (Jin et al., 2022, Zuo et al., 2022).

Human evaluations are crucial, revealing strong preferences for contextually fitting, natural rewrites versus overly generic, sentence-bound outputs.

5. Practical Design and Implementation Considerations

5.1 Architecture and Context Encoding

• Graph-based rewriting: Suitable for settings with explicit session/query structures (e.g., e-commerce search), supporting fine-grained propagation of user intent (Zuo et al., 2022).
• Hierarchical encoders: Preferable for open-domain dialogue, stylistic editing, or summarization, with context pooling depth tuned to 1–3 turns for relevance and scalability (Yerukola et al., 2023, Bao et al., 2022).
• Dual-encoder fusion: Essential for post-editing and cases requiring alignment between two input sequences (e.g., grammar correction, MT APE) (Lee et al., 2019).

5.2 Prompt and Rule Engineering

• LLM-based two-stage pipelines (rewrite then edit) with explicit prompting and curated demonstrations ensure rewrite correctness, clarity, informativeness, and non-redundancy (Ye et al., 2023).
• Rule extraction and clustering, as in HCT, are critical to limit the rule space and address compositional insertion problems endemic in span-only approaches (Jin et al., 2022).

5.3 Training Strategies

• Regularization: Dropout, label smoothing, and word dropout are employed to manage noise, especially with user-generated content (Bao et al., 2021).
• Auxiliary annotation: For SRL-guided and group-tagged models, high-quality and possibly cross-turn annotated datasets are mandatory for effective context modeling (Xu et al., 2020, Bao et al., 2021).

5.4 Evaluation

• Context-infused metric reporting is obligatory, with human “overall fit” correlation analysis validating setup (Yerukola et al., 2023).
• For task deployment, fallback to sentence-level rewrites in the absence of reliable context is recommended, with explicit confidence reduction (Yerukola et al., 2023).

6. Empirical Impact and Comparative Results

Context-aware post rewriters have demonstrably advanced state-of-the-art across multiple tasks:

• E-commerce query rewriting: +11.6% MRR, +20.1% Hit@16 over strong Transformer baselines by contextual session modeling (Zuo et al., 2022).
• Stylistic rewriting and social media: Contextual rewriters preferred by annotators at 50% versus 20–30% for non-contextual, with CtxSimFit metrics achieving $\rho=0.85$ with human fit (Yerukola et al., 2023).
• Conversational search: LLM rewrite–then–edit pipeline adds ~9.6 MRR over human rewrites for BM25 retrieval; distilled students maintain gains at an order of magnitude lower compute (Ye et al., 2023).
• Summarization: Group-tag contextualized models (BART-JointSR) outperform RL/copy-based non-contextual baselines by $+0.5$ to $+1.5$ ROUGE, with increased compression and cross-sentence coherence (Bao et al., 2022, Bao et al., 2021).
• Dialogue rewriting: HCT yields $+1.9$ BLEU-4, $+4.5$ ROUGE-L on CANARD vs. prior systems, and robust transfer across domains (Jin et al., 2022).

A plausible implication is that context encoding—when architecturally and statistically optimized—consistently yields non-trivial advances in both rewriting accuracy and downstream application performance, regardless of base model or domain.

7. Limitations, Generalization, and Future Directions

While context-aware post rewriters demonstrate marked improvements, several limitations and open challenges remain:

• Model capacity and computational overhead: Graph-based and dual-encoder architectures induce greater memory and compute requirements, necessitating lightweight distillation for production systems (Ye et al., 2023).
• Data requirements: Effective context modeling relies on abundant, high-quality parallel or pseudo-parallel data covering realistic context distributions, especially with annotation-intensive components (e.g., SRL, group-tagging).
• Robustness and domain adaptation: Noisy or missing context degrades output confidence; semi-supervised and transfer learning strategies may be required to maintain robustness in new settings (Bao et al., 2021, Zhou et al., 2019).
• Evaluation: Fully capturing human-perceived “fit” remains an unsolved problem in absence of large-scale context-sensitive reference sets, despite advances with metrics such as CtxSimFit (Yerukola et al., 2023).

Research continues into syntax- and semantics-aware fusion, more expressive rule or graph representations, and scalable self-supervised objectives capable of handling rapidly evolving user-generated content.