Masked Dialogue Generation (DialGen)

• The paper introduces DialGen, which reconstructs masked dialogue by jointly modeling token- and character-level contexts to ensure narrative coherence.
• It uses a Chinese BART_base backbone with a dedicated bi-directional Character Encoder to accurately attribute speakers and preserve story dynamics.
• Empirical results show improved BLEU scores, diversity, and human-judged coherence and informativeness compared to baseline models.

Masked Dialogue Generation (DialGen) is an approach developed to evaluate and advance machine understanding and generation of coherent, contextually grounded dialogue within narrative stories. Unlike traditional dialogue modeling in chitchat or task-oriented scenarios, DialGen is situated within stories that intrinsically encode complex character relationships and evolving interactive contexts, exposing unique challenges for discourse coherence, speaker differentiation, and narrative consistency (Yao et al., 2022). The principal objective is to automatically generate full dialogue turns that have been masked out of a narrative, such that the generated content is simultaneously fluent, contextually appropriate, informative, and coherent with existing story structure. Masked Dialogue Generation thus directly tests a model's ability to model global story-level context and dynamics between explicitly represented characters.

1. Formal Task Definition

In the Masked Dialogue Generation task, the input is a partially masked story represented as a sequence of tokens $S = (s_1, s_2, ..., s_T)$, where certain dialogue turns are replaced by a designated “[MASK]” placeholder. The model is required to output a concatenated token sequence $D = (d_1, d_2, ..., d_N)$ reconstructing the masked dialogue in order. At each step $n$, the output token generation probability is defined as:

$$P(d_n | d_{<n}) = \mathrm{softmax}(W \cdot f([H_n; C_{i_n}]) + b)$$

where $H_n \in \mathbb{R}^d$ is the decoder hidden state, $C_{i_n} \in \mathbb{R}^d$ is the representation of the character speaking token $d_n$, and $f(\cdot)$ implements SiLU activation followed by layer normalization. The model selects the speaking character for each generated token by computing

$$i_n = \arg\max_{i=1..K}(\hat{H}_n \cdot C_i),\quad \hat{H}_n = W_c H_n + b_c$$

and optimizes the negative log-likelihood loss:

$$\mathcal{L}_\text{DialGen} = -\sum_{n=1}^N \log P(d_n | d_{<n})$$

This explicit joint modeling of token- and character-level representations is designed to encode both textual and relational context crucial to narrative dialogue (Yao et al., 2022).

2. Dataset Construction and Statistics

The principal benchmark for Masked Dialogue Generation is DialStory, a large-scale dataset comprising 105,000 Chinese short-story chapters drawn from classical fairy tales and fiction. Only chapters with ten or more dialogue turns are included, with dialogue segments identified via quotation marks. Names are automatically extracted using a named-entity recognizer, and candidate character sets for each story are constructed. Heuristic speaker attributions are assigned via sentence-level analysis, with a manual verification on 150 chapters yielding 96.2% accuracy. Dialogue tokens constitute 30–50% of each story, enforcing a high density of conversational content. Key average metrics are summarized below:

Metric	Value
Total tokens	451.98
Sentences	22.14
Dialogue turns	12.78
Dialogue tokens/turn	20.68
Distinct characters	3.54

For the DialGen task, the dataset is split into 100k/2.5k/2.5k train/val/test, with input sequences containing on average ≈454 tokens and ≈10.4 dialogue turns per story (Yao et al., 2022).

3. Model Architecture

The baseline and proposed DialGen models are built on a Chinese BART_base backbone (6-layer encoder, 6-layer decoder, 768 hidden units, ≈116M parameters). Character representations are generated by a dedicated bi-directional Transformer-based Character Encoder, which pools encoder hidden states associated with character name mentions via mean-pooling:

$$C_i = \mathrm{Pool}(\text{CharEnc}(\{h_k \mid s_k \text{ mentions } C_i\}))$$

The DialGen decoder, at each generation step, (1) reads $H_n$ from BART, (2) selects the current character $C_{i_n}$ via cosine-style argmax, and (3) predicts the next token conditioned on $[H_n; C_{i_n}]$. Implementation includes SiLU activation, layer normalization, and trainable projection matrices. This architecture enables explicit modeling of character roles and their narrative evolution, capturing the discourse-level constraints unique to story dialogue.

4. Training Regime and Input/Output Formatting

The models are initialized from pretrained Chinese BART (CPT) and optimized with Adam, following BART's gradient clipping and scheduler. For DialGen, the model is trained for ~5.6 epochs at a learning rate of $1\times10^{-4}$ and batch-size matching BART_base. The input format replaces each masked dialogue turn with “[MASK]”; the decoder emits the concatenated sequence for all masked turns. Tokenization employs standard Chinese subwords. Baselines include BERT_base, RoBERTa, and MacBERT encoder-only architectures. Output is evaluated both as continuous token sequences and in terms of character selection at each dialogue step (Yao et al., 2022).

5. Evaluation Protocols and Empirical Results

Evaluation of DialGen employs both automatic and manual metrics:

• Automatic: BLEU-1/2 for n-gram overlap, Distinct-2/3/4 for intra-response diversity, and a BERT-based classifier for coherence (measuring the percentage of outputs classified as "coherent").
• Manual: 100 test stories are assessed by three annotators for fluency, coherence (on a 0–2 per turn, summed), and informativeness; outputs are compared pairwise against the BART baseline with Fleiss' κ for inter-annotator agreement.

Results on the test set are summarized:

Model	#Params	BLEU-1	BLEU-2	Dist-2	Dist-3	Dist-4
BART	116 M	21.04	7.85	6.67	20.57	33.78
Ours	125 M	23.88	8.47	7.72	25.35	43.10

Coherence is maintained at higher levels than BART as the number of masked turns increases. Human evaluation indicates statistically significant wins in coherence (39.9% vs. 15.0%, $p<0.01$) and informativeness (41.0% vs. 17.0%, $p<0.05$), but not in fluency.

6. Qualitative Analysis and Error Taxonomy

Qualitative analysis highlights that the explicit character-aware model is able to generate contextually specific, non-generic replies and to maintain correct speaker role assignments, differentiating, for example, between familial relationships that BART fails to resolve. Observed errors in generated dialogue are categorized as:

• Inter-sentence contradiction (4.85%)
• Inter-sentence repetition (16.50%)
• Intra-sentence contradiction (1.94%)
• Intra-sentence repetition (2.91%)

The majority of errors are of the inter-sentence repetition type, indicating challenges in modeling long-range context and avoiding redundancy (Yao et al., 2022).

7. Connections to Related Methodologies

Masked Dialogue Generation shares foundational principles with DialogBERT's masked utterance regression and utterance order ranking approach (Gu et al., 2020). While DialGen models operate at the token- and character-level within narrative contexts, DialogBERT extends BERT’s pretraining to dialogue by masking and regressing whole utterances and by explicitly learning discourse order. DialogBERT uses a hierarchical Transformer, first encoding utterances, then passing utterance embeddings through a context encoder, and decoding responses while attending to context-level embeddings. Its training involves the following:

• Masked Utterance Regression (MUR): Masking 10–15% of utterances and regressing to their embeddings.
• Distributed Utterance Order Ranking (DUOR): Ranking utterances in correct sequential order.
• Combined loss: $$\mathcal{L} = \mathcal{L}_\mathrm{gen} + \alpha \mathcal{L}_\mathrm{mask} + \beta \mathcal{L}_\mathrm{rank}$$ with $\alpha=\beta=1$.

Empirically, DialogBERT outperforms BART and DialoGPT in perplexity, BLEU-4, NIST, as well as in human judgments for coherence, informativeness, and human-likeness, suggesting the importance of masked, discourse-aware objectives for high-level dialogue modeling (Gu et al., 2020). This suggests that Masked Dialogue Generation and its variants contribute crucial advances in the effective modeling of discourse-level coherence, speaker identity, and narrative structure in data-driven dialogue systems.