AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs

Abstract: Recent advancements in LLMs have improved their ability to process extended conversational contexts, yet fine-tuning and evaluating short- and long-term memories remain difficult due to the absence of datasets that encode both short- and long-term conversational history. Existing conversational datasets lack memory grounding, overlook topic continuity, or rely on costly human annotation. To address these gaps, we introduce AgenticAI-DialogGen, a modular agent-based framework that generates persona-grounded and topic-guided conversations without human supervision. The framework uses LLM agents to extract knowledge graphs, identify topics, build speaker personas, and simulate topic-guided conversations from unstructured conversations. A QA module generates memory-grounded Question Answer (QA) pairs drawn from short- and long-term conversational histories. We also generated a new dataset entitled, TopicGuidedChat (TGC), where long-term memory is encoded as speaker-specific knowledge graphs and short-term memory as newly generated topic-guided conversations. Evaluations depict that AgenticAI-DialogGen yields higher conversational quality and LLMs fine-tuned on TGC dataset achieve improved performance on memory-grounded QA tasks.

• The paper introduces a modular agent-based framework that automates the generation of memory-grounded, topic-guided conversation datasets.
• It employs multi-stage processing including knowledge extraction, topic clustering, and persona-driven simulation for robust QA and memory probing.
• Empirical results demonstrate notable gains in dialogue coherence, topic consistency, and retrieval performance compared to legacy corpora.

AgenticAI-DialogGen: A Modular Topic-Guided Framework for Memory-Grounded Conversation Benchmark Creation

Motivation and Context

Despite recent progress in LLM context management, extant datasets for conversational modeling remain fundamentally limited for probing both short- and long-term memory. Existing QA datasets (e.g., CoQA, QuAC) focus on short multi-turn segments and neglect persona consistency and topic continuity. Open-domain dialog datasets (e.g., PersonaChat, DailyDialog, MSC) introduce elements of persona and session structure but still lack explicit topic-centric structure, QA supervision, and scalable memory grounding. Manual dataset curation is prohibitively expensive and insufficiently scalable for generating long-term, topic-consistent corpora. As a result, robust, realistic evaluation or fine-tuning of LLM conversational memory remains out-of-reach without significant architectural advances and automation.

AgenticAI-DialogGen: Architecture and Pipeline

AgenticAI-DialogGen introduces a comprehensive, fully automated, modular agent-based system leveraging LLMs for multi-stage synthetic dataset construction. The pipeline, instantiated on multi-session dialog corpora (primarily MSC), comprises the following stages:

• Preprocessing: Unstructured dialogues are canonicalized, speaker identities resolved, and text normalized to yield explicit turn-indexed conversational records.
• Knowledge Extraction: Joint LLM prompting extracts factual knowledge in subject–relation–object triple form, referencing source turn indices for traceability.
• Topic Extraction: Knowledge triples are clustered into coherent high-level topics via LLM prompts, with importance scoring to control downstream simulation.
• Knowledge Graph Construction: Topic-specific triples are assembled into directed MultiDiGraph knowledge graphs, supporting explicit relationship modeling and persona separation.
• Persona Generation: Speaker-wise knowledge graphs are abstracted into persona profiles, incorporating traits and topical interests detectable from prior discourse.
• Topic-Guided Conversation Simulation: LangGraph-based agents (DuelingChatAgent) interact, producing tightly turn-controlled, persona-consistent conversations on given topics, with evolving shared history context.
• Validation and Refinement: Outputs are quality- and topicality-validated. Detected issues trigger iterative refinement using LLM-provided suggestions, enforcing adherence to coherence and topical focus constraints.
• Memory-Grounded QA Generation: The QA module generates factual question-answer pairs grounded in both simulated (short-term) and underlying (long-term) contexts, probing recall at multiple abstraction levels.
• Postprocessing: All products are serialized, deduplicated, and exported in standardized schema for downstream use.

The framework is architected as model-agnostic (with GPT-4o as the main instantiation), modular with explicit error handling, and highly configurable. Each stage is independently testable and supports adaptable prompts and parameterization.

TopicGuidedChat (TGC): Dataset Design and Features

The product of the framework is the TopicGuidedChat (TGC) dataset, which fundamentally revises the structure of the underlying MSC corpus:

• Topic Organization: Conversations are organized around salient, semantically coherent topics, not just session/timeline boundaries.
• Long-Term and Short-Term Memory: For each persona pair, speaker-specific knowledge graphs constitute structured long-term memory while topic-driven simulated conversational turns provide explicit short-term context.
• Persona Grounding: Every conversation is governed by derived personas, linking discourse directly to individualized speaker-level knowledge graphs and inferred characteristics.
• QA Supervision: Each topic dialog is furnished with ~20 memory-grounded QA pairs (from either knowledge graphs or simulated discourse), supporting rigorous, fine-grained measurement of recall capabilities.
• Agent-Driven, Scalable, Human-Free Generation: The entire pipeline is synthetic and automatable, enabling efficient large-scale generation without human annotation.

Empirical Results

Data Quality Analysis

The TGC dataset demonstrates substantial improvements over the source MSC and peer datasets on both human and automatic metrics. Human evaluation yields:

• Topic consistency in TGC at 97.82% (+33.7 over MSC)
• Coherence at 94.67% (+7.5)
• Completeness and relevance at 88-94% (substantially higher than raw session data)
• Knowledge graphs in TGC are rated at 92.18% factual correctness and 90% relevance
• TGC QA pairs score 87%+ on factual correctness and relevance, confirming effective memory-grounded probing.

Automatic metrics further validate these results:

• Conversational flow (BERTScore F1): Highest for TGC with validation (e.g., 30.36 vs. 28.11 on MSC)
• Topic Consistency: Higher Self-BLEU, increased perplexity (controlled lexical variety within topic), decreased drift
• Content Similarity (ROUGE-L): Significant improvements
• Semantic Similarity & Entity Overlap: Both consistently outperform raw source datasets
• Readability: Maintained at a high level despite artificial generation

Memory-Grounded QA Evaluation

The efficacy of TGC for memory QA is demonstrated across major LLMs (GPT-4, Gemini, Claude, DeepSeek, Qwen, etc.) in both zero-shot and fine-tuned settings. Notably:

• Lightweight models fine-tuned on TGC (with structured KG memory) outperform larger zero-shot models by large margins (e.g., Mistral-7B F1 at 87.36 vs. GPT-4 zero-shot at 83.77).
• Structured memory (KG inclusion) yields a 10-15 F1 gain over unstructured or raw conversational context.
• The results decisively confirm the practical value of explicit, agent-generated KG-augmented datasets for downstream retrieval, recall, and memory probing over long conversational histories.

Theoretical and Practical Implications

The deployment of a fully agentic topic-guided generation and validation pipeline reframes the landscape of memory-grounded conversational modeling in three key ways:

1. Benchmark Advances: TGC enables systematic, large-scale, controlled evaluation/fine-tuning of LLM short-term and long-term memory, with persona and topic awareness tightly integrated—addressing a gap unfulfilled by any prior dataset.
2. Scalable Data Synthesis: The modular agent-based approach can adapt to new domains, persona schemes, or conversational styles, generalizing far beyond the MSC root data. The architecture seamlessly integrates with other conversational corpora such as PersonaChat and DailyDialog.
3. LLM Training Efficacy: The finding that weakly supervised, lightweight models outperform larger models on memory QA tasks when fine-tuned on TGC suggests systematic cost and efficiency benefits. Structured memory incorporation (KG) is not just beneficial but essential for robust recall capabilities, encouraging new architectures emphasizing structured symbolic context within neural systems.

Limitations

Inherent limitations remain regarding the factual, social, or semantic biases propagated from underlying LLMs, error propagation in the validation/refinement loop, and the computational demands of agent-based synthetic dataset construction. The resulting conversational style may be subtly shaped by prompt designs or system parameters, and the lack of human annotation (while enabling scale) restricts the representational diversity found in organic corpora.

Ethical Considerations

All generated personas and conversations are wholly synthetic; no identifying information or real-world individuals are represented. Bias mitigation is incorporated at validation, but synthetic generation and the use of preexisting LLMs do not eliminate bias risks. The dataset is intended for research, not end-user deployment, particularly in decision-critical environments.

Conclusion

AgenticAI-DialogGen offers a methodologically rigorous, scalable agent-based framework for generating topic-guided, persona-grounded dialog datasets with explicit short- and long-term memory supervision. The resultant TGC dataset surpasses legacy corpora in discourse coherence, topic alignment, and depth of memory probe, and empowers small LLMs with strong memory-aware conversational skills. This framework establishes a new standard for memory-grounded conversational AI research and performance measurement, with avenues for future work in reinforcement learning refinement and domain-adaptive topic and persona modeling.

Reference: "AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs" (2604.12179).