PodAgent: A Comprehensive Framework for Podcast Generation (2503.00455v1)

Abstract: Existing Existing automatic audio generation methods struggle to generate podcast-like audio programs effectively. The key challenges lie in in-depth content generation, appropriate and expressive voice production. This paper proposed PodAgent, a comprehensive framework for creating audio programs. PodAgent 1) generates informative topic-discussion content by designing a Host-Guest-Writer multi-agent collaboration system, 2) builds a voice pool for suitable voice-role matching and 3) utilizes LLM-enhanced speech synthesis method to generate expressive conversational speech. Given the absence of standardized evaluation criteria for podcast-like audio generation, we developed comprehensive assessment guidelines to effectively evaluate the model's performance. Experimental results demonstrate PodAgent's effectiveness, significantly surpassing direct GPT-4 generation in topic-discussion dialogue content, achieving an 87.4% voice-matching accuracy, and producing more expressive speech through LLM-guided synthesis. Demo page: https://podcast-agent.github.io/demo/. Source code: https://github.com/yujxx/PodAgent.

PodAgent: A Comprehensive Framework for Podcast Generation

The paper introduces PodAgent, a novel framework designed to generate complete and informative podcast audio programs. PodAgent addresses the limitations of existing automated methods that struggle with content depth, coherent dialogue generation, expressive speech synthesis, and suitable voice representation. This framework innovatively integrates a multi-agent system, voice-role matching, and LLM-enhanced speech synthesis, significantly enhancing the quality and professionalism of generated podcast content.

Core Components of PodAgent

1. Host-Guest-Writer System: The multi-agent collaboration within PodAgent is a cornerstone of its architecture. It organizes content creation using a Host-Guest-Writer paradigm:
   • Host-Agent: Responsible for setting interview outlines and expert guest roles aligned with the podcast topic.
   • Guest-Agents: Specialize in providing diverse insights corresponding to their assigned roles.
   • Writer-Agent: Synthesizes responses into coherent scripts, ensuring a natural, engaging flow and eliminating redundancy.
2. Voice-Role Matching: PodAgent constructs a diverse voice pool, allowing dynamic matching of voices to roles and maintaining consistency across topics. This matching is based on a comprehensive analysis of voice characteristics harnessing LLM capabilities, achieving an impressive 87.4% accuracy in experiments.
3. LLM-Guided Speech Synthesis: Leveraging open-source TTS models, PodAgent incorporates LLM-predicted speaking styles into its synthesis processes, enhancing speech expressiveness. This approach not only improves the prosodic and emotional quality of the output but also ensures alignment with the content's intent.

Evaluation and Results

Given the absence of standardized criteria in podcast generation, PodAgent introduces a comprehensive evaluation methodology, engaging both quantitative metrics and qualitative LLM-based assessments:

• Quantitative analysis involves lexical diversity, semantic richness, and information density, showcasing PodAgent's robustness in generating detailed conversation scripts over traditional models.
• Using LLMs as judges for qualitative assessment provided insights into coherence, engagement, diversity, and overall effectiveness. PodAgent consistently surpassed baseline models directly using GPT-4.

Implications and Future Directions

PodAgent's framework demonstrates significant potential in automating podcast generation with professional quality, serving as a benchmark for future developments in AI-driven multimedia content creation. The implications are vast, extending to educational, informational, and entertainment audio productions.

Looking ahead, further research could address the limitations in voice quality and diversity by expanding the voice database and exploring synthetic voice generation. Additionally, enhancing the nuanced incorporation of sound effects and music could deliver even more immersive audio experiences. The integration of more complex conversational dynamics, such as non-verbal cues, will also contribute to realism and listener engagement, promoting PodAgent as a leading tool in AI-generated audio content creation.