PodEval: Multimodal Podcast Evaluation Framework

• PodEval is a comprehensive multimodal framework for evaluating long-form podcast generation by separating content (text) from format (speech and audio).
• The framework integrates objective metrics and subjective tests, using tools like GPT-4 as a judge and real-world anchors such as the Real-Pod dataset.
• It provides actionable insights by combining reference-free textual measures with diagnostic evaluations of dialogue naturalness, loudness, and sound integration.

Searching arXiv for the PodEval paper and closely related podcast-generation work. PodEval is an open-source, comprehensive multimodal evaluation framework for podcast-like long-form audio generation. It is designed for the evaluation of open-ended podcast generation by jointly considering text, speech, and audio, and by separating Content from Format as distinct evaluation emphases. The framework addresses three problems identified as central for podcast generation: no reference standard answer, no unified evaluation metric, and uncontrollable and noisy human judgments. Its empirical design combines a real-world podcast dataset used as a human-level reference, objective metrics, model-based judgments, and structured subjective listening tests (Xiao et al., 1 Oct 2025).

1. Problem setting and methodological rationale

PodEval targets open-ended long-form podcast audio generation, not podcast understanding or short-form speech synthesis. The task is treated as unusually difficult to evaluate because it is simultaneously open-ended, long duration, multi-speaker, multimodal, and highly sensitive to subjective variability in listener ratings. In this setting, conventional reference-based text metrics are inadequate because a generated podcast can be high quality while being entirely different from any existing real episode. Standard speech metrics are also incomplete because they often emphasize sentence-level TTS rather than long-form dialogue, and standard audio metrics do not directly capture whether music and sound support or distract from the listening experience (Xiao et al., 1 Oct 2025).

The framework’s central methodological move is to decompose evaluation into text, speech, and audio. Text is treated as the primary carrier of Content, meaning the ideas, perspectives, informativeness, topical relevance, and richness of discussion. Speech and audio are treated as the primary carriers of Format, meaning the naturalness of spoken dialogue, speaker role distinction, expressiveness, loudness balance, use of music and sound effects, and the overall listening experience. This separation is not presented as absolute; the paper notes that the dimensions overlap, but the decomposition provides a practical evaluation structure.

A recurrent theme in the framework is that podcast evaluation should not collapse into a single scalar. Instead, PodEval treats podcast quality as a structured object whose components must be diagnosed separately. A plausible implication is that the framework is intended as much for system analysis as for ranking.

2. Real-Pod and the use of human-made podcasts as reference

PodEval introduces Real-Pod, a curated dataset of human-made real-world podcasts used as a reference for human-level creative quality, rather than as a gold target. The paper is explicit that Real-Pod is a reference, not an absolute answer. This distinction is fundamental: PodEval does not assume that a generated podcast should match a particular real episode. Instead, Real-Pod provides an anchor for what human-produced quality looks like across many topics and formats (Xiao et al., 1 Oct 2025).

Real-Pod is constructed in three stages. First, the dataset adopts the taxonomy of Apple Podcasts, using 17 categories. Second, for each category, GPT-4 is used to generate 5 popular and distinct topics, after which the topics are manually reviewed and refined; 3 representative topics per category are retained, producing 51 topics in total. Third, episodes are manually searched and screened according to topic relevance and rich format, with preference for multi-speaker conversation, background music and sound effects, and high audio quality.

The categories shown in the evaluation materials include Fiction, Education, Business, True Crime, Health & Fitness, Sports, Comedy, History, News, TV & Film, Society & Culture, Arts, Leisure, Music, Kids, Mental Health, and Science & Tech. The repository provides the Real-Pod dataset, topic and category metadata, and public download links with a download toolkit rather than redistributing audio directly.

Real-Pod is used in four distinct ways: as a human-level quality reference for text, speech, and audio comparisons; as the high-quality anchor (HQ) in subjective speech evaluation; as one of the systems compared in the experiments; and as a source of topic diversity for test design. This makes Real-Pod both a dataset and a calibration instrument.

3. Multimodal structure and the Content–Format distinction

The operational structure of PodEval is organized around three modalities and two evaluation emphases.

Dimension	Primary emphasis	Representative methods
Text	Content	Distinct-2, Sem-Div, MATTR, Info-Dens, GPT-4 as judge
Speech	Format	WER, DNSMOS, SIM, SPTD, Dialogue Naturalness Evaluation
Audio	Format	Loudness, SMR, CASP, questionnaire-based MOS

Within this structure, Content denotes what is being said: ideas, informativeness, topical relevance, and discussion richness. Format denotes how it is delivered: dialogue naturalness, speaker differentiation, expressiveness, loudness balance, and integration of music and sound effects. The framework therefore treats transcript quality and performed audio quality as related but non-identical objects.

This design has two important consequences. First, systems that only emit scripts can still be evaluated on the text axis. Second, systems with stronger delivery but weaker informational structure can be distinguished from systems with the opposite profile. The paper’s later results rely heavily on this decomposition, especially in showing that some systems are strong in script planning but weak in dialogue naturalness, while others are competitive in audio presentation.

4. Text evaluation and content assessment

The text component treats the conversation script or transcript as the core representation of podcast content. Because podcast generation is open-ended, PodEval rejects reference-dependent text metrics and instead uses reference-free quantitative metrics together with LLM-as-a-Judge evaluation. The script evaluation follows the approach from PodAgent (Xiao et al., 1 Oct 2025).

The four quantitative text metrics are Distinct-2, Semantic-Div (Sem-Div), MATTR, and Info-Dens. They are intended to measure lexical diversity, semantic richness or diversity, vocabulary richness, and information density, respectively. The paper reports overall averages across the 17 categories for four systems that provide scripts: GPT-4, PodAgent, MoonCast, and Real-Pod. The reported averages are:

• GPT-4: Distinct-2 0.9619, Info-Dens 6.4507, Sem-Div 0.1293, MATTR 0.6914
• PodAgent: Distinct-2 0.9741, Info-Dens 7.1767, Sem-Div 0.1372, MATTR 0.7216
• MoonCast: Distinct-2 0.9128, Info-Dens 6.0935, Sem-Div 0.1326, MATTR 0.6323
• Real-Pod: Distinct-2 0.9200, Info-Dens 8.1168, Sem-Div 0.1677, MATTR 0.6251

The interpretation given in the paper is comparatively specific. PodAgent outperforms directly prompted GPT-4 on every quantitative metric. Real-Pod scores lower on lexical-diversity-style metrics such as Distinct-2 and MATTR but higher on Sem-Div and Info-Dens; this is interpreted as reflecting more filler words, simpler language, and at the same time richer and longer discussions with more information. MoonCast, described as a spontaneous version of PodAgent, reduces lexical diversity and information density while keeping semantic diversity relatively close to PodAgent.

For more holistic script assessment, PodEval uses GPT-4 as judge under the criteria of coherence, engagingness, diversity, informativeness, and speaker diversity or speaker difference. The setup is comparative and evidence-based, with PodAgent vs GPT-4, score range -3 to 3, and positive values favoring PodAgent. The overall results are all positive: 0.7059 for coherence, 1.0294 for engagingness, 1.1765 for diversity, 1.6078 for informativeness, 1.0637 for speaker difference, and 1.3064 overall. The framework thus treats text evaluation as both property-based and judgment-based, with the second component intended to capture aspects that the first cannot.

5. Speech evaluation and dialogue naturalness

Speech is treated as the core delivery medium of podcasts, and the paper emphasizes that podcast speech evaluation differs from standard TTS evaluation because it requires dialogue-level naturalness rather than only sentence-level pronunciation accuracy. PodEval therefore combines objective metrics with a dedicated Dialogue Naturalness Evaluation (Xiao et al., 1 Oct 2025).

The objective speech metrics are WER, DNSMOS, SIM, and SPTD. WER, computed with Whisper, is used to measure pronunciation accuracy and robustness, particularly for insertion, repetition, or truncation failures. DNSMOS supplies SIG, BAK, OVRL, and P808_MOS, where SIG, BAK, and OVRL are trained according to P.835, and P808_MOS according to P.808. SIM measures speaker similarity using the cosine similarity of extracted speaker embeddings and is tied to the reference voice selected by PodAgent for the topic. SPTD, introduced by the authors, measures Speaker Timbre Difference:

$$\text{SPTD} = 1 - \frac{2}{N (N-1)} \sum_{i=1}^{N} \sum_{j=i+1}^{N} \text{sim}(\mathbf{e}_i, \mathbf{e}_j)$$

where $N$ is the number of distinct speakers and $\mathbf{e}_i$ is the speaker embedding for speaker $i$. Higher SPTD means greater timbral separation among speakers.

The subjective speech test is based on MUSHRA and is designed around five explicit difficulties: long duration, difficulty of multi-system comparison, the need to focus listeners on dialogue naturalness rather than content, balancing topic diversity against limited test time, and unreliable crowd responses. The test includes a high-quality anchor from Real-Pod and a low-quality anchor synthesized with eSpeak. PodEval automatically extracts dialogue segments with turn-taking, typically 15–25 seconds long, sampled across all 17 categories, with total test duration kept within 30 minutes. Samples from different systems are shown on the same page, and listeners rate them on a 0–100 slider divided into five stages with clear definitions.

Participants were recruited on Prolific, with a request for 20 native English-speaking participants from the US/UK. Two filtering rules were enforced: over 90% of low-quality samples had to be marked as the worst, and over 50% of high-quality samples had to rank in the top-2 best. Judgers 7, 18, and 20 were excluded, leaving 17 valid submissions.

The reported speech findings are differentiated by metric. All systems except Muyan-TTS achieve WER below 20%. The SIM ranking is Muyan-TTS, MOSS-TTSD, Dia, MoonCast, PodAgent. For SPTD, among PodAgent, NotebookLM, and Real-Pod, the ranking is PodAgent, NotebookLM, Real-Pod. DNSMOS places PodAgent, MoonCast, Muyan-TTS, MOSS-TTSD, and NotebookLM at similar speech quality, while Real-Pod and Dia score worse, with the paper attributing this to music/sound removal artifacts and real recording noise for Real-Pod, and to long-form synthesis failures for Dia. In subjective dialogue naturalness, the ranking after filtering is Real-Pod first, NotebookLM second, MoonCast / MOSS-TTSD strongest among open-source systems, and PodAgent lowest among the evaluated open-source podcast systems.

6. Audio evaluation and the mixed listening experience

At the audio level, PodEval evaluates the complete mixed podcast experience, including speech, music, sound effects, and their interactions. This level uses both objective and subjective measures (Xiao et al., 1 Oct 2025).

The objective audio metrics are loudness, SMR, and CASP. Loudness is measured with standards derived from ITU-R BS.1770-4, EBU, recommendations from AES, Apple Podcasts, and Spotify, and loudness-range guidance from Netflix. PodEval defines a practical reference region rather than a single absolute target:

• LOUD-IT: $-18$ to $-14$ LUFS
• LOUD-TP: $\leq -1$ dBTP
• LOUD-RA: 4 to 18 LU

The appendix gives scoring functions that assign full score inside the target region and exponential penalties outside it, with constants $k_1 = 0.0858$, $k_2 = 0.3291$, $k_3 = 4.605$, $N$0, and $N$1. SMR measures the balance between speech and music or sound effects, with SMR > 0 treated as the minimum acceptable condition, and SMR_SCORE defined as the proportion of cases where SMR exceeds 0. CASP evaluates MSE-Speech Harmony using the CASP model, pretrained on approximately 1,000 hours of podcast data, and the score used is the DualScore.

The subjective audio test is a questionnaire-based MOS-style evaluation. Rather than presenting full episodes, PodEval concatenates the first minute, middle minute, and final minute of each podcast, separated by a beep signal. The questionnaire contains 8 questions, combining perceptual questions such as “Information Delivery Effectiveness” with preference-based questions such as “Speaker Expression Preference,” and also asks about willingness to listen to the full episode and perceived human-likeness. Two validity safeguards are built into the design: attention-check questions, including “How many speakers are there?” and a music-related instruction question, and mandatory written justification for every score.

A pilot study preceded the final audio test. Based on feedback, the scale was reduced from 10 points to 5 points, definitions were clarified, the separation between perceptual and preference-based questions was refined, and justification requirements were added. The reported pass rate improved from 75% to 90%. The final MOS evaluation uses PodAgent, MOSS-TTSD, NotebookLM, and Real-Pod, with each sample around 3 minutes, each group containing 4 podcast samples from the same category, and an average completion time of about 24 minutes.

The objective loudness results show strong system-specific differences. For example, Dia and MoonCast align well with the target LOUD-IT range; NotebookLM is centered around about -25 LUFS; Muyan-TTS is best on LOUD-TP; and Real-Pod has highly scattered loudness distributions. The reported loudness scores are: Real-Pod 0.72 / 0.53 / 0.82, PodAgent 0.80 / 0.32 / 1.00, MoonCast 1.00 / 0.01 / 0.68, Muyan-TTS 0.88 / 1.00 / 0.83, Dia 0.98 / 0.01 / 0.95, MOSS-TTSD 0.88 / 0.02 / 0.99, and NotebookLM 0.51 / 0.56 / 1.00 for LOUD_IT_SCORE / LOUD_TP_SCORE / LOUD_LRA_SCORE. For systems with music and sound, SMR_BASIC_SCORE is 0.99 for Real-Pod and 1.00 for PodAgent, while CASP is 0.58 for Real-Pod and 0.56 for PodAgent.

At the subjective audio level, Real-Pod is reported as best on 5 of 7 metrics, especially Engagement Level and Human Likelihood. NotebookLM performs strongly and is at times close to or better than Real-Pod on production-oriented metrics, including Audio Quality. PodAgent performs poorly in audio MOS because its underlying speech is less natural in dialogue settings, despite relatively competitive music and sound integration.

7. Validation, system-level findings, and limitations

PodEval includes several explicit mechanisms for validation. In speech evaluation, the anchor-based MUSHRA design allows direct filtering of unreliable raters. In audio evaluation, the pilot-to-final redesign raised the pass rate from 75% to 90%. The paper also argues that the measured rankings agree with known system properties: PodAgent is strong in text and speaker distinction but weak in dialogue naturalness because its backend, CosyVoice2, is not designed for dialogue-level generation; NotebookLM is strong in naturalness; Real-Pod remains strongest in realism and engagement; and AI systems can outperform human-made podcasts on certain controllable production properties such as loudness consistency (Xiao et al., 1 Oct 2025).

The aggregate findings are clear. Human-made podcasts remain the strongest overall, especially in dialogue naturalness, engagement, human-likeness, and the holistic listening experience. Text generation appears more mature than speech and audio realism, as shown by PodAgent’s strong script results. Dialogue-level TTS is identified as a central bottleneck, since systems built for direct dialogue synthesis, such as MoonCast and MOSS-TTSD, outperform pipelines whose backend is optimized for single-sentence synthesis. At the same time, AI systems can be stronger on some standardized production metrics, and PodAgent’s CASP and SMR results suggest that music and sound integration is already practically viable, even if it does not compensate for unnatural dialogue.

The framework also identifies several limitations. Real-Pod is a reference, not a ground truth. Topic interest can confound subjective judgments, especially for Full Episode Willingness. Some metrics apply only to certain systems, such as text metrics for systems that emit scripts and SMR/CASP for systems with music or sound effects. PodEval also depends on GPT-4 in multiple places, including topic initialization, text judging, and summarization of written justifications, which introduces reliance on a proprietary evaluator. The ethics statement adds that Real-Pod uses publicly available podcasts under fair-use considerations, that audio is not directly redistributed, that crowd workers are paid above platform minimum wage, and that diverse topics and evaluators are used to mitigate bias.

Taken together, PodEval formalizes podcast evaluation as a structured multimodal problem rather than a single-score benchmark. Its most distinctive contribution is not merely the collection of metrics, but the integration of reference-free textual measures, speech and audio diagnostics, human-level reference data, and attention-controlled subjective protocols into one evaluation framework for long-form open-ended podcast generation.