---
title: 'PodEval: Multimodal Podcast Evaluation Framework'
url: https://www.emergentmind.com/topics/podeval
type: topic
---

# PodEval: Multimodal Podcast Evaluation Framework

Searching arXiv for the PodEval paper and closely related podcast-generation work.
PodEval is an open-source, comprehensive multimodal evaluation framework for podcast-like long-form audio generation. It is designed for the evaluation of open-ended podcast generation by jointly considering text, speech, and audio, and by separating **Content** from **Format** as distinct evaluation emphases. The framework addresses three problems identified as central for podcast generation: **no reference standard answer**, **no unified evaluation metric**, and **uncontrollable and noisy human judgments**. Its empirical design combines a real-world podcast dataset used as a human-level reference, objective metrics, model-based judgments, and structured subjective listening tests [2510.00485].

## 1. Problem setting and methodological rationale

PodEval targets **open-ended long-form podcast audio generation**, not podcast understanding or short-form speech synthesis. The task is treated as unusually difficult to evaluate because it is simultaneously **open-ended**, **long duration**, **multi-speaker**, **multimodal**, and highly sensitive to **subjective variability** in listener ratings. In this setting, conventional reference-based text metrics are inadequate because a generated podcast can be high quality while being entirely different from any existing real episode. Standard speech metrics are also incomplete because they often emphasize sentence-level TTS rather than long-form dialogue, and standard audio metrics do not directly capture whether music and sound support or distract from the listening experience [2510.00485].

The framework’s central methodological move is to decompose evaluation into **text**, **speech**, and **audio**. Text is treated as the primary carrier of **Content**, meaning the ideas, perspectives, informativeness, topical relevance, and richness of discussion. Speech and audio are treated as the primary carriers of **Format**, meaning the naturalness of spoken dialogue, speaker role distinction, expressiveness, loudness balance, use of music and sound effects, and the overall listening experience. This separation is not presented as absolute; the paper notes that the dimensions overlap, but the decomposition provides a practical evaluation structure.

A recurrent theme in the framework is that podcast evaluation should not collapse into a single scalar. Instead, PodEval treats podcast quality as a structured object whose components must be diagnosed separately. A plausible implication is that the framework is intended as much for **system analysis** as for ranking.

## 2. Real-Pod and the use of human-made podcasts as reference

PodEval introduces **Real-Pod**, a curated dataset of human-made real-world podcasts used as a **reference for human-level creative quality**, rather than as a gold target. The paper is explicit that Real-Pod is a **reference**, not an **absolute answer**. This distinction is fundamental: PodEval does not assume that a generated podcast should match a particular real episode. Instead, Real-Pod provides an anchor for what human-produced quality looks like across many topics and formats [2510.00485].

Real-Pod is constructed in three stages. First, the dataset adopts the taxonomy of **Apple Podcasts**, using **17 categories**. Second, for each category, **GPT-4** is used to generate **5 popular and distinct topics**, after which the topics are manually reviewed and refined; **3 representative topics per category** are retained, producing **51 topics** in total. Third, episodes are manually searched and screened according to **topic relevance** and **rich format**, with preference for multi-speaker conversation, background music and sound effects, and high audio quality.

The categories shown in the evaluation materials include Fiction, Education, Business, True Crime, Health & Fitness, Sports, Comedy, History, News, TV & Film, Society & Culture, Arts, Leisure, Music, Kids, Mental Health, and Science & Tech. The repository provides the Real-Pod dataset, topic and category metadata, and public download links with a download toolkit rather than redistributing audio directly.

Real-Pod is used in four distinct ways: as a human-level quality reference for text, speech, and audio comparisons; as the **high-quality anchor (HQ)** in subjective speech evaluation; as one of the systems compared in the experiments; and as a source of topic diversity for test design. This makes Real-Pod both a dataset and a calibration instrument.

## 3. Multimodal structure and the Content–Format distinction

The operational structure of PodEval is organized around three modalities and two evaluation emphases.

| Dimension | Primary emphasis | Representative methods |
|---|---|---|
| Text | Content | Distinct-2, Sem-Div, MATTR, Info-Dens, GPT-4 as judge |
| Speech | Format | WER, DNSMOS, SIM, SPTD, Dialogue Naturalness Evaluation |
| Audio | Format | Loudness, SMR, CASP, questionnaire-based MOS |

Within this structure, **Content** denotes what is being said: ideas, informativeness, topical relevance, and discussion richness. **Format** denotes how it is delivered: dialogue naturalness, speaker differentiation, expressiveness, loudness balance, and integration of music and sound effects. The framework therefore treats transcript quality and performed audio quality as related but non-identical objects.

This design has two important consequences. First, systems that only emit scripts can still be evaluated on the text axis. Second, systems with stronger delivery but weaker informational structure can be distinguished from systems with the opposite profile. The paper’s later results rely heavily on this decomposition, especially in showing that some systems are strong in script planning but weak in dialogue naturalness, while others are competitive in audio presentation.

## 4. Text evaluation and content assessment

The text component treats the conversation script or transcript as the core representation of podcast content. Because podcast generation is open-ended, PodEval rejects reference-dependent text metrics and instead uses **reference-free quantitative metrics** together with **LLM-as-a-Judge** evaluation. The script evaluation follows the approach from **PodAgent** [2510.00485].

The four quantitative text metrics are **Distinct-2**, **Semantic-Div (Sem-Div)**, **MATTR**, and **Info-Dens**. They are intended to measure lexical diversity, semantic richness or diversity, vocabulary richness, and information density, respectively. The paper reports overall averages across the 17 categories for four systems that provide scripts: **GPT-4**, **PodAgent**, **MoonCast**, and **Real-Pod**. The reported averages are:

- **GPT-4**: Distinct-2 **0.9619**, Info-Dens **6.4507**, Sem-Div **0.1293**, MATTR **0.6914**
- **PodAgent**: Distinct-2 **0.9741**, Info-Dens **7.1767**, Sem-Div **0.1372**, MATTR **0.7216**
- **MoonCast**: Distinct-2 **0.9128**, Info-Dens **6.0935**, Sem-Div **0.1326**, MATTR **0.6323**
- **Real-Pod**: Distinct-2 **0.9200**, Info-Dens **8.1168**, Sem-Div **0.1677**, MATTR **0.6251**

The interpretation given in the paper is comparatively specific. **PodAgent** outperforms directly prompted **GPT-4** on every quantitative metric. **Real-Pod** scores lower on lexical-diversity-style metrics such as Distinct-2 and MATTR but higher on **Sem-Div** and **Info-Dens**; this is interpreted as reflecting more filler words, simpler language, and at the same time richer and longer discussions with more information. **MoonCast**, described as a spontaneous version of PodAgent, reduces lexical diversity and information density while keeping semantic diversity relatively close to PodAgent.

For more holistic script assessment, PodEval uses **GPT-4 as judge** under the criteria of **coherence**, **engagingness**, **diversity**, **informativeness**, and **speaker diversity** or **speaker difference**. The setup is comparative and evidence-based, with **PodAgent vs GPT-4**, score range **-3 to 3**, and positive values favoring PodAgent. The overall results are all positive: **0.7059** for coherence, **1.0294** for engagingness, **1.1765** for diversity, **1.6078** for informativeness, **1.0637** for speaker difference, and **1.3064** overall. The framework thus treats text evaluation as both property-based and judgment-based, with the second component intended to capture aspects that the first cannot.

## 5. Speech evaluation and dialogue naturalness

Speech is treated as the core delivery medium of podcasts, and the paper emphasizes that podcast speech evaluation differs from standard TTS evaluation because it requires **dialogue-level naturalness** rather than only sentence-level pronunciation accuracy. PodEval therefore combines objective metrics with a dedicated **Dialogue Naturalness Evaluation** [2510.00485].

The objective speech metrics are **WER**, **DNSMOS**, **SIM**, and **SPTD**. **WER**, computed with **Whisper**, is used to measure pronunciation accuracy and robustness, particularly for insertion, repetition, or truncation failures. **DNSMOS** supplies **SIG**, **BAK**, **OVRL**, and **P808_MOS**, where SIG, BAK, and OVRL are trained according to **P.835**, and P808_MOS according to **P.808**. **SIM** measures speaker similarity using the cosine similarity of extracted speaker embeddings and is tied to the reference voice selected by PodAgent for the topic. **SPTD**, introduced by the authors, measures **Speaker Timbre Difference**:

$$
\text{SPTD} = 1 - \frac{2}{N (N-1)} \sum_{i=1}^{N} \sum_{j=i+1}^{N} \text{sim}(\mathbf{e}_i, \mathbf{e}_j)
$$

where \(N\) is the number of distinct speakers and \(\mathbf{e}_i\) is the speaker embedding for speaker \(i\). Higher SPTD means greater timbral separation among speakers.

The subjective speech test is based on **MUSHRA** and is designed around five explicit difficulties: long duration, difficulty of multi-system comparison, the need to focus listeners on dialogue naturalness rather than content, balancing topic diversity against limited test time, and unreliable crowd responses. The test includes a **high-quality anchor** from **Real-Pod** and a **low-quality anchor** synthesized with **eSpeak**. PodEval automatically extracts dialogue segments with turn-taking, typically **15–25 seconds** long, sampled across all **17 categories**, with total test duration kept within **30 minutes**. Samples from different systems are shown on the same page, and listeners rate them on a **0–100** slider divided into five stages with clear definitions.

Participants were recruited on **Prolific**, with a request for **20 native English-speaking participants from the US/UK**. Two filtering rules were enforced: over **90%** of low-quality samples had to be marked as the worst, and over **50%** of high-quality samples had to rank in the top-2 best. **Judgers 7, 18, and 20** were excluded, leaving **17 valid submissions**.

The reported speech findings are differentiated by metric. All systems except **Muyan-TTS** achieve **WER below 20%**. The **SIM** ranking is Muyan-TTS, MOSS-TTSD, Dia, MoonCast, PodAgent. For **SPTD**, among **PodAgent**, **NotebookLM**, and **Real-Pod**, the ranking is PodAgent, NotebookLM, Real-Pod. **DNSMOS** places PodAgent, MoonCast, Muyan-TTS, MOSS-TTSD, and NotebookLM at similar speech quality, while Real-Pod and Dia score worse, with the paper attributing this to music/sound removal artifacts and real recording noise for Real-Pod, and to long-form synthesis failures for Dia. In subjective dialogue naturalness, the ranking after filtering is **Real-Pod** first, **NotebookLM** second, **MoonCast / MOSS-TTSD** strongest among open-source systems, and **PodAgent** lowest among the evaluated open-source podcast systems.

## 6. Audio evaluation and the mixed listening experience

At the audio level, PodEval evaluates the complete mixed podcast experience, including **speech**, **music**, **sound effects**, and their interactions. This level uses both objective and subjective measures [2510.00485].

The objective audio metrics are **loudness**, **SMR**, and **CASP**. Loudness is measured with standards derived from **ITU-R BS.1770-4**, **EBU**, recommendations from **AES**, **Apple Podcasts**, and **Spotify**, and loudness-range guidance from **Netflix**. PodEval defines a practical reference region rather than a single absolute target:

- **LOUD-IT**: \(-18\) to \(-14\) LUFS
- **LOUD-TP**: \(\leq -1\) dBTP
- **LOUD-RA**: 4 to 18 LU

The appendix gives scoring functions that assign full score inside the target region and exponential penalties outside it, with constants \(k_1 = 0.0858\), \(k_2 = 0.3291\), \(k_3 = 4.605\), \(k_4 = 1.1513\), and \(k_5 = 0.2554\). **SMR** measures the balance between speech and music or sound effects, with **SMR > 0** treated as the minimum acceptable condition, and **SMR_SCORE** defined as the proportion of cases where SMR exceeds 0. **CASP** evaluates MSE-Speech Harmony using the **CASP model**, pretrained on approximately **1,000 hours of podcast data**, and the score used is the **DualScore**.

The subjective audio test is a questionnaire-based **MOS-style** evaluation. Rather than presenting full episodes, PodEval concatenates the **first minute**, **middle minute**, and **final minute** of each podcast, separated by a **beep signal**. The questionnaire contains **8 questions**, combining perceptual questions such as “Information Delivery Effectiveness” with preference-based questions such as “Speaker Expression Preference,” and also asks about willingness to listen to the full episode and perceived human-likeness. Two validity safeguards are built into the design: attention-check questions, including “How many speakers are there?” and a music-related instruction question, and **mandatory written justification** for every score.

A pilot study preceded the final audio test. Based on feedback, the scale was reduced from **10 points to 5 points**, definitions were clarified, the separation between perceptual and preference-based questions was refined, and justification requirements were added. The reported **pass rate** improved from **75%** to **90%**. The final MOS evaluation uses **PodAgent**, **MOSS-TTSD**, **NotebookLM**, and **Real-Pod**, with each sample around **3 minutes**, each group containing **4 podcast samples** from the same category, and an average completion time of about **24 minutes**.

The objective loudness results show strong system-specific differences. For example, **Dia** and **MoonCast** align well with the target **LOUD-IT** range; **NotebookLM** is centered around about **-25 LUFS**; **Muyan-TTS** is best on **LOUD-TP**; and **Real-Pod** has highly scattered loudness distributions. The reported loudness scores are: Real-Pod **0.72 / 0.53 / 0.82**, PodAgent **0.80 / 0.32 / 1.00**, MoonCast **1.00 / 0.01 / 0.68**, Muyan-TTS **0.88 / 1.00 / 0.83**, Dia **0.98 / 0.01 / 0.95**, MOSS-TTSD **0.88 / 0.02 / 0.99**, and NotebookLM **0.51 / 0.56 / 1.00** for **LOUD_IT_SCORE / LOUD_TP_SCORE / LOUD_LRA_SCORE**. For systems with music and sound, **SMR_BASIC_SCORE** is **0.99** for Real-Pod and **1.00** for PodAgent, while **CASP** is **0.58** for Real-Pod and **0.56** for PodAgent.

At the subjective audio level, **Real-Pod** is reported as best on **5 of 7** metrics, especially **Engagement Level** and **Human Likelihood**. **NotebookLM** performs strongly and is at times close to or better than Real-Pod on production-oriented metrics, including **Audio Quality**. **PodAgent** performs poorly in audio MOS because its underlying speech is less natural in dialogue settings, despite relatively competitive music and sound integration.

## 7. Validation, system-level findings, and limitations

PodEval includes several explicit mechanisms for validation. In speech evaluation, the anchor-based MUSHRA design allows direct filtering of unreliable raters. In audio evaluation, the pilot-to-final redesign raised the pass rate from **75%** to **90%**. The paper also argues that the measured rankings agree with known system properties: **PodAgent** is strong in text and speaker distinction but weak in dialogue naturalness because its backend, **CosyVoice2**, is not designed for dialogue-level generation; **NotebookLM** is strong in naturalness; **Real-Pod** remains strongest in realism and engagement; and AI systems can outperform human-made podcasts on certain controllable production properties such as loudness consistency [2510.00485].

The aggregate findings are clear. **Human-made podcasts remain the strongest overall**, especially in dialogue naturalness, engagement, human-likeness, and the holistic listening experience. **Text generation appears more mature than speech and audio realism**, as shown by PodAgent’s strong script results. **Dialogue-level TTS is identified as a central bottleneck**, since systems built for direct dialogue synthesis, such as MoonCast and MOSS-TTSD, outperform pipelines whose backend is optimized for single-sentence synthesis. At the same time, **AI systems can be stronger on some standardized production metrics**, and PodAgent’s **CASP** and **SMR** results suggest that music and sound integration is already practically viable, even if it does not compensate for unnatural dialogue.

The framework also identifies several limitations. **Real-Pod is a reference, not a ground truth**. **Topic interest** can confound subjective judgments, especially for **Full Episode Willingness**. Some metrics apply only to certain systems, such as text metrics for systems that emit scripts and SMR/CASP for systems with music or sound effects. PodEval also depends on **GPT-4** in multiple places, including topic initialization, text judging, and summarization of written justifications, which introduces reliance on a proprietary evaluator. The ethics statement adds that Real-Pod uses publicly available podcasts under fair-use considerations, that audio is not directly redistributed, that crowd workers are paid above platform minimum wage, and that diverse topics and evaluators are used to mitigate bias.

Taken together, PodEval formalizes podcast evaluation as a structured multimodal problem rather than a single-score benchmark. Its most distinctive contribution is not merely the collection of metrics, but the integration of **reference-free textual measures**, **speech and audio diagnostics**, **human-level reference data**, and **attention-controlled subjective protocols** into one evaluation framework for long-form open-ended podcast generation.

Source: https://www.emergentmind.com/topics/podeval