---
title: 'SayNext-PC: Multimodal Next-Utterance Prediction'
url: https://www.emergentmind.com/topics/saynext-pc
type: topic
---

# SayNext-PC: Multimodal Next-Utterance Prediction

SayNext-PC is a collection of multimodal datasets and personalization methodologies developed for next-utterance prediction in dialogue, with a particular focus on capturing individual- and context-specific word choice at the token level. It serves as the foundational dataset for the SayNext-Bench benchmark, enabling systematic evaluation of large language models (LLMs) and multimodal LLMs (MLLMs) in predicting human dialogue responses by integrating both linguistic history and fine-grained nonverbal cues. SayNext-PC also refers to a personalized next-token prediction system, grounded in psychological traits and privacy-aware engineering practices, enabling the modeling of user-aligned linguistic behavior in both synthetic and real-world conversational scenarios [2510.14398][2602.00327].

## 1. Dataset Composition and Construction

SayNext-PC exists in several versions, notably SayNext-PC2K and SayNext-PC19K, designed around three construction criteria: (1) dyadic, one-to-one interactions, (2) synchronized text and video modalities, and (3) fixed camera view on the responder. These constraints ensure clean speaker attribution and alignment of micro-expressions with utterances. 

Key statistics for the datasets include:

| Subset           | # Videos | Duration (min) | # Turns | # Responders | Years      |
|------------------|----------|----------------|---------|--------------|------------|
| SayNext-PC2K     |    359   |      2,092     |  5,432  |     72       | 2017–2019  |
| SayNext-PC19K    |   3,463  |     20,766     | 38,540  |    474       | 2017–2024  |

Modalities are exhaustively annotated: video (ViT-300M features, frame rate 25 fps, standardized resolutions), text (Whisper ASR, manual segmentation, mean WER 4.11%), and micro-body-language labels (aligned using iMiGUE protocol, with features such as lip movements, gaze direction, hand gestures). Record-level metadata includes clip/video IDs, subject/nationality, textual transcriptions, start/end timestamps, emotion annotations (Valence, Arousal, Dominance), micro-expression arrays, and a 20-dimensional cognitive priming vector (for modeling latent emotional/cognitive state activation during response) [2602.00327].

Data collection derives from large-scale post-match press conference videos from all four Grand Slam tennis tournaments, re-scraped and re-annotated for scale and continuous speaker focus.

## 2. Problem Formulation and Theoretical Foundations

SayNext-PC supports several task formulations:

- **Next-Utterance Prediction (Multimodal):** Given a question utterance $TA$ and a sequence of synchronized video frames $V$, predict the respondent’s next utterance $TE$. Formally, the system estimates $P(\text{truth\_response} \mid TA, V) \approx f_e(TA, V) \to \hat{TE}$.

- **Personalized Next-Token Prediction (YNTP paradigm):** For user $u$ with context $h = (x, y_{<t})$ (where $x$ is NPC input and $y_{<t}$ are previous user output tokens), plus user profile $U^u = (p^u, H^u)$ (MBTI type, dialogue history), the system predicts the distribution $p_\theta(w_t \mid h, U^u)$ across candidate next tokens. The global learning objective is minimization of token-level cross-entropy:
  $$
  L_{CE}(\theta) = -\sum_i \sum_{t=1}^{|y^u_i|} \log p_\theta(y^u_{i,t} \mid y^u_{i,<t}, x_i, U^u_i)
  $$
  [2510.14398].

The data supports analyses of anticipatory processing, affective alignment, and style-personalized language modeling grounded in psychologically interpretable parameters.

## 3. Evaluation Metrics and Benchmarking

Empirical assessment incorporates lexical, semantic, and affective metrics:

- **Lexical Overlap:** BLEU-4, ROUGE-L.
- **Semantic Similarity:** BERTScore-F1, SBERT cosine similarity.
- **Emotion Consistency:** NRC-VAD-derived metrics along Valence, Arousal, and Dominance dimensions.

For personalized next-token prediction, cross-entropy loss and perplexity are primary, sensitive to deviance from user-specific word-choice distributions. Macro-averaged English results for baseline and personalized adaptation methods are illustrated below:

| Method              | Cross-Ent ↓ | Perplexity ↓ | Sentence Sim. (M2) ↑ | Style Sim. (M5) ↑ | History Sim. (M6) ↑ |
|---------------------|-------------|--------------|----------------------|-------------------|---------------------|
| Zero-Shot Prompting |   1.82      |    6.17      |    0.43              |    2.88           |   0.43              |
| Few-Shot Prompting  |   1.65      |    5.21      |    0.46              |    3.28           |   0.45              |
| Few-Shot+MBTI       | **1.58**    | **4.85**     |    0.44              |    3.32           |   0.42              |
| LoRA Adapter FT     |   1.72      |    5.60      |    0.35              | **3.49**          |  **0.45**           |

Few-shot prompting substantially reduces perplexity. MBTI priming in prompts further improves style alignment. LoRA-driven fine-tuning yields sustained user-specific style and history resemblance, though it incurs costs due to per-user adaptation overhead [2510.14398].

SayNext-Chat, a dual-route MLLM, achieves state-of-the-art on SayNext-PC2K—BLEU-4 = 2.31%, ROUGE-L = 17.96%, BERTScore = 0.565, and Valence EC = 0.801—exceeding strong baselines such as GPT-4o by a factor of 2–6× in word-level match and 0.02–0.04 absolute in semantic/emotion metrics [2602.00327].

## 4. Personalization and Adaptation Methodologies

SayNext-PC provides a rich testbed for both external (prompt-based) and internal (fine-tuning-based) adaptation approaches:

- **Prompt-Based (External):**
  - *Zero-shot*: System prompt specifies desired user imitation.
  - *Few-shot*: Prompt includes $k$ (input, output) exemplars from the user.
  - *MBTI Augmentation*: Explicit MBTI summaries appended to prompt.
- **Fine-Tuning-Based (Internal):**
  - *LoRA (PEFT)*: Low-rank adapters are injected into transformer projections, $W = W_0 + AB$, where $A \in \mathbb{R}^{d \times r}, B \in \mathbb{R}^{r \times d}, r \ll d$; only $A, B$ are optimized.
  - *Prefix Tuning*: Trainable prefix tokens prepended to each attention layer.
  
Recommended hyperparameters include LoRA rank $r = 8$–$16$, prefix length $20$–$50$ tokens, learning rates $1\text{e}{-4}$ for adapters/prefixes, batch sizes $8$–$16$, and 3–5 training epochs per user [2510.14398].

The dataset supports multi-lingual experiments (English, Chinese, Japanese) and ablation studies on user history priming, MBTI conditioning, and scalability from PC2K to PC19K.

## 5. Annotation Protocols and Data Access

Annotation utilizes a combination of automated tools and human-in-the-loop validation:

- **Transcription:** Whisper ASR, manual WER verification (mean WER 4.11%).
- **Turn Segmentation:** Audio diarization plus manual correction for high-precision speaker boundary assignment.
- **Micro-annotations:** Adoption of iMiGUE-labeled micro-expressions; feature alignment to 25 fps video frames.
- **Priming vector extraction:** LLMs (GPT-4.1) cluster cognitive/emotional states, with each response annotated by cluster activation vector $[−1, 1]^{20}$.
- **File formats:** Metadata in CSV, video/text/annotation in standard formats (.mp4, .txt, .json, .npy). See [https://saynext.github.io/] for download and documentation [2602.00327].

Licensing is CC BY 4.0, enabling open research use.

## 6. Privacy and Engineering Considerations

SayNext-PC-derived systems, such as “SayNext-PC” as a next-token predictor, incorporate privacy-oriented practices:

- **User Adapter Storage:** Adapters or personalized prompts are stored strictly on-device to minimize leakage risk.
- **Profile Token Security:** MBTI/profile tokens are to be encrypted or hashed at rest.
- **Differential Privacy:** Optional DP-SGD is recommended for adapter optimization, bounding per-user information leakage.
- **Data Split Discipline:** Strict separation of train (first 80% of turns by user) and test (last 20%), mirroring realistic online adaptation constraints [2510.14398].

This design enables privacy-compliant deployment of personalized models in user-aligned dialogue applications.

## 7. Empirical Challenges and Future Insights

SayNext-PC experiments reveal persistent modeling challenges for LLMs and MLLMs:

- *Lexical overlap* remains low (<20% BLEU), even for top-performing models, indicating the intrinsic difficulty of verbatim utterance anticipation.
- Integration of nonverbal cues (gaze, gesture, micro-expressions) is essential but remains a limiting factor for model generalization and pragmatic nuance expression.
- Dual-route, cognitively inspired architectures demonstrably improve emotion consistency (by ∼3%) and facilitate affective/perspective alignment.
- Satisfactory generalization to unseen speakers or spontaneous scenarios remains an open challenge; pragmatic subtleties (sarcasm, metaphor) are often flattened to neutral predictions [2602.00327].

SayNext-PC thus defines a new research standard for anticipatory dialogue prediction systems, enabling advances toward context-sensitive and user-specialized AI interaction.

Source: https://www.emergentmind.com/topics/saynext-pc