Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating

Abstract: LLMs can predict interpersonal attraction from conversation transcripts, but it remains unclear what a speech predictor can add beyond transcript-only LLM prediction. Using Japanese speed-dating conversations, we combine predictions from a transcript-only LLM and a supervised speech predictor to estimate participants' reported liking of their partners. We show that speech can complement transcript-only LLM prediction, but that this complementarity is conditional rather than universal. Combining the two predictions significantly improves pairwise ranking accuracy over the transcript-only LLM alone in all evaluated conditions. By contrast, gains in per-participant Pearson $r$ vary across conversation rounds and rating directions, with none significant after correction. Retrospectively, these $r$ gains are concentrated among participants for whom the speech predictor is more accurate. Speech can therefore retain predictive value even when an LLM predicts attraction from transcripts. The relevant question is not simply whether speech helps, but where its complementarity emerges.

• The paper demonstrates that supervised speech predictors add conditional value over transcript-only LLMs in predicting interpersonal attraction.
• It employs a multimodal fusion approach combining HuBERT and Claude on 1,248 Japanese speed dating conversations using CCC and Pearson r metrics.
• Results reveal significant pairwise accuracy gains and underscore the need for participant-adaptive fusion strategies in multimodal models.

Speech Complements LLMs for Predicting Attraction in Speed Dating

Background and Motivation

The prediction of interpersonal attraction in dyadic interactions leverages both verbal and nonverbal cues. While LLMs have demonstrated utility in extracting relevant information from conversation transcripts for social outcome prediction, their coverage of nonverbal signals—especially from speech—is limited by design. Prior work integrates modalities at the data, feature, or prediction level; however, whether a supervised speech predictor can contribute incremental predictive value over a transcript-only LLM, and critically, under what conditions this occurs, remains an open empirical question. The present study addresses this gap using Japanese speed-dating conversations, direct rating elicitation, and multimodal predictive modeling.

Corpus, Problem Formulation, and Predictors

This study utilizes the Multi-Modal Speed Dating corpus, comprising 1,248 male-female conversations among 147 Japanese participants, each rated post-interaction via Rubin’s Liking Scale (13 items, 1-9 scale, high internal consistency). Two rounds (5-min and 10-min) of conversations were collected, enabling repeated measures. The fundamental predictive task is to recover each participant's differentiation among partners (within-participant ranking), with calibration-sensitive evaluation through CCC and association-sensitive assessment using per-participant Pearson $r$.

The modeling configuration comprises (1) a transcript-only LLM (Claude Sonnet 4.6, zero-shot, structured output, “extended thinking” disabled), and (2) a supervised speech predictor utilizing frozen HuBERT-Large representations, with attention pooling and regression confined to rater-side speech. The predictions from these models are fused at the score level via a weight selected by held-out validation.

Figure 1: The modeling and fusion pipeline, with parallel transcript (LLM) and rater-side speech (HuBERT + regression head) branches fused at score level.

Evaluation Design

Performance is evaluated through 25-fold leave-one-group-out cross-validation. Metrics include per-participant Pearson $r$ (ranking fidelity), CCC (calibration), and pairwise accuracy (PW, ordering). The analysis comprehensively reports results across conversation rounds (Session 1 and 2) and rating direction (female to male, F2M; male to female, M2F).

Auxiliary baselines include other supervised text predictors (BERT, Sentence-T5, J-LIWC), supervised acoustic features (openSMILE), as well as direct-input MLLM branches (Gemini 2.5 Flash and GPT-audio-mini on text+audio input).

Empirical Results

Unimodal Performance

The transcript-only LLM (Claude) outperforms all supervised text baselines and direct-input MLLMs, exhibiting per-participant $r$ in the range $.16$–$.25$ (F2M and M2F, both rounds). The HuBERT-based speech predictor is substantially weaker in Session 1 but approaches Claude in Session 2, particularly for F2M. Notably, supervised text+speech fusion (Sentence-T5 + HuBERT) fails to surpass Claude, highlighting the relative strength of LLMs on transcript-only inference.

Multimodal Fusion

Weighted late fusion of Claude and HuBERT shows significant improvements in pairwise accuracy (PW) across all session-by-direction conditions. Specifically, the fusion achieves a PW gain of $+0.05$ over Claude in both Session 1 F2M and Session 2 F2M, with statistically significant paired improvements. However, the gain in per-participant $r$ is not uniformly significant and is numerically largest in Session 2 F2M ($\Delta r=+0.08$), with negligible changes or null effects elsewhere.

Crucially, condition-dependent complementarity is observed: fusion benefits are tied to the relative strength of HuBERT for a given participant. Participants for whom HuBERT’s unimodal $r$ is high also show greater fusion gains, as illustrated in Figure 2.

Figure 2: Fusion gains ($\Delta r$ over Claude) are larger for participants where HuBERT obtains higher per-participant Pearson $r$0, highlighting the conditional value of speech features.

The late-fusion gain is not accounted for by redundancy: decomposition analyses demonstrate low variance overlap between Claude and HuBERT, and each independently explains variance beyond the other. Direct-input text+audio MLLMs do not show consistent improvements over transcript-only LLM configurations under the tested input pipelines.

Interpretation and Theoretical Implications

The findings support a nuanced view of multimodal complementarity: the incremental predictive value of rater-side speech is conditional—i.e., observed for participants and conditions where acoustic features are informative, but not a universal effect. The lack of a significant average per-participant $r$1 gain contradicts the notion that adding speech always improves outcome prediction. The improvement in PW indicates speech cues help preserve pairwise ranking of partners even when overall absolute scoring is less enhanced.

These results further suggest that model fusion incorporating independent speech and transcript processing can augment LLM-based social inference, provided the speech branch exhibits meaningful predictive signal. The condition-dependent nature of complementarity reflects the non-uniform distribution of acoustic cues signaling interpersonal attraction in spontaneous interaction, consistent with theories on nonverbal communication variability.

Methodological and Practical Considerations

Late score-level fusion offers a pragmatic modality integration strategy when model internals (e.g., LLM scalar outputs) do not allow for feature-level fusion. The selected fusion weights adapt to branch performance across conditions, increasing flexibility. The tested direct-input MLLM baselines fail to match the fusion’s pairwise accuracy benefits, likely due to prompt and input constraints. The modularity of score-level fusion enables rapid adaptation to evolving unimodal models.

Limitations include corpus specificity (Japanese, opposite-sex, face-to-face dyads), the absence of detailed speech-cue interpretability in HuBERT, and scope restriction to text and audio modalities. Partner-level consensus—which can predict liking scores across participants—is higher than any tested model, underscoring the value of integrating individual and group effects in future work.

Implications for AI and Future Directions

The observed findings restrain strong claims about universal benefits of speech addition to LLM-based social inference. They recommend empirical localization of multimodal complementarity and development of participant-adaptive or reliability-estimating fusion methods. Methodologically, score-level fusion provides a robust baseline for integrating disjoint modality experts. Advances in audio-native MLLMs and richer joint encoders may challenge the current fusion paradigm, but the need for careful per-condition and per-participant performance analysis will persist.

Future research should probe cue-level interpretability in self-supervised speech models, explore hierarchical and adaptive fusion strategies, expand to triadic and group interactions, and systematically examine the generalization of complementarity phenomena across languages, cultures, and pairing contexts.

Conclusion

This paper demonstrates that supervised speech predictors can confer conditional predictive gains over strong transcript-only LLMs in the context of speed-dating attraction prediction. The enhanced pairwise ranking accuracy supports the complementary—but non-uniform—role of acoustic cues in social inference. These findings underline the importance of granular, participant- and context-sensitive evaluation in multimodal affective computing and provide a precise empirical boundary for LLM capabilities in interpersonal outcome prediction (2607.23037).