FinCall-Surprise: Multimodal Earnings Call Benchmark
- The paper introduces FinCall-Surprise, a large-scale, multimodal dataset of 2,688 U.S. earnings calls (2019–2021) to predict binary future earnings surprises.
- It aligns full transcripts, audio recordings, and presentation slides to enable evaluation of both unimodal and multimodal models amid significant class imbalance.
- The benchmark reveals challenges in financial inference, highlighting the need for better minority-class detection, calibration, and effective multimodal integration.
FinCall-Surprise denotes a research benchmark for earnings surprise prediction from earnings conference calls. It was introduced as the first large-scale, open-source, and multi-modal dataset for earnings surprise prediction, comprising 2,688 unique corporate conference calls from 2019 to 2021 with word-to-word textual transcripts, full audio recordings, and corresponding presentation slides (Shu et al., 4 Oct 2025). Its primary task is to predict whether the next quarter’s earnings will exhibit a material positive or negative surprise relative to post-call analyst expectations. In adjacent research, the same label can also be used more broadly for earnings-call-specific surprise extracted from conference-call language, particularly in frameworks that distinguish immediate numeric surprise from slower qualitative information in earnings conference calls (Yu et al., 29 Jun 2026).
1. Dataset scope and corpus design
FinCall-Surprise is organized around a single event unit: one quarterly earnings call for one firm. Each instance includes a full transcript, a full call recording, a slide deck, and a binary label for the next earnings surprise (Shu et al., 4 Oct 2025). The calls cover 2019, 2020, and 2021, with 919 calls in 2019, 704 in 2020, and 1,065 in 2021. The firms are filtered to large, liquid U.S. equities with market capitalization greater than $1B and average daily trading volume greater than $50M.
The corpus is explicitly multi-modal. Text is sourced from Seeking Alpha and contains word-to-word transcripts with speaker identifiers such as Operator, Executives, and Analysts. Audio is sourced from EarningsCast and consists of full call recordings. Visual material is drawn from Bloomberg News, company websites, and similar sources and consists of the presentation slides used during the call. This design targets the practical observation that earnings calls contain not only propositional content but also prosodic, interactional, and visual information that may bear on subsequent earnings outcomes (Shu et al., 4 Oct 2025).
| Aspect | Specification |
|---|---|
| Coverage | 2,688 unique earnings conference calls, 2019–2021 |
| Modalities | Full transcripts, full audio recordings, presentation slides |
| Target | Binary Positive vs Negative material earnings surprise |
The dataset is intentionally aligned at the event level rather than the sentence or timestamp level. Transcript, audio, and slides are matched by quarter reporting period and by conference call titles across sources to ensure that all modalities correspond to the same call. This alignment supports evaluation of unimodal and multimodal systems under a common prediction target (Shu et al., 4 Oct 2025).
2. Label construction and prediction target
The benchmark uses a standardized earnings surprise definition based on reported EPS and analyst EPS forecasts. The continuous target is
The analyst consensus is defined as the mean of the most recent valid analyst forecasts issued within one month after the conference call, so analysts are allowed to revise their expectations using information from the call and other disclosures (Shu et al., 4 Oct 2025). This implies that the benchmark is not predicting pre-call surprise; it is predicting the next earnings surprise relative to post-call expectations.
The continuous surprise is converted into binary material-surprise labels with threshold :
with and cases with dropped. Thus, denotes a Positive earnings surprise and denotes a Negative earnings surprise (Shu et al., 4 Oct 2025).
This target design is consequential. The average gap between the call and the target earnings event is approximately three months, so the model must infer future fundamentals rather than merely parse contemporaneous quarter information. A plausible implication is that the task blends event interpretation with medium-horizon expectation formation, rather than reducing to a conventional same-day post-earnings-announcement problem.
3. Modalities, preprocessing, and benchmark protocol
The transcript modality is unusually long. The average transcript length is about 8.6k words, with maxima above 45k words in each year and above 57k words in 2019 (Shu et al., 4 Oct 2025). For benchmarking, transcript inputs are capped at 31K tokens to fit within a 32K token context window. About 20% of transcripts exceed this limit, and these are compressed with an iterative summarization pipeline using BART-Large-CNN. The summarization is applied progressively by speaker type: first Operator segments, then Executives, and finally Analysts if the text still exceeds the limit. The released dataset nevertheless contains the full original transcripts.
The audio modality consists of full recordings, typically about 30–60 minutes long. The paper does not define shared feature extraction at dataset level; raw audio is provided to audio-capable models, and the transcript is deliberately withheld when testing audio models so that results reflect audio understanding rather than transcript leakage (Shu et al., 4 Oct 2025).
The slide modality is based on PDF decks converted into page images. For benchmark evaluation, three images are sampled per deck, excluding the first and last pages, which are often title and closing slides. If a deck has three or fewer content slides, all are used. The released dataset includes the full original PDFs, so future work can alter the sampling strategy or use all pages (Shu et al., 4 Oct 2025).
The benchmark itself is zero-shot. Models are not fine-tuned on FinCall-Surprise; instead, frozen pre-trained systems are prompted with a standardized instruction that asks for a POSITIVE or NEGATIVE prediction and a structured output format. Reported results are given by year and overall rather than by train/validation/test split in the conventional supervised sense, because the study evaluates inference behavior rather than task-specific adaptation (Shu et al., 4 Oct 2025).
4. Evaluation methodology and empirical findings
Evaluation uses accuracy and macro-averaged precision, recall, and F1. This choice is central because the label distribution is highly imbalanced: Positive surprises account for 79% of labels in 2019, 86% in 2020, and 89% in 2021 (Shu et al., 4 Oct 2025). The benchmark therefore treats accuracy as insufficient on its own.
A recurrent finding is that high accuracy is often an illusion caused by significant class imbalance in the real-world data. Fifteen of the twenty-six evaluated models exceed 70% accuracy overall, yet macro-averaged precision, recall, and F1 are typically at or below 0.55 (Shu et al., 4 Oct 2025). The benchmark interprets this pattern as majority-class bias: many systems largely predict Positive and therefore benefit mechanically from the class distribution.
This point is illustrated by text-only Qwen-2.5-14B, which reaches overall accuracy 0.80, with year-specific accuracies of 0.78 in 2019, 0.77 in 2020, and 0.84 in 2021, while macro precision, recall, and F1 remain around 0.50 across years (Shu et al., 4 Oct 2025). Because 2021 has 89% Positive labels, the apparent year-over-year improvement in accuracy does not imply better minority-class discrimination.
The GPT family exhibits the opposite pattern. GPT-5, GPT-5-mini, GPT-5-nano, and GPT-4.1, evaluated as image-text models, post lower accuracies than some open models but better-balanced macro metrics. GPT-5 achieves overall accuracy 0.62, macro precision 0.55, macro recall 0.58, and macro F1 0.52, while GPT-4.1 reaches overall accuracy 0.69, macro precision 0.53, macro recall 0.55, and macro F1 0.53 (Shu et al., 4 Oct 2025). Manual inspection reported in the benchmark indicates that these models produce a more balanced mix of POSITIVE and NEGATIVE predictions instead of collapsing toward the majority class.
The study also documents unexpected weaknesses in some financially fine-tuned models. Finance-LLaMa3-8B reports overall accuracy about 0.12 and macro F1 about 0.11, while LLaMa-RAG-8B reports overall accuracy about 0.10 and macro F1 about 0.09 (Shu et al., 4 Oct 2025). The reported failure modes include response bias toward NEGATIVE predictions, poor instruction following, and degraded language generation. By contrast, Finance-LLaMa-8B performs materially better, with overall accuracy 0.74 and macro precision, recall, and F1 all at 0.47.
A second misconception addressed by the benchmark is that multimodality necessarily helps. The reported results are mixed. Within the Qwen-2.5 family, text-only Qwen-2.5-14B reaches overall accuracy 0.80, image-text Qwen-2.5-VL-7B reaches 0.82, and audio-text Qwen-2.5-Omni-7B reaches 0.84 (Shu et al., 4 Oct 2025). This suggests that slides and audio can add useful information. However, LLaMa-3.2-11B as an image-text model underperforms the text-only LLaMa-3.2-3B, and some audio models such as DeSTA-2.5-Audio-8B and Gemma-3n-E4B are substantially weaker. The benchmark therefore concludes that audio and visual modalities provide some performance gains, but current models still struggle to leverage these signals effectively (Shu et al., 4 Oct 2025).
5. Position within earnings-call research
FinCall-Surprise sits within a broader literature on extracting economically relevant signals from earnings calls. One line of work studies the call as a source of qualitative information that arrives after the press release. The unified framework in "Fast Numbers, Slow Language" shows a clean speed separation: quantitative surprise is priced immediately after the press release, whereas qualitative earnings conference call sentiment peaks on the next trading day (Yu et al., 29 Jun 2026). This suggests that conference calls encode information not fully reducible to headline EPS and revenue surprises.
Another line of work isolates specific subcomponents of call language. "Numerical Claim Detection in Finance" constructs a dataset of analyst reports and earnings calls, identifies numeric financial sentences, classifies them into in-claim and out-of-claim categories, and builds an optimism measure from forward-looking numeric claims (Shah et al., 2024). The paper reports that optimism is negatively related to earnings surprise and post-earnings cumulative abnormal returns, and a simple long-short rule based on optimism attains approximate 81% directional accuracy for . This suggests that narrow, claim-level representations can carry predictive information even when general sentiment is noisy.
Text-based finance research has also specialized toward other assets. "Credit Information in Earnings Calls" extracts credit-relevant segments from 202,788 U.S. quarterly earnings calls from 2009–2020, maps them into text-implied CDS information, and defines a residual credit score that forecasts future credit spread changes and future credit spread risk (Mamaysky et al., 2022). The paper interprets the residual between actual CDS-implied PVLGD and text-implied PVLGD as credit-relevant information not fully priced at the time of the call. A plausible implication is that FinCall-Surprise belongs to a wider family of call-based residual or disagreement measures, even when the downstream target is equity earnings rather than credit spreads.
Methodologically, the benchmark has already been complemented by text-only modeling work. "SAE-FiRE: Enhancing Earnings Surprise Predictions Through Sparse Autoencoder Feature Selection" uses only conference call transcripts from 2012–2014, frozen Gemma2 backbones, Sparse Autoencoders from Gemma Scope, feature selection, and an -regularized logistic regression classifier (Zhang et al., 20 May 2025). Its best reported configuration, Gemma2-9B with SAE feature selection, reaches accuracy 0.801, AUC 0.668, and weighted F1 0.757. This indicates that even without audio or slides, long-text representation learning remains a competitive methodological direction.
6. Conceptual significance, limitations, and future directions
FinCall-Surprise is significant chiefly because it changes the experimental conditions of earnings-call research. Earlier work on earnings surprises often relied on proprietary and text-only data; FinCall-Surprise instead provides a large-scale, open-source, and multi-modal benchmark that includes transcripts, audio, and slide decks in one aligned release (Shu et al., 4 Oct 2025). This lowers the barrier to reproducible work on event-driven financial prediction and exposes the practical difficulty of evaluating models under severe class imbalance.
Its main limitations are also explicit. The time span is restricted to 2019–2021, the firm universe is limited to large U.S. listed firms, and labels depend on the availability of analyst forecasts and their cross-sectional dispersion (Shu et al., 4 Oct 2025). Audio and slides are aligned only at the event level rather than at fine-grained timestamps, and the benchmark does not provide an explicit tri-modal baseline that jointly fuses text, audio, and slides. In addition, the use of transcript summarization for very long calls introduces a benchmark-specific approximation, even though the raw transcripts are released.
The benchmark’s central substantive lesson is methodological rather than purely predictive. High accuracy cannot be read as evidence of strong financial reasoning when class imbalance is extreme; financial fine-tuning does not reliably improve inference quality; and multimodal inputs are not yet exploited in a stable way (Shu et al., 4 Oct 2025). This suggests that future work will need better calibration, better minority-class detection, and stronger multimodal alignment rather than merely larger models.
A broader interpretation of “FinCall-Surprise” emerges from related literature. In the benchmark paper, it is a dataset and task definition for predicting future material earnings surprises from conference-call data. In surrounding research, the same expression can plausibly denote the earnings-call-specific component of surprise conveyed by tone, guidance, Q&A, and other qualitative signals that remain after headline numeric surprise has been absorbed (Yu et al., 29 Jun 2026). Under either interpretation, the common research object is the informational increment supplied by the earnings call beyond what is already encoded in simpler market-facing summaries.