SpeechCT-CLIP: Radiology Speech & CT Alignment Model

• The paper demonstrates that teacher-guided distillation from a text encoder recovers up to 88% of the performance gap in zero-shot CT abnormality classification.
• The methodology employs dual encoders—a frozen CT encoder and a trainable Whisper-based speech encoder using sliding-window processing—to manage long-form spoken reports.
• Empirical results on CT-RATE and RAD-ChestCT show improved retrieval and classification, supporting voice-driven radiology diagnostics and workflow integration.

SpeechCT-CLIP most specifically denotes a vision–speech CLIP-style foundation model for radiology that aligns 3D chest CT volumes and spoken radiology reports in a shared embedding space, enabling zero-shot abnormality classification, cross-modal retrieval, and related multimodal inference without requiring text at inference time (Buess et al., 24 Sep 2025). In adjacent literature, the term also functions as a broader conceptual label for CLIP-derived architectures that replace one CLIP branch with a speech or audio encoder and train a shared representation with contrastive objectives, as in text–mel text-to-speech, multilingual image–speech retrieval, and acoustic–phonetic alignment (Liu, 26 Feb 2025, Berry et al., 2022, Vallés-Pérez et al., 2023). Within that broader family, the radiology-specific SpeechCT-CLIP is distinguished by its use of a frozen CT-CLIP teacher and a speech pathway designed for long spoken reports.

1. Definition, scope, and conceptual lineage

SpeechCT-CLIP, in the narrow sense established by the 2025 radiology paper, is a voice-native multimodal CT analysis model trained on paired CT volumes and spoken findings sections from radiology reports (Buess et al., 24 Sep 2025). Its central objective is to determine whether multimodal pretraining can operate directly on dictation-style speech rather than on written report text. This design responds to a clinical workflow in which radiologists dictate reports, ASR systems transcribe them, and downstream medical AI ordinarily consumes only the written form.

In a wider methodological sense, SpeechCT-CLIP belongs to a class of models that inherit the basic CLIP recipe—dual encoders, a shared latent space, and a symmetric contrastive loss—but replace image or text branches with speech-related modalities. The Clip-TTS work describes a conceptual “SpeechCT-CLIP” as a Contrastive Text–Speech model aligning text and mel-spectrograms, with the explicit aim of learning “true semantics of the global context” for TTS (Liu, 26 Feb 2025). M-SpeechCLIP instantiates the same design pattern for multilingual image–speech retrieval, aligning HuBERT-based speech embeddings to CLIP’s image space (Berry et al., 2022). SCRAPS applies CLIP-style contrastive learning entirely within the speech domain, aligning mel-spectrograms and phoneme sequences in a shared latent space (Vallés-Pérez et al., 2023).

These variants clarify what is stable across the family and what is task-specific. The stable elements are a contrastive objective over paired modalities and a shared embedding geometry. The task-specific elements are the encoders, the unit of alignment, and the downstream use. In SpeechCT-CLIP for radiology, the visual branch is a 3D CT encoder, the speech branch must handle report-length audio, and the model is evaluated on zero-shot clinical classification and CT retrieval rather than on generic caption retrieval (Buess et al., 24 Sep 2025).

2. Clinical motivation and the Speech-RATE corpus

The immediate motivation for SpeechCT-CLIP is clinical. In routine radiology practice, radiologists dictate reports while reviewing CT volumes, and ASR systems produce the text used in electronic records. The paper argues that modern medical AI inherits ASR errors because multimodal foundation models such as CT-CLIP and BiomedCLIP are overwhelmingly text-based, and it therefore asks whether spoken reports can function as the language modality directly (Buess et al., 24 Sep 2025).

To study that question, the model is trained on Speech-RATE, a spoken-report resource obtained by augmenting CT-RATE with synthetic speech. CT-RATE provides 50,188 3D chest CT volumes, each paired with a written radiology report and annotated with 18 abnormality labels. Speech-RATE uses the findings section of each report, synthesizes speech with the Kokoro TTS engine using 8 distinct synthetic voices (4 female, 4 male) and randomized speaking rate, and resamples all audio to 24 kHz. The resulting corpus contains 50,188 spoken findings sections, 1,197 hours of audio, and an average report length of 86 seconds (Buess et al., 24 Sep 2025).

The paper also specifies the evaluation setting. It uses the official CT-RATE train/test splits, a lighter RadGenome-ChestCT representation for efficiency, and an external validation set, RAD-ChestCT, from another hospital. RAD-ChestCT uses 16 labels, because two calcification labels are merged and one label is removed. This external setup is important because the central claim is not merely within-dataset retrieval, but cross-domain robustness of a speech-native radiology representation (Buess et al., 24 Sep 2025).

A recurrent misconception is that substituting speech for text is mainly a data-format change. The Speech-RATE construction and the later ablations suggest otherwise: the shift from text to speech introduces long-sequence modeling, encoder selection, and a distributional mismatch between speech latent spaces and the pretrained CT-text space. The model’s design addresses those issues explicitly rather than assuming that raw spoken input will automatically preserve the semantics of dictated findings.

3. Architecture and optimization

SpeechCT-CLIP combines three components during training: a frozen CT encoder from CT-CLIP, a frozen text teacher from CT-CLIP, and a trainable speech encoder initialized from Whisper (base) (Buess et al., 24 Sep 2025). The CT encoder $g(\cdot)$ is a CT-ViT backbone from CT-CLIP / GenerateCT that processes 3D CT volumes into fixed-dimensional embeddings $v_i \in \mathbb{R}^d$. The teacher text encoder $h(\cdot)$ produces report-text embeddings $t_i$ in the same latent space. The trainable speech branch $f_\theta(\cdot)$ maps spoken findings waveforms to speech embeddings $a_i$.

Because many radiology reports are approximately one to two minutes long, the speech path cannot simply ingest the whole waveform in a single pass through Whisper. The model therefore uses a sliding-window strategy with 30 s windows and 2 s overlap. If a report waveform is segmented into $K$ windows, each window is encoded separately and the final report embedding is the average of the window embeddings:

$a_i = \frac{1}{K}\sum_{k=1}^{K} a_i^{(k)}.$

This choice treats the findings section as a single semantic object while staying within the input limits of the speech backbone (Buess et al., 24 Sep 2025).

Training uses two losses. The first is a symmetric audio↔CT contrastive loss over cosine similarities. For the audio-to-CT direction,

$$\mathcal{L}_{\text{con}}^{\text{a}\to\text{ct}} = -\frac{1}{N}\sum_{i=1}^N \log \frac{\exp\big(\cos(a_i, v_i)\big)}{\sum_{j=1}^N \exp\big(\cos(a_i, v_j)\big)},$$

with an analogous CT-to-audio term; the full contrastive objective is their sum. The second term is a speech–text distillation loss from the frozen CT-CLIP text teacher:

$\mathcal{L}_{\text{distill}} = 1 - \cos(a_i, t_i).$

The total objective is

$v_i \in \mathbb{R}^d$0

The teacher is used only during training; inference requires only the CT and speech branches (Buess et al., 24 Sep 2025).

Implementation details are also explicit. The CT and text encoders are frozen, Whisper (base) is the speech backbone, the optimizer is Adam, the learning rate is $v_i \in \mathbb{R}^d$1, the batch size is 8, and training runs for 100,000 steps on a single NVIDIA A100 80GB (Buess et al., 24 Sep 2025). The paper’s architecture section further notes that a projection head maps Whisper outputs into the CLIP embedding space, matching the CT-CLIP latent dimension.

This design differs materially from other speech-oriented CLIP variants. Clip-TTS inserts a duration predictor and length regulator so that text and mel sequences align at phoneme–frame granularity before pooling, because temporal alignment is indispensable for TTS (Liu, 26 Feb 2025). M-SpeechCLIP, by contrast, keeps CLIP’s image encoder frozen and adds a trainable speech-side transformer and projection head on top of HuBERT, targeting image–speech retrieval rather than long-form clinical dictation (Berry et al., 2022). SpeechCT-CLIP’s defining architectural move is therefore not merely “speech instead of text,” but speech plus teacher-guided transfer into an existing CT-text embedding space.

4. Empirical performance

The empirical result emphasized by the paper is that naive speech–CT contrastive training is insufficient, whereas distillation from CT-CLIP’s text encoder substantially narrows the gap to the text-based teacher (Buess et al., 24 Sep 2025). On CT-RATE internal zero-shot multi-label classification over 18 labels, the text-based CT-CLIP achieves F1 0.718 and AUROC 0.734. The naive speech-only baseline, denoted SpeechCT-CLIP$v_i \in \mathbb{R}^d$2, achieves F1 0.623 and AUROC 0.610. Full SpeechCT-CLIP, with distillation, improves to F1 0.705 and AUROC 0.708 (Buess et al., 24 Sep 2025).

The paper frames this as 88% performance-gap recovery. Using the reported values,

$v_i \in \mathbb{R}^d$3

so the recovered fraction is approximately $v_i \in \mathbb{R}^d$4. The abstract reports this as 88% (Buess et al., 24 Sep 2025). The same pattern holds externally on RAD-ChestCT: CT-CLIP reaches F1 0.660 and AUROC 0.643, the naive speech model reaches F1 0.596 and AUROC 0.552, and the distilled model improves to F1 0.623 and AUROC 0.603 (Buess et al., 24 Sep 2025).

Retrieval results reinforce the same conclusion. On CT-RATE, text-based CT-CLIP obtains R@5 0.048, R@10 0.085, R@50 0.281, and R@100 0.430 for report-to-CT retrieval. The naive speech model achieves 0.026, 0.049, 0.180, and 0.291, whereas full SpeechCT-CLIP reaches 0.042, 0.077, 0.244, and 0.377. Distillation improves retrieval substantially while leaving a residual gap to the text teacher (Buess et al., 24 Sep 2025).

Ablation on the speech backbone explains part of the design. When the authors compare wav2vec 2.0, HuBERT, and Whisper on direct abnormality classification from spoken reports, the reported weighted F1 scores under linear probing are 0.51, 0.65, and 0.67, respectively. With finetuning of Whisper’s upper layers, Whisper (top 2 layers) reaches F1 0.76, precision 0.84, and recall 0.71, while Whisper (top 3 layers) reaches F1 0.75, precision 0.82, and recall 0.71 (Buess et al., 24 Sep 2025). The paper interprets this as evidence that Whisper’s ASR pretraining, with its emphasis on semantic content, transfers well to radiology dictation.

One misconception that the reported numbers directly reject is that a CLIP-style objective alone is enough to recover text-grade semantics from speech. In this setting it is not. The naive contrastive model underperforms markedly, and the improvement comes from knowledge distillation from a pretrained text–image CLIP teacher, not from simple modality substitution (Buess et al., 24 Sep 2025).

5. Relation to other CLIP-derived speech systems

SpeechCT-CLIP sits within a broader ecosystem of CLIP-inspired speech models, but the neighboring systems solve different alignment problems and expose different inductive biases. Clip-TTS uses a text encoder and a mel encoder, both built from Feed-Forward Transformer (FFT) blocks, and aligns them with a CLIP-style contrastive loss before attaching a non-autoregressive Transformer mel decoder and HiFi-GAN vocoder. Its distinctive addition is a duration predictor and length regulator, which expand phoneme representations to frame-level resolution prior to global pooling; this makes the shared space explicitly phoneme-aware and time-aware (Liu, 26 Feb 2025). SpeechCT-CLIP does not solve sequence generation and therefore does not require this duration mechanism.

M-SpeechCLIP is closer in spirit to the retrieval side of SpeechCT-CLIP. It keeps CLIP-Large ViT-L/14 frozen on the image side, uses HuBERT-Large or HuBERT-Base as the speech backbone, and trains image–speech retrieval with Masked Margin Softmax (MMS). Its multilingual analysis shows that English-only pretraining creates systematic language asymmetries and that mixed-language batches and speech-backbone finetuning are important for non-English retrieval (Berry et al., 2022). That observation is orthogonal to SpeechCT-CLIP’s clinical setting, but it clarifies a general principle: the pretrained speech backbone matters not only for raw acoustic modeling but for the geometry of the shared CLIP space.

SCRAPS shifts attention from semantics to phonetic–acoustic alignment. It uses two Transformers plus a shared-weight LSTM integrator to embed mel-spectrograms and X-SAMPA phoneme sequences into a shared $v_i \in \mathbb{R}^d$5-dimensional space, trained with a symmetric CLIP-like loss. The resulting embeddings are sensitive to phonetic perturbation—reported as a 91.06\% score drop rate when 20\% of phonemes are randomly replaced—and comparatively robust to severe Gaussian noise (Vallés-Pérez et al., 2023). Relative to SpeechCT-CLIP, SCRAPS demonstrates that CLIP-style supervision can be applied inside the speech domain itself, without any visual branch.

SpeechCLIP+ and Segmental SpeechCLIP extend the family toward visually grounded speech. SpeechCLIP+ replaces fixed CLS-token segmentation with Continuous Integrate-and-Fire (CIF) and combines parallel retrieval and cascaded keyword-style learning in a hybrid multi-task setup; the CIF-based cascaded branch improves speech keyword extraction and can improve the parallel retrieval branch on Flickr8k (Wang et al., 2024). Segmental SpeechCLIP uses a hierarchical segmental speech encoder to generate word-like units from frozen Wav2Vec2 features and then feeds those continuous segment embeddings directly into a pretrained text encoder such as CLIP or mpnet, reporting stronger retrieval than cascaded SpeechCLIP and showing that audio-only training on semantically related utterance pairs can approach audio–visual performance (Bhati et al., 2023).

A broader methodological caution comes from SoundCLIP. In that work, projecting audio features aggressively into CLIP’s visual manifold improves audio↔video retrieval but degrades text generation quality in CLIP-based multimodal LLMs, yielding a reported trade-off with Pearson correlation $v_i \in \mathbb{R}^d$6 between retrieval gains and generation losses (Vosoughi et al., 12 Jun 2025). That result does not invalidate SpeechCT-CLIP, because SpeechCT-CLIP is evaluated on retrieval and zero-shot classification rather than on free-form generation. It does, however, caution against a common overgeneralization: stronger cross-modal alignment is not automatically optimal for every downstream objective.

6. Applications, limitations, and future directions

Within radiology, SpeechCT-CLIP is intended to support voice-driven diagnostic support. Because the model requires only CT and speech at inference, the paper proposes use cases in which radiologists interact with the system by speaking rather than typing, potentially avoiding ASR transcription errors and preserving aspects of spoken communication such as uncertainty markers and hedging (Buess et al., 24 Sep 2025). It also supports voice-based case retrieval, in which spoken descriptions are used to retrieve similar CT volumes, and more generally voice-controlled workflow integration for semantic search over CT archives.

The model’s immediate evaluated tasks are zero-shot multi-label classification and cross-modal retrieval, but the authors also identify possible extensions to segmentation, localization, multilingual training, real dictated reports, other imaging modalities such as X-ray, MRI, and ultrasound, and integration with LLMs for conversational assistants and structured report generation (Buess et al., 24 Sep 2025). These are described as future directions rather than completed results.

Several limitations are explicit. First, Speech-RATE uses synthetic voices, so real radiologist dictations with pauses, disfluencies, accent variability, and background noise remain out of distribution. Second, the residual performance gap to CT-CLIP indicates that speech still does not fully recover the text teacher’s semantic fidelity. Third, voice data are personally identifiable, so privacy regulation is a material deployment constraint. Fourth, the method remains an embedding-based system whose interpretability may be limited in clinical settings (Buess et al., 24 Sep 2025).

A second misconception is therefore that SpeechCT-CLIP proves speech can simply replace text in medical multimodal pretraining with no loss. The reported results do not support that interpretation. They support a narrower claim: speech can become a practical alternative to text when a pretrained text–image model transfers semantic structure into the speech encoder through distillation, and in this setting the resulting model reaches F1 0.705 versus 0.718 for the text teacher on CT-RATE while working without requiring text at inference time (Buess et al., 24 Sep 2025). A plausible implication is that future progress will depend less on replacing text outright than on better mechanisms for teacher transfer, long-form speech representation, and domain-robust spoken-report corpora.