---
title: 'XS-CoT: Cross-lingual Speech Chain-of-Thought'
url: https://www.emergentmind.com/topics/xs-cot-framework
type: topic
---

# XS-CoT: Cross-lingual Speech Chain-of-Thought

The semi-implicit Cross-lingual Speech Chain-of-Thought (XS-CoT) framework is an approach for enhancing non-core language instruction-following in Speech Large Language Models (SLLMs) by integrating cross-lingual chain-of-thought (CoT) reasoning and token compression. XS-CoT leverages a structured pipeline involving speech-to-text translation, core-language reasoning, and targeted token compression to address the scarcity of high-quality non-core language speech-text data and the limited multilingual reasoning capabilities characteristic of existing SLLMs. Empirical results demonstrate substantial improvements in instruction-following quality and inference latency, particularly in low-resource (non-core) languages, by exploiting the robust reasoning capacity of core-language models [2504.20835].

## 1. System Architecture

XS-CoT is built on a modular stack consisting of a speech encoder, modal adapter, and LLM. The processing pipeline operates as follows:

- **Input:** Speech in a non-core (target) language, denoted \(n\) (e.g., Japanese, German, French).
- **Stage 1 (Modal Alignment):** The speech encoder and adapter transcribe speech into a text instruction \(x^{\rm instr}_n\) in the same language.
- **Stage 2 (XS-CoT Fine-tuning):** The pipeline translates \(x^{\rm instr}_n\) to a core-language (English) instruction \(x^{\rm instr}_c\), performs chain-of-thought reasoning in English to produce \(x^{\rm resp}_c\), then translates the English response back into the target language as \(x^{\rm resp}_n\).
- **Stage 3 (Semi-Implicit CoT Compression):** The first three token types are progressively compressed, producing only a compact sketch of the reasoning chain during training to reduce inference latency, while retaining the global reasoning logic.

The pipeline is formalized as follows:

\[
\text{speech}_n \rightarrow x^{\rm instr}_n \rightarrow x^{\rm instr}_c \rightarrow x^{\rm resp}_c \rightarrow x^{\rm resp}_n
\]

where each transformation involves explicit sequence generation.

## 2. Token Typology and Cross-Lingual Transfer

XS-CoT introduces four explicit token types to facilitate cross-lingual reasoning transfer:

| Token Type                | Notation           | Function                                                      |
|---------------------------|--------------------|---------------------------------------------------------------|
| Target-language Instruction | \(x^{\rm instr}_n\) | Aligns speech-encoder output to textual instruction in target  |
| Core-language Instruction  | \(x^{\rm instr}_c\) | Enables leveraging the LLM's reasoning strength in core language (English) |
| Core-language Response     | \(x^{\rm resp}_c\)  | Encodes the English chain-of-thought reasoning and provisional answer |
| Target-language Response   | \(x^{\rm resp}_n\)  | Translates the English reasoning and answer to the target language |

The full output sequence is

\[
x = [x^{\rm instr}_n,\, x^{\rm instr}_c,\, x^{\rm resp}_c,\, x^{\rm resp}_n]
\]

By sandwiching the English reasoning within bidirectional translations, XS-CoT transfers core-language reasoning to low-resource domains.

## 3. Semi-Implicit Chain-of-Thought Compression

To address the significant inference latency incurred by long core-language CoT chains (typically exceeding 100 tokens), XS-CoT employs a semi-implicit compression mechanism:

- The English CoT response \(R = [s_1, s_2, ..., s_x]\) is partitioned into \(x\) sentences.
- Each sentence \(s_i\) is further divided into word-groups; only the first \(k\) groups are retained, followed by an ellipsis to indicate omission.
- At training epoch \(n\), the first \(m(n)\) sentences are compressed, where

\[
m(n) = 
\begin{cases}
\min(x,n) & n < m_0 \\
x         & n \ge m_0
\end{cases}
\]

- The compression operator \(c_k\) applied to a sentence \(s\) is defined as

\[
c_k(s) = [w_1, ..., w_k, ...] \quad \text{(where } w_j \text{ are word-groups, followed by “…”)}
\]

- The output at epoch \(n\) becomes

\[
c^{(n)}(R) = [c_k(s_1), ..., c_k(s_{m(n)}), s_{m(n)+1}, ..., s_x]
\]

- The training objective remains the standard next-token log-likelihood across all four token types:

\[
\mathcal{L}(\theta) = -\sum_{t=1}^{|x|} \log p_\theta\bigl(x_t \mid x_{<t},\, \text{speech}\bigr)
\]

No reconstruction loss is required; the progressive compression guides the model to infer missing reasoning details.

## 4. Inference Workflow and Latency Analysis

During inference, XS-CoT proceeds in four explicit decoding phases:

1. Decode \(x^{\rm instr}_n\) from speech.
2. Decode \(x^{\rm instr}_c\), conditioned on \(x^{\rm instr}_n\).
3. Decode a compressed CoT chain \(c^{(\infty)}(x^{\rm resp}_c)\), conditioned on previous outputs, where all sentences are fully compressed.
4. Decode \(x^{\rm resp}_n\) from the compressed CoT representation.

Because the majority of intermediate English tokens are compressed—each sentence retaining only \(k\) word-groups—the delay before producing target-language responses is significantly reduced. Empirical measurements indicate that for a full chain of \(D_{\rm full} \approx 107\) tokens, semi-implicit compression yields \(D_{\rm semi} \approx 53\), with a token delay reduction of approximately 50.5%. The compression ratio is defined as

\[
\rho = \frac{D_{\rm semi}}{D_{\rm full}}, \quad \Delta = 1-\rho
\]

where \(\Delta \approx 0.505\) quantifies the delay reduction.

## 5. Data Pipeline and Resource Utilization

XS-CoT employs the “Multilingual Alpaca Speech” dataset, constructed for English (60K samples), Japanese (30K), French (10K), and German (10K). Each training sample is generated via:

1. Extraction from Stanford Alpaca text instructions.
2. Filtering for noise and quality.
3. Translation to the target language.
4. Synthetic speech generation using fish-speech TTS.
5. Whisper ASR filtering (word error rate < 5%).

Crucially, the XS-CoT approach requires only a modest number of non-core language instruction-response speech examples. By leveraging core-language reasoning, the framework transfers chain-of-thought capability efficiently, achieving 2–3x higher sample efficiency over direct supervised fine-tuning.

## 6. Empirical Results and Trade-offs

XS-CoT delivers marked improvements in instruction-following quality for non-core languages, as measured by GPT-4 evaluation scores on Japanese test sets (OpenHermes & ALPACA):

| Model            | Direct SFT | XS-CoT | Absolute Gain | Relative Gain | Tokens (full) | Tokens (semi) |
|------------------|------------|--------|--------------|--------------|---------------|---------------|
| SALMONN-JA       | 28.4       | 50.3   | +21.9        | +77%         | 107           | 53            |

For SLLMs averaging over SALMONN and Qwen2Audio, XS-CoT achieves a 45% relative improvement over direct supervised fine-tuning. A comparative trade-off between compression and performance is observed:

| Method                | GPT-4 score | CoT tokens | Delay reduction |
|-----------------------|-------------|------------|-----------------|
| Full XS-CoT           | 50.3        | 107        | –               |
| Semi-Implicit XS-CoT  | 43.0        | 53         | 50.5%           |

Adopting the semi-implicit scheme saves roughly half the reasoning-token latency with only a 14.5% drop in GPT-4 score, representing a competitive balance between speed and answer quality [2504.20835].

Source: https://www.emergentmind.com/topics/xs-cot-framework