Soft Token Alignment for Cross-Lingual Reasoning

Abstract: Multilingual LLMs often produce inconsistent reasoning and answers for semantically equivalent prompts in different languages. Prior work suggests that intermediate representations can be relatively language-agnostic, but generation becomes increasingly language-specific as models commit to discrete output tokens. This is problematic because language-specific lexical choices can cause semantically equivalent reasoning paths to diverge across languages. These divergences motivate searching for a cross-lingual alignment signal that is less tied to any single vocabulary item or script. We propose SOLAR, an auxiliary objective for supervised fine-tuning that aligns soft-token representations across languages, using English as a pivot. Soft tokens are probability-weighted mixtures over the vocabulary embeddings, yielding continuous representations that can aggregate information from semantically related tokens across languages. We then align each non-English soft-token summary to its English counterpart in the shared embedding space. Across four multilingual reasoning benchmarks, SOLAR improves accuracy by up to +17.7 points over the base model and +3.8 over standard supervised fine-tuning, with the largest gains on low-resource languages. SOLAR also strengthens final-layer cross-lingual similarity and substantially reduces language-cluster separability, suggesting that aligning soft-token representations helps preserve shared semantic structure during multilingual reasoning.

• The paper introduces the SOLAR framework, which leverages probability-weighted soft tokens during supervised fine-tuning to preserve semantic alignment across languages.
• The methodology aligns pooled soft-token mixtures via an auxiliary cosine loss, yielding up to +17.7 task accuracy improvements and doubled cross-lingual similarity in low-resource settings.
• The approach maintains native script fidelity by suppressing dominant language interference, thereby ensuring robust reasoning performance across diverse languages.

Soft Token Alignment for Cross-Lingual Reasoning: An Analytical Essay

Motivation and Problem Formulation

Multilingual LLMs often exhibit inconsistent chain-of-thought (CoT) reasoning and answer accuracy across different languages when provided semantically equivalent prompts. Prior analyses indicate that intermediate transformer representations tend to be language-agnostic, whereas final layer activations increasingly specialize toward language-specific surface forms, triggering cross-lingual divergence at the output stage [wang-etal-2025-lost-multilinguality; lim2025languagespecific]. This specialization at the discrete token selection phase undermines semantic alignment between languages, especially in low-resource settings, where disparate vocabulary and script inhibit the construction of robust shared representations.

The SOLAR Framework

The proposed SOLAR (Soft Token Alignment for Cross-Lingual Reasoning) framework seeks to mitigate cross-lingual reasoning inconsistencies by introducing a soft-token alignment auxiliary objective during supervised fine-tuning (SFT). The central premise is that probability-weighted soft tokens—continuous mixtures over the vocabulary embedding matrix—preserve more cross-lingual semantic content than discrete token projections and can serve as a natural cross-lingual anchor.

During SFT, SOLAR processes English and non-English parallel translations with teacher forcing, generating, at each decoding position, a soft token constructed as the expectation of the model's next-token distribution post top-$k$ filtering and temperature scaling. These per-position soft tokens are then mean-pooled along the sequence to obtain a language-specific continuous summary in embedding space.

Figure 1: SOLAR framework. Parallel English and non-English inputs are processed by a shared LLM under teacher forcing, producing a per-position soft token that mixes vocabulary embeddings under the model's next-token distribution.

The core alignment loss minimizes the cosine distance between each non-English summary and its English counterpart (English is used as the pivot), applied with a stop-gradient on the English side. The overall per-language loss combines the standard cross-entropy objective and the alignment loss, modulated by a hyperparameter $\lambda$.

Key distinction: Unlike Soft Thinking approaches that use soft tokens only at inference [zhang2025soft], SOLAR leverages their algebraic structure to impart cross-lingual alignment pressure at training time, capitalizing on the model's shared embedding space.

Experimental Results

Benchmarks and Evaluation

Experiments target four competitive multilingual reasoning datasets—MGSM, AIME 2024, AIME 2025, and GPQA—spanning English and six non-English languages overlapping with the M-s1k training set. Performance is measured via task accuracy and cross-lingual consistency (CLC), the latter defined as the proportion of questions where both languages yield the correct answer.

Performance Gains

SOLAR consistently outperforms all baselines—including standard SFT, inference-time Soft Thinking, and state-of-the-art contrastive and RL-based alignment methods (MidAlign, MAPO, AlignX)—achieving up to +17.7 task accuracy points (low-resource Swahili, MGSM), with average improvements of +3.8 over SFT in overall benchmarks. CLC improvements closely track accuracy, reflecting enhanced cross-lingual agreement.

Improvement trends are especially pronounced for underrepresented languages, which see the largest absolute gains and a substantive reduction in generation collapse toward English.

Representation Analysis

The study conducts comprehensive representational analyses, focusing on cross-lingual similarity and language clustering in transformer hidden states. Usage of t-SNE visualizations on Gemma3-12B (Figure 2) demonstrates that, under discrete thinking, language clusters are separated in the final layer, whereas under soft thinking—and especially after SOLAR fine-tuning—overlap increases, indicating richer language-agnostic semantic mixing.

Figure 2: t-SNE of Gemma3-12B hidden states on MGSM at the middle and final layers. Language representations intermix at the middle layer under both thinking modes. At the final layer, discrete thinking yields separated clusters, while soft thinking substantially increases overlap.

Quantitatively, SOLAR more than doubles final-layer cross-lingual similarity (CLS) relative to SFT alone, and reduces silhouette-based language cluster separability, directly validating that soft-token alignment constrains the model to preserve shared semantic structure across languages in its final representations.

Behavioral Analysis and Token-Level Effects

Token-level and distribution-level analyses reveal that neither SFT nor SOLAR compromise the model's propensity to reason in the target script: post-fine-tuning, generated reasoning traces and top-$k$ distributions overwhelmingly favor the native script, with neutral/Latin token ratios nearly unchanged. Notably, SFT achieves script fidelity by increasing target-script token density among top predictions, while SOLAR achieves this by suppressing Latin (primarily English) competition in the output distribution—even more so for languages with historically weak coverage (e.g., Telugu).

Figure 3: Distribution-level token composition by language script on MGSM for Qwen3-4B, Qwen3-8B, and their training variants under top-k. SFT increases the density of target-script tokens in the top predictions, while SOLAR primarily suppresses Latin competition.

Ablations and Auxiliary Design Insights

Extensive ablations confirm the robustness and necessity of key SOLAR design choices:

• Soft-Token Target: Alignment over soft token mixtures (top-$k$ truncated) outperforms alternatives (hard token, untruncated mixtures, final layer hidden states) for accurate cross-lingual transfer.
• Alignment Granularity: Pooling over the entire sequence outperforms stepwise or answer-only alignment, likely due to non-parallel step structure in translation pairs.
• Alignment Space: Embedding-space alignment is essential; probability-space alignment yields inferior results, likely due to disjoint vocabularies.
• Pivot Language: English anchoring yields maximum gains, consistent with its centrality in pre-training corpora.

Sensitivity to the alignment strength ($\lambda$) and soft-token temperature ($\tau$) are moderate; performance degrades when alignment weight overwhelms cross-entropy or when soft-token distributions become excessively peaked (thus mimicking discrete tokens).

Broader Implications and Future Work

The SOLAR methodology provides a lightweight, model-agnostic, and resource-efficient paradigm for cross-lingual alignment, requiring neither architectural modification nor per-token supervision, and is highly compatible with both SFT and reinforcement-based fine-tuning stages. The gains on low-resource languages are particularly salient for narrowing the equity gap in LLM-supported languages.

Potential future directions include:

• Integrating SOLAR-like objectives with reward modeling or DPO to further enhance cross-lingual RL post-training [huang2025englishcentrictrainingreinforcementlearning].
• Exploring pivot-free or multi-pivot alignment to reduce English-centric bias, crucial for domains with cross-cultural variation.
• Scaling to tasks with more complex, less monotonic semantics (e.g., retrieval-augmented generation, non-mathematical reasoning).
• Adapting for broader model families (MoE, large-scale, or multimodal LLMs).

Conclusion

SOLAR advances the state of cross-lingual chain-of-thought reasoning in LLMs by explicitly aligning soft-token representations across languages at fine-tuning time. By leveraging the shared vocabulary embedding space, SOLAR imparts strong cross-lingual alignment pressure at training time, resulting in increased accuracy and cross-lingual consistency, most substantially in low-resource languages, without compromising target-language script fidelity. This work establishes probability-weighted soft tokens as a training signal for robust multilingual reasoning and provides empirical foundations for further bridging the multilingual reasoning gap.