SwimBird: Adaptive Multimodal Reasoning

• SwimBird is a reasoning-switchable multimodal model that adapts its cognitive substrate based on query demands for enhanced symbolic logic and perception.
• It employs distinct text-only, vision-only, and interleaved reasoning modes, chosen dynamically via a hybrid autoregressive mechanism.
• Evaluations demonstrate superior performance in benchmarks for visual, mathematical, and VQA tasks compared to fixed-pattern multimodal models.

SwimBird is a reasoning-switchable Multimodal LLM (MLLM) designed to dynamically adapt its internal reasoning modality—text-only, vision-only, or interleaved vision-text—based on the input query. It targets the core limitation of rigid, fixed-pattern reasoning in prior MLLMs by introducing a hybrid autoregressive mechanism capable of flexible, adaptive cognitive substrate selection, thereby improving both symbolic logical inference and perceptual scene understanding across a diverse set of multimodal tasks (Tong et al., 5 Feb 2026).

1. Motivation and Limitations of Fixed-Pattern Multimodal Reasoning

Traditional multimodal chain-of-thought (CoT) methods applied to MLLMs enforce one of three static reasoning templates for any given query: text-only, vision-only, or interleaved text ↔ vision. This inflexibility yields suboptimal results across the heterogeneous landscape of multimodal tasks. Text-only CoT achieves strong performance on symbolic and logical problems but falters on tasks requiring detailed visual reasoning—such as maze navigation or fine-grained spatial search—because compressing intermediate visual states into text induces brittle, lossy, and error-prone intermediate representations. Conversely, applying latent visual reasoning (i.e., inserting fixed spans of continuous “visual thoughts”) uniformly across queries can impair logical fidelity for tasks best handled by pure language. The lack of adaptive mode selection underscores heterogeneous demands: some tasks are innately symbolic, others purely perceptual, and many necessitate a nuanced, bidirectional interplay between vision and language. The SwimBird approach is motivated by the need to overcome this fixed-pattern modality rigidity, enabling query-adaptive selection of reasoning substrates for precise, robust multimodal cognition (Tong et al., 5 Feb 2026).

2. SwimBird Architecture and Mode-Switching Paradigm

SwimBird is initialized from a frozen vision encoder and a LLM (e.g., Qwen3-VL-8B) and augmented with mode-switching capabilities, implemented through distinct delimiters in its autoregressive generation. It supports three mutually exclusive reasoning modes:

• Text-Only Reasoning: Textual tokens are emitted within <reason>…</reason>; the model relies solely on next-token prediction, optimal for purely logical or language-dense tasks.
• Vision-Only Reasoning: The model emits a dynamically chosen span of continuous hidden state embeddings, demarcated as <|latent_start|> z₁, z₂, … <|latent_end|>, representing “visual thoughts.” This is suited for perceptual queries best addressed through non-linguistic internal representations.
• Interleaved Vision–Text Reasoning: The model alternates emission of <reason> and <|latent|> spans, adaptively grounding symbolic and perceptual steps as needed. This dynamic interplay targets queries requiring both modalities in sequence or alternation (e.g., localized image reading followed by comparison).

SwimBird’s mode selection is not manually prescribed but fully learned, with the model autoregressively deciding which reasoning modality to activate at every inference step, contingent on the evolving context and task structure.

3. Hybrid Autoregressive Objective and Training Formulation

SwimBird’s core learning procedure is a hybrid autoregressive objective that unifies sequence modeling for both discrete text and continuous visual clues:

• Textual Thought Prediction: For a span of $T$ reasoning words $\{w_1, \ldots, w_T\}$, SwimBird estimates the next-token probability $P(w_t | w_{<t}, I)$ using standard cross-entropy:

$$L_{text} = - \sum_{t=1}^{T} \log P(w_t \mid w_{<t}, I)$$

• Visual Thought Prediction: For a latent span of $K$ continuous intermediate embeddings $\{z_1, \ldots, z_K\}$, the model outputs $\hat{z}_k = f_\theta(z_{<k}, w_{\leq T}, I)$ and is optimized using mean squared error (MSE):

$L_{vis} = \sum_{k=1}^K \| \hat{z}_k - z_k \|_2^2$

• Composite Loss: When samples mix or alternate modal spans, the overall loss is a weighted sum of the two:

$$L = \lambda_{text} L_{text} + \lambda_{vis} L_{vis}$$

Here, $\lambda_{text}$ and $\{w_1, \ldots, w_T\}$0 balance the ratio, enabling the model to learn all three styles without preference-biasing unused patterns.

4. Reasoning-Mode Curation and SwimBird-SFT-92K Dataset

Effective adaptive reasoning necessitates a diverse, accurately labeled training corpus aligning each instance with its most appropriate modality. SwimBird-SFT-92K (approximately 92,000 instances) is a purpose-built dataset constructed as follows:

• Stage 1: Multimodal CoTs are sourced and filtered from Zebra-CoT, ThinkMorph, and MathCanvas. Instances solvable by the base model without hints are omitted.
• Stage 2: Reasoning dependency is empirically assessed using pass@8 statistics:
  • If introducing intermediate images increases pass@8 to accuracy $\{w_1, \ldots, w_T\}$1 without contemporaneous textual hints, the instance is labeled vision-only (8.8K samples).
  • If images help (pass_hint $\{w_1, \ldots, w_T\}$2 pass_base) but accuracy remains $\{w_1, \ldots, w_T\}$3, the instance is categorized as interleaved (33.5K).
• Stage 3: 50K pass@8-filtered text-only chain-of-thoughts are sourced from OpenMMReasoner.

The resulting dataset comprises approximately 50K text-only, 8.8K vision-only, and 33.5K interleaved samples. Coverage includes visual search, spatial navigation, geometric reasoning, mathematical problem solving, and general VQA.

Training Stage	Source Dataset(s)	No. of Instances
Stage 1	Zebra-CoT, ThinkMorph, MathCanvas	Filtered
Stage 2	N/A (mode labeled via pass@8)	8.8K/33.5K
Stage 3	OpenMMReasoner	50K

5. Benchmark Evaluation and Empirical Analysis

SwimBird has been evaluated on an array of vision-intensive, mathematical, and general multimodal benchmarks:

• Vision-Dense/Perception: V* Bench, HR-Bench 4K/8K, MME-RealWorld
• General VQA/Multimodal Reasoning: MMStar, RealWorldQA
• Multimodal Math: WeMath, DynaMath, MathVerse_MINI

Comparative performance against fixed-pattern models (e.g., Qwen3-VL-8B-Instruct) is robust:

Benchmark	SwimBird	Baseline (Fixed-Pattern)
V* Bench	85.5	83.8
HR-Bench 4K/8K	79.0/74.9	76.5/71.3
MMStar	71.2	64.7
RealWorldQA	73.1	71.8
WeMath	49.5	38.8
DynaMath	67.2	65.3
MathVerse_MINI	65.8	61.3

Ablation experiments reveal that a latent token budget $\{w_1, \ldots, w_T\}$4 of 32 achieves optimal high-resolution perception, with lower values degrading visual reasoning and higher values causing redundancy. The MSE loss weight $\{w_1, \ldots, w_T\}$5 is best set to 0.2; lower values diminish visual supervision, while higher values bias the model away from textual logic.

6. Flexible Reasoning and Prospective Extensions

SwimBird exhibits the capacity for on-the-fly modality selection tuned to the demands of each query: restricting output to text for logic-centric tasks, selecting vision-only when perceptual reasoning suffices, and interleaving modes for complex tasks requiring iterative grounding between modalities. This mitigates the risk of modality mismatch and yields increased robustness across domains. Furthermore, SwimBird dynamically adjusts the token budget for latent visual reasoning, emitting more tokens for high-resolution or information-dense visual queries and fewer for simplified cases, negotiating a tradeoff between computational efficiency and perceptual acuity.

Query-adaptive, hybrid autoregressive reasoning offers a promising paradigm for the development of general-purpose MLLMs capable of contextual reasoning substrate selection. Prospective research vectors include expansion to finer-grained mode taxonomies (e.g., sketch reasoning), integration with tool-usage policies, reinforcement learning approaches for more sophisticated mode selection, and extension to video or real-time interactive settings (Tong et al., 5 Feb 2026).