Continuous Audio Thinking (CoAT) Framework

• Continuous Audio Thinking (CoAT) is defined as the ability of audio models to preserve and reason over continuous acoustic data through either latent workspaces or structured audio embeddings.
• The framework employs expert distillation and specialized latent tokens to maintain detailed acoustic attributes such as pitch, timbre, and prosody, improving efficiency and accuracy.
• CoAT enables real-time, multi-turn, and streaming audio reasoning, supporting applications like controllable scene interpolation and tool-augmented waveform analysis.

Continuous Audio Thinking (CoAT) denotes a line of research in which an audio-capable model preserves, reorganizes, or reasons over acoustic information before or during response generation, rather than collapsing sound immediately into text-aligned outputs. The recent literature uses the term in related but non-identical ways. In one formulation, CoAT refers to compositional structure in audio representation spaces, where continuous transformations in sound scenes map to regular operations in embeddings. In another, it is a concrete framework that inserts a continuous latent thinking workspace into a Large Audio LLM (LALM) and supervises that workspace with audio-expert distillation. Closely related work extends the idea to explicit chain-of-thought, tool-augmented waveform analysis, long-form temporal reasoning, and real-time think-while-listening or think-while-speaking interaction (Chen et al., 14 Mar 2026, Han et al., 5 Jun 2026, Xiong et al., 26 Sep 2025, Shao et al., 24 Apr 2026).

1. Conceptual scope and competing formulations

The most explicit formalization of CoAT as a named framework appears in "Continuous Audio Thinking for Large Audio LLMs" (Han et al., 5 Jun 2026). There, the central claim is that conventional LALMs are trained to emit text, so their hidden states progressively drift toward text generation and away from preserving phonetic detail, prosody, affect, pitch, and sound events. CoAT addresses this by inserting a continuous latent workspace between the user input and the assistant response, then grounding that workspace with distillation from multiple frozen audio experts. The key design point is that the workspace is consumed during a single prefill, so it adds no extra autoregressive decoding cost.

A distinct but compatible use of the term appears in "Evaluating Compositional Structure in Audio Representations" (Chen et al., 14 Mar 2026). In that work, “Continuous Audio Thinking” is not introduced as a formal method name. Instead, it is described as the capacity to reason continuously about audio scenes by mapping continuous transformations in the signal domain—such as adding sources, varying gains, pitch shifts, or repetition-rate changes—to regular, predictable operations in representation space. Under this interpretation, CoAT is diagnosed rather than trained: a representation supports CoAT to the extent that source addition induces stable translations in embedding space and scene embeddings are reconstructible from attribute-level primitives.

Several adjacent papers use different terminology while clearly occupying the same conceptual territory. Audio-Cogito emphasizes deep audio reasoning with explicit free-form chain-of-thought but does not claim streaming continuity (Li et al., 14 Apr 2026). Thinking-with-Sound keeps the waveform “live” during inference and alternates linguistic reasoning with audio-domain manipulation, which its authors describe as a concrete instantiation of the same core idea, although the term CoAT is absent from the paper itself (Xiong et al., 26 Sep 2025). LAT-Audio frames long-form temporal awareness as a progressive global-to-local reasoning process with Think-With-Audio Chain-of-Thought, operationalizing temporally grounded CoAT over minutes-long recordings (Shao et al., 24 Apr 2026). This suggests that CoAT is best treated as an umbrella concept spanning compositional representation structure, latent acoustic workspaces, explicit reasoning traces, tool use, and continuous interaction.

2. Compositional structure as a representation-theoretic form of CoAT

The benchmark in "Evaluating Compositional Structure in Audio Representations" defines audio scenes as sets of sources $X=\{s_1,\dots,s_N\}$, where each source is parameterized by timbre, pitch, repetition rate, and amplitude, $s_n=[t_n,p_n,r_n,a_n]$, with each attribute discretized into $K=8$ classes (Chen et al., 14 Mar 2026). The paper proposes two tasks. A-COAT tests additive consistency under source mixing, and A-TRE tests whether an embedding can be reconstructed from primitive attribute tokens by a small composition model.

A-COAT constructs quadruples $(A,B,C,D)$ with a transformation set $T$ such that $B=A\cup T$ and $D=C\cup T$. For an encoder $f$ with embedding $z_X=f(X)$, the score is the cosine similarity between two difference vectors:

$$\mathrm{A\text{-}COAT}(A,B,C,D)= \frac{\langle z_B-z_A,\; z_D-z_C\rangle} {\|z_B-z_A\|\cdot\|z_D-z_C\|}.$$

High similarity means that adding the same sources acts like a consistent translation across contexts. The paper links this to the desiderata $s_n=[t_n,p_n,r_n,a_n]$0 and, more generally, $s_n=[t_n,p_n,r_n,a_n]$1.

A-TRE uses learnable primitive token vectors $s_n=[t_n,p_n,r_n,a_n]$2 for attribute classes $s_n=[t_n,p_n,r_n,a_n]$3. A source embedding is formed additively,

$s_n=[t_n,p_n,r_n,a_n]$4

and the set of source embeddings is processed by a single-layer Transformer encoder with a learnable $s_n=[t_n,p_n,r_n,a_n]$5 token to predict a scene embedding $s_n=[t_n,p_n,r_n,a_n]$6. The score is

$s_n=[t_n,p_n,r_n,a_n]$7

Where A-COAT probes algebraic consistency of transformations, A-TRE probes reconstructibility from attribute-level primitives.

To ensure oracle access to attributes, the benchmark uses synthetic 10 s clips at 32 kHz synthesized with learnfm’s DX7 FM synthesizer. Timbre is drawn from eight curated FM patches; pitch is binned from MIDI 36–84; rate from 0.2–3.0 Hz; and amplitude from $s_n=[t_n,p_n,r_n,a_n]$8 to $s_n=[t_n,p_n,r_n,a_n]$9 dB. Candidate pools contain 50,000 A-COAT quadruples and 150,000 A-TRE scenes, then entropy-balanced subsampling yields 2,000 quadruples for A-COAT and 10,000 scenes for A-TRE, split into 8,000 train, 1,000 validation, and 1,000 test.

The reported results establish a sharp separation between linear signal superposition and semantically structured composition. The Downsample baseline attains A-COAT $K=8$0 but A-TRE $K=8$1, showing that perfect additive consistency can arise trivially from signal linearity without meaningful attribute structure. Random embeddings score $K=8$2 on both tasks. Among pretrained encoders, AudioMAE and BEATs are strongest overall, with AudioMAE at A-COAT $K=8$3 and A-TRE $K=8$4, and BEATs at $K=8$5 and $K=8$6. Whisper reaches A-TRE $K=8$7 but a lower A-COAT of $K=8$8. CLAP scores $K=8$9 on A-COAT and $(A,B,C,D)$0 on A-TRE. PANNs and PaSST are weaker on additive consistency, at $(A,B,C,D)$1 and $(A,B,C,D)$2 respectively. The paper also reports that A-COAT usually declines as quadruple diversity increases, except for BEATs, whose positive slope suggests robustness to attribute diversity.

Within this benchmark lineage, CoAT is therefore an algebraic property of representation spaces: continuous changes in a scene should induce stable and interpretable changes in embeddings. A plausible implication is that such structure is a prerequisite for controllable scene interpolation, source addition and removal, and attribute-aware analysis, but the benchmark itself is careful to present these tasks as diagnostics rather than a full interactive reasoning system.

3. Continuous latent workspaces and expert-grounded acoustic memory

The CoAT framework in "Continuous Audio Thinking for Large Audio LLMs" introduces a deterministic thinking block of continuous tokens placed after the user input and before the assistant response (Han et al., 5 Jun 2026). If the frozen audio encoder produces $(A,B,C,D)$3 audio tokens at rate $(A,B,C,D)$4 Hz, the model extends the vocabulary with three special tokens $(A,B,C,D)$5, $(A,B,C,D)$6, and $(A,B,C,D)$7, and constructs

$(A,B,C,D)$8

with one $(A,B,C,D)$9 per audio token. The full prefix becomes $T$0. The final-layer hidden states at the $T$1 positions define the continuous thinking workspace $T$2.

The framework supervises $T$3 using frozen audio experts and lightweight projector heads. The experts cover both representational and task-specific dimensions: the backbone’s own audio tower or Sim-Whisper codec features for reconstruction, SPIDR for speech representation, PANNs for sound event detection, emotion2vec for paralinguistic information, and basic_pitch for pitch and harmonic structure. The losses are expert-specific: MSE for reconstruction, SPIDR, and emotion2vec; BCE in PANNs’ class-logit space; and a combined MSE plus focal-BCE for basic_pitch. Training proceeds in two stages: a reconstruction-only warm-up, then joint multi-expert distillation. The audio tower and expert encoders are frozen; learnable parameters are the LM LoRA adapters, projector heads, and embeddings for the three special tokens.

This formulation is motivated by what the paper calls supervision mismatch and hidden-state drift. The audio input contains rich frame-level structure, but the standard next-token objective only supervises assistant text tokens. CoAT’s central claim is that continuous latent thinking retains acoustic information more faithfully than textual chain-of-thought, which introduces an additional serialization bottleneck and extra autoregressive latency. Inference uses a single prefill: the thinking block is present, but the model does not decode extra reasoning tokens.

Empirically, the method is evaluated on Qwen2-Audio, Qwen2.5-Omni-7B, and Audio Flamingo 3. On Qwen2-Audio, MMAU improves from 52.50 to 66.90, MMAR from 47.10 to 52.60, and ClothoAQA from 75.52 to 79.96. On MELD, accuracy and class-weighted F1 rise from 25.5 and 21.8 to 58.0 and 56.1, and IEMOCAP improves from 54.00 to 72.70. The same model’s LibriSpeech clean WER drops from 4.14 to 2.30 and Switchboard from 19.06 to 9.25. On Qwen2.5-Omni-7B, MMAU improves from 65.60 to 69.40 and MMAR from 56.70 to 62.20, while IEMOCAP rises from 55.22 to 79.01. Audio Flamingo 3 improves MMAR from 55.70 to 59.60 and ClothoAQA from 80.10 to 85.30.

The efficiency argument is a major part of the framework. For Qwen2.5-Omni, text-CoT approximately doubles total latency at $T$4 and decodes about $T$5 more tokens, whereas CoAT raises total latency by only $T$6, with TTFT within about 10 ms of the baseline. For Audio Flamingo 3, text-CoT increases total latency by $T$7, while CoAT yields a $T$8 increase. The paper presents this as a direct advantage of continuous latent reasoning over discrete textual reasoning.

The broader significance of this work is that it turns CoAT into an architectural prior. Rather than asking whether a representation already exhibits continuous structure, it inserts a dedicated acoustic workspace and trains the model to preserve multiple expert views of the signal. This suggests a version of CoAT in which “thinking” is neither textual nor external-tool mediated, but an internal, expert-grounded latent buffer that conditions the eventual response.

4. Discrete chain-of-thought, tool use, reinforcement learning, and multi-turn audio state

A second major research line treats CoAT as explicit reasoning over audio, usually in text space. "Audio-CoT" is the first systematic exploration of chain-of-thought prompting for LALMs, using Qwen2-Audio-7B-Instruct and MMAU (Ma et al., 13 Jan 2025). It compares Manual-CoT, Zero-Shot-CoT, Desp-CoT, and self-consistency with $T$9. The best prompted configuration, Zero-Shot-CoT, reaches 57.80% total accuracy, versus 55.60% for the normalized baseline, and self-consistency increases this to 58.10%. The study finds that CoT improves easy and medium tasks but can hurt hard tasks, where longer chains may confuse the model rather than resolve ambiguity. It also reports a positive qualitative correlation between reasoning path length and accuracy.

Supervised fine-tuning scales this textual reasoning paradigm substantially. "Audio-Reasoner" trains on CoTA, a 1.2 million-sample reasoning-rich corpus spanning speech, music, and environmental sound, with an explicitly structured hidden channel: <PLANNING>, <[CAPTION](https://www.emergentmind.com/topics/caption)>, <REASONING>, and <SUMMARY> (Xie et al., 4 Mar 2025). The model achieves 61.71% on MMAU-mini versus 49.20% for Qwen2-Audio-Instruct, 7.94 on AIR-Bench chat versus 6.93, 65.2 on AIR-Bench foundation versus 59.2, and 53.9 on MELD versus 49.9. "Audio-Cogito" uses a different recipe—545k reasoning samples built by a four-stage Cogito-Pipe plus self-distillation from Qwen3-Omni-Thinking—and reports MMAR average accuracy 71.70%, rubrics score 62.22%, and Correct Reasoning Score 0.87 (Li et al., 14 Apr 2026).

A third branch replaces supervised rationale imitation with explicit reward shaping. "Audio-DeepThinker" fine-tunes Qwen3-Omni-30B-A3B-Instruct using pure RL exploration and a hybrid reasoning similarity reward combining an LLM evaluator with embedding similarity to reference reasoning chains (He et al., 20 Apr 2026). Stage 1 trains on 39,412 AVQA samples with answer correctness, format, consistency, and hybrid similarity; Stage 2 uses 29,483 acoustically challenging samples and switches to an LLM-only similarity reward. The resulting model reaches MMAR 74.0%, MMAU-test-mini 78.5%, and MMSU 77.26%, and the paper reports mechanistic evidence that RL primarily reshapes upper-layer MoE gating, with reasoning tokens crystallizing in upper transformer layers.

Tool use shifts the emphasis from text reasoning about a fixed audio encoding to reasoning with a live signal. "Thinking with Sound" introduces a training-free loop in which the model plans, decides whether to invoke an audio operator, applies it to the waveform, re-encodes the updated signal, and continues reasoning (Xiong et al., 26 Sep 2025). The operator suite includes denoising, enhancement, source separation, normalization, spectral and pitch analysis, and time-frequency transforms. On MELD-Hard1k, where all tested LALMs lose more than 50% accuracy under perturbations, TwS raises Qwen2.5-Omni 7B from 12.36% to 48.97%, Audio-Flamingo3 7B from 18.71% to 50.16%, and Voxtral 24B from 24.55% to 49.49%. The paper reports that denoising is the most critical operator in leave-one-out ablations and that most samples converge in 3–4 steps, with average steps around 2.8.

A multi-turn generative extension appears in "AudioChat", which frames audio stories as multi-source scenes that require understanding, editing, and generation within one system (Chen et al., 19 Feb 2026). Its Audio Transfusion Forcing objective combines causal language modeling for structured chain-of-thought with diffusion over continuous audio latents, while carrying previous reasoning chains as semantic memory across turns. The model uses a 3.6B Self-Cascaded Transformer, a continuous 48 kHz stereo tokenizer producing 40 Hz latents, and 6 million simulated conversations from AudioCopilot. Evaluation is performed with task-centric metrics such as multiFLAM, $B=A\cup T$0multiFLAM, and editFLAM, rather than only FAD or KAD. In this setting, CoAT becomes persistent planning over an evolving audio narrative rather than a single-turn recognition process.

Across these papers, explicit chain-of-thought, RL-emergent reasoning, waveform-in-the-loop tool use, and multi-turn audio generation all instantiate CoAT as sustained reasoning over acoustics. The mechanisms differ sharply—prompting, SFT, RL, tool calling, diffusion, or latent state carryover—but they converge on the claim that audio reasoning benefits from preserving intermediate acoustic structure instead of optimizing only for immediate answers.

5. Streaming, long-form, and full-duplex realizations

Long-form temporal awareness is a central pressure test for CoAT. "Listening with Time" introduces LAT-Chronicle, a 1.2k hour dataset with temporal annotations over audio up to 30 minutes, and LAT-Bench, a human-verified benchmark for Dense Audio Caption, Temporal Audio Grounding, and Targeted Audio Caption (Shao et al., 24 Apr 2026). LAT-Audio first constructs a coarse global timeline

$B=A\cup T$1

then performs iterative local reasoning via a single inference-time tool, crop_audio, under a Think-With-Audio Chain-of-Thought loop. The method reports Chinese and English TAG mIoU of 47.2 and 50.0, DAC average scores of 46.8 and 48.6, and TAC FENSE of 62.0 and 68.7, outperforming Gemini baselines on these long-form temporal tasks. The paper explicitly treats this progressive global-to-local paradigm as an operationalization of temporally grounded CoAT.

Real-time spoken interaction introduces a different constraint: reasoning must not block response onset. "STITCH" addresses this by alternating silent reasoning chunks with spoken response chunks inside a spoken LLM based on GLM-4-Voice-9B (Chiang et al., 21 Jul 2025). The system uses fixed chunk sizes $B=A\cup T$2, $B=A\cup T$3, and typically $B=A\cup T$4, exploiting the fact that audio playback time is longer than the time required to generate the corresponding tokens. On math QA, the average accuracies are 79.12 for full think-before-speak, 78.70 for Stitch-R, 78.04 for Stitch-S, 67.41 for a mix-reasoning model without CoT at inference, and 62.98 for a no-reasoning fine-tuned model. Stitch-S matches baseline first-chunk latency while preserving most of the reasoning gain.

"Thinking-while-speaking" in InterRS adopts a more explicitly controlled interleaving strategy (Du et al., 20 May 2026). A response is represented as $B=A\cup T$5, where $B=A\cup T$6 is a thinking segment and $B=A\cup T$7 is the corresponding spoken segment. The method enforces a global thinking:answer ratio near 4:1 and a local masking constraint $B=A\cup T$8. Single-token flags <|thinking|> and <|answer|> control state transitions. On SATA-Bench, KK-audio, SMQA, and Meta-audio, InterRS reaches an average of 49.33 while maintaining instant response, versus 36.92 for a fast-CoT instant baseline and 51.10 for a non-instant thinking SFT+RL system. Fluency improves from 1.65 with TA-only reward to 1.83 in the full InterRS configuration.

"Can Speech LLMs Think while Listening?" moves the trigger earlier, so reasoning begins before the user has finished speaking (Shih et al., 8 Oct 2025). Using Moshi, Mimi codec tokens at 12.5 Hz, and text-only chain-of-thought on the system text stream, the paper reports an average accuracy gain of about 2.4× over a suite of spoken reasoning tasks after CoT fine-tuning. It introduces a question-completeness metric

$B=A\cup T$9

which estimates when enough of the question has been heard to start reasoning. Under equivalent latency, this yields a 4% accuracy gain on ARC-Easy relative to heuristic triggers. A subsequent DPO stage reportedly reduces latency by 70% without loss in accuracy.

"Mind-Paced Speaking" decomposes the same problem into two coordinated streams: a Formulation Brain that emits think segments and an Articulation Brain that emits speech segments, both instantiated with the same Step-Audio 2 backbone (Wu et al., 10 Oct 2025). With fixed segment sizes $D=C\cup T$0 think tokens and $D=C\cup T$1 response tokens, MPS-thkfirst reaches 93.9% on Spoken-MQA and MPS-spkfirst reaches 92.8% under a zero-latency configuration, compared with 93.0% for a think-before-speak version and 68.6% for Mini-Omni-Reasoner. The same paper reports zero extra tokens before the first response token for MPS-spkfirst, 80 for MPS-thkfirst, and 762 for a think-before-speak baseline.

At the full-duplex systems level, "DuplexOmni" separates a fast interaction layer from a slower, pluggable thinking layer that can stream tool results back asynchronously (Huang et al., 8 Jun 2026). The interaction layer operates in 480 ms slices, continuously ingesting audio and video, generating text and six codec frames of speech per slice, and issuing control signals to start or stop background reasoning. Reported performance includes 72.6% Turn-on-Rate on Full DuplexBench v1.5, 77.2% on Big Bench Audio, 53.8% on Daily-Omni, LibriSpeech WER 0.1192, and real-time latency 0.506 s. Here CoAT becomes a systems property: listening, speaking, and thinking proceed concurrently, and the reasoning substrate is explicitly non-blocking.

6. Limitations, misconceptions, and unresolved directions

One recurrent misconception is that adding chain-of-thought automatically yields stronger acoustic grounding. "Listening or Reading?" directly challenges that assumption in speech-to-text translation (Romero-Díaz et al., 3 Oct 2025). Using attribution over speech DSUs, transcript tokens, and previously generated tokens, the paper finds that CoT largely mirrors cascaded behavior and relies mainly on transcripts. Layer-averaged speech attribution is only $D=C\cup T$2 in the Base model, rising to $D=C\cup T$3 with Direct S2TT mixing and $D=C\cup T$4 with noisy transcript injection. Robustness and prosody awareness improve under these interventions, but the study concludes that mere access to speech tokens during CoT is insufficient; explicit training pressure is needed to force acoustic use.

A second misconception is that strong additive structure alone implies semantic understanding. The A-COAT benchmark makes the opposite point vividly: the Downsample baseline is nearly perfect on additive consistency yet poor on reconstructibility from attribute primitives (Chen et al., 14 Mar 2026). This is why that paper pairs A-COAT with A-TRE and emphasizes that signal linearity can satisfy translation consistency without encoding semantics or attributes.

The literature also documents substantial domain and realism gaps. The compositional benchmark uses FM-synth tones with discrete attributes rather than real mixtures with masking, reverberation, and spatial cues (Chen et al., 14 Mar 2026). Audio-Cogito does not claim streaming or online inference, and its proposed CoAT extensions—streaming encoders, chunk-level CoT, explicit memory, and long-context attention or state-space models—remain future work rather than implemented components (Li et al., 14 Apr 2026). Thinking-with-Sound keeps the waveform live but operates over fixed clips rather than online audio streams (Xiong et al., 26 Sep 2025). LAT-Audio is offline rather than streaming, even though its global timeline and crop-based refinement are explicitly presented as amenable to incremental extension (Shao et al., 24 Apr 2026). AudioChat supports persistent multi-turn audio reasoning, but its 150-step diffusion sampling is not real time (Chen et al., 19 Feb 2026).

Real-time systems add their own failure modes. STITCH reports that GSM8K remains the hardest benchmark and that very long reasoning may exceed the playback-time budget, forcing premature termination of a reasoning chunk (Chiang et al., 21 Jul 2025). InterRS identifies timing misalignment between the next thought segment and the current spoken segment as a direct cause of pauses (Du et al., 20 May 2026). MPS-spkfirst slightly degrades on arithmetic because the first response segment is generated without any think prefix (Wu et al., 10 Oct 2025). DuplexOmni achieves strong duplex interaction metrics, but its full-duplex ASR is substantially worse on very short utterances than some baselines, reflecting the difficulty of low-context streaming recognition (Huang et al., 8 Jun 2026).

The dominant research directions are therefore converging. Multiple papers call for stronger streaming encoders, chunk-level supervision, persistent reasoning memory, alignment between reasoning steps and acoustic evidence, and evaluation sets with long-form, multilingual, and real-world audio (Li et al., 14 Apr 2026, Shao et al., 24 Apr 2026, He et al., 20 Apr 2026). Others emphasize better uncertainty control over when to start reasoning and how long to continue it (Shih et al., 8 Oct 2025, Du et al., 20 May 2026). The latent-workspace program suggests a different route: preserving acoustic information continuously inside the model may reduce the need to verbalize or tool-call every intermediate state (Han et al., 5 Jun 2026). A plausible implication is that the field is moving toward hybrid designs in which continuous latent acoustic memory, selective external tools, and asynchronous interaction are combined rather than treated as competing alternatives.