FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
Abstract: A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio LLM with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-LLM to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
First 10 authors:
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
This paper introduces FireRedAudio, an artificial intelligence model that can work with audio in many different ways. For example, it can:
- Turn speech into written words.
- Answer questions about speech, music, and other sounds.
- Create speech from written text.
- Copy a speaker’s voice after hearing a short example.
- Change what someone says or how they sound.
- Understand and organize recordings that last up to one hour.
The main idea is that understanding audio and creating audio are different jobs. FireRedAudio uses different ways of representing sound for each job, while still using the same main LLM.
2. What questions did the researchers want to answer?
The researchers focused on several main questions:
- Can one model both understand and generate audio well?
- Should audio be represented differently for understanding and for speech creation?
- Can continuous audio information produce more natural speech than systems that use compressed sound “tokens”?
- Can one model handle many tasks, such as speech recognition, text-to-speech, voice editing, and long recordings?
- Can the model keep track of when events happen in a long recording?
The most important question was whether using separate audio representations would help avoid a trade-off between understanding sounds accurately and recreating them naturally.
3. How did the researchers build and train the model?
Two different audio pathways
FireRedAudio uses two separate systems for turning sound into information.
The first is an Audio Encoder, which is mainly used for understanding. It is similar to a person taking notes: it keeps the important information, such as words, emotions, speakers, and background sounds, without saving every tiny detail of the original recording.
The second pathway uses RedAE, short for Red Audio Autoencoder. This system is mainly used for creating or editing speech. It keeps much more detailed information, including:
- The speaker’s voice quality, or timbre.
- Pitch.
- Speaking speed.
- Emotion.
- Pronunciation.
- Small details needed to rebuild the sound wave.
An autoencoder is like a machine that compresses a song into a smaller file and then tries to rebuild the original song. RedAE learns a compressed version of audio that can later be turned back into a waveform, which is the actual sound signal we hear.
These two audio pathways are not mixed together. Instead, they both send information to the same 9-billion-parameter LLM. A parameter is a value the model learns during training; more parameters generally give a model more ability to learn complex patterns.
The shared LLM
The shared LLM works like the model’s central brain. It receives:
- Audio information.
- Written instructions.
- Other text context.
For understanding tasks, it produces text one word at a time. For speech-generation tasks, it sends instructions to another system called a Diffusion Transformer, or DiT.
How speech is generated
The DiT begins with random noise and gradually changes it into meaningful speech information. This is somewhat like starting with a blurry, messy picture and repeatedly improving it until a clear image appears.
In this paper, the DiT creates continuous audio information instead of choosing from a limited list of sound pieces. The RedAE decoder then turns this information into a 24-kilohertz sound recording.
Training in stages
The model was trained gradually in five stages:
- Connecting audio to language: The model first learned to match audio with written transcripts.
- Improving audio understanding: It learned about speakers, emotions, music, noises, and timing.
- Learning many tasks together: It learned speech recognition, audio questions, text generation, and voice creation.
- Learning instructions: It learned to follow requests such as “speak more slowly” or “make the voice sound happier.”
- Learning long recordings: It was trained to understand audio lasting up to one hour.
This gradual process is similar to teaching a student basic skills before asking them to solve difficult, combined problems.
4. What did the researchers find?
Strong audio understanding
FireRedAudio was tested on two audio-understanding tests, MMAU and MMSU. These tests include questions about speech, music, environmental sounds, speakers, and speaking styles.
Its scores were:
| Test | FireRedAudio accuracy |
|---|---|
| MMAU test-mini | 82.0% |
| MMAU full test | 80.9% |
| MMSU | 83.3% |
FireRedAudio had the highest score among the systems listed in all three settings. This suggests that using a special representation for understanding helped the model notice important details in audio.
Good speech recognition in many languages
The model was also tested on automatic speech recognition, or ASR. ASR means turning spoken language into written text.
FireRedAudio performed especially well on:
- English speech from the LibriSpeech test-clean set.
- English speech in the FLEURS test.
- The average result across 102 languages in FLEURS.
On the 102-language test, it had an error rate of 14.94%. For error rates, lower is better because the number shows how often the system makes mistakes.
It also performed strongly on Chinese speech, meetings, dialects, and singing voices. This is important because real-world audio is often noisy, multilingual, or difficult to understand.
Natural and accurate voice creation
For zero-shot text-to-speech, the model creates speech in a voice it has only heard briefly before. The researchers measured:
- Whether the spoken words were correct.
- Whether the generated voice sounded like the example speaker.
FireRedAudio produced strong results in both Chinese and English. Its average word error score was 1.20%, while its speaker-similarity score was 0.71.
This means it was usually able to say the correct words while also keeping many features of the original speaker’s voice.
Following voice instructions
The model can follow instructions about how speech should sound, such as changing:
- Emotion.
- Volume.
- Speaking speed.
- Vocal style.
- Speaker characteristics.
The paper reports that FireRedAudio showed leading performance on instruction-following speech tests. This means it was better able to turn a written request into the desired style of speech.
Speech editing
FireRedAudio can also edit speech. For example, a user might ask it to:
- Replace certain words.
- Change the emotion.
- Make speech faster or slower.
- Change some acoustic details while preserving the speaker’s identity.
The researchers report substantial improvements over an earlier system called Ming-UniAudio-Edit in both meaning-based editing and sound-quality editing.
Understanding long recordings
FireRedAudio can analyze recordings lasting up to one hour. It can organize the recording into entries with timestamps, such as:
| Time | Event |
|---|---|
| 00:02 | Person A begins speaking |
| 00:18 | Music starts |
| 01:05 | Person B answers a question |
The model achieved timing accuracy within about one second in the reported tests. This could be useful for meetings, interviews, lectures, podcasts, and videos.
5. Why are these results important?
Many audio AI systems are good at only one or two tasks. One may recognize speech well but create unnatural voices. Another may generate realistic voices but struggle to understand long recordings.
FireRedAudio shows that a single model can perform many audio tasks while using different types of sound information for different purposes:
- A compact representation helps the model understand long and complicated recordings.
- A detailed, reconstructible representation helps it create realistic speech.
The results suggest that these two goals do not have to interfere with each other if the model keeps their audio pathways separate.
6. What could this research lead to?
FireRedAudio could eventually support tools such as:
- Automatic meeting notes and searchable recordings.
- Multilingual voice assistants.
- Audiobook and video narration.
- Voice-based accessibility tools.
- Dubbing and translation.
- Easier editing of podcasts and interviews.
- Personalized text-to-speech systems.
- Applications that understand emotion, music, and background sounds.
However, the paper mainly reports technical test results. More work would still be needed to study issues such as voice misuse, privacy, permission to copy someone’s voice, and whether the system works equally well for all languages and types of speakers.
In simple terms, the paper’s main message is: FireRedAudio works like one audio expert with two sets of ears—one set designed to understand sounds and another designed to recreate them. Sharing one language-model “brain” while keeping those two audio systems separate helps it perform many audio tasks effectively.
Practical Applications
Immediate Applications
The paper’s demonstrated capabilities—multilingual ASR, general audio understanding, long-form timestamping, zero-shot and instruction-controlled TTS, and semantic or acoustic speech editing—support the following applications that could be prototyped or deployed now, assuming access to suitable GPU infrastructure and the released implementation.
- Meeting, lecture, and interview transcription (enterprise productivity, education, research) Deploy FireRedAudio as a transcription service for meetings, interviews, lectures, podcasts, and customer calls. Its multilingual ASR and ability to process recordings of up to one hour can support searchable transcripts, speaker- and topic-oriented summaries, and question answering over recorded content. Dependencies: performance may vary with overlapping speakers, strong background noise, rare languages, dialects, and domain-specific terminology. Speaker diarization is not explicitly described and may require an external module.
- Timestamped long-form audio indexing (media, journalism, legal services, archives)
Use the model to convert long recordings into structured entries such as:
Second-level timestamp accuracy could enable clickable podcast chapters, evidence retrieval in legal recordings, archival cataloging, and navigation of lectures or public hearings. Dependencies: the reported timestamp accuracy is benchmark-dependent and should be validated for legal or compliance workflows. Boundary errors, incomplete transcripts, and uncertain attribution require human review.1 2 3
[00:12:34] Discussion of quarterly revenue [00:27:08] Witness describes the incident [00:44:51] Audience questions begin
- Audio search and content moderation (social media, broadcasting, platform operations) Integrate the model into audio pipelines that detect or retrieve spoken topics, environmental events, music-related content, and other acoustic evidence. A platform could index user-uploaded audio by transcript, topic, temporal location, speaking style, or sound event. Dependencies: the paper demonstrates general audio understanding on evaluation benchmarks but does not establish reliable safety classification, abuse detection, or production-scale false-positive rates. Moderation systems should use calibrated classifiers and human escalation.
- Multilingual accessibility tools (education, public services, assistive technology) Build live or offline captioning, multilingual transcription, lecture notes, and audio-to-text interfaces for users with hearing loss or users who speak different languages. The model’s FLEURS-102 evaluation suggests broad multilingual coverage, while its audio-understanding capabilities could support richer descriptions than transcription alone. Dependencies: language coverage and accuracy are uneven; accessibility products must report confidence, preserve timing, and provide correction interfaces. Deployment for medical, legal, or government communication requires domain validation.
- Podcast, video, and broadcast production workflows (entertainment and digital media) Producers could use ASR to generate subtitles, transcripts, chapters, highlight segments, and rough cuts. Long-form question answering can help locate quotes or scenes without manually reviewing the entire recording. Dependencies: post-production systems still need speaker identification, timecode alignment, profanity and name correction, and human editorial approval.
- Zero-shot voice adaptation for content creation (media, gaming, education, customer experience) Given a short reference recording, FireRedAudio can synthesize speech while attempting to preserve the reference speaker’s identity. Potential products include multilingual narration, prototype game dialogue, personalized educational content, and rapid voiceover generation without training a separate speaker model. Dependencies: the paper reports strong content accuracy and speaker preservation, but voice cloning requires explicit consent, identity verification, watermarking, and abuse monitoring. The quality and legality of reference recordings are important constraints.
- Instruction-controlled speech synthesis (assistive devices, virtual agents, localization)
Natural-language controls for vocal timbre, emotion, speaking rate, and volume can be exposed through a user interface or API:
This could support accessible reading tools, expressive game characters, audiobook prototyping, and customer-service agents. Dependencies: instruction adherence may be less predictable for combinations of attributes, languages, or extreme settings. Products should provide constrained presets and evaluate intelligibility and emotional appropriateness.1
“Read this announcement in a calm, warm voice, slightly slower than normal.”
- Speech correction and editorial editing (podcasting, advertising, journalism, education) The speech-editing pathway can rewrite semantic content or modify acoustic attributes such as delivery style, volume, or speaking rate. Editors could correct a mispronounced phrase, replace a sentence, remove an unwanted verbal segment, or make a recording more consistent without rerecording the speaker. Dependencies: FireRedAudio resynthesizes the complete target waveform rather than performing strictly local inpainting. This may alter surrounding prosody or background sound, so waveform comparison, human approval, and provenance labeling are needed.
- Audio-enabled research tools and reproducible baselines (academia and software engineering) Researchers can use the released code and model design as a baseline for unified audio-language experiments, including studies of continuous latent representations, multimodal instruction tuning, flow matching, long-context audio reasoning, and speech editing. Dependencies: reproducibility depends on access to the stated training data, hardware, checkpoints, and exact preprocessing. The paper reports aggregate benchmark results, so independent evaluation across more languages, speakers, and acoustic environments remains necessary.
- Personal audio organization (daily life and consumer software) A local or cloud application could summarize voice memos, search recordings by natural-language questions, extract action items, and generate short spoken reminders. Timestamped organization would allow users to jump directly to relevant portions of a recording. Dependencies: privacy is central because recordings may contain sensitive conversations. On-device inference may be difficult with a 9B-parameter backbone and a diffusion-based decoder; encrypted processing or a smaller distilled model may be required.
Long-Term Applications
The following applications are technically plausible but require additional research, validation, scaling, or safeguards before dependable deployment in high-stakes or fully autonomous settings.
- Clinical documentation and patient-accessible recordings (healthcare) FireRedAudio could transcribe consultations, summarize clinician–patient conversations, locate symptoms or treatment instructions, and generate accessible spoken explanations. Its ability to reason over long recordings could help review consultations or therapy sessions. Dependencies: clinical deployment requires high accuracy for medical terminology, robust speaker attribution, privacy compliance, auditability, uncertainty estimates, and extensive evaluation across accents and clinical settings. The model should assist clinicians rather than independently make diagnoses or treatment decisions.
- Legal evidence discovery and deposition analysis (law and public policy) Long-form transcription, timestamped retrieval, and question answering could accelerate discovery in hearings, depositions, police interviews, and public inquiries. Systems could identify relevant passages and organize evidence by topic or chronology. Dependencies: legal use requires demonstrable chain of custody, verbatim accuracy, preservation of uncertainty, protection against fabricated answers, and attorney verification. Second-level timestamp accuracy alone is insufficient for evidentiary use.
- Real-time multilingual interpreting and public-service communication (government, travel, emergency response) A unified model could combine speech recognition, reasoning, translation through a language-model interface, and speech synthesis to provide multilingual public announcements or conversational assistance. Dependencies: the paper evaluates ASR and TTS separately rather than end-to-end interpreting. Latency, turn-taking, translation fidelity, emergency terminology, network availability, and failure behavior require dedicated research.
- Robotic and embodied audio interaction (robotics, smart environments) Robots could use the understanding pathway to interpret spoken commands, environmental sounds, and conversational context, while using controllable TTS to respond with an appropriate voice, speed, or emotional style. Long-context memory could support multi-step tasks across extended interactions. Dependencies: reliable robotics requires low-latency streaming, wake-word handling, spatial audio, noise robustness, grounding in sensor and world state, and safe action policies. The current architecture is designed primarily for recorded audio and does not establish real-time embodied control.
- Interactive voice agents with persistent conversational audio memory (software, finance, customer service) Businesses could build agents that transcribe calls, reason over prior audio, answer questions, and generate personalized spoken responses. In finance or insurance, this might support call summarization and agent assistance rather than autonomous advice. Dependencies: the 9B LLM and flow-matching DiT may impose substantial inference cost and latency. High-stakes sectors also require authentication, consent recording, policy enforcement, secure data retention, and resistance to prompt injection or voice-based fraud.
- Personalized educational tutors and language-learning systems (education) The model could analyze pronunciation, speaking rate, prosody, and fluency, then generate targeted spoken feedback or examples in a learner’s preferred style. Long-form lecture analysis could produce individualized study materials. Dependencies: assessments of phonology and speaking style must be validated pedagogically and across learner populations. The model’s judgments should not be treated as definitive measures of ability, and generated feedback must avoid reinforcing accent or demographic bias.
- High-fidelity localization and synthetic media production (film, games, advertising) Instruction TTS and speech editing could enable multilingual dubbing, consistent character voices, rapid script changes, and localized emotional delivery while preserving a performer’s vocal identity. Dependencies: production quality requires better control of lip synchronization, dialogue context, background ambience, music mixing, and cross-language prosody. Performer consent, compensation, licensing, and provenance tracking are essential.
- Voice restoration and historical-audio preservation (cultural heritage, archives) Continuous-latent reconstruction could support denoising, controlled speech restoration, style-preserving rerecording, and searchable analysis of damaged historical recordings. The same model could describe environmental or musical content in archival collections. Dependencies: restoration must distinguish authentic signal from model-generated material. Archival workflows need reversible processing, uncertainty visualization, source preservation, and expert review to prevent historical alteration.
- Unified multimodal creative software (audio engineering and content tools) The architecture could evolve into a single application where users can ask questions about a recording, edit its words, modify its delivery, and generate new speech through one conversational interface. The decoupled pathways are particularly suitable for separating analysis from waveform reconstruction in such a workflow. Dependencies: robust production tools require non-destructive editing, exact segment control, deterministic or reproducible rendering, batch processing, plugin integration, and improved control over acoustic context. Full-waveform resynthesis may currently be unsuitable for precision editing.
- Energy-efficient edge deployment (mobile devices, IoT, automotive systems) A compressed or distilled version could provide offline transcription, voice interfaces, and environmental-audio understanding in phones, vehicles, hearing devices, or smart appliances. This would reduce cloud dependence and improve privacy. Dependencies: the reported 9B-parameter model, long-context processing, and diffusion-based generation are resource-intensive. Quantization, pruning, distillation, streaming architectures, and specialized hardware would be needed, with careful measurement of quality degradation.
- Standardized policy and governance infrastructure for synthetic voice (public policy and platform regulation) FireRedAudio’s combination of zero-shot cloning, natural-language voice control, and editing illustrates the need for technical governance tools: consent registries, speaker-identity verification, synthetic-audio detectors, provenance metadata, and mandatory labeling of generated or edited speech. Dependencies: detection alone is unlikely to remain reliable as generation improves. Effective governance will require cooperation among model providers, platforms, rights holders, regulators, and users, along with clear rules for biometric data and voice likeness.
- Research into representation-specialized unified models (academic machine learning) The decoupled Audio Encoder and RedAE pathways provide a testable design pattern for other multimodal systems: use task-optimized continuous representations for perception and reconstructible continuous latents for generation while retaining a shared language backbone. Future work could apply this principle to music, video, tactile sensing, or robotics. Dependencies: broader generalization is not established by the current experiments. Further ablations should measure representation interference, scaling behavior, latency, robustness, multilingual fairness, and whether the benefits persist at smaller or larger model sizes.
Glossary
- Acoustic conditioning: Information about local acoustic context used to guide speech generation. “The acoustic conditioning comprises eight historical RedAE latent frames from the two preceding audio steps”
- Acoustic–linguistic alignment: Correspondence between sound patterns and linguistic content. “thereby leveraging pretrained multilingual speech representations and acoustic--linguistic alignment priors”
- Adversarial loss: A training loss in which a generator is optimized to produce outputs that a discriminator cannot distinguish from real data. “Here, $\mathcal{L}_{\mathrm{rec}$, $\mathcal{L}_{\mathrm{adv}$, and $\mathcal{L}_{\mathrm{fm}$ denote the reconstruction loss, adversarial loss, and discriminator feature-matching loss, respectively.”
- Autoregressive modeling: Sequential prediction in which each output is generated based on previously generated elements. “This division allows the LLM to model sequence-level context while the DiT handles fine-grained continuous acoustics.”
- Causal architecture: A model design in which each output depends only on current or preceding inputs. “As shown in Figure~\ref{fig:redae_architecture}, the RedAE Encoder adopts a causal architecture.”
- Classifier-free guidance: A diffusion-generation technique that steers outputs using conditional and unconditional predictions without a separate classifier. “During training, we randomly drop the LLM conditioning to enable classifier-free guidance~\cite{ho2022classifier} while always retaining the acoustic conditioning.”
- Codebook: A finite collection of learned vectors used to quantize continuous representations into discrete codes. “Nevertheless, these methods still rely on quantized, codebook-based targets for audio generation”
- Continuous latent: A real-valued hidden representation from which an observable signal can be reconstructed. “Another line of work avoids quantized audio generation targets by directly modeling continuous latents that can be decoded back into waveforms.”
- Cross-entropy: A loss function that measures the discrepancy between a target probability distribution and a model’s predicted distribution. “The model is jointly optimized with cross-entropy over text tokens and a flow-matching loss for continuous-latent speech generation.”
- Cross-modal alignment: The process of mapping representations from different modalities into compatible spaces. “Isolating the additional temporal compression and cross-modal alignment in the Audio Adapter”
- Distillation loss: A loss that trains one model to reproduce information from another, usually larger or pretrained, model. “We therefore use a distillation loss to align the RedAE latents with high-level features from a frozen teacher Audio Encoder”
- Discrete audio token: A symbolic, quantized representation of audio used as a sequence-modeling target. “LongCat-Next~\cite{team2026longcat} represents both input and output audio as discrete tokens.”
- Diffusion Transformer (DiT): A Transformer architecture adapted to denoising or diffusion-based generative modeling. “The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents.”
- Element-wise addition: The operation of adding corresponding components of two vectors or tensors. “Kimi-Audio~\cite{ding2025kimi} combines continuous Whisper~\cite{radford2023robust} features with embeddings of discrete semantic tokens through element-wise addition.”
- Flow matching: A generative-training method that learns a vector field transporting noise to data samples. “We train the DiT with flow-matching~\cite{lipman2022flow} to learn a conditional velocity field from Gaussian noise to the target RedAE latents”
- Flow-matching loss: The regression objective used to train a model to predict the velocity along a prescribed probability path. “The flow-matching loss is”
- Frozen module: A neural-network component whose parameters are held fixed during training. “During FireRedAudio training, the pretrained RedAE Encoder is frozen”
- GAN-based objective: A training objective involving a generator and discriminator in an adversarial framework. “Following the X-Codec framework~\cite{ye2025codec}, RedAE is optimized with a hybrid GAN-based objective.”
- Inverse short-time Fourier transform (iSTFT): The operation that reconstructs a waveform from a sequence of short-time Fourier representations. “followed by a causal Qwen3 Transformer and an inverse short-time Fourier transform (iSTFT) head for waveform reconstruction”
- Kullback–Leibler (KL) regularization: A penalty that encourages a learned probability distribution to remain close to a reference distribution. “without variational posterior modeling or Kullback--Leibler (KL) regularization”
- Latent posterior: The probability distribution over hidden representations inferred from an observed input. “Because RedAE does not parameterize a latent posterior”
- Log-Mel feature: A time–frequency audio feature based on logarithmic energy in Mel-scaled frequency bands. “The understanding pathway therefore receives a mono waveform and extracts log-Mel features.”
- Long-context modeling: Modeling that preserves and uses dependencies across very long input sequences. “Understanding benefits from compact representations that organize task-relevant acoustic cues for long-context modeling”
- Masked discrete diffusion: A diffusion process that iteratively predicts or reconstructs masked discrete tokens. “UAT~\cite{wang2026uat} combines diffusion over continuous audio latents with masked discrete diffusion for text in a shared dual-stream backbone.”
- Mean-squared error (MSE): The average squared difference between predicted and target values. “The term $\mathcal{L}_{\mathrm{distill}$ is a mean-squared-error (MSE) semantic distillation loss”
- Mel scale: A perceptual frequency scale designed to approximate human judgments of pitch differences. “The understanding pathway therefore receives a mono waveform and extracts log-Mel features.”
- Multimodal token: A text token or continuous audio representation processed as an input to the LLM. “We use the term multimodal token to refer collectively to a text token or a continuous audio representation entering the LLM.”
- Non-overlapping window: A segmentation interval that does not share data with adjacent intervals. “the log-Mel sequence is first partitioned into non-overlapping windows of up to 30 seconds.”
- Paralinguistic information: Vocal information beyond literal linguistic content, such as emotion, emphasis, or speaking style. “A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information”
- Perceptual representation: A learned feature representation intended to preserve information relevant to interpreting sensory input. “The Audio Encoder and Audio Adapter into continuous perceptual representations at ”
- Probability path: A time-indexed sequence of probability distributions connecting a source distribution to a target distribution. “use the velocity of the corresponding probability path as the regression target.”
- Progressive multitask training: A staged training procedure that gradually introduces tasks and trainable components. “Through progressive multitask training, FireRedAudio supports ASR and audio understanding”
- Quantization: The conversion of continuous values into a finite set of discrete values or codes. “Nevertheless, these methods still rely on quantized, codebook-based targets for audio generation”
- RedAE latent: A continuous hidden representation produced by the Red Audio Autoencoder. “The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents.”
- Semantic distillation: Distillation that transfers high-level meaning or content information from a teacher representation to a student representation. “instead incorporates semantic distillation”
- Semantic module: A component that extracts or transforms representations associated with high-level meaning. “Ming-UniAudio~\cite{yan2025ming}, which appends a separate semantic module after the reconstructible latent representation”
- Short-time Fourier transform (STFT): A time-localized frequency analysis that represents a signal using Fourier transforms over overlapping windows. “an inverse short-time Fourier transform (iSTFT) head for waveform reconstruction”
- Temporal compression: Reduction of the number of time-step representations while retaining relevant information. “A separate Audio Adapter performs additional temporal compression”
- Temporal aggregation: Combining representations across adjacent time steps to form a lower-rate representation. “The second module aggregates each pair of adjacent frames”
- Temporal localization: Identifying when a particular event or content occurs in an audio recording. “FireRedAudio is trained for global summarization, temporal and content localization”
- Thinker–Talker decomposition: An architecture separating audio understanding and speech-generation functions into different model components. “Qwen3.5-Omni~\cite{team2026qwen3} and Qwen3-Omni~\cite{xu2025qwen3} instead adopt a Thinker--Talker decomposition”
- Variational autoencoder (VAE): A generative autoencoder that represents inputs with a probability distribution over latent variables. “Ming-UniAudio~\cite{yan2025ming} uses a semantic module to map low-dimensional latents produced by a variational autoencoder (VAE)”
- Velocity field: A function assigning a direction and speed to points in a continuous state space. “to learn a conditional velocity field from Gaussian noise to the target RedAE latents”
- Waveform reconstruction: Recovery of an audio signal from a compressed or latent representation. “while preserving waveform reconstructibility”
- Zero-shot text-to-speech (TTS): Speech synthesis that imitates a reference speaker without speaker-specific training or fine-tuning. “zero-shot TTS synthesizes target speech from text using reference speech”


